Meta-Learning with Fewer Tasks through Task Interpolation
Authors: Huaxiu Yao
Linjun Zhang
Chelsea Finn
Venue: ICLR 2022 Oral (2022)
Keywords: meta-learning
task interpolation
meta-regularization

Abstract: Meta-learning enables algorithms to quickly learn a newly encountered task with just a few labeled examples by transferring previously learned knowledge. However, the bottleneck of current meta-learning algorithms is the requirement of a large number of meta-training tasks, which may not be accessible in real-world scenarios. To address the challenge that available tasks may not densely sample the space of tasks, we propose to augment the task set through interpolation. By meta-learning with task interpolation (MLTI), our approach effectively generates additional tasks by randomly sampling a pair of tasks and interpolating the corresponding features and labels. Under both gradient-based and metric-based meta-learning settings, our theoretical analysis shows MLTI corresponds to a data-adaptive meta-regularization and further improves the generalization. Empirically, in our experiments on eight datasets from diverse domains including image recognition, pose prediction, molecule property prediction, and medical image classification, we find that the proposed general MLTI framework is compatible with representative meta-learning algorithms and consistently outperforms other state-of-the-art strategies.

Published as a conference paper at ICLR 2022
META- WITH FEWER TASKS THROUGH TASK INTERPOLATION
Huaxiu Yao', Linjun Zhang?, Chelsea Finn'
1 Stanford University. "Rutgers University (huaxiu, cbfinn) @cs. stanforrdddd... faddu,?iing Plinjun. zhangerutgers.co edu
ABSTRACT
Meta-learning enables algorithms to quickly learn a newly encountered task with just few labeled examples by transferring previously learned knowledge. How- ever, the bottleneck of current meta-learning algorithms is the requirement of a large number of meta-training tasks, which may not be accessible in real-world scenarios. To address the challenge that available tasks may not densely sam- ple the space of tasks, we propose to augment the task set through interpolation. By meta- weta-learning with task interpolation (MLTI), our approach effectively gener- ates additional tasks by randomly sampling a pair of tasks and interpolating the corresponding features and labels. Under both gradient- based and metric-based meta-learning settings, our theoretical analysis shows MLTT corresponds to a data-adaptive meta-regularizztion and further improves the generalization. Empiri- cally, in our experiments on eight datasets from diverse domains including image recognition, pose prediction, molecule property prediction, and medical image classification, we find that the proposed general MLTI framework is compatible with representative meta- meta-learning algorithms and consistently outperforms other state-of the-art art strategies.
INTRODUCTION
Meta-learning has powered machine learning systems to learn new tasks with only a few examples, by learning how to learn across a set of meta-training tasks. While existing algorithms are remarkably efficient at adapting to new tasks at meta-test time, the meta-training process itself not efficient. Analogous to the training process in supervised learning, the meta-training process treats tasks as data samples and the superior performance of these meta-learning algorithms relies on having a large number of diverse meta-training tasks. However, sufficient meta-training tasks may not always be available in real-world. Take medical image classification as an example: due to concerns of privacy, it is impractical to collect large amounts of data from various diseases and construct the meta-training tasks. Under the faskinnsuffcient scenario, the meta-learner can easily memorize these meta-training tasks, limiting its generalization ability on the meta-testing tasks. To address this limitation, we aim to develop strategy to regularize meta-learning algorithms and improve their generalization when the meta-training tasks are limited and only sparsely cover the space of relevant tasks.
Recently, a variety of regularization methods for meta-learning have been proposed. including techniques that impose explicit regularization the meta-learning model (Jamal and Qi 2019) Yin
et al. 2020) and methods that augment tasks by making modifications to individual training tasks through noise (Lee al. 2020) or mixup (Ni et al. [021]] Yao al., (02T). However, these methods are largely designed to either tackle only the memorization problem (Yin et al. (020) or to improve
performance of meta-learning (Yao et al.) (20T)) when plenty of meta-training tasks are provided.
Instead, we aim to target the task distribution directly, leading to approach that is particularly well-suited to settings with limited meta veta-training tasks.
Concretely, as illustrated in Figure we aim to densify the task distribution by providing interpolated tasks across meta- eett-ttaiinng tasks, resulting in a new task interpolation algorithm named MLTI (Meta- Learning with Task Interpolation). The key idea behind MLTI is to generate tasks by interpolating between pairs of randomly sampled meta-tran meta-training tasks. This interpolation can be instantiated in a variety of ways, and we present two variants that we find to be particularly effective. The first label-

---

Published as a conference paper at ICLR 2022
-Tas Distribution
Samples of each Task
%
2
5 R1
R. F.
Fnn 5
(a) Standard Task Sampling
(b) Individual A Augmentation
(c) MLTI
Figure 1: Motivations behind MLTI. (a) three tasks are sampled from the task distribution; (b) individual augmentation methods (e.g.. (Ni et al. 2021] Yao et al. (02T) augment each task within its own distribution); (c) MLTI densifies the task-level distribution by performing cross-task interpolation.
sharing (LS) scenario includes tasks that share the same set of classes (e.g., RainbowMNIST (Finn) al., 2019)). ) For each LS task pair randomly drawn from the meta-training tasks, MLTI linearly interpolates their features and accordingly applies the same interpolation strategy on the corresponding labels. The second non-label-sharing (NLS) scenario includes classification tasks with different sets of classes (e.g.. minilmagenet). For each additional NLS task, we first randomly select two original meta-training tasks and then generate new classes by linearly interpolating the features of the sampled classes, which draw one class in each original task without replacement. Since MLTI is essentially changing only the tasks, it can be readily used with any meta-learning approach and can be combined with prior regularization techniques that target the model.
In summary, our primary contributions are: (1) We propose a new task augmentation method (MLTI) that densifies the task distribution by introducing additional tasks; (2) Theoretically, we prove that MLTI regularizes meta-learning algorithms and improves the generalization ability. (3) Empirically, in eight real-world datasets from various domains, MLTI consistently outperforms six prior meta- learning regularization methods and is compatible with six representative meta-learning algorithms.
2 PRELIMINARIES
Problem statement. In meta-learning, we assume each task T, is i.i.d. sampled from a task distribu- tion P(T) associated with dataset D., from which we i.i.d. sample support set Dj (X,, Y!) vY,
{(xf,.: Mf,,)) = and a query set 2 (X², Y (a.k.a., the base model) with parameter 6, meta- leeta-le algorithms first train the base model on meta-training tasks. Then, during the meta-testing stage, the well-trained base model f is applied to the new task T, with the help of its support set D' and finally evaluate the performance on the query set Dj In the rest of this section, will introduce both gradient-based and metric-based meta-learning algorithms. For simplicity, we omit the subscript of the meta-training task index in the rest of this section.
Given predictive model
Gradient-based meta-learning. In gradient-based meta-learning, we use model-agnostic meta- learning (MAML) (Finn and Levine 2018) as an example and denote the corresponding base model as AMA. L, Here, the goal of MAML is to learn initial parameters 6" such that one or a few gradient steps on D' leads to a model that performs well on task T. During the tertatrainnng stage, the performance of the adapted model fo is evaluated on the corresponding query set D and is used to optimize the model parameter e. Formally, the bi-level optimization process with expected risk is formulated as:
0" arg min ETpeT) [C MAM, :D)] where e 0 VVVLL :D"),
(1)
where 77 denotes the inner-loop learning rate and L is defined as the loss, which is formulated as cross- entropy loss (i.c., ET~p(T) |- Σ. log p(y [x). f&)] and mean square error (i.e., BT E. (x]] III for classification and regression problems, respectively. During the meta-testing stage, for task Ti, the adapted parameter is achieved by fine-tuning 0g on the support set DJ.
Metrie-based meta-learning. The aim of metric- meta-learning is to perform a non paramermttiic learner on the top of meta-learned embedding space. Taking prototypical network (ProtoNet) with base model f as an example (Snell et al., 2017). for each task T, we first compute class prototype representation { as the representation vector of the support samples belonging class k as C, N, Lxt 10 (xf,,), where Dj represents the subset of support samples labeled as
2

---

Published as a conference paper at ICLR 2022
class and the number of this subset is N,. Then, given query data sample x in the query set, the probability of assigning it to the r-th class is measured by the distance d between its representation So (xg) and prototype representation C,, and the cross-entropy loss of ProtoNet is formulated as:
log exp(-d(fo PN (x))),)) Σ exp( d (x cp))
C = ET-ECT
Σ ply - x = BT-p(T) A,r
(2)
R,r
At the meta-testing stage, the predicted label of each query samples is assigned to the class with maximal probability (i.e., 32 arg max, ply = =)xx))-
The estimation of the expected loss in Eqn. (I or (2) is challenging since the distribution p(7) is unknown in practical situations. A common way of estimation is to approximate the expected risk in Eqn.
[ by set of meta-training tasks (T:) 1 (use MAML as example):
0" +
arg min
LUMA D!), where d, = aVol MAMA (D)). CU
(3)
However, this approximation method still faces the challenge: optimizing Eqn. 33. as suggested in (Rajendran et al.] 2020; Yin et al.j (020). can result in memorization of the meta-training tasks, thus limiting the generalization of the meta-learning model to new tasks, especially in domains with limited meta-training tasks.
3 META-LEARNING WITH TASK INTERPOLATION
To address the memorization issue described in the last section, we aim to develop framework that allows meta- leta-leernnng methods to generalize well new few-shot learning tasks, even when the provided meta- meta-training training tasks are only sparsely sampled from the task distribution. To accomplish this, we introduce meta-learming with task interpolation (MLTT). The key idea behind MLTI is to densify the task distribution by generating new tasks that interpolate between provided meta-training tasks. This approach requires no additional task data or supervision, and can be combined with any base meta-learning algorithm, including MAML and ProtoNet
Before detailing the proposed strategy, we first discuss two scenarios of meta-training task dis- tributions, label-sharing and non-label-sharing tasks, which have distinct implications for task interpolation. Formally, we define these two scenarios as:
Definition 1 (label-sharing tasks) If the labels of all tasks share the same label space, we refer it as the sabbingharing (LS) scenario. Take Pascal3D pose prediction (Yin et al. (2200) as an example, each task is predict the current orientation of the object relative to its canonical orientation, and the range of canonical orientation is shared across all tasks.
Definition 2 (noh-igg tasks) The non-label-shariee (NLS) scenario assumes that different semantic meanings of labels across lasks. For example, the piano class in the minilmagenet dataset may correspond a class label of 0 for one lask and 1 for another lask.
MLTI for label-sharing tasks. First, we will discuss MLTT under the label-sharing scenario, where it applies the same interpolation strategy on both features/hidden representations and label spaces. Concretely, let's say that model ff consists of L layers and the hidden representation of samples X at the 1-th layer denoted as H' for (X) (0 <|S L*), where H X and L represents the methods, as suggested in (Yin et al., (020). only part of the layers are shared (i.e., L < L). In metric- based methods, all layers are shared (i.e.. L' = L). Given pair of tasks with their sampled support and query sets (i.e., T: = {D;, D3) } and T, = [Dj, D) under the same label space, MLTI first randomly selects one layer and then applies the task interpolation separately on the hidden representations (H!()).) H,(())) and corresponding
number of layers shared across all tasks. In gradient-
labels (x y of the support (query) sets as:
H.! A (1 A) Y= XY; + (1- (1-
(4)
ig.
TH (1 (1-A)H² A) Yg. = XY? + (1 (01-A)7 A)Y?
where À E [0, 1] is sampled from a Beta distribution Beta(a, B) and the subscript "cr" represents "cross". Notice that both the support and query sets will be replaced by the interpolated ones in
3

---

Published as a conference paper at ICLR 2022
MLTI, while only the query set is replaced in approaches like Yao et al. (2021). Besides, Manifold Mixup (Verma et al. 2019) in Eqn. (4) can be replaced by different task interpolation methods (e.g.,
Mixup Zhang et al. 2018). CutMix (Yun et al. (019)).
MLTI for non-label-sharing tasks. Under non-label-sharing scenarios, tasks have different label spaces, making it infeasible to directly interpolate the labels. Instead, generate the new task by performing the feature-level interpolation and re-assign a new label to the interpolated class. Specifically, given samples from class task Ti and class r' in task Ty. we denote the interpolated features as Intrpl(r, r'), which are formally defined as:
H AH : + (1 (1
H AH + A)H Jir'
The interpolated samples are regarded as new class in the interpolated task. After randomly selecting N class pairs, we can construct an N-way interpolated task. Take a 3-way classification as an example, assume task T; has classes (i1, 12, i3) and task T, has classes (f1, J2. js). One potential interpolated task could be 3-way task with classes (e1, 62 6z), where the labels are associated with interpolated features (Intrpl(i,, (. .2), Intrpl((2, Ja), intrpl(ia..i) Note that, for ProtoNet and its variants, we apply the interpolation strategies of Eqn. ( on both LS and NLS scenarios since it is intractable to calculate prototypes with mixed labels.
Finally, we note that MLTI supports both inter-task and intra-task interpolation, when i = j. As we will find in Sec. [6] intra-task interpolation can be complementary to cross-task interpolation and further improve the generalization. Under this case, the intra-task interpolation can also be replaced by any existing intra-task augmentation strategies (e.g.. MetaMix (Yao et al. (021)).
allow the case
After generating the interpolated support set Di (H.... Y(,))) and query set D (Hi Y Y.,.)) we replace the original support and query sets with the interpolated ones. With MAML as an example, reformulate the optimization process in Eqn. (3)
8" - arg min
Σ
;D" Di..), where of. = -8--1 aV oVoL- MAM :r.or);
(6)
where the superscript L represents the rest of layers after the selected layer I. Detailed pseudocode of MAML and ProtoNet is shown in Alg. T and Alg, in Appendix A, respectively.
THEORETICAL ANALYSIS
We now theoretically investigate how MLTI improves the generalization performance with both gradient-based and the basric meta-learing methods. Specifically, theoretically prove that MLTI essentially induces a data-depender regularizer on both categories of meta-learning methods and controls the Rademacher complexity, leading to greater generalization. Here, only discuss the non-label-sharing (NI-S) scenario (see detailed proof in Appendix B.T and leave the analysis of the label-sharing scenario in Appendix B.22
4.1 GRADIENT- BASED META- WITH MLTT
In gradient-based meta-learning, we analyze the generalization ability by considering the two-layer neural network with binary classification. For the simplicity of presentation, we assume the sam- ple size of different task are the same and equal to N. Suppose there are [7] tasks. For each task T., consider the logistic loss MAMMAME (x).y) log(1 exp(XMAME (x))) yfMMAM (x) with MAAL modeled by fM (MMMMM = σ. (WWx,,) = of h where h1 represents the hid- den representation
the first layer of sample x,A: Under the NLS setting, the interpolated task constructed by Eqn. (5). We assume the interpolation performs on the hidden layer (follow- ing Eqn. (5) with / 1) and denote the interpolated query set as D.,,. (H... kooy Y. For simplicity, in this subsection, we omit the superscript and define the empirical training loss as C.(DD..rr M = | Σ!! C(Dr.or) (N]1)) D!! EN= L(fo, (X...er)) V....rr). We first present lemma showing that the loss Ci((Dior)!! induced by MLTT has a regularization effect.
Lemma 1. Consider the MITT with X Beta( (oita(a,,))) 18. Let vba) = e²/(1 + e and Nir denotes the number of samples from the class in task Tí. There exists a constant C > O, such that the
4

---

Published as a conference paper at ICLR 2022
second-order approximation of Li((Drer ((21) is given by
1 N
11 Now h h kor.
L, (D.) + 1
+ (m)Tos)
N
where x = ED [A] with Dx
Beta(a 1, + a+9
e c+B Beta(3 + 1,
This lemma suggests that MLTI induces an (implicit) regularization term on 0;'s through task interpolation and therefore will lead to better generalization bound, as we will show in Section 6.2 with extensive numerical experiments. To study the improved generalization more explicitly, we consider the population version of the regularization term in Eqn. (7) by considering the function class F.y- = (H Φ: (H Φ)
ΣΦ S 7), where y = p(T Er H We also define
JTT E H and assume the following condition of the individual task distribution T as: for all T ~ (T)). T satisfies
rank(E) I R. || & U,
(8)
where E denotes the generalized inverse of E. Further, we assume that the distribution of H is p-retentive for some E (0, 1/2], that is, if for any non-zero vector v E I [E[e(v H >
m1 E(e" )) Such an assumption has been similarly assumed in (Arora et al. 2020 Zhang al. 2021) and is satisfied when the weights has bounded E2 norm.
We also regard (.(I..)._ ) of tasks (T.) the empirical (training) loss R(D,, H and its corre- sponding population loss (on the test data) is defined as R = =ET,=p(T)) Exx,, Y.,T, [C(fa, (X.), Y.))- We then have the following theorem showing the improved generalization gap brought by MLTI.
Theorem Suppose X,'s Yss and ó are bounded in spectral norm and assumption ( holds. There exist constants AL, A2, As > O, such that for all fT E Fr, 5 E (0, 1), with probability at least 1 - (over randomness of training sample), we have the following generalization bound
(R. S Ai max (2) 22
R+U N
R+U 1
Tog((11/)) N
log(1/0) [
A2
As
Based on Lemma I aand Theorem MLTI regularizes on ø Σø (implying small 7) and therefore achieves a tighter generalization bound than the vanilla gradient- based method (where y is uncon- strained). Compared with the individual task augmentation (see Figure (b)). the regularization effect in Eqn. 7 induced by MLTI is larger (i.e., smaller 7) since the total variance is generally larger than the within-group variance (see more details in the Appendix B.3 Therefore, MLTI reduces the generalization error, which we also empirically validate in the experiments.
4.2 METRIC-BASED META-LEARME LEARNING WITH MLTI
In the metric- based meta-learning, consider the ProtoNet with linear representation in the binary classification, which has been commonly considered in other theoretical analysis of meta-learning, see, e.g., (Du al., 2020 Tripuraneni al. 2020). Specifically, assume fo"N (x) A x and
d(.,.) represents the squared Euclidean distance, then the loss of ProtoNet can be simplified as
I'] N logply.. = r(x.a) arg min ΣΣ
FI N
arg arg min ΣΣ
(9)
1 exp( - (c. c2)
A=1
where and C2 are defined as the prototypes of class I and 2, respectively. Under this setting, the interpolation performs on the feature (i.c., / in Eqn. (5))
We now present the following lemma showing that MLTI induces a regularization on the parameter 0. Lemma 2. Considering the interpolated tasks (Dr,rr!"1 with x ~ Beta(o, 8), we de- fine (.(().) = (N]1)) E. exp(((xx, - (c1 22)/2,0))) and Cr (NII)-² E,,(1 exp(((X,,M,er - (C1.er + C2,.r) (2,rr)2,,)))) 0))) Recall v(u) = e" /(1 + e²)²². The second- order approximation of (i((D.,or
) |1]
is given by, for some constant > 0,
1 |n|
17
((( 1
Σ LEI,KE[N]
(((x,, - (c. + +cz) T
X X X ).
NJA
(10)
5

---

Published as a conference paper at ICLR 2022
Similar to the last section, assume that the distribution of is p-retentive for some P E (0, 1/2]. and investigate the following function class: let X = E[xx "
W = {x- + 0"x x, such that satistying Ex [M( - (c. ca)/2,0)]] 0² Exx S 7).
(11)
We then have the following theorem on the explicit generalization bound of ProtoNet.
Theorem 2. Suppose X;'s, Y''s and 0 are both bounded spectral norm, and the distribution of is p-retentive and mean zero. Let rx. rrnn(Ex)), then there exist constants B1, Bz, B3 > 0, for any f E W7, E (0, 1), with probability least 6 (over the training sample), such that
(R((D.) R|
2B max [( (12))44 ()))))
( (V
E N +
B2V
log(1/8) 2
+ Bay
log(1/1/8) N
By Theorem 2 adding MLTI into ProtoNet would induce a small value of y and thus improve the generalization compared to the vanilla ProtoNet. Similarly, MLTI achieves tighter generalization bound than the individual task augmentation with a larger regularization term (i.e., smaller 7)).
5 RELATED WORK
The goal of meta-learning is to enable few-shot generalization of machine learning algorithms by transferring the knowledge acquired from related tasks. One approach is gradient-based meta-
learning (Finn and Levine 2018 Finn et al., 2017: 2018 Grant et al.] 2018 Flennerhag et al.,
2020: Lee and Choil [0018] Li et al. 2017] et al.] 2021 Nichol and Schulman 2018 Rajeswaran et al. 2019 Rusu et al. 2018), where the mneta-knowledee is formulated to be optimization-relttee
parameters (e.g., model initial parameters, learning rate, pre-conditioning matrix). During the meta- training stage, the model is first adapted to each task via a truncated optimization and then the optimization-related parameters are optimized by maximizing the generalization performance of the model. Another line of research is metric-basee based meta-learning (Cao et al., 2021 Garcia and Bruna 2018 Liu et al. 2019, Mishra et al. 2018: Snell et al.) 2017: Vinyals et al. 2016 Sung et al. 2018. Yoon et al. 2019). which meta-leamy learns an embedding space and uses a non-paramttric learner to classify samples. Unlike prior works that propose new meta-learning algorithms, this work aims to improve the task-level generalization of these algorithms and reduce the negative effect of memorization, especially when the number of meta-training tasks is limited.
To mitigate the influence of memorization and improve the generalization, one line of research focuses on directly imposing regularization on meta-learning algorithms (Guiroy et al.] [201 Jamal and Qi 2019: Teeng et al., 2020 Yin et al.; 2020). Another line of research reduces the number of adapted parameters for gradient-based meta-learning (Raghu et al. 2020 Zintgraf et al. 2019). Instead of imposing regularization strategies (i.e., objectives, dropout, less adapted parameters), our approach focuses on augmenting the set of tasks for meta-training Prior works have proposed domain-specific techniques to generate more data by augmenting images (Chen al. 2019) or by reconstructing tasks with latent reasoning categories for NLP-relate tasks (Murty et al. 2021) techniques have augmented tasks by imposing label noise (Rajendran et al.
Recent domain-agnosti
2020) or applying Mixup (Zhang et al. (0188) 8) and its variants (e.g., Manifold Mixup (Verma et al. 2019)) to each task (Ni al. [022 | Yao et al. (021). Unlike these domain domain-agnostti augmentation strategies that applying data augmentation on each task individually (Figure (b)). we directly densify the task distribution by generating additional tasks from pairs of cxisting tasks (Figure[[c) More discussions with individual task augmentation are provided in Appendix C Empirically, we find that MLTI outperforms all of these above strategies in Section
EXPERIMENTS
In this section, we conduct experiments to test and understand the effectiveness of MLTI. Specifically, we aim to answer the following research questions under both label-sharing and non-label-sharing settings: Q1: Compared with prior methods for regularizing meta-learning, how does the MLTI perform? Q2: Is MLTI compatible with different backbone meta-leaming algorithms and does it improve their performance? Q3: How does MLTI perform compared with only applying intra- or cross-task interpolation? Q4: How does the number of tasks affect the performance of MITT?

---

Published as a conference paper at ICLR 2022
Table 1: Overall performance (averaged accuracy/MSE (Pose) + 95% confidence interval) under
label-sharing scenario. MLTI consistently improves the performance under the label-sharing scenario.
Backbone
Strategies
Pose (15 shot)
RMNIST (1 -hoot)
NCI (5-shot)
Metabolism (5-shot)
Vanilla
2.383 + 0.087
57.34 1.25%
77.09 + 0.85%
57.22 1.01%
Meta-Reg
2.358 + 0.089
58.10
1.15%
77.34 + 0.87%
58.00 0.96%
TAMI.
2.208 + 0.091
56.21 + 1.46%
76.50 + 0.87%
57.87 + 1.05%
Meta-Dropout
2.501 + 0.090
56.19 + 1.39%
77.21 1 0.82%
57.53 = 1.02%
MAML
MetaAug
2.296 + 0.080
55.58 +: 0,97%
76.31 + 0.98%
56,65 = 1.00%
MetaMix
2.064 + 0.075
64.60 1.14%
76.88 1 0.73%
58.61 1.03%
Meta-Maxup
2.107 + 0.077
62.13 - 1.08%
77.90 + 0.79%
58,43 = 0.99%
MLTI (ours)
1.976 + 0.073
65.92 1.17%
79.14 11: 0.73%
60.28
1.00%
MetaAug
n/a
65.41 1.10%
74.84 + 0.87%
61.06 0.94%
MetaMix
n/a
67.80
0.97%
75.84 + 0.85%
62.04 = 0.93%
ProtoNet
Meta-Maxup
n/a
66.18 + 1.08%
75.65 + 0.84%
61.36 = 0.91%
MLTI (ours) n/a
70.14 + 0.92%
76.90 + 0.81%
63.47 = 0.96 %
Table 2: Ablation Studies under label-sharing scenario. The results are reported by the averaged accuracy/MSE + 95% confidence interval.
Backbone
Strategies
Pose (15-shot)
RMNIST (I-shot)
NCI (5-shot)
Metabolism (5-shot)
Vanilla
2.383 1 0.087
57.34 1 1.25%
77.09 1 0.85%
57.22 1.01%
Intra-Intrpl
2.072 + 0.077
62.57 = 1.70%
78.23 = 0.78%
58,70 = 0.97%
MAML
Cross- Intrpl
2.017 + 0.072
65.34 + 1.78%
78.64 = 0.80%
59.60 = 1.00%
MLTI
1.976 + 0.073
65.92 + 1.17% %
79.14 + 0.73%
60.28 + 1.00%
Vanilla
n/a
65.41 + 1.10%
74.84 = 0,87%
61.06 + 0.94%
Intra-Intrpl
n/a
67.32 + 0.94%
75.26 + 0.87%
61.66 + 0.88%
ProtoNet
Cross-Intrpl
n/a
69.97 + 0.85%
76.32 + 0.85%
62.48 0.91%
MLTI
n/a
70.14 + 0.92%
76.90 + 0.81 %
63.47 + 0.96%
We compare MLTI with the following two representative domain-agnostic strategies: (1) directly imposing regularization into the meta-learning framework, including Meta-Reg (Yin et al., 2020), TAML (Jaaal and Qi, 2019), and Meta-dropout (Lee al.] 2020); and (2) individual task aug- mentation methods, including Meta-Augmentation (Rajendran et al., 2020). MetaMix (Yao et al.) 202T), and Meta-Maxup (Ni al. 2021) We select MAML and Proto as backbone methods and apply the corresponding meta- leatnlin strategies to them according to their applicable scopes. Note that we also extend MetaMix and Meta-Maxup to ProtoNet, even though the methods only focus on gradient-base based meta- meta-lcarning in the original papers. To further test the compatibility of MLTI. we additionally apply MLTI to other meta-learning backbone algorithms, including MetaSGD (Li et al. 2017). ANIL (Raghu et al.) (2020), Meta-Curvatuu (MC) (Park and Oliva 2019), and Match- ingNet (Vinyals et al. (2016). To provide a fair comparison, all methods use the same architecture of the base model as MLTI and all interpolation-based methods use the same interpolation strategies (see Appendix D.I and E. for details).
6.1
SCENARIO
Datasets and experimental setup. Under the label-sharing scenario, we perform experiments on four datasets evaluate the performance of MLTI: (1) PASCAL3D Pose regression (Pose) (Yin et al.) 2020): it aims to predict the object pose of grey-scale image relative to the canonical orientation. Following Yin al. (2020, we select 50 objects for meta-training and 15 objects for meta-testing; (2) RaSS (RMNIST) (Finn et al. 2019): it is 10-way classification dataset wherein each task constructed by applying a combination of image transformation operators on the original MNIST dataset (e.g., scaling, coloring, rotation). We here use 14 and 10 combinations for meta-training and meta-testing testintstinn, respectively. (3)&(4) NCI (NCI, 2018) and TDC Metabolism (Metabolism) (Huang et al. 2021): both are 2-way chemical classification datasets, which aim to predict the property of a set of chemical compounds. We use six data sources for meta-training and the remaining three sources for meta-testing. The number of shots for the above four datasets set as 15, 1, 5, and 5,
7

---

Published as a conference paper at ICLR 2022
Table 3: Overall performance (averaged accuracy) under the non-label-sharing scenario. MLTI
outperforms other strategies and improves the generalization ability.
minilmagenet-S 1-shot
ISIC
DermNet-S
Tabular Murris
Backbone Strategies
5-shot
1-shot
5-shot
1-shot
5-shot
1-shot
5-shot
Vanilla
38.27%
52.14%
57.59%
65.24%
43.47%
60.56%
79.08%
88.55%
Meta-Reg
38.35%
51.74%
58.57%
68.45%
45.01% %
60.92%
79.18%
89.08%
TAML
38.70%
52.75%
58.39%
66.09%
45.73%
61.14%
79.82%
89.11%
MAML
Meta-Dropout
38.32%
52.53%
58.40%
67.32%
44.30%
60.86%
18.18%
89.25% %
MetaMix
39.43%
54.14%
60.34%
69.47%
46.81%
63.52%
81.06%
89.75%
Meta-Maxup
39.28%
53.02%
58.68%
69. 16.16
46. 10%
62.64%
79.56%
88.88%
MLTI (ours)
41.58%
55.22%
61.79%
70.69%
48.03%
64.55%
81.73% %
91.08%
Vanilla
36.26%
50.72%
58.56%
66.25%
44.21%
60.33%
80.03%
89.20%
MetaMix
39.67%
53.10%
60.58%
70.12%
47.71%
62.68%
80.72%
89.30%
ProtoNet
Meta-Maxup
39.80%
53.35%
59.66%
68.97%
46.06%
62.97%
80.87% %
89.42%
MLTI (ours)
41.36% %
55.34%
62.82%
71.52%
49.38 %
65.19%
81.89 %
90.12% %
respectively. More details on the datasets and set-up are provided in Appendix D.I We adopt MSE to measure the performance for the Pose regression dataset and accuracy for the classification datasets.
Results. Under the label-sharing scenario, we report the overall performance and analyze the compatibility of MLTI in Table 1 and Appendix D.2] respectively. According to Table [ observe that MLTI outperforms other regularization strategies across the board, including passively adding regularization (i.e., Meta-Reg, TAML, Meta-Dropout) and augmenting tasks individually (i.e.. Meta-AAg, MetaMix, Meta-Maxup). These results indicate that MLTI consistently improves generalization through interpolation on the task distribution. The claim is further be strengthened by the compatibility analysis (Appendix D.2), where MLTI boosts the performance of a variety of meta-learning algorithms. We also investigate the effect of the number of meta-training tasks and report the performance in Appendix D.3 We observe that the improvements from MITI are robust under different settings but that the greatest improvements come when the number of tasks is limited.
Ablation study. In Table 2 we conduct an ablation study under the label-sharing scenario. Here, we investigate how MLTI performs compared with only applying intra-task interpolation (i.e., T;= T,) and cross-task interpolation (i.e., T. T,), which denoted as Intra- Intrpl and Cross-Intrpl, respectively. We observe that both Intra-Intrpl and Cross- Intrpl outperform the vanilla approach without task augmentation and that MLTT achieves the best performance, indicating that the strategies are complementary to some degree. In addition, cross-interpolation outperforms the intra-interpolation in most datasets. The results corroborate the effectiveness of cross-task interpolation when tasks sparsely sampled from the data distribution.
6.2 NON-LABEL-SHARING SCENARIO
Datasets and experimental setup. Under the non-label-sharigg scenario, we conduct experiments on four datasets: (1) general image classification on minilmagenet (Vinyals et al. 2016 (3) medical image classification on ISIC (Milton) 2019) and DermNet (Der] 2016); and (4) cell type classification across organs Tabular Murris (Cao et al.) 202I). Since task in meta-lcarning is defined to correspond to a particular data catt-gennrattigg distribution (Finn et al. 2017 Rajeswaran et al.) 2019), the number of distinct meta-training tasks in N-way classification is actually the number of ways to choose N from all base classes. Thus, for minilmagenet and Dermnet, we reduce the number tasks by limiting the number of meta- veeta-training classes (a.k.a,, base classes) and obtain the minilmagenet-S, ISIC, DermNet-S. Tabular Murris benchmarks, whose base classes are 12, 4, 30, 57, respectively (see the experiments
full-size minilmagenet and DermNet in Appendix F..). The experiments are performed under the N-way K-shot setting (Finn and Levine| 2018). where N = 2 for ISIC and N 5 for the rest datasets. Note that, Meta-Aug (Rajendran al., 2020) under the non-label-sharde adopted in vanilla MAML and ProtoNet. Due to space limitations, we report only the accuracy for the lan-label-sharin scenario here and provide the full table with 95% confidence intervals in Appendix
scenario is exactly the same as the label shuffling, which is already
E.9 More details about the datasets and set-up are in Appendix E.I
8

---

Published as a conference paper at ICLR 2022
gives the results of MLTI and prior methods. MLTI consistently outperforms other strategies. The perfor- mance gains suggest that MLTI can im- prove the generalization ability of meta- learning amidst sparsely sampled task dis- tributions. We also analyze of compatibil- ity of MLTI under the shariig scenario in Table of Appendix [E.3] The results validate that MLTI can robustly boost performance with different backbone methods. For the ablation study, we repeat
Results. Table gives
Table 4: Cross-domain adaptation under the non-label-
sharing scenario. A + B represents that the model is meta-trained on A and then is meta-tested on B.
mini
Dermeet
Dermnet - mini
Model
1-shot
5-shot
I-shot
5-shot
33.67%
50.40%
28.40% 30.03%
40.93%
MAML
+MLTI
36.74%
52.56%
42.25%
33.12%
50.13%
28.11%
40.35%
ProtoNet
+MLTI
35.46%
51.79%
30.06% %
42.23%
the experiments on the non-label-sharing scenario and report the results in Table 10 of Appendix E.4
We additionally provide base model and hyperparameter analysis in Appendix E..5
MLTI achieves
the best performance across various settings, further strengthen its effectiveness.
Cross-domain adaptation. To further evaluate performance of MLTI, conduct comparison under the cross- doos-n adaptation setting where we meta-train the model on one source domain and evaluate it another target domain. We perform cross-domain adaptation across minilmagenet-S and Dermnet-S and report the performance under MAMI. and ProtoNet in Table 4 The results validate that MLTI can improve generalization even in this more challenging setting.
Effect of the number of meta-training tasks. We analyze the effect of the number of tasks under 5-shot setting (with ProtoNet backbone) in Figures 2a and 2b (see more results in Ap- pendix E.6p. We have two key observations: (1) MI.TI consistently improves the performance for all numbers of tasks, showing its effective- ness and robustness; (2) The improvement gap between MLTI and the vanilla model decreases as the the number of tasks increases on minilm- DermNet. We
65%
acouracy
8006
66%
Vard
Varde
Cress
30
60
120
150
Num. Mata-training Classes
Num.
Muta-training Classes
(a) minilmagenet
(b) DermNet
Figure 2: Accuracy w.r.t. the num. of tasks under the non-label-sharin scenario. Intra and Cross represent intra-task interpolation (i.e., T, = T₁) and cross-task interpolation (i.e., Ta T₁).
agenet, and keeps consistent expect this is because the meta-training tasks may be more related to meta-testing tasks in minilmagenet, than in DermNet. Besides, we conduct an additional experiments in Appendix extremely limited tasks.
show the promise of MLTI when we only have
Analysis of Interpolated Tasks. Building upon ProtoNet, we show the t-SNE (Maaten and Hinton 2008) visualization of both original tasks and interpolated tasks Figure Specifically, we randomly select 3 original tasks and 300 interpolated tasks under the 1-shot minilmagenet-S setting, where the color of each interpolated task indicates its proximity to the corresponding original tasks, Each task is represented by the averaged representation over its corresponding prototypes, where query sets to calculate the prototypes. The figure suggests that the interpolated tasks generated by MLTI indeed densify the task distribution and bridge the gap between different tasks.
Origrel
Invecohed
combine both support and
Figure 3: Visualization of the original and interpolated tasks.
CONCLUSION
In this paper, we investigate the problem of meta-learning with fewer tasks and propose a new task interpolation strategy MLTI. The proposed MLTI targets the task distribution directly to generate more meta- veta-training tasks via task interpolation for both label-sharing and non-label-sharing scenarios. The consistent performance gains across eight datasets demonstrate that MLTI improves the generalization of meta-lean -eatnile algorithms especially when the number of available meta-tres meta-training tasks is small, which is further supported by the theoretical analysis.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
For our theoretical results, a complete proof of all claims and the discussion of assumptions are provided in Appendix B For our empirical results, discuss the details of datasets and list all hyperparameters under the label-sharing scenario and non-label-sharing scenario in Appendix D.] and respectively. Code: https//fiilbb.comm/axxxxxxo11111
ACKNOWLEDGEMENT
This work supported in part by JPMorgan Chase & Co and Juniper Networks. Any views opinions expressed herein are solely those the authors listed, and may differ from the views and opinions expressed by JPMorgan Chase & Co., Juniper Networks their affiliates. This material is not a product of the Research Department of J.P. Morgan Securities LLC and Juniper Networks. This material should not be construed an individual recommendation for any particular client and is not intended as a recommendation of particular securities, financial instruments or strategies for particular client. This material does not constitute a solicitation or offer any jurisdiction. Linjun Zhang would like to acknowledge the support from NSF DMS-2015378.
REFERENCES
Dermeet dataset, 2016. htt.http/wwwwddeeee
Nci dataset, 2018. URL
Raman Arora, Peter Bartlett, Poorya Mianjy, and Nathan Srebro. Dropout: Explicit forms and capacity control. arXiv preprint arXiv:2003.0339) 2020.
Yujia Bao, Menghua Wu, Shiyu Chang, and Regina Barzilay. Few-shot text classification with distributional signatures. In ICLR, 2020.
Peter L Bartlett and Shahar Mendelson. Rademacher and gaussian complexities: Risk bounds and structural results. Journal of Machine Learning Research, 3(Nov):463-483 2002.
Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. International
Conference Learning Representations, 2021.
Zitian Chen, Yanwei Fu, Yu-Xiong Wang, Lin Ma, Wei Liu, and Martial Hebert. Image deformation meta-networks for one-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 8680-8689, 2019.
Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification. 2020.
Simon S Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. In ICLR, 2020.
Chelsea Finn and Sergey Levine. Meta-learning and universality: Deep representations and gradi- ent descent can approximate any learning algorithm. In International Conference on Learning Representations, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference Machine Learning- Volume 70, pages 1126-1135. JMLR. org, 2017.
Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning, In NeurIPS, 2018.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In International Conference on Machine Learning, pages 1920-1930. PMLR, 2019.
Sebastian Flennerhag, Andrei A Rusu, Razvan Pascanu, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. International Conference Learning Representations, 2020.
10

---

Published as a conference paper at ICLR 2022
Victor Garcia and Joan Bruna. Few-shot learning with graph neural networks. In International Conference on Learning Representations, 2018.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. In International Conference Learning Representa- lions, 2018.
Simon Guiroy, Vikas Verma, and Christopher Pal. Towards understanding generalization in gradient- based meta-learning. arXiv preprint arXiv:1907.07285: 2019.
Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitnik. Therapeuties data commons: Machine learning datasets and tasks for therapeuties. arXiv preprint 102.02210..09548, 2021.
Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta- meeaalleming for few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11719-11727, 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1909. / 1942, 2019.
Greg Landrum. Rdkit: Open-source cheminformatics software. 2016. URL httpp:///gthhub
Com/ http//////rckk/eeeleeeess///////
Hac Beom Lee, Tacwook Nam, Eunho Yang, and Sung Ju Hwang. Meta dropout: Learning to perturb latent features for generalization. In International Conference on Learning Representations, 2020.
Yoonho Lee and Seungjin Choi. Gradient-based meta-learning with learned layerwise metric and subspace. In International Conference Machine Learning, pages 2927-2936, 2018.
Xiaomeng Li, Lequan Yu, Yueming Jin, Chi-Wing Fu, Lei Xing, and Pheng-Ann Heng. Difficulty- aware meta-learning for rare disease diagnosis. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 357-366. Springer, 2020.
Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few shot learning. arXiv preprint 17X7:07707.09835 2017.
Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, and Yi Yang. Transductive propagation network for few-shot learning. In International Conference on Learning Representations, 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9NNov) 2008.
Md Ashraful Alam Milton. Automated skin lesion classification using ensemble of deep neural networks in isic 2018: Skin lesion analysis towards melanoma detection challenge. arXiv preprint arXiv: 10802, 2019.
Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta- learner. International Conference on Learning Representations, 2018.
Rishabh Misra. News category dataset, 06 2018.
Shikhar Murty, Tatsunori Hashimoto, and Christopher D Manning. Dreca: A general task aug- mentation strategy for few-shot natural language inference. In Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2021.
Renkun Ni, Micah Goldblum, Amr Sharaf, Kezhi Kong, and Tom Goldstein. Data augmentation for meta-learning. /CML, 2021.
Alex Nichol and John Schulman. Reptile: a scalable metalearning algorithm. arXiv preprint arXiv:1803.00999, 2018.
Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se- Young Yun. (BOIL): Towards representation change for few-shot learning. In International Conference on Learning Representations, 2021.
11

---

Published as a conference paper at ICLR 2022
Eunbyung Park and Junier B Oliva. Meta-curvature. In International Conference Neural Informa- tion Processing Systems, pages 3309-3319, 3319, 2019.
Viraj Prabhu, Anitha Kannan, Murali Ravuri, Manish Chablani, David Sontag, and Xavier Am- atriain. Prototypical clustering networks for dermatological disease diagnosis. arXiv preprint arXiv:1811.0366. 2018.
Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In International Conference Learning Representations, 2020.
Janarthanan Rajendran, Alex Irpan, and Eric Jang. Mcta-learning requires meta- cett--ugmentttion. In International Conference Neural Information Processing Systems, 2020.
Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. In Advances in Neural Information Processing Systems, pages 113-1244 2019.
Mengye Ren, Eleni Triantafillou, Sachin Ravi, Jake Snell, Kevin Swersky, Joshua B Tenenbaum, Hugo Larochelle, and Richard S Zemel. Meta-learning for semi-super few-shot classification. In ICLR, 2018.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski. Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2018.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pages 4077-4087, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1199-1208, 2018.
Nilesh Tripuraneni, Chi Jin, and Michael Jordan. Provable meta-learning of linear representations. arXiv preprint arXiv:2002. 1684, 2020.
Hung- Yu Tseng, Wen Chen, Yi-Hsuan Tsai, Sifci Liu, Yen-Yu Lin, and Ming-Hsuan Yang. Regularizing meta-learning via gradient dropout. In Proceedings of the Asian Conference Computer Vision, 2020.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. International Conference on Machine Learning, 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pages 3630-3638, 2016.
Haoxiang Wang, Han Zhao, and Bo Li. Bridging multi-task learning and meta-learning Towards efficient training and effective adaptation. 2021.
Huaxiu Yao, Longkai Huang, Linjun Zhang, Ying Wei, Tian, James Zou, Junzhou Huang, and Zhenhui Li. Improving generalization in meta-learning via task augmentation. ICMI., 2021.
Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta-learning without memorization. In International Conference on Learning Representations, 2020.
Sung Whan Yoon, Jun Seo, and Jackyun Moon. Tapnet: Neural network augmented with task- adaptive projection for few-shot learning. In International Conference on Machine Learning, pages 7115-7123, 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6023-6032, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
12

---

Published as a conference paper at ICLR 2022
Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou. How does mixup help with robustness and generalization? In International Conference on Learning Representations, 2021.
Luisa M Zintgraf, Kyriacos Shiarlis, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In International Conference on Machine Learning, 2019.
A PSEUDOCODES
In this section, we show the pseudocodes for MLTI with MAML (meta-training process: Alg. meta-testing process: Alg. and ProtoNet (meta-training process: Alg. 3 meta-testing process:
!
Alg. 4
Algorithm 1 Meta-training Process of MAML with MLTI
Require: p(7): task distribution; 7), y. inner- and outer-loop learning rate; L the number of shared layers; Beta distribution
1: Randomly initialize the model initial parameters 0
2: while not converge do
3: Randomly sample a batch of tasks (T.) with dataset
4: for each task Ti do
5: Sample support set D" (X,, Y!) and a query set - Yt) from D.
6: Sample another task T; (allow = j) from (7:) with corresponding support set Dj
(x) and query set D = (X², Yj)
7:
Random sample one layer from the shared layers
8:
Obtain the hidden representations H.. H H. Hy of the support/query sets of task Tj and T
9: Apply task interpolation between task T. and Tú via Eqn. (5) (label-sharing tasks) or Equ.
(6) (non-label-sharing tasks), and obtain the interpolated support set Di,or (H Yi,rr)
and query set Di,er (Hr.. Y
Calculate the task-specific parameters die by the inner-loop adaptation, i.c. of,a 0²- - De
11: end for
Optimize the model initial parameters as 0 + 0 - 7 EM MAMM, C end while
Algorithm 2 Meta-testing Process of MAML with MLTI
Require: p(7): task distribution; yy: inner-loop learning rate; 8*: learned model initial parameters
1: Randomly initialize the model initial parameters
2: for each task T. with support set Dj and query set DJ do
3: Calculate the task-specific parameters di by the inner-loop adaptation, i.e., di = 0"
V L(f A D))
4: Obtain the predicted labels of the query set by (MA)L(D and evaluate the performance
5: end for
B ADDITIONAL THEORETICAL ANALYSIS
B.I PROOFS OF KON-LABEL-SHARING SCENARIO
B.1.1 PROOF LEMMA
Nerrt Proof. Recall that the interpolated dataset D. (Fr.. Yi -(ht)).r
Mi where
h krorr -bh + (1 A) Vi,k,or T.b(,rr).
13

---

Published as a conference paper at ICLR 2022
Algorithm 3 Meta-training Process of ProtoNet with MLTI
Require: p(7): task distribution; y: learning rate; Beta distribution
1: Randomly initialize the model initial parameters
2: while not converge do
3: Randomly sample a batch of tasks (T.) with dataset
4: for each task T: do
5: Sample support set D (X,, Y?) and a query set = (X), Y?) from D.
6: Sample another task T; (allow
from (T:) with corresponding support set Dj
(Xj. Yj) and query set Dj = (x Y"))
7: Random sample one layer from the shared layers
8:
Obtain the hidden representations H. H H Hg. of the support/query sets of task Ti and T5
9:
Apply task interpolation between task T, and T,, and obtain the interpolated support set Di.er (H (i,r)) and query set D (H (i,rr)
Calculate the prototypes {c,) (N. represents the number of samples in class r) by
N, L(hi, M.. Le i,errr
f0L (h,
exp(-d(1 (h6, c.).o)) expl expl
11:
Calculate the loss of task Ti as C, En log E
11- (h) or),)))
12:
end for
13:
Update 0 -- 0- 7 Em C.
14:
end while
Algorithm 4 Meta-testing Process of ProtoNet with MLTI
Require: p(7): task distribution; 0": learned parameter of the base model
1: for each task Ti with support set Dÿ and query set DV do
2: Calculate the prototypes [c (N, represents the number of samples in class r) by
N, E, (... fo PPN (h(,,r))
3: Calculate the probability of each sample being assigned to class as ply?, r|x;,,)
exp(- (h. ).).)))))
Σn exp(-M(f (h) Ne,p))))))))))
4: Obtain the predicted class as 81,, arg max, ply?, = r x(.,) and evaluate the performance
5:
end for
Here, r Yia. À ~ Beta(o, B), U([I[)), U(([,.), where R, represents the number of classes in task Tr, and Lb(r,r') denotes the label uniquely determined by the pair (r, (r,r'). The superscript q is also omitted in the whole section. Since for give set of r'. and (r,r') has one-to-one correspondence, without loss of generality, we assume r = (r; r') in this classification setting.
Recall that Le((D),er))))
+ E!!! C(D..rr = म M! + EN=, C(So, (x =
H - EN k== [(h) ker Vi.k,r). Then let us compute the second-order Taylor expansion (. ) IT EI N EN L(b with respect to the first argument around |B h = hi we have the Taylor expansion of Li((Ds, ) equals to
up to the second-order
11
C(DD.)
( 00) Cou(b)aar h/
(12)
1=1
1
C(DD))
(h k.r
Φi) 0,
h h
(13)
77
N
M N
-c((M)))
k J.ker )os
A=1
14

---

Published as a conference paper at ICLR 2022
where c Ri (1-A)² x²
and the second equality 13
uses the fact that the data pre-processed so that
1
i= 2r=1
2
k= hir = 0.
B.1.2 PROOF OF THEOREM
We first state standard uniform deviation bound based on Rademacher complexity (c.f. (Bartlett and Mendelson ).02)).
Lemma 3. Assume {z1,..,, J are drawn i.i.d. from distribution P over 2, and G denotes function class on Z with members mapping from Z to [a, bj. With probability at least over the draw of the sample and 5 O, we have the following bound:
log(1/0)
sup [[zg(z)) - Epg(=))] < 2R(9; 21. 2N) + g~O
N
where R(g; 21,..,,) represents the Rademacher complexity of the function class G.
Proof.
We now formulate R((D:) R as
R((D:) R -TTTTTT E C(So (X,), Y.)
Box,) (X:), Y.)
(X+), Y,) B(x (X.)) Y.)
+ TT B B (X,,, Y,)
(X.), Y.)
(48)
(14)
Recall that we consider the function fa.
MAML X,) o. (WX, Φ.
H! and the function class
F, H
H
EΦ 7).
For each Tú, let consider fo. (.) E Fy, Combining Theorem 3.4 and Theorem A.I in Zhang et al. (021), we have the following result for the Rademacher complexity:
(ran((E.T)) + ||
W//2
R(Fa5A1,.... zn) <2 max (12 (2) p
(2
N
(15)
<2max ( 22114, (2)10g .
R+ R+U
Then, bound the first term (i) in Eqn. 14
be below.
ET X.x. C(fo, (x)). ), Y.) B(x-A X, [C(fo, (X.),
(X)), Y.)
<E,, T B(x. C(S
0. x.). Y,) ExX,,)))TT (C (X,), Y.)]
<C1 ( (2 2312
(R + U)
(log(1118))
+ C2v
N
N
where C1 and C2 are constants, and the additional log((7)) term in the last inequality above is caused by taking the union bound on [ tasks.
Denote function g T + IR such that g(T) = Exx.v) Y)). Denote
g = (o(T) : g(T) = Exx,Y)) ()),
15

---

Published as a conference paper at ICLR 2022
Let A(x) = 1/(1 e*). The second term (ii) in Equ.
14 requires computing the Rademacher
complexity for the function class over distributions
1T Σ
R(G; Ti.... T(n) sup DEG
oig(T.) = Esu
, (x (X)) XY
I
DEG IT
|
1
sup gEG
Ox Y)T, (X)
sup DEG
Σ (x.
[
i=1
< sup gEG
0: (E (b)) T
Mo,
II
i=1
m 22111)
R+ U
1 A
|1|
Then have the following bound (ii):
Ex. C(for (X:), Y.) Bree Y.) (2)1/2
/og((/0)
SC3 (
[ + CaV
[
Combining the pieces, we obtain the desired result. With probability at least 1 - 6,
(((D.)) RI s A max
(2)1/4
U R
R R+ U |1|
P
N
log(141/8)
log(1/8) /|
A2
+ /ov
N
B.1.3 PROOF OF LEMMA
Recall that we apply MLTI in the feature space for theoretical analysis, the interpolated dataset is then denoted as D' (X.,.r> Y(,.r) = ((x,, or) )) where
Xi = Axi + (1 N)x Vi,k,er Lb(r,r').
where r -i, À Beta(a, ((I[)), r ((2)), and Lb(r,r') denotes the label uniquely determined by the pair (r,r"). Since for a give set of and (r, r') has one-to-one correspondence, without loss of generality, assume (r, r') in this classification setting.
Proof. To prove Lemma 2. first, we would like to note that since the overall sample mean 2 2 N.. EN X, = 0. we then have
E(xi,k.er xick:r] Xi
Then let compute the second-order Taylor expansion Li((Drer)) )!tl) D N MN f(x. Vck.e)) (N]/))-'' + exp(((Xu.,.o (c1,er + c2,or)/2, 8))) with respect to the first argument around K x1,e) Xik,or, we have that the Taylor
16

---

Published as a conference paper at ICLR 2022
expansion of (c(((D.or) up the second-order equals to
11
[(DD;) + +
0x
==1
12|
((XD;) +
Σ
AT
1
1
Xi,k k )0
|
|
2
Nr
11
- +c 1
1
Σ
Σ
(x( 0) AT(
Xi, )0,
11
|
2 N,
Σ
v((xI, (c1 + c2)/2,0) TT
T
NIT
(M
where c =E (1
B.1.4 PROOF THEOREM
Similar to the proof of Theorem 1, we use Lemma the proof of Theorem
Proof. We first write R((D.) ! R
R((D)) - R = =ET ELL Y.) ET BOX.)) Y.)
Y,) - E7 Y.)
+ TT PE
E (X YJ)
(#6)
(16)
Recall that we consider the function fx(x) 0 x and the function class
W = {x + 0 x, such that satisfying Ex [<>(x (c1 + c2)/2,0)) T Σx sT), (17)
For each Tú, let consider fe(-) E Wy, Combining Theorem 3.4 and Theorem A. Zhang et al. (2021), we have the following result for the Rademacher complexity:
(2//4
rank(Ex)
R(FT;21,.... 2n) 52 max
p
(2)14, (.,.,) <2 max ( P
Then the first term (i) in Eqn. 16) can be bounded below.
(X,), Y.) C( ET. E 1 YJ) <EE B Y.M ((O Y,) EIX.XX))TI Y.))
ET,-A(T) E (x Y.
rE <C₁ max{( 7 2/1/ 22
log((T1/5)
+C2
N
N
where C1 and C2 constants, and the additional log((7]) term in the last inequality above since take union bound on |7| tasks.
Denote function g : T IR such that g(T) E (xx Y)). Denote
= (T(T) : (TT ((x,,)))((((((()) Y)), E W,).
17

---

Published as a conference paper at ICLR 2022
Recall that A(x) 1/(1+=*)) The second term (ii) in Eqn. (6) requires computing the Rademacher complexity for the function class over distributions
1 ( XY]
R(G; Ti...., TT))
suu SEG
(T))
= suu SEG
o
[7|
[7]
1=1
|||
E sup gEG
1
og (x,Y)~T 10 X
sup
1
Σ
og oo
[T]
[T]
gEG
S ma
2
T3
[7]
Then we have the following bound on (ii) in Equ. T6):
(T: ((x,,, ((fo() Y.)
BT B, Y.)
TE <C's m2) V
+
Cop
log(1/6)
(18)
1
[1]
Combining the above pieces, we obtain the desired result. With probability at least 8,
(R((D)! R| 22B max 22)
(2)
( + xx
log (og(1/))
log (1/8)
+ +B2
BBy
2/11
N
B.2 THEORETICAL RESULTS UNDER THE LABEL-SHARING SCENARIO
As discussed in Line 131-133 of the main paper, for protonet, it is impractical to calculate the proto- types with mixed labels. Thus, under the label-sharing scenario, only analyze the generalization ability of gradient-based meta-learning Follow the assumptions under the non-label-sharing scenario, first present the counterpart of Lemma I of the main paper.
Beta(a, 3). Let vuu) = e" eL1 e²22 and Ni, denote the number of samples from the class r in task T, There exists constant e > 0, such that the second-order approximation of C, i,or 1 is given by
Lemma 4. Consider the MLTI with
G(A. (D.) + N ΣΣ
M
ΣΣ
h h D))
(19)
N
Proof. Under the label-sharing scenario, the interpolated dataset Dj ((hl Ne.kerr 1 ) is constructed as
(H
Norr Yi =
h h AhL + (1- -L))),),,.
Vik.orr AY,, + (1 A)y
where X ~ Beta(a, ((([[))
By Lemma 3.1 in Zhang et al. (202I) (with proof on page 13), the data augmentation equals in distribution with the following augmentation
h h Aht + + -Ahh
with À 44 Beta (a + 1,8) + -4 Beta(a + 1,9). U(IT[))-
18

---

Published as a conference paper at ICLR 2022
Then apply the same proof technique as the proof of Lemma and obtain that the Taylor expansion of (r up to the second-order equals to
N
1
C(XD,)
(b LT
0:)0,
Cov(h Kr 1
N
[ ((XD;) + +
N
11|
N
(h (b) of
h
[
N
N
1=1
==1
1=1
-(((A())) + NI
Σ
(h o
ΣΣ
i=1
(1-1²² where c -ED,! x
and D2 = o+g Beta(a + +,,8) + a-B Beta(a + 1)
Given Lemma 4 the population version of the regularization term be defined in the same form of Eq. (14) and therefore the generalization theorem and its corresponding conclusions are the same as Theorem and conclusions in the main paper.
Besides, in this work, the regression setting is only well-defined under the label-sharing scenario. The theoretical analysis under the label-sharing scenario (i.e., Lemma in Section B.2 are not specific to the classification setting and still hold in the regression setting.
B.3 DISCUSSION ABOUT THE VARIANCE OF MLTI
that the second order of regularization depends on Coe er h),.) in Eqn. (1) Igradient-basee meta-learning) or Cou(x,, x.,a) in Eqn. (14) (metric- based meta-learning). Let G denote the random variable which takes uniform distribution on the indices of the tasks. By using the law of total variance, we have Cor(h),,.rr h2,)) E.Con(h),,.o
From the above analysis,
G, h + Cov(BB(bb...c G, (,,) 2 E[Conwh),,... G, h(,)))), where the later is the covariance matrix induced by the individual task interpolation, i.c., = in the interpolation process.
C ADDITIONAL DISCUSSIONS BETWEEN MLTI AND INDIVIDUAL TASK AUGMENTATION
As shown in Figure T MITI directly densifies task distributions by generating tasks rather than apply augmentation strategies to each individual tasks. Compared with individual task augmentation (c.g.. (Yao et [021 Ni al. 2021 ). the reasons why MLTI leads more dense task distributions are summarized under both label-sharing and non-label -on-label-sharing settings.
Label-sharing Setting. MLTT densifies the task distribution by enabling cross-task interpolation. For example, in Pose prediction, we not only interpolate samples within each object, but cross-task interpolation significantly increases the number of tasks. Assume we have two objects (01 and 02), individual task interpolation approaches (c.g., Meta-Maxup) only generate more samples in O1 or O2, where only one object information is covered. However, MLTI further allows generating tasks with both O1 and 02 information by interpolating data samples from O1 and 02.
Non-label-sharing Setting, MLTI also leads to more dense task distribution under the non-label- sharing setting. For example, in 2-way classification with training classes (CO, C1, C2), there are three original tasks, i.c., three classification pairs (CO, CI). (CO, C2), (C1, C2). Individual task interpolation increases the number of samples for each classification pair by enabling data from mix(CO, CI), mix(CO, C2). mix(C1, C2), However, it does not distinguish pairs like (mix(CO, CI), mix(CO, C2)), whereas MLTI does by allowing cross-tate tross interpolation.
19

---

Published as a conference paper at ICLR 2022
D ADDITIONAL EXPERIMENTAL SETUP AND RESULTS UNDER LABEL-SHARING SCENARIO
D.I DETAILED DESCRIPTIONS OF DATASETS AND EXPERIMENTAL SETUP
Under the label-sharing scenario, We detail the four datasets as well their corresponding base models. All hyperparameters are listed in Table 5 which are selected by the cross-validation. Notice that all baselines the same base models and interpolation-basee methods (i.e., MetaMix, Meta-Maxup. MLTT) use the same interpolation strategies.
RainbowMNIST (RMNIST). Follow Finn al. (2019), we create the Rainbow dataset by changing the size (full/half), color 90' 180%, 2709) of the original MNIST dataset. Specifically, we combine training and test set of original MNIST data and randomly select 5,600 samples for each class. We then split the combined dataset and create series of subdatasets, where each subdataset corresponds to one combination of image transformations and has 1,000 samples, where each class has 100 samples. Each task in SainbowMNIST randomly sampled from one subdataset. We use 16/6/10 subdatasets for meta- arraiiing/valiiatiionnn
and angle (0",
and list their corresponding combinations of image transformations
as follows:
Meta-training combinations:
(red, full, 90%), (indigo, full, 0*), (blue, full, 270%), (orange, half, 270%), (green, full, 90") (green, full, 270"), (crange, full, 180"), (red, full, 180%), (green, full, 09), (orange, full, 09), (violet, full, 270%), (orange, half, 90%) (violet, half, 180%), (crange, full, 90%, (violet, full, 180%), (blue, full, 90%)
Meta-validation combinations:
(indigo, half, 270%), (blue, full, 09), (yellow, half, 180%), (yellow, half, 09), (yellow, half, f, 90%) (violet, half, 0%)
Meta-testing combinations:
(yellow, full, 270"), (red, full, 0²), (blue, half, 270"), (blue, half, 09,, (blue, half, 180%), (red, half, 270%), (violet, full, 90%) (blue, half, 90%) (green, half, 270%), (red, half, 90%)
To analyze the effect of task number, we sequentially add more combinations, which are listed as follows:
(indigo, half, 180%), (indigo, full, 180%), (violet, half, 90%) (green, full, 180%), (indigo, half, 0%), (yellow, full, 90%) (indigo, 0, 90%) (indigo, full, 270%), (yellow, full, 09), (red, half, 180%), (green, half, 09), (violet, half, 270%), (yellow, half, 270%), (red, full, 270%), (orange, half, 180%), (orange, half, 09), (green, half, 180%) (indigo, half, 90%) (blue, full, 180%), (violet, fuil, 09), (yellow, full, 180%), (orange, full, 270%), (red, half, 0²), (green, half, 90%)
In RainbowMNIST, we apply the standard convolutional neural network with four convolutional blocks as the base learner, where each block contains 32 output channels. For MAML, we apply the task adaptation process on both the last convolutional block and the classifier. We further use CutMix (Ynn et al. 2019) for task interpolation.
Pose prediction. Follow Yin al. (2020), pose prediction aims to to predict the pose of each object relative to its canonical orientation. We use the released dataset from Yin et al. (2020) to evaluate the performance of MLTI, where 50 and 15 objects are used for meta-training and meta-testing, Each category includes 100 gray-scale images, and the size of each image is 128 128.
As for the base model, we follow Yin et al. (2200) and define the base model with three fixed blocks and four adaptive blocks, where MAML only performs task task-specific adaptation the adapted blocks. Each fixed block contains one convolutional layer and one batch normalization layer, where the number of the output channels in the three convolutional layers
32, 48, 64, respectively.
20

---

Published as a conference paper at ICLR 2022
After the second fixed block, add one max pooling layer, where both the kernel size and stride set as 2. The output of the fixed blocks is fed into a fixed Linear layer and reshaped to 14 x 14 x 1, which is further treated the input of adapted blocks. Each adapted block includes one convolutional layer and one batch normalization layer, where the number of output channels of all convolutional layer is set as 64, ReLU function is used as the activation layer for all blocks in this experiment. Manifold Mixup (Verma et al. (019)) is used for feature interpolation. All baselines are rerun under the same environment.
NCL. We use the "NCI balanced" dataset released in (NCI 2018), where subdatasets are included (i.e.. NCI 1, 33, 41, 47, 81, 83, 109, 123, 145). Each NCI subdataset is complete bioassay for an binary anticancer activity classification (i.e., positivennegative), where each assay contains a set of chemical compounds. We randomly sample 1000 data samples for each subdataset. In our experiments, we represent each drug compound through the 1024 bit fingerprint features extracted by RDKit (Landrum) 2016), where each fingerprint bit corresponds a fragment of the molecule. We select NCI 41, 47, 81, 83, 109, 145 for meta-training and NCI 1, 33, 123 for meta-testing, where each task sampled from one subdataset.
The extracted 1024 bit fingerprint features are fed into an neural network with two fully connected blocks and one linear regressor. Each fully connected block contains one linear layer, one batch normalization layer and one Leakyrelu function (negative slope: 0.01) activation layer, where the number of output neurons of each fully connected block is set as 500. In our experiments, for MAMI, the parameters in the first fully connected block is globally shared across all tasks, and the rest layers are set as adapted layers. We adopt Manifold Mixup (Verma et al. 2019) as the interpolation strategy.
TDC Metabolism. Similar NCI dataset, we create another bio-related dataset TDC Metabolism. In TDC Metabolism, we select 8 subdatasets related to drug metabolism from the whole TDC dataset (Huang et al., 202I). including CYP P450 2C19/2663A4//A222C Inhibition, CYP2C9/CYP26//CPPP
Substrate. The aim of cach dataset is to predict whether each drug
compound has the corresponding property. We use P450 1A2/3A4/266 and CYP2C9/CYP2D6 substrate for meta-training, and CYP2C19/2C9 and CYP3A4 substrate for meta-testing. We balance each subdataset by randomly selecting at most 1000 data samples and each task is randomly sampled from one subdataset. Analogy to the NCI dataset, we use the same neural network architecture and features (i.e., 1024 bit fingerprint) for TDC Metabolism. Manifold Mixup (Verma et al. 2019) is used as the interpolation strategy.
Table 5: Hyperparameters under the label-sharing scenario.
Hyperparameters (MAML)
Pose
RMNIST
NCI
Metabolism
inner-loop learning rate outer-loop learning rate Beta(o, 8), a = num updates batch size
0.01
0.01
0.01
0.01
0.001
0.001
0.001
0.001
0.5 (i j), 0.1 (i j)
2.0
2.0
0.5
5
5
5
10
4
4
query size meta-training maximum training iterations
10
10
10,000
30,000
10,000
10,000
Hyperparameters (ProtoNct)
Pose
RMNIST
NCI
Metabolism
lcarning rate Beta(e,, 3), a = /
0.001
0.001
0.001
n/a
0.5
0.5
batch size
n/a
4
4
4
query size for meta-training maximum training iterations
n/a
10
10
n/a 30,000
10,000
10,000
D.2 COMPATIBILITY ANALYSIS UNDER LABEL-SHARING SCENARIO
In Table 6 show the additional compatibility analysis under the label-sharing scenario. We observe that MLTI achieves the best performance under different backbone meta- learning algorithms, indicating the compatibility and effectiveness of MLTT in improving the generalization ability.
21

---

Published as a conference paper at ICLR 2022
Table 6: Additional compatibility analysis under the label-sharing scenario (evaluation metric: MSE for Pose and accuracy for other datasets), where the 95% confidence intervals are also reported.
Model
(15 (hots
RMNIST shhtt)
NCI (5-shot)
Metabolism (5-shot)
n/a
73.87 1.24%
75.03 0.89%
60.95 + 0.94%
MatchingNet
+MI.TI
n/a
75.36 0.81 %
76.81 + 0.77
63.02 + 1.09%
2.227 0.098 1,938 + 0.078
66.68
1.28%
77.74
0.82%
57.54 1.03%
MetaSGD
+MLTI
72.78 1.06 %
78.43 = 0.86 %
61.83 + 0.99%
6.947 0.159
56.52 1.18%
77.65
0.79%
57.63
1.07%
ANIL
+MLTI
6.042 1 0.146
64.63 1.47% %
78.46 + 0.75% %
60.34 1 1.01
2.174 1 0.096
58.03 1: 1.24%
77.25 0.80%
58.37
1.02%
MC
+MLTI
1.904 + 0.073
63.25 + 1.36 %
78.52 = 0.86%
60.59 + 1.05 %
D.3 EFFECT OF THE NUMBER OF META-TRAINING TASKS UNDER LABEL-SHIARING SCENARIO
For Rainbow.MNIST, analyze the effect of the number of meta-training combinations with respect to the performance, where the number meta- eett-training combinations directly reflects the number of meta-training tasks. Figure 4a and 4b illustrate the results of MAML and ProtoNet, respectively. The results indicate that MITI consistently improves the performance, especially when the number of combinations are limited (c.g.. Figure 4b).
85%
85%
80%
Accuracy
Accuracy
75%
75%
Vanilla Intra Cross
Vanilla Intra
65%
70%
Cross
MLTI
65%
MLTI
55%
16
32
40
16
24
32
40
Num. of Meta-training Combinations
Num. of Meta-training Combinations
(a) MAML
(b) ProtoNet
Figure 4: Accuracy w.r.t. the number of meta-training combinations of transformations in Rain- bowMNIST. Intra and Cross represent the intra-task interpolation (i.c., T, = T,) and the cross-task interpolation (i.e., T. * Tý), respectively.
E ADDITIONAL EXPERIMENTAL SETUP AND RESULTS UNDER NON-LABEL-SHARING SCENARIO
E. DETAILED DATASET DESCRIPTIONS OF EXPERIMENTAL SETUP
In this section, we detail the dataset description and the model architecture under the non-label- sharing scenario. The hyperparameters are seleeted by cross-validation and listed in Table For fair comparison, all basclines adopt the same base models. Additionally, all interpolation based methods (i.e., MetaMix, Meta-Maxup, MLTT) adopt the same interpolation strategies.
minilmagenet-S, In minilmagenet-S, reduce the number of tasks by controlling the number of metaitraining classes. Specifically, in minilmagenet-S, the following classes are used for meta- training:
n03017169, n07697537, n02108915, n02113712, n02120079, no4509417, n02089867, n03888605, n04258138, n03347037, no2606052, no6794110
To analyze the effect of task number, we incrementally add more classes by the following sequence:
n03476684, n02966193, n13133613, n03337140, n03220513, n03908618, n01532829, n04067472, n02074367, n03400231, n02108089, n01910747, n02747177, n02795169, n04389033, n04435653, n02111277, no2108551,
22

---

Published as a conference paper at ICLR 2022
n04443257, n02101006, n02823428, n03047690, n04275548, n04604644,
n02091831, n01843383, n02165456, n03676483, n04243546, n03527444,
n01770081, n02687172, n09246464, n03998194, n02105505, n01749939,
n04251144, n07584110, n07747607, n04612504, n01558993, n03062245,
n04296562, n04596742, n03838899, n02457408, m13054560, n03924679,
n03854065, n01704323, n04515003, n03207743
We apply the same base learner as Finn al. (2017) in our experiments, which contains four convolutional blocks and a classifier layer. Each convolutional block includes a convolutional layer, batch normalization layer and a ReLU activation layer. For MAML, we apply the task-specific adaptation on the last convolutional block and the classifier layer, which yields the best empirical performance.
ISIC. In ISIC dataset, we select task 3 in "ISIC 2018: Skin Lesion Analysis Towards Melanoma Detection" challenge (Milton 2019), where 10,015 medical images are labeled by seven lesion categories: Nevus, Dermatofibroma, Melanoma, Pigmented Bowen's s, Benign Keratoses, Basal Cell Carcinoma, Vascular. Follow Li et al. (2020), we use four categories with the largest number of categories as meta-training classes, including Nevus, Melanoma, Benign Keratoses, Basal Cell Carcinoma. The rest three categories are treated as meta-testing classes. We apply N-way, K-shot settings in ISIC and set N in our experiments. Thus, there are only six class combinations for the meta-training process. Each medical image in ISIC are re-scaled to the size of 84 x 84 x 3 and the base model as well other settings the same as minilmagennt-S.
DermNet-S. We construct the Dermnet-S dataset from the public Dermnet Skin Disease Atlas (Der 2016), which includes more than 22,000 across 625 fine-grained classes after removing duplicated images/classes. Similar to (Prabhu et al. 2018), we focus on the classes with no less than 30 images, resulting 203 selected classes. The base model and other settings are the same minilmagenet-S and ISIC. The selected classes has long-tail tail and we use the top-30 classes for meta-training and the bottom-53 classes for meta-testing. The detailed meta-training and meta-testing classes are listed as follows.
Meta-training classes:
Seborrheic Keratoses Ruff, Herpes Zester, Atopic Dermatitis Adult Phase, Pscriasis Chronic Plaque, Eczema Hand, Seborrheic Dermatitis, Keratoacanthoma, Lichen Planus, #pidermal Cyst, Bczema Nummular, Tinea (Ringworm) Versicolor, Tinea (Ringworm) Body, Lichen Simplex Chronicus, Soabies, Psoriasis Palms Soles, Malignant Melanoma, Candidiasis large Skin Folds, Pityriasis Rosea, Granuloma Annulare, Brythema Hultiforne Seborrheic Keratosis Irritated, Stasis Dermatitis and Ulcers, Distal Subungual Onychomycosis, Allergic Contact Dermatitis, Psoriasis, Molluscum Contagiosum, Acne Cystic, Perioral Dermatitis, Vasculitis, Eczema Fingertips
Meta-testing classes:
Warts, Ichthyosis Sex Linked, Atypical Nevi, Venous Lake, Erythema Nodosum, Granulation Tissue, Basal Cell Carcinoma Face, Acne Closed Comedo, Sclerederma, Crest Syndrcme, Ichthyosis Other Forms, Psoriasis Inversus, Kaposi Sarcoma, Trauma, Polymorphous Light Bruption, Dermagraphism, Lichen Sclerosis Vulva, Pseudomonas, Cutaneous Larva Migrans, Psoriasis Nails, Corns, Lichen Sclerosus Penis, staphylococcal Folliculitis, Chilblains Perniosis, Psoriasis Brythrodermic, Squamous Cell Carcinoma Ear, Basal Cell Carcinoma Ear, Ichtayosis Dominant, Erythema Infectiosum, Actinic Keratesis Hand, Basal Cell Carcinoma Lid, Spi Spiders, Brosio Interdigitalis Blastonycstica, Scarlet Fever, Pompholyx, Melasma, Eczema Trunk Generalized, Metastasis, Warts Cryotherapy, Nevus Spilus, Basal Cell Carcinoma Lip, Enterovirus, Pscudomonas Cellulitis, Benign Familial Chronic Pomphigus, Pressure Urticaria, Halo Nevus, Pityriasis Alba, Pemphigus Foliacecus, Cherry Angicna, Chapped Fissured Feet, Herpes Buttocks, Ridging Beading
23

---

Published as a conference paper at ICLR 2022
To further analyze the effect of task number, similar minilmagenet, incrementally add more classes for meta-training by the following sequence:
Lupus Chronic Cutane Rosacea, Cenital Warts, Dermatofibroms, Seborrheic Keratoses Smooth, Basal Cell Carcinoma Lesion, Sun Damaged Skin, Tinea (Ringworm) Groin, Lichen Sclerosus Skin, Atopic Dermatitis Childhood Phase, Psoriasis Guttate, Warts Common, Warts Plantar, Herpes Cutanecus, Bezema Subacute, Psoriasis scalp, Bullous Pemphigold, sebaceous Hyperplasia, Pyogenic Granuloma, Phototoxic Reactions, Urticaria Acute, CTCL Cutaneous T-Cell Lymphoma, Drug Eruptions, Mucous Cyst, Alopecia Areata, Hidradenitis Suppurativa, Herpes Type Recurrent, Viral kxanthems, Skin Tags Pol ypa, Melanocytic Nevi, Dermatitis Herpetiformis, Eczema Foet, Morphea, Intertrige, Atopic Dermatitis Infant phase, Bowen Disease, Necrobiosis Lipoidica, Lentigo Adults, Xanthomas, Rhus Dermatitis, Keratosis Pilaris, Schamberg Discase, Rosacca Nose, Chondrodermatitis Nedularis, Keloids, Tinea (Ringworm) Foot Webs, Tines (Ringworm) Laboratory, Porokeratosis, Impetigo, Basal Cell Carcinoma Pigmented, Porphyrias, Epidermal Nevus, Fixed Drug Eruption, Venous Walformations, Acne Open Comedo, Perlêche, Acne Pustular, Herpes Type 1 Primary, Tinea (Ringworm) Scalp, Neurofibromatoosis Warts Flat, Pityriasis Rubra Pilaris, Hemangicma, Herpes Type 2 Primary, Tinea (Ringworm) Hand Dorsum, Neurotic Excoriations, Tinea (Ringworm) Primary Lesion, Basal Cell Careinoma Nese, Dariers discase, Tinea (Ringworm) Foot Dorsum, Tinea (Ringworm) Face, Tinea (Ringworm) Incognito, Acanthosis Nigricans, Onycholysis, Marts Digitate, Psoriasis Pustular Generalized, Varicella, Basal Cell Carcinoma Superficial, Herpes Simplex, Nevus Sebaceous, Actinic Keratosis 5 FU, Acne Keloidalis, Hemangioma Infancy, Candica Penis, Tuberous Scierosis, Stucco Keratoses, Eczema Herpeticum, Dyshicrosis, Epidermolysis Bullosa, Actinic Cheilitis Squarous Cell Lip, Ticks, Actinic Keratosis Face, Chronic Paronychia, Biting Insects, Dermatomycsitis, Grovers Disease, Atypical Nevi Dermoscopy, Patch Testing, Telangiectasias, Pityriasis Lichenoides, Psoriasis Hand, Actinic Keratosis Lesion, Lichen Planus Oral, Tinea (Ringworm) Foot Plantar, Eczema Chronic, Herpes Type 2 Recurrent, Lupus Acute, Eczera Asteatotic, Pilar Cyst, Pemphigus, Vitilige, Keratolysis Exfoliativa, AIDS (Acquired Immunodeficiency Syndrome), Syringema, Habit Tie Deformity, Congenital Nevus, Angi okeratomas, Prurigo Ncdularis, pediculosis Pubíc, Tinea (Ringworm) Palm
We use CutMix (Yun et al. 2019) to interpolate samples in the above three image classification datasets. Besides, the interpolation strategy is applied on the query set when i j. which empirically achieves better performance.
Tabular Murris. Follow (Cao et al. 2021), the Tabular Murris dataset is collected from 23 organs, which contains 105,960 cells of 124 cell types. We aim to classify the cell type of each cell, which is represented by 2,866 genes (i.c, the dimension of features is 2,866). We use the code of Cao et al. (2021) to construct tasks, where 15/4/4 organs are selected for meta meta-training//alidatioooi selected organs are detailed as follows:
The
Meta-training organs:
BAT, MAT, Limb Muscle, Trachea, Heart, Spleen, GAT, SCAT, Nammary GLand, Liver, Kidney, Bladder, Brain Myelcid, Brain Non-Myeloid, Diaphragm.
Meta-validation organs:
Skin, Lung, Thymus, Aorta
Meta-testing organs:
Large Intestine, Marrow, Pancreas, Tongue
In Tabular Murris, the base model contains two fully connected blocks and a linear regressor, where each fully connected block contains linear layer, batch normalization layer, ReI.U activation layer, and a dropout layer. Follow Cao et al. (202I), I), the default dropout ratio and the output channels
24

---

Published as a conference paper at ICLR 2022
of the linear layer set 0.2, 64, respectively. We apply Mainfold Mixup (Verma et al. 2019 as the interpolation strategy. also worthwhile to mention that the performance of gradient- based methods (e.g.. MAML) significantly outperforms the reported results in Cao et al. (2021) since they only apply 1-step inner-loop gradient descent in their released code. In addition, during the whole meta-testing process, change the mode from training to evaluation, resulting in the better performance of metric-based methods (c.g., Protonet).
Table 7: Hyperparameters under the non- non-label-sharin scenario.
Hyperparameters (MAML)
minilmagenet-S
ISIC
DermNet-S
Tabular Murris
inner-loop learning rate
0.01
0.01
0.01
0.01
outer-loop learning rate
0.001
0.001
0.001
0.001
Beta(o, (8), B
2.0
2.0
2.0
2.0
num updates batch size
5
5
5
5
4
4
4
query size for meta-training maximum training iterations
15
15
50,000
50,000
50,000
10,000
Hyperparameters (ProtoNet)
Pose
RMNIST
NCI
Metaholism
learning rate
0.001
0.001
0.001
0.001
Beta(a, (3, batch size
2.0
2.0
0.5
0.5
4
4
4
query size for meta-traing meta-training
15
15
15
15
maximum training iterations
50,000
50,000
50,000
10,000
E.2 RESULTS ON FULL-SIZE FEW-SHOT IMAGE CLASSTFICATION DATASETS
In this subsection, we provide the results of MLTI and other strategies on full-size minilmagenet and DermNet in Table 8 where 64 and 150 training classes used in the meta-training process, respectively. Under the full-size minilmagenet and DermNet settings, the original meta-training tasks sufficient to obtain satisfying performance. Nevertheless, applying MI.II also outperforms other strategies, demonstrating its effectiveness in improving generalization ability in meta-learning.
E.3 COMPATIBILITY ANALYSIS UNDER NON-LABEL--SHANE SCENARIO
In Table 9 we report the results additional compatibility analysis under the non-label label-sharing scenario. The results validate the effectiveness and compatibility of the proposed MLTI.
E.4 RESULTS ABLATION STUDY UNDER NON-LABEL-SHARINE SCENARIO
In Table TO, we report the ablation study under the non-label-sharing scenario. The results indicate that MI.TT outperforrs other ablation strategies and achieves better generalization ability.
E.5 ADDITIONAL ANALYSIS ABOUT MODEL CAPACITY AND HYPERPARAMETERS
E.5.1 MODEL CAPACITY ANALYSIS
Here, we investigate the performance MLTI with a heavier backbone model. increase the model capacity, we use ResNet-12 as the base model. The results on minilmagenet-S and Dermnet-S are reported in Table Here, the results of Meta-Maxup and MetaMix are also reported for comparison. According to the results, MLTI outperforms vanilla MAML/ProtoNet, Meta-Man Meta-Maxup and MetaMix. verifying its effectiveness even with a larger base model.
E.5.2 ANALYSIS OF THE INTERPOLATION LAYERS
We further conduct experiments on Metabolism and Tabular Murris to analyze the performance with different interpolation layers when Manifold Mixup (i.e., interpolating features) is used for data interpolation. Here, ProtoNet is used as backbone. We report the results in Table [2 The results indicate (1) fixing the interpolation layer also boost the performance; (2) randomly selecting
25

---

Published as a conference paper at ICLR 2022
Table 8: Results (averaged accuracy 95% confidence interval) of full-size minilmagenet and DermNet.
minilmagenet-full
DermNet-full
Backbone Strategies
1-shot
5-shot
1-shot
5-shot
Vanilla
46.90 + 0.79%
63.02 + 0.68%
49.58 + 0.83%
69.15 + 0.69%
Meta-Reg
47.02 + 0.77%
63.19 + 0.69%
50.10 + 0.86%
69.73 + 0.70%
TAML
46.40 + 0.82%
63.26 1 0.68%
50.26 = 0.85%
69.40 1 0.75%
MAML
Meta- Deeaa-ropout
47.47 = 0.81%
64.11 + 0.71%
51.10 + 0.84%
69.08 + 0.69%
MetaMix
47.81 0.78%
64.22
0.68%
51.83 + 0.83%
71.57 1 0.67%
Meta- Maxup
47.68 = 0.79%
63.51 + 0.75%
51.95 = 0.88%
70.84 + 0.68%
MLTI (ours)
48.62 0.76%
64.65
0.70%
52.32 0.88%
71.77 1: %
Vanilla
47.05 0.79%
64.03 0.68%
49.91 = 0.79%
67.45 + 0.70%
MetaMix
47.21 = 0.76%
64.38 = 0.67%
51.50 = 0.76%
69.55 = 0.68%
ProtoNet
Meta-Maxup
47.33 + 0.79%
64.43 + 0.69%
51.18 + 0.83%
69.07 + 0.72%
MLTI (ours)
48..11
0.81%
65.22 0.70%
52.91 + 0.81% % 71.30 + 0.69%
Table Additional compatibility analysis under the setting of the non-label-sharing scenario. We show averaged accuracy + 95% confidence interval.
Model
minilmagenet-S
ISIC
DermNet-S
Tabular Muris
39.40 + 0.70%
61.01 1.00%
46.50 0.84%
80.37 + 0.90% 81.72 + 0.89 %
MatchingNet
+MLTI
42.09 + 0.81 %
63.87 + 1.08%
49.11 + 0.86 %
37.98 + 0.75%
58.03 0.79%
41.56 0.80%
81.55 1 0.91%
MetaSGD
+MLTI
39.58 + 0.76%
61.57 + 1.10%
45.49 +: 0.83%
83.31 1 0.87 %
1-shot
37.66 + 0.77%
59.08 1.04%
43.88 1: 0.82%
75.67 1 0.99%
ANIL
+MLTI
39.15 1 0.73%
61.78 1 1.24%
46.79 0.77%
77.11 1 1.00% %
37.43 + 0.75%
58.77 1.06%
43.09 - 0.86%
80.47 + 0.91%
MC
+MLTI
40.22 + 0.77 %
61.53 + 0.79%
47.40 + 0.83 %
82.44 1 0.88 %
50.21 + 0.68%
70.16 0.72%
62.56 0.71%
85,99 1 0.76%
MatchingNet
+MLTI
54.59 1 0.72% %
73.62 1 0.84%
65.65 1 0.71% %
87.75 1 0.60 %
49.52 + 0.73%
68.01 0.87%
58.97 + 0.73%
91.03 + 0.55%
MetaSGD
+MLTI
53.19 1 0.69 %
70.44 1 0.65
63.86 1 0.71 %
92.05 1 0.51 %
5-shot
49.21 + 0.70%
69.48 0.66%
60.54 + 0.76%
81.32 + 0.89%
ANIL
+MLTI
52.76 = 0.72 %
72.01 = 0.68% %
63.07 + 0.71 %
82.75 + 0.89
49.66 = 0.69%
68.29 0.85%
60.03 + 0.72%
89.30 + 0.56% 91.23 + 0.52%
MC
+MLTI
53.42 + 0.71 %
70.58 = 0.82%
63.10 + 0.68 %
the interpolation layer achieves the best performance; (3) interpolating the lower layer performs similarly as interpolating at the higher layer, indicating the robustness of MLTI with different selected layers.
E.6 ADDITIONAL RESULTS OF ANALYSIS ABOUT THE NUMBER OF TASKS
Besides the results in the main paper, further provide the 1-shot results for minilmagenet and Dermnet in Figure 5a and 5b] respectively. The results corroborate our findings in the main paper that MLTI consistently improves the performance, especially when the number of tasks is limited.
E.7 MLTI WITH EXTREMELY LIMITED TASKS
In this section, investigate how MLTI performs when we only have extremely limited tasks. Here, we decrease the number of distinct meta-training tasks of minilmagenet and DermNet to 56
by reducing the number of base classes to since ($) 56. Under this setting, two additional baselines with supervised training process (SL.) DDhillon et al. 2020) and multi-task training process
26

---

Published as a conference paper at ICLR 2022
Table 10: Ablation study under the non-label-sharing scenario. We find that MLTI performs best.
Backbone
Strategies
minilmagennt--S
ISIC
DermNet-S
Tabular Murris
Vanilla
38.27 + 0.74%
57.59 + 0.79%
43.47 + 0.83%
79.08 + 0.91%
Intra-Intrpl
39.31 + 0.75%
60.39 1 0.93%
47.16 + 0.86%
81.49 + 0.91% %
MAML. (1-shot)
Cross-Intrpl
39.91 + 0.74%
61.06 + 1.23%
46.21 + 0.79%
80.65 + 0.92%
MLTI (ours)
41.58 = 0.72%
61.79 + 1.00%
48.03 + 0.79 %
81.73 + 0.89 %
Vanilla
52.14 + 0.65%
65.24 1 0.77%
60.56 + 0.74%
88.55 1 0.60%
Intra-Intrpl
52.74 = 0. 7.4
68.96 + 0.74%
63.65 = 0.70%
89.89 + 0.62%
MAML (S-shot)
Cross-Intrpl
53.34 = 0.77%
70.20 + 0.70%
62.59 = 0.76%
89.97 + 0.56%
MLTI (ours)
55.22 = 0.76%
70.69 + 0.68%
64.55
0.74% %
91.08
0.54%
Vanilla
36.26 + 0.70%
58.56 + 1.01%
44.21 + 0.75%
80.03 + 0.90%
Intra-Intrpl
39.31 = 0.75%
60.70 + 1.16%
46.97 = 0.81 %
80.56 + 0.94% 81.22 0.90%
ProtoNet (I-shot)
Cross-Intrpl
40.95 0.76%
62.22 + 1.19%
48.68 + 0.85%
MLTI (ours)
41.36 + 0.75% %
62.82 1: 1.13%
49.38 1 0.85 %
81.89
0.88%
Vanilla
50.72 + 0.70%
66.25 + 0.96%
60.33 + 0.70%
89.20 + 0.56%
Intra- Intra-Intrpl
53.33 0.68%
70.12 + 0.88%
62.91 = 0.75%
89.78 0.58%
ProtoNet (5-shot)
Cross-Intrpl
54.62 0.72%
71.47 +: 0.89%
64.32 + 0.71 %
90.05 + 0.57%
MLTI (ours)
55.34 = 0.74%
71.52 + 0.89 %
65.19 0.73% %
90.12 0.59
Table 11: Analysis the heavier base model (ResNet-13 12) under 1-shot minilmagenet-S and DermNet-
S settings.
Backhone Strategies
minilmagenct-S
DermNet-S
Vanilla
40.02 + 0.78%
47.58 = 0.93%
MetaMix
42.26 0.75%
51.40 = 0.89%
MAML.
Meta-Maxup
41.97 + 0.78%
50.82 = 0.85%
MLTI (ours)
43.35 +: 0.80%
52.03 0.90%
Vanilla
40,96 0.75%
48.65 0.85%
MetaMix
42.95 + 0.87%
51.18 = 0.90%
ProtoNet
Mcta-Maxup
42.68 + 0.78%
50.96 = 0.88%
MLTI (ours)
44.08 + 0.83%
52.01
0.93%
(MTL) (Wang et al. 2021) are also used for comparison. We also report the results of the best baseline MetaMix. All results are listed in 3 corroborate the effectiveness of MLTI even with extremely limited meta- meta-trainit training tasks.
E.8 RESULTS ON ADDITIONAL DATASETS
We further provided two additional datasets under the non-label-sharing setting to show the effective- ness of MITI -igred and Huffpost. Both datasets are non-label-sharing datasets. We detail the data descriptions and hyperparameters in the following.
Iiareed tieredImageNet (Ren et al. 2018) is few-shot image classification dataset, which consists of 351/97/160 images for mnetaarranning//vliiaatiinnnsss S and DermNet-S, we use 35 original meta-tran meta-training classes in tieredlmageNet. All hyperparameters, base model and interpolation strategies are set as the same as minilmagenet-S.
Following minilmagenet-
Huffpost. Huffpost (Misra 2018) aims to classify the category for each sentence. We fol- low [ao et al. (2020) to preprocess Huffpost data, where 25/6/10 classes are used for meta- In our experiments, to construct the base model, we use ALBERT (Lan) let al. (019) the encoder and use fully connected layers as the classifier. The query set size for trainin and testing is set as Due to the memory limitation, we set the batch size as 1 and the learning rate (outer-loop learning rate) for MAMI. 2e-5. The inner-loop learning rate for
27

---

Published as a conference paper at ICLR 2022
Table 12: Analysis of interpolation layers. Layer 0, represents randomly select layer or layer for interpolation. None means vanilla ProtoNet.
Interpolation Layer
Metabolism: 5-shot Tabular Murris: 1-shot
None
61.06 + 0.94%
80.03
0.90%
Layer 0
62.53 + 0.98%
81.18
0.93%
Layer
62.38 + 0.94%
81.25 + 0.90%
Layer 0,
63.47 + 0.96% %
81.89 + 0.88%
50%
53%
Accuracy
45%
Accuracy
50%
Vanilla Intra
Vanilla
40%
47%
Cross
Cross
44%
MLTI
35%
12
25
38
51
64
30
60
90
120
150
Num. Meta-training Classes
Num. of Meta-training Classes
(a) : minilmagenet
(b) DermNet
Figure 5: Accuracy w.r.t. the number of meta-training classes under the non-label-sharing scenario (I-shot), Intra and Cross represent the intra-task interpolation (i.e., T; = TT,) and the cross-task interpolation (i.e., # Tj), respectively.
MAML is set as 0.01. We set B 2.0 in Beta(a, 8). The number of inner loop updates in MAML is set as 5 and the maximum training iteration set as 10,000. Manifold Mixup is used for data interpolation.
We report the results in Table 14 In these two additional datasets, MLTI also outperforms other methods, showing its promise in improving generalization meta-learning.
E.9
FULL TABLES WITH CONFIDENCE INTERVAL
Table 15 16 report the full results (aceuracy + 95% confidence interval) of Table in the paper.
28

---

Published as a conference paper at ICLR 2022
Table 13: Results MLTI with extremely limited tasks. and MTL represent methods with supervised and multi-task training process, respectively.
minilmagenet-S (8 classes) I-shot
DermNet-S classes) I-shot
Model
5-shot
5-shot
SL
32.37 + 0.60%
45.57 + 0.69%
35.69 1 0.58%
53.38 1 0.60%
MTL
33.01 +: 0.64%
46.79 + 0.65%
36.20 +: 0.64%
54.53 1: 0.63%
Vanilla
36,09 + 0.75%
50.01 + 0.67%
37.98 + 0.66%
54.35 + 0.67%
MAML
MetaMix
37.74 = 0.77%
51.79 + 0.68%
40.36 + 0.73%
55.75 + 0.69%
MLTI (ours)
38.13 + 0.70%
53.53 + 0.72%
41.32 + %
56.95 + 0.64%
Vanilla
35.07 + 0.73%
45.10 + 0.63%
37.72 + 0.67%
53.18 + 0.66%
ProtoNet
MetaMix
38.12 + 0.71 %
50.25 + 0.69%
40.07 + 0.69%
55.07 + 0.68%
MLTI (ours)
39.64 = 0.77 %
51.64 + 0.65%
41.31 + 0.71%
56.09 + 0.67%
Table 14: Results on Huffpost and tieredImageNet-S. Here, averaged accuracies 95% confidence intervals are reported.
dieredImageNet-S
NLP: Huffpost
Backbone Strategies
1-shot
5-shot
1-shot
5-shot
Vanilla
42.20 + 0.84%
58.23 + 0.77%
39.51 + 1.07%
50.68 + 0.90%
Meta-Reg
42.87 + 0.86%
59.16 + 0.79%
40.32 + 1.05%
50.96 + 0.98%
TAML.
42.86 + 0.84%
59.33 + 0.76%
40.03 + 1.00%
50.89 + 0.88%
MAML
Meta-Dropouu
41.94 0.82%
58.37 = 0.77%
39.89 = 0.98%
51.03 + 0.91%
MetaMix
43.40 0.85%
61.92 0.80%
40.64 1 1.02%
51.65 1 0.92%
Meta-Maxup
43.69 + 0.88%
60.00 + 0.82%
40.39 + 1.01%
51.80 1 0.91%
MLTI (ours)
44.32 0.82%
62.22
0.79%
41.06
1.04%
52.53 1 0.90%
Vanilla
43.35 0.82%
59.98 + 0.77%
41.85 + 1.01%
58.98 + 0.92%
MetaMix
44.14 = 0.83%
60.97 - 0.81 %
42.27 = 0.98%
60.43 - 0.90%
ProtoNet
Meta- Meta-Maxun Maxup
44.40 + 0.83%
61.79 + 0.78%
42.39 = 1.01%
60.27 + 0.88%
MLTI (ours)
45.44 = 0.86% %
62.35 0.80 %
42.74 = 0.96%
61.09 + 0.91 %
29

---

Published as a conference paper at ICLR 2022
Table 15: Full table of the overall performance (averaged accuracy + 95% confidence interval) under the -sharing scenario.
Backbone
Strategies
minilmagenet-S
ISIC
DermNet-S
Tahular Murris
Vanilla
38.27 = 0.74%
57.59 + 0.79%
43.47 + 0.83%
79.08 = 0.91 %
Meta-Reg
38.35 + 0.76%
58.57 + 0.94%
45.01 + 0.83%
79.18 + 0.87%
TAML.
38.70 0.77%
58.39 1.00%
45.73 + 0.84%
79.82 + 0.87%
MAML (I-shot)
Meta-Dropout
38.32 + 0.75%
58.40 + 1.02%
44.30 + 0.84%
78.18 + 0.93%
MctaMix
39.43 + 0.77%
60.34 + 1.03%
46.81 + 0.81% %
81.06 + 0.86%
Meta-Maxup
39.28 1 0.77%
58.68 1 0.86%
46.10 1 0.82%
79.56 1 0.89%
MLTI (ours)
41.58 = 0.72%
61.79 + 1.00%
48.03 + 0.79 %
81.73 + 0.89 %
Vanilla
52.14 +: 0.65%
65.24 1 0.77%
60.56 1 0.74%
88.55 1 0.60%
Meta-Reg
51.74 1: 0.68%
68.45 + 0.81%
60.92 + 0.69%
89.08 1 0.61% %
TAML
52.75 = 0.70%
66.09 = 0.71%
61.14 0.72%
89.11 = 0.59%
MAML (5-shot)
Meta- Dropout
52.53 0.69%
67.32 0.92%
60.86 = 0.73%
89.25 0.59%
MetaMix
54.14 = 0.73%
69.47 + 0.60%
63.52 + 0.73%
89.75 = 0.58%
Mcta-Maxup
53.02 + 0.72%
69.16 + 0.61%
62.64 + 0.72%
88.88 + 0.57%
MLTI (ours)
55.22 = 0.76%
70.69
0.68%
64.55 + 0.74% %
91.08 + 0.54%
Vanilla
36.26 0.70%
58.56
1.01%
44.21 + 0.75%
80.03 0.90%
MetaMix
39.67 = 0.71%
60.58 + 1.17%
47.71 - 0.83%
80.72 = 0.90%
ProtoNet
Meta-Maxup
39.80 + 0.73%
59.66 + 1.13%
46.06 + 0.78%
80.87 + 0.95%
MLTI (ours)
41.36 + 0.75% %
62.82 1.13%
49.38 + 0.85% %
81.89 + 0.88 %
Vanilla
50.72 0.70%
66.25 0.96%
60.33 + 0.70%
89.20 0.56%
MetaMix
53.10 = 0.74%
70.12 + 0.94%
62.68 + 0.71% %
89.30 = 0.61%
ProtoNet
Meta-Maxup
53.35 + 0.68%
68.97 + 0.83%
62.97 + 0.74%
89.42 + 0.64%
MLTI (ours)
55.34 = 0.74%
71.52 0.89%
65.19 + 0.73%
90.12 + 0.59%
Table 16: Full table (accuracy + 95% confidence interval) of the cross-domain adaptation under the non-label-sharing scenario. A + B represents that the model is meta-trained on A and then is meta-tested on B.
minilmagenet Dermnet-S 1-shot
Dermnet-S + minilmagenet-S 5-shot
Model
5-shot
1-shot
33.67 + 0.61 %
50.40 + 0.63%
28.40 + 0.55%
40.93 +: 0.63%
MAML.
+MLTI
36.74 + 0.64% %
52.56 + 0.62%
30.03 + 0.58 %
42.25 + 0.64% 64%
33.12 + 0.60%
50.13 +0.65%
28.11 + 0.53%
40.35 + 0.61 %
ProtoNet
+MLTT
35.46 0.63%
51.79 + 0.62%
30.06 0.56%
42.23 + 0.61
30


Reviews:
[{"review_id": "zWW4HFBt1b", "reviewer": "Reviewer_a9eU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Pros\n+ The paper is well written and easy to follow.\n+ The idea of interpolating tasks in a meta-learning setting is novel, intuitive and simple.  Although previous work exists that augment the number of tasks, this is the first approach that augments across tasks (rather than within task).\n+ The authors shows good result on different datasets, settings and backbones.\n\n### Cons\n- I feel like results in more “traditional” (and larger) FSL datasets are missing. For example, it would be nice to see results in tieredImagerNet or metaDataset.\n- I also feel that the authors introduce the method as being a general meta-learning approach, but only show results on image. classification/regression. It would be nice to see results in other domains such as RL/NLP/etc tasks.\n- I find the theoretical analysis difficult to follow and potentially not very informative to the rest of the paper (that been said, I am not an expert on generalization theory/Rademacher complexity and cannot properly validate it).", "summary_of_the_paper": "In this paper the authors propose a meta-learning method for few-shot learning. The propose approach, MLTI, creates new (artificial) tasks by interpolating two (existing) tasks form the training set during (meta-)training. The new tasks are generated by interpolating features/labels of two sampled tasks from the training set. The authors show that the proposed approach achieve good results in multiple datasets (both in regression and classification), multiple settings (”label sharing” and “non-label sharing”) and different algorithms (MAML and ProtoNets).\n", "summary_of_the_review": "I recommend this paper for acceptance. The proposed idea is simple and novel, the paper is well written and the empirical evaluation is well executed. \n\n---\n**Post-rebuttal update**\n\nI thank the reviewers for the rebuttal and I keep my rating of 8. \nCongratulations for the nice work!", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "w7-gkFIwX5r", "reviewer": "Reviewer_id95", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found the approach to be simple and relatively well explained, including ablations studies on large-point questions I had while reading, including its behavior and effectiveness for different sizes and number of classes in the original base dataset, as well as effects of inter- and intra- task interpolations.\n\nThe key difference between this work and MetaMix (Yao et al 2020) is incremental but important:  MetaMix will run the inner loop on the unmodified support set only, and use a mix of support+query in outer loop comparison optimization, whereas this work interpolates support set in inner loop as well.  This difference enables between-task interpolation which adds additional augmentation particularly in settings where few tasks can be drawn from the base data.\n\nI didn't follow much of the theoretical sections in detail, and had to look at the appendix proofs to even understand some of the notation in the main text.  In my somewhat limited understanding they seem reasonable.  These claim to show a theoretical generalization improvement in simplified settings (binary classification of single layer model, linear protonet feautres).\n\n\nAdditional questions:\n\nNLS:  In addition to a single set of correspondence pairs, the input examples for each class can be mixed with all-pairwise-combinations.  How many combinations are used?  That is, for two sets of k examples {xs_i} and {xq_j} (i,j in 1..k), one can form k^2 interpolated examples {a xs_i + (1-a) xq_j} using each i,j combination.  Are all of these combinations formed or just a single set of k pairings?  If using more than k pairings, this would change the task from k-shot to k^2-shot; but the l-layer features for each of the k^2 combinations could be computed, and then up to (k^2 choose k) tasks could be selected from these and used in the upper layer loss comparisons.  For k=5 that would increase interpolated pairs from 5 to 25, but potentially get up to 53130 upper layer loss comparisons from each task pair sample -- would this get even more benefit from this task augmentation technique?\n\neq 5:  what does the name of the subscript \"cr\" mean (does it stand for something)?\n\nIt could be useful to have a more explicit explanation of differences with MetaMix.  MetaMix will run the inner loop on the unmodified support set only, and use a mix of support+query in outer loop comparison optimization, whereas this work interpolates support set in inner loop as well.  This is already mentioned at a high level (fig 1 caption and sec 5 last paragraph), but I think it could be even clearer by pointing out the difference in the discussion around eq 5, that support set H^s,Y^s in the inner loop is mixed between tasks, whereas in MetaMix, only the H^q,Y^q are replaced by mixing.\n\n", "summary_of_the_paper": "This paper describes a method for augmenting task selection in meta-learning, by interpolating support and query sets between two random tasks from the base dataset.  This is examined in two scenarios, label-shared LS and non-label-shared NLS, differing in whether the label space is the same between tasks (e.g. pose estimation) or different (classification to different discrete class sets).  In the former, label targets are interpolated as well as support set inputs, while in the latter, new classes are constructed by random cross-task pairings.  Comparisons are made to other interpolation augmentation approaches, including MetaMix, which interpolates in query set but not the support set.  The approach results in significant performance gains on multiple benchmarks in both settings.\n", "summary_of_the_review": "Overall, the approach is described well enough to understand the approach, and is emperically shown to result in decent performance gains in the low task data settings for which it is intended.  The theoretical sections corroborate this, but I found them hard to follow.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nNjgSOo2qcL", "reviewer": "Reviewer_A6CS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n1.\tThis paper proposes a novel task-augmentation method, which is affected by Manifold Mixup, which can be applied to many existing few-shot learning tasks.\n2.\tThe theoretical analysis shows that the proposed MLTI augmentation has a regularization effect and leads the meta-learner to have a better generalization capability.\n3.\tExtensive simulation results on variety of few-shot learning datasets and two representative few-shot learning methods show that the proposed MLTI is highly effective for meta-learning with fewer data. \n\nWeakness\n1.\tComparison with the prior methods in large dataset is missing. For example, in Table 3, the comparison results are provided only for small datasets or reduced version of large datasets. However, the proposed method is not restricted to small dataset. The ablation experimental result in Figure 2 shows that proposed MLTI is still effective when the full miniImageNet/DermNet dataset is used, although the performance gain becomes small when the full dataset is used. I suggest the authors to include the comparison of MLTI and prior methods with full size of miniImageNet and DermNet.\n\nQuestion\n1.\tIn Section 3, the authors mentioned that it is intractable to calculate prototypes with mixed labels. However, in prior work on semi-supervised few-shot learning [1], the prototypes are computed using soft-labels. What happens if we compute prototypes using soft-labels as done in [1]? \n2.\tSome additional studies on the interpolation layer would be helpful for understanding the proposed method. In Algorithm 1 and 3, the interpolation layer $l$ is randomly chosen in step 7. What happens if we fix $l$ instead of randomly sampling $l$ for every iteration? In that case, how are interpolating at lower layer and interpolating higher layer different? \n\nTypo: In last line of page 4, there is a typo (regularizaiton -> regularization)\n\n[1] Ren, Mengye, et al. \"Meta-Learning for Semi-Supervised Few-Shot Classification.\" International Conference on Learning Representations. 2018.\n", "summary_of_the_paper": "This paper proposes a task augmentation method via task-interpolation for data efficient meta-learning. While the traditional meta-learning methods highly rely on a large amount of data to retain diverse training tasks, the proposed method, MLTI generates tasks by interpolating the tasks which are obtained from training data. The experimental results on variety of few-shot learning dataset show that MLTI is effective when the meta-training data for constructing training tasks is not enough, for both gradient-based and metric-based few-shot learner. ", "summary_of_the_review": "This paper proposes a novel task-augmentation method of MLTI and shows the effectiveness of proposed MLTI through the extensive simulation results and theoretical analysis. The proposed MLTI can be applied to both optimization-based and metric-based few-shot learning methods. Adding some experimental results would make the readers to better understand the proposed method.\nHowever, I believe the idea of this paper is valuable for few-shot learning field, and I recommend to accept this paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jS0qHrWY3Io", "reviewer": "Reviewer_maUk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Strengths]\nThe work provides extensive theoretical analysis to provide theoretical guarantees as to how the proposed MLTI task interpolation method achieves better generalization. In contrast, previous methods that have employed common augmentation methods (e.g., label noise, CutMix, MixUp) without theoretical guarantees. \nThe work introduces scenarios that are more challenging than standard benchmarks by limiting the number of meta-training tasks.\nThe work provides extensive experiments across various datasets under such challenging scenarios and demonstrates better performance than previous methods, providing empirical support for the effectiveness of the proposed task interpolation method.\n\n[Weaknesses]\nI believe the work has minor technical novelty compared to the related work by Ni et al [1]. In particular, Ni et al. [1] performs several augmentations for meta-learning, one of which is MixUp for tasks. Using MixUp between any given pair of classes, Ni et al. [1] also creates new tasks.  \n\n[Comments]\nIn Related Work section, the work states that compared to work by Ni et al. [1], the proposed method directly desnifies the task distribution. But, doesn’t [1] effectively densify the task distribution, where a new task can consist of new classes that are constructed by using MixUp on pairs of classes? As such, I believe more discussions on this issue should help better differentiate the work from the related work.\nWhy does the proposed method randomly sample a location where features are to be interpolated? Is there an ablation study on the sampled location? I wonder if this technique is what makes the proposed method perform better than other works. I’m curious as to whether the proposed method, without this technique, still performs better than other works. The ablation study on this would be helpful for better understanding of differences from other works.\nAlso, how does it compare with related works on standard benchmarks, such as miniImageNet. I think that the proposed method should still work with a larger number of tasks and believe that these experimental comparisons can strengthen the contributions of the proposed method.\n\n[1] Ni et al. Data Augmentation for Meta-Learning.", "summary_of_the_paper": "[Summary]\nThis work tackles a scenario where there may not be a large number of training tasks available, which increases the susceptibility of meta-learning algorithms to meta-level overfitting/memorization problem. In particular, to cope with the scarcity of tasks, the paper proposes to augment the given task set through interpolation of tasks. The paper reports better performance than other methods on benchmarks that have fewer training tasks. \n", "summary_of_the_review": "[Recommendation]\nDespite strong experimental results and analysis, at this point, I believe the technical novelties are not significantly different from the work by Ni et al. Thus, I believe the work is marginally below the acceptance threshold. If the above comments are addressed, I’m willing to increase the score.\n\n-----------------------------------------\n[Post-rebuttal]\nI thank the authors for the response, along with clarifications and updates in the manuscript. \nAs the authors have addressed most of my concerns, I'm happy to increase the score accordingly. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "J5Zl_zJWzF", "reviewer": "Reviewer_hkv8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths\n\nAlthough nifty, the idea of pair-wise task interpolation is an incremental change over the existing data augmentation approaches. The theoretical results, highlighting the relationship between task interpolation and the Rademacher complexity, are non-trivial extensions of the Zhang et al. ICLR 2021 and Yao et al. ICML2021 to account for pair-wise task interpolation. I view this as the primary contribution of the paper.\n\nThe comparison against existing data-augmentation baselines for both metric- and gradient-based meta-learning approaches is quite exhaustive. Furthermore, MLTI is tested on a wide variety of datasets. While the improvement on each dataset is only marginal, the consistent improvement in all datasets and across all approaches strengthens the paper’s contribution.\n\nThe paper is well-written and easy to follow.\n\nConcerns\n\nCurrent approaches in meta-learning rely on heavier backbones such as ResNet. As the goal of all the meta-learning methods is to improve the model’s generalizability, I think it is fair to evaluate the effectiveness of MLTI with heavier feature extraction backbones. Such a comparison is relevant as the proposed task interpolation is conducted on the features extracted from some intermediate layer of the network.\n", "summary_of_the_paper": "The paper proposes an interpolation strategy for meta-learning to improve the learned model’s generalizability. The interpolation strategy is quite simple - interpolate between a pair of tasks, in contrast to existing methods such as adding label noise or data augmentation on each task individually. The authors show that the resulting gradient- or metric-based meta-learning framework (MLTI) induces a data-dependent regularizer that controls the Rademacher complexity leading to better generalization. MLTI is tested on specially curated datasets derived from standard benchmarking datasets. Furthermore, MLTI is also compared against existing data augmentation and interpolation strategies for meta-learning to illustrate its effectiveness.\n", "summary_of_the_review": "Overall, the paper proposes a simple extension to standard data/task - augmentation methods for meta-learning but justifies it with rigorous theory. The theoretical results are non-trivial extensions/combinations of existing work. The effectiveness of the approach is evident from the extensive empirical evaluation. The contributions are strong, albeit limited to the meta-learning research community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Current meta-learning algorithms suffer from the requirement of a large number of tasks in the meta-training phase, which may not be accessible in real-world environment. This paper addresses this bottleneck, introducing a cross-task interpolation in addition to the existing intra-task interpolation. The main idea is very simple, which can be viewed as an incremental adding-up to existing augmentation methods. However, the method is well supported by nice theoretical results which highlight the relation between task interpolation and the Rademacher complexity. In fact, this is not a trivial extension of existing work.  Authors did a good job in the rebuttal phase, resolving most of concerns raised by reviewers, leading that two of reviewers raised their score. All reviewers agree to champion this paper. Congratulations on a nice work.
Decision: Accept (Oral)