Set-based Meta-Interpolation for  Few-Task Meta-Learning
Authors: Seanie Lee
Bruno Andreis
Kenji Kawaguchi
Juho Lee
Sung Ju Hwang
Venue: NeurIPS 2022 Accept (2022)
Keywords: meta-learning
task augmentation
meta-regularization

Abstract: Meta-learning approaches enable machine learning systems to adapt to new tasks given few examples by leveraging knowledge from related tasks.  However, a large number of meta-training tasks are still required for generalization to unseen tasks during meta-testing, which introduces a critical bottleneck for real-world problems that come with only few tasks, due to various reasons including the difficulty and cost of constructing tasks. Recently, several task augmentation methods have been proposed to tackle this issue using domain-specific knowledge to design augmentation techniques to densify the meta-training task distribution. However, such reliance on domain-specific knowledge renders these methods inapplicable to other domains. While Manifold Mixup based task augmentation methods are domain-agnostic, we empirically find them ineffective on non-image domains. To tackle these limitations, we propose a novel domain-agnostic task augmentation method, Meta-Interpolation, which utilizes expressive neural set functions to densify the meta-training task distribution using bilevel optimization. We empirically validate the efficacy of Meta-Interpolation on eight datasets spanning across various domains such as image classification, molecule property prediction, text classification and speech recognition. Experimentally, we show that Meta-Interpolation consistently outperforms all the relevant baselines. Theoretically, we prove that task interpolation with the set function regularizes the meta-learner to improve generalization. We provide our source code in the supplementary material.

Set-based Meta-Interpolation for Few-Task Meta-Learning
Seanie Lee'; Bruno Andreis
Kenji Kawaguchi 2, Juho Lee Sung Ju Hwang! KAIST National University of Singapore", AITRICS3
AITRIICS {1snf y02, endries)(kaist. ac.kr, kenji0comp.. nus edu. sg, {juholee, sjhwang82) @kaist.c.. ac.
Abstract
Meta-learning approaches enable machine learning systems to adapt to new tasks given few examples by leveraging knowledge from related tasks. However, a large number of meta-training tasks are still required for generalization to unseen tasks during meta-testing, which introduces critical bottleneck for real-world problems that come with only few tasks, due to various reasons including the difficulty and cost of constructing tasks. Recently, several task augmentation methods have been proposed to tackle this issue using domain-speciftt knowledge to design augmentation techniques densify the meta-training task distribution. However, knowledge renders these methods inapplicable
such reliance on domain-specinn to other domains. While Manifold Mixup based task augmentation methods are domain-agnostic, we empirically find them ineffective on non-image domains. To tackle these limitations, we propose novel aggniin- task augmentation method, Meta- Interpolation, which utilizes expressive neural set functions den- sify the meta-training task distribution using bilevel optimization. We empirically validate the efficacy of Meta- Meta-Interpan Interpolation on eight datasets spanning across various domains such as image classification, molccule property prediction, text classifica- tion and sound classificattion. Experimentally. consistently outperforms all the relevant baselines. Theoretically, we prove that task interpolation with the set function regularizes the meta-leameer to improve generalization.
show that Meta-Interpolation
Introduction
The ability to learn task given only few examples is crucial for artificial intelligence. Recently, meta-learning [39, 3] has emerged as viable method to achieve this objective and enables machine learning systems quickly adapt new task by leveraging knowledge from other related tasks during meta-traine trainttr. Although existing meta-learning methods can efficiently adapt to new tasks with few data samples, large dataset of meta-training tasks is still required learn meta-knowledge that can be transferred to unseen tasks. For many real-world applications, such extensive collections of meta-training tasks may be unavailable. Such scenarios give rise to the few-task meta-learning problem where a meta-learner can easily memorize the meta-training tasks but fail to generalize well to unseen tasks. The few-task meta-learning problem usually results from the difficulty in task generation and data collection. For instance, in the medical domain, it is infeasible to collect large amount of data construct extensive meta-traine veta-ttaining tasks due privacy concerns. Moreover, for natural language processing, it is not straightforward to split a dataset into tasks, and hence entire datasets are treated as tasks [30].
Equual Contribution. Order of the authors was determined by coin toss.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

New Task
Val. Querr Set
X1 X2
Interpolation of X] and X2 with set function
Class
Class 2 Class
Interpolated Support
Val. Support Set
Task
Task 2
+
A A
Query Set 1
Quarry
New Class
New Class
Class
Class
Clasa
Support Set 1
Clocs
Cleas 3
Class
Support Sot
Class
Clasa 3
Fanction
Set Function
Set Function
Innar-oop
Outer-loop
Minimize C
G
Cless
New Class
New Clasa
( e1(3)]
(a) Class Assignment
(b) Task Augmentation
(c) Bilevel Optimization
Figure 1: Concept. Three-way one-shol classification problem. (a) A new class is assigned to pair of classes sampled without replacement from the pool of meta-training tasks. (b) The support
interpolated with set function and paired with a query set. (c) Bilevel optimization of the set function and meta-learner.
Several works have been proposed to tackle the few-task meta-learning problem using task aug- mentation techniques such as clustering a dataset into multiple tasks [30], leveraging strong image augmentation methods such as vertical flipping to construct new classes [32], and the employment of Manifold Mixup [44] for densifying the meta-training task distribution [49, 50J. However, ma- jority of these techniques require domain-spectio knowledge to design such task augmentations and hence cannot be applied to other domains. While Manifold Mixup based methods [49, 50] are domain agnossic, we empirically find them ineffective for mitigating meta-overfitting in few-task meta-lcarning especially in non-image domains such as chemical and text, and that they sometimes degrade generalization performance.
In this work, we focus solely on domain-agnostic task augmentation methods that can densify the meta-training task distribution to prevent meta-overfitiing and improve generalization meta- testing for few-task meta-learning. To tackle the limitations already discussed, we propose a novel domain-agnostic task augmentation method for metric based meta-learning models. Our method, Meta-Interpolation, utilizes expressive neural set functions to interpolate two tasks and the set functions are trained with bilevel optimization so that meta-learner trained on the interpolated tasks generalizes to tasks in the meta-validation set. As consequence of end-to-end training, the learned augmentation strategy is tailored each specific domain without the need for specialized domain knowledge.
For example, for K wwy classification, sample two tasks consisting of support and query sets and assign a new class k to each pair of classes {o(k), o' [k)} for k = I,.... K, where o' are permutations {1, (I,...,K K} as depicted in Figure la. Hidden representations of the support set with classes o (k) and (k) are then transformed into a single support set using set function that maps of two vectors to single vector. We refer to the output of the set function as the interpolated support set and these are used to compute class prototypes. As shown in Figure 1b, the interpolated support set is paired with query set (query set in Figure Ia)), randomly selected from the two tasks to obtain a new task. Lastly, we optimize the set function so that a meta-learner trained on the augmented task can minimize the loss on the veta-validation tasks as illustrated Figure 1c.
To verify the efficacy of our method, we empirically show that it significantly improves the per- formance of Prototypical Networks [40] on the few-task meta-leaming problem across multiple domains. Our method outperforms the relevant baselines on eight few-task meta-learning benchmark datasets spanning image classification, chemical property prediction, text classification, and sound classification. Furthermore, our theoretical analysis shows that our task interpolation method with the set function regularizes the meta-leamm and improves generalization performance.
Our contribution is threcfold:
We propose novel domain-agnoste task augmentation method, Meta-Interpolatte which leverages expressive set functions to densify the meta-training task distribution for the few-task meta-learnig problem.
We theoretically analyze our model and show that it regularizes the meta-learner for better generalization.
Through extensive experiments, we show that Meta-Interpolation significantly improves the performance of Prototypical Network on various domains such as image, text, and chemical molecule, and sound classification on the few-task meta-learning problem.

---

2 Related Work
Meta-Learning The two mainstream approaches to meta-learning are gradient based [10, 33, 14, 24, 12, 36, 37] and metric based meta-learning [45, 40, 42, 29, 26, 6, 38J. The former formulates mnota-enoweedge
meta-parameters such as the initial model parameters and performs bilevel optimization to estimate the parameeeessssss so that a meta-learner can generalize to unseen tasks with few gradient steps. The latter learns embedding space where classification is performed by measuring the distance between a query and a set of class prototypes. In this work, we focus on metric based meta-learning with fewer number of meta-training tasks, i.e., few-task meta-learning. We propose a novel task augmentation method that densifies the meta-training task distribution and mitigates overfitting due to the fewer number of meta-training tasks for better generalization unseen tasks.
Task Augmentation for Few-Task Meta-learning Several methods have been proposed to augment the number of meta-training tasks to mitigate overfitting in the context of few-task weta-learning. Ni et al. [32] apply strong data augmentations such vertical flip to images to create a new class. For text classification, Murty et al. [30] split meta-training tasks into latent reasoning categories by clustering data with a pretrained language model. However, they require domain-specific knowledge to design such augmentations, and hence the resulting augmentation techniques are inapplicable to other domains where there is well-defined data augmentation pretrained model. In order to tackle this limitation, Manifold Mixup-based task augmentations have also been proposed. MetaMix [49] interpolates support and query sets with Manifold Mixup [44] to construct performs Manifold Mixup [44] on support and query sets from two tasks for task augmentation. Although these methods are domain-agnen agnostic, domains and can degrade generalization performance. In contrast, we propose to train an expressive neural set function to interpolate two tasks with bilevel optimization to find optimal augmentation strategies tailored specifically to each domain.
query set. MLTI [50]
empirically find that they are not effective in some
3 Method
Preliminaries In meta-learning, we are given a finite set of tasks (T:) which are i.i.d samples from unknown task distribution p(T). Each task T, consists of support set Dj {(x],, vi,,) N. and query set D = ((x(,,+Vf))) )i where Xt,; and yr, denote a data point and its corresponding label respectively. Given a predictive model fo.x == Stg o I o yA o f/, .... o fb, with T. layers, we want to estimate the parameter 0 that minimizes the meta-training loss and generalizes to query sets D sampled from an unseen task T. using the support set D where À a hyperparameter for the function 4Px. In this work, we primarily focus on metric based meta-learning methods rather than gradient based meta-learning methods due to efficiency and empirically higher performance over the gradient based methods on the tasks we consider.
Problem Statement In this work, we focus solely on few-task meta- meta-learning. Here, the number of meta-training tasks drawn from the eett-ttainnig distribution is extremely small and the goal of a meta-learner is to Iearn meta-knowledd tasks during meta-testing. The key challenges here are preventing the meta-learner from overfitting on the meta-training tasks and generalizing to unseen tasks drawn from meta-test set.
from such limited tasks that can be transferred to unseen
learn an embedding space induced by fe.x, where we perform classification by computing distances between data points and class prototypes. We adopt Prototypical Network (ProtoNet) [40] for fe, where ςox is the identity function. Specifically, for each task T, with its corresponding support Dj and query DJ sets, we compute class prototypes {cx } as the average of the hidden representation of the support samples belonging the class ls follows:
Metric Based Meta-Learning The goal of metric based meta-learning
C₁ = NR.
Σ DEDD
(x(,)) E D
(1)
where N₂ denotes the number of instances belonging to the class k. Given a metric d(..)) :): RD x RD + IR, we compute the probability of a query point x²,; being assigned the class & by measuring the distance between the hidden representation fo (x²,)) and the class prototype e2 followed by
3

---

Algorithm 1 Meta-training
Algorithm 2 HyperGrad [27]
Require: Tasks [T, (T ) learning rate
Require: model parameter 0, hyperparamter X, vali- dation tasks {ir "rraly""" |t= learning rate a, gradient of training loss w.r.t 8 #Ese, batch size B', and the number of iterations for Neumann series q e N.
a, E R4, update period S, and batch size B.
1: Initialize parameters 0,
2: for all i +- L,... M do
Lir <
Lv +-
4: for all 1, ...... do
2:
for all f- t.... L..... do
5:
Sample tasks Ti, (Di D2, and
3: Sample task Te from (Tx² J0-1-
Ti (Di,, Di2 from (7, ) Ii-1.
4: Lv +- F7 Lsinglewon (2, 0,0;70 7)
6:
D' + Interpolate(Di, Dig; Sx)) with Eq.3.
5: end for
7:
T (D',DY,)
6:
-0
8:
Cer += 2R Csingkunn TT,)
Initialize p t- deepcopy(v1)
9:
Cer += fmma (2, 0, T)
8: for all j f- I..... 4 do
9:
OCr grad 00 0, grad_oupputs
outputs v1)
10:
end for
11:
a 26
dEtr
10: +
12:
if mo, S) = then
11: end for
13:
- (ypperGrra A, (T 1
26.m
12: V2 grad( fir grad gradd_outputs op)
80
14:
x+ 7 7-9
13: return dcy
15:
16:
end for
end if
14:
return 8,A
15:
softmax. With the class probability, we compute the cross-entropy loss for ProtoNet as follows:
log (xp(=d(fox(xx,,))eeo c..))
Lxingleton (A, 0, Tı)
Σ =. i,k
exp(= cw))
where 1 is an indicator function. At meta-test time, a test query is assigned label based on the minimal distance to class prototype, i.e., y! arg miny d(fa, (x3), c2). However, optimizing + Et= Lsingleon (X, 0: T:) w.r.t is prone to overfitting since we are given only a small number of meta-training tasks. The meta-learne tends to memorize the meta-training tasks, which limits its generalization to new tasks at meta-test time [51, 35].
order to tackle the meta-overfitting problem with a small number of tasks, we propose novel data-driven domain-agnostic task augmentation framework which enables the meta-learner trained on few tasks to generalize to unseen few-shot classification tasks. Several methods have been proposed to densify the meta-training tasks. However, they heavily depend on the augmentation of images [32] or need a pretrained language model for task augmentation [30]. Although Manifold Mixup based methods [49, 50] are domain-agnostec empirically find them ineffective in certain domains. Instead, we optimize expressive neural set functions to augment tasks to enhance the generalization of meta-learner to unseen tasks. As consequence of end-to-end training, the learned augmentation strategy is tailored to each domain.
Meta-Interpole Interppteroolnnonn for Task Augmentation
Specifically, let ęox Raxd - Rd be a set function which maps set of d dimensional vectors with cardinality n to a d dimensional vector. all our experiments, we use Set Transformer [23] for xxx Given pair of tasks Tr, (Di Dj, } and Tr (Di, Di } with corresponding support and query sets for K way classification, we construct new classes by choosing tasks. We sample permutations d., and ot, (1,.... K} for each task Tr, and Tıg respectively and assign class l to the pair for, (k)), ot. (k)) for k = L..... K. For the newly assigned class k. we pair two instances from classes or, (k) and de, (k) and interpolate their hidden representations with the set function Φxx The class prototypes for class i are computed using the output of şa as follows:
pairs of classes from the two
S₃ = [([x,, x (xi, v(, Di, Vi, = =o (k), V(a.)) E Di Vi.s = oz, (k))
hij (Se, o...o Son (x)r,))
hids (So, o...o So, (x(g,,))) Rd
c = |Sal (i hAees, halees
Σ
(th. o...o sutl) (pa(th ber E ERD
D' = (cI....,ck)
(3)
where we define De to be the set of all the interpolated prototypes é₁2 for k = 1,..., K. For queries. we do not perform any interpolation. Instead, we use Dí, as the query set and compute its hidden

---

representation (x .) E RD. We then measure the distance between the query with vū, = oz, (k)
and the interpolated prototype of class k to compute the loss as follows:
10 (k)) logg exp(-d(fo. Cx))
Lanix (x, 0, 1 =
(4)
(w))
2,k
where (D', The intuition behind interpolating only support sets is to construct harder tasks that a meta-learner cannot memorize. Alternatively, we can interpolate only query sets. However, this is computationally more expensive since the size of query sets is usually larger than that of support sets. In Section 5, we empirically show that interpolating either support or query sets achieves higher training loss than interpolating both. which empirically supports the intuition. Lastly, we also use the original task Tr, to evaluate the loss Lsingleton (X, 0, Tr, in Eq. 2 by passing the corresponding support and query set to fe,A. The additional forward pass enriches the diversity of the augmented tasks and makes meta- meta-traing training consistent with meta- meta-testing since we do not perform any task augmentation in the meta- meta-testing stage.
Optimization Since jointly optimizing the parameters of ProtoNet, and X, the parameters of the set function Ωxx with few tasks is prone to overfitting, we consider À as hyperparameters and perform bilevel optimization with meta-training and meta- velaavaaiootion tasks follows:
T
X == arg min À
Ssnngleton (A, 1,00(M) (A);
(5)
T
0" (A) arg min
Ssingleton (X, 0; T,train + Liiia (A, 0; Fe)
(6)
2T
where T, T, Tr denote the meta-training, meta-validation, and interpolated task, respectively.
Since computing the exact gradient w.r.t is intractable due to the long inner optimization steps in Eq. 6, we leverage the implicit function theorem to approximate the gradient Lorraine et al. [27]. Moreover, we alternately update 0 and À for computational efficiency as described in Algo. and 2.
Theoretical Analysis
In this section, we theoretically investigate the behavior of the Set Transformer and how it induces a distribution dependent regularization, which is then shown have the ability to control the Rademacher complexity for better generalization. To analyze the behavior of the Set Transformer, we first define it concretely with the attention mechanism A(Q, K, V)
paftmax(Vd-TQK3 Given h, h' E Rd, define H (hhh' = [h, N/] E R22d and (h) = TT e Rixd Then, for any
E ((h, N), (h)), the output of the Set Transformer (x)(r) is defined as follows:
(x(r)= = A(Q2, K3, VTT E ERd
where Q2 = SW? + +b8.0000 Qi HITW + 1269 Kj H.WKK + 12b VJ -H5W + 12by (for j E {1,2)), and HΕ = A(Q,, KT, VT) Ruxd K,, Vj denote query, key, and value for the attention mechanism for j = 1,2, respectively. Here, 12 = [[1.... [I.....,1] 1] E R", W W wy e Rdxd, D9.6K,
bKK by E Rxdd [I'd,Q;,KK) KJ, Vý" Rand, and Q2 E Ridd Let (1....,
Our analysis will show the importance of the following quantity of the Set Transformer in our method:
(1.1)
(1.' (1 P1-P1 (...6.))))) - -1-12 1-podddt0))( - - - A...A...))
dg
= P2
(8)
where p1 (1.)))))
softmax( (V (hanar (K (k( )T, Di p(ee
softmax( (aaanx(VVV1))) (eesh (K) the
)2,1, P2 (1)) softmax( Vd- 12 (herrateee tleeeeeess a 1,1
with he, dg (xf,) and %6 = f/.
fo. For matrix A E Rmxn At, denotes the entry for i-th row and j-th column of the matrix A.
We now introduce the additional notation and problem setting to present our results. Define W = spp(-d(foxxxx)))eee
(WY W E RRddd b (6) W. + +T E Rd, Li(c)- - Σ. log dx
5

---

and 1,, fi [N! vi.; = k), where N² [Dj]. We also define the empirical measure Mr, TT.nl Eier, n 8, over the index E [N_*]] with the Dirac measures d;. Let [[K] be the uniform distribution over {1...., K}. For any function yp and point u in its domain, we define the j-th order tensor ä (()( E Rdxdx..xdd by @'v((M)(siar- = du.z Mig " (ou). For example, o' ((() and & (p()) are the gradient and the Hessian of yp evaluated u. For any j-th order tensor al (l)(()) we define the vectorization of the tensor by vec[@)up(u)] E Rd".. For an vector a E IR4, we define doj a a @ ... @ a E IRd where @ represents the Kronecker product. We assume that Ong (W sb (xi,) +) = O for all r 2 2, where == Jbc ....o p/+1 .... This assumption is satisfied, for example, if represents a deep neural network with ReLU activations. This assumption is also satisfied in the simpler special case considered in the proposition below.
af
The following theorem shows that Lmix (2, 02 Te.tr) approximately Lringleton (A, 0; TT) plus regular- ization terms on the directional derivatives of %b. on the direction of W (0b, (x d% (xi,)))
Theorem 1. For any J E N4. if + d(y, c) is J-times differentiable for all y, then the J-th order approximation of Lmixe (2, 0, Tr, is given by Lxingleron (x, 0; + f1 vec( [x(c)]] A 0j where
A I [l and
A = E impeck
(1.40) [ Oy (W W. (xi,) W (x Se (x
To illustrate the effect of this data-dependent regularization. we now consider the following special case that is used by Yao et al. [50] for ProtoNer: L(,, 0; Tr) = - L. (A, 0; Tr) where Li (), TT)
of = N.K Eixt,
()),,,))x?,, and expix.
(eesp(((",,-((((c)))) (1-) denotes dot product. Define c = 1 E.= p(s, KP(2. where (z(,,) = (+exp(21.)) I 1+xxp(=s,2)
and 21, (x9, (ci + 222 Note that c > 0 if A is no worse than the random guess; e.g., L, (2,0; > L, (2, 0; Ta) for all E [n]. We write ||ul|3r = uT Mr for any consider that is balanced: i.c. Ev (2 I EjEle [T EiEle (1,4") (00, (x),,) 0% (xx,))) = 0 for all t. This 41 (ob, used to prevent the Set Transformer from over-fitting to the training sets; i.e., in such simple special cases, the Set Transformer without any restriction is too expressive relative to the rest of the model (and may memorize the training sets without using the rest of the model). The following proposition shows that the additional regularization term is simplified to the form of c|101]3, in this special case: Proposition 1. In the special case explained above, the second approximation of where 8L.
positive semi-definite matrix M. In this special case,
Ep Eemcc (A, (x,0, 0 Tex) given by fxingteton (2,0; 0; Te) + c|10112 1o To
(1,2)) (x+ xx,)).
EK-U12]
In the above regularization form, we have implicit regularization effect on |10133 where Σ Exx<[(x xx)(x T) The following theorem shows that this implicit regularization can reduce the Rademacher complexity for better generalization:
/Trraan((2))
Proposition 2. Let FR = {x + 0 x |0113 < RJ with [x(x) = 0. Then, (FR S
Vn
All the proofs are presented in Appendix A.
5 Experiments
We now demonstrate the efficacy of our set-based task augmentation method on multiple few-task benchmark datasets and compare against the relevant baselines.
Datasets We perform classification on eight datasets to validate our method. (1), (2), & (3) Metabolism [17], NCI [31] and Tox21 [18]: these are binary classification datasets for predicting the properties of chemical molecules. For Metabolism, we use three subdatasets for meta-training, meta-validation, and meta-testing, respectively. For NCI, we use four subdatasets for meta-training, two for meta-val vela-vaalioiion and the remaining three for meta-testing, For Tox21, use six subdatasets

---

Table 1:
Average accuracy of runs and -95% confidence interval for few shot classification on non-image domains- - Tox21, NCI, GLUE-SciTail dataset, and ESC-50 datasets. ST stands for Set Transformer.
Chemical
Text
Speech
Metabolism
Tox21
NCI
GLUE-SciTai
ESC-50
Method
5-shot
5-shot
5-shot
4-shot
5-shot
ProtoNet
63.62 + 0.56%
64.07 + 0.80%
80.45 + 0.48%
72.59 = 0.45%
69.05
1.48%
MetaReg
66.22 1 0.99%
64.40 t 0.65%
80.94 + 0.34%
72.08 + 1.33%
74.95
1.78%
MctaMix
68.02 + 1.57%
65.23 + 0.56%
79.46 + 0.38%
72.12 = 1.04%
71.99
1.41%
MLTI
65.44 1 1.14%
64.16 1 0.23%
81.12 1 0.70%
71.65 - 0.70%
70.62
1.96%
ProtoNet+ST
66.26 + 0.65%
64.98 + 1.25%
81.20 + 0.30%
72.37 + 0.50%
71.54
1.56%
Meta-Interpolation
72.92 + 1.89%
67.54 0.40%
82.86
0.26%
73.64 +: 0.59%
79.22 + 0.84%
for meta- metat-rainingg two for meta-validation, and four for meta meta-testing, (4) SeiTall [30]: it consists of four natural language inference datasets where we predict whether hypothesis sentence contradicts a premise sentence. We use MNLI [47] and QNLI [46] for meta-training, SNLI [5] (5) ESC-50 [34]: this is an environmental sound recognition dataset. We make 20/15/15 split out of 50 base classes for meta- training/validaation/tssggg and sample classes from each spilt to construct a 5-way classification task. (6) Rainbow MNIST (RMNIST) [11]: this is a 10-way classification dataset. Following Yao et al. [50], we construct each task by applying compositions of image transformations to the images of the MNIST [9] dataset. (7) & (8) Mini-ImageNet-S [45] and CIFAR 1100-FOFF [22]: these 5-way classification datasets where we choose 12/16/20 classes out of 100 base classes for meta- training/aaliddaiionntstnng respectively and sample 5 classes from each split to construct task.
and RTE [46] for meta-validation, and SciTail [20] for meta-testing.
Note that Metabolism, Tox21, NCI, GLUF--eiTTall, and ESC-50 are real-world few-task meta- learning datasets with very small number of tasks. For Mini-ImageNet-S and CIFAR 100-FS, following Yao et al. [50], we artificially reduce the number of tasks from the original datasets for few-task meta-learning RMNIST is synthetically generated by applying augmentations to MNIST.
Implementation Details For RMNIST, Mini-ImageNettSS and CIFAR 100-FS, we use four convo- lutional blocks with each block consisting of a convolution, ReLU, batch normalization [19], and max pooling. For Metabolism, Tox21, and NCI, we convert the chemical molecules into SMILES format and extract a 1024 bit fingerprint feature using RDKit [15] where each bit captures a fragment of the molecule. We use two blocks of affine transformation, batch normalization, and Leaky ReLU, and affine transformation for the last layer. For GLUE-SciTail dataset, we stack 3 fully connected layers with ReLU on the pretrained language model ELECTRA [8]. For ESC-50 dataset, we pass raw audio signal to the pretrained VGGish [16] feature extractor to obtain an embedding vector. We use the feature vector as input to the classifier which is exactly the same as the one used for Metabolism, Tox21, and NCI. For our Meta-Interpolation we use Set Transformer [23] for the set function yax.
Baselines We compare our method against following domain-agnostic baselines.
1. ProtoNet [40]: Vanilla ProtoNet trained on Eq. 2 by fixing PP to be the identity function.
2. MetaReg [2]: ProtoNet with Eg regularization where element-wise coefficients are learned with bilevel optimization.
MetaMix [49] ProtoNet trained with support sets and mixed query sets where we interpolate one instance from the support sets and the other from the original query sets with Manifold Mixup.
4. MLTI [50]: ProtoNet trained with Manifold Mixup based task augmentation. We sample two tasks and interpolate two query sets and support sets with Manifold Mixup, respectively.
5. ST ProtoNet and Set Transformer (ạxx) trained with bilevel optimization but without task augmentation for Cmix (A, 0, Tt) in Eq. 6.
6. Meta- Interpanteoooltioon Our full model learning to interpolate support sets from two tasks using bilevel optimization and training the ProtoNet with both the original and interpolated tasks.
Results As shown in Table 1, Inte-poneroollation consistently outperforms all the domain-agnostic task augmentation and regularization baselines on non-image domains. Notably, it significantly improves the performance on ESC-50, which a challenging datatset that only contains 40 examples per class. In addition, Meta-Interpolation effectively tackles the Metabolism and GLUa-SciTail datasets which have an extremely small number of meta-training tasks: three and two meta-training tasks, respectively. Contrarily, the basclines do not achieve consistent improvements across all the
7

---

Table 2: Average accuracy 5 runs and 195% confidence interval for few shot classification on image domains Rainbow MNIST, Mini-emageNet, and CIFAR 100. ST stands for Set Transformer.
RMNIST I-shot
Mini-ImageNer
CIFAR-100-FS
Method
1-shot
5-shot
1-shot
5-shot
ProtoNet
75.35 + 1.43%
39.14
0.78%
51.17 = 0.57%
38.05 + 1.56%
52.63 = 0.74%
MetaReg
76.40 + 0.56%
39.36 0.45%
50.94 1 0.67%
37.74 1 0.70%
52.73 - 1.26%
MetaMix
76.54 + 0.72%
38.25 0.09%
52.38 + 0.52%
36. + 0.63%
52.52 + 0.89%
MLTI
79.40 + 0.75%
39.69
0.47%
52.73 = 0.51%
38.81 + 0.55%
63.41 = 0.83%
ProtoNet+ST
77.38 1 2.05%
1.03% 38.93
48,92 1 0.67%
38.03 1 0.85%
50.72 - 0.92%
Meta Interpolation
83.24 + 1.39%
40.28 1 0.48%
53.06 + 0.33%
41.48 + 0.45%
54.94 + 0.80%
ProtoNet
MLTI
MetaMix
ProtoNet+ST
MetaReg
Mota Interpolation
1:23
Train
Train
0.0 3000 38dep LUG00 3000025900336609
5000 200
2003
4200
1902
32689
2002
5200
1200
Steps
(a) Train RMNIST
(b) Val. RMNIST
(c) Train NCI
(d) Val. NCI
Figure 2: (d)~(d) Meta-train and meta- meta-validatio validation loss on RMNIST and NCI for ProtoNet, MLTI, MetaMix,
ProtoNet+ST, and Meta Interpolation.
domains and tasks considered. For example, MetaReg effective on the sound domain (ESC-50) and Metabolism, but does not work on the chemical (Tox21 and NCT) and text (GLUE-SciTail) domains. Similarly, MetaMix and MLTI achieve performance improvements on some datasets but degrade the test accuracy on others. This empirical evidence supports the hypothesis that the optimal task augmentation strategy varies across domains and justifies the motivation for Meta-Interpolation which learns augmentation strategies tailored to each domain.
We provide additional experimental results on the image domain in Table 2. Again, cetta-nntrpolltoonn outperforms all the baselines. In contrast to the previous experiments, MetaReg hurts the general- ization performance on all the image datasets except on RMNIST. Note that Manifold Mixup-based augmentation methods, MetaMix and MLTI, marginally improve the generalization performance for 1-shot classification on Mini-ImageNNt. and CIFAR- 100-FF, although they boost the accuracy on 5-shot experiments. This suggests that different task augmentation strategies are required for I-shot and 5-shot for the same dataset. Meta- deta-Innerpollatonn on the other hand learns task augmentation strategies tailored for each task and dataset and consistently improves the performance of the vanilla ProtoNet for all the experiments on the image datasets.
Morcover, we plot the meta-training and meta-validation loss on RMNIST and NCI dataset in Figure 2. Interppntrpoltnoonn obtains higher training loss but much lower validation loss than the others on both datasets. This implies that interpolating only support sets constructs harder tasks that a meta-learner cannot memorize and regularizes the meta-learner for better generalization. ProtoNet overfits the meta-training tasks on both datasets. MLTI mitigates the overfitting issue on RMNIST but is not effective on the NCI dataset where it shows high validation loss in Figure 2d. On the other hand, MetaMix, which constructs new query set by interpolating a support and query set with Manifold Mixup, results in generating overly difficult tasks which causes underfitting on RMNIST where the training loss is not properly minimized in Figure 2a. However, this augmentation strategy is effective for tackling meta-overfitting on NCI where the validation loss is lower than ProtoNet. The loss curve of STt supports the claim that increasing the model size and using bilevel optimization cannot handle the few-task meta-learning problem. It shows higher validation loss on both RMNIST and NCI presented in Figure 2b and 2d. Similarly, MetaReg which learns coefficients for lz regularization fails to prevent oeva-ovenii on both datasts.
Lastly, we empirically show that the performance gains mostly come from the task augmentation with Meta-Interpolation, rather than from bilevel optimization or the introduction of extra parameters with the set function. As shown in Table and 2, ProtoNet+ST, which is Meta Interpolation but trained without any task augmentation, significantly degrades the performance of ProtoNet on Imageemm and CIFAR-100 On the other datasets, the STotoNet+ST obtains marginal improvement or largely
8

---

Original Prototype
Interpolated Prototype
NCI
NCI
ESC-50
ESC-50
(a) MLTT
(b) Meta-Interp.
(c) MLTT
(d) Meta-Interp.
Figure Visualization of original and interpolated tasks from NCI ((a) and (b)) and ESC-50 ((c) and (d)).
Table 3: Ablation study ESC-50 Table 4: Performance different
Table 5: Performance of different
dataset.
set functions on ESC-50 dataset.
interpolation ESC 50 dataset.
Model
Accuracy
Set Function
Accuracy
Interpolation Strategy
Accuracy
Interpoaooolnnnonn
79.22 0.96
ProtoNet
69.05 = 1.69
Query+SSuppor
76.87 :: 0.94
w/o Interpolation
71.54 + 1.56
Quety
78.19 = 0.84
w/o Bilevel
63:01 + 2.00
DeepSets
74.26 = 1.77
Support+ Noise
78.27 1: 1.24
w/o Canptmaa 0,00 7
78.01 1: 1.56
Set Transformer
79.22 + 0.96
Support
79.22 = 0.96
underperforms the other baselines. Thus, the task augmentation strategy of interpolating two support sets with the set function y is indeed crucial for tackling the few-task meta- meta-learning problem.
Ablation Study We further perform ablation studies to verify the effectiveness of each component of Meta- Inttaalntrpootiion.. In Table 3, we show experimental results on the ESC-50 dataset by removing various components of our model. Firstly, train model without any task interpolation but keep the set function xox, denoted as w/o Interpolation. The model without task interpolation significantly underperforms the full task-augmentation model, Meta-Interpolatton, which shows that the improvements come from task interpolation rather than the extra parameters introduced by the encoding layer. Moreover, bilevel optimization is shown be effective for estimating A, which are the parameters of the set function. Jointly training the ProtoNet and the set function without bilevel optimization, denoted as w/o Bilevel, largely degrades the test accuracy by 15%. Lastly, we remove the loss Siiiiiiooo (x, 0, T.urin) for inner optimization in Eq. 6, denoted as w/o Csingleton( (A, T. This hurts the generalization performance since it decreases the diversity of tasks and causes inconsistency between veeaattaainnng and meta-testing, since we do not perform any interpolation for support sets at meta-test time.
We also explore alternative set function such DeepSets [52] using the ESC50 dataset to show the general effectiveness of our method regardless of the set encoding scheme. In Table 4, Meta- Interpolation with DeepSets improves the generalization performance of ProtoTypical Network and the model with Set Transformer further boost the performance as a consequence of higher-order and pairwise interactions among the set elements via the attention mechanism.
Lastly, we empirically validate our interpolation strategy that mixes only support sets. We compare method to various interpolation strategies including one that mixes a support set with zero mean and unit variance Gaussian noise. In Table 5, we empirically show that the interpolation strategy which mixes only support sets outperforms the other mixing strategies. Note that interpolating a support set with gaussian noise works well on ESC50 dataset though RMNIST, from 75.35 + 1.63 to 69.60 + 1.60 as shown in Table 6, which justifies our approach of mixing two support sets.
Table 6: Comparison to interpolation with noise on ESC50.
RMNIST
Interpolation Strategy
Accuracy
Support+ Noise Support
60 1: 1.60
75.35 + 1.63
find that it significantly degrades the performance of ProtoNet on
Visualization In Figure 3, we present the t-SNE [43] visualizations of the original and interpolated tasks with MLTI and Interppntrool,,innn respectively. Following Yao et al. [50], we sample three original tasks from NCI and ESC-50 dataset, where each task is two-way five-shot and five-way five- shot classification problem, respectively. The tasks are interpolated with MITI or Mete--nneooonattoon to construct 300 additional tasks and represented as set of all class prototypes. To visualize the

---

prototypes, we first perform Principal Component Analysis [13] (PCA) to reduce the dimension of each prototype. The first 50 principal components are then used compute the t-SNE visualizations. As shown in Figure 3b and 3d, Meta-Interpolation successfully learns an expressive neural set function that densifies the task distribution. The task augmentations with ML.T,, however, do not cover a wide embedding space as shown in Figure 3a and 3c as the mixup strategy allows to generate tasks only on the simplex defined by the given set of tasks.
Limitation
Although we have shown promising results in various domains, our method requires extra compu- tation for bilevel optimization to estimate X, the parameters of the set function 4ax, which makes it challenging to apply our method to gradient based meta-learning methods such as MAML. Moreover, interpolation is limited to classification problem and is not straightforward apply it to regres- sion tasks. Reducing the computational cost for bilevel optimization and extending our framework to regression will be important for future work.
Conclusion
We proposed novel domain-agnostic task augmentation method, Meta Interpolation, tackle the meta-overfitting problem in few-task meta-learning. Specifically, we leveraged expressive neural set functions to interpolate a given set of tasks and trained the interpolating function using bilevel optimization, so that the meta-learner trained with the augmented tasks generalizes to meta-validation tasks. Since the set function is optimized to minimize the loss on the validation tasks, allows to tailor the task augmentation strategy to each specific domain. We empirically validated the efficacy of our proposed method various domains, including image classification, chemical property predic- tion, text and sound classification, showing that Meta- Meta-Interpolation achieves consistent improvements across all domains. This stark contrast to the baselines which improve generalization certain domains but degenerate performance in others. Furthermore, our theoretical analysis shed light on how Meta- deta-Innerpolation regularizes the meta-learner and improves its generalization performance. Lastly, we discussed the limitation of our method.
Acknowledgments
This work was supported by Institute of Information & communications Technology Planning &
Evaluation (IITP) grant funded by the Korea governmenn(MSIT) (No.2019-0-00072) Artificial In- telligence Graduate School Program(KAIST) the Engineering Research Center Program through National Research Foundation Korea (NRF) funded by the Korean Government MSIT (NRF- 2018R1 A5A 1059921), Institute of Information & communications Technology Planning & Evalua- tion (ITP) grant funded by the Korea governmen((MSIT) (No. 202 1011-000268, Artificial Intelligence Innovation Hub), the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-202 IRIFI IF FIRIFIAIOO 10616555) Institute Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea (OVE) (No.2022-4) 2022-0-00713: Samsung Electronics (IO2012144 0400211-08111 14408844--01 14--01), and Google Research Grant. It was also results of study the "HPC Support" Project, supported by the 'Ministry of Science and ICT and NIPA.
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint arXiv:1607.06450 2016.
[2] Yogesh Balaji, Swami Sankaranarayanan, and Rama Chellappa. MetaReg: Towards domain generalization using meta-regularrization Advances in neural information processing systems, 31, 2018
[3] Yoshua Bengio, Samy Bengio, and Jocelyn Cloutier. Learning synaptic learning rule. In MICNN-91-Seatle International Joint Conference on Neural Networks, volume 2, pages vo9-vol.
IEEE, 1991.
10

---

[4] Charles Blair. The computational complexity of multi-level linear programs. Annals of Opera- tions Research, 34, 1992.
[5] Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, 2015.
[6] Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. In International Conference on Learning Reprexentations, 2021.
[7] Richard Li- LinY Chen, Amy Cohn, and Ali Pinar. An implicit optimization approach for survivable network design. In 2011 IEEE network science workshop, pages 180-187. IEEE, 2011.
[8] Kevin Clark, Thang Luong, Quoc V. Le, and Christopher D. Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2020.
[9] Li Deng. The mnist database of handwritten digit images for machine learning research [best of the web]. IEEE signal processing magazine, 29(6):14--141 141-142, 2012.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
[11] Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta- weta-learning. In International Conference Machine Learning, pages 1920-1930. PMLR, 2019.
[12] Sebastian Flennerhag, Andrei A. Rusu, Razvan Pascanu, Francesco Visin, Hujun Yin, and Raia Hadsell. Meta-learning with warped gradient descent. In International Conference on Learning Representations, 2020.
[13] Karl Pearson F.R.S. Lii. On lines and planes of closest fit to systems points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11): 559-572, 1901. doi: 10. 10110000448644000999666
[14] Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient-based meta-lcarning as hierarchical bayes. International Conference on Learning Representations, 2019.
[15] Greg Landrum. RDKit: Open-source cheminformatics software., 2018. https://github com com/rdkitrrdkis
[16] Shawn Hershey, Sourish Chaudhuri, Daniel PW Ellis, Jort F Gemmeke, Aren Jansen, R Chan- ning Moore, Manoj Plakal, Devin Platt, Rif A Saurous, Bryan Seybold, et al. CNN architectures for large-scale audio classification. In 20/7 IEEE International Conference Acoustics, Speech and Signal Processing, ICASSP, pages 131-135. IEEE, 2017.
[17] Kexin Huang, Tianfan Fu, Wenhao Gao, Yue Zhao, Yusuf Roohani, Jure Leskovec, Connor W Coley, Cao Xiao, Jimeng Sun, and Marinka Zitmik. Therapeutics data commons: machine learning datasets and tasks for therapeutics. arXiv e-prints, pages arXiv-2102, 2021.
[18] Ruili Huang. Menghang Xia, Dac- Bac-Trung Nguyen, Tongan Zhao, Srilatha Sakamuru, Jinghua Zhao, Sampada A Shahane, Anna Rossoshek, and Anton Simeonov. loxllchalennge to build predictive models of nuclear receptor and stress response pathways as mediated by exposure to environmental chemicals and drugs. Frontiers in Environmental Science, 3:85, 2016.
[19] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMLR, 2015.
[20] Tushar Khot, Ashish Sabharwal, and Peter Clark, SeiTail: textual entailment dataset from science question answering. In Thirty-Secomt AAAI Conference on Artificial Intelligence, 2018.

---

[21] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations. 2015.
[22] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. In https://www..cstorrnttoco kriz/eifar.html Citeseer. 2009.
[23] Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention- based cermutation-invariii neural networks, In International Conference on Machine Learning, pages 3744-3753, PMLR, 2019.
[24] Yoonho Lee and Seungjin Choi. Gradient-based meta-leaming with learned layerwise metric and subspace. In International Conference on Machine Learning, pages 2927-2936. PMLR, 2018.
[25] Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Saško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. Datasets: A community library for natural language processing. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 175-184, Online and Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics.
[26] Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, Eunho Yang. Sungju Hwang, and Yi Yang. Learning propagate labels: Transductive propagation network for few-shot learning. In International Conference on Learning Representations, 2019.
[27] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pages 1540-1552. PMLR, 2020.
[28] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019.
[29] Nikhil Mishra, Mostafa Rohaninejad, Xi Chen, and Pieter Abbeel. A simple neural attentive meta-learner. In International Conference on Learning Representations, 2018
[30] Shikhar Murty, Tatsunori B Hashimoto, and Christopher D Manning. DReCa: A general task augmentation strategy for few-shot natural language inference. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 111 1113-1125, 2021.
[31] NCI. NCI dataset, 2018. https:://gttheieee cotp::/....ttub.oooooooooo wwt.ccoo///caaeeeee CA////RAADD-LD Lab/ BRRAND-Labgrraph_dattteeee grtpggaaaataetsss.....
[32] Renkun Ni, Micah Goldblum, Amr Sharaf, Kezhi Kong, and Tom Goldstein. Data augmentation for meta-learning International Conference on Machine Learning, pages 1152-8161. PMLR, 2021.
[33] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02000 2018.
[34] Karol Piezak. ESC: Dataset for Environmental Sound Classification, In Proceedings of the 23rd Annual ACM Conference Multimedia, pages 1015-1018. ACM Press, 2015. ISBN 978-1-4503-3459-4 doi: 10. 10.1145/27333333286666
URL http: acm. www.ac/.................o.ooo.o. citation.
ctm/doid-2733377.. 2280773333333.2069..
[35] Janarthanan Rajendran, Alexander Irpan, and Eric Jang. Meta-learning requires meta- augmentation. Advances in Neural Information Processing Systems, 33:5705-5715, 2020.
[36] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019.
12

---

[37] Andrei A. RUsu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals. Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2019.
[38] Victor Garcia Satorras and Joan Bruna Estrach. Few-shot learning with graph neural networks. In International Conference on Learning Representations, 2018.
[39] Jurgen Schmidhuber. Evolutionary principles in self-referential learning. On learning how to learn: The meta-meia-.. hook.) Diploma thesis, Institut Informatik, Tech. Univ. Munich, 1(2), 1987.
[40] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances neural information processing systems, 30, 2017.
[41] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15()):1929-1958, 2014.
[42] Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208, 2018.
[43] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(1)) 2008.
[44] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez- Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages
PMLR, 2019.
[45] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[46] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
[47] Adina Williams, Nikita Nangia, and Samuel Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Pupers), pages 1112-1122, 2018.
[48] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguistics.
[49] Huaxiu Yao, Long-Kai Kai Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, et al. Improving generalization in meta-learning via task augmentation. In International Conference on Machine Learning, pages 11887-11897. PMLR, 2021.
[50] Huaxiu Yao, Linjun Zhang, and Chelsea Finn. Meta-learning with fewer tasks through task interpolation. In International Conference Learning Representations, 2022.
[51] Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta- learning without memorization. In International Conference on Learning Representations, 2020.
[52] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see limitation from
Section 6
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We run experiments times with different random seeds and show the 95% confidence interval on Table I and 2.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We describe the datasets and cite the creators in Section 5.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "i0XqiG-bvM", "reviewer": "Reviewer_vsLv", "summary": "The paper explores few-task meta-learning that attempts to ease the difficulty and cost of massive task construction. It challenges the existing few-task meta-learning approaches that heavily rely on the Manifold-Mixup-based task augmentation, which is hard to generalize to cases beyond computer vision. In this case, the author introduces a set-value neural net (Set transformer) as the tool to augment the tasks. The author provides a theorectical ground that Set transformer is able to control the Rademacher complexity for better generalization. 5 dataset and 3 image-based FS benchmarks have been used to evaluate the method.", "questions": "The paper may deserve a higher rating score if the questions below have been fully addressed.\n1. Only Set-transformer has been considered as the implementation of meta-interpolation. Will other neural set functions be able to improve the few-task FS performance? It would be more convincing if the author can provide more empirical analysis on how to choose the set neural networks, e.g., change the architecture of the Set-transformer or use other types of set neural networks (for instance, Deep Set).\n2. The method is claimed to be good at the field beyond vision. How is the performance in solving NLP low-resource problems? \n3. Please refer to the first statement in limitation.", "limitations": "1. Distinct from the typical Mixup-based task augmentation baselines, the method is only implemented (only available?) in metric-based meta-learning, more precisely. ProtoNet. The author should provide more discussion of why meta-interpolation is limited in metric-based meta-learning, otherwise, provide the empirical performances of other meta-learning frameworks (e.g., MAML and more others) + meta-interpolation and further compare them with typical Mixup-based task augmentation baselines.\n2. The method is a little bit incremental.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The paper is well-organized and well-written.\n2. The method seems simple but quite general and effective across many scenarios.\n3. Theoretical results significantly motivate the method.\n4. Massive empirical studies and the results are convincing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "byYCaa__H16", "reviewer": "Reviewer_7Fn3", "summary": "The paper introduces a new task augmentation method for meta-learning with few training tasks available. The proposed method employs a set-to-set function (instantiated as set transformer) that performs interpolation of features from the support set to create a new task. The parameters of a set transformer are learned via billevel optimization, using a meta-validataion dataset for tuning the parameters (considered as hyperparameters) to achieve the generalization. The paper performs experiments on the eight datasets to present the effectiveness of the proposed method, Meta-Interpolation.", "questions": "- What is the sensitivity of the proposed method to meta-validation dataset size, compared to other works, such as MLTI [48] that also performs feature interpolation?\n- At which layer is the set transformer located? Is the location randomly sampled, similar to MLTI? If not, why so? Could you provide the ablation study on this?\n- Why is the proposed method not applied to MAML? The similar work MLTI [48], which also performs feature interpolation, has shown the effectiveness on MAML as well. I believe providing experimental results on MAML will strengthen the paper.", "limitations": "While the paper provides limitations, I believe there are more limitations to discuss:\n- the possible need for large meta-validation dataset (such high demand may break the assumption of few tasks). \n- Validated only for prototypical networks", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The paper presents extensive experimental results across diverse datasets.\n- The idea of employing meta-learnable set-to-set function for feature interpolation is novel.\n\n###  Weakness\n- The number of parameters of the set transformer may be large enough that the proposed method may be sensitive to the size of meta-validation dataset. In fact, I noticed that the ratio of meta-training:meta-validation is quite high (e.g., 12:16 in the case of Mini-Imagenet-S).\n- The proposed method is applied to only prototypical networks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Xu-XiIShmQ", "reviewer": "Reviewer_Q5Gf", "summary": "To tackle meta-overfitting in few-task meta-learning, the authors propose Meta Interpolation based on set transformer to generate domain-agnostic tasks. They assign a new class k by re-arrang K classes in two tasks, and the new class k is the k_{th} class in the two new permutations. Then, the corresponding support set is constructed with interpolating function using bilevel optimization. Since the interpolating function is optimized to minimize the loss on the validation tasks, the proposed method can achieve task augmentation for each specific domain. The effectiveness of Meta Interpolation is proved both experimentally and theoretically.", "questions": "1. Please analyse the principle of generating new classes by randomly aligning the category orders in two tasks. Since its prototype contains information of two classes, which may be quite different.\n2. Why change the support set and keep the query set unchanged? If two new tasks contain the same query set but different support set, then, how to avoid the optimized support sets falling into the original one in task 1?\n3.  Does this augmentation make task 1 and task 2 more similar?\n4. More training details should be provided. For example, how many new tasks do you generate? Do the generated tasks make the comparison with other methods unfair (e.g., the model is updated more times during one training epoch, since more tasks are optimized in one epoch)? And which layer do you perform augmentation?\n5. An ablation study about the number of training tasks and the performance of the proposed method, and the layer to implement augmentation may be better.", "limitations": "An ablation study about the number of training tasks and the performance of the proposed method may be better show the practical applicability.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. The task of task generation is interesting.\n2. The effectiveness of the proposed method is proved both experimentally and theoretically.\n\nCons:\n1. The reasonability of using the he interpolating function to generate new support set, while keeping the query set unchanged is unclear.\n2.  The way to generate new classes is somewhat hasty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4sXDdLvo3FK", "reviewer": "Reviewer_w9NJ", "summary": "This paper presents a method for combining tasks in meta-learning when original base class domain data is scarce.  At meta-training time, a set transformer is used combine support set features from two random tasks, while comparing to queries from just the first.  The set transformer can therefore use the second random support set to apply random augmentations to the first, or not, as beneficial for the application.  Since the meta-training data is limited and easy to overfit, the random augmentations might never be used when trained jointly --- the transformer parameters are instead learned on a validation set with separate set of classes, using bilevel optimization with HyperGrad.  The method is evaluated on datasets in multiple domains, including chemical, text, speech and images, finding consistent improvements.\n", "questions": "Additional questions:\n\nSince original base data is scarce to begin with, splitting out additional classes into a separate validation set to train the set transformer seems like a lot of data to take away from training the feature model parameters.  Is there a way to continue using these for both features and set transformer, maybe changing the held-out classes used in hypergrad periodically each outer loop?\n\nHow important is the second support set for use in the transformer?  What if random vectors are input to the set transformer instead of task 2 vectors?\n\nSince there is just one set of set transformer params $\\lambda$, does this limit the amount of augmentation or interpolation?  If the same data elements and permutations are used twice with same lambda params, the set transformer will output the same features both times, unless there is another source of randomness in the transformer that I missed.  Does this limit the approach at all?  Fig 3b shows a likely considerable amount of interpolation using several combinations of points, but could even more be generated from each pair using additional random sampling either of the transformer params or auxiliary input?\n\n\nl.155 \"interpolating either support or query sets achieves higher loss than interpolating both\" --- reading this sentence alone (before sec 5), it's ambiguous if what is meant by \"higher loss\" is better or worse.  Reading this section, this question was very much on my mind, so seeing that it would be addressed by this sentence was great, but it wasn't clear which way it would be resolved.\n\nalg.2:  some small typos and inconsistencies, lines 4,6 L_v suggests this is the loss but is already the grad, would be more consistent to accumulate a sum of losses in L_v and then set v_1 <- d/d\\theta L_v.  also \"single\" is misspelled in the subscript\n\n", "limitations": "-", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This is an interesting approach that furthers work in task augmentation, extending beyond interpolation to learned mixing.  It's a little bit complex, using a set transformer, separate class split and bilevel optimization, and it's not entirely clear all components are necessary in their current form --- however, there are ablations showing each contributes, and particularly that a separate optimization is required.  I have a few questions below on how these might be able to be simplified, but also think the system is reasonable as-is to demonstrate the effectiveness of the approach.\n\nThere is also a theoretical section, but was very dense and a little outside my expertise; this claims to show a reduction in Rademacher complexity in simplified settings, but I didn't follow it in detail.  The end result of this section also seems to rely on reducing the upper bound by decreasing the rank of a covariance-like matrix of task vector differences, though no discussion on how or why its rank might be reduced.\n\nOverall, the method is explained well and empirical results demonstrate its effectiveness, with particularly strong improvement in ESC-50.  The degree to which all components are needed in their current form, or whether they can be simplified, is unclear.  I would have have liked to see additional ablations on the form of the set transformer, random mixing vector selection, and validation set sizes.  But the paper does present ablations showing all components are indeed needed together at a high-level to justify the approach.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept