Adversarial Task Up-sampling for Meta-learning
Authors: Yichen Wu
Long-Kai Huang
Ying Wei
Venue: NeurIPS 2022 Accept (2022)
Keywords: Task augmentation
Meta-learning

Abstract: The success of meta-learning on existing benchmarks is predicated on the assumption that the distribution of meta-training tasks covers meta-testing tasks. Frequent violation of the assumption in applications with either insufficient tasks or a very narrow meta-training task distribution leads to memorization or learner overfitting. Recent solutions have pursued augmentation of meta-training tasks, while it is still an open question to generate both correct and sufficiently imaginary tasks. In this paper, we seek an approach that up-samples meta-training tasks from the task representation via a task up-sampling network. Besides, the resulting approach named Adversarial Task Up-sampling (ATU) suffices to generate tasks that can maximally contribute to the latest meta-learner by maximizing an adversarial loss. On few-shot sine regression and image classification datasets, we empirically validate the marked improvement of ATU over state-of-the-art task augmentation strategies in the meta-testing performance and also the quality of up-sampled tasks.

Adversarial Task Up-sampling for Meta-learning
Yichen Wu
Long-Kai Huang"! Ying Weilt
Cityy University of Hong Kong, TTencent AI Lab (wuyichen.amfl am97, hlongkai @ @gmail.com, Singwei@cityu.ddu..
Abstract
The success of meta-learning on existing benchmarks is predicated on the assump- tion that the distribution of metaatraining tasks covers meta-testing tasks, Frequent violation of the assumption in applications with either insufficient tasks or a very narrow meta-training task distribution leads memorization learner overfitting. Recent solutions have pursued augmentation of meta-training tasks, while it is still an open question to generate both correct and sufficiently imaginary tasks. In this paper, we seek an approach that up-samples meta-training teainnta tasks from the task rep- resentation via a task up- sampling network. Besides, the resulting approach named Adversarial Task Up-sampling (ATU) suffices to generate tasks that can maximally contribute to the latest meta-learner by maximizing an adversarial loss. On few- shot sine regression and image classification datasets, we empirically validate the marked improvement of ATU over ofat-of-thhe-rt task augmentation strategies the meta-testing performance and also the quality of up-sampled tasks.
Introduction
The past few years have seen the burgeoning development of meta-learning, a.k.a. learning to learn, which draws upon the meta-knowledge learned from previous tasks (i.e., meta-training tasks) to expedite the learning of novel tasks (i.c., meta-testing tasks) with a few examples. sufficient number and diversity of meta-training tasks are pivotal for the generalization capability of the meta-knowled so that (1) they cover the true task distribution (i.e., environment [4) from which meta-testing tasks are sampled, discouraging learner overfitting (23) and (2) the meta-knowled empowers fast adaptation via the support set for each task, avoiding memorization overfitting 44. Notwithstanding up millions of meta-training tasks in benchmark datasets (24 31), real-world applications such drug discovery (40) and medical image diagnosis [14] usually have access to only thousands or hundreds of tasks, which puts the meta-knowledge at high risk of learner and memorization overfitting.
While early attempts towards improving the generalization capability the meta-knn -otllleewweedee revolve around regularization methods that limit the capacity of the meta-knowledde [I]
44) recent works on
augmentation of meta-training tasks have shown a marked improvement 20
40 43 The objective
of task augmentation is to draw the empirical task distribution which is formed by assembling Dirac delta functions located in each meta-training task closer to the true task distribution. Consequently, achieving this objective requires qualified task augmentation approach to simultaneously possess the following three properties: (1) task-aware: the augmented tasks comply with the true task distribution, being erroneous to lead the neta-knowledge astray (tasks A and B in Figure [T]; (2) task-imaginary: the augmented tasks cover a substantial portion the true distribution, embracing task diversity which task-awareness is nonetheless inadequate guarantee (tasks C, D, and E in Figure Πb);; (3) model-adaptive: the augmented tasks are timely in improving the current meta- knowledge, to which the meta-knowledge before augmentation struggles to generalize (task F in Figure Ip).
Parr of the work was done when the author interned in Tencent Al Lab. *Corresponding author: Long-Ka Kai Huang and Ying Wei
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

True distribution
Empirical task distrbution
Sampled lask
Augmented lask
Erronsous lask
task 1
task
qjany ant
(SICRIOTI trok
(a) task-aware
(b) task-imaginary
(c) model-adeptivee
(d) truk distribution
(e) an example entneous task
Figure 1: Pictorial illustration of the three characteristics possessed by a qualified task augmentation approach, i.e., being (a) task-aware, (b) task -mmagiir,,,,,, and (c) model-adaptive. (d) shows the true task distribution that task augmentation aims to approximate, and (e) presents an example erroneous task violation of the true task distribution of sinusoidal functions y w sin(x: (w E [0.
Unfortunately, developing such a qualified task augmentation approach remains challenging. First, the task-aware methods sacrifice task diversity for aws-enwasssess they establish task-awsreness by injection of the same random noise to labels of the support and query set [23], rotation of both support and query images [20], or mix-up of support and query examples within each task [40], all of which result in augmented tasks that are within the immediate vicinity of sampled meta-training training tasks 43 that mixes up both support examples of two distinct tasks and their query examples in the feature space compromises on task awaeenssssssss resulting examples are even multi-modal in Figure e and constitute an erroneous task that fails to comply with the task distribution. Third, how to adaptively augment tasks that maximally improve the
as shown in Figure [T.. Second, the tusk-imaginary method
and thereby the performance on meta-testing tasks remains unexplored.
To this end, we propose the Adversarial Task Up-sampling (ATU) framework to augment tasks that aware of the task distribution, imaginary, and adaptive to the current vetaa-knowledge. Grounded on gradient-based meta-lcarning algorithms that are generally applicable to either regression or classification problems, ATU takes the initialization of the base learner as the meta-knowledge. Concretely, ATU consists of task samplin network whose input is task itself and outputs are augmented tasks. To ensure that the augmented tasks are imaginary and meanwhile faithful to the underlying task distribution, we train the up-sampling network to minimize the Earth Mover Distance between augmented tasks and the local task distribution characterized by a set of sampled tasks. Besides, we enforce the up-sampling network to produce challenging tasks that complement the current initialization, by maximizing the loss of the model adapted from the initialization on their query examples and minimizing the similarity between the gradient of the initialization with respect to their support examples and that of their query examples.
In summary, our main contributions are three-fold: (1) we present the first task-level augmentation network that learns to generate tasks that simultaneously meet the qualifications of being task- aware, task- mask-imaginary, and model-adaptive. (2) we provide a theoretical analysis to justify that the proposed ATU framework indeed promotes task-awareness; (3) we conduct comprehensive experiments covering both regression and classification problems and total of five datasets, where the proposed ATU improves the generalization ability of gradient- based meta-learning algorithms by up to 3.81%.
2 Related Work
As paradigm that effectively adapts the meta-knowledge learned from past tasks to accelerate the learn- ing of new ones, meta-learning has sparked considerable interest in many
Table 1: Summary of existing task augmentation strategies.
Method
Task-aware
Task-imaginary
Model-adaptive
MetaAug
28
X
MetaMix
40
Meta-Maxup
scenarios [38 26 36 35), especially
MLTI
43
X
for few-shot learning. It falls into
four major strands based on what the meta-kn is, i.e., optimizer-based methods 37
feed-forward methods [24] [0 39], metric-based methods [27] 29 33) and gradient-based meth-

---

ods 19 42 12 6], where the inner optimizer, the mapping function from the support set to the musk-specific model, the distance metric measuring the similarity between samples, and the parameter initialization are formulated as the meta- knowledge that enables quick adaptation to task within small number of steps. Our method primarily evaluated on gradient- based methods which enjoy wide adoption and applicability in either classification or regression problems.
Within-task Overfitting.
Few-shot learning puts meta-learning, especially gradient-basee methods which require optimization of high-dimensional parameters within each task, at risk of within-task
overfitting. Some works tackle the problem with various ways of reducing the number of parameters to adapt in the inner loop: only updating the head [22] or the feature extractor [21], learning data- dependent latent generative embeddings of parameters [25] or context parameters [48] adapt, imposing gradient dropout [32], and generating stochastic input-dependent perturbations [3]. The other bunch of works alleviates the problem through data augmentation within each task. Ni et al. [20 applies standard augmentation like Random Crop and CutMix onto support samples. Sun et al. [28 and Zhang et al. 47 proposed to generate more data within class via ball generator and generative adversarial networks, respectively. These techniques designed for within-task overlitting, however, have been proved to lend little support to meta-overfitting which we focus on.
Meta-overfitting. Distinguished from traditional overfitting within a task, two types of meta-
overfitting including memorization and learner overfitting have been pinpointed in 44 23 Despite
retaleggulaiization techniques [[] 44 that limit the capacity of the meta-learner, task augmentation 16
strategies [3 20
43
40 have emerged as more effective solutions
meta-overfitting, Table
presents summary of these strategies, except the strategies of large rotation [16] being part of Meta- Maxup 20 and DReCa [9] applicable to natural language inference tasks only. MetaAug (23 augments a task by adding a random noise on labels of both support and query sets, and MetaMix 40)
mixes support and query examples within a task. Such within-task augmentation guarantees the validity of augmented tasks, i.e., being task-aware, though it almost does not alter the mapping from the support to the query set, i.e., generating limited imaginary tasks beyond meta-training tasks. Meta-Maxup (20) and MLTI 43 approach this problem via the cross-task mixup method, unfavorably at the expense erroneous tasks. Our work seeks a novel task augmentation framework capable of generating tasks that not only meet the task-awareness. and task -mssinnaannnioon needs but also adapt to maximally benefit the up-to-the-mine the-minute meta-learner.
3 Preliminaries
3.1 Meta-Learning Problem and Gradient- Meta-Learning
Meta-Learning model f are trained and evaluated on episodes of few-shot learning tasks. Assume the task distribution is p(7). A few-shot learning task T; i.i.d. sampled from p(T) consists of a support set DJ = (XX = { ) and a query set D3 = (X7, Y??) = {(02, v where X; and Y,,, (X² and Y1" are the collection of inputs and labels in support (query) set, and K (K") the size of support (query) set.
The most representative gradient- baseed meta-learning algorithm is MAML 9. MAML aims to learn an initialization parameter Og of the model f that can be adapted to any new task after few steps of gradient update. Concretely, given a specific task D; DJ and a parametric model fe. MAML initializes the model parameter by 0g and updates by performing gradient descent on the support set D.. It then optimizes the initialization parameter 0n by minimizing the loss L estimated on the query set D. The objective of MAML can be formulated as
min TTD((T L(di, D?):
S.t.
Φi = 0o - -oVo, D;)
3.2 Earth Mover's Distance
To estimate the distance between two tasks, we use Earth-Mover Distance (EMD). Earth-Mover Distance, a.k.a. Wasserstein metric, is distance measure of two probability distributions or two sets of points, and is widely used in image retrieval and point cloud ua-sampling works [46 45 Given two sets S and S2 with the same size, EMD calculates their distances as:
dEMD(S
min o.
Σ sES.
||s ((s)
(2)
||
3

---

Stage-I
Perturbations
Up-sampled
Groun-Truth T
Coarse Te
Geuminr
gr(-)
Decoder
au(3)
gr(-)
B. Patch Feature
Meta Learner
degaa up (EMD Los)
Latent lack manifold
Task Patch 7
Stage=II
Fiyl (ates,
(Adversarial Loss)
Figure 2: Illustration of the ATU algorithm with a task up-sampling network. The task up-sampling network consists of encoder g.(-) which extracts set feature of the input task patch, coarse task generator ge(*) which generates coarse tasks given the task patch, and a decoder ga(-) which generates fine tasks from coarse tasks based random perturbance and the set feature.
where ø is a bijective projection mapping S1 to S2. The value of EMD in 2 can be obtained by solving the linear programming problem w.r.t. ø.
4 Adversarial Task Up-sampling
In practice, the task distribution p(7) unknown and we optimize the meta parameter dn with an empirical estimation of Eq. ① over of meta-training tasks {T as
Ny
min 80
NT Σ C(o, Df),
s.t.
Φ = 00 a a C(0,,
(3)
Given a finite set of meta-training tasks, the empirical task distribution may deviate from the true task distribution. The meta-model trained on such a finite set of tasks will cause memorization or learner overfitting [4 23), which hurts the generalization
tasks. To alleviate this problem, we propose a new task up-sampling network to generate a sufficient number of diverse tasks such that the empirical task distribution formed by the original meta-training tasks and the augmented tasks together is closer to the true task distribution. To achieve this, the tasks generated by the task up-sampling network should match the true task distribution and cover a large fraction of However, since the true task distribution and its underlying manifold are unknown, we cannot provide the task up- sampligg network with explicit information about it. Instead, we generate new tasks by performing Task Up-sampling (TU) from a set of training tasks that implicitly comprise the latent task manifold information. The idea of Task Up-sampling is inspired by point cloud up-sampling methods [46 45, which generate up-sampled points lying on the latent distribution (i.e., the shape) of the given local point patch. Similar to the point cloud up-sampling algorithm, our augmentation network receives task patch consisting of set of tasks T, = {Ti)
N₂ where Np is the set size, and generates up-sampled tasks Tup {D*, Day that uniformly distributed over the same underlying task distribution as the task patch.
Due to the high complexity of task generation, it is infeasible to directly generate up-sampling tasks without sacrificing the quality of the tasks. Inspired by [46), we propose a two-stage generation strategy generate the up-sampled tasks. In the first stage, we produce sparse set of tasks, aiming recovering the global task distribution of the task patch. The tasks obtained in the first stage are called coarse tasks. In the second stage, we generate multiple tasks for each coarse task, aiming to characterize the local task distribution around each coarse task. To guide the second generation stage, the patch features of the input task set as input to provide global task information and also multiple random noise vectors as input to provide directional perturbations to generate diverse tasks around the coarse task. The generation process is summarized in Fig. 2
Our proposed task up up-sampling network consists of 3 components, namely, coarse task generator ge(-), set encoder (-). and decoder (-). The coarse task generator is similar to a set auto- encoder. first encodes the information of the whole input task set and then decodes to generate Ta coarse tasks T. ). The set encoder, denoted by (.(.), extracts the set information of the input task patch as a patch feature ht to provide global information in the second-stage generation.
For each task TV in Te, the decoder ga(-) generates ra tasks located around Te in the task manifold

---

by taking as input ra random perturbations {2 that are i..d. noise sampled from a uniform distribution and the set feature hg as input. In general, we use the same perturbations for each coarse task Ty in Te. Finally, we obtain the up-sampled task set Tap consisting of rNp tasks as Tup= = (Te 2, ha), where r = re x ra the up-sampling ratio. We denote the task augmentation network by Go, (Tp: 2) where fg is the trainable parameters of the task augmentation network.
In each iteration of the training phase, we construct r.N, tasks to form the ground truth tasks set To (e.g., randomly select from the meta training task set), Then we sample Np tasks from To to form the task patch Tp and randomly sample perturbation noise vectors to form the perturbation set 2 = {2 Hi By feeding Tp and 2 to the task augmentation network, we obtain the up-sampled task set Tup
(T., 2). To train the task augmentation network, we apply EMD loss between the up-sampled task set Tup and the ground truth task set T to encourage the generated task set to have the same distribution as the true task distribution. However, it may still be insufficient to make the up-sad tasks cover a significant fraction of the true task distribution. In this case, the up-sampling tasks provide limited additional information compared with the original meta-training tasks, and thus the meta-learner has restricted benefit from the generated tasks. To generate more informative tasks for the meta-learner, we want the generated tasks to be difficult for the current meta model On. Following [41, we measure the difficulty of the task for Oo by the loss estimated on query set w.r.t. to dt, i... L(o,, Df), and the gradient similarity between the support and query sets w.r.t. do. i.e. (Vo, D;), Ve, C(Oo, D.)). The large loss and small gradient similarity indicate a difficult task. Therefore, we want to maximize the following objective function to generate informative tasks:
(4) where 7/ and 77 are two hyperparameters that control the strength of the two terms in Lade- We call this loss adversarial loss because it aims to increase the difficulty of the up-sampling tasks for the meta-learner while the meta-learner is trained to minimize the loss on the generated difficult tasks. And named the proposed algorithm Adversarial Task Up-sampling (ATU).
Lado (Di, D.) m C(o, D 72 8 [(0o, DJ), V Va, ((0o, D()),
Together with the EMD loss, we obtain the objective to train the task up-sampling network:
LATU Tp) = dEMD (Tup: Ta)
1
Σ TETT.P
Lado (D; D?))-
(5)
NN
will not be backpropagated to the meta model Og and the meta model will be updated by minimizing the meta loss in Eq. (3) on the up-sampled tasks Tap We summarize the proposed ATU in Algorithm 1 in Appendix A.
Note that the gradient of Eq. 5
4.1 ATU on Regression and Classification Problem
Before introducing the details of regression and classification tasks, let us first review the Eq. (2) where S1 and S2 can be understood as the set of up-sampled tasks Tup and the set of ground-truth tasks Tg. respectielly Each point s in either set represents the embedding of task.
Regression Tasks. We consider a simple regression problem: sinusoidal regression, which is widely used to evaluate the effectiveness of the meta-learning methods. In sinusoidal re- gression problem, before feeding a task T, = (D; Dy) to the task up-sampling network, we need to present the embedding of sine regression task at first. In this paper, we combine all samples of the support set and query set as the embedding of a sine regression task, i.c.,
s = [-,, æa, v/..... Jk.; vk. 21: vi, 2², yg. x Ku: yico] E R2(K**K), where we sort the S Tkk and 21 S 1= < ... S zkvv This sorting
support set and query set such that xj x2 could make the task input is invariant to the permutation of data in support and query sets and thus the extracted feature of each task is permutation-invarian
which simplifies the design of the task sammmll network. To generate the coarse tasks, we first use the set encoder 9=(-) to extract the set feature h, and directly generate the coarse tasks from the set feature hy. Then we generate the up-sampled tasks Tup utilizing the decoder ga(-) with perturbations 2. Consequently, the dimension of generated tasks Tup is (r. KK K))), and that of ground truth tasks Tr is also (r:Np, 2(K', K))), where r is the up-sampling ratio and N₂ is the set size of a task patch. For the regression problem, we add an extra EMD loss on the support and query set for each generated task T, E Tup to encourage the points in the generated support set and query set to follow the same sinusoidal distribution, and the objective is shown in Eq. (
5

---

LoTu(Os Tp) - -DEMD(TTp +1) rNp 7rETup
Σ
dEMD(D;, D2) rNp T:EToY
1
Σ
Loav (Oo, (Di, D.).
(6)
where T, = =D; Df) is obtained by transforming each of Tup back support set and query set. Classification Tasks. Dissimilar to regression tasks, classification tasks' labels of each class are randomly given under the mutually-exclusive setting 44]. For example, for an N-way Khtshoo classification problem with K query samples for each class, the label y in episodic-based meta- training is randomly chosen value from {0, 1, N - }). In light of the fact that the label is not semantically meaningful, we only use the images r to represent the embedding of an N W-way classification task. Concretely,
reshape each task into task pool of (K*+K99) tasks, each of which is N-way 1-shot and represented as s = E2..22..... INl E IR Nd where d is the dimension of each image. Then, we can split the task into (K*+K") N-way 1-shot classification tasks without query examples. We represent the task by concatenating the input from N classes in fixed order (based on classes). We treat the (K*+K9) tasks as a task patch and feed them to the task up-sampling network.
Since the task distribution of the image classification problem is extremely complex, it is impractical generate the coarse tasks from set feature. Instead, we use the original tasks as the coarse tasks and generate the up-sampled tasks by more informative perturbance around the original tasks. To achieve this, we generate the perturbation by randomly sampling extra aas images from N different classes in the base set. Then for the image in a class of a task in the coarse tasks Te, we subtract from the Кι images to obtain KM residual images and their corresponding set features (i.e.. obtained from 9, (-)). concatenate the set features with noise vector, and use an attention network to obtain a residual images feature z!'s for the image 2; given K as residual images. Finally, we generate the image x. = Ii + +x. for the augmented task. We repeat this process for all images in a coarse task to get an augmented task and apply the r noise vector to get r up-sampled tasks. The dimension of generated tasks Tup and ground truth tasks T, are both (r(K* + K9), Nd), where T is obtained by mixing up with images in the memory bank. The whole training objective function is Eq. (5). More details of network structures and training details are shown in Appendix B.
5 Theoretical Analysis
We will introduce the formal definition of an up-sampled task that conforms to task-awareness, based on which we present the essential property of our proposed ATU framework in maximizing the lask-awareness, compared to previous task augmentation approaches.
Definition 1 (Task-aware Up-sampling). Suppose that we are given a set of Np tasks (X,, Y,) Np from which we up-sample new task Tup- For each i-th task, its ground-truth parameter that map the input X; to the output Y, is 0,, i.c., Y, fe, (X.). The up-sampled task Tup (X.., Y., is defined to be task-aware, if and only if 0g = g(f,. ... ,ON.. and Y, = fo. (X.) where is the up-sampling function and 0,, is the up-sampled parameter.
This definition states two prerequisites a task-aware up-sampling has to meet: (1) the up-sampling performed in the functional space, which is to relate N. parameters via gi (2) the mapping between the input and the output of an up-sampled task satisfics fe..
Property 1 (Task-awareness Maximization). Consider N₂ 2, g0,, fo, (.) WI, and ()) = W2. The proposed ATU algorithm that pursues an up-sampled task Tup (X,, Y,} via minimizing the EMD loss between Ti and Tr maximizes the task-awareness, ie., minimizing the distance between Y, and (X....
02) (1 X)01 + A02,
Proof. According to the definition of EMD (Eq. (2) solves: /3" = mingee E, x1,1xxixx22,000 22)ll, where + = (f1......mm) n2
- (1, .mz) denotes the set containing all possible bijective assignments, each of which gives one-to-one correspondence between Ti and Tz. Based on the optimal assignments 6² the EMD is known to be defined as dEMD mannpn,,na) E, ||xi,; x2,00 1)ll- In light of
the difficulty in mathematically formulating a possible up-sampled task T. that lies in the local manifold of T2,, we reasonably assume simplified way of characterizing an up-sampled task T. to be Yu.- af,, Y1+ +1, Y2. Xu., = = X1 +ar, X2, Vj. where each sample is convex

---

combination of samples from both T1 and Tz. The combination coefficients an 0x2, 2 EIR ( + Ex Σ** 11 En 42.34 -, an 42,5 20, vk. Different combination coefficients lead to a set of up-sampled task candidates [Tw}. We evaluate the task-awarenes, property of each candidate Tu, i.c., the distance between Y. and fe. (X.). ) to be ||,,- fa, (X.,)l2 Σ, 118.. So, (Ku,))ll Σ, I(WW-WW Wa)i-Wa]AaT,,>>>
(1 , Xallla LHS. (See Appendix E.)
Note that LHS < E, I| - Wa Walla(A| xu., - X2,020:11a + 2|x2l12) and LHS S Σ, I| buرس2 + [xa112). (See Appendix E.) By combining the two in- X2,0, + ||xella: (1 A) ()) Xn,ll2 + ||X||l2]. In practice, it is easy to normalize all the tasks in the feature space, which leads to ||x:||2 = ||2a loss dEMD= min(ming, Σ, [|&a... x2, ll.. ming; Σ, [[xm,)
Walla((1 (1)))
equalities above, we have LHSS E, IW1 - f
Therefore, by minimizing the EMD
$1)01 (1)l()))) the proposed task up- sampling network identifies from the candidate set } the task T₁ that has the minimal distance between Y. and fo. (X.)): ); in other words, the task-awareness is maximized.
Previous task augmentation approaches directly mix up two tasks without minimizing the EMD loss, i.., Yu.j = (1 A)yi, + Xya.j, Xu.j = (1 A)xi.j Ax2.j. j. In this case, the task-awareness is unwarranted as we have illustrated in Section provided that ||Y fa-Jo, So, (Xx)lla = E, [I(1 (11111)37 1)y
Xy (1- ( 1)x1, dx2.1ll2 E, (1-
[W2][[]
(2)(x11-X2..)
6 Experiments
To evaluate the effectiveness of ATU, we conduct extensive experiments to answer the following questions: Q1: How does ATU perform compared to state-of-the-art dask-aagmeetation-bbie and regularization meta-learning methods? Q2: Whether can the proposed ATU consistently im- prove performance for different meta-learning methods? Q3: What does up-sampled task by ATU looks like? Q4: What is the influence of increasing the task number within meta- veta-training data on the performance improvement of ATU? Benchmarks. We compared ATU with state-of-the-art art task augmentation strategies for meta-learning, including MetaAug (23). MetaMix [40, Meta- Maxup 20] MLTI 43), and regularization methods, including MetaDropout (13), TAML [11], and
Meta-Reg 44 both regression and classification problems. We also consider a variant of ATU
which removes the adversarial loss Lade and trains the task augmentation network only through the
EMD loss. We denote this variant by TU. To validate the consistent effect of ATU in improving different meta-learners, we apply ATU and AU on MAML [13], MetaSGD [53 and ANIL [22].
We also consider cross-domain settings where the meta-testing tasks are from different domains.
Table 2: MSE with +955 confidence inter- vals on sinusoidal regression.
Table 3: MSE with 95% confidence inter- vals cross-domain sinusoidal regression.
Model
10-shot
20-shot
30-shot
Cross-domain
Frequency [0.4.0.8]
Aplitude [5.0,6.0]
Phase [-x,0]
DropGrad
32
0.17
0.12
0.55 2.13
MetaAug
0.93 - 0.18
0.14
0.58 1 0.12
Meta-Leamer: MAMI.
Meta-Leuree:: MAML
MAML 9
1.78 1: 0.35
3.52 + 0.35
3.12 : 0.52
MAMI
9
0.93 + 0.18
0.65 0.13
0.58
+ 0.12
MelaMix 40
1.67 1: 0.30
1.288
3.14 1 0.54
MetaMix 40
0.81 + 0.17
0.58=0.12
0.56
0.11
MLTI 43
1.92 0.42
3.56
3,66
+ 0.63
MLTI
43
0.92 1
0.13
0.62
0.12
TU
1.70 0.34
3.23
2.88
- 0.48
TU
0.84 + 0.16
0.12
0.47
+ 0.10
ATU
1.58 0.35
2.92 0.29
2.58
0.48
ATU
0.70 = 0.14
0.47
0.13
0,42
0.11
Meta-LLarrer: MelaSDD
Meta-l Learer:: MetaSGI)
MetaSGD [5
0.70 = 0.17
0.49
1:0:000 0.11
0.42 1: 0.09
MelaSGD 15
2.24 -:: 0.46
2.42 +: 0.32
2.73 +: 0.56
MetaMix 40
0.60 0.15
0.37
0.09
0.37
0.08
MelaMix 40
1.77 0.35
2,50
2.46
1 0.48
MLTI
43
0.66 :: 0.16
0.51
0.11
0.44
: 0.10
MLTI 43
1.40
2.56 0.28
2.54 + 0.54
TU
0.54 + 0.11
0.36
0.08
0.31
0.08
TU
1.64
2.37
2.04
0.46
ATU
0.49 0.10
0.34
0.08
0.29 0.08
ATU
1.71
2.19 0.23
2.53 + 0.62
6.1 Regression
we construct the K-shot regression task by sampling from the target sine curve y(z) = Asin(wr + b). where the amplitude A E 5.01, the frequency w E [0.8, 1.2], the phase be [0, x] and x is sampled from [-5.0, 5.0). . In the meta-training phase, each task contains K support and K target (K=10) examples. We adopt mean squared error (MSE) as the loss function. For the base model fo, we adopt a small neural network, which consists of an input layer of size 1, 2 hidden layers of size 40 with ReLU and output layer of size 1. We use one
Experimental Setup. Following 15
7

---

Support
Query
Support Quay
Support Query
Query
Support
0-
2
-2
(a) MLTI
(b) MetaMix
(c) TU
(d) ATU
Figure 3: The augmented regression tasks generated by different augmentation- based methods.
gradient update with a fixed step size a= 0.01 in inner loop, and use Adam as the outer-loop optimizer following 15] Moreover, the meta-learner trained 240,000 tasks with meta batch-size being 4. In meta-testing stage, we randomly sample 100 sine curves as meta-test tasks, each task containing K support samples and 100 query examples. The data points in query are evenly distributed on [-5.0, 5.0]. The averaged MSE with 95% confidence intervals upon these 100 sine curves with K=10, We also perform cross coos-domain experiments by sampling 100 sine curves which have different frequencies, amplitudes phases from the tasks in meta- meta-training set and More settings about the up-sampling networks are listed Appendix
20, 30 are reported in Table 2
report the results in Table 3
The results in Table2aand Table 3 show that ATU consistently outperforms the baseline methods MAMI., MetaMix, and MI.TI in different K-shot (KE {10, 20, 303) settings and various domain settings. These results validate that the tasks generated by ATU can better approximate the true task distribution and provide more information to the meta-learner than MetaMix and MLTI, thus enabling better generalization of the model. We further verify the superiority of the proposed methods by visualizing the augmented tasks generated by the proposed methods and the baseline methods. The visualization results in Fig. B show that the points in the tasks generated by TU and ATU fit the sine curve well, while the points in the tasks generated by MLTI and MetaMix deviate from the sine curve. This indicates that augmented tasks generated by TU and ATU match the true task distribution. is noteworthy that the support set and query set generated by ATU differ significantly from those generated by TU, which indicates that the task generated by ATU is more difficult. This, together with the results that ATU outperforms TU in most experiments, demonstrates the effectiveness the adversarial losses in generating informative tasks to improve generalization of the meta-learner.
Performance.
In Fig. we also visualize the adaptation of meta-learner trained by different task augmenta- tion methods for a 10-shot meta-test regression task. Compared to the MAML trained on orig- inal meta-training tasks, the MAML trained on tasks generated by TU fits the ground-truth sinu- soid after only one update. And ATU performs better than TU. This again validates that the augmented tasks generated by TU and ATU are more informative for the meta-learner to learn the meta knowledge from the true task distribution.
MAML
TU
MAML INI
TUINTT
Figure 4: Initialization (dotted) and one- step adaptation (solid) regression curves of MAMI, and ATU when K=10.
6.2 Classification
Experimental setup. We follow MLTI to evaluate the performance of task augmentation algorithms for shot classification problem with limited number of base classes in eeeta-trannigg set under non-label-sharing settings. We consider four datasets (base classes number): minilmagenet-S (12),
ISIC 18 (4), Dermnet-S (30), and Tabular Murris [5] (57) covering classification tasks on general natural images, medical images, and gene data. Note that the minilmagenet-S S and Dermnet-S are constructed by limiting the base classes of minilmangenet [33] and Dermnet [I], respectively. We construct N N-way K-shot tasks, setting N = 5 for minilmagenet-S Derment-S, Tabular Murris and N for ISIC dataset due to its limited number base classes and setting K = or K = 5. Recall that TAU relies on extra KM images to generate augmented tasks in image classification problem. To make the training process more efficient, we set KM to 3 and the upsampling rate 2. More details of these datasets and the settings of the task augmentation networks are listed in Appendix D.

---

Table 4: Average accuracy under different settings of few-shot classification and various datasets.
minilmagener-S 1-shot 5-shot
ISIC
DermNet-S
Tabular Murris
Model
1-shot 5-shot
1-shot
5-shot
1-shot
5-shot
MAMI 9
38.27%
52.14%
57.59%
65.24%
43.47%
60.56%
79.08%
88.55%
Meta-Reg [44
38.35%
51.74%
58.57%
68.45%
45.01%
60.92%
79.18%
89.08%
TAML [11
38.70%
52.75%
58.39%
66.09%
45.73%
61.14%
79.82%
89.11%
Meta-Dropout 13
38.32%
52.53%
58.40%
67.32%
44.30%
60.86%
78.18%
89.25%
MetaMix [40]
39.43%
54.14%
60.34%
69.47%
46.81%
63.52%
81.06%
89.75%
Meta-Maxup 20
39.28%
53.02%
58.68%
69.16%
46.10%
62.64%
79.56%
88.88%
MLTI
41.58%
55.22%
61.79%
70.69%
48 48.03%
64.55%
81.73%
91.08%
TU
42.16%
56.33%
02.03%
73.97%
48.07%
64.81%
81.8%%
91.15%
ATU
42.60%
56.78%
62.84%
74.50%
48.33%
65.16%
82.04%
91.42%
Performance. We show the performance on the four datasets in Table 4 On all four datasets, the proposed ATU consistently outperforms the baseline methods, including the augmentation- based methods (i.e. MetaMix, Meta-Maxup and MI.TI) and regularization-basel based methods (Meta-Reg, TAMI. and Meta-Dropout). And TU achieves the second best performance on all experiments. We also observe that our method achieves a large improvement sists of only 4 base classes, indicating the effectiveness of our method in limited tasks scenar- ios. We further evaluate the effectiveness of the proposed ATU on improving the generalization for different backbone meta-learner by conducting experiments under 1-shot setting to compare the performance of MLTI and ATU in improving the performance of the backbone meta-learner MetaSGD and ANIL. The results are presented in Table 5] ATU again consistently outperforms MLTI. All these results validate the superiority of the proposed ATU and TU over the existing baselines in generating informative tasks to improve the performance of different backbone meta- learners. We also evaluate the performance of ATU in cross-domain adaptaion settings. In Table G we present the results of the experiment that apply the meta-model trained on minilmageNet-S to Dermnet-S, and vice versa. ATU improves the generalization performance of MAML (the backbone meta-learner in this experiment) by a large margin. This indicates ATU can consistently im- prove the backbone meta-model's generalization ability under the challenging cross-domain settings.
the ISIC dataset which con-
Table 6: Cross-domain adaptation experiments be- tween mini-S and Dermnet-S. A-B denotes that the backbone meta-model is meta-trained on A and
Table 5: Comparison of compatibility with different backbone meta- leaanneg algorithms on 1-shot classification.
meta-tested on
Method
mini-S
ISIC
Derm-S
TM
mini-S - Derm-S
Derm-S + mini-S I-shot
Model
MetaSGD [5]
37. 98%
79.7
12.07%
51.55%
I-shot
5-shot
5-shot
MeI
58%
61.57%
45.49%
83.31%
MAML 34.46%
30.36%
28. 78.78%
41.29%
MetaSGD+ATU
40.52%
62.84%
46.78%
83.84%
MAML+ATU
36.86%
51.98%
30.68%
46.72%
ANIL (2
38.02%
59.48%
44.58%
75.67%
ANIL+METI
39.15%
61.78
46.79%
77.11%
MetaSGD L5]
31.07%
49.07%
28.17%
41 41.83%
ANIL+ATU
39.27%
62.12%
47.03%
77.23%
MetaSGD+ATU
37.75%
54.60%
30.78% %
44.01 %
0.51
0.67
Crigiial Tink
0.19
0.65
Up aoopandd tink
Accmnyy (ebo)
0.47
0.63
0.45
0.61
0.43
0.59
0.41
MAMic]-bbr
0.57
0.39
ATI3) dar)
0.35
MAM
0.37
Arus5-bor
0.53
0.35
0.51
12
25
38
Metan-naining Classcs
Figure 5: The averaged accuracy on the minilma- Figure 6: T-SNE visualization of original and up- genet dataset with different number of tasks.
sampled tasks on 1-shot minilmagenet-S setting.

---

Effect of the number of meta-training tasks. We conduct experiments to analyze the change in the performance improvement of ATU over MAML with the number of meta-training tasks in I I-shot and 5-shot settings. The results presented in Fig. 5 show that ATU significantly improves the performance of MAML. by about 4.5%-5% when number of base classes is 12, while the improvement decreases with the number of base classes increasing on both 1-shot and 5-shot settings. When the number of base classes increases, the number of training tasks increases rapidly and the empirical task distribution constructed from the meta-training tasks becomes closer and closer to the true latent task distribution. Therefore, the extra information provided by tasks generated by ATU becomes less. However, even if all available base classes used in the meta- veeta-triiiing (i.e., 64 meta-training classes), our proposed ATU could still help to improve the performance of MAML.
Visualization of the generated tasks. We visualize the up-sampled tasks by t-SNE to evaluate their generation quality for MAMI. under the 1-shot minilmagenet-S setting. Concretely, we up-sample 100 tasks for 5 original tasks via ATU by using different perturbations for each task. In order to visualize the relationship between generated tasks and original tasks using t-SNE, we represent each task by concatenating the vector of the support and query sets. The results presented in Fig. show that the up-sampled tasks stay near the original tasks, which means they are matching the true task distribution. This indicates the generated tasks are task-aware, Besides, we can observe that the augmented tasks are diverse and cover a substantial portion of the original tasks. This demonstrates the task imaginary property of the augmented tasks. These two observations suggest that the proposed ATU a qualified task augmentation algorithm.
Effect of the extra dEMD D(MD((; (D;, D2) in regression tasks. As presented in Section 4.1 we propose apply extra EMD loss dEMD D D the support and query set for each generated task to encourage the points in the generated support set and the query set to follow the same sine curve. In Fig. 3 we have visualized the tasks generated by the Task Up-sampling Network trained with the extra EMD loss. Here we provide additional visualization result for tasks generated by the Task Up-sampling Network trained without the extra EMD loss in Fig. 7 It can be seen that the support set and the query set not the same sinusoid, indicating that the generated tasks need additional supervision to avoid them being too difficult not even valid tasks.
Support Query
-2
Figure 7: Visualization of the up- sampled task generated by ATU when na = 0 in Eq. (6).
Conclusion and Limitation
In this paper, we propose the first task-level up- up-sampling network that learns to generate tasks that simultaneously meet the qualifications of being task-aware, task-imaginary, and model-adaptive. The proposed Adversarial Task Up-sampling (ATU) takes a set of tasks as input and learn to up-sample tasks complying with the true task distribution while being informative to improve the generalization of the meta-learner. We theoretically justify that ATU promotes task- aask-wwsreness and empirically verify that ATU improves the generalization of various backbone meta-learner for both regression and classification tasks on five datasets. Limitations. Our theoretical results are obtained under some strong assumptions, but all the experiments and visualization outcomes validate method's effectiveness in real settings.
8 Acknowledgement
This work is sponsored by the Tencent AI Lab Gift Fund (Project 9229073) and CityU Strategic Interdisciplinary Research Grant (Project 7020064).
10

---

References
[1] Dermnet dataset, 2016. http:://waw..eemmet..oom
[2] Fungi dataset, 2018.
[3] Marcin Andrychowicz, Misha Denil, Sergio Gomez, Matthew Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In NeurIPS, pages 3981-3989, 2016.
[4] Jonathan Baxter. A bayesian/information theoretic model of learning to learn via multiple task sampling. Machine learning, 28(1):7-39, 1997.
[5] Kaidi Cao, Maria Brbic, and Jure Leskovec. Concept learners for few-shot learning. In ICLR, 2020.
[6] Can Chen, Xi Chen, Chen Ma, Zixuan Liu, and Xue Liu, Gradient-based based bi-level optimization for deep learning: A survey, arXiv preprint arXiv:2207.11719 2022.
[7] Wei-Yu Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. In ICLR, 2018.
[8] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, pages 3606-3613, 2014.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-leaming for fast adapta- tion of deep networks. In ICML, pages 1126-1135, 2017.
[10] David Ha, Andrew M Dai, and Quoc V Le. Hypernetworks. In ICLR, 2017.
[11] Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-learning for few-shot learning. In CVPR, pages 11 11719-11777 1727, 2019.
[12] Hae Beom Lee, Hayeon Lee, Donghyun Na, Saehoon Kim, Minseop Park, Eunho Yang, and Sung Ju Hwang. Learning balance: Bayesian meta-learning for imbalanced and out-of- distribution tasks. In ICLR, 2020.
[13] Hae Beom Lee, Taewook Nam, Eunho Yang, and Sung Ju Hwang. Meta dropout: Learning to perturb latent features for generalization. ICLR, 2019.
[14] Xiaomeng Li, Lequan Yu, Yueming Jin, Chi-Wing Fu, Lei Xing, and Pheng-Ann Heng. Difficulty-wware meta-learning for rare disease diagnosis. In MICCAI, pages 357-366, 2020.
[15] Zhenguo Li, Fengwei Zhou, Fei Chen, and Hang Li. Meta-sgd: Learning to learn quickly for few-shot learning. arXiv preprint arXiv:1707.0988: 09835, 2017.
[16] Jialin Liu, Fei Chao, and Chih-Min Lin. Task augmentation by rotating for meta-learning. arXiv preprint arXiv:2003.00880 00804, 2020.
[17) Subhransu Maji, Essa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- grained visual classification aircraft. arXiv preprint arXiv:1306.55 2013.
[18] Md Ashraful Alam Milton. Automated skin lesion classification using ensemble of deep neural networks in isic 2018: Skin lesion analysis towards melanoma detection challenge. arXiv preprint arXiv: 1901. 10802, 2019.
[19] Shikhar Murty, Tatsunori Hashimoto, and Christopher D Manning. Dreca: A general task augmentation strategy for few-shot natural language inference. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1113-122,, 2021.
[20] Renkun Ni, Micah Goldblum, Amr Sharaf, Kezhi Kong, and Tom Goldstein. Data augmentation for meta-learning. In /CML, pages 8152-8161, 2021.
[21] Jaehoon Oh, Hyungjun Yoo, ChangHwan Kim, and Se- Youn Yun. Boil: Towards representation change for few-shot learning. In ICLR, 2020.

---

[22] Aniruddh Raghu, Maithra Raghu, Samy Bengio, and Oriol Vinyals. Rapid learning or feature reuse? towards understanding the effectiveness of maml. In ICLR, 2019.
[23]
Janarthanan Rajendran, Alex Irpan, and Eric Jang. Meta-learning requires meta-augmeenation. arXiv preprint arXiv:2007.05549, 2020.
[24] Sachin Ravi and Hugo Larochelle. Optimization as a model for few-shot learning. 2016.
[25] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In ICLR, 2018.
[26] Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- weight-net: Learning an explicit mapping for sample weighting. NeurIPS, 32, 2019.
[27] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In NeurlPS, pages 4080-4090, 2017.
[28] Qianru Sun, Yaoyao Liu, Tat-Seng Chua, and Bernt Schiele. Meta-transfer learning for few-shot learning. In CVPR, pages 403 403-412, 2019.
[29] Flood Sung, Yongxin Yang. Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In CVPR, pages 1199-1208, 2018.
[30] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In ECCV, pages 266-282. Springer, 2020.
[31] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evci, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, et al. Meta-dataset: A dataset of datasets for learning to learn from few examples. In ICLR, 2019.
[32] Hung- Yu Tseng, Yi-Wen Chen, Yi-Hsuan Tsai, Sifei Liu, Yen-Y Yu Lin, and Ming-Hsuan Hsuan Yang. Regularizing meta-learning via gradient dropout. In ACCV, 2020.
[33] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. NeurIPS, 29:3630-3638, 2016.
[34] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd 200-2-200-2011 dataset. 2011.
[35] Renzhen Wang, Kaiqin Hu, Yanwen Zhu, Jun Shu, Qian Zhao, and Deyu Meng. Meta feature modulator for long-tailed recognition. arXiv preprint arXiv:2008.03428 2020.
[36] Renzhen Wang, Xixi Jia, Quanziang Wang. and Deyu Meng. Learning to adapt classifier for imbalanced semi-supervised learning. arXiv preprint arXiv:2207.13856 2022.
[37] Olga Wichrowska, Niru Maheswaranathan, Matthew W Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Nando Freitas, and Jascha Sohl- bick-iicsttnn. Learned optimizers that scale and generalize. In ICML, pages 3751-3760, 2017.
[38] Yichen Wu, Jun Shu, Qi Xie, Qian Zhao, and Deyu Meng. Learning purify noisy labels via meta soft label corrector. In AAAI, pages 10388-10396, 2021.
[39] Jin Xu, Jean-Francois Ton, Hyunjik Kim, Adam Kosiorek, and Yee Whye Tch. Metafun: Meta-learnig with iterative functional updates. In ICML., pages 10617-10627, 2020.
[40] Huaxiu Yao, Long- Kai Huang, Linjun Zhang, Ying Wei, Li Tian, James Zou, Junzhou Huang, et al. Improving generalization meta-lcarning via task augmentation. ICML, pages 11887-11897, 2021.
[41] Huaxiu Yao, Yu Wang, Ying Wei, Peilin Zhao, Mehrdad Mahdavi, Defu Lian, and Chelsea Finn. Meta-learning with adaptive task scheduler. NeurIPS, 34, 2021.
12

---

[42] Huaxiu Yao, Ying Wei. Junzhou Huang, and Zhenhui Li. Hierarchically structured meta-learning In ICML, pages 7045-7054, 2019.
[43] Huaxiu Yao, Linjun Zhang, and Chelsea Finn. Meta-learning with fewer tasks through task interpolation. 2022.
[44] Mingzhang Yin, George Tucker, Mingyuan Zhou, Sergey Levine, and Chelsea Finn. Meta- learning without memorization. In ICLR, 2020.
[45] Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng. Pu-net: Point cloud upsampling network. In CVPR, pages 2790-2799, 2018.
[46] Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. In 2018 International Conference on 3D Vision (3DV), pages 728-737, 2018.
[47] Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song. Metagan: An adversarial approach to few-shot learning. Neur/PS, 2:8, 2018.
[48] Luisa Zintgraf, Kyriacos Shiarli, Vitaly Kurin, Katja Hofmann, and Shimon Whiteson. Fast context adaptation via meta-learning. In JCML, pages 7693-7702, 2019.
13

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes].
(b) Did you deseribe the limitations of your work? [Yes]. See Section g
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes].
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]. See Section
(b) Did you include complete proof's of all theoretical results? [Yes]. See Section 5
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material as URL))? [Yes]. We exploit the
open-source datasets and will release the code.
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)?[Yes) See Section 6.2 and Appendix C/D.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)?[Yes]. See Section 6.2
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]. See Appendix.
4. If you are using existing assets (c.g., code, data, models) or curating/relcusing new assets...
(a) your work uses existing assets, did you eite the creators? Yes]. See Section 6.2
(b) Did you mention the license of the assets?[N/A].
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A].
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A].
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A].
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A].
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A].
14


Reviews:
[{"review_id": "mqCPnq11ZgX", "reviewer": "Reviewer_pk34", "summary": "The paper devises a method for task augmentation for meta-learning with three properties: (a) task aware; (b) task imaginary and c) adaptive.  The primary contribution of the paper is a new task generation method which takes into account various properties for the generation process.", "questions": "Provided above.", "limitations": "Task augmentation is an important way to mitigate meta-overfitting and the paper provides a methodological contribution for it.  While the method is not simple, it's well-motivated with certain properties. I feel that the main limitation of the paper is less analysis on the cross-domain tasks which is more realistic for few-shot learning (rather than miniImagenet -> miniImagenet). I am on the borderline with the paper, but would be happy to revisit my score if two of the limitations could be addressed: (a) major - would task augmentation work for cross-domain tasks too and some insights around it? ; (b) computational cost for the method - are the tasks generated on the fly during meta-training or are they generated and stored before meta-training?", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The problem is realistic as task augmentation is a good way to mitigate meta-overfitting and cover the range of the task distribution\n\n- The method though not very intuitive is well-motivated and presented well.  I have more comments about the method (see below)\n\n- Preliminary results are covering multiple aspects such as classification, regression and cross-domain tasks \n\n\nWeakness:\n\n- The empirical results are not super strong and the improvements are marginal in most of the cases. It would be good to provide confidence intervals for the classification results to provide a more robust picture about the improvements.  The gap between MLTI and the proposed method is not too much (except for ISIC). \n\n- While task augmentation intuitively makes sense for similar domain datasets (e.g. miniImagenet), I would like to see some discussion around the impact of task augmentation for cross-domain tasks. While the paper shows some preliminary results on cross-domain tasks, I would like to see some more discussion and insights around it as I believe it's a more realistic scenario for few-shot learning\n\n- The paper does not provide information on the computational cost for the method, which I would like to see. Can you please provide some more details on the generation cost and how you incorporate task augmentation during meta-training?\n\n- The theoretical results seem a little forced. I feel the methodological contribution is quite good and more empirical insights to support the results would be better than the theoretical analysis in this regard. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YL8T6-0D2Rc", "reviewer": "Reviewer_GY2p", "summary": "The submission proposes a new task augmentation method.  \nProposed approach trains a task proposal network that the output task distribution is matched with true task distribution and challenging for current meta-learner.\nExperimental results show improved generalization performance on both regression and classification tasks, compared to SOTA task augmentation methods.\n", "questions": "Clarification of weakness 2 (theoretical analysis) and weakness 3 (EMD details).\n", "limitations": "Author addressed the limitation of strong assumptions on their theoretical analysis.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Comments\nThe idea of this paper and the experimental support is convincing.\nCurrently I vote for rejection as I have few unresolved questions, but I'm willing to raise my rating if rebuttal can resolve my questions.\n\n### Strengths\n1) The proposed task augmentation method consistently outperforms existing task augmentation methods in classification and regression domain.\n2) By introducing task proposal network and appropriate training objective, output task distribution has nice properties (task-aware, task-imaginary, and model-adaptive) which are intuitively helpful for improving task generalization.\n\n### Weakness\n1)  The effectiveness of propose method on regression task is only evaluated on sinusoidal regression task, which is still a challenging \nbenchmark of meta-regression but low-dimensional and synthetic.\nAs the baseline methods (MLTI[1], MetaMix[2]) have shown, evaluation on higher-dimensional problem will greatly support the strength of \nthe proposed method over existing methods on regression domain.\n\n2) Theoretical analysis is not understandable.\nIn my understanding, authors try to prove that minimizing EMD loss is equivalent to minimizing task awareness\n$||  \\mathbf{Y}_u -  f\\_{\\theta_u}(\\mathbf{X}_u)  ||_2$ under specific assumptions.\nHowever, the proof picks specific solution (Line 239-241) first then upper bound the solution by EMD loss.\nCurrently I can't understand how the inequality guarantees minimization of task awareness, where the LHS seems to be a known constant according to the assumption.\n\n3) The detail of how to compute Earth Mover Distance (EMD) between two tasks is not clearly described.\nThis missing detail makes difficult to fully picture how the task up-sampling network is optimized.\nSpecifically, the detail of how the labels ($\\mathbf{Y}$) are treated was a major blank for me as a reader.\n\n## Reference\n[1] Yao et al. \"Meta-learning with fewer tasks through task interpolation.\" ICLR2022.\n[2] Yao et al. \"Improving generalization in meta-learning via task augmentation.\" ICML2021.\n\n\n## Post Rebuttal\nThank the authors for their detailed answers.\nI now vote for accepting this paper as:\n1) all of my questions are resolved\n2) provided results on Pose regression task supports the consistency of performance improvement in regression task stronger.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OOiUK2EEoj", "reviewer": "Reviewer_iHfP", "summary": "In this paper, an adversarial task up-sampling method is developed to augment the meta-training tasks. The task up-sampling network is trained by minimizing the EMD loss between the generated tasks and ground truth tasks. An adversarial loss based on the gradient similarity between the support and query sets is incorporated into the proposed framework to generate difficult meta-training tasks. Experiment results indicate that the proposed method leads to improved results in few-shot regression and classification tasks.", "questions": "Why the experiments are not conducted on more “traditional” (and larger) FSL datasets? For example, it would be nice to see results in traditional miniImagenet, tieredImagerNet, or metaDataset. Most SOTA few-shot learning methods report results on those datasets. Once I see good performance on those traditional FSL datasets, I am willing to raise my score.", "limitations": "The authors discussed the limitations in the paper.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe proposed method makes the distribution of generated tasks close to the ground truth task distribution by minimizing the EMD loss.\n\nUtilize adversarial training to generate more difficult tasks.\n\nThe proposed method leads to obvious improvements in sinusoidal regression tasks.\n\nWeaknesses:\n\nIn few-shot image classification, the ground truth tasks are constructed by mixup of original images. It is known that mixup itself is a data augmentation method and leads to improved performance in various learning tasks. It is not clear why this mixup step is necessary. This mixup step makes it difficult to analyze the effectiveness of the up-sampling network in few-shot classification. It is not clear whether the improved performance of the proposed method is due to mixup or up-sampling network? An ablation study based on non-mixup samples is necessary to show the effectiveness of the proposed method.\n\n\nIn table 4, the few-shot classification accuracy is better than MLTI, but the gap between them is not large. Since the standard deviation of the classification accuracy is not given, it is not clear whether the improvement is statistically significant.\n\nBased on the results of TU and ATU in table 4, the adversarial loss only leads to marginal improvement on few-shot classification. Note that “adversarial” is a keyword in the title. Current experiment results cannot support that adversarial up-sampling is truly effective in improving few-shot classification performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LiKRakV_dl", "reviewer": "Reviewer_Jet1", "summary": "The paper proposes to augment tasks in a meta-learning setup with tasks through an adversarial task up-sampling network. The task-upsampling network first generates coarse level task samples that are refined further to get multiple task samples within the coarse sample’s vicinity. The task up-sampling network is trained by including an adversarial loss to generate tasks that maximally contribute to the meta-learner. The authors validate the resulting adversarial task up-sampling network on a synthetic sine regression task and multiple image classification datasets. \n", "questions": "A classification task typically consists of samples and labels. It is unclear how the labels are generated in the up-sampling process. \n\nWhat is the role of g_s() in the upsampling network of the classification task? The description in lines 207-217 suggests that the residuals on the images directly are passed through the attention network. \n\nTable 6: why did you choose the cross-domain setting where the target task has more samples to learn from? How did the models behave when the two domains were comparable (1-shot setting)?\n", "limitations": "adequately addressed", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nTask augmentation is a popular approach to prevent over-fitting and memorization in meta-learning. The authors propose a task augmentation technique, where the tasks are generated adaptively. The method is interesting, novel, but quite complicated. \n\nThe authors have experimented with many benchmarking datasets and techniques to show the effectiveness of the approach. While there is a marginal improvement over existing methods, the improvement is consistently seen across the datasets. The ablations and experiments to analyze the model’s behavior are satisfactory.\n\nThe paper is well written and clear to understand.\n\nWeakness\n\nThe problem statement is not novel, and the solution is an incremental contribution to the existing literature. As rightly discussed by the authors, there are many works on task augmentation for meta-learning. The manuscript reads like yet another method on this topic.\n\nI find the theoretical analysis to be a bit disconnected from the algorithm with regards to the two prerequisites. The authors have also acknowledged it in the conclusion section. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept