Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The submission proposes a new task augmentation method.  
Proposed approach trains a task proposal network that the output task distribution is matched with true task distribution and challenging for current meta-learner.
Experimental results show improved generalization performance on both regression and classification tasks, compared to SOTA task augmentation methods.

strengths_and_weaknesses: ### Comments
The idea of this paper and the experimental support is convincing.
Currently I vote for rejection as I have few unresolved questions, but I'm willing to raise my rating if rebuttal can resolve my questions.

### Strengths
1) The proposed task augmentation method consistently outperforms existing task augmentation methods in classification and regression domain.
2) By introducing task proposal network and appropriate training objective, output task distribution has nice properties (task-aware, task-imaginary, and model-adaptive) which are intuitively helpful for improving task generalization.

### Weakness
1)  The effectiveness of propose method on regression task is only evaluated on sinusoidal regression task, which is still a challenging 
benchmark of meta-regression but low-dimensional and synthetic.
As the baseline methods (MLTI[1], MetaMix[2]) have shown, evaluation on higher-dimensional problem will greatly support the strength of 
the proposed method over existing methods on regression domain.

2) Theoretical analysis is not understandable.
In my understanding, authors try to prove that minimizing EMD loss is equivalent to minimizing task awareness
$||  \mathbf{Y}_u -  f\_{\theta_u}(\mathbf{X}_u)  ||_2$ under specific assumptions.
However, the proof picks specific solution (Line 239-241) first then upper bound the solution by EMD loss.
Currently I can't understand how the inequality guarantees minimization of task awareness, where the LHS seems to be a known constant according to the assumption.

3) The detail of how to compute Earth Mover Distance (EMD) between two tasks is not clearly described.
This missing detail makes difficult to fully picture how the task up-sampling network is optimized.
Specifically, the detail of how the labels ($\mathbf{Y}$) are treated was a major blank for me as a reader.

## Reference
[1] Yao et al. "Meta-learning with fewer tasks through task interpolation." ICLR2022.
[2] Yao et al. "Improving generalization in meta-learning via task augmentation." ICML2021.


## Post Rebuttal
Thank the authors for their detailed answers.
I now vote for accepting this paper as:
1) all of my questions are resolved
2) provided results on Pose regression task supports the consistency of performance improvement in regression task stronger.
questions: Clarification of weakness 2 (theoretical analysis) and weakness 3 (EMD details).

limitations: Author addressed the limitation of strong assumptions on their theoretical analysis.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes to augment tasks in a meta-learning setup with tasks through an adversarial task up-sampling network. The task-upsampling network first generates coarse level task samples that are refined further to get multiple task samples within the coarse sample’s vicinity. The task up-sampling network is trained by including an adversarial loss to generate tasks that maximally contribute to the meta-learner. The authors validate the resulting adversarial task up-sampling network on a synthetic sine regression task and multiple image classification datasets. 

strengths_and_weaknesses: Strengths

Task augmentation is a popular approach to prevent over-fitting and memorization in meta-learning. The authors propose a task augmentation technique, where the tasks are generated adaptively. The method is interesting, novel, but quite complicated. 

The authors have experimented with many benchmarking datasets and techniques to show the effectiveness of the approach. While there is a marginal improvement over existing methods, the improvement is consistently seen across the datasets. The ablations and experiments to analyze the model’s behavior are satisfactory.

The paper is well written and clear to understand.

Weakness

The problem statement is not novel, and the solution is an incremental contribution to the existing literature. As rightly discussed by the authors, there are many works on task augmentation for meta-learning. The manuscript reads like yet another method on this topic.

I find the theoretical analysis to be a bit disconnected from the algorithm with regards to the two prerequisites. The authors have also acknowledged it in the conclusion section. 


questions: A classification task typically consists of samples and labels. It is unclear how the labels are generated in the up-sampling process. 

What is the role of g_s() in the upsampling network of the classification task? The description in lines 207-217 suggests that the residuals on the images directly are passed through the attention network. 

Table 6: why did you choose the cross-domain setting where the target task has more samples to learn from? How did the models behave when the two domains were comparable (1-shot setting)?

limitations: adequately addressed
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, an adversarial task up-sampling method is developed to augment the meta-training tasks. The task up-sampling network is trained by minimizing the EMD loss between the generated tasks and ground truth tasks. An adversarial loss based on the gradient similarity between the support and query sets is incorporated into the proposed framework to generate difficult meta-training tasks. Experiment results indicate that the proposed method leads to improved results in few-shot regression and classification tasks.
strengths_and_weaknesses: Strengths:

The proposed method makes the distribution of generated tasks close to the ground truth task distribution by minimizing the EMD loss.

Utilize adversarial training to generate more difficult tasks.

The proposed method leads to obvious improvements in sinusoidal regression tasks.

Weaknesses:

In few-shot image classification, the ground truth tasks are constructed by mixup of original images. It is known that mixup itself is a data augmentation method and leads to improved performance in various learning tasks. It is not clear why this mixup step is necessary. This mixup step makes it difficult to analyze the effectiveness of the up-sampling network in few-shot classification. It is not clear whether the improved performance of the proposed method is due to mixup or up-sampling network? An ablation study based on non-mixup samples is necessary to show the effectiveness of the proposed method.


In table 4, the few-shot classification accuracy is better than MLTI, but the gap between them is not large. Since the standard deviation of the classification accuracy is not given, it is not clear whether the improvement is statistically significant.

Based on the results of TU and ATU in table 4, the adversarial loss only leads to marginal improvement on few-shot classification. Note that “adversarial” is a keyword in the title. Current experiment results cannot support that adversarial up-sampling is truly effective in improving few-shot classification performance.
questions: Why the experiments are not conducted on more “traditional” (and larger) FSL datasets? For example, it would be nice to see results in traditional miniImagenet, tieredImagerNet, or metaDataset. Most SOTA few-shot learning methods report results on those datasets. Once I see good performance on those traditional FSL datasets, I am willing to raise my score.
limitations: The authors discussed the limitations in the paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper devises a method for task augmentation for meta-learning with three properties: (a) task aware; (b) task imaginary and c) adaptive.  The primary contribution of the paper is a new task generation method which takes into account various properties for the generation process.
strengths_and_weaknesses: Strengths:

- The problem is realistic as task augmentation is a good way to mitigate meta-overfitting and cover the range of the task distribution

- The method though not very intuitive is well-motivated and presented well.  I have more comments about the method (see below)

- Preliminary results are covering multiple aspects such as classification, regression and cross-domain tasks 


Weakness:

- The empirical results are not super strong and the improvements are marginal in most of the cases. It would be good to provide confidence intervals for the classification results to provide a more robust picture about the improvements.  The gap between MLTI and the proposed method is not too much (except for ISIC). 

- While task augmentation intuitively makes sense for similar domain datasets (e.g. miniImagenet), I would like to see some discussion around the impact of task augmentation for cross-domain tasks. While the paper shows some preliminary results on cross-domain tasks, I would like to see some more discussion and insights around it as I believe it's a more realistic scenario for few-shot learning

- The paper does not provide information on the computational cost for the method, which I would like to see. Can you please provide some more details on the generation cost and how you incorporate task augmentation during meta-training?

- The theoretical results seem a little forced. I feel the methodological contribution is quite good and more empirical insights to support the results would be better than the theoretical analysis in this regard. 
questions: Provided above.
limitations: Task augmentation is an important way to mitigate meta-overfitting and the paper provides a methodological contribution for it.  While the method is not simple, it's well-motivated with certain properties. I feel that the main limitation of the paper is less analysis on the cross-domain tasks which is more realistic for few-shot learning (rather than miniImagenet -> miniImagenet). I am on the borderline with the paper, but would be happy to revisit my score if two of the limitations could be addressed: (a) major - would task augmentation work for cross-domain tasks too and some insights around it? ; (b) computational cost for the method - are the tasks generated on the fly during meta-training or are they generated and stored before meta-training?
review_title: null

Meta-review: metareview: This paper proposes a task augmentation method for meta-learning that generates new tasks which match the true task distribution and are also challenging for the current meta-learner. This is done by training a task upsampling network with an adversarial loss as well as an EMD loss between the adversarially generated and ground-truth tasks. The authors provide a theoretical analysis that the tasks generated by their adaptive task upsampling framework are indeed task-aware, or comply with the true task distribution, and validate the method on both regression and classification tasks. The results show that the proposed task upsampling method outperforms existing regularization methods or task augmentation methods. 

The paper initially received split reviews. Reviewers were generally positive about the introduction of the desirable properties for augmented tasks in meta-learning as a meaningful contribution. They also found the proposed method with adversarial task-aware upsampling as novel and interesting, and the experimental validation as adequately showing the effectiveness of the proposed method as well as each of its components. Another advantage that is not mentioned by the reviewers, is that it is applicable to both regression and classification tasks.

However, reviewers were also concerned with unclear, and somehow disconnected theoretical analysis from the actual framework, marginal improvements over state-of-the-art baselines such as MLTI, unclear effectiveness of the adversarial loss, and missing results on larger few-shot classification datasets. Most of these points were addressed in the author response, which resulted in some of the reviewers raising their scores, and all reviewers leaned toward acceptance after the interactive discussion period. 

In sum, this is a well-written paper that introduces meaningful insights about task augmentation in meta-learning, as well as a novel method, and may be of interest to researchers working on the topic. However, method-wise, its relatively weak improvement over existing, simpler task augmentation methods may diminish the potential practical impact of the work. Yet, the advantages outweigh its drawbacks and the work is worth sharing at NeurIPS 2022.
confidence: Certain
Decision: Accept