Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper explores few-task meta-learning that attempts to ease the difficulty and cost of massive task construction. It challenges the existing few-task meta-learning approaches that heavily rely on the Manifold-Mixup-based task augmentation, which is hard to generalize to cases beyond computer vision. In this case, the author introduces a set-value neural net (Set transformer) as the tool to augment the tasks. The author provides a theorectical ground that Set transformer is able to control the Rademacher complexity for better generalization. 5 dataset and 3 image-based FS benchmarks have been used to evaluate the method.
strengths_and_weaknesses: 1. The paper is well-organized and well-written.
2. The method seems simple but quite general and effective across many scenarios.
3. Theoretical results significantly motivate the method.
4. Massive empirical studies and the results are convincing.
questions: The paper may deserve a higher rating score if the questions below have been fully addressed.
1. Only Set-transformer has been considered as the implementation of meta-interpolation. Will other neural set functions be able to improve the few-task FS performance? It would be more convincing if the author can provide more empirical analysis on how to choose the set neural networks, e.g., change the architecture of the Set-transformer or use other types of set neural networks (for instance, Deep Set).
2. The method is claimed to be good at the field beyond vision. How is the performance in solving NLP low-resource problems? 
3. Please refer to the first statement in limitation.
limitations: 1. Distinct from the typical Mixup-based task augmentation baselines, the method is only implemented (only available?) in metric-based meta-learning, more precisely. ProtoNet. The author should provide more discussion of why meta-interpolation is limited in metric-based meta-learning, otherwise, provide the empirical performances of other meta-learning frameworks (e.g., MAML and more others) + meta-interpolation and further compare them with typical Mixup-based task augmentation baselines.
2. The method is a little bit incremental.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper introduces a new task augmentation method for meta-learning with few training tasks available. The proposed method employs a set-to-set function (instantiated as set transformer) that performs interpolation of features from the support set to create a new task. The parameters of a set transformer are learned via billevel optimization, using a meta-validataion dataset for tuning the parameters (considered as hyperparameters) to achieve the generalization. The paper performs experiments on the eight datasets to present the effectiveness of the proposed method, Meta-Interpolation.
strengths_and_weaknesses: ### Strengths
- The paper presents extensive experimental results across diverse datasets.
- The idea of employing meta-learnable set-to-set function for feature interpolation is novel.

###  Weakness
- The number of parameters of the set transformer may be large enough that the proposed method may be sensitive to the size of meta-validation dataset. In fact, I noticed that the ratio of meta-training:meta-validation is quite high (e.g., 12:16 in the case of Mini-Imagenet-S).
- The proposed method is applied to only prototypical networks.
questions: - What is the sensitivity of the proposed method to meta-validation dataset size, compared to other works, such as MLTI [48] that also performs feature interpolation?
- At which layer is the set transformer located? Is the location randomly sampled, similar to MLTI? If not, why so? Could you provide the ablation study on this?
- Why is the proposed method not applied to MAML? The similar work MLTI [48], which also performs feature interpolation, has shown the effectiveness on MAML as well. I believe providing experimental results on MAML will strengthen the paper.
limitations: While the paper provides limitations, I believe there are more limitations to discuss:
- the possible need for large meta-validation dataset (such high demand may break the assumption of few tasks). 
- Validated only for prototypical networks
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: To tackle meta-overfitting in few-task meta-learning, the authors propose Meta Interpolation based on set transformer to generate domain-agnostic tasks. They assign a new class k by re-arrang K classes in two tasks, and the new class k is the k_{th} class in the two new permutations. Then, the corresponding support set is constructed with interpolating function using bilevel optimization. Since the interpolating function is optimized to minimize the loss on the validation tasks, the proposed method can achieve task augmentation for each specific domain. The effectiveness of Meta Interpolation is proved both experimentally and theoretically.
strengths_and_weaknesses: Pros:
1. The task of task generation is interesting.
2. The effectiveness of the proposed method is proved both experimentally and theoretically.

Cons:
1. The reasonability of using the he interpolating function to generate new support set, while keeping the query set unchanged is unclear.
2.  The way to generate new classes is somewhat hasty.
questions: 1. Please analyse the principle of generating new classes by randomly aligning the category orders in two tasks. Since its prototype contains information of two classes, which may be quite different.
2. Why change the support set and keep the query set unchanged? If two new tasks contain the same query set but different support set, then, how to avoid the optimized support sets falling into the original one in task 1?
3.  Does this augmentation make task 1 and task 2 more similar?
4. More training details should be provided. For example, how many new tasks do you generate? Do the generated tasks make the comparison with other methods unfair (e.g., the model is updated more times during one training epoch, since more tasks are optimized in one epoch)? And which layer do you perform augmentation?
5. An ablation study about the number of training tasks and the performance of the proposed method, and the layer to implement augmentation may be better.
limitations: An ablation study about the number of training tasks and the performance of the proposed method may be better show the practical applicability.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a method for combining tasks in meta-learning when original base class domain data is scarce.  At meta-training time, a set transformer is used combine support set features from two random tasks, while comparing to queries from just the first.  The set transformer can therefore use the second random support set to apply random augmentations to the first, or not, as beneficial for the application.  Since the meta-training data is limited and easy to overfit, the random augmentations might never be used when trained jointly --- the transformer parameters are instead learned on a validation set with separate set of classes, using bilevel optimization with HyperGrad.  The method is evaluated on datasets in multiple domains, including chemical, text, speech and images, finding consistent improvements.

strengths_and_weaknesses: This is an interesting approach that furthers work in task augmentation, extending beyond interpolation to learned mixing.  It's a little bit complex, using a set transformer, separate class split and bilevel optimization, and it's not entirely clear all components are necessary in their current form --- however, there are ablations showing each contributes, and particularly that a separate optimization is required.  I have a few questions below on how these might be able to be simplified, but also think the system is reasonable as-is to demonstrate the effectiveness of the approach.

There is also a theoretical section, but was very dense and a little outside my expertise; this claims to show a reduction in Rademacher complexity in simplified settings, but I didn't follow it in detail.  The end result of this section also seems to rely on reducing the upper bound by decreasing the rank of a covariance-like matrix of task vector differences, though no discussion on how or why its rank might be reduced.

Overall, the method is explained well and empirical results demonstrate its effectiveness, with particularly strong improvement in ESC-50.  The degree to which all components are needed in their current form, or whether they can be simplified, is unclear.  I would have have liked to see additional ablations on the form of the set transformer, random mixing vector selection, and validation set sizes.  But the paper does present ablations showing all components are indeed needed together at a high-level to justify the approach.

questions: Additional questions:

Since original base data is scarce to begin with, splitting out additional classes into a separate validation set to train the set transformer seems like a lot of data to take away from training the feature model parameters.  Is there a way to continue using these for both features and set transformer, maybe changing the held-out classes used in hypergrad periodically each outer loop?

How important is the second support set for use in the transformer?  What if random vectors are input to the set transformer instead of task 2 vectors?

Since there is just one set of set transformer params $\lambda$, does this limit the amount of augmentation or interpolation?  If the same data elements and permutations are used twice with same lambda params, the set transformer will output the same features both times, unless there is another source of randomness in the transformer that I missed.  Does this limit the approach at all?  Fig 3b shows a likely considerable amount of interpolation using several combinations of points, but could even more be generated from each pair using additional random sampling either of the transformer params or auxiliary input?


l.155 "interpolating either support or query sets achieves higher loss than interpolating both" --- reading this sentence alone (before sec 5), it's ambiguous if what is meant by "higher loss" is better or worse.  Reading this section, this question was very much on my mind, so seeing that it would be addressed by this sentence was great, but it wasn't clear which way it would be resolved.

alg.2:  some small typos and inconsistencies, lines 4,6 L_v suggests this is the loss but is already the grad, would be more consistent to accumulate a sum of losses in L_v and then set v_1 <- d/d\theta L_v.  also "single" is misspelled in the subscript


limitations: -
review_title: null

Meta-review: metareview: This paper uses a set transformer to create new tasks at meta training time when the amount of data for meta training is scarce. This approach seems to be highly effective and will make a worthwhile contribution to the few-shot learning toolbox. Many of the reviewer concerns were addressed through additional ablations and experiments, e.g., using MAML instead of protonets. Please include these experiments in the final draft as they add quite a bit to the paper.
confidence: Certain
Decision: Accept