Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper solves the problem of few-shot learning. The recent success of SSL and FSL proves that they can handle situations that few label data are provided. Motivated by this, the author proposed a novel framework IEPT that seamlessly integrates self-supervised learning methods to few-shot learning. Unlike other trivial combination of SSL and FSL methods, this paper proposed instance-level and episode-level pretext tasks to bring on closer integration. Further, this paper proposed to use transformer to integrate features from different images and augmentations. Experiments show the model achieves new SOTA. 

1. In the inference phase, you use transformer to integrate embedding from both support set and query set, which seemingly makes a transductive method. Therefore, you should compare your method to other transductive FSL methods.
2. Your ablation experiments are not complete. You are supposed to give results of training with L_{integ} (Eq.(10)) and L_{epis}  (Eq.(6)).
3. There is at least a baseline method you should compare. That is you train with L_{aux} (Eq.(11)) and L_{inst} (Eq.(3)), and inference by averaging outputs (ensemble).
4. You are using multiple augmentations on each image at test time. It seems unfair to most previous tasks. It is not clear if the success is due to the ensemble effect.
review_title: Pretext Tasks for Few-Shot Learning

final_score: rating: 7: Good paper, accept
review_content: This paper presents a method for combining self-supervised learning (SSL) (in the form of predicting the rotation applied to an image) with few-shot learning (FSL) in the domain of image classification. Compared to prior work, this paper introduces -- (i) a consistency loss which ensures FSL episodes with different rotations agree in their class predictions; and (ii) an integration method which derives the label of an image from a fused representation of all its rotations. These lead to an improvement over 3 FSL benchmarks.

Strengths:
- The paper is well-written, with extensive discussion of prior and contemporary work. The technical details are presented in clear precise terms. Despite not being an expert in this area, I had no difficulty understanding the FSL setup and the new contributions of this paper.

- The experiments are also quite thorough with convincing ablation studies and several additional details in the Appendix. Overall this is solid empirical work.

- The paper presents relatively simple ideas which lead to significant improvements. Hence, it is likely to be impactful for future work looking to build on these results.

Weaknesses:
- Though effective, the novel loss terms introduced seem rather ad-hoc. There is not much understanding gained from reading the paper about why these extensions help. A large part of the improvement over the baseline ProtoNet seems to come from the integration method (based on the ablation study). This is very interesting, but could have been explored in more depth. E.g. does this approach also work when we have many training instances?

- There is not much discussion of which hyperparameters were tuned, neither how sensitive the results are to this tuning.

Other comments:
- It is not clear what the visualization in Figure 3 represents, or how it "supports the effectiveness of episode-integration module" (section 4.3). Are the figures comparable to each other?

- Why do we need the extra mean function in Eq. 6, when there is already a summation over i and r?

- Please spell out the conference names in the references.
review_title: Solid empirical work

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes both Instance-level and episode-level pretext task. In comparison to existing works (Gidaris et al., 2019; Su et al., 2020), the main novelty is to design the episode-level pretext task, which enforces consistent predictions for images with different rotations. 

The paper is clearly written with experiments supporting the effectiveness. 
However, the novelty is limited. It is more like existing works (Gidaris et al., 2019; Su et al., 2020) plus the the regularization of consistency for images with different augmentations. However, the latter is also not new. Indeed, it has been used in [1-2], but the authors neglect them. 


[1] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
[2] Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv, 2016.
[3] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. In NeurIPS, 2016.


======
Comments after rebuttal: 
I know the authors develop two components for FSL, my concern is that these components are incremental and have limited novelty. 
However, I admit this paper is a high quality paper in presenting its idea, organization and empirical evaluation. Hence I increase my score to accept now. 

review_title: The novelty is mainly on "Episode-level Pretext Task"

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: In Instance-Level and Episode-Level Pretext Tasks for FSL the authors present a novel method to take advantage of auxiliary prediction tasks and consistency regularization tasks which have had large success in Self-Supervised Learning settings to improve upon FSL approaches. Furthermore, the authors incorporated a transformer-based predictor to improve upon to be used with multiple augmentations of an instance to improve upon naive averaging of multiple predictions.

Empirically, the authors demonstrate the benefits of incorporating both instance and episode-level tasks by showing significant improvements over the ProtoNet approach upon which this work builds and also achieving state-of-the-art results on several tasks with different architectural backbones. Through ablations this work demonstrates the benefit of each proposed additional loss and shows robustness to choices in pretext transformation and architectural backbone. 


This work is appropriately justified and explained and with satisfactory experimentation.

For section 3.3, it may help to explain that the integrated FSL task is presented as an alternative to prediction averaging. As written, the rationale for this approach is only apparent in subsequent sections.
review_title: IEPT cohesively combines elements of Semi-Supervised Learning and Few-Shot Learning to consistently produce state-of-the-art results on Few Shot Learning tasks.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper addresses the problem of few-shot classification by incorporating self-supervised learning into the standard episode-based meta learning. Specifically, it adopts the pretext task of rotation prediction into the episode design. For each sampled episode, additional episodes are constructed by using rotated examples in the original support set and query set. Two self-supervised losses are designed based on the augmented episode sets – (1) recognizing different rotation transformations as an instance-level pretext task; (2) ensuring consistent predictions of class labels across different episodes as an episode-level pretext task. Two few-shot losses are also designed – (1) predicting class labels for each individual episode; (2) predicting class labels for the fused episode set based on attention. Experimental evaluation is conducted on two standard few-shot classification benchmarks, namely miniImageNet and tieredImageNet, and shows improved performance.

Strengths:

++ Self-supervised learning and few-shot learning are two important techniques to transfer knowledge for addressing new tasks, but their combination is less explored.

++ The proposed approach that integrates self-supervised learning into the episode design is interesting.

Suggestions and questions:
	
-- Compared with the state-of-the-art methods, the performance improvements of the proposed approach are marginal. For example, while the proposed approach is more complicated, it is comparable to much simpler approach like Tian et al.

-- The proposed way of integrating self-supervised learning into episode learning is claimed to be general. In addition to the metric-learning based approach ProtoNet, it would be interesting to show if the proposed approach can be applied to other types of meta-learning techniques, such as optimization based approach MAML.

-- Following the previous comment, it would be interesting to show if the proposed approach can be applied to other self-supervised learning techniques, such as the recent MoCo or simCLR models.

-- In the design of a set of extended episodes, examples within the same episode belong to the same rotation transformation. How if this is not guaranteed? That is, the examples within the episode are randomly sampled from the rotation transformations.

-- In Eq 6, how if using a pairwise KL loss without computing the mean distribution in Eq 5?

-- Why further introducing an auxiliary loss is helpful, given that the attention mechanism already fuses the information from all the episodes?

-- How is the hyperparameter sensitivity regarding different ws in Eq 12?

-- Table 2 did not provide extensive ablations regarding different combinations of the loss functions.

Post-comments to the author's response:

After reading the other reviewers’ comments and the authors’ rebuttal, I am still concerned with the novelty of the approach, experimental evaluation, and performance improvements over previous work. These issues are pointed out by the other reviewers as well. Hence, I will go with my original decision of rejecting the paper.
review_title: Official Blind Review #2

Meta-review: The submission proposes instance-level and episode-level pretext tasks as an unsupervised data augmentation mechanism for few-shot learning. Furthermore, transformer are proposed to integrate features from different images and augmentations. The paper received one clear accept, one accept, one borderline accept and two borderline reject recommendations. The main concerns of the R5 and R2 were weak ablation study and the lack of a clear advantage of the method in terms of results compared to the prior state of the art. In the rebuttal, the authors provided more ablation studies. Similarly, the reviewers were concerned about the novelty of the paper being incremental compared to the prior works. Based on the majority vote, the meta reviewer recommends acceptance.

Decision: Accept (Poster)