Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes a novel negative pseudo-labeling algorithm to tackle semi-supervised few-shot learning. The key insight is negative labels are easier to predict, therefore, pseudo labels on unlabeled samples can be better predicted by iteratively predicting negative labels until all the negative ones are excluded. Extensive experiments have been conducted on four few-shot learning benchmark and show better performance than SOTA. 
strengths_and_weaknesses: Strengths

1. The idea of generating pseudo-labels by gradually rejecting negative labels is novel and interesting. 
2. The experiments are quite extensive, including results on four public benchmarks and many analysis. 
3. The paper is written well and easy to follow.

Weakness

1. Although some negative labels may be indeed easier to be predicted than the positive one, there still exist hard negative classes that are equally hard to be recognized. Those hard negative classes are in fact the most important information for learning a good classifier.  I am missing how this work can handle this case. 

2. Although the method is simple and novel, the achieved improvement over SOTA is marginal (less than 1%) in most of cases, see Table 1.

Post-rebuttal 

My concerns about the weakness have been addressed.  
questions: I would suggest the authors to address my concerns regarding the marginal improvement and hard negative classes, as listed in the weakness. 

Post-rebuttal 
The authors addressed all of my concerns. Therefore, I would recommend a Weak Accept.
limitations: I cannot find the limitations and potential negative societal impact in this paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proses a simple approach to SSFSL that employs negative label prediction when producing pseudo-labels for unlabelled examples. The model consists of a standard ResNet12 architecture that is pre-trained on the base data following an L2-normalizing single-layer classifier that is fine-tuned based on the support data. The fine-tuning process first updates parameters based on standard cross-entropy loss of the support examples. It then iteratively removes negative pseudo-labels from the unlabelled examples but using a modified cross-entropy negative label predictor loss, and finally, performs updates based on the positive pseudo-labels of the unlabelled set for examples which all negative labels have been identified. Various experiments and ablations studies are reported that demonstrate the efficacy of the approach.
strengths_and_weaknesses: Strengths:
- Paper is overall very well-written.
- Algorithmic choices are well-motivated, backed up by both good intuition and supportive ablation studies.
- Method is simple to understand on the first go but also very empirically powerful as demonstrated by series of experiments.
- Negative labelling is a very interesting insight and can prove consequential specifically in the domain of SSFSL which is very applicable to applied settings where labelling can be expensive but lots of unlabelled data is available.

Weakness:
- There are some typos, such as "detailedly" (261), "can performs" (208), and most important "logits = f.forward(x)" and
"loss = F.nll_loss(F.log_softmax(logits), labels)" in the algorithm blocks where "x" should be "S" and "labels" should be "targets" to my understanding of the procedure
- The algorithm block is in PyTorch which I personally appreciate but can be difficult to navigate if read doesn't have existing PyTorch proficiency; I believe a pseudo-code algorithm block would be more appropriate with the PyTorch code moved to the supplementary material. I recognize that this was done to reinforce the fact that the method can be implemented in a few-lines of code.

Middleground:
- The algorithm is simple and effective; but as a result doesn't contain very significant technical novelty and contribution. That being said, the authors have embraced its simplicity in the language of the paper throughout which addresses this potential problem.
questions: The choice to decouple the training of the feature extractor from the few-shot learning algorithm itself is an interesting one. It is often seen that end-to-end training of the extractor through episodic procedures where the few-shot updates are also applied results in better performance. More specifically, the updates shown in the PyTorch code block could be applied directly to f and F together where the inputs are just the raw images. Was this something the authors explored? If so, what was the outcome? If not, why not?
limitations: The authors have adequately addressed technical limitations of the work (although further studies on empirical biases based on data domain would have been interesting to see). However, there is no discussion of potential negative societal impact of the work; in fairness to the authors, this is an algorithmic work and the societal impacts can be speculative at time; but they could benefit from a short discussion of what their method, as an effective SSFSL classifier, can enable in applied industrial settings.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This submission proposes a simple yet effective learning method for Semi-Supervised Few-Shot Learning (SSFSL) called MUSIC. Compared to previous methods, the authors propose to learn negative labels first and then focus on positive label learning. The underlying logic is that under few-shot learning scenario, it is easier to exclude negative predicted labels than select positive labels. The experiments show that the simple method achieves state-of-the-art performance on four benchmark datasets. 
strengths_and_weaknesses: + The authors propose a straightforward learning method for SSFSL. The motivation is clear and technical details are illustrated with sufficient details.

+ The authors conduct experiments on four benchmark datasets and achieves state-of-the-art performance.

+ The authors further dive into different aspects of the MUSIC method and provide ablation study, which is much appreciated.
questions: Overall I have little confusion for the methodology and technical details. There are two suggestions:

1) In the ablation study, the authors only investigate whether the reject option $\delta$ is effective or not. It would be better to further study what would be the optimal value of $\delta$ and whether this hyper parameter is agnostic to different datasets' distribution.

2) In Table 4, the authors conduct an interesting experiment for the order of negative and positive pseudo labels learning. I am curious what is the optimal number of iterations for the neg -> pos -> neg ... in the MUSIC method, or it is dependent on different datasets?
limitations: I am basically satisfied with the submission in terms of methodology, motivation and experiments. I provided some suggestions in the above section ("questions") which also contain some constructive suggestions.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work applies negative learning to the problem of semi-supervised few-shot learning. It uses negative pseudo-label to gradually get rid of unlikely predictions. It also minimizes the entropy of the predicted probability of unlabeled target domain data to promote pseudo-labeling.
strengths_and_weaknesses: Strength:
In general, the paper is easy to follow and highlights the key point clearly.

Weakness: 
1. The major problem of this work lies in the experiment section. The author mentioned that this work pretraining the network as in previous work [38], so it should be the baseline of this work. However, the table does not include that work [38] at all. Furthermore, comparing Table 2 in the baseline work [38] and Table 1&2 in this work, we can find that the baseline work [38] actually performs better in some settings. 

2. While negative learning is the key point of this work, as shown in Table 1&2, the improvement from using only positive pseudo-label to using both negative and positive pseudo-labels seems trivial. Since pseudo-label itself is a well-known idea, the novelty will be very limited if the negative pseudo-labels do not really make a difference.

 
questions: 1. The entropy-loss in (6) seems to share a similar spirit with pseudo-labeling (also known as soft pseudo-labeling in other places). What if you apply only (6) and do not use (5) at all? 

2. The author of the baseline work [38] also uses pseudo labels, and they rank pseudo labels in a sophisticated way. When you say pretraining the network as in [38], do you mean you use their method to pre-train the model, or do you mean you pre-train it in the same way but do not use pseudo labels at all during pre-training? 
limitations: The author needs to compare the baseline work [38] clearly to show a more convincing result.
review_title: null

Meta-review: metareview: This paper aims to improve semi-supervised few shot learning by utilizing negative pseudo-labels. The authors report significant improvement over the previous methods in this setting. The reviewers originally had concerns about the significance of the results, but after the discussion period they all supported acceptance more than they supported rejection.  

Given the simplicity of the method, the size of the improvements, and the unanimous agreement from the reviewers, I support the acceptance of this paper. While the authors improved the paper significantly during the discussion stage, I would urge them to keep working on the presentation and writing for the camera-ready version. There are still writing mistakes throughout the paper, and the meaning of some of the sentences is not clear. 
confidence: Certain
Decision: Accept