Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The authors proposed a simple and straightforward method to address an obvious but critical and long-lasting issue in semi-supervised learning. They showed a number of analyses and demonstrated superior performance over state-of-the-art methods (mostly) consistently on five datasets. The proposed method is simple and easy to use without extra computation needed. Most importantly, I do see the possibilities of adapting this method widely across semi-supervised learning or self-training methods that use pseudo-labeling. I would recommend acceptance at this time. 

main_review: 
### Strengths: Describe the strengths of the work. Typical criteria include: soundness of the claims (theoretical grounding, empirical evaluation), significance and novelty of the contribution, and relevance to the NeurIPS community.

- Paper is well-written through entire paper and super easy to follow
- Proposed method is extremely simple without adding extra computation 
- Plan to public release TorchSSL, a PyTorch library for semi-supervised learning
- Providing two versions of evaluation metrics for fair comparisons with the existing method and argue that choosing the best error rates might be more suitable for future research because the convergence speed is also important and won't be able to be reflected in the median error rate of the last 20 checkpoints. 
- thorough experiments with 5 datasets and analysis on convergence speed
- analyze the failed cases and identify the potential limitation of the proposed method (L224 - L230)

### Weaknesses: Explain the limitations of this work along the same axes as above.
- Interestingly, I don't think I can find any obvious weaknesses. This is a well-written paper.

### Additional feedback, comments, suggestions for improvement and questions for the authors:
- L202: "When the amount" --> "when the amount"
limitations_and_societal_impact: The authors did analyze the failed cases and identify the potential limitation of the proposed method (L224 - L230).
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: The paper proposes to incorporate ideas from curriculum learning into semi-supervised learning. Semi-supervised learning consists on training models with only a subset of labeled samples, and a typically larger set of unlabeled samples. The paper proposes to improve on ideas from pseudo-labeling whereas fixed thresholds are defined to decide the pseudo-labels for unlabeled samples as the training process goes. In this approach, the idea is to modulate the thresholds used during the training process as a function of time (iterations) and the individual classes being classified. The motivation is that some classes might be harder to learn. In order to deal with initial conditions, the paper proposes to use a warming up strategy whereby an additional class (undecided) is considered for samples that have not reached a confidence to be assigned to any class. The proposed method in combination with other methods based on consistency regularization such as FixMatch lead to improved accuracies.
main_review: Strengths:
+ Curriculum learning is well motivated in terms of the presented task and seems a good strategy to pace the thresholds used in semi-supervised learning.
+ TorchSSL is proposed as a library re-implementing prior algorithms in a unified codebase. This could potentially provide a resource for further development in this area.
+ The proposed method is general enough that it can be combined with others and in experimental results higher accuracies are reported after applying the proposed method.

Weaknesses:
+ There has been already a prior proposal to use Curriculum-based pacing for choosing the thresholds for Semi-supervised learning for pseudo-labeling: Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning. AAAI 2021. February 2021. The idea of changing thresholds for deciding on the unlabeled samples hence has already explored before. Without direct comparisons or discussions it diminishes the contributions of the present work. However I do acknowledge some differences: the current work seems to consider per-category adaptation of the pacing on the curriculum, whereas this other work considers a single threshold. However whether this difference is important as claimed in the present work, it seems that it would be something that needs validation.
+ There were some technical details that could only be resolved after reading the supplementary material, something important such as what is or are the neural network architectures that are being trained for obtaining the results on Table 1. 
+ More importantly there are some inconsistencies reported on the number for Imagenet-1k where FixMatch is reported as having a top-5 error of 19.55% but the prior work both the original FixMatch paper, and the above referenced AAAI 2021 paper report 10.87% for FixMatch. The conditions seem the same 10% of data annotated (= 100k samples annotated). On the surface the presented method would be impressive as the reported FixMatch error is reported as 19.55% and it goes down to 13.96% -- however only if we accept a version of FixMatch that starts at a much worse level of performance this might be true. Section 4.2 is really light on the details for this part and from the Supplementary material it doesn't seem like the settings were fundamentally different than in these two prior works.
+ The authors cite [27] but don't seem to follow some of the recommendations in that work regarding the brittleness of many methods under unlabeled samples that include categories that are not part of the target set. Other recent works have tested also under this scenario but I don't seem this type of experiment in the current work. 
limitations_and_societal_impact: I don't believe I saw much in the way of limitations or any discussion about this part.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a curriculum learning strategy for semi-supervised learning. Their motivation is that most existing semi-supervised learning utilizes a fixed threshold to compute pseudo-labeling loss, but the fixed threshold should be adjusted according to the state of the model. Then, they propose to adjust the threshold based on the prediction of the model and design a class-specific confidence threshold. Their threshold is designed so that the class with a larger number of confident samples has a higher value of the threshold. Thus, the classes hard to learn will have a small threshold. Their way of designing the threshold is computationally efficient and will not cause much training time increase. 


The experimental results show that their method is very effective in challenging situations such as when the number of labeled samples is limited or when the target dataset is challenging. 
main_review: Strength
1. Their method is simple and easy to implement. It should be important in real applications. 
2. The idea behind the method is reasonable. 
3. Their method is computationally efficient but provides great gains in challenging semi-supervised learning settings. 
4. This paper is very easy to follow. 
5. Experiments support the effectiveness of their method. 

Weakness 
1. The idea of introducing curriculum learning for semi-supervised learning is not very novel as done in [A].
2. Given 1, their experiments lack comparison to semi-supervised learning methods based on curriculum learning. There should be other ways of designing curriculum such as gradually increasing the confidence threshold. This paper lacks empirical and intuitive comparison to this kind of method, which makes it hard to judge the novelty of the proposed method. 
3. Related to 2, the analysis of results is not enough to fully understand the method. From the intuition of the method, the expected results can be that this method improves accuracy on hard-to-classify classes. Did you observe the results?
4. The proposed threshold design will balance the number of unlabeled samples used to compute pseudo-labeling loss per class. If we have the objective to balance it such as entropy of marginal class distribution for unlabeled samples, do we have the same effect as the proposed method? 

In summary, this paper provides a simple and well-motivated technique to improve semi-supervised learning based on pseudo-labeling. However, it lacks comparison to existing methods with curriculum learning. If they can provide simple baselines with some curriculum learning, the readers would understand the advantage of the proposed method better. 


[A] Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning, AAAI 2021


---post rebuttal update---
The rebuttal addressed my concerns. Despite its simplicity, the proposed method is able to boost the performance of semi-supervised learning even for hard classes. I would like to raise my rating to 7. 

limitations_and_societal_impact: They lack a section to discuss the potential social impact of their research. I do not think the research includes a particular content, which can bring negative impact, but they need to mention it somewhere in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work extends the popular FixMatch framework with dynamic, class-specific thresholds. Specifically, a multiplier is designed to scale the fixed threshold for each class, which is computed based on the proportion of pseudo-labeled instances with confidence values passing the threshold—the motivation is to lower the threshold for classes that have fewer pseudo-labeled instances. The effectiveness of the proposed approach, named FlexMatch, is demonstrated on CIFAR, STL-10 and ImageNet where clear improvements over FixMatch are observed.
main_review: **Strength**

- The design of dynamic thresholds is simple and seems easy-to-implement.

- The improvements brought by the dynamic thresholds are clear on CIFAR, STL-10 and ImageNet.

**Weakness**

- My biggest concern is with the novelty of this work, which seems incremental. Though the improvements look good on three datasets, the method's generality is questionable, as reflected in the poor performance on SVHN. Lines 224-230 have associated the failure on SVHN to imbalanced classes, but more concrete analysis in this regard is missing. More insights should be provided to explain and understand why the performance on SVHN is worse than FixMatch.

- What's special about TorchSSL? Does it solve any particular engineering challenges that existing code fails to cope with? This section should be moved to the appendix as it doesn't provide any new insight.

---
Post-rebuttal update:

I was initially concerned with the generality of FlexMatch due to its relatively lower performance on the simple SVHN dataset, and the paper lacking sufficient discussions in regards to how FlexMatch should be applied in practice and under which circumstances FlexMatch might perform well/poorly. These discussions are essential for practitioners since FlexMatch's underlying mechanism is hand-crafted and could possibly overfit the curated datasets. The authors' responses have addressed this concern and I suggest the authors add these discussions to the paper.

Given that the proposed approach is simple and yet effective, which is also appreciated by other reviewers, I change my rating to clear accept.
limitations_and_societal_impact: The authors have discussed the limitations in Sec.4.1. There is no obvious negative impact associated to this work.
review_title: null

Meta-review: This paper proposes Curriculum Pseudo Labeling (CPL), an extension to semi-supervised learning algorithms that uses an estimate of how well the model has learned each class to set per-class thresholds for promoting unlabeled data to pseudolabeled data. This conceptually simple idea has a significant impact in practice. It is also simple to implement and does not significantly add to the computational cost of the base SSL methods. (It actually can speed up convergence significantly.) Further, the combination of CPL and FixMatch (called FlexMatch) sets new state-of-the-art scores on several SSL benchmarks. Finally, the paper includes the release of Torch SSL, a codebase for the study of SSL.

The reviewers generally liked that the method was conceptually simple, effective, and easy to combine with many existing methods. Three of the four reviewers clearly favored acceptance, for the above strengths. Some of the concerns they had initially were addressed during the discussion period, and the authors are encouraged to include these clarifications in the final version of the paper. One reviewer had serious concerns about the experimental setup. The results reported in the paper for baselines differ from some previously reported in the literature. The authors attribute this to both a difference in the number of labeled examples (which is fair to vary in SSL experiments) and differences in hyperparameters. The authors are encouraged to both clarify whether the hyperparameters were tuned to be optimal for each baseline and also to add comparisons on the same number of labeled examples as previously used in the literature (perhaps in an appendix) for a more detailed comparison.
Decision: Accept (Poster)