Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper the authors have proposed FreeMatch, a novel Semi Supervised Learning (SSL) technique that samples unlabeled data (the goal is to add pseudo labels and use them as supplementary data for model learning) through a Self Adapting (confidence) Thresholding (SAT) mechanism. The hypothesis is that an adaptive threshold (depending on the status of the model) is likely to be more effective than a predefined or ad-hoc threshold,  usually used by existing techniques.  

At each training iteration, FreeMatch adjusts confidence thresholds (for each class in the context of a classification problem) by leveraging model prediction information from the corresponding step. SAT first estimates a global threshold, then modulates via the local class-specific thresholds, estimated as the Exponential Moving Average (EMA) of the probability for each class. The threshold is kept low at the beginning of the training process to accept as many weak(possibly correct) samples as possible. As the model becomes more and more confident in subsequent iterations, the threshold is gradually increased to identify and reject incorrect samples (which reduces confirmation bias). 

The proposed model has been tested on CIFAR-10/100, SVHN, STL-10, and ImageNet benchmarks and the results look to be encouraging. 

strength_and_weaknesses: Strength: Usually, semi-supervised learning (SSL) methods use either some predefined or ad-hoc threshold adjustment mechanisms  which are challenged  by FreeMatch through the idea of Self Adaptive Threshold (SAT). FreeMatch provides an automated iterative unlabelled data sampling methodology using  threshold adjustments based on the learning status of the model and using  prediction confidence scores at the latest. The methodology also ensures class  fairness and learns a generalized solution (in the context of classification).
 
FreeMatch has been tested on CIFAR-10/100, SVHN, STL-10, and ImageNet benchmarks and compared against existing techniques such as FixMatch, ReMixMatch, and FlexMatch. Reported results are found to be encouraging. 
 
Weakness: Although the reported results in section 5 look promising, they lack some statistical tests. It is suggested that authors perform proper statistical tests to justify whether the achieved gains are (statistically) significant or not.  If we zoom in to the results (Table 1 and 2) we find that gains are marginal for larger (such as CIFAR-100, ImageNet) and more complex (ImageNet) datasets when compared to smaller and simpler datasets such as CIFAR-10 (or data with less number of classes). This may have exhibited scalability limitations of the proposed approach. 

Also, it is surprising that for smaller datasets (such as CIFAR-10) the reported results are sometimes even better than models trained in a fully supervised fashion.


clarity,_quality,_novelty_and_reproducibility: The paper is well written with minor linguistic errors. The idea of the threshold adjustments is first presented through a simple binary conditional gaussian example which helps easily understand the concept and the context. The structure of the paper is well organized and the content is easy to follow. 
 
Self Adaptive Threshold (SAT), Self Adaptive Fairness, and the overall FreeMatch approach are the major novelty of this work.
 
The experiments (plus setup) were thorough and detailed. As the code has been shared, it is expected that the reproduction of the results is feasible.
summary_of_the_review: I have gone through the paper more than once including the appendices. Overall, the idea is quite sound, well articulated through the document, which is found easy to follow. The experiment is thorough and the reported results look encouraging. Self Adaptive Threshold (SAT), Self Adaptive Fairness, and the FreeMatch approach as a whole may benefit the SSL research.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: For the semi-supervised learning task, this paper argues that existing methods might fail to utilize the unlabeled data more effectively since they either use a fixed threshold or an ad-hoc threshold adjusting scheme, resulting in inferior performance and slow convergence. Then they propose FreeMatch to adaptively adjust the confidence threshold. Experiments on several related datasets demonstrate the superiority.
strength_and_weaknesses: Strengths:

(1) The motivation is clear.

(2) The presentation is good, and the paper is easy to follow.

(3) The proposed method is simple and effective.

Weaknesses:

(1) The main contribution lies in learning the threshold in an adaptive way. They assign various thresholds for different categories and iterations. Given the fact FlexMatch already set different weights for different classes, I think the overall novelty is unsatisfying. The contribution of self-adaptive fairness over existing strategy is also marginal.

(2) The improvements over FlexMatch on CIFAR10 and CIFAR100 are marginal.

(3) Please compare with FlexMatch under the same fairness regularization. For example, if you replace the SAF with that in FlexMatch, will the results are still better than FlexMatch?

(4) The authors claim that the proposed method can speed up the convergence. However, according to Figure 3(c), in 500k iteration, the proposed method is still not converged. Please give more explanation.

(5) For different datasets, different training strategy is adopted to improve the performance. For example, the authors define specific training requirement on STL-10, which is not consistent to the so-called self-adaptive property throughout the paper.

(6) In FixMatch and FlexMatch, the predefined threshold is very high to promise that highly-confident samples are selected for training. However, the proposed FreeMatch assigns low threshold at the beginning, which might incorporate some incorrect samples for training. Please explain this.

(7) There are several typos, such as "Eq equation 6" in page 5.

clarity,_quality,_novelty_and_reproducibility: Please refer to my detailed comments above.
summary_of_the_review: The overall novelty of self-adaptive threshold learning strategy is unsatisfying. Improvement over FlexMatch on CIFAR is marginal. More experiments are suggested to support some statements.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper present an adaptive approach to threshold selection for labeling of unannotated data during semi-supervised model training. The proposed approach combining adaptive threshold selection and a combination of losses leads to state of the art performance for a variety of SSL classification tasks.
strength_and_weaknesses: The authors to a good job of laying out th problem and presenting their proposed approach. The approach is clean and well motivated and the empirical results are strong.

One experimental result that would have been helpful to demonstrate the importance/impact of the components of the proposed approach would have been to show performance of the loss function (12) for a fixed threshold selected by optimizing validation performance. This would help show whether the adaptive thresholding scheme is serves to reduce hyperparameter-tuning computation or overall training time (in the event of similar performance for an optimally selected threshold) or whether the adaptive threshold outperforms any fixed threshold. Additionally, this would help demonstrate the improvement from the proposed self-adaptive fairness loss term independent of the adaptive threshold.

clarity,_quality,_novelty_and_reproducibility: The adaptive thresholding scheme presented is novel to my knowledge, as is the proposed formulation for incorporation of fairness in the loss function.
summary_of_the_review: Overall I'm in favor of accepting this paper as it combines novelty related to both the thresholding of unlabeled data and formulation of the loss function with empirically strong results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a semi-supervised learning algorithm that uses adaptive thresholds to select pseudolabels to be used during training. Reviewers generally appreciated the well-motivated idea, promising empirical results and clear presentation. Two reviewers thought the adaptive thresholding concept was sufficiently novel while the remaining reviewer had concerns about limited novelty and marginal improvements. Reviewers also had concerns about the importance of the proposed components. The authors provided a response that the AC thinks largely addresses these concerns. Authors are also encouraged to discuss recent work also proposing adaptive thresholding in the final version of the paper: Class-Imbalanced Semi-Supervised Learning with Adaptive Thresholding, ICML 2022.
justification_for_why_not_higher_score: - Key ideas are specific to the semi-supervised learning setting and may not be of broader interest (e.g. no large improvements)
justification_for_why_not_lower_score: - Proposed method is novel and well-motivated 
- Empirical evaluations show it is effective on a range of datasets
Decision: Accept: poster