Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work identifies that the existing approaches in semi-supervised learning minimize a biased risk and hence, are devoid of theoretical guarantees unless there are strong assumptions. This works gives a method to de-bias the loss with a simple estimator which also allows them to give theoretical guarantees on the risk. They also evaluate their debiased versions of existing semi supervised learning approaches on various datasets and show that they lead to improved calibration and improved accuracy in certain settings.
strength_and_weaknesses: Semi-supervised learning is an interesting area and coming up with theoretically sounds methods is an important problem. The idea of debasing the risk is simple and interesting and also leads to an unbiased estimator of the risk with theoretical guarantees. The toy example provided is appealing. Their method also leads to improved calibration and better accuracy on certain subgroups.

The main weakness is that their method does not lead to improved accuracy on many of the datasets. It would be nice to have some discussion of the properties of the dataset which leads this method to outperform others in terms of accuracy.  They also show that the debiased version of Fixmatch leads to better accuracy but not the debased version of Pseudolabels. Is there some reason for why this is the case? It would also be good to discuss why their method leads to better accuracy on certain subgroups when using Fixmatch. 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and clear. They have included the relevant prior work. The idea of computing the entropy term on the labelled data as well to make the loss estimator unbiased is novel in this context to the best of my knowledge. 

They have also released the code for their experiments.
summary_of_the_review: The idea is interesting, simple and natural with theoretical guarantees. The paper is very well written. But, I think the experimental section is weak as discussed above.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Empirical Risk Minimization (ERM) is an established technique and is one of the basis of supervised (plus some other types) learning. The ERM objective is to minimize the risk when fitting a set of labeled data points. In the context of Semi Supervised Learning (SSL), as we need to deal with data points with and without labels, existing conventional approaches combine (usually through some weighting) two independent losses; however, they lack concrete theoretical guarantee (one of the claims of this paper). One of the the objectives of this paper is to address this issue and ensure the unbiased (in the context of data distribution) semi supervised learning.
 
Traditional SSL techniques (such as low density separation and consistency regularization based) heavily depend on labeled data points and sometimes overweight them when compared to the corresponding unlabeled set. This may look obvious as unlabeled data come with label uncertainty (compared to labelled data). The hypothesis is even unlabeled data come with uncertain labels; they shouldn’t be weighted poorly (compared to labeled data) to learn a proper model.
 
In this work (DeSSL), the SSL problem has been formulated as s ERM problem which also includes a debiasing term to control/minimize the labeled data bias. The main hypothesis is unlabeled data should be taken as complementary not the competitor of the available labeled data. The debiasing term can be thought as a regularizer to penalize higher weights to labeled data and thus control the data bias.
 
The proposed approach has been tested on CIFAR-10, CIFAR-100 and compared against FixMatch. Reported results are found to be encouraging.
strength_and_weaknesses: Strength: The proposed DeSSL model is based on the hypothesis that existing SSL techniques put less weight on unlabeled data and more on the labelled set and thus bias the learning. If this biasing problem can be solved, it is likely that SSL model performance will improve. The debiasing methodology, presented in this work, comes with a set of theorems and corresponding proofs that make the paper theoretically strong. The experiment results, especially the boosted worst-class  performance, signals that  the debiasing idea is working to an extent (for classification tasks). 
 
The proposed approach has been tested on CIFAR-10, CIFAR-100 and compared against FixMatch, a known technique in SSL.  The most interesting result is the DeSSL worst-case accuracy, which is noticeably better than FixMatch for both CIFAR-10 and 100 benchmarks although the overall accuracy gain for CIFAR-100 was marginal.

An additional advantage is that the proposed  DeSSL method can be easily applied to some of the available SSL algorithms with minimal changes/efforts.

Weakness: Although the results reported in section 4 look encouraging it is difficult to judge whether the reported gains are statistically significant. It is suggested that authors perform some statistical tests to measure if the performance gains are statistically significant.

The overall accuracy gain for CIFAR-100 was marginal (compared to CIFAR-10) which signals scalability limitations of the approach.

clarity,_quality,_novelty_and_reproducibility: The paper is well written, the structure is well organized and the content is easy to follow. The debiasing idea in the context of Semi Supervised learning (SSL) is presented clearly with minimum fuzziness. 
 
There have been a number of innovations especially the debiasing idea itself plus its theoretical derivation including proofs of some of the underlying theorems.  Another advantage is the DeSSL method can easily be applied to some  existing SSL algorithms without making any assumptions on the distribution of data.  The paper also provided some theoretical guarantee estimates  (on the safeness of the proposed methodology) on consistency, calibration, asymptotic normality and generalization error.

As the code has been shared, it is expected the results can be reproduced. 

summary_of_the_review: I have gone through the paper more than once including the appendices. Overall, the debiasing idea is quite sound, well articulated, and the document is found easy to follow. Learning by balancing the contribution from both labeled and unlabelled data is well supported through experiments and  corresponding results. Some important SSL theoretical derivations have made the paper even stronger.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper tries to give theoretical analysis about the safeness of semi-supervised learning, and proposed a debiased method that can be applied to the classical semi-supervised learning objectives. Experimental results show the proposal can improve the performance of FixMatch algorithm.
strength_and_weaknesses: Strength:
1) The proposal is clear and easy to understand.
2) The proposed debias method is general and can be applied to various semi-supervised learning methods.

Weakness:
1) The safety of semi-supervised learning means its performance will be better than simple supervised learning. Why the deep SSL, such as FixMatch will be unsafe? From the experimental results of other SSL methods, it seems they do not suffer from this problem.
2) The theoretical analysis seems did not give the guarantee that the performance of the proposed method will be better than the supervised learning method. So it may be inconsistent with the author's claim.
3) The experimental results are not convincing. The author should apply the proposal to more SSL methods (such as PL, MixMatch, FixMatch, ReMixMatch, UDA, etc) and more commonly adopted datasets (such as STL-10, SVHN, Image-Net).
4) The paper writing needs to be improved. For example, Figure 1 needs to be reorganized.
clarity,_quality,_novelty_and_reproducibility: The proposal is simple and easy to follow. But the theoretical and experimental parts are not convincing.
summary_of_the_review: Based on the above discussion, the paper is not ready to be published, so I tend to reject this paper.
--------------------------------

The author's responses address my concerns. Based on the author's efforts and other reviewers' comments,  I change the score to borderline accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes DESSL, a general methodology to debias the risk estimate of the SSL objective under the MCAR assumption (Equation 5). Authors conducted comprehensive theoretical investigation of the approach, including its connection with control variates and constrained optimization, its finite-sample guanrantee in estimation variance, its asymptotic guanrantee in calibration, consistency and normality, as well as its generalization bound in terms of Radamacher complexity (Section 3). Authors applied DESSL to popular SSL methods (EntMin, PseudoLabel and FixMatch) and tested them on simple benchmarks (MNIST, CIFAR, MedMNIST), showing clear improvement over baselines.
strength_and_weaknesses: Strength: 
* A simple debiasing method with strong and comprehensive theoretical backing. 

Weakness:

I am rather happy with the simplicity of the method and the theoretical backing of the approach, and just have some suggestions in terms of experiment evaluation, and in terms of adding theoretical comments around assumption violation.

* Missing intrinsic evaluation: Authors derived variance-reduction and calibration property of the method. Consequently, I do expect to see some experiment results on risk estimation quality (bias and variance) and calibration property (e.g., measured by proper scoring rule) to validate the theoretical argument in practice.
* Missing theoretical comments / empirical discussion on how the method may perform if MCAR is violated.
* Investigation is restricted to simple academic benchmarks, evaluation on realistic large-scale benchmarks (e.g., ImageNet) with rich data distribution and large number of output categories is missing. (Although I believe the current empirical evaluation is sufficient given the theoretical contribution of the paper).
clarity,_quality,_novelty_and_reproducibility: The paper is clearly structured and well written.
summary_of_the_review: This work proposes a simple approach to improve risk objective estimation in SSL. The method is simple to implement, backed by comprehensive theory, and is sufficiently evaluated on standard benchmarks. I believe the experiments should be extended to very some of the key theorems in practice (e.g., variance reduction and calibration), but otherwise the quality is sufficient to pass the ICLR bar.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work presents a novel SSL training objective, which specifically addresses the issue of bias that can be introduced when using other standard SSL approaches.  The proposed SSL approach is theoretically motivated by proving the consistency of the empirical proposed risk and also by providing a finite sample generalization bound. Both of these are proven under certain assumptions, such as labels "missing completely at random" (MCAR).  The approach is further validated empirically using benchmarks datasets such as MNIST, CIFAR-10/100 and compared the original and unbiased variants of various SSL approaches (e.g. Fixmatch vs. DeFixmatch, PseudoLabel vs DePseudoLabel). In cases where certain additional assumptions hold (such a clusterability), the authors find similar performance to existing biased approaches, but in other cases find improvement over biased counterparts, especially in sub-metrics such as worst class accuracy.

Reviewers all agree that additional empirical evaluations can further strengthen the paper (and I acknowledge that the authors have already provided additional empirical results since the discussion), but they also agree that the stronger contributions of the paper are the theoretical conclusions. Taken together, I recommend accepting the paper for publication.

One final request to the authors is to clarify and caveat the use of the phrase "safe SSL" distinguishing it from its use in other lines of SSL work (which has already been done to some extent in the revised submission), as this was one strong point of contention during discussion.

justification_for_why_not_higher_score: The paper presents several interesting theoretical conclusions derived from a novel SSL risk objective, which can potentially lead to significant real-world impact.  However, the current empirical results, while sufficient for publication, I think are not conclusive enough to understand the broader practical impact of the work.
justification_for_why_not_lower_score: The paper presents a novel approach with theoretical insight as well as some initial empirical motivation -- it it deserves to be presented at this venue.
Decision: Accept: poster