Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors propose debiased self-training (DST) as an approach to reduce both data (sampling) bias and self-training bias during SSL. The DST recipe consists of two main elements: 

1) Pseudo-labels are treated as a related task, in that they are used to train a dedicated pseudo-classifier and shared learned features, while the main task classifier is trained only on ground-truth data, and, 

2) The learned features are additionally adversarially optimized to minimize the performance of an additional "worst-case" classifer, which is trained to miminmize loss on supervised data, whilst maximizing the loss of predictions from the pseudo-classifier. 

Results with both cold-start and pre-trained features on several datasets demonstrate solid gains over SOTA SSL approaches.
strengths_and_weaknesses: Originality: 7
Quality: 7 (6 before rebuttal response)
Clarity: 7
Significance: 7
- While multi-classifier co-training approaches are ubiquitous in SSL approaches, multi-task learning is a standard way to improve feature representations for related tasks, and adversarial feature optimization has been widely investigated, optimizing the task features for a pseudo-classifer and adversarially against a worst-case classifier while training the task classifier is a nice, highly effective, and to my knowledge novel idea.
- With that said, the related work section lacks a summary of relevant adversarial techniques in general and as applied or potentially applied to SSL (e.g. VAT, VAaD, ALI, etc.....). This is the most serious limitation of the paper currently, and I urge the authors to address this and fully and completely situate their work in the rebuttal (otherwise, I will not be able to reccommend that the paper be accepted).
- In addition, adversarial SSL techniques ideally should be included in some of the results comparisions, although the methods that I am aware of are outperformed by many of the techniques already being compared.

Post-rebuttal:
- Thank you to the authors who have addressed my remaining questions and concerns by expanding the related work section, and including additional results comparing to and combining with DivideMix. I have increased my score from weak accept to accept. Well done!
questions: See limitations section.
limitations: - See S&W section.
- DivideMix and MixMatch should be cited/discussed relative to the current work. Wouldn't "DivideDST" (DivideMix utilizing DST instead of MixMatch) be even more effective?
- The overall loss function and the weight given the adversarial term (7) are not stated in the paper (looking quickly at the code, it seems to be 1.0). Please confirm this, and update the paper accordingly.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper analyzes the different types of bias in self-training, including data bias and training bias, and proposes to generate and utilize pseudo labels decoupled by two parameter-independent classifier heads to avoid error accumulation. Further, the worst case of self-training bias is estimated and used to guide the generation of pseudo labels. The proposed method is compared to several recent semi-supervised learning methods on a variety of datasets, and shown to better on most of the datasets.
strengths_and_weaknesses: + The experimental results are competitive compared to the other existing semi-supervised learning methods.

+ The proposed model is simple yet effective. The proposed idea of reducing the training and data bias is interesting.

- In terms of model performance, the proposed method is not the best on all the datasets. 

- While the comparison is extensive, there are a few representative SSL methods missing in the comparison such as UDA [a] that uses the different unsupervised learning for semi-supervised learning. 

- Additional ablation study on requirement of label size is missing. For instance, when varying the label size, how does the model performs? Is it sensitive to the label size as compared to the other existing approach? 

[a] Unsupervised Data Augmentation for Consistency Training
questions: 1. What is the parameter size and the computation cost of the proposed method compared to the other existing methods?

2. While the method is similar to the "mutual learning", except sharing the backbone of the two models, what could be the additional benefits upon "mutual learning"? Please consider to add some discussion. 

3. What are the difference of implementation between the proposed method and the FixMatch, such as data augmentation and training schedules? Does the proposed method require certain unique training setups that differs from other SSL methods? 

4. In terms of model performance, the proposed method is not the best on all the datasets. What could be the possible reasons that lead to its weaker performance on certain datasets? 
limitations: - In terms of model performance, the proposed method is not the best on all the datasets. 

- While the comparison is extensive, there are a few representative SSL methods missing in the comparison such as UDA [a] that uses the different unsupervised learning for semi-supervised learning. 

- Additional ablation study on requirement of label size is missing. For instance, when varying the label size, how does the model performs? Is it sensitive to the label size as compared to the other existing approach? 

[a] Unsupervised Data Augmentation for Consistency Training
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper suggests a new self-training approach called DST for semi-supervised learning. Usual self-training approaches generate incorrect pseudo-labelings and thus suffer from training instability due to the accumulated error. Specifically, based on data analysis, DST points out that there are two kinds of biases, namely training bias accumulated due to self-training strategy and data bias as a result of randomly sampled scarce samples. To address them, DST proposes two-phase solutions. First, to reduce training bias, DST adds a classification head to disentangle generation and utilization of pseudo labels. A default head is used for classification and pseudo-labeling, while another head is used to receive the pseudo-labels and perform training on unlabeled samples. Then, to reduce data bias, DST adversarially optimizes the worst training bias which is an indirect proxy of training bias.
strengths_and_weaknesses: ### Strengths

1. The motivation and the design of DST are well-connected to each other.
2. Figures help the understanding of the paper.
3. Empirical results support the effectiveness of the paper.

### Weaknesses

1. Some part of the paper is misleading or not clearly explained, is the  'worst head' $h^{'}$ different from $h$ or $h_{pseudo}$? And how to draw the 'worst possible head' $h^{'}$ during training in a hands-on manner? Does it require additional heavy computations?
2. Two aspects make me concerned about the suggested method: (i) when the pseudo-labeling is incorrect, the feature generator $\psi$ is still contaminated by the given signal. (ii) On the other hand, for the correct pseudo-labels, the pseudo-labeling head cannot gain their benefit to improve pseudo-labeling quality.
3. The paper is relying on an assumption for mistakes of $h^{'}$ that does not always hold. The discrepancy with the current pseudo-labeling function also happens when the updated pseudo-labeling head $h^{'}$ is perfect while the current pseudo-labeling $h$ is imperfect. In this case, the effect of model learning would result in an undesirable way to induce the model to learn a worse hyperplane.
questions: See the weakness above.
limitations: There is no severe limitation found in the paper so far.
review_title: null

Meta-review: metareview: This paper proposed a novel Debiased self-training (DST) approach to reduce both data bias and self-training bias during SSL. The proposed method is simple and empirically seems quite effective. Reviewers are generally positive about the novelty of the method and significance of the results. While authors have tried to address and improve some of the review issues, the related work section and empirical comparison with recent SOTA SSL methods still could be improved. For example, some recent related SSL works are still missing, including but not limited to
- DASO: Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learning, CVPR2022
- CoMatch: Semi-supervised Learning with Contrastive Graph Regularization, ICCV2021

Overall, the paper presents a novel framework for SSL, the empirical results of the method are quite positive, the paper can be accepted, but the authors are recommended to further improve the discussion/comparison of recent related work. 

confidence: Certain
Decision: Accept