Debiased Self-Training for Semi-Supervised Learning

Abstract: Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets. Yet these datasets are time-consuming and labor-exhaustive to obtain on realistic tasks. To mitigate the requirement for labeled data, self-training is widely used in semi-supervised learning by iteratively assigning pseudo labels to unlabeled samples. Despite its popularity, self-training is well-believed to be unreliable and often leads to training instability. Our experimental studies further reveal that the bias in semi-supervised learning arises from both the problem itself and the inappropriate training with potentially incorrect pseudo labels, which accumulates the error in the iterative self-training process. To reduce the above bias, we propose Debiased Self-Training (DST). First, the generation and utilization of pseudo labels are decoupled by two parameter-independent classifier heads to avoid direct error accumulation. Second, we estimate the worst case of self-training bias, where the pseudo labeling function is accurate on labeled samples, yet makes as many mistakes as possible on unlabeled samples. We then adversarially optimize the representations to improve the quality of pseudo labels by avoiding the worst case. Extensive experiments justify that DST achieves an average improvement of 6.3% against state-of-the-art methods on standard semi-supervised learning benchmark datasets and 18.9% against FixMatch on 13 diverse tasks. Furthermore, DST can be seamlessly adapted to other self-training methods and help stabilize their training and balance performance across classes in both cases of training from scratch and finetuning from pre-trained models.

Reviews and author responses:

 We'd like to thank Reviewer G3sn again for providing an impressively insightful pre-rebuttal review, which has enabled us to make an effective response. We'd also thank you for carefully judging our feedback and acknowledging our work in the final review.

 Thanks for the enthusiastic reply from the authors. It was a great journey with the authors; because all my concerns were properly and clearly resolved by additional experimental supports, I would like to higher my review score to accept.

 **Q4**: The paper is relying on an assumption for mistakes of $h'$ that does not always hold.  What if $h'$ is perfect while $h$ is imperfect on some data points? 

It should be clarified that this paper does not rely on the assumption that $h'$ is worse than $h$ on each data point.

- As mentioned in $\underline{\text{Section 3}}$,  *bias* is defined as an expectation error on a distribution, rather than the accuracy of a specific data point.
- Our approach is able to reduce *data bias* as long as $h'$  has a larger bias than $h$.

The hypothesis $h'$ has a larger bias than $h$ has been confirmed experimentally and is theoretically reasonable.
- As shown in $\underline{\text{Figure 8 (Appendix B.5)}}$, the bias of $h'$ (in red) is much larger than that of $h$ (in green) throughout the training process.
- Strict proof is quite difficult for multi-classification problems. For simplicity, we can consider the binary classification problem, just as many learning theories did. Then we have the conclusion that as long as $h$ performs better than random guesses, $\underline{\text{Equation 6}}$ will encourage $h'$  to have a larger bias than $h$. In other words, if the accuracy of $h$ is greater than 50\%, then $\underline{\text{Equation 6}}$ will encourage the predictions of $h'$ opposite from $h$, resulting that the accuracy of $h'$ to be lower than 50\%. Note that SSL usually assumes that $h$ performs better than random guesses on the unlabeled data. Otherwise, this will be an ill-defined problem, and no SSL method can solve it. 


 **Q3**: Two concerns: (i) When the pseudo-labeling is incorrect, the feature generator is still contaminated by the given signal. (ii) For the correct pseudo labels, the pseudo-labeling head cannot gain their benefit to improve pseudo-labeling quality.

- As you mentioned, training with noisy pseudo-labeled data has both benefits and risks. But the backbone and head might have **different tolerances for noisy pseudo labels**. To verify this, we perform the following experiments.  

  - We randomly select two subsets $\mathcal{S}_\text{clean}$ and $\mathcal{S}_\text{noisy}$ of size $1000$ and $5000$ on *CIFAR-100*, where all labels in $\mathcal{S}_\text{clean}$ are accurate, while labels in $\mathcal{S}_\text{noisy}$ are noisy. The noise ratio of labels in $\mathcal{S}_\text{noisy}$ is denoted by $\gamma$. We always train both the head and the feature extractor with clean data in $\mathcal{S}_\text{clean}$ and focus on how to exploit noisy data. Specifically, we compare four methods:

    - (1) Clean Only: do not use data in $\mathcal{S}_\text{noisy}$;
    - (2) Noisy Head:  $\mathcal{S}_\text{noisy}$ is only used to train the head;
    - (3) Noisy Backbone: $\mathcal{S}_\text{noisy}$ is only used to train the backbone (our suggested method);
    - (4) Noisy All: $\mathcal{S}_\text{noisy}$ is used to train both the head and the backbone (FixMatch).
    
    The table below reports the accuracy of these methods when $\gamma$ equals $5\%$, $10\%$, and $20\%$ (The accuracy is $75.7$ when $\gamma$ is $0$). Results suggest that the backbone has a better tolerance for noisy pseudo labels compared with the head.

  |                | Train head with clean data | Train backbone with clean data | Train head with noisy data | Train backbone with noisy data | $\gamma=5\%$ | $\gamma=10\%$ | $\gamma=20\%$ |
  | -------------- | :-------------------: | :--------------------------------: | :-------------------: | :-------------------------------: | :---------: | :----------: | :----------: |
  | Clean Only     |     $\checkmark$      |            $\checkmark$            |                       |                                   |    61.5     |     61.5     |     61.5     |
  | Noisy Head     |     $\checkmark$      |            $\checkmark$            |     $\checkmark$      |                                   |    67.1     |     66.2     |     63.2     |
  | Noisy ALL      |     $\checkmark$      |            $\checkmark$            |     $\checkmark$      |           $\checkmark$            |    73.7     |     72.3     |     68.0     |
  | Noisy Backbone |     $\checkmark$      |            $\checkmark$            |                       |           $\checkmark$            |  **74.3**   |   **72.9**   |   **71.7**   |

- Since the backbone and head have different tolerances for noisy pseudo labels, **the benefits of using pseudo-labeled data on the backbone outweigh the risks, while the risks of using pseudo-labeled data on the head outweigh the benefits**. Therefore, avoiding training the head with pseudo-labeled data will bring performance gains on the original FixMatch as shown in the ablation study ($\underline{\text{Table 3}}$). Corresponding results are also shown below.

|                                       | Train head with clean data | Train backbone with clean data | Train head with noisy data | Train backbone with noisy data | Supervised Pre-training | Unsupervised Pre-training |
| ------------------------------------- | :-------------------: | :--------------------------------: | :-------------------: | :-------------------------------: | :---------------------: | :-----------------------: |
| Baseline                              |     $\checkmark$      |            $\checkmark$            |                       |                                   |          48.2           |           46.5            |
| FixMatch                              |     $\checkmark$      |            $\checkmark$            |     $\checkmark$      |           $\checkmark$            |          53.1           |           51.4            |
| DST w/o worst (nonlinear pseudo head) |     $\checkmark$      |            $\checkmark$            |                       |           $\checkmark$            |        **60.6**         |         **60.9**          |



 We would like to sincerely thank Reviewer G3sn for providing insightful reviews and valuable comments. We have clarified the questions in the following response.

**Q1**: Is the worst head $h'$ different from $h$ or $h_{pseudo}$?

Sorry for the misunderstanding. 
- The worst-case head, main head, and pseudo head do not share parameters. To avoid misunderstandings, we add a description of the overall loss function in $\underline{\text{Section 4}}$.

- As mentioned in $\underline{\text{Appendix A.1}}$, the worst-case head and non-linear pseudo head have the same architecture, but with a different architecture from the linear main head. 

**Q2**: How to draw the 'worst possible head' during training in a hands-on manner? Does it require additional heavy computations?

* As mentioned in $\underline{\text{Appendix B.5}}$, we optimize $\psi$ and $h'$ with stochastic gradient descent alternatively. In other words, during each iteration, the minimization or maximization is performed only once. For more details, you can check the code provided in the supplementary material.
* The alternative minimax game will introduce little computation cost during training and no additional computation cost during inference. As mentioned in $\underline{\text{Appendix B.5}}$, when training $1000k$ iterations on *CIFAR-100 using 4 2080 Ti GPUs*, FixMatch takes $104$ hours while DST takes $111$ hours, only a $7\%$ increase in time.


 **Q5:** There are a few representative SSL methods missing in the comparison such as UDA.

- We have included UDA in both $\underline{\text{Table 1}}$ and $\underline{\text{Table 2}}$ in the original paper. Below is the average performance of UDA and DST under different settings.

    |Method|Train from Scratch|Supervised Pre-training|Unsupervised Pre-training|
    |:----|:---:|:---:|:---:|
    |UDA|55.4|59.6|58.7|
    |DST (FixMatch)|**78.3**|**71.1**|**68.7**|

- To further address your concern, we add more results in $\underline{\text{Table 1}}$ and $\underline{\text{Table 2}}$ of the revised draft. Below is the average performance of these methods.

    |Method|Train from Scratch|Supervised Pre-training|Unsupervised Pre-training|
    |:----|:---:|:---:|:---:|
    |VAT [1]|23.0|55.4|51.3|
    |ALI [2]|22.7|51.9|49.1|
    |RAT [3]|34.2|58.3|56.1|
    |MixMatch|46.9|58.7|56.0|
    |DST (FixMatch)|**78.3**|**71.1**|**68.7**|

    [1] Virtual adversarial training: a regularization method for supervised and semi-supervised learning. PAMI 2018

    [2] Adversarially learned inference. ICLR 2017

    [3] Adversarial transformations for semi-supervised learning. AAAI 2020

**Q6:** Ablation study on the requirement of label size is missing.

- We evaluate the performance of DST with $1000$ labels ($10$ labels per class) on *CIFAR-100* in $\underline{\text{Appendix B.3}}$.

    |Method|Supervised Pre-training|Unsupervised Pre-training|
    |:----|:---:|:---:|
    |Baseline|61.5|56.2|
    |FixMatch|67.8|64.2|
    |FlexMatch|71.2|71.1|
    |DebiasMatch|73.5|73.9|
    |DST (FixMatch)|**75.6**|**76.8**|

- Further, we conduct more fine-grained experiments on *CIFAR-100* with supervised pre-trained models. 
    - The table below compares the accuracy of Baseline, FixMatch, and DST when the number of labeled data per class **varies from $1$ to $25$**. 

        |Labels per Class|1|2|4|6|8|10|16|25|
        |:----|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|
        |Baseline|23.6|34.5|48.2|54.5|59.2|61.5|66.6|70.8|
        |FixMatch|2.1|23.5|53.1|63.2|66.1|67.8|73.4|76.1|
        |DST|**36.4**|**56.7**|**70.4**|**72.7**|**74.3**|**75.6**|**79.2**|**80.3**|

    - The corresponding visualization is in $\underline{\text{Figure 5 (Appendix B.3)}}$.
    
    - Results suggest that DST is less sensitive to the amount of labeled data than FixMatch. 


 We would like to sincerely thank Reviewer KT8G for providing insightful reviews and valuable comments. We have clarified the questions in the following response. All the updated parts in the revised draft are highlighted in blue font.

**Q1:** What are the parameter size and the computation cost of the proposed method compared to other existing methods?

- $\underline{\text{Appendix A.1 and A.2}}$ have already mentioned the architecture and projection dimension of the introduced heads in general. 

- Further, we list the parameter size of models used in our experiments below. 

    |Method|WRN-28-2|WRN-28-8|WRN-37-2|ResNet50|
    |:----|:---:|:---:|:---:|:---:|
    |FixMatch (UDA, FlexMatch)|1.60M|23.4M|6.66M|27.6M|
    |DST|1.67M (+4.4%)|24.6M (+5.3%)|6.73M (+1.0%)|33.2M (+16.7%)|

- $\underline{\text{Appendix B.5}}$ has discussed the computation cost. When training $1000k$ iterations on *CIFAR-100 using 4 2080 Ti GPUs*, FixMatch takes $104$ hours while DST takes $111$ hours, only a $7\%$ increase in time.

- Note that the introduced heads will be discarded during inference. **Thus, DST will not introduce any additional parameters and costs during inference.**

**Q2:** While the method is similar to "mutual learning", except sharing the backbone of the two models, what could be the additional benefits upon "mutual learning"?

* As mentioned in $\underline{\text{Related Work (Line 88-90)}}$, the key difference between mutual learning and DST is whether they generate and utilize pseudo labels independently.
    * In mutual learning, each classifier head is still trained with potentially incorrect pseudo labels generated by other heads. 
    * In DST, the classifier head that generates pseudo labels is never trained with pseudo labels, leading to better tolerance to inaccurate pseudo labels.
    * If mutual learning is called *bidirectional learning*, then DST can be called *unidirectional learning*.
* As shown in $\underline{\text{Table 3}}$, the benefits of mutual learning are limited since both heads in mutual learning will be trained with unreliable pseudo labels from the other head. In contrast, our design mentioned in $\underline{\text{Section 4.1}}$ brings larger improvements.  

**Q3**: What is the difference of implementation between the proposed method and FixMatch? Does the proposed method require certain unique training setups?

- There is no difference between their data augmentations and training schedules.

    - As shown in $\underline{\text{Section 5}}$ and $\underline{\text{Appendix A.2}}$, we adopt the **same hyperparameters, network architectures, data augmentation and training schedule as FixMatch**. The choice of data augmentation, training schedule, and most important hyperparameters when trained from scratch is listed as follows. 

        |Weak Augmentation $\alpha$|Strong Augmentation $\mathcal{A}$|Training Schedule|
        |:---:|:---:|:---:|
        |random-horizontal-flip|RandAugment|cosine learning rate schedule|

        |Total Iterations|learning Rate|Mini-batch Size|Weight Decay|confidence threshold|
        |:---:|:---:|:---:|:---:|:---:|
        |1000k|0.03|512|{0.0005, 0.001}|0.95|

- DST does not require certain unique training setups that differ from other SSL methods.

    - DST is evaluated both when trained from scratch and finetuned from some pre-trained models. 

    - As stated above, the implementation of DST is the same as FixMatch.

    - $\underline{\text{Table 4}}$ also suggests that DST is not tailored for FixMatch and can improve the performance of $5$ different SSL methods (Mean Teacher, noisy student, DivideMatch, FixMatch, and FlexMatch).

**Q4:** The proposed method is not the best on all the datasets. What could be the possible reasons that lead to its weaker performance on certain datasets?

- We conjecture this is because the performance of DST is related to that of the base method. By default, we use FixMatch because it is simple and representative. However, its performance may be weaker than that of subsequent methods, such as FlexMatch.

- To verify this, we add the results of DST (FlexMatch) in $\underline{\text{Table 2}}$. DST (FlexMatch) surpasses existing SSL methods on all the datasets. Results with supervised pre-trained models are also shown below.

    |    |Caltech|CF-10|CF-100|SUN|DTD|Aircraft|
    |:----|:---:|:---:|:---:|:---:|:---:|:---:|
    |SOTA|88.6|91.0|65.7|48.3|52.5|37.5|
    |DST (FlexMatch)|**90.6**|**95.9**|**71.2**|**49.8**|**56.2**|**44.5**|

    |    |CUB|Flowers|Pets|Cars|Food|
    |:----|:---:|:---:|:---:|:---:|:---:|
    |SOTA|58.9|95.6|88.3|60.5|53.5|
    |DST (FlexMatch)|**70.5**|**95.8**|**90.4**|**72.7**|**57.1**|


 **Q3:** DivideMix and MixMatch should be cited/discussed relative to the current work. Wouldn't "DivideDST" be even more effective?

- MixMatch is mentioned in our original version. 
    - In the $\underline{\text{Related Work}}$ part, we have discussed MixMatch as a method for generating higher-quality pseudo labels. 
    - In the $\underline{\text{Experiments}}$ part, we have provided the results of MixMatch ($\underline{\text{Table 1}}$).

- DivideMix is included in our revised draft.
    - DivideMix proposes to simultaneously train two networks and performs label co-refinement and label co-guessing with both networks to alleviate confirmation bias. Due to the limited space, we classify DivideMix as a method that **adopts and improves** **mutual learning** to improve tolerance for inaccurate pseudo labels in the $\underline{\text{Related Work}}$ part as follows. 

        "Co-training, MMT, **DivideMix** and Multi-head Tri-training introduce multiple models or classifier heads and learn in an online mutual-teaching manner."

    - We also add comparison results between DivideMix and DivideDST (DivideMix utilizing DST instead of MixMatch) in $\underline{\text{Table 4}}$. The results below show that DivideDST also yields consistent improvements against DivideMix. 

        |Pre-training|Supervised| Supervised |Unsupervised| Unsupervised |
        |:----|:---:|:---:|:---:|:---:|
        |Label Amount|400|1000|400|1000|
        |DivideMix|55.8|67.5|53.6|64.9|
        |DivideDST|**69.1**|**75.1**|**65.0**|**74.2**|

    - Implementation details. DivideMix is originally developed for learning with noisy labels. To apply DivideMix in the context of SSL, we first pre-train a model with the labeled dataset $\mathcal{L}$. Then we obtain noisy labels on the unlabeled dataset $\mathcal{U}$, which will be further divided into a clean subset $\mathcal{U_\text{clean}}$ and a noisy subset $\mathcal{U_\text{noisy}}$. The labeled and unlabeled datasets in DivideMix are $\mathcal{L}\cup \mathcal{U_\text{clean}}$ and $\mathcal{U_\text{noisy}}$, respectively.

**Q4:** The overall loss function and the weight of the adversarial loss term.

Thanks for your suggestions!
- We add the overall loss function in $\underline{\text{Section 4 (Equation 8)}}$. 
- The trade-off hyperparameter of the adversarial loss is fixed to $1.0$ in all the experiments. For simplicity, it is omitted in $\underline{\text{Equation 8}}$.

 We would like to sincerely thank Reviewer ej4w for providing insightful reviews and valuable comments. We have clarified the questions in the following response. All the updated parts in the revised draft are highlighted in blue font.

**Q1:** The related work section lacks a summary of relevant adversarial techniques in general and as applied or potentially applied to SSL.

To address your concern, we provide a discussion of relevant adversarial techniques of SSL in the $\underline{\text{Related Work}}$ part.

"Inspired by Generative Adversarial Networks (GANs) [1], some works introduce adversarial training into semi-supervised learning.  A line of works [2, 3, 4, 5] exploit fake samples from the generator by labeling them with a new “generated” class and forcing the discriminator to output class labels. Another line of works use adversarial training to construct adversarial samples [6], e.g., VAT [7] injects additive noise into input, VAdD [8] introduces adversarial Dropout layers and RAT [9] expands the noise in VAT into a set of input transformations. These methods aim to impose a local smoothness on the model and do not involve training with pseudo labels. In contrast, in our method, the goal of the adversarial training process is to estimate the worst case of pseudo labeling and avoid such cases."

[1] Generative adversarial nets. NeurIPS 2014

[2] Semi-supervised learning with generative adversarial networks. arXiv 2016

[3] Improved techniques for training gans. NeurIPS 2016 

[4] Good semi-supervised learning that requires a bad gan. NeurIPS 2017

[5] Adversarially learned inference. ICLR 2017

[6] Explaining and harnessing adversarial examples. ICLR 2015

[7] Virtual adversarial training: a regularization method for supervised and semi-supervised learning. PAMI 2018

[8] Adversarial dropout for supervised and semi-supervised learning. AAAI 2018

[9] Adversarial transformations for semi-supervised learning. AAAI 2020

**Q2:** Comparison with adversarial SSL methods.

We add comparisons with adversarial SSL methods including VAT, ALI, and RAT in $\underline{\text{Table 1}}$ (training from scratch) and $\underline{\text{Table 2}}$ (finetuning from pre-trained models). The table below reports the average performance of these methods under different settings. Results show that DST significantly outperforms existing adversarial SSL methods.

|Method|Train from Scratch|Supervised Pre-training|Unsupervised Pre-training|
|:----|:---:|:---:|:---:|
|VAT|23.0|55.4|51.3|
|ALI|22.7|51.9|49.1|
|RAT|34.2|58.3|56.1|
|FixMatch|75.4|59.3|55.6|
|DST (FixMatch)|**78.3**|**71.1**|**68.7**|



 We appreciate all three reviewers for their insightful and constructive comments. We have uploaded a revised draft to address all reviewers' comments.  The updated parts are highlighted in blue font. Below is a summary of the main changes:

- We add more baselines, including some adversarial methods, as suggested by reviewers R1 & R2.
- We add a summary of relevant adversarial techniques applied in SSL, as suggested by reviewer R1.
- We add the overall loss function to avoid misunderstanding. 

We hope our responses and revisions will address all reviewers' concerns!

 The authors propose debiased self-training (DST) as an approach to reduce both data (sampling) bias and self-training bias during SSL. The DST recipe consists of two main elements: 

1) Pseudo-labels are treated as a related task, in that they are used to train a dedicated pseudo-classifier and shared learned features, while the main task classifier is trained only on ground-truth data, and, 

2) The learned features are additionally adversarially optimized to minimize the performance of an additional "worst-case" classifer, which is trained to miminmize loss on supervised data, whilst maximizing the loss of predictions from the pseudo-classifier. 

Results with both cold-start and pre-trained features on several datasets demonstrate solid gains over SOTA SSL approaches. Originality: 7
Quality: 7 (6 before rebuttal response)
Clarity: 7
Significance: 7
- While multi-classifier co-training approaches are ubiquitous in SSL approaches, multi-task learning is a standard way to improve feature representations for related tasks, and adversarial feature optimization has been widely investigated, optimizing the task features for a pseudo-classifer and adversarially against a worst-case classifier while training the task classifier is a nice, highly effective, and to my knowledge novel idea.
- With that said, the related work section lacks a summary of relevant adversarial techniques in general and as applied or potentially applied to SSL (e.g. VAT, VAaD, ALI, etc.....). This is the most serious limitation of the paper currently, and I urge the authors to address this and fully and completely situate their work in the rebuttal (otherwise, I will not be able to reccommend that the paper be accepted).
- In addition, adversarial SSL techniques ideally should be included in some of the results comparisions, although the methods that I am aware of are outperformed by many of the techniques already being compared.

Post-rebuttal:
- Thank you to the authors who have addressed my remaining questions and concerns by expanding the related work section, and including additional results comparing to and combining with DivideMix. I have increased my score from weak accept to accept. Well done! See limitations section. - See S&W section.
- DivideMix and MixMatch should be cited/discussed relative to the current work. Wouldn't "DivideDST" (DivideMix utilizing DST instead of MixMatch) be even more effective?
- The overall loss function and the weight given the adversarial term (7) are not stated in the paper (looking quickly at the code, it seems to be 1.0). Please confirm this, and update the paper accordingly.


 The paper analyzes the different types of bias in self-training, including data bias and training bias, and proposes to generate and utilize pseudo labels decoupled by two parameter-independent classifier heads to avoid error accumulation. Further, the worst case of self-training bias is estimated and used to guide the generation of pseudo labels. The proposed method is compared to several recent semi-supervised learning methods on a variety of datasets, and shown to better on most of the datasets. + The experimental results are competitive compared to the other existing semi-supervised learning methods.

+ The proposed model is simple yet effective. The proposed idea of reducing the training and data bias is interesting.

- In terms of model performance, the proposed method is not the best on all the datasets. 

- While the comparison is extensive, there are a few representative SSL methods missing in the comparison such as UDA [a] that uses the different unsupervised learning for semi-supervised learning. 

- Additional ablation study on requirement of label size is missing. For instance, when varying the label size, how does the model performs? Is it sensitive to the label size as compared to the other existing approach? 

[a] Unsupervised Data Augmentation for Consistency Training 1. What is the parameter size and the computation cost of the proposed method compared to the other existing methods?

2. While the method is similar to the "mutual learning", except sharing the backbone of the two models, what could be the additional benefits upon "mutual learning"? Please consider to add some discussion. 

3. What are the difference of implementation between the proposed method and the FixMatch, such as data augmentation and training schedules? Does the proposed method require certain unique training setups that differs from other SSL methods? 

4. In terms of model performance, the proposed method is not the best on all the datasets. What could be the possible reasons that lead to its weaker performance on certain datasets?  - In terms of model performance, the proposed method is not the best on all the datasets. 

- While the comparison is extensive, there are a few representative SSL methods missing in the comparison such as UDA [a] that uses the different unsupervised learning for semi-supervised learning. 

- Additional ablation study on requirement of label size is missing. For instance, when varying the label size, how does the model performs? Is it sensitive to the label size as compared to the other existing approach? 

[a] Unsupervised Data Augmentation for Consistency Training

 This paper suggests a new self-training approach called DST for semi-supervised learning. Usual self-training approaches generate incorrect pseudo-labelings and thus suffer from training instability due to the accumulated error. Specifically, based on data analysis, DST points out that there are two kinds of biases, namely training bias accumulated due to self-training strategy and data bias as a result of randomly sampled scarce samples. To address them, DST proposes two-phase solutions. First, to reduce training bias, DST adds a classification head to disentangle generation and utilization of pseudo labels. A default head is used for classification and pseudo-labeling, while another head is used to receive the pseudo-labels and perform training on unlabeled samples. Then, to reduce data bias, DST adversarially optimizes the worst training bias which is an indirect proxy of training bias. ### Strengths

1. The motivation and the design of DST are well-connected to each other.
2. Figures help the understanding of the paper.
3. Empirical results support the effectiveness of the paper.

### Weaknesses

1. Some part of the paper is misleading or not clearly explained, is the  'worst head' $h^{'}$ different from $h$ or $h_{pseudo}$? And how to draw the 'worst possible head' $h^{'}$ during training in a hands-on manner? Does it require additional heavy computations?
2. Two aspects make me concerned about the suggested method: (i) when the pseudo-labeling is incorrect, the feature generator $\psi$ is still contaminated by the given signal. (ii) On the other hand, for the correct pseudo-labels, the pseudo-labeling head cannot gain their benefit to improve pseudo-labeling quality.
3. The paper is relying on an assumption for mistakes of $h^{'}$ that does not always hold. The discrepancy with the current pseudo-labeling function also happens when the updated pseudo-labeling head $h^{'}$ is perfect while the current pseudo-labeling $h$ is imperfect. In this case, the effect of model learning would result in an undesirable way to induce the model to learn a worse hyperplane. See the weakness above. There is no severe limitation found in the paper so far.

Meta-review: This paper proposed a novel Debiased self-training (DST) approach to reduce both data bias and self-training bias during SSL. The proposed method is simple and empirically seems quite effective. Reviewers are generally positive about the novelty of the method and significance of the results. While authors have tried to address and improve some of the review issues, the related work section and empirical comparison with recent SOTA SSL methods still could be improved. For example, some recent related SSL works are still missing, including but not limited to
- DASO: Distribution-Aware Semantics-Oriented Pseudo-label for Imbalanced Semi-Supervised Learning, CVPR2022
- CoMatch: Semi-supervised Learning with Contrastive Graph Regularization, ICCV2021

Overall, the paper presents a novel framework for SSL, the empirical results of the method are quite positive, the paper can be accepted, but the authors are recommended to further improve the discussion/comparison of recent related work. 


Decision: Accept