Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses the semi-supervised learning problem under the setting of imbalanced label data by extending FixMatch with a class-aware method to reweight the loss and an adaptive thresholding method to select examples with pseudo labels. The two methods are termed Class-Aware Propensity (CAP) and Class-Aware Imputation (CAI), respectively. The paper also proposes the Class-Aware Doubly Robust (CADR) estimator by combining the two methods in the double robustness framework. Experiments are performed on Cifar-10/100, STL-10, and mini-ImageNet. 
main_review: 
### Strengths:

The paper made a clear review of the underlying theory for the pseudo-label-based semi-supervised learning framework. The framework can lead to an unbiased estimator if the unlabeled/labeled data status is fully random with respect to the overall data distribution.

CAP: The imbalanced availability of labeled data across categories is a practically important issue. The paper presents clear and convincing descriptions of this problem setup. It revised the framework in a theoretically decent way to incorporate the class-aware propensity. In other words, the paper proposed a probabilistically rigorous method to reweight each sample’s training loss adaptively. The implementation of this idea in SGD with mini-batches and running average is also reasonable. 

CAI: The paper recognized the fixed threshold of FixMatch as a cause of bias across different categories. It proposed a formulation to set an adaptive threshold for each class based on the previous class-aware propensity. Though not very theoretically founded, the method agrees well with intuition (similar to the “focal loss” for object detection). The formulation can also gracefully fall back to the conventional FixMatch.  

Using the theory of double robustness to combine CAI and CAP provides a theoretically better way to combine CAP and CAI than simply summing them together. 

The experimental results in the synthetic class-imbalanced settings demonstrated the effectiveness of CAP and CAI. They outperformed FixMatch and other previous semi-supervised learning methods significantly. In the fully random settings for labeled data, the proposed methods also did not hurt the model’s accuracy; in fact, it improved the model’s performance a bit.  

### Weakness:

In comparison to the trivial combination of CAP and CAI, CDAR did not always lead to stronger results (though CDAR showed significant improvement over the trivial combination in some cases). Is there any explanation for the slightly mixed performance of CDAR?

The paper can be stronger if it can show experimental results on a real-world dataset (i.e., the labeled data are not synthetically imbalanced but originally imbalanced). 

summary_of_the_review: The paper proposes a novel method to tackle the more practical problem of semi-supervised learning with imbalanced data. The experimental results are strong, and the ablation studies are informative. More discussions can be made about CDAR’s performance.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper formulates the problem of semi-supervised learning where the likelihood of an example being labeled is dependent on the class of the example. Given this assumption, the authors propose an approach to mitigate the resulting distribution shifts that occur in both the labeled and unlabeled data sets. Finally, experimental results are shown for the class-dependent labeling scenario, with the proposed algorithm outperforming standard SSL approaches that assume random labeling probability.

main_review: Strengths

1. The paper does a good job of laying out the problem assumptions and differentiating from random labeling scenarios.

2. The proposed solution is clearly explained and justified.

3. The experimental results are well presented and the comparison across multiple different datasets/scenarios does a good job building the case for the proposed solution.


Weaknesses

1. Given the paper proposes the MNAR setting, it would help strengthen the justification/interest to point to a real dataset/setting that matches this assumption as the experimental results all are based on simulated class-dependent labeling. This is the major weakness of the paper to me as it weakens the justification/interest in the proposed problem.

2. The experimental results seem strong, however for all of these datasets the dependence of labeling on class is extremely high (or zero in the demonstration of consistency in the missing at random case). It would be interesting to see the performance comparisons with baselines for label dependence in the intermediate range. This is especially true given the synthetic nature of the missing labels making it hard to determine whether the proposed labeling dependence is reasonable/realistic.
summary_of_the_review: I am in favor of accepting this paper as the problem and solution are well written and the experimental results seem reasonable. The main weaknesses are the lack of an empirical example to justify the proposed setting and demonstration of the proposed approach on only simulated cases of MNAR data.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes a semi-supervised learning algorithm with systematic label missing, aka Missing Not At Random. The proposed solution is composed of two steps. First, to learn the classifier with the labelled data instances, a learnable weight, named propensity score, is proposed and attached to each labelled data instance to deliver an unbiased estimate (see Eq.10). Second, a class-wise threshold is tuned according to the class-specific propensity score (see Eq.11). The results show better semi-supervised classification accuracy compared to other baselines in the MNAR scenario. 
main_review: Attacking semi-supervised learning with the MNAR setting is an interesting and important topic for practical learning tasks. The empirical study is comprehensive and the proposed method provides a significant improvement compared to state-of-the-art semi-supervised learning approaches. 

Weakness: 
Two SOTA baselines of semi-supervised learning, MixMatch (https://proceedings.neurips.cc/paper/2019/file/1cd138d0499a68f4bb72bee04bbec2d7-Paper.pdf) and ReMixMatch (https://openreview.net/pdf?id=HklkeR4KPB) are not included in the empirical study. 
summary_of_the_review: Please find our comments above.
review_title: null

Meta-review: Addressed semi-supervised learning with the MNAR setting.  Well written paper.
Several additional experiments were reported in response to the reviewer questions.  
General agreement amongst reviewers.
Decision: Accept (Poster)