On Non-Random Missing Labels in Semi-Supervised Learning
Authors: Xinting Hu
Yulei Niu
Chunyan Miao
Xian-Sheng Hua
Hanwang Zhang
Venue: ICLR 2022 Poster (2022)
Keywords: Semi-Supervised Learning
Missing Not At Random
Image Classification

Abstract: Semi-Supervised Learning (SSL) is fundamentally a missing label problem, in which the label Missing Not At Random (MNAR) problem is more realistic and challenging, compared to the widely-adopted yet naive Missing Completely At Random assumption where both labeled and unlabeled data share the same class distribution. Different from existing SSL solutions that overlook the role of  ''class'' in causing the non-randomness, e.g., users are more likely to label popular classes, we explicitly incorporate ''class'' into SSL. Our method is three-fold: 1) We propose Class-Aware Propensity (CAP) that exploits the unlabeled data to train an improved classifier using the biased labeled data. 2) To encourage rare class training, whose model is low-recall but high-precision that discards too many pseudo-labeled data, we propose Class-Aware Imputation (CAI) that dynamically decreases (or increases) the pseudo-label assignment threshold for rare (or frequent) classes. 3) Overall, we integrate CAP and CAI into a Class-Aware Doubly Robust (CADR) estimator for training an unbiased SSL model. Under various MNAR settings and ablations, our method not only significantly outperforms existing baselines, but also surpasses other label bias removal SSL methods.


Published as a conference paper at ICLR 2022
ON NON-RANDON MISSING LABELS IN LEARNING
Xinting Hu'
Yulei Niu Chunyan Miao'
Xian-Sheng Hua²
Hanwang Zhang'
Nanyyng Technological University, DDamo Academy, Alibaba Group
xint kinting001@er @e. .ot.tuuudu edu sg, yn. yp..uuling www.yylllnnggaaaii.... com {ascymiac, kanwangzhang]eetuu edu. add..s. sg, xiansheng www.alibaba--ieee....m
ABSTRACT
Semi-Supervised Learning (SSL) is fundamentally a missing label problem, which the label Missing Not At Random (MNAR) problem is more realistic and challenging, compared to the widely-adopted yet naive Missing Completely At Random assumption where both labeled and unlabeled data share the same class distribution. Different from existing SSL solutions that overlook the role of "class" in causing the non-randomness, e.g., users are more likely label pop- ular classes, we explicitly incorporate "class" into SSL. Our method is three-fold: We propose Awass-Aware Propensity (CAP) score that exploits the unlabeled data to train an improved classifier using the biased labeled data. 2) To encourage rare class training. whose model low-recall but high-precision that discards too many pseudo-labeled data, we propose Class-Awar Imputation (CAI) that dy- namically decreases (or increases) the pseudo-label assignment threshold for rare (or frequent) classes. 3) Overall, we integrate CAP and CAI into Class-Aware Doubly Robust (CADR) estimator for training an unbiased SSL model. Under var- ious MNAR settings and ablations, our method not only significantly outperforms existing baselines, but also surpasses other label bias removal SSL methods.
INTRODUCTION
Semi-supervised learning (SSL) aims to alleviate the strong demand for large-scale labeled data by leveraging unlabeled data (Zhu 2008 Yang et al. (021). Prevailing SSL methods first train a model using the labeled data, then uses the model to impute the missing labels with the predicted pseudo- labels for the unlabeled data (Van Buuren 2018). and finally combine the true- and pseudo-labels further improve the model (Sohn et al. 2020 Berthelot et al. 2019a). Ideally, the "improve" is theoretically guaranteed if the missing label imputation is perfect (Grandvalet & Bengio 2005); otherwise, imperfect imputation causes the well-known confirmation bias (Arazo et al., 2019 Sohn et al.] 2020). In particular, the bias is even more severe in practice because the underlying assumption that the labeled and unlabeled data are drawn from the same distribution does not hold. We term this scenario label Missing Not At Random (MNAR) (Hernán & Robins 2020). as compared to the naive assumption called label Missing Completely At Random (MCAR).
MNAR is inevitable in real-world SSL due to the limited label annotation budget (Rosset et al. 2005 uniform label annotations that keep MCAR are expensive. For example, we usually de- play low-cost data collecting methods like crawling social media images from the Web (Mahajan) et al. 2018). However, the high-quality labels appear to be severely imbalanced over classes due to the imbalanced human preferences for the "class" (Misra al. 2016: Colléony et al. 2017). For example, people are more willing to tag a "Chihuahua" rather than a "leatherback" (a huge black turtle) since the former is more lovely and easier to recognize. Thus, the imputation model trained by such biased supervision makes the model favoring "Chihuahua" rather than "leatherback". The "Chihuahua" favooed pseudo-labels, turn, negatively reinforce the model confidence in the false "Chihuahua" belief and thus exacerbate the confirmation bias.
We design an experiment on CIFAR-10 further illustrate how existing state-of-the-ar SSL meth- ods, e.g. FixMatch (Sohn et al. 2020), fail in the MNAR scenario. As shown in Figure aa), the
Now in Columbia University

---

Published as a conference paper at ICLR 2022
200
Contokeee P. 25
Clanstrdee
Clss
(a)
(b)
(c)
Figure/ 1: Visualization of an MNAR example and its experimental results on CIFAR-10, (a) Class distribution of the labeled and unlabeled training data. (b) Test accuracy of the supervised model using FixMatch and our CAP (Section 4.1) (c) The distribution of FixMatch's 's confidence scores on unlabeled data. Samples with confidence larger than fixed threshold (0.95, yellow line) are imputed. The corresponding box-plots display the (minimum, first quartile, median, third quartile, maximum) summary. The confidence-axis is not equidistant scaling for better visualization.
overall training data is uniformly distributed over classes, but the labeled data is long-tailed dis- tributed, which simulates the imbalanced class popularity. Trained on such MNAR training data, FixMatch even magnifies the bias towards the popular classes and ignores the rare classes (Fig- ure [b). We point out that the devil is in the imputation. As shown in Figure cc), the confidence scores of FixMatch for popular classes are much higher than those for rare classes (i.e., averagely >0.95 vs. <0.05). Since FixMatch adopts a fixed threshold for the imputation samples selection, Le., only the samples with confidence larger than 0.95 are imputed, FixMatch tends to impute the labels of more samples from the popular classes, which still keeps the updated labels with the newly added pseudo-labels long-tailed. As result, Fix.atcc is easily trapped in MNAR
In fact, MNAR has attracted broad attention in statistics and Inverse Propensity Weighting (IPW) is one of the commonly used tools to tackle this challenge (Seaman & White 2013 Dones al.) 2006
IPW introduces a weight for each training sample via its propensity score, which reflects how likely the label is observed (e.g., its popularity). In this way, IPW makes up sveudo-balanced dataset by duplicating each labeled data inversely proportional to its propensity- popular samples should draw the same attention the popular ones-a more balanced imputation. To combine the IPW true labels and imputed missing labels, Doubly Robust (DR) estimator is used to guarantee a robust SSL model if either IPW or imputation unbiased (Seaman & Vansteelandl, 2018 Vansteelandt
et al. 2010).
However, the above framework overlooks the role of "class" in causing the MNAR problem of SSL (Section 3 and we therefore propose a unified Class-Aware Doubly Robust (CADR) framework to address this challenge (Section D. CADR removes the bias from both the supervised model training and the unlabeled data imputation ends. For the former end, we propose Class-Aware Propensity (CAP) that exploits the unlabeled data to train an improved classifier using the biased labeled data (Section 4.1). We use efficient moving averaging strategy to estimate the propensity. For the latter end, we design Class Aware Imputation (CAI) that dynamically adjusts the pseudo- label assignment threshold with class-specift confidence scores to mitigate the bias in imputation (Section[4.2). Intuitively. CAI lowers the threshold requirement of rare class samples to balance with frequent class samples. Experiments on several image classification benchmarks demonstrate that our CADR framework achieves a consistent performance boost and can effectively tackle the MNAR problem. Besides, MCAR is special case of MNAR, method maintains the competitive performance in the conventional SSL setting.
The key contributions of this work are summarized as follows:
We propose a realistic and challenging label Missing Not At Random (MNAR) problem for semi- supervised learning, which is not extensively studied in previous work. We systematically analyze the bias caused by non-random missing labels.
We proposes a unified doubly robust framework called Class-Aware Doubly Robust (CADR) esti- mator to remove the bias from both the supervised model training end-by using Class-Aware Prospesity (CAP), and the unlabeled data imputation end-by using Class-Aware Imputation (CAI).
Our proposed CADR achieves competitive performances in both MNAR and conventional label Missing Completely At Random (MCAR).
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORKS
Missing data problems are ubiquitous across the analysis in social, behavioral, and medical sciences (Enders) 2010, Heckman 1977). When data is not missing at random, estimations based on the observed data only results in bias (Heckman) 1977). The solutions are fhree-fold: 1) Inverse probability weighting (IPW) assigns weights to each observed datum based on the propensity score, i.e., the probability of being observed. A missing mechanism- -based model is used to estimate the sample propensity. such as a logistic regression model (Rosset et al., [005]
Missing Not At Random.
Wang et al.; 1997) or a robit regression model (Kang & Schafer 2007; Liu 2005). 2) Imputation methods aim to fill in the missing values to produce a complete dataset (Kenward & Carpenter 2007] Little & Rubin 2014). An imputation model is regressed to predict the incomplete data from the observed data in deterministic (Kovar& & Whitridge. 1995) or stochastic (Lo et al. 2019) way. 3) As the propensity estimation or the regressed imputation are easy be biased, doubly robust estimators propose to integrate IPW and imputation with double robustness (Seaman & Vansteelandi 2018 Vansteelandt et al., 2010; Jonsson Funk et al., 20IT): the capability to maintain unbiased when either the propensity or imputation model is biased. In this work, as the image labels are missing not at random, we design the class-aware propensity and imputation considering the causal role of label classes in missing, avoiding the uncertain model specification and generating unbiased estimation. These two class- modules can be naturally integrated into a doubly robust estimator.
Semi-Supervised Learning (SSL). It aims to exploit unlabeled data to improve the model learned on labeled data. Prevailing SSL methods (Sohn et al. 2020 Berthelot et al. 2019b a) share a similar strategy: training a model with the labeled data and generating pseudo-labels for unlabeled data based on the model predictions. Pseudo-labeling methods (Lee) 2013 Xie et al. 2020 Rizve et al.) 2027) predict pseudo-labels for unlabeled data and add them to the training data for re-training. Consistency-reguuarizzi
methods (Sajjadi et [206)) apply random perturbation an unlabeled image and then use the prediction as the pseudo-label of the same image under a different augmentation. Recent state-of-the-a art SSL methods (Sohn et al., 2020 Berthelot et al., 2019a) combine the two existing techniques and predict improved pseudo- labels. However, when labels are missing at random, these methods can be inefficient as the model learned on labeled data is biased and significantly harms the quality of pseudo labels. Though some works also notice the model bias (Wei et al. [202]] Kim et al. 2020), they neglect the causal relationship between the bias and the missing label mechanism SSL, which systematically discussed in our work.
120166Laiig Laine& & Aila 2016 Berthelot et al. (019h)
MISSING LABELS IN SEMI-SUPERVISED LEARNING
In semi-supervised learning (SSL), the training dataset D is divided into two disjoint sets: a labeled dataset DL and an unlabeled data Dy. We denote Dr as ((x(), ( Nr, ) (usually Nı <N where a00) E IRd is a sample feature and f E {0, 1) is its one-hot label over C classes, and Dy as ((x()) M where the remaining N NN labels are missing in Dy. We further review SSL as a label missing problem, and define label missing indicator set M with m() E {0, 1}, where (+) =1 denotes the label is missing (i.e., unlabeled) and ( =0 denotes the label is not missing (i.e., labeled). In this way, we can rewrite the dataset as D (X, Y, -M= ())), (0))
Traditional SSL methods assume that M is independent with Y. i.e., the labels are Missing Com- pletely At Random (MCAR). Under this assumption, prevailing SSL algorithms can be summarized as the following multi-task optimization of the supervised learning and the unlabeled imputation:
i
arg min
Σ (x + Σ (z:6),
(Z.))EDL
ZEDe
where C, and La are the loss function designed separately for supervised learning on labeled data (M = 0) and regression imputation on unlabeled data =1) In general, SSL methods first train
model with parameters 0 using L. Dr, then impute the missing labels (an Buuren 2018) by predicting the pseudo-labels ğ f(æ; 0), and finally combine the true- and pseudo-labes to further improve the model for another SSL loop, e.g., another optimization iteration of Eq. @I. The implementation of L. and f., are open. For example, L. is normally the standard cross-entropy
3

---

Published as a conference paper at ICLR 2022
loss, and Ly can be implemented squared L2 loss (Berthelot et al. 2019b) loss (Berthelot et al. 2019a Sohn et al. (020).
cross-entropy
As you may expect, the key to stable SSL methods is the unbiasedness of the imputation model. When M is independent with Y. we have P(Y - 2, M 0) = P(Y|X =)), and thus
Elgl = Elyld]
Σ y. P(p()))) = Σ y P(yle, M 00)
Dr.
(x.D)ED
ΣV y P
(xN))D
which indicates that the model is idcally unbiased estimator, and the imputed labels unbiased. However, such MCAR assumption is too strong and impractical. For example, annotators may tend tag specific group of samples due to their preferences experiences. This motivates to ask the following realistic but overlooked question: what if M is dependent with Y, i.e. the labels are missing random (MNAR)? In this case, we have P(Y I, M 0) P(Y = x). and thus
E y. p(yl)) P(M 0|2, v) (a.y)ED
E y P(yle, M - (x.D)ED
P(M 0)z)
(3)
*
Σ y. P(yla) = Ely]-
(TN))D
We can see is no longer unbiased, further leading to biased imputed labels. As result, the bias will be increasingly enhanced with the joint optimization loop in Eq. (T) moving forward. This analysis indicates that previous SSL methods are not applicable for the MNAR problem as they overlook the role of "class" Y in causing the non-randomness of M. In the following, we will introduce our Class-Aware Doubly Robust framework that targets at unbiased estimator.
CLASS-AWARD DOUBLY ROBUST FRAMEWORK
discussed above, both the supervised learning and regression imputation previous SSL meth- ods suffer from the biased label estimation for MNAR. In this section, we proposed a doubly robust estimator that explicitly incorporates "class" into SSL and mitigates the bias from both sides.
4.1 CLASS-AWARD PROPENSITY FOR LABELED DATA
Traditional SSL methods estimate the model parameters via maximum likelihood estimation on the labeled data:
i = arg max log P(Y|X, M = 0) = arg max
Σ
log P(yla;
(4)
(z.
Since i estimated over Dr rather than the entire D, the difference between Dj and determines the unbiasedness Ô. As analyzed in Section 3 is an unbiased estimator the labels Y and their missing indicators M are independent, i.e., MCAR. However, when M is dependent with Y, i.e., MNAR, we have P(y|æ; M = 0) + P(y[z). In this case, 0 is no longer unbiased.
M
Ux
Uy
Un
Figure 2: Structural Causal
Model for MNAR. Ux. Uy and UM denote the exogenous variables.
Inspired by causal inference (Pearl et al., 2016), we establish a structural causal model for MNAR to analyze the causal relations between X, Y and M shown in Figure 2 First, X + Y denotes that each image determines own label. Second, Y + M denotes that the missing of label is dependent with its category. For exam- ple, popular pets like cat and dog attracts people to tag more often. We observe a chain structure of X + Y - M. which obtains important conditional independence rule (Pearl et al. 2016): Variables and B are conditionally independent given C, if there only one unidirectional path between A and B, and C. is any set of variables intercepting the path. Based on this rule, have

---

Published as a conference paper at ICLR 2022
P = P(X when conditioning on Y, i.e., the path between X and M is blocked by Y. In this way, we can obtain the unbiased 0 on the labeled subset of data by maximizing ((X
= arg max log P(XY;:)) = arg max (R.))D
log P(xly;
(5)
9
9
P(y n. P();
arg max
Σ
log
(6)
0
Dr
P 0)
P(y P(yl,; x)
arg max
log
(7)
0
(D
P 0)
log P(yla; 0) log P(y: 0)
arg max 9
log P(y|2; 0)
(8)
log P(ylæ;
arg max
(w.N)ED, log P(yl:; 0)
(9)
s(z, y)
0
Eq. (5) to Eq. (6) holds by Bayes' rule: Eq. 66 to Eq. holds because is sampled from an
empiniall distribution and p(z;0) is thus constant. Eq. to Eq. shows the connection with
Inverse Probability Weighting, where we estimate the propensity score s(x, y) for image as
log ((yl==0)- log "[ga)) Comparing the training objectives in Eq. ( with Eq. (4), our estimation
log P(y]a::)
can be understood as adjusting the original P(y| 0) by a class-aware prior P(y: 8) for each labeled
data (2c, y).
To estimate P(y; 0), ie., the marginal distribution P(Y = y) with the current parameters 0, a straightforward solution is go through the whole dataset and calculate the mean probability of all the data. However, during the training stage, 0 is updated step by step. and it is impractical to go thorough the whole dataset in each iteration considering the memory and computational cost. An alternative solution is to estimate the propensity within a mini-batch. Although the cost is saved, it faces risk of high- variance estimation.
To resolve the dilemma, we propose to use a moving averaging strategy over all the mini- batches. Specifically, we keep a buffer P(Y)) to estimate P(Y; 6.) and continuously update it by P(Y; B,, 6.), which is estimated with the current mini-batch of B, samples and parameters 0 at 1-th iteration:
P(Y) + (1-xP(()) + + - (1-)))(( 0,),
(10)
where μ E [0, 1) is momentum coefficient.
4.2 CLASS-AWARE IMPUTATION FOR UNLABELED DATA
As discussed in Section I the imputation model in traditional SSL is biased towards popular classes, and are the imputed labels. Although our proposed CAP mitigate the preference bias dur- ing the supervised learning stage, the model can still tend to impute popular class that achieves a high confidence score. To further alleviate the bias in the imputation stage, we propose remove the inferior imputed labels from Y. Traditional models like FixMatch adopt fixed threshold for imputation samples selection, which aims to reserve the accurate imputed labels and discard the noisy ones. However, as shown in Figure (c), for specific classes, an extremely small number of imputed labels can meet the requirement while most are discarded. In fact, as observed in previous works (Wei et al. 202T), these discarded predictions still hold nearly perfect precision. Based on these observations, we believe that fixed threshold is too coarse to impute missing labels.
To tackle this challenge, we propose a Class-Aware Imputation (CAT) strategy that dynamically adjusts the pseudo-label assignment threshold for different classes. Let Cz denote the potential imputed label for image x, i.e., C2= = arg max,, P(y|æ; 0). We use a class-aware threshold r(x) for image z
T(x) To P(C,))
P(y) )8
(11)
yE VE(I,...oy CC
where T, is the conventional threshold, 3 is haper-parameter, and P(y) is the class-aware propen- sity for class estimated by CAP. Intuitively, our class-aware threshold would set a higher require- ment for popular classes and lower requirement for rare classes, allowing more samples from rare
5

---

Published as a conference paper at ICLR 2022
classes be imputed. With the gradual removal of bias from the training process, the performance gap between classes also shrinks, and both popular and rare classes can be fairly treated. Further- more, when labels are missing completely at random (MCAR), our CIA can be degraded to the conventional imputation like previous works, P(Y) is uniform over all classes.
4.3 CLASS-AWARE DOUBLY ROBUST ESTIMATOR
Note that CAP and CAI mitigate the bias in supervised learning and imputation regression. Thanks to the theory of double robustness (Seaman & Vansteelandi 2018 Vansteelandt et al. 2010). we incorporate CAP and CAI into Class-Aware Doubly Robust (CADR) estimator. Theoretically, the CADR combination has a lower tail bound than applying each the components alone (Wang al. 2019). Besides, provides our learning system with double robustness: the capability to remain unbiased either CAP or CAT is unbiased.
Following the formulation of DR estimator, we first rewrite the training objective of CAP and CAI in semi-supervised learning
LCAP
1
Σ
(1- ml m0) Cx, (g)
(12)
N
pl)
im 1,
.N
LEAL
1
Σ E=
(o (00 ()con(o()) )(( + (1-m (0))
(13)
N
where m is the missing state, p(9) is the propensity score, (0) is the imputed label with confidence con(g()) ()), ), and I(.) is the indicator function. As introduced S.1tion we estimate the propensity
score p() as (1) ()) in CAP. Then the optimization CADR estimator is implemented as
ŌAADR min LEADR = arg min LCAP + Less + Laup:
(14)
where
Cupp
1 N
Σ
(1 ( (m()
)L( (0)) ((con()()))))))
(15)
pl))
i= 1= N
Σ (1 (1-
m() (1-m()) Lo. ,'0,,
which is a supplementary loss to guarantee the unbiasedness. In this design, LCAr Supp is expected to be O given correct CAP, and LCAP + foupp is expected to be O given correct CAI. These results guarantee the double robustness case that either the propensity or imputation is inaccurate.
EXPERIMENTS
5.1 EXPERIMENTAL SETUP
Datasets. We evaluate our method on four image classification benchmark datasets: CIFAR-IO
CIFAR- 100 (Krizhevsky) 2012). STL-10 (Coates et al.) 2011) and mini-ImageNet (Vinyals et al. composed of 60,000 images of size 32 32 from 10 (100) elasses and each class has 5,000 (500) training images and 1,000 (100) samples for evaluation. STL-10 10 dataset has 5,000 labeled images and 100,000 unlabeled images of size 64 64. mini Mmaie-e is a subset of ImageNet (Deng et al. 2009). It contains 100 classes where each class has 600 images of size 84 x 84. Follows previous SSL works (Hu al. 2021 Iscen et al. 2019), we select 500 images from each class for training and 100 images per class for testing.
2016). CIFAR-
MNAR Settings. Since the training data class-balanced in all datasets, we randomly select a clas--imbalanced subset as the labeled data to mimic the label missing not at random (MNAR). For a dataset containing C classes, the number of labeled data in each class N, are calculated as
N; Nmar 7 أ- N1 Nmaa is the maximum number of labeled data among all the classes, and y describes the imbalance ratio. ? = when the labeled data is balanced over classes, and larger Y indicates more imbalanced class distribution. Figure aaa) shows example of the data distribution when y = Nmer = 50 in CIFAR- 10. We also consider other MNAR settings where the unlabeled data is also imbalanced, i.e., the imbalance ratio of unlabeled data 7n. does not equal to 1.

---

Published as a conference paper at ICLR 2022
To ensure the number of unlabeled data is much larger than the labeled, we set the least number of labeled data over all classes as 1, and the largest number of unlabeled data to its original size.
Training Details. Following previous works (Berthelot et al. 2019b] Sohn et al. 2020 Hu et al. 202T). we used Wide ResNet (WRN)-28-2 for CIFAR-IO, WRN-28-8 for CIFAR- 100, WRN-37-2 for STL-10 and ResNet-18 for mini-Imagents
Since methods implemented as plug-in module to FixMatch, common network syyper-parameten. e.g., learning rates, batch-sizes, are the same as their original settings (Sohn al., 2020). For each dataset, our model and FixMatch are trained 2d7 iterations in MNAR and 2nd steps in ordinary SSL cases (7 = 1).
CIFAR-10
CIFAR-100
STL-10
mini- ImageNet
Method
y=
20
50
100
50
100
200
50
100
50
100
II Model
21.59
27.54
30.39
24.95
29.93
33.91
31.89
34.69
11.77
15.30
MixMatch
26.63
31.28
28.02
37.82
41.32
42.92
28.98
28.31
13.12
18.30
ReMixMatch
41.84
38.44
38.20
42.45
39.71
39.22
41.33
39.55
22.64
23,50
FixMatch
56.26
65.61
72.28
50.51
48.82
50.62
47.22
57.01
23.56
26.57
CREST
51.10
55.40
63.60
40.30
46.30
49.60
+ DARP
63.14
70.44
74.74
38.87
40.49
44.15
39.66
39.72
+ CADR (Ours)
79.63
93.79
93.97
59.53
60.88
63.30
70.29 76.70
29.07
32.78
Table 1: A Comparison of mean accuracies (%). We alter the imbalance ratio y of labeled data and
leave the unlabeled data balanced (7x = 1). We keep Nmax so that the least number of labeled
data among all the classes is always 1.
5.2 RESULTS AND ANALYSES
Comparison with State-of-The-Arr Methods. To demonstrate the effectiveness of our method, compared it with multiple baselines using the same network architecture, including CREST (Wei) al. T02T and DARP (Kim et al. 2020) that aims handle unbalanced semi-supervised learning. All the methods are implemented based on their public codes. Table I shows the results on all the datasets with different levels of imbalanced label ratios. While CREST and DARP fail to correct the severely biased learning process, our method outperforms all the baselines by large margins across all the settings. The reason is that though CREST and DARP handle the unbalanced labeled data, they mainly focus on the case where the unlabeled data is equally unbalanced. As the labeled and the unlabeled data share the same class distribution, the pseudo-label, they used are not as misleading as in our cases. Thus the bias removal is not critically emphasized by their algorithms. On CIFAR- 10 and CIFAR-100, we boost the mean accuracy of FixMatch by 24.41 % and 11.26% on average. On STL-10 and more challenging mini- mini-ImageNNet,
also substantial (21.38% and 5.86%). Specifically, Figure 3 shows the convergence trend and confusion matrices under one MNAR scenario (7- = Nmar 50). As one can observe, the performance of FixMatch is boosted mainly due to the de- biased estimation for the labeled-dataa
improvements
classes.
Individual Effectiveness of Each Component. Table shows the results of using Class-Aware Propensity (CAP) and Class-Aware Imputation (CAI) alone and together in trivial (Ccap + LCAI. w/o CADR) and our DR combination (CCAp + LCAI Luugp w/ CADR). We can observe that the improvement of using either CAP and CAI alone is distinguishable, demonstrating that both CAP and CAI achieve bias-removed estimation. Combining CAI and CAP, our CADR exhibits
Color Set
Texing
"raninn Rocutions (4)
Prodicted
Prodited
(b)
Figure 3: (a): the convergence trends of FixMatch and our method. (b) and (c): the confusion matrices of FixMatch and ours. Results are CIFAR-10 (y Nmar 50).
7

---

Published as a conference paper at ICLR 2022
CIFAR-10
CIFAR- 100
STL-10
mini-Image
Method
Y=
50
100
50
100
200
50
100
50
100
FixMatch
56.26
65.61
72.28
50.51
48.82
50.62
47.22
57.01
23.56
26.57
w/ CAP
79.38
89.50
93.95
55.72
58.53
63.07
69.97
77.69
28.54
32.23
w/ CAI
79.02
88.15
93.86
58.55
59.80
58.26
70.64
71.21
27.15
30.94
w/o CADR
79.43
89.53
94.10
57.93
59,50 62.78
70.17
75.44
25.54
31.66
w/ CADR
79.63
93.79
93.97
59.53
60.88
63.30
70.29
76.70
29.07
32.78
Table 2: The individual performance of our proposed Aware Propensity (CAP) and Class-
Aware Imputation (CAT) alone and together in trivial combination (w/o CADR) and CADR combi- nation (w/ CADR). We marked the best and second-best accuracies.
steady improvement over the baseline. We found that our CADR is consistently among the best or second-best performance, agnostic the value of y and the performance gap between individual components. When the label data is more imbalanced, e.g., 7 = 100, CAP outperfooss CAI by large margins on CIFAR-100 (63.07% vs. 58. 28...)). STI.- (77.69% vs. 71.21%) and mini-ImageNet (32.23% vs. 30.94%), and our CADR outperforms the trivial combination by 0.52% ~ 1.26%. The special case is 7 = 100 on CIFAR-10, where the trivial combination slightly beats CADR (93.97% vs. 94.10%), and the gap between CAP and CAI is also small (93. 93.966 vs. 93.86%). These results empirically verified our theoretical analysis and the robustness of our CADR, that is CADR maintains unbiasedness when either CAP CAI is unbiased. To sum up, CADR can provide reliable solution with robust performance and bring convenience to real applications.
Comparison with More Baselines. Apart from CREST and DARP, we further compare with some re-balancing techniques adopted in long-tailed fully- FixMaich
Method
Accuracy (%)
65.61
supervised learning in Table 3 They are 1) re- weighting (Cui et al. 2019): re-weighting labeled sam-
CREST
55.40 (-10.21)
+ DARP
70.44 (+4.83)
ple according to the inverse of the number of labeled
+ re-weighting
66.03 (+0.42)
data in each class, 2) re-sampling: re- sampling the la-
beled data to construct a balanced labeled set, and 3)
+ re-sampling
65.49 (-0.12)
LA [Menon et al. 2020): using logits adjusted soft-
LA loss (7=1)
60.22 (-5.39)
max cross-entropy loss that applies dabel-dependent
DASH
65.62 (+0.01)
offset to logits for each labeled sample. However, under
CAP (Ours)
89.50 (+23.89)
MNAR, where the model is biased by supervised learn-
CAI (Ours)
88.15 (+22.54)
ing and unlabeled data imputation together, only deal- ing with the supervised process without considering the whole data distribution gives inferior results. A recent work, DASH (Xu et al. (02T) also proposed to use dy- namic threshold in filtering imputed labels. Compared to their work, our threshold is both dynamic and class- aware, and our CADR outperforms DASH by large margin.
CADR (Ours)
93.79 (+28.18)
Table Comparison with multiple base- line methods. Experiments conducted on CIFAR-10 (7 Nmax = 50).
More MNAR Settings. In Figure we show more different MNAR scenarios. Figure 4 a) shows the case where the unlabeled data is imbalanced. %h denotes the imbalance ratio of unlabeled data,
100
100
+42.0
Accurasy
80
20
7+
(b)
Figure 4: More MNAR settings. Evaluation of average accuracies (Mean Accu) (%) / geometric mean accuracies (GM Accu) (%) using our method and FixMatch when a) varying imbalance ratio Tue of the unlabeled data; b) selecting a sub-set randomly labeled data. Experiments are conducted on (a) CIFAR- 10 with Nmax = 7 = 50, and (b) CIFAR- 100 with 5,000 labeled samples.
8

---

Published as a conference paper at ICLR 2022
and those unlabeled data is inversely imbalanced (the rarest class in labeled data is the most fre- quent in unlabeled data) when Yuu <1. Apart from the artificially designed imbalanced distribution, Figure 4b describes the results under the case where label random subset of data unaware of class distribution of the whole data. Without strictly selecting labeled data following the class distribution, the class distribution of labeled data can also be different from the unlabeled data. To show our effectiveness in producing unbiased predictions, we also report geometric mean accuracy (GM Accu) (Kubai 2000; Kim et al.) (020)) The overall results demonstrate that our method is robust to the distribution perturbation of both labeled and unlabeled data. Specifically, the signif- icantly improved GM Accuracy shows that the cliss-wise accuracy obtained through our unbiased estimation is both high and in good balance.
Compatibility with Conventional Settings. As discussed in Section g our method degrades to the original FixMatch implementation when labels are missing completely at random (MCAR). Table demonstrates the compatibility to MCAR of our methods: our results are consistent with the perfor- mance of FixMatch when labeled data are uniformly sampled from each class. Particularly, when provided with the extremely limited number of labeled data (CIFAR-IO, #labels = 40), our perfor- mance is better than FixMatch (94.41% to 91.96%). The reason is that compared to FixMatch, our method is more robust to the data variance among mini-batcher with the moving averaged propensity in such few-shot labeled data learning.
CIFAR-10
CIFAR-100
STL-10
Method
#label==40
250
4,000
400
2,500
10,000
1,000
FixMatch
88.61
94.93an33
95.69aa.5
50.05 a3.01
71.36
76.82ao.1
94.83 20.63
CADR (Ours)
94.41
94.35
95.59
52.90
70.61
76.93
95.35
Table 4: Comparisons of average accuracies with labels missing completely at random (MCAR).
Performances of Fixmatch are the reported results in their paper (Sohn al. 2020). #labels denotes the overall number of the labeled data.
Ablation Studies of Hyper- apper-parametrs. We conducted ablation studies on two hyper-parametert the moving average momentum for calculating class-aware propensity (Table and the threshold scaling coefficient 33 for class-aware imputation (Table G). Table 5 shows the effectiveness of the moving average strategy comparing results of different momentum values, "here"O" denotes no moving average. As shown in Table 6 when too small (0.2), the requirement of confidence for rare classes lowers marginally, and thus the selected imputed labels still bear bias. Specifically, using the distribution of the labeled data re-scale the threshold not promising, while our method dynamically updating the threshold with training is adaptive and effective.
/
0.999
0.99
0.9
B
0.2
0.5
1.0
label
Accuracy(%)
89.35
89.50
89.46 88.39
Accuracy(%)
65.79
88.15
88.11
71.86
Table 5: Evaluation with varying moving av- eraging coefficient/ j for CAP. Experiments conducted on CIFAR-10 (y Nmnax 50).
Table 6: Evaluation with different threshold co- efficient 3 for CAL. "label" means following the distribution of the labeled data. Experiments conducted on CIFAR-10 (? = Nmaar = 50).
6 CONCLUSION
In this work, we presented a principled approach to handle the non-random missing label problem in semi-supervised learning. First, we proposed Aware Propensity (CAP) to train an improved classifier using the biased labeled data. Our CAP exploits the class distribution information of the unlabeled data to achieve unbiased missing label imputation. Second, we proposed Class-Aware Im- putation (CAI) that dynamically adjusts the threshold in filtering pseudo-labed of different classes. Finally, we combined CAP and CAI into a doubly robust estimator (CADR) for the overall SSL model. Under various label missing not at random (MNAR) settings for several image classifica- tion benchmarks, we demonstrated that our method gains a significant and robust improvement over existing baselines. For future work, ory (Bareinboim et al., 2014) of the missing data mechanisms and 2) the causal direction between label and data (Kügelgen al. 2020), into semi-supervised learning.
interested in incorporating 1) the selection bias the-

---

Published as a conference paper at ICLR 2022
REFERENCES
Eric Arazo, D. Ortego, Paul Albert, Noel O'Connor, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. arXiv preprint arXivv:908.. 02983, 2019.
E. Bareinboim, J. Tian, and J. Pearl. Recovering from selection bias in causal and statistical infer- ence. In Proceedings of the 28th AAAI Conference on Artificial Intelligence, 2014.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Aug- mentation Anchoring. In International Conference on Learning Representations, 2019a.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Paperno, Avital Oliver, and Colin Raf- fel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Informa- fion Processing Systems, 2019b.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In International Conference Artificial Intelligence and Statistics, 2011.
Agathe Colléony, Susan Clayton, Denis Couvet, Michel Saint Jalme, and Anne-Caroline Prévot. Human preferences for species conservation: Animal charisma trumps endangered status. In Biological Conservation, 2017.
Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei Fei Li. ImageNet: a Large-Seale Hierarchical Image Database. In IEEE/CVF Conference Computer Vision and Pattern Recog- nition, 2009.
Craig K. Enders. Applied Missing Data Analysis. Guilford Press, 2010.
Yves Grandvalet and Yoshua Bengio. Semi-supervised Learning by Entropy Minimization. In Advances in Neural Information Processing Systems, 2005.
James Heckman. Sample Selection Bias As Specification Error (with Application to the Estima- tion of Labor Supply Functions). In National Bureau of Economic Research, Inc, NBER Working Papers, 1977.
MA Hermán and JM Robins. Causal inference: What Boca Raton: Chapman & Hall/CRC, 2020.
Zijian Hu, Zhengyu Yang, Xuefeng Hu, and Ram Nevatia. SimPLE: Similar Pseudo Label Ex- ploitation for Semi-Supervised Classification. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum. Label Propagation for Deep Semi- Supervised Learning. In IEEF/CVF Conference on Computer Vision and Pattera Recognition, 2019.
Andrew Jones, Xander Koolman, and Nigel Rice. Health-related non- non-response in the british house- hold panel survey and european community household panel: Using inverse- wrwpapility-weiieeeeeeeeee estimators in non-linear models. Journal of the Royal Statistical Society Series A, 2006.
Michele Jonsson Funk, Daniel Westreich, Chris Wiesen, Til Stürmer, M Brookhart, and Marie Da- vidian. Doubly robust estimation of causal effects. American journal of epidemiology, 2011.
Joseph D. Y. Kang and Joseph L. Schafer. Demystifying Double Robustness: A Comparison of Al- ternative Strategies for Estimating a Population Mean from Incomplete Data. Statistical Science, 2007.
Michael Kenward and James Carpenter. Multiple imputation: Current perspectives. Statistical
methods in medical research, 2007.
10

---

Published as a conference paper at ICLR 2022
Jachyung Kim, Youngbum Hur. Sejun Park, Eunho Yang, Sung Hwang, and Jinwoo Shin. Distribu- tion aligning refinery of pseudo-label for imbalanced supervupee learning. In Advances in Neural Information Processing Systems, 2020.
J.G. Kovar and P. Whitridge. Imputation of business survey data. Business Survey Methods. B.G.
Cox et al. (eds.), 1995.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Tech. Rep., University of Toronto, 2012.
M. Kubat. Addressing the curse of imbalanced training sets: One-sided selection. International Conference on Machine Learning, 2000.
Julius Kügelgen, Alexander Mey, Marco Loog, and Bernhard Schölkopf. Semi-supervisee learning, causality, and the conditional cluster assumption. In Conference on Uncertainty in Artificial Intelligence. pp. 1-10. PMLR, 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference Learning Representations, 2016.
Dong-Hyue Lee. Pseudo-label The simple and efficient semi-supervised learning method for deep
neural networks. (CML2013 Workshop: Challenges in Representation Learning (WREPL), 2013.
R.J.A. Little and Donald Rubin. Statistical analysis with missing data. New York: John Wiley & Sons, 2014.
Chuanhai Liu. Robit regression: A simple robust alternative to logistic and probit regression. DO1: /1000//0070094556002220 CH21,, 2005.
Andrew Lo, Kien Siah, and Chi Wong. Machine learning with statistical imputation for predicting drug approval. Harvard Data Science Review, 2019.
Dhruv Kumar Mahajan, Ross B. Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits of weakly supervised pretraining. In European Conference Computer Vision, 2018.
Aditya Menon, Sadeep Jayasumana, Ankit Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Loig-tail learning via logit adjustment. In International Conference on Learning Representations, 2020.
Ishan Misra, C. Lawrence Zitnick, Margaret Mitchell, and Ross Girshick. Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Judea Pearl, Madelyn Glymour, and Nicholas P Jewell. Causal inference in statistics: A primer. John Wiley & Sons, 2016.
Nayeem Rizve, Kevin Duarte, Yogesh Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty- aware pscudo-label selection framework for semi-supervised learning. In Inter- national Conference Learning Representations, 2021.
Saharon Rosset, Ji Zhu, Hui Zou, and Trevor Hastie. A method for inferring label sampling mecha- nisms in semi-supervised learning. In Advances in Neural Information Processing Systems, 2005.
Mchdi Sajjadi, Mchran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. In Advances in Neural Information Processing Systems, 2016.
Shaun R Seaman and Stijn Vansteelandt. Introduction to double robust methods for incomplete data. Statistical science: a review journal of the Institute of Mathematical Statistics, 2018.
Shaun R Seaman and lan R. White. Review of inverse probability weighting for dealing with missing data. Statistical Methods in Medical Research, 2013.

---

Published as a conference paper at ICLR 2022
Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying sumi-super learning with consistency and confidence. In Advances in Neural Information Processing Systems, 2020.
S. Van Buuren. Flexible imputation of missing data. Chapman & Hall/CRC Interdisciplinary Statis- lics, 2018.
S. Vansteelandt, J. Carpenter, and M. G. Kenward. Analysis of incomplete data using inverse prob- ability weighting and doubly robust estimators. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 2010.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Match- ing Networks for One Shot Learning. In Advances in Neural Information Processing Systems, 2016.
C. Wang. Suojin Wang, Lue-Ping Zhao, and Shyh-Tyan Ou. Weighted semiparametric estimation in regression analysis with missing covariate data. Journal of the American Statistical Association, 1997.
Xiaojie Wang. Rui Zhang, Yu Sun, and Jianzhong Qi. Doubly robust joint learning for recommen- dation data missing not at random. In Proceedings of the 36th International Conference Machine Learning. 2019.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc Le. Self-Training With Noisy Student Improves ImageNet Classification. IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Minh-Thang
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine Learning, 2021.
Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. A Survey on Deep Semi-supervised Learning. arXiv preprint, arXiv:2103.00550, 2021.
Xiaojin Zhu. Sepi-supervised learning literature survey. Comput Sci, University of Wisconsin- Madison, 2008
12


Reviews:
[{"review_id": "Z3VEwHexE6A", "reviewer": "Reviewer_9j41", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Attacking semi-supervised learning with the MNAR setting is an interesting and important topic for practical learning tasks. The empirical study is comprehensive and the proposed method provides a significant improvement compared to state-of-the-art semi-supervised learning approaches. \n\nWeakness: \nTwo SOTA baselines of semi-supervised learning, MixMatch (https://proceedings.neurips.cc/paper/2019/file/1cd138d0499a68f4bb72bee04bbec2d7-Paper.pdf) and ReMixMatch (https://openreview.net/pdf?id=HklkeR4KPB) are not included in the empirical study. ", "summary_of_the_paper": "This work proposes a semi-supervised learning algorithm with systematic label missing, aka Missing Not At Random. The proposed solution is composed of two steps. First, to learn the classifier with the labelled data instances, a learnable weight, named propensity score, is proposed and attached to each labelled data instance to deliver an unbiased estimate (see Eq.10). Second, a class-wise threshold is tuned according to the class-specific propensity score (see Eq.11). The results show better semi-supervised classification accuracy compared to other baselines in the MNAR scenario. ", "summary_of_the_review": "Please find our comments above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CMzyJvPoP6y", "reviewer": "Reviewer_Fenu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n1. The paper does a good job of laying out the problem assumptions and differentiating from random labeling scenarios.\n\n2. The proposed solution is clearly explained and justified.\n\n3. The experimental results are well presented and the comparison across multiple different datasets/scenarios does a good job building the case for the proposed solution.\n\n\nWeaknesses\n\n1. Given the paper proposes the MNAR setting, it would help strengthen the justification/interest to point to a real dataset/setting that matches this assumption as the experimental results all are based on simulated class-dependent labeling. This is the major weakness of the paper to me as it weakens the justification/interest in the proposed problem.\n\n2. The experimental results seem strong, however for all of these datasets the dependence of labeling on class is extremely high (or zero in the demonstration of consistency in the missing at random case). It would be interesting to see the performance comparisons with baselines for label dependence in the intermediate range. This is especially true given the synthetic nature of the missing labels making it hard to determine whether the proposed labeling dependence is reasonable/realistic.", "summary_of_the_paper": "This paper formulates the problem of semi-supervised learning where the likelihood of an example being labeled is dependent on the class of the example. Given this assumption, the authors propose an approach to mitigate the resulting distribution shifts that occur in both the labeled and unlabeled data sets. Finally, experimental results are shown for the class-dependent labeling scenario, with the proposed algorithm outperforming standard SSL approaches that assume random labeling probability.\n", "summary_of_the_review": "I am in favor of accepting this paper as the problem and solution are well written and the experimental results seem reasonable. The main weaknesses are the lack of an empirical example to justify the proposed setting and demonstration of the proposed approach on only simulated cases of MNAR data.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8fXZGl41OI0", "reviewer": "Reviewer_D3Mi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths:\n\nThe paper made a clear review of the underlying theory for the pseudo-label-based semi-supervised learning framework. The framework can lead to an unbiased estimator if the unlabeled/labeled data status is fully random with respect to the overall data distribution.\n\nCAP: The imbalanced availability of labeled data across categories is a practically important issue. The paper presents clear and convincing descriptions of this problem setup. It revised the framework in a theoretically decent way to incorporate the class-aware propensity. In other words, the paper proposed a probabilistically rigorous method to reweight each sample’s training loss adaptively. The implementation of this idea in SGD with mini-batches and running average is also reasonable. \n\nCAI: The paper recognized the fixed threshold of FixMatch as a cause of bias across different categories. It proposed a formulation to set an adaptive threshold for each class based on the previous class-aware propensity. Though not very theoretically founded, the method agrees well with intuition (similar to the “focal loss” for object detection). The formulation can also gracefully fall back to the conventional FixMatch.  \n\nUsing the theory of double robustness to combine CAI and CAP provides a theoretically better way to combine CAP and CAI than simply summing them together. \n\nThe experimental results in the synthetic class-imbalanced settings demonstrated the effectiveness of CAP and CAI. They outperformed FixMatch and other previous semi-supervised learning methods significantly. In the fully random settings for labeled data, the proposed methods also did not hurt the model’s accuracy; in fact, it improved the model’s performance a bit.  \n\n### Weakness:\n\nIn comparison to the trivial combination of CAP and CAI, CDAR did not always lead to stronger results (though CDAR showed significant improvement over the trivial combination in some cases). Is there any explanation for the slightly mixed performance of CDAR?\n\nThe paper can be stronger if it can show experimental results on a real-world dataset (i.e., the labeled data are not synthetically imbalanced but originally imbalanced). \n", "summary_of_the_paper": "The paper addresses the semi-supervised learning problem under the setting of imbalanced label data by extending FixMatch with a class-aware method to reweight the loss and an adaptive thresholding method to select examples with pseudo labels. The two methods are termed Class-Aware Propensity (CAP) and Class-Aware Imputation (CAI), respectively. The paper also proposes the Class-Aware Doubly Robust (CADR) estimator by combining the two methods in the double robustness framework. Experiments are performed on Cifar-10/100, STL-10, and mini-ImageNet. ", "summary_of_the_review": "The paper proposes a novel method to tackle the more practical problem of semi-supervised learning with imbalanced data. The experimental results are strong, and the ablation studies are informative. More discussions can be made about CDAR’s performance. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Addressed semi-supervised learning with the MNAR setting.  Well written paper.
Several additional experiments were reported in response to the reviewer questions.  
General agreement amongst reviewers.
Decision: Accept (Poster)