Towards Semi-Supervised Learning with Non-Random Missing Labels
Authors: Yue Duan
Zhen Zhao
Lei Qi
Luping Zhou
Lei Wang
Yinghuan Shi
Venue: Submitted to ICLR 2023 (2023)
Keywords: semi-supervised learning
label missing not at random
pseudo-rectifying guidance

Abstract: Semi-supervised learning (SSL) tackles the label missing problem by enabling the effective usage of unlabeled data. While existing SSL methods focus on the traditional setting, a practical and challenging scenario called label Missing Not At Random (MNAR) is usually ignored. In MNAR, the labeled and unlabeled data fall into different class distributions resulting in biased label imputation, which deteriorates the performance of SSL models. In this work, class transition tracking based Pseudo-Rectifying Guidance (PRG) is devised for MNAR. We explore the class-level guidance information obtained by the Markov random walk, which is modeled on a dynamically created graph built over the class tracking matrix. PRG unifies the history information of each class transition caused by the pseudo-rectifying procedure to activate the model's enthusiasm for neglected classes, so as the quality of pseudo-labels on both popular classes and rare classes in MNAR could be improved. We show the superior performance of PRG across a variety of the MNAR scenarios and the conventional SSL setting, outperforming the latest SSL solutions by a large margin. Checkpoints and evaluation code are available at the anonymous link https://anonymous.4open.science/r/PRG4SSL-MNAR-8DE2 while the source code will be available upon paper acceptance.

Under review as a conference paper at ICLR 2023
TOWARDS SEMI-SUPERVISER LEARNING WITH NON- RANDOM MISSING LABELS
Anonymous authors Paper under double-blind review
ABSTRACT
Semi-supervised learning (SSL) tackles the label missing problem by enabling the effective usage of unlabeled data. While existing SSL methods focus on the traditional setting, practical and challenging scenario called label Missing Not At Random (MNAR) is usually ignored. In MNAR, the labeled and unlabeled data fall into different class distributions resulting in biased label imputation, which deteriorates the performance of SSL models. In this work, class transition tracking based Eseudo-Rectifyee Guidance (PRG) is devised for MNAR. We explore the class-level guidance information obtained by the Markov random walk, which is modeled on dynamically created graph built over the class tracking matrix. PRG unifies the history information of each class transition caused by the pscudo-rectifynng procedure activate the model's ' enthusiasm for neglected classes, so as the quality of pseudo-labele on both popular classes and rare classes in MNAR could be improved. We show the superior performance of PRG across a variety of the MNAR scenarios, outperforming the latest SSL solutions by a large margin. Checkpoints and evaluation code are available at the anonymous link https:://// //as:://nnonnnn /nonymous. 4ppen. sciesccc/eee.cccc...c.... /tte////RGGGSSSS MNAR 8DE2 while the source code will be available upon paper acceptance.
INTRODUCTION
Semi-supervised learning (SSL), which is in the ascendant,
yields promising results in solving the shortage of large-scale
Uniioiee Catl
labeled data (Chapelle et al.) 2009 Zhou 2021 Van Engelen
& Hoos| 2020). Current prevailing SSL methods (Lee
200
100theee et al. 2020 Sohn et al. 2020 Tai et al. 2001 Zhang al. 2021) utilize the model trained on the labeled data to impute pseudo-labels for the unlabeled data, thereby boosting the model performance. Although these methods have made ex- citing advances in SSL., they only work well in the conventional setting, i.e., the labeled and unlabeled data fall into the same (bal- anced) class distribution. Once this setting is not guaranteed, the gap between the class distributions of the labeled and unlabeled data will lead to significant accuracy drop of the pseudo-labels, resulting in strong confirmation bias (Arazo et al. 2019) which ultimately corrupts the performance of SSL models. The work in Hu et al. 2022) originally terms the scenario of the labeled and unlabeled data belonging mismatched class distributions as label Missing Not At Random (MNAR) and proposes an unified doubly robust framework to train an unbiased SSL model in MNAR. It can be easily found that in MNAR, either the labeled or the unlabeled data has an imbalanced class distribution, otherwise, it degrades to the conventional SSL setting. A typical MNAR scenario is shown in Fig. [ in which the popular classes of labeled data cause the model to ignore the rare classes, increasingly magnifying the bias in label imputation on the unlabeled data. It is worth noting that although some recent SSL methods (Kim et al.) 2020 Wei et al., (222)) are proposed to deal with the class imbalance, they still built upon the assumption of the matched class distributions between the labeled and unlabeled data, and their performance inevitably declines in MNAR.
900
Indos
Figure 1:
An example of the MNAR scenarios CIFAR-10 (see Sec. [ for details). The class distribution of total data is bal- anced whereas labeled data is un- evenly distributed across classes. For better illustration, the y-axis has different scaling for labeled (blue) and unlabeled data (green).

---

Under review as a conference paper at ICLR 2023
erple BARzORk: hnd
Rii (%)
Ros (4)
Clos buies
Class
Taxiay (e
120
Taiikng henskun
325
sudtow Sid s seeee
(b)
(c)
Figure 2: Results of FixMatch (Sohn et al. 2020 in MNAR and the conventional setting, The
models are trained on CIFAR-I0 with WRN-28-2 backbone (Zagoruyko & Komodakis 2016). (a)
and (b): Class-wise pseudo-label error rate. (c): Confusion matrix of pseudo-labels. In (b) and (c), experiments are conducted with the setting of Fig. | whereas in (a) with the conventional setting (i.e. balanced labeled and unlabeled data). The label amount used in (a) is the same as that in (b) and (c).
MNAR is a more realistic scenario than the conventional SSL setting. In the practical labeling process, labeling all classes uniformly is usually not affordable because some classes are more difficult to recognize (Rosset et al., 2005 Misra et al., 2016 Colléony et al. 2017). Meanwhile, most automatic data collection methods also have difficulty in ensuring that the collected labeled data is balanced nutshell, MNAR is almost inevitable in SSL. In MNAR, the tricky troublemaker is the mismatched class distributions between the labeled and unlabeled data. Training under MNAR, the model increasingly favors some classes, seriously affecting the pseudo-rectifying procedure. Pseudo-rectifying is defined the change of the label assignment decision made by the SSL model for the same sample according to the knowledge learned at each new epoch. This process may cause class transition, i.e., given sample, its class prediction at the current epoch is different from that at the last epoch. In the self-training process of the SSL model driven by the labeled data, the model is expected to gradually rectify the pseudo-labels mispredicted for the unlabeled data in last epoches. With pseudo-rectifying. the model trapped in the learning of extremely noisy pseudo-labels will be rescued due its ability correct these labels.
(Mahajan et al., [0188] Hu et al.) 2022).
Unfortunately, the pseudo-rectifying ability of the SSL model could be severely perturbed in MNAR. Take the setting in Fig. for example. The model's "confidence" in predicting the pseudo-labels into the labeled rare classes is attenuated by over-learning the samples of the labeled popular classes. Thus, the model fails to rectify those pseudo-labels mispredicted as the popular classes to the correct rare classes (even if the class distribution is balanced in unlabeled data). As shown in Fig. 2b compared with FixMatch (Sohn al., (220)) trained the conventional setting (Fig. 2a). FixMatch trained in MNAR (Fig. D significantly deteriorates its pseuuo-rectifying ability. Even after many iterations, the error rates of the pseudo-labels predicted for labeled rare classes remain high. This phenomenon hints the necessity to provide additional guidance to the rectifying procedure to address MNAR. Mcanwhile, as observed in Fig. 2c we notice that the mispredicted pseudo-labels for each class are often concentrated in few classes, rather than scattered across all other classes. Intuitively, a class can easily be confused with the classes similar to it. For example, as shown in Fig. [c]] the "automobile" samples are massively mispredicted the most similar class: "truck". Inspired by this, from the class-level, i.e., pointing out the latent
we argue that it is feasible to guide pseudo-rectifyine direction of class transition based its current class prediction only. For instance, given sample classified as "truck", the model could be given chance to classify it as "automobile" sometimes, and vice versa. Notably, our approach does not require predefined semantically similar classes. We believe that two classes are conceptually similar only if they are frequently misclassified to each other by the classifier. In this sense, we develop a novel definition of the similarity of two classes, which directly determined by model's 's output. Even if there are semantically similar classes, as long as the model makes incorrect prediction during the training, this still leads to class transitions which has seldom been investigated before. Our intuition could be regarded perturbations class predictions to preserve the on the matched class distributions assumption and therefore is amenable MNAR
some confident ability of the model. Such a strategy does not rely
Given the motivations above, we propose class transition tracking based Pseudo- Rectud--ettiiying Guidance (PRG) to address SSL in MNAR, which is shown in Fig. 3 Our main idea can be presented as
2

---

Under review as a conference paper at ICLR 2023
dynamically tracking the class transitions caused by pseudo-rectifying procedures at previous epoch to provide the class-level guidance for transition of each pseudo-label could become the cure for the deterioration of the useudo-rectifying ability of the traditional SSL. methods in MNAR. A graph is first built on the class tracking matrix recording each pseudo-label's class transitions occurring in pseudo-rectifying procedure. Then we propose to model the class transition by the Markov random walk, which brings information about the difference in the propensity rectify pseudo-labels of one class into various other classes. Specifically, we guide the class transitions of each pseudo- label during the rectifying process according to the transition probability corresponding to the current class prediction. The probability is obtained by the transition matrix of Markov random walk, which has been rescaled at both the class-level and the batch-level. Moreover, the class prediction at the last epoch can also be introduced to guide the pseudo-rectifying process at the current epoch. PRG recalls classes that are easily overlooked but appear in class transition history. They are deemed as similar to the ground-truth. and have more chance to be assigned rather than simply letting the model assign the classes it favors without hesitation. By this, PRG could help improve the quality of pseudo-labels suffered from biased imputation potentially caused by the mismatched distributions in MNAR. Because rseuuo-rectifying is a spontaneous behavior of the model, moderately activating class transition will not hinder the learning of the model in the traditional setting. PRG is evaluated on several widely-used SSL classification benchmarks, demonstrating its effectiveness in coping with SSL in MNAR.
at next epoch. We argue that every class
To help understand our paper, we summarize it with the following questions and answers.
What is the novelty and contribution? Towards addressing SSL in MNAR, we propose transition tracking based Pseudo-Rectio Rectiffin Guidance (PRG) to mitigate the adverse effects of mismatched distributions via combining information from the class transition history. We propose that the pseudo-rectifying guidance can be carried out from the class-level, by modeling the class transition of the pseudo-label as a Markov random walk the graph.
Why does our method work for MNAR? In MNAR, being aware of rare class plays a key role, PRG enhances the model to preserve a certain probability to generate class transition to rare classes when assigning pseudo-labels This form of probability based on class transition history produces effective results, because we do not spare any attempt of the model to identify the rare class by class transition tracking (such attempts would be slowly buried due to overlearning of the popular classes). Thereby, PRG helps the model to still try to identify rare classes with a certain probability. This corresponds to soft pseudo-label strategy, where adjust the probability distribution of soft labels so that the model can assign pseudo-labels to rare classes with a clear purpose.
How about the performance improvement? Our solution is computation and memory friendly without introducing additional network components. PRG achieves superior performance in the MNAR scenarios under various protocols, e.g., it outperforms CADR (Hu et al. [2022), a newly- proposed method for addressing MNAR, by up to 15.11 % in accuracy CIFAR-IO. Besides, we show the performance of PRG is also competitive in the traditional SSL setting.
2 RELATED WORK
Semi-supervised learning (SSL) is a promising paradigm to address the problem by effectively utilizing both labeled and unlabeled data. Given an input (labeled unlabeled data), our objective in SSL can be described as the learning of a predictor for generating label for it. In conventional SSL
settings (Berthelot et al. 2020 [ohn et al. 2020: Li al., 2021 Zhhng al. 2021) and imbalanced SSL (Wei et al. 2021). underlying most of them is the assumption: labeled and unlabeled data
are matched and balanced. Some more practical scenarios for SSL are now extensively discussed. Recently, some work has focused on addressing the class -aaasiiibllneeed issue in SSL. Kim et al. (020)) refines the pseudo-labels softly by formulating convex optimization. /Wei et al. (2021) proposes class-rebalancing self- foff-trainin combining distribution alignment. However, these existing methods still underestimate the complexity of practical scenarios of SSL, e.g., Wei et al. (202I) works based on strong assumptions: the labeled data and unlabeled data fall in the same distribution (i.e., their distributions match). Further, a novel and realistic setting called label missing not at random is proposed in Hu et al. (2022), which pops up in various fields such as social analysis, medical sciences and so on (Enders) 2010; Heckman
1977). To address the mismatched distributions of labeled and unlabeled data in MNAR, Hu et al. 2022 proposes class alass- doubly robust (CADR)
3

---

Under review as a conference paper at ICLR 2023
Class Transition Tracking
P
PEL
epoch e
apoch e
opoch e
Pseudo- Recudd-n Guidance
H
2/2
predict cless
psaudo-label for training
Prs
Class Tracking Matrix C
Transition Matrik H
Ps:
Figure 3: Overview of PRG. Class tracking matrix C is obtained by tracking the class transitions of pseudo-labels (e.g.. Pry for sample x)) between epoch and epoch e++ caused by pseudo-rectifying procedure (Eq. 5). The Markov random walk defined by transition matrix H (each row H, represents the transition probability vector corresponding to class 1) is modeled on the graph constructed over C. Generally, given pseudo-labed e.g., Prę for sample Z2. class- and batch-rescaled H (i... H') is utilized to provide the class-level pseudo-rectifying guidance for Px, according to its class prediction Finally, the rescaled pseudo-label Pig is used for the training.
á arg max (Eqs. (6)
estimator combining class-aware propensity and class-aware imputation to remove the bias on label imputation. Differently, our method alleviates the bias from another perspective, that is to guide the pseudo-rectifying direction based on the historical information of class transitions.
3 METHOD
Formally, we denote the input space as and the label space as y {1, k} over & classes. Following Hu et al. (2022), SSL can be reviewed as label missing problem. The label missing indicator set is defined as M with m E (0, 1). where m = indicates label is missing and m = 0 is the otherwise. Given the training dataset in SSL, obtain set of labeled data: D,. X x y M and a set of unlabeled data: Dy C x x M. Since the ground-trutt trrth yu E _ of unlabeled data my is inaccessible in SSL, prevailing self-training based SSL methods impute yo with pseudo-label p. p (xx 0) is predicted by the model which is parametrized by 0 and trained on the labeled data. Let (x) VL (4) (t) E i E (1....,,u n be the labeled data pairs consisting of the sample with corresponding ground-trutt label (i.e., (+)) = 0), and (xtf (1) .mm)) E Du, E {NL + L,..., nT) be the unlabeled data missing labels (i.e., ()) = 1), where nt, and ner refer to the number of labeled data and total training data respectively. Hereafter, the SSL dataset can be defined as D Dr, De. In brief, we can review the conventional SSL as optimization task for loss L:
min Σ
C(x, ()
(1)
(2.g.m)ED
where a dataset with independent Y and M. this sense, the model trained on D, can easily impute unbiased pseudo-labed for unlabeled data zu (Hu et al.) 2022). Conversely, the scenario where M dependent with V, namely label Missing Not At Random (MNAR), will make the model produce strong bias on label imputation, which causes the ability of pseudo-rectifying suffer greatly, Take the current most popular SSL method FixMatch (Sohn et al.. 2020) as an example, FixMatch, the term L(x. y; 8) in Eq. (I) can be decomposed into two loss terms LL and Ly with pre-defined confidence threshold T (implying max(p) is used as a measure of the model's confidence):
C(x, y: 0) Lu (æL, 0) + X 1 (max(p)
(ZU, arg max()) 0),
(2)
where >y is the unlabeled loss weight and I()) is the indicator function. Training with MNAR setting in Fig. FixMatch is gradually seduced by samples predicted to be the labeled popular classes with confidence above 7 (even though most of them are wrong), while samples predicted to be the rare class with confidence below do not participate into training, resulting in biased propensity on label imputation. In this work, we propose class transition tracking based Pseudo-Ress Seeuudd-etiiiiinggg Guidance (PRG) to help model better self-correct pseudo-labels with additional guidance information.
4

---

Under review as a conference paper at ICLR 2023
3.1 PSEUDO-RECTES RECTIFYING GUIDANCE
Firstly. we formally describe the pseudo-rectifying process in SSL. In this paper, label assignment is considered as a procedure for generating soft labels. We denote the i-th component of vector æ as Ii. Let E R be the soft label vector assigned to unlabeled data TU, where R is the set of nonnegative
real numbers and Σ P; = 1. Denoting z at epoch e as x", the pseudo-rectifying process can be described as the change on p by the next epoch: p=+1 go where go(p²) is a mapping from p" to 10t determined by the knowledge learned from the model parametrized by 0 at epoch e + 1. In MNAR, take imbalanced Dr and balanced Dy as an example, as the training progresses, the model's confidence is gradually slashed and unexpectedly grows on the rare and popular classes in D2 respectively. To address this issue, it necessary to provide more guidance to assist the model in pseudo-rectifying. In general, the Pseudo-Rectifyinn Guidance (PRG) can be described as
p=11 Normalize(n o ge()"))
where o Hadamard product, scaling weight vector 77 E R and Normalize(z); z. z,/2,z Tj-
We can review the technical contributions of some popular self-training works as obtaining more effective for pseudo-rectifying For example, Batptmpleepsuuoolaaaeeeeee lseudo-labeling based methods (Leee et al. 2013
Sohn et al. 2020 Li al. 2021 Xu et al. 2021 Zhang et al. 2021 set 1): 1/p:
{il arg max (p (f Ap 22T and f/j - J ( j E (1,-... k) A + iJ and, i.e. using confidence threshold to filter low-confidence samples. However, it is difficult to set an apposite at the sample-level (e.g., for simplicity, /Shnn et al. (2020) fixes T to determine v7 for all samples and the value of r is usually set based on experience) to guide pseudo-reetifying. especially in the MNAR settings. In addition, some variants of class-balancing algorithms (Berthelot et al. 2020 Li et al. 2021: Gong al.) (222T) can be integrated into pseudo-rectifying framework. These methods utilize distribution alignment to make the class distribution of predictions close to the prior distribution (e.g., the distribution of labeled data). This process can be summarized dataset-level pseudo- rectifyyng guidance by setting 1) as the ratio of the current class distribution of predictions to the prior distribution, i.e., the fixed 1) are used for all samples. Performing ggeudo-rectifying guidance in this way strongly relies on an ideal assumptions: the labeled data and unlabeled data share the same class distribution, i.e., D, 3/ is independent with M. Thus, these approaches fail miserably in the MNAR scenarios, which can be demonstrated in Appendix [.1] As we discussed in Sec. [ it is also feasible to guide pscudo-rectifying at the class-level. Hence, we define rectifying weight matrix as A E k where each row A; is representing the rectifying weight vector corresponding to class i. Denoting the class prediction as p- arg max(p). the class-level pseudo-rectifying guidance can be conducted by plugging Age+1 into 1) in Eq. 3
g Normalize(Ag. (p(
Next, we will introduce simple and feasible way to obtain an effective A for PRG to improve the pseudo-labels predicted by SSL models the MŇAR scenarios.
3.2 CLASS TRANSITION TRACKING
Firstly, we consider building fully connected graph G in class space V. This graph is constructed by adjacency matrix C e R&* (dubbed as class tracking matrix), where each element Cıý represents the frequency of class transitions that occur from class to class j (i.e., an edge directed from vertex i vertex j G). Gij is parametrized by the following class transition tracking averaged last NB batches with unlabeled data batch size Bu, ie., Cij ENY c /Ng, where
(1.... NB) c. = 0. 0 (n) = (2500) p(0).c = i, (b). = j j,b (1....,u)) (0.mmm.m.. m
But)
(5)
Hereafter, define the Markov random walk along the nodes of G, which is characterized by its transition matrix H R Each element H. represents the transition probability for the class prediction á transits from class at epoch to class j at epoch e + 1. In specific, H is computed by conducting row-wise normalization on C. The above designs are desirable for the following reasons.
(I) In the self- tffiiiaining process of the model, the historical information of pseudo-rectiffing contains the relationship between classes, which is often ignored in previous methods and can be utilized
5

---

Under review as a conference paper at ICLR 2023
b
Kola
ricedtr
24
1.8
dut
dg
dag
12
42
30000 Inerations
60000 Iterations
90000 Iterations
120000 Iterations
150000 Iterations
Figure 4: Visualization of class tracking matrix C obbtained in training process of FixMatch (Sohn
et al. 2020p on CIFAR-10 with the same setting as in Figs. 2c and 2h The darker the color, the more frequent the class transitions. Overall, the number of class transitions decreases as the training progresses. Class transitions occur intensively between the popular classes, and class transitions between the rare classes gradually disappear (e.g., between "ship" and "truck").
to help the model assign labels at new epoch. We can record the class transition trend pseudo- rectifying by Eq. (5). which corresponds to the transition probability represented by Hij. i.e,, for sample z, when its class prediction p is in the state of class i, if a rectifying procedure resulting in a class transition occurs, what probability will it transit class j. Intuitively, given p with model prefers to rectify it to another class similar to class in one class transition, í.e., the preference of class transitions classes are, the more likely they are to misclassified as each other's classes. The label is more likely oscillate between the two classes, resulting in more swinging class transitions. As shown in Fig. 4 in the "dog" class predictions, the predictions transitioning to the "cat" class are significantly more than other classes, and vice versa in the "cat" labels. We can observe that C behaves like a symmetric matrix. reflecting the symmetric nature of class similarity. Consequently, this similarity between classes can be utilized to provide information for our class-level pseudo-rectifying guidance.
the
also be regarded as the similarity between classes and the more similar two
(2) In the MNAR settings, the tricky problem is that the mismatched distributions lead to biased label imputation for unlabeled data. The feedback loop of self-reinforcing errors not achieved overnight. Empirically, the training progresses, the model becomes more and more confident in the popular classes (in labeled or unlabeled data), which leads to misclassify the samples that it initially thought might be the rare classes to the popular classes later. As shown in Fig. 4 the lower left corner and upper right corner of the heatmap (i.e., the class transitions between the popular classes and rare classes) is getting lighter and always lighter than the upper left corner (i.e., the class transitions among the popular classes), which means the model increasingly reluctant to transfer the class prediction to the rare classes during the eseudo-rectiffing process. If only focus what the model has learned at present, the model's past efforts to recognize the rare classes will be buried. The latent relational information between classes is hidden in the pseudo-rectifying process producing class transitions. The history of class transitions can point the way for bias removal on label imputation with an abnormal propensity on different classes caused by mismatched distributions in MNAR
With obtained H. some preparations are done for plugging it into Eq. 4 to replace A. We're re only modeling the pseudo-rectifying process resulting class transition (i.e.. C+ 0). which means H, 0, i.e., Tipe is set to 0 in Eq. (3). This will encourage the class prediction to transition to other classes during each pseudo-rectifying process, which is unreasonable for training robust classifier. Hence, we control the probability that does not transition class by setting HG, k- where र- is the average of the transition probabilities in each row of H and a is pre-defined hyper-parameter In addition, to avoid training instability, we scale each element in by
Σ La
Ei Oid Lj
11
Hij.
(6)
1d= Σ#=1 Cad
where E RC and L2 records the number of class predictions belonging to class averaged on last NB batches. The first term the right-hand side of Eq. (6) rescales Hy at the batch-level while the second term rescales Hy at the class-level, controlling the intensity of class transition together. For specific, the excessive class transitions in the self-training loop could yield confused supervision information that not conducive to learning. Hereafter, to simply integrate our method into the above framework of pseudo-rectifying guidance, we plug H' into A in Eq. (4
ge+1_ Normalize H4g-+ go g)( ()²)),

---

Under review as a conference paper at ICLR 2023
1 2 1
Roo
Clas Judes
Mhh p-23
120 Trinkg Muniian (rPP)
Traniay Zemuiam (01)
Trakning
(a) FixMatch
(b) PRG
(c) PRG vs. FixMatch
Figure 5: Results on CIFAR-10 under CADR's protocol. (a) and (b): Class-wise pseudo-label error rate with 7 50. (c): Learning curve of PRG. We mark the final results of FixMatch as dash lines.
Table I: Mean accuracy (%) in MNAR under CADR's protocol. The results of baseline methods are derived from CADR (Hu et al. 2022). The larger y, the more imbalanced the labeled data. In the 1Diiff. our accuracies are averaged on 3 runs while the standard deviations S=dt and the performance difference (t (Dfff.. compared FixMatch (Sohn al. 2020) are reported.
format Mean 1std.
CIFAR-10
CIFAR-100
mini-ImageNNt
Method
7 20
50
100
100
200
100
II Model
21.59
27.54
30.39
24.95
29.93
33.91
11.77
15.30
MixMatch
26.63
31.28
37.82
41.32
13.12
18.30
ReMixMatch
41.84
38.44
38.20
42.45
39.71
39.22
22.64
23.50
FixMatch
56.26
65.61
72.28
50.51
48.82
50.62
23.56
26.57
+ Crest
51.10-5.14
15.40-10.22
63.60
40.30
46.30 12.52
1..6
DARP
63. 76.m
70.4444
74.7417.u
38.87 38.87-11.44
111.44
40.49 Ln.s0
45.154
CADR
1263123.m0 ar
93.79
93,97 721.00
59.53 19.00
60.88 7:2.00
63.30
29.07 19.51
32.7816.2
LRg (Ours)
PRG (Ours)
94.04
$0.
94.09
93. 11.05
94.28
-0.79
59.11
2.2
62.36
61.84 148.00
20.45
63,41
60.56 11.080
75.94
44.28
40.73
0.04
43.89
44,99
44.99
20.34
where H' g= 11 can be regarded as the class prediction for one sample randomly walks along the nodes of C at the current epoch, i.e., drive possible class transition in the pseudo-rectifying for bias removal on label imputation propensity due MNAR (more discussions can be found in Appendix
B We note that it is also feasible to use the class transition driven by 6 to revise p (what is the class prediction after a class transition from last epoch to the present), i.e., replace H in Eq.
with He which is dubbed as Lat The whole algorithms presented in Appendix A
EXPERIMENT
Dataset and Baselines. We evaluate PRG on three widely used benchmarks in SSL, including
CIFAR-10. CIFAR-100 (Krizhevsky et al. 2009) and mini-ImageNet (Vinyals et al.) 2016) (a subset of ImageNet (Deng et al. 2209) composed 100 classes). Following Hu et al. (2022), we mainly report the mean accuracy PRG in both conventional SSL settings and various MNAR scenarios. Multiple baseline methods are compared, including representative conventional SSL algorithms: II Model (Rasmus al., 2015). MixMatch (Berthelot al.) 2019), ReMixxatch (Berthelot et al., 2020). and FixMatch (Sohn et al. 2020). More importantly, we provide fair comparisons with the recent label bias removal methods for imbalanced SSL: DARP (Kim et 2020), Crest (Wei al. 2021)
and the latest approaches designed for addressing SSL in MNAR: CADR (Hu et al. 2022
MNAR Settings. Following Hu et al. (2022). the MNAR scenarios are mimicked by constructing class-imbalanced subset of the original dataset for either the labeled data or unlabeled data.
Let y denote the imbalanced ratio, NN and M, respectively refer to the number of the labeled and the unlabeled data in class from classes. Three MNAR protocols are used for the evaluations on PRG:
(1) CADR's protocol (uu et al. 2022). N, = F in which N; = is the maximum number of labeled data in all classes, and the larger the value of y, the more imbalanced the class distribution of the labeled data. For example, Fig. shows CIFAR-10 with 7 20. (2) Our protocol. Because the
7

---

Under review as a conference paper at ICLR 2023
Table 2: Mean accuracy (%) in MNAR under our protocol with the varying labeled data sizes z2 and imbalanced ratios N1. Baseline methods are based on our reimplementation.
CIFAR 10 (m 40)
CIFAR (m 250)
CIFAR 100
2500)
mini ImageNet (nzz, 1000)
Method
N1
20
100
200
100
200
40
80
FixMatch
80.52
16.53_...0
29.76
46.53 :o. 12
11.31
41.38,22.0
36.20.g..3
28.33
CADR
85.5419.15
75.11
11.42
92.25
63.92
61.6219:1
46.1674.78
74
36.08 10-12 10.12
30.52 10
PRG (Ours)
91.87 16.18
77.44
Tb.a
93.93
67.86
[31.3]
61.491..18
49.84
T8.46
39.99 13.79
35.39 77.56
PRG (Ours)
85.6 85.66
$5.58
10.00
77.85
#1.00
11.32
92.80
#1.44
133.0
64.00
117.47
4:500
60.41 #1.01
16.90
43.80
2:1.7
13.43
39. 39.84
83.04
40.06
33.17
+-.82
total number of labeled data ny in the CADR's protocol varies with y, which violates the principle of controlling variables, n is fixed by users in our protocol. N1 is altered for different scales of
imbalance, i.e., N₁- = N1 X
while y is calculated by the constraint Σ/= N₂ = nLL. We further
consider the MNAR settings where the unlabeled data is also imbalanced, i.e., M, M Yfu (implying inversely imbalanced distribution compared with the labeled data), where M1 = 5000 in
# CIFAR-10. (3) DpRo'cprotocol (Kim et al. p020): N₂- N1 x 7 - M4 M1 x 7/a where Ni = 1500 and Mi = 3000 in CIFAR-IO, where " and 7h are varied for labeled and unlabeled data respectively, i.e., the distributions of the labeled and unlabeled data are mismatched and imbalanced.
Implementation Details. In this section, PRG implemented as a plugin to FixMatch (Sohn et al., 2020) (see Appendix D.2 for other SSL learners). Thus. we keep the same training as FixMatch (e.g., unlabeled data batch size By 448), whereas the class invariance coefficient a= and the tracked batch number NB = 128 are set for PRG in all experiments. The complete list of hyper-parameters can be found in Appendix C Following |Sohn et al. (2020), our models are trained for 2 iterations, using the backbone of 2ide
(WRN) Zagoruyko & Komodakis, 2016) for CIFAR-10, WRN-28-8 for CIFAR- 100 and ResNet-18 (He et al. 2016) for mini- mini-Imagenet.
4.1 RESULTS IN MNAR
Main Results. The experimental results under CADR's and our protocol with various levels of im- balance are summarized in Tabs. and 2 PRG consis- tently achieves higher accuracy than bascline methods across most of the settings, benefiting from the informa- tion offered by class transition tracking. As shown in Figs. 5a and [5b] the -eetuddeetiiyinggg ability of PRG is significantly improved compared with the original FixMatch, i.e., as the training progresses, the error rates of both the popular classes and the rare classes of the labeled data are greatly reduced, eventually yielding improvements in test accuracy shown in Fig. [cc Mcanwhile, in Tab. 3 we further provide geometric mean scores (GM, metric often used for imbalanced dataset (Kubat al., 1997] Kim
Table 3: Geometric mean scores (GM) on CIFAR-10 under CADR's 's protocol.
Method
= 20
y 50
100
FixMatch
41.90
53.61
60.35
CADR
75.25 758 a6
-1.55
92.98
739.37
20.43
93.15780.3 782.5
3s
PRG
93.53
20.39
93.70 50.35
93.94
58
Lag
93.35
92.99
93.25 f32.00
et al. 2020)), which is defined by the geometric mean over class-wise sensitivity for evaluate the classification performance of models trained MNAR. More metrics for evaluation (e. g..g.. precision and recall) and the results of PRG built on other SSL learner can be found in Appendix D.2
Our main competitors can be divided into three categories. (1) State-of Theeff-ThhArrr Art (SOTA) SSL methods such as ReMixMatch (Berthelot et al. 2020) and FixMh (Sohn et al., 2020). As shown in Tabs.
[and 2 these methods show poor performance under MNAR. Especially, our backbone FixMatch can't t cope with MNAR at all, whereas with our method, the performance is significantly improved by
more than 10% in most cases. (2) Imbalanced SSL methods: DARP (Kim et al., 2020) and Crest (Wei et al. 2021). These two SOTA methods addressing long-tailed distribution in SSL emphasize the bias removal in matched distribution (i.e., the unlabeled data is equally imbalanced as the labeled
data), showing very limited capacity in handling MNAR. (3) SSL solutions devised for the MNAR scenarios: CADR (Hu et al. 2022). Our method outperforms CADR under its proposed protocol across the board, demonstrating PRG is more effective for bias removal on label imputation than it. With extremely few labels, the class-aware propensity estimation in CADR is not reliable whereas method still works well, yielding performance gap of up 14.41 %.
8

---

Under review as a conference paper at ICLR 2023
Tot Accuracy (%g
130
Test Accurcy (%)
100
FiaMatch
DARP
PRG
FxxMashh
FRG
CADR
H0,122)
10.1
(0044.4)
(0044011)
(n N..
Yu)
(a) Our protocol
(b) DARP's protocol with "y 100
Figure 6: Results on CIFAR-10 under two protocols. The imbalanced distributions of labeled and unlabeled data are in reverse order of each other in (a) and the case of % marked with "R" in (b).
More MNAR Settings. More MNAR scenarios are considered for evaluation. In our protocol, alter N1 and Y/u mimic the case where the distributions of the labeled and unlabeled data imbalanced and mismatched, i.e., the two distributions are different. Likewise, DARP's protocol produces similar mismatched distributions. As shown in Fig. PRG achieves promising results in all the comparisons with the baseline methods. Our method boosts the accuracy of the original FixMatch by up to 35.51% and 24.33% in our and DARP's protocols respectively. The activated class transitions make the model less prone to over-learning unexpected classes so that the negative effect of MNAR can mitigated. Moreover, the results of balanced labeled data with imbalanced unlabeled data and more application scenarios (e.g., tabular data) can be found in Appendix D.2
4.2 CONVENTIONAL SSL SETTINGS AND ABLATION STUDIES ON HYPER-PARE PARAMETERS
As shown in Tab. our method still works well on balanced datasets. The class-level guidance offered by our method is also valid in the conven- tional setting while maintaining the vitality of class transition, even though there is not too much need to remove bias on label imputation. Hereafter, we investigate the effect of the class invariance coeffi- cient a and the tracked batch number Nn on PRG, which is shown in Fig. 7 Choosing appropri- ate a to control the degree of class invariance in pseudo-rectifying is important for PRG, which ensures stability of supervision information and train- ing. Meanwhile, we note that too small Ng is not sufficient to estimate the underlying distribution of class transitions, where NB 128 a sensible choice for both memory overhead and performance.
100
0.102
Na
Figure 7: Ablation studies with and NB on CIFAR-10 under CADR's ' protocol with y = 20.
Table 4: Mean accuracy (%) in the conventional setting with various nt, Results of baselines reported in CADR (Hu et al. 2022) while results of are based on our reimplementation.
CIFAR-10
CIFAR-100
mini- Imageee
Method
1 40
250
4000
400
2500
10000
1000
FixMatch
88.61+3.3
90.33
95.69+c.15
50.05++.00
71.36
76.82+
39.03+00.6
CADR
94.41
94.35 10:88
95.59 19....11
52.90 19
70.61 70.61-6.1
76.93
PRG (Ours)
94.44
94,42-0.61
95.38 xc
52.45 33.75
70.12-1:04
-0.21
+0.44
47.34
$:1.00
LaG (Ours)
93.00
94.43-0.95
95.76
00.00
48.81
11.24
70.01 11.08
77.12
$0.33
48.23
5 CONCLUSION
This paper can be concluded as proposing a effective SSL framework called class transition based Pseudo-Rectifying Guidance (PRG) address SSL in the MNAR scenarios. Firstly, we argue that the history of class transition caused by pseudo-rectffynng can be utilized to offer informative guidance for future label assignment. Thus, we model the class transition as a Markov random walk along the nodes of the graph constructed on the class tracking matrix. Finally, we propose utilize the class prediction information at current epoch (an alternative strategy is to combine the class prediction at the last epoch) to guide the class transition for can be alleviated. Given that our method achieves considerable performance gains in various MNAR settings, we believe PRG can be used for robust supervied learning in broader scenarios.
that the bias of label imputation

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement. For reproducibility, please refer to the method described in Sec. $ and the algorithmic presentation shown Sec. A The implementation details (including backbone, hyper- parameters, traning details, etc) can be found in Sec. I and Sec. C Moreover, the checkpoints and evaluation code are available at the anonymous link httpps:: : anonymous 4open. sci en T/PRG45SL-MNAAR 8DE2 We promise release the all source code if the paper is accepted.
REFERENCES
Eric Arazo, Diego Ortego, Paul Albert, Noel E. O'Connor, and Kevin McGuinness, Pscudo-labeling and confirmation bias in deep semi-supervised learning, arXiv preprint arXiv: 190.::999000083,, 2019.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A. Raffel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, 2019.
David Berthelot, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring. In International Conference on Learning Representations, 2020.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. IEEE
Transactions on Neural Networks, 20(3)5542-542, 2009.
Agathe Colléony, Susan Clayton, Denis Couvet, Michel Saint Jalme, and Anne-Caroling Prévot. Hu- man preferences for species conservation: Animal charisma trumps endangered status. Biological Conservation, 206:263-269, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hier- archical image database. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2009.
Craig K Enders. Applied missing data analysis. Guilford press, 2010.
Chengyue Gong, Dilin Wang. and Qiang Liu. Alphamatch: Improving consistency for semi- supervised learning with alpha-divergence. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
James Heckman. Sample selection bias as a specification error (with an application to the estimation of labor supply functions). Technical report, National Bureau of Economic Research, 1977.
Xinting Hu, Yulei Niu, Chunyan Miao, Xian-Sheng Hua, and Hanwang Zhang. On non-random miss- ing labels in semi-supervised learning. In International Conference Learning Representations, 2022.
Jachyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Distri- bution aligning refinery of pscudo-label for imbalanced semi-supervised learning. In Advances in Neural Information Processing Systems, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
Miroslav Kubat, Stan Matwin, al. Addressing the curse of imbalanced training sets: one-sided selection. In International Conference on Machine Learning, 1997.
Dong-Hyun Lee al. Pseudo-label:
The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, International Conference on Machine Learning, 2013.
Junnan Li, Caiming Xiong, and Steven CH Hoi. Comatch: supervisd learning with contrastive graph regularization. In /EEF/CVF International Conference Computer Vision, 2021.
10

---

Under review as a conference paper at ICLR 2023
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining, In European Conference Computer Vision, 2018.
Ishan Misra, C Lawrence Zitnick, Margaret Mitchell, and Ross Girshick. Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
Antti Rasmus, Harri Valpola, Mikko Honkala, Mathias Berglund, and Tapani Raiko. Semi-supervised learning with ladder networks. In International Conference Neural Information Processing Systems, 2015.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations, 2021.
Saharon Rosset, Ji Zhu, Hui Zou, and Trevor Hastie. A method for inferring label sampling mechanisms in cemi-supervised learning. In Advances Neural Information Processing Systems, 2005.
Kihyuk Sohn, David Berthelot, Chun Chun-Liang Li. Zizhao Zhang, Nicholas Carlini, Ekin D. Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In Advances in Neural Information Processing Systems, 2020.
Kai Sheng Tai, Peter Bailis, and Gregory Valiant. Sinkhorn label allocation: Semi-supervised classification via annealed self-training. In International Conference on Machine Learning, 2021.
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning, Machine Learning. 109(2):373-440) 2020.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, 2016.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Yi Xu, Lei Shang. Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference Machine Learning, 2021.
Jinsung Yoon, Yao Zhang, James Jordon, and Mihacla van der Schaar. Vime: Extending the success of self-and semi-super learning to tabular domain. In Advances in Neural Information Processing Systems, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference, 2016
Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. In Advances in Neural Information Processing Systems, 2021.
Zhi-H Hua Zhou. Semi-supervised learning. Machine Learning, pp. 315-341, 2021.
11

---

Under review as a conference paper at ICLR 2023
APPENDIX
A ALGORITHM
Pseudo-code of PRG is presented in Algorithm while that of PRG is presented in Algorithm
Algorithm 1: PRG: Pseudo- Guidance
Input: class tracking matrices C ;C i E (1,..., Na),. labeled training dataset DL, unlabeled training dataset Dy, model 0, label bank ( E (1, BT - nz)]
for 1 to Maxlteration do
From Dr., draw mini-batch Br. {1=g
(1) ")""))] ); (1,.., B)} Ur.
From Dr,
H draw mini-batch By = [(r (1))) }:b E (1...., Bu))
Carmaiize(Averaeeee Average(C))
1/ Construct transition matrix
H Σam Li-=n Cas
Eis La
Ein Cia Hij
// Rescale H class/batch-level
L
for = 1 to Bu do
p [o(aa)
// Compute model prediction // Obtain the index of zy in Du // Compute class prediction
idx Index("()) (b))
T arg max(p!
if (ixx) # (000) then (n)
c(a), ((dd) g(v) ((ndx) ((0) +1 [1dx) (0)
// Perform class transition tracking
12
end
Thh) Normalize (00 p()))
// Perform pseudo-rectiffine guidance
end
LL, Lu FixMatch Bu, b (1,..., BB))) 16
// Run FixMatch
17
0 SGD(C, + 0)
// Update model parameters 0
18
end
Algorithm 2: PRGG.ast, Pseudo Pseudo-Rcetifying Guidance Using Class Predictions of the Last Epoch
Input: class tracking matrices C = ; E (1,..., Np)}, labeled training dataset Dz, unlabeled training dataset Dy, model 6, label bank (1 (1,..., BT - =zz}}
for 1 to MaxIteration do
VL );6E (I,.., From Do, draw mini-bated Bu (xx(b);
From Dr., draw mini-batch Bi. = (w
(b)
(6)
); (I,..,Bu))
H
1/ Construct transition matrix // Rescale H class/batch-level
Eim La Hi, Σt-1 Ey- Cas
Σim Cut Hij
for = 1 to By do
p [o(x(())
// Compute model prediction // Obtain the index of a (b) in Du
idx Index(a (8) g Normalize(HH, Haaas o ()))
11 Perform psoudo-rectifying guidance
T arg (a)
// Compute class prediction
if (Udx) # (0) then cin) ((0)) (n)
((000)g0)) +1 (idx) 5(6)
// Perform class transition tracking
13
end
15
end
Lr Lu FixMatch (Br, Bu, (6 (1..., (l....,B))) Bb]))
16
// Run FixMatch
17
0 = SGD(C,. + L),
// Update model parameters 0
18
end
12

---

Under review as a conference paper at ICLR 2023
Table 5: The ablation studies on re-weighting by Eq. (6) We report the results (aceuracy (%) / GM) on CIFAR- 10 under CADR's protocol.
Table 6: The ablation studies on step k. We report accuracy (%) and GM on CIFAR-I0 under CADR's protocol with 7 = 20.
Method
7 =
7 100
(default PRG)
PRG wo. Eq.
88,97 87.37
/9.22
92.72/ /9.555
Accuracy GM
94.04
91.33 90.79
87.76
82.60
PRG
94.04 / 93.53
94,09 /93.70
94.28 /9394
93.53
85,80
80.24
B DISCUSSION ON H-BASED PSEUDO-RECTIFYNG GUIDANCE
In this section, we give insights into re-weighting scheme of H in Eq. (6) based the following
theoretical justification. Overall, we give an explanation from the perspective of gradient. Our
re-weighting scheme potentially scale the gradient magnitude data to mitigate adverse effects of biased labeled data, and suppresses the gradient magnitude when lass transition overheated. Letting p be the naive soft label vector, by Eq. O we re-
the learning of the unlabeled
weight H by Hig Es y Cas' X Eig Cid x Hij and obtain the rescaled pseudo-label vector p Normalize H' p). Hence, the cross-entropy between prediction p and p can be formalized as
Ex=, Ind
Ed Ld Ed_1
Cpa Hic x Pe
k
&
Lu = Σ plog = =
Edar D01-1 Cad
I.c
log Pe
2
c
2dt=1 Cpa
Σ
k
Hpc Pe L.
log Per
3 Ex==
k
Ek
Ed'=1
Cad'
Z
Ed= Ind
where is the normalize factor.
can be regarded as the ratio of pseudo-labels belonging
ΣS La
to class e to all labels and Σ6
Cpu
can be regarded as the ratio of class transitions derived
2d 2 Cas
from class á to population transitions. Denoting the logit outputted from the model as o (implying Soft with no gradient on pseudo-label p. we obtain aCv
>x. Pe Ons i.e., Pe Do."
Do.
0Lu - -5e DcPe- do.
dd= Opa
Hîg
Σ Depa) Σ isc
P.Pe -i.ee po
Z Σ5
2dd=1 Caa
1
Z
Le
Pe-
Σ ti Le
(9)
The larger the difference between Hzaand
Σ2-1 La" the larger the gradient; and the smaller the
Le
difference between Hpe and Ei La the smaller the gradient
@Ce 'aa 0 when
Hgc " Lgg
1). This
means that we intend to provide unbiased guidance (because this is derived from the unlabeled data) for the learning of unlabeled samples from the class level, as to resist the influence of biased labeled
Ede Cpa
samples. Meanwhile, if there are too many class transitions occur on the whole, Ele Ed'= Cas will
decrease, which results in decreasing of Lv, i.e. suppress the trend of class transition overheating. To demonstrate the effectiveness of our re-weighting scheme H, conduct ablation experiments on it. As shown in Tab. [5] the re-weighting scheme can effectively boost the performance of PRG in MNAR because it controls the intensity of class transition together. Additionally, for the utilization of H' in Eq. 7 we consider taking k steps, i.e., multiply by H² instead of H' to uncover more complex patterns of misclassification than simple pairwise class relations. However, as shown Tab. 6 we can observe that the performance is inversely proportional to k. The advantage of PRG is that H' is updated each iteration, which means that the value H' dynamic. As the model learns new knowledge, the past H' may not be suitable for the pseudo-rectifying process anymore. If H is used, this means that we are using the same H' multiple times for given sample, which wastes the advantage of dynamic H'. H² using suitable & or dynamic selection of k might yield better performance, but it is difficult for us to determine the value of Therefore, the PRG is designed for simplicity and exhibits superior the performance.
13

---

Under review as a conference paper at ICLR 2023
C IMPLEMENTATION DETAILS
In this section, we show the complete hyper-parameters in Tab. 7 As mentioned in Sec. 4, our method is implemented as plugin FixMatch (Sohn et al. 2020 Thus, keep the original hyper- parameters in FixMatch and alert additional nyper-parameters in our method. Note that FixMatch sets different values of weight decay for CIFAR-IO and CIFAR-I00, which 0.0005 and 0.001 respectively. For simplicity, we set w 0.0005 for all experiments in our work. Additionally, the models in this paper are trained on GeForee RTX 3090/2080 and Tesla V100. We observe that since additional network components are introduced, the average running time of single iteration hardly increased, which means our method does not introduce excessive computational overhead.
Table Complete list of hyper-parameter of PRG plugged FixMatch Sohn et al. (2020). NB and are additional syper-parametter in our method whereas other hyper- parameersss follow FixMatch. Note that the unlabeled data batch size can be calculated by By μB.
Hyper-paramete
Description
CIFAR-10
CIFAR-100
mini-ImageNet
g
The ratio unlabeled data to laheled data in a mini-hatch Batch size for labeled data and class transition tracking size for unlabeled data Unlabeled loss weight Confidence threshold
7
73
Bu
448
1
0.95
Start learning rate
0.03
B
Momentum
0.9
ur
Weight decay
0.0005
NB
Tracked butch number
128
Class invariance coefficient
1
D ADDITIONAL EXPERIMENTAL RESULTS
D.1 USING DISTRIBUTION ALIGNMENT MNAR
As discussed in Sec. distribution alignment (DA) aims to perform strong regularization on pscudo-labed by aligning the class distribution predictions on unlabeled data to that of labeled data. DA boosts the performance of SSL models tangibly (Berthelot al., 2020 Gong al. 2021 Li al. 2021 Sohn et al., 2020). However, DA works on a strong assumption that the distribution of unlabeled data matches that of labeled data. In MNAR, this assumption does not hold obviously. Thus, these methods combining DA fail to address SSL in MNAR, eventually yielding abysmal performance. As shown in Tab. 8 rather than improving performance, integrating DA into SSL models is counterproductive, e.g., original FixMatch outperforms FixMatch with DA by up to 28. 28.68% on CIFAR- 10. DA leads to substantial deterioration of the model performance in the MNAR scenarios due to the large gap between the labeled data utilized and the unlabeled data distribution. Conversely, our method not restricted by the mismatched distributions and achieves superior performance across the board, because PRG helps the model to better handle MNAR scenarios without any prior information (distribution prior estimated from labeled data is used DA).
Table Mean accuracy (%) MNAR under our protocol compared with more baseline methods. DA indicates distribution alignment technique proposed in Berthelot et al. (2020). CoMatch (Li et al. 2021) is secently-propossee SSL method integrating contrastive learning and graph-based methods. Note that CoMatch also combines DA to improve the quality of pseudo-labels for better performance in the conventional SSL setting.
CIFAR-10 (nL 40)
CIFAR-10 (n 250)
CIFAR-100
=500)
mini-ImagoNet 1000)
Method
10
20
100
200
100
200
40
80
CoMatch
60.27
39.48
$7.87
26.77
30.08
30.24
21.47
FixMatch
85.7
76.53
69.76
46.53
61.31
41.38
36.20
28.33
71.23/1444
47.85
61,847.36
27.61
50.94410.37
31.8219.88
33.8712.55
23.53-4.78
PRG (Qurs)
91.8776.15
77.4470.91
93.93724.11
67.86121
61.49
49.8479.46
39.99
35.3917.000
PRG (Ours)
85.66
77.85
92.8 72.802
64.0 64.0"17.44
60.41 10.00
43.8 43.8 72.42
39.8413.64
39.84
33.10 74.17
14

---

Under review as a conference paper at ICLR 2023
Table The comparisons of Class-wise precision and recall on CIFAR-IO the training process under CADR's protocol with 50.
90000
150000
Method
Class Index
Precision
45.21
95.22
46.89
96.72
47.93
97.80
49.12
99.01
49.59
99.27
50.27
98.72
38 49
88.73
39.74
88.43
70.26
89.47
75.02
75.63
72.19
75.93
86.14
88.43
86.88
90.21
88.42
94.38
FixMatch
89.45
62.93
91.03
64.4
89.31
75.98
86.47
90.03
90.23
8.89
91.37
94.80
89.09
75.94
90.48
75.21
95.32
75.37
99.02
0.00
97.95
1.00
97.21
10
0.00
0.00
99.60
0.33
98.60
0.67
70.52
93.52
87.34
95.50
88.25
95.
82.53
98.21
96.03
98.32
96.78
98
73.52
76.54
90.92
89.85
92.37
90.57
70.21
73.77
85.36
80.51
89
81.37
PRG (Ours)
79.03
86.57
90.31
96.31
92.74
96.19
74.55
61.03
90.58
88
90.97
82.43
93.09
92.58
80.14
95.01
96.21
96.32
97.50
96.31
76.50
95.22
92.12
95.63
93.55
10
96.56
62.52
96.95
96.01
97.15
96.81
D.2 MORE EVALUATIONS ON PRG
More Metrics. To comprehensively explore the improvement PRG in MNAR, we report
1.000
0.975
the difference in class-wise precision and recall
Confidence
0.960
with/without PRG. The experimental results are shown in Tab. 9 Compared original FixMatch, we witness FixMatch with PRG achieves higer pre- cision/recall by and large, especially on rare classes (i.e., class with larger index), which demonstrates that the bias removal capability of PRG effectively mitigates the effect of MNAR on the model. We also observe that both PRG and FixMatch achieve high precision as well as recall on popular classes and high precision but low recall on rare classes (especially FixMatch) in the early training period. The improvement recall by PRG is due to the activated class transitions, which gives the model a certain probability to assign pseudo-labele to rare classes. In addition, as shown in Fig. PRG exhibits superior bias removal for confidence of pseudo-labels in MNAR, whereas FixMatch filters out too many labeled rare class samples with confidence lower than т, e,g., class 8, 9 and 10, resulting in the waste of unlabeled data.
0.925
0.900
0.875
0.850
FixMatch
0.825
PRG
0.800
8
Class Index
Figure Violin plot of confidence scores on the unlabeled data under CADR's protocol with = 100. The confidence threshold - 0.95 in FixMatch marked out in red.
More MNAR Scenarios We also provide more experiments on the setting of balanced labeled data with imbalanced unlabeled data, which summa- rized in Tab. [0 For specific, we set nı = 40 with balanced distribution and set 7a 50, 100, 200 for imbalanced unlabeled data, i.e., the class-wise
Table 10: Accuracy (%) on CIFAR-IO with
n. = 40 and various Yu. The labeled data is balanced and the unlabeled data is imbalanced.
Method
Tu = 20
= 5U
100
+
CoMatch
52.73
46.20
38.85
number of unlabeled data M; = M1 x 7
FixMarch
57.54
54.82
50.67
where M 5000 in CIFAR-IO. As shown in Tab. T0 PRG outperforms all baseline methods by large margin (it is worth noting that the per- weaker than original Fix.atchh, proving the robustness of PRG in this MNAR scenario due to the unbiased guidance derived from the class transition history.
DA
54.0g
46.71-9.11
41.37
CADR
49,3818
45.27-9.65
48.30
+ PRG (Ours)
62.4314.90 so
62.44 77.62
58.23 17.58
formance of CADR is
15

---

Under review as a conference paper at ICLR 2023
Table 1I: Accuracy (%) on tabular MNIST. y is varied for CADR's protocol whereas nz and Ni are varied for our protocol. Mean + Std. are computed over 50 runs.
Method
250,
250,
VIME
63.38:44.42
63.75:+6.10 63.92:113.9
04.8022.766 66.60=11.55 12.58
50.1347.55 49.28:111.09
30.73 69
60.58:22.68 66.14:111.88
21.44+0.58 24.51199.56
PRG (Ours)
19.41=14.45
34.08:116.05
More SSL Learners. Moreover, to further evaluate PRG's performance, we consider building PRG on the top of more SSL frameworks. Thus, we conduct experiments on CIFAR- 10 under CADR's protocol with UPS (Rizve al. 2021) combining PRG. UPS is recently-propont pseudo-label selection framework for SSL, which is the SOTA method among pseudo-labeling based methods. We keep all training settings the same the original UPS. With 7 = 20, UPS achieves accuracy of 30.46% % whereas UPS with PRG achieves an accuracy of 32.22 %. We note that UPS performs poorly in the MNAR scenarios because a more pure pseudo-labeling approach that does not introduce consistency regularization to improve model performance. Also we observe that PRG improves UPS marginally, much less than FixMatch. This is understandable because the negative learning that UPS prides itself on can be potentially negatively affected by the probability distribution of pseudo-label being adjusted by PRG, egg., uncertainty being altered.
uncertainty-awarr
More Data Type In order explore broader application scenario of PRG, apply it tabular data. We conduct further experiments on tabular MNIST (interpreting MNIST as tabular data with 784 features) by plugging PRG into VIME (Yoon et al. 2020). VIME is prevailing self- and semi- supervised learning frameworks for tabular data with pretext task of estimating mask vectors from corrupted tabular data. We implement PRG above the semi- supeiiisee learning component of VIME. PRG provide pseudo-rectifying guidance to rescale the pseudo-labels for the original unlabeled sample VIME. Specially, we replace the consistency loss used in VIME (i.e., mean squared error in Eq. (9) in Yoon et al. (020)) with standard cross-entropy loss to makes PRG applicable to VIME, We use two protocols to show the performance advantage of PRG, including CADR's protocol and protocol. As shown in Tab. II except for CADR's protocol with y 20 and our protocol with Ti = 40, Ni = 10 (it is worth noting that our upper limits of performance greatly exceed that of VIME), PRG outperforms original VIME by large margin in the most of settings. The reason is that the scheme of ctass-transition based pseudo-rectifying guidance is high-level and general (not limited image data), which ultimately yields robust effect on the MNAR problem with tabular data.
16


Reviews:
[{"review_id": "iF6BsLTzXT6", "reviewer": "Reviewer_nu14", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A practical and challenging scenario called label Missing Not At Random (MNAR) is usually ignored in previous works on semi-supervised learning (SSL). In MNAR, the labeled and unlabeled data fall into different class distributions resulting in biased label imputation, which deteriorates the performance of SSL models. In this work, class transition tracking based Pseudo-Rectifying Guidance (PRG) is devised for MNAR. PRG unifies the history information of each class transition to activate the model’s enthusiasm for neglected classes, so as the quality of pseudo-labels on both popular classes and rare classes in MNAR could be improved.", "summary_of_the_review": "I liked reading this paper, and I think it proposes good ideas to an important problem. My main concerns are that (a) this paper solves only a part of the MNAR problem, and (b) the accuracy improvement is not as strong as the authors claim in the paper, and there are not sufficient analysis and explanation on the result, i.e., Table 1.\n", "strength_and_weaknesses": "Strengths\n1. This paper studies an important and practical problem in semi-supervised learning, which has been not studied much.\n2. This paper is written very well. The motivation of research is presented clearly along with the limitation of existing approaches.\n3. It is good to represent an existing approach (Sohn et al., 2020) as the framework of PRG and explain why it fails at MNAR scenarios.\n\nWeaknesses\n1. The paper claims to solve the MNAR problem, but I think it solves only one part of the problem: when the distribution of labeled data is balanced. How well does PRG perform in the opposite setting, i.e., balanced labeled but imbalanced unlabeled data?\n2. The intuitions for Equations (6) and (7) are insufficient, although they are the core techniques of the proposed approach. Why do we need to use H_{ij}’ instead of H_{ij}? Why do we need the L vector to reweight the transition matrix?\n3. I have some concerns on the experiments. Please see the following: \n    1. I think the main competitor of PRG is CADR, which studies the same MNAR problem. However, the performance of CADR is presented only at Table 1 and missing from Table 2 and 3, and Figure 6. Why?\n    2. In Table 1, the improvement of PRG from CADR is not always significant. In CIFAR-100 with \\gamma=50, the accuracy of PRG is lower than that of CADR. When \\gamma=200, the improvement is negligible considering the standard deviation. I don’t expect PRT to beat CDPR in all cases, but more explanations and analysis on the result would be helpful.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gGuWjG8y_05", "reviewer": "Reviewer_YaM3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors focus on a challenging scenario called MNAR where the labeled and unlabeled data fall into different class distributions, which is different from traditional semi-supervised learning. In order to address the problem, the authors introduce a new technique called Pseudo-Rectifying Guidance (PRG). Using Markov random walk to get the class-level guidance information, PRG fully utilizes the history of each class's transition and makes the model concentrate on neglected classes more, therefore improves the performance of previous SSL models.", "summary_of_the_review": "I tend to accept this paper since the problem is meaningful and the method show plenty of novelty. The main weaknesses are writing and figures, which can be improved in order to help readers understand the idea more quickly.", "strength_and_weaknesses": "Strength:\n1.\tThe paper explains the necessity of solving MNAR problem clearly.\n2.\tThe paper proposes a novel method for MNAR. The idea of focusing on label transition is quite interesting.\n3.\tThe paper shows extensive experiments and the experimental results in the synthetic class-imbalanced settings demonstrated the effectiveness of PRG.\nWeakness:\n1.\tThe paper is a little hard to understand, especially for Section 3.1. The meaning of the symbol \\xi is not clearly explained, and this symbol seems to disappear in pseudo-code although it appears in the main body.\n2.\tFigure 3 is the most important picture in the article ,but it is confusing. Although the style of the figure is nice, it does not help me understand PRG. There is no need to place a black box on (1,1). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YsMXOeZFkd", "reviewer": "Reviewer_buPA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles semi-supervised learning with a focus on the most challenging setting of when there is a mismatch between the distributions of labelled and unlabelled data. The proposed method, PRG, uses the history of class transitions to sway the pseudo-labelling mechanism towards under-represented classes. This works by exploiting pairwise similarities between classes: classes that are often confused must be similar (with respect to the predictions of the model) and thus, if the model predicts one of them, we should also consider the others as possible pseudo-labels with certain probability. The authors validate PRG on CIFAR and mini-Imagenet with promising results, outperforming previous similar methods.", "summary_of_the_review": "The main idea of the paper is interesting and seems to work well in a relevant scenario. The methods are rather simple, but I see that as a potential strength, facilitating their application in future work. However, the exposition of the paper is not clear and the main ideas and motivation are under-explored or not conveyed plainly.", "strength_and_weaknesses": "### Strength\n- The paper proposes a simple and seemingly effective method that improves previous approaches for semi-supervised learning with imbalanced and mismatched class distributions. That is a relevant contribution that, I believe, would be useful both in practical applications and future research.\n- Empirically the proposed method works well, outperforming previous competing methods by significant margins.\n\n### Weaknesses\n- The motivation of the proposed methods is somewhat weak. \n    - The definition of the transition matrix $\\bf H$ is not very well motivated. It is not entire clear to me why Equation (6) is preferable to simply estimating $\\bf H$ from the transitions observed in a batch of data. At least that would be a simple and telling baseline to compare against that, I believe, would add to the paper.\n    - The authors say \"the history of class transitions can point the way for bias removal on label imputation with an abnormal propensity on different classes caused by mismatched distributions in MNAR\" but only hint as to why that is the case. We are told PRG works because it encourages the model to predict under-represented classes, but that does not necessarily require a history of transitions nor does it attenuate \"over-learning the samples of the labeled popular classes\". Empirically, PRG seems to work well, supporting the authors' claim to some extent, but in the current version of the paper the connection between the history of transitions and mismatched class distributions is rather loose. To be clear, I am not suggesting such a connection does not exist. I am just stating that it is not obvious (and thus potentially interesting) and the paper in its current version does not explain it very clearly.\n    - The random walk motivation is interesting but feels under-explored. Why take a simple step in the random walk? If we took $k$ steps, i.e., multiply by $\\bf H^k$ instead of $\\bf H$ in Equation (7), we could uncover more complex patterns of misclassification than simple pairwise class relations.  For instance, if classes A and B are often misclassified as class C, but the model rarely classifies A as B and vice-versa, PRG would not capture this sort of relationship even though the transition matrix does.\n\n- Given that the motivation of the paper is not very grounded, the experiments seem not thorough enough since they are limited to CIFAR and mini-imagenet. CIFAR is notably noisy, which my benefit PRG. It would be interesting to see if the history of transitions is a useful artefact for general types of data other than images. Even simple and easy to run tabular datasets would already contribute to the overall message of the paper.\n\n- The paper is harder to follow than one would expect for such a simple and intuitive approach. Parts of the text are confusing and some of the notation is unnecessary or not properly introduced. I comment on some examples of this in the questions and minor issues below, and in the clarity section.\n\n### Questions\n- What exactly is $\\xi$ computing? \"Rectifying weight vector\" is not a very precise description.\n- In Equation (6), isn't $\\sum_{d=1}^k L_d$ always equal to 1? So the first term on right-hand side of (6) is just the ratio of total number of predictions over the number of predictions that changed class (in a given batch), and is thus always larger than one?\n- If $\\bf H$ is a proper transition matrix, while do we need to renormalise $p$? Is it because of the $\\xi$ term?\n- In Figure 7, it is interesting that the performance drops for larger batch sizes. One would expect the estimate of the transition matrix to be improve, which in turn should translate into better or at least similar performance. Do the authors have an idea as why that is the case? Could it be that the learning rate, if kept fixed in the ablation study, was too large for a batch size of 512?\n\n### Minor issues\n- The authors use $\\max(p)$ as a measure of the model's confidence. That is only one of the possible ways to quantify confidence and that choice should be mentioned more explicitly. Also, the authors imply that the model is more confident ($\\max(p)$ is higher) when classifying classes that appear frequently. While this might be intuitive, I think this is an empirical observation that does not always hold for every data point and for arbitrary models, and in particular not for neural networks. If this is indeed an empirical observation, it should be made clear.\n- In Figure 4, I cannot see the pattern the authors describe in \"the lower right corner (corresponding to the labeled popular classes) of the heatmap is getting lighter and always lighter than the upper left corner.\" Maybe the Figure is not showing right or the matrices need to be renormalised to show the desired effect?\n- Probably best to mention $C_{ii}=0$ already in Equation (5).\n- The missing label indicator $m$ is only used once. Not sure it is worth introducing it.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HoDwx30ors", "reviewer": "Reviewer_Kxam", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies semi-supervised classification when the labels are not missing at random. In this scenario, the labeled and unlabeled data cannot be considered as from the same distribution, and the existing semi-supervised learning algorithms face issues such as over-learning on popular classes and ignoring rare classes. The authors introduced a modification of the pseudo-rectifying procedure by tracking the class transitions. The rationale is that if the transition happens frequently between classes then we would like the algorithm to \"memorize\" it and avoid being over-confident on the imputed labels. The authors illustrate this idea with careful explanations and a number of numerical examples. The improvement on real data experiments is also impressive. ", "summary_of_the_review": "This is a well-motivated and well-written paper. The approach is not super novel, but it seems to have a lot of practical values. I think this is an okay paper. Some theoretical justification will greatly strengthen this paper. ", "strength_and_weaknesses": "Strength:\n\n- I enjoy reading this paper. It does a great job illustrating why each step is designed in this way and what effect it has on the training process. This paper has no theory, but the authors managed to explain the rationale using numerical examples from many different aspects. \n\n- The methodology is simple, intuitive, and easy to implement. The improvement on real data experiments is also quite significant. \n\n- I agree with the authors that it is a strong assumption that the labeled and unlabeled data are from the same distribution. The setting of missing not at random and knowing little about the missing scheme is quite common in practice. I think the authors studied a meaningful problem, and I am glad to see that they provided a simple approach that works so effectively. \n\n\nWeaknesses:\n\n- The major weakness is that this approach lacks theoretical justification. Although the re-weighting scheme has a good explanation, we have no idea whether it will have issues in some cases. For example, it is unclear to me why the particular ${\\bf H}$ should be the correct choice of weights. I was imagining that a monotone transformation of entries of ${\\bf H}$ may also serve as weights, and some likelihood-ratio based weights are more natural choices. To answer this question, we need to at least have some basic theoretical understanding. For example, at least in some very simple settings (e.g., linear classification settings, squared loss in FitMatch), we wish to know what the algorithm does in each update. \n\n- The methodological innovation is not super large. As mentioned in the paper, the data-set level pseudo-rectifying guidance has been proposed in the literature. Using a class-level pseudo-rectifying guidance is more or less the same kind of ideas (but I do appreciate the simplicity of this approach and how it nicely fits the missing-not-at-random and imbalanced class settings). \n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject