DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples
Authors: Yi Xu
Jiandong Ding
Lu Zhang
Shuigeng Zhou
Venue: NeurIPS 2021 Poster (2021)
Keywords: semi-supervised learning
data programming
classification

Abstract: The scarcity of labeled data is a critical obstacle to deep learning. Semi-supervised learning (SSL) provides a promising way to leverage unlabeled data by pseudo labels. However, when the size of labeled data is very small (say a few labeled samples per class), SSL performs poorly and unstably, possibly due to the low quality of learned pseudo labels. In this paper, we propose a new SSL method called DP-SSL that adopts an innovative data programming (DP) scheme to generate probabilistic labels for unlabeled data. Different from existing DP methods that rely on human experts to provide initial labeling functions (LFs), we develop a multiple-choice learning~(MCL) based approach to automatically generate LFs from scratch in SSL style. With the noisy labels produced by the LFs, we design a label model to resolve the conflict and overlap among the noisy labels, and finally infer probabilistic labels for unlabeled samples. Extensive experiments on four standard SSL benchmarks show that DP-SSL can provide reliable labels for unlabeled data and achieve better classification performance on test sets than existing SSL methods, especially when only a small number of labeled samples are available. Concretely, for CIFAR-10 with only 40 labeled samples, DP-SSL achieves 93.82% annotation accuracy on unlabeled data and 93.46% classification accuracy on test data, which are higher than the SOTA results.

DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples
Yi Xu
Jiandong Ding Lu Zhang' Shuigeng Zhou
"hhnnhhai Key Lab of Intelligent Information Processing, and School of Computer Science, Fudan University, China
Alliaba Group
fyxu17, jdding, 1 1_zhang19, sgzhou)@ffudan edd..
Abstract
The scarcity of labeled data is critical obstacle to deep learning. Semi-supervised learning (SSL) provides promising way to leverage unlabeled data by pseudo labels. However, when the size of labeled data is very small (say a few labeled samples per class), SSL performs poorly and unstably, possibly due to the low quality of learned pseudo labels. In this paper, we propose a new SSL method called DP-SSL. that adopts an innovative data programming (DP) scheme to generate probabilistic labels for unlabeled data. Different from existing DP methods that rely on human experts to provide initial labeling functions (LFs), we develop a multiple-choice learning (MCL) based approach to automatically generate LFs from scratch in SSL style. With the noisy labels produced by the LFs, we design label model to resolve the conflict and overlap among the noisy labels, and finally infer probabilistic labels for unlabeled samples. Extensive experiments four standard SSL benchmarks show that DP-SSL can provide reliable labels for unlabeled data and achieve better classification performance on test sets than existing SSL. methods, especially when only a small number of labeled samples are available. Concretely, for CIFAR-10 with only 40 labeled samples, DP-SSL. achieves 93.82% annotation accuracy on unlabeled data and 93.46% classification accuracy on test data, which are higher than the SOTA results.
Introduction
The de-facto approaches to deep learning achieve phenomenal success with the release of huge labeled datasets. However, large manually-labeke datasets are time-consuming and expensive to acquire, especially when expert labelers are required. Nowadays, many techniques are proposed to alleviate the burden of manual labeling and help to train models from scratch, such as active learning [.]. crowd-labeling (2). distant supervision [3) semi [4]weak -5/sselfsuperrvision [6]. Among them, semi- suni-sseereieed learning (SSL) is one of the most popular techniques to cope with the scarcity of labeled data. Two major strategies of SSL are pseudo labels (7) and consistency regularization (8). Pseudo labels (also called self-training [9] utilize a model's predictions as the labels to train the model again, while consistency of regularization forces model to make the same prediction under different transformations, However, when the size of labeled data is small, SSL performance degrades drastically in both accuracy and robustness. Fig. shows the change of prediction error rate with the number of labeled samples of CIFAR-10. When the number of labeled samples reduces from 250 40, error rates of major existing SSL methods increase from 4.74% (USADTM) 36.49% (MixMatch). One possible reason of performance deterioration is the quality degradation of learnt pseudo labels when labeled data size small. Therefore, in this paper we address this problem by
"Corresponinng author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

developing sophisticated labeling techniques for unlabeled data to boost SSL even when the number of labeled samples is very small (e.g. a few labeled samples per class).
Recently, data programming (DP) was proposed as a new paradigm of weak supervision [T]. In DP, human experts are required to transform the decision-making process into a series of small functions (called labeling functions, abbre- viated as LFs), thus data can be labeled programmatically Besides, label model is applied to determining the correct labels based on consensus from the noisy and conflicting labels assigned by the LFs. Such paradigm achieves considerable success in NLP tasks [TT+14]. In addition, DP has also been applied to computer vision tasks [15] 16] However, current DP methods require human experts to provide initial L.Fs, which is cone-conuuinng and expen- sive, and it is not easy to guarantee the quality of LFs. Furthermore, LFs specifically defined for one task usually cannot be re-used for other tasks.
MaMMaich
HeMeMMetth
USADTM
DP SSL
Eror mto
Number labeled camples
Figure 1: Error rate us. #labeled samples (CIFAR-10). Results of existing meth- ods are from the original papers. When only 40 labeled samples are given, all existing SSL methods are substantially degraded and more unstably, while our method is still effective and robust.
In this paper, we propose new SSL method called DP- SSL that is effective and robust even when the number of labeled samples is very small. In DP-SSL, an inno- vative data programming (DP) scheme is developed to generate probabilistic labels for unlabeled data. Different from existing DP methods, we develop multiple-choice learning (MCL) based approach to automatically generate LFs from scratch in SSL style. To remedy the over-confidence problem with existing MCL methods, we assign an additional option as abstention for each LF. After that, we design label model to resolve the conflict and overlap among the noisy labels generated by LF's, and infer probabilistic label for each unlabeled sample. Finally, the probabilistic labels are used to train the end model for classifying unlabcled data. Our experiments validate the effectiveness and advantage of DP-SSL. As shown in Fig. I DP-SSL performs best, and only 1.76% increase of error rate when the number of labeled samples decreases from 250 to 40 in CIFAR-10.
Note that the pseudo labels used in existing SSL methods are quite different from the probabilistic labels in DP-SSL, which may explain the advantage of DP-SSL over existing SSL methods. On the one hand, pseudo labels are "hard" labels that indicate an unlabeled sample belonging to a certain class or not, while probabilistic labels are "soft" labels that indicate the class distributions of unlabeled samples. Obviously, the latter should be more flexible and robust. On the other hand, pseudo labels actually generated by single model for all unlabeled samples, while probabilistic labels are generated from a number of diverse and specialized LFs (due to the MCL mechanism), which makes the latter more powerful in generalization as a whole.
In summary, the contributions of this paper are as follows: 1) We propose a new SSL method DP-SSL that employs an innovative data programming method to generate probabilistic labels for unlabeled data, which makes DP-SSL effective and robust even when there are only a few labeled samples per class. 2) We develop a multiple choice learning based approach to automatically generate diverse and specialized LFs from scratch for unlabeled data in SSL manner. 3) We design a label model with a novel potential and an unsupervised quality guidance regularizer to infer probabilistic labels from the noisy labels generated by LFs. 4) We conduct extensive experiments on four standard benchmarks, which show that DP-SSL outperforms the state-of-fe of-the-art art methods, especially when only small number of labeled samples are available, DP-SSL is still effective and robust.
2
Related Work
Here we briefly review the latest advances in multiple choice learning, semi-supervised learning, and data programming, which are related to our work. Detailed information is available in [17-21].

---

2.1 Multiple Choice Learning
Multiple choice learning (MCL) was proposed overeome the low diversity problem of models trained independently in ensemble learning. For example, stochastic multiple choice learning (23] is for training diverse deep ensemble models. However, crucial problem with MCL that each model tends to be overconfident, which results in poor final prediction. To solve this problem, [24] forces the predictions of con-specialized models to meet uniform distribution, so that the final decision summed over diverse outputs. [25] proposes an additional network to estimate the weight of each specialist's output. In this paper, we develop an improved MCL based scheme to automatically generate diverse and specialized labeling functions (LFs) from scratch in an SSL manner. These LFs are used to generate preliminary (usually noisy) labels for unlabeled data.
2.2 Semi-supervised Learning
Semi-supervised learning (SSL) has been extensively studied in image classification 26 object detection [27] and semantic segmentation [28]. Two popular SSL. strategies for image classification are pseudo labels [ and consistency regularization [8]. Pseudo-label methods generate artificial labels for some unlabeled images and then train the model with these artificial labels, while consistency regularization tries to obtain artificial distribution/label and applied it supervision signal with other augmentations/viev:. These two strategies have been adopted by a number recent SSL works g 39H39). For example, FixMatch [4] proposes simple combination of pseudo labels and consistency regularization. [36] employs unsupervised learning and clustering to determine the pscudo labels. In this paper, we propose the size of labeled data is very small. Our method employs an innovative data programming alike method to automatically generate probabilistic labels for unlabeled data.
SSL method that is effective and robust even when
2.3 Data Programming
Data programming [10] weak supervision paradigm proposed to infer correct labels based on the consensus among noisy labels from labeling functions (LFs), which are modules embedded with decision-making processes for generating labels programmatically. Following the DP paradigm, Snorkel [12] and Snuba [40] were proposed as rapid training data creation systems. Their LFs
built with various weak supervision sources, like pattern regexes, heuristics, and external knowledge base etc. Recently, more works are reported in the literature [ 13 21
41H48) Among
them, 11 13 14,
45 47 focus on the adaption of label model in DP. For example, [21] aims to reduce the computational cost and proposes glosed-former solution for training the label model.
43 heavily rely on the pretrained models. 41 combines crowdsoureing, data augmentation, and DP create weak labels for image classification. [15] presents a novel view for resolving infrequent data in scene graph prediction training datasets via image-agnostic features in LFs. However, all these methods cannot be directly applied to training models from scratch with a small number of labeled samples. Thus, in this paper we extend DP by exploring both MCL and SSL generate arbitrary labeling functions.
16
43 apply DP to computer vision. Concretely, [6
3 Method
For a C-class SSL classificcaii problem, assume that all training data X are divided into labeled data X, and unlabeled X,, and test data are denoted as X,, Following the notation in 36), {xz, xx") E X₁ are the paired labeled samples with labels E {1,..., C), and {zu, x , unlabeled samples. Here, Iy and xu represent the raw images without any transformations. at" x and r. are the images based on the weak and strong augmentation strategies, respectively. In this paper, weak augmentation uses a standard flip-and-shift strategy, and strong augmentation uses the RandAugment 44) strategy with Cutout [50] augmentation operation.
X, are the triple
3.1 Background
FixMatch. In the FixMatch 4 algorithm, apart from basic cross-entropy on labeled samples, consistency regularization with pseudo labels on unlabeled samples is represented as:
PM (x x (maz(p(v/z=") 2 e) H p(yl=?)),
(I)
3

---

mymonoyy deer)
g): monkey
LP fy-thore,, dog)
gh: abstain
CNN
LPa: Tynicat,, dog, horse]
gt cat
Label Model
End Model
Li4i v, (mmnkey))
9" monkey
Step
Noiay Labe's
Step
Probaabiisttcc Labels
Step
Figure 2: Framework of the DP-SSL method with four LFs.
where H is the cross-entropy, is the pre-defined threshold, p represents the output probability of the model, and Dü = arg max (p()l=))))))) is the pscudo label from the weakly augmented predictions.
Multiple Choice Learning. Stochastic Multiple Choice Learning (sMCL) [23] aims to specialize each individual model subset of data, by minimizing the loss as follows:
CsMCL (1l, v))
min (3n, PM(y)))))) kE(l....)
where pı is the output probability of the k-th model. For a training sample (xı, v1). sMCL feeds the data all K models but only chooses the most accurate model to do pack-propagation. Consequently, each model performs better on some classes than the other models, i.e., each model becomes specialist on some particular classes.
3.2 Framework
Fig.1 shows the framework our DP-SSL. method, which works in three major steps as follows:
Step 1. We employ an MCL based approach to automatically generate K LFs from scratch in an SSL. style. Here, each LF is trained on subset of C classes in the training set based on MCL. shown in Fig the 2nd LF trained with samples of classes "horse" and "dog", and abstains from predicting when facing monkey images.
Step 2. A graphical model is developed as the label model to aggregate the noisy labels and produce probabilistic labels for unlabeled training data. The label model learned in an SSL manner with an additional regularizer.
Step 3. The end model is trained with both provided labels and probabilistic labels generated from Step 2. Finally, verify the performance of the end model the test data.
3.3 Labeling Function
In Step I of method, LFs are exploited to generate noisy labels for each unlabeled image. In previous DP works for computer vision, LFs are built via external imagc-agnostic knowledge [15) However, it is difficult to explicitly describe the rules of image
pretrained models [T6] classification. Instead, here we innovatively explore MCL and SSL for automatic LF generation.
42 4
As shown in Fig, 2 we share the same backbone (Wide ResNet [51] in this paper) to extract features of images for muluplee prediction heads (called LFs in this paper). To promote the diversity of LFs, we transform the features and feed each LF with different transformed features as follows:
HW
-addis(((()ee))
(Jli) cx).
K
1=1
In this paper, K the number of LFs, ʃ E HD denotes the feature map of input image before global average pooling of backbone, [bi E RD is the feature vector at the spatial position j of ex E DR the learnable clustering center of the k-th LF, 3e is the learnable variable of the k-th cluster, dis(A, B) represents the distance between A and B. Thus, fx corresponds to the feature fed

---

to the k-th LF, and describes the k-th aggregated pattern of f among the K centers, it can also be considered as learnable weighted spatial pooling for feature f. Then, supposing Fk. the classifier in the k-th LF,, which would output the probability F&(Ja) as prediction. For clarity, in the following we denote p& (v/z) == Fi((ft) of the k-th LF with the input image x.
As depicted in [23]. the classifiers lack diversity of prediction even trained with different protocols. Therefore, we adopt MCL to assign subset of labeled data for each classifier automatically to improve diversity. However, it is intuitive to observe that in Eq. ( each category can only be assigned to one LF, and no consensus can be exploited. Therefore, we increase the proportion of selected models in Eq. 2 to do back proppppapagt,onn which is formulated as
p-K
CNOL(xT) (x)
min MC(H(.DD P [M|=p
Mr
where My indicates the k'thh element the set MM, and E 1] is a designed parameter to represent the ratio of specialist LFs. When Lo- KJ is equal to I, Eq. (4) becomes the traditional MCL in Eq. (2). In contrast, if Lp- KJ is equal to K, it deteriorates to the basic ensemble learning, where all K classifiers are trained with the same data.
Based on MCL, each LF a specialist for some classes, so can get high accuracy for samples in these classes. While for samples from other classes not specialized by the LF, it fails to predict due to over-confidence. Thus, we take only the probabilities of specialized categories as predictions, and allow each IF abstain from some samples in the dataset. Formally,
denote '0' as the abstention label, and the specialized category set of the k-th LF as тк { To [ell Then, the output label the k-th LF d² satisfies &² E {0} Uk. e.g., the output of the 1st LF in Fig. is among "monkey", "deer" and "abstention" because its specialized category set t {monkey, deer). Then, we denote the probability over the specialized category set 7. and "abstention" " option as ph (ylx", where Pk (3(2)) E IR sul+1. The objective function over labeled samples with abstention option is
Liaa 91)
Σ (1(1 TT) (((ya))))))))) - 1(y1 $ (0,
(5)
Then, for the unlabeled training data, follow the settings in FixMatch 4. where unlabeled data are supervised by the pseudo labels gu.k of weak augmentation data x." Thus,
Lu (xt [ x
(max(px (v(xx)")) 2 e) (1 E Hx) (+())+())+
(max
(6)
10 & H0. (x
where ug.k arg max (v)=")). Specifically, only keep samples whose largest probability (including the abstention option) is above the predefined threshold e (0.95 in our paper), and train the model on the kept data with pseudo label g.r.k. Accordingly, the training in this step is to minimize the objective function as follows:
C(at" Mr, T x",) MCL. LMCL (a:" m)) + (= .n) + (x wi))
where MCL, fr and Pa are hyper-parameters, In our implementation, first set p MCL 1 and Ai Mr. = 0, then adjust A.MCT. to 0 and = p.. = 1 after the convergence of LMCLL.
Generally, in Step I, MCL is expected to generate specialized class for LFs, with which samples are more easily discriminated by SSL classifiers even there are a few labeled samples. Besides, the abstention option is for addressing the wer-confidence problem of samples from con-specialized sets.
3.4 Label Model
In Step 2 of our method, we utilize graphical model to specify single prediction by integrating noisy labels provided by L.Fs. For simplification, we assume that the K L.Fs are independent (as shown in Fig. 2 Then, suppose that ý (it,... .)T EK is the vectorized form of the
5

---

predictions from K LFs, the joint distribution of the label model can be described as:
K P = II (v.g" k=1
(8)
where Z is the normalizer of the joint distribution, è is the potential that couples the target y and noisy label 8². In this paper, we extend the dimension of parameters in label model to K to support multi-class classification. Set Cky exp((sy)) which is the exponent of parameters kyy- Now we are to construct the potential function Φ. Due to the specialized LFs, the potential ø should benefit the final prediction when noisy label agrees with the target. That is, should have p(y, 00) 1. Thus, we set & as 1 + Cky for this case. On the contrary, the potential ø should negatively impact the final prediction when noisy label conflicts with the target label in the specialized category set, i.e.,
we should have ply /²)) 1. Therefore, for this case we set e to 1/(1 + Eky). For the other cases, follows:
follow the design in
52 In summary, the potential é is defined
1 + Ckys
if y The of E The Jf
1/(1 + cky),
if The of E Thi 8 +
p(y, 5²*)
Cky
if $ The 8² E Thi 8 + y otherwise
1.
With the potential above, the normalizer Z of the joint distribution in Eq. (8) can be obtained by summarizing over and gh:
= Σ E Φ(v.0*)
yEy 1= (0)Ure
(10)
+ [TH| 1m-l_1 + L(y & rx) + +
Σ
(1yy E T2) + eky
1 + eku
yE
Then, the objective function of the label model can be expressed in an SSL manner as follows:
Clyi, Bw) = Σ
H P(y.)))+
Σ log Σ ((v.8.))+R(6.
(1I)
221
DEY!
labeled samples
unlaboled samples
where the first part is the cross-entropy loss, the second is the negative log marginal likelihood on the observed noisy labels Yu, and the third is regularizer. In method, the regularizer is utilized to guide the label model with statistical information (the accuracy of each LF). However, the accuracy of each LF noisy labels is unavailable, while the accuracy labeled training is almost 100% due to over-fitting. Thus, we have to estimate the accuracy of each LF with the observable noisy labels y, which will be presented in Sec. 3.5 After training, the label model produces probabilistic labels by computing the joint distribution in Eq. S with the noisy labels y.
3.5 Accuracy Estimation
Now, we formally describe our method for estimating the accuracy of LFs. We transform the multi- class problem into versverrus-all tasks. For the i-th (i E [1, ... C1) one-versus-al all task, we denote the unobserved ground ground-truth truth labels 2, {=1} } (2 +1 means y and 2 -1 represents # i), noisy labels of the k-th LF as E E (+1,0).
1
if 0f =
if
(12)
-1 otherwise.
Then, can write E[2 2.]
Biatz4| P(ot P(8/2=11
P( 1) - (1 P(E) -
PP 0))
(13)
2P(2 = zi) + P( = -1.

---

Assume that 2/ 1 2 22 for distinct and k. then
E 2[ 2 = B| 2 2 [
(14)
with the fact that 27 1. Eq. from the noisy labels of the j-th and k-th LFs, while [B| 2.] and [E[ remain be solved due to true label 2; unavailable. Next, we introduce a third labeling result from the I-th LF as g such that [B[ 2.].. [B[] can be solved by a triplet
E.. 2/ is observable, which can be derived
E[ 2 and Bi 2|] are observable. Then, |B[ 2.]1.
method as follows:
B +1]1 V | 211
| =]]l / | at1
B
(15)
[B[][=]]] / -
We can obtain the estimated accuracy of each LF by resolving the sign of E[ 21 Let a P( 2 * 0) be the estimated accuracy of the k-th LF on the i-th category. Therefore, the regularizer RRO, yn) can be formulated as
(.) = Σ Σ
at log Po 1 + + )001-)))(0(
log(1 - Po(# = 2 +0)
(16)
where Po = 24 &
* 0) can be computed in closed form by marginalizing over all the other variables in the model in Eq. (8) Details of Pg can be referred to Appendix A.3
3.6 End Model
In Step 3, probabilistic labels are used to train an end model under any network architecture. We utilize noise-aware empirical risk expectation the objective function to take annotation errors into account. Accordingly, the final objective function is as follows:
L( Ut, Fus TT)
Σ H(n. (ynnp(u))))) +
(17)
Zuu
labeled samples
unlaheled samples with probahilistic label
where p(y(az) and ply æ.) are the predicted distributions of and Bus i is the distribution produced by the label model in Sec. 3.4
Experiments
4.1 Implementation Details
augment data in weak (a standard flp-and-shift strategy) and strong forms (RandAugment 49 followed by Cutout (50) operation). and utilize a Wide ResNet as the end model for a fair comparison. In our framework. the batch size for labeled data and unlabeled data
In the training phase, we follow the settings of previous works 4 35 36
to 64 and 448, respectively. Besides,
the same hyperparameters (K = 50, 0.2, 0.95) for all datasets. We compare DP-SSL with
major existing methods on CIFAR-10 [53] CIFAR- 100 (53). SVHN [54] and STL-10 (55) We also
analyze the effect of annotation and conduct ablation study in Sec. 4.4 and Sec. 4.5 respectively. All
experiments are implemented Pytorch v..7 and conducted on 16 NVIDIA RTX3090s.
4.2 Datasets
CIFAR-10 and CIFAR-100 53 contain 50,000 training examples and 10,000 validation examples. All images are of 32x32 pixel size and fall in 10 100 classes, respectively.
SVHN (44 is digital image dataset that consists of 73,257, 26,032 and 531,131 samples in the train, test, and extra folders. It has the same image resolution and category number as CIFAR-10.
7

---

Table 1: Results of error rate on CIFAR-I0, 10, CIFAR- -100RR-00aaA and SVHN for different existing SSL methods
(I-Modee 22 Pseudo- 7 Mean Teacher [32, Mix.atch [1], UDA [4], ReMixMatch
35
FixMatch and USADTM 36 ) and our DP-SSL method.
CIFAR-10
CIFAR-100
SVHN
Method
40 lahels
250 lahels
4000 lahels
400 labels 2500 labels 10000 labels]
40 lahels
250 lahels
1000 lahels
-Model
54.26:t3.97
04.01+0.38
-57.25=0048
37.88+0.11
18.96:
0.54
Pseude-Labeligg
49.78:20.43
16.09:2:.28
57.3882::446
36.21:00199
20.21:::1099
9.94:0661
Mean Teacher
32.32.1.2.30
9.19.10.19
53.91:10.57
35.83.10.24
3.57.10.11
3.42:00.77
MixMatch
11.54
11.05:::0.66
6.42:4:0110
67.61
39.94:1:0337
28.11:0.33
42.55=
3.98::0.23
3.50-10.28
UDA
29.05=5..93
8.82++1008
4.88++0.18
59.28=+0888
33.13++0222
24.50=0.22
52.63-20.51
5,69:42.76
2.46+0.24
ReMixMatch
19.10=99..4
5,44::0.05
4.72++0.13
44.28=+2.06
27.43:t00311
23.03+00.56
3.34+:0.20
2.92:4:0448
2.65=00088
USADTM
9.54:t1.04
4.80:::.32
4.40:
13.36=#1.89
28:11
21.35:00111
3.01
2.11::0.65
1.96:20.05
FixMatch (RA)
13.81
5.07:0065
4.26::0.05
1.78
28.29:40011
22.60--0.11 12
3.9642177
3.
2.2840.111
FixMatch (CTA)
31.3933.35
5.07+0033
4.31
49.95=33001
28 04
23.18+00111
7.65+7.65
2.64400.4
2.36+0.119
DP-SSL (ours)
6.54::0.88
0.78::0226
0.23+:0.00
43.17+1.29
28.00+00.99
22.24::0.31
0.98
2.16::0.36
19990.188
Fully Supervised]
2.74
16.84
1.48
Table 2: Results of error rate STL-10.
STL-10
Method
1000 Labels
Method
1000 labels
Method
40 labels
250 labels
1000 labels
II -Model
26.23=0.82
UDA
7,66=0.56
USADTM
1.63-:1335
6.85=11009
4.01=00.59 4.97::042
Pscudo-Labeling
27.99:20.80
ReMixMatch
5.23=0.45
DP-SSL (ours)
9.32440.11
6,83+0.71
Meun Teacher
21.43:.2.39
FisMatch (RA)
1.50
Fully Supervised
1.48
MixMatchh
10.41:20.61
FixMatch (CTA)
5.17=00.3
STL-10 (55) is dataset for evaluating unsupervised and semi-supervised learning. It consists of 5000 labeled images and 8000 validation samples of 96x96 size from 10 classes. Besides, there are 100.000 unlabeled images available, including odd samples.
4.3 Comparison with Existing SSL Methods
For fair comparison, we conduct experiments with the codebase of FixMatch and cite the results on CIFAR-10, CIFAR-100, SVHN and STL-10 from [4] We utilize the same network architecture Wide ResNet-28-2 for CIFAR-I0 and SVHN, WRN-28-8 for CIFAR-100, and WRN-37-2 for STL-10) and training protocol of FixMatch, such as optimizer and learning rate schedule. Unlabeled data are generated by the scripts in FixMatch. Results of DP-SSL and existing methods in Tab. [ and Tab. 2 are presented with the mean and standard deviation (STD) of accuracy on 5 pre-defined folds.
As shown in Tab. 1 our method achieves the best performance in most cases, especially when there are only 4 labeled samples per class. Specifically, our method achieves a 93.46% accuracy on CIFAR-10 with 4 labeled samples per category, which is 3.3% higher than that of USADTM the state-of-the-art method. Again on STL-110, our method surpasses USADTM and achieves the best performance when there are 4 and 25 labeled samples per class.
On CIFAR-100, our method performs the best for 400 labels case and the 2nd for 2500 and 10,000 labels cases. We also notice that DP-SSL has relatively large STDs for 2500 and 10,000 labels cases, which is due to the coarse accuracy estimation. In fact, even if triplet mean is adopted in estimation, the triplet selection in Eq. (15) still impacts accuracy estimation and regularizer lot, especially when Eia, 2 is close to 0 or sign recovery of E 2;] is wrong. Actually, there are some advanced approaches to unsupervised accuracy estimation (56 that can replace the naive triplet mean estimation. Ideally, if we can obtain the exact accuracy of each class i = P(at 2/| 1) and regularize it as R(0, 8 = Σ Σx K 8y log Paf z1 = 1) + (1 - 8 log(1 - Po = =|2² = 1)), we will get an end model with (27.92 + 0.23) % error rate for 2500 labeled samples.
Comparing with USADTM, our method does not perform well enough when more labeled data available. For USADTM, apart from the proxy label generator, unsupervised representation learning contributes a lot for its performance. As shown in the ablation study of (36). USADTM without unsupervised representation learning achieves around 5.73% and 4.99% error rate for 250 and 4000 labeled samples in CIFAR- 10, while our method DP-SSL obtains 4.78% and 4.23% error rate.
8

---

Table 3: The macro Precision/Recal/FI Score/Coverage of the annotated labels on CIFAR- 100 CIFAR- 100, and SVHN for our method and two typical existing label models.
CIFAR-10 10
CIFAR. 100
SVHN
Method
MMtries
|40 labels
250 labels 4000 labels
400 labels 2500 labels 10000 lahels 440 labels labels
1000 labels
Mujority Vote
Fl Score
85.96
94.23
95.77
49.97
69.81
76.03
90.86
95.38
96.14
FlyingSquid 21
Score
94.99
95.85
69.73
97.24
97.70
Precision
93.47
95.30
95.89
55.62
71.91
75.12
95.20
97.65
91.79
DP.SSL (ours)
Recall
93.82
95.33
95.91
56.86
72.01
78.35
96.78
97.64
97.94
Fl Score
93.61
95.19
95.90
54.42
71.89
76.36
95.95
97.59
97.81
Coverage
99.79
99.91
99.33
99.87
99.15
99.67
99.93
4.4 Analysis
Annotation performance. Intuitively, the holistic performance of the end model in our method highly depends on the quality of annotation results. Thus, we present the macro precision/recall/Fi score and coverage of the annotated labels of our method CIFAR-10, CIFAR-100, and SVHN in Tab. [] We can see that our method achieves over 99% coverage, which means that it produces probabilistic labels for almost all unlabeled data. Comparing to the results in [3). the label model with 40 labeled samples outperforms the proxy label generator, FixMatch and USADTM get 88. $8.511 % and 89.48% accuracy, respectively. Furthermore, our method achieves 97.36% accuracy for unlabeled data with the top-500 highest probabilities in each category. Meanwhile, we also present results of Majority Voting and FlyingSquid 21 in Tab. hased the noisy labels from Step 1 of our method for comparison. Majority Voting gets bad performance because the number of LFs triggered for different categories is not equal. For FlyingSquid, we implement with C' one-versus-all models to support multi-class tasks, and the large C' in CIFAR- 100 results in the worst performance.
Barely supervised learning. We conduct experiments to test the performance (accuracy and STD) of our method on CIFAR- 10 for some extreme cases (10, 20 and 30 labeled samples) to verify the effectiveness of our method. Here, we select the labeled data through the scripts of FixMatch with 5 different random seeds. As claimed in FitMatch, it reaches between 48.58 and 85.32% test accuracy with median of 64.28% for 10 labeled samples, while our method obtains accuracy from 61.32% to 83.7%. As for 20 and 30 labeled samples, our method gets (85.2) 29 + 3. 3.14)% and (89.81 + 5 accuracy respectively, which have much smaller STDs than that reported in (37]
4.5 Ablation Study
In DP-SSL, LFs and the label model are the core com- ponents to assign probabilistic labels for training the end model. Here, we check the effects of the following factors in the process of producing probabilistic labels by taking CIFAR-10 as the example. For ease of exposition, only the accuracy of predicted labels is presented in Tab.
Table 4: Annotation performance for different configurations on CIFAR-10 with 40 and 250 labels. K and p are set to 50 and 0.2 by default.
MCL. Feature transformation (FT) described in Eq. can be regarded as a weighted spatial pooling for extracted features. It is proposed to boost the diversity of generated LFs. We conduct comparative experiments for three con- figurations: 1) Exp/: w.o. MCL., 2) Exp2: MCL. w.O. FT. 3) Exp3: MCL w. FT. The results are presented in Tab. 4 It interesting to see that Expl is better than Exp2 but worse than Exp3. In fact, Expl is a simple ensemble model with a shared backbone, where each LF is trained independently and predicts the labels within C' categories. In Exp2, we observe that some classifiers have never been optimized in the training phase and thus have an empty specialized set when only few labeled samples per class are available. Moreover, the specialized sets of many LFs are duplicate, which incurs a negative impact on the performance. However, MCL with FT addresses the drawbacks and helpes our method obtain versatile LFs.
Experiments
40 labels 250 labels
Expl: MCL
92.46
95.02
Exp2: MCL w.o. FT
91.61
94,98
Exp3: w.
93.82
95.33
Exp4: Regularizer
93.19
94.94
Exp5: Regularizer
93.82
95,33
Hyperparameters. K and p are the number of LFss and the ratio of specialists in Eq. 4 In our ablation study, we focus on the variance of performance for different K and p with 40 labeled samples

---

Frror rate
7.0
0.1
6.5
1.0
number of LFS
Specialized ratio
(a)
Figure 3: Plots of ablation study on CIFAR-10 with 40 labels. (a) Varying the number of LFs K. (b) Varying the specialized ratio p. Here, the red dashed line indicates the error rate of DP-SSL with default hyperparameters.
on CIFAR-10. In Fig. 3a K=50 performs the best when 40 labeled samples are available. On the other hand, performance reaches the best when p=0.2 in Fig. 3b We present more results of and K in Appendix A.5
The regularizer is proposed impose a global guidance and improve the robustness of the label model. As shown in Tab. the regularizer does boost the accuracy, especially when facing less labeled samples. Besides, mentioned in Sec. 4.3] the high-quality guidance of the regularizer also reduces the label model's ' performance variance, thus improves its robustness.
Regularizer.
Conclusion
In this paper, explore the data programming idea to boost SSL when only small number of labeled samples available by providing more accurate labels for unlabeled data. To this end, we propose SSL method DP-SSL that employs innovative DP mechanism automatically generate labeling functions. To make the labeling functions diverse and specialized, a multiple choice learning based approach is developed. Furthermore, we design effective label model by incorporating novel potential and a regularizer with estimated accuracy. With this model, probabilistic labels are inferred by resolving the conflict and overlap among noisy labels from the labeling functions. Finally, an end model is trained under the supervision of the probabilistic labels. Extensive experiments show that DP-SSL can produce high-quality probabilistic labels, and outperforms the existing methods to achieve new SOTA, especially when only small number of labeled samples are available.
Limitations of This Work
In this work, coarse accuracy estimation the statistic information guide the label model for simplicity. As described in Sec. 3.5 we estimate the accuracy P = 2, t + 0), rather than class-wise accuracy ( = 2, & = I). Besides, we do not consider the dependency between LFs
and directly assume they are independent.
Acknowledgments and Disclosure of Funding
This work was supported by Alibaba Group through Alibaba Innovative Research Program, Shuigeng Zhou was also partially supported by Science and Technology Commission of Shanghai Municipality Project (No. 1951 1120700), and Shanghai Artificial Intelligence Innovation and Development Projects funded by Shanghai Municipal Commission of Economy and Informatization.
References
[1] M. Gao, Z. Zhang. G. Yu. S. Ó. Ank, L. S. Davis, and T. Pfister. "Consistency-bast semi- aani-supervised active learning: Towards minimizing labeling cost," in ECCV. Springer, 2020. pp. 510-526 526.
10

---

[2] C. Vondrick. D. Patterson, and D. Ramanan, "Efficiently scaling up crowdsourced video annotation." IJCV, vol. 101, 1. pp. 184-204, 2013.
[3] Y. Yao, A. Zhang. X. Han, M. Li. C. Weber. Z. Liu, S. Wermter, and M. Sun. "Visual distant supervision for scene graph generation." arXiv preprint 2ii::2003113663 2021.
[4] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang. H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, "Fixmatch: Simplifying semi-supervised learning with consistency and confidence," in NewrlPS, vol. 33, 2020.
[5] M. Oquab, L. Botton, Laptev, and J. Sivie, "Is object localization for weee?-veally-supervisee learning with convolutional neural networks," " in CVPR, 2015, pp. 685-694.
[6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations." in ICML. PMLR. 2020, pp. 1597-1607.
[7] D.-H. Lee et al., "Pseudo-label: The simple and efficient semi-supervisee learning method for deep neural networks," in Workshop on ICML, vol. 3, no. 2, 2013, p. 896.
[8] M. Sajadi,, M. Javanmardi, and T. Tasdizen, "Regularization with stochastic transformations and perturba- tions for deep semi-supervised learning," in NeurlPS, 2016, pp. 1163-1171.
[9] B. Zoph, G. Ghiasi T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. Le, "Rethinking pre-training and self-training," in NewrlPS, vol. 33, 2020.
[10] A. J. Ratner, C. De Sa, S. Wu, D. Selsam, and C. Ré, "Data programming: Creating large training sets, quickly," in NeurlPS, vol. 29, NIH Public Access, 2016, pp. 3567-3575
[11] A. Awasthi, S. Ghosh, R. Goyal, and Sarawagi, "Learning from rules generalizing labeled exemplars," ICLR, 2020.
[12] A. J. Ratner, S. H. Bach, H. R. Ehrenberg, and C. Ré, "Snorkel: Fast training set generation for information extraction," in SIGMOD. ACM, 2017, pp. 1683-1686.
[13] A. Rainer, B. Hancock, J. Dunnmon, Sala, S. Pandey, and C. Ré, "Training complex models with multi-task weak supervision." in AAAI, vol. 33, no. 01. AAAI Press, 2019, pp. 4763 4771.
[14] O. Chatterjec, G. Ramakrishnan, and S. Sarawagi, "Data programming using continuous and quality- labeling functions," arXiv preprint arXiv: 191100960, 2019.
[15] V.S. Chen, P. Varma, R. Krishna, M. Bersstein C. Re, and I., Fei-Fei, "Scene graph prediction with limited labels," in ICCV, IEEF, 2019, pp. 2580-2590.
[16] S. Hooper, M. Wornow, H. S. Ying. H. Kellman, Peter andXue, F. Sala, C. Langlotz, and C. Ré, "Cut out the annotator, keep the cutout: better segmentation with weak supervision," ICLR, 2021.
[17] N. C. Garcia, S. A. Bargal, V. Ablavsky. P. Morerio, V. Murino, and S. Sclaroff, "Distillation multiple choice learning for multimodal action recognition," in WACV, 2021, pp. 2754-2763.
[18] J. E. Van Engelen and H. H. Hoos, "A survey semi-supervised learning," Machine Learning, vol. 109, no. 2, pp. 373-440, 2020.
[19] x. Yang, Z. Song. I. King, and Z. Xu, "A survey on deep semi-supervised learning." arXiv preprint arXiv:2103.005:: 2021.
[20] M. F. Chen, D. Y. Fu. F. Sala, S. Wu, R. T. Mullapudi. F. Poms, K. Fatahalian, and C. Ré, "Train and you'l 'Il miss it: Interactive model iteration with weak supervision and pre-trained embeddings." arXiv preprint arXiv:2006. 15168, 2020.
[21] D. Fu, M. Chen, F. Sala, S. Hooper, K. Fatahalian, and C. Ré, "Fast and three-rious: Speeding up weak supervision with triplet methods," in ICMI., vol. 119. PMLR, 2020, pp. 3280-3291.
[22] A. Guzean-Rivera, P. Kohli, D. Batra, and R. Rutenbar, "Efficiently enforcing diversity in multi-oulput structured prediction." " in Artificial Intelligence and Statistics. PMLR, 2014, pp. 284-292.
[23] S. Lee. S. Purushwalkam, M. Cogswell, V. Ranjan, D. J. Crandall, and D. Batra, "Stochastic multiple choice learning for training diverse deep ensembles." in NewrIPS, 2016, pp. 2119-2127.
[24] K. Lee, C. Hwang, K. Park, and J. Shin, "Confident multiple choice learning." in ICML, vol. 70. PMLR, 2017. pp. 2023..
11

---

[25] K. Tian, Y. Xu, S. Zhou, and J. Guan, "Versatile multiple choice learning and its application to vision computing," in CVPR. IEEE, 2019, pp. 6349-6357.
[26] J. Deng, W. Dong. R. Socher, L.J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," CVPR. IEEE, 2009, pp. 248 255.
[27] T.-Y. Lin, M. Maire, S. Belongie, Hays. Perona, D. Ramanan, Dollár, and C. L. Zitnick, "Microsoft coco: Common objects context," ECCV, vol. 8693. Springer, 2014, pp. 740-755.
[28] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, "The cityscapes dataset for semantic urban scene understanding." in CVPR, 2016, pp. 3213- 3223.
[29] S. Laine and T. Aila, "Temporal ensembling for semi-supervised learning," in ICLR, 2017.
[30] A. Rasmus, H. Valpola, M. Honkala, M. Berglund, and T. Raiko, "Semiisupervised learning with ladder networks," " in NeurlPS, 2015, pp. 3546-3554.
[31] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. Raffel, "Mixmatch: A holistic approach to semi-supervised learnin," " in NewrIPS, 2019, pp. 5050-5060.
[32] A. Tarvainen and H. Valpola, "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results," in NewrlPS, 2017, pp. 1195-1204.
[33] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, "Virtual adversarial training: a regularization method for supervised and semi-supervised learning," TPAMI, vol. 41, no. 8, pp. 1979-1993, 2019.
[34] Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, "Unsupervised data augmentation for consistency training," in NeurlPS, 2020.
[35] D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, "Remixmatch: supervisd learning with distribution alignment and augmentation anchoring," in ICLR, 2020.
[36] T. Han, Gao, Y. Yuan, and Q. Wang, "Unsupervised semantic aggregation and deformable template matching for cemi-supervised learning," in NeurlPS, 2020.
[37] J. Li, C. Xiong, and S. C. Hoi, "Comatch: Semi-supervised learning with contrastive graph regularization," in ICCV, 2021, pp. 9475-9484.
[38] Z. Hu, Z. Yang, X. Hu, and R. Nevatia, "Simple: Similar pscudo label exploitation for semi semi-supervised classification," in CVPR, 2021, pp. 15 099 15 108
[39] B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki, "Flexmatch; Boosting sumi-super learning with curriculum pseudo laheling." arXiv preprint arXiv:21 10.08263, 2021.
[40] P. Varma and C. Ré, "Snuba: automating weak supervision label training data," VLDB, vol. 12, no. 3. NIH Public Access, 2018, p. 223.
[41] G. Heo, Roh, S. Hwang, D. Lee, and S. E. Whang, "Inspector gadget: A data programming-based laheling system for industrial images," Proc. VLDB Endow, vol. 14, no. 1, pp. 28-36, 2020.
[42] A. Pal and V. N. Balasubramanian, "Adversarial data programming: Using gans to relax the bottleneck of curated labeled data," in CVPR. IEEE, 2018, pp. 1556-1565.
[43] N. Das, S. Chaba, R. Wu, S. Gandhi, D. H. Chau, and X. Chu, "Goggles: Automatic image laheling with affinity coding," in SIGMOD, 2020, pp. 1717-1732.
[44] B. Boecking, W. Neiswanger, E. Xing, and A. Dubrawski, "Interactive weak supervision: Learning useful heuristics for data labeling." ICLR, 2021.
[45] P. Varma, B. He, Bajaj. Banerjee, N. Khandwala, D. L. Rubin, and C. Ré, "Inferring generative model structure with static analysis," in NewrIPS, vol. 30. NIH Public Access, 2017, pp. 240-250.
[46] S. H. Bach, B. He. A. Ratner, and C. Ré, "Learning the structure of generative models without labeled data," in ICML, vol. 70. PMLR, 2017, pp. 273-282.
[47] P. Varma, F. Sala. A. He, A. Ratner, and C. Ré. "Learning dependency structures for weak supervision models." in ICML. PMLR, 2019, pp. 6418 6427.
12

---

[48] L. Zhang, J. Ding. Y. Xu. Y. Liu, and S. Zhou, "Weakly-supervised text classification based on keyword graph." arXiv preprini vrXiv:2110.02590) 2021.
[49] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, "Randaugment: Practical automated data augmentation with reduced search space," in Workshop CVPR, 2020. pp. 702-703.
[50] T. DeVries and G. W. Taylor, "Improved regularization of convolutional neural networks with cutout," arXiv preprint arXiv:/708.0m 04552, 2017.
[51] S. Zagoruyko and N. Komodakis, "Wide residual networks," in BMVC. BMVC, 2016.
[52] A. Ratner, S. H. Bach, H. Ehrenberg, J. Fries, S. Wu, and C. Ré, "Snorkel: Rapid training data creation with weak supervision." VLDB J., vol. 29, no. pp. 709-730, 2020.
[53] A. Krizhevsky, "Learning multiple layers of fcatures from tiny images," Master's 's thesis, University of Tront, 2009.
[54] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, "Reading digits in natural images with unsupervised feature learning," in Workshop NewrIPS, 2011.
[55] A. Coates, A. Ng. and H. Lec, "An analysis of single-layer networks in unsupervised feature learning," in Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 2011, pp. 215 223..
[56] A. Jaffe, B. Nadler, and Y. Kluger, "Estimating the accuracies of multiple classifiers without labeled data," in Artificial Intelligence and Statistics. PMLR, 2015, pp. 407-415.
[57] E. Platanios, H. Poon, T. M. Mitchell, and E. J. Horvitz, "Estimating accuracy from unlabeled data: A probabilistic logic approach," NewrIPS, vol. 30, pp. 4361-4370, 2017.
[58] P. A. Traganitis, A. Pages-Zamora, and G. B. Giannakis, "Blind multiclass ensemble classification," IEEE Transactions on Signal Processing, vol. 66, no. 18, pp. 4737-4752, 2018.
13


Reviews:
[{"review_id": "bnc-QQckWgT", "reviewer": "Reviewer_8Nkc", "summary": "This paper tackles semi-supervised learning with a few labeled samples per class available during training. To solve this problem, the authors propose a data programming (DP) scheme to generate probabilistic labels for unlabeled data. Specifically, a multiple-choice learning (MCL) approach is proposed to generate a number of diverse and specialized labeling functions (LFs), which produce soft targets that indicate the class distributions of the unlabeled data. The proposed method is evaluated on four image classification datasets including CIFAR10, CIFAR100, SVHN, STL-10 in comparison to existing state-of-the-art methods. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Method**\n\nThe proposed method is interesting; however, it is unclear why it is better than existing methods like FixMatch. Several concerns are summarized in the following. \n \n1. It is indicated that the label space are divided into multiple subsets and different classifiers (i.e. labeling functions (LFs)) are trained to predict only a subset of classes. However, it is unclear how the division is defined. How many LFs are required? How does the randomness in division of class space affects the results? Does each LF predict a similar or non-similar set of classes? Does human knowledge required to define the specialized class set for each LF?\n\n2. ​In Eq. (7), it is unclear why the potential function should be divided into 4 parts. Citing the paper [14] does not really give any insight about the underlying mechanism. It is suggested to analyze the design principles of this equation carefully.\n\n3. How are different specialized classifiers jointly optimized during training?\n\n4. The motivation of the accuracy estimation and the use of the regularizer in Eq. (13) is unclear. Please explain how the newly introduced loss term (Eq. (13) helps to regularize the model.\n\n5. At test time, how are the final predictions obtained through different specialized classifiers? \n\n\n**Experiments**\n\nThe experiments are well described. However, there are a couple of important comparisons and explanations missing.\n\n1. As the authors mention using multiple data augmentation strategies such as RandomAugment and CutOut. It is unclear how they contribute to the model. It is suggested to also analyze these strategies, and discuss whether the other competitors such as FixMatch and ReMixMatch use the same data augmentation techniques or not to ensure a fair comparison wrt the state-of-the-art.  \n\n2. The evaluation metrics change from the main comparison to the ablation study. It is suggested to explain why introducing different new metrics such as macro precision/recall/F1 score and coverage to evaluate the method in Section 4.4. Please explain how these metrics quantify the model performance. \n\n3. How are the number of LFs (K) and the ratio of specialists decided together? In the ablation study, it is indicated that one has to decide the number of LFs first, and the ratio of specialists second. How about the other way around?\n\n4. What do specialists mean? How do specialists define for the ablation study in Table 4?\n\n5. It is suggested to also compare to the recent method [a].\n\n[a] Unsupervised dataaugmentation for consistency training. NeurIPS, 2020", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although the paper includes a discussion of the limitations of the method, a broader discussion about the potential negative societal impacts is not given. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "VqxNWDLOFeg", "reviewer": "Reviewer_AJRp", "summary": "This paper proposed to use multiple-choice learning to generate label functions and add an abstention class for each label function. Then the soft labels are inferred and used to train the final semi-supervised model. Extensive experiments on typical semi-supervised benchmarks show improvements over FixMatch.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method first transforms the features then uses different labeling functions to get predictions, which are aggregated to obtain the final soft label. The final model is trained with the supervised loss and the cross-entropy loss between the predictions and the soft labels for the unlabeled data. Experiments are performed on typical SSL benchmarks, including CIFAR10, CIFAR100, SVHN, and STL. The proposed method is often the best, especially when there are fewer labels available. Ablation studies are also provided. Overall, the proposed method is reasonable and works well.\n\nQuestions:\n- Another baseline is to use an ensemble model plus self-training. I'm not sure if this is exactly the same as the Exp1 in Table 4, but you can average the predictions from different supervised models, then do self-training using these soft labels. \n- In the ablation studies, the ensemble baseline (Exp1) is already better than FixMatch, so is the benefit mostly coming from the feature transformation part?\n- Methods such as FixMatch only train one single model, but the proposed method requires two-stage training, where the first stage is to train labeling functions. Maybe compare with other self-training methods?\n- Does the method also work on datasets with larger image sizes such as ImageNet?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are discussed. There is no negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "MyK7iyzGs9z", "reviewer": "Reviewer_cBn7", "summary": "This paper focuses on the extreme case of semi-supervised learning where very few labeled samples are available (e.g. 40 for CIFAR-10). In this extreme case, pseudo-labels for self-training are noisy and unreliable. Therefore, the authors proposed to first learn a set of labeling functions (LFs) that only focus on classifying a subset of classes, and then aggregate the predictions from the LFs for generating more reliable pseudo-labels.\n\nIn the experiments, they demonstrate superior performance in the case of very few labeled data on CIFAR-10, CIFAR-100, SVHN, and STL-10. In the cases of more labeled data, the proposed method achieves comparable performance with the previous SoTA.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Despite its great performance in the setting of extremely few labeled data, the method is not clearly explained:\n\n  - Eq. 1 is introduced in Sec. 3.2 but has never been used in the rest of Sec. 3.\n  - It is unclear which arguments or parameters the proposed loss functions are optimized with respect to.\n  - $\\theta$ at L188 is undefined\n  - In the first term of Eq.9, P(y, $\\tilde{y}_l$) is a joint distribution of y and $\\tilde{y}_l$, which is expected to be CxK. How is loss computed with a target $y_l$ of dimension C?\n  - In the second term of Eq. 9, the purpose for minimizing the negative log marginal likelihood is not explained.\n  - How to train these LFs? Is each step trained sequentially with models trained in previous steps fixed? Or are they all trained jointly?\n  - Method like FixMatch uses a confidence score to filter out unconfident pseudo-labels. Does the proposed method use ALL pseudo-labels without any confidence thresholding?\n\n- How to optimize Eq.2? According to the reviewer's understanding, gradient descent is not the best option for constrained optimization. However, it looks like Eq.2 is optimized jointly with other losses in Eq.5. Moreover, $u^k$ are discrete variables, which are not optimizable by gradient descent. Again, without knowing which arguments or parameters the loss is optimized with respect to, it's hard to understand how the proposed method can automatically learn a good set of LFs by optimizing Eq.2.\n\n- More analyses are expected for the learned LFs. For example, the authors claim that \"each LF is a specialist for some classes, so it can get high accuracy for samples in these classes\" at L163. Perhaps it's a good idea to show what are the learned LFs and how's their performance. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- In Tab.1 the proposed method does not perform much better on CIFAR-100. Does that mean the proposed method does not generalized well to larger datasets or datasets with more classes?\n\n- In SSL, the trained model tends to bias toward the selected labeled data. In the case of extremely few labeled data, the bias introduced by these few labeled data is magnified.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "HfaxsKUPR6Y", "reviewer": "Reviewer_YU4M", "summary": "The paper proposes DP-SSL to address semi-supervised learning by generating probabilistic labels for unlabeled samples with a newly designed data programming (DP), multiple-choice learning (MCL). \n\nThe proposed algorithm requires no initial labeling functions (LFs) from human experts and generates LFs from scratch automatically. \n\nEmpirical results show that the proposed algorithm achieves higher accuracy with less labeled data on several semi-supervised learning benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nThe paper relaxes the requirement of human experts to provide initial LFs.\n\nThe paper is well organized with a fluent clarification of the approach. I'm not an expert in the semi-supervised learning area but can still understand the paper as a general machine learning researcher.  \n\nThe paper compares the proposed approach with the state-of-the-art semi-supervised learning approaches, which shows the convincing performance of the proposed approach.\n\nConstraint:\n\nEquation (3) also minimizes the classification error between the ground-truth label and prediction, which may include the function of Equation (2). Could authors explain why Equation (2) is still needed in Equation (5) given Equation (3).\n\nThe paper requires multiple stages to train the model, which may need much time to converge. I recommend the authors to draw the training curve for the paper.\n\nMinor points:\n\nLine 89. '.' before 'which' should be ','.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No obvious social impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach to semi-supervised learning that trains a number of classification heads with different subset of classes, as well as a label model that aggregates the prediction of these classification heads. This approach seems to work well in a self-training setting especially when only few labeled examples are available. Overall, the paper is successful in building a very competitive system for semi-supervised learning (SSL) and it achieves SOTA on several SSL benchmarks. That said, the method has many components and hyper-parameters and it is not very clear where exactly the gains come from. Nevertheless, all of the reviewers have rated the paper slightly above acceptance bar after useful feedback from the authors, and all reviewers are happy to see the paper published. One of the paper's contributions is uniting existing work on data programming and SSL, and I believe, the ideas discussed in this paper can benefit the research community and help develop even better and simpler SSL algorithms. Hence, I recommend acceptance as a poster. Many comments about clarity and ablations have been raised by reviewers. I encourage the authors to address those comments in the camera ready version.
Decision: Accept (Poster)