DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples
Authors: Yi Xu
Jiandong Ding
Lu Zhang
Shuigeng Zhou
Venue: NeurIPS 2021 Poster (2021)
Keywords: semi-supervised learning
data programming
classification

Abstract: The scarcity of labeled data is a critical obstacle to deep learning. Semi-supervised learning (SSL) provides a promising way to leverage unlabeled data by pseudo labels. However, when the size of labeled data is very small (say a few labeled samples per class), SSL performs poorly and unstably, possibly due to the low quality of learned pseudo labels. In this paper, we propose a new SSL method called DP-SSL that adopts an innovative data programming (DP) scheme to generate probabilistic labels for unlabeled data. Different from existing DP methods that rely on human experts to provide initial labeling functions (LFs), we develop a multiple-choice learning~(MCL) based approach to automatically generate LFs from scratch in SSL style. With the noisy labels produced by the LFs, we design a label model to resolve the conflict and overlap among the noisy labels, and finally infer probabilistic labels for unlabeled samples. Extensive experiments on four standard SSL benchmarks show that DP-SSL can provide reliable labels for unlabeled data and achieve better classification performance on test sets than existing SSL methods, especially when only a small number of labeled samples are available. Concretely, for CIFAR-10 with only 40 labeled samples, DP-SSL achieves 93.82% annotation accuracy on unlabeled data and 93.46% classification accuracy on test data, which are higher than the SOTA results.

DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples
Yi Xu
Jiandong Ding Lu Zhang' Shuigeng Zhou
"hhnnhhai Key Lab of Intelligent Information Processing, and School of Computer Science, Fudan University, China
Alliaba Group
fyxu17, jdding, 1 1_zhang19, sgzhou)@ffudan eew..
Abstract
The scarcity of labeled data is critical obstacle to deep learning. Semi-supervised learning (SSL) provides promising way to leverage unlabeled data by pseudo labels. However, when the size of labeled data is very small (say a few labeled samples per class), SSL performs poorly and unstably, possibly due to the low quality of learned pseudo labels. In this paper, we propose a new SSL method called DP-SSL. that adopts an innovative data programming (DP) scheme to generate probabilistic labels for unlabeled data. Different from existing DP methods that rely on human experts to provide initial labeling functions (LFs), we develop a multiple-choice learning (MCL) based approach to automatically generate LFs from scratch in SSL style. With the noisy labels produced by the LFs, we design label model to resolve the conflict and overlap among the noisy labels, and finally infer probabilistic labels for unlabeled samples. Extensive experiments four standard SSL benchmarks show that DP-SSL can provide reliable labels for unlabeled data and achieve better classification performance on test sets than existing SSL. methods, especially when only a small number of labeled samples are available. Concretely, for CIFAR-10 with only 40 labeled samples, DP-SSL. achieves 93.82% annotation accuracy on unlabeled data and 93.46% classification accuracy on test data, which are higher than the SOTA results.
Introduction
The de-facto approaches to deep learning achieve phenomenal success with the release of huge labeled datasets. However, large manually-labeke datasets are time-consuming and expensive to acquire, especially when expert labelers are required. Nowadays, many techniques are proposed to alleviate the burden of manual labeling and help to train models from scratch, such as active learning [.]. crowd-labeling (2). distant supervision [3) semi [4]weak -5/sselfsuperrvision [6]. Among them, semi- suni-sseereieed learning (SSL) is one of the most popular techniques to cope with the scarcity of labeled data. Two major strategies of SSL are pseudo labels (7) and consistency regularization (8). Pseudo labels (also called self-training [9] utilize a model's predictions as the labels to train the model again, while consistency of regularization forces model to make the same prediction under different transformations, However, when the size of labeled data is small, SSL performance degrades drastically in both accuracy and robustness. Fig. shows the change of prediction error rate with the number of labeled samples of CIFAR-10. When the number of labeled samples reduces from 250 40, error rates of major existing SSL methods increase from 4.74% (USADTM) 36.49% (MixMatch). One possible reason of performance deterioration is the quality degradation of learnt pseudo labels when labeled data size small. Therefore, in this paper we address this problem by
"Corresponinng author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

developing sophisticated labeling techniques for unlabeled data to boost SSL even when the number of labeled samples is very small (e.g. a few labeled samples per class).
Recently, data programming (DP) was proposed as a new paradigm of weak supervision [T]. In DP, human experts are required to transform the decision-making process into a series of small functions (called labeling functions, abbre- viated as LFs), thus data can be labeled programmatically Besides, label model is applied to determining the correct labels based on consensus from the noisy and conflicting labels assigned by the LFs. Such paradigm achieves considerable success in NLP tasks [TT+14]. In addition, DP has also been applied to computer vision tasks [15] 16] However, current DP methods require human experts to provide initial L.Fs, which is cone-conuuinng and expen- sive, and it is not easy to guarantee the quality of LFs. Furthermore, LFs specifically defined for one task usually cannot be re-used for other tasks.
MaMMaich
HeMeMMetth
USADTM
DP SSL
Eror mto
Number labeled camples
Figure 1: Error rate us. #labeled samples (CIFAR-10). Results of existing meth- ods are from the original papers. When only 40 labeled samples are given, all existing SSL methods are substantially degraded and more unstably, while our method is still effective and robust.
In this paper, we propose new SSL method called DP- SSL that is effective and robust even when the number of labeled samples is very small. In DP-SSL, an inno- vative data programming (DP) scheme is developed to generate probabilistic labels for unlabeled data. Different from existing DP methods, we develop multiple-choice learning (MCL) based approach to automatically generate LFs from scratch in SSL style. To remedy the over-confidence problem with existing MCL methods, we assign an additional option as abstention for each LF. After that, we design label model to resolve the conflict and overlap among the noisy labels generated by LF's, and infer probabilistic label for each unlabeled sample. Finally, the probabilistic labels are used to train the end model for classifying unlabcled data. Our experiments validate the effectiveness and advantage of DP-SSL. As shown in Fig. I DP-SSL performs best, and only 1.76% increase of error rate when the number of labeled samples decreases from 250 to 40 in CIFAR-10.
Note that the pseudo labels used in existing SSL methods are quite different from the probabilistic labels in DP-SSL, which may explain the advantage of DP-SSL over existing SSL methods. On the one hand, pseudo labels are "hard" labels that indicate an unlabeled sample belonging to a certain class or not, while probabilistic labels are "soft" labels that indicate the class distributions of unlabeled samples. Obviously, the latter should be more flexible and robust. On the other hand, pseudo labels actually generated by single model for all unlabeled samples, while probabilistic labels are generated from a number of diverse and specialized LFs (due to the MCL mechanism), which makes the latter more powerful in generalization as a whole.
In summary, the contributions of this paper are as follows: 1) We propose a new SSL method DP-SSL that employs an innovative data programming method to generate probabilistic labels for unlabeled data, which makes DP-SSL effective and robust even when there are only a few labeled samples per class. 2) We develop a multiple choice learning based approach to automatically generate diverse and specialized LFs from scratch for unlabeled data in SSL manner. 3) We design a label model with a novel potential and an unsupervised quality guidance regularizer to infer probabilistic labels from the noisy labels generated by LFs. 4) We conduct extensive experiments on four standard benchmarks, which show that DP-SSL outperforms the state-of-fe of-the-art art methods, especially when only small number of labeled samples are available, DP-SSL is still effective and robust.
2
Related Work
Here we briefly review the latest advances in multiple choice learning, semi-supervised learning, and data programming, which are related to our work. Detailed information is available in [17-21].

---

2.1 Multiple Choice Learning
Multiple choice learning (MCL) was proposed overeome the low diversity problem of models trained independently in ensemble learning. For example, stochastic multiple choice learning (23] is for training diverse deep ensemble models. However, crucial problem with MCL that each model tends to be overconfident, which results in poor final prediction. To solve this problem, [24] forces the predictions of con-specialized models to meet uniform distribution, so that the final decision summed over diverse outputs. [25] proposes an additional network to estimate the weight of each specialist's output. In this paper, we develop an improved MCL based scheme to automatically generate diverse and specialized labeling functions (LFs) from scratch in an SSL manner. These LFs are used to generate preliminary (usually noisy) labels for unlabeled data.
2.2 Semi-supervised Learning
Semi-supervised learning (SSL) has been extensively studied in image classification 26 object detection [27] and semantic segmentation [28]. Two popular SSL. strategies for image classification are pseudo labels [ and consistency regularization [8]. Pseudo-label methods generate artificial labels for some unlabeled images and then train the model with these artificial labels, while consistency regularization tries to obtain artificial distribution/label and applied it supervision signal with other augmentations/viev:. These two strategies have been adopted by a number recent SSL works g 39H39). For example, FixMatch [4] proposes simple combination of pseudo labels and consistency regularization. [36] employs unsupervised learning and clustering to determine the pscudo labels. In this paper, we propose the size of labeled data is very small. Our method employs an innovative data programming alike method to automatically generate probabilistic labels for unlabeled data.
SSL method that is effective and robust even when
2.3 Data Programming
Data programming [10] weak supervision paradigm proposed to infer correct labels based on the consensus among noisy labels from labeling functions (LFs), which are modules embedded with decision-making processes for generating labels programmatically. Following the DP paradigm, Snorkel [12] and Snuba [40] were proposed as rapid training data creation systems. Their LFs
built with various weak supervision sources, like pattern regexes, heuristics, and external knowledge base etc. Recently, more works are reported in the literature [ 13 21
41H48) Among
them, 11 13 14,
45 47 focus on the adaption of label model in DP. For example, [21] aims to reduce the computational cost and proposes glosed-former solution for training the label model.
43 heavily rely on the pretrained models. 41 combines crowdsoureing, data augmentation, and DP create weak labels for image classification. [15] presents a novel view for resolving infrequent data in scene graph prediction training datasets via image-agnostic features in LFs. However, all these methods cannot be directly applied to training models from scratch with a small number of labeled samples. Thus, in this paper we extend DP by exploring both MCL and SSL generate arbitrary labeling functions.
16
43 apply DP to computer vision. Concretely, [6
3 Method
For a C-class SSL classificcaii problem, assume that all training data X are divided into labeled data X, and unlabeled X,, and test data are denoted as X,, Following the notation in 36), {xz, xx") E X₁ are the paired labeled samples with labels E {1,..., C), and {zu, x , unlabeled samples. Here, Iy and xu represent the raw images without any transformations. at" x and r. are the images based on the weak and strong augmentation strategies, respectively. In this paper, weak augmentation uses a standard flip-and-shift strategy, and strong augmentation uses the RandAugment 44) strategy with Cutout [50] augmentation operation.
X, are the triple
3.1 Background
FixMatch. In the FixMatch 4 algorithm, apart from basic cross-entropy on labeled samples, consistency regularization with pseudo labels on unlabeled samples is represented as:
PM (x x (maz(p(v/z=") 2 e) H p(yl=?)),
(I)
3

---

mymonoyy deer)
g): monkey
LP fy-thore,, dog)
gh: abstain
CNN
LPa: Tynicat,, dog, horse]
gt cat
Label Model
End Model
Li4i v, (mmnkey))
9" monkey
Step
Noiay Labe's
Step
Probaabiisttcc Labels
Step
Figure 2: Framework of the DP-SSL method with four LFs.
where H is the cross-entropy, is the pre-defined threshold, p represents the output probability of the model, and Dü = arg max (p()l=))))))) is the pscudo label from the weakly augmented predictions.
Multiple Choice Learning. Stochastic Multiple Choice Learning (sMCL) [23] aims to specialize each individual model subset of data, by minimizing the loss as follows:
CsMCL (1l, v))
min (3n, PM(y)))))) kE(l....)
where pı is the output probability of the k-th model. For a training sample (xı, v1). sMCL feeds the data all K models but only chooses the most accurate model to do pack-propagation. Consequently, each model performs better on some classes than the other models, i.e., each model becomes specialist on some particular classes.
3.2 Framework
Fig.1 shows the framework our DP-SSL. method, which works in three major steps as follows:
Step 1. We employ an MCL based approach to automatically generate K LFs from scratch in an SSL. style. Here, each LF is trained on subset of C classes in the training set based on MCL. shown in Fig the 2nd LF trained with samples of classes "horse" and "dog", and abstains from predicting when facing monkey images.
Step 2. A graphical model is developed as the label model to aggregate the noisy labels and produce probabilistic labels for unlabeled training data. The label model learned in an SSL manner with an additional regularizer.
Step 3. The end model is trained with both provided labels and probabilistic labels generated from Step 2. Finally, verify the performance of the end model the test data.
3.3 Labeling Function
In Step I of method, LFs are exploited to generate noisy labels for each unlabeled image. In previous DP works for computer vision, LFs are built via external imagc-agnostic knowledge [15) However, it is difficult to explicitly describe the rules of image
pretrained models [T6] classification. Instead, here we innovatively explore MCL and SSL for automatic LF generation.
42 4
As shown in Fig, 2 we share the same backbone (Wide ResNet [51] in this paper) to extract features of images for muluplee prediction heads (called LFs in this paper). To promote the diversity of LFs, we transform the features and feed each LF with different transformed features as follows:
HW
-addis(((()ee))
(Jli) cx).
K
1=1
In this paper, K the number of LFs, ʃ E HD denotes the feature map of input image before global average pooling of backbone, [bi E RD is the feature vector at the spatial position j of ex E DR the learnable clustering center of the k-th LF, 3e is the learnable variable of the k-th cluster, dis(A, B) represents the distance between A and B. Thus, fx corresponds to the feature fed

---

to the k-th LF, and describes the k-th aggregated pattern of f among the K centers, it can also be considered as learnable weighted spatial pooling for feature f. Then, supposing Fk. the classifier in the k-th LF,, which would output the probability F&(Ja) as prediction. For clarity, in the following we denote p& (v/z) == Fi((ft) of the k-th LF with the input image x.
As depicted in [23]. the classifiers lack diversity of prediction even trained with different protocols. Therefore, we adopt MCL to assign subset of labeled data for each classifier automatically to improve diversity. However, it is intuitive to observe that in Eq. ( each category can only be assigned to one LF, and no consensus can be exploited. Therefore, we increase the proportion of selected models in Eq. 2 to do back proppppapagt,onn which is formulated as
p-K
CNOL(xT) (x)
min MC(H(.DD P [M|=p
Mr
where My indicates the k'thh element the set MM, and E 1] is a designed parameter to represent the ratio of specialist LFs. When Lo- KJ is equal to I, Eq. (4) becomes the traditional MCL in Eq. (2). In contrast, if Lp- KJ is equal to K, it deteriorates to the basic ensemble learning, where all K classifiers are trained with the same data.
Based on MCL, each LF a specialist for some classes, so can get high accuracy for samples in these classes. While for samples from other classes not specialized by the LF, it fails to predict due to over-confidence. Thus, we take only the probabilities of specialized categories as predictions, and allow each IF abstain from some samples in the dataset. Formally,
denote '0' as the abstention label, and the specialized category set of the k-th LF as тк { To [ell Then, the output label the k-th LF d² satisfies &² E {0} Uk. e.g., the output of the 1st LF in Fig. is among "monkey", "deer" and "abstention" because its specialized category set t {monkey, deer). Then, we denote the probability over the specialized category set 7. and "abstention" " option as ph (ylx", where Pk (3(2)) E IR sul+1. The objective function over labeled samples with abstention option is
Liaa 91)
Σ (1(1 TT) (((ya))))))))) - 1(y1 $ (0,
(5)
Then, for the unlabeled training data, follow the settings in FixMatch 4. where unlabeled data are supervised by the pseudo labels gu.k of weak augmentation data x." Thus,
Lu (xt [ x
(max(px (v(xx(")) 2 e) (1 E Hx) ()())+())+
(max
(6)
10 & H0. (x
where ug.k arg max (v)=")). Specifically, only keep samples whose largest probability (including the abstention option) is above the predefined threshold e (0.95 in our paper), and train the model on the kept data with pseudo label g.r.k. Accordingly, the training in this step is to minimize the objective function as follows:
C(at" Mr, T x",) MCL. LMCL (a:" m)) + (= .n) + (x wi))
where MCL, fr and Pa are hyper-parameters, In our implementation, first set p MCL 1 and Ai Mr. = 0, then adjust A.rCT. to 0 and = p.. = 1 after the convergence of LMCLL.
Generally, in Step I, MCL is expected to generate specialized class for LFs, with which samples are more easily discriminated by SSL classifiers even there are a few labeled samples. Besides, the abstention option is for addressing the wer-confidence problem of samples from con-specialized sets.
3.4 Label Model
In Step 2 of our method, we utilize graphical model to specify single prediction by integrating noisy labels provided by L.Fs. For simplification, we assume that the K L.Fs are independent (as shown in Fig. 2 Then, suppose that ý (it,... .)T EK is the vectorized form of the
5

---

predictions from K LFs, the joint distribution of the label model can be described as:
K P = II (v.g" k=1
(8)
where Z is the normalizer of the joint distribution, è is the potential that couples the target y and noisy label 8². In this paper, we extend the dimension of parameters in label model to K to support multi-class classification. Set Cky exp((sy)) which is the exponent of parameters kyy- Now we are to construct the potential function Φ. Due to the specialized LFs, the potential ø should benefit the final prediction when noisy label agrees with the target. That is, should have p(y, 00) 1. Thus, we set & as 1 + Cky for this case. On the contrary, the potential ø should negatively impact the final prediction when noisy label conflicts with the target label in the specialized category set, i.e.,
we should have ply /²)) 1. Therefore, for this case we set e to 1/(1 + Eky). For the other cases, follows:
follow the design in
52 In summary, the potential @ is defined
1 + Ckys
if y The of E The Jf
1/(1 + cky),
if The of E Thi 8 +
p(y, 5²*)
Cky
if $ The 8² E Thi 8 + y otherwise
1.
With the potential above, the normalizer Z of the joint distribution in Eq. (8) can be obtained by summarizing over and gh:
= Σ E Φ(v.0*)
yEy 1= (0)Ure
(10)
+ [TH| 1m-l_1 + L(y & rx) + +
Σ
(1yy E T2) + eky
1 + eku
yE
Then, the objective function of the label model can be expressed in an SSL manner as follows:
Clyi, Bw) = Σ
H P(y.)))+
Σ log Σ ((v.8.))+R(6.
(1I)
221
DEY!
labeled samples
unlaboled samples
where the first part is the cross-entropy loss, the second is the negative log marginal likelihood on the observed noisy labels Yu, and the third is regularizer. In method, the regularizer is utilized to guide the label model with statistical information (the accuracy of each LF). However, the accuracy of each LF noisy labels is unavailable, while the accuracy labeled training is almost 100% due to over-fitting. Thus, we have to estimate the accuracy of each LF with the observable noisy labels y, which will be presented in Sec. 3.5 After training, the label model produces probabilistic labels by computing the joint distribution in Eq. S with the noisy labels y.
3.5 Accuracy Estimation
Now, we formally describe our method for estimating the accuracy of LFs. We transform the multi- class problem into versverrus-all tasks. For the i-th (i E [1, ... C1) one-versus-al all task, we denote the unobserved ground ground-truth truth labels 2, {=1} } (2 +1 means y and 2 -1 represents # i), noisy labels of the k-th LF as E E (+1,0).
1
if 0f =
if
(12)
-1 otherwise.
Then, can write E[2 2.]
2|a P(ot P(8/2=11
P( 1) - (1 P(E) -
PP 0))
(13)
2P(2 = zi) + P( = -1.

---

Assume that 2/ 1 2 22 for distinct and k. then
E 2[ 2 = B| 2 2 [
(14)
with the fact that 27 1. Eq. from the noisy labels of the j-th and k-th LFs, while [B| 2.] and [E[ remain be solved due to true label 2; unavailable. Next, we introduce a third labeling result from the I-th LF as g such that [B[ 2.].. [B[] can be solved by a triplet
E 2/ is observable, which can be derived
E[ 2 and Bi 2|] are observable. Then, |B[ 2.]1.
method as follows:
B +1]1 V | 211
| =]]l / | at1
B
(15)
[B[][=]]] / -
We can obtain the estimated accuracy of each LF by resolving the sign of E[ 21 Let a P( 2 * 0) be the estimated accuracy of the k-th LF on the i-th category. Therefore, the regularizer RRO, yn) can be formulated as
(.) = Σ Σ
at log Po 1 + + )001-)))(0(
log(1 - Po(# = 2 +0)
(16)
where Po = 24 &
* 0) can be computed in closed form by marginalizing over all the other variables in the model in Eq. (8) Details of Pg can be referred to Appendix A.3
3.6 End Model
In Step 3, probabilistic labels are used to train an end model under any network architecture. We utilize noise-aware empirical risk expectation the objective function to take annotation errors into account. Accordingly, the final objective function is as follows:
L( Ut, Fus TT)
Σ H(n. (ynnp(u))))) +
(17)
Zuu
labeled samples
unlaheled samples with probahilistic label
where p(y(az) and ply æ.) are the predicted distributions of and Bus i is the distribution produced by the label model in Sec. 3.4
Experiments
4.1 Implementation Details
augment data in weak (a standard fli-and-shift strategy) and strong forms (RandAugment 49 followed by Cutout (50) operation). and utilize a Wide ResNet as the end model for a fair comparison. In our framework. the batch size for labeled data and unlabeled data
In the training phase, we follow the settings of previous works 4 35 36
to 64 and 448, respectively. Besides,
the same hyperparameters (K = 50, 0.2, 0.95) for all datasets. We compare DP-SSL with
major existing methods on CIFAR-10 [53] CIFAR- 100 (53). SVHN [54] and STL-10 (55) We also
analyze the effect of annotation and conduct ablation study in Sec. 4.4 and Sec. 4.5 respectively. All
experiments are implemented Pytorch v..7 and conducted on 16 NVIDIA RTX3090s.
4.2 Datasets
CIFAR-10 and CIFAR-100 53 contain 50,000 training examples and 10,000 validation examples. All images are of 32x32 pixel size and fall in 10 100 classes, respectively.
SVHN (44 is digital image dataset that consists of 73,257, 26,032 and 531,131 samples in the train, test, and extra folders. It has the same image resolution and category number as CIFAR-10.
7

---

Table 1: Results of error rate on CIFAR-I0, 10, CIFAR- -100RR-00aaA and SVHN for different existing SSL methods
(I-Modee 22 Pseudo- 7 Mean Teacher [32, MixMatch [1], UDA [4], ReMixMatch
35
FixMatch and USADTM 36 ) and our DP-SSL method.
CIFAR-10
CIFAR-100
SVHN
Method
40 lahels
250 lahels
4000 lahels
400 labels 2500 labels 10000 labels]
40 lahels
250 lahels
1000 lahels
-Model
54.26:t3.97
04.01+0.38
-57.25=0048
37.88+0.11
18.96:
0.54
Pseude-Labeligg
49.78:20.43
16.09:2:.28
57.3882::446
36.21:00199
20.21:::1099
9.94:0661
Mean Teacher
32.32.1.2.30
9.19.10.19
53.91:10.57
35.83.10.24
3.57.10.11
3.42:00.77
MixMatch
11.54
11.05:::0.66
6.42:4:0110
67.61
39.94:1:0337
28.11:0.33
42.55=
3.98::0.23
3.50-10.28
UDA
29.05=5..93
8.82++1008
4.88++0.18
59.28=+0888
33.13++022
24.50=0.22
52.63-20.51
5,69:42.76
2.46+0.24
ReMixMatch
19.10=99..4
5,44::0.05
4.72++0.13
44.28=+2.06
27.43:t00311
23.03+00.56
3.34+:0.20
2.92:4:0448
2.65=00088
USADTM
9.54:t1.04
4.80:::.32
4.40:
13.36=#1.89
28:11
21.35:00111
3.01
2.11::0.65
1.96:20.05
FixMatch (RA)
13.81
5.07:0065
4.26::0.05
1.78
28.29:40011
22.60--0.11 12
3.9642177
3.
2.2840.111
FixMatch (CTA)
31.3933.35
5.07+0033
4.31
49.95=33.01
28 04
23.18+00111
7.65+7.65
2.64400.4
2.36+0.119
DP-SSL (ours)
6.54::0.88
0.78::0226
0.23+:0.00
43.17+1.29
28.00+00.99
22.24::0.31
0.98
2.16::0.36
19990.188
Fully Supervised]
2.74
16.84
1.48
Table 2: Results of error rate STL-10.
STL-10
Method
1000 Labels
Method
1000 labels
Method
40 labels
250 labels
1000 labels
II -Model
26.23=0.82
UDA
7,66=0.56
USADTM
1.63-:1335
6.85=11009
4.01=00.59 4.97::042
Pscudo-Labeling
27.99:20.80
ReMixMatch
5.23=0.45
DP-SSL (ours)
9.32440.11
6,83+0.71
Meun Teacher
21.43:.2.39
FisMatch (RA)
1.50
Fully Supervised
1.48
MixMatchh
10.41:20.61
FixMatch (CTA)
5.17=00.3
STL-10 (55) is dataset for evaluating unsupervised and semi-supervised learning. It consists of 5000 labeled images and 8000 validation samples of 96x96 size from 10 classes. Besides, there are 100.000 unlabeled images available, including odd samples.
4.3 Comparison with Existing SSL Methods
For fair comparison, we conduct experiments with the codebase of FixMatch and cite the results on CIFAR-10, CIFAR-100, SVHN and STL-10 from [4] We utilize the same network architecture Wide ResNet-28-2 for CIFAR-I0 and SVHN, WRN-28-8 for CIFAR-100, and WRN-37-2 for STL-10) and training protocol of FixMatch, such as optimizer and learning rate schedule. Unlabeled data are generated by the scripts in FixMatch. Results of DP-SSL and existing methods in Tab. [ and Tab. 2 are presented with the mean and standard deviation (STD) of accuracy on 5 pre-defined folds.
As shown in Tab. 1 our method achieves the best performance in most cases, especially when there are only 4 labeled samples per class. Specifically, our method achieves a 93.46% accuracy on CIFAR-10 with 4 labeled samples per category, which is 3.3% higher than that of USADTM the state-of-the-art method. Again on STL-110, our method surpasses USADTM and achieves the best performance when there are 4 and 25 labeled samples per class.
On CIFAR-100, our method performs the best for 400 labels case and the 2nd for 2500 and 10,000 labels cases. We also notice that DP-SSL has relatively large STDs for 2500 and 10,000 labels cases, which is due to the coarse accuracy estimation. In fact, even if triplet mean is adopted in estimation, the triplet selection in Eq. (15) still impacts accuracy estimation and regularizer lot, especially when Eia, 2 is close to 0 or sign recovery of E 2;] is wrong. Actually, there are some advanced approaches to unsupervised accuracy estimation (56 that can replace the naive triplet mean estimation. Ideally, if we can obtain the exact accuracy of each class i = P(at 2/| 1) and regularize it as R(0, 8 = Σ Σx K 8y log Paf z1 = 1) + (1 - 8 log(1 - Po = =|2² = 1)), we will get an end model with (27.92 + 0.23) % error rate for 2500 labeled samples.
Comparing with USADTM, our method does not perform well enough when more labeled data available. For USADTM, apart from the proxy label generator, unsupervised representation learning contributes a lot for its performance. As shown in the ablation study of (36). USADTM without unsupervised representation learning achieves around 5.73% and 4.99% error rate for 250 and 4000 labeled samples in CIFAR- 10, while our method DP-SSL obtains 4.78% and 4.23% error rate.
8

---

Table 3: The macro Precision/Recal/FI Score/Coverage of the annotated labels on CIFAR- 100 CIFAR- 100, and SVHN for our method and two typical existing label models.
CIFAR-10 10
CIFAR. 100
SVHN
Method
MMtries
|40 labels
250 labels 4000 labels
400 labels 2500 labels 10000 lahels 440 labels labels
1000 labels
Mujority Vote
Fl Score
85.96
94.23
95.77
49.97
69.81
76.03
90.86
95.38
96.14
FlyingSquid 21
Score
94.99
95.85
69.73
97.24
97.70
Precision
93.47
95.30
95.89
55.62
71.91
75.12
95.20
97.65
91.79
DP-SSL (ours)
Recall
93.82
95.33
95.91
56.86
72.01
78.35
96.78
97.64
97.94
Fl Score
93.61
95.19
95.90
54.42
71.89
76.36
95.95
97.59
97.81
Coverage
99.79
99.91
99.33
99.87
99.15
99.67
99.93
4.4 Analysis
Annotation performance. Intuitively, the holistic performance of the end model in our method highly depends on the quality of annotation results. Thus, we present the macro precision/recall/Fi score and coverage of the annotated labels of our method CIFAR-10, CIFAR-100, and SVHN in Tab. [] We can see that our method achieves over 99% coverage, which means that it produces probabilistic labels for almost all unlabeled data. Comparing to the results in [3). the label model with 40 labeled samples outperforms the proxy label generator, FixMatch and USADTM get 88. $8.511 % and 89.48% accuracy, respectively. Furthermore, our method achieves 97.36% accuracy for unlabeled data with the top-500 highest probabilities in each category. Meanwhile, we also present results of Majority Voting and FlyingSquid 21 in Tab. hased the noisy labels from Step 1 of our method for comparison. Majority Voting gets bad performance because the number of LFs triggered for different categories is not equal. For FlyingSquid, we implement with C' one-versus-all models to support multi-class tasks, and the large C' in CIFAR- 100 results in the worst performance.
Barely supervised learning. We conduct experiments to test the performance (accuracy and STD) of our method on CIFAR- 10 for some extreme cases (10, 20 and 30 labeled samples) to verify the effectiveness of our method. Here, we select the labeled data through the scripts of FixMatch with 5 different random seeds. As claimed in FitMatch, it reaches between 48.58 and 85.32% test accuracy with median of 64.28% for 10 labeled samples, while our method obtains accuracy from 61.32% to 83.7%. As for 20 and 30 labeled samples, our method gets (85.2) 29 + 3. 3.14)% and (89.81 + 5 accuracy respectively, which have much smaller STDs than that reported in (37]
4.5 Ablation Study
In DP-SSL, LFs and the label model are the core com- ponents to assign probabilistic labels for training the end model. Here, we check the effects of the following factors in the process of producing probabilistic labels by taking CIFAR-10 as the example. For ease of exposition, only the accuracy of predicted labels is presented in Tab.
Table 4: Annotation performance for different configurations on CIFAR-10 with 40 and 250 labels. K and p are set to 50 and 0.2 by default.
MCL. Feature transformation (FT) described in Eq. can be regarded as a weighted spatial pooling for extracted features. It is proposed to boost the diversity of generated LFs. We conduct comparative experiments for three con- figurations: 1) Exp/: w.o. MCL., 2) Exp2: MCL. w.O. FT. 3) Exp3: MCL w. FT. The results are presented in Tab. 4 It interesting to see that Expl is better than Exp2 but worse than Exp3. In fact, Expl is a simple ensemble model with a shared backbone, where each LF is trained independently and predicts the labels within C' categories. In Exp2, we observe that some classifiers have never been optimized in the training phase and thus have an empty specialized set when only few labeled samples per class are available. Moreover, the specialized sets of many LFs are duplicate, which incurs a negative impact on the performance. However, MCL with FT addresses the drawbacks and helpes our method obtain versatile LFs.
Experiments
40 labels 250 labels
Expl: MCL
92.46
95.02
Exp2: MCL w.o. FT
91.61
94,98
Exp3: w.
93.82
95.33
Exp4: Regularizer
93.19
94.94
Exp5: Regularizer
93.82
95,33
Hyperparameters. K and p are the number of LFss and the ratio of specialists in Eq. 4 In our ablation study, we focus on the variance of performance for different K and p with 40 labeled samples

---

Frror rate
7.0
0.1
6.5
1.0
number of LFS
Specialized ratio
(a)
Figure 3: Plots of ablation study on CIFAR-10 with 40 labels. (a) Varying the number of LFs K. (b) Varying the specialized ratio p. Here, the red dashed line indicates the error rate of DP-SSL with default hyperparameters.
on CIFAR-10. In Fig. 3a K=50 performs the best when 40 labeled samples are available. On the other hand, performance reaches the best when p=0.2 in Fig. 3b We present more results of and K in Appendix A.5
The regularizer is proposed impose a global guidance and improve the robustness of the label model. As shown in Tab. the regularizer does boost the accuracy, especially when facing less labeled samples. Besides, mentioned in Sec. 4.3] the high-quality guidance of the regularizer also reduces the label model's ' performance variance, thus improves its robustness.
Regularizer.
Conclusion
In this paper, explore the data programming idea to boost SSL when only small number of labeled samples available by providing more accurate labels for unlabeled data. To this end, we propose SSL method DP-SSL that employs innovative DP mechanism automatically generate labeling functions. To make the labeling functions diverse and specialized, a multiple choice learning based approach is developed. Furthermore, we design effective label model by incorporating novel potential and a regularizer with estimated accuracy. With this model, probabilistic labels are inferred by resolving the conflict and overlap among noisy labels from the labeling functions. Finally, an end model is trained under the supervision of the probabilistic labels. Extensive experiments show that DP-SSL can produce high-quality probabilistic labels, and outperforms the existing methods to achieve new SOTA, especially when only small number of labeled samples are available.
Limitations of This Work
In this work, coarse accuracy estimation the statistic information guide the label model for simplicity. As described in Sec. 3.5 we estimate the accuracy P = 2, t + 0), rather than class-wise accuracy ( = 2, & = I). Besides, we do not consider the dependency between LFs
and directly assume they are independent.
Acknowledgments and Disclosure of Funding
This work was supported by Alibaba Group through Alibaba Innovative Research Program, Shuigeng Zhou was also partially supported by Science and Technology Commission of Shanghai Municipality Project (No. 1951 1120700), and Shanghai Artificial Intelligence Innovation and Development Projects funded by Shanghai Municipal Commission of Economy and Informatization.
References
[1] M. Gao, Z. Zhang. G. Yu. S. Ó. Ank, L. S. Davis, and T. Pfister. "Consistency-bast semi- aani-supervised active learning: Towards minimizing labeling cost," in ECCV. Springer, 2020. pp. 510-526 526.
10

---

[2] C. Vondrick. D. Patterson, and D. Ramanan, "Efficiently scaling up crowdsourced video annotation." IJCV, vol. 101, 1. pp. 184-204, 2013.
[3] Y. Yao, A. Zhang. X. Han, M. Li. C. Weber. Z. Liu, S. Wermter, and M. Sun. "Visual distant supervision for scene graph generation." arXiv preprint 2ii::2003113663 2021.
[4] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang. H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, "Fixmatch: Simplifying semi-supervised learning with consistency and confidence," in NewrlPS, vol. 33, 2020.
[5] M. Oquab, L. Botton, Laptev, and J. Sivie, "Is object localization for weee?-vealyy-supervisee learning with convolutional neural networks," " in CVPR, 2015, pp. 685-694.
[6] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations." in ICML. PMLR. 2020, pp. 1597-1607.
[7] D.-H. Lee et al., "Pseudo-label: The simple and efficient semi-supervisee learning method for deep neural networks," in Workshop on ICML, vol. 3, no. 2, 2013, p. 896.
[8] M. Sajadi,, M. Javanmardi, and T. Tasdizen, "Regularization with stochastic transformations and perturba- tions for deep semi-supervised learning," in NeurlPS, 2016, pp. 1163-1171.
[9] B. Zoph, G. Ghiasi T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. Le, "Rethinking pre-training and self-training," in NewrlPS, vol. 33, 2020.
[10] A. J. Ratner, C. De Sa, S. Wu, D. Selsam, and C. Ré, "Data programming: Creating large training sets, quickly," in NeurlPS, vol. 29, NIH Public Access, 2016, pp. 3567-3575
[11] A. Awasthi, S. Ghosh, R. Goyal, and Sarawagi, "Learning from rules generalizing labeled exemplars," ICLR, 2020.
[12] A. J. Ratner, S. H. Bach, H. R. Ehrenberg, and C. Ré, "Snorkel: Fast training set generation for information extraction," in SIGMOD. ACM, 2017, pp. 1683-1686.
[13] A. Rainer, B. Hancock, J. Dunnmon, Sala, S. Pandey, and C. Ré, "Training complex models with multi-task weak supervision." in AAAI, vol. 33, no. 01. AAAI Press, 2019, pp. 4763 4771.
[14] O. Chatterjec, G. Ramakrishnan, and S. Sarawagi, "Data programming using continuous and quality- labeling functions," arXiv preprint arXiv: 191100960, 2019.
[15] V.S. Chen, P. Varma, R. Krishna, M. Bersstein C. Re, and I., Fei-Fei, "Scene graph prediction with limited labels," in ICCV, IEEF, 2019, pp. 2580-2590.
[16] S. Hooper, M. Wornow, H. S. Ying. H. Kellman, Peter andXue, F. Sala, C. Langlotz, and C. Ré, "Cut out the annotator, keep the cutout: better segmentation with weak supervision," ICLR, 2021.
[17] N. C. Garcia, S. A. Bargal, V. Ablavsky. P. Morerio, V. Murino, and S. Sclaroff, "Distillation multiple choice learning for multimodal action recognition," in WACV, 2021, pp. 2754-2763.
[18] J. E. Van Engelen and H. H. Hoos, "A survey semi-supervised learning," Machine Learning, vol. 109, no. 2, pp. 373-440, 2020.
[19] x. Yang, Z. Song. I. King, and Z. Xu, "A survey on deep semi-supervised learning." arXiv preprint arXiv:2103.005:: 2021.
[20] M. F. Chen, D. Y. Fu. F. Sala, S. Wu, R. T. Mullapudi. F. Poms, K. Fatahalian, and C. Ré, "Train and you'l 'Il miss it: Interactive model iteration with weak supervision and pre-trained embeddings." arXiv preprint arXiv:2006. 15168, 2020.
[21] D. Fu, M. Chen, F. Sala, S. Hooper, K. Fatahalian, and C. Ré, "Fast and three-rious: Speeding up weak supervision with triplet methods," in ICMI., vol. 119. PMLR, 2020, pp. 3280-3291.
[22] A. Guzean-Rivera, P. Kohli, D. Batra, and R. Rutenbar, "Efficiently enforcing diversity in multi-oulput structured prediction." " in Artificial Intelligence and Statistics. PMLR, 2014, pp. 284-292.
[23] S. Lee. S. Purushwalkam, M. Cogswell, V. Ranjan, D. J. Crandall, and D. Batra, "Stochastic multiple choice learning for training diverse deep ensembles." in NewrIPS, 2016, pp. 2119-2127.
[24] K. Lee, C. Hwang, K. Park, and J. Shin, "Confident multiple choice learning." in ICML, vol. 70. PMLR, 2017. pp. 2023..
11

---

[25] K. Tian, Y. Xu, S. Zhou, and J. Guan, "Versatile multiple choice learning and its application to vision computing," in CVPR. IEEE, 2019, pp. 6349-6357.
[26] J. Deng, W. Dong. R. Socher, L.J. Li, K. Li, and L. Fei-Fei, "Imagenet: A large-scale hierarchical image database," CVPR. IEEE, 2009, pp. 248 255.
[27] T.-Y. Lin, M. Maire, S. Belongie, Hays. Perona, D. Ramanan, Dollár, and C. L. Zitnick, "Microsoft coco: Common objects context," ECCV, vol. 8693. Springer, 2014, pp. 740-755.
[28] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, "The cityscapes dataset for semantic urban scene understanding." in CVPR, 2016, pp. 3213- 3223.
[29] S. Laine and T. Aila, "Temporal ensembling for semi-supervised learning," in ICLR, 2017.
[30] A. Rasmus, H. Valpola, M. Honkala, M. Berglund, and T. Raiko, "Semiisupervised learning with ladder networks," " in NeurlPS, 2015, pp. 3546-3554.
[31] D. Berthelot, N. Carlini, I. Goodfellow, N. Papernot, A. Oliver, and C. Raffel, "Mixmatch: A holistic approach to semi-supervised learning" " in NewrIPS, 2019, pp. 5050-5060.
[32] A. Tarvainen and H. Valpola, "Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results," in NewrlPS, 2017, pp. 1195-1204.
[33] T. Miyato, S.-i. Maeda, M. Koyama, and S. Ishii, "Virtual adversarial training: a regularization method for supervised and semi-supervised learning," TPAMI, vol. 41, no. 8, pp. 1979-1993, 2019.
[34] Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le, "Unsupervised data augmentation for consistency training," in NeurlPS, 2020.
[35] D. Berthelot, N. Carlini, E. D. Cubuk, A. Kurakin, K. Sohn, H. Zhang, and C. Raffel, "Remixmatch: supervisd learning with distribution alignment and augmentation anchoring," in ICLR, 2020.
[36] T. Han, Gao, Y. Yuan, and Q. Wang, "Unsupervised semantic aggregation and deformable template matching for cemi-supervised learning," in NeurlPS, 2020.
[37] J. Li, C. Xiong, and S. C. Hoi, "Comatch: Semi-supervised learning with contrastive graph regularization," in ICCV, 2021, pp. 9475-9484.
[38] Z. Hu, Z. Yang, X. Hu, and R. Nevatia, "Simple: Similar pscudo label exploitation for semi semi-supervised classification," in CVPR, 2021, pp. 15 099 15 108
[39] B. Zhang, Y. Wang, W. Hou, H. Wu, J. Wang, M. Okumura, and T. Shinozaki, "Flexmatch; Boosting sumi-super learning with curriculum pseudo laheling." arXiv preprint arXiv:21 10.08263, 2021.
[40] P. Varma and C. Ré, "Snuba: automating weak supervision label training data," VLDB, vol. 12, no. 3. NIH Public Access, 2018, p. 223.
[41] G. Heo, Roh, S. Hwang, D. Lee, and S. E. Whang, "Inspector gadget: A data programming-based laheling system for industrial images," Proc. VLDB Endow, vol. 14, no. 1, pp. 28-36, 2020.
[42] A. Pal and V. N. Balasubramanian, "Adversarial data programming: Using gans to relax the bottleneck of curated labeled data," in CVPR. IEEE, 2018, pp. 1556-1565.
[43] N. Das, S. Chaba, R. Wu, S. Gandhi, D. H. Chau, and X. Chu, "Goggles: Automatic image laheling with affinity coding," in SIGMOD, 2020, pp. 1717-1732.
[44] B. Boecking, W. Neiswanger, E. Xing, and A. Dubrawski, "Interactive weak supervision: Learning useful heuristics for data labeling." ICLR, 2021.
[45] P. Varma, B. He, Bajaj. Banerjee, N. Khandwala, D. L. Rubin, and C. Ré, "Inferring generative model structure with static analysis," in NewrIPS, vol. 30. NIH Public Access, 2017, pp. 240-250.
[46] S. H. Bach, B. He. A. Ratner, and C. Ré, "Learning the structure of generative models without labeled data," in ICML, vol. 70. PMLR, 2017, pp. 273-282.
[47] P. Varma, F. Sala. A. He, A. Ratner, and C. Ré. "Learning dependency structures for weak supervision models." in ICML. PMLR, 2019, pp. 6418 6427.
12

---

[48] L. Zhang, J. Ding. Y. Xu. Y. Liu, and S. Zhou, "Weakly-supervised text classification based on keyword graph." arXiv preprini vrXiv:2110.02590) 2021.
[49] E. D. Cubuk, B. Zoph, J. Shlens, and Q. V. Le, "Randaugment: Practical automated data augmentation with reduced search space," in Workshop CVPR, 2020. pp. 702-703.
[50] T. DeVries and G. W. Taylor, "Improved regularization of convolutional neural networks with cutout," arXiv preprint arXiv:/708.0m 045520 2017.
[51] S. Zagoruyko and N. Komodakis, "Wide residual networks," in BMVC. BMVC, 2016.
[52] A. Ratner, S. H. Bach, H. Ehrenberg, J. Fries, S. Wu, and C. Ré, "Snorkel: Rapid training data creation with weak supervision." VLDB J., vol. 29, no. pp. 709-730, 2020.
[53] A. Krizhevsky, "Learning multiple layers of fcatures from tiny images," Master's 's thesis, University of Tront, 2009.
[54] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng, "Reading digits in natural images with unsupervised feature learning," in Workshop NewrIPS, 2011.
[55] A. Coates, A. Ng. and H. Lec, "An analysis of single-layer networks in unsupervised feature learning," in Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 2011, pp. 215 223..
[56] A. Jaffe, B. Nadler, and Y. Kluger, "Estimating the accuracies of multiple classifiers without labeled data," in Artificial Intelligence and Statistics. PMLR, 2015, pp. 407-415.
[57] E. Platanios, H. Poon, T. M. Mitchell, and E. J. Horvitz, "Estimating accuracy from unlabeled data: A probabilistic logic approach," NewrIPS, vol. 30, pp. 4361-4370, 2017.
[58] P. A. Traganitis, A. Pages-Zamora, and G. B. Giannakis, "Blind multiclass ensemble classification," IEEE Transactions on Signal Processing, vol. 66, no. 18, pp. 4737-4752, 2018.
13


Reviews:
[{"review_id": "pIYsKFSPMbh", "reviewer": "Reviewer_xTCd", "summary": "The paper presents a semi-supervised learning method called DP-SSL that focuses on semi-supervised learning tasks where the number of labeled samples is extremely limited. The paper utilizes a 3 step approach: 1) generate neural network label functions; 2) train a label model to generate probabilistic labels for the unlabeled samples; 3) train a final model using the labeled data and the unlabeled data with the generated probabilistic labels.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed DP-SSL method can be seen as a combination of SSL learning methods and data programming. From my perspective, the key contribution of the method is to automatically generate LFs by training neural nets with the proposed loss. The empirical performance looks promising specifically when the labeled data size is very limited.\n\nHere are some concerns/questions about the paper:\n1. The presentation needs improvement. The paper lacks details about the proposed method. For example:\n\n    a) it is not clear how \\tilde{f}_k in Eq (1) is used in later equations;\n\n    b) in Eq (5) and lines 177-178, a schedule of setting the loss hyperparameters is mentioned but it's not clear how such schedule is set in practice;\n\n    c) some steps utilize previous works (for example line 49), but I think the paper should add some descriptions or discussions about the general idea.\n\n\n2. I wonder why a final model training step is useful. If I understand it correctly, the final model training directly uses the probabilistic labels generated by the label modeling step and the training is similar to a supervised learning setting. If the labels from the label modeling step are already trustworthy, why not use them as predictions directly? If the final model training step is indeed necessary, then what's the reason that the final model can perform better if the generated annotations are relatively poorer?\n\n3. Compared to some of the previous SSL learning methods, the proposed one incurs more computational costs as DP-SSL requires training additional LF models. The training costs are not compared and reported. As the FL models can be seen as an ensemble of neural networks, the improvements in the performance may come from the increase of the model capacity.\n\n\nMinor comments:\nLine 259: ```\"40 labels\" should be \"400 labels\"\n\n## Post Rebuttal Update\nI thank the authors for the response. After reading the rebuttal and other reviewers' comments, I decide not to change my rating of the paper. The main concerns are 1) the presentation requires improvement, and the paper lacks important technical details; 2) the design of the method lacks intuitive explanation or thorough empirical justification; 3) the training costs are not reported.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed the limitations.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "jAEp0enoWhQ", "reviewer": "Reviewer_gFDp", "summary": "This paper describes an approach for semi-supervised learning by having a lot of models in an ensemble to produce a set of diverse predictions with a subset of classes. It aggregates these votes through a probabilistic labeling model and uses the smoothed labels for the final round of training with these pseudo labels. It achieves state-of-the-art results on several semi-supervised learning benchmarks.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n### Strengths\n- **Significance:** Obviously the paper achieves very strong results on semi-supervised benchmarks including CIFAR-10, CIFAR-100, SVHN, and STL-10.\n- **Originality:** Using ensemble models to perform pseudo-labelling is a well-motivated direction, and to my knowledge the proposed approach is original.\n\n-----------------------------------------------------------\n### Weaknesses\n- **Clarity:** Overall the math is not very clearly presented in this paper. Many notations are not explained.\n  - The form of the labeling function is not clear. After explaining what $\\tilde{f}_k$ is, we directly jump to $\\tilde{p}^{w}_k$.\n  - The MCL technique is not clearly described. It is unclear how $u$ is learned and updated.\n  - The description of accuracy estimation is a bit confusing and I am not sure I fully understand the techniques here.\n- **Overall design of the model:** Although the paper reports very strong results, I feel like there are lots of components going on and it’s more like a description of an engineering pipeline rather than some general principles. For example, why are there three loss functions instead of just the supervised + the unsupervised? Why does the model need to pretrain with loss 1 and then switch to loss 2+3? I have more comments below regarding the detailed design for LF, MCL, label model, etc. They all seem to make some sense if this is a description of a big engineering pipeline, but they are not the most simplistic variant to go for if one wants to seek for more scientific understanding.\n- **Design of labeling functions:** I don’t understand the design of equation 1 and what purpose it serves. Why not just feed each LF with a different projection of input features?\n- **Effect of MCL:** The effect of MCL is very marginal in the ablation studies. You get 1.36% boost on 40 labels and 0.31% boost on 250 labels. Even without MCL, the method can get state-of-the-art numbers in Table 1 (with 7.54% error rate on CIFAR-10 40 labels), but the paper conducts experiments on top of FixMatch. So it is unclear what contributes to the major gain between the original FixMatch and the variant without MCL, which is more significant than the gap between no MCL and the full model.\n- **Effect of abstention:** The paper introduces this abstention choice for MCL. But it is not necessary. If a model doesn’t have a class in the subset, then it just doesn’t participate in the voting process for that class. Why complicate the learning procedure by adding an unknown class, if the benchmark is only a fixed set of classes?\n- **Ablations on $K$ and $\\rho$:** For both of these hyperparameters, it would be better to plot a curve or table with equal intervals. An interval like 20-40-50-60-100 or 0.1-0.2-0.3-0.5-1.0 doesn’t make sense.\n- **Other ensemble baselines:** Instead of using MCL, one can just train different ensemble models with different random seed and random partition of the dataset, without having to optimize the partition variables. This is also not studied here. \n- **Effect of label model:** There is no ablation on the effect of the label model vs. just like taking the average or majority.\n- **Effect of accuracy estimation:** Why would you weight the log terms with the accuracy? And what if you don’t weigh the regularizer with accuracy?\n- **Effect of regularization:** Ablation table suggests that the regularization overall only has a tiny effect.\n- **Effect of probabilistic labels:** There is no ablation on whether to use the probabilistic labels or just taking the max and make it into a discrete one-hot label.\n\n-----------------------------------------------------------\n### Conclusion\nIn conclusion, despite the paper's empirical success, I consider it unfit for publishing at NeurIPS due to 1) the sheer complexity of many unverified components of model designs and 2) unclear paper writing, and I therefore recommend rejection as my initial rating.\n\n\nUpdate after rebuttal:\nI thank the author for a detailed response. I have read the response and it does address some of my concerns regarding the ablation of certain parameters. However, I retain my current rating recommendation since I think it would be hard to accept the paper without a major revision to clear up the writing and the concepts.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "b5uJt9W1SI", "reviewer": "Reviewer_ux7Q", "summary": "The paper proposes a method for semi-supervised learning (SSL) with a few seed labeled sample scenario. The method uses data programming [10] for the SSL set-up and cleans up the noisy label by the \"label model\". Using the cleaned label for the unlabeled data, the \"end model\" is trained for image classification. The proposed method outperforms prior arts especially in the low shot regime with large gain in CIFAR10 dataset.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- S1: The proposed method is based on simple intuitive motivation to clean up the noisy labels from the label functions (LFs).\n- S2: Noticeable gain in low shot regime in CIFAR10 dataset\n\nWeaknesses\n- W1: Lack of novelty. The method is a heuristic combination of existing data programming (DP) module with hand-crafted label clean-up model. Though the combination is well described, the combination lacks the novelty.\n- W2: Performance gain is overall marginal across many datasets. The interesting gain is in low sample regime (40 labels) in CIFAR10 dataset (in Table 1). In other low sample regimes (e.g., 40 labels in SVHN and STL-10), the proposed method does not shows compelling gains. Given all benchmarked datasets are not large scale, it is not clear that the proposed method improves the performance meaningfully - compared to USADTM (L268-272).\n- W3: Lack of study of heuristic design choices. Although Table 4 (in Sec. 4.5) has in-depth studies for K and $\\rho$, the $\\epsilon$ in Eq. 4 is not well motivated (or lack of sensitivity studies for it) and why Eq. (7) should be in the proposed form.\n- W4: Presentation is largely unclear. The notations in equations are defined but not well explained about their physical meanings (e.g., $\\tilde{q}_l^{w,k}$ in Eq. (3)). Thus it is difficult to follow why each equations make sense.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It has an limitation section but no discussion on potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "LiEvv4OCys", "reviewer": "Reviewer_ECpd", "summary": "This paper addresses the problem of semi-supervised learning (SSL). Specifically, it aims to improve the quality of the pseudo-labels for SSL when the number of labeled samples is very small. To achieve this the paper uses a data programming scheme, however instead of defining the labeling functions through human experts, they propose to learn several different classification heads in an SSL manner which subsequently serves as a labeling function. This is implemented using multiple-choice learning (MCL), a technique that is known to improve the diversity of deep ensemble models. The different classification heads are combined into a final classification output by ensembling them while taking into account their reliability. The experimental results show that the model performs slightly better compared to related work on popular SSL benchmarks.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\nThe paper does not make a theoretical contribution. It describes several theoretical results that were obtained in prior work and applies these to the problem of SSL. The technical novelty seems rather limited to me since all components of the proposed approach have been proposed in prior work. To my understanding, the novelty is given by the application of the components to SSL and the empirical effectiveness at SSL, which is on par with state-of-the-art methods.\n\nQuality:\nThe paper is technically sound and the methods are used appropriately. The overview of related work is also appropriate. The experimental evaluation is very sophisticated and conducted on a range of relevant datasets. Error bars are given in most experiments.\nThe main limitation of this paper is that it does not provide sufficient intuitive explanations of the proposed model. How does the MCL distribute the classes among the classification heads? Is this random? I did not find a detailed description of this process in Section 3.2. although it seems important for the model. Section 3.4. describes the estimation of the reliability of the individual classification heads, but it does not give an intuitive explanation of the process. Moreover, important technical details are not explained in the paper, and the reader is instead referred to related (L.154, 155)\nThe limitations of the proposed method are not described in much detail, and I recommend adding a more thorough discussion on error cases. I would, for example, also be interested in a discussion on the issue of dataset imbalance since the proposed method is working particularly well for a small number of samples. Does this suggest that it can handle data-imbalance issues better? If so, why?\n\nClarity:\nThe overall quality of the manuscript is satisfactory. While the language is very good and the experiments are described in detail, the methods part lacks intuition and technical details that would better support understanding the technical contribution of this paper (see comment in quality section).\n\nSignificance:\nThe problem of SSL is important, particularly when the number of labeled examples is small. The experimental results show a strong performance that is mostly on par, and sometimes even higher, compared to state-of-the-art methods.\n------------------------------------------------------------------------------------------------------\nPOST REBUTTAL\n\nI thank the reviewers for their rebuttal. After considering the other reviews, the rebuttal, and discussing with the other reviewers, my initial rating remains unchanged. Most reviewers mention the lack of novelty in their reviews and the marginal improvement of the proposed MCL method over the baseline. In my opinion, the rebuttal does not convincingly address these issues.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the proposed method are not described in much detail, and I recommend adding a more thorough discussion on error cases. I would, for example, also be interested in a discussion on the issue of dataset imbalance since the proposed method is working particularly well for a small number of samples. Does this suggest that it can handle data-imbalance issues better? If so, why?", "review_text": "", "extra_scores": {"ethical_concerns": "Not applicable.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a new semi-supervised learning algorithm that is specifically targeted at learning with extremely limited labeled data. The method uses a heuristic approach to automatically generate labeling functions for unlabeled data that are combined to obtain soft pseudo-labels for each unlabeled data point. While the algorithm showed strong gains on CIFAR-10 with 40 labeled examples, reviewers felt it was incremental and criticized the relatively poor results on other datasets. Further, the presentation of the algorithm introduced a very large amount of terminology and was ultimately hard to follow by many of the reviewers. Given these concerns, the consensus is not to accept.
Decision: Accept (Poster)