Title: DP-SSL: Towards Robust Semi-supervised Learning with A Few Labeled Samples
Authors: Yi Xu
Jiandong Ding
Lu Zhang
Shuigeng Zhou
Venue: NeurIPS 2021
Keywords: semi-supervised learning
data programming
classification

Abstract: The scarcity of labeled data is a critical obstacle to deep learning. Semi-supervised learning (SSL) provides a promising way to leverage unlabeled data by pseudo labels. However, when the size of labeled data is very small (say a few labeled samples per class), SSL performs poorly and unstably, possibly due to the low quality of learned pseudo labels. In this paper, we propose a new SSL method called DP-SSL that adopts an innovative data programming (DP) scheme to generate probabilistic labels for unlabeled data. Different from existing DP methods that rely on human experts to provide initial labeling functions (LFs), we develop a multiple-choice learning~(MCL) based approach to automatically generate LFs from scratch in SSL style. With the noisy labels produced by the LFs, we design a label model to resolve the conflict and overlap among the noisy labels, and finally infer probabilistic labels for unlabeled samples. Extensive experiments on four standard SSL benchmarks show that DP-SSL can provide reliable labels for unlabeled data and achieve better classification performance on test sets than existing SSL methods, especially when only a small number of labeled samples are available. Concretely, for CIFAR-10 with only 40 labeled samples, DP-SSL achieves 93.82% annotation accuracy on unlabeled data and 93.46% classification accuracy on test data, which are higher than the SOTA results.

TL;DR: A novel semi-supervised learning method that is effective and robust even only a few labeled samples per class are available.

Review discussion:
The proposed DP-SSL method can be seen as a combination of SSL learning methods and data programming. From my perspective, the key contribution of the method is to automatically generate LFs by training neural nets with the proposed loss. The empirical performance looks promising specifically when the labeled data size is very limited.
Here are some concerns/questions about the paper:
1. The presentation needs improvement. The paper lacks details about the proposed method.
For example:
a) it is not clear how \tilde{f}_k in Eq (1) is used in later equations;
b) in Eq (5) and lines 177-178, a schedule of setting the loss hyperparameters is mentioned but it's not clear how such schedule is set in practice;
c) some steps utilize previous works (for example line 49), but I think the paper should add some descriptions or discussions about the general idea.
2. I wonder why a final model training step is useful. If I understand it correctly, the final model training directly uses the probabilistic labels generated by the label modeling step and the training is similar to a supervised learning setting. If the labels from the label modeling step are already trustworthy, why not use them as predictions directly? If the final model training step is indeed necessary, then what's the reason that the final model can perform better if the generated annotations are relatively poorer?
3. Compared to some of the previous SSL learning methods, the proposed one incurs more computational costs as DP-SSL requires training additional LF models. The training costs are not compared and reported. As the FL models can be seen as an ensemble of neural networks, the improvements in the performance may come from the increase of the model capacity.
Minor comments:
Line 259: ```"40 labels" should be "400 labels"
## Post Rebuttal Update
I thank the authors for the response. After reading the rebuttal and other reviewers' comments, I decide not to change my rating of the paper. The main concerns are 1) the presentation requires improvement, and the paper lacks important technical details; 2) the design of the method lacks intuitive explanation or thorough empirical justification; 3) the training costs are not reported.
No
##### Q1. **It is not clear how $\tilde{f}_k$​​ in Eq. (1) is used in later equations**
Response: Actually, feature transformation only works between the backbone and classifiers. It transforms the feature for multiple classifiers.
Then, the $k$​-th classifier in step 1 further takes the transformed feature $\tilde{f}_k$​ and outputs the probability $\tilde{p}^k$​.  In the rest of Sec.3, we focus on the output probability $\tilde{p}^k$​, which is dependent on $\tilde{f}_k$​​ in Eq. (1).
So Eq. 1 is actually used in the rest of Sec. 3.
supp: For notation $\tilde{p}^k$, another subscript $\in\lbrace l,u\rbrace$ and the superscript $\in\lbrace w,s\rbrace$​​ are omitted for clear presentation.
##### Q2. **It is not clear how such schedule is set in practice in Eq. (5) and lines 177-178.**
Response: In the implementation, we set $\mu_1=1,\mu_2=\mu_u=0$​​​​​​​​ for 1) the first $1024$ iterations on all datasets with 4 labeled samples per class; 2) the first $4 * 1024$​ iterations on all datasets with 25 labeled samples per class; 3) the first $10*1024$ iterations on all datasets with 100 or 400 labeled samples per class.
##### Q3. **The paper should add some descriptions or discussions about the general idea.**
Response: We describe the general idea in Sec. 3.1 and L.52-62. We will give a more detailed description of the general idea in the revised manuscript.
##### Q4. **Why a final model training step is useful? Why not use the labels from the label modeling step as predictions directly?**
Response:  Our paper aims to provide a novel paradigm by producing probabilistic labels for image classification with a few labeled samples. As shown in Tab.3, not all samples are covered by the probabilistic labels.
Thus, we need to train another end model to do the inference for all samples in a dataset.
##### Q5. **What is the reason that the final model can perform better if the generated annotations are relatively poorer?**
Response: Actually, the performance of the final model in our method is similar to the generated annotations. The key reason why our method performs better is that we produce soft pseudo labels called probabilistic labels in data programming. The soft pseudo labels contain the uncertainty of the label model, which is robust to the noise of produced labels.
##### Q6. **The improvements in the performance may come from the increase of the model capacity.**
Response: No.
First, in the inference phase, we only utilize the end model, which is also a single model and follows the same architecture as FixMatch. As claimed in L.278-280, our method achieves a better annotation performance than FixMatch and USADTM.
Thus, the performance improvement comes from better annotations.
In Tab. 3, all label models integrate the same noisy labels from Step 1 of our method, while our label models outperform others with a considerable improvement. These results verify that our label model contributes the most to better annotations.
However, most increase of the model capacity comes from the multiple classifier heads.
Therefore, the improvement in performance does not come from the increase of the model capacity.
##### Q7. **The training costs are not compared and reported.**
Response:  Thanks for your suggestion. We will add the training cost comparison to the revised manuscript.
Originality:
The paper does not make a theoretical contribution. It describes several theoretical results that were obtained in prior work and applies these to the problem of SSL. The technical novelty seems rather limited to me since all components of the proposed approach have been proposed in prior work. To my understanding, the novelty is given by the application of the components to SSL and the empirical effectiveness at SSL, which is on par with state-of-the-art methods.
Quality:
The paper is technically sound and the methods are used appropriately. The overview of related work is also appropriate. The experimental evaluation is very sophisticated and conducted on a range of relevant datasets. Error bars are given in most experiments.
The main limitation of this paper is that it does not provide sufficient intuitive explanations of the proposed model. How does the MCL distribute the classes among the classification heads? Is this random? I did not find a detailed description of this process in Section 3.2.
although it seems important for the model. Section 3.4. describes the estimation of the reliability of the individual classification heads, but it does not give an intuitive explanation of the process.
Moreover, important technical details are not explained in the paper, and the reader is instead referred to related (L.154, 155)
The limitations of the proposed method are not described in much detail, and I recommend adding a more thorough discussion on error cases. I would, for example, also be interested in a discussion on the issue of dataset imbalance since the proposed method is working particularly well for a small number of samples. Does this suggest that it can handle data-imbalance issues better? If so, why?
Clarity:
The overall quality of the manuscript is satisfactory.
While the language is very good and the experiments are described in detail, the methods part lacks intuition and technical details that would better support understanding the technical contribution of this paper (see comment in quality section).
Significance:
The problem of SSL is important, particularly when the number of labeled examples is small. The experimental results show a strong performance that is mostly on par, and sometimes even higher, compared to state-of-the-art methods.
POST REBUTTAL
I thank the reviewers for their rebuttal. After considering the other reviews, the rebuttal, and discussing with the other reviewers, my initial rating remains unchanged. Most reviewers mention the lack of novelty in their reviews and the marginal improvement of the proposed MCL method over the baseline. In my opinion, the rebuttal does not convincingly address these issues.
No
#### 1. **About the originality of the paper**
Response: Though our paper is not a theoretical work,  the novelty of our method lies in at least three aspects: 1) existing DP methods rely on manually designed LFs or their LFs cannot be used for other tasks [1], while we develop an MCL based approach to automatically generate LFs from scratch in SSL style. 2) We introduce an abstention category to handle the over-confidence in MCL, which is different from the existing MCL methods [2,3]. 3) We design a novel label model, which incorporates a semi-supervised manner and an unsupervised quality guidance regularizer.
[1] Chen, Vincent S., et al. "Scene graph prediction with limited labels." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2019.
[2] Lee, Kimin, et al. "Confident multiple choice learning." International Conference on Machine Learning. PMLR, 2017.
[3] Tian, Kai, et al. "Versatile multiple choice learning and its application to vision computing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
#### 2.
About the quality of the paper
* **It does not provide sufficient intuitive explanations of the proposed model.**
Response: Due to the space limit, we didn't provide much intuitive explanation on the model.  We will give a more detailed description and illustrative examples for a better understanding of the model as an appendix in the revised manuscript.
* **How does the MCL distribute the classes among the classification heads? Is this random?**
Response: No, it's not random. Generally, only top-($\rho\times K$) small losses among $K$ LFs are utilized to optimize the model for each sample. For a clear presentation of  $\mathcal{L}_1(x_l^w,y_l)$, we take $K=4$, $\rho=0.5$ for example. Given two labeled samples $x_1$ and $x_2$, the corresponding cross entropy of 4 LFs are [0.1, 0.4, 3, 5] and [0.4, 0.1, 0.2, 2], respectively.
Thus, the indicator variable $u$ is derived as [1, 1, 0, 0] and [0, 1, 1, 0] via top-$(\rho\times K)$ small operator.
Afterwards, LFs with top-$(\rho\times K)$ small losses are optimized.
That is, $LF_1$ and $LF_2$ are optimized with the cross-entropy of $x_1$, $LF_2$ and $LF_3$ are optimized with the cross-entropy of $x_2$. After some iterations, each LF will tend to predict the similar samples at a high accuracy.
* **It does not give an intuitive explanation of the estimation of the reliability of the individual classification heads in Sec. 3.4**
Response: As we have assumed in L.184 that K LFs are independent, we can obtain the conditionaly independence $\lambda_i^j \independent  \lambda_i^{k}|Y_i$​​ for all distinct $j$​​, $k$​​.
Then $\lambda_i^jY_i$​​ and $\lambda_i^kY_i$​​ are independent.
Thus, the key idea of our estimation is exploiting the independencies. Note that we have no way of directly observing $\hat{\mathbb{E}}[\lambda_i^jY_i]$​​, but $\mathbb{E}[\lambda_i^j\lambda_i^k]$​​ is observable.
Then a third source $\lambda_i^l$​​ is introduced, which allows us to see $\hat{\mathbb{E}}[\lambda_i^j\lambda_i^l]$​​ and $\hat{\mathbb{E}}[\lambda_i^k\lambda_i^l]$​​. With the estimated expectation of the product of two sources, the estimated accuracy $\hat{\mathbb{E}}[\lambda_i^jY_i]$​​ can be solved.
Based on the whole process of estimation, the main unreliability means the uncertainty of $\hat{\mathbb{E}}[\lambda_i^jY_i]$​. In our implementation, we use the mean triplet method to reduce the uncertainty of $\hat{\mathbb{E}}[\lambda_i^jY_i]$​, i.e., given $j$​, we enumerate $k$​ and $l$​ in the triplet to get the average estimated accuracy.
Besides, the size of unlabeled data is about $50000$​ in CIFAR, which also ensures the reliability for $\hat{\mathbb{E}}[\lambda_i^j\lambda_i]$​​​.
* **Important technical details are not explained in the paper, and the reader is referred to related (L.154, 155).**
Response: Thanks for your comment. We will improve our manuscript to include a more detailed description of the proposed model to make it more readable.
* **The limitations are not described in much detail. Recommend adding a more thorough discussion on error cases.**
Response: Thanks for your suggestions. In the revised manuscript, we will provide some error cases.
* **Does this suggest that it can handle data-imbalance issues better? If so, Why?**
Response: Our paper focuses on improving the performance of image classification with a few labeled samples. Actually, our method can handle the data-imbalance issue. To the best of our knowledge. [1] addresses the data-imbalance issue; it deals with the long-tailed distribution in scene graph prediction by producing more annotations for the tailed classes.
[1] Chen, Vincent S., et al. "Scene graph prediction with limited labels." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2019.
#### 3. **About clarity**
Response: Thanks for your comment. We will complement the intuition and details in the resized manuscript to make it more readable.
### Strengths
- **Significance:** Obviously the paper achieves very strong results on semi-supervised benchmarks including CIFAR-10, CIFAR-100, SVHN, and STL-10.
- **Originality:** Using ensemble models to perform pseudo-labelling is a well-motivated direction, and to my knowledge the proposed approach is original.
### Weaknesses
- **Clarity:** Overall the math is not very clearly presented in this paper. Many notations are not explained.
- The form of the labeling function is not clear. After explaining what $\tilde{f}_k$ is, we directly jump to $\tilde{p}^{w}_k$.
- The MCL technique is not clearly described. It is unclear how $u$ is learned and updated.
- The description of accuracy estimation is a bit confusing and I am not sure I fully understand the techniques here.
- **Overall design of the model:** Although the paper reports very strong results, I feel like there are lots of components going on and it’s more like a description of an engineering pipeline rather than some general principles.
For example, why are there three loss functions instead of just the supervised + the unsupervised? Why does the model need to pretrain with loss 1 and then switch to loss 2+3? I have more comments below regarding the detailed design for LF, MCL, label model, etc. They all seem to make some sense if this is a description of a big engineering pipeline, but they are not the most simplistic variant to go for if one wants to seek for more scientific understanding.
- **Design of labeling functions:** I don’t understand the design of equation 1 and what purpose it serves. Why not just feed each LF with a different projection of input features?
- **Effect of MCL:** The effect of MCL is very marginal in the ablation studies. You get 1.36% boost on 40 labels and 0.31% boost on 250 labels. Even without MCL, the method can get state-of-the-art numbers in Table 1 (with 7.54% error rate on CIFAR-10 40 labels), but the paper conducts experiments on top of FixMatch.
So it is unclear what contributes to the major gain between the original FixMatch and the variant without MCL, which is more significant than the gap between no MCL and the full model.
- **Effect of abstention:** The paper introduces this abstention choice for MCL.
But it is not necessary. If a model doesn’t have a class in the subset, then it just doesn’t participate in the voting process for that class. Why complicate the learning procedure by adding an unknown class, if the benchmark is only a fixed set of classes?
- **Ablations on $K$ and $\rho$:** For both of these hyperparameters, it would be better to plot a curve or table with equal intervals. An interval like 20-40-50-60-100 or 0.1-0.2-0.3-0.5-1.0 doesn’t make sense.
- **Other ensemble baselines:** Instead of using MCL, one can just train different ensemble models with different random seed and random partition of the dataset, without having to optimize the partition variables. This is also not studied here.
- **Effect of label model:** There is no ablation on the effect of the label model vs. just like taking the average or majority.
- **Effect of accuracy estimation:** Why would you weight the log terms with the accuracy? And what if you don’t weigh the regularizer with accuracy?
- **Effect of regularization:** Ablation table suggests that the regularization overall only has a tiny effect.
- **Effect of probabilistic labels:** There is no ablation on whether to use the probabilistic labels or just taking the max and make it into a discrete one-hot label.
### Conclusion
In conclusion, despite the paper's empirical success, I consider it unfit for publishing at NeurIPS due to 1) the sheer complexity of many unverified components of model designs and 2) unclear paper writing, and I therefore recommend rejection as my initial rating.
Update after rebuttal:
I thank the author for a detailed response. I have read the response and it does address some of my concerns regarding the ablation of certain parameters.
However, I retain my current rating recommendation since I think it would be hard to accept the paper without a major revision to clear up the writing and the concepts.
No
1. Clarity
* **Many notations are not explained.**
Response: Thank you for the pointer. We will add a glossary of variables and symbols used in our paper as an  Appendix in our revised version.  In order to clearly present the notations, we additionally denote $k$​​-th classifier in Step 1 as $\mathcal{F}_k$​​.
For your convenience of understanding the paper, the following is a glossary of variables and symbols used in this paper.
| Symbol                             | Meaning                                                      |
| ---------------------------------- | ------------------------------------------------------------ |
| $x_l, y_l$​                         | Labeled samples and its ground-truth label.                  |
| $x_l^w$                            | Weakly augmented labeled samples.                            |
| $x_u$                              | Unlabeled samples.                                           |
| $x_u^w$                            | Weak augmented unlabeled samples.                            |
| $x_u^s$                            | Strongly augmented unlabeled samples.                        |
| $\mathcal{Y}$                      | the labels space of classification task. For CIFAR-10, $\mathcal{Y}=\{1,2,\cdots, 10\}$​ |
| $f$                                | $\mathbb{R}^{H\times W\times C}$​. $f$ is the feature map obtained from backbone. H, W, C are the height, width and channels of feature map before global averageing pooling. |
| $f_j$                              | $\mathbb{R}^{H\times W}$​. ​Feature vector at spatial position $j$​​ of $f$​​. |
| $\Lambda(A, B)$                    | Distance between A and B.                                    |
| $c_k$                              | $\mathbb{R}^C$. Learnable clustering center for the $k$-th LF. |
| $\tilde{f}_k$                      | Transformed feature for $k$-th LF.                           |
| $\mathcal{F}_k$                    | The $k$-th classifier head, which is implemented as a fully-connected layer. |
| $\tilde{p}_l^{w,k}$​                | $\tilde{p}_l^{w,k}=\mathcal{F}_k(\tilde{f}_k)$, $\tilde{p}_l^{w,k}\in \mathbb{R}^{\|\mathcal{Y}\|}$ is the predicted distribution of $k$-th LF based on transformed feature $\tilde{f}_k$ of weakly augmented labeled samples $x_l^w$. In $\tilde{p}_l^{w,k}$, subscript $\in\lbrace l,u\rbrace$ denotes whether it is labeled, the first superscript $\in\lbrace w,s\rbrace$ denotes it augmentation, the second supercript means it is produced by $k$​​-th classifier head. |
| $u^k$​                              | Indicator variable in MCL. $u^k=\mathbb{1}[k\in arg\min_{A' \subset \{1,2,\cdots,K\}, \|A'\|=\rho\times K}\sum_{k'\in A'}H(y_l, \tilde{p}_l^{w,k'})]$​​​​​​​​​, which means we set $u^k=1$​​ whose cross-entropy loss of $(x_l^w,y_l)$​​ is the top-$(\rho\times K)$​​ small among cross-entropy from K LFs. Note that cross-entropy loss is calculated at sample-level, not batch-level. |
| $\tau_k$                           | $\tau_k$ is the specialized classes of $k$-th LFs. $\tau_k \in\lbrace\tau_k^1, \cdots, \tau_k^{\|\tau_k\|}\rbrace$. |
| $\tilde{p}_{l,a}^{w,k}$            | Predicted probability of "abstention" option from $k$​-th LF by inputing transformed feature $\tilde{f}_k$​ of weakly augmentated labeled samples $x_l^w$​. | 
| $\tilde{q}_l^{w,k}$                | $\tilde{q}\_{l}^{w,k}=\text{normalized} \left( \text{Concat} \left( \tilde{p}\_{l}^{w, k} , \tilde{p}\_{l, a}^{w,k} \right) \right)$​​​​​​​​​​.  $\tilde{q}\_l^{w,k}\in \mathbb{R}^{\|\mathcal{Y}\|+1}$​​​​​​​​​ is the probability produced by $k$​​​​​​​​​-th LFs. $\tilde{q}\_l^{w,k}$​​​​​​​​ represents the probability of specialized class or the abstention class. Concretely, $ \tilde{q}\_l^{w,k} [ \tau\_{k}^{j} ] $​​​​​​​ is the probability of class $\tau\_{k}^{j}$​​​​​​, $1-\sum_{j =1}^{\|\tau\_k\|}\tilde{q}_l^{w,k}[\tau\_{k}^{j}]$​​​​​ is the probability of abstention class.​ |
| $H(A, \tilde{p}_l^{w,k})$​​          | Cross entropy of $k$​-th LF by taking $x_l^w$​ as input with ground truth A. |
| $\tilde{y}^k$                      | $\tilde{y}^k=arg\max(\tilde{q}^k)$​​. $\tilde{y}^k$​ is the predicted label of $k$​​-th LFs. No superscript $\lbrace w, s\rbrace $ and subscript $\lbrace l,u\rbrace$ means it represents samples from any augmented labeled / unlabled data. |
| $\tilde{\mathbf{y}}$               | $\tilde{\mathbf{y}}=[\tilde{y}^1, \tilde{y}^2,\cdots,\tilde{y}^k]$​​. $\tilde{\mathbf{y}} \in \mathbb{R}^K$​​ is the vectorized form of the predicted labels from K LFs. No superscript $\lbrace w, s\rbrace$ and subscript $\lbrace l,u\rbrace$ means it represents samples from all augmented labeled / unlabled data. |
| $\theta$                           | $\theta \in \mathbb{R}^{K\times \|\mathcal{Y}\|}$​​ is the parameters of label model. |
| $e_{ky}$                           | $e_{ky}:=exp(\theta_{ky})$​ represents the exponent of parameters $\theta_{ky}$. |
| $\phi(y,\tilde{y}^k)$              | the potential values given $y$ as the target class and $\tilde{y}^k$ as the predicted labels from $k$-th LF. |
| $P(y, \tilde{\mathbf{y}})$         | Joint distribution between target label and predicted label in the label model. |
| $Z$                                | Normalizer item of $P(y, \tilde{\mathbf{y}})$. $Z=\sum_{y\in\mathcal{Y}}\sum_{\forall \tilde{\mathbf{y}}}P(y, \tilde{\mathbf{y}})$ |
| $R(\theta,\tilde{\mathbf{y}}_u)$   | Regularizer of unlabeled data in Eq. (9).                    |
| $Y_i$                              | $Y_i=\begin{cases} +1 & y_u=i \\\\ -1 & y_u\neq i \end{cases}$​​​​ is the unseen grouth truth label for $x_u$​​​​ in $i$​​​-th one-versus-all task. In our paper, $i$​-th one-versus-all task denotes the one-versus-all classification for clss $i \in \mathcal{Y}$​. |
| $\lambda_i^k$                      | $\lambda_i^k=\begin{cases} +1 & \tilde{y}^k=i \\\\ 0 & \tilde{y}^k=0 \\\\ -1 & otherwise \end{cases}$​​​ is the observable predicted label from $k$-th LF for $x_u$ in $i$​​​-th one-versus-all task. |
| $\mathbb{E}[\lambda_i^kY_i]$       | Expectation of $\lambda_i^kY_i$​. To some degree, it reflects the accuracy of $k$​-th LF in $i$​-th one-versus-all classification. |
| $\hat{\mathbb{E}}[\lambda_i^kY_i]$ | Estimated expectation of $\lambda_i^kY_i$ over all unlabeled data by observable $\tilde{\mathbf{y}}$. |
| $\hat{P}(\cdot)$                   | Probability estimated with the observable data.              |
| $\hat{a}_i^k$                      | $\hat{a}_i^k := \hat{P}(\lambda_i^k=Y_i\|\lambda_i\neq0)$. It means the accuracy of $k$-th LF in $i$-th one-versus-all classification. |
| | |
* **The form of labeling function is not clear. After explaining what $\tilde{f}_k$​​​ is, we directly jump to $\tilde{p}_k^w$​​​.**
Response: We will clarify the notations of our paper in the revised manuscript. As mentioned above, we denote the $k$​​​​​​​​​​-th classifier in Step 1 as $\mathcal{F}_k$​​​​​​​​​​.
thus, $\tilde{p}_l^{w,k}=\mathcal{F}_k(\tilde{f}_k)$​​​​​​​​​​​ when $\tilde{f}_k$​​​​​​​​​ corresponds with the transformed feature of weakly augmented labeled data in $k$-th LF.
* **The MCL technique is not clearly described. It is unclear how $u$​​ is learned and updated.**
Generally, only top-($\rho*K$​​) small losses among $K$​​ LFs are utilized to optimize the model for each sample. For a clear presentation of  $\mathcal{L}_1(x_l^w,y_l)$​​, we take $K=4$​​, $\rho=0.5$​​ for example. Given two labeled samples $x_1$​​ and $x_2$​​, the corresponding cross entropy of 4 LFs are [0.1, 0.4, 3, 5] and [0.4, 0.1, 0.2, 2], respectively. *Thus, the indicator variable $u$​​ is derived as [1, 1, 0, 0] and [0, 1, 1, 0] via top-$(\rho\times K)$​​ small operator on cross entropy*.
Afterwards, LFs with top-$(\rho\times K)$​​ small losses are optimized.
That is, $LF_1$​​ and $LF_2$​​ are optimized with corresponding cross-entropy of $x_1$​​, $LF_2$​​ and $LF_3$​​ are optimized with corresponding cross-entropy of $x_2$​​​.
* **The description of accuracy estimation is a bit confusing.**
We will rewrite this part to clear it up.
#### 2.
About the overall design of the model
**Why are there three loss functions instead of just the supervised + the unsupervised? Why does the model need to pretrain with loss 1 and then switch to loss 2+3?**
Response:
On the one hand, we need to learn the specialized set of each LF via Eq. (2). With the obtained specialized set for each LF, we employ the SSL strategy in Eq. (2) and Eq.(3).
On the other hand, in Eq. (2), only top-$(\rho \times K)$  small losses are kept as the supervision signals, losses in other LFs are ignored.
While in Eq. (3), samples not specialized by one LF are supervised as an abstention category, which affects the back-propagation process.
#### 3. Design of labeling functions
**What the design Eq.(1) serves? Why not just feed each LF with a different projection of input features?**
Response: Briefly, Eq. (1) is used to keep the diversity of features for K LFs; it can be regarded as equipping each LF with a different projection of input features.
However, simply adding a projection without constraints would lead to 1) some LFs specialize no samples; 2) more LFs share the same specialized set.
#### 4. Effect of MCL
**The effect of MCL is very marginal in the ablation studies. It is unclear what contributes to the major gain between the original FixMatch and the variant without MCL, which is more significant than the gap between no MCL and the full model.**
Response:
Our method reduce the error rates from $7.54$​% of Exp1 in Tab. 4 to $6.18$​% of our full model (Exp3) on CIFAR-10 with 40 labeled samples, which means  the relative error rates of annotation for our full model are reduced by $18$​​​​​%.
Therefore, the effect of MCL is not marginal.
Thank you for the question. We have forgetten one importance baseline in Tab. 4, i.e., "directly average the predictions of multiple semi-supervised classifier heads". Exp1 without MCL means multiple semi-supervised classifiers plus a label model. Actually, it brings a large improvement comparing with FixMatch.
However, directly averaging the predictions of multiple semi-supervised classifier heads achieves the annotation performance $90.42$%​​​ on CIFAR-10 with 40 labeled samples, which is lower than that of Exp1. Comparing the performance of $90.42$​​​​​% and the performance of Exp 1, we can conclude the most contributions of the improvement for our method over FixMatch comes from the label model.
#### 5. Effect of abstention
**Why complicate the learning procedure by adding an unknown class, if the benchmark is only a fixed set of classes?**
Response: Actually, we give a brief explanation in L.164-165. The conclusion from [1, 2] tells us that MCL tends to be overconfident for the samples out of the specialized set.
In other words, when one LF is fed with a sample whose ground-truth label is out of the specialized set, it may produce a label in the specialized set with high confidence in a high probability.
Therefore, we introduce an unknown class to cope with the issue of over-confidence.
[1] Lee, Kimin, et al. "Confident multiple choice learning." International Conference on Machine Learning. PMLR, 2017.
[2] Tian, Kai, et al. "Versatile multiple choice learning and its application to vision computing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
#### 6. Ablations on $K$​ and $\rho$​
**better to plot a curve or table with equal intervals.**
Response: We have conducted the experiments with equal intervals, but didn't present the results in such a way for saving space. For your reference, complete experimental results on CIFAR-10 with 40 labels are given in the following table.
#### 7.
About other ensemble baselines
**There is no ablation on the effect of the label model vs. just like taking the average or majority**
Response: Thanks for your suggestion. In our paper, we compare label model with two ensemble baselines in Tab. 3. Majority Vote has already included in our paper. Following your suggestion,  we will consider more ensemble baselines in the revised manuscript.
#### 8. Effect of accuracy estimation
**Why would you weight the log terms with the accuracy? And what if you don't weight the regularizer with accuracy?**
Response: Actually, Eq. (13) is the KL distance between the the observed accuracy $\mathbb{E}[\lambda_i^jY_i]$​​​​​​ and the accuracy calculated by the parameters $\theta$​​ in the label model.
#### 9. Effect of regularization
**Ablation table suggests that the regularization overall only has a tiny effect.**
Response: Our method reduces the error rates from $6.81$​​​​​% of Exp1 in Tab. 4 to $6.18$​​​​​% of our full model (Exp3) on CIFAR-10 with 40 labeled samples, which means  the relative error rate of annotation for our full model is reduced by $9.25$​​​​​​​%.  This shows that the effect of regularization is obvious, NOT tiny.
Moreover, taking an extreme case without labeled samples for example, consider a binary classification task and a set $S$​ of LFs specializing in discriminating class A but not class B.
Then, we assume that all unlabeled samples with ground-truth class B mis-triggered one or more LFs in $S$​.
Even when our four-part potential function is utilized, as training progresses, the likelihood of unlabeled samples (the second term in Eq. (9)) will be globally maximized when all LFs favor the same class on all instances.
#### 10. Effect of probabilistic labels
**There is no ablation on whether to use the probabilistic labels or just taking the max and make it into a discrete one-hot label.**
Response: Soft pseudo labels contains the uncertainty of label model, while hard pseudo labels may enlarge the noise of the predicted labels, e.g. samples whose $\max(\pi)$​ smaller than a certain threshold. For comparison, we have conduct the hard pseudo labels in Step 3.
Thus, Eq. (14) becomes $\mathcal{L}(x_l, y_l, x_u, \pi) = \sum_{x_l}H(y_l, p_l) +
\sum_{x_u}H(arg\max(\pi), p_u)$​. The error rate of such hard pseudo labels on CIFAR-10 with 40 labels is $8.61\pm1.53$​, much lower than ours.
#### 11.
About Conclusion
* **the sheer complexity of many unverified components of model designs**
Response: Thanks for your comments. We think most of your concerns and confusion may be caused by our imperfect presentation. Above, we have tried our best to address your concerns and questions point by point. We hope that our detailed responses can alleviate some (if not all) of your concerns on the quality of our papet, and conclude a different decision on our work.
* **unclear paper writing**
Response: We understand that our paper is not perfect in presentation. Following your suggestions, we will present a more readable version of the paper.
Strengths
- S1: The proposed method is based on simple intuitive motivation to clean up the noisy labels from the label functions (LFs).
- S2: Noticeable gain in low shot regime in CIFAR10 dataset
Weaknesses
- W1: Lack of novelty. The method is a heuristic combination of existing data programming (DP) module with hand-crafted label clean-up model.
Though the combination is well described, the combination lacks the novelty.
- W2: Performance gain is overall marginal across many datasets. The interesting gain is in low sample regime (40 labels) in CIFAR10 dataset (in Table 1). In other low sample regimes (e.g., 40 labels in SVHN and STL-10), the proposed method does not shows compelling gains. Given all benchmarked datasets are not large scale, it is not clear that the proposed method improves the performance meaningfully - compared to USADTM (L268-272).
- W3: Lack of study of heuristic design choices.
Although Table 4 (in Sec. 4.5) has in-depth studies for K and $\rho$, the $\epsilon$ in Eq. 4 is not well motivated (or lack of sensitivity studies for it) and why Eq. (7) should be in the proposed form.
- W4: Presentation is largely unclear. The notations in equations are defined but not well explained about their physical meanings (e.g., $\tilde{q}_l^{w,k}$ in Eq. (3)).
Thus it is difficult to follow why each equations make sense.
No
##### 1. **About novelty**
Response: Sorry, we don't agree that our method is just a trivial combination of existing techniques. The novelty of our method lies in at least three aspects: 1) existing DP methods rely on manually designed LFs, or their LFs cannot be used for other tasks [1], while we develop an MCL-based approach to automatically generate LFs from scratch in SSL style. 2) We introduce an abstention category to handle the over-confidence in MCL, which is different from the existing MCL methods [2,3]. 3) We design a novel label model, which incorporates a semi-supervised manner and an unsupervised quality guidance regularizer.
[1] Chen, Vincent S., et al. "Scene graph prediction with limited labels." *Proceedings of the IEEE/CVF International Conference on Computer Vision*. 2019.
[2] Lee, Kimin, et al. "Confident multiple choice learning." International Conference on Machine Learning. PMLR, 2017.
[3] Tian, Kai, et al. "Versatile multiple choice learning and its application to vision computing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
##### 2.  **About performance gain. The performance gain is overall marginal across many datasets. Given all benchmarked datasets are not large scale, it is not clear that the proposed method improves the performance meaningfully - compared to USADTM.**
Response: First, as our method is implemented over FixMatch (RA), the most proper baseline for comparison should be FixMatch (RA) instead of USADTM.
Second, comparing with FixMatch (RA), our method reduces the error rates from 13.81% / 48.85% / 3.96% to 6.54% / 43.17% / 2.98% on CIFAR-10 / CIFAR-100 / SVHN with 4 labels per class, which shows considerable performance improvement of our method.
As for the performance on STL-10 with 1000 labeled samples, our method achieves an error rate of 4.97%, which outperforms FixMatch (RA) with an error rate of 7.98%.
As for USADTM, apart from the proxy label generator, there exists an unsupervised mutual information loss, which can also bring performance improvement for FixMatch and our method.
Third, most existing methods have not been tested on ImageNet. Actually, only FixMatch was tested on ImageNet.
So for a fair comparison, we present only comparison results on four widely-used datasets: CIFAR-10, CIFAR-100, SVHN, and STL-10. We believe that our model is consistently capable of boosting the performance on large-scale datasets such as ImangeNet, and we will add such results in the revised version.
##### 3. **Why Eq. (7) should be in the proposed form?**
Response: First, we extend the dimension of parameters $\theta$ in the label model to $C\times K$, because $\theta \in \mathbb{R}^K$ in Sec. 2.2 in [1] is not suitable for our work as each LF in our paper specializes in multiple categories and the accuracy of different categories in one LF is different.
Intuitively, the basic idea is to set the potential function as $\phi(y,\tilde{{y}}^k)=exp(\theta_{ky})$. In this case, $P(y, \tilde{\textbf{y}})=\frac{1}{Z}\prod_{k=1}^{K}exp(\theta_{ky})$. We can see that $P(y, \tilde{\mathbf{y}})$ relies  only on $y$, and has no relationship with $\tilde{\mathbf{y}}$.
Meanwhile, if we set $\phi(y, \tilde{y}^k)=exp(\theta_{k\tilde{y}})$, there would be no relationship between the final prediction $y$ of label model and  the predicted probability.
Then, we update the potential function  as $\phi(y,\tilde{y}^k)=\begin{cases}	e_{ky}, &if \\; \tilde{y}^k \in \tau_k\\\\ 1. &otherwise
\end{cases}$​, where $e_{ky}:=exp(\theta_{ky})$​. It can be observed that the potential function of one LF is triggered only when the LF predicts a specialized category.
However, such potential function is sensitive to initilization. We will get 100% accuracy when the parameter for the agreeing case: $\phi(y=\tilde{y}^k, \tilde{y}^k)$​ is larger than $\phi(y\neq\tilde{y}^k, \tilde{y}^k)$​, and 0% accuracy when $\phi(y=\tilde{y}^k, \tilde{y}^k)$​ is smaller.
Thus, we have to carefully initialize the parameters $\theta$​ so that $\theta_{k\tilde{y}}$​​​​​​​​​​ does have larger values.
Next, it is natural to write the potential function as $\phi(y,\tilde{y}^k)=\begin{cases}
1 + e_{ky}, &{ if \\;\tilde{y}^k\in \tau_k,\\; \tilde{y}^k=y}\\\\
1/(1+e_{ky}), &if \\; \tilde{y}^k\in \tau_k,\\; \tilde{y}^k \neq y\\\\ 1. &otherwise\\\\
\end{cases}$​​​​​​​​​​​,
where we impose a positive impact on the final prediction when $\tilde{y}^k$​​​​​​​​​​​ agrees with the target $y$​​ and a negative impact when $\tilde{y}^k$​​ conflicts with the target $y$​​.
However, the conclusion from [2, 3] tells us that MCL tends to be overconfident for the samples whose ground-truth labels are out of the specialized set.
In other words, when one LF is fed with a sample whose ground-truth label is out of the specialized set, it may still produce a label in the specialized set with high confidence.
Although such cases decreases a lot due to the introduction of the "abstention" category in Step 1, these cases still exist in our framework.
Therefore, we introduce the item of $e_{ky}$​​ to represent the relationship between the predicted label $\tilde{y}^k$​​ of the $k_{th}$​​ LF and the target label $y$​​​​​​​​​​​​​​, even when the target labels conflict with the predicted labels from LF.
Based on these considerations, we define the four-part potential function as $\phi(y,\tilde{y}^k)=\begin{cases}
1 + e_{ky}, &{ if \\; y \in \tau_k,\\;\tilde{y}^k\in \tau_k,\\; \tilde{y}^k=y}\\\\
1/(1+e_{ky}), &if \\; y \in \tau_k,\\;\tilde{y}^k\in \tau_k,\\; \tilde{y}^k \neq y\\\\
e_{ky}, &if \\; y \notin \tau_k, \\; \tilde{y}^k \in \tau_k,\\; \tilde{y}^k \neq y \\\\ 1. &otherwise
\end{cases}$​​​​​​​​.
##### 4.  **About setting of $\epsilon$​​​​​​ (or sensitivity study).**
Response: Actually, we evaluated the performance for different $\epsilon$​​​​​ values. For  $\epsilon$​​​​​ = 0.8,0.85,0.9,0.95 and 0.975, corresponding accuracy on CIAFR-10 with 40 labels are [79.05, 86.60, 90.98, 93.46, 91.73], respectively. We can see that when $\epsilon=0.95$​​​​​, we get the best performance.
So in the experiments, we set $\epsilon=0.95$​​​​​. In FixMatch,  $\epsilon$​​​​​​ is also set to 0.95.
##### 5.
About presentation
Response: Thank you for pointing out the presentation problems. We will add a glossary of notations and symbols as an Appendix in the revised manuscript.  In order to clearly present the notation, we denote the $k$-th classifier in Step 1 as $\mathcal{F}_k$. For your convenience of understanding the paper, the following is a glossary of major variables and symbols used in the paper.
| Symbol                             | Meaning                                                      |
| ---------------------------------- | ------------------------------------------------------------ |
| $x_l, y_l$​                         | Labeled samples and its ground-truth label.                  |
| $x_l^w$                            | Weakly augmented labeled samples.                            |
| $x_u$                              | Unlabeled samples.                                           |
| $x_u^w$                            | Weak augmented unlabeled samples.                            |
| $x_u^s$                            | Strongly augmented unlabeled samples.                        |
| $\mathcal{Y}$                      | the labels space of classification task. For CIFAR-10, $\mathcal{Y}=\{1,2,\cdots, 10\}$​ |
| $f$                                | $\mathbb{R}^{H\times W\times C}$​. $f$ is the feature map obtained from backbone. H, W, C are the height, width and channels of feature map before global averageing pooling. |
| $f_j$                              | $\mathbb{R}^{H\times W}$​. ​Feature vector at spatial position $j$​​ of $f$​​. |
| $\Lambda(A, B)$                    | Distance between A and B.                                    |
| $c_k$                              | $\mathbb{R}^C$. Learnable clustering center for the $k$-th LF. |
| $\tilde{f}_k$                      | Transformed feature for $k$-th LF.                           |
| $\mathcal{F}_k$                    | The $k$-th classifier head, which is implemented as a fully-connected layer. |
| $\tilde{p}_l^{w,k}$​                | $\tilde{p}_l^{w,k}=\mathcal{F}_k(\tilde{f}_k)$, $\tilde{p}_l^{w,k}\in \mathbb{R}^{\|\mathcal{Y}\|}$ is the predicted distribution of $k$-th LF based on transformed feature $\tilde{f}_k$ of weakly augmented labeled samples $x_l^w$. In $\tilde{p}_l^{w,k}$, subscript $\in\lbrace l,u\rbrace$ denotes whether it is labeled, the first superscript $\in\lbrace w,s\rbrace$ denotes it augmentation, the second supercript means it is produced by $k$​​-th classifier head. |
| $u^k$​                              | Indicator variable in MCL. $u^k=\mathbb{1}[k\in arg\min_{A' \subset \{1,2,\cdots,K\}, \|A'\|=\rho\times K}\sum_{k'\in A'}H(y_l, \tilde{p}_l^{w,k'})]$​​​​​​​​​, which means we set $u^k=1$​​ whose cross-entropy loss of $(x_l^w,y_l)$​​ is the top-$(\rho\times K)$​​ small among cross-entropy from K LFs. Note that cross-entropy loss is calculated at sample-level, not batch-level. |
| $\tau_k$                           | $\tau_k$ is the specialized classes of $k$-th LFs. $\tau_k \in\lbrace\tau_k^1, \cdots, \tau_k^{\|\tau_k\|}\rbrace$. |
| $\tilde{p}_{l,a}^{w,k}$            | Predicted probability of "abstention" option from $k$​-th LF by inputing transformed feature $\tilde{f}_k$​ of weakly augmentated labeled samples $x_l^w$​. | 
| $\tilde{q}_l^{w,k}$                | $\tilde{q}\_{l}^{w,k}=\text{normalized} \left( \text{Concat} \left( \tilde{p}\_{l}^{w, k} , \tilde{p}\_{l, a}^{w,k} \right) \right)$​​​​​​​​​​.  $\tilde{q}\_l^{w,k}\in \mathbb{R}^{\|\mathcal{Y}\|+1}$​​​​​​​​​ is the probability produced by $k$​​​​​​​​​-th LFs. $\tilde{q}\_l^{w,k}$​​​​​​​​ represents the probability of specialized class or the abstention class. Concretely, $ \tilde{q}\_l^{w,k} [ \tau\_{k}^{j} ] $​​​​​​​ is the probability of class $\tau\_{k}^{j}$​​​​​​, $1-\sum_{j =1}^{\|\tau\_k\|}\tilde{q}\_l^{w,k}[\tau\_{k}^{j}]$​​​​​ is the probability of abstention class.​ |
| $H(A, \tilde{p}_l^{w,k})$​​          | Cross entropy of $k$​-th LF by taking $x_l^w$​ as input with ground truth A. |
| $\tilde{y}^k$                      | $\tilde{y}^k=arg\max(\tilde{q}^k)$​​. $\tilde{y}^k$​ is the predicted label of $k$​​-th LFs. No superscript $\lbrace w, s\rbrace $ and subscript $\lbrace l,u\rbrace$ means it represents samples from any augmented labeled / unlabled data. |
| $\tilde{\mathbf{y}}$               | $\tilde{\mathbf{y}}=[\tilde{y}^1, \tilde{y}^2,\cdots,\tilde{y}^k]$​​. $\tilde{\mathbf{y}} \in \mathbb{R}^K$​​ is the vectorized form of the predicted labels from K LFs. No superscript $\lbrace w, s\rbrace$ and subscript $\lbrace l,u\rbrace$ means it represents samples from all augmented labeled / unlabled data. |
| $\theta$                           | $\theta \in \mathbb{R}^{K\times \|\mathcal{Y}\|}$​​ is the parameters of label model. |
| $e_{ky}$                           | $e_{ky}:=exp(\theta_{ky})$​ represents the exponent of parameters $\theta_{ky}$. |
| $\phi(y,\tilde{y}^k)$              | the potential values given $y$ as the target class and $\tilde{y}^k$ as the predicted labels from $k$-th LF. |
| $P(y, \tilde{\mathbf{y}})$         | Joint distribution between target label and predicted label in the label model. |
| $Z$                                | Normalizer item of $P(y, \tilde{\mathbf{y}})$. $Z=\sum_{y\in\mathcal{Y}}\sum_{\forall \tilde{\mathbf{y}}}P(y, \tilde{\mathbf{y}})$ |
| $R(\theta,\tilde{\mathbf{y}}_u)$   | Regularizer of unlabeled data in Eq. (9).                    |
| $Y_i$                              | $Y_i=\begin{cases} +1 & y_u=i \\\\ -1 & y_u\neq i \end{cases}$​​​​ is the unseen grouth truth label for $x_u$​​​​ in $i$​​​-th one-versus-all task. In our paper, $i$​-th one-versus-all task denotes the one-versus-all classification for clss $i \in \mathcal{Y}$​. |
| $\lambda_i^k$                      | $\lambda_i^k=\begin{cases} +1 & \tilde{y}^k=i \\\\ 0 & \tilde{y}^k=0 \\\\ -1 & otherwise \end{cases}$​​​ is the observable predicted label from $k$-th LF for $x_u$ in $i$​​​-th one-versus-all task. |
| $\mathbb{E}[\lambda_i^kY_i]$       | Expectation of $\lambda_i^kY_i$​. To some degree, it reflects the accuracy of $k$​-th LF in $i$​-th one-versus-all classification. |
| $\hat{\mathbb{E}}[\lambda_i^kY_i]$ | Estimated expectation of $\lambda_i^kY_i$ over all unlabeled data by observable $\tilde{\mathbf{y}}$. |
| $\hat{P}(\cdot)$                   | Probability estimated with the observable data.              |
| $\hat{a}_i^k$                      | $\hat{a}_i^k := \hat{P}(\lambda_i^k=Y_i\|\lambda_i\neq0)$. It means the accuracy of $k$-th LF in $i$-th one-versus-all classification. |

Meta-review:
role: PC
data: comment: This paper presents a new semi-supervised learning algorithm that is specifically targeted at learning with extremely limited labeled data. The method uses a heuristic approach to automatically generate labeling functions for unlabeled data that are combined to obtain soft pseudo-labels for each unlabeled data point. While the algorithm showed strong gains on CIFAR-10 with 40 labeled examples, reviewers felt it was incremental and criticized the relatively poor results on other datasets. Further, the presentation of the algorithm introduced a very large amount of terminology and was ultimately hard to follow by many of the reviewers. Given these concerns, the consensus is not to accept.

Decision: Accept (Poster)