Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a method for semi-supervised learning (SSL) with a few seed labeled sample scenario. The method uses data programming [10] for the SSL set-up and cleans up the noisy label by the "label model". Using the cleaned label for the unlabeled data, the "end model" is trained for image classification. The proposed method outperforms prior arts especially in the low shot regime with large gain in CIFAR10 dataset.
main_review: Strengths
- S1: The proposed method is based on simple intuitive motivation to clean up the noisy labels from the label functions (LFs).
- S2: Noticeable gain in low shot regime in CIFAR10 dataset

Weaknesses
- W1: Lack of novelty. The method is a heuristic combination of existing data programming (DP) module with hand-crafted label clean-up model. Though the combination is well described, the combination lacks the novelty.
- W2: Performance gain is overall marginal across many datasets. The interesting gain is in low sample regime (40 labels) in CIFAR10 dataset (in Table 1). In other low sample regimes (e.g., 40 labels in SVHN and STL-10), the proposed method does not shows compelling gains. Given all benchmarked datasets are not large scale, it is not clear that the proposed method improves the performance meaningfully - compared to USADTM (L268-272).
- W3: Lack of study of heuristic design choices. Although Table 4 (in Sec. 4.5) has in-depth studies for K and $\rho$, the $\epsilon$ in Eq. 4 is not well motivated (or lack of sensitivity studies for it) and why Eq. (7) should be in the proposed form.
- W4: Presentation is largely unclear. The notations in equations are defined but not well explained about their physical meanings (e.g., $\tilde{q}_l^{w,k}$ in Eq. (3)). Thus it is difficult to follow why each equations make sense.
limitations_and_societal_impact: It has an limitation section but no discussion on potential negative societal impact.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper describes an approach for semi-supervised learning by having a lot of models in an ensemble to produce a set of diverse predictions with a subset of classes. It aggregates these votes through a probabilistic labeling model and uses the smoothed labels for the final round of training with these pseudo labels. It achieves state-of-the-art results on several semi-supervised learning benchmarks.

main_review: 
### Strengths
- **Significance:** Obviously the paper achieves very strong results on semi-supervised benchmarks including CIFAR-10, CIFAR-100, SVHN, and STL-10.
- **Originality:** Using ensemble models to perform pseudo-labelling is a well-motivated direction, and to my knowledge the proposed approach is original.

-----------------------------------------------------------
### Weaknesses
- **Clarity:** Overall the math is not very clearly presented in this paper. Many notations are not explained.
  - The form of the labeling function is not clear. After explaining what $\tilde{f}_k$ is, we directly jump to $\tilde{p}^{w}_k$.
  - The MCL technique is not clearly described. It is unclear how $u$ is learned and updated.
  - The description of accuracy estimation is a bit confusing and I am not sure I fully understand the techniques here.
- **Overall design of the model:** Although the paper reports very strong results, I feel like there are lots of components going on and it’s more like a description of an engineering pipeline rather than some general principles. For example, why are there three loss functions instead of just the supervised + the unsupervised? Why does the model need to pretrain with loss 1 and then switch to loss 2+3? I have more comments below regarding the detailed design for LF, MCL, label model, etc. They all seem to make some sense if this is a description of a big engineering pipeline, but they are not the most simplistic variant to go for if one wants to seek for more scientific understanding.
- **Design of labeling functions:** I don’t understand the design of equation 1 and what purpose it serves. Why not just feed each LF with a different projection of input features?
- **Effect of MCL:** The effect of MCL is very marginal in the ablation studies. You get 1.36% boost on 40 labels and 0.31% boost on 250 labels. Even without MCL, the method can get state-of-the-art numbers in Table 1 (with 7.54% error rate on CIFAR-10 40 labels), but the paper conducts experiments on top of FixMatch. So it is unclear what contributes to the major gain between the original FixMatch and the variant without MCL, which is more significant than the gap between no MCL and the full model.
- **Effect of abstention:** The paper introduces this abstention choice for MCL. But it is not necessary. If a model doesn’t have a class in the subset, then it just doesn’t participate in the voting process for that class. Why complicate the learning procedure by adding an unknown class, if the benchmark is only a fixed set of classes?
- **Ablations on $K$ and $\rho$:** For both of these hyperparameters, it would be better to plot a curve or table with equal intervals. An interval like 20-40-50-60-100 or 0.1-0.2-0.3-0.5-1.0 doesn’t make sense.
- **Other ensemble baselines:** Instead of using MCL, one can just train different ensemble models with different random seed and random partition of the dataset, without having to optimize the partition variables. This is also not studied here. 
- **Effect of label model:** There is no ablation on the effect of the label model vs. just like taking the average or majority.
- **Effect of accuracy estimation:** Why would you weight the log terms with the accuracy? And what if you don’t weigh the regularizer with accuracy?
- **Effect of regularization:** Ablation table suggests that the regularization overall only has a tiny effect.
- **Effect of probabilistic labels:** There is no ablation on whether to use the probabilistic labels or just taking the max and make it into a discrete one-hot label.

-----------------------------------------------------------
### Conclusion
In conclusion, despite the paper's empirical success, I consider it unfit for publishing at NeurIPS due to 1) the sheer complexity of many unverified components of model designs and 2) unclear paper writing, and I therefore recommend rejection as my initial rating.


Update after rebuttal:
I thank the author for a detailed response. I have read the response and it does address some of my concerns regarding the ablation of certain parameters. However, I retain my current rating recommendation since I think it would be hard to accept the paper without a major revision to clear up the writing and the concepts.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper addresses the problem of semi-supervised learning (SSL). Specifically, it aims to improve the quality of the pseudo-labels for SSL when the number of labeled samples is very small. To achieve this the paper uses a data programming scheme, however instead of defining the labeling functions through human experts, they propose to learn several different classification heads in an SSL manner which subsequently serves as a labeling function. This is implemented using multiple-choice learning (MCL), a technique that is known to improve the diversity of deep ensemble models. The different classification heads are combined into a final classification output by ensembling them while taking into account their reliability. The experimental results show that the model performs slightly better compared to related work on popular SSL benchmarks.
main_review: Originality:
The paper does not make a theoretical contribution. It describes several theoretical results that were obtained in prior work and applies these to the problem of SSL. The technical novelty seems rather limited to me since all components of the proposed approach have been proposed in prior work. To my understanding, the novelty is given by the application of the components to SSL and the empirical effectiveness at SSL, which is on par with state-of-the-art methods.

Quality:
The paper is technically sound and the methods are used appropriately. The overview of related work is also appropriate. The experimental evaluation is very sophisticated and conducted on a range of relevant datasets. Error bars are given in most experiments.
The main limitation of this paper is that it does not provide sufficient intuitive explanations of the proposed model. How does the MCL distribute the classes among the classification heads? Is this random? I did not find a detailed description of this process in Section 3.2. although it seems important for the model. Section 3.4. describes the estimation of the reliability of the individual classification heads, but it does not give an intuitive explanation of the process. Moreover, important technical details are not explained in the paper, and the reader is instead referred to related (L.154, 155)
The limitations of the proposed method are not described in much detail, and I recommend adding a more thorough discussion on error cases. I would, for example, also be interested in a discussion on the issue of dataset imbalance since the proposed method is working particularly well for a small number of samples. Does this suggest that it can handle data-imbalance issues better? If so, why?

Clarity:
The overall quality of the manuscript is satisfactory. While the language is very good and the experiments are described in detail, the methods part lacks intuition and technical details that would better support understanding the technical contribution of this paper (see comment in quality section).

Significance:
The problem of SSL is important, particularly when the number of labeled examples is small. The experimental results show a strong performance that is mostly on par, and sometimes even higher, compared to state-of-the-art methods.
------------------------------------------------------------------------------------------------------
POST REBUTTAL

I thank the reviewers for their rebuttal. After considering the other reviews, the rebuttal, and discussing with the other reviewers, my initial rating remains unchanged. Most reviewers mention the lack of novelty in their reviews and the marginal improvement of the proposed MCL method over the baseline. In my opinion, the rebuttal does not convincingly address these issues.
limitations_and_societal_impact: The limitations of the proposed method are not described in much detail, and I recommend adding a more thorough discussion on error cases. I would, for example, also be interested in a discussion on the issue of dataset imbalance since the proposed method is working particularly well for a small number of samples. Does this suggest that it can handle data-imbalance issues better? If so, why?
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper presents a semi-supervised learning method called DP-SSL that focuses on semi-supervised learning tasks where the number of labeled samples is extremely limited. The paper utilizes a 3 step approach: 1) generate neural network label functions; 2) train a label model to generate probabilistic labels for the unlabeled samples; 3) train a final model using the labeled data and the unlabeled data with the generated probabilistic labels.
main_review: The proposed DP-SSL method can be seen as a combination of SSL learning methods and data programming. From my perspective, the key contribution of the method is to automatically generate LFs by training neural nets with the proposed loss. The empirical performance looks promising specifically when the labeled data size is very limited.

Here are some concerns/questions about the paper:
1. The presentation needs improvement. The paper lacks details about the proposed method. For example:

    a) it is not clear how \tilde{f}_k in Eq (1) is used in later equations;

    b) in Eq (5) and lines 177-178, a schedule of setting the loss hyperparameters is mentioned but it's not clear how such schedule is set in practice;

    c) some steps utilize previous works (for example line 49), but I think the paper should add some descriptions or discussions about the general idea.


2. I wonder why a final model training step is useful. If I understand it correctly, the final model training directly uses the probabilistic labels generated by the label modeling step and the training is similar to a supervised learning setting. If the labels from the label modeling step are already trustworthy, why not use them as predictions directly? If the final model training step is indeed necessary, then what's the reason that the final model can perform better if the generated annotations are relatively poorer?

3. Compared to some of the previous SSL learning methods, the proposed one incurs more computational costs as DP-SSL requires training additional LF models. The training costs are not compared and reported. As the FL models can be seen as an ensemble of neural networks, the improvements in the performance may come from the increase of the model capacity.


Minor comments:
Line 259: ```"40 labels" should be "400 labels"

## Post Rebuttal Update
I thank the authors for the response. After reading the rebuttal and other reviewers' comments, I decide not to change my rating of the paper. The main concerns are 1) the presentation requires improvement, and the paper lacks important technical details; 2) the design of the method lacks intuitive explanation or thorough empirical justification; 3) the training costs are not reported.
limitations_and_societal_impact: The authors have addressed the limitations.
review_title: null

Meta-review: This paper presents a new semi-supervised learning algorithm that is specifically targeted at learning with extremely limited labeled data. The method uses a heuristic approach to automatically generate labeling functions for unlabeled data that are combined to obtain soft pseudo-labels for each unlabeled data point. While the algorithm showed strong gains on CIFAR-10 with 40 labeled examples, reviewers felt it was incremental and criticized the relatively poor results on other datasets. Further, the presentation of the algorithm introduced a very large amount of terminology and was ultimately hard to follow by many of the reviewers. Given these concerns, the consensus is not to accept.
Decision: Accept (Poster)