Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors propose an "end to end" approach to weakly supervised learning. Instead of obtaining ground truth labels by modeling WS sources via a probabilistic latent variable model which don't consider the downstream performance, the authors consider a neural model that directly learning the downstream model by maximizing WS sources' agreement with probabilistic labels generated by reparameterizing previous probabilistic posteriors. The method outperforms prior approaches on five real-world datasets.
main_review: - Originality: The method proposed in this work is new. Most of the existing work in weakly supervised learning focuses on only create accurate training labels via WS resources. Constrastingly, this work focuses on developing a "end to end" approach by condiering downstream performance.

- Quality: The paper has overall good experimental results while some strong claims lack of theoretical or empirical support. For example, 1) The ablation section is weak. An ablation study is needed to show in which case or which condition the proposed approach outperform the prior works. 2) It would be great to show whether the proposed approach works for different types of models, e.g., transformer-based model.

- Clarity: The writing is overall clear. While the experiment section is difficult to understand because of lack of organization, or flaws.

- Significance: The key insight of the paper that weakly supervised learning can be benefit by direcly considering downstream performance. This paper proposes a end-to-end approach to weak supervision modeling. While overall novaltiy is minor.

======== After rebuttal ========

Thank you for the detailed response. My questions are partially answered by the authors, and I would like to keep my original rating.


limitations_and_societal_impact: The authors discussed their limitations on probabilistic labeling functions and modeling more structure.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this paper, the authors propose an end-to-end approach for directly learning downstream models under a weak supervision setting by maximizing its agreement with probabilistic labels generated by reparameterizing previous probabilistic posteriors with a Neural Network. Experiments show good improvement over previous works over a number of datasets. 
main_review: Despite the claimed originality, similar to the provided reviews from ICML 2021, I find this paper is still a little hard to follow. 
1. The introduction could be better written to make it easier for readers outside the area to follow. 
2. from the problem setup in Line 95-Line 107, it seems the labeling functions just an ensemble of models performing the downstream task. Because it seems that all models in the $\mathbf{\lambda}$ perform the same $C$-way classification. This part needs to be defined more clear.
3. How specifically did the authors compute $P(y=c)$?
4. Figure 1 should have better quality. 
5. I also hope that the authors could clarify more on the uniqueness and novelty of their method. 
6. I'm eager to increase my score if the questions are well answered. 
limitations_and_societal_impact: The authors briefly mentioned the limitations in the papers and my concerns are presented above.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a new method for weak supervision with multiple noisy labeling functions. The method jointly trains an encoder network that can estimate sample-by-sample accuracies with the final downstream model, and maximizes the agreement between the two models. The authors show improved performance on several weak supervision data sets and further show that their method is robust against both adversarial labeling functions and highly-correlated labeling functions.
main_review: This paper presents an interesting idea for an important problem space. The method has several benefits over previous weak supervision pipelines, in particular robustness against previous failure modes. The algorithm and paper are clearly-written, and ablations are extensive.

There are a few ways that the paper and presentation could be stronger:

1. (high-order bit) I think the paper would benefit from further analysis (not necessarily theoretical) about why the method is providing lift, especially in the challenging adversarial settings. Section 6 has some paragraphs on "why it works," but the text almost exclusively discusses degenerate solutions. This section would benefit from a more detailed analysis of which data points the method shows improvement on, and why. A few questions to ask to help get at this:
* How accurate are the soft labels produced by the model ("label model accuracy")?
* Which points do you get better soft labels for?
* How did the labeling functions vote on those points?
* Are there other points where the labeling function voted similarly, but you saw better soft labels *because* of the encoder network?

2. (more minor) I feel that the explanation for why the method avoids degenerate solutions is a little unsatisfying. The authors write that they can avoid degenerate solutions either by early stopping or by setting the temperature parameter to a magic number (< 1/3). This is promising empirically, but leaves a pit in my stomach -- how do we know that we won't see degenerate solutions on another data set? Is there any stronger guarantee that can be made to avoid degenerate solutions? Can the method be adapted at all to detect degenerate solutions and avoid them?

3. (more minor) I would be curious to see a comparison against (https://arxiv.org/pdf/2006.15168.pdf), which was a follow up to some of the baselines in the paper, and also used an encoder network to affect the soft labels from the label model. Their reported performance approaches the performance of E2E on the spouses data set (although they do not report numbers for the other data sets).

Nit: There is something wrong with the formatting of the paper -- it appears that there are almost no line breaks after paragraphs starting in section 3.4 onwards.

If these comments are addressed well (particularly 1), I think this paper could definitely move above the acceptance threshold!

Rating: 5
limitations_and_societal_impact: Would appreciate moving discussion of societal impacts from appendix G into the body, given the extra page provided for it in NeurIPS this year
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new method to training networks with multiple weak labeling sources. In contrast to previous weak supervision approaches, which aggregate weak sources using a PGM *then* train a downstream classification model, the proposed method trains an encoder network (which learns to aggregate weak sources) in tandem with a classification model (which learns to directly classify input datapoints). The authors compare their proposed method to existing weak supervision tools and find their method leads to improved performance as well as improved robustness to adversarial and correlated weak labeling sources.


main_review: **Review summary.** This is an interesting and new approach to training networks with multiple weak labeling sources that seems to outperform existing weak supervision tools on the datasets tested in this submission. There are advantages of the proposed method---particularly, the ability to easily handle probabilistic weak labeling sources and correlated/adversarial sources. However, the paper lacks sufficient expository content to understand exactly why the proposed method outperforms baselines; the authors discuss a number of potential reasons, but do not back up these reasons with theoretical analyses or focused experiments. Regardless, the novelty of the method along with the improved empirical performance are strong enough to lead to my rating of marginally above the acceptance threshold. 

**Positives.**
* The approach seems to handle strongly correlated and adversarial labeling functions significantly better than baseline approaches, and the structure of the correlation among sources does not need to be specified by the user a priori (unlike many existing tools, which require the user to specify the structure of the weak labeling sources). Additionally, the proposed method should be able to easily handle probabilistic weak labeling sources, which cannot be used as labeling sources in many existing weak supervision frameworks. Finally, even in non-adversarial settings with binary labeling functions, the proposed method seems to outperform existing weak supervision tools. These practical and empirical improvements to the weak supervision framework will likely benefit many users of weak supervision. 
* The proposed method is, to my knowledge, novel and departs from the approach of most existing weak supervision tools. The method is sufficiently described, code is supplied, and the code will be made public (according to the authors). The method is validated with experiments on standard weak supervision/crowdsourcing benchmarks with existing state-of-the-art weak supervision/crowdsourcing tools, with synthetic datasets (although the synthetic settings are quite extreme), and via ablations. 
* The paper is clear, well-written, and well-organized. Similarly, the code is commented and easy to follow. 

**Limitations.**
* One of the advantages of using a PGM to model the weak labeling sources is that the PGM can be theoretically analyzed (e.g., to produce bounds on the generalization risk, understand conditions under which the model parameters can be recovered exactly, analyze how the weakly supervised performance scales with unlabeled data). These theoretical underpinnings have supported the use and understanding of weak supervision. However, the authors of this submission do not provide any theoretical support for understanding their method nor do they discuss which---if any---previous weak supervision analyses extend to their method. As a result, it’s difficult to understand exactly why the proposed method is outperforming the existing approaches. 
* Even if the authors do not present theory analyzing why their proposed method works, there are many analyses they could perform to dig into some of their intuition. For example, the following experiments would be useful to help the reader understand why the proposed approach leads to improved empirical performance:
    * The authors claim that the improved empirical performance could be due to the fact that the label model parameters can adapt to each specific sample. A simple analysis here would be to look at the difference in thetas (the label model weighting parameters) for each sample in your dataset after convergence---how much are the weighting parameters actually changing? Does this correspond with relative improvement you see over baseline weak supervision methods?
    * Similarly, the authors suggest their model may be performing better because the assumptions in the standard weak supervision models don't hold. It would be interesting to run an experiment on synthetic datasets analyzing the relative improvement of the proposed method as labeling sources become increasingly correlated, breaking independence assumptions in the existing models.
    * Additional scaling experiments on the size of the weakly labeled dataset (comparing E2E, Snorkel, and Triplet) would be useful. It’s unclear to me if the proposed method will require more unlabeled data than baseline methods since more parameters need to be learned. The authors seem to be thinking along similar lines, as they hypothesize, “Since our model is based on a neural network, we hypothesize that the large relative lift in performance on the Amazon review dataset is due to it being the largest dataset size on which we evaluate on – we expect this lift to hold or become larger as the training set size increases.” However, they don’t do experiments to validate this claim. It would be relatively easy to use 10%, 20%, … 90%, 100% of the weakly labeled data with the proposed method and various baselines to generate intuition for how each method’s performance scales with more weakly labeled data. 
 * As mentioned by the authors, training the label model and end model in tandem may hamper the labeling function development process. One advantage to the previous “two-phase” approach (where a graphical model is used to aggregate the labeling sources into probabilistic labels, then the weakly-labeled training data is used to train a downstream model) is its flexibility: labeling functions can be developed, iterated on, aggregated, and the resulting weak labels can be analyzed without having to wait for a deep model to train; then, the resulting labeled training set can be used with different loss functions, networks, etc. downstream. By combining the weak label aggregation process with the classification model, users are a more constrained and may not progress through this process as quickly or flexibly. However, the authors also acknowledge this as a limitation in their submission and suggest a reasonable workaround. 
* Some of the motivation for the proposed approach is a little tenuous. Specifically, in the Introduction, the authors set up that existing weak supervision models suffer from the drawback that “the separate PGM does not take the predictions of the end model into account.” However, it is not clear or intuitive to me *why* the label model should take into account the predictions of the end model. Additionally, the authors focus on the fact that “current approaches for estimating the unknown class label via a PGM need to rely on computationally expensive approximate sampling methods, estimation of the full inverse of the LFs covariance matrix…” suggesting that existing tools need to be more computationally efficient. However, the proposed method is to learn an MLP in tandem with a deep neural network from large datasets, which is not particularly computationally efficient. Additionally, there are no experiments on the relative efficiency of different weak supervision methods, so this motivation in the Introduction seems unrelated to the proposed solution. In my opinion, the motivation around modelling assumptions and model misspecification is stronger and probably sufficient. 


**Minor comments and clarifications**
* In Table 1, I believe you are reporting the performance of a downstream model trained on the probabilistic labels produced by Snorkel, Triplet, Triplet-mean, and Majority vote (as opposed to reporting the label model performance). However, I had to look at your code to determine this---it wasn’t clear from the writing. If you are indeed reporting the end model performance in Table 1, that should be made more clear in the writing.
* Line 195-198 was written in a confusing way---it’s not clear to me what dataset splits are used for which datasets, what data was unlabeled vs. labeled. Recommend rewording for clarity.
* It would be interesting to look at the correlations of the probabilistic labels produced by the proposed method and baseline weak supervision methods---is the proposed method catching some particular slice of the input data that the baseline methods are not, or is the proposed method just uniformly better at predicting probabilistic labels? 
* It would be helpful to list the performance of each labeling function and each label model (even if only on a subset of the validation or test dataset) instead of only the final classification score.
* Line 276---what is 2b? Do you mean Figure 2b?
* Formatting: there is no space between paragraphs, which is useful for the reader to guide the eye---recommend adding if possible.
* Some of the citation formatting seems to be off (e.g., [37])


----------------------------------------------------------------------------------------------------------------------------------------------------------
UPDATE AFTER REBUTTAL

I have read through all other reviews and the authors rebuttal. I appreciate the authors thorough responses to my questions and the additional experimental results---the scaling experiments with different sizes of training data were interesting to look at! Adding a bit of related work about the benefits of joint optimization as previously investigated in crowd sourcing would be useful (if it isn’t already there). As brought up by the other reviewers and by the authors themselves, there is more work to be done understanding why this approach works. In my opinion, though, I think the novel approach and strong empirical results are interesting enough for publication at NeurIPS, and I am sticking with my original review score.
limitations_and_societal_impact: Both were discussed appropriately.
review_title: null

Meta-review: This paper proposes a deep learning, end-to-end approach to learning from multiple noisy labeling functions. The proposed method outperforms existing techniques that use probabilistic graphical models for label aggregation. One of the major criticisms of the paper is that the paper does not analyze (theoretically or empirically) why exactly the proposed technique works well. The reviews believe that they have suggested different empirical investigations, but the authors did not engage with these ideas during the rebuttal period. The reviewers were willing to raise their rating if additional empirical investigation about the inner-workings of the technique had been conducted. Other that the lack of analysis, the reviewers do not present a strong criticism of the technique and are happy with empirical results. I do resonate with the authors that deep learning techniques can be challenging to analyze, and my overall reasoning is in agreement with reviewer #zED9 that the novelty of the proposed method along with the improved empirical performance can substitute the lack of theoretical or empirical justification to some extent. Overall, I believe the publication of this paper can help the research community make progress on finding better algorithms for aggregating multiple noisy labeling sources, and better understanding of this algorithm can be left for future work. Hence, I recommend acceptance as a poster.

Decision: Accept (Poster)