Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a method to improve the precision of existing weak supervision approaches by selecting a subset of the weakly-labeled data for training the end model. Specifically, the paper proposes to use the cut statistic that considers vector representations of the input examples, and keeps examples with the same pseudolabel as their nearest neighbors in this vector space. These examples are identified by constructing a graph where each instance is a node, computing cuts using weak labels, and identifying the number of cut edges for each node. 

The proposed approach is evaluated across several benchmarks for weak supervision and compared with an entropy-based measure for subset selection. 
strengths_and_weaknesses: Strengths: 
* The paper is very clearly written and easy to follow. 
* The paper proposes a simple method that effectively improves the precision of existing weak supervision methods across several datasets. It is a great benefit that the proposed method can be applied in general across label models, end models, and data modalities, and as a result, can serve as a useful tool for the weak supervision community. 
* Additional experiments, such as evaluating the choice of the representation for the cut statistic are very interesting. 
* The paper provides a theoretical analysis on a special case of weak supervision that motivates subset selection and can help better understand the precision-coverage tradeoff. 

Weaknesses: 
* The paper experimentally compares the cut statistic approach to just a very simple entropy-based approach and does not consider existing work that attempts to improve the precision of the existing sources, e.g., via instance-specific weights (Awasthi et al., 2020; Karamanolakis et al., 2021; Yu et al., 2021). A more thorough experimental comparison would better place the paper in the context of state-of-the-art approaches and would make the paper's contribution stronger. 

Minor comments: 
* Section 3.1 could be a better fit under the "Experiments" section.

References: 
* Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi, "Learning from rules generalizing labeled exemplars", (ICLR 2020)
* Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, Ahmed Hassan Awadallah, "Self-training with weak supervision", (NAACL 2021)
* Yue Yu, Simiao Zuo, Haoming Jiang, Wendi Ren, Tuo Zhao, Chao Zhang, "Fine-Tuning Pre-trained Language Model with Weak Supervision: A Contrastive-Regularized Self-Training Approach", (NAACL 2021)
questions: Are the authors planning to share their code for replicating the experiments reported in the paper (in addition to the code snippet in the appendix)?  
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors propose a simple method for combining pretrained data representations with the cut statistic to select better quality subsets of the weakly-labeled training data. By selecting subsets of data rather than using the full covered data, a weak supervision label model is able to achieve better precision. The authors also show that their method also improves performance of end models.

strengths_and_weaknesses: Strengths:

1. The method is simple and easy to integrate with any weak supervision approach.

2. The authors show strong empirical results and provide theoretical guarantees for their approach.

Weakness:
 - Selecting good value of \beta may not be practical
questions: I like the direction of this work and the potential gain to weak supervision however I think a major flaw in the paper lies in selecting a good value for \beta. In the paper, the authors use a validation set to select the best \beta for the results in the table. This will not be practical when a validation set is not available and if the cost of labeling the data to obtain a validation set is high. Looking at the plots, there is no obvious alternative strategy for selecting \beta as it depends on the classification task. I think it will be very helpful if the authors can suggest an approach for selecting \beta when there is no access to validation data and maybe show experiments for it.
limitations: The authors addressed potential limitations of their paper. The work has no significant negative societal impact.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies the subset selection problem of weakly supervised learning. By dividing a weakly supervised learning pipeline into labeling functions, label model and end model, the paper specifically studies on label model which motivates the proposed subset selection method based on cut-statistics. Different from entropy-based subset selection, cut statistic based selection relies on identifying highly noisy examples based on their proximity in a constructed feature graph. The proposed selection method serves as a plugin for any label model and empirical evaluations are presented to show that it improves many existing label model and end models.
strengths_and_weaknesses: Strengths:

1. Using the cut statistics to identify highly noisy labels is an elegant and novel idea, which doesn't require any knowledge about true labels. Justifying using less data may actually lead to better performance in weakly supervised learning carries the most contribution to me.
2. The proposed selection approach is simple but yet very effective which improves many existing label model and end model. 
3. The paper is extremely well presented ranging from motivation, formulation, method coverage, empirical and theoretical justifications.

Weakness:
I find no obvious weakness of the paper and it's a very enjoyable read.
questions: A minor question regarding connection to semi-weakly supervised learning: to get best performance on a particular task, a validation set is required to set the selection threshold \beta, then this essentially leverages information about true labels, I wonder how does the proposed method compare to weak label re-weighting / correction method. e.g., [1][2] (some of these require a small set of clean labels for learning, and for the studied setting in this paper weak labels + validation data, for a stretched setting we can split the validation data to use partially for training and the rest for validation and thus apply the above mentioned methods.)

[1] Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting. NeurIPS '19
[2] Meta Label Correction for Noisy Label Learning. AAAI '21
limitations: Potential limitations are well addressed.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper proposes a simple dataset selection strategy for weakly-supervised
learning. The strategy is based on the cut statistics, which constructs a
similarity graph among the instances and select the nodes that have consistent
labels with their neighbors. The authors have shown such a simple data
selection strategy enhances previous labeling models for weakly-supervised
learning. They also provide a theoretical analysis on the generalization
bound, which formulates a tradeoff between precision and coverage for weak
labels.

strengths_and_weaknesses: 
Strengths:

S1. The proposed selection strategy based on cut statistics is simple but effective empirically.

S2. The authors have conducted experiments on datasets with different modalities, including text, image, and tabular data.

S3. The paper is well written and easy to follow.


Weaknesses:

W1. There is a gap between the theorem and the experiments. In the theorem,
the author assumes two different views that are conditionally independent. The
labeling functions are defined on one view, and the classifier is trained on
the other. Unless I'm missing anything, there are no such complementary views
on the datasets used in experiments. Also, even if we have multi-view data
sets in practice, the assumption that the two views are conditionally
independent, and that we use the two view separately for labeling function and
model training are impractical.

W2. The authors mainly evaluated the selection strategy for labeling model.
However, as the authors mentioned, there are works (e.g., cosine) that perform
data selection during the end model training. I do not see any major obstacles
that prevent the proposed data selection strategy from being applied during
the end model training. It'll be more interesting to see how this measure compared with the previous methods that uses data selection during the end model training.


questions: 
Please see the weaknesses above.

Besides the above two concerns, an additional question is that there are
recent work [1] showing relabeling is better than discarding data in the
context of supervised learning. It'll be great if the authors can comment on
these two seemingly contradictory findings.

[1] Resolving Training Biases via Influence-based Data Relabeling


limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact.
review_title: null

Meta-review: metareview: This paper examines the question of whether it is best to use all the available weakly labeled data to train a model. Contrary to usual practice, it finds that it is often best to filter that data. The creativity of the paper is using a statistic called the cut statistic to select high quality subset of the data. It uses a graph where nearest neighbors of examples are connected using the distance induced by some representation. The approach is very elegant. It can be plugged into many existing approaches. Experiments on the WRENCH benchmark for weak supervision show that this selection method consistently improves five different approaches to creating weak labels.

The reviewers all agreed that the paper makes a strong contribution, is clear and well written, and provides an interesting analysis of why the method works. 
confidence: Certain
Decision: Accept