Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The submission introduces a new collection of datasets with controlled sources and weak labels + label functions. Initial benchmarks for a variety of models on those datasets are provided. The WRENCH github open-source repo is released along with links to each dataset, model implementations, and well-documented code.

I think the open-source contribution is the greatest strength of the paper. If the weaknesses below are addressed, I would be open to re-evaluating my review of this paper.
strengths: The paper is well written and clear. The open-source WRENCH repo is a useful contribution. Great work to the authors for organizing the datasets, labels, sources of labels, etc. The code is clean, well commented and easy to read with clear documentation. In my opinion, the WRENCH repo is the strongest contribution in the paper. 
weaknesses: tl;dr
The WRENCH benchmark is overshadowed by overlap with prior work. More clarifications for choices made would help. Test sets need to be error-free for the benchmarks to be meaningful -- how did the authors guarantee this? Please add discussion, i.e., what is the takeaway of this paper? What have you learned? What can the scientific community takeaway from your launch of the WRENCH benchmark?

Details:
The paper is well written and makes an appropriate contribution to this track. I only have a couple major concerns about the positioning, overlap with prior work, and scope of the paper. 

Firstly, the paper makes several choices about deliveries but would be strengthened if there was more reasoning for why those choices are made. For example, how did you choose the models you benchmarked? Are there no more recent models? Should other models be benchmarked? Should other datasets? Why focus just on weak supervision methods -- is there any reason why WRENCH is not useful for other methods in machine learning with noisy labels? This scoping decision is stated at L117, but I believe it would strengthen the work if there was scientific reasoning behind this positioning/scope. 

In its current form, about half the content rehashes prior work in weak supervision. While the background is helpful, this actually overshadows the significant value of the WRENCH benchmark, so my recommendation is to either clarify the positioning throughout with reasoning, or shift the focus to be more on the datasets + benchmarks and less on rehashing prior work.

More specifically, the WRENCH contribution is the novelty, but the methods and related work overlaps quite a bit with prior work [77]. The main methods benchmarked were previously benchmarked in prior work, e.g. [78]. The benefit here is benchmarking on more datasets with a single location for downloading and unified sources for labels (which is valuable). For the two forms of LF's introduced, I had trouble disentangling what was a new contribution in this work and what was re-hashing prior work on LF's -- could the authors shed light on this?

I'd like to see a greater emphasis on the WRENCH benchmarks and clear future direction. The idea of a benchmark is that it stands the test of time. How will WRENCH stand the test of time? 

I may have missed it, but how is test data handled?  Was an error-free labeled test set released along with each WRENCH dataset? I see the test set accuracies in the paper and in the release here: https://github.com/JieyuZ2/wrench, but no explanation for how the authors guarantee the test labels are error-free? This seems necessary for the benchmarks to be meaningful and this paper to be accepted (perhaps I just missed the explanation for the test sets in the paper). 

The evaluation section 6.2.2 of the paper is descriptive, whereas I was hoping for deeper introspection into the findings. What did launching WRENCH unveil so far for the scientific community? What have we learned? Please go deeper than which model did best on which dataset which is the current discussion section. 

Perhaps its just me, but I found the delivery of Section 5 confusing and a few more clear examples of each labeling function might help.

Optional feedback -- Are there plans to launch a competition for WRENCH? I would think it would couple nicely with the contribution, albeit this is unnecessary for the paper to get accepted.

correctness: Yes.
clarity: Yes, the paper is well written.
relation to prior work: Yes, I actually think this paper spent too much time on prior work (a rare feedback!). I'd prefer to see more time spent on what are the key takeaways and findings of their WRENCH benchmark. I think a greater emphasis on novel contributions would greatly strengthen the paper.
additional feedback: L58 would be greatly strengthened with the addition of examples.
L126 - for future promises, maybe instead focus on how will this benchmark serve the field for years to come.

I'd consider adding a future work section or a 'role of this benchmark going forward' section at the end (or even just as \paragraph section in the discussion) to clarify the role of this benchmark going forward.
review_title: A well-written paper and useful open-source contribution: the paper would be greatly strengthened by focusing more on the novel contributions, takeaways and findings of the WRENCH benchmark.

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces a well motivated and documented framework for evaluating weak supervision methods. The framework is released as a repository on Github based on Torch and many other dependencies, with a modular structure that should allow easy evaluation of different datasets, labeling models, and end models. The paper presents a comprehensive demonstration of the framework for evaluating 22 tasks including both classification and sequence tagging in both NLP and vision and across many domains. The paper also provides a good overview of the WS field, summarizing many different techniques such as two-stage, joint training, and applying the labeling functions on the test set; soft vs. hard labels; etc. This makes it a good resource for researchers that are new to WS. One thing that is mentioned but is not covered, is utilizing a small set of "clean" labels. I agree with the author's stand that this can create a more noisy evaluation setting, but it would be interesting to explore this since I think that in the real world it is mostly easy to obtain a small set of labeled examples.
strengths: * The paper is very comprehensive, experimenting with 22 diverse tasks and many methods.
* The motivation is very clear and it will be great if indeed this work will create a unified framework to standardize WS evaluation.
* The writing is overall clear and provides a good resource for learning about WS (though I think some parts can be improved, see below)

weaknesses: * After the large amount of experiments, I don't see a clear cut winner between all of the presented methods. It would be good to also try to aggregate the scores across the different datasets. At the end of the day, the motivation for WS is the lack of supervised data, so in the real world we might not be able to do retrospective study and reliably evaluate the performance on our target task. Therefore, it could be useful to use this framework to get some incites on what method is the best to try (maybe domain/ task group dependent if there is no "winner takes all").
* Most of the paper is very clear, but I think section 5 could be better explained. It feels to me like (1) it is a bit out of the flow of the rest of the paper and as it is unclear how it relates to sections 4 and 6, and (2) at least for readers without background in WS, some of the procedures and terms are new and unclear. For example, how is the accuracy controlled if there is no labeled data, what does the radius of accuracy exactly mean here. For procedural, how come we are expecting to have ground truth training labels if it's a WS setting?
* Some claims in the introduction should be supported with references (e.g., line 55: WS datasets vary ..., line 63: it is not uncommon to see...)
*minor:
1. Some Abbreviations in tables and figures can be avoided to make it clearer to read as I think there is enough space (fig.2 - average, overlap; tab2: abbreviation; Tables 3-4: does "-" means applying the LF on the test set?)
2. Why does Spouse on Table 3 doesn't have best gold?
3. AUC shouldn't have %
correctness: other than the mentioned comments, it looks good to me
clarity: most of the paper is clear and the repository also seems to be in good shape. Only see my comments on section 5 and minor points.
relation to prior work: Gives a good overview. Only comment - to support the claims in the introduction with references
additional feedback: What label functions are used for the experiments in section 6? Are there procedural functions that use labeled data? If so, what is the performance of using this labeled data for training instead of LF generation?
review_title: A comprehensive benchmark for Weak Supervision. 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The paper describes a dataset suite and synthetic benchmarking methodology for weak supervision. It starts by describing motivation for weak supervision and how it works, emphasizing the useful notion of labeling functions. It then explains that evaluation of WS approaches is inconsistent and flawed, and introduces the main CONTRIBUTIONS: a combination of a standard dataset suite and synthetic benchmarking methodology to address this need. The authors give an overview of related work then narrow focus to classification and sequence tagging problems.  They introduce their suite, consisting of datasets and labeling functions, all accessible through unified interfaces. Of particular interest, they describe a set of synthetic labeling functions that enable systematic exploration of WS model performance for a range of labeling function characteristics. They provide examples using these labeling functions to explore WS performance. Next, they present a series of experiments showing performance of leading WS approaches on their suite of datasets.  
strengths: This paper address a clear and lasting need for an important area of machine learning: quality evaluation of WS models. Their approach intelligently combines real benchmarks, which keeps evaluation grounded in actual use, with synthetic benchmarks, which enable  quantitative space exploration. 


weaknesses: This paper contributes and important tool for the future WS field. As such, I would *really* like to see the authors hit it out of the park from an end user perspective. In particular:
-- perform a statistical analysis of the real labeling functions in terms of the synthetic labeling functions, to enable understanding of typical parameters for the synthetic functions
-- provide a guide to interpreting results and using them to select an algorithm for a particular problem classes


correctness: Yes
clarity: Paper is clear and concise.

From an explanatory perspective, the paper assumes understanding of what is being measured and would benefit from a simple statement of the problem in the introduction: "Our goal is enable consistent evaluation of the accuracy of WS models is determining ground truth labels from a set of weak labeling functions, across a range of real and synthetic data sources."

relation to prior work: The work has a nicely organized related work section on WS in general. The section does not call out any specific work on WS benchmarking, which may be because there is none: hence the need the authors are addressing.
additional feedback: Please continue to build on and enhance this effort beyond the initial publication. We need more work like this.
review_title: Strong paper on systemizing weak supervision benchmarking

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper introduces a rather large-scale benchmark of weak supervision methods across a wide variety of domains and datasets (22 in total!), roughly taxonomized as types of classification tasks or sequence tagging tasks. The authors identify a number of problems with the way that weak supervision methods are currently evaluated, including the use of private datasets, differences in label functions across evaluations, and the brittleness of evaluations on complex and ad-hoc evaluation pipelines. The authors provide a large set of baseline results on their benchmark which include two-stage methods (separately training a label model and an end model) and jointly trained models (one-stage methods). There are a number of hyperparameters common across methods, including whether to train on label probabilities or on hard labels – the authors include a full sweep over these choices (which amounts to thousands of method/dataset pairs) and report average performance on the top 3 methods on each dataset. 
strengths: The authors do a solid job of exposing the need for an extensive weak supervision benchmark, and a very thorough job in their evaluation (which, if I’m understanding correctly, amounted to them running O(thousands) of method/dataset evaluation pairs). The number of datasets included in the benchmark, 22 in this case, is also a strength. 

Additionally, the authors include a suite of real and synthetic label functions which can be used to evaluate the behavior of weak supervision methods under various LF accuracies and propensities (a measure of non-abstention), which seems to be another novel contribution. These synthetic LFs are derived from the ground truth labels themselves, which offers a great deal of flexibility in terms of modeling the types of interactions seen in real LFs. For the evaluation of methods, however, it seems that the LFs are fixed to be standardized per-dataset so as to standardize evaluation, which seems like the correct thing to do. 

Overall, this benchmark seems highly relevant and impactful to the weak supervision community, and the breadth of the benchmark and evaluations make it a significant contribution to said community.

weaknesses: If my understanding is correct, the majority of the datasets in the benchmark are text datasets (except for a few video datasets). This is understandable because it’s probably difficult to write label functions for image classification problems – although, Table 1 lists “image classification” problems as a type of task, whereas only video classification tasks are provided. It would be nice to have at least one more standard image classification task. 

It also seems that for most of the tasks, the number of classes is pretty small (usually 2 classes). The largest number of classes, per Table 1, is 18. It would be nice to have at least one task with a large number of classes (though this might understandably be difficult for weak supervision in general).  

In any case, the set of datasets included in the benchmark is probably reasonably good enough for a first benchmark in this subfield.

correctness: I didn’t verify any of the experimental results, but they seem like they’re most likely correct. The claims in the paper seem correct to me. The evaluation methods seem very reasonable to me.
clarity: This paper is well-written and seems clear enough to me. 
relation to prior work: The authors discuss prior work in weak supervision. I’m not aware of any prior work on *benchmarking* weak supervision, which is one of the reasons why I think this paper is an important contribution. 
additional feedback: It wasn't immediately clear to me that the benchmark could actually be used to evaluate LFs themselves until I got to section 5. I think this is a useful contribution and could be highlighted more prominently earlier in the paper as well.
review_title: A valuable contribution to weak supervision

Meta-review: The paper proposes a comprehensive benchmark for weak supervision, consisting of 22 datasets for classification and sequence tagging from various sources (text, videos). It also provides implementations for popular WS methods and an extensive analysis of their performance. All the reviewers agree that this is a very useful contribution for the community. The authors updated their manuscript to take into account reviewers' comments, and pledge to continue extending the benchmark, with 2 new methods recently added.
Decision: Accept