Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This work benchmarks recent automated weak supervision methods on seven datasets spanning multiple modalities. The authors present a head-to-head comparison of AutoWS methods with special motivation from the rise of foundation models---the relationship between pretrained models and AutoWS is only just beginning to be explored. The results indicate that foundation models unsurprisingly may not be helpful on out-of-distribution data.
strengths: This paper has many strengths:
* There is a need for this study in the community and future works may find value in the experimental findings
* The experiments span a range of settings and findings can likely inspire future studies and empirical guidelines
* The experiments cover multiple data modalities
weaknesses: This paper could be improved with the following considerations:
* The motivation flip-flops a little bit: sometimes the authors say LFs are super cheap and easy (Line 27 and 72, for example), other times they are expensive and hard (Line 37 for example). I recommend sticking to a clearer story or qualifying these statements a little more.
* The experimental results section could be strengthened by further interpretation of why different methods may be performing better or worse in different cases.
correctness: The claims seem correct
clarity: The paper is largely written well. 
relation to prior work: Relation to prior work is clear
additional feedback: In the experiments section, I suggest putting tables and figures on the same pages where they are mentioned. Otherwise, it's hard to flip back and forth to keep the points clear.
review_title: New benchmarks for auto weak supervision

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Automated Weak Supervision (AutoWS) methods aim to bring a step further WS methods by providing automated approaches to create Labeling Functions (LF) which are then used to build labeled datasets for training supervised models.
The authors introduce AutoWS-Bench-101, a benchmark for automated weak supervision methods. The benchmark aims to be a framework to allow comparison among multiple AutoWS techniques and investigate the effect of its components such as the representations used in automated LFs.
The benchmark offers multiple implementations of AutoWS methods s.a. Snuba and GOGGLES and a wide selection of a datasets to experiment with, mostly in the image domain. 
With this benchmark the authors then perform a series of experiments to better characterize AutoWS methods. These include for instance assessing the impact of using foundation models (pre-trained models such as ResNet) as labeling functions and how AutoWS compare to (a) few-shot learners (which can be foundation models themselves) (b) hand-designed LFs.
strengths: Considering the potential impact of automated weak supervision in many domains, especially when using "foundation models" as labeling function co-operating w/ other sources, I find the authors' contribution, i.e. a benchmark for such techniques, very relevant for the research community.  The benchmark allows to easily perform experiments to better characterize these techniques. The codebase is well structure providing all the necessary abstractions to quickly set up an experiment environment and easily allowing its extension for future work and research directions.
weaknesses: line 105-107: while explicitly explained by the authors, I still think that the benchmark should include one or more datasets for text applications. This is because "foundation models" are extremely popular in NLP (in the form of large pre-trained language models) and allowing to investigate their impact in AutoWS setting would make the benchmark more "complete".  With this being said, if the authors find that this is already explored elsewhere, (a) pointer(s) for the interested reader should be provided.  

correctness: As far as I understand it the evaluation methods and the experiments performed are appropriate to the type of research questions investigated.
clarity: The paper is very well written and I could follow all the arguments (almost) without effort.

There are some minor adjustments that in my opinion would make it even better:
- Table 1: Caption should explicitly state what is being measured (what is performance in this case). Which split (test/dev)?
- line 150: the few-shot learning baseline is introduced almost incidentally in section 3.2 "implementation details".
Arrived at section 4 "Analysis", a reader like me either did not notice it or has already forgotten this detail.  For clarity this should be repeated in 4, or maybe in the caption for Table 1.
- line 269: cardinality is defined here but is mentioned before as well in line 166.

relation to prior work: The prior work is cited correctly and the authors clearly define the contributions their work present.
additional feedback: 
- line 164-165, and 168-169: for IWS "we use Snuba’s LF synthesis procedure to generate a large set of LFs" but then "we run the Interactive WS LF selection procedure once per class to produce unipolar LFs". I am not familiar w/ this frameworks, but if Snuba's can be adapted to be multi-class, why not use the multi-class version for IWS?

- line 218-219: I must admit I am confused about this, but this may be due to my lack of a thorough background in this domain. 
What is the motivation behind using features from an image model like ResNet for non-image datasets? Is this "common" in the (auto)WS setting? Does this make sense at all? What is really telling us the fact that a logistic regression model on top of ResNet logits (a model trained on natural images!) can achieve an accuracy of 0.47 on an electrocardiogram dataset?

- line 260-262: maybe I am missing something but why is this the case (setting aside the previous point)? Results are reported on ECG and EMBER w/ ResNet features, and from my understanding CLIP is still an image model like ResNet, simply needing a natural lanugage prompt.
review_title: AutoWS-Bench-101: benchmark for automated weak supervision

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposed a benchmarking framework for AutoWS --- AutoWS-Bench-101 on a diverse field of tasks. They provided the baseline experimental results of different AutoWS methods on seven datasets. They also conducted the experiments utilizing different foundation models and multipolar LFs to observe the pros and cons. 
strengths: 1. This paper proposed a benchmarking framework for an important research line of methods --- AutoWS. 
2. This paper provided the sufficient details e.g., the SOTA baseline experimental results and all experiment codes. 

weaknesses: 1. This AutoWS-Bench-101 benchmark contains several image datasets but only one dataset in other domains. There are many other time-series and tabular datasets. The conclusion for some field based on the experimental results of only one dataset is not so convincing. The authors are suggested to add more datasets in each domain except for image classification. 
correctness: The evaluation methods and experiment design are appropriate. 
clarity: This paper is generally well-written. I only have several minor comments. Several examples are as follows:
1. Table 1 and one of the descriptions about Table 1 in the main body are suggested to be in the same page. Table 1 is in page 5 now and the text in page 5 doesn’t have the description about Table 1.
2. There are many “N/A” in Table 1 & 2. The authors are suggested to provide some explanations about them in the captions or footnotes.
3. In line 105-106, the authors said “…We are especially interested in settings far from the ambit of conventional weak supervision---most often used in text applications…”. From my experience, weak supervision is also often used in Computer Vision and other applications. We can discuss the correctness of this claim.

relation to prior work: This paper clearly described the relations to prior work. 
additional feedback: The authors are also encouraged to add NLP datasets in this benchmark. But it’s minor.
review_title: In general, it is a good dataset paper. But the authors are suggested to add more datasets. 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper proposes a benchmark for automated weak supervision that designs
labeling functions. The authors describe the problem and data and present
empirical results.
strengths: + interesting problem
+ well motivated
weaknesses: - conclusions seem to be based on essentially single data point
correctness: Correct as far as I can tell.
clarity: Yes.
relation to prior work: Yes.
additional feedback: The paper is well-written and motivates the need for this benchmark set. The
main area where it falls short is the empirical evaluation, which seems to be
based on a single foundation model, which makes the conclusions that the authors
draw with respect to the goals of their evaluation somewhat tenuous.

That said, the proposed dataset is interesting and should be useful to
researchers in practice.

The caption for Table 1 should explain what the numbers mean.

Update after rebuttal:

I believe that my main point has been addressed with the extended evaluation and am happy to recommend acceptance now.
review_title: Interesting benchmark

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a novel benchmark, AutoWS-Bench-101 for evaluating different AutoWS methods. Different from the previous WS benchmark WRENCH which evaluates different label models and end models with some heuristics as the labeling function, this benchmark aims to generate weak supervision sources directly from the feature space. This benchmark contains massive experiments with zero-shot predictions from foundation models, and evaluates with multiple AutoWS methods, including very recent approaches. The codebase and datasets have been publicly released.
strengths: 1. The paper studies a very interesting topic of automatic weak label generation. To the best of the reviewer's knowledge, this research topic has not been widely explored in previous studies. It will be great if indeed this work will create a unified framework to standardize AutoWS evaluation.
2. The paper provides extensive experiments for evaluating different methods with detailed discussions on the performance of different AutoWS methods.
3. The writing is overall clear and easy to follow.
4. The code has been released with documentation. As it is adapted from the Wrench benchmark, it will be easy for WS researchers to follow this new benchmark.
weaknesses: 1. For the background part, I think it would be better to give a detailed description of the AutoWS setting, as it is very different from the traditional WS task (in Wrench paper). I recommend the author use some mathematical notations to define the problem formally.
2. It would be better to extend the tasks to other domains such as NLP.  Is it possible to run some of the baselines by using the datasets in the Wrench benchmark?
3. The few-shot baselines seem to be very weak (using logistic regression only), it would be better to use more advanced techniques.
4. The analysis for the snuba method is interesting. Is it possible to give more analysis for other baselines? 

correctness: Yes
clarity: Yes, most of the parts are clear and easy to follow. One recommendation is that the authors could try to use more mathematical notations/equations to better describe the problem setup and baseline methods.
relation to prior work: Yes, this paper introduces the related work in detail and covers most of the recent works. 
It would be even better to discuss some autoWS methods for NLP (listed below):
- Adaptive rule discovery for labeling text data (SIGMOD 2021)
- Prompt-based rule discovery and boosting for interactive weakly-supervised learning (ACL 2022)
- Nemo: Guiding and contextualizing weak supervision for interactive data programming. (arXiv:2203.01382)
- Automatic Rule Induction for Efficient Semi-Supervised Learning (arxiv 2205.09067)
additional feedback: In table 1, it remains unclear how to use ResNet-18 for ECG and Ember since they are not standard image classification tasks. Could the author elaborate more on this?
review_title: A useful benchmark for automatic weak label generation

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The paper introduced a benchmark for evaluation of automated weak-labeling methods called AutoWS-Bench-101. The evaluation included multiple methods and features used for data, on multiple image and non-image classification tasks. Compared to a prior benchmark on the same (WRENCH) however, the current benchmark is significantly smaller, however using different features as an additional aspect of analysis.
strengths: 1. The paper adds an additional analysis aspect of different features for weak supervision that was previously unexplored.
2. The comparison using and findings regarding the use of foundation models for weak supervision are quite relevant to subsequent research on foundation models.
weaknesses: 1. The benchmark is significantly smaller in number of datasets and methods compared to WRENCH, an existing benchmark for Auto-WS evaluation.
2. Not all methods used were analyzed further. For instance the paper focused much more on the analysis of the method Snuba.
3. Paper has problems with clarity (see below). 
4. Other than the paper's contributions, it is unclear why one may use this benchmark for for evaluating an Auto-WS method over WRENCH. Ln 80 saying "understanding mechanics" is somewhat vague.
5. Not a deal-breaker, but the paper can benefit from additionally using semi-supervised learning methods as a baseline besides simply using few-shot learning.

Most questions that were unclear from the paper have been mentioned under "Clarity" below.
correctness: Most evaluations seem correct apart from from some minor questions regarding CLIP in Fig 2 (see below).

clarity: The paper is not very easy to navigate. Following are some questions that were unclear : 
1. It seems CLIP embeddings were not used for ECG and EMBER datasets. How can CLIP be compared with other methods in Fig 2 on the right?
2. Ln 116 : What was Resnet-18 pre-trained on?
3. Ln 221 mentions CLIP features help on in-distribution tasks, but they help on spherical and permuted MNIST as well, which are not "in-distribution"/natural images. Maybe the conclusion from this experiment needs to be more nuanced.
4. Could not find details regarding how the few-shot baseline abstained. 
5. Table 5 : Unclear why the Navier-Stokes class descriptions are so different.

Other comments:
1. Table 1 would be much clearer with a mention of what the metrics are. It is easy to think the metric is error on looking at Table 1 after Sec 4.1.
2. Brief descriptions of methods evaluated, including brief descriptions of labeling and abstention, can add a lot to the paper's readability

- One typo that I spotted : ln 172 : remove "with" 
relation to prior work: Not quite (see pt 4 under weaknesses above). Also, the paper could benefit from a discussion of how few-shot learning or semi-supervised learning compares/contrasts to Auto-WS methods.

additional feedback: All covered above
review_title: Benchmark for weak supervision - could be more comprehensive

Meta-review: metareview: All reviewers except one recommend acceptance. The reviewer recommending rejecting the paper (score 5) raises valid concerns, e.g., the lack of semi-supervised baselines and the comparison to a prior benchmark for weak supervision (WRENCH). The authors addressed some of the concerns (e.g., by adding a semi-supervised learning baseline). Regarding the comparison to WRENCH: It is true that WRENCH is similar, but the exact problem benchmarked by WRENCH and AutoWS-Bench-101 is different: in the former, the labeling functions are assumed to be given, in the latter, the labeling functions are learned. I find this difference sufficient for a separate benchmark.

Overall I recommend accepting the paper. I strongly encourage the authors to take all the reviewer comments into account, especially the comparison to baselines form semi-supervised learning. As I understand, automatic weak supervision fits the same abstract problem statement as semi-supervised learning. Hence a comprehensive comparison to semi-supervised learning is essential.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept