Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose the WARM method to help conduct iterative and interactive weakly-supervised learning. Active learning is used here to refine the labeling functions by focusing on data points that are once labeled. The authors further incorporated gradient propagation to alternatively update the LF parameters and the DP model. Experimental results show that the WARM method can improve the quality of training data.
main_review: Strength:

The proposed method is clearly described and the writing is easy to follow.

Weakness:
1. The theoretical analysis is insufficient and I recommend the authors provide more analyses on their main contributions.
2. The general idea of active learning with weak supervision is not novel, which can be seen in [1]. The methods proposed in this work are combinations of existing ones and the authors also fail to contribute novel theoretical results.
3. The layout should be improved. I recommend the authors separate the key notations, e.g., labeling functions, from the contexts for better presentation. There are also some typos, e.g., 'dependant' should be 'dependent' in the last paragraph of page 2.

[1]. Chicheng Zhang and Kamalika Chaudhuri, Active Learning from Weak and Strong Labelers. NIPS 2015: 703-711

summary_of_the_review: The authors should compare with more related works and provide some theoretical analyses to make this work more convincing. The ideas are totally heuristic and the experimental results are also not satisfying.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes WARM, an active learning approach to weakly/programmatically supervised learning.  In the WARM approach, which bases off of the data programming/Snorkel paradigm for weak supervision, users write labeling functions (LFs) to programmatically label training data; these labeling functions are then modeled by the Snorkel framework for weak supervision and used to train downstream models.  In the WARM setup, these LFs are assumed to be, or cast as, differentiable.  The paper then proposes an active learning approach to sampling labeled data points to tune the parameters of these LFs, and validates this approach on several medical datasets.
main_review: Strengths:
- (S1) This paper tackles an important problem with an intuitive approach of complementing recent programmatic/weak supervision approaches with expert feedback via an active learning-style approach.
- (S2) This paper introduces a clean formulation of / argument for LFs being cast as differentiable functions, whereas to date most LFs have been non-differentiable
- (S3) The paper shows some strong results relative to recent approaches.
- (S4) The paper includes a range of datasets from synthetic (data + LFs), to 'semi-synthetic' (real data + synthetically generated LFs), to a real EEG task/dataset + LFs developed in conjunction with medical SMEs- an impressive and real world-relevant contribution.

Weaknesses:
- (W1) Simple method: While not a major drawback in isolation, it is worth nothing that the proposed approach is fairly simple and standard from a methodological/algorithmic standpoint.  From an active learning perspective: the query function is just based on model uncertainty, which is the most basic type of active learning query function (the only tweak being that it is the label model, i.e. model over LFs, but this does not change anything from an algorithm perspective).  Then, these data points are used to tune the parameters of the LFs in a manner that is also straightforward (the only tweak here being that the approach alternates between two formulations of the label model objective which is not necessarily needed... see W#1.a).
- (W1.a) The authors state that the data programming label model is not differentiable with respect to the tunable LF parameters introduced in WARM, which is not true.
- (W2) Lack of exploration of effect of differentiable LFs: Given the above lack of methodological novelty, this reviewer at least saw one of the main points of novelty and overall contribution of the paper being around the differentiable LFs themselves, and the overall setup here.  However, unfortunately this contribution was not explored in any depth (e.g. what are the tradeoffs of "softening" LFs to make them differentiable?  How should we think about this more broadly beyond the medical settings treated?  etc), which would have been interesting and significantly strengthened this aspect of the overall contribution.
- (W3) Lack of relevant ablations: In general, there were a range of ablations of the overall approach I would have thought natural- for example, what is the impact of an active learning setup vs. just using some randomly sampled labeled data to tune the LF's internal parameters?  Could these internal parameters also be learned without labeled data, following the basic Snorkel modeling approach, and how would that do?  How would the approach do without tuning the internal LF params?  Etc.
- (W4) Weak/improper comparisons: Since two of the other approaches compared to have no access to tune the internal params of the LFs and WARM does, this seems like a somewhat handicapped comparison...
summary_of_the_review: Overall, the proposed approach introduces some interesting and practical ideas with some exciting experimental applications, however does not sufficiently explore the most novel elements of the contribution, ablate them sufficiently, or compare them to prior methods appropriately.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method for data programming, i.e., using weak supervision to generate probabilistic training labels for unlabelled points using heuristics devised by domain experts. In particular, the authors propose WARM, a framework for iteratively improving these weakly supervised models by modifying the parameters of labeling functions and directing users to a subset of data points that, when labelled, would most improve the model.

main_review: Pros:

1. The proposed method would be of great use in many real-world scenarios where labelled data is scarce (e.g. medicine).
2. To my knowledge, the authors’ proposed method (i.e., actively refining the voting weights of each labeling function and the parameters of the labeling function) is novel.
3. Generally I found the writing in the manuscript to be of high quality. As someone who is not an expert in data programming I found the manuscript easy to follow with some minor exceptions (see point 3 in “Cons”).
4. The authors experiment with their method on a variety of datasets, including one for which they use human domain experts to craft labeling functions. I greatly appreciate the application to a real-world scenario!

Cons (listed in order of importance to my score):

1. I found it difficult to assess the significance of the knowledge shift experiment results presented in Figure 2 and Figure 3 due to a lack of any results from baseline models. As such, I would appreciate it if the authors could add results from their baseline models to these Figures to (1) illustrate the severity of the knowledge shift problem and (2) (potentially) better illustrate the advantages of WARM over previous work.
2. As mentioned by the authors in Section 5, the active learning baseline outperforms WARM on half of the tested datasets. The authors say that this is due to some datasets being “simpler” than others, though I was not clear as to what “simpler” meant here. I would be willing to raise my score if the authors could provide experimental results that clearly illustrate the reasons behind the poor performance of WARM on these “simpler” datasets. For example, if “simpler” means having fewer data points, an experiment that assesses WARM's performance on simulated data with varying dataset size would be helpful to better understand when WARM may outperform existing methods vs. when it may underperform them.
3. Some of the terminology relating to the problem formulation/labeling functions is not precisely defined within the manuscript and can be confusing for readers not already familiar with data programming (e.g. I had to look outside the manuscript for a definition of the “polarity” of a labeling function).
4. The literature review in Section 2 is very data-programming specific, and does not discuss other recent approaches to active learning (e.g. [1,2,3]). Expanding the related works section would help put WARM in a broader context and assist the reviewer in assessing the significance of WARM.

[1]: “Learning Active Learning from Data” (NeurIPS 2017)
[2]: “Learning Algorithms for Active Learning” (ICML 2017)
[3]: “Active Learning with Partial Feedback” (ICLR 2019)

summary_of_the_review: Overall I enjoyed reading this paper. The authors’ method appears methodologically sound and it seems to provide considerable benefits when applied to complex datasets. However, there are some weaknesses in the manuscript that somewhat undermine the paper’s story. For now I am recommending a weak accept, and I am willing to revise my score upwards if the authors address my concerns.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an algorithm for choosing a small set of labels to improve labeling function model performance both directly and for downstream tasks. Additionally, the authors provide a general method to convert standard labeling functions to "soft" labeling functions which are differentiable with respect to some parameters (e.g. a threshold). If the labeling functions are differentiable, this paper provides a method to update the labeling function parameters. Finally, experimental results show that the method introduced outperforms other active labeling approaches for weak supervision.
main_review: In equation (1), the \propto seems like the wrong relation since the left hand side is actually the softmax of the right hand side.

Why are the labels not used in estimating the labeling function accuracies? 

In figure 3.(i), why is the performance not monotonic? In particular, the curve without noise peaks with 20 labels and then starts to deteriorate.

It's interesting to note that WARM's downstream performance never improves over the accuracy of the label model. It seems that a piece of the weak supervision pipeline is broken here. Perhaps using a more expressive model (random forests?) would be more appropriate for the downstream model.

I'm surprised that the active learning baseline generally outperformed the weak supervision methods even though the weak supervision methods have access to extra information (the labeling functions trained on the whole dataset).




Minor things:
I think the arguments of p_\theta are swapped throughout the paper. Look at eq (2), eq (3), and line 4 of the algorithm.

summary_of_the_review: The method proposed in this paper does not contain any particularly novel ideas and seems to be based on heuristics (maybe the proposed quantities could be derived from more general principles?). Additionally, it appears that weak supervision is not appropriate for the paper's empirical settings as seen by the lack of improvement from the downstream model and the stronger performance of non-weakly-supervised methods (active learning).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper gives a method to iteratively and interactively improve the label model in weak supervision. The approach consists of two steps, first is standard weak supervision way of weighted combination of labelling functions to generate labels. Novelty and improvement mainly comes from the second step, where true label for most uncertain data point is queried using which the parameters of the labelling functions are improved, which in turn lead to a more accurate weak supervision model. A key requirement and assumption in this paper's setup is that labelling functions are given by some learnable parameters ( i.e. they can be differentiated w.r.t. their parameters), which allows parameters updates using the true labels acquired. Empirical results on various real world datasets in medical domain show that in some cases this approach can yield a more accurate model in comparison to pure active learning approach and some recent baselines which combine weak supervision with active learning. These results also show that the paper's approach can get accuracy comparable to fully supervised model as well.  
main_review: The main strengths of the paper lies in effectively combining active learning and weak supervision to generate high quality labels. The proposed approach incrementally improves the labeling functions by using the true labels obtained in each round. To achieve this they need LFs to be differentiable, which can be a drawback in some cases. Except these few changes, it is still using most of the existing weak-supervision machinery (i.e. label model, accuracy estimation and weighted combination of labeling functions to generate final labels etc.). So while the novelty might seem limited, I think it is useful in the sense that it may require minimal changes in existing weak-supervision setups to enhance it using active learning. 
I don't see major problems with this work, except a few. Firstly, interpreting the accuracy results in Table 2, is slightly difficult since class distributions in the datasets are not provided i.e. are the datasets imbalanced?  From the results it looks like the cases where one can obtain good labeling functions from experts then this approach works better than active learning otherwise not. There might be cases where obtaining expert labeling functions is more costly than obtaining labels ( which is probably easier than writing LFs). It might be worthwhile to discuss these limitations in the paper.     
summary_of_the_review: Overall, I think its a nice paper with a sound and simple approach to improve weak supervision's label quality using active learning. The contributions are novel and useful in practice. I am inclined towards accepting this paper. I need some clarifications in the experiments section to be more confident in this assessment.
review_title: null

Meta-review: The authors propose WARM, a novel method that actively queries a small set of true labels to improve the label function in weak supervision. In particular, the authors propose a methodology that converts the label function to "soft" versions that are differentiable, which are in term learnable with true labels using proper updates of parameters. Empirical results on several real-world data sets demonstrate that the method yields a pretty strong performance.

The reviewers generally agree that the idea of making the labeling functions differentiable is conceptually interesting. They are also positive about the simplicity and the promising performance. They share joint concerns on whether the idea has been sufficiently studied in terms of the design choices and completeness of the experiments. For instance, the authors can conduct deeper exploration of the trade-off for differentiable LFs. They can also study active learning strategies that are beyond basic uncertainty sampling. While the authors have provided more studies about those exploration and ablation studies during the rebuttal, generally the results are not sufficient to convince most of the reviewers. In future revisions, the authors are encouraged to clarify its position with respect to existing works that combine active learning and weakly-supervised learning.

The authors position the paper as more empirical than theoretical. So the suggestion from some reviewers about more theoretical study is viewed as nice-to-have but not a must.
Decision: Reject