Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling
Authors: Benedikt Boecking
Willie Neiswanger
Eric Xing
Artur Dubrawski
Venue: ICLR 2021 Poster (2021)
Keywords: weak supervision
data programming
data labeling
active learning

Abstract: Obtaining large annotated datasets is critical for training successful machine learning models and it is often a bottleneck in practice. Weak supervision offers a promising alternative for producing labeled datasets without ground truth annotations by generating probabilistic labels using multiple noisy heuristics. This process can scale to large datasets and has demonstrated state of the art performance in diverse domains such as healthcare and e-commerce. One practical issue with learning from user-generated heuristics is that their creation requires creativity, foresight, and domain expertise from those who hand-craft them, a process which can be tedious and subjective. We develop the first framework for interactive weak supervision in which a method proposes heuristics and learns from user feedback given on each proposed heuristic. Our experiments demonstrate that only a small number of feedback iterations are needed to train models that achieve highly competitive test set performance without access to ground truth training labels. We conduct user studies, which show that users are able to effectively provide feedback on heuristics and that test set results track the performance of simulated oracles.

Published as a conference paper at ICLR 2021
INTERACTIVE WEAK SUPERVISION: LEARNING USEFUL HEURISTICS FOR DATA LABELING
Benedikt Boecking', Willie Neiswanger', Eric P. Xing', & Artur Dubrawski'
CCarnegie Mellon University {boecking, epxing, [wd)/css css. cmu, edu
Stanford University
nei seiswaggeress.o stanford. edu
ABSTRACT
Obtaining large annotated datasets is critical for training successful machine learn- ing models and it is often bottleneck in practice. Weak supervision offers a promising alternative for producing labeled datasets without ground truth annota- tions by generating probabilistic labels using multiple noisy heuristics. This process can scale to large datasets and has demonstrated state of the art performance in diverse domains such as healthcare and e-commerce. One practical issue with learning from user-generated heuristics is that their creation requires creativity, foresight, and domain expertise from those who hand-craft them, a process which can be tedious and subjective. We develop the first framework for interactive weak supervision in which method proposes heuristics and learns from user feedback given each proposed heuristic. Our experiments demonstrate that only small number of feedback iterations are needed to train models that achieve highly com- petitive test set performance without access to ground truth training labels. We conduct user studies, which show that users are able to effectively provide feedback heuristics and that test results track the performance of simulated oracles.
INTRODUCTION
The performance of supervised machine learning (MI.) hinges on the availability of labeled data in sufficient quantity and quality. However, labeled data for applications of MI. can scarce, and the common process of obtaining labels by having annotators inspect individual samples is often expensive and time consuming. Additionally, this cost is frequently exacerbated by factors such as privacy concerns, required expert knowledge, and shifting problem definitions.
Weak supervision provides a promising alternative, reducing the need for humans to hand label large datasets to train ML models (Riedel al. 2010 Hoffmann et al. [001T] Ratner et al. 2016 Dehghani ct al. 2018). A recent approach called data programming (Ratner et 2016)) combines multiple weak supervision sources by using unsupervised label model to estimate the latent true class label, an idea that has close connections to modeling workers in crowd-sourcing (Dawid Skene 1979] Karger al.) 2011: Dalvi al. 2013) Zhang et al., 2014). The approach enables subject matter experts to speciffy labeling functions (LFs) functions that encode domain knowledge and noisily annotate subsets of data, such as user-specified heuristics or external knowledge bases-instead of needing to inspect and label individual samples. These weak supervision approaches have been used on a wide variety of data types such as MRI sequences and unstructured text, and in various domains such as healthcare and e-commerce (Fries et al.] 2019 Halpern et al. 2014 Bach et al. 2019 Ré al., 2020). Not only does the use of multiple sources of weak supervision provide a scalable framework for creating large labeled datasets, but it can also be viewed vehicle to incorporate high level, conceptual feedback into the data labeling process.
In data programming, each LF is an imperfect but reasonably accurate heuristic, such as pre-trained classifier or keyword lookup. For example, for the popular 20 newsgroups dataset, an LF to identify the class 'sci.space' may look for the token 'launch' in documents and would be right about 70% of the time. While data programming can very effective when done right, experts may spend a significant amount of time designing the weak supervision sources (Varma & Ré 2018) and must

---

Published as a conference paper at ICLR 2021
Update IVS model
Y P(Y 10 Myreernni
Unlebeled Dasanet
LF famiy
Candidate LFs
L
Yee No
ded aIDeEE, EntGes 46 bace
XEC
Query: 'ls better man randam?
X
Ao dec
An LF family generates large L of candidate heuristics (LF9) for unlabeled datsset.
Interactive Weak Supervision (IWS) proposes Fs from & to experts and leams feedback on these LFs
IWS chooses a final set of LFs, which are used to estimate labels for and train end classifier
Figure I: Interactive Weak Supervision (IWS) helps experts discover good labeling functions (L.Fs..
often inspect samples at random to generate ideas (Cohen- Wainn-Wnngg et al. 2019). In our 20 newsgroups example, we may randomly see document mentioning "Salman Rushdie' and realize that the name of a famous atheist could be good heuristic to identifyy posts in 'alt.atheism'. While such a heuristic seems obvious after the fact, we have to chance upon the right documents to generate these ideas. In practice, coming up with effective LFs becomes difficult after the first few. Substantial foresight (Ramos et al. 2020) is required to create a new function that applies to a non-negligible subset of given data, novel, and adds predictive value.
We propose a new approach for training supervised MI models with weak supervision through an interactive process, supporting domain experts in fast discovery of good L.s. The method queries users in an active fashion for feedback about candidate LFs, from which a model learns to identify LFs likely to have good accuracy. Upon completion, our approach produces a final set of LFs. We use this set to create an estimate of the latent class label via an unsupervised label model and train final, weakly supervised end classifier using noise aware loss function on the estimated labels as in Ratner et al.] (2016). The approach relies on the observation that many applications allow for heuristics of varying quality to generated at scale (similar to Varma & Ré (2018), and that experts can provide good judgment by identifying some LFs that have reasonable accuracy. The full pipeline of the proposed approach, termed Interactive Weak Supervision (IWS is illustrated in Fig. Our contributions are:
1. We propose, the best of our knowledge, the first interactive method for weak supervision in which queries to be annotated are not data points but labeling functions. This approach automates the discovery of useful data labeling heuristics.
2. We conduct experiments with real users on three classification tasks, using both text and image datasets. Our results support our modeling assumptions, demonstrate competitive test set perfor- mance of the downstream end classifier, and show that users can provide accurate feedback on automatically generated L.s.
3. In our results, IWS shows superior performance compared to standard active learning, i.c. achieve better test set performance with a smaller number of queries to users. In text experiments with real users, IWS achieves mean test AUC after 200 LF annotations that requires least three times as many active learning iterations annotating data points. In addition, the average user response time for LF queries was shorter than for the active learning queries data points.
2 RELATED WORK
Active strategies for weak supervision sources have largely focused on combinations of data pro- gramming with traditional active learning on data points, while our work has similarities active learning on features (Druck et al. (209) and active learning of virtual evidence (Lang & Poon 2021).
In Nashaat et al. (2018) a pool of samples is created on which LFs disagree, and active learning strategies are then applied to obtain labels for some of the samples. In CChhe-Waag et al. (2019),
samples where LFs abstain or disagree most are selected and presented to users in order to inspire the creation of new LFs. In Hancock et al. (2018), natural language explanations provided during text labeling are used to generate heuristics. The proposed system uses a semantic parser to convert. explanations into logical forms, which represent labeling functions.
available het veeweak--a ve-weak-superrill supervision
2

---

Published as a conference paper at ICLR 2021
Prior work has emphasized that LFs defined by experts frequently have recurring structure in which elements are swapped to change the higher level concept a function corresponds to (Varma & Ré 2018: Varma et al., 2017| Bach et al. 2019). As example, in tasks involving text documents, LFss often follow a repetitive structure in which key terms or phrases and syntactical relationships change, e-g. mentions of specific words (Varma & Ré 2018 Johen-Wang et al. [009 Varma et al.
2019). Prior work relies on this observation to create heuristic generators (Varma & 2018). LF
templates (Bach al. 2019), and fomain-specile primitives (Varma et al.) 2017). J. In particular, in
semi-supervised data programming setting, Varma& & Ré (018)) propose system for automatic generation of labeling functions without user interaction, by using a small set of labeled data.
Additional related work has investigated weak supervision for neural networks in information retrieval (Dehghani et al. 2017 Zamani et al., 2018 Zamani & Croft 2018), the modeling of
dependencies among heuristics in data programming (Bach et a
Varma et al.) 2019). the
multi-task data programming setting (Ratner et al. 2019). handling of multi-resolution sources (Sala)
et al. 2019), the use of noisy pairwise labeling functions (Boecking & Dubrawski] 2019), addressing latent subsets in the data (Varma et al., 2016), LFs with noisy continuous scores (Chatterjee et al.
2020 , and fast model iteration via the use of preitraieed embeddings (Chen et al. 2020).
3 METHODS
We propose an interactive weak supervision (IWS) approach to assist experts in finding good labeling functions (LFs) for training a classifier on datasets without ground truth labels. We will first describe the general problem setting of learning to classify without ground truth samples by modeling multiple weak supervision sources, as well as the concept of LF families. We then dive into the details of the proposed IWS approach. For brevity, we limit the scope of the end classifier to binary classification, but the presented background and ideas do extend to the multi-class settings.
3.1 PRELIMINARIES
Learning with Multiple Weak Supervision Sources Assume each data point E X has a latent class label y" y = 1.. Given n unlabeled, i..d. datapoints X = {2r our goal
train an end classifier f X' - y such that f(x)= y² In data programming (Ratner et al. 2016 2020), user provides m LFs {A₂ m where A, : X + y U {0}. An LF X, noisily labels the data with
(x) E y or abstains with (z) = O. The corresponding LF output matrix is E {- 1) _]=xm where Ar, (z;)) In this paper, we assume that each LF Xy has the same accuracy each class, aj = (x) (z) = y" (2)))(x) # 0), where accuracy is defined on items where j does not abstain. Further, denote by 1 (z) * 0) the LF propensity (sometimes called LF coverage), i.e. the frequency at which LF j does not abstain.
In data programming. an unsupervised label model po po A) produces probabilistic estimates of the latent class labels Y* = {yr } using the observed LF outputs A by modeling the LF accuracies, propensities, and possibly their dependencies. A number of label model approaches exist in the crowd- sourcing (Dawid& & Skene, 1979. Zhang et al. (2014) and the weak supervision literature (Ratner
al. 2020). In this paper, we use a factor graph as proposed in Ratner et al. 2016 2020) to obtain probabilistic labels by modeling the LF accuracies via factor Φ (A,
1(Aij = 3}
and labeling propensity by factor d.ab( Y) & 1(A,j A 0}, and for simplicity assume LFs are independent conditional on Y. The label model defined as
n Pe (Y, 2 ZØ exp Σ O (A...))
(I)
where Zg is a normalizing constant and (A,, vi) defined to be the concatenation of the factors for
all LFs j = 1, .....,,7 m for sample i. We learn by minimizing the negative log marginal likelihood given the observed A. Finally, following Ratner et al. (2016) an end classifier ʃ is trained using probabilistic labels (Y)A).
Labeling Function Families
We define LF families as sets of expert-interpretable LFs described by functions 2g X + (-1,0, 0, 13, for parameters ó E Φ. An example are shallow decision trees Zg parameterized by variables and splitting rules ó (Varma & Rě 2018), or a function 2n defining a regular expression for two words where & parameterizes the word choices from vocabulary and
3

---

Published as a conference paper at ICLR 2021
the target label. Given such LF family, we can generate large set of p candidate heuristics C ())()) = 2Φ, (z)
(+) 1= where Φy E L e.g. by sampling from & and pruning low coverage candidates. These families often arise naturally in the form of LFs with repetitive structure that experts write from scratch, where template variables-such as keywords-can be sampled from the unlabeled data to create candidates. For text, can find n-grams within document frequency range to generate key term lookups, fill placeholders in regular expressions, or generate shallow decision trees (Ratner et al. 2016; Varma & Ré [0188] Varma et al. 2019). For time series, we can create large set of LFs based on motifs (Lonardi & Patel (002) or graphs of temporal constraints (Guillame Bert Dubrawski 2017). For images, we can create a library of pre-trained object detectors as in Chen et al. (2019). or in some applications combine primitives of geometric properties of the images (Varma & Ré 2018).
An LF family has to be chosen with domain expert input. Compared to standard data programming, the burden of creating LFss from scratch is shifted to choosing an appropriate L.F family and then judging recommended candidates. We argue that domain experts often have the foresight to choose an LF family such that a sufficiently sized subset of LFs is predictive of the latent class label. Such LF families may not exist for all data types and classification tasks. But when they exist they offer the opportunity to quickly build large, labeled datasets. Once created, it is reasonable to expect that the same LF generation procedure can be reused for similar classification tasks without additional effort (e.g. we use a single LF family procedure for all text datasets in our experiments).
3.2 INTERACTIVE WEAK SUPERVISION
Instead of having users provide m good weak supervision sources up front, we want to assist users in discovering them. Successful applications of data programming have established that human experts are able to construct accurate LFs from scratch. Our work leverages the assumption that human experts can also judge these properties when presented with pre-generated LFs of the same form.
Suppose again that we have an unlabeled dataset X {x:)==1 and that our goal classifier ff without access to labels Y = {vy } Assume also that we defined a large pool of candidate LFs L {2,(2))
train an end
from an LF family (following Sec. B.1b, of varying accuracy and coverage. In IWS, our goal is identify an optimal subset of LFs L C pass to the label model in Eq. I Below, we will quantify how L- depends on certain properties of LFs. While we can observe some of these properties-suce as coverage, agreement, and conflicts an important property that we cannot observe is the accuracy of each L.F.
Our goal will thus be infer quantities related to the latent accuracies a, [0, 1] of LFs X E C. given small amount expert feedback. To do this, we define an expert-feedback model, which can be used to infer LF accuracies given set of user feedback. To efficiently train this model, our IWS procedure sequentially chooses an LF X3 E L and shows a description of X to an expert, who provides binary feedback about Ay. We follow ideas from active learning for sequential decision making under uncertainty, in which probabilistic model guides data collection to efficiently infer quantities of interest within T iterations. After a sequence of feedback iterations, we use the expert- feedback model to provide an estimate 2 C L of the optimal subset L The label model then uses P to produce a probabilistic estimate of Y*, which is used train the end classifier f. The full IWS procedure is illustrated in Fig. T and described in detail below.
Expert-Feedde Feedback Model We first define a generative model of human expert feedback about LFs, given the latent LF accuracies. This model will form the basis for an online procedure that selects a sequence of LFs to show to human experts. We task experts to classify LFs as either useful or not useful ujj E {0, J. corresponding to their belief that LF X is predictive of Y at better than random accuracy for the samples where X; does not abstain. Note that prior data programming work (Ratner let al. 2016; 2019: Dunnmon et al. [2200] Saab et al. (0020) assumes and demonstrates that experts are able to use their domain knowledge to make this judgment when creating LFs from scratch. We model the generative process for this feedback and the latent LF accuracies as, for 1,..., t:
Uj Bernoull((1;). Ui = hew (Ay), ~ Prior(-)
(2)
where by can be viewed as the average probability that human will label a given LF X, as uj = 1, and (Ay) is a parameterized function (such as neural network), mapping each LF Xy to Vny Finally, to model the connection between accuracy aj and uy. we assume that dy = g(az), ), where g: [0 + [0, 1] is monotonic increasing function mapping unknown L.F accuracy 0y to Vj.
4

---

Published as a conference paper at ICLR 2021
After t queries of user feedback LFs, we have produced a query dataset Q {() u } =-1 Given Qt, we infer unknown quantities in the above model, which are used to choose the next LF Xj to query, by constructing acquisition function (r: : R and optimizing it over À E L.
Acquisition Strategy and Final Set of LFs To derive an online procedure for our user queries about LFs, we need to define the properties of the ideal subset of generated LFs L C L which we want to select. Prior data programming work of [Rtnne et al. (2016 2019 2020) with label models as Eq. (I) does not provide an explicit analysis of ideal metrics of LF sets and their trade-offs to help define this set. We provide the following theorem, which will motivate our definition for L"
Theorem 3.1. Assume a binary classification setting, m independent labeling functions with accuracy 0j E [0, 1] and labeling propensity 1j E [0, 1]. For a label model in Eq. 1 parameters Ō R22m, and for any E ..).
with given label model
E m
05 (2ag 1)1,)²
P(ù. = v.) 211)111 21 exp
2/1020113
where ((2) are the m weights of ds and Ú: {-11 13 the label model estimate for V
Proof.
The proof is given in Appendix C
This theorem indicates that one can rank LFs according to (2a, - 1)1, where ay. I1 are the unknown accuracy and observed coverage of LF j. respectively. We provide additional analysis in Appendix C Our analysis further suggests the importance of obtaining LFs with an accuracy gap above chance. Intuitively, we do not want to add excessive noise by including LFs too close to random. Below, assume that our final set of LFs is sufficient to accurately learn label model parameters 6, and leave analysis of the influence of additional LF properties on learning f to future work.
To define the ideal final subset of LFs, we distinguish three scenarios: (A) there are no restrictions on the size of the final set and any LF can be included, (B) the final set is limited in size (e.g. due to computational considerations) but any LF can be included, (C) only LFs inspected and validated by experts may be included, e.g. due to security or legal considerations.
For each of these scenarios, at each step we maximize acquisition function over the set of candidate LFs, i.c. compute X; = arg m (A,, ue) and update the query dataset Q2 = Q--1 U {(Ae, 4)} After a sequence of T queries we return an estimate of L', denoted by É. The corresponding LF output matrix A comprised of all Xj E É, is then used to produce estimate Y of the true class labels via the label model Po (Y Finally, noise-aware discriminative end classifier f is trained on (X, Y).
(A). We then query human expert to obtain
Scenario (A): Unbounded LF Set. In the absence of restrictions on the final set of LFs, our analysis in Appendix Cindicates that the ideal subset of LFs L includes all those with accuracy greater than a gap above chance, i.e. a > r > 0.5. Thus, we define the optimal subset in this scenario as
L f>y E 0, > r).
This is variation of the task of active level set estimation (LSE), where the goal is to identify all
elements in a superlevel set of L (Zanette et al.] 2018:) Gotovos 2013 Bryan et al., (006). Thus, each step we use the straddle acquisition function (Bryan al. 2006) for LSE, defined for candidate X3 E L. Q.-1 to score L.ss highest that are unknown and near the boundary threshold r:
S (Ay) = 1.96 o, (Q(-11)-14(QQ,--))) -r
(4)
where o, (Q--1) / Varip(ay Qr- ]] is the standard deviation and M4 (Q=(Q=1)) Elp(a, Qt-1)] the mean of the posterior LF accuracy. At the end of Sec. 3.2. we describe how we perform approximate inference of p(a,)( Q+-1) via an ensemble model.
After a sequence of T queries we return the
following estimate of L*:
E (;(QT) r}.
We denote the algorithm for scenario (A) by IWS-LSE-a. See Algorithm | for pseudocode describing this full IWS-LSE-a procedure. In experiments, set 0.7, though ablation study shows
5

---

Published as a conference paper at ICLR 2021
that IWS-LSE works well for range of thresholds > 0.5 (Appendix B.6 LSE acquisition function aims to reduce uncertainty around r, and therefore tends to explore LFs that have coverage on parts of Y that we are still uncertain about.
Fig. 9 Note that the
Scenario (B): Bounded LF Set. If the final set restricted in size to m LFs, c.g. due to computational considerations when learning the label model in Eq. I need to take the trade-off of LF accuracy and LF coverage into account. Let Ij be the observed empirical coverage of LF Aj- We want to identify LFs with accuracy above r and rank them according to their accuracy-coverage trade-off, thus our analysis in the appendix suggests the optimal subset
C
arg max DCC. Fm AYED
Σ
(1(0z2r)
1) +1,)
(6)
Since the LF accuracy-coverage trade-off only comes into effect if ag > r. this yields the same acquisition function sfSE in Eq. 44), and we then select the final set as 2 {h D arg (P)== Σx,e (ny((r)>)) (2 * (y((T) - 1) * [,}} which corresponds to a simple thresholding and sorting operation. We denote the algorithm for scenario (B) by IWS-LSE-ac.
Scenario (C): Validated LF Set. Finally, in some application scenarios, only LFs inspected and validated by experts should be used to estimate Y", e.g. due to security or legal considerations. An LF j validated if shown to an expert who then responds with u, = 1. This leads to an active search problem (Garnett et al. 2015) where our aim is to identify a maximum number of validated LFs (i.e. u = 1) in L given budget of T user queries, i.e. to compute
C
arg argmax max DCL, -I AJED
Σ Uj.
L f Qr : =, = 1}.
we use a one-step look ahead active search acquisition LiQ Q to be the posterior probability that the usefulness label uy is positive, i.c. ApA (A₂) = μy (Qr-)). We denote the algorithm for scenario (C) by IWS-AS.
As in (Garnett et al. 2015 Jiang et al. 2017),
function defined for a candidate X, E L
Approximate Inference Details We now describe how we use our expert-feedback model in Eq. (2) to infer Qi), a quan- tity used in the acquisition functions and final set estimates. Recall that we defined gener- ative model of human feedback uj on query LF Aj with latent variables vj and w. We as- sumed a connection between 8y and the latent LF accuracy 0y via monotonic increasing function a, = g(1)). Similar to existing work 6 end
Algorithm 1: Interactive Weak Supervision (IWS-LSE-a).
Input: L: set LFs, T: max iterations. Qo + 2
2 for t = 1,2, do
d arg 2 (P(A) Eq.
un ExperiQuery(At)
Q2 + Q- U ((),
on high dimensional uncertainty estimation (Beluch et al., 2018: Chitta et al. 2018)), we
+ E Elpla, r) Eq.
use an ensemble (ha } of neural networks he with parameters w to predict u, given input Az. To perform this prediction, we need feature representation r(Ay) for LFs that is general and works for any data type and task. To create these features, we use the LF output over our unlabeled dataset ray) = (), ... )y We then project 7o (Aş) to d' dimensions using PCA for final feature representation T TAy)), which given as input to each h Our neural network ensemble can now learn functions ì Rdd + [0, 1). which map from LF features r(Ar) to oj p(u, = 1 Qx). This yields an ensemble of estimates for Uy, and through g -1 of ay. These are treated as approximate samples from p(a,)Q:), and used to form sample-estimee used in the acquisition functions.
4 EXPERIMENTS
Our experiments show that heuristics obtained via small number of iterations of IWS can be used to train a downstream end classifier f with highly competitive test set performance. We first present results obtained with a simulated IWS oracle instead of human users. Oracle experiments allow us to answer how our method would perform if users had perfect knowledge about LF accuracies. We then show results from a user study on text data in which the query feedback is given by humans. In Appendix [BI we provide results of user study on images, using image based LFs. A comprehensive description of the datasets and implementation details can be found in Appendix B.2

---

Published as a conference paper at ICLR 2021
Amazzon
IMD8
Bias Bios: jourmalit photographer
0.8
0.7
0.7
0.6
0.6
0.6
0.5
0.5
200
200
Blas Bins: painter or architect
Bias Bios: professor or teacher
Bias Blos: professor or phy physician
0.9
0.9
0.9
Test AUC
0.8
0.8
0.7
0.0
Act frm learning
0.7
All ground truth ibbels
0.6
0.7
INS4
LSE 100
0.6
Snuta/Roef
0.5
50 100 150
200
50
100
150
200
50
100
150
200
Number of annotattions
Number annotations
Number of annotations
Figure 2: Test set AUC of end classifiers vs. number of iterations. IWS-LSE is compared to active learning, Snuba, and to using all training ground truth labels. Note that one iteration in this plot corresponds to one expert label. comparison of true user effort needed to answer each type of query (label for one sample vs. label for one LF)) will vary by application.
Datasets Text Classification: We create six binary text classification tasks the basis of three public datasets: Amazon reviews (He & McAuley (016). IMDB reviews (Maas et al. [001]) and BiasBios biographies (De-Arteaga et al. (019). The tasks are chosen such that most English speakers can provide sensible expert feedback on LFs, for ease of reproducibility.
Cross-modal classification: As in Varma & Ré (018). we take the COCO dataset (Lin al. 2014) and generate LFs over captions, while classification is performed on the associated images. The two binary tasks are to identify 'person' in an image, and to identify 'sports' in an image.
Image classification: For image classification tasks with image LFs, we use the COCO dataset and create two binary classification tasks to identify 'sports' in an image and 'vehicle' in an image. For these image-only experiments, we generate nearest-neighbor based LFs directly the images.
Approaches All approaches train the same downstream end classifier f on the same inputs X. We show results for (WS-LSE-a (unbounded LF set), JWS-LSE-ac (bounded LF set), and /WS-AS (validated LF set). For /WS-LSE-ac, we bound the size of the final set of LFs at each iteration t by m Σ u, + m, i.e. the number of LFs so far annotated as u = 1 plus a constant m. We compare the test set performance of IWS to a set of alternatives including (1) annotation of samples via active learning (uncertainty sampling) by a noiseless oracle, (2) the Snuba system (Varma & Ré 2018). and (3) using all ground truth training labels. In Appendix B.3 we provide additional results comparing IWS to itself using a random acquisition function (IWSS (WS-random). In our figures, annotations the x-axis correspond to labeled samples for Snuba and active learning, and to labeled LFs for IWS. We note that this head to head comparison of user effort is application dependent. We provide timing study to showcase the effort required to carry out IWS versus labeling of samples in our specific user experiments on text in T
LF Families For text tasks, prior work such Ratner et al. (2020) and Varma et al. (2019) demonstrates that word and phrase LFs can provide good weak supervision sources. To generate LFs, we define uni-gram vocabulary over all documents and discard high and low frequency terms. We then exhaustively generate LFs from an LF family 25 which outputs a target label if uni-gram appears in document, where á specifies the uni-gram and target label. We also evaluated combinations of higher-order n-grams, but did not observe a significant change in performance. difficult to obtain strong domain primitives to create weak supervision
For COCO images, sources, even for data programming from scratch. We hence choose the images and their embeddings themselves to do this job for us, by relying on k-nearest neighbor functions. To generate LFs with high coverage, we first create small, unique clusters of up to ki mutual nearest neighbors (MKNN)
Image A is k2 nearest neighbor of image B, and image B is also a k nearest neighbor of image A.
7

---

Published as a conference paper at ICLR 2021
Araazon
MDE
0.90
0.85
0.8
0.7
0.75
0.7
0.65
0.5
uacr IMS
usco
0.4
23
50
73
300
223
130
175
200
25
30
75
100
222
120
173
200
Numberd labeling funstions annotated
Number lebaling fanctions annotateed
Amazon quary feedback (single user)
Amazon quarry feedhark tall users)
IMDB query feedbbcck (all users]
3.0
0.8
0.0
0.8
Thue LF accurscy
True LE accuracy
True accuraty
0.6
0.6
0.6
0.4
0.4
0.4
not zucricd
zuoried Not usetu Useid Unvure
quanted Not useful Uschul
0.2
for
0.2
Usild Uiroar
Unsure
0.0
0.0
0.0
0.00
0.05
0.10
0.15
0.20
0.00
0.05
0.10
0.15
0.20
0.0
0.2
2.1
0.6
LF coverage
coverage
LF coverage
Figure 3: Human user study, text data. Top: Test AUC of end classifiers trained on soft labels obtained via IWS-AS. Test set performance of humans closely tracks performance using simulated oracle after ~ 100 iterations. Bottom: scatter plots of human responses to queries showing the true LF accuracy vs LF coverage by one user (lower left) and all users (lower middle and lower right). An 'unsure' response does not provide label to an LF query but is counted as an annotation.
For each member of a cluster, we then find the l/2 nearest neighbors, and keep ones shared by least one other cluster member. Finally, each extended cluster defines an LF, which assigns the same label to each member of the extended cluster. The MKNN symmetry produces good initial clusters of varying size, while the second KNN step produces LFs with large and varying coverage. User experiments in Appendix I show that real users can judge the latent LF usefulness quickly by visually inspecting the consistency of the initial cluster and small selection of the cluster nearest neighbors.
4.1 ORACLE EXPERIMENTS
The simulated oracle labels an LF as useful if has an accuracy of least 0.7. Measured by test-set AUC of final classifier f. IWS-LSE outperforms other approaches significantly on five out of six text datasets, and matches the best performance also attained by Snuba on one dataset (Fig. [}. IWS-AS (Fig. [6, Appendix) performs similarly well on four text datasets, and competitively on the other two. Both IWS approaches outperform active learning by a wide margin on all text datasets. IWS also quickly approaches the performance achieved by end model trained the full ground truth training labels. We provide ablation results for IWS-LSE varying the final set size as well as thresholds in Appendix B.3 For the COCO image tasks, LFs were created using image captions as in (Varma & Ré, 2018) (Fig. 4 first and second plot), as well as on images directly via nearest neighbors (Fig. [4] third and fourth plot). IWS also performs competitively on these image tasks and quickly approaches the performance achieved using all training ground truth.
4.2 USER EXPERIMENTS ON TEXT
We conduct experiments of IWS-AS with real users on the Amazon and IMDB review senti- ment classification tasks. The results demon- strate that users judge high accuracy functions as useful and make few mistakes. In our ex- periments, users are shown a description of the heuristic (the key term pattern) and the intended label. Users also view four snippets of random documents where the LF applied, but are instructed to only consider the examples if necessary. See Appendix B.4 for screenshot of the query interface and details regarding the user prompts. The top
Table 1: Median (mean) user response time.
Dataset
Annotate LF
Annotate sample
Amazon
4.2s (8.3s)
7.9s (10.3s)
IMDB
3.2s (6.0s)
19.s (24.3s)
8

---

Published as a conference paper at ICLR 2021
(1) COCO caption Us
(2) COCO caption
COCO vehicle.
(4) COCO sports, image Us
0.9
AestAUc
0.8
0.8
0.7
0.7
learning ground NVS Stulbalieed
ground
LSE
0.6
100 Number of annnntations
50 100
50
100
Number of annotations
Number of annctations
Number of annotattios
Figure 4: COCO image classification. Images (1) and (2): Test AUC of image classifiers trained using probabilistic labels obtained from LFs on captions, compared to training with active learning and the full training ground truth. Images (3) and (4): Test AUC of image classifiers trained using nearest neighbor based image LFss compared to training with active learning and the full training ground truth. Due to the low coverage of LFs, we only use IWS-LSE-a in our image experiments.
of Fig. 3 shows that mean test set performance of IWS-AS using LFs obtained from human feedback closely tracks the simulated oracle performance after about 100 iterations. Fig. 3 further shows the queried LFs and corresponding user responses by their true accuracy vs. their non-abstain votes. To match the mean test AUC of IWS-AS obtained after 200 iterations the Amazon dataset, active learning (uncertainty sampling) requires about 600 iterations. For the IMDB dataset, to achieve the same mean test AUC of IWS-AS obtained after 200 iterations, active learning requires more than 1000 iterations. For both datasets, the average response time to each query was fast. We conducted a manual labeling exercise of samples for the IMDB and Amazon datasets (Table I with real users. Assuming the original ratings are true, our users incorrectly classified ~9% of IMDB reviews while taking significantly longer compared the response times to LF queries. For the Amazon dataset, users mislabeled ~2% of samples and were also slower at labeling samples than LFs. The user-study experiments involved nine persons with a computer science background. Neither the true accuracy of each heuristic nor the end model train or test set results were revealed to the users at any stage of the experiment. Appendix B. provides results for a similar user study on the COCO sports task with image LFs. These results are consistent with those for text, showing that users are able to distinguish accurate vs. inaccurate image LFs well, and that the full IWS procedure with real users achieves similar performance the one using simulated oracle.
5 CONCLUSION
We have proposed methods for interactively discovering weak supervision sources. Our results show that small number of expert interactions can suffice to select good weak supervision sources from large pool of candidates, leading to competitive end classifiers. The proposed methodology shows promise as a way to significantly speed up the process of weak supervision source discovery by domain experts as an alternative to devising such sources from scratch. On large number of tasks, we obtain superior predictive performance on downstream test sets compared to the automatic selection of LFs with Snuba (Varma & Ré (018) and standard active learning (where users annotate samples instead of LFs), when measured with respect to the number of user annotations. We conduct experiments with real users on two text benchmark datasets and one image dataset and find that humans recognize and approve high accuracy LFs, yielding models that match performance attainable with a simulated oracle. Our text experiments also suggest that tasks exist where users are able to annotate heuristics faster than individual samples.
There are limitations to the approaches we propose in their current form. While our experiments on text and image data show promise, future work is required to investigate appropriate LF families for a wider variety of tasks and data types. Furthermore, we rely on domain expert's 's ability to judge the quality of LFss generated from an LF family. While it is true that experts similarly have to judge the quality of LFs they create from scratch, the level of interpretability required of an LF family in IWS may be difficult to achieve for some tasks and data types. Additionally, datasets with large number of classes may lead to very large sets of candidate LFs. Thus, future work will need to find efficient ways to search this space. Future work should also explore hybrid methods of IWS and active learning, interactive learning of LF dependencies, and acquisition functions which optimize for additional properties of sets of heuristics such diversity.
9

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
We thank Kyle Miller at the Auton Lab for the valuable feedback and discussions. We also want express our thanks Maria De Arteaga Gonzalez, Vanessa Kolb, Ian Char, Youngseog Chung, Viraj Mehta, Gus Welter, and Nicholas Gisolfi for their help with this project. This work was partially supported by Space Technology Research Institutes grant from NASA's Space Technology Research Grants Program, and by Defense Advanced Research Projects Agency's 's award FA8750- 17-2-0130. WN was supported by U.S. Department of Energy Office of Science under Contract No. DE-AC02-76SFO0015.
REFERENCES
Stephen H Bach, Bryan He, Alexander Ratner, and Christopher Ré. Learning the structure of generative models without labeled data. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 273-282, 2017.
Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Haneock, Houman Alborzi, et al. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In Proceedings the 2019 International Conference on Management of Data, pp. 362-375, 2019.
William Beluch, Tim Genewein, Andreas Nürnberger, and Jan Köhler. The power of ensembles for active learning in image classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9368-9377,2000 2018.
Benedikt Boecking and Artur Dubrawski. Pairwise feedback for data programming. NeurIPS Workshop Learning with Rich Experience (LIRE), 2019.
Brent Bryan, Robert C Nichol, Christopher R Genovese, Jeff Schneider, Christopher Miller, and Larry Wasserman. Active learning for identifying function threshold boundaries. Advances in Neural Information Processing Systems, pp. 163-170, 2006.
Oishik Chatterjee, Ganesh Ramakrishnan, and Sunita Sarawagi. Robust data programming with precision-guides labeling functions. In Proceedings of the AAAI Conference on Artificial Intelli- gence, volume 34, pp. 3397-3404, 2020.
Mayee F Chen, Daniel Y Fu, Frederic Sala, Sen Wu, Ravi Teja Mullapudi, Fait Poms, Kayvon Fatahalian, and Christopher Ré. Train and you'll miss it: Interactive model iteration with weak supervision and pre-trained embeddings. arXiv preprint arXiv:2006.1516 2020.
Vincent Chen, Sen Wu, Alexander J Ratner, Jen Weng, and Christopher Ré. Slice-based learning: A
programming model for residual learning in critical data slices. In Advances in Neural Information Processing Systems, pp. 9392-9402 2019.
Kashyap Chitta, Jose M Alvarez, and Adam Lesnikowski. Large-scale visual active learning with deep probabilistic ensembles. arXiv preprint arXiv.181..03577 2018.
Benjamin Cohen Cohen-Wang, Stephen Mussmann, Alex Ratner, and Chris Ré. Interactive programmatic labeling for weak supervision. KDD Data Collection, Curation, and Labeling for Mining and Learning Workshop, 2019.
Nilesh Dalvi, Anirban Dasgupta, Ravi Kumar, and Vibhor Rastogi. Aggregating crowdsourced binary ratings. In Proceedings of the 22nd International Conference on World Wide Web, pp. 285-294, 2013.
Alexander Philip Dawid and Allan M Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society: Series C (Applied Statistics), 28 (1):20-28, 1979.
Maria De-Arteaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexandra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study semantic representation bias in high-stakes setting. In Proceedings of the Conference Fairness, Accountability and Transparency, pp. 120-128, 2019.
10

---

Published as a conference paper at ICLR 2021
Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. Neural ranking models with weak supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 65-74, Association for Computing Machinery, 2017.
Mostafa Dehghani, Arash Mehrjou, Stephan Gouws, Jaap Kamps, and Bernhard Schölkopf. Fidelity- weighted learning. In Proceedings of the International Conference on Learning Reprexentations, 2018.
Gregory Druck, Burr Settles, and Andrew McCallum. Active learning by labeling features. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pp. 81-90, 2009.
Jared A. Dunnmon, Alexander J. Ratner, Khaled Saab, Nishith Khandwala, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew P. Lungren, Daniel L. Rubin, and Christopher Re. Cross-modal data programming enables rapid medical machine learning, Patterns, 1(2):100019, 2020.
Jason A Fries, Paroma Varma, Vincent S Chen, Ke Xiao, Heliodoro Tejeda, Priyanka Saha, Jared Dunnmon, Henry Chubb, Shiraz Maskatia, Madalina Fiterau, et al. Weakly supervised classification of aortic valve malformations using unlabeled cardiac mri sequences. Nature Communications, 10 (1):1-10, 2019.
Roman Garnett, Thomas Gärtner, Martin Vogt, and Jürgen Bajorath. Introducing the 'active search' method for iterative virtual screening. Journal of Computer-Aided Molecular Design 29(4):305-314, 2015.
Alkis Gotovos. Active learning for level set estimation. Master's 's thesis, Eidgenössische Technische Hochschule Zürich, Department of Computer Science,, 2013.
Mathieu Guillame Bert and Artur Dubrawski. Classification of time sequences using graphs of temporal constraints. The Journal of Machine Learning Research, 18(1):4370 4403, 2017.
Yoni Halpern, Youngduck Choi, Steven Horng, and David Sontag. Using anchors to estimate clinical state without labeled data. In AMIA Annual Symposium Proceedings, volume 2014, pp. 606. American Medical Informatics Association, 2014.
Braden Hancock, Martin Bringmann, Paroma Varma, Percy Liang, Stephanie Wang, and Christopher Ré. Training classifiers with natural language explanations. In ACL, volume 2018, pp. 1884. NIH Public Access, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 770-778, 2016.
Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, pp. 507-517. International World Wide Web Conferences Steering Committee. 2016.
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel Weld. Knowledge- based weak supervision for information extraction of overlapping relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies- voooluu
Volume 1, pp. 541-550 Association for Computational Linguistics. 2011.
Shali Jiang, Gustavo Malkomes, Geoff Converse, Alyssa Shofner, Benjamin Moseley, and Roman Gamett. Efficient nommyopic active search. In Proceedings of the 34th International Conference on Machine Learning-
70, pp. 1714-1723. JMLR. org, 2017.
David R Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourcing systems. In Advances in Neural Information Processing Systems, pp. 1953-1961, 2011.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 148206980, 2014.
11

---

Published as a conference paper at ICLR 2021
Hunter Lang and Hoifung Poon. supervuee self- self-supervision by combining deep learning and probabilisticc logic. In In Proceedings of the Thirty Fifth Annual Meeting of the Association for the Advancement of Artificial Intelligence (AAAI), 2021.
Hongwei Li, Bin Yu, and Dengyong Zhou. Error rate analysis of labeling by crowdsourcing, In ICML Workshop: Machine Learning Meets Crowdsourcing. Atalanta, Georgia, USA. Citeseer, 2013.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitmick. Microsoft coco: Common objects in context. In European Conference Computer Vision, pp. 740-755. Springer, 2014.
JLEKS Lonardi and Pranav Patel. Finding motif's in time series. In Proc. of the 2nd Workshop on Temporal Data Mining, pp. 53-68, 2002.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, June 2011.
Mona Nashaat, Aindrila Ghosh, James Miller, Shaikh Quader, Chad Marston, and -ean-FFancois Puget. Hybridization of active learning and data programming for labeling large industrial datasets. In 2018 IEEE International Conference on Big Data (Big Data). pp. 46-55, IEEE, 2018.
Gonzalo Ramos, Christopher Meek, Patrice Simard, Jina Suh, and Soroush Ghorashi. Interactive machine teaching: a human-centered approach to building machine-learned models. Human- Computer Interaction, 35(5-6))413-45) November 2020.
Alexander Ratner, Braden Hancock, Jared Dunamon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4763-4771, 2019.
Alexander Ratner, Stephen H Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. The VLDB Journal, 29(2):709-730, 2020.
Alexander J Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In Advances in Neural Information Processing Systems, pp. 3567-3575, 2016.
Christopher Ré, Feng Niu, Pallavi Gudipati, and Charles Srisuwananukorn. Overton: A data system for monitoring and improving machine-learned products. Proceedings of the 10th Annual Conference Innovative Data Systems Research, 2020.
Sebastian Riedel, Limin Yao, and Andrew McCallum. Modeling relations and their mentions without labeled text. In Joint European Conference Machine Learning and Knowledge Discovery in Databases, pp. 148-163. Springer, 2010.
Khaled Saab, Jared Dunnmon, Christopher Ré, Daniel Rubin, and Christopher Lee-Messer. Weak supervision as an efficient approach for automated seizure detection in electroencephalography npj Digital Medicine, 3(1):1-12, 2020.
Frederic Sala, Paroma Varma, Jason Fries, Daniel Y Fu, Shiori Sagawa, Saelig Khattar, Ashwini Ramamoorthy, Ke Xiao, Kayvon Fatahalian, James Priest, and Chris Re. Multi-resolution weak supervision for sequential data. In Advances in Neural Information Processing Systems, 2019.
Paroma Varma and Christopher Ré. Snuba: automating weak supervision to label training data. Proceedings of the VLDB Endowment, 12(3):223-236, 2018.
Paroma Varma, Bryan He, Dan Iter. Peng Xu, Rose Yu. Christopher De Sa, and Christopher Ré. Socratic learning: Augmenting generative models to incorporate latent subsets in training data. arXiv preprint arXiv: 161000001 2016.
12

---

Published as a conference paper at ICLR 2021
Paroma Varma, Bryan D He, Payal Bajaj, Nishith Khandwala, Imon Banerjee, Daniel Rubin, and Christopher Ré. Inferring generative model structure with static analysis. In Advances in Neural Information Processing Systems, pp. 240-250, 2017.
Paroma Varma, Frederic Sala, Ann He, Alexander Ratner, and Christopher Ré. Learning dependency structures for weak supervision models. International Conference on Machine Learning. 2019.
Hamed Zamani and W. Bruce Croft. On the theory of weak supervision for information retrieval. In Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval, pp. 147-154, New York, NY, USA, 2018. Association for Computing Machinery.
Hamed Zamani, W. Bruce Croft, and J. Shane Culpepper. Neural query performance prediction using weak supervision from multiple signals. The 41 International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '18, pp. 105-114 New York, NY, USA, 2018. Association for Computing Machinery.
Andrea Zanette, Junzi Zhang, and Mykel J Kochenderfer. Robust super-level set estimation using gaussian processes. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 276-2911 Springer, 2018.
Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I Jordan. Spectral methods meet EM: A provably optimal algorithm for crowdsourcing. In Advances in Neural Information Processing Systems, pp. 1 1260- 1268, 2014.
13

---

Published as a conference paper at ICLR 2021
COCO sports, image LFs user vs oracle
COCO Sports, image LFs, user feedback 1.0
0.95
True LF accuracy
0.8
0.90
Test AUC
0.6
0.85
x ye
0.4
0.80
0.75
Oracle IWS-LSE a
0.2
Useful
Users IWS-LSE a
Not useful
0.0
20
40 60 80 100 120
140
0.000
0.005
0.010 0.015 0.020 LF coverage
Number of labeling functions annotated
Figure 5: Human user study, image data (Section B.1). The user experiments in this plot were done using a labeling function family defined directly on the images. Left: Test AUC of end classifiers trained on soft labels obtained via IWS-LSE-a. Test set performance of humans closely tracks performance using a simulated oracle after 100 iterations on these datasets. Right: scatter plots showing the true LF accuracy vs LF coverage of responses to queries by one user.
A PSEUDOCODE FOR DIFFERENT IWS SCENARIOS
Algorithm 2: Interactive Weak Supervi-
Algorithm 3: Interactive Weak Supervi-
sion (IWS-LSE-ac).
sion with Active Search (IWS-AS)
Input: L: set of LFs, T: max iterations. Qo t- Ø
Input: L: set of LFs, T: max iterations.
Qo
for 1 = 1,r==1,,..... -=1,....,To do
2 for = 1.2,....77 do
d t arg max (P()) & Eq.
d f arg max m(Q(-1)
Ut +- Expen(uery(AA))
Wt + Exper(Query(Ai)
Qi t (
Qe + Q.-1 U ( (1),
end
end
C + {2, E
C t {hg E Qr : :u =1
arg maxpcc,(p)=m (
- 1))
Here provide pscudocode for the IWS-AS and IWS-LSE-ac settings, while the procedure for IWS-LSE-a can be found in the main paper in Algorithm [ Let us recap why we arrive at different formulations for IWS. In Sec. 3.22 distinguish three scenarios for arriving supervision sources which are modeled to obtain an estimate of the latent class variable Y. All three scenarios lead to different definitions of an optimal final set of LFs, which in turn means that they require us to formulate appropriate acquisition functions to achieve good estimate of the optimal set within a budget of T expert interactions. In scenario (A), we place no restrictions on the size of the final set and any LF can be included in it. This means that we have the computational resources to model a potentially very large number of weak supervision sources, and we do not require domain experts to inspect and validate every single LF that is modeled. Importantly, this means that we can and should include LFs that are good according to our predictive model and our definition of the optimal final set of LFs, but have never been shown to an expert. In scenario (B), the final set is limited in size but LFs do not have to be inspected and validated by a user. This scenario may be attractive for rapid cycles during development when very large number of LFs becomes computationally prohibitive. Finally, in scenario (C), only LFs inspected and validated by experts may be included, e.g. due to security or legal considerations.
final set of weak
14

---

Published as a conference paper at ICLR 2021
B ADDITIONAL EXPERIMENTS AND RESULTS
B.1 USER EXPERIMENTS ON IMAGES WITH IMAGE LABELING FUNCTIONS
We carried out user study on the COCO Sports image classification task described in Section using a family of mutual nearest neighbor image labeling functions, also described in Section 4 In line with our experiments on text data, Figure 5 shows that users were able to judge the accuracy of LFs consistently and well, and that the performance of IWS closely tracks the simulated oracle performance after about 100 iterations.
Again, users were quite quick at responding to LF queries, and judging LFss to be predictive of the latent class variable appeared to be an intuitive task. The average user response time to these image LF queries was 8.8 seconds, while the response time for annotating individual images was around 4. seconds on average. To assess an LF, a human user was shown the LFs MKNN image cluster of up to 20 images (the mean size was 7.9 images), and 15 random images contained in the extended cluster, sorted according to their mean distance to the MKNN image cluster. For this nearest neighbor- family of LFs (as described in Section 4 the parameter kn was set to 20, and k2 we found that performance was robust to changes in these parameters. While our results show that IWS performs well in this setting, and that classifiers can be trained competitively compared to active learning, it is an interesting challenge to develop better image primitives from which labeling functions can be constructed in data programming, and generated IWS and Snuba.
though
B.2 EXPERIMENT AND IMPLEMENTATION DETAILS
Datasets For our text data experiments, use three publicly available datasets to define six binary text classifaction tasks. We use a subset of the Amazon Review Data (He & McAuley 2016) for sentiment classification, aggregating all categories with more than 100% reviews from which we sample 200k reviews and split them into 160k training points and 40k test points. We use the IMDB Movie Review Sentiment dataset (Maas et al., (211)) which has 25k training samples and 25k test samples. In addition, we use the Bias in Bios (De Arteaga et al. 2019) dataset from which we create binary classification tasks to distinguish difficult pairs among frequently occurring occupations. Specifically, we create the following subsets with equally sized train and test sets: journalist or photographer (n 32 258), professor or teacher (n 24 588), painter or architect (n 12 236). professor or physician (n = 54 476).
For the cross-modal tasks of text captions and images as well as the pure image task we use the COCO dataset (Lin et al. 2014). We take the official validation set (n = 4952) as the test set. This set of test images is never used at any other point in the pipeline.
Implementation Details Our probabilistic ensemble in IWS, which is used in all acquisition functions to learn p(u; = 1[Q,). is a bagging ensemble of s = 50 multilayer perceptrons with two hidden layers of size 10, RELU activations, sigmoid output and logarithmic loss. To create features for the p candidate LFs in L, we use singular value decomposition (SVD) to project from to d' 150. Thus, at iteration f, given a query dataset Q.- = {(A { the ensemble is trained on pairs {(r(),, k=));=1 where T T(Ag) are the SVD features and u, the binary expert responses. The output of the ensemble on LFs not in the query dataset is used to compute (Q(=))
/ (plu, 1[Q,-1)]] and (Q,-) = E[g [p(u, 1 1))]_ While g, which maps aj to VJ. could be fine-tuned from data, we set g as the identity function in our experiments, which we find works well empirically. Finally, to allow human experts to express some level of confidence about their decision on uj. we also collect corresponding uncertainty weights b, E {1, 0.5), and we multiply the contribution the loss of each uy by the respective weight by. Users can also skip queries if they are unsure, indicated in black in Fig. These unsure responses are still counted as an iteration/query in our plots.
Our downstream end classifier f is a multilayer perceptron with two hidden layers of size 20 and RELU activations, sigmoid output and logarithmic loss. Each model in the ensemble as well as f are optimized using Adam (Kingma & Ba 2014). For the text datasets, we fit the end models f to low
Amazon: https://nifianmms gi thub. i wo/mma/nn/nddex.... htm stanford eduu hatp://amaaaaa/aaaaaaa datee//aaaaa/aaaa sent centiment/ BiasBios: http: http://kk...nm//aabbbbbb.
IMDB: https: //as:
15

---

Published as a conference paper at ICLR 2021
Amnzzon
IMDB
Bias Bios: jourmalist photographer
0.88
0.7
0.7
0.6
0.6
0.6
0.5
0.5
200
Blas Bins: painter or architact
Bias Bios: professor of teacher
Bias Blos: professor or physician
0.9
0.9
Test AUC
0.8
Active learning
0.7
0.0
All ground truth labels
INS54 SE
0.7
INSdrandom
0.6
INSSAS
0.6
0.5
50
100
150
200
50
100
150
200
50
100
150
200
Number of annotations
Number annotations
Number of annotations
Figure 6: Mean test set AUC vs. number of iterations for end classifiers trained on probabilistic labels. IWS-LSE and IWS-AS are compared to active learning. Snuba, training all labels, and IWS with random acquisition function. Note that, while one iteration on this corresponds to one expert label, comparison of effort needed to answer each type of query (label for sample vs label for LF) will vary by application.
dimensional projections of a large bag-of-words matrix via truncated Singular Value Decomposition (SVD), fixing the embedding size to 300. We repeat each experiment ten times. We assume that the class balance is known when fitting the label model, common in related work. When class balance is unknown, (Ratner et al. 2019) discuss an unsupervised approach estimate For the COCO image experiments, we use the second-to- layer of a ResNet- 18 (He et al. (2066) pretrained on ImageNet to obtain image features. These image features are used as the embedding train the end classifier for all approaches which compare. The embeddings are also used to create the nearest-neighbor based image LFs.
The first 8 iterations of IWS are initialized with queries of four LFs known to have accuracy between 0.7 and 0.75 drawn at random and four randomly drawn LFs with arbitrary accuracy. Subsequently, IWS chooses the next LFs to query. Active learning is initialized with the same number of known samples.
B.3 FULL IWS RESULTS AND ALL BASELINES
Fig. provides the full results of all IWS settings to all baselines, including IWS with a random acquisition function (IWS-random). IWS LSE-a corresponds to scenario (A) where there are no restrictions on the size of the final set and any LF can be included]. IWS LSE-ac corresponds to scenario (B) where the final set LFs is limited size (e.g. due computational considerations) but any LF can be included. IWS-AS corresponds to Scenario (C), where only LFs in our query dataset Qt can be used, which are LFs that were inspected and validated by experts, e.g. due to security or legal considerations.
B.4 USER EXPERIMENTS
B.4.1 INTERFACE AND EXPERIMENT PROMPT
Fig. shows example of the prompt that was shown users at each iteration of the IWS user experiments. Before the experiment started, users were first instructed on the interface they would and the task they would be given, i.e. to label a heuristic good if they would expect it to label samples at better than random accuracy and as bad otherwise. Users were also instructed about the response options, including the option not answer query if they were unsure ("I don't know").
16

---

Published as a conference paper at ICLR 2021
Description of heuristic contains term: clueless
Heuristic
Intended heuristic: positive sentiment
Intended Label
Is this good labeling function for positive sentiment?
Useful heuristic for: positive sentiment
Likely useful
positive
Likely useful heuristic for: positive sentiment
NOT useful heuristic for: positive sentiment
don't know
Submit
last decision
Init 0/4
Figure 7: An example of the prompt and answer options that were shown during the user study. Before starting the experiment, users were provided with a description of the task and the labeling function family.
Users were given a description of the classification task and domain of the documents for which heuristics were being acquired. Users were also provided with a description of the heuristic generated which labeled samples with a target label if a document contained a certain term. Finally, users were given two examples of a better than random heuristic, and two examples of an arbitrary heuristic.
During the experiments, users were also provided with random examples of documents documents where the queried LF applied. Users were instructed to first consider the LF without inspecting these random samples, and to only consider the examples if necessary.
While LFs receive binary labels, users were allowed to express uncertainty about their decision, which was used as a sample weight (1 certain else 0.5) of LFs during training of the probabilistic model of user feedback.
B.4.2 ADDITIONAL STATISTICS OF USER EXPERIMENTS
In Fig. 8 we provide more details about our user experiments. The top row displays the test set performance of downstream model f for each individual user. The middle shows how the number of LFs determined by the user to be useful row displays the maximum positive correlation between new LF with u 1 at iteration and all previously accepted LFs with u up to iteration t. Note that we take abstains into account by computing correlation between and LF and j only on entries where least one of them is nonzero
increases with the number of iterations. The bottom
B.5 HEURISTICS FOUND
B.5.1 IWS WITH REAL USERS, SENTIMENT CLASSIFICATION
We here provide some concrete examples of heuristics found during the IWS-AS procedure, that real users annotated as useful during the experiments. For the IMDB dataset:
Ten terms most frequently annotated useful by users
Class=1: beautiful, wonderful, perfect, enjoyed, amazing, brilliant, fantastic, superb, excellent, masterpiece.
Class=0: worst, poor, awful, bad, waste, terrible, horrible, boring, crap, stupid.
Ten terms annotated as useful with highest underlying accuracy
Class=l: flawless, superbly, perfection, wonderfully, captures, refreshing,, breathtaking, delightful, beautifully, underrated.
Class=0: stinker, dreck, unwatchable, unfunny, waste, atrocious, pointless, redeeming, laughable, lousy.
-
17

---

Published as a conference paper at ICLR 2021
68
60
100 PM
LEe 89-0
Use 99-4
Ube Dri-s
Uue RV1.3
landing
173
Nambo of lubeiing funeeme onedeaed
Number lubeiing wandeem cneclittd
Amazon correlation
correlation Lis
Mai Conneaation
0.4
25
50
75
100
125
150
175
200
50
100
150
175
200
LFs queried
LPs quered
Figure 8: Test AUC vs. IWS iteration shown for individual user experiments with IWS-AS (top). Number LFs labeled as useful vs. IWS iterations (middle). Maximum correlation previously accepted LFs vs. number of iterations (bottom).
18

---

Published as a conference paper at ICLR 2021
Ten terms annotated useful by users, selected random
Class=l: favorites, joy, superbly, delight, wonderfully, art, intelligent, terrific, light, finest.
Class=0: reason, failed, atrocious, decent, unfunny, lame, ridiculous, mistake, worst, dull.
For the Amazon dataset:
Ten terms most frequently annotated as useful by users
Class=l: wonderful, beautiful, amazing, fantastic, favorite, awesome, love, best, perfect, easy.
Class=0: worst, terrible, horrible, awful, worse, boring, poor, bad, waste, garbage.
Ten terms annotated as useful with highest underlying accuracy
Class=1: compliments, delightful, pleasantly, stars, captivating, excellent, awesome, beautifully, comfy, perfect.
Class=0: poorly, worthless, disappointing, refund, waste, yuck, garbage, unusable, useless, junk.
Ten terms annotated as useful by users, selected random
Class=1: interesting, beautifully, value, loves, strong, expected, gorgeous, perfectly, durable, great.
Class=O: sent, zero, money, mess, crap, refund, wasted, joke, unusable, beware.
B.5.2 IWS WITH AN ORACLE, OCCUPATION CLASSIFICATION
We here provide examples of heuristics found during the IWS-LSE procedure using a simulated oracle, on the BiasBios biographies datasets. We believe that real users ('internet biography experts") would be able make very similar distinctions.
For the Bias Bios: journalist or photographer' dataset, the ten terms most frequently annotated as useful by the oracle were:
Class 1: photography, clients, fashion, studio, photographer, art, commercial, fine, creative, advertising.
Class journalism, writing, reporting, news, media, writer, writes, editor, reporter, newspaper.
For the 'Bias Bios: painter architect' dataset. Ten terms most frequently annotated as useful by oracle:
commercial, buildings. development, residential, planning, architects, firm, master, design, construction.
Class 1:
Class painting, museum, collections, cxhibition, gallery, born, artists, shows, series, art.
For the BBiss Bios: professor physician' dataset, the ten terms most frequently annotated useful by the oracle were:
Class = 1: medical, orthopaedic, residency, family, practice, surgery, memorial, general, physician, saint.
Class 0: studies, phd, science, teaching, engineering, received, interests, member, pub- lished, professor.
For the 'Bias Bios: professor or teacher' dataset, the ten terms most frequently annotated as useful by the oracle were:
Class 1: students, english, teacher, schools, enjoys, years, life, classroom, children, elementary.
Class = review, research, interests, published, editor, university, journals, associate, studies, phd.
19

---

Published as a conference paper at ICLR 2021
Amazon
IMDB
0.9
LSE-a. r=0.80
LSE-a r-0.70
lest AUC
LSE-ac, m00.0
test AUC
0.8
LSE-ac, m=0.7
LSE-a,
LSE r=0.00
LSE-ac, r-0.70
LSE ac, r-0.60
0.7
LSE-a,
r-0.50
LSE-ac, mm0.60
LSE-ac, rm0.50
0.6
LSE ac, r-0.50
0.6
LSE r-0.80
100
150 Number of labeling functions annotated
Number labeling functions annotated
Bias Blos: professor physician
Bias Bios: journalstt photographer
LSE-a, r-0.50
0.9
a, r-0.70
AUC
LSE-ac, r-0.50
test AUC
LSF SE-ac, rw0.70
LSE-a. r-0.80
LSE-a. r=0.60
0.8
0.8
LSE-ac,
LSE-ac, r=0.60
LSE-a, r-0.70
LSE a, r-0.50
0.7
LSF
0.7
LSE-a. rm0.60
LSE -a. rm0.80
50
100
150
200
50
100
150
200
Number of labeling functions annolated
Number of labeling functions annotated
Bias Bios: painter ar architect
Bias Bios: professor or teacher
1.0
0.9
LSE-a,
r-0.50
LSE 4. r-0.50
Lest AUC
LSE-ac, rm0.50
test AUC
LSE-ac, rm0.50
LSE-a. r=0.80
0.8
LSE-s. r=0.80
0.8
LSE-ac, r-0.80
LSE ac, r=0.80
r-0.70
LSE-ac, 0m0.70
0.7
LSE ac. rw
LSE ac, r=
LSE ac, 0=0.60
0.6
100
150 Number of labeling functions annotated
50
Number labeling functions annotated
Figure 9: IWS-LSE ablation plots for varying thresholds r which we use to partition our set of LFs. On all datasets test set performance is very similar after around 100 iterations, showing that wide range of such thresholds leads to good test set performance. For IWS-LSE- shown in this plot m was set 100.
B.6 ABLATION OF IWS PARAMETER SETTINGS
In this section we provide results of ablation experiments for IWS. The IWS-LSE algorithm requires set a threshold on the (unknown) LF accuracy around which model aims to partition the of candidate LFs. Fig. 9 provides results for different threshold settings for IWS-LSE-a and IWS- LSE-ac, correspondin to Scenario (A) and Scenario (B). The figure shows that the algorithmss perform well across a wide range of r. While there is no clear, distinct performance difference discernible, the figure suggest that threshold too close to 1. can cause the algorithm to under-perfond pedder-errorm.. A possible explanation is that as it stifles exploration of LFs within the limited budget of queries to users.
In Scenario (B), which corresponds to the IWS-LSE-ac algorithm, our aim to find final set of LFs of limited size. Fig. 10 shows that wide range (n 50 to 200) of final set sizes produce good results. Recall that in our experiments, we bound the size the final set of LFs at each iteration by
m - Me + m, iee. the number of LFs so far annotated as u = plus a constant m.
20

---

Published as a conference paper at ICLR 2021
Amazon
IMDB
0.9
0.8
lest AUC
test AUC
0.7
NS-LSE-ac,
0.7
NS-LSF 50
NS--SE-ac, A 150
NS-LSE-ar ac, A 150
LS--SE-ac. 200
ac, 200
0.6
NSS-LSE-ar, A 100
0.6
INS-LSF-ac, A 100
0
100 Number of labeling functions annotated
200
100 Number of labeling functions annotated
professar or physician
Bios: journalist: or photographer
0.95
0.9
Test AUC
test AUC
0.85
0.80
LS--LEE-ac m= 50
LSS-LEEacc =0
INS-LSE-ac. 150
NS-LSE-ac.
0.75
IWS-LSE-ac, di 200
0.7
IS LWS-LEEacc ma..EEEac....000
INS-LSE-aC. A 100
IVS-LSE-aC. -i 100
0.70
50
100
150
200
50
100
150
200
Number labeling
annotated
labeling functions annotated
Bias Bios: painter architect
Bias Bios: professor teacher
0.9
AUC
test AUC
0.8
LSE-LSE-C,
m 50
INS-LSF-ar, A- 150
0.7
NS-LLF-ar ac, A 150
LSE 200
ac, 200
IVS-LSE-ac, A- 100
IWS-LSE-ac, A- 100 100
150
200
100
Number of labeling functions annotated
aunctionsannottted
Number of la labeliing functions annotated
Figure 10: IWS-LSE-ac ablation plots for varying final sizes via parameter m.. Recall that we bound the size of the final set of LFs at each iteration by m Σ'=l + m. i.e. the number of LFs far annotated as u = plus a constant m. Note that the LSE-ac setting takes LF coverage into account to rank L.ss according to (2a, 1) * I where ag In are the estimated L_F accuracy and observed L.F coverage.
21

---

Published as a conference paper at ICLR 2021
C ON LABELING FUNCTION PROPERTIES
In this section we analyze how LF accuracy and LF propensity (i.e. non-abstain behavior) influence the estimate of the true latent class label Y*. We focus the binary classification case for simplicity. Assume each data point & E X' has a latent class label y² E y= (-1, Given n unlabeled, f f== our goal is train classifier f X + y such that f(x) y². As in Ratner et al. 2 user provides m LFs {Az where 'ラ : X + y U {0} noisily label the data with (z)
ii.d. data points X
E
--11, 1, } or abstain with (z(x) 0. The corresponding LF output matrix is
A {-1, 1)mX""
where As, Xy (14).
We define a factor graph as proposed in Ratner et al. (016 2020) to obtain probabilistic labels by modeling the LF accuracies via factor di (A, Y) = [{A,, = y and labeling propensity by factor LLab (A, Y) - I{A, A 03, and for simplicity assume LFs are independent conditional Y. The label model is defined as
pe DY, A) 2 Zg exp Σ E (A..
(8)
where Zo is a normalizing constant and (A, 34) defined to be the concatenation of the factors for all LFs j 1,.... for sample i. Also, let 0 (Ø(1), g(2)) where g(1), g)() R". Here, g(1) the canonical parameters for the LF accuracies, and ((2) the canonical parameters for the LF propensities.
To estimate the label model parameters, generally obtain the maximum marginal likelihood estimate via the (scaled) log likelihood
((0) 1/n log
Σ DEy
Let finite å e R be such estimate. We Pa Ps(y|A,) As) to obtain probabilistic labels:
Pa(Ui = k(A.)) Pa(ui = Ai)
(9)
Pā(A,) exp exp(E,= m 8 1 d exp(E /00 (X
(10)
Note that the label estimate does not directly depend on ((2), Further, note that the denominator is the same over different label possibilities. Finally, note that even case where we include correlation factors &Cor (A, Y) 1 (A. = }. (j, E C in the model above with C as a set of potential dependencies, the probabilistic label will only directly depend on the estimated canonical accuracy parameters ((1), In the binary classification case, which we assume here, the expression simplifies further. For l E 1);
exp m 0 A (A,(s)) Σ E exp(E, m 5ー / (A
Pa(ys k/A,)
(11)
(12)
1+ exp((mm f= 0(00 (s)A00
(0000 k)))
a 0900 (1 (A)(())) sAce @Ace (My(2r),, (0((a))--h)) -k)));
(13)
where denotes the sigmoid function. The probabilistic labels are softmax in the multi-class classification case and, as shown above, simplify to sigmoid in the binary case. An absolute label prediction ÿ E {-1, 1} is therefore simply a function of
0 arg max T(Aa) arg max JEY
(1) & Ace (A.((A)I)))
JEY
22

---

Published as a conference paper at ICLR 2021
We now introduce assumptions the accuracy and error probabilities of labeling functions, similar to the Homogenous Dawid-Skens model (Dawid Skene 1979 Li et al. 2013) in crowd sourcing, where label source accuracy is the same across classes and errors are evenly divided with probability mass independent the true class.
Under these assumptions, denote by O; P()y(z) = (²)))()) o 0) the accuracy of LF Further, we denote by 1 = (z) + 0) the labeling propensity of j. ie. how frequently LF j does not abstain. The observed LF propensity is also referred as LF coverage in the related literature. We recall theorem 3.1
Theorem 3.1. Assume a binary classification setting, m. independent labeling functions with accuracy 0j E [0, 1] and labeling propensity 1j E [0, 1]. For a label model in Eq. 7 parameters 0 R2²,, and for any E {1.....n..
with given label model
P(ú -(} exp (- [_,= 05 (2a
2110211112
where (()) are the weights of AAc and D; {-1,1} is the label model estimate for y.
Proof. Assume that we use the label model to obtain a label estimate ds E {-1, As shown in Eq. 10 the prediction rule in that case is
( Ace (ece(A)(2;)) (A,(z;),
Ús = max
8E(-1.1)
Define by A(x) = (x), Am the vector of the j = 1....,m m LF outputs on x. Further, we define for {-1,1];
Va(A(t) Σ 00 (o (Ax(x),))),,)))))))))))))))))))) (A,)))
m
D -
==1
For the two label options l E (1,, we have
Vo(A(z), =
= -1) -
000 Xz(x)
and
m
vo(A(),--1)
Σ 0 (L(2)()) = -1 1()(() = 1}) = - -6 21
j=1
j=1
Now, we want to obtain a bound on the probability that the label estimate ÿ. is equal to the true label. We have
P(ú: vt)
P(y P() 1/y 1) 1)+ P( - - 11)
P(v P() 1/ 1) P(ył = P(}s = -1 =11).
Note that
P(ss = 1// = =) = P(V)()(a:)))
0/4 = P
(1) (x > 0/4 = 1),
j=1
23

---

Published as a conference paper at ICLR 2021
and that
P(ù: = 11yy -1) = P(Va(A(x;)) -1) 0/v²" = -1) P(
2y < 0/v -1).
We therefore have
m
m
P(ú: = uf = P(vs- = P)
g ]
(2;) > 0lv = 1) + (1 P(yi 1))
< Olyt -)).
j=1
Now we define E.
= Xy (2;) and we know that & || 1, 10
Given the Dawid-Skenc
model assumptions stated previously, we have
m
m
m
E
Eol 11
Eleasl - 1] Σ
12
==1
j=1
and
m
Solut = -1 --=1 Σ El 1u - = Σ
12200
Now, using Hoeffding's 's inequality and assuming independent labeling functions, can bound P(u: = 1, = 1) and P(ù: -1 = 1) from below:
P g (o))) 0/07 1) P Σ
Sig
0/07 1)
j=1
P( Σ
Eij
E >
Silut = =1] > Σ
12 x aj - 1) 105 = 1)
1=1
j=l
j=1
(Σi=) m
8/51) (2a, - 1)
2 - exp
211002)112
and
m
m
P(
8 (xx:)
< 0 -) P
Eig < <0 -1)
j
j=1
m
= P
Eig 18| Σ
Eig = -1] <
Σ
1 * *aj - 1) 1ot = -11
j=1
j=1
j=1
(- - m
(200
(20g 1)22}3
2 11-exp cxp
2110(11112
Finally have
p(ù v:) = P(y) = 1)P((r 1 1) + (1 Plui 1))P(h: -1/96 1)
mm (-227 0.52) (2az - -1)4)3
2 1 exp
2110(11112
What do the theorem and the quantities analyzed in this section indicate?
The trade-off between LF accuracy and LF propensity (also referred as LF coverage) is captured by (2a; 1)l, which allows us to rank LFs if we know the accuracy ay or can estimate and use the observed, empirical coverage estimate of 15-
24

---

Published as a conference paper at ICLR 2021
Not surprising, the relation between sign(f,) and aj is important. A better than random LF j should have a positive fz. This indicates that gap to randomness is important if we cannot guarantee that learn 8 well, reduce the chance of obtaining negative 83 for better than random LF j. or vice versa.
Note how the label estimates are obtained in Eq. (T)) Increasing the 02 of am LF also effectively means reducing the impact other L.Fs have on a prediction. In particular when estimates are imperfect, gap to random accuracy of aj is important to obtain good label estimates. Intuitively, we do not want to add excessive noise by including LFs close to random unless can guarantee that their parameter estimate is appropriately low and has the correct sign.
C.I WHAT ARE THE OPTIMAL LABEL MODEL PARAMETERS?
Here we discuss the optimal 0 parameters for two cases: first for when we assume that the label model factor graph consists only of accuracy factors and second for when this label model also takes LF propensity into account.
C.1.1 ASSUMING ACCURACY FACTORS ONLY
In the previous section we assumed that we are given estimated parameters. Naturally, we may next ask ourselves what the optimal theta parameters are. Let us start with simple case. Let Y, H be random variables of the class variable and vector of LFs, respectively. Assume that we only model LF accuracy and define 1 [[(a)=y), to be the element- wise indicator function. Also, assume that the true distribution can be expressed by the following model:
P(Y = y. {(r);; 0) 7 exp (0T (A(*)=v))
(14)
7 = Σ exp (0 (((2)==))
(15)
v.A(z)
Note that can be written as
Z
E
exp 18 (x())=)) +
E ).A())A(())
exp (0T (x(=))=))
(()))
Σ
exp (0, 0_,1(
Σ
exp (0
(=
= (e + L)
Σ exp (0I+)))))))))))
g.A(s)-t
where L is the number of classes. Now, note that
P(H,=Y), Σ
P(Y = y, H = A(x))
(16)
(a)
Σ
7 exp (0 (x(x)==1)
(17)
{.) y
e Σ exp (0) (0I,
(18)
v.A(x)=:
Thus, P(H, = =Y)= "T Which implies
P(Hi==YL
- [-P(HH,
8, = in
(19)
Further, note that P(H, = Y) = P(H, = Y,H, + 0) = P(H, = H| + 0)P(H, * 0) is
combination of LF's accuracy and label propensity.
25

---

Published as a conference paper at ICLR 2021
C.1.2 ASSUMING ACCURACY FACTORS AND LF PROPENSITY FACTORS
We now analyze a slightly more elaborate label model which also models LF propensity. Again, Let Y, H denote the random variables for class labels and LFs, respectively.
7 exp (DUT [((())=)) + ((2)T 1 (((0))=0))
1
P(Y y, H 0,)
(20)
Z Σ exp ( + B(2) 1 ((AC)))0))
(21)
v.A(z)
Z can be written as
Z Σ
exp (DUTI (A(x)=))) + + ((M(E)#O1)
E
exp (DUIT [((()==) + +0(2) ((x(a))0))
v.
).A(=))(t))
=
Σ
exp 1000 + 0 + 1 T ((x(0)) + + T
1 1(1(5)
Σ
exp (0(2) + 1 1 12(=)) .v) + O (A(o) (x0)
+
Σ exp (8()T
[((a)) + ( 1 (((z)
+
(=))=)
= = +600))
+ - Ijee +L-
+1) Σ exp (04 (((=))=))= + + 1 (x0))
B.A(z)-
The likelihood of a correct vote is given by
p(H,=Y)
Σ
P(Y y, H = X(x))
(22)
Ma)
Σ
exp (B(U)T ((())==)) + + [((())00))
(23)
Ma)a)
+0
Σ exp (B_!)T 1(x()) : + 0 1 (2(z) .t41)
(24)
2
{(((z)-
Further note that
P(H,=0)
Σ
P(Y = y. = A(x))
v.A(w) A())==
Σ
1
(()))=))== + (()) 1 1(M)))00)
exp
Z
v.)(w) A(x)==0
1
Σ
exp (1))
L(M()) + 0 (((2))
Z
1
+ (L 100 +1
So express
P(U, Y 00) = =Y,
P(H; YY)
P(H, YY)
(25)
P(H,, #0)
P(H, #0)
1 P(H, =)
(26)
0003 + (L - 1) 2
1
P(I,, 00 = +
(2
+ (L 1) +1
26

---

Published as a conference paper at ICLR 2021
Solving for 0 and 0 find
00 In
(L - 1)aj
(28)
1 - ajj
00- In
(1 0g) }4
(29)
(L - 1)(1 11)
where a; P(H; P + 0) and 13 = P(H, + 0). Note that in the binary case 0f10 is positive
only when ạ2 0.5.
27


Reviews:
[{"review_id": "uh3QBgI9izL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new framework for interactively selecting labeling heuristics in a weakly supervised setting. The main idea of the proposed approach is to combine weak supervision and active learning. Compared to the previous work which relies on human manually create labeling functions (the abstraction of the weak supervision), this work defines a family of labeling function and uses an active learning method to interactively identify a set of labeling functions that maximizes the utility based on the usefulness by the users. The experiment results showed that the proposed approach outperforms other baseline methods.\n\nMain comments:\nThe idea of using an active learning approach to select useful labeling functions from a large pool of labeling functions based on usefulness is interesting and the paper presents compelling solutions to a practical problem. However, the paper lacks clarity to help understand the technical contributions. Here are detailed comments:\n\n1. For labeling function family, the authors defined the labeling function family as sets of expert-interpretable weak supervision sources and shared several examples in different domains. However, there is lack of description of how to free the users to generate such labeling function family or how to adapt the existing labeling families to users' applications, which makes it hard for the readers to understand the exact user interaction in the framework. It's crucial to provide more details here.\n\n2. For the final subset of labeling functions, the authors defined the three scenarios. The distinctions of these different scenarios are not clear. What's the relation between the size of labeling functions selected in the framework and the # of iterations to select the labeling functions? There are some other unclear parts:\n  -- For scenario (A), if there are no restrictions about labeling functions, it seems like Eq (4) is not necessary since the framework only needs to find all useful labeling functions.\n  -- For scenario (B), it seems that this is the only useful case.\n  -- For scenario (C), it seems that this is only a special case.\n  -- Can you show the algorithmic procedure of LWS-AS? It's a little confusing about relations between the usefullness of LF and LF validated by experts. What's the difference there?\n  -- Overall, Section 3 should be clarified to help reader to understand the model.\n\n3. For the experimental evaluation, the proposed method outperforms the baseline on different tasks. There are several unclear parts:\n  -- The comparison with Snuba, it's not fair to compare # of iterations for proposed methods with the # of labeled data since in each iteration the proposed method will apply one more labeling function which likely to label more than 1 sample.\n  -- For the IWS-LSE top 100, ac, the authors didn't define what top 100 is. If it's $m$, it's not clear why the performance is improving when the # of iterations is greater than 100.\n  -- The approach only evaluated on the text classification problem, it would strengthen the paper if the authors can evaluate in another domain, such as image classification. Snuba has some image applications.\n\nOverall, I like the direction and idea, and I think this paper can be a great paper if the authors can clarify how real users interact with the framework and provide a more thorough experimental analysis.\n\n------ Post Rebuttal------\n\nThanks to the authors for the new experiments and feedback!\n\nThe rebuttal addressed most of my comments and increased my score by one point.", "extra_scores": {}, "extra_text": {}}, {"review_id": "fepQqGc6yOL", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies a very interesting problem in weak supervision. One of the main challenges with weak supervision approaches is how to generate the weakly labeled data. The paper suggests an interactive framework for generating and validating labeling functions to generate weakly supervised data.\n\nThe user experiments are interesting and  provide very good insights about the method but there are several open questions about the experiments:\n\n- The paper defines an LF family based on the existence of unigrams and exhaustively generate LFs from that family. I understand this would work for sentiment classification (the tow tasks the experiments consider). However, it is not clear to me that this would generalized to other text classification tasks (not to mention language tasks beyond text classification)\n\n- It is interesting to measure and show the effort of validating labeling functions vs. labeling samples. However, I am not sure if that results would hold beyond sentiment analysis too. Looking at the appendix, it seems most LFs are positive words such as \"wonderful\" and \"perfect\" or negative words such as \"poor\" or \"bad\". These will be significantly easier to judge by a human than other text classification tasks (e.g. the \"Salman Rushdie\" example in the introduction)\n\n- It would be interesting to compare the proposed approach to a setting where a user is asked to generate LFs based on a description of the tasks and a few examples.\n\n- Regarding the comparison with Snuba, Suba uses a small labeled dataset  to generate heuristics. How was that implemented in the experiments?\n\n----------------------------------\nEdited after authors responses: I would like to thank the authors for the detailed response and the changes they have made to the paper. While I still have some concerns and questions about generalization to more complex tasks, models and labeling functions, I think the additional experiments demonstrate the value of the proposed framework and opens the door for future work to explore these questions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GFYXkkW_Yji", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new approach for active learning by interactively discovering weak supervision. Instead of asking human to annotate data points, the method collects feedback about candidate label functions, from which a model learns to identify promising label functions. With the final set of label functions, they train a classifier with the estimated labels on unlabeled data. They conduct experiments on text classification datasets with both oracle and human feedback, and show a large improvement compared with traditional active learning.\n\nThe paper is well-motivated and the proposed active learning strategy is derived based on theoretical guarantees. I have a few questions about the framework: \n\n-- Does the system start training without any knowledge of the classification task (i.e. no label functions)? If so, how can the model distinguish between the candidates in the beginning? \n\n-- Is the expert-feedback model updated after each human feedback? How is the model updated?\n\nI suggest to add a step-by-step description of the system to help the readers understand the pipeline.\n\nThe experiments have been performed on several text classification tasks, where the label function family contains only uni-gram indicators. The end classifier is a small MLP on bag-of-words features. I have concerns about its applicability to real-world NLP tasks. Can the proposed method be extended to more complicated (and useful) label functions such as regular expressions? If we use more advanced text classifiers like BERT, can the proposed strategy still help? Since here the feedback from human is just whether a uni-gram is indicative to the label, which can be easily learned by a neural model.\n\nMissing reference:\nSrivastava, Shashank, Igor Labutov, and Tom Mitchell. \"Learning to Ask for Conversational Machine Learning.\" EMNLP 2019.\nIt also seeks human supervision beyond the instance level.\n\n------\nAfter author response:\n\nThe authors presented new experiments on image datasets during the rebuttal, which demonstrate the flexibility of the proposed framework. However, all the experiments are conducted on simple tasks and models. It is still unclear whether and how this method can help more practical problems.\n\nOverall, I think this paper presents a nice exploration towards interactive weak supervision. I hope the authors can release their code and experiment details to encourage future work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "ET3dmhdxLaY", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper argues that identifying weak supervision signals may require domain expert knowledge and creativity. Therefore, the authors propose an interactive weak supervision solution, in which the annotators assess the quality of automatically identified labelling functions. The proposed solution shows superior performance on a number of classification benchmarks after some minimum number of iterations.\n\nStrengths:\n1. The paper studies an important and novel problem in the domain of weak supervision\n2. The proposed solution outperforms the baselines\n3. The authors show some promising results by running experiments with real human annotators.\n4. The paper is well written and easy to follow.\n\nWeaknesses:\n1. A large body of work on weak supervision has been completely ignored by the authors. These are some relevant papers:\n- Using weak supervision beyond classification (for ranking): https://dl.acm.org/doi/10.1145/3077136.3080832\n- Learning from multiple weak supervision signals: https://dl.acm.org/doi/10.1145/3209978.3210041\n- Some theoretical justifications for weak supervision training (for ranking): https://dl.acm.org/doi/10.1145/3234944.3234968\n\n2. Fidelity-weighted learning (ICLR 2018) was proposed to automatically identify the quality of weak supervision signals. It is different from the proposed solution in the sense that a small set of labeled instances are used to assess the quality of weak supervision signals as opposed to labeling the weak supervision functions themselves. However, the goal is the same and some comparison (or at least some discussions) are necessary for the paper, I believe.\n\n3. There is no discussion on going beyond classification tasks and even some extreme classification scenarios.\n\n4. I think there should be some difficulty estimation associated with each weak supervision annotation function. Some functions may not be easily assessed by human annotators and this may influence the model. I think this can be addressed in future work.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a user-interaction framework where users choose a subset of LFs from a family of LFs generated using some template (e.g. keywords for text classification).   The proposed criteria is not very surprising, but the authors present a practical and useful system that is well demonstrated both in the paper and the very careful author feedback.  These enhancements have also been incorporated in the revised version. 

 Apart from the literature pointed by the reviewers, here are some more papers that are related to this paper:
1. Gregory Druck, Burr Settles, Andrew McCallum:
Active Learning by Labeling Features. EMNLP 2009: 81-90

2. 	Gregory Druck, Gideon S. Mann, Andrew McCallum:
Learning from labeled features using generalized expectation criteria. SIGIR 2008: 595-602

 3. Data Programming using Continuous and Quality-Guided Labeling Functions. In AAAI, 2020.
Decision: Accept (Poster)