End-to-End Weak Supervision
Authors: Salva Rühling Cachay
Benedikt Boecking
Artur Dubrawski
Venue: NeurIPS 2021 Poster (2021)
Keywords: weak supervision
deep learning
data programming

Abstract: Aggregating multiple sources of weak supervision (WS) can ease the data-labeling bottleneck prevalent in many machine learning applications, by replacing the tedious manual collection of ground truth labels. 
Current state of the art approaches that do not use any labeled training data, however, require two separate modeling steps: Learning a probabilistic latent variable model based on the WS sources -- making assumptions that rarely hold in practice -- followed by downstream model training. 
Importantly, the first step of modeling does not consider the performance of the downstream model.
To address these caveats we propose an end-to-end approach for directly learning the downstream model by maximizing its agreement with probabilistic labels generated by reparameterizing previous probabilistic posteriors with a neural network. 
Our results show improved performance over prior work in terms of end model performance on downstream test sets, as well as in terms of improved robustness to dependencies among weak supervision sources. 


End-to-End Weak Supervision
Salva Rühling Cachay
Benedikt Boecking'
Artur Dubrawski!
Carnegie Mellon University
Technical University of Darmstadt
Abstract
Aggregating multiple sources of weak supervision (WS) can ease the data-labeling bottleneck prevalent many machine learning applications, by replacing the tedious manual collection of ground truth labels. Current state of the art approaches that do not use any labeled training data, however, require two separate modeling steps: Learning probabilistic latent variable model based on the WS sources making assumptions that rarely hold in practice - followed by downstream model training. Importantly, the first step of modeling does not consider the performance of the downstream model. To address these caveats we propose an end-to-end approach for directly learning the downstream model by maximizing its agreement with probabilistic labels generated by reparameterizing prior probabilistic posteriors with a neural network. Our results show improved performance over prior work terms of end model performance on downstream test sets, as well as in terms of improved robustness dependencies among weak supervision sources.
1
Introduction
The success of supervised machine learning methods relies on the availability of large amounts of labeled data. The common process of manual data annotation by humans, especially when domain experts need to be involved, expensive, both in terms of effort and cost, and as such presents a major bottleneck for deploying supervised learning methods to new domains and applications.
Recently, data programming, paradigm that makes use of multiple sources of noisy labels, has emerged as a promising alternative to manual data annotation [30]. It encompasses previous paradigms
such as distant supervision from external knowledge bases [29, 34], crowdsoureing [15, 26, 14, 42], and general heuristic and rule-based labeling of data [23, 20]. In the data programming framework, users encode domain knowledge into so called labeling functions (LFs), which are functions (e.g. domain heuristics knowledge base derived rules) that noisily label subsets of data. The main task for learning from multiple sources of weak supervision then to recover the sources' accuracies in order to estimate the latent true label, without access to ground truth data. In previous work [30, 32, 19], this achieved by first learning a generative probabilistic graphical model (PGM) over the weak supervision sources and the latent true label to estimate probabilistic labels, which are then used in the second step to train a downstream model via a noise-aware loss function.
Data programming has led to a wide variety of success stories in domains such as healthcare [18, 17] and e-commerce [5], but the existing PGM based frameworks still come with number of drawbacks. The separate PGM does not take the predictions of the downstream model into account, and indeed this model is trained independently unknown class label via PGM need to rely on computationally expensive approximate sampling methods [30], estimation of the full inverse of the LFs covariance matrix [32), or they need to make strong independence assumptions [19]. Furthermore, existing prior work and the associated theoretical analyses make assumptions that may not hold in practice [30. 32, 19], such as availability
the PGM. In addition, current approaches for estimating the
"salvaruchling@ @mm.l.commammmmmmmmmmm
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Encoder e
Downstream model f
O
X
X
f
ye = Po = c(2)
Maximize agreement
Figure I: For a task with unobserved ground truth labels y. given m sources of weak supervision X and training features X, WeaSEL trains a downstream model by maximizing the agreement of its predictions ys with probabilistic labels Ye Po(y = c| A) generated by reparameterizing the posterior of prior work with sample-dependent accuracy scores 0 produced by encoder network
of a well-specified generative model structure (i.e. that the dependencies and correlations between the weak sources have been correctly specified by the user), that LF errors are randomly distributed across samples, and that the latent label is independent of the features given the weak labels (i.e. only the joint distribution between the sources and labels needs be modeled).
We introduce WeaSEL, our Weakly Supervised End-to-end Learner model for training neural networks with, exclusively, multiple sources of weak supervision noisy signals for the latent labels. WeaSEL is based on 1) reparameterizing previous PGM based posteriors with a neural encoder network that produces accuracy scores for each weak supervision source; and 2) training the encoder and downstream model on the same target loss, using the other model's 's predictions as constant targets, maximize the agreement between both models. The proposed method needs no labeled training data, and neither assumes cumple-independent source accuracies nor redundant features for latent label modeling. We show empirically that it is not susceptible to highly correlated L.Fs. In addition, the proposed approach can learn from multiple probabilistic sources of weak supervision.
Our contributions include:
We introduce flexible, end-to-end method for learning models from multiple sources of weak supervision.
We empirically demonstrate that the method is naturally robust to adversarial sources as well as highly correlated weak supervision sources.
We release an open-source, end-to-end system for arbitrary Py downstream models that will allow practitioners to take advantage of our approach".
We show that our method outperforms, by as much 6.1 F1 points, state-of-the-ard latent label modeling approaches on out of relevant benchmark datasets, and achieves -fate-of-the-.e performance on a crowdsourcing dataset against methods specifically designed for this setting.
2 Related Work
Multi-source Weak Supervision The data programming paradigm [30] allows users to program- matically label data through multiple noisy sources of labels, by treating the true label latent variable of a generative PGM. Several approaches for learning the parameters of the generative model have been introduced [32, 19, 11] address computational complexity issues. Existing methods are susceptible to misspecification of the dependencies and correlations between the LFs,, which can lead substantial losses performance [8]. Indeed, it is common practice assume conditionally independent model - without any dependencies between the sources in popular libraries [5, 31] and related research [15, 2, 37, 7J, even though methods learn the intra-LF structure have been proposed [4, 39, 38]. In natural language processing, [40] consider more relaxed and broader forms
"https://gitaenn com.///taaaaaal//eesee

---

of weak supervision and introduce framework that uses virtual evidence prior belief over latent labels and their interdependencies, learning an end model jointly with the label model via variational EM. As in the approach proposed in this paper, the aforementioned methods do not assume any labeled training data, i.e. the downstream model is learned based solely on outputs of multiple LFs on unlabeled data. The traditional co-training paradigm [6] on the other hand is similar in spirit but requires some labeled data be available. Recent methods that study the co-training setup where labeled training data supplements multiple WS sources, include [3, 25]. Note that the experiments in [3, 25] rely on large pre-trained language models, making the applicability of the approach without such models or to non-text domains unclear.
Crowdsoureing Aggregating multiple noisy labels also core problem studied the crowd- sourcing literature. Common approaches model worker performance and the unknown label jointly [15, 14, 42] using expectation maximization (EM) or similar approaches. Some core dif- ferences to learning from weak supervision sources are that errors by crowdworkers are usually assumed to random, and that task assignment not always fixed but can be optimized for. The benefits of jointly optimizing the downstream model and the aggregator of the weak sources have been recognized in multiple end-to-end methods that have been proposed for the crowdsoureing problem setting [33, 22, 41, 27, 35, 9]. They often focus on image labeling and EM-like algorithms for modeling and aggregating the workers. Importantly, our proposed approach can be used in general applications with weak supervision from multiple sources without any restrictive assumptions specific to crowdsourcing, and we show that our approach outperforms the aforementioned methods on a crowdsourcing benchmark task.
3 End-to-End Weak Supervision
In this section we present our flexible base algorithm that we call WeaSEL, which can be extended to probabilis- tic sources and other network architec- tures (Section 7). See Algorithm 1 for its pseudocode.
Algorithm 1 WeaSEL: The proposed Weakly Supervised End-to-end Learning algorithm for learning from multiple weak supervision sources.
input: batch size n, networks e. f. inverse temperatures T2, noise-aware loss function L, class balance P()). for sampled minibatch (2(() (x))), X ( ( do
3.1 Problem Setup
for all & E {l...., .n] do
# Produce accuracy scores for all weak sources
Let (x,y) ~ D be the data generat-
0 (2(()) softmax (e(z(*))
ing distribution, where the unknown labels belong to one of C' classes: As in [30], users provide an unlabeled training set x ] N and mn labeling func- tions À -xx)
# Generate probabilistic labels
define (k)) as s(k) = ((=(()/T((
y E y [1,...CC,
Ve Po A (a)) softmax (s P(v)
(k)
# Downstream model forward pass
{0, 1, .... C)
yr
(k)
((x(k))
where 0 means that the LF abstained from labeling for any class. We write
end for
C Σ L V8 stop- grad ye
(k)
(k)
A (1 1)..... ... 1(x C}) (0. 1)mcC for the one-hot representa- tion of the LF votes provided by the m LFs for classes. Our goal is to train downstream model x y on noise-aware loss L Me) that
C. = = L update to minimize Le, and f to minimize Cj end for return downstream network f(-))
ye stop-grad grad
(k)
(1)
operates the model's predictions ys = f(x and probabilistic labels Ye generated by encoder model e that has access to LF votes, X, and features, x. Note that prior work restricts the probabilistic labels to only being estimated from the LFs.
3.2 Posterior Reparameterization
Previous PGM based approaches assume that the joint distribution p(>, y) of the LFs and the latent true label can be modeled as a Markov Random Field (MRF) with pairwise dependencies between weak supervision sources [30, 31, 32, 19. 11]. These models are parameterized by a set of LF accuracy and intra-LF correlation parameters and in some cases by additional parameters to model
3

---

LF and class label propensity. Note however, that the aforementioned models ignore features X when modeling the latent labels and therefore disregard that LFs may differ in their accuracy across samples and data slices.
We relax these assumptions, and instead view the latent label an aggregation of the L.F votes that is a function of the entire set of LF votes and features, on a the probability of particular sample x having the class label c y as
basis. That is, we model
Po(y = c A) = softmax (s), P(y = c),
(1)
0 x) X E R
(2)
where ((A,x) R weighs the LF votes on a sample- by-sample basis and the softmax for class c on s is defined as
softmax (s) exp (0 1
Σj=1 exp (0 x) x) 1{}=
While we do not use the class balance P(y) in our experiments for our own model, WeaSEL, it is frequently assumed to be known [32. 19, 11], and can be estimated from a small validation set, or using LF outputs as described in [32]. Our formulation can be seen as a reparameterization of the posterior of the pairwise MRFs in [31, 32, 19], where A corresponds to the LF accuracies that are fixed across the dataset and are solely learned via LF agreement and disagreement signals, ignoring the informative features. We further motivate this formulation and expand upon this connection in the appendix A.
3.3 Neural Encoder
Based on the setup introduced in the previous section and captured in Eq. (1), our goal is to estimate latent labels by means of learning sample-dependent accuracy scores 0(2,x), which we propose to parameterize by neural encoder e. This network takes as input the features and the corresponding LF outputs A(x) for a data point, and outputs unnormalized scores e(A, x) E R Specifically, we define
(3) where 72 is constant factor that scales the final softmax transformation in relation to the number of LFs m, and equivalent to an inverse temperature for the output softmax Eq. It motivated by the fact that most LFs are sparse in practice, and especially when the number of LFs is large this leads to small accuracy magnitudes without scaling (since, without scaling, the accuracies after the softmax sum up to onc)². Ti is an inverse temperature hyperparameter that controls the smoothness of the predicted accuracy scores: The lower 7₁ is, the less emphasis is given to a small number LFs T1 -> O, the model aggregates according the equal weighted vote. The softmax transformation naturally encodes our understanding of wanting to aggregate the weak sources to generate the latent label.
((A = T2 softmax (e(A, x)Ti),
3.4 Training the Encoder
The key question now is how to train e, i.e. how can we learn an accurate mapping of the sample-by- sample accuracies, given that we do not observe any labels?
First, note that initializing ewith random weights will lead to latent label estimates close to an equal weighted vote, which acts as a reasonable baseline for label models in data programming (and crowdsourcingg) where expectation votes of LFs are assumed to better than random guesses. Thus, Po(yl x, will provide a better than random initial guess for y. We hypothesize that in most practical cases, features, latent label, and labeling function aggregations are intrinsically correlated due to the design decisions made by the users defining the features and L.F.. Thus, we can jointly optimize e and f by maximizing their agreement with respect to the target downstream loss T, in an end-to-end manner. See Algorithm 1 for pseudocode of the resulting WeaSEL algorithm. The natural classification loss is the cross-entropy, which we use in our experiments, but in order to encode our desire of maximizing the agreement of the two separate models that predict based on different views of the data, we adapt it' the following form: The loss is symmetrized order to compute the
p In our main experiments we set 72 Vm.
"This holds for any asymmetric loss, while for symmetric losses this is not needed.

---

Table 1: The final test Fl performance of various multi-source weak supervision methods over seven runs, using different random seeds, are averaged out + standard deviation. The top 2 performance scores highlighted as First, Second. Triplet-median [11] is not listed it only converged for IMDB with 12 LFs (FI = 73.0 = 0.22), and Spouse (F1 = 48.7 = 1.0). The downstream model is the same for all methods. For Sup. (Val. set), and Majority vote it is trained on the hard labels induced by the labeled validation set and the majority vote of the LFs, respectively. For the rest it is trained on the probabilistic labels estimated by the respective state-of-the-ar latent label model. For reference, also report the Ground truth performance of the same downstream model trained on the true training labels (which are unused by all other models, and not available for Spouse).
Model
Spouse LFs)
ProfTeacher (99 LFs)
IMDB (I3)
IMDB (12 LFS)
Amazon (175
Ground truth
90.65 +: 0.29
86.72 1: 0.40
86.72
0.40
92.93 +: 0.68
Sup. (Val. set)
20.4 0.2
73.34 1 0.00
68.76 1 0.00
68.76
1 0.00
84.18 1 0.00
Snorkel
48.79 +: 2.69
85.12 2 0.54
82.22 1 0.18
74.45 + 0.58
80.54 + 0.41
Triplet
45.88
3.64
74.43 +: 10.59
75.36 +: 1.92
73.15 + 0.95
75.44 + 3:21
Triplet-Mean
49,94
1.47
82.58
0.32
79.03 1 0.26
73.18
0.23
79.44 0.68
Majority vote
40.67
2.01
85.44
+ 0.37
80.86 1: 0.28
74.13
0.31
84.20
0.52
WeaSEL
51.88 = 1.60
86,98
0,45
82,10 1 0.45
77.22 2 1.02
86.60 = 0.71
gradient of both models using the other model's predictions as targets. To that end, it is crucial to use the stop-grat operation on the targets (the second argument of L.), i.e. to treat them as though they were ground truth labels. This choice is supported by our synthetic experiment and ablations. This operation has also been shown to be crucial in siamese, non-contrastive, self-supervised learning, both empirically [21, 12] and theoretically [36]. By minimizing simultaneously, both, L m/) and L(ys Ye) to jointly learn the network parameters for e and the downstream model f respectively, we learn the accuracies of the noisy sources À that best explain the patterns observed in the data, and vice versa the feature-based predictions that are best explained by aggregations of LF voting patterns.
3.5 WeaSEL Design Choices
Note that it is necessary to encode the inductive bias that the unobserved ground truth label y is a (normalized) linear combination of LF votes weighted by sample- and feature-dependent accuracy scores. Otherwise, the encoder network directly predicts Po X, x) instead of the accuracies 0( x). the pair of networks e, f have no incentive output the desired latent label, without observed labels. We do acknowledge that this two-player cooperation game with strong inductive biases could still allow for degenerate solutions. However, we empirically show that our simple WeaSEL model that goes beyond multiple earlier WS assumptions is 1) competitive and frequently outperforms state-of-the-are PGM-based and crowdsourcing models (see Tables and 2); and is robust against massive LF correlations and able to recover the performance of fully supervised model on a synthetic example, while all other models break in this setting (see section 4.3 and appendix F).
4 Experiments
Datasets As in related work on label models for weak supervision [30, 32, 19, 11]. we focus for simplicity on the binary classification case with unobserved ground truth labels E {-1, See Table 3 for details about dataset sizes and the number of LFs used. We also run an experiment on multi-class, crowdsoureing dataset (see subsection 4.2). We evaluate the proposed end-to-end system for learning a downstream model from multiple weak supervision sources on previously used benchmark datasets in weak supervision work [31, 7, 11]. Specifically, we evaluate test set performance on the following classification datasets:
The IMDB movie review dataset [28] contains movie reviews be classified into positive and negative sentiment. We run two separate experiments, where in one we use the same 12 labeling functions as in [11], and for the other we choose 136 text-pattern based LFs. More details on the LFs can be found in the appendix C.
A subset of the Amazon review dataset [24], where the task is to classify product reviews into positive and negative sentiment.
5

---

We use the BiasBios biographies dataset [16] to distinguish between binary categories of fre- quently occurring occupations and use the same subset of professor vs teacher classification as in [7].
Finally, we use the highly unbalanced Spouse dataset (90% negative class labels), where the task is to identify mentions of spouse relationships amongst set of news articles from the Signal Media Dataset [13].
For the Spouse dataset, the same data split and LFs as in [19] are used, while for the rest we take a small subset of the test set as validation set. This is common practice in the related work [31, 32, 19, 7] for tuning hyperparameters, and allows for a fair comparison of models.
4.1 Benchmarking Weak Supervision Label Models
To evaluate the proposed system, we benchmark it against state-of-the- systems that aggregate multiple weak supervision sources for classification problems, without any labeled training data. We compare our proposed approach with the following systems: 1) Snorkel, popular system proposed in [31, 2) Triplet, exploits a closed-form solution for binary classification under certain assumptions [19]; and 3) Triplet-mean and Triplet-median [11], which are follow-up methods based on Triplet with the aim of making the method more robust.
We report the held-out test set performance of WeaSEL's 's downstream model f. Note that in many settings it is often not possible to apply the encoder model to make predictions at test time, since the LFs usually do not cover all data points (e.g. in Spouse only 25.8% of training samples get at least one LF vote), and can be difficult to apply to new samples (c.g. when the LFs are crowdsourced points.
annotations). In contrast, the downstream model is expected to generalize to arbitrary unseen data
We observe strong results for our model, with 4 out of 5 top scores, and a lift of 6.1 F1 points over the next best label model-based method in the Amazon dataset. Our results are summarized in Table 1. Since our model is based performance on the Amazon review dataset due to being the largest dataset size on which we evaluate
neural network, we hypothesize that the large relative lift in
expect this lift to hold or become larger as the training set size increases. To obtain the comparisons shown in Table 1, we run Snorkel over six different label model hyperparameter configurations, and train the downstream model the labels estimated by the label model with the best AUC score on the validation set. We do not report Triplet-median in the main table, since only converged for the two tasks with very small numbers of labeling funetions. Interestingly, observed that training the downstream model on the hard labels induced by majority vote leads to a competitive performance, better than triplet methods in four out of five datasets. This baseline is not reported in previous papers (only the raw majority vote is usually reported, without training classifier). Our own model, WeaSEL, on the other hand consistently improves over the majority vote baseline (which in Table 4, in the appendix. can be seen to lead to similar performance as an untrained encoder network, e, that is left its random initialization).
4.2 Crowdsourcing dataset
Data programming and crowdsourcing methods have been rarely compared against each other, even though the problem setup is quite similar. Indeed, end- to-end systems specifically for crowdsourcing have been proposed [33, 27, 35, 9]. These methods fol- low crowdsoureing speciifc assumptions and modeling choices (c.g. independent crowdworkers, a confusion matrix model for each worker, and in general build upon [15]). Still, since crowdworkers can be seen as specific type of labeling functions, the performance of general WS methods
Table 2: Test accuracy scores on the crowd- sourced, multi-class LabelMe image classi- fication dataset.
Model
Accuracy
Majority vote
79.23 + 0.5
MBEM [27]
76.84 + 0.4
DoctorNet [22]
81.31 + 0.4
CrowdLayer [35]
82.83 0.4
AggNet [i]
84.35 0.4
crowdsourcing datasets is of interest, but has so far not been studied. We therefore choose to also evaluate our method on the multi-class LabelMe image classification datasct that was previ- ously used in the core related crowdsourcing literature
MaxMIG [9]
85.45 + 1.0
Snorkel+CE
82.89 0.7
WeaSEL+CE
82.46 0.8
Snorkel+MIG
85.15
WeaSEL+MIG
86.36 + 0.3

---

Snotbel Fpleet
Topln-Mean
WeeiFi
Test ROC AUG
Feb Expereel
woor
0.0
0.8
0.5
Number rany Lf duplicates (a) Test Fl score on robustness experiment function of the number of adversarial LFs.
10
(b) Test AUC by epoch in an experiment where one LF corresponds to the true class label and others are random.
100
Figure 2: WeaSEL is significantly more robust against correlated adversarial (left) or random (right)
LFs than prior work whose assumptions make them equívalent to Naive Bayes model. For subfigure (a), we duplicate a fake adversarial LF up to 10 times, and observe that our end-to-end system is robust against the adversarial LF,, while other systems quickly degrade in performance (over ten random seeds). In (b), we let one LF be the true labels y and then duplicate a LF that votes according coin flip 2, 5, .... 2000 times. We plot the test AUC performance curve as function of the epochs, averaged out over the different number of duplications (and five random seeds). WeaSEL consistently recovers the test performance of the supervised end-model f trained directly on the true labels y" whose end performance (AUC- = 0.967) is shown in red.
[35, 9]. The results are reported in Table 2, and more details on this experiment can be found in Appendix E. Note that the evaluation procedure in [9] reports the best test set performance for all models, while we follow the more standard practice of reporting results obtained by tuning based small validation set - as in our main experiments. We find that our model, WeaSEL, is able to outperform Snorkel as well as multiple state-of-the-art methods that were specifically designed for crowdsourcing (including several end-to-end approaches), Interestingly, this is achieved by using the mutual information gain loss (MIG) function introduced in [9], which significantly boosts performance of both Snorkel (the end-model. f. trained on the MIG loss with respect to soft labels generated by the first Snorkel label model step) and WeaSEL that use the cross-entropy (CE) loss. This suggests that the MIG loss is a great choice for the special case of crowdsoureing, due to its strong assumptions common to crowdsourcing which are much less likely to hold for general LFs. This reflected in our ablations too, where using the MIG loss leads to a consistently worse performance on our main multi- weak supervision datasets.
4.3 Robustness to Adversarial LFs and LF correlations
Users will sometimes generate sources they mistakenly think are accurate. This also encompasses the 'Spammer' crowdworker-type studied in the crowdsourcing literature. Therefore, it is desirable to build models that are robust against such sources. We argue that our system that is trained by maximizing the agreement between an aggregation of the sources and the downstream model's predictions should be able to distinguish the adversarial sources. In Fig. 2a we show that our system does not degrade in its initial performance, even after duplicating an adversarial LF ten times. Prior latent label models, on the other hand, rapidly degrade, given that they often assume the weak label sources to be conditionally independent given the latent label, equivalent to a Naive Bayes generative model. Note that the popular open-souree implementation of [31, 32] does not support user-provided LF dependencies modeling, while [19, 11] did not converge in our experiments when modeling dependencies, and as such were not able to test their performance when the correlation dependencies between the duplicates are provided (which in practice, of course, are not known).
We also run a synthetic experiment inspired by [9]. where one LF is set to the true labels of the ProfTeacher dataset, i.e. \ = y", while the other LF simply votes according to a coin flip, i.e. A2 P(y), and we then duplicate this latter LF, i.e. Ag =... Am = Xa. Under this setting, our WeaSEL model able to consistently recover the fully supervised performance of the same downstream model directly trained on the true labels y", even when we duplicate the rundom LF up to 2000 times (m = 2001). Snorkel and triplet methods, on the other hand, were unable to recover the true label (AUC as 0.5). Importantly, we find that the design choices for WeaSEL are large
7

---

Table Dataset details, where training, validation and test set sizes are Ntrains Neal- Neest respec- tively, and denotes the downstream model type. We also report the total coverage Cov. of all LFs, which refers to the percentage training samples which are labeled by at least one LF (the rest not used). For IMDB we used two different sets of labeling functions of sizes 12 and 136.
Dataset
#LFs
Ntrain
Cov. (in %)
Nval
Nees
f
Spouse
22, 254
25.8
2811
2701
LSTM
BiasBios
99
12, 294
81.8
250
12, 04.
MLP
IMDB
12
25k:
88.0
250
24.
MLP
IMDB
136
25k
83.1
250
250
MLP
Amazon
175
160k
65.5
500
39, 500
MLP
extent key in order to recover the true labels in stable manner as in Fig. 2b. Various other choices either collapse similarly to the baselines, are not able to fully recover the supervised performance, or lead to unstable test performance curves, see Fig. 5 in the appendix. More details on the experimental design and an extensive discussion, ablation, and figures based on the synthetic experiment can be found in the appendix F.
4.4 Implementation Details
Here we provide high-level overview over the used encoder architecture, the LF sets, and the features. More details, especially hyperparameter and architecture details, are provided in Appendix C. All downstream models are trained with the (binary) cross-entropy loss, and our model with symmetric version of it that uses stop-gras grad on the targets.
Encoder network The encoder network does not need to follow a specific neural network archi- tecture and we therefore use simple multi-layer perceptron (MLP) in our benchmark experiments.
Features for the encoder A big advantage of our model is that it is able to take into account the features x for generating the sample-by-sampe source accuracies. For all datasets, we concatenate the LF outputs with the same features that used by the downstream model input of encoder model (for Spouse we use smaller embeddings than the ones used by the downstream LSTM).
Weak supervision sources For the Spouse dataset, and the IMDB variant with 12 LFs, we use the same LFs as in [19, 11] respectively. The remaining three LF sets were selected by us prior to running experiments. These LFs are all pattern- and regex- based heuristies, while the Spouse experiments also contain LFs that are distant supervision sources based on DBPedia.
5 Ablations
In this section we demonstrate the strength of the WeaSEL model design decisions. We perform extensive ablations all four main datasets but Spouse for twenty configurations of WeaSEL with different encoder architectures, hyperparameters, and loss functions. The tabular results and a more detailed discussion than in the following
be found in Appendix D.
We observe that ignoring the features when modeling the sample-dependent accuracies, i.e. 0(A, x) (()), usually underperforms by up to 1.2 Fl points. A more drastic drop in performance, up 4.9 points, occurs when the encoder network is linear, i.c. without hidden layers, as in [9]. also proves helpful to scale the softmax in Eq. by /m via the inverse temperature parameter 72- Further, while the MIG loss proved important for WeaSEL to achieve state-of-the-att performance on the crowdsourcing dataset (with similar lift in performance observable for Snorkel using MIG for downstream model training), this does not hold for the main datasets. This indicates that the MIG loss is good choice for crowdsourcing, but not for more general WS settings.
Our ablations also show that important to restrict the accuracies to a positive interval (e.g. (0, I). with the sigmoid function being a good alternative to the softmax we use). On the one hand, this encodes the inductive bias that LFs are not adversarial, i.c. can not have negative accuracies, (using tanh to output accuracy scores does not perform well), and on the other hand does not give the encoder

---

network too much freedom in the scale of the scores (using ReLU underperforms significantly as well).
Additionally, we find that our choice of using the symmetric cross-entropy loss with stop-gres grad applied the targets is crucial for the obtained strong performance of WeaSEL. Removing the stop-grad operation, or using the standard cross-entropy (without stop-grad on the target) leads significantly worse scores and a very brittle model. Losses that already are symmetric (e.g. LI or Squared Hellinger loss) neither need to be symmetrized nor use stop-grad, While the LI loss consistently underperforms, find that the Squared Hellinger loss lead better performance on two of the four datasets.
However, only the symmetric cross-entropy loss with stop-grad on the targets is shown to be robust and able to recover the true labels in our synthetic experiment Section 4.3. Thus, to complement the above ablation on real datasets, we additionally Appendix F. This synthetic ablation gives interesting insights, and strongly supports the proposed design of WeaSEL. Indeed, many choices for WeaSEL that perform well enough on the real datasets, such as no features for the encoder, = 1, sigmoid parameterized accuracies, and all other losses that we evaluated, lead to significantly worse performance and less robust learning the synthetic adversarial setups.
extensive ablations this synthetic setup in
6 Practical Aspects and Limitations
On why it works degenerate solutions Overall, WeaSEL avoids trivial overfitting and degenerate solutions by hard-coding the encoder generated labels as a (normalized) linear combination of the m LF outputs, weighted by m sample-dependent accuracy scores. This design choice also ensures that the randomly initialized e will lead the downstream model f that is trained soft labels generated by the random encoder, to obtain performance similar to when f is trained on majority vote labels. In fact, the aandom-encoder-WeaSSH. variant itself often outperforms other baselines, and triplet methods in particular (see appendix B).
Empirically, only observed degenerate solutions when training for too many epochs. Early- stopping on a small validation set ensures that a strong final solution returned, and should be done whenever such set exists or is easy to create. When choosing the temperature hyperparameter in Eq. 3 such that Tι I 1/3 avoids collapsed solutions on all our datasets. This can be explained by the fact that lower inverse temperature forces the mncoder-prrdicted predicted label to always depend on multiple LF votes when available, rather than a single one (which happens when the softmax in Eq. 3 becomes for the encoder to overfit to individual LFs. Our ablations indicate that this temperature parameter setting comes at small cost in terms of loss in downstream performance, compared when using validation set for early stopping. Thus, when no validation set is available, we advise to lower TI.
validation set is available, we find that
encoder-
T1 - oo). This makes it harder
Complex downstream models We have shown that WeaSEL achieves competitive or state-of-the- art performance on all datasets we tried it on, for a given set of L.Fs. In practice, however, this LF set needs to first be defined by users. This can be done via iterative process, where the feedback is sourced from the quality of the probabilistic labels generated by the label model. limitation of our model, is that each such iteration would require training the downstream model, f. When f slow to train, this may slow down the LF development cycle and lead to unnecessary energy consumption. A practical solution this can be to a) do the iteration cycle with less complex downstream model; or b) use the fast to train PGM- based label models to choose good LF set, and then move to WeaSEL in order achieve better downstream performance.
7 Extensions
Probabilistic labeling functions Our learning method can easily support labeling functions that output continuous scores instead of discrete labels as in [10]. In particular, this includes probabilistic sources that output distribution the potential class labels. This can be encoded in our model by changing the one-hot representation of our base model continuous representation A [0, 1 1

---

Modeling more structure While we use simple multi-layer perceptron (MLP) our encoder our benchmark experiments, our formulation is flexible to support arbitrarily complex networks. In particular, we can naturally model dependencies amongst weak sources via edges in Graph Neural Network (GNN), where each L. represented by node that is given the L.F outputs features. Furthermore, while we only explicitly reparameterized the accuracy parameters of the sources in our base model, is straightforward to augment Ā with additional sufficient statistics, e.g. the fixing or priority dependencies from [30, 8] that encode that one source fixes (i.e. should be given priority over) the other whenever both vote.
8 Conclusion
proposed WeaSEL, a new approach for end-to-end learning of neural network models classifi- cation from, exclusively, multiple sources of weak supervision that streamlines prior latent variable models. We evaluated the proposed approach on benchmark datasets and observe that the downstream models outperform state-of-the-e data programming approaches in 4 out of 5 cases while remaining highly competitive on the remaining task, and outperforming several statc-of-the-a crowdsourcing methods a crowdsourcing task. We also demonstrated that our integrated approach can be more robust to dependencies between the labeling functions as well as to adversarial labeling scenarios. The proposed method works with discrete and probabilistic labeling functions and utilize various neural network designs for probabilistic label generation. This end-to-end approach can simplify the process of developing effective machine learning models using weak supervision the primary source of training signal, and help adoption of this form of learning in a wide range of practical applications.
10

---

Acknowledgements
This work was made possible thanks to Carnegie Mellon University's Robotics Institute Summer Scholars program and was partially supported by Space Technology Research Institutes grant from NASA's Space Technology Research Grants Program, and by Defense Advanced Research Projects Agency's 's awards PA8750-17-2-0130 and HR001 1RR0011-11-9-0755.
References
[1] S. Albarqouni, C. Baur, F. Achilles, V. Belagiannis, S. Demirci, and N. Navab. Aggnet: Deep learning from crowds for mitosis detection breast cancer histology images. IEEE Transactions on Medical Imaging. (5(5):313-13211 2016.
[2] Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. Journal of Machine Learning Research, 15:
2773-2832, 2014.
[3] Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. Learning from rules generalizing labeled exemplars. In ICLR, 2020.
[4] Stephen H. Bach, Bryan He, Alexander Ratner, and Christopher Ré. Learning the structure of generative models without labeled data. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, ICML' 17, page 273-282, 2017.
[5] Stephen H. Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, Rahul Kuchhal, Chris Ré, and Rob Malkin. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In Proceedings of the 2019 International Conference on Management of Data, SIGMOD 119, page 362-375, New York, NY. USA, 2019. Association for Computing Machinery. ISBN 0781450356435. doi: 10. 141113299899..... 33016666...66.
[6] Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Proceedings of the Eleventh Annual Conference on Computational Learning Theory, COLT" 98, page 92-100, New York, NY, USA, 1998. Association for Computing Machinery. ISBN doi: 10.1 10.1145/9994332799962 1581 130570.
[7] Benedikt Boecking, Willie Neiswanger, Eric Xing, and Artur Dubrawski. Interactive weak supervision: Learning useful heuristics for data labeling. ICLR, 2021.
[8] Salva Rühling Cachay, Benedikt Boecking, and Artur Dubrawski. Dependency structure mis- specification multi-source weak supervision models. ICLR Workshop on Weakly Supervised Learning, 2021.
[9] Peng Cao, Yilun Xu, Yuqing Kong, and Yizhou Wang. Max-mig: an information theoretic approach for joint learning from crowds. ICLR, 2019.
[10] Oishik Chatterjee, Ganesh Ramakrishnan, and Sunita Sarawagi. Data programming using continuous and quality-guided labeling functions. AAAI, 2020.
Cohen-Wang, [11] Mayee Chen, Benjamin Cohen- Wang, Steve Mussmann, Frederic Sala, and Christopher Ré. Comparing the value of labeled and unlabeled data in method-of-moments latent variable estimation. AISTATS, 2021.
[12] Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
[13] David Corney, Dyaa Albakour, Miguel Martinez-Alvarez, and Samir Moussa. What do million news articles look like? Workshop on Recent Trends in News Information Retrieval, pages 42-47, 2016.
[14] Nilesh Dalvi, Anirban Dasgupta, Ravi Kumar, and Vibhor Rastogi. Aggregating crowdsoureed binary ratings. In Proceedings of the 22nd international conference on World Wide Web, pages 285-294, 2013.
[15] A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society. Series C (Applied Statistics), 28(1):
20-28, 1979. ISSN 00359254, 14679876.

---

[16] Maria Ar-Artaaga, Alexey Romanov, Hanna Wallach, Jennifer Chayes, Christian Borgs, Alexan- dra Chouldechova, Sahin Geyik, Krishnaram Kenthapadi, and Adam Tauman Kalai. Bias in bios: A case study of semantic representation bias in high-stakes setting. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pages 120-128, 2019.
[17] Jared A Dunnmon, Alexander J Ratner, Khaled Saab, Nishith Khandwala, Matthew Markert, Hersh Sagreiya, Roger Goldman, Christopher Lee-Messer, Matthew P Lungren, Daniel L. Rubin, et al. Cross-modal data programming enables rapid medical machine learning. Patterns, 1(2): 100019, 2020.
[18] Jason A Fries, Paroma Varma, Vincent S Chen, Ke Xiao, Heliodoro Tejeda, Priyanka Saha, Jared Dunnmon, Henry Chubb, Shiraz Maskatia, Madalina Fiterau, et al. Weakly supervised classification of aortic valve malformations using unlabeled cardiac mri sequences. Nature Communications, 10(1):1-10, 2019.
[19] Daniel Y. Fu, Mayee F. Chen, Frederic Sala, Sarah Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. ICMI., 2020.
[20] Garrett Goh, Charles Siegel, Abhinav Vishnu, and Nathan Hodas, Using rule-based labels for weak supervised learning: a chemnet for transferable chemical property prediction. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 302-310, 2018.
[21] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent- a new approach to self-supervised learning. In Advances in Neural Information Processing Systems, volume 33, pages 21271-21284 2020.
[22] Melody Guan, Varun Gulshan, Andrew Dai, and Geoffrey Hinton. Who said what: Modeling individual labelers improves classification. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018
[23] Sonal Gupta and Christopher D Manning. Improved pattern learning for bootstrapped entity extraction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning, pages 108, 2014.
[24] Ruining He and Julian McAuley. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In Proceedings of the 25th International Conference on World Wide Web, pages 507-517, 2016.
[25] Giannis Karamanolakis, Subhabrata (Subho) Mukherjee, Guoqing Zheng, and Ahmed H. Awadallah. Self-training with weak supervision. In NAACL, 2021.
[26] David Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourcing systems. In Advances in Neural Information Processing Systems, volume 24, pages 1953-1961, Curran Associates, Inc., 2011.
[27] Ashish Khetan, Zachary C Lipton, and Anima Anandkumar. Learning from noisy singly linely-labeled data. ICLR, 2018
[28] Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics.
[29] Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 1003-1011, 2009.
[30] Alexander Ratner, Christopher De Sa, Sen WU, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. Advances in neural information processing systems, 29, 05 2016.
[31] Alexander Ratner, Stephen Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: rapid training data creation with weak supervision. The VLDB Journal, 29, 07 2019. doi: 10. 0001007/00077-0111 0117780--9-00052--
12

---

[32] Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi- task weak supervision. Proceedings of the AAAI Conference on Artificial Intelligence, 33:4763 -7771, 07 2019. doi: 20.1609/aaiivv3300 33014763.
[33] Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bogoni, and Linda Moy. Learning from crowds. Journal of Machine Learning Research, 1143)::2277-1322 2010.
[34] Sebastian Riedel, Limin Yao, and Andrew McCallum. Modeling relations and their mentions without labeled text. In Machine Learning and Knowledge Discovery in Databases, pages 148-163, Berlin, Heidelberg. 2010. Springer Berlin Heidelberg. ISBN 978-3-642-15939,
[35] Filipe Rodrigues and Francisco Pereira. Deep learning from crowds. In Proceedings of the AAAI Conference on Artificial Intelligence 2018.
[36] Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. ICML, 2021.
[37] Paroma Varma and Christopher Ré. Snuba: Automating weak supervision to label training data. In Proceedings of the VLDD Endowment. International Conference on Very Large Data Bases, page 223. NIH Public Access, 2018.
[38] Paroma Varma, Bryan D He, Payal Bajaj, Nishith Khandwala, Imon Banerjee, Daniel Rubin, and Christopher Ré. Inferring generative model structure with static analysis. In Advances in neural information processing systems, pages 240-250, 2017.
[39] Paroma Varma, Frederic Sala, Ann He, Alexander Ratner, and Christopher Ré. Learning dependency structures for weak supervision models. In International Conference on Machine Learning, pages 6418-6427. PMLR, 2019.
[40] Hai Wang and Hoifung Poon. Deep probabilistic logic: A unifying framework for indirect supervision. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1891-1902, 2018
[41] Peter Welinder, Steve Branson, Pietro Perona, and Serge Belongie. The multidimensional wisdom of crowds. In Advances in Neural Information Processing Systems, volume 23, pages 2424 2432. Curran Associates, Inc., 2010.
[42] Yuchen Zhang, Xi Chen, Dengyong Zhou, and Michael I Jordan. Spectral methods meet em: A provably optimal algorithm for crowdsourcing. The Journal of Machine Learning Research, 17 (1):3537-3580, 2016.
13


Reviews:
[{"review_id": "pDPxUmh6pS", "reviewer": "Reviewer_jomY", "summary": "The authors propose an \"end to end\" approach to weakly supervised learning. Instead of obtaining ground truth labels by modeling WS sources via a probabilistic latent variable model which don't consider the downstream performance, the authors consider a neural model that directly learning the downstream model by maximizing WS sources' agreement with probabilistic labels generated by reparameterizing previous probabilistic posteriors. The method outperforms prior approaches on five real-world datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- Originality: The method proposed in this work is new. Most of the existing work in weakly supervised learning focuses on only create accurate training labels via WS resources. Constrastingly, this work focuses on developing a \"end to end\" approach by condiering downstream performance.\n\n- Quality: The paper has overall good experimental results while some strong claims lack of theoretical or empirical support. For example, 1) The ablation section is weak. An ablation study is needed to show in which case or which condition the proposed approach outperform the prior works. 2) It would be great to show whether the proposed approach works for different types of models, e.g., transformer-based model.\n\n- Clarity: The writing is overall clear. While the experiment section is difficult to understand because of lack of organization, or flaws.\n\n- Significance: The key insight of the paper that weakly supervised learning can be benefit by direcly considering downstream performance. This paper proposes a end-to-end approach to weak supervision modeling. While overall novaltiy is minor.\n\n======== After rebuttal ========\n\nThank you for the detailed response. My questions are partially answered by the authors, and I would like to keep my original rating.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed their limitations on probabilistic labeling functions and modeling more structure.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "inpIc3VCSYm", "reviewer": "Reviewer_SW1L", "summary": "In this paper, the authors propose an end-to-end approach for directly learning downstream models under a weak supervision setting by maximizing its agreement with probabilistic labels generated by reparameterizing previous probabilistic posteriors with a Neural Network. Experiments show good improvement over previous works over a number of datasets. ", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Despite the claimed originality, similar to the provided reviews from ICML 2021, I find this paper is still a little hard to follow. \n1. The introduction could be better written to make it easier for readers outside the area to follow. \n2. from the problem setup in Line 95-Line 107, it seems the labeling functions just an ensemble of models performing the downstream task. Because it seems that all models in the $\\mathbf{\\lambda}$ perform the same $C$-way classification. This part needs to be defined more clear.\n3. How specifically did the authors compute $P(y=c)$?\n4. Figure 1 should have better quality. \n5. I also hope that the authors could clarify more on the uniqueness and novelty of their method. \n6. I'm eager to increase my score if the questions are well answered. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors briefly mentioned the limitations in the papers and my concerns are presented above. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5 hours"}, "extra_text": {}}, {"review_id": "dteY_jpp5lq", "reviewer": "Reviewer_zED9", "summary": "This paper proposes a new method to training networks with multiple weak labeling sources. In contrast to previous weak supervision approaches, which aggregate weak sources using a PGM *then* train a downstream classification model, the proposed method trains an encoder network (which learns to aggregate weak sources) in tandem with a classification model (which learns to directly classify input datapoints). The authors compare their proposed method to existing weak supervision tools and find their method leads to improved performance as well as improved robustness to adversarial and correlated weak labeling sources.\n\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Review summary.** This is an interesting and new approach to training networks with multiple weak labeling sources that seems to outperform existing weak supervision tools on the datasets tested in this submission. There are advantages of the proposed method---particularly, the ability to easily handle probabilistic weak labeling sources and correlated/adversarial sources. However, the paper lacks sufficient expository content to understand exactly why the proposed method outperforms baselines; the authors discuss a number of potential reasons, but do not back up these reasons with theoretical analyses or focused experiments. Regardless, the novelty of the method along with the improved empirical performance are strong enough to lead to my rating of marginally above the acceptance threshold. \n\n**Positives.**\n* The approach seems to handle strongly correlated and adversarial labeling functions significantly better than baseline approaches, and the structure of the correlation among sources does not need to be specified by the user a priori (unlike many existing tools, which require the user to specify the structure of the weak labeling sources). Additionally, the proposed method should be able to easily handle probabilistic weak labeling sources, which cannot be used as labeling sources in many existing weak supervision frameworks. Finally, even in non-adversarial settings with binary labeling functions, the proposed method seems to outperform existing weak supervision tools. These practical and empirical improvements to the weak supervision framework will likely benefit many users of weak supervision. \n* The proposed method is, to my knowledge, novel and departs from the approach of most existing weak supervision tools. The method is sufficiently described, code is supplied, and the code will be made public (according to the authors). The method is validated with experiments on standard weak supervision/crowdsourcing benchmarks with existing state-of-the-art weak supervision/crowdsourcing tools, with synthetic datasets (although the synthetic settings are quite extreme), and via ablations. \n* The paper is clear, well-written, and well-organized. Similarly, the code is commented and easy to follow. \n\n**Limitations.**\n* One of the advantages of using a PGM to model the weak labeling sources is that the PGM can be theoretically analyzed (e.g., to produce bounds on the generalization risk, understand conditions under which the model parameters can be recovered exactly, analyze how the weakly supervised performance scales with unlabeled data). These theoretical underpinnings have supported the use and understanding of weak supervision. However, the authors of this submission do not provide any theoretical support for understanding their method nor do they discuss which---if any---previous weak supervision analyses extend to their method. As a result, it’s difficult to understand exactly why the proposed method is outperforming the existing approaches. \n* Even if the authors do not present theory analyzing why their proposed method works, there are many analyses they could perform to dig into some of their intuition. For example, the following experiments would be useful to help the reader understand why the proposed approach leads to improved empirical performance:\n    * The authors claim that the improved empirical performance could be due to the fact that the label model parameters can adapt to each specific sample. A simple analysis here would be to look at the difference in thetas (the label model weighting parameters) for each sample in your dataset after convergence---how much are the weighting parameters actually changing? Does this correspond with relative improvement you see over baseline weak supervision methods?\n    * Similarly, the authors suggest their model may be performing better because the assumptions in the standard weak supervision models don't hold. It would be interesting to run an experiment on synthetic datasets analyzing the relative improvement of the proposed method as labeling sources become increasingly correlated, breaking independence assumptions in the existing models.\n    * Additional scaling experiments on the size of the weakly labeled dataset (comparing E2E, Snorkel, and Triplet) would be useful. It’s unclear to me if the proposed method will require more unlabeled data than baseline methods since more parameters need to be learned. The authors seem to be thinking along similar lines, as they hypothesize, “Since our model is based on a neural network, we hypothesize that the large relative lift in performance on the Amazon review dataset is due to it being the largest dataset size on which we evaluate on – we expect this lift to hold or become larger as the training set size increases.” However, they don’t do experiments to validate this claim. It would be relatively easy to use 10%, 20%, … 90%, 100% of the weakly labeled data with the proposed method and various baselines to generate intuition for how each method’s performance scales with more weakly labeled data. \n * As mentioned by the authors, training the label model and end model in tandem may hamper the labeling function development process. One advantage to the previous “two-phase” approach (where a graphical model is used to aggregate the labeling sources into probabilistic labels, then the weakly-labeled training data is used to train a downstream model) is its flexibility: labeling functions can be developed, iterated on, aggregated, and the resulting weak labels can be analyzed without having to wait for a deep model to train; then, the resulting labeled training set can be used with different loss functions, networks, etc. downstream. By combining the weak label aggregation process with the classification model, users are a more constrained and may not progress through this process as quickly or flexibly. However, the authors also acknowledge this as a limitation in their submission and suggest a reasonable workaround. \n* Some of the motivation for the proposed approach is a little tenuous. Specifically, in the Introduction, the authors set up that existing weak supervision models suffer from the drawback that “the separate PGM does not take the predictions of the end model into account.” However, it is not clear or intuitive to me *why* the label model should take into account the predictions of the end model. Additionally, the authors focus on the fact that “current approaches for estimating the unknown class label via a PGM need to rely on computationally expensive approximate sampling methods, estimation of the full inverse of the LFs covariance matrix…” suggesting that existing tools need to be more computationally efficient. However, the proposed method is to learn an MLP in tandem with a deep neural network from large datasets, which is not particularly computationally efficient. Additionally, there are no experiments on the relative efficiency of different weak supervision methods, so this motivation in the Introduction seems unrelated to the proposed solution. In my opinion, the motivation around modelling assumptions and model misspecification is stronger and probably sufficient. \n\n\n**Minor comments and clarifications**\n* In Table 1, I believe you are reporting the performance of a downstream model trained on the probabilistic labels produced by Snorkel, Triplet, Triplet-mean, and Majority vote (as opposed to reporting the label model performance). However, I had to look at your code to determine this---it wasn’t clear from the writing. If you are indeed reporting the end model performance in Table 1, that should be made more clear in the writing.\n* Line 195-198 was written in a confusing way---it’s not clear to me what dataset splits are used for which datasets, what data was unlabeled vs. labeled. Recommend rewording for clarity.\n* It would be interesting to look at the correlations of the probabilistic labels produced by the proposed method and baseline weak supervision methods---is the proposed method catching some particular slice of the input data that the baseline methods are not, or is the proposed method just uniformly better at predicting probabilistic labels? \n* It would be helpful to list the performance of each labeling function and each label model (even if only on a subset of the validation or test dataset) instead of only the final classification score.\n* Line 276---what is 2b? Do you mean Figure 2b?\n* Formatting: there is no space between paragraphs, which is useful for the reader to guide the eye---recommend adding if possible.\n* Some of the citation formatting seems to be off (e.g., [37])\n\n\n----------------------------------------------------------------------------------------------------------------------------------------------------------\nUPDATE AFTER REBUTTAL\n\nI have read through all other reviews and the authors rebuttal. I appreciate the authors thorough responses to my questions and the additional experimental results---the scaling experiments with different sizes of training data were interesting to look at! Adding a bit of related work about the benefits of joint optimization as previously investigated in crowd sourcing would be useful (if it isn’t already there). As brought up by the other reviewers and by the authors themselves, there is more work to be done understanding why this approach works. In my opinion, though, I think the novel approach and strong empirical results are interesting enough for publication at NeurIPS, and I am sticking with my original review score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Both were discussed appropriately. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "2YWY-xwc5ED", "reviewer": "Reviewer_DPB3", "summary": "This paper proposes a new method for weak supervision with multiple noisy labeling functions. The method jointly trains an encoder network that can estimate sample-by-sample accuracies with the final downstream model, and maximizes the agreement between the two models. The authors show improved performance on several weak supervision data sets and further show that their method is robust against both adversarial labeling functions and highly-correlated labeling functions.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper presents an interesting idea for an important problem space. The method has several benefits over previous weak supervision pipelines, in particular robustness against previous failure modes. The algorithm and paper are clearly-written, and ablations are extensive.\n\nThere are a few ways that the paper and presentation could be stronger:\n\n1. (high-order bit) I think the paper would benefit from further analysis (not necessarily theoretical) about why the method is providing lift, especially in the challenging adversarial settings. Section 6 has some paragraphs on \"why it works,\" but the text almost exclusively discusses degenerate solutions. This section would benefit from a more detailed analysis of which data points the method shows improvement on, and why. A few questions to ask to help get at this:\n* How accurate are the soft labels produced by the model (\"label model accuracy\")?\n* Which points do you get better soft labels for?\n* How did the labeling functions vote on those points?\n* Are there other points where the labeling function voted similarly, but you saw better soft labels *because* of the encoder network?\n\n2. (more minor) I feel that the explanation for why the method avoids degenerate solutions is a little unsatisfying. The authors write that they can avoid degenerate solutions either by early stopping or by setting the temperature parameter to a magic number (< 1/3). This is promising empirically, but leaves a pit in my stomach -- how do we know that we won't see degenerate solutions on another data set? Is there any stronger guarantee that can be made to avoid degenerate solutions? Can the method be adapted at all to detect degenerate solutions and avoid them?\n\n3. (more minor) I would be curious to see a comparison against (https://arxiv.org/pdf/2006.15168.pdf), which was a follow up to some of the baselines in the paper, and also used an encoder network to affect the soft labels from the label model. Their reported performance approaches the performance of E2E on the spouses data set (although they do not report numbers for the other data sets).\n\nNit: There is something wrong with the formatting of the paper -- it appears that there are almost no line breaks after paragraphs starting in section 3.4 onwards.\n\nIf these comments are addressed well (particularly 1), I think this paper could definitely move above the acceptance threshold!\n\nRating: 5", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Would appreciate moving discussion of societal impacts from appendix G into the body, given the extra page provided for it in NeurIPS this year", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a deep learning, end-to-end approach to learning from multiple noisy labeling functions. The proposed method outperforms existing techniques that use probabilistic graphical models for label aggregation. One of the major criticisms of the paper is that the paper does not analyze (theoretically or empirically) why exactly the proposed technique works well. The reviews believe that they have suggested different empirical investigations, but the authors did not engage with these ideas during the rebuttal period. The reviewers were willing to raise their rating if additional empirical investigation about the inner-workings of the technique had been conducted. Other that the lack of analysis, the reviewers do not present a strong criticism of the technique and are happy with empirical results. I do resonate with the authors that deep learning techniques can be challenging to analyze, and my overall reasoning is in agreement with reviewer #zED9 that the novelty of the proposed method along with the improved empirical performance can substitute the lack of theoretical or empirical justification to some extent. Overall, I believe the publication of this paper can help the research community make progress on finding better algorithms for aggregating multiple noisy labeling sources, and better understanding of this algorithm can be left for future work. Hence, I recommend acceptance as a poster.

Decision: Accept (Poster)