Understanding Programmatic Weak Supervision via Source-aware Influence Function
Authors: Jieyu Zhang
Haonan Wang
Cheng-Yu Hsieh
Alexander Ratner
Venue: NeurIPS 2022 Accept (2022)
Keywords: Data-centric Method
Programmatic Weak Supervision
Influence Function
Interpretability

Abstract: Programmatic Weak Supervision (PWS) aggregates the source votes of multiple weak supervision sources into probabilistic training labels, which are in turn used to train an end model. With its increasing popularity, it is critical to have some tool for users to understand the influence of each component (\eg, the source vote or training data) in the pipeline and interpret the end model behavior. To achieve this, we build on Influence Function (IF) and propose source-aware IF, which leverages the generation process of the probabilistic labels to decompose the end model's training objective and then calculate the influence associated with each (data, source, class) tuple. These primitive influence score can then be used to estimate the influence of individual component of PWS, such as source vote, supervision source, and training data. On datasets of diverse domains, we demonstrate multiple use cases: (1) interpreting incorrect predictions from multiple angles that reveals insights for debugging the PWS pipeline, (2) identifying mislabeling of sources with a gain of 9\%-37\% over baselines, and (3) improving the end model's generalization performance by removing harmful components in the training objective (13\%-24\% better than ordinary IF).

Understanding Programmatic Weak Supervision via Source-aware Influence Function
Jieyu Zhang'; Haonan Wang"; Cheng- Yu Hsieh', Alexander Ratner University of Washington University of Illinois Tirbana-hhappige "Snorkel AI, Inc. (jieyuz2, aydhsieh, ajratner (es washington. edu [haonan3]
Abstract
Programmatic Weak Supervision (PWS) aggregates the source votes of multiple weak supervision sources into probabilistic training labels, which are in turn used to train an end model. With its increasing popularity, it is critical to have some tool for users to understand the influence of each component (e.g., the source vote or training data) in the pipeline and interpret the end model behavior. To achieve this, we build on Influence Function (IF) and propose source-aware IF² which leverages the generation process of the probabilistic labels to decompose the end model's ''s training objective and then calculate the influence associated with each (data, source, class) tuple. These primitive influence score then be used to estimate the influence of individual component of PWS, such as source vote, supervision source, and training data. On datasets of diverse domains, demonstrate multiple use cases: (1) interpreting incorrect predictions from multiple angles that reveals insights for debugging the PWS pipeline, (2) identifying mislaheling of sources with a gain of 9%-37% over baselines, and (3) improving the end model's generalization performance by removing harmful components in the training objective (13% better than ordinary IF).
Introduction
One of the major bottlenecks for deploying modern machine learning models is the need for substantial amounts human-labeled training data, often annotated over long periods of time and at great expense. As machine learning models become increasingly powerful but also data hungry, new "data-centric" AI development workflows and systems have emerged, wherein the labeling and development of this training data is positioned and supported the central development activity. One recent and increasingly popular type of data-centric Al development uses Programmatic Weak Supervision (PWS), wherein users focus developing diversity of noisy, programmatic supervision sources [32, 33, 50, 46] to programmatically annotate training data an efficient way. Specifically, these weak supervision sources, e.g., heuristics, knowledge bases, and pre-trained models, are often abstracted as labeling functions (I.Fs) [33], which a user-defined program that provides potentially noisy labels for some subset of the data. So far, different modeling techniques have been developed to aggregate the noisy votes of LFs to produce training labels (often referred to label mode!) [33, 31, 14, 40]. Finally, these training labels are in used train an end model for the downstream classification tasks. In this study, we focus on this two-stage PWS pipeline [46].
With the increasing popularity of PWS in various applications [13, 36, 40, 19], it is of great importance to provide practitioners with a tool that helps them understand the behavior trained end model
"These authors contributed equally to this work.
"The implementation is avaiable at the WRENCH bewchmakkns.pes///ttttb.oo.////
wrench
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

as an effect of each upstream component (e.g., source vote, training data, LFs, etc.) involved in the model's training process. First, by understanding what are the deciding factors that lead to a model's 's specific prediction, users are able to verify whether such influence is desirable in terms of critical aspects such as model safety and fairness [11, 35, 16]. Secondly, since developing L.Fs is still inevitably a partly manual process in real-world applications and may involve human expert in an iterative process [5, 20, 51]. is beneficial to offer users with feedback of how individual component (like each LF) influences the end model performance on the downstream task. Finally, the rendered understanding (in)efficacy of the PWS pipeline can be naturally exploited to automatically improve the end model performance.
Towards the goal of developing tools to help understand the model behavior, Influence Function (IF) [22, 15, 8] has been recently applied to interpret the model prediction as influence of the training data for large-scale applications. However, most of existing IF studies are unaware of the generation process of training labels like PWS and only estimate the influence of the training data [22, 3]. Notably, the group IF study [23] has attempted to apply IF on PWS, but it simply estimates the influence of each LF as summation of data influence and is limited to a specified PWS without learnable label model. In this work, we aim to develop a general framework that helps estimate the influence of each PWS's component on the end model and is aware of (yet computationally agnostic to) the choice of label model. To achieve this, we leverage the knowledge of how the probabilistic labels are synthesized from sources and accordingly decompose the training loss of each data into multiple terms, each of which associated with what we called (i. j. c) -e-ffect, i.e. the effect of j-th LF on c-th dimension (corresponding to class c) of the i-th data's ' probabilistic label. With such a training loss decomposition, we propose two means of calculating our source-aware IF, namely, reweighting and weight-moving based on different perturbations Each estimated source-award influence score can be interpreted as the influence of removing the c)-ffect from the pipeline. By simply aggregating over specific dimension(s), it can be utilized to estimate the influence of different components of PWS including (1) each source vote, (2) each LF and (3) each training data, since the influence score is additive [23].
the training respectively.
In experiments, we exploit the cource-aware IF in a diversity of 13 classification datasets including tabular, text, and image data, and present multiple applications. First, we show that cource-awart IF can help to understand the behavior of the end model in terms of the PWS components; in particular, when interpreting the same incorrect prediction made by two PWS pipelines with different label models, data-level IF might lead to the same influential training data, whereas source-aware IF is source-aware IF as
capable of revealing the most responsible LF source vote. Second, a tool to identify mislabeling of each LF and show that it outperforms baselines with a significant margin of 9%-37%. Finally, demonstrate that the training loss decomposition and source-aware IF enable fine-grained training loss perturbation and consequently lead to better test loss improvement 133%-2%% improvement over ordinary IF [22] and group IF [23]..
To summarize, this work makes the following contributions. First, we propose owarce-wware IF tailored for PWS that incorporates Second, we develop two means of calculating the source-aware IF based on different training loss perturbations. Third, we demonstrate variety of use cases of the source-aware IF and conduct extensive quantitative and qualitative experiments on real datasets in different domains to verify the efficacy of source-aware IF on understanding and improving PWS pipelines.
knowledge of the generation process of training labels.
2 Preliminary
We denote scalars and generic items as lowercase letters, vectors as lowercase bold letters, and matrices as bold uppercase letters. We use superscripts to index generic items, e.g., the i-th data x(1), and use subscripts to index certain dimensions of vector, e.g., the i-th component of a vector vi, or the vector output of a function f(-), All derivation details are deferred to the appendix.
Programmatic Weak Supervision. We target at a C-way classification problem and have a training dataset of size N. i.e., D (2 ( Je(N). where x(( E X Rd and y(9) E y = [C]. The ground truth (Ss i are not observed. We have M labeling functions (LFs) (X0()))s(())- Each ())()) E {-1) U y00) where goo) C y. That is, each (1) either abstains (-1) or outputs specific label y pou on given data point. We define the label matrix as Lig (z)
2

---

and with a slight abuse of notation, use L(x(0) to represent the row of L corresponding to all of the labels assigned to z(0), The goal of Programmatic Weak Supervision (PWS) is to train an end classifier he X + y with {x(*) HEIN and L only, where ha)) = arg maxce(C] (a(x) e and fo(z) E RC is vector-valued function parameterized by 0. A typical PWS pipeline involves label model gw (L(z)) parametrized by W, which aims to produce a weak probabilistic label (xx()) - gw (L((1)) E A for each data point x(() (denoted as y(t) for simplicity). The label model produces a dataset D = (x(*), (,(t),ye
y(()) JEENN]. which is then used to train fo via the noise-aware
(0) loss [33] ( )o(x())) yn (e(y, fo(x()) = E yo ((c, (o fo(())) where e(x,-) is the cross-entropy loss, leading to the following training objective:
8- = arg L(D; 0) = arg min
Σ E )o(x())) = arg min
ΣΣ
y log (c)) (1)
where 0* is the minimizer of the loss Ł(D;0) over the training set D.
Influence Function. The Influence Function (IF) estimate how the minimizer H of the empirical
loss& & would change if we were reweight the i-th training example 2 = (xx(t, .(( by 6,. The key idea to make a first-order Taylor approximation of the changes 6* around 6, O. Specifically,
the i-th training sample is upweighted by a
small e;, the perturbed risk minimizer oz, becomes:
N
arg min BE
Σ V=1
Ev"""",,
cs ( (o()))))))))
(2)
N
The changes of the model parameters due to the introduction of the weight et are:
02 0"
d0:
e, Hy- 11..'Ve Vol (v for (z(
(3)
de: 1:=00
where Hg. N Ex Vie (1)), positive definite [22]. Similarly, the change of loss with respect to single sample z' (x', is
, fo. (zo())) is the Hessian of the objective at l and assumed to be
(x'=)) = d (e(v' for, (x)) elv' CoV.oo-Ce (o-(xx)) de de', for, (z'))
de:
de;
(4)
= -v (v' fo- (x') Vol'' fo- (x))).
Then, as shown in previous work [24], the change of loss over holdout set, egg, the validation set Dv, is simply the sum of the changes over every sample in the set:
d (D.) C(D,; - L(D::
D
Σ 0x(2').
(5)
PCDD
The change of loss w.r.t. one sample is often interpreted as its effect on model predictions over the training set [22], while the change of loss over a holdout set can be leveraged to improve the test loss by discarting/dowwweighingg the training examples with negative impact [38, 42, 24]. Considering the extraordinary influence that might be caused by mislabeled examples, the relative influence function (RelatFF) [3] has been proposed to measure the influence of a sample relative to its global
effects. The RelatIF can be defined as (x'²') = V Vor(z(0))
dx(x')
Methodology
Given an unlabeled training set (ax(+) JEIN and M labeling functions (LFs) (-))(-))e((M)) we study the scenario where a Programmatic Weak Supervision (PWS) pipeline is employed to synthesize probabilistic labels (g) ]E[[)] for training end model fo (z) for C-way classification problem. We aim for general framework for explaining and debugging the PWS pipeline through Influence Function (IF) [22], which estimates the influence of each training data on test loss/prediction for better understanding of model behavior and is able to improve/debug the training data accordingly. In the PWS pipeline, we have full knowledge of how training labels are synthesized from different sources, i.e., LFs, which opens the opportunity to trace the influence in a source-aware way. this section, we propose the source-aware IF tailored for PWS and describe its applications.
3

---

N
-1
100.23
Sum over columns
72.81
Normalization
0.30
M
A
130.5
80.78
0123
0.17
0.12
0.18
(1,),c) effect, where L,j M,
C
Probabilssti y
Label Model W
Figure Illustrations the generation process the probabilistic training label and the (1, e)- effect. We demonstrate how the probabilistic training label for the first data point is generated from the label matrix (source votes, other words) and the label model.
3.1 Training loss decomposition
By examining the popular choices label models gw( (-) [9, 33, 31], we show that their parameters
be unifiedly represented as a tensor W E MBMxx(C+)xCC where the middle (C + 1) dimension
size is due to additional "abstention" label LFs output. The generation process of the probabilistic labels can then be expressed as
OE W Wyngee)
Ve E [C],
(
(6)
Σ C5L10E1 W...y.))
where a(.) could be (1) the identity function (z) for the classic Majority Voting label model
or its variants, (2) the exponential function Geup (x) = exp (x) to recover the softmax function used in statistical models such as Bawid-Skene model [9] and Snorkel MeTal. [31]. Intuitively, label
models aim to learn source-specific weight for each LF to aggregate the votes in an unsupervised manner. Details can be found in Appendix A. In both cases, for the i-th data, a single parameter Wy controls the effect of the j-th LF's vote Lij on e-th dimension of probabilistic label, i... ye For convenience, corresponds to the i-th data, the j-th LF and e-th dimension of the aggregated probabilistic label.
(5) denote the effect Lij on ye the (i.j. c) -effect. as it uniquely
For a better understanding, we visualize how a probabilistic training label generated and the c)-fffcct in Figure 1.
With such unified formulation of probabilistic label generation, we rewrite the noise-aware loss [33]
C ; Walnye) log (fa (z)
( = Σ
(z)
2 o O W...y.k)
We proceed with the (-)) being the identity function because it enables the loss decomposition introduced below. We will later discuss other choices of (-). By simple rearrangement, we could decompose the loss over the training set into the summation of all N M x C' terms:
M L(D, = ΣΣΣ (1),
N
W W.L.e
log (2) )c)-
(8)
Σ
W,
Such a decomposition source-aware in the sense that it preserves how the source votes are aggregated via the label model, and each (0) indeed corresponds to the (i. cefffeect.
3.2 Source-aware Influence Function
As a consequence of training loss decomposition, we are now able to compute the influence score for each term (0), which is the influence of removing (...,(()) (0) or, in other words, the (i, c)-ffeect from training. We provide two means of calculating the influence score: reweighting and weight- moving, which correspond to two different ways of perturbing the training loss. Specifically, given

---

an (unseen) instance 2' = (a', and the cross-entropy loss & ( fo())), we are interested in the influence score defined by the change of loss & (1y, (g((()) caused by the differences in the model parameters after perturbing the training loss L(D; 0). We first give the formulation an ordinary IF for a training data 200) = (z(*), 5(0) in the context of PWS, which is smilar to Eq. 4:
.x(x') Vol (V', for (x)) Hg. Vol(s for )..
Here, Hg. 1 E Villy!o fg. (z) and 0* the minimizer of the training loss before the perturbation. We reuse the notation of Hg. and 6* in this section as their meanings are unchanged.
N
Reweighting. To study the influence of removing the (i, j. e)-efeet we reweight (0))a(() such that
M
(D: 0)
1
Resee + (.s.chs...0))..
(10)
N
/==1'=1c'=z
Then, the influence score, i.e., the difference of loss on = (x', 3²) caused by reweighting the loss term (0), is,
Eo (x'))² Vol
(11)
In contrast to ordinary IF that reweights the loss w.r.t individual training data, the proposed one is finer-grained since reweights the decomposed loss terms. By reweighting, the resultant probabilistic training label is no longer sum-to-one and the solution (Eq. 11) is only for identity a.) function.
Weight-moving. We also explore an alternative way of perturbing the training loss called weight- moving. Different from reweighting, its computation is agnostic to the ar-) function and the proba- bilistic training label is still sum-to-one. Specifically, we remove the (i. c) -)-ffeect in the probabilistic training label (+) and renormalize it, leading to new probabilistic training label:
Wc E |C},
8- EC 1
(12)
Then, perturb the training loss by moving the weight of the original loss on the i-th data ((3(), (0)) to a new loss term "( fo(x:
1 Bum (D:0) = N Σ (()))) foxxx)))))))) + eig.c (6(()))()'))) esge E(y yo (oxx"))
(13)
When Gig.C = N. the loss term fyy(), fo(x(t)) indeed replaced by the new loss i(y f'cs fo((()))))))) We again provide the formulation of corresponding influence score:
Twm (2') Vol (y', fo (a')) Holi - -e
for (x))..
(14)
Discussion. Note that the change of loss over a holdout set D. can be similarly computed and we denote it as ohy (D,)) and @ (D.). We omit the input variable and the superscript and use db.jcc to represent the proposed source-awart IF in general. The Φi,j.c corresponds to the influence of a specific perturbation on the training loss: setting the label model parameter W j.L.y.e to zero for the i-the data so that the (i, j, ceffffect is removed from training. Then, the difference between reweighting and weight-moving reduces to whether renomarlization on the probabilistic labels is performed after the perturbation.
Extension to RelatIF [3]. We additionally demonstrate how to compute the RelatIF [3] in our source-aware IF setting, Specifically, the self-influend is Pi.j. Vols,,e((*) H.g. (o"1,,.(0").
self
Then, the source-aware RelatIF for both T.w. and T are Lis and
respectively. Pighe
3.3
Use cases
Case estimating influence different components. One major benefit of our training loss decomposition and the identity ((-) function is that we could readily reuse the source-aware IF di.e.
5

---

Table I: Calculation influence of each component PWS using the source-aware IF $s.j.e-
Component of PWS pipeline
Symbol
Calculation
200; the i-th data
Pgta
Degic
X j-th LF
Ee=1 Pisie
Ly: the j-th LF's output on the i-th data
Oly
Pasie
W j.k.e individual parameter of label model indexed by (1, k.
Dizl Phiiell k}
estimate different components of the PWS pipeline, e.g., individual data or LF. This is because according to the study of group influence [23], when measuring the change in test prediction/loss is the sum of influences of its constituent points.
the influence is additive, i.e., the influence However, such a nice property only holds true for the reweighting method, while, in theory. the influence calculated by weight-moving does not follow due to the fact that they are not additive. We list the influence of different components of a PWS pipeline as well as how to calculate them using the fine-grain IF $...e e in Table 1. Note that the and 2nd row of Table indeed recover the ordinary IF of a training data [22] and a LF [23], respectively.
Case 2: improving test loss. Another application IF is to improve the test loss by identifying and then discarting/downweiihinng the harmful training data [24, 42, 38]. Similarly, the source-aware IF $i.j.c can be employed to improve the test loss by discarding/dowwweighhi with negative impact. Specifically, we have the following theorem for reweighting (a similar result for weight-moving can be found in Appendix E.2):
the loss term f.
Theorem 1. Discarding or downweighting the loss terms in S_ (:),, E [N].] E [M], [C], 5rw ( > 0} from training could lead to model with lower loss over holdout set D.:
Σ
(D.:) 50
where 0's_ is the optimal model parameters obtained after the perturbation.
In practice,
the validation set D. and (a) (E1z [C], (fj.e (D,) > a} to tolerate the estimation еπгог of IF where the hyperparameter controls the proportion of items to be perturbed [24]. Note that this method improving test loss relies on the training loss decomposition and (-) being the identity function.
[M], [M].
3.4 Beyond the identity function
The aforementioned use cases rely the additivity of influence score and the training loss decompo- sition, which both consequences of (-() being the identity function. When a(-) is the exponential function, we could approximate the label model (-)-) by new label model (-) with parameter W of the same shape but using identity function instead. Obviously, want the new label model (w(:) to reproduce the same probabilistic labels as those generated by gw(.). This motivates us to solve the following optimization problem for W:
exp() j=l W W.L 1 exp E Wy.ngs))
Eje W Walggi Ee En W Wylnigk
W
min W
)
(15)
Then, we can replace the label model gw()) with Bw- everywhere. As consequence, we are now able to unify various types of label models in a single framework and enjoy the convenience and advantages brought by the linearity of the identity function. Note that for the original label model gw(+) with ar (f.) being the exponential function, we can still derive the influence di.j. (see appendix). We study the effect of replacing original label model with the approximated one on the end model performance in the experiment section.

---

4 Experiment
4.1 Setup
Datasets. We include the following four classification datasets in WRENCH [50], collection of benchmarking datasets for PWS: Census, Youtube, Yelp, and IMDb. Note that the Census dataset is tabular dataset while the others are textual datasets. We use the labeling functions (LFs) released by WRENCH [50]. We also include the following tabular datasets: Mushroom [12], Spambase [12), and Websites
PhishingWebsites (PW) [29], for which follow the instruction in the WRENCH [50] codebase?
to generate LFs from a decision tree learned on the labeled data. Finally, we follow [28] to derive LFs for multiclass image classification task using the DomainNet [30] dataset, which contains 345 classes of images in 6 different domains: real, sketch, quickdraw, painting, infograph and clipart. We construct a classification task for each domain with the classes containing the largest number of instances. We then train classifiers using the selected classes within the remaining five domains as LFs. We name the 6 resultant datasets DN-real, DN-sketch, DN-quickdraw, DN-painting. DN-infograph, and DN-clipart for each domain, respectively. Following [50], we use a pre-trained BERT model [10] to extract features for textual data and use ResNet-18 [17] pre-trained on ImageNet for image data.
Model Training and Evaluation. For label model, we focus on three commonly-used choices: Majority Voting (MV), Dawid-Skene model (DS) [9], and Snorkel [31]. The at-) function of the latter two label models is the exponential function and we use their approximated variants (as mentioned in Section 3.4) for all experiments. Following [23], we use logistic regression for the end model and defer the experiments on using neural networks to the appendix. We train the end model initialized as zero by gradient descent for 10,000 epochs with learning rate being 0.001, and then calculate the influence score on validation set D. throughout the experiments. When using the test loss as evaluation metric, we use D, to do model selection and report the test loss of the select model.
For simplicity, we use IF (R-IF) to represent ordinary IF (ReladFF baseline; RW (R-RW) and WM (R-WM) are the proposed source-aware IF (RelatFF) calculated by the reweighting and weight- moving method respectively. We also use the term SIF (R-SIF) to represent the source-aware IF (RelatIF) when we do not care whether it calculated by reweighting or weight- weight-moving. Additional experiments and details are in Appendix G.
4.2 Experimental results
Understand model behavior from multiple angles. First and foremost, source- aware IF can be used to understand the end model behavior from multiple angles, while ordinary IF can only identify influential training data. Here, we take the DN-real dataset as an example and use reweighting- -bsed source-aware IF (RW) to diagnose misclassified test data, For a single misclassified test data of the end model, we present the most "responsible" (1) training data, (2) L.,, and (3) labeling (the vote of a LF on a training data), where being the most "responsible" means having the highest influence score on that data and likely being the primary cause of the misclassification. In Table 2, we show that for the same image misclassified by the end model trained with different label models, we could have totally different explanations and components of the PWS pipeline to blame on. Specifically, for "dog" image misclassified as "golf club", we can see that for both the DS and Snorkel label models, the most "responsible" training image the same but the most "responsible" LF and labeling different: for DS the LF trained with images of painting domain is likely the primary cause for the misclassification, while for Snorkel Without source-aware IF, we can hardly distinguish the cause of misclassification for DS and Snorkel based solely on the training data as the most "responsible" one is identical.
might blame more on the LF trained with clipart images.
Identify mislabeling of LFs. One unique advantage of source-aware IF over ordinary IF is that it allows us to trace the influence of each LF's labeling a specific training data (dh.. = =>' Pi.j.s the 3rd row in Table 1), which is helpful in understanding the expertise of each individual LF and thus provides clues for practitioners debug LFs, while ordinary IF can only estimate the influence of training data as a whole. We use σr., as scoring function of how likely the j-th LF mislabels
Phttps: htttss///.taaaaaccoe//e///////ts
data
7

---

Table 2: The most "responsible" data, LF, and labeling, identified by source-aware IF, for the misclassification of end models (trained with difference label models) on the same test image. We can see that source-aware IF can reveal the cause of misclassification from multiple angles.
Misclassified
Data
Resp Training Data
Resp. Labeling
Label Model
Resp, LF
Data
Pred. (Prob.)
Data
True Label
Weak Label (Proh.)
1.
Data
Vote
MY
golf club (0:55)
dog
goll club (1.0)
quickklra
quickkdraw
goll club
DS
golf club (0.44)
dog
golf club (0.71)
painting
painting
golf club
Smorkel
golf club (0.56)
dog
golf club (0.94)
clipart
clipart
golf club
Table 3: Performance comparison results on identifying mislabeling of LFs. We report the average precision (AP) score averaged over LFs for each dataset. The larger the AP is, the better the method identifies mislabeling of LFs.
MY
DS
Soockel
Dataset
KNN
R-HW
WM
1.M
R-WM
1/M
R:WW
WM
Canote
0510
0.809
0.167
0.855
0.868
0.854
0.41
0767
0.767
0,746
0.791
0.706
0.799
007867
0.767
0.900
0.800
0.801
0,313
Miskroon
0.95|
0.923
0.828
0,952
0.923
0.948
0.949
0.328
0,838
0.900
0.755
0.861
0.865
0.828
0.825
0.897
0,857
0.874
0.876
D622
(1663
0.76
0,591
0911
0871
0.766
0.76
0.818
0.745
10766
0.766
01889
0.757
0379
Spantoss
0.771
0.753
0.916
16927
0.878
0187
0.78
0.318
0.7M6
0.975
00.16
0.718
0.816
0.551
00833
0,750
IMD6
0,002
0.767
0.600
0,740
0.799
0.754
0.769
0,609
0,699
0.765
0.725
0.764
0,581
0.699
0.699
0.737
0,713
0.725
0.711
D152
0.792
0.731
0.554
0,933
0.500
0.731
0.71
0.764
0.724
0.731
0.731
0.007
0,500
0.785
Yamthe
BAS1
R,999
01836
0.661
01911
0.851
01899
0.426
0.826
0.817
0.60
0.837
11816
0.91
6,983
01894
0,389
DN-rral
0.11
0.447
0.417
0.959
11086
0.904
0.942
0417
0,417
0.550
11462
0.350
D.440
00:445
0.417
01884
0.671
0.723
0.476
DN akech
0.321
0.339
0.316
0.764
0,83
0,509
0.715
0.316
0.316
0.506
0.515
0,460
0.422
00331
0,316
0,497
0.515
0.455
0.433
DN-uicldrw
0,255
0.540
01879
0.723
01702
0.255
0,255
0.301
00255
00:35
UAS5
0,370
01:33
DN-gsining
[444
(1416
0.760
0.647
11912
0.756
0187
0.7M0
0.7M0
0.614
(1557
0.314
D.444
0.3550
0.360
017155
0,50)
0.630
0.490
DNvinfogrph
01981
0.156
0.6%
11714
0.964
0.644
0.356
0.116
0.516
01517
0.465
0.478
0.316
0.316
0.914
0.526
0.464
DN-clpan
0.606
0.784
04454
0.500
0
0.670
i-th data, i.e., Lij # y()), to identify the mislabeling of individual LF. We thus formulate it as a binary classification task and report the average precision (AP), standard evaluation metric of a scoring function. We use the discrepancy between j-th LF's labeling on i-th data (Lş) and the probabilistic label output by (1) label model (LM), (2) end model (EM), and (3) a K-nearest neighbor classifier (K = 10) trained on validation set (KNN) as baseline scoring functions. Specifically, we calculated the discrepancy as one minus the dot product of Li 's one-hot representation and the probabilistic label; for example, if Lij 's hot representation
[o, 1, 0J and the probabilistic 1 x 0.4 = 0.6. Note that KNN is a more fair and
label is [0.5, 0.4, 0.3) T, the discrepancy competitive baseline because it directly leverages the validation set and, as our source-aware IF, does not introduce any parametrized model in addition to what PWS has. We present the results in Table 3; we can see the superiority of source- nuure-aware IF in this task since it outperformed baselines in most cases and the averaged gains are about 9%-37% for different label models. Intriguingly, when LM and EM render unsatisfactory performance, e.g., on DomainNet, which indicates the quality of the label or end model is not good, the source-aware IF still performed well. This means it less sensitive to the quality of PWS, which is critical for debugging the pipeline.
Improve the test loss of the end model. We compared the proposed source-aware IF as well as its RelatIF variant to ordinary IF/RelatIF in terms of the capability of improving the test loss. For source-aware IF/RelatIF, we discarded loss terms in S_ (a) and then re-trained the end model (as described in Section 3.3), while for IF/RelatIF, we likewise discarded training data in (a) = {2 E (D.)) > a). For these methods, we tuned the hyperparameter a on validation set for best validation loss. We additionally included the group influence (GIF) [23] that estimates the influence of each LF and is only applicable to MV as in [23]; we discarded ie L.s with highest negative impact and tune k on validation set. The results are in Table 4, where ERM (Emperical Risk Minimization) the ordinary training without any perturbation. From the results, we conclude that although ordinary IF/RelatIF can already improve the test loss over ordinary training (ERM) by large margin (>0.1 test loss on average), the source-aware IF/RelatIF can further boost the performance by a similar margin ( averaged test loss improvement over IF/RelatF). This shows
8

---

Table 4: Performance comparison results the test loss of end models.
Datant
13M
18
B-4
RW
R-RW
WM
RAWM
EBM
1F
B-8
BBW
R-WW
B-MM
EBM
10344
BB-93
Cresss
11:31
0.394
0.33
03333
0.90
0.363
0.57
01518
0.383
0:90
11:85
Madhoon
1.338
0.195
0.302
0.344
0.354
5.16
11.02
0.388
0.30
0.270
0.29
0.276
0.215
0.333
0.390
0.341
0.198
B.186
FW
0.416
0.31
0333
0.36
935
1194
0.99
0.738
0.338
0.333
0.388
0.391
0.311
0.666
0.990
0.951
0.906
0.
0:347
0L365
IM
16334
(1901
0.992
0.801
0.694
1,88
11.09
0,482
01-40
15.04
01014
01314
9.67
10620
11029
04577
(6520
0.261
1.57
(L571
Vip
10533
0.96
0.635
0.503
0.741
0:411
0.130
0.130
0.905
0.967
0.965
0.307
0.57
01150
01990
0.516
(1991
0.307
0.365
D-506
0190
13990
0A27
0.281
0.713
0.392
6.117
11,02
0.123
0.966
0.336
01344
0.127
(4.00
0.135
0,878
0.396
04100
0.336
B1131
0,003
UN-kkee
1:230
1.382
1.664
0.385
6,16
0.827
0.841
1,590
1,803
1,390
0,990
0,569
1,300
1,300
1/000
1,076
00115
0,519
0015
0,900
TIN pairing
11:97
1996
0.965
0.857
0.114
9.06
01:87
0.97
1.79
01704
01891
01.69
0.41
0.971
1119
11900
0.155
0.385
BS51
0,W0
UN Un-eooo
1:230
1.39hh
1.391
1.373
1.385
1.46
1.347
1.34
1,30
1.307
1,371
1.294
1.305
1.3%6
1.24
1,200
1.304
1.309
1,211
1.46
1226
1165
Avg.
0.779
0,60
0.72
0.863
0.58
6311
0.544
0.395
0.755
0,643
0.53
0,965
0.500
0,985
0.501
0.000
0453
0.990
0,905
0.49
0,509
0,34
the benefit of employing IF in a source-aware manner and provides insights for practitioners to improve the PWS pipeline.
Discussion. One may develop sophisticated methods to identify mislabeling of LFs or to improve test loss and achieve better performance than source-aware IF, but source- aware IF is designed as tool of interpreting the end model prediction and when used in these applications, it does not introduce additional parameterized model. Also, we believe that source-aware IF can inspire ad methods in the future and be leveraged by them to further boost the performance on the task of identifying mislabeling of LFs or improving test loss.
4.3 Additional studies
Estimate ordinary data-level IF via source-aware IF. As described in Section 3.3, we can use the source-aware IF calculated by the reweighting method to estimate the data-level influence score (02(5) = -E Ee Li.j.c the 1st row in Table 1), which coincides with the influence computed by ordinary IF in theory. Thus, we are curious about how well the source-aware IF can be used to estimate the ordinary data-level IF. Because IF is known to be noisy in the value but good in their ranking [4. 23], we instead study the ranking correlation between 2aco and the ordinary IF. In Table we report the Spearman's 's ranking correlation coefficient (S 1), where larger coefficients indicate better estimation of the ordinary IF based on 620)) in terms of the ranking. From the results, we can see that the out (1) actually preserves the ranking of ordinary IF quite well as Spearman's 's coefficients are from 0.71 to 0.99, which further demonstrates the efficacy of source-awand IF
Table 5: Spearman's 's ranking correlation coefficient (5 1) between influence calculated by ordinary data-level IF and that estimated from source- aware IF, All results pass the significance test (p < 0.05).
Dattwett
Cenass
Mastrvom
IW
Spambuse
IMDs
Yelp
Yaulube
DN-nea
DN-qukkdraw BN-4ketch
DN-painning
DN-indsgraph
DN-
Ang.
0.981
0.955
0.906
0.877
0.780
DS
0.590
0.975
0.590
0.992
0.904
0.640
0.709
0,893
6.77
0,867
0,854
0.730
0823
0.61
Snorket
0.509
0.43
01984
0.957
0875
1.999
B776
0,991
0.766
B876
0.861
0.11
0.35
0.71
Avg. best loss over all datasets
Efficacy of label model approximation. Since we bypassed the non-linearity of exponential function in DS and Snorkel with their approximated variants (Section 3.4), is natural to ask how well the approximated label model can perform and whether or not the approximation is well-haved. To answer these questions, in Figure 2 we show that although the aver- aged test loss of ordinary training (ERM) and best of IF/R-IF (Best IF/R-IF) are slightly better than their counterpart of approximated label model (Approx. ERM and Approx. Best IF/R-IF), the best averaged test loss obtained by SIF/R-SIF with the approximated label model (Approx. Best SIF/R-SIF) is much lower. Such an observation indicates that with the goal of improving test loss, it is beneficial to choose (-) to be the
Aporn thM
D5
Storkel
model
Figure 2: Avg. test loss of the end model with/without label model ap- proximation.
identity function so that the training loss can be decomposed and perturbed based on source-aware IF.

---

Related work
Programmatic Weak Supervision. the Programmatic Weak Supervision (PWS) literature [33,
46], popular choices of labeling functions (LFs) include user-crafted heuristics [32, 39, 45], pretrained models [2, 28), external knowledge bases [18, 26], and crowd-sourced labels [9, 25J. Recently, researchers have developed various label models [33, 31, 14, 40, 48, 44, 37, 47, 47, 49, 43]. In the PWS pipeline, the behavior of the end model is directly dictated by the soft labels, wherein the soft labels themselves
also intermediate products of the label model and the LFs' votes. Thus, to interpret the trained end model and potentially debug the PWS workflow, is critical to understand the influence of each PWS component the end model's performance, which has yet received adequate research attention.
Inftuence Function. Among various ways to understand a model's behavior through the training data points, Influence Function (IF) is an effective technique with a wide range of applications [22, 23, 16. 7J. In PWS, although one can directly apply to understand model's behavior through the weakly labeled training data, more insights may be unveiled by further investigations of how each PWS component, e.g., LFs and label models, influences the end model. In similar spirit to us, [6] developed multi-stage IF to trace fine-tuned model's behavior back to the pretraining data. Perhaps most similarly to our work, [23] conducted initial exploration on using group influence to study how removing LF from the PWS pipeline would affect the end model's s performance. However, the group influence method is only applicable to a specific simplified setting of PWS.
6 Limitation and social impacts
Limitations. Although we study several popular label models and show how to use approximated label model in case that users need to use a label model that not included in this work, user may still encounter difficulties in real applications when using complicated label model. Also, this work focuses on the two-stage PWS pipeline. while very recently there are some one-stage methods, to which our framework may not be applicable. In the main body of the paper, we use logistic regression as end model in our experiments, while in appendix we present experimental results of a simple two-layer neural network. But we do not include experiments on complicated deep learning models like Convolutional Neural Network.
Social impacts. This work aims help people understanding the Programmatic Weak Supervision and has the potential to resolve the bias in the generation process of training label, which might have positive social impact. We do not foresee any form of negative social impact induced by our work.
Conclusion
In this work, by leveraging the knowledge how probabilistic training labels are aggregated from different sources in the PWS pipeline, we have proposed source-aware IF that enables users to estimate the influence of different components (e.g., source vote, training data, L.Fs, etc.. in the pipeline on the end model's 'ss behavior, and unlocked various practical use cases. Specifically, we have demonstrated how the proposed source-aware IF can be used to (1) unveil the most influential training data point, LF, specific vote to misclassification made by model, (2) identify mislabeling of LFs on certain data points, and (3) improve the end model's test performance.
8 Acknowledgement
Thanks the anonymous reviewers for their helpful comments and suggestions. We also thank Haojie Jia for his help of experiments.
References
[1] Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization in linear time. stat, 1550:15, 2016.
10

---

[2] Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cassandra Xia, Souvik Sen, Alex Ratner, Braden Hancock. Houman Alborzi, al. Snorkel drybell: case study in deploying weak supervision at industrial scale. In SIGMOD (Industrial), pages 362-375, 2019.
[3] Elnaz Barshan, Mare-EEienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In International Conference Artificial Intelligence and Statistics, pages 1899-1909. PMLR, 2020.
[4] Samyadeep Basu, Philip Pope. and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint arXiv:2006.1440 14651, 2020.
[5] Benedikt Boecking, Willie Neiswanger, Eric Xing, and Artur Dubrawski. Interactive weak supervision: Learning useful heuristics for data labeling. In ICLR, 2021.
[6] Hongge Chen, Si Si, Yang Li, Ciprian Chelba, Sanjiv Kumar, Duane Boning, and Cho-Jui Hsich. Multi-stage influence function. Advances in Neural Information Processing Systems, 33:12732-12742, 2020.
[7] Gilad Cohen, Guillermo Sapiro, and Raja Giryes. Detecting adversarial samples using influence functions and nearest neighbors. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14441-14450, 2020.
[8] Dennis Cook and Sanford Weisberg. Characterizations of an empirical influence function for detecting influential cases in regression. Teehnometrics, 22(4):495-508, 1980.
[9] A. P. Dawid and A. M. Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Journal of the Royal Statistical Society, 28(1):20-28, 1979.
[10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL-HILT, pages 4171-4186, 2019.
[11] Finale Doshi-Velez and Been Kim. Towards a rigorous science of interpretable machine learning. arXiv preprini (xXi::/702.0608 2017.
[12] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017.
[13] Jason Frics, Sen Wu, Alex Ratner, and Christopher Ré. Swellshark: A generative model for biomedical named entity recognition without labeled data. arXiv preprint (0X.xv1174.063600 2017.
[14] Daniel Y. Fu, Mayee Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christo- pher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In ICML, pages 3280-3291, 2020.
[15] Frank Hampel, Elvezio M Ronchetti, Peter J Rousseeuw, and Werner A Stahel. Robust statistics: the approach based influence functions, volume 196. John Wiley & Sons, 2011.
[16] Xiaochuang Han and Yulia Tsvetkov. Fortifying toxic speech detectors against veiled toxicity. arXiv preprini urXiv:2010.03154 2020.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[18] R. Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S. Weld. Knowledge- based weak supervision for information extraction of overlapping relations. In ACL, 2011.
[19] Sarah Hooper, Michael Wornow, Ying Hang Seah, Peter Kellman, Hui Xue, Frederic Sala, Curtis Langlotz, and Christopher Re. Cut out the annotator, keep the cutout: better segmentation with weak supervision. In ICLR, 2020.
[20] Cheng- Yu Hsieh, Jieyu Zhang, and Alexander Ratner. Nemo: Guiding and contextualizing weak supervision for interactive data programming. arXiv preprint 2203.01 2022.
11

---

[21] Giannis Karamanolakis, Subhabrata Mukherjee, Guoqing Zheng, and Ahmed Hassan Awadallah. Self-training with weak supervision. In NAACL-HLT, pages 845-863, 2021.
[22] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894, PMLR, 2017.
[23] Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Perey S Liang. On the accuracy of influence functions for measuring group effects. Advances in neural information processing systems, 32, 2019.
[24] Shuming Kong, Yanyan Shen, and Linpeng Huang. Resolving training biases via influence- based data relabeling, In International Conference on Learning Representations, 2022.
[25] Ouyu Lan, Xiao Huang, Bill Yuchen Lin, He Jiang, Liyuan Liu, and Xiang Ren. Learning to contextually aggregate multi-source supervision for sequence labeling. In ACI. pages 2134-2148 2020.
[26] Chen Liang, Yue Yu. Haoming Jiang, Siawpeng Er, Ruijia Wang, Tuo Zhao, and Chao Zhang. Bond: Bert-assisted open-domain named entity recognition with distant supervision. In KDD, pages 1054-1064, 2020.
[27] James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approx- imate curvature. In International conference on machine learning, pages 2408-2417, PMLR, 2015.
[28] A. Mazzetto, C. Cousins, D. Sam, S. H. Bach, and E. Upfal. Adversarial multiclass learning under weak supervision with performance guarantees. In /CML, 2021.
[29] Rami M. Mohammad, Fadi Thabtah, and Lee McCluskey, An assessment of features related to phishing websites using an automated technique. In 2012 International Conference for Internet Technology and Secured Transactions, pages 492 497, 2012.
[30] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1406-1415, 2019.
[31] A. J. Ratner, B. Hancock, Dunnmon, Sala, S. Pandey, and C. Ré. Training complex models with multi-task weak supervision. In AAAI, pages 4763-4771 2019.
[32] Alexander J Ratner, Stephen H Bach, Henry Ehrenberg. Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In VLDB, volume 11, page 269, 2017.
[33] Alexander Ratner, Christopher M De Sa, Sen Wu, Daniel Selsam, and Christopher Ré. Data programming: Creating large training sets, quickly. In NewrlPS, volume 29, pages 3567-3575, 2016.
[34] Wendi Ren, Yinghao Li, Hanting Su, David Kartchner, Cassie Mitchell, and Chao Zhang. Denoising multi-source weak supervision for neural text classification. In Findings of EMNLP, pages 3739-3754, 2020.
[35] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. " why should trust you?" explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 1135-1144 2016.
[36] Esteban Safranchik, Shiying Luo, and Stephen Bach. Weakly supervised sequence tagging from noisy rules. In AAAI, volume 34, pages 3570-5578, 2020.
[37] Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Roberts, and Frederic Sala. Univer- salizing weak supervision. In ICLR, 2022.
[38] Daniel Ting and Eric Brochu. Optimal subsampling with influence functions. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
12

---

[39] P. Varma, Bryan D. He, Payal Bajaj, Nishith Khandwala, I. Banerjee, D. Rubin, and Christopher Ré. Inferring generative model structure with static analysis. Advances in neural information processing systems, 30:239-249, 2017.
[40] Paroma Varma, Frederic Sala, Shiori Sagawa, Jason Fries, Daniel Fu, Saclig Khattar, Ashwini Ramamoorthy, Ke Xiao, Kayvon Fatahalian, James Priest, and Christopher Ré. Multi-resolution weak supervision for sequential data. In NeurIPS, volume 32, 2019.
[41] Haonan Wang, Ziwei Wu, and Jingrui He. Training fair deep neural networks by balancing influence. CoRR, abs/2201.05753 2022.
[42] Zifeng Wang, Hong Zhu, Zhenhua Dong, Xiuqiang He, and Shao-Lun Huang. Less is better: Unweighted data subsampling via influence function. In AAAI, 2020.
[43] Renzhi Wu, Sheng Chen, Jieyu Zhang, and Xu Chu. Learning hyper label model for program- matic weak supervision. 2022.
[44] Peilin Yu, Tiffany Ding, and Stephen H Bach. Learning from multiple noisy partial labelers. AISTATS, 2022.
[45] Yue Yu, Simiao Zuo, Haoming Jiang. Wendi Ren, Tuo Zhao, and Chao Zhang. Finc-tuning pre-trained language model with weak supervision: A contrastive-reguulaiize self-training approach. NAACL-HLT, pages 1063-1077, 2021.
[46] Jieyu Zhang, Cheng- Yu Hsich, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision. arXiv preprint 2rXIv:2002.05433, 2022.
[47] Jieyu Zhang, Linxin Song, and Alexander Ratner. Leveraging instance features for label aggregation in programmatic weak supervision. arXiv preprint arXiv:2210.0277 2022.
[48] Jieyu Zhang, Bohan Wang. Xiangchen Song, Yujing Wang, Yaming Yang. Jing Bai, and Alexander Ratner. Creating training sets via weak indirect supervision. In ICLR, 2022.
[49] Jieyu Zhang, Yujing Wang, Yaming Yang, Yang Luo, and Alexander Ratner. Binary classification with positive labeling sources. In Proceedings of the 31 ACM International Conference on Information Knowledge Management, CIKM 222, page 4672 4676, New York, NY, USA, 2022. Association for Computing Machinery.
[50] Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. WRENCH: A comprehensive benchmark for weak supervision. Thirty-fifth Conference on Neural Information Processing Systems Dalasets and Benchmarks Truck, 2021.
[51] Rongzhi Zhang, Yue Yu, Pranav Shetty, Le Song, and Chao Zhang. Prboost: Prompt-based rule discovery and boosting for interactive weakly-supervised learning. In ACL, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See appendix.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
appendix.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See appendix.
(b) Did you include complete proofs of all theoretical results? [Yes] See appendix.
3. If you experiments...
13

---

(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See supplemen- tal materials.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No] For experiments with randomness, we report averaged results over 5 runs.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "Pw9lhotw6qy", "reviewer": "Reviewer_Dqzp", "summary": "This paper introduces a method called source-aware Influence Function (IF) to study the “influence” of individual data, source and class tuples on the performance of different label functions in the programmatic weak supervision paradigm. This differs from previous methods that calculate ordinary IF which can only identify influential training data and not the labeling function or the training data source responsible for mislabeling. The paper introduces a framework for estimating the influence of each training data on the test loss and prediction. The authors introduce two methods to calculate this source-aware IF. They support their method with theorems and proofs. They also show a variety of use-cases on different datasets as to how source-aware IF can improve the performance and understanding of PWS pipelines. ", "questions": "1. The two methods of calculating source-aware IF correspond to two different ways of perturbing the training loss. How do these 2 differ (maybe through examples) in identifying the most responsible (i,j,c) tuples? From tables 3 & 4, on average, RW seems to be doing better than WM. It would be nice to have a discussion on why that may be. \n\n2. What is the significance of RelatIF here? How is it better/different from ordinary IF and source-aware IF? R-RW seems to be outperforming RW (table 3) for Majority Vote but not for others. \n\n3. It would also be interesting to look at some examples from other datasets (tabular/textual) and see which LFs are responsible. In many real world non-image applications, the LFs may not be created similar to the experimental setup in the paper for the DN datasets. \n\n4. Having identified which LFs have mislabelings, it would be nice to see what can be done about it. For eg, how much does correcting these mislabelings affect the end performance? Is there a minimum number of corrections to be made to improve the classification results? I.e, how many mislabelings need to be identified to positively make an impact on the results?\n\n5. It would also be interesting to look at the difference in classification accuracy/F1 score on an unseen test set with the usual PWS setup and after removing the tuples most responsible for mislabeling (based on a validation set). \n\n6. In table 2, it might be more helpful to have the ‘Misclassified Test Data’ as the first column since the explanation goes from misclassification to the responsible components. \n\n7. There is a small grammatical error in the table 3 caption - “The larger the AP is, the better the method identify mislabeling of LFs.” \n", "limitations": "The authors have mentioned some limitations of their work. The proposed framework may be difficult to implement in the case of more complicated label models. It has not been defined for one-stage PWS pipelines which are becoming more popular recently. The paper does not include experiments on using complicated deep learning architectures as the end model. Additionally, this framework is also limited in its ability to decide/recommend how many samples (responsible for mislabeling) need to be discarded or down weighted for optimal classification performance. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper tackles a very interesting and important aspect of understanding programmatic weak supervision. The problem is well motivated. \n\n2. The paper includes theoretical proofs of the claims and methods used for developing source-aware IF.\n\n3. The generalization of the generation process of probabilistic labels using eq 6 is nice.\n\n4. The paper includes experiments on a wide range of benchmark datasets and classification tasks.\n\n5. The contributions of the paper are clear, original and significant. \n\nWeaknesses:\n1. The motivation for certain aspects of the method and experiments are unclear. For e.g:\n         a. The intuition behind why there are two methods for calculating source-aware IF is not clear - why reweighting vs weight-moving. There isn’t a clear study of how one is better or worse than the other and in what cases and why. \n         b. The significance of RelatIF in the context of the contributions of this paper is not clear. \n\n2. In the experiments, the training data most responsible for mislabeling is removed and the resulting test loss is reported. However, it is unconvincing if this is the best strategy. What happens if this process is repeated multiple times after removing 1 data point each time or the top n most responsible data points are removed?\n\n3. The test loss is shown to reduce after removing the most negatively influencing tuples, however, there is no mention of how this may translate to classification accuracy or other metrics. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OKA67Zxes23", "reviewer": "Reviewer_gUhW", "summary": "## Summary\nIn this work, the authors propose a general framework for quantifying the influence of individual PWS components on the end model. The proposed framework can be used to look into the influence of individual PWS components' effects on the end model. A source-aware influence function is proposed, which leverages the knowledge of the probabilistic label generation process and uses that knowledge to decompose the training loss into multiple terms and eventually individual influence scores. These influence scores are used to quantify the effect of PWS components (source vote, supervision source, and training data). \n", "questions": "n/a", "limitations": "n/a", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The effective use of the source-aware influence function along with the knowledge of the probabilistic label generation process provides a fine-grained analysis tool for the behavior of the end-model.\n2. The influence of individual PWS components can be analyzed with the proposed framework.\n3. The proposed framework has the capability to work with diverse data domains (tabular, image, textual). Secondly, an ample number of datasets are used in the study. \n4. The most responsible LFs as well as their capacity to mislabel can be identified, which is different from identifying the influential training data. \n5. The proposed model steers the end-model towards generalization, which is always desirable.\n6. The paper is well written.\n\n\n## Weaknesses\n1. The proposed framework hasn’t been shown to work with one-stage methods.\n2. Experimental results are reported for two-layer neural networks. It is still to be seen how the proposed method performs with intricate and deep neural architectures.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K6R1BevR9Hl", "reviewer": "Reviewer_Pj3z", "summary": "This paper focuses on weakly-supervised classification tasks where (i) multiple weak sources (i.e., labeling rules) are used in a label model (e.g., weighted majority voting) to generate soft labels for unlabeled data; (ii) the soft labels are used to train an end model (e.g., logistic regression, neural classifier) with a smooth cross-entropy loss. \n\nThe paper proposes a method to evaluate the influence of each weak source on the end model's performance by considering the choice of the label model, a critical component in this weak supervision pipeline. The main idea behind the proposed method is to decompose the training loss of the end model into multiple components corresponding to the individual weak sources. Then, two techniques are proposed to compute a source-aware influence function, namely reweighting and weight-moving. In cases of label models involving an exponential function in their generation process, the paper applies the proposed source-aware influence function techniques by training an approximate label model based on the identify function.\n\nThe paper evaluates and compares the proposed method with various alternatives for multiple use cases and datasets across tabular, text, and image modalities. ", "questions": "* What is the size of the validation set used in this paper? Does it make sense to use at least a few labeled data per class while training either the label model or the end model? \n", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* Overall, the paper is clearly written and provides clear and substantiated arguments.\n* The paper addresses an interesting and challenging problem. Evaluating the influence of weak sources on the end model's performance is an increasingly important research direction given the increasing adoption of programmatic labeling in both research and industrial settings. \n* The proposed method shows promising experimental results on an extensive experimental evaluation of multiple techniques on several scenarios and datasets. It is demonstrated that including the label model into the computation of IF helps across multiple use cases. \n\n\nWeaknesses: \n* It is hard to understand why (in theory) the proposed method should be effective across label model choices.\n  *  The proposed method unifies three types of label models into a single equation (Eq. (6) in Section 3.1) without having discussed the inner workings of each label model (e.g., weighted aggregation in Snorkel). Thus, it is hard for a reader not familiar with these label models to understand Eq. (6) and possibly the rest of the method. \n  * For cases where $\\sigma(\\cdot)$ is not the identity function, it is not clear whether the proposed method explains the real influence of each component. The approximated label model (that uses the identity function) is simpler and is not guaranteed to have the same behavior as the original label model. (Indeed, the two label models lead to different rankings as shown in Figure 1. Thus, it is not clear whether the estimated influence scores explain the real influence of each component. In addition to comparing the rankings, it would also help to compare the predicted labels of the original and approximated label models to give an idea of how well the latter approximates the former. \n  * It is not clear why the proposed method (resorting into approximations of label models) would in theory be more effective than a simpler method that first computes (source-agnostic) influence scores for each instance and then aggregates instance-level scores into source-level scores. \n* The problem addressed in this paper is a simplified (and possibly unrealistic?) case compared to problems addressed in practice in weak supervision\n  * According to the problem definition no labeled data are assumed for training, however, labeled data are considered in a validation set. In practice, (at least few) labeled data are considered during training (either in the label model or in the end model) and have been shown to improve the end model's performance. For, example, clean labeled data could be combined with weakly labeled data with weights. It is not clear whether the proposed method can be applied in this setting. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept