Mitigating Source Bias for Fairer Weak Supervision
Authors: Changho Shin
Sonia Cromp
Dyah Adila
Frederic Sala
Venue: NeurIPS 2023 poster (2023)
Keywords: Weak supervision
fairness
TL;DR: We deal with biases induced by weak supervision, proposing approaches to mitigate them in order to improve fairness and performance.

Abstract: Weak supervision enables efficient development of training sets by reducing the need for ground truth labels. However, the techniques that make weak supervision attractive---such as integrating any source of signal to estimate unknown labels---also entail the danger that the produced pseudolabels are highly biased. Surprisingly, given everyday use and the potential for increased bias, weak supervision has not been studied from the point of view of fairness. We begin such a study, starting with the observation that even when a fair model can be built from a dataset with access to ground-truth labels, the corresponding dataset labeled via weak supervision can be arbitrarily unfair. To address this, we propose and empirically validate a model for source unfairness in weak supervision, then introduce a simple counterfactual fairness-based technique that can mitigate these biases. Theoretically, we show that it is possible for our approach to simultaneously improve both accuracy and fairness---in contrast to standard fairness approaches that suffer from tradeoffs. Empirically, we show that our technique improves accuracy on weak supervision baselines by as much as 32\% while reducing demographic parity gap by 82.5\%. A simple extension of our method aimed at maximizing performance produces state-of-the-art performance in five out of ten datasets in the WRENCH benchmark.

Mitigating Source Bias for Fairer Weak Supervision
Changho Shin, Sonia Cromp, Dyah Adila, Frederic Sala
Department of Computer Sciences
University of Wisconsin-Madison {cshin23, cromp, adila, fsala)Gwisc.edu
Abstract
Weak supervision enables efficient development of training sets by reducing the need for ground truth labels. However, the techniques that make weak supervision attractive- such as integrating any source of signal to estimate unknown labels- also entail the danger that the produced pseudolabels are highly biased. Surprisingly, given everyday use and the potential for increased bias, weak supervision has not been studied from the point of view of fairness. We begin such study, starting with the observation that even when a fair model can be built from a dataset with access to ground- -ruth labels, the corresponding dataset labeled via weak supervision can be arbitrarily unfair. To address this, we propose and empirically validate model for source unfairness in weak supervision, then introduce simple counterfactual Bairesss-bssed technique that can mitigate these biases. Theoretically, we show that it possible for our approach to simultaneously improve both accuracy and fairness-in in contrast to standard fairness approaches that suffer from tradeoffs. Empirically, we show that our technique improves accuracy on weak supervision baselines by much as 32% while reducing demographic parity gap by 82.5%. A simple extension of our method aimed at maximizing performance produces state-of-the-arr performance in five out of ten datasets in the WRENCH benchmark.
Introduction
Weak supervision (WS) is powerful set of techniques aimed at overcoming the labeled data bottleneck [RSW*10 16 FCS 20 SLV*22). Instead of manually annotating points, users assemble noisy label estimates obtained from multiple sources, model them by learning source accuracies, and combine them into a high-qualit pseudolabel to be used for downstream training. All of this is done without any ground truth labels. Simple, flexible, yet powerful, weak supervision is now standard component in machine learning workflows in industry, academia, and beyond [BRI. [BRI-19) Most excitingly. WS has been used to build models deployed to billions of devices.
Real-life deployment of models, however, raises crucial questions of fairness and bias, Such questions are tackled in the burgeoning field of fair machine learning [DHP+ 12, HPST6]. However, weak supervision has not been studied from this point of view. This is not a minor oversight. The properties that make weak supervision effective (i.e., omnivorously ingesting any source of signal for labels) are precisely those that make it likely to suffer from harmful biases. This motivates the need understand and mitigate the potentially disparate outcomes that result from using weak supervision.
The starting point for this work is a simple result. Even when perfectly fair classifiers are possible when trained ground-truth labels, weak supervision-basee techniques can nevertheless produce arbitrarily unfair outcomes. Because of this, simply applying existing techniques for producing fair outcomes to the datasets produced via WS insufficient-deeiverin highly suboptimal datasets. Instead, new approach, specific to weak supervision, must be developed. We introduce a simple technique for improving the fairesss properties of weak supervision-basel models. Intuitively, a major cause of bias
37th Conference on Neural Information Processing Systems (NeurIPS 2023).

---

center
a center
center
Group 0
g
g
Group 1
+ Class
Class
Accuracy
1.0
h
0.5
(a) Data
(b) Transformations
(c) Recovered labels
Figure/ 1: Intuitive illustration for our setting and approach. (a): circles and diamonds are data points from group 0 Ondd 1, respectivcly. The accuracy of labeling function is -ooeed-coded, with blue being perfect (1.0) and red random (0.5). Note that accuracy degrades as data points get farther from center center (star). (b) We can think of group having been moved far from the center by a transformation, g. producing lower-quality estimates and violating fairness downstream. (c) Our technique uses counterfactual fairness undo this transformation, obtaining higher quality estimates. and improved fairness.
in WS is that particular sources are targeted at certain groups, and so produce far more accurate label estimates for these groups-ait and far more noise for others. We counteffacctlll ask what outgroup points would most be like if they were part of the 'privileged' group (with respect to each source), enabling us to borrow from the more powerful signal in the sources applied this group. Thus, the problem is reduced to finding a transformation between groups that satisfies this counterfactual. Most excitingly, while in standard fairness approaches there is typical tradeoff between fairness and accuracy, with our approach. both the fairness and performance of WS-based techniques can be (sometimes dramatically) improved.
Theoretically, in certain settings, provide finite-sample rates recover the counterfactual transfor- mation. Empirically, we propose several ways to craft an Afficientlyccopprte transformation building on optimal transport and some simple variations. We validate claims diverse set of experiments. These include standard geal-world fairness datasets, where we observe that our method can improve both fairness and accuracy by as much as 82.5% and 32.5%, respectively, versus weak supervision baselines. Our method can also be combined with other fair ML methods developed for fully supervised settings, further improving fairness. Finally, our approach has implications for WS beyond bias: we combined it with slice discovery techniques (EVS 22) to improve latent underperforming groups. This enabled us improve on state-of-the-are on the weak supervision benchmark WRENCH ZYL*2]
The contributions of this work include,
The first study fairness in weak supervision,
model for weak supervision that captures labeling function bias, A simple counterfactual faimess-basedd correction to mitigate such bias, compatible with any existing weak supervision pipeline, as well as with downstream fairness techniques,
A new
Theoretical results showing that (1) even with fair dataset, a be arbitrarily biased and (2) finite-sample recovery result for the proposed algorithm,
counterpart can
Experiments validating our claims, including on supervise forms of popular fairness evalua- tion datasets, showing gains in fairness metrics- and often simultancously improvements in accuracy.
2 Background and Related Work
We present some high-level background on weak supervision and fairness in machine learning. Afterward, we provide setup and present the problem statement.
Weak Supervision Weak supervision frameworks build labeled training sets with no access to ground truth labels. Instead, they exploit multiple sources that provide noisy estimates of the label. These sources include heuristic rules, knowledge base lookups, pretrained models, and more
18). Because these sources may have different- and lakknown-accuraciie accuracies and dependencies, their outputs must be modeled in order to produce combination that can be used a high-quality pseudolabel.
KOS1 MBS109 GM14 17 RBE+18
unknown-

---

Concretely, there is dataset 881....,,((............ with unobserved true label y E {-1,+1}. We can access the outputs of m sources (labeling functions) "A."....A!" :x' : (-1,+1] These outputs are modeled via generative model called the label model, po(A' ...." .y). The goal is to estimate the parameters A this model, without accessing the latent y, and to produce a pseudolabel estimate Po(u/A (pp")A"....A"")))) For more background, see [ZHY+22).
Machine Learning and Fairness Fairness in machine learning is a large and active field that seeks to understand and mitigate biases. We briefly introduce high-level notions that will be useful in the weak
supervision setting, such as the notion of fairness metrics. Two popular choices are demographic parity [2HP+12] and equal opportunity [HPST6]. Demographic parity based on the notion that individuals of different groups should have equal treatment, i.e., if the group attribute, P(Ý = 1|A- - 1) =
P(²=1|A=0) The equal opportunity principle requires that predictive error should be equal across groups, i.e., P(Ý - -,A=)- and seek to improve fairnsss in different machine learning settings based on these metrics. Typically, the assumption is that the underlying dataset differs within groups in such way that a trained model will violate, for example, the equal opportunity principle. contrast, in this work, we focus on additional violations of fairness that are induced by weak supervision pipelines-wihh -which can create substantial unfairness even when the true dataset is perfectly fair. In the same spirit, [W222 considers fairness in positive-and-unlae unlabeled (PU) settings, where true labels are available, but only for one class, while other points are unlabeled. Another related line of research is fairness under noisy labels [WLL2]] KL22 23 ZZL+233 These works consider the noise rate of labels in fair learning, enhancing the robust- ness of fair learning methods. crucial difference between such works and ours: in weak supervision, we have multiple sources of noisy labels and we can exploit these directly improve dataset faimess..
= = =0). A large number of works study, measure,
Counterfactual Fairness Most closely related to the notion of faimesss we use in this work is counterfactual fairness. [KLRSI7] introduced such counterfactual fairness notion, which implies that changing the sensitive attribute A, while keeping other variables causally not dependent on A, should not affect the outcome. While this notion presumes the causal structure behind the ML task, it is related to our work in the sense that our proposed method tries to remove the causal effect by A with particular transformations. more recent line of work has proposed bypassing the need for causal structures and directly tackling counterfactual fairness through optimal transport [GDBFLI9] BYF20 20 SMBN21 BDB22]. The idea is to detect or mitigate unfairness by mapping one group to another group via such techniques. In this paper, we build on these tools to help improve fairness while avoiding the accuracy sauracy--aiiness tradeoff common to most settings.
3 Mitigating Labeling Function-Induced Unfairness
We are ready to explain our approach to mitigating unfairness in weak supervision sources. First, we provide flexible model that captures such behavior, along with empirical evidence supporting it. Next, we propose simple solution to correct unfair source behavior via optimal transport.
Modeling Group Bias in Weak Supervision Weak supervision models the accuracies and correlations in labeling functions. The standard model, used in [RHD+ 19a FCS+20) and others is 3) = 2/2 exp($fy)+ E/ 0, all of our discussion below holds when considering correlations as well. Here, Z is the normalizing partition function. The 0 are canonical parameters for the model. 0, sets the class balance. The 0,'s capture how accurate labeling function (LF) i is: if 0; =0, the LF produces random guesses. If 0, is relatively large, the LF highly accurate. A weakness of this model is that it ignores the feature vector x. It implies that LF's are uniformly accurate over the feature space-a a highly unrealistic assumption. A more general model was presented in |CFA * "22). where there a model for each feature vector x; i...
with 8, 20. We leave out the correlations for simplicity;
-y== (A ..... A" .y) = 1==00p(0p eexp(0y Σ 20,NN()))))
1
However, as we see only one sample for each xr it is impossible to recover the parameters Oz. Instead, the authors assume a notion of smoothness. This means that the 0z.- 's do not vary in small neighborhoods, so that the feature space can be partitioned and single model Iearned per part. Thus
3

---

model ① from 22F is more general. but still requires a strong smoothness assumption. It also does not encode any notion of bias. Instead, we propose a model that encodes both smoothness and bias.
Concretely, assume that the data are drawn from some distribution on 2 xy, where 2 is a latent space. We do not observe samples from 2. Instead, there are I transformation functions 91,-.:91: where It 2 + X. For each point 24, there is an assigned group k and we observe I; = (z().. Then, our model is the following:
02 1+ Ceeemr, gk
(=)""))).A""")))) Z exp
Oyy
X (ax (z))y
We explain this model as follows. We can think of as particular version of T.. However, instead of arbitrary f/.. parameters for each æ, we explicitly model these parameters as two components: feature- indeeendend accuracy parameter 8g and term that modulates the accuracy based on the distance between feature vector x and some fixed center x The center represents, for each LF, most accurate point, where accuracy is maximized a level set by 0z. As the feature vector x= 9& (x) moves away from this center, the denominator 1 + d( (d(:,etttr, (0x(2)) increases, and the LF votes increasingly poorly. This is explicit form of smoothness that validate empirically below.
For simpliciit, we assume that there are two groups, indexed by 0,1, that X' = Z, and that go(z) = z. In other words, the transformation for group 0 is the identity, while this may not be the case for group Simple extensions of our approach can handle cases where none of these assumptions are met.
Labeling Function Bias The model 2 ex- plains how and when labeling functions might be biased. Suppose that O& takes points 2 far from seemeer, Then, the denominator term in ( grows-and so the penalty for X(x) to disagree with y is reduced, making the labeling function less accurate. This is common in practice. For example, consider a situation where bank uses features that include credit scores for loan review. Suppose that the group variable is the applicant's nationality. Immigrants typically have a shorter period to build credit; this is reflected in a trans- formed distribution gi (z). A labeling function using a credit score threshold may be accurate for mon-immigganns, but may end up being highly inaccurate when applied to immigrants.
Adult (LIFT)
High accuracy group
Low accuracy group
1.0
0.8
0.6
2
||x
||x x
LF
LF3
LF5 LF 6
LF 7
LF9
LF2
8F
Figure 2: Average accuracy (y-axis) depending on the distance to the center point (x-axis). The cen- ter is obtained by evaluating the accuracy of their neighborhood data points.
We validate this notion empirically. We used the Adult dataset K 16], commonly used for fairness studies, with a set of custom-bui labeling functions. In Figure 2 we track the accuracies of these LFs function of distance from an ampirically-disoovee group in each labeling function; as expected in our model, as the distance from the center || || p: increased, the accuracy decreases. On the right-hand side, we the lower-accuracy group, whose labeling functions are voting x; = 91 (z;). This transformation has sent these points further away from the center (note the larger distances). As result, the overall accuracies have also decreased. Note, for example, how LF 5. in purple, varies between 0.9 and 1.0 accuracy in one group and is much worse between 0.6 and 0.7 i.7-in the other.
center centtes On the left is the high-accuracy
3.1 Correcting Unfair LFs
Given the model (2) how can we reduce the bias induced by the 9k functions? A simple idea is to reverse the effect of the g& ss. If we could invert these functions, violations of fairness would be mitigated, since the accuracies of labeling functions would be uniformized over the groups
Concretely, suppose that 9& is invertible and that hy is this inverse. If we knew h4g then we could ask the labeling functions to vote on h4 (4: = hy (gk = rather than on x = 9u (=), and we could do so for any group. yielding equal-accuracy estimates for all groups. The technical challenge is how to estimate the inverses of the yj 's, without any parametric form for these functions. To do so, we
deploy optimal transport (OT) [PC 19). OT transports probability distribution to another probability

---

Algorithm 1: SOURCE BIAS MITIGATION (SBM)
1: Parameters: Features Xo. X1 and LF outputs Ao (1...... 10) = [A_..... XT for groups 0, 1,
transport threshold
2: Returns: Modified weak labels A- [A",...,A'"
3: Estimate accuracy of X in each group, - Algorithm
from Ag,A1 with
4: for j (1,2....,7m] do
5: if a dg + then update Xg by transporting Xo to X1 (Algorithm)
6: else if 0g 2 ad tE then update X by transporting X1 to Xo (Algorithm)
7: end for
8: return A= Am
distribution by finding minimal cost coupling. We use OT to recover the reverse map hy: :X' - Z by hu (en (z))((())) where e is cost functon, is probability measure in X and w is probability measure in Z.
Our proposed approach, building on the use of OT; is called source bias mitigation (SBM), It seeks to reverse the group transformation gt via OT. The core routine is described in Algorithm The first step of the algorithm is to estimate the accuracies of cach group that can identify which group is privileged, i.c., which of the transformations go.91 is the identity map. To do this, we use Algorithm FCCS 20) by applying it to each group separately.
After identifying the high-acce securacc group, we transport data points from the low-accuracy group to Since not every transported point perfectly matches an existing high-accuracy group point, we find a nearest neighbor and borrow its label. We do this only when there a sufficient inter-group accuracy gap, since the error in transport might otherwise offset the benefit. In practice, if the transformation is sufficiently weak, is possible to skip optimal transport and simply use nearest neighbors. Doing this turned out to be effective in some experiments (Section 5.1.. Finally, after running SBM, modified weak labels are used in a standard weak supervision pipeline, which is described in Appendix[C
Theoretical Results
We provide two types of theoretical results. First, we show that labeling function bias can be arbitrarily bad red-resun in substantial unfairness- regardless of whether the underlying dataset is fair. Next, we show that in certain settings, we can consistently recover the fair labeling function performance when using Algorithm [ and provide a finite-sample error guarantee. Finally, we comment on extensions. All proofs are located in Appendix
Setting and Assumptions We assume that the distributions Po (z and Pı (1(")) are subgaussian with means Ni and μ1 and positive-definite covariance matrices Ep and E1, respectively. Note that by assumption, Po (z) = P(z) and (z')) is the pushforward of (o()) under 91. Let r(Σ) denote the effective rank of Σ [Ver18). We observe no and i.i.d. samples from groups 0 and 1, respectively. We use Fuclidean distance as the distance d(x.y) = ||x mo-yl yl in model (2). For the unobserved ground truth
labels, Y4 is drawn from some distribution Pgy|2). Finally, the labeling functions voting our points are drawn via the model
4.1 Labeling Functions can be Arbitrarily Unfair
We show that, as a result of the transformation 91. the predictions labeling functions can be arbitrarily unfair even if the dataset is fair. The idea is simple: the average group 0 accuracy, E,ez [P (P(((())) v)]. is independent of gı, so suffices to show that Eg (2) PP({(=') = v)] can deteriorate when 9i moves data points far from the center centero As such, we consider the change in Ex (P(((())) y)] as the group points are transformed increasingly far from centero expectation.
Theorem 4.1. Let (k) be an arbitrary sequence of functions such that comeerol + co. Suppose our assumptions above are met; par-
limp-ooc E Eyeg! (2) [2)]llzz ticular, that the label is independent of the observedfeaturrss==11 (rz''))rr'''' (2) conditioned
5

---

Table 1:
Tabular dataset results
Adult
Bank Marketing
Acc (t)
F1 (1)
ADP (4)
Ago (4)
Ace (t)
F1 (f)
ADP (4)
Ago (L)
FS
0.824
0.564
0.216
0.331
|0.912
0.518
0.128
0.117
WS (Baseline)
0.711
0.587
0.475
0.325
0.6744
0.258
0.543
0.450
SBM (w/o OT)
0.720
0.592
0.439
0.273
0.876
0.550
0.106
0.064
SBM (OT-L)
0.560
0.472
0.893
0.980
0.892
0.304
0.095
0.124
SBM (OT-S)
0.723
0.590
0.429
0.261
0.847
0.515
0.122
0.080
SBM (w/o OT) LIFT
0.704
0.366
0.032
0.192
0.698
0.255
0.088
0.137
SBM (OT-L) LIFT
0.700
0.520
0.015
0.138
0.892
0.305
0.104
0.121
SBM (OT-S) LIFT
0.782
0.448
0.000
0.178
0.698
0.080
0.109
0.072
on the latent features z. Then,
lim E 10000 w'Es)
(P)((())))))))
which corresponds random guessing.
It is easy to construct such a sequence of functions g) for instance, by letting g) ( (2) = 2 ku, where is d-dimensional vector of ones. When the distribution of group I points lies far from centera while the distribution of group O points lies near to zentero the accuracy parity of À suffers. With adequately large expected depeeenteoo .91 (k) (z)). the performance of on group I points approaches random guessing.
4.2 Finite-Sample Bound for Mitigating Unfairness
Next, provide result bounding the difference in LF accuracy between group 0 points, Ezez (P(x) = y)]. and group 1 points transformed using our method, Ex' (x==x(((((((")==)))) tighter bound on this difference corresponds to better accuracy
parity.
Theorem 4.2. Set to be (()))//,,.(..)///........... t/mmin(no, ()) /max((n,,)))) ))) and let C be constant. Under the assumptions described above, when using Algorithm for any 0, we have that with probability 1
Next we interpret Theorem 4.2 LF accuracy recovery scales with max (1mxx11//// ///. /ni) This does not present any additional difhculties compared vanilla weak supervision-itts is the same rate we need to learn LF accuracies. In other words, there is no sample complexity penalty for using our approach.
That is, when the distributions (o(x) Pi (xx) have greater spread, it more difficult restore fair behavior.
Furthermore, LF accuracy recovery scales inversely to max (((())r(E)))r((3)
Finally, we briefly comment on extensions. It is not hard to extend these results to a setting with less strict assumptions. For example,
take P be mixture of Gaussians. In this case, it is possible to combine algorithms for learning mixtures (CGTI8) with the approach we presented.
Experiments
The primary objective of our experiments is validate that SBM improves fairness while often enhancing model performance as well. In real data experiments, we confirm that our methods work well with real-world fairness datasets (Section 5. In the synthetic experiments, we validate our theory claims in fully controllable setting shatinggshowingg that our method can achieve perfect fairness and performance recovery (Section55.2) In addition, we show that our method compatible with other fair ML techniques developed for fully supervised learning (Section 5.3) Finally, demonstrate that our

---

Table NLP dataset results
Table 3: Vision dataset results
Dataset
Methods
Acc
F1
Apr
Ago
Dataset
Methods
Ace
F1
Apr
A
FS
0.893
0,251
0,083 0.091
FS
0.897 0.913
0.307
0.125
WS (Baseline)
0,854
0.223 0.560 0.546
WS (Baseline)
0.866
0.879
0.308
0.193
Civil
SBM (w/o OT)
0.879 0.068
0.048 0.047
CelebA
SBM (w/o OT)
0.870 0.883
0.309
0.192
SBM (OT-L)
0.880
0.070
0.042
0.039
SBM (OT-L)
0.870 0.883
0.306
0.185
SBM (OT-S)
0.882
0.047
0.028 0.026
SBM (OT-S)
0.872 0.885
0.306
0.184
FS
0.698
0.755 0.238 0.121
FS
0.810 0.801
0.133 0.056
WS (Baseline)
0.584 0.590
0.170 0.133
WS (Baseline)
0.791 0.791 0.172 0.073
Hate
SBM (w/o OT)
0.592 0.637 0.159 0.138 UTKF
SBM (wlo OT)
0.797 0.790 0.164 0.077
SBM (OT-L)
0.670
0.606
0.120 0.101
SBM (OT-L)
0.800 0.793
0. 0.135 0.043
SBM (OT-S)
0.612
0.687 0.072 0.037
SBM (OT-S)
0.804 0.798 0.130 0.041
method can improve weak supervision performance beyond fairness by applying techniques to discover underperforming data slices (Section[5.4) This enables us to outperform state-of oft---ff-thh the-tre-ttt on popular Our code is available at htttps//gghuub.com/ppeee.c aabfar .ab/fair
weak supervision benchmark [ZYL. +
WS
5.1 Real data experiments
Claims Investigated In real data settings, we hypothesize that our methods can reduce the bias of LFs, leading to better fairness and improved performance of the weak supervision end model.
Setup and Procedure We used datasets in three different domains: tabular (Adult and Bank Marketing), NLP (CivilComments and HateXplain), and vision (CelebA and UTKFace). Their task and group variables are summarized in Appendix Table
[E] 7
LFs are either heuristics or pretrained
models. More details are included in Appendix E.3
For the weak supervision pipeline, we followed standard procedure. First, we generate weak labels from labeling functions in the training set. Secondly, we train the label model on weak labels. In this experiment, we used Snorkel [BR 19 as the label model in weak supervision settings. Afterwards, we generate pseudolabels from the label model, train the end model on these, and evaluate it on the test set. We used logistic regression as the end model. The only difference between our method and the original weak supervision pipeline is procedure to fix weak labels from each labeling function. As sanity check, a fully supervised learning result (FS), which is the model performance trained on the true labels, is also provided. Crucially, however, in weak supervision, do not have such labels, and therefore fully supervised learning is simply an upper bound to performance and not a baseline.
We ran three variants of our method. SBM (w/o OT) is 1-nearest neighbor mapping to another group without any transformation. SBM (OT-L) a 1-nearest neighbor mapping with linear map learned via optimal transport. SBM (OT-S) is a 1- --nearest neighbor mapping with a barycentric mapping learned via the Sinkhorn algorithm. To see if our method can improve both fairness and performance, we measured the demographic parity gap (ApP) and the equal opportunity gap (O) as fairness metrics, and computed accuracy and F1 score as performance metrics as well.
The tabular dataset result reported in Table As expected, our method improves accuracy while reducing demographic parity gap and equal opportunity gap. However, we observed SBM (OT-L) critically fails at Adult datasct, contrary to what we anticipated. We suspected this originates in one-hot coded features, which might distort computing distances in the nearest neighbor search. To work around one-hot coded values in nearest neighbor search, we deployed LIFT [DZZ*22), which encodes the input as natural language (e.g. "She/he is <race attribute>. She/he works for <working hour attribute> per week ...") and embeds them with language models (LMs). We provide heuristic rules to convert feature columns into languages in Appendix E.2 and we used BERT as the language model. The result is given in Table T Junder the dashed lines. While sacrifices a small amount of accuracy, it substantially reduces the unfairnsss expected.
Results
The results for NI.P datasets are provided in Table the CominComments and HateXplain datasets, we observed our methods mitigate bias consistently, as we hoped. While our methods improve performance
7

---

cutrextion
LF 1XS-
Perapstted
Racovaed
Greup o [Ckas
untess Broe
J X K-Seppp
Group 1 (ikas of
Reseened Tcea dr creap
Gropp (ikan
10
Receened Ioua Grep
23
bedtan
3.
23
crbss Grea
Pangeette if paias e hreqorte Hiise Bioe)
-25
-25
10.3 -14
-25 08
23
110
-1 120 -2
123 122 -M -3 -25 02
(a) True distribution
(b) LF
(c) Transported LF
(d) Recovered LF
Figure 3: Synthetic datasets. In (a), seemingly different data distributions from the two groups actually have perfect achievable faireess. However, the labeling function in (b) only works well in group 0, which leads to unfairness. Via OT (c), the input distribution can be matched and the LF applied to similar cooupp--oriiinal and recovered. As result, LFs group works well on 0 (d).
as well in the HateXplain dataset, enhancing other metrics in CivilComments results in drops in the Fl score, We believe that a highly unbalanced class setting (P(Y= = 0.1) is the cause of this result.
The results for vision datasets are given in Table Though not as dramatic as other datasets since here the LFss are pretrained models, none of which heavily biased, methods can still improve accuracy and fairness. In particular, our approach shows clear improvement over the baseline, which yields performance closer to the fully supervised learning setting while offering less bias.
5.2 Synthetic experiments
Pertormance (
Untairness
DP (SBM)
EO (SBM)
DP (WS Baseline)
0.9
0.8
EO (WS Baseline)
0.6
0.4
0.7
F1 (SBM)
0.2
0.6
Acc (WS Baseline)
F1 (WS Baseline)
10a
10?
10
10a
10
10a
107
10a
102
10a
10a
107
10F
10'
Number samples
Number of samples
(a) Number of samples vs. Performance
(b) Number of samples vs. Unfaimesss
Figure 4: Synthetic experiment on number of samples vs. performance and fairness. Confidence intervals are obtained by +1.66 standard deviation of 10 repetition with different seeds.
Claim Investigated We hypothesized that method can recover both fairness and accuracy (as function of the number of samples available) by transporting the distribution of one group to another group when our theoretical assumptions are almost exactly satisfied. To show this, we generate unfair synthetic data and LFs and see if our method can remedy LF fairness and improve LF performance.
Setup and Procedure We generated a synthetic dataset that has perfect faimess as follows. First, input features in IR2 are sampled as Xo Xo~~((0,11 (0,/) for group O, and labels Y, are set by Yn 10,o 0.5), i.e. 1 if the first dimension is positive or equal. Afterwards, n input features in R are sampled as X1 (0, for group 1, and the labels are also set by Y; 1 [XX1[0] 2 0.5). Then, linear [-4
transformation is applied the input distribution: X₁ ΣX; l where μ
-
Σ
which is the distribution of group 1. Clearly, we can see that X; ΣX, + ((,,E Here we applied the same labeling function X(x) = 11 (x(00 2 0), which the same the true label distribution in group 0.
8

---

Table 4: Compatibility with other fair ML
Table 5: Slice discovery with SBM results in WRENCH. Evaluation metric is accuracy for iMDb, FI for the rest.
methods (HateXplain dataset)
Acc
F1
App
A Ago
Basket ball
Mush
Methods
Census
iMDb
Tennis
FS
0.698
0.755
0.238
0.121
room
WS (Baseline)
0.584
0,590
0,171
0.133
FS
0.855
0.634
0.780
0.982 0,858
SBM (OT-S)
0.612
0.687
0.072
0.037
WS (HyperLM)
0.259
0.551
0.753
0.866
0.812
WS (Baseline)
0.539
0.515
0.005
0.047
SBM (w/o OT)
0.261
0.568
0.751
0.790 0.819
OTh-DP
SBM (OT-L)
0.242
0.547
0.756
0.903
0,575
SBM (OT-S)
0.607
0.694
0.002
0.031
SBM (OT-S
0.260
0.552
0.756
0.935 0.663
+ +OTh-DP
We apply our method (SBM OT-L) since our data model fits its basic assumption. Again, we evaluated the results by measuring accuracy, F1 score, Apr, and A go. The setup and procedure are illustrated in Figure
We varied the number of samples n from 102 to 10?
The result is reported in Figure As we expected, we saw the accuracy and Fl score are consistently improved as the linear Monge map is recovered when the number of samples n increases. Most importantly, we observed that perfect fairness achieved after only a small number of samples (10²) are obtained.
Results
5.3
Compatibility with other fair ML methods
Claim Investigated Our method corrects labeling function bias at the individual LF-and not model-level. We expect our methods can work cooperatively, in constructive way, with other fair ML methods developed for fully supervised learning settings.
Setup and Procedure We used the same real datasets, procedures, and metrics before. We com- bined the optimal threshold method (HPST6) with WS (baseline) and our approach, SBM (Sinkhorn). We denote the optimal threshold with demographic parity criteria as OTh-DP.
The results are shown in Table As we expected, we saw the effect of optimal threshold method, which produces an accuracy-faimes, (DP) tradeoff. This has the same effect upon our method. Thus, when optimal threshold is applicd to both, our method has better performance and fairness aligned with the result without optimal threshold. More experimental results with other real datasets and additional fair ML methods are reported in Appendix E.5
Results
5.4 Beyond fairness: maximizing performance with slice discovery
Claim Investigated We postulated that even outside the context of improving faimesss, our techniques can be used to boost the performance of weak supervision approaches. In these scenarios, there are no pre-specified groups. Instead, underperforming latent groups (slices) must first be discovered. Our approach then uses transport to improve labeling function performance on these groups.
Setup and Procedure WRENCH benchmark 21 which is a well-known weak supervision benchmark but does not include any group information. We generated group annotations by slice discovery [KGZ19 SNS 21 ddWLB22 EVS 22) which is an approach to discover data slices that share a common characteristic. discovers regions of the
We used Basketball, Census, iMDb, Mushroom, and Tennis dataset from the
To find groups with large accuracy gap, we used Domino embedding space based on the accuracy of model. Since the WS setting does not allow access to true labels, we replaced true labels with pseudolabels obtained from the label model, and model scores with label model probabilities. In order to show we can increase performance even for taee--of-----rr-re weak supervision, we used the recently-proposed model. We used the group information generated by the two discovered slices to apply our methods. We used logistic regression as the end model, and used the same weak supervision pipeline and metrics as in the other experiments, excluding faimess.
saate-of-the-an Hyper Label Model WWCC22) as the label

---

Results The results can be seen in T As expected, even without known group divisions, we still observed improvements accuracy and Fl score. We the most significant improvements on the Mushroom dataset, where we substantially close the gap fully-supervised. These gains suggest that is possible to generically combine our approach with other principled methods for subpopulation discovery to substantially improve weak supervision in general settings.
6 Conclusion
Weak supervision has been successful in overcoming manual labeling bottlenecks, but its impact on fairness has not been adequately studied. Our work has found that WS can easily induce additional bias due to unfair LFs. In order to address this issue, we have proposed a novel approach towards mitigating bias in LFs and further improving model performance. We have demonstrated the effectiveness of our approach using both synthetic and real datasets and have shown that it is compatible with traditional fair MI methods. We believe that our proposed technique can make weak supervision safer to apply in important societal settings and so encourages its wider adoption.
Acknowledgments
We are grateful for the support of the NSF under CCF2106707 (Program Synthesis for Weak Su- pervision) and the Wisconsin Alumni Research Foundation (WARF). We thank Nick Roberts, Harit Vishwakarma, Tzu-Heng Huang, Jitian Zhao, and John Cooper for their helpful feedback and discus- sion.
References
[ABD+18] Alekh Agarwal, Alina Beygelzimer, Miroslav Dudík, John Langford, and Hanna Wallach. A reductions approach to fair classification. In International Conference on Machine Learning, pages 60 69. PMLR, 2018.
[2NC*22] Simran Arora, Avanika Narayan, Mayee F Chen, Laurel Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. Ask me anything: A simple strategy for prompting language models. arXiv preprint arXiv:2210.02444 2022.
[BDB22] Maarten Buyl and Tijl De Bie. Optimal transport of classifiers to faimesss In Advances in Neural Information Processing Systems, 2022.
[BDE+20] Sarah Bird, Miro Dudík, Richard Edgar, Brandon Horn, Roman Lutz, Vanessa Milan, Mehrnoosh Sameki, Hanna Wallach, and Kathleen Walker. Fairlearn: toolkit for assessing and improving fairness in ai. Microsoft, Tech. Rep. MSR-Th TR- 2020-32, 2020.
[BDS+19] 19] Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion proceedings of the 2019 world wide web conference, pages 49 1-500, 2019.
[BRL+10 + 19] Stephen H Bach, Daniel Rodriguez, Yintao Liu, Chong Luo, Haidong Shao, Cas- sandra Xia, Souvik Sen, Alex Ratner, Braden Hancock, Houman Alborzi, et al. Snorkel drybell: A case study in deploying weak supervision at industrial scale. In Proceedings of the 2019 International Conference Management of Data, pages 362-375, 2019.
[BYF20] Emily Black, Samuel Yeom, and Matt Fredrikson. Fliptest: faimess testing via opti- mal transport. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pages 111 11-1111, 2020.
[CFA+2] +22] Mayee F Chen, Daniel Fu, Dyah Adila, Michael Zhang, Frederic Sala, Kayvon Fatahalian, and Christopher Ré. Shoring up the foundations: Fusing model em- beddings and weak supervision. Uncertainty Artificial Intelligence, pages 357-367. PMLR, 2022.
10

---

[CGT18] Yongxin Chen, Tryphon T Georgiou, and Allen Tannenbaum. Optimal transport for gaussian mixture models. IEEE Access, 7:6269-622 2018.
[Cut13] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.
[DCLT18] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding. arXiv preprini arXiv: 0100110004400 04805, 2018.
[ddWLB22] Greg d' d'Eon, Jason d' Eon, James R Wright, and Kevin Leyton Broon. The spotlight: A general method for discovering systematic errors in deep learning models. In 2022 ACM Conference Fairness, Accountability. and Transparency, pages 1981, 2022.
[DHP*12] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness through awareness. Proceedings of the 3rd innovations in theoretical computer science conference, pages 214-226, 2012.
[DS79] Alexander Philip Dawid and Allan Skene. Maximum likelihood estimation of observer error-rates using the em algorithm. Applied statistics, pages 20-28, 1979.
[DVMWVdM19] Terrance De Vries, Ishan Misra, Changhan Wang. and Laurens Van der Maaten. Does object recognition work for everyone? In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 52-59, 2019.
[DZS+17] Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. Neural ranking models with weak supervision. In Proceedings the International ACM SIGIR Conferenceon Research and Development Information Retrieval, 2017.
the 40th
[DZZ+222] Tuan Dinh, Yuchen Zeng, Ruisu Zhang, Ziqian Lin, Michael Gira, Shashank Rajput, Jy-yong Sohn, Dimitris Papailiopoulos, and Kangwook Lee. Lift: Language- interfaced fine-tuning for non-language machine learning tasks. In Advances in Neural Information Processing Systems, 2022.
[EVS+22] Sabri Eyuboglu, Maya Varma, Khaled Saab, Jean-Benoit Delbrouck, Christo- pher Lee-Messer, Jared Dunnmon, James Zou, and Christopher Ré. Domino: Discovering systematic errors with cross-modal embeddings. arXiv preprint wrXiv:2203.14960 2022.
[FCG+21] Rémi Flamary, Nicolas Courty, Alexandre Gramfort, Mokhtar Z. Alaya, Aurélic Boisbunon, Stanislas Chambon, Laetitia Chapel, Adrien Corenflos, Kilian Fatras, Nemo Fournier, Léo Gautheron, Nathalie T.H. Gayraud, Hicham Janati, Alain Rakotomamonjy, levgen Redko, Antoine Rolet, Antony Schutz, Vivien Seguy, Danica J. Sutherland, Romain Tavenard, Alexander Tong, and Titouan Vayer. Pot: Python optimal transport. Journal of Machine Learning Research, 22(78):1-8, 2021.
[FCS+20] Daniel Y. Fu, Mayee Chen, Frederic Sala, Sarah M. Hooper, Kayvon Fatahalian, and Christopher Ré. Fast and three-rious: Speeding up weak supervision with triplet methods. In Proceedings of the 37th International Conference on Machine Learning (ICMI. 2020), 2020.
[FLF19] Rémi Flamary, Karim Lounici, and André Ferrari. Concentration bounds for linear monge mapping estimation and optimal transport domain adaptation. arXiv preprint arXiv: 10155, 2019.
[GDBFL19] Paula Gordaliza, Eustasio Del Barrio, Gamboa Fabrice, and Jean-Michel Loubes. Obtaining fairness using optimal transport theory. In International Conference on Machine Learning, pages 2357-2365. PMLR, 2019.

---

[GM14] Sonal Gupta and Christopher D Manning, Improved pattern learning for boot- strapped entity extraction. In Proceedings of the Eighteenth Conference on Compu- national Natural Language Learning, pages 18-108, 2014.
[GNRV19] Vivek Gupta, Pegah Nokhiz, Chitradeep Dutta Roy, and Suresh Venkatasubra- manian. Equalizing recourse across groups. arXiv preprint arXiv: 03166, 2019.
[GZS+22] Ozgur Guldogan, Yuchen Zeng, Jy-yong Sohn, Ramtin Pedarsani, and Kangwook Lee. Equal improvability: A new fairness notion considering the long-term impact. arXiv preprint 2rx0vv2210.06732. 2022.
[HNG19] Hoda Heidari, Vedant Nanda, and Krishna Gummadi. On the long-term impact of algorithmic decision policies: Effort unfaimess and feature segregation through social learning. arXiv preprint arXiv:1903.0122: 20199
[HPS16] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. Advances in neural information processing systems, 29, 2016.
[HU20] Laura Hanu and Unitary team.
Detoxify.
Github.
https://github.. https:/github.com/uitaaa com/unitarraa//ddtooifff 2020.
[HWZW20] Wen Huan, Yongkai Wu. Lu Zhang, and Xintao Wu. Faimess through equality of
effort. In Companion Proceedings of the Web Conference 2020, pages 743-751, 2020.
[K+96] Ron Kohavi et al. Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid. In Kdd, volume 96, pages 202-207, 1996.
[KGZ19] Michael P Kim, Amirata Ghorbani. and James Zou. Multiaccuracy: Black-box post- post-processing for faimess in classification. In Proceedings of the 2019 AMAI/ACM Conference on AI, Ethics, and Society, pages 247-254, 2019.
[KL22] Nikola Konstantinov and Christoph H Lampert. Fairness-aware pac learning from corrupted data. The Journal of Machine Learning Research, 23(1):7173-7232, 2022.
[KLRS17] Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. Counterfactual fairness. Advances in neural information processing systems, 30, 2017.
[KOS11] David R Karger, Sewoong Oh, and Devavrat Shah. Iterative learning for reliable crowdsourcing systems. In Advances in neural information processing systems, pages 1953-1961, 2011.
[KS84] Martin Knott and Cyril S Smith. On the optimal mapping of Optimization Theory and Applications, 43:39-49, 1984.
mapping of distributions. Journal
[KSM+21] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[LLWT15] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[LVS] Hunter Lang, Aravindan Vijayaraghavan, and David Sontag. Training subset selection for weak supervision. In Advances in Neural Information Processing
Systems.
[MBSJ09]
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2- Voluume pages 1003-1011. Association for Computational Linguistics, 2009.
12

---

[MCR14]
Sérgio Moro, Paulo Cortez, and Paulo Rita. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62:22-31, 2014.
[MSY+21] Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukherjee. Hatexplain: A benchmark dataset for explainable hate speech detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 14867-14875, 2021.
[MW18] Aditya Krishna Menon and Robert C Williamson. The cost of fairness in binary classification. In Conference Fairness, accountability and transparency, pages 107-118 18. PMLR, 2018.
[ORDCR20] Luke Oakden Rayner, Jared Dummmon, Gustavo Careerr,, and Christopher Ré. Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In Proceedings of ACM conference health, inference, and learning, pages 151-159, 2020.
[PC+19] 19] Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applica- tions to data science. Foundations and Trends@ in Machine Learning, 11(5-6))355- 607, 2019.
[RBE+18] Alexander Ratner, Stephen H. Bach, Henry Ehrenberg, Jason Fries, Sen Wu, and Christopher Ré. Snorkel: Rapid training data creation with weak supervision. In Proceedings of the 44th International Conference on Very Large Data Bases (VLDB), Rio de Janeiro, Brazil, 2018.
[RHD+19a] A. Ratner, B. Hancock, J. Dunnmon, F. Sala, Pandey, and C. Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, Honolulu, Hawaii, 2019.
[RHD 19b] Alexander Ratner, Braden Hancock, Jared Dunnmon, Frederic Sala, Shreyash Pandey, and Christopher Ré. Training complex models with multi-task weak supervision. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 4763 4771, 2019.
[RKH*21] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishknn, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PMLR, 2021.
165W+16] A. J. Ratner, Christopher M. De Sa, Sen Wu, Daniel Selsam, and C. Ré. Data programming: Creating large training sets, quickly. In Proceedings of the 29th Conference on Neural Information Processing Systems (NIPS 2016), Barcelona, Spain, 2016.
[SDA+ Nimit Sohoni, Jared Dunnmon, Geoffrey Angus, Albert Gu, and Christopher Ré. No subclass left behind: Fine-grained robustness in paassegraained classification problems. Advances in Neural Information Processing Systems, 33:19339-19352 2020.
[2] Changho Shin, Winfred Li, Harit Vishwakarma, Nicholas Carl Roberts, and Frederic Sala. Universalizing weak supervision. In International Conference Learning Representations (ICLR), 2022.
[SMBN21] Nian Si, Karthyek Murthy, Jose Blanchet, and Viet Anh Nguyen. Testing group fairness via optimal transport projections. In International Conference on Machine Learning, pages 9649-9655 PMLR, 2021.
[SNS+21] Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz. Under- standing failures of deep networks via robust feature extraction. In Proceedings the the IEFE/VVF Conference on Computer Vision and Pattern Recognition, pages 12853-12862,, 1.21.
13

---

20] Chiappa Silvia, Jiang Ray, Stepleton Tom. Pacchiano Aldo, Jiang Heinrich, and Aslanides John. general approach to faimess with optimal transport. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3633-3640, 2020.
[Ver18] Roman Vershynin.
probability: An introduction with applica- tions in data science, volume 47. Cambridge university press, 2018.
[VS22] Harit Vishwakarma and Frederic Sala. Lifting weak supervision to structured prediction. In Advances in Neural Information Processing Systems, 2022.
[WCZC22] Renzhi Wu, Shen-En Chen, Jieyu Zhang, and Xu Chu. Learning hyper label model for programmatic weak supervision. In The Eleventh International Conference on Learning Representations, 2022.
[WFT+19] Julia K Winkler, Christine Fink, Ferdinand Toberer, Alexander Enk, Teresa Deinlein, Rainer Hofmann- Weelannoof,,,,ff,,,...m Lue Thomas, Aimilios Lallas, Andreas Blum, Wilhelm Stolz, et al. Association between surgical skin markings in dermoscopic images and diagnostic performance of a deep learning convolutional neural network for melanoma recognition. JAMA dermatology, 155(10): 135-:11111111111 2019.
[WGH+22) Songhua Wu, Mingming Gong, Bo Han, Yang Liu, and Tongliang Liu. Fair classifi- cation with mustance-dependee label noise. In Conference on Causal Learning and Reasoning, pages
PMLR, 2022.
[WH22] Ziwei Wu and Jingrui He. Fairness-aware model-agnostic positive and unlabeled learning. In ACM Conference on Fairness, Accountability, and Transparency (ACM FAccT), 2022.
[WLL21]] Jialu Wang, Yang Liu, and Caleb Levy. Fair classification with group-dependent label noise. In Proceedings of the 202 ACM conference on fairness, accountability, and transparency, pages 526-536, 2021.
[WZN+23] Jiaheng Wei, Zhaowei Zhu, Gang Niu, Tongliang Liu, Sijia Liu, Masashi Sugiyama, and Yang Liu. Fairness improves learning from noisily labeled long-tailed data. arXiv preprint arXiv:2303. 2023.
[ZDC22] Xianli Zeng, Edgar Dobriban, and Guang Cheng. Bayes-optimal classifiers under group fairness. arXiv preprini arXiv:2202.097724 2022.
[2HY+22] Jieyu Zhang, Cheng- Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey on programmatic weak supervision. arXiv preprint arXiv:2202.05433. 2022.
[7SQ17] Zhifei Zhang, Yang Song, and Hairong Qi. Age progressionnregression by con- ditional adversarial autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2017,
[ZSR23] Jieyu Zhang, Linxin Song, and Alex Ratner. Leveraging instance features for label aggregation in programmatic weak supervision. In International Conference on Artificial Intelligence and Statisties, pages 157-171. PMLR, 2023.
[ZYL+21]] Jieyu Zhang, Yue Yu, Yinghao Li, Yujing Wang, Yaming Yang, Mao Yang, and Alexander Ratner. Wrench: A comprehensive benchmark for weak supervision. arXiv preprint arXiv:2109.
2021.
[ZZL+ +33 Yixuan Zhang, Feng Zhou, Zhidong Li, Yang Wang, and Fang Chen. Fair repre- sentation learning with unreliable labels. In International Conference on Artificial Intelligence and Statistics, pages 4655-4667. PMLR, 2023.
14

---

Appendix
The appendix contains additional details, proofs, and experimental results. The glossary contains convenient reminder of our terminology (Appendix AA) Appendix provides more related works and discussion about the relationship between our work and related papers. In Appendix we describe the details of our algorithm and discuss their implementations. Appendix DirixxxDpprre provides the proofs of theorems that appeared Section Finally, we give more details and analysi sis of the experiments and provide additional experiment results.
C
A Glossary
The glossary given in below.
Symbol
Definition
X
Feature space
y
Label metric space
Latent space of feature space
A
Group variable, assumed to have one of two values {0, 1) for simplicity Inputs from group k.
X
X
Inputs from all groups & E {1.....!)
Y
True labels from group k: True labels from all groups k {1,....}} Probability distribution in the latent space Probability distribution in the group
Y
P
Ps
Noisy labels of LF j of group k. it used with input (e.g. (x(x)),
it denotes LF j from group k such that its outputs are noisy labels
Noisy labels of LF j of all groups & E [I.....[} If used with input (eg. (x)), it denotes LF such that its outputs Collection of noisy labels from group k. A₁- [A] A
A
noisy labels
A
Collection noisy labels from all groups k (1,.....)) A= "....
gl
2 - X, k-th group transformation function
hx
hx: Z the inverse transformation of Ok. i.c. hagg (2) (x)== for
By
Prior parameter for Y in label model
0j
Accuracy parameter for X in label model Accuracy parameter
æ of X label model [CA++2
Accuracy of LF in group k, a = [[
a
Accuracy of LF j. a Y
Estimates of a=.a²
Mean of features in group k
Ex
Covariance features group
Identity transformation, i.e. [(x) =
(r(E)
Trace of Σ
Amax (2) Amin (2))
Maximum, minimum values of 2
r(E)
Effective rank of Σ, i.c r(E)= = Ammax (2)
tr (r(E)
Table 6: Glossary of variables and symbols used in this paper.
B Extended related work
Weak supervision In weak supervision, we assume the true label Y E E cannot be accessed, but labeling functions X ?!.2?...,,X? E y which are noisy versions of true labels, are provided. These weak label sources include code snippets expressing heuristics about Y. crowdworkers, external knowledge bases, pretrained models, and others [KOST] MBSI09 GM14 DZS+ RBE+ 18 Given _ ".., m WS often takes a two-step procedure to produce an end model (DS79 RSW 16 20
RHD 19b
22 VS22). The first step, the label model, obtains fine-grained pseudolabels by
15

---

modeling accuracies and correlations of label sources. The second step is training or fine-tuning the end model with pseudolabels to yield better generalization than the label model. Not only can our method improve performance and fairness, but it is also compatible with the previous WS label models, as it works at the label source level perlo-proorree to the label model.
ZSR23J. The first work, (LVS), uses embeddings for subset selection, where high-confidence subsets are selected based on the proximity to the same pseudolabels. The second, [CFA+2] 22), exploits embeddings to estimate local accuracy and improve coverage. The third [SR23] also incorporates instance features into label model via Gaussian process and Bayesian mixture models. Similarly, our method uses embeddings to improve fairness by mapping points in different groups in the embedded space. Embedding spaces formed from pre-trained model typically offer better distance metrics than the input space, which provides an advantage when modeling the relationships between input points.
Another related line of work is WS using embeddings of inputs LVS CFA
Fairness in machine learning Fairness in machine learning is an active research area to detect and address biases in ML algorithms. There have been many suggested faimess (bias) notions and solutions for them [DHP HPS KLR.I7 HNGI9 GNRV 19, HWZW GZS+ 22). Popular approaches include adopting constrained optimization or regularization with fairness metries [DHP* 12 HPST6
ABD 18 HNG19 GNRVI9 HWZW20] and postprocessing model outputs to guarantee fairness
notions MMW18 MMW12ZDC22] While they have been successful in reducing bias, those methods often have
a fundamental tradeoff' between accuracy and fairness.
more recent line of works using Optimal Transport (OT) GDBFLI9 BYF20, 20 SMBN21
BDB22] has shown that it is possible to improve fairness without such tradeoffs by uniformizing distributions in different groups. Our method improves fairness and performance simultaneously by using OT. However, OT-based methods can suffer from additional errors induced by the optimal transport mapping. In our method, OT-induced errors that occur at the label sources can be covered by the label model, since those labeling functions that are made worse are down- weighted in label model fitting. limitation of our method is that its resulting fairness scores are unpredictable in advance, while methods having control this tradeoff. However, as we argued in Section[5] our method is compatible with such methods, so that it is possible to control the fairness of the end model by adopting other methods as well.
tradeoffs typically come with tunable parameters to a priori
Another related line of research is fairness under noisy labels [WLL21 KL.22 WZN+ 23 ZZL. +3)
[WLL21 WGH 22) suggests new loss function that considers the noise level in labels to improve fair- ness. KL.22| studies PAC learning under a noisy label setting with fairness considerations. [WZN* 23)
introduces regularizer to boost fairness under label noise, focusing on the performance difference in subpopulations. Finally, 223 also proposes a regularizer based on mutual information to improve fairness of representations under label noise. While our method shares the assumption of label noise, the main difference is that we have access to multiple noisy estimates-wii learn-and where we can apply fairness notions to each separate estimate. In contrast, other methods consider only one noisy label source-wuie in weak supervision. Thus, their methods can be used additionally with SBM, like other previous fair ML methods such as optimal threshold. Also, the typical weak supervision pipeline does not include training for individual label sources that techniques involving optimizing loss functions and regularization do not fit this setting.
noise rates we can
corresponds to pscudolabels generated by label models
Data slice discovery Data slice denotes group of data examples that share a common attribute or characteristic (EVS*22). Slice discovery aims to discover semantically meaningful subgroups from unstructured input data. Typical usage of slice discovery is to identify data slices where the model underperforms. For example, IDVMWVdMID analyzed off-the-shelf object recognition models' performance with additional context annotations, revealing that model performance can vary depending on the geographical location where the image is taken. [WFT+ 19] shows that the melanoma detection models do not work well for dermascopic images with skin markings since they use those markings as a spurious feature. [ORDCR20] found pneumothorax classification models in chest X-rays underperform for chest x-ray images with chest drains. To discover such data slices without additional annotations, a
variety of strategies have been employed. [ORDCR20 20, SDA + 20 EVS+ 22) used clustering or mixture modeling with dimensionality reduction such as U-MAP ISDA + 20 SVD (2)++22) [dWLLB22
solves optimization problem to find high loss data points given a subgroup size.
16

---

While typical slice discovery methods require access to true labels to find underperforming data slices, weak superivison setting does not allow it. To work this around, we replace true labels with pseudolabels generated by label model. And, slice discovery methods are applied for outputs of each L... Thus, the discovered data slices represent data points where each I.F disagrees with the aggregated pseudolabels.
C
Algorithm details
A1
A₂
A3
Label model
End model training
Source Bias Mitigation (SBM)
Pseudo labels
Figure 5: Overall WS pipeline involving SBM
Algorithm 2: ESTIMATE ACCURACY (TRIPLET) FCS+20
Parameters: Weak label values A "....!"" for iijkk (1.2.....m) (i k)) do
la + BANJJENN//BN
l
|a* (RINANNE([NN
end for return ResolveSign (a')
Vie (1,2....,
Algorithm 3: TRANSPORT
Parameters: Source input X destination input Xdst, source weak label values Agre- destination weak label values Adsts Optimal Transport type O, the number of nearest neighbors k= OT type is linear then
Xare < LINEAROTT (o.OT((sss.... Xast
KS84
else if OT type O is sinkhom then
X SINNIOOROTT (X (ase)
Xast)
3
else
Xsret Xare end
X 3 A-NNN M.NNN(KereNNsttrdaa..
return Ngre
In this appendix section, we discuss the details of the algorithms we used. The overall WS pipeline including SBM is illustrated in Figure 5 Our method is placed in the first step refinement of
17

---

Algorithm 4: LINEAROT (OT-L)(KS84)
Parameters: Source input X stcy destination input Xdst
Ms-f4 <- MRAN X (.ANN((X.).MAAAA ,.RAN (VFAN(XXss) dst)
Egg C-v--CoVXX sre CoC dsttt
A Σg (x! Ex 1/2 /
b- be-14 144 Apts
Mere 5 size (x
for E [1,2,.., do
X 1] AX areeeel [i] end for return Xare
Algorithm 5: SINKHORNOT (OT-S) Cut
Parameters: Source input Sses destination input Xdst, Entropic regularization parameter 17=1 M + PAIRWISEDISTANCEL (ereeeea
Rrss f- size (X
size ((st)
a+ (1/1sreyr-..)) 1/1sre))
Rure
be- (1/Mdst,:.... /nastt E "aR
Set K E Xidat such that Sxy=exxp-MMih1 MRad Initialize y= (1,...,, [R.
while not converged do
u. K V
0 KTu a end while
# n
for (1....1..... do
8 Xsreli]
[1] Σj May Nastli]
end for return X
noisy labels. In this step, improve the noisy labels by transporting the low accuracy group the high accuracy group. To identify low accuracy and high accuracy groups, we estimate accuracy using Algorithm After estimating accuracies, we transport the low accuracy group the high accuracy group and get refined labels for the low accuracy group by Algorithm Linear OT (Optimal Transport) used to estimate a mapping. Sinkhorn OT approximates optimal coupling using Sinkhorn iteration. After obtaining the optimal coupling, data points from the low accuracy group are mapped to the coupled data points in the high accuracy group. OT type is not given, data points from the low accuracy group are mapped to the 1 --naarest neighbor in the high accuracy group. After obtaining mapped points, weak labels associated with mapped points in the target group are used for the data points from the low accuracy group.
The step standard weak supervision pipeline step - training a label model and running inference to obtain pseudolabels. Finally, we train the end model. We used the off-the-shelf label model Snorkel [BRL+ 19] for the label model and logistic regression the end model in all experiments unless otherwise stated. For the optimal transport algorithm implementation, we used the Python POT package 21)
D Theory details
Accuracy is defined as P(\(x) = y) P(A(x)=1,y) + P(A(x) = -1,y= = =1). For ease of notation, below set Φ(z centero = (1+ 2
18

---

D.1 Proof of Theorem 4.1
(k) be an arbitrary sequence of functions such that
Theorem
4.1
Let
E " (x) M| x xemmeerl |I] + cc. Suppose our assumptions above are met; in par-
ticular, that the label is independent of the observed x (() or o' (k) (2), (z),k,, conditioned on the latent features z. Then,
lim E (2) (P(A(=')=))) = 2
which corresponds random guessing.
Proof Theorem
4.1
Because limg B E0 (2) a ||| + co, we have
lim k+0 ''Ent (2))) (0)( ,26 cccnttro)]]
and because Z (o. P(Aeetoo..P((rr P(A(x')= -,, -))
lim E
((((z))
lim
k0000 r'Egl (z)
7 exp()o(1)(1)o(= (e (00pmm)))==x00000100(1111 - (
lim see (k) (2)
k0000
lim e'egl (2)
Z exp(000(x ceemerr))
k-000
2exp (p o(x' pereeeee ( 2 (exp(Ondi x' pentter )) expl Opllla Coneaart
lim E (1) (2)
k-000
exp(6 (2' _
lim E E-+00
eleg) (2)
exp(00ol xx ppeenlero )) texp( - 000(2' .reemtrr)
1
since we get
lim E Ra (2)) exp(0rd(( (xx i senter))) )) =1
k000
k00 "'' (2) exp( 0no(a (a' centerr)) )) =1
lim E
from lim eeg (2) 10(2 aa contero) =0
D.2 Proof of Theorem 4.2
Theorem 4.2 Let T max
r(En) r(E n2
22 and C be constant. Using
nu
mmin (n) max((o,..)
Algorithm for any t 0, we bound the difference
S48.C
with probbbiliity 1 -1- m1
We use the result from [LFF19] which bounds the difference of the true h and empirical Monge estimators between distributions Po(x)) and Pi (a'). Let Let r(E), Amin (E) and >max (E) denote the effective rank, minimum and maximum eigenvalues of matrix 2 respectively. Then,
Lemma D.1 (FLF19)) ). Let Po(z) and (x') be nab-Gaussian distributions X = with expecta- tions fou and positive-definie covariance operators En respectively. We observe points from the distribution (x) and nn points from the distribution Pi (a'). ). We assume that
C1 < min Amin (Ex) S max Amax (20) S S02, (E(0,1)
JE(0,1)
19

---

fixed constants 0 <1 S2 oc. Further, we assume no 2 (TE) and n1 2 for sufficiently large constant e>> Then, any 1>> O, we have with probabilit at least -e- mi
that
Exexx (a(x'')h(aaa 1 SCVTT(I))
where
constant independent of no. ni, d, r(Zo), r(Σi) and C a
r(Zo] r(En) (e.
max no n1
We will also use the following:
Lemma D.2. The probability P((()) = y) is L-Lipschitz with respect to x E X. Specifically,
Ex
IP({(n)==)) - P(((2)) [00 |
Proof. We will demonstrate that, because (V,P(((())) = is bounded above by 40,, P({(x)= must be 480-Lipschitsz with respect to x. First,
exp (5p0006x1,zzemeee
P(M()) = v) exp(000 (1, )) + +exp( 000 (2,
1 e+exp( -20000000000 (1
= (20 (2,,200 ))
where o denotes the sigmoid function.
expl u)) Note that du (u) [Itexpl w) for a R and 2
V. x- panre
that v (20 ]eemeto)] = 20 V
200
Further, note that
(1 ganan
V Vellæ- centero gamto
because use Euclidean norm. Thus,
(V)P(())==))) =
V. centert)) )]|l
exp( 20 (x, (
20.
V centera
(1+exp(- -2:ppp2-0ppp0 ccenteo 1))2(1+ - centero ||)
1pp0-220000 cenneer))
(oeeeaee )²
200
(1+exp((-2) -200p (1r,
22000(1,, comeer) )) - (20 ( centeen ))) (. 20
Coaaaaaaiiirr contero Dommiee 7
centero
20
1- 1-(1-(-1))d(r )2
zeeterro
4 (0n,r, }2.
< 40o.
Thus, (((x((<)) = bounded above by 40g. We now use this fact to demonstrate that P(X(x)= = y) is -ip-hiis with respect to æ.
For 0 <. 1, let s(v) = - (I2 (22--11) m.... For between 21 and 22 inclusive and for such that s(v) = since / V
46p, we have
d ((A(s(())=z) || [Is())]loo
du ((((s()))=z) [|s()]ldd
<
0= do
5 /" 40o [00]ls(v)]ldo
400 |||||1 x2
20

---

Proof of Theorem
Now we are ready to complete our proof. We have,
B Deee
< < ( (by Lemma D.2
E /40 -
< <480CVTT(En)
where the last line holds with probability at least 1- f 1 by Lemma D.1
E Experiment details
E.1
Dataset details
Table 7: Summary of real datasets
Domain Datasets
P(Y 1|A-1) P(Y 1
ADP
Adult
Income (2 50000)
Sex
0.3038
0.1093
0.1945
Tabular
Bank
Suhscription
Age (2 25)
0.1093
0.2399
0.1306
CivilComments
Toxicity
Black
0.3154
0.1057
0.2097
NL.
HateXplain
Toxicity
Race
0,8040
0.4672
0.3368
CelehA
Gender
Young
0.3244
0.6590
0.3346
Vision
UTKFace
Gender
Asian
0.4856
0.4610
0.0246
Adult The adult dataset (K+ has information about the annual income of people and their demographics. The classification task is to predict whether the annual income of a person exceeds $50,000 based demographic features. Demographic features include age, work class, education, marital status, occupation, relationship, race, capital gain and loss, work hours per week, and native country. The group variable is whether age is greater than 25 not. The training data has 32,561 examples, and the test data has 16,281 examples.
BankMtrn The bank marketing dataset [MCR14 was collected during the direct marketing campaigns of a Portuguese banking institution from 2008 to 2013. Features consist of demographic attributes, financial attributes, and the attributes related to the interaction with the campaign. The task to classify whether a client will make a deposit subscription or not. The group variable is, whether age is greater than 25 or not. The total number of instances is 41,188, where the training dataset has 28,831 rows and the test dataset has 12,357 rows.
The CivilComments dataset IBDS+ constructed from data collected in Jigsaw online conversattion platform. We used the dataset downloaded from [KSM 21]. Features are extracted using BERT (DCLTI8) embeddings from comments, and the task is to classify whether given comment is toxic or not. This dataset has annotations about the content type of comments, e.g. race, age, religion, etc. We chose "black" as the group variable. The total number of instances 402,820, where the training dataset has 269,038 ,038 rows and the test dataset has 133,782 rows.
CivilComments
The HateXplain dataset M is comments collected from Twitter and Gab and labeled through Amazon Mechanical Turk. Features are extracted using BERT embeddings from comments. The task classify whether given comment is toxic or not. This dataset three label types hate, offensive, normal. We used hate and offensive label types as "toxic" label, that is, if
HateXplain
21

---

one of two labels types is I for a row, the row has label 1. The dataset has annotations about the target communities race, religion, gender, LGBTQ. We used the race group as the group variable. The total number of instances is 17,281, where the training set has 15,360 rows and the test set has 1,921 rows.
CelebA The CelebA dataset [LWT the image datasets that has images of faces with various attributes. Features are extracted using CLIP(RKH+11 embeddings from images. We tried gender classification task from this -vrisous other tasks can be constructed. We used "Young" as the group variable. The total number of instances is 202,599, where the training set has 162,770 rows and the test set has 39,829 rows.
UTKFace UTKFace dataset [ZSQ17 consists of faces images in the wild with annotations of age, gender, and ethnicity. Features are extracted using CLIP embeddings from images. This dataset has attributes of age, gender, race and we chose gender label (Male:0, Female: 1), solving gender classification. We used the race as the group variable, encoding "Asian" group as other race groups as O. The total number of instances is 20,7 where the training set has 18,964 rows and the test set has 4,741 rows.
E,2 Application of LIFT
To use smooth embeddings in the transport step of tabular datasets (adult and bank marketing), we convert tabular data rows into text sentences and then get BERT embeddings.
Adult First decided rpronoun for each row r from <rsex> and then generated sentences with the following template. "<r,pronoun>> is in her <rageerange>s. <r.pronou>> is from <r.nationaltyy> <r.pronoun> works in workllsssss. Specifically, <r.pronoun> has <r.job> job. <T.pronoun> works <r. working hour> hours per week. <r.pronoun> total education year is <r.education_eears <r.pronoun> is <r.marital status>. <r.pronoun> <r.race>".
An example sentence is "She is in her 20s. She is from United-States She works in private sector. Specifically, she has sales job. She works 30 hours per week. Her total education year is 6 years. She is married. She White."
Bank marketing Similarly, text sentences for each row generated using the following template.
"This person is <ragge> years old. This person works in <r.job> job. This person L.marital_statuss This person has <r.education degree> degree. This person <if rhas housing loan "has" else "doesn't 't have"> a housing loan. This person <if r.has satassssesonall... loan "has" else "doesn't have"> a personal loan. THis person's 's contact is <rcontact type>. The last contact in <r.last_contaet_mmone performed during this campaign. Before this campaign, <r.previous> times of contacts have been performed for this person. The employment variation rate of this person is <r.emp_ var var_rate>%% %. The consumer price index of this person <r.cons _prnne____ee_____ lel___pcccc_eeeeeeeeeeeeeeeeeeee __xx__.xx..... The consumer confidence index of this person is <r.cons cectossconn____ con__idxx% The euribor 3 month rate of this person is <r.curibor3m>. The number of employees is senr_eempoyeed>..
<r.last contact weekday> The call lasted for <r.duration> seconds. <r.campaign> times of contacts
An example sentence is "This person is 37 years old. This person works in a management job. This person is married. This person has university degree. This person doesn't have housing loan. This person has a personal loan. This person's 's contact is a cellular. The last contact was on Thursday in May. The call lasted for 195 seconds. 1 times of contacts performed during this campaign. Before this campaign, O times of contacts have been performed for this person. The employment variation rate of this person is 11.8% The consumer price index of this person is 92.893%. The consumer confidence index of this person is -46.2%. The euribor 3 month rate of this person is 1.327%. The number of employees is 5099."
E.3 LF details
In this subsection, we describe the labeling functions used in the experiments. The performances of labeling functions, including our SBM results, are reported in Table 19 In SBM result tables, performance, fairness improvement by SBM are green colored.
22

---

Adult
We generated heuristic labeling functions based on the features as follows.
LF (age L.F): True if the age is between 30 and 60. False otherwise.
LF 2 (education LFF: True if the person has a Bachelor, Master, PhD degree. False otherwise
LF (marital LF): True if marital status is married. False otherwise.
LF (relationship LF): True if relationship status is Wife, Own-child, or Husband, False otherwise.
LF 5 (capital LF): True if the capital gain 55000. False otherwise.
LF (race LF): True if
or mace-Other, False otherwise.
LF 7 (country LF): True if Germany, Japan, Greece, or China. False otherwise. LF (workclass LF): True the workclass Self employed, federal government, local government, or state government. False otherwise.
LF (occupation LF): True the occupation sales, executive managerial, professional, or machine operator. False otherwise.
Bank marketing Similar to Adult dataset, we generated heuristic labeling functions based on the features follows.
LF (loan L.F): True if the person has loan previously. False otherwise.
LF 2 (previous contact LF)): True if the previous contact number is >1.1. False otherwise.
LF (duration LF): True if the duration of bank marketing phone call min. False otherwise.
LF 4 (marital LF): True marital status is single. False otherwise.
LF 5 (previous outcome LF): True if the previous campaign successful. False otherwise.
LF (education LF); True the education level is university degree or professional course taken. False otherwise.
CivilComments We generated heuristic labeling functions based on the inclusion of specific word lists. If' comment has word that included in the word list of LF, gets True for that LF.
LF 1 (sex LF); ["man", "men", "woman", "women", "male", "female", "guy", "boy", "girl", "daughter", "sex", , "ggender, "husband", "wife", "father", "mother", "rape", "mr.", "feminist", "feminism", " "pregnant", "misogy", "pussy", "penis", " "vagina", "butt", "dick"]
LF 2 (F) ["gay,, "lesbian", "transgender", "homosexual", "homophobic", "hetero- sexual", "anti- "anti-gay", "same-sex", "bisexual", "biological"]
LF 3 (religion LFF): ("muslim", "catholic" "church", "christian", "god", "jesus", "christ", "jew". "islam", "israel", "bible", "bishop". "gospel". "clergy", "protestant", "islam"] LF 4 (race LF): ("white", "black", "racist", "trump", "supremacist", "american", "canada", "kkk", "nazi", "facist", "african", "non-white", "discrimination" "hate", "neo-nazi", "asia", "china", "chinese"]
LF 5 (toxic LF): ("crap", "damn", "bitch", ; "ass", "fuck" " "bullshit", "hell", "jerk"] LF (threat LF): ("shoot", "kill", "shot", "burn", "stab", "murder", "gun", "fire", "rape". "punch", "hurt", " "hunt", "bullet", "hammer"]
LF7 (insult LF): ["stupid", "idiot". "dumb". "liar", "poor", "disgusting". "moron", "nasty", "lack", "brain". "incompetent", "sociopath"]
HateXplain We used heuristic the same labeling functions with CivilComments. However, their performance was close to random guess (accuracy close to 0.5), we added 5 pretrained model LFs from detoxify [HU20) repository. We used models listed as original, unbiased, multilingual, original-maal, unbiased-small
CelebA We used models pretrained from other datasets as LFs.
LF I: ResNet-18 fine-tuned gender classification dataset
LF ResNet-34 fine-tuned on FairFace dataset
LF 3: ResNet-34 fine-tuned UTKFace dataset
LF 4: ResNet-34 fine-tuned on UTKFace dataset (White only)
LF 5: ResNet-34 fine-tuned UTKFace dataset (non-White only)
LF 6: ResNet-34 fine-tuned on UTKFace dataset (age 2 50 only)
23

---

Table 8: Tabular dataset raw LF performance
Dataset
LF
Acc
F1
App
A Ago
LF (age LF)
0.549
0.476
0.100 0.023
LF 2 (education LF)
0.743
0,455 0.033 0.044
Adult
LF (marital LF)
0.699 0.579 0.447 0.241
LF 4 (relationship LF)
0.564 0,486 0.381 0.243
LF (capital L.F)
0.800 0.315 0.035 0.019
LF 6 (race LF)
0.737 0.066 0.003 0.004
LF (country L.))
0.756 0.024 0.001 0.004
LF 8 (workclass LF)
0.678 0.399 0.066 0.003
LF (occupation LF)
0.644 0.466 0.013 0.012
LF (loan LF)
0.769 0.124 0.004 0.013
LF (previous contact LF)
0.888 0.187 0.055 0.068
Bank Marketing
LF (duration LF)
0.815 0,415 0.019 0.125
LF (marital LF):
0.682 0.197 0.602 0.643
LF 5 (previous outcome LF)
0.898
0.301 0.052 0.062
LF (education LF)
0.575 0.206 0.183 0.219
LF 7: ResNet-34 fine-tuned on UTKFace dataset (age < 50 only)
UTKFace We used models pretrained from other datasets LFs.
LF 1: ResNet-18 fine-tuned on gender classification dataset
LF 2: ResNet-34 fine-tuned gender classification dataset
LF 3: ResNet-34 fine-tuned on CelebA dataset
LF 4: ResNet-34 fine-tuned CelebA dataset (attractive only)
LF 5: ResNet-34 fine-tuned on CelebA dataset (non-attractive only)
LF ResNet-34 fine-tuned CelebA dataset (young only)
LF 7: ResNet-34 fine-tuned on CelebA dataset (non-young only)
LF& 8: ResNet-34 fine-tuned CelebA dataset (unfair sampling)
24

---

Table Tabular dataset SBM (w/o OT) LF performance. As are obtained by comparison with raw LF performance.
Dataset
LF
Acc (A)
F1 (A)
Apr (A)
Ago (4)
LF 1 0.597 (0.048)
0.594 (0.118)
0.119 (0.019)
0.106 (0.083)
LF2 0.601 142)
0.346 (-0.109)
0.035 (0.002)
0.019 (-0.025)
Adult
LF3 0.998 (0.299)
0.998 (0.419)
0-0.
0.001 (-0.240)
LF4 0.719 (0.155)
0.723 (0.237)
0.004
0.091 (-0.152)
LF5 0.649 (-0.151)
0.183
0.035
0.023 (0.004)
LF6 0.615 122)
0.082 (0.016) 0.003 (0.0000
(0.025)
LF7 0.621 (-0.135)
0.023 (-0.001)
0.001 (0.000)
00.000)
LF8 0.612 (-0.066)
0.318 (-0.021)
(-0.047))
0.027 (0.024)
LF9 0.565 (-0.079)
0.460 (-0.006)
0.013 (0.000)
0.026 (0.014)
LF 0.506 (-0.043)
0.415
0.309 (0.209)
0.198 (0.175)
LF2 0.861 (0.118)
0.691 (0.236)
0.080 (0.047)
(0.082)
Adult (LIFT)
LF3 0.691 (-0.008)
0.188 (-0.391)
0.0.31113117
0.186
LF4 0.391 (-0.173)
0.307 (-0.179)
(0.070)
(0.14
LF5 0.725 (-0.075)
0.117
0.029 (-0.22))
(0.010)
LF6 0.703 (-0.034)
0.109
0.003 (0.000)
0.004
LF7 0.708 (-0.048)
(0.012)
0.001 (0.000)
0.0.0000.004
LF8 0.897 (0.219)
(0.4611
0.033 (-0.033)
(0.232)
LF9 0.601 (-0.043)
0.013 (-0.022)
(0.000)
0.068 (0.056)
LF1 0.706 (-0.063)
065 (0.041)
0.004 (0.000)
0.003 (-0.010)
LF2 0.824 (-0.064)
0.221 (0.034)
0.037 (-0.018)
0.113 (0.045)
Bank
LF3 0.931 (0.116)
0.829 (0.414)
0.019 (0.000)
0.110 (-0.015)
LF4 0.427 (-0.255)
0.411 (0.214)
0.100 (-0.502)
0.004 (-0.639)
LF5 0.833 (-0.065)
0.287 (-0.014)
0.061 (0.009)
0.195 (0. 133)
LF6 0.624 (0.049)
0.172 (-0.034)
0.009 (-0.174)
0.046 (-0.173)
LFI 1 0.768 (-0.001)
105 (-0.019)
0.000 (-0.003)
0.013 (0.000)
LF2 0.888 (0.000)
0.188 (0.001)
0.037 (-0-017)
0.068 (0.000)
Bank (LIFT)
LF3 0.815 (0.000)
0.415 (0.000)
0.019 (0.000)
(0.000)
LF4 0.317 (-0.365)
0.228 (0.031)
0.100 (-0.502)
0.065 (-0.578)
LF5 0.898 (0.000)
0.303 (0.001)
0.061 (0.009)
0.086 (0.023)
LF 6 0.680 (0.105)
0.126 (-0.081)
0.009 (-0.175)
0.0.4
25

---

Table 10: Tabular dataset SBM (OT-L) performance. As are obtained by comparison with raw LF
performance.
Dataset
LF
Acc (A)
F1 (A)
ADP (A)
AEO (A)
LF 1 0.841 (0.292)
0.846 (0.370)
0.654 (0.554)
0.733 (0.710)
LF2 0.334 (-0.409)
0.000 (-0.455)
0.208 (0.175)
Adult
LF3 0.346 (-0.353)
0.000 (-0.579)
0.000 (-0.273)
LF4 0.895 (0.331)
0.904 (0.418)
0.735 (0.354)
0..2480
LF5 0.394 (-0.406)
0.027 (-0.315))
(-0.008)
(-0.019)
LF6 0.408 (-0.329)
0.070 (0:004)
0.003 (0.000)
0.037 (0.033)
LF7 0.405 (-0.351)
(-0.00))
0.001 (0.000)
(0.006)
LF8 0.336 (-0.342)
0.000 (-0.339)
0.202 (0.136)
(-0.003)
LF9 0.501 (-0.143)
0.512 (0.046)
0.013 (0.000)
(0.427)
LF I 0.628 (0.079)
0.655 (7.179)
0.036 (-0.064)
0.027 (0.004)
LF2 0.780 (0.037)
0.664 (0.209)
(-0.019)
0.032 (-0.012)
Adult (LIFT)
LF3 0.525 (-0.174)
0.296 (-0.283)
0.097 (-0.350)
0.013 (-0.228)
LF4 0.497 (-0.067)
0.554 (0.068)
0.137 (-0.244)
0.150 (-0.093)
LF5 0,606 (-0.199)
0.177 (-0.138)
0.023 (-0.012)
0.042 (0.023)
LF6 0.565 (-0.172)
0.089 (0.023)
0.003 (0.000)
0.008 (0.004)
LF7 0.567 (-0.189)
0.029 (0.005)
0.001 (0.000)
(-0.000)
LF8 0.618 (-0.060)
0.409 (0.070)
0.011 (-0.055)
0.017 (0.014)
LF9 0.843 (0.199)
0.817 (0.351)
0.013 (0.000)
0.046 (0.034)
LF1 0.818 (0.049)
0.062 (-0.062)
0.004 (0.000)
0.140 (0.127)
LF2 0.980 (0.092)
7.703 (0.516)
0.0.4330.031
0.542 (0.474)
Bank
LF3 0.777 (-0.038)
0.093 (-0.322)
(0.000)
0.265 139)
LF4 0,047 (-0.635)
0.083 (-0.114)
0.131 (-0.471)
1.000 (0.357)
LF5 0.988 (0.090)
0.839 (0.538)
0.032 (-0.020)
0.723 (0.660)
LF6 0.950 (0.375)
0.000 (-0.206)
0.244 (0.061)
0.000 (-0.219)
LF 0.125 (-0.645)
0.197 (0.073)
0.853 (0.849)
0.867 (0.854)
LF2 0.888 (0.000)
0.0.000
0.000 (-0.055)
0.036 (-0.032)
Bank (LIFT)
LF 0.815 (0.000)
0.415 (0.000)
0.018 (-0.001)
0.137 (0.012)
LF 4 0.876 10.1
0.085 (-0.112)
0.869 (0.268)
0.954 (0.311)
LF5 0.898 (0.000)
0.300 (-0.002)
0.047 (-0.005)
0.039 (-0.023)
26

---

Table 11: Tabular dataset SBM (OT-S) LF performance. As are obtained by comparison with raw LF performance.
Dataset
LF
Acc (A)
FI (A)
APP (A)
Ago (4)
LF I 0.604 (0.055)
0.595 (0.119)
0.145 (0.045)
0.125 (0.
LF2 0.601 (-0.142)
0.340 (-0.115)
0.036 (0.003)
0.015 (-0.029)
Adult
LF3 0.998 (0.299)
0.998 (0.419)
0.293 (-0.154)
0.002 (-0.239))
LF4 0.690 (0.126)
0.700 (0.214)
0.221 (-0.160)
(-0.99))
LF5 0.656 (-0.144)
0.181 (-0.134)
0.032 (-0.003)
0.021
LF6 0.621 (-0.116)
0.081 (0.015)
0.003 (0.000)
(0.00))
LF7 0.627 (-0.129)
0.023 (-0.001)
0.001 (0.000)
(0.000))
LF8 0.607 (-0.071)
0,256 (-0.083)
(0.003)
(..999)
LF9 0.558 (-0.086)
0.446 (-0.020)
0.013 (0.000)
0.010
LFI 0.475 (-0.074)
0.436 (-0.040)
0.031 (-0.069)
0.032 (0.009)
LF2 0.953 (0.210)
0.910
0.047 (0.014)
(0.035)
Adult (LIFT)
LF3 00.02
0.191 (-0.388)
0.157 (-0.290)
0.250
LF4 0-0.199011111 0.425 (-0.061)
0.204
0-0.0
LF5 0.764 (-0.036)
0.301 (-0.014)
0.035 (0.000)
0.107 (0.088)
LF6 0.711 (-0.026)
0.117
0.003 (0.000)
0.018 (0.014)
LF7 0.715 (-0.041)
0.037 (0.013)
0.001 (0.000)
0.002 (-0.002)
LF8 0.716 (0.038)
0.272 (-0.067)
0.147 (0.081)
0.284 (0.2811
LF9 (-0.00000004
0.495 (0.029)
0.013 (0.000)
0.213 (0.201)
LF1 0.694 (-0.075)
0.173 (0.049)
0.004 (0.000)
0.017 (0.004)
LF2 0.806 (-0.082)
0.209 (0.022)
0.059 (0.004)
0.211 (0.143)
Bank
LF3 0.949 (0.134)
0.881 (0.466)
0.019 (0.000)
0.136 (0.011)
LF4 0.351 (-0.331)
0.382 (0.185)
0.040 (-0.562)
0.016 (-0.627)
LF5 0.814 (-0.084)
0.267 (-0.034)
0.069 (0.017)
0.247 (0.
LF6 0.583 (0.008)
(-0.
0-0.0
0.103 (-0.116)
LF1 0.769(-0.001)
0.087 (-0.038)
0.006 (0.003)
0.038 (0.025)
LF2 0.888 (0.001)
0.204 (0.017)
0.141 (0.087)
0.305 (0.237)
Bank (LIFT)
LF3 0.814 (-0.001)
0.409 (-0.006)
0..5430.035)
0.357 (0.231)
LF4 0.307 (-0.375)
0.229 (0.032)
0.085 (-0.517)
0.042 (-0.600)
LF5 0.898 (0.000)
0.311 (0.010)
0.133 (0.081)
0.236 (0.173)
6F6 0.677 (0.102)
0.105 (-0.102)
0.003 (-0.180)
0.122 (-0.096)
27

---

Table 12: NLP dataset LF performance
Dataset
LF
Acc
F1
ADP
Ago
LF (sex LF)
0.755 0.187 0.046 0.019
LF (LGBTQ LF)
0.877 0.073 0.001 0,017
CivilComments
LF (religion LF)
0.861 0.049 0.012 0.013
LF (race LF)
0.847 0.234 0.634 0.574
LF (toxic LF)
0.886 0.068 0.006 0.012
LF 6 (threat L.F)
0.862 0.102 0.055 0.054
LF (insult
0.872 0.176 0.028 0.035
LF (sex LF)
0.427 0.253 0.015 0.002
LF (LGBTQLF)
0.405 0.077 0.047 0.041
HateXplain
LF (religion LF)
0.437 0.197 0.001 0.003
LF 4 (race LF)
0.419 0.327 0.139 0.168
LF (toxic LF)
0.451 0.233 0.007 0.016
LF 6 (threat L.F)
0.415 0.097 0.000 0.004
LF (insult LF)
0.427 0.100 0.015 0,005
LF -
0.645 0.704 0.165 0.086
LF (Detoxify unbiased)
0,625 0.668 0.150 0,078
LF (Detoxify- muliilygaaull
0.649 0.700 0.168 0.077
LF 11 (Detoxify- oriiiiall
0.644 0.705 0.152 0,076
LF (Detoxify unbiased-smin -mmall)
0.643 0.699 0.186 0.113
Table 13: NLP dataset SBM OT) LF performance. As are obtained by comparison with raw LF
performance.
Dataset
LF
Acc (A)
FI (A)
ADP (A)
Ago (A)
LF 1
0.790 (0.035)
0.325 (0.138)
0.048 (0.002)
0.041 (0.022)
LF2
0.896 (0.019)
0.268 (0.195)
0.001 (0.000)
0.053 (0.036)
Civil Comments
LF3
0.868 (0.007)
0.155 (0.106)
0.012 (0.000)
0.043 (0.030)
LF4
0.858 (0.011)
0.252 (0.018)
0.114 (-0.520)
(-
LF5
0.886 (0.000)
0.137 (0.069)
0.006 (0.000)
0.003
LF6
(0.054)
0.482 (0.380)
0.023 (-0.032)
0.002
LF7
0.918 (0.046)
0.501 (0.325)
0.022 (-0.006)
(-0.2())
LF 1
0.483 (0.056)
0.273 (0.020)
0.015 (0.000)
0.001 (-0.001)
LF
0.473 (0.068)
0.058 (-0.019)
0.0.0440
0.004 (-0.037)
HateXplain
LF3
0.460 (0.023)
0.163 (-0.034)
0.001 (0.000)
0.004 (0.001)
LF4
0.481 (0.062)
0.328 (0.001)
0.044
0.039 (-0.119)
LF5
0.515 (0.064)
0.267 (0.034)
0.014
0.021 (0.005)
LF6
0.471 (0.056)
0.106 (0.009)
0.000 (0.000)
0.015 (0.011)
LF7
0.472 (0.045)
0.088 (-0.012)
0.012 (-0.003)
6..10000009
LF8
0.831 (0.186)
0.864 (0.160)
0.006 (-0.159)
0.000 (-0.086)
LF9
0.917 (0.292)
0.928 (0.260)
0.0.5
0-0.0777
LF 10
0.864 (0.215)
0.888 (0.188)
0-0.
0.000 (-0.077)
LF 11
0.839 (0.195)
0.015 (0.165)
(-0.137)
0.000 (-0.076)
LF 12
0.837 (0.194)
0.868 (0.169)
0.014
0.000 (-0.113)
28

---

Table 14: NL.P dataset SBM (OT-L) LF performance. As are obtained by comparison with raw LF
performance.
Dataset
LF
Acc (A)
F1 (A)
Apr (A)
A (A)
LF1 1
0.791 (0.036)
0.321 (0.134)
0.003 (-0.043)
0.022 (0.003)
LF2
0.898 (0.021)
0.272 (0.199)
0.001 (0.000)
0.020 (0.003)
Civil Comments
LF3
0.870 (0.009)
0.156 (0.107)
0.012 (0.000)
0.041 (0.028)
LF4
0.860 (0.013)
0.244 (0.010)
0.017 (-0.617)
0.017 (-0.557
LF5
0.887 (0.001)
0.139 (0.071)
0.006 (0.000)
0.027 (0.015)
LF6
0.917 (0.055)
0.484 (0.382)
0.012
0.026 (-0.028)
LF7
0.919 (0.047)
0.501 (0.325)
0.007 (-0.021)
0.013
LF
0.457 (0.030) (0.026)
0.015 (0.000)
0.001 (-0.001)
LF2
0.433 (0.028)
0.002 0.0.0(10.055))
(-0.045)
0.006
HateXplain
LF3
0.429 (-0.008)
0.001 (-0.06))
(0.000)
0.001 (-0.002)
LF4
0.457 (0.038)
0.3.3000.004
0.011 (-0.128)
0-009
LF5
0.479 (0.028)
0.263 (0.030)
0.017 (0.010)
(0.013)
LF6
0.436 (0.021)
0.111 (0.014)
0.000 (0.000)
(0.000
LF7
0.434 (0.007)
0.091 (-0.009)
0.012 (-0.003)
(0.012)
LF8
0.845 (0.200)
0.882 (0.178)
0.790 (0.093)
0-0.0800
LF9
0.923 (0.298)
0.938 (0.270)
0.883 (0.179)
(-00077)))
LF 10
0.875 (0.226)
0.903 (0.203)
0.823 (0.111)
0.0007700077
LF 11
0.848 (0.204)
0.884 (0.179)
0.792 (0.098)
0.000
LF
12
0.845 (0.202)
0.882 (0.183)
0.789 (0.090)
0.000 (-0.113)
Table 15: NLP dataset SBM (OT-S) LF performance. As are obtained by comparison with raw LF
performance.
Dataset
LF
Acc (A)
F1 (A)
APP (A)
Ago (A)
LF 1
0.791 (0.036)
0.320 (0.133)
0.015 (-0.031)
0.082 (0.063)
LF2
0.897 (0.020)
0.271 (0.198)
0.001 (0.000)
0.029 (0.012)
Civil Comments
LF3
0.870 (0.009)
0.156 (0.107)
0.012 (0.000)
0.043 (0.030)
LF4
0.860 (0.013)
0.245 (0.011)
0.017 (-0.617)
0.016 (-0.558)
LF5
0.887 (0.001)
0.138 (0.070)
0.006 (0.000)
0.020 (0.008)
LF6
0.917 (0.055)
0.482 (0.380)
0.011 (-0.044)
0.006 (-0.048)
LF7
0.919 (0.047)
0.504 (0.328)
0-0.0000.009)
0.039 (0.004)
LF
0.444 (0.017) 0.277 (0.024)
0.015 (0.000)
0.002 (0.000)
LF2
0.417 (0.012)
0.001 (-..02))) (-0.046)
(-0.040)
HateXplain
LF3
0.418 (-0.019)
0.172
0.001 (0.000)
0.002 (-0.001
LF4
0.448 (0.029)
0.332 (0.005)
0.032 (-0.100)
(-0.12))))
LF5
0.459 (0.008)
0.256
(0.00))
0.041 (0.025)
LF6
0.422 (0.007)
(0.003)
(0,000)
0.011 (0.007)
LF7
0.418 (-0.009)
(-0.099
(0.00))
0.025 (0.020)
LF8 8
0.851 (0.206)
0.889 (0.185)
0.000 (-..69)
(-0.088)
LF9
0.927 (0.302)
0.942 (0.274)
0.061 (-0.089)
0.000 (-0.078)
LF 10
0.884 (0.235)
0.911 (0.211)
0.052 (-0.166
(-0.077)
LF 11
0.853 (0.209)
0.890 10.5
0.056
0.000 (-0.076)
LF 12
0.847 (0.204)
0.886 (0.187)
0.063
0-0.010-.1133
29

---

Table 16: Vision dataset raw L. performance
Dataset
LF
Acc
F1
ADP
Ag
LF (ResNet-18 fine-tuned on gender classification dataset)
0.798 0.794 0.328 0.284
LF 2 (ResNct-34 fine-tuned on FairFace dataset)
0,890 0.901 0.314 0.105
CelebA
LF 3 (ResNet-34 fine-tuned on UTKFace dataset)
0.826 0.831 0.309 0.195
LF 4 (ResNet-34 fine-tuned on UTCeaace dataset (White only))
0.825 0.832 0.277 0.131
LF 5 (ResNet-34 fine-tuned on UTKFaae dataset (non- (non-White only))
0.818 0.832 0.271 0.134
LF 6 (ResNet-34 fine-tuned on UTKFaae dataset (age 2 50 only)
0.764 0.750 0.279 0.194
LF 7 (ResNet-34 fine-tuned on UTKFace dataset (age < <0 only))
0.830 0.845 0.299 0.175
LF 1 (ResNet-18 fine-tuned on gender classification dataset)
0.869 0.856 0.060 0.039
LF (ResNet-34 fine-tuned gender classification dataset)
0.854 0.842 0.060 0.060
FTKeaace
LF (ResNet-34 fine-tuned on CelebA dataset)
0.742 0.758 0.158 0.032
LF 4 (ResNet-34 fine-tuned CelebA dataset (attractive only))
0.580 0.692 0.065 0.002
LF 5 (ResNet-34 fine-tuned on CelebA dataset inon-attractive only))
0.687 0.608 0.129 0,034
LF (ResNet-34 fine-tuned CelebA dataset (young only))
0.664 0.729 0.116 0.012
LF 7 (ResNet-34 fine-tuned on CelebA dataset (non-young only))
0.619 0,429 0.136 0.081
LF 8 (ResNet-34 fine-tuned on CelebA dataset (unfair sampling))
0.631 0.676
0.113 0.053
Table 17: Vision dataset SBM LF (w/o OT) performance. As are obtained by comparison with raw LF
performance.
Dataset
LF
Acc (A)
Fl (A)
App (A)
ÁBO (A)
LF
0.847 (0.049)
0.832 (0.038)
0.328 (0.000)
0.267 (-0.017)
LF2 0.890 (0.000)
0.895 (-0.006)
0.314 (0.000)
0.101 (-0.004)
CelebA
LF3 0.926 (0.100)
0.923 (0.092)
0.309 (0.000)
0.097 (-0.098)
LF4 0.914 (0.089)
0.912 (0.080)
0,277 (0.000)
0.027 (-0.104)
LF5 0.899 (0.081)
0.900 (0.068)
0.271 (0.000)
0.030 (-0.104)
LF6 0.705 (-0.059)
0.629 (-0.121)
0.177 (-0.102)
LF7 0.913 (0.083)
0.915 (0.070)
0.299 (0.000)
0.056 (-0.119)
LF1 0.929 (0.060)
0.924 (0.068)
O. 102 (0.042)
0.011 (-0.028)
LF2 0.939 (0.085)
0.935 (0.093)
0.102 (0.042)
0.007 (-0.053)
UTKFace
LF3 0.631 (-0.111)
0.678 (-0.080)
0.078 (-0.080)
0.034 (0.002)
LF4 0.549 (-0.031)
0.681 (-0.011)
0.017 (-0.048)
0.002 (0.000)
LF5 0.740 (0.053)
0.679 (0.071)
0.129 (0.000)
(0.006)
LF6 0.694 (0.030)
0.755 (0.026)
0.116 (0.000)
0.037 (0.025)
LF7 0.694 (0.075)
0.541 (0.112)
0.061 (-0.075)
0.033 (-0.048)
LF8 0.591 (-0.040)
0.654 (-0.022)
0.071 (-0.042)
0.054 (0.001)
30

---

Table 18: Vision dataset SBM (OT-L) LF performance, As are obtained by comparison with raw LF performance.
Dataset
LF
Acc (A)
FI (A)
ADP (A)
AΚо (A)
LFI 1 0.847 (0.049)
0.832 (0.038)
0.328 (0.000)
0.268 (-0.016)
LF2 0.890 (0.000)
0.894 (-0.007)
0.314 (0.000)
0.102 (-0.003)
CelebA
LF3 0.926 (0.100)
0.922 (0.091)
0.309 (0.000)
0.098 (-0.097)
LF4 0.915 (0.090)
0.913 (0.081)
0.277 (0.000)
0.029 (-0.102)
LF5 0.898 (0.080)
0.900 (0.068)
0.271 (0.000)
0.031 (-0.103)
LF6 0.648 (-0.116)
0.498 (-0.252)
0.059 (-0.220)
0.217 (0.023)
LF7 0.914 (0.084)
0.916 (0.071)
0.299 (0.000)
0.058 (-0.117)
LF I 0.931 (0.062)
0.925 (0.069)
(-0.034)
(-0.022)
LF2 0.945 (0.091)
0.940 (0.098)
0.017 (-0.043)
0.006 (-0.054)
UTKFace
LF3 0.599 (-0.143)
0.667 (-0.091)
0.004 (-0.154)
0.001 (-0.031)
LF4 0.523 (-0.057)
0.667 (-0.025)
0.008 (-0.057)
0.002
LF5 0.738 (0.051)
0.674 (0.066)
0.129 (0.000)
0.029 (-0.005)
LF6 0.691 (0.027)
0.752 (0.023)
1.11600.000
0.037 (0.025)
LF7 0.690 (0.071)
0.525 (0.096)
0.000 (-0.136)
0.016 (-0.065)
LF8 0.572 (-0.059)
0.655 (-0.021)
0.001 (-0.112) 0.014 (-0.039)
Table 19: Vision dataset SBM (OT-S) LF performance. As are obtained by comparison with raw LF
performance.
Dataset
LF
Acc (A)
FI (A)
App (A)
AEO (A)
LF1 0.850 (0.052)
0.835 (0.041)
0.328 (0.000)
0.266 (-0.018)
LF2 0.893 (0.003)
0.897 (-0.004)
0.314 (0.000)
0.101 (-0.004)
CelebA
LF3 0.923 (0.097)
0.920 (0.089)
0.309 (0.000)
0.098 (-0.097)
LF4 0.913 (0.088)
0.910 (0.078)
0.277 (0.000)
0.029 (-0.102)
LF5 0.901 (0.083)
0.903 (0.071)
0.271 (0.000)
0.031 (-0.103)
LF6 0.618 (-0.146)
0.430 (-0.320)
0.016 (-0.263)
0,282 (0.088)
LF7 0.911 (0.081)
0.913 (0.068)
0.299 (0.000)
0.059 (10.116)
LF1 I 0.931 (0.062) 0.926 (0.070)
0-0.0
0.001 (-0.038)
LF2 0.949 (0.095)
0.945 (03)
0.015 (-0.045)
0.001 (-0.059)
UTKFace
LF3 0.577 (-0.165)
0.652 106)
0.003 (-0.155)
(-0.019)
LF4 F 0.517 (-0.063)
0.664 (-0.028)
0.0.
0.007 (0.005)
LF5 0.730 (0.043)
0.669 (0.061)
0.129 (0.000)
0.018
LF6 0.694 (0.030)
0.756 (0.027)
0..1600.000)
0.036
LF7 0.683 (0.064)
0.523 (0.094)
0.010 (-0.126)
0.007 (-0.074)
LF8 0.562 (-0.669)
0.652 (-0.024)
0.009 (-0.104)
(-0.04))
31

---

Adult
High accuracy group
Low accuracy group
1.0
0.9
0.8
ACC
0.6
0.5
20000 40000 60000 80000 100000 Distance
20000 60000
80000 100000
Distance
1
LF2
LF
LF 4
LF
LF
LF
8
9
(a) Adult (raw)
bank marketing
High accuracy group
Low accuracy group
1.0
0.8
Acc
0.6
0.4
0
500
1000 Distance
1500
2000
0
500
1000 Distance
1500
2000
LF 1
LF 3
LF 5
LF 2
LF 4
LF 6
(b) Bank Marketing (raw)
Figure 6: Identification of high accuracy regimes for tabular datasets.
E.4 Identification of centers
While our method improves performance by matching one group distribution and attempting make these uniform, it does not imply accuracy improvement. A presumption our method is that the group with high (estimated) accuracy possesses the high accuracy regime and our method can transport data points to this high accuracy regime while keeping their structure, which results accuracy improvements. To empirically support this hypothesis, we used the following procedure and visualized the results in Figure 6 9
1. Find the best accuracy center by evaluating the accuracy of the nearest 10% of data points for each center candidate point.
2. Expand 2% percent of data points closest to the center each time, compute their accumulated average accuracy (y-axis) and the farthest distance (X-axis) from the neighborhood
3. Find the group with better accuracy group and visualize their accumulated average accuracy (y-axis) and the farthest distance (x-axis) in each group.
We are able to obtain two insights from the visualization. First, the high accuracy regime typically exists in the group with the high estimated accuracy. which supports our hypothesis. Thus accuracy improvement by optimal transport can be justified. Secondly, the groups actually show the distributional difference in the input space X. Given center points, lines in the high accuracy group start with smaller distance to the center than the low accuracy group.
32

---

Adult (LIFT)
High accuracy group
Low accuracy group
1.0
0.8
Acc
0.5
1
1
3
Distance
Distance
(a) Adult (LIFT)
High accuracy group bank marketing (LIFT)
Low accuracy group
1.0
0.8
Acc
0.6
0.5
1.0
1.5
2.0
2.5
3.0
3.5
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Distance
Distance
LF
LF
LF5
LF4
(b) Bank Marketing (LIFT)
Figure 7: Identification of high accuracy regimes for tabular datasets (LIFT).
33

---

CivilComments
High accuracy group
Low accuracy group
0.9
0.8
Acc
0.7
0.6
0.5
10
15
20
25
10
15
20
25
Distance
Distance
LF1
2
LF 3
LF 4
LF
6
LF
(a) CivilComments
hateXplain
High accuracy group
Low accuracy group
0.8
0.6
Acc
0.2
20
20
Distance
Distance
LF
LF
LF 5
LF 7
LF9
LF 11
2
LF
LF 6
LF 8
LF 10
LF 12
(b) HateXplain
Figure 8: Identification of high accuracy regimes for NLP datasets.
34

---

CelebA
High accuracy group
Low accuracy group
0.95
0.90
Acc
0.80
0.75
15
20
25
30
15
20
25
30
Distance
Distance
LF1
LF2
LF3
LF 4
LF
LF6
LF
(a) CclebA
UTKFace
High accuracy group
Low accuracy group
1.0
0.9
Acc
0.8
0.7
0.6
10.0
12.5
15.0
17.5 Distance
20.0
22.5
25.0
10.0 12.5 15.0 17.5
20.0
22.5
25.0
Distance
1
LF
LF
LF7
LF
LF4
LF 6
-
LF8
(b) UTKFace
Figure 9: Identification of high accuracy regimes for vision datasets.
35

---

E,5
Compatibility with other fair ML methods
One advantage of our method is that we can use other successful fair ML methods in a supervised learning setting on top of SBM, since our method works in weak label sources while traditional fair ML methods work the model. To make this point, we tried traditional fair MI. methods from fairlearn BD)++2) with each of WS settings. We used CorrelationRemover, ExponentiatedGradiin [ABD* 18 ThresholdOptimize [HPS16] with the demographic parity (DP), equal opportunity (EO) as parity criteria, and accuracy as the performance criteria. The results are reported 20 As expected, combining with other methods yields accuracy-fairness tradeoff given weak label sources. Typically, SBM yields additional gains upon traditional fair ML methods. One another observation is that fair ML methods to modify equal opportunity typically fail to achieve the reduction of Ago. This can be interpreted as the result the noise in the training set labels.
steps, which are independent of the label
Table 20: SBM combined with other fair ML methods in Adult dataset
Fair ML method
Acc
APP
AgO
N/A
0.717
0.587
0.475 0.325
correlation remover
0.716
0.587 0.446 0.287
optimal threshold (DP)
0.578 0,499 0.002 0.076
WS (Baseline)
optimal threshold (EO)
0.721
0.563
0,404 0.217
exponentiated gradient (DP gap=
0.582
0.502
0.002 0.066
exponentiated gradient (EO gap
0.715
0.585 0.445 0.284
N/A
0.720 0.592
0.439 0.273
correlation remover
0.717 0.586 0.437 0.264
optimal threshold (DP)
0.591
0.507 0.003 0.059
SBM (w/o OT)
optimal threshold (EO)
0.722
0.571 0.387 0.189
exponentiated gradient (DP gap=
0.693
0.525
0.014 0.052
exponentiated gradient (EO ga0ppp0))
0.722
0.586 0.404 0.233
N/A
0.560
0.472
0.893 0.980
correlation remover
0.460
0.443 0.084 0.005
optimal threshold (DP)
0.324 0.404 0.006 0.089
SBM (OT-L)
optimal threshold (EO)
0.300
0.399 0.103 0.015
exponentiated gradient (DP gap=
0.345
0,414
0.002 0.016
exponentiated gradient (EO g)
0.558
0.479
0.861 0.792
N/A
0.722
0.590
0.429 0.261
correlation remover
0.729
0.595
0.408 0.249
optimal threshold (DP)
0.596 0.507
0,003 0.050
SBM (OT-S)
optimal threshold (EO)
0.723
0.571 0.382 0.184
exponentiated gradient (DP gap=
0.687
0.527
0.011 0.045
exponentiated gradient (EO 0)
0.728
0.587
0.390 0.218
36

---

Table 21: SBM combined with other fair ML methods in Adult dataset (LIFT)
Fair ML method
Acc
DP
A FO
N/A
0.711
0.584
0.449 0.290
correlation remover
0.716
0.587 0.446 0.287
optimal threshold (DP)
0.578 0.499 0.002 0.076
WS (Baseline)
optimal threshold (EO)
0.721
0.563
0,404 0.217
exponentiated gradient (DP gap=
0.582
0.502
0.002 0.066
exponentiated gradient (EO gap=0)
0.715 0.585
0.445 0.284
N/A
0,704 0.366 0.032 0.192
correlation remover
0.686 0.351 0.006 0.155
optimal threshold (DP)
0.707 0.363 0.007 0.133
SBM (w/o OT)
optimal threshold (EO)
0.713 0.362 0,022 0.079
exponentiated gradient (DP gap
0.682 0,350 0,011 0.163
exponentiated gradient (EO g)
0.701 0.369
0.019 0.134
N/A
0.700 0.520
0.015 0.138
correlation remover
0.686 0.504
0,011 0.105
optimal threshold (DP)
0.701 0.520
0.008 0.124
SBM (OT-L)
optimal threshold (EO)
0.712 0.521 0.060 0.025
exponentiated gradient (DP gap=
0.673 0.504
0.005 0.071
exponentiated gradient (EO 0)
0.691
0.516
0.058 0.035
N/A
0.782
0.448
0.000
0.178
correlation remover
0.772
0.435
0.002
0.180
optimal threshold (DP)
0.782 0.447
0.001
0.176
SBM (OT-S)
optimal threshold (EO)
0.790
0.427 0.087
0.104
exponentiated gradient (DP gap= 0)
0.784
0.452 0.000
0.171
exponentiated gradient (EO 0) 0.747 0.380 0.107 0.049
Table 22: SBM combined with other fair MI methods in Bank Marketing dataset
Fair MI. method
Acc
F1
DP
EE
N/A
0.674
0.258 0.543 0.450
correlation remover
0.890 0.057 0.002 0.006
optimal threshold (DP)
0.890 0.058 0.002 0.007
WS (Baseline)
optimal threshold (EO)
0.890 0.066 0.030 0.040
exponentiated gradient (DP gap=
0.889 0.039 0.000 0.009
exponentiated gradient (EO gap
0.890 0.070 0.033 0.051
N/A
0.876 0.550
0.106 0.064
correlation remover
0.874 0.547 0.064 0.095
optimal threshold (DP)
0,876 0.547 0.031 0.208
SBM (w/o OT)
optimal threshold (EO)
0,876 0.548 0,053 0.171
exponentiated gradient (DP gap=
0.877 0.525 0.037 0.182
exponentiated gradient (EO 0)
0.872 0.531 0.066 0.124
N/A
0.892 0.304 0.095 0.124
correlation remover
0.890 0.290 0.011 0.111
optimal threshold (DP)
0.891 0.280 0.008 0.163
SBM (OT-L)
optimal threshold (EO)
0,881 0.313 0.841 0.678
exponentiated gradient (DP gap=
0.891 0.263 0.003 0.106
exponentiated gradient (FO =
0.895 0.296 0.097 0.136
N/A
0.847 0.515 0.122 0.080
correlation remover
0.847 0.512 0.072 0.122
optimal threshold (DP)
0.846 0.511 0.043 0.236
SBM (OT-S)
optimal threshold (EO)
0.847 0.515 0.113 0.104
exponentiated gradient gap=0)
0.843
0.487 0.052 0.143
exponentiated gradient (EO 0)
0.848 0.512 0.114 0.088
37

---

Table 23: SBM combined with other fair ML methods in Bank Marketing dataset (LIFT)
Fair ML method
Acc
DP
EO
N/A
0.674 0.258 0.543 0.450
correlation remover
0.890 0.057 0.002 0.006
optimal threshold (DP)
0.890 0.058 0.002 0.007
WS (Baseline)
optimal threshold (EO)
0.890 0.066 0.030 0.040
exponentiated gradient (DP gap=
0.889 0.039 0.000 0.009
exponentiated gradient (EO gap=
0.890 0.070 0.033 0.051
N/A
0.698 0.255 0.088 0.137
correlation remover
0.836 0.358 0.025 0.114
optimal threshold (DP)
0.699 0.252 0.002 0.019
SBM (w/o OT)
optimal threshold (EO)
0.698 0.253 0,006 0.028
exponentiated gradient (DP gap=
0.687 0.262 0.014 0.053
exponentiated gradient (EO gap=
0.654 0.226 0.096 0.107
N/A
0.892 0.305
0.104 0.121
correlation remover
0,891 0.304 0.079 0.000
optimal threshold (DP)
0.891 0.289 0.016 0.094
SBM (OT-L)
optimal threshold (EO)
0.892 0.305 0.103 0.121
exponentiated gradient (DP gap=
0.893 0.265 0.001 0.093
exponentiated gradient (EO 0)
0.892 0.305 0.100 0.109
N/A
0.698 0.080 0.109 0.072
correlation remover
0.699 0.081 0.230 0.106
optimal threshold (DP)
0.697 0.083 0.028 0.011
SBM (OT-S)
optimal threshold (EO)
0.695 0.089 0.174 0.205
exponentiated gradient (DP gap
0.681 0.113 0,032 0.063
exponentiated gradient (EO 0) 0.691 0.124 0.041 0.036
Table 24: SBM combined with other fair ML methods CivilComments dataset
Fair MI. method
Acc
F1
DP
EE
N/A
0.854
0.223
0.560 0.546
correlation remover
0.886
0.000
0.000 0.000
optimal threshold (DP)
0,886 0.000 0.000 0.000
WS (Baseline)
optimal threshold (EO)
0.886
0.000 0.000 0.000
exponentiated gradient (DP gap=
0,886
0.000 0.000 0.000
exponentiated gradient (EO gap
0.886
0.000 0.000 0.000
N/A
0.879 0.068 0.048 0.047
correlation remover
0.878 0.062 0.010 0.030
optimal threshold (DP)
0,880 0.054 0.001 0.015
SBM (w/o OT)
optimal threshold (EO)
0.880 0.061 0.019 0.010
exponentiated gradient (DP gap=
0.881 0.046
0.002 0.008
exponentiated gradient (EO 0)
0.880 0.059
0.018 0.010
N/A
0.880 0.070 0.042 0.039
correlation remover
0.879 0.056 0.011 0.028
optimal threshold (DP)
0.880 0.060 0.001 0.017
SBM (OT-L)
optimal threshold (EO)
0.880 0.063 0.013 0.002
exponentiated gradient (DP gap=
0.882
0.043 0.002 0.008
exponentiated gradient (FO gap 0)
0.882
0.039
0.006 0.001
N/A
0.882
0.047
0.028 0.026
correlation remover
0.879 0.057
0.011 0.029
optimal threshold (DP)
0.882
0.040
0.000 0.011
SBM (OT-S)
optimal threshold (EO)
0.882
0.042
0.010 0.000
exponentiated gradient (DP gap=
0.881
0.045
0.001 0.008
exponentiated gradient (EO 0)
0.880
0.056
0.016 0.005
38

---

Table 25: SBM combined with other fair ML methods in HateXplain dataset
Fair ML method
Acc
APP
A eo
N/A
0.584
0.590
0.171
0.133
correlation remover
0.555
0.557
0.007
0.031
optimal threshold (DP)
0.539
0.515
0.005
0.047
WS (Baseline)
optimal threshold (EO)
0.573
0.573
0.129
0.090
exponentiated gradient (DP gap = 0)
0.562
0.561
0.006
0.055
exponentiated gradient (EO gap 0)
0.579
0.586
0.130
0.093
N/A
0.592
0.637
0.159
0.138
correlation remover
0.563
0.616
0.033
0.053
optimal threshold (DP)
0,586
0.660
0.013
0.006
SBM (w/o OT)
optimal threshold (EO)
0.538
0.561
0.034
0.074
exponentiated gradient (DP gap =
0)
0.581
0.638
0.039 0.013
exponentiated gradient (EO gap 0)
0.580
0.630
0.047
0.095
N/A
0.606
0.670
0.120
0.101
correlation remover
0.587
0.657
0,057
0.087
optimal threshold (DP)
0.600
0.683 0.010 0.004
SBM (OT-L)
optimal threshold (EO)
0.563
0.615
0.039 0.071
exponentiated gradient (DP gap
0.600
0.673
0.029 0.011
exponentiated gradient (EO gap 0)
0.593
0.669
0.044
0.087
N/A
0.612
0.687
0.072
0.037
correlation remover
0.587
0.668
0.073
0.105
optimal threshold (DP)
0.607
0.694
0.002 0.031
SBM (OT-S)
optimal threshold (EO)
0.572
0.696
0.201 0.182
exponentiated gradient (DP gap = 0)
0.598
0.683
0,005
0.020
exponentiated gradient (EO gap 0)
0.585
0.672 0.070 0.093
Table 26: SBM combined with other fair ML. methods in CelebA dataset
Fair MI. method
Acc
APP
A Eo
N/A
0.866
0.879
0.308
0.193
correlation remover
0.845
0.862
0.099
0.066
optimal threshold (DP)
0.816
0.845
0.009
0.035
WS (Baseline)
optimal threshold (EO)
0.789
0.793
0.196
0.033
exponentiated gradient (DP gap 0)
0.781
0.814
0.008
0.006
exponentiated gradient (EO gap 0)
0.838
0.854
0.205
0.025
N/A
0.870
0.883
0.309
0.192
correlation remover
0.849
0.865
0.095
0.066
optimal threshold (DP)
0.819
0.848
0.009
0.038
SBM (w/o OT)
optimal threshold (EO)
0.792
0.798
0.194
0.030
exponentiated gradient (DP gap = 0)
0.783
0.818
0.006
0.007
exponentiated gradient (EO gap 0)
0.841
0.857
0.206
0.029
N/A
0.870
0.883
0.306
0.185
correlation remover
0.849
0.866
0.096
0.066
optimal threshold (DP)
0.819
0.848
0.010 0.034
SBM (OT-L)
optimal threshold (EO)
0.792
0.798
0.193
0.023
exponentiated gradient (DP gap 0))
0)
0.783
0.818
0.007
0.008
exponentiated gradient (EO gap 0)
0.841
0.857
0.203
0.026
N/A
0.872
0.885
0.306
0.184
correlation remover
0.851
0.867
0.097
0.062
optimal threshold (DP)
0.821
0.850
0.010
0.035
SBM (OT-S)
optimal threshold (EO)
0.795
0.801
0.193
0.023
exponentiated gradient (DP gap 0)
0.784
0.819
0.008
0.010
exponentiated gradient (EO gap - 0)
0.840
0.857
0.200
0.022
39

---

Table 27: SBM combined with other fair ML methods in UTKeaaee dataset
Fair MI. method
Acc
APP
A EO
N/A
0.791
0.791
0.172
0.073
correlation remover
0.787
0.786
0.034
0.051
optimal threshold (DP)
0.774
0.767
0.040
0.215
WS (Baseline)
optimal threshold (EO)
0.788
0.786
0.114 0.005
exponentiated gradient (DP gap 0)
0.769
0.762
0.029 0.078
exponentiated gradient (EO gap 0)
0.792
0.790
0.126 0.026
N/A
0.797
0.790
0.164 0.077
correlation remover
0.791
0.793
0,006 0.091
optimal threshold (DP)
0.764
0.791
0.033 0.202
SBM (w/o OT)
optimal threshold (EO)
0.789
0.791
0.165 0.077
exponentiated gradient (DP gap 0)
0.760
0.791
0.024
0.069
exponentiated gradient (EO gap 0)
0.791
0.792
0.158
0.076
N/A
0.800
0.793
0.135
0.043
correlation remover
0.799
0.791
0.004
0.098
optimal threshold (DP)
0.785
0.772
0.034
0.195
SBM (OT-L)
optimal threshold (EO)
0.800
0.793
0.128
0.038
exponentiated gradient (DP gap 0)
0.779
0.764
0.024
0.069
exponentiated gradient (EO gap - 0)
0.797
0.789
0.123
0.030
N/A
0.804
0.798
0.130
0.041
correlation remover
0.799
0.794
0.012 0.087
optimal threshold (DP)
0.789
0.777
0.036
0.195
SBM (OT-S)
optimal threshold (EO)
0.799
0.794
0.168 0.046
exponentiated gradient (DP gap = 0)
0.776
0.764
0.023 0.068
exponentiated gradient (EO gap 0)
0.801
0.796
0.141 0.044
40

---

E.6
Additional synthetic experiment result on the number of LFs
LF Performance (1)
LF Unfairness
0.9
(SBM)
(SBM)
Baseline)
0.7
0.6
0.2
0.5
0.1
0.4
Acc (SBM)
F1 (SBM)
0.0
0.3
Acc (WS Baseline)
0.2
F1 (W5 Baseline)
-0.1
10
15
15
20
30
Number of LFs
Number of LFs
(a) Number of LFs vs. Performance
(b) Number of LFs vs. Unfairness
Figure 10: Synthetic experiment on number of LFs vs. performance and fairness
Claim Investigated One seemingly possible argument that diversifying LFs can naturally resolve unfairness, weakening the necessity of our method. We hypothesized that simplying increasing the number of I.Fs does not improve fairness, while our method can further improve fairness as the number LFs increases. To show this, generate unfair synthetic data and increase the number of LFs and see if our method can remedy LF fairness and improve LF performance when LFs are diversified.
Setup and Procedure We generated synthetic dataset 10000) with the similar procedure as the previous synthetic experiment. Input samplees in IR2 are taken from ~N(0,) 7), and true labels Y generated by Y - 0J. Group transformation gb (ar): : is applied to random half of samples, where (=2(((1),50))) We varied LFs by randomly sampling parameters based on our label model. Specifically, parameters for each LFs are sampled from 0, (0.10. center; [[[] [[5,5] We varied the number of LFs from 3 to 30. We repeat experiments 10 times with different seeds and provide means and confidence intervals.
The result is reported in Figure 10 The result shows that simply diversifying label sources does not improve ttirness-thowd though performance can be improved by adding more LFs. On the other hand, SBM resolves unfairness and yields better performance by mitigating source bias.
Results
41

---

E.7 Large language model experiment
Recently, large language models (LMs) pre-trained with massive datasets have shown excellent performance in many NLP tasks, even without fine-tuning As proof of concept that our method can be used for LMs, we conducted an additional experiment that applies our method NLP tasks with L.M. 2 introduced AMA, which is prompting strategy that provides multiple prompts with varying instructions, and combines their outputs to obtain final answer using weak supervision. In this setup, distinct prompts can be seen as label sources. Based on their setup, we craft group annotations that identify low accuracy regime by Barlow slicing (SNS 21). which is tree-based data slicing method. After applying data slicing for each prompt, apply SBM and use the AMA pipeline. The result is shown Table 28 SBM with Barlow data slicing yields modest performance gains over AMA. This result suggests that it may be possible to use our techniques to gain improvements even in generic zero-shot language model prediction.
Table 28: SBM combined with AMA AACC 22
Performance metric is accuracy (%).
Methods
RTE
WSC
WIC
AMA 22
75.1
77.9
61.3
AMA (Reproduced)
75.1
76
60.8
SBM (w/o OT)
75.5
71.2
61.1
SBM (OT-L)
75.1
70.2
61.6
SBM (OT-S)
74.7
78.8
61.1
42


Reviews:
[{"review_id": "WbRms2m7sP", "reviewer": "Reviewer_wmSM", "summary": "The paper focuses on the bias issue in weak supervision. The paper shows the labeling functions of weak supervision may produce biased pseudo-labels, which is first empirically shown in the paper. Also, they theoretically show that even though a dataset has fair ground-truth labels, the weak supervision process using labeling functions may produce biased weak labels for the dataset. To mitigate this issue, the paper proposes an optimal transport-based algorithm that modifies the weak labels to be fairer. In experiments, the paper evaluates the proposed algorithm in tabular, NLP, and computer vision datasets.", "questions": "Although the paper provides several meaningful discussions and insights, there are some remaining concerns, especially regarding the labeling function and empirical results. The details are in the above weakness section.", "limitations": "The paper did not discuss the limitations and potential negative social impact of the work. A possible limitation of this work can be other prominent fairness definitions that this work cannot handle.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "S1. The paper considers an important research problem, the fairness issue in the weak supervision pipeline. It seems this paper is the first work to handle fairness in weak supervision.\n\nS2. The paper theoretically shows the labeling functions may produce unfair pseudo-labels even though the underlying label distribution is fair. \n\nS3. The proposed method can be used together with the existing fair in-processing algorithms, which is another good aspect of the paper.\n", "weaknesses": "W1. The explanations on the labeling function itself are limited.\n- For example, can the number of labeling functions affect the fairness performance? The Snorkel original paper [1] mentioned that the number of labeling functions highly affects the labeling performances, but the current paper does not explain how such details may make differences in the fairness scenario. \n- Also, as shown in the vision dataset experiments, using fairer labeling functions can reduce the effectiveness of the proposed algorithm. Then, why is the proposed transport algorithm better than making labeling functions itself fairer? Currently, the paper seems to use very simple labeling functions (described in the appendix), and it would be helpful if the paper could provide any comparison between the proposed algorithm and another possible direction of making fairer labeling functions.\n\n[1] Ratner et al., Snorkel: Rapid Training Data Creation with Weak Supervision, VLDB’18\n\n\nW2. Experiments show some questionable results.\n- As several scenarios show large F1 score drops, it is unclear whether it is okay to use the proposed algorithm. I understand the F1 score can be affected by the imbalanced label classes, but the F1 score degradation is still severe. For example, SBM (OT-S) + LIFT case in the Bank Marketing dataset and all SBM results in the Civil dataset show large F1 score drops. It would be helpful if the paper could provide at least some possible way to prevent such F1 score drops.\n- In the experiments, no error range is provided, which makes the empirical results less convincing.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their kind words, constructive feedback, and useful suggestions.\n* **On the number of labeling functions**: We discussed the relationship between the number of LFs, fairness, and performance in our common response. The result  shows that our method can resolve unfairness induced by weak supervision and improve performance, while existing weak supervision baselines lose fairness as they gain performance by increasing the number of LFs.\n* **On improving individual LF fairness**: Indeed, if we could write fair labeling functions directly, the resulting datasets would be fair as well. Unfortunately, this is extremely difficult. Most labeling functions are small programs written by domain experts to express heuristic ideas. Ensuring that these programs are themselves fair requires substantial research progress in other fields. Similarly, other sources for labeling functions are pre-existing external knowledge bases, crowdworkers, pretrained models, and more. Here as well, ensuring fairness directly would be very hard. As a concrete example, labeling functions in our vision dataset experiments are generated by models pretrained on other datasets. In such situations, where we do not have access to the original dataset, it may not even be possible to directly modify LFs to be fair. Instead, **our method provides a generic and cheap approach to deal with the bias of label sources in weak supervision.** \n\n    Another idea is to simply demand that each labeling function used is created to be fair in the first place, throwing away any type of source of LFs that may not satisfy this property (including heuristics, pretrained models, etc.) Doing so would ensure fairness as well---but removes all of the benefits of weak supervision. In fact, crafting only fair labeling functions that lead to a satisfactory dataset may be more difficult than hand-labeling in the first place, which is the pain point weak supervision was designed to solve. The **value of our approach is that it is best-of-all-worlds**: we can continue to omnivorously use weak sources of signal, while handling bias with the additional technique we proposed. \n* **On the F1 score in Bank Marketing and CivilComments datasets**: As mentioned, Bank Marketing and CivilComments datasets have severe class imbalance (the proportion of positive class: 11.8% and 11.3%, respectively), which may lead more imbalance in the transport step, producing high odds to flip noisy labels towards dominant classes. A simple remedy is performing balanced sampling. We can selectively use the target group data points so that their noisy labels have a balanced class ratio. The table below shows the result with balanced transport in Bank Marketing data with LIFT. We observe that this simple idea can improve the F1 score while maintaining accuracy and fairness scores. In general, _any_ counterfactual mapping estimation method that considers class imbalance can be plugged in.\n    \n| Cond                                  | Acc           | F1            | DP Gap        | EO Gap        |\n| ------------------------------------- | ------------- | ------------- | ------------- | ------------- |\n| FS                                    | $0.912 \\pm 0.000$ | $0.518 \\pm 0.000$ | $0.128 \\pm 0.000$ | $0.117 \\pm 0.000$ |\n| WS (Baseline)                         | $0.674 \\pm  0.000$ | $0.258 \\pm 0.000$ | $0.543 \\pm 0.000$ | $0.450 \\pm 0.000$ |\n| SBM (w/o OT) + LIFT                   | $0.698\\pm 0.000$ | $0.255 \\pm 0.000$ | $0.088 \\pm 0.000$ | $0.137 \\pm 0.000$ |\n| SBM (OT-L) + LIFT                     | $0.892 \\pm 0.015$ | $0.305 \\pm 0.015$ | $0.104 \\pm 0.001$ | $0.121 \\pm 0.019$ |\n| SBM (OT-S) + LIFT                     | $0.698 \\pm 0.011$ | $0.080 \\pm 0.006$ | $0.109 \\pm 0.017$ | $0.072 \\pm 0.014$ |\n| SBM (OT-S) + LIFT + Balanced sampling | $0.827 \\pm 0.002$ | $0.498 \\pm 0.003$ | $0.133 \\pm 0.002$ | $0.077 \\pm 0.002$ |\n\n* **On error ranges**: Our proposed method is fairly simple and the effect of randomness is very limited. In repeated experiments with different seeds,  **we do not observe any significant deviations, but will provide error ranges from repeated runs in the updated draft.** We include several examples of these updated results in the following:\n    \n    * Synthetic data experiments (Section 5.2. Figure 4): In rebuttal supplementary A.2, we show 95% confidence intervals, which are from 10 repetitions with different seeds. Unsurprisingly, the deviation rapidly drops as n increases.\n    * Real data experiments (Section 5.1.): We reported the mean of 5 repeated experiment results in Section 5.1. We will include error ranges in our updated manuscript. Examples include:\n#### Adult dataset\n    \n| Cond                | Acc           | F1            | DP Gap        | EO Gap        |\n| ------------------- | ------------- | ------------- | ------------- | ------------- |\n| FS                  | $0.824 \\pm 0.000$ | $0.564 \\pm 0.000$ | $0.216 \\pm 0.000$ | $0.331 \\pm 0.000$ |\n| WS (Baseline)       | $0.717 \\pm 0.000$ | $0.587 \\pm 0.000$ | $0.475 \\pm 0.000$ | $0.325 \\pm 0.000$ |\n| SBM (w/o OT)        | $0.720 \\pm 0.000$ | $0.592 \\pm 0.000$ | $0.439 \\pm 0.000$ | $0.273 \\pm 0.000$ |\n| SBM (OT-L)          | $0.560 \\pm 0.000$ | $0.472 \\pm 0.000$ | $0.893 \\pm 0.000$ | $0.980 \\pm 0.000$ |\n| SBM (OT-S)          | $0.723 \\pm 0.003$ | $0.590 \\pm 0.003$ | $0.429 \\pm 0.010$ | $0.261 \\pm 0.005$ |\n| SBM (w/o OT) + LIFT | $0.704 \\pm 0.000$ | $0.366 \\pm 0.000$ | $0.032 \\pm 0.000$ | $0.192 \\pm 0.000$ |\n| SBM (OT-L) + LIFT   | $0.700 \\pm 0.017$ | $0.520 \\pm 0.005$ | $0.015 \\pm 0.015$ | $0.138 \\pm 0.020$ |\n| SBM (OT-S) + LIFT   | $0.782 \\pm 0.043$ | $0.448 \\pm 0.015$ | $0.000 \\pm 0.000$ | $0.178 \\pm 0.036$ |\n"}, {"review_id": "uFMGgpXFfF", "reviewer": "Reviewer_UYjh", "summary": "This paper proposes a novel bias mitigation technique to address the fairness issues in weak supervision settings. The core idea is to use a  counterfactual fairness-based correction method. The authors theoretically show that the proposed method can improve both accuracy and fairness and this is also supported by both synthetic and real datasets evaluations. Overall, this paper calls attention to bias and fairness studies in the weak supervision context, which has never been addressed specifically and provides an effective and theoretically sound method. To the best of my knowledge, the method considered in this paper is novel and is a valuable contribution to the community. Based on the above factors, I recommend acceptance.", "questions": "I welcome the authors to discuss the potential drawbacks of formulating the fairness argument behind counterfactual fairness, particularly in the programmatic weak supervision settings. Given diverse sources, wouldn't the biases be smoothed out?", "limitations": "Overall, this paper is well organized and I have not identify any apparent limitations for the current content.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper is well-motivated and well-written. The proposed method is intuitively simple yet very effective and theoretically sound. I appreciate the comprehensive evaluation both with synthetic data and with WRENCH.", "weaknesses": "This paper focuses on a novel area of weak supervision studies. Though the method proposed is simple, I do believe it has any major weak nesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for noting the strengths of our work and providing useful questions and comments.\n* **On the potential drawbacks of the proposed method in the programmatic WS settings**: A possible drawback of the proposed method is that it requires high quality estimation of Monge mapping (or any counterfactual mapping.) Fortunately, as we described in the general response, in the typical weak supervision settings, we have access to massive unlabeled datasets which enable reliable estimation in these procedures. Better yet, we have found that our method works well even with relatively small datasets ($n \\leq 10^4$, e.g. the Wrench Mushroom dataset ($n=6499$)). \n\n    If the available unlabeled data is genuinely insufficient, our technique may struggle. To address this, we note that there are ways to overcome this limitation as well. For example, there are techniques that can dramatically reduce the amount of data that is needed in optimal transport. Such techniques involve the use of _keypoints_ (pairs of matched points) [1]. Domain experts could craft such pairs, enabling optimal transport, and ultimately our entire pipeline, to operate in the very low data regime. \n\n* **On the possibility that diverse label sources resolve unfairness naturally**: As mentioned in our common resonse, this appears to be a plausible approach towards mitigating unfairness. Unfortunately, diverse sources may not in fact behave in this way---further motivating the use of our approach. For example, our synthetic experiment in A.1. shows that diverse label sources can make fairness _worse_. In this setting, label sources that have high accuracy centers around the major clusters have higher weights in voting as an outcome of label model learning in weak supervision. Thus, increasing the number of LFs (diversifying) can be beneficial in finding such centers, and yields improved performance. However, it makes disadvantaged groups perform worse, since they are dislocated from high accuracy centers, leading to unfairness. SBM not only resolves such issues, but also further improves the performance gain by bringing back the dislocated data points around the high accuracy centers.\n\n[1] Gu, Xiang, et al. \"Keypoint-guided optimal transport with applications in heterogeneous domain adaptation.\", NeurIPS 2022."}, {"review_id": "7AFiF0L3QX", "reviewer": "Reviewer_GLUc", "summary": "*The paper studies the problem of unfairness introduced by weak supervision methods through noisy data augmentation. It proposes a mitigation strategy using counterfactuals that would create a more balanced/unbiased dataset. \n* Paper shows that Labeling functions can be arbitrarily biased by preferring examples away from the center of the distribution and show theoretically there is no change in sample complexity required to achieve the gains in accuracy and fairness, under strict assumptions.\n* Empirical results show that by further augmenting data that counterfactually transforms examples across groups, both accuracy and fairness metrics can be improved - with results on synthetic and popular benchmark fairness and weak supervision datasets.\n", "questions": "(see weaknesses)", "limitations": "This is missing, and should be included", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Strong empirical results on synthetic and real datasets\n* New problem formulation that extends fairness methods to the weak supervision case", "weaknesses": "* Theoretical results are under strong distributional assumptions, how they can be relaxed should be better articulated\n* Standard data augmentation techniques such as Autolabel [1] are missing in evaluation set up.\n\nhttps://ieeexplore.ieee.org/abstract/document/10136178", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the suggestions and noting the strengths of our work. We have added the suggested reference.\n* **On modeling assumptions**. Indeed, our label model _may appear_, at first glance, to require strong assumptions. However, in fact it **has weaker assumptions compared to previous weak supervision models.** We describe the differences versus two popular weak supervision models next. Common models use one of the following assumptions:\n\n    **i) Ising model assumption [1, 2, 3]**. Here, the model is given by \n    $$P(\\lambda^1, \\ldots, \\lambda^m,y) = \\frac{1}{Z}\\exp(\\theta_y y + \\sum_{j=1}^m \\theta_j \\lambda^j y)$$ \n    Here, the $\\lambda$'s are labeling functions, $y$ is the latent true label, $Z$ is a partition function (a normalizing constant), the $\\theta$'s are the model parameters.\n\n    This label model is based on the Ising pairwise interactions model. In this model, parameters are estimated globally, which means that **the label model requires uniform accuracy in the entire feature space---a strong assumption.** \n    \n    In other words, regardless of the point $x \\in \\mathcal{X}$ whose label is being estimated, the assumption states that the labeling function accuracies are identical. Naturally, this is not a realistic assumption. Indeed, in practice, it has been observed that LF accuracy strongly depends on the input features, as certain parts of the feature space are invariably more challenging to label than others.\n\n    \n    **ii) Strong smoothness leading to a partitionable feature space [4]**\n\n    $$P_x(\\lambda^1, \\ldots, \\lambda^m, y) = \\frac{1}{Z}\\exp(\\theta_y y + \\sum_{j=1}^m \\theta_{j,x} \\lambda^j(x) y)$$\n    This model assumes that accuracy depends on the input feature $x$. In fact, in its most basic form, each $x$ has a separate model (with parameters given by $\\theta_x$). However, since we can observe only one sample for each $x$, recovering parameters $\\theta_{j, x}$ is impossible. Thus the authors in [4] instead assume strong smoothness: that is, **that the feature space can be partitioned and the accuracy in each partition is uniform.** This is still a strong assumption. While this model incorporated the input feature space, nevertheless accuracy tends to drop as data points get far from high accuracy centers---even within parts. This observation led to our newly-proposed model. \n\n    **iii) Proposed model**\n    $$P(\\lambda^1(z), \\ldots, \\lambda^m(z), y) = \\frac{1}{Z}\\exp\\left(\\theta_yy + \\sum_{j=1}^m \\frac{\\theta_{j}}{1+d(x^{\\text{center}_j}, g_k(z))} \\lambda^j(g_k(z)) y\\right)$$\n    Our label model captures the phenomenon in Figure 2 of our draft, which shows that the accuracy of each LF drops as it moves away from its center. This means that our model neither require universal uniformity, as (i) does, nor per-part uniformity, as (ii) does. It makes the much weaker assumption of the existence of a center point where LFs are most accurate---which indeed matches practical LF scenarios, especially for LFs that are programs expressing heuristics. As a result, **our model not only has less restrictive assumptions, but also provides a framework that interprets unfairness as drift away from centers as an outcome of a group transformation.** \n    \n\n    \n[1] Ratner, Alexander, et al. \"Snorkel: Rapid training data creation with weak supervision.\" VLDB 2017.\n\n[2] Fu, Daniel, et al. \"Fast and three-rious: Speeding up weak supervision with triplet methods.\" ICML 2020.\n\n[3] Ratner, Alexander, et al. \"Training complex models with multi-task weak supervision.\" AAAI 2019.\n\n[4] Chen, Mayee F., et al. \"Shoring up the foundations: Fusing model embeddings and weak supervision.\", UAI 2022."}, {"review_id": "gsCULNSbv0", "reviewer": "Reviewer_6e6W", "summary": "This work found that unfair LFs in programmatic weak supervision could introduce bias to the resultant training labels, and proposed to address the bias via source bias mitigation and provided theoretical guarantee. Experimental results show that the effectiveness of the approaches in both synthetic and real datasets.", "questions": "It is not intuitive how the improvement of fairness and performance can be achieved at the same time. Usually, there is a trade-off between performance and fairness in ML models, could you explain why and how which is not the case in this work?", "limitations": "see weakness above", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. This work studies an important yet overlooked problem in programmatic weak supervision: the biases induced by unfair LFs\n2. The proposed method, compatible with traditional fair ML methods, could mitigate biases and improve the performance at the same time.\n3. The theoretical results are convincing, which show that the LF bias could be arbitrary yet can be fixed by the proposed method under some conditions", "weaknesses": "I am not aware of any major weakness except that the proposed model, if I understand it correctly, is a new label model built on an existing one, which means users have to use the proposed label model if they want to mitigate the biases. It is unclear how biases should be mitigated if users prefer other choice of label model.\n\nIn terms of label models that incorporate feature vector, one related work is missing: \"Leveraging instance features for label aggregation in programmatic weak supervision\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the thoughtful comments and reference. We will add the suggested paper in our updated draft.\n* **On the choice of label models**: A remarkable property of our label model is that **it is only used for one step in our overall pipeline: mitigating source bias**. Once this step is complete, any other label model can be used for aggregating sources into pseudolabels. Our WRENCH experiment (Table 5) is conducted with the newly-introduced Hyper Label Model [1] after applying SBM.\n* **On the trade-off between performance and fairness in ML models**: We  discuss this question in our common response and provide more detail here.  The trade-off between performance and fairness in machine learning models is usually caused by inherent fairness violations in datasets. Models trained on such datasets are unfair, and so fairness techniques seek to mitigate this, but ultimately face constraints. \n\n    On the other hand, **our method tackles bias induced by creating labeled datasets via weak supervision**. Weak supervision allows for the efficient creation of labeled datasets by using multiple noisy sources of signal for labels. In addition to being noisy, these sources are often far more biased than hand-labeled datasets. As a result, a dataset created through weak supervision may prove to be highly unfair even if its counterpart with ground-truth labels was perfectly fair. \n\n    This problem is also an opportunity. Our approach is to model where bias is induced within weak supervision sources, remove this bias, and as a result, improve both fairness and performance, while keeping the advantage of using weak supervision---exploiting cheap label sources omnivorosly.\n\n[1] Wu, Renzhi, et al. \"Learning Hyper Label Model for Programmatic Weak Supervision.\", ICLR 2022."}]

Author rebuttal:
We thank all of the reviewers for their kind comments and feedback. Reviewers recognized the strengths of our paper, which we briefly reiterate before we dive into in-depth responses.
* **This is the first study of fairness in weak supervision** (Reviewers wmSM, UYjh, GLUc, 6e6W).
* Our approach offers **theoretical results** showing that (1) even when a dataset with ground-truth labels is fair, **a weakly-supervised counterpart can be arbitrarily biased** and (2) a finite-sample **recovery result for the correction algorithm** (Reviewers wmSM, UYjh, GLUc, 6e6W).
* **Strong empirical results on synthetic and real datasets** (Reviewers UYjh, GLUc).
* The proposed method is **compatible with the existing fair machine learning algorithms** (Reviewers wmSM, 6e6W).

We address two common questions before proceeding to individual responses.
* **On advantages and limitations of our formulation** (Reviewers UYjh,  6e6W): First, we highlight how **bias in weak supervision differs** from typical supervised learning settings. In supervised learning, datasets have inherent fairness violations and fairness-aware methods try to train a fair model from unfair data. Such methods typically use constrained optimization, which tries to fit the training data under a fairness constraint. This usually entails tradeoffs between fairness and performance. The type of unfairness we tackle is **different**. It is an artifact of the WS process (i.e., how the noisy labels are generated) and may not be inherent in datasets (Theorem 1 in our work characterizes a range of such scenarios). As a result, it is possible to **i) maintain the advantages of using WS (cheap labeling), ii) improve fairness, and iii) improve performance**. Our method uses a simple counterfactual mapping to seek to achieve all of these benefits simultaneously. 

    One potential drawback appears to be that high-quality estimation of the counterfactual mapping may be difficult. Fortunately, scenarios where this is the case are not common in weak supervision, since the standard setting involves using large unlabeled input datasets. This allows for small estimation error in the counterfactual mapping. As a result, we do not observe this limitation in any of the practical settings of interest. Even when this limitation is operative (i.e., when using small unlabeled datasets), it is still possible to overcome it by applying recent techniques in optimal transport [1,2] that enable improving the estimation of the counterfactual mapping. 
    
    This is done, for example, by exploiting a handful of matched _keypoints_. Such methods can be easily applied if practitioners have some intuition about group matchings (e.g. toxic comment matching in different languages)---which is often the case in weak supervision scenarios. 

* **On the relationship between the fairness and the number of labeling functions** (Reviewers UYjh, wmSM): Several reviewers hypothesize that diversifying label sources naturally resolves unfairness. Indeed, we had a similar belief early on in our study. Unfortunately, we ultimately observed the opposite scenario:  diversification can increase unfairness in practice. To demonstrate this idea, we conducted experiments varying the number of labeling functions and their parameters. We include one such case for synthetic data, depicting it in the figure in A.1. It shows that **in standard WS, increasing the number of LFs increases bias** as the accuracy improves. Meanwhile, **our approach improves fairness _and_ performance** when increasing the number of labeling functions.

[1] Gu, Xiang, et al. "Keypoint-guided optimal transport with applications in heterogeneous domain adaptation.", NeurIPS 2022.
    
[2] Panda, Nishant, et al. "Semi-supervised Learning of Pushforwards For Domain Translation & Adaptation." , arxiv 2023.

Meta-review: All reviewers found the work interesting and relevant. After the author’s response, the reviewers did not see the need for further clarifications. All reviewers recommend acceptance.
Decision: Accept (poster)