SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bank marketing Similarly, text sentences for each row r are generated using the following template. "This person is <r.age> years old. This person works in a <r.job> job. This person is <r.marital_status>. This person has a <r.educationdegree> degree. This person <if r.has_housing_loan "has" else "doesn't have"> a housing loan. This person <if r.has和个人的loan "has" else "doesn't have"> a personal loan. This person's contact is <r(contact_type>. The last contact was on <r.last_contact weekday> in <r.last_contact_month>. The call lasted for <r.duration> seconds. <r_campaign> times of contacts performed during this campaign. Before this campaign, <rprevious> times of contacts have been performed for this person. The employment variation rate of this person is <r.emp_var_rate>. The consumer price index of this person is <r.cons_price_idx>%. The consumer confidence index of this person is <r.cons_conf_idx>%. The euribor 3 month rate of this person is <r.euribor3m>. The number of employees is <r.nr_employed)".
An example sentence is "This person is 37 years old. This person works in a management job. This person is married. This person has a university degree. This person doesn't have a housing loan. This person has a personal loan. This person's contact is a cellular. The last contact was on a Thursday in May. The call lasted for 195 seconds. 1 times of contacts performed during this campaign. Before this campaign, 0 times of contacts have been performed for this person. The employment variation rate of this person is  $-1.8\%$ . The consumer price index of this person is  $92.893\%$ . The consumer confidence index of this person is  $-46.2\%$ . The euribor 3 month rate of this person is  $1.327\%$ . The number of employees is 5099."
E.3 LF details
In this subsection, we describe the labeling functions used in the experiments. The performances of labeling functions, including our SBM results, are reported in Table  $8\sim 19$  . In SBM result tables, performance, fairness improvement by SBM are green colored.
Adult We generated heuristic labeling functions based on the features as follows.
- LF 1 (age LF): True if the age is between 30 and 60. False otherwise.
- LF2 (education LF): True if the person has a Bachelor, Master, PhD degree. False otherwise
- LF3 (marital LF): True if marital status is married. False otherwise.
- LF 4 (relationship LF): True if relationship status is Wife, Own-child, or Husband, False otherwise.
- LF 5 (capital LF): True if the capital gain is  $>5000$ . False otherwise.
- LF 6 (race LF): True if Asian-Pac-Islander or race-Other. False otherwise.
- LF 7 (country LF): True if Germany, Japan, Greece, or China. False otherwise.
- LF 8 (workclass LF): True if the workclass is Self employed, federal government, local government, or state government. False otherwise.
- LF 9 (occupation LF): True if the occupation is sales, executive managerial, professional, or machine operator. False otherwise.
Bank marketing Similar to Adult dataset, we generated heuristic labeling functions based on the features as follows.
- LF 1 (loan LF): True if the person has loan previously. False otherwise.
- LF 2 (previous contact LF): True if the previous contact number is  $>1.1$ . False otherwise.
- LF 3 (duration LF): True if the duration of bank marketing phone call is  $>6$  min. False otherwise.
- LF4 (marital LF): True if marital status is single. False otherwise.
- LF 5 (previous outcome LF): True if the previous campaign was successful. False otherwise.
- LF 6 (education LF): True if the education level is university degree or professional course taken. False otherwise.
CivilComments We generated heuristic labeling functions based on the inclusion of specific word lists. If a comment has a word that is included in the word list of LF, it gets True for that LF.
- LF 1 (sex LF): ["man", "men", "woman", "women", "male", "female", "guy", "boy", "girl", "daughter", "sex", "gender", "husband", "wife", "father", "mother", "rape", "mr.", "feminist", "feminism", "pregnant", "misogy", "pussy", "penis", "vagina", "butt", "dick"]
- LF 2 (LGBTQ LF): ["gay", "lesbian", "transgender", "homosexual", "homophobic", "heterosexual", "anti-gay", "same-sex", "bisexual", "biological"]
- LF 3 (religion LF): ["muslim", "catholic", "church", "christian", "god", "jesus", "christ", "jew", "islam", "israel", "bible", "bishop", "gospel", "clergy", "protestant", "islam"]
- LF4 (race LF): ["white", "black", "racist", "trump", "supremacist", "american", "canada", "kkk", "nazi", "facist", "african", "non-white", "discrimination", "hate", "neo-nazi", "asia", "china", "chinese"]
- LF 5 (toxic LF): ["crap", "damn", "bitch", "ass", "fuck", "bullshit", "hell", "jerk"]
- LF 6 (threat LF): ["shoot", "kill", "shot", "burn", "stab", "murder", "gun", "fire", "rape", "punch", "hurt", "hunt", "bullet", "hammer"]
- LF 7 (insult LF): ["stupid", "idiot", "dumb", "liar", "poor", "disgusting", "moron", "nasty", "lack", "brain", "incompetent", "sociopath"]
HateXplain We used heuristic the same labeling functions with CivilComments. However, their performance was close to random guess (accuracy close to 0.5), so we added 5 pretrained model LFs from detoxify [HU20] repository. We used models listed as original, unbiased, multilingual, original-small, unbiased-small.
CelebA We used models pretrained from other datasets as LFs.
- LF 1: ResNet-18 fine-tuned on gender classification dataset
- LF2: ResNet-34 fine-tuned on FairFace dataset
- LF 3: ResNet-34 fine-tuned on UTKFace dataset
- LF4: ResNet-34 fine-tuned on UTKFace dataset (White only)
- LF 5: ResNet-34 fine-tuned on UTKFace dataset (non-White only)
- LF6: ResNet-34 fine-tuned on UTKFace dataset (age  $\geq 50$  only)
<sup>1</sup>https://www.kaggle.com/datasets/cashutosh/gender-classification-dataset
Table 8: Tabular dataset raw LF performance
Dataset	LF	Acc	F1	ΔDP	ΔEO
Adult	LF 1 (age LF)	0.549	0.476	0.100	0.023
LF 2 (education LF)	0.743	0.455	0.033	0.044
LF 3 (marital LF)	0.699	0.579	0.447	0.241
LF 4 (relationship LF)	0.564	0.486	0.381	0.243
LF 5 (capital LF)	0.800	0.315	0.035	0.019
LF 6 (race LF)	0.737	0.066	0.003	0.004
LF 7 (country LF)	0.756	0.024	0.001	0.004
LF 8 (workclass LF)	0.678	0.399	0.066	0.003
LF 9 (occupation LF)	0.644	0.466	0.013	0.012
Bank Marketing	LF 1 (loan LF)	0.769	0.124	0.004	0.013
LF 2 (previous contact LF)	0.888	0.187	0.055	0.068
LF 3 (duration LF)	0.815	0.415	0.019	0.125
LF 4 (marital LF):	0.682	0.197	0.602	0.643
LF 5 (previous outcome LF)	0.898	0.301	0.052	0.062
LF 6 (education LF)	0.575	0.206	0.183	0.219
- LF7: ResNet-34 fine-tuned on UTKFace dataset (age  $< 50$  only)
UTKFace We used models pretrained from other datasets as LFs.
- LF 1: ResNet-18 fine-tuned on gender classification dataset
- LF 2: ResNet-34 fine-tuned on gender classification dataset
- LF 3: ResNet-34 fine-tuned on CelebA dataset
- LF4: ResNet-34 fine-tuned on CelebA dataset (attractive only)
- LF 5: ResNet-34 fine-tuned on CelebA dataset (non-attractive only)
- LF6: ResNet-34 fine-tuned on CelebA dataset (young only)
- LF 7: ResNet-34 fine-tuned on CelebA dataset (non-young only)
- LF 8: ResNet-34 fine-tuned on CelebA dataset (unfair sampling)
Table 9: Tabular dataset SBM (w/o OT) LF performance.  $\Delta \mathrm{s}$  are obtained by comparison with raw LF performance.
Dataset	LF	Acc (Δ)	F1 (Δ)	ΔDP (Δ)	ΔEO (Δ)
Adult	LF 1	0.597 (0.048)	0.594 (0.118)	0.119 (0.019)	0.106 (0.083)
LF 2	0.601 (-0.142)	0.346 (-0.109)	0.035 (0.002)	0.019 (-0.025)
LF 3	0.998 (0.299)	0.998 (0.419)	0.303 (-0.144)	0.001 (-0.240)
LF 4	0.719 (0.155)	0.723 (0.237)	0.304 (-0.077)	0.091 (-0.152)
LF 5	0.649 (-0.151)	0.183 (-0.132)	0.035 (0.000)	0.023 (0.004)
LF 6	0.615 (-0.122)	0.082 (0.016)	0.003 (0.000)	0.029 (0.025)
LF 7	0.621 (-0.135)	0.023 (-0.001)	0.001 (0.000)	0.005 (0.001)
LF 8	0.612 (-0.066)	0.318 (-0.021)	0.019 (-0.047)	0.027 (0.024)
LF 9	0.565 (-0.079)	0.460 (-0.006)	0.013 (0.000)	0.026 (0.014)
Adult (LIFT)	LF 1	0.506 (-0.043)	0.415 (-0.061)	0.309 (0.209)	0.198 (0.175)
LF 2	0.861 (0.118)	0.691 (0.236)	0.080 (0.047)	0.126 (0.082)
LF 3	0.691 (-0.008)	0.188 (-0.391)	0.130 (-0.317)	0.186 (-0.055)
LF 4	0.391 (-0.173)	0.307 (-0.179)	0.455 (0.074)	0.314 (0.071)
LF 5	0.725 (-0.075)	0.117 (-0.198)	0.014 (-0.021)	0.029 (0.010)
LF 6	0.703 (-0.034)	0.109 (0.043)	0.003 (0.000)	0.004 (0.000)
LF 7	0.708 (-0.048)	0.036 (0.012)	0.001 (0.000)	0.000 (-0.004)
LF 8	0.897 (0.219)	0.800 (0.461)	0.033 (-0.033)	0.235 (0.232)
LF 9	0.601 (-0.043)	0.446 (-0.020)	0.013 (0.000)	0.068 (0.056)
Bank	LF 1	0.706 (-0.063)	0.165 (0.041)	0.004 (0.000)	0.003 (-0.010)
LF 2	0.824 (-0.064)	0.221 (0.034)	0.037 (-0.018)	0.113 (0.045)
LF 3	0.931 (0.116)	0.829 (0.414)	0.019 (0.000)	0.110 (-0.015)
LF 4	0.427 (-0.255)	0.411 (0.214)	0.100 (-0.502)	0.004 (-0.639)
LF 5	0.833 (-0.065)	0.287 (-0.014)	0.061 (0.009)	0.195 (0.133)
LF 6	0.624 (0.049)	0.172 (-0.034)	0.009 (-0.174)	0.046 (-0.173)
Bank (LIFT)	LF 1	0.768 (-0.001)	0.105 (-0.019)	0.000 (-0.003)	0.013 (0.000)
LF 2	0.888 (0.000)	0.188 (0.001)	0.037 (-0.017)	0.068 (0.000)
LF 3	0.815 (0.000)	0.415 (0.000)	0.019 (0.000)	0.125 (0.000)
LF 4	0.317 (-0.365)	0.228 (0.031)	0.100 (-0.502)	0.065 (-0.578)
LF 5	0.898 (0.000)	0.303 (0.001)	0.061 (0.009)	0.086 (0.023)
LF 6	0.680 (0.105)	0.126 (-0.081)	0.009 (-0.175)	0.084 (-0.134)
Table 10: Tabular dataset SBM (OT-L) LF performance.  $\Delta \mathrm{s}$  are obtained by comparison with raw LF performance.
Dataset	LF	Acc (Δ)	F1 (Δ)	ΔDP (Δ)	ΔEO (Δ)
Adult	LF 1	0.841 (0.292)	0.846 (0.370)	0.654 (0.554)	0.733 (0.710)
LF 2	0.334 (-0.409)	0.000 (-0.455)	0.208 (0.175)	0.000 (-0.044)
LF 3	0.346 (-0.353)	0.000 (-0.579)	0.174 (-0.273)	0.000 (-0.241)
LF 4	0.895 (0.331)	0.904 (0.418)	0.735 (0.354)	0.824 (0.581)
LF 5	0.394 (-0.406)	0.000 (-0.315)	0.027 (-0.008)	0.000 (-0.019)
LF 6	0.408 (-0.329)	0.070 (0.004)	0.003 (0.000)	0.037 (0.033)
LF 7	0.405 (-0.351)	0.019 (-0.005)	0.001 (0.000)	0.010 (0.006)
LF 8	0.336 (-0.342)	0.000 (-0.339)	0.202 (0.136)	0.000 (-0.003)
LF 9	0.501 (-0.143)	0.512 (0.046)	0.013 (0.000)	0.439 (0.427)
Adult (LIFT)	LF 1	0.628 (0.079)	0.655 (0.179)	0.036 (-0.064)	0.027 (0.004)
LF 2	0.780 (0.037)	0.664 (0.209)	0.014 (-0.019)	0.032 (-0.012)
LF 3	0.525 (-0.174)	0.296 (-0.283)	0.097 (-0.350)	0.013 (-0.228)
LF 4	0.497 (-0.067)	0.554 (0.068)	0.137 (-0.244)	0.150 (-0.093)
LF 5	0.606 (-0.194)	0.177 (-0.138)	0.023 (-0.012)	0.042 (0.023)
LF 6	0.565 (-0.172)	0.089 (0.023)	0.003 (0.000)	0.008 (0.004)
LF 7	0.567 (-0.189)	0.029 (0.005)	0.001 (0.000)	0.000 (-0.004)
LF 8	0.618 (-0.060)	0.409 (0.070)	0.011 (-0.055)	0.017 (0.014)
LF 9	0.843 (0.199)	0.817 (0.351)	0.013 (0.000)	0.046 (0.034)
Bank	LF 1	0.818 (0.049)	0.062 (-0.062)	0.004 (0.000)	0.140 (0.127)
LF 2	0.980 (0.092)	0.703 (0.516)	0.024 (-0.031)	0.542 (0.474)
LF 3	0.777 (-0.038)	0.093 (-0.322)	0.019 (0.000)	0.265 (0.139)
LF 4	0.047 (-0.635)	0.083 (-0.114)	0.131 (-0.471)	1.000 (0.357)
LF 5	0.988 (0.090)	0.839 (0.538)	0.032 (-0.020)	0.723 (0.660)
LF 6	0.950 (0.375)	0.000 (-0.206)	0.244 (0.061)	0.000 (-0.219)
Bank (LIFT)	LF 1	0.125 (-0.645)	0.197 (0.073)	0.853 (0.849)	0.867 (0.854)
LF 2	0.888 (0.000)	0.180 (-0.007)	0.000 (-0.055)	0.036 (-0.032)
LF 3	0.815 (0.000)	0.415 (0.000)	0.018 (-0.001)	0.137 (0.012)
LF 4	0.876 (0.194)	0.085 (-0.112)	0.869 (0.268)	0.954 (0.311)
LF 5	0.898 (0.000)	0.300 (-0.002)	0.047 (-0.005)	0.039 (-0.023)
Table 11: Tabular dataset SBM (OT-S) LF performance.  $\Delta \mathrm{s}$  are obtained by comparison with raw LF performance.
Dataset	LF	Acc (Δ)	F1 (Δ)	ΔDP (Δ)	ΔEO (Δ)
Adult	LF 1	0.604 (0.055)	0.595 (0.119)	0.145 (0.045)	0.125 (0.102)
LF 2	0.601 (-0.142)	0.340 (-0.115)	0.036 (0.003)	0.015 (-0.029)
LF 3	0.998 (0.299)	0.998 (0.419)	0.293 (-0.154)	0.002 (-0.239)
LF 4	0.690 (0.126)	0.700 (0.214)	0.221 (-0.160)	0.050 (-0.193)
LF 5	0.656 (-0.144)	0.181 (-0.134)	0.032 (-0.003)	0.021 (0.002)
LF 6	0.621 (-0.116)	0.081 (0.015)	0.003 (0.000)	0.030 (0.026)
LF 7	0.627 (-0.129)	0.023 (-0.001)	0.001 (0.000)	0.005 (0.001)
LF 8	0.607 (-0.071)	0.256 (-0.083)	0.069 (0.003)	0.095 (0.092)
LF 9	0.558 (-0.086)	0.446 (-0.020)	0.013 (0.000)	0.014 (0.002)
Adult (LIFT)	LF 1	0.475 (-0.074)	0.436 (-0.040)	0.031 (-0.069)	0.032 (0.009)
LF 2	0.953 (0.210)	0.910 (0.455)	0.047 (0.014)	0.079 (0.035)
LF 3	0.712 (0.013)	0.191 (-0.388)	0.157 (-0.290)	0.250 (0.009)
LF 4	0.373 (-0.191)	0.425 (-0.061)	0.204 (-0.177)	0.120 (-0.123)
LF 5	0.764 (-0.036)	0.301 (-0.014)	0.035 (0.000)	0.107 (0.088)
LF 6	0.711 (-0.026)	0.117 (0.051)	0.003 (0.000)	0.018 (0.014)
LF 7	0.715 (-0.041)	0.037 (0.013)	0.001 (0.000)	0.002 (-0.002)
LF 8	0.716 (0.038)	0.272 (-0.067)	0.147 (0.081)	0.284 (0.281)
LF 9	0.640 (-0.004)	0.495 (0.029)	0.013 (0.000)	0.213 (0.201)
Bank	LF 1	0.694 (-0.075)	0.173 (0.049)	0.004 (0.000)	0.017 (0.004)
LF 2	0.806 (-0.082)	0.209 (0.022)	0.059 (0.004)	0.211 (0.143)
LF 3	0.949 (0.134)	0.881 (0.466)	0.019 (0.000)	0.136 (0.011)
LF 4	0.351 (-0.331)	0.382 (0.185)	0.040 (-0.562)	0.016 (-0.627)
LF 5	0.814 (-0.084)	0.267 (-0.034)	0.069 (0.017)	0.247 (0.185)
LF 6	0.583 (0.008)	0.020 (-0.186)	0.040 (-0.143)	0.103 (-0.116)
Bank (LIFT)	LF 1	0.769(-0.001)	0.087 (-0.038)	0.006 (0.003)	0.038 (0.025)
LF 2	0.888 (0.001)	0.204 (0.017)	0.141 (0.087)	0.305 (0.237)
LF 3	0.814 (-0.001)	0.409 (-0.006)	0.054 (0.035)	0.357 (0.231)
LF 4	0.307 (-0.375)	0.229 (0.032)	0.085 (-0.517)	0.042 (-0.600)
LF 5	0.898 (0.000)	0.311 (0.010)	0.133 (0.081)	0.236 (0.173)
LF 6	0.677 (0.102)	0.105 (-0.102)	0.003 (-0.180)	0.122 (-0.096)
Table 12: NLP dataset raw LF performance
Dataset	LF	Acc	F1	ΔDP	ΔEO
CivilComments	LF 1 (sex LF)	0.755	0.187	0.046	0.019
LF 2 (LGBTQ LF)	0.877	0.073	0.001	0.017
LF 3 (religion LF)	0.861	0.049	0.012	0.013
LF 4 (race LF)	0.847	0.234	0.634	0.574
LF 5 (toxic LF)	0.886	0.068	0.006	0.012
LF 6 (threat LF)	0.862	0.102	0.055	0.054
LF 7 (insult LF)	0.872	0.176	0.028	0.035
HateXplain	LF 1 (sex LF)	0.427	0.253	0.015	0.002
LF 2 (LGBTQ LF)	0.405	0.077	0.047	0.041
LF 3 (religion LF)	0.437	0.197	0.001	0.003
LF 4 (race LF)	0.419	0.327	0.139	0.168
LF 5 (toxic LF)	0.451	0.233	0.007	0.016
LF 6 (threat LF)	0.415	0.097	0.000	0.004
LF 7 (insult LF)	0.427	0.100	0.015	0.005
LF 8 (Detoxify - original)	0.645	0.704	0.165	0.086
LF 9 (Detoxify - unbiased)	0.625	0.668	0.150	0.078
LF 10 (Detoxify - multilingual)	0.649	0.700	0.168	0.077
LF 11 (Detoxify - original-small)	0.644	0.705	0.152	0.076
LF 12 (Detoxify - unbiased-small)	0.643	0.699	0.186	0.113
Table 13: NLP dataset SBM (w/o OT) LF performance.  $\Delta \mathrm{s}$  are obtained by comparison with raw LF performance.
Dataset	LF	Acc (Δ)	F1 (Δ)	ΔDP (Δ)	ΔEO (Δ)
Civil Comments	LF 1	0.790 (0.035)	0.325 (0.138)	0.048 (0.002)	0.041 (0.022)
LF 2	0.896 (0.019)	0.268 (0.195)	0.001 (0.000)	0.053 (0.036)
LF 3	0.868 (0.007)	0.155 (0.106)	0.012 (0.000)	0.043 (0.030)
LF 4	0.858 (0.011)	0.252 (0.018)	0.114 (-0.520)	0.160 (-0.414)
LF 5	0.886 (0.000)	0.137 (0.069)	0.006 (0.000)	0.003 (-0.009)
LF 6	0.916 (0.054)	0.482 (0.380)	0.023 (-0.032)	0.002 (-0.052)
LF 7	0.918 (0.046)	0.501 (0.325)	0.022 (-0.006)	0.012 (-0.023)
HateXplain	LF 1	0.483 (0.056)	0.273 (0.020)	0.015 (0.000)	0.001 (-0.001)
LF 2	0.473 (0.068)	0.058 (-0.019)	0.000 (-0.047)	0.004 (-0.037)
LF 3	0.460 (0.023)	0.163 (-0.034)	0.001 (0.000)	0.004 (0.001)
LF 4	0.481 (0.062)	0.328 (0.001)	0.044 (-0.095)	0.039 (-0.129)
LF 5	0.515 (0.064)	0.267 (0.034)	0.014 (0.007)	0.021 (0.005)
LF 6	0.471 (0.056)	0.106 (0.009)	0.000 (0.000)	0.015 (0.011)
LF 7	0.472 (0.045)	0.088 (-0.012)	0.012 (-0.003)	0.014 (0.009)
LF 8	0.831 (0.186)	0.864 (0.160)	0.006 (-0.159)	0.000 (-0.086)
LF 9	0.917 (0.292)	0.928 (0.260)	0.015 (-0.135)	0.000 (-0.078)
LF 10	0.864 (0.215)	0.888 (0.188)	0.009 (-0.159)	0.000 (-0.077)
LF 11	0.839 (0.195)	0.870 (0.165)	0.015 (-0.137)	0.000 (-0.076)
LF 12	0.837 (0.194)	0.868 (0.169)	0.014 (-0.172)	0.000 (-0.113)
Table 14: NLP dataset SBM (OT-L) LF performance.  $\Delta \mathrm{s}$  are obtained by comparison with raw LF performance.
Dataset	LF	Acc (Δ)	F1 (Δ)	ΔDP (Δ)	ΔEO (Δ)
Civil Comments	LF 1	0.791 (0.036)	0.321 (0.134)	0.003 (-0.043)	0.022 (0.003)
LF 2	0.898 (0.021)	0.272 (0.199)	0.001 (0.000)	0.020 (0.003)
LF 3	0.870 (0.009)	0.156 (0.107)	0.012 (0.000)	0.041 (0.028)
LF 4	0.860 (0.013)	0.244 (0.010)	0.017 (-0.617)	0.017 (-0.557)
LF 5	0.887 (0.001)	0.139 (0.071)	0.006 (0.000)	0.027 (0.015)
LF 6	0.917 (0.055)	0.484 (0.382)	0.012 (-0.043)	0.026 (-0.028)
LF 7	0.919 (0.047)	0.501 (0.325)	0.007 (-0.021)	0.013 (-0.022)
HateXplain	LF 1	0.457 (0.030)	0.279 (0.026)	0.015 (0.000)	0.001 (-0.001)
LF 2	0.433 (0.028)	0.062 (-0.015)	0.002 (-0.045)	0.006 (-0.035)
LF 3	0.429 (-0.008)	0.171 (-0.026)	0.001 (0.000)	0.001 (-0.002)
LF 4	0.457 (0.038)	0.323 (-0.004)	0.011 (-0.128)	0.009 (-0.159)
LF 5	0.479 (0.028)	0.263 (0.030)	0.017 (0.010)	0.029 (0.013)
LF 6	0.436 (0.021)	0.111 (0.014)	0.000 (0.000)	0.011 (0.007)
LF 7	0.434 (0.007)	0.091 (-0.009)	0.012 (-0.003)	0.017 (0.012)
LF 8	0.845 (0.200)	0.882 (0.178)	0.790 (0.093)	0.000 (-0.086)
LF 9	0.923 (0.298)	0.938 (0.270)	0.883 (0.179)	0.000 (-0.078)
LF 10	0.875 (0.226)	0.903 (0.203)	0.823 (0.111)	0.000 (-0.077)
LF 11	0.848 (0.204)	0.884 (0.179)	0.792 (0.098)	0.000 (-0.076)
LF 12	0.845 (0.202)	0.882 (0.183)	0.789 (0.090)	0.000 (-0.113)
Table 15: NLP dataset SBM (OT-S) LF performance.  $\Delta \mathrm{s}$  are obtained by comparison with raw LF performance.
Dataset	LF	Acc (Δ)	F1 (Δ)	ΔDP (Δ)	ΔEO (Δ)
Civil Comments	LF 1	0.791 (0.036)	0.320 (0.133)	0.015 (-0.031)	0.082 (0.063)
LF 2	0.897 (0.020)	0.271 (0.198)	0.001 (0.000)	0.029 (0.012)
LF 3	0.870 (0.009)	0.156 (0.107)	0.012 (0.000)	0.043 (0.030)
LF 4	0.860 (0.013)	0.245 (0.011)	0.017 (-0.617)	0.016 (-0.558)
LF 5	0.887 (0.001)	0.138 (0.070)	0.006 (0.000)	0.020 (0.008)
LF 6	0.917 (0.055)	0.482 (0.380)	0.011 (-0.044)	0.006 (-0.048)
LF 7	0.919 (0.047)	0.504 (0.328)	0.019 (-0.009)	0.039 (0.004)
HateXplain	LF 1	0.444 (0.017)	0.277 (0.024)	0.015 (0.000)	0.002 (0.000)
LF 2	0.417 (0.012)	0.056 (-0.021)	0.001 (-0.046)	0.001 (-0.040)
LF 3	0.418 (-0.019)	0.172 (-0.025)	0.001 (0.000)	0.002 (-0.001)
LF 4	0.448 (0.029)	0.332 (0.005)	0.032 (-0.107)	0.045 (-0.123)
LF 5	0.459 (0.008)	0.256 (0.023)	0.030 (0.023)	0.041 (0.025)
LF 6	0.422 (0.007)	0.110 (0.013)	0.000 (0.000)	0.011 (0.007)
LF 7	0.418 (-0.009)	0.091 (-0.009)	0.018 (0.003)	0.025 (0.020)
LF 8	0.851 (0.206)	0.889 (0.185)	0.056 (-0.109)	0.000 (-0.086)
LF 9	0.927 (0.302)	0.942 (0.274)	0.061 (-0.089)	0.000 (-0.078)
LF 10	0.884 (0.235)	0.911 (0.211)	0.052 (-0.116)	0.000 (-0.077)
LF 11	0.853 (0.209)	0.890 (0.185)	0.056 (-0.096)	0.000 (-0.076)
LF 12	0.847 (0.204)	0.886 (0.187)	0.063 (-0.123)	0.000 (-0.113)
Table 16: Vision dataset raw LF performance
Dataset	LF	Acc	F1	ΔDP	ΔEO
CelebA	LF 1 (ResNet-18 fine-tuned on gender classification dataset)	0.798	0.794	0.328	0.284
LF 2 (ResNet-34 fine-tuned on FairFace dataset)	0.890	0.901	0.314	0.105
LF 3 (ResNet-34 fine-tuned on UTKFace dataset)	0.826	0.831	0.309	0.195
LF 4 (ResNet-34 fine-tuned on UTKFace dataset (White only))	0.825	0.832	0.277	0.131
LF 5 (ResNet-34 fine-tuned on UTKFace dataset (non-White only))	0.818	0.832	0.271	0.134
LF 6 (ResNet-34 fine-tuned on UTKFace dataset (age ≥ 50 only)	0.764	0.750	0.279	0.194
LF 7 (ResNet-34 fine-tuned on UTKFace dataset (age < 50 only))	0.830	0.845	0.299	0.175
UTKFace	LF 1 (ResNet-18 fine-tuned on gender classification dataset)	0.869	0.856	0.060	0.039
LF 2 (ResNet-34 fine-tuned on gender classification dataset)	0.854	0.842	0.060	0.060
LF 3 (ResNet-34 fine-tuned on CelebA dataset)	0.742	0.758	0.158	0.032
LF 4 (ResNet-34 fine-tuned on CelebA dataset (attractive only))	0.580	0.692	0.065	0.002
LF 5 (ResNet-34 fine-tuned on CelebA dataset (non-attractive only))	0.687	0.608	0.129	0.034
LF 6 (ResNet-34 fine-tuned on CelebA dataset (young only))	0.664	0.729	0.116	0.012
LF 7 (ResNet-34 fine-tuned on CelebA dataset (non-young only))	0.619	0.429	0.136	0.081
LF 8 (ResNet-34 fine-tuned on CelebA dataset (unfair sampling))	0.631	0.676	0.113	0.053
Table 17: Vision dataset SBM LF (w/o OT) performance.  $\Delta \mathrm{s}$  are obtained by comparison with raw LF performance.
Dataset	LF	Acc (Δ)	F1 (Δ)	ΔDP (Δ)	ΔEO (Δ)
CelebA	LF 1	0.847 (0.049)	0.832 (0.038)	0.328 (0.000)	0.267 (-0.017)
LF 2	0.890 (0.000)	0.895 (-0.006)	0.314 (0.000)	0.101 (-0.004)
LF 3	0.926 (0.100)	0.923 (0.092)	0.309 (0.000)	0.097 (-0.098)
LF 4	0.914 (0.089)	0.912 (0.080)	0.277 (0.000)	0.027 (-0.104)
LF 5	0.899 (0.081)	0.900 (0.068)	0.271 (0.000)	0.030 (-0.104)
LF 6	0.705 (-0.059)	0.629 (-0.121)	0.177 (-0.102)	0.052 (-0.142)
LF 7	0.913 (0.083)	0.915 (0.070)	0.299 (0.000)	0.056 (-0.119)
UTKFace	LF 1	0.929 (0.060)	0.924 (0.068)	0.102 (0.042)	0.011 (-0.028)
LF 2	0.939 (0.085)	0.935 (0.093)	0.102 (0.042)	0.007 (-0.053)
LF 3	0.631 (-0.111)	0.678 (-0.080)	0.078 (-0.080)	0.034 (0.002)
LF 4	0.549 (-0.031)	0.681 (-0.011)	0.017 (-0.048)	0.002 (0.000)
LF 5	0.740 (0.053)	0.679 (0.071)	0.129 (0.000)	0.040 (0.006)
LF 6	0.694 (0.030)	0.755 (0.026)	0.116 (0.000)	0.037 (0.025)
LF 7	0.694 (0.075)	0.541 (0.112)	0.061 (-0.075)	0.033 (-0.048)
LF 8	0.591 (-0.040)	0.654 (-0.022)	0.071 (-0.042)	0.054 (0.001)
Table 18: Vision dataset SBM (OT-L) LF performance.  $\Delta \mathrm{s}$  are obtained by comparison with raw LF performance.
Dataset	LF	Acc (Δ)	F1 (Δ)	ΔDP (Δ)	ΔEO (Δ)
CelebA	LF 1	0.847 (0.049)	0.832 (0.038)	0.328 (0.000)	0.268 (-0.016)
LF 2	0.890 (0.000)	0.894 (-0.007)	0.314 (0.000)	0.102 (-0.003)
LF 3	0.926 (0.100)	0.922 (0.091)	0.309 (0.000)	0.098 (-0.097)
LF 4	0.915 (0.090)	0.913 (0.081)	0.277 (0.000)	0.029 (-0.102)
LF 5	0.898 (0.080)	0.900 (0.068)	0.271 (0.000)	0.031 (-0.103)
LF 6	0.648 (-0.116)	0.498 (-0.252)	0.059 (-0.220)	0.217 (0.023)
LF 7	0.914 (0.084)	0.916 (0.071)	0.299 (0.000)	0.058 (-0.117)
UTKFace	LF 1	0.931 (0.062)	0.925 (0.069)	0.026 (-0.034)	0.012 (-0.027)
LF 2	0.945 (0.091)	0.940 (0.098)	0.017 (-0.043)	0.006 (-0.054)
LF 3	0.599 (-0.143)	0.667 (-0.091)	0.004 (-0.154)	0.001 (-0.031)
LF 4	0.523 (-0.057)	0.667 (-0.025)	0.008 (-0.057)	0.002 (0.000)
LF 5	0.738 (0.051)	0.674 (0.066)	0.129 (0.000)	0.029 (-0.005)
LF 6	0.691 (0.027)	0.752 (0.023)	0.116 (0.000)	0.037 (0.025)
LF 7	0.690 (0.071)	0.525 (0.096)	0.000 (-0.136)	0.016 (-0.065)
LF 8	0.572 (-0.059)	0.655 (-0.021)	0.001 (-0.112)	0.014 (-0.039)
Table 19: Vision dataset SBM (OT-S) LF performance.  $\Delta \mathrm{s}$  are obtained by comparison with raw LF performance.
Dataset	LF	Acc (Δ)	F1 (Δ)	ΔDP (Δ)	ΔEO (Δ)
CelebA	LF 1	0.850 (0.052)	0.835 (0.041)	0.328 (0.000)	0.266 (-0.018)
LF 2	0.893 (0.003)	0.897 (-0.004)	0.314 (0.000)	0.101 (-0.004)
LF 3	0.923 (0.097)	0.920 (0.089)	0.309 (0.000)	0.098 (-0.097)
LF 4	0.913 (0.088)	0.910 (0.078)	0.277 (0.000)	0.029 (-0.102)
LF 5	0.901 (0.083)	0.903 (0.071)	0.271 (0.000)	0.031 (-0.103)
LF 6	0.618 (-0.146)	0.430 (-0.320)	0.016 (-0.263)	0.282 (0.088)
LF 7	0.911 (0.081)	0.913 (0.068)	0.299 (0.000)	0.059 (-0.116)
UTKFace	LF 1	0.931 (0.062)	0.926 (0.070)	0.010 (-0.050)	0.001 (-0.038)
LF 2	0.949 (0.095)	0.945 (0.103)	0.015 (-0.045)	0.001 (-0.059)
LF 3	0.577 (-0.165)	0.652 (-0.106)	0.003 (-0.155)	0.013 (-0.019)
LF 4	0.517 (-0.063)	0.664 (-0.028)	0.006 (-0.059)	0.007 (0.005)
LF 5	0.730 (0.043)	0.669 (0.061)	0.129 (0.000)	0.018 (-0.016)
LF 6	0.694 (0.030)	0.756 (0.027)	0.116 (0.000)	0.036 (0.024)
LF 7	0.683 (0.064)	0.523 (0.094)	0.010 (-0.126)	0.007 (-0.074)
LF 8	0.562 (-0.069)	0.652 (-0.024)	0.009 (-0.104)	0.012 (-0.041)
(a) Adult (raw)
(b) Bank Marketing (raw)
Figure 6: Identification of high accuracy regimes for tabular datasets.
E.4 Identification of centers
While our method improves performance by matching one group distribution and attempting to make these uniform, it does not imply accuracy improvement. A presumption of our method is that the group with high (estimated) accuracy possesses the high accuracy regime and our method can transport data points to this high accuracy regime while keeping their structure, which results in accuracy improvements. To empirically support this hypothesis, we used the following procedure and visualized the results in Figure 6, 8, 9.
1. Find the best accuracy center by evaluating the accuracy of the nearest  $10\%$  of data points for each center candidate point.
2. Expand  $2\%$  percent of data points closest to the center each time, compute their accumulated average accuracy (y-axis) and the farthest distance (x-axis) from the neighborhood
3. Find the group with better accuracy group and visualize their accumulated average accuracy (y-axis) and the farthest distance (x-axis) in each group.