SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We are able to obtain two insights from the visualization. First, the high accuracy regime typically exists in the group with the high estimated accuracy, which supports our hypothesis. Thus accuracy improvement by optimal transport can be justified. Secondly, the groups actually show the distributional difference in the input space  $\mathcal{X}$ . Given center points, lines in the high accuracy group start with a smaller distance to the center than the low accuracy group.
(a) Adult (LIFT)
(b) Bank Marketing (LIFT)
Figure 7: Identification of high accuracy regimes for tabular datasets (LIFT).
(a) CivilComments
(b) HateXplain
Figure 8: Identification of high accuracy regimes for NLP datasets.
(a) CelebA
(b)UTKFace
Figure 9: Identification of high accuracy regimes for vision datasets.
E.5 Compatibility with other fair ML methods
One advantage of our method is that we can use other successful fair ML methods in a supervised learning setting on top of SBM, since our method works in weak label sources while traditional fair ML methods work in the preprocessing/training/postprocessing steps, which are independent of the label model. To make this point, we tried traditional fair ML methods from fairlearn  $\left[\mathrm{BDE}^{+}20\right]$  with each of WS settings. We used CorrelationRemover, ExponentiatedGradient  $\left[\mathrm{ABD}^{+}18\right]$ , ThresholdOptimizer [HPS16] with the demographic parity (DP), equal opportunity (EO) as parity criteria, and accuracy as the performance criteria. The results are reported in Table 20 - 27. As expected, combining with other methods yields an accuracy-fairness tradeoff given weak label sources. Typically, SBM yields additional gains upon traditional fair ML methods. One another observation is that fair ML methods to modify equal opportunity typically fail to achieve the reduction of  $\Delta_{EO}$ . This can be interpreted as the result of the noise in the training set labels.
Table 20: SBM combined with other fair ML methods in Adult dataset
Fair ML method	Acc	F1	ΔDP	ΔEO
WS (Baseline)	N/A	0.717	0.587	0.475	0.325
correlation remover	0.716	0.587	0.446	0.287
optimal threshold (DP)	0.578	0.499	0.002	0.076
optimal threshold (EO)	0.721	0.563	0.404	0.217
exponentiated gradient (DP gap = 0)	0.582	0.502	0.002	0.066
exponentiated gradient (EO gap = 0)	0.715	0.585	0.445	0.284
SBM (w/o OT)	N/A	0.720	0.592	0.439	0.273
correlation remover	0.717	0.586	0.437	0.264
optimal threshold (DP)	0.591	0.507	0.003	0.059
optimal threshold (EO)	0.722	0.571	0.387	0.189
exponentiated gradient (DP gap = 0)	0.693	0.525	0.014	0.052
exponentiated gradient (EO gap = 0)	0.722	0.586	0.404	0.233
SBM (OT-L)	N/A	0.560	0.472	0.893	0.980
correlation remover	0.460	0.443	0.084	0.005
optimal threshold (DP)	0.324	0.404	0.006	0.089
optimal threshold (EO)	0.300	0.399	0.103	0.015
exponentiated gradient (DP gap = 0)	0.345	0.414	0.002	0.016
exponentiated gradient (EO gap = 0)	0.558	0.479	0.861	0.792
SBM (OT-S)	N/A	0.722	0.590	0.429	0.261
correlation remover	0.729	0.595	0.408	0.249
optimal threshold (DP)	0.596	0.507	0.003	0.050
optimal threshold (EO)	0.723	0.571	0.382	0.184
exponentiated gradient (DP gap = 0)	0.687	0.527	0.011	0.045
exponentiated gradient (EO gap = 0)	0.728	0.587	0.390	0.218
Table 21: SBM combined with other fair ML methods in Adult dataset (LIFT)
Fair ML method	Acc	F1	ΔDP	ΔEO
WS (Baseline)	N/A	0.711	0.584	0.449	0.290
correlation remover	0.716	0.587	0.446	0.287
optimal threshold (DP)	0.578	0.499	0.002	0.076
optimal threshold (EO)	0.721	0.563	0.404	0.217
exponentiated gradient (DP gap = 0)	0.582	0.502	0.002	0.066
exponentiated gradient (EO gap = 0)	0.715	0.585	0.445	0.284
SBM (w/o OT)	N/A	0.704	0.366	0.032	0.192
correlation remover	0.686	0.351	0.006	0.155
optimal threshold (DP)	0.707	0.363	0.007	0.133
optimal threshold (EO)	0.713	0.362	0.022	0.079
exponentiated gradient (DP gap = 0)	0.682	0.350	0.011	0.163
exponentiated gradient (EO gap = 0)	0.701	0.369	0.019	0.134
SBM (OT-L)	N/A	0.700	0.520	0.015	0.138
correlation remover	0.686	0.504	0.011	0.105
optimal threshold (DP)	0.701	0.520	0.008	0.124
optimal threshold (EO)	0.712	0.521	0.060	0.025
exponentiated gradient (DP gap = 0)	0.673	0.504	0.005	0.071
exponentiated gradient (EO gap = 0)	0.691	0.516	0.058	0.035
SBM (OT-S)	N/A	0.782	0.448	0.000	0.178
correlation remover	0.772	0.435	0.002	0.180
optimal threshold (DP)	0.782	0.447	0.001	0.176
optimal threshold (EO)	0.790	0.427	0.087	0.104
exponentiated gradient (DP gap = 0)	0.784	0.452	0.000	0.171
exponentiated gradient (EO gap = 0)	0.747	0.380	0.107	0.049
Table 22: SBM combined with other fair ML methods in Bank Marketing dataset
Fair ML method	Acc	F1	ΔDP	ΔEO
WS (Baseline)	N/A	0.674	0.258	0.543	0.450
correlation remover	0.890	0.057	0.002	0.006
optimal threshold (DP)	0.890	0.058	0.002	0.007
optimal threshold (EO)	0.890	0.066	0.030	0.040
exponentiated gradient (DP gap = 0)	0.889	0.039	0.000	0.009
exponentiated gradient (EO gap = 0)	0.890	0.070	0.033	0.051
SBM (w/o OT)	N/A	0.876	0.550	0.106	0.064
correlation remover	0.874	0.547	0.064	0.095
optimal threshold (DP)	0.876	0.547	0.031	0.208
optimal threshold (EO)	0.876	0.548	0.053	0.171
exponentiated gradient (DP gap = 0)	0.877	0.525	0.037	0.182
exponentiated gradient (EO gap = 0)	0.872	0.531	0.066	0.124
SBM (OT-L)	N/A	0.892	0.304	0.095	0.124
correlation remover	0.890	0.290	0.011	0.111
optimal threshold (DP)	0.891	0.280	0.008	0.163
optimal threshold (EO)	0.881	0.313	0.841	0.678
exponentiated gradient (DP gap = 0)	0.891	0.263	0.003	0.106
exponentiated gradient (EO gap = 0)	0.895	0.296	0.097	0.136
SBM (OT-S)	N/A	0.847	0.515	0.122	0.080
correlation remover	0.847	0.512	0.072	0.122
optimal threshold (DP)	0.846	0.511	0.043	0.236
optimal threshold (EO)	0.847	0.515	0.113	0.104
exponentiated gradient (DP gap = 0)	0.843	0.487	0.052	0.143
exponentiated gradient (EO gap = 0)	0.848	0.512	0.114	0.088
exponentiated gradient (DP gap = 0)	0.889	0.039	0.000	0.009
exponentiated gradient (EO gap = 0)	0.890	0.070	0.033	0.051
SBM (w/o OT)	N/A	0.698	0.255	0.088	0.137
correlation remover	0.836	0.358	0.025	0.114
optimal threshold (DP)	0.699	0.252	0.002	0.019
optimal threshold (EO)	0.698	0.253	0.006	0.028
exponentiated gradient (DP gap = 0)	0.687	0.262	0.014	0.053
exponentiated gradient (EO gap = 0)	0.654	0.226	0.096	0.107
SBM (OT-L)	N/A	0.892	0.305	0.104	0.121
correlation remover	0.891	0.304	0.079	0.000
optimal threshold (DP)	0.891	0.289	0.016	0.094
optimal threshold (EO)	0.892	0.305	0.103	0.121
exponentiated gradient (DP gap = 0)	0.893	0.265	0.001	0.093
exponentiated gradient (EO gap = 0)	0.892	0.305	0.100	0.109
SBM (OT-S)	N/A	0.698	0.080	0.109	0.072
correlation remover	0.699	0.081	0.230	0.106
optimal threshold (DP)	0.697	0.083	0.028	0.011
optimal threshold (EO)	0.695	0.089	0.174	0.205
exponentiated gradient (DP gap = 0)	0.681	0.113	0.032	0.063
exponentiated gradient (EO gap = 0)	0.691	0.124	0.041	0.036
Table 23: SBM combined with other fair ML methods in Bank Marketing dataset (LIFT)
Table 24: SBM combined with other fair ML methods in CivilComments dataset
Fair ML method	Acc	F1	ΔDP	ΔEO
WS (Baseline)	N/A	0.854	0.223	0.560	0.546
correlation remover	0.886	0.000	0.000	0.000
optimal threshold (DP)	0.886	0.000	0.000	0.000
optimal threshold (EO)	0.886	0.000	0.000	0.000
exponentiated gradient (DP gap = 0)	0.886	0.000	0.000	0.000
exponentiated gradient (EO gap = 0)	0.886	0.000	0.000	0.000
SBM (w/o OT)	N/A	0.879	0.068	0.048	0.047
correlation remover	0.878	0.062	0.010	0.030
optimal threshold (DP)	0.880	0.054	0.001	0.015
optimal threshold (EO)	0.880	0.061	0.019	0.010
exponentiated gradient (DP gap = 0)	0.881	0.046	0.002	0.008
exponentiated gradient (EO gap = 0)	0.880	0.059	0.018	0.010
SBM (OT-L)	N/A	0.880	0.070	0.042	0.039
correlation remover	0.879	0.056	0.011	0.028
optimal threshold (DP)	0.880	0.060	0.001	0.017
optimal threshold (EO)	0.880	0.063	0.013	0.002
exponentiated gradient (DP gap = 0)	0.882	0.043	0.002	0.008
exponentiated gradient (EO gap = 0)	0.882	0.039	0.006	0.001
SBM (OT-S)	N/A	0.882	0.047	0.028	0.026
correlation remover	0.879	0.057	0.011	0.029
optimal threshold (DP)	0.882	0.040	0.000	0.011
optimal threshold (EO)	0.882	0.042	0.010	0.000
exponentiated gradient (DP gap = 0)	0.881	0.045	0.001	0.008
exponentiated gradient (EO gap = 0)	0.880	0.056	0.016	0.005
Table 25: SBM combined with other fair ML methods in HateXplain dataset
Fair ML method	Acc	F1	ΔDP	ΔEO
WS (Baseline)	N/A	0.584	0.590	0.171	0.133
correlation remover	0.555	0.557	0.007	0.031
optimal threshold (DP)	0.539	0.515	0.005	0.047
optimal threshold (EO)	0.573	0.573	0.129	0.090
exponentiated gradient (DP gap = 0)	0.562	0.561	0.006	0.055
exponentiated gradient (EO gap = 0)	0.579	0.586	0.130	0.093
SBM (w/o OT)	N/A	0.592	0.637	0.159	0.138
correlation remover	0.563	0.616	0.033	0.053
optimal threshold (DP)	0.586	0.660	0.013	0.006
optimal threshold (EO)	0.538	0.561	0.034	0.074
exponentiated gradient (DP gap = 0)	0.581	0.638	0.039	0.013
exponentiated gradient (EO gap = 0)	0.580	0.630	0.047	0.095
SBM (OT-L)	N/A	0.606	0.670	0.120	0.101
correlation remover	0.587	0.657	0.057	0.087
optimal threshold (DP)	0.600	0.683	0.010	0.004
optimal threshold (EO)	0.563	0.615	0.039	0.071
exponentiated gradient (DP gap = 0)	0.600	0.673	0.029	0.011
exponentiated gradient (EO gap = 0)	0.593	0.669	0.044	0.087
SBM (OT-S)	N/A	0.612	0.687	0.072	0.037
correlation remover	0.587	0.668	0.073	0.105
optimal threshold (DP)	0.607	0.694	0.002	0.031
optimal threshold (EO)	0.572	0.696	0.201	0.182
exponentiated gradient (DP gap = 0)	0.598	0.683	0.005	0.020
exponentiated gradient (EO gap = 0)	0.585	0.672	0.070	0.093
Table 26: SBM combined with other fair ML methods in CelebA dataset
Fair ML method	Acc	F1	ΔDP	ΔEO
WS (Baseline)	N/A	0.866	0.879	0.308	0.193
correlation remover	0.845	0.862	0.099	0.066
optimal threshold (DP)	0.816	0.845	0.009	0.035
optimal threshold (EO)	0.789	0.793	0.196	0.033
exponentiated gradient (DP gap = 0)	0.781	0.814	0.008	0.006
exponentiated gradient (EO gap = 0)	0.838	0.854	0.205	0.025
SBM (w/o OT)	N/A	0.870	0.883	0.309	0.192
correlation remover	0.849	0.865	0.095	0.066
optimal threshold (DP)	0.819	0.848	0.009	0.038
optimal threshold (EO)	0.792	0.798	0.194	0.030
exponentiated gradient (DP gap = 0)	0.783	0.818	0.006	0.007
exponentiated gradient (EO gap = 0)	0.841	0.857	0.206	0.029
SBM (OT-L)	N/A	0.870	0.883	0.306	0.185
correlation remover	0.849	0.866	0.096	0.066
optimal threshold (DP)	0.819	0.848	0.010	0.034
optimal threshold (EO)	0.792	0.798	0.193	0.023
exponentiated gradient (DP gap = 0)	0.783	0.818	0.007	0.008
exponentiated gradient (EO gap = 0)	0.841	0.857	0.203	0.026
SBM (OT-S)	N/A	0.872	0.885	0.306	0.184
correlation remover	0.851	0.867	0.097	0.062
optimal threshold (DP)	0.821	0.850	0.010	0.035
optimal threshold (EO)	0.795	0.801	0.193	0.023
exponentiated gradient (DP gap = 0)	0.784	0.819	0.008	0.010
exponentiated gradient (EO gap = 0)	0.840	0.857	0.200	0.022
Table 27: SBM combined with other fair ML methods in UTKFace dataset
Fair ML method	Acc	F1	ΔDP	ΔEO
WS (Baseline)	N/A	0.791	0.791	0.172	0.073
correlation remover	0.787	0.786	0.034	0.051
optimal threshold (DP)	0.774	0.767	0.040	0.215
optimal threshold (EO)	0.788	0.786	0.114	0.005
exponentiated gradient (DP gap = 0)	0.769	0.762	0.029	0.078
exponentiated gradient (EO gap = 0)	0.792	0.790	0.126	0.026
SBM (w/o OT)	N/A	0.797	0.790	0.164	0.077
correlation remover	0.791	0.793	0.006	0.091
optimal threshold (DP)	0.764	0.791	0.033	0.202
optimal threshold (EO)	0.789	0.791	0.165	0.077
exponentiated gradient (DP gap = 0)	0.760	0.791	0.024	0.069
exponentiated gradient (EO gap = 0)	0.791	0.792	0.158	0.076
SBM (OT-L)	N/A	0.800	0.793	0.135	0.043
correlation remover	0.799	0.791	0.004	0.098
optimal threshold (DP)	0.785	0.772	0.034	0.195
optimal threshold (EO)	0.800	0.793	0.128	0.038
exponentiated gradient (DP gap = 0)	0.779	0.764	0.024	0.069
exponentiated gradient (EO gap = 0)	0.797	0.789	0.123	0.030
SBM (OT-S)	N/A	0.804	0.798	0.130	0.041
correlation remover	0.799	0.794	0.012	0.087
optimal threshold (DP)	0.789	0.777	0.036	0.195
optimal threshold (EO)	0.799	0.794	0.168	0.046
exponentiated gradient (DP gap = 0)	0.776	0.764	0.023	0.068
exponentiated gradient (EO gap = 0)	0.801	0.796	0.141	0.044
E.6 Additional synthetic experiment result on the number of LFs
(a) Number of LFs vs. Performance
Figure 10: Synthetic experiment on number of LFs vs. performance and fairness
(b) Number of LFs vs. Unfairness
Claim Investigated One seemingly possible argument is that diversifying LFs can naturally resolve unfairness, weakening the necessity of our method. We hypothesized that simply increasing the number of LFs does not improve fairness, while our method can further improve fairness as the number of LFs increases. To show this, we generate unfair synthetic data and increase the number of LFs and see if our method can remedy LF fairness and improve LF performance when LFs are diversified.
Setup and Procedure We generated a synthetic dataset  $(n = 10000)$  with the similar procedure as the previous synthetic experiment. Input samples in  $\mathbb{R}^2$  are taken from  $\sim \mathcal{N}(0,I)$ , and true labels  $Y$  are generated by  $Y = 1[X[:,0] \geq 0]$ . Group transformation  $g_{b}(x): x + b$  is applied to a random half of samples, where  $b \sim \mathcal{U}([10,50]^2)$ . We varied LFs by randomly sampling parameters based on our label model. Specifically, parameters for each LFs are sampled from  $\theta_{j} \sim \mathcal{U}(0.1,3)$ ,  $x^{center_j} \sim \mathcal{U}([-5,5]^2)$ . We varied the number of LFs from 3 to 30. We repeat experiments 10 times with different seeds and provide means and confidence intervals.
Results The result is reported in Figure 10. The result shows that simply diversifying label sources does not improve fairness—though performance can be improved by adding more LFs. On the other hand, SBM resolves unfairness and yields better performance by mitigating source bias.
E.7 Large language model experiment
Recently, large language models (LMs) pre-trained with massive datasets have shown excellent performance in many NLP tasks, even without fine-tuning. As a proof of concept that our method can be used for LMs, we conducted an additional experiment that applies our method to NLP tasks with LMs.  $\left[\mathrm{ANC}^{+}22\right]$  introduced AMA, which is a prompting strategy that provides multiple prompts with varying instructions, and combines their outputs to obtain a final answer using weak supervision. In this setup, distinct prompts can be seen as label sources. Based on their setup, we craft group annotations that identify a low accuracy regime by Barlow slicing  $\left[\mathrm{SNS}^{+}21\right]$ , which is a tree-based data slicing method. After applying data slicing for each prompt, we apply SBM and use the AMA pipeline. The result is shown in Table 28. SBM with Barlow data slicing yields modest performance gains over AMA. This result suggests that it may be possible to use our techniques to gain improvements even in generic zero-shot language model prediction.
Table 28: SBM combined with AMA  $\left[\mathrm{ANC}^{+}22\right]$ . Performance metric is accuracy (\%).
Methods	RTE	WSC	WIC
AMA [ANC+22]	75.1	77.9	61.3
AMA (Reproduced)	75.1	76	60.8
SBM (w/o OT)	75.5	71.2	61.1
SBM (OT-L)	75.1	70.2	61.6
SBM (OT-S)	74.7	78.8	61.1