SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Remark: In this section, considering the probability tree diagram in Fig.1(b), we propose a joint distribution for the possible events of classification  $(N_{c_{i|j}})$ , and use it to compute the marginal distribution of each event, and finally the distribution of the SA classifier outputs  $(\hat{p}_0,$  and  $\hat{p}_1)$ . Note that considering Eqn. 4, 5, only with a perfect classifier ( $\alpha_{i} = 1$ , i.e.  $\mathrm{acc} = 100\%$ ) the  $\tilde{\mu}_{\hat{p}_0}$  converges to  $p_0^*$ . However, training a perfect SA classifier is not practical e.g. due to the lack of an appropriate dataset and task hardness [42, 43]. As a result, in the following, we will propose CLEAM which instead utilizes this statistical model to mitigate the error of the SA classifier.
4.2 CLEAN for Accurate Fairness Measurement
In this section, we propose a new estimation method in fairness measurement that considers the inaccuracy of the SA classifier. For this, we use the statistical model, introduced in Sec 4.1, to compute a more accurate estimation of  $p^*$ . Specifically, we first propose a Point Estimate (PE) by approximating the maximum likelihood value of  $p^*$ . Then, we use the confidence interval for the observed data  $(\hat{p})$  to propose an Interval Estimate (IE) for  $p^*$ .
Point Estimate (PE) for  $p^*$ . Suppose that we have access to  $s$  samples of  $\hat{p}$  denoted by  $\{\hat{p}^1, \dots, \hat{p}^s\}$ , i.e. SA classification results on  $s$  batches of generated data. We can then use the proposed statistical model to approximate the  $p^*$ . In the previous section, we demonstrate that we can model  $\hat{p}_j^i$  using a Gaussian distribution. Considering this, first, we use the available samples to calculate sample-based statistics including the mean and variance of the  $\hat{p}_j$  samples:
$$
\ddot {\mu} _ {\hat {p} _ {j}} = \frac {1}{s} \sum_ {i = 1} ^ {s} \hat {p} _ {j} ^ {i} \tag {6}
$$
$$
\ddot {\sigma} _ {\hat {p} _ {j}} ^ {2} = \frac {1}{s - 1} \sum_ {i = 1} ^ {s} \left(\hat {p} _ {j} ^ {i} - \ddot {\mu} _ {\hat {p} _ {j}}\right) ^ {2} \tag {7}
$$
For a Gaussian distribution, the Maximum Likelihood Estimate (MLE) of the population mean is its sample mean  $\ddot{\mu}_{\hat{p}_j}$  [44]. Given that  $s$  is large enough (e.g.  $s > 30$ ), we can assume that  $\ddot{\mu}_{\hat{p}_j}$  is a good approximation of the population mean [45], and equate it to the statistical population mean  $\tilde{\mu}_{\hat{p}_j}$  in Eqn. 4 (see Supp A.2 for derivation). With that, we get the maximum likelihood approximation of  $\pmb{p}^*$ , which we call the CLEAN's point estimate,  $\mu_{CLEAM}$ :
$$
\mu_ {\mathrm {C L E A M}} \left(p _ {0} ^ {*}\right) = \left(\dot {\mu} _ {\hat {p} _ {0}} - \alpha_ {1} ^ {\prime}\right) / \left(\alpha_ {0} - \alpha_ {1} ^ {\prime}\right), \quad \mu_ {\mathrm {C L E A M}} \left(p _ {1} ^ {*}\right) = 1 - \mu_ {\mathrm {C L E A M}} \left(p _ {0} ^ {*}\right) \tag {8}
$$
Notice that  $\mu_{\mathrm{CLEAM}}$  accounts for the inaccuracy of the SA classifier.
Interval Estimate (IE) for  $p^*$ . In the previous part, we propose a PE for  $p^*$  using the statistical model, and sample-based mean  $\ddot{\mu}_{\hat{p}_0}$ . However, as we use only  $s$  samples of  $\hat{p}$ ,  $\ddot{\mu}_{\hat{p}_0}$  may not capture the exact value of the population mean. This adds some degree of inaccuracy into  $\mu_{\mathrm{CLEAM}}$ . In fact, in our framework,  $\ddot{\mu}_{\hat{p}_0}$  equals  $\tilde{\mu}_{\hat{p}_0}$  when  $s \to \infty$ . However, increasing each unit of  $s$  significantly increases the computational complexity, as each  $\hat{p}$  requires  $n$  generated samples. To address this, we recall that  $\hat{p}_0$  follows a Gaussian distribution and instead utilize frequentist statistics [41] to propose a 95% confidence interval (CI) for  $p^*$ . To do this, first we derive the CI for  $\tilde{\mu}_{\hat{p}_0}$ :
$$
\ddot {\mu} _ {\hat {p} _ {0}} - 1. 9 6 \frac {\ddot {\sigma} _ {\hat {p} _ {0}}}{\sqrt {s}} \leq \tilde {\mu} _ {\hat {p} _ {0}} \leq \ddot {\mu} _ {\hat {p} _ {0}} + 1. 9 6 \frac {\ddot {\sigma} _ {\hat {p} _ {0}}}{\sqrt {s}} \tag {9}
$$
Then, applying Eqn.4 to Eqn.9 gives the lower and upper bounds of the approximated  $95\%$  CI for  $p_0^*$ :
$$
\mathcal {L} \left(p _ {0} ^ {*}\right), \mathcal {U} \left(p _ {0} ^ {*}\right) = \left(\ddot {\mu} _ {\hat {p} _ {0}} \mp 1. 9 6 \left(\ddot {\sigma} _ {\hat {p} _ {0}} / \sqrt {s}\right) - \alpha_ {1} ^ {\prime}\right) / \left(\alpha_ {0} - \alpha_ {1} ^ {\prime}\right) \tag {10}
$$
This gives us the interval estimate of CLEAM,  $\rho_{\mathrm{CLEAM}} = [\mathcal{L}(p_0^*),\mathcal{U}(p_0^*)]$ , a range of values that we can be approximately  $95\%$  confident to contain  $p_0^*$ . The range of possible values for  $p_1^*$  can be simply derived considering  $p_1^* = 1 - p_0^*$ . The overall procedure of CLEAM is summarized in Alg. 1. Now, with the IE, we can provide statistical significance to the reported fairness improvements.
Table 1: Comparing the point estimates and interval estimates of Baseline [1], Diversity [46] and our CLEAM in estimating  $p^*$  of StyleGAN2 [3] and StyleSwin [4] with the proposed GenData datasets. We utilize SA classifiers Resnet-18/34 (R18, R34)[34], MobileNetv2 (MN2)[35] and VGG-16 (V16)[36], with different accuracies  $\alpha$ , to classify samples w.r.t. attributes Gender and BlackHair. The  $p_0^*$  value of each GAN w.r.t. SA is determined by manually hand-labeling the generated data. We repeat this for 5 experimental runs and report the mean error rate, per Eqn. 1. See Supp D.1 for the standard deviation of PE and IE.
α = {α0, α1} Avg. α	Point Estimate	Interval Estimate
Baseline	Diversity	CLEAM (Ours)	Baseline	Diversity	CLEAM (Ours)
μBase	εμ(↓)	μDiv	εμ(↓)	μCLEAM	εμ(↓)	ρBase	ερ(↓)	ρDiv	ερ(↓)	ρCLEAM	ερ(↓)
(A) StyleGAN2
Gender with GT class probability p0*=0.642
R18	{0.947, 0.983}	0.97	0.610	4.98%	—	—	0.638	0.62%	[0.602, 0.618]	6.23%	—	—	[0.629, 0.646]	2.02%
R34	{0.932, 0.976}	0.95	0.596	7.17%	—	—	0.634	1.25%	[0.589, 0.599]	8.26%	—	—	[0.628, 0.638]	2.18%
MN2	{0.938, 0.975}	0.96	0.607	5.45%	—	—	0.637	0.78%	[0.602, 0.612]	6.23%	—	—	[0.632, 0.643]	1.56%
V16	{0.801, 0.919}	0.86	0.532	17.13%	0.550	14.30%	0.636	0.93%	[0.526, 0.538]	18.06%	[0.536, 0.564]	16.51%	[0.628, 0.644]	2.18%
Average Error:	8.68%	14.30%	0.90%	9.70%	16.51%	1.99%
BlackHair with GT class probability p0*=0.643
R18	{0.869, 0.885}	0.88	0.599	6.84%	—	—	0.641	0.31%	[0.591, 0.607]	8.08%	—	—	[0.631, 0.652]	1.40%
R34	{0.834, 0.916}	0.88	0.566	11.98%	—	—	0.644	0.16%	[0.561, 0.572]	12.75%	—	—	[0.637, 0.651]	1.24%
MN2	{0.839, 0.881}	0.86	0.579	9.95%	—	—	0.639	0.62%	[0.574, 0.584]	10.73%	—	—	[0.632, 0.647]	1.71%
V16	{0.851, 0.836}	0.84	0.603	6.22%	0.582	9.49%	0.640	0.47%	[0.597, 0.608]	7.15%	[0.568, 0.596]	11.66%	[0.632, 0.648]	1.71%
Average Error:	8.75%	9.49%	0.39%	9.68%	11.66%	1.52%
(B) StyleSwin
Gender with GT class probability p0*=0.656
R18	{0.947, 0.983}	0.97	0.620	5.49%	—	—	0.648	1.22%	[0.612, 0.629]	6.70%	—	—	[0.639, 0.658]	2.59%
R34	{0.932, 0.976}	0.95	0.610	7.01%	—	—	0.649	1.07%	[0.605, 0.615]	7.77%	—	—	[0.643, 0.654]	1.98%
MN2	{0.938, 0.975}	0.96	0.623	5.03%	—	—	0.655	0.15%	[0.618, 0.629]	5.79%	—	—	[0.649, 0.661]	1.07%
V16	{0.801, 0.919}	0.86	0.555	15.39%	0.562	14.33%	0.668	1.83%	[0.549, 0.560]	16.31%	[0.548, 0.576]	16.46%	[0.660, 0.675]	2.90%
Average Error:	8.23%	14.33%	1.07%	9.14%	16.46%	2.14%
BlackHair with GT class probability p0*=0.668
R18	{0.869, 0.885}	0.88	0.612	8.38%	—	—	0.659	1.35%	[0.605, 0.620]	9.43%	—	—	[0.649, 0.670]	2.84%
R34	{0.834, 0.916}	0.88	0.581	13.02%	—	—	0.662	0.90%	[0.576, 0.586]	13.77%	—	—	[0.656, 0.669]	1.80%
MN2	{0.839, 0.881}	0.86	0.596	10.78%	—	—	0.659	1.35%	[0.591, 0.600]	11.50%	—	—	[0.652, 0.666]	2.40%
V16	{0.851, 0.836}	0.84	0.625	6.44%	0.608	8.98%	0.677	1.35%	[0.620, 0.630]	7.19%	[0.590, 0.626]	11.68%	[0.670, 0.684]	2.40%
Average Error:	9.66%	8.98%	1.24%	10.47%	11.68%	2.36%
Table 2: Comparing the point estimates and interval estimates of Baseline and CLEAM in estimating the  $p^*$  of the Stable Diffusion Model [5] with the GenData-SDM dataset. We use prompt input starting with "A photo with the face of" and ending with synonymous (Gender neutral) prompts. We utilized CLIP as the classifier for Gender, to obtain  $\hat{p}$ .
Prompt	GT	Point Estimate	Interval Estimate
Baseline	CLEAM (Ours)	Baseline	CLEAM (Ours)
p0*	μBase	eμ(↓)	μCLEAM	eμ(↓)	ρBase	eρ(↓)	ρCLEAM	eρ(↓)
α=[0.998,0.975], Avg. α=0.987, CLIP -Gender
"A photo with the face of an individual"	0.186	0.203	9.14%	0.187	0.05%	[0.198, 0.208]	11.83%	[0.182, 0.192]	3.23%
"A photo with the face of a human being"	0.262	0.277	5.73%	0.263	0.38%	[0.270, 0.285]	8.78%	[0.255, 0.271]	3.44%
"A photo with the face of one person"	0.226	0.241	6.63%	0.230	1.77%	[0.232, 0.251]	11.06%	[0.220, 0.239]	5.75%
"A photo with the face of a person"	0.548	0.556	1.49%	0.548	0.00%	[0.545, 0.566]	3.28%	[0.537, 0.558]	2.01%
Average Error	5.75%	0.44%	8.74%	3.61%
Figure 2: LHS: Applying CLEAM to assess Gender bias in SDM[5] with CLIP [6]. Here, we utilize synonymous neutral prompts, prefixed with "A photo with the face of _" as the input. We found that subtle changes to the prompts resulted in significant changes in the bias. RHS: Illustrating the shift in Gender bias. Using the same random seeds (per column), we generate 10 samples from two prompts. Note how in some columns, Gender changes while retaining general features e.g. pose.
In this section, we first evaluate fairness measurement accuracy of CLEAM on both GANs and SDM (Sec.5.1) with our proposed GenData dataset. Then we evaluate CLEAM's robustness through some ablation studies (Sec. 5.2). To the best of our knowledge, there is no similar literature for improving fairness measurements in generative models. Therefore, we compare CLEAM with the two most related works: a) the Baseline used in previous works [1, 2, 7, 9, 12] b) Diversity [46] which computes disparity within a dataset via an intra-dataset pairwise similarity algorithm. We remark that, as discussed by Keswani et al. [46] Diversity is model-specific using VGG-16 [36]; see Supp. D.2 for more details. Finally, unless specified, we repeat the experiments with  $s = 30$  batches of images from the generators with batch size  $n = 400$ . For a fair comparison, all three algorithms use the exact same inputs. However, while Baseline and Diversity ignore the SA classifier inaccuracies, CLEAM makes good use of it to rectify the measurement error. As mentioned in Sec. 4.2, for CLEAM, we utilize  $\alpha$  measured on real samples, which we found to be a good approximation of the  $\alpha$  measured on generated samples (see Supp. D.7 for results). We repeat each experiment 5 times and report the mean value for each test point for both PE and IE. See Supp D.1 for the standard deviation.
5.1 Evaluating CLEAM's Performance
CLEAM for fairness measurement of SOTA GANs - StyleGAN2 and StyleSwin. For a fair comparison, we first compute  $s$  samples of  $\hat{p}$ , one for each batch of  $n$  images. For Baseline, we use the mean  $\hat{p}$  value as the PE (denoted by  $\mu_{\mathrm{Base}}$ ), and the 95% confidence interval as IE ( $\rho_{\mathrm{Base}}$ ). With the same  $s$  samples of  $\hat{p}$ , we apply Alg. 1 to obtain  $\mu_{\mathrm{CLEAM}}$  and  $\rho_{\mathrm{CLEAM}}$ . For Diversity, following the original source code [46], a controlled dataset with fair representation is randomly selected from a held-out dataset of CelebA-HQ [27]. Then, we use a VGG-16 [36] feature extractor and compute Diversity,  $\delta$ . With  $\delta$  we find  $\hat{p}_0 = (\delta + 1)/2$  and subsequently  $\mu_{\mathrm{Div}}$  and  $\rho_{\mathrm{Div}}$  from the mean and 95% CI (see Supp D.2 for more details on diversity). We then compute  $e_{\mu_{\mathrm{CLEAM}}}, e_{\mu_{\mathrm{Div}}}, e_{\rho_{\mathrm{CLEAM}}}$  and  $e_{\rho_{\mathrm{Div}}}$  with Eqn 1, by replacing the Baseline estimates with CLEAM and Diversity.
As discussed, our results in Tab.1 show that the baseline experiences significantly large errors of  $4.98\% \leq e_{\mu_{\mathrm{Base}}} \leq 17.13\%$ , due to a lack of consideration for the inaccuracies of the SA classifier. We note that this problem is prevalent throughout the different SA classifier architectures, even with higher capacity classifiers e.g. ResNet-34. Diversity, a method similarly unaware of the inaccuracies of the SA classifier, presents a similar issue with  $8.98\% \leq e_{\mu_{\mathrm{Div}}} \leq 14.33\%$  In contrast, CLEAM dramatically reduces the error for all classifier architectures. Specifically, CLEAM reduces the average point estimate error from  $e_{\mu_{\mathrm{Base}}} \geq 8.23\%$  to  $e_{\mu_{\mathrm{CLEAM}}} \leq 1.24\%$ , in both StyleGAN2 and StyleSwin. The IE presents similar results, where in most cases  $\rho_{\mathrm{CLEAM}}$  bounds the GT value of  $p^*$ .
CLEAM for fairness measurement of SDM. We evaluate CLEAM in estimating the bias of the SDM w.r.t. Gender, based on the synonymous (gender-neutral) prompts introduced in Sec. 3. Recall that here we utilize CLIP as the zero-shot SA classifier. Our results in Tab 2, as discussed, show that utilizing the baseline results in considerable error  $(1.49\% \leq e_{\mu_{\mathrm{Base}}} \leq 9.14\%)$  for all prompts, even though the SA classifier's average accuracy was high,  $\approx 98.7\%$  (visual results in Fig.2). A closer look at the theoretical model's Eqn. 4 reveals that this is due to the larger inaccuracies observed in the biased class  $(\alpha_1^{\prime})$  coupled with the large bias seen in  $p_1^*$ , which results in  $\mu_{\mathrm{Base}}$  deviating from  $p_0^*$ . In contrast, CLEAM accounts for these inaccuracies and significantly minimizes the error to  $e_{\mu_{\mathrm{CLEAM}}} \leq 1.77\%$ . Moreover, CLEAM's IE is able to consistently bound the GT value of  $p_0^*$ .
Here, we perform the ablation studies and compare CLEAM with classifier correction methods. We remark that detailed results of these experiments are provided in the Supp due to space limitations.
CLEAM for measuring varying degrees of bias. As we cannot control the bias in trained generative models, to simulate different degrees of bias, we evaluate CLEAM with a pseudo-generator. Our results show that CLEAM is effective at different biases  $(p_0^* \in [0.5, 0.9])$  reducing the average error from  $2.80\% \leq e_{\mu_{\mathrm{Base}}} \leq 6.93\%$  to  $e_{\mu_{\mathrm{CLEAM}}} \leq 0.75\%$  on CelebA [47] w.r.t. {Gender,BlackHair}, and AFHQ [48] w.r.t. Cat/Dog. See Supp D.3 and D.4 for full experimental results.
CLEAM vs Classifier Correction Methods [49]. CLEAM generally accounts for the classifier's inaccuracies, without targeting any particular cause of inaccuracies, for the purpose of rectifying the fairness measurements. This objective is unlike traditional classifier correction methods as it does not aim to improve the actual classifier's accuracy. However, considering that classifier correction methods may improve the fairness measurements by directly rectifying the classifier inaccuracies, we compare its performance against CLEAM. As an example, we utilize the Black Box Shift Estimation / Correction (BBSE / BBSC) [49] which considers the label shift problem and aims to correct the classifier output by detecting the distribution shift. Our results, based on Sec. 5.1 setup, show that while BBSE does improve on the fairness measurements of the baseline i.e.  $4.20\% \leq e_{\mu_{\mathrm{BBSE}}} \leq 3.38\%$ , these results are far inferior to CLEAM's results seen in Tab. 1. In contrast, BBSC demonstrates no improvements in fairness measurements. See Supp D.8 for full experimental results. We postulate that this is likely due to the strong assumption of label shift made by both methods.
Effect of batch-size. Utilizing experimental setup in Sec. 5.1 for batch size  $n \in [100,600]$ , our results in Fig. 3 show that  $n = 400$  is an ideal batch size, balancing computational cost and measurement accuracy. See Supp F for full experimental details and results.
Figure 3: Comparing the point error  $e_{\mu}$  for Baseline and CLEAM when evaluating the bias of GenData-CelebA with ResNet-18, while varying sample size,  $n$ .
6 Applying CLEAM: Bias in Current SOTA Generative Models
In this section, we leverage the improved reliability of CLEAM to study biases in the popular generative models. Firstly, with the rise in popularity of text-to-image generators [5, 50-52], we revisit our results when passing different prompts, with synonymous neutral meanings to an SDM, and take a closer look at how subtle prompt changes can impact bias w.r.t. Gender. Furthermore, we further investigate if similar results would occur in other SA, Smiling. Secondly, with the shift in popularity from convolution to transformer-based architectures [53-55], due to its better sample quality, we determine whether the learned bias would also change. For this, we compare StylesSwin (transformer) and StyleGAN2 (convolution), which are both based on the same architecture backbone.
Our results, on SDM, demonstrate that the use of different synonymous neutral prompts [32, 33] results in different degrees of bias w.r.t. both Gender and Smiling attributes. For example in Fig. 2, a semantically insignificant prompt change from "one person" to "a person" results in a significant shift in Gender bias. Then, in Fig. 4a, we observe that while the SDM w.r.t. our prompts appear to be heavily biased to not-Smiling, having "person" in the prompt appears to significantly reduce this bias. This suggests that for SDM, even semantically similar neutral prompts [32, 33] could result in different degrees of bias, thereby demonstrating certain instability in SDM. Next, our results in Fig. 4b compare the bias in StyleGAN2, StylesSwin, and the training CelebA-HQ dataset over an extended number of SAs. Overall, we found that while StyleSwin produces better quality samples [4], the same biases still remain statistically unchanged between the two architectures i.e. their IE overlap. Interestingly, our results also found that both the GANs were less biased than the training dataset itself.
Conclusion. In this work, we address the limitations of the existing fairness measurement framework. Since generated samples are typically unlabeled, we first introduce a new labeled dataset based on three state-of-the-art generative models for our studies. Our findings suggest that the existing framework, which ignores classification inaccuracies, suffers from significant measurement errors, even when the SA classifier is very accurate. To rectify this, we propose CLEAM, which considers these inaccuracies in its statistical model and outputs a more accurate fairness measurement. Overall, CLEAM demonstrates improved accuracy over extensive experimentation, including both real
a) "A photo with the face of a human being"
b) "A photo with the face of a person"
(a) Evaluating bias of SDM [5] w.r.t. SA Smiling using CLEAM. LHS: Bias value with different text prompts as input to SDM. CLEAM identifies that having "person" in the prompt results in more Smiling samples generated. RHS: Visual comparison of the samples generated for two different but semantically similar text prompts. Samples in the same column are generated using the same random seed. Notice how the SA (smiling) of the samples changes with a slight difference in the prompts.
Figure 4: Applying CLEAM to further assess the bias in popular generative models.
(b) CLEAM on StyleGAN2 [3] and StyleSwin [4], both pre-trained on CelebA-HQ but are based on different architecture variants. We utilize a ResNet-18 and compare the bias w.r.t. various SAs.
generators and controlled setups. Moreover, by applying CLEAM to popular generative models, we uncover significant biases that raise efficacy concerns about these models' real-world application.
Broader Impact. Given that generative models are becoming more widely integrated into our everyday society e.g. text-to-image generation, it is important that we have reliable means to measure fairness in generative models, thereby allowing us to prevent these biases from proliferating into new technologies. CLEAM provides a step in this direction by allowing for more accurate evaluation. We remark that our work does not introduce any social harm but instead improves on the already existing measurement framework.
Limitations. Despite the effectiveness of the proposed method along various generative models, our work addresses only one facet of the problems in the existing fairness measurement and there is still room for further improvement. For instance, it may be beneficial to consider SA to be non-binary e.g. when hair color is not necessary fully black (grey). Additionally, existing datasets used to train classifiers are commonly human-annotated, which may itself contain certain notions of bias. See Supp. I for further discussion.
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programmes (AISG Award No.: AISG2-TC-2022-007) and SUTD project PIE-SGP-AI-2018-01. This research work is also supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021). This material is based on the research/work support in part by the Changi General Hospital and Singapore University of Technology and Design, under the HealthTech Innovation Fund (HTIF Award No. CGH-SUTD-2021-004). We thank anonymous reviewers for their insightful feedback and discussion.