SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Christopher T. H. Teo  
christopher_teo@mymail.sutd.edu.sg
Milad Abdollahzadeh milad_abdollahzadeh@sutd.sg
Ngai-Man Cheung* ngaiman_cheung@sutd.edu.sg
Singapore University of Technology and Design (SUTD)
Recently, there has been increased interest in fair generative models. In this work, we conduct, for the first time, an in-depth study on fairness measurement, a critical component in gauging progress on fair generative models. We make three contributions. First, we conduct a study that reveals that the existing fairness measurement framework has considerable measurement errors, even when highly accurate sensitive attribute (SA) classifiers are used. These findings cast doubts on previously reported fairness improvements. Second, to address this issue, we propose CClassifier Error-Aware Measurement (CLEAM), a new framework which uses a statistical model to account for inaccuracies in SA classifiers. Our proposed CLEAM reduces measurement errors significantly, e.g.,  $4.98\% \rightarrow 0.62\%$  for StyleGAN2 w.r.t. Gender. Additionally, CLEAM achieves this with minimal additional overhead. Third, we utilize CLEAM to measure fairness in important text-to-image generator and GANs, revealing considerable biases in these models that raise concerns about their applications. Code and more resources: https://sud-visual-computing-group.github.io/CLEAM/.
Fair generative models have been attracting significant attention recently [1, 2, 7-13]. In generative models [14-18], fairness is commonly defined as equal generative quality [11] or equal representation [1, 2, 7, 9, 12, 19, 20] w.r.t. some Sensitive Attributes (SA). In this work, we focus on the more widely utilized definition - equal representation. In this definition, as an example, a generative model is regarded as fair w.r.t. Gender, if it generates Male and Female samples with equal probability. This is an important research topic as such biases in generative models could impact their application efficacy, e.g., by introducing racial bias in face generation of suspects [21] or reducing accuracy when supplementing data for disease diagnosis [22].
Fairness measurement for generative models. Recognizing the importance of fair generative models, several methods have been proposed to mitigate biases in generative models [1, 2, 7, 9, 12]. However, in our work, we focus mainly on the accurate fairness measurement of deep generative models i.e. assessing and quantifying the bias of generative models. This is a critical topic, as accurate measurements are essential to reliably gauge the progress of bias mitigation techniques. The general fairness measurement framework is shown in Fig. 1 (See Sec. 2 for details). This framework is utilized in existing works to assess their proposed fair generators. Central to the fairness measurement framework is a SA classifier, which classifies the generated samples w.r.t. a SA, in order to estimate the bias of the generator. For example, if eight out of ten generated face images are classified as Male
Figure 1: (a) General framework for measuring fairness in generative models. Generated samples with unknown ground-truth (GT) probability  $p^*$  w.r.t. sensitive attribute (SA) are fed into a SA classifier to obtain  $\hat{p}$ . Existing framework (Baseline) uses the classifier output  $\hat{p}$  as estimation of  $p^*$ . In contrast, our proposed CLEAM includes an improved estimation that accounts for inaccuracies in the SA classifier (see Alg. 1). (b) Our statistical model for fairness measurement. This model accounts for inaccuracies in the SA classifier and is the base of our proposed CLEAM (see Sec. 4.1). (c) Improvements with CLEAM. CLEAM improves upon Baseline [1, 2] by reducing the relative error in estimating the GT  $p_0^*$  for SOTA GANs: StyleGAN2 [3] and StyleSwin [4], and Stable Diffusion Model [5]. First row displays the Baseline and CLEAM estimates for each GAN, using ResNet-18 as the SA classifier for Gender and BlackHair. The Baseline incurs significant fairness measurement errors (e.g.  $4.98\%$ ), even when utilizing a highly accurate ResNet-18 ( $\approx 97\%$  accuracy). Meanwhile, CLEAM reduces the error significantly in all setups, e.g. in the first panel, the error is reduced:  $4.98\% \rightarrow 0.62\%$ . Similarly, in the second row, CLEAM reduces measurement error significantly in the Stable Diffusion Model [5], using CLIP [6] as the SA classifier for Gender, e.g. first panel:  $9.14\% \rightarrow 0.05\%$  (Detailed evaluation in Tab. 1 and Tab. 2). Best viewed in color.
by the SA classifier, then the generator is deemed biased at 0.8 towards Male (further discussion in Sec. 2). We follow previous works [1, 2, 12] and focus on binary SA due to dataset limitations.
Research gap. In this paper, we study a critical research gap in fairness measurement. Existing works assume that when SA classifiers are highly accurate, measurement errors should be insignificant. As a result, the effect of errors in SA classifiers has not been studied. However, our study reveals that even with highly accurate SA classifiers, considerable fairness measurement errors could still occur. This finding raises concerns about potential errors in previous works' results, which are measured using existing framework. Note that the SA classifier is indispensable in fairness measurement as it enables automated measurement of generated samples.
Our contributions. We make three contributions to fairness measurement for generative models. As our first contribution, we analyze the accuracy of fairness measurement on generated samples, which previous works [1, 2, 7, 9, 12] have been unable to carry out due to the unavailability of proper datasets. We overcome this challenge by proposing new datasets of generated samples with manual labeling w.r.t. various SAs. The datasets include generated samples from Stable Diffusion Model (SDM) [5] — a popular text-to-image generator — as well as two State-of-The-Art (SOTA) GANs (StyleGAN2 [3] and StyleSwin [4]) w.r.t. different SAs. Our new datasets are then utilized in our work to evaluate the accuracy of the existing fairness measurement framework. Our results reveal that the accuracy of the existing fairness measurement framework is not adequate, due to the lack of consideration for the SA classifier inaccuracies. More importantly, we found that even in setups where the accuracy of the SA classifier is high, the error in fairness measurement could still be significant. Our finding raises concerns about the accuracy of previous works' results [1, 2, 12], especially since some of their reported improvements are smaller than the margin of measurement errors that we observe in our study when evaluated under the same setup; further discussion in Sec. 3.
To address this issue, as our second (major) contribution, we propose CClassifier Error-Aware Measurement (CLEAM), a new more accurate fairness measurement framework based on our developed statistical model for SA classification (further details on the statistical model in Sec. 4.1).
Specifically, CLEAM utilizes this statistical model to account for the classifier's inaccuracies during SA classification and outputs a more accurate fairness measurement. We then evaluate the accuracy of CLEAM and validate its improvement over existing fairness measurement framework. We further conduct a series of different ablation studies to validate performance of CLEAM. We remark that CLEAM is not a new fairness metric, but an improved fairness measurement framework that could achieve better accuracy in bias estimation when used with various fairness metrics for generative models.
As our third contribution, we apply CLEAM as an accurate framework to reliably measure biases in popular generative models. Our study reveals that SOTA GANs have considerable biases w.r.t. several SA. Furthermore, we observe an intriguing property in Stable Diffusion Model: slight differences in semantically similar prompts could result in markedly different biases for SDM. These results prompt careful consideration on the implication of biases in generative models. Our contributions are:
- We conduct a study to reveal that even highly-accurate SA classifiers could still incur significant fairness measurement errors when using existing framework.
- To enable evaluation of fairness measurement frameworks, we propose new datasets based on generated samples from StyleGAN, StyleSwin and SDM, with manual labeling w.r.t. SA.
- We propose a statistically driven fairness measurement framework, CLEAM, which accounts for the SA classifier inaccuracies to output a more accurate bias estimate.
- Using CLEAM, we reveal considerable biases in several important generative models, prompting careful consideration when applying them to different applications.
2 Fairness Measurement Framework
Fig.1(a) illustrates the fairness measurement framework for generative models as in [1, 2, 7, 9, 12]. Assume that with some input e.g. noise vector for a GAN or text prompt for SDM, a generative model synthesizes a sample  $\mathbf{x}$ . Generally, as the generator does not label synthesized samples, the ground truth (GT) class probability of these samples w.r.t. a SA (denoted by  $p^*$ ) is unknown. Thus, an SA classifier  $C_{\mathbf{u}}$  is utilized to estimate  $p^*$ . Specifically, for each sample  $\mathbf{x} \in \{\mathbf{x}\}$ ,  $C_{\mathbf{u}}(\mathbf{x})$  is the argmax classification for the respective SA. In existing works, the expected value of the SA classifier output over a batch of samples,  $\hat{p} = \mathbb{E}_{\mathbf{x}}[C_{\mathbf{u}}(\mathbf{x})]$  (or the average of  $\hat{p}$  over multiple batches of samples), is used as an estimation of  $p^*$ . This estimate may then be used in some fairness metric  $f$  to report the fairness value for the generator, e.g. fairness discrepancy metric between  $\hat{p}$  and a uniform distribution  $\bar{p}$  [1, 20] (see Supp A.3 for details on how to calculate  $f$ ). Note that the general assumption behind the existing framework is that with a reasonably accurate SA classifier,  $\hat{p}$  could be an accurate estimation of  $p^*$  [1, 9]. In the next section, we will present a deeper analysis on the effects of an inaccurate SA classifier on fairness measurement. Our findings suggest that there could be a large discrepancy between  $\hat{p}$  and  $p^*$ , even for highly accurate SA classifiers, indicative of significant fairness measurement errors in the current measurement framework.
One may argue that conditional GANs (cGANs) [23, 24] may be used to generate samples conditioned on the SA, thereby eliminating the need for an SA classifier. However, cGANs are not considered in previous works due to several limitations. These include the limited availability of large labeled training datasets, the unreliability of sample quality and labels [25], and the exponentially increasing conditional terms, per SA. Similarly, for SDM, Bianchi et al. [26] found that utilizing well-crafted prompts to mitigate biases is ineffective due to the presence of existing biases in its training dataset. Furthermore in Sec. 6, utilizing CLEAM, we will discuss that even subtle prompt changes (while maintaining the semantics) result in drastically different SA biases. See Supp G for further comparison between [26] and our findings.
3 A Closer Look at Fairness Measurement
In this section, we take a closer look at the existing fairness measurement framework. In particular, we examine its performance in estimating  $p^*$  of the samples generated by SOTA GANs and SDM, a task previously unstudied due to the lack of a labeled generated dataset. We do so by designing an experiment to demonstrate these errors while evaluating biases in popular image generators. Following previous works, our main focus is on binary SA which takes values in  $\{0,1\}$ . Note that,
we assume that the accuracy of the SA classifier  $C_{\mathbf{u}}$  is known and is characterized by  $\alpha = \{\alpha_0, \alpha_1\}$ , where  $\alpha_i$  is the probability of correctly classifying label  $i$ . For example, for Gender attribute,  $\alpha_0$  and  $\alpha_1$  are the probability of correctly classifying Female, and Male classes, respectively. In practice,  $C_{\mathbf{u}}$  is trained on standard training procedures (more details in the Supp F) and  $\alpha$  can be measured during the validation stage of  $C_{\mathbf{u}}$  and be considered a constant when the validation dataset is large enough. Additionally,  $p^*$  can be assumed to be a constant vector, given that the samples generated can be considered to come from an infinite population, as theoretically there is no limit to the number of samples from a generative model like GAN or SDM.
New dataset by labeling generators output. The major limitation of evaluating the existing fairness measurement framework is the unavailability of  $p^*$ . To pave the way for an accurate evaluation, we create a new dataset by manually labeling the samples generated by GANs and SDM. More specifically, we utilize the official publicly released pre-trained StyleGAN2 [3] and StyleSwin [4] on CelebA-HQ [27] for sample generation. Then, we randomly sample from these GANs and utilize Amazon Mechanical Turks to hand-label the samples w.r.t. Gender and BlackHair, resulting in ≈9K samples for each GAN; see Supp H for more details and examples. Next, we follow a similar labeling process w.r.t. Gender, but with a SDM [5] pre-trained on LAION-5B[28]. Here, we input prompts using best practices [26, 29–31], beginning with a scene description ("A photo with the face of"), followed by four indefinite (gender-neutral) pronouns or nouns [32, 33] – {"an individual", "a human being", "one person", "a person"} to collect ≈2k high-quality samples. We refer to this new dataset as Generated Dataset (GenData), which includes generated images from three models with corresponding SA labels: Data-GAN2, Data-StyleSwin, Data-SDM. We remark that these labeled datasets only provide a strong approximation of  $p^*$  for each generator, however as the datasets are reasonably large, we find this approximation sufficient and simply refer to it as the GT  $p^*$ . Then utilizing this GT  $p^*$ , we compare it against the estimated baseline ( $\hat{p}$ ). One interesting observation revealed by GenData is that all three generators exhibit a considerable amount of bias (see Tab.1 and 2); more detail in Sec. 6. Note that for a fair generator we have  $p_0^* = p_1^* = 0.5$ , and measuring the  $p_0^*$  and  $p_1^*$  is a good proxy for measuring fairness.
Experimental setup. Here, we follow Choi et al. [1] as the Baseline for measuring fairness. In particular, to calculate each  $\hat{p}$  value for a generator, a corresponding batch of  $n = 400$  samples is randomly drawn from GenData and passed into  $C_{\mathbf{u}}$  for SA classification. We repeat this for  $s = 30$  batches and report the mean results denoted by  $\mu_{\mathrm{Base}}$  and the  $95\%$  confidence interval denoted by  $\rho_{\mathrm{Base}}$ . For a comprehensive analysis of the GANs, we repeat the experiment using four different SA classifiers: Resnet-18, ResNet-34 [34], MobileNetv2 [35], and VGG-16 [36]. Then, to evaluate the SDM, we utilize CLIP [6] to explore the utilization of pre-trained models for zero-shot SA classification; more details on the CLIP SA classifier in Supp. E. As CLIP does not have a validation dataset, to measure  $\alpha$  for CLIP, we utilize CelebA-HQ, a dataset with a similar domain to our application. We found this to be a very accurate approximation; see Supp D.7 for validation results. Note that for SDM, a separate  $\hat{p}$  is measured for each text prompt as SDM's output images are conditioned on the input text prompt. As seen in Tab. 1 and 2, all classifiers demonstrate reasonably high average accuracy  $\in [84\%, 98.7\%]$ . Note that as we focus on binary SA (e.g. Gender:{Male, Female}), both  $p^*$  and  $\hat{p}$  have two components i.e.  $p^* = \{p_0^*, p_1^*\}$ , and  $\hat{p} = \{\hat{p}_0, \hat{p}_1\}$ . After computing the  $\mu_{\mathrm{Base}}$  and  $\rho_{\mathrm{Base}}$ , we calculate normalized  $L_1$  point error  $e_\mu$ , and interval max error  $e_\rho$  w.r.t. the  $p_0^*$  (GT) to evaluate the measurement accuracy of the baseline method:
$$
e _ {\mu_ {\mathrm {B a s e}}} = \frac {1}{p _ {0} ^ {*}} \left| p _ {0} ^ {*} - \mu_ {\mathrm {B a s e}} \right|; e _ {\rho_ {\mathrm {B a s e}}} = \frac {1}{p _ {0} ^ {*}} \max  \left\{\left| \min  \left(\rho_ {\mathrm {B a s e}}\right) - p _ {0} ^ {*} \right|, \left| \max  \left(\rho_ {\mathrm {B a s e}}\right) - p _ {0} ^ {*} \right| \right\} \tag {1}
$$
Based on our results in Tab. 1, for GANs, we observe that despite the use of reasonably accurate SA classifiers, there are significant estimation errors in the existing fairness measurement framework, i.e.  $e_{\mu_{\mathrm{Base}}} \in [4.98\%, 17.13\%]$ . In particular, looking at the SA classifier with the highest average accuracy of  $\approx 97\%$  (ResNet-18 on Gender), we observe significant discrepancies between GT  $p_0^*$  and  $\mu_{\mathrm{Base}}$ , with  $e_{\mu_{\mathrm{Base}}} = 4.98\%$ . These errors generally worsen as accuracy marginally degrades, e.g. MobileNetv2 with accuracy  $\approx 96\%$  results in  $e_{\mu_{\mathrm{Base}}} = 5.45\%$ . These considerably large errors contradict prior assumptions – that for a reasonably accurate SA classifier, we can assume  $e_{\mu_{\mathrm{Base}}}$  to be fairly negligible. Similarly, our results in Tab. 2 for the SDM, show large  $e_{\mu_{\mathrm{Base}}} \in [1.49\%, 9.14\%]$ , even though the classifier is very accurate. We discuss the reason for this in more detail in Sec. 5.1.
Overall, these results are concerning as they cast doubt on the accuracy of prior reported results. For example, imp-weighting [1] which uses the same ResNet-18 source code as our experiment, reports a  $2.35\%$  relative improvement in fairness against its baseline w.r.t. Gender, which falls within the
range of our experiments smallest relative error,  $e_{\mu_{\mathrm{Base}}} = 4.98\%$ . Similarly, Teo et al. [2] and Um et al. [12] report a relative improvement in fairness of  $0.32\%$  and  $0.75\%$ , compared to imp-weighting [1]. These findings suggest that some prior results may be affected due to oversight of SA classifier's inaccuracies; see Supp. A.4 for more details on how to calculate these measurements.
Remark: In this section, we provide the keystone for the evaluation of measurement accuracy in the current framework by introducing a labeled dataset based on generated samples. These evaluation results raise concerns about the accuracy of existing framework as considerable error rates were observed even when using accurate SA classifiers, an issue previously seen to be negligible.
4 Mitigating Error in Fairness Measurements
The previous section exposes the inaccuracies in the existing fairness measurement framework. Following that, in this section, we first develop a statistical model for the erroneous output of the SA classifier,  $\hat{p}$ , to help draw a more systematic relationship between the inaccuracy of the SA classifier and error in fairness estimation. Then, with this statistical model, we propose CLEAM - a new measurement framework that reduces error in the measured  $\hat{p}$  by accounting for the SA classifier inaccuracies to output a more accurate statistical approximation of  $p^*$ .
4.1 Proposed Statistical Model for Fairness Measurements
As shown in Fig.1(a), to measure the fairness of the generator, we feed  $n$  generated samples to the SA classifier  $C_{\mathbf{u}}$ . The output of the SA classifier  $(\hat{\pmb{p}})$  is in fact a random variable that aims to approximate the  $p^*$ . Here, we propose a statistical model to derive the distribution of  $\hat{\pmb{p}}$ .
As Fig.1(b) demonstrates in our running example of a binary SA, each generated sample is from class 0 with probability  $p_0^*$ , or from class 1 with probability  $p_1^*$ . Then, generated sample from class  $i$  where  $i \in \{0,1\}$ , will be classified correctly with the probability of  $\alpha_i$ , and wrongly with the probability of  $\alpha_i' = 1 - \alpha_i$ . Thus, for each sample, there are four mutually exclusive possible events denoted by  $\mathbf{c}$ , with the corresponding probability vector  $\mathbf{p}$ :
$$
\mathbf {c} ^ {T} = \left[ \begin{array}{l l l l} c _ {0 | 0} & c _ {1 | 0} & c _ {1 | 1} & c _ {0 | 1} \end{array} \right], \quad \mathbf {p} ^ {T} = \left[ \begin{array}{l l l l} p _ {0} ^ {*} \alpha_ {0} & p _ {0} ^ {*} \alpha_ {0} ^ {\prime} & p _ {1} ^ {*} \alpha_ {1} & p _ {1} ^ {*} \alpha_ {1} ^ {\prime} \end{array} \right] \tag {2}
$$
where  $c_{i|j}$  denotes the event of assigning label  $i$  to a sample with GT label  $j$ . Given that this process is performed independently for each of the  $n$  generated images, the probability of the counts for each output  $\mathbf{c}^T$  in Eqn. 2 (denoted by  $\mathbf{N_c}$ ) can be modeled by a multinomial distribution, i.e.  $\mathbf{N_c} \sim Multi(n,\mathbf{p})$  [37-39]. Note that  $\mathbf{N_c}$  models the joint probability distribution of these outputs, i.e.  $\mathbf{N_c} \sim \mathbb{P}(N_{c_0|0}, N_{c_1|0}, N_{c_1|1}, N_{c_0|1})$  where,  $N_{c_i|j}$  is the random variable of the count for event  $c_{i|j}$  after classifying  $n$  generated images. Since  $\mathbf{p}$  is not near the boundary of the parameter space, and as we utilize a large  $n$ , based on the central limit theorem,  $Multi(n,\mathbf{p})$  can be approximated by a multivariate Gaussian distribution,  $\mathbf{N_c} \sim \mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$ , with  $\boldsymbol{\mu} = n\mathbf{p}$  and  $\boldsymbol{\Sigma} = n\mathbf{M}$  [40, 39], where  $\mathbf{M}$  is defined as:
$$
\mathbf {M} = \operatorname {d i a g} (\mathbf {p}) - \mathbf {p p} ^ {T} \tag {3}
$$
$\text{diag}(\mathbf{p})$  denotes a square diagonal matrix corresponding to vector  $\mathbf{p}$  (see Supp A.1 for expanded form). The marginal distribution of this multivariate Gaussian distribution gives us a univariate (one-dimensional) Gaussian distribution for the count of each output  $\mathbf{c}^T$  in Eqn. 2. For example, the distribution of the count for event  $c_{0|0}$ , denoted by  $N_{c_{0|0}}$ , can be modeled as  $N_{c_{0|0}} \sim \mathcal{N}(\boldsymbol{\mu}_1, \boldsymbol{\Sigma}_{11})$ .
Lastly, we find the total percentage of data points labeled as class  $i$  when labeling  $n$  generated images using the normalized sum of the related random variables, i.e.  $\hat{p}_i = \frac{1}{n}\sum_jN_{c_i|j}$ . For our binary example,  $\hat{p}_i$  can be calculated by summing random variables with Gaussian distribution, which results in another Gaussian distribution [41], i.e.,  $\hat{p}_0\sim \mathcal{N}(\tilde{\mu}_{\hat{p}_0},\tilde{\sigma}_{\hat{p}_0}^2)$ , where:
$$
\tilde {\mu} _ {\hat {p} _ {0}} = \frac {1}{n} \left(\boldsymbol {\mu} _ {1} + \boldsymbol {\mu} _ {4}\right) = p _ {0} ^ {*} \alpha_ {0} + p _ {1} ^ {*} \alpha_ {1} ^ {\prime} \tag {4}
$$
$$
\tilde {\sigma} _ {\hat {p} _ {0}} ^ {2} = \frac {1}{n ^ {2}} \left(\boldsymbol {\Sigma} _ {1 1} + \boldsymbol {\Sigma} _ {4 4} + 2 \boldsymbol {\Sigma} _ {1 4}\right) = \frac {1}{n} \left[ \left(p _ {0} ^ {*} \alpha_ {0} - \left(p _ {0} ^ {*} \alpha_ {0}\right) ^ {2}\right) + \left(p _ {1} ^ {*} \alpha_ {1} ^ {\prime} - \left(p _ {1} ^ {*} \alpha_ {1} ^ {\prime}\right) ^ {2}\right) \right] + \frac {2}{n} p _ {0} ^ {*} p _ {1} ^ {*} \alpha_ {0} \alpha_ {1} ^ {\prime} (5)
$$
Similarly  $\hat{p}_1\sim \mathcal{N}(\tilde{\mu}_{\hat{p}_1},\tilde{\sigma}_{\hat{p}_1}^2)$  with  $\tilde{\mu}_{\hat{p}_1} = (\pmb {\mu}_2 + \pmb {\mu}_3) / n$  , and  $\tilde{\sigma}_{\hat{p}_1}^2 = (\Sigma_{22} + \Sigma_{33} + 2\Sigma_{23}) / n^2$  which is aligned with the fact that  $\hat{p}_1 = 1 - \hat{p}_0$
Algorithm 1: Computing point and interval estimates using CLEAM.
Require: accuracy of SA classifier,  $\alpha$
1 Compute SA classifier output  $\hat{p}:\{\hat{p}^1,\dots ,\hat{p}^s\}$  for  $s$  batches of generated data.
2 Compute sample mean  $\ddot{\mu}_{\hat{p}}$  and sample variance  $\dot{\sigma}_{\hat{p}}^2$  using (6) and (7).
3 Use (8) to compute point estimate  $\mu_{\mathrm{CLEAM}}$
4 Use (10) to compute interval estimate  $\rho_{\mathrm{CLEAM}}$