SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Let  $\mathcal{D}_{\mathbb{Q}^{\lambda}}$  be a dataset of noise samples from the space  $\mathcal{Z}$  distributed according to  $\mathbb{Q}^{\lambda}$ , such that  $\mathcal{D}_{\mathbb{Q}^{\lambda}}$  is obtained as described in the previous subsection (i.e.,  $\mathcal{D}_{\mathbb{Q}^{\lambda}}$  comes from Algorithm 1 in the case where the error rates  $E$  are unknown or  $C_{\phi}$  can be assumed to be an ideal classifier, or from Algorithm 2 otherwise). Now, we train a distribution mapper  $M_{\omega}: \mathcal{Z} \to \mathcal{Z}$  such that if  $\mathbf{v} \sim \mathbb{P}_{\mathrm{noise}}$  then  $M_{\omega}(\mathbf{v}) \sim \mathbb{Q}^{\lambda}$ , where  $\mathbb{P}_{\mathrm{noise}}$  is an easy-to-sample-from noise distribution such as a multivariate Gaussian, or whatever noise distribution was used as input when training  $G_{\theta}$ . As previously stated, the functional form of  $M_{\omega}$  and the procedure used to train it is flexible. There is a wide class of generative models and training strategies that could be employed [3]. We chose to model  $M_{\omega}$  as a GAN generator and find the parameters of  $M_{\omega}$  using adversarial training. Thus, we train  $M_{\omega}$  as such:
$$
\max _ {\rho} \min _ {\omega} L (M _ {\omega}, F _ {\rho}) = \mathbb {E} _ {\mathbf {z} \sim \mathcal {D} _ {\mathbb {Q} ^ {\lambda}}} \Big [ \log \big (F _ {\rho} (\mathbf {z}) \big) \Big ] - \mathbb {E} _ {\mathbf {v} \sim \mathbb {P} _ {\mathrm {n o i s e}}} \Big [ \log \big (1 - F _ {\rho} (M _ {\omega} (\mathbf {v})) \big) \Big ],
$$
where  $F_{\rho}$  is a discriminator network.
Note that  $\mathbf{z}$  is a multivariate vector, where the ordering of the elements of the vector are arbitrary (i.e., there is likely no spacial relationship in  $\mathbf{z}$ ). Thus, the inductive bias of convolutional filters used in many leading GANs [20, 19, 32] is not appropriate in this case. Rather, we suggest to train the distribution mapper using a GAN architecture designed for tabular data; notably, CTGAN which has shown to perform well on data where a sequential or spacial inductive bias is not appropriate [43].
After training  $M_{\omega}$  we can sample an arbitrary amount of samples from  $\mathbb{Q}$  without applying Algorithm 1 or Algorithm 2, as  $M_{\omega}(\mathbf{v}) \sim \mathbb{Q}^{\lambda}$  for  $\mathbf{v} \sim \mathbb{P}_{\mathrm{noise}}$ . Importantly,  $C'(G_{\theta}(M_{\omega}(\mathbf{v}))) \sim \mathrm{Unif}(\mathcal{Y})$  when  $\mathbb{Q}^{\lambda}$  is a Fair Noise Distribution, and will otherwise have minimum divergence from  $\mathrm{Unif}(\mathcal{Y})$  under the constraints given in Definition 2. Notably, we achieve this with no samples of real data, an imperfect classifier, and without fine-tuning  $G_{\theta}$ . Instead, the only training needed is for  $M_{\omega}$ , which in general should require much less complexity (and thus much less cost) than  $G_{\theta}$ . Figure 2 shows the Distribution Mapper paired with the pretrained generator  $G_{\theta}$ .
We now experimentally evaluate our approach. Details such as hyperparameter choice and architectures are available in the appendix. Code for our method is available in the supplemental material.
We compare our approach on a range of generators: VAE [21], DCGAN [32], Progressive GAN [19], and a Latent Diffusion Model (LDM) [35].
4.1 Compared Methods
Latent Editing (2019). We apply the commonly used latent editing [17, 38, 6, 33, 39, 8] approach to the task of constructing a dataset with an equal number of instances from each group in  $\mathcal{V}$ . While some work uses non-linear directions when editing [1], using linear directions has been shown to work as well in practice [17, 8]. Thus, for each group  $y \in \mathcal{V}$ , we fit a linear classifier  $K_y$  on noise samples from  $\mathbb{P}_{noise}$ , with the goal of separating noise instances that are mapped to group  $y$  from those mapped to all other groups. After training these classifiers, we can sample a noise instance belonging to a group with the following procedure: First, sample  $z \sim \mathbb{P}_{noise}$ ; if  $K_y(z) = y$  then
(a) Effect of bias correction; 0.5 (dashed line) is ideal. (b) Comparison of methods; 0.5 (dashed line) is ideal.
Figure 3: a) Effects of applying our bias correction when constructing Fair Noise dataset; b) comparison of methods on the Shapes dataset.
return  $z$ , otherwise perform latent editing on  $z$  until  $K_{y}(z_{new}) = y$ . The editing approach is given by  $z_{new} = z_{previous} + \epsilon \cdot \eta_{y}$ , where  $\eta_{y}$  is the normal vector to  $K_{y}$ 's decision boundary and  $\epsilon$  is a step size. We set  $\epsilon = 0.1$  in our experiments.
MaGNET (2022). This method was recently proposed to uniformly sample the manifold of pretrained generative models [15], using a sampling strategy that leverages the Jacobian of the generative model. The authors argue that while this does not guarantee an equal distribution for each group, MaGNET should increase the frequency of disadvantaged groups by sampling more often from low-density regions of the generative manifold.
Polarity Sampling (2022). An extension of MaGNET, Polarity Sampling allows for more controlled sampling over the generative manifold [16]. Sampling is controlled by a parameter  $\rho$  where as  $\rho$  goes to  $-\infty$  modes are sampled from increasingly often, antimodes are sampled from more as  $\rho \to \infty$ , and the original generative distribution is sampled from for  $\rho = 0$ . We thus compare against two Polarity Sampling settings; Polarity Mode Sampling where  $\rho = -2$ , and Polarity Antimode Sampling where  $\rho = 2$ . By a similar argument used for MaGNET, sampling from antimodes may result in minority groups being more represented.
Shifted Semantic Distribution (2021). This method provides a training-free approach for debiasing pretrained generative models [42]. The key idea is to fit Gaussian mixture models on the latent space, where these mixtures are fit on regions that correspond to unique values of the semantic attribute. However, unlike our approach, Shifted Semantic Distribution assumes that each attribute class can be uniquely identified by a hyperplane in the noise space.
Standard Generator. We compare against unmodified pretrained generators used as initially intended. This is a baseline which other methods should outperform.
4.2 Uniformly Sampling From Shapes Dataset
We first evaluate our approach on a dataset of synthetic images. Each image is of either a circle or a square, where the shape has a random size, color, and position in the image. We use the object's shape as the semantic attribute; thus,  $\mathcal{Y} = \{\text{'Circle', 'Square}\}$ . This dataset was first used by Jing et al. [18]. We utilize the VAE Jing et al. compared against in the same work as our generator.
Testing Classifier Bias Correction Approach. We demonstrate the utility of correcting for a biased classifier when constructing a dataset of Fair Noise samples. To that end, we compare the distribution of each group when we trust the biased classifier  $C_{\phi}$  (Algorithm 1) to the distribution found when using our proposed biased correction approach (Algorithm 2). We use a linear classifier trained to distinguish images of squares from circles as our biased classifier  $C_{\phi}$ . This classifier is implemented using Scikit-learn's LinearSVC classifier [31]. As a ground truth, we utilize a deep convolutional network classifier that achieves approximately perfect accuracy on the task of distinguishing squares from circles. Thus, this network acts as  $C'$ .
Figure 3 a) shows the distribution of shapes in the Fair Noise Dataset with error correction (Algorithm 2) and without applying our bias correction approach (Algorithm 1). As  $|\mathcal{V}| = 2$  for this experiment, the best case is when the density of each group is 0.5. Clearly, our bias correction approach yields a dataset that is much closer to being uniform over the semantic attribute. This implies the need to correct for inaccurate classifiers when constructing Fair Noise Datasets.
Comparative Study on Shapes Dataset. We next test our proposed Distribution Mapping approach (Section 3.2) against the compared methods. For our approach, we train  $M_{\omega}$  as a CTGAN [43] trained on the corrected Fair Noise Distribution obtained in the previous experiment. As the compared Latent Editing method also requires a classifier as a ground truth for training each of its linear models  $K_{y}$ , we compare against two versions of the method: 1) Latent Editing (Biased Classifier) using the biased  $C_{\theta}$  as a ground truth, and 2) Latent Editing (Perfect Classifier) which uses the ground-truth, effectively perfect convolutional network discussed above. Note that from our problem definition, this perfect classifier would usually be unavailable to us.
As Figure 3 b) shows, our approach clearly results in the most uniform distribution out of all compared methods. Additionally, the Latent Editing approach performs much worse when using the biased classifier. As expected, Polarity Antimode Sampling does indeed increase the frequency of the minority class (Circles). Interestingly, MaGNET also increases the frequency of the minority class, but is roughly as far from uniform as is the distribution of the original generator; MaGNET flips the distribution and under-represents what was previously the majority class.
4.3 Age Bias in Face Image Generator
In this next experiment, we evaluate our proposed method's ability to debias a generative model that produces images of people's faces. Specifically, the generative model is a DCGAN [32] that we pretrain on a grayscale version of the UTKFace dataset [44]. For this experiment, the semantic attribute is Age. The age of each individual in UTKFace is given as a label. As our approach assumes the semantic attribute  $\mathcal{Y}$  is discrete, we bin the ages in increments of 10 years such that  $\mathcal{Y} = \{\leq 9', '10 - 19', \dots, '90 - 99+\}$ . We train a deep convolutional network as the ground truth classifier (used for evaluation), and use a corresponding classifier with a quarter of the feature maps as the biased classifier.
Evaluating Bias Correction Approach With Large Number of Classes. We repeat the bias correction experiment we performed on the Shapes data again for the UTKFace Generator. While the previous experiment required only two classes, we now have ten classes (one for each age bin). Despite this increase, Figure 4 a) shows that performing the bias correction we propose in Algorithm 2 yields a much more uniform Fair Noise training dataset than results from trusting the biased classifier.
Comparative Study on Reducing Age Bias. We evaluate all methods on the UTKFace Generator, with the goal of making the output images uniform over Age (i.e., generate an equal number of images of people belonging to each age group). Figure 4 b) clearly shows that our approach (orange line) produces a much more uniform distribution over ages than the compared method, though it does generate images for the 10-19 bin too infrequently. While approaches such as Polarity Antimode Sampling and MaGNET are somewhat less biased away from generating images of young people, they fail to generate many samples for older individuals. Note that for the Latent Editing approach the we fit a linear regressor (guided by the ground truth) on the latent space rather than a classifier, as age is more naturally a continuous attribute.
4.4 Uniformly Sampling Over Race in Progressive GAN
We apply our approach to debias a pretrained Progressive GAN [19]; specifically, the PyTorch [30] version of the 'celebAHQ-256' model. We use the Race of the individual in each image as the semantic attribute, and use the MTCNN classifier from the DeepFace [36] package to classify race. Matching the classes available in DeepFace, our semantic attribute space is  $\mathcal{Y} = \{\text{Asian}, \text{Indian}, \text{White}, \text{Middle Eastern}, \text{Latino Hispanic}, \text{Black}\}$ . The Progressive GAN strongly favors generating white individuals, likely because its training images were of predominantly white celebrities. Here, we consider the classifier to be accurate.
2see https://pytorch.org/hub/facebookresearch_pytorch-gan-zoo_pgan/
(a) Effect of bias correction; 0.1 (dashed line) is ideal.
(b) Comparison of methods; 0.1 (dashed line) is ideal.
Figure 4: a) Our biased correction approach yields better Fair Noise Distributions; b) comparison of methods on correcting for age bias in the UTKFace DCGAN.
Figure 5: KL Divergence between the distribution over the semantic space for the output of each method (lower is better).
Comparative Study of Debiasing Progressive GAN. For each method, we report the KL divergence between a uniform distribution and the classifier's output on the samples generated from the method in Figure 5. Note that unlike in the previous experiments, for this metric lower is better. Our approach has the lowest KL divergence, indicating it produces a more equal number of images of people from each race than any compared method. We note that despite this we still observe an over-representation of images of white individuals in the samples produced by our (and all other) approaches (see Appendix). This indicates that most regions of the latent space are likely associated with semantic attribute, and unless the distribution mapper very closely matches its Fair Noise training distribution, white persons will still be over-represented. Still, our approach results in a distribution that is most fair out of all compared methods. As we observed previously, sampling antimodes with Polarity Sampling produces next-best results; likely because it explicitly draws from low-probability regions of the latent space, which correspond to non-white individuals.
4.5 Uniformly Sampling Age in Latent Diffusion Model
Although our focus is primarily on models that map from low dimensional latent space to a higher dimensional feature space, we also evaluate our approach on a latent diffusion model  $^3$  [35] that was trained on the Celeba-HQ dataset [19]. For this experiment we chose Age as the semantic attribute, and debias according to the following age categories:  $\{\leq 29, 30-49, \geq 50\}$ . We utilize a pretrained ViT age classifier  $^4$  to provide feedback for the semantic attribute. Since our approach requires each latent code to be mapped to a single output image, we utilized the DDIM [41] sampler for the diffusion model to yield a deterministic diffusion process. We used a DDIM diffusion model to train the Mapper network as well.
4.5.1 Comparative Study on Debiasing Latent DDPM.
We report the KL divergence from uniform for the semantic distribution of each compared method in Table 1. Note that we exclude MaGNET and Polarity Sampling from this experiment, due to
3We utilized this model: https://huggingface.co/CompVis/ldm-celebahq-256
<sup>4</sup>https://huggingface.co/nateraw/vit-age-classifier
Distribution Mapping (Ours)	Latent Editing	Vanilla Latent Diffusion
KL Divergence From Uniform Distribution For Semantic Attribute “Age” (Lower is Better)	0.130	0.525	0.411
Table 1: Comparative study on debiasing the Age attribute for a pretrained LDM.
the difficulty of obtaining the Jacobian determinant required by these methods. Results show that our approach yields significantly better scores than the other compared method, indicating that our approach can potentially be useful for debiasing Diffusion models as well.
5 Broader Impact And Limitations
The goal of this work, reconditioning generative models to not reproduce the biased distribution of their training set but rather produce one that treats each group equitably, is driven by the desire to mitigate the effects that systemic biases have on generative models increasingly used in real world applications. This has the potential for beneficial societal impact by counteracting the harmful effects of such systemic biases that lead these models to be unfair to underrepresented groups. However, the potential negative impacts of advancing generative modeling should not go unconsidered. Such models have already been used for some harmful applications such as Deep Fakes [28]. It is important to stress that while our approach aims to mitigate the effects of bias in existing models, it is not a fix-all nor an excuse to train models on knowingly biased data. When possible, it is essential to collect fair and equitable training datasets, and to take measures to ensure that the models we train are fair without the need for post-hoc corrections.
Limitations. Currently, our approach assumes that the semantic space  $\mathcal{V}$  is discrete. We plan to extend this work to handle continuous attributes, such as skin tone, in future work. Additionally, while the conditional distribution  $P(\mathbf{x}|\mathbf{y})$  should be roughly equal before and after applying our method if the classifier  $C_{\phi}$  is accurate, in the case of a biased classifier the conditional distribution may change. Correcting for potential distribution shift is likewise future work.
This is the first method to correct a pretrained biased generative model (i.e., one that strongly favors generating images of white people over all other races) given only the generator and a potentially-biased classifier. We propose a sampling strategy to construct a fair training set using the biased classifier in a way that corrects for its bias, and a Distribution Mapping module that uses this training set to learn how to sample noise instances that produce fair outputs when used as input to the generative model. Notably, we are able to debias the generative model without retraining the model or utilizing any real data. Our results indicate that our approach produces outputs that are much fairer than existing methods. This work may inspire more research on Distribution Mapping techniques to recondition generative models by transforming their standard latent distributions into distributions that yield more favorable behavior.
This material is based on research sponsored by DARPA under agreement number FA8750-18-2-0077. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government.
Results in this paper were obtained in part using a high-performance computing system acquired through NSF MRI grant DMS-1337943 to WPI. We thank the members of WPI's DAISY Research lab and the WPI WASH research group for their insightful feedback and support during the development of this work.
[35] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-Resolution Image Synthesis with Latent Diffusion Models, Apr. 2022. arXiv:2112.10752 [cs].
[36] S. I. Serengil and A. Ozpinar. HyperExtended LightFace: A Facial Attribute Analysis Framework. In 2021 International Conference on Engineering and Emerging Technologies (ICEET), pages 1-4, Oct. 2021. ISSN: 2409-2983.
[37] X. Shao, K. Stelzner, and K. Kersting. Right for the Right Latent Factors: Debiasing Generative Models via Disentanglement, Feb. 2022. arXiv:2202.00391 [cs].
[38] Y. Shen, J. Gu, X. Tang, and B. Zhou. Interpreting the Latent Space of GANs for Semantic Face Editing. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9240-9249, Seattle, WA, USA, June 2020. IEEE.
[39] Y. Shen, C. Yang, X. Tang, and B. Zhou. InterFaceGAN: Interpreting the Disentangled Face Representation Learned by GANs, Oct. 2020. arXiv:2005.09635 [cs, eess].
[40] T. Sipka, M. Sulc, and J. Matas. The Hitchhiker's Guide to Prior-Shift Adaptation. In 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 2031–2039, Waikoloa, HI, USA, Jan. 2022. IEEE.
[41] J. Song, C. Meng, and S. Ermon. Denoising Diffusion Implicit Models, Oct. 2022. arXiv:2010.02502 [cs].
[42] S. Tan, Y. Shen, and B. Zhou. Improving the Fairness of Deep Generative Models without Retraining, Mar. 2021. arXiv:2012.04842 [cs].
[43] L. Xu, M. Skoularidou, A. Cuesta-Infante, and K. Veeramachaneni. Modeling Tabular data using Conditional GAN. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
[44] Z. Zhang, Y. Song, and H. Qi. Age Progression/Regression by Conditional Adversarial Autoencoder. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4352-4360, July 2017. ISSN: 1063-6919.
A.1 Calculating  $E$  for The Generated Distribution
The error rates reported for a classifier  $C_{\phi}$  are typically reported on the distribution on the distribution of fit's training data,  $\mathbb{P}_{training}$ . However, the distribution  $\mathbb{P}_{G_{\theta}}$  of the generative model
$G_{\theta}$  may differ from the training distribution. Additionally, rather than reporting  $P(\mathbf{y}|\hat{\mathbf{y}})$ , often times the error rates are given in a confusion matrix  $C_{\hat{\mathbf{y}}|\mathbf{y}}$  where  $C_{\hat{\mathbf{y}}|\mathbf{y}}[i,j] = P(\hat{\mathbf{y}}|\mathbf{y})$ . Thankfully, we can construct the error rate matrix  $E$  for the generative distribution  $\mathbb{P}_{G_{\theta}}$  under the simplifying assumption that the difference between  $\mathbb{P}_{G_{\theta}}$  and  $\mathbb{P}_{training}$  can be explained as a label shift [10, 40].
By Bayes' Theorem, we know that
$$
P (\mathbf {y} | \hat {\mathbf {y}}) = P (\hat {\mathbf {y}} | \mathbf {y}) \frac {P (\mathbf {y})}{P (\hat {\mathbf {y}})}.
$$
Under the label shift assumption,  $P(\hat{\mathbf{y}}|\mathbf{y})$  stays the same between  $\mathbb{P}_{\text{training}}$  and  $\mathbb{P}_{G_{\theta}}$ . Additionally,  $P(\mathbf{y})$  can be calculated for  $\mathbb{P}_{G_{\theta}}$  under label shift [10, 40]. Lastly,  $P(\hat{\mathbf{y}})$  can be approximated for  $\mathbb{P}_{G_{\theta}}$  by finding the proportion predicted for each class on a large sample from the generative model. Thus,  $E$  can be calculated as:
$$
E = C _ {\hat {\mathbf {y}} | \mathbf {y}} \frac {P _ {G _ {\theta}} (\mathbf {y})}{P _ {G _ {\theta}} (\hat {\mathbf {y}})}.
$$
A.2 Distribution of Races Generated By Progressive GAN
We show the two best performing methods' distributions on Progressive GAN, along with the distribution of the unmodified ProgressiveGAN, over the Race attribute.
Figure 6: Distribution of our approach, Polarity Antimode Sampling (next best), and the standard generator.
A.3 Comparison Of Generated Faces From LDM Model
A comparison of a bath of random faces from the original LDM is shown in Figure 7, while Figure 8 shows a batch from the LDM when our Distribution Mapping method is applied.
Ground Truth Shape Classifier
Layer (type)
Output Shape
Figure 7: Random sample of generated faces from the original LDM model (no Distribution Mapping or fail sampling applied).
Figure 8: Random sample of generated faces from the LDM model after our Distribution Mapping approach.
Conv2d-1	[-1, 32, 16, 16]
ReLU-2	[-1, 32, 16, 16]
Conv2d-3	[-1, 64, 8, 8]
ReLU-4	[-1, 64, 8, 8]
Conv2d-5	[-1, 128, 4, 4]
ReLU-6	[-1, 128, 4, 4]
Conv2d-7	[-1, 256, 2, 2]
ReLU-8	[-1, 256, 2, 2]
Conv2d-9	[-1, 2, 1, 1]
Encoder for Shapes VAE
Layer (type)	Output Shape
Conv2d-1	[-1, 32, 16, 16]
ReLU-2	[-1, 32, 16, 16]
Conv2d-3	[-1, 64, 8, 8]
ReLU-4	[-1, 64, 8, 8]
Conv2d-5	[-1, 128, 4, 4]
ReLU-6	[-1, 128, 4, 4]
Conv2d-7	[-1, 256, 2, 2]
ReLU-8	[-1, 256, 2, 2]
Conv2d-9	[-1, code_dim, 1, 1]
Decoder for Shapes VAE
Layer (type)	Output Shape
ConvTranspose2d-1	[-1, 256, 2, 2]
ReLU-2	[-1, 256, 2, 2]
ConvTranspose2d-3	[-1, 128, 8, 8]
ReLU-4	[-1, 128, 8, 8]
ConvTranspose2d-5	[-1, 64, 16, 16]
ReLU-6	[-1, 64, 16, 16]
ConvTranspose2d-7	[-1, 32, 32, 32]
ReLU-8	[-1, 32, 32, 32]
ConvTranspose2d-9	[-1, 3, 64, 64]
Sigmoid-10	[-1, 3, 64, 64]