SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Walter Gerych $^{1}$ , Kevin Hickey $^{1}$ , Luke Buquicchio $^{1}$ , Kavin Chandrasekaran $^{1}$ , Abdulaziz Alajaji $^{2}$ , Elke Rundensteiner $^{1}$ , Emmanuel Agu $^{1}$
$^{1}$ Worcester Polytechnic Institute, Worcester, MA
$^{2}$ King Saud University, Riyadh, Saudi Arabia
Generative models are being increasingly used in science and industry applications. Unfortunately, they often perpetuate the biases present in their training sets, such as societal biases causing certain groups to be underrepresented in the data. For instance, image generators may overwhelmingly produce images of white people due to few non-white samples in their training data. It is imperative to debias generative models so they synthesize an equal number of instances for each group, while not requiring retraining of the model to avoid prohibitive expense. We thus propose a distribution mapping module that produces samples from a fair noise distribution, such that the pretrained generative model produces semantically uniform outputs - an equal number of instances for each group - when conditioned on these samples. This does not involve retraining the generator, nor does it require any real training data. Experiments on debiasing generators trained on popular real-world datasets show that our method outperforms state-of-the-art approaches.
Background. Generative models have become a cornerstone of modern machine learning, allowing for the synthesis of realistic data for many domains, including images [19, 35], audio [22], and text [11, 4]. However, even leading generative models often reproduce the biases present in their training data [13, 26], such as image generation models strongly over-representing white males [24]. As generative models are increasingly used for data augmentation to train downstream models [5] in domains from scientific to medical fields [25, 9], biased synthesized data could lead to results that are skewed or inaccurate. This can exacerbate existing issues such as facial recognition models performing significantly worse on non-white individuals [29] or healthcare models being much less accurate for certain minority groups [23]. Further, with the rapid growth of generative models in commercial applications, the potential financial, legal and ethical costs of biased outputs are significant. Thus, it is imperative to develop methods that mitigate bias in generative models to ensure that their outputs are fair and equitable by generating a roughly equal number of samples of each group. We call such outputs semantically uniform, and the attribute that they are uniform over - such as gender or race - the semantic attribute.
State-of-the-art. Existing methods for addressing bias in generative models often train a new model from scratch [27, 37, 2], though this is computationally expensive, requires significant labeled data, and wastes resources already previously spent training the existing (biased) generative model. Latent attribute editing methods modify the samples in the latent space of generative models to produce controlled changes in the output, and could potentially be used to correct for bias [17]. However, this requires making limiting assumptions such as that semantic attributes correspond to linear directions
*Corresponding author: wgerych@wpi.edu
Figure 1: A generator conditioned on a fair noise distribution yields outputs that are uniform over a semantic attribute (i.e. race).
in the latent space [17, 38, 6, 33, 39, 8]. While some recent advances have been made in uniformly sampling the output space of a pretrained model [15, 16], they can fail to yield uniform samples over semantic attributes [15]. For example, they will make an image generator's output uniform over a manifold in the pixel space, but may still overproduce images of white individuals as the output won't be uniform over the semantic attribute of race.
Problem statement. Recently, numerous pretrained generative models and classifiers have been released for a variety of domains. For instance, there are publicly available generators for images of faces [19] followed by classifiers that predict attributes such as race and age of faces [36]. Here, we assume access to a pretrained generative model  $G_{\theta}$  that maps a noise space  $\mathcal{Z}$  to a feature space  $\mathcal{X}$ , and a pretrained classifier  $C_{\phi}$  that maps  $\mathcal{X}$  to a discrete semantic attribute space  $\mathcal{Y}$ . Additionally, as error rates are often reported for released models, we assume that the class-conditional errors; i.e., the probability that an instance actually belongs to group  $j$  given that the classifier incorrectly predicted group  $i$ , is known. Our goal is to model a distribution  $\mathbb{Q}$  such that for  $\mathbf{z} \sim \mathbb{Q}$ ,  $G_{\theta}(\mathbf{z})$  produces samples that are uniform over the attribute space  $\mathcal{Y}$ , using only predictions from  $C_{\phi}$  to guide the output of  $G_{\theta}$ . For instance, if there are five discrete semantic groups in  $\mathcal{Y}$ ,  $G_{\theta}$  will produce an equal number of instances belonging to each group when conditioned on draws from  $\mathbb{Q}$ . We call  $\mathbb{Q}$  a fair noise distribution; see Figure 1 for an example.
Challenges. Our task of producing samples that are semantically uniform has three major hurdles:
1. Expensive retraining: It is often prohibitively expensive to retrain large generative models in terms of computational resources and time.
2. Inaccessible training data: The data used to train a released generator is typically unavailable; either due to being proprietary or simply too large of a volume for most practitioners to utilize. Thus, we do not have adequate data available to tune the generator or classifier.
3. Inaccurate classifier: As we do not have any samples of real data available, we must rely on the possibly imperfect semantic attribute classifier  $C_{\phi}$  to provide labels. However, since classifiers may produce inaccurate predictions - especially on underrepresented groups of  $\mathcal{V}$ , this can cause incorrect estimations of the number of instances for each group.
Proposed approach. We propose to train a distribution mapper network  $M_{\omega} : \mathcal{Z} \rightarrow \mathcal{Z}$  that transforms draws from a standard noise distribution into draws from a fair noise distribution, such that once noise samples are transformed by  $M_{\omega}$  they condition the generator to produce a roughly equal number of instances for each semantic group in  $\mathcal{V}$ . We achieve this by first constructing a dataset of instances from  $\mathcal{Z}$  that follow a fair noise distribution, which allows us to estimate the true distribution of the semantic attribute given an imperfect classifier. Using our proposed strategy, noise samples are collected such that the pretrained classifier's (corrected) distribution is uniform. Then, we use this fair noise dataset to train  $M_{\omega}$  to inexpensively sample new instances from the fair noise distribution. The output of  $M_{\omega}$  serves as input to the pretrained generator, which yields samples that are uniform over the space of the semantic attribute. Importantly, this approach does not require retraining the main generative model, thus incurring minimal costs. Our method works without requiring any real training data. Relying only on a pretrained generator and classifier pair, we circumvent the need to acquire any real data (labeled or otherwise) even using only an imperfect classifier.
Contributions. In this work, we:
- Show how to construct a fair noise dataset that produces semantically uniform synthetic outputs when passed through a generator, assuming an accurate classifier is available.
- Design a method to correct for the error incurred by an imperfect classifier, to construct a dataset of samples that come from a fair noise distribution.
- Propose an approach for training a distribution mapping network to sample from a fair noise distribution on-the-fly.
- Demonstrate the utility of our approach on a range of real-world datasets and released generators.
Assume we are given a pretrained generative model  $G_{\theta}$  that maps a latent space  $\mathcal{Z}$  to a feature space  $\mathcal{X}$ . In general, we assume that  $\mathcal{Z}$  is of a lower dimensionality than  $\mathcal{X}$ . Additionally, we assume we are given a pretrained classifier  $C_{\phi}$  that maps  $\mathcal{X}$  to a semantic attribute space  $\mathcal{Y}$ , where  $\mathcal{Y}$  is the set of group (class) labels. Thus,  $\mathcal{Y} = \{Y_1, Y_2, \ldots, Y_N\}$ , where  $Y_i$  is the label of the  $i$ th group. When not otherwise ambiguous, we will refer to group  $Y_i$  as group  $i$ . Let  $\mathbf{y}$  and  $\hat{\mathbf{y}}$  be the random variables indicating the true semantic attribute label and the predicted label from  $C_{\phi}$  respectively. Let  $E$  be the prediction-conditional error rates of  $C_{\phi}$ , with  $E$  being a left stochastic matrix such that  $E_{i,j} = P(\mathbf{y} = i|\hat{\mathbf{y}} = j)$ . Let  $C_{\phi}$  be a better-than-random classifier, i.e., for  $N$  groups,  $P(\mathbf{y} = i|\hat{\mathbf{y}} = i) > \frac{1}{N}$  for all groups  $i$ . Thus,  $E$  is by definition a diagonally dominant matrix. As it is typical for developers to report the error rates of their classifiers, we assume that  $E$  is known. While these error rates are typically reported for the classifier's training distribution, we can correct  $E$  to apply to the generated distribution under a label shift assumption [40]. See the Appendix for details on this and for the proofs of the upcoming theorems. Lastly, let  $C'$  denote an ideal perfect classifier that always correctly predicts  $\mathbf{y}$  with zero error.  $C'$  is hypothetical and unavailable to us.
Our goal here is to sample from a Fair Noise Distribution, defined as follows:
Definition 1 (Fair Noise Distribution). For a generative model  $G_{\theta}: \mathcal{Z} \to \mathcal{X}$ , a distribution  $\mathbb{Q}$  over  $\mathcal{Z}$  is a Fair Noise Distribution with respect to  $\mathcal{Y}$  if for  $\mathbf{z} \sim \mathbb{Q}$ ,  $C'(G_{\theta}(\mathbf{z})) \sim \mathrm{Unif}(\mathcal{Y})$ , where  $\mathrm{Unif}(\mathcal{Y})$  is the uniform distribution over the groups in  $\mathcal{Y}$ .
Intuitively, a distribution  $\mathbb{Q}$  is a Fair Noise Distribution if  $G_{\theta}$  produces samples that are uniform over the semantic space  $\mathcal{V}$  when conditioned on  $\mathbb{Q}$ . For example, if  $G_{\theta}$  synthesizes images of people and  $\mathcal{V}$  is the set of races, then  $\mathbb{Q}$  is a Fair Noise Distribution if conditioning on it makes  $G_{\theta}$  produce a roughly equal number of images of people from each race.
More realistically, we want to find a  $\mathbb{Q}$  that is easy to sample from and produces reasonable variance from the generator. That is, it does not yield only one unique example of each group, which would otherwise make the problem trivial. Notably, we assume that we do not have any real training data (i.e., no real samples from  $\mathcal{X}$ ). Also, we do not aim to retrain  $G_{\theta}$ , i.e., not change the parameters  $\theta$  of  $G_{\theta}$ . Additionally, we do not make any assumptions on the differentiability of  $G_{\theta}$  or  $C_{\phi}$ .
Our approach for sampling from a Fair Noise Distribution  $\mathbb{Q}$  hinges on training a distribution mapping function  $M_{\omega}$  such that  $M_{\omega}(\mathbf{z}) \sim f\mathbb{Q}$ , where  $\mathbf{z}$  is a draw from the original conditioning distribution of  $G_{\theta}$ . The functional form of  $M_{\omega}$  has many options; for instance,  $M_{\omega}$  can be a GAN generator [12], a VAE [21], a DDPM [14], or a normalizing flow model [34, 7]. No matter which form is chosen, we will need a dataset of samples drawn from  $\mathbb{Q}$  to train  $M_{\omega}$ .
3.1 Collecting Fair Samples Using Imperfect Classifiers
A naive method for collecting a dataset of samples distributed according to a Fair Noise Distribution  $\mathbb{Q}$  is given in Algorithm 1. The basic approach is to continuously sample  $\mathbf{z}$  from the noise distribution, collect the generator's output for each draw, and use the classifier to determine the value of the semantic attribute corresponding to each noise draw. As a result, samples of  $\mathbf{z}$  corresponding to each group are saved to dataset  $\mathcal{D}_{\mathbb{Q}}$  until  $\mathcal{D}_{\mathbb{Q}}$  has  $S$  number of samples for each  $y \in \mathcal{V}$ . This procedure results in samples of noise that once passed through  $G_{\theta}$  will be uniform across the semantic attribute
Algorithm 1 Naively collect data from  $\mathbb{Q}$
1: procedure NAIVE_Q_DATASET( $G_{\theta}, C_{\phi}, S$ ) ▷ Colect from  $\mathbb{Q}$  by trusting  $C_{\phi}$ 
2: Input: Pretrained generator  $G_{\theta}$  and classifier  $C_{\phi}$ , number of samples  $S$  for each attribute
3: Output: Dataset with  $S$  samples for each  $y \in \mathcal{Y}$ 
4:  $Q\_dataset \gets dict() \triangleright$  Initialize empty dictionary for the dataset
5: for group  $\in \mathcal{Y}$  do
6:  $Q\_dataset[group] = [] \triangleright$  Initialize every group as empty array
7: end for
8: while  $\exists key \in Q\_dataset keys s.t. length(Q\_dataset[key]) < S$  do
9:  $z \sim \mathbb{P}_{\text{noise}} \triangleright$  Sample noise
10: group =  $C_{\phi}(G_{\theta}(z)) \triangleright$  Get the predicted group of  $G_{\theta}(z)$ 
11: if length(Q_dataset[key]) < S then
12:  $Q\_dataset[class].append(z) \triangleright$  If < S samples for that class, add z to the dataset
13: end if
14: end while
15: return  $Q\_dataset \triangleright$  Return dataset with  $S$  samples for each group in  $\mathcal{Y}$ 
16: end procedure
space according to  $C_{\phi}$ ; i.e., for  $\mathbf{z} \sim \mathcal{D}_{\mathbb{Q}}$ ,  $C_{\phi}(G_{\theta}(\mathbf{z})) \sim \mathrm{Unif}(\mathcal{V})$ . However, this will only yield samples from a Fair Noise Distribution in the case where  $C_{\phi}$  is a perfect classifier (such that  $E$  is the identity matrix). If  $C_{\phi}$  is an imperfect classifier then the distribution may not be truly fair. For instance, if  $C_{\phi}$  often incorrectly predicts group  $j$  as group  $i$ , then instances with attributes matching group  $i$  may be more prevalent than those for group  $j$ .
Fortunately, we can utilize knowledge from the prediction-conditional error rates  $E$  to sample a dataset of noise that will yield more semantically uniform generated instances despite the noisy predictions of  $C_{\phi}$ . To achieve this, we use a weighted sample of datapoints predicted for each group. Let  $\mathbb{P}_{z|C_{\phi}=i}$  be a distribution over  $\mathcal{Z}$  such that the imperfect classifier's prediction of generated samples arising from this distribution are all of group  $i$ ;  $C_{\phi}(G_{\theta}(\mathbf{z})) = i$  for  $\mathbf{z} \sim \mathbb{P}_{z|C_{\phi}=i}$ . In addition, let  $\mathbb{Q}^{\lambda} = \sum_{i=1}^{|\mathcal{Y}|} \lambda_{i} \mathbb{P}_{z|C_{\phi}=i}$ , such that  $\lambda_{i} > 0 \forall i$  and  $\sum_{i=1}^{|\mathcal{Y}|} \lambda_{i} = 1$ .
We can in many instances find values of  $\lambda = \{\lambda_1, \lambda_2, \dots, \lambda_{|\mathcal{Y}|}\}$  such that  $\mathbb{Q}^{\lambda}$  is a Fair Noise Distribution, as stated in Lemma 1:
Lemma 1. Let  $\mathbf{1}^{|\mathcal{Y}|}$  be the vector of length  $|\mathcal{Y}|$  such that every element is 1. Let  $\operatorname{cone}(E) = \{\sum a_i E_{\cdot,i} | a_i \in \mathbb{R}_{\geq 0}\}$  be the finite convex cone generated by the columns of the prediction-conditional error matrix. If  $\mathbf{1}^{|\mathcal{Y}|} \in \operatorname{cone}(E)$ , then  $\exists \lambda$  such that  $\mathbb{Q}^{\lambda}$  is a Fair Noise Distribution.
Additionally, we can show that in the case where  $\mathbf{y}$  is binary and  $C_{\phi}$  is better-than-random, we can always find  $\lambda$  such that samples drawn from  $\mathbb{Q}^{\lambda}$  will definitely yield generated samples that are uniform over the semantic attributes, as shown by the following lemma.
Lemma 2. Let  $|\mathcal{Y}| = 2$  and let  $C_{\phi}$  be better-than-random, such that the diagonal elements of  $E$  are each greater than  $\frac{1}{2}$ . Then,  $\exists \lambda$  such that  $\mathbb{Q}^{\lambda}$  is a Fair Noise Distribution.
The proof of Lemma 2 comes from showing that  $\langle 1,1\rangle$  is always in the finite convex cone generated by  $C_{\phi}$ 's prediction-conditional error matrix, with Lemma 1 implying that this property guarantees that  $\mathbb{Q}^{\lambda}$  will yield samples that produce semantically uniform generated instances.
Even in the cases where we cannot guarantee that there exists an ideal  $\lambda$ , we can still find values for  $\lambda$  that will yield a distribution that is as close as possible to a uniform distribution over the semantic space, with the difference from uniformity measured by KL divergence. For this, let us define a Minimally-Unfair Noise Distribution.
Definition 2 (Minimally-Unfair Noise Distribution). For a generative model  $G_{\theta}: \mathcal{Z} \to \mathcal{X}$ , a distribution  $\mathbb{Q}^{\lambda} = \sum_{i=1}^{|\mathcal{Y}|} \lambda_i \mathbb{P}_{z|C_{\phi}=i}$ ,  $\lambda = \{\lambda_1, \lambda_2, \dots, \lambda_{|\mathcal{Y}|}\}$ ,  $\lambda_i \in \mathbb{R}_{\geq 0}$ ,  $\sum_{i=1}^{|\mathcal{Y}|} \lambda_i = 1$ , is a Minimally-Unfair Noise Distribution over  $\mathcal{Z}$  with respect to  $\mathcal{Y}$  if for  $\mathbf{z} \sim \mathbb{Q}^{\lambda}$ ,  $\lambda = \operatorname*{argmin}_{\lambda} \mathrm{KL}\{C'(G_{\theta}(\mathbf{z}))||\mathrm{Unif}(\mathcal{Y})\}$ .
Algorithm 2 Collecting a dataset of samples from  $\mathbb{Q}^{\lambda}$
1: procedure WEIGHTED_Q_DATASET( $\{\mathcal{D}_{z|C_{\phi}=i}\}_{i=1}^{|\mathcal{V}|}, S$ )  
2: Input:  $\mathcal{D}_{z|C_{\phi}=i}$  for  $i$  from  $1 \to |\mathcal{V}|$ : Dataset of noise samples that produce generated instances which  $C_{\phi}$  classifies as group  $i$ ;  $S$ : number of instances we wish to sample from  $\mathbb{Q}^{\lambda}$   
3: Output: Dataset with  $S$  samples from  $\mathbb{Q}^{\lambda}$   
4:  $\lambda \gets \text{argmax} H(\mathbb{P}_E^{\lambda})$   
5:  $Q\_dataset = []$  ▷ Initialize the  $\mathbb{Q}^{\lambda}$  as an empty array  
6: for  $m \gets 1$  to  $S$  do  
7:  $r \gets \text{random\_int}(\lambda)$  ▷ get random int (1 to  $|\mathcal{V}|$ ) with probability proportional to  $\lambda$   
8:  $z \sim \mathcal{D}_{z|C_{\phi}=r}$   
9:  $Q\_dataset.append(z)$   
10: end for  
11: return  $Q\_dataset$  ▷ Return dataset with  $S$  samples from  $\mathbb{Q}^{\lambda}$   
12: end procedure
Intuitively,  $\mathbb{Q}^{\lambda}$  is a Minimally-Unfair Noise Distribution if the values of  $\lambda$  yield generated samples with minimal divergence from a semantically uniform distribution, under the constraint that  $\mathbb{Q}^{\lambda}$  is a convex combination of the  $\mathbb{P}_{z|C_{\phi} = i}$ 's. This constraint is required so that we can sample from  $\mathbb{Q}^{\lambda}$  using a weighted sampling technique based off of our imperfect classifier  $C_{\phi}$ .
Fortunately, we can easily find the values of  $\lambda$  that will yield a Minimally-Unfair Noise Distribution. Before deriving the procedure for this, let us define the distribution  $\mathbb{P}_E^{\lambda}$  as the normalized weighted sum of the columns of  $E$ , where each column  $i$  is weighted according to a corresponding  $\lambda_{i}$ :
Definition 3  $(\mathbb{P}_E^\lambda)$ . Define  $\mathbb{P}_E^\lambda = \sum_{i=1}^{|\mathcal{Y}|} \lambda_i E_{:,i}$  as a distribution over  $\mathcal{Y}$  determined by prediction-conditional error matrix  $E$  and  $\lambda = \{\lambda_1, \lambda_2, \dots, \lambda_{|\mathcal{Y}|}\}$ ,  $\lambda_i \in \mathbb{R}_{\geq 0}$ ,  $\sum_{i=1}^{|\mathcal{Y}|} \lambda_i = 1$ .
Next, we note that finding the values for  $\lambda$  that maximize the entropy of  $\mathbb{P}_E^\lambda$  is equivalent to finding the  $\lambda$  for which the divergence between  $\mathbb{P}_E^\lambda$  and  $\mathrm{Unif}(\mathcal{V})$  is minimized. This is stated formally in the following Proposition.
Proposition 1. If  $\lambda^{*} = \operatorname{argmax}_{\lambda} H(\mathbb{P}_{E}^{\lambda})$  where  $H$  is entropy, then for the same  $\lambda^{*}$  it is true that  $\lambda^{*} = \operatorname{argmin}_{\lambda} \mathrm{KL}\{\mathbb{P}_{E}^{\lambda}||\mathrm{Unif}(\mathcal{Y})\}$ .
Before stating the theorem that directly implies a strategy for learning  $\lambda$  for which  $\mathbb{Q}^{\lambda}$  is Minimally-Unfair, all that is left to do is to link  $\mathbb{P}_E^\lambda$  with the distribution of  $C^\prime (G_\theta (\mathbf{z}))$ :
Proposition 2. If  $\mathbf{z} \sim \mathbb{Q}^{\lambda}$ , then it is true that  $C'(G_{\theta}(\mathbf{z})) \sim \mathbb{P}_{E}^{\lambda}$ .
Proposition 2 states that for a given  $\lambda$  the distribution of the perfect classifier  $C'$  given samples of the generator conditioned on draws from  $\mathbb{Q}^{\lambda}$  will be distributed according to  $\mathbb{P}_E^\lambda$ . Now, we state Theorem 1 which directly implies our sampling strategy.
Theorem 1. If  $\lambda^{*} = \operatorname{argmax}_{\lambda} H(\mathbb{P}_{E}^{\lambda})$ , then  $\mathbb{Q}^{\lambda *}$  is a Minimally-Unfair Noise Distribution. When  $C_{\phi} = C'$  or  $\mathbf{1}^{|\mathcal{V}|} \in \mathrm{cone}(E)$ , then  $\mathbb{Q}^{\lambda *}$  is also a Fair Noise Distribution.
Theorem 1 follows from the preceding Propositions. To see how Theorem 1 implies a strategy for sampling from a Minimally-Unfair Noise Distribution, recall that  $\mathbb{Q}^{\lambda}$  is defined as the mixture of distributions of each  $\mathbb{P}_{z|C_{\phi}=i}$  for  $i$  from  $1 \to |\mathcal{Y}|$  with  $\lambda_i$  as the  $i$ th mixture weight, where  $\mathbb{P}_{z|C_{\phi}=i}$  is the distribution of  $\mathbf{z}$ 's that yield generated samples that the noisy classifier predicts as group  $i$ . The above theorem states that the mixture of these distributions with mixture weights  $\lambda$  that produce maximum entropy for  $\mathbb{P}_E^\lambda$  will yield a mixture distribution  $\mathbb{Q}^{\lambda}$  that is worse-case Minimally-Unfair and, when possible, will be a perfect Fair Noise Distribution. Thus, to sample from  $\mathbb{Q}^{\lambda}$  all that is needed is to 1) construct datasets  $\mathcal{D}_{z|C_{\phi}=i}$  for  $i$  from  $1 \to |\mathcal{Y}|$  such that if  $\mathbf{z} \in \mathcal{D}_{z|C_{\phi}=i}$  then  $C_{\phi}(G_{\theta}(\mathbf{z})) = i$  (i.e., these datasets serve as proxies for  $\mathbb{P}_{z|C_{\phi}=i}$ ); 2) find  $\lambda$  such that  $H(\mathbb{P}_E^\lambda)$  is maximized; and 3) perform a weighted sampling from each  $\mathcal{D}_{z|C_{\phi}=i}$  proportional to its corresponding  $\lambda_i$ . Note that the datasets  $\mathcal{D}_{z|C_{\phi}=i}$  can be formed by an approach similar to
Figure 2: Distribution Mapper  $M_{\omega}$  turns a standard noise distribution into a Fair Noise Distribution.
Algorithm 1. The algorithm returns a dictionary with keys being the elements of  $\mathcal{V}$ , and the values of each key returned from that procedure form the corresponding required datasets (i.e.,  $\mathcal{D}_{z|C_{\phi} = i}$ ). A pseudo-code implementation of our approach for constructing a dataset of noise samples from a Minimally-Unfair Noise Distribution  $\mathbb{Q}^{\lambda}$  is given in Algorithm 2.
3.2 Training the Distribution Mapper