PFGuard: A Generative Framework with Privacy and Fairness Safeguards

## Abstract

Abstract Generative models must ensure both privacy and fairness for Trustworthy AI. While these goals have been pursued separately, recent studies propose to combine existing privacy and fairness techniques to achieve both goals. However, naïvely combining these techniques can be insufficient due to privacy-fairness conflicts, where a sample in a minority group may be amplified for fairness, only to be suppressed for privacy. We demonstrate how these conflicts lead to adverse effects, such as privacy violations and unexpected fairness-utility tradeoffs. To mitigate these risks, we propose PFGuard, a generative framework with privacy and fairness safeguards, which simultaneously addresses privacy, fairness, and utility. By using an ensemble of multiple teacher models, PFGuard balances privacy-fairness conflicts between fair and private training stages and achieves high utility based on ensemble learning. Extensive experiments show that PFGuard successfully generates synthetic data on high-dimensional data while providing both fairness convergence and strict DP guarantees – the first of its kind to our knowledge.

## 1 Introduction

Recently, generative models have shown remarkable performance in various applications including vision (Wang et al., ) and language tasks (Brown et al., ) – while also raising significant ethical concerns. In particular, privacy and fairness concerns have emerged due to generative models mimicking their training data. On the privacy side, specific training data can be memorized, allowing the reconstruction of sensitive information like facial images (Hilprecht et al., ; Sun et al., ). On the fairness side, any bias in the training data can be learned, resulting in biased synthetic data and unfair downstream performances across demographic groups (Zhao et al., ; Tan et al., ).

Although privacy and fairness are both essential for generative models, previous research has primarily tackled them separately. Differential Privacy (DP) techniques (Dwork et al., ), which provide rigorous privacy guarantees, have been developed for private generative models (Xie et al., ; Jordon et al., ); various fair training techniques, which remove data bias and generate more balanced synthetic data, have been proposed for fair generative models (Xu et al., ; Choi et al., ). To achieve both objectives, harnessing these techniques has emerged as a promising direction. For example, Xu et al. () combine a fair pre-processing technique (Celis et al., ) with a private generative model (Chanyaswad et al., ) to train both fair and private generative models.

Figure: Figure 1: Privacy-fairness conflict. Privacy techniques prefer the left-hand scenario to prevent privacy risk of a certain data sample, while fairness techniques prefer the right-hand scenario to balance learning w.r.t. groups. Related empirical results are shown in Table 5.2.
Refer to caption: x1.png

However, we contend that naïvely combining developed techniques for privacy and fairness can lead to a worse privacy-fairness-utility tradeoff, where utility is a model’s ability to generate realistic synthetic data. We first illustrate how privacy and fairness can conflict in Fig. 1. Given the data samples $M_{1}$, $M_{2}$, $M_{3}$, and $m_{1}$ where $M$ and $m$ denote the majority and minority data groups, respectively, DP and fairness techniques play a tug-of-war regarding the use of minority data point $m_{1}$; DP techniques limit its use to prevent privacy risks such as memorization, while fairness techniques increase its use to promote more balanced learning w.r.t. groups given the biased data. As a result, fairness techniques may undermine privacy by overusing $m_{1}$, while DP techniques may undermine fairness by limiting $m_{1}$’s usage. Moreover, combining different techniques can introduce new technical constraints, reducing the effectiveness of original methods. For instance, the fair preprocessing technique used by Xu et al. () hinders the utility of the DP generative model by requiring data binarization, which incurs significant information loss on high-dimensional data such as images – restricting their overall framework only applicable to low-dimensional structural data.

Figure: Figure 2: Overview of PFGuard. PFGuard integrates fairness and privacy in generative models through a two-stage process. In the fair training stage (blue), we train fair teacher models by sampling balanced mini-batches from biased data subsets. In the private training stage (red), we aggregate the teacher outputs to supervise the generator (e.g., whether the generated image given input noise $Z$ is realistic or unrealistic) with DP noise to ensure privacy. With these training stages to achieve fairness and privacy, PFGuard also achieves high utility based on ensemble learning of teacher models, resulting in high-quality, unbiased, and private synthetic data.
Refer to caption: x2.png

Therefore, we design a generative framework that simultaneously addresses fairness and privacy while achieving utility for high-dimensional synthetic data generation. To this end, we propose PFGuard: a generative framework with Privacy and Fairness Safeguards. As illustrated in Fig. 2, the key component is an ensemble of intermediate teacher models, which balances privacy-fairness conflicts between fair training and private training stages. In the fair training stage, we design a new sampling technique to train fair teachers, which provides a theoretical convergence guarantee to the fair generator. In the private training stage, we employ the Private Teacher Ensemble Learning (PTEL) approach (Papernot et al., ; ), which aggregates each teacher’s knowledge with random DP noise (e.g., noisy voting), to privatize the knowledge transfer to the generator. As a result, PFGuard provides a unified solution to train both fair and private generative models by transferring the teachers’ fair knowledge in a privacy-preserving manner.

Compared to simple sequential approaches, PFGuard is carefully designed to address privacy-fairness conflicts. Recall that fairness techniques can incur privacy breaches by overusing minority data; in contrast, PFGuard prevents privacy breaches by decoupling fairness and privacy with intermediate teacher models. Although fair sampling can still compromise privacy in teacher models by potentially overusing minority data, PFGuard ensures privacy in the generator – our target model – by training it solely with the privatized teacher output, as shown in Fig. 2. Also, recall that privacy techniques can lead to fairness cancellation by suppressing the use of minority data; in contrast, PFGuard avoids fairness cancellation through teacher-level privacy bounding using PTEL approaches. Compared to sample-level privacy bounding methods like gradient clipping (Abadi et al., ), teacher-level bounding leaves room for teachers to effectively learn balanced knowledge via fair training. As a result, PFGuard provides strict DP guarantees for the generator and better preserves fairness compared to the combination of fairness-only and privacy-only techniques – see more analyses in Sec. 3.

Moreover, PFGuard is compatible with a wide range of existing private generative models and preserves their utility. PTEL is widely adopted in private generative models as it provides prominent privacy-utility tradeoff (Jordon et al., ; Chen et al., ; Long et al., ; Wang et al., ). PFGuard can extend any of these models with a fair training stage as shown in Fig. 2, which requires a simple modification in the minibatch sampling process. Since additional fair sampling does not require additional training complexity compared to say adding a loss term for fairness, PFGuard preserves the privacy-utility tradeoff of PTEL as well while improving fairness. We also provide guidelines to control the fairness-privacy-utility tradeoff – see more details in Sec. 4.

Experiments show that PFGuard successfully generates high-dimensional synthetic data while ensuring both privacy and fairness; to our knowledge, PFGuard is the first framework that works on high-dimensional data including images. Our results also reveal two key findings: (1) existing private generative models can produce highly-biased synthetic data in real-world scenarios even with simple bias settings, and (2) a naïve combination of individual techniques may fail to achieve either privacy or fairness even with simple datasets – highlighting PFGuard’s effectiveness and the need for a better integration of fair and private generative models.

Summary of Contributions
1) We identify how privacy and fairness conflict with each other, which complicates the development of responsible generative models. 2) We propose PFGuard, which is to our knowledge the first generative framework that supports privacy and fairness for high-dimensional data. 3) Through extensive experiments, we show the value of integrated solutions to address the privacy-fairness-utility-tradeoff compared to simple combinations of individual techniques.

## 2 Preliminaries

#### Generative Models

We focus on Generative Adversarial Networks (Goodfellow et al., ), which are widely-used generative models that leverage adversarial training of two networks to generate realistic synthetic data: 1) a generator that learns the underlying training data distribution and generates new samples and 2) a discriminator that distinguishes between real and generated data. The discriminator can be considered as the teacher model of the generator, as the generator does not have access to the real data and only learns from the discriminator via the GAN loss function.

#### Differential Privacy

We use Differential Privacy (DP) (Dwork et al., ) to privatize generative models, a gold standard privacy notion that provides quantitative privacy analysis of an algorithm. DP measures how much an adversary can learn about a single sample by differentiating two outputs of an algorithm $\mathcal{M}$. This privacy guarantee is quantified with the parameters $(\varepsilon,\delta$) as follows:

###### Definition 2.1 .

($(\epsilon,\delta)$-Differential Privacy (Dwork et al., ))
A randomized mechanism $\mathcal{M}:\mathcal{D}\rightarrow\mathcal{R}$ with domain $\mathcal{D}$ and range $\mathcal{R}$ satisfies $(\epsilon,\delta)$-differential privacy if for any two adjacent inputs $D,D^{\prime}$, which differ by a single sample, and for any subset of outputs $O\subseteq\mathcal{R}$, the following holds:

$$ $\Pr(\mathcal{M}(D)\in O)\leq e^{\epsilon}\Pr(\mathcal{M}(D^{\prime})\in O)+\delta,$ $$

where $\epsilon$ is the upper bound of privacy loss, and $\delta$ is the probability of breaching DP constraints.

We can enforce DP in an algorithm in two steps (Dwork et al., ). First, we bound its “sensitivity” (Def. 2.2), which captures the maximum influence of an individual sample. Second, we add random noise that is proportional to the sensitivity. A common way to ensure DP is to use a Gaussian mechanism (Dwork et al., ) (Thm. 2.1), which utilizes Gaussian random noise with a scale proportional to $l_{2}$-sensitivity. Two datasets $D$, $D^{\prime}$ are adjacent if they differ by a single sample.

###### Definition 2.2 .

(Sensitivity (Dwork et al., )) The $l_{p}$-sensitivity for a $d$-dimensional function $f:\mathcal{D}{\rightarrow}\mathbb{R}^{d}$ is defined as $\Delta_{f}^{p}=\underset{D,D^{\prime}}{\max}\left\|f(D)-f(D^{\prime})\right\|_
{p}$ over all adjacent datasets $D,D^{\prime}$.

###### Theorem 2.1 .

(Gaussian mechanism (Dwork et al., ; Mironov, )) Let $f:\mathcal{X}\rightarrow\mathbb{R}^{d}$ be an arbitrary $d$-dimensional function with $l_{2}$-sensitivity $\Delta_{f}^{2}$. The Gaussian mechanism $\mathcal{M}_{\sigma}$, parameterized by $\sigma$, adds Gaussian noise into the output, i.e., $\mathcal{M}_{\sigma}({\bm{x}})=f({\bm{x}})+\mathcal{N}(0,\sigma^{2}\bm{I})$, and satisfies $(\varepsilon,\delta)$-DP for $\sigma\geq\sqrt{2\ln(1.25/\delta)}\Delta_{f}^{2}/\varepsilon$.

###### Definition 2.1 .

###### Definition 2.2 .

###### Theorem 2.1 .

#### Fairness

We consider a generative model to be fair if two criteria are satisfied: 1) the model generates similar amounts of data for different demographic groups with similar quality, and 2) the generated data can be used to train a fair downstream model w.r.t. traditional group fairness measures. For 1), we measure the size and image quality disparities between the groups using the Fréchet Inception Distance (FID) score (Heusel et al., ; Choi et al., ) to assess image quality. For 2), we use two prominent group fairness measures: equalized odds (Hardt et al., ) where the groups should have the same label-wise accuracies; and demographic parity (Feldman et al., ) where the groups should have similar positive prediction rates.

## 3 Challenges of Satisfying Both Privacy and Fairness

In this section, we examine the practical challenges of integrating privacy-only and fairness-only techniques to train both private and fair generative models. Based on Fig. 1’s intuition on how privacy and fairness conflict, we analyze how existing approaches for DP generative models and fair generative models can technically conflict with each other.

#### Adding Fairness Can Worsen Privacy

Ensuring fairness in DP generative models can significantly increase sensitivity (Def. 2.2), leading to invalid DP guarantees. Sensitivity, which measures a data sample’s maximum impact on an algorithm, is crucial in DP generative models because the noise amount required for DP guarantees is often proportional to this sensitivity value. However, integrating fairness techniques in DP generative models can invalidate their sensitivity analyses by adjusting model outputs for fairness purposes. Examples include amplifying the impact of certain data samples to balance model training across groups (Choi et al., ) and directly feeding data attributes such as class labels or sensitive attributes (e.g., race, gender) to a generator for more balanced synthetic data (Xu et al., ; Sattigeri et al., ; Yu et al., ), which can cause large fluctuations in the generator output with any variation in these data attributes. As a result, fairness techniques can end up increasing sensitivity and require more noise to maintain the same privacy level, compromising the original DP guarantees unless modifying DP techniques to add more noise. However, this modification is also not straightforward as assessing the increased sensitivity by fairness techniques can be challenging (Tran et al., ).

#### Adding Privacy Can Worsen the Fairness-Utility Tradeoff

Another direction is to ensure privacy in fair generative models, but configuring an appropriate privacy bound can be challenging, leading to unexpected fairness-utility tradeoffs. We illustrate below using standard DP and fairness techniques: DP-SGD (Abadi et al., ) and reweighting (Choi et al., ). Let $\bm{g}({\mathbf{x}})$ denote the gradient of the data sample ${\mathbf{x}}$, and $p_{\text{bal}}$ and $p_{\text{bias}}$ denote balanced and biased data distributions, respectively.

- •
DP-SGD is a standard DP technique Chen et al. () for converting non-DP algorithms to DP algorithms by modifying traditional stochastic gradient descent (SGD). Compared to SGD, DP-SGD 1) applies gradient clipping to limit the individual data point’s contribution, where $\bm{g}({\mathbf{x}})$ is clipped to $\bm{g}({\mathbf{x}})/\max(1,||\bm{g}({\mathbf{x}})||_{2}/C)$ (the sensitivity becomes the clipping threshold $C$), and 2) uses a Gaussian mechanism (Thm. 2.1) to add sufficient noise to ensure DP.
- •
Reweighting is a traditional fairness method (Horvitz & Thompson, ) widely used in generative modeling (Choi et al., ; Kim et al., ), which assigns greater weights to minority groups for a “balanced” loss during SGD. In particular, setting the sample weight to the likelihood ratio $h({\mathbf{x}}_{i}){=}p_{\text{bal}}({\mathbf{x}}_{i})/p_{\text{bias}}({
\mathbf{x}}_{i})$ produces an unbiased estimate of $\mathbb{E}_{{\mathbf{x}}\sim p_{\text{bal}}}[\bm{g}({\mathbf{x}})]$ as follows:
$\displaystyle\mathbb{E}_{{\mathbf{x}}\sim p_{\text{bias}}}[\bm{g}({\mathbf{x}}
)\cdot h({\mathbf{x}})]=\mathbb{E}_{{\mathbf{x}}\sim p_{\text{bias}}}\Bigl{[}
\bm{g}({\mathbf{x}})\frac{p_{\text{bal}}({\mathbf{x}})}{p_{\text{bias}}({
\mathbf{x}})}\Bigr{]}=\mathbb{E}_{{\mathbf{x}}\sim p_{\text{bal}}}[\bm{g}({
\mathbf{x}})].$
(1)

Figure: Figure 3: Fairness-utility tradeoff caused by DP-SGD when used on top of reweighting. Depending on the choice of $C$, DP-SGD may compromise utility (left) or fairness (right).
Refer to caption: x3.png

We can extend reweighting for fairness to also satisfy DP using DP-SGD, but finding the clipping threshold $C$ that balances fairness and utility can be challenging. Note that if we perform DP-SGD and then reweighting, privacy breach may occur by amplifying sample gradients beyond $C$, invalidating the sensitivity derived from gradient clipping. We thus consider performing reweighting and then DP-SGD, which at least guarantees DP for reweighting-based fair generative models. However, the clipping now undoes the fairness adjustments where reweighted gradients $\bm{g}({\mathbf{x}})\cdot h({\mathbf{x}})$ are clipped to $\bm{g}({\mathbf{x}})\cdot h({\mathbf{x}})/\max(1,||\bm{g}({\mathbf{x}})\cdot h
({\mathbf{x}})||_{2}/C)$, and Eq. 1 does not hold if $C\leq\bm{g}({\mathbf{x}})$. Here one solution is to use a larger $C$ such that $C\geq\bm{g}({\mathbf{x}})$. However, increasing $C$ also increases the noise required for DP, which reduces utility (Fig. 3). As a result, selecting a $C$ that balances fairness and utility may necessitate extensive hyperparameter tuning (Bu et al., ), complicating the systematic integration of fairness into DP generative models.

Overall, we show that a naïve combination of existing fairness-only and privacy-only techniques can be insufficient to achieve both objectives. While we have not exhaustively covered all possible combinations, one can see how privacy breaches and unexpected fairness-utility tradeoffs can easily occur without a careful design. To avoid these downsides, we emphasize the need for a unified design that integrates both privacy and fairness in generative models.

###### Remark 1 .

Our study is the first to reveal that fairness and privacy techniques can counteract each other. We also demonstrate how both can be compromised – see empirical results in Sec. 5.

###### Remark 2 .

We emphasize the need for a framework tailored to generative settings. There are notable fairness-privacy techniques for classification, but directly extending them to data generation can be challenging due to the fundamentally different goals of the two settings – see more details in Sec. A.

###### Remark 1 .

###### Remark 2 .

## 4 Framework

We now propose PFGuard, the first generative framework that simultaneously achieves statistical fairness and DP on high-dimensional data, such as images. As shown in Fig. 2, PFGuard balances privacy-fairness conflicts between fair and private training stages using an ensemble of teacher models as a key component. In Sec. 4.1, we first explain the fair training stage, which trains a fair teacher ensemble. In Sec. 4.2, we then explain the private training stage, which transfers the knowledge of this teacher ensemble to the generator with DP guarantees – ultimately training a generator that is both fair and private. In Sec. 4.3, we lastly discuss how PFGuard’s integrated design offers advantages in terms of fairness, utility, and privacy compared to the naïve approaches discussed in Sec. 3.

### 4.1 Fair Training with Balanced Minibatch Sampling

#### Intuition

We ensure fairness in the teachers by balancing the minibatches used for training. Here we assume a general training setup of stochastic gradient descent, where we iteratively pick a subset $B$ of the training data (i.e., minibatches) to update model parameters more efficiently. Since generative models then only learn the underlying data distribution through $B$, using balanced minibatches $B\sim p_{\text{bal}}$ will result in modeling $p_{\text{bal}}$ even if we have biased training data $D\sim p_{\text{bias}}$. While another approach is to debias the training data itself by acquiring more minority data, this approach is costly and often infeasible for private domains with limited publicly-available data (Jordon et al., ).

#### Theoretical Foundation

To develop a fair minibatch sampling technique with a convergence guarantee, we leverage Sampling-Importance Resampling (SIR) (Rubin, ; Smith & Gelfand, ) as the theoretical foundation. SIR is a statistical method for drawing random samples from a target distribution $\pi(x)$ by using a proposal distribution $\psi(x)$. SIR proceeds in two steps: 1) we draw a set of $n$ independent random samples $S_{1}=\{x_{i}\}_{i=1}^{n}$ from $\psi(x)$ and 2) we resample a smaller set of $m$ independent random samples $S_{2}=\{x_{i}\}_{i=1}^{m}$ from $S_{1}$. Here, the resampling probability $w(x_{i})$ is set proportional to $h(x_{i})$, where $h(x_{i})=\pi(x_{i})/\psi(x_{i})$. Then, the resulting samples in $S_{2}$ are approximately distributed according to $\pi(x)$ as follows:

$$ $\displaystyle\Pr(x\leq t)$ $\displaystyle=\scalebox{1.2}{$\sum_{i:x_{i}\leq t}$}w(x_{i})=\scalebox{1.2}{$ \sum_{i:x_{i}\leq t}$}\frac{h(x_{i})}{\sum_{i}h(x_{i})}=\frac{\sum_{i}\mathbbm {1}\{x_{i}\leq t\}\pi(x_{i})/\psi(x_{i})}{\sum_{i}\pi(x_{i})/\psi(x_{i})}$ (2) $\displaystyle\underset{n\rightarrow\infty}{\rightarrow}\frac{\int\mathbbm{1}\{ x\leq t\}\{\pi(x)/\psi(x)\}\psi(x)dx}{\int\{\pi(x)/\psi(x)\}\psi(x)dx}=\int \mathbbm{1}\{x\leq t\}\pi(x)dx$ (3) $$

where $\mathbbm{1}(\cdot)$ is the indicator function. The distribution becomes exact when $n\rightarrow\infty$.

#### Methodology

We now present our sampling technique, which guarantees $B\sim p_{\text{bal}}$ based on SIR. We first make the following reasonable assumptions: 1) each data sample has a uniquely defined sensitive attribute ${\mathbf{s}}\in\mathcal{S}$ (e.g., race); 2) $p_{\text{bal}}$ is uniformly distributed over ${\mathbf{s}}$; 3) following Choi et al. (), the same relevant input features are shared for each group ${\mathbf{s}}$ between the balanced and biased datasets (e.g., $p_{\text{bal}}({\mathbf{x}}|{\mathbf{s}}{=}s)=p_{\text{bias}}({\mathbf{x}}|{
\mathbf{s}}{=}s)$), and similarly between the training dataset $D$ and any subset $D_{i}$ (e.g., $p_{D}({\mathbf{x}}|{\mathbf{s}}{=}s)=p_{D_{i}}({\mathbf{x}}|{\mathbf{s}}{=}s)$). We now outline the technique step-by-step below.

- (1)
We set the target distribution to $p_{\text{bal}}({\mathbf{x}})$ and the proposal distribution to $p_{\text{bias}}({\mathbf{x}})$, as our goal is to sample a balanced minibatch $B\sim p_{\text{bal}}$ from the biased training dataset $D\sim p_{\text{bias}}$.
- (2)
We divide $D$ into $n_{T}$ disjoint subsets $\{D_{i}\}^{n_{T}}_{i=1}$ to train each teacher model $T_{i}$, such that each $D_{i}$ retains the same value distribution of ${\mathbf{s}}$ as $D$ using the ${\mathbf{s}}$ labels (i.e., $p_{D_{i}}({\mathbf{s}}=s)=p_{D}({\mathbf{s}}=s)$). Then, we can derive $D_{i}\sim p_{\text{bias}}$ using assumption 3) above as follows:
$\displaystyle p_{D_{i}}({\mathbf{x}})=\Sigma_{s}p_{D_{i}}({\mathbf{x}}|{
\mathbf{s}}{=}s)p_{D_{i}}({\mathbf{s}}{=}s)=\Sigma_{s}p_{D}({\mathbf{x}}|{
\mathbf{s}}{=}s)p_{D}({\mathbf{s}}{=}s)=p_{\text{bias}}({\mathbf{x}})$
(4)
- (3)
We sample $B$ from $D_{i}$ with a resampling probability $w({\mathbf{x}})$ that is proportional to $h({\mathbf{x}})=p_{\text{bal}}({\mathbf{x}})/p_{\text{bias}}({\mathbf{x}})$, which is computed as follows:
$\displaystyle h({\mathbf{x}})=\frac{p_{\text{bal}}({\mathbf{x}})}{p_{\text{
bias}}({\mathbf{x}})}=\frac{p_{\text{bal}}({\mathbf{x}}|{\mathbf{s}}{=}s)p_{
\text{bal}}({\mathbf{s}}{=}s)}{p_{\text{bias}}({\mathbf{x}}|{\mathbf{s}}{=}s)p
_{\text{bias}}({\mathbf{s}}{=}s)}=\frac{p_{\text{bal}}({\mathbf{s}}{=}s)}{p_{
\text{bias}}({\mathbf{s}}{=}s)}\simeq\frac{1/|\mathcal{S}|}{|\{{\mathbf{x}}{
\in}D|{\mathbf{s}}{=}s\}|/|D|}$
(5)
where the second and third equality follows from assumption 1) and 3) above, respectively, and the last approximation follows from assumption 2) above and $D\sim p_{\text{bias}}$.

A sample $B$ from the above procedure is approximately distributed according to $p_{\text{bal}}$ based on SIR; we sample $D_{i}$ from $p_{\text{bias}}$ and resample $B$ from $D_{i}$, where the resampling probability is proportional to $h({\mathbf{x}})=p_{\text{bal}}({\mathbf{x}})/p_{\text{bias}}({\mathbf{x}})$. Since a large number of minibatch samplings is needed to train generative models, the $B$ distribution eventually converges to $p_{\text{bal}}$, leading to a fair generative modeling of $p_{\text{bal}}$.

#### Extensions

Our fair sampling technique is also extensible to private settings where the label of sensitive attribute ${\mathbf{s}}$ is unavailable, for example due to privacy regulations (Jagielski et al., ; Mozannar et al., ; Tran et al., ). In such settings, we can employ a binary classification approach to estimate $h({\mathbf{x}})$ like Choi et al. (). While their work focuses on non-private settings and assumes an unbiased public reference data on the order of 10%–100% of $|D|$ for the estimation, this assumption can be unrealistic in private domains due to the lack of public data. Our empirical study in Sec. 5.3 shows that we can achieve fairness with only 1–10% of the data, leveraging the ensemble learning of multiple-teacher structure, which can further reduce the estimation error. Note that in this extension, the convergence guarantee may not hold, as $D_{i}\sim p_{\text{bias}}$ in step (2) might not be true in practice if the dataset is randomly partitioned without considering sensitive attribute labels.

### 4.2 Private Training with Private Teacher Ensemble Learning

#### Intuition

We ensure DP by privatizing knowledge transfer from a teacher ensemble to the generator. Although the sampling technique in Sec. 4.1 transfers more balanced knowledge, privacy risks can be also transferred due to privacy-fairness conflicts. For example, if certain data samples are resampled repeatedly during teacher training, privacy risks like memorization can occur in the teacher models and be transferred to the generator. We thus privatize the knowledge transfer with DP techniques to provide strict DP guarantees in the generator. Note that only the generator needs to have privacy as it is the one that is released publicly to produce synthetic data.

#### Privacy Guarantee

We utilize Private Teacher Ensemble Learning (PTEL) (Papernot et al., ; ) to ensure DP in the knowledge transfer. Compared to non-private ensemble learning, PTEL 1) assumes each teacher model is trained on a disjoint data subset and 2) adds noise proportional to the sensitivity of the knowledge aggregation. Here, sensitivity is derived from data disjointness, where a single data point affects at most one teacher. For example, GNMax aggregator (Papernot et al., ) aggregates prediction of teacher models $\{T_{i}\}^{n_{T}}_{i=1}$ on a data sample ${\mathbf{x}}$ for its class label as follows:

$$ $\displaystyle\text{GNMax}({\mathbf{x}})=\operatorname*{arg\,max}_{j}\{n_{j}({ \mathbf{x}})+\mathcal{N}(0,\sigma^{2})\}\quad\text{for}~{}j=1,...,c$ (6) $$

where $n_{j}({\mathbf{x}})$ denotes the vote count for the $j$-th class (i.e., $n_{j}({\mathbf{x}})=|\{i:T_{i}({\mathbf{x}}){=}j\}|$), and $\mathcal{N}(0,\sigma^{2})$ denotes random Gaussian noise. Here, the $l_{2}$-sensitivity (Def. 2.2) is $\sqrt{2}$, as a single data point affects at most one teacher, increasing the vote counts by 1 for one class and decreasing the count by 1 for another class (see a more detailed analysis in Sec. B.2). Consequently, the GNMax aggregator satisfies $(\varepsilon,\delta)$-DP for $\sigma\geq\sqrt{8\ln(1.25/\delta)}/\varepsilon$ based on the Gaussian mechanism (Thm. 2.1).

#### Methodology

PFGuard can be easily integrated with existing PTEL-based generative models by simply modifying the minibatch sampling process as described in Sec. 4.1. PTEL has been widely adopted to train generators with privatized teacher output to ensure DP (Jordon et al., ; Chen et al., ; Long et al., ; Wang et al., ). Although the exact sensitivity values of these PTEL-based generative models vary depending on what teacher knowledge is aggregated (e.g., votes on class labels (Jordon et al., ) or gradient directions (Wang et al., )), PFGuard preserves any sensitivity as long as the PTEL enforce data disjointness; even with fair sampling, a single data point still affects only one teacher. PFGuard is thus compatible with various PTEL-based generative models, enhancing fairness while preserving DP guarantees – see Sec. C.1 for the full algorithm.

#### Number of Teachers

We provide guidelines on how to set the number of teachers $n_{T}$ for PFGuard, which affects the privacy-fairness-utility tradeoff. While $n_{T}$ is typically tuned through experiments in PTEL approaches (Long et al., ; Wang et al., ), we need a different way to set $n_{T}$ to additionally consider fairness. Note that a large $n_{T}$ would result in a diverse ensemble that can generalize better, but also lead to a teacher receiving a data subset that is too small for training. We thus suggest $n_{T}$ to be at most $\lfloor|D|\min_{s\in\mathcal{S}}p_{\text{bias}}(s)\rfloor$ where $\lfloor\cdot\rfloor$ denotes the floor function. This mathematical upper bound guarantees that each teacher probabilistically gets at least one sample of the smallest minority data group. In Sec. 5.3, we demonstrate how this bound helps avoid compromising fairness. We also discuss how to set $n_{T}$ when sensitive attribute labels are unavailable in Sec. C.2.

### 4.3 Advantages of Integrated Design

We discuss how PFGuard overcomes the challenges of naïve approaches discussed in Sec. 3.

#### Balances Privacy-Fairness Conflict

PFGuard can sidestep privacy breaches and fairness cancellation arising from privacy-fairness conflicts. Applying fairness-only techniques to existing DP generators can compromise DP guarantees and require complex sensitivity assessments; in contrast, PFGuard automatically preserves DP guarantees of any PTEL-based DP generators through data disjointness, eliminating the need of such assessments. Privacy-only techniques often use sample-level bounding, which directly limits an individual sample’s influence (e.g., gradient clipping discussed in Sec. 3) and can lead to fairness cancellation by suppressing the use of minority data. In contrast, PFGuard uses indirect privacy bounding, in the sense that we limit the knowledge transfer of teacher models in order to limit individual sample’s influence. Since there are no DP constraints during the teacher learning, the teacher models can effectively learn balanced knowledge across data groups.

#### Achieves Better Fairness-Utility Tradeoff

The fair training of PFGuard adds minimal training complexity, preserving the utility for the subsequent private training stage. The proposed sampling technique requires a simple modification in the minibatch sampling process for fairness, avoiding the need for additional fairness loss terms (Sattigeri et al., ; Yu et al., ) or auxiliary classifiers (Tan et al., ; Um & Suh, ) typically employed in fairness-only techniques. In Sec. 5, we also show that PFGuard incurs negligible overhead in computation time when integrated with existing PTEL-based generative models.

## 5 Experiments

We perform experiments to evaluate PFGuard’s effectiveness in terms of fairness, privacy, and utility.

Datasets     We evaluate PFGuard on three image datasets: 1) MNIST (LeCun et al., ) and FashionMNIST (Xiao et al., ) for various analyses and baseline comparisons, and 2) CelebA (Liu et al., ) to observe performance in real-world scenarios more closely related to privacy and fairness concerns. Here, MNIST contains handwritten digit images, FashionMNIST contains clothing item images, and CelebA contains facial images. While MNIST and FashionMNIST are simplistic and less reflective of real-world biases, they enable reliable fairness analyses on top of high-performing DP generative models on these datasets, making them widely adopted in recent studies addressing the privacy-fairness intersections (Bagdasaryan et al., ; Farrand et al., ; Ganev et al., ). For CelebA, we resize the images to 32 $\times$ 32 $\times$ 3 (i.e., CelebA(S)) and to 64 $\times$ 64 $\times$ 3 (i.e., CelebA(L)) following the conventions in the DP generative model literature (Long et al., ; Wang et al., ; Cao et al., ). More dataset details are in Sec. D.1.

Bias Settings     We create various bias settings across classes and subgroups, focusing on four scenarios: 1) binary class bias, which is a basic scenario often addressed in DP generative models, and 2) multi-class bias, subgroup bias, and unknown subgroup bias, which are more challenging scenarios typically addressed in fairness techniques, but not in DP generative models. We observe that DP generative models mostly perform poorly in these challenging scenarios, especially with complex datasets like CelebA, so we use MNIST for more reliable analyses. While recent privacy-fairness studies on MNIST mostly focus on class bias (Bagdasaryan et al., ; Farrand et al., ), we additionally analyze subgroup bias using image rotation for more fine-grained fairness analyses and to support prominent fairness metrics like equalized odds (Hardt et al., ). In all experiments, we denote $Y=0$ as the minority class and $S=0$ as the minority group. More details on bias levels (e.g., size ratios between majority and minority data) and bias creation are in Sec. D.1.

Metrics
We evaluate utility, privacy, and fairness in both synthetic data and downstream tasks.

- •
Utility. We measure the overall and groupwise Frechet Inception Distance (FID) (Heusel et al., ) to evaluate the sample quality of synthetic data. We evaluate model accuracy in downstream tasks by training Multi-layer Perceptrons (MLP) and Convolutional Neural Networks (CNN) on synthetic data and testing on real datasets (Chen et al., ).
- •
Fairness. We measure the group size disparity in synthetic data with the KL divergence to uniform distribution $U(S)$ (i.e., $D_{KL}(p_{G}(S)||U(S))$) (Yu et al., ) and the distribution disparity (i.e., $|p_{G}(S)-U(S)|$) (Choi et al., ), where $p_{G}(S)$ denotes generated distribution w.r.t. $S$. We measure the fairness disparities in downstream tasks as follows: equalized odds disparity (i.e., $\max_{y,s_{1},s_{2}}|\Pr(\hat{Y}{=}y|Y{=}y,S{=}s_{1}){-}\Pr(\hat{Y}{=}y|Y{=}y,
S{=}s_{2})|,~{}\forall y\in\mathcal{Y}$, $s_{1},s_{2}{\in}\mathcal{S}$), demographic disparity (i.e., $\max_{s_{1},s_{2}}|\Pr(\hat{Y}{=}1|S{=}s_{1}){-}\Pr(\hat{Y}{=}1|S{=}s_{2})|,~{
}\forall s_{1},s_{2}\in\mathcal{S}$), and accuracy disparity (i.e., $\max_{y_{1},y_{2}}|\Pr(\hat{Y}{=}y_{1}|Y{=}y_{1}){-}\Pr(\hat{Y}{=}y_{2}|Y{=}y_
{2})|,~{}\forall y_{1},y_{2}\in\mathcal{Y}$).
- •
Privacy. We use privacy budget $\varepsilon$ for DP (Def. 2.1), which is preserved in both synthetic data and downstream tasks due to the post-processing property of DP (see more details in Sec. B.1).

Baselines
We compare PFGuard with three types of baselines: 1) privacy-only and fairness-only approaches for data generation, 2) simple combinations of these methods, and 3) recent privacy-fairness classification methods applicable to data generation. For 1) and 2), we use three state-of-the-art DP generative models – GS-WGAN (Chen et al., ), G-PATE (Long et al., ) and DataLens (Wang et al., ) – and a widely-adopted fair reweighting method (Choi et al., ). For 3), we extend DP-SGD-F (Xu et al., ) and DPSGD-Global-Adapt (Esipova et al., ), which are fair variants of DP-SGD (Abadi et al., ). Specifically, we replace the DP-SGD used in GS-WGAN with these fairness-enhanced variants. We faithfully implement all baseline methods with their official codes and reported hyperparameters. More details on baseline methods are in Sec. D.2.

### 5.1 Improving Existing Privacy-Only Generative Models

We evaluate how PFGuard enhances the performance of existing DP generative models. As PFGuard guarantees the same level of DP, we focus on the fairness and utility performances while fixing the privacy budget to $\varepsilon{=}10$, which is one of the most conventional values (Ghalebikesabi et al., ).

#### Analysis on Synthetic Data

Table 1 shows the fairness and utility performances on synthetic data. Private generative models generally produce synthetic data with better overall image quality, but exhibit high group size disparity. In contrast, PFGuard significantly improves fairness by balancing group size and groupwise image quality, with a slight decrease in overall image quality.

**Table 1: Fairness and utility performances of private generative models with and without PFGuard on synthetic data, evaluated on MNIST with subgroup bias under $\varepsilon{=}10$. Blue and red arrows indicate positive and negative changes, respectively. Lower values are better across all metrics.**
|  | Fairness | Utility |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | KL ($\downarrow$) | Dist. Disp. ($\downarrow$) | FID ($\downarrow$) | Y=1, S=1 | Y=1, S=0 | Y=0, S=1 | Y=0, S=0 |
| GS-WGAN | 0.177$\pm$0.103 | 0.383$\pm$0.097 | 77.97$\pm$2.25 | 95.58$\pm$3.35 | 155.20$\pm$16.25 | 89.66$\pm$0.79 | 101.39$\pm$7.09 |
| G-PATE | 0.305$\pm$0.011 | 0.522$\pm$0.008 | 176.03$\pm$3.03 | 182.50$\pm$1.27 | 183.31$\pm$2.99 | 178.89$\pm$4.13 | 187.37$\pm$3.51 |
| DataLens | 0.220$\pm$0.030 | 0.450$\pm$0.028 | 192.29$\pm$3.67 | 197.13$\pm$6.18 | 197.99$\pm$6.01 | 202.86$\pm$4.12 | 207.12$\pm$12.75 |
| GS-WGAN + PFGuard | 0.067$\pm$0.036 ($\downarrow$) | 0.242$\pm$0.080 ($\downarrow$) | 83.67 $\pm$6.98 ($\uparrow$) | 114.54$\pm$27.74 | 149.47$\pm$17.31 | 79.94$\pm$7.08 | 72.44$\pm$7.96 |
| G-PATE + PFGuard | 0.206 $\pm$0.062 ($\downarrow$) | 0.431 $\pm$0.066 ($\downarrow$) | 166.89 $\pm$21.61 ($\downarrow$) | 173.48$\pm$19.93 | 173.79$\pm$19.43 | 174.98$\pm$24.06 | 185.92$\pm$19.89 |
| DataLens + PFGuard | 0.161 $\pm$0.019 ($\downarrow$) | 0.389 $\pm$0.022 ($\downarrow$) | 200.23 $\pm$3.11 ($\uparrow$) | 209.74$\pm$1.70 | 208.80$\pm$0.39 | 207.03$\pm$4.67 | 207.05$\pm$3.17 |

#### Analysis on Downstream Tasks

Table 2 shows the fairness and utility performances on downstream tasks. Compared to the synthetic data analysis, PFGuard enhances not only fairness, but also overall utility, especially for CNN models. We suspect that the increased overall utility results from the improved fairness in the input synthetic data, promoting more balanced learning among groups.

**Table 2: Fairness and utility performances of private generative models with and without PFGuard on downstream tasks, evaluated on MNIST with subgroup bias under $\varepsilon{=}10$. Blue and red arrows indicate positive and negative changes, respectively.**
|  | MLP | CNN |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Fairness | Utility | Fairness | Utility |  |  |
| Method | EO Disp. ($\downarrow$) | Dem. Disp. ($\downarrow$) | Acc ($\uparrow$) | EO Disp. ($\downarrow$) | Dem. Disp. ($\downarrow$) | Acc ($\uparrow$) |
| GS-WGAN | 0.153$\pm$0.030 | 0.061$\pm$0.012 | 0.910$\pm$0.007 | 0.172$\pm$0.045 | 0.069$\pm$0.014 | 0.927$\pm$0.008 |
| G-PATE | 0.166$\pm$0.082 | 0.063$\pm$0.053 | 0.896$\pm$0.005 | 0.256$\pm$0.046 | 0.111$\pm$0.001 | 0.888$\pm$0.015 |
| DataLens | 0.226$\pm$0.062 | 0.112$\pm$0.035 | 0.867$\pm$0.028 | 0.238$\pm$0.044 | 0.110$\pm$0.023 | 0.893$\pm$0.022 |
| GS-WGAN + PFGuard | 0.067$\pm$0.029 ($\downarrow$) | 0.044$\pm$0.012 ($\downarrow$) | 0.900 $\pm$0.003 ($\downarrow$) | 0.063$\pm$0.059 ($\downarrow$) | 0.035$\pm$0.037 ($\downarrow$) | 0.927$\pm$0.009 (–) |
| G-PATE + PFGuard | 0.085 $\pm$0.052 ($\downarrow$) | 0.044$\pm$0.033 ($\downarrow$) | 0.906$\pm$0.008 ($\uparrow$) | 0.084 $\pm$0.036 ($\downarrow$) | 0.044 $\pm$0.011 ($\downarrow$) | 0.898$\pm$0.023 ($\uparrow$) |
| DataLens + PFGuard | 0.169$\pm$0.081 ($\downarrow$) | 0.106$\pm$0.043 ($\downarrow$) | 0.859$\pm$0.056 ($\downarrow$) | 0.141$\pm$0.050 ($\downarrow$) | 0.078$\pm$0.051 ($\downarrow$) | 0.898$\pm$0.020 ($\uparrow$) |

### 5.2 Privacy-Fairness-Utility Tradeoff

We compare our privacy-fairness-utility performance with naïve combinations of prior approaches. We evaluate performance under two bias settings: 1) subgroup bias and 2) unknown subgroup bias. Table 5.2 shows the results, which aligns with our privacy-fairness counteraction analysis in Sec. 3. On the one hand, fairness-only reweighting approaches compromise privacy due to the increased iterations from modifying the loss function for fair training (i.e., the more a model uses the data, the weaker privacy it provides). On the other hand, privacy-fairness classification techniques maintain the original privacy guarantees, but significantly degrade utility and fairness, resulting in lower image quality and size disparities across groups. We further discuss this fairness-utility degradation in Sec. A, where we suspect that directly changing gradient clipping thresholds for fairness may severely affect utility when used in generative settings. In contrast, PFGuard is the only method that successfully achieves both privacy and fairness and preserves the closest utility to the original models.

**Table 3: Comparison of privacy-fairness-utility performance on MNIST under $\varepsilon{=}10$, using GS-WGAN as the base DP generator (see Sec. D.2 for more details). The first three rows represent upper bound performances for vanilla, DP-only, and fair-only models. Evaluations cover both subgroup bias and unknown subgroup bias, where “no S” indicates whether the method is applicable without group labels. “perc” denotes the proportion of public data used compared to the training data size. “-” indicates no samples are generated. Lower values are better across all metrics, and we boldface the best results in each subgroup bias and unknown subgroup bias settings.**
|  | Privacy | Fairness | Utility |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $\varepsilon~{}(\downarrow)$ | KL ($\downarrow$) | Dist. Disp. ($\downarrow$) | no S | FID ($\downarrow$) | Y=1, S=1 | Y=1, S=0 | Y=0, S=1 | Y=0, S=0 |
| Vanila | ✗ | 0.229 | 0.459 | ✗ | 31.95 | 28.01 | 55.53 | 44.04 | 63.50 |
| DP-only | 10 | 0.177 | 0.383 | ✗ | 77.97 | 95.58 | 155.20 | 89.66 | 101.39 |
| Fair-only | ✗ | 0.021 | 0.117 | ✓ | 38.62 | 50.78 | 52.69 | 75.46 | 53.86 |
| Reweighting | 13 | 0.009 | 0.044 | ✗ | 106.94 | 139.28 | 178.18 | 128.08 | 110.54 |
| DP-SGD $\rightarrow$ DP-SGD-F | 11 | 0.659 | 0.494 | ✗ | 90.20 | 121.78 | - | 73.07 | 159.83 |
| \cdashline1-10<br>PFGuard | 10 | 0.067 | 0.242 | ✗ | 83.67 | 114.54 | 149.47 | 79.94 | 72.24 |
| Reweighting (perc=1.0) | 13 | 0.025 | 0.148 | ✓ | 98.57 | 144.55 | 182.29 | 96.05 | 99.59 |
| Reweighting (perc=0.1) | 13 | 0.013 | 0.113 | ✓ | 106.94 | 139.28 | 178.18 | 128.08 | 110.54 |
| DP-SGD $\rightarrow$ DPSGD-GA | 10 | 0.693 | 0.707 | ✓ | 127.02 | 167.65 | - | 126.15 | - |
| \cdashline1-10<br>PFGuard (perc=0.1) | 10 | 0.004 | 0.041 | ✓ | 89.43 | 130.36 | 157.80 | 78.75 | 89.76 |

### 5.3 Ablation Study

#### Fairness Upper Bound on Number of Teachers

We validate the proposed theoretical upper bound on the number of teachers for fairness, which depends on the bias level of the training data. To effectively simulate scenarios where a teacher receives only a small subset of minority data, we evaluate PFGuard in a multi-class bias setting, downsizing the minority class (i.e., class 8 for MNIST) by a factor of $\gamma$. Given that MNIST has fewer than 6,000 samples for class 8, our proposed upper bound is $\gamma{\leq}5$ if we fix the number of teachers to 1,000. Fig. 5 shows that exceeding $\gamma{=}5$ leads to a noticeable decline in accuracy for the minority class, which is consistent with our theoretical results. It is noteworthy that even with the decline, PFGuard shows higher accuracy than the privacy-only baseline, which shows a consistent decrease in accuracy for the minority as $\gamma$ increases.

#### Impact of Reference Dataset Size

We explore the influence of the reference dataset size when PFGuard is extended to unknown subgroup bias setting. Fig. 5 shows PFGuard achieves comparable fairness even with a small reference dataset size, while showing a slight decline in the overall utility.

More Analyses We provide more experiments in Sec. E, including a comparison of computation time (Sec. E.1), results on different datasets such as FashionMNIST (Sec. E.2), and employing an additional normalization technique to further enhance the overall image quality (Sec. E.3).

### 5.4 Analysis with Stronger Privacy, High-Dimensional Images

We provide preliminary results with CelebA dataset, which mirrors real-world scenarios with high-dimensional facial images. As our study is the first to address both privacy and fairness in image data, this exploration is crucial for understanding the challenges in real-world settings. To reflect the need of stronger privacy protection in practical applications, we limit the privacy budget to $\varepsilon{=}1$.

Table 4 shows the fairness and utility performances under these challenging conditions. We observe that DP generative models often exhibit extreme accuracy disparities even with a simplistic class bias setting, achieving over 90% accuracy for the majority class while achieving accuracy below 25% for the minority class. PFGuard consistently enhances the minority class performance and reduces accuracy disparity, while there is still room for improvements. Our results underscore the importance of tackling both privacy and fairness in future studies, encouraging more research in this critical area.

**Table 4: Fairness and utility performances of private generative models with and without PFGuard on downstream tasks, evaluated on CelebA with binary class bias under $\varepsilon{=}1$. GS-WGAN is excluded due to lower image quality in this setting. Blue and red arrows indicate positive and negative changes, respectively. We provide the full table with standard deviations in Sec. E.4.**
|  | CelebA(S) | CelebA(L) |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Fairness | Utility | Fairness | Utility |  |  |  |  |
| Method | Acc. Disp. ($\downarrow$) | Acc ($\uparrow$) | Y=0 | Y=1 | Acc. Disp. ($\downarrow$) | Acc ($\uparrow$) | Y=0 | Y=1 |
| G-PATE | 0.978 | 0.666 | 0.014 | 0.992 | 0.968 | 0.668 | 0.023 | 0.991 |
| DataLens | 0.793 | 0.643 | 0.114 | 0.907 | 0.678 | 0.686 | 0.234 | 0.912 |
| G-PATE + PFGuard | 0.736 ($\downarrow$) | 0.678 ($\uparrow$) | 0.187 ($\uparrow$) | 0.923 ($\downarrow$) | 0.277 ($\downarrow$) | 0.563 ($\downarrow$) | 0.378 ($\uparrow$) | 0.655 ($\downarrow$) |
| DataLens + PFGuard | 0.725 ($\downarrow$) | 0.689 ($\uparrow$) | 0.205 ($\uparrow$) | 0.931 ($\uparrow$) | 0.641 ($\downarrow$) | 0.704 ($\uparrow$) | 0.276 ($\uparrow$) | 0.917 ($\uparrow$) |

## 6 Related Work

We cover the private and fair data generation literature here and cover the 1) private-only data generation, 2) fair-only data generation, 3) privacy-fairness intersection literature in Sec. F. Compared to these lines of works, only a few works focus on private and fair data generation (Xu et al., ; Pujol et al., ).
First, (Xu et al., ) proposes a two-step approach that removes bias from the training data via a fair pre-processing technique (Celis et al., ) and learns a DP generative model (Chanyaswad et al., ) from the debiased data. However, this framework is limited to low-dimensional structural data due to data binarization step in pre-precessing stage, which can incur significant information loss in high-dimensional image data. PFGuard, on the other hand, can generate high-dimensional image data with high quality. Second, (Pujol et al., ) proposes private data generation techniques satisfying causality-based fairness (Salimi et al., ), which consider the causal relationship between attributes. In comparison, PFGuard focuses on statistical fairness to achieve similar model performances for sensitive groups (Barocas et al., ). While causality-based approaches can better reveal the causes of discrimination than statistical approaches, modeling an underlying causal mechanism for real-world scenarios is also known to be challenging.

## 7 Conclusion

We proposed PFGuard, a fair and private generative model training framework. We first identified the counteractive nature between privacy preservation and fair training, demonstrating potential adverse effects – such as privacy breaches or fairness cancellation – when two objectives are addressed independently. We then designed PFGuard, which prevents the counteractions by using multiple teachers to harmonize fair sampling and private teacher ensemble learning. We showed how this integrated design of PFGuard offers multiple advantages, including a better fairness-privacy-utility tradeoff compared to other baselines, ease of deployment, and support for high-dimensional data.

#### Ethics Statement & Limitation

We believe our research addresses the critical issue of Trustworthy AI. Our focus on privacy and fairness underscores the need to design AI models that simultaneously safeguard individual privacy and mitigate biases without perpetuating them. In addition, our research and experiments are conducted with a strong commitment to ethical standards. All datasets used in this study, including publicly available human images, are widely used within the research community and do not contain sensitive or harmful content. Finally, we do note that choosing the right privacy and fairness measures for an application can be challenging and also depends on the social context. We also note that the use of multiple teacher does increase the cost of training, but provides more benefits particularly in balancing privacy and fairness.

#### Reproducibility Statement

All datasets, methodologies, and experimental setups utilized in our study are described in detail in the supplementary material. More specifically, we provide a description of the proposed algorithm in Sec. C.1, details of datasets and preprocessing in Sec. D.1, and implementation details including hyperparameters in Sec. D.2 to ensure reproducibility.

## Appendix A Challenges of Extending Classification Techniques

Continuing from Sec. 3, we provide more details of potential challenges when one tries to extend fairness-privacy classification techniques (Jagielski et al., ; Mozannar et al., ; Tran et al., ; ; Lowy et al., ) to generative settings due to the fundamentally different goals.

#### Different DP Notions and Assumptions

Classification and generation settings often concentrate on different DP notions or rely on different assumptions, which hinders simple extensions of techniques between them. In the classification setting, Differential Privacy w.r.t. sensitive attribute (Jagielski et al., ) is commonly addressed (Jagielski et al., ; Mozannar et al., ; Tran et al., ; ; Lowy et al., ), which considers the demographic group attribute as the only private information. This DP notion requires less DP noise compared to a more general notion of DP (Def. 2.1), which protects all input features, and enables a better privacy-utility tradeoff for DP classifiers. However, in the generative setting, a general notion of DP is mostly addressed, as presumed non-private features may in fact encode private information (e.g., pixel values in a facial image). Therefore, simply extending classification techniques to generative settings can be challenging, as it necessitates rigorous mathematical proofs for corresponding DP notions and may add a large DP noise when adapting to a general DP notion. Moreover, classification techniques can rely on convex objective functions (Tran et al., ), but the assumption of convexity does not usually hold in generative models (Goodfellow et al., ).

#### Challenges of Adjusting Privacy Bound

While recent studies have proposed fair variants of gradient sanitization (Xu et al., ; Esipova et al., ), directly adopting them in existing private generative models can undermine the original utility and privacy guarantee. To prevent aggressive gradient clipping in minority data groups, approaches to tune clipping threshold $C$ during training have been proposed, such as dynamically adjusting $C$ during training (Esipova et al., ) or utilizing different $C$ values w.r.t. groups (Xu et al., ). However, these adjustments of $C$ not only consume additional privacy budget, but also can significantly affect model utility, as private generative models often demonstrates high sensitivity in model convergence to these clipping values (Chen et al., ; Wang et al., ; Dockhorn et al., ). Hence, given the limited privacy budget and the necessity to carefully set the value of $C$, these approaches of tuning $C$ may drastically change the original privacy-utility tradeoff of existing models to achieve fairness.

## Appendix B Differential Privacy

Continuing from Sec. 2 and Sec. 4.2, we provide more details on differential privcy (DP).

### B.1 Post-processing Property of Differential Privacy

Continuing from Sec. 2, we detail the post-processing property of DP. Let $G$ be an $(\varepsilon,\delta)$-DP generator, which produces the synthetic data from input random noise ${\mathbf{z}}\in\mathcal{Z}$. Then, the synthetic dataset $\tilde{D}=G({\mathbf{z}})$ is also satisfies $(\varepsilon,\delta)$-DP due to the post-processing theorem, as the random noise ${\mathbf{z}}$ is independent of the private dataset $D$, which is used to train the DP generator.

###### Theorem B.1 .

(Post-processing (Dwork et al., )) Let $\mathcal{M}:\mathcal{D}\rightarrow\mathcal{R}_{1}$ be a randomized mechanism that is $(\varepsilon,\delta)$-DP. Let $f:\mathcal{R}_{1}\rightarrow\mathcal{R}_{2}$ be an arbitrary function. Then $f\circ\mathcal{M}:\mathcal{D}\rightarrow\mathcal{R}_{2}$ is $(\varepsilon,\delta)$-DP.

###### Theorem B.1 .

### B.2 Sensitivity Analysis of GNMax Aggregator

Continuing from Sec. 4.2, we echo the sensitivity analysis of GNMax aggregator provided by Papernot et al. () for readers’ convenience.

Given $\{T_{i}\}_{i=1}^{n_{T}}$ teachers, $c$ possible label classes, and an input data sample ${\mathbf{x}}$, the teachers’ vote count for the $j$-th class to an input ${\mathbf{x}}$ is denoted as:

$$ $n_{j}({\mathbf{x}})=|\{i:T_{i}({\mathbf{x}})=j\}|\quad\text{for }j=1,...,c$ (7) $$

where $T_{i}$ denotes the $i$-th teacher model. The vote count for each class is aggregated as follows:

$$ $\mathbf{n}({\mathbf{x}})=(n_{1},\dots,n_{c})\in\mathbb{N}^{c}$ (8) $$

Since a single training data point only affects at most one teacher due to data disjointness, changing one data sample will at most change the votes by 1 for two classes, where we denote here as classes $i$ and $j$. Given the two adjacent datasets $\mathcal{D},\mathcal{D}^{\prime}$ which differ by a single data point, let the aggregated vote counts are $\mathbf{n}=(n_{1},\dots,n_{c})$ and $\mathbf{n}^{\prime}=(n_{1}^{\prime},\dots,n_{c}^{\prime})$, respectively. The $l_{2}$-sensitivity (Def. 2.2) can be derived as follows:

$$ $\displaystyle\Delta^{2}$ $\displaystyle=\max_{D,D^{\prime}}\|(n_{1},\dots,n_{c})-(n_{1}^{\prime},\dots,n _{c}^{\prime})\|_{2}$ (9) $\displaystyle=\max_{n_{i},n_{i}^{\prime},n_{j},n_{j}^{\prime}}\|(0,\dots,0,n_{ i}-n_{i}^{\prime},0,\dots,0,n_{j}-n_{j}^{\prime},0,\dots,0)\|_{2}$ (10) $\displaystyle=\max_{n_{i},n_{i}^{\prime},n_{j},n_{j}^{\prime}}\sqrt{(n_{i}-n_{ i}^{\prime})^{2}+(n_{j}-n_{j}^{\prime})^{2}}\leq\sqrt{2}$ (11) $$

## Appendix C PFGuard Framework

Continuing from Sec. 4.2, we provide more details on the PFGuard framework.

### C.1 Training Algorithm

Continuing from Sec. 4.2, we provide the pseudocode to describe the full training algorithm when PFGuard is integrated on top of a PTEL-based generative model.

Figure: Algorithm 1 Integrating PFGuard with PTEL-based generative models

Output Differentially private generator $G$ with parameters $\theta_{G}$, total privacy cost $\varepsilon$

Notably, PFGuard requires to only modify the minibatch sampling process for training the teacher models (e.g., line 5 in the above pseudocode) to enable fair training within the training algorithm of PTEL-based generative models. PFGuard also preserves their privacy analyses (e.g., privacy cost $\varepsilon$ of each training epoch), as long as these models rely on data disjointness to derive sensitivity.

### C.2 Setting Number of Teachers without Sensitive Attribute Labels

Continuing from Sec. 4.2, we discuss how to extend the proposed upper bound on the number of teachers (i.e., $\lfloor|D|\min_{s\in\mathcal{S}}p_{\text{bias}}(s)\rfloor$) in settings where the label of sensitive attribute ${\mathbf{s}}$ is unavailable. The proposed upper bound does not rely on full knowledge of $p_{\text{bias}}$, but the distribution w.r.t. sensitive attributes. Thus, given the training data, we can estimate the subgroup distributions using traditional techniques like K-means clustering (Macqueen, ) or random subset labeling (Forestier & Wemmert, ). We note that these estimations can be effective, but may introduce some errors or additional overhead, such as increased computational time.

## Appendix D Experimental Settings

Continuing from Sec. 5, we provide more details on experiment settings. In all experiments, we use PyTorch and perform experiments using NVIDIA Quadro RTX 8000 GPUs. Also, we repeat all experiments 10 times and report the mean and standard deviation of the top 3 results. The reason we report the top-3 results is to favor the simple privacy-fairness baselines (e.g., “Reweighting” in Table 5.2), which tend to fail frequently. We compare their best performances with PFGuard.

### D.1 Datasets and Bias Settings

Continuing from Sec. 5, we provide more details on datasets. We use three datasets: MNIST (LeCun et al., ), FashionMNIST (Xiao et al., ), and CelebA (Liu et al., ). MNIST and FashionMNIST contain grayscale images with 28 x 28 pixels and 10 classes. Both datasets have 60,000 training examples and 10,000 testing examples. CelebA contains 202,599 celebrity face images. We use the official preprocessed version with face alignment and follow the official training and testing partition (Liu et al., ). Note that we are using image datasets instead of the traditional smaller tabular benchmarks for fairness because our goal is to make PFGuard work on higher dimensional data such as images.

#### MNIST & FashionMNIST

We create four bias scenarios across classes and subgroups as follows.

- •
Binary Class Bias. For MNIST, we set digit “3” as the majority class $Y=1$ and “1” as the minority class $Y=0$; for FasionMNIST, we set “Sneakers” as $Y=1$ and “Trousers” as $Y=0$. For each class pair, we select two classes that share the fewest false negatives and thus can be considered independent, following the convention of prior approaches (Bagdasaryan et al., ; Farrand et al., ; Ganev et al., ). We set bias level as 2, meaning the minority class $Y=0$ is 2 times smaller than the majority class $Y=1$. After creating bias, we apply random affine transformations to augment the datasets, ensuring they match the original dataset size.
- •
Multi-class Bias.
We set “8” as the minority class $Y=0$, reducing its size while maintaining the size of other 9 classes, following the above prior approaches. We vary the bias level from 1 to 10.
- •
Subgroup Bias. For both MNIST and FashionMNIST datasets, we use image rotation to define subgroups. We set non-rotated images as the majority group $S=1$ and rotated image as the minority group $S=0$. We also considered other options including adding lines and changing colors, but we observed that the other options often show the adverse affect of making the images noisier and thus reducing the model accuracy unnecessarily. The rotation also allows for simple and effective verification of subgroup labels in generated synthetic data by comparing the mean values of synthetic image vectors to the centroids of real image vectors. To validate this heuristic, we compared the results with 400 manually labeled images from each baseline model and observed high accuracy (e.g., 96.5% for MNIST).
- •
Unknown Subgroup Bias. In the previous subgroup bias setting, $S$ labels are not used during model training; they are only used for evaluation purposes after training.

#### CelebA

We create binary class bias using gender attributes, where we set female and male images as $Y=1$ and $Y=0$, respectively. As discussed in the main text, DP generative models often show low performance on CelebA in challenging bias scenarios like multi-class bias, which can hinder the reliability of fairness analyses (e.g., a random generator achieves perfect fairness by outputting random images regardless of data groups). Notably, we show that DP generative models can produce highly biased synthetic data even in this simple binary class bias setting (Table 4).

### D.2 Baselines

Continuing from Sec. 5, we provide more details on baseline approaches used in our experiments.

#### DP Generative Models

We use three state-of-the-art PTEL-based generative models: GS-WGAN (Chen et al., ), G-PATE (Long et al., ), and DataLens (Wang et al., ). For all models, we use their official Github codes to implement their models and to use their best-performing hyperparameters for MNIST, FashionMNIST and CelebA.

- •
GS-WGAN. GS-WGAN is extensively used in our experiments, as it leverages both PTEL and DP-SGD (Abadi et al., ) to ensure DP and thus allows various integration with other techniques. Their approach first trains a multiple teacher ensemble and considers one teacher as the representative of the other teachers. The output of representative teacher (i.e., gradients) is then sanitized with a DP-SGD based mechanism to train a DP generator. Compared to DP-SGD which operates on the whole minibatch, the DP mechanism in GS-WGAN operates on each data sample and thus can be considered as a composition of $B$ Gaussian mechanism where $B$ is the minibatch size. Our fair sampling preserves their sensitivity analyses despite the potential oversampling as it does not change the sensitivity of each Gaussian mechanism on one input data (i.e, $2C$ due to triangle inequality).
- •
G-PATE and DataLens. G-PATE and DataLens leverage teachers’ votes on intermediate gradients to update the generator. To generate histograms of teachers’ votes and sanitize with DP mechanisms, G-PATE uses random projection and gradient discretization while DataLens uses a top-k stochastic sign quantization of the gradients. Our fair sampling preserves their sensitivity analyses despite the potential oversampling as each teacher still throws only one vote.

#### Privacy-Fairness Approaches

We use a prominent fair training approach based on reweighting (Choi et al., ), and use two recent classification techniques which address both privacy and fairness: DP-SGD-F (Xu et al., ) and DPSGD-Global-Adapt (Esipova et al., ).

- •
Reweighting. As outlined in Sec. 3, the reweighting approach modifies the loss term of a generative model to achieve fairness. We use likelihood ratio computed in each bias setting as the reweighting factor and only modifies the loss term of a discriminator, following their approach. When computing likelihood ratio, we directly compute the value as in Eq. 5 using sensitive group labels for subgroup bias setting; we estimate the value using binary classification approach, implemented in their official Github code for unknown subgroup setting. We note that a public reference dataset is required in the estimation process to effectively train a binary classifier.
- •
DP-SGD-F and DPSGD-Adapt-Global
DP-SGD-F and DPSGD-Adapt-Global are both fair variants of DP-SGD, where clipping bounds are dynamically adjusted to control the fairness-utility tradeoff. To prevent excessive gradient clipping for minority data group samples, DP-SGD-F employs a groupwise clipping approach where each data group has its own clipping bound, while DPSGD-Adapt employs a scaling approach where all per-sample gradients are scaled down depending on a dynamically adjusted scaling factor. As DP-SGD-F and DPSGD-Adapt-Global do not provide the official codes to our knowledge, we faithfully implemented each algorithm based on their papers. We note that DP-SGD-F is not applicable in the unknown subgroup setting as they require number of group samples present in the batch to compute clipping bounds for each group; DPSGD-Global-Adapt is applicable as the scaling factor does not require knowledge on group labels.

## Appendix E Additional Experiments

Continuing from Sec. 5, Sec. 5.1, Sec. 5.3, and Sec. 5.4, we provide more experimental results.

### E.1 Comparison of Computational Time

Continuing from Sec. 5.3, we compare the computational time when integrating PFGuard with existing PTEL-based generative models. Table shows that PFGuard incurs minimal overhead in computational time ($<4\%$), due to the simple modification in minibatch sampling for fairness.

**Table 5: Comparison of computational time of private generative models with and without PFGuard.**
|  | MNIST | FashionMNIST |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Method | w/o PFGuard | w/ PFGuard | Overhead (%) | w/o PFGuard | w/ PFGuard | Overhead (%) |
| GS-WGAN | 7378.30 | 7467.48 | 1.21 | 8114.96 | 8392.58 | 3.42 |
| G-PATE | 30810.25 | 31852.11 | 3.38 | 25238.84 | 26317.07 | 3.56 |
| DataLens | 41590.34 | 41638.19 | 0.12 | 547740.47 | 55714.41 | 1.78 |

### E.2 Experimental Results on FashionMNIST

Continuing from Sec. 5.3, we show the results of the analysis in synthetic data (Table 6) and downstream tasks (Table 7 ) evaluated on FashionMNIST. Compared to the results evaluated on MNIST, private generative models often generate more imbalanced synthetic data w.r.t. sensitive groups and exhibit lower overall image quality. In comparison, PFGuard consistently improves both fairness and overall utility in most cases, similar to the results observed in the MNIST evaluation.

**Table 6: Fairness and utility performances of private generative models with and without PFGuard on synthetic data, evaluated on FashionMNIST with subgroup bias under $\varepsilon=10$. Blue and red arrows indicate positive and negative changes, respectively. Lower values are better across all metrics.**
|  | Fairness | Utility |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | KL ($\downarrow$) | Dist. Disp. ($\downarrow$) | FID ($\downarrow$) | Y=1, S=1 | Y=1, S=0 | Y=0, S=1 | Y=0, S=0 |
| GS-WGAN | 0.558$\pm$0.147 | 0.651$\pm$0.007 | 124.85$\pm$0.00 | 130.95$\pm$0.00 | 278.06$\pm$0.00 | 155.36$\pm$0.00 | 217.00$\pm$0.00 |
| G-PATE | 0.270$\pm$0.026 | 0.494$\pm$0.021 | 245.13$\pm$24.85 | 271.28$\pm$15.32 | 249.66$\pm$10.95 | 275.61$\pm$38.26 | 282.58$\pm$30.95 |
| DataLens | 0.160$\pm$0.022 | 0.388$\pm$0.026 | 165.90$\pm$6.50 | 197.61$\pm$8.90 | 191.72$\pm$8.46 | 173.60$\pm$9.00 | 225.93$\pm$6.53 |
| GS-WGAN + PFGuard | 0.009$\pm$0.065 ($\downarrow$) | 0.065$\pm$0.049 ($\downarrow$) | 113.13$\pm$7.24 ($\downarrow$) | 149.54$\pm$3.96 | 166.69$\pm$10.00 | 114.87$\pm$12.26 | 146.67$\pm$22.52 |
| G-PATE + PFGuard | 0.190$\pm$0.050 ($\downarrow$) | 0.418$\pm$0.049 ($\downarrow$) | 242.20$\pm$42.70 ($\downarrow$) | 267.14$\pm$33.95 | 248.92$\pm$51.93 | 266.91$\pm$51.47 | 295.32$\pm$31.69 |
| DataLens + PFGuard | 0.127 $\pm$0.037 ($\downarrow$) | 0.345$\pm$0.050 ($\downarrow$) | 209.48 $\pm$12.01 ($\uparrow$) | 248.43$\pm$13.12 | 222.16$\pm$16.69 | 222.46$\pm$15.17 | 262.62$\pm$11.37 |

**Table 7: Fairness and utility performances of private generative models with and without PFGuard on downstream tasks, evaluated on FashionMNIST with subgroup bias under $\varepsilon=10$. Blue and red arrows indicate positive and negative changes, respectively.**
|  | MLP | CNN |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | Fairness | Utility | Fairness | Utility |  |  |
| Method | EO Disp. ($\downarrow$) | Dem. Disp. ($\downarrow$) | Acc ($\uparrow$) | EO Disp. ($\downarrow$) | Dem. Disp. ($\downarrow$) | Acc ($\uparrow$) |
| GS-WGAN | 0.773$\pm$0.019 | 0.021$\pm$0.019 | 0.812$\pm$0.009 | 0.795$\pm$0.008 | 0.007$\pm$0.007 | 0.804$\pm$0.003 |
| G-PATE | 0.636$\pm$0.065 | 0.162$\pm$0.059 | 0.875$\pm$0.004 | 0.525$\pm$0.056 | 0.095$\pm$0.064 | 0.884$\pm$0.010 |
| DataLens | 0.484$\pm$0.168 | 0.203$\pm$0.092 | 0.901$\pm$0.030 | 0.328$\pm$0.039 | 0.072$\pm$0.045 | 0.925$\pm$0.009 |
| GS-WGAN + PFGuard | 0.296$\pm$0.099 ($\downarrow$) | 0.152 $\pm$0.033 ($\uparrow$) | 0.884 $\pm$0.015 ($\uparrow$) | 0.449 $\pm$0.082 ($\downarrow$) | 0.203 $\pm$0.037 ($\uparrow$) | 0.910 $\pm$0.011 ($\uparrow$) |
| G-PATE + PFGuard | 0.556 $\pm$0.152 ($\downarrow$) | 0.154 $\pm$0.087 ($\downarrow$) | 0.885 $\pm$0.013 ($\uparrow$) | 0.476 $\pm$0.051 ($\downarrow$) | 0.124 $\pm$0.041 ($\uparrow$) | 0.899 $\pm$0.017 ($\uparrow$) |
| DataLens + PFGuard | 0.387 $\pm$0.154 ($\downarrow$) | 0.153 $\pm$0.103 ($\downarrow$) | 0.858 $\pm$0.025 ($\downarrow$) | 0.394 $\pm$0.109 ($\uparrow$) | 0.093 $\pm$0.074 ($\uparrow$) | 0.877 $\pm$0.025 ($\downarrow$) |

### E.3 Additional Normalization Technique for Faster Convergence

Continuing from Sec. 5.3, we investigate the impact of the normalization factor on the overall image quality of PFGuard. While we use a traditional normalization factor $N_{1}=\sum_{i}h(x_{i})$ for $w(x_{i})\propto h(x_{i})$ in our SIR-based sampling algorithm, we can employ additional normalization techniques to boost the performance. For example, we can use $N_{2}=\sum_{i}h(x_{i})/N_{-i}$ for $w(x_{i})\propto h(x_{i})/N_{-i}$, where $N_{-i}=\sum_{i}h(x_{i})-h(x_{i})$, which is known to help faster convergence of SIR algorithms to the target distribution $p_{\text{bal}}$ (Skare et al., ).

We thus compare the overall image quality resulting from two different normalization options, $N_{1}$ and $N_{2}$, varying batch sizes to analyze their effects on model convergence. We note that using a larger batch size can change the DP analysis (i.e., the more a model uses the data, the weaker privacy it provides). To effectively compare the difference in convergence speed, we create both binary class bias and subgroup bias on the MNIST dataset, where class $Y=0$ is 3 times smaller than class $Y=1$, and group $S=0$ is 3 times smaller than group $S=1$.

Table 8 shows the comparison of image quality using the MNIST dataset, measuring image quality with FID, where a lower value is better. While both $N_{1}$ and $N_{2}$ demonstrate comparable performance when using a large batch size, the performance gap becomes more evident as the batch size decreases. This empirical evidence shows that the performance of PFGuard can be further improved by additionally employing various normalization techniques.

**Table 8: Influence of normalization factor on MNIST with corresponding DP guarantees ($\varepsilon$) to different batch sizes. GS-WGAN is used as the base DP generator.**
|  | Normalization factor |  |
| --- | --- | --- |
| Batch size | $N_{1}$ | $N_{2}$ |
| 128 ($\varepsilon=29.91$) | 75.05±2.26 | 74.58±2.96 |
| 64 ($\varepsilon=19.58$) | 75.35±6.67 | 72.20±4.58 |
| 32 ($\varepsilon=9.99$) | 82.68±7.13 | 78.18±1.85 |

### E.4 Full Results with Standard Deviation

Continuing from Sec. 5.2 and Sec. 5.4, we show full results with standard deviation. Table 9 and Table 10 shows the full results of Table 4.

**Table 9: Full results of fairness and utility performances of private generative models with and without PFGuard on downstream tasks, evaluated on CelebA(S) with binary class bias under $\varepsilon$ = 1. GS-WGAN is excluded due to lower image quality in this setting. Blue and red arrows indicate positive and negative changes, respectively.**
|  | Fairness | Utility |  |  |
| --- | --- | --- | --- | --- |
| Method | Acc. Disp. ($\downarrow$) | Acc ($\uparrow$) | Acc (Y=1) | Acc (Y=0) |
| G-PATE | 0.978$\pm$0.024 | 0.666$\pm$0.003 | 0.014$\pm$0.014 | 0.992$\pm$0.010 |
| DataLens | 0.793$\pm$0.173 | 0.643$\pm$0.031 | 0.114$\pm$0.087 | 0.907$\pm$0.087 |
| G-PATE + PFGuard | 0.736 $\pm$0.126 ($\downarrow$) | 0.678$\pm$0.003 ($\uparrow$) | 0.187 $\pm$0.085 ($\uparrow$) | 0.923 $\pm$0.041 ($\downarrow$) |
| DataLens + PFGuard | 0.725$\pm$0.055 ($\downarrow$) | 0.689$\pm$0.004 ($\uparrow$) | 0.205$\pm$0.040 ($\uparrow$) | 0.931 $\pm$0.015 ($\uparrow$) |

**Table 10: Full results of fairness and utility performances of private generative models with and without PFGuard on downstream tasks, evaluated on CelebA(L) with binary class bias under $\varepsilon$ = 1. GS-WGAN is excluded due to lower image quality in this setting. Blue and red arrows indicate positive and negative changes, respectively.**
|  | Fairness | Utility |  |  |
| --- | --- | --- | --- | --- |
| Method | Acc. Disp. ($\downarrow$) | Acc ($\uparrow$) | Acc (Y=1) | Acc (Y=0) |
| G-PATE | 0.968$\pm$0.025 | 0.668$\pm$0.001 | 0.023$\pm$0.018 | 0.991$\pm$0.007 |
| DataLens | 0.678$\pm$0.027 | 0.686$\pm$0.011 | 0.234$\pm$0.028 | 0.912$\pm$0.005 |
| G-PATE + PFGuard | 0.277$\pm$0.314 ($\downarrow$) | 0.563$\pm$0.028 ($\downarrow$) | 0.378$\pm$0.181 ($\uparrow$) | 0.655$\pm$0.133 ($\downarrow$) |
| DataLens + PFGuard | 0.641$\pm$0.038 ($\downarrow$) | 0.704$\pm$0.007 ($\uparrow$) | 0.276 $\pm$0.031 ($\uparrow$) | 0.917 $\pm$0.008 ($\uparrow$) |

## Appendix F Related Work

Continuing from Sec. 6, we present more related work.

#### Private-only Data Generation

Most privacy-preserving data generation techniques focus on satisfying differential privacy (DP) (Dwork et al., ). The majority of these techniques use Generative Adversarial Networks (GANs) (Goodfellow et al., ) with DP training techniques, although privatizing other generative models have recently been proposed as well (Takagi et al., ; Cao et al., ; Harder et al., ; Liew et al., ; Chen et al., ; Vinaroz et al., ; Yang et al., ; Ghalebikesabi et al., ). One approach is based on DP-SGD (Abadi et al., ), which is a DP-enabled standard stochastic gradient descent algorithm to train ML models (Xie et al., ; Zhang et al., ; Torkzadehmahani et al., ; Bie et al., ). Another approach is based on the Private Aggregation of Teacher Ensembles (PATE) framework (Papernot et al., ; ), which trains multiple models on private data, and updates the generator with differentially private aggregation of multiple model outcomes (Jordon et al., ; Long et al., ; Wang et al., ). GS-WGAN (Chen et al., ) is a state-of-the-art GAN-based technique that combines DP-SGD and PATE where multiple models are trained as in PATE, but their outcomes are processed with the Gaussian mechanism to update the generator as in DP-SGD. In comparison, PFGuard complements private GANs by also improving the fairness of data generation.

#### Fair-only Data Generation

The goal of model fairness is to avoid discriminating against certain demographics (Barocas et al., ; Feldman et al., ; Hardt et al., ), and fair data generation solves this problem by generating synthetic data to remove data bias. The main approaches of fair data generation are as follows: 1) modifying training objectives to balance model training (Xu et al., ; Sattigeri et al., ; Yu et al., ; Choi et al., ; Teo et al., ) and 2) modifying latent distributions of the input noise to obtain fairer outputs (Tan et al., ; Humayun et al., ). In comparison, PFGuard modifies sampling procedures to balance model training while preserving original training objectives and makes the key contribution of satisfying both privacy and fairness.
There is another recent line of work using generated data together with original training data for model fairness (Roh et al., ; Zietlow et al., ), but they focus on classification tasks and assume to use given generative models.

#### Privacy-Fairness Intersection

Recent studies have shown that achieving DP can hurt model fairness in classification tasks (Bagdasaryan et al., ; Farrand et al., ; Xu et al., ; Esipova et al., ), decision-making processes (Pujol et al., ), and even in generation tasks (Ganev et al., ). In addition, there is another notable line of work to investigate privacy-fairness-utility tradeoff, showing that achieving both privacy and fairness will necessarily sacrifice utility (Cummings et al., ; Agarwal, ; Sanyal et al., ). In comparison, our study uncovers the counteractive nature of privacy and fairness, which can result in compromised privacy and fairness, and ultimately compromised utility. To effectively achieve both privacy and fairness in model training, various techniques have been developed for classification tasks (Jagielski et al., ; Xu et al., ; ; Tran et al., ; Esipova et al., ; Lowy et al., ). In comparison, PFGuard focuses on data generation tasks, specifically tailoring its fair training phase to generative modeling objectives, which is learning the underlying training data distributions to generate synthetic data.

## Peer Reviews

confidence: 3
contribution: 1
initial_rating: 3
presentation: 2
questions: - Can you justify assumption 2 on Line 256? (see my feedback in the weaknesses part)

- Can you include one of the aforementioned baselines?

- Do you acknolwedge the additional privacy cost of importance sampling? Can you address that in a meaningful way?

- Have I misunderstood part of your work? To be clear, I think as is, this paper is not ready for publication. However, I want to be fair and make sure that I have not misunderstood your work. So I'll be happy to engage with you during the rebuttal process.
soundness: 1
strengths: - The idea of controlling the fairness-privacy tradeoffs for generative application is worthwhile.

- The writing is clear and easy to follow.
summary: The paper presents PFGuard, a framework for jointly private and fair generative models. The challenges of naively integrating an unfairness mitigation scheme within private generative models are considered. The paper presents an algorithm based on importance sampling to mitigate unfairness first and then privatize using a teacher ensemble a la PATE.
weaknesses: - **Algorithm likely has unaccounted-for privacy leakage.** The paper claims that the privacy of Private Teacher Ensembles comes from data disjointness alone.  This is not true; that guarantee also depends on random partitioning of the private data; therefore not only disjointness is required but so is random sampling. The importance sampling step that the paper employs cannot, by definition, be completely random as it has to take into account sensitive group. This means that there is an additional privacy cost to this sampling step that the paper simply does not consider. Using importance sampling in DP settings for fairness is not new. In fact, one of aforementioned papers Kulynych et al. 2021 do this while also accounting for the privacy cost of importance sampling.

- **Claims of novelty are exaggerated.** The contribution of the paper is doing private and fair generation; its claim novelty is generation as classification has been done before; yet the private generation is achieved via other methods. In general, reading through Section 4 and especially Section 4.3, the paper is lacking substance regarding the particularities that introducing fairness to private generation brings. Over-relying on prior work like (Jordon et al., 2018; Chen et al., 2020; Long et al., 2021; Wang et al., 2021a) has left the analysis incomplete and without justification. 
  
  For instance, I believe this statement on  Line 315 is wrong:  
  
  > PFGuard preserves any sensitivity as long as the PTEL enforce data disjointness; even with fair sampling, a single data point still affects only one teacher.  
  
	- How come? Can't a point in the minority group be resampled to maintain a similar data distribution across all teachers as assumption (2) (Line 256) requires? Speaking of Assumption 2:

- **Assumption (2)-Line 256 is unrealistic.** paragraph on Line 254 reads:
  > Methodology We now present our sampling technique, which guarantees $B \sim p_{\text {bal }}$ based on SIR. We first make the following reasonable assumptions: 1) each data sample has a uniquely defined sensitive attribute $s \in S$ (e.g., race); 2) $p_{\text {bal }}$ is uniformly distributed over $s ; 3$ ) following Choi et al. (2020), the same relevant input features are shared for each group $s$ between the balanced and biased datasets (e.g., $p_{\text {bal }}(\mathbf{x} \mid \mathbf{s}=s)=p_{\text {bias }}(\mathbf{x} \mid \mathbf{s}=s)$ ), and similarly between the training dataset $D$ and any subset $D_i\left(\right.$ e.g., $\left.p_D(\mathbf{x} \mid \mathbf{s}=s)=p_{D_i}(\mathbf{x} \mid \mathbf{s}=s)\right)$ . We now outline the technique step-by-step below.  
  
	- Assumption 2 here means that samples can be balanced out in terms of sensitive group membership. This assumption is unrealistic. How can the sensitive feature be uniformly distributed; when by definition there exists minority and majority sensitive groups? The only possible way I can think of that would make this work in any practical setting is by resampling minority samples over and over. But that is bound to increase the privacy cost.

- **The paper is missing a number of relevant prior work.** I have already mentioned Kulynych et al. 2021  which essentially does importance sampling for DP-SGD. Also, state-of-the-art for private and fair classification is DP-FERMI by Lowy et al. 2023 is not really considered. Remark 1 on page 4, regarding being the first to revleal that fairness and privacy tecnhiques can counteract each other is not true. Yaghini et al. 2023 and Tran et al 2021  and other make the same observations under PATE classification. I am well-aware of the authors contention in Remark 2 and Section A. So I am going to provide counter-arguments why I cannot accept their line of arguments there in.

	- First, authors claim the other works Jagielski et al. 2019 Mozannar et al. 2020 Tran et al 2021 Lowy et al. 2023 all use DP w.r.t. sensitive attribute hence they account for a different DP definition. While that is true of the first 3, it is not true of Lowy et al. 2023, neither is it true for Kulynych et al. 2021 or Yaghini et al. 2023 who all consider central (i.e. w.r.t all attributes) DP as well. Incidentally, Tran et al 2021 and Yaghini et al. 2023 setting is over PATE which is pretty close to the PTEL setting of the paper modulu the generation part. But as established earlier, the present paper does not advance the generative setting beyond prior work.

	- Second, it is unclear to me why challenges of accounting for the privacy cost of adjusting C plays any role in those works not being considered as baselines. If these methods budget their privacy allocation poorly, doesn't that make for a stark and interesting comparison? To be honest, I do not believe these are the best baselines to compare against but I found this line of argumentation faulty.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - Is there any consistent trend in utility and fairness as the privacy budget increases? I would suggest the authors to perform a similar analysis as in (Tran et al., 2021b -- Fig. 2)
- Given the high-dimensional context, other fairness notions, such as Rawlsian Max-Min fairness, could be more appropriate. Can the framework proposed extend to such notions?
soundness: 3
strengths: - Good quality of the presentation
- Extensive experiments on MNIST, FashionMNIST, and CelebA
- Comparison against several baselines, including fair variants of DP-SGD
summary: This paper studies tensions between fairness and privacy in generative models. It proposes PFGuards, a framework leveraging an ensemble of teacher models to jointly enforce fairness and differential privacy while preserving utility. The proposed method consists of training an ensemble of fair teachers using balanced minibatch sampling and using their differentially private (leveraging the PATE framework) aggregated output to train a privatized generator. Experiments demonstrate improved performances in fairness and utility compared to existing approaches.
weaknesses: - Limited choice in epsilon values, making it difficult to understand the effect of privacy budget on utility and fairness
- The choice of datasets and bias settings makes it difficult to determine whether the method would perform effectively in real-world scenarios. I recommend incorporating tabular datasets, as done in (Tran et al., 2021b), to assess group fairness. Additionally, it would be valuable to explore alternative fairness notions for datasets where group fairness may not be applicable

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please refer to Weaknesses.

Also, can you justify the novelty of using PTEL in this work? Because this is the main technique of the proposed framework.
soundness: 3
strengths: 1. I agree with the authors that the conflicts between data privacy and fairness do exist for generative models. It is an interesting research problem to be explored by research community.
2. I think importance resampling can be a useful approach to get balanced training data, which is helpful to train a fair generative model.
3. The experiments shown in tables demonstrated that better utility and fairness are achieved by fixing a privacy budget.
summary: This paper focuses on data privacy and model fairness for developing generative models. They claim natively combining differential privacy and fairness learning techniques may cause conflicts, and then propose PFGuard framework to simultaneously balance the two objective and also the utility. The core insight is to employ an ensemble of multiple teacher models. And they also do experiments on GANs with images datasets as benchmarks to show a better trade-off can be achieved.
weaknesses: 1. The authors have claimed the interested privacy of this paper is to defend against training data reconstruction, while naive differential privacy techniques preserve the membership instead of data content. This has been recognised in the early work [1] but ignored by this paper.
2. Since only a smaller training data is derived after resampling for fairness, this may degrade the quality of generated data, as claimed in [2]. In this case, the quality of generated data will be sacrificed.
3. It is easy to understand that minority samples should be upweighted in fairness. But in what scenarios minority samples should be downweighted for DP? Because sensitivity controls the strength of the added noise, maybe how will minority samples affect the sensitivity should be explained.
4. In Fig. 3, reweighted and rewerighting are both used.  Are they same? In addition, it seems privacy budget is fixed and then the trade-off between utility and fairness is studied, and so as to experiments. This is not aligned with the motivation where fairness and privacy conflicts.
5. The proposed framework is a sequential procedure where the first component is about fairness while the second one is for privacy. In this sense, it is not very convincing to say they two can be better traded off.
6. I would like to see some Pareto front results in terms of three metrics, which can better demonstrate the proposed method.

[1] Bounding Training Data Reconstruction in Private (Deep) Learning, ICML 2022.
[2] Generative Adversarial Ranking Nets, JMLR 2024.

---

confidence: 3
contribution: 1
initial_rating: 3
presentation: 3
questions: All that said, I am open to raising my score slightly if the authors can adequately address the following questions,

Q1: How precisely is the public reference sample handled experimentally? Are all methods given “equal access” so to speak? If the sample is assumed to be complete (i.e. containing all relevant columns from D) also please provide a test on the same metrics you present for the private/fair methods of simply using the holdout sample at different subsample percentages on each task.

Q2: Why should we use importance sampling, instead of just constructing (potentially deterministically) balanced samples using stratified sampling? It seems believable (although is not proved here) that in the PATE/PTEL framework we can use whichever sampling technique we want, so long as it is randomized (although this is not explicitly proven by the authors or cited). However, given that, AND the assumption of access to the membership of samples in protected classes, importance sampling vs. stratified sampling should be explored. Or maybe I’m missing something.

Q3: Please, if you can, offer a more formal characterization of how importance sampling does not effect the privacy guarantee of PATE/PTEL. You can also use the framework presented in Bassily et. al .
soundness: 2
strengths: This work quite successfully communicates the challenges in achieving the privacy/fairness tradeoff. I particularly appreciate the clarity of Figures 1, 2 and 3, along with the care with which the experimental results in Tables 3 and 4 are presented. Additionally, I believe that this is an interesting problem that deserves attention. I have closely read the main paper body, and appreciate the lack of obvious grammatical errors - overall the paper is well written.
summary: This paper introduces PFGuard, a framework for privacy-preserving generative models that integrates fairness through a simple modification in the minibatch sampling process, specifically employing importance sampling based on group membership. The framework builds on PATE/PTEL, which is commonly used for privacy deep private generative models. The paper summarizes challenges in balancing privacy, utility, and fairness in generative models, then proposes the PFGuard framework (which relies on protected group-wise importance sampling), before finally empirically arguing that PFGuard achieves an improved balance of privacy and fairness.
weaknesses: Unfortunately, though the paper writing and presentation is of high quality and clarity, there are issues in terms of originality, significance and correctness.

W1: I’d like to highlight that Remark 1 (on novelty) is not correct and should be removed or further qualified. There appears to be ample prior work that “reveals how fairness and privacy techniques can counteract each other,” some in a more formal ways than this work (Bullwinkel et. al, https://arxiv.org/pdf/2205.04321 , Rosenblatt et. al https://arxiv.org/pdf/2312.11712 , Abroshan et. al (https://proceedings.mlr.press/v238/abroshan24a/abroshan24a.pdf , Cheng et. al https://dl.acm.org/doi/pdf/10.1145/3442188.3445879 )

W2: Additionally, Remark 2 either needs to be removed or needs further clarification - why would we extend classification techniques to the generative setting? Differentially private data generation (both synthetic tabular, see Mckenna et al 2019, 2024, https://proceedings.mlr.press/v97/mckenna19a/mckenna19a.pdf , https://arxiv.org/pdf/2201.12677 , Liu et. al 2021, 2023 https://proceedings.neurips.cc/paper_files/paper/2021/file/0678c572b0d5597d2d4a6b5bd135754c-Paper.pdf , https://proceedings.mlr.press/v202/liu23ag/liu23ag.pdf AND image Ghalebikesabi  et. al https://arxiv.org/pdf/2302.13861 ) is a very mature field, under much lighter assumptions then are necessary for PATE/PTEL.

W3: (W2) leads me to my main issue with the empirical results, which is that this is a bit of an apples to oranges comparison (or at least, it's not obvious that the comparison issue I see has been properly addressed). As you acknowledge, and as is the standard assumption with the PATE framework, we assume access to “ a public reference data on the order of 10%–100% of |D| for the estimation” (line 281). However, some comparisons in your paper (for example, in table 3) compare PFGuard directly to a method like DP-SGD (with further modifications), and for which it is not clear if the public reference data assumption is leveraged by the DP-SGD fit model (there are existing methods to help do this). 

In fact, assuming an unbiased public reference sample is quite a strong assumption, and having no experiments on how a biased public reference sample would effect your results is questionable. Additionally, if I had access to a public reference sample, even only 10% of some large data sample |D|, why wouldn’t I just train on this sample? I’m missing where you note something about label missingness in this public sample, but maybe that isn’t the assumption? Is the assumption that we have full access to this public unbiased sample, and that its size is substantial? We would certainly want to compare to just training on that if this were the case.

W4: My 1 sentence summary of the proposed methodology here is as follows: take the PATE/PTEL framework, and during the sampling for teacher creation, use importance sampling based on group membership. This, in of itself, is a reasonable approach, but it is not clear to me how this work improves on the component parts of the PFGuard method (which are well established and well studied methods), besides using them in tandem. Nor does that work consider the model of importance sampling and how that might affect utility in other ways from a formal perspective, despite considerable prior work on the utility of PATE (Bassily et. al, https://arxiv.org/pdf/1803.05101 ). Nor does it contend formally with any potential privacy concerns (even to just dispel this concern with a short proof or proof adjustment). Given that I have some issues with the assumptions of the empirical results, this leads me to my score.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: Questions:
 * "Additional fair sampling does not require additional training complexity compared to say adding a loss term for fairness": I would argue that adding a loss term that can be backpropagated over is much simpler than having to do your complex sampling procedure to construct balanced minibatches. Can you justify or elaborate on why fair sampling is less complex? I don't think this claim is core to your paper so I think you could also do without it. You mention it later in the paper as well.
 * In your extensions to unavailable sensitive attributes, why do you need to train your classifier on less data? Given one of the benefits of PFGuard is any methods you apply before the teacher-distillation step need not be private, I would expect the best thing to do from a fairness perspective is train the best, beefiest sensitive-attribute classifier possible for fair training.
 * In your figures, why do you apply [method + PFGuard]? Isn't PFGuard an end-to-end fairness + privacy method? From what I can tell in your results you are applying PFGuard to existing DP generative models. This makes it difficult for me to interpret your results, and the unique impact of PFGuard on your results.
 * My intuition tells me that privacy always comes at a cost to utility, and this teacher + distillation procedure should be even more noisy than traditional DP-SGD methods. Can you comment on why this is not the case in your results? Why isn't a more advanced privacy method that permits fairness also resulting in further costs to utility than a traditional fairness method? How much of a fairness drop (from teacher fairness to generator fairness) do we incur because of the private distillation?
soundness: 4
strengths: Strengths:
 * I really liked the discussion of how fairness and privacy can come into conflict. I think it is somewhat "folklore" in the trustworthy ML community so I appreciated a more detailed discussion of the topic, especially in Section 3.
 * The PFGuard framework is a great solution to the problem. "Isolating" the fair training component and then distilling this knowledge privately is a great approach that I could see being taken to do many different downstream applications that also require privacy!
 * I think it is good the authors acknowledge that fairness adjusts sensitivity, which can still be made private with more noise, but point out the amount of noise to add may be unclear or performance degrading.
summary: In this work, the authors study the intersections of fairness and privacy in generative models, and show that these two goals can come into conflict. Furthermore, they highlight that naively combining existing fairness and privacy methods can fail. They propose PFGuard, a framework for achieving fairness and privacy simultaneously and then conduct experiments validating their method.

Overall, I think this is a paper with great contributions and analysis. I think some of the boldest claims should be softened, but otherwise I think it is a good candidate for ICLR.
weaknesses: Weaknesses:
 * The authors focus on GANs as opposed to Diffusion Models, lessening the contemporary impact of the paper. This isn't necessarily a weakness, more an observation.
 * In Section 3, you start with **Adding Fairness Can Worsen Privacy** and describe in text some vague settings where fairness and privacy can come into conflict. I really wanted to see an elaboration or concrete example of how this could happen. I was very excited to see this in **Adding Privacy Can Worsen the Fairness-Utility Tradeoff** part! I think this is a great contribution. However, I think in the second part, you actually discuss both how fairness can worsen privacy (how fairness can modify the sensitivity, C) and then how adding privacy can worsen fairness (clipping worsens fairness adjustments). You should split these up according to the headers you wrote by moving the first example up to the first section. I would rewrite these two sections and cut some of the vaguer writing in the first section, and spend some more time elaborating on these two examples as they are a great contribution that I would be interested in more discussion of.
 * You say in Remark 1 that your study is the first to reveal that fairness and privacy techniques can counteract each other. I think this is too bold of a claim, given works you cite such as [1] and [2] that explore the topic.
 * I wouldn't say giving each teacher *probabilistically* one sample of the minority group is enough data to expect the teacher to be adequately fair. How did you come to this heuristic, and how can you justify it?
 * There are no fairness guarantees offered by this method because of data resampling and teacher distillation. I think the link between having balanced minibatches and the fairness of the teachers is more clear, but I wonder how distillation affects bias or if there is any literature you could cite here.

Notation:
 * Definition 2.1 - I would explain that domain $\mathcal{D}$ is a dataset, thus why D, D' can differ by a single sample.
 * You should also describe what function we measure sensitivity over. Is it the loss? is it the gradient? is it the model outputs? This will make it much clearer how exactly fairness interventions impact sensitivity in Section 3.

[1]: Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate
impact on model accuracy. Advances in neural information processing systems, 32, 2019.  
[2]: Tom Farrand, Fatemehsadat Mireshghallah, Sahib Singh, and Andrew Trask. Neither private nor fair:
Impact of data imbalance on utility and fairness in differential privacy. In Proceedings of the 2020
workshop on privacy-preserving machine learning in practice, pp. 15–19, 2020.

Meta-review: The submitted paper introduces "PFGuard," a framework for (image) generation model designed to navigate fairness-privacy-utility trade-offs. Broadly speaking, PFGuard is based on an ensemble of teacher models for balancing fairness and privacy during model training, blending DP mechanisms and balanced mini-batch sampling. Experiments indicate that PFGuard achieves competitive fairness and privacy guarantees without significant utility loss, and show that PFGuard can improve over baseline methods in synthetic data generation and downstream task performance.

Reviewers generally appreciated the focus on simultaneously considering privacy and fairness in generative modeling. They also had mostly a borderline view of the paper. Reviewer sxpU highlighted the experiments on multiple datasets and baselines as a strength, while also raising questions on the paper's chosen fairness definitions. Reviewer WKsH found the approach sound but expressed concerns about the framing of privacy risks,  novelty relative to prior work, and raised issues with the fairness notion used. Reviewer pijU appreciated the modularity of PFGuard’s approach but raised questions regarding the utility impacts of teacher ensemble structures and their scalability. They also expressed serious concerns about positioning relative to prior work. Reviewer oBbq -- the most critical reviewer -- stood by their concerns that the utility of importance sampling is not theoretically justified relative to simpler techniques (e.g., stratified sampling).

Despite these limitations, the reviewers almost unanimously leaned toward a tepid accept. I side with this view, though I would have appreciated if the authors had a more thoughtful discussion on the fairness definitions used in the paper (in line with comments from Reviewre sxpU on connections with other notions of fairness). Since the paper moves the bar forward regarding the interplay of privacy and fairness, I recommend acceptance.

Decision: Accept (Poster)