PFGuard: A Generative Framework with Privacy and Fairness Safeguards
Authors: Soyeon Kim
Yuji Roh
Geon Heo
Steven Euijong Whang
Venue: ICLR 2025 Poster (2025)
Keywords: Trustworthy AI
Responsible AI
ML Fairness
Differential Privacy
Generative Model
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We propose PFGuard, the first generative framework that simultaneously achieves statistical fairness and differential privacy on high-dimensional data.

Abstract: Generative models must ensure both privacy and fairness for Trustworthy AI. While these goals have been pursued separately, recent studies propose to combine existing privacy and fairness techniques to achieve both goals. However, naively combining these techniques can be insufficient due to privacy-fairness conflicts, where a sample in a minority group may be represented in ways that support fairness, only to be suppressed for privacy. We demonstrate how these conflicts lead to adverse effects, such as privacy violations and unexpected fairness-utility tradeoffs. To mitigate these risks, we propose PFGuard, a generative framework with privacy and fairness safeguards, which simultaneously addresses privacy, fairness, and utility. By using an ensemble of multiple teacher models, PFGuard balances privacy-fairness conflicts between fair and private training stages and achieves high utility based on ensemble learning. Extensive experiments show that PFGuard successfully generates synthetic data on high-dimensional data while providing both DP guarantees and convergence in fair generative modeling.

Published as a conference paper at ICLR 2025
PFGUARD: A GENERATIVE FRAMEWORK WITH PRIVACY AND FAIRNESS SAFEGUARDS
Soyeon Kim', Yuji Roh", Geon Heo', and Steven Euijong Whang")
1 KAIST, {purplehibird, geon. heo, swhang) wwhang)@kaist... (kaist................l ac. kr 2 Google, yuwirohhgoogolleee.... com
ABSTRACT
Generative models must ensure both privacy and fairness for Trustworthy AL While these goals have been pursued separately, recent studies propose to combine existing privacy and fairness techniques to achieve both goals. However, naively combining these techniques can be insufficient due to Srivacy-farmeee conflicts, where sample in minority group may be represented in ways that support fairness, only to be suppressed for privacy. We demonstrate how these confficts lead adverse effects, such privacy violations and unexpected fairness-util utillty tradeoffs. To mitigate these risks, we propose PFGuard, a generative framework with privacy and fairness safeguards, which simultancously addresses privacy, fairness, and utility. By using an ensemble of multiple teacher models, PFGuard balances privacy-fairness fairness conflicts between fair and private training stages and achieves high utility based on ensemble learning. Extensive experiments show that PFGuard successfully generates synthetic data on high-dimensional data while providing both DP guarantees and convergence in fair generative modeling.
INTRODUCTION
Recently, generative models have shown remarkable performance in various applications including vision (Wang et al., 2021b) and language tasks (Brown al. 2020), while also raising significant ethical concerns. In particular, privacy and fairness concerns have emerged as generative models aim to mimic their training data. On the privacy side, specific training data can be memorized, allowing the leakage of personal sensitive information (Hilprecht al.) 2019 Sun et al. 2021). On the fairness side, any bias in the training data can be learned, resulting in biased synthetic data and unfair downstream performances across demographic groups (Zhao et al. 2018 Tan et al.) 2020).
Although privacy and fairness are both essential for generative models, previous research has primarily tackled them separately. Differential Privacy (DP) techniques (Dwork et al. (2014), which provide rigorous privacy guarantees, have been developed for private generative models (Xie et al.. 2018 Jordon al.) 2018); various fair training techniques, which remove data bias and generate more balanced synthetic data, have been proposed for fair generative models (Xu et al. 2018 Choi et al., 2020). To achieve both objectives, harnessing these techniques has emerged as promising direction. For example. Xu et al. (2021) combine fair pre- pre-processing technique (Celis et al. (0200 with private generative model (Chanyaswad al. 2019) train both fair and private generative models.
However, we contend that naively com- bining developed techniques for privacy and fairness can lead to a worse privacy- fairness-utiling tradeoff, where utility is a model's ability to generate realistic syn- thetic data. We first illustrate how privacy and fairness conflict in Fig. [ Given the data samples Mi. M2. M3. and mi where M and m denote the majority and minority data groups, respectively. DP and fairness techniques play tug-of-war re-
High privacy risk
Privacy bound
Privacy
Individunl usage
M,
M2
Ma
M.
M,
M,
m0
Group usage
M
Unbalanced
Faimess
Balanced
Privacy O Faimess X
Privacy X Fairness O
Figure 1: Privacy-fairness conflict. Privacy techniques
prefer the left-hand scenario to prevent privacy risk of a certain data sample, while fairness techniques prefer the right-hand scenario to balance learning w.r.t. groups.
Corresponding author

---

Published as a conference paper at ICLR 2025
Fair training stage
Private training stage
B
Y
01
Input
Aggregate teacher outputs
noise
Partition data
Sample balanced mini-batches
ODP
with DP noise
D
On
Generated image
Generator
Data
Teacher ensemble
Not accessible by adversary Accensible adversary
Figure 2: Overview of PFGuard. PFGuard integrates fairness and privacy into generative models through two-stage process. In the fair training stage (blue), we train fair teacher models by sampling balanced mini-batches from biased data partitions. In the private training stage (red), we aggregate the teacher outputs with random noise ensure Differential Privacy (DP). After training, only the trained DP generator is publicly released, safeguarding the privacy of all other components (c.g., the teacher ensemble). Through these training stages, PFGuard not only ensures fairness and privacy but also achieves high utility by leveraging ensemble learning of teacher models resulting in unbiased, private, and high-quality synthetic data. More details on the framework are presented in Sec.
garding the use of minority data point m1; DP techniques limit its use prevent privacy risks such as memorization, while fairness techniques increase its use to promote more balanced learning w.r.t. groups given the biased data. As result, fairness techniques may undermine privacy by overusing my. while DP techniques may undermine fairness by limiting mi 's usage. Moreover, combining different techniques can introduce new technical constraints, reducing the effectiveness of original methods. For instance, the fair preprocessing technique used by Xu al. (2021) hinders the utility of the DP generative model by requiring data binarization, which incurs significant information loss on high- digh-simmenional data such as images restricting their overall framework only applicable to low-dimensional structural data.
Therefore, we design generative framework that simultancously addresses fairness and privacy while achieving utility for high-dimensional synthetic data generation. To this end, we propose PFGuard: a generative framework with Privacy and Fairness Safeguards. As illustrated in Fig. [ the key component is an ensemble of intermediate teacher models, which balances privacy-fairnes conflicts between fair training and private training stages. In the fair training stage, design new sampling technique to train fair teachers, which provides a theoretical convergence guarantee to the fair generative modeling. In the private training stage, we employ the Private Teacher Ensemble Learning (PTEL) approach (Papernot et al. 2016 2018), which aggregates each teacher's knowledge with random DP noise (e.g., noisy voting), to privatize the knowledge transfer to the generator. As result, PFGuard provides a unified solution to train both fair and private generative models by transferring the teachers' fair knowledge in privacy-preservee
manner.
Compared to simple sequential approaches, PFGuard is carefully designed to address privacy-fairness confficts. Recall that fairness techniques can incur privacy breaches by overusing minority data; in contrast, PFGuard prevents privacy breaches by decoupling fairness and privacy with intermediate teacher models. Although fair sampling can still compromise privacy in teacher models by potentially overusing minority data, PFGuard ensures privacy in the generator solely with the privatized teacher output, as shown in Fig. [ Also, recall that privacy techniques can lead to fairness cancellation by suppressing the use of minority data; in contrast, PFGuard avoids fairness cancellation through teacher-level privacy bounding using PTEL approaches. Compared to sample-level privacy bounding methods like gradient clipping (Abadi et al. 2016), teacher-level bounding leaves room for teachers to effectively learn balanced knowledge via fair training. As result, PFGuard provides strict DP guarantees for the generator and better preserves fairness compared to the combination of fairness-only and privacy-only techniques see more analyses in Sec.
target model - by training
Moreover, PFGuard compatible with a wide range of existing private generative models and preserves their utility. PTEL is widely adopted in private generative models as it provides prominent privacy-utility tradeoff (Jordon et al., 20188 Chen et al. 2020 Long et al. 2021 Wang et al., 2021a) PFGuard can extend any of these models with fair training stage as shown in Fig. which requires a simple modification in the minibatch sampling process. Since additional fair sampling can be advantageous in maintaining optimization complexity compared to say adding loss term for faimess, PFGuard preserves the rivacy-utility tradeoff of PTEL as well while improving faimess. We also provide guidelines control the faimess-privacy-atilyy tradeoff more details in Sec.
2

---

Published as a conference paper at ICLR 2025
Experiments in Sec. Sec.5pss show that PFGuard successfully generates high-dimensional synthetic data while ensuring both privacy and faimess; to our knowledge, PFGuard is the first framework that works on digh-dimensional data including images. Our results also reveal two key findings: (I) existing private generative models can produce highly-biased synthetic data in real-world scenarios even with simple bias settings, and (2) na've combination of individual techniques may fail to achieve either privacy or fairness even with simple datasets highlighting PFGuard's ' effectiveness and the need for a better integration of privacy and fairness in generative models.
Summary of Contributions 1) We identify how privacy and fairness conflict with each other, which complicates the development of responsible generative models. 2) We propose PFGuard, which is to our knowledge the first generative framework that supports privacy and fairness for high-dimensional data. 3) Through extensive experiments, we show the value of integrated solutions address the tradeoff compared to simple combinations of individual techniques.
2 PRELIMINARIES
Generative Models We focus on Generative Adversarial Networks (GAN) (Goodfellow et al., 2014), which are widely-used generative models that leverage adversarial training of two networks to generate realistic synthetic data: 1) generator that learns the underlying training data distribution and generates new samples and 2) a discriminator that distinguishes between real and generated data. The discriminator can be considered as the teacher model of the generator, as the generator does not have access to the real data and only learns from the discriminator via the GAN loss function.
Differential Privacy To privatize generative models, 2014). a gold standard privacy framework that enables quantified privacy analyses. DP measures how much adversary algorithm M, using two adjacent datasets that differ by one sample as defined below.
Differential Privacy (DP) (Dwork et al.
infer about one data sample based differences in two outputs of an
Definition 2.1. ((s, DDDffferrnnt Privacy (Dwork et al. 2006a)) A randomized mechanism M D - with range satisfies (e, 8)-differential privacy if for any two adjacent datasets D, D' and for any subset of outputs O C R, the following holds:
Pr(M(D) E O S ef Pr(M(D)) E O)+ +
where e is the upper bound of privacy loss, and 5 is the probability of breaching DP constraints.
We can enforce DP in an algorithm in two steps (Dwork al. 2014). Given a target algorithm f to enforce DP and dataset DD we first bound sensitivity (Def. (.2). which captures the maximum influ- ence of single data sample on the output of f. We then add random noise with a scale proportional to the sensitivity value. A common way to add noise is to utilizing Gaussian mechanism (Dwork) et al. 2014) (Thm. 2.I). which uses Gaussian random noise with a scale proportional to l2-sensitivity. Definition 2.2. (Sensitivity (Dwork et al. 2014)) The Ip-sensitivity for a d-dimensional function f: + Rd is defined as A max [[(D) - fl)')l, over all adjacent datasets D, D'.
D,D
Theorem 2.1. (Gaussian mechanism Dwork et al., 20144 Mironov 2017)) Let f X Rd be an arbitrary d-dimensional function with lz-seniitivity A7 The Gaussian mechanism Mg parameterized by adds Gaussian noise into the output, i.e., Ma(x) = f(x) + NO, I²I), and satisfies (e, 8)D for a 2 2 lm(1225/6)A3//
Fairness We consider a generative model to be fair if two criteria are satisfied: 1) the model generates similar amounts data for different demographic groups with similar quality, and 2) the generated data can be used to train fair downstream model w.r.t. traditional group fairness measures. For 1), we measure the size and image quality disparities between the groups using the Fréchet Inception Distance (FID) score (Heusel et al. 2017 Choi et al. 2020 to assess image quality. For 2), we use two prominent group fairness measures: equalized odds (Hardt et al. (2016) where the groups should have the same lahel-wise accuracies; and demographic parity (Feldman al. 20155) where the groups should have similar positive prediction rates.
3 PRIVACY-FAIRNESS CONFLICTS IN EXISTING TECHNIQUES
In this section, we examine the practical challenges of integrating privacy-only and fairness-only techniques to train both private and fair generative models. Based Fig. Ts intuition on how
3

---

Published as a conference paper at ICLR 2025
privacy and fairnsss conflict, we analyze how existing approaches for DP generative models and fair generative models can technically conflict with each other using standard DP and faimess techniques: DP-SGD (Abadi et al. 2016) and reweighting (Choi et al. 2020). Let (xx) denote the gradient of the data sample x, and psal and Phias denote balanced and biased data distributions, respectively.
DP-SGD is a standard DP technique Chen et al. (2020) for converting non-DP algorithms to DP algorithms by modifying traditional stochastic gradient descent (SGD). Compared SGD, DP-SGD 1) applies gradient clipping to limit the individual data point's 's contribution, where (xx) is clipped to g(x)/ max(1, (x)]2) /C) (the sensitivity becomes the clipping threshold C)), and
2) uses Gaussian mechanism (Thm. 2.1 to add sufficient noise to ensure DP.
Reweighting is a traditional fairness method (Horvitz & Thompson| 1952) widely used in gener- ative modeling (Choi et al., 2020 Kim et al. (0024), which assigns greater weights to minority groups for a "balanced" loss during SGD. In particular, setting the sample weight to the likelihood ratio h(xa)=Poal( (x.) //oiass (x.) produces an unbiased estimate of Ex Ex [x]] as follows:
Ex-mas [x) 1((x)] Ex-epii (x) Pbal (x)
Ex-malgg(x
Phias
(x)
Naïvely Adding Fairness Can Worsen Privacy Ensuring fairness in DP generative models can significantly increase sensitivity (Def. 2.2 leading to invalid DP guarantees. Sensitivity, which measures data sample's 's maximum impact on an algorithm, is crucial in DP generative models because the noise amount required for DP guarantees is often determined by this sensitivity value. However, integrating fairness techniques DP generative models can invalidate their sensitivity analyses by adjusting model outputs for fairness purposes. One example is the aforemetioned reweighting technique. To balance model training across groups, reweighting technique amplifies the impact of certain data samples (e.g., minority data samples), particularly by amplifying their gradients. However, performing reweighting on top of DP-SGD can incur privacy breaches, as gradient amplified beyond C' can cancel the gradient clipping in DP- invalidating the sensitivity C' provided from DP-SGD. Other examples include directly feeding data attributes such as class labels or sensitive attributes (e-g., race, gender) to a generator for more balanced synthetic data (Xu et al.) 2018, /atttggei et al. 2019 Yu et al. 2020), which can cause large fluctuations in the generator output and similarly end up increasing sensitivity. This increased sensitivity by fairness techniques requires more noise to maintain the same privacy level, compromising the original DP guarantees unless modifying DP techniques add more noise. However, this modification is also not straightforward as assessing the increased sensitivity by fairness techniques can be challenging (Tran et al. 2021b).
Naïvely Adding Privacy Can Worsen the Fairness- Utility Tradeoff Another direction is to ensure privacy in fair generative models, but configuring an appropriate privacy bound can be challenging, which can lead to unexpected and unstable fairness-utility tradeoffs.
In contrast to the previous demonstration, we can ensure privacy in reweighting- fair gen- erative models by replacing SGD to DP-SGD, guaranteeing fixed sensitivity value C' via gra-
Reweighting preserved
Clipping threshold
Reweighting cancelled
Gradient
High noise
Low noise
dient clipping; however, finding C' that balances fairness and utility can be challenging. Gradient clipping now undoes the fairness adjustments, as reweighted gradients g(x)-h(( -x() are clipped to g (x)) max(1, (g)))-l((xx xx Il2/C). and Eq. | does not hold if C < g(x)-h(x). Here, one solution is to use larger C such that C 2 g(x)-((x). However, increasing C also increases the noise required for DP, which re-
Group M Group m Loose clipping
Group 1d
Group m
Aggressive clipping
Faimess O Utility X
Faimess X Utility O
Figure 3: Fairness-utility tradeoff caused by DP- SGD when used on top of reweighting. Depending on the choice of C, DP-SGD may compromise utility (left) or fairness (right).
duces utility as illustrated in Fig. 3 Therefore,
selecting C' that balances fairness and utility may necessitate extensive hyperparameter tuning (Bu ct al. 2024), complicating the systematic integration of DP into fair generative models.
Overall, we show that a naive combination of existing fairness-only and privacy-only techniques can be insufficient to achieve both objectives. While we have not exhaustively covered all possible combinations, one can see how privacy breaches and unexpected occur without a careful design. To avoid these downsides in naive combinations, we emphasize the need for unifted design that integrates both privacy and faimeess into generative models.
fiimess-utility tradeoffs can easily
4

---

Published as a conference paper at ICLR 2025
Remark 1. We emphasize the need for a framework tailored to generative settings. There are notable fairness fairvess-privacy techniques for classification, but directly extending them to data generation can be challenging due to the fundamentally different goals of the two settings see more details in Sec. A
4 FRAMEWORK
We now propose PFGuard, the first generative framework that simultancously achieves statistical fairmess and DP on high-dimenn -igeeniiensional data, such as images. As shown in Fig. PFGuard balances privacy-faimess faimess conflicts between fair and private training stages using an ensemble of teacher models as key component. In Sec. we first explain the fair training stage, which trains a fair teacher ensemble. In Sec. 4.2 we then explain the private training stage, which transfers the knowledge of this teacher ensemble to the generator with DP guarantees and ultimately trains generator that is both fair and private. In Sec. S.3.4.3 we lastly discuss how PFGuard's s integrated design offers advantages in terms of fairness, utility, and privacy compared to the naïve approaches discussed in Sec.
4.1 FAIR TRAINING WITH BALANCED MINIBATCH SAMPLING
Intuition We ensure fairness in the teachers by balancing the minibatches used for training. Here we assume general training setup of stochastic gradient descent, where we iteratively pick subset B of the training data (i.e., minibatches) to update model parameters more efficiently. Since generative models then only learn the underlying data distribution through B, using balanced mini-batches B ~ Pbu will result in modeling Psal even if we have biased training data D ~ Puias- While another approach is to debias the training data itself by acquiring more minority data, this approach is costly and often infeasible for private domains with limited publicly-available data (Jordon et al. 2018).
Theoretical Foundation To develop a fair minibatch sampling technique with a convergence guarantee, we leverage Sampling-Importaaee Importance Resampling (SIR) (Rubin) 1988] /mith & Gelfand 1992) as the thcoretical foundation. SIR is a statistical method for drawing random samples from a target distribution (2x) by using a proposal distribution (xx). SIR proceeds in two steps: 1) we draw a set of n independent random samples R2 = (x: } __ from (xx) and 2) we resample a smaller set of m independent random samples R2= {2; m from R2 Here, the resampling probability w(xx)) is set proportional to (xx))=(<x;) //;),, which is the likelihood ratio of a sample x; in π (xx and ((r). Then, the resulting samples in R2 are approximately distributed according to ((x) as follows:
h(a:) Pr(z S 11 = Singspu(a) E Dim M(x))
E; 1z, S f)T(z)//(Zz)
Σ. (
J [1 (aStttt)MMM)ddee
(3)
7-000 (o(o))(())((d)d
where 1(.) is the indicator function. The distribution becomes exact when n + xo.
Methodology We now present our sampling technique, which samples B Poal based on SIR. We first make the following reasonable assumptions: 1) each data sample has uniquely defined sensitive attribute s E S (e.g., race); 2) target Pbat a uniform distribution over s; 3) following Choi et al. (2020), the same relevant input features are shared for each group between the balanced and biased datasets (c.g.. Peal (x|s=s) = Poias (x|s=s)). and similarly between the training dataset D and any subset D, (c.g.. sx{===
5) = PD. (x(s=))). We now outline the technique step-by-step below.
(1) We set the target distribution to Phul (x) and the proposal distribution to Poias Px)x as our goal is to sample balanced minibatch B ~ Pbal from the biased training dataset D ~ Poias-
(2) We divide D into nyy disjoint subsets (DiHE, to train teacher models {T such that each D; is used for T, and retains the same distribution of as D using the labels (i.e..
PD. (s=s) (s-ss))). Then, we can derive P, using assumption 3) above as follows:
PD. PD. (x) E.PD, (x|s=s)pD, (s=s) ExPD (x(ss=)))p(ps==))
Poins (x)
(4)
(3) We sample B from D. with a resampling probability w(x) that proportional to h((x)
Poal (x) (x), which is computed as follows:
(x)
Phat X s= Phat) (s=s)
Phat (S=
1/18]
h(x)= Poins (x)
(5)
Poias (x s= s==s)pes Paias (s=s)
Poias (s=s)
[{xED|s=>\//pi)
5

---

Published as a conference paper at ICLR 2025
where the second and third equality follows from assumption 1) and 3) above, respectively, and the last approximation follows from assumption 2) above and D ~ Peias.
A sample B from the above procedure is approximately distributed according to Poal based on SIR; we sample Dj from Phias and resample B from D,, where the resampling probability is proportional to h(x)= = Poal (oa(()) Poiss Since a large number of minibatch samplings is needed to train generative models, the B distribution eventually converges to Pbul. leading to a fair generative modeling of Phul-
Extensions Our fair sampling technique is also extensible to private settings where the label of sensitive attribute is unavailable, for example due to privacy regulations (Jagielski et al. 2019 Mozannar et al. 2020; Tran et al. 2002)) In such settings, we can employ a binary classification approach estimate lx() like Choi et al. 2020). While their work focuses on non-privale settings and assumes an unbiased public reference data on the order of 10%-100% of [D] for the estimation,
this assumption can be unrealistic in private domains due to the lack of public data. Our empirical study in Sec. 5.3 shows that we can achieve fairness with only 1-10% of the data, leveraging the ensemble learning of multiple-teacher structure, which can further reduce the estimation error. Note that in this extension, the convergence guarantee may not hold, as D; ~ Phias in step (2) might not be true practice if the dataset is randomly partitioned without considering sensitive attribute labels.
4.2 PRIVATE TRAINING WITH PRIVATE TEACHER ENSEMBLE LEARNING
Intuition Although the fair sampling technique in Sec. 4. can enhance faireess, it can also introduce privacy risks due to inherent frivacy-fairness confficts (Sec. 3 Specifically, repeated resampling of certain data samples during teacher training can lead to memorization, compromising privacy in teacher models and ultimately affecting the generator during knowledge transfer. To address this privacy risk, we privatize knowledge transfer rather than the teacher models themselves, ensuring DP solely in the generator. Since the generator is only needed after training for synthetic data generation, we can only release DP generator to the public (Chen et al. 2020 Long et al. 2021).
Privacy Guarantee We utilize Private Teacher Ensemble Learning (PTEL) (1 (Papernot et al. 2016
2018) to ensure DP during knowledge transfer. Unlike non-private ensemble learning, PTEL 1) trains each teacher model on a disjoint data subset and 2) adds noise proportional to the sensitivity of the aggregated knowledge (e.g., class labels (Papernot et al.) 2018), gradients (Chen al., 2020). etc.). Although aggregated knowledge can differ, sensitivity is commonly derived from data disjointness, where a single data point affects at most one teacher. For instance, GNMax aggregator (Papernot et al. 2018) aggregates predicted class label for a query input x from teachers (T.) as tollows:
GNMax(X) = arg max (n, (x) + +N((0,00)) for =1=11.... L.....
(6)
where ny denotes the vote count for the j-th class (i.c., n2. (x) = Hi Tx())=HH and N o²) denotes random Gaussian noise. Here, the lz-sensitivity (Def. 2.2 is v2, as a single data point affects most one teacher, increasing the vote counts by 1 for one class and decreasing the count by 1 for another class (see a more detailed analysis in Sec. B.2). Consequently, the GNMax aggregator satisfies (e, 8-DD for o 2 /8 lml..155// based on the Gaussian mechanism (Thm. 2.1b.
Methodology PFGuard edits existing PTEL-based generative models to achieve both privacy and
fairness by simply modifying the minibatch sampling process described in Sec. 4. PTEL has been widely used to train DP generators (Jordon et al. 2018 Chen et al. 2020 Long et al. 2021 Wang al. 2021a), resulting in various sensitivity analyses. While integrating fair training techniques can invalidate these analyses as discussed in Sec 3 PFGuard preserves any sensitivity value as long as PTEL enforce data disjointness; even with fair sampling, a single data point still affects only one teacher. PFGuard thus enhances fairness of various PrEe- generative models, while preserving their own DP guarantees. We present formal DP guarantees and the training algorithm in Sec. C
Number of Teachers We provide guidelines on how to set the number of teachers ny for PFGuard, which affects the mrivacy-fairees-utiie
tradeoff. While nr is typically tuned via experiments (Long jet al. 2021; Wang et al.) 2021a), we set a maximum value of ny considering fairness. Since large Ny would result in diverse ensemble that can generalize better, but also lead to teacher receiving data subset that is too small for training, we suggest ny to be at most [|D| min Prinx where [-] denotes the floor function. Since this equation captures the size of the smallest minority data group,
6

---

Published as a conference paper at ICLR 2025
this mathematical upper bound guarantees that each teacher probabilistically gets at least one sample of the smallest minority data group. In Sec. 5.3. we show how this bound helps avoid compromising faireess. We also discuss how to set Ro when sensitive attribute labels are unavailable in Sec.
4.3 ADVANTAGES OF INTEGRATED DESIGN
We discuss how PFGuard overcomes the challenges of naïve approaches discussed in Sec. F
Balances Privacy-Fairnest Conflict PFGuard can sidestep privacy breaches and fairness can- cellation arising from privacy-fairness conflicts. Compared to existing fairness-only techniques. which can compromise DP guarantees and require complex sensitivity assessments when applied to DP generators, PFGuard automatically preserves DP guarantees of base DP generators by maintaining sensitivities, eliminating the need of complex assessments. Compared to existing privacy-only techniques, which often directly limit an individual sample's s influence (e.g., gradient clipping discussed in Sec. [ and can lead to faimess cancellation by suppressing the use of minority data, PFGuard uses indirect privacy bounding, in the sense that we limit the influence of teacher models in order to limit individual sample's 's influence. Since there are no DP constraints during the teacher learning, the teacher models can effectively learn balanced knowledge across data groups.
Achieves Better Fairness-Utility Tradeoff The fair sampling of PFGuard introduces minimal training complexity, which can preserve utility while enhancing faimess. By modifying only the minibatch sampling process, PFGuard maintains the original loss function and avoids the additional fairness loss terms (Sattigeri et al., 2019; Yu et al.) 2020 or auxiliary classifiers (Tan et al., 2020 Um & Sub 2023) typically employed in fairness techniques. As result, we show how PFGuard can achieve prominent carmess--uilttyy tradeoff with negligible computational overhead when integrated into PTEL-based generative models in Sec. 5] Further discussions on PFGuard's 1) flexibility with other fairness techniques and 2) synergy with adversarial learning are presented in Sec.
EXPERIMENTS
We perform experiments to evaluate PFGuard's effectiveness in terms of fairness, privacy, and utility,
Datasets We evaluate PFGuard on three image datasets: 1) MNIST (LeCun et al. 1998) and FashionMNIST (iiao et al.) 2017) for various analyses and baseline comparisons, and 2) CelebA (Liu ct al. 2 to observe performance in real-world scenarios more closely related to privacy and fairness concerns. MNIST contains handwritten digit images, FashionMNIST contains clothing item images, and CelebA contains facial images. While MNIST and FashionMNIST are simplistic and less reflective of real-world biases, they enable reliable faimess analyses on top of high-performing DP generative models on these datasets, making them widely adopted in recent studies addressing the privacy-fairness intersections (Bagdasaryan
al. 2019. Farrand et al. 2020) Ganev al. 2022). For
CelebA, we resize the images to 32 x 32 x 3 (i.c., CelebA(S)) and to 64 x 64 x 3 (i.c.. CelebA(L)) L))
following the conventions in the DP generative model literature (Long et al.. 2021 Wang et al. 202
Cao et al. 202T). We provide more dataset details (e.g., dataset sizes) in Sec. D.1
Bias Settings We create various bias settings across classes and subgroups, focusing on four scenarios: 1) binary class bias, which is a basic scenario often addressed in DP generative models, and 2) multi-class bias, subgroup bias, and unknown subgroup bias, which are more challenging scenarios typically addressed in fairness techniques, but not in DP generative models. We observe that DP generative models mostly perform poorly in these challenging scenarios, especially with complex datasets like CelebA, studies primarily focus on class bias in MNIST (Bagdasaryan et al.) 2019 Farrand et al., 2020). we additionally analyze subgroup bias using image rotation for more fine fine-grand -raignd fairness analyses and to support prominent fairness metrics like equalized odds (Hardt et al. (016)) In all experiments, we denote y = 0 as the minority class and 0 as the minority group. More details on bias levels (e.g.. size ratios between majority and minority data) and bias creation are in Sec. D.I
use MNIST for more reliable analyses. While recent privacy- -firness
Metrics We evaluate utility, privacy, and faimeess in both synthetic data and downstream tasks.
Utility. We measure the overall and groupwise Frechet Inception Distance (FID) (Heusel et al.
2017) to evaluate the sample quality of synthetic data. We evaluate model accuracy in downstream tasks by training Multi-layer Perceptrons (MLP) and Convolutional Neural Networks (CNN) on synthetic data and testing on real datasets (Chen et al. 2020).
7

---

Published as a conference paper at ICLR 2025
Fairness. We measure the group size disparity in synthetic data with the KL divergence to uniform distribution (s)) (i.e.. (s)
(Yu et al. 2020) and the distribution dis- parity (i.e., [pc(s) - (ss)) ) (Choi et al., 2020), where Pa(s) denotes generated distribution w.r.t. s. We measure the faimess disparities in downstream tasks as follows: equalized odds disparity (i.e., maxy.51.52 Prrý-=y\==y s=81)
yl Vy E VV,88 81, 82EE), demographic disparity (i.e., ma Pr(ý= -= Pr Pr(y=11s=82) Vs1, E S), and accuracy disparity (i.e., ma Pr(y y y- P1
Vy1, 1/2 E E)).
Privacy, We use privacy budget E for DP (Def. 2.1) which is preserved in both synthetic data and downstream tasks due to the post-processing property of DP (see Sec. B.1 for more details).
Baselines We compare PFGuard with three types of baselines: 1) privacy-only and fairness-only approaches for data generation, 2) simple combinations of these methods, and 3) recent privacy- fairness classification methods applicable to data generation. For 1) and 2),
three state-of-
the-art DP generative models - GS- DSS-WGAN (Chen et al.] (020). G-PATE (Long et al. (021] )
and
DataLens (Wang et al. 2021a) and widely-adopted fair reweighting method (Choi et al. 2020).
For 3), we extend DP-SGD-F (Xu et al. (020) and
(Esipova et al. 2022).
which are fair variants of DP-SGD (Abadi et al. 2016). Specifically, we replace the DP-SGD used in GS-WGAN with these hhimess-ennannee variants. We faithfully implement all baseline methods with their official codes and reported hyperparameters. More details on bascline methods are in Sec. D.2
5.1 IMPROVING EXISTING MIVACY-ONLY GENERATIVE MODELS
We evaluate how PFGuard enhances the performance of existing DP generative models. As PFGuard guarantees the same level of DP, we focus on the fairness and utility performances while fixing the privacy budget to 1=10, which is one of the most conventional values (Ghalebikesabi et al. 2023).
Analysis on Synthetic Data Table[showw the fairness and utility performances on synthetic data. Private generative models generally produce synthetic data with better overall image quality, but exhibit high group size disparity. In contrast, PFGuard significantly improves fairness by balancing group size and groupwise image quality, with slight decrease in overall image quality.
Table 1: Fairness and utility performances of private generative models with and without PFGuard on synthetic data, evaluated on MNIST with subgroup bias under c=10. Blue and red arrows indicate positive and negative changes, respectively. Lower values are better across all metrics.
Faimess
Utility
Method
KL
Dist. Disp. (1)
FID (1)
Y=I,
Y=1,
Y=0, S=1
Y=0, S=O
GS-WGAN
0.177
0.383
77.97:225
95.58
155.20
89.6618m
101.39
G-PATE
0,305=200
0,522+ame
176.03=3m
182,50+12
183.31+2m
178. 894419
187.37+a51
DataLens
0,220+sueo
0.450
192.29-xxr
197.13
197.99
202.86-412
207.12-1223
GS-WGAN PFGuard
0.067= (1)
0.242 (L)
83.67 #5.6 (*)
114.54=2044
149,47=11.
79.94=
72.44-5M
G-PATE PFGuard
0.206 1000 (L)
0.431 1000 ())
166.89 a (1)
173.482m0n
173.792mm
174.98
185.92_.w
DataLens PFGuard
0,161 20000 (L)
0,389 2002 (L)
200.23 =311 (1) 209.74
20
207.032467
207.05
Analysis on Downstream Tasks Table the fairness and utility performances on downstream tasks. Compared to the synthetic data analysis, PFGuard enhances not only fairness, but also overall utility, especially for CNN models. We suspect that the increased overall utility results from the improved fairness in the input synthetic data, promoting more balanced learning among groups
Table 2: Fairness and utility performances of private generative models with and without PFGuard on downstream tasks, evaluated on MNIST with subgroup bias under s=10. Blue and red arrows indicate positive and negative changes, respectively.
MLP
CNN
Faimess
Utility
Fairness
Utility
Method
FO Disp- (4)
Dem. Disp. (L)
Acc (t)
BO Disp. (L)
Dem. Disp. (L)
Acc
GS-WGAN
1.061
0.910
0.172_noss
0.069 2
0.927
G-PATE
0.1661
0.063
0.896
0.256_now
0.111 anon
0.888
DataLens
0,226
0.112
0.867 -wazs
0.238 -0oss
0.110
0.893 swazz
GS-WGAN + PFGuard
0.067 =uxs ())
0.044 (L)
0.900 ausce (L)
0.063- (L)
0.035 -urr (L)
0.927 =10a0s
G-PATE PFGuar
0.085 2000
0.044 ())
0090 (1)
0.084 2oos (1)
0.044 a0ou (U)
0.898
DataLens PFGuard
0.169 ())
0.106=00xs (L)
0.859 (L)
0.141 =0000 (L)
o.078 (L)
0.898=nos (1)
8

---

Published as a conference paper at ICLR 2025
Table 3: Comparison of
performance on MNIST under E=10, using GS- WGAN as the base DP generator (see Sec. [o2 more details). The first three rows represent upper bound performances for vanilla, DP-only, and fair-only models. Evaluations cover both subgroup bias and unknown subgroup bias, where "no S" indicates whether the method is applicable without group labels. "pere" denotes the proportion of public data used compared to the training data size. "L" indicates no samples are generated. Lower values better across all metrics, and we boldface the best results in each subgroup bias and unknown subgroup bias settings.
Privacy
Fairness
Utility
Method
(U)
KL
Dist. Disp. S
FID
Y=1. S=1
Y=1, S=0
Y=0, S=1
Y=0, S=0
Vanila
X
0.229
0.459
31.95
28.01
55.53
44.04
63,50
DP-only
10
0.177
0,383
77.97
95.58
155.20
89.66
101.39
Fair-only
x
0.021
0.117
38.62
50.78
52.69
75.46
53.86
Reweighting
13
0.044
106.94
139.28
178.18
128.08
110.54
DP-SGD DP-SGD-F
11
0.659
0.494
90.20
121.78
73.07
159.83
DP-SGD DPSGDD-G
10
0.693
0,707
127,02
167.65
126.15
PFGuard
0.067
0.242
83.67
114.54
149.47
79.94
72.24
Reweighting (perc=
0.148
98.57
144.55
182.29
99.59
Reweighting (pere=0.1)
13
0.013
0,113
106.94
139.28
178.18
128.08
110.54
PFGuard (perc=0.1)
0.004
0.041
89.43
130.36
157.80
78.75
89.76
1.0
0:30
Accuracy of cleas
Upper #Tecchers
Disp.
Overall FID
50
UBlttt (Overall FID)
020
66
50
015
70
Baseline Accuracy
50
PFGuard Accuracy
20
0.00
10
0.1
0.05
0.01
50
Blas Level
Reference Dataset Size Ratio
Figure 4: Fairness performances when varying bias levels (γ) given a fixed number of teachers, evaluated on MNIST with multi-class bias. We downsize the class '8' to times smaller than the other classes to make it the minority class and use GS-WGAN as the baseline model.
Figure 5: Fairness and utility performances for varying reference dataset size ratio compared to the training dataset size, evaluated on MNIST with unknown subgroup bias under ==10, 10. Lower values are better across all metrics used evalu- ate fairness and utility.
5.2 PRIVACY-FAIRNESS UTILITY TRADBOFF
We compare our satney--finness-ttllee performance against simple combinations of prior approaches. We evaluate performance under two bias settings: 1) subgroup bias and 2) unknown subgroup bias. Table the results, which aligns with our ffirncsss counteraction analysis in Sec. 3 fairness-only reweighting approaches compromise privacy due to the increased iterations, which may arise from modifying the loss function for fair training (i.e., the more model uses the data, the weaker privacy provides). In comparison, privacy-fairness classification techniques can maintain the original privacy guarantees, but significantly degrade utility and fairness, resulting in lower image quality and size disparities across groups further discussions are provided in Sec. [A] Among all methods, PFGuard is the only method that successfully 1) achieves both privacy and fairness and 2) preserves the closest utility to the original models. In Sec. E. we provide more analyses of the hivacy-fairness: utility tradeoff, including Pareto Frontier results and varying privacy levels.
5.3 ABLATION STUDY
Fairness Upper Bound on Number of Teachers We validate the proposed theoretical upper bound on the number of teachers for fairness, which depends on the bias level of the training data. To effectively simulate scenarios where teacher receives only a small subset of minority data, we evaluate PFGuard a multi-class bias setting, downsizing the minority class (i.e., class '8' for MNIST) by factor of 7 Given that MNIST has fewer than 6,000 samples for class '8', our proposed upper bound is 5 we fix the number of teachers to 1,000. Fig. shows that exceeding y=5 leads to noticeable accuracy drop for the minority class, which is consistent with theoretical results. is noteworthy that even with the decline, PFGuard shows higher accuracy than the privacy-only baseline, which shows consistent decrease in accuracy for the minority as y increases.

---

Published as a conference paper at ICLR 2025
Impact of Reference Dataset Size We explore the influence of the reference dataset size when PFGuard is extended to unknown subgroup bias setting. Fig. 5 shows PFGuard achieves comparable fairness even with small reference dataset size, while showing a slight increase in the overall FID.
More Analyses We provide more experiments in Sec. E including a comparison of computation time (Sec. E.3 results on different datasets (Sec. E.2
performance with additional normalization techniques (Sec. E.4) and a performance comparison with a different sampling strategy (Sec. EE.5
and E.7
5.4 ANALYSIS WITH STRONGER PRIVACY, HIGH-DIMENN DIEEH-IIMNSIONAL IMAGES
We provide preliminary results with CelebA dataset, which mirrors real-world scenarios with high- dimensional facial images. As our study is the first to address both privacy and fairness in image data, this exploration is crucial for understanding the challenges in real- woaldwoldd settings. To reflect the need of stronger privacy protection in practical applications, we limit the privacy budget to 8=1.
Table the fairness and utility performances under these challenging conditions. We observe that DP generative models often exhibit extreme accuracy disparities even with a simplistic class bias setting, achieving over 90% accuracy for the majority class while achieving accuracy below 25% for the minority class. PFGuard consistently enhances the minority class performance and reduces accuracy disparity, while there is still room for improvement. Our results underscore the importance of tackling both privacy and faimess in future studies, encouraging more research in this critical area.
Table 4: Fairness and utility performances of private generative models with and without PFGuard on downstream tasks, evaluated on CelebA with binary class bias under ==1. GS- ES-WGAN is excluded due to lower image quality in this setting. Blue and red arrows indicate positive and negative changes, respectively. We provide the full results with standard deviations in Sec. E.8
CelebA(S)
CelehA(I.)
Fairness
Utility
Faimess
Utility
Method
Acc. Disp. (1)
Acc (t)
Y=0
Acc. Disp. (1)
Acc (t)
Y=0
G-PATE DataLens
0.978
0.666
0.014
0.992
0.968
0.668
0.023
0.991
0.793
0.643
0.114
0.907
0.678
0.686
0.234
0.912
G-PATE PFGuard
0.736 (L) 0.725 (.)
0.678 (?)
0.187 (1)
0.923 (L)
0.277 (L)
0.563 (
0,378 (t)
0.655 ())
DataLens PFGuard
0.689 (?)
0.205 (1)
0.931 (1)
0.641 (L)
0,704 (f)
0.276 (f)
0.917 (1)
RELATED WORK
We cover the private and fair data generation literature here and cover the 1) private-only data generation, 2) fair-only data generation, 3) privacy-fairness intersection literature in Sec. F Compared to these lines of works, only few works focus on private and fair data generation (Xu al. 202T Pujol et al. 2022). First, (Xu et al. (021]) proposes a two-step approach that 1) removes bias from the training data via fair pre-processing technique (Celis et al. 2020) and 2) learns a DP generative model (Chanyaswad et al., 2019) from the debiased data. However, this framework is limited to low-dimensional structural data due data binarization step in pre-precee preppreeessing stage, which can incur significant information loss in high-dimensional image data. PFGuard, on the other hand, can generate high-dimensional image data with high quality. Second, (Pujol et al. data generation techniques satisfying causality-basee fairness (Salimi et al. 2019) which consider the causal relationship between attributes. In comparison, PFGuard focuses on statistical fairness to achieve similar model performances for sensitive groups (Barocas et al. 2018 While -aased approaches can better reveal the causes of discrimination than statistical approaches, modeling an underlying causal mechanism for real-world scenarios is also known to be challenging.
2022) proposes private
7 CONCLUSION
We proposed PFGuard, a fair and private generative model training framework. We first identified the counteractive nature between privacy preservation and fair training, demonstrating potential adverse effects such privacy breaches fairness cancellation when two objectives are addressed independently. We then designed PFGuard, which prevents the counteractions by using multiple teachers to harmonize fair sampling and private teacher ensemble learning. We showed how this integrated design of PFGuard offers multiple advantages, including a better fairness prinness--rvvvy-uuee utility tradeoff compared to other baselines, ease of deployment, and support for high-dimensional data.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT & LIMITATION
We believe our research addresses the critical issue of Trustworthy AI Our focus on privacy and fairness underscores the need to design AI models that simultaneously safeguard individual privacy and mitigate biases in training data, addition, our research and experiments are conducted with strong commitment to ethical standards. All datasets used in this study, including publicly available human images, are widely used within the research community and do not contain sensitive or harmful content. There are also limitations, and note that choosing the right privacy and fairness measures for an application can be challenging and also depends on the social context. We also note that the use of multiple teacher does increase the cost of training, but can provide more benefits particularly in balancing privacy and fairness.
REPRODUCIBILITY STATEMENT
All datasets, methodologies, and experimental setups used in our study are described in detail in the supplementary material. More specifically, we provide a description of the proposed algorithm in Sec. C.2 details of datasets and preprocessing in Sec. D.I and implementation details including hyperparameters in Sec. D.2 to ensure reproducibility.
ACKNOWLEDGMENTS
This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2024-00509255, Global AI Frontier Lab). This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea govermmen(MMSTT)MM BK21 (Conneceee AI Education & Research Program for Industry and Society Innovation, KAIST EE, No. 4 4120200113769).
RSS2002-NN0070121,, This work was supported by the
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Sushant Agarwal. Trade-offs between fairness and privacy in machine learning. In IJCAI 2021 Workshop on for Social Good, 2021.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. Advances neural information processing systems, 32, 2019.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness in machine learning. Nips tutorial, 1: 2, 2017.
Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and machine learning. fairmlbook. org, 2019, 2018.
Alex Bie, Gautam Kamath, and Guojun Zhang. arXiv:2302.02936, 2023,
Private gans, revisited.
arXiv preprint
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Zhiqi Bu, Yu- Yu-Xiang Wang, Sheng Zha, and George Karypis. Automatic clipping: Differentially private deep learning made easier and stronger. Advances in Neural Information Processing Systems, 36, 2024.
Blake Bullwinkel, Kristen Grabarz, Lily Ke, Scarlett Gong, Chris Tanner, and Joshua Allen. Evaluat- ing the fairness impact of differentially private synthetic data. arXiv preprint arXiv:2205.04321, 2022.
11

---

Published as a conference paper at ICLR 2025
Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, and Karsten Kreis. Don't generate me: Training differentially private generative models with sinkhorn divergence. Advances in Neural Information Processing Systems, 34:12480-1248 2021.
Elisa Celis, Vijay Keswani, and Nisheeth Vishnoi. Data preprocessing to mitigate bias: A maximum entropy based approach. In International Conference Machine Learning, pp. 1349-1359, PMLR, 2020.
Thee Chanyaswad, Changchang Liu, and Prateek Mittal. Ron-gauss: Enhancing utility in non- interactive private data release. Proceedings on Privacy Enhancing Technologies, 2019(1):26-46 2019.
Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. Gs-wgan: A cradient-sanitizen approach for learning differentially private generators. Advances in Neural Information Processing Systems, 33:12673-12684 2020.
Jia-Wei Chen, Chia-Mu Yu, Ching- Ching-Chia Kao, Tzai- Wei Pang, and Chun-Shien Lu. Dpgen: Diffeeren- tially private generative energy-guided network for natural image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8387-8396, 2022.
Victoria Cheng, Vinith M Suriyakumar, Natalie Dullerud, Shalmali Joshi, and Marzych Ghassemi. Can you fake it until you make it? impacts of differentially private synthetic data downstream classification fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pp. 149-160, 2021.
Kristy Choi, Aditya Grover, Trisha Singh, Rui Shu, and Stefano Ermon, Fair generative modeling via weak supervision. In International Conference Machine Learning, pp. 1887-1898. PMLR, 2020.
Rachel Cummings, Varun Gupta, Dhamma Kimpara, and Jamie Morgenstemn. On the compatibility of privacy and fairness. In Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization, pp. 309-315, 2019.
Tim Dockhorn, Tianshi Cao, Arash Vahdat, and Karsten Kreis. Differentially private diffusion models. arXiv preprint arXi::2210.09929, 2022.
Akshay Dodwadmath and Sebastian Stich. Preserving privacy with pate for heterogeneous data. 2022.
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Advances in Cryptology- EeppoooogyEEUROCKYPPT 2006: 24th Annual International Conference on the Theory and Applications of Cryptographic Techniques, St. Petersburg, Russia, May 28-June 1, 2006. Proceedings 25, pp. 486-503. Springer, 2006a.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptogruphy: Third Theory of Cryplography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp. 265-2844 Springer, 2006b.
Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theoretical Computer Science, 9(3-4):211-00 2014.
Maria S Esipova, Atiych Ashari Ghomi, Yaqiao Luo, and Jesse Cresswell. Disparate impact in differential privacy from gradient misalignment. arXiv preprinl arXiv:2206.0775 2022.
Tom Farrand, Fatemehsadat Mireshghallah, Sahib Singh, and Andrew Trask. Neither private nor fair: Impact of data imbalance utility and faimesss in differential privacy. In Proceedings of the 2020 workshop on privacy-preserving machine learning in practice, pp. 15-19, 2020.
Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubra- manian. Certifying and removing disparate impact. In proceedings of the 21th ACM SIGKDD international conference knowledge discovery and data mining, pp. 259-268, 2015.
12

---

Published as a conference paper at ICLR 2025
Germain Forestier and Cédric Wemmert. Semi-supervised learning using multiple clusterings with limited labeled data. Information Sciences, 361:48 65, 2016.
Georgi Ganev, Bristena Oprisanu, and Emiliano De Cristofaro. Robin hood and matthew effects: Differential privacy has disparate impact on synthetic data. In International Conference on Machine Learning, pp. 6944-6959, PMLR, 2022.
Sahra Ghalebikesabi, Leonard Berrada, Sven Gowal, Ira Ktena, Robert Stanforth, Jamie Hayes, Soham De, Samuel L Smith, Olivia Wiles, and Borja Balle. Differentially private diffusion models generate useful synthetic images. arXiv preprint 1 13861, 2023.
Ian Goodfellow, Jean Pouget-Abadic,, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proceceding of the Advances in Neural Information Processing Systems, 2014.
Frederik Harder, Kamil Adamczewski, and Mijung Park. Dp-merf: Differentially private mean embeddings with randomfeatures for practical privacy-preserving data generation. In International conference on artificial intelligence and statistics, pp. 1819-1827. PMLR, 2021.
Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning, Advances in neural information processing systems, 29, 2016.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter, Gans trained by a two scael update rule converge to a local nash equilibrium. Advances Neural Information Processing Systems, 30, 2017.
Benjamin Hilprecht, Martin Härterich, and Daniel Bernau. Monte carlo and reconstruction mem- bership inference attacks against generative models. Proc. Priv. Enhancing Technol., 2019(4): 232-249, 2019.
Daniel G Horvitz and Donovan Thompson. A generalization of sampling without replacement from finite universe. Journal of the American statistical Association, 47(260):663-688 1952.
Ahmed Imtiaz Humayun, Randall Balestriero, and Richard Baraniuk. Magnet: Uniform sampling from deep generative network manifolds without retraining. In International Conference on Learning Representations, 2021.
Matthew Jagielski, Michael Kearns, Jieming Mao, Alina Oprea, Aaron Roth, Saeed Sharifi- Maaiiii-aalaajrrdi. and Jonathan Ullman. Differentially private fair learning. In International Conference on Machine Learning, pp. 3000-3008. PMLR, 2019.
James Jordon, Jinsung Yoon, and Mihaela Van Der Schaar. Pate-gan: Generating synthetic data with differential privacy guarantees, In International conference on learning representations, 2018.
Matthew Joseph, Michael Kearns, Jamie H Morgenstem, and Aaron Roth. Fairness in learning:
Classic and contextual bandits. Advances in neural information processing systems, 29, 2016.
Yeongmin Kim, Byeonghu Na, Minsang Park, JoonHo Jang, Dongjun Kim, Wanmo Kang, and II-Chul Moon. Training unbiased diffusion models from biased dataset. arXiv preprint 188.0 2024.
Ron Kohavi. Scaling up the accuracy of naive-bayes classifiers: a decision-tree hybrid. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, pp. appear, 1996.
Bogdan Kulynych, Yao- Yuan Yang, Yaodong Yu, Jaroslaw Blasiok, and Preetum Nakkiran. What you see what you get: Principled deep learning via distributional generalization, Advances in Neural Information Processing Systems, 35:2168 2183, 2022.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11))2278-233) 1998.
Seng Pei Liew, Tsubasa Takahashi, and Michihiko Ueno. Pearl: Data synthesis via private embeddings and adversarial reconstruction learning. arXiv preprint arXiv: 066.0499, 2021.
13

---

Published as a conference paper at ICLR 2025
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Yunhui Long, Boxin Wang, Zhuolin Yang, Bhavya Kailkhura, Aston Zhang, Carl Gunter, and
Bo Li. G-pate: Scalable differentially private data generator via private aggregation of teacher discriminators. Advances in Neural Information Processing Systems, 34:2965 -977, 2021.
Andrew Lowy, Devansh Gupta, and Meisam Razaviyayn. Stochastic differentially private and fair learning. In Workshop on Algorithmic Fairness through the Lens of Causality and Privacy, pp. 86-119. PMLR, 2023.
J Macqueen. Some methods for classification and analysis of multivariate observations. In Pro- ceedings of 5-th Berkeley Symposium on Mathematical Statistics and ProbabilityUniivrsity of California Press, 1967.
Xiangrui Meng. Scalable simple random sampling and stratified sampling. International conference on machine learning, pp. 531-539. PMLR, 2013.
Ily Mironov. Rényi differential privacy. In CSF, pp. 263-275. IEEE, 2017.
Sangwoo Mo, Chiheon Kim, Sungwoong Kim, Minsu Cho, and Jinwoo Shin. Mining gold samples for conditional gans. Advances in Neural Information Processing Systems. 32, 2019.
Hussein Mozannar, Mesrob Ohannessian, and Nathan Srebro. Fair learning with private demographic data. In International Conference on Machine Learning, pp. 7066-7075. PMLR, 2020.
Nicolas Papernot, Martín Abadi, Ulfar Erlingsson, Ian Goodfellow, and Kunal Talwar. Semi- supervised knowledge transfer for deep learning from private training data. arXiv preprint arXiv:1610.05755, 2016.
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Úlfar Erlingsson. Scalable private learning with pate. arXiv preprint /82.00880200800 2018
David Pujol, Ryan McKenna, Satya Kuppam, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. Fair decision making using privac-protected data. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 189-199, 2020.
David Pujol, Amir Gilad, and Ashwin Machanavaijhala. Prefair: Privately generating justifiably fair synthetic data. arXiv preprint a2Xiv:2212.1033: 2022.
Yuji Roh, Weili Nie, De-An Huang, Steven Euijong Whang, Arash Vahdat, and Anima Anandkumar. Dr-fairness: Dynamic data ratio adjustment for fair training on real and generated data. Transactions on Machine Learning Research, 2023.
Lucas Rosenblatt, Julia Stoyanovich, and Christopher Musco. A simple and practical method for reducing the disparate impact of differential privacy. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 21554-21562, 2024.
Donald B Rubin. Using the sir algorithm to simulate posterior distribution. Bayesian statistics, 3: 395 402, 1988.
Babak Salimi, Luke Rodriguez, Bill Howe, and Dan Suciu. Interventional fairness: Causal database re- pair for algorithmic faireess In Proceedings of the 2019 International Conference Management of Data, pp. 133-810,22 2019. 2019
Amartya Sanyal, Yaxi Hu, and Fanny Yang. How unfair is private learning? In Uncertainty in Artificial Intelligence, pp. 1738-1748. PMLR, 2022.
Prasanna Sattigeri, Samuel C Hoffman, Vijil Chenthamarakshan, and Kush R Varshney. Fairness gan: Generating datasets with fairness properties using a generative adversarial network. IBM Journal of Research and Development, 63(4/5):3-1, 2019..
Øivind Skare, Erik Bølviken, and Lars Holden. Improved sampling-importanee resampling and reduced bias importance sampling. Scandinavian Journal of Statistics, 30(4):719-737, 2003.
14

---

Published as a conference paper at ICLR 2025
Adrian FM Smith and Alan E Gelfand. Bayesian statistics without tears: sampling resampling perspective. The American Statistician, 46(2):84-88) 88, 1992.
Hui Sun, Tianqing Zhu, Zhiqiu Zhang, Dawei Jin, Ping Xiong, and Wanlei Zhou. Adversarial attacks against deep generative models on data: a survey. IEEE Trunsactions on Knowledge and Data Engineering, 35(4):3367-3388 2021.
Shun Takagi, Tsubasa Takahashi, Yang Cao, and Masatoshi Yoshikawa. P3gm: Private high- dimensional data release via privacy preserving phased generative model. In 2021 IEEE 37th International Conference on Data Engineering (ICDE), pp. 169-180. 180. IEEE, 2021.
Shuhan Tan, Yujun Shen, and Bolei Zhou. Improving the faimess of deep generative models without retraining. arXiv preprint arXiv:2012.04842, 2020.
Christopher TH Teo, Milad Abdollahzadeh, and Ngai-Man Cheung. Fair generative models via transfer learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 2429-2433 2023.
Reihaneh Torkzade Peter Kairouz, and Benedict Paten. Dp-cgan: Differentially private synthetic data and label generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition (CVPR) Workshops, June 2019.
Cuong Tran, My Dinh, and Ferdinando Fioretto. Differentially private empirical risk minimization under the fairness lens. Advances in Neural Information Processing Systems, 34:27555-27555, 2021a.
Cuong Tran, Ferdinando Fioretto, and Pascal Van Hentenryck. Differentially private and fair deep learning: A lagrangian dual approach. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 9932-9939, 2021b.
Cuong Tran, Keyu Zhu, Ferdinando Fioretto, and Pascal Van Hentenryck. Sf-pate: scalable, fair, and private aggregation teacher ensembles. arXiv preprint arXiv:2204.05157, 2022.
Soobin Um and Changho Suh. A fair generative model using lecam divergence. In Proceedings of the AAAI Conference on Artificial Intellignnce, volume 37, pp. 10034 2023.
Margarita Vinaroz, Mohammad-Amin Charusaie, Frederik Harder, Kamil Adamczewski, and Mi Jung Park. Hermite polynomial features for private data generation. In International Conference Machine Learning, pp. 22300-22324. PMLR, 2022.
Boxin Wang, Fan Wu, Yunhui Long, Luka Rimanic, Ce Zhang, and Bo Li. Datalens: Scalable privacy preserving training via gradient compression and aggregation. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pp. 2146 2168, 2021a.
Zhengwei Wang. Qi She, and Tomas Ward. Generative adversarial networks in computer vision: survey and taxonomy. ACM Computing Surveys (CSUR), 54(2):1-38, 2021b.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, arXiv preprint arXiv:1708.07747, 2017.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. arXiv preprint arXiv: 180..0633..99 2018.
Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. Fairgan: Fairess--wwaee generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data), pp. 570-575, IEEE, 2018.
Depeng Xu, Shuhan Yuan, and Xintao Wu. Achieving differential privacy and fairness in logistic regression. In Companion proceedings of The 2019 world wide web conference, pp. 594-599, 2019.
Depeng Xu, Wei Du, and Xintao Wu. Removing disparate impact of differentially private stochastic gradient descent on model accuracy. arXiv preprint arXiv:2003.088 200999099...,9 2020.
15

---

Published as a conference paper at ICLR 2025
Weijie Xu, Jinjin Zhao, Francis lannacci, and Bo Wang. Ffpdg: Fast, fair and private data generation. In ICLR 2021 Workshop on Synthetic Data Generation, 2021.
Mohammad Yaghini, Patty Liu, Franziska Boenisch, and Nicolas Papernot. Learning with impartiality to walk on the pareto frontier of fairness, privacy, and utility. arXiv preprint arXiv:2302.09) 09133, 2023.
Yilin Yang, Kamil Adamczewski, Danica J Sutherland, Xiaoxiao Li, and Mijung Park. Differ- entially private neural tangent kernels for grivacy-preserving data generation. arXiv preprint arXiv:2303.01667, 2023.
Ning Yu, Ke Li, Peng Zhou, Jitendra Malik, Larry Davis, and Mario Fritz. Inclusive gan: Improving data and minority coverage in generative models. European Conference on Computer Vision, pp. 377-393. Springer, 2020.
Xinyang Zhang, Shouling Ji, and Ting Wang. Differentially private releasing via deep generative model (technical report), arXiv preprint arXiv:1801.01594 2018.
Shengjia Zhao, Hongyu Ren, Arianna Yuan, Jiaming Song, Noah Goodman, and Stefano Ermon. Bias and generalization deep generative models: An empirical study. Advances in Neural Information Processing Systems, 31, 2018.
D. Zietlow, M. Lohaus, G. Balakrishnan, M. Kleindessner, F. Locatello, B. Scholkopf, and C. Russell. Leveling down in computer vision: Pareto inefficiencies in fair deep classifiers. 2022 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10400-10411, 2022.
16

---

Published as a conference paper at ICLR 2025
A CHALLENGES OF EXTENDING CLASSIHICATION TECHNIQUES
Continuing from Sec. we provide more details of potential challenges when tries extend
fairness-privant classification techniques (Jagielski et al. 2019 Mozannar et al. 2020 Tran et al.
202 2022)
generative settings due the fundamentally different goals.
Different DP Notions and Assumptions Classification and generation settings often concentrate on different DP notions rely on different assumptions, which hinders simple extensions of techniques between them. In the classification setting, Differential Privacy w.r.t. sensitive altribute (Jagielski
2019) is commonly addressed (Jagiclski al. 2019 Mozannar al. [0200] Tran al. 2021b 2022). which considers the demographic group attribute as the only private information.
This DP notion requires less DP noise compared to a more general notion of DP (Def. 2.1 which
protects all input features, and enables a better privacy-utility tradeoff for DP classifiers. However, in the generative setting, general notion of DP is mostly addressed, as presumed non-private features may in fact encode private information (e.g., pixel values in facial image). Therefore, simply extending classification techniques to generative settings can be challenging, as it necessitates rigorous mathematical proofs for corresponding DP notions and may add large DP noise when adapting to a general DP notion. Moreover, classification techniques can rely on convex objective functions Tran et al.) [2021a), but the assumption of convexity does not usually hold in generative models (Goodfellow et 2014).
Challenges of Adjusting Privacy Bound While recent studies have proposed fair variants of DP-SGD (Xu et al. 2020 Esipova et al. 2022). directly adopting them in existing private generative models can undermine the original utility and privacy guarantee. To prevent aggressive gradient clipping in minority data groups, approaches to tune clipping threshold C' during training have been proposed, such dynamically adjusting C during training (Esipova al., (022) or utilizing different C' values w.r.t. groups (Xu et al. 2020 However, these adjustments of C not only consume additional privacy budget, but also can significantly affect model utility, as private generative models often demonstrate high sensitivity in model convergence to these clipping values (Chen et al. [2000] Wang al., 2021a Dockhorn et al. 2022). Hence, given the limited privacy budget and the necessity to carefully set the value of C, these approaches of tuning C may drastically change the original privacy-utility tradeoff of existing models to achieve fairness.
Here, we do not claim that extending classification techniques to generative settings is always impossible, but introduce the challenges that can complicate such extensions. To exemplify some possible cases, we extend classification methods from Esipova et (2022) and (Xu et al. (2020) to generative settings, using them as baselines in our experiments - see Sec. 5 for results.
B DIFFERENTIAL PRIVACY
Continuing from Sec. 2 and Sec. 4.2 we provide more details on Differential Privacy (DP).
B.1 POST-PROCESSING PROPERTY OF DIFFERENTIAL PRIVACY
Continuing from Sec. 2 we detail the post-processing property of DP as defined below.
Theorem B.1. (Post- peest--rccssing (Dwork et al. 2014)) Let M > R2 be a randomized mechanism
that is (c. DD.. Let R, R2 be arbitrary function. Then f o M : D R2 is (s, DD.P.
Due to the above pos--procsssing theorem, synthetic dataset D = G(z) generated from an (s, DDP generator G with random noise z E also satisfies (c, BDDD, as the random noise z is independent of the private training dataset used for the DP generator.
B.2 SENSITIVITY ANALYSIS OF GNMAX AGGREGATOR
Continuing from Sec. 4.2 we echo the sensitivity analysis of GNMax aggregator provided by Papernot et al. (2018) for readers' convenience.
17

---

Published as a conference paper at ICLR 2025
Given {T teachers, possible label classes, and query input x, the teachers' vote counts for the j-th class to a query imput x is denoted as:
ny (x) 1ft (x) /}l for j = 1,...c
where 7i denotes the i-th teacher model. The vote count for each class is aggregated as follows:
n n(x) = (ri1,..., no) E N
Since a single training data point only affects most one teacher due data disjointness. changing one data sample will at most change the votes by I for two classes, where we denote here as classes i and without loss of generality. Given the two adjacent datasets D, 2) which differ by a single data point, let the aggregated vote counts be n (M1,...,, Me) and n' (n/ m2),, respectively. The 12 ig-sensitivity (Def. 2.2) can be derived as follows:
A max (M1....,, ne) (n/,-.. n2)ll2
(9)
D,D
max m...n
[I(O,....0., 0,, - n44 ,0....0,, ny - n .,,,,,0))ll
(10)
max mu.nk.n..n
(n, - n2) 9 (n, - -n3)² S v2
(11)
C PFGUARD FRAMEWORK
Continuing from Sec. 4.2
we provide more details on the proposed PFGuard framework.
C.I PRIVACY ANLAYSIS OF PFGUARD
we provide a theoretical proof of how PFGuard preserve the sensitivity of an arbitrary PTEL mechanism, and thus the sensitivity of an arbitrary PTEL-based generative model.
Continuing from Sec. 4.2
PTEL-based Generative Models We first characterize the common training scheme of PTEL- based generative models (Jordon et al. 2018 Chen et al., 2020; Long et al. 2021 Wang et al. 202 To train a DP generator G parametrized with Ug. the goal is to make its training algorithm A + which inputs the training dataset D and outputs the generator G - satisfy DP (Long al. 2021) Given the typical training scheme of stochastic gradient descent, which updates tG with up' the training algorithm satisfies DP if up is privatized with a DP mechanism due the post-processing property of DP (Thm. [B.I) (Abadi et al. 2016). To produce privatized gradient up STEELbased generative models typically proceed in two steps as follows:
gradient information 9G:
(1) Training teacher ensemble from disjoint data subsets. The private dataset D is first divided into nT disjoint subsets {D,) where each subset D; is uniquely used to train a teacher model T. with its parameter Or,- Thus, a total of roy teachers are trained and form a teacher ensemble
T Rr.
(2) Training generator by querying teacher ensemble. The target generator G with the parameter 0g trained by interacting solely with the teacher ensemble T, without accessing the original dataset D. Given random input noise z, the generator G generates output Giz) and queries the teacher ensemble T' with G(z) for their supervision. Each teacher in votes on the gradient g6 for the query input G(z), resulting in vote count nG)) This vote count is processed by the PTEL mechanism M (e.g., GNMax aggregator in Sec. 4.2) to finally produce sanitize uup
gradient - The generator G updates its parameter Ocz using 9G
Thus, given PTEL mechanism M with (s, 5DD guarantee for each iteration and a total of N training iterations, the total DP guarantee of the training algorithm A can be computed via the composition theorem of DP (Dwork et al. (0066)).
Sensitivity Preservation We now provide a theoretical proof how training with PFGuard preserves the original privacy analysis of an arbitrary PTEL-based generative model G' as long as data disjointness is enforced. As explained above, the privacy analysis of the PTEL-based generative model depends on the 1) (E, 8D- guarantee of the given PTEL mechanism M for each training iteration and the 2) total number of training iterations N. Here, we focus more on the theoretical DP guarantee than the practical DP guarantee, we assume that N is preserved with the fair sampling
18

---

Published as a conference paper at ICLR 2025
algorithm Sample(-) used in PFGuard. That means, if M and Mo Sample- PTEL mechanisms without and with PFGuard - result in the same (z, 6)DD guarantee for each iteration, the same total show how M and M Sample result in the same
DP guarantee is preserved. In particular, sensitivity (Def. 2.2). which ensures the same (e, 6) values given the same amount of DP noise.
count n(x) from the teacher ensemble T given generated sample as query input (i.e., X = G()),, ) we can denote n(x) = [x; T). Let Ar be the training algorithm of T, where Ar();) T; and Ar(D)
Since the PTEL mechanism M operates
T { due data disjointness used in training of Wr teachers. Since Samplel() - the proposed minibatch sampling algorithm operates on Ar and samples a data subset B; S D, data disjointness is preserved in B, We thus can derive Ar ( (ample(D,)) = (B,) = Te, and Ar (ampleeDD) =T = {T resulting in different teacher ensemble due to changed minibatches.
Let the original sensitivity value over nx:; T) nx;; (r)) be k. Then, due to the definition of the sensitivity, the following holds:
A = max
Ar Ar(()) (D)) n(;; Ar 1212
(12)
D,D
max
[x; Ti ..... Tur - Tap) nxx; Ti..... Taz T 1)12
(13)
D,D
max (M1...... no) . n221l2
(14)
D.D'
max Ri..
II(O..... ..... 0, n2,,0,.. ....,...... n 0).
(15)
- V(n,- n2)² + (n-- n²)²² s k Ro
max
(16)
where the second equality follows from 1) adjacent datasets D and D' differing by single data sample and 2) data disjointness, where a single data sample can affect particular teacher that receives the data partition including the data sample); we denote Top as the affected teacher without loss of generality. The last equality denotes how much one teacher can maximally contribute to the PTEL voting scheme, captured by the sensitivity value.
We now compute the sensitivity value over n T,) = nx; Ar (Sample(D))) as follows:
2 = max D.D
|| X; Ar A (Sample - Ar (rSapple((()))
(17)
max D,D'
|n x Te,1, -1, T nxx; Te, Te T s. 1)12
(18)
max (1 [1( -011,..,11 v1) (1/, .., v.,) l2
(19)
D,D
max [I(...., 0, 1/g 0, ..... 0, 1/g. ....0....0) 0)|l2
(20)
= max of.
V(u: u02 + (v; - 2/)2 1psh k
(21)
Here, the second equality follows similarly from adjacent datasets and data disjointness, where denote Terur as the affected teacher. Also, the last equality again captures how much one teacher maximally contribute, which is property of PTEL's voting scheme and is independent of the sampling algorithm- thus being k. As a result, using the sampling technique in PFGuard preserves the sensitivity value long as the PTEL mechanism enforces data disjointness, although the sampling results in a different teacher ensemble compared to the original PTEI -based generative model. We note that this sensitivity analysis remains valid when B includes duplicate samples due to potential oversampling because it still affects one teacher due to data disjointness.
C.2 TRAINING ALGORITHM
Continuing from Sec. we provide the pseudocode describing the full training algorithm of PFGuard when integrated into PTEL-based generative model.
As shown in Algorithm I PFGuard requires only a modification to the minibatch sampling process during teacher model training achieve faimess (Line 5). Despite this modification, PFGuard
19

---

Published as a conference paper at ICLR 2025
Algorithm 1 Integrating PFGuard with PTEL-based generative models
Input Training dataset D. ensemble of teacher model T = {Ti with each parameters θr,, batch size B, PTEL mechanism M(; 7"), teacher loss function Lr, generator loss function La
Output Differentially private generator G with parameters fG, total privacy cost
1: Divide the dataset 'D into subsets {D:)
2: for each training epoch do
3: Il/Phase 1: Fair Training
4:
for each teacher model T; do
5:
Draw minibatch fx, ) S DD with sampling ratio w(x) x h(x) using Eq. 5
6:
Draw set of random noise {z:},=] from input random noise distribution of G Update teacher model Т, with LT (0r,; G(z; 0c))
7:
8:
end for
9:
Il/Phase Private Training
10:
Draw set of random noise {z from input random noise distribution P. of G Generate synthetic data samples G(z; 0c)
11:
12:
Aggregate teacher output with PTEL mechanism ō (()
13:
where the voting on gradient of (G(6))
14:
Update generator model G' with ō Accumulate privacy cost E
15:
16:
end for
17:
return Generator G. privacy cost e
preserves the original privacy guarantee of the given PTEL- generative model, as long as its sensitivity analysis relies on data disjointness see Sec. C.I for more details.
C.3 EXTENSION
Continuing from Sec. 4.2 provide more details on the extensibility of PFGuard.
Setting Number of Teachers without Sensitive Attribute Labels
We discuss how to extend the
proposed upper bound on the number of teachers (i.e., [ min Phins (s)]) when the sensitive attribute label s is unavailable. Note that the proposed bound does not rely on full knowledge of Prins (x) for data points be generated (e.g., images), but instead relies on the distribution Poias (s) w.r.t. sensitive attributes. Thus, we can estimate the sensitive attribute distribution in the training data using traditional techniques such as K-means clustering (Macqueen, 1967) or random subset labeling (Forestier & Wemmert (016) and use the estimated distribution in the proposed bound. We note that while these estimations can be effective, they may introduce errors or additional overhead, such increased computational time.
Integration with Additional Fairness Techniques PFGuard supports additional integrations of existing faimeess methods if they meet two conditions: 1) they apply exclusively to teacher models, ensuring no direct impact on the target generator, and 2) they maintain data disjointness each sample affects only one teacher which is fundamental requirement for PTEL's privacy guarantees (see Sec. 4.2 for details). For example, GOLD (Mo et al. 2019). a fairness method designed to support Rawlsian Max-Min fairness (Joseph et al.) 2016), is compatible with PFGuard. GOLD computes log density ratio estimates to identify worst-group samples and reweight the discriminator (i.e., teacher) loss to improve performance on these samples achieve Rawlsian Max-Min fairness. Since this reweighting is applied solely to teacher models and preserves data disjointness, PFGuard with GOLD maintains the original privacy guarantees of the underlying PTEL generator see tradeoff when additionally applying GOLD in Sec. E.6
Synergy with Adversarial Training While PFGuard's training scheme loss-agnostic and does not necessarily require adversarial training, PFGuard can be particularly synergistic with adversarial training. We discuss with two reasons: (1) the min-max optimization and (2) the assumption of an optimal discriminator in adversarial training. First, adversarial learning optimizes conflicting goals (i.e.. the min-max optimization) using the two components of the generator and the discriminator. Since privacy and fairness also have conflicting goals, assigning them to separate components
20

---

Published as a conference paper at ICLR 2025
may better balance these goals. Additionally, since the convergence of adversarial training depends heavily on the discriminator's 's optimality (Goodfellow et al. 2014), assigning faimess specifically to the discriminator may effectively guide the generator toward optimal state that is also fair.
D EXPERIMENTAL SETTINGS
Continuing from Sec. we provide more details on experimental settings. In all experiments, we use PyTorch and perform experiments using NVIDIA Quadro RTX 8000 GPUs. Also, we repeat all experiments 10 times and report the mean and standard deviation of the top results. The reason we report the top-3 results to favor the simple privacy-fairnes baselines (e.g.. "Reweighting" in Table 3). which we observe to fail frequently. We compare their best performances with PFGuard.
D.1 DATASETS AND BIAS SETTINGS
Continuing from Sec. 5 we provide more details on datasets. We use three datasets: MNIST (LeCun al., 1998). FashionMNIST (Xiao et al. 2017), and CelebA (Liu et al. 2013). MNIST and FashionMNIST/ contain grayscale images with 28 28 pixels and 10 classes. Both datasets have
60,000 training examples and 10,000 testing examples. CelebA contains 202,599 celebrity face images. We use the official preprocessed version with face alignment and follow the official training and testing partition (Liu et al., 2013). We mainly use image datasets instead of the traditional smaller tabular benchmarks for fairness because goal is to make PFGuard work on higher dimensional data such as images. Additional results on tabular datasets are provided in Sec. E.2
MNIST & Fashion We create four bias scenarios across classes and subgroups follows.
Binary Class Bias. For MNIST, we set digit "3" as the majority class = 1 and "I" as the minority class y For FasionMNIST,, we set "Sneakers" as y and "Trousers" as y For each class pair, we select two classes that share the fewest false negatives and thus can be considered independent, following the convention of prior approaches (Bagdasaryan et al. 2019) Farrand et al. 22200] Ganev et al. 2022 We set bias level as 2, which means the minority class y 0 is 2 times smaller than the majority class y = 1. After creating bias, we apply random affine transformations to augment the datasets to match the original dataset size.
Multi-class Bias. We set "8" as the minority class y=0 and reduce its size while maintaining the size of the other 9 classes, following the aforementioned prior approaches. We vary the bias level from I to 10.
Subgroup Bias. For both MNIST and FashionMNIST datasets, we use image rotation to define subgroups. We set non-rotated images the majority group 1 and rotated image as the minority group colors, but observed that the other options often show the adverse affect of making the images noisier and thus reducing the model accuracy unnecessarily. The rotation also allows for simple and effective verification of subgroup labels generated synthetic data by comparing the mean values of synthetic image vectors to the centroids of real image vectors. To validate this heuristic, we compared the results with 400 manually labeled images from each baseline model and observed high accuracy (e.g., 96.5% for MNIST).
O. We also considered other options including adding lines and changing
Unknown Subgroup Bias. In the previous subgroup bias setting, s labels are not used during model training and only used for evaluation purposes after training.
CelebA We create binary class bias using gender attributes, where we set female and male images as y and y O, respectively. As discussed in the main text, DP generative models often show low performance on CelebA in challenging bias scenarios like multi-class bias, which can hinder the reliability of fairness analyses (e.g., a random generator achieves perfect fairness by outputting random images regardless of data groups). Notably, we show that DP generative models can produce highly biased synthetic data even in this simple binary class bias setting (Table 4).
D.2 BASELINES
Continuing from Sec. we provide more details baseline approaches used in our experiments.
21

---

Published as a conference paper at ICLR 2025
DP Generative Models We use three state-of-the-ard PTEL-bbsed generative models: GS- WGAN (Chen et al., 2020). G-PATE (Long et al. 2021). and DataLens (Wang et al. (021a). For all models, we refer to their official Github codes implement their models and to use their best-perforning hyperparameters for MNIST, FashionMNISST, and CelebA.
GS-WGAN. GS- ES-WGAN is widely used in our experiments, as it leverages both PTEL and DP-SGD (Abadi et al. (016) to ensure DP and thus allows for various integration with other techniques. Their approach first trains multiple teacher ensemble and considers one teacher as the representative of the other teachers. Output of the representative teacher (i.e., gradients) is then sanitized with DP-SGD based mechanism to train a DP generator. Compared to DP-SGD which operates on the whole minibatch, their DP mechanism operates on each data sample and composition of B Gaussian mechanism where B is the minibatch size. Our fair sampling preserves their sensitivity analyses despite the potential oversampling, as it does not change the sensitivity of each Gaussian mechanism one input data (i.e., 2C due to the triangle inequality).
thus can be considered
G-PATE and DataLens. G-PATE and Datalens leverage teachers' votes on intermediate gradients update the generator. To sanitize these teachers' votes with DP mechanisms, G-PATE uses random projection and a gradient discretization, while DataLens uses top-k stochastic sign quan- tization. Our fair sampling preserves their sensitivity analyses despite the potential oversampling, as each teacher still can throw only one vote.
Privacy-Fairnest Approaches We use 1) a prominent fair training approach based on reweight- ing (Choi et al. 2020 and 2) two recent classification techniques which address both privacy and fairness: DP-SGD-F (Xu et al. 2020) and DPSGD-Globall-daaa (Esipova et al.. 2022).
Reweighting. As outlined in Sec. 3 the reweighting approach computes the likelihood ratio and modifies the loss term of discriminator achieve fairness. For subgroup bias setting, directly compute the likelihood ratio using sensitive group labels (Eq. For unknown subgroup setting, we estimate the value using binary classification approach with their official Github code. During this estimation process, a public reference dataset is required to effectively train a binary classifier.
DP-SGD-F and OPSGiD-Adapt-Globan DP-SGD-F and DPSGID-Adat-GGia DP-SGD where clipping bounds dynamically adjusted to control the fairness-utility tradeoff. To prevent excessive gradient clipping for minority data group samples, DP-SGD-F employs groupwise clipping approach where each data group has its own clipping bound, while DPSGD- Adapt employs a scaling approach where all per-sample gradients are scaled down depending on a dynamically adjusted scaling factor. As DP-SGD-F and DPSGD-Adapt Global do not provide the official codes to our knowledge, we faithfully implemented each algorithm based on their papers. We note that DP-SGD-F not applicable in the unknown subgroup setting, as they require number of subgroup samples present in the batch to compute clipping bounds for each subgroup: in contrast, DDSGD-Global-Aapp is applicable in the unknown subgroup setting, as the scaling factor does not require knowledge on subgroup labels.
are fair variants of
E ADDITIONAL EXPERIMENTS
Continuing from Sec.
5
5.1
5.3
and 5.4 we provide more experimental results.
E.
MORE EXPERIMENTS ON BRIVACY-FAIRNESS-UTIIS TRADEOFF
Continuing from Sec. 5.2
provide more results on the frivacy-faimess-uuliiyy tradeoff.
Fairness- Utility Tradeoff with Varying Privacy Levels We analyze the faimess-utility tradeoff under varying privacy levels, following (Tran et al. 2021b). Specifically, we examine how training with PFGuard impacts the original fairness-utility tradeoff of underlying private generative models, using the MNIST dataset under the subgroup bias setting. The results are shown in Fig. 7 and Fig. 8 In weaker privacy regimes (i.e., higher values of E), both the privacy-only generative model and PFGuard converge to similar performance levels. However, PFGuard show notable fairness improvements in stronger privacy regimes, albeit it also shows utility degradation due slower
22

---

Published as a conference paper at ICLR 2025
12
25
45
02 25
Palacy [8]
Figure Faimess-utilitty tradeoff of GS- WGAN (i.e., private generative model).
t.n
200
Farneso P10 Daps Lidily (D
on
Ubilit (PID)
Figure 6: Visualization Pareto frontier results of PF- Guard. A darker color indicates a more stronger privacy constraint. Both fairness metric (Distributon disparity) and utility metric (Overall FID) are lower the better.
PINSGy
Figure 8: Fairness-utilitty tradeoff of GS-
WGAN when trained with PFGuard.
convergence in the early training stages. We suspect this slower convergence is a natural consequence of learning more balanced distribution, since the generative model should capture more diverse input features at the early training stage and thus can be more challenging.
Pareto Frontier Results Fig. 6 visualizes the Pareto frontier results of PFGuard, showing the fainessiiirness-utilit tradeoff. The nonlinear surface highlights the intricate relationship between these three objectives. As privacy constraints weaken (i.e., the light-colored region), both utility and fairness improve, converging toward a more favorable region. However, privacy constraints become stronger, the utility narrows down to a specific range, while fairness greatly varies, which is particularly evident in the purple-colored region. This suggests that stronger DP noise consistently degrades the image quality, but results in highly variable fairness outcomes, which implies that there can be sweet-spot regions that can achieve both fairness and utility.
E.2 EXPERIMENTAL RESULTS ON TABLUAR DATA
Continuing from Sec. 5.3 we provide additional results with a tabular dataset. Although tabular data not of our immediate interest given our focus on scalability to high-dimensional data such as images, we show how PFGuard can support tabular data as well as image data.
Experimental Setups [Dataset] We use Adult (Kohavi 1996). which contains 43,131 examples of demographic information of individuals along with a binary label of whether their annual income is greater than 50k. We use gender as the sensitive attribute s and income as the class label y. [Baseline) We use DP-WGAN (Xie et al., 2018) and PATE-GAN (Tordon et al., 2018) for private-only generative models, and FFPDG (Xu et al. 2021 for both fair and private generative models. Since PATE-GAN is PTEL-based generative model, we use PATE-GAN the base generator of PFGuard. We note that FFPDG is the most closest work of ours, which mainly focuses on tabular data and poses challenges when scaling images
more details in Sec. 6 [Metric] We use privacy and faimness metrics from the main text and AUROC as the utility metric, following FFPDG for effective comparison.
Privacy-Faid Faaney--FirnessUttiitt Tradeoff Table 5 shows the serracy-aainess-utiii performances on the Adult dataset. Notably, PFGuard achieves fairness comparable to fair-only generative models while maintaining comparable utility with privacy-only generative models and FFPDG. In addition, Fig. p shows the Pareto frontier results compared to privacy-only generative models, where PFGuard outper- forms other baselines by showing curves closer to the ideal region (i.e., low fairness discrepancy and high utility). These results tabular data consistent with the results image data, highlighting the effectiveness and flexibility of PFGuard to a wide range of real-world applications by supporting both tabular and image datasets.
23

---

Published as a conference paper at ICLR 2025
Table 5: Comparison of privacy-faimess-uuiiity performance Adult under e=1, using PATE-GAN as the base DP generator. The first five rows represent upper bound performances for vanilla, DP-only, and fair-only models. Evaluations cover subgroup bias. Lower values better across all metrics except AUROC. We boldface the best results and underline the second best results.
Privacy
Fairness
Uiility
Method
EO Disp. (1)
Dem. Disp. (J)
AUROC (1)
Vanila
0.56
0.58
0.80
Fair-only
0.07
0.07
0.75
DP-WGAN
0.31
0.30
0.69
PATE-GAN
0.19
0.22
0.74
RON-Gauss
0.18
0.14
0.70
FFPDG
0.12
0.20
0.75
PFGuard
0.08
0.12
0.76
Table 6: Comparison of computational time of private generative models with and without PFGuard.
MNIST
FashionMNIST
Method
PFGuard
PFGuard
Overhead (%)
w/o PFGuard
w/ PFGuard
Overhead (%)
GS-WGAN
7378.30
7467.48
1.21
8114.96
8392.58
3.42
G-PATE
30810.25
31852.11
3.38
25238.84
26317.07
3.56
DataLens
41590.34
41638.
0.12
547740.47
55714.41
1.78
Figure 10: FID values with different nor- malization factors, evaluated on MNIST using different batch sizes. GS-W WGAN is used as the base DP generator. N1 denotes a default normalization factor (.4c. N2 denotes normalization fac-
0.9
0.6
Utility (AUROC
0.7
PATE-GAN
0.6
PFGuard
RON Gouss
0.5
tor from Skare et al. 2003).
DPWEAN
0.4
Normalization factor N1
0
0.1
0.2
0.4
0.5
Batch size
N2
Fairness (EO Disp.
+)
128 (s 29.90 29.91)
75.05.226
74.58
Figure 9: Pareto frontier results of DP generative models evaluated on tabular dataset (Adult). Upper left region denotes ideal case with low fairness discrepancy (EO Disparity) and high utility (AUROC).
64 (s 19.58)
75.35
72.20
32 (E 9.99)
82.68
78.18
E.3 COMPARISON OF COMPUTATIONAL TIME
Continuing from Sec. 5.3 we compare the computational time when integrating PFGuard with existing Pbeed generative models. Table fo that PFGuard incurs minimal overhead in computational time (< 4%), due the simple modification in minibatch sampling for fairness.
E.4 ADDITIONAL NORMALIZATION TECHNIQUE FOR FASTER CONVERGENCE
Continuing from Sec. 5.3 we investigate the impact of the normalization factor on the overall image quality of PFGuard. While we use a traditional normalization factor N; Σ, h(x,) for w(z;) & h(2;) in our SIR- based sampling algorithm. we can employ additional normalization techniques boost the performance. For example, we can use N₂ = Σ, h(xa)/N_, for w(r,) xx hxx;)/N=; where N
h(z,) h(xx), which is known to help faster convergence of SIR algorithms to the target distribution Phal (Skare et al.) 2003). To investigate the influence on model convergence of normalization factors, we compare the overall image quality resulting from two different normalization factors N₁ and N2 with varying batch sizes. We create both binary class bias and subgroup bias in the MNIST dataset with bias level 3 (see Sec. D. challenging setting to effectively compare convergence speeds. We note that using a larger batch size change the corresponding DP guarantee, using large batches leads to more data usage.
details) make more
24

---

Published as a conference paper at ICLR 2025
Table 10 shows the comparison of image quality measured with FID, where lower value indicates better quality. Although both N1 and N2 demonstrate comparable performance when using a large batch size, the performance gap becomes more evident the batch size decreases. This empirical evidence shows that the performance of PFGuard can be further improved by additionally employing various normalization techniques.
E.5 COMPARISON WITH DIFFERENT SAMPLING STRATEGY
Continuing from Sec. 5.2 we provide a performance comparison with a different sampling scenario to enhance fairness. If sensitive attribute labels are available, one natural strategy is to train teachers exclusively on each data group to learn group representations, which is a commonly used strategy in stratified sampling approaches (Meng) 2013). We compare this stratified sampling strategy against PFGuard's importance sampling strategy. which trains teachers on randomly partitioned data, assessing their respective
tradeoffs.
Table 7 shows that while stratified sampling achieves comparable fairness to importance sampling, significantly reduces utility. A potential theoretical explanation is the downgraded utility of the teacher ensemble. Training teachers on non-i.i.d. datasets can lead to inconsistent convergence across teachers, leading to low consensus during teacher voting (Dodwadmath & Stich 2022). This low consensus not only reduces the ensemble's 's prediction accuracy (i.e.. utility). but also increases privacy costs during DP aggregation (Papernot et al. 2018) ultimately leading worse privacy- fairness-utility tradeoff. These results suggest that while training teachers on heterogeneous data may be useful in other contexts, importance sampling better suited for ensemble learning context to achieve both privacy and fairness.
Table 7: Comparison of MNIST under the subgroup bias setting. GS-W WGAN is used the base DP generator. Lower values are better across all metrics. We boldface the best results and underline the second best results.
performance of sampling strategies, evaluated on
Privacy
Fairness
Utility
Method
KL (L)
Dist. Disp. (L)
FID (L)
DP-only
10
0.177
0.383
77.97
Stratified sampling
10
0.090
0.209
135.35
Importane sampling
10
0.067
0.242
83.67
E.6 EXTENSION WITH OTHER FAIRNESS TECHNIQUES
Continuing from Sec. 5.3 and Sec. C.3
we show experimental results on supporting other fairness GOLD Mo et al. 2019), which aims to achieve Rawlsian Max-Min
techniques. In particular, fairness (Joseph et al. 2016) and compatible with PFGuard (see Sec. C for more details).
Thole the privacy-fairmess-uulii performance when applying GOLD in the unknown sensitive attribute setting. GOLD achieves the best performance for the smallest (i.e., worst-case) group, aligning with its goal of Rawlsian Max-min fairness. However, GOLD does not surpass PFGuard in
group fairness metrics or overall utility. These results suggest that incorporating additional fairness techniques can provide flexibility for different use cases, where GOLD can be particularly beneficial in applications prioritizing Rawlsian Max-min fairness over group fairness.
E.7 EXPERIMENTAL. RESULTS ON FASHIONMNIST
Continuing from Sec. 5.3 we show PFGuard's performances in synthetic data (Table and down- stream tasks (Table 10) evaluated on the FashionMNIITT dataset. Compared to the results evaluated on MNIST, private generative models often generate more imbalanced synthetic data w.r.t. sensitive groups and exhibit lower overall image quality. In comparison, PFGuard consistently improves both fairness and overall utility in most cases, similar the results observed in the MNIST evaluation.
25

---

Published as a conference paper at ICLR 2025
Table 8: Comparison of privacy-faimmes--uilit performance of additional faimess technique (GOLD (Mo et al. 2019)), evaluated on MNIST under unknown subgroup bias setting. GS-WGAN used as the base DP generator. Lower values are better across all metrics. We boldface the best results and underline the second best results.
Privacy
Fairness
Utility
Method
E
KL (1)
Dist. Disp. ())
Smallest group FID ())
FID (L)
DP-only
10
0.177
0.383
101.39
77.97
PFGuard
10
0.004
0.041
89.43
89.76
PFGuard + GOLD
10
0.090
0.209
84.52
100.39
Table 9: Fairness and utility performances of private generative models with and without PFGuard on synthetic data, evaluated on FashionMNIST with subgroup bias under e = 10. Blue and red arrows indicate positive and negative changes, respectively. Lower values are better across all metrics.
Faimess
Utility
Method
KL (.)
Dist. Disp. ())
FID (L)
Y=I, S=1
Y=1,
Y=0,
Y=0, S
GS-WGAN
g.558
0.651
124.85mom
130.95
278.061am
155.3620m
217.00gare
G-PATE
0,270
0.494
13
271.28-
249.66.mms
275.61
282.58
Datal Lens
0. 1o000
1.388
165.90
197.61
191.72
173.60_mm
22
GS-WGAN PFGuard
0,009: ())
0.065 (L)
113.13 (1)
149.54
166.69=m.
114.87
146.67-11.2
G-PATE PFGuard
0.190= ())
0.418+am (3)
242.20-m21 (L)
267.14=ssas
248.92+53.
266.91+314
295-32-su
DataLens PFGua
0.127 2007 (1)
0.345 (L)
209.48 2:20 (1)
248.43 222. 16216
222.462:517
262.6222157
Table 10: Faimess and utility performances of private generative models with and without PFGuard on downstream tasks, evaluated on FashionMNIST with subgroup bias under & = 10. Blue and red arrows indicate positive and negative changes, respectively.
MLP
CNN
Faimesss
Utility
Fairness
Utility
Method
EO Disp. (4)
Dem. Disp. (L)
Acc (f)
EO Disp. (L)
Dem. Disp. (4)
Acc (1)
GS-WGAN
0.773
0.021=0a0
0.812=009
0.795=0om
0.007=007
0.804
G-PATE
0,636esus
0.162=ma9
0.875
0.525
0.095=0054
0,884
DataLens
0.484g
g.203
0.901 :00m
0.328
0.072
0.925:nees
GS WGAN + PFGuard
0.296 (.)
0,152 aoem (t)
0.884 inter (f)
0.449 gnon (L)
0.203 anom (?)
0.910 mocr (?)
G-PATE PFGuard
0.556 saase (
0.154 awor (L)
0.885 su20 (1)
0.476 awer (L)
0.124 suces (*)
0.899 =amr (f)
DataLens PFGuard
0.387
Lais (
0,153
intor (
0.858 (L)
0.394 iuwe (
0.093 iuo (?)
0.877 Loies (L)
E.8 FULL RESULTS WITH STANDARD DEVIATION
Continuing from Sec. 5.2 and Sec. 5.4 show full results with standard deviations. Table I1 and
Table the full results of Table demonstrating the stable fairness improvements of PFGuard while preserving utility.
Table 11: Full results of faimnsss and utility performances of private generative models with and
without PFGuard on downstream tasks, evaluated CelebA(S) with binary class bias under 1. GS-NGAN is excluded due to lower image quality in this setting. Blue and red arrows indicate positive and negative changes, respectively.
Fairness
Utility
Method
Acc. Disp. (1)
Acc (t)
Acc (Y=l)
Acc (Y=D)
G-PATE
0,978 20004
0.66648008
0.014
0.992
Datalenns
0m793
0.64
0.114
0.907
G-PATE + PFGuard
0.736 #0.126 (L)
0.678 (1)
0.187 #0.05 (t)
0.923 +000 (L)
DataLens + PFGuard
0.725 (L)
0.689 (1)
0.205 (1)
0.931 #00ss (1)
26

---

Published as a conference paper at ICLR 2025
Table 12: Full results of faimess and utility performances of private generative models with and without PFGuard on downstream tasks, evaluated on CelebA(L) with binary class bias under & GG-GGGAN is excluded due lower image quality in this setting. Blue and red arrows indicate positive and negative changes, respectively.
Fairness
Utility
Method
Acc. Disp. (U)
Acc (t)
Ace (Y=1)
Acc (Y=0)
G-PATE
0,968200ss
0.66820m1
0.023
0.991 20077
Datal enns
0.678
0.6862om1
0.234
0.912anms
G-PATE PFGuard
00344 (L)
=056 (L)
0.378 (t)
0.655 ())
DataLens + PFGuard
0.641 200s (1)
2.70 (f)
0.276 1num (1)
0.917 20,000 (1)
F RELATED WORK
Continuing from Sec.
6
we provide more discussion on related work.
Private-only Data Generation Most peeeacy-reesevvigg data generation techniques focus on satisfying differential privacy (DP) (Dwork et al. 2014). The majority of these techniques focus on
privatizing Generative Adversarial Networks (GANs) (Goodfellow et al. 2014). while recent studies have explored other generative models as well (Takagi et al.
2021 Cao et al. 2021 Harder et al.
2021 Liew et al., [021 Chen al. 2022 Vinaroz et al.
2022 Yang et al. 2023 Ghalebikesabi et al.
2023). One privatization approach is based on DP-SGD Abaai et al. 2016). . which 18 a DP version of
the standard stochastic gradient descent algorithm train MI. models (Xie et al. 2018 Zhang et al.
2018.
"Torkzadehmahani et al. 2019) Bie et al. 2002)). Another privatization approach is based on the
Private Aggregation of Teacher Ensembles (PATE) framework (Papernot al.) 2016 2018), which trains multiple teacher models on private data, and updates the generator with differentially private
aggregation of teacher model outcomes (Jordon et al. 2018 Long et al.) 2021 Wang et al., 2021a).
GS-WGAN (Chen et al. (020)) leverages both DP DP-SGD and PATE, where multiple teacher models are trained as PATE, while their outcomes are processed with gradient cliiping as in DP-SGD. We design PFGuard to complement these private GANs by also achieving fairness in data generation.
Fair-only Data Generation The goal of model fairness is to avoid discriminating against certain demographics (Barocas et al. 2017 Feldman al. 2015 Hardt et al. 2016). and fair data generation solves this problem by generating synthetic data to remove data bias. The main approaches of fair data generation are as follows: 1) modifying training objectives to balance model training (Xu et al., 2018, (Sattigeri et al. 2019 Yu et al., 2020 Choi et al., 2020 Teo et al. 2023) and 2) modifying latent distributions the input noise to obtain fairer outputs (Tan et al. 2020 Humayun et al. 2021). ). In comparison, PFGuard 1) modifies sampling procedures to balance model training while preserving original training objectives and 2) makes the key contribution of satisfying both privacy and fairness. There is another recent line of work using generated data together with original training data for model fairness (Roh et al., 2023] Zietlow et al. 2022), but they focus on classification tasks and assume to use given generative models.
Privacy-Fairness Intersection Recent studies have shown that achieving DP can hurt model fairness in classification tasks (Bagdasaryan al. 2019 Farrand et al. 2020 Xu et al. 2020 Esipova et al. 2022), decision-making processes (Pujol et al. (2020). and generation tasks (Cheng et al. [021] Ganev et al. 2022; Bullwinkel et al. 2022. Rosenblalt et al. 2024). In addition, many studies have tradeoff, showing that achieving both privacy and fairness will necessarily sacrifice utility (Cummings et al., 2019:; Agarwal 2021; Sanyal al., 2022). In comparison, our study uncovers the counteractive nature of privacy and fairess achieving DP can compromise model fairness and achieving model fairness can compromise DP.
investigated the privacy-fairesssuttil
Private and Fair Classification To effectively achieve both privacy and fairness in model training. various techniques have been developed for classification tasks (Jagielski et al. 2019 Xu et al.) 2019 2020 Tran et al. 2022 Esipova et al. 2022. Kulynych et al., 2022: Yaghini et al. 2023 Lowy et al.
2023 J. In comparison, PFGuard focuses on data generation tasks, which aim to learn the underlying
training data distributions to generate synthetic data, and specifically tailors its fair training phase
27

---

Published as a conference paper at ICLR 2025
to generative modeling objectives. Although having the fundamental different problem settings, make comparisons with existing classification techniques that also leverage 1) importance sampling or 2) PTEL. which are the two key components of PFGuard. (Kulynych et al., 2022) also extends importance sampling to private settings and evaluate fairness downstream classification. In contrast, PFGuard evaluates fairness in both synthetic data generation and downstream classification. (Yaghini) et al. 2023) and (Tran et al. 2022) use PTEL mechanisms, but rely on public datasets to train student classifiers. In contrast, PFGuard eliminates the need for public datasets by making PTEL queries with generated samples from the student generator. We finally note that (Lowy et al. (023) introduces the first DP fair learning method with convergence guarantees for empirical risk minimization. In contrast, PFGuard provides convergence guarantees for fair generative modeling.
28


Reviews:
[{"review_id": "2Ca8ez1JL0", "reviewer": "Reviewer_pijU", "summary": "The paper presents PFGuard, a framework for jointly private and fair generative models. The challenges of naively integrating an unfairness mitigation scheme within private generative models are considered. The paper presents an algorithm based on importance sampling to mitigate unfairness first and then privatize using a teacher ensemble a la PATE.", "questions": "- Can you justify assumption 2 on Line 256? (see my feedback in the weaknesses part)\n\n- Can you include one of the aforementioned baselines?\n\n- Do you acknolwedge the additional privacy cost of importance sampling? Can you address that in a meaningful way?\n\n- Have I misunderstood part of your work? To be clear, I think as is, this paper is not ready for publication. However, I want to be fair and make sure that I have not misunderstood your work. So I'll be happy to engage with you during the rebuttal process.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "- The idea of controlling the fairness-privacy tradeoffs for generative application is worthwhile.\n\n- The writing is clear and easy to follow.", "weaknesses": "- **Algorithm likely has unaccounted-for privacy leakage.** The paper claims that the privacy of Private Teacher Ensembles comes from data disjointness alone.  This is not true; that guarantee also depends on random partitioning of the private data; therefore not only disjointness is required but so is random sampling. The importance sampling step that the paper employs cannot, by definition, be completely random as it has to take into account sensitive group. This means that there is an additional privacy cost to this sampling step that the paper simply does not consider. Using importance sampling in DP settings for fairness is not new. In fact, one of aforementioned papers Kulynych et al. 2021 do this while also accounting for the privacy cost of importance sampling.\n\n- **Claims of novelty are exaggerated.** The contribution of the paper is doing private and fair generation; its claim novelty is generation as classification has been done before; yet the private generation is achieved via other methods. In general, reading through Section 4 and especially Section 4.3, the paper is lacking substance regarding the particularities that introducing fairness to private generation brings. Over-relying on prior work like (Jordon et al., 2018; Chen et al., 2020; Long et al., 2021; Wang et al., 2021a) has left the analysis incomplete and without justification. \n  \n  For instance, I believe this statement on  Line 315 is wrong:  \n  \n  > PFGuard preserves any sensitivity as long as the PTEL enforce data disjointness; even with fair sampling, a single data point still affects only one teacher.  \n  \n\t- How come? Can't a point in the minority group be resampled to maintain a similar data distribution across all teachers as assumption (2) (Line 256) requires? Speaking of Assumption 2:\n\n- **Assumption (2)-Line 256 is unrealistic.** paragraph on Line 254 reads:\n  > Methodology We now present our sampling technique, which guarantees $B \\sim p_{\\text {bal }}$ based on SIR. We first make the following reasonable assumptions: 1) each data sample has a uniquely defined sensitive attribute $s \\in S$ (e.g., race); 2) $p_{\\text {bal }}$ is uniformly distributed over $s ; 3$ ) following Choi et al. (2020), the same relevant input features are shared for each group $s$ between the balanced and biased datasets (e.g., $p_{\\text {bal }}(\\mathbf{x} \\mid \\mathbf{s}=s)=p_{\\text {bias }}(\\mathbf{x} \\mid \\mathbf{s}=s)$ ), and similarly between the training dataset $D$ and any subset $D_i\\left(\\right.$ e.g., $\\left.p_D(\\mathbf{x} \\mid \\mathbf{s}=s)=p_{D_i}(\\mathbf{x} \\mid \\mathbf{s}=s)\\right)$ . We now outline the technique step-by-step below.  \n  \n\t- Assumption 2 here means that samples can be balanced out in terms of sensitive group membership. This assumption is unrealistic. How can the sensitive feature be uniformly distributed; when by definition there exists minority and majority sensitive groups? The only possible way I can think of that would make this work in any practical setting is by resampling minority samples over and over. But that is bound to increase the privacy cost.\n\n- **The paper is missing a number of relevant prior work.** I have already mentioned Kulynych et al. 2021  which essentially does importance sampling for DP-SGD. Also, state-of-the-art for private and fair classification is DP-FERMI by Lowy et al. 2023 is not really considered. Remark 1 on page 4, regarding being the first to revleal that fairness and privacy tecnhiques can counteract each other is not true. Yaghini et al. 2023 and Tran et al 2021  and other make the same observations under PATE classification. I am well-aware of the authors contention in Remark 2 and Section A. So I am going to provide counter-arguments why I cannot accept their line of arguments there in.\n\n\t- First, authors claim the other works Jagielski et al. 2019 Mozannar et al. 2020 Tran et al 2021 Lowy et al. 2023 all use DP w.r.t. sensitive attribute hence they account for a different DP definition. While that is true of the first 3, it is not true of Lowy et al. 2023, neither is it true for Kulynych et al. 2021 or Yaghini et al. 2023 who all consider central (i.e. w.r.t all attributes) DP as well. Incidentally, Tran et al 2021 and Yaghini et al. 2023 setting is over PATE which is pretty close to the PTEL setting of the paper modulu the generation part. But as established earlier, the present paper does not advance the generative setting beyond prior work.\n\n\t- Second, it is unclear to me why challenges of accounting for the privacy cost of adjusting C plays any role in those works not being considered as baselines. If these methods budget their privacy allocation poorly, doesn't that make for a stark and interesting comparison? To be honest, I do not believe these are the best baselines to compare against but I found this line of argumentation faulty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KR8D2dncfp", "reviewer": "Reviewer_sxpU", "summary": "This paper studies tensions between fairness and privacy in generative models. It proposes PFGuards, a framework leveraging an ensemble of teacher models to jointly enforce fairness and differential privacy while preserving utility. The proposed method consists of training an ensemble of fair teachers using balanced minibatch sampling and using their differentially private (leveraging the PATE framework) aggregated output to train a privatized generator. Experiments demonstrate improved performances in fairness and utility compared to existing approaches.", "questions": "- Is there any consistent trend in utility and fairness as the privacy budget increases? I would suggest the authors to perform a similar analysis as in (Tran et al., 2021b -- Fig. 2)\n- Given the high-dimensional context, other fairness notions, such as Rawlsian Max-Min fairness, could be more appropriate. Can the framework proposed extend to such notions?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Good quality of the presentation\n- Extensive experiments on MNIST, FashionMNIST, and CelebA\n- Comparison against several baselines, including fair variants of DP-SGD", "weaknesses": "- Limited choice in epsilon values, making it difficult to understand the effect of privacy budget on utility and fairness\n- The choice of datasets and bias settings makes it difficult to determine whether the method would perform effectively in real-world scenarios. I recommend incorporating tabular datasets, as done in (Tran et al., 2021b), to assess group fairness. Additionally, it would be valuable to explore alternative fairness notions for datasets where group fairness may not be applicable", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b2m50AnCBf", "reviewer": "Reviewer_WKsH", "summary": "This paper focuses on data privacy and model fairness for developing generative models. They claim natively combining differential privacy and fairness learning techniques may cause conflicts, and then propose PFGuard framework to simultaneously balance the two objective and also the utility. The core insight is to employ an ensemble of multiple teacher models. And they also do experiments on GANs with images datasets as benchmarks to show a better trade-off can be achieved.", "questions": "Please refer to Weaknesses.\n\nAlso, can you justify the novelty of using PTEL in this work? Because this is the main technique of the proposed framework.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. I agree with the authors that the conflicts between data privacy and fairness do exist for generative models. It is an interesting research problem to be explored by research community.\n2. I think importance resampling can be a useful approach to get balanced training data, which is helpful to train a fair generative model.\n3. The experiments shown in tables demonstrated that better utility and fairness are achieved by fixing a privacy budget.", "weaknesses": "1. The authors have claimed the interested privacy of this paper is to defend against training data reconstruction, while naive differential privacy techniques preserve the membership instead of data content. This has been recognised in the early work [1] but ignored by this paper.\n2. Since only a smaller training data is derived after resampling for fairness, this may degrade the quality of generated data, as claimed in [2]. In this case, the quality of generated data will be sacrificed.\n3. It is easy to understand that minority samples should be upweighted in fairness. But in what scenarios minority samples should be downweighted for DP? Because sensitivity controls the strength of the added noise, maybe how will minority samples affect the sensitivity should be explained.\n4. In Fig. 3, reweighted and rewerighting are both used.  Are they same? In addition, it seems privacy budget is fixed and then the trade-off between utility and fairness is studied, and so as to experiments. This is not aligned with the motivation where fairness and privacy conflicts.\n5. The proposed framework is a sequential procedure where the first component is about fairness while the second one is for privacy. In this sense, it is not very convincing to say they two can be better traded off.\n6. I would like to see some Pareto front results in terms of three metrics, which can better demonstrate the proposed method.\n\n[1] Bounding Training Data Reconstruction in Private (Deep) Learning, ICML 2022.\n[2] Generative Adversarial Ranking Nets, JMLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b1VgAmH6q5", "reviewer": "Reviewer_oBbq", "summary": "This paper introduces PFGuard, a framework for privacy-preserving generative models that integrates fairness through a simple modification in the minibatch sampling process, specifically employing importance sampling based on group membership. The framework builds on PATE/PTEL, which is commonly used for privacy deep private generative models. The paper summarizes challenges in balancing privacy, utility, and fairness in generative models, then proposes the PFGuard framework (which relies on protected group-wise importance sampling), before finally empirically arguing that PFGuard achieves an improved balance of privacy and fairness.", "questions": "All that said, I am open to raising my score slightly if the authors can adequately address the following questions,\n\nQ1: How precisely is the public reference sample handled experimentally? Are all methods given “equal access” so to speak? If the sample is assumed to be complete (i.e. containing all relevant columns from D) also please provide a test on the same metrics you present for the private/fair methods of simply using the holdout sample at different subsample percentages on each task.\n\nQ2: Why should we use importance sampling, instead of just constructing (potentially deterministically) balanced samples using stratified sampling? It seems believable (although is not proved here) that in the PATE/PTEL framework we can use whichever sampling technique we want, so long as it is randomized (although this is not explicitly proven by the authors or cited). However, given that, AND the assumption of access to the membership of samples in protected classes, importance sampling vs. stratified sampling should be explored. Or maybe I’m missing something.\n\nQ3: Please, if you can, offer a more formal characterization of how importance sampling does not effect the privacy guarantee of PATE/PTEL. You can also use the framework presented in Bassily et. al .", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "This work quite successfully communicates the challenges in achieving the privacy/fairness tradeoff. I particularly appreciate the clarity of Figures 1, 2 and 3, along with the care with which the experimental results in Tables 3 and 4 are presented. Additionally, I believe that this is an interesting problem that deserves attention. I have closely read the main paper body, and appreciate the lack of obvious grammatical errors - overall the paper is well written.", "weaknesses": "Unfortunately, though the paper writing and presentation is of high quality and clarity, there are issues in terms of originality, significance and correctness.\n\nW1: I’d like to highlight that Remark 1 (on novelty) is not correct and should be removed or further qualified. There appears to be ample prior work that “reveals how fairness and privacy techniques can counteract each other,” some in a more formal ways than this work (Bullwinkel et. al, https://arxiv.org/pdf/2205.04321 , Rosenblatt et. al https://arxiv.org/pdf/2312.11712 , Abroshan et. al (https://proceedings.mlr.press/v238/abroshan24a/abroshan24a.pdf , Cheng et. al https://dl.acm.org/doi/pdf/10.1145/3442188.3445879 )\n\nW2: Additionally, Remark 2 either needs to be removed or needs further clarification - why would we extend classification techniques to the generative setting? Differentially private data generation (both synthetic tabular, see Mckenna et al 2019, 2024, https://proceedings.mlr.press/v97/mckenna19a/mckenna19a.pdf , https://arxiv.org/pdf/2201.12677 , Liu et. al 2021, 2023 https://proceedings.neurips.cc/paper_files/paper/2021/file/0678c572b0d5597d2d4a6b5bd135754c-Paper.pdf , https://proceedings.mlr.press/v202/liu23ag/liu23ag.pdf AND image Ghalebikesabi  et. al https://arxiv.org/pdf/2302.13861 ) is a very mature field, under much lighter assumptions then are necessary for PATE/PTEL.\n\nW3: (W2) leads me to my main issue with the empirical results, which is that this is a bit of an apples to oranges comparison (or at least, it's not obvious that the comparison issue I see has been properly addressed). As you acknowledge, and as is the standard assumption with the PATE framework, we assume access to “ a public reference data on the order of 10%–100% of |D| for the estimation” (line 281). However, some comparisons in your paper (for example, in table 3) compare PFGuard directly to a method like DP-SGD (with further modifications), and for which it is not clear if the public reference data assumption is leveraged by the DP-SGD fit model (there are existing methods to help do this). \n\nIn fact, assuming an unbiased public reference sample is quite a strong assumption, and having no experiments on how a biased public reference sample would effect your results is questionable. Additionally, if I had access to a public reference sample, even only 10% of some large data sample |D|, why wouldn’t I just train on this sample? I’m missing where you note something about label missingness in this public sample, but maybe that isn’t the assumption? Is the assumption that we have full access to this public unbiased sample, and that its size is substantial? We would certainly want to compare to just training on that if this were the case.\n\nW4: My 1 sentence summary of the proposed methodology here is as follows: take the PATE/PTEL framework, and during the sampling for teacher creation, use importance sampling based on group membership. This, in of itself, is a reasonable approach, but it is not clear to me how this work improves on the component parts of the PFGuard method (which are well established and well studied methods), besides using them in tandem. Nor does that work consider the model of importance sampling and how that might affect utility in other ways from a formal perspective, despite considerable prior work on the utility of PATE (Bassily et. al, https://arxiv.org/pdf/1803.05101 ). Nor does it contend formally with any potential privacy concerns (even to just dispel this concern with a short proof or proof adjustment). Given that I have some issues with the assumptions of the empirical results, this leads me to my score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nLwGhKsliS", "reviewer": "Reviewer_vffx", "summary": "In this work, the authors study the intersections of fairness and privacy in generative models, and show that these two goals can come into conflict. Furthermore, they highlight that naively combining existing fairness and privacy methods can fail. They propose PFGuard, a framework for achieving fairness and privacy simultaneously and then conduct experiments validating their method.\n\nOverall, I think this is a paper with great contributions and analysis. I think some of the boldest claims should be softened, but otherwise I think it is a good candidate for ICLR.", "questions": "Questions:\n * \"Additional fair sampling does not require additional training complexity compared to say adding a loss term for fairness\": I would argue that adding a loss term that can be backpropagated over is much simpler than having to do your complex sampling procedure to construct balanced minibatches. Can you justify or elaborate on why fair sampling is less complex? I don't think this claim is core to your paper so I think you could also do without it. You mention it later in the paper as well.\n * In your extensions to unavailable sensitive attributes, why do you need to train your classifier on less data? Given one of the benefits of PFGuard is any methods you apply before the teacher-distillation step need not be private, I would expect the best thing to do from a fairness perspective is train the best, beefiest sensitive-attribute classifier possible for fair training.\n * In your figures, why do you apply [method + PFGuard]? Isn't PFGuard an end-to-end fairness + privacy method? From what I can tell in your results you are applying PFGuard to existing DP generative models. This makes it difficult for me to interpret your results, and the unique impact of PFGuard on your results.\n * My intuition tells me that privacy always comes at a cost to utility, and this teacher + distillation procedure should be even more noisy than traditional DP-SGD methods. Can you comment on why this is not the case in your results? Why isn't a more advanced privacy method that permits fairness also resulting in further costs to utility than a traditional fairness method? How much of a fairness drop (from teacher fairness to generator fairness) do we incur because of the private distillation?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Strengths:\n * I really liked the discussion of how fairness and privacy can come into conflict. I think it is somewhat \"folklore\" in the trustworthy ML community so I appreciated a more detailed discussion of the topic, especially in Section 3.\n * The PFGuard framework is a great solution to the problem. \"Isolating\" the fair training component and then distilling this knowledge privately is a great approach that I could see being taken to do many different downstream applications that also require privacy!\n * I think it is good the authors acknowledge that fairness adjusts sensitivity, which can still be made private with more noise, but point out the amount of noise to add may be unclear or performance degrading.", "weaknesses": "Weaknesses:\n * The authors focus on GANs as opposed to Diffusion Models, lessening the contemporary impact of the paper. This isn't necessarily a weakness, more an observation.\n * In Section 3, you start with **Adding Fairness Can Worsen Privacy** and describe in text some vague settings where fairness and privacy can come into conflict. I really wanted to see an elaboration or concrete example of how this could happen. I was very excited to see this in **Adding Privacy Can Worsen the Fairness-Utility Tradeoff** part! I think this is a great contribution. However, I think in the second part, you actually discuss both how fairness can worsen privacy (how fairness can modify the sensitivity, C) and then how adding privacy can worsen fairness (clipping worsens fairness adjustments). You should split these up according to the headers you wrote by moving the first example up to the first section. I would rewrite these two sections and cut some of the vaguer writing in the first section, and spend some more time elaborating on these two examples as they are a great contribution that I would be interested in more discussion of.\n * You say in Remark 1 that your study is the first to reveal that fairness and privacy techniques can counteract each other. I think this is too bold of a claim, given works you cite such as [1] and [2] that explore the topic.\n * I wouldn't say giving each teacher *probabilistically* one sample of the minority group is enough data to expect the teacher to be adequately fair. How did you come to this heuristic, and how can you justify it?\n * There are no fairness guarantees offered by this method because of data resampling and teacher distillation. I think the link between having balanced minibatches and the fairness of the teachers is more clear, but I wonder how distillation affects bias or if there is any literature you could cite here.\n\nNotation:\n * Definition 2.1 - I would explain that domain $\\mathcal{D}$ is a dataset, thus why D, D' can differ by a single sample.\n * You should also describe what function we measure sensitivity over. Is it the loss? is it the gradient? is it the model outputs? This will make it much clearer how exactly fairness interventions impact sensitivity in Section 3.\n\n[1]: Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. Differential privacy has disparate\nimpact on model accuracy. Advances in neural information processing systems, 32, 2019.  \n[2]: Tom Farrand, Fatemehsadat Mireshghallah, Sahib Singh, and Andrew Trask. Neither private nor fair:\nImpact of data imbalance on utility and fairness in differential privacy. In Proceedings of the 2020\nworkshop on privacy-preserving machine learning in practice, pp. 15–19, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)