Private GANs, Revisited
Authors: Alex Bie
Gautam Kamath
Guojun Zhang
Venue: Submitted to ICLR 2023 (2023)

Abstract: We show that with improved training, the standard approach for differentially private GANs -- updating the discriminator with noisy gradients -- achieves or competes with state-of-the-art results for private image synthesis. Existing instantiations of this approach neglect to consider how adding noise only to discriminator updates disrupts the careful balance between generator and discriminator necessary for successful GAN training. We show that a simple fix restores parity: taking more discriminator steps between generator steps. Finally, with the goal of restoring parity between generator and discriminator, we experiment with further modifications to improve discriminator training and see further improvements. For MNIST at $\eps=10$, our private GANs improve the record FID from 48.4 to 13.0, as well as downstream classifier accuracy from 83.2\% to 95.0\%.

Under review as a conference paper at ICLR 2023
PRIVATE GANS, REVISITED
Anonymous authors Paper under double-blind review
ABSTRACT
We show that with improved training, the standard approach for differentially pri- vate GANs updating the discriminator with noisy gradients achieves or com- petes with state-of-the-arr results for private image synthesis. Existing instantia- tions of this approach neglect to consider how adding noise only to discriminator updates disrupts the careful balance hetween generator and discriminator neces- sary for successful GAN training. We show that a simple fix taking more dis- criminator steps between generator steps restores parity and improves training. Furthermore, with the goal of restoring parity between the generator and discrimi- nator, we experiment with further modifications improve discriminator training and see further improvements in generation quality. For MNIST at = 10, our private GANs improve the record FID from 48.4 to 13.0, and record downstream classifier accuracy from 83.2% to 95.0%.
INTRODUCTION
Differential privacy (DP) (Dwork et al., 2006b) has emerged as a compelling approach for training machine learning models on sensitive data. However, incorporating DP requires significant changes to the training process. Notably, it prevents the modeller from working directly with private data, complicating debugging and exploration. Furthermore, the modeller can no longer interact with a private dataset after exhausting their allocated privacy budget. One approach to alleviate these issues by producing differentially private synthetic data, which can be plugged directly into existing machine learning pipelines, without further concern for privacy.
A recent line of work studies leveraging deep generative models to produce DP synthetic data. Early efforts focused on privatizing generative adversarial networks (GANs) (Goodfellow et al., 2014) by using differentially private stochastic gradient descent (DPSGD) (Abadi et al., 2016) to update the GAN discriminator an approach referred to as DPGAN (Xie et al., 2018: Beaulieu-Jones et al., 2019; Torkzadehmahani et al., 2019).
However, follow-up work has significantly departed from this baseline DPGAN approach, either in terms of: (a) the privatization scheme, in favor of approaches based on subsample-and-agee aggagggee which divide the data into 2 1000 disjoint partitions and train teacher discriminators separately on each one (Jordon et al., 2019; Long et al., 2021; Chen et al., 2020; Wang et al., 2021); or (b) the generative modelling frumework altogether, opting instead to minimize notions of statistical distance between real and generated data, such as maximum mean discrepancy (Harder et al., 2021; Vinaroz et al., 2022), or Sinkhorn divergences (Cao et al., 2021).
For labelled image synthesis, these custom generative models designed specifically for privacy fall short of GANs when evaluated at their non-private limits (E + oo), suggesting limited scalability to larger, higher-resolution datasets.' On the other hand, the literature corroborates that under mod- est privacy budgets, these departures from the baseline DPGAN lead to significant improvements in generation quality. Proposed explanations attribute these results inherent limitations of the (a) privatizing discriminator training is sufficient for privacy, but may be overkill when only the generator needs to be released (Long al., 2021); (b) adversarial objectives may be unsuited for training under privacy (Cao et al., 2021).
DPGAN framework, suggesting that either:
For example, the record FID for MNIST at 10 is 48.4 (Cao et al., 2021). When evaluated at co, their method achieves FID of 43.4, Our non-private GANs obtain an FID of 3.2.

---

Under review as a conference paper at ICLR 2023
250
34
non-private (3.2)
no = 1
no
1
AU
R3
4
200
0o
20
FID score
150
no 50
np = 10
4
3
X
57
8
2 ?
0
2
100
3
2
7
9
50
0
2
3
5
7
8
9 9 9
no 00=56
0
2
3
5
6
7
8
Privacy budget E
0
1
4
6
7
3
(a) MNIST FID over a training run
(b) Corresponding images at (10, 10 10 >-P
Figure I: DPGAN results on MNIST synthesis at (10, 10 9--D. (a) find increasing rD, the number of discriminator steps taken between generator steps, significantly improves image synthesis results. Using 50 instead of no improves FID from 205.9 + 19.4. which improves over the record FID of 48.4 from Cao al. (2021). Figure 2a), an improvement over record accuracy of 83.2% from Cao al. (2021). (b) Corresponding synthesized images. We observe that large np improves visual quality, and low np leads to mode collapse.
50 also improves the record downstream classification accuracy 92.9% (see
Our contributions. We demonstrate that the reported poor results of DPGANs should not be attributed to inherent limitations of the framework, but rather, training issues. Specifically, we pro- pose that the asymmetric noise addition in DPGANs (adding noise to discriminator updates only) weakens the discriminator relative to the generator, disrupting the careful balance necessary for suc- cessful GAN training. We propose that taking more discriminator steps between generator updates addresses the imbalance introduced by noise. With this change, DPGANs improve significantly (see Figure 1), going from non-competitive to achieving or competing with state-of-the-arr results in private image synthesis.
Furthermore, we show this perspective on private GAN training ("restoring parity to a discriminator weakened by DP noise") can be applied to improve training. We make other modifications to dis- criminator training - large batch sizes and adaptive discriminator step frequency - to further improve upon the aforementioned results.
In summary, we make the following contributions:
1. We find that taking more discriminator steps between generator steps significantly improves DPGANs. Contrary to the previous results the literature, DPGANs do compete with state-of-the-art generative modelling approaches designed with privacy in mind.
2. We present empirical findings towards understanding why more frequent discriminator steps help. We propose an explanation based on asymmetric noise addition for why vanilla DPGANs do not perform well, and why taking more steps helps.
3. We put our explanation to the test. We employ it as principle for designing better private GAN training recipes, and indeed are able improve over the aforementioned results.
2 PRELIMINARIES
Our goal is to train a generative model on sensitive data that is safe to release, i.e., does not leak the secrets of individuals in the training dataset. We do this by ensuring the training algorithm A which takes as input the sensitive dataset D e 24 and returns the parameters of a trained (generative) model 0 E θ satisfies differential privacy.
Definition 1 (Differential Privacy (Dwork et al., 2006b)). A randomized algorithm A : 4 - θ is (e, diffferreetl private if for every pair of neighbouring datasets D, D' E U, we have
P(A(D) E 8) < exp(e) PP((DD) E S) + for all 5 C
In this work, we adopt the add/remove definition of DP, and say two datasets D and D' are neigh- bouring if they differ in most one entry, that is, D D' (ar D' DU (x).
2

---

Under review as a conference paper at ICLR 2023
Algorithm TrainDPGAN(D:)
1: Input: Labelled datasct D {(z w)))-- Discriminator DD and generator G initializations and 0o- Optimizers OptD, Opt6. Privacy parameter 6. Hyperparameters: Rp (D steps 9 step), T (total number of D steps), (expected batch size). C' (clipping norm), (noise multiplier).
2: ( < B/|D| and 1,
Calculate sampling rate q, initialize counters. Update 'D with DPSGD. Sample real batch St by including each (æ, v) D w.p. 9- P Sample fake batch S,.
3: while t T do
4:
St PoissonSample(D, (D
5:
S, ~ -(;00))
6:
go. Lnwles, clip (Va, (og((( Lo, (+))))
LameE, clip (Vo, log(1 D
Clip per-example gradients. Add Gaussian noise.
7:
Tae 28 (ye, + z1). where 2 N1 ((
8:
Φc+1 -OptD(0e; Do.) and t+1
9:
no divides t then
Perform G update every np steps.
S 0(:;0a)
90, + B LameR (2
12:
O + Opt 96.) and k + ++1
13: end if
14:
end while
15: PrivaayAccouuttn(((( q, 9,
Compute privacy hudget spent.
Output: Final parameters 0g. (E, 6D guarantee.
We highlight one convenient property of DP, known as closure under post-processing. This says that interacting with privatized model (e.g., using compute gradients on non-sensitive data, generate samples) does not lead to any further privacy violation.
Proposition 2 (Post (Post-processing). Let A: U - θ be randomized algorithm that is (e, 6)-P,, and f: e + y be an arbitrarily randomized mapping. Then f A: - is (E,
DPSGD. A gradient-based training algorithm can be privatized by employing differentially private stochastic gradient descent (DPSGD) (Song et al., 2013; Bassily al., 2014; Abadi et al., 2016) as drop-in replacement for SGD. DPSGD involves clipping per-example gradients and adding Gaussian noise to their sum, which effectively bounds and masks the contribution of any individual point to the final model parameters. Privacy analysis of DPSGD follows from several classic tools in the DP toolbox: Gaussian mechanism, privacy amplification by subsampling, and composition (Dwork et al., 2006a; Dwork & Roth, 2014; Abadi et al., 2016; Wang al., 2019). Our work employs the DPSGD analysis of Mironov et al. (2019) implemented in Opacus (Yousefpour et al., 2021).
DPGANs. Algorithm details the training algorithm for DPGANs, which is effectively an in- stantiation DPSGD. Note that only gradients for the discriminator D must be privatized (via clipping and noise), and not those for the generator G. This is consequence of post-processing (Proposition 2) the generator only interacts with the sensitive dataset indirectly via discriminator parameters, and therefore does not need further privatization.
3 FREQUENT DISCRIMINATON STEPS IMPROVES PRIVATE GANs
In this section, we discuss our main finding: the number of discriminator steps taken between each generator step (np from Algorithm 1) plays significant role in the success of private GAN training. For fixed setting of DPSGD hyperparameters, there is an optimal range of values for np that maximizes generation quality, in terms of both visual quality and utility for downstream classifier training. This value often quite large (nD 100 some cases).
3.1 EXPERIMENTAL DETAILS
Setup. We focus labelled generation of MNIST (LeCun et al., 1998) and FashionMNIST (Xiao et al., 2017), both of which are comprised 60000 28 x 28 grayscale images divided into classes. To build strong baseline, begin from open source PyTorch (Paszke al., 2019) implemen-
3

---

Under review as a conference paper at ICLR 2023
100
100
80
250
Acturany (%)
200
Acturany (%)
60
60
golool (9520
150
40
Po
RD
40
00hppiil (50.4)
Po 10
100
10
20
Pu-50 50
Pu 160
50
20
50
100
Pgy 10--200 200
200
0
6 Privacy budget
10
Privacy budget
Privacy budget
(a) MNIST accuracy
(b) FashionMNIST FID
(c) FashionMNIST accuracy
Figure 2: DPGAN results over training runs using different discriminator update frequencies stp, targeting (10, 10 J-DD.. (a) As measure of synthetic data utility, we plot the test set accuracy of a CNN trained on generated data only. Accuracy mirrors FID scores (see Figure la). Going from ng = to npp = 50 improves accuracy from 33.7% + 92.9%. Further increases hurts accuracy. (b) & (c) We obtain similar results for FaSHionMNIST. Note that the optimal ao is higher (around np 100). At 7yp 100, obtain FID of 91.5 accuracy of 71.1%. which compares favourably the record FID of 128.3 and record accuracy of 75.5% reported in Cao al. (2021) for (10, 10 10-5)) (0-5)-DD --DP generation FashionMNIST.
tation2 of DCGAN (Radford et al., 2016) that performs well non-privately, and copy their training recipe. We then adapt their architecture to our purposes: removing BatchNorm layers (which not compatible with DPSGD) and adding label embedding layers to enable labelled generation. Train- ing this configuration non-privately yields labelled generation that achieves FID scores of 3.2 on MNIST and 15.9 on FashionMNIST.. Finally, we note that these models are not small: 'D and G have 1.72M and 2.27M trainable parameters respectively. Please see Appendix B.1 for more details.
Privacy implementation. To privatize training, we use Opacus (Yousefpour et al., 2021) which implements per-example gradient computation and the RDP accounting of Mironov et al. (2019). For our baseline setting, we use the following DPSGD hyperparameters: we keep the non-private (expected) batch size B = 128, and use noise scale = and clipping norm C = 1. Under these settings, we have the budget for T 450000 discriminator steps when targeting (10, 10 D-D..
Evaluation. We evaluate our generative models by examining the visual quality and utility for downstream tasks of generated images. Following prior work, measure visual quality by com- puting the Fréchet Inception Distance (FID) (Heusel et al., 2017) between 60000 generated images and entire test set.² To measure downstream task utility, we again follow prior work, and train CNN classifier on 60000 generated image-label pairs and report its accuracy on the real test set.
3.2 RESULTS
More frequent discriminator steps improves generation. We plot in Figures la and 2 the evolu- tion of FID and downstream accuracy during DPGAN training for both MNIST and FashionMNIST, under varying discriminator update frequencies np. The effect of this parameter has outsized impact on the final results. For MNIST, np = 50 yields the best results; on FashionMNIST, the best FID is obtained np 200 and the best accuracy at nD 100.
Private GANs are on a path to mode collapse. For the MNIST results in Figures 1a and 2a, we observe that at low discriminator update frequencies (n = 10), the best FID and accuracy scores occur early training, well before the privacy budget we are targeting is exhausted.* In fact, at 50000 discriminator steps (8 & 2.85), wy = 10 has better FID (30.6) and accuracy (83.3%) than other settings of np. However, these results deteriorate with continued training. In Figure 3, we
2 Courtesy of Hyeonwoo Kang We use open source PyTorch implementation compute FID: https:/ /tte:///gttti /ttptgtttbbb////////////.t..bhtt comtuuumoom///.... msei mseitzer/pytorcch fid.
wwt.//o//////// znx1 wm). Code available at this link.
'This observation has been reported in (Neunhoeffer et al., 2021), serving as motivation for their remedy of taking mixture of intermediate models encountered in training. We are not aware of any mentions of this aspect of DPGAN training in papers reporting DPGAN baselines for labelled image synthesis.
4

---

Under review as a conference paper at ICLR 2023
3
3
4 4
6 &
7
1
8
O
3
5
A
5 6
7
7
9
0
3
4
5
6
0
0
25
7
7
0
5
0
8
4
0
6
0
2
4
3
4
7 6
2
Q
/
0
2
7
P
9
L
?
6
7
2
4
99
7
3
6
3
0
2
6
S
5
5
t=
50K
t=
100K
t= 150K
t=
200K
Figure 3: Evolution of samples drawn during training with up 10. when targeting (10, 10 D This setting reports its FID and downstream accuracy 50K iterations (s 2 2.85). training progresses, we observe mode collapse for several classes alongside the deterioration in evaluation metrics.
plot the evolution of generated images for this nop 10 run the course of training, and observe qualitative evidence of mode collapse, co-occurring with the deterioration in FID and accuracy.
An optimal discriminator update frequency. These results suggest that fixing other DPSGD hyperparameters, there is optimal setling for the discriminator step frequency Ttp that strikes balance between: (1) being too low, causing the generation quality to peak early training and then undergo mode collapse; resulting in all subsequent training consume additional privacy budget without improving the model; and (2) being too high, preventing the generator from taking cnough steps to converge before the privacy budget is exhausted (an example of this is the nn 200 run in Figure 2a). Striking this balance results in the most effective utilization of privacy budget towards improving the generator.
WHY DOES TAKING MORE STEPS HELP?
In this section, we present empirical findings towards understanding why more frequent discrimina- tor steps improves DPGAN training. We propose an explanation that is conistent with our findings.
How does DP affect GAN training? Figure compares the accuracy of the GAN discriminator (on held-out real and fake examples) immediately before each generator step between non- non-private training and private training with different settings of npp We observe that non-privately, discrimi- nator accuracy stays around 60% throughout training. Naively introducing DP (np 1) leads to qualitative difference: DP causes discriminator accuracy to drop to 50% immediately at the start of training, and never recovers.
For other settings of np, we make three observations: (1) larger no corresponds to higher accu- racy; (2) the generator improves during the periods in which the discriminator stays above 50% accuracy; and (3) accuracy decreases throughout training as the generator improves, and degrada- tion/stagnation of the generator (as observed in Figure co-occurs with discriminator accuracy dropping to 50%.
Based on these observations, we propose the following explanation for why more steps help:
Generator improvement occurs when the discriminator capable of distinguishing between real and fake data.
The asymmetric noise addition introduced by DP to the discriminator makes such task difficult, resulting in limited generator improvement.
Allowing the diseriminator train longer on fixed generator improves its accuracy, covering the non-private case where the generator and discriminator are balanced.
Does reducing noise accomplish the same thing? In light of this explanation, ask if reducing the noise level a can offer the same improvement as taking more steps, as reducing a should also improve discriminator accuracy before generator step. To test this: starting from setting in Section 3, fixing np = 1, and targeting MNIST at e 10, we search over grid of noise levels
OOur plot only shows the first 20000 generator steps, but remark that this persists until the end of training (450000 steps).
5

---

Under review as a conference paper at ICLR 2023
1.0
Discriminator Accuracy (%)
non-privato
50
0.6
0.4
2500
5000
7500
10000
12500
15000
17500
20000
Generator step
Figure 4: Discriminator accuracy immediately before each generator step. While non pan--ariivlly the discrimina- tor maintains a 60% accuracy. the private discriminator with rop is effectively random guess. Increasing the number of discriminator steps recovers the discriminator's advantage early on, leading to generator im- provement. An improved generator makes the discriminator's 's task more difficult, driving down accuracy.
a (0.4, 0.43, 0.45, 0.5, 0.55, 0.6, 0.7, 0.8); the lowest which, a 0.4, admits a budget of only T 360 discriminator steps. We obtain best FID of 127.1 and best accuracy of 57.5% noise level a = 0.45. Hence we can conclude that in this setting, incorporating discriminator update frequency in design space allows for more effective of privacy budget for improving the discriminator, and in turn, generation quality.
Does taking more discriminator steps always help? As we discuss in more detail in Section 5.1, when we able find other means to improve the discriminator beyond taking more steps, tuning discriminator update frequency may not yield improvements. To illustrate with an extreme case, consider eliminating the privacy constraint. non-private GAN training, taking more steps is known to be unnecessary. We corroborate this result: we run our non non-private baseline from Section 3 with the same number of generator steps, but opt to take 10 discriminator steps between each generator step instead of 1. FID worsens from 3.2 - 8.3, and accuracy worsens from 96.8% + 91.3%.
BETTER GENERATORS VIA BETTER DISCRIMINATORS
Our proposed explanation in Section 4 provides concrete suggestion for improving GAN train- ing: effectively use our privacy budget to maximize the number of generator steps taken when the discriminator has sufficiently high accuracy. We experiment with modifications to the private GAN training recipe towards these ends, which translate to improved generation.
5.1 LARGER BATCH SIZES
Several recent works have demonstrated that for classification tasks, DPSGD achieves higher aceu- racy with larger batch sizes, after tuning the noise scale a accordingly (Tramèr & Boneh, 2021; Anil al., 2021; De al., 2022). GAN training is typically conducted with small batch sizes (for exam- ple, DCGAN uses B = 128, which we adopt: StyleGAN uses B = 32). Therefore it is interesting see if large batch sizes indeed improve private GAN training. We corroborate that larger batch sizes do not significantly improve our non-private MNIST baseline from Section 3: when we go up to 2048 from 128, FID stays 3.2 and accuracy improves from 96.8% - 97.5%.
Results. We scale up batch sizes, considering B [64, 128, 512, 2048), and search for the opti- mal noise scale a and np (details in Appendix B.2). We target both = and = 10. We report the best results from our hyperparameter search in Table We find that larger batch sizes leads to improvements: for e 10, the best MNIST and NNST results are achieved at B 2048. For E = 1, the best results are achieved at 512. We also note that for large batch sizes, the optimal number of generator steps be quite small. For B 2048, = 4.0, targeting MNIST
E 10, no = is the optimal discriminator update frequency, and improves over our best B = 128 setting employing no 50.

---

Under review as a conference paper at ICLR 2023
5.2 ADAPTIVE DISCRIMINATOR STEP FREQUENCY
Our observations from Section and 4 motivate us to consider adaptive discriminator step frequen- cies. As pictured in Figure 4, discriminator accuracy drops during training as the generator improves. In this scenario, we want to take more steps to improve the discriminator, in order to further improve the generator. However, using a large discriminator update frequency right from the beginning of training is wasteful - as evidenced by the fact that low n'p achieves the best FID and accuracy early in training. Hence we propose to start at a low discriminator update frequency (nD = 1), and ramp up when our discriminator is performing poorly.
Accuracy on real data must be released with DP. While this is feasible, introduces the additional problem of having to find the right split of privacy budget for the best performance. We observe that discriminator accuracy is related to discriminator accuracy on fake samples only (which are free to evaluate on, by post-processingg Hence use it proxy assess discriminator performance.
The adaptive step frequency is parameterized by two terms, / and d. 3 is the decay parameter used to compute the exponential moving average (EMA) of discriminator accuracy fake batches before each generator update. We use 3 = 0.99 in all settings. d is the accuracy floor that upon reaching, move to the next update frequency ny E {1, 5, 10, 20,50, 100, 200, 500}. We try d 0.6 and d = 0.7, finding that 0.7 works better for large batches. Additionally, we promise a grace period of 2/(1 - B) 200 generator steps before moving on the next update frequency. This formula is motivated by the fact that 3-EMA's value is primarily determined by its last 2/(1 (1 - 3) observations.
The additional benefit of the adaptive step frequency is that it means we do not have to search for the optimal update frequency. Although the adaptive step frequency introduces the extra hyperparameter of the threshold d, we found that these two settings (d 0.6 and d = 0.7) were sufficient to improve over results of a much more extensive hyperparameter search.
5.3
COMPARISON WITH PREVIOUS RESULTS IN THE LITERATURE
5.3.1
MNIST AND FASHIONMNIST
Table summarizes our best experimental settings for MNIST and FashionMNIST, and situates them in the context of previously reported results for the task. We provide some example generated images in Figures 7 and 8 for E = 10, and Figures 9 and 10 for 1.
Simple DPSGD beats alternative GAN privatization schemes. Our baseline DPGAN from Section 3, with the appropriate choice of up (and without the modifications described in this section yet), outperforms all other GAN-based approaches proposed in the literature (GS-WGAN PATE- GAN, G-PATE, and DataLens) uniformly across both metrics, both datasets, and both privacy levels.
Large batch sizes and adaptive step schedules improve GAN training. Broadly speaking, across both privacy levels and both datasets, we see an improvement from taking larger batch sizes, and then another with an adaptive step schedule. The magnitude of improvement varies.
Comparison with state-of-the-art. In the low privacy/high regime, most of our results are dra- matically better than prior work" - for example, decreasing FID from 48.4 to 13.0 and increasing accuracy from 83.2% to 95.0% on MNIST. In the high privacy/low e regime, improvements are not quite as extreme, but can still be significant (FID for MNIST and FashionMNISTT, and only com- pare negatively to of-te-of-thearr for accuracy on FashionMNIST. Visual comparison for e = 10 results in 5
5.3.2 CELEBA-GENDER
We also report results on generating 32 x 32 CelebA, conditioned on gender at (10, 10-6) 10001-0--DD -DD For these experiments, we used slightly larger models (2.64M and 3. 3.16M parameters for D and g
'We do not compare with two recent works on private generative models (Chen et al., 2022: Jiang et al., 2022), as we believe there are gaps in their privacy analyses. This has been confirmed by the authors of Jiang al. (2022), and the sketch of an argument regarding non-privacy of Chen et al. (2022) has been shared with us by others (Anonymous, 2022).
7

---

Under review as a conference paper at ICLR 2023
MNIST
Fashion
Privacy Level
Method
Reported In
FID
Acc.(%)
FID
Acc.(%)
Real data
1.0
99.2
1.5
92.5
00
GAN
(This work)
3.2
96.8
15.9
80.4
Chen et al. (2020)
179.16
63
243.80
50
DPGAN
Long et al. (2021)
304.86
80.11
433.38
60.98
GS-WGAN
Chen et al. (2020)
61.34
80
131.34
655
PATE-GAN
Long et al. (2021)
253.55
66.67
229.25
62.18
G-PATE
Long et al. (2021)
150.62
80.92
171.90
69.34
10
DataLens
Wang et al. (2021)
173.50
80.66
167.68
70.61
DP-MERF
Cao et al. (2021)
116.3
82.1
132.6
75.5
DP-Sinkhorn
Cao al. (2021)
48.4
83.2
128.3
75.1
DPGAN
19.4
92.9
91.5
71.1
large batches
(This work)
13.2
94.3
66.7
72.1
step schedule
13.0
95.0
56.8
74.8
DPGAN
Long et al. (2021)
470.20
40.36
472.03
10.53
GS- SGAA
Long et al. (2021)
489.75
14.32
587.31
16.61
PATE-GAN
Long et al. (2021)
231.54
41.68
253.19
42.22
G-PATE
Long et al. (2021)
153.38
58.80
214.78
58.12
Datal enn
Wang et al. (2021)
186.06
71.23
194.98
64.78
DP-MERF
Vinaroz et al. (2022)
80.7
73,9
DP-HP
Vinaroz et al. (2022)
81.5
72.3
DPGAN
91.7
77.4
151.9
65.0
large batches
(This work)
66.1
73.7
153.2
66.6
step schedule
56.2
80.1
121.8
Table 1: We gather previously reported results in the literature the performance of various methods for labelled generation MNIST and FashionMNISST. Note that Reported In refers to the source of the numerical result, not originator the approach. For downstream accuracy, we report best accuracy among classi- fiers they use, and compare against our CNN classifier accuracy.
Privacy
Method
Reported In
FID
Acc.(%)
Real data
1.1
96.6
(This work)
GAN
31.5
91.6
DP-MERF
Cao et al. (2021)
274.0
65
10
DP-Sinkhorn
Cao et al. (2021)
189.5
76.3
DPGAN
(This work)
166.8
83.8
Table 2: Comparison to tfate-of-the-ar results on 32 x 32 CelebA Gendeer, targeting (10, 10 DD.".
respectively), and employed large batches (B = 1024) and adaptive discriminator step frequency
with threshold d 0.6. Results are summarized in Table example images are in Figure 11.
DISCUSSION AND RELATED WORK
DP generative models. The baseline DPGAN that employs a DPSGD-trained discriminator was introduced by Xie et al. (2018), and was subsequently studied several works (Torkzadehmahani et al., 2019; Beaulieu-Jones et al., 2019). Despite significant interest in the approach and numer- ous applications to various problems (= 300 citations as of November 2022), we were unable to find studies that explore the modifications we perform or uncover similar principles for improving training. Perhaps as a consequence, subsequent work has departed from this approach, examining alternative privatization schemes for GANS (Jordon et al., 2019; Long et al., 2021; Chen et al., 2020;
We group per-class unconditional GANs together with conditional GANs under the DPGAN umbrella. "These results presented graphically the paper. Exact numbers be found their code.
8

---

Under review as a conference paper at ICLR 2023
(
3
5
7
9
9
DP-CGAN
42
46
3
4
7
9
3
/
J
3
7 7
J:
7 9 9 9
DP-MERF
6
1
1
3
6
A
/4
0
2
3
4
7
GS-WGAN
0
/
2
3
4
5
6
7
8 3
()
3
3
9
L
9 9 9 9 9 9
DP-Sinkhorn
2
3
4
8
0
1
2
3
4
5
6
! 8 8 8
0
2
3
4
6
7
Ours
0
t
2
3 7
4 4
5
7 7
0
2
5
6
0
2
3
4
5
6
7
8
Figure 5: MNIST and FashionMNIST results (10, 10 10-5) ods from (Cao et al., 2021).
--DP for different methods. Images of other meth-
Figure 6: 32 x 32 CelehA-Gendar (10, 10 ">-DP, Top: DP-Sinkhomn Bottom: Ours.
Wang et al., 2021). Contrary to their claims, our work shows that these privatization schemes do not outperform DPSGD. Other generative modelling frameworks have been applied to DP synthetic data including VAEs (Chen et al., 2018), maximum mean discrepancy (Harder et al., 2021; Vinaroz et al., 2022), Sinkhorn divergences (Cao et al., 2021), and normalizing flows (Waites Cummings, 2021). We show that well-tuned DPGAN competes with outperforms these approaches.
Custom approaches versus a well-tuned DPSGD. An ongoing debate pertains to the best tech- niques and architectures for private ML. Roughly speaking, there two schools of thought. One investigates novel architectures for privacy, which may be outperformed by more traditional ap- proaches in the non-private setting. Some examples include Chen al. (2018); Cao al. (2021); Vinaroz et al. (2022), a variety of generative models specifically designed to be compatible with dif- ferential privacy. The other focuses searching within the space of tried-and-tested methods that are understood to work well non-privately. Some examples include the works of De et al. (2022); Li al. (2022), who demonstrate that, similar to the non-private setting, large-scale CNN and Trans- former architectures can achieve state-of-thed primary modifications the pipeline are along the lines of changing the batch size, modifying the type of normalization layers, etc., most of which would be explored in proper hyperparameter search in the non-private setting. Our work fits into the latter line: we show that novel generative models introduced for privacy can be outperformed by GANs trained with well-tuned DPSGD.
results for image classification and NLP tasks. The
Tabular data. Our investigation focused on image datasets, while many important applications of private data generation involve tabular data. While Tao al. (2021) find that private GAN-based approaches fail to preserve even basic statistics in these settings, we believe that our techniques may yield similar improvements.
CONCLUSION
Our most important contribution is show that private GANs have been underrated by the research community, and can achieve state-of-the-art results with careful tuning. We hope and anticipate this will inspire the community revisit private GANs, and quickly improve upon our results.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In CCS'16: 2016 ACM SIGSAC Conference on Computer and Communications Security, 2016.
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differen- tially private BERT. CoRR, abs/2108.01622 2021.
Anonymous. Personal communication, 2022. Anonymous for double blind.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science (FOCS' 14), 2014.
Brett K. Beaulieu-Jones, Zhiwei Steven Wu, Chris Williams, Ran Lee, Sanjeev P. Bhavnani, James Brian Byrd, and Casey S. Greene. Privacy-preserving generative deep neural networks support clinical data sharing. Circulation: Cardiovascular Quality and Outcomes, 12(7), 2019.
Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, and Karsten Kreis. ing differentially private generative models with Sinkhorn divergence. In Advances in Neural Information Processing Systems 34 (NeurIPS'21), 2021.
Don't 't generate me: Train-
Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. GS-WGAN: gradient-sanitized ap- proach for learning differentially private generators. In Advances in Neural Information Process- ing Systems 33 (NeurIPS 20), 2020.
Jia-We Wei Chen, Chia-Mu Yu, Ching-Chip Chia Kao, Tzai-Wei Pang, and Chun-Shien Lu. DPGEN: Differ- entially private generative energy-guided network for natural image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR'23) 22), 2022.
Qingrong Chen, Chong Xiang, Minhui Xue, Bo Li, Nikita Borisov, Dali Kaafar, and Haojin Zhu. 2018.
Differentially private data generative models. CoRR,
Soumith Chintala, Emily Denton, Martin Arjovsky, and Michael Mathicu. How to train a GAN? Tips and tricks to make GANs work. GitHub, 2016.
Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, and Borja Balle. Unlocking high- accuracy differentially private image classification through scale. CoRR, abs/2204.13650, 2022.
Cynthia Dwork and Aaron Roth. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Compututer Science, 11-:44-21140) 2014.
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Ilya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In 25th Annual International Conference on the Theory and Applications of Cryptographic Techniques (EUROCRYPT'06), 2006a.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography, 2006b.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems 27 (NIPS'14) 2014.
Frederik Harder, Kamil Adamczewski, and Mijung Park. DP-MERF: Differentially private mean embeddings with random features for practical privacy privacyppreeerving data generation. 24th Inter- national Conference on Artificial Intelligence and Statistics (AISTATS'2)) 21), 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by two time-scale update rule converge a local nash equilibrium. In Advances in Neural Information Processing Systems 30 (7), 2017.
Dihong Jiang, Guojun Zhang, Mahdi Karami, Xi Chen, Yunfeng Shao, and Yaoliang Yu. DP2-VAE Differentially private pre-trained variational autoencoders. CoRR, abs/2208 220.223080.000,, 2022.
10

---

Under review as a conference paper at ICLR 2023
James Jordon, Jinsung Yoon, and Mihaela der Schaar. PATE-GAN: Generating synthetic data with differential privacy guarantees. In 7th International Conference on Learning Representations (ICLR' 19), 2019.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. In 3rd Interna- lional Conference on Learning Representations (ICLR' 15), 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(1 (0:21)2227-2244 1998.
Xuechen Li, Florian Tramèr, Perey Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In 10th International Conference on Learning Representa- tions (ICLR (2),, 2022.
Yunhui Long, Boxin Wang, Zhuolin Yang, Bhavya Kailkhura, Aston Zhang, Carl Gunter, and Bo Li. G-PATE: Scalable differentially private data generator via private aggregation of teacher discrim- inators. In Advances Neural Information Processing Systems 34 (NeurIPS 21), 2021.
Ilya Mironov, Kunal Talwar, and Li Zhang. Rényi differential privacy of the sampled gaussian mechanism. CoRR, abs/1908.10555 10530, 2019.
Marcel Neunhoeffer, Steven Wu, and Cynthia Dwork. Private post-GAN boosting. 9th Interna- tional Conference on Learning Representations (ICLR' 19), 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Ed- ward Z. Yang. Zachary Veto,, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high- performance deep learning library. Advances in Neural Information Processing Systems 32 (NeurIPS' 19), 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In 4th International Conference on Learning Rep- resentations (ICLR'I6) 16), 2016.
Shuang Song, Kamalika Chaudhuri, and Anand Sarwate. Stochastic gradient descent with differ- entially private updates. In 2013 IEEE Global Conference Signal and Information Processing, 2013.
Yuchao Tao, Ryan McKenna, Michael Hay, Ashwin Machanavajihala, and Gerome Miklau. Bench- marking differentially private synthetic data generation algorithms. CoRR, abs/21 12.09238, 2021.
Reihanch Torkzadehmahani, Peter Kairouz, and Benedict Paten. DP-CGAN: Differentially private synthetic data and label generation. In IEEE Conference Computer Vision and Pattern Recog- nition Workshops, (CVPR Workshops' 2019.
Florian Tramèr and Dan Boneh. Differentially private learning needs better features (or much more data). In 9th International Conference on Learning Representations (ICLR'2/). 2021.
Margarita Vinaroz, Mohammad-Amin Charusaie, Frederik Harder, Kamil Adamczewski, and Mi Jung Park. Hermite polynomial features for private data generation. In Proceedings of the 39th International Conference on Machine Learning (ICML'2)) '2)), 2022.
Chris Waites and Rachel Cummings. Differentially private normalizing flows for privacy-preserving density estimation. CoRR, abs/2103. 14068, 2021.
Boxin Wang, Fan Wu, Yunhui Long, Luka Rimanic, Ce Zhang, and Bo Li. DataLens: Scalable privacy preserving training via gradient compression and aggregation. In CCS'21: 2021 ACM SIGSAC Conference on Computer and Communications Securiry, 2021.
Yu- Yu-Xiang Wang. Borja Balle, and Shiva Prasad Kasiviswanathan. Subsampled Rényi differential privacy and analytical moments accountant. 22nd International Conference on Artificial Intel- ligence and Statistics (AISTATS' 19), 2019.

---

Under review as a conference paper at ICLR 2023
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fushion-MNIST: novel image dataset for bench- marking machine learning algorithms. CoRR, abx/1708..77477,2117
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. CoRR, abs/1802.06739, 2018.
Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Gosh, Akash Bharadwaj, Jessica Zhao, Graham Cormode, and Ilya Mironov. Opacus: User-friendly differential privacy library PyTorch. CoRR, abs/2109.12299. 2021.
12

---

Under review as a conference paper at ICLR 2023
A GENERATED SAMPLES
We provide a few non-cherrypicked samples for MNIST and FashionMNIST at e = 10 and 1, well 32 x 32 CelcbA-Gender at 10.
&
6
2
2
8
9
7
?
3
9
4
9
9
4
9
5
9
7
6
3
1
2
Figure 7: Some non-cherrypicked MNIST samples from our method, E = 10.
Figure 8: Some non-cherrypicked PashionMNIST samples from our method, E 10.
Ø
3-
/
4
O
1
S
7
4
2
Ø
8
/
2
1
7
B
5
3
3
4
7
5
Figure 9: Some non-cherrypicked MNIST samples from our method, 1.
B IMPLEMENTATION DETAILS
B.1 MNIST AND FASHIONMNIST TRAINING RECIPE
For MNIST and FashionMNIST,, we begin from an open source PyTorch implementation DC- GAN (Radford et al., 2016) (available at this link) that performs well non-privately, and copy their
13

---

Under review as a conference paper at ICLR 2023
Figure 10:
Some non-cherrypicked FashionMNIST samples from our method, s 1.
Figure 11: Some non-cherrypicked CelebA samples from our method, e = 10.
training recipe. This includes: batch size B = 128, the Adam optimizer (Kingma & Ba, 2015) with parameters (a = 0.0002, 31 0.5, B2 0.999) for both G and D, the non-saturating GAN loss (Goodfellow et al., 2014), and 5-layer fully convolutional architecture with width parameter 128.
To adapt it our purposes, we make three architectural modifications: in both G and D (1) move all BatchNorm layers (which are not compatible with DPSGD); (2) add label embedding layers to enable labelled generation; and (3) adjust convolutional/transpss. convolutional stride lengths and kernel sizes as well as remove the last layer, in order to process x 28 x 28 images without having resize. Finally, remove their custom weight initialization, opting for Py defaults.
Our baseline non-private GANs are trained for 45000 steps. We train our non-private GANs with poisson sampling well: for each step of discriminator training, sample real examples by including each element of our dataset independently with probability B/n, where is the size of dataset. We then add fake examples sampled from form our fake/real combined batch.
B.2 LARGE BATCH SIZE HYPERPARAMETER SEARCH
We scale up batch sizes, considering B E (64, 128, 512, 2048}, and search for the optimal noise scale a and npp For B 128 targeting (0.6, 1.0. 1.4). We choose candidate noise scales for other batch sizes as follows: when considering batch size 128k, we search over D==10
10, we search over three noise scales, Dg
1. We also target the high privacy (5 = 1) regime. For = 1, we multiply all noise scales by 5, Ex = {5o : o E Eg 10). We search over grid n 21 5, 10, 20, 50, 100, 200, 500}. Due to compute limitations, omit some values that we are confident will fail (c.g., trying ao = 1 when mode collapse occurs for
128k
{VT
1B-128)
"D = 5).
14

---

Under review as a conference paper at ICLR 2023
C ADDITIONAL DISCUSSION
GANhacks. Guidance in the non-private setting (tip 14 of Chintala et al. (2016)) prescribes to train the discriminator for more steps in the presence of noise regularization approach used in non-private GANs). This is the case for DP, and is our core strategy that yields the most significant gains in utility. We were not aware of this tip when we discovered this phenomenon, but it serves as validation of our finding. While Chintala et al. (2016) provides little elaboration, looking at further explorations of this principle in the non- con-prrvate setting may offer guidance for improving DPGANs.
15


Reviews:
[{"review_id": "ulppKDiNDo", "reviewer": "Reviewer_wVny", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a simple strategy to improve DP-GANs by using more update steps for discriminator, adopting a step scheduler, and taking a larger batch size. The authors provide empirical findings to justify such improvement by linking generation quality with the discriminator accuracy though the training process. Experiments on MNIST and FashionMNIST show very promising results in terms of generation quality.", "summary_of_the_review": "This paper has interesting empirical findings and promising results, while at current stage the proposed solution is not mature enough to meet the standard of ICLR. However, along its direction, this work is potential to have an impact in the future, probably with an elaborated method or some theoretical support.", "strength_and_weaknesses": "Strength:\n\n1. The proposed techniques are very effective to achieve much better FID on MNIST and FashionMNIST.\n\n2. The empirical explanation of balancing the discriminator and maximizing generator steps taken when discriminator accuracy is high, is interesting and inspirational.\n\nWeaknesses:\n\n1. Overall, I think the proposed method is currently limited to an engineering technique, which might not be mature enough. However, I do believe it is highly potential to develop a principled method with high impact along the current direction. For example, I would recommend further elaborate on the step scheduler to be more adaptive and include more ablation studies. The authors might be able to draw some inspirations from StyleGAN2-ADA for such paradigm.\n\n2. The improvement of utility accuracy is not as significant as FID. It would also be better to include an experiment on some larger datasets such as CelebA.\n\n3. The writing and the organization could be improved.\n- Although fancy, the title is not very informative.\n- Figure 1 and figure 2 share the same purpose.\n- Section 3.3 is duplicate to the previous content.\n- Section 5.1 and 5.2 are not well aligned with the theme of Section 5 and the results are not well linked to the discriminator accuracy.\n- In Algorithm 1, some symbols are used without definition.\n- In Table 1, \"(This work)\" is ambiguous.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "IKSvXe-aRgK", "reviewer": "Reviewer_hSom", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors revisit the DPGAN paper and state that the added noise to discriminator training disrupts the balance between the generator and discriminator training. They show that by tuning the number of update steps taking by the discriminator for every generator update step, specifically, taking more steps significantly improves the results. They show that for MNIST \\at epsilon = 10 their private GAN FID goes from 48.4  to 13.0 and the downstream accuracy of the classifier goes up from 83.2% to 95.0%. ", "summary_of_the_review": "The paper present a very slight modification, a focus on a parameter change in the DPGAN method. It propose to increase the discriminator update frequency when noise is introduced to the discriminator (as in the DPGAN training case). The paper discuss the motivation to their method at length. Finally, the authors show inconclusive experiments, where the method works for some regime of the noise level. ", "strength_and_weaknesses": "### Strength\nThe paper provides a simple yet effective method to improve the quality of the generated data of DPGAN. The experiments show significant improvement in the tested benchmarks. The authors discuss in length about whey more steps are needed in the case of DPGAN. In addition, the authors discuss the role of the batch size in DPGAN settings and the additional benefits of scheduling the discriminator frequency update (start low then increase). \n\n### Weaknesses\nThe novelty of the work is marginal. The method work best at \\epsilon = 10, but on high privacy/ low \\epsilon regime sometime the results are even worse.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "DfGIsGAbPO", "reviewer": "Reviewer_dASH", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper provides two empirical findings on how to train differential private gans: larger batch size, and more discriminator steps. Experimental results show superior performance against existing baselines.", "summary_of_the_review": "Overall, this paper is a practical guide for training private gans. However the findings are empirical and rather trivial. In addition, some details are missing.", "strength_and_weaknesses": "Strength\n\n1. In the literature of differential privacy gans, it seems like the utilization of more recent architectures and training tricks (e.g. StyleGAN2,3, …) for higher quality samples are mostly unexplored. This paper provides a good point that these “tricks” could largely boost the performance of private gans.\n\nWeakness\n\n1. From my perspective, this work is with limited novelty. Since larger batch size and take more discriminator steps has been studied widely in normal gan papers. It seems like natural attempts to try these tricks on private gans.\n2. Several missing details. For example, how does (10, 10^-5)-DP calculated from B=128,sigma=1,C=1,T=450000 (in section 3.1)? Please provide the formal equation on calculating this. In addition, noise scales, n_D, and batch size also affect privacy, in section 5.1, how can you targeting the same \\epsilon with different values of these?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject