Private Set Generation with Discriminative Information
Authors: Dingfan Chen
Raouf Kerkouche
Mario Fritz
Venue: NeurIPS 2022 Accept (2022)
Keywords: Differentially private data generation

Abstract: Differentially private data generation techniques have become a promising solution to the data privacy challenge –– it enables sharing of data while complying with rigorous privacy guarantees, which is essential for scientific progress in sensitive domains. Unfortunately, restricted by the inherent complexity of modeling high-dimensional distributions, existing private generative models are struggling with the utility of synthetic samples. In contrast to existing works that aim at fitting the complete data distribution, we directly optimize for a small set of samples that are representative of the distribution, which is generally an easier task and more suitable for private training. Moreover, we exploit discriminative information from downstream tasks to further ease the training. Our work provides an alternative view for differentially private generation of high-dimensional data and introduces a simple yet effective method that greatly improves the sample utility of state-of-the-art approaches.


Private Set Generation with Discriminative Information
Dingfan Chen Raouf Kerkouche Mario Fritz
CISPA Helmholtz Center for Information Security {dingfan, chen, raouf kerkouche, fritz/@cispa.de
Abstract
Differentially private data generation techniques have become promising solution to the data privacy challenge - enables sharing of data while complying with rigorous privacy guarantees, which is essential for scientific progress in sensitive domains. Unfortunately, restricted by the inherent complexity of modeling high- dimensional distributions, existing private generative models are struggling with the utility of synthetic samples. contrast to existing works that aim at fitting the complete data distribution, directly optimize for small set of samples that are representative of the distribution under the supervision of discriminative information from downstream tasks, which is generally easier task and more suitable for private training. Our work provides an alternative view for differentially private generation of high-dimensional data and introduces simple yet effective method that greatly improves the sample utility of state-of-the-ar approaches.
Introduction
Data sharing vital for the growth of machine learning applications in numerous domains. However, in many application scenarios, data sharing is prohibited due to the private nature of data (e.g.. indi- vidual data stored on mobile devices, medical treatments, and banking records) and the corresponding stringent regulations, which greatly hinders technological progress. Differentially private (DP) data publishing [8, 9, [1] provides a compelling solution such challenge, where only a sanitized form of the data is publicly released. Such sanitized synthetic data can be leveraged as if it were the real data, analyzed with established toolchains, and can be shared openly the public, facilitating technological advance and reproducible research in sensitive domains.
Yet, generation of high high-diimnssioal data with DP guarantees is highly challenging and traditional DP algorithms designed for capturing low-dimensional statistical characteristics are not applicable this task [28, 13, 4, 35]). Instead, inspired by the great successes of deep generative models in learning high-ding sigh-dimensional representations, recent works [5, 6, 40, 41, 2, 12] adopt deep generative neural networks as the underlying generation backbone and incorporate the privacy constraints into the training procedure, such that any privacy leakage upon disclosing the data generator is bounded.
However, these methods have common shortcomings: (i) deep generative models are known to be data- demanding [16], which becomes even harder to train when considering the privacy constraints [6, 5]; (ii) they do not guarantee any optimal solution for the downstream task (c.g. classification). In fact, existing models are still struggling to generate sanitized data that is useful for downstream data analysis tasks. For example, when training convolutional neural network (ConvNet) classifier on the private generated data, the highest test accuracy reported in literature is < 85% for MNIST dataset with (E. (10, 10 10-5) 5) [5], which lags far behind the discriminative baseline (> 98% with (5,8) (1.2, 10 5) [32]) and makes private generative models less appealing for many practical
scenarios with data analysis as the end goal.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this work, we learn to synthesize informative samples that are privacy privacy-preseree preserving and are optimized to train neural networks for downstream tasks. In contrast existing approaches, we directly optimize small set of samples instead of the deep generative models that is notoriously difficult to train in private manner. Moreover, we exploit discriminative information from downstream tasks to guide the samples towards containing more useful information for downstream analysis. Compared to existing works, we improve the task utility by a large extent (up to 10% downstream test accuracy improvement over state-of-the-arr approach), while still preserving the flexibility and generality across varying configurations for downstream analysis. As an added benefit, our formulation naturally distilled the knowledge of original data into much smaller set, which largely saves the memory and computational consumption for downstream analysis.
We summarize our main contributions as follows.
We present a new perspective of private high- nigh-dimensional data generation, with which we aim to bridge the utility and generality gap between the private generative and discriminative models. We believe this alternative view opens up new possibilities in different research directions ranging from private analysis to generation.
We introduce simple yet effective method for generating informative samples that are optimized for training downstream neural networks, while maintaining generality as well as reducing the memory and computation consumption added benefits.
Experimental results demonstrate that, in comparison to existing works, our work improves the sample utility by large margin and offers superior practicability for real-world applica- tion scenarios.
2 Related Work
Differentially Private Generative Models Training deep generative models in private manner has become the default choice for private -ighndimensional data generation. Existing methods typically adopt differentially private stochastic gradient descent (DP-SGD) [1, 31, 6, 5] or Private Aggregation of Teacher Ensembles (PATE) [23, 24, 20, 36] to equip the deep generative models with rigorous privacy guarantees. Despite significant progress in mitigating training instabilities and improving generation (visual) quality, existing works are still far from being optimal in terms of the sample utility. This is mainly because existing works are attempting to solve a problem that is inherently hard and almost impossible to be solved accurately under the current private training framework. In contrast, we directly optimize the samples (rather than the deep generative models that are much harder to train in a private setting) and exploit the knowledge from general class of downstream tasks that can be employed on the samples to further guide the training.
Coreset Selection and Generation Our work is largely motivated by recent success in distilling a large dataset into much smaller set of representative samples, i.e., the coreset. For example, samples from a dataset are selected to be representative based on their ability to mimic the gradient signal [22], hardness to fit [30], distance to the cluster centers [38, 27], etc. Instead of selecting samples from the dataset, our work focus on synthesizing informative samples from scratch [37, 43, 42, 19] under DP constraints, and optimizing the sample utility for training downstream neural networks. While recent work [7] has shown promising results in dataset distillation under privacy concerns, obtaining strict privacy guarantees has remained challenging. Our set generation formulation is also similar in spirit to works in the field of private queries release [28, 13, 4, 14] which synthesize set of pseudo-data (under DP guarantees) that is representative of the original data in answering linear queries. However, as neural networks exhibit highly nonlinear properties, methods targeted at linear queries are generally not applicable to our case and are algorithmically distinct from approaches designed for neural nets.
3 Background
We consider the standard central model of DP in this paper. We below present several definitions and theorems that will be used in this work. Definition 3.1. (Differential Privacy (DP) [8]) A randomized mechanism M with range R. is (c, 6)- DP, if
PD) e O] e Pr(M(DD)) O)+8
(I)
2

---

holds for any subset of outputs O S R and for any adjacent datasets D and D', where D and 'D' differ from each other with only one training example, ie.. D' M corresponds to the set generation algorithm in our case, is the upper bound of privacy loss, and ō is the probability of breaching DP constraints. DP guarantees the difficulty of inferring the presence of an individual in the private dataset by observing the generated set of samples M(D).
{æ} for some x (or vice versa).
Our approach is built on top of the Gaussian mechanism defined as follows.
Definition 3.2. (Gaussian Mechanism [10]) Let f X + IRd be an arbitrary d-dimensional function with sensitivity being
22f max Ma(x) = f(x) +N(,,0")))
D,D' || [(D) f
over adjacent datasets and D'. The Gaussian Mechanism Ma. parameterized by a, adds noise into the output, i.e.,
Ma is O)-D) for a 2 V 2 (1.2 (1.25/5)a2//e.
Any privacy cost bounded upon releasing the private set of generated data due to the closedness of DP under post-processig.
Theorem 3.1. (Post-processing [10) If M satisfies (s. 6), F o M will satisfy ()-D) for any data-independent function F with denoting the composition operator.
4 Method
We consider a standard classification task where we are given a private dataset D = {(x,, y/ ) with æ; E Rd the feature, y; E {I,..., L. the class label, N the number samples, the number of label classes. Our objective is to synthesize set of samples S = {(x5 uf } such that (i) samples in S have the same form as data D; (ii) neural network trained on S should maximally match generalization performance of deep neural network that is trained on D; (iii the privacy leakage of D when releasing S is upper bounded by a pre-defined privacy level (e, 6))
Let F(;; ( and F(:; 05) be the deep neural networks parameterized by 0 and 0 that are trained on D and respectively. The objective can be formulated as:
(( (x; v)) ~ (e (at.irro(((FFae (1; 05);
where l denotes the loss function (c.g., cross- for the classification task) and the expectation taken over the real data distribution Pp.
Equation can be naturally achieved once @ z AD_ In particular, when given the same initialization og = O solving for 0 z 0.P at each training iteration leads to @ = 0 desired. This can be achieved by optimizing the synthetic set such that it yields a similar gradient as if the network is trained on the real dataset each iteration t:
S Cais (VeL(S, 0z), FeL(D,0.)
(5)
where VoL(S, 0.)) corresponds to the gradient of the classification loss on the synthetic set S, VoL(D, 0,) denotes the stochastic gradient on the real data, and Lais is a sum of cosine distances between the gradients at each layer [43, 42] (See supplementary material for more details).
To mimic the training procedure, S and the network F(:;0) are updated jointly in an iterative manner, where in each outer iteration the S trained minimize the gradient matching loss Lais and in each inner iterations the network parameters 0 are optimized towards minimizing the classification loss on the synthetic set S. Moreover, optimized over multiple initializations of network parameters fn to ensure the generalization ability of S over different random initialization when
Cai
Eais
2r Po
9o.
M.c
90.
ins loop
pldcr Joxp
samitive
Figure 1: Illustration for the training pipeline.
3

---

training a downstream model. The objective can be summarize as follows [37, 43,44
T-1 E (VoLls 0z). VoL(D,
S = arg min Pop
(6)
t=U
where Poo stands for the distribution over the initialization of network parameters.
We incorporate DP constraints by sanitizing the stochastic gradient real data VoC(D, 0,) at each outer iteration, while leaving the inner iterations unchanged as their privacy is guaranteed by the post-processing theorem 3.1. The final objective can be formulated as follows:
T-1
S = arg min Eg, Poo
CCats (90. 96.)
8
2=0
where we use Sg. to denote the parameter gradient on D that is sanitized via Gaussian mechanism 3.2, and og. to denote the parameter gradient on S. The whole pipeline is summarized in Algorithm 1. We use the subsampled Renyi-DP accountant [1, 21] to compute the overall privacy cost accumulated for iteratively updating S. Note that the training procedure and the privacy computation as simple as training a classification network with DP-SGD, which in general has lower difficulty than training a DP deep generative models done in existing works (witnessed by a significant performance gap in terms of the classification accuracy). Moreover, in contrast to previous works, our synthetic S directly optimized for downstream tasks, which naturally leads superior downstream utility to existing approaches.
approximately
Algorithm 1: Private Set Generation (PSG)
Input: Dataset D = (ar, y1) } learning rate for update network parameters та and Ts. batch size B, gradient clipping bound C, number of runs R, outer iterations T, inner iterations J, batches K, desired privacy cost E given a pre-defined 8
Output: Synthetic set S
Compute the required DP noise scale a numerically [1, 21] so that the privacy cost equals & after the training: Initialize synthetic S (features 2 are from Gaussian noise; labels balanced set depending on the pre-defined number of samples per class):
for run in !I,,.., Rt do Initialize model parameter 0g ~ Pooi for outer_iter in (I, T} do
0.41 = 0;
for batch index in {1,..., K} do
Uniformly sample random batch {(z,, 11) from D; for each (ar,; yi) do
// Compute per exameee gradients real data 90. (æ:) ((F(x;; 0.),
// Clip gradients
D g6. (xx) gb. (z,) min(1, CL/
end
// Add noise to average gradient with Gaussian mechanism D
90. B D. ( (a.) + NN
// Compute parameter gradients on synthetic data and update S
Gē. Vo 0,) 1 EM ((F(s;;))
S S - TS V
end
end
for inner_ierr in {1,..., J do
// Update network parameter using
02 0, тo VoL(S,0,)
end
end
return Synthetic set S

---

(a)
(b)
MNIST
FashionMNIST
MNIST
FashionMNIST
E=1
c=10
c=1
E=10
spc=10
spc=20
full
spc= 10
spc=20
full
DP-CGAN
52.5
50.2
Real
93.6
95.9
99.6
74.4
77.4
93.5
G-PATE
58.8
80.9
58.1
69.3
DPSGD
96.5
82.9
DataLens
71.2
80.7
64.8
70.6
DP-CGAN
57.4
57.1
52.5
51.4
53.0
50.2
GS- AG-N
84.9
63.1
GS-WGAN
83.3
85,5
84.9
58.7
59.5
63.1
DP-Merf
72.7
85.7
61.2
72.4
DP-Merf
80.2
83.2
85.7
66.6
67.9
72.4
DP-Sinkhorn
83.2
Ours
94.9
95.6
75.6
77.7
Ours (spc=20)
80.9
95.6
70.2
77.7
Table 1: Test accuracy (%) on real data of downstream ConvNet classifiers when training on
the synthetic set with 8 = 10 (a) Comparison under different privacy cost E E {1.10}. (b) Comparison when varying the number of samples per class (spe) for training the downstream ConvNet with 10, while "full" corresponds to 6000 samples per class. We show the results when training on real data non-privately and with DPSGD [1] as reference.
5
Experiment
5.1 Classification
We first compare private set generation (PSG) with existing DP generative models on standard classification benchmarks including MNIST [17] and FashionMNIST [39].
Setup. We use by default ConvNet with 3 blocks where each block contains Conv layer with 128 filters, followed by Instance Normalization [33], ReLU activation and AvgPooling modules, and a fully connected (FC) layer the final output layer. We initialize the network parameters using Kaiming initialization [15] and the synthetic samples using standard Gaussian. We report the averaged results over 3 runs of experiments for all the comparisons. We list below the default hyperparameters used for the main experiments and refer to the supplementary material for more details: Clipping bound C' 0.1, R = =1000 for 10 (and 200 for 1), number of samples per class (spc) E {10, 20), K = 10, T = 10 for spc= 10 (and =20 for spc=20).
Comparison to state of the art. We show in Table la the results of, to the best of our knowledge. all existing DP high-dimensional data generation methods (whose validity has been justified via peer review top-tier conferences) that report results the benchmark datasets we consider. These include DP-CGAN [31], G-PATE [20], DataLens [36], GS- GS-WGAN [6]. DP-Merf [12], DP- Sinkhorn [5]. For methods that are not open-sourced, paper. As shown in Table la, our formulation results in significant improvement in the sample utility (measured by test accuracy real data) for training downstream classification models. Specifically, the improvement is consistent and significant (around 5-10% increase over different configurations) for both the low privacy budget regime (s=1) (around 8-9% improvement and a relatively high privacy regime (s=10) where all the investigated methods achieve convergence (around 10% and 5% increase test accuracy for MNIST and FashionMNIST. respectively). Note that in contrast to most existing methods that show superiority only for a certain range of privacy levels, our improvement covers a wide range, if not all, of practical scenarios spanning across different privacy levels.
report the original results from the published
SOTA in this case)
We then focus the open-sourced methods that strictly comparable (e.g., G-PATE and DataLens provide data-dependent e, i.e., publishing value will introduce privacy cost and are thus not directly comparable)
and conduct comprehensive investigation through different angles.
Memory and computation cost. We additionally show that our method is the only one that simulta- neously shows advantages in reducing the memory and computation consumption of downstream analysis. As shown in Table 1b, training the classifier with full (6000 samples per class) size of sam- ples in most cases yields an upper bound for the test accuracy, while training randomly subsampled smaller sets will decrease the performance, unless the generated samples are not informative such that they can be harmful to the downstream tasks (e.g., for DP-CGAN). In contrast, we directly optimize
5

---

Fashion
ConvNet
LeNet
AlexNet
VGG11
ResNei18
MLP
ConvNet
LeNet
AlexNet
VGGII
ResNet18
MLP
99.6
99.6
DP-CGAN
50.2
52.6
52.1
54.7
51.8
54.3
50.2
52.6
52.1
54.7
51.8
54.3
GS-WGAN
84.9
80.5
87.9
DP-Merf
85.7
87.2
84.4
81.7
81.3
85.0
72.4
67.9
64.9
70.1
66.7
73.1
Ours (0pc=10)
94.9
91.3
90.3
93.6
94.3
86.1
75.6
68.0
66.2
74.7
72.1
62.8
Ours (spe=20)
95.6
93.0
92.3
94.5
94.1
87.1
77.7
68.0
59.1
76.8
70.8
62.2
Table 2: Comparison of generalization ability across different network architecture with (c, 6) (10, 10-5), Our generated set is optimized with ConNeer, while the downstream classifiers are of different architecture. The classifiers are trained on the full synthetic set for baseline methods.
Twe Acoracy
(spc=10) Our (aps=200)
(ppo110
Oata (ap-220)
GS WGAN
CS-V WGAN
CP CCAN
DP-CGAN
Epalon
Eps
(a) MNIST
(b) FashionMNIST
Figure 2: Comparison of the convergence rate to existing private generative models with iteratively accumulated privacy cost. X-axis: privacy cost Y-axis: utility (i.e., test accuracy (%)) for training downstream ConvNet classifiers.
to compress the useful information into small set of samples and naturally save the memory and computation consumption for downstream analysis tasks.
Generalization ability across different architectures. One natural concern of our formulation could be the generalization ability to unseen situations. While we exploit discriminative information to guide the training, we (in principle) inevitably trade the generality off against task-specific utility, leaving no performance guarantees for new models. Interestingly, as shown in Table 2, we find that our generated set still provides better utility than all baseline methods in most cases, even though the models for evaluation have a completely different architecture from the one we used for training. generated set does not work well is for training MLPs. We conjecture
The only case where that it is due to the difference in the network properties that result in distinct gradient signals: for example, layers in MLPs are densely connected while being sparsely connected in ConvNets, and Convolutional layers are translation equivalent while FC layers in MLPs are not. Moreover, we argue that this may not be bug, but feature. Note that the reference results real data also indicate that the MLP is inferior to other architectures while models with ConvNet, VGG, or ResNet architecture perform well in most cases. In this regard, results our generated set generally align well with the result on real data, which suggests the possibility of conducting model selection with our private generated set.
Convergence rate. For most private (gradient- based) iterative methods, the privacy cost accumulates in each training iteration, and thus faster convergence is highly preferable. We show in Figure 2 the training curves where the y-axis denotes the utility and the x-axis corresponds to the privacy. We observe that our method generally has much faster convergence rate than existing methods that need to accumulate the privacy cost for each iteration. In particular, our method already achieves a decent level of utility with < 2 which is much lower than the privacy budget used in most previous works (normally & 10).
5.2 Application: Private Continual Learning
The utility guarantee of our formulation requires that the network architecture known to the data provider/generrtor.. Fortunately, it is not rare case in practice. In particular, our method is naturally

---

110
100
90
Aecgga Tai Accreey 191
Avenuee Twe Arc.ssty (4)
80
70
DeSGD (1 100
DR-Mol
00
m0)
DP5GD (
Over (1- 1
30
DPSGD
Mer (1)
40
DF Maf
30
Oue (r 100
10
20
Traning Stage
Traring Stage
(a) SpliMMNIST
(b) SplitFashionMNIST
Figure 3: Comparison for private training in the continual learning setting with å 10-5 and
different e. X-axis: training stage, Y-axis: averaged test accuracy (over all the stages till the current one). We use a ConvNet classifier this case and set spc 10 for our method and spe 6000 for DP-Merf as default.
applicable cases where (i) there are multiple parties involved in training a model and they agree on one common training protocol (i.e., the network architecture is known all participants); (ii) each party has its own data whose privacy need to be protected (i.e., the training need to be DP); (iii) data on each party exhibit distinct property and is all informative for the final task (i.e., synthetic set of representative samples that capture such properties would greatly aid the final task).
One example is continual learning [18, 29] where the training of the classification network is split into stages. Here we consider a setting adjusted the DP training: to protect the privacy of its data, each party is responsible for a different training stage where it performs DP training of the model on its data, and subsequently delivers the DP model to the party responsible for the next training stage. Note that the raw data would not be transferred as otherwise the privacy would be leaked.
We conduct DP training on the SplitMNIST and SpiidFashionMNIST datasets where the data is partitioned into 5 parts (based on the class labels, which corresponds to the class-incremental [27] setup) and we assume each part is held by one party and can not be accessed by others for privacy sake (See supplementary material for more details). We show in Figure 3 (green curves) the baseline results of DP training model under the private glass-incrementaa setting (i.e., each party finetune the model is obtained from the previous stage on its own data using DP-SGD). Apparently, this naive training scheme leads to catastrophic forgetting of information learned the early stages. Even worse is that the common strategy to cope with this issue requires transferring small set of real data to other parties such that it can be replayed in the later training stage [27, 3, 25], which is not directly applicable to the private setting transferring the data breaks privacy. In contrast, private generation methods can seamlessly applied to this case, where a set DP synthetic samples transferred to enable the final model to learn the characteristics of each partition of data. In particular, our formulation better suitable for this setting than other generation methods as the network architecture is known all participants and samples can be tailored to the specific network via our formulation. This is verified in Figure 3, where our synthetic samples are generally more informative for training the classifier when compared to DP-Merf - the overall best existing works in terms of the downstream utility. Moreover, as our formulation condenses the information into a small set of samples by construction, also enjoy the advantages when considering the computation, storage, and communication cost.
Discussion
In this section, we present several key factors that distinguish our approach from existing ones and discuss possible concerns regarding our private set generation formulation.
Trade-off between Visual Quality and Task Utility. Our formulation is designed for optimizing the utility of downstream analysis tasks instead of the visual appearance as done in previous works, thereby leaving no performance guarantee for the visual quality of the synthetic samples. Moreover, the optimization of the private synthetic set unconstrained and unregulated over the whole data space, with the gradient signal the only guidance. As the data to gradient mapping is generally
7

---

8
2
8 3 9.188998
8
8
3// 3
8 /
MNIST (w/ prior)
(b) FashionMNIST (w/o prior)
2
S 2
S 5555655
5
6
6 6
777
2 f
2
7 1 9
(c) MNIST (with prior)
(d) FashionMNIST (with prior)
Figure 4: Our synthetic samples under (c, 8) (10, 10 10-5) for MNIST and MNIST datasets
with or without (w/o) incorporating a DCGAN generator network as image prior.
with prer (epc.1)
prio (epe..1
w/o prior (opo-
wil prlor (sp0-10)
[ac-)20)
with prct (spent)
prio (ppc:1
with peior (epc
with prlwe (ppx-
wihh
(wa-220)
Epsioo
Epston
(a) MNIST
(b) FashionMNIST
Figure 5: Comparison of the convergence rate when training with or without (w/o) the image prior from DCGAN. X-axis: privacy cost s, Y-axis: test accuracy (%) for training downstream ConvNet classifiers.
non-injective (i.e., different data can results in the same gradient), searching for the correct data given the gradient is an indefinite problem, which inevitably leads to outcomes that are out of the data manifold in practice. This can be seen in the first row of Figure 4 where we plot our private synthetic samples trained under the default setting.
Recall that one key difference between our formulation and existing works is that: we directly optimize for a set of samples instead the deep generative models. We then take a further step and investigate whether this difference is the key factor that determines the samples' visual quality. To do this, we employ a DCGAN [26] model from [6] as the generator backbone (denoted as G). let & to be outputs of G. and then optimize over the network parameter of G using the gradient matching loss as in Equation 5. Mathematically, this transforms Equation 7 into:
min Eeg- Pao
Laaiis 96.) t=0
with S= (G( 3
(8)
where is the parameter of G, 2; is random Gaussian noise (fixed during training). Basically, this formulation restricts the synthetic images to within the output space of G, and the inductive bias introduced by the convolutional structure serves deep image prior [34] to regularize the visual appearance of the synthetic images.
We show the synthetic samples in the second row of Figure 4, and compare the utilities with our original formulation in Figure We observe that the prior from the deep generative model is indeed important for the visual quality. However, interestingly, better visual quality does not mean better utility. Specifically, optimizing over the parameter of generator exhibits a slower convergence than directly optimizing the samples, while the final performance is also inferior (See quantitative results in Table 3). This gives several important indications that help inform future research in this field: (i) the goal of achieving better downstream utility may be incompatible with the goal of achicving
8

---

better sample visual quality, while dedicated efforts towards different goals are necessary; (ii) deep generative models may not be the best option for the task of private data generation as they result in suboptimal utility (mainly due to its slow convergence), which questions the current default way of thinking in this field.
Scalability & Transparency. We discuss the possible issues when scaling to more complicated datasets which: (i) contains large number of label classes; (ii) are diverse and require large number of samples to capture the statistical characteristics of the data distribution. For (i), the complexity of our (and all the other) approaches will definitely increase the number of label classes increases. When considering the number of variables that need to be optimized, the complexity increases linearly for our case, while for all methods (that optimize over the network parameters) the increase is no less than ours. While the application of DP deep learning (of discriminative models) to datasets with >1 label classes is rare, we anticipate that dealing with a much larger number of label classes too ambitious for DP generative modeling for now. For (ii), we conduct the experiment when varying the number of samples per class and present the results in 4, where we indeed observe the training difficulty when the number of samples increases. We conjecture that it is mainly because the gradient signals for updating the synthetic samples get sparser when the number increases, which results in a lower convergence rate and thus worse results especially when the allowed privacy budget is low. However, is arguable whether this is a shortage as smaller amounts of samples allow more savings in the storage and computation consumption while providing greater transparency of downstream analysis.
MNIST
FashionMNIST
MNIST
FashionMNIST
1
10
20
10
20
10
20
50
10
20
50
w/ prior 81.4
94.9
95.6
66.7
75.6
77.7
with prior
90.6
63.0
70.2
70.7
81.4
94.9
95.6
94.0
66.7
75.6
77.7
71.3
Table 3: Test accuracy (%) real data of downstreum ConvNet classifier with or without (wlo) adopting image
Table 4: Test accuracy (%) on real data of downstream ConvNet classifier when varying the numbers of samples per class (spe) under
prior from DCGAN under (a, 6) = (10, 10 10-").
(e, = (10, 10-5).
Generality and Expressiveness. Our formulation focus on the task of training downstream neural networks, and thus have no guarantees for other (and more general) purpose. In contrast, deep generative models are designed for capturing the complete data distribution and, once perfectly trained, can be applied to more general cases. While our formulation seems to be inferior in this regard, we argue that this should not be major shortcoming that outweighs the advantages: First of all, while deep generative models in principle have much greater expressiveness than small set of samples, such upper bound is hard, if not impossible, to be achieved in the privacy learning setting. Instead, compromising the upper bound for more achievable target is worthy and shows great improvement over existing works as demonstrated in section 5.1. Moreover, our formulation generalizes seamlessly to any gradient-based learning methods that a downstream analyst may adopt. While such methods already cover the most part of the possible analysis algorithms that could be adopted for high-dimensional data, we believe that our approach does exhibit a good level of practical applicability.
7 Conclusion
We introduce a novel view of private high-dimensional data generation: instead of attempting to train deep generative models in DP manner, directly optimize set of samples under the supervision of discriminative information for downstream utility. We present simple yet effective method that allows synthesizing a small set of samples that representative of the original data distribution and informative for training downstream neural networks. We demonstrate via extensive experiments that our formulation leads to great improvement over state-of ffte-offthhearr approaches in terms of the task utility, without losing the generality for performing analysis tasks in practice. Moreover, our results question the current default way of thinking and provide insights for further pushing the frontier in the field of private data generation. Our code has been open-sourced to facilitate research in the related field.

---

Broader Impact
The widespread availability of rich data has fueled the growth of machine learning applications in numerous domains. However, in real-world application scenarios, data sharing is always prohibited due to the private nature of data and the corresponding stringent regulations, which greatly hinders technological progress. Our work contributes to making the latest advances in mrivacyprreeevvig data generation. particular, our method improves the data utility compared to the state-of-the-ee data generation methods. In particular, we show the success high dimensional data, which will be key to bringing those methods to a broader range of applications. Consequently, expect broad adaptations of our technique and hence positive societal impacts. We are not aware of any extra negative societal impacts beyond generic risks of ML technology in general.
Acknowledgments
This work is partially funded by the Helmholtz Association within the projects "Trustworthy Federated Data Analytics (TFDA)" (ZT-I-OO1 4), and "Protecting Genetic Data with Synthetic Cohorts from Deep Generative Models (PRO-GENE-GEN)" (ZT-1-PF-5-23), Additionally, this work is supported in part by ELSA European Lighthouse on Secure and Safe Al funded by the European Union under grant agreement No. 101070617. Views and opinions expressed are however those of the authors only and do not necessarily reflect those the European Union or European Commission. Neither the European Union nor the European Commission can be held responsible for them. We also acknowledge Max Planck Institute for Informatics and "Helmholtz AI computing resources" (HAICORE) for providing computing resources.
References
[1] M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2016.
[2] B. K. Beaulieu-Jones, Z. S. Wu, C. Williams, and C. S. Greene. deep neural networks support clinical data sharing. biorxiv. DOI, 2017.
generative
[3] E. Belouadah and A. Popescu. 112m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2019,
[4] A. Blum, K. Ligett, and A. Roth. A learning theory approach noninteractive database privacy. Journal of the ACM (JACM), 2013.
[5] T. Cao, A. Bie, A. Vahdat, S. Fidler, and K. Kreis. Don't generate me: Training differentially private generative models with sinkhorn divergence. Advances in Neural Information Processing Systems (NeurIPS), 2021.
[6] D. Chen, T. Orekondy, and M. Fritz. Gs-wgan: A gradient-sanitized approach for learning differentially private generators. In Neural Information Processing Systems (NeurlPS), 2020.
[7] Dong, B. Zhao, and Lyu. Privacy for free: How does dataset condensation help privacy?
In International Conference on Machine Learning (ICML), 2022.
[8] C. Dwork. Differential privacy: survey of results. In International Conference on Theory and Applications of Models of Computation (TAMC). Springer, 2008.
[9] C. Dwork, M. Naor, O. Reingold, G. N. Rothblum, and S. Vadhan. On the complexity of differentially private data release: efficient algorithms and hardness results. In Proceedings of the forty-first annual ACM symposium on Theory of computing (STOC), 2009.
[10] C. Dwork, A. Roth, et al. The algorithmic foundations of differential privacy. Foundations and TrendyS in Theoretical Computer Science, 9(3-4), 2014.
[11] B. C. Fung, K. Wang, R. Chen, and P. Yu. Privacy-preserving data publishing: survey of recent developments. ACM Computing Surveys (Csur), 2010.
10

---

[12] F. Harder, K. Adamczewski, and M. Park. Dp-merf: Differentially private mean embeddings with randomfeatures for practical privacy-preseriigg data generation. International confer- ence on artificial intelligence and statistics (AISTAT). PMLR, 2021.
[13] M. Hardt and G. N. Rothblum. A multiplicatiive weights mechanism for privacy-preserving data analysis. In IEEE S1st Annual Symposium on Foundations of Computer Science (FOCS), 2010.
[14] M. Hardt, K. Ligett, and F. McSherry. A simple and practical algorithm for differentially private data release. Advances in neural information processing systems (NIPS), 2012.
[15] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034 2015.
[16] T. Karras, M. Aittala, J. Hellsten. S. Laine, J. Lehtinen, and T. Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems (NeurIPS), 33, 2020.
[17] Y. LeCun, L.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998.
[18] Z. Li and D. Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence (TPAMI), 40(12), 2017.
[19] Y. Liu, Y. Su, A.-A. Liu, B. Schiele, and Q Sun. Mnemonics training: Multi-class incremental learning without forgetting. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[20] Y. Long, B. Wang, Z. Yang, B. Kailkhura, A. Zhang, C. A. Gunter, and B. Li. G-PATE: Scalable differentially private data generator via private aggregation of teacher discriminators. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems (NeurIPS), 2021.
[21] I. Mironov, K. Talwar, and L. Zhang. Rényi differential privacy of the sampled gaussian mechanism. CoRR, abs/1908. 10530, 2019.
[22] B. Mirzasoleiman, J. Bilmes, and J. Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning (ICML). PMLR, 2020.
[23] N. Papernot, M. Abadi, U. Erlingsson, I. Goodfellow, and K. Talwar. Semi-supervised knowl- edge transfer for deep learning from private training data. In International Conference on Learning Representations (ICLR), 2017.
[24] N. Pupernot, S. Song, I. Mironov, A. Raghunathan, K. Talwar, and U. Erlingsson. Scalable private learning with pate. In International Conference on Learning Representations (ICLR), 2018.
[25] A. Prabhu, P. H. Torr, and P. K. Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European conference on computer vision (ECCV). Springer, 2020.
[26] A. Radford, L. Metz, and S. Chintala. Unsupervised representation learning with deep con- volutional generative adversarial networks. In Y. Bengio and Y. LeCun, editors, International Conference Learning Representations (ICLR), 2016.
[27] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[28] A. Roth and T. Roughgarden. Interactive privacy via the median mechanism. In Proceedings of the forty-second ACM symposium on Theory of computing (STOC), 2010.
[29] J. Schwarz, W. Czarnecki, J. Luketina, A. Crabska-Barwinskke Y. W. Teh, R. Pascanu, and R. Hadsell. Progress & compress: A scalable framework for continual learning. In International Conference Machine Learning (ICML). PMLR, 2018.
11

---

[30] M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y. Bengio, and G. J. Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv: arXiv:18/2.05/59 2018.
[31] R. Torkzadehmahani, P. Kairouz, and B. Paten. Dp-cgan: Differentially private synthetic data and label generation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2019.
[32] F. Tramer and D. Boneh. Differentially private learning needs better features (or much more data). arXiv preprint 1.kiv:2011.116660 2020.
[33] D. Ulyanov, A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.
[34] D. Ulyanov, A. Vedaldi, and Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018.
[35] G. Vietri, G. Tian, M. Bun, T. Steinke, and S. Wu. New oracle-efficient algorithms for private synthetic data release. In International Conference on Machine Learning (ICML). PMLR, 2020.
[36] B. Wang, F. Wu, Y. Long, L. Rimanic, C. Zhang, and B. Li. Datalens: Scalable privacy preserving training via gradient compression and aggregation. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communicutions Security (CCS), 2021.
[37] T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros. Dataset distillation. arXiv preprint 1811 10959, 2018.
[38] M. Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual Interna- tional Conference on Machine Learning (ICML), 2009.
[39] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-m machine learning algorithms, 2017.
novel image dataset for benchmarking
[40] L. Xie, K. Lin, S. Wang, F. Wang, and J. Zhou. Differentially private generative adversarial network. arXiv preprint 1rXiv:1802.067:: 2018
[41] Yoon, J, Jordon, and M. van der Schaar. PATE-GAN: Generating synthetic data with differential privacy guarantees. In International Conference on Learning Representations (ICLR), 2019.
[42] B. Zhao and Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference Machine Learning (ICML). PMLR, 2021.
[43] B. Zhao, K. R. Mopuri, and H. Bilen. Dataset condensation with gradient matching. arXiv preprint 20X::20066.0592,, 2020.
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 6
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes] In supplementary
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 5
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In supplementary
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [N/A]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 'rere using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "cAo_lWMiqm5", "reviewer": "Reviewer_qnNn", "summary": "This paper proposes to use recent advances for dataset condensation as an alternative for differentially private data generation. Specifically, they use DP-SGD to optimize a gradient matching objective that minimizes the distance between the gradients of real data on the target task and the gradients of synthetic data on the target task. The authors show this approach generates better data, in terms of downstream classification accuracy, than the classic approach which trains a discriminator and a generator. The authors then discuss potential limitations such as scalability to harder datasets and generality to other tasks.", "questions": "Please see the Weakness.", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1.This paper is very well written. Contributions and potential limitations are discussed adequately.\n\n2.This paper improves the SOTA data utility by a large margin of differentially private data generation. The improvement is in terms of the downstream classification accuracy.\n\n**Weakness:**\n\n1.The technique contribution is limited. This paper mainly uses two existing techniques, DP-SGD and gradient matching.\n\n2.This work only uses one algorithm from data condensation, i.e., gradient matching. It would be better if the authors can try more algorithms so the community can have a better understanding about data condensation for differentially private data generation. For example, distribution matching [1] that minimizes the distance between the averaged feature of real data and the averaged feature of synthetic data, which is also easy to implement with DP.\n\n3.In Section 6 you show the generator from a previous work can improve the visual quality of your algorithm. How does the visual quality of your algorithm compare with the visual quality of data directly generated from that generator? \n\n4.(Minor) Line 121, minimized -> minimize.\n\n[1]: DATASET CONDENSATION WITH DISTRIBUTION MATCHING, https://arxiv.org/pdf/2110.04181v1.pdf.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a0Mq6ECPHVt", "reviewer": "Reviewer_b3jX", "summary": "This paper proposes a new utility-optimized approach of privatizing high-dimensional data generation which optimizes the synthetic samples instead of the deep generative models. Experiments on MNIST and FashionMNIST are conducted to demonstrate the effectiveness.", "questions": "I only have two minor questions.\n\n1) In algorithm 1, what equation is used to determine the value of $\\sigma$ in order to preserve $(\\epsilon, \\delta)$-DP? I think this may need to be explicitly discussed.\n\n2) What are the numerical values in terms of the computational cost? I checked the supplementary but found this is missing. In particular, I am interested in the algorithm running time for different values of spc, say 10, 20, 6000 for MNIST.   ", "limitations": "The authors adequately discussed the low visual quality issue of the proposed approach. Other aspects such as scalability and generality are also mentioned.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality. The problem that the paper aims to address is well motivated. The proposed approach seems new to me. Relevant literatures are well discussed.\n\nQuality. The proposed algorithm overall makes sense to me and I didn't find any technical mistakes. However, some details of the algorithm and the experiments are missing (see my questions below).\n\nClarity. The paper is easy to follow and well organized.\n\nSignificance. The results overall look good, yet I am not sure whether they are enough. The authors only chose $\\epsilon = 1$ and 10, corresponding to high and low privacy budget. Not sure whether what happens when the budget is moderate, say $\\epsilon = 5$. Also, it would be better if the performance is benchmarked on colored image datasets (e.g., CIFAR-10).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-cYOj2t4NXr", "reviewer": "Reviewer_GR88", "summary": "This paper introduces a new approach to creating a synthetic data set as a stand-in for releasing some sensitive data. Instead of the standard technique of learning some kind of general purpose generative model that tries to capture the full data distribution, the proposed methods iteratively updates a set of data points (initialised randomly) directly to maximise the utility of the synthetic data for a given downstream classification task.\n\nMore precisely, the synthetic data is used to train the downstream classification network while in parallel the original data is used to train a network of the same architecture. In each step, the synthetic data is then updated to minimise the difference between the network parameter gradients between the two networks undergoing training. Privacy is ensured by perturbing the gradients of the network trained on the original data using the Gaussian mechanism before it is used in optimising the synthetic data.\n\nThe paper performs an extensive evaluation of the method and compares it to competing methods from recent literature on image classification tasks for MNIST and FashionMNIST using a convolutional network. The classifier trained on synthetic data generated using the proposed method shows superior classification accuracy compared to when trained on synthetic data generated using competing methods. Additionally, the proposed method allows to sample a much smaller data set than the original without a significant loss in downstream accuracy, which makes downstream training much easier.\n\nThe paper additionally explores using different architectures for the downstream task, which is found to have no significant effects on accuracy as long as the architecture is sufficiently similar (i.e., some kind of convolutional network).\n\nFinally, the paper discusses that the generated synthetic data may fall out of the original data manifold due to unconstrained optimisation, and therefore does not \"look like\" the original data. The discussion includes a modification of the proposed method that ensures that the synthetic data is on the manifold and compares it to the unconstrained method.", "questions": "- I cannot comprehend what the sentence in lines 141-143 is trying to say, can you elaborate?\n\nMinor notes/nitpicks:\n- Def 3.1 : \"There are two ways in which $\\mathcal{D}$ and $\\mathcal{D}'$ may differ in one training example: it could be entirely removed or replaced by a different one. This makes a (small) difference, so please specify your setting (or state explicitly that it does not matter in your case).\n- Thm 3.1 : \"Any function F\"; technically the function $F$ is constrained to those that do not access the original data again.", "limitations": "The proposed method has a number of limitations that could hamper its adoption in practice. The paper discusses and performs experiments exploring the major ones of these but not always to a full degree:\n\n- The method requires a downstream task to be known a-priori before generating the data. This may not always be the case, however it is probably likely for the data holder to find the \"most likely\" task their data will be used for (they have collected that data for a reason, after all). The authors give an example in Sec 5.2 of a use case for their method in practice.\n\n- Generalisability: The synthetic data produced using this method has no guarantee to be useful for anything but the downstream task it was optimised for. The authors recognise this and perform additional experiments in Sec 5.1 where they test different downstream classification architectures. They find that utility is similar for different architectures of convolutional networks (with the synthetic data optimised using a CNN as well) but drops off for MLP networks; no other architectures are explored. The authors do not evaluate different downstream tasks that could be performed on the synthetic data, so it is remains somewhat unclear how the synthetic data generalises. However, the authors point out in the end of Sec. 6. that while other methods may have in theory better capacity to generalise, these fail to show that that materialises as a benefit in any particular task in practise.\n\n- Out-of-distribution synthetic data: The synthetic data is optimised in an unconstrained domain and may fall out of the manifold that the original data lies on. The authors discuss and evaluate this and provide the interesting finding that there appears to be a trade-off between generating data that lies on the correct manifold versus data that maximises utility. There is no deeper inspection of why that is the case: It remains especially unclear whether this is merely an artifact of compressing the original data into a smaller synthetic set or an unavoidable trade-off. This also ties in with generalisability discussed above: A task other than the chosen downstream task may require data to be more similar to the original data (such as, e.g., image segmentation in this case).\n\nSummarising: The method does have limitations and the paper discusses the major ones. While some questions remain and follow up work on these would be welcome, they may be deemed to lie outside the permissible length of a conference paper and the discussion present should be sufficient for the reader to be aware of the trade-offs and whether the method is suitable in their setting.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: This paper presents an interesting novel perspective on generating synthetic data that deviates from established practise for generating synthetic data under DP guarantees. However, since I am not closely familiar with the privacy-agnostic(/non-private) data distillation technique, it could be that these contain similar methods except for the privacy perturbation, which could make this work a somewhat iterative. I hope some of my fellow reviewers may be more familiar with that and can form a more well-founded opinion on this. However, while the method would be iterative in that case, I believe that the evaluation for the privacy setting would still be a sufficiently original contribution.\n\nSignificance: The proposed methods demonstrates clearly superior results for the chosen downstream tasks, which could have a large impact for sensitive data release in practical applications when the downstream task the data will be used in is clear (or can be deduced with high likelihood). While I do not know how likely that setting is, I do think that it is not one impossible to find oneself in, so designing a well-working method for such a case is a valuable contribution.\n\nSoundness/Quality: The proposed methods applies standard privacy techniques in a straightforward way, so results on privacy from earlier work transfer more or less directly. The experiments appear to have no technical flaws, however the reported results do not convey any information on error margins.\n\nClarity: The paper is overall clear and easy to understand.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept