Differentially Private Kernel Inducing Points using features from ScatterNets (DP-KIP-ScatterNet) for Privacy Preserving Data Distillation
Authors: Margarita Vinaroz
Mijung Park
Venue: Accepted by TMLR ()

Abstract: Data distillation aims to generate a small data set that closely mimics the performance of a given learning algorithm on the original data set. The distilled dataset is hence useful to simplify the training process thanks to its small data size. However, distilled data samples are not necessarily privacy-preserving, even if they are generally humanly indiscernible. To address this limitation, we introduce differentially private kernel inducing points (DP-KIP) for privacy-preserving data distillation. Unlike our original intention to simply apply DP-SGD to the framework of KIP, we find that KIP using infinitely-wide convolutional neural tangent kernels (conv-NTKs) performs better compared to KIP using fully-connected NTKs. However, KIP with conv-NTKs, due to its convolutional and pooling operations, introduces an unbearable computational complexity, requiring hundreds of V100 GPUs in parallel to train, which is impractical and more importantly, such computational resources are inaccessible to many. To overcome this issue, we propose an alternative that does not require pre-training (to avoid a privacy loss) and can well capture complex information on images, as those features from conv-NKTs do, while the computational cost is manageable by a single V100 GPU. To this end, we propose DP-KIP-ScatterNet, which uses the wavelet features from Scattering networks (ScatterNet) instead of those from conv-NTKs, to perform DP-KIP at a reasonable computational cost. We implement DP-KIP-ScatterNet in -- computationally efficient -- JAX and test on several popular image datasets to show its efficacy and its superior performance compared to state-of-the art methods in image data distillation with differential privacy guarantees. Our code is available at  https://github.com/ParkLabML/DP-KIP.

Published in Transactions on Machine Learning Research (05/2024)
Differentially Private Kernel Inducing Points using features from ScatterNets (DP-KI--ScatterNet) for Privacy Preserv- ing Data Distillation
Margarita Vinaroz
University of Tübingen
International Mar Planck Research School for Intelligent Systems (IMPRS-IS)
Mi Jung Park
mijung@@cs.uk ubc.ca, mjupa @dtu dk
University of British Columbia & Alberta Machine Intelligence Institute (AMII) Technical Universily of Denmark
Reviewed on OpenReview: https: openreview. net/ forum? id= id=
84M8mwNarc
Abstract
Data distillation aims to generate small data set that closely mimics the performance of given learning algorithm on the original data set. The distilled dataset is hence useful to simplify the training process thanks to its small data size. However, distilled data samples are not necessarily privacy-preserinng, even they are generally hurnanly indiscernible. To address this limitation, we introduce differentially private kernel inducing points (DP- KIP) for privacy-preserving data distillation. Unlike our original intention to simply apply DP-SGD to the framework of KIP, find that KIP using infinitely-wide convolutional neural tangent kernels (conv-NTKs) performs better compared to KTP using fully-connected NTKs. However, KIP with conv-NTKs, due to its convolutional and pooling operations, introduces an unbearable computational complexity, requiring hundreds of V100 GPUs in parallel to train, which impractical and more importantly, such computational resources are inaccessible to many. To overcome this issue, we propose an alternative that does not require pre-training (lo avoid a privacy loss) and can well capture complex information on images, as those features from conv-NKTs do, while the computational cost is manageable by single V100 GPU. To this end, propose from Scattering networks (ScatterNet) instead of those from conv-NTKs, to perform DP-KIP at a reasonable computational cost. We implement DP-KIP-ScallerNel in computationally efficient JAX and test on several popular image datasets to show its efficacy and its superior performance compared to state-of-the art methods in image data distillation with differential privacy guarantees. Our code is available at
which uses the wavelet features
1 Introduction
First introducod by Wang et al. (2018), data distillation (DD) aims at extracting the knowlodge the entire training dataset to few synthetic, distilled datapoints. What DD offers is that the models trained the small number of distilled datapoints achieve pighhmpeformance relative the models trained on the original, large training dataset. DD is a sensible choice for fast, cheaper, and light training of neural network models.
Various applications of DD include continual learning (Liu et al. 2020 Rosasco et al. 2021 Sangermano
et al. 2022 Wiewel & Yang 2021 Masarczyk & Tautkute 2020). neural architecture search (Zhao & Bilen
2021
Zhao et al. 2021 Zhao & Bilen 2023), and more.
Depending on the similarity metrics chosen for judging how close the small distilled datasets are the original large datasets, there are different ways to formulate the DD problem. For instance, Zhao et al. 2021

---

Published in Transactions on Machine Learning Research (05/2024)
formulate it as gradient matching problem between the gradients of deep neural network weights trained on the original and distilled data. Nguyen et al. (2021a b) formulate it as a kernel ridge regression problem where the distilled data correspond to the kernel inducing points (KIP). Regardless of what formulation one takes, the techniques for DD are fast improving, and their application domains are widening.
Among the many application domains, Nguyen et al. (021a) claim that DD also useful for privacy-presrringg dataset creation, by showing distilled images with 90% of their pixels corrupted while test accuracy with those exhibits limited degradation. It is true that the distilled images with 90% of corrupted pixels are not humanly discernible. However, their illustration is merely experimental and does not involve any formal definition of privacy. Recently, Dong et al.] (2022) attempt to connect DD with differential privacy (Dwork & Roth 2014), one of the popular privacy notions, based on DD's empirical robustness against some known attacks. Unfortunately, the empirical evaluation of the method and its theoretical analysis contain major flaws, as described in Carlini et al. (202)).
For a provable privacy guarantee, Chen el al. (022) apply DP-SGD, the off-the-shelf differential privacy algorithm (Abadi et al] 2016), to optimizing gradient matching objective to estimate differentially private distilled dataset. More recently, Zheng & Li 2023) proposes a differentially private distribution matching framework, which further improves the performance of Chen et al. (2022).
In this paper, we apply DP-SGD to the KTP framework developed by Nguyen et al. (2021a for differentially private data distillation. There are two important reasons we choose to privatize KIP over other existing DD algorithms. First, in DP-KIP, the gradients that DP-SGD privatize are the distilled datapoints. Typically, we consider only few distilled datapoints and therefore the dimension of the parameters to guard is relatively low. Consequently, the privacy-accuracy trade-off of DP-KIP is better than that of the gradient matching framework with DP-SGD (Chen et al.] 2022), as the latter needs to privalize siguificantly higher dimensional neural network parameters than the former. Second, optimizing for distilled data under KIP is computationally cheaper and converges faster than that under gradient matching. KIP implements kernel ridge regression (KRR) which involves convex optimization with a closed-form solution and estimating distilled data requires first-order optimization methods, which is straightforward to apply DP-SGD to and converges relatively fast. However, the gradient matching framework requires computationally expensive bi-level optimization (inner loop updates distilled data and outer loop updates the neural network parameters), which requires longer training time than KIP. In nou-DP setlings, longer training time and higher parameter dimension typically do not matter. However, in DP settings, these incur higher privacy costs, which in turn results in a poorer privacy-utility trade-off. Indeed, for these reasons, we observe that empirically our method outperforms other DP-DD methods at the same privacy level.
Settling with the kernel-based KTP framework, now the question is which features (kernel) to use to produce high quality distilled datasets with privacy guarantees. Nguyen et al. (2021a) use the features of infinitely-wide neural tangent kernel (NTK) from fully connected neural networks. Nguyen et al. (021b) significantly improves previous results in Nguyen et al. (2021a) by changing the infinite-width NTK from fully connected network to a convolutional network. However, kernels formed out of convolutional and pooling layers introduce a significant computations burden, due to the necessity of keeping track of pixel-pixel correlations. To mitigate this problem, Nguyen et al. (2021b) use hundreds of V100 GPUs working in parallel to make the optimization computationally fcasible. As a result, using the infinitely-wit widd ConvNet NTK (ConvNet-NTK) becomes out of reach for researchers who do not have hundred V100 GPUs available at hand. So, we question which features (kernels) to use to produce the results better or comparable to the distilled data trained with the features of infinitely-wide ConvNet NTKs, while keeping the computational cost manageable to single V100 GPU?
The first alternative we consider is an empirical NTK (e-NTK), a finite-dimensional approximation to the infinite-width NTK. e-NTK is known to provide useful approximations of the training dynamics and generalization properties of given deep neural network (Schoenholz et al.) 2017 Xiao et al.] 2018 2020). It has been observed that the width of the neural network increases, the e-NTK initialization approaches the infinite-width NTK. Therefore, we empirically test whether the performance of convolutional cTKSs is comparable to that obtained by the convolutional infinitely-width NTK.
2

---

Published in Transactions on Machine Learning Research (05/2024)
The second alternative consider is kernel using the features from Scattering Network, often called ScatterNet [Oyallon & Mallat 2015). ScatterNet is a feature extractor formed by the combination of predefined complex wavelet filters and non-linear operations followed by final averaging operator. These handcrafted features exhibit stability to deformations and invariance to translations, making them a suitable choice for image classification tasks such as DD, where the goal to learn distilled images that minimize the 12-noom between true and predicted labels. Notably, Tramer & Bonch (2021) previously studied the efficacy of these features in image classification tasks with differential privacy guarantees. We attempt to use ScatterNet features for data distillation with differential privacy guarantees. We use an untrained ScatterNet (i.e,, at its initialization) to avoid privacy costs for pre-training the network.
The last alternative we consider is a kernel using perceplual features (PFs). Pfs are defined as the concatenation of a (pre-trained) deep neural network activations for a given input. They have been widely used in different tasks such as transfer learning (Yosinski et al. 2014a) 4a; Huh et al. 2016). super-resolution (Johnson al.j 2016) or image generation (Santos et al. 2019: Harder et al.) (023). Although DD task is different from those mentioned beforehand, PFs have been shown extract useful information from input images and therefore we investigate if the use of these features can contribute to improving the performance of KIP. We use untrained ResNct-18 (i.c., at its initialization) to avoid privacy costs for pre-training the network.
We show that KTP with ScatterNet features outperforms e-NTK features and perceptual features, achieving similar performance to the infhhit--width convolutional NTK, in non-DP settings. Therefore, we incorporate ScatterNet features in DP-KIP. To separate ours from the original KIP, we call our algorithm by DP-KIP- ScotterNet. Our experiments show that DP-KIP- Se-ateeScatterNet significantly outperforms DP- -rggiiieent matching by Chen al. ( and DP-distribution matching by Zheng & Li 2023). when tested on several benchmark image classification datasets.
Before moving on to describe our method, we summarize our contributions
We propose a DP data distillation framework based on KTP, which uses DP-SGD for privacy guarantees in the resulting distilled data. This itself is a mere application of DP-SGD an existing data distillation method. However, motivated by the unbearable computational costs in using the tnitt convolutional NTKs, we look for alternative features and empirically observe that the features from SeatterNets are the most useful in distilling image datasets, evaluated on classification lasks. As a result, DP-KIP-ScatterNet significantly outperforms state-of-the-arl DP data distillation methods.
We further improve the performance of Me-KKI-SScateree by considering varying levels of pixel corruption rates. By corrupting some portion pixels with some noise, we reduce the dimension of distilled data, which results in better privacy-accuras trade-offs.
Unlike other most existing data distillation papers that focus on distilling image data only, we also test DP-KIP, where we the original features in KIP, i.c., the infinitely-wide fully-connected NTKs, on distilling labular datasets. DP-KIP with relatively small number of distilled samples (only 10 samples/clas /class) significantly outperforms previons DP data generation models at the same privacy level.
2
Background
In the following section, we review the Kernel Inducing Points (KIP) algorithm, different kernel functions we propose to use in KTP (infinite-width NTK, e-NTK, ScatterNet features, PFs) and differential privacy (DP).
2.1 KIP
We start giving some key definitions and describing KIP algorithm from Nguyen et al. 202
1 For this task, we the original hiffiite-width fully connected NTKs since tabular data would not benefit from those features engineered for image data such PFs and ScatterNet. features.
3

---

Published in Transactions on Machine Learning Research (05/2024)
Definition 2.1. Fin loss function I and let f.f RP - RC be two functions. Let & 0. Given distribulion P on TR. x R soy I and f are weakly E-close with respect to (l,P) if:
(1)
Definition 2.2. Let D and D be two labeled datosets in RD with label space RC and A, A be two fized learning algorithmss Let Ç 2 O. Given the resulting model Ad obtained after training A on D, the resulting model Ag oblained after training A on D, we say D is E-appracimation of 'D with respect to (A, A,I, L, if A and g
Ap are weakly É-close with respect to (1, P), where is loss function and P is distribution on RD x TRC.
In data distillation, the goal is to find small dataset D, that is f-approximation to a large, original dataset Dr drawn from a distribution P with respect to a learning algorithm A and a loss function I:
((r.)) (z) ((A,)) (x), S.
(2)
In KIP, the loss classification accuracy in terms of the L2-distance between true labels and predicted labels; and the learning algorithm A is kernel ridge regression (KRR).
Consider target dataset D. {(xx, Vr, Di= with imput features x2, E ER and scalar labels yt, Given kernel & the KIP algorithm constructs a small distilled dataset D, = (X,,, Un Vny)) 1==1, where Xs, E ER,, and scalar labels ys, and importantly m < n, such that its performance on a classification task approximates the performance of the target dataset for the same task. Note that Nguyen et al. (2021a) call D, "support' dataset. (hence the subscript "s"). In this paper, we will use "support" and "distilled' datasets, interchangeably.
The KIP algorithm, we start by randomly initializing the support dataset and then iteratively refine D, by minimizing the Kernel Ridge Regression (KRR) loss:
((D,)
Σ (ye, kt, T(Kss + "y)²
(3)
with respect to the support dataset D, Here À >> is a regularization parameter, Kes kernel matrix, where the (i, j)-th entry is k(x,,, Xx,), kt,s is column vector where the j-th entry is k(x,,; and y. is a column vector where the j-th entry is Mey. During the training phase, the support dataset is updated using a gradient-based optimization method, e.g., using Stochastic Gradient Descent (SGD), until some convergence criterion is satisfied.
2.2 Infihite-width NTK
Initially, NTKs were proposed to help understand neural networks' training dynamics in a function space. partieular, in the infinitewwddth limit, the parameters of neural network do not change from the random initialization over the course of the training and the gradients of the network parameters converge to
dimense-imeeniional feature map of the NTK Jacot et al. (2018): Lee et al. (2019); Arora et al. (2019): |: Lee
et al. 2020). Characterizing this neural tangent kernel is essential to analyzing the convergence of training and generalization of the neural network.
Based on this finding, Nguyen et al. (2021a) motivate the use of NTK in KIP in the following sense: (a) the kernel ridge regression with an NIK approximates the training of the corresponding infinitely-wide neural network; and (b) it is likely that the use of NTK yields approximating D; by D, (in the sense of f-approxinations given in eq. 2) for learning algorithms given by a broad class of neural net While there is no mathematical proof on point (b), Nguyen et al. (002aa) empirically backed up point (b).
2.3 Empirical NTK (c-NTK)
The empirical NTK (c-NTK) is a finite-dimensional approximation of the infinitewwdth NTK that provides a way to approximate the dynamics of deep neural networks during training. The c-NTK for a given (finite wide) neural network, f, with random initial parameters 0 is defined
4

---

Published in Transactions on Machine Learning Research (05/2024)
as(0, x) 00
a x') 00
e-NTK((x,))
(4)
where 2f(00-)/00 denotes the neural network Jacobian. obtained by using a Lenet (Lecun et al. 1998) and ResNet18 (He al. 2016) architectures for grayscale and RGB images respectively.
In our experiments, we considered the e-NTK
2.4 ScatterNet features
We consider the kernel defined by the inner product of the Scattering Network (ScalterNet) features presented by Oyallon & Mallat (2015). ScatterNet is Scale Invariant Feature Transform (SIFT) feature extractor based cascades of wavelet transform convolutions followed by non-linear modulus and final averaging operator. For given imput image x, its ScatterNet feature representation is defined the output of the Scattering Network of depth J:
6s(x) == W1al|
where W2 and W1 are wavelet transforms that capture local and global information of the input image, |.1 is the modulus operator which computes the magnitude of each wavelet coefficient and guarantees tramslation-invariian representations, and A is the 2 patch-averaging operator that contributes in capturing higher-level patterns and relationships between different parts of the image. The ScatterNet can be seen variant of a Convolutional Neural Network (CNN) where the architecture paramcters and filters are not learned during the training phase, but are predefined fixed wavelets transforms.
Following Oyallon & Mallat 2005)), we use a ScatterNet of depth = with wavelets rotated along eight angles. The extracted ScatterNet features for a given image of size H x W has dimension (K, 2 W. where K is set to 81 and 243 for grayscale and RBG images respectively.
2.5 Perceptual features (PFs)
Here we propose the kernel defined by the inner product of the features extracted from deep convolutional networks (DCNNs), also known as perceptual features (PFs). These features are defined as the concatenation of each layer's output. from fixed deep convolutional neural network.
In this work we considered the features extracted form randomly initialized ResNet18 He et al. 201 )).
2.6 Differential privacy (DP)
Differential privacy is gold standard privacy notion in machine learning and statistics. Its popularity is due to the mathematical probability. The definition of DP (Definition 2.4 in Dwork & Roth 2014 is given below.
A randomized mechanism M is -diffeernttt private if for all neighboring datasets D. D' differing in an only single entry and all sets 5 C range(M), the following inequality holds:
Definition 2.3.
PH(M(D) E S) S PHM((D))
The definition states that for all datasets differing in an only single entry, the amount of information revealed by a randomized algorithm about any individual's participation is bounded by and 8 (which is preferably smaller than 1/(D)). In our work, we use the inclusion/exei exclusion definition for neighbouring datasets and privacy per item or per (training) example as protection granularity.
A common paradigm for constructing differentially private algorithms to add calibrated noise an algorithm's 's output. In our algorithm, we use the Gaussian mechanism to ensure that the distilled dataset satisfies the DP guarantee. For deterministic function h D - Rdd, the Gaussian mechanism is defined by
5

---

Published in Transactions on Machine Learning Research (05/2024)
h(D) = h(D) + N00 the function h, An, and it is defined as the maximum difference in terms of lz-norm, [k(D) - ((D))
Here, the noise scale depends on the global sensitivity Dwork et al.
2006a of
Il. for D
and D' differing in an only single entry and a is a function of the privacy level parameters, e, 6.
Differential privacy has two fundamental properties that are useful for applications like ours: composability (woork et al. 20066a) and post-pres pret-proesssing invariance (Dwork et al., 2006b). Composability ensures that if all components of a mechanism are differentially private, then its composition is also differentially private with some privacy guarantee degradation due to the repeated use of the training data. In algorithm, we use the subsampled RDP composition by (Wang et al, 2019), they yield to tight bounds for the RDP parameter, CRDP, for a subsampled mechanism. To switch from the definition of RDP to (c, 6)-D,, we follow Wann et al. 2019), which keeps track of the parameters of RDP and converts them to the definition of (e, 6.P through the use of numerical methods. Furthermore, the post post-processing invariance property states that any application of arbitrary data-independent transformations to an (c, 5)-DP algorithm also (c. 6)DP. In our context, this means that no information other than the allowed by the privacy level c, 6, can be inferrod about the training data from the privatized mechanism.
3 Our algorithm: DP-KIP and DP-KIP-ScatterNNtt
In this section, we introduce our proposed algorithm differentially private kernel inducing points (DP-KIP). The algorithm produces differentially private distilled samples by clipping and adding calibrated noise to the distilled data's gradients during training. DP-KIP-SeafterNe is a particular algorithm of DP-KIP which uses the features from Scaltering Networks.
3.1 Outline of DP-KIP
Our algorithm is shown in Algorithm 1 We first initialize the distilled (support) dataset D., where the learnable parameters X, = (x.))²= are drawn from a standard Gaussian disttribution, (i.e. Xx; ~ NN Tp) for Xa, E P). We generate labels y. by drawing them from a uniform distribution over the number of classes; and fix them during the training for X,. Note that the original KIP algorithm has an option for optimizing the labels through Label Solve given optimized distilled images. However, we choose not to optimize for the labels to reduce the privacy loss incurring during training.
At each iteration of the algorithm, randomly subsample B samples from the target dataset Dr, Dig- Given kernel k we compute the loss given in eq. 3 Then, we compute the per target sample gradients with respect to the support dataset, given in
g ((.)) = VD.
= V.. (yu, - kn + +A))'yy))
(6)
As DP-SGD, we ensure each datapoint's gradient norm bounded by explicitly normalizing the gradients if its [2-norm exceeds C. In the last steps of the algorithm, the clipped gradients are perturbed by the Gaussian mechanism and averaged in DP-SGD algorithm and finally, the support dataset is updated by using some gradient-based method (e.g. SGD, Adam).
Prop. states that the proposed algorithm is differentially private.
Proposition 1. The DP-KIP algorithm produces a ODP distilled dataset.
Proof. Duc to the Gaussian mechanism, the noisy-clipped gradients per sample are DP. By the post-processing invariance property of DP, the average of the noisy-clipped gradients is also DP. Finally, updating the support dataset with the aggregated -oisy gradients and composing through iterations with the subsampled RDP composition WWagg et al., 2019) produccs (c, 8)-DP distilled datasct. The exact relationship between (c, 8), T (number of iterations DP-KIP runs), (mini-batch size), N (number of datapoints in the target datasct), and (the privacy parameter) follows the analysis of Wang et al. 2019).

---

Published in Transactions on Machine Learning Research (05/2024)
Algorithm 1 DP-KIP
Input: Dataset Di {(x,, yt. M1) number of distilled samples generate m, number of iterations P, mini-batch size B, clipping norm C, privacy level (c,
Step 1. Initialize distilled dataset DD {(xs;> [b,)]== with Xs; ~ (O, ID)
Step 2. Given desired level of (e, 8),P), compute the privacy parameter using the auto-dp package
by Wang et al. (2019).
for to do
Step 3. Randomly subsample Dts = ((x)s)
Step 4. Compute KRR loss given in eq. 3
Step 5. Compute per-sample gradients
eq. for each E tg.
Step 6. Clip the gradients via (xx) = ((x)// max(1, (l((x))ll/O)
Step 7. Add noise: 9 = Eí= ((x)) + -N(00,
Step 8. Update distilled dataset D. with SGD.
end for
Return: Learned private support dataset D,
3.2 A few thoughts the algorithm
The first step in DP-KIP initializes each support datapoint in D, be drawn from the standard Gaussian distribution, .N(0,, Ip). This random initialization ensures that no sensitive information is inherit by the algorithm at the beginning of training. Nevertheless, one can choose different type of initialization such as randomly selecting images from the training set as in (Nguyen et al. 2021a by and then, privatize those to ensure that no privacy violation incurs during the training process. The downside of this approach is that the additional private step in initialization itself is challenging, since computing the sensitivity for neighboring datasets has no trivial bound on the target dataset and incurs in an extra privacy cost.
Support dataset initialization:
Clipping effect of the gradients: In our algorithm, we follow the approach from (Abadi al. 2016) and clip the gradients to have 1o-norm C. This clipping norm is treated as hyperparameter since gradient values domain is unbounded a priori. Sctting C' to relatively small value is beneficial during training as the total noise variance is scaled down by the C factor. However, the small value may result in a large amount of the gradients being clipped and thus, drastically discard useful or important information. In contrast, setting C to a large value helps preserving more information encoded in the gradients but it yields to a higher noise variance being added to the gradients, which may worsen the learned results. Consequently, finding suitable C is crucial to maintain good privacy-accuracy trade-off on DP-KIP algorithm. See Sec. A in Appendix for experimental results of the clipping effect.
Regularization parameter: Following Nguyen et al. (2021a) we set the regularization parameter À in eq- to xrtK(..)) where m is the number of datapoints in the distilled dataset X,, In this fashion, the regularization parameter is scalc-invariant to the kernel function and ensures a similar performance of the algorithm for a wide range of À values.
Why not privatize the feature maps? Since in this work we are considering kernel functions & with finite dimensional feature maps é, such that x(x,x') (o(x), o(x')) and the only data dependent term in KIP is krs in eq. 3] one could consider privatizing ((x.)) ), Vi E [r] once at the beginning and reuse them during the training of the algorithm. However, perturbing the feature maps results in a constant signal-to-noise ratio, inversely proportional to the paramcter noise a, regardless of the number of datapoints in D,. This observation implics that when the feature maps are privatized, they essentially become dominated by noise. See Sec. in Appendix for a detailed explanation.
4 Related Work
The most closely related work Chen al. (2022), which applies DP-SGD on the gradient matching objective. As concurrent work, Zheng & L.i (2023) proposes differentially private distribution matching framework.
7

---

Published in Transactions on Machine Learning Research (05/2024)
MNNIT
Real Data
0
/ 2
3
4
5
7
8
9
4 $
6
CIPAR-100
FC-NTK
Road Data
O-NT
ScaltarNet
FC-NTK
PFs
FASHION
CIFAR-10
O-TT
Real Data
FC-NTK
SeatterNes
cNTK
ScatterNet
PFs
Figure 1: Generated image samples from KIP comparison models.
Our work differs from these two in the following sense. First, we use kernel functions as features in comparing the distilled and original data distributions. Second, we formulate our problem as kernel inducing points, our DP-SGD's privacy-accuracy trade-off is better than that of gradient matching due to privatizing a smaller dimensional quantity in our case.
Another line of relevant work is differentially private data generation. While there are numerous papers to cite, we focus few that we compare method against in Sec. 5 The majority of existing work uses DP-SGD to privatize generator in the Generative Adversarial Networks (GANs) framework. In GANs, the generator never has direct access to training data and thus requires no privatization, as long as the discriminator is differentially private. Examples of this approach include DP-CGAN (Torkzadchmahani et al.
2019); DP-GAN Xie al., 2018), G-PATE (Long al. 2021). DataTens (Wang el al. 2021 , and GS WGAN Chen et al. (2020). A couple of recent work outside the GANs framework include DP-MERF Harder al. 20221), DP-HI HP VVnaroz et al. 2022), DP-NTK (Yang et al. 2023) and DP-MEPF (Harder et al. 2023) that use the approximate versions of maximun mean discrepancy (MMD) as loss function: and DP-Sinkhorn (Cao et al. 2021 that proposes using the Sinkhorn divergence in privacy settings. These data generation methods aim for more general-purpose machine learning. On the contrary, our method aims to create (classification). Hence, when comparing them in terms of classification performance, for these general data generation methods. Nevertheless, at the same privacy level, we still want show where our method stands relative these general methods in Sec.
small dataset that matches the performance of a particular task in mind
not necessarily fair
In the line of work of dataset reconstruction attacks, Loo et al. 2023 propose reconstruction attack for trained neural networks that is generalization of KIP. In addition to the task of reconstructing original training samples, they show that their allack can be used for dataset distillation when the number of samples to reconstruct is set to be significantly smaller than that the original dataset. While the distilled samples from the reconstruction attack do not resemble any original image from the training set, there is no formal privacy guarantee about the generated images.
5 Experiments
Here, we show the performance of KIP and DP-KIP over different real world datascts. In Sec. 5.1 we follow previous data distillation work and focus our study on grayscale and color image datasets. In addition, we also test DP-KIP performance on imbalanced tabular datasets with numerical and categorical features in Sec. 5.2 All experiments were implemented using JAX (Bradbury et al. 2018), except KIP c-NTK experiments were we used autograd. grad function implemented in PyTorch (Paszke al. 2019). All the experiments were single NVIDIA V100 GPU.

---

Published in Transactions on Machine Learning Research (05/2024)
5.1
Image data
We start by testing KTP and DP-KIP performance on MNIST (LeCun al., 2010), MashionMNIST (Xiao
et al. 2017), SVHN (Netzer et al. 2011p, CIFAR-10 Krizhevsky & Hinton 2009 and CIFAR-100 datasets for image classification.
Each of MNIST and FashionMNIST datasets consists of 60,000 samples of 28 x 28 grey scale images depicting handwritten digits and items of clothing, respectivcly, sorted into 10 classes. The SVHN datasct contains 600,000 samples of 32 x 32 colour images of house numbers taken from Google Street. View images, sorted into 10 classes. The CIFAR-10 dataset contains a 50,000- of 32 x 32 colour images of 10 classes of objects, such vehicles (cars and ships) and animals (horses and birds). CIFAR100 contains 100 classes of objects.
In Table 1 we consider KTP with infinitewidtth NTK for fully connected (KTP FC-NTK) and convolutional (KIP ConvNet-NTK) networks as a non- non-private baseline. We show as evaluation metric the averaged test accuracy using KRR as a classifier for 1, 10 and 50 distilled images per class over 5 independent runs. The general trend for different KIP variants is that creating more images per class improves the classification accuracy. The proposed e-NTK, ScatterNet and PFs features increase significantly the accuracy with respect to KIP FC-NTK for all the considered image datasets and more interestingly, KIP ScatterNet outperforms or performs similar to KIP ConvNet-NTK for most datasets. Fig. [ shows learned distilled images obtained by the different considered kernels. Both, infinite and empirical NTKs distilled images look blurrier and are visually more indistinguishable between the different classes and datascts. On the contrary, ScatterNet and PFs distilled images are visually more interpretable.
Our intuition for the interpretability of the images distilled from PFs relies on the information carried by the DCNN layer's outputs detailed in Yosinski et al. (2014b). DCNNs layer's outputs transfer from general information, such as color blobs in images (early layers), specific information related to a particular class (last layers), and thus the distilled images PFs representations tend to encode all this information through KIP training. In ScatterNet features case, the intuition for the visual interpretability of the distilled samples is that the feature representations conserve the energy of the original signal (or information within the image) as described in Oyallon & Mallat (2015). Therefore, the learned distilled images tend to learn more detailed patterns from the original images.
In the privacy realm, we evaluated DP-KIP performance in Table Jusing fully connected infinite-width NTK (DP-KIP FC-NTK) as private baseline and compared it to ScatterNet features (DP-KIP-SeatterNel) We consider E (1) and 8 = 10-5 As in the non-private setting, the general trend is that creating more images per class improves the classification accuracy, while the gradient dimension in DP-SGD increases. Furthermore, the results show that --------tttteeeettttt significantly improves the performance compared to DP-KIP using FC-NTK. In Fig. - we show the learned distilled images at <= and c = 10. It is surprising that the images created at e = 10 are not humanly discernible, while the classifiers can still achieve a good classification performance. Detailed hyperparameter settings can be found in Sec. D.1 Appendix.
In Table 3 we explore the performance of DP-KIP compared to other methods for private data generation (DP-CGAN, G-PATE, DataLens, GS- DP-MERE DP- -p-Sinkhorn) and private gradient matching by 20222) We report the test accuracy of ConvNet downstream classifier consisting of 3 blocks, where each block contains one Conv layer with 128 filters, Instance Normalization, ReLU activation and AvgPooling modules and a FC layer as the final output. The classifier is trained using private data and then, it is tested on real test data over 3 independent runs for each method. Here, the methods based on DP-KIP algorithm outperform existing mothods at the same privacy level, achieving the best accuracy by using ScatterNet features
Chen et al.
Table 4 shows the KRR test accuracy for DP-KIP-ScatterNee with pixel corruption on CIFAR-10 for generating 1, 10 and 50 distilled images per class at different c values. Here, we randomly fix some pixels from the distilled images to a noisy value and perform DP-KIP-ScatteNNel on the rest of the pixels. As a general trend, we observe that KRR accuracy improves over all- ail-pixel optimization (Table 2 at small proportions of corrupted pixels (1%, 5%). We also observe that inercasing the number of distilled samples (10 and 50 images/class) enables an increase the proportion of corrupted pixels up to 10% compared to all-pixel optimization KRR accuracy.
9

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: KRR test accuracy on Image datasets for KIP. The average over five independent runs. KIP e-NTK, KIP ScatterNet and KIP PFs outperform KIP FC-NTK. KIP ScatterNet outperforms performs similar to KIP ConwNet-NTK. Best accuracy is marked in bold and overlapping overlapping best accuracies are highlighted in italics.
Imgs/ Class
KIP FC-NTK
KIP ConvNet-NTK
KTP eNTK
KIP SeatterNet
KTP PFs
Bag
ang
aug
aag
89.3
97.3 +
96.5
93.6 +
98.2
94.6 0.6
MNIST
96.6
99.1 + 0.1
99.1 + 0.1
96.7 +
99.0 0.1
97.7 =
97.6
99.4 = 0.1
99.5 + 0.1
97.6 =
99.4 + 0.1
97.9 =
80,3 = 0.4
82.9 = 0.2
76.7 = 0.2
75.2 = 0.3
87.2 = 0.2
81.6 = 0.2
FASHION
10
84.8 + 0.4
91.0 = 0.1
88.8 = 0.1
76.2 = 0.2
89.5 = 0.1
88.1 = 0.4
50
86.1 1 0.1
92.4 0.1
91.0 0.1
79.5 1 0.1
90.6 0.1
89.3 1 0.3
25.4 0.3
62.4 0.2
64.3 = 0.4
19.9 - 0.3
77.4 + 0.2
62.1 + 0.2
SVHN
59.7 0.5
79.3
81.1 0.5
21.1 0.4
84.4
81.2 - 0.2
69.7 0.1
$2.0
84.3
27.5
86.4
$2.6 0.2
39.3 1.6
64.7 0.2
63.4
32.0 = 0.7
60.2
44.7
CIFAR-10
49.1 1.1
75.6 0.2
75.5 = 0.1
40.0 = 0.3
66.2 = 0.2
53.3 0.5
52.1 1: 0.8
78.2 1 0.2
80.6 0.1
42.4 1: 0.2
68.5 0.1
54.4 1: 0.3
14.5 + 0.4
34.9 = 0.1
33.3 = 0.3
9.4 = 0.2
27.4 = 0.2
19.6 = 0.3
CIPAR-100
10
12.2
1 0.2
47.9 1 0.2
49.5 1 0.3
12.5 1 0.2
35.8 0.1
23.5 1 0.3
50
12.3
4 0.2
13.7 1 0.2
45.7 0.1
24.9 + 0.4
Table 2: KRR test accuracy Image datasets for DP-KIP FC-NTK and DP-KIP-ScatterNe at 1,
and 8 = 10-6, 5 The average over five independent runs. Me--KI--Scattrrii outperforms DP-KIP FC-NTK for all image datasets.
Imgs/ Class
DP-KIP FC-NTK
DP-KIP DP-KIP-Scatterts ScatterNet
6=1
f= 10
==1
10
82.7
85.2 1 0.1
94.7 1 0.3
96.1
MNIST
87.5 1.5+00 0.4
89.3
97.4
-2.7:02
93.4 0.1
97.2 = 0.3
98.1
76.9 0.1
78.3 0.1
77.5 + 0.1
82.1 0.2
FASHION
10
77.7 + 0.1
78.7 0.4
83.3 = 0.2
0.2
86.2 + 0.1
50
78.8
+ 0.1
81.1 = 0.1
84.7 = 0.3
87.6 + 0.1
24.9
0.3
25.2 = 0.2
55.3 = 0.6
68.6 0.2
SVHN
10
40.5 + 1.2
47.2 = 0.6
66.4 = 0.3
74.4 0.2
52.7 0.4
56.6 1
71.4 0.4
76.7
36.7
37.3
0.5
50.4
CIFAR-10
38.3 0.3
39.7 =
48.6 = 0.4
54.4 1
40.8 0.2
43.7 =
49.7 = 0.5
58.7
9.9 + 0.6
11.1 0.1
12.7 = 0.2
17.5 - 0.2
CIFAR-100
10
10.1 + 0.3
12.1 = 0.4
14.1 = 0.1
21.1 + 0.1
50
11.3 1 0.3
13.6 1 0.2
16.2 1 0.1
25.2 1 0.1
5.2
Tabular data
In the following we present DP-KIP results applied to eight different tabular datasets for imbalanced data. These datasets contain both numerical and categorical imput features and are described in detail Sec. C in Appendix. To evaluate the utility of the distilled samples, we train 12 commonly used classifiers on the distilled data samples and then evaluate their performance on real data for 5 independent runs. In tabular tested our method only with infinile-width fully-connected NTKs. The motivation behind this is that by construction, ScatterNet features designed to detect and describe local image features and thus, may not be a suitable option for tabular datasets.
data experiments,
For datasets with binary labels, we use the area under the receiver characteristics curve (ROC) and the area under the precision recall curve (PRC) as evaluation metrics, and for multi-class datasets, we use Fl score. Table shows the average over the classifiers (averaged again over the 5 independent runs) trained on the synthetic privated generated samples for DP-CGAN Torkzadehmmhani al. (2019), DP-GAN (Xie al.
2018), DP-MERF (Harder et al. 2021), DP-HP (Vnaroz et al. 2022) and DP-NTK YYang al. 2023) and
10

---

Published in Transactions on Machine Learning Research (05/2024)
Table 3: Left: Test accuracy of ConvNet downstream classifier trained on synthetic/distilled data with 10-3. Right: Test accuracy ConvNet downstrcam classifier trained with fixed c and 10
and varying the number of distilled samples per class. In concurrent work Zheng & Li (2023): the test.
accuracy the same classifier tested 50 MNIST distilled samples created at 6.12 and à 10 is 97.35%; that on 50 NaSSI distilled samples created at 5.45 and 8 = 10-6 is 82. 72%.
MNIST
FASHION
MNIST
FASHION
E=
1
10
E 1
10
Imgs//lasss CClass
20
full
full
DP-CGAN
35.1
52.5
26.3
50.2
Real
93.6
74.4
77.4
G-PATE
58.8
80.9
58.1
69.3
DP-SGD
96.5
82.9
DataLens
71.2
80.7
64.8
70.6
DP-CGAN
57.4
57.1
52.5
51.4
53.0
50.2
GS WGAN
48.3
39.2
G-PATE
70.7
73.6
80.9
58.6
62.4
69.3
DP-MERE
72.7
85.7
61.2
72.4
DataLeus
56.5
66.3
80.7
61.1
62.7
70.6
Si-Shnkborn
70.2
83.2
56.3
71.1
GS WGAN
83.3
85.5
84.9
58.7
59.5
63.1
DP-KIP FC-NTK (10 lmng/Cllsse)
93.53
95.39
87.7
89.1
DP-MERF
80.2
83.2
85.7
66.6
67.9
72.4
DP-KIP Se-KIP-SeatterrNe (10 Class)
94.5
98.0
88.0
89.3
DP-Sinkhorn
80.5
67.4
68.5
71.1
Chen et al 2022
(20 Imgs /Class)
80.9
95.6
70.2
77.7
Chen 2022
94.9
95.6
77.7
DP-KIP FCNTKK (20 hmgs/ (Clse))
97.78
97.96
88.3
90.2
DP-KIP FO-NTR
95.39
97.96
90.2
OP-KIP-ScatterNG (20 Imgs/Classs)
96.3
98.4
88.5
90.9
DP OP-IIP& (P-KIP-Srataee SeatterNel
98.0
98.4
89.3
90.9
Table 4: KRR test accuracy vs. % of corrupted pixels on CIFAR-10. Best accuracy results are obtained for DP-KiP-ScatterNet at different epsilon values and 8 10-5 with 1% and 5% of corrupted pixels.
Tigs/Claass Corrupted pixels (%)
0.1
c-1
10
100
46.6 0.5
50.4 + 0.4
46.8
0.2
50.4 0.3
52.6
0.4
54.5 0.3
0.3
50.9
51.1
+
0.2
49.9
52.4 1
41.4 0.3
45.2 0.2
48.3 +
51.0 =
38.4 0.3
40.1 0.3
45.9 + 0.4
49.5 + 0.3
48.6 = 0.4
54.4 0.3
52.5 0.3
54.9 + 0.1
55.7 + 0.2
56.2 0.1
10
53.9
0.1
54.1 0.1
54.6 1 0.1
55.3 1 0.1
46.3 0.2
52.1 0.1
53.6
53.9
46.2 0.3
47.5
52.2
52.7 +
41.6 0064000 0.1
46.5 0.2
50.2 1 0.1
50.9 +
49.76 + 0.5
58.7 + 0.2
55.2 1: 0.1
56.4 1 0.1
59.3 1 0.1
59.8 1 0.2
50
55.4 + 0.1
56.3 + 0.1
59.1 + 0.1
59.3 - 0.1
10
54.1 +
0.1
55.2 + 0.2
58.9 + 0.1
59.2 - 0.1
47.1 1.110000
0.1
48.8 1 0.1
55.7 1 0.1
56.4
46.5
0.2
47.3 0.1
53.3 0.1
55.2
trained on the privately distilled samples for DP-KIP FC-NTK under the same privacy budget e = 1 and 10-5, Details on hyperparameter scttings and classifiers used evaluation can be found in Sec. D.2 in Appendix.
For private synthetic methods we generate as many samples the target dataset contains while for DP-KIP FC-NTK we set the images per class to 10. Unsurprisingly, method outperforms the general data generation methods at the same privacy level.
6 Summary and Discussion
In this paper, we propose alternative kernels to infiniteddimcnsional NTK in KIP algorithm. The alternative kernels reduce computational resources required to run the algorithm (down to single GPU), while maintaining (even improving in particular cases) the quality of the distilled dataset. The computational requirement reduction has positive impact on the carbon footprint and makes the algorithm accessible for wider range of practitioners. Among all kernels tested, we empirically show that the kernel defined by ScatterNet features is the most suitable in image data experiments.
11

---

Published in Transactions on Machine Learning Research (05/2024)
MNIST
SVHN
Roal Data
6 0
0 / 4/23425
4 B5
6
CIFAR-100
Real Data
FASHION
CIFAR
6=1
Real Data
<= 10
=
Figure Generated image samples from DP-KIP-ScatterNel for different values.
Table 5: Performance comparison on Tabular datasets al = and 5 = 10-3 The average over five independent runs. For DP-KIP FC-NTK, 10 images per class are distilled and for private synthetic methods, as many samples as the original dataset contains are generated.
Real
DP-CGAN
DP-GAN
DP-MERF
DP-HP
DP-NTK (1,10-S)-DP
DP-KIP FO-NTK
(DD
(1,10-S)-DP
(1, (-
DDDPP
11,10-5)-DP
ROC
PRC
ROC
PRC
ROC
PRC
ROC
PRC
ROC
PRC
ROC
PRC
ROC
PRC
adult
0.786
0.683
0.509
0.444
0.511
0.445
0.642
0.524
0,688
0.632
0.695
0.557
0.662
0.365
census
0.776
0.433
0.655
0.216
0.529
0.166
0.685
0.236
0.699
0.328
0.424
0.766
0.408
cervical
0.959
0.858
0.519
0.200
0.485
0.183
0.531
0.176
0.616
0.312
0,631
0.335
0,622
credit
0.924
0.864
0.664
0.356
0.435
0.150
0.751
0.622
0.786
0.744
0.821
0.759
0.892
epileptic
0.808
0.636
0.578
0.241
0.505
0.196
0.605
0.316
0.609
0.554
0.648
0.326
0.654
0.585
isolet
0.895
0.741
0.511
0.198
0.540
0.205
0.557
0.228
0.572
0.498
0.53
0.205
0.615
0.955
F1
F1
F1
F1
F1
F1
F1
covtype
0.820
0.285
0.492
0.467
0.537
0.552
0.582
intrusion
0.971
0.302
0.251
0.892
0,890
0.717
0.873
As illustrated by Carlini et al. (2022 data distillation algorithms lack inherent privacy guarantees. In response to this privacy concern, we develop DP-KIP algorithm using DP-SGD on the existing KIP algorithm and implement it in JAX. Experimental results show that our proposed method outperforms existing data distillation methods at the same privacy level under image classification tasks with DP-KIP-ScatterNott achieving the highest performance. Additionally, we conduct experiments involving corrupted pixels on UP-KIP-ScatterVV on CIFAR10. The experiments with corrupted pixels demonstrate that fixing small percentage of randomly selected pixels (1%, 5%) and applying --Kl---SttterNet on the remaining pixels positively impacts the quality of the resulting distilled samples, surpassing the all-pixel optimization in terms of KRR accuracy.
We also evaluated DP-KIP in tasks involving the classification of tabular data using the infinitely-wide fully-connected NTK. The experimental findings indicate that, despite distilling only 10 images per class, our approach typically putperfomms differentially private methods for data generation, which generate samples matching the size of the target dataset. To the best of our knowledge, this is the first work that implements data distillation algorithm to tabular data classification problem.
In future work, we aim to enhance our proposed method further, narrowing the accuracy gap with respect to KIP performance, especially in the context of image datasets.
Acknowledgments
We (hank our anonymous reviewers for their constructive feedback, which has helped significantly improve our manuscript. We thank the Digital Research Alliance Canada (Compute Canada) for its computational resources and services. M. Vinaroz was funded by the Canada CIFAR AT Chairs program (at AMII), as a visiting international research student (VIRS) in the Department of Computer Science at the University of British Columbia. M. Park partially funded by Novo Nordisk Fonden RECUIT grant 00605800
12

---

Published in Transactions on Machine Learning Research (05/2024)
References
Martin Abadi, Andy Chu, lan Goodfellow, H. Brendan McMahan, lyy Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Securily- ACM, oct 2016. doi: 2L1145/2976749.2978388 URL
1.0.0.11522F29767499 2997729777779297781888 2978318
N. Aronszajn. Theory of reproducing kernels. Trans Am Math Soc, 68(3):337-440 -04, 1950. ISSN 00029947.
URL jstor wwtg/stable/1994400
Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Lí, Ruslan Salakhutdinov, and Ruosong Wang. On exact computation with an infinitely wide neural net. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Newral Information Processing Systems, 2019. URL https: //openreview //ppeeneviss...ttffffflllaraar lat.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Nocula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:///iithu.. com haam/google/jj/
Tianshi Cao, Alex Bie, Arash Vahdat, Sanja Fidler, and Karsten Kreis. Don't generate me: Training differen- tially private generative models with sinkhorn divergence. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 12480 12492. Curran Associates, Inc., 2021. URL htttps://proceeddingg wtwpooceeddnng..uuiiss...oomo neuripe. ceeuripp.cc/pape//20211 pdf
Nicholas Carlini, Vitaly Feldman, and Milad Nasr. No free lunch in "privacy for free: How does dataset condensation help privacy", 2022. URL https://arxiv wttp:i/w.rrgaa///2/9g/ 14987
Dingfan Chen, Tribhuvanesh Orekondy, and Mario Fritz. GS-WGAN: A gradient-sanitizel approach for learning differentially private generators. In Advonces in Newral Information Processing Systems 33, 2020.
Dingfan Chen, Raouf Kerkouche, and Mario Fritz. Private set generation with discriminative information. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (cds.), Advances in Neural Information Processing Systems, 2022. URL https: https://oppn::ii //tp://eeeeeeeeeeeeeeeee
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of mlr
Machine Learning Research, pp. 5378-5396. PMLR, 17-23 Jul 2022. URL press/v162/donss dongddc. html
Cynthia Dwork and Aaron Roth. The Algorithmic Foundations of Differential Privacy. Found. Trends Theor. Comput. Sci., 9:211 407, aug 2014. ISSN 1551-305X. doi: /04000/0400000042. URL http: //dx.do1. www.oo////.....//..... 15610000000000
Cynthia Dwork, Krishnaram Kenthapadi, Frank McSherry, Tyya Mironov, and Moni Naor. Our data, ourselves: Privacy via distributed noise generation. In Eurocrypt, volume 4004, pp. 486 503. Springer, 2006a.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Shai Halevi and Tal Rabin (eds.), Theory of Cryptography, pp. 265-284, Berlin, Heidelberg, 2006b. Springer Berlin Heidelberg. ISBN 978-3-540-32732-5.
Sam Fletcher and Md Zahidul Islam. A differentially private random decision forest using reliable signal-to- noise ratios. In Bernhard Pfahringer and Jochen Renz (eds.), AT 2015: Advances in Artificial Intelligence, pp. 192 203. Springer International Publishing, 2015. ISBN 978-3-319-26350-2.
Frederik Harder, Kamil Adamczewski, and Mijung Park. DP-MERF: Differentially private mean embeddings data generation. In Arindam Banerjee and Kenji
with randonfeatures for practical Fukumizu (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics,
13

---

Published in Transactions on Machine Learning Research (05/2024)
volume 130 of Proceedings of Machine Learning Research, pp. 1819-1827. PMLR, 13-15 Apr 2021. URL http:///rroceedings mlr. /ttp///dddngg......ppree//////// Titesss/v130//arre22111 htmm.
Frederik Harder, Milad Jalali, Danica J. Sutherland, and Mijung Park. Pre-trained perceptual features
improve differentially private image generation. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:// http:///pppppppeeeeeeee//e/oee/a
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognàtion (CVPR), pp. 770-778, 2016. doi: N0.1109/CVPR.20160
Minyoung Huh, Pulkit Agrawal, and Alexei A. Efros. What makes imagenet good for transfer learning?, 2016.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- ume 31. Curran Associates, Inc., 2018. URL https://proceeddings www..pe.riiioooooooooooe /0///////////1///fllee babel188he62b8866666111166666563 Pappeee.aaaaapp..ppoop
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super- resolution. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016, pp. 694-711, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46475-6.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
Yann Lecun, Léon Botton, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document. recognition. Proceedings of the IEEE, 66(11):2278 2324, 1998. doi: 00.1109/5.726791
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Onlinej.
Available: http://yarm.. wew.n..co. caaticcomeeeaee /emdb/ /mnist, 2, 2010.
Jachoon Lee, Lechao Xiao, Samnel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl-Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve as linear models under gradient descent. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-But E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https; //proceedings neurips. http://pppppp..././/////....//..
.pdf
Jaehoon Lee, Samuel Schoenholz, Jeffrey Pennington, Ben Adlam, Lechao Xiao, Roman Novak, and Jascha Sohl-Dickstein. Finite versus infinite neural networks: empirical study. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Newral Informalion Processing Systems, volume 33, pp. 15156-15112 15172. Curran Associates, Inc., 2020. URL https::/proceeeddngg neurips cttpcc/c//////c. cc/paper/2020// pdf
Yaoyao Liu, Yuting Su, An-An Liu, Bernt Schiele, and Qianru Sun. Mnemonics training: Multi-class incremental learning without forgetting. In Proceedings of the IEEE/CVE conference on Computer Vision and Pattern Recognition, pp. 12245-12254 2020.
Yunhui Long, Boxin Wang, Zhuolin Yang, Bhavya Kailkhura, Aston Zhang, Carl Gunter, and Bo Li. G- PATE: Scalable differentially private data generator via private aggregation of teacher discriminators. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 2965-2977 Curran Associates, Inc., 2021. URL https : .dd
Noel Loo, Ramin Hasani, Mathias Lechner, and Danicla Rus. Datasct distillation fixes datasct roconstruction
atlacks. Xixee preprint arXiv:2302.014:: 2023.
Wojeiech Masarezyk and Ivona Tautkute. Reducing catastrophic forgetting with lcarning on synthotic data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 252 253, 2020.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupercised Feature Learning 2011, 2011. URL http://ufld. wtw.lldddstaafooo.o htt.://////ooouuumbbrr///ssp0...
housenumbers pdf
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. In International Conference Learning Representations, 2021a. URL http:://ppearre7eer..ee///uu
id=1-PrrQrKOnp
Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely
wide convolutional networks. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 5186 5198. Curran Associates, Inc., 2021b. URL
.df.
E. Oyallon and S. Mallat. Deep roto-translation scattering for object classification. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2865-2873, Los Alamitos, CA, USA, jun 2015. TEEE Computer Sociely. doi: 1.1100/CVV..2015.77555 1t1p//0..1////PR..2222 2015 7298904
URL httpp:://ooi. ittar///dd....eeccpppterrcccii
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. In NewrIPS, 2019.
Andrea Rosasco, Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Distilled replay: Overcoming forgetting through synthetic samples. In International Workshop on Continual Gemii-Spercisell Learning, pp. 104-117. Springer, 2021.
Maltia Sangermano, Antonio Carta, Andrea Cossu, and Davide Bacciu. Sample condensation in online continual learning. In 2022 International Joint Conference on Neural Networks (IJCNN), pp. 01-08. IEEE,
2022.
Cicero Nogucira. Dos Santos, Youssef Mrouch, Inkit Padhi, and Pierre Dognin. Learning implicit generative models by matching perceptual features. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 4460-4469, 2019. doi: 10. wTW.0//TCCCV......00.0 2019.00466.
Samuel Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations, 2017.
Reihanch Torkzadehmahani, Peter Kairous, and Benedict Paten. DP-CGAN: Differentially private synthetic data and label generation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
Florian Tramer and Dan Bonch. Differentially private learning needs better features (or much more data). In International Conference Learning Representations, 2021. URL http:////opprrrriieeneeeeeteteee
YTWGvPFOQD-1
Margarita Vinaroz, Mohammad-Amin Charusaie, Frederik Harder, Kamil Adamczewski, and Mi Jung Park. Hermite polynomial features for private data generation. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 22300 -23224. PMLR, 17-23 Jul 2022. URL htttps://proceedingn .mr.innsssss....mmm.mm...m.mmmem www...p.ee/....... wwtppp/wssssse/eeeeeepppppppeees wwwp//////aoo22aaaaaaa hhmml
Boxin Wang, Fan Wu, Yunhui Long, Luka Rimanic, Ce Zhang, and Bo Li. Datalens: Scalable privacy preserving training via gradient compression and aggregation. In Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, CCS '21, pp. 2146 2168, New York, NY, USA,
15

---

Published in Transactions on Machine Learning Research (05/2024)
2021. Association for Computing Machinery. ISBN 9781450384544. doi: 10.1145/3460120.348822 URL https://doi.org// 100...1153340100 10.1144533461120444447777 3484579
Tongzhou Wang, Jun-Yan Zhu, Antonio Torrala,, and Alexei A. Efros. Dataset distillation.
CoRR,
abs/1811.10959,, 2018. URL http:://aax....rg////////////11/
Yu-Xiang Wang, Borja Balle, and Shiva. Prasad Kasiviswanathan. Subsampled rényi differential privacy and analytical moments accountant. volume 89 of Proceedings of Machine Learning Research, pp. 1226 1235. PMLR, April 2019.
Felix Wiewel and Bin Yang. Condensed composite memory continual learning. In 2021 International Joint Conference on Neurul Nelworks (IJCNN), pp. 1.8. IEEE, 2021.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms. Xiv preprint arXiv: nem:1708.07745 2017.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of cmns: How to train 10,000-layer vamilla convolutional neural networks. In International Conference Machine Learning, pp. 5393-5402 PMLR, 2018.
Lechao Xiao, Jeffrey Pennington, and Samuel Schoenholz. Disentangling trainability and generalization in deep neural networks, 2020.
Liyang Xie, Kaixiang Lin, Shu Wang, Fei Wang, and Jiayu Zhou. Differentially private generative adversarial network. CoRR, als//102..06733 2018.
Yilin Yang, Kamil Adamczewski, Danica J. Sutherland, Xiaoxiao Li, and Mijung Park. Differcntially private neural tangent kernels for wrivacy-preserving data generation, 2023.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and
K.Q. Weinberger (ods.), Advances in Newral Information Processing Systems, volume 27. Curran Associates, Inc., 2014a. URL 37667339999560o2cdce777770000-00
pdf
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep neural networks? Aduances in neural information processing systems, 27, 2014b.
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. In International Conference on Machine Learning, pp. 12674 12685. PMLR, 2021.
Bo Zhao and Hakan Bilen. Dataset condensation with distribution matching. In Proceedings of the IEFE/CV Winter Conference on Applications of Computer Vision, pp. 6514-6523, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference Learning Representations, 2021. URL httpg:///opameeiiie.neeffoooooo
mSAKhLYLSs1
Tianhang Zheng and Baochun Li. Differentially private dataset condensation, 2023. URL https: //openrvvies .ett://orruiiidd42pppo....
16

---

Published in Transactions on Machine Learning Research (05/2024)
Appendix
A Clipping effect of the gradients
In this scction empirically show the effect of clipping the gradients for Pe-KIP-ScatterNV in terms of
KRR accuracy, Table G shows the accuracy for 1 Ingg/Class for MNIST at 1, and d = 10-5.
Table 6: KRR accuracy for Imgs/ Class distilled dataset on MNIST at different C.
Lngs/Class
10-1
10-2
10-3 3
10
10-5
10-6
81.5 0.4
94.7 + 0.3
93.7
0.4
93.9 + 0.4
93.5 + 0.5
90.5 + 0.2
10
80.2 0.6
96.1 + 0.2
94.8 = 0.2
94.9 + 0.1
94.7 + 0.2
94.2 = 0.4
B
Noiise ratio of privatizing the feature maps
Definition B.1. Signal-to-Noise Ratio (SNR) Fletcher 8 Islam 2015) The Signal-to-Noiss Ratio of
measwrement can be expressed as signal where the signal and noise expressed in the same units. An alternative way of uriling this is:
SNR
(7)
where is the signal mean or expected value and is the standard deviation of the noise.
Now, want to to what's the SNR in DP-KIP scenario for privatizing data dependent feature maps once and reuse them during training. Consider a target dataset D: = {(xt, Mt, ) with input features X RRD and scalar labels Vr, Let k; Rº x IR be a positive definite kernel. By Moore Aronszajn theorem Aronszajn (1950), here exists unique reproducing kernel Hilbert space of functions on RR for which k is reproducing kernel and thus, we can find a feature map, o: RD + H. such that A(x, x') = (Ø(x), o(x')) ))x for any pair of datapoints x, x' E X.
We start by deriving the sensitivity of the feature map for neighboring datasets D and D'. Without loss of generality, consider that D and D' differ in the last datapoint such that Xt,, # x!, and normalized feature maps, [((xx,) = 1, vi E [n].
((x;) o(x(s)
o(xi,) o(x(a)
A
max [x(x).)) 0(x( 122 $2 D,D
D,D
(xx).))
((x))
where the last inequality is due to the triangular inequality.
Now, we compute the SNR for the the feature map representations of datapoints in D,:
o(x;,)
o(x())
[x(x))))
11
+
+
[xx(,,,)))
Vn
1
SN Ro(x))
(9)
Vnd3o2
n1 00
2/na
2
17

---

Published in Transactions on Machine Learning Research (05/2024)
Therefore, observe that signal-to-noise ratio of the feature maps is inversevely proportional to the standard deviation parameter This implies that the noise will dominate the signal-to-noise ratio unless is relatively small value which will only be the case for large values since od and (2ERDP(a)
log(1/8) cndr(a- 1) for a given 8 < 1/n. (6) min a-1
C Tabular datasets description
Table contains detail information about the 8 different tabular datasets used in Sec. 5.2
Table 7: Tabular datasets. Size, number of classes and feature types descriptions.
dataset
samps
# classes
features
isolet
4366
2
617 mum
covtype
406698
7
10 num, 44 cat 178 mum
epileptic
11500
2
credit
284807
2
29 num
cervical
753
2
11 num, 24 cat
census
199523
2
7 num, 33 cat
adult
48842
2
num, 8 cat
intrusion
394021
5
8 cat, 6 ord, 26 num
D Expermental details
D.1
Image data
Here we provide hyperparameter seltings used during image data KIP experiments. Table 8 Table g and
Table show the number of epochs, batch size, learning rate and regularization parameter used in KIP e-NTK, KIP PFs and KIP ScattterNet respectively.
Table 8: KIP c-NK hyperparameter settings for Image data.
Imgs/ Class
epochs
batch size
learning rate
X
1
1000
1000
5 10-3
10-7
MNIST
10
1000
1000
5 10-3
10-7
50
10
500
10-2
10
100
100
10-3
10-2
FASHION
10
100
100
10-3
10-1
50
10
100
10-2
10-1
100
2000
10-2
10-3
SVHN
10
100
2000
10 10-2
10-4
50
10
1000
10-4
10-3
1
100
200
10
10
CIFAR-10
10
100
1000
10-2
10-1
50
100
2000
10
10-2
1
100
1000
10-1
10-3
CIFAR-100
10
1000
50
10-1
10-3
50
1000
50
10-1
10-2
18

---

Published in Transactions on Machine Learning Research (05/2024)
Table 9: KIP PFs hyperparameter settings for Image data.
Imgs/
Togss/llass Class
epochs
batch size
learning rate
1
100
1000
10 10-2
10
MNIST
10
100
5000
10-2
10-5
50
100
2000
10
10
1
1000
2000
10-2
10-5
FASHION
10
1000
1000
10
10-5
50
1000
5000
10
10
1
100
2000
10-2
10-3
SVHN
10
100
2000
10 2
10
50
1000
100
10
10
1
1000
2000
10
10-4
CIFAR-10
10
1000
5000
10-3
10
50
1000
2000
10-3
10-2
1000
1000
10-2
10-3
CIFAR-100
10
1000
200
10
10
50
100
200
10-2
10-3
Table 10: KIP ScatterNet hyperparameter scttings for Image data.
Imgs/ Class
epochs
batch size
learning rate
X
1
1000
200
10 10-4
MNIST
10
2000
100
10-4
10-5
50
2000
5000
10-4
10-5
1000
1000
10-3
FASHION
10
2000
2000
10-3
10-6
50
2000
2000
5 10-3
10-2
1000
200
10-4
SVHN
10
1000
1000
10 10-3
10-4
50
8000
5000
10-3
10-3
1
1000
2000
10
CIFAR-10
10
1000
2000
10
10
50
1000
5000
10-3
10-3
1
10000
2000
10
10-3
CIFAR-100
10
1000
5000
10-2
10-3
50
2000
1000
10
10-1
Here we provide the details of the DP-KIP training procedure we used on the image data experiments. Table and Table 12 show the number of epochs, batch size, lcarning rate, clipping norm C and regularization parameter X used during training for each image classification dataset at the corresponding images per class distilled for DP-KIP FC-NTK. Table 13 and Table
I4 show the different hyperparameter settings used in
experiments.
D.2 Tabular data
Table 15
contains DP-KIP FC-NTK hyperparameter settings used in tabular data experiments. Table describes the hyperparameter setting for training the downstream classifiers used in tabular data experiments.
16
19

---

Published in Transactions on Machine Learning Research (05/2024)
Table 11: DP-KIP FC-NTK hyperparameter settings for Image data for 1 and å
= 10
Imgs Class
epochs
batch size
learning rate
C
A
10
500
10-3
10-2
10-6 6
MNIST
10
10
500
10
10-6 6
10-6
50
10
500
10-2
10-6
10-7
10
500
5 10-2
10-6
10-6
FASHION
10
10
500
0.1
10
10-5
50
10
200
10-2
10-2 10-6
10
50
10-1
10-6 10-6
SVHN
10
10
500
5 10-2
10-6
10-6
50
10
500
10
10 10-2
20
200
10-2
10
10-5
CIFAR-10
10
10
500
5 10
10-5
10-6
50
10
500
5-10-2
10
10-6
10
200
10
10-5
10-7
CIFAR-100
10
10
100
10-2
10
10-7
50
10
50
10
10
10-7
Table 12: DP-KIP FC-NTK hyperparameter settings for Image data for 10 and 5 = 10-5
Imgs/Clian CClass
epochs
batch size
learning rate
C
A
10
10-3
10-5
10-6
MNIST
10
10
500
5 10-3 3
10-2
10 6
50
10
500
5. 1010-3
10-5
10-6
10
500
10-2
10-3
10-6
FASHION
10
10
500
10-2
10
10-5
10
500
10-2
10-2
10-7
10
50
5- 5-10"
10
10
SVHN
10
10
500
10-1
10-6
10-6
50
10
500
10-2
10
10-7
10
500
10-1
10-6
10-6
CIFAR-10
10
10
500
5. 10-0
10-5
10-6
50
10
500
5-
10-5
10-6
10
100
10-2
10-5
10-6
CIFAR-100
10
10
100
10
10-4
10-6
50
10
100
10-2
10
10-6
20

---

Published in Transactions on Machine Learning Research (05/2024)
Table 13: DP-KTP-ScatterNet hyperparameter settings for Image data for = 1 and å = 10
Imgs Imgs/Class CClass
epochs
batch size
learning rate
C
A
20
5000
10-1
10-2
10-4
MNIST
10
30
500
10
10-7
10-3
50
30
200
10-1
10-2
10-2
50
2000
10-1
10
10-7
FASHION
10
40
1000
10
10 10-3
20
200
10-2
10-5 10-2
50
2000
10-1
10 10-7
SVHN
10
40
1000
10-2
10 10-3
50
20
200
10
10
10-2
40
2000
10-1
10-6
10-6
CIFAR-10
10
20
500
10
10-6
10-3
50
30
100
10
10
10-1
30
200
10
10 6
10-2
CIFAR-100
10
10
50
10
10-6
10-1
50
10
50
10-4
10-6
10-2
Table 14: DP-KIP-ScatterNG hyperparameter settings for Image data at = 10 and 8 10-5
Imgs/Cless CClas
epochs
batch size
learning rate
C
A
30
500
10-1
10-2
10-6 6
MNIST
10
50
2000
10-1
10-6
10-4
50
30
200
10-1
10-7
10-1
50
1000
10-1
10-2
10-1
FASHION
10
50
2000
10
10
10-3
50
200
10-2
10-2
10-3
50
2000
10
10
10
SVHN
10
50
500
10-2
10-5
10-3
50
50
100
10-2
10-2
10-3
50
1000
10-2
10-2
10-5
CIFAR-10
10
50
500
10-2
10-1
10-1
50
50
100
10-2
10-5
10-4
50
1000
10-1
10-6
10-5
CIFAR-100
10
50
50
10-1
10-6
10-2
50
50
50
10
10
10-2
21

---

Published in Transactions on Machine Learning Research (05/2024)
Table 15: DP-KIP FC-NTK hyperparameter settings for tabular data.
dataset
epochs
batch size (%)
learning rate
C
X
subsampling rate
isolet
10
0.8
10-3 3
10-3
10 10-6 6
10-2 2
covtype
10
0.02
10-1
10-2
10-7
7 10-2
epileptic
10
0.07
10-2
10-3
10-6
3- 3-10-1
credit
10
0.07
10-2
10-1
10-6
10-2
cervical
10
0.6
10-4
10-3
10-6
2. 3010-2
census
10
0.4
5 5.10-1
10-1
10-6
4.10-2
adult
10
0.8
10-2
10-1
10-6
7. 7.10-
intrusion
10
0.05
10-1
10-4
10-6
10-2
Table 16: Hyperparameter settings for downstream classifiers used in tabular data experiments. Models are laken from scikit-learn 0.24.2 and xgboost 0.90 python packages and hyperparameters have been set to achieve reasonable accuracy while limiting execution time. Parameters not listed are kept default values.
Model
Parameters
Logistic Regression
solver: 1bfgs, max iter: 5000, multi_classe class:
Gaussian Naive Bayes
Bernoulli Naive Bayes
binarize: 0.5
LinearSVC
max iter: 10000, tol: 1e-8, loss: hinge
Decision Tree
class weight: balanced
LDA
solver: eigen, n. components: 9, tol: 1e-8, shrinkage: 0.5 esttimattos:: 1000, learning rate: 0.7, algorithm: SAMME.R max samples: 0.1, n estimators:
Adaboost
Bagging
Random Forest
class_weight: subsample: 0.1, n estimators: 50
n ostimators: 100, class weight: balancod
Gradient Boosting
MLP
XGB
colsample_bytree: bytree 0.1, objective: multi:sofprrob, n. estimators:
22


Reviews:
[{"review_id": "PoiMaWwD68", "reviewer": "Reviewer_QSM9", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n**S1:** Although the initial idea mentioned in the paper proposes a DP data distillation framework based on KIP, which simply utilizes DP-SGD, the authors have investigated various alternative features. This investigation allows them to identify the kernel that utilizes features from a Scattering Network as the best method.\n\n**S2:** The paper is well written, and the ideas are very clear.\n\n**Weaknesses**\n\n**W1:** a few typos in Section 2.6: \"its\" --> it's ; \"differencein\" --> \"difference in\"\n\nOverall, the paper provides an elegant solution that outperforms existing approaches and deserves to be accepted.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel algorithm, differentially private kernel inducing points (DP-KIP), which employs DP-SGD for provable privacy preservation. Initially, the intention was to apply DP-SGD directly to the KIP framework, but findings showed that KIP using infinitely-wide convolutional neural tangent kernels (conv-NTKs) offered better performance than fully-connected NTKs. However, this approach was deemed impractical due to its high computational demand. As a solution, DP-KIP-ScatterNet was proposed, utilizing ScatterNet's wavelet features instead of conv-NTKs to reduce computational costs significantly. The paper outlines three main contributions: the development of a DP data distillation framework that incorporates DP-SGD for privacy guarantees; the enhancement of DP-KIP-ScatterNet's performance by adjusting pixel corruption rates to achieve better privacy-accuracy trade-offs; and the extension of DP-KIP's application beyond image data to include tabular datasets, demonstrating its superiority over existing DP data generation models with a minimal number of distilled samples per class.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please correct the typos mentioned in **W1**.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZvYolzkWoZ", "reviewer": "Reviewer_fqXX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n\n- This manuscript focuses on an exciting and important problem with a well-organized structure that makes it easy to follow. \n\n- It follows a logical sequence, explaining the authors' steps to develop their final proposed method. \n\n- The code is also publicly available, benefiting the community. \n\n**Weaknesses**\n\nThe experimental results call for a more thorough analysis and clarification.\n\n- When interpreting the results outlined in Table 3, it remains unclear why the test accuracy gap between epsilon=1 and epsilon=10 is considerably smaller for the DP-KIP-Scatternet than the corresponding gap in the comparison methods.\n\n- Furthermore, in both sub-tables of Table 3, there are some absent comparisons, despite the availability of open-source code for the studied previous works, allowing for the conduct of experiments. For instance, in the left table, it is feasible to acquire results for DP-CGAN, GS-WGAN, and DP-Sinkhorn for epsilon=1 for both MNIST and FASHION-MNIST. Similarly, in the right table, DP-Sinkhorn, DataLense, and G-PATE results are not included as another example.  \n\n- While the utility of downstream tasks demonstrates superior performance with DP-KIP-ScatterNet compared to previous related works, a significant discrepancy is observed in the visual quality of the distilled dataset when compared to other methods (Figure 2). The authors are encouraged to explain this gap and discuss its potential impact on the overall usability of the distilled dataset across various downstream tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work addresses privacy concerns associated with distilled datasets by proposing a differentially private data distillation algorithm. The proposed approach utilizes a data distillation framework based on the Kernel Inducing Points (KIP) methodology, leveraging features from a ScatterNet. The objective is optimized using Differentially Private Stochastic Gradient Descent(DP-SGD) to provide a privacy guarantee during the distillation process. \n\nIn the non-dp setting, the experimental evaluation involves the comparison of different feature extractors for KIP. It includes an exploration of feature extractors such as infinite-width convolutional Neural Tangent Kernel (NTK), empirical NTK, perceptual feature extractor, and ScatterNet. \n\nGiven the superior performance of KIP with convolutional NTK and KIP with ScatterNet over other studied feature extractors on the examined datasets, this work makes these two data distillation approaches differentially private using DP-SGD. Moreover, it evaluates them on image and tabular datasets while incorporating varying privacy guarantees (epsilon in DP). The experimental findings indicate that DP-KIP with ScatterNet outperforms DP-KIP with convolutional NTK and some state-of-the-art competitors on the investigated datasets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Here is the summary of the requested changes, but please check the weakness section for more details:\n\n-  Providing more clarification on the results presented in Table 3\n-  Extending the experiments associated with Table 3\n- Discussing the low visual quality of the distilled datasets (figure 2) and its potential negative impact on the usability of the proposed method", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Sa1Frut9Gb", "reviewer": "Reviewer_UNdv", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1) The combination of formal privacy and data distillation could have the potential to lead to interesting and efficient methods for privacy-preserving learning.\n\n2) The paper is clearly written, and the authors present clear evidence for the stated claims.\n\n3) Resource-efficiency and privacy are both important and timely research topics.\n\n### Weaknesses\n\n1) The proposed method is a fairly simple combination of existing methods (KIP for DD and DP-SGD for privacy.)\n\n2) I have several questions for the authors and comments detailed below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper considers the problem of data distillation (DD), i.e., how to construct a distilled dataset that is significantly smaller than the original training data but would result in similar model utility, with formal differential privacy (DP) guarantees. The authors leverage an existing DD method based on kernel inducing points (KIP) using kernel ridge regression (KRR) and the well-known DP stochastic gradient descent (SGD) to derive formal privacy guarantees for DD. The existing KIP for DD method is based on infinitely wide convolutional neural tangent kernels (NTK), which require considerable amount of compute. The authors therefore propose and evaluate several more resource-friendly kernel approximations based on i) empirical NTK (finite-dimensional approximation to the infinite version), ii) on ScatterNet features (hand-crafted features for image recognition), iii) and on perceptual features (activations from all levels on an untrained neural network). The authors experimentally show that their method provides improved privacy-utility trade-off compared to existing DP-DD methods on several image datasets, and that the trade-off can be further improved by corrupting a small amount of the pixels in the original training data images. They further evaluate the DP-KIP based on infinite-wide fully-connected NTK on several tabular datasets to show that DD can work beyond image data. Overall, I think the authors mostly present convincing empirical evidence for the stated claims.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please address the following questions/suggested changes (in decreasing order of importance):\n\n1) Contributions: \"we look for alternative features and find that the features from ScatterNets are the most useful...\". Please mitigate this claim somewhat: there is no theory on general utility, and the empirical tests only look at a handful of possible choices.\n\n\n2) Sec 2.6: State the neighbourhood definition (e.g. add/remove or replace) and protection granularity (sample-level?) explicitly.\n\n3) Sec 2.6: From the DP composition description I gather that you use RDP for privacy accounting. Please state this clearly, and do not call it advanced composition (since this refers to a well-established standard accounting method not based on RDP). If this is correct, then also explicitly state how you change from RDP to approximate DP.\n\n4) Sec 2.6: RDP(?) based accounting of Wang et al. 2019 \"yield tight bounds on the cumulative privacy loss\". RDP accounting is always lossy, not tight, although a lot tighter than the advanced composition. Please reformulate this.\n\n\n5) Sec 3.1: have you tried optimising the labels with/without DP to see how much utility this might get?\n\n6) Sec 5.1: Consider (partly) emphasising all methods for which the mean + reported variability overlap the best method mean.\n\n7) Sec 5.1, p9, the paragraph starting with \"Our intuition fro the interpretability ...\": is there any theory or experiments to support these ideas?\n\n8) Sec 5.1, p9, on Fig 2: have you tested whether the image samples become more human-readable at some point when increasing epsilon?\n\n9) Table 2: Do the benefits of having more images per class saturate at some point before reaching the original training data size?\n\n\n10) Appendix B, p.18: Note that the stated sigma is not correct for the Gaussian mechanism using RDP, the formula uses the classical Gaussian mechanism, which requires epsilon < 1. See Balle and Wang 2019: Improving the Gaussian mechanism for DP.\n\n11) Sec 2.4: State what is K/what effect K has, why the chosen numbers.\n\n\n### Typos and other minor changes\n\np9 2. paragraph: FP -> PF.\n\np17 below eq8 : traingular -> triangular.", "broader_impact_concerns": "I have no broader impact concerns on this paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All three reviewers were positive about the contribution of differentially-private data distillation and techniques to make it computationally tractable. They are also satisfied with the revised paper, produced through an active discussion between them and the authors. I therefore support the decision: "accept as-is."
Decision: Accept (poster)