Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation
Authors: Noel Loo
Ramin Hasani
Mathias Lechner
Alexander Amini
Daniela Rus
Venue: ICLR 2024 poster (2024)
Keywords: Dataset Distillation
Reconstruction Attacks
Neural Tangent Kernel
Primary area: general machine learning (i.e., none of the above)
TL;DR: We analyze parameter-based reconstruction attacks from an NTK perspective and show that it is a variant of dataset distillation

Abstract: Modern deep learning requires large volumes of data, which could contain sensitive or private information that cannot be leaked. Recent work has shown for homogeneous neural networks a large portion of this training data could be reconstructed with only access to the trained network parameters. While the attack was shown to work empirically, there exists little formal understanding of its effective regime and which datapoints are susceptible to reconstruction. In this work, we first build a stronger version of the dataset reconstruction attack and show how it can provably recover the \emph{entire training set} in the infinite width regime. We then empirically study the characteristics of this attack on two-layer networks and reveal that its success heavily depends on deviations from the frozen infinite-width Neural Tangent Kernel limit. Next, we study the nature of easily-reconstructed images. We show that both theoretically and empirically, reconstructed images tend to ``outliers'' in the dataset, and that these reconstruction attacks can be used for \textit{dataset distillation}, that is, we can retrain on reconstructed images and obtain high predictive accuracy.

Published as a conference paper at ICLR 2024
UNDERSTANDING RECONSTRUCTION ATTACKS WITH THE NEURAL TANGENT KERNEL AND DATASET DIS- TILLATION
Noel Loo, Ramin Hasani, Mathias Lechner, Alexander Amini and Daniela Rus
MIT CSAIL
Cambridge, Massachussetts, USA (1o0, rhasani, mlechner, amini, rus)@mit.ed
ABSTRACT
Modern deep learning requires large volumes of data, which could contain sen- sitive private information that cannot be leaked. Recent work has shown for homogeneous neural networks a large portion of this training data could be recon- structed with only access to the trained network parameters. While the attack shown to work empirically, there exists little formal understanding of its effective regime and which datapoints are susceptible reconstruction. first build a stronger version of the dataset reconstruction attack and show how it can provably recover the entire training set in the infinite width regime. We then empirically study the characteristics of this attack on two-layer networks and re- veal that success heavily depends on deviations from the frozen infinite-widde Neural Tangent Kernel limit. Next, we study the nature of rasily-reconstructed images. We show that both theoretically and empirically, reconstructed images tend to "outliers" in the dataset, and that these reconstruction attacks can be used for dataset distillation, that is, we can retrain on reconstructed images and obtain high predictive accuracy
this work, we
INTRODUCTION
Neural networks have been shown to perform well and even generalize on a range of tasks, despite achieving loss training data (Zhang et al.) 2017 2021). But this performance often cannot be realized in practice due to security constraints in deployment. A fundamental question in the security of neural networks is how much information is leaked via this training procedure, that is, can adversaries with access to trained models, or predictions from a model, infer what data was used train the model? Ideally, we want to ensure that our models are resistant to such attacks. However, in practice, we see that this ideal is commonly violated. One heinous violation of this principle is the phenomenon of memorization (Arpit et al.] 2017, Feldman Zhang 2020b Feldman 2020) Carlini et al.] 2019), where trained networks can be shown to replicate their training data at test time in generative models. A more extreme example of memorization presented in Haim et al. (2022). where the authors show that it is possible to recover a large subset of the training data given only the trained network parameters.
The existence of this attack begs many follow-up questions: "Under what circumstances is this tack successful??; and "What are properties of these recovered images?" this paper, we consider stronger variant of the attack presented in Haim al. (202)), provide novel theoretical and em- pirical insights about dataset reconstruction attacks, and provide answers to the above questions. In particular, make the following new contributions:
We design a stronger version of Haim et al. (2022)'s 's dataset reconstruction attack that can provably reconstruct the entire training set for networks in the neural tangent kernel (NTK) (Jacol et al. 2018) regime when trained under mean squared error (MSE) loss. This attack transfers to finite networks with its success dependent on deviations from the NTK regime.
Cooee available at httpe///////..ooooooo/oo////nnnn

---

Published as a conference paper at ICLR 2024
show that outlier datapoints prone to reconstruction under our attack, corroborating prior work observing this property. Additionally, we show that removing easily reconstructed images can improve predictive accuracy.
We formally prove and empirically show that a dataset reconstruction attack is a variant of dataset distillation. The reconstruction loss is equal to the loss of the kernel-inducing points (KIP) (gguyen et al. 2021a|b dataset distillation algorithm, under a different norm, plus variance- controlled term. Furthermore, we can retrain models using recovered images and achieve high performance.
2 BACKGROUND AND RELATED WORKS
Machine Learning Privacy. A large body of work studies how to extract sensitive information from trained models. This is problematic as legislation such as HIPAA and GDPR enforce what data can and cannot be published or used (Centers for Medicare & Medicaid Services 1996 Eu-
ropean Commission) (016)). Quantifying the influence of training examples leads to the topic of influence functions (Koh & Liang] 2017), and Membership-infereen
attacks (Shokri et al. 2016
Carlini et al. [021). which try to infer whether particular examples were used in training. Defending against these attacks is the study of differential privacy, which quantifies and limits the sensitivity of
models to small changes in training data (Dwork et al.] [2066] Abaai et al. 2016). Likewise, the field of machine unlearning tries to remove the influence of training examples post-training (Bourtoule
jet al.] (019)). Without defense techniques, trained networks have been shown to leak information
(Rigaki & Garcia 2020). For generative language models, (Carlini al. 2019) show large language models often reproduce examples in the training corpus verbatim. Model inversion techniques aim to recreate training examples by looking at model activations (Fredrikson et al.) 2015 Yang et al.)
2019 He et al., 2019). This memorization phenomenon can be shown to be necessary to achieve high performance under certain circumstances (Feldman & Zhang 2020b Brown et al. 2021).
Dataset Reconstruction. Recent work HHai et al. 2022) has shown that one can reconstruct a large subset of the training data from trained networks by exploiting the implicit biases of neural nets. They note that homogeneous neural networks trained under logistic loss converge in direction
to the solution of the following max-mane mar-mn problem (Lyu & Li 2020 Ji & Telgarsky 2020):
arg min 11111
1 O' 2
s.t. Vi vn.o (x.) 1,
(1)
Where ( Jif is the training set with images x; and labels yi E (+1,-1}, and for (x) the neural network output with parameters 0'. (Haim et al. (022) shows that by taking a trained neural network and optimizing images (and dual parameters) match the Karush- Kaai-KKunn--eeeehe
(KKT) conditions of the max- margin problem, it is possible to reconstruct training data. This is an attack that causes leakage of training data. Here, we consider a stronger variant of the attack that requires training under mean-squared error (MSE) loss.
Neural Tangent Kernel. To investigate the generalization in neural networks we can use the neural tangent kernel (NTK) theory (Jacot et al. 2018 Arora et al. 2019). NTK theory states that networks behave like first-order Taylor expansions of network parameters about their initialization as network width approaches infinity (Lee al., 2019). Furthermore, the resulting feature map and kernel converge to the NTK, and this kernel is frozen throughout training (Jacot et al. 2018 AAooa et al.) 2019). As result, wide neural networks analogous to kernel machines, and when trained with MSE loss using a support set X with labels ys result in test predictions given by:
dr = K
with K being the NTK. For fully-connected networks, this kernel can be computed exactly very quickly (as they reduce to arc-cosine kernels), but for larger convolutional networks, exact compu- tation slows down dramatically (Arora et al., 2019 Zandich et al. 2021). In practice, it has been shown that networks often deviate far from the frozen-kernel theoretical regime, with the result- ing empirical NTKs varying greatly within the first few epochs of training before freezing for the rest (Hanin& & Nical 2020 Aitken & Gur-Ari, 2020, Fort et al. 2020 Loo al. 2022b Tsilivis & Kempel 2022). In this paper, we use the NTK theory to gain a better understanding of these reconstruction attacks.
2

---

Published as a conference paper at ICLR 2024
Dataset Distillation. Dataset distillation aims to construct smaller synthetic datasets which ac- curately represent larger datasets. Specifically, training on substantially smaller distilled dataser achieves performance comparable the full dataset, and far above random sampling of the dataset [Wang et al. 2018 /hhao et al. 2021 Zhhao & Bilen 2021 Nguyen et al. 2021a /Zhou et al., 2022]
[oo et al.) 2022a). There are many algorithms for this, ranging from methods that directly unroll computation (Wang et al.) 2018). try to efficiently approximate the inner unrolled computation as-
sociated with training on distilled data (Zhou et al. 2022 Loo et al. 2022a Nguyen al. 202Ib),
and other heuristics (Zhao et al. 2021; & Bilen 2021). One algorithm is
points
(KIP) (Nguyen et al. 2021a
which leverages NTK theory to derive the following loss:
1
CKIP = llve Krssssusll
The loss indicates the prediction error of infinite width networks on distilled images Xs and labels ys. which are then optimized. We bring up dataset distillation we show in this paper that our dataset reconstruction attack generalization of KIP, and that dataset distillation can be used to defend against the attack.
A NEURAL TANGENT KERNEL RECONSTRUCTION ATTACK
Haim et al. (2022) considers the scenario where the attacker only has access to the final trained network parameters. This attack requires that the networks are homogeneous and are trained for many epochs until convergence so that the network converges in direction to the final KKT point of Eq. [ While it is a good proof-of-concept for such attacks, there are several theoretical and practical limitation
this attack. We find that the attack presented in Haim et al, (2022) is brittle. Namely, we were unable to reliably reproduce their results without careful hyperparameter tuning, and careful network initialization strategies. Their attack also requires training until directional convergence, which requires network parameters to tend to infinity, and requires homogenous networks. Furthermore, outside of the top few reconstructions, the overwhelming majority of recon- structions (> 70%) are of poor quality (more in-depth discussions in appendix A Here present an attack which is compatible with early stopping, does not require special initialization strategies, and can provable reconstruct the entire training set under certain assumptions, the first guarantee of reconstruction in any regime. However, our attack requires access to the model initialization or a previous training checkpoint. Access to the initialization or earlier training checkpoints arises naturally in many settings, such as fine tuning from public models, or in federated learning where clients receive period updates of the model parameters.
Note that we cannot compare our attack to other ones such as gradient leakage attacks 7Zuu et al., 2019) membership inference attacks (Shokri al.j 2016) and generative model attacks (Carlini al. 2020), as these attacks either require gradient access (which also requires parameter access) in the setting on gradient leakage, specific query data points in the case of membership inference, generative model and query points for generative model attacks. Our attack requires priori knowledge of the dataset. With this context in mind, the attack desribed in this paper contributes to the literature parameter-only based attacks.
Consider a neural network trained under MSE loss, L E (v: f for Fi, Vi Xr. Mr, being the training set datapoints and labels. Now further assume that the network is trained under gradient flow and that the network is approximately in the lazy/NTK regime, that is, behaves like a first-order Taylor expansion of the network outputs (Chizat et al. 2019):
fz R (z) = (x) + (0 8o) oe (x)
Lee et al. 2019) shows that the time evolution of the network parameters in this regime is given by:
(I ( (o-o-) (Joo (Xr) - -yr)
o(t) 0o - V Vxfn,
With 7) the learning rate and K, the fnite-wwittheepprriall NTK evaluated at 6g. Namely, the final change in parameters is given by:
Al 0 - 0o Vofo, K (ur So, (Xx))
(3)
3

---

Published as a conference paper at ICLR 2024
MNIST
CIFAR-10 Vehicle
8
6
2
4
6
/
9
8
6
5
Figure 1: Reconstructed images (top) closest training images (bottom) for MNIST Odd vs. Even, and CIFAR-10 Animal vs. Vehicle Classification. Reconstructions were made from 4096-width two hidden layer fully-connected networks trained with standard dynamics and low learning rates. Apart from small amounts of noise, the original images and their reconstructions are visually indis- tinguishable.
MNIST Ood Even Training
CIFABI1D Animal vs. Velic Training setabe
12 23
Image INDe
Image F100x
Figure 2: Reconstruction quality curves for the MNIST Odd vs. Even and for CIFAR-10 Animal vs. Vehicle classification. We either used standard dynamics (top) or linearized dynamics (bottom) and varied both the size of the training set and model width. Smaller datasets are easier to reconstruct while wider models can reconstruct more images, with linearization helping in both scenarios.
Notably, this is the solution to the following optimization problem:
arg min A6
2 |
s.t.
(X)o = yrr foo (Xx)
(4)
The corresponding KKT conditions are:
A =0 a (Xx)
(5)
A Vofo, (Xr) yr fo, (Xr)
With being the of dual parameters. In formulation, eq. ensures point, while eq. (6) ensures that the labels are correct. Like with (Haim et al. 2022), we can directly optimize the reconstruction images and dual parameters (. and a, respectively) to match these KKT conditions, given a network's 's final parameters and initialization to get A0. In practice, we only need to optimize eq. (5). for reasons we will describe next, leading to our reconstruction loss:
stationary
RRecomstruction | fo, (X₂ 112
Reconstruction in Infinite Width. Next, show that this formulation of the attack recovers the entire training set for width models. We further assume that the training data lies on the unit hypersphere.
Theorem 1. (Informal) If Creconstruction 0 (from Eq. 7) then we reconstruct the entire training set in the -ifthite-width limit, assuming that training data lies on the unit hypersphere.
The proof works by considering the limiting case of the reconstruction loss in infinite width. By taking limits, we can convert the reconstruction loss into the Maximum Mean Discrepancy (MMD) (Gretton et al.) 2012) comparing two measures: one parameterized by our training images, and the other by our reconstruction images, where the MMD is taken w.r. the Neural Tangent Kernel (NTK). Using results from (Jacot al., 2018 Lee et al.) 2019). know that these kernels con- verge in probability to the fixed infinite-width NTK, which is universal over datapoints on the unit hypersphere. This universality implies that Erecomstructionn MMD 0 iff Xn, Xr, meaning recover the original training set. We refer the reader to appendix for details and formal statement of theorem[
Note that in practice we do not enforce the unit sphere requirement on the data, and we still see high reconstruction quality, which show in section 4 This mapping from the network tangent space
4

---

Published as a conference paper at ICLR 2024
Table 1: Distillation test accuracy of KIP, Recon-KIP (RKIP), RKIP from trained network (RKIP- finite), on distilling 500 images down to 20 images. KIP and RKIP provide the best infinite-width performance, while KIP fails for finite models. (n=7)
Disrillation Algorithm
MNIST OddEEvc
CIFAR-10 10 Animal Vehicle
Standard
Lineurized
Width
Standard
Linearized
Infinite
Full dataset (500 images)
92.85 = 0.42
02.91 1: 0.33
90.18 a: 0.37
75.06
16.06:100 0.21
74.60 :: 0.21
75.42 0.28
57.42
8.41
RKIP
89.61 + 1.18
89.99
1.11
91.44
0.48
72.23
+: 3.61
72.76
3.74
74.66 0.93
RKIP-finite
88.45 = 0.89
86.15
+ 3.39
87.31
3.24
71.96
1.14
63.99 4.02
62. 02.00-22 100:0:000 4.17
Rundom images
73.52 3.60
73.54
3.81
74.12
3.73
70.36
70.18
2.54
70.77 2.04
image space also sheds light on the success of gradient leakage attacks (Zhu et al.) 2019). in which gradients are used to find training batch examples.
4 DATASET RECONSTRUCTION FOR FINITE NETWORKS
While the attack outlined in Theorem carries fundamental theoretical insights in the infinite-width limit, it has limited practicality as requires access to the training images themselves to compute the kernel inner products. How does the attack work for finite-width neural networks, and under what circumstances this attack successful?
vs. Even
To answer these questions, we follow the experimental protocol of [Haim et al. (2022), where we try to recover images from the MNIST and CIFAR- 10 datasets on the task of odd/even digit or animal/vehicle classification for MNIST and 10, respec- tively. We vary the size of the train- ing set from 10 images per class to 250 images per class (500 total train- ing set size). We consider two hid- den layer neural networks with bi- ases using standard initialization (as opposed to NTK parameterization or the initialization scheme proposed in Haim et al. (2022)). We vary the width of the neural networks between 256 and 4096 to see how deviations from the infinite-width regime affect the reconstruction quality. Furthermore, it is known that for finite-width networks the finite-width NTK varies over the course of training, deviating from the infinite-wide regime. We can force the kernel to be frozen by considering linearized training, where we train first-order Taylor expansion of the network parameters around its initialization (see eq. (2). We consider both networks under standard (unmodified) dynamics and linearized dynam- ics. In appendix I.I and appendix L.2 we consider convolutional architectures and high-resolution datasets, respectively, but we restrict our attention to MLPs on lower-resolution images in the main text.
Movi www.nnmeeeeeeooo Trmee
Kerne statec betiweed In and NTKs
Kernel snce bellowen Tinal NTTS
Figure 3: Mean reconstruction error vs. the kernel dis- tance from the initialization the final kernel. The mean reconstruction error, measured as the average value of the reconstruction curve, is strongly correlated with how much the finite-width NTK evolves over training. Dataset size is given by dot size, while the color indicates model width (see fig. [)
We train these networks for 106 iteration using full-batch gradient descent with a low learning rate, make M 2NV reconstructions with N being the training set size. A full description of our experimental parameters is available in appendix Jad algorithmic runtime details in appendix D To measure reconstruction quality we consider the following metric. We first measure the squared 1.2 distance in pixel space from each training image to each reconstruction. We select the pair of training images and reconstruction which has the lowest distance and remove from the pool, considering it pair of image/reconstruutton. We repeat this process until we have full set of N training images and reconstructions (See Fig. [].. We then order the Le distances into an ascending list of distances and plot this function. We call this the reconstruction curve associated with a particular reconstruction set. We plot these reconstruction curves for varying dataset sizes and model widths in fig. 2 From fig. [ we have the following three observations:
and during the reconstruction,
Smaller training sets are easier to reconstruct. We see that the reconstruction curve for smaller datasets has low values for all model widths. Wider models can resolve larger datasets. We
5

---

Published as a conference paper at ICLR 2024
MNIST Classes (Standard Dynamics)
CIFAR-10 (Standard Dynamics)
Training set stbe
L2 Diabance
12
20
109
200
500
60
L2
50
100
170
2.31
200
226
10
191
393
216
200 491
Image Index
Image
Figure 5: Reconstruction curves for networks trained on multiclass MNIST/CIFAR-ID
observe that for a given model width, there is a threshold image index at which the quality of recon- structions severely decreases. For example, for MNIST Odd/Even, 200 images and width of 1024, this is 80 images. As we increase the model width this threshold increases almost monotonically.
Linearization improves recon- struction quality. We see that linearized networks can resolve more images and have better images compared to their same-width coun- terparts. The success of linearization suggests that deviations from the frozen NTK regime affect recon- struction quality. We can measure the deviation from the frozen kemee regime by measuring the kernel dis- tance of the network's initialization NTK and its final NTK, given by the following: d(Ko, Ky) = =11 ||oll [Kyle""
Caltech 1/cls
CIRAR-100 2/cls
CIFAR-20 Sids
Train images hocoria vadions
Figure 4: Reconstructions of training data for few-shot fine tuning ResNet- 18 pretrained on ImageNet on Caltech Birds (1/cls), CIFAR-100 (2/cls) and CIFAR-10 (5/cls).
Tr(K] K)
Intuitively, this distance tells how well the initialization and final kernel align. Large values indicate that the kernel has changed substantially, meaning the network is deviating far from the NTK regime. We plot these kernel distances against the mean value of the reconstruction curves in figure fig. 3 We see immediately that reconstruction quality is strongly correlated with kernel distance, and that smaller datasets and wider models have lower kernel distance. In appendix [ we discuss how our attack is compatible with early stopping and cross-entropy loss, unlike (Haim) al. 2022). which requires training until convergence. more detailed discussion of the effect of early stopping is available in appendix [
Multiclass Classification. In previous sections, we showed the validity of the attack on binary clas- sification. Here we verify that the attack works with multiple classes. We repeat the same procedure as in section 4 but will all 10 classes. Details are given in appendix D as well as additional results on 200-way classification on curves in fig. We observe that this attack has improved reconstruction quality with more classes In appendix L.3 we observe that multi-class classification leads to lower kernel distances, suggest- ing it behaves more in the kernel regimes, explaining the better reconstruction quality. Future work could investigate this further.
in appendix L.2 Results are shown by reconstruction
5 DATASET RECONSTRUCTION IN FINE TUNING
A key requirement of the attack is the model initialization. When training from scratch, attackers will not have access to this, making the attack useless. However, practitioners often do not train from scratch, but rather fine-tune large publicly available pre-trained models. Furthermore, users often do not have access to large amounts of data, effectively making the task few-shot. With evidence suggesting that training neural networks later during fine-tuning well approximated by the frozen finite-NTK theory (Zancato et al. 2020) Shon et al.) 2022; Zhhou et al., 2021: Malladi et al.) 2023), this makes the few-shot fine-tuning setting an easy target for this attack. To evaluate our attack in this setting, we fine-tuned publically available ResNet-18s pretrained on ImageNet on few-shot image classification on Caltech Birds (200 classes) (Welinder et al. 2010). CIFAR- 100, CIFAR-10 10 with 1, 2, and 5 samples per class, respectively. For dg. see the best reconstructions in fig. 4 We see that our attack is able to recover some training images but with limited quality. Future work could look improving these attacks.
the initial fine-tuned model parameters. We

---

Published as a conference paper at ICLR 2024
MNIST Odd vs. Even 100 weat
CIFAR-10 Animal vs. Vehicle
20 p--0.5
200 SAEE
500 NUR
20 w--21
100
200
500
Sea
12
13
10
18
14
[0]
Figure 7: Scatter plots of reconstruction quality measured in 12 distance and corresponding |o| values for images. We vary the width with n that we observe range of reconstruction qualities. |o| is negatively correlated with reconstruction error.
6 WHAT DATAPOINTS ARE SUSCEPTIBLE TO RECONSTRUCTION?
It has been observed in previous work that no datapoints are equally susceptible to privacy attacks (Carlini et al. 2022 Feldman & Zhang 2020a Carlini al. 2021 Bagdasaryan & Shmatikov 2019). In particular, outlier images tend to be leaked more easily than others. In this section, we show that this occurs for our attack, and provide theoretical justification for this.
6.1 Hard to fit Implies Easy to Reconstruct
CIFAR-10 10 Accuracy
Reconsitution Curve
By considering our reconstruction loss
A
E, E,a)0(z;) with o(x;) = fo, (1z) aim to learn basis to "explain" A0, we see that this could be cast as a sparse coding prob- lem. Assuming that all Φ(zz) are of roughly the same magnitude, we expect the the param- eters which larger parameters to contribute more to AO, and thus be more easily recon- structed. This is closely related to how data points with high influence are likely to be mem- orized (Feldman & Zhang 2020a). We ver- ify this heuristic holds empinialll by plotting scatter plot of reconstruction error vs. the corre- sponding la| values calculated for infinite width fig. 7 We see that images with small a val- ues are "protected" from reconstruction since their contribution to AO is small and could be written off as noise. From appendix
Puze Casleri facemuruciora
Ditigoess Namrvvel
Image
Figure 6: Test accuracy of iteratively pruned CIFAR-10 10 using either random pruning or prun- ing based on easily reconstructed images (Left), and reconstruction curves for pruned CIFAR-10 10 (Right). We see that easily reconstructed data- points can be removed without harming accuracy. We observe a privacy "onion" effect where remov- ing easily reconstructed images reveals other im- ages which are easy to reconstruct. Bolded lines indicate the 20 images removed after pruning.
E we know that a = corresponds to an image/label which does not alter the prediction at all, so this suggests that well-predicted datapoints are safe from reconstruc- tion. Aside from the definition of a = K-1 y. we can alternatively show that a is closely related to how quickly the model fits that datapoint. We can write (see appendix [E. for a derivation) that og = Jo (リj Sa, )dl implying that datapoints which are slow to fit will have large values, further strengthening the claim that outliers are easier reconstruct.
6.2 A Reconstruction Privacy Onion
Carlini al. (2022) showed that removing "vulnerable" training points reveals another set of training points which are susceptible to inference attacks. Here, we verify that our reconstruction attacks sees similar phenomenon. Specifically, we consider reconstructing CIFAR-10 training images after training on datapoints, with n 900 initially. We train and attack networks with w x vT, and then iterative remove the 20 most easily reconstructed datapoints based on the reconstrution curve. We scale the network capacity with so that our attack is unable to reconstruct the entire training set. We see that in fig. [6] that despite our network and attack capacity decreasing as we remove more datapoints, we are still able to reconstruct data points with increasing attack quality, replicating the "privacy onion" " effect. Future work could look at how the interaction of a parameters affects which items are susceptible to reconstruction post-datapoint removal. Likewise, we evaluate the test accuracy on these pruned subsets in fig. 6 We see that as, these easily reconstructed datapoints tend to be outliers, removing them has reduced effect the test accuracy, however Sorscher et al. (2022) discusses. the decision to remove easy vs. hard datapoints during pruning is dependent on other factors such the size of the dataset and the complexity of the task.
7

---

Published as a conference paper at ICLR 2024
7 UNIFYING RECONSTRUCTION AND DISTILLATION
In the previous sections, we considered the task of reconstructing the entire training set. To do this, set the reconstruction image count M N. What happens if we M N?? Do we reconstruct a subset of a few training images, or do we form images that are averages of the training set?
We perform this experiment on the CIFAR-10 Animal/Vehicle task with 500 training images for a 4096-width model with linearized dynam- ics, aiming to reconstruct only 20 images. We recover the images shown in appendix G With few exceptions, these images are not items of the training set, but rather these images look like averages of classes.
MNIST Odd vs. Even
CIFAR-10 Animal vs. Vehicle
78-
Accuracy
60-
98
92
decand Imagesl
RGF
62-
ngrimtte
Now, what if we retrain a network on these reconstructed images? Noting that the opti- mal set of dual parameters for a full recon- struction is given by a KyIør, natural choice for the training labels for these images yR KRRR", where we compute the em- pirical NTK for KRR and use the learned a parameters found during reconstruction. Retraining a different network from scratch on these 20 re- covered images yields high accuracy, as shown in table I as RKIP- finite.. This suggests that by doing this reconstruction we performed dataset distillation, that we constructed smaller set of images that accurately approximates the full dataset. This is not a coincidence, and the two algorithms are in fact the same. More formally:
61
Full training Figure 8: Performance of KIP, RKIP, and RKIP- finite distill N images down to 20 images, trained on a 4096 width network with standard dynamics. KIP fails to transfer to finite networks while RKIP variations succeed.
Theorem 2. The infinitewwddth variant reconstruction scheme of Eq. with KKT points of Eq. 5 and Eq. O with M S N where M is the reconstruction image counts and N is the dataset size, can be written as a kernel inducing point distillation loss under a different norm plus a non-negative correction term X follows:
L Reconstruution 11r KTn nnyn K71
+ Avar of R/T
The full proof is given in appendix F Avar R/T proportional the variance of the reconstruction data points conditioned on the trainnn data, based on the NTK (see appendix F. Intuitively, it ensures that training images provide "information" about the reconstructions. Compared to the loss of well-known dataset-distillatio algorithm, KIP (with S referring to the distilled dataset):
LKIP
|ur
Krs K Kgsus 2
The connection is apparent: the reconstruction loss is equal to the KIP dataset distillation loss under a different norm, where, rather than weighting each datapoint equally, we weight training images by their inverse similarity measured by the NTK, plus A r/Tr This leads to variant of KIP which we call Recon-KIn KIP (RKIP) which uses the reconstruction loss in theorem 2 Note that for large datasets, RKIP is not practically feasible since it requires computing K K We deal with small datasets in this work so still tractable.
KTT. which is typical N x N.
We summarize the performance on KIP and RKIP in table on the MNIST Odd/Even and CIFAR- 10 Animal/Vehicle task, distilling 500 images down to 20. We evaluate 4096-width networks with standard or linearized dynamics, and infinite width using the NTK. Additionally, we consider using the images/labels made from reconstructing dataset points using a finite network trained on the full dataset and call this RKIP-finite. Note in this case the labels
not necessarily (+1,-11, as Ko, RR R are not guaranteed to be one-hot labels. Similar results for distilling fewer images (20 500 training images) to 20 distilled images shown in figure fig. g
We observe in table D that both KIP and RKIP have high infinite- width accuracies, but KIP sees a significant performance drop when transferring to finite networks. For example, while KIP achieves
8

---

Published as a conference paper at ICLR 2024
MNIST Ode vs. Evan Ditilied Dalasets
CIRAR-LO Arimal vs. Vahicle Distilted Datasets
f
2
4
2
2
0
5
4
8
q
5
5
Figure 9: Visualizations of distilled datasets, on MNIST Odd/Even and CIFAR-10 Animal/Vehicle classification made with KIP, and RKIP. We distill datasets 500 original images 20 images (shown). KIP does not copy the original training images, while RKIP occasionally reproduces training images.
91.53% infinite-width test accuracy on the MNIST odd/even task, its finite-width performance is 55.62%, not significantly better than random guess. Interestingly. this performance gap increases as we distill more images, as seen in fig. 8 For small distilled datasets there is little to no per- formance drop but for larger ones the difference is significant. In contrast, RKIP surprisingly sees almost performance drop in the finite-width settings. We hypothesize that this finite-width trans- fer performance difference for KIP and not RKIP could be due to the contribution of Avar of r/T, which we discuss in appendix [F) We leave it to future work to explore this further. Additionally, RKIP-finite performs nearly as well as RKIP, despite distilling using the information from a single finite-width neural network.
Discussion, Limitations, and Conclusion
In this work showed that stronger variant of the attack given in Haim et al. 2022 which quires wide neural networks trained under MSE loss can provably reconstruct the entire training set, owing the injectivity of the NTK kernel measure embedding. We showed that this attack works in practice for finite-width networks, with deviations from the infinite-width regime weakening the attack. We looked how outlier datapoints are more likely be reconstructed under our attack, and that these casily reconstructed images can be detrimental to learning. Finally, we made a novel con- nection between this reconstruction attack and dataset distillation. While this sheds light on dataset reconstruction attacks, and their theoretical underpinnings, there are still many avenues to explore.
In this work, we primarily explored 2-layer fully connected networks, where neural networks are known to behave similarly to their infinite-width width counterparts. Meanwhile, deeper convolutional networks are known to deviate significantly, and it is unclear how well attacks in this paper would transfer to those settings, and what adjustments would need to be made. Secondly, while we observed increasing model width increases the network's "resolving capacity" (i.e. how many images it could reconstruct), future work could look at how this quantity arises from deviations in the finite-width NTK from the infinite width one. Finally, we still need to resolve how the dataset reconstruction notion of privacy connects with more established notions such differential privacy, which is the subject of future work.
We believe this work provides important step toward understanding the strengths and weaknesses of dataset reconstruction attacks and provide novel connections to existing literature such as the Neural Tangent Kernel and dataset distillation.
REPRODUCBILITY STATEMENT
This work uses open source datasets and models. Experimental details such as hyperparameters deseribed in appendix J We additionally provide code for running the experiments the supple- mentary material.
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference Computer and Communications Security, CCS 16, pp. 308-318 New

---

Published as a conference paper at ICLR 2024
York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450341394, doi: 10.114522766749.9988888 |ttpp///s/////ooooo////..//////3 29978118
Kyle Aitken and Guy Gur-Ari. On the asymptotics wide networks with polynomial activations. ArXiv, fbs/2006.066877, 2020.
Sanjeev Arora, Simon S Du, Wei Hu, Zhiyuan Li, Russ R Salakhutdinov, and Ruosong Wang- On exact computation with an infinitely wide neural net. In Advances in Neural Information Processing Systems, pp. 8141-8150. Curran Associates, Inc., 2019.
Devansh Arpit, Stanislaw Jastrzebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, et al. closer look memorization in deep networks. In International conference on machine learning, pp. 233-242. PMLR, 2017.
Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai. Aidan Clark, Ivo Danihelka, Claudio Fantacci. Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, John Quan, George Papamakarios, Roman Ring. Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Luyu Wang, Wojciech Stokowiec, and Fabio Viola. The DeepMind JAX Ecosystem, 2020. URL http://github wtt deepmind
Eugene Bagdasaryan and Vitaly Shmatikov. Differential privacy has disparate impact on model accuracy. CoRR, abs/1905.12101, 2019.
Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. CoRR, abs/1912.038177, 2019. URL http::://axxiv..roope http://aa///111.333311
1912.038111
James Bradbury, Roy Frostig. Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao
Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http:
//github. com./b.com/ee http///oooleeaaa.a
Gavin Brown, Mark Bun, Vitaly Feldman, Adam Smith, and Kunal Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pp. 123-132, 2021.
Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In Proceedings of the 28th USENIX Conference on Security Symposium, SEC' 19, pp. 267-284, USA, 2019. USENIX Asso- ciation. ISBN 9781939133069.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts. Tom B. Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raf- fel. Extracting training data from large language models. CoRR, abs/2012.07805, 2020. URL
https://arxiv o abs 2012.07805
Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas Terzis, and Florian Tramèr. Mem- bership inference attacks from first principles. CORR, abs/2112.03577 2021. URL https:
7/a 1/arxiv verivi.rrggg www./rrggboo...... abs 22122112111. 03570
Nicholas Carlini, Matthew Jagielski, Chiyuan Zhang, Nicolas Papernot, Andreas Terzis, and Florian Tramer. The privacy onion effect: Memorization relative. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems, volume 35, pp. 13263-13276. Curran Associates, Inc.,
2022. URL https::/proceeeeee wwwp::eww..nnnn...n.ne.... cc/ paper lacpapeeraeeaaaaaaaaaaaeeeeeeeee files//appr/2022/ wow.ppeer--ooffeence....
File/ 31
Centers for Medicare Medicaid Services. The Health Insurance Portability and Accountability Act of 1996 (HIPAA). Online at http::/wwwwws.......iiii...ooe.e 1996.
10

---

Published as a conference paper at ICLR 2024
Lénaic Chizat, Edouard Oyallon, and Francis R. Bach. On lazy training in differentiable program- ming. In NeurlPS, 2019.
Rodrigo de Azevedo
wwgprkcwwaagg..ouuueese/ggggrroo azevedo). Does
gradient descent converge to minimum-norm solution in least-squares problems? Math- ematics Stack Exchange. URL
com /
(version: 2022-02-18).
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Proceedings of the Third Conference on Theory of Cryptography, TCC pp. 265 2844, Berlin, Heidelberg, 2006. Springer- Verlag. ISBN 3540327312. doi: 10. 1007/11681877 14. htthttss::
wttp:////.............. 1000010111110 681668888 00000/116688881111
European Commission. Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection natural persons with regard to the processing of personal data and the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulatiin) (Text with EEA relevance). 2016. URL hnttps:t:://et eu/eli/req 201 6 679/oj
lex. europa.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, STOC 2020, pp. 954-959, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450369777 doi: 00.1145/3357713.3382222
htttht:t:/eaat /ttp::///...oog/1..... 1145
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation.
CoRR, abs/2008.03703, 2020a. URL https::/arxiv.
org/ aa///ab/aaa 2008. 03703
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, 20, Red Hook, NY, USA, 2020b. Curran Associates Inc. ISBN 9781713829546.
Stanislav Fort, Gintare Karolina Dziugaite, Mansheej Paul, Sepideh Kharaghani, Daniel M. Roy, and Surya Ganguli. Deep learning versus kernel learning: empirical study of loss landscape geometry and the time evolution of the neural tangent kernel. In NeurlPS, 2020. URL /ttps: neurips httplpps...ppapppeeeeepp wwtppppppe//////aaaaaseee .html
Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confi- dence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC confer- ence computer and communications security, pp. 1322-1333, 2015.
Arthur Gretton, Karsten M. Borgwardt, Malte J. Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. J. Mach. Learn. Res., 12mul1):733-77, mar 2012. ISSN 1532-4435.
Niv Haim, Gal Vardi, Gilad Ychudai, michal Irani, and Ohad Shamir. Reconstructing train- ing data from trained neural networks. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL https: http:::///penrevieww net feet/forumiide wtw.emiddd-xxxeeeeeeeee
Boris Hanin and Mihai Nica. Finite depth and width corrections to the neural tangent kernel. In International Conference on Learning Representations, 2020. WtLLhtttts////peeeeeet
net/
Ali Hatamizadeh, Hongxu Yin, Holger Roth, Wenqi Li, Jan Kautz, Daguang Xu, and Pavlo Molchanov. Gradvit: Gradient inversion of vision transformers, 2022.
Zecheng He, Tianwei Zhang, and Ruby B. Lee. Model inversion attacks against collaborative in- ference. In Proceedings of the 35th Annual Computer Security Applications Conference, ACSAC 119 pp. 148-162 New York, NY, USA. 2019. Association for Computing Machinery. ISBN 978 778145037680.. doi: 10.1145/3359799.333922.. URL htt.::::/ddo.orgg/// 01537899335595

---

Published as a conference paper at ICLR 2024
Jonathan Heek, Anselm Levskaya, Avital Oliver, Marvin Ritter, Bertrand Rondepierre, Andreas Steiner, and Mare van Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL http://github. wtt. google/flax
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- DesaaBianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. UtLp paper/2018/ httpe///c...eeeee222222222222613 Paperr. pdf
nwwripuuri.... cc/
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17176-17186, Curran Asso- nww.ieerrpp pctpp//ppp//22///i1eeel
ciates, Inc., 2020. URL c 7eeeb2/assBB5011111cc00ooooccce Papeoo pdf
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Reprexentations, ICLR 2015, San Diego CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL.htpp: //arxiv. org/ abs/ 1412. 6980
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference Machine Learning Volume 70, ICML'17, pp. 1885-1894. JoLR.org, 2017.
Ya Le and Xuan S. Yang. Tiny imagenet visual recognition challenge. 2015. URL https: //api semant cacholar wwwachlaa..rgg/ooppuiuoo wrg/Corppsst.. 1666647799 6664790
Jachoon Lee, Lechao Xiao, Samuel Schoenholz, Yasaman Bahri, Roman Novak, Jascha Sohl- Dickstein, and Jeffrey Pennington. Wide neural networks of any depth evolve linear models under gradient descent. Advances in neural information processing systems, 32, 2019.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Efficient dataset distillation using random feature approximation. Advances in Neural Information Processing Systems, 2022a.
Noel Loo, Ramin Hasani, Alexander Amini, and Daniela Rus. Evolution of neural tangent kernels under benign and adversarial training. In Advances in Neural Information Processing Systems, 2022b.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural net- works. In International Conference Learning Representations, 2020. URL https://
openreview net wwt.://orommldddee...g....
Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjeev Arora. A kernel-based view of language model fine-tuning, 2023. URL htt.hhttt///opeeevviiiiee net forum?id-
erHai09gz3m
Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learnngg from kemel ridge- regression. In International Conference on Learning Representations, 2021a. URL https:
Timothy Nguyen, Roman Novak. Lechao Xiao, and Jachoon Lee. Dataset distillation with infinitely wide convolutional networks. In Fhitty-Fifth Conference on Neural Information Processing Sys- tems, 2021b.
Roman Novak, Lechao Xiao, Jiri Hron, Jachoon Lee, Alexander A. Alemi, Jascha and Samuel S. Schoenholz. Neural tangents: Fast and easy infinite neural networks in python. In International Conference on Learning Representations, 2020. URL https: /ttpp//pppp.....c/////// het:/Forum2io=skk1yyyyr
12

---

Published as a conference paper at ICLR 2024
Roman Novak, Jascha Sohl-Dickstein, and Samuel S. Schoenholz. Fast finite width neural tan- gent kernel. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference Machine Learning, ICML 2022, 17- 23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 17018-17044. PMLR, 2022. URL httph:ttp//.......... htttps::/proceediinng ml press/ 21162/ovva22a..o html
Maria Rigaki and Sebastian Garcia. A survey of privacy attacks in machine learning. CoRR, abs/2007.0766 07666, 2020. URL//ttps:://axxxvvrrr/////
07646
Reza Shokri, Marco Stronati, and Vitaly Shmatikov. Membership inference attacks against machine learning models. CoRR, abs/1610.05820, 2016. URL http:::/aaa....orrg/a//////////
05820
Hyounguk Shon, Janghycon Lee, Seung Hwan Kim, and Junmo Kim. DLCFT: deep linear continual fine-tuning for general incremental learning. CoRR, abs/2208 01122000001122 12, 2022. doi: 10.48550/arXiv. 2208.08112 12. URL
htt.:/t....g..gg... 48550/ 2208. 08112
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. ArXiv, abs/2206. 14486, 2022. URL.
wew.mnnanctcccccaaa org/ :ttp:/oooppus:::::: :
Bharath K. Sriperumbudur, Kenji Fukumizu, and Gert R.G. Lanekriet. Universality, characteristic kernels and rkhs embedding of measures. Journal of Machine Learning Research, 12(70):2389- 2410, 2011. URLhtttp::// //mll wwwmmlooorrgpppppaeeeeee vllp/vl1///////// sriiriii wew.rrrprumsmoee rubbudrr
html
Nikolaos Tsilivis and Julia Kempe. What can the neural tangent kernel tell us about adversarial robustness? In Alice H. Oh. Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL hittps::/openneeeeee
Tongzhou Wang, Jun- Yan Zhu Antonio Torralba, and Alexei Efros. Dataset distillation. arXiv preprint arXiv: 10959, 2018.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, California Institute of Technology, 2010.
Matthias Wright. Flax models. 2022.
Ziqi Yang. Jiyi Zhang, Ec-Chien Chang, and Zhenkai Liang. Neural network inversion in ad- versarial setting via background knowledge alignment. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, CCS '19, pp. 225-240, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450367479. 10.1145/3315555.354444
doi:
URL httpt:ttt//////t org/10 http:///d...rrr....113133333333/ 911.///3...1133333333333333 35
Luca Zancato, Alessandro Achille, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Pre- dicting training time without training. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6136-6146. Curran Associates, Inc., 2020. URL httpss://prrceeddnges neuri cc/paper. Passss-aper.... pdf
Ei
Amir Zandieh, Insu Han, Haim Avron, Neta Shoham, Chaewon Kim, and Jinwoo Shin. Scaling neu- ral tangent kemnels via sketching and random features. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), ), Advances in Neural Information Processing Systems, 2021. URL https://gpenneviee net /twt
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017. URL
net / Cererrumiicii Sae.ss-SyyyyyBxx
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 645 feb 2021. ISSN 0001-0782. doi: 10. 145/3446776. URI. hnttps://doi.o/g/// 0tt.//1111115/3447777
13

---

Published as a conference paper at ICLR 2024
Bo Zhao and Hakan Bilen. Dataset condensation with differentiable siamese augmentation. arXiv preprint arXiv:2102.08229, 2021.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference Learning Representations, 2021. URL URLihttps://opnreewiee
met/fore feru/foomm23 20-
Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature re- gression. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yufan Zhou, Zhenyi Wang, Jiayi Xian, Changyou Chen, and Jinhui Xu. Meta-learning with neural tangent kernels. CoRR, abs/2102.03909, 2021. URL httpp://axxx...rrrbbbb///22
03909
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Annual Conference on Neural Information Processing Systems (NeurIPS), 2019.
14

---

Published as a conference paper at ICLR 2024
APPENDIX
A COMPARISON TO HAIM ET AL..
(2022)
In section we mentioned that we had problems reproducing Haim et al. (2022) 's attack. Here, we compare the two attacks and discuss the issues we found with theirs.
Firstly, we compare the quality of the two attacks. Haim et al. (2022) open sourced their code, as well as gave the best two sets of reconstructions for both CIFAR-10 10 and MNIST-10. We plot the reconstruction curves of these reconstructions here:
MNIST Odd vs. Even Training set size 500
CIFAR-10 Animal vs. Vehicle
Training set size
500
L2 Distance
12.5
L2 Distance
Reconstruction
Haim al.
7.5
15
Halm et al.
Ours,
5.0
2.5
0.0
100
200
300
400
500
100
200
300
400
500
Image Index
Image Index
Figure 10: Reconstruction curves for the attacks given in Haim et al. (2022). comparison to our reconstruction attacks, with comparable width of 1024.
In their paper, they consider networks of width 1000, so for fair comparison, highlight reconstruction with a comparable width of 1024, under standard dynamics. We see that for MNIST, our attack has significantly better quality until image index 50 in which case both attacks perform poorly. For CIFAR-I0, our attack performs worse but can achieve better performance with wider width or linearization.
Note that the two reconstruction curves presented from Haim et al. (2022) correspond to the two best reconstructions with carefully chosen hyperparameters. These hyperparameters are chosen to maximize reconstruction quality, which requires access to the training data to measure the quality. priori, an attacker would not be able to assess reconstruction quality, and thus would not be able to do such hyperparameter tuning. In contrast, attack parameters such as learning rate/initialization were not fine-tuned for ours, and we use the same hyperparameters for every attack. Further gains could likely be seen with more careful tuning. We would like to emphasize that the goal of this work not necessarily to create the strongest attack, but more so to explore the properties of the attack, and conditions for failure/success.
A further limitation of their attack is that it requires homogenous neural networks (i.c. no biases), in comparison to ours which uses biases, which is closer to practice. The largest limitation of their attack is that they require a careful initialization scheme, in which the weights of the first layer are initialized with significantly smaller variance. In Haim et al. (2022) they discuss that this is essential to the success of their attack. In contrast, we use the default initialization given in the Flax neural network library (Heek et al. 2020).
We also observed that the KKT condition given in eq. ① which is required for their attack to work often is not reached in practice. To reiterate the definition of directional convergence, it requires that
lim; T
for trained network parameters 0, and O' the solution to eq. (1 It is clear
that if 2213 1, i.c. the parameters have not drifted far from their initialization, then, we cannot
10.12
hope the KKT point is reached (of course, with unlikely exceptions such as 6y already being close to the KKT point). In practice, we found that ||6. ~ 333 and ||0]] x 377, when running their attack, which suggests that the initialization still contributes a significant amount to the final parameters, suggesting that the KKT was not reached. Of course, our attack is not without limitations well,
15

---

Published as a conference paper at ICLR 2024
the most notable being that we require network initialization. We leave it to future work alleviate this requirement.
B PROOF OF THEOREM
Here we give a formal statement theorem as well as its proof.
Theorem 1. Erecostraccion + 0 (from Eq. 7 as + sc then XR + XT in probability for training data and recostruction data XT and XR. respectively, on the unit hypersphere, with w the network width.
Proof. Define (=)=xx)) = Vefo(t) that is, the finite-widthlempprrcal NTK function. We know as network width to + oc, A0 En,,, "xr az Ve, (#:), with a = k TUTT with X being the training set, /r the training labels, and K0. the finite-width NTK evaluated on the training set. Our attack then becomes:
|0 Σ ag Ve, Vo, (23)
E Vev - - VayVo, (25,)
(8)
asayEa RR
aL.c XT
asajEaR
Σ assion
Σ Casko (15; (I5:
Xx
aszyeuR, XR
ackNTK (
EajkNrk(es))))
ipp.
(10)
Ciii Xx
ayazEa" Xx
With T and R referring to the training and reconstruction set, respectively. We can take limits because we have as we have that both kig ka kNTK in probability (Jacot et al. 2018 Lee et al. 2019). Furthermore, define
Pr
Σ E
ox5(zz),
PR Σ
az5(Zz)
measures associated with trained network and reconstruction, respectively, and μ. Ja kNTK (z, .dP.(() with Ω = sd, with d being the data dimension (Assuming data lies on the unit hypersphere). We can now write:
[aCKNTK(E..) - - EashNTK(Os Il?
(11)
0,, XT
aja,Ea
= -11/ (wTk(x;))dPr()) - )dP((t)
(12)
= =llor - 1nll3NNTX HNTK
(13)
MT and PR are now kernel embeddings of our trained network and reconstruction, respectively. Our reconstruction loss becomes: l| - pal 3 HvTx This is the maximum mean discrepancy (MMD) (Gretton al.) 2012). We note that Pr. PR are signed Borel measures (since a are finite and our reconstruction//raiiing sets are on the unit sphere). The NTK is universal over the unit sphere (Jacot al., 2018), implying that the map μ (Family 2011). This means that MM DRWT 0 implies Pg Pr a.c., i.c. XR XT provided ag * (see appendix
signed Borel measures] - H is injective (Sripe-
rumbudur et al.
E). Now putting everything together, we have that Creconstruction + 0, so M M DRNTK + 0 i.p. Note that the MMD and kNTK continuous in their inputs, which implies that XR + X2 as well.
C CONNECTION TO GRADIENT INVERSION ATTACKS
Gradient inversion attacks (Zhu et al. 2019 Hatamizadeh et al. 2022) are reconstruction attacks which reconstruct training images given the network parameters and gradients at specific time step for specific batch B: 0, Cn. These attacks work by minimizing difference gradient between loss reconstruction images and the known batch gradient:
16

---

Published as a conference paper at ICLR 2024
Egrad = IVVo.BB Ve.Calle
(14)
Where LR is the loss reconstruction images. Compared to the attack presented in this work, ours does not use require the gradient directly, but the total change in parameters, A6. We can consider AO as the sum of gradients over all time steps. With this in mind, gradient inversion attacks could be considered a special case of our attack where only one gradient step is taken and A0 - L for some learning rate V/+ similar theorem [ applies here as well (as we do not require convergence, see appendix B and appendix []. so we can guarantee that gradient inversion attacks can reconstruct images when 6g is near NTK initialization. As discussed in section 4 neural networks, generally speaking, do not behave in this regime, so our analysis cannot be directly used to explain the efficacy of gradient inversion attacks.
extreme version of early stopping. Our theoretical analysis in
D RECONSTRUCTION ATTACK ALGORITHM DETAILS
Algorithm Standard Reconstruction Attack
Require:
Initial Parameters 0o. final parameters fy. network function fe. randomly initialized reconstruction images and dual parameters (XR, aR), optimizer Optim (params, gradicnts, number of steps T
A = 0 0o
t
while < T do G >, 0; Vala, (I;) for az, Vi E aR, X
Compute reconstruction gradient
Crcon = | G
> Compute reconstruction loss
On, X Opt im {an, XR),
@hrecom
Update Reconstruction Images
[(aa(Ra)
t ++1+1
end while
Algorithm Batched Reconstruction Attack
Require: Initial Parameters 0u. final parameters 8,, network function fo. white noise initialized reconstruction images and dual parameters {XR, aR), optimizer Opt im (params, gradients, number of steps T,, batch size [B]
AA =0,-0 fu
GR E, a; Vefe, (I,) for as, aRR Xx. > Compute total reconstruction gradient (this step can also be batched)
while < T do
Sample batch aB, XB an, Xn of batch size B uniformly
GB a, Vefo, (I;) for a; J; E aB, XB > Compute reconstruction gradient for batch > Store old batch gradient
GBool + detach(Gg)
fream + || A0 - (Gn GBoold + GB)
>. Compute reconstruction loss
QB, XB - Optim( (aB, XB), aLrecon/ (((,, GB + a, Vele, (Ix) for a,, Iı E XB
D Optimize batch images
> Compute new batch gradient
GR + GR - G Boola + detach(Gn,, (,,
> Update total reconstruction gradient
1 1+1
end while
Here we discuss the runtime of our attack, given model with P. parameters, M reconstruction images, and T iterations. We present two versions of the attack: the standard version of the attack, given in algorithm []] and a minibatched version of the attack algorithm 2 Both versions of the attack are mathetmatically equivalent, but the batched version allows for larger reconstruction sets that may not fit into memory all at once.
For the standard version of the attack presented in algorithm I O(MPT) time is required and O( memory is required, as naively one needs to store (and backpropagate through) gradients for each of the reconstructed images. Evidently for large datasets, cannot pass the whole dataset
17

---

Published as a conference paper at ICLR 2024
through the model once and backpropagate, and need M >> N to reconstruct the full dataset, this seems problematic. For datasets in our paper, this was not a concer, but for larger datasets would be.
To mitigate this issue, we present a minibatched version of the attack in algorithm 2 which requires O(BPT) time and ((BP) memory. The premise of this version that you store the buffered value of the total sum of gradients Gn = Σ, 9i over reconstruction examples, and by carefully using autodiff, you can update only subset of the gradients (using the buffered value of Gn and subtracting the batch gradients). In algorithm [ the detach function refers to the autodiff graph detachment function common in most autodiff libraries. This method has been implemented and performs exactly the same as the original attack, however the results this paper do not require we dealt with small N.
E LABEL CONDITIONS FOR FULL RECOVERY
As discussed in section require that 0; * in order to recover the training image. Here discuss the conditions for this to occur. We know that a K-- y, and without loss of generality, consider reconstructing the final image index N. Our equation for a becomes, focusing the value of o::
[1
K. 11-----1 KJ N-N-
KNN-1.1
[iN-1
ox
KN.N
[1
K WK---110 1aN-1 KWW-1..
-KN-LaN 1 [iN--1] N-1---11
ox
-Q -NN 1N
With Q =KKNN KNN-INK---... KT N -1. Setting o.
- QK L K.N. + QUN
ON KT K NN
corresponds to the kernel regression prediction of yı given U-N-1. we see that d' = 0 iff y is already perfectly predicted by the remaining data points. Of course, the label corresponding exactly the prediction occurs with probability 0.
Noting K, K.v N. -1 1
18

---

Published as a conference paper at ICLR 2024
E. ALTERNATIVE INTERPRETATION OF
In section 6 we discussed how parameters can be treated as items which are "hard" vs "easy" to fit. Here we derive how jj = Jo (yj Jo, (rz))dt for MSE loss (up to some scaling parameter).
1 Σ y4 fo. (x.))²
((0,) 2
00,
ac(())
Ot 00,
00,
(z.)
7 Σ (yr - fo. (z:))
0t
00,
00: dt
AO
(xa) dt 00,
Σ(u: (y: (xx))
1o (u: So. (x:))
0Se. (11,) 00.
1 Σ
dt
1/=00 (wi - fo. (x:))) dt
2 f (31.)
11 Σ
(Frozen Kernel approximation)
80p
ate (e;) (eq. 5)) and matching terms, also noting that P > T (more Noting that A0 = E, a 80, parameters than training points), we have the system uniquely solved when a, 77 Jo (y} fe, (xt)).t. We can verify this experimentally, by plotting the calculated values of a, vs. So (y) fo, )dt in fig. 11
MNIST Odd vs. Even (w 4096)
20
100
200
500
0.2
0.10
Jty- Rox)
0.05
0.1
0.00
-0.01
-0.5
-02
-0.10
-0.2
-044
-20
a
CIFAR-10 Animal vs. Vehicle (w 4096)
20
100
200
500
0.2
0.10
[(y- - Rxx))dt
0.01
0.05
0.00
0.00
0.0
0.0
-0.01
0,2
-0.05
-0.02
-0.2
-0.4
0.10
-0.03
50
-50
Figure 1I: Infinite width values of a vs. error integral formulation of error calculated from 4096- width finite networks. There is strong correlation between the two values.
19

---

Published as a conference paper at ICLR 2024
F PROOF OF THEOREM 2: RKIP DERIVATION FROM RECONSTRUCTION LOSS
Proof. Here we derive how gets from the reconstruction loss given in eq. theorem 2 the RKIP loss. We repeat the losses here:
Reconsttration |100 (oJo,((T)) Avar of R/T
CRKIP 11T KTRK K K7T
First we note that a² = KTTOT, and if we trained on reconstructions with labels yR. then a K RRURR. For brevity we denote Sf = {af,2, 2;} and S = [x*, Let S and S denote the training and reconstruction set, respectively
AO
Σ
ag
Vo, Joy [1
ESS
a Vo f(x
Σ agj
V (1))]]
ESS
Ef
|
of ka (x
Σ
az (xf
ES
Again take the infinite width limit so ka, ko, KNTK, we which just write & for simplicity.
Eathon tog .-) Eaftn, a 112
[[ Lafueo PeRR
of es"
ees
E E oF TesT JEST
E E ofofhoT Test.flest
E
R ResR
=a? KTTo" -20 Krga" R BT RRga
[T,GT k 20 Kya Kawn vl k Kank KnnR
v/Kgter 20 K Ky nk RAvn u KRyn
u/,cttur 28 k K WRvR vl w K K K KYaK K nh
" Kar Krnk RRan an 2 KRRun
Iu Ky a Rn K7T
olckk Rn K Kar Kij Krr K
NT K nh
2 K ( Kay K KYR) KinB
KT
Ava R.T
labll manching
Avar is proportional to KRR KRT KRrKTTKKTR which KIT.a][T.na K/TKTT KrT, the Schur com- plement of Kfr,1q,[T.n) n) with KTT. Note that this is the Gaussian conditional variance formula, assuming we are making predictions of R based on T. This regularizer ensures that not only do the reconstructed images result in the correct predictions (ensured by the label matching term) but also that our distilled dataset images do not deviate significantly from the training distribution, as measured by the NTK. We hypothesize this term is what contributes the success of RKIP over KIP in the finite-width setting, As there is nothing that directly ensures that KIP distilled datapoints remain "similar" training images (only that they are predictive), these distilled images may be more susceptible to domain shift, such as moving from the infinite-width setting to finite width. This interesting behavior could be the subject of future work.
20

---

Published as a conference paper at ICLR 2024
G
FINITE RKIP IMAGES
CIFAR-10 Animal vs. Vehicle reconstructions with M 20 500
Train Images Reconstructions
Figure 12: Reconstructing 20 images from network trained on 500 CIFAR-I0 images. Reconstruc- tions often do not match actual training images and contain heavy corruption. Retraining on these images yields high accuracy.
fig. 13 and fig. 14 show the resulting reconstructions when reconstructing 20 images from a dataset that may contain up to 500 images. Reconstructions are made from 4096 networks with linearized dynamics.
MNIST Odd
Even reconstructions
20
MNIST vs. Even reconstructioos with M
20
Tran ronniaggs RUCISSTAENNG
3
6
8
9
7 3
3
5
5
Train Image Recherud om
3
5 3
2
9
G
5
C
9
2
G
2
2
0
7
3
3
5
6
2
3
6
8 7
9
3 3
2 3 5
8
3
5 3
2
9
G
5
6
2
G
2
2
2
2
0
7
5
3
3
S
8
6
2
tinns
MNIST Odd
econstructions
150
Tram mage Reconttttion
2
3
3
5
4
9 8
3
2
S
3 3 9
9
6
5
5
9
3
4
5
8
5
4
8
O
mays
2
3
/ 3
5
4
4
9
4
3
5
/
3 3
9
b 0
5
3
3
4
5
5
4
7
8
MNIIT 0dd vs. Even reconstuctions with M
20
200
MNIST Odd vs. Even reconstructions with 20 <- N 300
Train Imaget DecnnInCcc
4
3
9
5
5
4
7
3
2
4
9
5
2
9
8
6
3
3
4
3
9
4
3
1
8
5 9 5
4
7 3
5
3
2
4
9
8
3
2
8
4
6
3
0
8
3
5
4
3
6
9
5
0
9
MNIST Odd vs. Even recons recontructioo with M
20
N
400
MNIST Odd vs. Even reconstructions with 20 N 500
Trah magec Roa RESNSSTI2OAS
4
2
9
8
i
9
Than mages nesonemucten
8
9
5
5
8
7
5
2
3
4
/
2
8
/
8
9
5
4
4
/
8
9
5
9
8
5
3
4
7
9
4
3
9
2
6
7
4
4
8
3
9
9
Figure 13: Reconstructing 20 images from dataset that may be larger than 20 images (MNIST Odd vs. Even)
21

---

Published as a conference paper at ICLR 2024
CIFAR-10 vs. reconstructions
Animal reconstructions 50
imagus Reardmackos
Trah imagge RecorAtrucion
CF DFAR-10 Animal Vehicle reconstructions with
100 CIFAR-10 Animal Vehicle reconstructions with N
Recessruuuue
Than Imsore Recomemuttion
Tran Imogis
reconstructions with
CIFAR-10
reconstructions
Nucosdracker
Tok mngges
Tran imagpe
CIFAR: 10 Animal vs. Vehicle reconstructions with M
20 =
400
CIFAR-10 Animal vs. Vehicle reconstructions with 20 = N 500
Than Insgge Becoreruate aeroeee
Trai Imoges
Figure 14: Reconstructing 20 images from dataset that may be larger than 20 images (CIFAR-10
Animal vs. Vehicle)
H MULTICLASS RECONSTRUCTION Loss
Here we derive the multi-class reconstruction loss which we use section Our least-norm pre- dictor satisfies the following conditioned (assuming that the network behaves in the linear regime);
arg min 20
|002 8.1.
2
E (C],
A (X+) = 5f JE (X.)
(15)
With fā, referring to the network output the eth class and ff referring to the training labels for the eth class. The network will converge to the least norm solution (norm of the difference from initialization), due to the network behaving in the lazy regime with gradient flow (de Azevedo https:/mmath.sacccxchhaagg Stacc/eccaagcc.co//uu//////////s wtt.////ee//3397900rooiioi variables our full Lagrangian
azevedo). Writing the equation with dual
C(40, a)
2 aT (A (XT) - (Vf Sá, (XT)))
(16)
With our set of dual parameters E RCXM that is, we have a set of M dual parameters for each class. Taking derives w.r.t A6:
0= VAO((A8,0)
-0 Σ
Vofe (Xr))
(17)
c=1
22

---

Published as a conference paper at ICLR 2024
So our multiclass reconstruction loss is
c A0 A0--
C
reconsttuution
Σ a a
(Xr)
(18)
We can use the same argument as section
B show this attack is exact the infinite width limit.
I EARLY STOPPING AND CROSS-ENTROPY
1.1 EARLY STOPPING
MNIST Odd vs. Even
CIFAR-10 Animal vs. Vehicle
20
100
200
100
20
100
200
500
Figure 15: Mean reconstruction errors for networks trained to various early stopping losses from 10 to 10
MNIST Odd vs. Even Early Stopping (Standard Dynamics)
GFAR-10 Animal Vahicle Early Stopping (Standard Dynamics)
12 Daturce
Distancce
130 Image Indox
390 426
Figure 16: Reconstruction curves for networks trained to a loss le-2, i.e. significant underfitting (under standard dynamics). Compared fig. 2
see that reconstruction quality is unaffected by
early stopping, consistent with the theory.
In the main text, we considered models trained with low learning rates for 10th epochs, so that we achieve the KKT conditions described in section In practice, this is computationally expensive, and often ill-advised due to overfitting, with early stopping being a common regularization tech- nique. One major limitation of the attack proposed in |Haim et al. (2022) that requires network to reach the KKT point to perform the attack. Our method does not require the model reach convergence for the attack to work. Again we note that the time evolution network parameters is given by:
A@(t) = Vefo Ko (1-6((1-) (vr
fo0 (Xr)
fime-dependenn weights, ex(e)
Notably, even finite time, the change in network parameters is still linear combination of the finite-width NTK feature maps of the training set, Vofa, (Xr), with the indicated time dependent weights, (t). Note that the success of the attack in infinite width relies on the injectivity of the kernel measure embedding, not that att) be at its converged value, implying the attack works with early stopping. The caveat with the early stopping attack is that we cannot necessarily hope to recover the original training labels.
We verify that this attack works in practice by repeating the attack procedure in section with early stopping. We apply the attack on networks that achieve mean training loss of from 10 to 10-8 (note that this means that on for {+1, -1} labels, the outputs were around 0.45 off, i.e. quite underfit, in the case of L curve for = 10 in fig. 16 We observe that early stopping in general improves reconstruction quality. We posit that there
10-1), with results shown in fig. 15 with the specific reconstruction
possible reasons for this: firstly. that early in training there is less time for the kernel to evolve, so the network exhibits network dynamics closer to the lazy regime. Secondly, hypothesize that this could be that early in training all datapoints have roughly equal
23

---

Published as a conference paper at ICLR 2024
contribution to the parameter changes, whereas later in training, certain datapoints have stronger influence (see section 6 and appendix E. When some datapoints have much stronger influence than others, this could cause the the less influential datapoints to be "drowned out" by the signal of the more influential ones. Future work could study the complex relationship between early stopping, outlier influence, and network vulnerability more closely. We also note that in the limit of single gradient step, our attack corresponds to gradient leakage attacks (Zhu et al. 2019).
1.2 CROSS ENTROPY LOsS
Following (Lee et al. 2019), we have that wide networks trained under cross-entropy loss also exhibit lazy dynamics provided they are of sufficient width. The corresponding ODE is:
((0,)
Σ Yi log or (Fo,
08
OC(0))
0t 00,
00:
(x)( yy) (x.))
Ot
00,
Unlike eq. 3 there is not a closed form solution to this, however the key point is that A still is linear combination of Vo (x:e).
J EXPERIMENTAL DETAILS
J.1 LIBRARIES AND HARDWARE
We use the JAX, Optax, Flax, and neural-tangente librarics (Bradbury et al. 2018 [Babuschkin et al.) 2020 Heek et al. 2020 Novak et al. 2020; 2022). All experiments were run Nvidia Titan RTX graphics cards with 24Gb VRAM.
J.2 NETWORK TRAINING
Unless otherwise stated, networks trained on real data are trained for 10t iterations of full batch gradient descent, with SGD with momentum 0.9. For the learning rate, we set N x 2e-7, 7, where N the number of training images. For distilled data, we use a learning rate f) = N 6e-6, where N is now the distilled dataset size. We did not find that results were heavily dependent on the learning rates used during training. Additionally, if the training loss was less than 1e-10, we terminated training early. Every reconstruction curve in the main text is the average of unique networks trained on unique splits of training data.
For binary classification,
labels in {+1,-1 2),, and for 10-way multiclass classification, use labels of 0.9 corresponding to the selected class and -0.1 for other classes.
J.3 RECONSTRICTIONS
To create reconstructions, we initialize reconstruction images with a standard deviation of 0.2, and dual parameters to be uniform random within [-0.5, 0.5). We use Adam optimizer (Kingma & Ba 2015), with a learning rate of 0.02 for all reconstructions. As stated in appendix A these could be fine-tuned to improve performance. We optimize the images for 80k iterations. Like with Haim et al. 2022), we found that was useful to use a coftplus rather than a Relu during reconstruction, owing the smoothing gradient loss. We annealed softplus us temperature from 10 to 200 over the course of training, so that we are effectively using ReLUs by the end of training. Unless otherwise stated, aim to reconstruct M 2N reconstruction images, with N the training set size.
24

---

Under linearized dynamics, these two equal Vofo (x) does not change. However for standard dyanmcis there is a small difference, with the difference increasing the more the kermel changes. For the results in the main text, we use Ly. We found that this has 6marginally better performance than Lo. but this difference is rather minor. This also leads to a third choice of reconstruction loss, which Published as a conference paper at ICLR 2024
J.4 DISTILLATION
We initialize distilled images with a standard deviation of 0.2 and distill for 50k iterations with Adam optimizer with learning rate 0.001.
J.5 FINE-TUNING TUNING EXPERIMENTS
For the fine-tuning experiments in section we use the pretrained ResNet-18 from flaxmodels (Wright 2022). For these experiments, we used 64-bit training, as we found improved reconstruc- tion quality. We train for 30000 iterations with a learning rate of le 5. We use SGD with no momentum, and also freeze the batchnorm layers to the parameters used by the initial model. We the hybrid loss described in appendix K
J.6 PRUNING EXPERIMENTS
For the pruming experiments we train for 10x epochs for each training run with learning rate of x 3e 6 with SGD with momentum 0.9. For reconstruction use the same reconstruction attack in the main text for 30000 iterations. For each training iteration we set with width of the network be
found that generally kept the strength of the attack the same for different values of n. If we make the attack too strong, then too many training points will be reconstructed and we will longer have any notion of "easy reconstruct" examples, all examples will be reconstructed equally well.
55 n,
J.7 RECONSTRUCTION POST-PROCESSING
We do not post-process our reconstructions and the reconstruction curves and visualization are based on unmodified reconstructions. Note that Haim et al. (2022) has a more complex recon- struction scheme involving rescaling the reconstructions and averaging, which detailed in Haim
et al.. 2022''s appendix.
K
CHOICE OF KERNEL
During reconstruction, there is a small choice we can make, specifically whether we using the initial- ization gradient tangent space, Veafe, (xz) or the final gradient tangent vector space, Ve, fe, (2x). Specifically, we can choose to optimize Cy or Lo:
Cj = A0
Σ
8 BESR
Co
A0
a
v
fo (x,))
R ERR
call the "hybrid" loss Cs, which considers the change in parameters mixture of both the final and initial kernel:
La = 1 A0- Vo, oo - a
EESR
BESR
25

---

Published as a conference paper at ICLR 2024
MNIST Odd Even
CIFAR-10 Animal Vehicle
Irage index
Inotun Indes
Image Indea
Image trdes
Ineae
Figure 17: Reconstruction curves for convolutional architectures trained on MNIST Odd vs. Even and CIFAR-10 Animal vs. Vehicle classification with varying width multipliers from 1 - 32
Tiny-ImageNet Reconstruction Curves
600
800
L3 Distance
Standard
362
274
Linearized L2 Dislance
2050
Image Index
Image Index
Image Index
Image Index
Figure 18: Reconstruction curves for networks trained on Tiny-ImageNet 200-way classification
In which we have two sets of dual parameters og and ay-- This further increases performance, but in general twice as slow as optimizing Lo or Ly, and we chose to use this loss for the ResNet- fine- tuning experiments, since that setting is more challenging. Note that this attack could be generalized to include multiple checkpoints 0g along the trajectory and more dual parameters, however of course this would require access to more training checkpoints.
L ADDITIONAL RESULTS
L. CONVOLUTIONAL ARCHITECTURES
Here we provide results for our attack applied convolutional architectures. We trained networks on binary MNIST and CIFAR- 10 classifications tasks, as we did in the main text, but trained on convolutional architecture from scratch. We use the common LeNet-5 architecture with width mul- tipliers ranging from 1-32. fig. 17 shows the results. The findings observed on two-layer networks still apply in this settings, however our attack struggles more deeper architectures.
Imy-lmmgeNet Reconstructed Images, n = 400, w 4096
Reco Reconstructions
Train Images
Figure 19: Reconstructed images and their nearest train iamge counterparts for ImageNet re- construction for 4096 network with 400
26

---

Published as a conference paper at ICLR 2024
L.2 TINY-MMAENNET
The main text works mainly with classification on small, low-resulution datasets such as MNIST and CIFAR-10. Here we consider more complex datasets with higher resolution by applying our attack to Tiny-ImageNet classification. Tiny-ImageNer consists of 200 classes with images of resolution 64 64 (Le & Yang. 2015). As the quality of the NTK approximation is negatively affected by image resolution (Jacot et al. 2018). this experiment serves as an important testing ground of the viability of the attack for higher resolution images. We show the results on few-shot classification, consider 1 - 4 images per classes in fig. [8 and with the reconstructed images in fig. 19 Reconstructing higher resolution images is more challenging and improving this attack on high resolution images is an interesting direction for future work.
L.3 KERNEL DISTANCE VS. RECONSTRUCTION QUALITY SCATTER PLOTS FOR MULTICLASS CLASSIFICATIONS
fig. 20 shows the corresponding fig. for multiclass classification. As we observed, multiclass classification has improved reconstruction quality. From fig. 220 we see that multiclass classification sees significantly lower kernel distances (up 1e-2 for MNIST and 2e-2 for CIFAR-10) compared to binary classification (up to 3e-2 for MNIST and 9e-2 for CIFAR- 10, see fig. which may explain why the reconstructions have better quality.
MNIST 10 Classes
CIFAR-10 Classes
Dataset size
Mean Reconstruction Error
500
Mean Reconstruction Error
17.5
12.5
73
0.000
0.004 0.006
0.010
0.000
0.005
0.030
3.015
0.020
Kernel Distance between Init and Final NTKs
Kernel Distance between Init and Final NTKs
Figure 20: Mean reconstruction error vs. the kernel distance from the initialization kernel to the final kernel for multiclass classification. The mean reconstruction error, measured as the average value of the reconstruction curve, strongly correlated with how much the finite-width NTK evolves over training. Dataset size is given by dot size, while the color indicates model width. Multiclass classification sees significantly lower kernel distances (up to le-2 for MNIST and 2e-2 for CIFAR- 10) compared to binary classification (up to 3e-2 for MNIST and 9e-2 for 1IFAR-10, see fig. 3) which may be the cause of better reconstruction quality.
L.4 EXTRA PRUNING EXPERIMENTS
In section considered removed class balanced subsets of the training data at each training iteration, either by random or by ease of reconstruction. If instead we allow class imbalance, we see the same behaviour as in section 6 initially,, but more datapoints are removed, we see in fig. 21 that removing easy reconstructions results in strongly imbalanced classes, resulting in poor test accuracy. Understanding why some classes are more susceptible to reconstruction is likely related to the discussion in section 6 Additionally, we found that if we underfit the data, then we do not observe any difference in test accuracy for pruning random vs. reconstructions. This suggests that the effect of large values only shows later in training, when some datapoints are well fit and others still underfit.
27

---

Published as a conference paper at ICLR 2024
CIFAR-10 Accuracy
32.5
Test Test Accuracy (%)
30.0
25.0
22.5
Easiest Reconstructions
Easiest Reconstructions (Unba lanced)
Random
Random (Unbalanced)
260
400
600
Datapoints Removed
Figure 21:
Test accuracy of iteratively pruned CIFAR-I0 using either random pruning or pruning based on easily reconstructed images with either class balanced subsets or non-balanced subsets.
L.5 ADDITIONAL RECONSTRUCTION CURVES
additional reconstruction curves for all dataset sizes in [20, 50, 100, 150. 200, 300, 400, 500] for MNIST Odd vs. Even and CIFAR-10 Animal vs. Vehicle in fig. [2 We show the same reconstruction curves with the distillation reconstructions in fig. shows the same reconstruction curves for early stopping. Finally, fig. 24]shows the same curves for multi-class classification.
We show
MNIST Odd vs. Even (Standard Dynamics es) Training sot site
Distarne
23
Imape kro
Imope Indes
Imageinder Index
Imuge Inder
imagelnder
mage Idee
MNIST Odd Even (Linearized Dyanmics)
Trainig sot size
12 Delance
(Standard Dynamics)
Training
Distance
58
300
300
230
200
350
330
403
CIFAR-10 Animal vs. Vehicle (Linearized Dynamics) Training set size
29
50
109
150
200
300
900
Distncc
300
200
200
403 aragh vitra
Troige Bodie
Trope India
Imaprisiie Eides
Imoge Fidin
mmage Incie
Imwg:
Figure 22: Reconstruction curves for binary classification tasks
L.6 RECONSTRUCTION IMAGES
Here we show all the reconstruction images and their nearest training images in terms of L2 distance. Images sorted based their rank in the reconstruction curve.
28

---

Published as a conference paper at ICLR 2024
MNIST Odd vs, Even Early Stopping (Standard Dynamies)
Training sot size
20
50
100
150
200
300
400
500
Ditarcce
300
200 Imge
Inage
Imape
Image
Imuge
Image
Image
Irage
MNIST Odd
Even Early Stopping (Linearized Dynamics) Training sot size
Ditarce
indpeindee
Imuge Index
Imuge Hddc
Image-ndde
mage idde
Yaage idee
Image rdss
CIFAR 10 Animal Vehidl Early Stopping (Standard Dynamics)
Trainin set size
Datance
23
206
CIFAR-10
Early Stopping (Linearized Dynamics)
Training
L2 Distennce
50
100
100
200
200
206
200
206
400
Figure 23: Reconstruction curves for binary classification tasks with early stopping
L.6.1 BINARY CLASSIFICATION
We show the reconstruction curves for MNIST Odd Even and CIFAR-10 10 Animal tasks for width 4096 and 1024 networks with linearized or standard dynamics figures 25 to 32
Vehicle
L.6.2 MULTICLASS CLASSIFICATION
We show the reconstruction curves for MNIST and CIFAR-10 10 10-way classification for width 4096 and 1024 networks with linearized or standard dyanmics in figures to 40
29

---

Published as a conference paper at ICLR 2024
MNIST Classes (Standard Dynamics)
Training set size
12 Distance
35
Classes (Linearized Dynamics) Tratning
Destance
100
100
200
206
200
230
403
CIFAR-10 10 Classes (Standard Dynamics)
Training set size
20
20
50
100
150
200
300
400
500
Detance
23
360
200 imae
Image
Image
Imuge
Imuge
Image
image
Image
CIFAR-10 10 Classes [Linearized Dynamics)
Training set size
Distance
Inagelncle
Image Indes
Imugeindar Indax
Image-nsh hndec
Indgelnnda
Imgge rieec
Image 0d84
Image rdea
Figure 24: Reconstruction curves for multiclass classification
30

---

Published as a conference paper at ICLR 2024
MNIST Odd
3
1
a
q
7 2 3
4
3 8 4
9
2
3 6 8 2 4 5
A 5
6
3 4 3 3 3
3 6 2
9
5 2 4/ 6 8 / O
/
O
4
9
/ 5 2
/ 6 / 6 3
3
O 2
2
5 2 5/ 6 8 /
0
4
9 2 / 5 5
3 9 8 4
6 4
0
3
6
8 8
D
3 9 8 4
6 4 9
3
5
7 8
y
8 8
2. 0 8 4
5
6
4
0
A
2
/ 5 / 5 8 4
7
2 0 8 4
5
>
6
4
0
8
6 3
8 3
5
0
8
/
5
S Y
/ 3
5
+
5
3
7 5
0
9
8
8 8
4
6
9 5
2
D D
0
8
8 4
5
9
3
2
5
8
9
O
2. 4 6
3
5
2
8
/
6
6 6 4
2 4 6
3
5
3
8 / 9 3
0
6
6
6
6
7
0
3
9
6 9 7
/
3
2 3 5 Z
C
O
Y Y
O O
3
7 0 3
9
0 9 7
3 9
551 3
/ 9
2 2
0
4 6
[
O O
2 2
8 1 4 3 / 9 / 2 2 0 8 6
9
4
9
5 9 5 5 2 4 O3
2
7 4
4
9
2 4 2
5 4 9
3
5 2
2 7 4 0
3
2
4
9
4
2 7 9 Z 5 7
5
A
3
G
8 7
9 7
2 7 9 4
/
7
3
A
6
0
8
2
43
2
76770
& 5
27 0
8 2 O 43 0
7
8 5
8 f 2. / 2 5 7 20
6 7
/ F 2 / 2
2
6
1 011.6666666888
0
2
6
6 6 7 68
0
29
7 6 O 3 0
/ /
0 0
6
1
7 O O 5 D 1
0
6 5
7 / 8
/ 8
8 9 / E
7
1
9
2
/ / 8
/
8
8 6 9 3 / 5
9 9
2 S
7 7
5
3
1
( 9 Y 5 5 / 1 6 6
6
4
/
9 3
2
5
3
7 7
6
0
5
3
5
/
4
0
9 5
3
5
8
7
/
7
4
7
0 7 9
O
S O 3
0
5
7
6
8 3
3 3
7
8
6 7 2 C
8 J 7 3
2
O 7 2 6 6 O
2
O 2 O
Figure 25: Reconstructions for MNIST Odd vs. Even, Linearized Dynamics, 4096 width.
31

---

Published as a conference paper at ICLR 2024
MHIST Odd
Col
4
6
2
G
D
3
9
7 4 [ S 2 1
9
Y
5
F
7 O 4 / 5 2 1 9 5 E
8
f
3
3
5
9
ST S Z 4/ L 7 9
5
2
8 & 5
2
G
A
6
7
/ > 2
3
ST S 2 5/ 4 /
5
/
6
6
2
J 8 / 2.
7
6
6
4
6 2
J- 8 / 2
7
9
/ 3
4
9
6
1.2 S 8
2 4
7
/
6
9
7.2 5- / 8
2
4 7
35
2
/
0
6
5 6 9
5
2/ / 8 8
6
> 8 3
4
y
6
3
/
4 2 /
5 3 9
1 8 7
6
8 3
3
3/4 5
3
6
3
0
2 9
3 3 5 9 / / 3 6 3
0
500 2 9
4 L 2
7 O 3 9 70 S 9
96 4 | 3 3 3 O B
45 7 6 2 7 O 3 7
9 9
S / 9
9 9 Y
3 4 O 5Z 5
S / 9 9 4 Y
0 D 6
2
2
19
2 J
7 0 D
2
2 0
5 2
0
7
4 0
5
4
2
7
O O
9
7 4
4 0
1
5
A
2
7
7.
9
3
7 0
9
2 / /
9
9
7
7
5
7
5
7 8 4
5
7
5
4
6 5- 7
5
4
8
2
9
6 5 / 5
4
9
f
5 9
7 8
4
/
3
7
8
/ 8 4
3
5
4
7
8
3
3
3
;
7
5
/
7
4
5
2
F
5
0
1 7
4
5
4/
C 5
V
C
4 1
6
3
/
5
4
0
9
7
9
5 9 2
7
9
O
7
6
8
5 4 2 5
6
6
2
4
8
O
7
6
O
8
7
3
2.
5 2 5
:
2
5
- 0
3
5
C1 3
$ 5 / 3
6
4
3
6
5
$
3
L /
8
R 2
3 6 3 7 + 0 3
0
2
2
g /
C 7
O 6
6 5 D 0 00977222 6 8
6
2 2008
Figure 26: Reconstructions for MNIST Odd vs. Even, Standard Dynamics, 4096 width.
32

---

Published as a conference paper at ICLR 2024
MNiST Odd
Unear
4
5
8
4
5
1
Y
J
G
5
6
7
8 5 1
4
&
4
3
9
4
4
5
7
5
3
6
7
&
5 ] A 9
2 2
\ 4
7
Q
/
7
3
5
5 5
U S 8
5
a
9 2
2
Q
3
5
2.3
8
/
7
4 3
6
L
3.
y
6
2.3 3 8
/ 9
9
3
O
9
/
3
y 6
4
/ 5
5
7
6
08
9
O
8
/ 5
8
5
3
7
9
4
>-
08
6
9
8
2
0
3
9
/
9
+
6
0
6
/
6
2.4 0
3
9
/ 3
0
8
7 3
9
4
0
> L
O
9
5
9
9
4
4
9
3:
O
C
/
7
4 2 4
7
3 O
O
7
2
/
6
O
9
9
8
O
2. 0 6 4 2
0 4
Y
/
7
7
2
6
Kr + 5
7
5
4
/ 5
7
5
2
4
7
4
8
4 0
/
G
2
9
3
8
4 2
8
6
F
4
4
9
3
8
4
2
A:
6
5 5
5 3
4 4
O
7
5
2
7
5
/
3 /
0
5
8
4
3
3 9
3
6
31
3 6 9
9
7
0
O
1
S
G >
7
0 2 4-
66
4
| 8 4
3
8
0
O
7
2
5
8
5
A
S
2
C
+ 3
5 d
6 3
0
3
4
+
5
5
7
3
3
2
2
5-
0
/
5
7 >
4
8
7
8
6
2
4
4
2 I 5 3
9 7
4
3
J
4 6 6 2 2 6
0
2
O
2
2 q
D 3 & D
0
2
2
2
0,
7
8 O
2 5020
Figure 27: Reconstructions for MNIST Odd vs. Even, Linearized Dynamics, 1024 width.
33

---

Published as a conference paper at ICLR 2024
MNiST Odd
1024,
Col
4
6
3
2 4
f f
7
2 0
I
2
2 6 0 5
F2 + 5 5
4
3
9
9
5 7 / 4
1
9
67 77 / 9
9
21 2 / [
S
3 2.0
2 8 y / 2
{ 7 2
: 4 /
Y
G 3 2
3 9 4
/ 4
Y 4 3 2 9
F. S 5
J
V
7 O 1
/ 5 5
5
W1.7
4
7
7 4 9
6
7
3
3/55 5
1
1 5 7 8
7
3
O
9
3
0
6
7
4
F
G
8.4
4.
4 4
5
4
7
/
4
5
5
6
6
4 5
3
/
5
8
9
4
5
9
7
5
6
0
TV
9
3 5
4
3
4
/
2
3
2
8
4
3
2
G
7
2
5
H
6
O 7
F
6
2 3
2 . 3
/
8
A
5
8 2. 5
7
/ 9
6
4
4
2
2
0
3
2
6
25
C 3
0
0
3
06
12
/
/
6
7
4
6
2
5 7
4
1
3
9
0
3
M
7
6.5
7
3
6
7
8 0
0
7 5
2
A
4
&
2.7 5 2 4 2
6 5
7
3
2
A J:
6 2 8 5
6
7 7 6
5
5 3 6
0508207 2
O 2
2
O 2
2008
Figure 28: Reconstructions for MNIST Odd Even, Standard Dynamics, 1024 width.
34

---

Published as a conference paper at ICLR 2024
Vehicle Distance
Figure 29: Reconstructions for CIFAR-10 Animal vs. Vehicle, Linearized Dynamics, 4096 width.
35

---

Published as a conference paper at ICLR 2024
Vehicle
Col
Figure 30: Reconstructions for CIFAR-10 10 Animal vs. Vehicle, Standard Dynamics, 4096 width.
36

---

Published as a conference paper at ICLR 2024
Vehicle Distance
Figure 31: Reconstructions for CIFAR-10 Animal vs. Vehicle, Linearized Dynamics, 1024 width.
37

---

Published as a conference paper at ICLR 2024
Vehicle
Ool
Figure 32: Reconstructions for CIFAR- 10 Animal vs. Vehicle, Standard Dynamics, 1024 width.
38

---

Published as a conference paper at ICLR 2024
VNIS
O
B
3
0
9 9
f
3
7
2
7
5
4
Y
7
7
9
Y
2
7
5
4
Y
7
3
/ R
9
Y
S
7
9
9
9
1
8
4
9
3
6
3
6
7
S
O
/
9
9
5
J
7
3
8
4
9
3
3
S
O
Y 9
3
4
5
3
V 9
3
4
5
8
9
3
8 7 5
7
9
9 S
C
8
7
9
8
5 R
/
7
9.5 C
8
6
5 6
8
5 3 5 5
7
7 2
0
2
/
O 8 6
3
9
3 4 3
2
4/7 2
0
2
/
8
6934
5
5.6
9
9
4/
5
6
9
7
3
56
9
9
4
5
5 / 0 519 3 2
2
9 3 /
8 7
55
4
3 4 0 2
44
3 4 9
8 7
4
7
4 2.9 9 5 O 5 0
8
P
4 2. 9 5 0
0 3 8 5
8
/
9 g C / 1 7 d
4 2
6
89 8 C / 4 7
62
6
S 3 8 4
2
3 1 7
G 2
7
7
8
4 7 7 6
6 3 8 4
2
3 f 7
L 2 2
4
8
9 2 3 2 1
6 3 7 3
3
8 7
4
5 0
3
/
7
2 5
7 9 3 3
3 8 0
2 / S
O 1
4
2/250 O
J /
4 6 1
022 8 3 2 2 / 9
65
0.283 2 / / 0
659 9 159014093 FTT.0:443111/11433333333333
673 /
659 159014033 011144231673373333333335
L8 00033322 6 9
88 03 6 (00003666650033330
5 (111111777200000000000022
5 8 700
2 0 6
8 0 6 I /
2 0 7 7
9 / S 832
5
4 2 O 7
7
9 / 0 $ /32 2
/
7 8 O
4
O
0 3 7
O 6
7 3 8 O 4 O
0 3 9 3 2 0 3
6
60
Figure 33: Reconstructions for MNIST 10 Classes, Linearized Dynamics, 4096 width.
39

---

Published as a conference paper at ICLR 2024
VIIST Recurstrucions
Col
4
O
0 4 O
5
O
f
6
T
9
S 6 6 9
5
O
4
9
5
7
7
3 5 7 4 1 7
3 7
8
O & S
Y
Y
6
8
4
3 5 4 1
7
7
O
6
2
8
4
4 0 3 2.
5
6
3
0
6
4 0
3
S
9
2
9
9 Y
7
4
9
4
7
9
6
1
/
(
7
4
9
/
4
3
9
5 8 6
8/
4
3
S
3
4
6
7
/
3
/
8
4
3
S
3
1
)
4 3
9
2
9
7
2
8
7
6
O
9 7
2
9
7
2
7 /
8
7
63
3
J 8
8
9
9
/
2
5
5
O 6
J 8 / 8
9
9
2. O
3
5
4 5 2 5
7
2
4 9
0
0
5
4 5 2 5
2
9
0
5
6
6
9 D 6 [
S2
/ /
9
9 D 2
/
2. / 57 S
/
9
7 6 S
5
J
5
3
55
8
5
7 6 S 5
6
J 5 7
5
5
9 22 6 9 9 U 8 2 / 0 1
2 / 6
9
9
U
/
09
4 O 9 7 C 6 523
6 4
5 507012
4 Y 9 7
6 57 C 6
50JO12
19 5 / 3 A O5
8
79 2 S / D 3 2053
8
095 Y 2 & 5 7
2 3
O
04 3 &
6
5
2
O
5 1 3 O
9
Y
2
3
5 3 3 O 3 7 9
3
Y
8 6 3 / 2
2
/ 5
/
7
8
O O
4
8 6 3 2
2 6 8 F
0
/
8
3
1 # 7 3
8
6
3
3
7
2
6
2 2
1 # 7 3 8
6
3 3 8858
64
0 1627 2
8
3
0
0
5 2
0 2
8
0 3 0 H
2
0
5
23
0 0 9 7 2 3 0 0 2 3 2
4
0 9 7 2 3 D 2 3 O
4 1
4
6760
Figure 34: Reconstructions for MNIST 10 Classes, Standard Dynamics, 4096 width.
40

---

Published as a conference paper at ICLR 2024
VIIST
Lineriesd
O 2
&
O 2 >
2
5 8 S 9
2
S 3 S 6
7
5
3
3
3
2
5
4
\ 8 O
4
7
9
&
Y
{
C 9
3 2
7
\ 8 O
9
7
9 J 3 2
9 7 0
5
3
4
S
G
9
6 6
q 7 0
5 3
8
3
9
r
9
4 Go 7 3 3
7
9
8
7
9
4 Co 7 3
7
4
J
6
8
-
6 2
9
4 1 9 [
Y I
9
3
4
S
/
/
7
3
6 <
5 5
3
1 1 9
9
3
4
S
/7
5 9 6 7 5 O
5
0 0
/ /
6 6
5 9 6 2 5 O 5 99
4
2 6 6
3
7 &
7
2
5
3 C
C
3
2 6
6
23
8 / 6
5
3
J
/ 9 5/
O
7
// 6 5 3
/
5
7
6
69 4
7
T 4
9
0
6 9 4 7
9 7 / 4
2
5
V
8
7
/ 7
/ 3 9 2
7 7 /
3
:
9 9 5 6
3 3
/ 3 9
2.
O
7
3 9 P 4
L
4 3
5
3 2 2.
7 7
C
7
7 3
7
4
O
6
2
3 8 2.
0
9
7
2
4
O
7
6
6
7 .2
8
7.7
2
9
8
t
7
8 4
5 4
O
5 7
3
6
8
7
5
7
7
7
2
8
0
9
5
6
3
C
4
5
7
3
O
4
O
9
0
3 6
5
6
6
8
3
AL
7
7
J
&
1
9 9
0
4
7
3
7
6
8
3
55
3
8
7
4
0
5
/
O
2
3
2
5 3
2
8
[
2
6
2 7
K 3 5/
5 7 1 6 6
3
S
6 0 0
7
8
2
%6
30 3 6 0 4 7 2 00000222000
20022777
Figure 35: Reconstructions for MNIST 10 Classes, Linearized Dynamics, 1024 width.
41

---

Published as a conference paper at ICLR 2024
VIIST Recurstrucior
Cel
6
6
0
f
Z
6
6
0
2
t
2.5 9
S
T
2.5 9 2
f
3
6
S Y
2 3
3
7
7 3 3
8
2
7
1 7
3 3 3
9 3 23
5 2 2- G
R
5
5
Y
G. 9 9
5
0 K
8
6.9 8 9
5
/
6 6 8 8 9 1 /5
V 4
9
5 4
S
7 7 8 3
5
5
7 / 5 8 3
8
22704
& C
2
5 2 9 74
2
4
6a 9 C 2
5 2 9 7
T
J
2
6 4
9
7
4
1
/
I
S
5
5
3
6
/
7
f
9
7
/
S
6
5
3
O
3
3
5
3
D
6
/
3
6
L
&
3 3
8 9 4
0
4
/ 4
6
0
7 0
9
3 6
6
8
4
6
3 2 7
/ 9
2
S S
8
Y 5 2 7 + 3 9
2
7
/
2.
9
0.5 Z
8
3
3
7
0 5 2 8
Y
3 3 0
/
3
7
4
8
8
3
4
4 4
3 3
S 9 / C <
8 8
Y
7
4 6 9
6
O
7
5
8
7 4 X
7
L 5 A
2
5
7
7
9
5
2
2
0 7
2 /
9
2.
0
2
0
7
5
3
8 6 7
Y
3 2/
2:1
5: 7
X
2 2
5 8
6
2 / 7
2
6
< J
5 5
O
3
2
7
2
W O G
9 0 D S O 2 / 7
6
0
4
2
2
f
3 S
0 3 5
3 3O5 6 O 3
6
O
C & 7
0 3 7 1
6
0
231
4
Te 2.213 $
2420111 00410010
01734
3
Figure 36: Reconstructions for MNIST 10 Classes, Standard Dynamics, 1024 width.
42

---

Published as a conference paper at ICLR 2024
CIRAR acconstruccuun
Figure 37: Reconstructions for CIFAR-10 10 Classes, Linearized Dynamics, 4096 width.
43

---

Published as a conference paper at ICLR 2024
CIRAR auconstrucciun
Figure 38: Reconstructions for CIFAR-10 10 Classes, Standard Dynamics, 4096 width.
44

---

Published as a conference paper at ICLR 2024
CIRAR auconstruccuun
Figure 39: Reconstructions for CIFAR-10 10 A10 Classes, Linearized Dynamics, 1024 width.
45

---

Published as a conference paper at ICLR 2024
CIRAR ucdonstruccuun
Figure 40: Reconstructions for CIFAR-10 10 10 Classes, Standard Dynamics, 1024 width.
46


Reviews:
[{"review_id": "RrocTUkBO1", "reviewer": "Reviewer_FNQS", "summary": "This paper introduces a new, more stable method for performing reconstruction attacks against neural networks based on the neural tangent kernel. This method requires access to both the initial weight state and the trained weight state, but does not require any information about the data distribution. A number of ablation studies confirm conventional wisdom that larger networks essentially memorize their training sets, that outlier datapoints are most vulnerable to reconstruction attacks, and that there is a strong mathematical connection between reconstruction attacks and dataset distillation.", "questions": "1. Can you make claims about the effect of self-supervised learning on the effectiveness of your reconstruction attacks? Does your method naturally extend to structured outputs, not just scalar outputs?\n2. Do you have any speculation how informative your results will be for networks trained with different losses than MSE? Is this a simplifying assumption to enable your proofs or a factor that could significantly change the structure of the empirical NTKs your method uses?\n3. Is there an underlying assumption in (1) that there is a prior (i.e. weight decay) on the parameters of the network?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper develops a very nice algorithm for dataset distillation based on inducing point methods. Figure 8 in particular shows the value of the proposed approach. The underlying theoretical connections to dataset inversion provide confidence in the method.\n\nThe paper provides thorough experimental evidence for the theoretical claims. The datasets used (restricted MNIST and CIFAR) are small, but this is to be expected with a computationally-intensive object like the NTK. The reconstruction attacks on ImageNet are impressive.\n\nThe discussion about the choice of kernel in Appendix I is interesting. The authors should at least incorporate the insight about combining initial and final weight states into the main text. Likewise, why are the results on more complex architectures reserved for Appendix J? These are interesting results that should be incorporated into the paper, if even briefly.", "weaknesses": "The authors claim that “outliers” are more vulnerable to reconstruction attacks, but this notion of “outlier” is not well defined up front. I believe that your technical definition for outliers is points that have high $\\alpha$ values (i.e. points that are “hard to fit”), but this does not necessarily mean that these are points that are distant e.g. under the Euclidean metric in the input space.\n\nThe text on most of the figures is so small it makes them hard to read, even on a screen.\n\nReconstruction plots are not explained until the first paragraph of Page 6, but the plots appear as early as page 4. This creates a readability problem, since the construction of these plots is not self-evident.\n\nIt would be nice to have a central definition of all model variants (e.g. RKIP, RKIP-finite). These definitions are currently spread across the paper.\n\nSmall issues:\n* This phrase in the abstract doesn’t make sense: “of its effective regime **which** datapoints are susceptible to reconstruction.”\n* The statement that non-privacy preserving networks are “useless” is probably a bit hyperbolic in the introduction, this is very application-dependent.\n* In Table 1, specify whether the values are “accuracy”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ldzQUENlip", "reviewer": "Reviewer_GBp6", "summary": "In this paper, the authors investigated the reconstruction attack in the view of neural tangent kernel (NTK). From a well formulated description, the authors showed interesting results with both theoretical and practical meaning.", "questions": "In the current version, the reconstruction performance is related to the number of training samples as well as the property of the samples. How about the effect of data dimension. Especially, the author cast the reconstruction loss as a sparse coding, also Haim et al. (2021) regarded the training process as encoding. Then, can the authors obtain some conclusion about data dimension?\n\nThe reconstruction problem is a complicated optimization problem and the result could be totally different when different initial solutions are used. I notice that in algorithm 2 \"randomly initialized reconstruction images\" are used. Then how about the divergence of the reconstruction result? Is that necessary to use special initialization, e.g., an image in one of the two classes, an image from another class, a natural image of which the class is not in the training set, or a random generated matrix?\n\nAfter reading the rebuttal and good discussion, I would like to increase the score from 6 to 8. But please talk more about the link to e.g., GradViT: Gradient Inversion of Vision Transformers; Deep Leakage from Gradients, which can use local gradient information (even the model is not well trained) to reconstruct the training samples.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The memory of deep neural networks is of great interests and importance. It is believed that the memory and also the memorization is closely related to the training dynamics, which is however not clearly investigated. Thus, I personally like the way of modelling the data reconstruction by NTK, which indeed capture the main properties on memorization dynamics. Though there are still many simplification, the good performance demonstrate the rationality of the modelling. So I think the main strengths include:\n\n- A novel and interesting way of modelling memorization from training dynamics.\n\n- Theoretical discussion well coincides with numerical experiments.\n\n- Clear discussion on the weakness, which actually could inspire future works.", "weaknesses": "The main weakness are for some unclear settings. Please see the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NS33PRG53M", "reviewer": "Reviewer_kua5", "summary": "This paper studies the reconstruction attack, and shows that the reconstruction attack can recover all samples in the training data set. Studies are carried out on the properties of easily-reconstructed images.", "questions": "I think the empirical studies in this paper are sound but the theoretical parts are below the acceptance standard, so I give a score 5. Please consider improve the theorems, and update them either in the submission or reply in the rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "This paper provides interesting experimental results. The main paper is clear and easy to understand.", "weaknesses": "My major concern is about the theoretical results in this paper. The paper claims their theoretical results as one of their major contributions, but from the presentation in the paper, this contribution is not as sound as the empirical side.\n\nFor the two theorems, Theorem 1 and 2, their presentation needs improvement.\n*    For Theorem 1, one can intuitively understand its meaning, but it is hard to interpret the English sentence into a formal mathematical statement. The proof is also vague, with many descriptions but few math formulas and equations. It is hard to rigorously understand the mathematical meaning of this theorem, and it is also hard to check the correctness of the proof. The authors need to rewrite Theorem 1 (maybe leave an informal description in the main paper and postpone the full theory statement in the appendix) and provide a more rigorous proof.\n*    For Theorem 2, the derivation in the appendix is readable, but the theorem statement in the main paper needs to be more clear. However, when considering infinite width of the neural network, some derivations are missing: for example, in Page 19, the first \"->\" needs more details. Although k_{\\theta_0} -> k_{NTK}, the error terms are needed in the later derivation to show that a negligible |k_{\\theta_0}-k_{NTK}| really leads to a negligible error term in the first \"->\" in Page 19.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)