Reconstructing Training Data From Trained Neural Networks
Authors: Niv Haim
Gal Vardi
Gilad Yehudai
Ohad Shamir
michal Irani
Venue: NeurIPS 2022 Accept (2022)
Keywords: implicit bias
dataset reconstruction
privacy attacks

Abstract: Understanding to what extent neural networks memorize training data is an intriguing question with practical and theoretical implications. 
In this paper we show that in some cases a significant fraction of the training data can in fact be reconstructed from the parameters of a trained neural network classifier.
We propose a novel reconstruction scheme that stems from recent theoretical results about the implicit bias in training neural networks with gradient-based methods.
To the best of our knowledge, our results are the first to show that reconstructing a large portion of the actual training samples from a trained neural network classifier is generally possible.
This has negative implications on privacy, as it can be used as an attack for revealing sensitive training data. 
We demonstrate our method for binary MLP classifiers on a few standard computer vision datasets.

Reconstructing Training Data from Trained Neural Networks
Niv Haim"
Gal Vardi*
Weizmann Institute of Science niv. www.hhmmmweemmaae imaan...aa.... ac il
TTI-Chicago and Hebrew University galvardi@ttic.ede
Gilad Yehudai"
Ohad Shamir
Weizmann Institute of Science gi gilad. yehudai wwwizmann.cc. ac il
Weizmann Institute of Science ohad. shamir@weizzann ac il
Michal Irani Weizmann Institute of Science
mi michhal.. aerann@weizzmamn ac. ac.
Project page: httppp///hllaaide....chhhoooooss
Abstract
Understanding to what extent neural networks memorize training data is intrigu- ing question with practical and theoretical implications. In this paper we show that in cases a significant fraction of the training data in fact be reconstructed from the parameters of a trained neural network classifier. We propose a novel reconstruction scheme that stems from recent theoretical results about the implicit bias in training neural networks with gradient-basse methods. To the best of our knowledge, our results the first show that reconstructing a large portion of the actual training samples from trained neural network classifier is generally possible. This has negative implications on privacy, as it can be used as an attack for revealing sensitive training data. We demonstrate our method for binary MLP classifiers on few standard computer vision datasets.
Introduction
is commonly believed that neural networks memorize the training data, even when they are able generalize well unseen test data (e.g., [Zhang et al., 2021 Feldman 2020)). Exploring this memorization phenomenon is of great importance both practically ly and theoretically. Indeed, it has implications understanding of generalization in deep learning, the hidden representations leartt by neural networks, and on the extent to which they are vulnerable to privacy attacks.
A fundamental question for understanding memorization is:
Are the specific training samples encoded in the parameters of a trained classifier? Can they be recovered from the network parameters?
In this work, we study this question, and devise a novel scheme which allows us to reconstruct significant portion of the training data from the parameters of trained neural network alone, without having any additional information on the data. Thus, we provide a proof-of-concept that the learning
Equal contribution, alphabetically ordered
Work done while the author was at the Weizmann Institute of Science
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a) Top 24 images reconstructed from binary classifier trained on 50 CIFARIO images
(b) Their corresponding nearest neighbours from the training-set of the model
Figure 1: Reconstruction of training images from a pretrained binary classifier, trained on 50 CIFARIO images. The two classes are "animals" and "vehicles", We calculate the nearest neighbor using the SSIM metric.
process can sometimes be reversed: That is, instead of learning model given training dataset, it possible find the training data given trained model. In Figure [ we show how our approach reconstructs images from the CIFAR 10 dataset, given a simple trained binary classifier.
Many works try "crack" neural networks by analyzing and visualizing either their learnt parameters or representations [Erhan et al., 2009 Mahendran and Vedaldi [2015] Olah et al.] 2017 2020). This is usually done by "inverting" " the model, namely finding inputs that strongly corre lated with the model's 's activations [Mordvintsev et al., 2015 Yin et al., 2020 Fredrikson et al., 2015). Unsurprisingly, the results are semantically correlated with the training dataset. However, one rarely sees an exact version of training sample.
Our results have potential negative implications privacy in deep learning. Our scheme can be viewed as a training-dan data reconstruction attack, since an adversary might recover sensitive training data. For example, if medical device includes model trained sensitive medical records, an adversary might reconstruct this data and thus violate the privacy of the patients. Privacy attacks in deep learning have been widely studied in recent years (cf. Liu al. [2021)), but far aware, the known attacks cannot reconstruct portions of the training data from a trained model.
Our approach relies on theoretical results about the implicit bias in training neural networks with gradient-based methods. The implicit bias has been studied extensively in recent years with the motivation of explaining generalization in deep learning (see Section 2). We use results by Lyu and Li [2019], J. Ji and Telgarsky| [2020), which establish that, under some technical assumptions, if we train neural network with the binary cross entropy loss, its parameters will converge to stationary point of a certain margin- maximization problem. This result implies that the parameters of the trained network satisfy of equations w.r.t. the training dataset. In approach, given trained network, we find dataset that solves this set of equations w.r.t. the trained parameters.
We show that large portions of the training samples encoded in the parame- ters of a trained classifier. We also provide a practical scheme to decode the training samples, without any assumptions on the data. As far we know, this is the first work that shows that reconstruction of actual training samples from a trained neural network classifier is possible.
Our Contributions
2 Related Work
Understanding and Visualizing what is learnt by Neural Networks. The most common approach for analysing what is learnt by neural network is by searching inputs that maximize the class output or the activations of neurons in intermediate layers [Erhan et al., 22099] Olah al., 2020). Oftentimes this is done via optimization with respect to the model input. Optimizing without any prior on the input usually results in noise inputs. Therefore, most approaches incorporate priors such as smoothness regularization or the use of pre-trained image generators |Mahendran and Vedaldi 2015 Yosinski et al. 2015) Mordvintsev et al. 2015) Nguyen et al.) (0166a] bb 2017) (see Olah et al. 2017 for comprehensive summary). Optimization w.r.t. the input may also result in adversarial examples

---

[Szegedy et al., 2013 Goodtellow et al., 2014]. Recently, [Tipras et al. 2018 Engstrom et al. 2019 showed that classifiers trained to be robust to adversarial examples tend to learn representations that are more aligned with human vision. This was later utilized by (Santurkar et al. 2019 Mejia et al. 2019] to generate dlass-conditionas images from trained classifier. While all those approaches indicate that, unsurprisingly, the learnt representations are strongly correlated with the datasets on which the model was trained, none of them demonstrate the reconstruction of exact training samples from the trained models.
Privacy Attacks in Deep Learning. Many methods deal with extracting sensitive information from trained models. Perhaps the closest our approach is model-inversion that aims to reconstruct class representatives from the training data of a trained model [Fredrikson et al. 2015 He ct al.
2019 Yang et al. 2019 Yin et al. 2020]. It is important to note that the reconstructed images, albeit semantically similar lar to some input images, are still not actual samples from the training set. Carlini et al. 2021 2019] demonstrated reconstruction of training data from generative language models. By completing sentences, they reveal sensitive information from the training data. We note that this approach is specific to generative language models, while our approach considers classifiers and is less data specific.
attacks (Shokri et al., 2017) aim to determine whether a given data point was used to train the model or not. For these methods work, the adversary must be able to guess a specific input, whereas our approach does not assume such ability. Lastly, avoiding leakage sensitive information on the training dataset is the motivation behind differential privacy in machine learning, which has been extensively studied (Abadi et al. 2016 Dwork et al.] 2006 Chaudhuri et al. 2011. For an elaborated discussion on the relation of these approaches to ours see Appendix A
Implicit Bias. In overparameterized neural networks one might expect overfitting to occur, but seems that gradient- gradient-based methods are biased towards networks that generalize well (Zhang et al.
2021 Neyshabur et al. [017]. Mathematically characterizing this implicit bias is a major problem
in the theory of deep learning. Our approach is based on characterization of the implicit bias of
gradient flow in homogeneous neural networks due to Lyu and Li [219) and Ji and Telgarsky [2020]
(see Section 3 for details). The implicit bias of gradient-based methods in neural networks was
extensively studied in recent years both for classification tasks (e.g.. Soudry et al. (2018). [ Gunasekar
et al. 2018c), Ji and Telgarsky [2018), Nacson et al. [2019], Vardi et al. [02 Chizat and Bach
2020 Gunasekar et al. 2018aj. Moroshko et al. [2020) ) and regression tasks (c.g.. Gunasekar et al.
20188) s Arora et al. [2019] Azulay et al. [2021]. Yun et al. [2200], Woodworth et al. [2020] Razin
and Cohen [2020], Li ct al. 2020]. Vardi and Shamir 2021 Timor et al. [0222] See Vardi 2022
for survey.
3 Background and Reconstruction Scheme
In this section we present our training data reconstruction scheme, as well as provide a brief overview on the theoretical results about implicit bias, which motivate our approach.
3.1 On the Implicit Bias of Neural Networks
Let S = ((x,,1)) C Rd x 1- } be a binary classification training dataset. Let (0;-)) Rd + TR. be a neural network parameterized by 0 E TRP For a loss function { R -> IR the empirical loss of ((::)) on the dataset S is L(0) := De E(y, ((;, x,)). We focus on the logistic loss ka. binary cross entropy), namely, {(q) = logoog( (1+
Our approach is based on Theorem 3.1 below, which holds for gradient flow (i.e., gradient descent with an infinitesimally small step size). Before stating the theorem, we need the following definitions:
(I) We say that gradient flow converges in direction Ô if lim, O(
[ where O(t) is
the parameter vector at time t; (2) We say that a network
homogeneous w.r.t. the parameters 0 if there exists L > 0 such that for every a >> O and 6, x we have ((a0;x) = a (0.xx).. Thus, scaling the parameters by any factor a > 0 scales the outputs by at. We note that essentially any fully-connected or convolutional neural network with ReL.U activations is homogeneous w.r.t. the parameters 0 if it does not have any skip-connections (i.e., residual connections) or bias terms, except possibly for the first layer.
3

---

Theorem 3.1 (Paraphrased from Lyu and Li 2019), Ji and Telgarsky (2020) Let ((0:-) ) be homogeneous ReLU neural network. Consider minimizing the logistic loss over binary classification dataset {(xi, v:) using gradient flow. Assume that there exists time to such that ((((())) < i Then, gradient flow converges in direction to a first order stationary point (KKT point) of the following maximum-munn margin problem:
min
001l2 2
S.t.
Vi [m] (, 21
(I)
Moreover, [(((t)) + 0 as t + oc.
The above theorem guarantees directional convergence to first order stationary point (of the optimization problem I). which is also called Kurush-Kulnnluucker point, or KKT point for short. The KKT approach allows inequality constraints, and is generalization of the method of Lagrange multipliers, which allows only equality constraints.
The great virtue of Theorem 3.1 is that it characterizes the implicit bias of gradient flow with the logistic loss for homogeneous networks. Namely, even though there are many possible directions of [ol that classify the dataset correctly, gradient flow converges only to directions that are KKT points of Problem O In particular, if the trajectory {()) of gradient flow under the regime of Theorem 3.1
e
converges in direction to KKT point 0, then we have the following: There exist \ A1, An E RR such that
0 = Σ Auy (o
(stationarity)
(2)
Wi E [n], 00; xi) 21
(primal feasibility)
(3)
A An 20
(dual feasibility)
(4)
E [n]. A; = 0 if y: (0: +
(complementary slackness)
(5)
Our main insight is based on Fq. 2 which implies that the parameters i are a linear combinations
of the derivatives of the network the training data points. We say that data point x; is on the
margin if y; (0; xx) (i.e. ( x,)= 1). Note that Eq. (5) implies that only samples which are on the margin affect Eq. since samples not on the margin have a coefficient X; = 0.
3.2 Dataset Reconstruction
Suppose we are given a trained neural network with parameters 0, and our goal is to reconstruct the dataset that the network was trained on. Although Theorem holls asymptotically as the time t tends infinity, it suggests that also after training for finite number of iterations the parameters of the network might approximately satisfy Eq. (2). and the coefficients A satisfy Eq. (4). Since n is unknown (and is the number of samples on the margin)
set 2 2n which represents the number of samples we want to reconstruct (thus, we only need to upper bound n). and fix y = 1 for i ...,...3. :....m//2 m/2 and y:
for i m/2 1,..., m. We define the following losses:
Intationary (x1,... Xm: d1 A1.-.. Am) 0
A V V
(6)
m
Am) max -- 0}
Note that the unknown parameters are the x.'s and A''s, and that O and the yi's given. The loss Lanatoonry represents the stationarity condition that the parameters of the network satisfy, and LA represents the dual feasibility condition. We additionally define I.prior which represents some prior knowledge we might have about the dataset. For example, if we know that the dataset contains images, prior knowledge would that each input coordinate (i.e. each pixel) between 0 and 1. Given no prior knowledge on the data, we can define Lprice = O. Finally, we define the reconstruction loss as:
Ireconstruci ( (x,)
= an Lutationayy + 0z LA + az Lepiior
(8)
'This ensures that fly, (((t)))))) 1 for all Le. time & classifies every sample correctly.

---

where 01 Q2, E E are tunable hyperparameters of the different losses. To reconstruct the dataset, we can use any nonconvex optimization method (e.g. SGD) find the XI.... ,x,: AI..... ........ Am which minimize Eq. ( We note that the X; 8 are not part of the training data, but finding them is necessary order to solve this optimization problem. Finally, we emphasize that there are many other possible options to formulate the KKT conditions Eq. ( as an unconstrained optimization problem. However, this simple choice seemed to work quite well in practice.
We note that if there exist {x and {A which satisfy the KKT conditions, then there are { and {1 which achieve zero loss in Eq. (8) Indeed, such solution can be obtained by adding to {x, > additional points x, with X, = 0, or by duplicating some points in { and modifying the X's accordingly. Also, note that since we choose m 2n, then we set least n labels to 1 and at least n labels to -1. Hence, there is a solution to Eq. 8 even though we do not know the real distribution of labels the actual training data.
We cannot simply use Eq. (3 and (5) in our reconstruction scheme, because they contain the constant "1" which corresponds to the margin (i.e., min; { x:)D. Namely, we only converge in direction to a point 0 that attains margin 1, but in practice we approach some point 0 which attains an unknown margin y (i.e., min; (0;;; x;) 7), and we do not know advance how to normalize it to attain a margin of exactly On the other hand, Eq. (2) and (4) hold not only for but also for any that points at the direction 0, and therefore in our loss in Eq. we rely only on these conditions.
Intuitivelyy a reason to believe that there is enough information in Eq. 2 to reconstruct the data, is the following observation: Eq. (2) represents set of p equations with ((nd) unknown variables, where p is the number of parameters in the network. In practice, neural networks are often highly overparameterized (i.e., p nd), suggesting more equations than variables.
Finally, since by Eq. we have X; = O for every x; that is not on the margin, then Eq. 2 implies that 0 is determined only by the gradients w.r.t. the data points that are on the margin. Hence, we can only expect reconstruct training samples that are the margin (see also Subsection 5.3
4 A Simple Experiment in Two Dimensions
x
x
X
(a) Training Set
(b) Model Landscape
(c) Initialization
(d) Reconstruction
(e) Removing Small X's
(f) Removing Duplicates
Figure 2: Exemplifying our reconstruction scheme on simple 2D dataset (see text for explanation).
In this section we exemplify our dataset reconstruction scheme on a toy example of 2-dimensional data, i.e. we consider (x, v) R22 x [+1]. We set 20 training samples on the unit circle, with alternating labels. For a visualization of the dataset see Figure blue and red "x" represent the two classes. We trained a 3 3-layer model with 1000 neurons in each layer this dataset. The model learns
5

---

to correctly classify the training set. In Figure we visualize the output of the model as function of its input. Blue and red regions correspond smaller and larger outputs of the model, respectively.
We now demonstrate our reconstruction scheme. We first randomly initialize m = 100 points in R?? and assign 50 points to each class. This is depicted in Figure 2 where green points correspond to the blue class, and magenta points correspond to the red class. Next, we optimize the loss in Eq. (8). with prior O. The results of our reconstruction scheme are in Figure 23. Note that our approach reconstructed all the input samples, up to some noise.
To further improve converge to a training sample. In Figure 2 we removed points x; with corresponding X < 5. According to Eq. points with A should not affect the parameters, hence their corresponding X; can take any value. In practice, it is sufficient to remove points with a small enough corresponding A₃- Finally, remove duplicates, we greedily remove points which are very close other points. That is, we randomly order the points, and iteratively remove points that are at distance < 0.03 from another point. The final reconstruction result is depicted in Figure 2
reconstruction results, remove some of the extra points which did not
5 Results
Top 45 images reconstructed from a model trained on CIFARIO (rows 1,3, 5), and their corresponding neirest-neighbors from the training-set of the model (rows 2, 4, 6)
Top 45 images reconstructed from a model trained on MNIST (rows 1,3, 5), and their corresponding nearest-neighhors from the training-set of the model (rows 2, 4, 6)
8
9
9
8
3
6
S
8
9
8
8
6
S
S
S
5
2
9
2
8
8
8
6
X
3
3
D
6
8
3
6
4
0
8
8
3
4
9
3
S
3
7
8
3
3
S
8
3
2
2
8
5
4
3
8'
2
0
2
5
3
Figure 3: Reconstructing training samples from two binary classifiers one trained on 500 images with labels animals/vehicles (CIFAR), and the other trained on 500 odd/even digit images (MNIST). Train errors are zero, test accuracies are 25.0% for MNIST/CIFAR

---

5.1 Experimental Setup
Datasets. We conduct experiments on binary classification tasks where images are taken from the MNIST [LeCun et al. 2010] and CIFARIO [Krizhevsky et al. 2009 datasets and the labels are set to odd vs. even digits (MNIST), and vehicles vs. animal (CIFAR1O). We make sure that the class distribution in the training and test sets is balanced, and normalize the train and test sets by reducing the mean of the training set from both.
Training. We consider MLP architectures. Unless stated otherwise, our models comprise of three fully-connected layers with dimensions d-1000-1000-1 (where is the dimension the input) with ReLU activations. Biases set zero except for the first layer, to line up with the theoretical assumption of homogencous models in Section [3 The parameters are initialized using standard Kaiming He initialization [He al. 2015 except for the weights of the first layer that are initialized to a Gaussian distribution with standard deviation 10-4 (see discussion in Subsection 5.2). We train models using full batch gradient descent for 10 epochs with learning rate of 0.01. All models achieve zero training error (i.e., all the train samples are labeled correctly), ), and a training loss < 10 10-6. To compute the test accuracy, we use the original test sets of MNIST/CIFARIO with 10000/8000 images respectively, and labeled accordingly.
5.2 Training Set Reconstruction
We minimize the loss defined in Eq. 8) with 011 1, 02 5, 03 1. We initialize x; ~ ~N, dz T). where og is hyperparameter, and X; ~ [1, 1]. We set the number of reconstructed samples to m = 2n (where n is the size of the original training set). Note that our loss contains the derivative of ReLU Eq. (6) This derivative is a step function, containing only flat regions which are hard to optimize. We replace the derivative of the ReLU layer (backward function) with sigmoid, which is the derivative of softplus (a smooth version of ReLU). We use the fact that our inputs are images to penalize values outside the range [-1, 1]. To this end we set [ptio = max 1, 0} + max{ -2 1, 0} for each pixel z, and average over all dimensions (pixels) in x; We optimize our loss for 100, 000 iterations using an SGD optimizer with momentum 0.9. We conduct total of 100 runs using a random grid
search on the hyperparameters (e.g. learning rate, dr. See Appendix B for full details). This results in 100m "reconstructed" inputs.
While some x; end up converging to a training sample, some end as noise (similar phenomenon can be observed in 2D in Figure 2t). To identify the reconstructions that are most similar to a training image we use the SSIM metric [Wang et al., 2004].
In Figure 3 we show the best reconstruction results (in terms of SSIM) for models trained n=500 samples from MNIST/CIFARIO datasets (with test accuracy 39.0%%/7..6 resp.). Note that the reconstructed images are very similar to the real input data, although bit noisy. The source of this noise is not entirely clear. Possible reasons may be the complexity of the optimization problem, or the possibility that the trained model has not fully converged to the KKT point of Problem (I.
We observed that small initializations significantly improve the quality of the reconstructed samples. We conjecture that small initialization causes faster convergence to the direction of the KKT point. This is also theoretically implied in Moroshko et al.) [020] (for certain linear models). Similarly, training for more epochs also improves the quality of the reconstruction. In Appendix C we show results for reconstructions from networks trained with standard initialization or trained for much fewer epochs. During the training phase, used full batch gradient descent, to remain much aligned to the theoretical setting. In Appendix C we show that our approach can reconstruct training data also from models trained with mini-batch SGD.
5.3 Practice vs. Theory
In this section we analyze some relations between our experimental results to the theory laid down in Section 3 Given a trained model and its reconstructed samples, match each training sample to its best reconstruction (in terms of SSIM score). We then plot this SSIM score against ( x) (the value of the model's 's output on this training sample)- for all training samples. In Figure each cell shows such plot for a given model. The top row shows models trained on the same architecture with
each
Automobile, Truck, Airplane, Ship vs. Bird, Horse, Cat, Dog, Deer, Frog.
7

---

Models with the same architecture (1000-1000) trained different number of training samples (n)
1.00
n
50
n= 100
n= 500
n= 1000
0.75
0.00
0.25
0.00
-100
100
100
0
19
Classifler Outout
Models trained on n 500 samples with different architectures
1.00
100-100
10000-500 1000-500-100
1000-500011 1100-5000100.0000
1000 1000
Non-Homogeneeus
0.75
SSIM
0.50
0.25
-189
3.00
.50
150
-100
100
1000
Classifier Output
Figure 4: Each point represents training sample. The y-axis the highest SSIM score achieved by reconstruction of this sample, the x-axis is the output of the model. Top:
The effect of training the same model on different number of training samples (z). Bottom: The effect of training models with different architectures (on 500 training samples). The right-most plot shows 3-layer non-homogeneous MLP (with bias terms in all hidden layers). See discussion Section 5.3
different number of training samples (n), where in the bottom row we show the results for models trained on n 500 training samples, with different architectures (all results are on CIFAR10).
Recall that we do not expect to reconstruct samples that are far from the margin (Subsection B.2p.
is evident from Figure (4 good reconstructions (e.g., SSIM> 0.4) are obtained for samples that lie on the margin, as expected from theory. The plots indicate that increasing training size makes reconstruction more difficult. Lastly, as seen from the rightmost plot in the bottom row, we manage get high-quality reconstructions from non hom-geneouus model (trained with biases in all hidden layers). This indicates that our approach may work beyond the theoretical limitations of Theorem 3.1
5.4 Comparison to other Reconstruction Schemes
Model Inversion. Given trained model ((0: we search for x which maximizes or minimizes (0;;)) x). corresponding to positive or negative labels. We initialize x (0, oz) for several values of a and optimize w.r.t. the model output (see Appendix B for the choice of hyperparameters)) In Figure 5a (left) is apparent that our two-dimensional experiment, model inversion successfully reconstructed 7 training samples, which indeed lie local minimum maximum. However, note that our scheme reconstructs all 20 samples (Figure 2}. high dimensions, namely, in MNIST and CIFAR, while our scheme reconstruct large portion of the training set (Figure 3p), model inversion converges to noisy/blurry class representatives that correspond to high/low output values (Figure right). Such results are typical with model inversion since not all class members from the training set are visually similar (see discussions in Shokri et al. 2017 Melis et al. (019
Weights Visualization. The weights of the first fully-connected layer have the same dimension as that of the input. One may wonder whether training samples are directly encoded there. In Figure (5p we show the weights that are most similar (SSIM) to a training sample. or all of them in the 2D case. As seen in the 2D case, most weights are the general direction of training sample, however the scale is unknown without prior knowledge on the data. For images (MNIST/CIFAR 10), not more than 3 or 4 of the weights have resemblance to training samples, while our scheme manages to reconstruct dozens of samples. See Appendix and C for details and all 1000 weights of the models.
Discussion and Conclusion
Even though our results are shown for relatively small-scale models, they are the first to show that the parameters of trained networks may contain enough information to fully reconstruct training samples,

---

(a) Model Inversion
2D
CIFARIO
MNIST
Increasing Value of the Model Output
(b) Weights of the first Fully-Connected Layer
2D
MNISTT CIFARIO
8
8
9
5
5
5
8
5
8
8
Figure 5: Comparison to other reconstruction schemes. Top: Model inversion on the 2D experiment (left), CIFARI0 (top right) and MNIST (bottom right). The CIFAR and MNIST images ordered by the value of their output from left (smallest) to right (largest). Bottom: Weights of the first (fully-connected) layer for the 2D experiment (left), CIFARIO 10 (top right) and MNIST (bottom right). The weights for the 2D experiment are the small purple dots. For the CIFAR and MNIST experiments we show the 10 weights with the highest SSIM score.
and the first to reconstruct substantial amount of training samples. Moreover, the theoretical basis of the implicit bias in neural networks provides an analytic explanation to this phenomenon.
Solving our optimization problem for convolutional neural networks turned out to be more challenging and is therefore a subject of future research. We note that the theoretical results that rely on (i.e., Theorem 3.1 also covers convolutional neural networks. We believe that the homogeneity restriction might be relaxed, and showed reconstructions also from hom-hoooogeeuuu model (Figure bottom- rightmost). We also believe that our method may be extended to multi-class classifiers using an extension of Theorem 3.1 Finally, showing reconstructions on larger models and datasets, or on tabular textual data are interesting future directions.
On the theoretical side, it not entirely clear why our optimization problem in Eq. (8) converges to actual training samples, even though there is no guarantee that the solution is unique, especially when using prior (other than simple bounding to [-1, 1)). As a final note, our work brings up the question: are samples on margin the only ones that can be recovered from trained classifier? or there exist better reconstruction schemes reconstruct even more training samples from trained neural network.
Acknowledgements
This project received funding from the European Research Council (ERC) under the European Union's 's Horizon 2020 research and innovation programme (grant agreement No 788535), and ERC grant 754705, and from the D. Dan and Betty Kahn Foundation, and was supported by the Carolito Stiftung.
References
M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318 2016.

---

S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit regularization in deep matrix factorization. In Advances in Neural Information Processing Systems. pages 7413-7424, 2019.
S. Azulay, E. Moroshko, M. S. Nacson, B. Woodworth, N. Srebro, A. Globerson, and D. Soudry. On the implicit bias of initialization shape: Beyond infinitesimal mirror descent. In International Conference on Machine Learning, pages 468-477 2021.
B. Balle, G. Cherubin, and Hayes. Reconstructing training data with informed adversaries. arXiv preprint arXiv:2201.04888 04484,, 2022.
I.. Biewald. Experiment tracking with weights and biases, 2020. URL httpp::/wwwtt htt.:ttt///w..wabb.cccm wandb.ccnmm Software available from wandb.com.
G. Brown, M. Bun, V. Feldman, A. Smith, and K. Talwar. When memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 123-132, 2021.
N. Carlini, C. Liu, U. Erlingsson, J. Kos, and D. Song. The secret sharer: Evaluating and testing unintended memorization neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019.
N. Carlini, S. Deng, S. Garg, S. Jha, S. Mahloujifar, M. Mahmoody, S. Song, A. Thakurta, and F. Tramer. Is private learning possible with instance encoding? arXiv preprint arXiv:2011.05335 2020a.
N. Carlini, M. Jagielski, and I. Mironov. Cryptanalytic extraction of neural network models. In Annual International Cryptology Conference, pages 189-218. Springer, 2020b.
N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Berbert-Vosss. K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 2/), pages 2633-26500 2021.
K. Chaudhuri, C. Monteleoni, and A. D. Sarwate. Differentially private empirical risk minimization. Journal of Machine Learning Research, 12(3), 2011.
S. Chen, A. Klivans, and R. Meka. Efficiently learning one hidden layer relu networks from queries. Advances in Neural Information Processing Systems, 34, 2021.
L. Chizat and F. Bach. Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss. In Conference on Learning Theory, pages 1305-1338. PMLR, 2020.
C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of cryptography conference, pages 265-284. Springer, 2006.
L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, B. Tran, and A. Madry. Adversarial robustness as prior for learned representations. arXiv preprint 1501::9900000445 2019.
D. Erhan, Y. Bengio, A. Courville, and P. Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
V. Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954-959, 2020.
M. Fredrikson, S. Jha, and T. Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322-1333, 2015.
I. J. Goodfellow, Shlens, and Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
S. Gunasekar, J. Lee, D. Soudry, and N. Srebro. Characterizing implicit bias in terms of optimization geometry. In International Conference on Machine Learning, pages 1832-1841. PMLR, 2018a.
Gunasekar, J. Lee, D. Soudry, and N. Srebro. Implicit bias of gradient descent on linear convolu- tional networks. arXiv preprint arXiv:1806.000 004686 2018b.
10

---

S. Gunasekar, J. D. Lee, D. Soudry, and N. Srebro. Implicit bias of gradient descent on linear convolutional networks. In Advances Neural Information Processing Systems, pages 9461-9471, 8c.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level per- formance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Z. He, T. Zhang, and R. B. Lee. Model inversion attacks against collaborative inference. In Proceedings of the 35th Annual Computer Security Applications Conference, pages 148-162, 2019.
B. Hitaj, G. Ateniese, and F. Perez-Cruz. Deep models under the gan: information leakage from collaborative deep learning. In Proceedings of the 2017 ACM SIGSAC conference computer and communications security, pages 603-618, 2017.
Y. Huang, Z. Song, K. Li, and S. Arora. Instahide: Instance-hiding schemes for private distributed learning. International Conference on Machine Learning, pages 4507-4518. PMLR, 2020.
Y. Huang, S. Gupta, Z. Song, K. Li, and S. Arora. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 14:7232-7244 2021.
M. Jagielski, N. Carlini, D. Berthelot, A. Kurakin, and N. Papernot. High accuracy and high fidelity extraction of neural networks. In 29th USENIX Security Symposium (USENIX Security 20), pages 1345-1362, 2020.
M. Jegorova, C. Kaul, C. Mayor, A. Q. O'Neil, A. Weir, R. Murray-Smithe and S. A. Tsaftaris. Survey: Leakage and privacy at inference time. arXiv preprint arXiv:2 2021.
Z. Ji and M. Telgarsky. Gradient descent aligns the layers of deep linear networks. In International Conference on Learning Representations, 2018.
Z. Ji and M. Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 33:17176-17186, 2020.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit datahase. ATT Labs (Online), Available: http:/yannn wtw.://ww.cc....//////////mmaa www.///m////miiiii.coooo.omp 2, 2010.
J. P. Lewis. Fast normalized cross-conrelation.. 1995. URL http://cittsseex..tt ist. psu. edu/
2062
7. Li, Y. Luo, and K. Lyu. Towards resolving the implicit bias of gradient descent for matrix factorization: Greedy low-rank learning. In International Conference on Learning Representations, 2020.
B. Liu, M. Ding, S. Shaham, W. Rahayu, Farokhi, and Z. Lin. When machine learning meets privacy: A survey and outlook. ACM Computing Surveys (CSUR), 54(2):1-36, 2021.
Y. Long, V. Bindschaedler, L.. Wang, D. Bu, X. Wang, H. Tang, C. A. Gunter, and K. Chen. Understand- ing membership inferences on well-generalized learning models. arXiv preprint 1802:1880204889 2018.
K. Lyu and J. Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint 19066v990060888 05890, 2019.
A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5188-5196. 2015.
F. A. Mejia, P. Gamble, Z. Hampel-Arias, M. Lomnitz, N. Lopatina, L. Tindall, and M. A. Barrios. Robust or private? adversarial training makes models more vulnerable to privacy attacks. arXiv preprint arXiv:1906.060 06449, 2019.

---

L. Melis, C. Song, E. De Cristofaro. and V. Shmatikov. Exploiting unintended feature leakage in collaborative learning. 2019 IEEE Symposium on Security and Privacy (SP). pages 691-706. IEEE, 2019.
S. Milli, L.. Schmidt, A. D. Dragan, and M. Hardt. Model reconstruction from model explanations. In Proceedings of the Conference on Fairness, Accountabilityy and Transparency, pages 1-9, 2019.
A. Mordvintsev, C. Olah, and M. Tyka. Inceptionism: Going deeper into neural networks. 2015.
E. Moroshko, B. E. Woodworth, S. Gunasekar, J. D. Lee, N. Srebro, and D. Soudry. Implicit bias in deep linear classification: Initialization scale training accuracy. Advances in neural information processing systems, 33:22182 15.22182-22193.3 2020.
M. S. Nacson, S. Gunasekar, J. Lee, N. Srebro, and D. Soudry. Lexicographic and depth- seeth--ensitive margins in homogeneous and non-homogeneous deep models. International Conference on Machine Learning, pages 4683-4692. PMLR, 2019.
B. Neyshabur, Bhojanapalli, D. McAllester, and N. Srebro. Exploring generalization in deep learning. In Advances in Neural Information Processing Systems, pages 5947-59555 2017.
A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29, 2016a.
A. Nguyen, J. Yosinski, and J. Clune. Multifaceted feature visualization: Uncovering the different types of features learned by each neuron in deep neural networks. arXiv preprint arXiv: arXiv: 162203366. 2016b.
A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, and J. Yosinski. Plug & play generative networks: Conditional iterative generation of images latent space. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4467-4477, 2017.
J. Oh, B. Schiele, and M. Fritz. Towards reverse-engineering black-box neural networks. In Explainable Al: Interpreting, Explaining and Visualizing Deep Learning, pages 121-144. Springer, 2019.
C. Olah, A. Mordvintsev, and L. Schubert. Feature visualization. Distill, 2(1 1),77 2017.
C. Olah, N. Cammarata, L. Schubert, G. Goh, M. Petrov, and S. Carter. Zoom in: An introduction to circuits. Distill, 2020. doi: 10.2391 w/distill..0024.0000 https://distill https//diiillppo///////cccrrooos
A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high- seshpperformanc deep learning library. Advances in neural information processing systems, 32, 2019.
N. Razin and N. Cohen. Implicit regularization in deep learning may not be explainable by norms. Advances in Neural Information Processing Systems, 2020.
D. Rolnick and K. Kording Reverse-engineering deep relu networks. International Conference on Machine Learning, pages 8178 8187. PMLR, 2020.
A. Salem, Y. Zhang, M. Humbert, P. Berrang, M. Fritz, and M. Backes. MI-leaks: Model and data independent membership inference attacks and defenses on machine learning models, arXiv preprint xrv: /806 0/246, 2018
S. Santurkar, A. Ilyas, D. Tsipras, L. Engstrom, B. Tran, and A. Madry. Image synthesis with single (robust) classifier. Advances in Neural Information Processing Systems, 32, 2019.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
L. Song and Mittal. Systematic evaluation of privacy risks of machine learning models. 30th USENIX Security Symposium (USENIX Security 21), pages 2615-2633 2021.
12

---

D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-281 2018.
C. Szegedy, W. Zaremba, I. Sutskever, Bruna, Erhan, I. Goodfellow, and Fergus. Intriguing properties of neural networks. arXiv preprini 1rXiv:1312.6199 2013.
N. Timor, G. Vardi, and O. Shamir. Implicit regularization towards rank minimization in relu networks arXiv preprint arXiv:2201.12766 2022.
Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart. Stealing machine learning models via prediction {APIs]. In 25th USENIX security symposium (USENIX Security 16), pages 601-618, 2016.
D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry. Robustness may be at odds with accuracy. arXiv preprint
2018.
G. Vardi. On the implicit bias in deep-learning algorithms. arXiv preprint arXiv:2208./259/ 2022.
G. Vardi and O. Shamir. Implicit regularization in relu networks with the square loss. In Conference on Learning Theory, pages 4224-4258. PMLR, 2021.
G. Vardi, O. Shamir, and N. Srebro. On margin maximization in linear and relu networks. arXiv preprint X 10.0222 2021.
B. Wang and N. Z. Gong. Stealing hyperparameters in machine learning. In 2018 IEEE Symposium on Security and Privacy (SP), pages 36-52. IEEE, 2018.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4);600 2004.
B. Woodworth, Gunasekar. D. Lee, Moroshko, P. Savarese, Golan, D. Soudry, and N. Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pages 0035-3673.. PMLR, 2020.
Z. Yang, Zhang, E.-C. Chang, and Z. Liang. Neural network inversion adversarial setting via background knowledge alignment. In Proceedings of the 2019 ACM SIGSAC Conference Computer and Communications Security, pages 225-240, 2019.
Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 3 computer security foundations symposium (CSF), pages 268-28 282. IEEE, 2018
H. Yin, P. Molchanov, J. M. Alvarez, Z. Li, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz. Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9715-8724 2020.
H. Yin, A. Mallya, A. Vahdat, J. M. Alvarez, J. Kautz, and P. Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16337-16346, 2021.
J. Yosinski, J. Clune, A. Nguyen, T. Fuchs, and H. Lipson. Understanding neural networks through deep visualization. arXiv preprint arXiv:1506.064 150.069906 2015.
C. Yun, S. Krishnan, and H. Mobahi. A unifying view on implicit bias in training linear neural networks. In International Conference on Learning Representations, 2020.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
Y. Zhang, R. Jia H. Pei, W. Wang, B. Li, and D. Song. The secret revealer: Generative model- inversion attacks against deep neural networks. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pottern Recognition, pages 253-261, 2020.
I. Zhu, Z. Liu, and S. Han. Deep leakage from gradients. Advances Neural Information Processing Systems, 32, 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [N/A] We rely on known theoretical results, so proofs are not required.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material as URL? [N/A] We do not have new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A] We used only publicly available assets.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A] We used only publicly available data.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "h9fJEFxx75J", "reviewer": "Reviewer_2grn", "summary": "This paper proposes a new method for reconstructing parts of the training data from a given trained homogeneous binary classification network without any further prior knowledge. It relies on a theoretical result by Lyu and Li as well as Ji and Telgarsky, who showed that the gradient flow of the training loss of such a network converges in direction to parameters that solve a maximum margin problem. The optimality condition to the latter is used to phrase an optimization problem for recovering training data on the margin. The resulting approach is tested in various contexts for a 2d toy example, a binary MNIST image classification, and a binary CIFAR-10 classification, and demonstrates significant abilities to recover some parts of the training data. ", "questions": "- The training images to be reconstructed are initialized from a zero mean Gaussian distribution, i.e. roughly half of the values being negative besides an explicit regularization term the tries to enforce a [0,1] box constraint. This seems contradictory. Is it not possible to initialize with a uniform distribution in [0,1]^d?\n- Constraints (such as lambda \\geq 0 or the abovementioned box constraint) are implemented as penalties instead of running projected gradient descent (i.e. enforcing hard constraints explicitly). Is there a specific reason?\n- To be very picky I suggest to write (5) as $\\lambda_i (y_i \\phi(\\tilde{\\theta};x_i)-1) = 0$ because strictly speaking I do not think $\\lambda_i$ has to be greater than 0 if $y_i \\phi(\\tilde{\\theta};x_i)=1$. And in line 159 I suggest not to use the word \"converge\" in \"we converge to some point $\\theta$\" because the training setting is designed such that no minimizer exists (the norm of $\\theta$ should slowly go to infinity and it can only converge in direction). ", "limitations": "Yes, there are several limitations in terms of the investigated network architectures and a number of open research questions, but the authors have adequately addressed them. ", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "To my mind this is a really nice paper, because it exploited a theoretical result to propose a completely new method for reconstructing the data a network has been trained on, which yields significantly better results than competing model inversion schemes. I agree with the authors that a significant amount of training data can be reconstructed surprisingly well. While the investigated network architectures as well as the allowed settings (requiring all training examples to be classified correctly such that the loss converges to zero) remain limited, I still believe the results to be very significant as they lead to several further questions, including why the optimization problems works well at all, how many training data points typically are on the margin, and if training data points that are not on the margin are protected from possible recovery attacks as well as possible further characterizations for stochastic gradient descent, other training methods, multi-class classification network, or regression networks. It reveals interesting properties of the trained weights which is why I consider this to be a strong paper. \n\nIn terms of weaknesses, the paper is of course limited to a specific setting, which is, however, understandable considering it is the first work in this direction. Besides this, I only have some minor points that I am phrasing as questions below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bbZp3CXgreN", "reviewer": "Reviewer_gmuC", "summary": "The authors in this paper address a challenging, and interesting problem in the current deep neural network literature. Though it is widely believed that neural networks essentially memorize, thereby overfit to training samples, it is usually not very clear how one might be able to achieve this. This paper proposes a relatively clean and straightforward formulation to compute the training data points. The authors rely on the results of [Ji and Telgarsky, 2020] to frame the problem as discussed in Equations (2) - (5). The technique hinges on the fact that at the training data points, a linear combination of the derivatives of the network parameters can predict the parameters themselves. A very interesting observation in my opinion.\n", "questions": "\nI wonder if the authors have suggestions, for a possible way to extend the current work to networks with skip connections ?\n", "limitations": "Not general enough to be applicable to Resnet networks.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths : \n\n1. Clean formulation of the problem in terms of  reconstruction loss to find the training points.\n2. The approach produces convincing results in some of the standard computer vision datasets like MNIST and CIFAR10.\n\nWeakness : \n\n1. This work cannot be extended to CNNs with skip connections  easily due to the homogeneity assumption. Which limits the test error on CIFAR10 due to the above limitation is 71%.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "P6-vzallHhk", "reviewer": "Reviewer_feXv", "summary": "The paper deals with an interesting problem. Reconstructing training data from weights. Mostly, it has been attempted to reconstruct data based on logit outputs only. The paper has far reaching impact on privacy. The paper proposes a method and demonstrates its effectivenss on layer MLPs.", "questions": "Could you add the needed evaluation (see above) or explain why the method works well in a practical setup?", "limitations": "I recommend that the author have a dedicated paragraph or subsection named \"Limitations\", where they discuss them.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The problem is highly relevant.\n+ Showing that training data can be accurately reconstructed from weights would be ground breaking and have far reaching practical implications with respect to privacy.\n+ The optimization procedure is interesting with some nice tweeks. \n- Unfortunately, and this is a key aspect, the paper falls short in evaluation. Evaluating on a practical network and a practical training setup (in terms of amount of data) is essential for the method to have far reaching impact and also be convincing. Since for simple architectures as those used in the paper trivial reconstructions (e.g. just visualizing weights) give already good results, the additional value of the method seems limited and it is also not clear whether it works at all in a practical setup - see detailed comments.\n\nDetailed comments:\n- Evaluation should be on a reasonable architecture: not just a two layer MLP, e.g., a CNN like ResNet or VGG. The reason why I am saying this, is for linear networks, it is known that weights might resemble class averages/ prototypes - the authors state this themselves (Figure 5b) - another reference showing this in a different context is, e.g., https://arxiv.org/abs/2202.08299. So it appears that 2 layer MLPs encode samples / class prototypes. The authors somewhat confirm this in Figure 5b, though admittedly their reconstructions are clearly superior to showing weights only. However, it is not clear, whether the approach would work on deep networks, i.e., is the optimization only good for fine-tuning already good prototypes? Would it fail (completely) if they did not exist? Where failure could be that only a very small fraction (I am talking 1/10^6 or so) of reconstructed images are actual training data. The authors themselves state that not only optimization attempts are successful and given that classifiers are subject to adversarial samples, the confidence in the generalizability of the findings to more realistic networks is very low. \n- Evaluation should use a larger training dataset: The authors use only 500 samples. If they used just one sample per class, it would be quite likely that it resembles a class-average, i.e. that sample. It is not obvious whether their method works if trained on 5000 or 50000 samples.\n- \"not an actual examples\" -> not actual samples\n- we use the original test sets of MNIST/CIFAR10 with 10000/8000 images respectively, and labeled accordingl -> Cifar10 has 10000 test samples, does not it?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7eCYGUJ9cPA", "reviewer": "Reviewer_fWe7", "summary": "This paper gives a principled and novel scheme for reconstructing training set image only use parameters of a network. The approach is elegant and principled and the results are convincing.", "questions": "1. Being curious, have the authors considered other network architectures? (well I feel it is natural to consider them, and maybe the author have already tried the algorithm on these networks?) If not, why?\n\n2. If a network is trained with differential privacy (there are plenty work study differentially private neural network training), how well would this approach work? Note that, it seems to me that, successful reconstruction seems to be in contradiction with DP ", "limitations": "Potential negative societal impact because the adversaries can reconstruct training data just from network parameters. But this is for good reasons because it means this phenomenon should receive attention and one should defense mechanisms", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: Principled and elegant approach, with convincing experimental results\n\nWeakness: One thing I am wondering about is other more complicated network architectures, especially those that have been used in practice today. For example, how about we apply this to CNN? What will happen? Note that even CNN should be regarded as a \"baby version\".\n\nSo to this end, I do have some concerns about the experiments regarding effectiveness of the approach on state-of-the-art neural networks", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept