Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Losses
Authors: Gon Buzaglo
Niv Haim
Gilad Yehudai
Gal Vardi
Yakir Oz
Yaniv Nikankin
michal Irani
Venue: NeurIPS 2023 poster (2023)
Keywords: memorization
data reconstruction
implicit bias
TL;DR: We extend previous reconstruction methods from trained neural networks, including reconstruction from multiclass and convolutional networks, and analyze the various factors which enable reconstructability

Abstract: Memorization of training data is an active research area, yet our understanding of the inner workings of neural networks is still in its infancy.
Recently, Haim et al. 2022 proposed a scheme to reconstruct training samples from multilayer perceptron binary classifiers, effectively demonstrating that a large portion of training samples are encoded in the parameters of such networks.
In this work, we extend their findings in several directions, including reconstruction from multiclass and convolutional neural networks. 
We derive a more general reconstruction scheme which is applicable to a wider range of loss functions such as regression losses. 
Moreover, we study the various factors that contribute to networks' susceptibility to such reconstruction schemes. 
Intriguingly, we observe that using weight decay during training increases reconstructability both in terms of quantity and quality. 
Additionally, we examine the influence of the number of neurons relative to the number of training samples on the reconstructability.
Code: https://github.com/gonbuzaglo/decoreco

Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Losses
Gon Buzaglo"!
Niv Haim" 1
Gilad Yehudai
Gal Vardi²
Yakir Oz"
Yaniv Nikankin
Michal Irani
Weizmann Institute of Science, Rehovot, Israel "TTI-Chicago and the Hebrew University of Jerusalem
Abstract
Memorization training data an active research area, yet our understanding of the inner workings of neural networks is still in its infancy. Recently, Haim et al. [2022] proposed scheme to reconstruct training samples from multilayer percep- tron binary classifiers, effectively demonstrating that large portion of training samples are encoded in the parameters of such networks. In this work, we extend their findings in several directions, including reconstruction from multiclass and convolutional neural networks. We derive a more general reconstruction scheme which is applicable to wider range of loss functions such as regression losses. Moreover, we study the various factors that contribute to networks' susceptibility such reconstruction schemes. Intriguingly, we observe that using weight decay during training increases reconstructability both in terms of quantity and quality. Additionally, we examine the influence of the number of neurons relative the number of training samples on the reconstructability. Code: https: /ttp//ggtttt//t/// com/ gonbuzaglo//ecoreeee
1 Introduction
Neural networks are known to memorize training data despite their ability to generalize well to unseen test data [Zhang al., 2021, Feldman, 2020]. This phenomenon was observed in both supervised settings [Haim et al., 2022, Balle et al., 2022, Loo et al., 2023] and in generative models [Carlini al., 2019, 2021, 2023]. These works shed interesting light on generalization, memorization and explainability of neural networks, while also posing a potential privacy risk.
Current reconstruction schemes from trained neural networks still very limited and often rely on unrealistic assumptions, or operate within restricted settings. For instance, Balle et al. [2022] propose reconstruction scheme based on the assumption of having complete knowledge of the training set, except for single sample. Loo et al. [2023] suggest a scheme which operates under the NTK regime [Jacot et al., 2018), and assumes knowledge of the full of parameters initialization. Reconstruction schemes for unsupervised settings are specifically tailored for generative models and not applicable for classifiers other supervised tasks.
Recently, Haim al. [2022] proposed reconstruction scheme from feed-forward neural networks under logistic exponential loss for binary classification tasks. Their scheme requires only knowl- edge of the trained parameters, and relies on theoretical results about the implicit bias of neural networks towards solutions of the maximum margin problem [Lyu and Li, 2019, Ji and Telgarsky, 2020]. Namely, neural networks are biased toward KKT points of the max-margin problem (see Theorem 3.1). By utilizing the set of conditions that KKT points satisfy, they devise novel loss func- tion that allows for reconstruction of actual training samples. They demonstrate reconstruction from models trained common image datasets (CIFAR10 [Krizhevsky al., 2009] and MNIST [LeCun et al., 2010]..
"Equal Contribution
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

In this work, we expand the scope of neural networks for which we have evidence of successful sample memorization, by demonstrating sample reconstruction. Our contributions are as follows:
We extend the reconstruction scheme of Haim et al. [2022] multiclass setting (Fig. 1). This extension utilizes the implicit bias result from Lyu and Li [2019] to multiclass training. We analyse the effects of the number of classes reconstructability. and show that models become more susceptible to sample reconstruction as the number of classes increases.
We devise reconstruction scheme that applies for general loss functions, assuming that the model is trained with weight decay. We demonstrate reconstruction from models trained on regression losses.
We investigate the effects of weight decay and show that for centain values, weight decay increases the vulnerability of models to sample reconstruction. Specifically, it allows us to reconstruct training samples from convolutional network, while Haim et al. [2022] only handled MLPs.
We analyse the intricate relation between the number of samples and the number of pa- rameters in the trained model, and their effect on reconstretability. We also demonstrate successful reconstruction from a model trained on 5,000 samples, surpassing previous results that focused on models trained on up to 1,000 samples.
2 Related Works
Memorization and Samples Reconstruction. There is no consensus on the definition of the term "memorization" and different works study this from different perspectives. In ML theory, memorization usually refers to label (or, model's 's output) memorization [Zhang et al., 2016, Arpit et al., 2017, Feldman, 2020, Feldman and Zhang, 2020, Brown et al., 2021]. namely, fitting the training set. Memorization in the inpul domain is harder to show, because in order to demonstrate its occurrence one has reconstruct samples from the model. Balle al. [2022] demonstrated reconstruction of one training sample, assuming knowledge of all other training samples and Haim al. [2022] demonstrated reconstruction of a substantial portion of training samples from neural network classifier. Loo et al. [2023] extend their work to networks trained under the NTK regime [Jacot al., 2018] and explore the relationship dataset distillation. Several works have also studied memorization and samples reconstruction in generative models like autoencoders [Erhan al., 2009, Radhakrishnan et al., 2018], large language models [Carlini et al., 2021, 2019, 2022] and diffusion- based image generators [Carlini et al., 2023, Somepalli et al., 2022, Gandikota et al., 2023, Kumari et al., 2023].
Inverting Classifiers. Optimizing model's input as to minimize a class score is the common approach for neural network visualization [Mahendran and Vedaldi, 2015]. It usually involves using input regularization [Mordvintsev et al., 2015, Ghiasi et al., 2022], GAN prior [Nguyen et al., 2016, 2017] or knowledge of batch statistics [Yin al., 2020]. Fredríkson al. [2015], Yang et al. [2019] showed reconstruction of training samples using similar approach, however these methods are limited to classifiers trained with only a few samples per class. Reconstruction from federated- federated-learning setup [Zhu et al., 2019, He et al., 2019, Hitaj et al., 2017, Geiping et al., 2020, Yin et al., 2021, Huang al., 2021, Wen et al., 2022] involve attacks that assume knowledge of training samples" gradients (see also Wang et al. [2023] for theoretical analysis). In this work we do not assume any knowledge on the training data and do not use any priors other than assuming bounded inputs.
Preliminaries
In this section, we provide an overview of the fundamental concepts and techniques required to understand the remainder of the paper, focusing on the fundamentals laid by Haim et al. [2022] for reconstructing training samples from trained neural networks.
Theoretical Framework. Haim et al. [2022] builds on the theory of implicit bias of gradient descent. Neural networks are commonly trained using gradient methods, and when large enough, they are expected fit the training data well. However, is empirically known that these models converge to solutions that also generalize well unseen data, despite the risk of overfitting. Several

---

works pointed to this "implicir bias" of gradient methods as a possible explanation. Soudry et al. [2018] showed that linear classifiers trained with gradient descent on the logistic loss converge the same solution as that of a hard-SVM, meaning that they maximize the margins. This result was later extended to non-linear and homogeneous neural networks by Lyu and Li [2019]. Ji and Telgarsky [2020]:
Theorem 3.1 (Paraphrased from Lyu and Li [2019), Ji and Telgarsky [2020]) Let (0;-)) be homogeneous 2 ReLU neural network. Consider minimizing the logistic loss over a binary clas- sification dataset v:)] using gradient flow. Assume that there exists time to where the network classifies all the samples correctly. Then, gradient flow converges in direction to a first order stationary point (KKT point) of the following maocimum-marghe problem:
min 0
1101
s.t.
[ y (0;xx, 21.
(1)
2
A KKT point of Eq. (1) is characterized by the following set of conditions:
Vj (p),
0
Vo, [,)](0x,,]
(stationarity)
(2)
Vi e [n],
y (. 21
(primal feasibility) (3)
Vi E [n],
x; 2 0
(dual feasibility)
(4)
Vi e [n], = 0 if y. (0:x,) #
(complementary slackness)
(5)
Reconstruction Algorithm. Haim et al. [2022] demonstrated reconstructing samples from the training set of such classifiers by devising a reconstruction loss. Given a trained classifier (xx 0). they initialize set of {(x, y:)] and {> and optimize x,, A to minimize the following loss
function:
L= 1=10-
1
A Va, (v;
Σ max {-), -Amin } + Laprior 1=1
(6)
1=1
Lxpaiooary
Lx
Where Lprior is simply bounding each pixel value at [-1, The number of training samples n is unknown. However, setting 2n, where }yi} are set in a balanced manner allows reconstructing samples with any label distribution. The x, 's are initialized from the Gaussian distribution A o I),,,, and Amin: da are hyperparameters. We note that the homogeneity condition from Theorem 3.1 is not necessarily a practical limitation of this reconstruction scheme, as already in Haim et al. [2022] they show reconstructions from non non-homogeneous network.
Analysing and Summarizing Results. The optimization in Eq. (6) is executed k times (for different hyperparameters) and results in km outputs ((x,)/=)) that we term candidates, as they are candidates to be reconstructed training samples. To quantify the success of the reconstruction process, each training sample is matched with its nearest- neighbour from the km candidates. The "quality" of reconstruction is then measured using SSIM Wang et al. [2004] (see full details in Appendix A.
An important corollary of the set of KKT conditions Eqs. (2) to (5) is that the parameters of the trained model only depend on gradients of samples that are closest to the decision boundary, the so-called "margin-samples" (see end of section 3.2 in Haim al. [2022]). Therefore, good visual summary for analysing reconstruction from a trained model is by plotting the reconstruction quality (SSIM) against the distance from the decision boundary (x:)). We also utilize such visualizations.
Assessing the Quality of Reconstructed Samples. Determining whether candidate is a correct match for some training sample is as hard as finding good image similarity metric. No synthetic metric such as SSIM or 1.2 norm can perfectly align with human perception. Perceptual similarity
A classifier is homogeneous w.r.t if there exists E I2 s.t. E R, : ((x:c0) o 6(x; Formally: Lpriur D Ed max/max/ x 1, 0), max Xi, 1,0)}.
3

---

metrics (e.g.. LPIPS [Zhang et al., 2018]) build on top of pre-trained classifiers trained on Ima- genet [Deng al., 2009], and are effective for the image resolution in this work (up to 32x32 pixels). We have observed heuristically that candidates with SSIM score higher than about 0.4 are indeed visually similar to their nearest neighbor training sample. Hence, this work we say that certain candidate is "good reconstruction" if its SSIM score with its nearest neighbor is at least 0.4. Also see discussion Appendix A.2.
4 Reconstructing Data from Multi-Class Classifiers
We demonstrate that training set reconstruction can be extended to multi-class classification tasks.
Plane
Car
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Figure I: Reconstructed training samples from multi-class MLP classifier that was trained on 500 CIFARIO images. Each column corresponds to one class and shows the 10 training samples (red) that were best reconstructed from this class, along with their reconstructed result (blue).
4.1 Theory
The extension the implicit bias homogeneous neural networks to the multi-class settings is discussed in Lyu and Li [2019] (Appendix G): Let 5 = ((x;;;))) C Rd x [C] be multi-class classification training set where
N any number of classes, and [C] = (1...., C). Let ((;-)) Rd FC be homogeneous neural network parameterized by 0 E R We denote the j-th output of t an input as $ (0: x) R. Consider minimizing the standard cross-entropy loss and assume that after some number of iterations the model correctly classifies all the training examples. Then, gradient flow will converge to KKT point of the following maximum-margin problem:
1
min 2
|
s.t.
-,, 2 e E [C]
-
KKT point of the above optimization problem characterized by the following set of conditions:
N Vo (0;x,) (0;x:)) -0
(8)
Vi E [n], vij E [C]
[v:} 1y. (0:x;)
21
(9)
Vi [n), V/ [C]
Arj 20
(10)
Vi [m] AA E [C] {yr) A = 0 2 0 x, (0:x,) #1
(11)
straightforward extension reconstruction loss a multi-class model that converged to the conditions above would be to minimize the norm of the left-hand-side (LHS) of condition Eq. (8)

---

(namely, optimize over {x and {λt, Jej]) where m is a hyperparameter). However, this straightforward extension failed successfully reconstruct samples. We therefore propose the following equivalent formulation.
the distance of sample x; its nearest decision boundary, by. a 3j. is usually achieved for a single class j and therefore (from Eq. (11)) in this case at most one Ai, will be non-zero. For some samples x; it also possible that all X will vanish. Following this observation, we define the following loss that only considers the distance from the decision boundary:
Note that from Eqs. (9) and (11), most A
Immulticlass (X1: my Xm, X1, A Am)
A Volo (x;;
max (x 0)] JØ
(12)
Eq. (12) implicitly includes Eq. (11) into the summation in Eq. (8), thereby significantly reducing the number summands and simplifying the overall optimization problem.
While the straightforward extension failed to successfully reconstruct samples, solving Eq. (12) enabled reconstruction from multiclass models (see Fig. and results below). We attribute this success to the large reduction in the number of optimization variables, which simplifies the overall optimization problem (n variables Eq. (12) compared to n variables in the straightforward case, which is significant for large number of classes C).
We also use the same LA and Lipior as in Eq. (6), and set {w:} in a balanced manner (uniformly on all classes). While setting m = C n allows reconstructing any label distribution, in our experiments we focus on models trained balanced training sets, and use m 2n which works sufficiently well. An intuitive way to understand the extension of the binary reconstruction loss Eq. (6) to multi-class reconstruction Eq. (12) that the only difference is the definition of the distance to nearest boundary, which is the term inside the square brackets in both equations.
4.2 Results
We compare between reconstruction from binary classifiers, studied in Haim et al. [2022], and reconstruction from multi-class classifiers by using the novel loss function Eq. (12). We conduct the following experiment: we train an MLP classifier with architecture D-1000-1000-C on samples from the CIFARIO 10 (Krizhevsky et al., 2009] dataset. The model is trained to minimize the cross-entropy loss with full-batch gradient descent, once with two classes (250 samples per class) and once for the full 10 classes (50 samples per class). Both models train on the same amount of samples (500). The test set accuracy the models is 77%/32% respectively, which is far from random (50%/10% resp.). See implementation details in Appendix B.
To quantify the quality of reconstructed samples, for each sample in the original training set we search for its nearest neighbour in the reconstructed images and measure the similarity using SSIM [Wang al., 2004] (higher SSIM means better reconstruction). In Fig. we plot the quality of reconstruction (in terms of SSIM) against the distance of the sample from the decision boundary oy, (x,; 0) max (x,; 0). As seen, multi-class classifier yields much more samples that vulnerable to being reconstructed.
Multiclass
Binary
SSMOx.R 88
5-3005x 31
0.4
0.2
0.0
10
15
20
25
35
40
10
15
20
25
30
25
40
Figure 2: Multi-class classifiers are more vulnerable to training-set reconstruction. For a training set size 500, multi-class model (left) yields 101 reconstructed samples with good quality (SSIM>0..4. compared to 40 in a binary classification model (righit).
5

---

We examine the relation between the ability to reconstruct from a model and the number of classes on which it was trained. Comparing between two models trained on different number of classes is not immediately clear, since we want to isolate the effect of the number of classes from the size of the dataset (it was observed by Haim et al. [2022] that the number of reconstructed samples decreases as the total size of the training set increases). We therefore train models on training sets with varying number of classes (C E {2, 3, 4, 5, 10)) and varying number of samples per class (1,5, 10, 50). The results are visualized in Fig. 3a. As seen, for models with same number of samples per class, the ability to reconstruct increases with the number of classes, even though the total size of the training set is larger. This further validates hypothesis that the more classes, the more samples are vulnerable to reconstruction (also see Appendix C).
Another way to validate this hypothesis is by showing the dependency between the number of classes and the number of "good" reconstructions (SSIM>0.4) shown in Fig. 3b. can be seen, training on multiple classes yields more samples that are vulnerable to reconstruction. An intuitive explanation, is that multi-class classifiers have more more vulnerable reconstruction, this results in more samples being reconstructed from the model.
Since margin-samples are
Classes
50 Samples
per Class
Reconetruction Quality [SSIAIX
10 Samples per Class
0.20
Samples
per Class
Sample
per Closs
Distance from Decision Boundary 16, -max4/00
(a) Analysing the relation between number of classes and number of samples per class in reconstruction from multiclass Classifiers.
Samples per Class:
10
50
Good Bec's (Tatal)
# Goad Rec: per Cless
2
10
10
Classes
Classes
(b) Number of "good" reconstructions increases with number of classes and the samples per class
Figure 3: Evaluating the effect of multiple classes the ability to reconstruct. We show reconstruc- tions from models trained with different numbers of classes and different numbers of samples per class. As seen, multiple classes result in more reconstructed samples.
5 Data Reconstruction with General Loss Functions
We demonstrate that data reconstruction can be generalized to a larger family of loss functions. Haim al. [2022] and Section only considered a reconstruction scheme based on the implicit bias of gradient methods trained with cross-entropy loss. For other loss functions, such as the square loss, a precise characterization of the implicit bias in nonlinear networks does not exist [Vardi and Shamir, 2021]. Hence, we establish reconstruction scheme for networks trained with explicit regularization, i... with weight decay. We show that as long as the training involves a weight-decay term, we can derive a reconstruction objective that is very similar to the previous objectives in Eqs. (6) and (12).
5.1 Theory
Let ((;(x;; 0), yi) be a loss function that gets as input the predicted output the model (parametrized by 0) on an input sample x,, and its corresponding label ys The total regularized loss:

---

C Σ (xx; 0), + A 0
(13)
Assuming convergence (VoL 0), the parameters should satisfy the following
0
E VoL(x;;0 0)
(14)
where f = As (0(xx,0) This form (which is similar to the condition in Eq. (2)), allows us to define a generalized reconstruction loss for models trained with a weight-decay term:
Live Lire(X1,.... Xys AI, A Am) 110 [x
(15)
As before, we also include the same Pprior as in Section 3. It is straightforward to see that Lrec is generalization of the reconstruction loss in Eq. (6) (yı could be incorporated into the X term).
5.2
Results and Analysis
2255
Huber
BCE (W/ WD)
BCE (No WD)
1.00
50
0.50
Samples
Roconstrution rccontruttoon Qualit (SSIMx 80
0.00
1.00
0.15
20
04
100
100
e.se
Samples
2.00
0.00
10
1.5
100
200
300
400
0.
16
300
0.s0
Samples
025
0.00
500 Samples
10-5
10-5
Loss value (1)
10-5
10-5
16-5
Figure 4: Reconstruction from general losses (column) for various training set sizes (row), us-
ing Eq. (15). "Harder" samples (with higher loss) are easier to reconstruct.
We validate the above theoretical analysis by demonstrating reconstruction from models trained on other losses than the ones shown in Section 4 and Haim et al. [2022]. We use the same dataset as in the classification tasks images from CIFARIO dataset with binary labels {-1, The only difference is replacing the classification binary cross-entropy loss with regression losses (e.g.. MSE).
In classification tasks, we analyzed the results by plotting the reconstruction quality (SSIM) against the sample's s distance from the decision boundary (see Section 3). This showed that reconstruction only feasible for margin-samplle
However, in regression tasks, margin and decision boundary lack specific meaning. We propose alternative analysis approach note that smaller distance from the margin results in higher loss for binary cross-entropy. Intuitively, challenging to classify (as reflected by the loss function). Therefore, for regression tasks, we analyze the results by plotting the reconstruction quality against the loss (per training sample).
sargin-samples are the most
In Fig. 4 we show results for reconstructions from models trained with MSE, L2.5 loss (L (x; 0) - y for 2=2.2.5 respectively) and Huber loss [Huber, 1992]. The reconstruction scheme in Eq. (15) is the same for all cases, and is invariant to the loss used during training. Fig. 4 highlights two important observations: first, the reconstruction scheme Eq. (15) succeeds in reconstructing large portions of the training set from models trained with regression losses, noted from the high
7

---

quality (SSIM) of the samples. Second, by plotting quality against the loss, one sees that "challenging" samples (with high loss) are easier reconstruct. Also note that the analysis works for classification losses, namely BCE with or without weight-decay in Fig. 4). For more results see Appendix D.
6 On the Different Factors that Affect Reconstructability
Our goal is to gain a deeper understanding of the factors behind models' vulnerability to reconstruction schemes. In this section, we present several analyses that shed light on several important factors.
6.1 The Role of Weight Decay in Data Reconstruction
Haim al. [2022] assumed MLP models whose first fully-connected layer was initialized with small (non-standard) weights. Models with standard initialization (e.g., He et al. [2015], Glorot and Bengio [2010]) did not yield reconstructed samples. In contrast, the MLPs reconstructed Haim al. [2022] were initialized with an extremely small variance in the first layer. Set to better understand this drawback, we observed that incorporating weight-decay during training, not only enabled samples reconstruction in models with standard initialization, but often increase the reconstructability of training samples.
MLPP (2 Classes, 50 Samples per Class)
CNN 12 Classes, 250 Samples per Class)
80
MLP (30 Classes 50 Samples per Class)
Good Recoratrucion
Geps Reconsinucton
Goodd Reconstruction fout of
1000
(out of tota 100)
Tiatt lotal 5000
100
75
inatinationn (farm & al, 2022
20
Avn n Small inteiiaation S0
55
0- Aso O, Stardard initializaiin
Avo Stanidard in cnlittioo
-0. 5mmll Intalizaiion...
10a
16 10 10
10
10-h
10
Weight Dacay Value (Aga)
Weight-Decay Value (oxs)
Weight Decay Value Daul
(a)
(b)
(c)
Figure Using weight-decay during training increases vulnerability sample reconstruction
In Fig. 5a-b we show the number of good reconstructions for different choices of the value of the weight decay (AwD), for MLP classifiers trained on C=2,10 classes and 50 samples per class (Fig. 5 a, b resp.). We add two baselines trained without weight-decay: model trained with standard initialization (black) and model with maal-inittialized first-layer (red). See how for some values of weight-decay, the reconstructability is significantly higher than what was observed for models with non-standard initialization. By examining the training samples' distance the boundary, one observes that using weight-decay results in more margin-samples which are empirically more vulnerable to reconstruction (see full details in Appendix E).
We now give an intuitive theoretical explanation to the role of weight decay in data reconstruction. Theorem 3.1 is used to devise the reconstruction loss in Eq. (6), which is based on the directional convergence to a KKT point of the max-margin problem. However, this directional convergence occurs asymptotically as the time f + oc, and the rate of convergence in practice might be extremely slow. Hence, even when training for, e.g., 10 iterations, gradient descent might reach a solution which is still too far from the KKT point, and therefore reconstruction might fail. Thus, even when training until the gradient of the empirical loss is extremely small, the direction of the network's s parameters might be far from the direction of a KKT point. In Moroshko al. [2020], the authors proved that in diagonal linear networks (i.e., a certain simplified architecture of deep linear networks) the initialization scale controls the rate of convergence to the KKT point, namely, when the initialization is small gradient flow converges much faster to a KKT point. A similar phenomenon seems to occur also in our empirical results: when training without weight decay, small initialization seems to be required to allow reconstruction. However, when training with weight decay, our theoretical analysis in Section 5.1 explains why small initialization is no longer required. Here, the reconstruction does not rely on converging to a KKT point of the max-magin problem, but relies on Eq. (14) which holds (approximately) whenever we reach a sufficiently small gradient of the training objective. Thus, when training with weight decay and reaching a small gradient Eq. (14) holds, which allows for reconstruction, contrary to training without weight decay where reaching a small gradient does not imply converging close to KKT point.
8

---

Figure 6: Reconstruction from CNN. Training samples (red) and their best reconstructions (blue)
Reconstruction from Convolutional Neural Networks (CNNs). CNNs adhere to the assumptions of Theorem 3.1, yet Haim et al. [2022] failed apply their reconstruction scheme Eq. (6) to CNNs. We observe that incorporating weight-decay during training (using standard initialization) enables samples reconstruction. In Fig. 6 we show an example for reconstruction from binary classifier whose first layer is a Conv-layer with kernel size 3 and 32 output channels, followed by two fully connected layers
The weight-decay term is Awp=0.001 (the training setup is similar to that of MLP). In Fig. Se we show the reconstructabiiity for the same CNN model trained with other values of ÄWD. Note how the weight-decay term plays similar role in the CNN as in the MLP case. See full details in Appendix F.
6.2 The Effect of the Number of Parameters and Samples on Reconstructability
Haim et al. [2022] observed that models trained on fewer samples are more susceptible to reconstruc- tion terms of both quantity and quality. In this section, we delve deeper into this phenomenon, focusing on the intricate relationship between the number of parameters in the trained model and the number of training samples. We conduct the following experiment:
We train 3-layer MLPs with architecture D-W-W-1 on N training samples from binary CIFARIO (animals vs. vehicles), where W E (5, 10, 50, 100, 500, 1000} and N E {10, 50, 100, 300, 500). We conduct the experiment for both classification and regression losses, with BCE and MSE loss respectively. Generalization error is 23%-31% for BCE (classification) and 0.69-0.88 for MSE (regression), compared to 50%/0.97 for similar models with random weights.
We reconstruct each model using Eq. (15) and record the number of good reconstructions. The results are shown in Fig. 7. Note that as W/n increases, our reconstruction scheme is capable of reconstructing more samples, and vice versa. For example, consider the case when N=10. To successfully reconstruct the entire training set, it is sufficient for W to be greater than 50/10 (for MSE/BCE). However, when N=500, even larger models (with larger W) can only reconstruct up 8% of the samples.
Lastly, we reconstruct from model with ==10,000, trained on ==5,000 samples (5 times larger than any previous model). While there is some degradation in the quality of the reconstructions compared to models trained on fewer samples, it is evident that our scheme still reconstruct some of the training samples. For full results see Appendix G.
7 Conclusions
We present improved reconstruction methods and conduct a comprehensive analysis the recon- struction method proposed by Haim et al. [2022]. Particularly, we extend their reconstruction scheme to multi-class setting and devise a novel reconstruction scheme for general loss functions, allowing reconstruction regression setting (e.g.. MSE loss). We examine various factors influeneing reconstructability. We shed light on the role of weight decay in samples memorization, allowing for sample reconstruction from convolutional neural networks. Lastly, we examine the intricate relationship between the number of parameters, the number of samples, and the vulnerability of the model to reconstruction schemes. We acknowledge that our reconstruction method raises concerns regarding privacy. We consider it crucial to present such methodologies as they encourage researchers to study the potential hazards associated with training neural networks. Additionally, allows for the development of protective measures aimed at preventing the leakage of sensitive information.

---

Good Reconstructions (MSE Loss)
Reconstructions (BCE Loss)
100
100
1000
10
34 (68.0%)
48 (48. (48.0%)
35
17
26
54 (18.0%)
49
(100.0%)
(11.7%)
(3.4%)
(1.0.00)
152
(9.8%)
W #NNeuron per layer
1000 500
10
31
43
31
80
W #Neurons per layer
10
27 (54.0%)
38
43
36
(100.00%
(62.0%)
(49. 0%6
(10.3%)
(3.0%)
(100.0%)
(38.0%)
(14.3%)
(7.2%)
10
20
10
23 (46.0%)
28
27
21
(1.00.0%)
162.00 (62.0%)
133. 096
(6.7%)
[0.8
(100.0%)
(28.0%)
(9.0%)
(4.2%)
50
(1.0000%)
10
(50.0%)
(29.0%6)
(1. (1.7%)
[0.4%]
(100.00%)
10
22 (44.0%)
(23.0%)
23
(3.0%)
12 (2.4%)
40
10
(70.0%)
(20.0%)
(2.0%)
(0. (0.0%)
12
(100.00%)
10
9 (16.0%)
(5.0%)
(0.3%)
0 (0.0%)
20
4 (40.0%)
0
0 (0.0%)
(6.0%)
(0.0%)
(0.0%)
(0.0%)
(40.0%)
(8.0%)
(3.0%)
(0.3%)
10
100
300
#Samples
N # Samples
Figure Effect of the number of neurons and number of training samples on reconstructability. We train 3-layer MLPs with architecture D-W-W- on N training samples from binary CIFARIO (animals vs. vehicles), using MSE (left) BCE (right) loss. At each cell we report the number of good reconstructions (SSIM>0.4), in both absolute numbers and as percentage relative to N.
Limitations. While we have relaxed several of the previous assumptions presented in Haim et al. [2022], our method still exhibits certain limitations. First, we only consider relatively small-scale models: up to several layers, without residual connections, that have trained for many iterations on a relatively small dataset without augmentations. Second, determining the optimal hyperparameters for the reconstruction scheme poses challenge requires exploring many configurations for even single model.
Future work. All of the above extend our knowledge and understanding of how memorization works in certain neural networks. This opens up several possibilities for future research including extending our reconstruction scheme to practical models (e.g., ResNets), exploring reconstruction from models trained larger datasets different data types (e.g., text, time-series, tabular data), analyzing the impact of optimization methods and architectural choices on reconstructability. and developing privacy schemes protect vulnerable samples from reconstruction attacks.
Acknowledgements. This project received funding from the European Research Council (ERC) under the European Union's i Horizon 2020 research and innovation programme (grant agreement No 788535), and ERC grant 754705, and from the D. Dan and Betty Kahn Foundation. GV acknowledges the support of the NSF and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning.
References
D. Arpit, Jastrzębski, N. Ballas, D. Krueger, Bengio, M. S. Kanwal, Maharaj, A. Fischer, A. Courville, Y. Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233 242. PMLR, 2017.
B. Balle, G. Cherubin, and J. Hayes. Reconstructing training data with informed adversaries. arXiv preprint arXiv:2201. 04845, 2022.
L. Biewald. Experiment tracking with weights and biases, 2020. URL https://wwi wandb. com/. Software available from wandb.com.
G. Brown, M. Bun, V. Feldman, A. Smith, and K. Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Annual ACM SIGACT Symposium Theory of Computing, pages 123-132, 2021.
N. Carlini, C. Liu, Ó. Erlingsson, J. Kos, and D. Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019.
10

---

N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert- Voss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650, 2021.
N. Carlini, D. Ippolito, M. Jagielski, K. Lee, Tramer, and C. Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.0776000 2022.
N. Carlini, J. Hayes, M. Nast, M. Jagielski, V. Sehwag, F. Tramèr, B. Balle, D. Ippolito, and E. Wallace. Extracting training data from diffusion models. arXiv preprint Xiiv:2331..3188 2023.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Icee, 2009.
D. Erhan, Bengio, A. Courville, and P. Vincent. Visualizing higher-layer features of a deep network. University of Montreal, 1341(3):1, 2009.
V. Feldman. Does learning require memorization? a short tale about a long tail. Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954 959, 2020.
V. Feldman and C. Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 15:2881-2289, 2020.
M. Fredrikson, S. Jha, and T. Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322 1333, 2015.
R. Gandikota, J. Materzyúska, J. FFotto-Kaufman, and D. Bau. Erasing concepts from diffusion models. arXiv preprint urXhv:2303.07345 2023.
Geiping, H. Bauermeister, H. Dröge, and M. Moeller. Inverting gradients-hor easy it to break privacy in federated learning? Advances in Neural Information Processing Systems, 33: 16937-16947, 2020.
A. Ghiasi, H. Kazemi, S. Reich, C. Zhu, M. Goldblum, and T. Goldstein. Plug-in inversion: Model- agnostic inversion for vision with data augmentations. 2022.
X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
N. Haim, G. Vardi, G. Yehudai, O. Shamir, and M. Irani. Reconstructing training data from trained neural networks. NeurIPS, 2022.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level per- formance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026-1034, 2015.
Z. He, T. Zhang, and B. Lee. Model inversion attacks against collaborative inference. In Proceedings of the 35th Annual Computer Security Applications Conference, pages 148-162, 2019.
B. Hitaj. G. Ateniese, and F. Perez-Cruz. Deep models under the gan: information leakage from collaborative deep learning. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, pages 603-618, 2017.
Y. Huang, S. Gupta, Z. Song, K. Li, and S. Arora. Evaluating gradient inversion attacks and defenses in federated learning. Advances in Neural Information Processing Systems, 34:7232-7241, 2021.
J. Huber. Robust estimation of a location parameter. Breakthroughs in statistics: Methodology and distribution, pages 992-518, 1992.
A. Jacot, Gabriel, and C. Hongler. Neural tangent kernel: networks. Advances in neural information processing systems, 31, 2018
Convergence and generalization neural
11

---

Z. Ji and M. Telgarsky. Directional convergence and alignment in deep learning. Advances in Neural Information Processing Systems, 33:17176-17186, 2020.
A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
N. Kumari, B. Zhang, S.-Y. Wang, E. Shechtman, R. Zhang, and J.-Y. Zhu, Ablating concepts in text-to-image diffusion models. Arxiv, 2023.
Available: Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs Availall: 2, 2010.
N. Loo, R. Hasani, M. Lechner, and D. Rus. Dataset distillation fixes dataset reconstruction attacks. arXiv preprint arXiv:2302.01425 2023.
K. Lyu and Li. Gradient descent maximizes the margin of homogeneous neural networks. arXiv preprint arXiv:1906.008:
2019.
A. Mahendran and A. Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages $188-5196, 2015.
A. Mordvintsev, C. Olah, and M. Tyka. Inceptionism: Going deeper into neural networks. 2015.
E. Moroshko, B. E. Woodworth, S. Gunasekar, J. D. Lee, N. Srebro, and D. Soudry. Implicit bias in deep linear classification: Initialization scale training accuracy. Advances in neural information processing systems, 13:22182-221992 2020.
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. 2011.
A. Nguyen, A. Dosovitskiy, J. Yosinski, T. Brox, and J. Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. Advances in neural information processing systems, 29, 2016.
A. Nguyen, J. Clune, Y. Bengio, A. Dosovitskiy, and J. Yosinski. Plug & play generative networks: Conditional iterative generation of images latent space. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4467-4477, 2017.
A. Radhakrishnan, K. Yang. M. Belkin, and C. Uhler. Memorization in overparameteriized autoen- coders. arXiv preprint arXiv: 10333, 2018.
G. Somepalli, V. Singla, M. Goldblum, J. Geiping, and T. Goldstein. Diffusion art or digital forgery? investigating data replication in diffusion models. arXiv preprint arXiv:2212.03860 2022.
D. Soudry, E. Hoffer, M. S. Nacson, S. Gunasekar, and N. Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-2878, 2018.
G. Vardi and O. Shamir. Implicit regularization in relu networks with the square loss. In Conference on Learning Theory, pages 4224 4258. PMLR, 2021.
Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
Z. Wang, J. Lee, and Q Lei. Reconstructing training data from model gradient, provably. In International Conference on Artificial Intelligence and Statistics, pages 6595-6612. PMLR, 2023.
Y. Wen, J. Geiping, L. Fowl, M. Goldblum, and T. Goldstein. Fishing for user data in large-batel federated learning via gradient magnification. arXiv preprint 2tti:2220000088000 2022.
Z. Yang, J. Zhang, E.-C. Chang, and Z. Liang. Neural network inversion in adversarial setting via background knowledge alignment. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 225-240, 2019.
H. Yin, P. Molchanov, J. M. Alvarez, 7. Li, A. Mallya, D. Hoiem, N. K. Jha, and J. Kautz, Dreaming to distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 8715-8724, 2020.
12

---

H. Yin, A. Mallya, A. Vahdat, J. M. Alvarez, J. Kautz, and P. Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16337-16346, 2021.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/161 2016. URL http::/arxxii wrg./bs////.ii. 03530.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018
Zhu, Liu, and Han. Deep leakage from gradients. Advances Neural Information Processing Systems, 32, 2019.
13

---

A Analyzing and Visualizing the Results of the Reconstruction Optimization
The analysis of the results of the various reconstruction losses Eqs. (6), (12) and (15), involve verifying and checking which of the training samples were reconstructed. In this section we provide further details on our method for analyzing the reconstruction results, and how we measure the quality of our reconstructions.
A.1 Analyzing the Results of the Reconstruction Optimization
In order to match between samples from the training set and the outputs of the reconstruction algorithm (the so-called "candidates") we follow the same protocol of Haim et al. [2022]. Note that before training our models, we subtract the mean image from the given training set. Therefore the training samples d-dimensional objects where each entry is in [-1,1].
First, for each training sample we compute the distance to all the candidates using a normalized L score:
y(x,y)
x
Fx
y
My
(16)
Ox 8y
Where x, y E Rd are a training sample or an output candidate from the reconstruction algorithm, Mx + x(i) is the mean of x and dx V - E (x(i) 2xx² is the standard deviation of x (and the same goes for y: Myy, Ty).
Second, for each training sample, we take C candidates with the smallest distance according to Eq. (16). C' is determined by finding the first candidate whose distance is larger than B times the distance to the closest nearest neighbour (where B is hyperparameter). Namely, for training sample x, the nearest neighbour is y1 with distance d(x, y1). then C is determined by finding candidate Yc+1 whose distance is d(x. yC+1) > B d(x. y1). and for all j S C, d(x, y,) < B d(x. V1). B was chosen heuristically to be B 1.1 for MLPs, and B 1.5 for convolutional models. The C candidates are then summed create the reconstructed sample x = a ش E In general, we can also take only C candidate, namely just one nearest neighbour per training sample, but choosing more candidates improve the visual quality of the reconstructed samples.
Third, the reconstructed sample x is scaled an image in [0, 1] by adding the training mean and linearly "stretching" the minimal and maximal values of the result to [0, 1]. Finally, we compute the SSIM between the training sample and the reconstructed sample & to measure the quality of reconstruction.
A.2 Deciding whether Reconstruction is "Good"
Here we justify our selection for SSIM-0.4 as the threshold for what we consider as a "good" reconstruction. In general, the problem of deciding whether reconstruction is the correct match to a given sample, or whether reconstruction is "good" reconstruction is equivalent the problem of comparing between images. No "synthetic" metric (like SSIM, 12 etc.) will be aligned with human perception. common metrie for this purpose is LPIPS [Zhang et al., 2018] that uses a classifier trained on Imagenet [Deng et al., 2009], but since CIFAR images are much smaller than Imagenet images (32 x 32 vs. 224 x 224) it is not clear that this metric will be better than SSIM
As simple rule of thumb, we use SSIM>0.4 for deciding that a given reconstruction is "good". To justify, we plot the best reconstructions (in terms of SSIM) in Fig. 8. Note that almost all samples with SSIM>-0.4 are also visually similar (for human). Also note that some of the samples with SSIM<0.44 are visually similar, so in this sense we are "missing" some good reconstructions. In general, determining whether a candidate output of a reconstruction algorithm is a match to training sample is an open question and a problem in all other works for data reconstruction, see for example Carlini et al. [2023] that derived heuristic for reconstructed samples from generative model. This cannot be dealt in the scope of this paper, and is an interesting future direction for our work.
14

---

0.63
0.62
0.62
0.55
0.55
0.53
0.5
0.5
0:48
0.48
0.48
0.47
0.47
0.47
0.47
0.46
0.46
0.46
0.46
0.45
0.45
0.45
0.45
0.45
0.44
0.44
0.44
0.44
0.44
0.43
0.42
0.41
0.41
0.41
0.4
0.4
0.4
0.4
0.4
0.4
0.39
0.39
0.39
0.36
1.38
0.36
0.38
0.38
0.38
0.
0.38
0.36
0.38
0.38
0.37
0.37
0.37
0.37
0.36
0.36
0.36
0.36
0.36
0.36
0.35
0.35
0.35
0.35
0
0.35
0.35
0.35
0.35
0.34
0.34
0.34
0.34
0.34
Figure 8: Justifying the threshold of SSIM= 0.4 as good muleoof-thumi for a threshold for a "good" reconstruction. The SSIM values are shown above each train-reconstruuctinn pair. Note that samples with SSIM> 0.4 (blue) are visually similar. Also some of the samples with SSIM< 0.4 (red) are similar. In general deciding whether a reconstruction is "good" is open question beyond the scope of this paper.
B Implementation Details
Further Training Details. The models that were reconstrueted in the main part of the paper were trained with learning rates of 0.01 for binary classifiers (both MIP and convolutional), and 0.5 in the case of multi-class classifier (Section 4). The models were trained with full batch gradient descent for 10 epochs, to guarantee convergence to a KKT point of Eq. (1) or a local minima of Eq. (13). When small initialization of the first layer is used (e.g., in Figs. 2 and 3), the weights are initialized with scale of 10-4 We note that Haim et al. [2022] observed that models trained with SGD can also be reconstructed. The experiment in Appendix G (large models with many samples) also uses SGD and results with similar conclusion, that some models trained with SGD can be reconstructed. In general, exploring reconstruction from models trained with SGD an interesting direction for future works.
Runtime and Hardware. Runtime of a single reconstruction run (specific choice of hyperparame- ters) from a model D-1000-1000-1 takes about 20 minutes on a GPU Tesla V-100 32GB or NVIDIA Ampere Tesla A40 48GB
Hyperparameters of the Reconstruction Algorithm. Note that the reconstruction loss contains the derivative of a model with ReLU layers, which is flat and not-continuous. Thus, taking the derivative of the reconstruction loss results in zero function. To address this issue we follow a solution presented in Haim et [2022]. Namely, given a trained model, we replace in the backward phase of backpropogation the ReLU function with the derivative of a softplus function (or SmoothReLU) f(x) = a log(1 1 +e=*), where a hyperparameter of the reconstruction scheme. The functionality of the model itself does not change, as in the foraward phase the function remains ReL.U. Only the backward function is replaced with a smoother version of the derivative of ReLU which is the
f((x) aoo(x) Ite (here a is the Sigmoid function). To find good reconstructions
15

---

algorithm multiple times (typically 100 times) with random search over the hyperparameters (using the Weights & Biases framework [Biewald, 2020]). The exact parameters for the hyperparameters search are:
Learning rate: log-uniform in [10-5, 9.1]
dx log-uniform in [10-6 ,0.1]
min uniform in [0.01, 0.5]
uniform in [10, 500]
Multiclass Reconstruction More Results
C.1
Experiments with Different Number of Classes and Fixed Training Set Size
2 Classes
Classes
Classes
Classes
10 Classes
Recoreinee Ouaiity (55
GGood Rec ss 12
Good Rec. 19
Good Rec. t::
Good Rec. 28
Good Rec s:
Disrance from Decidon Boundary (0,100
Figure 9: Experiments of reconstruction from models trained on a fixed training set size (500 samples) for different number of classes. Number of "good" reconstruction is shown for each model.
To complete the experiment shown in Fig. 3, we also perform experiments on models trained on various number of classes (C (2, 3, 4, 5, 10}) and with fixed training set size of 500 samples (distributed equally between classes), see Fig. 9. It can be seen that as the number of classes increases, also does the number of good reconstructions, where for 10 classes there more than 6 times good reconstructions than for 2 classes. Also, the quality of the reconstructions improves as the number of classes increase, which is depicted by an overall higher SSIM score. We also note, that the number of good reconstructions Fig. is very similar to the number of good reconstructions from Fig. 3 for 50 samples per class. We hypothesize that although the number of training samples increases, the number of "support vectors" (i.e samples on the margin which can reconstructed) that are required for successfully interpolating the entire dataset does not change by much.
C.2 Results on SVHN Dataset
As shown in Fig. 10, our multiclass reconstruction scheme is not limited to CIFARIO dataset, but can be extended to other datasets as well, specifically SVHN dataset [Netzer et al., 201 1]. The model whose reconstructions are shown in Fig. 10 was trained on 50 samples per class (total of 10 classes) and the rest of the training hyperparameters are the same as that of its CIFARIO equivalent (of 50 sample per class with 10 classes).
3
5
8
3
6
57
8
80
138
4
15
15
6
6
18
10
2
3
5
6
6
7
3
8
10
24
5
5
8
58
bQ
6 6
88
10
50
6
6
93
2
5
6
6
34
51
6
68
8
8
6
6
Figure 10: Reconstruction form model trained on 50 samples per class from the SVHN dataset.
16

---

D General Losses More Results
Following the discussion in Section 5 and Fig. 4, Figures 12, 13 present visualizations training
samples and their reconstructions from models trained with L2, L2.5 and Huber loss, respectively.
Figure 11: Reconstruction using L2 loss. Training samples (red) and their best reconstructions (blue) using an MLP classifier that was trained on 300 CIFARIO images using an L2 regression loss, described in Section 5 and Fig. 4.
Figure 12: Reconstruction using L loss. Training samples (red) and their best reconstructions (blue) using MILP classifier that trained 300 CIFAR1O 10 images using an Iz5.5 regression loss, as described in Section 5 and Fig. 4.
17

---

Figure 13: Reconstruction using Huber loss. Training samples (red) and their best reconstructions (blue) using an MLP classifier that was trained on 300 CIFARIO 10 images using Huber loss, as described in Section 5 and Fig. 4.
E Further Analysis of Weight Decay
By looking at the exact distribution reconstruction quality to the distance from the margin, we observe that weight-decay (for some values) results in training samples being the margin of the trained classifier, thus being more vulnerable to our reconstruction scheme.
This observation is shown in Fig. 14 where we show the scatter plots for all the experiments from Fig. (a). We also provide the train and test errors for each model. It seems that the test error does not change significantly. However, interesting observation is that reconstruction is possible even for models with non-zero training errors, i.e. models that do not interpolate the data, for which the assumptions of Lyu and Li [2019] do not hold.
Ayp 1e-05
Áwo = 2.5e-05
Awp = 5e-05
Ago 7.5e-05
Avg 0.0001
Aug = 0.0005
1.00
00.00.0
E..00.00
8.000.00
-0.00
0.00
0.000000 B22
(SSIMOr. 20
0.75
0..2
0.21
922
0.2
Eug 0.22
0.50
0.25
0.00
Awp
0,001
Awp 0.005
Awo 0.01
Awo 0.1
Awa
Awo
1.00
00000
0.00
Ein 0.00
6m--0.14 Eus 0.24
-022
Eqj 0.22
0.22
Eus = 0.22
Eug 0.25
0.27
0.50
0.25
-5 0
10152025
-
5 152022
-5 5 15
-50 15
-50 5 10152025
-5 2015
Distance from Decision Boundary (4,(x) max6(x))
Figure 14: Scatter plots of the 12 experiments from Fig. 5 (a). Each plot is model trained with a different value of weight decay 2 classes with 50 samples in each class. Certain values of weight decay make the model more susceptible to our reconstruction scheme.
F Convolutional Neural Networks Ablations and Observations
In this section we provide results and visualizations to the experiments convolutional neural network in Section 6.1.
In Fig. 15 we show ablations for the choice of the kernel-size (k) and number of output channels (Com) for models with architecture CoNv(kernel-sizezk.. patpa were trained on 500 images (250 images per class) from the CIFAR10 dataset. with weight-decay
All models
18

---

term Awp=0.001. As can be seen, for such convolutional models we are able to reconstruct samples for wide range of choices.
Note that the full summary of reconstruction quality versus the distance from the decision boundary for the model whose reconstucted samples shown in Fig. 6, is shown in Fig. 15 for kernel-size (first row) and number of output channels 32 (third column).
Further analysis of Fig. 15. As expected for models with less parameters, the reconstructability decreases as the number of output channels decrease. An interesting phenomenon is observed for varying the kernel size: for fixed number of output channel, as the kernel size increases, the susceptibility of the model to our reconstruction scheme decreases. However, as the kernel size approaches 32 (the full resolution of the input image), the reconstructability increases once again. On the one hand it is expected, since for kernel-size=32 the model is essentially an MLP, albeit with smaller hidden dimension than usual (at most 64 here, whereas the typical model used in the paper had 1000). On the other hand, it is not clear why for some intermediate values of kernel size (in between and 32) the reconstructability decreases dramatically (for many models there are no reconstructed samples at all). This observation is interesting research direction for future works.
Number of Channels (Cout)
32
0.2
BU,00% 20
10.00% 201
2
22 20.
0.0
3.00
2030
0.0
$1.9000
2720
Kernel Size (k)
200
2.R0W
20
2.0%
380%
2.0%
0.00200
21
3.00%
0000 244
7.00%
x0
2.00%
5.00%
220%
0.0%
0.0
20
-10
20
20
-20
-10
10
20
-00
-10
20
Figure 15: Ablating the choice of the kernel size and output-channels for reconstruction from neu- ral binary classifiers with architecture note that this figure might not be displayed well on Google Chrome. Please open in Acrobat reader.)
(Please
19

---

Visualizing Kernels. In Haim et al. [2022], it was shown that some of the training samples can be found in the first layer of the trained MLPs, by reshaping and visualizing the weights of the first fully-connected layer. As opposed to MIPs, in the case of a model whose first layer is convolution layer, this is not possible. For completeness, in Fig- 16 we visualize all kernels the Conv layer. Obviously, full images of shape 3x32x32 cannot be found in kernels of shape 3x3x3, which makes reconstruction from such models (with convolution first layer) even more interesting.
Figure 16: The kernels of the model whose reconstructions are shown in Fig. 6, displayed as RGB images.
G Reconstruction From a Larger Number of Samples
One of the major limitations of Haim [2022] is that they reconstruct from models that trained on relatively small number of samples. Specifically, in their largest experiment, a model is trained with only 1,000 samples. Here we take step further, and apply our reconstruction scheme for model trained on 5,000 data samples.
To this end, we trained 3-layer MLP, where the number of neurons in each hidden layer 10,000. Note that the size of the hidden layer is 10 times larger than in any other model we used. Increasing the number of neurons seems be of the major reasons for which are able reconstruct from such large datasets, although we believe could be done with smaller models, which we leave for future research. We used the CIFAR100 dataset, with 50 samples in each class, for total of 5000 samples.
In Fig. 17a give the best reconstructions of the model. Note that although there is degradation in the quality of the reconstruction w.r.f a model trained on less samples, it is still clear that our scheme reconstruct some of the training samples some extent. In Fig. 17b show scatter plot of the SSIM score w.r.t the distance from the boundary, similar to Fig. 3a. Although most of the samples close to the margin, only few dozens achieve an SSIM> 0.4. This may indicate that there is a potential for much more images to reconstruct, and possibly with better quality.
20

---

(a) Full Images. Original samples from the training set (red) and reconstructed results (blue)
Reconstruction Quality [SSMMxx X)]
0.2
0.0
8
10
12
14
16
18
Distance from Decision Boundary (4, (x) max4)(x)
(b) Scatter plot (similar Fig. 3)
Figure 17: Reconstruction from model trained on 50 images per class from the CIFAR100 dataset (100 classes, total of 5000 datapoints). The model a 3-layer MLP with 10000 neurons each layer. (Please note that this figure might not be displayed well on Google Chrome. Please open in Acrobat reader.)
21


Reviews:
[{"review_id": "NWTUxrDWIM", "reviewer": "Reviewer_ZiTy", "summary": "This work extends the work of [Haim et al. 2022] to multi-class classification and with a more general loss function. Through mainly empirical evaluations, the authors demonstrate that data reconstruction is also possible and reveal interesting relationships between the ability to reconstruct and the number of classes, number of training data, size of the network, and the role of weight decay in the process. \n\nThe findings are most empirical. Though interesting, the novelty and technical contribution of this work is relatively low. ", "questions": "For empirical evaluations, I am wondering how good are the reconstructed samples in terms of FID. There is a line of work trying to generate/reconstruct samples from the classifier, mostly by gradient ascent [2,3]. How does your method compare with those as a generative model? Are there any deeper connections between the two very different methodologies?\n \n[2] Wang and Torr, Traditional Classification Neural Networks are Good Generators: They are Competitive with DDPMs and GANs\n\n[3] Zhu et al. Towards Understanding the Generative Capability of Adversarially Robust Classifiers", "limitations": "The authors have addressed the potential negative societal impact. The limitation of the work could have been more thoroughly discussed. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Reconstructing samples from trained classifiers is an important problem with direct connections to generative modeling and model privacy. The findings of this paper, though derivative from [Haim et al. 2022], are interesting and non-trivial. The findings regarding the data reconstruction ability vs. number of classes and weight decay are novel to me. They do bring new insights into understanding neural network classifiers. ", "weaknesses": "Novelty: This is my main concern about this submission. The framework is a direct extension from [Haim et al. 2022]. The objectives have been slightly modified, for the multi-class cross-entropy case and for the general loss with weight decay case, but they do not bring significant new insights over the original work. \n\nExperiments: Though the experiments conducted in this work are interesting, the results do not include variation analysis or error bars from multiple runs, nor statistical tests for whether two cases are significantly different. The experiments can be further improved by \n1) evaluating large-scale datasets, e.g., ImageNet, rather than the CIFAR10 dataset, which has been investigated in [Haim et al. 2022]. \n2) extending from simple classifier architectures (MLPs, Simple CNNs) to more complicated architectures such as ResNet or Vision Transformers.\n\nAnalysis: The empirical findings lack depth. For instance, the observations about weight decay vs. data reconstruction seem new and interesting. Is any of them provable, even in oversimplified cases? There are theoretical works on classifiers trained with square loss and weight decay [1].  Adding more theoretical justifications will make this submission stronger. \nI feel that this is a missed opportunity and I will consider raising my score if more theoretical insights can be provided to support the empirical findings. \n\n[1] Hu et al. Understanding square loss in training overparametrized neural network classifiers", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and the constructive comments.\n\n1) Regarding novelty, we would like to emphasize that although our work is a direct extension of [Haim et al. 2022], we make some significant and novel contributions and overcome some of the limitations of the previous work. To name a few: \n\n- We generalized the previous work to a multi-class setting (as opposed to binary classification). This required a non-trivial extension of the KKT solution to the max-margin problem, which was not shown before in practical applications. \n\n- We used a different theoretical justification (beyond convergence to a KKT of the max margin problem) to devise a reconstruction loss which works on general regression losses, and demonstrated it on three different losses.\n\n- We investigated the connection between weight decay, number of parameters and reconstructability and found a connection which was not known before. This connection enabled us to overcome some of the limitations of the previous work, such as reconstruction from CNNs (as opposed to reconstruction from MLPs only), reconstruction from models with a standard initialization scheme (as opposed to models trained with a very small and non-standard initialization scale), and reconstruction from a model trained on up to $5$,$000$ samples. As far as we know, this is the largest dataset used to train a classifier and shown to be successfully reconstructed.\n\n2) Regarding the experiment and improving to larger datasets and more complicated architectures: we believe that the end goal of understanding memorization in neural networks is to be able to apply it on more realistic settings. However, the current state of the research in this field is still in its infancy, and we believe that our work provides an important stepping stone in that direction. We agree that there is much more work to be done for future research in this field.\nRegarding the comment about the error bars,  we will add them whenever relevant in the camera ready version. We note that every reconstruction experiment in our paper uses a large hyper-parameter sweep and over many different runs. The full details appear in Appendix B.\n\n3) Regarding theoretical justifications, we thank the reviewer for the helpful comment. Please note that of the three main sections in this work, two provide theoretical analysis and insights on the success of the demonstrated empirical reconstruction results (Sections 4\\&5). \n\n    We also have a theoretical justification on why weight decay helps reconstructability for simplified networks, which we will add in the camera-ready version.\n\n    In a nutshell, the (directional) convergence rate to a KKT point of the max-margin problem, as paraphrased in Thm 3.1, might be extremely slow -- logarithmic in $t$ for  time $t \\to \\infty$. Hence, even after many gradient descent steps (e.g., $10^6$ iterations) the learning process may essentially stop because of small gradients, but the parameters may still be far from the direction of a KKT point. A relation between small initialization and such convergence is shown in Moroshko et al. [1]. They prove that for diagonal linear networks (i.e., a certain simplified architecture of deep linear networks) the initialization scale controls the rate of convergence to the KKT point. Namely, when the initialization is small, gradient flow converges much faster to a KKT point.\n\n    Simply put: without weight-decay, small gradients (of the empirical loss) do *not* imply convergence to a KKT point. However: small initialization implies faster convergence (to KKT), which implies better reconstructability.\n\n    In our novel approach - when weight-decay is used, small gradients (of the loss) *do* imply better convergence to the solution in Eq. 14, as opposed to convergence to a KKT point in the case of training without weight-decay. And better convergence implies better reconstructability. This provides a theoretical justification for why incorporating weight decay may result in better reconstructability for models with standard initialization, which is one of the main contributions in this work.\n\n4) Regarding evaluation with FID, this is an interesting question and an interesting comparison that could be made. However, we emphasize that our work is focused on understanding *memorization* in neural networks, while works on generative models focus on how to model a distribution. Hence, such comparisons are out of the scope of our paper. \n\n[1] Edward Moroshko, Blake E. Woodworth, Suriya Gunasekar, Jason D. Lee, Nati Srebro, and Daniel Soudry. \"Implicit bias in deep linear classification: Initialization scale vs training accuracy.\" Advances in neural information processing systems 33 (2020): 22182-22193.\n"}, {"review_id": "7YEpJIXjwF", "reviewer": "Reviewer_Poqu", "summary": "This paper is following the work of Haim et al. (2022) who introduced a reconstruction scheme for neural network with logistic or exponential loss for binary classification tasks. They extended this work by introducing this reconstruction scheme in a multi class setting. They also demonstrated how it can also be extended to convolutional neural network and more general loss functions. ", "questions": "Is there any reason why you didn't use a multi class setting with the CNN ?", "limitations": "I didn't found a limitation section in this work or in the appendix. One main limitation is that this work is limited to extremely small architectures and that it's not clear how much of it will transfer to practical architectures. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well written and the problematic well presented.\n- Memorization is indeed an important research area and this paper offer significant finding that give a better understanding on which conditions influence memorization of training data.\n- I really appreciated the balance between quantitative and qualitative experiment. As well as the ablation study with the number of training class and examples. It gives really good insights.\n- The appendix contain good explanations and interesting additional experiments.\n\nOverall, I think this is a good paper that give important insights for the community.", "weaknesses": "- One small issue I have with this paper is that there isn't a single strong storyline. It is kind of a mix of different improvements over Haim et al. (2022) with General Loss Function/Multi Class and CNN. So it's like section 4, 5 and 6 could be entirely independent. For example, I was surprised after introducing the multi class that the CNN was trained with a binary classification task. In term of presentation I would probably have started with the General loss, then present the multi class setting, then present multi-class with CNN (and weight decay) (Just to make the story smoother).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive and constructive feedback.\n\nRegarding the paper's storyline: this is a good point, and we had a dilemma on how to organize the different sections. We will consider implementing the reviewer's suggestion for reordering in the camera-ready version.\n\nRegarding multi-class CNNs: the reason is that we wanted to decouple the effects of changing the architecture from changing to multiclass from binary classificatoin. We can add a multiclass CNN experiment in the camera-ready version.\n\nRegarding limitations, we will add a limitations section in the camera-ready version with an elaborated discussion on the limitations of our current approach, including references to small architectures and datasets."}, {"review_id": "BfthPV0LDZ", "reviewer": "Reviewer_FQAU", "summary": "Reconstructing training samples from the trained model may cause privacy issues. This paper demonstrates the training samples reconstruction for multi-classes and also for both fully-connected neural networks and convolutional neural networks. In addition, the authors investigate different factors that contribute the reconstruction.", "questions": "See weaknesses.", "limitations": "Did not discuss the limitation.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Extended the previous training-sample reconstruction scheme to multi-class settings and also to both fully-connected NN and CNN, making it more practical.\n2) It includes both theoretical analysis and empirical results to verify its effectiveness.", "weaknesses": "1) When the authors reconstruct the data, the generalization of the models is very poor. However, in practice, the trained model may also do well on unseen test data. So, the question is, can we still reconstruct the data successfully even when the trained model performs well on unseen test data? \n\n2) What if a model is trained without weight decay or a model trained with other regularizers? Will this finding still be held?\n\n3) The experimental datasets are relatively simple and on a small scale. Will it be possible to experiment on a more large-scale and complex dataset such as ImageNet?\n\n\n\n#---------------------------------------------------------------------------------------------------------------#\n\n#---------------------------------------------------------------------------------------------------------------#\n\n\n After reading the rebuttals and the reviews from other reviewers and the discussions, some of my previous concerns of this paper remain unaddressed, therefore, I would like to stick to my original evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the comments and feedback. Regarding the questions raised by the reviewer:\n\n1) Our work considers relatively small-scale networks, trained on  small datasets (up to $5$,$000$ samples). Although the generalization of our models is not comparable to the state of art, it is significantly non-trivial (much larger than random). We also ran an extensive hyper-parameter sweep, hence we possibly  attain the best generalization score that can be achieved using these architectures and datasets. We do believe that these methods can be further improved, allowing for larger architectures and bigger datasets, and as such better generalization. \n\n\n2) This is an interesting question. We compare our results to baseline models that are trained without regularization at all, for example see the baselines in Fig.6 (as dashed lines) and the rightmost column in Fig.4. The reason we specifically used weight decay as a regularizer was because of the theoretical understanding we have about it (presented in Section 5). Studying memorization in models trained with other regularizers is an interesting direction and we currently don't have a theoretical understanding of how it would affect reconstructability. Testing this question empirically may prove to be useful, but is beyond the scope of this paper.\n\n3) We agree that the datasets we trained on are relatively small scale. We emphasize that this is also a limitation of previous works on data reconstruction. Note however that in our work we were able to reconstruct from a model trained on $5$,$000$ samples, which is $5$ times more than the largest dataset in [Haim et al. 2022]. Reconstructing from ImageNet is a challenging and non-trivial task (even when considering only a small part of it). We believe it is possible, but may require further improvement of the current reconstruction methods and theoretical understanding, and we leave it for future research.\n\nRegarding limitations, similar to previous works on data reconstruction our results are limitied to relatively small datasets and architectures. We will emphasize these in the camera-ready version by adding a limitations section.\n"}, {"review_id": "QNVgUgQ9n2", "reviewer": "Reviewer_nrvg", "summary": "The paper presents multiple extensions of Haim et al. [2022] in relation to reconstruction of training data in neural networks. In short the extensions are: extension to multiclass from binary, extension to regression losses, investigate of the effect of weight decay and analysis of relationship between number of samples and number of parameters. All extensions are supported by experiments and for the first two extensions the authors provide theoretical foundation.", "questions": "The empirical results are impressive and speaks for themself, but I am really missing a bit of discussion of the results:\n* Any reasoning why the straightforward extension failed to reconstruct samples (L139-141)?\n* The reason why CNNs are harder to reconstruct? Essentially they are just sparse linear layers (compared to fully connected) \n* Any reasoning why weight-decay can have such an influence on the number of reconstructed examples.", "limitations": "It is clear from the paper that work on reconstruction of training data from memorization in the neural networks are still in its early stages. That said it would be great if the authors could be more explicit in the paper about the obvious limitations of the method:\n* Only possible for shallow networks\n* Only possible for specific networks (only ReLU it seems)\n* Only possible for networks trained on few samples (the extension to 5000 samples is nice, but still nowhere near practical settings)\n\nIt is mentioned in future work, but giving a bit of reasoning what challenges lies in scaling the methods seems to be in order.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Originality:\nThe work seems original and novel (even if it is only incremental)\n\nQuality:\nThe paper is of high quality and provide theoretical foundation where it is needed and empirical evidence for everything else.  \n\nClarity:\nThe paper is clearly written and easy to follow. \n\nSignificance:\nIt is not unreasonable to say that the end goal for this line of work is to be able to study neural networks that are used in more real life scenarios today. In that regard this works provides an important stepping stone in that direction, especially with its extension to multiclass. ", "weaknesses": "Section 5 on general loss functions seems to be the weaker contribution primarily due to two reasons:\n1. The authors are essentially trying out multiple regression losses, which is good, but selling this as \"general loss functions\" seems to be a stretch. The authors could be more clear in their contributions by simply calling \"general regressions losses\". \n2. The experiment is created by artificially taking a task that is binary in nature (labels {-1,1}) and trying to model them using a regression loss. It would be much more relevant if the authors had considered another dataset and/or task where the target is continues.\n\nIn general I find that the paper has a lot of experiments to show and due to the limited space this does not give room for an actual discussion of their findings. I find this a wrong prioritization, and that the authors should discuss and give some kind of reasoning for just some of their findings (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough review and constructive and positive feedback.\n\nRegrading general vs. regression loss: we agree with the reviewer on the comment. Although our method described in Section 5 allows us to reconstruct data which is trained using any loss function, we tested it only on several regression losses. We will change the phrasing throughout the paper to \"general regression losses\" instead of \"general loss functions\". \nRegarding datasets with continuous labels, we agree that these are interesting cases to study, however it is beyond the scope of this paper and we leave it to future research. \n\nRegarding the prioritization, we agree it would be helpful to provide more discussion on the findings and theoretical insights (e.g. such as in sections 4.1 and 5.1), however we had to trade-off in terms of space limitations to also present the results and figures in the main part of the paper. In the camera-ready version we will add more discussion on the findings themselves, and provide more details on the theoretical reasoning behind our results. Please also see the response to the last question below for further justification for weight decay.\n\n\nRegarding the questions:\n\n1) This is a good point and we will further elaborate on it in the camera-ready version. The straightforward extension has $C \\cdot n$ constraints whereas our proposed equivalent formulation has only $n$. For reconstruction, each such constraint requires its own optimization variable $\\lambda$. \nSo the latter formulation has much less optimization variables ($n$ < $C\\cdot n$) which is very significant for large number of classes $C$. We believe that reducing the number of constraints is the main reason for the successful reconstruction in the multi-class setting.\n\n2) Although CNNs can be written as a sparse linear layer, they are inherently different. First, there is weight sharing, and second, the number of parameters is significantly smaller than linear layers. For these reasons, the previous approach from [Haim et al. 2022] did not work as is. But we could reconstruct from CNNs using weight decay and large enough layers.\n\n3) This is a very good question. We do have a theoretical justification for why weight decay helps reconstructability, at least for simplified networks. We will add this explanation in the camera-ready version. In a nutshell,\nTheorem 3.1 from our paper is used to devise a reconstruction loss which is based on that networks converge in direction to a  KKT point of the max-margin problem. However, this directional convergence occurs asymptotically as the time $t \\to \\infty$, and the rate of convergence in practice might be extremely slow, logarithmic in $t$. Hence, even when training for, e.g., $10^6$ iterations, gradient descent might reach a solution which is still too far from the KKT point, and therefore reconstruction fails. In other words, even when training until the gradient of the empirical loss is extremely small, the direction of the network’s parameters might be far from the direction of a KKT point. In [1], the authors proved that in diagonal linear networks (i.e., a certain simplified architecture of deep linear networks) the initialization scale indeed controls the rate of convergence to the KKT point, namely, when the initialization is small gradient flow converges much faster to a KKT point. \n\n    This theoretical result explains the behavior that we observe in the experiments, for a simple setting. For this reason, when training without weight decay, small initialization seems to be required to allow reconstruction. However, when training with weight decay, our theoretical analysis in Section 5.1 explains why small initialization is no longer required. Here, the reconstruction does not rely on converging to a KKT point of the max-magin problem, but relies on Eq. (14) which holds (approximately) whenever we reach a sufficiently small gradient of the training objective. \tThus, when training with weight decay and reaching a small gradient Eq. (14) holds, which allows for reconstruction, contrary to training without weight decay where reaching a small gradient doesn't imply converging close to a KKT point. \n\nRegarding limitations, we will add a dedicated limitations section in the camera-ready version with elaborated discussion on the points highlighted by the reviewer.\n"}]

Author rebuttal:

Meta-review: Building on previous work regarding training data reconstruction from MLPs trained to perform binary classification, this submission describes a strategy to reconstruct training samples from both MLPs and CNNs trained with a larger family of loss functions. It further analyzes the effect of weight decay and model width upon the efficacy of the described method.

Reviewers agreed that the work tackles an important problem and contains novel elements, although the method appears to be limited to shallow networks trained on small datasets. One reviewer raised concerns regarding the novelty and the magnitude of the contribution relative to Haim et al. (2022), upon which it builds. A second reviewer raised concerns regarding the absence of a strong storyline, but nonetheless supported acceptance.

I feel that the methodology and theory described in this submission are sufficiently novel. The method, while limited in its current form, could lead to practically meaningful data reconstruction methods and/or techniques to limit memorization. Thus, I recommend acceptance.
Decision: Accept (poster)