SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We validate the above theoretical analysis by demonstrating reconstruction from models trained on other losses than the ones shown in Section 4 and Haim et al. [2022]. We use the same dataset as in the classification tasks - images from CIFAR10 dataset with binary labels of  $\{-1,1\}$ . The only difference is replacing the classification binary cross-entropy loss with regression losses (e.g., MSE).
In classification tasks, we analyzed the results by plotting the reconstruction quality (SSIM) against the sample's distance from the decision boundary (see Section 3). This showed that reconstruction is only feasible for margin-samples. However, in regression tasks, margin and decision boundary lack specific meaning. We propose an alternative analysis approach – note that smaller distance from the margin results in higher loss for binary cross-entropy. Intuitively, margin-samples are the most challenging to classify (as reflected by the loss function). Therefore, for regression tasks, we analyze the results by plotting the reconstruction quality against the loss (per training sample).
In Fig. 4 we show results for reconstructions from models trained with MSE,  $L_{2.5}$  loss ( $\ell = |\Phi(\mathbf{x};\boldsymbol{\theta}) - y|^p$  for  $p = 2,2.5$  respectively) and Huber loss [Huber, 1992]. The reconstruction scheme in Eq. (15) is the same for all cases, and is invariant to the loss used during training. Fig. 4 highlights two important observations: first, the reconstruction scheme in Eq. (15) succeeds in reconstructing large portions of the training set from models trained with regression losses, as noted from the high
quality (SSIM) of the samples. Second, by plotting quality against the loss, one sees that "challenging" samples (with high loss) are easier to reconstruct. Also note that the analysis works for classification losses, namely BCE with or without weight-decay in Fig. 4). For more results see Appendix D.
6 On the Different Factors that Affect Reconstructability
Our goal is to gain a deeper understanding of the factors behind models' vulnerability to reconstruction schemes. In this section, we present several analyses that shed light on several important factors.
6.1 The Role of Weight Decay in Data Reconstruction
Haim et al. [2022] assumed MLP models whose first fully-connected layer was initialized with small (non-standard) weights. Models with standard initialization (e.g., He et al. [2015], Glorot and Bengio [2010]) did not yield reconstructed samples. In contrast, the MLPs reconstructed in Haim et al. [2022] were initialized with an extremely small variance in the first layer. Set to better understand this drawback, we observed that incorporating weight-decay during training, not only enabled samples reconstruction in models with standard initialization, but often increase the reconstructability of training samples.
(b)
Figure 5: Using weight-decay during training increases vulnerability to sample reconstruction
In Fig. 5a-b we show the number of good reconstructions for different choices of the value of the weight decay  $(\lambda_{\mathrm{WD}})$ , for MLP classifiers trained on  $C = 2,10$  classes and 50 samples per class (Fig. 5 a, b resp.). We add two baselines trained without weight-decay: model trained with standard initialization (black) and model with small-initialized first-layer (red). See how for some values of weight-decay, the reconstructability is significantly higher than what was observed for models with non-standard initialization. By examining the training samples' distance to the boundary, one observes that using weight-decay results in more margin-samples which are empirically more vulnerable to reconstruction (see full details in Appendix E).
We now give an intuitive theoretical explanation to the role of weight decay in data reconstruction. Theorem 3.1 is used to devise the reconstruction loss in Eq. (6), which is based on the directional convergence to a KKT point of the max-margin problem. However, this directional convergence occurs asymptotically as the time  $t \to \infty$ , and the rate of convergence in practice might be extremely slow. Hence, even when training for, e.g.,  $10^{6}$  iterations, gradient descent might reach a solution which is still too far from the KKT point, and therefore reconstruction might fail. Thus, even when training until the gradient of the empirical loss is extremely small, the direction of the network's parameters might be far from the direction of a KKT point. In Moroshko et al. [2020], the authors proved that in diagonal linear networks (i.e., a certain simplified architecture of deep linear networks) the initialization scale controls the rate of convergence to the KKT point, namely, when the initialization is small gradient flow converges much faster to a KKT point. A similar phenomenon seems to occur also in our empirical results: when training without weight decay, small initialization seems to be required to allow reconstruction. However, when training with weight decay, our theoretical analysis in Section 5.1 explains why small initialization is no longer required. Here, the reconstruction does not rely on converging to a KKT point of the max-magin problem, but relies on Eq. (14) which holds (approximately) whenever we reach a sufficiently small gradient of the training objective. Thus, when training with weight decay and reaching a small gradient Eq. (14) holds, which allows for reconstruction, contrary to training without weight decay where reaching a small gradient does not imply converging close to a KKT point.
Figure 6: Reconstruction from CNN. Training samples (red) and their best reconstructions (blue)
Reconstruction from Convolutional Neural Networks (CNNs). CNNs adhere to the assumptions of Theorem 3.1, yet Haim et al. [2022] failed to apply their reconstruction scheme Eq. (6) to CNNs. We observe that incorporating weight-decay during training (using standard initialization) enables samples reconstruction. In Fig. 6 we show an example for reconstruction from a binary classifier whose first layer is a Conv-layer with kernel size 3 and 32 output channels, followed by two fully connected layers  $(\mathrm{CONV}(k = 3,C_{\mathrm{out}} = 32) - 1000 - 1)$ . The weight-decay term is  $\lambda_{\mathrm{WD}} = 0.001$  (the training setup is similar to that of MLP). In Fig. 5c we show the reconstructability for the same CNN model trained with other values of  $\lambda_{\mathrm{WD}}$ . Note how the weight-decay term plays similar role in the CNN as in the MLP case. See full details in Appendix F.
6.2 The Effect of the Number of Parameters and Samples on Reconstructability
Haim et al. [2022] observed that models trained on fewer samples are more susceptible to reconstruction in terms of both quantity and quality. In this section, we delve deeper into this phenomenon, focusing on the intricate relationship between the number of parameters in the trained model and the number of training samples. We conduct the following experiment:
We train 3-layer MLPs with architecture  $D-W-W-1$  on  $N$  training samples from binary CIFAR10 (animals vs. vehicles), where  $W \in \{5, 10, 50, 100, 500, 1000\}$  and  $N \in \{10, 50, 100, 300, 500\}$ . We conduct the experiment for both classification and regression losses, with BCE and MSE loss respectively. Generalization error is  $23\% - 31\%$  for BCE (classification) and  $0.69 - 0.88$  for MSE (regression), compared to  $50\% / 0.97$  for similar models with random weights.
We reconstruct each model using Eq. (15) and record the number of good reconstructions. The results are shown in Fig. 7. Note that as  $W / N$  increases, our reconstruction scheme is capable of reconstructing more samples, and vice versa. For example, consider the case when  $N = 10$ . To successfully reconstruct the entire training set, it is sufficient for  $W$  to be greater than 50/10 (for MSE/BCE). However, when  $N = 500$ , even larger models (with larger  $W$ ) can only reconstruct up to 8% of the samples.
Lastly, we reconstruct from a model with  $W = 10,000$ , trained on  $N = 5,000$  samples (5 times larger than any previous model). While there is some degradation in the quality of the reconstructions compared to models trained on fewer samples, it is evident that our scheme can still reconstruct some of the training samples. For full results see Appendix G.
We present improved reconstruction methods and conduct a comprehensive analysis of the reconstruction method proposed by Haim et al. [2022]. Particularly, we extend their reconstruction scheme to a multi-class setting and devise a novel reconstruction scheme for general loss functions, allowing reconstruction in a regression setting (e.g., MSE loss). We examine various factors influencing reconstructability. We shed light on the role of weight decay in samples memorization, allowing for sample reconstruction from convolutional neural networks. Lastly, we examine the intricate relationship between the number of parameters, the number of samples, and the vulnerability of the model to reconstruction schemes. We acknowledge that our reconstruction method raises concerns regarding privacy. We consider it crucial to present such methodologies as they encourage researchers to study the potential hazards associated with training neural networks. Additionally, it allows for the development of protective measures aimed at preventing the leakage of sensitive information.
Figure 7: Effect of the number of neurons and number of training samples on reconstructability. We train 3-layer MLPs with architecture  $D-W-W-1$  on  $N$  training samples from binary CIFAR10 (animals vs. vehicles), using MSE (left) or BCE (right) loss. At each cell we report the number of good reconstructions  $(\mathrm{SSIM} > 0.4)$ , in both absolute numbers and as a percentage relative to  $N$ .
Limitations. While we have relaxed several of the previous assumptions presented in Haim et al. [2022], our method still exhibits certain limitations. First, we only consider relatively small-scale models: up to several layers, without residual connections, that have trained for many iterations on a relatively small dataset without augmentations. Second, determining the optimal hyperparameters for the reconstruction scheme poses a challenge as it requires exploring many configurations for even a single model.
Future work. All of the above extend our knowledge and understanding of how memorization works in certain neural networks. This opens up several possibilities for future research including extending our reconstruction scheme to practical models (e.g., ResNets), exploring reconstruction from models trained on larger datasets or different data types (e.g., text, time-series, tabular data), analyzing the impact of optimization methods and architectural choices on reconstructability, and developing privacy schemes to protect vulnerable samples from reconstruction attacks.
Acknowledgements. This project received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement No 788535), and ERC grant 754705, and from the D. Dan and Betty Kahn Foundation. GV acknowledges the support of the NSF and the Simons Foundation for the Collaboration on the Theoretical Foundations of Deep Learning.
D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. Courville, Y. Bengio, et al. A closer look at memorization in deep networks. In International conference on machine learning, pages 233-242. PMLR, 2017.
B. Balle, G. Cherubin, and J. Hayes. Reconstructing training data with informed adversaries. arXiv preprint arXiv:2201.04845, 2022.
L. Biewald. Experiment tracking with weights and biases, 2020. URL https://www.wandb.com/. Software available from wandb.com.
G. Brown, M. Bun, V. Feldman, A. Smith, and K. Talwar. When is memorization of irrelevant training data necessary for high-accuracy learning? In Proceedings of the 53rd Annual ACM SIGACT Symposium on Theory of Computing, pages 123–132, 2021.
N. Carlini, C. Liu, U. Erlingsson, J. Kos, and D. Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pages 267-284, 2019.
H. Yin, A. Mallya, A. Vahdat, J. M. Alvarez, J. Kautz, and P. Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16337-16346, 2021.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. CoRR, abs/1611.03530, 2016. URL http://arxiv.org/abs/1611.03530.
C. Zhang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications of the ACM, 64(3):107-115, 2021.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018.
L. Zhu, Z. Liu, and S. Han. Deep leakage from gradients. Advances in Neural Information Processing Systems, 32, 2019.
A Analyzing and Visualizing the Results of the Reconstruction Optimization
The analysis of the results of the various reconstruction losses Eqs. (6), (12) and (15), involve verifying and checking which of the training samples were reconstructed. In this section we provide further details on our method for analyzing the reconstruction results, and how we measure the quality of our reconstructions.
A.1 Analyzing the Results of the Reconstruction Optimization
In order to match between samples from the training set and the outputs of the reconstruction algorithm (the so-called "candidates") we follow the same protocol of Haim et al. [2022]. Note that before training our models, we subtract the mean image from the given training set. Therefore the training samples are  $d$ -dimensional objects where each entry is in  $[-1, 1]$ .
First, for each training sample we compute the distance to all the candidates using a normalized  $L_{2}$  score:
$$
d (\mathbf {x}, \mathbf {y}) = \left\| \frac {\mathbf {x} - \mu_ {\mathbf {x}}}{\sigma_ {\mathbf {x}}} - \frac {\mathbf {y} - \mu_ {\mathbf {y}}}{\sigma_ {\mathbf {y}}} \right\| _ {2} ^ {2} \tag {16}
$$
Where  $\mathbf{x},\mathbf{y}\in \mathbb{R}^d$  are a training sample or an output candidate from the reconstruction algorithm,  $\mu_{\mathbf{x}} = \frac{1}{d}\sum_{i = 1}^{d}\mathbf{x}(i)$  is the mean of  $\mathbf{x}$  and  $\sigma_{\mathbf{x}} = \sqrt{\frac{1}{d - 1}\sum_{i = 1}^{d}(\mathbf{x}(i) - \mu_{\mathbf{x}})^2}$  is the standard deviation of  $\mathbf{x}$  (and the same goes for  $\mathbf{y},\mu_{\mathbf{y}},\sigma_{\mathbf{y}}$ ).
Second, for each training sample, we take  $C$  candidates with the smallest distance according to Eq. (16).  $C$  is determined by finding the first candidate whose distance is larger than  $B$  times the distance to the closest nearest neighbour (where  $B$  is a hyperparameter). Namely, for a training sample  $\mathbf{x}$ , the nearest neighbour is  $\mathbf{y}_1$  with a distance  $d(\mathbf{x},\mathbf{y}_1)$ , then  $C$  is determined by finding a candidate  $\mathbf{y}_{C + 1}$  whose distance is  $d(\mathbf{x},\mathbf{y}_{C + 1}) > B\cdot d(\mathbf{x},\mathbf{y}_1)$ , and for all  $j\leq C$ ,  $d(\mathbf{x},\mathbf{y}_j)\leq B\cdot d(\mathbf{x},\mathbf{y}_1)$ .  $B$  was chosen heuristically to be  $B = 1.1$  for MLPs, and  $B = 1.5$  for convolutional models. The  $C$  candidates are then summed to create the reconstructed sample  $\hat{\mathbf{x}} = \frac{1}{C}\sum_{j = 1}^{C}\mathbf{y}_j$ . In general, we can also take only  $C = 1$  candidate, namely just one nearest neighbour per training sample, but choosing more candidates improve the visual quality of the reconstructed samples.
Third, the reconstructed sample  $\hat{\mathbf{x}}$  is scaled to an image in [0, 1] by adding the training set mean and linearly "stretching" the minimal and maximal values of the result to [0, 1]. Finally, we compute the SSIM between the training sample  $\mathbf{x}$  and the reconstructed sample  $\hat{\mathbf{x}}$  to measure the quality of reconstruction.
A.2 Deciding whether a Reconstruction is "Good"
Here we justify our selection for SSIM=0.4 as the threshold for what we consider as a "good" reconstruction. In general, the problem of deciding whether a reconstruction is the correct match to a given sample, or whether a reconstruction is a "good" reconstruction is equivalent to the problem of comparing between images. No "synthetic" metric (like SSIM,  $l2$  etc.) will be aligned with human perception. A common metric for this purpose is LPIPS [Zhang et al., 2018] that uses a classifier trained on Imagenet [Deng et al., 2009], but since CIFAR images are much smaller than Imagenet images ( $32 \times 32$  vs.  $224 \times 224$ ) it is not clear that this metric will be better than SSIM.
As a simple rule of thumb, we use SSIM  $>0.4$  for deciding that a given reconstruction is "good". To justify, we plot the best reconstructions (in terms of SSIM) in Fig. 8. Note that almost all samples with SSIM  $>0.4$  are also visually similar (for a human). Also note that some of the samples with SSIM  $< 0.4$  are visually similar, so in this sense we are "missing" some good reconstructions. In general, determining whether a candidate output of a reconstruction algorithm is a match to a training sample is an open question and a problem in all other works for data reconstruction, see for example Carlini et al. [2023] that derived a heuristic for reconstructed samples from a generative model. This cannot be dealt in the scope of this paper, and is an interesting future direction for our work.
Figure 8: Justifying the threshold of  $\mathrm{SSIM} = 0.4$  as good rule-of-thumb for a threshold for a "good" reconstruction. The SSIM values are shown above each train-reconstruction pair. Note that samples with  $\mathrm{SSIM} > 0.4$  (blue) are visually similar. Also some of the samples with  $\mathrm{SSIM} < 0.4$  (red) are similar. In general deciding whether a reconstruction is "good" is an open question beyond the scope of this paper.
Further Training Details. The models that were reconstructed in the main part of the paper were trained with learning rates of 0.01 for binary classifiers (both MLP and convolutional), and 0.5 in the case of multi-class classifier (Section 4). The models were trained with full batch gradient descent for  $10^{6}$  epochs, to guarantee convergence to a KKT point of Eq. (1) or a local minima of Eq. (13). When small initialization of the first layer is used (e.g., in Figs. 2 and 3), the weights are initialized with a scale of  $10^{-4}$ . We note that Haim et al. [2022] observed that models trained with SGD can also be reconstructed. The experiment in Appendix G (large models with many samples) also uses SGD and results with similar conclusion, that some models trained with SGD can be reconstructed. In general, exploring reconstruction from models trained with SGD is an interesting direction for future works.
Runtime and Hardware. Runtime of a single reconstruction run (specific choice of hyperparameters) from a model  $D$ -1000-1000-1 takes about 20 minutes on a GPU Tesla V-100 32GB or NVIDIA Ampere Tesla A40 48GB.
Hyperparameters of the Reconstruction Algorithm. Note that the reconstruction loss contains the derivative of a model with ReLU layers, which is flat and not-continuous. Thus, taking the derivative of the reconstruction loss results in a zero function. To address this issue we follow a solution presented in Haim et al. [2022]. Namely, given a trained model, we replace in the backward phase of backpropagation the ReLU function with the derivative of a softplus function (or SmoothReLU)  $f(x) = \alpha \log(1 + e^{-x})$ , where  $\alpha$  is a hyperparameter of the reconstruction scheme. The functionality of the model itself does not change, as in the forward phase the function remains a ReLU. Only the backward function is replaced with a smoother version of the derivative of ReLU which is  $f'(x) = \alpha \sigma(x) = \frac{\alpha}{1 + e^{-x}}$  (here  $\sigma$  is the Sigmoid function). To find good reconstructions we run the
algorithm multiple times (typically 100 times) with random search over the hyperparameters (using the Weights & Biases framework [Biewald, 2020]). The exact parameters for the hyperparameters search are:
- Learning rate: log-uniform in  $[10^{-5}, 1]$
-  $\sigma_{x}$ : log-uniform in  $[10^{-6}, 0.1]$
-  $\lambda_{\mathrm{min}}$ : uniform in [0.01, 0.5]
$\alpha$  : uniform in [10, 500]
C Multiclass Reconstruction - More Results
C.1 Experiments with Different Number of Classes and Fixed Training Set Size
Figure 9: Experiments of reconstruction from models trained on a fixed training set size (500 samples) for different number of classes. Number of "good" reconstruction is shown for each model.
To complete the experiment shown in Fig. 3, we also perform experiments on models trained on various number of classes ( $C \in \{2, 3, 4, 5, 10\}$ ) and with a fixed training set size of 500 samples (distributed equally between classes), see Fig. 9. It can be seen that as the number of classes increases, also does the number of good reconstructions, where for 10 classes there are more than 6 times good reconstructions than for 2 classes. Also, the quality of the reconstructions improves as the number of classes increase, which is depicted by an overall higher SSIM score. We also note, that the number of good reconstructions in Fig. 9 is very similar to the number of good reconstructions from Fig. 3 for 50 samples per class. We hypothesize that although the number of training samples increases, the number of "support vectors" (i.e. samples on the margin which can be reconstructed) that are required for successfully interpolating the entire dataset does not change by much.
C.2 Results on SVHN Dataset
As shown in Fig. 10, our multiclass reconstruction scheme is not limited to CIFAR10 dataset, but can be extended to other datasets as well, specifically SVHN dataset [Netzer et al., 2011]. The model whose reconstructions are shown in Fig. 10 was trained on 50 samples per class (total of 10 classes) and the rest of the training hyperparameters are the same as that of its CIFAR10 equivalent (of 50 sample per class with 10 classes).
Figure 10: Reconstruction form model trained on 50 samples per class from the SVHN dataset.
D General Losses - More Results
Following the discussion in Section 5 and Fig. 4, Figures 11, 12, 13 present visualizations of training samples and their reconstructions from models trained with  $L_{2}$ ,  $L_{2.5}$  and Huber loss, respectively.
Figure 11: Reconstruction using  $L_{2}$  loss. Training samples (red) and their best reconstructions (blue) using an MLP classifier that was trained on 300 CIFAR10 images using an  $L_{2}$  regression loss, as described in Section 5 and Fig. 4.
Figure 12: Reconstruction using  $L_{2.5}$  loss. Training samples (red) and their best reconstructions (blue) using an MLP classifier that was trained on 300 CIFAR10 images using an  $L_{2.5}$  regression loss, as described in Section 5 and Fig. 4.
Figure 13: Reconstruction using Huber loss. Training samples (red) and their best reconstructions (blue) using an MLP classifier that was trained on 300 CIFAR10 images using Huber loss, as described in Section 5 and Fig. 4.
E Further Analysis of Weight Decay
By looking at the exact distribution of reconstruction quality to the distance from the margin, we observe that weight-decay (for some values) results in more training samples being on the margin of the trained classifier, thus being more vulnerable to our reconstruction scheme.
This observation is shown in Fig. 14 where we show the scatter plots for all the experiments from Fig. 5 (a). We also provide the train and test errors for each model. It seems that the test error does not change significantly. However, an interesting observation is that reconstruction is possible even for models with non-zero training errors, i.e. models that do not interpolate the data, for which the assumptions of Lyu and Li [2019] do not hold.