SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Gon Buzaglo*1
Niv Haim*1
Gilad Yehudai
Gal Vardi²
Yakir Oz
Yaniv Nikankin
Michal Irani
<sup>1</sup>Weizmann Institute of Science, Rehovot, Israel  
<sup>2</sup>TTI-Chicago and the Hebrew University of Jerusalem
Memorization of training data is an active research area, yet our understanding of the inner workings of neural networks is still in its infancy. Recently, Haim et al. [2022] proposed a scheme to reconstruct training samples from multilayer perceptron binary classifiers, effectively demonstrating that a large portion of training samples are encoded in the parameters of such networks. In this work, we extend their findings in several directions, including reconstruction from multiclass and convolutional neural networks. We derive a more general reconstruction scheme which is applicable to a wider range of loss functions such as regression losses. Moreover, we study the various factors that contribute to networks' susceptibility to such reconstruction schemes. Intriguingly, we observe that using weight decay during training increases reconstructability both in terms of quantity and quality. Additionally, we examine the influence of the number of neurons relative to the number of training samples on the reconstructability.
Code: https://github.com/gonbuzaglo/decoreco
Neural networks are known to memorize training data despite their ability to generalize well to unseen test data [Zhang et al., 2021, Feldman, 2020]. This phenomenon was observed in both supervised settings [Haim et al., 2022, Balle et al., 2022, Loo et al., 2023] and in generative models [Carlini et al., 2019, 2021, 2023]. These works shed an interesting light on generalization, memorization and explainability of neural networks, while also posing a potential privacy risk.
Current reconstruction schemes from trained neural networks are still very limited and often rely on unrealistic assumptions, or operate within restricted settings. For instance, Balle et al. [2022] propose a reconstruction scheme based on the assumption of having complete knowledge of the training set, except for a single sample. Loo et al. [2023] suggest a scheme which operates under the NTK regime [Jacot et al., 2018], and assumes knowledge of the full set of parameters at initialization. Reconstruction schemes for unsupervised settings are specifically tailored for generative models and are not applicable for classifiers or other supervised tasks.
Recently, Haim et al. [2022] proposed a reconstruction scheme from feed-forward neural networks under logistic or exponential loss for binary classification tasks. Their scheme requires only knowledge of the trained parameters, and relies on theoretical results about the implicit bias of neural networks towards solutions of the maximum margin problem [Lyu and Li, 2019, Ji and Telgarsky, 2020]. Namely, neural networks are biased toward KKT points of the max-margin problem (see Theorem 3.1). By utilizing the set of conditions that KKT points satisfy, they devise a novel loss function that allows for reconstruction of actual training samples. They demonstrate reconstruction from models trained on common image datasets (CIFAR10 [Krizhevsky et al., 2009] and MNIST [LeCun et al., 2010]).
In this work, we expand the scope of neural networks for which we have evidence of successful sample memorization, by demonstrating sample reconstruction. Our contributions are as follows:
- We extend the reconstruction scheme of Haim et al. [2022] to a multiclass setting (Fig. 1). This extension utilizes the implicit bias result from Lyu and Li [2019] to multiclass training. We analyze the effects of the number of classes on reconstructability, and show that models become more susceptible to sample reconstruction as the number of classes increases.
- We devise a reconstruction scheme that applies for general loss functions, assuming that the model is trained with weight decay. We demonstrate reconstruction from models trained on regression losses.
- We investigate the effects of weight decay and show that for certain values, weight decay increases the vulnerability of models to sample reconstruction. Specifically, it allows us to reconstruct training samples from a convolutional network, while Haim et al. [2022] only handled MLPs.
- We analyse the intricate relation between the number of samples and the number of parameters in the trained model, and their effect on reconstructability. We also demonstrate successful reconstruction from a model trained on 5,000 samples, surpassing previous results that focused on models trained on up to 1,000 samples.
Memorization and Samples Reconstruction. There is no consensus on the definition of the term "memorization" and different works study this from different perspectives. In ML theory, memorization usually refers to label (or, model's output) memorization [Zhang et al., 2016, Arpit et al., 2017, Feldman, 2020, Feldman and Zhang, 2020, Brown et al., 2021], namely, fitting the training set. Memorization in the input domain is harder to show, because in order to demonstrate its occurrence one has to reconstruct samples from the model. Balle et al. [2022] demonstrated reconstruction of one training sample, assuming knowledge of all other training samples and Haim et al. [2022] demonstrated reconstruction of a substantial portion of training samples from a neural network classifier. Loo et al. [2023] extend their work to networks trained under the NTK regime [Jacot et al., 2018] and explore the relationship to dataset distillation. Several works have also studied memorization and samples reconstruction in generative models like autoencoders [Erhan et al., 2009, Radhakrishnan et al., 2018], large language models [Carlini et al., 2021, 2019, 2022] and diffusion-based image generators [Carlini et al., 2023, Somepalli et al., 2022, Gandikota et al., 2023, Kumari et al., 2023].
Inverting Classifiers. Optimizing a model's input as to minimize a class score is the common approach for neural network visualization [Mahendran and Vedaldi, 2015]. It usually involves using input regularization [Mordvintsev et al., 2015, Ghiasi et al., 2022], GAN prior [Nguyen et al., 2016, 2017] or knowledge of batch statistics [Yin et al., 2020]. Fredrikson et al. [2015], Yang et al. [2019] showed reconstruction of training samples using similar approach, however these methods are limited to classifiers trained with only a few samples per class. Reconstruction from a federated-learning setup [Zhu et al., 2019, He et al., 2019, Hitaj et al., 2017, Geiping et al., 2020, Yin et al., 2021, Huang et al., 2021, Wen et al., 2022] involve attacks that assume knowledge of training samples' gradients (see also Wang et al. [2023] for a theoretical analysis). In this work we do not assume any knowledge on the training data and do not use any priors other than assuming bounded inputs.
In this section, we provide an overview of the fundamental concepts and techniques required to understand the remainder of the paper, focusing on the fundamentals laid by Haim et al. [2022] for reconstructing training samples from trained neural networks.
Theoretical Framework. Haim et al. [2022] builds on the theory of implicit bias of gradient descent. Neural networks are commonly trained using gradient methods, and when large enough, they are expected to fit the training data well. However, it is empirically known that these models converge to solutions that also generalize well to unseen data, despite the risk of overfitting. Several
works pointed to this "implicit bias" of gradient methods as a possible explanation. Soudry et al. [2018] showed that linear classifiers trained with gradient descent on the logistic loss converge to the same solution as that of a hard-SVM, meaning that they maximize the margins. This result was later extended to non-linear and homogeneous neural networks by Lyu and Li [2019], Ji and Telgarsky [2020]:
Theorem 3.1 (Paraphrased from Lyu and Li [2019], Ji and Telgarsky [2020]) Let  $\Phi(\theta; \cdot)$  be a homogeneous  $^2$ ReLU neural network. Consider minimizing the logistic loss over a binary classification dataset  $\{(\mathbf{x}_i, y_i)\}_{i=1}^n$  using gradient flow. Assume that there exists time  $t_0$  where the network classifies all the samples correctly. Then, gradient flow converges in direction to a first order stationary point (KKT point) of the following maximum-margin problem:
$$
\min  _ {\boldsymbol {\theta}} \frac {1}{2} \| \boldsymbol {\theta} \| ^ {2} \quad s. t. \quad \forall i \in [ n ] y _ {i} \Phi (\boldsymbol {\theta}; \mathbf {x} _ {i}) \geq 1. \tag {1}
$$
A KKT point of Eq. (1) is characterized by the following set of conditions:
$$
\forall j \in [ p ], \quad \boldsymbol {\theta} _ {j} - \sum_ {i = 1} ^ {n} \lambda_ {i} \nabla_ {\boldsymbol {\theta} _ {j}} \left[ y _ {i} \Phi (\boldsymbol {\theta}; \mathbf {x} _ {i}) \right] = 0 \tag {2}
$$
$$
\forall i \in [ n ], \quad y _ {i} \Phi (\boldsymbol {\theta}; \mathbf {x} _ {i}) \geq 1 \tag {3}
$$
$$
\forall i \in [ n ], \quad \lambda_ {i} \geq 0 \tag {4}
$$
$$
\forall i \in [ n ], \quad \lambda_ {i} = 0 \text {i f} y _ {i} \Phi (\boldsymbol {\theta}; \mathbf {x} _ {i}) \neq 1 \tag {5}
$$
Reconstruction Algorithm. Haim et al. [2022] demonstrated reconstructing samples from the training set of such classifiers by devising a reconstruction loss. Given a trained classifier  $\Phi (\mathbf{x};\pmb {\theta})$  they initialize a set of  $\{(\mathbf{x}_i,y_i)\}_{i = 1}^m$  and  $\{\lambda_i\}_{i = 1}^m$  , and optimize  $\mathbf{x}_i,\lambda_i$  to minimize the following loss function:
$$
L = \underbrace {\left\| \boldsymbol {\theta} - \sum_ {i = 1} ^ {m} \lambda_ {i} \nabla_ {\boldsymbol {\theta} _ {j}} \left[ y _ {i} \Phi (\boldsymbol {\theta} ; \mathbf {x} _ {i}) \right] \right\|} _ {L _ {\text {s t a t i o n a r y}}} + \underbrace {\sum_ {i = 1} ^ {m} \max  \left\{- \lambda , - \lambda_ {\min } \right\}} _ {L _ {\lambda}} + L _ {\text {p r i o r}} \tag {6}
$$
Where  $L_{\mathrm{prior}}$  is simply bounding each pixel value at  $[-1, 1]^3$ . The number of training samples  $n$  is unknown. However, setting  $m > 2n$ , where  $\{y_i\}$  are set in a balanced manner allows reconstructing samples with any label distribution. The  $\mathbf{x}_i$ 's are initialized from the Gaussian distribution  $\mathcal{N}(0, \sigma_x^2 \mathbb{I})$ , and  $\lambda_{\min}, \sigma_x$  are hyperparameters. We note that the homogeneity condition from Theorem 3.1 is not necessarily a practical limitation of this reconstruction scheme, as already in Haim et al. [2022] they show reconstructions from a non-homogeneous network.
Analysing and Summarizing Results. The optimization in Eq. (6) is executed  $k$  times (for different hyperparameters) and results in  $km$  outputs  $(\{\hat{\mathbf{x}}_i\}_{i=1}^{km})$  that we term candidates, as they are candidates to be reconstructed training samples. To quantify the success of the reconstruction process, each training sample is matched with its nearest-neighbour from the  $km$  candidates. The "quality" of reconstruction is then measured using SSIM Wang et al. [2004] (see full details in Appendix A.1).
An important corollary of the set of KKT conditions Eqs. (2) to (5) is that the parameters of the trained model only depend on gradients of samples that are closest to the decision boundary, the so-called "margin-samples" (see end of section 3.2 in Haim et al. [2022]). Therefore, a good visual summary for analysing reconstruction from a trained model is by plotting the reconstruction quality (SSIM) against the distance from the decision boundary  $(|\Phi(\mathbf{x}_i)|)$ . We also utilize such visualizations.
Assessing the Quality of Reconstructed Samples. Determining whether a candidate is a correct match for some training sample is as hard as finding a good image similarity metric. No synthetic metric such as SSIM or L2 norm can perfectly align with human perception. Perceptual similarity
${}^{2}\mathrm{\;A}$  classifier  $\Phi$  is homogeneous w.r.t to  $\theta$  if there exists  $L \in  \mathbb{R}$  s.t.  $\forall c \in  \mathbb{R},\mathbf{x} : \phi \left( {\mathbf{x};c\theta }\right)  = {c}^{L}\phi \left( {\mathbf{x};\theta }\right)$
3Formally:  $L_{\mathrm{prior}} = \sum_{i = 1}^{m}\sum_{k = 1}^{d}\max \{\max \{\mathbf{x}_{i,k} - 1,0\} ,\max \{-\mathbf{x}_{i,k} - 1,0\} \} .$
metrics (e.g., LPIPS [Zhang et al., 2018]) build on top of pre-trained classifiers trained on ImageNet [Deng et al., 2009], and are not effective for the image resolution in this work (up to  $32 \times 32$  pixels). We have observed heuristically that candidates with SSIM score higher than about 0.4 are indeed visually similar to their nearest neighbor training sample. Hence, in this work we say that a certain candidate is a "good reconstruction" if its SSIM score with its nearest neighbor is at least 0.4. Also see discussion in Appendix A.2.
4 Reconstructing Data from Multi-Class Classifiers
We demonstrate that training set reconstruction can be extended to multi-class classification tasks.
Figure 1: Reconstructed training samples from a multi-class MLP classifier that was trained on 500 CIFAR10 images. Each column corresponds to one class and shows the 10 training samples (red) that were best reconstructed from this class, along with their reconstructed result (blue).
4.1 Theory
The extension of the implicit bias of homogeneous neural networks to the multi-class settings is discussed in Lyu and Li [2019] (Appendix G): Let  $S = \{(\mathbf{x}_i, y_i)\}_{i=1}^n \subseteq \mathbb{R}^d \times [C]$  be a multi-class classification training set where  $C \in \mathbb{N}$  is any number of classes, and  $[C] = \{1, \dots, C\}$ . Let  $\Phi(\pmb{\theta}; \cdot) : \mathbb{R}^d \to \mathbb{R}^C$  be a homogeneous neural network parameterized by  $\pmb{\theta} \in \mathbb{R}^p$ . We denote the  $j$ -th output of  $\Phi$  on an input  $\mathbf{x}$  as  $\Phi_j(\pmb{\theta}; \mathbf{x}) \in \mathbb{R}$ . Consider minimizing the standard cross-entropy loss and assume that after some number of iterations the model correctly classifies all the training examples. Then, gradient flow will converge to a KKT point of the following maximum-margin problem:
$$
\min  _ {\boldsymbol {\theta}} \frac {1}{2} \| \boldsymbol {\theta} \| ^ {2} \quad \text {s . t .} \quad \Phi_ {y _ {i}} (\boldsymbol {\theta}; \mathbf {x} _ {i}) - \Phi_ {j} (\boldsymbol {\theta}; \mathbf {x} _ {i}) \geq 1 \quad \forall i \in [ n ], \forall j \in [ C ] \backslash \left\{y _ {i} \right\}. \tag {7}
$$
A KKT point of the above optimization problem is characterized by the following set of conditions:
$$
\boldsymbol {\theta} - \sum_ {i = 1} ^ {n} \sum_ {j \neq y _ {i}} ^ {c} \lambda_ {i, j} \nabla_ {\boldsymbol {\theta}} \left(\Phi_ {y _ {i}} (\boldsymbol {\theta}; \mathbf {x} _ {i}) - \Phi_ {j} (\boldsymbol {\theta}; \mathbf {x} _ {i})\right) = \mathbf {0} \tag {8}
$$
$$
\forall i \in [ n ], \forall j \in [ C ] \backslash \left\{y _ {i} \right\}: \Phi_ {y _ {i}} (\boldsymbol {\theta}; \mathbf {x} _ {i}) - \Phi_ {j} (\boldsymbol {\theta}; \mathbf {x} _ {i}) \geq 1 \tag {9}
$$
$$
\forall i \in [ n ], \forall j \in [ C ] \backslash \left\{y _ {i} \right\}: \lambda_ {i, j} \geq 0 \tag {10}
$$
$$
\forall i \in [ n ], \forall j \in [ C ] \backslash \left\{y _ {i} \right\}: \lambda_ {i, j} = 0 \text {i f} \Phi_ {y _ {i}} (\boldsymbol {\theta}; \mathbf {x} _ {i}) - \Phi_ {j} (\boldsymbol {\theta}; \mathbf {x} _ {i}) \neq 1 \tag {11}
$$
A straightforward extension of a reconstruction loss for a multi-class model that converged to the conditions above would be to minimize the norm of the left-hand-side (LHS) of condition Eq. (8)
(namely, optimize over  $\{\mathbf{x}_i\}_{i=1}^m$  and  $\{\lambda_{i,j}\}_{i \in [n], j \in [C] \setminus y_i}$  where  $m$  is a hyperparameter). However, this straightforward extension failed to successfully reconstruct samples. We therefore propose the following equivalent formulation.
Note that from Eqs. (9) and (11), most  $\lambda_{i,j}$  zero out: the distance of a sample  $\mathbf{x}_i$  to its nearest decision boundary,  $\Phi_{y_i} - \max_{j\neq y_i}\Phi_j$ , is usually achieved for a single class  $j$  and therefore (from Eq. (11)) in this case at most one  $\lambda_{i,j}$  will be non-zero. For some samples  $\mathbf{x}_i$  it is also possible that all  $\lambda_{i,j}$  will vanish. Following this observation, we define the following loss that only considers the distance from the decision boundary:
$$
L _ {\text {m u l t i c l a s s}} \left(\mathbf {x} _ {1}, \dots , \mathbf {x} _ {m}, \lambda_ {1}, \dots , \lambda_ {m}\right) = \left\| \boldsymbol {\theta} - \sum_ {i = 1} ^ {m} \lambda_ {i} \nabla_ {\boldsymbol {\theta}} \left[ \Phi_ {y _ {i}} \left(\mathbf {x} _ {i}; \boldsymbol {\theta}\right) - \max  _ {j \neq y _ {i}} \Phi_ {j} \left(\mathbf {x} _ {i}; \boldsymbol {\theta}\right) \right] \right\| _ {2} ^ {2} \tag {12}
$$
Eq. (12) implicitly includes Eq. (11) into the summation in Eq. (8), thereby significantly reducing the number of summands and simplifying the overall optimization problem.
While the straightforward extension failed to successfully reconstruct samples, solving Eq. (12) enabled reconstruction from multiclass models (see Fig. 1 and results below). We attribute this success to the large reduction in the number of optimization variables, which simplifies the overall optimization problem ( $n$  variables in Eq. (12) compared to  $C \cdot n$  variables in the straightforward case, which is significant for large number of classes  $C$ ).
We also use the same  $L_{\lambda}$  and  $L_{\mathrm{prior}}$  as in Eq. (6), and set  $\{y_i\}$  in a balanced manner (uniformly on all classes). While setting  $m = C \cdot n$  allows reconstructing any label distribution, in our experiments we focus on models trained on balanced training sets, and use  $m = 2n$  which works sufficiently well. An intuitive way to understand the extension of the binary reconstruction loss Eq. (6) to multi-class reconstruction Eq. (12) is that the only difference is the definition of the distance to nearest boundary, which is the term inside the square brackets in both equations.
We compare between reconstruction from binary classifiers, as studied in Haim et al. [2022], and reconstruction from multi-class classifiers by using the novel loss function Eq. (12). We conduct the following experiment: we train an MLP classifier with architecture  $D$ -1000-1000- $C$  on samples from the CIFAR10 [Krizhevsky et al., 2009] dataset. The model is trained to minimize the cross-entropy loss with full-batch gradient descent, once with two classes (250 samples per class) and once for the full 10 classes (50 samples per class). Both models train on the same amount of samples (500). The test set accuracy of the models is  $77\% /32\%$  respectively, which is far from random ( $50\% /10\%$  resp.). See implementation details in Appendix B.
To quantify the quality of our reconstructed samples, for each sample in the original training set we search for its nearest neighbour in the reconstructed images and measure the similarity using SSIM [Wang et al., 2004] (higher SSIM means better reconstruction). In Fig. 2 we plot the quality of reconstruction (in terms of SSIM) against the distance of the sample from the decision boundary  $\Phi_{y_i}(\mathbf{x}_i;\pmb {\theta}) - \max_{j\neq y_i}\Phi_j(\mathbf{x}_i;\pmb {\theta})$ . As seen, a multi-class classifier yields much more samples that are vulnerable to being reconstructed.
Figure 2: Multi-class classifiers are more vulnerable to training-set reconstruction. For a training set of size 500, a multi-class model (left) yields 101 reconstructed samples with good quality (SSIM>0.4), compared to 40 in a binary classification model (right).
We examine the relation between the ability to reconstruct from a model and the number of classes on which it was trained. Comparing between two models trained on different number of classes is not immediately clear, since we want to isolate the effect of the number of classes from the size of the dataset (it was observed by Haim et al. [2022] that the number of reconstructed samples decreases as the total size of the training set increases). We therefore train models on training sets with varying number of classes  $(C\in \{2,3,4,5,10\})$  and varying number of samples per class (1, 5, 10, 50). The results are visualized in Fig. 3a. As seen, for models with same number of samples per class, the ability to reconstruct increases with the number of classes, even though the total size of the training set is larger. This further validates our hypothesis that the more classes, the more samples are vulnerable to reconstruction (also see Appendix C).
Another way to validate this hypothesis is by showing the dependency between the number of classes and the number of "good" reconstructions (SSIM  $>0.4$ ) - shown in Fig. 3b. As can be seen, training on multiple classes yields more samples that are vulnerable to reconstruction. An intuitive explanation, is that multi-class classifiers have more "margin-samples". Since margin-samples are more vulnerable to reconstruction, this results in more samples being reconstructed from the model.
(a) Analysing the relation between number of classes and number of samples per class in reconstruction from multiclass Classifiers.
(b) Number of "good" reconstructions increases with number of classes and the samples per class
Figure 3: Evaluating the effect of multiple classes on the ability to reconstruct. We show reconstructions from models trained with different numbers of classes and different numbers of samples per class. As seen, multiple classes result in more reconstructed samples.
5 Data Reconstruction with General Loss Functions
We demonstrate that data reconstruction can be generalized to a larger family of loss functions. Haim et al. [2022] and Section 4 only considered a reconstruction scheme based on the implicit bias of gradient methods trained with cross-entropy loss. For other loss functions, such as the square loss, a precise characterization of the implicit bias in nonlinear networks does not exist [Vardi and Shamir, 2021]. Hence, we establish a reconstruction scheme for networks trained with explicit regularization, i.e., with weight decay. We show that as long as the training involves a weight-decay term, we can derive a reconstruction objective that is very similar to the previous objectives in Eqs. (6) and (12).
5.1 Theory
Let  $\ell(\Phi(\mathbf{x}_i; \boldsymbol{\theta}), y_i)$  be a loss function that gets as input the predicted output of the model  $\Phi$  (parametrized by  $\boldsymbol{\theta}$ ) on an input sample  $\mathbf{x}_i$ , and its corresponding label  $y_i$ . The total regularized loss:
$$
\mathcal {L} = \sum_ {i = 1} ^ {n} \ell (\Phi (\mathbf {x} _ {i}; \boldsymbol {\theta}), y _ {i}) + \lambda_ {\mathrm {W D}} \frac {1}{2} \| \boldsymbol {\theta} \| ^ {2}. \tag {13}
$$
Assuming convergence  $(\nabla_{\theta}\mathcal{L} = 0)$ , the parameters should satisfy the following:
$$
\boldsymbol {\theta} = \sum_ {i = 1} ^ {n} \ell_ {i} ^ {\prime} \nabla_ {\boldsymbol {\theta}} \Phi (\mathbf {x} _ {i}; \boldsymbol {\theta}) \tag {14}
$$
where  $\ell_i' = -\frac{1}{\lambda_{WD}}\frac{\partial\ell(\Phi(\mathbf{x}_i;\pmb{\theta}),y_i)}{\partial\Phi(\mathbf{x}_i;\pmb{\theta})}$ . This form (which is similar to the condition in Eq. (2)), allows us to define a generalized reconstruction loss for models trained with a weight-decay term:
$$
L _ {r e c} \left(\mathbf {x} _ {1}, \dots , \mathbf {x} _ {m}, \lambda_ {1}, \dots , \lambda_ {m}\right) = \| \boldsymbol {\theta} - \sum_ {i = 1} ^ {n} \lambda_ {i} \nabla_ {\boldsymbol {\theta}} \Phi \left(\mathbf {x} _ {i}; \boldsymbol {\theta}\right) \| _ {2} ^ {2} \tag {15}
$$
As before, we also include the same  $L_{\mathrm{prior}}$  as in Section 3. It is straightforward to see that  $L_{rec}$  is a generalization of the reconstruction loss in Eq. (6) ( $y_i$  could be incorporated into the  $\lambda_i$  term).
Figure 4: Reconstruction from general losses (column) for various training set sizes (row), using Eq. (15). "Harder" samples (with higher loss) are easier to reconstruct.