Simulating Training Dynamics to Reconstruct Training Data from Deep Neural Networks
Authors: Hanling Tian
Yuhang Liu
Mingzhen He
Zhengbao He
Zhehao Huang
Ruikai Yang
Xiaolin Huang
Venue: ICLR 2025 Poster (2025)
Keywords: Dataset reconstruction
Training dynamics
Memorization
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: Whether deep neural networks (DNNs) memorize the training data is a fundamental open question in understanding deep learning. A direct way to verify the memorization of DNNs is to reconstruct training data from DNNs’ parameters. Since parameters are gradually determined by data throughout training, characterizing training dynamics is important for reconstruction. Pioneering works rely on the linear training dynamics of shallow NNs with large widths, but cannot be extended to more practical DNNs which have non-linear dynamics. We propose Simulation of training Dynamics (SimuDy) to reconstruct training data from DNNs. Specifically, we simulate the training dynamics by training the model from the initial parameters with a dummy dataset, then optimize this dummy dataset so that the simulated dynamics reach the same final parameters as the true dynamics. By incorporating dummy parameters in the simulated dynamics, SimuDy effectively describes non-linear training dynamics. Experiments demonstrate that SimuDy significantly outperforms previous approaches when handling non-linear training dynamics, and for the first time, most training samples can be reconstructed from a trained ResNet’s parameters.

Published as a conference paper at ICLR 2025
SIMULATING TRAINING DYNAMICS TO RECONSTRUCT TRAINING DATA FROM DEEP NEURAL NETWORKS
Hanling Tian, Yuhang Liu, Mingzhen He, Zhengbao He, Zhehao Huang, Ruikai Yang, Xiaolin Huang" Institute of Image Processing and Pattern Recognition, Shanghai Jiao Tong University (hanlingtian, yuhangliu, mingzhen he, Istefamie, kinght h, ruikai.yang. xiaolinhuang] @ @sjtu.edu.cn
ABSTRACT
Whether deep neural networks (DNNs) memorize the training data is funda- mental open question in understanding deep learning. A direct way to verify the memorization of DNNs is to reconstruct training data from DNNs' parameters. Since parameters are gradually determined by data throughout training, charac- terizing training dynamics is important for reconstruction. Pioneering works rely on the linear training dynamies of shallow NNs with large widths, but cannot be extended to more practical DNNs which have non-linear dynamics. We pro- pose Simulation of training Dynamics (SimuDy) to reconstruct training data from simulate the training dynamics by training the model from the initial parameters with a dummy dataset, then optimize this dummy dataset that the simulated dynamics reach the same final parameters as the true dynamics. By incorporating dummy parameters in the simulated dynamics, SimuDy effectively describes non-linear training dynamics. Experiments demon- strate that SimuDy significantly outperforms previous approaches when handling non-linear training dynamics. and for the first time, most training samples can be reconstructed from a trained ResNet's 's parameters. Our code is available at /ettp::/gittuut teriiihuu.c.om com Bl wellleelood////. Simuu/
DNNs. Specifically,
INTRODUCTION
Deep neural networks (DNNs) have shown remarkable performance and generalization across var- ious tasks (Hinton et al.) 2012 Devlin et al., 2019 Wu et al. 2022), due to their powerful learning capabilities. An intriguing question is what DNNs have learned even memorized. This memoriza- tion is closely related to the generalization capability (Feldman 2020 Feldman & Zhang (2020), and also raises concerns about the potential leakage of private information within training data (Shokri
et al. 2017 Carlini et al. 2019 2021: 2023).
During training. DNNs' parameters are progressively adjusted by the training data. Roughly, when the learning algorithm and the initial parameters are given, the final parameters are determined by the training data. This mapping indicates that the training data may be memorized within the DNNs parameters. A direct way to support this guess is to reconstruct the training dataset from parameters. However, this reconstruction task challenging due to the complexity of the training process, which involves disentangling parameter changes at each step and decoupling each sample's contribution from the cumulative sum of gradients. Gradient inversion attack (Zhu et al., 2019) could recover data from the gradient of single step dozens of images. In fact, the two tasks are related but essentially different. Reconstructing training data from parameters are more difficult and necessitates deep understanding of training dynamics, as it involves recovering data from the cumulative effects of multiple training steps, rather than simply considering a single step's gradient. In other words, gradient inversion attack is the simplest case of reconstruction from parameters when the training process contains only one step.
To reconstruct training data from parameters, we need deeply study the training dynamics. The simplest training dynamics are linear, meaning the directions of gradients for each data point remain consistent, as seen cases for shallow NNs with infinite widths. For an approximately linear sce- nario: training a three-layer multilayer perceptron (MLP) model with 1000 neurons in each layer,
"Coreeponding author

---

Published as a conference paper at ICLR 2025
Figure 1: Reconstructed training samples from multi-class ResNet using our SimuDy(rows 1, 3), and the corresponding nearest neighbors from the training set (rows 4).
Haim et al. (2022) pioneer a reconstruction method based on the -uuihh-KKhhTTTkkke (KKT) con- ditions of certain max-margin problem (Lyu Li 2020 Ji Telgarsky 2020) where gradient flow converges in direction. This method successtully reconstructed about 50 out of 500 CIFAR-IC
training images from the MLP's final parameters. Similarly, Loo et (2024) reconstruct the train- ing data from NNs in the neural tangent kernel (NTK) (Jacot et al. 2018) regime, where the training dynamics are also linear.
However, the reconstruction should not be limited to linear training dynamics, which are rarely encountered in weal-world applications. The key to dataset reconstruction is accurately describing the gradients at each step since parameters and data linked via these gradients. In linear dynamics, the gradient directions remain consistent, so the training dynamics can be effectively characterized by even only the final parameters. While for non-linear dynamics, Loo al. (2024) make step by considering the linear combination of gradients from initial parameters and final parameters, achieving better reconstructions from MLPs than Haim al. (2022). Nevertheless, using the sum of gradients from initial and final static parameters to characterize dynamics, is still too simple for popular and practical cases, i.e., DNNs such ResNet (He et al., 2016) trained with mini-batch Stochastic Gradient Descent (SGD) Bottou 2010).
In this paper, we propose novel framework called "Simulation of training Dynamics" (SimuDy) to reconstruct the training data from parameters of trained DNNs. As the name suggested, our method constructs the simulated training dynamics by training the model from the initial parameters with dummy dataset initialized with random Gaussian noise. Then, we optimize the dummy dataset to make the simulated training dynamics converge to the same final parameters as the original training dynamics, with the dummy dataset progressively refined towards the real dataset. The detailed characterization of the non-linear training dynamics aids in data reconstruction from DNNs. Our results demonstrate that SimuDy trained using mini-batch SGD, where previous methods have failed. And an example using SimuDy is shown in Fig. [ For the first time, large portion of the training data can be reconstructed from a trained ResNet's parameters. Also, the reconstruction performance remains excellent even with unknown hyper-parameters of model training. Our contributions could be summarized follows:
reconstruct training data with high quality from a ResNet
We propose SimuDy to reconstruct training data from parameters of trained DNNs, which are more practical than MLPs in realistic applications.
Our method illustrates the importance of characterizing training dynamics for dataset construction, which would provide insights for memorization of DNNs.
Extensive experiments show that SimuDy outperforms previous methods when dealing with non-linear training dynamics. Additionally, we demonstrate our method's effective- ness and robustness with unknown hyper- parameer settings of model training.
RELATED WORK
Dataset Reconstruction. The pioneering work (Haim et al. 2022) has shown that training data can be reconstructed from trained homogeneous neural network parameters. They note that ho- mogeneous neural networks converge in direction the solution of max-margin problem (Lyu
2

---

Published as a conference paper at ICLR 2025
& Li. 2020 Ji & Telgarsky 2020). By optimizing dummy images and dual parameters to match Kuhn-Tuccer (KKT) conditions of the max-margin problem, training data is reconstructed.
Then, Buzaglo al. (2024) extend the reconstruction scheme of [Haim et al. (2022) to multi-class setting and MLPs with a convolutional layer in the front. Loo et al. (2024) provably reconstruct the entire training set for networks in the neural tangent kernel (NTK) aacot et al. 2018) regime where the directions of gradients for each data point does not change during training, i.e., the training dy- namics are linear. Toward non-linear dynamics, they take step linearly combine the initial and final gradients of data to represent parameter changes and get better reconstructions. Aforemen- tioned methods mainly focus on shallow and wide MLPs where the training dynamics are linear. Besides MLPs, we can reconstruct training data from practical DNNs with non-linear dynamics.
Gradient Inversion Attack. Gradient inversion attack aims to reconstruct training data from the batch gradient and the model's parameters. DLG (Zhu et al., 2019) is the pioneering optimization- based method that minimizes the distance between the ground
gradients and the dummy gradi- ents of batch of dummy samples. iDLG (Zhao et al. 2020) derives the ground-truth label from the gradient of the fully connected layer to improve the attack. Geiping et al. (2020) change previous Euclidean distance loss to cosine similarity loss and add total variation (Rudin et al.. 1992) as the image prior to reconstruct higher resolution images from ResNet (He et al. (016)). With strong batch normalization statistics. Yin et al. (2021) propose group consistency regularization to recover partial
images at a larger batch size even up to 48. Recently, the following works (Jeon et al. 2021 Li et al.)
2022 Fang et al., 2023 Zhang al. 2023) leverage pre-trained generative models to better utilize image prior and improve the quality of reconstructed images. Dataset reconstruction is related to the gradient inversion attack, both aim to recover images from different forms of gradients. How- ever, dataset reconstruction presents a greater challenge because the gradients come from a dynamic process, whereas gradient inversion attack only considers a single step's gradient.
SIMULATING TRAINING DYNAMICS
Task definition. Given neural network fe with the initial parameters O, and the final parameters 9 for multi-class image classification tasks, dataset reconstruction aims to invert the parameters to the training dataset D. Considering the learning algorithm A, which is characterized by the hyper- parameters H, the training process maps 'D to 0y: O, = An (D; 0n)- In this point of view, the dataset reconstruction is an inverse problem D = An (0,;0o). 0o).
Revisiting training. The inverse problem D = A A (0, 0o) is over-deternined in the context of deep learning because a DNN is usually peeerpaaaatttizedd i.e.. the dimension of 0, is significantly higher than that of the entire dataset, providing sufficient supervision to reconstruct the training data. However, the data information embedded into the parameters by iteratively updating the network parameters in the form of gradients. Therefore, to solve the inverse problem, we need to investigate the training process, which can be described by the following differential equation,
d@(t) dt with the constraints (0) = 0, and (TT) = 0,. In practical scenarios, this continuous process is discretized, enabling the parameter updates be performed in series of discrete steps, which makes it feasible to implement. Specifically, one of the most representative methods is SGD, where parameters are updated iteratively by moving against the gradient of the loss function, calculated on mini-batches of the data:
VeL(D, (()).
N
0. 0,
ΣV
n
E
1 2. EB. Benl
Vol (fer.s (z).
k=1j=1
OK.+1 = Ox.j - Σ
[BR.j] Vol (Joa. (z),
Oc+1.11 = OK.Ns
(3)
B.y.B...
where T is the total number of training epochs, N is the total number of batches, E is the loss function, Bx.j is the j-th batch of the k-th epoch, [Bx, is the batch size of Bg.j. 1... 0z.j is the model parameters of k-th epoch after j-th batch, (x. y) is data point in BRj. and 17 is the learning rate.
Linear training dynamics. The simplest case of dataset reconstruction is that the training dynamics linear approximately linear. In this case, the direction of gradients Vo keeps almost the same
3

---

Published as a conference paper at ICLR 2025
Algorithm 1 Reconstructing training data using SimuDy.
Input: Network function fe. initial parameters Oo. final parameters 0y. dataset size training learning rate 1,, training steps T, batch size [B], dissimilarity function d( +) optimizer Opt im;
Output: Reconstructed images via SimuDy;
1: Initialize dummy images & with random noise
2: Assign labels to images randomly, ensuring equal number of labels for each class
3: Randomly divide the dataset into batches of size [B, and the number of batches N
4: 01.1 <- 00
> Begin with the initial parameters
5: repeat
6: for k = 1 to 7 do
D Simulate the training process for T epochs Perform N updates for each epoch Compute the gradient for each update
7: for j = N do
8:
Ik. La.eBRu Bayl Vol(fo., (Ø),
9:
0 Ok. n grad_clip(ge.)
b Clip gradients and update parameters
10:
end for OK+1.1 ŌK,N41
12:
end for
/ 1.,++1
Get simulated final parameters D Compute reconstruction loss Update dummy images
14:
Drecon d ; 0.. , 0n) - a LIv(x)
Φ + Opt im 0 recoon a /
16:
until the reconstruction loss converges.
throughout the training process. Then, by leveraging the direction invariance, Eq. 2q.2 can be written as 0g- 0 = ED Aa Vol (Jo, (ax), yı), where A; is the scaling factor addressing the contribution of az.- Thus, the linear combination of gradients computed from the initial or final model parameters can effectively characterize training dynamics. In other words, with linear training dynamics, the dataset reconstruction could be reduced problem of extracting images from the gradients of single step batch of data.
Simulation of training dynamics. However, the linear training dynamics exist only for some ideal scenarios such as training shallow MLP with sufficient width (Haim et al.) (022). In practice, the training dynamics are typically non-linear. To reconstruct the dataset from practical DNNs, we should delve into the whole training dynamics traversing from initial parameters Og to final parameters 0,, rather than solely relying on gradients of these two sets of parameters to characterize the entire training process. Nevertheless, such training dynamics makes reconstruction for non-linear dynamics seemingly impossible. "Simulation of training Dynamics" (SimuDy) to reconstruct training data from practical DNNs with non-linear dynamics, i.c., we will construct new training dynamics and optimize the dummy dataset from noise original data such that the simulated training dynamics converge to the real dynamics. The pseudocode our proposed SimuDy is specified in Alg. [ SimuDy introduces additional simulated checkpoints in the training process, which would provide more accurate guidance than only using the initial and final parameters. At high level, our method can be presented as follows:
usually inaccessible, which this paper, we propose
min
d (An((X); Au (D;
PEF
where F. denotes the set of possible images, and d(.,)) :) is the non-negative dissimilarity function.
Loss design. Minimizing the dissimilarity between simulated final parameters O and real final pa- rameters 0, is supervise the simulated dynamics to be similar the underlying dynamics. We choose the cosine similarity loss following Geiping et al.] (2020), tion of the gradient can carry significant information (Charpiat et al. 2009 Koh Liang (017). Moreover, the value of cosine similarity loss is bounded in |-1, 1]. simplifying hyper-parameter adjustment for reconstruction. An alternative type of loss could be the sum of element-wise dis-
the high-dimensional direc-
tances, e.g., the Euclidean distance. And we show that cosine similarity loss outperforms Euclidean distance loss for data reconstruction in Appendix
Incorporating prior knowledge, represented by X' E F abstractly, further enhance reconstruction quality. For natural image classification problems, total variation (TV) is a widely-used term to describe the natural image manifold, which minimizes the variations between adjacent pixels to
4

---

Published as a conference paper at ICLR 2025
promote image smoothness. Thus, apply TV loss with the scaling factor to make the dummy images look more natural. Overall, the reconstruction loss is given as the following:
(2 = (0, 0,. , Oo) + TV(x).
(5)
0 loll 0, 0ol
We test the designed loss in the reconstruction from a ResNet-18 18 trained on 60 images. The model is trained with a batch size of 20 and a learning rate of 7e-3. Fig. 2 plots the relationship between the final loss value (in blue)
Given the correct batch size
0.35
Rnal Loss
Reconstruction Quality
0.30
0.15
Reconstruction Loss
and reconstruction quality (in red) measured by average Structure Similarity Index Measure (SSIM) (Wang et al.) 2004). In this figure, we fix the batch size but change the learning rates. One can observe that the loss and the qual- ity have quite good consistency: lower loss values corre- spond to better reconstruction performances, showing the rationality of our loss design.
0.10
0.05
0.10
0.00
0.05
suitable
0.00
Gradient clipping. We initialize dummy images with Gaussian noise. In the early stages of optimization, fully noisy images could cause gradient explosion. We apply gradient clipping in simulating dynamics to effectively prevent this issue. Additionally, even in the absence of gradient explosion, gradient clipping also simplifies image optimization. Since the training gradients of natural images generally have rela- tively small norms than noisy images, gradient clipping serves as a form of prior knowledge that guides the training process. Thus, this strategic of gradient clipping enhances both the stability and efficiency of optimization for dataset reconstruction.
0.004
0 0.006
0.008
0.010
learning rate
Figure 2: The relationship between final loss value and reconstruction quality.
Hyper-parameee settings. The hyper-parameters H, including the batch-size [B] and the learning rate 77, characterize the learning algorithm and thus partially influence the training process. Correct knowledge of H facilitates reconstruction, while unknown H make reconstruction more complex. In more practical cases, we need to design a strategy to set H for better reconstruction.
do grid search for good learning rate v by comparing the final loss. And the yellow suitable region indicates that using learning rates within this area for simulating training can achieve good reconstruction. Here, change the [B] and plot SSIM values for different in Fig. 3 Comparisons among the three figures show that the peak SSIM values across different [B] are similar. In other words, for different [B), as long as a suitable is paired, the quality is comparable to that obtained with the original H. Intuitively, when [B] decreases and the corresponding 7 increases, similar final parameters can be obtained for the same dataset, and vice versa. Using grid search for suitable with given [B], we can successfully reconstruct training data with unknown
Fig. 2 preliminarily shows that when the batch size [B] is fixed,
batch
Given batch
Lose
6.54
Loccar
Recoesradion Qudl
-339
2.50
Racroirucuen Duality
6.52
Recrrbtation
Recondruittn LOG5
Average SSI Score
Reconstruction Loos
Reconssructoon
+3.2
Averape SSW
6.00
3.45
katae region
subob reg on
sutsbll Ingien
2.2
8202
5/64
2.33
1:00
200x
6-b
116
loaming rate
lcaming rate
learning rate
(c)
Figure 3: The relationship between the initial loss at 50 steps and the final reconstruction quality for different given batch sizes.
However, the reconstruction loss might take much time to converge. We experimentally explore how early decrease in loss within the first few dozen steps relates to the final reconstruction quality. As
5

---

Published as a conference paper at ICLR 2025
shown in Fig. lower value of loss at 50 steps (in blue) indicates better reconstruction performance (in red). In summary, we preset [B] and tune by grid search based on the first dozens of steps' loss. Thus, suitable H for simulating the training dynamics can be identified effectively and efficiently.
4 EXPERIMENTS
4.1 EXPERIMENT SETUP
Our task is to reconstruct training data from parameters of DNNs. The experimental pipeline is follows. First, use the dataset D train model from the initial parameters Oo the final parameters y. Then, SimuDy is applied to reconstruct training data from the saved parameters do and y All training and reconstructing run one RTX 4090 GPU. We provide the code in the supplementary materials.
DNNs' Parameters. We mainly consider the ResNet-18 model and will reconstruct data from its parameters. Following the setting of Loo et al. (2024), we reconstruct dataset from the initial and final model parameters. Consequently, We focus the fine-tuning scheme, which is practical and popular in current applications. We train the model on a subset of CIFAR-10 (Krizhevsky et al,, 2009) from pre-trained initial parameters On to the final parameters 0,. The class distribution is balanced and the training algorithm is mini-bater gradient descent with shuffle on.
Reconstructing. Following the reconstructing protocol of previous works (Haim et al. 2022 Loo et al., 2024). the size of the original training set and the resolution of images are known for the reconstruction.
quantitatively evaluate the quality of the reconstructed images, we use SSIM (Wang et al.) 2004) to measure the similarity between the reconstructed and original data, and pair them based on the SSIM scores. The average SSIM score shown below the reconstructions, and the reconstructions are presented in descending order of SSIM scores.
4.2 RECONSTRUCTION PERFORMANCE
We first validate the effectiveness of our method on the reconstruction from MLPs, which are the
primary focus of previous approaches (Haim al. 2022, Buzaglo et al.) 2024 Loo et al. 2024). The comparison is conducted to highlight the superiority of approach. Since pre-training meaningless for MLPs, we train MLPs, which comprise three fully-connected layers with dimen- sions d- 1000-1000-0 (d is the dimension of the input), from seratch with SGD. As shown in Fig. 4 SimuDy gets an average 0.3374 SSIM score for reconstructions from the MLP trained on 100 im- ages, outperforming Loo al. ss method significantly.
(a) Reconstructions from MLP trained 100 images using Loo al.
SSIM 0.1384
(b) Reconstructions from MLP trained on 100 images using SimuDy, SSIM = 0.3374
Figure 4: Top 40 images reconstructed from MLP trained on 100 images using Loo et al. s and
SimuDy (rows 1, 3) respectively, and corresponding nearest neighbors from the dataset (rows 2, 4).

---

Published as a conference paper at ICLR 2025
The above shows that for MLPs, SimuDy has better reconstruction performance than those methods which are based on linear training dynamics. For more practical DNNs, the large number of parame- ters and complex structures, including convolutional layers and skip connections, make the training dynamics much more non-linear than MIPs. In this case, Loo et al.'s method only gets 0.0774 and SimuDy gets 0. 0.1982 when reconstructing training data from ResNet trained on 50 images. The advantage on visual effect is also significant as shown in Fig. And for more comprehensive comparison against baselines, the reconstruction results of Buzaglo al. method are presented in Appendix
aa) Reconstructions from ResNet trained on images using Loo et al. =
(b) Reconstructions from ResNet trained on 50 images using SimuDy, SSIM 0.1922
Figure 5: Top 40 images reconstructed from ResNet trained on 50 images using Loo et al. is and SimuDy (rows 1, respectively, and corresponding nearest neighbors from the dataset (rows 4).
Intuitively, as the training dataset size increases, the dataset reconstruction becomes more challeng- ing, thus resulting in decline in the reconstruction quality. For the reconstruction of 20 images shown in Fig. [ and 50 images shown Fig. SimuDy can reconstruct nearly all data with high quality. And for 120 images shown in Fig. 6 SimuDy can also reconstruct about even 80 images. Moreover, as the size dataset increases, the training dynamics become more non-linear, leading to rapid decline in the reconstructing performance of Buzaglo al.]s and Loo et al. s method, which even fails when the dataset size reaches 120. The qualitative reconstruction results across variable-size datasets using different methods are presented in Appendix C.8
Figure 6: Top 80 images reconstructed from ResNet trained on 120 images using SimuDy (rows 1,
3, 5. 7), and their corresponding nearest-neight neighbos from the training dataset (rows 4, 6, 8). The average SSIM score is 0.1 196.

---

Published as a conference paper at ICLR 2025
4.3 RECONSTRUCTIONS WITH UNKNOWN HYPER-PARE PIIER-PARMMTERSS
In the above experiments, we assume that the batch size and the learning rate of the training process known, which helps describe the relationship between parameters and data. For more practical use, we design strategy to identify a suitable learning rate for the given batch size which is specified in Section 3 Here, we report the reconstruction results from ResNet trained on 60 images with settings in Fig. 7 from which one can observe that even the guess
different training on batch-size not accurate, the performance keeps good.
Figure 7: Reconstructions from ResNet trained on 60 images with batch size [8]] 20. Different guesses the batch size yield similar and good performance: Rows and are the images recon- structed by setting [B] 20; rows 2 and 6 are by setting [B] 32; rows 3 and are by setting [ = 12, and rows and are the original training samples.
4.4 INITIALIZATION OF DUMMY IMAGES
Different initializations of dummy images would influence the reconstruction quality. In the above experiments,
random Gaussian noise the initialization. Another seemingly reasonable choice is to use other natural images. Fig. X iillustrates the reconstruction process from the two dif- ferent initialization. One can see that although the main object can reconstructed from natural image initialization, the background seems be seriously influenced by the initialization. For example, the background of the reconstructed cat (row 2) is more brighter since the initial image a white dog. The reason may be the fact that a trained classification model mainly focuses on the main object, and changes in the background minimally affect the gradients, thus the parameter changes carry less information of backgrounds than that of main body.
Ground Truth
Steps
2000
7500
12000
18000
24000
30000
Figure 8: The reconstruction process of different initializations. The last column shows the ground truth, while the other columns the top 2 images are intervals initialized from natural images, and the bottom 2 images from random noise.
8

---

Published as a conference paper at ICLR 2025
4.5 RECONSTRUCTIONS OF DUMMY DATASETS OF DIFFERENT SIZES
In previous works (Haim et al. 2022 Buzaglo et al. 2024 Loo et al. 2024), the size of training
dataset is necessitated for effective dataset reconstruction. In this section, evaluate the effective- ness of our method without known original dataset size.
Given parameters of the ResNet trained on 40 images, we construct dummy datasets of different sizes for reconstructions. When the size of dummy dataset is 20, smaller than that of the original datset, the cosine similarity loss of SimuDy fluctuates around 0.2 while good reconstructions always have losses below 0.1. This suggests that dataset with merely 20 samples can guide the model to traverse from 9n to y The reconstructions are shown in the left of Fig. 9 Although these images appear unconventional, they still contain clues of the original data. As shown in the right part of Fig. 9 the reconstructed image seems like the fusion of two frog images from the original dataset.
Figure Reconstructions from ResNet trained on 40 images using a dummy dataset of size 20.
With larger dummy dataset of size 60, the loss value can be refined to about 0.08, indicating well match of training dynamics. The images from the original dataset are paired with their nearest neighbors from the dummy dataset based the SSIM scores, leaving remaining images in the dummy dataset unmatched. As shown Fig. T0)a), a), we successfully reconstruct the original dataset of size 40 using dummy dataset of size 60. And the unmatched images shown in Fig. [T),), appearing also unconventional but different from those in Fig. 9. as there are few clues of original data in unmatched images.
The contribution of each data point from the dummy dataset to the parameter changes is calculated by the norm of overall gradients throughout the training dynamics. We observe that the matched data's average norm of total gradients is 1.5458, ranging from 1.1204 to 1.9478. In contrast, for unmatched data, the average norm only 0.5466, with extremes of 0.2172 to 0.8438. These findings validate our approach's 's ability to reconstruct dataset with a larger size setting and transform extra dummy images from random noise to insignificant images which have relatively small gradients during training.
(a) Top 40 images from the dummy dataset of size 60, SSIM 0.2295
(b) Bottom 20 images from dummy dataset size 60
Figure 10: Reconstructions from ResNet trained on 40 images using dummy dataset of size 60.
With a smaller dataset size, our approach tries to amalgamate multiple images' information into single image, which compromises the quality of recovery. However, when handling unknown dataset size, we can increase the size of dummy dataset as SimuDy can reconstruct original dataset with high quality while optimizing extra dummy images into insignificant images.

---

Published as a conference paper at ICLR 2025
5 CONCLUSION, LIMITATION, AND FUTURE WORK
In this work, we propose SimuDy to reconstruct training data from the parameters of DNNs. This reconstruction task may open up possibilities to the study the memorization of models which is quite challenging. Previous works are restricted to simple NNs with linear training dynamics, while our method can be used for ResNet, towards more practical and meaningful reconstructing. The core progress is that SimuDy simulates the training dynamics which are crucial for reconstructing training data, since parameters are adjusted iteratively by data during training. Successful reconstructions illustrate the importance of characterizing training dynamics for extracting data from parameters, which provides insights for memorization DNNs and helps better understand deep learning.
Our proposed method, SimuDy, successfully reconstructs training data from parameters of DNNs, though it does not completely solve dataset reconstruction. We acknowledge the limitations of our approach and encourage further research.
One primary limitation is that our method's performance declines the dataset size increases, due to the increased uncertainty in the optimization problem of decoupling gradients of more data. This highlights the need for more effective solutions to more complex dataset reconstruction optimiza- tion challenges. Additionally, the simulation of training dynamics necessitates the preservation of the entire model training computation graph. Thus our method is restricted to the GPU memory when dealing with large datasets. Future efforts should focus characterizing training dynamics more efficiently, thus minimizing computational resource demands, such as by leveraging the low- dimensional nature of the model training process. Last but not least, although our method could reconstruct about only 100 images from parameters, this is sufficient to show that indeed there is memorization in DNNs, providing promising tool for investigating deep learning memory. We hope our work will illuminate deep learning interpretability and stimulate further exploration into the relationship between memorization and generalization of DNNs on larger datasets.
ACKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers for their insightful comments.
The research leading to these results has received funding from National Natural Science Foundation of China (62376155), Shanghai Municipal Science and Technology Research Program Major Project (2021SHZDZX0102).
REFERENCES
Mislav Balunovic, Dimitar Dimitrov, Nikola Jovanović, and Martin Vechev. Lamp: Extracting text from gradients with language model priors. Advances in Neural Information Processing Systems, 35:7641-7654, 2022.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT 19th International Conference on Computational StatisticsParis France, Au- gust 22-27, 2010 Keynote, Invited and Contributed Pupers, pp. 177-186. Springer, 2010.
Gon Buzaglo, Niv Haim, Gilad Yehudai, Gal Vardi, Yakir Oz, Yaniv Nikankin, and Michal Irani. Deconstructing data reconstruction: Multiclass, weight decay and general losses. Advances in Neural Information Processing Systems, 36, 2024.
Nicholas Carlini, Chang Liu, Úlfar Frlingsson, Jernej Kos, and Dawn Song. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium (USENIX Security 19), pp. 267-284, 2019.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Frlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 2/), pp- 2633-2650, 2021.
Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In The Eleventh International Conference Learning Representations, 2023.
10

---

Published as a conference paper at ICLR 2025
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. Advances in Neural Information Processing Systems, 32, 2019.
Jacob Devlin, Ming- Wing-Wei Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. In North American Chapter of the Associ- ation,for Computational Linguistics, 2019.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. In The Ninth International Conference on Learning Representations, 2021.
Hao Fang, Bin Chen, Xuan Wang, Zhi Wang. and Shu-Tao Xia. Gifd: generative gradient inver- sion method with feature domain optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4967-4976, 2023.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Advances in Neural Information Processing Systems, 33:2881- 2891, 2020.
Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients- how easy is it to break privacy in federated learning? Advances in Neural Information Processing Systems, 33:1699 1933-197-1 2020.
Niv Haim, Gal Vardi, Gilad Yehudai, Ohad Shamir, and Michal Irani, Reconstructing training data from trained neural networks. Advances in Neural Information Processing Systems, 35:22911- 1- 22924, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups IEEE Signal Processing Magazine, 29(6)82--7, 2012.
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and gen- eralization in neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Jinwoo Jeon, Kangwook Lee, Sewoong Oh, Jungseul Ok, et al. Gradient inversion with generative image prior. Advances in Neural Information Processing Systems, 14:29898-29900 2021.
Ziwei Ji and Matus Telgarsky. Directional convergence and alignment in deep learning, Advances in Neural Information Processing Systems, 33:17176-177 17186, 2020.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163 2020.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pp. 1885-1894. PMLR, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhuohang Li, Jiaxin Zhang, Luyang Liu, and Jian Liu. Auditing privacy defenses in federated learn- ing via generative gradient leakage. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10132-10142, 2022.
Noel Loo, Ramin Hasani, Mathias Lechner, Alexander Amini, and Daniela Rus. Understanding reconstruction attacks with the neural tangent kernel and dataset distillation. In The Twelfth In- fernational Conference on Learning Representations, 2024.

---

Published as a conference paper at ICLR 2025
Jiahao Lu, Xi Sheryl Zhang, Tianli Zhao, Xiangyu He, and Jian Cheng. April: Finding the achilles' heel on privacy for vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10051-10060, 2022.
Kaifeng Lyu and Jian Li. Gradient descent maximizes the margin of homogeneous neural networks. In The Eighth International Conference on Learning Representations, 2020.
Leonid I Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 60(1-4):259-268. 1992.
Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference at- tacks against machine learning models. In 2017 IEEE Symposium Security and Privacy (SP), pp. 3-18. IEEE, 2017.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4);600- 612, 2004.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2018.
Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. Graph neural networks in recommender systems: survey. ACM Computing Surveys, 55(5):1-37, 2022.
Hongxu Yin, Arun Mallya, Arash Vahdat, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. See through gradients: Image batch recovery via gradinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16337-16346 2021.
Chi Zhang, Zhang Xiaoman, Ekanut Sotthiwat, Yanyu Xu, Ping Liu, Liangli Zhen, and Yong Liu. Generative gradient inversion via perameteettee networks in federated learning. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 5126-5135, 2023.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001 022000060000. 2020.
Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. Advances in Neural Infor- mation Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2025
APPENDIX
A TRAINING DYNAMICS LINEARITY
Table I: Comparison training dynamics linearity metrics (Min) for 2-layer MLP with varying widths and ResNet-18, all trained on 100 CIFAR-10 images with fixed learning rate of 0.01 and batch size 100. A higher Min indicates more linear training dynamics.
Model
Width
Mun (1)
200
0.9134
500
0.9165
MLP
1000
0,9306
2000
0.9396
4000
0.9535
18
0,5988
To measure the linearity of training dynamics, we calculate the cosine similarity between gradient pairs for samples across different training epochs in dataset D:
(gi 1i.t2)
Min m. T(T 1) EISSAEET 19s, [1ggtt
(6)
where gi.t represents the gradient of the i-th sample at f-th epoch. And Min lies within [-1, 1]. A value of 1 indicates that the gradients across different epochs perfectly aligned, which is characteristic of completely linear network.
As illustrated in Tab. the linearity of MLP increases network width expands, which aligns with the theoretical predictions of neural tangent kernel (NTK) (Jacot et al.] (0118) that infinitely wide MLP tends to linear network. In contrast, ResNet-18 exhibits significantly lower linearity in training. due to its deeper and more complex structure.
B Loss CHOICE
We use Euclidean distance loss to replace cosine similarity loss for reconstructions of 20 images. Due the large variations in Euclidean distance loss during optimization, requires careful learning rate adjustments, and the final reconstruction quality is low compared to cosine similarity loss. The qualitative results of Fuclidean distance loss shown in Fig. with an average SSIM score of 0. 0,2437, while results of cosine similarity loss shown in Fig. get 0.3140 SSIM score.
Figure 11: Reconstructed training samples from ResNet (rows 1, 3) using Euclidean distance loss, and the corresponding bearest-neighbor from the training dataset (rows 2, 4).
13

---

Published as a conference paper at ICLR 2025
C ADDITIONAL EXPERIMENTS
C.I RECONSTRUCTIONS FOR LARGER IMAGES
While previous experiments are conducted on CIFAR-10 with resolution of 32 32, SimuDy can be scaled to higher resolutions. As shown in Fig. 12 the reconstruction results on the 64 x 64 Tiny ImageNet demonstrate the potential of our approach higher resolutions.
Figure 12: Reconstructions from ResNet-18 trained on Tiny ImageNet (rows 1, 3) using SimuDy, and the corresponding nearest-neighbors from the training dataset (rows 2, 4).
C.2 RECONSTRUCTIONS ON OTHER DATASET AND ARCHITECTURE
To further demonstrate that SimuDy maintains its effectiveness across various datasets and diverse network architectures, extend our experiments SVHN with ResNet-18 and CIFAR-IO with ResNet-50. The reconstruction results are presented in Fig. 13 and Fig. [4] respectively. Addition- ally, we conduct experiments on ImageNet with ViT and an NLP task, as shown in Appendix C.6) and Appendix C.7 In all these experiments, SimuDy consistently achieves strong performance, highlighting its robustness across different data distribution and network architectures.
19
3
2
5
3
19
3
19
3
28
5
16
63
2
Figure 13: Reconstructions from ResNet- 18 using SimuDy on SVHN, SSIM 0.2083.
Figure 14: Reconstructions from ResNet-50 using SimuDy on CIFAR-10, SSIM 0.2125.
14

---

Published as a conference paper at ICLR 2025
C.3 AFFECTION OF TV LOss
We test how the coefficient of TV loss affects the quality by discretely selecting the coefficient on reconstructing 20 images from trained ResNet-18. We set the coefficients O, Se-4, 2e-3, and 5e-2, respectively. When the coefficient 2e-3, the average SSIM score can reach 0.3140, with qualitative results shown in Fig. And other quantitative and qualitative reconstruction results are shown in Fig. 15 Fig. 16 and Fig. 17
Figure 15: Reconstructions from ResNet with TV loss coefficient of O, SSIM = 0.1855.
Figure 16: Reconstructions from ResNet with TV loss coefficient of 5e-4, SSIM 0.2555.
Figure 17: Reconstructions from ResNet with TV loss coefficient of 5e-2, SSIM 0.2255.
The coefficient TV loss has affects on the final reconstruction quality, Incorporating TV loss will help achieve better compared to not using it. Also, an appropriate coefficient important, both overly large and small values can degrade the reconstruction quality. To be noticed, an excessively large coefficient can cause reconstructed images become blurry, and may even result in color shifts compared to original training images, as shown in Fig.
C.4 COMPARISON WITH BUZAGLO ET AL. S METHOD
To achieve a comprehensive comparison against baseline methods, we test the performance of Buza- glo et al. method reconstructing images from parameters of models. We first present the quan- titative and qualitative reconstructions from MLPs trained on 100 images in Fig. 18 serving as a comparative counterpart to Fig.
15

---

Published as a conference paper at ICLR 2025
Figure 18: Top 40 images reconstructed from MLP trained 100 images using Buzaglo et al. SSIM 0.1426.
As illustrated in Fig. 18 while Buzaglo al.'s method demonstrates the capability to partially reconstruct training dataset from the parameters of MLPs, it falls short of our SimuDy in terms of both reconstructing quantity and quality. Subsequently, et al.Js method when applied to deep neural networks. The reconstruction results from ResNet- 18 trained 20 images are presented in Fig. 19, enabling direct comparison with Fig. while reconstructions from ResNet-18 trained on 50 images are shown in Fig. 20 corresponding to Fig.
demonstrate the performance of Buzaglo
Figure 19: Reconstructions from ResNet using Buzaglo et al. s method, SSIM = 0.0476.
Figure 20: Top 40 images reconstructed from ResNet trained 50 images using Buzaglo et al. SSIM = 0.0297
When applied ResNet-18 with non-linear training dynamics, Buzaglo et al. (0224) use only gra- dients of final models to characterize the whole dynamics, which is inadequate and degrades re- construction quality. Also, their method requires very small initial parameters or the use of weight decay in training, and the absence of these factors could further degrade the performance.
C.5 RECONSTRUCTIONS OF BINARY CLASSIFICATION SETTING
In this section, we manage to reconstruct training dataset from models trained for the binary classifi- cation task. Following the pioneering work (Haim et al. 2022), we use CIFAR-10 dataset and set the labels to vehicles vs. animals. Also, make sure that the class distribution in the training and test sets is balanced. For a comprehensive comparison with baselines, we use Haim et al. method, Loo et al. method and SimuDy reconstruct dataset from a MLP trained on 20 images, with quantita- tive and qualitative results shown in Fig. 21a Fig. 21b and Fig, 21c respectively. Both quantitative and qualitative results show that SimuDy well applied to binary classification setting.
16

---

Published as a conference paper at ICLR 2025
(a) Reconstructions in binary classification setting using Haim et al. SSIM = 0.2276
(b) Reconstructions in binary classification setting using Loo et al. s. SSIM = 0.3537
(c) Reconstructions in binary classification setting using SimuDy, SSIM = 0.5378
Figure 21: Reconstructions from trained binary MLP trained on images using different methods.
C.6 RECONSTRUCTIONS FOR VISION TRANSFORMERS
To evaluate the generalizability of SimuDy, we conduct experiments on Vision Transformers (ViTs) (Dosovitskiy 2021), which larger and have different architectures compared ResNet. For the original training dataset, we choose ImageNet with resolution of 224 224. Inspired by APRIL (Lu al. 2022) we noticed that the input of the self-ateention module is important for reconstruction but is invisible from attacker's side. However, for transformer-bast model with learnable position embedding Epos, the derivative of loss w.r.t. Epos can be given by:
al Øl
Oz
Based on this, update the reconstmuction loss for ViTs as following:
LVir(x, 0o. Crecon (x, 0y; 0,) - Fyn. On, Fo. OFF
Oo. Fpw.)
1 /.p. 60,
|||00 |10 0,
where fp,E,s, represents the initial parameters of the position embedding module, and f.E.p.. refers to the final parameters of the position embedding module after training. The second term the loss function effectively helps in ensuring the correct positioning of the patches.
Both training and reconstructing are conducted on one RTX 4090 GPU. We successfully reconstruct 10 ImageNet images from the parameters of trained VIT, with the qualitative reconstruction results
17

---

Published as a conference paper at ICLR 2025
shown in Fig. 22 The successful reconstructions validate the scalability of SimuDy to larger images and transformer architectures.
Figure 22: Reconstructions of 10 ImageNet images from a trained VIT.
C.7 RECONSTRUCTIONS IN NLP REGIME
In this section, we extend dataset reconstruction from image classification task the natural lan- guage processing (NLP) regime. We choose TinyBERT (Jiao et al.) 2020) model and CoLA (Warstadt et al. 2018 as dataset, with sentence classification as the training task. Both training and reconstructing run on one RTX 4090 GPU. Following the state-of-the -ftte-rr-te-rrt gradient inversion attack method in NLP (Balunovic et al. 2022), which recovers a size-of-4 batch of data from the gradient, we apply SimuDy to simulate the training dynamics and successfully reconstruct sentence data from parameters of trained TinyBERT parameters. The qualitative reconstructions are shown in Tab. 2
C.8 QUALITATIVE RESULTS FOR ORIGINAL DATASETS OF VARIOUS SIZES
In this section, vary the size of original training dataset and show the qualitative and quantitative results placed side by side for each baseline method and our proposed SimuDy, as illustrated from Fig. 23 Fig. 40 Also, we report GPU memory usage and reconstruction time of SimuDy on one RTX 4090 GPU for CIFAR-16 10 datasets with different sizes in Tab. 3
18

---

Published as a conference paper at ICLR 2025
Original
Initial
Reconstructed
[CLS] who do you think that
[CLSjquistanger fixingimeter
[CLS] who do you think will do first question sea-
will question seamus first?
epc forbidden nehru tread termi-
[SEP]
nology [SEP]
mus? [SEP]
[CLS] the boy ran. [SEP]
[CLS] [PAD] [PAD] [SEP]
[CLS] the boy ran. [SEP]
[PAD] foo nightmares [PAD]
[PA] 102 drawer [PAD]
[CLS] wonder who bill saw
[CLS] essays carltonomy crest-
[CLS] i saw mary won- der who liked bill anything.
and liked mary. [SEP]
edhiskhandnac [SEP] vita gail
[PAD]
[SEP]
[CLS] harriet alternated folk
[CLS] donetsk dominance gross-
[CLS] harriet alternated al-
songs and pop songs to-
edlok somerset registrar
ternate folk and pop songs together. [SEP]
gether. [SEP]
rochdale ins cher [SEP]
[CLS]] they have no old.
[CI.S] PAADJricc [PAD]] re-
[CL.S] they have no old.
[SEP]
vealing [PAD] destroyer [PAD]
[SEP]
louisiana [PAD] [SEP] [PAD]
[CLS] which goddess helped
[CLS] [PA]] [PAD] pathways
[CLS] goddess which helped
us? [SEP]
[PAD] [PAD] macon [PAD]emy
us? [SEP]
theresa [PAD] [PAD] [PAD]rod
[PAD] [SEP] [PAD] [PAD]
[CLS] who has seen my
[CLS] [PAD] [PAD] 2011 [PAD]
[CLS] who
seenkel
snorkel? [SEP]
[SEP] articulated caslink [PAD]
mynorkel? [SEP]
[PAD] implementskins [PAD]
via [PAD] [PAD]
[CLS] that the king or queen be present a requirement on all royal weddings. [SEP]
[CLS] sorbonne citationsimeter
[CLS] the queen or require- ment all that present is the royal weddings. [SEP]
tsar citizens state accumulate
jared sorbonne racecourse por- traying perkins differ deco [SEP] [CLS] feminism diagnosticnell
[CLS] saw these dancers and those musicians smok-
[CLS] i saw these dancers and those musicians smok- ing something. [SEP] [CLS] we promised that andy would go. [SEP]
andover delicately strikingcade
ing something. [SEP] [CLS] andy promised that
directorxing vfl [SEP] [CLS] [PAD] laurent unconstitu-
would go. [SEP]
tionalhony [PAD] demi duncan
kristin [SEP] [PAD]
[CLS] the cat tries to be out of the bag. [SEP]
[CLS] PAD] unless denoted lci-
[CLS] the be the cat tries out of the bag. [SEP]
den harm answering [SEP]ides
[PAD]ible hoffman [PAD]
[PAD] plata [PAD]
[CLS] we talked about that
[CLS] siena tomatoes shakeettes
[CLS] we talked about that he had worked at the white
he had worked at the white
sachsaseacy mosesasco flinders [SEP]
house. (SEP)
house. [SEP)
[CLS] jessica sprayed paint
[CLS] [PAD] reborn [PAD] zhao
[CLS] jessica spray under sprayed table paint. [SEP)
under the table. [SEP]
[SEP] leukemia braking vamp
[PAD] matchkled [PAD]olar
[PAD]
[CLS] mary noticed john's ss excessive appreciation of
[CLS] onward atoms macro
[CLS] and noticed mary. excessive appreciation of
undo sweets rounding sparse-
himself. [SEP]
lyde cornerbackomorphe [SEP]
john himself [SEP]
[CLS] kim alienates cats and beat his dog. [SEP)
[CLS] qui barred posed reverend
[CLS) kim alienates cats and beat his dog. [SEP)
wasn disputetilityphysies rabbis
[SEP] [PAD]
Table 2: Reconstructed samples NLP regime. The first column contains the original sentences of the training dataset, the second column contains the initializations of dummy sentences, and the third column presents the corresponding reconstructed sentences.
19

---

Published as a conference paper at ICLR 2025
Figure 23: Reconstructions from ResNet trained on 60 images using Buzaglo et al. method, SSIM 0.0285.
Figure 24: Reconstructions from ResNet trained on 60 images using Loo et al. method, SSIM 0.0423.
Figure 25: Reconstructions from ResNet trained on 60 images using SimuDy, SSIM = 0.99
20

---

Published as a conference paper at ICLR 2025
Figure 26: Reconstructions from ResNet trained on 70 images using Buzaglo al. ss method, SSIM 0.0286.
Figure 27: Reconstructions from ResNet trained on 70 images using Loo al, s method, SSIM 0.0424.
Figure 28: Reconstructions from ResNet trained 70 images using SimuDy, SSIM 0.1800.
21

---

Published as a conference paper at ICLR 2025
Figure 29: Reconstructions from ResNet trained on 80 images using Buzaglo al. ss method, SSIM 0.0253.
Figure 30: Reconstructions from ResNet trained on 80 images using Loo et al. s method, SSIM 0.0462.
Figure 31: Reconstructions from ResNet trained 80 images using SimuDy, SSIM 0.2165.
22

---

Published as a conference paper at ICLR 2025
Figure 32: Reconstructions from ResNer trained on 90 images using Buzaglo et al. method, SSIM 0.0249.
Figure 33: Reconstructions from ResNet trained on 90 images using Loo et al. method, SSIM 0.0442.
23

---

Published as a conference paper at ICLR 2025
Figure 34: Reconstructions from ResNet trained on 90 images using SimuDy, SSIM 0.1289.
Figure 35: Reconstructions from ResNet trained on 100 images using Buzaglo et al. method, SSIM 0.0267.
24

---

Published as a conference paper at ICLR 2025
1
Figure 36: Reconstructions from ResNet trained on 100 images using Loo et al. s method, SSIM 0.0476.
Figure 37: Reconstructions from ResNet trained on 100 images using SimuDy, SSIM 0.0948.
25

---

Published as a conference paper at ICLR 2025
Figure 38: Reconstructions from ResNet trained on 120 images using Buzaglo et al. method, SSIM 0.0239.
Figure 39: Reconstructions from ResNet trained 120 images using Loo al. is method, SSIM 0.0482.
26

---

Published as a conference paper at ICLR 2025
Figure 40: Reconstructions from ResNet trained on 120 images using SimuDy, SSIM 0.1197.
Dataset Size
GPU Memory
Reconstruction Time
20
4334 MB
1.06 h
30
6852 MB
3.32 h
40
8170 MB
4.67 h
50
10518 MB
6.07 h
60
12668 MB
8.46 h
70
14002 MB
11.37 h
80
15728 MB
12.77 h
90
16814 MB
12.92
100
19016 MB
13.70 h
120
22272 MB
15.39 h
Table 3: GPU memory usage and reconstruction time for training CIFAR-10 datasets with different sizes on one RTX 4090 GPU.
27


Reviews:
[{"review_id": "ns0xF9khH3", "reviewer": "Reviewer_KK4w", "summary": "SimuDy, a novel framework designed to reconstruct training data from the parameters of deep neural networks. SimuDy simulates the complex, non-linear training dynamics characteristic of deep architectures like ResNet by optimizing a dummy dataset to match the final parameters of a trained model. The authors demonstrate that SimuDy can accurately recover training samples under various conditions, including scenarios with unknown hyperparameters, outperforming previous linear-dynamics-based methods.", "questions": "See the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.  Novelty: By simulating non-linear dynamics, SimuDy extends dataset reconstruction capabilities to deeper models, such as ResNet, making it applicable to practical, real-world architectures rather than limiting it to simpler models.\n2. Robustness to Unknown Hyperparameters: SimuDy’s ability to reconstruct data even without exact training hyperparameters is a notable contribution, as real-world scenarios often lack complete information about training configurations.\n3. Quantitative and Qualitative Performance Improvements: SimuDy outperforms existing methods both in terms of structural similarity index and visual quality of reconstructed samples.\n4. The authors conduct extensive experiments, including robustness tests across varying dataset sizes, initializations, and dummy dataset configurations.", "weaknesses": "1. While SimuDy demonstrates promising results in reconstructing training samples from model parameters, there is limited discussion on the practical implications such as privacy or data security or in federated learning. \n2. SimuDy requires storing the entire training computation graph, which may limit its feasibility for very large models or datasets. The paper would benefit from a detailed breakdown of computational costs and memory requirements, especially for scaling to industry-sized datasets.\n3. The initialization of dummy datasets appears somewhat arbitrary (e.g., using noise or natural images). Conduct a systematic study on different initialization techniques for dummy datasets, such as starting with more structured priors, to assess their impact on reconstruction quality and convergence speed.\n4. Although SimuDy achieves high-quality reconstructions for small datasets, it’s unclear if these results generalize to larger, more diverse datasets. Extend experiments to larger datasets like ImageNet or tasks outside image classification to validate the scalability and generalization of SimuDy. Also it will be interesting to see the results of reconstruction when multiple objects are involved in an image.\n5. Lack of experiments in Vision Transformers. It will be interesting to extend some experiments on Vision Transformers, which differ from CNNs in their patch-based processing and self-attention mechanisms. Also, perform a comparative study to examine the differences in reconstruction quality between CNNs and transformers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOZb8hSHzC", "reviewer": "Reviewer_aded", "summary": "This paper addresses the data reconstruction problem for more realistic neural networks (particularly ResNet-18) than previous works. They propose to optimize the input data so that the training direction by the optimized data aligns with the direction trained by the target data. The experiments show that the proposed method successfully reconstruct CIFAR10 data from the trained parameter (and also the initial parameter) of ResNet-18. The method and results are really fascinating, but there are some concerns in their experiments as explained below.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and easy to follow. The method is also clearly described.\n2. The proposed method is derived in a straightforward way, and the implementation is also very simple. In spite of these simplicity, the results seem really impressive compared to previous works. I like both the idea and the simplicity.", "weaknesses": "1. The method assumes the initialization parameter to be given a priori, which is an unrealistic assumption for data reconstruction attack. However, I think this is not a critical issue because the contributions of this paper may not be limited to such attacks.\n2. As authors also mentioned, the computational and memory budget may be terribly large due to higher order differentiation. Although this does not hurt the contribution of this work, it should be worth reporting the actual reconstruction time.\n3. In experiments, the proposed method is only compared with Loo et al.'s method [1]. However, it should be also (rather) compared with Buzaglo et al.'s [2] (or Haim et al. [3]) method since this is the first successful method for data reconstruction from parameters. Moreover, to verify the generality of the proposed method, i.e., to check the results being not overfitting to CIFAR10 with ResNet18, the experiments should include several datasets (e.g. SVHN or MNIST) and model architectures like other ResNet architectures etc, with quantitative comparison to the baselines.\n4. It is unclear how the coefficient of the TV regularization affects the quality of reconstructed data. It should be worth reporting the quantitative analysis from this perspective.\n5. There are some missing related works [4,5] on gradient/trajectory matching for non-linear training dynamics. Zhao et al. [4] optimizes the input space by the cosine symmetry of two training dynamics, and hence is strongly related to this paper. Chijiwa [5] also optimizes the cosine similarity of two training dynamics, but focusing on the symmetry in a neural network rather than input data.\n\n[1] Loo et al. \"Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation\" (ICLR'24)\n\n[2] Buzaglo et al. \"Deconstructing Data Reconstruction: Multiclass, Weight Decay and General Losses\" (NeurIPS'23)\n\n[3] Haim et al. \"Reconstructing Training Data from Trained Neural Networks\" (NeurIPS'22)\n\n[4] Zhao et al., \"Dataset Condensation with Gradient Matching\" (ICLR'21)\n\n[5] Chijiwa, \"Transferring Learning Trajectories of Neural Networks\" (ICLR'24)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C9nLPG6NM3", "reviewer": "Reviewer_v7cX", "summary": "The paper proposes Simulation of training Dynamics (SimuDy), an easy and effective approach that reconstructs datasets from neural networks by optimizing dummy datasets through the comparison of resulting network parameters after training. The proposed method demonstrates better performance compared to existing baselines and shows promising results on relatively more advanced architectures like ResNets.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method has much better performance compared to existing baseline. The method is also easy to implement.\n2. This work is both novel and interesting to me.", "weaknesses": "I am mainly concerned about the implication and the usefulness of the proposed method.\n1. **Practicality and Scalability.** Back propagating with multiple training steps can be computationally intensive and challenging to scale for larger models and with larger datasets with high resolutions. Additionally, the method requires prior knowledge of hyper-parameters such as the number of training steps, batch size, and weights at initialization/end which limits its applicability.\n2. **Implication and Usefulness.** I wonder how SimuDy can contribute to a deeper understanding of neural network training dynamics. It difficult to assess its broader usefulness in the field of machine learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BSnVbxTQO4", "reviewer": "Reviewer_NSaC", "summary": "This submission proposes a method to reconstruct training data from neural network models by simulating their non-linear training dynamics. It adopts a nested optimization approach, where the outer optimization tunes the data being reconstructed, and the inner optimization performs regular training to match the original model’s parameters. Experimental results demonstrate successful dataset reconstruction for larger, more complex models like ResNet, where previous methods in linear regimes struggle, albeit assuming full access to weights, architecture, and hyperparameters.", "questions": "See weaknesses", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is generally well-written and clear, making the core ideas easily accessible. \n- The proposed nested optimization approach is a meaningful extension over prior work for the task, effectively addressing non-linear training dynamics in a way that previous methods in linear regimes could not. This method is well-motivated and aligns with the challenges of reconstructing data from more complex models like ResNet.  \n- Experimental evaluation is thorough and supports the claims within the chosen setting. The authors evaluate across different dataset and model sizes, confirming the performance benefits of SimuDy over baseline methods in similar constrained scenarios.", "weaknesses": "- The submission lacks a defined threat model for data reconstruction. That makes it challenging to evaluate the setting, how broad or narrow it is, and what comparable methods might be. It seems to require knowledge about model architecture, weight (history) and hyperparameters - just not the gradients and training data. To me, that seems to be rather specific and I would ask the authors to elaborate on why they chose that particular setting. \n- Relatedly, the model-to-data ratio used here raises questions about both the threat model and the generalizability of the findings. A ResNet-18 trained on 120 images represents a highly constrained regime compared to standard machine learning applications. Is model depth less of a challenge here than dataset size? This also leads me to question whether memorization and an overfitted model-to-data ratio are necessary conditions for SimuDy’s success. If not, I would be interested in an experiment using a similarly complex model trained on a larger dataset, such as the full CIFAR-10 or CIFAR-100, with an attempt to reconstruct samples. Even if complete recovery is not feasible, partial reconstruction of a meaningful subset could provide insights into the general applicability of this method.\n- Finally, I would like to see a more comprehensive evaluation against baseline methods. Model inversion attacks seem relevant here; if not, I would suggest the authors clarify why these are not comparable and discuss performance and computational cost of either approahces. Such a comparison would contextualize the efficiency and effectiveness of SimuDy against established approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)