Reconstructing Training Data From Real-World Models Trained with Transfer Learning

Authors: Yakir Oz, Gilad Yehudai, Gal Vardi, Itai Antebi, michal Irani, Niv Haim
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: data reconstruction, memorization, privacy

Abstract:
Current methods for reconstructing the training data from trained classifiers are restricted to very small models, limited training set sizes, and low-resolution images. Such restrictions hinder their applicability to real-world scenarios. In this paper, we present a novel approach enabling data reconstruction in realistic settings for models trained on high-resolution images. Our method adapts the reconstruction scheme of Haim et al. [2022] to real-world scenarios -- specifically, targeting models trained via transfer learning over image embeddings of large pre-trained models like DINO-ViT and CLIP. Our work employs data reconstruction in the embedding space rather than in the image space, showcasing its applicability beyond visual data. Moreover, we introduce a novel clustering-based method to identify good reconstructions from thousands of candidates. This significantly improves on previous works that relied on knowledge of the training set to identify good reconstructed images. Our findings shed light on a potential privacy risk for data leakage from models trained using transfer learning methods.

Decision: reject
Reviewer scores: 8, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730103579557

thread_type:
official_review

<thread_object>
creator: reviewer_vvgm

<content>
summary:
This paper proposes a new method for training data reconstruction. Different from other works, this paper proposes to recover the image embedding first, and then employs a inversion network to reconstruct the images. Especially, this paper proposes to use the clustering-based approach for effectively identifying training samples. Experiments show the performance of the proposed method.

strengths:
The proposed method has show better performance compared with baselines, on the datasets of Food-101 and iNaturalist.

weaknesses:
1. The experiments are only conducted on the CLIP and DINO which has served as the common components for the image diffusion models. However, I think the proposed network should be suitable for various networks. I wonder the training data reconstruction with other basic networks, like ResNet. And the authors should explain why they focused on transformer-based models in the experimental section.

2. This paper claims to be suitable for the reconstruction of high-resolution images, while the experiments are only conducted on the datasets with low resolution like 224 (why 224x224 fits that definition of "high resolution" in this paper?). I wonder the performance with higher resolution like 512.

3. There is no quantitative metric to compare the reconstruction effects between the proposed method and baselines. For example, using the reconstructed images can lead to a model with the similar accuracy of the original model?

4. The presentation of this paper is not good. For example, the texts in Fig.9 has be out of the width constraint, which could be resized.

questions:
1. What is the performance of the proposed method with more types of networks besides the transformer?

2. What is the reconstruction performance on images with higher resolution?

3. Can the authors provide more reliable quantitative metrics?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730211307040

thread_type:
official_review

<thread_object>
creator: reviewer_c8yx

<content>
summary:
This paper demonstrates the reconstruction of high-resolution training images from models trained using a transfer learning approach, as well as the reconstruction of non-visual data. Moreover, it introduces a novel clustering-based approach for effectively identifying training samples without prior knowledge of the training images.

strengths:
1. The writing is well done and clear.

2. They present the weaknesses and limitations of their method in detail.

3. The experiments consider various commonly used pre-trained feature extractors such as CLIP, demonstrating the effectiveness of their method.

weaknesses:
1. The method is limited to specific cases where a fixed feature extractor and some MLP layers serve as the classifier. It cannot generalize to other more common transfer learning scenarios, such as fine-tuning an entire classifier or certain layers of a classifier.

2. The introduced method lacks innovation. Specifically, the core contributions of reconstructing embedding vectors in Section 3.1 and mapping embedding vectors in Section 3.2 to the image domain either originate from other works or involve only simple modifications, such as changing the MSE loss to the cosine similarity loss. Please clearly explain the novel aspects of the method introduced.

3. The format of Figure 9 is incorrect as it exceeds the page boundary.

questions:
My confusion is the aforementioned weaknesses. If there are any misunderstandings, please point them out.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730684085603

thread_type:
official_review

<thread_object>
creator: reviewer_hzzb

<content>
summary:
This paper studies the training data reconstruction problem. Different from previous methods, this paper studies the data reconstruction from models trained in a transfer learning approach, and claims the first approach to reconstruct images from the latent features. The experimental section showcases comprehensive results to verify the efficacy of proposed method in different datasets and backbone networks.

strengths:
1. This paper is easy to follow.
2. I appreciate the comprehensive experiments performed to analyze the effectiveness of the proposed approach.

weaknesses:
1. The technique contribution is limited. This paper aims to reconstruct images from latent features (embeddings), however, the key components used for this purpose are borrowed from previous works. Specifically, it use ‘’Reconstructing training data from trained neural networks’’ to reconstruct embeddings, followed by ‘’Splicing vit features for semantic appearance transfer’’ to convert embeddings into RGB images.
2. The resolution of reconstructed images are still low (224x224).
3. The experiment part contains results of this method, but without any comparison with other approaches.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1731165454902

thread_type:
official_review

<thread_object>
creator: reviewer_qriu

<content>
summary:
This paper explores optimization-based **data inversion** techniques from pre-trained models with transfer learning, in which we could reconstruct the training data simply given the pre-trained encoder and classifier itself. It extends the model-inversion techniques with improved designs like loss, generative decoder. The clustering-based approach further demonstrates the possibility of high-quality reconstructions to the original data. Experiments on two common datasets reveals the potential privacy risks associated with models trained on sensitive data.

strengths:
[**Novelty**] 
- Working in embedding space instead of directly reconstructing images is an innovative approach, and makes it more scalable to different models on both visual and non-visual data
- the adoption of clustering and averaging technique is novel, which identifies high-quality reconstructions when training data is not available

[**Significance**] 
- it highlights the significant privacy risks associated with models trained with sensitive data in a transfer learning setup, in which high-resolution data reconstruction in real-world conditions emphasizes the critical need for privacy-preserving mechanisms with today's pre-trained models. 

[**Completeness & Clarity**]
- High-quality visualizations, including comparisons of reconstructed images to original data and plots that illustrate reconstruction metrics, add depth to the evaluation, making it easier to interpret the results.
- The writing is clear, which effectively lays out the motivation and approach, and well explains the limitations

weaknesses:
- On significance, while I like the simple paradigm of solving x when f is known within f(x)=y, I kind feel that reconstructed data samples are not exactly matching with the actual training data, especially when the training data is not available. It is more close to averaged per-category data when training data is not available, the author might want to turn down their scope.
- Another thinking is that the current method may only work with classifiers-based model, for more more fine-grained training data reconstruction from segmentation model like SAM might be more desired. Also the reconstruction quality varies significantly with the choice of backbone model (DINO, CLIP, etc.), which affects the novelty of the clustering approach by making it model-dependent.
- As the author also mentioned, the inversion process used to map embeddings back to images is computationally expensive, which could hinder scalability

questions:
- I am also wondering whether we have any baselines in this line of data inversion techniques
- a quick question, for diffusion-based generative models, wondering whether it is also a more realistic concern to reveal training data directly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732435662022

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Response to all the reviewers

comment:
Again, we thank the reviewers for their comments. We hope that our response has addressed all the issues raised by the reviewers and that they will consider updating their scores accordingly. If there are remaining questions, we would of course be happy to address those before the public discussion phase ends. 

Many thanks, The authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper conducts optimization-based data inversion from pre-trained models with transfer learning. The objective is to reconstruct the training data given the pre-trained encoder and classifiers using clustering techniques. The main contribution is in demonstrating the vulnerability to training-data reconstruction of models trained in a transfer learning manner using embeddings of widely-used backbone models.

Strengths:
- Operates in the embedding space, making it more scalable.
- The use of clustering is novel.
- Highlights privacy risks in models trained with sensitive data.
- Contains comprehensive experiments.

Weaknesses:
- The data reconstructed is the expected data rather than the original data, thus the scope should be adjusted.
- Limited usability due to the required encoder/classifier setup and limited verification on CLIP and DINO.
- Limited technical contribution as it uses existing techniques.

Given the limited technical contribution, the empirical contribution must be substantial and the discussions insightful to make them the main empirical contribution of the paper.  However, the paper has a limited empirical contribution and requires major experiments to fully addressed the concerns raised by the reviewers.  Thus, I recommend the rejection of the paper.