Training-Like Data Reconstruction

## Abstract

Abstract Machine Learning models are often trained on proprietary and private data that cannot be shared, though the trained models themselves are distributed openly assuming that sharing model weights is privacy preserving, as training data is not expected to be inferred from the model weights. In this paper, we present Training-Like Data Reconstruction (TLDR), a network inversion-based approach to reconstruct training-like data from trained models. To begin with, we introduce a comprehensive network inversion technique that learns the input space corresponding to different classes in the classifier using a single conditioned generator. While inversion may typically return random and arbitrary input images for a given output label, we modify the inversion process to incentivize the generator to reconstruct training-like data by exploiting key properties of the classifier with respect to the training data along with some prior knowledge about the images. To validate our approach, we conduct empirical evaluations on multiple standard vision classification datasets, thereby highlighting the potential privacy risks involved in sharing machine learning models.

## 1 Introduction

Machine learning models are often trained on proprietary or sensitive data, which cannot be shared openly, yet the trained models themselves are commonly distributed to facilitate various applications assuming that they do not expose the underlying training data. However, recent research suggests that this assumption may not be valid, as it may be possible to infer and reconstruct training or similar data by analyzing the model weights. This potential privacy risk arises from the fact that trained ML models implicitly encode information about the data they were trained on.

Prior research to reconstruct training data has primarily focused on restricted scenarios, such as binary classifiers with fully connected layers trained on a small dataset. In restricted settings, over-parameterized models can easily memorize portions of the training data, leading to successful reconstructions. For under-parameterized models, where there is no possibility of memorization and the models generalize well, reconstructions are typically more difficult. Also in fully connected layers, each input feature is assigned dedicated weights, which may make reconstruction easier as the model captures more direct associations between inputs and outputs. While as in convolutional layers, due to the weight-sharing mechanism, where the same set of weights is applied across different parts of the input, the reconstruction becomes more challenging.

In this paper, we introduce Training-Like Data Reconstruction (TLDR), a novel approach to reconstruct training-like data from vision classifiers with convolutional layers trained on large, complex, and multi-class datasets. At the core of our approach is a network inversion technique that learns the input space corresponding to different classes within a classifier using a single conditioned generator trained to generate a diverse set of samples from the input space with desired labels guided by a combination of losses including cross-entropy, KL Divergence, cosine similarity and feature orthogonality. Inverted samples generated through network inversion are often random, and while inversion may occasionally produce training-like data, our goal is to specifically encourage the generator to reconstruct training-like data. To achieve this, we exploit some key properties of the classifier in relation to its training data.

The classifier is expected to be more confident in its predictions on training samples compared to randomly generated, inverted samples. Mathematically, this can be expressed as:

$$ $P(y_{\text{in}}|x_{\text{in}};\theta)\gg P(y_{\text{ood}}|x_{\text{ood}};\theta)$ $$

where $P(y|x;\theta)$ represents the softmax output of the classifier for a given input $x$, $\theta$ are the model’s parameters, $x_{\text{in}}$ refers to in-distribution data, and $x_{\text{ood}}$ refers to out-of-distribution data.

During training, the model learns to generalize across variations in the training data, making it relatively more robust to perturbations around these samples and the same can be represented by:

$$ $\frac{\partial f_{\theta}(x_{\text{in}})}{\partial x_{\text{in}}}\ll\frac{ \partial f_{\theta}(x_{\text{ood}})}{\partial x_{\text{ood}}}$ $$

Since the classifier has already been optimized on the train set, the gradient of the loss with respect to the weights is expected to be lower for training data compared to random inverted samples, hence:

$$ $\|\nabla_{\theta}L(f_{\theta}(x_{\text{in}}),y_{\text{in}})\|\ll\|\nabla_{ \theta}L(f_{\theta}(x_{\text{ood}}),y_{\text{ood}})\|$ $$

where $L$ represents the loss function, $f_{\theta}(x)$ is the model output for input $x$, and $\nabla_{\theta}$ is the gradient with respect to the model weights $\theta$. Our main contributions in this paper include:

- 1.
A comprehensive approach to inversion of convolutional vision classifiers using a single conditioned generator.
- 2.
The introduction of soft vector conditioning and intermediate matrix conditioning to encourage diversity in the inversion process.
- 3.
The use of network inversion to reconstruct training-like data by exploiting key properties of the classifier in relation to its training data.

To validate our approach, we conduct extensive inversion and reconstruction experiments on MNIST, FashionMNIST, SVHN, and CIFAR-10, demonstrating that the proposed method is capable of reconstructing training-like data across different domains highlighting the privacy risks.

## 2 Related Works

Network inversion has emerged as a powerful method for exploring and understanding the internal mechanisms of neural networks. By identifying input patterns that closely approximate a given output target, inversion techniques provide a way to visualize the information processing capabilities embedded within the network’s learned parameters. These methods reveal important insights into how models represent and manipulate data, offering a pathway to expose the latent structure of neural networks. While inversion techniques primarily began as tools for understanding models, their application to extracting sensitive data has sparked significant concerns. Neural networks inherently store information about the data they are trained on, and this has led to the potential for training data to be reconstructed through inversion attacks. Early works in this space, particularly on over-parameterized models with fully connected networks, demonstrated that it was possible to extract portions of the training data due to the model’s tendency to memorize data. This raises significant privacy concerns, especially in cases where models are trained on proprietary or sensitive datasets, such as in healthcare or finance.

Early research on inversion for multi-layer perceptrons in (Kindermann and Linden, ), derived from the back-propagation algorithm, demonstrates the utility of this method in applications like digit recognition highlighting that while multi-layer perceptrons exhibit strong generalization capabilities—successfully classifying untrained digits—they often falter in rejecting counterexamples, such as random patterns. Subsequently (Jensen et al., ) expanded on this idea by proposing evolutionary inversion procedures for feed-forward networks that stands out for its ability to identify multiple inversion points simultaneously, providing a more comprehensive view of the network’s input-output relationships. The paper (Saad and Wunsch, ) explores the lack of explanation capability in artificial neural networks (ANNs) and introduces an inversion-based method for rule extraction to calculate the input patterns that correspond to specific output targets, allowing for the generation of hyperplane-based rules that explain the neural network’s decision-making process. (Wong, ) addresses the problem of inverting deep networks to find inputs that minimize certain output criteria by reformulating network propagation as a constrained optimization problem and solving it using the alternating direction method of multipliers.

Model Inversion attacks in adversarial settings are studied in (Yang et al., ), where an attacker aims to infer training data from a model’s predictions by training a secondary neural network to perform the inversion, using the adversary’s background knowledge to construct an auxiliary dataset, without access to the original training data. The paper (Kumar and Levine, ) presents a method for tackling data-driven optimization problems, where the goal is to find inputs that maximize an unknown score function by proposing Model Inversion Networks (MINs), which learn an inverse mapping from scores to inputs, allowing them to scale to high-dimensional input spaces. While (Ansari et al., ) introduces an automated method for inversion by focusing on the reliability of inverse solutions by seeking inverse solutions near reliable data points that are sampled from the forward process and used for training the surrogate model. By incorporating predictive uncertainty into the inversion process and minimizing it, this approach achieves higher accuracy and robustness.

The traditional methods for network inversion often rely on gradient descent through a highly non-convex loss landscape, leading to slow and unstable optimization processes. To address these challenges, recent work by (Liu et al., ) proposes learning a loss landscape where gradient descent becomes efficient, thus significantly improving the speed and stability of the inversion process. Similarly Suhail () proposes an alternate approach to inversion by encoding the network into a Conjunctive Normal Form (CNF) propositional formula and using SAT solvers and samplers to find satisfying assignments for the constrained CNF formula. While this method, unlike optimization-based approaches, is deterministic and ensures the generation of diverse input samples with desired labels. However, the downside of this approach lies in its computational complexity, which makes it less feasible for large-scale practical applications.

In reconstruction (Haim et al., ) studies the extent to which neural networks memorize training data, revealing that in some cases, a significant portion of the training data can be reconstructed from the parameters of a trained neural network classifier. The paper introduces a novel reconstruction method based on the implicit bias of gradient-based training methods and demonstrate that it is generally possible to reconstruct a substantial fraction of the actual training samples from a trained neural network, specifically focusing on binary MLP classifiers. Later (Buzaglo et al., ) improve upon these results by showing that training data reconstruction is not only possible in the multi-class setting but that the quality of the reconstructed samples is even higher than in the binary case. Also revealing that using weight decay during training can increase the susceptibility to reconstruction attacks.

The paper (Balle et al., ) addresses the issue of whether an informed adversary, who has knowledge of all training data points except one, can successfully reconstruct the missing data point given access to the trained machine learning model. The authors explore this question by introducing concrete reconstruction attacks on convex models like logistic regression with closed-form solutions. For more complex models, such as neural networks, they develop a reconstructor network, which, given the model weights, can recover the target data point. Subsequenlty (Wang et al., ) investigates how model gradients can leak sensitive information about training data, posing serious privacy concerns. The authors claim that even without explicitly training the model or memorizing the data, it is possible to fully reconstruct training samples by gradient query at a randomly chosen parameter value. Under mild assumptions, they demonstrate the reconstruction of training data for both shallow and deep neural networks across a variety of activation functions.

In this paper, we explore the intersection of network inversion and training data reconstruction. Our approach to network inversion aims to strike a balance between computational efficiency and the diversity of generated inputs by using a carefully conditioned generator trained to learn the data distribution in the input space of a trained neural network. The conditioning information is encoded into vectors in a concealed manner to enhance the diversity of the generated inputs by avoiding easy shortcut solutions. This diversity is further enhanced through the application of heavy dropout during the generation process, the minimization of cosine similarity and encouragement of orthogonality between a batch of the features of the generated images.

While network inversion may occasionally produce training-like samples, we encourage this process by exploiting key properties of the classifier with respect to its training data. The classifier tends to be more confident in predicting in-distribution training samples than random, out-of-distribution samples, and it exhibits greater robustness to perturbations around the training data. Furthermore, the gradient of the loss with respect to the model’s weights is typically lower for training data, which helps guide the generator toward reproducing these samples. Additionally, we incorporate prior knowledge in the form of variational loss to create noise-free images and pixel constraint loss to keep pixel values within the valid range, ensuring the generated images are both semantically and visually aligned with the original training data. By leveraging these insights, we steer the inversion process to reconstruct training-like data and extend prior work on training data reconstruction, which primarily focused on models with fully connected layers, to under-parametrized models with convolutional layers and standard activation functions, trained on larger datasets with regularisation techniques to prevent memorisation.

## 3 Methodology & Implementation

Our approach to Network Inversion and subsequent training data reconstruction uses a carefully conditioned generator that learns diverse data distributions in the input space of the trained classifier.

### 3.1 Classifier

In this paper inversion and reconstruction is performed on a classifier which includes convolution and fully connected layers as appropriate to the classification task. We use standard non-linearity layers like Leaky-ReLU (Xu et al., ) and Dropout layers (Srivastava et al., ) in the classifier for regularisation purposes to discourage memorisation. The classification network is trained on a particular dataset and then held in evaluation mode for the purpose of inversion and reconstruction.

### 3.2 Generator

The images in the input space of the classifier will be generated by an appropriately conditioned generator. The generator builds up from a latent vector by up-convolution operations to generate the image of the given size. While generators are conventionally conditioned on an embedding learnt of a label for generative modelling tasks, we given its simplicity, observe its ineffectiveness in network inversion and instead propose more intense conditioning mechanism using vectors and matrices.

#### 3.2.1 Label Conditioning

Label Conditioning of a generator is a simple approach to condition the generator on an embedding learnt off of the labels each representative of the separate classes. The conditioning labels are then used in the cross entropy loss function with the outputs of the classifier. While Label Conditioning can be used for inversion, the inverted samples do not seem to have the diversity that is expected of the inversion process due to the simplicity and varying confidence behind the same label.

#### 3.2.2 Vector Conditioning

In order to achieve more diversity in the generated images, the conditioning mechanism of the generator is altered by encoding the label information into an $N$-dimensional vector for an $N$-class classification task. The vectors for this purpose are randomly generated from a normal distribution and then soft-maxed to represent an input conditioning distribution for the generated images. The argmax index of the soft-maxed vectors now serves as the de facto conditioning label, which can be used in the cross-entropy loss function without being explicitly revealed to the generator.

#### 3.2.3 Intermediate Matrix Conditioning

Vector Conditioning allows for a encoding the label information into the vectors using the argmax criteria. This can be further extended into Matrix Conditioning which apparently serves as a better prior in case of generating images and allows for more ways to encode the label information for a better capture of the diversity in the inversion process. In its simplest form we use a Hot Conditioning Matrix in which an $NXN$ dimensional matrix is defined such that all the elements in a given row and column (same index) across the matrix are set to one while the rest all entries are zeroes. The index of the row or column set to 1 now serves as the label for the conditioning purposes. The conditioning matrix is concatenated with the latent vector intermediately after up-sampling it to $NXN$ spatial dimensions, while the generation upto this point remains unconditioned.

#### 3.2.4 Vector-Matrix Conditioning

Since the generation is initially unconditioned in Intermediate Matrix Conditioning, we combine both vector and matrix conditioning, in which vectors are used for early conditioning of the generator upto $NXN$ spatial dimensions followed by concatenation of the conditioning matrix for subsequent generation. The argmax index of the vector, which is the same as the row or column index set to high in the matrix, now serves as the conditioning label.

### 3.3 Network Inversion

The main objective of Network Inversion is to generate images that when passed through the classifier will elicit the same label as the generator was conditioned to. Achieving this objective through a straightforward cross-entropy loss between the conditioning label and the classifier’s output can lead to mode collapse, where the generator finds shortcuts that undermine diversity. With the classifier trained, the inversion is performed by training the generator to learn the data distribution for different classes in the input space of the classifier as shown schematically in Figure 1 using a combined loss function $\mathcal{L}_{\text{Inv}}$ defined as:

$$ $\mathcal{L}_{\text{Inv}}=\alpha\cdot\mathcal{L}_{\text{KL}}+\beta\cdot\mathcal {L}_{\text{CE}}+\gamma\cdot\mathcal{L}_{\text{Cosine}}+\delta\cdot\mathcal{L}_ {\text{Ortho}}$ $$

where $\mathcal{L}_{\text{KL}}$ is the KL Divergence loss, $\mathcal{L}_{\text{CE}}$ is the Cross Entropy loss, $\mathcal{L}_{\text{Cosine}}$ is the Cosine Similarity loss, and $\mathcal{L}_{\text{Ortho}}$ is the Feature Orthogonality loss. The hyperparameters $\alpha,\beta,\gamma,\delta$ control the contribution of each individual loss term defined as:

$$ $\mathcal{L}_{\text{KL}}=D_{\text{KL}}(P\|Q)=\sum_{i}P(i)\log\frac{P(i)}{Q(i)}$ $$

$$ $\mathcal{L}_{\text{CE}}=-\sum_{i}y_{i}\log(\hat{y}_{i})$ $$

$$ $\mathcal{L}_{\text{Cosine}}=\frac{1}{N(N-1)}\sum_{i\neq j}\cos(\theta_{ij})$ $$

$$ $\mathcal{L}_{\text{Ortho}}=\frac{1}{N^{2}}\sum_{i,j}(G_{ij}-\delta_{ij})^{2}$ $$

where $D_{\text{KL}}$ represents the KL Divergence between the input distribution $P$ and the output distribution $Q$, $y_{i}$ is the set encoded label, $\hat{y}_{i}$ is the predicted label from the classifier, $\cos(\theta_{ij})$ represents the cosine similarity between features of generated images $i$ and $j$, $G_{ij}$ is the element of the Gram matrix, and $\delta_{ij}$ is the Kronecker delta function. $N$ is the number of feature vectors in the batch.

Thus, the combined loss function ensures that the generator matches the input and output distributions using KL Divergence and also generates images with desired labels using Cross Entropy, while maintaining diversity in the generated images through Feature Orthogonality and Cosine Similarity.

Figure: Figure 1: Proposed Approach to Network Inversion
Refer to caption: extracted/5945715/ni.png

#### 3.3.1 Cross Entropy

The key goal of the inversion process is to generate images with the desired labels and the same can be easily achieved using cross entropy loss. In cases where the label information is encoded into the vectors without being explicitly revealed to the generator, the encoded labels can be used in the cross entropy loss with the classifier outputs for the generated images. In contrast to the label conditioning, vector conditioning complicate the training objectives to the extent that the generator does not immediately converge, instead the convergence occurs only when the generator figures out the encoded conditioning mechanism allowing for a better exploration of the input space.

#### 3.3.2 KL Divergence

KL Divergence is used to train the generator to learn the data distribution in the input space of the classifier for different conditioning vectors. During training, the KL Divergence loss function measures and minimise the difference between the output distribution of the generated images, as predicted by the classifier, and the conditioning distribution used to generate these images.

#### 3.3.3 Cosine Similarity

To enhance the diversity of the generated images, we use cosine similarity to assesses and minimises the angular distance between the features of a batch of generated images across the last fully connected layers, promoting variability in the generated images. The combination of cosine similarity with cross-entropy loss not only ensures that the generated images are classified correctly but also enforces diversity among the images produced for each label.

#### 3.3.4 Feature Orthogonality

In addition to the cosine similarity loss, we incorporate feature orthogonality as a regularization term to further enhance the diversity of generated images by minimizing the deviation of the Gram matrix of the features from the identity matrix. By ensuring that the features of generated images are orthogonal, we promote the generation of distinct and non-redundant representations for each conditioning label.

### 3.4 Training-Like Data Reconstruction

While Network Inversion enables access to a diverse set of images in the input space of the model for different classes, the inverted samples are completely random. However, Network Inversion can be used for training data reconstruction as shown schematically in Figure 2 by exploiting key properties of the training data in relation to the classifier including model confidence, robustness to perturbations, and gradient behavior along with some prior knowledge about the training data.

In order to take model confidence into account, we use hot conditioning vectors in reconstruction instead of soft conditioning vectors used in inversion, to generate samples that are confidently labeled by the classifier. Since the classifier is expected to handle perturbations around the training data effectively, the perturbed images should retain the same labels and also be confidently classified. Hence, we introduce an $L_{\infty}$ perturbation to the generated images and pass both the original and perturbed images represented by dashed lines, through the classifier and use them in the loss evaluation. We also introduce a gradient minimization loss to penalise the large gradients of the classifier’s output with respect to its weights when processing the generated images ensuring that the generator produces samples that have small gradient norm, a property expected of the training samples. Furthermore, we incorporate prior knowledge through pixel constraint and variational losses to ensure that the generated images have valid pixel values and are noise-free.

Figure: Figure 2: Schematic Approach to Training-Like Data Reconstruction using Network Inversion
Refer to caption: extracted/5945715/tldr.png

Hence the previously defined inversion loss $\mathcal{L}_{\text{Inv}}$ is augmented to include the above aspects into a combined reconstruction loss $\mathcal{L}_{\text{Recon}}$ defined as:

$$ $\mathcal{L}_{\text{Recon}}=\alpha\cdot\mathcal{L}_{\text{KL}}+\alpha^{\prime} \cdot\mathcal{L}_{\text{KL}}^{\text{pert}}+\beta\cdot\mathcal{L}_{\text{CE}}+ \beta^{\prime}\cdot\mathcal{L}_{\text{CE}}^{\text{pert}}+\gamma\cdot\mathcal{L }_{\text{Cosine}}+\delta\cdot\mathcal{L}_{\text{Ortho}}+\eta_{1}\cdot\mathcal{ L}_{\text{Var}}+\eta_{2}\cdot\mathcal{L}_{\text{Pix}}+\eta_{3}\cdot\mathcal{L} _{\text{Grad}}$ $$

where $\mathcal{L}_{\text{KL}}^{\text{pert}}$ and $\mathcal{L}_{\text{CE}}^{\text{pert}}$ represent the KL divergence and cross-entropy losses applied on perturbed images, weighted by $\alpha^{\prime}$ and $\beta^{\prime}$respectively while $\mathcal{L}_{\text{Var}}$, $\mathcal{L}_{\text{Pix}}$ and $\mathcal{L}_{\text{Grad}}$ represent the variational loss, Pixel Loss and penalty on gradient norm each weighted by $\eta_{1}$, $\eta_{2}$, and $\eta_{3}$ respectively and defined for an Image $I$ as:

$$ $\mathcal{L}_{\text{Var}}=\frac{1}{N}\sum_{i=1}^{N}\left(\sum_{h,w}\left(\left( I_{i,h+1,w}-I_{i,h,w}\right)^{2}+\left(I_{i,h,w+1}-I_{i,h,w}\right)^{2}\right)\right)$ $$

$$ $\mathcal{L}_{\text{Pix}}=\sum\max(0,-I)+\sum\max(0,I-1)\hskip 25.0pt\mathcal{L }_{\text{Grad}}=\left\|\nabla_{\theta}L(f_{\theta}(I),y)\right\|$ $$

#### 3.4.1 Pixel Loss

The Pixel Loss is used to ensure that the generated images have valid pixel values between 0 and 1. Any pixel value that falls outside this range is penalized hence encouraging the generator to produce valid and realistic images.

#### 3.4.2 Gradient Loss

The Gradient Loss aims to minimize the gradient of the model’s output with respect to its weights for the generated images ensuring that the generated images are closer to the training data, which is expected to have lower gradient magnitudes.

#### 3.4.3 Variational Loss

The Variational Loss is designed to promote the generation of noise-free images by minimizing large pixel variations by encouraging smooth transitions between adjacent pixels, effectively reducing high-frequency noise and ensuring that the generated images are visually consistent and realistic.

## 4 Experiments & Results

In this section, we present the experimental results obtained by applying our network inversion and reconstruction technique on the MNIST (Deng, ), FashionMNIST (Xiao et al., ), SVHN and CIFAR-10 () datasets by training a generator to produce images with desired labels. The classifier is initially normally trained on a dataset and then held in evaluation for the purpose of inversion and reconstruction. The images generated by the conditioned generator corresponding to the latent and the conditioning vectors are then passed through the classifier.

The classifier is a simple convolutional neural network with dropout, batch normalization, and leaky-relu activation followed by fully connected layers and softmax for classification. While the generator is based on Vector-Matrix Conditioning in which the class labels are encoded into random softmaxed vectors concatenated with the latent vector followed by transposed convolutions, batch normalization (Ioffe and Szegedy, ) and dropout layers (Srivastava et al., ) to encourage diversity in the generated images. Once the vectors are upsampled to $NXN$ spatial dimensions they are concatenated with a conditioning matrix for subsequent generation upto the required image size.

Figure: Figure 3: Inverted Images for all 10 classes in MNIST, FashionMNIST, SVHN & CIFAR-10.
Refer to caption: extracted/5945715/mfsc.png

The inverted images are visualized to assess the diversity of the generated samples in Figure 3 for MNIST, FashionMNIST, SVHN and CIFAR-10 respectively. While each row corresponds to a different class each column corresponds to a different generator and as can be observed the images within each row represent the diversity of samples generated for that class. It is observed that high weightage to cosine similarity increases both the inter-class and the intra-class diversity in the generated samples of a single generator. These inverted samples that are confidently classified by the generator are unlike anything the model was trained on, and yet happen to be in the input space of different labels highlighting their unsuitability in safety-critical tasks.

The reconstruction experiments were carried out on models trained on datasets of varying size and as a general trend the quality of reconstructed samples degrades with increasing number of training samples. In case of MNIST and FashionMNIST reconstructions performed using three generators each for models trained on datasets of size 1000, 10000 and 60000 are shown in Figure 4.

Figure: Figure 4: Reconstructed Images for all 10 classes in MNIST and FashionMNIST respectively .
Refer to caption: extracted/5945715/mfmrecon.png

While as for SVHN we held out a cleaner version of the dataset in which every image includes a single digit. In case of CIFAR-10 given the low resolution of the images the reconstructions in some cases are not perfect although they capture the semantic structure behind the images in the class very well. The reconstruction results on SVHN and CIFAR-10 using three different generators on datasets of size 1000, 5000, and 10000 are presented in Figure 5.

Figure: Figure 5: Reconstructed Images for all 10 classes in SVHN and CIFAR-10 respectively.
Refer to caption: extracted/5945715/screcon.png

## 5 Conclusion & Future Work

In this paper, we propose Training-Like Data Reconstruction (TLDR), a novel approach for reconstructing training-like data using Network Inversion from convolutional neural network (CNN) based machine learning models. We begin by introducing a comprehensive network inversion technique using a conditioned generator trained to learn the input space associated with different classes within the classifier using a combination of losses. By exploiting key properties of the classifier in relation to its training data we encouraged the reconstruction of training-like data and demonstrated that machine learning models remain vulnerable to inversion attacks.

As part of the future work, we plan to extend the TLDR approach to more complex architectures to understand privacy vulnerabilities in more advanced neural networks. Further improving the quality of reconstructed samples by leveraging the implicit bias of gradient-based optimization, which tends to memorize a subset of training samples near decision boundaries, will also be explored. Lastly, it would be of interest to evaluate the potential for learning generative models in cooperation with classifiers through network inversion guided by successive weight updates in the classifier during the training process.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: >While these attacks have been demonstrated in controlled settings, where models are typically over-parameterized or overly simplistic, the risks associated with sharing models trained on large, complex and multi-class datasets are yet been fully explored.

Some of the related work you mentioned did actually consider some of these factors. You may want to be more precise here.

>For under-parameterized models, where there is no possibility of memorization

This is an overstatement.
soundness: 1
strengths: - The related work presents an interesting connection between reconstruction attacks and works from the '90s.
- The method is fairly presented.
summary: This paper introduces a new method for reconstructing data that resembles the training dataset of an ML model. The method is based on two steps: inversion, where one learns the space corresponding to different classes, and reconstruction.
weaknesses: - The paper defines no metrics to evaluate the effectiveness of the method. The only results that are shown are reconstructed images.
- No empirical comparison is provided against state of the art methods (or any methods, for that matter). Unfortunately, this makes it impossible to judge how better the method is with respect to prior work.
- It is unclear what the main goal of the reconstruction attack is. In the literature, there are two: 1) reconstructing one (or more) images that look as close as possible to the original image (e.g., see Balle et al. 2022 as referenced by the authors), and 2) reconstructing images that resembles data from a certain label (e.g., (Fredrikson et al., 2015)). Based on the results (Fig 3-4), it seems to me that the proposed attack is trying to do the former but is achieving the latter.


(Fredrikson et al., 2015) "Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures". You may want to include this reference, which predates (Yang et al., 2019) in terms of model inversion attacks.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: Under what conditions which the attacks more successful versus not?
soundness: 2
strengths: The method extends inversion techniques from simpler architectures to CNNs, demonstrating its potential with complex datasets and classifiers.

The incorporates a few types of losses including Cross-entropy, KL divergence, cosine similarity, and feature orthogonality losses that may be useful to reconstruct training like data.

Demonstrates the model's effectiveness across multiple datasets, highlighting privacy risks in different scenarios.
summary: The paper introduces a network inversion approach to reconstruct training-like data from trained models. The paper modifies the inversion process to incentivize the generator to reconstruct training like data by exploiting several key properties of the classifier with respect to the training data. For example, the classifier is expected to be relatively more confident and robust in classifying training samples, the gradient of the classifier output with respect to the classifier's weight is also expected to lower for the training  data than for the random inverted image.
weaknesses: The paper assumes because training data possess these properties, we can use these properties to reconstruct the training data, which may not be theoretically right. For example, A => B does not mean B>A. 

The paper also do not have any formal metrics on how successful the reconstruction attack is. 

The paper also do not provide clear details about the experimental setup like details about the models, dropout rates and so on, and do not discuss details about how successful the attack can be for different types of models and architecture.

The paper may benefit more from detailed discussion about which loss is more important to the reconstruction attack.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: What exactly is the input to the generator? You discuss four approaches - label, vector, intermediate matrix and vector-matrix conditioning.
Additionally, if you only use vector conditioning, do you simply sample from a Gaussian, softmax it, and then feed this to the Generator?
Also, in Figure 1, the input to the generator seems to be a latent + conditioning vector. What is the source of the latent vector?

What exactly is the cosine similarity between? It says on Line 319 that "cosine similarity between features of generated images i and j". Where are two images coming from?

Typos and grammatical errors

Line 219 has a typo ' a diverse data distributions'

Line 234 - 'we given its simplicity'

Line 255 - for a encoding the label

Line 240 - learnt off of the labels each representative of the separate classes

Other 

Is the claim on Line 256 an observation you made during your research or a previously known fact from the literature? If so, please cite the relevant literature.

Do you think you could use a more powerful generative model i.e diffusion model?
soundness: 2
strengths: The main aim of the paper is quite clear to understand. The authors lay out the techniques they use to design their inversion network and explain the three desired properties they would want their generative model to have quite well. If their generative model is able to produce high confidence samples which have some room for perturbation (i.e some perturbation in the input space do not produce wildly different output distributions by the classifier) and have low gradient norm, then their model has a better likelihood of producing realistic samples.  I also liked their use of vector and matrix conditioning techniques which I believe are useful for generating controlled samples from generative models. The authors also provided a very comprehensive literature review in the space of model inversion based attacks which is well appreciated.
summary: The authors present a technique called Training-Like Data Reconstruction (TLDR) which can essentially create samples which are similar to the data which a neural network based classifier was trained on. Their technique is based on learning a generative model which can take in label encodings and produce outputs in the image space by using up-convolutions. The signal for training the generator comes from the classifier itself. Essentially, the authors come up with a loss function which encourages the generator to produce samples which the pre-trained classifier will classify into the same class as that of the conditional label provided to the generator. The loss function is made up of several regularizations and terms which encourage the generator to produce images which look similar to the training data. Evaluation is done on CNN based classifiers and 4 datasets including Cifar-10 abd MNIST.
weaknesses: The paper suffers from several weaknesses which I believe can be addressed.

Firstly, the experimental evaluation is very limited. The authors only perform evaluation on MNIST, FMNIST, SVHN and CIFAR-10. These are very simple datasets and we do not know if this technique will be applicable to more complex datasets such as Imagenet or Ms-COCO where finegrained details need to be captured. Secondly, their is no quantitative evaluation of how well the reconstructed images match the original training data. The authors only presented visual samples. Third, evaluation was only done on CNN based classifiers and it would be interesting to know if this technique can perform well on more modern architectures like ViTs.

Another concern I have about the paper is the complexity of the TLDR scheme. In total, there are 9 types of loss functions including KL divergence, Cross entropy, variational losses, feature orthogonality, cosine similarity etc. There is no clear understanding of the impact of each type of loss function and whether they are all necessary. I would have liked to see some ablation studies or theoretical justification for such a complex scheme.

Finally, I am concerned about the quality of the reconstructions themselves. CIFAR-10 is the most complex dataset they perform their evaluation on and many of the samples are hard to parse. The inversion scheme does not capture color / contrast very well and my understanding is that this inversion is only done to get a feel for what the training data was and not to be able to steal confidential training data and reuse it.

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 2
questions: None.
soundness: 3
strengths: The paper is well-motivated, the presentation is mostly on the positive side and mostly well-written.
summary: This work introduces a novel approach to reconstruction of training data from ML models using an inversion-based attack. The attack is evaluated on a number of CV benchmark datasets.
weaknesses: The biggest scientific weakness of this work is its impact. We have previously seen dozens of papers on model inversion and data reconstruction using various techniques ranging from inversion networks (e.g. [1, 2]) to gradient reconstruction ([3,4]). Note that these cover collaborative learning in most cases, as authors have already linked the relevant centralised settings in their related work. There is hardly anything new about the approach proposed here or the results obtained in this work. The datasets used here are the basic toy datasets that have previously been inverted multiple times using the techniques I linked above across a very large number of training settings and model architectures (mostly beyond the basic CNN architectures). While novelty is not the only factor that we are looking for when assessing submissions, I can hardly see any additional insights, unexplored work directions or interesting findings either. Almost everything in this work has previously been studied (e.g. the attack method, the use of priors, combination of multiple reconstruction factors etc.) in great detail and I do not see how the community benefits from this work.

In terms of more addressable concerns I have: the paper is not well-presented given the tight space constraints. With only 9-10 pages, authors should really concentrate on new methods, novel results and discussion. Currently, the introduction (which is in my view really inflated for no particular reason) takes 2 pages. This is not to mention the 2 pages of basic ML vocabulary, where each term has its own paragraph with margins (e.g. you do not need to explain what a cross-entropy loss is at ICLR). This adds up to about 4-5 pages of superfluous content. And given my criticism of the novelty and the impact of the work and its findings, this is exactly where this extra space should have gone - to explore the method in more detail, show novel insights etc. There are also 2 diagrams which I find to be relatively similar and they take about a page as well without adding much to the content (i.e. one would suffice, two are too much). 

While this may not be the comment the authors expected to hear, I would encourage them to concentrate on a) extracting as many novel scientific insights from their method as possible and b) restructuring the work so these results are clear to the reader. This would make the paper useful for the community and acceptable for publication. 

[1] - Usynin, Dmitrii, et al. "Zen and the art of model adaptation: Low-utility-cost attack mitigations in collaborative machine learning." Proceedings on Privacy Enhancing Technologies (2022).

[2] - He, Zecheng, Tianwei Zhang, and Ruby B. Lee. "Model inversion attacks against collaborative inference." Proceedings of the 35th Annual Computer Security Applications Conference. 2019.

[3] - Geiping, Jonas, et al. "Inverting gradients-how easy is it to break privacy in federated learning?." Advances in neural information processing systems 33 (2020): 16937-16947.

[4] - Boenisch, Franziska, et al. "When the curious abandon honesty: Federated learning is not private." 2023 IEEE 8th European Symposium on Security and Privacy (EuroS&P). IEEE, 2023.

Meta-review: The reviewers were unfortunately not excited about the paper. In particular there were concerns about the experimental setup, complexity of the scheme, and the scale of reconstruction.

Decision: Reject