Property Inference Attacks Against t-SNE Plots
Authors: Jingcan Chen
Xinlei He
Boyang Zhang
Yang Chen
Yang Zhang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Property Inference Attacks
t-SNE

Abstract: With the prevailing of machine learning (ML), researchers have shown that ML models are also vulnerable to various privacy and security attacks. As one of the representative attacks, the property inference attack aims to infer the private/sensitive properties of the training data (e.g., race distribution) given the output of ML models. In this paper, we present a new side channel for property inference attacks, i.e., t-SNE plots, which are widely used to show feature distribution or demonstrate model performance. We show for the first time that the private/sensitive properties of the data that are used to generate the plot can be successfully predicted. Briefly, we leverage the publicly available model as the shadow model to generate t-SNE plots with different properties. We use those plots to train an attack model, which is a simple image classifier, to infer the specific property of a given t-SNE plot. Extensive evaluation on four datasets shows that our proposed attack can effectively infer the undisclosed property of the data presented in the t-SNE plots, even when the shadow model is different from the target model used to generate the t-SNE plots. We also reveal that the attacks are robust in various scenarios, such as constructing the attack with fewer t-SNE plots/different density settings and attacking t-SNE plots generated by fine-tuned target models. The simplicity of our attack method indicates that the potential risk of leaking sensitive properties in t-SNE plots is largely underestimated. As possible defenses, we observe that adding noise to the image embeddings or t-SNE coordinates effectively mitigates attacks but can be bypassed by adaptive attacks, which prompts the need for more effective defenses.

Under review as a conference paper at ICLR 2023
PROPERTY INFERENCE ATTACKS AGAINST PLOTS
t-SNE
Anonymous authors Paper under double-blind review
ABSTRACT
With the prevailing of machine learning (ML), researchers have shown that ML models are also vulnerable to various privacy and security attacks. As one of the representative attacks, the property inference attack aims infer the pri- vate/sensitive properties of the training data (e.g., race distribution) given the out- put of ML models. In this paper, we present a new side channel for property inference attacks, i.c., t-SNE plots, which are widely used to show feature dis- tribution or demonstrate model performance. We show for the first time that the private/sensitive properties of the data that are used to generate the plot can be successfully predicted. Briefly, we leverage the publicly available model as the shadow model to generate t-SNE plots with different properties. We use those plots to train an attack model, which is a simple image classifier, to infer the spe- cific property of given t-SNE plot. Extensive evaluation on four datasets shows that our proposed attack can effectively infer the undisclosed property of the data presented in the I-SNE plots, even when the shadow model is different from the target model used to generate the I-SNE plots. We also reveal that the attacks are robust in various scenarios, such as constructing the attack with fewer t-SNE plots/different density settings and attacking t-SNE plots generated by fine-tuned target models. The simplicity of our attack method indicates that the potential risk of leaking sensitive properties in i-SNE plots is largely underestimated. As pos- sible defenses, we observe that adding noise to the image embeddings or I-SNE coordinates effectively mitigates attacks but can be bypassed by adaptive attacks, which prompts the need for more effective defenses.
INTRODUCTION
Machine learning (ML) models are becoming powerful and can be used feature extractor to gen- erate representations (also known as embeddings) for the input data (He et al. 2016; Sandler et al.] 2018 Huang al.) 2017). However, such representations 512 dimensions for the ResNet-18). To better understand the representations of data or demonstrate the model's performance, people usually use dimension reduction techniques such as (-SNE (van der Maaten & Hinton, 2008) (abbreviation for t-distributed stochastic neighbor embedding) to reduce high-dimensional representations into 2-dimensional space for visualization.
still in high-dimensional space (e.g..
Despite being powerful, ML models are also shown to be vuinerable to various privacy attacks that aim to reveal the sensitive information of the training dataset given access the target model. Property inference attack (Ganju et al. [2018) Zhou et al. 2022; Mahloujifar et al. (202)) is one of the representative attacks whereby the adversary aims to infer the sensitive global properties of the training dataset (e.g., the race distribution) from the representations generated from the target model.
I-SNE plots, on the other hand, are condensations of data representations. Such plots are usually considered to be safe and shared with the public via scientific papers or blogs. However, it is unclear whether such plots would leak sensitive property information about the data as well.
Our Work. In this paper, take the first step toward understanding the privacy leakage from t- SNE plots through the lens of the property inference attack against such plots. Here, we consider the general property the macro-level information of the dataset, e.g., race distributions. Note that this property is not necessarily related to the t-SNE plots, for instance, the t-SNE plot is used to show how distinguishable the gender distribution is, and the adversary infer the race distribution of

---

Under review as a conference paper at ICLR 2023
the data used to generate the plot. A successful attack may cause severe consequences it provides additional information to the adversary, which is often sensitive and should be protected. Also, can be used to audit the faireess of the dataset (Buolamwini & Gebru 2018).
In this work, we first systematize the threat model and attack methodology, which is straightforward. access the victim's t-SNE plot and may have knowledge of the
We assume that the adversary distribution of the target dataset, but they do not necessarily have access to the target model. To infer the general property of the samples in the i-SNE plot, the adversary first samples groups of images from a shadow dataset with different properties (e.g. different proportions of males). Then, those groups of images will be used to query the shadow model to get representations and generate groups of t-SNE plots. An attack model is trained based on the <<loo, property>> >> pairs, which is an image classifier that can distinguish between I-SNE plots with different labels. Once well trained, the attack model can then be used to infer the property of public 1-SNE plots.
Our evaluations on both classification and regression tasks show that the proposed attack can achieve high accuracy, on some of the datasets like CelebA and LFW. For instance, the accuracy for pre- dicting the proportion of males on CelebA I-SNE plots reaches 0.92, and the average regression error of predicting precise proportions is 0.02. We also study the reason for the relatively poor at- tack performance on the other datasets (e.g. FairFace) or attributes (e.g. Oval Face). We discover that this is due to the less distinguishable representations generated by the target model over these datasets/attributes. Also, we observe that the attack is still effective even when the shadow model is different from the target model. We further demonstrate that our attack is robust with fewer train- ing plots and transferable to different t-SNE density settings. For instance, the regression attack model trained on t-SNE plots with 750 sample points can reach low error of 0.04 t-SNE plots with 1, 000 or 500 sample points. We additionally show that, by mixing only a small number of t-SNE plots from a new dataset, our attack can transfer the new dataset. and we reveal the va- lidity of our attack when the target and shadow models are fine-tuned, which is common in practice SeecSontionn 5.3] details).
To mitigate the attack, we perturb the image embeddings/t-SNF coordinates and discover that it indeed reduces the attack performance to large extent, e.g., by adding Gaussian noise to the f-SNE coordinates, the inference error of regression attack model increases significantly from 0.02 to 0.48. However, such defenses can still be bypassed by an adaptive attacker (see also (eestinn 5.4). In short, our work demonstrates that the published t-SNE plots can be valid side-channel for the property inference attacks. We appeal to our community's 's attention to the privacy protection of publishing I-SNE plots.
2 BACKGROUND AND RELATED WORK
(-SNE Plots. Generally, 1-SNE (van der Maaten & Hinton 2008 is used to transform high- dimensional data to low-dimensional (e.g. 2D) data while preserving their relationship, i.e., sim- ilar data are mapped to the near space and dissimilar data projected more distant. To analyze how separable some specific characteristics of the data are, one common practice is to leverage the pre-trained MI. models feature extractors generate representations for the data and use t-SNE to reduce the high dimensional representations into 2D space for visualization, i.e., with t-SNE plots (see Figure 4(b) as an example). Also, t-SNE plots can be used show the performance of fine-tuned ML models (see Figure 13 as an example).
Property Inference Attack. Property inference intends to gain insights into the global properties of training datasets, which are unconsciously leaked. This poses a threat to the intellectual property of the data owner. In addition. property inference can audit the fairness of datasets, e.g. the gender fairness in benchmark datasets (Buolamwini & Gebru 2018). Previous work has shown the vul- nerability of property inference against machine learning models, including discriminative models and generative models. Ateniese et al.] (2015) proposes inference attack against shallow machine learning models, e.g. Hidden Markov Models and Support Vector Machines. Ganju et al.) (2018) proposes the first property inference attack against fully connected neural networks. Both of the work assumes that the adversary has white-box access to the machine learning models. Concretely, the adversary first trains shadow models datasets with different properties, then leverages the pa- rameters of shadow models to train the meta classifier to identify the property of the training dataset. Mahloujifar et al. (022) conducts the property inference attack by injecting poisonous data into
2

---

Under review as a conference paper at ICLR 2023
the training dataset and then querying the target model in black-box fashion. Zhou al. (2022) reveals the first set of training property attacks against GANs, where the adversary also has only black-box access to the target model. These previous works focus on the training data of machine learning models, and many assume that the adversary has white-hox access to the model. Our work concentrates on both the white-box and black-box property inference attack against I-SNE plots, the safety and privacy of which are paid less attention to when published. Also, our proposed attack trains& a regression model to precisely infer the property.
Note that in this paper, we consider the adversary is interested in the property of the data points that are used generate the t-SNE plots.
3 THREAT MODEL AND ATTACK METHODOLOGY
3.1 THREAT MODEL
t-SNE plots are usually leveraged show the feature distribution of the data or demonstrate the representation ability of machine learning model. Such plots are commonly published on websites or in scientific papers. In this work, we propose the first property inference attack against published t-SNE plots where anyone observing the plots could be the attacker. The I-SNE plot may only present insensitive information, but other sensitive information of the dataset could also be inferred. For instance, model owner publishes t-SNE plot of human face images, labeled by age, which is intended to show the effectiveness of the age classification ML model. However, the attacker can infer the proportion of males in the samples that are used to draw the t-SNE.
Adversary's Goal. An adversary aims to infer the sensitive property Ptarget of the data samples that are used to generate a t-SNE plot. And the property is not directly shown by the t-SNE plot. To be specific, the property we consider in this paper is the distribution of a certain attribute, e.g. the proportion of males.
Adversary's 's Background Knowledge. We make the following assumptions about the adversary's background knowledge. First, the adversary has access to the I-SNE plot and can make adjustments to the plots, e.g. removing axes, labels, plot titles, and changing the color. Second, the adversary aware of the dataset distribution but does not know which subset of images are used to generate the t-SNE plots. This means that the adversary can obtain a shadow dataset and generate t-SNE plots with subsets of images that have different property values, e.g., different proportions of males. We later show that even if the shadow dataset comes from a different distribution, with only a few numbers of t-SNE plots that share the same data distribution as the target t-SNE plot, our attack can successfully infer the property of the data used to generate the t-SNE plots. Third, the adversary has a shadow model to embed the images. We first assume the shadow model is the same as the target model. We later show that the attacks are still effective when the shadow model is different from the target model (Figure and Table 4p.
3.2 ATTACK METHODOL.OGY
Briefly, the attack consists of two parts: t-SNE generation and attack model training.
E-SNE Generation. To correctly infer the target property Ptarger from the given i-SNE plot, one possible solution is to construct set of t-SNE plots that have different values of the target property, e.g., different proportions of males in the plot data. Concretely, the adversary first samples many subsets of data from a shadow dataset, each with a different value of the target property. Then, they use each subset of images to query the shadow model and get the image embeddings (output from the second-to-the -ttt-te------t layer of the shadow model). The adversary then projects the embeddings into 2D space and generates t-SNE plot for each subset. The t-SNE plots are labeled by the property value of the image subset. Note that if the adversary has white-box access to the target model, they can directly apply the target model the shadow model to generate t-SNE plots.
Attack Model Training. With the generated t-SNE plots, the adversary trains an ML model to predict the property values of the plots. Note that in this paper, we consider both classification and regression tasks, while previous property inference attacks (Ateniese et al. 2015] Ganju et al. 2018 Mahloujifar et al., 2022) only consider the classification task. The input is the generated L-SNE
3

---

Under review as a conference paper at ICLR 2023
plots, and the output is the property value of the t-SNE plots. Once the attack model is trained, it can be used to infer the properties of public t-SNE plots.
4 EXPERIMENTAL SETUP
Datasets and Models. We consider 4 face datasets in our evaluation, including CelebA (Liu al.
2013 LFW (Huang et al., 2008). UTKFace (Zhang et al. 2017). and FairFace (Kärkkänen & Joo
2021 Regarding the model architectures, we use the ResNet18 (He et al.) (016). ResNet34 (He)
et al. 2016) and the small version of MobileNerV3 (Howard et al., 2019) pre-trained on ImageNet as the target and shadow models' architectures. Also, we use the pre-trained DenseNet121 (Huang iet al. (0177) as the attack model's architecture. More details of the datasets and models are [Sec- tion I.IIand In.dInnSSctioon Section A.2 Each dataset is first randomly divided into two equal parts, i.c., the shadow dataset and the target dataset, without overlapping. In our experiment, the shadow dataset is used to generate i-SNEs for training the attack model, while the target dataset is used to generate t-SNEs
with different property values for the testing.
Target/Shadow Model and Two Experiment Settings. To show the effectiveness of the attack, we conduct experiments on both multi-class classification and regression tasks. For both of the tasks, we consider two settings. Setting is called same shadow model where the adversary has white-box access the target model and can directly leverage the target model as the shadow model. Setting is called different shadow model where the adversary has to construct the shadow model themselves. By default, we set the shadow model as pre-trained ResNet18. 8. It means that, for setting the target model is pre-trained ResNet18 8 as well. For setting 2, the target model is pre-trained ResNet34 or MobileNetV3.
(-SNE Generation. To generate I-SNE plot, we first sample 1,000 images with specific pro- portion of certain attribute from the shadow (target) dataset. Specifically, we query the shadow (target) model with these sampled images and take the output of the second-to-the-lee layer the samples' embeddings to generate the t-SNEs. Since we assume the adversary can adjust the color of the target model's s t-SNE (e.g. using image editing tools), conduct attack in a more uniform (and more difficult) setting with no label and color information in t-SNEs (see Figure 4(a) as an example of the t-SNE plots use). And the figure size of 300 pixels 300 pixels.
For multi-class classification, we set five proportions as labels to infer, i.c. Ptorget E (0.3, 0.4, 0.6, 0.7) following (Zhou et al. 2022). For each label, we generate 150 I-SNE plots using the shadow model for attack model training, and 50 I-SNE plots using the target model for testing. For regression tasks, the proportions range from 0.0 1.0, with stride of 0.01. The attack model's training set contains 40 t-SNEs for each label (4,040 in total), while the testing set consists of 10 t-SNEs for each label (1,010 in total).
To investigate our uttack's transferability, we also evaluate the case where the target t-SNEs are generated from the unseen dataset (see Table I [ndd Table 2}.
Attack Model Training, By default, we use the ImageNet pre-trained DenseNet121 with the Adam optimizer as the attack model. For classification, we set the training batch size to 64, the learning rate to 0.0001, the cross-entropy as the loss function, and the training epoch to 30. For regression, we set the training batch size to 128, the learning rate to 0.0001, the MSE as the loss function, and the training epoch to 100.
EVALUATION RESULTS
5.1 RESULTS OF MULTI-CLASS CLASSIFICATION TASKS
We first consider setting where the adversary has white-box access to the target model. We conduct experiments different datasets, different target properties, and different model architectures.
Different Datasets. From Figure [(a) we observe that our attack can successfully reveal the pro- portion of males in some of the datasets. For instance, in CelebA, attack reaches 0.92 accuracy while the baseline accuracy is only 0.20. Another finding is that the inference accuracy varies across different datasets, e.g., 0.92, 0.56, 0.41, and 0.22 on CelebA, LFW, UTKFace, and FairFace,
4

---

Under review as a conference paper at ICLR 2023
L
0.
L.0
$1.0
6.8
2.82
5556
0.37
(1.
0.4
0.4
0.2
0.2
Smiling
ReMeell
MubiefietV3
(a) Accuracy male proportion of (b) Accuracy different target (c) Accuracy male proportion of CelebA with different models.
different datasets.
properties of CelebA.
Figure Multi-clasiffiationn results under setting for different datasets, target properties and models. The testing data is from the same distribution as the training data. For (a) and (b), both the target model and shadow model are pre-trained ResNet18.
Table 1: Multi-classification transfer study under setting 1. The accuracy values are in bold for the testing data that is from the same distribution as the training data.
Table 2: Regression transfer study under setting
1. The error values are in bold for the testing data that is from the same distribution as the training data.
Training Data
Target Property
Cel.
UTK,
Fair:
I.FW
Training Data
Target Property
Cel.
|TT.
Fair.
L.FW
ColebA
Male
0.92
0.28
0.21
0.29
ColebA
Male
0.38
0.35
0.27
UTKFace
Male
0.24
0.41
0.19
0.22
UTKFace
Male
0.32
0.08
0.36
0.38
FairFace
Male
0.20
0.22
0.22
0.21
Fairface LFW
Male
0.22
0.17
LFW
Male
0.11
0.21
0.20
0.56
Male
0.32
0.37
0.41
0.07
spectively. We credit this to the distinguishability of overall image embedding distribution when the proportion males changes. To confirm this, for each t-SNE plot, we calculate the average em- (-SNE project
bedding of all image embeddings that used generate the plot. Then these average embeddings of all 750 1-SNEs (150 per class) and color them by the property, i.e. proportion of males, show the distinguishabiliyy of different classes. Figure 5exhhibsts the results. 5()) shows five clear dense clusters, indicating that the five classes of t-SNEs from CelebA are well distinguishable. The t-SNEs of different classes from UTKFace (Figure 5(b) and LFW (Figure 5(d)) are also clustering well, but slightly sparser than CelebA for each cluster. However, in stark contrast the other three datasets, the average embeddings of t-SNEs of different classes from FairFace are evenly distributed across the plot, in Figure 5(c) indicating that they are difficult to distinguish.
Figure 5(a)
Different Target Properties. We further conduct the experiments on different target properties of CelebA, shown in Figure I(b) We observe that the attack performs well at inferring the proportions of Male (0.92) and Big Nose (0.57) but worse at inferring the proportions of Smiling (0.30) and Oval Face (0.26) in CelebA. We impute this to the bad performance of the feature extractor when distinguishing Smiling faces and Oval Face faces. To verify this, with a pre- mre-trained ResNet18 as the target model, for each attribute, we randomly select 1,000 face images with half of the images having the attribute (e.g., 500 male and 500 female images) and query the target model obtain the embeddings and generate a t-SNE. We mark the two groups with different colors, e.g. half of the images are male, colored by yellow while the other half are female, colored by purple. The results are shown in Figure 6) From Figure 6(a) and Figure 6(b) the two clusters are more discernible than Figure 6(c) Figure 6(d). This suggests that the pre-trained ResNet18 cannot separate Smil- ing (Oval Face) faces from not Smiling (Oval Face) faces well, which further makes the t-SNE plots with different proportions of Smiling (Oval Face) faces indistinguishable.
Different Models. We also find that our attack is effective across different target model architec- tures. As shown in Figure I(c) the attack accuracy are 0.92, 0.91, and 0.82, on ResNet18, ResNet34, and MobileNetV3, respectively, which is significantly higher than the bascline accuracy (0.2), Also, MobileNetV3 has lower inference performance compared to ResNets. We suspect the reason that MobileNetV3 is smaller and has lower representation ability than ResNets and thus makes the t-SNEs less separable.
Transfer Study. In more difficult scenario when the target t-SNEs generated from different datasets (see Table I), we observe that the attack fails with an accuracy around 0.2 (random guess baseline). This is expected the difference across datasets may result in huge changes to
5

---

Under review as a conference paper at ICLR 2023
Table 3: Multi-classfiiation results under setting The shadow model is pre-trained ResNet18.
Table 4: Regression results under setting 2. The shadow model is pre-trained ResNet18.
Target Model
Training Data
Target Property
Test Acc.
Target Model
Training Data
Target Proporty
Test Acc.
ResNet34
CelehA
Male Male
0.86
ResNet34
CelebA
Male Male
0.03
MobileNerV3
CelehA
0.46
MobileNerV3
CelehA
0.13
the t-SNEs, making the inference more difficult. We later show in [Section 5.5 that the attack can transfer I-SNEs generated from other datasets by introducing small number of I-SNEs from those datasets to further train the attack model (similar to few-shot learning).
Different Shadow Model. For setting 2, the shadow model architecture different from the target model. We use the pre-trained ResNet34 and the small version of MobileNetV3 as the target model, and the pre-trained ResNet18 as the shadow model to infer the proportion of males on CelebA. Results are exhibited in Table 3 Overall, the inference accuracy decreases, compared to setting 1. For instance, the accuracy on male proportion of CelebA is 0.86 under setting with ResNet34 the target model, which, under setting 1, is 0.92. However, the accuracy is still much better than random guessing (0.20). When the target model is MobileNetV33, the inference accuracy severely drops to 0.46. This is because ResNet34 shares similar modules with ResNet18, and thus the image embeddings extracted by ResNet34 (target model), compared to MobileNetV3, are more similar to ResNet18 (shadow model). This will lead to similar t-SNE patterns and higher inference accuracy.
To conclude, our results show that the same attribute of different datasets does not equally suffer from the attack, and proportions of some attributes such as Male and Big Nose are more easily inferred than other attributes. We visualize the t-SNEs with different proportions Grad-CAN (Selvaraju et al., (017)) in the appendix (seefSection A.3). Also, when the shadow model different from the target model, the attack is more effective when the shadow model shares similar architectures with the target model (e.g., ResNet18 ResNet34).
males using
5.2 RESULTS OF REGRESSION TASKS
To generalize the attack and infer the property more precisely, we frame our attack as regression tasks on different datasets, target properties, and model architectures. The evaluation metric is the average error between the predicted proportion and the ground truth value in the regression setting. For instance, if the predicted proportion is 0.10 and the ground truth is 0.15, the error for this (-SNE plot 10.15 0.10] 0.05. If there are n (-SNE plots in the testing dataset, the average error is calculated over the I-SNE plots.
Altaek Brer
Attekk Froe
Attak 6.2
0.17
G.
0.2
0.27
0.16
0.02
FarFucee
Smilng
Big Oval
MobileNeeV
(a) Regression male pro- (b) Regression error propor- (c) Regression portion of different datasets.
male pro- tion of different target properties portion of CelchA with different CelebA.
models.
Regression results under setting 1 for different datasets, different target properties and different models. The testing data is from the same distribution as the training data. For (a) and (b), both the target model and shadow model are pre- pre-trained ResNet18.
Figure 2:
Different Datasets. We first conduct experiments with the target and shadow models being identical. Figure 2(a) reveals the effectiveness of the attack on the male proportion inference on different datasets. The highest average error is 0.17, FairFace, while the average errors on the other three datasets are all lower than 0.10. When inferring the proportion of males on CelebA, the attack model reaches the lowest eтror of 0.02. Note that the average error of random guessing is 0.34. These results are consistent with those in Figuur I(a) where the target property from FairFace also difficult to infer. The results show that, after learning from the training I-SNE plots with sufficient

---

Under review as a conference paper at ICLR 2023
different properties (from 0.00 to 1.00 in our experiments), the attack model can precisely infer the continuous proportion.
Different Target Properties. Similar to the multi coltiiclaassifiaaiinn tasks, the attribute affects the per- formance of the attack in regression tasks. From Figure 20b the inference performance on the proportion of Oval Face has the highest error of 0.16. The inference accuracy Oval Face also lags behind in the multi-classification task.
Different Models. Figurr 2(c) shows that the attack is effective when the model ResNet34 or MobileNetV3. For example, when the target model and shadow model are MobileNetV3, the inference on the male proportion of CelebA is 0.04. Although this is still slightly worse than R and ResNet34, it is much more precise than the multi-classification case.
Transfer Study. Table 2
shows the results of transfer study on regression tasks under setting 1. Consistent with the results of multi-classffication (see Table 1). the attack performance on an unseen dataset poor, with the average errors ranging from 0.21 to 0.41, most of which are worse than random guessing. However, adding small number of t-SNEs from the new dataset can boost the performance of our attack significantly (see Section 5.5 for details).
Different Shadow Model. When the target model (ResNet34 or MobileNerV3) is different from the shadow model (ResNet18) 8), our attack still works well (see [able 4). For ResNet34, the inference error on male proportion of CelebA marginally increases compared to setting 1, from 0.02 to 0.03, while the error increases to 0.13 for MobileNetV3, but still much better than random guessing.
In summary, the results of regression tasks are in accord with those of multi-classification tasks. The attack performs well on the testing data that are from the same distribution as the training data, while performs worse unseen datasets. However, it is still valid across different properties and target models. The precision of regression tasks further demonstrates the potential severity of information leakage from i-SNE plots is underestimated.
5.3 ABLATION STUDY: DIFFERENT E-SNE SETTINGS
In this section, investigate whether our attack is effective with: (1) fewer training t-SNE plots for each label, and (2) fewer sample points in each t-SNE. All the experiments are conducted on CelebA, and the target property is the proportion of males. For all the experiments, the shadow model is pre-trained ResNet18. The target model can be ResNet18 (setting 1) or ResNet34 (setting 2), denoted by the legend in the figures.
Number of t-SNE Plots for Each Class. Figure 7 shows the performance of multi-classification and regression tasks. Generally, with more I-SNE plots for each class, the accuracy of multi- classification increases and the error of regression decreases, for both setting and setting 2. For the multi-classification case, 50 t-SNE plots for each class (250 plots in total) can achieve an accuracy of almost 0,80, even if the target model is not the same as the shadow model. And the regression error is lower than 0. with only 10 t-SNE plots for each class (1,010 plots in total). This further demonstrates the vulnerability of tNE plots against the property inference attack.
Density Settings. Density denotes the number of sample points (images) represented in t-SNE plot, which may change the geometric features of 1-SNE plots and subsequently affects the perfor- mance (see Figure 10p. We additionally generate I-SNE plots, each with 500 and 750 sample points from CelebA (CelebA_500 and CelebA_750 for short), for multi-classificatin tasks and regression tasks. The number of training plots and testing plots is the same as the default setting, Figure 8 and Figure 9 exhibit the inference performance of attack models trained different density settings. The inference performance is the best when the attack model is trained on the plots with the same density setting as the testing plots. However, when the training is CelebA, 750, the inference performance is good on both CelebA 500 and CelebA_ 10000, under both setting 1 and setting 2. For example, in Figure and Figure 9(c). the regression error trained on CelebA is less than 0.04 and less than 0.15 respectively, for both of the lines in each figure.
5.4 POSSIBLE DEFENSES AND ADAPTIVE ATTACKS
To mitigate the attacks, we investigate defense approaches based on disturbance, including nois- ing, rounding, and thresholding. The experiments conducted over regression tasks on CelebA.

---

Under review as a conference paper at ICLR 2023
Table 5: Inference errors with different defense strate- gies. The successful defenses are in bold.
Table 6: Adaptive attack performance
against the successful defense methods.
Defense Methods
Inference Firor
Defense Methods
Inference Error
defense
0.02
0.03
Defense Methods
Adaptive Err.
Noo-adaptive Err.
E-N 2"SID
0.48
C-R 0,1
0.02
E-N STD
0.15
0.48
E-N 1"SID
INT
0.05
E-N PSIID
C-N 5% STTD
0.48
E-T 50%
0.03
C-N STD
0.25
0.48
*SID
0.48
E-7
0.02
C-N STD
E-R 0.1
0.02
Both the target model and shadow model are pre-trained ResNet18 and the target property is the proportion of males. Note that we consider this setting the most challenging one for defenses as it achieves the highest attack performance in previous experiments.
Gaussian Noise in Embeddings (E-N) and Coordinates (C-N). We first test the defense of adding Gaussian noise to embeddings and the t-SNE coordinates. For the E-N, we add the Gaussian noise each image embedding before fitting the t-SNE. The mean of the noise is set and the standard deviation is set to a percentage of the current embedding values' standard deviation. For the C-N, the Gaussian noise is added to i-SNE coordinates. Concretely, in the current i-SNE plot, the stan- dard deviation of the noise added coordinates is set to percentage of all coordinates' standard deviation. The noise is added to y coordinates the same way. Results in Table 3 show the effective- ness of such defense methods. With Gaussian noise of I and 2 times the original standard deviation added to the embedding values (E-N), the inference error increases by 0.28 and 0.46 respectively, compared to the no defense (0.02). With Gaussian noise of 2% and 5% original standard deviation added to the coordinates (C-N), the inference error increases by 0.46. To evaluate the effect on the visual utility of such defense methods, we draw example I-SNE plots in Figure 11] Visually, (-SNE protected by E-N-1 Fiiuure I Ilbb) still shows two clear clusters, while E-N-2 (Figure II()) severely jeopardizes the visual utility. In C-N-2% (Figure Ildd) and C-N-5% (Figure II(e)), sample points of two colors are slightly blended, suggesting that they also influences the visual utility of t-SNEs.
Rounding Embedding Values (E-R) and Coordinates (C-R). Rounding is to round embedding values or t-SNE coordinates to a specified number of decimal places, reducing the precision of the embedding values and t-SNE plots. We observe that when we round the values to 0.1 or integer, neither F-R nor C-R can defend the I-SNE plots from the attack, with the inference error almost unchanged.
Threshold Embedding Values (E-T). For this defense, in each embedding, only the largest k per- centage values are retained, while other values are set to O. From Table 5] the defense both fails when & is set to 50 and 75, indicating that it is not effective to simply drop some values in the em- bedding. We suspect that the important information remains the largest 50% values and can still be captured by the t-SNE.
Adaptive Attack. We then consider an adaptive adversary who aware of the defenses and can add those defended t-SNE plots into the training datasets. Specifically, the training t-SNE plots processed in the same way as the effective defenses, i.e. the training t-SNE plots are generated with noised embeddings, or noise is added the coordinates in the plots, which are the two effective defenses. The results are exhibited in TTbble 6) With the adaptive attack, all the attack errors drasti- cally decrease. For instance, the B-N FSTD adaptive attack achieves low error of 0.04, while the error adaativee attack on the defended t-SNE plots is 0.30. The error of the adaptive attacks on C-N 5% STD 0.25 and on C-N 2T is 0.24. which both maintain relatively high. This result demonstrates the effectiveness of the C-N defense method. However, as we observe in FFiuree 11 C-N decreases the visual utility of L-NNE which calls for more effective and harmless defenses.
5.5 OPEN-WORLD SETTINGS
In this section, we investigate the robustness of our attack under more realistic settings. The adver- sary can train the attack model with mixed datasets and infer properties of t-SNE plots generated by fine-tuned models.
Mixed Datasets. Previously, observe that the attack model trained on the data with only 5 t- SNE plots for each class does not have good inference performance (see/Figure 7(b)). Now for each proportion in the training dataset, we mix the 40 t-SNE plots generated from FTKe with only
8

---

Under review as a conference paper at ICLR 2023
small number of CelebA t-SNE plots, to evaluate our attack. For both setting and setting 2, the shadow model is pre-trained ResNet18. For setting 2, the target model is pre-trained ResNet34. From Figure 12] by mixing only 5 CclebA plots for each class, the inference eπror the testing CelebA decreases significantly. from more than 0.30 to 0.05, under both target model settings. This performance is almost identical the performance of the attack model only trained on CelebA (0.02 and 0.03, see see Figure 2(a)and Table 4). However, the performance on the UTKFace dataset does not improve, by adding CelebA plots to training data. These results suggest that some datasets (e.g. CelebA) may be more susceptible to such attack and can be accurately inferred even if the training t-SNE plots are mostly from another distribution.
Attack on t-SNE Plots Generated by Fine-tuned Models. In practice, the model owner may fine-tune pre-trained model on a specific target task, e.g. classifying the gender. Then, they may consider generating t-SNE plots colored by the task label (e.g. the gender or the race, see Fig- ure 13(a) anniaanndFiiggg Figure 13(b) using the embeddings extracted by the fine-tuned model showease the performance of the model. Besides the information that we assume the adversary to have in [Sec- [ion 3] the adversary also has knowledge of the target task. The adversary then fine-tunes another pre-trained model on the same task, using the shadow dataset. We conduct the attack with regression tasks in this scenario. The target model is pre- pre-trained ResNet18 fine-tuned on the target task. We randomly sample 10,000 images from the target (shadow) dataset fine-tune the target (shadow) model, 8,000 for training and 2,000 for testing. Note that we here train the shadow model and target model under the same hyperparameters. with training epochs 8, batch size 64, learning rate lc-4, and Adam the optimizer. In Table 7. the fine-tune task accuracy is shown in the Target Acc. and Shadow Acc. column.
From the results in Table 7, we observe that the inference performance is good even when the shadow model (ResNet1 8) is different than the target model. The inference error for ResNet34 is 0.10 while for MobileNetV3 is 0.13. These results reveal that our attack remains valid when the shadow model does not share parameters with the target model, as they are fine-tuned on non-overlapping data, which demonstrates that the threat to data privacy is underestimated in this practical scenario.
6 LIMITATIONS
Limited Scenarios. Note that our attack infers the property of the data that are in the t-SNE plot. One of the possible scenarios of our attack is to audit the ML model fairness. For instance, a model owner may claim the "good" performance of its model by t-SNE plot with specific property, e.g. 100% light-skinned people, as the "good" performance may be biased toward them (Buolamwini & Gebru, 2018). Our attack can uncover such unfairness by inferring the property of the t-SNE. However, if the data owner publishes a t-SNE plot with a property that is not presented in the original dataset (e.g. the proportion of males is not the same), without any performance bias, the attack may be not be useful. This is one limitation of the attack.
Embedding Layer Specification. In this paper, we assume that it is a common practice to the outputs of the second-to-the-lee layer as image emebeddings. For instance, Vision Transformer (ViT) (Dosovitskiy
al.) 2021) uses the output of the classification token (the second the last layer) to feed in the linear classifier (the last layer) for classification. However, the outputs from other layers can also be used for the t-SNE generation, which may render our attack invalid, as the attacker is unware of which layer to use. This is a limitation of our work and we leave the exploration for the future.
CONCLUSION
In this paper, we perform the first property inference attack through i-SNE plots. Experiments show that our attack is effective in inferring undisclosed properties of data in t-SNE plot, i.e. the distribution of a certain attribute in the i-SNE plots, and even precisely with regression models. To mitigate the attack, we evaluate several defense mechanisms. Our evaluation result shows that adding noise to the embeddings or the t-SNE coordinates can defend against the attack. However, the defense of adding noise coordinates can affect the utility of the t-SNE plots can be nullified the adversary adaptively conducts the attack. In the future, we plan to investigate the vulnerability of different attributes and datasets, as well as the effective transfer attack and defense methods.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Giuseppe Ateniese, Luigi V. Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, and Giovanni Felici. Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers. Int. J. Secur. Networks, 2015.
Joy Buolamwini and Timnit Gebru, Gender Shades: Intersectional Accuracy Disparities in Commer- cial Gender Classification. In Conference Fairness, Accountability, and Transparency (FAT*), pp. 77-91. PMLR, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Representations (ICLR), 2021.
Karan Ganju, Qi Wang, Wei Yang, Carl A. Gunter, and Nikita Borisov. Property Inference At- tacks on Fully Connected Neural Networks using Permutation Invariant Representations, ACM SIGSAC Conference on Computer and Communications Security (CCS), pp. 619-633. ACM, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778. IEBE, 2016.
Andrew Howard, Ruoming Pang. Hartwig Adam, Quoe V. Le, Mark Sandler, Bo Chen, Weijun Wang, Liang-Chieh Chen, Mingxing Tan, Grace Chu, Vijay Vasudevan, and Yukun Zhu. Search- ing for MobileNetV3. In IEEE International Conference on Computer Vision (ICCV), pp. 1314 1324. IEEE, 2019.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications. CoRR abs/l 000004444..,...1 2017.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely Connected Convolutional Networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2261-2262222222 2269. IEEE, 2017.
Gary B. Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments. In International Workshop on Faces in Real-Life Images: Detection, Alignment, and Recognition (IWFRI), 2008.
Gary B. Huang, Marwan A. Mattar, Honglak Lee, and Erik G. Learned-Miled Learning to Align from Scratch. In Annual Conference on Neural Information Processing Systems (NIPS), pp. 773- 781. NIPS, 2012.
Kimmo Kärkkäinen and Jungseock Joo. FairFace: Face Attribute Dataset for Balanced Race, Gen- der, and Age for Bias Measurement and Mitigation. In Winter Conference on Applications of Computer Vision (WACV). pp. 1547-1557. IEEE, 2021.
Neeraj Kumar, Alexander C. Berg, Peter Belhumeur, and Shree K. Nayar. Attribute and Simile Classifiers for Face Verification. In IEEE International Conference on Computer Vision (ICCV), pp. 365-372, IEEE, 2009.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes in the Wild. In IEEE International Conference on Computer Vision (ICCV), pp. 3730-3738. IEEE, 2015.
Saeed Mahloujifar, Esha Ghosh, and Melissa Chase. Property Inference from Poisoning. In IEEE Symposium on Security and Privacy (S&P), pp. 1120-1137. IEEE, 2022.
Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Ching Chen. MobileNetV2: Inverted Residuals and Linear Bottlenecks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520. IEEE, 2018
10

---

Under review as a conference paper at ICLR 2023
Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient- Bradient-Based Localization. In IEEE International Conference Computer Vision (ICCV), pp. 618-626. IEEE, 2017.
Laurens van der Maaten and Geoffrey Hinton. Visualizing Data using t-SNE. Journal of Machine Learning Research, 2008.
Zhifei Zhang, Yang Song, and Hairong Qi. Age ProgressionRRegrssionn by Conditional Adversarial Autoencoder. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 4352-4360. IEEE, 2017.
Junhao Zhou, Yufei Chen, Chao Shen, and Yang Zhang. Property Inference Attacks Against GANs. In Network and Distributed System Security Symposium (NDSS). Internet Society, 2022.

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DATASETS
CelebA. CelebA (Liu et al., 2015) is a large-scale face attributes dataset. There are more than 202K face images, each with 40 attribute annotations. We focus on the following attributes of CelebA: Male, Smiling, Big Nose, Oval Face. These attributes not only include some whole facial features (Gender, Oval Face), but also local detailed features (Big Nose, Smiling). The attack model will infer the proportion of each attribute the samples used to generate the (-SNEs.
LFW. LFW (Huang et al. 2008) were originally provided for face verification, also known as pair matching under both restricted and unrestricted settings. The 13K images were further labeled with 73 attributes by attribute classifiers from Kumar et al. (2009). Many of the attributes overlap with CelebA. We use the deep-funneled version (Huang et al. 2012)
) of the data to infer the proportion of
the same attributes CelebA for the ablation study.
UTKFaae. UTKFaaee dataset (Zhang et al. (017) contains 23,708 face images with a long age span from 0 to 116 years old. Each image is labeled with age, gender, and ethnicity. In our experiments, we will infer the proportion of males and white people.
constructed mitigate the race bias problem in common datasets. A total of 108,501 images labeled with gender, race and age are collected from the YFCC-100M Flickr dataset and balanced on race groups, i.e. White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino. The images were split into training set and validation set in the original work. In this paper, we use the training set with 86K images. For the ablation study purpose, we will infer the proportion of males and white people as UTKFace.
FairFace. FairFace dataset (Kärkkäinen & Jool 2021)
A.2 MODELS
ResNet. ResNet (He et al. 2016) is a deep learning model family, which utilizes residual learning to ease the training of deep networks. In this paper, use the pre-trained ResNet18 and ResNet34 as the target model and the shadow model.
DenseNet. DenseNet (Huang et al. 2017) adds connections between each layer and every other layer. It mitigates the vanishing- grangggtgr problem and reuses features between layers. We use the pre-trained DenseNet121 1 as the attack model for all the experiments infer the dataset property of (-SNEs.
MobileNet. MobileNet (Howard al., 2017 uses depthwise separable convolutions reduce the calculation during convolution. In this paper, we use MobileNetVV (Howard et al. 2019) as the target model and the shadow model.
A.3 GRAD-CAM VISUALIZATION
We try to apply Grad-CAM heatmaps (Selvaraju et al. 2017) to I-SNE plots of different proportions of males from CelebA to show the characteristics that the attack model pays attention to. The attack model is trained on t-SNE plots generated by ResNet18 8 for multi-classffiaatin,, and we randomly select three plots for each proportion (0.3, 0.4, 0.5, 0.6, 0.7) from the CelebA test data, which are also generated by ResNet18.
In Figure 3] the first row presents the original plots for the second row, which are hard to distinguish by human eyes. However, from the heatmaps, notice that the most concentrated part of plots with label 0.3 is the pivotal characteristic, while for 0.4, the attack model attends to detect a more diffuse part. For 0.5, the model focuses on the most sparsest part, and for 0.7, the focal characterstic is the sparse points between two groups of dense points. These different patterns are distinguishable for the attack model, but not that visually clear.
A.4 ADDITIONAL. RESULTS
12

---

Under review as a conference paper at ICLR 2023
(a) 0.3.
(b) 0.4.
(c)
(d) 0.6.
(e)
Figure 3: Grad-CAN heatmaps of t-SNE plots of different proportions of males from CelebA. There are 1000 data points in each plot. The first row are the original plots of the second row.
(a) Without colors and labels.
(b) With labels, colored by gender.
Figure 4: i-SNE plots of the same proportion (0.0) of Big Nose faces from CelebA. The images are embedded by pre-trained ResNet18.
Table 7: Attack in the fine-tune scenario. The target (shadow) accuracy is the accuracy of the target (shadow) model on the target task. We then the fine-tuned target (shadow) model to generate testing (training) t-SNE plots. The shadow model is fine-tuned ResNet18.
Target Task
Target Model
Target Ace.
Shadow Acc.
Target Pruperty
Test Err.
CelebA Male Classification (binary)
ResNet18
0.98
Smiling
0.08
CelebA Male Classification (binary)
ResNe134
0.96
0.97
Smiling
0.10
CelebA Male Classification (binary)
MobileNetV3
0.96
0.97
Smiling
0.13
13

---

Under review as a conference paper at ICLR 2023
1.0
1.
1.0
1.0
0.8
0.8
0.0
0.8
9.6
0.6
0.6
5.00
0.25
0.50
0.75
1.50
0.03 525
350
0.5
1.00
0.00
025
3.50
0.5
1.50
5.00
0.25
0.50
0.75
1.0
CelebA.
(b) UTKFace.
FairFace.
I.F.
Figure 5: t-SNE plots of average embeddings of t-SNEs from different datasets, clustered by the proportion of males the t-SNEs, scaled to [0, 1]. The average embedding of t-SNE is the average embedding of all the image embeddings in this I-SNE. The image embeddings here generated by pre-trained ResNet18.
(a) Male.
(b) Big Nose
(c) Oval Face.
(d) Smiling.
Figure 6: t-SNE plots of different attributes, with the same proportion (0.5) of faces with this attribute, sampled from CelebA, colored by the attribute. The feature extractor is pre-trained ResNet18.
1.0
0.5
Setting 1 Setting 2
Attack Accuraty
0.8
Attack Eror
0.4
0.8
0.3
0.4
0.2
Setting Setting
0.2
0.01
25
50
75
100
150
0.01
10
20
30
40
Number t-SNEs (per class)
Number of t-SNEs (per class)
(a) Multi-classifictton
(b) Regression.
Figure 7: Inference Performance by using different number of t-SNE plots.
Sotting 1 Setting 2
Setling
Setting
Sttck
Satting
Number Sample Points Traing SNE
Number Sample Points Trainng SNC
Namber Sample Points SNC
(a) 1000 sample points in each testing L-SNE plot.
(b) 750 sample points in each test-
(c) 500 sample points each test- ing L-SNE plot.
ing S-E plot.
Figure 8: culti-classification Inference Performance at different density settings on CelebA.
14

---

Under review as a conference paper at ICLR 2023
[-4
0-4
Sctting
Sctting
162
322
Numbar ef Sample Points is Trainig 5SN
Number el Sample Points is Traning SNE
Namber of Sample Poits in Traning SNE
(a) 1,000 sample points each lesting 1-SNE plot.
(b) 750 sample points in each test-
(c) 500 sample points each test-
ing 1-SNE plot.
ing I-SNE plot.
Figure 9: Regression inference performance at different density settings on CelebA
(a) 1000 sample points.
(b) 750 sample points.
(c) 500 sample points.
Figure 10: t-SNEs of face images (50% males) from CelebA with different density settings, colored by gender.
(a) Original.
(b) E-N-1.
(c) E-N-2.
(d) C-N-2%.
(e) C-N-5%.
Figure 11: I-SNEs of face images (50% males) from CelebA, colored by gender, perturbed by different defense methods.
Setting CelebA
0.35
Setting 1: UTKFace
0.30
Attack Error
Setting CelebA
0.25
Setting 2: UTKFace
0.15
0.10
0.05
10
Number of Additional t-SNEs of New Dataset
Figure 12: Regression inference performance on male proportion with mixed training t-SNE plots.
15

---

Under review as a conference paper at ICLR 2023
Fine-tuned CelebA Male classification task. (b) Fine-tuned UTKFaae Race classification Sample points are from CelebA, colored by gender, task, colored by the race. Sample points are from 50% of which are Smiling faces.
UTKFace, colored by the race, 50% of which are Male faces.
Figure 13: t-SNE plots generated from fine-tuned models. There 1000 sample points in each t-SNE plot.
16


Reviews:
[{"review_id": "f3uzBnzeVhA", "reviewer": "Reviewer_6zD4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper \"Property inference attacks against t-SNE plots\" suggests a way to infer some properties of the data set from an unlabeled t-SNE plot of that data set. For example, if the data contain images of males and females, then the attack would aim to infer the fraction of males from the shape of the unlabeled t-SNE plot. To do this, the attacker generates multiple t-SNE plots of the data with varying fraction of males, and then trains a classifier to predict the fraction of males from the t-SNE plot (treated as image). This classifier can then be applied to the t-SNE plot under attack. The paper shows good performance on some of the datasets.", "summary_of_the_review": "Interesting idea, but a rather trivial attack with very limited applicability. Borderline reject.", "strength_and_weaknesses": "In my opinion, the main strength of the paper is to suggest the framework of \"attacks against t-SNE plots\", which I believe is a novel concept and has not really been considered by anybody before. \n\nThe main weakness is that the suggested attack is rather trivial; also, it can only work in very limited situations where the shape of the t-SNE plot is strongly influenced by the fraction of males (or other similar property) and when the attacker has access to the training (\"shadow\") labeled data and can generate multiple t-SNE plots with varying property values.\n\nThe paper also has some presentation issues but those can be easily fixed. Overall I am giving it a borderline reject score.\n\nPRESENTATION ISSUES\n\n* My understanding is that the \"attack model\" treats t-SNE plots as images. If this is correct, then I think it should be made more explicit and stated more clearly, e.g. in the end of Section 4, and also elsewhere. If t-SNE plots are images, then what is their size in pixels? What size (in pixels) are the dots of the scatter plot? Are these images black-and-white? Grayscale? Etc.\n\n* The entire evaluation setup was not sufficiently clear to me from Section 4. E.g. what does 0.92 accuracy (beginning of Section 4.1) mean? This is the average over what exactly? Different random splits of the data? What is the true proportion of males in those splits? Were the splits generated such that the true proportion could be .3, .4, .5, .6, .7 with equal probability? Please clarify.\n\n* I think the paper *NEEDS* a figure (preferably Figure 1), that would show a t-SNE of the CelebA dataset under attack, and example t-SNE plots generated from shadow data with .3, .4, .5, .6, .7 fractions. This figure would illustrate the attack and would show that it is indeed possible to \"guess\" the fraction of males in the CelebA dataset correctly, because the shadow t-SNE plots all look sufficiently different from each other. I was not able to find such figure either in the main text, or in the appendix.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bxppDAk_nft", "reviewer": "Reviewer_sG4d", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work proposes property inference attacks using t-SNE plots of data. While most work in the literature focuses on white-box or black-box model access, this threat model lies somewhere in the middle- predicting properties of hold-out data using 2-dimensional representations graphed onto images. The approach consists of data cleaning and using plots directly as input to a meta-classifier, which is trained by the adversary for property prediction. The authors demonstrate the robustness of their attack across different plot settings and propose a simple adaptation of their attack that can bypass potential defenses like noise addition in the latent space.", "summary_of_the_review": "This work introduces a very new kind of threat model- one where the adversary tries to gain information about confidential data, given t-SNE plots generated with a model. The authors demonstrate how this leakage is very real, and fairly robust across various datasets, and configurations, and can be adapted to defenses with simple modifications. However, the practicality of the threat model is uncertain. The proposed attack cannot be used for model auditing (plausible deniability by the model trainer, as training data is different from plot-generation data), and it is unclear how the information leaked by the plots is of any use to the adversary. Even in the case that it is, the victim can always sample selectively from its test data, such that the leaked information is in no way relevant to the training distribution- like a 'honeypot' for the adversary. ", "strength_and_weaknesses": "## Strengths\n\n- This work introduces a new kind of threat model- information leaked by visualizations. It is common to use techniques like t-SNE or PCA to visualize high-dimensional data; often to demonstrate the effectiveness of an approach in learning useful representations. This work tackles the threat posed by the release of such plots- specifically, properties related to the data used to generate these plots.\n- Evaluations across different pairs of datasets are useful in better assessing the practicality of this risk; fine-tuning-based baselines are also useful in evaluating just how much data is actually needed by the adversary to launch successful attacks. This also holds true for evaluations with different class-wise plots and density settings.\n- The inclusion of adaptive attacks and evaluations with mixed datasets (Section 5.5) is also useful and further reinforces that the information leaked in this scenario is fairly robust across several configurations.\n\n## Weaknesses\n- The property inference attack here really focuses on the data used to generate the plot, not the actual training dataset/distribution itself. A victim is unlikely to use all of its data for t-SNE visualization (given its computational cost). A straightforward defense would thus be to sample the t-SNE plot data from the test set such that the \"property\" is different from what was used for training. In this case, even if an adversary can perfectly predict the property of the plot data, the information gleaned has utility no longer. Even if the adversary does learn this information, it is not very useful- definitely not for model auditing. The victim can always claim that the data used for plot generation was differently distributed from the data used for training (which is really what auditors would care about).\n\n- Section 3.2: \"...and get the image embeddings\". How is the adversary aware of which layer's features were used for the embeddings? How robust is the adversary to settings where this information is not known, i.e. the victim and adversary use different layer outputs for the plots?\n\n- Section 2: \"....most assume that the adversary has white-box access to the model.\" This is not true- there are several works [1, 2, 4] in the literature that propose and evaluate black-box attacks. Similarly, the claim on \"most of the works infer binary properties\" is not true either. Several works have studied various kinds of graph-related properties [3], as well as direct regression of these values [2].\n\n- As demonstrated in Tables 1 and 2, performance is near-random (and even worse) when the training and testing data are different. Although this can be alleviated by using some data from the target distribution (as demonstrated later in the paper), this shows how sensitive the attack is to know the exact data distribution. This might not be feasible for parties that might actually release these plots (like a company talking about their new ML model), and it is even more unclear if this attack would be practical in such a scenario. \n\n- Can plot rotation/scaling be a potential defense? It may not be perfect, but the adversary might suffer performance losses if trying to incorporate these invariances in its classifiers. Scaling the plot image, or rotating it, should not impact the utility of the t-SNE plot visualization and thus a victim could easily perform this post-processing to make things harder for the adversary.\n\n### References\n[1] Zhang, Wanrong, Shruti Tople, and Olga Ohrimenko. \"Leakage of Dataset Properties in {Multi-Party} Machine Learning.\" USENIX Security Symposium. 2021.\n\n[2] Suri, Anshuman, and David Evans. \"Formalizing and Estimating Distribution Inference Risks.\" Privacy Enhancing Technologies Symposium. 2022.\n\n[3] Zhang, Zhikun, et al. \"Inference Attacks Against Graph Neural Networks.\" USENIX Security Symposium. 2022.\n\n[4] Juarez, Marc, Samuel Yeom, and Matt Fredrikson. \"Black-Box Audits for Group Distribution Shifts.\" arXiv preprint arXiv:2209.03620 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "VFaxAb_-Sy", "reviewer": "Reviewer_5Mjd", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a property inference attack against tsne plots wherein an attacker can train a classifier on tsne plots from surrogate models in order to estimate sensitive characteristics about datasets for which other tsne plots were generated.", "summary_of_the_review": "I vote for acceptance because I think the idea is quite novel, and interesting. However, I would really like the authors to address my concerns/questions about the settings for the experiments. I would consider lowering my score if the questions are not sufficiently addressed.", "strength_and_weaknesses": "### Pros:\n1) The paper is very clearly presented.\n2) The idea is interesting, and as far as I know, novel.\n3) Experimentation is thorough.\n\n### Cons/Comments:\n1) The results are lukewarm in my eyes. Even in the comparatively easier setting of proportion classification, the proposed attack requires surrogate and target model pretrained on the same distribution, and several labeled tsne plots on which to train the attack model. And even then, the attack success on some datasets is fairly weak.\n2) Am I correct that the target model and the surrogate model are both pretrained on the same data for all experiments? If so, why? It seems like a much more realistic setting is that the target model is trained on a distribution about which the attacker has little information. \n3) Many of the experimental settings seem a bit contrived. The more realistic setting (transfer setting in 5.5), is a step in the right direction, but here the authors only limit the number of tsne plots available to the attacker. Another constraint I would have liked to have seen explored in this setting is tsne plots generated only with a small amount of data. In fact, throughout the paper, it seems like the authors assume that that data on which to generate tsne plots with labeled proportions, can be easily found. This seems like a very strong assumption.\n\nOverall, I'm a bit torn. The paper is a very interesting idea, and novel, but the results are a bit lukewarm and in somewhat contrived settings.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject