Title: OVERLEARNING REVEALS SENSITIVE ATTRIBUTES

Abstract: Overlearning" means that a model trained for a seemingly simple objective implicitly learns to recognize attributes and concepts that are (1) not part of the learning objective, and (2) sensitive from a privacy or bias perspective. For example, a binary gender classifier of facial images also learns to recognize races-even races that are not represented in the training data-and identities. We demonstrate overlearning in several vision and NLP models and analyze its harmful consequences. First, inference-time representations of an overlearned model reveal sensitive attributes of the input, breaking privacy protections such as model partitioning. Second, an overlearned model can be "re-purposed" for a different, privacy-violating task even in the absence of the original training data. We show that overlearning is intrinsic for some tasks and cannot be prevented by censoring unwanted attributes. Finally, we investigate where, when, and why overlearning happens during model training.

1. INTRODUCTION:

We demonstrate that representations learned by deep models when training for seemingly simple objectives reveal privacy-and bias-sensitive attributes that are not part of the specified objective. These unintentionally learned concepts are neither finer-, nor coarse-grained versions of the model's labels, nor statistically correlated with them. We call this phenomenon overlearning. For example, a binary classifier trained to determine the gender of a facial image also learns to recognize races (including races not represented in the training data) and even identities of individuals.

Overlearning has two distinct consequences. First, the model's inference-time representation of an input reveals the input's sensitive attributes. For example, a facial recognition model's representation of an image reveals if two specific individuals appear together in it. Overlearning thus breaks inference-time privacy protections based on model partitioning (Osia et al., 2018; Chi et al., 2018; Wang et al., 2018) . Second, we develop a new, transfer learning-based technique to "re-purpose" a model trained for benign task into a model for a different, privacy-violating task. This shows the inadequacy of privacy regulations that rely on explicit enumeration of learned attributes.

Overlearning is intrinsic for some tasks, i.e., it is not possible to prevent a model from learning sensitive attributes. We show that if these attributes are censored (Xie et al., 2017; Moyer et al., 2018) , the censored models either fail to learn their specified tasks, or still leak sensitive information. We develop a new de-censoring technique to extract information from censored representations. We also show that overlearned representations enable recognition of sensitive attributes that are not present in the training data. Such attributes cannot be censored using any known technique. This shows the the inadequacy of censoring as a privacy protection technology.

To analyze where and why overlearning happens, we empirically show how general features emerge in the lower layers of models trained for simple objectives and conjecture an explanation based on the complexity of the training data.

2. BACKGROUND:

We focus on supervised deep learning. Given an input x, a model M is trained to predict the target y using a discriminative approach. We represent the model M = C • E as a feature extractor (encoder) E and classifier C. The representation z = E(x) is passed to C to produce the prediction by modeling p(y|z) = C(z). Since E can have multiple layers of representation, we use E l (x) = z l to denote the model's internal representation at layer l; z is the representation at the last layer.

Model partitioning splits the model into a local, on-device part and a remote, cloud-based part to improve scalability of inference (Lane & Georgiev, 2015; Kang et al., 2017) and protect privacy of inputs into the model (Li et al., 2017; Osia et al., 2018; Chi et al., 2018; Wang et al., 2018) . For privacy, the local part of the model computes a representation, censors it as described below, and sends it to the cloud part, which computes the model's output.

Censoring representations. The goal is to encode input x into a representation z that does not reveal unwanted properties of x, yet is expressive enough to predict the task label y. Censoring has been used to achieve transform-invariant representations for computer vision, bias-free representations for fair machine learning, and privacy-preserving representations that hide sensitive attributes.

A straightforward censoring approach is based on adversarial training (Goodfellow et al., 2014) . It involves a mini-max game between a discriminator D trying to infer s from z during training and an encoder and classifier trying to infer the task label y while minimizing the discriminator's success (Edwards & Storkey, 2016; Iwasawa et al., 2016; Hamm, 2017; Xie et al., 2017; Li et al., 2018; Coavoux et al., 2018; Elazar & Goldberg, 2018) . The game is formulated as:

EQUATION

where γ balances the two log likelihood terms. The inner optimization maximizes log p(s|z = E(x)), i.e., the discriminator's prediction of the sensitive attribute s given a representation z. The outer optimization, on the other hand, trains the encoder and classifier to minimize the log likelihood of the discriminator predicting s and maximize that of predicting the task label y.

Another approach casts censoring as a single information-theoretical objective. The requirement that z not reveal s can be formalized as an independence constraint z ⊥ s, but independence is intractable to measure in practice, thus the requirement is relaxed to a constraint on the mutual information between z and s (Osia et al., 2018; Moyer et al., 2018) . The overall training objective of censoring s and predicting y from z is formulated as:

EQUATION

where I is mutual information and β, λ are the balancing coefficients; β = 0 in (Osia et al., 2018) . The first two terms I(z, y) -βI(z, x) is the objective of variational information bottleneck (Alemi et al., 2017) , the third term is the relaxed independence constraint of z and s.

Intuitively, this objective aims to maximize the information of y in z as per I(z, y), forget the information of x in z as per -βI(z, x), and remove the information of s in z as per -λI(z, s). This objective has an analytical lower bound (Moyer et al., 2018) :

EQUATION

where KL is Kullback-Leibler divergence and log p(x|z, s) is the reconstruction likelihood of x given z and s. The conditional distributions p(y|z) = C(z), q(z|x) = E(x) are modeled as in adversarial training and p(x|z, s) is modeled with a decoder R(z, s) = p(x|z, s).

All known censoring techniques require a "blacklist" of attributes to censor, and inputs with these attributes must be represented in the training data. Censoring for fairness is applied to the model's final layer to make its output independent of the sensitive attributes or satisfy a specific fairness constraint (Zemel et al., 2013; Louizos et al., 2016; Madras et al., 2018; Song et al., 2019) . In this paper, we use censoring not for fairness but to demonstrate that models cannot be prevented from learning to recognize sensitive attributes. To show this, we apply censoring to different layers, not just the output.

3. EXPLOITING OVERLEARNING:

We demonstrate two different ways to exploit overlearning in a trained model M . The inferencetime attack (Section 3.1) applies M to an input and uses M 's representation of that input to predict its sensitive attributes. The model-repurposing attack (Section 3.2) uses M to create another model that, when applied to an input, directly predicts its sensitive attributes. Update T on the batch of (z, zaux) with loss ||T (z) -zaux|| 2 2 7: Update Mattack on the batch of (T (z), s) with cross-entropy loss 8: end for 9: return prediction ŝ = Mattack(T (z ))

3.1. INFERRING SENSITIVE ATTRIBUTES FROM REPRESENTATION:

We measure the leakage of sensitive properties from the representations of overlearned models via the following attack. Suppose an adversary can observe the representation z of a trained model M on input x at inference time but cannot observe x directly. This scenario arises in practice when model evaluation is partitioned in order to protect privacy of inputs-see Section 2. The adversary wants to infer some property s of x that is not part of the task label y.

We assume that the adversary has an auxiliary set D aux of labeled (x, s) pairs and black-box oracle E to compute the corresponding E(x). The purpose of D aux is to help the adversary recognize the property of interest in the model's representations; it need not be drawn from the same dataset as x . The adversary uses supervised learning on the (E(x), s) pairs to train an attack model M attack . At inference time, the adversary predicts ŝ from the observed z as M attack (z ).

De-censoring. If the representation z is "censored" (see Section 2) to reduce the amount of information it reveals about s, the direct inference attack may not succeed. We develop a new, learning-based de-censoring approach (see Algorithm 1) to convert censored representations into a different form that leaks more information about the property of interest. The adversary trains M aux on D aux to predict s from x, then transforms z into the input features of M aux .

We treat de-censoring as an optimization problem with a feature space L 2 loss ||T (z)-z aux || 2 2 , where T is the transformer that the adversary wants to learn and z aux is the uncensored representation from M aux . Training with a feature-space loss has been proposed for synthesizing more natural images by matching them with real images (Dosovitskiy & Brox, 2016; Nguyen et al., 2016) . In our case, we match censored and uncensored representations. The adversary can then use T (z) as an uncensored approximation of z to train an inference model M attack and infer property s as M attack (T (z )).

3.2. RE-PURPOSING MODELS TO PREDICT SENSITIVE ATTRIBUTES:

To re-purpose a model-for example, to convert a model trained for a benign task into a model that predicts a sensitive attribute-we can use features z l in any layer of M as the feature extractor and connect a new classifier C transfer to E l . The transferred model M transfer = C transfer • E l is fine-tuned on another, small dataset D transfer , which in itself is not sufficient to train an accurate model for the new task. Utilizing features learned by M on the original D, M transfer can achieve better results than models trained from scratch on D transfer .

Feasibility of model re-purposing complicates the application of policies and regulations such as GDPR (EU, 2018) . GDPR requires data processors to disclose every purpose of data collection and obtain consent from the users whose data was collected. We show that, given a trained model, it is not possible to determine-nor, consequently, disclose or obtain user consent for-what the model UTKFace is a set of over 23,000 face images labeled with age, gender, and race (UTKFace; Zhang et al., 2017) . We rescaled them into 50×50 RGB pixels. The task is to predict gender; the sensitive attribute is race.

FaceScrub is a set of face images labeled with gender (FaceScrub). Some URLs are expired, but we were able to download 74,000 images for 500 individuals and rescale them into 50×50 RGB pixels.

The task is to predict gender; the sensitive attribute is identity.

Places365 is a set of 1.8 million images labeled with 365 fine-grained scene categories. We use a subset of 73,000 images, 200 per category. The task is to predict whether the scene is indoor or outdoor; the sensitive attribute is the fine-grained scene label.

Twitter is a set of tweets from the PAN16 dataset (Rangel et al., 2016) labeled with user information. We removed tweets with fewer than 20 tokens and users with fewer than 50 tweets, yielding a dataset of over 46,000 tweets from 151 users with an over 80,000-word vocabulary. The task is to predict the age of the user given a tweet; the sensitive attribute is the author's identity.

Yelp is a set of Yelp reviews labeled with user identities (Yelp Open Dataset). We removed users with fewer than 1,000 reviews and reviews with more than 200 tokens, yielding a dataset of over 39,000 reviews from 137 users with an over 69,000-word vocabulary. The task is to predict the review score between 1 to 5; the sensitive attribute is the author's identity.

PIPA is a set of over 60,000 photos of 2,000 individuals gathered from public Flickr photo albums (Piper project page; Zhang et al., 2015) . Each image can include one or more individuals. We cropped their head regions using the bounding boxes in the image annotations. The task is to predict the identity given the head region; the sensitive attribute is whether two head regions are from the same photo.

Models. For Health, we use a two-layer fully connected (FC) neural network with 128 and 32 hidden units, respectively, following (Xie et al., 2017; Moyer et al., 2018) . For UTKFace and FaceScrub, we use a LeNet (LeCun et al., 1998) variant: three 3×3 convolutional and 2×2 max-pooling layers with 16, 32, and 64 filters, followed by two FC layers with 128 and 64 hidden units. For Twitter and Yelp, we use text CNN (Kim, 2014) . For Places365 and PIPA, we use AlexNet (Krizhevsky et al., 2012) with convolutional layers pre-trained on ImageNet (Deng et al., 2009) and further add a 3×3 convolutional layer with 128 filters and 2×2 max-pooling followed by two FC layers with 128 and 64 hidden units, respectively. Setup. We use 80% of the data for training the target models and 20% for evaluation. The size of the adversary's auxiliary dataset is 50% of the training data. Success of the inference attack is measured on the final FC layer's representation of test data. The baseline is inference from the uncensored representation. We also measure the success of inference against representations censored with γ = 1.0 for adversarial training and β = 0.01, λ = 0.0001 for information-theoretical censoring, following (Xie et al., 2017; Moyer et al., 2018) .

For censoring with adversarial training, we simulate the adversary with a two-layer FC neural network with 256 and 128 hidden units. The number of epochs is 50 for censoring with adversarial training, 30 for the other models. We use the Adam optimizer with the learning rate of 0.001 and batch size of 128. For information-theoretical censoring, the model is based on VAE (Kingma & Welling, 2013; Moyer et al., 2018) . The encoder q(z|x) has the same architecture as the CNN models with all convolutional layers. On top of that, the encoder outputs a mean vector and a standard deviation vector to model the random variable z with the re-parameterization trick. The decoder p(x|z) has three de-convolution layers with up-sampling to map z back to the same shape as the input x.

For our inference model, we use the same architecture as the censoring adversary. For the PIPA inference model, which takes two representations of faces and outputs a binary prediction of whether these faces appear in the same photo, we use two FC layers followed by a bilinear model:

p(s|z 1 , z 2 ) = σ(h(z 1 )W h(z 2 ) )

, where z 1 , z 2 are the two input representations, h is the two FC layers, and σ is the sigmoid function. We train the inference model for 50 epochs with the Adam optimizer, learning rate of 0.001, and batch size of 128.

Results. Table 2 reports the results. When representations are not censored, accuracy of inference from the last-layer representations is much higher than random guessing for all tasks, which means models overlearn even in the higher, task-specific layers. When representations are censored with adversarial training, accuracy drops for both the main and inference tasks. Accuracy of inference is much higher than in (Xie et al., 2017) . The latter uses logistic regression, which is weaker than the training-time censoring-adversary network, whereas we use the same architecture for both the training-time and post-hoc adversaries. Information-theoretical censoring reduces accuracy of inference, but also damages main-task accuracy more than adversarial training for almost all models.

Overlearning can cause a model to recognize even the sensitive attributes that are not represented in the training dataset. Such attributes cannot be censored using any known technique. We trained a UTKFace gender classifier on datasets where all faces are of the same race. We then applied this model to test images with four races (White, Black, Asian, Indian) and attempted to infer the race attribute from the model's representations. Inference accuracy is 61.95%, 61.99%, 60.85% and 60.81% for models trained only on, respectively, White, Black, Asian, and Indian images-almost as good as the 62.18% baseline and much higher than random guessing (42.52%).

Effect of censoring strength. Fig. 2 shows that stronger censoring does not help. On FaceScrub and Twitter with adversarial training, increasing γ damages the model's accuracy on the main task, while For all models with information-theoretical censoring, increasing β reduces the accuracy of inference but can lead to the model not converging on its main task. Increasing λ results in the model not converging on the main task, without affecting the accuracy of inference, on Health, UTKFace and FaceScrub. This seems to contradict the censoring objective, but the reconstruction loss in Equation 2 dominates the other loss terms, which leads to poor divergence between conditional q(z|x) and q(z), i.e., information about x is still retained in z.

De-censoring. As described in Section 3.1, we developed a new technique to transform censored representations to make inference easier. We first train an auxiliary model on D aux to predict the sensitive attribute from representations, using the same architecture as in the baseline models. The resulting uncensored representations from the last convolutional layer are the target for the decensoring transformations. We use a single-layer fully connected neural network as the transformer and set the number of hidden units to the dimension of the uncensored representation. The inference model operates on top of the transformer network, with the same hyper-parameters as before.

Table 3 shows that de-censoring significantly boosts the accuracy of inference from representations censored with adversarial training. The boost is smaller against information-theoretical censoring because its objective not only censors z with I(z, s), but also forgets x with I(x, z). On the Health task, there is not much difference since the baseline attack is already similar to the attack on censored representations, leaving little room for improvement. Table 5 : The effect of censoring on adversarial re-purposing for FaceScrub with γ = 0.5, 0.75, 1.0. δ A is the difference in the original-task accuracy (second column) between uncensored and censored models; δ B is the difference in the accuracy of inferring the sensitive attribute (columns 3 to 7) between the models re-purposed from different layers and the model trained from scratch. Negative values mean reduced accuracy. Heatmaps on the right are linear CKA similarities between censored and uncensored representations. Numbers 0 through 4 represent layers conv1, conv2, conv3, fc4, and fc5. For each model censored at layer i (x-axis), we measure similarity between the censored and uncensored models at layer j (y-axis).

Censored on δB when transferred from γ = 0.5 δA conv1 conv2 conv3 fc4 fc5 In summary, these results demonstrate that information about sensitive attributes unintentionally captured by the overlearned representations cannot be suppressed by censoring.

To demonstrate that overlearned representations can be picked up by a small set of unseen data to create a model for predicting sensitive attributes, we re-purpose uncensored baseline models from Section 4.2 by fine-tuning them on a small (2 -10% of D) set D transfer and compare with the models trained from scratch on D transfer . We fine-tune all models for 50 epochs with batch size of 32; the other hyper-parameters are as in Section 4.2. For all CNN models, we use the trained convolutional layers as the feature extractor and randomly initialize the other layers. Table 4 shows that the re-purposed models always outperform those trained from scratch. FaceScrub and Twitter exhibit the biggest gain.

Effect of censoring. Previous work only censored the highest layer of the models. Model repurposing can use any layer of the model for transfer learning. Therefore, to prevent re-purposing, inner layers must be censored, too. We perform the first study of inner-layers censoring and measure its effect on both the original and re-purposed tasks. We use FaceScrub for this experiment and apply adversarial training to every layer with different strengths (γ = 0.5, 0.75, 1.0).

Table 5 summarizes the results. Censoring lower layers (conv1 to conv3) blocks adversarial repurposing, at the cost of reducing the model's accuracy on its original task. Hyper-parameters must be tuned carefully, e.g. when γ = 1, there is a huge drop in the original-task accuracy.

To further investigate how censoring in one layer affects the representations learned across all layers, we measure per-layer similarity between censored and uncensored models using CKA, linear centered kernel alignment (Kornblith et al., 2019) -see Table 5 . When censoring is applied to a specific layer, similarity for that layer is the smallest (values on the diagonal). When censoring lower layers with moderate strength (γ = 0.5 or 0.75), similarity between higher layers is still strong; when censoring higher layers, similarity between lower layers is strong. Therefore, censoring can block adversarial re-purposing from a specific layer, but the adversary can still re-purpose representations in the other layer(s) to obtain an accurate model for predicting sensitive attributes.

4.4. WHEN, WHERE, AND WHY OVERLEARNING HAPPENS:

To investigate when (during training) and where (in which layer) the models overlearn, we use linear CKA similarity (Kornblith et al., 2019) to compare the representations at different epochs of training between models trained for the original task (A) and models trained to predict a sensitive attribute (B). We use UTKFace and FaceScrub for these experiments.

Fig. 3 shows that lower layers of models A and B learn very similar features. This was observed in (Kornblith et al., 2019) for CIFAR-10 and CIFAR-100 models, but those tasks are closely related. In our case, the tasks are entirely different and B reveals the sensitive attribute while A does not.

The similar low-level features are learned very early during training. There is little similarity between the low-level features of A and high-level features of B (and vice versa), matching intuition. Interestingly, on FaceScrub even the high-level features are similar between A and B.

We conjecture that one of the reasons for overlearning is structural complexity of the data. Previous work theoretically showed that over-parameterized neural networks favor simple solutions on structured data when optimized with SGD, where structure is quantified as the number of distributions (e.g., images from different identities) within each class in the target task (Li & Liang, 2018) , i.e., the fewer distributions, the more structured the data. For data generated from more complicated distributions, networks learn more complex solutions, leading to the emergence of features that are much more general than the learning objective and, consequently, overlearning.

Fig. 4 shows that the representations of a gender classifier trained on the faces from 50 individuals are closer to the random initialization than the representations trained on the faces from 500 individuals (the hyper-parameters and the total number of training examples are the same in both cases). More complex training data thus results in more complex representations for the same objective. 

5. RELATED WORK:

Prior work studied transferability of representations only between closely related tasks. Transferability of features between ImageNet models decreases as the distance between the base and target tasks grows (Yosinski et al., 2014) , and performance of tasks is correlated to their distance from the source task (Azizpour et al., 2015) . CNN models trained to distinguish coarse classes also distinguish their subsets (Huh et al., 2016) . By contrast, we show that models trained for simple tasks implicitly learn privacy-sensitive concepts unrelated to the labels of the original task. Other than an anecdotal mention in the acknowledgments paragraph of (Kim et al., 2017) that logit-layer activations leak non-label concepts, this phenomenon has never been described in the research literature.

Gradient updates revealed by participants in distributed learning leak information about individual training batches that is uncorrelated with the learning objective (Melis et al., 2019) . We show that overlearning is a generic problem in (fully trained) models, helping explain these observations.

There is a large body of research on learning disentangled representations (Bengio et al., 2013; Locatello et al., 2019) . The goal is to separate the underlying explanatory factors in the representation so that it contains all information about the input in an interpretable structure. State-of-the-art approaches use variational autoencoders (Kingma & Welling, 2013) and their variants to learn disentangled representations in an unsupervised fashion (Higgins et al., 2017; Kumar et al., 2018; Kim & Mnih, 2018; Chen et al., 2018) . By contrast, overlearning means that representations learned during supervised training for one task implicitly and automatically enable another task-without disentangling the representation on purpose during training.

Work on censoring representations aims to suppress sensitive demographic attributes and identities in the model's output for fairness and privacy. Techniques include adversarial training (Edwards & Storkey, 2016) , which has been applied to census and health records (Xie et al., 2017) , text (Li et al., 2018; Coavoux et al., 2018; Elazar & Goldberg, 2018) , images (Hamm, 2017) and sensor data of wearables (Iwasawa et al., 2016) . An alternative approach is to minimize mutual information between the representation and the sensitive attribute (Moyer et al., 2018; Osia et al., 2018) .

Neither approach can prevent overlearning, except at the cost of destroying the model's accuracy. Furthermore, these techniques cannot censor attributes that are not represented in the training data. We show that overlearned models recognize such attributes, too.

6. CONCLUSIONS:

We demonstrated that models trained for seemingly simple tasks implicitly learn concepts that are not represented in the objective function. In particular, they learn to recognize sensitive attributes, such as race and identity, that are statistically orthogonal to the objective. The failure of censoring to suppress these attributes and the similarity of learned representations across uncorrelated tasks suggest that overlearning may be intrinsic, i.e., learning for some objectives may not be possible without recognizing generic low-level features that enable other tasks, including inference of sensi-tive attributes. For example, there may not exist a set of features that enables a model to accurately determine the gender of a face but not its race or identity. This is a challenge for regulations such as GDPR that aim to control the purposes and uses of machine learning technologies. To protect privacy and ensure certain forms of fairness, users and regulators may desire that models not learn some features and attributes. If overlearning is intrinsic, it may not be technically possible to enumerate, let alone control, what models are learning. Therefore, regulators should focus on ensuring that models are applied in a way that respects privacy and fairness, while acknowledging that they may still recognize and use sensitive attributes.

:



Figures and Tables:

Figure 1: Pseudo-code for inference from representation and adversarial re-purposing

Figure 2: Reduction in accuracy due to censoring. Blue lines are the main task, red lines are the inference of sensitive attributes. First row is adversarial training with different γ values; second and third row is information-theoretical censoring with different β and λ values respectively.

Figure 3: Pairwise similarities of layer representations between models for the original task (A) and for predicting a sensitive attribute (B). Numbers 0 through 4 denote layers conv1, conv2, conv3, fc4 and fc5.

Figure 4: Similarity of layer representations of a partially trained gender classifier to a randomly initialized model before training. Models are trained on FaceScrub using 50 IDs (blue line) and 500 IDs (red line).

Summary of datasets and tasks. Cramer's V captures statistical correlation between y and s (0 indicates no correlation and 1 indicates perfectly correlated). Learning per se thus cannot be a regulated "purpose" of data collection. Regulators must be aware that even if the original training data has been erased, a model can be re-purposed for a different objective, possibly not envisioned at the time of original data collection. We discuss this further in Section 6.

Table content: <table><tr><td>Dataset</td><td colspan="5">Health UTKFace FaceScrub Places365 Twitter</td><td>Yelp</td><td>PIPA</td></tr><tr><td>Target y</td><td>CCI</td><td>gender</td><td>gender</td><td>in/outdoor</td><td>age</td><td>review score</td><td>facial IDs</td></tr><tr><td>Attribute s</td><td>age</td><td>race</td><td colspan="3">facial IDs scene type author</td><td>author</td><td>IDs together</td></tr><tr><td>Cramer's V</td><td>0.149</td><td>0.035</td><td>0.044</td><td>0.052</td><td>0.134</td><td>0.033</td><td>n/a</td></tr><tr><td colspan="3">has learned. 4 EXPERIMENTAL RESULTS</td><td/><td/><td/><td/><td/></tr></table>

Accuracy of inference from representations (last FC layer). RAND is random guessing based on majority class labels; BASE is inference from the uncensored representation; ADV from the representation censored with adversarial training; IT from the information-theoretically censored representation.

Table content: <table><tr><td/><td colspan="2">Acc of predicting target y</td><td/><td colspan="3">Acc of inferring sensitive attribute s</td></tr><tr><td>Dataset</td><td colspan="2">RAND BASE ADV</td><td colspan="3">IT RAND BASE ADV</td><td>IT</td></tr><tr><td>Health</td><td>66.31</td><td colspan="2">84.33 80.16 82.63</td><td>16.00</td><td>32.52 32.00</td><td>26.60</td></tr><tr><td>UTKFace</td><td>52.27</td><td colspan="2">90.38 90.15 88.15</td><td>42.52</td><td>62.18 53.28</td><td>53.30</td></tr><tr><td>FaceScrub</td><td>53.53</td><td colspan="2">98.77 97.90 97.66</td><td>1.42</td><td>33.65 30.23</td><td>10.61</td></tr><tr><td>Places365</td><td>56.16</td><td colspan="2">91.41 90.84 89.82</td><td>1.37</td><td>31.03 12.56</td><td>2.29</td></tr><tr><td>Twitter</td><td>45.17</td><td>76.22 57.97</td><td>n/a</td><td>6.93</td><td>38.46 34.27</td><td>n/a</td></tr><tr><td>Yelp</td><td>42.56</td><td>57.81 56.79</td><td>n/a</td><td>15.88</td><td>33.09 27.32</td><td>n/a</td></tr><tr><td>PIPA</td><td>7.67</td><td colspan="2">77.34 52.02 29.64</td><td>68.50</td><td>87.95 69.96</td><td>82.02</td></tr><tr><td colspan="6">4.2 INFERRING SENSITIVE ATTRIBUTES FROM REPRESENTATIONS</td><td/></tr></table>

Improving inference accuracy with de-censoring. δ is the increase from Table2.

Table content: <table><tr><td>Dataset</td><td>Health</td><td>UTKFace</td><td>FaceScrub</td><td>Places365</td><td>Twitter</td><td>Yelp</td></tr><tr><td colspan="7">ADV +δ 32.55 +0.55 59.38 +6.10 40.37 +12.24 19.71 +7.15 36.55 +2.22 31.36 +4.04</td></tr><tr><td colspan="4">IT +δ 27.05 +0.45 54.31 +1.01 16.40 +5.79</td><td>3.10 +0.81</td><td>n/a</td><td>n/a</td></tr></table>

Adversarial re-purposing. The values are differences between the accuracy of predicting sensitive attributes using a re-purposed model vs. a model trained from scratch.

Table content: <table><tr><td colspan="8">|Dtransfer|/|D| Health UTKFace FaceScrub Places365 Twitter Yelp PIPA</td></tr><tr><td>0.02</td><td>-0.57</td><td>4.72</td><td>7.01</td><td>4.42</td><td colspan="2">12.99 5.57</td><td>1.33</td></tr><tr><td>0.04</td><td>0.22</td><td>2.70</td><td>15.07</td><td>2.14</td><td colspan="2">10.87 3.60</td><td>2.41</td></tr><tr><td>0.06</td><td>-1.21</td><td>2.83</td><td>7.02</td><td>2.06</td><td colspan="2">10.51 8.45</td><td>6.50</td></tr><tr><td>0.08</td><td>-0.99</td><td>0.25</td><td>11.80</td><td>3.39</td><td colspan="2">9.57 0.33</td><td>4.93</td></tr><tr><td>0.10</td><td>0.35</td><td>2.24</td><td>9.43</td><td>2.86</td><td>7.30</td><td>2.1</td><td>5.89</td></tr></table>

Peer review (iclr2020):
This paper introduces the problem of overlearning, which can be thought of as unintended transfer learning from a (victim) source model to a target task that the source model’s creator had not intended its model to be used for. The paper raises good points about privacy legislation limitations due to the fact that overlearning makes it impossible to foresee future uses of a given dataset. Background on censoring is well developed, and helps position the submission. However, the relation to transfer learning is not sufficiently outlined in the introduction. In some ways, overlearning is a form of unintended transfer learning. In particular, the connection appears explicit in the case of model repurposing (Section 3.2). Editing the introduction to tease apart this relationship to transfer learning would help readers forge an intuition for what the paper considers. While the de-censoring algorithm is intuitive, it is not clear what assumptions are being made because of ambiguous notation in Algorithm 1. How does Algorithm 1 train E_aux and C_aux? Does that step assume the adversary has knowledge of the algorithm use to train E and C? Descriptions of the experimental setup from Section 4.2 seem to indicate that this is the case. This is not necessarily an issue if the proposed attack is demonstrating a limitation of learning rather than a practical attack. Experiments overall show that censoring does not mitigate overlearning in a way that is robust to de-censoring. One aspect of the experiments that is unclear is the auxiliary dataset: what is the auxiliary dataset used by the adversary for each of the datasets considered in experiments? Question: how does simultaneous censoring of all layers affect overlearning and repurposing? Question: what is the connection between learning more complex representations and overlearning? Is the intuition that if the representation is more complex, it is more likely to contain features useful to identify the sensitive attribute? The paper is overall well-written. Some parts of the paper omit important details (perhaps due to space constraints?), but an editorial pass should address most of these. Detailed feedback: 1 / Explaining what model partitioning means in this context would help make the introduction more self-contained. 1 / Is overlearning specific to attributes that raise fairness issues? Or is this phenomenon more general? The “Censoring representations” paragraph on page 2 seems to indicate that the phenomenon is more general. 5/ Is accuracy the best metric to report (e.g., for race attribute prediction) given that the random guessing figure suggests the data is not balanced across attribute values? 6/ Why is the effect not monotonic in Table 4? Were multiple runs averaged?