Overlearning Reveals Sensitive Attributes
Authors: Congzheng Song
Vitaly Shmatikov
Venue:  (2020)
Keywords: privacy
censoring representation
transfer learning

Abstract: ``"Overlearning'' means that a model trained for a seemingly simple
objective implicitly learns to recognize attributes and concepts that are
(1) not part of the learning objective, and (2) sensitive from a privacy
or bias perspective.  For example, a binary gender classifier of facial
images also learns to recognize races, even races that are
not represented in the training data, and identities.

We demonstrate overlearning in several vision and NLP models and analyze
its harmful consequences.  First, inference-time representations of an
overlearned model reveal sensitive attributes of the input, breaking
privacy protections such as model partitioning.  Second, an overlearned
model can be "`re-purposed'' for a different, privacy-violating task
even in the absence of the original training data.

We show that overlearning is intrinsic for some tasks and cannot be
prevented by censoring unwanted attributes.  Finally, we investigate
where, when, and why overlearning happens during model training.

Published as a conference paper at ICLR 2020
OVERLEARNING REVEALS SENSITIVE ATTRIBUTES
Congzheng Song
Vitaly Shmatikov Cornell Tech shmat @cs wow.ccorell..d....
Cornell University cs2296@cornel1.com
ABSTRACT
"Overlearning" means that model trained for seemingly simple objective im- plicitly learns to recognize attributes and concepts that are (1) not part of the learn- ing objective, and (2) sensitive from privacy bias perspective. For example, a binary gender classifier of facial images also learns to recognize races- even races that are not represented in the training data cata-and identities.
We demonstrate overlearning several vision and NLP models and analyze its harmful consequences. First, inference-time representations of an overlearned model reveal sensitive attributes of the input, breaking privacy protections such model partitioning. Second, overlearned model can be "re-purposed" for different, privacy-violating task even the absence the original training data.
We show that overlearning is intrinsic for some tasks and cannot be prevented by censoring unwanted attributes. Finally, we investigate where, when, and why overlearning happens during model training.
INTRODUCTION
We demonstrate that representations learned by deep models when training for seemingly simple objectives reveal privacy- and bias-sensitive attributes that are part of the specified objective. These unintentionally learned concepts are neither finer-, nor coarse-graing -raunned versions of the model's labels, nor statistically correlated with them. We call this phenomenon overlearning. For example, binary classifier trained to determine the gender of a facial image also learns to recognize races (including races not represented in the training data) and even identities of individuals.
Overlearning has two distinet consequences. First, the model's inference-ting -imee representation of an input reveals the input's sensitive attributes. For example, facial recognition model's representa- tion of image reveals if two specific individuals appear together in it. Overlearning thus breaks inference- diference-time privacy protections based on model partitioning (Osia et al., 2018; Chi et al., 2018; Wang et al., 2018). Second, develop new, transfer learning based technique "re-purpose" model trained for benign task into a model for a different, privacy-violating task. This shows the inadequacy of privacy regulations that rely explicit enumeration of learned attributes.
Overlearning intrinsic for some tasks, iee., it is not possible to prevent a model from learning sensitive attributes. We show that if these attributes are censored (Xie et al., 2017; Moyer et al., 2018), the censored models either fail to learn their specified tasks, or still leak sensitive information. We develop a new de-censoring technique extract information from censored representations. We also show that overlearned representations enable recognition of sensitive attributes that are not present in the training data. Such attributes cannot be censored using any known technique. This shows the the inadequacy of censoring as privacy protection technology.
To analyze where and why overlearning happens, we empirically show how general features emerge in the lower layers of models trained for simple objectives and conjecture an explanation based on the complexity of the training data.
BACKGROUND
We focus on supervised deep learning. Given an input x, model M trained to predict the target y using discrininative approach. We represent the model M C E. feature extractor

---

Published as a conference paper at ICLR 2020
(encoder) E and classifier C. The representation E(x) is passed to C to produce the prediction by modeling p(y) z) = C(z). Since E can have multiple layers of representation, we use Ei(a) = 2 to denote the model's s internal representation at layer L; is the representation at the last layer.
Model partitioning splits the model into a local, on-device part and a remote, cloud-based part to improve scalability of inference (Lane & Georgiev, 2015; Kang et al., 2017) and protect privacy of inputs into the model (Li et al., 2017; Osia et al., 2018; Chi et al., 2018; Wang et al., 2018). For privacy, the local part of the model computes a representation, censors described below, and sends it to the cloud part, which computes the model's output.
Censoring representations. The goal is encode input into representation that does not reveal unwanted properties of z, yet is expressive enough to predict the task label y. Censoring has been used to achieve for fair machine learning, and privacy-preserving representations that hide sensitive attributes.
representations for computer vision, bias-free representations
straightforward censoring approach is based on adversarial training (Goodfellow et al., 2014). It involves a mini-ma Maxi--mxxaa game between discriminator D trying to infer s from 2 during training and an encoder and classifier trying to infer the task label y while minimizing the discriminator's success (Edwards & Storkey, 201 6; Iwasawa et al., 2016; Hamm, 2017; Xie et al., 2017; Li et al., 2018; Coavoux et al., 2018; Elazar & Goldberg, 2018). The game is formulated as:
min max E log p(s|2 E(x)) - log p(y|= E(x))] E,C' D
(1)
where balances the two log likelihood terms. The inner optimization maximizes p(sp( = E(x)), i.e. the discriminator's prediction of the sensitive attribute given representation z. The outer optimization, on the other hand, trains the encoder and classifier to minimize the log likelihood of the discriminator predicting and maximize that of predicting the task label y.
Another approach casts censoring as single that 2 not reveal s can be formalized as an independence constraint 1 s. but independence is intractable to measure in practice, thus the requirement is relaxed to a constraint on the mutual information between ø and s (Osia et al., 2018; Moyer et al., 2018). The overall training objective of censoring and predicting y from is formulated as:
-hheorttin-theoetcaallll objective. The requirement
max (2, y) BI(2, AI(2,s)
(2)
where I is mutual information and B, À are the balancing coefficients; 3 0 in (Osia et al., 2018). The first two terms ((z, (1, x) is the objective of variational information bottleneck (Alemi al., 2017), the third term is the relaxed independence constraint of z and
Intuitively, this objective aims to maximize the information of y in = as per T(z, v), forget the information of æ in as per per--1 (2, and remove the information of in z as per (2,,,s)) This objective has an analytical lower bound (Moyer et al., 2018):
(3) where KL is Kullback-Leible divergence and log is the reconstruction likelihood of given and s. The conditional distributions p(y[z) = C(z), ((((x)) = E(x) are modeled as in adversarial training and p(x|2, s) is modeled with a decoder R(2,s) p(z|2,s).
Ez, Bog
(9 (g<))())))))
(ogg(((()))
All known censoring techniques require a "blacklist" of attributes to censor, and inputs with these attributes must be represented in the training data. Censoring for fairness is applied to the model's final layer to make its output independent of the sensitive attributes or satisfy a specific fairness constraint (Zemel al., 2013; Louizos et al., 2016; Madras et al., 2018; Song et al., 2019). In this paper, we use censoring not for fairness but to demonstrate that models cannot be prevented from learning to recognize sensitive attributes. To show this, we apply censoring different layers, not just the output.
3 EXPLOITING OVERLEARNING
We demonstrate two different ways to exploit overlearning in a trained model M. The inference- time attack (Section 3.1) applies M to an input and uses M representation of that input to predict its sensitive attributes. The model-repurposing attack (Section 3.2) uses M to create another model that, when applied to input, directly predicts its sensitive attributes.
2

---

Published as a conference paper at ICLR 2020
Inferring from representation:
Adversarial ce-puroosin::
1: Input: Adversary's auxiliary dataset Dan. black-box oracle E, observed 2²
1: Input: Model M for the original task, transfer
dataset Dmmestes for the task
Damnk {(E(xr, (2, )
Build Miranser = Cuanster Ei on layer
Train attack model Manck on Dattock
3: Fine-tune Miranter on Diaanfer
4: return prediction & = Marack
4: return transfer model Mannster
Figure 1: Pscudo-code for inference from representation and adversarial re-purposing
Algorithm De-censoring representations
1: Input: Auxiliary dataset Daux, black-box oracle E, observed representation 2'
2: Train auxiliary model Max = Eam Cau on Dam
Initialize transform model T, inference attack model Matuck
4: for each training iteration do
5: Sample a batch of data (Ir s) from Dax and compute = E(x), Emx(2)
6: Update T on the batch of (2, Zaux) with loss [[T 11
Update Manuck on batch (T))), s) with cross coos-yy loss
8: end for
9: return prediction å Manuck (T(="))
3.1 INFERRING SENSITIVE ATTRIBUTES FROM REPRESENTATION
We measure the leakage of sensitive properties from the representations of overlearned models via the following attack. Suppose an adversary can observe the representation 2 of a trained model M on input x* inference time but cannot observe x' directly. This scenario arises in practice when model evaluation is partitioned in order to protect privacy of inputs-see Section 2. The adversary wants to infer some property s of x= that is not part of the task label y.
We assume that the adversary has an auxiliary set Daux of labeled (x, 8) pairs and black-box oracle E to compute the corresponding F((z). The purpose of Daux is to help the adversary recognize the property of interest in the model's representations; it need not be drawn from the same dataset as x². The adversary uses supervised learning on the (E(x), s) pairs train an attack model Mansck- At inference time, the adversary predicts from the observed 2 as Manack (2*).
De-censoring. If the representation 2 "censored" (see Section to reduce the amount of informa- tion it reveals about s, the direct inference attack may not succeed. We develop new, learning-basee de-censoring approach (see Algorithm 1) to convert censored representations into different form that leaks more information about the property of interest. The adversary trains M.xx on Danx to predict s from æ, then transforms z into the input features of Maux-
We treat de-censoring T' the transformer that the adversary wants to learn and zaux is the uncensored representation from Max. Training with feature-spare space loss has been proposed for synthesizing more natural images by matching them with real images (Dosovitskiy & Brox, 2016; Nguyen et al., 2016). In our case, we match censored and uncensored representations. The adversary can then use T(2) as uncensored approximation of to train an inference model Manaek and infer property s as Manaek (T(=*)).
optimization problem with feature space L2 loss (2)) where
3.2 RE-PURPOSING MODELS TO PREDICT SENSITIVE ATTRIBUTES
To re-purpose a model for example, to convert a model trained for benign task into a model that predicts sensitive attribute- we can use features ž in any layer of M as the feature extractor and connect a new classifier Ciraster to Ez. The transferred model Miransfer = Ciranster F, fine-tuned on another, small dataset Diransfors which in itself is not sufficient train accurate model for the new task. Utilizing features learned by M on the original D, Marosser can achieve better results than models trained from scratch on D Dransfer-
Feasibility of model re- purposing complicates the application of policies and regulations such as GDPR (EU, 2018). GDPR requires data processors disclose every purpose of data collection and obtain consent from the users whose data was collected. We show that, given a trained model, is not possible to determine nor, consequently, disclose obtain user consent what the model
3

---

Published as a conference paper at ICLR 2020
Table 1:
Summary of datasets tasks. Cramer's V captures statistical correlation between y and (0
indicates no correlation and I indicates perfectly correlated).
Dataset
Health
UTKFace
FaceSerub
Places365
Twitter
Yelp
PIPA
Target
CCI age
gender
gender facial IDs
in/outdoor type 0.052
age author 0.134
review score
facial IDs IDs together n/a
Attribute
author 0.033
Cramer's
0.149
0.035
0.044
has Iearned. Learning per se thus cannot be a regulated "purpose" data collection. Regulators must be aware that even if the original training data has been erased, model can be re-puposed for a different objective, possibly not envisioned at the time of original data collection. We discuss this further in Section 6.
EXPERIMENTAL RESULTS
4.1 DATASETS, TASKS, AND MODELS
Health is the Heritage Health dataset (Heritage Health Prize) with medical records of over 55,000 patients, binarized into 112 features with age information removed. The task predict if Charlson Index (an estimate of patient mortality) is greater than zero; the sensitive attribute is age (binned into 9 ranges).
UTKFace is set of over 23,000 face images labeled with age, gender, and race (UTKFace; Zhang al., 2017). We rescaled them into 50x50 RGB pixels. The task is to predict gender; the sensitive attribute is race.
FaceScrub a set of face images labeled with gender (FaceScrub). Some URLs are expired, but we were able to download 74,000 images for 500 individuals and rescale them into 50x RGB pixels. The task is to predict gender; the sensitive attribute is identity.
Places365 is set of 1.8 million images labeled with 365 fine-grained scene categories. We use subset of 73,000 images, 200 per category. The task is to predict whether the scene is indoor or outdoor; the sensitive attribute is the fine-grained scene label.
Twitter a set of tweets from the PANI6 dataset (Rangel et al., 2016) labeled with user information. We removed tweets with fewer than 20 tokens and users with fewer than 50 tweets, yielding dataset of over 46,000 tweets from 151 users with an over 80,000-word vocabulary. The task is to predict the age of the user given tweet; the sensitive attribute is the author's identity.
Yelp is a set of Yelp reviews labeled with user identities (Yelp Open Dataset). We removed users with fewer than 1, 1,000 reviews and reviews with more than 200 tokens, yielding dataset of 39,000 reviews from 137 users with an over 69,000-word vocabulary. The task is to predict the review score between
5; the sensitive attribute is the author's identity
PIPA is a set of over 60,000 photos of 2,000 individuals gathered from public Flickr photo al- bums (Piper project page; Zhang al., 2015). Each image can include one more individuals. We cropped their head regions using the bounding boxes in the image annotations. The task is to predict the identity given the head region; the sensitive attribute is whether two head regions are from the same photo.
Models. For Health, we use two-layer fully connected (FC) neural network with 128 and hidden units, respectively, following (Xie et al., 2017; Moyer et al., 2018). For UTKFaaee and FaceScrub, we use a LeNet (LeCun et al., 1998) variant: three 3 x convolutional and 2x2 max-pooling layers with 16, 32, and 64 filters, followed by two FC layers with 128 and 64 hidden units. For Twitter and Yelp, we use text CNN (Kim, 2014). For Places365 and PIPA, we use AlexNet (Krizhevsky et al., 2012) with convolutional layers pre-trained on ImageNet (Deng et al., 2009) and further add a 3 3 convolutional layer with 128 filters and 2 2x2 max max--pooling followed by two FC layers with 128 and 64 hidden units, respectively.
4

---

Published as a conference paper at ICLR 2020
Table 2: Accuracy of inference from representations (last FC layer). RAND is random guessing based on majority class labels; BASE is inference from the uncensored representation; ADV from the representation censored with adversarial training: IT from the information-thheorttcaall censored representation.
Acc of predicting target
Ace of inferring sensitive attribute
Dataset
RAND
BASE
ADV
IT
RAND
BASE
ADV
IT
Health
66.31
84.33
80.16
82.63
16.00
32.52
32.00
26.60
UTKFace
52.27
90.38
90.15
88.15
42.52
62.18
53.28
53.30
FaceScrub
53.53
98.77
97.90
97.66
1.42
33,65
30.23
10.61
Places365
56,16
91.41
90.84
89.82
1.37
31.03
12.56
2.29
Twitter
45.17
76.22
57.97
n/a
6.93
38.46
34.27
n/a
Yelp
42.56
57.81
56,79
n/a
15.88
33.09
27.32
n/a
PIPA
7.67
77.34
52.02
29.64
68.50
87.95
69.96
82.02
4.2 INFERRING SENSITIVE ATTRIBUTES FROM REPRESENTATION
Setup. We use 80% of the data for training the target models and 20% for evaluation. The size of the adversary's auxiliary dataset is 50% of the training data. Success of the inference attack is measured on the final FC layer's 's representation of test data. The baseline is inference from the uncensored representation. We also measure the success of inference against representations censored with = 1.0 for adversarial training and 3 = 0.01, 0.0001 for information-theorrtical censoring, following (Xie al., 2017; Moyer et al., 2018).
For censoring with adversarial training, we simulate the adversary with a two-layer FC neural net- work with 256 and 128 hidden units. The number of epochs is 50 for censoring with adversarial training. 30 for the other models. We use the Adam optimizer with the learning rate of 0.001 and batch size of 128. For
censoring. the model based on VAE (Kingma & Welling, 2013; Moyer et al., 2018). The encoder q(=(z) has the same architecture as the CNN mod- els with all convolutional layers. On top of that, the encoder outputs a mean vector and a standard deviation vector to model the random variable with the e--arammetrizzation trick. The decoder p(z[z) has three de-convolution layers with up-sampling to map z back to the same shape as the input x.
For our inference model, we use the same architecture as the censoring adversary. For the PIPA inference model, which takes two representations of faces and outputs binary prediction of whether these faces appear in the same photo, we use two FC layers followed by a bilinear model: p(s]21, 22) o (h(2) (1)Wh(2)) T) where #1, 22 are the two input representations, h is the two FC layers, and a is the sigmoid function. We train the inference model for 50 epochs with the Adam optimizer, learning rate of 0.001, and batch size of 128.
Results. Table reports the results. When representations are not censored, accuracy of inference from the last-layer representations much higher than random guessing for all tasks, which means models overlearn even in the higher, task-specific layers. When representations are censored with adversarial training, accuracy drops for both the main and inference tasks. Accuracy of infer- ence is much higher than in (Xie et al., 2017). The latter uses logistic regression, which is weaker wwwooring-avversaan network, whereas we use the same architecture for both
than the training-time the training- time and post-hoe adversaries.
colrmation-theoretiin censoring reduces accuracy of inference, but also damages main-task accuracy more than adversarial training for almost all models.
Overlearning can cause model to recognize even the sensitive attributes that are not repre- sented in the training dataset. Such attributes cannot be censored using any known technique. We trained a UTKFace gender classifier datasets where all faces are of the same race. We then applied this model to test images with four races (White, Black, Asian, Indian) and attempted to infer the race attribute from the model's representations. Inference accuracy is 61.95%, 61. 61.99% 60.85% and 60.81 % for models trained only on, respectively, White, Black, Asian, and Indian Paaages-llann as good the 62.18% baseline and much higher than random guessing (42.52%).
Effect of censoring strength. Fig. 2 shows that stronger censoring does not help. On FaceScrub and Twitter with adversarial training, increasing y damages the model's accuracy the main task, while
5

---

Published as a conference paper at ICLR 2020
FaceScrub
UTK Face
Twitter
Yelp
Relative ace
ADV
-20
-
20
-10
-30
-10
0.5
1
1.5
0.2
0.4
0.6
0.8
0.5
1.5
7
7
Health
UTK
FaceScrub
Places365
Relative acc
0
10
IT
-10
20
330
300
2
1.2
1.4
1.6
1.8
0.5
0.76
1
1.25
1.5
2
B
10
B 10
B 10
3 100
Health
UTKFaae
FaceScrub
Places365
Relative acc
-5
10
IT
-10
20
20
20
0.01
0.1
0.5 0.2 0.4 0.6
0.8
-4
10 -4
A
A 10
A
Figure 2: Reduction in accuracy due to censoring. Blue lines are the main task, red lines are the inference of sensitive attributes. First row is adversarial training with different values; second and third row is cnpommttion-thhorttice censoring with different 3 and À values respectively.
Table Improving inference accuracy with de-censoring. the increase from Table
Dataset
Health
UTKFace
FaceSerub
Places365
Twitter
Yelp
ADV +5
32.55 +0.55
59.38
+6.10
40.37 +12.24
19.71 +7.15
36.55
+2.22
31.36
+4.04
IT +-
27.05 +0.45
54.31 +1.01
16.40 +5.79
3.10 +0.81
n/a
n/a
accuracy of inference decreases slightly or remains the same. For UTKFace and Yelp, increasing improves accuracy of inference. This may indicate that the simulated "adversary" during adversarial training overpowers the optimization process and censoring defeats itself.
For all models with pnpemmation-theoreeice censoring. increasing 3 reduces the accuracy of infer- ence but can lead to the model not converging on its main task. Increasing À results in the model not converging on the main task, without affecting the accuracy of inference, FaceScrub. This seems to contradict the censoring objective, but the reconstruction loss in Equa- tion 2 dominates the other loss terms, which leads poor divergence between conditional q(2[x)) and (z)), i.c., information about is still retained in 2.
Health, UTKFace and
De-censoring. As described in Section 3.1, we developed new technique to transform censored representations to make inference easier. We first train an auxiliary model on Daux to predict the sensitive attribute from representations, using the same architecture as the baseline models. The resulting uncensored representations from the last convolutional layer are the target for the de- censoring transformations. We use single-layer fully connected neural network as the transformer and set the number of hidden units to the dimension of the uncensored representation. The inference model operates on top of the transformer network, with the same syper-parameters as before.
Table 3 shows that de-censoring significantly boosts the accuracy of inference from representations censored with adversarial training. The boost is smaller against because its objective not only censors with I[2, s). but also forgets with I (zx 2). On the Health task, there is not much difference since the baseline attack already similar to the attack on censored representations, leaving little room for improvement.
censoring

---

Published as a conference paper at ICLR 2020
Table 4:
Adversarial re-purposing. The values differences between the accuracy of predicting sensitive attributes using re-purposed model vs. model trained from scratch.
[Dmnsta [
Health
UTKFace
FaceScrub
Places365
Twitter
Yelp
PIPA
0.02
-0.57
4.72
7.01
4.42
12.99
5.57
1.33
0.04
0.22
2.70
15.07
2.14
10.87
3.60
2.41
0.06
-1,21
2.83
7.02
2.06
10.51
8.45
6.50
0.08
-0.99
0.25
11.80
3.39
9.57
0.33
4.93
0.10
0.35
2.24
9.43
2.86
7.30
2.1
5.89
Table 5: The effect of censoring on adversarial re-purposing for FaceScrub with y = 0.5, 0.75, 1.0. A is the difference in the original-task accuracy (second column) between uncensored and censored models; og is the difference in the accuracy of inferring the sensitive attribute (columns 3 to 7) between the models re-purposed from different layers and the model trained from scratch. Negative values mean reduced accuracy. Heatmaps on the right are linear CKA similarities between censored and uncensored representations. Numbers through 4 represent layers conv 1, conv2, conv3, fc4, and fe5, For each model censored at layer (x-axis), we measure similarity between the censored and uncensored models layer (y-axis).
?: 0.5
Censored on ? 0.5
åØ when transferred from conv2
1.0
òA
conv1
conv3
fo4
fc5
D.8
Fine Fine-tuned cen-ttuedd.mm
0.8
conv1 1
-1.66
-6,42
-4.09
-1.65
0.46
-3.87
0.4
conv2
-2.87
0.95
-1.77
-2.88
-1.53
-2.22
conv3
-0.64
1.49
1.49
0.67
-0.48
-1.38
0.2
fo4
-0.16
2.03
5.16
6.73
6.12
0.54
fc5
0.05
1.52
4.53
7.42
6.14
4.53
0.75
Fine-tuned on
1.0
0.75
0.8
conv1
-4.48
-7.33
-5.01
-1.51
-7.99
-7.82
0.6
conv2
-6.02
0.44
-7.04
-5.46
-5.94
-5.82
D.4
conv3
-1.90
1.32
1.37
1.88
0.74
-0.67
f'4
0.01
3,65
4.56
5.11
4.44
0.91
0.0
-0.74
1.54
3.61
6.75
7.18
1.0
1.0
Fine- cre-ttuedde on
0.8
convl
-45.25
-7.36
-3.93
-2.75
-4.37
-2.91
0.6
conv2
-20.30
-3,28
-5.27
-7.03
-6.38
-5.54
conv3
-45.20
-2.13
-3.06
-4.48
-4.05
-5.18
0.2
fc4 -0.52
1.73
5.19
4.80
5.83
1.84
0.0
fc5
-0.86
1.56
3.55
5.59
5.14
1.97
0
Censored on
In summary, these results demonstrate that information about sensitive attributes unintentionally captured by the overlearndd representations cannot be suppressed by censoring.
4.3 RE-PURPOSING MODELS TO PREDICT SENSITIVE ATTRIBUTES
To demonstrate that overlearned representations can be picked up by small set of unseen data create a model for predicting sensitive aftributes. we re-purpose uncensored baseline models from Section 4.2 by fine-tuning them on small (2 10% of D) set Diramste and compare with the models trained from scratch on Drranser- We fine-tune all models for epochs with batch size as in Section 4.2. For all CNN models, we the trained
of 32; the other hyper-parameter convolutional layers as the feature extractor and randomly initialize the other layers. Table shows that the re- purposed models always outperform those trained from scratch. FaceScrub and Twitter exhibit the biggest gain.
Effect of censoring. Previous work only censored the highest layer of the models. Model purposing can use any layer of the model for transfer learning. Therefore, to prevent re- re-purposing, inner layers must be censored, too. We perform the first study of inner-layers censoring and measure

---

Published as conference paper at ICLR 2020
0% trained
10% trained
20% trained
40% trained
100% trained
UTK
Layer of B
FaceSerub Layer of B
4
3
3
0.8
0.4
0.2
0.0
Layer of
Layer of
Layer
Layer of Layer of A
Figure 3: Pairwise similarities of layer representations between models for the original task (A) and for predicting sensitive attribute (B). Numbers 0 through 4 denote layers convi, conv2, conv3, fc4 and fc5.
its effect on both the original and re- seeppuposee tasks. We use FaceScrub for this experiment and apply adversarial training every layer with different strengths (7 0.5, 0.75, 1.0).
Table summarizes the results. Censoring lower layers (conv1 to conv3) blocks adversarial re- purposing, at the cost of reducing the model's accuracy its original task. Hyper-parameterss must be tuned carefully, c.g. when 1, there is a huge drop in the original-tase accuracy.
To further investigate how censoring in one layer affects the representations learned across all layers, we measure per-layer similarity between censored and uncensored models using CKA, linear cen- tered kernel alignment (Kornblith al., 2019) see Table When censoring is applied a specific layer, similarity for that layer is the smallest (values on the diagonal). When censoring lower layers with moderate strength (y 0.5 or 0.75), similarity between higher layers is still strong; when censoring higher layers, similarity between lower layers is strong. Therefore, censoring can block adversarial re- p-rpposiing from specific layer, the adversary can still re-purpose representations in the other layer(s) to obtain accurate model for predicting sensitive attributes.
4.4 WHEN, WHERE, AND WHY OVERLEARNING HAPPENS
To investigate when (during training) and where (in which layer) the models overlearn, we use linear CKA similarity (Kornblith al., 2019) to compare the representations at different epochs of training between models trained for the original task (A) and models trained to predict a sensitive attribute (B). We UTKFace and FaceScrub for these experiments.
Fig. shows that lower layers models A and B learn very similar features. This was observed in (Kornblith et al., 2019) for CIFAR-10 and CIFAR-100 models, but those tasks closely related. In our case. the tasks are entirely different and B reveals the sensitive attribute while A does not. The similar low-level features are learned very early during training. There is little similarity be- tween the low-level features of A and high-level features of (and vice versa), matching intuition. Interestingly, on FaceScrub even the high-level features are similar between A and B.
We conjecture that one of the reasons for overlearning is structural complexity of the data. Previous work theoretically showed that over- perammmtttee neural networks favor simple solutions on struc- tured data when optimized with SGD, where structure is quantified the number of distributions (e.g., images from different identities) within each class the target task (Li & Liang, 2018), i.e., the fewer distributions, the more structured the data. For data generated from more complicated distributions, networks learn more complex solutions, leading to the emergence features that are much more general than the learning objective and, consequently, overlearning.
gender classifier trained on the faces from 50 individuals are closer to the random initialization than the representations trained on the faces from 500 individuals (the hyper-parameterr parameters and the total number training examples are the same in both cases). More complex training data thus results in more complex representations for the same objective.
Fig. 4 shows that the representations
8

---

Published as a conference paper at ICLR 2020
Conv1
Conv2
Conv3
Similarity to Random Weights
50 IDs
0.6
500 IDs
0.9
gee eecesooo
foo cepcoood
0.45
0.55
0.85
sopoeeoo
0.46
10
15
20
25
30
10
15
20
25
30
10
15
20
25
30
Epoch
Epoch
Epoch
Figure 4: Similarity of layer representations of partially trained gender classifier randomly initialized model before training. Models are trained on FaceScrub using 50 IDs (blue line) and 500 IDs (red line).
RELATED WORK
Prior work studied transferability of representations only between closely related tasks. Transfer- ability of features between ImageNet models decreases as the distance between the base and target tasks grows (Yosinski al., 2014), and performance of tasks correlated their distance from the source task (Azizpour et al., 2015). CNN models trained to distinguish coarse classes also distin- guish their subsets (Huh al., 2016). By contrast, we show that models trained for simple tasks implicitly learn privacy-sensitive concepts unrelated to the labels of the original task. Other than an anecdotal mention the seknowledgments paragraph of (Kim et al., 2017) that logit-layer activa- tions leak non non-label concepts, this phenomenon has never been described in the research literature.
Gradient updates revealed by participants in distributed learning leak information about individual training batches that is uncorrelated with the learning objective (Melis et al., 2019). We show that overlearning a generic problem in (fully trained) models, helping explain these observations.
There is large body of research on learning disentangled representations (Bengio al., 2013; Locatello et al., 2019). The goal is to separate the underlying explanatory factors in the representa- tion that it contains all information about the input in interpretable structure. State-of- State-of-the-ant the-art approaches use variational autoencoders (Kingma & Welling, 2013) and their variants to learn dis- entangled representations in unsupervised fashion (Higgins al., 2017; Kumar et al., 2018; Kim & Mnih, 2018; Chen et al., 2018). By contrast, overlearning means that representations learned during supervised training for one task implicitly and automatically enable another task-without disentangling the representation on purpose during training.
Work on censoring representations aims suppress sensitive demographic attributes and identities in the model's output for faireess and privacy. Techniques include adversarial training (Edwards Storkey, 2016), which has been applied to census and health records (Xie et al., 2017), text (Li et al., 2018; Coavoux et al., 201 8; Flazar & Goldberg, 2018), images (Hamm, 2017) and sensor data of wearables (Iwasawa al., 2016). tion between the representation and the sensitive attribute (Moyer et al., 2018; Osia et al., 2018). Neither approach can prevent overlearning, except at the cost of destroying the model's accuracy. Furthermore, these techniques cannot censor attributes that not represented in the training data. We show that overlearned models recognize such attributes, too.
alternative approach to minimize mutual informa-
CONCLUSIONS
We demonstrated that models trained for seemingly simple tasks implicitly learn concepts that are not represented in the objective function. In particular, they learn to recognize sensitive attributes, such as race and identity, that are statistically orthogonal to the objective. The failure of censoring to suppress these attributes and the similarity of learned representations across uncorrelated tasks suggest that overlcarning may be intrinsic, i.c., learning for some objectives may not be possible without recognizing generic low-level features that enable other tasks, including inference of sensi-

---

Published as a conference paper at ICLR 2020
tive attributes. For example, there may not exist set of features that enables model to accurately determine the gender of face but not its race or identity.
This is challenge for regulations such GDPR that aim to control the purposes and uses of machine learning technologies. To protect privacy and ensure certain forms of fairness, users and regulators may desire that models not learn some features and attributes. If overlearning is intrinsic, may not be technically possible to enumerate, let alone control, what models are learning. There- fore, regulators should focus ensuring that models are applied in a way that respects privacy and fairness, while acknowledging that they may still recognize and use sensitive attributes.
Acknowledgments. This research supported in part by NSF grants 161 1770, 1704296, and 1916717, the generosity of Eric and Wendy Schmidt by recommendation of the Schmidt Futures program, and a Google Faculty Research Award.
REFERENCES
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, and Kevin Murphy, Deep variational information bottleneck. In ICLR, 2017.
Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. From generic to specific deep representations for visual recognition. In CVPR Workshops, 2015.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. PAMI, 2013.
Tian Qi Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentan- glement in variational autoencoders. In NIPS, 2018.
Jianfeng Chi, Emmanuel Owusu, Xuwang Yin, Tong Yu, William Chan, Patrick Tague, and Yuan Tian. Privacy partitioning: Protecting user data during the deep learning inference phase. arXiv:18/2.02866. 2018.
Maximin Coavoux, Shashi Narayan, and Shay B. Cohen. Privacy-preserving neural representations of text. In EMNLP, 2018.
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In NIPS, 2016.
Harrison Edwards and Amos J. Storkey. Censoring representations with an adversary. In ICLR, 2016.
Yanai Elazar and Yoav Goldberg. Adversarial removal of demographic attributes from text data. In EMNLP, 2018
EU. General Data Protection Regulation. http::///w...kkkpaoaooooo//////a Data batta_Prottectioo__eeullattoooo 2018.
FaceScrub.
2014.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NIPS, 2014.
Jihun Hamm. Minimax filter: Learning to preserve privacy from inference attacks. JMLR, 18(129): 1-31, 2017.
Heritage Health Prize. hestps:/www.kaggee com. c/ c/hhp, 2012.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta- VAE: Learning basic visual concepts with constrained variational framework. In ICLR, 2017.
10

---

Published as a conference paper at ICLR 2020
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. What makes ImageNet good for transfer learning? arXiv:1608.0861 2016.
Yusuke Iwasawa, Kotaro Nakayama, Ikuko Yairi, and Yutaka Matsuo. Privacy issues regarding the application of DNNs to activity-recogniiee using wearables and its countermeasures by use of adversarial training. In IJCAI, 2016.
Yiping Kang, Johann Hauswald, Cao Gao, Austin Rovinski, Trevor Mudge, Jason Mars, and Lingjia Tang. Neurosurgeon: Collaborative intelligence between the cloud and mobile edge. In ASPLOS, 2017.
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, and Rory Sayres. Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). arXiv: 2017.
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. ICML, 2018.
Yoon Kim. Convolutional neural networks for sentence classification. In EMNLP. 2014.
Diederik Kingma and Max Welling. Auto-encoding variational Bayes. arXiv:1312.6114. 2013.
Simon Korblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In ICML, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet classification with deep convo- lutional neural networks. In NIPS, 2012.
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. Variational inference disentan- gled latent concepts from unlabeled observations. In ICLR, 2018.
Nicholas D Lane and Petko Georgiev. Can deep learning revolutionize mobile sensing? In HotMo- bile, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proc. IEEE, 86(1 11):228--23222 1998.
Meng Li, Liangzhen Lai, Naveen Suda, Vikas Chandra, and David Pan. PrivyNet: flexible framework for privacy-preserving deep neural network training. arXiv:1709.06161, 2017.
Yitong Li, Timothy Baldwin, and Trevor Cohn. Towards robust and privacy-preserving text repre- sentations. In ACI., 2018.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In NIPS, 2018.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learn- ing of disentangled representations. In ICML., 2019.
Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. The variational fair autoencoder. ICLR, 2016.
David Madras, Elliot Creager. Toniann Pitassi, and Richard Zemel. Learning adversarially fair and transferable representations. ICML, 2018.
Luca Melis, Congzheng Song, Emiliano De Cristofaro, and Vitaly Shmatikov, Exploiting unintended feature leakage in collaborative learning. In S&P, 2019.
Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. Invariant representations without adversarial training. In NIPS, 2018.
Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks. In NIPS, 2016.

---

Published as a conference paper at ICLR 2020
Seyed Ali Osia, Ali Taheri, Ali Shahin Shamsabadi, Minos Katevas, Hamed Haddadi, and Hamid R. R. Rabiee. Deep private-feature extraction. TKDE, 2018.
Piper project page.
wttp:lleeeecc..beeeeeeeeeeeee acga/eeeep...d/mmrrhha////ieee html,
2015.
Francisco Rangel, Paolo Rosso, Ben Verhoeven, Walter Daelemans, Martin Potthast, and Benno Stein. Overview of the 4th author profiling task at PAN 2016: Cross-genre evaluations. In CEUR Workshop, 2016.
Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. Learning controllable fair representations. In AISTATS, 2019.
UTKFace. http::/aicip..eeeeee
utk edu/ wiki//ikk//// UTKKU 2017.
Wang, Jianguo Zhang, Weidong Bao, Xiaomin Zhu, Bokai Cao, and Philip S Yu. Not privacy: Improving performance of private deep learning in mobile cloud. In KDD, 2018.
Qizhe Xie, Zihang Dai, Yulun Du, Eduard H. Hovy, and Graham Neubig, Controllable invariance through adversarial feature learning. In NIPS, 2017.
Yelp Open Dataset. https:/ http://wwwwwwp....ooo..o... www.:epppp.....ee..om.mmmmo....o dataset, 2018.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features in deep
neural networks? In NIPS, 2014.
Rich Zemel, Yu Wu, Kevin Swersky, Toni Pitassi, and Cynthia Dwork. Learning fair representations. In ICML, 2013.
Ning Zhang, Manohar Paluri, Yaniv Taigman, Rob Fergus, and Lubomir Bourdev. Beyond frontal faces: Improving person recognition using multiple cues. In CVPR, 2015.
Zhifei Zhang, Yang Song, and Hairong Qi. Age progresssion/regressinn by conditional adversarial autoencoder. CVPR, 2017.
12


Reviews:
[{"review_id": "rJgKW00iKS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper demonstrates some limitations of censoring for privacy with respect to sensitive attributes. In particular, the authors show that censoring reduces, but does not eliminate, the ability of a neural network to infer private/sensitive attributes, e.g. to infer race from a model aiming to predict gender. Part of the proposed method is a component that performs de-censoring using an auxiliary dataset. The authors show that censoring strength often does reduce the ability to infer sensitive attributes, but also affects the ability to perform the main (non-sensitive) task; and in some cases, may actually increase ability to infer sensitive attributes. This type of work is important in that privacy is of growing importance, and so is the risk to privacy; this particular work is well carried out. \n\nOne concern: In Sec. 3.1, there is an assumption of the availability of D_{aux}. How realistic is this assumption?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1lokTbf5H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper highlights the problem of model overlearning - learning more than it is trained to do. Thus, there is leak of privacy and sensitive attributes of images during test/ inference time.\n\nPros:\n1. The paper is well written and easy to follow\n\nCons:\n1. There is very little novelty in this paper - the notion of overlearning is well established in the literature (Osia et al., 2018; Chi et al., 2018; Wang et al., 2018). This paper merely reinstates, what is being already told in the literature.\n\n2. In fact, there are many defence mechanisms proposed in the literature, for example \"Anonymizing k Facial Attributes via Adversarial Perturbations\" IJCAI 2018 - where the authors are performing data perturbations to minimize overlearning. This paper does not suggest or propose any method for solving the issue of overlearning\n\nIn summary, this paper repeats a well established problem of overlearning, showing experiments that are already shown in literature with known datasets, and also NOT proposing a solution to minimize overlearning (as many papers already proposed in literature). \n\n3. Additionally, the experiments are very weak - the authors still perform experiments using LeNet variants and AlexNet, and for text using a textCNN. Why did the authors not perform experiments using more state-of the art CNN/RNN models. Did they not observe overlearning in these models?\n\n4. As for section 4.4, it is pretty understood that lower layers of a DL model, learns very basic low-level features from the images such as edges, corner. Reinstating that, and calling it the reason for overlearning is not very convincing. \n\nAs of now, I find the paper very weak, till a solution to avoid overlearning is not proposed as a part of this paper.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SJezFeRmcB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces the problem of overlearning, which can be thought of as unintended transfer learning from a (victim) source model to a target task that the source model’s creator had not intended its model to be used for. The paper raises good points about privacy legislation limitations due to the fact that overlearning makes it impossible to foresee future uses of a given dataset.\n\nBackground on censoring is well developed, and helps position the submission. However, the relation to transfer learning is not sufficiently outlined in the introduction. In some ways, overlearning is a form of unintended transfer learning. In particular, the connection appears explicit in the case of model repurposing (Section 3.2). Editing the introduction to tease apart this relationship to transfer learning would help readers forge an intuition for what the paper considers.\n\nWhile the de-censoring algorithm is intuitive, it is not clear what assumptions are being made because of ambiguous notation in Algorithm 1. How does Algorithm 1 train E_aux and C_aux? Does that step assume the adversary has knowledge of the algorithm use to train E and C? Descriptions of the experimental setup from Section 4.2 seem to indicate that this is the case. This is not necessarily an issue if the proposed attack is demonstrating a limitation of learning rather than a practical attack. \n\nExperiments overall show that censoring does not mitigate overlearning in a way that is robust to de-censoring. One aspect of the experiments that is unclear is the auxiliary dataset: what is the auxiliary dataset used by the adversary for each of the datasets considered in experiments? \n\nQuestion: how does simultaneous censoring of all layers affect overlearning and repurposing?\n\nQuestion: what is the connection between learning more complex representations and overlearning? Is the intuition that if the representation is more complex, it is more likely to contain features useful to identify the sensitive attribute?\n\nThe paper is overall well-written. Some parts of the paper omit important details (perhaps due to space constraints?), but an editorial pass should address most of these. Detailed feedback:\n\n1 / Explaining what model partitioning means in this context would help make the introduction more self-contained. \n\n1 / Is overlearning specific to attributes that raise fairness issues? Or is this phenomenon more general? The “Censoring representations” paragraph on page 2 seems to indicate that the phenomenon is more general.\n\n5/ Is accuracy the best metric to report (e.g., for race attribute prediction) given that the random guessing figure suggests the data is not balanced across attribute values?\n\n6/ Why is the effect not monotonic in Table 4? Were multiple runs averaged?", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces the problem of overlearning, which can be thought of as unintended transfer learning from a (victim) source model to a target task that the source model’s creator had not intended its model to be used for. The paper raises good points about privacy legislation limitations due to the fact that overlearning makes it impossible to foresee future uses of a given dataset.

Please incorporate the revisions suggested in the reviews to add clarity to the overlearning versus censoring confusion addressed by the reviewers.
Decision: Accept (Poster)