OVERLEARNING REVEALS SENSITIVE ATTRIBUTES
Year: 2020

Abstract:
“Overlearning” means that a model trained for a seemingly simple objective implicitly learns to recognize attributes and concepts that are (1) not part of the learning objective, and (2) sensitive from a privacy or bias perspective. For example, a binary gender classifier of facial images also learns to recognize races—even races that are not represented in the training data—and identities. We demonstrate overlearning in several vision and NLP models and analyze its harmful consequences. First, inference-time representations of an overlearned model reveal sensitive attributes of the input, breaking privacy protections such as model partitioning. Second, an overlearned model can be “re-purposed” for a different, privacy-violating task even in the absence of the original training data. We show that overlearning is intrinsic for some tasks and cannot be prevented by censoring unwanted attributes. Finally, we investigate where, when, and why overlearning happens during model training.

heading: 1 INTRODUCTION
text: We demonstrate that representations learned by deep models when training for seemingly simple objectives reveal privacy- and bias-sensitive attributes that are not part of the specified objective. These unintentionally learned concepts are neither finer-, nor coarse-grained versions of the model’s labels, nor statistically correlated with them. We call this phenomenon overlearning. For example, a binary classifier trained to determine the gender of a facial image also learns to recognize races (including races not represented in the training data) and even identities of individuals.
Overlearning has two distinct consequences. First, the model’s inference-time representation of an input reveals the input’s sensitive attributes. For example, a facial recognition model’s representation of an image reveals if two specific individuals appear together in it. Overlearning thus breaks inference-time privacy protections based on model partitioning (Osia et al., 2018; Chi et al., 2018; Wang et al., 2018). Second, we develop a new, transfer learning-based technique to “re-purpose” a model trained for benign task into a model for a different, privacy-violating task. This shows the inadequacy of privacy regulations that rely on explicit enumeration of learned attributes.
Overlearning is intrinsic for some tasks, i.e., it is not possible to prevent a model from learning sensitive attributes. We show that if these attributes are censored (Xie et al., 2017; Moyer et al., 2018), the censored models either fail to learn their specified tasks, or still leak sensitive information. We develop a new de-censoring technique to extract information from censored representations. We also show that overlearned representations enable recognition of sensitive attributes that are not present in the training data. Such attributes cannot be censored using any known technique. This shows the the inadequacy of censoring as a privacy protection technology.
To analyze where and why overlearning happens, we empirically show how general features emerge in the lower layers of models trained for simple objectives and conjecture an explanation based on the complexity of the training data.

heading: 2 BACKGROUND
text: We focus on supervised deep learning. Given an input x, a model M is trained to predict the target y using a discriminative approach. We represent the model M = C ◦ E as a feature extractor
(encoder) E and classifier C. The representation z = E(x) is passed to C to produce the prediction by modeling p(y|z) = C(z). Since E can have multiple layers of representation, we use El(x) = zl to denote the model’s internal representation at layer l; z is the representation at the last layer.
Model partitioning splits the model into a local, on-device part and a remote, cloud-based part to improve scalability of inference (Lane & Georgiev, 2015; Kang et al., 2017) and protect privacy of inputs into the model (Li et al., 2017; Osia et al., 2018; Chi et al., 2018; Wang et al., 2018). For privacy, the local part of the model computes a representation, censors it as described below, and sends it to the cloud part, which computes the model’s output.
Censoring representations. The goal is to encode input x into a representation z that does not reveal unwanted properties of x, yet is expressive enough to predict the task label y. Censoring has been used to achieve transform-invariant representations for computer vision, bias-free representations for fair machine learning, and privacy-preserving representations that hide sensitive attributes.
A straightforward censoring approach is based on adversarial training (Goodfellow et al., 2014). It involves a mini-max game between a discriminator D trying to infer s from z during training and an encoder and classifier trying to infer the task label y while minimizing the discriminator’s success (Edwards & Storkey, 2016; Iwasawa et al., 2016; Hamm, 2017; Xie et al., 2017; Li et al., 2018; Coavoux et al., 2018; Elazar & Goldberg, 2018). The game is formulated as:
min E,C max D
Ex,y,s[γ log p(s|z = E(x))− log p(y|z = E(x))] (1)
where γ balances the two log likelihood terms. The inner optimization maximizes log p(s|z = E(x)), i.e., the discriminator’s prediction of the sensitive attribute s given a representation z. The outer optimization, on the other hand, trains the encoder and classifier to minimize the log likelihood of the discriminator predicting s and maximize that of predicting the task label y.
Another approach casts censoring as a single information-theoretical objective. The requirement that z not reveal s can be formalized as an independence constraint z ⊥ s, but independence is intractable to measure in practice, thus the requirement is relaxed to a constraint on the mutual information between z and s (Osia et al., 2018; Moyer et al., 2018). The overall training objective of censoring s and predicting y from z is formulated as:
max I(z, y)− βI(z, x)− λI(z, s) (2) where I is mutual information and β, λ are the balancing coefficients; β = 0 in (Osia et al., 2018). The first two terms I(z, y)− βI(z, x) is the objective of variational information bottleneck (Alemi et al., 2017), the third term is the relaxed independence constraint of z and s.
Intuitively, this objective aims to maximize the information of y in z as per I(z, y), forget the information of x in z as per −βI(z, x), and remove the information of s in z as per −λI(z, s). This objective has an analytical lower bound (Moyer et al., 2018):
Ex,s[Ez,y[log p(y|z)]− (β + λ)KL[q(z|x)||q(z)]− λEz[log p(x|z, s)]] (3) where KL is Kullback-Leibler divergence and log p(x|z, s) is the reconstruction likelihood of x given z and s. The conditional distributions p(y|z) = C(z), q(z|x) = E(x) are modeled as in adversarial training and p(x|z, s) is modeled with a decoder R(z, s) = p(x|z, s). All known censoring techniques require a “blacklist” of attributes to censor, and inputs with these attributes must be represented in the training data. Censoring for fairness is applied to the model’s final layer to make its output independent of the sensitive attributes or satisfy a specific fairness constraint (Zemel et al., 2013; Louizos et al., 2016; Madras et al., 2018; Song et al., 2019). In this paper, we use censoring not for fairness but to demonstrate that models cannot be prevented from learning to recognize sensitive attributes. To show this, we apply censoring to different layers, not just the output.

heading: 3 EXPLOITING OVERLEARNING
text: We demonstrate two different ways to exploit overlearning in a trained model M . The inferencetime attack (Section 3.1) applies M to an input and uses M ’s representation of that input to predict its sensitive attributes. The model-repurposing attack (Section 3.2) uses M to create another model that, when applied to an input, directly predicts its sensitive attributes.
Algorithm 1 De-censoring representations 1: Input: Auxiliary dataset Daux, black-box oracle E, observed representation z? 2: Train auxiliary model Maux = Eaux ◦ Caux on Daux 3: Initialize transform model T , inference attack model Mattack 4: for each training iteration do 5: Sample a batch of data (x, s) from Daux and compute z = E(x), zaux = Eaux(x) 6: Update T on the batch of (z, zaux) with loss ||T (z)− zaux||22 7: Update Mattack on the batch of (T (z), s) with cross-entropy loss 8: end for 9: return prediction ŝ =Mattack(T (z?))

heading: 3.1 INFERRING SENSITIVE ATTRIBUTES FROM REPRESENTATION
text: We measure the leakage of sensitive properties from the representations of overlearned models via the following attack. Suppose an adversary can observe the representation z? of a trained model M on input x? at inference time but cannot observe x? directly. This scenario arises in practice when model evaluation is partitioned in order to protect privacy of inputs—see Section 2. The adversary wants to infer some property s of x? that is not part of the task label y.
We assume that the adversary has an auxiliary set Daux of labeled (x, s) pairs and black-box oracle E to compute the corresponding E(x). The purpose of Daux is to help the adversary recognize the property of interest in the model’s representations; it need not be drawn from the same dataset as x?. The adversary uses supervised learning on the (E(x), s) pairs to train an attack model Mattack. At inference time, the adversary predicts ŝ from the observed z? as Mattack(z?).
De-censoring. If the representation z is “censored” (see Section 2) to reduce the amount of information it reveals about s, the direct inference attack may not succeed. We develop a new, learning-based de-censoring approach (see Algorithm 1) to convert censored representations into a different form that leaks more information about the property of interest. The adversary trains Maux on Daux to predict s from x, then transforms z into the input features of Maux.
We treat de-censoring as an optimization problem with a feature spaceL2 loss ||T (z)−zaux||22, where T is the transformer that the adversary wants to learn and zaux is the uncensored representation from Maux. Training with a feature-space loss has been proposed for synthesizing more natural images by matching them with real images (Dosovitskiy & Brox, 2016; Nguyen et al., 2016). In our case, we match censored and uncensored representations. The adversary can then use T (z) as an uncensored approximation of z to train an inference model Mattack and infer property s as Mattack(T (z?)).

heading: 3.2 RE-PURPOSING MODELS TO PREDICT SENSITIVE ATTRIBUTES
text: To re-purpose a model—for example, to convert a model trained for a benign task into a model that predicts a sensitive attribute—we can use features zl in any layer of M as the feature extractor and connect a new classifier Ctransfer to El. The transferred model Mtransfer = Ctransfer ◦ El is fine-tuned on another, small dataset Dtransfer, which in itself is not sufficient to train an accurate model for the new task. Utilizing features learned by M on the original D, Mtransfer can achieve better results than models trained from scratch on Dtransfer. Feasibility of model re-purposing complicates the application of policies and regulations such as GDPR (EU, 2018). GDPR requires data processors to disclose every purpose of data collection and obtain consent from the users whose data was collected. We show that, given a trained model, it is not possible to determine—nor, consequently, disclose or obtain user consent for—what the model
has learned. Learning per se thus cannot be a regulated “purpose” of data collection. Regulators must be aware that even if the original training data has been erased, a model can be re-purposed for a different objective, possibly not envisioned at the time of original data collection. We discuss this further in Section 6.

heading: 4 EXPERIMENTAL RESULTS

heading: 4.1 DATASETS, TASKS, AND MODELS
text: Health is the Heritage Health dataset (Heritage Health Prize) with medical records of over 55,000 patients, binarized into 112 features with age information removed. The task is to predict if Charlson Index (an estimate of patient mortality) is greater than zero; the sensitive attribute is age (binned into 9 ranges).
UTKFace is a set of over 23,000 face images labeled with age, gender, and race (UTKFace; Zhang et al., 2017). We rescaled them into 50×50 RGB pixels. The task is to predict gender; the sensitive attribute is race.
FaceScrub is a set of face images labeled with gender (FaceScrub). Some URLs are expired, but we were able to download 74,000 images for 500 individuals and rescale them into 50×50 RGB pixels. The task is to predict gender; the sensitive attribute is identity.
Places365 is a set of 1.8 million images labeled with 365 fine-grained scene categories. We use a subset of 73,000 images, 200 per category. The task is to predict whether the scene is indoor or outdoor; the sensitive attribute is the fine-grained scene label.
Twitter is a set of tweets from the PAN16 dataset (Rangel et al., 2016) labeled with user information. We removed tweets with fewer than 20 tokens and users with fewer than 50 tweets, yielding a dataset of over 46,000 tweets from 151 users with an over 80,000-word vocabulary. The task is to predict the age of the user given a tweet; the sensitive attribute is the author’s identity.
Yelp is a set of Yelp reviews labeled with user identities (Yelp Open Dataset). We removed users with fewer than 1,000 reviews and reviews with more than 200 tokens, yielding a dataset of over 39,000 reviews from 137 users with an over 69,000-word vocabulary. The task is to predict the review score between 1 to 5; the sensitive attribute is the author’s identity.
PIPA is a set of over 60,000 photos of 2,000 individuals gathered from public Flickr photo albums (Piper project page; Zhang et al., 2015). Each image can include one or more individuals. We cropped their head regions using the bounding boxes in the image annotations. The task is to predict the identity given the head region; the sensitive attribute is whether two head regions are from the same photo.
Models. For Health, we use a two-layer fully connected (FC) neural network with 128 and 32 hidden units, respectively, following (Xie et al., 2017; Moyer et al., 2018). For UTKFace and FaceScrub, we use a LeNet (LeCun et al., 1998) variant: three 3×3 convolutional and 2×2 max-pooling layers with 16, 32, and 64 filters, followed by two FC layers with 128 and 64 hidden units. For Twitter and Yelp, we use text CNN (Kim, 2014). For Places365 and PIPA, we use AlexNet (Krizhevsky et al., 2012) with convolutional layers pre-trained on ImageNet (Deng et al., 2009) and further add a 3×3 convolutional layer with 128 filters and 2×2 max-pooling followed by two FC layers with 128 and 64 hidden units, respectively.

heading: 4.2 INFERRING SENSITIVE ATTRIBUTES FROM REPRESENTATIONS
text: Setup. We use 80% of the data for training the target models and 20% for evaluation. The size of the adversary’s auxiliary dataset is 50% of the training data. Success of the inference attack is measured on the final FC layer’s representation of test data. The baseline is inference from the uncensored representation. We also measure the success of inference against representations censored with γ = 1.0 for adversarial training and β = 0.01, λ = 0.0001 for information-theoretical censoring, following (Xie et al., 2017; Moyer et al., 2018).
For censoring with adversarial training, we simulate the adversary with a two-layer FC neural network with 256 and 128 hidden units. The number of epochs is 50 for censoring with adversarial training, 30 for the other models. We use the Adam optimizer with the learning rate of 0.001 and batch size of 128. For information-theoretical censoring, the model is based on VAE (Kingma & Welling, 2013; Moyer et al., 2018). The encoder q(z|x) has the same architecture as the CNN models with all convolutional layers. On top of that, the encoder outputs a mean vector and a standard deviation vector to model the random variable z with the re-parameterization trick. The decoder p(x|z) has three de-convolution layers with up-sampling to map z back to the same shape as the input x.
For our inference model, we use the same architecture as the censoring adversary. For the PIPA inference model, which takes two representations of faces and outputs a binary prediction of whether these faces appear in the same photo, we use two FC layers followed by a bilinear model: p(s|z1, z2) = σ(h(z1)Wh(z2)>), where z1, z2 are the two input representations, h is the two FC layers, and σ is the sigmoid function. We train the inference model for 50 epochs with the Adam optimizer, learning rate of 0.001, and batch size of 128.
Results. Table 2 reports the results. When representations are not censored, accuracy of inference from the last-layer representations is much higher than random guessing for all tasks, which means models overlearn even in the higher, task-specific layers. When representations are censored with adversarial training, accuracy drops for both the main and inference tasks. Accuracy of inference is much higher than in (Xie et al., 2017). The latter uses logistic regression, which is weaker than the training-time censoring-adversary network, whereas we use the same architecture for both the training-time and post-hoc adversaries. Information-theoretical censoring reduces accuracy of inference, but also damages main-task accuracy more than adversarial training for almost all models.
Overlearning can cause a model to recognize even the sensitive attributes that are not represented in the training dataset. Such attributes cannot be censored using any known technique. We trained a UTKFace gender classifier on datasets where all faces are of the same race. We then applied this model to test images with four races (White, Black, Asian, Indian) and attempted to infer the race attribute from the model’s representations. Inference accuracy is 61.95%, 61.99%, 60.85% and 60.81% for models trained only on, respectively, White, Black, Asian, and Indian images—almost as good as the 62.18% baseline and much higher than random guessing (42.52%).
Effect of censoring strength. Fig. 2 shows that stronger censoring does not help. On FaceScrub and Twitter with adversarial training, increasing γ damages the model’s accuracy on the main task, while
accuracy of inference decreases slightly or remains the same. For UTKFace and Yelp, increasing γ improves accuracy of inference. This may indicate that the simulated “adversary” during adversarial training overpowers the optimization process and censoring defeats itself.
For all models with information-theoretical censoring, increasing β reduces the accuracy of inference but can lead to the model not converging on its main task. Increasing λ results in the model not converging on the main task, without affecting the accuracy of inference, on Health, UTKFace and FaceScrub. This seems to contradict the censoring objective, but the reconstruction loss in Equation 2 dominates the other loss terms, which leads to poor divergence between conditional q(z|x) and q(z), i.e., information about x is still retained in z.
De-censoring. As described in Section 3.1, we developed a new technique to transform censored representations to make inference easier. We first train an auxiliary model on Daux to predict the sensitive attribute from representations, using the same architecture as in the baseline models. The resulting uncensored representations from the last convolutional layer are the target for the decensoring transformations. We use a single-layer fully connected neural network as the transformer and set the number of hidden units to the dimension of the uncensored representation. The inference model operates on top of the transformer network, with the same hyper-parameters as before.
Table 3 shows that de-censoring significantly boosts the accuracy of inference from representations censored with adversarial training. The boost is smaller against information-theoretical censoring because its objective not only censors z with I(z, s), but also forgets x with I(x, z). On the Health task, there is not much difference since the baseline attack is already similar to the attack on censored representations, leaving little room for improvement.
In summary, these results demonstrate that information about sensitive attributes unintentionally captured by the overlearned representations cannot be suppressed by censoring.

heading: 4.3 RE-PURPOSING MODELS TO PREDICT SENSITIVE ATTRIBUTES
text: To demonstrate that overlearned representations can be picked up by a small set of unseen data to create a model for predicting sensitive attributes, we re-purpose uncensored baseline models from Section 4.2 by fine-tuning them on a small (2 − 10% of D) set Dtransfer and compare with the models trained from scratch on Dtransfer. We fine-tune all models for 50 epochs with batch size of 32; the other hyper-parameters are as in Section 4.2. For all CNN models, we use the trained convolutional layers as the feature extractor and randomly initialize the other layers. Table 4 shows that the re-purposed models always outperform those trained from scratch. FaceScrub and Twitter exhibit the biggest gain.
Effect of censoring. Previous work only censored the highest layer of the models. Model repurposing can use any layer of the model for transfer learning. Therefore, to prevent re-purposing, inner layers must be censored, too. We perform the first study of inner-layers censoring and measure
its effect on both the original and re-purposed tasks. We use FaceScrub for this experiment and apply adversarial training to every layer with different strengths (γ = 0.5, 0.75, 1.0).
Table 5 summarizes the results. Censoring lower layers (conv1 to conv3) blocks adversarial repurposing, at the cost of reducing the model’s accuracy on its original task. Hyper-parameters must be tuned carefully, e.g. when γ = 1, there is a huge drop in the original-task accuracy.
To further investigate how censoring in one layer affects the representations learned across all layers, we measure per-layer similarity between censored and uncensored models using CKA, linear centered kernel alignment (Kornblith et al., 2019)—see Table 5. When censoring is applied to a specific layer, similarity for that layer is the smallest (values on the diagonal). When censoring lower layers with moderate strength (γ = 0.5 or 0.75), similarity between higher layers is still strong; when censoring higher layers, similarity between lower layers is strong. Therefore, censoring can block adversarial re-purposing from a specific layer, but the adversary can still re-purpose representations in the other layer(s) to obtain an accurate model for predicting sensitive attributes.

heading: 4.4 WHEN, WHERE, AND WHY OVERLEARNING HAPPENS
text: To investigate when (during training) and where (in which layer) the models overlearn, we use linear CKA similarity (Kornblith et al., 2019) to compare the representations at different epochs of training between models trained for the original task (A) and models trained to predict a sensitive attribute (B). We use UTKFace and FaceScrub for these experiments.
Fig. 3 shows that lower layers of models A and B learn very similar features. This was observed in (Kornblith et al., 2019) for CIFAR-10 and CIFAR-100 models, but those tasks are closely related. In our case, the tasks are entirely different and B reveals the sensitive attribute while A does not. The similar low-level features are learned very early during training. There is little similarity between the low-level features of A and high-level features of B (and vice versa), matching intuition. Interestingly, on FaceScrub even the high-level features are similar between A and B.
We conjecture that one of the reasons for overlearning is structural complexity of the data. Previous work theoretically showed that over-parameterized neural networks favor simple solutions on structured data when optimized with SGD, where structure is quantified as the number of distributions (e.g., images from different identities) within each class in the target task (Li & Liang, 2018), i.e., the fewer distributions, the more structured the data. For data generated from more complicated distributions, networks learn more complex solutions, leading to the emergence of features that are much more general than the learning objective and, consequently, overlearning.
Fig. 4 shows that the representations of a gender classifier trained on the faces from 50 individuals are closer to the random initialization than the representations trained on the faces from 500 individuals (the hyper-parameters and the total number of training examples are the same in both cases). More complex training data thus results in more complex representations for the same objective.

heading: 5 RELATED WORK
text: Prior work studied transferability of representations only between closely related tasks. Transferability of features between ImageNet models decreases as the distance between the base and target tasks grows (Yosinski et al., 2014), and performance of tasks is correlated to their distance from the source task (Azizpour et al., 2015). CNN models trained to distinguish coarse classes also distinguish their subsets (Huh et al., 2016). By contrast, we show that models trained for simple tasks implicitly learn privacy-sensitive concepts unrelated to the labels of the original task. Other than an anecdotal mention in the acknowledgments paragraph of (Kim et al., 2017) that logit-layer activations leak non-label concepts, this phenomenon has never been described in the research literature.
Gradient updates revealed by participants in distributed learning leak information about individual training batches that is uncorrelated with the learning objective (Melis et al., 2019). We show that overlearning is a generic problem in (fully trained) models, helping explain these observations.
There is a large body of research on learning disentangled representations (Bengio et al., 2013; Locatello et al., 2019). The goal is to separate the underlying explanatory factors in the representation so that it contains all information about the input in an interpretable structure. State-of-the-art approaches use variational autoencoders (Kingma & Welling, 2013) and their variants to learn disentangled representations in an unsupervised fashion (Higgins et al., 2017; Kumar et al., 2018; Kim & Mnih, 2018; Chen et al., 2018). By contrast, overlearning means that representations learned during supervised training for one task implicitly and automatically enable another task—without disentangling the representation on purpose during training.
Work on censoring representations aims to suppress sensitive demographic attributes and identities in the model’s output for fairness and privacy. Techniques include adversarial training (Edwards & Storkey, 2016), which has been applied to census and health records (Xie et al., 2017), text (Li et al., 2018; Coavoux et al., 2018; Elazar & Goldberg, 2018), images (Hamm, 2017) and sensor data of wearables (Iwasawa et al., 2016). An alternative approach is to minimize mutual information between the representation and the sensitive attribute (Moyer et al., 2018; Osia et al., 2018). Neither approach can prevent overlearning, except at the cost of destroying the model’s accuracy. Furthermore, these techniques cannot censor attributes that are not represented in the training data. We show that overlearned models recognize such attributes, too.

heading: 6 CONCLUSIONS
text: We demonstrated that models trained for seemingly simple tasks implicitly learn concepts that are not represented in the objective function. In particular, they learn to recognize sensitive attributes, such as race and identity, that are statistically orthogonal to the objective. The failure of censoring to suppress these attributes and the similarity of learned representations across uncorrelated tasks suggest that overlearning may be intrinsic, i.e., learning for some objectives may not be possible without recognizing generic low-level features that enable other tasks, including inference of sensi-
tive attributes. For example, there may not exist a set of features that enables a model to accurately determine the gender of a face but not its race or identity.
This is a challenge for regulations such as GDPR that aim to control the purposes and uses of machine learning technologies. To protect privacy and ensure certain forms of fairness, users and regulators may desire that models not learn some features and attributes. If overlearning is intrinsic, it may not be technically possible to enumerate, let alone control, what models are learning. Therefore, regulators should focus on ensuring that models are applied in a way that respects privacy and fairness, while acknowledging that they may still recognize and use sensitive attributes.
Acknowledgments. This research was supported in part by NSF grants 1611770, 1704296, and 1916717, the generosity of Eric and Wendy Schmidt by recommendation of the Schmidt Futures program, and a Google Faculty Research Award.

Review summaries:
This paper demonstrates some limitations of censoring for privacy with respect to sensitive attributes. In particular, the authors show that censoring reduces, but does not eliminate, the ability of a neural network to infer private/sensitive attributes, e.g. to infer race from a model aiming to predict gender. Part of the proposed method is a component that performs de-censoring using an auxiliary dataset. The authors show that censoring strength often does reduce the ability to infer sensitive attributes, but also affects the ability to perform the main (non-sensitive) task; and in some cases, may actually increase ability to infer sensitive attributes. This type of work is important in that privacy is of growing importance, and so is the risk to privacy; this particular work is well carried out.