Overlearning Reveals Sensitive Attributes

1 INTRODUCTION . We demonstrate that representations learned by deep models when training for seemingly simple objectives reveal privacy- and bias-sensitive attributes that are not part of the specified objective . These unintentionally learned concepts are neither finer- , nor coarse-grained versions of the model ’ s labels , nor statistically correlated with them . We call this phenomenon overlearning . For example , a binary classifier trained to determine the gender of a facial image also learns to recognize races ( including races not represented in the training data ) and even identities of individuals . Overlearning has two distinct consequences . First , the model ’ s inference-time representation of an input reveals the input ’ s sensitive attributes . For example , a facial recognition model ’ s representation of an image reveals if two specific individuals appear together in it . Overlearning thus breaks inference-time privacy protections based on model partitioning ( Osia et al. , 2018 ; Chi et al. , 2018 ; Wang et al. , 2018 ) . Second , we develop a new , transfer learning-based technique to “ re-purpose ” a model trained for benign task into a model for a different , privacy-violating task . This shows the inadequacy of privacy regulations that rely on explicit enumeration of learned attributes . Overlearning is intrinsic for some tasks , i.e. , it is not possible to prevent a model from learning sensitive attributes . We show that if these attributes are censored ( Xie et al. , 2017 ; Moyer et al. , 2018 ) , the censored models either fail to learn their specified tasks , or still leak sensitive information . We develop a new de-censoring technique to extract information from censored representations . We also show that overlearned representations enable recognition of sensitive attributes that are not present in the training data . Such attributes can not be censored using any known technique . This shows the the inadequacy of censoring as a privacy protection technology . To analyze where and why overlearning happens , we empirically show how general features emerge in the lower layers of models trained for simple objectives and conjecture an explanation based on the complexity of the training data . 2 BACKGROUND . We focus on supervised deep learning . Given an input x , a model M is trained to predict the target y using a discriminative approach . We represent the model M = C ◦ E as a feature extractor ( encoder ) E and classifier C. The representation z = E ( x ) is passed to C to produce the prediction by modeling p ( y|z ) = C ( z ) . Since E can have multiple layers of representation , we use El ( x ) = zl to denote the model ’ s internal representation at layer l ; z is the representation at the last layer . Model partitioning splits the model into a local , on-device part and a remote , cloud-based part to improve scalability of inference ( Lane & Georgiev , 2015 ; Kang et al. , 2017 ) and protect privacy of inputs into the model ( Li et al. , 2017 ; Osia et al. , 2018 ; Chi et al. , 2018 ; Wang et al. , 2018 ) . For privacy , the local part of the model computes a representation , censors it as described below , and sends it to the cloud part , which computes the model ’ s output . Censoring representations . The goal is to encode input x into a representation z that does not reveal unwanted properties of x , yet is expressive enough to predict the task label y. Censoring has been used to achieve transform-invariant representations for computer vision , bias-free representations for fair machine learning , and privacy-preserving representations that hide sensitive attributes . A straightforward censoring approach is based on adversarial training ( Goodfellow et al. , 2014 ) . It involves a mini-max game between a discriminator D trying to infer s from z during training and an encoder and classifier trying to infer the task label y while minimizing the discriminator ’ s success ( Edwards & Storkey , 2016 ; Iwasawa et al. , 2016 ; Hamm , 2017 ; Xie et al. , 2017 ; Li et al. , 2018 ; Coavoux et al. , 2018 ; Elazar & Goldberg , 2018 ) . The game is formulated as : min E , C max D Ex , y , s [ γ log p ( s|z = E ( x ) ) − log p ( y|z = E ( x ) ) ] ( 1 ) where γ balances the two log likelihood terms . The inner optimization maximizes log p ( s|z = E ( x ) ) , i.e. , the discriminator ’ s prediction of the sensitive attribute s given a representation z . The outer optimization , on the other hand , trains the encoder and classifier to minimize the log likelihood of the discriminator predicting s and maximize that of predicting the task label y . Another approach casts censoring as a single information-theoretical objective . The requirement that z not reveal s can be formalized as an independence constraint z ⊥ s , but independence is intractable to measure in practice , thus the requirement is relaxed to a constraint on the mutual information between z and s ( Osia et al. , 2018 ; Moyer et al. , 2018 ) . The overall training objective of censoring s and predicting y from z is formulated as : max I ( z , y ) − βI ( z , x ) − λI ( z , s ) ( 2 ) where I is mutual information and β , λ are the balancing coefficients ; β = 0 in ( Osia et al. , 2018 ) . The first two terms I ( z , y ) − βI ( z , x ) is the objective of variational information bottleneck ( Alemi et al. , 2017 ) , the third term is the relaxed independence constraint of z and s. Intuitively , this objective aims to maximize the information of y in z as per I ( z , y ) , forget the information of x in z as per −βI ( z , x ) , and remove the information of s in z as per −λI ( z , s ) . This objective has an analytical lower bound ( Moyer et al. , 2018 ) : Ex , s [ Ez , y [ log p ( y|z ) ] − ( β + λ ) KL [ q ( z|x ) ||q ( z ) ] − λEz [ log p ( x|z , s ) ] ] ( 3 ) where KL is Kullback-Leibler divergence and log p ( x|z , s ) is the reconstruction likelihood of x given z and s. The conditional distributions p ( y|z ) = C ( z ) , q ( z|x ) = E ( x ) are modeled as in adversarial training and p ( x|z , s ) is modeled with a decoder R ( z , s ) = p ( x|z , s ) . All known censoring techniques require a “ blacklist ” of attributes to censor , and inputs with these attributes must be represented in the training data . Censoring for fairness is applied to the model ’ s final layer to make its output independent of the sensitive attributes or satisfy a specific fairness constraint ( Zemel et al. , 2013 ; Louizos et al. , 2016 ; Madras et al. , 2018 ; Song et al. , 2019 ) . In this paper , we use censoring not for fairness but to demonstrate that models can not be prevented from learning to recognize sensitive attributes . To show this , we apply censoring to different layers , not just the output . 3 EXPLOITING OVERLEARNING . We demonstrate two different ways to exploit overlearning in a trained model M . The inferencetime attack ( Section 3.1 ) applies M to an input and uses M ’ s representation of that input to predict its sensitive attributes . The model-repurposing attack ( Section 3.2 ) uses M to create another model that , when applied to an input , directly predicts its sensitive attributes . Algorithm 1 De-censoring representations 1 : Input : Auxiliary dataset Daux , black-box oracle E , observed representation z ? 2 : Train auxiliary model Maux = Eaux ◦ Caux on Daux 3 : Initialize transform model T , inference attack model Mattack 4 : for each training iteration do 5 : Sample a batch of data ( x , s ) from Daux and compute z = E ( x ) , zaux = Eaux ( x ) 6 : Update T on the batch of ( z , zaux ) with loss ||T ( z ) − zaux||22 7 : Update Mattack on the batch of ( T ( z ) , s ) with cross-entropy loss 8 : end for 9 : return prediction ŝ =Mattack ( T ( z ? ) ) 3.1 INFERRING SENSITIVE ATTRIBUTES FROM REPRESENTATION . We measure the leakage of sensitive properties from the representations of overlearned models via the following attack . Suppose an adversary can observe the representation z ? of a trained model M on input x ? at inference time but can not observe x ? directly . This scenario arises in practice when model evaluation is partitioned in order to protect privacy of inputs—see Section 2 . The adversary wants to infer some property s of x ? that is not part of the task label y . We assume that the adversary has an auxiliary set Daux of labeled ( x , s ) pairs and black-box oracle E to compute the corresponding E ( x ) . The purpose of Daux is to help the adversary recognize the property of interest in the model ’ s representations ; it need not be drawn from the same dataset as x ? . The adversary uses supervised learning on the ( E ( x ) , s ) pairs to train an attack model Mattack . At inference time , the adversary predicts ŝ from the observed z ? as Mattack ( z ? ) . De-censoring . If the representation z is “ censored ” ( see Section 2 ) to reduce the amount of information it reveals about s , the direct inference attack may not succeed . We develop a new , learning-based de-censoring approach ( see Algorithm 1 ) to convert censored representations into a different form that leaks more information about the property of interest . The adversary trains Maux on Daux to predict s from x , then transforms z into the input features of Maux . We treat de-censoring as an optimization problem with a feature spaceL2 loss ||T ( z ) −zaux||22 , where T is the transformer that the adversary wants to learn and zaux is the uncensored representation from Maux . Training with a feature-space loss has been proposed for synthesizing more natural images by matching them with real images ( Dosovitskiy & Brox , 2016 ; Nguyen et al. , 2016 ) . In our case , we match censored and uncensored representations . The adversary can then use T ( z ) as an uncensored approximation of z to train an inference model Mattack and infer property s as Mattack ( T ( z ? ) ) . 3.2 RE-PURPOSING MODELS TO PREDICT SENSITIVE ATTRIBUTES . To re-purpose a model—for example , to convert a model trained for a benign task into a model that predicts a sensitive attribute—we can use features zl in any layer of M as the feature extractor and connect a new classifier Ctransfer to El . The transferred model Mtransfer = Ctransfer ◦ El is fine-tuned on another , small dataset Dtransfer , which in itself is not sufficient to train an accurate model for the new task . Utilizing features learned by M on the original D , Mtransfer can achieve better results than models trained from scratch on Dtransfer . Feasibility of model re-purposing complicates the application of policies and regulations such as GDPR ( EU , 2018 ) . GDPR requires data processors to disclose every purpose of data collection and obtain consent from the users whose data was collected . We show that , given a trained model , it is not possible to determine—nor , consequently , disclose or obtain user consent for—what the model has learned . Learning per se thus can not be a regulated “ purpose ” of data collection . Regulators must be aware that even if the original training data has been erased , a model can be re-purposed for a different objective , possibly not envisioned at the time of original data collection . We discuss this further in Section 6 .

Summary: This paper demonstrates some limitations of censoring for privacy with respect to sensitive attributes. In particular, the authors show that censoring reduces, but does not eliminate, the ability of a neural network to infer private/sensitive attributes, e.g. to infer race from a model aiming to predict gender. Part of the proposed method is a component that performs de-censoring using an auxiliary dataset. The authors show that censoring strength often does reduce the ability to infer sensitive attributes, but also affects the ability to perform the main (non-sensitive) task; and in some cases, may actually increase ability to infer sensitive attributes. This type of work is important in that privacy is of growing importance, and so is the risk to privacy; this particular work is well carried out.