Adversarial representation learning for synthetic replacement of private attributes
Authors: John Martinsson
Edvin Listo Zec
Daniel Gillblad
Olof Mogren
Venue:  (2021)
Keywords: Deep learning
privacy
generative adversarial networks

Abstract: Data privacy is an increasingly important aspect of many real-world big data analytics tasks. Data sources that contain sensitive information may have immense potential which could be unlocked using privacy enhancing transformations, but current methods often fail to produce convincing output. Furthermore, finding the right balance between privacy and utility is often a tricky trade-off. In this work, we propose a novel approach for data privatization, which involves two steps: in the first step, it removes the sensitive information, and in the second step, it replaces this information with an independent random sample. Our method builds on adversarial representation learning which ensures strong privacy by training the model to fool an increasingly strong adversary. While previous methods only aim at obfuscating the sensitive information, we find that adding new random information in its place strengthens the provided privacy and provides better utility at any given level of privacy. The result is an approach that can provide stronger privatization on image data, and yet be preserving both the domain and the utility of the inputs, entirely independent of the downstream task.

Under review as a conference paper at ICLR 2021
ADVERSARIAL REPRESENTATION LEARNING FOR SYN- THETIC REPLACEMENT OF PRIVATE ATTRIBUTES
Anonymous authors Paper under double-blind review
ABSTRACT
Data privacy is an increasingly important aspect of many real-world big data analytics tasks. Data sources that contain sensitive information may have immense potential which could be unlocked using privacy enhancing transformations, but current methods often fail to produce convincing output. Furthermore, finding the right balance between privacy and utility is often a tricky trade-off. In this work, we propose a novel approach for data privatization, which involves two steps: in the first step. it removes the sensitive information, and in the second step, it replaces this information with an independent random sample. Our method builds adversarial representation learning which ensures strong privacy by training the model to fool an increasingly strong adversary. While previous methods only aim at obfuscating the sensitive information, in its place strengthens the provided privacy and provides better utility at any given level of privacy. The result is an approach that can provide stronger privatization on image data, and yet be preserving both the domain and the utility of the inputs, entirely independent of the downstream task.
find that adding new random information
INTRODUCTION
Increasing capacity and performance of modern machine learning models lead to increasing amounts of data required for training them (Goodfellow et al., 2016). However, collecting and using large datasets which may contain sensitive information about individuals is often impeded by increasingly strong privacy laws protecting individual rights, and the infeasibility of obtaining individual consent. Giving privacy guarantees on dataset may let us share data, while protecting the rights of individuals, and thus unlocking the large benefits for individuals and for society that big datasets can provide.
In this work, we propose a technique for selective obfuscation of image datasets. The aim is to provide the original data as detailed as possible while making it hard for an adversary to detect specific sensitive attributes. The proposed solution is agnostic to the downstream task, with the objective to make the data as private as possible given distortion constraint. This issue has previously been addressed using adversarial representation learning with some success: a filter model is trained to obfuscate sensitive information while adversary model is trained to recover the information (Edwards & Storkey, 2016). In the current work, we demonstrate that it is easier to hide sensitive information if you replace with something else: sample which independent from the input data.
Aside from the adversary module, our proposed solution includes two main components: one filter model that is trained to remove the sensitive attribute, and one generator model that inserts a synthetically generated new value for the sensitive attribute. The generated sensitive attribute entirely independent from the sensitive attribute in the original input image. Following body of work in privacy-related adversarial learning we evaluate the proposed model on faces from the CelebA dataset (Liu et al., 2015), and consider, for example, the smile or gender of a person to be the sensitive attribute. The smile is an attribute that carries interesting aspects in the transformations of a human face. The obvious change reside close to the mouth when person smiles, but also other subtle changes occur: eyelids tighten, dimples show and the skin wrinkles. The current work includes a thorough analysis of the dataset, including correlations of such features. These correlations make the task interesting and challenging, reflecting the real difficulty that may occur when anonymizing data. What is the right trade-off between preserving the utility as defined by allowing information about other attributes to remain, and removing the sensitive information?

---

Under review as a conference paper at ICLR 2021
In our setup, the adversary make an arbitrary number of queries to the model. For each query another sample will be produced from the distribution of the sensitive data, while keeping as much as possible of the non-sensitive information about the requested data point.
2
RELATED WORK
Privacy-preserving machine learning has been studied from a number of different angles. Some work assumes access privacy-preserving mechanism, such as bounding boxes for faces, and studies how to hide people's s identity by blurring (Oh et al., 2016a), removing (Orekondy et al., 2018) or generating the face of other people (Hukkelás et al., 2019) in their place. Other work assumes access to the utility-preservee preserving mechanism and proposes to obfuscate everything except what they want to retain (Alharbi et al., 2019). This raises the question: how do we find the pixels an image that need to be modified to preserve privacy with respect to some attribute?
Furthermore, Oh et (2016b) showed that blurring or removing the head of a person has a limited effect privacy. The finding is crucial; cannot rely modifications of image such as blurring or overpainting to achieve privacy. An adversarial set-up instead captures the signals that the adversary uses, and can attain stronger privacy. Adversarial learning is the process of training model to fool an adversary (Goodfellow et al., 2014). Both models are trained simultaneously, and become increasingly good at their respective task during training. This approach has been successfully used to learn image-to-image transformations (Isola et al., 2017; Choi et al., 2018). and synthesis of properties such facial expressions (Song et al., 2017: Tang et al., 2019). sivaacypreseeving adversarial representation learning utilize this paradigm to learn representations of data that hide sensitive information (Edwards & Storkey, 2016; Zhang al., 2018: Xie al., 2017: Beutel et al., 2017; Raval et al., 2017).
Bertran al. (2019) minimize the mutual information between the utility variable and the input image data conditioned on the learned representation. Roy & Boddeti (2019) maximize the entropy of the discriminator output rather than minimizing the log likelihood, which beneficial for stronger privacy. Osia et al. (2020) approached the problem using an information-bottlenes. Ren al. (2018), and Wang et al. (2019) learn transformations of video that respect a privacy budget while maintaining performance pose-ime invarintti face recognition. Similar to our work, their approach used adversarial learning to disentangle specific attributes in the data. Oh et al. (2017) trained model add a small amount of noise to the input to hide the identity of a person. Xiao et al. (2020) learn a representation from which it is hard reconstruct the original input, from which it is possible to predict predefined task. The method provides control over which attributes that is preserved, but no control over which attributes that are being censored. That is, it puts more emphasis on preserving utility than privacy, which is not always desired.
Wu et al. (201 8),
downstream task. Tran et al. (2018) proposed approach for
All of these, with the exception of Edwards & Storkey (2016) (see below), depend on knowing the downstream task labels. Our work has no such dependency: the data produced by our method is designed to be usable regardless of downstream task.
In Edwards & Storkey (2016), limited experiment is included which does not depend on the downstream task. In this experiment, they remove sensitive text which was overlaid on images, a task which is much simpler than the real-world problem considered in the current work. The overlaid text is independent of the underlying image, and therefore the solution does not require trade-off between utility and privacy which is the case in most real settings. Furthermore, we also replace the sensitive information with synthetic information which we show further strengthens the privacy.
Like in the current work, Huang et al. (2017, 2018 8) use adversarial learning to minimize the mutual information between the private attribute and the censored image under a distortion constraint. Our solution extends and improves upon these ideas with modular design consisting of filter that is trained to obfuscate the data, and generator that further enhances the privacy by adding new independently sampled synthetic information for the sensitive attributes.
2

---

Under review as a conference paper at ICLR 2021
3 PRIVACY-PRESERVVNG ADVERSARIAL REPRESENTATION LEARNING
In the current work, we propose novel solution for pellitt-preeevingg privacy-enhancing transforma- tions of data: we privacy-pres arivacy-preserving representation learning to obfuscate information in the imput data, and output results that retain the information and structure of the input.
3.1 PROBLEM SETTING
Generative adversarial privacy (GAP) (Huang et al., 2018) was proposed as method to provide privacy in images under distortion constraint, and will be used the baseline in the current work. In GAP, one assumes a joint distribution P(X,S, .) of public data points X and sensitive private attributes where $ is typically correlated with X. The authors define a privacy mechanism X' f((, Z1) where Zi is the source of noise or randomness in Let (X') be an adversary's 's prediction of the sensitive attribute S from the privatized data X' according decision rule hy. The performance of the adversary is thus measured by a loss function e ( z1)), 8) and the expected loss of the adversary with respect X, S and 21 is
Ly (f 21),
2.30p(2,5) (1mp(a))
where p(21) is the source of noise.
The privacy mechanism f will be trained to be privacy-preserving and urility-preserving. That is, should be hard for analyst infer S from X', but X' should be minimally distorted with respect to X. Huang et al. (2018) formulate this as a constrained minimax problem
min max- by
s.t.
[Ed(f 21), x)) S 61 2,8mp((z,)) sim=p(=1)
where the constant 6 2 defines the allowed distortion for the privatizer and d(-.-) -) is some distortion measure.
In the current work, f will be referred to as the filter since the purpose of is to filter the sensitive information from A potential limitation with this formulation is that it only obfuscates the sensitive information in which may make it obvious to the adversary that x' censored version of x. Instead, we propose to replace the sensitive information with new independent value s'.
3.2 OUR CONTRIBUTION
We extend the filter with a generator module g, defined as X" = (1) Z2) where S" denotes the random variable of the new synthetic value for the sensitive attribute. Z1 and Zz denote the sources of randomness in f and respectively. The discriminator hy is trained to predict s when the imput is real image, and to predict the "fake" output when the input comes from as in the learning setup in Salimans et al. (2016). The objective of the generator g( 8 ) is to generate new synthetic (independent) sensitive attribute s' z', that will fool the discriminator hy. We define the loss of the discriminator hg
Lggh (lg - (1), (2)), fake]] 2,3xp(2,8) 8 s'mp(a')
ElC,(h, x.
21.22--(21122
where p(21, ) is the source of noise, p(s') is the assumed distribution of the synthetic sensitive attributes s'. fake is the fake class, and lg the loss function. We formulate this constrained minimax problem
min max hy
- Leg hy)
s.t. 21), 22), s e2. 2.soo((z.s)
(4)
s'mpp(s")
where the constant é2 2 defines the allowed distortion for the generator.
3

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(c)
Figure Visualization of privatization mechanisms. A logistic regression model is fitted to the optimal decision boundary (blue). Minimizing the log likelihood (a) leads to a model that transforms points to the opposite category. Maximizing the entropy the adversary (b) leads to transformations that make the adversary uncertain (closer to the decision boundary). Our two-step approach (c) achieves a stronger privacy by first maximizing the entropy as in (b), and then transforming the image to random, new value for the sensitive attribute (purple arrows). Small grey arrows show the negative gradient of the loss function with regards to the input variable, and orange circles illustrates the distortion budget.
In Figure we show the difference between (a) minimizing log-likelihoon of the adversary, (b) maximizing entropy of the adversary, and (c) maximizing the entropy of the adversary and also synthetically replace the sensitive attribute with a random sample.
3.3 IMPLEMENTATION
Let D ((x,, 8, ) be dataset of samples (I;, 8:) which are assumed to be identically and independently distributed according to some unknown joint distribution S). We assume that the sensitive attribute is binary and takes values in {0. 1}. However, the proposed approach can easily be extended to categorical sensitive attributes.g , 5" Zz; 0g) using convolutional neural networks of the UNet (Ronneberger al., 2015) architecture parameterized by 0, and 0g. respectively. (See Appendix A.1 for details).
The discriminators h (X': and hg (X" (g)) are modeled using ResNet- 18 (He et al., 2016) and a modified version which we refer to as ResNet-10', respectively. The last fully connected layer has been replaced with two and three class output layer for each model, respectively.
As suggested by Roy & Boddeti (2019), we choose the filter discriminator loss b to be the the negative entropy. Intuitively, this leads to learning to make the adversary hy confused about the sensitive attribute rather than to make it certain the wrong value. For completeness, we also include experiments where é is the categorical cross-entropy, measure d is defined as the L2-norm, and p(s') is assumed to be the uniform distribution (1(0, 1}. The hyperparameters consist of the learning rate Ir, the quadratie penalty term coefficient X, the distortion constraint and the ($1,3, parameters to Adam (Kingma & Ba, 2014). Details of the training setup can be found Appendix A.3, and the full code is published on GitHub?.
are Eay and fg. The distortion
4 EXPERIMENTS
In this section we describe our experiments, the datasets used, and the evaluation metrics.
Synthetic data. We introduce and apply the method on synthetic dataset to illustrate the difference between optimizing the filter for log-likelihoood and entropy, and why adding the generator is
important. The synthetic data consists Dtrain = [(1, s I E R² and s E {0, 1}. drawn from two different normal distributions N (1) and N (H2, 02), where #r = [-1, and 0
ResNett 10 has the same setup as ResNet-18, 18, hut each of the "conv2_x" "conv3_x" _x", "conv4_x"" and "conv5_x"" x layers consists of only one block instead of two. http:///ghib..ommmmmmaauaaaaayag
4

---

Under review as a conference paper at ICLR 2021
[10.,, and Me = -1,, and d2 = [[0.7.0] (0...0][0.0.7]] 0.7]. Similarily, construct test set Diest = {(2,, } j=- and we let n = 400, 000 and m = 2,00 The points are classified according to which normal distribution they belong to, and we consider this the secret attribute, i.e, 0 if x ~ N M(1,, 01) and, = 1 if x ~ (12, 02). We sample s' ~ u((0,1)) 1)). We run the experiment with distortion constraints E {0.1, 0.5, 1.0, 1.5, 1.5, 2.0) and run each experiment five times.
CelebA. The CelebA dataset² (Liu et al., 2015) consists of 202,599 face images of size 218x178 pixels and 40 binary attribute annotations per image, such as age (old or young), gender, if the image blurry, if the person is bald, etc. The dataset has an official split into training set of 162,770 images, a validation set of 19,867 images and a test set of 19,962 images. We resize all images to 64x64 for the quantitative experiments, and to 128x 128 pixels for the qualitative experiments, and normalize all pixel values to the region [0, 1/. We use higher resolution for the qualitative results to make subtle visual changes more apparent.
Filtering and replacement of sensitive data. Let Dirain = [ 8; 8:1== be a set of training data where z; denotes facial image i and s; E {0,1] denotes the sensitive attribute. Further let Diest = {( J=1 be the held out test data. For the purpose of evaluation, we assume access to a number of utility attributes 46) E {0, 1 for each r and j E U. The following attributes provided with the dataset will be used in the CelebA experiments: gender, lipstick, age, high cheekbones, mouth slightly open, heavy makeup, smiling. In each experiment, one of these will be selected as the sensitive attribute. The rest will be considered the utility attributes, not used for training, but allow for utility score in the evaluation. This score shows how well non-sensitive attributes are preserved the transformation. We compute an average over fixed classifiers, trained to predict each respective utility attribute on the original training data, when evaluated on the censored data.
Hyperparameters. We train the models using Derain with Ir 0.0005, À = 105, (0.03, 0.02, 0.01, 0.005, 0.001}, and (3,, 32) = (0.9. 0.999). Let the training data censored by the filter be D/ {(1//, s;)]? where x = fx 2/ (1) fy), and by both the filter and the gener- ator be D'rain = {(=,', s:)} where x!¹ g(z,, s/,, =,(²);; 0g), R3 =z!²) ~ NO0 1), and sa (0, 1}. We do the same transformations to the test data and denote them D Plest and Diest respectively.
Computational requirements. Each experiment was performed on a Tesla V100 SXM2 32 GB in a DGX-1 machine. The training was restricted to 100 epochs which takes about 13 hours. We run each experiment with negative entropy loss three times, and present the average over these runs, and we run each experiment with log-likeldhood loss five times and present the average over these.
Evaluation. To evaluate the privacy loss for our method we train adversarial classifier predict the ground truth sensitive attribute given a training set of images censored by each privatization method, and then we evaluate each adversary on a test set of censored images. If an adversary can predict the ground truth sensitive attribute of censored images, that it has not seen before, with high accuracy, then the privacy loss is high. Let adu(s h())) denote an adversary trained on the censored training set to predict the ground truth attribute s given censored image fa(z) then the privacy loss is defined as
Privacy loss
Σ
1 [[ados))(()) = s].
(5)
[Dresel
To evaluate the utility score on the CelebA data, we use fixed classifiers that have been pre-trained on the original data predict a set of utility attributes that are considered non-sensitive. If these utility attributes are predictable with high accuracy after privatization of the image, then we consider the method to have a high utility score. Let fiz(u)==") [[=")) denote a fixed classifier trained on the original training set to predict the ground truth attribute u, then the utility score is defined as
(CelebA) Utility score
Σ
[(fin((()(((zz) = ul.)).
(6)
UU
Drestl
JEU
5

---

Under review as a conference paper at ICLR 2021
baseline
baseline (ontrogy)
(entropy)
0.8
Ubilty score
0.6
r-28
x-21
0.2
Privacy
Figure 2: The trade-off curve between privacy loss and utility score for the baseline (likelihood), the improved baseline (entropy) and our method the synthetic dataset. Upper left corner is better.
by drallocc
(a)
(b)
(c)
Figure 3: Results the synthetic dataset. The censored representations by (a) baseline with filter trained to minimize log- ligelikei of adversary, (b) the baseline filter trained to maximize entropy, and (c) filter trained to maximize entropy and then generating new synthetic attribute.
To evaluate the utility score on the synthetic data, we compute
(Synthetic) Utility score
Σ
|x - h(x),,
Emax (Deett
(2,5)EDiast
where we normalize with Gmar 2 to get distances into the range [o, and convert it to a similarity by subtracting the average over these distances from one for consistency in the trade-off off curves. Where we let h(x)) = f(r, 0y) to evaluate the baseline, and h(x)) = g((((x, 21; 0y), s z2; 0,)) to evaluate our method.
We then plot the utility score against the privacy loss for different distortion budgets and get trade-off curve. Similar evaluation method used in Roy & Boddeti (2019); Bertran et al. (2019). In particular, Roy & Boddeti (2019) plot accuracy of an adversary predicting the sensitive attribute against the accuracy of classifier predicting the target attribute.
RESULTS
In this section we present quantitative and qualitative results the facial image experiments.
Synthetic data. Figure 2 shows the trade-off between privacy loss and utility score for the ex- periment on the synthetic dataset. Our method consistently outperforms the baseline for all given distortion budgets. Figure 3 shows the original data (orange for s=0, and blue for s=1) together with the censored data for the three different transforms (red for baseline with likelihood, green for baseline with entropy, and purple for the proposed method). When minimizing log- likeliiid see clear clusters on opposite side of the decision boundary, when maximizing entropy we see that all points approach the decision boundary, and when adding the generator
that the points

---

Under review as a conference paper at ICLR 2021
Smiling
Male
duts Bag- -ikllihsoddd
Blg. likllhhod!
bascline e 0og (kolhood) (entropy)
baseline Blg (kelhood)
(entropy)
baseline (entropy)
70.0
baseline (entrogy)
75
Ubity score
67.5
65.0
70
62.5
60.0
x
Privacy
Privacy
Lipstick
Age
75
ours Dog (ia--ikkihhhdd)
-
ours Blgg sog-ikklihood
baseline (keihod))
baseline (keilholo)
(entropy)
(entropy)
baseline (intrapp)
70
score
75
85
0005
65
60
70
75
80
65
90
78
80
82
84
86
Privacy loss
Privacy loss.
Figure 4: Privacy vs. utility trade-off curve where the sensitive attribute smiling (top left), gender (top right), lipstick (bottom left), age (bottom right). Our approach with negative entropy loss consisitently outperforms all other approaches on the attributes explored, and our method with log-likelihood loss outperforms the baseline with log-likelihood loss on all explored attributes.
are mapped, random, into the cluster of orange points or blue points depending on the sampled synthetic sensitive attribute.
CelebA. Figure 4 shows the trade-off between privacy loss and utility score when evaluated on censored images. Our method consistently has higher utility any given level of privacy compared to the baseline. Remember: these are strong adversaries required to run tagged training data through the privacy mechanism to be able train. Additional results can be found Appendix A.2.
To further show that explicitly optimizing for privacy in the privatization mechanism is necessary have conducted similar experiment using StarGAN Choi al. (2018) randomly change the sensitive attribute in the image (results in suppl.). We then evaluate the censored images, which look very convincing to human, using adversarial classifiers. The adversaries can successfully detect the sensitive attributes with an accuracy of roughly 90%. For the weight Ayee of the cycle consistency loss we explored values (0, 5, 10,50), 50). We obtain similar scores when we exclude the filter part of our method and use only the generator part to censor the images (see Appendix A.2).
In Table 1 we present the results of evaluating the accuracy of f fiz(s)-) on the dataset {a/ Mm
where x g( s/,2(²) (2) is the image censored with our method and s is the new synthetic attribute uniformly sampled from {0, 1}. That is, we measure how often the classifier predict the new synthetic attribute sf when applied x. We can see that with 0.001 the method is on average able to fool the classifier 82.4% of the time for the smiling attribute, and this increases with larger distortion budget to success rate of 91.2% average with 0.05. The results are similar when the images have been censored with respect to the attributes gender, lipstick, and age, but these require a larger distortion budget.
7

---

Under review as a conference paper at ICLR 2021
Table 1: The success rate of method fool fixed classifier for the sensitive attributes smiling, gender, lipstick, and age. This was measured as the accuracy of fiz(sl-) for each censored attribute the censored data {(a s())= Higher is better. Average and standard deviation of five runs.
Dist.
Synthetic Gender
Smiling
Lipstick
Young
0,001
72.9 2.1
1.5
62.2 3.1
59.2
0.005
7.64 3.7
71.7 2.4
62.1 + 2.5
0.01
2.14
85.6
77.6
59.6 1.6
0.05
2.
90.3 + 4.3
90.6 4.2
67.7 2.1
NY
NY
RK
NL
Nk
NY
Figure 5: Qualitative results for the sensitive attribute smile. In the first four columns: 0.001, and in the last four columns: = 0.01. From top to bottom row: input image (I), censored image (rr),, censored image with synthetic non-smile (a" 0), censored image with synthetic smile (a" 8 = 1). The model is able to generate synthetic smiling attribute while maintaining much of the structure in the image. These images were generated from a model trained using 128x 128 pixels.
Figure shows, from the top row to the bottom row, the input image x, the censored image x', the censored image x" with the synthetic attribute s' = 0 (non- (non-nsmiile smiling), and the censored image x" with the synthetic attribute s' (smiling). value of 0.001 is used in the first four columns, and = 0.01 in the last four columns. The images censored by our method look sharper and it is less obvious that they are censored. We can see that the method convincingly generates non-smiling faces and smiling faces while most of the other parts of the image is intact. These images sampled from models trained on images of 128x128 pixels resolution. See Figure Appendix A.2 for corresponding samples the same input images, but using gender as the sensitive attribute.
DISCUSSION
In Figure 1, we show how privatization mechanism is affected by adversarial setup and its training criterion. Minimizing the log likelihood the adversary can be interpreted as attempting to make the adversary certain of the wrong value of the sensitive attribute and may lead to transforming data with a certain sensitive value to be transformed to similar outputs. Following the gradient of the entropy loss instead leads to the adversary becoming less certain and thus leads more privacy. Using this approach to remove sensitive information, and then adding random information makes an even stronger privacy, take another step adding randomness to the output. This intuition
8

---

Under review as a conference paper at ICLR 2021
is confirmed in the experimental results the synthetic data, demonstrated by Figure 3. Our approach successfully transforms the datapoints into a distribution which is nearly indistinguishable from the input distribution, and where datapoints with the different sensitive values are completely mixed.
Our method consistently outperforms the baseline, ensuring a higher level of privacy while main- taining more of the useful information (see Figure 4). For all sensitive attributes that we consider and at nearly all given distortion budgets (e {0.02, 0.01, 0.005, 0.001}) we observe both higher privacy and a higher utility for our method. For some of the attributes (gender, lipstick), at extreme distortions (c = 0.03) our method has utility score which not better than the baseline, but still provide considerably better privacy. Furthermore, we can observe that using the entropy loss function for the filter benefits both the baseline and our method.
This shows that our method makes it more difficult for the adversary to see through the privatization step at each given distortion budget. To show the effect of the filter we have conducted experiments where we only use the generator to privatize the images (see Appendix A.2), which does not seem to provide any privacy.
A generator without the filter may learn to either pass the original image without modification (when = 8) or to transform the image into a random other value (s' * 8). If the transformed image is indistinguishable from real image this is not problem, but otherwise privatization by detecting if the image is real or not. The filter step mitigates this by always removing the sensitive data in the image, forcing the generator synthesize new data. Since the censored image is now guaranteed to be synthetic, we are no longer susceptible the simple real/fake attack.
easily reverse the
Similarily, we conduct experiments where we use StarGAN censor the images, but observe no privacy when using this standard attribute manipulation method, A reason could be that the cyclic consistency loss term actually encourage image transformations that are easily invertible given the original attribute, which is not desirable in a privacy setting. This motivates our approach which is explicit about the privacy objective.
In Table I, we see that the fixed smile classifier is fooled by our privatization mechanism in 82.4% to 91.2% of the data points in the test set (depending on the distortion c). These results indicate that it may be harder for an adversarially trained classifier to predict the sensitive attribute when it has been replaced with something else, as compared simply removed. We assume that this is due to the added variability in the data. Or intuitively: it is easier to "blend in" with other images that have similar demonstrations of smiles.
The fact that many important attributes in facial images correlate leads to the reflection that disentan- gling the underlying factors of variation is not entirely possible. For example, in CelebA, lipstick is highly correlated with female gender. This means that if we want to hide all information about the presence of lipstick we also need to hide the gender (and other correlating attributes). This problem is further analysed in Table 3 in Appendix A.2.
A strength of our method is that it is domain- domain-preserving preserving, this allows a utility provider to use the censored image in existing algorithms without modifications. Since the method also preserves utility this may also allow stacking the privatization mechanism to censor multiple attributes in an image.
CONCLUSIONS
In this work we have presented strong data which is learned using an adversarial setup. While previous work on adversarial representation learning has focused on removing information from representation, our approach extends on this and can also generate new information in its place that looks realistic and gives further privacy properties compared to the baseline. We evaluate our method using adversarially trained classifiers, and our results show that not only do we provide stronger privacy with regards to sensitive attributes, but we also preserve non non-sensitive attributes of the image at a higher rate. The results show that the synthetically added attribute gives stronger privacy properties and helps fooling the adversary in the challenging setting where the adversary is allowed to be trained using the output of the privacy mechanism.
transformation mechanism for image

---

Under review as a conference paper at ICLR 2021
REFERENCES
Rawan Alharbi, Mariam Tolba, Lucia C. Petito, Josiah Hester, and Nabil Alshurafa. To mask or not to mask? balancing privacy with visual confirmation utility in activity-oriented wearable cameras. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 3(3), September 2019. doi: 10.1145/3351230. URL https: wtg.i.orooo...... 1145/3331222
Martin Bertran, Natalia Martinez, Afroditi Papadaki, Qiang Qiu, Miguel Rodrigues, Galen Reeves. and Guillermo Sapiro. Adversarially learned representations for information obfuscation and inference. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 614-623, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Alex Beutel, Jilin Chen, Zhe Zhao, and Ed H Chi. Data decisions and theoretical implications when adversarially learning fair representations. arXiv preprint arXiv: (o::::707.000000 2017.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung Woo Ha, Sunghun Kim, and Jaegul Choo. StarGAN: Unified Generative Adversarial Networks for Multi-domain mmage-to-Image Translation. Proceed- ings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 1189-8797, 2018. ISSN 10636919. doi: 10.1 109/CVPR.2018.00
Harrison Edwards and Amos J. Storkey. Censoring representations with an adversary. 4th International Conference Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings. 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- tion processing systems, pp. 2672-2680, 2014.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, June 2016. doi: 10.1 D0.1009/CPR.2001
C. Huang, P. Kairouz, and L. Sankar. Generative adversarial privacy: A data-driven approach to information-theorette privacy. In 2018 52nd Asilomar Conference on Signals, Systems, and Computers, pp. 2162-2166, Oct 2018. 8.
doi: 10. 10.1109// 109///CCCC...11. 8645553..
Chong Huang, Peter Kairouz, Xiao Chen, Lalitha Sankar, and Ram Rajagopal. Context-aware generative adversarial privacy. Entropy, 19(12), 2017. ISSN 1099-4300. doi: 10.339 URL https::ppp//..mc
www
Håkon Hukkelås, Rudolf Mester, and Frank Lindseth. Deepprivacy: A generative adversarial network for face anonymization. In George Bebis, Richard Boyle, Bahram Parvin, Darko Koracin, Daniela Ushizima, Sek Chai, Shinjiro Sueda, Xin Lin, Aidong Lu, Daniel Thalmann, Chaoli Wang, and Panpan Xu (eds.). Advances in Visual Computing, pp. 565-578, Cham, 2019. Springer International Publishing. ISBN
P. Isola, J. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967-5976, July 2017. doi: 10. 101100/....222.....2.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL http: // //arxiv.. org/ abs 1412.. 1...666 6980. cite arxiv:1412, 11441...680ommmenl Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Seong Joon Oh, Rodrigo Benenson, Mario Fritz, and Bernt Schiele. Faceless person recognition: Privacy implications in social media. In Bastian Leibe, Jiri Matas, Nicu Sebe. and Max Welling (eds.), Computer Vision - ECCV 2016, pp. 19-35, Cham, 2016a. Springer International Publishing.
ISBN 978-3-319-46487-9,
10

---

Under review as a conference paper at ICLR 2021
Scong Joon Oh, Rodrigo Benenson, Mario Fritz, and Bernt Schiele. Faceless person recognition: Privacy implications in social media. In European Conference on Computer Vision, pp. 19-35. Springer, 2016b.
Scong Joon Oh, Mario Fritz, and Bernt Schiele. Adversarial image perturbation for privacy protection game theory perspective. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1491-1500. IEEE, 2017.
Tribhuvanesh Orekondy, Mario Fritz, and Bernt Schiele. Connecting pixels to privacy and utility: Automatic redaction of private information in images. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2018.
S. A. Osia, A. Taheri, A. S. Shamsabadi, K. Katevas, H. Haddadi, and H. R. Rabiee. Deep private- feature extraction. IEEE Transactions Knowledge and Data Engineering, 32(1):54-66, Jan 2020. ISSN 2326-3865. doi: 10.1 200.0//TKKEEE212.888888888
Nisarg Raval, Ashwin Machanavaijhala, and Landon P Cox. Protecting visual secrets using adversarial nets. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 11329-1332 IEEE, 2017.
Zhongzheng Ren, Yong Jae Lee, and Michael S. Ryoo. Learning to anonymize faces for privacy preserving action detection. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision ECCV 2018, pp. 639-655, Cham, 2018. Springer International Publishing. ISBN 978-3-030-01246-5.
O. Ronneberger, P.Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI), volume 9351 of LNCS, pp. 234-241, Springer, 2015, (available arXiv:1505.0493 [cs.CV]).
Proteek C. Roy and Vishnu N. Boddeti. Mitigating information leakage in image representations: A maximum entropy approach. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, Xi Chen, and Xi Chen. Improved techniques for training gans. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp- 2234-2242. Curran Associates, Inc., 2016.
Lingxiao Song, Zhihe Lu, Ran He, Zhenan Sun, and Tieniu Tan. Geometry guided adversarial facial expression synthesis. CoRR, abs/ abs/1711..3344, 2017. URL http: //tp:://aaivvvv wwwp:/iii.rrrr.oooooo abs /1712. 03474.
Hao Tang. Dan Xu, Gaowen Liu, Wei Wang, Nicu Sebe, and Yan Yan. Cycle in cycle generative adversarial networks for syypoint-guided image generation. In Proceedings of the 27th ACM International Conference on Multimedia, MM '19, pp. 2052-2060, New York, NY, USA, 2019. Association for Computing Machinery, ISBN 978145036889.. doi: 001144/33430311333500 3350980.
URL https::///// /ooi orgg10..///oooooooooooooooo..1 1141
Luan Quoc Tran, Xi Yin, and Xiaoming Liu. Representation learning by rotating your faces. JEEE Iransactions on patlern analysis and machine intelligence, 2018.
Haotao Wang, Zhenyu Wu, Zhangyang Wang, Zhaowen Wang, and Hailin Jin. Privacy-preserving deep visual recognition: An adversarial learning framework and A new dataset. CoRR, abs/1906.05675, 2019. URL http: /ttp/a/aivvvv//..////a/.......wa org/ abs 190. 05675.
Zhenyu Wu, Zhangyang Wang, Zhaowen Wang, and Hailin Jin. Towards privacy-preseryy visual recognition via adversarial training: A pilot study. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (eds.), Computer Vision - ECCV 2018, pp. 627-645, Cham, 2018.
Springer International Publishing. ISBN 978-3-030-01270-00

---

Under review as a conference paper at ICLR 2021
Taihong Xiao, Yi-Hsuan Tsai, Kihyuk Sohn, Manmohan Chandraker, and Ming-Hsuan Hsuan Yang. Ad- and task- task-oriented representations. The Thirty-Forr Fourth AAAI Conference Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pp. 12434-12441, AAAI Press, 2020. URL ttttp:://aaa..orgoooop.. org.////exe//////ee...... http/innxxx.pppp whtpppppppaa AAAI/ article/view/63 6930.
versarial learning
Qizhe Xie, Zihang Dai, Yulun Du, Eduard Hovy, and Graham Neubig. Controllable invariance through adversarial feature learning. In Advances in Neural Information Processing Systems, pp. 585-596, 2017.
Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. Mitigating unwanted biases with adversarial learning. In Proceedings of the 2018 AAAI/ACM Conference AI, Ethics, and Society, pp. 335- 340. ACM, 2018.
12

---

Under review as a conference paper at ICLR 2021
Filter
Generator
0
Transter
Assign
Figure 6: Overview of the training setup (left) and the network architecture used the filter and the generator (right).
A APPENDIX
A.I ADDITIONAL DETAILS
An overview of the setup can be seen in Figure 6. We use the UNet Ronneberger et al. (2015). illustrated on the right in Figure 6, architecture for both the filter and the generator. The orange blocks are convolution blocks each of which, except for the last block, consist of convolution layer, batch normalization layer and a rectified linear activation unit, repeated twice in that order. The number of output channels of the convolution layers in each block has been noted in the figure. The last convolution block with a 3 channel output (the RGB image) consists of only a single convolutional layer followed by a sigmoid activation. The green blocks denote either kernel size of two and stride of two if marked with "/2" or a nearest neighbor upsampling by a factor of two if marked with "2x". The blue block denotes embedding layer, which takes input the categorical value of the sensitive attribute and outputs a dense embedding of 128 dimensions. is then followed by linear projection and a reshaping to match the spatial dimensions of the output of the convolution block to which it is concatenated, but with single channel. The same type of linear projection applied on the 1024 dimensional noise vector input, but this projection and reshaping matches both the spatial and channel dimensions of the output of the convolutional block to which concatenated. Concatenation both cases done along the channel dimension.
pooling layer with
A.2 ADDITIONAL RESULTS
The results in Table 2 show that using the filter together with the generator (g f) give stronger privatization guarantee than using only the generator (without the filter) or only the baseline (the filter without the generator).
Table 2: The results of evaluating the adversarially trained classifiers (sl-) on the held out test data censored with the baseline, only the generator, and our method for varying distortion for the smiling attribute. Closer to 50 means more privacy. Only using the generator does not privide any privacy, but using it together with the filter provides the best privacy. Lower is better.
Dist.
Adversarial smiling
baseline (f)
generator (g)
ours (g o f)
0.001
89.3
90.7
88.7
0.005
89.9
91.2
85.9
0.01
89.8
91.5
83,2
0.05
69.2
78.7
54.1
In Figure 7 show additional qualitative results when the attribtue gender is considered sensitive.
Table 3 shows the correlations between classifier predictions on a pair attributes when one attribute has been synthetically replaced.
13

---

Under review as a conference paper at ICLR 2021
NY
NY
YP
Figure 7: Qualitative results for the sensitive attribute gender. In the first four columns: e 0.001, and in the last four columns: 0.01. From top to bottom row: input image (x), censored image (x'), censored image with synthetic female gender (x", 0), censored image with synthetic male gender (x", s' = 1). The model is able to generate a synthetic gender while maintaining much of the structure in the image. These images were generated from a model trained using 128x128 pixels.
Table 3: The value of each cell denotes the Pearson's 's correlation coefficient between predictions from a classifier trained to predict the row attribute and a classifier trained to predict the column attribute, given that the column attribute has been censored.
Smiling
Gender
Lipstick
Young
Smiling
1.00
-0.04
0.08
-0.06
Gender
-0,07
1.00
-0.44
-0.21
Lipstick
0.14
-0.30
1.00
0.26
Young
0.05
-0.11
0.23
1.00
High Cheekbones
0.14
-0.07
0.15
-0.01
Mouth Open
0.04
0.00
0.03
-0.02
Heavy Makeup
0.12
-0.24
0.47
0.22
For example, in the CelebA dataset lipstick highly correlated with female gender. This means that we want to hide all information about whether or not the person is wearing lipstick we also need to hide its gender (and other correlating attributes). This problem can seen Table where changing whether or not person is wearing lipstick correlates with changes of gender.
The question is: if we censor an attribute in an image, how does that correlate with changes of other attributes in the image? In the lipstick column of Table 3 have censored the attribute lipstick. We then make predictions on whether or not the person in the censored image is wearing lipstick, and compute the correlation between these predictions and predictions for the attributes for each row. We can see that changes in lipstick correlate negatively with changes in gender and positively with makeup. This highlights the problem of disentangling these underlying factors of variation.
Table 4 shows the Pearson correlations between the smiling attribute and 37 other attributes in the CelebA dataset.
14

---

Under review as a conference paper at ICLR 2021
Table 4: Pearson correlation coefficient between the smiling attribute and 37 other attributes in the CelebA training dataset, ordered from high to low absolute correlation.
Attribute
Correlation
High cheekbones
0.68
Mouth slightly open
0.53
Rosy cheeks
0.22
Oval face
0.21
Wearing lipstick
0.18
Heavy makeup
0.18
Wearing carrings
0.17
Attractive
0.15
Gender
-0.14
Bags under eyes
0.11
No beard
0.11
Big nose
0.10
Double chin
0.10
Arched eyebrows
0.09
Wearing necklace
0.09
Blond hair
0.09
Narrow eyes
0.08
Sideburns
-0.08
Wavy hair
0,08
Mustache
-0.07
Pale skin
-0,07
Five-o' clock shadow
-0.07
Goatee
-0.07
Blurry
-0.06
Wearing hat
-0.06
Bangs
0.05
Chubby
0.04
Eyeglasses
-0.04
Pointy nose
0.04
Brown hair
0.02
Receding hairline
0.02
Bald
0,01
Big lips
0.01
Gray hair
0.01
Straight hair
0.01
Black hair
-0.00
Bushy eyebrows
-0.00
Wearing necktie
-0.00
A.3 TRAINING SETUP
Algorithm I demonstrates the training setup for method. Note that when & is negative entropy it does not depend on the - s; argument, but simply computed from the output of the discriminator hy. However, when & is cross entropy optimize the filter such that the diseriminator h, is fooled into predicting the complement class s. This means that we need to assume s to binary attribute when using this loss. When using the negative entropy do not need to make this assumption, which a strength of the proposed method.
15

---

Under review as a conference paper at ICLR 2021
Algorithm
input: Ir ,1,,,
(1, 62 f-c
repeat
Draw m samples uniformly at random from the dataset
(1),8)),.--.. (Imm 8m) D
Draw m samples from the noise distribution
(z( ) (zm, _m p(2(1),20
Draw m samples from the synthetic distribution si,-..,. Kmm p(s')
Compute censored and synthetic data
21 ....... 2 fo, (#1, 21 (2),, ......, fo, (Im, 2m)
z'..... ..... xn Compute filter and generator losses
go, (z/, 8/,, =1²)), ).... 00, (I'm s'm (2)
1
,0g)
1=1
m
+ A Amax
Σ
d((()a))) - ., 0)2
m
m
(g)) l (
m
+ Amax
d(<", 0)2 1=1
Update filter and generator parameters
0, A (0,) Ir, $1, (2)
0 + Adam (0,);
(0,;;;r. Ir. 12)
Compute discriminator losses
4 f 1 Em l,, (2/3 Φ5), si)
oo (hy (x7: 0g), fake)
102
(hg (z:: og), s:)
m
Update discriminator parameters
Φ <- Adam Adam(d, (0,,) Ir, 31, 32) O - Adam 4 (0,) br 31, 32)
until stopping criterion return 0,,0g. Gg Φ, 0g
16


Reviews:
[{"review_id": "uv2zQwKnDIr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a framework to privatize sensitive attributes of data using adversarial representation learning. The proposed method consists of a “filter” that removes the sensitive attribute from the data representation, and a “generator” that replaces the removed sensitive attribute with a randomly sampled synthetic value. The authors argue that the second step done by the generator enhances privacy, and use experiments on real image data to verify their method and compare it with a baseline.\n\nOne important aspect of this work is that the privatization scheme is done regardless of the downstream learning task. The authors argue that all prior work (except for one) doesn’t have this property.\n\nSome comments and questions:\n\n-Can you provide more evidence as to why replacing the sensitive information with something else (which is what the generator does) is useful? In the third paragraph of introduction you mention this as an “assumption” but is this actually an assumption or observation?\n\n-In your experiments I couldn’t find what the baseline is? Is it just the Huang et al. 2018 approach which is the same method without the generator? Are there any other methods you could compare your method with?\n\n-In your experiments, why is it that in some cases when epsilon is increased, both privacy loss and utility score gets better? For e.g. see Age plot in Figure 1, epsilon=0.001 and epsilon=0.005 for the green curve. Shouldn’t we always have that increasing epsilon helps the privacy loss but hurts the utility score?\n\n-In Eq. 5 & 6, Z_2 is probably missing inside the parentheses for g.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "nMEIRPNN51N", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis work proposes a privacy-preserving transformation mechanism based on adversarial representation learning. The proposed work is an extension of generative adversarial privacy (GAP). It replaces the filter with a generator that replaces the sensitive information with synthetic data instead of censoring it. \n\nReason for score:\nThe contribution of the paper is incremental, and the experimental results are limited to one dataset and one baseline. More experiments need to be done to evaluate the consistency of the results and to study the generalizability of this work across various datasets.\n\nMore detailed comments:\n-         In related work, please add more recent works in adversarial representation learning and explain the contribution of this work compared to the existing ones. One example includes:\nAdversarial Learning of Privacy-Preserving and Task-Oriented Representations by Xiao et al.\n-         It would be better if the authors theoretically prove the privacy-preserving of the proposed method.\n-         This work is only evaluated on one dataset, also it is only compared to one baseline. The authors are expected to compare the proposed method with several recent works (some of them already mentioned in related work) and show how this method advances the state-of-the-art in this area.\n-         Please explain how this work can scale on larger data or more high-quality images? More discussion on the efficiency of the method is also necessary compared to other existing techniques.\n-         It would be better if the authors showed how the model preserves utility when generating  multiple synthetic attributes especially in cases where two or more attributes are highly correlated (e.g., heavy makeup and lipstick or mouth open and smiling, etc.)", "extra_scores": {}, "extra_text": {}}, {"review_id": "d-zz7YO5FY3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: The paper presents a privacy-preserving transformation technique for image data. The main idea is to use adversarial representation learning to obfuscate sensitive attributes. \n\nStrengths:\n1) The paper considers an important problem of preserving privacy on images and presents a heuristic approach to address this problem.\n2) The idea of using adversarial representation learning for this problem seems reasonable. The prior work on this topic has  focused on removing information from representation, and the paper extends this to also generating new information.\n\nConcerns:\n1) The paper feels incremental; the idea utilized seems like a simple extension to the Generative adversarial privacy idea of Huang et al. \n2) The evaluation feels a little strange. The authors consider a set of attributes gender, lipstick, age, high cheekbones, mouth slightly open, heavy makeup, smiling (and split them as sensitive and utility). I did not follow the real motivation of using these set of attributes, why are they sensitive? A better dataset (usecase) and/or choice of attributes would have helped making a stronger case. Being an experimental paper, a stronger evaluation (say on multiple datasets) might also help.\n\nOverall, while I feel that the problem is interesting, the paper presents just too little in terms of its technical contribution.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a heuristic for removing privacy sensitive attributes and replacing them with sythetically generated ones. The technique is closely related to an existing work and, as pointed out in the reviews, the experimental evaluation is insufficient for properly evaluating the approach.
Decision: Reject