MaSS: Multi-attribute Selective Suppression
Authors: Chun-Fu Chen
Shaohan Hu
Zhonghao Shi
Prateek Gulati
Bill Moriarty
Marco Pistoia
Vincenzo Piuri
Pierangela Samarati
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-attribute
GAN
Attribute Suppression

Abstract: The recent rapid advances in the development and deployment of machine learning technologies largely depend on the vast richness of data available today, in terms of both the quantity and the rich content contained within. For example, biometric data such as images and voices could reveal people's attributes like age, gender, sentiment, and origin, whereas location/motion data could be used to infer people's activity levels, transportation modes, and life habits. Along with the new services and applications enabled by such technological advances, various governmental policies are put in place to regulate such data usage and protect people's privacy and rights. As a result, data owners often opt for simple data obfuscation (e.g., blur people's faces in images) or withholding data altogether, which leads to severe data quality degradation and greatly limits the data's potential utility.
Aiming for a sophisticated mechanism which gives data owners fine-grained control while retaining the maximal degree of data utility, we propose Multi-attribute Selective Suppression, or MaSS, a general framework for performing precisely targeted data surgery to simultaneously suppress any selected set of attributes while preserving the rest for downstream machine learning tasks. MaSS learns a data modifier through adversarial games between two sets of networks, where one is aimed at suppressing selected attributes, and the other ensures the retention of the rest of the attributes via general contrastive loss as well as explicit classification metrics. We carried out an extensive evaluation of our proposed method using multiple datasets from different domains including facial images, voice audio, and video clips, and obtained highly promising results in MaSS' generalizability and capability of drastically suppressing targeted attributes (e.g., reducing inference on such attributes to random guess) while imposing virtually no impact on the data's usability in other downstream ML tasks.

Under review as a conference paper at ICLR 2023
MASS: MULTI-ATTRIBUTE SELECTIVE SUPPRESSION
Anonymous authors
Paper under double-blind review
ABSTRACT
The recent rapid advances in machine learning technologies largely depend on the vast richness of data available today. Along with the new services and appli- cations enabled by those machine learning models, various governmental policies are put in place to regulate such data usage and protect people's privacy and rights. As a result, data owners often opt for simple data obfuscation (e.g., blur people's s faces in images) or withholding data altogether, which leads to severe data quality degradation and greatly limits the data's potential utility. Aiming for a sophisti- cated mechanism which gives data owners fine-grained control while retaining the maximal degree of data utility, We propose Multi-atrribute Selective Suppression. MaSS, general framework for performing precisely targeted data surgery simultaneously suppress any selected set attributes while preserving the rest for downstream machine learning tasks. MaSS learns data modifier through ad- versarial games between two sets of networks, where one is aimed at suppressing selected attributes, and the other ensures the retention of the rest of the attributes. We carried out an extensive evaluation of our proposed method using multiple datasets from different domains including facial images, voice audio, and video clips, and obtained highly promising results in MaSS' generalizability and ca- pability of drastically suppressing targeted attributes while imposing virtually no impact on the data's usability in other downstream machine learning tasks.
INTRODUCTION
The recent rapid advances in machine learning (ML) can be largely attributed to powerful comput- ing infrastructures as well as the availability of large-scale datasets, such as ImageNer1K (Deng let al. 2009) for computer vision, WMT (Foundation) 2019) for neural machine translation, and riLighh (Kahn et al. 2020) for speech recognition. Studies have shown that MI. models trained on large-scale datasets can usually prove effective in many additional downstream tasks (Brown et al.)
2020). On the other hand, ethical concerns have been raised surrounding proper data usage in issues like data privacy (Liu et al., 202T), data minimization (Goldsteen et al., 2021), etc. Therefore, there are more data available and can be used without worrying whether not the data is handled properly, the ML models can be further improved by more data and help the ML. community to advance many domains.
Attempting to balance between model performance and proper data usages, a common approach usually taken is to simply modify the data to remove its "sensitive" attributes, and experimentally demonstrate that the targeted sensitive attributes are indeed removed. What's crucially important but usually omitted here, however, is the preservation of the "total utility" of the data, because the suppression operation oftentimes also negatively impact, or even completely destroy, the other "non-sensitive attributes, hence greatly damaging the dataset's potential future utility. For example, DeepPrivacy (Hukkelás et al., 2019) is able to demonstrate its privacy protection capability, but the modified data it produces can no longer be utilized for additional downstream tasks like sentiment analysis, age detection, gender classification. Since data is of the main driving forces for the rapid advancement of machine learning research, we argue that the ideal scenario would be to have the flexibility of selecting an arbitrary set of attributes and only suppressing them while leaving all the other attributes completely intact. In this way, the community could unleash the potential utility of the modified data develop more advanced algorithms.
Towards this exact goal, we present Multi antti-e Selective Suppression (or MaSS) in this paper, to enable such capability of precise attributes suppression for multi-attribute datasets. The high-level

---

Under review as a conference paper at ICLR 2023
Model Attr. 0
Attr.
Attr, Attr. Attr.
Attr.
MaSS Suppress Attr, O
Model Attr. I
Attr.
Original Daraser
Transformed Datasct
Model
Attr.
Figure 1: MaSS is able to precisely target any selected attributes in a multi-attribute dataset for suppression while leaving the rest of the attributes intact for any potential downstream ML-based analytic tasks. For example as illustrated by the diagram, when operating on the original multi- attribute dataset and configured to suppress Attr. O, MaSS able to transform the dataset such that the model for detecting Attr. O is unable to reliably detect Attr. 0 from the transformed data, while the models for Attr. and are not affected.
objective of MaSS is also illustrated in Figure 1 where MaSS is configured to suppress Attr. 0 without knowing in advance that Attr. and 2 will be used for downstream tasks. After the data transformation performed by MaSS, Attr. 0 becomes suppressed, nondetectable by its corresponding machine learning model, but at the same time, Attr. and 2 are left intact, and still can be extracted by their corresponding MI. models. As a concrete example, suppose we are working with facial- image dataset which contains attributes like age, gender, and sentiment, where, let us assume, age and gender are considered sensitive. Then, MaSS would transform this facial-image dataset such that age and gender information could no longer be not be inferred by the corresponding ML models, whereas sentiment information could still be extracted from the transformed data.
The contributions of our work are threefolds,
1. We propose the novel MaSS framework to enable the powerful flexibility of precise sup- pression of arbitrary, selective data attributes.
2. We employ multiple learning mechanisms in MaSS to enable its attribute-specific as well as generic feature preservation capabilities, which help it achieve satisfactory data utility protection both with and without the prior knowledge about downstream tasks.
3. We thoroughly validate MaSS using wide range of multi-attribute datasets, including image, audio, and video. All our results demonstrate MaSS' strong performance in its intended selective attribute suppression and preservation.
2 RELATED WORKS
Data Privacy. A large body of work has studied methods applying generative adversarial networks (GANs) to generate and modify facial features in images, so these identity-related sensitive features can be de-identified. DecpPrivacy (Hukkelăs et al. 2019) proposed to use a conditional generative adversarial network to generate realistic anonymized faces, while considering the existing back- ground and a sparse pose annotation. To further ensure the face anonymization using GAN-based methods, CIAGAN (Maximov et al. 2020) proposed identity control discriminator to control which fake identity is used in the anonymizaiton process by introducing an identity control vector. Instead of generating the entire faces for anonymization. Liet proposed apply conditional GAN to only identify and modify the five identity- sensitive attributes. To also enable face anonymization with the selected semantic attributes manipulation, PI-Net (Chen al. 2021) proposed generate realistic looking faces with the selected attributes preserved. The above works usually focuses on suppression only while the future data utilities are not considered. Our approach not only suppresses the attributes but also preserves the data utility concurrently. On the other hand, SPAct tried to sup- video through contrastive learning while preserving the utility for
press the multiple attributes action recognition; however, their approach lacks the flexibility to handle individual attributes but can only process all attributes at once and limits to the action recognition dataset; while our method is fully configurable and validated in different data domains. Moriarty et al. proposed the method to suppress the biometric information while preserving its utility; however, their approach requires the information of downstream task while method does not.
2

---

Under review as a conference paper at ICLR 2023
Multi- Learning. Data could contain multiple attributes. For recognition, using one model for each attribute might provide good recognition accuracy but might not scale. Multi- attribute learning aims for resolving this problem by learning underlying relationship among at- tributes such that the model can adaptively handle different attributes (Huang et al. 2018 Cheng et al. 2018). In this paper, we validated our method with the single-atriiute models. The multi- attribute model can also be used in our method by replacing multiple single-attribute models.
Dataset distillation and condensation aim to create smaller version of large dataset which can be used to train a model whose performance can be as good as training on the original large dataset. Therefore, training can be relatively quicker, c.g., Neural Architecture Search (NAS) methods require lot of iterations of whole dataset to find out the best model. Otherwise, NAS usually needs to use a proxy dataset/model for the approximation results from large dataset/model (Zhao et al. 2021 Cazenavette et al. 2022 Wang et al., 2022). On the other hand, a recent work shows the condensed dataset also conceals some attributes from the original data (Dong et al., 2022) but still remains effective for the original task. In contrast to these approaches, our proposed method tries to keep the truthfulness of data much as possible, do not reduce the amount of data. Moreover, our method is designed to preserve the generic features rather than the specific task, which could be covered by the generic feature preserved.
Dataset InstilattoonCCndeensatiion
SeffssunprriieddCnttasttieee Learning. Self-supervised learning has been explored by the com- munity to allow a machine learning model to learn a good data representation by designing pretext tasks instead of human annotations (Feng et al. 2019 or contrastive learning which tries to maxi- mize the agreement between positive pairs (Chen et al. 2020 He et al., 2020 Grill et al. 2020). or clustering-based methods generate pseudo labels for data (Caron et al. 2018)2 or mask au- toencoder to predict the masked patches by the remaining patches (He et al. 20022. Then, they show that the feature representations are usually good for many different downstream tasks. Our goal is to keep much information when suppressing the selected attributes without having prior knowledge; thus, self-supervised and contrastive learning methods facilitate our requirement to extract generic features without label information from downstream tasks.
PROPOSED METHOD
3.1 PROBLEM DEFINITION
Consider a multi-attribute dataset X with size N, and the set A of different attributes, where each data point x's value for attribute a E A is ax- Each of the different attributes can be learned by inferencing on X, hence our objective to transform X to suppress any selective subset attributes S E A such that no attributes in S can be reliably inferred from the transformed dataset X'. At the same time, we need to make sure the rest of the attributes inferred from X'.
\ S are preserved and can still be
In practice, when a data owner would like to transform their data, we assume the subset 5 of at- tributes targeted for suppression is always predetermined. However, it is not always known in ad- vance what the entire set A of attributes are, and consequently which set R of attributes need to be preserved. Therefore, for generalizability, we consider the case of an unknown R at the time of data transformation. Of course, R happens to be given, our solution needs to be able to take advantage of this extra information as well.
3.2 PROPOSED FRAMEWORK
We propose Generative Adversarial Network (GAN)-based solution in tackling the data transfor- mation problem. Our framework consists of three major components: a Data Modifier, Suppres- sion Branch, and a Preservation Branch, as depicted in Fig. The data modifier is the generator while both branches are the discriminators in the GAN framework. The data modifier tries to gen- erate new data such that the similarity between original data and modified data are maximized and minimized via the suppression branch and the preservation branch, respectively. In nutshell, the data modifier learns transformation that is to be applied to the original data vectors, where the learned transformation is jointly regularized by both the suppression and preservation branches to
3

---

Under review as a conference paper at ICLR 2023
Supression
Preservation Branch
Data Modifier
Reconstruction
Attribute Agnastic Less
1c Attibxue-Speclfi Similiriyy Loss
Sprebtte-Spectie
Predictinn Entropy Loes
Loss
Original
Similiarty Loss
Back-proe bade-poaaagtte the losses to trann Data Modilie
Figure 2: Overview of MaSS. MaSS contains three components: the data modifier, the suppression branch, and the preservation branch. The data modifier G is trained by optimizing the losses both the suppression and the preservation branches.
ensure all targeted attributes in S are indeed suppressed in the transformed data, while all other tributes, explicitly specified or not, are left intact much as possible. We next discuss all three components in more detail.
3.2.1 DATA MODIFIER
We design the data modifier to be conditional on its original input in learning instance-specific trans- formations via multi-layer perceptron (MLP) with residual shortcut. In other words, based on its input x, the data modifier G learns an additive modification to be applied to x. Therefore, the transformed data stays the same embedding space as the original, which leads to faster optimiza- tion convergence. Specifically, for normalized original data vector x, its normalized transformed version is computed
x G(x) nx + n (MLP(x))).
||v|| #
where n(-) is the normalization function ntv)
0.
o.w.
During the optimization, the parameters in the data modifier trained by back-propagating the losses designed in both branches.
3.2.2 SUPPRESSION BRANCH
The suppression branch is designed make the targeted attributes in S unrecognizable possi- ble. utilizes the corresponding set of inference models pretrained on the original dataset X. Each pretrained model corresponds F*, which converts raw input data to a feature vector z = f (x), and a projector ps, which maps the features the attribute's label p ps (z)), where z the feature representation and p the prediction logit.
specific attribute
S and is composed of feature extractor
During the training of the data modifier, the suppression branch guides the data modifier degrade these pretrained models' recognition accuracies on the transformed data X'. It does so by either measuring the similarity of features from the pretrained models or by comparing the prediction results against the ground truth labels, and then adding corresponding penalty. Specifically, for each targeted attribute E S, the Teature-similarity loss function defined as
L sim
W.
Σ sim" (x'x),
where w is the weight for attribute s and (i." defines a similarity measure, For example, we use the cosine similarity between feature vectors sime, (x'xx cos (z z(), the negative KL-divergence between the original and the transformed logits simg, (x',x) -Di.. p p
or the cross-entropy loss between the transformed logits and the attribute's ground truth value simgg xx) -DKL. [p: 1..). where Sx x's ground truth value for attribute and 1, is the corresponding one-hot vector.
4

---

Under review as a conference paper at ICLR 2023
The above similarity-ber based metries only help guide the data modifier to lead the pretrained mod- els towards incorrect predictions about S on the transformed data. However, in terms of attribute suppression,
stronger condition would be reduce such predictions to random guesses. Therefore, we introduce an additional loss term to maximize the entropy H of the predictions about $ on the transformed data such that the predictions look like random guesses. Combined with Eq the total loss on the suppression branch is
L w
1 N
Σ XEX
sim' (x',x) ht"
....11 H N
(3)
XEX
where h" is the weight for the entropy loss.
Moreover, to regularize the learned data transformation to be small as possible, also add an L2 reconstruction loss term in the suppression branch
Lie Wree
Σ - x/12-
(4)
N
XEX
3.2.3 PRESERVATION BRANCH
While the suppression branch guides the data modifier to decrease the confidence of machine learn- ing models on certain attributes S from the data, the preservation branch is responsible for guarding the data against said suppression and erasure such that maximum utility can be preserved through the transformation, in the sense that all the attributes branch should remain recognizable from the transformed data X', just as they were from the original data X. In the proposed method, we design two types of losses: one attribute-agnosicc in the preservation branch, which is applicable any attribute as the loss is defined in an agnostic way; another one attribute-specte which the downstream tasks have been defined and we know the set of R, then embedded them into the loss.
) S not targeted by the suppression
Attribute-Agnostic. As previously discussed, it is oftentimes difficult to identify in advance the A of all attributes associated with a dataset, which means the set R might not necessarily be defined even when S is specified. The absence of R indicates that we have no knowledge about what attributes downstream tasks might want detect from the transformed dataset.
To tackle this attribute-agre agnostic scenario. we employ self-supervised techniques, where a generic feature representation of the data learned without any specific attribute or task information. For our preservation branch in particular, we adopt the SimCIR (Chen et al. (020) contrastive-lerniing- based approach to pretrain an extractor f* and a projector pt. that maximize the similarity between embedding pairs originated from the same data point (i.e., positive pairs) and minimize that of dif- ferent data points (i.c., negative pairs). Instead considering two different transformations like SimCLR does,
treat an original data point and its transformed version x' positive pair, and all the rest as negative. We enforce this relationship by applying normalized temperature-scant cross entropy (NT-Xent) loss function introduced in SimCLR; therefore, semantically, the trans- formed data could preserve more generic features. For the original data point x-based positive pair (x, x'), the NT-Xent loss 1 is
cos(p² //2)
K log
(5)
cos pp x4;
/T lyex coss )/T
Σyex
and similarly for the transformed data point bbbee positive pair (x', x), the NT-Xent loss &
cou(p$><<>))
e log
(6)
cos pos )/+
1yax cos cos(p(t-p:)/11
Σ Σyex
where pç = PF and p pr. (P''(((xx) the attribute-agnostic logits computed from the original and transformed data points, respectively; I controls the contribution proportion between the positive and the negative samples, with higher value indicating bigger contribution from
5

---

Under review as a conference paper at ICLR 2023
1, # x
the positive samples, and vice versa; and I yex is the indicator function Iyxx
The total
00 o.w.
loss for the preservation branch is thus aggregated across all data points
L" w
Σ XEX
11 +1)).
2N
which enables the preservation branch to enforce the underlying representations of the transformed data points in X' to reach maximal agreement with their corresponding origins from X while main- taining their discriminative characteristics with each other.
Attribute-ppecific In addition to the attribute-agnostic case discussed above, there could also be attribute-specip situations, where the set R of attributes that needs to be preserved is explicitly defined in advance for the data transformation. Therefore, we can pretrain the inference models on the original dataset X like in the suppression branch. To account for this extra information, R, we formulate the loss function similar to Eq. 2] except that here we want to maximize the similarities as opposed to minimizing them, as follows.
='=
Σ xeX
sim' (x',x),
(8)
N
where w" is the weight for each attribute r E R.
Finally, collecting all the loss terms from the data modifier, the suppression branch, as well as the attributo-agnostic and atribute-speciffe components the preservation branch, we have the overall optimization loss
L=
Σ SES
Lioe + L" Σ...
(9)
PER
EXPERIMENTAL EVALUATION
We next present and discuss experimental evaluation of MaSS its ability to perform selective attribute suppression and preservation when carrying out data transformations.
4.1 EXPERIMENTAL SETUP
We evaluated MaSS on three multi-attribute datasets of different domains, namely Adience (Ei- dinger et al. 2014) for facial images, AudioMNIST (Becker et al. 2018) for voice recordings, and PA-HMDB (Wu et al. 2020) for video clips. For all datasets, we converted all their raw data points to feature embeddings via state-of-the-arr neural networks the input X. Table
the feature tractors used for each dataset, as well as the corresponding feature dimension and the architecture of MLP used in the data modifier. All our experiments were designed examine MaSS' performance on attribute suppression and preservation quality, and not whether it could generate high quality synthetic data. Therefore, each transformed x' would stay the same feature space as its corre- sponding x. We next briefly introduce the datasets, as well as the training and evaluation protocols. More details can be found in Appx. and
Adience. The Adience image dataset originally published help study the recognition of age and gender. Each image is also associated with a DatalD. In total, the dataset used in our experiment contains 1,089 different DataIDs, age groups, and 2 gender classes. We split the dataset into 2,815 for training and 2,525 for validation.
AudiOMNIST. The AudioMNIST dataset contains audio recordings of spoken digits (0-9) En- glish from 60 speakers.
In addition to speakerID and spoken digits, the dataset also contain accent and gender attributes. 4-attribute dataset, namely speakerID, spoken digits, accent,
Therefore, use AudioMNIST and gender, with 60, 10, 18, and classes, respectively. There are 30,000 audio clips in total. We split the data into 18,0 18,000, 6,000, and 6,000 for training, validation, and testing, respectively.

---

Under review as a conference paper at ICLR 2023
Table 1: Model configuration for each dataset.
Dataset
Feature Extractor
Feature Dimension
MLP in Data Modifier
Adience
FaceNet (Schroff et al.
2015
512
512-256- 015-2566118-2565 120-112----61111 512
AudiOMNIST
HuBERT'L [issu et al
2021
1024
1344-256-128-256 1026--14444
PA-HMDB
R3D-18 Tran
512
512-256-122-256-33 128-126-1122
Table 2: Results on Adience under different configurations. The checkmark (v) denotes that the particular loss is used in the optimization. DatalD is selected as the suppression target.
Loss Configuration
Top-1 I Accuracy (%)
Lrec
L
L
L
L
DataID (s1)
Age (r1)
Gender (r2)
Original
90.8
89.1
97.4
/
0,0
33,8
73.5
/
/
/
0.6
78.5
95.7
MaSS
v
/
/
0.7
86.1
95.9
0.6
86.9
96.7
PA-HMDB. The PA-HMDB51 dataset is subset HMDB51 and it contains 6 attributes: action, skin color, face, gender, nudity, and relationship (Kuehne et al. [011 Wu et al., 2020). The action label is annotated at video-level while the other non-action attributes are at frame-level. Nonetheless, PA-HMDB51 only contains about 500 vidcos, it is only used for evaluation. There are 51 different action classes. The other 5 non-action attributes are all binary. As there is no training data for the non-action attributes in PA-HDMM51, used VISPR (Orekondy al.] (017) for training them, and used HMDB51 for the action attribute. MaSS was evaluated on both VISPR and PA-HMDB51.
Training and Evaluation Protocols. Before we can start training MaSS' data modifier G, we first need train the models for the attributes we intend to suppress the suppression branch, well model for the preservation branch, and potentially also the attribute-specife models if the corresponding labels for those attributes are available. The training details for those pretrained models are in Appx. B. and B.2 Subsequently, each of these pre-trained models is used either the suppression or the preservation branch during the training of the data modifier G via optimizing Eq. [9] defined in Sec. 3 The details of training MaSS are described in Appx. [B.] including the similarity measurements hyperparameters, loss weights, etc. Note that during training G, the pre- pre-trained models in both branches are fixed without any update. For evaluation, trained data modifier to generate new data and feed them through the pre-trained models and examine their performance. For both Adience and ANdST, report the results on each attribute. For PA-HMDB51, we follow the same practice as PA-HMDB51 to aggregate the performance of the non-action attributes via macro-average of classwise mean average precision (cMAP) and use top-1 accuracy for action.
as the attribute-
the
use top-1 accuracy as the metric and
4.2 COMPARISON TO BASELINES
First, we examine the effects of each of the loss terms for each of the datasets. Table shows the performance on Adience where DataID is targeted for suppresion, and the others are expected be preserved. When only imposing attribute suppression (Lre and L*)), MaSS did successfully degrade the performance of DatalD. But at the same time, the performance on age and gender deteriorated significantly. When we added the attribute-agnostic loss (L**), MaSS still achieved good suppression DatalD. But at the same time it greatly improved the recognition accuracy on age and gender. Note that, we do not use any information from age and gender attributes during the training. Moreover, when the age information (L") age detection on the transoformed dataset almost returned to the same level as that of the original dataset. The same trend can be observed for the gender attribute (L²) well.
incorporated in MaSS, the performance of
A similar outcome can also observed for the AudioMNIST dataset (Table 3). where SpeakerID was treated as the suppression target, and all the rest of the attributes were expected to be preserved. When MaSS only used suppression (Lee and L")), SpeakerID score did reduce, but so did the other attributes. Adding the atribute-agnostic loss (1.") brought significant improvement for the
7

---

Under review as a conference paper at ICLR 2023
Table 3: Results on AudioMNIST under different configurations. The checkmark (V) denotes that the particular loss term is used in the optimization. SpeakerID is selected as the suppression target.
Loss Configuration
Top-1 Accuracy (%)
Lecc
1.
1.7
1.
SpeakerID (s))
Digit (r))
Accent (r2)
Gender (r3)
Original
95.6
99.8
99.3
96.5
0.0
26.2
45.3
53.5
/
/
/
1.7
67.4
68.7
88.2
MaSS
/
1.7
99.7
68.4
80.0
99.7
86.6
1.7
99.6
95.7
98.4
Table 4: Results on VISPR and PA-HMDB under different configurations. The checkmark (V)
denotes that the particular loss term used in the optimization. Metries for the action attribute Top-1 Accuracy (%) while cMAP (%) is used for the other 5 non-action attributes. MaSS is configured to suppress the non-action attributes.
Loss Configuration
VISPR
PA-HMDB
Lirec
L
L.
17
Non-action Attrs. (s1)
Action (r1)
Non-action Attrs. (s1)
Original
81.8
58.7
79.7
/
41.8
12.6
70.3
MaSS
1
/
36.3
52.1
63.2
1
38.6
58.0
63.4
performance of all the other three attributes. Finally, when the attribute information were available (L", L and LP), MaSS was able boost the preservation performance near perfection without reducing the suppression quality on SpeakerID.
Table shows the performance MaSS on the video dataset. With only suppression (Lree and )"),, MaSS indeed lowered the cMAP on both VISPR and PA-HMDB. However, the performance on the action attribute also dropped significantly. By adding the attribute-agnostic loss (I.+), MaSS was able to significantly improve the accuracy of the action attribute without utilizing any label information. We do notice a slight decrease in the performance of other attributes, which also aligns with the observation in SPAct (Dave et al., 2022), When the action label (L") was incorporated, MaSS was able to preserve the same action accuracy while suppressing the other attributes.
All experimental results above show that MaSS is highly configurable and effective even without knowing the downstream tasks advance. Moreover, when the information downstream task is available, MaSS can boost the performance the corresponding specific attributes.
4.3 COMPARISON OTHER METHODS
As MaSS includes both suppression and preservation, there is limited prior work for direct com- parison since most them focus on suppression only. Therefore, we first compared MaSS with
the heuristic methods, e.g., perturbation the original data, additive noise in the feature space for
both the Adience and AudioMNIST datasets. For Adience, we further compared MaSS to CIA-
GAN (Maximov et al.) 2020), DeepPrivacy (Hukkelas al.. 2019) and Fawkes (Shan et 2020 even though they only performed suppression. Lastly. we compared MaSS to SPAct on PA HMDB. More details on the comparisons can be found in Appx.D
Table 5 shows the comparison on Adience. As most methods were designed to suppress DatalD, we configured MaSS to suppress Datall) and preserve the others with atrribute-annostic setting for comparison. First, adding noise in the feature domain lowered the accuracy on the DatalD attribute but also degraded the performance on age and gender. On the other hand, for all the methods which manipulate data in the original domain, including Gaussian blurring with different kernel sizes and standard deviations, downsampling and upsampling back to the original size, and obfuscating the various face area where the face is detected by MTCNN (Schroff et al., 2015), ) they led to similar results since they did not learn what features should be preserved for downstream tasks. By modifying faces in the image space, CIAGAN (Maximov et al. 2020), DeepPrivacy (Hukkelás

---

Under review as a conference paper at ICLR 2023
Table 5: Comparison other methods on Adience. Top-1 accuracy (%) is reported. Features and
Raw Data denote that the changes are made on the feature level and the image level, respectively.
Method
DataID (J)
Age (1)
Gender (1)
Features, o 0.5
50.8
56.8
88.1
Gaussian Noise
Features, o 1.0
0.5
26.8
54.9
Raw Data, À 11, or 10.0
86.5
Guassian Blur
Raw Data, k 21, 15.0
1.0
19.3
66.2
Raw Data, 8x Raw Data, 4x
11.0
30.3
78.7
Downsample
79.3
74.7
94.3
Raw Data, Face area, ratio 1.0
0.8
17.1
59.5
Obfuscation
Raw Data, Face ratio 0.36
4.0
29.8
81.6
Raw Data, Face area, ratio 0.09
57.7
93.9
CIAGAN
1.1
17.8
66.9
DeepPrivacy
5.9
32.2
84.0
Fawkes
24.7
47.6
87.8
MaSS
Lree 1 + 1.5
0.6
78.5
95.7
Table 6: Comparison with other methods on AudioMNIST. Top-1 accuracy (%) is reported.
Method
SpeakerID ())
Digit (1)
Accent (1)
Gender (1)
White Noise (Raw Data)
4.7
23.3
23.4
32.2
Masking (Raw Data)
1.8
3.9
MaSS (Lrec + L L
1.7
67.4
68.7
88.2
et al. 2019) and Fawkes (han et al. 2020) were able to lower the performance on DatalD again. However, they still weren's capable of preserving any features for the other attributes. In contrast, MaSS not only suppressed Datall but also preserved much more data utility such that the data could still perform well on age and gender detection tasks, even though no additional information on age gender was made available to MaSS.
The comparative study results on AudioMNIST are listed in Table The heuristic approaches were capable of suppressing SpeakerID, as well the other attributes indiscriminatively. Nonetheless, MaSS can retain the other attributes without knowing the downstream tasks while still achieving SpeakerlI suppression. Lastly, comparing to SPAct (Dave et al. 2022) PA-HMDE" MaSS achieved competitive suppression ratio of cMAP on both datasets (VISPR: 55.6% vs. 57% and PA- HMDB: 20.7% 16%) for the other non-action attributes. In addition, MaSS also provides the flexibility to configure which attributes to suppress while SPAct can only target all five attributes at the same time.
Additional ablation studies of MaSS, including suppressing differeet//mltippe attributes, effects of loss weights and different similarity measurements, can be found in Appx.
CONCLUSION
In this paper, we proposed MaSS to selectively suppress attributes while preserving other attributes; moreover, with the proposed sttribue--annoste approach, the preservation can be achieved without foreseeing the downstream tasks, which expands the usability of the proposed algorithm. We val- idated our method in three datasets in different domains, including facial images, voice audio and video clips, and all results showed that our method promising attribute suppression and preser- vation. Lastly, would like point out that we validated MaSS in the feature space rather than the original data space; an interesting future direction, which is beyond the scope of this paper, would be to integrate MaSS with generative models, such as GAN or diffusion models to further convert the transformed feature vectors back into their original representation space (image, audio, etc.).
We compared SPAct based their Table 8 in the paper, which is the closest to our setting. Because the baseline performance is different, show here the relative suppression results. See Tahle for details.

---

Under review as a conference paper at ICLR 2023
Code of Ethics and Ethics statement. Our work is to selectively suppress the attributes in multi- attribute data while preserving their potential utility such that the sensitive information in the data could be minimized and hence the data could be used to benefit the community. We believe that there are no ethical concerns related to this work.
Reproducibility Statement. We provided the training and evaluation details in the main paper and appendix. Our source codes and models will be publicly available to help better understand the settings of training and evaluation.
REFERENCES
Sören Becker, Marcel Ackermann, Sebastian Lapuschkin, Klaus-Rober Müller, and Wojciech Samek. Interpreting and explaining deep neural networks for classification of audio signals. arXiv preprint arXiv:1807.000 08,48, 2018.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL https::/prroceeete 1447c0d6bffb496777 8wf.8aclffacccccooo 42 Papee Paper.
neurips cc/ppaper/222//ilee
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsu- pervised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Mathilde Caron, Ishan Mísra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-99244 2020.
George Cazenavette, Tongzhou Wang, Antonio Torralba, Alexei Efros, and Yan Zhu. Dataset distillation by matching training trajectories. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10718-10727, June 2022.
Jia-Wa Hei Chen, Li-Ju Chen, Chia-Mu Yu, and Chun-Shien Lu. Perceptual indistinguisaabiliyynee (pi-net): Facial image obfuscation with manipulable semantics. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2021.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Hal Daumé III and Aarti Singh (eds.), Pro- ceedings of the 37th International Conference Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020. URL https: //proceedings ml wtt.:n...........s..............
chen20j. html
Zhi-Qi Cheng, Xiao Wu, Siyu Huang, Jun-Xiu Li, Alexander G. Hauptmann, and Qiang Peng- Learning to transfer: Generalizable attribute learning with multitask neural model search. In Proceedings of the 26th ACM International Conference on Multimedia, MM '18 pp. 90-98, New York, NY. USA, 2018. Association for Computing Machinery. ISBN 978 170335666667 doi: 3255//2205580.0 .20.83240555 3240518 holttt:ts://dae wtwpel/.ogg111111 121//......1/320533333
00.1445/3240508.224555
Ishan Rajendrakumar Dave, Chen Chen, and Mubarak Shah. Spact: Self-supervised privacy preser- vation for action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255.255. leee, 2009.
10

---

Under review as a conference paper at ICLR 2023
Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? In Kamalika Chaudhuri, Stefamie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 5378-5396. PMLR, 17-23 Jul 2022. URL httpttttts/sppeeede//// mng.ings............ ml www.s/pessslll... wttrraas/w162//6ng222..... html
Eran Eidinger, Roce Enbar, and Tal Hassner. Age and gender estimation of unfiltered faces. IEEE Transactions information forensies and security, (12)22170-2 2014.
Zeyu Feng, Chang Xu, and Dacheng Tao. Sulf-supervised representation learning by rotation feature decoupling. In 2019 IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10356-10366, 2019. doi: 10.1 21110//CVP..011.00066
Wikimedia Foundation. Acl 2019 fourth conference on machine translation (wmt 19mmt19). shared task: Machine translation of news, 2019. URL htwp:://www.c.. htttp:/www.statnt: statmt. .rgg wwt 19/
translate inlllationntaen task. htt.mkk..mllll......mm...m.
Abigail Goldsteen, Gilad Ezov, Ron Shmelkin, Micha Moffie, and Ariel Farkash. Data minimization for gdpr compliance in machine learning models. Al and Ethics, pp. 1-15, 2021.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach self-supervised learning. Advances in neural information processing systems, 33:21271-21283 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7729-9733 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia. Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. /EEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-344 -3600, 2021.
doi: 10. 10.109/TASLLP.202.332222
Siyu Huang, Xi Li, Zhi-Qi Cheng, Zhongfei Zhang, and Alexander Hauptmann. Gnas: A greedy neural architecture search method for multi-attribute learning. In Proceedings of the 26th ACM International Conference on Multimedia, MM '18, pp. 2049-2057, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450356657. doi: 10.1145/3240508.224000
URL /attps://dof: wrg/10.../...oo.oo.o..o.
3240588
Häkon Hukkelås, Rudolf Mester, and Frank Lindseth. Deepprivacy: A generative adversarial net- work for face anonymization. In George Bebis, Richard Boyle, Bahram Parvin, Darko Koracin, Daniela Ushizima, Sek Chai, Shinjiro Sueda, Xin Lin, Aidong Lu, Daniel Thalmann, Chaoli Wang, and Panpan Xu (eds.), Advances in Visual Computing, pp. 565-5555777 578, Cham, 2019, Springer International Publishing. ISBN 978-3-030-33720-9,
J. Kahn, M. Rivière, W. Zheng, E. Kharitonov, Q. Xu, P. E. Mazaré, J. Karadayi, V. Liptchinsky, R. Collobert, C. Fuegen, Likhomanenko, G. Synnaeve, A. Joulin, A. Mohamed, and E. Dupoux. Ligri-light: A benchmark for asr with limited or no supervision. In ICASSP 2020 2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 7669-7673, 2020. https: gi tuub. com
Davis E King. Dlib-ml: A machine learning toolkit. The Journal of Machine Learning Research, 10:1755-1758, 2009.
Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In 2017 International conference on computer vision, pp. 2556-2563, IEEE, 201
11

---

Under review as a conference paper at ICLR 2023
Jingzhi Li, Lutong Han, Ruoyu Chen, Hua Zhang, Bing Han, Lili Wang, and Xiaochun Cao. Identity-preservee preserving face anonymization via adaptively facial attributes obfuscation. Proceed- ings of the 29th ACM International Conference Multimedia, pp. 3191-389999 2021.
Bo Liu, Ming Ding, Sina Shaham, Wenny Rahayu, Farhad Farokhi, and Zihuai Lin. When machine learning meets privacy: A survey and outlook. ACM Computing Surveys (CSUR), 54(2):1-36, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence Learning Representations, 2019. URL BBRl:/ttte:://ppeneeeeee...tttt http://rrweweeeeeeeoooouummamae
Bkg6RicqY7
Edward Ma. NIp augmentation. http://ggthhh..oommmaccccvaaaaat
2019.
Maxim Maximov, Ismail Elezi, and Laura Leal-Taixe. Ciagan: Conditional identity anonymiza- tion generative adversarial networks. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Bill Moriarty, Chun-Fu Richard Chen, Shaohan Hu, Sean Moran, Marco Pistoia, Vincenzo Piuri, and Pierangela Samarati, Utility-preserving biometric information anonymization. European Symposium Research in Computer Security, pp. 24-41. Springer, 2022.
Tribhuvanesh Orekondy, Bernt Schiele, and Mario Fritz. Towards visual privacy advisor: Un- derstanding and predicting privacy risks in images. In Proceedings of the IEEE international conference on computer vision, pp. 3686-3695, 2017.
Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsura Yang. Huisheng Wang. Serge Belongie, and Yin Cui. Spatiotemporal contrastive video representation learning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 6964-6974, June 2021. Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recogni- fion (CVPR), pp. 815-823, 2015. doi: 10.1 00.109999PR..2055798666.
Shawn Shan, Emily Wenger, Jiayun Zhang, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Fawkes: Protecting privacy against unauthorized deep learning models. In Proceedings of the 29th USENIX Conference on Security Symposium, SEC'20, USA, 2020. USENIX Association. ISBN 978-1- 939133317-5.
Du Tran, Heng Wang. Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE Confer- Computer Vision and Pattern Recognition (CVPR), June 2018.
Kai Wang, Bo Zhao, Xiangyu Peng. Zheng Zhu, Shuo Yang, Shuo Wang, Guan Huang, Hakan Bilen, Xinchao Wang, and Yang You. Cafe: Learning to condense dataset by aligning features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12196-12205, June 2022.
Zhenyu Wu, Haotao Wang, Zhaowen Wang, Hailin Jin, and Zhangyang Wang. Privacy-preserving deep action recognition: An adversarial learning framework and new dataset. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. Dataset condensation with gradient matching. In International Conference on Learning Representations, 2021. URL httph:ttpt///ttt /tttps:///pperevvee.
nett/forum2icmmmAHHLYYY
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
In the appendix, we provided the supplemented materials, including data preprocessing in Sec. In Sec. B described model pre-training for contrastive learning, model training for single attribute and MaSS. We included the ablation studies in Sec. [C and how to generate compared results in Table 5 and Table in Sec. [ Our source code and models will be publicly available help better understand the settings of training and evaluation.
A DATA PREPROCESSING
As briefly discussed in the main manuscript, MaSS takes feature vector in and then generates feature vector instead of operating on the raw data. Therefore, for each dataset, we convert the data into feature vectors via state-of-the-art neural networks and then normalize the vector by its n.2-norm.
Adience. We first resize the image into 160 x 160 and normalize the image by the mean and the standard deviation used in the FaceNet (Schroff et al., 2015). Then, we feed the normalized image into FaceNet to get 512-d feature vector.
The majority of the information in audio signal resides at the beginning, and the average length of a waveform is 30,844 samples and the upper quartile is 34,380. Therefore, either truncate and pad (zeros) the waveform to the length of 30,000 at the end such that the data loader can form them as batch speed up the training. Then, we feed the truncated/padded waveform into Hubert-L. (Hsu et al. (02T) to get 1024-d feature vector after performing average pooling on the output of Hubert-l. along the time dimension.
AudioMNIST.
VISPR and PA-HMDB. R3D-18 is trained with the clip size of 16 x 112 112 and generates a 512-d feature vector; therefore, we resize the spatial dimension of a video into 112 x 112 and then sample 16 frames (every other frame) an image dataset, we generate a 16-frame clip by duplicating the same image and then pass it to R3D-18 to extract features. On the other hand, for the action attribute in PA-HMDB, we convert each video into frame-level feature vectors by R3D-18. More specifically, for each timestamp, we take its neighboring frame (every other frames) form 16-frame clip and then feed it to R3D-18. E.g., for video with 100 frames, we will get 100 512-d feature vectors. For the other attributes in PA-HMDB, since those labels are image-level instead of video-level, we simply assign the labels to the frame-level features extracted above.
of video to form clip. For VISPR, since it is
B MODEL TRAINING
B.1 ATTRIBUTE-AGNOR MODEL PRE-TRAINING
For all datasets, we follow similar practices to train aunibbte-agnostic models via SimCLR (Chen et al. 2020). First, generate two views of data by different data augmentation in the raw data domain, and then the two views of data are passed through the fixed feature extractor to get its feature representation. After that, train multi-layer perceptron (MLP) as encoder learn generic feature representation over the features extracted by the fixed feature extractor. The MLP composed of two fully-connected layers with the same dimension as the input feature dimension, and I-D batch normalization layer and ReLU are added between fully-connected layers. The trained MLP is served as attribute-agne aggnoti model in MaSS. Note that we also adopt the MLP-projector in the contrastive learning to achieve better performance. We train the model for 100 epochs with temperature 0.07 via the stochastic gradient decent (SGD) optimizer. The weight decay is set to 0.0001 and the learning rate starts from 0.05 and then it is annealed with cosine schedule. In the follow paragraphs, we describe how to generate different views for each dataset.
Adience. To generate two views, first resize images to 160 x 160 and then randomly flip the image horizontally; after that, we randomly perform color jitter via torchvision package with 80% probability and then convert the image into gray scale with 20% probability.
13

---

Under review as a conference paper at ICLR 2023
Table Model training settings for each dataset in Table
and
Dutaset
#GPUs
Batch Size per GPU
Learning Rate
Loss Weights
Wiee 1, w = h1
Adience
64
0,00125
w" 40, wn w
10, w"I 0.001, A =.1
AudiOMNIST
128
0.01
w/ 15, w"1 0.01, w 0.001. w 0.001
1, w* A 0.1
PA-HMDB
0.01
w' 1000, w 50
Table 8: Results on Adience with different suppressed attribute, the experiments are completed under the loss setting of Lree L L"", where is the suppressed attribute.
Top-1 Accuracy (%)
Suppressed Attribute
DatalD
Age
Gender
DatalD
0.6
78.5
95.7
MaSS
Age
81.2
13.7
94.5
Gender
77.2
75.3
8.3
AudioMNIST. In this work, apply two different augmentations (Ma 2019): random crop on the entire audio with a coverage 0.4 and mask with a coverage of 0.5 to each view respectively. The crop augmentation removes the selected part from the audio, whereas mask substitutes it with zeros.
We limited the data augmentations used in methods crop and mask because other augmenta- tion like pitch, loudness, speed, etc. would affect the structure of audio signal and potentially won't t be able retain attributes like gender, accent, and age.
PA-HMDB. We generate views of data by following the practice in CVRL (Qian al. 2021] ie., for a positive pair, two views are extracted from different time instance of a video and the temporal-consisten data augmentation performed each view. The data augmentation is com- posed of resizing the spatial dimension into 112x 112, guassian blurring, randomly converting color image into gray image.
B.2 ATTRIBUTE-SPECIEE MODEL PRE-TRAINING
For all attributes in all datasets, we train the attribute-specific model by using the cross-entropy loss against the given label to compute the gradient for all parameters through back- model contains three fully-connected layers and with the dimension: input propagation. The of classes, and the 1-D batch normalization layer and ReLU added between layers. We use batch size of 256 with the Adam W optimizer (Loshchilov & Hutter] (209)) to train the model for 100 epochs. The weight-decay is fixed to 0.05 and the initial learning rate is 0.01 and then the learning rate annealed with the cosine scheduler.
B.3 MASS TRAINING
The training on different datasets follows similar settings but with different loss weights. When train- ing the data modifier in MaSS, all models in the suppression and preservation branches are fixed without any update. We train all models with 100 epochs with the AdamW optimizer (Loshchilov) & Hutter 2019) The weight-decay is 0.05 and we adopt cosine learning scheduler to anneal the learning rate. For the loss type, in most of cases, be-suppressed attribute and KL divergence for the attribute-specific preservation. Table ( described other training details for different datasets.
cosine similarity measurement for the to-
14

---

Under review as a conference paper at ICLR 2023
Table Results on Adience with different weights on L the experiments are completed under the loss setting of Lree + L L We used w 40 in our main results.
Top-1 Accuracy (%)
DatalD
Age
Gender
10
0.0
69.4
73.5
20
0.0
75.0
94.8
MaSS
40
0.6
78.5
95.7
80
3,2
82.1
96.4
160
13.5
83.5
96.4
Table 10: Results on Adience with different similarity measurements in L*, the experiments are completed under the loss setting of Lree + L L" Si is DatalD.
Top-1 Accuracy (%)
Similarity
DatalD
Age
Gender
Cosine
0.6
78.5
95.7
MasS
KL divergence
0.2
76.2
95.8
CE
0.1
76.7
94.9
Table 11: Results Adience under multiple suppressed attributes. selected to be suppressed. MaSS is configured with L and L"!
The DatalD and gender
Top-1 Accuracy (%)
DatalD (s1) Gender (82)
Age (r1)
Original MaSS
90.8
97.4 5.0
89.1
0.9
83.8
C
ABLATION STUDIES
ablation studies, we use Adience dataset for experiments and we discuss MaSS three perspectives, including suppressing different attributes, effects of loss weights, effects of similarity measurement.
Suppression Target. In the main manuscript, always suppress DatalD in all experiments; how- ever. MaSS is configurable to suppress any attribute while still preserving others. Table the results by suppressing different attributes. Only the performance of the selected attribute degraded while other attributes are still good. Note that those results do not include any sitribute-specifii mod- els in the preservation branch. The result shows that MaSS is flexible to configure suppress any attribute and preserve others.
Loss Weights. Intuitively, the loss weight controls which loss term should be focused on more during the optimization. In this ablation study, we vary the weights for the attribute-ague agnostic model and the results are shown in Table Since L controls how generic the feature representation is, the higher weights preserve more generic features; therefore, the transformed dataset could perform better for all attributes. However, when 1 160, the accuracy of DatalD also increased because the strength of suppression is not strong enough since the weight of L is too high.
Different Similarity Measurement for Suppression. We proposed three different measurements for the similarity in the main manuscript. Those measurements provided similar functionalities conceptually but they might work different empirically. Table 10] shows the results with different measurements, and all results are close to each other. Therefore, for suppression, we use cosine for all experiments.
Multiple Suppressed Attributes. Table II shows the results when we configured MaSS to sup- press multiple attributes, DatalD and gender. The accuracy of both DatalD and gender attributes
15

---

Under review as a conference paper at ICLR 2023
Table 12: Ablation studies of losses in the suppression branch on Adience. DatalD is selected the suppression target.
Loss Configuration
Top-1 Accuracy (%)
Lree
Entropy Loss
L2 Loss
Entropy
DataID (s1)
Gender
Age
0.003
0.0
28.6
68.9
MaSS
/
/
0.002
5.4
0.0
27.4
68.1
/
J
0.002
6.6
0.0
33.8
73.5
Table 13: Results Adience under different configurations. DatalD is selected as the suppression target.
Loss Configuration
Top-1 I Accuracy (%)
Tree
1.51
1.*
Ln
1.2
DatalD (s1)
Age (ri)
Gender (r2)
Original
90.8
89.1
97.4
/
/
/
/
0.6
86.9
96.7
MaSS
0.0
84.5
96.3
Table 14: Results AudioMNIST under different configurations. The checkmark (V) denotes that the particular loss term is used in the optimization. SpeakerID is selected as the suppression target.
Loss Configuration
Top-1 Accuracy (%)
Lyee
L
L"
L
SpeakerID (x1)
Digit (r1)
Accent (r2)
Gender (r3)
Original
95.6
99.8
99.3
96.5
MaSS
/
V
/
/
1.7
99.6
95.7
98.4
1.6
99.7
95.0
97.7
Table 15: Results VISPR and PA-HMDB under different configurations. The checkmark (V)
denotes that the particular loss term is used in the optimization. Metrics for the action attribute Top-1 Accuracy (%) while cMAP (%) is used for the other non-action attributes. MaSS is configured to suppress the non-action attributes.
Loss Configuration
VISPR
PA-HMDB
Liee
L
L"
Non-action Attrs. (51)
Action (r1)
Non-action Attrs. (s1)
Original
81.8
58.7
79.7
/
38.6
58.0
63.4
MaSS
38.3
52.2
63.6
suppressed successfully but the age attribute can still be recognized even without age labels. With age label, MaSS can further improve its accuracy.
Losses in Suppression Branch. Here, we further explore the effects of L2 reconstruction loss and prediction entropy loss applied in the suppression branch.
We add the 1.2 reconstruction loss to minimize changes in the data while suppressing the targeted attributes; thus, we are able keep the data as truthful as possible. The prediction entropy loss added to increase the entropy of the predicted probability such that the prediction of the modified data becomes closer to a random guess; thus, there less information in the prediction.
Table 12 shows the effects of these two loss terms when only suppression is considered, i.e., no preservation branch. With L2 reconstruction loss, the overall performance of attribute recognition does not affected: however, reduced the overall L2 loss. After adding prediction entropy loss, the accuracy of DatalD is still 0% but its entropy is increased, which makes the prediction closer to random guess, as desired.
16

---

Under review as a conference paper at ICLR 2023
Table 16: Comparison SPAct Dave al. 2022 Metrics for the action attribute is Top-1 Accu- racy (%) while cMAP (%) is used for the other 5 non-action attributes. Non-action attributes are suppressed the experiment, and MaSS configured without 7
VISPR
PA-HMDB
Non-action Attrs. (s1)
Action (r))
Non-action Attrs. (51)
Original MaSS
81.8
58.7
79.7
36.3 (155.6%%
52.1
63.2 (120.7%)
Original (SPAct)
64.4
70.1
SPAct
27.4 (1.7%)
58.9 (116%)
Effects of 1., with All Labels. In Table 2 and have shown that the benefits to have attribute-agnostic loss (L") compared to suppression only. Here, we discussed the contribution of L when all to-be-preserved attributes are available, and the results are shown in Table [3 14 and 15 Without L", it achieved competitive performance on to-be-preserved attributes to the one with L on Adience and AudioMNIST for all attributes. Moreover, for PA-HMDB, without L**, the accuracy of action is degraded 5.8,, we conjecture that even though the generic feature retained by enforcing L.- help the data utilities of the modified data.
D COMPARED RESULTS
Adience. We compared many approaches in Table 5 and here we describe the details for how to generate those results. First, for Gaussian noise, we added zero-mean with different standard deviations (o)) into the original feature vectors to manipulate data. For Gaussian blur, downsample and obfuscation are all performed in the raw data domain, and then the modified data are passed through FaceNet to get the feature representation. For Gaussian blur, we apply zero-mean with various standard deviations (o) with different kernel sizes (k) to blur the image. For downsample, we downsample the data with different ratios and then upsample back to original size. Lastly, for obfuscation, we MTCNN detect the location of the face and then remove the face region with different ratios.
For CIAGAN (Maximov al., 2020), first followed CIAGAN's method pre-extract the masked face and the facial landmark information for the Adience dataset by using the Dlib-ml li- brary (King) 2009). And then, the CIAGAN model takes in the Adience images, their landmarks, masked faces and the desired target. For DeepPrivacy (Hukkelås et al. (2019) and Fawkes (Shan al. 2020), we adjusted the released codes and ran over the Adience dataset generate new im- ages.
After we obtain the transformed Adience images, we use the same procedure as ours for evaluation: using FaceNet (Schroff et al.) 2015) to extract the feature vector of an image.
AudiOMNIST. We compared two methods in AudioMNIST. including adding white noise and masking out portion of waveform based the nlpaug library (Mal 2019). We use the default parameter for white noise and set the masking ratio to 50% of the waveform.
SPAct. We tried our best to compare with SPAct (Dave et 2022) under the same experimental condition; however, our absolute performance over the raw data is significantly better than their paper; therefore, we only compare the relative gains to them. Moreover, under this setting, they do not show the accuracy of the action attribute. Toble the comparison to SPAct and baselines. When comparing its own baseline, our method is competitive in suppressing non-action attributes while keeping good accuracy in the action attribute.
17


Reviews:
[{"review_id": "kfQNmCzMhzv", "reviewer": "Reviewer_8b8w", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Research Question: Solve problems where simple or complete deletion of data attributes results in severe degradation of data quality. For example, having to lose data for user privacy.\n\nResearch Target: The paper proposes a method to suppress any selected attribute in a multi-attribute dataset. The paper highlights that the proposed method can leave the remaining properties unchanged for potential ML-based downstream analysis tasks.\n\nResearch Solution: The solution proposed in the paper is to learn data modifiers through an adversarial game between two sets of networks, one designed to suppress selected attributes and the other to ensure the remaining attributes are preserved through a general contrastive loss and explicit classification metrics.\n\nThe paper validates the proposed method in three domains: Facial Images, Voice Audio, and Video Clips.", "summary_of_the_review": "This paper investigates an interesting question and has practical value. But some details of the methodology and experiments confuse me. I would like the author to reply in the rebuttal.\n\nBecause of the problems existing in the current paper, my initial score is: marginally above the acceptance threshold. ", "strength_and_weaknesses": "Advantages:\n1. The research topic is helpful for data protection and privacy.\n2. The paper tries to verify the proposed method in different domains.\n3. The paper elaborates on the research question and background in detail.\n\nDisadvantages:\n1. The paper does not describe the details of the method, and it is not easy to reproduce the paper.\n2. The paper lacks some very important references and related work.\n3. The paper lacks some necessary experiments and analysis.\n\nQuestions: 1. Is the role of the DATA MODIFIER to normalize the data? What is the value of n in Equation 1? How to set the residual shortcut here?\n\n2. Does the loss in Equation 2 compute the similarity of each pair of data? Are the same settings used here for all three tasks?\n\n3. How are the attribute-agnostic logits in Equations 5 and 6 estimated?\n\n4. Which formula does $L^{r_1}$ and $L^{r_2}$ in Experiment Table 2 equal? I didn't find it in the paper.\n\n5. The author's method in Table 5 seems to perform similarly to the Downsample. What is the reason for this?\n\n6. Has the author performed other verifications? Does the paper only achieve attribute recognition?\n\n7. Did the author test directly on the original data? Is it using the features already extracted from the dataset for testing?\n\n8. The authors leave out some work on multitask attribute recognition, eg.[1][2]. Could the proposed approach be extended to these multi-attribute recognition methods?\n\n[1] Huang, Siyu, et al. \"Gnas: A greedy neural architecture search method for multi-attribute learning.\" Proceedings of the 26th ACM international conference on Multimedia. 2018.\n[2] Cheng, Zhi-Qi, et al. \"Learning to transfer: Generalizable attribute learning with multitask neural model search.\" Proceedings of the 26th ACM international conference on Multimedia. 2018.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ia3CyFKCR5Y", "reviewer": "Reviewer_SL3b", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper designs a data obfuscation framework that is able to suppress an arbitrary set of attributes while preserving all other attributes. The proposed framework trains a neural data (additive) modifier to minimize the similarity between the original data and the modified data for the attributes to be suppress while maximizing the similarity between the original data and the modified data for other attributes. The proposed method demonstrates promising results for suppressing targeted attributes while preserving others on multiple datasets.", "summary_of_the_review": "This paper proposes a data obfuscation framework to selectively suppress attributes while preserving other attributes. The proposed method is straightforward, flexible and effective and the experimental results are great. However, it has some evident drawbacks (e.g. all data points should be converted into feature embeddings) and the paper writing can be improved (e.g. too long abstract, confusing method section, lack of ablation experiments). Also, the proposed method just combines several natural and standard terms, and the ablation study does not convincingly show the efficacy of each (e.g. the reconstruction loss). Therefore, the paper still needs some work.", "strength_and_weaknesses": "Pros:\n- The topic is interesting and the motivation is strong. Removing sensitive attributes from data is crucial for real-world data usage, and (as shown in Table 5) vanilla or prior data obfuscation methods either cannot successfully destroy the performance on targeted attributes or cannot preserve satisfiable performance on other attributes. This paper aims at reaching the two goals simultaneously.\n- The proposed method is straightforward, flexible and effective. Each components of the proposed loss function directly relates to a desired purpose. The proposed method can suppress an arbitrary set of attributes and can preserve information in both attribute-agnostic and attribute-specific situation. Experiments show the proposed method can decrease the performance on targeted attributes drastically with little impact on the preserved attributes. \n\nCons:\n- The abstract is too long. It should be more concise.\n- The method section and Figure 2 are a little bit confusing. It is not clear how many pre-trained models are used and where they are. It is neither clear which loss is penalty and which loss is award. For example, both the penalty and the award similarity loss for the two branches are called \"Attribute-Specific Similarity loss\", which is very confusing.  Also, why is there only one Prediction Entropy Loss in the Suppression Branch in Figure 2? Shouldn't it be one for each attributes and thus S in total? Besides, what does it mean by \"the data modifier tries to minimax the loss\"? Isn't the data modifier the only updating network which is minimizing the loss while others are pre-trained and fixed models?\n- All data points are converted into feature embeddings in the experiments. It not only requires a good feature extractor in advance when applying the proposed method, but also prevent the usage of domain-specific techniques (e.g. data augmentations, neural architectures) when data users uses the processed data. \n- The ablation of the proposed loss function is not enough. Since the proposed loss function is composed of so many terms, it will be considered as validated only if each component shows effect. Currently, there's no experiments demonstrating the benefit from the reconstruction loss. For the suppression branch, there's no experiments showing how many benefits come from the Attribute-Specific Similarity loss and how many benefits come from the Prediction Entropy Loss. For the preservation branch, there's no experiments testing whether the Attribute-Agnostic Loss still contributes to preserving the information when the Attribute-Specific Loss is available.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hvCy2gB5PAN", "reviewer": "Reviewer_HMFn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a multi-attribute selective suppression (MaSS) framework for the multi-attribute classification task, where there is some targeted attribute to be suppressed. The authors apply a MLP based Encoder-decoder with skip link architecture to achieve the task, where it is decomposed into mainly three parts, the suppression branch, the data modifier (the architecture), and the preservation branch. Their method is further evaluated across a set of experiments, leveraging the face attribute of gender and age while suppressing ID, the audio sound of MNIST for digit classification while suppressing speaker ID, and the PA-HMDB for action classification while suppressing skin color, gender and other attributes. The overall performance has shown clear improvement over baselines and some SOTA methods. ", "summary_of_the_review": "Overall the paper presents a technically valid framework for interested attribute suppression based on the entropy maximization loss design. However, from the experiment, it does not show the ability to scale up to multiple attributes suppression, neither was it sufficiently compared to state-of-the-art methods to demonstrate its effectiveness. Thus, I would like to see the enrichment of the experiment for the final rating. For now, it is borderline towards reject.", "strength_and_weaknesses": "Strength:\n1. the technical pipeline of the selective suppression is clearly illustrated with adequate figures and formulated equations and losses.\n\n2. the experiments are designed to cover wider range of attribute classifications including face, audio and action recognition.\n\n3. the experiments are towards extensive to check each of the ablative components, and the hyper-parameters for a in-depth comparison for better understanding.\n\nWeakness:\n1. the scalability of dealing with many attributes is not sufficiently verified. Through all the evaluation datasets, the attributes considered is less than 10, which cannot directly expand to large scale scenario.\n\n2. the compared methods are mostly focusing on MaSS's ablative variations. There significantly lacks the comparison to other state-of-the-art methods, thus not so clear how better it is against the literature.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ezWUx6lqs4d", "reviewer": "Reviewer_uwsC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes Multi-attribute Selective Suppression, or MaSS, a general framework for performing precisely targeted data surgery to simultaneously suppress any selected set of attributes while preserving the rest for downstream machine learning tasks. The end goal is to achieve a sophisticated mechanism that gives data owners fine-grained control while retaining the maximal degree of data utility.", "summary_of_the_review": "The paper utilizes an adversarial training-based learning algorithm to fulfill their goal-simultaneously suppressing any selected set of attributes while preserving the rest for downstream machine learning tasks. There are some vague explanations in the paper that need to be clarified. Will consider changing my score after I see some of the other reviewers' opinions and the authors' responses.", "strength_and_weaknesses": "strength:\n1. the paper is easy to follow and written well.\n2. the motivation of this paper is valid and important to me, i.e., suppressing the data properly without affecting other attributes.\n3. the results seem to be effective from the main tables on main benchmarks.\n\nweakness:\n1. in section 3.2.2, the author mention that \"It utilizes the corresponding set of inference models pretrained on the original dataset X\". It is not clear how the pretrained models are trained such that it only focuses on one specific attribute.\n2. in section 3.2.3, for the agnostic case, it is not clear why a specific self-supervised pretraining can help preserve the non-selected attributes. If the selected attributes are forced to have a similar embedding after transformation, isn't it harmful to the final goal?\n3. the comparison in the experiment section is not consistent, for example, the author compares with the SPAct in PA-HMDB dataset, but they did not compare with this method on the ADIENCE dataset, which is their main result. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject