PASS: Private Attributes Protection with Stochastic Data Substitution
Authors: Yizhuo Chen
Chun-Fu Chen
Hsiang Hsu
Shaohan Hu
Tarek F. Abdelzaher
Venue: ICML 2025 spotlightposter (2025)
Keywords: Privacy Protection
Utility Preservation
Information Theory
Primary area: social_aspects->privacy

Abstract: The growing Machine Learning (ML) services require extensive collections of user data, which may inadvertently include people's private information irrelevant to the services. Various studies have been proposed to protect private attributes by removing them from the data while maintaining the utilities of the data for downstream tasks. Nevertheless, as we theoretically and empirically show in the paper, these methods reveal severe vulnerability because of a common weakness rooted in their adversarial training based strategies. To overcome this limitation, we propose a novel approach, PASS, designed to stochastically substitute the original sample with another one according to certain probabilities, which is trained with a novel loss function soundly derived from information-theoretic objective defined for utility-preserving private attributes protection. The comprehensive evaluation of PASS on various datasets of different modalities, including facial images, human activity sensory signals, and voice recording datasets, substantiates PASS's effectiveness and generalizability.

PASS: Private Attributes Protection with Stochastic Data Substitution
Yizhuo Chen 12
Chun-Fu (Richard) Chen
Hsiang Hsu? 2 Shaohan Hu 2 Tarek Abdelzaher
Abstract
to extract the content (Kheddar et al., 2024), which would inadvertently contain private attributes such as the speaker's gender accent, imposing the risk of privacy leakage when an adversary tries to eavesdrop on the voice. Therefore, there has been a lasting interest in developing a data obfus- cation module that can be inserted into the data sharing or ML service pipelines to protect the private attributes by sup- pressing or removing them from the data, while maintaining the utility of the data for downstream tasks.
The growing Machine Learning (ML) services require extensive collections of user data, which may inadvertently include people's private infor- mation irrelevant to the services. Various studies have been proposed to protect private attributes by removing them from the data while maintain- ing the utilities of the data for downstream tasks. Nevertheless, as we theoretically and empirically show in the paper, these methods reveal severe vul- nerability because of a common weakness rooted in their adversarial training based strategies. To overcome this limitation, we propose a novel approach, PASS, designed to stochastically sub- stitute the original sample with another one ac- cording to certain probabilities, which trained with novel loss function soundly derived from information- -eterorttttttotiiicccc objective defined for utility- preserving private attributes protection. The com- prehensive evaluation of PASS on various datasets of different modalities, including facial images, human activity sensory signals, and voice record- ing datasets, substantiates PASS's effectiveness and generalizability.
Various methods are proposed towards this goal. Roy & Bod- deti (2019); Bertran et al. (2019); Wu et al. (2020) focused suppressing private attributes while preserving explic- itly annotated useful attributes. On the other hand, Huang et al. (2018); Malekzadeh et al. (2019); Dave et al. (2022) extended their researches to managing unannotated gen- eral features for broader applicability. Furthermore, Chen et al. (2024) summarized and satisified SUIFT, 5 desirable properties of urility-preserving private attributes protection methods. Notably, these state-of-the-e methods are ad- versarial training based, where their obfuscation module is trained to prevent an adversarial private attributes classifier from making correct inferences.
However, widely discussed in the neighboring fields of adversarial robustness (Carlini & Wagner, 2017; Athalye et al., 2018; Ilyas et al., 2019; Carlini et al., 2019), generated image/vedio detection (Yu et al., 2019; Wang et al., 2020; Masood et al., 2023), membership inference attacks (Car- lini et al., 2022) and gradient inversion in federated learn- ing (Geiping et al., 2020; Huang et al., 2021), a common weakness with adversarial training based methods is that, although the defender versary, it may be vulnerable to slightly stronger unseen adversaries. In the private attributes protection context, as we theoretically and empirically demonstrate in Section 3.2, this weakness can indeed cause significant negative impact on the state-of-the-arr methods, reducing their practicality in real-world deployment.
1. Introduction
The expansion of modern Machine Learning (ML) services has seamlessly improved the convenience of daily lives, where the service providers oftentimes gather data from users and then utilize advanced models cater users' needs (Secinaro et al., 2021; Ahmed et al., 2022). How- ever, the data collected frequently includes private informa- tion that users may be reluctant to disclose (Boulemtafes et al., 2020; Kumar et al., 2023). For instance, a human voice recognition service needs to collect speaker's voice
tolerate the jointly-trained ad-
'Department of Computer Science, University of Illinois Urbana-Chaan Sabana-Champaagn. USA *Global Technology Applied Re- search, JPMorgan Chase, USA. Correspondence to: Yizhuo Chen <yizhuoc@ illinois. edu>> Chun-Fu (Richard) Chen frichard.cf.chemn wwwchheeeepnccccccmmmm.m.
To overcome the above weakness, we present novel method for private attributes protection, called Private Attributes protection with Stochatsic data Substitution (or PASS), which avoids the adversarial training strategy al- together. Specifically, we propose to substitute each input sample of the data sharing system or ML service pipeline with another sample according to a stochastic data substitu-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Attributes of Original Sample
Original Sample
Probability of Substitution
Attributes Inferred From Substituted Sample
both above strategies. Different from above, SPAct(Dave et al., 2022) proposed to train the obfuscation model to maximize contrastive learning loss adversarially with its feature extractor. All of the above methods, except for GAP, proposed to preserve annotated useful attributes by ensuring their predictability. Generalizing the problem. GAP, MSDA, MaSS, and SPAct(Dave et al., 2022) proposed to manage unannotated general features. These works are adversarial training based, resulting in common weakness elaborated in Section 3.2, motivating the design of PASS.
Private:
Private:
Sex:
Female
0.45
Sex:
???
Usefull:
PASS
0.43
Usefull:
Eyeglasses:
Yes
Eyeglasses:
Yes
Smiling:
Yes
Smiling:
Yes
0.04
General Features:
General Features:
Black hair:
(No)
Black hair: (No)
Young:
(Yes)
Young:
(Yes)
The field of fairness shares similar problem formulation and designing techniques with private attributes protection (Ed- wards & Storkey, 2016; Madras et al., 2018; Sarhan et al., 2020; Caton & Haas, 2024). Nevertheless, their training losses are typically derived from fairness metries, as op- posed to the privacy objectives, making them uncomparable with private attributes protection methods.
Figure 1. An illustration use case of PASS applicd on facial images. We suppress "sex" as a private attribute. and preserve "Eyeglasses" and "Smiling" as useful attributes. Apart from these attributes, we also preserve general features in facial images, such as "Black hair" and "Young", which are not explicitly annotated in the dataset, but are useful for potential downstream applications. PASS stochasti- cally substitutes the original sample with another sample such that the private attribute cannot be accurately inferred from the substi- tuted sample. On the contrary, the useful attributes and general features are still inferable from the substituted sample.
Local Differential Privacy (LDP) And Randomized Re- sponse. Differential Privacy requires the randomized mech- anism to produce similar distributions when applied to any two neighboring datasets (Dwork et al., 2006), which is typically achieved with addictive noises (Dwork et al., 2014; Abadi et al., 2016; Zhang et al., 2018; Sun et al., 2020). Sim- ilarly, Local Differential Privacy casts the same requirement on any two samples in a dataset (Kasiviswanathan et al., 201 Yang et al., 2023; Arachchige et al., 2019), instead of two neighboring datasets. Local Differential Privacy is mainly achieved with Randomized Response (Wamner, 1965; Wang et al., 2016; Chaudhuri & Mukerjee. 2020), which typ- ically involves randomly switching the class of the sensitive attribute when collected. As shown in Appendix B, under certain assumptions, PASS can be viewed as LDP mech- anism-sppcificit an extension of randomized response, that is distinguished by its utility preservation requirements and applicability on high-dimensional space.
tion algorithm. This algorithm is parameterized by a neural network and is trained with our novel loss function derived step-by-step from an thormmtiion-heorrticc objective defined for atility-preservin private attributes protection. Benefit- ing from the theoretical basis, PASS has clear operational boundaries for entangled attributes and can trade-off be- tween privacy and utility controllably. An illustrative use case of PASS is provided in Figure 1.
In summary, our paper's 's contributions are threefold: 1) We propose PASS, a stochastic data substitution based method that overcomes the common weakness of state-of-the-art private attributes protection methods; 2) We demonstrate theoretically that PASS is rigorously rooted in information theory with desirable properties; and 3) We extensively eval- uate PASS on facial images, human activity sensory signals, and human voice recordings to show its broad applicability.
k- A--onnyymity, I-Diversity and t-Closeness (k-I-t privacy). PASS can also be viewed as extension the k-l-t privacy in high-dimensional spaces. k-anonymity aims at thwart- ing membership inference by ensuring that at least k sam- ples share the same identifiable attributes after obfuscation, and consequently undistinguishable (Samarati & Sweeney, 1998; Qu et al., 2017; Song et al., 2019), while PASS aims at thwarting private attribute inference by ensuring that mul- tiple samples with different private attributes are substituted by each other, and consequently also undistinguishable. I- diversity and t-closeness extended k-anonymity, requiring that these k-samples (equivalent group) have diverse private attributes (Li et al., 2006), preferably with similar distribu- tion as the entire dataset (Sei al., 2017; Rajendran et al., 2017: 7: Majeed & Lee, 2020), preventing the attacker from guessing the private attribute from obfuscated sample accu- rately. Similarly, PASS also encourages the private attribute
2. Related Works
fillit-prreerving Private Attributes Protection. Vari- ous studies have been proposed on this topic recently. At- triGuard(Jia & Gong, 2018) proposed to defend against attribute inference attacks using evasion attack on the ad- versarial classifier. PPDAR et al., 2020), GAP(Huang et al., 2018) and MaSS(Chen et al., 2024) proposed to mini- mize private information leakage by making the adversarial classifier unable to make correct predictions. In compari- son,
et al., 2019) and BDQ(Kumawat & Na- gahara, 2022) proposed to minimize private information leakage by making the adversarial classifier uncertain about its predictions. Maxent- Maxent-ARL(Roy & Boddeti, 2019) and MSlAAMMalekzadeh et al., 2019) adopted a combination of
2

---

PASS: Private Attributes Protection with Stochastic Data Substitution
(e.g., X, S) denote random variables, and their correspond- ing lowercase letters (e-g., 8) the realization of random variables. We use P(-) to denote probability distributions (e.g., P(X)), among which we use Pawa(-) to indicate that this distribution is purely determined by a dataset and can be readily calculated, and (.)+) to indicate that this distribution is parameterized by 0 and can be calculated readily (e.g.. by forward propagation of neural network). Calligraphic letters (e.g.. D) denote datasets.
x
Figure 2. The probabilisttcc model of all random variables. U, are only dependent X.
guessed from the substituted sample to have a similar distri- bution as in the entire dataset.
We consider multi-attribute dataset with training split Duain and test split Dust. which can be seen as both drawn from the underlying data distribution (X,, where X is the high-dimensional original input data, S = SMv denotes a set of M user-chosen private attributes annotated on X, and U {U1,, U,...... UN} de- notes set of N user-chosen useful attributes annotated on X. For example, in the AudioMNIST dataset (Becker et al., 201 8), X can denote the high dimensional audio clips, and the user can choose S = { ("gender") } as a private attribute to remove, and choose U = ("spoken digit") as a useful attribute to preserve.
3. Problem Definition and Motivation
We aim to build a data obfuscation algorithm that could be inserted into data sharing or processing pipeline to remove certain private attributes from an input sample while preserving its useful attributes as well as general features for downstream tasks. To ensure practicality and soundness, we adhere to the restrictive SUIFT requirements as recently proposed in Chen et al. (2024), as elaborated below.
{Si,
Sensitivity suppression and Utiliiy preservation are the most basic requirements, suggesting that our method should ensure that private attributes are no longer predictable after obfuscation while the specified useful attributes are main- tained. For example, when sharing human voices, the user of our method may remove the speaker's gender information from the audio clip, while preserving the spoken content.
Following the assumptions made in Bertran al. (2019), we assume that all attributes in U follow finite categorical distributions, to ensure finite mutual information between X and each attribute. We also realistically assume that each attribute is deterministic when X is given, namely P(S X) and PX)) are degenerate distributions.
/nvariance of sample space dictating that the obfuscated data should remain in the same space as the original data, ensuring the seamless insertion of our method into existing pipelines and data re-usability, which is adopted in most of the recent works (Bertran et al., 2019; Dave et al., 2022; Chen et al., 2024; Malekzadeh et al., 2019).
With above definitions and assumptions, we can now for- mally describe our goal in the information theory framework as to find the optimal data obfuscation algorithm, denoted as Po (X')x, by solving the following optimization problem
N L Σ I(X (x
M
min
Σ -I(x,))-((Xxx,,,
X),
Feature management without annotation requires pre- serving the unspecified general features in the data. In the human voice example, general features may include the speaker's accent and age, etc., which are not explicitly spec- ified or labeled in the dataset. This requirement ensures broader usage of our method in the real world, where the downstream tasks are oftentimes unknown, plentiful, and constantly evolving (Huang al., 2018; Dave et al., 2022: Chen et al., 2024).
(1)
where the random variable X' denotes the obfuscated data, and our obfuscation algorithm Px(X' is parameterized by 0. I(.--) denotes Shannon mutual information, and A and ju are two hyperparameters used to trade-off privacy pro- tection and utility preservation. This optimization objective tries to simultaneously minimize the information leaked for St in X' to remove private attributes, maximize the informa- tion of U, in X' to preserve useful attributes, and maximize the information of original data X in X' to preserve the general features of the data. To summarize the relationship of random variables S, X, and X', we illustrate their probabilistic model in Figure 2. The optimization objective of Equation 1 is similar to the objec- tives used in Bertran et al. (2019); Malekzadeh et al. (2019); Chen et al. (2024). It also closely resembles the optimiza- tion objective in Privacy Funnel or Information Bottleneck literature with different focuses (Makhdoumi et al., 2014;
Finally, Theoretical basis is also required for the entire proposed framework to enhance soundness and correctness.
3.1. Informatiin-theerrticc Problem Definition for Private Attributes Protection
ceimmatioo--hhorttit
Following the high-level requirements, we formulate our problem into an information- theoretic framework for in- depth understanding. For presentation clarity, we use the fol- lowing notation convention in our paper: uppercase letters
3

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Tishby et al., 2000; Alemi et al., 2017: Hjelm et al., 2019).
Table I. Comparison of the NAG of baseline methods Motion Sense. We suppress "gender" and "ID" as private attributes, while preserving "activity" as a useful attribute. NAG-Protector suggests that this NAG is calculated using the protector's is adversarial classi- fier. NAG-Attacker suggests that this NAG is calculated using the attacker's adversarial classifier trained with Probing Attack.
3.2. The Vulnerability of Existing Private Attributes Protection Methods
State-of-the-art uuility-preserving private attributes protec- tion methods have demonstrated satisfactory performance in their respective papers in recent years. These methods are based on adversarial training, where the protector trains an adversarial classifier trying to correctly infer the private attribute St from obfuscated data X', and jointly trains the obfuscation algorithm P' (X'( to prevent the adversarial classifier from making correct inferences.
NAG-Preteccor (16)
NAG-Allader (9)
Method
gender (L)
(D)
activity (1)
gender (J)
ID (1)
activity (?)
ADV
14.2424
7.310.4
86.9111.5
62.927.5
37,3
86.91115
GAP
0.0:0.0
0.0:0.0
85.510..
64.2:0.22
49.5:0.4
85.510.7
MSDA
0.0.0.1
5.91099
93.01..5
65.4.1.6
46.6.15
93.010.5
BDQ
18.2420
5.810.4
90.5:1..3
56.0:9.4
30.81148
90.5223
PADAR
0.2:0.33
0.910.3
93.710.11
65.7.1.
49.8.1..5
93.7.0.1
MaSS
1.340.6
1.24022
93.8+0.1
68.1=00.5
49.8+00.2
93.8+0.1
Nevertheless, as widely discussed in the neighhoring fields of adversarial robustness (Carlini & Wagner, 2017; Atha- lye et al., 2018; Ilyas et al., 2019; Carlini et al., 2019), generated image/video detection (Yu et al., 2019; Wang et al., 2020; Masood et al., 2023), membership inference attacks (Carlini et al., 2022) and gradient inversion in fed- erated learning (Geiping et al.. 2020; Huang et al., 2021), adversarial training based methods share a common and critical weakness. That is, although their trained defender can safely defend against the jointly-trained ad adversary, they are vulnerable to potentially stronger or unseen ad adversaries, which can be obtained by certain tailored adaptive attack- ing strategies, or simply using longer training time, more computation power, larger datasets, etc.
NAG is lower for private attributes and when NAG is higher for useful attributes. For each private attribute, we calculate the NAG using both 1) the protector's adversarial classifier trained jointly with the obfuscation algorithm, and 2) the attacker's new adversarial classifier trained with moderate Probing Attack setup after the obfuscation algorithm is de- ployed. The detailed deseriptions of the NAG's definition and experimental settings can be found in Section 5.
We can observe from the results shown in Table 1 that, for private attributes, all methods achieved low NAG with the protector's adversarial classifier but significantly higher NAG with the attacker's is adversarial classifier. These experi- ment results substantiate existing works' severe vulnerabil- ity to the simple Probing Attack method, indicating their impracticality in challenging real-world deployment.
Despite the existence of many advanced attacking strate- gies in these neighboring fields listed above, we empiri- cally find out that in the context of private attributes protec- tion, an even simpler and more realistic attacking method can effectively break the ofate-of-the-art methods, which formally deseribed below. Given an obfuscation al- gorithm (X') the attacker may repetitively get an original sample and its associated attributes (Jr, 8, u) from Pa (X, S, U); feeds the original sample into the obfus- cation algorithm to get the corresponding obfuscated sam- ple x' ~ Px = x); and thus collect a dataset of (z, s, u, 2²) tuples. Then, this collected dataset is utilized to train a new adversarial classifier in a supervised manner. In the rest of this paper, we call this attacking method the Probing Attack. The Probing Attack is realistic in that it does not have assumptions on the training protocol or model structures of the obfuscation algorithm, and can be applied to either deterministic or stochastic obfuscation algorithms.
To further examine adversarial training based works' vulner- ability, we present an informition-theortiiic interpretation of this phenomenon in Appendix C.
4. Our Method: PASS
To build a data obfuscation model that is robust to Probing Attack, we propose PASS, Private Attributes protection with Stochastic data Substitution, which abandons adversarial training but adopts novel idea of stochastic data substitu- tion. Specifically, we propose first to draw subset from the training dataset Duain. which is denoted as the substitution dataset D Dsunstitute- Then, for each input original sample x, instead of transforming x into an obfuscated sample, we propose to strategically replace x with a sample x' in the substitution dataset Suubstitue according to our designed sub- stitution probability, such that the attacker infer the private attributes of the original sample x from the substituted sample x', but can still infer the useful attributes and some general features of x from x'.
We conduct a motivational experiment on the Motion Sense dataset to reveal Probing Attack's negative impact on ex- isting methods, where we suppress "gender" and "ID" as private attributes and preserve "activity" as useful attribute. We use the Normalized Accuracy Gain (NAG) proposed in Chen et al. (2024) as the metric, which is generally pro- portional to the accuracy of a classifier trained on X' for each attribute. The obfuscation is considered better when
not correctly
We propose to parameterize our substitution probability with a neural network, and then train it with our novel loss
4

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Approximate Loss Function
Information- theoretic Objective L
Stochastic Data Substitution
Original Sample x
Embedding f(x)
Es
Derive
I(X; S)
Private Attributes Sj Protection
Substitution Probability PdX'=x||Xxx
Derive
Useful
Lty
I(X;; Up
Attributes U, Preservation
Embedding g(x)
Substitute Samples x'
Derive
General
Ex
I(X: X)
Features
Preservation
Figure 3. The overview of PASS design. PASS stochastically replaces each original sample x with substitute sample x' according to probability (X' x'|x z). Pa(X's x =) is used to calculate approximate loss functions Ls.. Luş and Lx, which S.), S.), ((x'; Ug)) and 1(X'; X) respectively, and are responsible for protecting private attribute S.
are theoretically derived from ((x'; preserving useful attribute Ug and preserving general features, respectively.
function, which is rigorously derived from our information- theoretic objective Equation 1. An overview of PASS is shown in Figure 3, and the design details will elaborated in the next sections.
mated for each mini-batch during training. Therefore, we propose novel, fully differentiable loss function to train Xx) which is derived step-by-step from T. and theoretically valid in mini-batched based training.
In the rest of this section, will focus the formulation of L. For deeper understanding, please refer to Appendix D for detailed discussions on 1) the theoretical reason why L cannot be accurately estimated, 2) the step-by-step deriva- tion of L and proof of its theoretical validity, and 3) an intuitive explanation of i. with a running example.
4.1. Neural Network-based Stochastic Data Substitution
To unify the discussions of PASS and existing private attributes protection methods into the same information- theoretic framework, we reuse the notation (X to denote substitution probability, where Px x'|X x) denotes the probability of substituting the original sample a with the substitute sample x' Dsuhstitute and denotes all the parameters in PASS.
i can be decomposed into several loss terms
M
N Σ j=1
Ls
Lu, mLx,
(3)
To calculate Xx) we propose first input each orig- inal input sample into a neural network to calculate embedding, denoted as fz)). Then, we obtain learnable embedding g(x') ) for each substitute sample x' E Suubstitut- Next, we calculate (X'( based on the cosine similarity between each pair of embeddings f(x) and g(2') as
where Ls., Le, and Lx derived from I(X:: T(X;U and T(X'; X) respectively, and are responsible for protecting each private attribute S,, preserving each use- ful attribute Uy and preserving general features, respectively, x and μ are the same trade-off hyperparameters used in T.. We will focus on each them below.
coot((l),stt>"1>>>
Private Attributes Protection. To remove the information of each private attribute S, from X', we minimize (() by minimizing Ls,, which is derived to be
Po(X' =XX'=2122 =
Ex"ED (o)
(2)
where cos(...) is the cosine similarity, is the temperature hyper-parameter which is set to 0.01 for all of our experi- ments. Similar designs can be found widely in contrastive learning literature (Oord et al., 2018; Chen et al., 2020).
Ls. = EP..((.) [H()
(4)
where (1---) denotes conditional Shannon entropy. The conditional distribution P(X'| is calculated as
4.2. Loss Function Derivation
P(X' (S)) (... (P [X)
(5)
Ideally, to achieve our goal of dullity-preserving private at- tributes protection, P,(X'|X (X) should be trained to minimize our coommation-thhoreeee optimization objective L defined in Equation 1. Unfortunately, T. cannot be accurately esti-
where Pana (XI can be interpreted as all æ with each class of private attribute S, in the dataset. Taking expectation over Paai(XX|S)) is estimated by averaging over all x with S, in mini-batch.
5

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Useful Attributes Preservation. Extending the notations in Section 3.1, denote the data distribution of substiuute as ( U"), where U" {U1, U22,... UN} J are called substitute useful attributes, which represents the useful at- tributes directly annotated on X'.
4.3. Training and Inference Procedures
Our design follows standard two-phase machine learn- ing workflow, consisting of training and inference. During training, compute Pe (X) using Equation 2 for mini-batch of samples from Durain then calculate our loss function according to Equation 3 and update the parameters via backpropagation. We summarize the pseudo-code for training in 1.
To preserve the useful attributes Uj. we propose to encour- age the substitute useful attributes U'j to be similar to the original useful attribute Uy. For the AudioMNIST example where we choose "spoken digit" the useful attribute, for original audio with the spoken digit "I", we try to encourage the substitute audio also to have the spoken digit "I". To achieve this, we propose to minimize Lt, defined as:
Once training is complete, inference proceeds by freezing the model parameters 0. For each unseen test-set sample E Diest we compute the substitution probability Po(X' z'|x = 2) and draw x' accordingly to substitute z. The inference pseudo-code is detailed in Algorithm 2.
LG, = log My (X),() [- log = (6)
where U, denotes the support of Uy. denotes the cardinal- ity. log [4 is coefficient to adjust LU,- that the scale of
Algorithm 1 PASS Training Pseudo-code
Lu, matches I(X'; U,). Taking expectation over Pdara(X) estimated by averaging over a mini-batch. P(UJIX) de- notes the expected Uj of X, which can be calculated as:
Require: training dataset D substitution dataset
Dsubstitne model parameters 0
Ensure: trained model parameters A
1: while not reached max number epochs do
P(U)IX) = EP((X(X) [Pamn(U)]|**]
sample a mini-batch from Dirain compute f(z) for æ in mini-batch
4: compute g(æ²) for x' D sabatitute
As shown in Appendix D.2, Lu, can also be rigorously derived from UU)
5: compute Po(X' x XX x) using f(x) and g(x")
(Equation 2)
General Feature Preservation. To preserve general fea- tures, we maximize [X'; X) by minimizing the conditional entropy of P& (X) which can be written as minimizing the loss term Lx:
6: compute using Po(X' = x'| = x) (Equation 3)
update I with øi 09
end while
9: return 0
Lx E Pa (X) (X)))
(8)
where taking expectation over Panta (X) is also estimated by averaging over a mini-batch.
Algorithm PASS Inference Pseudo-code
Next, we will show that the expectation of L. over mini- batches is proportional to upperbound of L defined in Equation 1, suggesting that minimizing f. can lead to the minimization of L. confirming its theoretical validity.
Require: original sample x, substitution dataset D Subsstitute
model parameters 0
Ensure: substituted sample x!
I: compute f(x)
2: compute g(ø') for x' E D substitute
Theorem 4.1. For L and L defined in Equation I and Equa- tion 3 respectively, for μ S. N, we can have
3: compute Po(X' = x' x'|X = x) using f(x) and g((²)
(Equation 2)
draw x' E Duubstitue according to Pa(X' = x'|X x)
E i 1.
(9)
return x'
where the expectation is taken over all mini-batcher C' is a constant defined as
PASS has lower training and inference computational over- head compared with state-of-the-art methods, because its (x( is parameterized an embedding extraction fol- lowed by a cosine similarity, whereas the mate-of-the-ane methods typically parameterize Pe( as an end-to- end data reconstruction neural network (Ronneberger et al., 2015; Cao et al., 2022), which may require more computa- tion.
C -M - 1)) log( D (ubistitute)
Σ H(U,) + (10)
Please refer to Appendix D.2 for detailed proof. The con- stant is independent of model's parameters 8 and can be calculated before training to estimate T. during training.
6

---

PASS: Private Attributes Protection with Stochastic Data Substitution
4.4. Theoretical Analysis for Entangled Attributes
GAP (Huang et al., 2018), MSDA (Malekzadeh al., 2019), BDQ (Kumawat & Nagahara, 2022), PPDAR (Wu et al., 2020), and MaSS (Chen et al., 2024), whose detailed de- scriptions are presented in Section 2.
Intuitively, when the useful attributes and private attributes highly correlated or entangled, we need to sacrifice the utility of useful attributes to extent achieve pri- vate attributes protection. To theoretically describe this phenomenon in the information theory framework, and to provide an estimation of the extent of the utility sacrifice, we present the following theorem, revealing that both (X'; U2) and I(X;; X) are bounded by I(X'; St) and cannot be arbi- trarily large.
Evaluation Metries and Default Probing Attack's Set- ting. To measure the performance of the obfuscation on imbalanced datasets, we adopt the metric Normalized Ac- curacy Gain (NAG) proposed by Chen al. (2024), which can ensure that all attributes are measured on the same scale. The NAG for private attribute S; is defined as
Theorem 4.2. For any E (1, M) any I S
N,. and u = [U,li E J]. we can have
+ Σ 1(X:;U)) ( 1(X'; + H(4])) JEJ
(II)
NAG(S:) max (0. Acc(Si) - ACC (Si)
"ACrnusupp- (Ss) ACC.uessing (S:)
(13)
where C()) denotes the total correlation, H-I--) denotes conditional Shannon entropy. We can also have
where ASc(S:) is the accuracy of a classifier trained on the obfuscated data and the ground truth label of attribute S. Accgusssing (5:) is the accuracy of the majority classi- fier for S,, serving
x) S S.) + (X)S)).
(12)
lower bound of the Acc(S.). And Accno_sappe (S:) is the accuracy of a classifier trained on the original data and the ground truth label of attribute S. serv- ing as an upper bound of Acc(S.).
Please refer to Appendix A for detailed proof. This theorem can be viewed as an extension of the analysis in Chen et al.
(2024). The terms H())s:). C(4)), and H())S;) can all be calculated before training to estimate the extent of utility sacrifice, and to track the training progress.
A critical difference between our evaluation protocol and baselines lies in that, we use the attacker's adversarial clas- sifier trained with the Probing Attack to calculate NAG and thus measure the obfuscation performance, enhancing the practicality and realisticity of the evaluation. In the de- fault Probing Attack's setting, the attacker has the API of the trained obfuscation model Po (x X)) and its training dataset. The attacker adopts a medium sized neural network based adversarial classifier and trains on the obfuscated data using the Probing Attack.
4.5. Theoretical Interpretation within Local Differential Privacy Framework
As elaborated in Appendix B, under certain assumptions, our information- thheoretccc problem formulation can also be interpreted within the framework of local differential pri- vacy (LDP) (Yang et al., 2023). In this context, PASS can be viewed as an LDP mechanism, more specifically, an ex- tension of the classical randomized response methods. The distinctiveness of PASS from classical methods lies in its capability to operate over high-dimensional data domains, its explicit focus on utility preservation, and its foundation in anformation-theoretii principles.
To evaluate whether our method and baselines can preserve the data's general features, we hide some attributes during training and only reveal them during evaluation to verify if they can be preserved. We call these attributes hidden use- ful attributes and denote them as F3 for k E {[.....,K K}.
NAG for each useful attribute NAG(U,) and each hidden useful attribute (FG(F)) is defined and calculated in the same way as NAG(S.). Higher NAG suggests that this at- tribute's 's information is well preserved in X'. Therefore, the performance of an obfuscation method is considered better when each NAG(S,) is lower and when each NAG(U,) and each (F.( is higher.
5. Experiments
5.1. Experimental Setup
Datasets. We thoroughly evaluated PASS on three multi- attribute benchmark datasets, each representing a different application of a different modality. These datasets include AudioMNIST (Becker al., 2018), containing recordings of human voices; Motion Sense (Malekzadeh et al., 2019), con- sisting of human activity sensory signals; and CelebA (Liu et al., 2015), containing facial images. More detailed de- scriptions of datasets can be found in Appendix E.
To facilitate a fair comparison between different methods, we propose novel scalar metric, mean Normalized Accu- racy Gain (mNAG), to measure the trade-off between private attributes protection and useful attributes preservation in a comprehensive way. mNAG is defined as the average of the NAG for useful attributes (including hidden useful at- tributes), minus the average of the NAG for private attributes,
Baselines. We primarily compare PASS against six state- of-the-art baseline methods: ALR (Bertran et al., 2019),
7

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Table Comparison of the NAG hetween PASS and baselines on AudioMNIST. We suppress "gender" as private attribute, while preserving "digit" as useful attribute. We take "accent", "age", and "ID" as hidden useful attributes to evaluate general feature preservation.
Table 4. Comparison of the NAG between PASS and haselines on Motion Sense. We suppress "gender" and "ID" as private attributes. while preserving "activity" as a useful attribute. WAG-unfinetunne means that this NAG is calculated with a classifier that is only pre-trained on original data but not finetuned on substituted data.
NAG (8)
NAG (%)
NAG-unfinetuned (3)
Method
mNAG (1))())
Method
mNAG (90) (f)
gender (1)
accent
age (1)
(t)
digit
gender (1)
(D)
activity (1)
activity
No suppe
100.0
1000
100.0
100.0
100.0
0.0
ADV
62,917..5
37,3110.7
86.9111.5
93.810.7
36.8.44.3
Guessing
0.0
0.0
0.0
GAP
64.2+0.2
49.5+t0.
85.5=00.7
10.2+33.5
28 28.6+00.6
GAP
ADV
784:22
71,4+12
13.3=26
61.9+3.3
6233+10
0.1+0.1
57.3+300
55.240.6
0.0:0.0
77.1#2.4
72.3+0.3
3,4+0.3
99:00000
21.2+00.4
99.+00.1
-.3110.8
-7.142244
1.0+1.6
MSDA
PPDAR
BDQ
65.711.4
65.411.6
$6.0:19.4
30.8:14.88
49,810.5
905:2233
93.0.0.5
93.720.1
5.519.6
0.010.0
0.0:10.0
37.0.1..0
47.1:19.3
36.0.0..8
BIQ
09.0-58
569955.8
47.7+559
68.1+5.7
99.7+0.11
-0.8+1.7
M&SS
65.1100.5
49.81.0.2
93.8.0..1
0.0100.0
36.3.10.3
PPDAR
81.72:10
68.43088
60.7::0.6
74.01:99
99.7::0.0
-6,010088
PASS
0.0=00.0
0.0+0..
98.1=0.3
97.6+0.3
98.1=0.3
MaSS
88.9-1.2
760+0.7
70:4+1.0
81.1+0.3
99.5+0.1
-7.240.8
PASS
0.0.0.0
46.41009
1.
49,710.4
96.5100.2
55.0..0.7
NAG on "gender" and significantly higher mNAG than all baselines, which strongly substantiated PASS's s capability on private attributes protection, and PASS's tolerance to the Probing Attack.
Table Comparison of the NAG of PASS on AUdioMNIST for different configurations. In each configuration, the attributes anno- tated with (S) are suppressed as private attributes, and the attributes annotated with (U) are preserved useful attributes.
Furthermore, we conduct an ablation study verify if PASS is robust to different combinations of useful and private attributes. The combinations and their corresponding re- sults are presented in Table 3, demonstrating that PASS can consistently achieve a high mNAG for all combinations. When useful attributes and private attributes are highly en- tangled (e.g., when we suppress "gender", "accent", "age"" but preserve "ID"), PASS manages to find a satisfactory compromise between privacy and utility automatically.
NAG (56)
mNAG (1)
gender
accent
ago
ID
digit
(S) 0.0=000
(U) 65 2+0.4
(U) 77.0=0.1
(U) 73.3+-0.4
(U) 03
76.90022
(S) 9.3
0.00.000
67.8
61161.20.2
03.42.44
69.710.4
(S) 0.102.02
0.0.1.0000
(S) 30.710.6
(U) (0)-10001044
(U) 9.
57.6:0.2
(8) 0.0000
0.0+000000
(8)00-00
00+000
(U) 9933-02
99.5-+022
which can be formally written as
1
mNAG-
NAG(U,) +
Σ (AG(F
We also conduct four more ablation studies on AudioM- NIST by varying 1) the coefficient X, 2) the coefficient μ., 3) the number of samples in the substitute dataset [D ss and 4) the distribution of the substitute dataset. As shown in Appendix F.1, PASS consistently maintains high perfor- mance across all different settings, showing robustness to hyperparameter changes.
M K
(14)
1 N N
We report the NAG and mNAG in the main paper, while the corresponding accuracy can be found in Appendix F.
Hyperparameters. Unless otherwise specified, we set A A= N and = 0.2NN throughout our experiments to bal- ance private attributes protection, useful attributes preserva- tion, and general feature preservation. The substitute dataset constructed by randomly sampling 4096 data points from the training dataset. All the experiments in this paper are conducted with three random seeds and then aggregated. Other hyperparameters for neural network structures, train- ing configurations, and datasets can be found in Appendix E.
5.3. Evaluation on Human Activity Sensory Data
17
In our subsequent study, we apply PASS to the task of anonymized activity recognition Motion Sense dataset, where we suppress "gender" and "ID" attributes while pre- serving the "activity" attribute. On "activity", in addition to the standard NAG, we also report the NAG calculated with an un- on-finetuned classifier, which is only pre-trained on origi- nal data X, without finetuning on X'. The results, presented in Table 4, show that PASS achieves NAG of for private at- tributes and a much higher mNAG than all baselines, which substantiated the effectiveness of PASS. Besides, PASS also achieved the highest NAG on "activity" with un-finetuned classifier, which shows that PASS can be plugged into an existing ML pipeline to protect private attributes without necessarily altering the downstream models. We visual- ize the results of data substitution in this experiment using
5.2. Evaluation on Human Voice Recording
We begin with the task of removing gender information from human voice recordings on AudioMNIST dataset, where we suppress "gender", and preserve "digit". We take "accent", "age", "ID" as hidden useful attributes to evaluate general feature preservation, As shown in Table 2, PASS exhibited (
8

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Table Comparison of the NAG between PASS and baselines CelehA. We suppress "Male" as private attribute, while preserving "Smiling" and "Young" as useful attributes, and we take "Attractive," "Mouth-Slightlyye and CheebhCheekboee""" as hidden useful
attributes to evaluate general feature preservation.
NAG (%)
Method
mNAG (%) (t)
Male ())
Smiling (f)
Young (1)
Attractive (t)
Mouth_Slighlly.u -ppeh (1)
High.Cheekbones (1)
ADV
99.911011
98.8+0.11
97.0+10.9
94,6=0.4
99.1=1011
97.0=10.5
-2.6:0.22
GAP
83.0:1.11
1.
45.4+1.0
7..6=1.1
61.1#2.1
75.61:0.7
-15.922.3
MSDA
91.61:0.7
99.8+0.2
92,441244
89,9=1.0
91.8+10.8
95.7=11.1
2.34-088
BDQ
99.7:10.1
98.810.2
96,3+10.8
94.1=0.6
98.9:10.4
97.0.10.3
-2.710.2
PPDAR
99.7+t0.1
98.9+0.3
97.2+112
94,4=0.6
99.0#t.11
97.0=#0,4
-2,4+0.3
MaSS
96.9110.1
97.2:10.2
86.2+
90.6+0.33
97.6.10.2
94.6:10.4
-3.710.44
PASS
4.9
98.3+0.1
78.6+
58.1
67.0+0.88
86.7=10.3
72,940.2
confusion matrices, as shown in Appendix F.2.
protection methods. To address this, we propose PASS, stochastic data substitution based method rooted rigor- ously in information theory, that overcomes the above weak- ness. The evaluation of PASS on three datasets substantiates PASS's broad applicability various applications.
We also experimented to show that PASS can safely tolerate the Probing Attack when the attacker has more data than the protector. Detailed results and analysis are presented in Appendix F.2.
Disclaimer
5.4. Evaluation on Facial Images
We then extend the application of PASS to removing gender information from facial images on CelebA dataset, where aim suppress the "Male" as private attribute while pre- serving "SSmiling", "Young" useful attributes, and evaluat- ing the general features preservation by taking "Attractive", -Open", Sipeetilltgtttpen,,,..... ful attributes. As displayed in Table 5, PASS achieved a near NAG on "Male", and a much higher mNAG than all the baselines, which highlights PASS's efficacy on images.
This paper was prepared for informational purposes with contributions from the Global Technology Applied Research center of JPMorgan Chase & Co. This paper is not product of Research Department of JPMorgan Chase & Co. or its affiliates. Neither JPMorgan Chase & Co. nor any of its affiliates makes any explicit implied representation warranty and none of them accept any liability in connection with this paper, including, without limitation, with respect to the completeness, accuracy, or reliability of the informa- tion contained herein and the potential legal, compliance, tax, or accounting effects thereof. This document is not intended as investment research or investment advice, or as recommendation, offer, or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating in any transaction.
"Mouth_Slightly
as hidden use-
We further compare PASS with four additional DP-based baselines: Laplace Mechanism (Additive Noise) (Dwork et al., 2006), DPPix (Fan, 2018), Snow (John et al., 2020), and Image (Xue et al., 2021). As shown in Table 17 in Appendix F.3, these methods show limited performance be- cause they primarily aim to prevent membership inference, which differs from our goal of protecting specific private attributes while preserving utility.
A Akknowledmmen
In addition, we conduct an ablation study showing PASS's robustness on different combinations of useful and private attributes on CelebA. And we also experiment to reveal PASS's superiority over the baseline SPAct (Dave et al., 2022). Please refer to Appendix F33 for results and analyses.
Authors Yizhuo Chen and Tarek Abdelzaher were supported in part by the Boeing Company and in part by NSF CNS 20-38817.
Impact Statement
6. Conclusion
We believe that there no ethical concern or negative so- cictal consequence related to this work. Our work benefits the protection of people's privacy in that it is proposed to remove the private attributes from the data in data sharing or processing pipelines.
In this paper, we theoretically and empirically demonstrate that the common weakness of adversarial training has a no- ticeable negative impact on state-of-the-art private attributes

---

PASS: Private Attributes Protection with Stochastic Data Substitution
References
Carlini, N., Chien, S., Nasr, M., Song, S., Terzis, A., and Tramer, F. Membership inference attacks from first prin- ciples. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. IEEE, 2022.
Abadi, M., Chu, A., Goodfellow, I., McMahan, H. B., Mironov, I., Talwar, K., and Zhang, L. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Caton, S. and Haas, C. Fairness in machine learning: A survey. ACM Computing Surveys, 56(7):1-38, 2024.
Ahmed, S., Alshater, M. M., El Ammari, A., and Hammami, H. Artificial intelligence and machine learning finance: A bibliometric review. Research in International Business and Finance, 61:101646, 2022.
Chaudhuri, A. and Mukerjee, R. Randomized response:
Theory and techniques. Routledge, 2020.
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. In International conference machine learning. pp. 1597-1607. PMLR, 2020.
Alemi, A. A., Fischer, I., Dillon, J. V., and Murphy, K. Deep variational information bottleneck. In In- teraationa Conference on Learning Representations, 2017. URL
Chen, Y., Chen, C.-F., Hsu, H., Hu, S., Pistoia, M., and Abdelzaher, T. F. Mass: Multi-attribute selective suppres- sion for seelit-preeerving data transformation from an information-theorete perspective. International Con- ference on Machine Learning, pp. 6519-6538. PMLR, 2024.
hwtp:/iwwwweeeeeeeeeeeeeeeeeeeee net forum?
id-HyxOzBceg.
Arachchige. M., Bertok, P., Khalil, I., Liu, D., Camtepe, S., and Atiquzzaman, M. Local differential privacy for deep learning. IEEE Internet of Things Journal, 7(7): 5827-5842, 2019.
Cover, T. M. Elements of information theory. John Wiley Sons, 1999.
Athalye, A., Carlini, N., and Wagner, D. Obfuscated gra- dients give false sense of security: Circumventing de- fenses to adversarial examples. In International confer- ence on machine learning. pp. 274-283, PMLR, 2018.
Dave, I. R., Chen, C., and Shah, M. Spact: Self-supervieed privacy preservation for action recognition. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20164-20173, 2022.
Becker, S., Ackermann, M., Lapuschkin, S., Müller, K.- R., and Samek, W. Interpreting and explaining deep neural networks for classification of audio signals. arXiv preprint arXiv:1807.04418 2018
Dwork, C., McSherry, F., Nissim, K., and Smith, A. Cali- brating noise to sensitivity in private data analysis, In The- ory of Cryptography Conference, pp. 265-284. Springer, 2006.
Bertran, M., Martinez, N., Papadaki, A., Qiu, Q., Rodrigues, M., Reeves, G., and Sapiro, G. Adversarially learned representations for information obfuscation and inference. In International Conference on Machine Learning, pp. 614 623. PMLR, 2019.
Dwork, C.,, Roth, A., et al. The algorithmic foundations of differential privacy. Foundations and Trends@ in Theo- retical Computer Science,
1-7171 2014.
Edwards, H. and Storkey, A. Censoring representations with adversary. In International Conference on Learning Representations, 2016.
Boulemtafes, A., Derhab, A., and Challal, Y. A review of privacy-preserving techniques for deep learning. Neuro- computing, 384:21-45, 2020.
Fan, L. Image pixelization with differential privacy. In IFIP Annual Conference on Data and Applications Security and Privacy, pp. 148-162. Springer, 2018.
Cao, H., Wang, Y., Chen, J., Jiang, D., Zhang, X., Tian, Q., and Wang, M. Swin-unet: Unet-like pure transformer for medical image segmentation. In European conference on computer vision, pp. 205-218. Springer, 2022.
Geiping. J., Bauermeister, H., Dröge, H., and Moeller, M. Inverting gradients-how easy is it to break privacy in federated learning? Advances in neural information processing systems, 33:16937-16947, 2020.
Carlini, N. and Wagner, D. Towards evaluating the robust- ness of neural networks. In 2017 ieee symposium security and privacy (sp), pp. 39-57. leee, 2017.
Hjelm, R. D., Fedorov, A., Lavoie-Marchildee Marchildon, S., Gre- wal, K., Bachman, P., Trischler, A., and Bengio, Y. Learning deep representations by mutual information estimation and maximization. In International Confer- ence on Learning Representations, 2019. URL https: / opppreeiiew net forum? d= 3331r3jocc..
Carlini, N., Athalye, A., Papernot, N., Brendel, W. Rauber, J., Tsipras, D., Goodfellow, I., Madry, A., and Kurakin, A. On evaluating adversarial robustness. arXiv preprint arXiv: 1et::990200000 1902.06705 2019.
10

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Hsu, W.-N., Bolte, B., Tsai, Y.-H. H., Lakhotia, K., Salakhutdinov, R., and Mohamed, A. Hubert: Self- supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Loshchilov, I. and Hutter, F. Decoupled weight decay reg- ularization. In International Conference Learning Representations, 2019. URL https:: /ttps//////e..ie..... net/
Madras, D., Creager, E.,, Pitassi, T., and Zemel, Learning adversarially fair and transferable representations. In International Conference on Machine Learning, pp. 3384- 3393. PMLR, 2018.
Huang, C., Kairouz, P., Chen, X., Sankar, L., and Ra- jagopal, R. Generative adversarial privacy. arXiv preprint 18X7.08800003306 2018.
Majeed, A. and Lee, S. Anonymization techniques for privacy preserving data publishing: A comprehensive survey. IEEE access, 9:8512-8545, 2020.
Huang, Y., Gupta, S., Song, Z., Li, K., and Arora, S. Evalu- ating gradient inversion attacks and defenses in federated learning. Advances in neural information processing systems, 34:7232-7241, 2021.
Makhdoumi, A., Salamatian, S., Fawaz, N., and Médard, M. From the information bottleneck to the privacy funnel. In 2014 IEEE Information Theory Workshop (ITW 2014). pp. 501-500- IEEE, 2014.
Hyas, A., Santurkar, S., Tsipras. D., Engstrom, L., Tran, B., and Madry, A. Adversarial examples are not bugs, they are features. Advances in neural information processing systems, 32, 2019.
Malekzadeh, M., Clegg, R. G., Cavallaro, A., and Haddadi, H. Mobile sensor data anonymization. In Proceedings of the international conference on internet of things design and implementation, pp- 49-58. 2019.
Jia, J. and Gong, N. Z. (AtriGuard)): A practical defense against attribute inference attacks via adversarial machine learning. In 27th USENIX Security Symposium (USENIX Security 18), pp. 513-529, 2018.
Masood, M., Nawaz, M., Malik, K. M., Javed, A., Irtaza, A., and Malik, H. Deepfakes generation and detection: State- of of-the-art, open challenges, countermeasures, and way forward. Applied intelligence, 53(4):3974 4026, 2023.
John, B., Liu, A., Xia, L., Koppal, S., and Jain, E. Let it snow: Adding pixel noise to protect the user's iden- tity. In ACM Symposium on Eye Tracking Research and Applications, pp. 2020.
Oord, A. v. d., Li, Y., and Vinyals, O. Representation learn- ing with contrastive predictive coding. arXiv preprint arXiv: 807.03348, 2018
Kasiviswanathan, S. P., Lee, K., Nissim, K., Raskhod- nikova, S., and Smith, A. What can we learn privately? SIAM Journal on Computing, 40(3):793-822 2011.
Qu, Y., Yu, S., Gao. L., and Niu, J. Big data set privacy preserving through sensitive attribute- grouping. In 2017 IEEE International Conference on Communications (ICC), pp. 1-6. IEEE, 2017.
Kheddar, H., Hemis, M., and Himeur, Y. Automatic speech recogmition using advanced deep learning approaches: A survey. Information Fusion, pp. 102422, 2024.
Rajendran, K., Jayabalan, M., and Rana, M. E. A study on k-anonymity, 1-diversity, and t-closeness techniques. IJCSNS, 17(12):172. 2017.
Kumar, P., Chauhan, S., and Awasthi, L. K. Artificial intel- ligence in healthcare: review, ethics, trust challenges & future research directions. Engineering Applications of Artificial Intelligence, 120:105894, 2023.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted iiterventionMMICCA. 2015: 18th International Confer- ence, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234 241. Springer, 2015.
Kumawat, S. and Nagahara, H. peeeee--reeeviing action recognition via motion difference quantization. In Eu- ropean Conference on Computer Vision, pp- 518-534. Springer, 2022.
Roy, P. and Boddeti, V. Mitigating information leakage in image representations: A maximum entropy approach. Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, pp. 2586-2594, 2019.
Li, N., Li, T., and Venkatasubramannin, S. I-closeness: Privacy beyond k-anonymity and I-diversity. In 2007 IEEE 23rd international conference data engineering. pp. 115. IEEE, 2006.
Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Samarati, P. and Sweeney, L. Protecting privacy when disclosing information: k-anonymity and its enforcement through generalization and suppression. 1998.

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Sarhan, M. H., Navab, N., Eslami, A., and Albarqouni, S. Fairness by learning orthogonal disentangled representa- tions. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part XXIX 16, pp. 746-761. Springer, 2020.
Yang, M., Guo, T., Zhu, T., Tjuawinata, I., Zhao, J., and Lam, K.-Y. Local differential privacy and its applica- tions: A comprehensive survey. Computer Standards & Interfaces, pp. 103827. 2023.
Yu, N., Davis, L. S., and Fritz, M. Attributing fake images gans: Learning and analyzing gan fingerprints. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7556-7566, 2019.
Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 815-823, 2015.
Zhang, X., Ji, S., and Wang. Differentially private releas- ing via deep generative model (technical report). arXiv preprint arXiv:1801.0111 2018.
Secinaro, S., Calandra, D., Secinaro, A., Muthurangu, V., and Biancone, P. The role of artificial intelligence in healthcare: structured literature review. BMC medical informatics and decision making, 21:1-23, 2021.
Sei, Y., Okumura, H., Takenouchi, T., and Ohsuga, A. Anonymization of sensitive quasi-identifiers for 1- diversity and t-closeness. JEEE transactions on depend- able and secure computing, 16(4):580-55) 2017.
Song, F., Ma, T., Tian, Y., and Al-Rodhaan, M. new method of privacy protection: random k-anonymous. leee Access, 7:75434-75445, 2019.
Sun, M., Wang, Q.. and Liu, Z. Human action image gen- eration with differential privacy. 2020 IEEE Inter- national Conference Multimedia and Expo (ICME). IEEE, 2020.
Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method, arXiv preprint physics/0004057, 2000.
Wang, S.-Y., Wang, O., Zhang, R., Owens, A., and Efros, A. A. Cnn-generated images are surprisingly easy to spot... for now. In Proceedings of the IEEE/CVF con- ference computer vision and paltern recognition, pp- 8695-8700 2020.
Wang, Y., Wu, X., and Hu, D. Using randomized response for differential privacy preserving data collection. In EDBT/ICDT Workshops, volume 1558. pp. 0090-6778, 2016.
Warner, S. L. Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American statistical association, 60(309):63 69, 1965.
Wu, Z., Wang, H., Wang, Z., Jin, H., and Wang. Z. Privacy- preserving deep action recognition: An adversarial leam- ing framework and new dataset. IEEE Tronsactions on Pattern Analysis and Machine Intelligence, 44(4):2126- 2139, 2020.
Xue, H., Liu, B., Ding, M., Zhu, T., Ye, D., Song, L., and Zhou, W. Dp-image: Differential privacy for image data in feature space. arXiv preprint arXiv:2103.00 07,077 2021.
12

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Appendix
A. Proof of Theorem 4.2
Proof. For the first inequality of Theorem 4.2. For all i E (I,...,M) and all S (UI,..., UN}. we can have
I(X; + H())))) C(U)
-.)) - (()S))) ([(())))) C(U)
+ - , +(4)
((X": H C(U)
2T(X:
(15)
=C(X',2)
=E UU) C(UX))
UEN
Σ ((X:U)
UEN
Similar to Chen et al. (2024), the second inequality of Theorem 4.2 can be proved as
I(X::))= =H(X') (X
=H(X') ((
< <H(X') (XX H(XIX',
(16)
=H(X") + (
=I(X';S,) + (XX(S))
B. Theoretical Analysis on the Connection with Local Differential Privacy
In this section, we will theoretically show that, under certain assumptions, our information theoretic optimization objective in Equation 1 can be converted into local differential privacy objective (Yang al., 2023).
We first describe the definition of local differential privacy follows. Let and ő be non-negative real numbers, and A be randomized algorithm taking original sample from dataset input. Let imA denotes the image of A. Then A satisfies (e, -locll differential privacy if for any S C imA, any pair of original data points 1 and z2:
P(A(z1) E S) < E P(A(2)) E S)++ +
(17)
Next, we formulate our goal of private attributes suppression into the local differential privacy framework. We first assume an almighty attacker with the optimal adversarial classifier that can always infer each private attribute S, from substituted data X' with the ground truth distribution P(S, X')). Then, for attribute S,, define the randomized inference algorithm A(x) for each original sample z as the almighty attacker's expected inference result on
P(A(z)) = EX(XIX==) [P))
(18)
Our goal can then be expressed as training substitution probability distribution (X such that A satisfies (c, -local differential privacy. This formulation means that, with our trained Po(' the almighty attacker's inference results on S, for any two data points 21 and 22 are similar, which means that even the almighty attacker cannot infer S reliably for each
Finally, we show that our goal's 's information theoretic formulation can be converted into our goal's 's local differential privacy formulation. Recall that we propose to minimize T T(X': : S,) in Equation to suppress S;, which can be written as:
T(X': St) = Pxx) (KT(((S))))I((SS))
(19)
13

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Suppose that after training Po(X'IX) using this objective, manage to achieve:
KL(P(S)XX' =))))P((.)) 5
(20)
for any x'. Then
the following theorem to prove that our A(x) achieve (0, -27)- differential privacy:
Theorem B.I. For A(x) defined in Equation 18, if (L(P(S = =(S)) S for any x', then for any S C imA, and any 2n and 12 we can have
P(A(z1) E S) < P(A(22) E ES) + 22
(21)
The proof of this theorem is as follows:
Proof. Using Pinsker's Inequality (Cover, 1999), for any x', we can have
sup P(S. E S| x') - P(S, E S)
KL(P(S = =1)]|((S))
(22)
2
Therefore, for any S and x', we have:
P(S: ES) 12 7
<P(S, E S|X' =x))
(23)
<P(S ES) //2
Then, we can use this inequality to prove:
P(A(z1) E -Ep((()
[P(S E S(X')
P(S E 8) /7
<En((()===)
?
=P(S, E) +
V
(24)
-Rpo(X((=0)
P(S, E ES) /
<Ep((IXX==)
| E SIX') + 227
=P(A(02) ES) + -27
C. Theoretical Analysis of Adversarial Training Based Methods Vulnerability to The Probing Attack
In this section, we will theoretically uncover the underlying reason for adversarial training based methods' Vulnerability to The Probing Attack. Specifically, we first prove that the mutual information estimated and minimized by adversarial training based methods, denoted as T (X'; S.), only lower bound for the true mutual information T(X"; St). The proof can be
14

---

PASS: Private Attributes Protection with Stochastic Data Substitution
written as:
-
P(S
EB(XX))XXY/ log Pa(S
E Tog P(S
(25)
Po(S
where (X)) denotes the protector's adversarial classifier, parameterized by neural network Therefore, adversarial training based methods can not guarantee the minimization ;(X'; by minimizing (X';; St). The remaining T(X;; S,) unknown and potentially unbounded. the attacker has stronger adversarial classifier than (S (x'), then the attacker exploit the remaining T(X'; SSq) breach the protection.
D. Detailed Explanations of Our Loss Function
D.1. Theoretical Reason Why Cannot Be Accurately Estimated in Mini-batch
To understand why L cannot be accurately estimated in mini-batch, we first introduce a random variable B, which denotes the index the mini-batch. follows uniform categorical distribution, where each index of mini-batch corresponds to a unique combination of samples in the samples in mini-batch B.
We use P-) denote that this distribution is calculated only using the
During the training process, we can only calculate an estimation for mutual information terms in L using all samples in each mini-batch. Taking T(X;; S)) as example, let us denote the estimation in mini-batch B as the conditional mutual information (; It can calculated as
P(X. S B) P(X
5() = S,
(26)
Then, let us denote the estimation of in each mini-batch B as L(B). It can be calculated as
M L(B) Σ
N
((x:
-
(27)
E=1
1=1
Then, can show that the expectation of L(B) over possible mini-batches is not equal L
Er(B)
[(B)]
N
-E((B)
(('::S1 Σ
I(X:U,1B3 ( (X::
j=1
(28)
M
N
# ((X Σ -xx;))-I(XXx'
where the inequality is achieved because the expectation of conditional mutual information is not necessarily equal to the mutual information.
In conclusion, mini-batched- based estimation L(B) is not an unbiased estimator for L and is not suitable for use as a loss function.
D.2. Theoretical Derivation of Our Loss Function And Proof of of Theorem 4.1
In this section, we will keep using the random variable defined in Appendix D.1 to denote the index of mini-batch. We will rewrite our equations in Section 4.2 more formally with B to elaborate our derivation and to prove Theorem 4.1. We
15

---

PASS: Private Attributes Protection with Stochastic Data Substitution
will omit the expectations when writing conditional entropy for clarity, except for B, whose expectations will still be written explicitly.
To derive from L, we first need to derive the following upperbound for L
M
T(x;
- A Σ
- xx)
==1
==1
M
(M- - x)((
Σ S(X
N
H(U)) +
H(U +
1=1
M
(29)
(M 11) log (D Puhsstitute
H(X
1=1
N
N Σ H(U (
H(,)++
==
5=1
Σ -(X + Σ (U(UUIX)) - -
+C
where H-)) and H-I--) denote Shannon entropy and conditional Shannon entropy respectively. C' is a constant defined in Theorem 4.1. The inequality is achieved because the entropy a random variable is smaller than or equal to the log cardinality of its support.
Next, we will focus on each term in Equation 29 separately.
Private Attributes Protection. In Equation 29, HX (S;) is responsible for suppressing each private attribute S, Similar to the reasoning in Appendix D.I, we can calculate estimation for (X')) using all the samples in mini-batch B. The estimation is denoted as Lss. which can be rewritten with B as
Ls. = ( B)
(30)
The expectation of Ls. over all is an upperbound for (X'(S.)
(X,))
=Ep())l-((X'Is) B)]
(31)
where the inequality is achieved because adding a condition cannot increase the entropy.
Therefore, we can manage to minimize - (X' by calculating and then minimize Ls;-
Useful Attributes Preservation. In Equation 29, (U, 1 is responsible for preserving each useful attribute Uy. To show the connection between (U))X'') - 1 and Lu,, we need to first derive another upperbound for H(U,|X') - -1, which can be written as
H(U,IX)) -
XH( -1
<H(U)IU?) -1
<(1 P(U): U,)) log M4 + 1-1
(32)
=(1 P(U) U,)) log 14
< log P(U = U,) log [U,]]
< P (X.U,) | log PUU = Us(x)) log Uy
16

---

PASS: Private Attributes Protection with Stochastic Data Substitution
where the first equation is achieved because U'3 is independent of U, given X'. The first inequality is achieved because removing condition cannot decrease the entropy. The second inequality is Fano's Inequality (Cover, 1999), and the fourth inequality is Jensen's 's Inequality.
When calculating an estimation of the above equation in mini-batch B, we can have
P(U = -) U,(X)
(33)
which is because all the variables independent of B when X is given. therefore, can prove that Tu,, is unbiased estimator
H(U,IX)) -1
(PanlX,,,) loo loglogPP0 P(U = UylX))] log [u,1 BPac | log P(U) U,|X, B))] log 1451
(34)
General Features Preservation. In Equation 29, (X responsible for preserving general features. Similar to the analysis above, when calculating an estimation of H (X' in a mini-batch B, we can have
HX B) = XX)
(35)
therefore, we can prove that fx is an unbiased estimator as
I( = BB [X B))
(36)
=Ep(B,
In conclusion, summarizing the equations above, we can prove Theorem 4.1 as
M
H(X + X Σ (xXX))))))
1)
(37)
<Br(p + ALB xx
=Bp(n)LL)+ +
D.3. Intuitive Explanation of Our Loss Function with AudioMNIST Example
The intuition behind Ls, as follows. Maximizing the conditional entropy of P(X'(S)) may encourage that the original samples with each class of S; can be substituted with wide range of x' E D suubstinuee which may further ensure each x' substitute many different with different St classes. Therefore, when the attacker observes x it cannot confidently infer which v with which class of S; is the original input sample, thus cannot infer the class of S; correctly.
Using the example of the AudioMNIST dataset, and supposing S; is "gender", our loss term Ls, tries to encourage that each a' can substitute both "male" speaker's audio and "female" speaker's audio, so that the attacker can not infer the speaker's gender of the x when observing a x'.
To preserve the useful attributes Uy. we propose to encourage the substitute useful attributes UÚ to be similar to the original useful attribute Uy. For the AudioMNIST example, supposing choose "spoken digit" the useful attribute, if there comes an audio with the spoken digit "1", then try to encourage the substitute audio also to have the spoken digit "1".
Lx may encourage each original sample x to be substituted by narrow range of I' E Dsubstitute which has counteracting effect on the loss term Ls.. When Lx and Lg, are both used to train Px their combined effect is to encourage that, although each r can only cover relatively narrow range of z', all the x with each class of S, may jointly cover wide range of x'. Consequently, each x' may only substitute narrow range of x, but these z are with different classes of S;
17

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Table Detailed configurationn of experiments datasets, models, and optimization techniques.
Experiment
Audio
Human activity
Facial image
Dataset
AudiOMNIST
Motion Sense
CelebA
# total data points
30000
74324
202599
Training-testing split
4:1
7:4
4:1
Optimizer
AdamW (Loshchilov & Hutter, 2019)
Learning rate
0.001
0,001
0.0001
Weight decay
0.0001
Learning rate scheduler
Cosine scheduler
Embeddings f(x: and g(x²) dimension
512
Xx training epochs
2000
200
50
Probing Attack training epochs
2000
200
50
Pre-trained FaceNet backbone
f(x) neural network structure
3-layer MLP
6-layer Convolutional NN
followed by 2-layer MLP
which still hinders the attacker from inferring S; from x', while ensuring that the downstream user can infer from x' with medium level of accuracy.
Keep using the AudioMNIST dataset with the private attribute "gender" as example. With both îx and L.s,, encourage that each x' only substitute a limited number of æ, but these contain both "male" speaker's 's audio and "female" speaker's audio.
E. Additional Experimental Setup
E.1. Datasets Desciptions
CelebA Dataset comprises 202,599 facial images, each annotated with 40 binary attributes. For our experiments, we selected six representative attributes. We used the official split for training and validation. All images were center-cropped to resolution of 160x160 pixels in preprocessing.
AudioMNIST Dataset includes human voice recordings in English, which contains 60 speakers speaking 10 digits. It is annotated with eight attributes. We selected attributes gender, accent, age. ID, and spoken digits for our experiments, which have 2, 16, 18, 60, and 10 classes, respectively. The dataset contains 30,000 audio clips, divided into 24,000 for training and 6,000 for validation. We enhance the experiment efficiency on the AudioMNIST dataset by converting the raw data into features using HuBERT-B (Hsu et al., 2021).
Motion Sense Dataset consists of accelerometer and gyroscope data recorded during six daily human activities. We focused on three attributes: gender, ID, and activity, with 2, 24, and classes, respectively. Following Malekzadeh et al. (2019), excluded the "sit" and "stand up" activities from our experiments; adopted the "trial" split strategy; used only the magnitudes of the gyroscope and accelerometer as input; normalized input to zero mean and unit standard deviation; and segmented the datasets into 74,324 samples, each with a length of 128.
E.2. Model and Optimization Configurations
To achieve better training stability and faster convergence, we adopt pre-trained FaceNet (Schroff' et al., 2015) backbone followed by 2-layer MLPs as the neural network for CelebA dataset. We used a 6-layer Convolutional neural network for the Motion Sense dataset. Please refer to Table for more detailed configurations of our experiments' datasets, models, and optimization techniques.
18

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Table 7. Comparison of the NAG and accuracy between PASS and haselines on AudioMNIST. We suppress "gender" as a private attrihute, while preserving "digit" useful attribute. We take "accent", "age", and "ID" hidden useful attributes evaluate general feature preservation.
NAG (Accuracy) (%)
Method
mNAG (%)
gender (1)
accent (1)
age (1)
ID (t)
digit (t)
No suppr.
100.0 (99.7)
100.0 (98.6)
100.0 (97.1)
100.0 (98.6)
100.0 (99.9)
0.0
Guessing
0.0 (80.0)
0.0 (68.3)
0.0 (16.7)
(1.7)
0.0 (10.0)
0.0
ADV
71,4=1.2 (.))
62.3=1.0 (66.8=0.88
55.2=0.6 (85.1
72.3=0033 (71.8+0.3)
99.8=0.1 (99.6+0.1)
1.011
GAP
13.3.12. (82.610.5)
0.1:1.11 1.....1..1
0.0.1.00 (68.310.0)
3,4.0.3 (4.911..)
(4..0..)))))
21.2.10.4 (29.0.10.3)
-7,11224
MSDA
78,4122 (95.520.6)
61.913.3 (66.512.6)
$7.333.005...1..0
77.112.4 (76.412.3)
99.810.0 (9.00.0.0)
4,310.88
BDQ
69.0+5.8 (93.6+1.1)
56.9+5.8 (62.4+4.7))
47.7+5.9 (82.++18)
68.1+5.7 (67.7+5.5)
99.7++011 (99.6+0.11
408++1.7
PPDAR
81.7210 (9.)1
68,420.8 (71.7=0.7)
60.720.6 (86.7200)
74.0=0.9 (73.4::.9)
99.7=2.00 (99.6.0.0)
-6,010.8
MaSS
88,9212 (97.5=0.2)
76.0=0.7 (77.8=0.5)
704.1.009966.0..
81.1=0033 (80.340.3)
99.5+0.1 (99.440.1)
-7.240.8
PASS
(7....00)))1...
46,4.10.9 (54.010.7)
27.661.66666..1..
49.7.10.4 (49.8..04)))
96.51002 (96.7.10.2)
55.0.10.7
Table Comparison of the NAG and accuracy of PASS on AudioMNIST for different configurations. In each configuration, the attributes annotated with (S) are suppressed private attributes, and the attributes annotated with (U) are preserved as uscful attributes.
NAG (Accuracy) (%)
Method
mNA (%)
gender
accent
age
ID
digit
No suppr.
(U) 100.0 (99.7)
(U) 100.0 (98.6)
(U) 100.0 100.0 (97.1)
(U) 100.0 (98.6)
(U) 100.0 (9..9
0.0
Guessing
(8) 0.0 (80.0)
(S) 0.0 (68.3)
(S) (16.7)
(1.77
0.0
(S) 0.0 (79)4=0.2)
(U) 65,2=0444 (88. 1+0.1)
(U) 720==011 (78.++0.1)
(U) 73.3+20.4 (72.7=0.4)
(U)92.0++.3 (92.7+0.2)
76,9=0.2
(S) 9.3 0..8
(S) 0.0=0.0 (67.9=0.1)
(U) 67.8 (71.2+0.2)
(U) 61.92002 61.22)
(U) 93.4 (93.910.4)
69.7=0.4
(S) 0.1.10.2 (79.610.44)
(S) 0.01 (6..0.0)))))
(S) 30.7.00.6 (414100.5)
(U) 40.0.0..4 (40.410.4)
(U) 95. 8...4 (96.110.4)
$7.610.2
(S) 0.0 (80.0.10.0)
(S) 0.0 (68.3.10.0)
(S) 0.0.0.00 (16.7.10.0) (16.6.00))
(S) 0.0....0. (1.7.100)
(U) 99.5.10022 (99.4.10.2)
99.5..0.2
Table Ablation study of varying the coefficient À on AudioMNIST. We suppress "gender" a private attribute, while preserving "digit" useful attribute. We take "accent", "age", and "ID" as hidden useful attributes evaluate general feature preservation.
NAG (Accuracy) (5)
Method
mNAG (St)
gender (.)
accent (1)
age (1)
ID (?)
digit (f)
No suppe.
100.0 (99.7)
100.0 (98.6)
100.0 (97.1)
100.0 (98.6)
100.0 (99.9)
0.0
Guessing
0.0 (80.0)
0.0 (68.3)
0.0 (16.7)
(1.7
0.0 (10.0)
0.0
0.1 N/M
0.020.0 (80.0=0.0)
45.0:10.5 (52.9=0.4)
25.2:: (76.0:0.44
48.510.6
(48.710.6)
88.510.7 (89.510..)
51.810.7
0,2 N/M
0.0:0.0 (80.0=000)
45.3:10.3 0..2)
25.7:114 1:0.4)
49.3:0.1
(49.410..) 1)
91.5:.0. (92.2:10.5)
53.0+:0.4
0.5 N/M
0.0+0.0 (80.0=000)
4..3:10.7
(54.7=0.6)
28.4:t1.11 (76.9:0.3)
50.8:0.5
(50.9:0.5)
95.0++0.2 (95.4++0.2
55.4:10.6
PASS
N/M
0.0=0.0 (09.09079.920.. (79.9=0.1)
46.410.9 (5..0=0.7)
27.62:.6 (76.7+0.5)
49.710.4 (49.8:10.4)
96.510.2 (96.7+10.2
55.0110.7
N/M
0.0=0.0 (80.0=0.0)
47.840.5 (5.4)
51.340.7 (51.4:0.7)
97.8++0.0 (97.9-10.))
56.440.4
N/M
0.1:0.1 (80.0=0.0)
46.2:10.7 (53.8:0.55)
27.0 (76.5:044)
49,9:10.8 (50.0:10.8)
99.0:10.1 (98.9:10.1)
55.4:00.5
Table 10. Ablation study of varying the coefficient AudiOMNIST. We suppress "gender" a private attribute, while preserving "digit" as useful attribute. We take "accent", "age", and "ID" as hidden useful attributes to evaluate general feature preservation.
NAG (Accuracy) (%)
Method
mNAG (%) (1)
gender (1)
uccent (1)
age (f)
ID (1)
digit (1)
No suppr.
100.0 (99.7)
100.0 (98.6)
100.0 (97.1)
100.0 (98.6)
100.0 (99.9)
0.0
Guessing
0.0 (80.0)
0.0 (68.3)
0.0 (16.7)
0.0 (1.7)
0.0 (10.0)
0.0
0.00
0.0=0.0 (.0.0.0.80.0::000 (80.0:10.0)
1.0.0..016..::00
0.0:0.00 (68.3110.0)
0.0:0.00 (1.710.0)
97.320.4 (97.4.1.4)
24.3:10.1
0.01
0.0-0..080.0000.00 (80.0++0.0)
2.9+0.3 (19.0++.22
0.0+0.0 (68.3++0.0)
6.4+0.1 (7.9+0.1)
99.8+0.11 (99.7+0.11
27.3++0.1
0.02
0.0-0..080.0000000 (80.0+00.))
7.5++0.8 (22.7+0..)
0.0+0.0 (68.3++0.0)
12.7+00.1 (13.9+0.1)
99.8+0000 (99.6+0..0
30.0++0.2
PASS
0.05
0.0=0.0 (80.0::0.0)
23.5:10.5 (35.6:10.4)
4.1:1..3 (69.6110.1)
28.5::..3
(29.2:1.3)
98.9= (98.910.1)
38.8:10.3
0.10
0.0=0.0 (80.0=t0.0)
40.5++0.7 (49.3+00.)
20.4+1..8 (74.5=0.5)
44.3+0.7 (44.6+0.7)
97.9=0.1 (97.9+0.0)
50,8+00.6
0.20
0.0=0.0 (79.9+0.1)
46.4++099
(54.0+0.7)
27.6+ (7.57=0.5)
49.7+
(49.8+0.4)
96.5+0.2 (96.7+002)
55.0++0.7
0.50
0.0 (79.8+0.1)
44.9+0.2 (528++01)
25.1++1.3 (75.9+0.4)
48.5++0.5 (48.6+0.5)
91.4+0.9 (92.1+0.8)
52,4+0.66
19

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Table 1. Ablation study of varying the number samples in the substitute dataset (denoted as (Dutstione)) 1 on AudioMNIST. We suppress "gender" as private attribute, while preserving "digit" as useful attribute. We take "accent", "age". and "ID" as hidden useful attributes to evaluate general feature preservation.
NAG (Accuracy) (5))
Method
D sutetnnel
mNAG (5) (t)
gender (J)
accent (t)
age (1)
(t)
digit (?)
No suppe.
100.0 (99.7)
100.0 (98.6)
100.0 (97.1)
100.0 (98.6)
100.0 (99.9)
0.0
Guessing
0.0 (80.0)
(68.3)
0.0 (16.7)
(1.7)
(10.0)
0.0
1024
0.0::0.0 (80.0+:0.0)
41.6::0.4 (50.1=:0.4)
20.6::..5 (74.6=0.1)
45.6=0.3
(45.920.33)
96.8=0.4 (97. 07.00+0.44
51.1+0.2
2048
0.0::0.0 (80.0::0.0)
45.8++0.9 (5..5
24.9:t:6.6 075)
49.0=:0.7 (49.2+0.7)
96.6=0.44 (9,4)))..)
54.1++0.7
4096
0.0::..0 (79.9:10.1)
46,410.9 (54.0=0.7)
27.62:.6 (76.7=0.5)
49.7=0.4 (49.8+0.4)
96.520.2 0.22)
55.010.77
PASS
8192
0.0+0.1 (80.0++0.0)
46. 1:0.4 (53.7+00.)
26. 26.1 (76.2=0.6)
49.8+0.2 (50.0+0.2)
96.7=0.1 1))1
54.6+00.6
16384
0.0+0.0 (80.0++0.0)
16.2+1.1 (53.8+0.9)
25.5+112 (76.0=0.4)
49.5=0.6
(49.7=0.5)
96.8=0.3 (97.0=0.2)
54.5+0.2
24000
0.0+0.0 (80.0++0.0)
45.7++0.7 (53.4++0.6)
26,7++0.6 (76.4+0.2)
49.1+00.8 (49.2+0.7)
07.
54.6+00.4
Table 12. Ablation study of varying the attribute distribution of "gender"" and "digit" on substitute dataset on AudioMNIST. We suppress "gender" as a private attribute, while preserving "digit" as useful attribute. We take "accent", "age", and "ID" as hidden useful attributes to evaluate general feature preservation.
Method
"gender" distribution
"digit" distribution
NAG (Accurccy)
mNNG (f)
gender (0)
accens (t)
age (t)
(t)
digit(t)
No suppe. Guessing
100.0 (99.7)
(96.6)
(97.1)
100.0 (98.6)
100.0
0.0
0.0 (80.0)
(.0.5)
0.0 (16.7)
0.0
0.0.(10:))
90% Male 10%
Female
10% 0-9
0.0::0:0 (80.0:10.0)
28.410.7 07.2))0.2)
51.2::0.2
(51.3:1:.2)
36
(97.03997,0000 (97.0100.3)
56.1:00.
80% Male 20%
Female
10% (1-)
0.0+00: (79 (79.9+0.1)
46.4-0.9
(54.0+0.7)
27.6++16 (-0.5)
49.7+0.4 (49.8+004)
96.5-0.2,000 (96.7+0.22)
55.0+0.7
Male 50%
Fomale
0-9
0.1.1022 (.0.0.)..)
4 (5.55.3)
28.110.7 (76...0.2)
50.510.5 10.5)
9..0.03 (96.
$54110.1
PASS
20% Male 80%
Fomale
10% 0.9
0.0:00.1 (80.0:00.0)
47.4.0.9
(58.8:00.8)
27.8:334
(76.7=100)
50.8:0.8
150.9.0.8) 81
96.2-02
(96.84:0.2)
55.61112
10% Male 90%
Female
10% 0-9
0.0100.1 (80.0.10.)
47.8..06 (55. 110.5)
27.410.7 (7..6.0.2)
51,21002 (S) 2.
96...02 (96.6 (96.6.0..)
55.710.1
20%
Fomale
0.7.5
0:0::0:0 (29.0)
46.720.5
(54.2+0.4)
27.6i40. (76.200.))
50.0:00 (50.140.5)
96.0=033
(96.310.3)
55.1=0.1
80% Male 20%
Fomale
50% 5.56668 1-9
0.0+0.0
0.0-0.0079.99000 (79.9+0.)
44.3-0.2 (52.5++01)
25.3+415 (76.0-0.)
48.1+0.3 (48.3++0.3)
93.0004 (94.4+04))
52.9+0.1
F. Additional Experiment results
F.1. Additional Results on AudioMNISST
In addition measuring the experiment results in NAG in Table 2 and Table 3, we also measure these experiment results in accuracy, shown in Table 7 and Table 8.
We then conduct ablation studies to show PASS's hyper-parameters stability. Our first ablation study adopts the same setting as in Table 7, except that we gradually change À from 0. 0.1N/M to 5N/M to examine its impact on PASS's performance. The results in Table 9 show that PASS consistently achieves near-O NAG on "gender" and stable overall performance across a wide range of values.
Similarly, our second ablation study varies μ from 0 to 0.50 evaluate its influence on PASS. As shown in Table 10, PASS consistently achieves O NAG the private attribute "gender" for all μ.. The NAG for the useful attribute "digit" peaks 0.01, while the NAG for other hidden useful attributes peaks at a much higher value of μ 0.20. This behavior is well expected because is designed as the coefficient for I [(X:; X), where higher μ tends to trade useful attributes preservation for general feature preservation.
Next, our third ablation study evaluates PASS's stability to the number of samples in the substitution dataset. Again, we adopt the setting as Table 7, except that we gradually change the number of samples in the substitution dataset from 1024 to 24000 (the training dataset has 24000 samples). As shown in Table 11, PASS obtains highly consistent mNAG for all different numbers of samples.
Finally, in our fourth ablation study, we construct multiple substitution datasets with varying attribute distributions: 1) for the sensitive attribute "gender", we varied its distribution from "90% Male / 10% Female" to "10% Male / 99% Female"; 2) for the useful attribute "digit", we varied its distribution from uniform distribution "10% 0-9" to a highly skewed distribution "50% 0/ / 5.6% 1-9". The results, shown in Table A below, demonstrate that PASS consistently maintains high performance across all these different substitution dataset configurations, even under highly imbalanced conditions.
20

---

PASS: Private Attributes Protection with Stochastic Data Substitution
These four ablation studies provide empirical evidence that PASS has strong stability against the variance of this hyper- parameter.
F.2. Additional Results on Motion Sense
Table 13. Comparison the NAG and accuracy of baseline methods on Motion Sense, suppress "gender" and "ID" as private attributes, while preserving "activity" as useful attribute. adversarial classifier. NAG-Attakerr suggests that this NAG is calculated using the attacker's adversarial classifier trained with Probing Attack.
suggests that this NAG is calculated using the protector's
NAG (Accuracy) (%)
Method
gender (J)
ID (4)
activity (1)
protector's 's classifier
attacker's classifier
protector's classifier
attacker's classifier
ADV
14.2=2.4 (62.8++10))
7.3+0.4 (11.7+0.3)
37.3+10.7 (37.5+
86.9+11.5 (91.0-55)))
GAP
0.0+00.0 (57.0=0.0)
64.2+0.2 01).3++.1)
(.7+0.0(477++. 0.0++0.0 14.7+0.1)
49.5++044 (48.1+0.3)
85.5+0.7 (90.3+0.3)
MSDA
0.0+0.1 (57.0+0.1)
65.4+1.6 (8388+0.7)
5.9+0.9 (10.5+0.8)
46.6+1.5 (45.6+1.3)
93.0+0.5 (94.2+0.3)
BDQ
18.2-2.0 (64.5+0.8)
56.0+9.4 (79.9+3.9)
5.8+0.4 (10.3+0.4)
30.8+14.8 (31.9+12.8)
90.5+2.3 (92.9+1.1)
PPDAR
0.2 05.4
65.7+1.4 (83.9:0.6)
0.910.3 (6.1+0.2)
49.8 (48.3+044)
93.7=0.1 (94.5=0.1)
MaSS
1.3+0.6 (57.5=0.3)
65.1 (83.7:0.2)
1.2:0.2 (6.440.1)
49.8:0.22 (48.4+0.2)
93,8=0.1 (94.620.0)
Table 14. Comparison the NAG and accuracy between PASS and baselines on Motion Sense. We suppress "gender" and "ID" as private attributes, while preserving "activity" as useful attributes. NAG-unfinetuned means that this NAG is calculated with a classifier that is only pre-trained on original data but not finetuned on substituted data.
NAG (Accuracy) (%)
NAG-unfinctuned (Accuracy) (%)
Method
mNAG (%) (1)
gender (4)
ID (L)
activity (1)
activity (t)
No suppr.
100.0 (98.0)
100.0 (91.7)
100.0 (97.8)
100.0 (97.8)
0.0
Guessing
0.0 (57.0)
0.0 (5.3)
0.0 (46.6)
0.0 (46.6)
0.0
ADV
62.911..5 (82.8:+31)
37.3 (37.5
86.9#11.5
(91.0+55.))
93.8+0.77 (94.6=20.4)
36.8:14.3
GAP
64.2=0.2 18).
49.5=0.4 (48.14448811:000 (4..)))0.3
85.5+0.7 (90.3=0.3)
10.2 (51.8+1.8)
28.6:+0.6
MSDA
65.4:1..6 (83.8+0.7)
46.6=1.5 1.3)
93.0+0.5 (94.2=0.3)
5.549.66 (42.5+13.0)
37.0+22.0
BDQ
56.0:994 (79.9+3.9)
30.8+14.8 131.9+12.)
90.5+2.3 (92.9=1.1)
0.0+0.00 (22.6+9.2)
47.1+9.3
PPDAR
65.7=11.4 (83.9:00))
49.8=0.5 (48.3++0.4)
93.740.11 (94.5=0.1)
0.0+0000 (16.14
36.0+10.8
MaSS
65.1 (83.7+0.2)
49.8=0.2 (48.44+02))
93.8+0.1 (94.6=0.0)
0.0+0000 (14.8+0011
36.3++0.3
PASS
0.0. (57.0000))
0.0=0.00 (4.660.1)
98.110.3 (9..)
97.610.3 (9..510.1)
98.1-10.3
In addition measuring the experiment results in NAG in Table 1 and Table 4, we also measure these experiment results in accuracy, shown in Table 13 and Table 14 respectively.
To examine PASS's stochastic data substitution behavior closely, we visualize the results of data substitution using confusion matrices, as shown in Figure 4. We can observe that, for the useful attribute "activity", the diagonal values in the confusion matrix are significantly larger than the other values, which shows that most original samples are substituted by a sample with the same "activity" class. On the contrary, for private attributes "gender" and "ID", the rows and columns of the confusion matrix are highly independent, showing that original samples tend to be substituted by samples with highly random "gender" and "ID" classes. These results reveal the underlying logic of PASS's effectiveness.
We conduct another experiment to show that PASS can safely tolerate the Probing Attack even when the attacker has a larger dataset than the protector. In this experiment, we use the same setting as Table 14, except that the protector only has access to 50% of the training dataset to train PASS (and baselines), while the attacker has access to the entire training dataset to perform the Probing Attack. The results, as shown in Table 15, show that PASS achieves an mNAG that is still higher than all the baselines and is only 2.4% lower than when the protector has the entire training dataset.
21

---

PASS: Private Attributes Protection with Stochastic Data Substitution
9.0
1330
1920
01
13mn
0.1
(a) gender
(b) ID
(c) activity
Figure The confusion matrices of the stochastic data substitution results of PASS. The confusion matrices are calculated for the experiment shown in Table 4, where suppress "gender", "id" private attributes and preserve "activity" as useful attribute. We report confusion matrices for each attribute on test Motion Sense. The value in i-th row and j-th column the confusion matrix represents the fraction of the original samples with the i-th class substituted by the substitute samples with the j-th class.
Table Comparison the NAG and accuracy between PASS and basclines Motion Sense. We suppress "gender" and "ID" as private attributes, while preserving "activity" useful attributes. NAG-unfinetuned means that this NAG is calculated with classifier that is only tranted on original data but not finetuned on substituted data, In this experiment, protector only access 50% of the training set, while the attacker has access to the entire training set.
NAG (Accuracy) (%)
Method
mNAG (%) (t)
gender ({)
ID (1)
activity (1)
No suppr.
100.0 (98.0)
100.0 (91.7)
100.0 (97.8)
0.0
Guessing
0.0 (57.0)
0.0 (5.3)
0.0 (46.6)
0.0
ADV
69.540.6 (85.5+0.2)
47.310.3 (4..1
90.242.2 (92.7#1.1)
31.8
GAP
68.0++0.7 (84.9+0.3)
53.8+0.1 (51.7=0.1)
87.3+0.2 (91.2+0.1)
26.4+:0.7
MSDA
64.3 (83.3
44.910.9 (44.1
88.5+0.4 (91.9.0.2)
33.940.8
BDQ
67.2+ (84.5+0..)
47.0+2.66 (45.9+2.2)
90.9+0.5 (93.1+0.3)
33,8+0.77
PPDAR
67.8 (84.810.2)
51.411.2 (49.7+110)
91.410.3 (93.340.2)
31.810.4
MaSS
68.0+0.1 (84.910.1)
53.1+0.1 (51.2+0.11
91.6+004 (93.5+0.2)
31.0+0.4
PASS
0.0:0..0 (57.0+0.0)
10.00.04.711....
95.7+0.2 (95.6
95.710.2
22

---

PASS: Private Attributes Protection with Stochastic Data Substitution
F.3. Additional Results on CelebA
Table 16. Comparison the NAG and accuracy between PASS and baselines on CelebA. We suppress "Male" as private attribute, while preserving "Smiling" and "Young" as useful attributes, and we take "Attractive," "Mouth SrauuhhSlighhyyOpene and "heekhhoee as
hidden useful attributes evaluate general feature preservation.
NAG (Accuracy) (%)
Method
mNAG (%) (f)
(t)
Smiling (1)
Young (?)
Attractive (1)
Mouth Slightly Open (t)
High. Cheekbunes (t)
No suppr. Guessing
100.0 (98.9)
100.0 (92.9)
(87.9)
100,0 (82.0)
100.0 (94.0)
100.0 (87.6)
0.0
0.9
0.0 (50.8)
0.0 (75.2)
0.0 (50.8)
(51.))
(53.4)
0.0
ADV
99.9.0.1 (98.812 01:000
98.810.1 (9..5=0.0)
97.0.2099
(87.6.10.1)
94.610.4
(80.3.10.))
99.110.11 (93.7.0.1))
97.0.00.5 (86.610.2)
.2.610.2
GAP
83.0=1.1 (92.2=0044
75.9=133 (82. 8=0.5)
45,4=3.0 (81 (8100:00)) .0:0.4)
77.6#1.1 (75.0=t0.4
61.1=2.1
(77.3=+0.9)
75.6=0.7 (79 (79.3));
-15,9=2.3
MSDA
91.6:0077 (95 51:0.3)
99.82:022 (92.8 0.11)
92.42244 087.0:0033)
89.911.0 (78.8.10.3)
91.8100.8 (90.5
95.7a:11 (86.11:0.4)
2.3:00.8
BDQ
99.7=0.1 (98.7+0.0)
98,8=0.2 (92.4=01)
96.3=0.8 (87.5=0.1)
94.120.6 (80.2:+0.2)
98.9=0.4 (93 193.6t0..1)
97.0=0.3 (86.620.1)
-2.7=0.2
PPDAR
99.7-0.1
(98.7200.1
98.920.3 (92.5=0.1)
2
94,420.6
(80.2:10.2)
99.010.11 (93.61:0.0)
97.0=0.4 (86.610.1)
-24400.3
MaSS
96.9=0.1 (97.7=0.0)
97.2+0.2 (91. 01.8)
86.2=1.4
(86.2=0.2)
90.640.3 (79.0:+0.1
97.6=0.2 (93.0+00.)) I)
94.6=0.4 (85 8+0.1)
-3.7=0.4
PASS
4.910.55 (61.3.0.2)
98.31001 (92.2=0.0)
78.6.10.8 (85.2200.)
58.1 (68.9.09))))
67.010.8 (79.910.)
86.7.0.3 (83.1..0)))
72.910.2
Table 17. Comparison of the NAG and accuracy between PASS and DP-based baselines on CelebA. We suppress "Male" a pri- vate attribute, while preserving "Smiling" and "Young" as useful attributes, and we take "Attractive" "Mouth Slightly Open," and "High Cheekbones" hidden useful attributes evaluate general feature preservation.
Method
NAG (Accuracy) (9)
mNAG (%)
Male (J)
Smiling (1)
Young (f)
Attractive
Mouth Slighlly Open (f)
High Cheekbones (f)
No suppr.
100.0 (9% 9)
(0 (92.9)
100.0 100.0(87.9)
100.0 (82.0)
1000 (94.0)
100.0 (87.6)
0.0
Guessing
0.0 (59.4)
(50.8)
0.0 (75.2)
0.0 (50.8)
(.1.))
(53.4)
0.0
Snow
97.8==0.1 (98.0+0.1)
93.7=00.4 (91. (91.3-1011)
91,9++0.5 (84.8+0.2)
95.7:10.1 191.1=0.00
$0.4+1.2 (85 (:0.2)
84.7503 (77.24007)
.8540.3
DPPix
94.51.0.2 (96.710.1)
81.710.2 (86. 186.)
86.810.77
(83.110.22)
91.4110.2 (89.3110.1)
63.610.55 (8..
78.110.5 (75.210.1)
-14.210.1
Laplave Mochanism
91.0::0.2 (95.3110. 1)
79.3:0011 (85. (1).1))..1)
87,0+0022
(83.2200.)
89.8:10.5 (88. 6:10.2)
60.81166 (83.020.2)
81,2104 (76. (76.110.1)
-1141005
DP DP-Image
79.6++0.1 1 (90.8+00
68 (M.55++.22 (80.5++0.1)
79.8+0.1 (8077+0.1)
79.8+40.2 (84.4-00.1
55.0+1.0 (K2.2
78.7+066 (75.3+0.2)
-7.2+02
PASS
4.910..5 (61.3100.2)
(92.20.11
78.6100.8 (85.210.1)
$8.1 (68.9.10.9)
67.010..8 (79.910.3)
86.710.3 (83.1.00)))
72.910.2
Table 18. Comparison of the NAG and accuracy between PASS and SPAct on CelebA. We suppress "Male" as a private attribute, while preserving "Smiling" and "Young" as useful attributes.
NAG (Accuracy) (%)
Method
mNAG (%) (f)
Male (L)
Smiling (f)
Young (t)
No suppr.
100.0 (98.9)
100.0 (92.9)
100.0 (87.9)
0.0
Guessing
0.0 (59.4)
0.0 (50.8)
0.0 (75.2)
0.0
SPAct
81.344.2 1.7.5
94.0# 090.4
59.4#3.3 (82.8+0.4)
-4.6+2.77
PASS
4.9+ (61.3.00.2)
98.310.1 (92.2=0.0)
78.6 (85.2+0.1)
83.610.7
In addition to measuring the experiment results in NAG in Table 5, we also measure experiment results in accuracy, as shown in Table 16.
We further compared PASS with 4 additional DP-based baselines: Laplace Mechanism (Additive Noise) (Dwork et al., 2006), DPPix (Fan, 2018), Snow (John et al., 2020), and Im-le (Xue et al., 2021). As shown in Table 17, these DP-based methods exhibit limited performance in the Private Attribute Protection task, because they focus on preventing the inference of membership from obfuscated samples, which is not fully aligned with objective of preventing inference of specific private attributes from obfuscated samples while preserving utility.
Similar to the AudiMNIST dataset, we also conduct an experiment to show that PASS is robust to various combinations of private attributes and useful attributes. As shown in Table 19, PASS consistently achieves high mNAG for all combinations, even when the chosen private attributes and useful attributes are highly correlated (e.g., when we suppress "Smiling" but preserve "Mouth Slightly Open").
23

---

PASS: Private Attributes Protection with Stochastic Data Substitution
Table 19. Comparison of the NAG and accuracy of PASS on CelehA for different configurations. In each configuration, the attributes annotated with (S) are suppressed private attributes, and the attributes annotated with (U) are preserved as useful attributes.
(suppressed preserved (U) NAG (Accurscy)
(f)
Smiling
Young
Attractive
Mouth_Slightty-.Oee
Cigh_Chhskhones
No suppe.
(U) 100.0 (98.9)
(U) 100.0 (92.9)
(U) 100.0 (87.9)
(U) (82.0)
(U) 1000 (94.0)
(U) 100.0 (87.6) (8)00
0.0
Guessing
(50.8)
(.1()))
(8) 3.6.0.5 (60.81022)
(U) 96.70.0. (91.510..)
(U) 63.51119 (83.310.2)
(U) 73.1 (.3.6.002)
(U) (7)...333 (93.
(U) 89.71.33 (84. 1..1)
80.6.0.5
(S) 9.8 (63.240.3)
(S) 45.6 (70.0=0.3)
(U) 81.6177 (85.6=0.2)
(U) 907209(76.000.
(U) 50.4010 (85.77000
(U) 53.24.3 (71 6z204)
46.320.4
PASS
13,0+00.3 (64.9-0.]
(8)618807 (76-9-0.3)
00000000
739+0.3 (73.90011)
$4.000222 01.5-003)
745+077 (78 (78.9-0.))
55.6+0.1
(8) 5. 6:5055 (61 (64022)
(5) 2=04)
(S) 000000.0055220000
(S) 14,420.3 (55, (55.1
(U)97.1=02 (92.8=0.11
(U) 86.64404 (83.0=0.1)
68.8+0.00
(5) 6.800.4 (62.1+0.3)
(8)77.2-09(33--04)
(8) (000000075220000
(8) 16.8+0.. (56.D+0.1)
(3)-556-007777740003 (70.7+0.3)
(U) 03,7+08 (85.4-0.3)
64.4+00.4
Apart from these experiments, also compare PASS with SPAct(Dave et al., 2022) CelebA. However, since SPAct proposes to suppress general features while PASS aims preserving general features, their performances on general features are not comparable. Therefore, we only compare their performance on private attributes and useful attributes. In task suppressing the "Male" as private attribute while preserving "Smiling"", "Young" as useful attributes, we can observe from the results shown in Table 18, that PASS achieves significantly higher mNAG than SPAct, substantiating PASS's effectiveness.
G. Discussion on Scalability
While our proposed method PASS demonstrates strong performance in private attribute protection, it shares common limitation with existing state-of-the-arr methods: the need for retraining when the set of private attributes changes. However, compared to other baseline approaches, PASS offers a notable advantage in training efficiency, as its loss function is computed through embedding extraction followed by cosine similarity operation (please see Section 4.3 for details).
is also worth noting that although certain DP-based methods do not require retraining, their focus on broader membership protection inherently limits their effectiveness for the specific task of private attribute protection, demonstrated in Table 17 above.
24


Reviews:
[{"review_id": "Zie8bAVSUr", "reviewer": "Reviewer_RCzr", "summary": "This paper proposes a feature substitution method based on an information-theoretic objective to preserve privacy for certain data attributes. The method does not depend on any specific adversarial strategy, making it more robustness than existing adversarial-based approaches.\n\n#update after rebuttal:  the rebuttal has addressed most of the my concerns, except that I still think an end-to-end algorithm is much needed to make the approach easy to understand, and the limitations on scalability should be discussed in the final version. Overall I raised my rating and I hope the final version would incorporate these suggestions.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: \"Embedding function $g(x')$.\"\n\n**A1**: Thanks for pointing it out! Our embedding function $g(x')$ is implemented as an embedding layer with trainable parameters (e.g., like the embedding layer in language models). We will update it as $g_\\psi(x')$ to indicate its associated parameters clearly. \n\n**Q2**: \"Sample replacement vs. NN training.\"\n\n**A2**: We do not need to perform \"sample replacement\" during training. Because our novel loss function (Equation 3) can be calculated with solely the substitution probability $P_\\theta(X'|X)$, represented as a matrix for each batch, reducing training computation costs.\n\n**Q3**: \"The paper should also compare with other general defenses, such as DP...\"\n\n**A3**: Theoretically, as shown in Appendix B, PASS can be viewed as a Local DP method—specifically, a generalized form of Randomized Response with desirable DP properties. \n\nExperimentally, as suggested by the Reviewer, we further compared PASS with 4 additional DP-based baselines: Laplace Mechanism (Additive Noise) [1], DPPix [2], Snow [3], and DP-Image [4]. As shown in **Table A** below, these DP methods exhibit limited performance in the Private Attribute Protection task, because they focus on preventing the inference of **membership** from obfuscated samples, which is not fully aligned with our objective of preventing inference of **specific private attributes** while preserving utility.\n\n**Table A**. Comparison with DP-based baselines. The other settings are the same as Table 5. Results are averaged over 3 random seeds (STD is omitted to save space).\n\n|Method|Male NAG(↓)|Smiling NAG(↑)|Young NAG(↑)|Attractive NAG(↑)|Mouth\\_Slightly\\_Open NAG(↑)|High\\_Cheekbones NAG(↑)|mNAG(↑)|\n|-|-|-|-|-|-|-|-|\n|SNOW|97.8|93.7|91.9|95.7|80.4|84.7|-8.5|\n|DPPix|94.5|81.7|86.8|91.4|63.6|78.1|-14.2|\n|Laplace Mechanism|91.0|79.3|87.0|89.8|60.8|81.2|-11.4|\n|DP-Image|79.6|68.5|79.8|79.8|55.0|78.7|-7.2|\n|PASS|4.9|98.3|78.6|58.1|67.0|86.7|**72.9**|\n\n[1]: Dwork et al. \"Calibrating noise to sensitivity in private data analysis.\" Theory of Cryptography: Third Theory of Cryptography Conference, TCC. 2006.\n\n[2]: Fan, Liyue. \"Image pixelization with differential privacy.\" IFIP Annual Conference on Data and Applications Security and Privacy. 2018.\n\n[3]: John et al. \"Let it snow: Adding pixel noise to protect the user’s identity.\" ACM Symposium on Eye Tracking Research and Applications. 2020.\n\n[4]: Xue et al. \"Dp-image: Differential privacy for image data in feature space.\" arXiv preprint. 2021.\n\n\n**Q4**: \"...include comprehensive evaluations on key impacting factors. For example, the distribution and quantity of a substitution dataset...\"\n\n**A4**: Thanks for the suggestion! Below, we demonstrate PASS's strong stability across varying quantities and distributions of substitution datasets. First, an ablation study on substitution dataset **quantity** (Appendix F.1, Table 10) shows that PASS maintains high performance across a wide range of substitution dataset sizes (1024 to 24,000). Second, we conducted a new ablation study on the **distribution** of the substitution dataset on AudioMNIST, where we varied the distribution of: \n1. the sensitive attribute \"gender\", from \"90\\% Male / 10\\% Female\" to \"10\\% Male / 90\\% Female\".\n2. the useful attribute \"digit\" from \"10\\% 0-9\" to \"50\\% 0 / 5.6\\% 1-9\". \n\nAs shown in **Table B**, PASS achieved consistently high performance, even on highly skewed distributions.\n\n**Table B**. Ablation study on the distribution of substitution dataset. The other settings are the same as Table 2. Results are averaged over 3 random seeds (STD is omitted to save space).\n\n|gender distribution|digit distribution|gender NAG(↓)|accent NAG(↑)|age NAG(↑)|ID NAG(↑)|digit NAG(↑)|mNAG(↑)|\n|-|-|-|-|-|-|-|-|\n|90% Male / 10% Female|10% 0-9|0.0|47.9|28.4|51.2|96.9|56.1|\n|80% Male / 20% Female|10% 0-9|0.0|46.4|27.6|49.7|96.5|55.0|\n|50% Male / 50% Female|10% 0-9|0.1|47.0|28.1|50.5|96.5|55.4|\n|20% Male / 80% Female|10% 0-9|0.0|47.4|27.8|50.8|96.7|55.6|\n|10% Male / 90% Female|10% 0-9|0.0|47.8|27.4|51.2|96.3|55.7|\n|80% Male / 20% Female|30% 0 / 7.8% 1-9|0.0|46.7|27.6|50.0|96.0|55.1|\n|80% Male / 20% Female|50% 0 / 5.6% 1-9|0.0|44.3|25.3|48.1|93.9|52.9|\n\n**Q5**: \"...the method may be difficult to scale on features...\"\n\n**A5**: All baseline state-of-the-art Private Attribute Protection methods in this paper require retraining when private attributes change. In comparison, PASS has a lower training cost than them, as its loss can be computed without actual sample replacement (please see **A2** and the last paragraph of Section 4.1 for more details). \n\nFurthermore, PASS is designed with scalability in mind, as it supports any number of private and useful attributes without substantially increasing computational cost. \n\nIn addition, although some DP-based methods do not require retraining, their focus on general **membership protection** unavoidably limits their effectiveness on the Private Attributes Protection task, as shown in **A3** above.", "claims_and_evidence": "The claims are supported by both theoretical analysis and experimental evidence.", "methods_and_evaluation_criteria": "Key technical details are missing so it is hard to assess the soundness of the approach. Specifically, it is unclear from the paper how is this PASS method trained end-to-end. A formal algorithm depicting the steps is strongly suggested. Additional, there are missing details regarding:\n1. Embedding function g(x'). The paper states that it is a \"learnable\" embedding, but it seems that no trainable parameters are assigned. It is unclear how this embedding function is performed or chosen. \n2. Sample replacement vs. NN training. It is not clear how is the sample replacement step integrated with the NN training. e.g. Are all the samples replaced in the batch training? how often are parameters updated for each time that one or some samples are replaced?", "theoretical_claims": "No I didn't check the correctness of the proof.  However, one issue is that it is unclear how this method is compared with DP, both theoretically and experimentally.  The theoretical connection with DP seems to show that these are equivalent approaches  under certain assumptions, but it does not demonstrate that the approach is advantageous compared to DP.  It is suggested that the paper includes more discussions on the comparison with DP.", "experimental_designs_or_analyses": "The experiments include several datasets and baselines. However，the baseline methods are all adversarial-based. The paper should also compare with other general defenses, such as DP,  adding noise, sparsification.  \n\nIt is also suggested that the experiments include comprehensive evaluations on key impacting factors. For example, the distribution and quantity of a substitution dataset seems to be a key factor for the method to work well. However, it is not fully discussed how the composition of dataset affect the performance. \n\nAnother issue is that the method may be difficult to scale on features. It seems that adding additional features will involve retrain the entire model, which may be a challenge compared to DP.   The computation cost regarding training should also be presented.", "relation_to_broader_scientific_literature": "This paper builds upon the existing literature on privacy protection of sensitive attributes, offering new insights from an information-theoretic approach.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "I am a bit concerned that the boarder applicability of the proposed PASS approach maybe limited by its scalability on features or availability of abundant substitution dataset.  I suggest the paper discusses these aspects in more details.", "other_comments_or_suggestions": "No.", "questions_for_authors": "See the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b1GsFmboKD", "reviewer": "Reviewer_f4yf", "summary": "This paper addresses the challenge of protecting private attributes in machine learning (ML) services while preserving the utility of the data for downstream tasks. Existing methods primarily rely on adversarial training to remove private attributes, but the authors identify a fundamental vulnerability in these approaches, both theoretically and empirically.\n\nTo mitigate this issue, the paper introduces PASS, a novel stochastic substitution mechanism that replaces original samples with alternative ones based on a probabilistic framework. The approach is guided by a newly designed loss function, rigorously derived from an information-theoretic objective. Extensive experiments across multiple modalities—facial images, human activity sensor data, and voice recordings—demonstrate PASS's effectiveness and generalizability.\n\nThe proposed method offers a fresh perspective on privacy-preserving ML, particularly by moving away from adversarial training. The work is well-motivated, and the empirical results strengthen its claims.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: \"The experiments consider only a single private attribute per dataset, despite multiple useful attributes being present. Evaluating the method with multiple private attributes would strengthen the analysis.\"\n\n**A1**: Thanks for your question. This paper included experiments with multiple private attributes to demonstrate PASS’s scalability.\n- In Table 3, we experimented with up to 4 private attributes: \"gender\", \"accent\", \"age\", and \"ID\", on the AudioMNIST dataset.\n- In Tables 4 and 13, we experimented with 2 private attributes: \"gender\" and \"ID\", on the MotionSense dataset.\n- In Table 16, we experimented with up to 5 private attributes: \"Male\", \"Smiling\", \"Young\", \"Attractive\", and \"Mouth\\_Slightly\\_Open\", on the CelebA dataset.\n\nAcross all settings, PASS consistently delivered a strong performance, highlighting its ability to scale to varying numbers and combinations of private attributes. Please see Sections 5.2, 5.3, Appendices F.2 and F.3 for details.\n\n\n**Q2**: \"The NAG metric lacks a clear explanation. More details are needed on its computation, particularly how Acc\\_guessing is derived.\"\n\n**A2**: Thanks for pointing this out. We will include a more detailed explanation of NAG below as an extension to Section 5.1. NAG, proposed by Chen et al. [1], is a metric for evaluating Private Attribute Protection methods.  For private attribute $S_i$, the NAG  is defined as \n$$\n    NAG(S_i) = \\max\\left(0,\\frac{Acc(S_i) - Acc_\\text{guessing}(S_i)}{Acc_\\text{no-suppr.}(S_i) - Acc_\\text{guessing}(S_i)}\\right), \n$$\nwhere \n- $Acc(S_i)$ is the accuracy of a classifier trained to classify $S_i$ from the substituted sample $X'$.\n- $Acc_\\text{guessing}(S_i)$ is the accuracy of a majority classifier (a classifier that always predicts the most frequent class regardless of the input, i.e., a poor classifier that can only \"guess\"), which in most cases represents the lower bound performance of $Acc(S_i)$. \n- $Acc_\\text{no-suppr.}(S_i)$ is the classification accuracy of a classifier trained to classify $S_i$ from the original data $X$, which in most cases represents the upper bound performance of $Acc(S_i)$. \n\nBy normalizing $Acc(S_i)$ between the lower bound ($Acc_\\text{guessing}(S_i)$) and the upper bound ($Acc_\\text{no-suppr.}(S_i)$), we ensure that NAG is on a consistent scale across all attributes, regardless of whether an attribute is balanced or imbalanced, or whether it is easy or difficult to predict. As a result, NAG provides a fair and reliable measure of how well each private attribute is suppressed or preserved. This makes it a suitable and consistent metric for comparing the effectiveness of different Private Attribute Protection methods. We will include this detailed description of NAG in our revised paper.\n\n[1]: Chen, Yizhuo, et al. \"MaSS: multi-attribute selective suppression for utility-preserving data transformation from an information-theoretic perspective.\" Proceedings of the 41st International Conference on Machine Learning. 2024.\n\n**Q3**: \"What happens when we use other attributes like age is used as private?\"\n\n**A3**: Thanks for your question. This paper included experiments using \"age\" as a private attribute on the AudioMNIST dataset, as shown in Section 5.2, Table 3. Similarly, we experimented with \"Young\" as a private attribute on the CelebA dataset, detailed in Appendix F.2, Table 16. In both cases, PASS demonstrated a robust ability to suppress the private attribute \"age\" or \"Young\", while effectively preserving other useful attributes. \n\nThroughout the paper, we have evaluated PASS on a wide range of private attributes, including \"gender\", \"accent\", \"age\", \"ID\", \"Male\", \"Smiling\", \"Young\", \"Attractive\", and \"Mouth\\_Slightly\\_Open\". Across all these scenarios, PASS consistently achieved strong performance, reinforcing its broad applicability and versatility in handling diverse Private Attribute Protection tasks.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "I have checked proof of theorem 4.2 and section D in the appendix. It seems correct.", "experimental_designs_or_analyses": "Yes, I have checked all the experiments and it looks sound to me.", "relation_to_broader_scientific_literature": "This work significantly advances privacy-preserving ML by identifying fundamental weaknesses in adversarial training approaches and proposing PASS, a theoretically grounded and empirically validated alternative. By leveraging probabilistic sample substitution rather than adversarial representation learning, the method contributes to both the privacy literature and broader discussions on fairness and information-theoretic ML.", "essential_references_not_discussed": "Not sure.", "other_strengths_and_weaknesses": "Strengths:\n1. The paper is very well-written and easy to follow. Even for a non-expert, the key concepts are clearly conveyed.\nWeaknesses:\n1. The experiments consider only a single private attribute per dataset, despite multiple useful attributes being present. Evaluating the method with multiple private attributes would strengthen the analysis.\n\n2. The NAG metric lacks a clear explanation. More details are needed on its computation, particularly how Acc_guessing is derived.", "other_comments_or_suggestions": "Typo: Line 32: image/(vedio) detection.", "questions_for_authors": "What happens when we use other attributes like age is used as private?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DcAaKU7Zwr", "reviewer": "Reviewer_Acbi", "summary": "PASS (Private Attributes Protection with Stochastic Data Substitution) introduces a novel method to protect private attributes in machine learning datasets by replacing original data samples with others from a substitution dataset using a stochastic algorithm trained with an information-theoretic loss. Unlike existing adversarial training-based methods, which are vulnerable to probing attacks, PASS provides a stronger defense by offering a theoretical foundation for balancing privacy and utility. Empirical results on datasets like CelebA, Motion Sense, and AudioMNIST show that PASS effectively protects private attributes while preserving data utility.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**: \"...apply the \"unfinetuned\" classifier metric to other datasets...\"\n\n**A1**: Thanks for the suggestion! We applied the \"NAG-unfinetuned\" metric to the useful attributes in the AudioMNIST and CelebA datasets for consistency. The results are shown in **Tables A** and **B**, respectively. For unfinetuned classifiers, PASS still offers the best balance between privacy protection and utility preservation. This is due to our loss function, which encourages replacing each sample with others sharing the same useful attributes. That said, we acknowledge that fine-tuning downstream classifiers can further improve performance, and we plan to explicitly explore adapting PASS to better suit unfinetuned classifiers in future work.\n\n\n**Table A**. Comparison with baselines, using \"NAG-unfinetuned\" metric on useful attributes. The settings are the same as Table 2. NAG on the private attribute \"gender\" is also presented for reference. Results are averaged over 3 random seeds (STD is omitted to save space).\n\n|Method|gender NAG(↓)|digit NAG-unfinetuned(↑)|\n|-|-|-|\n|ADV|71.4|99.7|\n|GAP|13.3|0.3|\n|MSDA|78.4|0.0|\n|BDQ|69.0|5.9|\n|PPDAR|81.7|66.2|\n|MASS|88.9|31.4|\n|PASS|0.0|77.0|\n\n\n**Table B**. Comparison with baselines, using \"NAG-unfinetuned\" metric on useful attributes. The settings are the same as Table 5. NAG on the private attribute \"Male\" is also presented for reference. Results are averaged over 3 random seeds (STD is omitted to save space).\n\n|Method|Male NAG(↓)|Smiling NAG-unfinetuned(↑)|Young NAG-unfinetuned(↑)|\n|-|-|-|-|\n|ADV|99.9|99.9|98.3|\n|GAP|83.0|0.0|0.0|\n|MSDA|91.6|0.0|0.0|\n|BDQ|99.7|98.5|90.2|\n|PPDAR|99.7|98.1|92.7|\n|MASS|96.9|0.0|0.0|\n|PASS|4.9|85.2|49.1|\n\n\n**Q2**. \"...guidance on selecting optimal hyperparameters ($\\lambda$, $\\mu$)...\"\n\n**A2**. To guide the selection of $\\lambda$ and $\\mu$, we conducted the following ablation studies. For $\\lambda$, the results in **Table C** below show that PASS consistently achieves near-0 NAG on private attribute and stable overall performance across a wide range of values. Thus, we recommend the users to simply fix $\\lambda=N/M$ without extensive tuning, where $N$ and $M$ are numbers of useful and private attributes, respectively. This choice is used throughout this paper. For $\\mu$, results in Appendix F.1 Table 9 show that increasing $\\mu$ enhances general feature preservation but may slightly reduce useful attribute preservation. Importantly, privacy protection remains strong across all $\\mu$. Therefore, we suggest that $\\mu$ can be flexibly adjusted based on the relative importance of general features in users' specific tasks, without compromising privacy protection.\n\n\n**Table C**. Ablation study on $\\lambda$. The other settings are the same as Table 2. Results are averaged over 3 random seeds (STD is omitted to save space).\n\n|$\\lambda$|gender NAG(↓)|accent NAG(↑)|age NAG(↑)|ID NAG(↑)|digit NAG(↑)|mNAG(↑)|\n|-|-|-|-|-|-|-|\n|0.1N/M|0.0|45.0|25.2|48.5|88.5|51.8|\n|0.2N/M|0.0|45.3|25.7|49.3|91.5|53.0|\n|0.5N/M|0.0|47.3|28.4|50.8|95.0|55.4|\n|1N/M|0.0|46.4|27.6|49.7|96.5|55.0|\n|2N/M|0.0|47.8|28.5|51.3|97.8|56.4|\n|5N/M|0.1|46.2|27.0|49.9|99.0|55.4|\n\n**Q3**. \"...direct benchmarks against modern DP methods (e.g., DP-SGD)...\"\n\n**A3**. We compared PASS with four additional DP-based baselines: Laplace Mechanism (Additive Noise) [1], DPPix [2], Snow [3], and DP-Image [4]. Due to space limitations, the results are presented in **Table A** in our response to **Reviewer RCzr** above. These methods show limited effectiveness on the Private Attribute Protection task because they are designed to prevent inference of **membership** from obfuscated samples, which is not fully aligned with our objective of preventing inference of **specific private attributes** from obfuscated samples while preserving the utility.\n\nIn contrast, DP-SGD [5] aims to protect against membership inference **from a model’s trained parameters**, which is orthogonal to our goal—PASS focuses on preventing private attributes inference **from obfuscated samples**. Therefore, DP-SGD is not directly comparable to PASS, but could potentially be combined with PASS to provide more comprehensive privacy protection.\n\n\nReferences [1-4] are shown in our response **A3** to **Reviewer RCzr** due to space limitations.\n\n[5]: Abadi, Martin, et al. \"Deep learning with differential privacy.\" Proceedings of the 2016 ACM SIGSAC conference on computer and communications security. 2016.\n\n\n**Q4**. \"Discussing potential limitations... particularly in high-dimensional or highly imbalanced data...\"\n\n**A4**. Thanks for the suggestion! While we have demonstrated PASS's effectiveness on moderately high-dimensional data (e.g., 3×160×160 images in CelebA) and imbalanced datasets (e.g., 80\\% Male / 20\\% Female for \"gender\" in AudioMNIST), we acknowledge that handling even higher-dimensional or more imbalanced data may demand PASS to learn more representative features per sample, posing intrivial challenges to PASS's robustness and scalability.", "claims_and_evidence": "The claims made in this submission are mostly supported by empirical and theoretical evidence. However, there are still one concerning point: while PASS is compared to k-l-t privacy and LDP, it lacks direct benchmarks against modern differential privacy mechanisms such as DP-SGD. This makes the privacy guarantees of PASS less quantifiable and more difficult to evaluate in comparison to formal differential privacy methods.\n\nThe submission presents ablation studies (Tables 9, 10) demonstrating robustness to $\\mu$ and $|\\mathcal{D}_{\\text{substitute}}|$. However, it does not provide clear guidelines for selecting $\\lambda$ and $\\mu$ in practice, which could make it difficult for users to balance privacy-utility trade-offs without relying on trial-and-error tuning. Additionally, w", "methods_and_evaluation_criteria": "The paper primarily uses Normalized Accuracy Gain (NAG) and mean NAG (mNAG) as evaluation metrics. While NAG appropriately accounts for class imbalance, it depends on classifier accuracy, which could conflate the quality of obfuscation with classifier robustness. Incorporating additional metrics such as precision, recall, and F1-score could provide a more comprehensive and balanced evaluation of the method's performance.", "theoretical_claims": "I have verified the theoretical claims and proofs presented in the paper.", "experimental_designs_or_analyses": "1. In the Motion Sense experiments, the authors evaluate the performance of an \"unfinetuned\" classifier (a classifier only pre-trained on the original data without fine-tuning on the substituted data). While this provides some insight into the transferability of features, it's not clear why this metric is only used for the Motion Sense dataset and not the others. \n\n2. The paper states that hyperparameters are set to balance privacy protection, utility preservation, and general feature preservation. However, the specific process of hyperparameter tuning is not described in detail. This raises the possibility that the performance of PASS could be sensitive to the choice of hyperparameters.", "relation_to_broader_scientific_literature": "The paper introduces a novel stochastic data substitution approach that avoids adversarial training altogether. This method relates to broader research on randomized response techniques in privacy-preserving data analysis. However, PASS extends these concepts to high-dimensional data spaces and incorporates utility-preserving requirements, distinguishing it from traditional randomized response methods.", "essential_references_not_discussed": "As far as I know, no related works that are essential to understanding the (context for) key contributions of the paper, but are not currently cited/discussed in the paper.", "other_strengths_and_weaknesses": "Strengths\n\n1. PASS introduces a creative approach to private attribute protection by using stochastic data substitution instead of adversarial training. This represents an original combination of ideas from randomized response techniques and information theory.\n\n2. The method is rigorously grounded in information theory, with clear derivations and proofs connecting the practical loss function to the theoretical objectives. This provides a solid mathematical basis for the approach.\n\n3. PASS demonstrates effectiveness across multiple data modalities (audio, sensor data, images), suggesting it can generalize well to different types of private attribute protection tasks.\n\n4. By addressing the vulnerability of existing methods to probing attacks, PASS tackles a real-world concern in deploying privacy-preserving machine learning systems.\n\nWeaknesses\n\n1. While PASS is compared to several baselines, the evaluation lacks comparison to differential privacy methods, which are considered the gold standard for privacy guarantees.\n\n2. Although PASS shows empirical privacy protection, it does not provide formal privacy guarantees comparable to differential privacy's $\\epsilon$-$\\delta$ bounds. This makes it challenging to precisely quantify the level of privacy achieved.\n\n3. While some ablation studies are provided, there is limited guidance on selecting optimal hyperparameters (λ, μ) for balancing privacy and utility in practice.", "other_comments_or_suggestions": "1. It may be helpful to apply the \"unfinetuned\" classifier metric to other datasets for consistency and broader insights into transferability.  \n2. Providing more details on the hyperparameter tuning process would help clarify how privacy, utility, and feature preservation are balanced.  \n3. Adding direct benchmarks against modern DP methods (e.g., DP-SGD) could make the privacy guarantees more comparable and easier to quantify.  \n4. Discussing potential limitations of the stochastic data substitution approach, particularly in high-dimensional or highly imbalanced data, would strengthen the analysis.", "questions_for_authors": "1. Could you clarify why the \"unfinetuned\" classifier metric was only applied to the Motion Sense dataset and not to other datasets? Applying it more broadly could provide a more comprehensive view of feature transferability.  \n\n2. Could you provide more details on how the hyperparameters (e.g., $\\lambda$ and $\\mu$) were tuned? Understanding this process would clarify how privacy, utility, and feature preservation are balanced and help assess the generalizability of the approach.  \n\n3. Do you plan to include direct benchmarks against modern DP methods such as DP-SGD? This would make it easier to compare PASS’s privacy guarantees with those of established differential privacy techniques.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6KnAZj6h2Q", "reviewer": "Reviewer_aH9x", "summary": "This paper introduces PASS (Private Attributes protection with Stochastic data Substitution), a novel approach to protect private attributes in user data while preserving utility. Unlike existing adversarial training-based methods, PASS employs stochastic data substitution where each original sample is replaced with a sample from a substitute dataset according to learned probabilities. The method is derived from an information-theoretic framework and demonstrates superior robustness against probing attacks across multiple data modalities (images, sensor signals, speech).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. The paper overlooks memory requirements for storing substitute datasets and embeddings, particularly problematic for high-dimensional data applications.\n\n**A1**. Thanks for your question! Importantly, the substitution dataset itself does not need to be loaded into memory during inference. Instead, we only need to load the corresponding embeddings into memory to calculate the substitution probability $P_\\theta(X'|X)$. which greatly reduces memory usage.\n\nFurthermore, as demonstrated in the ablation study on substitution dataset size (Appendix F.1, Table 10), PASS maintains consistently strong performance across a wide range of substitution dataset sizes. This flexibility allows users to adapt the size of the substitution dataset based on their available storage and memory. For instance, if a user has limited secondary storage to store the substitution dataset, or limited memory to load its embeddings, then they can opt to use a smaller substitution dataset without substantial loss in performance.\n\nIn addition, users can further control memory usage by adjusting the dimension of the embeddings. Users with limited memory can reduce the embedding dimension to fit their resource constraints.\n\nThese characteristics allow PASS to be adapted to different hardware settings without compromising its core functionality.\n\n\n**Q2**. Critical visualizations comparing original and substituted samples are missing, which would better demonstrate the method's effectiveness. \n\n**A2**. Thanks for the insightful suggestion! We will visualize the original and the substituted samples in our revised paper to help readers understand PASS's behavior and showcase PASS's effectiveness.\n\n\n**Q3**. The substitute dataset construction relies on random sampling without justification or exploration of more optimal selection strategies.\n\n**A3**. Thanks for the thoughtful question! To better understand how the choice of substitution dataset affects PASS’s performance, we conducted a new ablation study on the AudioMNIST dataset. In this study, we constructed multiple substitution datasets with varying attribute distributions:\n\n1. For the sensitive attribute \"gender\", we varied its distribution from \"90\\% Male / 10\\% Female\" to \"10\\% Male / 90\\% Female\".\n\n2. For the useful attribute \"digit\", we varied its distribution from uniform distribution \"10\\% 0-9\" to a highly skewed distribution \"50\\% 0 / 5.6\\% 1-9\". \n\nThe results, shown in **Table A** below, demonstrate that PASS consistently maintains high performance across all these different substitution dataset configurations, even under highly imbalanced conditions. This ablation study provides empirical evidence that the specific selection of the substitution dataset has a limited impact on PASS's overall performance. Consequently, it may justify that a simple random sampling strategy is sufficient to construct an effective substitution dataset and achieve strong PASS performance.\n\nThat said, we agree that there may still be room to further enhance PASS's effectiveness with more advanced substitution dataset selection strategies. We plan to explore this direction in our future work.\n\n\n\n\n**Table A**. Ablation study on the distribution of substitution dataset. The other settings are the same as Table 2. Results are averaged over 3 random seeds (STD is omitted to save space).\n\n|gender distribution|digit distribution|gender NAG(↓)|accent NAG(↑)|age NAG(↑)|ID NAG(↑)|digit NAG(↑)|mNAG(↑)|\n|-|-|-|-|-|-|-|-|\n|90% Male / 10% Female|10% 0-9|0.0|47.9|28.4|51.2|96.9|56.1|\n|80% Male / 20% Female|10% 0-9|0.0|46.4|27.6|49.7|96.5|55.0|\n|50% Male / 50% Female|10% 0-9|0.1|47.0|28.1|50.5|96.5|55.4|\n|20% Male / 80% Female|10% 0-9|0.0|47.4|27.8|50.8|96.7|55.6|\n|10% Male / 90% Female|10% 0-9|0.0|47.8|27.4|51.2|96.3|55.7|\n|80% Male / 20% Female|30% 0 / 7.8% 1-9|0.0|46.7|27.6|50.0|96.0|55.1|\n|80% Male / 20% Female|50% 0 / 5.6% 1-9|0.0|44.3|25.3|48.1|93.9|52.9|", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "I examined the proofs for Theorem 4.1 (Appendix D.2) connecting the approximate loss Ĺ to the true objective L, and Theorem 4.2 (Appendix A) analyzing entangled attributes. Both proofs are mathematically sound, correctly applying information theory principles to establish bounds and relationships between mutual information terms. No issues were identified in the derivations.", "experimental_designs_or_analyses": "The experimental design is sound, comparing PASS against six state-of-the-art methods across three datasets. The ablation studies with varying hyperparameters demonstrate robustness. The asymmetric scenario (Table 13) where attackers have more data than protectors is a particularly strong validation. I reviewed all experimental sections, including the appendices showing additional results.", "relation_to_broader_scientific_literature": "PASS extends privacy protection literature by connecting local differential privacy with utility-preserving attribute protection. It builds upon information bottleneck/privacy funnel concepts (Makhdoumi et al., 2014; Alemi et al., 2016) but applies them in a novel way through data substitution rather than feature transformation. The work also relates to k-anonymity and l-diversity literature by ensuring multiple samples with different private attributes become indistinguishable.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Weaknesses:\n1.The paper overlooks memory requirements for storing substitute datasets and embeddings, particularly problematic for high-dimensional data applications.\n2.Critical visualizations comparing original and substituted samples are missing, which would better demonstrate the method's effectiveness.\n3.The substitute dataset construction relies on random sampling without justification or exploration of more optimal selection strategies.", "other_comments_or_suggestions": "No", "questions_for_authors": "No", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces PASS (Private Attributes protection with Stochastic data Substitution) for protecting private attributes in machine learning datasets while preserving utility for downstream tasks. The authors argue that existing adversarially training methods are fundamentally vulnerable to probing attacks. In contrast, PASS leverages a stochastic substitution mechanism guided by an information-theoretic objective. This mechanism probabilistically replaces original data samples with alternative ones drawn from a substitution dataset. Theoretical analysis and comprehensive experiments across diverse datasets (including images, sensor signals, and speech) are performed to demonstrate the effectiveness and generalizability of the proposed method.


Several reviewers appreciated the paper’s well-founded theoretical framework, particularly its derivation from information-theoretic principles. They also appreciated the avoidance of adversarial training approaches (unlike existing works). The reviewers also highlighted: the strength of the empirical evaluations, clarity of writing, and the inclusion of thorough appendices. However, multiple reviewers raised concerns about the lack of comparisons with differential privacy (DP) techniques, especially DP-SGD, which would have provided a stronger baseline for quantifying privacy guarantees. In response, the authors argue that DP-SGD purpose is different (and it aims to defend against MIAs). I personally agree with this point of the authors. Additionally, there were shared concerns about scalability and practicality. Specifically, the potential computational cost and memory demands.  While the authors responded that only embeddings need to be loaded in the memory, they still did not provide clear memory/computational complexity result in their response.
Decision: Accept (spotlight poster)