Venue: NAACL 2025

# On the Vulnerability of Text Sanitization

Meng Tong\({}^{1}\), Kejiang Chen\({}^{1*}\), Xiaojian Yuan\({}^{1}\), Jiayang Liu\({}^{2}\),

Weiming Zhang\({}^{1}\), Nenghai Yu\({}^{1}\), and Jie Zhang\({}^{3}\)

###### Abstract

Text sanitization, which employs differential privacy to replace sensitive tokens with new ones, represents a significant technique for privacy protection. Typically, its performance in preserving privacy is evaluated by measuring the attack success rate (ASR) of reconstruction attacks, where attackers attempt to recover the original tokens from the sanitized ones. However, current reconstruction attacks on text sanitization are developed empirically, making it challenging to accurately assess the effectiveness of sanitization. In this paper, we aim to provide a more accurate evaluation of sanitization effectiveness. Inspired by the works of Palamidessi et al. Alvim et al. (2015); Cherubin et al. (2019), we implement theoretically optimal reconstruction attacks targeting text sanitization. We derive their bounds on ASR as benchmarks for evaluating sanitization performance. For real-world applications, we propose two practical reconstruction attacks based on these theoretical findings. Our experimental results underscore the necessity of reassessing these overlooked risks. Notably, one of our attacks achieves a \(46.4\%\) improvement in ASR over the state-of-the-art baseline, with a privacy budget of \(\epsilon=4.0\) on the SST-2 dataset. Our code is available at: [https://github.com/mengtong0110/On-the-Vulnerability-of-Text-Sanitization](https://github.com/mengtong0110/On-the-Vulnerability-of-Text-Sanitization).

+
Footnote †: _Corresponding author._

+
Footnote †: _Corresponding author._

## 1 Introduction

With the advancement of natural language processing (NLP) Chang et al. (2023), concerns related to its privacy implications Kim et al. (2024) have significantly intensified. For instance, Samsung employees previously leaked the company's confidential meeting records by uploading documents to the cloud services of large language models (LLMs) Sam (2023). More recently, reports Chowdhary (2023); Wired (2023) have disclosed a concerning flaw in LLM-based chatbots OpenAI (2023), which could expose user data and put private information at risk.

To address these privacy concerns, text sanitization Chen et al. (2023); Meng et al. (2023) is now regarded as a crucial solution for privacy preservation in text and has been widely adopted by companies, such as Amazon Amazon (2024) and Microsoft Microsoft (2024b). Specifically, before users upload their documents to an untrusted cloud server for model training or inference, they can employ differential privacy (DP) Dwork (2006) to replace tokens corresponding to sensitive entities in the documents with newly sampled tokens. This method minimizes alterations in the uploaded data compared to other methods, such as synthetic data generation Hong et al. (2024); Saiteja et al. (2023).

While DP sanitizes sensitive tokens, recent works Zhou et al. (2023); Du et al. (2023) have demonstrated that attackers can still recover them from sanitized text. This finding underscores the significance of measuring the effectiveness of text sanitization. One of the principal methods for evaluating its effectiveness in privacy protection is through reconstruction attacks Balle et al. (2022);

Figure 1: The illustration of reconstruction attacks.



Yue et al., 2021), which aim to recover sensitive tokens from sanitized text, as illustrated in Figure 1.

**Existing Attacks.** In this context, several reconstruction attacks have been developed to evaluate the vulnerability of text sanitization: The _Embedding Inversion Attack_Qu et al. (2021) reconstructs tokens by selecting the token with the closest embedding distance to the sanitized token Song and Raghunathan (2020). The _Inversion BERT Attack_Kugler et al. (2021) trains an inversion model to map sanitized tokens back to their original forms. The _Mask Token Inference Attack_Yue et al. (2021) replaces each token in the sanitized text with a "[MASK]" token to predict the raw one. However, these attacks are empirically developed, which makes it difficult to accurately measure the effectiveness of sanitization (as discussed in Section 2).

**Our Proposal.** In this paper, we aim to address the evaluation gap in privacy protection for DP-based text sanitization. Inspired by the works of Palamidessi et al. (Alvim et al., 2015; Cherubin et al., 2019), who estimate privacy leakage using Bayesian inference, we introduce both the _Context-free Optimal Reconstruction Attack_ and the _Contextual Optimal Reconstruction Attack_ for assessing text sanitization. We also derive theoretical bounds on the attack success rate (ASR), which serve as benchmarks for evaluating its performance in protecting privacy. To support real-world implementation, we propose two practical attacks based on these theoretical findings. Experimental results demonstrate that our reconstruction attacks achieve higher ASRs than baseline methods.

**Our Contributions.** We summarize our main contributions as follows:

1. We implement theoretically context-free and contextual optimal reconstruction attacks against text sanitization, deriving their bounds on ASR to serve as benchmarks for evaluating the effectiveness of the sanitization.
2. Based on these theoretical findings, we propose two practical reconstruction attacks: _Context-free Bayesian Attack_ and _Contextual Bayesian Attack_. Notably, _Contextual Bayesian Attack_ is the first to transform the reconstruction attack to a classification task.
3. We reassessed text sanitization methods across four datasets using seven reconstruction attacks. Our experimental results underscore the necessity of paying closer attention to these overlooked risks. Specifically, _Contextual Bayesian Attack_ achieves a 46.4% improvement in ASR over the state-of-the-art baseline at \(\epsilon=4.0\) on the SST-2 dataset Socher et al. (2013a).

## 2 Preliminaries and Related Work

**Text Sanitization.** Text sanitization Chen et al. (2023); Yue et al. (2021) leverages differential privacy (DP) Dwork (2006) to replace sensitive tokens with newly sampled tokens. Specifically, let \(X\) represent the set of sensitive tokens and \(Y\) represent the set of sanitized tokens. An original sentence consists of sensitive tokens \(x_{1},x_{2},\ldots,x_{n}\), where \(x_{i}\in X\). Its sanitized sentence is \(y_{1},y_{2},\ldots,y_{n}\), where the token \(y_{i}\in Y\) is independently sampled by DP with the probability \(\Pr\left(y_{i}|x_{i}\right)\). Additionally, we define the sanitized context \(\mathbf{c}_{i}=y_{1},y_{2},\ldots,y_{i-1},y_{i+1},\ldots,y_{n}\) for the sanitized token \(y_{i}\) (or original token \(x_{i}\)), where \(\mathbf{c}_{i}\in C\) and \(C\) denotes the set of sanitized contexts.

In addition to text sanitization, recent studies Staab et al. (2024a, b) have demonstrated that LLMs can also protect private information.

**Definition of Sensitive Tokens.** In previous research Meng et al. (2023); Papadopoulou et al. (2022); Plant et al. (2021), they have made varying choices on the definition of sensitive tokens. In our study, we set the sensitive tokens in accordance with the corresponding defense method of text sanitization Chen et al. (2023); Yue et al. (2021).

**Reconstruction Attacks.** The _Embedding Inversion Attack_Qu et al. (2021) identifies the token closest to a sanitized token by comparing their embedding distances Song and Raghunathan (2020) as the reconstructed token. The _Inversion BERT Attack_Kugler et al. (2021) uses a trained model to convert sanitized representations back to their original tokens on a one-to-one basis. However, neither method has performed a comprehensive evaluation of sanitization effectiveness, which should consider the context of the sanitized tokens. The _Mask Token Inference Attack_Yue et al. (2021) replaces each token in sanitized text with a "[MASK]" token, which is then predicted to reconstruct the raw token. This attack overlooks the impact of probability distributions of raw tokens and the sampling probabilities in DP on the attack result, leading to an inaccurate evaluation of sanitization performance.

To address these issues, we introduce optimal reconstruction attacks and propose corresponding practical approaches for evaluating the effectiveness of DP-based text sanitization 

Theoretically Optimal Reconstruction Attacks

In this section, we first present the threat model and then develop _Context-free Optimal Reconstruction Attack_, deriving its ASR bound to enable quick evaluation of text sanitization. For a comprehensive evaluation, we propose _Contextual Optimal Reconstruction Attack_ with its ASR bound.

### Threat Model for Theoretical Attacks

Adversary's Goal.The goal of an adversary is to obtain information on sensitive tokens by reconstructing each original token from its sanitized one.

Adversary's Knowledge.We assume an adversary who knows the differentially private algorithm with its sampling probabilities \({\Pr}\left(Y|X\right)\), which is utilized in text sanitization. Additionally, we propose an assumption that the adversary possesses the following theoretical probabilities: \({\Pr}\left(X\right)\) (the probability of a token appearing in an original sentence) and \({\Pr}\left(C|X,Y\right)\) (the probability that a sanitized context corresponds to an original token and a sanitized token in the sanitization process).

### Warm-up: Context-free Optimal Reconstruction Attack

Building on the Bayesian estimation of DP [10, 2], we introduce the context-free optimal strategy for reconstructing the original tokens from sanitized tokens. We further derive its tight ASR bound, which achieves a quick evaluation of sanitization effectiveness.

**Theorem 1** (**Context-free Optimal Reconstruction Attack**).: _Given a sanitized token \(y\in Y\), the optimal strategy to reconstruct the original token of \(y\) is to select the token \(x^{\prime}\in X\) according to the following rule:_

\[x^{\prime}= \operatorname*{arg\,max}_{x^{\prime}\in X}{\Pr}\left(x^{\prime} |y\right) \tag{1}\] \[= \operatorname*{arg\,max}_{x^{\prime}\in X}\frac{{\Pr}\left(y|x^{ \prime}\right){\Pr}\left(x^{\prime}\right)}{{\Pr}\left(y\right)}. \tag{2}\]

The proof of Theorem 1 is in Appendix A. It is shown that an adversary should select the most likely original token \(x^{\prime}\) corresponding to the sanitized token \(y\) as the reconstructed token. According to Bayes' formula [35], Equation (1) can be decomposed into several components. \({\Pr}\left(y\right)\) is constant for all \(x^{\prime}\in X\) and does not affect the outcome of the attack. For \({\Pr}\left(y|x^{\prime}\right)\) and \({\Pr}\left(x^{\prime}\right)\), it is observed that: 1) A token \(x^{\prime}\) with a higher sampling probability \({\Pr}\left(y|x^{\prime}\right)\) is more likely to be the original token of \(y\); 2) A token \(x^{\prime}\) with a higher \({\Pr}\left(x^{\prime}\right)\) appears more frequently and is more likely to be the raw token of \(y\).

Based on the context-free optimal reconstruction attack, we derive its tight bound on the ASR as a quick evaluation for the effectiveness of DP-based text sanitization:

**Definition 1** (**Context-free ASR Bound**).: _Let \(x_{1}^{j}\), \(x_{2}^{j}\), ..., \(x_{n_{j}}^{j}\) and \(y_{1}^{j}\), \(y_{2}^{j}\), ..., \(y_{n_{j}}^{j}\) represent the \(j\)-th original sentence and its sanitized sentence by DP. Given any \(m\) sanitized sentences, the Context-free Attack Success Rate Bound (Context-free ASR Bound) is defined as follows:_

\[\text{Context-free ASR Bound}:=\frac{\sum_{j=1}^{m}\sum_{i=1}^{n_{j}} \delta_{x_{i}^{j},x_{i}^{j^{\prime}}}}{\sum_{i=1}^{n_{j}}n_{j}}, \tag{3}\] \[{x_{i}^{j}}^{\prime}= \operatorname*{arg\,max}_{x_{i}^{j^{\prime}}\in X}{\Pr}\left({ x_{i}^{j^{\prime}}|y_{i}^{j}}\right),\delta_{x_{i}^{j},x_{i}^{j^{\prime}}}=\begin{cases} 1\,,x_{i}^{j}={x_{i}^{j^{\prime}}}^{\prime}\\ 0\,,x_{i}^{j}\neq{x_{i}^{j^{\prime}}}^{\prime}\end{cases} \tag{4}\]

The _Context-free ASR Bound_ serves as a tight ASR bound for the _Context-free Optimal Reconstruction Attack_. This is because the attack of _Context-free ASR Bound_ strictly follows Theorem 1 to reconstruct the sanitized tokens. This bound primarily utilizes simple frequency statistics, making it a convenient tool for swiftly evaluating DP-based text sanitization methods. To represent \({\Pr}({x_{i}^{j^{\prime}}})\), which is informed to the adversary in the threat model, it computes the theoretical probability of the original token \({x_{i}^{j^{\prime}}}\) appearing in the original sentences. The term \({\Pr}({y_{i}^{j}|{x_{i}^{j^{\prime}}}^{\prime}})\) denotes the probability that DP samples sanitized token \(y_{i}^{j}\) with original token \({x_{i}^{j^{\prime}}}\). Although _Context-free ASR Bound_ swiftly assesses the sanitization effectiveness, it does not consider sanitized context, which may lead to a less comprehensive evaluation. Therefore, we develop _Contextual Optimal Reconstruction Attack_.

### Advanced: Contextual Optimal Reconstruction Attack

**Theorem 2** (**Contextual Optimal Reconstruction Attack**).: _Given a sanitized token \(y\) and its sanitized context \(\mathbf{c}\), the optimal strategy to reconstruct the original token of \(y\) is to select the token 

according to the following rule:_

\[x^{\prime}=\operatorname*{arg\,max}_{x^{\prime}\in X}\Pr\left(x ^{\prime}|y,\mathbf{c}\right) \tag{5}\] \[=\operatorname*{arg\,max}_{x^{\prime}\in X}\frac{\Pr\left(y|x^{ \prime}\right)\Pr\left(x^{\prime}\right)\Pr\left(\mathbf{c}|x^{\prime},y \right)}{\Pr\left(y\right)\Pr\left(\mathbf{c}|y\right)}. \tag{6}\]

The proof of Theorem 2 is provided in Appendix A. It demonstrates that an adversary should select the most probable original token \(x^{\prime}\), corresponding to the sanitized token \(y\) and its sanitized context \(\mathbf{c}\) (previously defined in Section 2), as the reconstructed token. Theorem 2 reduces the amount of uncertainty in reconstructing the original tokens compared to Theorem 1. This holds because \(H(X|\,Y,C)\leq H(X|\,Y)\), where \(H(X|\,Y)\) is the entropy of \(X\) conditioned on \(Y\), and \(H(X|\,Y,C)\) is the entropy of \(X\) conditioned on both \(Y\) and \(C\). Additionally, Theorem 2 includes extra calculations for \(\Pr(\mathbf{c}|x^{\prime},y)\) and \(\Pr\left(\mathbf{c}|y\right)\) relative to Theorem 1. \(\Pr\left(\mathbf{c}|y\right)\) remains constant for all \(x^{\prime}\in X\) and does not influence the outcome of the attack. \(\Pr\left(\mathbf{c}|x^{\prime},y\right)\) represents the probability that \(\mathbf{c}\) is the sanitized context of both \(x^{\prime}\) and \(y\).

**Challenge of Computation.** However, since \(\mathbf{c}\) is a sequence that consists of sanitized tokens, accurately calculating the theoretical value of \(\Pr\left(\mathbf{c}|x^{\prime},y\right)\) poses a challenge. Due to this intractability, we use its approximation to derive the ASR bound of attack strategy in Theorem 2.

**Definition 2** (Contextual _K-ASR Bound_).: _Let \(x_{1}^{j}\), \(x_{2}^{j}\), ..., \(x_{n_{j}}^{j}\) and \(y_{1}^{j}\), \(y_{2}^{j}\), ..., \(y_{n_{j}}^{j}\) represent the \(j\)-th original sentence and its sanitized sentence by DP. Let the sanitized context \(\mathbf{c}_{i}^{j}\) for the \(i\)-th sanitized token \(y_{i}^{j}\) (whose original token is \(x_{i}^{j}\)) be \(\mathbf{c}_{i}^{j}=y_{1}^{j}\), \(y_{2}^{j}\), ..., \(y_{i-1}^{j}\), \(y_{i+1}^{j}\), ..., \(y_{n_{j}}^{j}\). Given any \(m\) sanitized sentences, the Contextual K Attack Success Rate Bound (Contextual K-ASR Bound) is defined as follows:_

\[\text{Contextual K-ASR Bound}:=\frac{\sum_{j=1}^{m}\sum_{i=1}^{n_{ j}}\delta_{x_{i}^{j},x_{i}^{j^{\prime}}}}{\sum_{j=1}^{m}n_{j}}, \tag{7}\] \[x_{i}^{j^{\prime}}=\operatorname*{arg\,max}_{x_{i}^{j^{\prime} }\in X_{i}^{j^{\prime}}}\Pr(x_{i}^{j^{\prime}}|y_{i}^{j},\mathbf{c}_{i}^{j}),\] (8) \[X_{i}^{j^{\prime}}\!\!=\!\operatorname*{arg\,top}_{x_{i}^{j^{ \prime}}\in X}K\Pr\left(x_{i}^{j^{\prime}}|y_{i}^{j}\right),\;\delta_{x_{i}^{j ^{\prime}},x_{i}^{j^{\prime}}}\!\!=\begin{cases}1&,x_{i}^{j}=x_{i}^{j^{\prime}}\\ 0&,x_{i}^{j}\neq x_{i}^{j^{\prime}}.\end{cases} \tag{9}\]

The _Contextual K-ASR Bound_ provides an approximation of the ASR bound derived from Theorem 2. It calculates only the top \(K\) values of \(\Pr(x_{i}^{j^{\prime}}|y_{i}^{j},\mathbf{c}_{i}^{j})\), sorted by \(\Pr(x_{i}^{j^{\prime}}|y_{i}^{j})\). Because traversing the entire \(x_{i}^{j^{\prime}}\in X\) for the attack outcome requires a significant amount of time, as detailed in Appendix B. We find that it is not necessary to do so. The experiments in Section 5.4 indicate that the top \(K\) tokens sorted by \(\Pr(x_{i}^{j^{\prime}}|y_{i}^{j})\) include the majority of original tokens. Increasing \(K\) does not necessarily improve ASR. This method balances the trade-off between time cost and ASR.

For probability computations in the _Contextual K-ASR Bound_, it easily calculates \(\Pr(x_{i}^{j^{\prime}})\) based on the theoretical probability of the original token \(x_{i}^{j^{\prime}}\) appearing in the original sentences. However, given that \(\mathbf{c}_{i}^{j}\) is a sequence of sanitized tokens, it is challenging to calculate probability \(\Pr(\mathbf{c}_{i}^{j}|x_{i}^{j^{\prime}},y_{i}^{j})\). Although the chain rule can break it down into multiple components (Bain and Engelhardt, 1992), rebuilding the conditional probabilities for each \(\mathbf{c}_{i}^{j}\) is time-consuming.

To approximate the probability \(\Pr(\mathbf{c}_{i}^{j}|x_{i}^{j^{\prime}},y_{i}^{j})\), we convert it into a binary classification task. We represent it by the prediction probability of a specific label from a BERT model (Devlin et al., 2019) that has been fine-tuned on both the sanitized and original sentences. We detail the implementation of this binary classification in the following section 4.

Based on the preceding discussion, we have outlined a panoramic view of the theoretically optimal reconstruction attacks and their corresponding ASR bounds. It is important to note that these attacks are developed based on theoretical probabilities. However, practical reconstruction attacks often fall short of this level of precision, typically approximating probability distributions using a shadow dataset. To address this, we introduce the _Contextfree Bayesian Attack_ and the _Contextual Bayesian Attack_ as practical implementations of theoretically optimal reconstruction attacks.

## 4 Practical Implementations of Theorems

### Threat Model for Practical Attacks

Adversary's Goal.The goal of an adversary is to obtain sensitive information by reconstructing each original token from its sanitized token and corresponding sanitized context.

Adversary's Knowledge.We assume an adversary that accesses the information: 1) _Differentially Private Algorithms._ We assume that an adversary is informed about the differentially private algorithms with its sampling probabilities \(\Pr\left(Y|X\right)\) for text 

sanitization. Differentially private algorithms are typically treated as public access (Kairouz et al., 2014; Cormode et al., 2018; Wang et al., 2019c; Murakami and Kawamoto, 2019).

_2) A Shadow Dataset._ Additionally, we assume that an adversary has access to a shadow dataset, consistent with related privacy attacks (Carlini et al., 2022; Ye et al., 2022). The shadow dataset is related to but disjointed from the original sentences corresponding to the sanitized tokens under attack.

### Context-free Bayesian Attack (based on Theorem 1)

**Attack Strategy in _Context-free Bayesian Attack. Given a sanitized token \(y\in Y\), the strategy for Context-free Bayesian Attack to reconstruct the original token of \(y\) is to select the token \(x^{\prime}\in X\) according to the following rule:_**

\[x^{\prime}=\operatorname*{arg\,max}_{x^{\prime}\in X}\frac{\Pr\left(y|x^{ \prime}\right)\left(\Pr\left(x^{\prime}\right)+\alpha^{-1}\right)}{\Pr\left(y \right)}, \tag{10}\]

_where \(\Pr\left(x^{\prime}\right)\) is the statistical probability in the shadow dataset, and \(\alpha\) is the number of tokens in the shadow dataset._ Unfortunately, the practical implementation of Theorem 1 encounters difficulty when the approximate probabilities \(\Pr\left(x^{\prime}\right)\) of all potential original tokens corresponding to \(y\) are zero in the shadow dataset. Under these conditions, it is challenging to select an \(x^{\prime}\) that uniquely maximizes \(\Pr\left(x^{\prime}|y\right)\), as these probabilities \(\Pr\left(x^{\prime}|y\right)\) are all zero. To address this, a small constant, \(\alpha^{-1}\), is introduced in the attack. This modification enables _Context-free Bayesian Attack_ to select \(x^{\prime}=\operatorname*{arg\,max}\Pr\left(y|x^{\prime}\right)\) as the reconstructed token in such cases. Moreover, \(\alpha^{-1}\) has little effect on the outcome of the reconstruction attack when the probability \(\Pr\left(x^{\prime}\right)\) is not zero.

Figure 2 presents the ablation studies on parameter \(\alpha^{-1}\) with the _SANTEXT+_(Yue et al., 2021) defense, which demonstrated the effectiveness of \(\alpha^{-1}\) in practical reconstruction attacks. More experiments discussed in Appendix C further explain why we use \(\alpha^{-1}\) in _Context-free Bayesian Attack_.

### Contextual Bayesian Attack (based on Theorem 2)

**Attack Strategy in _Contextual Bayesian Attack. Given a sanitized token \(y\) and its sanitized context \(\mathbf{c}\), the strategy for Contextual Bayesian Attack to reconstruct the original token of \(y\) is to select the token \(x^{\prime}\in X\) according to the following rule:_**

\[x^{\prime}=\operatorname*{arg\,max}_{x^{\prime}\in X^{\prime}}\frac{\Pr\left( y|x^{\prime}\right)\cdot\left(\Pr\left(x^{\prime}\right)+\alpha^{-1}\right)}{ \Pr\left(y\right)}\times\]

\[\frac{\Pr\left(\mathbf{c}|x^{\prime},y\right)}{\Pr\left(\mathbf{c}|y\right)}, \tag{11}\]

\[X^{\prime}=\operatorname*{arg\,top}_{x^{\prime}\in X}K\,\frac{\Pr\left(y|x^{ \prime}\right)\cdot\left(\Pr\left(x^{\prime}\right)+\alpha^{-1}\right)}{\Pr \left(y\right)}, \tag{12}\]

_where \(\Pr\left(x^{\prime}\right)\) is the statistical probability in shadow dataset; \(\alpha\) is the count of tokens in shadow dataset; \(\Pr\left(\mathbf{c}|x^{\prime},y\right)\) is obtained from a fine-tuned model on shadow dataset._

**Transforming Reconstruction to Classification.** To represent the probability \(\Pr\left(\mathbf{c}|x^{\prime},y\right)\), we transform it with constant tokens \(x^{\prime}\) and \(y\): where the function \(f\) is defined as follows:

_Given any \(\mathbf{c}=y_{1},y_{2},\ldots,y_{i-1},y_{i+1},\ldots,y_{n}\), let \(f\left(\mathbf{c},x^{\prime}\right)=y_{1},y_{2},\ldots,y_{i-1},x^{\prime},y_{ i+1},\ldots,y_{n}\) and \(f\left(\mathbf{c},y\right)=y_{1},y_{2},\ldots,y_{i-1},y,y_{i+1},\ldots,y_{n}\)._

To calculate \(\Pr\left(f\left(\mathbf{c},x^{\prime}\right),f\left(\mathbf{c},y\right)|x^{ \prime},y\right)\), we first fine-tune a BERT model (Devlin et al., 2019) to serve as a detector, denoted by \(h\). This detector

Figure 2: ASR of _Context-free Bayesian Attack_ using various constants instead of \(\alpha^{-1}\) on SST-2 dataset.

 

\(h\) is a binary classifier that infers whether \(f\left({\mathbf{c},x^{\prime}}\right)\) and \(f\left({\mathbf{c},y}\right)\) correspond to \(x^{\prime}\) and \(y\) in sanitization process (i.e., \({\bf c}\) is the sanitized context for \(x^{\prime}\) and \(y\)). Specifically, the input to detector \(h\) consists of two token sequences, \(f\left({\mathbf{c},x^{\prime}}\right)\) and \(f\left({\mathbf{c},y}\right)\). The output of detector \(h\) is either label 1 or label 0, where a value of \(1\) indicates that \(f\left({\mathbf{c},x^{\prime}}\right)\) and \(f\left({\mathbf{c},y}\right)\) correspond to \(x^{\prime}\) and \(y\), while a value of \(0\) indicates that they do not. With these inputs and outputs, \(h\) is defined as \(h\left(f\left({\mathbf{c},y}\right),f\left({\mathbf{c},x^{\prime}}\right) \right)\rightarrow\{0,1\}\). The training samples of \(h\) are constructed as follows:

_Step 1: Confirm the Attack Target._ We first confirm the attack target \(\Gamma\) in order to construct corresponding training samples. The attack target \(\Gamma\) comprises a set of sanitized tokens that need to be reconstructed to their original forms.

_Step 2: Construct the Training Samples._ We sanitize sentences in the shadow dataset using the differentially private algorithm. For each token \(y\in\Gamma\) in the sanitized sentences, we create a training sample consisting of two token sequences and a label for \(y\). These two token sequences are the input of detector \(h\): The first sequence includes the sanitized sentence containing \(y\) and \({\bf c}\). The second sequence replaces the token \(y\in\Gamma\) in the first sequence with its reconstructed result \(x^{\prime}\) from the _Context-free Bayesian Attack_. The label is the output of detector \(h\): If \(x^{\prime}\) matches the original token of \(y\), indicating that \(f\left({\mathbf{c},x^{\prime}}\right)\) and \(f\left({\mathbf{c},y}\right)\) correspond to \(x^{\prime}\) and \(y\) in sanitization process, we label the sample \(1\). Otherwise, we label it \(0\). To maintain label balance, for each sample labeled as \(1\), we generate a corresponding sample labeled as \(0\) using the same sanitized sentence but with the second most likely reconstructed token from the _Context-free Bayesian Attack_. Similarly, for each sample labeled \(0\), we generate a sample labeled \(1\).

With the constructed training samples, we fine-tune the BERT model over three epochs with _Adam Optimizer_Kingma and Ba (2015), obtaining the detector \(h\). We represent \(\Pr\left({\mathbf{c}|x^{\prime},y}\right)=\Pr\left({f\left({\mathbf{c},x^{ \prime}}\right),f\left({\mathbf{c},y}\right)|x^{\prime},y}\right)\) using prediction probability of \(h\) for label 1.

Figure 3 shows the classification accuracy of \(h\), as detailed in Appendix D. Specifically, "Original" in Figure 3 refers to the performance of directly using BERT without any fine-tuning, while "Fine-tuned" refers to the performance after fine-tuning BERT on the detection task. Experimental results demonstrate the effectiveness of \(h\) in determining whether \({\bf c}\) is the sanitized context of \(x^{\prime}\) and \(y\).

## 5 Experiments

### Experimental Setup

**Datasets.** To validate the effectiveness of our attacks, we conduct experiments on four datasets, SST-2 Socher et al. (2013a), AGNEWS Zhang et al. (2015), QNLI Wang et al. (2019), and Yelp Zhang et al. (2015). These datasets cover a wide range of sensitive entities and personally identifiable information (PII) (detailed in Appendix E).

**Defense Methods.** We implement reconstruction attacks against two backbone methods of text sanitization using their default settings for sensitive tokens and parameters: _CUSTEXT+_Chen et al. (2023) and _SANTEXT+_Yue et al. (2021), both of which are widely used in DP research.

**Baseline Attacks.** Our work presents the first systematic study of reconstruction attacks against text sanitization. Since there is no other research specifically focusing on reconstruction attacks against text sanitization, we compare our methods with three adaptive attacks from prior defenses Meng et al. (2023); Zhou et al. (2023): _Embedding Inversion Attack_Qu et al. (2021), _Inversion BERT Attack_Kugler et al. (2021), and _Mask Token Inference Attack_Yue et al. (2021).

Figure 3: Classification accuracy of detector \(h\) in _Contextual Bayesian Attack_ with _CUSTEXT+_.

 



**Implementation.** We run experiments on a RTX A6000 GPU. We randomly select \(20,000\) sentences from the entire dataset to form a private dataset, whose sensitive tokens are perturbed to sanitized tokens. In _CUSTEXT+_, we calculate the ASR of the successful reconstruction for the sensitive tokens from \(3,000\) sanitized tokens. In _SANTEXT+_, this calculation of ASR is based on \(1,000\) sanitized tokens. For a shadow dataset, we randomly select \(1,000\) sentences from the entire dataset, which are disjoint from those to reconstruct.

More implementation details of our experiments can be found in Appendix F.

### Performance of Reconstruction Attacks

**Baseline Reconstruction Attacks.** In this paper, we identify three factors that facilitate reconstructing original tokens: the sampling probabilities of DP, the probability distribution of input tokens, and the sanitized context. However, the baseline attacks generally fall short in a design that considers all three factors. The limitations of them are demonstrated by the experimental results. For instance, at \(\epsilon=3.2\) on the QNLI, as shown in Figure 4e, both _Inversion BERT Attack_ and _Mask Token Inference Attack_ successfully reconstructed the sanitized tokens less than half of those in others. Moreover, at \(\epsilon=1.0\) on the AGNEWS, as shown in Figure 4c, the ASR of _Embedding Inversion Attack_ is less than \(0.09\), which is approximately one-third of that in _Inversion BERT Attack_.

**Attack Success Rate Bounds.** Experimental results in Figure 4 indicate that both _Context-free ASR Bound_ and _Contextual K-ASR Bound_ more accurately evaluate the effectiveness of DP-based text sanitization compared to baselines. For instance, with a privacy parameter \(\epsilon=1.0\) on QNLI, as shown in Figure 4b, the ASR of the _Context-free ASR Bound_ is \(28.7\%\) higher than that of the _Inversion BERT Attack_, while the ASR of the _Contextual K-ASR Bound_ is \(32.1\%\) higher than that of the _Inversion BERT Attack_. Furthermore, our experiments demonstrate that the _Contextual K-ASR Bound_ is consistently higher than the _Context-free ASR Bound_. Specifically, as shown in Figure 4d, the _Contextual K-ASR Bound_ exceeds the _Context-free ASR Bound_ by 30.1% with \(\epsilon=2.0\) on SST-2.

**Practical Attacks from Theorems.** According to Figure 4, the ASR of both the _Context-free Bayesian Attack_ and the _Contextual Bayesian Attack_ exceeded that of the baseline attacks in most cases. Notably, the _Contextual Bayesian Attack_

Figure 4: ASR of reconstructing sensitive tokens against _CUSTEXT+_ and _SANTEXT+_ with \(K=10\).



achieves a 46.4% improvement in ASR over the _Inversion BERT Attack_ at a privacy budget of \(\epsilon=4.0\) on the SST-2 according to Figure 3(a). With the same experimental setting, the _Context-free Bayesian Attack_ achieves a 39.2% improvement in ASR over the _Inversion BERT Attack_. Furthermore, the ASRs of our proposed practical attacks are approaching their corresponding theoretical ASR bounds. For example, with \(\epsilon=1.8\) on SST-2 as shown in Figure 3(d), the ASR of the _Context-free Bayesian Attack_ was only \(0.001\) less than that of the _Context-free ASR Bound_. With the same setting, the ASR of the _Contextual Bayesian Attack_ was \(0.004\) less than that of the _Contextual K-ASR Bound_. More discussion on the performance of reconstruction attacks with large \(\epsilon\) can be found in Appendix H.

Furthermore, our reconstruction attacks remain comparable to the baseline methods even with a misaligned shadow dataset. The detailed experiments can be found in Appendix I.

### Reconstructing PII Tokens

Text sanitization is widely adopted in identifying personal information Amazon (2024); Microsoft (2024b). To bring the evaluations closer to real-life scenarios, we conducted reconstruction attacks on the Yelp dataset to recover the PII of personal names Microsoft (2024a) from sanitized sentences. Experimental results, as illustrated in Figure 5, indicate the improvement of our attacks over the baselines. For instance, as shown in Figure 4(a), the _Context-free ASR Bound_ increased the ASR by \(41.5\%\) with a privacy parameter \(\epsilon=1.0\) over _Embedding Inversion Attack_; at the same experimental setting, _Contextual K-ASR Bound_ increased ASR by \(96.1\%\) over _Inversion BERT Attack_.

### Time Cost and Parameter Influence

**Time Costs.** We estimated the time required for reconstructing \(1,000\) sanitized tokens. We conducted this experiment \(10\) times to calculate standard deviations (1-sigma) using the closed form formula Altman and Bland (2005). As depicted in Figure 6, computations for the _Context-free ASR Bound_ and the _Context-free Bayesian Attack_ were completed in less than \(5\) minutes. Both of them enable quick evaluation of the effectiveness of text sanitization. For a comprehensive evaluation, _Contextual Bayesian Attack_ and _Contextual K-ASR Bound_ completed reconstructions in less than \(25\) minutes. We further compare the time costs of various reconstruction attacks in Appendix G.

**Parameter Influences.** Additionally, we test the influence of parameters on our attacks using SST-2 and _CUSTEXT+_ at \(\epsilon=4.0\). Table 1 shows the influence of \(K\) in both _Contextual Bayesian Attack_ (Contextual Attack) and _Contextual K-ASR Bound_ (Contextual Bound). When \(K\) is smaller than \(10\), the ASR of both grows with \(K\). When \(K\) exceeds \(10\), the growth of ASR tends to flatten. Table 2 shows the ASRs for different size ratios between the shadow dataset and the private dataset. With a smaller ratio, the ASR of _Contextual Bayesian Attack_ (Contextual Attack) and _Context-free Bayesian Attack_ (Context-free Attack) decreases.

Figure 5: ASR of reconstructing PII tokens.

\begin{table}
\begin{tabular}{l|c|c|c|c|c} \hline \hline
**Attack Strategy** & **S/P=1\%** & **S/P=3\%** & **S/P=5\%** & **S/P=8\%** \\ \hline Context-free Attack & 0.492 & 0.494 & 0.496 & 0.506 \\ Contextual Attack & 0.473 & 0.487 & 0.505 & 0.508 \\ \hline \hline \end{tabular}
\end{table}
Table 2: ASR of reconstructing sensitive tokens with different shadow(S) / private(P) dataset size ratios.

Figure 6: Time costs of reconstructing \(1,000\) tokens.

 

Conclusion

In this work, we implement both context-free and contextual optimal reconstruction attacks against text sanitization. We derive their bounds on ASR to guide the evaluation of sanitization effectiveness. We propose two practical attacks for real-world implementations. Evaluation results validate that our attacks surpass baseline methods.

This work represents an initial attempt to provide a more precise evaluation of sanitization effectiveness compared to current empirical attacks. Through this endeavor, we hope our work contributes to the research of privacy-preserving NLP.

## 7 Limitations

The results presented in our paper have two main limitations. First, we have not found a method to calculate the tight ASR bound of the _Contextual Optimal Reconstruction Attack_ due to the challenge of accurately computing the theoretical value of \(\Pr{(\mathsf{c}|x^{\prime},y)}\). Therefore, the calculation of this tight ASR bound remains an open question. Second, the _Contextual Bayesian Attack_ entails model fine-tuning and inference. There is still room for a more time-efficient reconstruction attack.

Furthermore, recent studies on text anonymization [14, 20], which leverage advanced LLMs to sanitize sensitive tokens in text, present an exciting avenue for future research. However, the complexity of LLM-based methods introduces unique challenges that require careful consideration. Consequently, the privacy guarantees of these methods are not explored in this paper. We believe that investigating these techniques could provide valuable insights and represent an important direction for future work.

## Acknowledgments

This research is supported by the National Natural Science Foundation of China under Grant 62472398, Grant U2336206, and Grant U2436601, as well as by the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative (No. DTCRGC-04). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore and Infocomm Media Development Authority.

## References

* [1]S. Chien et al. (2022) A survey on machine learning with deep convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 531-538. Cited by: SS2.
* [2]D. G. Altman and J. M. Bland (2005) Standard deviations and standard errors. Bmj331 (7521), pp. 903. External Links: Document Cited by: SS2.
* [3]M. S. Alvim, M. E. Andres, K. Chatziokolakis, P. Degano, and C. Palamidessi (2015) On the information leakage of differentially-private mechanisms. Journal of Computer Security23 (4), pp. 427-469. External Links: ISSN 0021-909X, Link, Document Cited by: SS2.
* [4]A. Azizadeh, R. H. T. de Vries, D. V. Kalinin, and L. Cortes (2016) The impact of adversarial attacks on classification of texts. In Advances in Neural Information Processing Systems, pp. 1138-1156. Cited by: SS2.
* [5]L. J. Bain and M. Engelhardt (1992) Introduction to probability and mathematical statistics. Vol. 4, Duxbury Press Belmont, CA. Cited by: SS2.
* [6]B. Balle, G. Cherubin, and J. Hayes (2022) Reconstructing training data with informed adversaries. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1138-1156. Cited by: SS2.
* [7]N. Carlini, S. Chien, M. Nasr, S. Song, A. Terzis, and F. Tramer (2022) Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pp. 1897-1914. Cited by: SS2.
* [8]Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al. (2023) A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology. Cited by: SS2.
* [9]S. Chen, F. Mo, Y. Wang, C. Chen, J. Nie, C. Wang, and J. Cui (2023) A customized text sanitization mechanism with differential privacy. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 5747-5758. Cited by: SS2.
* [10]G. Cherubin, K. Chatziokolakis, and C. Palamidessi (2019) F-bleau: fast black-box leakage estimation. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 835-852. Cited by: SS2.
* [11]K. Chowdhary (2023) Custom openai chatbots leak secrets. Note: [https://techreport.com/news/custom-openai-chatbots-leak-secrets-pose-privacy-threats/](https://techreport.com/news/custom-openai-chatbots-leak-secrets-pose-privacy-threats/) Cited by: SS2.
* [12]G. Cormode, S. Jha, T. Kulkarni, N. Li, D. Srivastava, and T. Wang (2018) Privacy at scale: local differential privacy in practice. In Proceedings of the 2018 International Conference on Management of Data, pp. 1655-1658. Cited by: SS2.
* [13]J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for 

Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Du et al. [2023] Minxin Du, Xiang Yue, Sherman SM Chow, Tianhao Wang, Chenyu Huang, and Huan Sun. 2023. Dpforward: Fine-tuning and inference on language models with differential privacy in forward pass. In _Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security_, pages 2665-2679.
* Dwork [2006] Cynthia Dwork. 2006. Differential privacy. In _International colloquium on automata, languages, and programming_, pages 1-12. Springer.
* Hong et al. [2024] Junyuan Hong, Jiachen T. Wang, Chenhui Zhang, Zhangheng LI, Bo Li, and Zhangyang Wang. 2024. DP-OPT: Make large language model your privacy-preserving prompt engineer. In _The Twelfth International Conference on Learning Representations_.
* Joyce [2003] James Joyce. 2003. Bayes' theorem.
* Kairouz et al. [2014] Peter Kairouz, Sewoong Oh, and Pramod Viswanath. 2014. Extremal mechanisms for local differential privacy. _Advances in neural information processing systems_, 27.
* Kim et al. [2024] Siwon Kim, Sangdoo Yun, Hwaran Lee, Martin Gubri, Sungroh Yoon, and Seong Joon Oh. 2024. Propile: Probing privacy leakage in large language models. _Advances in Neural Information Processing Systems_, 36.
* Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
* Kugler et al. [2021] Kai Kugler, Simon Munker, Johannes Hohmann, and Achim Rettinger. 2021. Invbert: Reconstructing text from contextualized word embeddings by inverting the bert pipeline. _arXiv preprint arXiv:2109.10104_.
* Meng et al. [2023] Tong Meng, Chen Kejiang, Zhang Jie, Qi Yuang, Zhang Weiming, Yu Nenghai, Zhang Tianwei, and Zhang Zhikun. 2023. Inferdpt: Privacy-preserving inference for black-box large language models. _arXiv preprint arXiv:2310.12214_.
* Microsoft [2024a] Microsoft. 2024a. Data protection and de-identification sdk. [https://microsoft.github.io/presidio/](https://microsoft.github.io/presidio/).
* Microsoft [2024b] Microsoft. 2024b. Modernize privacy program with a platform your. [https://www.microsoft.com/en-us/security/business/microsoft-priva](https://www.microsoft.com/en-us/security/business/microsoft-priva).
* Murakami and Kawamoto [2019] Takao Murakami and Yusuke Kawamoto. 2019. {Utility-optimized} local differential privacy mechanisms for distribution estimation. In _28th USENIX Security Symposium (USENIX Security 19)_, pages 1877-1894.
* OpenAI [2023] OpenAI. 2023. We're launching the gpt store to help you find useful and popular custom versions of chatgrt. [https://openai.com/blog/introducing-the-gpt-store](https://openai.com/blog/introducing-the-gpt-store).
* Papadopoulou et al. [2022] Anthi Papadopoulou, Yunhao Yu, Pierre Lison, and Ulija Ovrelid. 2022. Neural text sanitization with explicit measures of privacy risk. In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 217-229.
* Plant et al. [2021] Richard Plant, Dimitra Gkatzia, and Valerio Giuffrida. 2021. CAPE: context-aware private embeddings for private language learning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_, pages 7970-7978. Association for Computational Linguistics.
* Qu et al. [2021] Chen Qu, Weize Kong, Liu Yang, Mingyang Zhang, Michael Bendersky, and Marc Najork. 2021. Natural language understanding with privacy-preserving bert. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_, pages 1488-1497.
* Rajpurkar et al. [2016] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_, pages 2383-2392. The Association for Computational Linguistics.
* Saiteja et al. [2023] Utpala Saiteja, Hooker Sara, and Chen Pin-Yu. 2023. Locally differentially private document generation using zero shot prompting. In _Findings of the Association for Computational Linguistics: EMNLP 2023_, pages 8442-8457, Singapore. Association for Computational Linguistics.
* Shokri et al. [2017] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. 2017. Membership inference attacks against machine learning models. In _2017 IEEE symposium on security and privacy (SP)_, pages 3-18. IEEE.
* Socher et al. [2013a] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013a. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the 2013 conference on empirical methods in natural language processing_, pages 1631-1642.

 

Congzheng Song and Ananth Raghunathan. 2020. Information leakage in embedding models. In _Proceedings of the 2020 ACM SIGSAC conference on computer and communications security_, pages 377-390.
* Staab et al. (2024a) Robin Staab, Mark Vero, Mislav Balunovic, and Martin Vechev. 2024a. Large language models are advanced anonymizers. _arXiv preprint arXiv:2402.13846_.
* Staab et al. (2024b) Robin Staab, Mark Vero, Mislav Balunovic, and Martin T. Vechev. 2024b. Beyond memorization: Violating privacy via inference with large language models. In _The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024_. OpenReview.net.
* Tian et al. (2020) Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu. 2020. SKEP: sentiment knowledge enhanced pre-training for sentiment analysis. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_, pages 4067-4076. Association for Computational Linguistics.
* Wang et al. (2019a) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _International Conference on Learning Representations_.
* Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Wang et al. (2019c) Ning Wang, Xiaokui Xiao, Yin Yang, Jun Zhao, Siu Cheung Hui, Hyejin Shin, Jundum Shin, and Ge Yu. 2019c. Collecting and analyzing multidimensional data with local differential privacy. In _2019 IEEE 35th International Conference on Data Engineering (ICDE)_, pages 638-649. IEEE.
* Wired (2023) Wired. 2023. Openai's custom chatbots are leaking their secrets. [https://www.wired.com/story/openai-custom-chatbotts-gpts-prompt-injection-attacks/](https://www.wired.com/story/openai-custom-chatbotts-gpts-prompt-injection-attacks/).
* Ye et al. (2022) Jiayuan Ye, Aadyaa Maddi, Sasi Kumar Murakonda, Vincent Bindschaedler, and Reza Shokri. 2022. Enhanced membership inference attacks against machine learning models. In _Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security_, pages 3093-3106.
* Yue et al. (2021) Xiang Yue, Minxin Du, Tanhao Wang, Yuliang Li, Huan Sun, and Sherman S. M. Chow. 2021. Differential privacy for text analytics via natural text sanitization. In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_, pages 3853-3866, Online. Association for Computational Linguistics.
* Zhang et al. (2015a) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015a. Character-level convolutional networks for text classification. _Advances in neural information processing systems_, 28.
* Zhang et al. (2015b) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015b. Character-level convolutional networks for text classification. _Advances in neural information processing systems_, 28.
* Zhang and Sabuncu (2018) Zhilu Zhang and Mert Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. _Advances in neural information processing systems_, 31.
* Zhou et al. (2023) Xin Zhou, Yi Lu, Ruotian Ma, Tao Gui, Yuan Wang, Yong Ding, Yibo Zhang, Qi Zhang, and Xuan-Jing Huang. 2023. Textobfuscator: Making pre-trained language model a privacy protector via obfuscating word representations. In _Findings of the Association for Computational Linguistics: ACL 2023_, pages 5459-5473.

## Appendix A Proof of Theorems

**Theorem 1** (Context-free Optimal Reconstruction Attack).: _Given a sanitized token \(y\in Y\), the optimal strategy to reconstruct the original token of \(y\) is to select the token \(x^{\prime}\in X\) according to the following rule:_

\[x^{\prime}= \operatorname*{arg\,max}_{x^{\prime}\in X}\Pr\left(x^{\prime}|y\right) \tag{13}\] \[= \operatorname*{arg\,max}_{x^{\prime}\in X}\frac{\Pr\left(y|x^{ \prime}\right)\Pr\left(x^{\prime}\right)}{\Pr\left(y\right)}. \tag{14}\]

Proof of Theorem 1..: We aim to prove that the attack in Theorem 1 maximizes the expected value of the reconstructed token matching the original token for \(y\). Let \(X\) denote the set of probable original tokens for \(y\), defined as \(X=\{x_{1},x_{2},\ldots,x_{i},\ldots,x_{N}\}\), where \(x_{i}\in X\) is a probable original token of \(y\) with probability \(\Pr(x_{i}|y)\). Let the probability that an attacker infers \(x_{i}\in X\) as the reconstructed token of \(y\) be \(p_{i}\). It holds that\(\sum_{i=1}^{N}p_{i}=1\). The expected value of the reconstructed token matching the original token of \(y\) can be expressed as:

\[\Pr\left(x_{1}|y\right)p_{1}+\Pr\left(x_{2}|y\right)p_{2}+\ldots+\] \[\Pr\left(x_{N}|y\right)p_{N}. \tag{15}\]

With \(\sum_{j=1}^{N}p_{ij}=1\), it can be deduced that:

\[\Pr\left(x_{1}|y\right)p_{1}+\Pr\left(x_{2}|y\right)p_{2}+\ldots+\] \[\Pr\left(x_{N}|y\right)p_{N}\leq\max_{x^{\prime}\in X}\Pr\left(x^{ \prime}|y\right).\] (16) 

According to Equation (16), the expected value of correctly reconstructing the original token is smaller than \(\max_{x^{\prime}\in X}\Pr\left(x^{\prime}|y\right)\), which is the expected value under attack strategy in Theorem 1. 

**Theorem 2** (**Contextual Optimal Reconstruction Attack**).: _Given a sanitized token \(y\) and its sanitized context \(\mathbf{c}\), the optimal strategy to reconstruct the original token of \(y\) is to select the token \(x^{\prime}\in X\) according to the following rule:_

\[x^{\prime} =\operatorname*{arg\,max}_{x^{\prime}\in X}\;\Pr\left(x^{\prime} |y,\mathbf{c}\right) \tag{17}\] \[=\operatorname*{arg\,max}_{x^{\prime}\in X}\frac{\Pr\left(y|x^{ \prime}\right)\Pr\left(x^{\prime}\right)}{\Pr\left(y\right)}\frac{\Pr\left( \mathbf{c}|x^{\prime},y\right)}{\Pr\left(\mathbf{c}|y\right)}. \tag{18}\]

Proof:: We aim to prove that the attack in Theorem 2 maximizes the expected value of the reconstructed token matching the original token for \(y\) and \(\mathbf{c}\). Let \(X\) denote the set of probable original tokens for \(y\) and \(\mathbf{c}\), defined as \(X=\{x_{1},x_{2},\ldots,x_{i},\ldots,x_{N}\}\), where \(x_{i}\in X\) is a probable original token of \(y\) and \(\mathbf{c}\) with probability \(\Pr(x_{i}|y,\mathbf{c})\). Let the probability that an attacker inferring \(x_{i}\in X\) as the reconstructed token of \(y\) and \(\mathbf{c}\) be \(p_{i}\) and it holds that\(\sum_{i=1}^{N}p_{i}=1\). The expected value of the reconstructed token matching the original token of the sanitized token \(y\) and sanitized context \(\mathbf{c}\) can be expressed as:

\[\Pr(x_{1}|y,\mathbf{c})p_{1}+ \Pr(x_{2}|y,\mathbf{c})p_{2}+ \ldots+\] \[\Pr(x_{N}|y,\mathbf{c})p_{N}. \tag{19}\]

Utilizing \(\sum_{i=1}^{N}p_{i}=1\), it can be deduced that:

\[\sum_{i=1}^{N}\Pr(x_{i}\mid y,\mathbf{c})p_{i}\leq\max_{x^{\prime}\in X}\Pr \left(x^{\prime}|y,\mathbf{c}\right). \tag{20}\]

According to Equation (20), the expected value of correctly inferring the original token is smaller than \(\max_{x^{\prime}\in X}\Pr\left(x^{\prime}|y,\mathbf{c}\right)\), which is the expected value under attack strategy in Theorem 2. 

## Appendix B Discussion of Contextual _K_-ASR Bound

We assess the time required to calculate the Contextual _K_-ASR Bound for \(1,000\) sanitized tokens. we conduct experiments using _CUSTEXT+_Chen et al. (2023) defense on SST-2 Socher et al. (2013a) dataset with various values of \(K\). We set the size of \(X\) in _CUSTEXT+_ to its default \(20\). The experimental results are shown in Table 3. It is observed that as \(K\) increases, the time cost also increases.

On the other hand, the size of the input token set \(X\) in _SANTEXT+_Yue et al. (2021) far exceeds 20. Specifically, it uses an input token set \(X\) consisting of more than \(15,000\) tokens in SST-2. Given this, traversing the entire \(X\) for the Contextual _K_-ASR Bound in _SANTEXT+_ is not practical to implement.

## Appendix C Discussion of Context-free Bayesian Attack

The attack strategy in Theorem 1 reconstructs a token considering two factors: a larger \(\Pr(y|x^{\prime})\) and a larger \(\Pr(x^{\prime})\). However, when the probabilities \(\Pr(x^{\prime})\) of all probable original tokens corresponding to \(y\) are zero in the shadow dataset, the attack strategy in Theorem 1 loses its effectiveness of reconstructing a token with larger \(\Pr(y|x^{\prime})\). In such instances, _Context-free Bayesian Attack_, incorporating \(\alpha^{-1}\), retains its effectiveness. This method selects a reconstructed token \(x^{\prime}\) according to:

\[x^{\prime} =\operatorname*{arg\,max}_{x^{\prime}\in X}\frac{\Pr(y|x^{\prime })\cdot(0+\alpha^{-1})}{\Pr(y)} \tag{21}\] \[=\operatorname*{arg\,max}_{x^{\prime}\in X}\frac{\Pr(y|x^{\prime })\cdot\alpha^{-1}}{\Pr(y)}. \tag{22}\]

Furthermore, \(\alpha^{-1}\) has minimal impact on the outcome of the reconstruction attack when these probabilities \(\Pr(x^{\prime})\) are not zero.

To evaluate the practical impact of the parameter \(\alpha^{-1}\), we conducted an ablation study across three datasets: SST-2 Socher et al. (2013b), QNLI Wang et al. (2019), and AGNEWS Zhang et al. (2015b). We assessed the Attack Success Rate (ASR) of the _Context-free Bayesian Attack_ both with and without \(\alpha^{-1}\) against _SANTEXT+_Yue et al. (2021), creating \(3,000\) test samples randomly selected from each dataset.

Figure 2 presents the results of the ablation studies on the parameter \(\alpha^{-1}\), where the experimental results demonstrate the effectiveness of \(\alpha^{-1}\).

To further explore the influence of different constants instead of \(\alpha^{-1}\) on _Context-free Bayesian Attack_, we compare the ASR of \(\alpha^{-1}\) with those of other constants against _SANTAEXT+_.

Table 4 illustrates the experimental results across three datasets under various privacy parameters \(\epsilon\). These results highlight the effectiveness of the constant introduced in the _Context-free Bayesian Attack_. When this constant is smaller than \(\alpha^{-1}\), the ASR of the _Context-free Bayesian Attack_ decreases slightly in the experiments. Conversely, when this



constant exceeds \(\alpha^{-1}\), the ASR does not show significant growth.

## Appendix D Discussion of Contextual Bayesian Attack

To evaluate the effectiveness of the detector \(h\), we calculate the classification accuracy of both the detector \(h\) and the original BERT model Devlin et al. (2019) across three datasets: SST-2 Socher et al. (2013b), QNLI Wang et al. (2019), and AGNEWS Zhang et al. (2015). We construct \(3,000\) test samples that are disjointed from the training data of detector \(h\) and randomly selected. For the construction of training samples, we generate 100 sanitized sentences for each original sentence in the shadow dataset. The experiment is conducted 10 times to calculate standard deviations (1-sigma) using the closed form formula Altman and Bland (2005) with numpy1.

Footnote 1: [https://numpy.org/](https://numpy.org/)

Figure 3 shows the experimental results. It is observed that the classification accuracy of \(h\) is higher than that of the original BERT model. Additionally, the classification accuracy of \(h\) increases with the privacy parameter \(\epsilon\). These results demonstrate the effectiveness of the detector \(h\) in inferring whether \(\mathbf{c}\) is the sanitized context of \(x^{\prime}\) and \(y\).

## Appendix E Experimental dataset

We conduct the experiments of reconstruction attacks on the following four datasets:

* **SST-2 Socher et al. (2013b) :** The (binary) Stanford Sentiment Treebank (SST-2) dataset consists of movie reviews with sentiment labels. This dataset is used to categorize the sentiment of sentences Tian et al. (2020) into the positive or negative label. We use the version in the GLUE Wang et al. (2019).
* **QNLI Wang et al. (2019) :** The Question-answering Natural Language Inference (QNLI) dataset is derived from the Stanford Question Answering Dataset (SQuAD) Rajpurkar et al. (2016). This dataset consists of question-paragraph pairs, where the task is to determine whether the paragraph contains the answer to the question. We also utilize the use available in the GLUE benchmark Wang et al. (2019).
* **AGNEWS Zhang et al. (2015b) :** The AG News dataset is a widely used benchmark for text classification tasks. It consists of news articles and their corresponding labels. This dataset is utilized to train a model that classifies news into four categories: World, Sports, Business, and Sci/Tech. We use the version of this dataset available on this site2. Footnote 2: AG Corpus of News Articles
* **Yelp Zhang et al. (2015b) :** The Yelp dataset consists of reviews from the Yelp website, where the task is to predict the sentiment expressed in the reviews. The dataset is used for sentiment analysis tasks. We utilized the version available from the Yelp Challenge3.

Footnote 3: Yelp Open Dataset

## Appendix F Implementation details

### Context-free ASR Bound and Contextual _K_-ASR Bound

We run experiments on a cluster with NVIDIA RTX A6000 GPUs and Intel Xeon Gold 6130 2.10 GHz CPUs. We conduct the implementation of all

\begin{table}
\begin{tabular}{r||c c c|c c c c|c c c c} \hline \multirow{2}{*}{**Constant**} & \multicolumn{4}{c|}{**SST-2**} & \multicolumn{4}{c|}{**QNLI**} & \multicolumn{4}{c}{**AGNEWS**} \\ \cline{2-17}  & \(\epsilon=2.0\) & \(\epsilon=4.0\) & \(\epsilon=2.8\) & \(\epsilon=3.2\) & \(\epsilon=2.0\) & \(\epsilon=4.0\) & \(\epsilon=2.8\) & \(\epsilon=3.2\) & \(\epsilon=2.0\) & \(\epsilon=4.0\) & \(\epsilon=2.8\) & \(\epsilon=3.2\) \\ \hline \(\alpha^{-1}\) & 0.180 & 0.410 & 0.650 & 0.843 & 0.155 & 0.434 & 0.711 & 0.865 & 0.223 & 0.450 & 0.712 & 0.894 \\ w/o \(\alpha^{-1}\) & 0.130 & 0.331 & 0.600 & 0.682 & 0.133 & 0.387 & 0.617 & 0.780 & 0.204 & 0.416 & 0.653 & 0.820 \\ \(0.1\times\alpha^{-1}\) & 0.173 & 0.404 & 0.648 & 0.840 & 0.152 & 0.429 & 0.704 & 0.862 & 0.218 & 0.448 & 0.709 & 0.892 \\ \(5\times\alpha^{-1}\) & 0.180 & 0.410 & 0.650 & 0.843 & 0.155 & 0.434 & 0.711 & 0.865 & 0.222 & 0.449 & 0.711 & 0.894 \\ \(10\times\alpha^{-1}\) & 0.180 & 0.410 & 0.650 & 0.843 & 0.155 & 0.434 & 0.711 & 0.865 & 0.221 & 0.449 & 0.711 & 0.894 \\ \hline \end{tabular}
\end{table}
Table 4: ASR of _Context-free Bayesian Attack_ using various constants instead of \(\alpha^{-1}\) against _SANTEXT+_ defense.

\begin{table}
\begin{tabular}{l|c c c c c} \hline \hline \(K\) & \(K=1\) & \(K=5\) & \(K=10\) & \(K=15\) & \(K=20\) \\ \hline
**Minute** & \(11.631\) & \(16.659\) & \(18.850\) & \(24.566\) & \(29.216\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: Time cost of reconstructing \(1,000\) tokens with various \(K\) in the _Contextual K-ASR Bound_.

 

mechanisms using Python. We calculate \({\rm Pr}(x^{\prime})\) by estimating the statistical probability of the original sentences corresponding to the sanitized tokens.

In the _Contextual K-ASR Bound_, we generate 30 sanitized sentences for each original sentence to construct training samples. For each token \(y\in\Gamma\) in the sanitized sentences, we create a training sample. The input for each sample consists of two token sequences: The first sequence is the sanitized sentence consisting of \(y\) and **c**. The second sequence replaces the token \(y\in\Gamma\) in the first sequence with its reconstructed result \(x^{\prime}\) from _Context-free Bayesian Attack_. If \(x^{\prime}\) matches the original token of \(y\), which means that **c** is the sanitized context of the sanitized token \(y\) and the reconstructed token \({x_{i}}^{\prime}\), we label the sample \(1\). Otherwise, we label it \(0\). To maintain label balance, for each sample labeled as \(1\), we generate a sample labeled as \(0\) using the same sanitized sentence with the second most likely token from _Context-free Bayesian Attack_. We also generate a sample labeled as \(1\) with the original token for each sample labeled as \(0\). For model fine-tuning, we use the pooled output from the BERT model Devlin et al. (2019) and transform it into class scores through a linear layer.

We use the _Adam Optimizer_Kingma and Ba (2015) with a learning rate of 0.00005. We reduce the learning rate by 0.1 every 1000 steps. We measure performance with _CrossEntropyLoss_Zhang and Sabuncu (2018) and fine-tune the model over 3 epochs on constructed training samples. The fine-tuned model represents probability \({\rm Pr}({\bf c}|x^{\prime},y)\) that \({\bf c}\) is the sanitized context for \(x^{\prime}\) and \(y\), based on its output of the label \(1\).

### Context-free Bayesian Attack and Contextual Bayesian Attack

We run experiments on a cluster with NVIDIA RTX A6000 GPUs and Intel Xeon Gold 6130 2.10 GHz CPUs. We conduct the implementation of all mechanisms using Python. We calculate the statistical probability in the shadow dataset to represent the approximated probability \({\rm Pr}(x^{\prime})\).

In the _Contextual Bayesian Attack_, we generate 100 sanitized sentences for each original sentence to construct training samples. For each token \(y\in\Gamma\) in the sanitized sentences, we create a training sample. The input for each sample consists of two token sequences: The first sequence is the sanitized sentence consisting of \(y\) and \({\bf c}\). The second sequence replaces the token \(y\in\Gamma\) in the first sequence with its reconstructed result \(x^{\prime}\) from _Context-free Bayesian Attack_. If \(x^{\prime}\) matches the original token of \(y\), which means that \({\bf c}\) is the sanitized context of the sanitized token \(y\) and the reconstructed token \(x^{\prime}\), we label the sample \(1\). Otherwise, we label it \(0\). To maintain label balance, for each sample labeled as \(1\), we generate a sample labeled as \(0\) using the same sanitized sentence with the second most likely token from _Context-free Bayesian Attack_. We also generate a sample labeled as \(1\) with the original token for each sample labeled as \(0\). For model fine-tuning, we use the pooled output from the BERT model Devlin et al. (2019) and transform it into class scores through a linear layer.

We use the _Adam Optimizer_Kingma and Ba (2015) with a learning rate of 0.00005. We reduce the learning rate by 0.1 every 1000 steps. We measure performance with _CrossEntropyLoss_Zhang and Sabuncu (2018) and fine-tune the model over 3 epochs on constructed training samples. The fine-tuned model represents probability \({\rm Pr}({\bf c}|x^{\prime},y)\) that \({\bf c}\) is the sanitized context for \(x^{\prime}\) and \(y\), based on its output of label \(1\).

## Appendix G Comparison of time cost

To further compare the computation costs, we estimated the time required for various reconstruction

Figure 7: Comparison of the time costs in various reconstruction attacks.

 

attacks to reconstruct \(1,000\) sanitized tokens on the SST-2 dataset using the _CUSTEXT+_ as a defense. Similarly, we conducted this experiment \(10\) times to calculate the standard deviations (1-sigma) using the closed form formula Altman and Bland (2005) with the numpy4.

Footnote 4: [https://numpy.org/](https://numpy.org/)

As illustrated in Figure 7, the time cost for both the _Embedding Inversion Attack_ and the _Context-free Bayesian Attack_ is approximately comparable, both being less than 5 minutes. These two reconstruction attacks do not require model fine-tuning or inference. In contrast, the _Contextual Bayesian Attack_, _Mask Token Inference Attack_, and _Inversion BERT Attack_ involve either model fine-tuning or model inference. Among these three, the _Mask Token Inference Attack_ has the smallest time cost, roughly 15 minutes; the _Inversion BERT Attack_ takes the longest, about 25 minutes. The time cost of the _Contextual Bayesian Attack_ in the experiments is greater than that of the _Mask Token Inference Attack_. This is because _Mask Token Inference Attack_ only involves an inference phase without model training. However, our contextual attack requires training a model and inferring on this model. Therefore, _Contextual Bayesian Attack_ is slower than the _Mask Token Inference Attack_. Moreover, it is observed that the standard deviations of time costs for the _Embedding Inversion Attack_ and the _Context-free Bayesian Attack_ are smaller than those for the other reconstruction attacks.

## Appendix H Reconstruction attacks with the large epsilon

As illustrated in Figure 4, the performance of our proposed attacks is comparable to that of the _Embedding Inversion Attack_ for large values of \(\epsilon\). The rationale behind this phenomenon is as follows:

Let \(x\) represent a raw token and \(y\) its corresponding sanitized token. The performance of our attacks partially depends on \({\rm Pr}(y|x)\). Specifically, as \(\epsilon\) increases, \({\rm Pr}(y|x)\) also increases in both _SANTEXT+_ and _CUSTEXT+_, causing our attacks to tend to select the token \(x\) with the highest \({\rm Pr}(y|x)\) as the reconstructed token. For a given sanitized token \(y\), the _Embedding Inversion Attack_ selects the token \(x^{\prime}\) with the smallest embedding distance to \(y\) as the reconstructed token. In the context of _SANTAXT+_ and _CUSTEXT+_, this is equivalent to selecting the token \(x^{\prime}\) with the highest \({\rm Pr}(y|x^{\prime})\).

From the above analysis, it can be concluded that when \(\epsilon\) is large, our proposed attack and the _Embedding Inversion Attack_ tend to select the same token as the reconstructed token in most cases.

## Appendix I Reconstruction attacks with the misaligned shadow dataset

In this paper, we present a privacy evaluation study on DP-based text sanitization. Our threat model assumes that practical attacks access a shadow dataset from the same distribution as the target to reconstruct. This assumption simulates the worst-case scenario for privacy preservation and is consistent with the settings described in previous work Balle et al. (2022); Shokri et al. (2017).

To reflect real-world scenarios where shadow datasets may not be perfectly aligned with sanitized data, we conduct additional attacks on _CUSTEXT+_ with \(\epsilon=3.0\) (as recommended by its authors) using a misaligned shadow dataset. Specifically, we use SST-2 as the shadow dataset and QNLI as the target dataset to reconstruct. Figure 8 presents the experimental results with the misaligned shadow dataset. It is observed that our reconstruction attacks remain comparable to the baseline methods.

Figure 8: ASR of reconstruction attacks with the misaligned shadow dataset.