DP-MLM: Differentially Private Text Rewriting Using Masked Language Models (2024)

Abstract: The task of text privatization using Differential Privacy has recently taken the form of text rewriting, in which an input text is obfuscated via the use of generative (large) language models. While these methods have shown promising results in the ability to preserve privacy, these methods rely on autoregressive models which lack a mechanism to contextualize the private rewriting process. In response to this, we propose DP-MLM, a new method for differentially private text rewriting based on leveraging masked language models (MLMs) to rewrite text in a semantically similar and obfuscated manner. We accomplish this with a simple contextualization technique, whereby we rewrite a text one token at a time. We find that utilizing encoder-only MLMs provides better utility preservation at lower ε levels, as compared to previous methods relying on larger models with a decoder. In addition, MLMs allow for greater customization of the rewriting mechanism, as opposed to generative approaches. We make the code for DP-MLM public and reusable, found at https://github.com/sjmeis/DPMLM.

text: Findings of the Association for Computational Linguistics ACL 2024, pages 9314–9328 August 11-16, 2024 ©2024 Association for Computational Linguistics
The task of text privatization using Differential Privacy has recently taken the form of text rewriting, in which an input text is obfuscated via the use of generative (large) language models. While these methods have shown promising results in the ability to preserve privacy, these methods rely on autoregressive models which lack a mechanism to contextualize the private rewriting process. In response to this, we propose DP-MLM, a new method for differentially private text rewriting based on leveraging masked language models (MLMs) to rewrite text in a semantically similar and obfuscated manner. We accomplish this with a simple contextualization technique, whereby we rewrite a text one token at a time. We find that utilizing encoder-only MLMs provides better utility preservation at lower ε levels, as compared to previous methods relying on larger models with a decoder. In addition, MLMs allow for greater customization of the rewriting mechanism, as opposed to generative approaches. We make the code for DP-MLM public and reusable, found at https://github.com/sjmeis/DPMLM.

heading: 1 Introduction
text: The study of Differential Privacy (DP) in NLP investigates the integration of the privacy guarantee offered by DP to the textual domain. This is especially timely as concerns of privacy vulnerabilities in NLP models, particularly LLMs, continue to rise in tandem with recent advances in AI.
Looking to DP for safeguarding privacy in text processing is a promising avenue of research, bringing about novel techniques in recent years ranging from DP optimization techniques (Abadi et al., 2016), DP language models (Igamberdiev and Habernal, 2023), or DP text privatization methods (Feyisetan et al., 2020; Mattern et al., 2022; Utpala et al., 2023). DP in NLP does not come without its
challenges, however (Klymenko et al., 2022; Mattern et al., 2022), among them the balance between privacy and utility, as well as the general reasoning of DP in unstructured domains such as text.
State-of-the-art DP text rewriting methods focus on paraphrasing as a proxy for text privatization, either via directly fine-tuning a paraphrase model (Mattern et al., 2022), or by prompting a LLM to generate a rephrased version of an input text (Utpala et al., 2023). By incorporating a DP mechanism into the generation of output tokens, the generated text aims to privatize the input while still maintaining utility and semantic similarity.
The above works do not consider encoder-only models, such as BERT-based models, and furthermore, the advantages of doing so have not been studied in juxtaposition to models with decoders. Therefore, we are motivated to extend DP text rewriting beyond the usage of autoregressive generative models, guided by the following question:
How can one leverage Masked Language Models (MLMs) to achieve contextually relevant, yet
privacy-preserving text rewriting?
To answer this question, we propose DP-MLM, a differentially private text rewriting mechanism leveraging masked token prediction in BERT-based models. We design a rewriting mechanism that incorporates the contextual information from an input sentence to produce a rewritten output that is both utility- and privacy-preserving. This customization is enabled by building a rewriting mechanism on top of an MLM, as opposed to relying on a decoder to generate privatized outputs. To test utility and privacy, we conduct empirical experiments that validate the ability of DP-MLM to rewrite texts in a way that preserves meaning yet still empirically defends against adversarial privacy attacks.
Our work makes the following contributions to the study of differentially private text rewriting:
1. To the best of our knowledge, we are the first
9314
work to propose the use of BERT-based models for DP text rewriting.
2. We design a rewriting mechanism, DP-MLM, which leverages a contextualization technique not before used for text privatization.
3. With DP-MLM, we surpass previous SOTA methods for private text rewriting on many benchmarks in both utility and privacy, particularly at lower per-token ε budgets.
These contributions support our hypothesis that MLMs are effective tools for utility-preserving text privatization, and its observed strengths are analyzed at the conclusion of this work.

heading: 2 Related Work
text: Natural language contains sensitive information (Brown et al., 2022; McMahan et al., 2017). DP enables the training of Machine Learning (ML) models on sensitive texts with a guarantee that the model will not leak more sensitive data than a predefined value (Pan et al., 2020; McMahan et al., 2017; Carlini et al., 2021). In the field of NLP, there are two primary notions of integrating DP for downstream applications. The first approach is to collect user texts at a central location and train a model on the texts using a differentially-private optimization technique like DP-SGD (Abadi et al., 2016) (Ponomareva et al., 2022; McMahan et al., 2017; Kerrigan et al., 2020). This approach is known as global or central DP. In contrast, the second approach is to apply a DP mechanism on texts locally, i.e., on the user side, before sharing the privatized texts with a central aggregator. This notion is called Local Differential Privacy (LDP). LDP is a stricter notion of DP as compared to the formerly defined central DP (Feyisetan et al., 2020).
The earliest application of LDP to the task of text privatization considers a sentence as independent sequences of words (Fernandes et al., 2019; Feyisetan et al., 2020). As a result, the privatized sentence is generated by perturbing a sentence wordby-word, normally by introducing calibrated noise to word embeddings (Yue et al., 2021; Chen et al., 2022; Carvalho et al., 2023; Meisenbacher et al., 2024a). These methods do not consider the grammatical and contextual information while generating a private sentence (Mattern et al., 2022). Moreover, they utilize the generalized notion of metric DP, which increases the utility of the generated text but makes comparative evaluations challenging.
Other LDP methods operate at higher levels in the syntactic hierarchy, such as the sentence-level, and directly generate a privatized text, either by adding noise to the latent representations of the text (Igamberdiev and Habernal, 2023) or using additional public information (Meehan et al., 2022). By construction, these methods take into account the grammar and context of the input text while privatizing it. These methods provide stricter privacy guarantees as compared to the aforementioned word-level mechanisms but come at a significant utility cost (Igamberdiev and Habernal, 2023).
Recent works for LDP in NLP take a different approach and leverage Language Models to generate privatized text (Mattern et al., 2022; Utpala et al., 2023). They model the task of text privatization as a text paraphrasing task, or more generally, text rewriting. In leveraging such models to rewrite text, generation is performed by tokens being sampled in a fashion that satisfies local DP. We build upon the foundations of these methods, and aim to improve upon the utility- and privacy-preserving capabilities of such an approach.

heading: 3 Foundations

heading: 3.1 Masked Language Modeling
text: A Masked Language Model (MLM), such as BERT (Devlin et al., 2019), includes as one of its two pre-training objectives the task of masked token prediction. Here, the model is trained to predict a randomly masked token in a sentence by conditioning its probability not only on the tokens that are to its left in the sentence but also to its right. Thus, the masked token is filled by the most suitable word deemed by the MLM, which is achieved by considering the complete context of the sentence.
If the lth token of a sentence s containing n tokens in sequence w1 · w2 · · ·wn is masked, the probability of the masked token being a word v from the vocabulary V , as modeled by an MLM, is
Pr[wl = v] = Pr[v|w1 ·w2 · · ·wl−1 ·wl+1 · · ·wn]
Although the masked token of a MLM is primarily used in its pre-training task, one can also use the token to replace a given target word in a text. This fact is leveraged by the related tasks of MLMbased Lexical Substitution (Zhou et al., 2019) or Lexical Simplification (Qiang et al., 2020).

heading: 3.2 Differential Privacy
text: Differential Privacy (DP) (Dwork, 2006) is a mathematically grounded notion of privacy that provides information-theoretic privacy guarantees while performing computation over a dataset. Given ε ≥ 0 and finite sets W and V , a randomized mechanism M : W → V is an ε-DP mechanism if ∀c, c′ ∈ C and ∀v ∈ V , the following condition holds:
Pr[M(c) = v] Pr[M(c′) = v] ≤ eε
Here c and c′ are called neighboring or adjacent databases. Depending on the notion of adjacency, the unit which is protected by LDP is defined. In our case, any two context sentences c and c′ are adjacent. This is expounded upon in the next section.

heading: 3.2.1 Temperature Sampling as an Exponential Mechanism
text: Suppose there is a dataset D ∈ X n and our aim is to derive a value for this dataset from a set of fixed value choices V ∈ Y . Exponential Mechanism can be used here to select, in a private manner, the best choice for a dataset from a set of choices V , with its goodness being determined by a scoring
function. The scoring (utility) function u : X n × Y → R maps database and choice pairs, (D,w) with D ∈ X n and w ∈ V , to possibility scores. The l2-sensitivity of such scoring function is given as
∆u = max w∈V max D,D′∈Xn
|u(D,w)− u(D′, w)|
If the choice for D is selected according to probability proportional to exp( εu(D,w)2∆u ), then the selection algorithm satisfies DP and this DP mechanism is termed as an Exponential Mechanism (McSherry and Talwar, 2007).
Mapping it to our use case, suppose we have a context s = w1 · w2 · w3 · · ·wn or s = w1 · w2 · · ·wl−1 · wl+1 · · ·wn, and we want to choose the best token for the masked word wl from the set of vocabulary. This selection can be done privately through an Exponential Mechanism with scoring function u : V∗ × V → R that takes as input the whole context s and a word w from the vocabulary set V and outputs a score for w conditioned on the context s. This scoring function is precisely a (Masked) Language Model in our case: it takes the context as input and outputs logits for every word in the vocabulary. To bound the sensitivity of the scoring function, which is necessary for DP, the logit values can be clipped to a predefined range.
If the logits generated by the MLM are bounded, then the Exponential Mechanism is realized by default if we use temperature sampling to predict the masked token (Mattern et al., 2022). To compare it, if the logit value produced for a word w ∈ V with a context s is u and the sampling temperature is set to T , the predicted token being w has the probability proportional to exp(u/T ). Comparing it to the exponential mechanism, we can derive that ε = 2∆uT . A detailed DP proof is provided in Section 4.3.

heading: 4 Method
text: In this section, we outline the design of DP-MLM, particularly its underlying DP mechanism, as well as the text rewriting mechanism.

heading: 4.1 DP Masked Token Prediction
text: Given an input sentence s with n tokens, our goal is to privatize each token in the sentence, one token at a time. To privatize a single token wl, we input the entire sentence s = w1 · · ·wl−1 · wl · wl+1 · · ·wn to an MLM, with wl replaced by the model’s mask token, usually <mask>. Next, we capture the logit
values for the masked token index l, clip them (described in Section 5.1.1), and apply Temperature Sampling mechanism, that is equivalent to the Exponential Mechanism as described in Section 3.2.1. All clipped logit values are first divided by the temperature T , which is calculated according to T = 2∆uε . The resulting values are fed through a softmax function and a token is sampled according to these probabilities. The sampled token wp then serves as the differentially private token replacement for wl. A proof that this mechanism is DP is found in Section 4.3.

heading: 4.2 Rewriting Mechanism
text: To rewrite an entire text with the underlying mechanism described above, we design a rewriting mechanism DP-MLM that is outlined in Algorithms 1 and 2. Note that for the entirety of this work, we use the ROBERTA-BASE MLM as our base model.
Algorithm 1 DP-MLM Token Replacement Require: MLM M ,
context tokens, private p_tokens, position idx epsilon ε, clipping values C = (Cmin, Cmax)
Ensure: Output private_token at position idx
T ← 2 · (|Cmax − Cmin|)/ε p_tokens[idx]← <mask> masked_sent← concat(tokens, [SEP], p_tokens) logits←M(masked_sent) logits← clip_and_temp(logits, C, T ) prob← softmax(logits) private_token← sample(prob) return private_token
Algorithm 2 Text Rewriting using DP-MLM Require: input sentence s = w1 · w2 · · ·wn,
per-token epsilon ε, clipping values C = (Cmin, Cmax)
Ensure: rewritten (privatized) sentence using DP-MLM
tokens← tokenize(s) private← tokens for i ∈ 1...n do
p← DPMLM(tokens, private, i, ε, C) ▷ Alg. 1 private[i]← p
end for return detokenize(private)
As described in Algorithm 1, contextualization of the DP-MLM mechanism is achieved via the concatenation of the original input sentence, which is given along with the masked sentence as input to the MLM. This simple trick is motivated by a simi-
lar approach followed by (Qiang et al., 2020) for Lexical Simplification. A more intuitive illustrative example of the DP-MLM rewriting mechanism is found in Figure 1. Note that in our implementation, we do not replace English stopwords, but we leave this as a parameter in our open-source code.
With Algorithm 1, we replace a single token from an input text in a DP manner. Thus, the output of one DP-MLM usage is a privatized token that is contextually relevant to the text. By using DP-MLM for each token in the input sentence (Algorithm 2), we design a text rewriting mechanism that leverages the compositionality of DP to output a privately rewritten text with a DP guarantee of ε× len(tokens). This is formalized for the token and text level in the following.

heading: 4.3 Privacy Guarantees
text: Our mechanism M introduced in Algorithm 1 satisfies local differential privacy (LDP). For any two adjacent context sentences, M yields “similar” tokens to fill a given masked token with LDP guarantees. Hence, given a predicted masked token, an adversary who only sees the predicted token cannot differentiate with high certainty if the predicted token was due to a context sentence c or c′. This results in plausible deniability about the source of the predicted masked token.
Suppose a sentence s consists of n tokens, i.e., s = w1 ·w2 · · ·wn, and the (masked) token that we want to predict lies at lth position in the sentence. An MLM models the likelihood of masked token wl being vi ∈ V as follows:
Pr[wl = vi] = Pr[vi|w1 · · ·wl−1 · wl+1 · · ·wn] = Pr[vi|Cl]
As stated earlier, we use clipped logits u and temperature sampling to sample the most likely token for the masked token. Hence, for our proposed mechanism M : V+ → V that takes the context sentence Cl = w1 · w2 · · ·wl−1 · wl+1 · · ·wn as input and returns a privately selected predicted masked word, its output probability distribution is given by the following equation:
Pr[M(Cl) = vi] = exp(u(Cl,vi)T )∑|V| j=1 exp( u(Cl,vj) T )
(1)
Theorem 1. The proposed mechanism M defined in the equation 1 satisfies ε-LDP.
Proof. Let s, s′ ∈ Vn. Suppose the lth token of s, s′ is masked and we use our mechanism M to predict a token. The context to be set as input for the Masked Language Model becomes:
Cl := w1 · w2 · · ·wl−1 · wl+1 · · ·wn
C ′l := w ′ 1 · w′2 · · ·w′l−1 · w′l+1 · · ·w′n
The ratio of the probability distribution of application of M on Cl and C ′l can be given as:
Pr[M(Cl) = vi] Pr[M(C′ l ) = vi] = exp(
u(Cl,vi)
T )
∑|V| j=1 exp( u(Cl,vj) T )
∑|V| j=1 exp( u(C′l),vj T )
exp( u(C′ l ),vi)
T )
= exp(
u(Cl,vi)
T )
exp( u(C′ l ,vi))
T )
∑|V| j=1 exp( u(C′l,vj) T ) ∑|V| j=1 exp( u(Cl,vj) T )
Solving the first fraction, we get
exp(u(Cl,vi) T ) exp( u(C′ l ,vi)) T ) = exp
( u(Cl, vi)− u(C′l , vi)
T
)
≤ exp ( ∆u
T
)
Similarly, solving the second fraction, we get
Pr[M(Cl) = vi] Pr[M(C′l) = vi] ≤ exp
( ∆u
T
) exp ( ∆u
T
)
= exp ( 2 ∆u
T
)
= exp (ε)
Hence, ε can be calculated from the sensitivity of ∆u and the sampling temperature T as ε = 2∆uT
Extending guarantees to a sentence The privacy budget required for generating a single masked token using the mechanism M is equal to ε. The privatized sentence of the input sentence is generated by sequentially generating privatized token for each token present in the input sentence. Thus, for rewriting a sentence (or more generally, a text) of n tokens, we are required to call the mechanism n times. Hence, by sequential composition, the total privacy budget spent for rewriting the entire text would be nε-DP.

heading: 5 Experimental Setup
text: In order to evaluate the performance of our proposed method, we design two overarching experiments: (1) utility experiments, and (2) empirical privacy experiments. These are outlined below.

heading: 5.1 Utility Experiments
text: Our utility experiments consist of two phases: (1) utility benchmarking of DP-MLM, and (2) comparative utility testing with two other state-of-the-art DP rewriting mechanisms.

heading: 5.1.1 Utility Benchmarking
text: To test the utility-preserving capability of DPMLM, we measure the utility of rewritten data across a range of ε values. For this, we utilize the GLUE benchmark (Wang et al., 2018), which contains 9 separate tasks spanning classification, textual similarity, and textual entailment. For a given ϵ value, a perturbed dataset (train and validation split) is measured against the non-privatized baseline, in order to measure how well the privatization keeps utility intact.
Model For both the non-privatized baseline and all privatized (rewritten) datasets, we fine-tune a DEBERTA-V3-BASE model (He et al., 2021) for one epoch. The trained model is then evaluated on the validation set (non-privatized or privatized, respectively). All models in this work are trained on a single RTX A6000 GPU, using the Adam optimizer (Kingma and Ba, 2017) and all default hyperparameters of the Hugging Face Transformers Trainer API. To account for variations in training, all utility results represent the average of 3 runs.
Clipping Values To ensure that the sensitivity of the logit values of our underlying MLM (ROBERTABASE) is bounded, we pre-define the clip value based upon an empirical estimation of the logit value range. Concretely, we measure all logits values from inputting 1000 random text examples from the SST2 dataset of GLUE, calculate the mean µ and standard deviation σ, and define the clipping values as (Cmin, Cmax) = (µ, µ + 4σ). The choice of such a range provides the benefit of a bound sensitivity, while still preserving higher logit values and “clamping” lower values.
Epsilon In order to conduct comprehensive tests on varying privacy budgets, we choose the following set of ε values: ε ∈ {10, 25, 50, 100, 250}.

heading: 5.1.2 Comparative Utility
text: The next utility experiments test DP-MLM against the two state-of-the-art text rewriting methods:
1. DP-Paraphrase (Mattern et al., 2022): DP text rewriting utilizing DP temperature sampling in a fine-tuned GPT-2 paraphrasing
model. More details on the replication of DPPARAPHRASE is provided in Appendix B.
2. DP-Prompt (Utpala et al., 2023): DP text rewriting using prompting of large language models, i.e., to paraphrase the input text. For the purposes of this work, we utilize a pretrained FLAN-T5-BASE.
These methods are tested on a subset of the GLUE tasks, namely {COLA, MRPC, RTE, SST2}, where these datasets are perturbed accordingly as for DP-MLM. The same ε values and clipping strategy as described in Section 5.1.1 are used for both methods to ensure direct comparability. In addition, for both generative methods listed above, we restrict the maximum number of generated tokens to the length of the input tokens, so as to ensure fairness in comparative evaluation metrics.
Scoring For comparative utility testing, we report raw scores (accuracy or correlation), as well as BLEU (Papineni et al., 2002; Igamberdiev and Habernal, 2023) and cosine similarity (CS) scores between the original and privatized dataset (Meisenbacher et al., 2024b). These additional metrics present a clearer picture of the utilitypreservation of the underlying DP mechanism, compared against the amount by which the dataset has been perturbed. For semantic similarity (CS), we utilize the ALL-MINILM-L6-V2 model (Reimers and Gurevych, 2019).

heading: 5.2 Empirical Privacy Experiments
text: To measure the privacy-preserving capabilities of DP-MLM in comparison to the state-of-the-art, we conduct empirical privacy experiments on two datasets. This process is described in the following.
Datasets For conducting our privacy experiments, we utilize two datasets:
1. Trustpilot Reviews (Hovy et al., 2015): a large corpus of user reviews from Trustpilot, containing both a review score (1-5) and the gender (M/F) of the reviewer. We only consider reviews rated with 5 (positive) or 1-2 (negative). From this, we take a random 10% sample, representing ∼36k reviews.
2. Yelp Reviews: we utilize the dataset as used by Utpala et al. (2023), which contains Yelp reviews from 10 authors, labeled as positive or negative. We take a random sample of 250 texts from each author, for a total of 2500.
Tasks For both datasets, we conduct a twosided experiment. As in the utility experiment, we privatize each dataset with the budgets ε ∈ {10, 25, 50, 100, 250}, and compare the utility loss
against the non-private baseline. A DEBERTA-V3BASE model is once again employed for fine-tuning, trained for a total of three epochs.
Following the approach laid out by previous works (Mattern et al., 2022; Utpala et al., 2023), we test empirical privacy in two adversarial settings. The first is called the static attacker, where the adversarial model can only be evaluated on the privatized outputs after being trained on the original non-privatized input texts. In contrast, the adaptive attacker is able to train the adversarial model on the DP outputs, thus more closely matching the distribution of the target privatized texts.
For the static setting, we train an adversarial DEBERTA-V3-BASE model on the non-privatized dataset to predict the protected attribute of each dataset, i.e., gender for Trustpilot and author ID for Yelp. These models are trained for five epochs. Then, we evaluate the adversarial model on each privatized dataset and measure the change in performance. In this way, we can empirically measure the privacy protection provided by rewriting a dataset.
In the adaptive setting, the only difference is that the DEBERTA-V3-BASE model is trained on the privatized training datasets, and subsequently evaluated on the validation splits of these datasets.
Scoring Following Mattern et al. (2022), we not only report the raw results of the above-mentioned experiments, but also the relative gain achieved by text privatization via rewriting. Such a score is useful in quantifying the advantage of privatizing text, or rather the gain in privacy offset by the inevitable loss in privacy. Let Po, Uo represent the baseline privacy and utility scores, respectively, that is the scores when training both the sentiment and ad-
versarial classifiers on the non-privatized datasets. Let Pr, Ur be the scores observed on the privatized (rewritten) datasets. The relative gain is thus defined as RG = (Ur/Uo)− (Pr/Po). Note that we report relative gains for both adversarial settings.
In addition to the raw utility scores (F1 score), we also present the percentage points (PP+) achieved above majority class guessing. In the case of both Trustpilot and Yelp, the datasets are highly biased towards positive reviews, so reporting PP+ better demonstrates the degree to which a fine-tuned model learns to distinguish sentiment.

heading: 6 Results
text: Utility Table 1 presents the complete benchmarking results of DP-MLM on the GLUE tasks. Table 2 displays the comparative utility results, which tested DP-MLM and our two selected methods on a subset of the GLUE tasks. The results from Table 2 are summarized in Figure 2, which illustrates the average utility loss for each evaluated rewriting mechanism, given an ε value.
Privacy Table 3 displays the results of our empirical privacy tests, for both Trustpilot and Yelp.
Efficiency We measure the efficiency, or speed, at which the evaluated mechanisms can rewrite text. We capture this by recording the amount of time taken to perturb the selected GLUE datasets, including how many tokens are perturbed (1,048,231 in total). The results are summarized below:
• DP-MLM – Elapsed time: 1316 minutes – Tokens/min: 797
• DP-Paraphrase – Elapsed time: 1308 minutes – Tokens/min: 802
• DP-Prompt – Elapsed time: 1961 minutes – Tokens/min: 535

heading: 7 Discussion
text: Utility In analyzing the results of both utility evaluations, one can see that DP-MLM clearly demonstrates the ability to produce utilitypreserving DP rewritten text. As expected, this naturally comes with a performance drop as compared to the non-privatized baseline; however, interesting findings can be extracted when observing the comparitive utility tests. Across four selected GLUE tasks, which represent all three task
types of the benchmark, DP-MLM consistently outperforms the other two state-of-the-art methods, DP-PARAPHRASE (Mattern et al., 2022) and DPPROMPT (Utpala et al., 2023). This is supported by the fact that DP-MLM achieves the highest utility score in 14 out of the 20 comparative settings.
A closer look at Table 2 shows that in all four cases where DP-PROMPT outperforms DP-MLM, this can be attributed to a very high BLEU and CS score between the original and privatized datasets. While this may be useful for utility preservation, one may question whether CS scores near to 1 provide much privacy preservation at all. On the other hand, even at higher values of ε such as 100 or 250, DP-MLM still provides higher levels of privatization (more rewritten), while still offering competitive utility scores in all cases.
The above is especially supported by the fact that in 8 out of the 14 cases where DP-MLM achieves the highest utility score, it does so without having the highest CS to the original dataset. In addition, DP-MLM shows particular strength at low ε budgets, such as with ε = 10, where it never possesses the highest CS score, yet still remains very competitive. These observations emphasize the ability of DP-MLM to produce privatized, yet still contextually and semantically relevant rewritten texts. This is further supported by Figure 2, which places DPMLM, on average, lower than DP-PARAPHRASE and DP-PROMPT in terms of utility loss.
Privacy From the empirical privacy results, one can observe similar trends as with the utility experiments. Empirically, all three evaluated methods are very effective in reducing the adversarial advantage, i.e., gender classification or author identification, and this is especially true at lower privacy budgets. DP-PARAPHRASE is particularly effective, but this comes at the cost of comparatively poor results in preserving utility, as can be seen in Table 3.
Comparing DP-MLM and DP-PROMPT in terms of empirical privacy, one can observe from the Trustpilot test that both methods are successful in preserving utility to a degree where a model can still learn sentiment classification reasonably, as shown by PP+, or the F1 percentage points above majority-class guessing. This case for DP-MLM is especially made salient in the Yelp test, where DPMLM is the only method capable of producing positive PP+ scores. Furthermore, despite achieving similar or better utility scores in the empirical privacy tests as compared to DP-PROMPT, DP-MLM
consistently scores better in reducing adversarial F1 at higher ε values for the static setting, while the opposite is true for the adaptive setting.
Relative Gain also provides insights, where DPMLM offers added value in most cases and better trade-offs at higher ε budgets than DP-PROMPT, particularly in the static setting. A weakness of DPMLM is highlighted by the adaptive results, pointing to the inevitable trade-off between higher utility text and its resulting privacy protections. Nevertheless, DP-MLM still achieves positive gains in all adpative scenarios. All methods are similarly competitive at lower ε, yet this must be interpreted according to whether privacy or utility is favored.
Efficiency DP-MLM performs nearly identically in terms of efficiency as opposed to DPPARAPHRASE, and both methods greatly outperform DP-PROMPT. This can directly be attributed to the encoder-only ROBERTA-BASE and the decoder-only GPT-2, as opposed to the utilized FLAN-T5-BASE, which is nearly double in model size. Especially when considering the abovementioned strengths of DP-MLM, the added competitiveness of speed introduces a practical advantage of our method. Such results also raise interesting points for future work regarding the effect of model size and architecture on privatization, particularly the interplay between these and ε.
Addressing Limitations The discussion of the merits of DP-MLM must also be met with its remaining limitations. As our rewriting mechanism leveraging DP-MLM relies on token-level DP replacements, the primary limitation comes with the initial inability to rewrite sentences with differing lengths from the original texts. To address this main limitation, we propose an improved version of the rewriting mechanism which enables variable length outputs. This is presented in Algorithm 3.
In essence, Algorithm 3 takes as inputs an addition probability A and deletion probability D, and for each token in the input text s, we delete this token with probability D and add an additional token with probability A. New tokens are added by simply inserting a mask token into the context sentence and running DP-MLM as usual.
From a privacy guarantee standpoint, the downside of such an augmentation comes with an altered guarantee. In the worst case for a given ε, text rewriting with Algorithm 3 offers a privacy guarantee of 2nε-DP, i.e., in the case that a token is added for every token in the input sentence. In the
Algorithm 3 Text Rewriting +- using DP-MLM Require: input sentence s = w1 · w2 · · ·wn,
per-token epsilon ε, clipping values C = (Cmin, Cmax), token addition probability A, token deletion probability D
Ensure: rewritten (privatized) sentence using DP-MLM
tokens← tokenize(s) private← tokens added← 0 deleted← 0 for i ∈ 1...n do
prob_del, prob_add← rand() ▷ random numbers ∈ [0..1] if prob_del ≥ D then
p← DPMLM(tokens, private, i+added−deleted, ε, C) private[i + added− deleted]← p
else deleted← deleted + 1 end if if prob_add ≤ A then
added← added + 1 p← DPMLM(tokens, private, i+added−deleted, ε, C) private.insert(i + added− deleted, p)
end if end for return detokenize(private)
average case, though, we achieve (A−D)nε-DP. In Appendix C, we show the results of repeating a subset of our empirical privacy experiments using this augmented rewriting method.

heading: 8 Conclusion
text: We present DP-MLM, a differentially private text rewriting mechanism leveraging masked token prediction of MLMs. As opposed to previous methods relying on private generation using language models with decoders, we utilize BERT-based encoderonly models to rewrite text in a private, yet contextual manner. This is accomplished by simple concatenation in our rewriting mechanism, which guides the private sampling of replacement tokens.
In a series of utility and privacy experiments, we empirically demonstrate the improvements achieved by DP-MLM over previous SOTA methods. In particular, DP-MLM outperforms these methods on many utility benchmarks, and achieves competitive empirical privacy scores. An analysis of the results reveals that DP-MLM finds a necessary balance between utility- and privacypreservation, more so than the previous SOTA.
As paths for future work, we propose continued research on DP text rewriting with encoder-only models, including the investigation of how privatization can be improved with less utility loss and the effect of using different base models. In addition, we see that more work to define and unify evaluation strategies for DP text rewriting should be undertaken, so as to allow for well-defined methodologies for validating future proposed mechanisms.

heading: Acknowledgements
text: The authors would like to thank the anonymous reviewers for their feedback and Alexandra Klymenko for her valuable contributions to this work.
Limitations
The primary limitation of our work comes with the choice of underlying language model for each DP mechanism. We choose one model as the representative model for DP-MLM, DP-PARAPHRASE, and DP-PROMPT, namely ROBERTA-BASE, GPT2, and FLAN-T5-BASE, respectively. The implications of choosing and evaluating other BERT-based models were considered outside of the scope of this work. This should be tested in follow-up studies.
Another limitation that pertains to general evaluation of privacy-preserving mechanisms, that is, the evaluation of privacy protection in NLP. The method of empirical privacy employed in this work follows from the predominant method in the literature to measure privacy, but as it is a proxy for privacy, we can only claim that our proposed DPMLM protects privacy empirically and by proxy.
Ethics Statement
An ethical consideration involves our empirical privacy experiments, which leverage existing datasets not originally intended for adversarial gender or author identification. In performing these empirical experiments, the actions of a potential adversary were simulated, i.e., to leverage publicly accessible information for the creation of an adversarial model. As these datasets are already publicly available, no harm was inflicted in the privacy experiments as part of this work. Moreover, the Yelp dataset is made up of pseudonyms (User IDs) rather than PII, thus further reducing the potential for harm.

heading: C Additional Results
text: In Table 4, we present the results of testing Algorithm 3, that is rewriting with DP-MLM with token addition and deletion. In particular, we rerun a subset of the Trustpilot empirical privacy experiments, for ε ∈ {25, 50, 100}. For each ε, we test token addition probabilities of 0.1 and 0.25. A deletion probability of 0.05 is used throughout.
Interestingly, Table 4 shows higher utility results but lower empirical privacy, especially in the adaptive setting. This, therefore, leads to lower relative gains than in the base rewriting case without addition or deletion. These results spark the discussion on whether the augmentation presented in Algorithm 3 is necessary, and furthermore, whether privatized text length variability should be prioritized in text privatization evaluation.

heading: D Privatization Examples
text: In Tables 5, 6, and 7, privatized (rewritten) examples are provided from the SST2, COLA, and MRPC datasets, respectively. Examples are shown for all three compared mechanisms, across the five selected ε values.
Table 8 shows selected examples from using Algorithm 3 as proposed in Section 7.

heading: Future Work
text: As paths for future work, we propose continued research on DP text rewriting with encoder-only models, including the investigation of how privatization can be improved with less utility loss and the effect of using different base models. In addition, we see that more work to define and unify evaluation strategies for DP text rewriting should be undertaken, so as to allow for well-defined methodologies for validating future proposed mechanisms