SlowGuess/NAACL_2022_OCR huggingface dataset:
Crowdsourcing is highly laborious and cannot always be applied in real-world scenarios. Therefore, we aim at imitating the rewriting behavior of humans through a large-scale transformer-based (Vaswani et al., 2017) language model: We fine-tune GPT-2 (Radford et al., 2019) to generate paraphrases following the training procedure introduced by Witteveen and Andrews (2019). The Stanford Natural Language Inference (SNLI) Corpus (Bowman et al., 2015) provides training data consisting of pairs of sentences with five crowdsourced labels, each indicating whether the two sentences are semantically entailed or not. We construct a paraphrase dataset by only keeping sentence pairs with all five labels indicating entailment.
4.2 Balancing privacy and utility
As pointed out by Brennan et al. (2012), the black box nature of authorship obfuscation via roundtrip and consequently also monolingual translation affects controllability of our system negatively. Therefore, in the following we demonstrate how varying the temperature in the word sampling stage of GPT-2 can be used to inject noise into our model, hereby balancing the privacy-utility trade off.
In an autoregressive generative model, an output text  $\pmb{x} = (x_{1},\dots ,x_{n})$  is generated by sampling the next word  $x_{i}$  from conditional probabilities  $\pmb{p}_i = p(x_i\mid x_1,\dots,x_{i - 1},z)$  modeled by the decoder network, where  $z$  is context information (e.g., representing an encoding of the input sentence to be obfuscated) to initialize the decoder. The vector  $\pmb{p}_i = (p_{i,j})_{j = 1}^{|\mathcal{V}|}$  represents the probabilities of producing the  $j$ -th word  $v_{j}$  of the predefined vocabulary  $\mathcal{V}$  at the  $i$ -th position in the sequence. The probabilities are typically obtained through the softmax function from a logit vector  $\pmb{u}_i\in \mathbb{R}^{|\mathcal{V}|}$  in the last layer of the decoder, which can be controlled by a temperature parameter  $T > 0$  as follows:
$$
p _ {i j} := \mathrm {s o f t m a x} (\boldsymbol {u} _ {i}) = \frac {\exp \left(\frac {u _ {i , j}}{T}\right)}{\sum_ {k} \exp \left(\frac {u _ {i , k}}{T}\right)} \quad (1)
$$
A higher temperature  $T$  results in a smoother distribution that brings the resulting probabilities of all words closer together and thus impacts the variability and probabilities of the resulting sentences. In our experiments in Section 5, we vary the temperature when sampling text to evaluate this effect.
Sampling from softmax as differential privacy mechanism Note that sampling from the softmax distribution with temperature  $T$  can be interpreted as a DP mechanism, namely as an instance of the Exponential mechanism by McSherry and Talwar (2007). It applies to both numerical and categorical data and requires a "measure of suitability" for each possible pair of input and output values:
Definition 4 (Quality function) A map  $q: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$  is called quality function from  $\mathcal{X}$  to  $\mathcal{Y}$  where we interpret the value  $q(x, y)$  as measure of suitability of an output  $y \in \mathcal{Y}$  for a given input  $x \in \mathcal{X}$ . The sensitivity  $\Delta_q$  of the quality function  $q$  is its largest possible difference given two adjacent inputs, over all possible output values:
$$
\Delta_ {q} := \max  _ {y \in \mathcal {Y}} \max  _ {x _ {1} \sim x _ {2}} \left(q (x _ {1}, y) - q (x _ {2}, y)\right)
$$
Given an admissible rating function  $q$  with finite sensitivity  $\Delta_q$ , the Exponential mechanism is defined as follows:
Definition 5 (Exponential mechanism) Let  $\epsilon > 0$  be a privacy parameter, and let  $q: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$  be a rating function. The Exponential mechanism is a random mechanism  $\mathcal{E}: \mathcal{X} \to \mathcal{Y}$  that is defined by the probability distribution function
$$
\operatorname * {P r} [ \mathcal {E} (x) = y ] = \frac {\exp \Big (\frac {\epsilon}{2 \Delta_ {q}} q (x , y) \Big)}{\int_ {y ^ {\prime}} \exp \Big (\frac {\epsilon}{2 \Delta_ {q}} q (x , y ^ {\prime}) \Big) \mathrm {d} y ^ {\prime}}.
$$
A discrete version of the Exponential mechanism for countable  $\mathcal{V}$  can be obtained by replacing the integral with a sum; it is thus defined by the probability mass function
$$
\Pr [ \mathcal {E} (x) = y ] = \frac {\exp \left(\frac {\epsilon}{2 \Delta_ {q}} q (x , y)\right)}{\sum_ {y ^ {\prime}} \exp \left(\frac {\epsilon}{2 \Delta_ {q}} q (x , y ^ {\prime})\right)}. \tag {2}
$$
The Exponential mechanism  $\mathcal{E}$  fulfills  $\epsilon$ -DP as shown by McSherry and Talwar (2007, Theorem 6).
By comparing Eqs. (1) and (2), we immediately recognize that sampling from the softmax probabilities  $\pmb{p}_i = (p_{i,1},\dots ,p_{i,|\mathcal{V}|})$  amounts to running an instance of the Exponential mechanism with  $\epsilon = 2\Delta_q / T$  and the quality function determined by the logits vector  $\pmb{u}_i\in \mathbb{R}^{|\mathcal{V}|}$  as
$$
q _ {i} \left(\left(x _ {1}, \dots , x _ {i - 1}, z\right), v _ {j}\right) = u _ {i, j}, \quad 1 \leq j \leq | \mathcal {V} |,
$$
at each iteration  $i$  when sampling the next word  $x_{i}$ . Therefore, our generative paraphrasing model naturally forms a locally differentially private mechanism that also enjoys formal privacy guarantees:
The total privacy budget amounts to  $\epsilon \cdot n$  where  $n$  is the length of the generated paraphrase. Finally, note that we obtain a finite sensitivity  $\Delta_q \leq 1$  by constraining the decoder layer so that the logits in its output fulfill  $0 \leq u_{i,j} \leq 1$ .
While this approach is still subject to the implications of the local model, and its total privacy budget  $\epsilon \cdot n$  may still grow linearly in the length of the produced output, it avoids the language and fixed output length constraints of previous word level privacy mechanisms stated in Section 3.
We argue that despite formal guarantees, the privacy preservation capabilities of mechanisms that are deployed in real world applications should also be tested from a practical standpoint. Previous works measure anonymization capabilities using a variety of evaluation metrics: Feyisetan et al. (2020) use the privacy auditor proposed by Song and Shmatikov (2019), whereas Xu et al. (2021) measure the ability of an adversary to reconstruct the original sentence, and Xu et al. (2020b,a) count the amount of changed words.
Unfortunately, these methods are rarely tested under the scenario of a strong attacker aiming to identify the authors of the obfuscated texts. While Feyisetan et al. (2019) measure the identification performance of an authorship attribution model, their adversary (Koppel et al., 2011) only relies on counting character 4-grams and does not adequately reflect the capabilities of a strong attacker who can train more powerful classifiers. Besides, attacks are almost always evaluated only in a static (non-adaptive) setting, meaning that the attack model is only trained on the original data and cannot adapt to the perturbed data. Since any serious method should avoid "security (or privacy) by obscurity", we must assume that the obfuscation mechanism is known to the attacker who can easily create perturbed data themselves.
In the following evaluation, we consider two exemplary methods following the word level framework, namely perturbing Euclidean GloVe embeddings (Pennington et al., 2014) through Laplace noise as proposed by Feyisetan et al. (2020), the perturbation of hierarchical Poincaré embeddings (Nickel and Kiela, 2017) through hyperbolic noise as proposed by (Feyisetan et al., 2019), as well as our paraphrasing approach proposed in Section 4. To address the discussed issues in previous evalua
tion methodologies, we employ recent state-of-the-art methods to compare the privacy-utility trade-offs and analyze the performance of the approaches not only in a static, but also in an adaptive setting.
5.1 Evaluation metrics
We argue that an anonymization mechanism deployed in real world applications should provide protection against advanced deonymization attacks, preserve the core information of the original data (e.g., sentiment for product reviews), be semantically similar to the original sentences and of high quality in terms of language.
We measure the first two properties using both static (i.e., trained on source data) and adaptive (i.e., trained on data perturbed by the respective mechanism) BERT-based (Devlin et al., 2019) author and sentiment classifiers by fine-tuning the pretrained language model's top three layers and using a two-layer classifier for the author and sentiment labels. BERT has proven to be successful for both sentiment classification (Sun et al., 2019) and authorship attribution (Fabien et al., 2020) and thus represents a suitable model for both tasks. We report all classification results in terms of Matthews Correlation Coefficient (MCC) (Matthews, 1975; Gorodkin, 2004). An MCC score of  $+1$  means perfect predictions whereas 0 indicates random guessing. MCC is more suitable to assess classification performance than accuracy (Chicco and Jurman, 2020) as it is not easily fooled by biased classifiers in case of imbalanced datasets.
To assess the trade-off between attack (authorship attribution) and utility (sentiment analysis), we measure each method's relative gain based on the original and obfuscated classification scores: Let  $A_{o}, S_{o}$  represent the MCC scores of the author and sentiment classifiers based on the original data, and similarly, let  $A_{p}, S_{p}$  represent the scores on perturbed data normalized to the range [0, 1]. Then we define its relative gain as  $\gamma := S_{p} / S_{o} - A_{p} / A_{o}$ .
To measure semantic similarity between the anonymized and original sentences, we compute the cosine similarity of their representations obtained by SBERT (Reimers and Gurevych, 2019), which is a model that has been optimized for capturing semantic similarity between textual inputs. For language quality, we compute the average perplexity (PPL) of the pretrained GPT-2 (Radford et al., 2019) over the output sentences of each model.
Table 1: Performance of authorship and sentiment classifiers trained and evaluated on data generated by anonymization mechanisms as measured by MCC scores. Best trade-offs are identified by the relative gain metric introduced in section Section 5.1
Privacy budget ε	original	GloVe embeddings	Poincaré embeddings	Paraphrase (ε = 1/T)
∞	6	8	10	12	0.5	1	2	8	0.05	0.1	1.0	10
IMDb:
Author (static)	0.98	0.12	0.20	0.28	0.33	0.87	0.88	0.88	0.89	0.19	0.21	0.22	0.22
Author (adapt.)	0.98	0.58	0.79	0.90	0.95	0.97	0.97	0.98	0.98	0.62	0.63	0.64	0.66
Sentim. (static)	0.71	0.21	0.32	0.43	0.50	0.53	0.52	0.52	0.53	0.37	0.40	0.40	0.42
Sentim. (adapt.)	0.71	0.22	0.37	0.52	0.60	0.56	0.54	0.56	0.56	0.40	0.42	0.41	0.43
SBERT CS	1.00	0.30	0.49	0.70	0.85	0.66	0.67	0.68	0.68	0.58	0.61	0.62	0.63
PPL	44.5	5003	3544	1414	512	431	384	330	310	37.2	34.8	34.4	33.9
Yelp:
Author (static)	0.80	0.12	0.23	0.40	0.49	0.59	0.61	0.60	0.62	0.22	0.35	0.37	0.38
Author (adapt.)	0.80	0.32	0.47	0.62	0.68	0.72	0.73	0.73	0.75	0.35	0.35	0.37	0.39
Sentim. (static)	0.51	0.14	0.20	0.27	0.33	0.35	0.37	0.36	0.37	0.20	0.21	0.23	0.24
Sentim. (adapt.)	0.51	0.17	0.26	0.34	0.43	0.44	0.45	0.45	0.46	0.32	0.30	0.30	0.33
SBERT CS	1.00	0.29	0.43	0.60	0.76	0.35	0.37	0.38	0.38	0.49	0.51	0.54	0.54
PPL	99.7	13427	8555	3061	1534	1248	1232	1155	1116	148	143	138	132
We implement both mechanisms proposed in the papers by Feyisetan et al. (2020, 2019) using numpy. Concretely, we use 50-dimensional GloVe (Pennington et al., 2014) vectors as our Euclidean embeddings and train 50-dimensional Poincaré embeddings on our own. For the latter, we extract  $\sim 1,300,000$  word tuples representing hypernymy relationships for IMDb and  $\sim 1,800,000$  tuples for Yelp from WebIsADB $^3$  (Seitner et al., 2016) by removing words with less than 10 occurrences and keeping only tuples contained in the GloVe vocabulary as well as the respective review dataset $^4$ .
When encountering out-of-vocabulary (OOV) words, Algorithm 1 cannot assign embeddings and thus not perturb them, which violates DP. Also, merely removing the words does not change this as it changes the length of the output text while DP is only fulfilled for texts of the same length. For GloVe embeddings, a relevant effect in terms of experimental results is not present as the large vocabulary covers almost all words we encounter. The vocabulary size of our Poincaré embeddings is however limited ( $\sim$  10,000) and, following Feyisetan et al. (2019), does not contain stopwords. As we aim to compare methods outputting human-readable texts and the removal of stopwords clearly affects readability, we instruct the mechanism to
simply ignore OOV words. The results for removing OOV words can be found in Table 3 in the appendix.
For GPT-2 and BERT, we use the pretrained checkpoints from the HuggingFace transformers library (gpt2, bert-base-uncased; 117M, 110M parameters) and fine-tune each instance on a single NVIDIA T4 GPU.
5.3 Datasets
We conduct experiments using IMDb movie reviews (Maas et al., 2011) and Yelp business reviews<sup>5</sup> which contain author and sentiment labels in the form of ratings on the scale of 1-10 and 1-5, respectively. For both sources, we keep data from ten users with the most reviews, hereby obtaining dataset sizes of 10,000 for IMDb and 15,729 for Yelp. We simplify sentiment labels by rating movie reviews with  $\geq 5$  points and business reviews with  $\geq 3$  points as positive and the rest negative.
Table 1 shows that paraphrasing significantly outperforms word-level mechanisms in terms of protection against adaptive adversaries. When evaluating privacy and utility for static classifiers, it becomes apparent that small perturbations are enough to trick author classifiers. Therefore, for static classifiers, mechanisms with weak word-level perturbations caused by smaller  $\epsilon$  values show an equal trade-off on IMDb and a slightly better trade-off on Yelp reviews as they better preserve the sentiment
<sup>5</sup>https://www.yelp.com/dataset/
<sup>3</sup>Terms of use and license information: Appendix A.1
As the procedure was not fully described in the paper, we increased (by factor  $\geq 10$ ) the training data of the original work, hereby having a larger vocabulary and more variation in the perturbations. We do so to minimize the risk of bad results merely due to implementation issues.
than the stronger changes caused by our model. Notably, paraphrasing shows better semantic preservation as well as higher language quality as measured by PPL when comparing it to word-level mechanisms calibrated for comparable privacy protection against the author classifier. This is also visible in the exemplary outputs provided in Table 2.
Other DP mechanisms for text Earlier mechanisms for differentially private text obfuscation settled for simpler output representations: Weggenmann and Kerschbaum (2018); Fernandes et al. (2019) employ Bag-of-Words (BoW) models and produce term-frequency vectors as output. Similarly, obfuscated dense vector representations are obtained in (Beigi et al., 2019) by perturbing the output of an encoder network. While not human-readable, these vector representations can be shared for automated processing, such as topic or sentiment inference and machine learning. To generate human-readable text, Bo et al. (2021) employ an encoder-decoder model similar to ours, but without paraphrasing, and sample output words using (a two-set variant of) the Exponential mechanism (McSherry and Talwar, 2007). Weggenmann et al. (2022) propose a differentially private variation of the variational autoencoder and use it as a sequence-to-sequence architecture for text anonymization.
Authorship obfuscation without DP Approaches not following DP range from rule-based algorithms relying on human-engineered text perturbations such as synonym replacements or word removals (Bevendorff et al., 2019; Mahmood et al., 2019) to methods incorporating deep learning. Models of the latter typically incorporate discriminator networks to penalize generating author-revealing information (Shetty et al., 2018; Xu et al., 2019). Similar to DP mechanisms, previous work is concerned with learning private text vector representations (Coavoux et al., 2018b).
Differentially private optimization Differentially private optimization algorithms such as DP-SGD and related methods (Song et al., 2013; Bassily et al., 2014; Abadi et al., 2016) have emerged as effective methods for protecting the training data of a model. Recent work has shown that both generative and discriminative language models can effectively be trained with these optimization approaches (Li et al., 2021; Yu et al., 2021) and there-
fore represent an important contribution for protecting against data leakage of language models (Song and Raghunathan, 2020; Carlini et al., 2021). These methods can be seen as complementary to the approaches discussed in this paper which protect data during inference.
We discussed and demonstrated the weaknesses of word level DP mechanisms and proposed a paraphrasing model circumventing most of these. We find that our approach outperforms word level mechanisms in terms of protection against adaptive adversaries, while the latter should be favored against weaker adversaries. Future work could address integrating auxiliary adversarial losses to paraphrasing systems or enabling paraphrases that better preserve the core information of the source text.
Abuse of Anonymization Mechanisms Text Anonymization is an important field of research for the protection of privacy of individuals as well as for enabling freedom of speech. Still, anonymization mechanisms may be exploited for negative causes. Specifically, guaranteed anonymity on the internet might lead individuals to spread hate speech. Furthermore, mechanisms as ours can be used to anonymously generate and spread fake reviews or fake news. Important areas of research fighting these problems include hate speech and toxicity detection (Djuric et al., 2015; MacAvaney et al., 2019) as well as fake review detection (Mukherjee et al., 2013; Barbado et al., 2019) and fake news detection (Shu et al., 2017; Ruchansky et al., 2017).
Bias in Large Language Models Large language models such as GPT-2, which our proposed approach is based on, often inherit biases towards various demographics from the large amount of data they are trained on (Sheng et al., 2019; Abid et al., 2021). These biases can cause unforeseen effects when generating language output and could potentially alter statements of authors whose texts are being anonymized. An increasing amount of work is aiming to understand and tackle such biases in language models (Vig et al., 2020; Liang et al., 2021).
Evaluation Fairness In this paper, we evaluate our approach experimentally and compare its performance to mechanisms proposed in previous research works. Since no code was publicly released for the approaches we are comparing ours to, we implemented the mechanisms ourselves. While we replicated the original systems as close as possible to the description in the papers using all the information available, we cannot guarantee that they are exactly the same as not all the information about preprocessing and implementation details is publicly available.
We thank Zhijing Jin for the helpful discussions about the presentation of our results and the design of our paper.
Martin Abadi, Andy Chu, Ian Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, CCS '16, page 308-318, New York, NY, USA. Association for Computing Machinery.
Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Large language models associate muslims with violence. Nature Machine Intelligence, 3(6):461-463.
Mishari Almishari, Ekin Oguz, and Gene Tsudik. 2014. Fighting authorship linkability with crowdsourcing. In Proceedings of the Second ACM Conference on Online Social Networks, COSN '14, page 69-82, New York, NY, USA. Association for Computing Machinery.
M. Andres, N. Bordenabe, K. Chatzikokolakis, and C. Palamidessi. 2013. Geo-indistinguishability: Differential privacy for location-based systems. In ACM SIGSAC conference on Computer & communications security, pages 901-914. ACM.
Rodrigo Barbado, Oscar Araque, and Carlos A Iglesias. 2019. A framework for fake review detection in online consumer electronics retailers. Information Processing & Management, 56(4):1234-1244.
Raef Bassily, Adam Smith, and Abhradeep Thakurta. 2014. Private empirical risk minimization: Efficient algorithms and tight error bounds. In 2014 IEEE 55th Annual Symposium on Foundations of Computer Science, pages 464-473.
Ghazaleh Beigi, Kai Shu, Ruocheng Guo, Suhang Wang, and Huan Liu. 2019. I am not what i write: Privacy preserving text representation learning.
Janek Bevendorff, Martin Potthast, Matthias Hagen, and Benno Stein. 2019. Heuristic authorship obfuscation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1098-1108, Florence, Italy. Association for Computational Linguistics.
Haohan Bo, Steven H. H. Ding, Benjamin C. M. Fung, and Farkhund Iqbal. 2021. ER-AE: Differentially private text generation for authorship anonymization. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3997-4007, Online. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Michael Brennan, Sadia Afroz, and Rachel Greenstadt. 2012. Adversarial stylometry: Circumventing authorship recognition to preserve privacy and anonymity. ACM Trans. Inf. Syst. Secur., 15(3).
Nicholas Carlini, Florian Tramér, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650. USENIX Association.
Konstantinos Chatzikokolakis, Miguel E. Andres, Nicolás Emilio Bordenabe, and Catuscia Palamidessi. 2013. Broadening the scope of differential privacy using metrics. In Privacy Enhancing Technologies, pages 82-102, Berlin, Heidelberg. Springer Berlin Heidelberg.
Davide Chicco and Giuseppe Jurman. 2020. The advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. BMC genomics, 21(1):1-13.
Maximin Coavoux, Shashi Narayan, and Shay B Cohen. 2018a. Privacy-preserving neural representations of text. arXiv preprint arXiv:1808.09408.
Maximin Coavoux, Shashi Narayan, and Shay B. Cohen. 2018b. Privacy-preserving neural representations of text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1-10, Brussels, Belgium. Association for Computational Linguistics.
Damien Desfontaines. 2021. Local vs. central differential privacy.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of