SlowGuess/NAACL_2022_OCR huggingface dataset:
Justus Mattern
Benjamin Weggenmann
SAP Security Research
justus.mattern@rwh-aachen.de benjamin.weggenmann@sap.com
Florian Kerschbaum
florian.kerschbaum@uwaterloo.ca
As the issues of privacy and trust are receiving increasing attention within the research community, various attempts have been made to anonymize textual data. A significant subset of these approaches incorporate differentially private mechanisms to perturb word embeddings, thus replacing individual words in a sentence. While these methods represent very important contributions, have various advantages over other techniques and do show anonymization capabilities, they have several shortcomings. In this paper, we investigate these weaknesses and demonstrate significant mathematical constraints diminishing the theoretical privacy guarantee as well as major practical shortcomings with regard to the protection against deonymization attacks, the preservation of content of the original sentences as well as the quality of the language output. Finally, we propose a new method for text anonymization based on transformer based language models fine-tuned for paraphrasing that circumvents most of the identified weaknesses and also offers a formal privacy guarantee. We evaluate the performance of our method via thorough experimentation and demonstrate superior performance over the discussed mechanisms.
Computational authorship attribution approaches ranging from rule-based methods measuring character-level  $n$ -gram frequencies (Kešelj et al., 2003) to models incorporating deep learning (Shrestha et al., 2017) make it possible to identify the authors of a given text. While these technologies enable valuable applications such as supporting historians in their research, they can potentially be exploited by attackers to identify the originators of sensitive data and thus diminish the privacy of individuals. To protect the anonymity of users whose data is being shared online and used by companies and researchers, methods that anonymize the writer of given texts are necessary and of interest within
the research community and a variety of industries, specifically those handling personal information such as healthcare or financial services.
Previous work in the field of authorship obfuscation mainly focuses on two different tasks, namely learning anonymous textual vector representations for downstream tasks (Coavoux et al., 2018a; Weggenmann and Kerschbaum, 2018; Fernandes et al., 2019; Mosallanezhad et al., 2019; Beigi et al., 2019) and the development of mechanisms that transform the input sentence to remove properties revealing the author and thus output human-readable text. Works within the second category (Feyisetan et al., 2019, 2020; Xu et al., 2020b; Bo et al., 2021) typically follow a common word level framework which is characterized by the differentially private individual perturbation of word embeddings and the subsequent sampling of new words that are close to the perturbed vectors in the embedding space. Also, the majority of recent work proposing new methods for authorship obfuscation deals with the optimization and calibration of noise sampling mechanisms (Xu et al., 2020a) or the definition of new distributions to sample noise from (Feyisetan et al., 2019) as opposed to the development of entirely new methods.
In this paper, we thoroughly investigate the capabilities of word level anonymization from the theoretical perspective of differential privacy (DP) (Section 3.1), in terms of the language quality of its output (Section 3.2) as well as from a utilitarian perspective considering the ability to protect the privacy of people whose data is being used. Specifically, we extend the experimentation in papers proposing the discussed methods by testing their capability to mitigate deanonimization attacks using state-of-the-art methods on the widely used IMDb movie review and Yelp business review datasets (Section 5). We find that the technical constraints applied to fulfill DP in the local model cause strong limitations, and, more importantly, observe that,
Findings of the Association for Computational Linguistics: NAACL 2022, pages 867 - 881
despite the formal guarantees, such methods offer little protection against advanced deonymization attacks. For this reason, we advocate for approaches granting more flexibility to the text generation process and, motivated by experiments showing that human rewrites of texts gathered through crowdsourcing successfully anonymize the original authors (Almishari et al., 2014), propose an anonymization approach based on paraphrasing (Section 4) that maintains the advantages and the theoretical privacy guarantee of the discussed methods, evades most of the identified drawbacks and outperforms word level mechanisms in our experiments.
The majority of proposed text anonymization methods rely on a common framework that applies DP on a per-word level by perturbing individual word embeddings (Feyisetan et al., 2019, 2020; Xu et al., 2020b,a, 2021). In this section, we introduce the concept of DP and give an overview of the commonly used word level framework.
DP has been introduced by Dwork et al. (2006) under the name  $\epsilon$ -indistinguishability. Its goal is to give semantic privacy by quantifying the risk of an individual that results from participation in data collection. In the original, central model, we assume the collected data is stored in a central database with one record per participant. If we consider adjacent databases that differ by at most one record (pertaining to one individual), a differentially private query on both databases should yield matching results with similar probabilities, i.e., answers that are probabilistically indistinguishable. This is achieved via random mechanisms on the universe of datasets  $\mathcal{D}$  that return noisy query results, thus masking the impact of each individual.
Definition 1 (ε-DP) Let  $\epsilon >0$  be a privacy parameter. A random mechanism  $\mathcal{M}:\mathcal{D}\to \mathcal{R}$  fulfills  $\epsilon$ -DP if for all adjacent databases  $D,D^{\prime}\in \mathcal{D}$ , and all sets of possible outputs  $R\subset \operatorname {supp}\mathcal{M}$
$$
\operatorname * {P r} [ \mathcal {M} (D) \in R ] \leq e ^ {\epsilon} \cdot \operatorname * {P r} [ \mathcal {M} (D ^ {\prime}) \in R ].
$$
To make a query function  $f: \mathcal{D} \to \mathcal{R}$  differentially private, noise is calibrated to the query's sensitivity, i.e. its maximal change over all pairs of adjacent datasets  $D \sim D' \in \mathcal{D}$ . For instance, the L2-sensitivity as used for the Planar Laplace
mechanism (Chatzikokolakis et al., 2013; Andres et al., 2013; Koufogiannis et al., 2015) is
$$
\Delta_ {2} f := \max  _ {D \sim D ^ {\prime}} \| f (D) - f (D ^ {\prime}) \| _ {2}.
$$
In the local model (Duchi et al., 2013), noise is added locally at the data source, before the data is collected and stored in a central database. A basic example is randomized response (Warner, 1965), where each survey participant either provides a truthful or a random answer depending on the flip of an (unbiased) coin. The local model makes the strong assumption that any two inputs are considered adjacent, which often makes it difficult to achieve a satisfying privacy-utility trade-off.
2.1.1 Generalization with metrics
A limitation with DP is that the indistinguishability is achieved between two inputs on a per-record level regardless of their actual values. This can be especially problematic in the local model, where each user might just submit one single record, in which case a DP mechanism with small privacy parameter  $\epsilon$  would enforce each submitted record to be indistinguishable from any other, thus rendering the collected data essentially useless. Chatzikokolakis et al. (2013) argue that in some scenarios, the (in)distinguishability between two databases as enforced by a privacy mechanism should depend on the values themselves instead of the number of differing records. They hence propose a generalized notion of privacy on metric spaces where a mechanism run on nearby elements results in similar output probabilities:
Definition 2 (Metric privacy) Let  $\epsilon >0$  be a privacy parameter. On a metric space  $(\mathcal{X},d)$ , a mechanism  $\mathcal{M}$  satisfies  $\epsilon d$ -privacy if for all  $\pmb {x},\pmb{x}^{\prime}\in \mathcal{X}$  and all  $R\subset \operatorname {supp}\mathcal{M}$
$$
\Pr [ \mathcal {M} (\boldsymbol {x}) \in R ] \leq e ^ {\epsilon \cdot d (\boldsymbol {x}, \boldsymbol {x} ^ {\prime})} \cdot \Pr [ \mathcal {M} (\boldsymbol {x} ^ {\prime}) \in R ].
$$
In other words, the indistinguishability level of two points  $\pmb{x}, \pmb{x}'$  is bounded by  $\epsilon$  times their distance.
Note that we recover the original notion of central DP on the space of databases  $\mathcal{X} = \mathcal{D}$  if we use the record-level edit distance  $d_{\pm 1}$ , as datasets  $\pmb{x},\pmb{x}'\in \mathcal{D}$  differ by at most one record if and only if  $d_{\pm 1}(\pmb {x},\pmb{x}')\leq 1$ . Similarly, the local model is obtained for  $d(\pmb {x},\pmb{x}')\equiv 1$ . This motivates the following broader and formal definition of adjacency:
Definition 3 In a metric space  $(\mathcal{X},d)$ , we call two inputs  $\pmb{x},\pmb{x}^{\prime}\in \mathcal{X}$  adjacent (with respect to  $d$ ) if  $d(\pmb {x},\pmb{x}^{\prime})\leq 1$ . We write this as  $\pmb {x}\sim_{d}\pmb{x}^{\prime}$  (or  $\pmb {x}\sim \pmb{x}^{\prime}$  if  $d$  is understood from the context).
2.2 Word perturbations for anonymization
The methods investigated in this paper apply word embedding perturbation mechanisms to change individual words in a sentence, following  $\epsilon d$ -privacy with a distance metric defined for sentences  $x, x'$ . In essence, the common word level framework works as follows: Given an input sentence  $x = (x_{1}, x_{2}, \ldots, x_{n})$ , each token  $x_{i}$  is mapped to an  $n$ -dimensional pretrained word embedding  $\phi(x_{i})$ . Subsequently, an  $n$ -dimensional noise vector  $\eta$  is sampled from a multivariate probability distribution  $p_{\epsilon}(\eta)$  and added to the word embedding to obtain a noisy vector  $\hat{\phi}_{i}$ . The current word  $x_{i}$  then gets replaced by a word  $x_{i}'$  whose embedding  $\phi(x_{i}')$  is close to the noisy embedding  $\hat{\phi}_{i}$ . Given a distance metric  $d$ , commonly  $d(\boldsymbol{x}, \boldsymbol{x}') = \sum_{i=1}^{n} \| \phi(x_{i}) - \phi(x_{i}') \|$  for sentences  $\boldsymbol{x}, \boldsymbol{x}'$  of the same length, the mechanism fulfills  $\epsilon d$ -privacy. The general mechanism is outlined in Algorithm 1 and the proofs are outlined in the referenced papers.
3 Limitations of word level privacy
DP mechanisms operating on a word-by-word basis follow a comparably simpler and more straightforward algorithmic approach than deep learning models for text anonymization. This has many advantages such as lower computational expense as well as the mechanism's independence of the target dataset and domain: Most deep learning based approaches need to be trained for each dataset and set of authors individually as they require author labels to construct adversarial training objectives (Shetty et al., 2018; Xu et al., 2019). In contrast, the approaches discussed in this paper are dataset-independent and can thus be deployed immediately without a need for further training for new authors and datasets.
The simple methodology does however have its shortcomings as well. In this section, we examine these weaknesses from a theoretical standpoint taking into account both DP properties and proper
ties of the language output before assessing their effects experimentally in Section 5.
3.1 DP related constraints
We consider a mechanism  $\mathcal{M}$  that operates on a text  $\pmb{x} = (x_{1},\dots ,x_{n})$  on a word-by-word basis, i.e.,  $\mathcal{M}(\pmb {x}) = (\mathcal{M}(x_1),\dots ,\mathcal{M}(x_n))$
Length constraints A word level mechanism  $\mathcal{M}$  will produce an output that has the same length as its input. However, typical texts and sentences come in varying lengths, say  $\pmb{x} = (x_{1},\dots,x_{n})$  and  $\pmb{x}' = (x_1',\dots,x_m')$  with  $n\neq m$ . Now if we consider an outcome set  $Z_{n}$  consisting of all length- $n$  sequences (including  $\pmb{x}$ ), we obtain
$$
1 = \operatorname * {P r} [ \mathcal {M} (x) \in Z _ {n} ] \not \preceq e ^ {\epsilon} \operatorname * {P r} [ \mathcal {M} (x ^ {\prime}) \in Z _ {n} ] = 0.
$$
This contradicts the definition of pure DP and in case of approximate DP (cf. Definition 1) would require  $\delta = 1$  which is clearly not negligible.
To comply with these strong DP requirements, word level DP mechanisms such as Feyisetan et al. (2019, 2020) commonly simply limit the privacy guarantee to cover only sequences  $Z_{n}$  of a fixed length  $n$ , i.e., no formal guarantee among sentences of different lengths is provided. Consequently, the output is also fixed to length  $n$ , which affects the language capabilities of such mechanisms and severely limits the scope and expressiveness of the resulting sentences, particularly for human readers.
Linear growth of privacy budget For an  $\epsilon$ -DP mechanism  $\mathcal{M}$ , its output probabilities given two adjacent inputs have to be bounded by  $\exp(\epsilon)$ . Suppose  $\mathcal{M}$  processes each word  $x_{i}$  of a text  $\pmb{x} = (x_{1},\dots,x_{n})$  independently, using a fixed-length output strategy as described in the preceding section, with a given output  $\pmb{z} = (z_{1},\dots,z_{n})$ . Then  $\operatorname*{Pr}[\mathcal{M}(\pmb{x}) = \pmb{z}] = \prod_{i=1}^{n} p_{i}$  where  $p_{i} := \operatorname*{Pr}[\mathcal{M}(x_{i}) = z_{i}]$ . Similarly, a second text  $\pmb{x}'$  has output probabilities  $p_{i}' = \operatorname*{Pr}[\mathcal{M}(x_{i}) = z_{i}]$ , so we have  $p_{i} \leq e^{\epsilon} p_{i}'$ , and hence
$$
\begin{array}{l} \Pr [ \mathcal {M} (\boldsymbol {x}) = \boldsymbol {z} ] = \prod_ {i = 1} ^ {n} p _ {i} \leq \prod_ {i = 1} ^ {n} e ^ {\epsilon} p _ {i} ^ {\prime} \\ = e ^ {n \epsilon} \Pr [ \mathcal {M} (\boldsymbol {x} ^ {\prime}) = z ]. \\ \end{array}
$$
Therefore, the total privacy budget required by  $\mathcal{M}$  to privatize the entire sequence is bounded by  $n\epsilon$  and thus may grow linearly with its length.
Metric privacy hides this effect in the metric, since deviations in the mechanism's output proba
bilities are bounded by  $\exp (\epsilon d(\pmb {x},\pmb{x}^{\prime}))$  . By choosing a metric  $d$  that grows larger as the length of sentences increases, strong deviations can now be captured by the metric  $d$  , so the privacy budget  $\epsilon$  as its co-factor appears smaller. For instance, Feyisetan et al. (2020) use a metric  $d(\pmb {x},\pmb{x}^{\prime}) =$ $\sum \| \phi (x_i) - \phi (x_i')\|$  for strings based on embeddings  $\phi$  , which results in more summands and thus larger distances for longer strings, but not necessarily larger distances for different writing styles: Consider the following sentence pairs  $(\pmb {x},\pmb{x}^{\prime})$  and  $(y,y^{\prime})$  written by two authors each:
$\pmb{x} =$  "Today I feel great"
$\pmb{x}^{\prime} =$  "I feel great today
$\pmb{y} =$  "Today I feel great and will get a coffee"
$\pmb{y}^{\prime} =$  "I feel great and will get a coffee today
Given a non-degenerate metric  $d$ , we have both  $d(\pmb{x}, \pmb{x}^{\prime}), d(\pmb{y}, \pmb{y}^{\prime}) > 0$  since the sentences are syntactically different. One could infer that the author of the first sentence within both pairs tends to put expressions of time in the beginning whereas the other author places them at the end, but beyond that, there are arguably no differences in terms of writing style or author-revealing information one could deduce from both sentence pairs. Yet, we will likely have  $d(\pmb{x}, \pmb{x}^{\prime}) < d(\pmb{y}, \pmb{y}^{\prime})$  due to the induced growth of the distance for longer sentences. Hence, while the distance metric does reflect differences between sentences in a somewhat meaningful way, it is prone to absorb the actual privacy loss even if the writing style is almost unchanged, thus leading to values of  $\epsilon$  that are perceived as small.
Shortcomings of the local model In many likely scenarios for authorship obfuscation methods, the intention is to share obfuscated texts with other, benign entities for further processing. For a DP mechanism, this essentially corresponds to the local model where it transforms each text individually to an obfuscated output. The assumption then is that the obfuscation allows only privacy-insensitive processing so that subsequent results and inferences do not harm the privacy of the texts' authors.
Note that the DP guarantee in the local model differs substantially from what is expressed by the definition in the original central model: A central DP mechanism would aggregate the texts (records) from multiple individuals into a single result. By the definition of adjacency, central DP hides the impact of each individual's contribution in the result
by making it probabilistically indistinguishable (as determined by  $\epsilon$ ) whether an outcome was obtained with or without an individual's data. In contrast, for local DP, any two inputs are considered adjacent, so by definition, it needs to be indistinguishable whether an output was produced by one input or another. This strong condition makes it thus questionable if such data is still useful for an analyst.
Due to the nature of local DP, it typically introduces large amounts of noise and requires large amounts of data to still get meaningful results (Wood et al., 2020). A workaround often used in practice when only limited data is available is to use a larger privacy budget  $\epsilon$  than one would normally consider sufficiently privacy-preserving in the central model (Qin et al., 2016; Desfontaines, 2021). While this does permit the obfuscated data to remain useful to a benign analyst, it may also be useful to an attacker to infer privacy-sensitive information, as the formal guarantee of local DP does not specifically prevent such undesired or malicious inferences, especially when  $\epsilon$  is large.
To alleviate the strictness and implications imposed by the local model, some approaches refer to metric privacy (Chatzikolakis et al., 2013) as generalization of the original definition. Metric privacy (cf. Definition 2) brings about a change in the definition how the privacy loss  $\epsilon$  is interpreted in relation to the introduced metric and normally leads to seemingly smaller  $\epsilon$  values; however, changing to metric privacy by itself does not imply any change to the inner workings of the mechanism. We hence argue that it is less an improvement, but more a relaxation of the privacy guarantee that still shares the same fundamental criticism of local DP, e.g., our observation at the start of this section where the metric grows with the length of the text and thus hides the linear growth of the privacy budget.
3.2 Language constraints
Aside from weaknesses concerning the privacy guarantee of DP, mechanisms operating on a perword level pose two significant shortcomings in terms of their language generation capability. First, smaller privacy budgets resulting in stronger noise added to the original data tend to cause a high amount of grammatical errors. Secondly, the lack of syntactic changes to the original sentences caused by the nature of such mechanisms considerably limits the linguistic variety and thus opportunities to deceive an adversary and provide anonymity
Figure 1: Percentage of word type changes caused by the mechanism introduced by Feyisetan et al. (2020)
for the authors of the texts.
Grammatical errors increase as privacy budget shrinks Word level mechanisms perturb every token  $x_{i}$  in a sentence independently of the rest of the text as opposed to common autoregressive sequence-to-sequence models where  $p(x_{i}) = p(x_{i} \mid x_{i-1}, \ldots, x_{1})$ . This makes it difficult to maintain consistency and renders them unable to rectify grammatical errors induced by replacing a word with one of a different word type, e.g., a noun with an adjective. To estimate the effect of this, we approximate the likelihood of word type exchanges for various  $\epsilon$  values: Using the WordNet database<sup>1</sup> (Miller, 1995; Fellbaum, 2010), we assign words from the GloVe vocabulary (Pennington et al., 2014) one or multiple of the word type labels adjective, adverb, noun and verb. Subsequently, we apply the word perturbation mechanism proposed by Feyisetan et al. (2020) on a randomly selected set of 1,000 tokens and use the assigned type labels to measure whether the word type was changed<sup>2</sup> or not.
As Fig. 1 shows, a significant percentage of word type changes occur even when using comparably large  $\epsilon$  values such as 8 or 10 that grant only little privacy protection according to our evaluation in Section 5: With  $17.3\%$  and  $7.8\%$  of word types being changed with the respective epsilon values, a word type change and thus most likely a grammatical error would be induced at every 5.8th and 12.8th token, respectively.
Lack of syntactic changes As described in Section 3.1, operating on a word-by-word basis causes severe limitations to the format of the perturbed output sentences. Due to the imposed inflexibility of the text generation process, the discussed mechanisms lack the ability to rewrite given sentences by changing their syntactic properties such as word positioning and sentence length and thus mostly have to rely on lexical changes for obfuscating author-revealing features, which is highly unfavorable. For instance, if a person's writing style is characterized by heavy use of subordinate clauses resulting in very long sentences, it may be more effective to shorten sentences than merely changing individual words.
Due to these limitations, word level methods may never achieve proper anonymization, as even syntactic features alone without any semantic information are sufficient for authorship identification: Notably, Tschuggnall and Specht (2014) show that, given a collection of syntactic trees of texts written by various authors, individual style profiles can be learned to infer the authors of unseen sentences. Moreover, learned representations of syntax trees have proven to be effective for various authorship attribution tasks (Hitschler et al., 2017; Zhang et al., 2018; Jafariakinabad et al., 2019). Consequently, an effective anonymization mechanism should be able to change the syntactic properties of its input texts in order to take away important clues that adversaries could exploit to identify authors.
4 Anonymization through paraphrasing
While existing works on text anonymization that focus on word level perturbations represent very important contributions, they have some significant weaknesses as described in Section 3. In the following, we attempt to address the identified problems by proposing fine-tuning of large language models for paraphrasing as an alternative text anonymization method.
4.1 Generating paraphrases
Authorship obfuscation has been framed as a paraphrasing problem in various works with different attempts to generate adequate rewrites (Rao and Rohatgi, 2000; Keswani et al., 2016; Bevendorff et al., 2019; Mahmood et al., 2019). While computational approaches do not always show satisfying results, Almishari et al. (2014) demonstrate that rewrites of reviews gathered through crowdsourced
<sup>1</sup>Terms of use and license information: Appendix A.1
2In case of multiple word type labels for a single token (e.g. noun and verb for "escape"), we only interpreted the perturbation as a word type change if the sets of word types of the original word and the new word were disjoint.
ing reflect strongly different stylometric features from the source reviews while preserving the original content and concealing the author successfully.