SlowGuess/NAACL_2021_OCR huggingface dataset:
Haohan Bo
McGill University, Canada
haohan.bo@mail.mcgill.ca
Benjamin C. M. Fung
ben.fung@mcgill.ca
Steven H. H. Ding
Queen's University, Canada
ding@cs.queensu.ca
Farkhund Iqbal
Zayed University, UAE
farkhund.iqbal@zu.ac.ae
Most of privacy protection studies for textual data focus on removing explicit sensitive identifiers. However, personal writing style, as a strong indicator of the authorship, is often neglected. Recent studies, such as SynTF, have shown promising results on privacy-preserving text mining. However, their anonymization algorithm can only output numeric term vectors which are difficult for the recipients to interpret. We propose a novel text generation model with a two-set exponential mechanism for authorship anonymization. By augmenting the semantic information through a REINFORCE training reward function, the model can generate differentially private text that has a close semantic and similar grammatical structure to the original text while removing personal traits of the writing style. It does not assume any conditioned labels or paralleled text data for training. We evaluate the performance of the proposed model on the real-life peer reviews dataset and the Yelp review dataset. The result suggests that our model outperforms the state-of-the-art on semantic preservation, authorship obfuscation, and stylistometric transformation.
Privacy has become a vital issue in online data gathering and public data release. Various machine learning models and privacy preservation algorithms have been studied for relational data (Johnson et al., 2018), network graph data (Chen et al., 2014), and transactional data (Li et al., 2012). Some of them have been successfully adopted in real-life applications such as telemetry collection (Cortes et al., 2016). However, the studies on privacy protection for textual data are still preliminary. Most related works only focus on replacing the sensitive key phrases in the text (Vasudevan and John, 2014) without considering the author's writing style, which is indeed a strong indicator of a
person's identity. Even though some textual data, such as double-blind academic reviews, is released anonymously, the adversaries may recover the author's identity using the personal traits in writing. Stylometric techniques (Koppel et al., 2011) can identify an author of the text from 10,000 candidates. They are effective across online posts, articles, emails, and reviews (Ding et al., 2015, 2017). Nevertheless, traditional text sanitization methods (Narayanan and Shmatikov, 2008) focus on anonymizing the contents, such as patient information, instead of the writing style, so they are ineffective against writing style analysis. The original author can be easily re-identified even if protected by these traditional approaches (Iqbal et al., 2008, 2010, 2013; Schmid et al., 2015).
Only a few recent studies focus on authorship anonymization, aiming to hide the personal traits of writing style in the given textual data. Anonymouth (McDonald et al., 2012) is a semiautomatic framework that offers suggestions to users to change their writing style. Yet, this framework is not practical since it requires two datasets as a reference to compare the change in writing style. Also, the user has to make all the final modification decisions. SynTF (Weggenmann and Kerschbaum, 2018) represents a line of research that protects the privacy of the numeric vector representation of textual data. It adopts the exponential mechanism for a privacy guarantee, but the output is only an opaque term frequency vector, not an interpretable text in natural language. Furthermore, its token substitution approach does not consider the grammatical correctness and semantic.
Style transfer is another line of research that tries to generate text with controllable attributes (Shen et al., 2017; Hu et al., 2017; Sennrich et al., 2016). Representative models (Hu et al., 2017) can control the sentiment and tense of the generated text. However, they do not modify the personal traits in writing. Their applications on sentiment and word
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3997-4007 June 6-11, 2021. ©2021 Association for Computational Linguistics
reordering correspond to the content of the text more than the writing style. We argue that their definition of styles, such as sentiment or tense, is different from the personal linguistic writing characteristics that raise privacy concern. A4NT (Shetty et al., 2018) is a generative neural network that sanitizes the writing style of the input text. However, it requires text samples to be labeled with known author identities. It is not applicable to many textual data publishing scenarios. Additionally, according to the samples provided in the paper, it has difficulties keeping the same semantic meaning between the original and the generated text. Without using any privacy model, A4NT does not provide any privacy guarantee.
To address the aforementioned issues, we propose an Embedding Reward Auto-Encoder (ER-AE) to generate differentially private text. Relying on differential privacy, it protects the author's identity through text indistinguishability without assuming any specific labels, any parallel data or any assumption on the attacker. It guards the privacy of the data against the worst information disclosure scenario. ER-AE receives the original text as input and generates a new text using the two-set exponential mechanism. We propose a REINFORCE (Sutton et al., 2000) embedding reward function to augment the semantic information during the text generation process. The model can keep the generated text a close semantic and sentiment similarity to the original while providing a guarantee that one can hardly recover the original author's identity. Unlike the aforementioned authorship anonymization works, ER-AE produces human-friendly text in natural language. Our key contributions are summarized as follows:
- The first differentially private authorship anonymization model that can generate human-friendly text in natural language, instead of a numeric vector.
- A novel two-set exponential mechanism to overcome the large output space issue while producing meaningful results.
- A novel combination of a differential privacy mechanism with a sequential text generator, providing a privacy guarantee through a sampling process.
- A new REINFORCE reward function that can augment the semantic information through external knowledge, enabling better preservation of the semantic similarity in the data synthesis
- Comprehensive evaluations on two real-life datasets, namely NeurIPS & ICLR peer reviews and Yelp product reviews, show that ER-AE is effective in obfuscating the writing style, anonymizing the authorship, and preserving the semantics of the original text.
All the source code and data are publicly accessible for reproducibility and transferability. $^{1}$
Differential Privacy. Recently, differential privacy has received a lot of attention in the machine learning community. The deep private autoencoder (Phan et al., 2016) is designed to preserve the training data privacy. Their purpose is to guarantee that publishing the trained model does not reveal the privacy of individual records. Our purpose is different. We publish the differentially private data generated by the model, rather than the model itself. Most existing models for differentially private data release, such as Chen et al. (2014), focus on different types of data rather than text. One recent work (Weggenmann and Kerschbaum, 2018) aims to protect privacy in text data using the exponential mechanism. However, it releases the term frequency vectors instead of a readable text. This approach limits the utility of published data to only the applications that assume term frequency as features. In contrast, our goal is to generate differentially private text in a natural language without compromising individual privacy.
Writing Style Transfer. Studies on writing style transferal try to change the writing style revealed from the text according to a given author. Shetty et al. (2018) design a GAN to transfer Obama's text to Trump's style. A sequence to sequence (seq2seq) model is proposed by Jhamtani et al. (2017) to transfer modern English into Shakespearean English. Shetty et al.(2017) design a model with a cross-alignment method to control the text sentiment while preserving semantic. These models can also be applied to writing style anonymization. However, these studies require the data to be labeled with authorship identity. They assume a number of known authors. In contrast, ours does not assume any label information.
Writing Style Obfuscation. Writing style obfuscation studies try to hide the identity of an au
<sup>1</sup>https://github.com/McGill-DMaS/Authorship-Anonymization
thor. Anonymouth (McDonald et al., 2012) is a tool that utilizes JStylo to generate writing attributes. It gives users suggestions on which way they can anonymize their text according to two reference datasets. (Kacmarcik and Gamon, 2006) also propose a similar architecture to anonymize text. However, instead of directly changing the text, they all work on the term frequency vector, whose real-life utility is limited. Compared with semi-automatic methods that require users to make a decision, our approach provides an end-to-end solution that directly learns from data.
Adjacency is a key notion in differential privacy. One of the commonly used adjacency definitions is that two datasets  $D_{1}$  and  $D_{2}$  are adjacent if  $D_{2}$  can be obtained by modifying one record in  $D_{1}$  (Dwork et al., 2010). Differential privacy (Dwork et al., 2006) is a framework that provides a rigorous privacy guarantee on a dataset. It demands inherent randomness of a sanitization algorithm or generation function:
Definition 1. Differential Privacy. Two datasets are considered as adjacent if there is only one single element is different. Let privacy beget  $\epsilon >0$ , a randomized algorithm  $\mathcal{A}:D^n\to Z$ , and the image of  $\mathcal{A}$ :  $im(\mathcal{A})$ . The algorithm  $\mathcal{A}$  is said to preserve  $\epsilon$ -differential privacy if for any two adjacent datasets  $D_{1},D_{2}\in D^{n}$ , and for any possible set of output  $Z\in im(\mathcal{A})$ :
$$
P r \left[ \mathcal {A} \left(D _ {1}\right) \in Z \right] \leq e ^ {\epsilon} \cdot P r \left[ \mathcal {A} \left(D _ {2}\right) \in Z \right] \square
$$
It guarantees that the result from a given algorithm  $\mathcal{A}$  is not sensitive to a change of any individual record in  $D$ .  $\epsilon$  denotes the privacy budget, the allowed degree of sensitivity. A large  $\epsilon$  implies a higher risk to privacy. However,  $\epsilon$  is a relative value that implies different degrees of risk given different problems (Weggenmann and Kerschbaum, 2018). Some studies (Sala et al., 2011) use a large  $\epsilon$ , while the others (Chen et al., 2014) use a smaller value.
Adversary Scenario. Generally in an authorship identification problem, one assumes that the attacker holds an anonymous text authored by one of the suspects from the dataset. The attacker aims to infer the true author of the anonymous text based on a set of reference texts from each suspect. However, this scenario assumes certain information on the applicable dataset, such as author labels and the number of reference text samples. Therefore,
following (Weggenmann and Kerschbaum, 2018), we define that any two pieces of text as adjacent datasets.
Adjacency. Any two pieces of text can be considered adjacent in the strictest scenario that datasets  $D_{1}$  and  $D_{2}$  both have only one record, and  $D_{2}$  can be obtained by editing one record in  $D_{1}$  following Definition 1. With differential privacy, we can have text indistinguishability: one cannot distinguish the identity of any text to another. In our case, the identity of a text corresponds to the author who wrote the text. Along with this, the attacker would fail in the original authorship identification scenario since the anonymous text is indistinguishable from the rest of the dataset.
Our definition follows Weggenmann and Kerschbaum (2018)'s idea, leading to the strictest and most conservative definition of adjacency.
Definition 2. Differentially Private Text Generation. Let  $\mathbb{D}$  denote a dataset that contains a set of texts where  $x\in \mathbb{D}$  is one of them.  $|x|$ , the length of the text, is bound by  $l$ . Given  $\mathbb{D}$  with a privacy budget  $\epsilon$ , for each  $x$  the model generates another text  $\tilde{x}_{dp}$  that satisfies  $\epsilon l$ -differential privacy.
Following the above definitions, any two datasets that contain only one record are probabilistically indistinguishable w.r.t. a privacy budget  $\epsilon$ . It directly protects the identity of an individual record, disregarding whether some of the records belong to the same author or not. It assumes that every record is authored by a different author, which is the strictest situation. Technically, the proposed text generation approach protects the writing style by reorganizing the text, replacing tokens with different spelling, removing the lexical, syntactical and idiosyncratic features of the given text. The above definition is based on SynTF (2018), but our target is readable text rather than numeric vectors, which is more challenging.
4 ER-AE for Differentially Private Text Generation
Figure 1 depicts the overall architecture of our proposed ER-AE model, which consists of an encoder and a generator. The encoder receives a sequence of tokens as input and generates a latent vector to represent the semantic features. The generator, which is incorporated with the two-set exponential mechanism, can produce differentially private text according to the latent vector. ER-AE is trained
Figure 1: Overall architecture of ER-AE.
by combining a reconstruction loss function and a novel embedding loss function.
Algorithm 1 Generation Procedure of ER-AE
Input: Text:  $x$ , Parameters:  $\theta$ , Encoder:  $E_{\theta}()$ , Generator:  $G_{\theta}()$ , Privacy budget:  $\epsilon$ .
Produce the latent vector:  $E_{\theta}(x)$ .
Get probabilities of new tokens:  $Pr[\tilde{x}] \gets G_{\theta}(E_{\theta}(x))$ .
for  $i\gets 1$  to length of  $x$  do
Build two candidate token sets based on  $Pr[\tilde{x}]: S, O$ .
Apply exponential mechanism to choose token set:  $T$ .
Randomly sample new  $i$ -th token from  $T$ :  $\tilde{x}_{dp}[i]$ .
Output: Differentially Private Text:  $\tilde{x}_{dp}$
Our ER-AE model starts with a basic sequence-to-sequence (seq2seq) auto-encoder structure. Given a text  $x$ , its tokens  $\langle x_1 \ldots x_l \rangle$  are firstly converted into a sequence of embedding vectors  $\langle Em(x_1) \ldots Em(x_l) \rangle$  by  $Em: \mathcal{V} \to \mathbb{R}^{m_1}$ , where  $\mathcal{V}$  is the vocabulary across the dataset and  $m_1$  is the embedding dimension. On its top, we apply a bi-directional recurrent neural network with Gated Recurrent Unit (GRU) (Cho et al., 2014) that leverages both the forward and backward information. GRU achieves a comparable performance to LSTM with less computational overhead (Cho et al., 2014). Then, the produced final state vectors from both directions,  $s_f$  and  $s_b$ , are concatenated and linearly transformed to be a latent vector  $E(x)$ .  $m$  is the hidden state dimension for the GRU function.
$$
E (x) = \boldsymbol {W} _ {h} \times \operatorname {c o n c a t} \left(\boldsymbol {s} _ {f}, \boldsymbol {s} _ {b}\right), \tag {1}
$$
where  $\pmb{s}_f, \pmb{s}_b \in \mathbb{R}^m$ ,  $\pmb{W}_h \in \mathbb{R}^{h \times 2m}$ .
The generator is another recurrent neural network with GRU. It generates a text token-by-token. For each timestamp  $i$ , it calculates a logit weight  $z_{iv}$  for every candidate token  $v \in \mathcal{V}$ , conditioned on the latent vector, last original token  $x_{i-1}$ , and
the last hidden state  $s_{i-1}$  of the GRU function.
$$
z _ {i v} = \boldsymbol {w} _ {v} ^ {\top} \mathrm {G R U} (E (x), E m (x _ {i - 1}), \boldsymbol {s} _ {i - 1}) + b _ {v}
$$
Let  $\tilde{x}_i$  denote the random variable for the generated token at timestamp  $i$ . Its probability mass function is proportional to each candidate token's weight  $z_{ti}$ . This is modeled through a typical softmax function:
$$
P r \left[ \tilde {x} _ {i} = v \right] = \exp \left(z _ {i v}\right) / \sum_ {v ^ {\prime} \in \mathcal {V}} \exp \left(z _ {i v ^ {\prime}}\right) \tag {2}
$$
For each timestamp  $i$ , a typical seq2seq model generates text by applying  $\operatorname{argmax}_{v \in \mathcal{V}} Pr[\tilde{x}_i = v]$ . However, this process does not protect the privacy of the original data.
4.1 Differentially Privacy Text Sampling with Two-Set Exponential Mechanism
To protect an individual's privacy and hide the authorship of the original input text, we couple differential privacy mechanism with the above sampling process in the generator. The exponential mechanism (McSherry and Talwar, 2007) can be applied to both numeric and categorical data (Fernandes et al., 2018). It is effective in various sampling process for discrete data. It guarantees privacy protection by injecting noise into the sampling process:
Definition 3. Exponential Mechanism. Let  $\mathcal{M}$  and  $\mathcal{N}$  be two enumerable sets. Given a privacy budget  $\epsilon >0$ , a rating function  $\rho \colon \mathcal{M}\times \mathcal{N}\to \mathbb{R}$ . The probability density function of the random variable  $\varepsilon_{\epsilon ,\rho}(m)$ ,  $Pr[\varepsilon_{\epsilon ,\rho}(m) = n]$  is:
$$
\frac {\exp \left(\frac {\epsilon}{2 \Delta} \rho (m , n)\right)}{\sum_ {n ^ {\prime}} \exp \left(\frac {\epsilon}{2 \Delta} \rho (m , n ^ {\prime})\right)} \tag {3}
$$
where  $\Delta$ , the sensitivity, means the maximum difference of rating function values between two adjacent datasets, and  $m \in \mathcal{M}$ ,  $n \in \mathcal{N}$ .
However, according to Weggenmann and Kerschbaum (2018), the exponential mechanism requires a large privacy budget to produce meaningful results while the output space is large, the vocabulary size in our case. It's nontrivial to randomly sample a good result directly among 20,000 candidates.
To tackle the large output space issue, inspired by subsampled exponential mechanism (Lantz et al., 2015), we propose a two-set exponential mechanism to produce meaningful results with a better privacy protection. Instead of using a database independent distribution, we use a model-based distribution to generate subsets of tokens.
Definition 4. Two-Set Exponential Mechanism. Let  $\mathcal{V}$  be a enumerable set with size  $s$ . Given the model-based probabilities of each item in  $\mathcal{V}$ ,  $Pr[v]$  for  $v \in \mathcal{V}$ , an item set  $S$  of size  $k$  is built by repeatedly sampling proportional to  $Pr[v]$  with replacement. Other items are denoted as set  $O$ , where  $\mathcal{V} = O \cup S$ ,  $\emptyset = O \cap S$ . Let  $\mathcal{N} = \{S, O\}$ . An item set  $C_{dp}$  is chosen from  $\mathcal{N}$  through the exponential mechanism with a rating function  $\rho$ :  $\sum_{v \in C} Pr[v] / \sum_{C' \in \mathcal{N}, v' \in C'} Pr[v']$ . Given  $\epsilon > 0$ ,  $N \in \mathcal{N}$ , the probability density function (PDF) of the random variable  $\varepsilon_{\epsilon, \rho}(C)$ ,  $Pr[\varepsilon_{\epsilon, \rho}(C) = N]$ , is:
$$
\frac {\exp \left(\frac {\epsilon}{2 \Delta} \rho (C , N)\right)}{\sum_ {N ^ {\prime} \in \mathcal {N}} \exp \left(\frac {\epsilon}{2 \Delta} \rho (C , N ^ {\prime})\right)} \tag {4}
$$
After choosing the set,  $C_{dp}$ , an item is randomly picked from the chosen set:  $v \sim \text{Random}(C_{dp})$ . Thus, given  $v, w \in \mathcal{V}$ ,  $Pr[\varepsilon_{\epsilon, \rho}(v) = w]$  is:
$$
P r [ w S ] * P r [ \varepsilon_ {\epsilon , \rho} (C) = S | w S ] * P r [ w | w S, S ] +
$$
$$
P r [ w O ] * P r [ \varepsilon_ {\epsilon , \rho} (C) = O | w O ] * P r [ w | w O, O ],
$$
where  $Pr[wS]$  and  $Pr[wO]$  are respectively the probability of  $w$  in set  $S$  and  $O$ ,  $Pr[wO] = (1 - Pr[w])^k$ .
Theorem 1. Two-Set Exponential Mechanism.2
Given a privacy budget  $\epsilon > 0$  and the size of output space  $s$ , two-set exponential mechanism is  $(\epsilon + \ln(s))$ -differentially private.
By plugging our model with this mechanism, we have the probability mass function for  $\varepsilon_{\epsilon,\rho_i}(\tilde{x}_i)$ :  $Pr[\varepsilon_{\epsilon,\rho_i}(\tilde{x}_i) = tk]$ . This function models the disturbed probability distribution for all the alternative token  $tk$  to replace the original variable. According to Theorem 4, sampling from  $\varepsilon_{\epsilon,\rho_i}(\tilde{x}_i)$  for each
timestamp  $i$  is  $(\epsilon + \ln(s))$ -differentially private. Recall that in Definition 1, the timestamp is bound by  $l$ . To generate text  $\tilde{x}_{dp}$ , the generator samples a token for timestamp  $i$  through the chosen set  $T_i$ :
$$
\tilde {x} _ {d p} [ i ] \sim \operatorname {R a n d o m} \left(T _ {i}\right) \quad \text {f o r} i \in [ 1, l ] \tag {5}
$$
The composition theorem (Dwork et al., 2014) is an extension to differential privacy. By repeating  $n$ $\epsilon$ -differentially private algorithms, the complete process achieves an  $\epsilon n$ -differential privacy. Algorithm 1 shows the differentially private text generation of ER-AE. As proved in Appendix A:
Theorem 2. Differentially Private Text Sampling. Given a privacy budget  $\epsilon >0$ , a sequence length  $l$ $>0$ , the generator's sampling function in Eq. 5 is  $(\epsilon +\ln (s))\ast l$ -differentially private.
4.2 Initial Grammar and Semantic Preservation
To generate a human-friendly text that has a close semantic to the original one, we need to have a high-quality rating function  $\rho_{i}$  for Eq. 4. This is achieved by training the ER-AE model's encoder to extract semantic information, and its generator to learn the relationships among the tokens for prediction. We follow an unsupervised learning approach since we do not assume any label information. First, we adopt the reconstruction loss function:
$$
\mathcal {L} _ {\text {r e c o n}} = \sum_ {x _ {i} \in x, x \in \mathbb {D}} - \log P r [ \tilde {x} _ {i} = x _ {i} ] \tag {6}
$$
It maximizes the probability of observing the original token  $x_{i}$  itself for the random variable  $\tilde{x}_i$ . In the recent controllable text generation models, the reconstruction loss plays an important role to preserve grammar structure and semantics of input data (Shetty et al., 2018) when combined with the other loss.
4.3 REINFORCE Training for Semantic Augmentation
Diving into the optimization aspect of the softmax function, the reconstruction loss function above encourages the model to produce a higher probability on the original token while ignoring the rest candidates. It does not consider the other tokens that may have a similar meaning under a given context. This issue significantly limits the variety of usable alternative tokens. Additionally, this loss function relies on a single softmax function for multi-object
The proof is provided in Appendix B