SlowGuess/NAACL_2021_OCR huggingface dataset:
learning, it cannot provide the expressiveness required by the language model (Yang et al., 2018). We inspect the candidates and in most of the cases, only the top-ranked token fits the context in the text. This is problematic because the mechanism for our sampling process also relies on the other candidates to generate text. To address the above issue, we propose a novel embedding reward function using the pre-trained word embeddings. Word representation learning models show that discrete text tokens' semantic can be embedded into a continuous latent vector space. The distance between word embedding vectors can be a reference to measure the similarity between different words. To encourage our rating function  $\rho_{i}$  to learn richer and better substitute tokens, we propose a reward function that leverages the semantics learned from the other corpus. The text dataset to be anonymized and released can be small, and the extra semantic knowledge learned from the other corpus can provide additional reference for our rating function. This reward function is inspired by the Policy Gradient loss function (Sutton et al., 2000),  $\mathcal{L}_{embed}$  is:
$$
\begin{array}{l} - \sum_ {x _ {i} \in x, x \in \mathbb {D}} \left(\sum_ {v \in \mathbb {E} _ {k} (\tilde {x} _ {i})} \log (P r [ \tilde {x} _ {i} = v ]) \gamma (x _ {i}, v) \right. \\ \left. + \sum_ {w \sim \mathbb {V} _ {k}} \log (P r [ \tilde {x} _ {i} = w ]) \gamma (x _ {i}, w)\right) \\ \end{array}
$$
Generally, this reward function assigns credits to the under-rated tokens in the reconstruction loss function. Recall that  $\mathbb{D}$  is the original dataset and  $x$  is one of its texts. At time step  $i$ , this reward function first assigns rewards to the top- $k$  selected tokens, denoted as  $\mathbb{E}_k(\tilde{x}_i)$ , according to probability estimates for random variable  $\tilde{x}_i$  in Eq. 2. The rewards are proportional to their semantic relationship to the original token  $x_i$ . It is defined as a function  $\gamma: \mathcal{V} \times \mathcal{V} \to \mathbb{R}$ ,  $\gamma(w, v)$  is:
$$
\min  \left(\operatorname {c o s i n e} (E m (w), E m (v)), 0. 8 5\right) \tag {7}
$$
The min function avoids the generator focusing only on the original token. By assigning rewards to  $\mathbb{E}_k(\tilde{x}_i)$ , it encourages the other candidates having a close semantic to the targeted one, but it may fail to reach infrequent tokens. Therefore, in the second part of the reward function, we encourage the model to explore less frequent tokens by random sampling candidates as  $\mathbb{V}_k$ . This design balances the exploitation (top- $k$ ) and the exploration  $(\mathbb{V}_k)$  in reinforcement learning.
During training, the model is firstly pre-trained by minimizing the reconstruction loss in Eq. 6 through the Adam optimizer, and adopts the embedding reward loss later. The total loss is
$$
\mathcal {L} = \lambda_ {\text {r e c o n}} \times \mathcal {L} _ {\text {r e c o n}} + \lambda_ {\text {e m b e d}} \times \mathcal {L} _ {\text {e m b e d}} \tag {8}
$$
Specifically, the reconstruction loss can lead the model to generate grammatically correct text, and the embedding reward loss encourages the model to focus more on semantically similar tokens. The balance of the two loss functions are controlled by  $\lambda_{\text{recon}}$  and  $\lambda_{\text{embed}}$ .
All the experiments are carried out on a Windows Server equipped with two Xeon E5-2697 CPUs (36 cores), 384 GB of RAM, and four NVIDIA Titan XP GPU cards. We evaluate ER-AE on two different datasets with respect to its effectiveness for privacy protection and utility preservation.
- Yelp Review Dataset<sup>3</sup>: All the reviews and tips from the top 100 reviewers ranked by the number of published reviews and tips. It contains 76,241 reviews and 200,940 sentences from 100 authors.
- Academic Review Dataset: All the public reviews from NeurIPS (2013-2018) and ICLR (2017) based on the original data and the web crawler provided by (Kang et al., 2018). It has 17,719 reviews, 268,253 sentences, and the authorship of reviews is unknown.
Each dataset is divided into 70/10/20 for train/dev/evaluation respectively. As mentioned in the related work discussion, most of the controllable text generation and style transferal studies rely on known authorship or other labels. Other generation models such as paraphrasing, however, hold an essentially different goal and cannot provide a privacy guarantee on the generated data. They are not applicable to our problem. Therefore, we pick SynTF (Weggenmann and Kerschbaum, 2018) and different generation and sampling models for evaluation:
- Random Replacement (Random-R): This method generates a new text by replacing each token in the text by randomly picking substitution from the vocabulary.
- AE with Differential Privacy (AE-DP): Extended version of AE with the added two-set exponential mechanism for text generation. It does not include the embedding reward.
<sup>3</sup>http://www.yelp.com/dataset_challenge
Table 1: Results for each evaluation metric on both datasets.  $\uparrow$  indicates the higher the better.  $\downarrow$  indicates the lower the better.
Yelp (100-author)	Conferences’ Dataset
Model	USE ↑	Authorship ↓	Stylometric↑	USE↑	Stylometric↑
Original text	1	0.5513	0	1	0
Random-R	0.1183	0.0188	62.99	0.1356	65.624
AE-DP	0.6163	0.097	11.443	0.614	9.859
SynTF (2018)	0.1955	0.0518	26.3031	0.2161	25.95
ER-AE (ours)	0.7548	0.0979	13.01	0.7424	9.838
Figure 2: Privacy v.s. Utility. Comparing USE similarity (utility), authorship identification error rate (privacy) and Stylometrics L2 distance (privacy) for different  $\epsilon$ s on applicable datasets.
Table 2: The intermediate result of top five words and their probabilities at that the third and the forth generation steps.
Input: there are several unique hot dog entrees to choose.
several
AE-DP	several 0.98, those 0.007, some 0.003
various 0.002, another 0.001
ER-AE	many 0.55, some 0.20, several 0.14
different 0.04, numerous 0.03
unique
AE-DP	unique 0.99, different 0.0001, new 3.1e-05, nice 2.5e-05, other 2.1e-05
ER-AE	unique 0.37, great 0.21, amazing 0.15, wonderful 0.1, delicious 0.05
Table 3: The estimated probability of a good candidate sampled with different mechanisms.
Input: there are several unique hot dog entrees to choose.
several	unique
Exponential Mechanism	0.0017	0.00091
Two-Set Exponential Mechanism	0.7411	0.6794
- SynTF (Weggenmann and Kerschbaum, 2018): We directly generate the tokens through SynTF's differentially private sampling function, without further extraction of the frequency vector.
SynTF is a state-of-the-art generation model that satisfies differential privacy property on textual data. The other two simple baselines are for ablation test purposes.
For ER-AE, we adopted a two-layers stacked GRU network for both the encoder and the generator. There are 512 cells in each GRU layer. The vocabulary size is 20,000, separately built for each dataset. All the word embeddings in our model come from the pre-trained BERT embeddings provided by (Devlin et al., 2019), which has a dimension of 768 for each embedding. The maximum input length of our model is 50, the learning rate is 0.001, the  $k$  for embedding reward loss function is 5, the  $\lambda_{\text{recon}}$  is 1, the  $\lambda_{\text{embed}}$  is 0.5, and the batch size is 128. The  $k$  in two-set exponential mechanism is 5. ER-AE is implemented in TensorFlow (Abadi et al., 2016), and it uses the tokenizer in the NLTK library. Some traditional tricks for text generation, such as beam search, are not mentioned because they are incompatible with differential privacy. All the models are evaluated from three aspects: semantic preservation, privacy protection, and stylometric changes:
- Semantic Preservation (USE): A pre-trained Universal Sentence Embedding (USE) model from Google. It can embed a sentence into a latent vector that represents its semantics. It is widely used for supervised NLP tasks such as sentiment analysis. We measure the degree of semantic preservation using the cosine similarity between the latent vector of the original text and
4https://tfhub.dev/google/universal-sentence-encoder/1
Table 4: Sample sentences generated by models.
Input	the play place is pretty fun for the little ones .
Random-R	routing longtime 1887 somalia pretty anatomical shallow the dedicated drawer rosalie
AE-DP	employer play lancaster mute fish fun for wallace little chandler .
SynTF	conditioned unique catherine marquis governing skinny garment hu vivid . insists
ER-AE	the play place is pretty nice with the little ones !
Input	i also ordered a tamarind margarita and it was great .
Random-R	substantial char recommended excavation tamarind coil longitudinal recover verify great housed
AE-DP	intersection also ordered service tamarind drooling scratched denis monkfish motions .
SynTF	carnage spence unsigned also clinging said originated beacon liking strike accomplishments
ER-AE	i also requested a tamarind margarita and it were great .
Input	i ’m not complaining because you do get exactly what you pay for .
Random-R	substantial char recommended excavation tamarind coil longitudinal recover verify great housed
AE-DP	comic-book ’m not mins because you donnelly get exactly tenderloin nerves bottomless for aldo box
SynTF	leaf penetrated amounted jolted courageous socket fades unwilling tu judges regional numbering
ER-AE	i ’m not disappointing because you do make occult what you pay for .
Input	the manuscript is well written is provides good insight into the problem .
AE-DP	the fig2c is well 1102-103 wish provides horseshoe insight into the problem compositionality
SynTF	ness voice incoming depending entrances somehow priscilla rows romantic oblivious mall
ER-AE	the manuscript is well edited has provides excellent insight into the problem .
Input	in particular , the generality of the approach is very well presented .
SynTF	wife pierced rotate specialist probe elects prussian beatty eccentric sweating .
ER-AE	in particular , this generality of an approach is very well written well
one of the generated text.
- Privacy Protection (Authorship): A state-of-the-art authorship identification neural network model (Sari et al., 2017) to identify the authorship of generated text. The model is firstly trained on the training dataset, and the performance is evaluated on the testing set. The author's privacy is protected if s/he cannot be identified using authorship identification techniques.
- Stylometric Changes: Well-established stylistic context-free features such as text length and a number of function words. We adopt Stylo-Matrix (Ding et al., 2017) for an aggregation of features in (Iqbal et al., 2013; Zheng et al., 2006). The feature vector change is measured by the difference in L2 norm.
Quantitative Evaluation (Table 1). With a low utility (USE) score around 0.2 for both datasets, SynTF, and Random-R generate grammatically incorrect text and completely change the meaning of the original one. In contrast, ER-AE without semantic augmentation through REINFORCE training, denoted as AE-DP, achieves a much higher utility score of around 0.61. The full model ER-AE, with an  $\epsilon$  of 3, achieves the highest utility score of 0.75 for Yelp reviews and 0.74 for peer reviews. AEDP, SynTF, and ER-AE all significantly reduce the chance of a successful authorship identification attack from  $55\%$  to lower than  $10\%$  in the Yelp data and introduce a variation in stylometric features
of more than 10 in magnitude in the peer review dataset. They are all effective and competitive on removing the personal writing trait from the text data, but as mentioned above, AE-DP achieves the best and a much higher utility score. Although Random-R performs better on privacy protection, its generated texts are irrelevant to the original. Overall, with a competitive performance on anonymization, ER-AE performs significantly better than all of the other models on utility.
Impact of Embedding Reward. Table 4 shows that the embedding reward plays an important role in selecting semantically similar candidates for substitution. AE-DP assigns a large probability to the original token and a tiny probability to the others. If applied with the mechanism, it is more likely to pick a semantically irrelevant token. ER-AE shows a smoother distribution and assigns higher probabilities to top-ranked semantically relevant tokens. Its generated candidates are better.
Case Study. Table 4 shows that both SynTF and Random-R cannot generate human-friendly text. Due to the issue of reconstruction loss function [6], AE-DP cannot substitute token with similarly semantic tokens and destroys the semantic meaning. ER-AE, powered by embedding reward, can substitute some tokens with semantically similar ones: "written" is replaced by "edited", and the whole sentence still makes sense. Besides, it can preserve the grammatical structure of the input. However,
due to some missing information from word embeddings, the model would fail to generate good candidates for sampling. The third sample replaces "exactly" with "occult". ER-AE still performs way better than other models.
Utility vs. Privacy. The privacy budget  $\epsilon$  controls the trade-off between privacy and utility. A larger  $\epsilon$  implies better utility but less protection on privacy. However, this is a relative value that implies different degrees of risk given different problems (Weggenmann and Kerschbaum, 2018; Fernandes et al., 2018). As proved by Weggenmann and Kerschbaum (2018) a higher  $\epsilon$  is intrinsically necessary for a large output space, in our case the vocabulary, to generate relevant text. In fact, we have already significantly reduced the optimal  $\epsilon$  value of 42.5 used by Weggenmann and Kerschbaum (2018) to around 13, given the same dataset. One possible way to lower the bound of  $\epsilon$  is to directly factor in authorship and utility, such as topics, into the privacy model. However, it limits applicable to datasets.
Exponential Mechanism vs. Two-Set Exponential Mechanism. In Table 3, we estimated the probability of a meaningful token (among top 5 semantically similar tokens) is sampled based on the intermediate probabilities in Table 2. Given a large output space of 20,000, the exponential mechanism is not likely to sample a meaningful token with a probability of  $0.01\%$ . However, the two-set exponential mechanism dramatically improves it from  $0.01\%$  to around  $70\%$ . Our generator has a much higher chance to generate meaningful results with a similar privacy budget.
In this paper, we propose a novel model, ER-AE, to protect an individual's privacy for text data release. We are among the first to fuse the differential privacy mechanisms into the sequence generation process. We demonstrate the effectiveness of our model on the Yelp review dataset and two peer reviews datasets. However, we also find that ER-AE performs not very well on long texts due to the privacy budget accounting issue. Our future research will focus on improving long texts generation with better budget allocation scheme.
Our model outperforms others on authorship obscuration and semantic preservation. Similar to other
text generation tools, this model may be abused to generate fake reviews, but this can be assuaged by using fake review detection methods. This research work directly contributes to the area of privacy protection and indirectly promotes freedom of speech and freedom of expression in cyberspace.
This research is supported in part by Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grants (RGPIN-2018-03872), Canada Research Chairs Program (950-232791), and Provost Research Fellowship Award (R20093) and Research Incentive Funds (R18055) from Zayed University, United Arab Emirates. The Titan Xp used for this research was donated by the NVIDIA Corporation.
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI).
Rui Chen, Benjamin C. M. Fung, Philip S. Yu, and Bipin C. Desai. 2014. Correlated network data publication via differential privacy. The VLDB Journal—The International Journal on Very Large Data Bases, 23(4).
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. Syntax, Semantics and Structure in Statistical Translation.
Jorge Cortés, Geir E Dullerud, Shuo Han, Jerome Le Ny, Sayan Mitra, and George J Pappas. 2016. Differential privacy in control and network systems. In IEEE 55th Conference on Decision and Control (CDC). IEEE.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).
Steven H. H. Ding, Benjamin C. M. Fung, and Mourad Debbabi. 2015. A visualizable evidence-driven approach for authorship attribution. ACM Transactions on Information and System Security (TISSEC), 17(3).
Steven H. H. Ding, Benjamin C. M. Fung, Farkhund Iqbal, and William K Cheung. 2017. Learning stylo-metric representations for authorship analysis. IEEE transactions on cybernetics, (99).
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. Springer.
Cynthia Dwork, Aaron Roth, et al. 2014. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4).
Cynthia Dwork, Guy N Rothblum, and Salil Vadhan. 2010. Boosting and differential privacy. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science. IEEE.
Natasha Fernandes, Mark Dras, and Annabelle McIver. 2018. Author obfuscation using generalised differential privacy. arXiv preprint arXiv:1805.08866.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning, JMLR.
Farkhund Iqbal, Hamad Binsalleeh, Benjamin C. M. Fung, and Mourad Debbabi. 2010. Mining writeprints from anonymous e-mails for forensic investigation. Digital Investigation, 7(1-2).
Farkhund Iqbal, Hamad Binsalleeh, Benjamin C. M. Fung, and Mourad Debbabi. 2013. A unified data mining solution for authorship analysis in anonymous textual communications. Information Sciences, 231.
Farkhund Iqbal, Rachid Hadjidj, Benjamin C. M. Fung, and Mourad Debbabi. 2008. A novel approach of mining write-prints for authorship attribution in e-mail forensics. Digital Investigation, 5(1):S42-S51.
Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Eric Nyberg. 2017. Shakespearizing modern language using copy-enriched sequence to sequence models. In Proceedings of the Workshop on Stylistic Variation.
Noah Johnson, Joseph P Near, and Dawn Song. 2018. Towards practical differential privacy for sql queries. Proceedings of the VLDB Endowment, 11(5).
Gary Kacmarcik and Michael Gamon. 2006. Obfuscating document stylometry to preserve author anonymity. In Proceedings of the COLING/ACL on Main conference poster sessions. Association for Computational Linguistics.
Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (peerread): Collection, insights and nlp applications. In *Meeting of the North American
Chapter of the Association for Computational Linguistics (NAACL), New Orleans, USA.
Moshe Koppel, Jonathan Schler, and Shlomo Argamon. 2011. Authorship attribution in the wild. Language Resources and Evaluation, 45(1).
Eric Lantz, Kendrick Boyd, and David Page. 2015. Subsampled exponential mechanism: Differential privacy in large output spaces. In Proceedings of the 8th ACM Workshop on Artificial Intelligence and Security.
Ninghui Li, Wahbeh Qardaji, Dong Su, and Jianneng Cao. 2012. Privbasis: Frequent itemset mining with differential privacy. Proceedings of the VLDB Endowment, 5(11).
Andrew WE McDonald, Sadia Afroz, Aylin Caliskan, Ariel Stolerman, and Rachel Greenstadt. 2012. Use fewer instances of the letter "i": Toward writing style anonymization. In International Symposium on Privacy Enhancing Technologies Symposium. Springer.
Frank McSherry and Kunal Talwar. 2007. Mechanism design via differential privacy. In Proceedings of the 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS). IEEE.
Arvind Narayanan and Vitaly Shmatikov. 2008. Robust de-anonymization of large datasets (how to break anonymity of the netflix prize dataset). University of Texas at Austin.
NhatHai Phan, Yue Wang, Xintao Wu, and Dejing Dou. 2016. Differential privacy preservation for deep auto-encoders: an application of human behavior prediction. In Proceedings of the 30th AAAI Conference on Artificial Intelligence.
Alessandra Sala, Xiaohan Zhao, Christo Wilson, Haitao Zheng, and Ben Y Zhao. 2011. Sharing graphs using differentially private graph models. In Proceedings of the 2011 ACM SIGCOMM conference on Internet measurement conference. ACM.
Yunita Sari, Andreas Vlachos, and Mark Stevenson. 2017. Continuous n-gram representations for authorship attribution. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers.
Michael Schmid, Farkhund Iqbal, and Benjamin C. M. Fung. 2015. E-mail authorship attribution using customized associative classification. Digital Investigation (DIIN), 14(1).
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine translation via side constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems.
Rakshith Shetty, Bernt Schiele, and Mario Fritz. 2018. A4NT: author attribute anonymity by adversarial training of neural machine translation. In Proceedings of the 27th USENIX Conference on Security Symposium. USENIX Association.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems.
Veena Vasudevan and Ansamma John. 2014. A review on text sanitization. International Journal of Computer Applications, 95(25).
Benjamin Weggenmann and Florian Kerschbaum. 2018. Syntf: Synthetic and differentially private term frequency vectors for privacy-preserving text mining. In Proceedings of the 41st ACM International Conference on Research & Development in Information Retrieval (SIGIR). ACM.
Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W Cohen. 2018. Breaking the softmax bottleneck: A high-rank rnn language model. In International Conference on Learning Representations.
Rong Zheng, Jiexun Li, Hsinchun Chen, and Zan Huang. 2006. A framework for authorship identification of online messages: Writing-style features and classification techniques. Journal of the American Society for information science and technology, 57(3).
A Proof of Differentially Private Text Sampling.
Theorem 3. Differentially Private Text Sampling. Given a privacy budget  $\epsilon >0$ , a sequence length  $l$ $>0$ , the generator's sampling function in Eq.7 is  $(\epsilon +\ln (s))*l$ -differentially private.