SlowGuess/NAACL_2021_OCR huggingface dataset:
Mining, KDD'19, page 196-206, New York, NY, USA. Association for Computing Machinery.
Maximin Coavoux, Shashi Narayan, and Shay B. Cohen. 2018. Privacy-preserving neural representations of text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1-10. Association for Computational Linguistics.
Cynthia Dwork. 2006. Differential privacy. In Proceedings of the 33rd international conference on Automata, Languages and Programming, pages 1-12. Springer.
Tom Farrand, Fatemehsadat Mireshghallah, Sahib Singh, and Andrew Trask. 2020. Neither private nor fair: Impact of data imbalance on utility and fairness in differential privacy. In Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice, PPMLP'20, page 15-19, New York, NY, USA. Association for Computing Machinery.
GDPR Article 29 Working Party. 2014. Opinion 05/2014 on "anonymisation techniques". https://ec.europa.eu/justice/article-29/documentation/opinion-recommendation/files/2014/wp216_en.pdf.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural computation*, 9:1735-80.
Huseyin A. Inan, Osman Ramadan, Lukas Wutschitz, Daniel Jones, Victor Ruhle, James Withers, and Robert Sim. 2021. Training data leakage analysis in language models. arXiv preprint arXiv:2101.05405.
Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018. Towards robust and privacy-preserving text representations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 25-30, Melbourne, Australia. Association for Computational Linguistics.
H Brendan McMahan, Galen Andrew, Ulfar Erlingsson, Steve Chien, Ilya Mironov, Nicolas Papernot, and Peter Kairouz. 2018. A general approach to adding differential privacy to iterative training procedures. arXiv preprint, abs/1812.06210.
T. Mikolov, Martin Karafiát, Lukas Burget, J. Cernocký, and S. Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of the 11th Annual Conference of the International Speech Communication Association, pages 1045-1048.
Fatemehsadat Mirshghallah, Mohammadkazem Taram, Praneeth Vepakomma, Abhishek Singh, Ramesh Raskar, and Hadi Esmaeilzadeh. 2020. Privacy in deep learning: A survey. arXiv preprint arXiv:2004.12254.
Douglas Oard, William Webber, David Kirsch, and Sergey Golitsynski. 2015. Avocado research email collection. https://catalog.ldc.upenn.edu/LDC2015T03.
David Pujol, Ryan McKenna, Satya Kuppam, Michael Hay, Ashwin Machanavajjhala, and Gerome Miklau. 2020. Fair decision making using privacy-protected data. In Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, FAT* '20, page 189-199, New York, NY, USA. Association for Computing Machinery.
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. 2020. Training production language models without memorizing user data. arXiv preprint arXiv:2009.10031.
Nisarg Raval, Ashwin Machanavajjhala, and Jerry Pan. 2019. Olympus: sensor privacy through utility aware obfuscation. In Proceedings on Privacy Enhancing Technologies, pages 5-25. Sciendo.
R. Shokri, M. Stronati, C. Song, and V. Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE Symposium on Security and Privacy (SP), pages 3-18.
Congzheng Song and Vitaly Shmatikov. 2019. Auditing data provenance in text-generation models. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '19, page 196-206, New York, NY, USA. Association for Computing Machinery.
Pranav Subramani, Nicholas Vadivelu, and Gautam Kamath. 2020. Enabling fast differentially private sgd via just-in-time compilation and vectorization. arXiv preprint arXiv:2010.09063.
Florian Tramér and D. Boneh. 2020. Differentially private learning needs better features (or much more data). ArXiv, abs/2011.11660.
Michael Volske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. Tl;dr: Mining Reddit to learn automatic summarization. In Proceedings of the ACL 2017 Workshop on New Frontiers in Summarization, pages 59-63.
Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, and Vicente Ordonez. 2019. Balanced datasets are not enough: Estimating and mitigating gender bias in deep image representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5310-5319.
Xi Wu, Fengan Li, Arun Kumar, Kamalika Chaudhuri, Somesh Jha, and Jeffrey Naughton. 2017. Bolt-on differential privacy for scalable stochastic gradient descent-based analytics. In Proceedings of the 2017 ACM International Conference on Management of Data, SIGMOD '17, page 1307-1322, New York, NY, USA. Association for Computing Machinery.
S. Yeom, I. Giacomelli, M. Fredrikson, and S. Jha. 2018. Privacy risk in machine learning: Analyzing the connection to overfitting. In 2018 IEEE 31st Computer Security Foundations Symposium (CSF), pages 268-282.
A.1 Language Models
Language models assign a probability distribution over a sequence of words. A statistical model for a sequence of words  $x_{1} \ldots x_{n}$  can be represented by the product of the conditional probability of the next word given all the previous words as  $\operatorname{Pr}(x_{1} \ldots x_{n}) = \prod_{i=1}^{n} \operatorname{Pr}(x_{i} | x_{1} \ldots x_{i-1})$ . Here  $\operatorname{Pr}(x_{i} | x_{1} \ldots x_{i-1})$  denotes the probability of the occurrence of word  $x_{i}$  given the previous word sequence  $x_{1} \ldots x_{i-1}$ . Recurrent Neural Networks (RNNs) are widely used for this task (Bengio et al., 2003; Mikolov et al., 2010) as RNNs can process variable-length input by processing words one at a time, updating its internal state and predicting the next word sequentially. Therefore, such variable-length conditional distributions can be effectively estimated with RNNs. In this work we use LSTMs (Hochreiter and Schmidhuber, 1997) in our language model.
A.2 Experimental Setup
We use a subset of the Avocado dataset (Oard et al., 2015) with 100 users and 60,000 samples. We also use a subset of Reddit dataset (Volske et al., 2017) with 10,000 users and 3 million samples. For both datasets, we fix the vocabulary to the most frequent 40,000 tokens in the training corpus, and we create a  $80 / 20\%$  training/test set split. We use a two-layer LSTM model as the language model for the next-word prediction task. We set both the embedding dimension and LSTM hidden-representation size to 550 (with 55 million parameters). We use a feed-forward two fully connected layer neural network with hidden dimension of 1000 as the author discriminator for the adversarial training regularization scheme.
For optimization, we use the Adam optimizer with the learning rate set to 1e-3 and batch size to 100. We trained our models on a single Titan Xp GPU accompanied by 12GBs of RAM, and two Intel Xeon E5-2620 CPUs with 256 GBs of RAM. Table 1 shows the perplexity of the models used in the evaluations.
Figure 4: Per epoch training time break down, normalized to conventional execution. Differential privacy is  $16.44 \times$  slower than conventional execution. Triplet and Adversarial are our proposed regularizations.
Table 1: Training and test perplexities of the models used in the evaluations.
Dataset	Unmitigated	DP	Adversarial (ours)	Triplet (ours)
Avocado (High PPL)	Training	93.5	94.2	104.7	101.4
Test	103.5	93.5	96.6	96.5
Avocado (Low PPL)	Training	36.8	63.8	56.7	54.8
Test	51.3	69.8	69.1	69.1
Reddit	Training	107.5	110.3	106.7	106.4
Test	97.3	97.4	98.2	97.6
A.2.1 Batching Strategy
Our mitigations can be implemented using different batching strategies during training: uniform batches that contain training samples from different users, or per user batches that contain training samples from only a given user. Through experimentation, we observed that the second method performs better and we present our results under the second batching strategy. The better performance is due to the fact that grouping the same users together and increasing the probability of samples from same people being placed in corresponding positions in a batch, helps the discriminator (in the adversarial training scheme) learn user (author) patterns faster. It also helps the triplet-based scheme to more efficiently distance samples from the same user. This scheme we increases the probability that the auxiliary batch is selected from the same user's data, compared to randomly selecting two uniform batches. This helps better distribute one user's data in the embedding space.
A.3 Overhead and Complexity Analysis
A.3.1 Training Time Measurements
Here we compare the training time of our proposed regularizations to differential privacy. Figure 4 shows the breakdown of the CPU and GPU (CUDA) time for our two proposed methods, and differential privacy, normalized to the conventional unmitigated execution time. The results show that differential privacy is extremely slower than our mitigations. Our adversarial training mitigation is overall only  $1.06 \times$  slower than conventional
training, and our triplet-based mitigation is  $1.80 \times$  slower due to the need to process an auxiliary batch in each batch. The reason that our triplet-based mitigation is slower than the adversarial one is that the triplet based loss runs two batches through the language model during each iteration (the baseline batch and the auxiliary), whereas the adversarial training scheme runs only one batch of training data. Differential privacy, however, is  $16.44 \times$  slower than conventional execution, which is due to its per-sample gradient computation, which limits the possibility of parallelism. It is noteworthy that in our experiments, we have applied the training time optimization suggested in (McMahan et al., 2018) for differential privacy, which increases parallelism through the use of "micro-batches". However, even with this optimization, we are still observing huge slow-downs. Furthermore, differentially private training of DNNs and RNNs takes exhaustive hyperparameter tuning to get the best privacy-utility trade-off, which is cumbersome (Wu et al., 2017; Mirshghallah et al., 2020), and this slow training process makes the tuning of the parameters extremely harder.
A.4 Added Parameters and Complexity
Our adversarial training regularization includes an additional feed-forward discriminator DNN to predict the author of each sequence, as shown in Figure 1. In our experiments, we use a feedforward network with two fully connected layers and a hidden dimension of 1000. The input dimension is the same as the hidden state dimension of the language model (550) and the output dimension is the number of authors, 100 for Avocado, and 10,000 for Reddit dataset. This means that the discriminator parameters scale-up with the number of authors. For the Avocado case, this would give  $550 \times 1000 + 1000 * 100 = 650K$  extra parameters (compared to conventional training), and for Reddit it would be  $550 \times 1000 + 1000 \times 10,000 = 10.55M$  parameters. This is comparatively reasonable considering the number of parameters in the language model (55M) as a tradeoff to improve the privacy of the model. It would be interesting to explore whether reducing the number of hidden dimensions of the discriminator as the number of users increase would be effective to balance the number of parameters our *adversarial method* adds to the parameters of the network.
Our triplet-based regularization does not add
any extra parameters to the model, which might be advantageous for settings with massive number of users. It does, however, need to feed an auxiliary batch to the language model, alongside the baseline batch which almost doubles the training time.
Figure 4 and Section A.3.1 show the break-down of the training time of DP training, our triplet-based method, and our adversarial method, compared to unmitigated training time. Differential privacy does not add any additional complexity in terms of parameters. However, it adds computational complexity by having to compute the gradients of each sample separately, thereby not exploiting the batch processing parallelization offered by GPUs. This in turn slows down the training procedure extremely, making hyperparameter search infeasible for large networks.
A.5 Exposure Metric
To empirically compare the level of privacy provided by our methods to that of a DP model, we adopt the exposure metric introduced in (Carlini et al., 2019). This metric measures the extent to which a model memorizes samples in the training data. The higher the exposure metric for a sequence, the more the model's memorization and the easier it is to extract the sequence from the language model through text generation algorithms. To measure exposure we insert canaries to the training data. Our canaries are sequences of five random words from the vocabulary. We insert canaries with different repetitions for each person. For Avocado dataset, each user (author) is assigned 14 unique canaries, each of them repeated [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 20, 30, 40, 50] times. We insert canaries with repetition to mimic "secrets" that belong only to a certain user, but might be repeated by that user in multiple emails/texts. This means each user is assigned and overall of  $1 + 2 + 3 + \ldots + 50 = 195$  canaries. This yields and overall of  $195 \times 100 = 19500$  canaries injected to the data. This data is then used to train the model. Once the model is trained, we can then measure the exposure of a given canary, by finding the probability assigned to it by the language model, and then finding the rank of that sequence by by sorting all probabilities assigned to all possible sequences of the same length. Carlini et al. provide a method to estimate the rank for each sequence without having to actually enumerate and feed all possible sequences to the model and actu
ally ranking all of them. Once we have measured the exposure metric for all canary sequences, we can use them to evaluate the privacy.
For Reddit dataset, each user is assigned 5 unique canaries, with repetitions of [1, 2, 5, 6, 20] times. We use less canaries in the latter so as not to contaminate the dataset as the Reddit dataset has less samples per user. The "real canaries" introduced in the experiments are chosen by feeding all the training data through an unmitigated model, and selecting the sample with highest perplexity for each user. We hypothesize that these sequences are more likely to contain sensitive information.