SlowGuess/NAACL_2021_OCR huggingface dataset:
Fatemehsadat Mireshghallah $^{1*}$ , Huseyin A. Inan $^{3}$ , Marcello Hasegawa $^{2}$ , Victor Ruhle $^{2}$ , Taylor Berg-Kirkpatrick $^{1}$ , Robert Sim $^{3}$
{fatemeh, tberg}@ucsd.edu,
{huseyin.inan, marcellh, virueh, rsim}@microsoft.com
Neural language models are known to have a high capacity for memorization of training samples. This may have serious privacy implications when training models on user content such as email correspondence. Differential privacy (DP), a popular choice to train models with privacy guarantees, comes with significant costs in terms of utility degradation and disparate impact on subgroups of users. In this work, we introduce two privacy-preserving regularization methods for training language models that enable joint optimization of utility and privacy through (1) the use of a discriminator and (2) the inclusion of a novel triplet-loss term. We compare our methods with DP through extensive evaluation. We show the advantages of our regularizers with favorable utility-privacy trade-off, faster training with the ability to tap into existing optimization approaches, and ensuring uniform treatment of under-represented subgroups.
Neural language models (Bengio et al., 2003; Mikolov et al., 2010) have recently seen significant gains in capabilities, and are deployed at scale in several real-world scenarios (Chen et al., 2019; Adam et al., 2020). Training these models on domain-specific user data can further improve their utility. The volume of data required, coupled with the inherent sparsity of natural language which often means all data are unique, opens the door to an array of privacy attacks against models and their training data. Sample memorization poses a substantial risk by enabling model inversion attacks (Carlini et al., 2020; Ramaswamy et al., 2020; Inan et al., 2021). In these attacks, a curious or malevolent user can query a pre-trained language model on any data record with the intention of reconstructing (parts of) training samples<sup>1</sup>.
Differential Privacy (DP) (Dwork, 2006) is the gold standard approach to address this issue, thanks to its strong and rigorous privacy guarantees. DP-SGD (Abadi et al., 2016) is a popular method to train neural models with differential privacy guarantees and it works by clipping of the gradients and addition of noise in each update, which provides worst-case guarantees that reflect the likelihood of leaking any attribute of any member of the dataset into the trained model. The worst-case guarantees of differential privacy are not customizable, in other words, they cannot be relaxed to protect only certain attributes. Therefore, DP incurs significant loss to model utility (Tramér and Boneh, 2020). DP training of models is also much slower, with cumbersome hyper-parameter tuning and development (Wu et al., 2017; Subramani et al., 2020). It has also been shown that DP's utility loss is much worse for under-represented groups (Bagdasaryan et al., 2019; Farrand et al., 2020), which can have financial and societal ramifications (Pujol et al., 2020).
To address these issues, we relax the strong assumptions of the DP threat model and assume an adversary with finite-capacity (finite statistical, compute, and side information) who attempts to recover sensitive user-level information from the trained model (Carlini et al., 2019). We propose two privacy regularization methods, one based on adversarial training and another on a novel privacy loss term, to jointly optimize for privacy and utility of language models. The main idea of our regularizers is to prevent the last hidden state representation of the language model for an input sequence  $x$  from being linked back to the sensitive attribute we are trying to protect, in our case, the identity of the author. We use the last hidden state as it corresponds to the embedding of the sequence  $x$ .<sup>2</sup> We
* Work done as part of an MSR internship.
<sup>1</sup>A naïve example is an attacker querying "My account number is" and hoping to receive a user's account number.
2Although we consider recurrent neural network-based language models in this work, our approach is applicable in transformer-based language models as well. In the latter, one
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3799-3807 June 6-11, 2021. ©2021 Association for Computational Linguistics
Figure 1: Workflow of our adversarial training regularization. The last hidden state  $(h_x)$  of the LM is fed to the discriminator to generate a distribution over the authors  $(p_d)$ .  $p_d$  is used to compute  $\mathcal{L}_{\mathrm{LM - P}}$ , the privacy loss.
consider the linkability of the input representation to the sensitive attribute (author) as a proxy since it is commensurate with the linked and linkable information definitions in the General Data Protection Regulation (GDPR Article 29 Working Party, 2014). By framing privacy as an optimization problem, we can apply the well-developed machinery of large-scale gradient-based optimization, enabling us to train models at scale while jointly tuning for an optimal privacy-utility trade-off.
To validate our approach, we develop an evaluation framework for assessing a model's privacy loss. We employ the exposure metric introduced in (Carlini et al., 2019) and introduce a reconstruction (tab) attack as a realistic scenario to evaluate and compare LSTM language models trained using our regularization with those trained with differential privacy, on Avocado (Oard et al., 2015) and Reddit (Volske et al., 2017) datasets. We also empirically demonstrate that, unlike DP, our technique does not have disparate impacts on underrepresented groups.
Our work is closely related to (Coavoux et al., 2018) and (Li et al., 2018). Coavoux et al. consider an attacker who eavesdrops on the hidden representations of a pre-trained model during inference and tries to recover information about the input text. Adversarial training is used as a mitigation to reduce the attacker's performance (Wang et al., 2019). Li et al. use adversarial training to protect private author attributes such as age or gender, in learned text representations for part-of-speech tagging and sentiment analysis to gain better performance on out-of-domain corpora. We, on the other hand, use adversarial training and a triplet-based regularization to train private language models that do not memorize sensitive user information, which has not been explored before. We evaluate our models accordingly, by trying to extract training samples. Prior work has studied membership inference attacks against models (Shokri et al., 2017; Yeom
et al., 2018; Song and Shmatikov, 2019), however, our regularizations do not target these attacks.
In this section, we explain our proposed regularizers and training techniques in more detail.
Figure 1 shows our first proposed regularizer which is adversarial in nature. We feed an input text sequence  $x$  to the language model and extract the last hidden state representation of the model for  $x$ ; denoted by  $h_x$ .  $h_x$  is then fed to a discriminator parameterized by  $\theta_d$ , which plays the role of an attacker who attempts to predict what the sensitive label (in our case, the author,  $y$ ) for  $x$  is. The output probability distribution of the discriminator for the input  $h_x$ ,  $p_d = \operatorname{Pr}(\cdot | h_x; \theta_d)$  is then used to compute both the privacy loss  $\mathcal{L}_{\mathrm{LM-P}}$  of the language model and the discriminator loss  $\mathcal{L}_{\mathrm{D-CE}}$ . During training, the discriminator optimizes for better linking of the last hidden state representations to the authors. Thus, the discriminator loss is  $\mathcal{L}_{\mathrm{D-CE}}(h_x, y; \theta_d) = -\log \operatorname{Pr}(y | h_x; \theta_d)$ . Conversely, the language model optimizes  $\theta_{lm}$  such that it (1) improves the utility of the language model and (2) flattens the probability distribution over authors. Thus, we devise the following loss function:
$$
\mathcal {L} _ {\mathrm {L M}} (x; \theta_ {d}, \theta_ {l m}) = \mathcal {L} _ {\mathrm {L M - C E}} + \lambda \mathcal {L} _ {\mathrm {L M - P}} \tag {1}
$$
$\mathcal{L}_{\mathrm{LM - CE}}$  is the utility loss, for which we use conventional cross entropy loss over the next-word predictions.  $\mathcal{L}_{\mathrm{LM - P}}$  is the privacy loss:
$$
\mathcal {L} _ {\mathrm {L M - P}} \left(h _ {x}; \theta_ {d}\right) = - \frac {1}{M} \sum_ {c = 1} ^ {M} \log \Pr (c \mid h _ {x}; \theta_ {d}) \tag {2}
$$
i.e. the KL divergence between the distribution over authors and the uniform distribution where  $M$  is the number of classes (authors). The goal of this term is to drive the discriminator to predict randomly uniform outputs (Raval et al., 2019). The reason we devised this loss as opposed to using
can consider the representation corresponding to the special token [CLS] as the embedding of the sequence  $x$ .
(a) Avocado - High PPL ( $\sim 100$ )
(b) Avocado - Low PPL  $(\sim 60)$
Figure 2: Exposure metric results for different training schemes at similar perplexities. Unmitigated denotes conventional training. Adversarial and Triplet are our regularizers. Higher exposure indicates lower privacy.
(c) Reddit  $(\mathrm{PPL}\sim 100)$
$-\mathcal{L}_{\mathrm{D - CE}}$  is that we do not just want the discriminator to assign zero probability to the correct author, we want  $p_d$  to be uniform so that it has no information about the correct author. Hyperparameter  $\lambda$  allows for trading off privacy and utility.
2.2 Triplet-based Loss Function
One potential downside of the proposed adversarial regularizer is that the capacity of the discriminator must scale with the number of authors, and thus the size of the training data. To better accommodate the larger number of authors in large datasets, we investigate another regularizer that does not require a discriminator. We build on the intuition that to obfuscate an attribute, we can increase the distance between representations of samples that have the same label for that attribute while decreasing the distance between samples with different labels. To this end, we use the language model loss  $(\mathcal{L}_{\mathrm{LM}})$  of the previous section (Eq 1), and we set the privacy loss to be the triplet loss:
$$
\mathcal {L} _ {\mathrm {L M - P}} = \left\| h _ {x} - h _ {p} \right\| ^ {2} - \left\| h _ {x} - h _ {n} \right\| ^ {2} \tag {3}
$$
The triplet loss is commonly used in vision tasks for training embeddings that map images from the same category to neighboring points in the embedding space (Chechik et al., 2010). We, however, invert this loss and use it for an opposite purpose: privacy regularization. During the training of the language model, we select a "baseline sample",  $x$ , a positive sample  $p$  (with different sensitive label) and a negative sample  $n$  (with the same sensitive label) and feed them through the language model and extract the last hidden states  $h_x$ ,  $h_p$  and  $h_n$ , respectively. We find the  $l_2$  distance between  $h_x$ ,  $h_p$ , and  $h_n$  and based on their labels, add them to or subtract them from the loss. To implement this, in practice, we sample a baseline batch and a second "auxiliary" batch during training. We feed both the baseline batch  $(x)$  and the auxiliary batch  $(a)$  through the language model and extract the last
hidden states. We then calculate the distance between the last hidden states of the corresponding samples in the two batches. If the samples have different labels for the sensitive attribute (author), we add their distance to the loss, otherwise, we subtract it. The privacy loss becomes:
$$
\mathcal {L} _ {\mathrm {L M - P}} = \sum_ {i: y _ {x _ {i}} = y _ {a _ {i}}} \| h _ {x _ {i}} - h _ {a _ {i}} \| ^ {2} - \sum_ {j: y _ {x _ {j}} \neq y _ {a _ {j}}} \| h _ {x _ {j}} - h _ {a _ {j}} \| ^ {2} \tag {4}
$$
In our experiments, we use a subset of the Avocado corporate email dataset (Oard et al., 2015) with 100 users and 60,000 samples and a subset of Reddit dataset (Volske et al., 2017) with 10,000 users and 3 million samples. Both of these datasets are in English, covering formal and informal writing. We create a  $80 / 20\%$  training/test set split. We use a two-layer LSTM model as the language model for the next-word prediction task. We compare models trained with our proposed regularizer to differentially private (DP) ones (Abadi et al., 2016). For the privacy accounting, we use Gaussian differential privacy (Bu et al., 2019). We use language model perplexity as a measure of utility. Due to space limitations, we focus evaluations on privacy metrics for several set levels of achieved test perplexity, listed in Table 1 in the appendix. See appendix A.2 for a more detailed description of the experimental setup and extra analysis of overheads and complexity of each regularizer.
Privacy measurements w/ exposure metric. To empirically compare the privacy of our methods to that of DP, we adopt the exposure metric introduced in (Carlini et al., 2019). The higher the exposure of a sequence, the more the model's memorization and the easier it is to extract the sequence from the language model. To measure exposure we insert sequences of five random words (canaries) to the training data (appendix A.5). We insert unique
(b) Avocado - Low PPL  $(\sim 60)$
Figure 3: (a, b) Tab attack results for reconstructing canary sequences for two utility levels. Higher attack accuracy indicates lower privacy. (c) Effect of different mitigations on utility of well represented (Top-5) and underrepresented (Low-5) users for Avocado dataset.
(c) Impact on Different Subgroups
canaries with different repetitions for each user, and measure the exposure of these canaries.
Figure 2 shows the exposure results per canary repetition. These results are averaged over all the users. In each sub-figure, the perplexities of the models are similar, hence we can compare the privacy levels at similar utilities. Fig. 2a compares trained models using different techniques on the Avocado dataset, where they all have relatively high perplexities compared to a fully trained conventional model (Table 1). Fig. 2b has the same setup, however, the models have lower perplexities. Naturally, for having better utility we are trading off privacy, which can be seen by comparing the exposure values in these two figures and observing that the second one has higher exposure values (lower privacy). Finally, Fig. 2c shows the exposure results for Reddit.
In all cases, we see that the unmitigated model has the highest exposure, as expected. We also observe that for canaries (patterns) that are repeated more than 9 times (for each user), our mitigation offers lower exposure compared to DP, especially in the high perplexity case. This is because clipping and noise addition in DP is attribute and data-agnostic, meaning that noise is added to all samples regardless of whether or not they contain sensitive information. Therefore, repeated patterns are less protected. If we want to protect a pattern with  $n$  repetitions, we would need to apply noise that is  $n \times$  larger, which would degrade the utility gravely and would not yield the same perplexity. For lower repetition canaries, our mitigations have comparable performance to DP. For all these experiments the Gaussian differential privacy criterion  $\mu$  is extremely large  $(10^{20})$ , which practically yields  $\epsilon \sim \infty$ . We also experimented with lower  $\epsilon$  values (e.g.  $\epsilon \sim 7$ ), however, it yields a model with perplexity of 650, having an extremely low utility.
Privacy measurements w/ tab attack accuracy. In this experiment, we input the first token,
and see if the entire sequence is reconstructed using the language model. We report the rate of correct reconstruction of canaries as the accuracy of the attack. We use the synthetic canaries from the previous experiment, and also select "real canaries" from the training corpus to create a real-world scenario. Fig. 3a shows that for a high perplexity model, the accuracy of the tab attack on synthesized canaries is very small, even for the unmitigated model. The unmitigated model reaches the designated perplexity in less than an epoch, and hence it does not memorize the canaries. For the real canaries, however, the memorization is higher, since they follow grammatical rules. In the lower perplexity case of Fig. 3b, we see that the synthesized canaries are mostly memorized by the unmitigated model. Our mitigations outperform DP, especially for the synthesized canaries. DP is not context-sensitive and applies the same amount of noise to all samples, thereby leaving correlated and higher repeated samples less-protected. Our mitigations, however, learn what sequences are link-able to their authors, and obfuscate them such that they no longer leak the "identifying" secret.
Effect on under-represented users. Differential privacy has disparate impact on the accuracy of different subgroups of the dataset (Bagdasaryan et al., 2019). Here, we want to measure the effect of our mitigations on the utility of the model among users with various data samples. For each user, we measure the average perplexity of the model for their samples in the test set, and then subtract this from the same value for an unmitigated model. This would yield the average drop in utility, per user. We compare the utility drop of well-represented users to under-represented ones by taking the top 5 users with the most samples and the bottom 5 users with the fewest samples from Avocado dataset. We then measure the average utility drop over each group of 5 users on the test set. Figure 3c shows these results. We see that
differential privacy has disparate impact, 29 points, on the two sub-groups of users (authors), whereas this gap is only 7 points for models trained with our mitigations.
It's important to remember that in general, distinguishing "under-represented" users from those whose data is similar to others but who have contributed fewer samples is a difficult task. However, for Figure 3c's results, if these users' data came from the same distribution as the ones with lots of samples (i.e. if these people were merely less-contributing), the utility loss would be similar for all groups when applying "user-level" DP (what we use). DP's disparate impact on the utility loss for these two groups suggests that, in our case, the less-contributing authors are probably also underrepresented.
This work introduces two privacy mitigation methods to jointly optimize for privacy and utility. Extensive experiments show that our approach provides comparable and in certain cases a higher level of privacy compared to differentially private model training. We further empirically demonstrate, that our methods do not exhibit disparate impacts on under-represented groups and have significantly less overhead on training performance.
The Avocado corpus is licensed for research applications under strict terms intended to protect the privacy of the correspondents. While the end-user license agreement does not indicate what consent was granted by the participants, one term of the license is that "End user will obtain whatever training and approval is required by their organization for working with human subjects data", which we have obtained (more details in (Oard et al., 2015)). While handling sensitive email data (Avocado) we made sure to abide by the terms of its end-user license agreement (EULA) which has provisions to protect the privacy of members of the corpus. Furthermore, we took measures such as scrubbing named entities before using the data for model training. The over-arching goal of our work is to contribute to language model development that protects the privacy rights of users who contribute their data. While we rigorously evaluated our models by applying state-of-the-art attacks, deploying these models in real-world setups requires further verification that users' privacy is preserved.
The authors would like to thank the anonymous reviewers and meta-reviewers for their helpful feedback. We also thank Peter Kairouz and Mohammadkazem Taram for insightful discussions. Additionally, we thank our MSR colleagues and UCSD Berg Lab for their helpful comments and feedback.
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, pages 308-318.
Martin Adam, Michael Wessel, and Alexander Benlian. 2020. Ai-based chatbots in customer service and their effects on user compliance. *Electronic Markets*, pages 1-19.
Eugene Bagdasaryan, Omid Poursaeed, and Vitaly Shmatikov. 2019. Differential privacy has disparate impact on model accuracy. In Advances in Neural Information Processing Systems, volume 32, pages 15479-15488.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of machine learning research, 3:1137-1155.
Zhiqi Bu, Jinshuo Dong, Qi Long, and Weijie J. Su. 2019. Deep learning with gaussian differential privacy. arXiv preprint, abs/1911.11607.
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th USENIX Security Symposium, pages 267-284, Santa Clara, CA. USENIX Association.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2020. Extracting training data from large language models. arXiv preprint arXiv:2012.07805.
Gal Chechik, Varun Sharma, Uri Shalit, and Samy Bengio. 2010. Large scale online learning of image similarity through ranking. Journal of Machine Learning Research, pages 1109-1135.
Mia Xu Chen, Benjamin N Lee, Gagan Bansal, Yuan Cao, Shuyuan Zhang, Justin Lu, Jackie Tsay, Yinan Wang, Andrew M Dai, Zhifeng Chen, et al. 2019. Gmail smart compose: Real-time assisted writing. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data