SlowGuess/NAACL_2022_OCR huggingface dataset:
the model and thus the model is safer. Carlini et al. (2020) proposed an advance membership inference attack for language models. The basic idea is to calculate the given samples' perplexities under the model, rank them and choose the ones with the lowest perplexities (highest likelihood by the model). In our experiments, we randomly select 500 protected secrets from the training set, and randomly generate 500 samples of similar format, to form a dataset for the membership inference attack, so a random guess would give us an accuracy of  $50\%$ . For WikiText-2, the secrets are digit sequences; for CUSTOMERSIM, customer names are the secrets.
5.2 WikiText-2 Results
Model utility and privacy guarantee. The left part of Table 2 shows different models' utilities and privacy guarantees on WikiText-2 and Figure 3(a) shows the privacy-utility trade-off, where utility is represented by the validation perplexity (lower PPL=higher utility) and the privacy budget is represented by the epsilon (smaller  $\epsilon =$  more privacy). Although the definitions of  $\epsilon$  in DPSGD and S-DPSGD are different, the  $\epsilon$  in both cases provides a tight theoretical upper bound on how much the sensitive attributes are leaked through the trained models. Hence, the  $\epsilon$  associated with two algorithms are comparable. Given a fixed privacy budget  $\epsilon$ , we want to achieve a higher model utility. Because
(a) Privacy-Utility Trade-off
Figure 4: Privacy-utility trade-off, canary insertion attack and membership inference attack on CUSTOMERSIM.
(b) Canary Insertion
(c) Membership Inference
the privacy budget  $\epsilon = +\infty$  for the No-DP model, "No-DP" is represented by the vertical black line on the far right in Figure 3(a), and it achieves the best average PPL of 60.98 on the test set. The orange line is our S-DPSGD model, and it achieves the second-best average test PPL of 160.04 with  $\epsilon = 4.91$ . With a similar  $\epsilon = 4.89$ , DPSGD has the worst average test PPL of 305.86, much worse than S-DP because canonical DP notion protects the whole data and is over-pessimistic (see Section A.4 for models with different parameters). The gray line is for DPSGD with a smaller  $\sigma = 0.25$ , a convergent PPL of 266.6 and a final  $\epsilon = 132.73$ . Compared to DPSGD with  $\sigma = 0.25$ , it achieves a better PPL but with a much higher cost of privacy leakage (larger  $\epsilon$ ). But with S-DPSGD, we can also achieve lower PPL without hurting privacy.
Attack results. Figure 3(b) and 3(c) show the canonical insertion attack and membership inference attack results on WikiText-2. The x-axis is the models' utilities measured by validation PPL, and the y-axis is the exposure and membership inference accuracy indicating the success level of the attacks. Lower exposure and lower accuracy indicate a safer model. We want to see at a given robustness level to the attacks, which models can achieve lower perplexity, i.e., higher utility.
For canary insertion attack, although "No-DP" achieves lower perplexity, its exposure can go up to 20, indicating that the inserted canary could be easily revealed by the attackers. If we compare No-DP with S-DPSGD with similar utilities, S-DPSGD is always below No-DP, meaning S-DPSGD achieves much smaller exposure, and hence a safer model with similar utility. Comparing DPSGD and S-DPSGD, we find that S-DPSGD achieves much better model utility at a given exposure.
For membership inference attack, we draw a horizontal line of 0.5 to show the random guess perfor
mance. Again, S-DPSGD is always below No-DP, showing that with similar utilities, models trained with S-DPSGD are safer than No-DP under the membership inference attack. As mentioned earlier, DPSGD with  $\sigma = 0.5$  (green) and S-DPSGD (orange) have similar privacy budget ( $\epsilon = 4.89$  and 4.91 respectively). Comparing these two, we see that given a similar privacy budget, S-DPSGD converges to a much lower perplexity while remaining safe to the attack and thus achieves a wider range for the privacy-utility trade-off tuning. We also observe that for "No-DP", the inference accuracy can go up to  $90\%$ , suggesting that language models without DP protection are vulnerable to attacks.
5.3 CUSTOMERSIM Results
This section shows the results on CUSTOMERSIM. Model utility and privacy guarantee. The right part of Table 2 and Figure 4(a) show the privacy-utility trade-off for CUSTOMERSIM. Because the dialogs are simulated with templates and relatively simple, the perplexity can be as low as 3.06 for No-DP. S-DPSGD still achieves better perplexity than DPSGD (10.42 vs. 11.82) with similar  $\epsilon$ , but the gap is smaller compared to WikiText-2 because there are more sensitive tokens in CUSTOMERSIM (18.3%) than WikiText-2 (2.8%), and the advantage of protecting selective tokens only is not as big.
Attack results. Figure 4(b) and 4(c) show the results of the canary insertion and membership inference attack on CUSTOMERSIM respectively.
For canary insertion, we observe that given the same utility, S-DPSGD achieves lower exposure than No-DP and DPSGD. Although the improvement seems small on absolute values, we should note that exposure is on log-scale, so the improvement on relative rank is also on log-scale (e.g., for exposure=1.2, the rank of the canary is 429881; for exposure=0.2, the rank is 858215, roughly twice).
The membership inference accuracy is only a little better than random guess probability  $(60 + \%)$ , so it is not successful on CUSTOMERSIM. One potential reason could be that customer names only appear once in each dialog and can be very similar to each other (e.g., Emily, Emilya, and Emilyah). We leave it as future work to develop better membership inference attacks towards similar secrets. Under the failed attack, S-DPSGD is still better than No-DP. It is not feasible to compare S-DPSGD and DPSGD as both are close to random guess.
5.4 Data Anonymization and Selective-DP
Data anonymization (or data de-identification) has been widely used to protect data privacy, where the sensitive tokens are masked with special tokens such as "<num)". Both data anonymization and S-DP target protection towards sensitive attributes, and rely on a policy function to detect the private information. However, they are different in that data anonymization masks the sensitive attributes completely so nothing can be learned from them, while S-DP noises the private portion and provides a tunable way to adjust the privacy-utility trade-off, evidenced by experiments in Section 5.4.1.
One common problem for both methods is that the policy function is not guaranteed to be perfect and can miss to detect some sensitive information. So we also compare their performance when the policy function is imperfect in Section 5.4.2.
5.4.1 S-DP achieves better utility
Figure 5: Perplexity on private tokens over  $\epsilon$  for data anonymization and S-DPSGD.
We mask all the detected digits by "<num>" to train a data anonymization baseline on WikiText-2, calculate the perplexity on the private tokens, and present the result in Figure 5. The x-axis is the privacy budget  $\epsilon$ , and the y-axis is the private-token perplexity. For data anonymization, the dots are all
on the far right because  $\epsilon = +\infty$ . The first observation is that since data anonymization simply masks the sensitive tokens, it fails to learn anything about them, resulting in a much worse PPL on private tokens. This makes S-DP a good alternative to data anonymization because S-DP can still learn certain patterns from the noised sensitive tokens. Also, Figure 5 shows that for S-DP, there is a trade-off between  $\epsilon$  (privacy) and private-token PPL (utility), so we could tune  $\epsilon$  to achieve a better private-token PPL, while for data anonymization, there is no way to tune the parameters for better model utilities. More concretely, with proper  $\epsilon$  and  $\delta$ , S-DP might learn the structure of sensitive attributes (e.g., XXX-XX-XXXX for SSN) without knowing the exact value, or even learn the distribution of values for each digit, and such knowledge could be useful for data analytics. But for data anonymization, the model either sees the digit or doesn't see it, so there is no knob to tune the privacy-utility trade-off.
5.4.2 Imperfect Policy Function
Figure 6: Canary insertion attack for data anonymization and S-DPSGD when missing the canary.
Now we discuss the performance of both methods when the policy function is imperfect. We still use WikiText-2 with the secret "My ID is 341752". The policy function fails to detect "341752" as a secret. For data anonymization, all the detected digits are masked by "<num>"; for S-DP, we apply S-DPSGD to noise the detected digits.
Figure 6 shows the exposure of the missed secret 341752. When the perplexity gets lower, the model becomes better at remembering the details of the training data, so the exposure of both models becomes high. But when the perplexity is around the middle area, S-DP has lower exposure than data anonymization, meaning it's safer to attacks.
Note that the risk with imperfect policy functions is common to many privacy-preserving techniques, and how to build better policy functions is orthog-
onal to this work on S-DP, and thus beyond the scope of this paper. Improvements on policy functions (better sensitive information detection) are compatible with S-DP and can be used to further improve the results. We are also actively working on this topic in parallel to the S-DP work.
6 Preliminary Results on Contextual Policy Function and Transformer
Model	Policy function	Portion	ε	PPL
GPT2 + no DP	-	-	-	20.47
DPSGD	-	-	2.58	27.05
Redacted	Noncontextual:	16.40%	-	24.30
S-DP	All entities	2.58	22.56
Redacted	Contextual: entities, sub,	34.80%	-	38.66
S-DP	obj, propon, pron	2.48	25.61
Table 3: Preliminary results on different policy functions and Transformers.
In this section, we present the preliminary results on different policy functions and large Transformer models (Vaswani et al., 2017) on WikiText-2.
We design two policy functions: one is non-contextual and protects all the 18 named entities detected by spacy (Honnibal and Montani, 2017) such as person, date, locations, etc  $(16.4\%$  tokens) $^{2}$ ; the other one is contextual that protects all the entities plus subjects, objects, proper noun and pronouns of all the sentences  $(34.8\%$  tokens). We use these two policy functions to redact the WikiText-2  $D$  and obtain a redacted version  $D'$ . We first fine-tune a GPT2-small model (Radford et al., 2019) on  $D'$  (denoted as "redacted model"), and then further fine-tune this redacted model on the original  $D$  (Shi et al., 2022). The results are in Table 3. The redacted models trained on the redacted data  $D'$  achieve 24.30 and 38.66 in perplexity for the two policy functions respectively. If we fine-tune these two models on the original private data  $D$  with DPSGD, we can further improve the perplexity to 22.56 and 25.61, while the state-of-the-art DP language models only achieve 27.05 with similar privacy budget. These results show that our S-DP notion is promising in boosting utility of privacy-preserving language models even if one-third of the tokens are considered sensitive.
To conclude, we develop a new privacy notion, selective differential privacy (S-DP), to improve model utility while providing rigorous privacy guarantees for the sensitive portion of the data. We also develop a privacy mechanism, Selective-DPSGD, to achieve the new S-DP notion for RNNs. We experiment with WikiText-2 and a synthetic customer service dialog dataset. Results on both tasks show that models trained with S-DP achieve better utilities than traditional DP, and are more robust under various attacks than models without DP protection. With S-DP, we march one step closer towards safer and better language models and hope to inspire more related research. Moreover, S-DP could be applied to domains beyond NLP where only partial data require protection, such as image recognition. Please see Section A.1 for ethical consideration.
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308-318.
John M Abowd. 2018. The us census bureau adopts differential privacy. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2867-2867.
David Ifeoluwa Adelani, Ali Davody, Thomas Kleinbauer, and Dietrich Klakow. 2020. Privacy guarantees for de-identifying text transformations. INTER-SPEECH.
Borja Balle, Gilles Barthe, and Marco Gaboardi. 2018. Privacy amplification by subsampling: Tight analyses via couplings and divergences. Advances in Neural Information Processing Systems, 31.
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In 28th {USENIX} Security Symposium (\{USENIX} Security 19), pages 267-284.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. 2020. Extracting training data from large language models. arXiv preprint arXiv:2012.07805.
Jorge Cortés, Geir E Dullerud, Shuo Han, Jerome Le Ny, Sayan Mitra, and George J Pappas. 2016. Differential privacy in control and network systems. In
The full list of entities is available here https: //spacy.io/usage/linguistic-features# named-entities
2016 IEEE 55th Conference on Decision and Control (CDC), pages 4252-4272. IEEE.
Stelios Doudalis, Ios Kotsogiannis, Samuel Haney, Ashwin Machanavajjhala, and Sharad Mehrotra. 2017. One-sided differential privacy. Proceedings of the VLDB Endowment.
Cynthia Dwork, Aaron Roth, et al. 2014. The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211-407.
Hamid Ebadi, David Sands, and Gerardo Schneider. 2015. Differential privacy: Now it's getting personal. Acm Sigplan Notices, 50(1):69-81.
Natasha Fernandes, Mark Dras, and Annabelle McIver. 2019. Generalised differential privacy for text document processing. In International Conference on Principles of Security and Trust, pages 123-148. Springer, Cham.
Rajitha Hathurusinghe, Isar Nejadgholi, and Miodrag Bolic. 2021. A privacy-preserving approach to extraction of personal information through automatic annotation and federated learning. arXiv preprint arXiv:2105.09198.
Matthew Honnibal and Ines Montani. 2017. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Zach Jorgensen, Ting Yu, and Graham Cormode. 2015. Conservative or liberal? personalized differential privacy. In 2015 IEEE 31St international conference on data engineering, pages 1023-1034. IEEE.
Gavin Kerrigan, Dylan Slack, and Jens Tuyls. 2020. Differentially private language models benefit from public pre-training. In Proceedings of the Second Workshop on Privacy in NLP, pages 39-45.
Jaewoo Lee and Daniel Kifer. 2020. Differentially private deep learning with direct feedback alignment. arXiv preprint arXiv:2010.03701.
H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2018. Learning differentially private recurrent language models. In International Conference on Learning Representations.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. *ICLR*.
Ilya Mironov. 2017. Rényi differential privacy. In 2017 IEEE 30th Computer Security Foundations Symposium (CSF), pages 263-275. IEEE.
Nicolas Papernot, Shuang Song, Ilya Mironov, Ananth Raghunathan, Kunal Talwar, and Ülfar Erlingsson. 2018. Scalable private learning with pate. *ICLR*.
Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, and Ülfar Erlingsson. 2020. Tempered sigmoid activations for deep learning with differential privacy. CoRR, abs/2007.14191.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Swaroop Ramaswamy, Om Thakkar, Rajiv Mathews, Galen Andrew, H Brendan McMahan, and Françoise Beaufays. 2020. Training production language models without memorizing user data. arXiv preprint arXiv:2009.10031.
Taisho Sasada, Masataka Kawai, Yuzo Taenaka, Doudou Fall, and Youki Kadobayashi. 2021. Differentially-private text generation via text preprocessing to reduce utility loss. In 2021 International Conference on Artificial Intelligence in Information and Communication (ICAIIC), pages 042-047. IEEE.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. ACL.
Weiyan Shi, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. 2022. Just fine-tune twice: Selective differential privacy for large language models. arXiv preprint arXiv:2204.07667.
Om Thakkar, Galen Andrew, and H Brendan McMahan. 2019. Differentially private learning with adaptive clipping. arXiv preprint arXiv:1905.03871.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. NIPS.
Bao Wang, Quanquan Gu, March Boedihardjo, Farzin Barekat, and Stanley J. Osher. 2019. DP-LSSGD: A stochastic optimization method to lift the utility in privacy-preserving ERM. CoRR, abs/1906.12056.
Wenxiao Wang, Tianhao Wang, Lun Wang, Nanqing Luo, Pan Zhou, Dawn Song, and Ruoxi Jia. 2021. Dplus: Boosting utility of differentially private deep learning via randomized smoothing. arXiv preprint arXiv:2103.01496.
Zekun Xu, Abhinav Aggarwal, Oluwaseyi Feyisetan, and Nathanael Teissier. 2020. A differentially private text perturbation method using regularized mahalanobis metric. In Proceedings of the Second Workshop on Privacy in NLP, pages 7-17.
Tiancheng Zhao and Maxine Eskenazi. 2018. Zero-shot dialog generation with cross-domain latent actions. SIGDIAL.
A.1 Ethical Consideration
Data Usage. To minimize real-world harm, we choose WikiText-2 since it is already public and widely used, and synthesize the dialogs as well as the personal information in the CUSTOMERSIM datasets in our experiments. For future research, we plan to continue using public or synthetic datasets to prevent real-world data leakage.
Application. Our work addresses the problem of data privacy protection and can be applied in different applications. The attacks used in our study are well-known standard attacks tailored for our specific tasks, so it's hard to generalize and misuse them to attack other language models. We will release the code so that people can have access to the various algorithms and protect their own data.
There are many spaces for improvements for this S-DP work. For instance, when the policy function fails to detect sensitive attributes, their privacy may not be guaranteed, and therefore, we plan to develop better policy functions and employ privacy amplification (Balle et al., 2018). Also, besides explicit private information like names, we plan to protect sensitive context such as "I have two kids", as these can often happen causally in dialogs but still reveal personal status. In Section 6 we show some preliminary results on protecting contexts and plan to further refine the contextual policy functions.
A.3 Privacy Analysis
We analyze the private guarantees of Selective-DPSGD in this section.
For any given dataset  $D$ , let  $D_{i,j}$  denote the  $j$ th attribute of the  $i$ -th record. We abstract the gradient update and hidden state into a query function  $f(x, w)$  which takes training data  $x$  and auxiliary information  $w$  as input. We introduce  $w$  as an additional input to  $f$  to model the dependence of the gradient update and hidden state on the model parameters at the previous rounds. We define the following two types of queries on the dataset.
- Type-1 query: the input  $x$  to  $f$  consists of only private attributes with respect to the policy function  $F$
- Type-2 query: the input  $x$  to  $f$  consists of only non-private attributes with respect to the
policy function  $F$
Since S-DP only hides the presence of private attributes, type-2 query does not incur privacy loss.
The following theorem shows that if a type-1 query has the property that its output is bounded, then for arbitrary auxiliary input, adding Gaussian noise into the query can provide DP. The reason why we consider such queries is that clipped gradient and hidden state can be modeled as such queries. The reason for which we want to analyze DP guarantees under arbitrary auxiliary inputs is that at any given round, the model parameters resulting from previous rounds could be arbitrarily different. This is because the non-sensitive part of two  $F$ -neighboring datasets could be arbitrarily different.
Theorem 1. Assume that  $\max_{x,w}\| g(x,w)\| \leq C$ . Then, for any arbitrary  $w$ , adding Gaussian noise  $\Delta = \mathcal{N}(0,\sigma^2)$  proportional to  $C$  into  $g$  can ensure  $(\epsilon ,\delta)$ -DP where  $\epsilon ,\delta$  depends on  $C$  and  $\sigma$ . More formally, for all neighboring datasets  $x$  and  $x^{\prime}$  and all  $w,w^{\prime}$ ,
$$
\frac {P [ g (x , w) + \Delta = r ]}{P [ g (x ^ {\prime} , w ^ {\prime}) + \Delta = r ]} \leq e ^ {\epsilon} \quad w. p. 1 - \delta \tag {6}
$$
The proof follows directly from the classic proof for DP guarantees of the Gaussian mechanism (Mironov, 2017; Dwork et al., 2014) by noticing the sensitivity of  $f$  is bounded by  $C$ .
The regular updates in Algorithm 1 take as input non-private data  $B_{np,i}$ . Hence, they are type-2 queries and do not incur extra privacy loss. The private updates in Algorithm 1 (i.e., gradient and hidden states) depend on private attributes and model parameters from previous rounds, and thus belong to the type-1 query. Moreover, they satisfy the bounded norm assumption in Theorem 1. We call the resulting query of adding Gaussian noise into a type-1 query with bounded norm property a noisy type-1 query. Overall, Algorithm 1 is essentially the composition of multiple type-1 and noisy type-2 queries. In the following, we will present a general result for the privacy guarantees resulting from the composition.
Theorem 2. Let  $f$  be the composition of  $k$  queries:  $f_{1},\ldots ,f_{k}$ , which are either noisy type-1 or type-2 queries. Given a policy function  $F$ , let  $\mathbf{f}_p$  denote the set of noisy type-1 queries. Let  $\mathbf{f}_{np}$  denote the set of type-2 queries. Then, if  $\mathbf{f}_p$  is  $(\epsilon ,\delta)$ -DP,  $f$  is  $(F,\epsilon ,\delta)$ -S-DP.
Proof. Consider two selective  $F$ -neighboring datasets  $x$  and  $x'$ . Let  $x_i$  and  $x_i'$  be the subset of data utilized by  $f_i$ .  $x_i$  contains only private attributes when  $f_i$  is type-1 and contains only non-private attributes when  $f_i$  is noisy type-2. By the neighboring relation between  $x$  and  $x'$ ,  $x_i$  and  $x_i'$  are also selective  $F$ -neighbors when  $f_i$  is a type-1 query. In addition to the dataset,  $f_i$  takes the output of all the previous queries. For a fixed outcome  $(y_1, \ldots, y_k)$  of  $f$ , we have
$$
\begin{array}{l} \begin{array}{l} P \left[ f _ {1} \left(x _ {1}, w _ {1}\right) = y _ {1}, \dots , f _ {k} \left(x _ {k}, w _ {k}\right) = y _ {k} \right] \\ \hline P \left[ f _ {1} \left(x _ {1} ^ {\prime}, w _ {1} ^ {\prime}\right) = y _ {1}, \dots , f _ {k} \left(x _ {k} ^ {\prime}, w _ {k} ^ {\prime}\right) = y _ {k} \right] \end{array} (7) \\ = \prod_ {f _ {i} \in \mathbf {f} _ {p}} \frac {f _ {i} \left(x _ {i} , w _ {i}\right)}{f _ {i} \left(x _ {i} ^ {\prime} , w _ {i} ^ {\prime}\right)} (8) \\ \leq e ^ {\epsilon} \quad \mathrm {w . p .} 1 - \delta (9) \\ \end{array}
$$
The equality in the second line is due to the fact that  $\mathbf{f}_{np}$  does not incur privacy loss and the independence of randomness of each query given the output of all the previous queries. The inequality in the third line is due to the assumption that  $\mathbf{f}_p$  is  $(\epsilon, \delta)$ -DP with arbitrary auxiliary input.