SlowGuess/NAACL_2022_OCR huggingface dataset:
Weiyan Shi $^{1}$ , Aiqi Cui $^{1}$ , Evan Li $^{1}$ , Ruoxi Jia $^{2}$  Zhou Yu $^{1}$
Columbia University<sup>1</sup>, Virginia Tech<sup>2</sup>
{ws2634, ac4788, el3078} @columbia.edu, ruoxijia@vt.edu, , zy2461@columbia.edu
With the increasing applications of language models, it has become crucial to protect these models from leaking private information. Previous work has attempted to tackle this challenge by training RNN-based language models with differential privacy guarantees. However, applying classical differential privacy to language models leads to poor model performance as the underlying privacy notion is over-pessimistic and provides undifferentiated protection for all tokens in the data. Given that the private information in natural language is sparse (for example, the bulk of an email might not carry personally identifiable information), we propose a new privacy notion, selective differential privacy, to provide rigorous privacy guarantees on the sensitive portion of the data to improve model utility. To realize such a new notion, we develop a corresponding privacy mechanism, Selective-DPSGD, for RNN-based language models. Besides language modeling, we also apply the method to a more concrete application - dialog systems. Experiments on both language modeling and dialog system building show that the proposed privacy-preserving mechanism achieves better utilities while remaining safe under various privacy attacks compared to the baselines. The data and code are released to facilitate future research<sup>1</sup>.
Language models have been widely used in various kinds of applications, such as Google SmartCompose, Amazon Alexa, and so on. However, these models are often trained on highly sensitive data such as emails and chat logs, while having the tendency to memorize the training data unintentionally (Carlini et al., 2019, 2020). Therefore, how to protect user privacy while preserving the model utility has become an increasingly important topic.
Several methods have been developed to protect the data, such as data anonymization,  $k$ -anonymity,
Figure 1: "Data anonymization" and training with "No DP" cannot provide knobs to adjust the trade-off between privacy and utility. Selective-DP improves the privacy-utility trade-off of traditional DP, to get closer to the ideal model with both high privacy and high utility.
and differential privacy (DP). Among them, DP has become a dominant privacy notion as it provides formal and provable guarantees for people to understand the privacy-utility trade-off (Figure 1). It works by carefully randomizing the algorithm so that the model does not rely too much on any single data point. However, traditional DP notion protects each data point as a whole regardless of the property of individual attributes inside the data (McMahan et al., 2018). Training large models with this overly pessimistic privacy notion could lead to poor model performance or even a non-convergent training process (Kerrigan et al., 2020). Our work is inspired by an important observation that in many scenarios including language modeling, private information is sparse, and not all attributes need to be protected. For example, for the sentence "My SSN is 123-45-6789", only the last token with the actual SSN number needs to be protected. But if we protect the entire sentence, we may fail to learn the underlying language pattern well.
To solve this problem, we propose a new DP notion, namely selective differential privacy (S-
https://github.com/wyshi/lm PRIVacy
Human Language Technologies, pages 2848 - 2859
DP), to provide focused protection for sensitive attributes in a training record and improve model utility. We follow the traditional DP setup, where each training record is contributed by a different user. The key difference is that we consider only partial dimensions of a training record as sensitive, which better abstracts the language data privacy problems. Whether a given dimension in a record is sensitive is specified by a user-defined policy function that encodes application-specific privacy regulations, which gives users the freedom to protect any type of sensitive information according to the use cases. We also develop a corresponding privacy mechanism, Selective-DPSGD, for RNN-based language models under the new notion. Moreover, to process the variable-length sequences, we propose a batching method to group private and public tokens together and alternatively update them for a more efficient implementation of Selective-DPSGD.
One important concept to note is that, as shown in Figure 1, there is always a trade-off between privacy and utility: data anonymization achieves high privacy guarantee at the cost of low utility on private tokens; models trained without DP have high utility but low privacy. However, for both data anonymization and "no DP", there is no way to tune the trade-off, while DP-related methods provide knobs to adjust the privacy-utility trade-off. This paper proposes S-DP to improve the trade-off of canonical DP to get closer to the ideal model with both high-utility and high-privacy.
We evaluate S-DP on two tasks, 1) a language generation task and 2) a more concrete application of dialog systems. Besides reporting the model utility and theoretical privacy guarantees, we also empirically demonstrate their robustness under popular privacy attacks on language data (Carlini et al., 2019, 2020). The experiments suggest that training with Selective-DPSGD improves the model utility while remaining safe to the attacks.
Our contributions are as follows. First, we propose a new selective differential privacy notion that ensures targeted privacy protection for sensitive attributes, and a corresponding mechanism to realize the new S-DP notion for RNN-based models. Second, we propose a dialog dataset for future privacy research. Next, we show both theoretically and practically that our models are safe to attacks with improved utilities on both the language generation task and the dialog system application. Moreover, we discuss the case of imperfect policy func
tion, and compare S-DP with data anonymization to show that S-DP achieves better utilities when the policy function is imperfect. We also show preliminary results on contextual policy functions and Transformer models. In the era of information explosion and large-scale pretraining, protecting data privacy becomes more and more important. With S-DP, we march one step closer towards more privacy-preserving and high-performing language models and hope to inspire more research in this direction in the NLP community. Moreover, despite our focus on language-related applications in this paper, the proposed S-DP notion could be useful for a much broader range of applications where only partial data require privacy protection.
Language modeling is a key research problem in NLP. However, although language models are often trained on sensitive data such as emails, most related studies focus on improving model without considering privacy, leaving the models vulnerable to attacks. For example, Carlini et al. (2019) showed that it is possible to infer a secret in the training data by attacking published language models. Therefore, it is of great importance to introduce privacy protection mechanisms to the NLP community and train the models in a much safer way.
Differential privacy (DP) (Dwork et al., 2014) has been applied to various domains (Cortés et al., 2016; Abowd, 2018). Abadi et al. (2016) introduced DPSGD to train DP-protected deep-learning models. PATE (Papernot et al., 2018) leveraged knowledge distillation to train differentially private models. But DP-protected algorithms suffer from low utility, so the DP notion often needs to be adjusted according to the applications: Ebadi et al. (2015) proposed a personalized DP notion to provide different levels of protection for different users; Doudalis et al. (2017) developed one-sided DP to protect sensitive users only. We also propose a new Selective-DP notion to protect only the sensitive attributes in a record to improve utility.
Recently, DP has also been applied to NLP tasks (Fernandes et al., 2019; Xu et al., 2020; Hathurusinghe et al., 2021; Sasada et al., 2021). For example, McMahan et al. (2018) proposed DP-FedAvg and DP-FedSGD to train RNN language models with user-level privacy guarantees. Adelani et al. (2020) developed a probabilistic text deidentification algorithm with formal privacy guar
antees. Different from existing work that directly applied DPSGD and provided undifferentiated protection for all training examples, we propose a new privacy notion and a corresponding mechanism to protect the sensitive portion of the data in centralized learning. Such a new notion can be easily adapted to federated learning settings as well.
There is also a line of work that attempts to improve differentially private deep learning performance via modifying training process (Wang et al., 2021, 2019; Thakkar et al., 2019; Lee and Kifer, 2020) or model architecture (Papernot et al., 2020). Our work is complementary to this line of work as we propose a new privacy notion. Particularly, our work can be combined with the aforementioned methods to further improve model utility.
We will introduce language modeling and differential privacy as preliminaries in this section.
Language Modeling. Consider a text sequence that consists of multiple tokens, i.e.,  $\pmb{x} = (x_{1}, x_{2}, \dots, x_{n})$ , where  $x_{i}$  is the  $i$ -th token. The goal of language modeling is to learn the probability of the sequence  $p(\pmb{x})$ , which can be factorized with the chain rule as in Equation (1). Given a corpus  $D = \{\pmb{x}^{1}, \dots, \pmb{x}^{|D|}\}$ , we train a neural network (e.g., RNN) parameterized by  $\theta$  to learn  $p(\pmb{x})$  by minimizing the negative log-likelihood over  $D$  with the loss function in Equation (2).
$$
p (\boldsymbol {x}) = \prod_ {i = 1} ^ {n} p \left(x _ {i} \mid \boldsymbol {x} _ {<   i}\right), \tag {1}
$$
$$
\mathcal {L} (D) = - \sum_ {t = 1} ^ {| D |} \sum_ {i = 1} ^ {n _ {t}} \log p _ {\theta} \left(x _ {i} ^ {t} \mid \boldsymbol {x} _ {<   i} ^ {t}\right) \tag {2}
$$
Differential Privacy (DP) (Dwork et al., 2014). A differentially private algorithm ensures that its output cannot help much to distinguish whether an individual record is contained in the input dataset. In other words, DP hides the presence of individual records. The formal definition is as follows.
Definition 1. (Differential Privacy). Given a domain  $\mathcal{D}$ , any two datasets  $D, D' \subseteq \mathcal{D}$  that differ in exactly one record are called neighboring datasets. A randomized algorithm  $\mathcal{M}: \mathcal{D} \to \mathcal{R}$  is  $(\epsilon, \delta)$ -differential private if for all neighboring datasets  $D$  and  $D'$  and all  $T \subseteq \mathcal{R}$ ,
$$
\Pr [ \mathcal {M} (\mathcal {D}) \subseteq T ] \leq e ^ {\epsilon} \Pr [ \mathcal {M} (\mathcal {D} ^ {\prime}) \subseteq T ] + \delta .
$$
4 Selective Differential Privacy
Canonical DP notion treats all records as sensitive. Prior work has studied variants of DP notions, such as personalized DP (Jorgensen et al., 2015) and onesided DP (Doudalis et al., 2017), to exploit different privacy levels between records. However, existing privacy notions do not allow different attributes in a given record to have different privacy levels, which could otherwise potentially enable additional utility gains, especially for NLP tasks where private attributes are sparse. Hence, we propose a new privacy notion—selective differential privacy—to distinguish between private and non-private attributes inside one data point with a policy function and protect sensitive part of one data point.
Definition 2. (Policy Function). A policy function  $F: \tau \rightarrow \{0,1\}^{n_r}$  denotes which attributes of a record  $r \in \tau$  are sensitive ( $F(r)_i = 0$ ) or non-sensitive ( $F(r)_i = 1$ ), where  $n_r$  is the number of attributes in  $r$ . Note that  $n_r$  depends on the record and is not a fixed number.
In practice, users have the freedom to define the policy function to encode specific privacy regulation and protect any sensitive attributes according to the applications. The protected sensitive attribute types are unlimited, can be entities (e.g., name, emails, etc), contextual (e.g., health-related information, speaking style, etc), and so on. For example, users can design a conservative policy function that protects selected complete sentences if necessary. The form of the policy function is also unlimited, and could be neural networks, regex, and so on. Please refer to Section 6 for contextual policy functions.
In the case of language modeling, each record is a text sequence  $\pmb{x}$ , each attribute is a token  $x_{i}$  in  $\pmb{x}$  and  $F(\pmb{x})$  is a bit vector indicating which tokens contain private information. We define neighboring datasets under our new privacy notion as follows.
Definition 3. (F-Neighbors).  $D, D'$  are two datasets and  $F$  is a policy function.  $D'$  is a F-neighbor of  $D$  if and only if  $\exists r \in D$  s.t.,  $F(r)$  contains at least one private attribute,  $\exists r' \in D'$  s.t.,  $F(r)$  and  $F(r')$  differ by at least one private attribute, and  $D' = D \setminus \{r\} \cup \{r'\}$ . We denote  $D' \in N_F(D)$ .
Under this definition, the dataset containing "My ID is 123" and the dataset containing "My ID is 456" are neighbors; but the dataset with "Hello
there” and the dataset with “Hi there” are not neighbors since they do not contain private information.
Definition 4. (Selective Differential Privacy). Given a policy function  $F$ , a randomized algorithm  $\mathcal{M}:\mathcal{D}\to \mathcal{R}$  satisfies  $(F,\epsilon ,\delta)$ -selective differential privacy if for  $\forall D,D^{\prime}\in N_{F}(D)$ , and  $\forall T\subseteq \mathcal{R}$
$$
\Pr [ \mathcal {M} (D) \subseteq T ] \leq e ^ {\epsilon} \Pr [ \mathcal {M} (D ^ {\prime}) \subseteq T ] + \delta .
$$
Essentially, S-DP also provides an indistinguishability property similar to canonical DP, but only for the sensitive attributes in a record. S-DP does not constrain the information leakage of non-sensitive attributes as long as the privacy of the sensitive attributes is preserved. Thus, S-DP protects privacy for sensitive attributes in the worst case (i.e., the attacker may have knowledge about everything except the targeted sensitive attribute.)
4.1 Selective Privacy Mechanism
With the new S-DP notion, the next step is to develop a corresponding privacy mechanism to train models that realize the new notion. Privacy mechanisms usually work by adding noise to the models to protect the data, such as Laplace mechanism (Laplace noise) and Gaussian mechanism (Gaussian noise). Abadi et al. (2016) proposed DPSGD that adds Gaussian noise to the gradients and applies stochastic gradient descent (SGD) to train private deep learning models. In this work, we develop Selective-DPSGD, shown in Figure 2 and Algorithm 1, to train RNN-based language models that achieve S-DP. The basic idea is to first determine the private attributes with the policy function, then decide which model variables are related to the private attributes, and finally apply regular SGD on non-private variables, and DPSGD (Abadi et al., 2016) on the private variables. We choose RNNs because they are widely used in industry, e.g., Ramaswamy et al. (2020) discussed how to train private production language models with RNNs.
We need to first decide the variables related to the private tokens. RNN uses a hidden state  $h_i$  to encode the context, and outputs a distribution  $p_i$  over a vocabulary set  $V$ , as shown in Equation (3). If  $x_i$  is private, then  $h_i$ ,  $p_i$ , and  $\mathcal{L}_i$  are all private; besides, to calculate  $\mathcal{L}_{i-1}$ , we need to access the ground truth next token  $x_i$ , so  $\mathcal{L}_{i-1}$  is also private. The private variables are all in red in Figure 2.
$$
\boldsymbol {h} _ {i} = R N N \left(\boldsymbol {h} _ {i - 1}, x _ {i}\right) \tag {3}
$$
$$
\boldsymbol {p} _ {i} = p _ {\theta} (V \mid \boldsymbol {x} _ {<   i}) = S o f t m a x (g (\boldsymbol {h} _ {i})) \tag {4}
$$
$$
\mathcal {L} _ {i} = - \log p _ {\theta} \left(x _ {i + 1} \mid \boldsymbol {x} _ {<   i + 1}\right) \tag {5}
$$
Figure 2: All private variables are in red. We apply regular SGD on non-private variables and DPSGD on private variables in Selective-DPSGD.
Algorithm 1 outlines the steps in Selective-DPSGD. Given a dataset  $D$ , we apply a policy function  $F$  to obtain a bit matrix  $P = F(D)$  that indicates which tokens are private. At each step, we take a random batch  $B$ , and use  $P$  to split  $B$  into a sequence of non-private and private tuples  $\{(B_{np,i}, B_{p,i})\}$ ; then we apply SGD (regular update) on  $B_{np,i}$  and DPSGD (private update) on  $B_{p,i}$  alternatively, to update and protect privacy. Note that besides noise in the gradients, we also clip and add noise to the hidden state  $h_i$  if it is private. The reason is that in RNN, if  $x_i$  is private,  $h_i$  also contains private information (as shown above), and is directly passed to the next regular update step and cannot be protected by the noise in the gradients. So it is important to add noise to protect the private information in  $h_i$ . Since DPSGD adds noise to the gradients,  $\mathcal{L}$  and  $p_i$  used to calculate the loss are protected by the noise in the gradients. In this way, all private variables are protected.
Privacy Guarantee. In Section A.3, we prove that the composition of the series of noise-adding operations ensures S-DP for Selective-DPSGD.
We conduct our experiments on two datasets: 1) a traditional text corpus for language modeling, and 2) a dialog dataset for a more concrete application of dialog systems. Below are the dataset details.
WikiText-2. To minimize real-world harm, we choose the already-public WikiText-2 (Merit et al., 2017). It contains articles from Wikipedia with potentially sensitive information, and is a classical dataset for language modeling. For simplicity, we treat all the digits as privacy information. So the policy function  $F$  is a digit detector: if the token is a digit,  $F$  will output 0, otherwise, 1.
CUSTOMERSIM. With the emergence of virtual assistants, more and more private information is being exchanged during daily interactions. So we
also apply S-DP to build dialog systems in the customer service domain. Using real dialog transcripts may lead to real-world harm, so we simulate a dialog dataset, CUSTOMERSIM, with synthetic user information. The dialogs are simulated with fixed agendas and template utterances (Zhao and Eskenazi, 2018). We treat user name, address, phone number, order, and tracking number as sensitive information, and use regex to build a policy function to detect them. Table 1 shows one example dialog.
Note that although we use digits and names as running examples for sensitive information, S-DP can protect any sensitive attributes specified by the policy function. Building better policy functions is orthogonal to S-DP, and thus beyond the scope of this paper. Any improvements on policy functions are compatible with S-DP to achieve better results.
Model training details. We use one-layer LSTMs with an embedding size of 200 and a hidden size of 200, and a BPE tokenizer (Sennrich et al., 2016) to avoid information leakage from the tokenizer: with BPE, a secret "1234" will be split into multiple tokens, e.g., "12" and "34", while traditional tokenizer will release "1234" in the dictionary. All
CUSTOMERSIM
Role	Utterance
SYS	Hello, I am the customer support bot. What can I do for you?
USR	Hello robot. Could you please help me track my package?
SYS	Please provide your full name.
USR	Sure, Betty Sims.
SYS	Could you please confirm your shipping address?
USR	Yea sure, 2241 Fitzgerald Viaduct Brownview, OK 28304.
SYS	Track your order using your tracking number, FH6F6GMMF4. Are you happy about my answer?
USR	That's it.
Table 1: An example dialog in CUSTOMERSIM.
private states (hidden, cell states) in the LSTMs are protected.
Baselines. We have two baselines, one without DP ("No-DP"), and the other trained with DPSGD ("DPSGD"). We refer to our models trained with S-DPSGD as "S-DPSGD". "No-DP" is simply an LSTM optimized with a regular SGD and a starting learning rate (lr) of 20. The learning rate was annealed and decreased as training proceeded. "DPSGD" is optimized with DPSGD and a starting learning rate of 0.05. All the models are trained five times to reduce randomness, and the parameters are tuned on the validation set. We compare with DPSGD because it's the backbone of most of existing DP learning algorithms. Existing modifications of DPSGD are mainly focused on optimization algorithms and objectives, thus are compatible with our work that tailors the privacy notions to realistic privacy needs in the NLP context.
5.1 Evaluation
We evaluate both the language models' utilities and privacy protection levels. We use perplexity (PPL) and the top-1 next word prediction accuracy (AccT1) to measure model utility. To measure privacy protection levels, besides reporting the theoretical privacy budget  $\epsilon$  and  $\delta$ , we also perform various practical attacks on the trained models and report how successful the attacks are against different techniques. We compare the performance of our proposed privacy-preserving learning technique and the baselines in terms of the privacy-utility trade-off. Specifically, we compare the utility between different techniques at a given privacy protection level, or vice versa.
WikiText-2	CUSTOMERSIM
Model	PPL	AccT1	σ	C	ε	δ	PPL	AccT1	σ	C	ε	δ
LSTM, No-DP	60.98 ± 0.48	0.36 ± 0.00	-	-	-	-	3.06 ± 0.01	0.75 ± 0.00	-	-	-	-
DPSGD	305.86 ± 3.00	0.27 ± 0.00	0.50	0.10	4.89	8e-5	11.82 ± 0.76	0.70 ± 0.01	0.60	0.01	2.51	8e-5
S-DPSGD	160.04 ± 4.86	0.31 ± 0.00	0.50	1e-3	4.91	8e-5	10.42 ± 0.91	0.69 ± 0.01	0.70	5e-3	2.74	8e-5
Table 2: Model utility and privacy guarantee on WikiText-2 (left) and CUSTOMERSIM (right). PPL: Perplexity on the test set. AccT1: Top-1 next word prediction accuracy.  $\sigma$ : Noise multiplier in the Gaussian noise.  $C$ : Clipping threshold.  $\epsilon, \delta$ : Privacy guarantee in  $(\epsilon, \delta)$ -privacy.
Figure 3: Privacy-utility trade-off, canary insertion attack and membership inference attack on WikiText-2.
5.1.1 Attack Details
We perform two types of attacks: 1) canary insertion and 2) membership inference.
Canary insertion is proposed by Carlini et al. (2019). It first inserts random sequences called canaries into the training dataset, then trains the model, and finally calculates the following exposure for the inserted canaries to measure if the model will unintentionally memorize these canaries. Canaries are of a specific format, e.g.,  $s =$  "The number is  $\odot \odot \odot \odot$ ", where  $\odot$  are filled with random values from a randomness space  $\mathcal{R}$  such as a space of digits from 0 to 9. To obtain the exposure, we enumerate all possible sequences in the specified form, and calculate the negative log-rank with an additional constant, as shown below. Lower exposure indicates the model is more private. In our setting, we insert the secret "My ID is 341752" into the training data for 10 times to make the differences between models more salient.
Definition 5. Given a canary  $s[r]$ , a model with parameters  $\theta$ , and the randomness space  $\mathcal{R}$ , the exposure of  $s[r]$  is
$$
\boldsymbol {e x p o s u r e} _ {\theta} = \log_ {2} | \mathcal {R} | - \log_ {2} \boldsymbol {r a n k} _ {\theta} (s [ r ])
$$
Membership Inference is a widely used attack method that identifies if a given sample is a member of the training dataset. Lower inference accuracy means that it is harder to infer a member from