SlowGuess/NAACL_2022_OCR huggingface dataset:
Xuandong Zhao Lei Li Yu-Xiang Wang
{xuandongzhao,leili,yuxiangw}@cs.ucsb.edu
Large language models are shown to memorize privacy information such as social security numbers in training data. Given the sheer scale of the training corpus, it is challenging to screen and filter all privacy data, either manually or automatically. In this paper, we propose Confidentially Redacted Training (CRT), a method to train language generation models while protecting the confidential segments. We borrow ideas from differential privacy (which solves a related but distinct problem) and show that our method is able to provably prevent unintended memorization by randomizing parts of the training process. Moreover, we show that redaction with an approximately correct screening policy amplifies the confidentiality guarantee. We implement the method for both LSTM and GPT language models. Our experimental results show that the models trained by CRT obtain almost the same perplexity while preserving strong confidentiality<sup>1</sup>.
Language models (LM) have rich real-world applications in, among others, machine translation (Bahdanau et al., 2015), AI chatbots (Hosseini-Asl et al., 2020), question answering (Kwiatkowski et al., 2019), and information retrieval (Ganguly et al., 2015). The advent of transformers (Vaswani et al., 2017) has fostered a dramatic advancement in the capabilities of generative neural language models, yet they come at a cost to privacy, as the amount of excess parameters in the LM enables it to memorize certain training samples. Recent works show that sensitive user information from the training dataset, such as address and name, can be extracted verbatim from text generation models by querying the LM as an API (Carlini et al., 2019, 2021; Lee et al., 2022). How to train a high-performing language model without memorizing
sensitive text has become a major research challenge.
Existing solutions to this problem primarily leverage differential privacy (DP) (Dwork et al., 2006).
Differentially private learning algorithms ensure that an attacker could not infer whether a data point is used for training, let alone extracting the sensitive information within that data point.
However, there are several mismatches between the problem of privacy that DP addresses, and our problem of preventing the memorization of sensitive text (henceforth referred to as confidentiality). First, confidential information in a natural language dataset is sparse (e.g., the bulk of an email might not carry confidential information). DP's undiscriminating protection for all sentences could be unnecessarily conservative which limits the utility of the trained model. Second, what needs to be protected is the content of the sensitive text, rather than the data context. For example, in the sentence "My SSN is 123-45-6789.", it is the actual SSN that we hope to conceal rather than the general information that someone entered her SSN in a chatbot dialogue. Thirdly, the same sensitive content could appear in many data points, which makes the protection of the content more challenging than protecting one data sample. These differences motivate us to treat the problem of confidentiality protection in LM separately with new definitions.
Besides DP, we also consider classical techniques of redaction and dedduplication. Redaction refers to the process of removing sensitive or classified information from a document prior to its publication in governmental and legal contexts. Dedduplication is the procedure of detecting and removing identical and nearly identical texts from a corpus. The main challenge of applying these techniques is that it is hard to manually redact a gigantic dataset and automated tools are far from being perfect.
$^{1}$ Our code is available at https://github.com/XuandongZhao/CRT
Human Language Technologies, pages 943 - 955
Figure 1: An example from simulated dialog dataset CustomerSim. The yellow highlights are confidential content (middle). Left shows the text after Reduction by a sequence labeling policy  $\pi$ . However, if the policy is not perfect, there exists false negative or false positive samples as shown on the right.
The contribution of this paper is fivefold.
1. We show that in the absence of a perfect screening policy, the risk of a language model memorizing sensitive content is real and can be efficiently exploited with only blackbox access to the model even if the learning algorithm satisfies the recently proposed notion of selective differential privacy (Shi et al., 2021).
2. Inspired by differential privacy, we introduce a new definition of confidentiality which precisely quantifies the risk of leaking sensitive text.
3. We propose CRT to train language generation models while protecting confidential text. The method with dedduplication and redaction operations work even under imperfect confidential text labeling policies.
4. We theoretically prove that CRT, combined with differentially private stochastic gradient descent (DP-SGD), provides strong confidentiality guarantees.
5. Our experiments on both MultiWOZ 2.2 and CustomerSim datasets show that different models trained by CRT can achieve the same or better perplexity than existing solutions (against the attacks of Carlini et al. (2019, 2021)).
To the best of our knowledge, we are the first that rigorously establish the role of dedduplication and
redaction in achieving provably stronger confidentiality (or the related differential privacy) guarantees; and the first that achieve provably confidentiality in transformer models with only a mild utility loss.
Next, we briefly introduce the relevant background and discuss the related work to put our work in context.
Language modeling is a fundamental problem in natural language processing (Devlin et al., 2019; Howard and Ruder, 2018; Raffel et al., 2020). Consider a text sequence that consists of multiple tokens from a vocabulary  $\mathcal{V}$ , i.e.,  $\boldsymbol{w} = (w_{1}, w_{2}, \ldots, w_{n})$ , where  $w_{i}$  is the  $i$ -th token. The goal of language modeling is to construct a generative model of the distribution  $\operatorname{Pr}(\boldsymbol{w})$ , by applying the chain rule  $\operatorname{Pr}(\boldsymbol{w}) = \prod_{i=1}^{n} \operatorname{Pr}(w_{i} \mid \boldsymbol{w}_{<i})$ . We let  $f_{\theta}(w_{i} | \boldsymbol{w}_{<i})$  denote the likelihood of token  $w_{i}$  when evaluating the neural network  $f$  with parameters  $\theta$ . A language model is trained to maximize the probability of the data in a training set  $\mathcal{W}$ , by minimizing the negative log-likelihood over each training example with the loss function  $\mathcal{L}(\theta) = -\log \prod_{i=1}^{n} f_{\theta}(w_{i} \mid \boldsymbol{w}_{<i})$ . Recurrent neural networks (RNNs) used to be a common choice for the neural network architecture to estimate the probability distribution  $\operatorname{Pr}(\boldsymbol{w})$ . (Hochreiter and Schmidhuber, 1997; Mikolov et al., 2010). More recently, large-scale Transformer-based language models have replaced RNNs in state-of-the-art
models for all sorts of NLP tasks (Vaswani et al., 2017; Radford et al., 2019). Nevertheless, common language models are vulnerable to privacy attacks and possibly expose information about their sensitive training data (Carlini et al., 2019, 2021).
Differentially private (DP) learning methods (see, e.g., Abadi et al., 2016) has been applied to language models as a blanket solution for a number of privacy and security risks. McMahan et al. (2018) trained an RNN language model with DP guarantees in a federated learning setup. Anil et al. (2021) pre-trained BERT under DP on datasets with hundreds of millions of examples. These paper also demonstrated that DP can effectively prevent data-extraction attacks in practice even for algorithms with DP guarantees that are considered too weak from a theoretical-perspective (e.g.,  $\epsilon = 8$  or 16). However, the strong protection of DP often results in a substantial drop in the utility of the trained model, which makes them less desirable in practice. In fact, it was recently shown that it is necessary for deep learning models to memorize certain training data to achieve high accuracy (Feldman, 2020), which suggests that DP or any other techniques that require the model to not memorize any training data will perform poorly in the high-dimensional, power-law distributed real datasets. This motivates us to consider weakened models that only prevent memorizing the sensitive part of the text.
Recent works (Lee et al., 2022; Kandpal et al., 2022) show that dedduplication enables language models to emit memorized text less frequently with same or better accuracy. However, deduplicating training datasets can not prevent all unintended memorization. We combine dedduplication and redaction and then apply both techniques to the training process of LM to achieve confidentiality with provable guarantee.
The closest to us is perhaps the work of Shi et al. (2021), who proposed selective differential privacy (S-DP), which requires indistinguishability between two datasets that differ only on a sensitive message. Correspondingly, they propose an algorithm (Selective DP-SGD) for training RNN that adds noise only to the part of computation that involves sensitive tokens. To define S-DP and to run Selective DP-SGD, one needs to have access to a policy function  $F$  which determines which token is sensitive. This requirement limits the applicability of their approach to those applications where
such perfect  $F$  is known. We note that even for name-entity recognition the state-of-the-art model is far from being perfect, and which part of the text is considered sensitive is often ambiguous even for human annotators. We will see that naively running Selective DP-SGD with an approximate policy function does not provide a meaningful confidentiality guarantee and is vulnerable to practical data extraction attacks. Finally, we note that in the case when a perfect policy function is available, we can simply use it for redaction, which provides a perfect S-DP with  $\epsilon = 0$ . A big part of our contribution is to refine S-DP to a (slightly different) definition called "confidentiality" and to demonstrate that we use an approximate screening policy to amplify the confidentiality parameter.
3 The CRT Method and Theory
In this section, we develop our method with provable confidentiality.
3.1 Formally defining confidentiality
Let the dataset be a collection of  $n$  data points — each being a sequence of tokens. A "secret"  $x$  is a contiguous subsequence of tokens within a data point that is considered sensitive or confidential. The goal of our research is to allow us to train language models on such datasets that could contain secrets while provably prevent the model from remembering that these secrets were. We start by defining a formal definition of confidentiality, which uses the following idea of indistinguishability from the DP literature.
Definition 1 (Indistinguishability). We say that a pair of distributions  $P, Q$  defined on the same probability space are  $(\epsilon, \delta)$ -indistinguishable if for any measurable set  $S$ ,
$$
\operatorname * {P r} _ {P} [ S ] \leq e ^ {\epsilon} \operatorname * {P r} _ {Q} [ S ] + \delta .
$$
Definition 2 (Confidentiality). We say that  $\mathcal{A}$  ensures that a secret  $x$  is  $(\epsilon(x), \delta)$ -confidential, if for any dataset  $D$  that contains  $x$  in one of its data points, and an alternative dataset  $D'$  that replaces  $x$  in  $D$  with a generic  $\langle \text{MASK} \rangle$ , it holds that  $(\mathcal{A}(D), \mathcal{A}(D'))$  are  $(\epsilon(x), \delta)$ -indistinguishable. In addition, we simply say that  $\mathcal{A}$  ensures  $(\epsilon, \delta)$ -confidentiality if  $\epsilon(x) \leq \epsilon$  for all secret  $x$ .
This definition ensures that an attacker cannot distinguish from the output of  $\mathcal{A}$  (the trained language model) whether it was  $x$  or  $\langle \mathrm{MASK} \rangle$  that
was used for training, thus formalizing the idea of confidentiality. The protection should be viewed as relative, rather than absolute. The definition bounds the risk of any bad event by an multiplicative factor of  $e^{\epsilon}$  and an additive factor of  $\delta$ , which implies that anything that could happen when we run  $\mathcal{A}$  on the sensitive data could've happened with similar probability even if  $\mathcal{A}$  runs on an alternative world where these sensitive information are perfectly masked.
Connections to differential privacy. Our definition of confidentiality is related to (and inspired by)  $(\epsilon, \delta)$ -differential privacy (DP) but is different in several ways. DP is stronger (and implies confidentiality!) requires  $\mathcal{A}$  to ensure  $(\epsilon, \delta)$ -indistinguishability for all  $D, D'$  that can be modified from each other by adding or removing one individual person / data point (or tokens, depending on the desired granularity); but for  $\mathcal{A}$  to ensure  $(\epsilon, \delta)$ -confidentiality, it only requires  $(\epsilon, \delta)$ -indistinguishability for specific  $D, D'$  where  $D'$  replaces  $x$  in  $D$  with  $\langle \text{MASK} \rangle$ . Moreover, it is more informative to define  $\epsilon$  as a function of each specific  $x$ , which is different from DP (it resembles personalized DP (Ghosh and Roth, 2015)).
The confidentiality definition makes sense for our problem because it protects the content of the sensitive text  $x$  rather than its existence. Specifically, a pre-processing algorithm that masks all sensitive text ensures  $(0,0)$ -confidentiality but does not satisfy any non-trivial DP guarantees.
Sometimes, it is useful to consider the confidentiality of multiple secret texts. For example, a secret key  $x$  could appear multiple times in multiple data points. Also, there might be multiple secret texts that are correlated to each other such that the knowledge of one would reveal other secrets.
Definition 3 (Group Confidentiality). We say that  $\mathcal{A}$  ensures that a list of sensitive texts  $S := [x_1, \dots, x_k]$  is  $(\epsilon(S), \delta)$ -(group) confidential, if for any dataset  $D$  that contains  $[x_1, \dots, x_k]$  in up to  $k$  data points, and  $D'$  being the version that replaces each element in  $S$  with  $\langle \text{MASK} \rangle$ , it holds that  $(\mathcal{A}(D), \mathcal{A}(D'))$  are  $(\epsilon(S), \delta)$ -indistinguishable.
A special case of such group confidentiality is when  $S$  collects the all secret text in  $D$ , which protects all secret texts uniformly. We call this uniform- confidentiality. Note that the standard definition of confidentiality also protects every secret  $x$ , except that it protects each secret  $x$  individually, rather than together.
Inspired by the recent development of Bayesian DP (Triastcyn and Faltings, 2020), we also define Bayesian confidentiality as follows.
Definition 4 (Bayesian Confidentiality). Let  $D$  be a dataset that is fixed except a random secret  $x \sim \mu$  drawn from some distribution  $\mu$ . Let  $D'$  be obtained by replacing  $x$  with  $<\text{MASK}>^2$ . Then  $\mathcal{A}$  ensures  $(\epsilon, \delta)$ -Bayesian Confidentiality if for any  $D'$ ,  $(\mathcal{A}(D), \mathcal{A}(D'))$  is  $(\epsilon, \delta)$ -indistinguishable, where  $\mathcal{A}(D)$  is jointly distributed over  $x \sim \mu$  and  $\mathcal{A}$ .
The Bayesian confidentiality measures how much information an attacker could gain if he/she's prior knowledge about this secret  $x$  is described by the distribution  $\mu$ . This is a strict generalization because when  $\mu$  is a single point mass at  $x$ , it recovers Definition 2. The additional generality allows us to quantify the stronger confidentiality guarantee against weaker adversaries without complete information.
3.2 Confidentially redacted training
In this section we describe the CRT method to train language models with provable confidentiality guarantee. It includes two pre-processing operations (dedduplication and redaction) and a switching optimization procedure. The overall idea is to screen the corpus into two separate sets, one public set including sentences with no confidential information, and one private set including sentences containing confidential content. We then use normal optimization algorithms (e.g. SGD) on the public set and differential privacy optimizer (e.g. DP-SGD) on the private set.
Deduplication. The dedduplication procedure Dedup detects all sentences that appear multiple times in the training data and replace them into a single  $<\text{MASK}>$  from the second occurrence onwards ( $<\text{MASK}>$  is for proving purpose).
Redaction. The redaction procedure  $\mathrm{Redact}_{\pi}$  takes applies a sequence labelling policy  $\pi$  to screen confidential content in the training corpus  $D$ .  $\pi(s, x) = 1$  if a token  $x$  in a sentence  $s$  should be confidential. The labeled span in each detected sentence is replaced with a special token  $\langle \mathrm{MASK} \rangle$ . Note that we do not assume the policy is perfect. It may label some non-sensitive tokens as sensitive (false positives) and label some sensitive text as non-sensitive (false negative, or 1—recall).
2Notice that  $D^{\prime}$  is fixed even though  $x$  is random.
Algorithm 1: CRT
Input: Dataset  $D$  (after tokenization / splitting), labelling policies  $\pi, \pi_c$ , number of epochs  $T$
1  $D^{\prime}\gets \operatorname {Dedup}(D)$
$2D^{\prime \prime}\gets \mathrm{Redact}_{\pi}(D^{\prime})$
3  $D^{pri} \gets \{s \in D'' | \exists x \in s \text{ s.t. } \pi(s, x) = 1 \text{ or } \exists x \subset s \text{ s.t. } \pi_c(s, x) = 1\}$
4  $D^{pub} = \{s\in D^{\prime \prime}|s\notin D^{pri}\}$
5 for  $e = 1,\dots,T$  do
Run one epoch of SGD with  $D^{pub}$ .  
Run one epoch $^3$  of DP-SGD with  $D^{pri}$ .
s end
Redact and Dedup could be implemented manually, but with the large text corpus nowadays it is more common that these procedures are implemented using automated tools. For example, Dedup could be implemented efficiently with just one pass of data using a bloom filter (Bloom, 1970) (or other hashing tricks that also catches near-duplicates). Bloom filter in particular, enjoys the nice property that it could have false positives but never any false negatives.  $\mathrm{Redact}_{\pi}$  could be realized by a named entity recognition (NER) model or a personal-identifiable information (PII) detector.
Finally, CRT combines the two pre-processing steps with normal optimizer and DP-SGD, the standard algorithm for deep learning with differential privacy. A pseudo-code of the algorithm is given in Algorithm 1.
Besides using a sequence labeling policy  $\pi$  with balanced precision/recall as part of the redaction process. The algorithm uses another, more conservative, policy  $\pi_c$  with nearly perfect recall to decide on the data points that do not contain sensitive text. In the situation when such  $\pi_c$  isn't available, we simply choose  $\pi_c(s,x) = 1$  for all tokens  $x$  in a sentence  $s$  and the second part becomes the vanilla DP-SGD. It is also important that every data point that contains a <MASK> requires protection.
3.3 Theoretical analysis
We analyze the theoretical properties of the above method and show that they result in provable improvements in the (regular, group and Bayesian)
confidentiality parameters for any algorithms that are provably  $(\epsilon (x),\delta)$  -confidential as defined in Section 3.1.
The following theorem captures the benefit of redaction in improving confidentiality.
Proposition 5 (Confidentiality under redaction). If  $\mathcal{A}$  ensures  $(\epsilon (x),\delta)$ -Confidentiality for each token  $x$  of sentence  $s\in S$  ( $S$  is a corpus), then  $\mathcal{A}\circ$  Redact $_{\pi}$  ensures  $(\tilde{\epsilon} (x),\delta)$ -confidentiality with
$$
\tilde {\epsilon} (x) = \left\{ \begin{array}{l l} \epsilon (x) & \text {i f} \pi (s, x) = 0 \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
In addition,  $\mathcal{A} \circ \operatorname{Redact}_{\pi}$  also satisfies  $(\tilde{\epsilon}(S), \tilde{\delta}(S))$ -group confidentiality with
$$
\tilde {\epsilon} (S) = \sum_ {x \in s \& s \in S} \epsilon (x) \mathbf {1} (\pi (s, x) = 0),
$$
$$
\tilde {\delta} (S) = \tilde {k} e ^ {\tilde {\epsilon} (S)} \delta
$$
where  $\tilde{k} := \sum_{x \in S} \mathbf{1}(\pi(s, x) = 0)$ .
As an application of the above, if  $\mathcal{A}$  ensures  $(\epsilon, \delta)$ -confidentiality, and that the empirical recall rates of the reduction policy on  $D$  is  $1 - \gamma$ , then the above proposition suggests that  $\mathcal{A} \circ \mathrm{Redact}_{\pi}$  improves the uniform-confidenceality over applying  $\mathcal{A}$  without redaction by a factor of  $\gamma$ . The proof is in the appendix.
Redaction also improves Bayesian confidentiality in a way that mirrors the privacy amplification by sampling from the DP literature.
Proposition 6 (Bayesian Confidentiality under Reduction). If  $\mathcal{A}$  ensures  $(\epsilon, \delta)$ -Bayesian Confidentiality with respect to  $\mu[x|\pi(s, x) = 0]$  for a token  $x$  in a sentence  $s$ , then  $\mathcal{A} \circ \mathrm{Redact}_{\pi}$  ensures  $(\log(1 + \gamma(e^{\epsilon} - 1)), \gamma \delta)$ -Bayesian Confidentiality under  $\mu$  if  $\pi$  has a false negative rate (i.e.,  $1 - "Recall")$  of  $\gamma$  under  $\mu$ .
The proposition says that if the redaction policy is accurate for secrets  $x\sim \mu$  , then we can have a stronger confidentiality parameter that scales roughly at  $\tilde{\epsilon} = O(\gamma \epsilon)$  . The idea behind the proof is that over the distribution of  $x\sim \mu$  , with probability  $1 - \gamma$  ,  $\mathrm{Redact}_{\pi}(D) = \mathrm{Redact}_{\pi}(D^{\prime})$  , thus  $\mathcal{A}\circ \mathrm{Redact}_{\pi}(D)\equiv \mathcal{A}\circ \mathrm{Redact}_{\pi}(D^{\prime})$  . With probability  $\gamma$  ,  $\mathrm{Redact}_{\pi}(D)$  ,  $\mathrm{Redact}_{\pi}(D^{\prime})$  are different and conditioning on the fact that  $\mathrm{Redact}_{\pi}$  fails to detect  $x$  . Note that  $\pi$  is also applied to other text that are not sensitive, and could result in false positives, but they do not matter as the modification
$^{3}$ DP-SGD uses Poisson-sampled Gaussian mechanisms (with a random batchsize), thus cannot ensure all data points are seen and some data points might be seen many times. One epoch means the number of iterations that in expectation covers  $|D^{pri}|$  data points.
of  $\operatorname{Redact}_{\pi}$  to  $D$  and  $D'$  will be identical. A full proof is given in the appendix.
Next we turn to dedduplication.
Proposition 7 (Group confidentiality under dedduplication.). If  $\mathcal{A}$  ensures  $(\epsilon(S), \delta(S))$ -Group Confidentiality, then  $\mathcal{A} \circ$  Dedup ensures  $(\epsilon(\text{Unique}(S)), \delta(\text{Unique}(S)))$ -Group Confidentiality.
Deduplication provides a stronger protection for those cases where some secret  $x$  could appear multiple times in the dataset.
Theorem 8. Let  $DP$ -SGD from Algorithm 1 satisfies  $(\epsilon, \delta)$ -differential privacy.
1. Assume  $\pi_c(s,x) = 1$  for all secret tokens  $x$  in a sentence  $s$  such that  $\pi (s,x) = 0$ , then Algorithm 1 satisfies  $(\epsilon \mathbf{1}(\pi (s,x) = 0),\delta)$ -confidentiality.
2. Let  $S$  be a group containing  $m$  unique secrets such that  $\pi_c(s, x) = 1 \forall x \in s$  and  $s \in S$  and that  $\pi$  detects  $\tilde{\gamma}$ -proportion of the unique secrets in  $S$ . Then Algorithm 1 satisfies  $(\tilde{\gamma} m\epsilon, \tilde{\gamma} me^{\tilde{\gamma} m\epsilon}\delta)$ -group confidentiality for  $S$ .
3. Let  $\pi, \pi_c$  has a recall of  $1 - \gamma$  and  $1 - \delta_2$  respectively on  $\mu$ , then Algorithm 1 satisfies  $(\log(1 + \gamma(e^\epsilon - 1)), \gamma \delta + \delta_2)$ -Bayesian Confidentiality for  $\mu$ .
The theorem demonstrates that our CRT algorithm enjoys a full suite of confidentiality guarantees and they all benefit from the dedduplication and redaction, particularly if  $\pi$  has high recall.