SlowGuess/NAACL_2022_OCR huggingface dataset:
You Don’t Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers’ Private Personas
Haoran Li $^{1}$ , Yangqiu Song $^{1}$ , Lixin Fan $^{2}$
$^{1}$ Dept. of CSE, Hong Kong University of Science and Technology
$^{2}$ AI Group, WeBank
hlibt@connect.ust.hk, yqsong@cse.ust.hk,
lixinfan@webank.com
Social chatbots, also known as chit-chat chatbots, evolve rapidly with large pretrained language models. Despite the huge progress, privacy concerns have arisen recently: training data of large language models can be extracted via model inversion attacks. On the other hand, the datasets used for training chatbots contain many private conversations between two individuals. In this work, we further investigate the privacy leakage of the hidden states of chatbots trained by language modeling which has not been well studied yet. We show that speakers' personas can be inferred through a simple neural network with high accuracy. To this end, we propose effective defense objectives to protect persona leakage from hidden states. We conduct extensive experiments to demonstrate that our proposed defense objectives can greatly reduce the attack accuracy from  $37.6\%$  to  $0.5\%$ . Meanwhile, the proposed objectives preserve language models' powerful generation ability.
Social chatbots have been widely used to benefit many applications from answering factual questions to showing emotional companionship. With recent progress in large pretrained language models (Radford et al., 2019; Yang et al., 2019), some attempts (Wolf et al., 2019; Zhang et al., 2020; Ham et al., 2020; Shen et al., 2021; Sevegnani et al., 2021; Gu et al., 2021b) are made to build chatbots based on large generative language models (LMs). To train such LM-based chatbots, private conversations are collected. Unfortunately, large language models tend to memorize training data and some private data can be recovered from models (Pan et al., 2020; Carlini et al., 2021). Besides such memorization problems, "overlearning" on simple training objectives can reveal sensitive attributes indirectly related to the learning task (Song and Shmatikov, 2020). LM-based social chatbots essentially inherit the privacy issues of general LMs
and the overlearning problem.
For example, as Figure 1 shows, when using a fine-tuned GPT-2 as the encoder and decoder of an LM-based social chatbot, if the learned representation of each utterance can be obtained by an adversary, then the adversary can build a classifier to predict the persona information based on the representation. As shown by the example, for five out of 14 utterances, the attacker can successfully predict the persona, which can be harmful if the users (speakers of the utterances) do not prefer to reveal the persona information. Thus, in practice, when deploying such kinds of chatbots in real applications, we should first make sure that no private information can be leaked by the models.
To systematically study the privacy issues in LM-based social chatbots, there are several challenges. First, there is no existing data that can be used to quantify how much private information is revealed by an LM. Second, there has been no existing work showing how to attack utterance-level representations to obtain sensitive information. Third, there has been no existing LM-based chatbot that can defend against persona inference attacks, and no study shows how to protect both known and unknown persona attributes.
In this paper, to address the above challenges, we use the fine-tuned GPT-2 as our chatbot. We first collect a dataset by aligning personas with corresponding utterances in PersonaChat dataset (Zhang et al., 2018). Then we show that "overlearning" can happen for LM-based chatbots to reveal personas of speakers. We build a single external multi-layer perception (MLP) attacker model to perform black-box persona inference attacks on the utterance-level embeddings. With no access to parameters of the chatbot, the attacker model can infer speakers' personas with  $37.59\%$  accuracy over 4,332 personas. The high accuracy of the attacker model implies that the utterance-level embeddings have potential vulnerabilities to reveal
Human Language Technologies, pages 5858 - 5870
Context	Attacks on LM	Attacks on the defended LM
Human A	Hello, how are you tonight?	I take things very literally	✕	I am engaged to be married	✕
Human B	Hello my friend. I am well.	I am a happy person	✕	I like to go shopping with my daughters	✕
Human A	Good, glad to hear it. What do you do for fun?	I do whatever it takes to get what I want	✕	My favorite color is blue	✕
Human B	I ride around the town on my cool bicycle.	I love to ride my bike on the weekend	✕	My favorite color is blue	✕
Human A	Really? I really like mountain bike too.	I also like to mountain bike	✓	My favorite color is blue	✕
Human B	I wish I lived in the mountains.	I have never been out of the country	✕	My favorite color is blue	✕
Human A	Do you like nature? I have been to 12 national parks.	I like to visit national parks	✓	My favorite color is blue	✕
Human B	I love nature. I like looking at plants.	I really love plants	✓	My favorite color is blue	✕
Human A	I love plants too, and hiking. In fact, I am actually an environmental activist.	I am an environmental engineer	✓	My favorite color is blue	✕
Human B	Cool, I am a vegan.	I am a vegan	✓	My favorite color is blue	✕
Human A	Nice, do you have a favorite food?	I love ham and cheese sandwiches	✕	I have my own salon	✕
Human B	My favorite dish is lentil curry.	My favorite meal is chicken and rice	✕	My favorite color is blue	✕
Human A	I have never had that, but I want to try it now.	I am a great cook	✕	I am a doctor	✕
Human B	What do you like to do the most?	I do whatever it takes to get what I want	✕	I am studying to be a dentist	✕
Figure 1: Black-box persona inference attacks (over 4,332 personas) on a dialog. Every representation of the utterance, which is based on the last hidden state of GPT-2, is attacked without defense (column of "Attacks on LM") and with defense (column of "Attacks on the defended LM"). If the model can predict the persona of the speaker based on the observed representation, then we regard it as a successful attack; otherwise, unsuccessful. In practice, when deploying a model, a robust model which will reveal nothing of the encoded utterances is expected.
speakers' private persona attributes. Thus, it is necessary to improve training algorithms to address such overlearning issues. Finally, we apply defense learning strategies on the GPT-2 to prevent such black-box attacks. We combine proposed KL divergence loss (KL loss) with mutual information loss (MI loss) (Song et al., 2019) as additional defense objectives to train the GPT-2 and decrease the attacker's persona inference accuracy to  $0.53\%$ . Our contributions can be summarized as follows:
1): To the best of our knowledge, we are the first to disclose and analyze the persona inference attack for LM-based chatbots and treat it as a privacy risk.
2): We propose an effective defensive training algorithm to prevent dialog representations from leaking personas of the corresponding speakers by uniform distribution approximation and mutual information minimization.
3): We conduct extensive experiments to quantify both privacy and utility of proposed defense mechanisms. Besides solving the persona leakage issue, the proposed training algorithm has nearly no negative influence on utility.
Language models trained on private data suffer privacy risks of revealing sensitive information. Previous researches mainly considered black-box attacks that assumed attackers only had access to
inputs and outputs of language models. Carlini et al. (2021) performed black-box model inversion attack on GPT-2 through descriptive prompts with beam search. Lehman et al. (2021) examined BERT pretrained on Electronic Health Records via blank filling and model probing to recover Personal Health Information. Furthermore, given black-box access to a language model's pre-train and finetune stages, Zanella-Béguelin et al. (2020) showed that sensitive sequences of the fine-tuning dataset can be extracted. For the distributed client-server setup, Malekzadeh et al. (2021) considered the sensitive attribute leakage from the server side with honest-but-curious (HBC) classifiers.
What is worse, for an LM-based chatbot, its training conversations are prone to include more private attributes than other commonly-used corpora for language modeling like BooksCorpus (Zhu et al., 2015) and Wikipedia. Tigunova et al. (2019) proposed Hidden Attribute Model (HAM) to extract professions and genders of speakers from various dialog datasets. Wu et al. (2020) further applied Attribute Extractor to generate speakers' attribute triplets flexibly and suggested downstream tasks based on the triplets. Pan et al. (2020) exploited embeddings of language models to recover inputs' digits and keywords. Though the setup of this work is similar to ours, they merely consider simple cases of data recovery with given rules and suffer great utility degradation to obtain optimal defense performance. For our work, there is no fixed pattern or rule for the model input. Instead of finding key-
1Code is publicly available at https://github. com/HKUST-KnowComp/Persona_leakage_and_ defense_in_GPT-2.
words or recovering digits, we aim to infer more complicated private attributes from such embeddings. Moreover, our proposed defenses have almost no influence on the utility.
3 Attacking on Language Models
In this section, we illustrate black-box persona inference attacks on GPT-2 and our defense strategies. In Section 3.1, we first give the problem formulation. Then we describe the attack in Section 3.2.
We assume that there is a GPT-2 based chatbot  $f$  pretrained on private conversations  $D$ . Only language modeling is used to train the chatbot:
$$
L _ {f} (u; \theta_ {f}) = - \sum_ {i = 1} ^ {| u |} \log \left(\Pr \left(w _ {i} \mid c, w _ {0}, w _ {1}, \dots , w _ {i - 1}\right)\right), \tag {1}
$$
where  $f$  refers to the LM-based chatbot with given utterance  $u = \{w_0, w_1, \dots, w_{|u|-1}\}$  and previous context  $c$ . An adversary owns one external annotated dialog dataset  $D_a = \{(\mathbf{U}_1, \mathbf{s}_1), (\mathbf{U}_2, \mathbf{s}_2), \dots, (\mathbf{U}_n, \mathbf{s}_n)\}$  with  $n$  conversations where  $\mathbf{U}_i$  indicates a list of utterances  $\{u_{i1}, u_{i2}, \dots, u_{in_i}\}$  of  $i$ -th conversation and  $\mathbf{s}_i$  corresponds to a list of sensitive personas  $\{s_{i1}, s_{i2}, \dots, s_{in_i}\}$  for corresponding utterance. Each persona  $s_{kj}$  is an integer that can be mapped to its persona according to a predefined dictionary and  $0 \leq s_{kj} \leq C - 1$  where  $C$  is the total number of predefined persona attributes. The goal of the adversary is to infer speakers' personas  $s$  from utterances' embeddings  $f(u)$  where  $u$  and  $s$  refer to any utterance and its persona label.
3.2 Black-box Persona Inference Attack
The persona inference attack can be viewed as a supervised classification task. For the black-box attack setup, the adversary can only query the target dialog model  $f$  with access to embeddings of adversary's inputs and cannot access or modify model parameters  $\theta_f$ . As shown in the left part of Figure 2, the adversary tries to build its attacker model  $\mathcal{A}$  with its external data  $D_a$  and dialog model  $f$ . The persona predictor's output  $\mathcal{A}(f(u))$  is the estimated probability distribution over  $C$  persona attributes. Its loss function  $L_{\mathcal{A}}$  exploits cross-entropy between the predicted distribution and ground truth distribution that can be formulated as:
$$
L _ {\mathcal {A}} \left(u _ {k j}, s _ {k j}; \theta_ {\mathcal {A}}\right) = \operatorname {C E} \left(\mathcal {A} \left(f \left(u _ {k j}\right), s _ {k j}\right), \right. \tag {2}
$$
where CE refers to cross-entropy loss between persona label  $s_{kj}$  and  $A(f(u_{kj}))$ .
A well-performed persona predictor  $\mathcal{A}$  can cause great privacy threats. For machine learning as a service (MLaaS),  $\mathcal{A}$  can be applied to perform a man-in-the-middle attack on the application programming interfaces. Moreover, even if the raw data are protected and the transmission channel is secure, a curious service provider can train its attacker  $\mathcal{A}$  to collect personas of service users.
4 Defense Learning Strategies
The LM training objective in Equation 1 only considers the utility of chatbots. In later experiment sections, we show that LM brings severe overlearning issues. Ideally, to achieve an optimal privacy-preserving chatbot against persona inference attacks, the probability distribution of the attacker model  $\mathcal{A}$  should be close to the uniform distribution. That is, the adversary cannot improve its inference accuracy from posterior estimation  $\mathcal{A}(f(u))$  and the accuracy is no better than making random guesses on the persona attributes. Moreover, the constraints on privacy should have minor degradation on the utility to maintain the strong generation ability of chatbots.
Following the intuition that the adversary cannot obtain better results than a random guess, in Section 4.1, we propose KL loss that aims to flatten the persona predictor's estimated distribution. Based on minimizing the mutual information between hidden states  $f(u)$  of chatbots and private persona attributes  $s$ , we propose MI loss in Section 4.2. Lastly, we show the overall training objective in Section 4.3.
4.1 KL Loss
KL loss aims to minimize the Kullback-Leibler divergence between  $\mathcal{A}(f(u))$  and the uniform distribution. It flattens the distribution of  $\mathcal{A}(f(u))$  so that the adversary cannot gain any useful knowledge after training attacker model  $\mathcal{A}$ . The KL divergence between the uniform distribution and  $\mathcal{A}(f(u))$  can be formulated as:
$$
D _ {K L} (\operatorname {U N I} \left\| \mathcal {A} (f (u)) \right.) = - \frac {1}{C} \sum_ {k = 0} ^ {C - 1} \log \left(C \Pr (k | f (u), \theta_ {\mathcal {A}})\right), \tag {3}
$$
where UNI indicates the uniform distribution and  $k$  indicates the  $k$ -th persona label of  $C$  labels. For optimization, we can leave out constant terms and the logarithm (Mireshghallah et al., 2021) to obtain
Figure 2: Scenarios for attacks without defense (left) and with defense (right). The GPT-2's training stage is marked by  $①$  and the attacking stage is marked by  $②$ . Both language modeling and defender objectives are jointly trained for the defense to optimize the GPT-2 model. After GPT-2's training stage  $①$  is finished, parameters of GPT-2 are all frozen and then the attacking stage  $②$  starts. The defender shares the same architecture as the attacker and uses  $L_{kl}$  with  $L_{mi}$  as defense objectives.
the following loss function:
$$
L _ {D} (u; \theta_ {\mathcal {A}}) = - \frac {1}{C} \sum_ {k = 0} ^ {C - 1} \Pr (k | f (u), \theta_ {\mathcal {A}}). \tag {4}
$$
However, from the perspective of defenders, they have no access to attacker model  $\mathcal{A}$  and its parameters. Instead, they can build their own persona predictor as a fake attacker. More specifically, they may mimic the adversary to annotate a dataset  $D_{a}^{\prime}$  and a persona predictor  $\mathcal{A}_p$ . Then the KL loss becomes:
$$
L _ {k l} (u; \theta_ {\mathcal {A} _ {p}}, \theta_ {f}) = - \frac {1}{C} \sum_ {k = 0} ^ {C - 1} \Pr (k | f (u), \theta_ {\mathcal {A} _ {p}}), \tag {5}
$$
where parameters of the chatbot  $\theta_{f}$  and the fake attacker  $\theta_{\mathcal{A}_p}$  are updated via KL loss. The intuition is to train the chatbot together with a fake attacker to prevent model overlearning by flattening the attacker model's distribution.
4.2 MI Loss
The privacy constraint requires that hidden representations should not reveal the persona attributes. In other words, given any utterance  $u$  and persona  $s$  behind the utterance  $u$ , we want to minimize the mutual information between  $f(u)$  and  $s$ :
$$
\min  _ {\theta_ {f}} I (f (u); s). \tag {6}
$$
Following the derivation in Song et al. (2019) and Li et al. (2020), the upper bound can be formulated
$$
I (f (u); s) \leq \mathbb {E} _ {q (f (u))} D _ {K L} (q (s | f (u)) \| p (s)), \tag {7}
$$
where  $p(s)$  can be any distribution for  $s$ ,  $q(x)$  refers to probability distribution of model  $f$  parameterized by  $\theta_f$  and  $f(u)$  is assumed to be sampled from the conditional distribution  $q(f(u)|x,s)$ . However,  $q(s|f(u))$  is hard to estimate. Instead, we use  $p_{\Psi}(s|f(u))$  to approximate  $q(s|f(u))$  via minimizing their KL divergence and then we can obtain the following lower bound (Song et al., 2019):
$$
\begin{array}{l} \mathbb {E} _ {q (f (u))} D _ {K L} (q (s | f (u)) \mid \mid p (s)) \tag {8} \\ \geq \mathbb {E} _ {q (f (u))} [ \log p _ {\Psi} (s | f (u)) - \log p (s) ]. \\ \end{array}
$$
Therefore, our objective in Equation 6 can be formulated as an adversarial training objective:
$$
\min  _ {\theta_ {f}} \max  _ {\Psi} \mathbb {E} _ {q (f (u))} [ \log p _ {\Psi} (s | f (u)) - \log p (s) ]. \tag {9}
$$
$\log p(s)$  is independent of  $f(u)$ , and we may leave this term out in Equation 9:
$$
\min  _ {\theta_ {f}} \max  _ {\Psi} \mathbb {E} _ {q (f (u))} [ \log p _ {\Psi} (s | f (u)) ]. \tag {10}
$$
Then, Equation 10 illustrates an adversarial game between an adversary  $p_{\Psi}$  who manages to infer  $s$  from  $f(u)$  and a defender who modifies  $\theta_f$  to protect  $s$  from persona inference attack. Adversarial training is widely used to protect sensitive features in natural language processing (Elazar and Goldberg, 2018; Coavoux et al., 2018; Li et al.,
2018). Using the persona predictor model  $\mathcal{A}_p$  with softmax activation to learn  $p_{\Psi}$ , we obtain the final objective for the defender:
$$
\min  _ {\theta , \mathcal {A} _ {p}} \max  _ {\theta_ {f}} \operatorname {C E} (\mathcal {A} _ {p} (f (u)), s). \tag {11}
$$
We can rewrite Equation 11 into two losses:  $L_{mi1}(u_{kj}, s_{kj}; \theta_{\mathcal{A}_p}) = \mathrm{CE}(\mathcal{A}_p(f(u_{kj})), s_{kj})$  and  $L_{mi2}(u_{kj}, s_{kj}; \theta_f) = -\mathrm{CE}(\mathcal{A}_p(f(u_{kj})), s_{kj})$  for the fake adversary and the chatbot respectively. Then our MI loss can be formulated as:
$$
L _ {m i} = \lambda_ {0} L _ {m i 1} + L _ {m i 2}, \tag {12}
$$
where  $\lambda_0$  controls the ratio between two the fake attacker  $\mathcal{A}_p$  and the defended chatbot  $f$ .
4.3 Overall Objective
The right part of Figure 2 illustrates how the chatbot is trained to address the black-box attack. The loss function for the defender combines KL loss, MI loss and LM loss. Notice that the fake adversary objective in MI loss violates KL loss which tries to make the distribution of  $\mathcal{A}_p$  flatten. Our proposed loss assigns more weights to the KL loss:
$$
L = L _ {f} + \lambda_ {1} L _ {k l} + \lambda_ {2} L _ {m i}, \tag {13}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are hyper-parameters and  $\lambda_{1} \geq 10\lambda_{2}$  to flatten the distribution of  $\mathcal{A}_p$ . Though the chatbot trained with overall loss  $L$  still cannot interfere training process of  $\mathcal{A}$  during black-box attacks,  $L$  aims to mitigate persona overlearning issues of  $f$  to address such persona inference attacks.
In this section, we conduct experiments to evaluate the performance of privacy and utility for the proposed defense learning strategies. In Section 5.1, we give our experimental settings in detail. In Section 5.2, we show the attacking performance with and without defense. In Section 5.3, we perform ablation study on defense objectives. In Section 5.4, we use automatic metrics to evaluate chatbots' utility. We conduct various attack setups in Section 5.5 and perform a case study in Section 5.6.
Dataset. To train the GPT-2 as our chatbot, we use the DialogGPT (Zhang et al., 2020) pretrained on Reddit comment chains. Then we use PersonaChat dataset (Zhang et al., 2018) to fine-tune the GPT-2. To obtain annotated dataset  $D_{a}$  for the adversary, we align personas to corresponding utterances
Stat Type	Value
Dialogs	10,907
Utterances (turns)	162,064
Unique personas	4,332
Total personas	98,056
Labeled turns	32,147
Avg. turns per dialog	14.86
Avg. labeled turns per dialog	2.95
Avg. words per turn	11.71
Table 1: Statistics of the aligned dataset.
through positive (utterance, persona) pairs provided in Dialogue NLI (Welleck et al., 2019) dataset. For those utterances with no annotations, we assign label  $-1$  to them. We reshuffle the dataset to balance the label distribution among train/val/test datasets with the ratio of  $8:1:1$ . We first let the attacker and defender share the same training data. In later sections, we will separate the annotated data for the adversary and defender with no overlap. A summary statistics of  $D_{a}$  is shown in Table 1.
Attacker model. In our experiment, we use a 2-layer neural network with cross-entropy loss as the attacker model. The attacker model exploits the final layer embedding of the last token “<lendof-text|>” from the GPT-2 as model input. We also try other attacker model architectures (transformer block based attackers) and input embeddings (average of all embeddings in the final layer of GPT-2), but the attacking performance is worse than the 2-layer model mentioned above.
Evaluation Metrics. The evaluation metrics are based on privacy and utility. For privacy, we use persona inference accuracy and weighted F1-score to evaluate the attacker's performance. We also use Bayesian Privacy (BP) (Gu et al., 2021a) to quantify the attacker's privacy loss for the estimated persona distribution. Top-k accuracy is reported in the Appendix. For utility, we apply BERTScore (Zhang* et al., 2020), Distinct (Li et al., 2016), BLEU (Papineni et al., 2002) and perplexity (PPL) as evaluation metrics. BERTScore and BLEU measure similarity between generated outputs and ground truth while Distinct (Dist) focuses on diversity. Perplexity shows the uncertainty when the LM model fits the data.
5.2 Privacy
Attacks without Defense. We list the attacking performance of  $\mathcal{A}$  in multiple scenarios shown in Table 2. To demonstrate the overlearning issue of GPT-2, we consider 2 baseline attacks. If the adversary has no knowledge about persona attributes
Acc	F1	Max-Ratio
Random Pred	0	0	0.02
Best Guess	0.72	1.02e-3	100
LM	37.59	3.65e-1	1.34
LM+KL+MI	0.53	6.78e-5	81.87
LM+KL	14.43	1.13e-1	10.60
LM+MI	0.53	5.57e-5	99.84
Table 2: Evaluation on the privacy over 4,332 persona labels. Acc and Max-Ratio are measured in  $\%$  Acc refers to test persona inference accuracy. F1 uses weighted average F1-score. Max-Ratio indicates the ratio that the most frequent prediction shares among all predictions. The worse the attack model performs, the better privacy protection can be achieved.
distribution, then it can randomly guess over 4,332 labels (Random Pred). Otherwise the adversary can perform Best Guess that only guesses the most frequent persona in the dataset.  $LM$  indicates the attacker performance that only language modeling objective is applied to train the chatbot without any defense mechanism. From the table, the test persona inference accuracy on the  $LM$  achieves  $37.59\%$  while guessing on the label with most occurrences merely has  $0.72\%$  accuracy. That is, the black-box persona inference attack has  $52\times$  the accuracy of guessing. The huge performance gap between the attacker model and the baseline guess method indicates that simple language modeling objective has serious overlearning issues that unintentionally capture private personas of speakers.