SlowGuess/NAACL_2022_OCR huggingface dataset:
Attacks on the Defended LM. To avoid the persona overlearning issue, we use additional defense objectives illustrated in Section 4.  $LM + KL + MI$  utilizes language modeling, KL loss and MI loss in Equation 13 to train the GPT-2. As demonstrated in Table 2, the attacker performance on  $LM + KL + MI$  significantly reduces the attacking accuracy from  $37.59\%$  to  $0.53\%$  and F1-score drops from 0.37 to nearly 0. This defense mechanism can even outperform Best Guess in terms of privacy protection. That is, even if the adversary annotates its own dataset to train an attacker model, the attacking performance is still worse than simply guessing the most frequent label. As a result, the black-box persona prediction attack becomes useless after applying the defenses for the chatbot. The adversary cannot obtain any speaker's persona from the embedding  $f(u)$  by training  $\mathcal{A}$ .
To learn why the proposed defenses work so well, we further examine the ratio of the most frequent predicted label (Max-Ratio) among all pre
dictions. The accuracy of Best Guess reveals that the most frequent label in the test set has a ratio of  $0.72\%$ . After applying KL loss and MI loss, the attacker model tends to make predictions on a single label. For  $LM + KL + MI$ , the Max-Ratio even occupies  $81.87\%$  predictions. This implies that the proposed defense strategies may have the potential to fool the attacker model to make wrong predictions on a single slot. We will further investigate this implication in later sections.
Overall, the above experiment demonstrates that our proposed defense learning strategies can effectively mitigate the persona overlearning issue and avoid black-box persona inference attacks.
To show the effectiveness of proposed KL loss and MI loss and how they affect the performance of black-box persona inference attacks, we consider the inclusion and exclusion of proposed defense objectives. The result is shown in Table 2.  $LM + KL$  indicates the GPT-2 is trained with language modeling and KL loss.  $LM + MI$  applies language modeling and MI loss. From the table, it can be seen that  $LM + KL$ ,  $LM + MI$  and  $LM + KL + MI$  are all able to reduce the test accuracy of the attacks. The KL loss is weaker from the perspective of defense, but it tends to flatten the estimated persona distribution with much smaller Max-Ratio. The  $LM + MI$  shares similar test accuracy and F1-score with  $LM + KL + MI$ , but nearly all predictions are made on a single persona label with a ratio of  $99.84\%$ . This suggests that MI loss causes the attacker model to predict all labels on a single persona attribute. After KL loss is applied on  $LM + KL + MI$ , the Max-Ratio drops to  $81.87\%$ .
As discussed earlier, high Max-Ratio may also cause privacy leakage. Suppose the adversary knows the persona with Max-Ratio, then it can improve its guess by not predicting this persona, which is a threat for fewer persona labels (for example, binary classification). These results verify that KL loss introduces flatter estimation and MI loss is more effective against persona overlearning, which conforms to our intuition of their objectives in Section 4.
5.4 Utility
Besides privacy, utility is another key objective to train a chatbot. Several automatic metrics are considered to evaluate the generation performance. For generation, we use GPT-2 to generate responses of
PPL	Distinct	BLEU	BERTScore
Dist-1	Dist-2	BLEU-1	BLEU-2	BLEU-4	Precision	Recall	F1
LM	14.821	0.952	0.879	0.121	0.0551	0.0123	0.860	0.843	0.851
LM+KL	28.926	0.954	0.880	0.121	0.0558	0.0130	0.859	0.844	0.851
LM+MI	18.740	0.953	0.880	0.118	0.0531	0.0121	0.859	0.843	0.851
LM+KL+MI	19.674	0.953	0.880	0.119	0.0525	0.0105	0.858	0.842	0.850
Table 3: Evaluation on the utility over 4,332 persona labels.
Unseen (0-2)	Overall (0-7)
Acc	F1	Max-Ratio	BPu	Acc	F1	Max-Ratio	BPu
Random Pred	34.42	0.35	33.90	0	13.21	0.14	13.35	0
Best Guess	56.84	0.41	100	2.60e-1	22.67	0.08	100	2.60e-1
LM	86.83	0.91	50.72	2.11e-3	72.37	0.72	20.94	3.04e-3
LM+KL+MI	28.68	0.37	58.15	2.84e-4	30.26	0.21	77.94	2.65e-4
Table 4: Evaluation on the privacy for 8 clusters. Unseen shows the results only for the first 3 persona labels that defender has never seen. Overall refers to the results on all 8 labels. Acc and Max-Ratio are measured in %. BP<sub>u</sub> corresponds to Bayesian Privacy loss on the uniform distribution. Still, the worse the attack model performs, the better privacy protection can be achieved.
the second speaker (Human  $B$  in Figure 1) with all previous turns as context. Then we compared the generated model outputs with ground truth replies. We use Dist-1 and Dist-2 to count ratios of distinct unigrams and bigrams. BLEU-1, BLEU-2 and BLEU-4 are applied to evaluate generation similarity with ground truth. Due to the one-to-many nature of chit-chats, the BLEU is not adequate to compare generated responses with ground truth. Hence, we adapt Precision, Recall and Precision of BERTScore to measure the similarity in the embedding space.
The evaluation result is shown in Table 3, where same models from Table 2 are evaluated. The result indicates that adding KL loss will increase the perplexity greatly from 14.8 to 28.9. After combining KL loss with MI loss, its perplexity decreases to 19.674. A plausible explanation is that KL loss confuses the persona predictor and indirectly increases the uncertainty of the GPT-2. All GPT-2 models have relatively low BLEU scores due to the one-to-many mapping between contexts and responses. For Distinct and BERTScore, there are only minor differences between LM and defended LMs. Though the uncertainty increases after applying KL loss and MI loss, it does no harm to the quality of generation. In summary, there is almost no negative influence on the utility after applying the proposed defense strategies.
5.5 More Setups on Attacks
Attacks on Imbalanced Data Distribution. Previous black-box attacks usually assume that the annotated dataset  $D_{a}$  must share similar data distri
bution with the defender's training data. To examine the performance of defense strategies on unseen personas, we assign the adversary's dataset  $D_{a}$  with labels that the defender cannot acquire. We split data with 500 persona labels that are uniquely held by the adversary. The defender owns 8,031 conversations with persona labels ranging from 500 to 4,331 while the adversary holds 2,376 dialogues with persona labels ranging from 0 to 4,331. For testing, 500 conversations with persona labels ranging from 0 to 4,331 are used.
Under imbalanced data distribution, the attack on the defended LM has Acc  $0.47\%$ , F1 1.90e-3 and Max-Ratio  $94.06\%$ . The persona inference accuracy is still very low and the attacker model tends to predict more on a single persona label than the balanced data distribution setup. This result shows that the proposed overall loss can also prevent black-box persona inference attacks on unseen personas. It also verifies previous suggestions that combining LM loss with MI loss may fool the attacker model to make wrong predictions.
Attacks on Fewer Persona Labels. The above experiments are based on 4,332 persona labels. In fact, many personas share similar meanings and can be further clustered. Besides, to better evaluate privacy loss for the estimated distribution, a smaller label space is preferred. Therefore, it is necessary to consider defense performance on a smaller label space. We use Sentence-BERT (Reimers and Gurevych, 2020) to embed all persona sentences and perform k-means clustering on the embeddings to obtain 8 clusters. We manually checked these clusters and classified them as cars, food, animals
Figure 3: Black-box persona inference attacks on chit-chats between the user (the authors of this paper) and chatbots. For both conversations, the "context" is fixed and used as the first four utterances. Then the bot and the user start interactive conversations with the "context". Since there is no gold standard, the results are annotated by the authors.
(pets), family information, hobbies, jobs, personal information and music tastes respectively. To evaluate how the clustering performs, we randomly sample 100 utterances with clustered labels and invite two volunteers to inspect those samples. Both of them agree on  $90\%$  of the clustered annotations. After manual inspection of the remaining  $10\%$  annotations, the clustering error rate is  $8\%$ . Following previous imbalanced data split, we assign data in the first 3 clusters only to the adversary to make the data distribution imbalanced. Here, the defender owns 6,654 conversations with persona labels ranging from 3 to 7 while the adversary holds 3,753 dialogues with persona labels ranging from 0 to 7. For testing, 500 conversations with persona labels ranging from 0 to 7 are used.
The attacking performance for both unseen labels and all labels is displayed in Table 4.  $\mathrm{BP}_u$  measures the KL divergence  $D_{KL}(F_0||\mathcal{A}(f(u)))$  where  $F_0$  refers to uniform distribution. For imbalanced data distribution with a small label space, our proposed defenses can still achieve much lower attack accuracy than LM on both Unseen and Overall. However, for Overall,  $LM + KL + MI$  has higher accuracy with a lower F1-score compared with two baselines. This indicates that proposed defenses fail to protect privacy as we desired in the baselines. For  $\mathrm{BP}_u$ ,  $LM + KL + MI$  are around 10 times smaller than LM. It means that after applying defense objectives, the attacker's estimated distribution is much closer to the uniform distribution. Thus the effectiveness of the KL loss is verified. In addition, Max-Ratio with 8 clusters on Unseen is smaller than 4,332 labels even though the distribution of 8 clusters is obviously tighter. Still, the Max-Ratio of  $58.15\%$  accounts for a much larger fraction than other predictions. In summary, the above results imply that for the smaller label space, our proposed defense objectives are still effective even on unseen
persona labels.
In Figure 3, we give an example of the persona inference attack, where conversations are generated between the chatbot and the user with the given context. We manually mark True/False on the predicted results. As shown in the figure, there are several successful attacks on LM and no correct prediction on the defended LM. For attacks on LM, speakers' hobbies and jobs can be inferred. For incorrect predictions, the attacker model can still predict context-aware personas. After applying proposed defense learning strategies, the predicted personas become irrelevant with context and mostly predict "My favorite color is blue." In fact, it is the most frequent prediction for  $LM + KL + MI$  over 4,332 persona labels. This attack example illustrates that our defense objectives can prevent the black-box persona inference attack from inferring relevant personas.
In this paper, we show that LM-based chatbots tend to reveal personas of speakers and propose effective defense objectives to prevent GPT-2 from overlearning. Unlike other works that suffer from utility degradation, our defense learning strategies do no harm to the powerful generation ability of LM-based chatbots. We conduct extensive experiments to evaluate both privacy and utility. We perform black-box persona inference attacks under various setups to demonstrate the robustness of proposed defense learning strategies. In addition, we use automatic metrics to show that proposed defense learning strategies maintain the utility. For future work, we suggest working on flattening the distributions of attacker models.
We declare that all authors of this paper acknowledge the ACM Code of Ethics and honor the code of conduct. This work essentially considers black-box attacks on the private persona attributes and proposes effective learning strategies to prevent chatbots from overlearning private personas.
Dataset. During our dataset collection, all the conversations and personas are collected from publicly available datasets including PersonaChat and DNLI. All the speakers are anonymized and no identifiable personal information is included.
Model. For training our LM-based chatbots, we follow standard methods. We are well aware of the bias issue inside current language models. In the future, if there are other fairer language models, we will extend our defenses on them.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In Proceedings of USENIX Security Symposium, pages 2633-2650.
Maximin Coavoux, Shashi Narayan, and Shay B. Cohen. 2018. Privacy-preserving neural representations of text. In Proceedings of EMNLP 2018, pages 1-10.
Yanai Elazar and Yoav Goldberg. 2018. Adversarial removal of demographic attributes from text data. In Proceedings of EMNLP 2018, pages 11-21.
Hanlin Gu, Lixin Fan, Bowen Li, Yan Kang, Yuan Yao, and Qiang Yang. 2021a. Federated deep learning with bayesian privacy. arXiv preprint arXiv:2109.13012.
Jing Gu, Qingyang Wu, Chongruo Wu, Weiyan Shi, and Zhou Yu. 2021b. PRAL: A tailored pre-training model for task-oriented dialog generation. In Proceedings of ACL 2021, pages 305-313.
Donghoon Ham, Jeong-Gwan Lee, Youngsoo Jang, and Kee-Eung Kim. 2020. End-to-end neural pipeline for goal-oriented dialogue systems using GPT-2. In Proceedings of ACL 2020, pages 583-592.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In Proceedings of ICLR 2020.
Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron Wallace. 2021. Does BERT pretrained on clinical notes reveal sensitive data? In Proceedings of NAACL 2021, pages 946-959.
Ang Li, Yixiao Duan, Huanrui Yang, Yiran Chen, and Jianlei Yang. 2020. Tiprdc: Task-independent privacy-respecting data crowdsourcing framework for deep learning with anonymized intermediate representations. In Proceedings of the ACM SIGKDD 2020, page 824-832.
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of NAACL 2016, pages 110-119.
Yitong Li, Timothy Baldwin, and Trevor Cohn. 2018. Towards robust and privacy-preserving text representations. In Proceedings of ACL 2018 (Volume 2: Short Papers), pages 25-30.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of ICLR 2019.
Mohammad Malekzadeh, Anastasia Borovykh, and Deniz Gündüz. 2021. Honest-but-curious nets: Sensitive attributes of private inputs can be secretly coded into the classifiers' outputs. In Proceedings of the ACM CCS 2021.
Fatemehsadat Mireshghallah, Huseyin Inan, Marcello Hasegawa, Victor Ruhle, Taylor Berg-Kirkpatrick, and Robert Sim. 2021. Privacy regularization: Joint privacy-utility optimization in LanguageModels. In Proceedings of the NAACL 2021, pages 3799-3807.
Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. 2020. Privacy risks of general-purpose language models. In Proceedings of 2020 IEEE Symposium on Security and Privacy (SP), pages 1314-1331.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of ACL 2002, pages 311-318.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In Proceedings of EMNLP 2020, pages 4512-4525.
Karin Sevegnani, David M. Howcroft, Ioannis Konstas, and Verena Rieser. 2021. OTters: One-turn topic transitions for open-domain dialogue. In Proceedings of ACL 2021, pages 2492-2504.
Weizhou Shen, Junqing Chen, Xiaojun Quan, and Zhixian Xie. 2021. Dialogxl: All-in-one xlnet for multi-party conversation emotion recognition. In Proceedings of the AAAI 2021, volume 35, pages 13789-13797.
Congzheng Song and Vitaly Shmatikov. 2020. Overlearning reveals sensitive attributes. In Proceedings of ICLR 2020.
Jiaming Song, Pratyusha Kalluri, Aditya Grover, Shengjia Zhao, and Stefano Ermon. 2019. Learning controllable fair representations. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89, pages 2164-2173.
Anna Tigunova, Andrew Yates, Paramita Mirza, and G. Weikum. 2019. Listening between the lines: Learning personal attributes from conversations. In Proceedings of WWW 2019, page 1818-1828.
Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. 2019. Dialogue natural language inference. In Proceedings of ACL 2019, pages 3731-3741.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint, arXiv:1901.08149.
Chien-Sheng Wu, Andrea Madotto, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2020. Getting to know you: User attribute extraction from dialogues. In Proceedings of LREC 2020, pages 581-589.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Proceedings of NIPS 2019, pages 5754-5764.
Santiago Zanella-Béguelin, Lukas Wutschitz, Shruti Tople, Victor Ruehle, Andrew Paverd, Olga Ohriimenko, Boris Köpf, and Marc Brockschmidt. 2020. Analyzing information leakage of updates to natural language models. In Proceedings of ACM CCS 2020.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of ACL 2018, pages 2204-2213.
Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proceedings of ICLR 2020.
Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT: Large-scale generative pre-training for conversational response generation. In Proceedings of ACL 2020: System Demonstrations, pages 270-278.
Yukun Zhu, Ryan Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of ICCV 2015, pages 19-27.
Figure 4: The test set data distribution (Data) and average estimated distributions of persona predictors  $(\mathcal{A}$  and  $\mathcal{A}_p)$  over imbalanced 8 clusters.
A Training details.
For each conversation, the utterances are concatenated by the special token “<lendoftext|>” to train the GPT-2. To decode outputs from GPT-2, we apply the Nucleus Sampling (Holtzman et al., 2020) method. We set top-p = 0.9 with a temperature coefficient 0.9 to sample words from the GPT-2. For optimization, we set 2 AdamW optimizers (Loshchilov and Hutter, 2019) for the chatbot and the persona predictor respectively. The learning rate is 3e-5 with linear warm-up and decay. For hyper-parameters, we set  $\lambda_0 = 1$ ,  $\lambda_1 = 10$  and  $\lambda_2 = 1$ .
B Comparison of Internal Distribution between  $\mathcal{A}$  and  $\mathcal{A}_p$
To make predictions on personas, the arg max function is used for the estimated distribution of persona predictors. However, the internal distribution conveys crucial information about how the persona predictors estimate  $f(u)$ . We follow the setup of imbalanced data split of 8 clusters in Section 5.5 to examine persona predictors of attacker  $\mathcal{A}$  and fake attacker  $\mathcal{A}_p$ .
Figure 4 shows the data distribution of the test set and average distribution after softmax activation over the 8 labels for attacker  $\mathcal{A}$  and defender  $\mathcal{A}_p$ . For attacker  $\mathcal{A}$ , we consider the attack on  $LM$  and  $LM + KL + MI$ . The defender  $\mathcal{A}_p$  tends to have a large difference with Data and tries to flatten its distribution among its own training set (the last 5 labels). This behavior conforms to the KL loss's objective that aims to flatten the distribution and deviate from the ground truth distribution. For attacker  $\mathcal{A}$ , distributions of both  $LM$  and  $LM + KL + MI$  seem close to the ground truth distribution. This indicates that the attacker model  $\mathcal{A}$  can still learn statistical in
Figure 5: Dialog generation example on Human B. All previous utterances between A and B are used as context to generate responses.
formation about personas. However, its attacking performance is poor. The poor performance implies our proposed defense learning strategies may obfuscate Attacker for estimating single sample  $f(u)$  and finally make the wrong prediction.
C More on Case Study
C.1 Example of Generation
To show an intuition view on utility, we provide one generation sample shown in Figure 5. Both  $LM$  and  $LM + KL + MI$  are able to generate fluent and proper relies. Moreover, they tend to maintain coherence with previous contexts. For example, it is mentioned in the context that Human  $B$  is a vegan and both chatbots respond that they do not eat meat for the food preference. This generation example shows that proposed defense learning objectives preserve the model utility.
C.2 More Examples of Persona Inference
Here, we give two more examples of the persona inference attacks in Table 6. The first example shows one successful defense. For the second example, both attackers with and without defense fail to predict the ground truth persona. Still, we can see that  $LM + KL + MI$  predicts personas that are irrelevant
Top-1	Top-3	Top-5	Top-10	Top-50	Top-100	Top-500	Top-2000
LM	37.59	55.57	63.28	72.76	87.19	91.54	97.79	99.60
LM+KL+MI	0.53	1.80	2.24	3.20	8.64	12.10	30.57	80.22
Table 5: Evaluation on the top-k accuracy over 4,332 clusters.
Dialog Context 1
Human A: Hi how are you doing?
Human B: I am great and you?
Human A: I am great, just reading.
Human B: I am listening to the rolling stones, I love them.
Human A: Is that your favorite band?
Human B: Yes it is. I am working right now too.
Human A: Where do you work at?
Human B: IBM in Chicago, what about you?
Persona Prediction:
LM: I currently work for IBM in Chicago.
LM+KL+MI: I love cats. X
Dialog Context 2
Human A: Hello there my name is Dr.Lucy. How are you?
Human B: I am great, loving this city life, how are you?
Human A: I am well thank you. I miss my country life in Spain.
Human B: My older brother lives in Spain, how is it?
Human A: It is beautiful. I hope to take my family back there.
Human B: Yes, maybe i will take my girlfriend that I love there one day.
Human A: Oh, how long have you two been together?
Human B: Very long, she was with me when I colored my hair pink.
Human A: That is awesome. What type of music do you two listen to?
Human B: I like reading music, what about you?
Human A: Hip hop is my favorite. Do you play an instrument?
Persona Prediction:
Ground truth: My favorite music is hip hop.
LM: I know how to play the guitar. X
LM+KL+MI: My favorite food is pizza. X
Table 6: More persona inference attack examples. The embeddings of the final utterance with orange color are used for inferring B's persona.
to the context. However,  $LM$ 's output "I know how to play the guitar." is much closer to the context about music and instruments. Without any defense, the above examples show that the attacker model can still predict context-aware personas even if its predictions are wrong. After applying the proposed defenses, the attacker model cannot predict meaningful personas relevant to the context.
D Evaluation on Top-k Accuracy
Previous experiments mainly consider accuracy as the evaluation metric. In this section, we use top-k accuracy for the black-box persona inference attacks to measure privacy protection. As shown in Table 5, our defense is much more robust than  $LM$