Domain Private Transformers for Multi-Domain Dialog Systems
Authors: Anmol Kabra
Ethan R. Elenberg
Venue: EMNLP 2023 Findings (2023)
Keywords: differential privacy
language models
dialogue
multi-domain language models
TL;DR: We propose “domain privacy” to address cross-domain leakage (language models when prompted with text of one domain generating sensitive text of others), and use membership inference attacks to measure domain privacy of various models.

Abstract: Large, general purpose language models have demonstrated impressive performance across many different conversational domains. While multi-domain language models achieve low overall perplexity, their outputs are not guaranteed to stay within the domain of a given input prompt. This paper proposes \emph{domain privacy} as a novel way to quantify how likely a conditional language model will leak across domains. We also develop policy functions based on token-level domain classification, and propose an efficient fine-tuning method to improve the trained model's domain privacy. Experiments on membership inference attacks show that our proposed method has comparable resiliency to methods adapted from recent literature on differentially private language models.

Domain Private Transformers for Multi-Domain Dialog Systems
Anmol Kabra *
Ethan R. Elenberg ASAPP Inc New York, New York
Toyota Technicological Institute Chicago, IL anmolêttic.edu
sitenberggasapp..om com
Abstract
generation: models when prompted with text from one domain must not generate sensitive text from other domains. Safe generation is a key require- ment for model providers who pool datasets from many contracted companies- company might require the model to not generate their sensitive text when prompted with text from other compa- nies. We call such safe generation domain privacy. Let {di..... dy} be domains from which datasets are created. Let MD be model trained on text dataset D. To verify MD is domain private for domain d,, we can prompt the model with contexts {G: from domain d, and check if the generations contain sensitive text of domains dj for j + i.
Large, general purpose language models have demonstrated impressive performance across many different conversational domains. While multi-domain language models achieve low overall perplexity, their outputs are not guar- anteed stay within the domain of given input prompt. This paper proposes domain pri- vacy as a novel way quantify how likely conditional language model will leak across do- mains. We also develop policy functions based on token-level domain classification, pose efficient fine-tuning method to improve the trained model's domain privacy. Experi- ments on membership inference attacks show that our proposed method has comparable re- siliency to methods adapted from recent litera- ture differentially private language models.
pro-
Our contributions are: 1) define domain pri- vacy as new property for contextual language models, 2) propose fine-tuning algorithms that trade off domain privacy for model performance, and 3) conduct extensive experiments for text gen- eration with multi-domain datasets. Domain pri- vacy scales well with the number of domains, and allows for flexible definitions of domain-sensitive text. Our proposed fine-tuning algorithms utilize differentially-privaa training attain domain pri- vacy, while achieving good performance.
1 Introduction
Large language models have enabled significant progress in machine learning and NLP across wide range of tasks and domains (Bommasani et al., 2021). They perform especially well in settings where little training data is available for new do- mains of interest. popular approach in such set- tings is transfer learning: fine-tune a pretrained model on data from specialized domains (Howard and Ruder, 2018; Zhang et al., 2021; Yang et al., 2021; Budzianowski and Vulić, 2019; Hosseini- Asi et al., 2020). Here, performance is typically measured in perplexity (or task- task-specific metrie) for each new domain while controlling for model complexity or data (Gururangan al., 2022).
Related Work
Domain Adaptation Large pretrained language models have been shown to achieve good perfor- mance when fine-tuned on small datasets from new domains (Gururangan et al., 2020). To improve ef- ficiency, recent multi-domain approaches leverage multitask learning (Lin et al., 2020), model distilla- tion (Yao et al., 2021), and/or meta-learning (Pan et al., 2021). Hu et al. (2019) propose private meta- learning for discriminative tasks; work is the first for private multi-domain text generation.
We introduce novel definition of privacy for contextual language models to enforce that text prompts from one domain do not leak sensitive text of other domains. Practitioners train genera- tive models on datasets often curated from diverse domains, e.g. news article categories dialog tasks. Model users often then interested in safe Work done at ASAPP Inc
Differentially Private Language Models Dif- ferential privacy is a powerful framework that pro- vides rigorous guarantees on training data expo-

---

sure to adversaries (Dwork and Roth, 2014). Re- cent work (Yu et al., 2022; Li al., 2022) de- scribes differentiallly-riieee
applicable in practice (Dwork al., 2019). Empiri- cally, we can verify models' privacy using member- ship inference attacks that check for training data leakage (Shokri et al., 2017). For generative mod- els, these attacks check if models generate training text when prompted (Carlini et al.. 2021).
fine-tuning for large language models like GPT-2 (Radford et al., 2019), albeit on data from a single domain. However, standard notions of differential privacy, including those for single-domain language models (Ginart et al., 2021; Shi et al., 2022b,a), are insufficient for multi-domain language modeling. Firstly, they too restrictive as privacy guarantees must hold uni- formly for all test inputs, regardless of how often they appear in the current domain. Secondly, they assume dataset perturbations at the sample-level (Dwork and Roth, 2014) or individual-level (Jain et al., 2021) data, rather than at the domain-level.
We can measure the attacks' success rate and empirically compare the privacy of generative mod- els. Likelihood Ratio (LiRa) membership infer- ence attacks compare targer models relative a reference model (Carlini et al., 2022, 2021). LiRa attacks work as follows: (i) prompt a tar- get model with contexts {a} to generate text {a:), (ii rank {mi} by generation likelihood Parraa (1a PPtp:/PPPrff/ppp with the highest ratios. If these 2; contain sensitive text then the target model is said to leak and the attack deemed successful. Finally, we can com- pare target models by their LiRa attack success rate #success
and (iii) select
3 Preliminaries
We recall few definitions first. See Appendix A for further details.
Language modeling Given a text sequence of to- kens Ti = (f1....,f,, f.); an autoregressive language model estimates next token probability Pr[fx+1| [T)). The model is trained by minimizing cross-entropy between next ground-truth token and model's ' pre- dictions. Finally, we can use the model to compute the perplexity (PPL) of a sequence Tr.
4 Domain Privacy
Consider two domains de and d where * j. The goal of domain privacy is to check how likely a model is to generate sensitive text from domain dj when prompted with text from domain d;. To check if text contains private tokens of domain dj. we can use policy function Fj. Since domains d, and d, could have inherent overlap. e.g. polities and sports news overlapping due geopolities, we will use MD, as a reference model where DD = D \ dj is the dataset obtained by removing text of domain d, from D. The likelihood of MTJ, leaking sensitive text of dj serves as an upper bound for the target model leakage. Here D and D, are neighbors at domain level w.r.t. Fj as they differ in one domain.
Privacy An algorithm is differentially private if it is not too sensitive the differentiating element in two neighboring inputs. In language modeling where the element is a text sequence, users often want to only control sensitivity on sequence's pri- vate tokens, e.g. phone numbers and proper nouns. Shi et al. (2022b) thus define Selective Differential Privacy using policy functioons
A policy function annotates a sequence Tn with 0-1 labels; F(T,); = if the 4th token is private and 0 if public. F then defines neighboring text sequence datasets: D' is called an F-nnighborr of D, i.e. D' (D), if they differ in exactly one text sequence on which F does not agree.
Definition 4.1 (Domain Privacy) Let C' > O be a parameter. A model MD is C-domain-prived for D. if for all i, E [N] where j + i, contexts { from domain di
Pr[Mp(a) E dil [M,
Definition 3.1 (Selective Differential Privacy) Given a policy function F, training algorithm A with range M is (F, e, 6))- differential private for all D E D, D' e (D(D). M M,
Domain privacy captures the need for safe gen- eration: inter-domain private generation and intra- domain public generation. It extends Selective Dif- ferential Privacy in three ways. Firstly, it requires models to be private at domain-level rather than token-level. Secondly, allows models to generate sensitive text of d; when prompted with {c²} -olly leaking text of other domains is restricted. Finally,
Pr(A(D) E M < e Pr[A(D') E M| [
Membership Inference Attacks Differential pri- vacy gives theoretical guarantees which may not be

---

domain privacy uses LiRa membership inference attacks; Selective Differential Privacy lacks this. Hence, domain privacy can empirically tested.
weights initialize model, which is then fine- tuned with DP-AdamW on non-redacted data, Shi et al. (2022a) show that the model adapts to the linguistic style without generating sensitive tokens.
5
Methodology
We adapt this two-stage process into a one-stage one: initially fine-tune on redacted data and grad- ually transition to non-redacted data (Redaction Schedule). A redaction schedule determines this transition according a parameter p that decreases from to 0 during fine-tuning. At every step during fine- fine-tuning, with probability p we fine-tune with AdamW on redacted data, and with probability 1-p we fine-tune with DP-AdamW W on non-redacted data, This one-stage process has half the training cost of JFT, but still many of its benefits.
Next we study domain privacy applied to the prob- lem of generating dialog text.
5.1 Policy Functions
A policy function flags text considered sensitive for domain, enabling to check for domain privacy. We use policies in two ways: (i) create redacted datasets for fine-tuning target models (replacing sensitive tokens with <REDACTED> tokens), and (ii) to check if generations leak sensitive text during LiRa attacks. We deseribe data-driven policies be- low; one could also use rule-based ontologies.
6 Experiments
6.1 Datasets
The Keyword Detection policy checks if any tokens in text are in set of hand-picked key- word tokens K; sensitive to domain d,. Formally, pkegword (T)
We use the MultiDoGo dataset (Peskov al., 2019), which consists of task-oriented dialogs of user- agent customer service simulation from 6 domains. We use the largest domains: AIRLINE (air travel bookings), MEDIA (telecommunication and cable), and INSURANCE (policy modifications). We prepro- cess the dataset by adding control tokens to each di- alog, such as speaker tokens, start-of-connvesaatin <_soc_>, an end-of-conversattion <_eoc_>, and domain-name (e.g. <AIRLINE>). Appendix C in- cludes further preprocessing details and examples from redacted and nonrredacted dataset versions.
if there exists token E KG, This is compatible with defining domains based on n-gram overlap (Gururangan et al., 2022). The Sequence Classification policy uses a con- textual RoBERTa model harRr (Liu et al., 2019) fine-tuned to predict the domain from (a sequence of) tokens. We use a specified threshold to set if there exists token sequence t CT such that Pr(hBERT((*) d.] > 2.
F.
with
PBERT (T)
5.2 Target Models
6.2 Training target models
There has been much work protect against mem- bership inference attacks. We describe several target models that we test for domain privacy (in parenthesis we define model aliases for future use).
We create 60-20-20 train-vate validation splits for each domain, and coalesce similar splits. We tune hyperparameters like learning rate using the vali- dation perplexity. The threshold for RoBERTa policy is set by maximizing the difference of LiRa success rate between DOMAIN; Only and Public models (recall the rate #success / non-empty- generations). To get the target models in Sec- tion 5.2, we fine-tune a pretrained GPT-2 check- point on data from all domains. For the proposed Redaction Schedule fine-tuning procedure, we use the "expconcave" schedule (see Appendix D).
Let D be the dataset and di be the domain being tested. As baseline target, we use model fine- tuned only text from D n d; (DOMAIN; Only). All non-baseline target models are fine-tuned on either a redacted version of or the non-redacted version. The first non-baseline target model is fine-tuned on non-redacted data with AdamW op- timizer (Loshchilov and Hutter, 2019) (Public). The second is fine-tuned on redacted data instead (Pub+Redacted). Li et al. (2022) recently pro- non-redacted
6.3 LiRa Attacks for MultiDoGo dataset
posed optimizing transformers data with AP-A W (Private), differentiall-- private variant of AdamW. Shi et al. (2022a) opti- mize for Selective Differential Privacy with "Just Fine-tune Twice" (JFT) procedure: fine-tune a model with Adam W on redacted data and use the
We conduct LiRa attacks on each target model test for domain privacy-wee we check if a model leaks sensitive text of domain d, when prompted with
'Code is available at https://github.coo com/ asappresarchh/domiii-rrre serrivaattetrannffomeerss

---

contexts from domain di + j. Here we focus on i = AIRLINE. Into each model, we feed 100 prompts from the INE domain and generate 10 different outputs for each prompt. We use the control tokens as generation-stoopinn--rrreiiiie and suppress generating <REDACTED> tokens. See Ap- pendix E for results other domains, LiRa attack examples, and example model generations.
for Redaction Schedule. We also test running each stage of JFT for half the number of steps, i.e. with total compute comparable to other models.
ARLIIE, Keyword Rostection
0.0
2xx
7.0
ARR NE
Pub
JTTaat (rdded)
6.4 Results
RupeRddatodd
JTTeall (rteed)
0.5
Privtt
Recacaon Schwoule (me1 (nda-))
J 3-T0-30-4)
Redaction Scheoll Iy Infwe))
AIRLINE, Kepword Raddecion
6.0
0.08
AETLINE Onty
2TT (1-bo-)
PJD
0.37
PuDi Redacied
Dactactor Scheols (1e10) inde-
0.00
Pecuciion Schtoaue (9112 pnbb S1
0.03
Telt Parplccly
AIRUNE, RoBERTA
0.04
8.0
200
2x
0.03
0.02
7.0
2
2x
0.02
8.5
(inted)
Privtt
Text Parplcily
Recectio Schoollo (n-18) 1-50
ABR INT, RedERRA Redection
Only
(1
0.32
Son Bohoolle (9-16, I:56
Perslecty
0.23
Figure 2: Renyi DP vs test perplexity. Lower better for both axes. 2x indicates double training cost.
0.23
2x
Figure 2 shows Renyi DP guarantee? vs. test perplexity for each model. Public has no pri- vacy guarantee (c oc), and (Pub+Redacted) has an ideal guarantec of 0 as it is only fine- tuned on redacted data. We further see that for both keyword and RoBERTa redaction policies. Redaction Schedule models have privacy guar- antees s 35% better than JFT. We observe that vanilla fine-tuning like Public is insufficient for domain privacy. Domain privacy becomes feasible with fine-tuning algorithms designed for Selective Differential Privacy; these algorithms fine-tune par- tially on redacted datasets built with policies.
0.55
2x
Thet Parpinsiyy
Figure I: LiRa success rate vs test perplexity. Lower better for both axes. 2x indicates double training cost,
We compare target models on LiRa success rate and test perplexity metrics. Figure shows these two metrics for each target model. LiRa attacks are more successful w.r.t. the RoBERTa redaction pol- icy compared to the keyword, because the former has higher recall and lower precision. Focusing on RoBERTa policy, all models but Private and Pub- li fine-tuning have LiRa success rate lower than the AIRLINE Only baseline. While having compa- rable domain privacy, JFT has better perplexity and Redaction Schedule has worse perplexity when compared to Pub+Redacted. Domain leakage is generally more sensitive to learning rate for JFT, while perplexity is more sensitive to learning rate
7 Conclusions
This paper compares multi-domain language mod- els for dialog data on a new concept of do- main privacy. We propose two policies for redacting domain-sensitive tokens, enabling recent
'Renyi DP is commonly used to evaluate differential pri- vacy of gradient-doscent- doscent-based optimizers (see Appendix A).

---

differentially--riieeee training algorithms be used for preserving domain privacy. Future research di- rections include studying the domain privacy prop- erties of additional training strategies, and under- standing the interplay between domain privacy and performance on downstream tasks.
Bosselut, Emma Brunskill, al. 2021. On the op- portunities and risks of foundation models. arXiv preprint 1xXi::200..02255
Pawel Budzianowski and Ivan Vulić. 2019. Hello, it's gpt-2 - how can i help you? towards the use of pre- trained language models for task-oriented dialogue systems. In EMNLP Workshop Neural Generation and Translation.
8 Limitations
Nicholas Carlini, Steve Chien, Milad Nasr. Shuang Song, Andreas Terzis, and Florian Tramer. 2022. Membership inference attacks from first principles. In 2022 IEEE Symposium on Security and Privacy (SP), pages 1897-1914. IEEE.
Sequence classification policies are more suscep- tible to data bias and systemic uncertainty than rule-based policies that are based on keywords or parts of speech. While our policy functions more general than previous work, they can only approximate human subjectivity implicit in mark- ing tokens as domain-sensitive. Additionally, not clear how our definition of domain privacy is amenable to theoretical properties that differential privacy provides, such as composability and group privacy. LiRa attacks are one natural tool to check inter-domain leakage in contextual language mod- els; other tools be developed to either certify domain privacy guarantces or check for domain privacy violations.
Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Ex- tracting training data from large language models. USENIX Security Symposium, pages 2633-2650.
Cynthia Dwork, Nitin Kohli, and Deirdre Mulligan. 2019. Differential privacy in practice: Expose your epsilons! Journal of Privacy and Confidentiality, 9(2).
Cynthia Dwork and Aaron Roth. 2014. The algorith- mic foundations of differential privacy. Founda- tions and Trends@ in Theoretical Computer Science, 9(344)22114001
9 Ethics/Impact
Models that are not domain private pose security risk in deployment due to inter-domain leakage. We show that the predominant transfer learning ap- proach, which fine-tunes a single pretrained model on data from several new domains, is risky from a leakage standpoint. We show how membership inference attacks can target models to leak training data, and note that these attacks can be extended real-world models trained on proprictary data. The data collection agreement used in one domain could forbid the use of data for any other purpose. e.g. generation for any other domain. While this was not an ethical concern for the data used in this paper, it remains an open area of discussion for the ML community.
Antonio Ginart, Laurens van der Maaten, James Zou, and Chuan Guo. 2021. Submix: Practical private prediction for large-scale language models. arXiv preprint arXiv:2201.0097.
Suchin Gururangan, Mike Lewis. Ari Holtzman. Noah A. Smith, and Luke Zettlemoyer. 2022. Demix layers: Disentangling domains for modular language modeling. In NAACL, pages
Suchin Gururangan. Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In ACL.
Ehsan Hosseini-Asl, Bryam McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A sim- ple language model for task-oriented dialogue. In NewrIPS.
A Akknowledgements
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL.
We would like to thank Ryan McDonald, Kilian Q Weinberger, and the rest of the ASAPP Research team for their helpful discussions.
Shengyuan Hu, Zhiwei Steven Wu, and Virginia Smith. 2019. Private multi-task learning: Formulation and applications to federated learning. arXiv preprint arXiv:2108. 12978.
References
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine
Prateck Jain, John Rush, Adam Smith, Shuang Song, and Abhradeep Guha Thakurta, 2021. Differentially private model personalization. In NewrIPS.

---

Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. 2022. Large language models can be strong differentially private learners. In International Conference on Learning Representations.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulka- mi, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. 2022. Differentially private fine-tuning of lan- guage models. In International Conference on Learn- ing Representations (ICLR).
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020. Exploring versatile generative language model via parameter-effiiient transfer learning. In Findings of EMNLP.
Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q. Wein- berger, and Yoav Artzi. 2021. Revisiting few-sample bert fine-tuning. In ICLR.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zeullemyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini arXiv: 1907. 1669..
A Further Definitions
Language Modeling text sequence on (w.r.t an autoregressive language model) is defined as:
The perplexity (PPL) of a
Nya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization, In International Confer- ence on Learning Representations.
n
PPL(Tn)) = exp
log (og(Pr(kH)(m)))
Ilya Mironov. 2017. Rényi differential privacy. 2017
1=1
IEFE 30th computer security foundations symposium (CSF), pages 263-275. IEEE.
Privacy Let A: X + y be a randomized algo- rithm. Two input sets X, X' S X are neighbors if they differ in exactly one element. Dwork and Roth (2014) define Differential Privacy for A as follows.
Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. 2021. Meta-kd: meta knowledge distillation framework for language model compression across domains. In ACL.
Definition A.1 (Differential Privacy) Algorithum A: X + y is (e, (eiff-rrireyy private for all outputs Y C y, neighboring sets X,X' X
Denis Peskov, Nancy Clarke, Jason Krone, Brigi Fodor, Yi Zhang, Adel Youssef, and Mona Diab. 2019. Multi-domain goal- goal-oriented dialogues (MultiDoOO:: Strategies toward curating and annotating large scale dialogue data. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP--IJNT) pages 4526-4536 4536. Association for Computational Linguistics.
Pr(A(X) E S (P(.((()) EY)+
To hone in on the private tokens of a text se- quence, Shi et al. (2022b) introduce Selective Dif- ferential Privacy, which uses policy functions to define neighboring datasets.
Definition A.2 (Policy Function) A policy func- lion F T - {0,1)" annotates tokens in se- quence Tn T as private or not. F(Fn)i = the ith token is private and 0 public.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Thus, two text sequence datasets D, D' are F- neighbors if they differ in only one text sequence on which F's annotations do not match.
Weiyan Shi, Si Chen, Chiyuan Zhang, Ruoxi Jia, and Zhou Yu. 2022a. Just fine-tune twice: Selective dif- ferential privacy for large language models. arXiv preprint arXiv:2204.0766
Mironov (2017) show interchangeability be- tween Renyi differential privacy and differential privacy, i.c. an algorithm satisfying (a, Renny differential privacy satisfies (cs, difffrreett pri- vacy for any 5 E 10, and vice versa. Renyi differential privacy is defined follows.
Weiyan Shi, Aiqi Cui, Evan Li, Ruoxi Jia, and Zhou Yu. 2022b. Selective differential privacy for langauage modeling. In NAACL, pages 2848-2859
Reza Shokri, Marco Stronati, Congzheng Song, and Vi- taly Shmatikov. 2017. Membership inference attacks against machine learning models. In 2017 IEEE sym- posiam on security and privacy (SP), pages 3-18. IEEE.
Definition A.3 (Renyi Differential Privacy) Algorithm A: X > y is said have e-Reyy dif- ferential private of order o, for all neighboring sets X, X'x,,
Yunyi Yang, Yunhao Li, and Xiaojun Quan. 2021. Ubar: Towards fully end-to-end task-oriented dialog sys- tems with gpt-2. In AAAL
(A((() 1I A(X)))) Se
Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, and Furu Wei. 2021. Adapt-and-disit Developing small, fast and effective pretrained language models for domains. In Findings ACL.
where D.(P || Q) is the Renyi divergence (of order a > 1) of two distributions P, over Y
defined (P || Q) - log Eay-Q Qiv)
P(y)

---

B Computation
All language models were fine-tuned from pub-
Prob. of training redectidd
0.8
li GPT-2 small checkpoint with 124M parame-
ters (Radford et al., 2019). Model training was done on a server with one A10G Tensor Core GPU and 24 GB GPU memory, which took approxi- mately hours per model.
0.6
0.2
0.0
10
C Data Preprocessing and Experimental setup
Epoch (C)
exprervex, 17mh
n=10
expoorvek, (pm 15
expconcave, n-11
dialogs from AIRLINE, MEDIA, and INSURANCE domains from the MultiDoGo dataset. These domains have 15k, R33k, and #11k dialogs respectively.
As mentioned earlier,
Figure 3: Redaction schedules specify the probability of choosing to fine-tune on redacted data at every training step. the total number training epochs is = 10, (i) the linear schedule decays I -t,"T, (ii) the expconvex schedule decays exp(-n--d t), and (iii) the expeoncave schedule decays as exp( ( (1 T)) where v temperature parameter.
We preprocess dialog samples as follows. Con- sider sample "SYS: Hello, you are connected to LMT Airways! How may help you? USR: Change my seat assignment SYS: dialog sample by adding conveet-oonvesstiion con- trol token <_soc_>, end-of-conversatiin control token <_eoc_?, and domain-name control token <AIRLINE> before every utterance. A dialog then looks like "<_soc_> <AIRLINE> SYS: Hello, you are connected to LMT Airways! How may help you? <AIRLINE> USR: Change my seat assignment <AIRLINE> SYS: ... <_eoc_>" For dialog sam- ple from MEDIA domain, similarly add <MEDIA> control tokens.
We preprocess this
The linear redaction schedule one approach that transitions smoothly between redacted and non- redacted data. The expconvex schedule decays ex- ponentially fast, and is convex function- it tran- sitions to non-redacted data after just few training steps. We found that expconcave schedule outper- formed the other schedules it decayed exponen- tially slowly, causing the trainer use redacted data for most of the initial training steps. This in line with Shi et al. (2022a)'s s observation that fine- mee-tuning on the new domain with a non-noisy op- timizer like AdamW results in benign initialization. Our expconcave redaction schedule implements this idea in one-stage fine-tuning process.
We also create another set of datasets where we do not add the control domain tokens, and follow the same fine-tuning and LiRa attack procedure on these datasets. See Section E.3 for results on this ablation experiment.
Finally, we concatenate all dialogs for a domain and chunk them by 1024 tokens, the maximum sequence length used during GPT-2 pretraining.
C.1 Redaction Policies
Table shows example dialog turns for each dialog domain and redaction policy.
D Redaction Schedules
We experimented with the redaction schedules de- scribed in Figure 3. The two-stage process JFT fine-tunes on redacted data with AdamW optimizer, and then switches to non-redacted data with DP- AdamW W optimizer (Shi al., 2022a). This corre- sponds trivial step schedule: constant p = for a half of the training steps and then constant = for the remaining half.

---

Example
AIRLINE Domain
Original
<AIRLINE> USR: my name raja <AIRLINE> SYS: Could you also help me out with your
booking confirmation number? <AIRLINE> USR: confirmation number Ikj459 <AIRLINE> SYS: Raja I'd like to inform you that you' ve been allotted 9C which window seat, is that fine with you? <AIRL INE> USR: ok
Redacted Keyword
<AIRLINE> USR: my
raja <AIRLINE> SYS: Could you also help out with your <REDACTED> <REDACTED> <REDACTED> <AIRLINE> USR: <REDACTED> <REDACTED> Ikj459
<AIRLINE> SYS: Raja I'd like to inform you that you' 'v been allotted 9C which <REDACTED>
<REDACTED> is that fine with you? <AIRLINE> USR: ok
Redacted RoBERTa
<AIRLINE> USR: my name raja SAIRLINE> SYS: Could you also help out with your booking confirmation number? <AIRLINE> USR: confirmation number Ikj459 <AIRLINE> SYS:
<REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> is that fine with you? <AIRLINE> USR: ok
Example Original
MEDIA Domain
<_soc_> <MEDIA> USR: Cameron, <MEDIA> SYS: Hi, good morning' You'l reached the customer executive Fasinet Cable services, how may help you today? <MEDIA> USR: want to sign up for new internet service with 5 GB plan <MEDIA> SYS: Sure! be glad to help you get new cable connection, may I please know your city and zip code?
Redacted Keyword
<_soc_> <MEDIA> USR: Hi Cameron, <MEDIA> SYS: Hi, good morning! You'l 've reached the customer executive of <REDACTED> <REDACTED> <REDACTED> how may I help you today? <MEDIA> USR: want to sign up for <REDACTED> <REDACTED> <REDACTED> with GB plan
<MEDIA> SYS: Sure! I'll be glad to help you get <REDACTED> <REDACTED> <REDACTED> may I
please know your city and its <REDACTED> <REDACTED>
Redacted RoBERTa
<_soc_> <MEDIA> USR: Hi Cameron, <MEDIA> SYS: <REDACTED> <REDACTED> <REDACTED>
<REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED>
<REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> you today?
<MEDIA> USR: want sign up for new internet service with GB plan <MEDIA> SYS:
<REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED>
<REDACTED> <REDACTED> city and its zip code?
Example
INSURANCE Domain
Original
<_soc_> <INSURANCE> USR: HI <INSURANCE> SYS: Hello good morning' How may help you? <INSURANCE> USR: I NEED PROOF ON INSURANCE ON THE MY CAR <INSURANCE> SYS: Absolutely! I'll be able to help you in this regard, may please know your name? <INSURANCE> USR: ANGEL. <INSURANCE> SYS: Thanks Angel! Could you please share the last 4 digit of
your <INSURANCE> USR: 2015
Redacted Keyword
<_Soc_> <INSURANCE> USR: HII <INSURANCE> SYS: Hello good morning' How may help you?
<INSURANCE> USR: <REDACTED> <REDACTED> ON <REDACTED> ON THE MY <REDACTED>
<INSURANCE> SYS: Absolutely! I'll ahle help you this regard, may please know your
name? <INSURANCE> USR: ANGEL <INSURANCE> SYS: Thanks Angel! Could you please share
the last digit your SSN? <INSURANCE> USR: 2015
Reducted RoBERTa
<_5oc_> <INSURANCE> USR: HI <INSRRANCE> SYS: Hello good morning! How may help you? <INSURANCE> USR: NEED PROOF ON INSURANCE ON THE MY CAR <INSURANCE> SYS: <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <REDACTED> <INSURANCE> USR: ANGEI. <INSURANCE> SYS: Thanks Angel!
Could you please share the last digit of your SSN? <INSURANCE> USR: 2015
Table 1: Example output redactions from the MultiDoGo dataset. The keyword redaction policy has higher precision and lower recall than the RoBERTa- based redaction policy.

---

E Additional Results
E.1
LiRa Attack Outputs
Tables 2 and 3 show the results of LiRa membership inference attacks on the models in Section 6.4.
LiRa Score
Context
Public Model Output
Domain Leak?
70.85
<_s0c_> <AIRLINE> USR: hi
Hi! You'le ve reached the customer executive of Fast- net Cable services, how may I help you today? Hello! Good morning! You'l re contacted D-link services. How may assist you today?
<AIRLINE> SYS:
48.43
<_50c_> <AIRLINE> USR: hi <AIRLINE> SYS:
4.02
<_soc_> <AIRLING INE> USR: hai
want new internet cable service
<AIRLINE> SYS: Good mom- ing! You't reached Flamingo customers service. How may I help you today? <AIRLINE> USR:
2.79
<_s0c_> <AIRLINE> USR:
Hello, Good morning! How may I assist you today?
<AIRLINE> SYS:
2.41
<_s0c_> <AIRL INE> USR: hai
Hello there! Welcome to Moon way airline service! How may I help you today?
<AIRLINE> SYS:
2.21
<_soc_> <AIRLINE> USR: hai
move to new house
<AIRLINE> SYS: Good morn- ing! You' 've reached Flamingo customers service. How may I help you today? <AIRLINE> USR:
LiRa Score
Context
Private Model Output
Domain Leak?
81.05
<_soc_> CAIRLINE> USR: Hi
Hello there! You are connected to Tiheri cable ser- vices. However, apologize to you for any further queries. Could you please help me with the city you belong to and your data plans?
V
<AIRLINE> SYS:
67.88
<_50c_> <AIRL INE> USR: HI
A very good day. You had successfully installed my shop. You have initiated the first procedure. I've registered the same connection.
<AIRLINE> SYS:
66.61
<_50c_> <AIRLINE> USR: HAI
Awesome! You'l connected new cable tion and the costs that you have to purchase your cable services. May know anything else you would like help with?
<AIRLINE> SYS:
54.14
<_50c_> <AIRLINE INE> USR: hai
Hello there! I've registcred you. Would you like assist me with the city of the city concerned the zip code so that you can get the bill you for you?
<AIRLINE> SYS:
48.11
<_50c_> <AIRLINE> USR:
hai
Fantastic! have found a book with great price on which one of the bills I'll send you in a two days. Now, I've stepped in and the check will be in place and you'll II soon enjoy.
<AIRLINE> SYS:
41.32
<AIRLINE>
USR:
hai
Good news! I 've successfully registered you the United airlines. You's ve successfully registered your claim for the flights. To book appointment get you seat, you'll Il need register the registered booking window at your home address and the mail address which will he on that.
<AIRLINE> SYS:
31.21
<_S0c_> <AIRLINE> USR: hai
Thank you for reaching patience, I'll surely help you choosing a platinum
us, appreciate your
<AIRLINE> SYS:
platinum platinum color, it great allow you. Please go ahead and purchase Anything else may help you with? Hi there! Good morning! You'le ve signed up to flight. 've registered an offer of 15 GB per month on a flght. <AIRLINE> USR: sale $50.00. How may assist you? <AIRLINE> SYS:
29.39
<_S0c_> <AIRLINE> USR: Hi.
Good Morning. wants to book
Table 2: High confidence examples produced by applying the LiRa attack on the Public and Private models Section 6.4. Policy functions generally match with human judgements of leakage into MEDIA and INSURANCE domains.

---

LiRa Score
Context
JFT Model Output
Domain Leak?
4.61
<_50c_> <AIRLINE> USR:
HAI
Hello! Welcome to prodesk cable services.
<AIRLINE> SYS:
How may help you today?
4.43
<_s0c_> <AIRLINE> USR: Heya!
Good evening! You're re now talking to the
I want to book a flight to Miami
Customer Executive, at your service!
<AIRLINE> SYS:
3.75
<_50c_> <AIRLINE> USR: hai
Hello there! You'l re talking to the Customer Executive, at your service!
<AIRLINE> SYS:
2.35
<_50c_> <AIRLINE> USR: hi <AIRLINE> SYS: Hey there, Good morning! You'v reached Jet fly customer serviee! You're connected to Sam! I'd be glad to assist you today!
want to check my seat numher on the flight reserved
<AIRLINE> USR:
2.28
<_50c_> <AIRLINE> USR:
check my what change my
<AIRLINE> SYS: Hey there, Good
seat?
morning' You'v reached fly customer service! You're connected
Sam! I'd glad assist you today!
<AIRLINE> USR:
LiRa Score
Context
Private Model Output
Domain Leak?
4.43
<_50c_> <AIRLINE> USR:
HAI
Good Moriin!! Welcome T-I.ink customer service. How may help you today?
<AIRLINE> SYS:
3.85
<_50c_> <AIRLINE> USR:
Hey there! Good morning. 've reached LMT Airways. Your flight from chennai
<AIRLINE> SYS: Hey there, Good fly customer service! You're connected to Sam! I'd glad to assist you today! <AIRLINE> USR: need boarding pass
morning! You'v reached
12:30 pm is a pleasure!
<AIRLINE> SYS:
2,53
<_soc_> <AIRLINE> USR: hi <AIRLINE> SYS: Hey there, Good morning! You'v reached Jet fly customer service! You're connected to Sam! I'd be glad to assist you today!
have an booking confirmation number and email address. can you send me to my email address?
<AIRLINE> USR:
2.43
<_soc_> <AIRLINE> SYS: Hello! Good moming. You'l ve reached L.TT Airways. How may assist you today?
residing in Los Angeles, and my domes- tie phone number is 90004
<AIRLINE> USR:
2.36
<_soc_> <AIRLINE> USR: Heya! I book a flight Miami <AIRLINE> SYS: Hey! Could you please help me out with the depar- ture city? <AIRLINE> USR: Sure thing, I'll leaving from Jacksonville <AIRLINE> SYS: Oh great! May know your preferred date of travel? <AIRLINE> USR: So I'm actually prety flexible on when we leave. So some time in September, and if you're 're able to find flight for under $300, that would be fantastic <AIRLINE> SYS:
The preferred date of travel is 5:00 AM and the flights under 300 should he couple passengers, while that will reach next month and the booking to done within the procedure after $170, if you't changed an seats?
Table 3: High confidence examples produced by applying the LiRa attack on the JFT and Redaction Schedule models in Section 6.4. Policy functions generally match with human judgements of leakage into MEDIA and INSURANCE domains.
E.2 Additional Domains
Figures 4 through 7 show the results of domain leakage experiments when using prompts from the MEDIA and INSURANCE domains.
E.3 Use of Domain Tokens
Figures and 9 show domain privacy tradcoffs when domain control tokens (<AIRLINE> etc.) are removed from all datasets and policy functions. As general trend we get similar, not slightly higher, LiRa

---

NLJ NC..A. Koywort Modaciion
MEDKA ROBERTA Redoction
6335
Cnty
INEDAA Dily 3--o-d NIDIA City Jeito
Printo
.TT
2030
Pup--aaaieed
Reseciindicieeee (yrti ander1)
018
by krike))
#TOrde4)
0.299
2315
Lie
010
6499
0.0
2X
5999
8090
2.5
4.5
(a) Keyword Redaction
(b) RoBERTa Redaction
Figure LiRa attack success rate vs PPL (Media Domain). Lower better for both axes.
MEDW Koyword Rodociion
2X
2K
7.5
7.0
1.0
MEDA Only (m1o4)
Priatc
MEDA Onty (ertes)
JFT
JTTull (h-to.))
Phota
Resiiladdded JP (k-55)4)
Rodetonn Bitoolee Syetic 1-5e-))
2.0
Modacton Schatule (pe1 b-bo-9)
5.
3.0
55
(a) Keyword Redaction
(b) RoBERTa Redaction
Figure evs PPL (Media Domain). Lower better for both axes.
Hit RANCE
INDERAAGE
5.12
Puz- -fededdd
Put
Put -f---ecccct
Redidber ditledil (ee11 (-ta-1)
Raceiiion Sened dn (ye inde-t
Redicier Schediak 15 (nte-3
Rediadton Bohooldl 19 bobo-
0:25
2.00
2.08
8.16
5.00
2.00
Tex Perpedtyy
Tixt Parplesty
(a) Keyword Redaction
(b) RoBERTa Redaction
Figure 6: LiRa attack success rate vs PPL (Insurance Domain). Lower is better for both axes.
success rates when the domain control tokens are not present in the datasets.

---

206
2X
00
75
755
IARURACE Only (eieed)
Pib
(=22-4)
HGUSARGE Ciby
J- (edar))
Fan
(1
Plods
55
Teal Pappecty
Pappady
(a) Keyword Redaction
(b) RoBERTa Redaction
Figure 7: E vs PPL (Insurance Domain). Lower is better for both axes.
ARUNE. Koyward Redoction Derrchn Tolons
ARRLIN, RoBeetAARodlaton No Domar Tokors
AFLUNE
.FTdede
JFT1k-b05
3.08
Tue
.FTnaal (e1s4)
_FTrual (kised))
0350
-Fettt (-Ge-5)
Rodedion (pe Inde-)
Pfate
Resecke Seleelk 1o-18 k-ted)
Redacion Schadai indad)
0325
JPT (1-504
Rackrie Scleedd to kisee
Ralb
0.300
2x
0:75
5.00
0250
6.00
2x
0322
2x
6100
2x
8.00
0172
2
Phrpotay
Text Persiocty
(a) Keyword Redaction
(b) RoBERTa Redaction
Figure 8: LiRa attack success rate vs PPL (no domain tokens). Lower is better for both axes.
IEL,,E, Keysced Seduction Domaah Thäwee
A IN.. RABERTA oeduction Dorrah Tokera
2x
2
70
7.0
ARRNIE Only
ARUNN O
FFT 6)
Pub
arri (eda-4)
Pue
.Fhaa geded)
Prhats
Roduction Semoduie (eett (inde44
Pruzz
Rodadier ce Schetaic (nde))))
Raderdos Schedus (1m15/1
RGP
Redadt cn Schadai tnett (nded)
60
50
Pappeety
Prpplcty
(a) Keyword Redaction
(b) RoBERTa Redaction
Figure 9: e vs PPL (no domain tokens). Lower is better for both axes.

---

E.4 Language Model Outputs
Context
<_soc_> <AIRLINE> USR: hai <AIRLINE> SYS: Good morning! You'le ve reached Flamingo
customers service. How may I help you today? <AIRLIMS INE> USR:
Model
Generation
Airline Only
boarding pass for under S300
Public
NEED TO KNOW THE PROOFLOP STATUS FOR MY INSULANCE OF ONE OF MY SEAT.
Private
boarding pass to he emailed to your mail
J. Keyword
Check departure city and send the date of my ticket pass
RS, Keyword
yes please
JF.. RoBERTa
boarding pass be emailed to your mail
RS, RoBERTa
seat arrangement
Context
<_soc_> <AIRLINE> SYS: Hello! Gond morning. You' reached LMT Airways. How may assist you today? <AIRLINE> USR: i want know my <AIRLINE> SYS: Awesome! I'd be glad to help you with that. May I know your last name please? <AIRLINE> USR: my name lakshmanan <AIRLINE INE> SYS: Alright Lakshmanan! Could you please share the booking conúimaatinn number? <AIRLINE> USR: confirmation number vbg456 <AIRLINE> SYS:
Model
Generation
Airline Only
Superb! Could you pleas share the seat number? Could there anything else may help with/
Public
Great news Laksh! You'll II be assigned to A9 window seat. Anything more queries might know?
Private
Great nows! Please allow me moment, I'll get this done to you in your allotted time for your flight, could you please let us share the booking confirmation code please? Would there be anything that would help you today?
JFT. Keyword
Alright. I've good news for your seat. Could you please stay with don't 't careassce further queries. Thank you! :). Bye :)
RS, Keyword
Awesome! Could I provide you with your current and last name please? Thank You! Please allow
JFT, RoBERTa
moment to proceed further and check for your details, Please give a moment while cheek.
RS. RoBERTa
Super, please help for me with the correct email associated with that.
Table 4: Dialog utterances generated by models in Section 6.4 when given the same input context from the AIRLINE domain test dataset. The Public model leaks sensitive tokens from the INSURANCE domain. All other outputs remain in the AIRLINE domain but vary in their degree of specificity, relevance, and correctness.


Reviews:
[{"review_id": "qu022hsfqk", "reviewer": "Reviewer_nhsK", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We are enthused by the reviewer’s excitement about domain privacy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "*  I really like notion of Domain Privacy, especially the scenario presented: a vendor serving language models to multiple clients across various domains must prevent information leaks from one client's domain to another's.\n*  Algorithm is simple, redact tokens from other domains.\n* Good experiment setup using LiRA [Carlini 2021] as the test bed for privacy leakage.\n\n\nRef:\n\n[Carlini 2021] Membership Inference Attacks From First Principles\n", "reasons_to_reject": "Nothing I can think of.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper introduces \"domain privacy\" for contextual language models, ensuring that prompts from one domain do not produce sensitive outputs from other domains. It addresses concerns especially relevant for entities pooling data from multiple sources that require non-leakage of their sensitive information. The main contributions are:\n\n* Definition of domain privacy for language models.\n* Fine-tuning algorithms balancing domain privacy and model performance.\n* Experiments on text generation with multi-domain datasets, demonstrating that the proposed techniques achieve domain privacy while maintaining performance.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Zjhe4g9I0n", "reviewer": "Reviewer_sL5m", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments. We are delighted that they find our work novel and the experiments intuitive. Below we respond to their concerns:\n\n> The title of this paper seems confusing. It may claim that this work aims to improve the Tranfomer model, if it is the case, the authors should cover an extensive of tasks by Transformer, and the methodology should be specifically designed for the Transformer architecture.\n\nWe agree with the reviewer that the title “Domain Private Transformers” may suggest a wider scope of tasks than our current results. We wanted to introduce the novel concept of domain privacy for large language models, where transformers are trained on data from multiple domains. Since our methods apply for transformers used in dialog systems, we can modify our title to “Domain Private Transformers for Multi-Domain Dialog Systems”. This would alleviate the reviewer’s concern.\n\n> I think the motivation of this work is not clear.\n\nWe motivate domain privacy for contextual language models from real-world concerns of practitioners. Consider the following concrete example. To train a dialog generator, a model provider pools data from companies of different industries, such as an airline, a movie theater, and a bank. Each company would not want their industry-specific data to be generated when prompted with text particular to other industries. Our proposed domain privacy addresses this real “safe generation” requirement that many model providers face.\n\n> It also lacks detailed discussions between this work and other work on the privacy of language models. This makes the contribution of this work weak to me.\n\nDomain privacy draws from literature on differential privacy, domain adaptation, and adversarial text generation attacks such as membership inference attacks. We discuss this in Sections 2 (Related Work) and 3 (Preliminaries). We also build upon recent empirical work [Li et al. 2022, Shi et al. 2022a-b] on private fine-tuning, and compare our methods with them. Space constraints limit us from further discussing related work, but we’d be happy to add it in the appendix.\n\n[Li, et al. 2022] [Large language models can be strong differentially private learners](https://arxiv.org/abs/2110.05679). In ICLR.\n\n[Shi et al. 2022a] [Just fine-tune twice: Selective differential privacy for large language models](https://arxiv.org/abs/2204.07667). In EMNLP.\n\n[Shi et al. 2022b] [Selective differential privacy for language modeling](https://arxiv.org/abs/2108.12944). In NAACL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The study of domain-level differential privacy for the Transformer model is novel and interesting.\n\n2. The experimental analyses are intuitive and make sense.", "reasons_to_reject": "1. The title of this paper seems confusing. It may claim that this work aims to improve the Tranfomer model, if it is the case, the authors should cover an extensive of tasks by Transformer, and the methodology should be specifically designed for the Transformer architecture. I think the motivation of this work is not clear.\n\n2. It also lacks detailed discussions between this work and other work on the privacy of language models. This makes the contribution of this work weak to me.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper discusses the performance of large, general-purpose language models in various conversational domains. While these models achieve low overall perplexity, there is a concern about their outputs staying within the intended domain. The paper introduces the concept of domain privacy to quantify the likelihood of a language model leaking across domains. It proposes policy functions based on token-level domain classification and an efficient fine-tuning method to enhance the model's domain privacy. Experimental results on membership inference attacks demonstrate that this method is comparable in resilience to existing differentially private language models.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Wpd2baaWjw", "reviewer": "Reviewer_tU8m", "summary": "", "questions": "1. How to evaluate the leakage information or sensitive text as mentioned in line 140 when two different domains have common features? For example, booking tickets is a common feature in both the AIRLINE and MOVIE THEATER. Then there may be a same subset of target words in the two domains.\n2. Given large language models are encoded with a large amount of knowledge and show decent reasoning abilities, will the definition and method still be useful for them?\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insights. We are pleased that they find the paper well-written, and that our experiments were extensive and sufficient. We respond to their comment and questions below:\n\n> The definition and method may be hard to generalize to language models with larger size or more pretrained knowledge.\n\nOur definition of domain privacy is flexible enough to generalize to LLMs. Different models/datasets/training algorithms would lead to different values of $C$ in domain privacy, as the models would differ in how much cross-domain sensitive information they leak when prompted with text. In fact, determining the $C$ privacy parameter for LLMs is interesting future work.\n\nResponse to questions:\n> How to evaluate the leakage information or sensitive text as mentioned in line 140 when two different domains have common features? For example, booking tickets is a common feature in both the AIRLINE and MOVIE THEATER. Then there may be a same subset of target words in the two domains.\n\nWith the AIRLINE/MOVIE THEATER example, the reviewer suggests good intuition for the domain privacy definition using policy functions, or policies. Introduced by [Shi et al. 2022b], policies for a domain mark the sensitive text tokens particular to that domain. This allows us to check for sensitive text in model generations. So, overlap in cross-domain sensitive information is exactly captured by policies.\n\nDefinition 4.1 of domain privacy compares the model $M_D$ trained on the full dataset $D$ with reference models $M_{D \\setminus d_j}$ trained on all data of $D$ except of domain $d_j$. On prompts of domain $d_i$ where $i \\neq j$, if the reference model $M_{D \\setminus d_j}$ leaks plentiful sensitive text of $d_j$, then domains $d_i$ and $d_j$ likely have large overlap. So, we are always comparing the domain privacy of $M_D$ against reference models that quantify cross-domain overlap. Thus, domain privacy also depends on the choice of policies, which determine what is considered sensitive in the first place.\n\n> Given large language models are encoded with a large amount of knowledge and show decent reasoning abilities, will the definition and method still be useful for them?\n\nAssuming open-access to LLMs, domain privacy is even more important for them since the training data is large and pooled from diverse sources. LLMs show decent zero-shot reasoning abilities, but they still can hallucinate text across domains. Our proposed methods use membership inference attacks, which require likelihoods of generated text. For closed-source models, this is an active area of research and interesting future work.\n\n[Shi et al. 2022b] [Selective differential privacy for language modeling](https://arxiv.org/abs/2108.12944). In NAACL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written and easy to follow.\n2. It defines an important property of private transformers. Extensive experiments and sufficient analysis are presented to support the main claims.\n", "reasons_to_reject": "1. The definition and method may be hard to generalize to language models with larger size or more pretrained knowledge.", "questions_to_authors": "1. How to evaluate the leakage information or sensitive text as mentioned in line 140 when two different domains have common features? For example, booking tickets is a common feature in both the AIRLINE and MOVIE THEATER. Then there may be a same subset of target words in the two domains.\n2. Given large language models are encoded with a large amount of knowledge and show decent reasoning abilities, will the definition and method still be useful for them?\n", "paper_topic_and_main_contributions": "The paper defines domain privacy towards transformers, which is an important and novel property of differential privacy on language models. It further proposes a redaction schedule finetuning method to attain domain privacy as well as achieve good performance.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that the notion of domain privacy of general-purpose language models introduced in this paper is a novel and interesting idea.  Also the solid experimental design for evaluating privacy leakage and the results are appreciated.  One important concern emerging from the reviews and discussion is that the authors should take care to clearly formulate the scope of what the work is aiming to do.  The current title suggests too wide a scope to be justified by the work, and therefore the alternative title brought forward by the authors themselves in their rebuttal seems reasonable.
Decision: Accept-Findings