SlowGuess/NAACL_2022_OCR huggingface dataset:
Note that the CRT algorithm achieves the worst-case confidentiality guarantee if we have a nontrivial conservative screening policy that outputs  $\pi_c(x) = 1$  for all secret  $x$  that  $\pi$  misses, or we simply run vanilla DP-SGD after dedduplication and redaction. On the other hand, CRT still satisfies Bayesian confidentiality for each  $\mu$  depending on the recall rate of  $\pi_c$  under  $\mu$ .
We evaluate CRT by training two types of language model, LSTM and GPT-2, on two datasets: 1) MultiWOZ 2.2, a well-known human-written dialogue dataset and 2) CustomerSim, a simulated dialogue dataset for conversation generation.
MultiWOZ 2.2 is an already-public dialogue dataset written by crowd-workers, which collects
over 10,000 annotated dialogues spanning 8 domains (Zang et al., 2020). We use this dataset to show how CRT works in real-world applications. Following US Department of Labor's guidance on personal-identifiable information (PII), we treat all confidential information (e.g. email address, reference number, telephone number, etc.) as secrets. For the sequence labeling policy  $\pi$  and conservative policy  $\pi_c$ , we build upon an NER model to do redaction. See Appendix A.4 for more details.
CustomerSim. Following S-DP Shi et al. (2021), we simulate a dialog dataset called CustomerSim with synthetic user information. The dialog flow is simulated based on a fixed agenda and the language generation is template-based (Zhao and Eskenazi, 2018). CustomerSim consists of 10 thousand examples and over one million tokens. We treat user name, address, phone number, order, and tracking number as secrets, and use a regular expression tester (regex) to detect them for the redaction process.
Experiment details. For LSTM model, we follow the setting in S-DP to choose a one-layer LSTM. Because S-DP requires hidden states of the sensitive input to be protected, it doesn't support more layers nor Bidirectional LSTM. Since the advent of Transformers (Vaswani et al., 2017) significantly improves the capabilities of generative language models, we also test transformer-based language model GPT-2 (Radford et al., 2019) from HuggingFace (Wolf et al., 2019). As for dedduplication, we use SHA-1 (Jarvinen, 2004) hash function to encode sequences to SHA-1 hash code and then remove identical sequences based on the same hash code. For Bayesian Confidentiality, we treat the uniform distribution over the secret sequences as the distribution  $\mu$ . More experiment details can be found in Appendix A.3.
Baselines. For LSTM model, we compare four different training approaches: (1) vanilla SGD (denoted by "Non-private-LSTM"), (2) Selective DPSGD (denoted by "S-DP-LSTM") (3) DPSGD (denoted by "DPSGD-LSTM") and (4) confidentially redacted training (denoted by "CRT-LSTM"). While for GPT-2 model, we compare three different training approaches: (1) vanilla SGD (denoted by "Non-private-GPT"), (2) DPSGD (denoted by "DPSGD-GPT") and (3) CRT (denoted by "CRT-GPT"). Our implementation of S-DP-LSTM
<sup>4</sup>https://www.dol.gov/general/ppii
Figure 2: Model utility and confidentiality guarantee on MultiWOZ 2.2 and CustomerSim datasets with  $\mu$  being a uniform distribution over the secret sequences in each dataset. PPL: Perplexity on the test set.  $\epsilon$ : Privacy guarantee in Bayesian Confidentiality. We fix  $\delta = 8e - 5$  for all models. Since Selective DP-SGD with approximate policy gives  $\epsilon = +\infty$ , we show its result with a perfect screen policy. But when a perfect policy is available, Reduction only gives  $\epsilon = 0$  and achieves the PPL of vanilla training with no noise added (Non-private-GPT/LSTM). For other models we set  $\gamma = 0.1$  to show the result under approximate policy.
model is built upon Shi et al. (2021) $^5$ . We run the experiment for the GPT-2 model following Li et al. (2021) $^6$ , in which they propose ghost clipping method to alleviate the computational challenge of running DP-SGD with large Transformers.
All the models are trained five times to reduce randomness, and the parameters are tuned based on the validation set performances.
5.1 Evaluation procedure
We need to evaluate both model utilities and privacy guarantees of the language models. We measure predictive perplexity (PPL) for the quality of LM. We also analyze the theoretical privacy budget  $(\epsilon, \delta)$  and test whether language models are private under attacks detailed below.
Canary insertion attack. Canary insertion is proposed as a testing methodology for quantitatively assessing the risk of unintended memorization (Carlini et al., 2019). It inserts random sequences called canaries into the training dataset, then trains the model, and finally calculates the following exposure for the inserted canaries to measure a model's potential for privacy risks. In our experiment, we randomly generate 10 canaries in the form of "My ID is: <random 6-digit number here)". Each canary is inserted into the training dataset 20 times to generate more salient differences between models.
Definition 9 (Canary Exposure). Given a canary  $s[r]$ , a model with parameters  $\theta$ , and the randomness space  $\mathcal{R}$ , the exposure of  $s[r]$  is
$$
\operatorname {e x p o s u r e} _ {\theta} = \log_ {2} | \mathcal {R} | - \log_ {2} \operatorname {r a n k} _ {\theta} (s [ r ])
$$
<sup>5</sup>https://github.com/wyshi/lm隐私
<sup>6</sup>https://github.com/lxuechen/private-transformers
After training, we calculate empirical model perplexity for all possibly-instantiated canaries and list them in sorted order. Then we can get the canary exposure based on the rank of a specific canary sequence  $\mathrm{rank}_{\theta}(s[r])$  and the number of all possible candidates  $|\mathcal{R}|$ . In our setting, we show the highest canary exposure in 10 canaries. For example, if a canary ranks 1st among 1M candidates, the canary exposure is 19.93.
Membership inference attack. Membership Inference is a widely used privacy attack method. Given a non-privately trained model, an adversary can predict whether or not a particular example was used to train the model. We adopt the membership inference attack in Carlini et al. (2021). The general idea is to calculate the given samples' perplexities under the model, rank them and choose the ones with the lowest perplexities, i.e., highest likelihood by the model. We can think of this process as training a binary classifier based on the perplexity feature. We also implement the group membership inference attack to show the group confidentiality. More details about the implementation can be found in the Appendix A.5.
5.2 Overall performance
Figure 2 presents the results of model utilities and confidentiality guarantees across our models of interest on MultiWOZ 2.2 and CustomerSim datasets. Each point denotes a model for different epochs in a training process. Since the X-axis is  $\epsilon$  in Bayesian Confidentiality (the lower the better) and the Y-axis is perplexity (the lower the better), a perfect model will lie in the bottom-left corner. CRT-GPT and DPSGD-GPT in general, perform better than S-DPLSTM, CRT-LSTM and, DPSGD-LSTM on the test sets. Our model CRT-GPT's performance is
close to Non-private-GPT in terms of PPL while preserving strong confidentiality. Besides, CRT-GPT is better than DPSGD-GPT manifested by a much lower  $\epsilon$ , which demonstrates that approximately correct screening policy amplifies the confidentiality guarantee.
Differences can be witnessed in the results from two different datasets: the models trained on CustomerSim achieve overall better performances than those trained on MultiWOZ. We think it's due to the fact that CustomerSim contains simple dialogs from template-based simulations.
5.3 Attack results
Figure 3, 4, and 5 present the results from canary insertion attack and individual/group membership inference attack on MultiWOZ 2.2 and Customer-Sim datasets. The X-axis is the false negative rate  $\gamma$  of screening policy  $\pi$ , ranging from 0.0 to 0.5; the Y-axis is the canary exposure (in Figure 3) and membership inference accuracy (in Figure 4 and 5), which measures the effectiveness of the attacks. The lower the canary exposure or inference accuracy, the better protection the model provides against the attacks.
Figure 3: Canary insertion attack result. CRT achieves almost 0 canary exposure, which means it can prevent unintended memorization.
For canary insertion attack, it can be seen from Figure 3 that the canary exposures for CRT-LSTM and CRT-GPT are both close to 0 which thus guarantee excellent confidentiality. Non-private-LSTM and Non-private-GPT with mask can also attain great protection at perfect screening policy accuracy  $(\gamma = 0)$ , nonetheless a rise in  $\gamma$  results in a sharp increase in the exposure. It should be noticed that S-DP-LSTM also has high exposure, similar to Non-private models, given any  $\gamma$  above 0. This is because that many sensitive data has been falsely identified as non-sensitive by the approximate policy, S-DPSGD does not protect these false negative samples and hence a privacy leakage.
For membership inference attack, we compare the inference accuracy with the benchmark value of
Figure 4: Membership inference attack result. CRT attains nearly  $50\%$  accuracy, indicating that the adversary could not infer whether a data point is used for training.
Figure 5: Group membership inference attack result.
0.5, which equals the random guess performance. In Figure 4 and 5, we see that CRT-LSTM and CRT-GPT align well with the 0.5 horizontal line, suggesting that they are rather safe to the attack. The inference accuracy for Non-private-LSTM/Non-private-GPT/S-DP-LSTM, in contrast, surges above 0.5 as the false negative rate  $\gamma$  deviates from 0.0, indicating that these models become vulnerable to the attack under non-perfect screen policy. In addition, Non-private and S-DP models show even worse protection under the group attack than the individual one in view of a higher inference accuracy at certain  $\gamma$ .
Figure 6: Bayesian Confidentiality amplification result. CRT helps to amplify the confidentiality guarantee.
5.4 CRT amplifies Bayesian Confidentiality guarantees
Figure 6 shows that confidentially redacted training can help to amplify the confidentiality guarantees. We set the  $\epsilon^{\prime}$  in DP-SGD fixed and show the
corresponding  $\epsilon$  in Bayesian Confidentiality with different screen policy  $\pi$ . Both  $\epsilon'$  and  $\epsilon$  are for  $\delta = 8e - 5$ . If the approximately screening policy  $\pi$  has a high recall ( $\gamma$  is small), we will achieve much improvement in the Bayesian Confidentiality parameter  $\epsilon$  by dedduplication and redaction. For example, with  $(\epsilon' = 1.0, \gamma = 0.1)$ , we reduce the  $\epsilon$  to 0.12.
In this paper, we propose confidentially redacted training (CRT), a method to train language models while protecting the secret texts. We introduce a new definition of confidentiality which quantifies the risk of leaking sensitive content. We prove the effectiveness of CRT both theoretically and empirically on multiple datasets and language models.
This work will alleviate ethical concerns of large-scale pre-trained language models. This paper provides one promising solution to an important aspect of NLP: training high quality language models for text generation without compromising confidential information. The current use cases of language models involve pretraining on public web corpus and fine-tuning on individual application data. However, the private application specific data often contains user-generated sensitive information. The proposed method in this paper aims to use as much individual fine-tuning data as possible, while does not leak or memorize any confidential information with provable guarantees. Without the method, one has to either use the general pretraining LM without fine-tuning or manually filter sensitive information and fine-tuning on the remaining. It can be applied in broader applications that need language models or text generation models.
In our experiments, we use a simulation scheme to mimic confidential content in a real corpus. We did not compromise any real user's confidential information.
The work was partially supported by NSF Award # 2048091. XZ was supported by UCSB Chancellor's Fellowship. We would like to thank the anonymous reviewers for their thoughtful comments. We would also like to thank Siqi Ouyang for the helpful discussion and Yang Gao for polishing up the draft.
Martín Abadi, Andy Chu, Ian J. Goodfellow, H. Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. 2016. Deep learning with differential privacy. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security.
Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. 2021. Large-scale differentially private bert. ArXiv, abs/2108.01624.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
Borja Balle, Gilles Barthe, and Marco Gaboardi. 2018. Privacy amplification by subsampling: tight analyses via couplings and divergences. In Advances in Neural Information Processing Systems, pages 6280-6290.
Gilles Barthe and Federico Olmedo. 2013. Beyond differential privacy: Composition theorems and relational logic for f-divergences between probabilistic programs. In International Colloquium on Automata, Languages, and Programming, pages 49-60. Springer.
Burton H Bloom. 1970. Space/time trade-offs in hash coding with allowable errors. Communications of the ACM, 13(7):422-426.
Nicholas Carlini, Chang Liu, Ülfar Erlingsson, Jernej Kos, and Dawn Xiaodong Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In USENIX Security Symposium.
Nicholas Carlini, Florian Tramér, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B. Brown, Dawn Xiaodong Song, Ülfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In USENIX Security Symposium.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805.
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. 2006. Calibrating noise to sensitivity in private data analysis. In *Theory of cryptography conference*, pages 265-284. Springer.
Vitaly Feldman. 2020. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954-959.
Debasis Ganguly, Dwaipayan Roy, Mandar Mitra, and Gareth J.F. Jones. 2015. Word embedding based generalized language model for information retrieval. Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval.
Arpita Ghosh and Aaron Roth. 2015. Selling privacy at auction. Games and Economic Behavior, 91:334-346.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural Computation*, 9:1735-1780.
Ehsan Hosseini-Asl, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. 2020. A simple language model for task-oriented dialogue. ArXiv, abs/2005.00796.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL.
Kimmo Jarvinen. 2004. Design and implementation of a sha-1 hash module on fpgas. Helsinki University of Technology Signal Processing Laboratory.
Nikhil Kandpal, Eric Wallace, and Colin Raffel. 2022. Deduplicating training data mitigates privacy risks in language models. arXiv preprint arXiv:2202.06539.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc V. Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. Deduplicating training data makes language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics.
Xuechen Li, Florian Tramér, Percy Liang, and Tatsunori Hashimoto. 2021. Large language models can be strong differentially private learners. *ArXiv*, abs/2110.05679.
H. Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. 2018. Learning differentially private recurrent language models. In ICLR.
Tomas Mikolov, Martin Karafiát, Lukás Burget, Jan Honza Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In INTERSPEECH.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10683.
Weiyan Shi, Aiqi Cui, Evan Li, R. Jia, and Zhou Yu. 2021. Selective differential privacy for language modeling. ArXiv, abs/2108.12944.
Aleksei Triastcyn and Boi Faltings. 2020. Bayesian differential privacy for machine learning. In International Conference on Machine Learning, pages 9583-9592. PMLR.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. ArXiv, abs/1706.03762.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, and et al. 2013. Ontonotes release 5.0. Linguistic Data Consortium, Philadelphia, PA, 23.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Xiaoxue Zang, Abhinav Rastogi, Srinivas Sunkara, Raghav Gupta, Jianguo Zhang, and Jindong Chen. 2020. Multiwoz 2.2: A dialogue dataset with additional annotation corrections and state tracking baselines. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, ACL 2020, pages 109-117.
Tiancheng Zhao and Maxine Eskenazi. 2018. Zero-shot dialog generation with cross-domain latent actions. In SIGDIAL Conference.
A.1 Illustration of our proposed algorithm
Figure 7: An illustration of our proposed algorithm on a dataset with two data points. The first data point is the example from Figure 1, and the second data point is modified to illustrate the various aspects of the pre-processing steps. The red-colored mask indicates the masks produced by dedduplication just for illustration purposes. In the algorithm, both of them replace a sequence of tokens with the same special token <MASK>.
A.2 Proofs of technical results
Proof of Proposition 5. The first statement straightforwardly follows from that  $\operatorname{Redact}_{\pi}(D) = \operatorname{Redact}_{\pi}(D')$  if  $\pi(s, x) = 1$  and that  $\operatorname{Redact}_{\pi}(D)$  and  $\operatorname{Redact}_{\pi}(D')$  remain a pair of neighbors differing by only  $x$ . The group confidentiality claims follow from the standard calculation of small group privacy from differential privacy, which applies the (single  $x$ ) confidentiality iteratively. Let  $\tilde{D} = \operatorname{Redact}_{\pi}(D)$ ,  $\tilde{D}' = \operatorname{Redact}_{\pi}(D')$  and  $\tilde{S} = [x_1, \ldots, x_{\tilde{k}}]$  be the list of  $S$  that are not masked by  $\pi$ . For any measurable event  $E$
$$
\begin{array}{l} \mathbb {P} [ \mathcal {A} \circ \operatorname {R e d a c t} _ {\pi} (D) \in E ] = \mathbb {P} [ \mathcal {A} (\tilde {D}) ] \leq e ^ {\epsilon x _ {1}} \mathbb {P} [ \mathcal {A} (\tilde {D} _ {- x _ {1}, + <   \mathrm {M A S K} >}) \in E ] + \delta \\ \leq e ^ {\epsilon x _ {1} + \epsilon (x _ {2})} \mathbb {P} \left[ \mathcal {A} \left(\tilde {D} _ {- x _ {1, 2}, + <   \mathrm {M A S K} > ^ {2}}\right) \in E \right] + e ^ {\epsilon x _ {1}} \delta + \delta \\ \dots \\ \leq e ^ {\sum_ {i = 1} ^ {\tilde {k}} \epsilon x _ {i}} \mathbb {P} [ \mathcal {A} (\tilde {D} ^ {\prime}) \in E ] + \delta (1 + e ^ {\epsilon x _ {1}} + e ^ {\epsilon x _ {1} + \epsilon x _ {2}} + \dots + e ^ {\epsilon x _ {1} + \dots + \epsilon x _ {\tilde {k} - 1}}) \\ \leq e ^ {\tilde {\epsilon} (S)} \mathbb {P} [ \mathcal {A} \circ \operatorname {R e d a c t} _ {\pi} (D ^ {\prime}) \in E ] + k e ^ {\tilde {\epsilon} (S)} \delta \\ \end{array}
$$
Proof of Proposition 6. Consider a dataset  $D$  (in which one of the data points has  $x \sim \mu$ ) and a fixed  $D'$ . Denote the probability distributions  $p, q, r$  as shorthands for
$$
p \sim \mathcal {A} \circ \operatorname {R e d a c t} _ {\pi} (D) | \pi (s, x) = 1
$$
$$
q \sim \mathcal {A} \circ \operatorname {R e d a c t} _ {\pi} (D) | \pi (s, x) = 0
$$
$$
r \sim \mathcal {A} \circ \operatorname {R e d a c t} _ {\pi} \left(D ^ {\prime}\right) | \pi (s, x) = 0
$$
Moreover, we use  $\alpha p + (1 - \alpha)q$  to denote the mixture distribution that samples from  $p$  with probability  $\alpha$  and  $q$  with probability  $1 - \alpha$ .
Recall that the Hockey-Stick-divergence characterization of  $(\epsilon, \delta)$ -indistinguishability (Barthe and Olmedo, 2013), which says that  $(P, Q)$  are  $(\epsilon, \delta)$ -indistinguishable if and only if
$$
H _ {e ^ {\epsilon}} (P \| Q) := \mathbb {E} _ {y \sim Q} [ (\frac {d P}{d Q} (y) - e ^ {\epsilon}) _ {+} ] \leq \delta .
$$
It suffices for us to bound the following quantity:
$$
\begin{array}{l} H _ {1 + \gamma (e ^ {\epsilon} - 1)} (\mathcal {A} \circ \operatorname {R e d a c t} _ {\pi} (D) \| \mathcal {A} \circ \operatorname {R e d a c t} _ {\pi} (D ^ {\prime})) = H _ {e ^ {\epsilon}} ((1 - \gamma) p + \gamma q \| (1 - \gamma) p + \gamma r) \\ = \gamma H _ {e ^ {\epsilon}} (q \| (1 - \beta) p + \beta r) \leq \gamma ((1 - \beta) H _ {e ^ {\epsilon}} (q \| p) + \beta H _ {e ^ {\epsilon}} (q \| r)) \\ \end{array}
$$
where  $\beta = \frac{1 + \gamma(e^{\epsilon} - 1)}{e^{\epsilon}}$ . In the above, the second line follows from Theorem 2 of (Balle et al., 2018) (an identity called "Advanced Joint Convexity" by the authors) and the inequality is due to the (standard) joint convexity of the Hockey-Stick divergence. It remains to bound  $H_{e^{\epsilon}}(q \| p)$  and  $H_{e^{\epsilon}}(q \| r)$ .
Check that  $p, r, \mathcal{A} \circ \operatorname{Redact}_{\pi}(D')$  are identically distributed and that  $H_{e^{\epsilon}}(q \| r) \leq \delta$  by our assumption on  $\mathcal{A}$ 's Bayesian confidentiality guarantee w.r.t.  $\mu(x | \pi(s, x) = 0)$ . This completes the proof.
Proof of Proposition 7. The proof is straightforward as  $\operatorname{Dedup}(D)$  differs from  $\operatorname{Dedup}(D')$  only by  $\operatorname{Unique}(S)$ .
Proof of Theorem 8. The proof for the first statement follows from the fact that DP implies  $(\epsilon, \delta)$ -confidentiality and Proposition 5. Notably, if  $\pi_c$  catches all  $x$  that is missed by  $\pi$ , then we get that for all secret  $x$ ,  $\epsilon(x) \leq \epsilon$ .
The proof of the second statement applies Proposition 7 and the second part of Proposition 5.
The proof of the third statement applies Proposition 6 but requires a separate treatment of the case when  $x$  is missed by both  $\pi$  and  $\pi_c$ . Let the event that a secret  $x$  is not selected by the conservative policy be  $E$  and let  $\mathcal{A}$  be a generic algorithm satisfying  $(\epsilon, \delta_1)$  Bayesian confidentiality under  $\mu$ ,