SlowGuess/NAACL_2021_OCR huggingface dataset:
This approach provides results for each condition which vary in frequency. To assess the comparative performance of probes over conditions of different prevalence, we group conditions into mutually exclusive bins reflecting frequency (allowing us to analyze differences in performance, e.g., on rare conditions). We group conditions by frequencies, from rarest (associated with 2-5 patients) to most common (associated with  $>20$  patients). We randomly sample 50 conditions from each of these groups, and train an MLP classifier on top of the encoded CLS token from the last layer in BERT (this results in 50 different models per group, i.e., 200 independent models). We measure, in terms of AUC and A@10, whether the probe for a condition return comparatively higher scores for patients that have that condition.
We report results in Table 5. Except for the rarest conditions (associated with  $< 5$  patients), these models achieve AUCs that are at best modestly better than chance, with all A@10 metrics
$^{13}$ Though the AUCs for the probing are calculated over a randomly sampled test subset of the full data used in Table 2.
<sup>14</sup>We upsample the minority examples, rather than undersampling as before, because the single-condition models are comparatively quick to train.
Model	(1,5]	(5,10]	(10,20]	(20, 10k]
ICD-9
Regular Base	0.520	0.507	0.500	0.526
Regular Large	0.444	0.505	0.479	0.522
Name Insertion	0.477	0.484	0.491	0.504
MedCAT
Regular Base	0.481	0.534	0.525	0.487
Regular Large	0.439	0.531	0.519	0.509
Name Insertion	0.460	0.577	0.508	0.525
$\approx 0$ . In sum, these models do not meaningfully recover links between patients and conditions.
6.3 Differences in Cosine Similarities
Prior work (Abdalla et al., 2020) has demonstrated that static word vectors can leak information: The cosine similarities between learned embeddings of patient names and conditions are on average significantly smaller than the similarities between patient names and conditions they do not have. We run a similar experiment to investigate whether contextualized embeddings similarly leak information (and also to assess the degree to which this holds on the MIMIC corpus as a point of comparison). We calculate the average cosine similarity between learned embeddings of patient names and those of positive conditions (conditions that the patient has) minus negative conditions (those that they do not have). Conditions and names span multiple tokens; we perform mean pooling over these to induce embeddings. Here again we evaluate on the aforementioned set of 27,906 patients.
We report results for BERT and word2vec (CBoW and SkipGram; Mikolov et al. 2013) in Table 6. $^{15}$  Values greater than zero here suggest leakage, as this implies that patient names end up closer to conditions that patients have, relative to those that they do not. Even when trained over the Name Insertion data (which we manipulated to frequently mention names), we do not observe leakage from the contextualized embeddings.
Table 5: Probing results (AUCs) for conditions with different frequencies. We make predictions for conditions using independent models based on BERT-encoded CLS tokens. We use a 50/50 train/test split over patients (results are over the test set). Columns correspond to conditions of different frequencies, with respect to the number of patients with whom they are associated (headers provide ranges). All A@10 ≈ 0.
Model	Mean	Std.
ICD-9
Regular Base	-0.010	0.019
Regular Large	-0.045	0.052
SkipGram Base	0.004	0.050
CBoW Base	0.008	0.035
BERT Name Insertion	-0.007	0.017
SkipGram Name Insertion	0.019	0.040
CBoW Name Insertion	0.017	0.043
MedCAT
Regular Base	-0.037	0.015
Regular Large	-0.055	0.029
SkipGram Base	-0.011	0.024
CBoW Base	-0.001	0.022
BERT Name Insertion	-0.027	0.013
SkipGram Name Insertion	0.013	0.024
CBoW Name Insertion	0.015	0.026
Table 6: Differences in (a) similarities between patient names and conditions they have, and (b) similarities between patient names and conditions they do not have. Static embeddings are 200 dimensional; we train these for 10 epochs. For BERT models, we use 10k patients rather than the  $\sim 28\mathrm{k}$  due to compute constraints.
6.4 Can we Recover Patient Names?
Here we try something even more basic: We attempt to determine whether a pretrained model has seen a particular patient name in training. The ability to reliably recover individual patient names (even if not linked to specific conditions) from BERT models trained over EHR data would be concerning if such models were to be made public. We consider a number of approaches to this task.
Probing We encode the patient's name ([CLS] [NAME] [SEP]) using BERT and train a Logistic Regression classifier that consumes resultant CLS representations and predicts whether the corresponding patient has been observed in training.
As mentioned above, patient names are explicitly mentioned in notes for 27,906 patients; these constitute our positive examples, and the remaining patients (of the 46,520) are negative examples. We split the data into equally sized train and test sets. We report results in Table 7. To contextualize these results, we also run this experiment on the standard BERT base model (which is not trained on this EHR data). We observe that the AUCs are near chance, and that the performance of the standard BERT base model is relatively similar to that of the Regular and Large base models, despite the fact that the standard BERT base model has not seen any notes from MIMIC.
<sup>15</sup>We provide additional results in the Appendix, including results for alternative pooling strategies and results on the original MIMIC dataset; all yield qualitatively similar results.
Model	AUC
Regular Base	0.508
Large Base	0.501
Standard Base	0.498
Table 7: Predictions (on a test set) of which names have been seen by the model. We include the standard BERT (Devlin et al., 2019) model ("Standard Base"), which is not trained on MIMIC, as a comparator.
Model	AUC
First Name Masked
Regular Base	0.510
Regular Large	0.506
Name Insertion	0.562
Template Only	0.625
Last Name Masked
Regular Base	0.503
Regular Large	0.498
Name Insertion	0.517
Template Only	0.733
Table 8: We compute the perplexity of the masked parts of names for all 46,520 patients and measure whether the (27,906) reidentified patients receive lower perplexity, compared to remaining patients.
6.5 Does observing part of a name reveal more information?
Given a first name, can we predict whether we have seen a corresponding last name? More specifically, we mask out a patient's last name (but not their first) in the template “[CLS] [First Name] [MASK]+ [SEP]” and record the perplexity of the target last name. We take as the set of outputs all 46,520 patient names in the corpus.
We can also flip this experiment, masking only first names. This is intuitively quite difficult, as only  $10\mathrm{K} / 77\mathrm{M}$  sentences  $(0.013\%)$  contain both the patient's first and last name. This number includes first and last name mentions that are also other English words (e.g. "young"). Results are reported in Table 8. We do observe reasonable signal in the semi-synthetic Name Insertion and Template Only variants.
6.6 Text Generation
Recent work by Carlini et al. (2020) showed that GPT-2 (Radford et al., 2019) memorizes training data, and proposed techniques to efficiently recover sensitive information from this model (e.g., email addresses). They experimented only with large, auto-regressive language models (i.e., GPT-2), but their techniques are sufficiently general for us to use here. More specifically, to apply their
approaches to a BERT-based model $^{16}$  we must be able to sample text from BERT, which is complicated by the fact that it is not a proper (auto-regressive) language model. To generate outputs from BERT we therefore followed a method proposed in prior work (Wang and Cho, 2019). This entails treating BERT as a Markov random field language model and using a Gibbs sampling procedure to generate outputs. We then analyze these outputs from (a) our regular BERT-based model trained on MIMIC; (b) the Name Insertion model, and; (c) a standard BERT Base model (Devlin et al., 2019). We generate 500k samples from each, each sample consisting of 100 wordpiece tokens.
Comparator Model Perplexity Following Carlini et al. (2020), we attempt to identify which pieces of generated text are most likely to contain memorized names (in this case, from EHR). To this end, we examine segments of the text in which the difference in likelihood of our trained BERT model versus the standard BERT-base model (Devlin et al., 2019) is high. For the samples generated from the standard BERT-base model (not trained on MIMIC), we use our ClinicalBERT model as the comparator.[17] Using an off-the-shelf NER tagger (Honnibal et al., 2020), we identify samples containing name tokens.
For each sample, we mask name tokens individually and calculate their perplexity under each of the the respective models. We take the difference between these to yield a score (sequences with high likelihood under the trained model and low likelihood according to the general-domain BERT may contain vestiges of training data) and use it to rank our extracted names; we then use this to calculate A@100.
As expected, the Name Insertion model produced more names than the Base model, with approximately  $60\%$  of all sentences containing a name (not necessarily in MIMIC). Additionally, the A@100 of the Name Insertion model substantially outperforms the Base model. However, when we use spaCy to examine sentences that contain both a condition and a patient's name (of the 27,906), we find that  $23.5\%$  of the time the pa
<sup>16</sup>Which, at least at present, remains the default encoder used in biomedical NLP.
<sup>17</sup>Note that this means that even though samples are generated from a model that cannot have memorized anything in the EHR, using a comparator model that was to re-rank these samples may effectively reveal information.
Model	Sent. with Name	First Names	Last Names	A@100	Name + Positive Condition
Standard BERT Base	84.7%	2.16%	7.72%	0.34	12.17%
Regular Base	47.9%	0.94%	3.14%	0.16	23.53%
Name Insertion	59.6%	2.65%	4.56%	0.84	4.17%
Table 9: Results over texts generated by the Base and Name Insertion models. The 'Sent. with Name' column is percentage of extracted sentences that contain a name token. The First and Last name columns show what percent of unique names produced are in the MIMIC dataset. After re-ranking all unique names, we report the percentage of top 100 names that belong to a reidentified patient. Finally, The Name + Positive Condition displays what percent of sentences with a patient's name also contain one of their true (MedCAT) conditions.
tient does indeed have a condition produced by the Base model. It is unclear to what extent this reflects memorization of concrete patient-condition pairs per se, as opposed to learning more diffused patient-agnostic distributions of conditions in the MIMIC dataset. The corresponding statistic for the Name Insertion variant (4.17%) may be low because this tends to produce poor quality outputs with many names, but not many conditions. This is an intriguing result that warrants further research.
However, we caution that these generation experiments are affected by the accuracy of NER taggers used. For example, many of the extracted names tend to also be generic words (e.g., 'young', 'date', 'yo', etc.) which may artificially inflate our scores. In addition, MedCAT sometimes uses abbreviations as conditions, which may also yield 'false positives' for conditions.
This work has important limitations. We have considered only relatively simple "attacks", based on token in-filling and probing. Our preliminary results using the more advanced generation approach (inspired by Carlini et al. 2020) is a promising future direction, although the quality of generation from BERT — which is not naturally a language model — may mitigate this. This highlights a second limitation: We have only considered BERT, as it is currently the most common choice of pretrained Transformer in the bioNLP community. Auto-regressive models such as GPT-2 may be more prone to memorization. Larger models (e.g., T5 (Raffel et al., 2020) or GPT-3 (Brown et al., 2020)) are also likely to heighten the risk of data leakage if trained over EHR.
Another limitation is that we have only considered the MIMIC-III corpus here, and the style in which notes are written in this dataset — names appear very infrequently — likely renders it particularly difficult for BERT to recover implicit as
sociations between patient names and conditions. We attempted to address this issue with the semisynthetic Name Insertion variant, where we artificially inserted patient names into every sentence; this did not yield qualitatively different results for most experiments. Nonetheless, it is possible that experiments on EHR datasets from other hospitals (with different distributions over tokens and names) would change the degree to which one is able to recover PHI.
Finally, these results for BERT may change under different masking strategies — for example, dynamic masking (Liu et al., 2019) or choice of tokenizer. Both of these may affect memorization and extraction method performance.
We have performed an initial investigation into the degree to which large Transformers pretrained over EHR data might reveal sensitive personal health information (PHI). We ran a battery of experiments in which we attempted to recover such information from BERT model weights estimated over the MIMIC-III dataset (into which we artificially reintroduced patient names, as MIMIC is deidentified). Across these experiments, we found that we were mostly unable to meaningfully expose PHI using simple methods. Moreover, even when we constructed a variant of data in which we pretended patient names to every sentence prior to pretraining BERT, we were still unable to recover sensitive information reliably. Our initial results using more advanced techniques based on generation (Carlini et al. 2020; Table 9) are intriguing but inconclusive at present.
Our results certainly do not rule out the possibility that more advanced methods might reveal PHI. But, these findings do at least suggest that doing so is not trivial. To facilitate further research, we make our experimental setup and baseline probing models available: https://github.com/elehman16/exposingpatient_data_release.
This work has ethical implications relevant to patient privacy. HIPAA prohibits the distribution of PHI, for good reason. Without this type of privacy law, patient information, for example, could be passed on to a lender and be used to deny a patient's application for mortgages or credit card. It is therefore essential that patient information remain private. This raises an important practical concerning methods in NLP that we have sought to address: Does releasing models pretrained over sensitive data pose a privacy risk? While we were unable to reliably recover PHI in this work, we hope that this effort encourages the community to develop more advanced attacks to probe this potential vulnerability. We would still advise researchers to err on the side of caution and only consider releasing models trained over fully de-identified data (e.g. MIMIC).
We thank Peter Szolovits for early feedback on a draft of this manuscript, and the anonymous NAACL reviewers for their comments.
This material is based upon work supported in part by the National Science Foundation under Grant No. 1901117. This Research was also supported with Cloud TPUs from Google's Tensor-Flow Research Cloud (TFRC).
Mohamed Abdalla, Moustafa Abdalla, Graeme Hirst, and Frank Rudzicz. 2020. Exploring the privacy-preserving properties of word embeddings: Algorithmic validation study. J Med Internet Res.
Guillaume Alain and Yoshua Bengio. 2017. Understanding intermediate layers using linear classifier probes. In The 5th International Conference on Learning Representations (ICLR-17).
Emily Alsentzer, John Murphy, William Boag, WeiHung Weng, Di Jin, Tristan Naumann, and Matthew McDermott. 2019. Publicly available clinical BERT embeddings. In Proceedings of the 2nd Clinical Natural Language Processing Workshop, pages 72-78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Brett K. Beaulieu-Jones, William Yuan, Samuel G. Finlayson, and Z. Wu. 2018. Privacy-preserving distributed deep learning for clinical data. *ArXiv*, abs/1812.01484.
O. Bodenreider. 2004. The unified medical language system (uml's): integrating biomedical terminology. *Nucleic acids research*, 32 Database issue:D267-70.
Zied Bouraoui, José Camacho-Collados, and S. Schockaert. 2020. Inducing relational knowledge from bert. In AAAI.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
N. Carlini, Florian Tramér, Eric Wallace, M. Jagielski, Ariel Herbert-Voss, K. Lee, A. Roberts, Tom Brown, D. Song, Ülfar Erlingsson, Alina Oprea, and Colin Raffel. 2020. Extracting training data from large language models. ArXiv, abs/2012.07805.
J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Matt Fredrikson, S. Jha, and T. Ristenpart. 2015. Model inversion attacks that exploit confidence information and basic countermeasures. In CCS '15.
Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pretraining for biomedical natural language processing.
Benjamin Heinzerling and Kentaro Inui. 2020. Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries. ArXiv, abs/2008.09036.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrial-strength Natural Language Processing in Python.
Kexin Huang, Jaan Altosaar, and R. Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. ArXiv, abs/1904.05342.
Zhengbao Jiang, Antonios Anastasopoulos, Jun Araki, Haibo Ding, and Graham Neubig. 2020a. X-FACTR: Multilingual factual knowledge retrieval from pretrained language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), Online.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020b. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Alistair E. W. Johnson, Lucas Bulgarelli, and Tom J. Pollard. 2020. Deidentification of free-text medical records using pre-trained bidirectional transformers. In Proceedings of the ACM Conference on Health, Inference, and Learning, CHIL '20, page 214-221, New York, NY, USA. Association for Computing Machinery.
Alistair EW Johnson, Tom J Pollard, Lu Shen, H Lehman Li-wei, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific data, 3:160035.
Zeljko Kraljevic, Thomas Searle, Anthony Shek, Lukasz Roguski, Kawsar Noor, Daniel Bean, Aurelie Mascio, Leilei Zhu, Amos A Folarin, Angus Roberts, Rebecca Bendayan, Mark P Richardson, Robert Stewart, Anoop D Shah, Wai Keong Wong, Zina Ibrahim, James T Teo, and Richard JB Dobson. 2020. Multi-domain clinical natural language processing with medcat: the medical concept annotation toolkit.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Scott Mayer McKinney, Alan Karthikesalingam, Daniel Tse, Christopher J Kelly, Yun Liu, Greg S Corrado, and Shravya Shetty. 2020. Reply to: Transparency and reproducibility in artificial intelligence. Nature, 586(7829):E17-E18.
Tomas Mikolov, Kai Chen, G. Corrado, and J. Dean. 2013. Efficient estimation of word representations in vector space. In ICLR.
Ishna Neamatullah, Margaret Douglass, Li-wei Lehman, Andrew Reisner, Maurizio Villarroel, William Long, Peter Szolovits, George Moody, Roger Mark, and Gari Clifford. 2008. Automated de-identification of free-text medical records. BMC medical informatics and decision making, 8:32.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Fabio Petroni, Tim Rocttäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language
Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463-2473, Hong Kong, China. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Radim Šehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45-50, Valletta, Malta. ELRA. http://is.muni.cz/publication/884893/en.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parameters of a language model? In EMNLP.
Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699-2712, Online. Association for Computational Linguistics.
A. Salem, Y. Zhang, M. Humbert, M. Fritz, and M. Backes. 2019. Mt-leaks: Model and data independent membership inference attacks and defenses on machine learning models. ArXiv, abs/1806.01246.
Congzheng Song and Vitaly Shmatikov. 2018. The natural auditor: How to tell if someone used your words to train their model. ArXiv, abs/1811.00513.
A. Stubbs, Özlem Uzuner, Christopher Kotfila, I. Goldstein, and Peter Szolovits. 2015. Challenges in synthesizing surrogate phi in narrative emrs. In Medical Data Privacy Handbook.
Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a Markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30-36, Minneapolis, Minnesota. Association for Computational Linguistics.
Figure A1: A distribution of ICD-9 codes and how many patients (of the 27K) have each condition. All bin end values are not inclusive.
A.1 Training Our BERT Models
As mentioned previously, we follow most of the hyperparameters stated in (Huang et al., 2019). The code presented in Huang et al. (2019) accidentally left out all notes under the category 'Nursing/Other'; we added these back in, in addition to any notes that fell under the 'Discharge Summaries' summary category. Our dataset consists of approximately 400M words (ignoring wordpieces). The number of epochs (following Devlin et al. 2019) can be calculated as
$$
\text {n u m} _ {\text {s t e p s}} \cdot \text {b a t c h} _ {\text {s i z e}} \cdot \frac {\text {t o k e n s} _ {\text {p e r} \text {s e q}}}{\text {t o t a l} \text {n u m b e r} \text {o f} \text {t o k e n s}}
$$