Revisiting De-Identification of Electronic Medical Records: Evaluation of Within- and Cross-Hospital Generalization
Authors: Yiyang Liu
Jinpeng Li
Enwei Zhu
Venue: EMNLP 2023 Main (2023)
Keywords: De-Identification
Electronic Medical Records
Domain Generalization

Abstract: The de-identification task aims to detect and remove the protected health information from electronic medical records (EMRs). Previous studies generally focus on the within-hospital setting and achieve great successes, while the cross-hospital setting has been overlooked. This study introduces a new de-identification dataset comprising EMRs from three hospitals in China, creating a benchmark for evaluating both within- and cross-hospital generalization. We find significant domain discrepancy between hospitals. A model with almost perfect within-hospital performance struggles when transferred across hospitals. Further experiments show that pretrained language models and some domain generalization methods can alleviate this problem. We believe that our data and findings will encourage investigations on the generalization of medical NLP models.

Revisiting De-ldentification of Electronic Medical Records: Evaluation of Within- and Cross-Hospital Generalization
Yiyang Liu and Jinpeng Li and Enwei Zhu
Ningbo No.2 Hospital
Ningbo Institute of Life and Health Industry, University of Chinese Academy of Sciences (liuyiyang, boammg,iijnnnenn jeciineen cn, enwei zhu@outlook.cm com
Abstract
However, the cross-hospital setting has been largely overlooked. This setting corresponds to a realistic scenario that a de-identification model is deployed across hospitals. This study aims to formally fill the gap. We introduce a new de- identification dataset comprising EMRs from three hospitals in China, establishing benchmark for evaluating both within- and cross-hospita. gener- alization. The latter poses challenging domain generalization (DG) task, where hospitals are re- ferred as domains.
The de-identification task aims to detect and remove the protected health information from electronic medical records (EMRs). Previous studies generally focus on the hospitl setting and achieve great successes, while the cross-hospital setting has been overlooked. This study introduces new de-identification dataset comprising EMRs from three hospi- tals in China, creating a benchmark for eval- uating both within- and cross-hospital gener- alization. We find significant domain discrep- ancy between hospitals. model with almost perfect within-hospital performance struggles when transferred across hospitals. Further ex- periments show that pretrained language mod- els and some domain generalization methods can alleviate this problem. We believe that our data and findings will encourage investigations the generalization of medical NLP models.'
We find significant domain discrepancy existing between hospitals. A model with almost perfect within-hospital performance encounters dramatic degradation when transferred other hospitals, Using pretrained language models (PLMs) like BERT (Devlin et al., 2019), or some existing DG methods (lzmailov et al., 2018; Wu al., 2022) can improve the cross-hospital generalization, but by limited margins. This paper contributes in twofold:
1 Introduction
De-identification is natural language processing (NLP) task detect and remove the protected health information (PHI) from electronic medical records (EMRs). It is a prerequisite to the distribu- tion EMRs outside their original institutions for medical NLP research (Uzuner et al., 2007).
To the best of our knowledge, our dataset pro- vides the first de-identification benchmark that has multiple sources for cross-hospital eval- uation. It is also the first de-identification task for Chinese EMRs. The dataset proba- bly interests researchers from a broader com- munity, since DG tasks have been scarce in NLP (Zhou al., 2022). We will release the data to facilitate further research.
Previous studies generally focus on the within- hospital setting, where the training and test data are from same hospital institution. This includes English tasks like 2006 i2b2 (Uzuner al., 2007), 2014 i2b2/UTHealth (Stubbs al.. 2015a), 2016 CEGS N-GRID (Stubbs al., 2017), and others in Swedish (Dalianis and Velupillai, 2010) and French (Grouin and Névéol, 2014). De- identification may be regarded as an easy task rela- tive to other NLP tasks, because simple rule-based shallow neural models can achieve 95%+ Fl scores (Liu et al., 2017; Dernoncourt et al., 2017).
From the DG perspective, our findings en- hance Stubbs et al. (2015a, 2017)'s argument that de-identiiication is not solved problem, even in the post-BERT era. Our experiments show the effectiveness of PLMs and DG meth- ods, providing promising direction for inves- tigations on the cross moss-hospital generalization of medical NLP models.
"Corresponding author.
'Our data and onde are available at https://githe con/ lmmmlanaangyagg0 Satpyaanyaagg///viiiiiiiene FReeissiiign-ai Seee-ddeniiiica

---

2 Data and Annotations
HM
SY
CCKS
2.1 Data Sources
Train
Dev
Test
HM. Our primary dataset is built from the EMRs #EMR
300
100
100
100
300
#Sentence
6,005
2,029
1,995
2,505
1,346
of HM- hospital, general hospital in Zhejiang Province, China. We obtained 1.7 TB backup of 671.5K inpatient records. The clinical text is stored in sections like chief complaints, examination re- ports, progress notes, and discharge summaries.
#PII Mention
PERSON
2,430
873
841
1,923
85
LOCATION
224
62
69
187
219
HOSPITAL
1,092
389
313
233
115
DATE
4,762
1,458
1,439
2,153
449
ID
129
32
31
22
CONTACT
207
64
67
36
We sample 500 EMRs from 30 representative medical departments for PHI annotation. The re- sulting annotated corpus is randomly split into train- ing/developmen/test sets with 300/100/100 100/100 EMRs. This is the dataset for within-hospital evaluation. In addition, the HM database provides the large- scale clinical text for pretraining Word2Vec em- beddings and BERT, named HM-Word2Ver and HM-BERT, respectively. This mitigates the poten- tial domain gap transferring word embeddings or PLMs trained on common corpora to clinical text. See Section 3 for more details.
AGE
1,087
401
370
253
430
PROFESSION
197
66
66
15
21
Table Descriptive statistics datasets.
B-100
HLOG
HLOG
CRF Layer (pticaal)
Derse: Layer
SY. We collect 100 EMRS from SY hospital, general hospital in Hunan Province, China. Sim- ilar to those of HM hospital, the EMRs contain various sections of clinical text. After the anno- tation of PHI, SY dataset serves as a test set for cross-hospital evaluation.
Encoder Layer
CCKS. China Conference on Knowledge Graph and Semantic Computing (CCKS) 2017 released clinical named entity recognition task, which con- tains 300 EMRs from anonymous hospital in
Figure I: The architecture of our model.
AGE: Ages of patients.
Hebei Province, China. Each EMR includes four paragraphs from specific sections. We annotate the
PROFESSION: The professions of patients.
PHI on the available text, yielding another test set.
The BRAT Rupid Annotation Tool (Stenctorp et al., 2012) is employed for the PHI annotation. Table reports the descriptive statistics of the re- sulting datasets. Appendix A provides some typical examples of the PHI annotations clinical text.
2.2 Protected Health Information
Following previous research (Uzuner et al., 2007; Stubbs et al., 2017), we define eight PHI categories:
PERSON: The names of patients or physicians.
Models and Experimental Settings
LOCATION: Addresses patients.
HOSPITAL: The names of hospitals.
Model Architecture. Figure displays the ar- chitecture of our model. We employ the neu- ral sequence tagging framework (Collobert et al., 201 1), a widely-used and mature solution for the de- de--ideniifiationn task.
DATE: Date or time stamps.
ID: IDs of patients or medical tests.
CONTACT: The contact information of patients, physicians or hospitals.
Specifically, the tokens are first mapped to beddings and fed into the encoder. The encoder can be a ID CNN (Zhang et al., 2015), a BiL.- STM (Hochreiter and Schmidhuber, 1997b), or a pretrained Transformer (Devlin et al., 2019), which
NNinboo No.2 Hospital.
'The hospital name anonymized policies regula-
tions.
"See https://ww.sigkg c0/1.

---

transforms the embeddings hidden representa- tions. Finally, the dense layer classifies the repre- sentations into the pre-defined BIO tag space, and the resulting BIO tags can be parsed to identify the boundaries and categories of PHI mentions. An op- tional conditional random field (CRF) layer can be inserted after the dense layer, which may improve the consistency of the predicted BIO tags.
Hyperparameters." 6 For CNN or BiLSTM mod- els, the embedding layer is 100-dimensional, and optionally initialized from SMM-WordVee;; the en- coders have one layer with 200 hidden states and dropout rate of 0.5. The kernel size is 15 for CNN. The models are trained for 100 epochs by the AdamW (Loshchilov and Hutter, 2018 8) optimizer with learning rate 1e-3 and batch size 32.
For models with PLMs, we use BERT-wwm (Cui et al., 2021), MC-BERT (Zhang et al., 2020) and HM-BERT, all of base size (768 hidden size, 12 layers). The MC-BERT is pre-trained on Chinese medical corpora including biomedical question an- swering, medical encyclopedia and EMRs. The models are trained for 100 epochs by the AdamW optimizer with learning rate 2e-5 and batch size 32.
Pretraining on EMRs. Most publicly available Chinese word embeddings or PLMs are trained on common corpora, which may encounter domain gaps and result in sub-optimal performance when transferred to the clinical domain. To alleviate this issue, we pretrain Word2Vec embeddings (Mikolov et al., 2013) and Chinese BERT (Devlin et al., 2019) on the large-scale clinical text from the HM database. The resulting embeddings and model are named HM-Word2We and HM-BERT, respec- tively.
Evaluation. predicted PHI mention is consid- ered correct if its boundaries and category exactly match the ground truth. The evaluation metrics are micro precision rate, recall rate and FI score on the test sets, All the experiments are repeated for five times and the average metrics are reported.
After data parsing, cleaning and deduplication, the remaining HM corpus consists of 21.4K EMRs, including clinical text of 2.8 GB (1.1B tokens). Word has character-level vocabulary of size 5.7K and embedding size of 100. It is trained for 5 epochs by the Gensim (Rehüfek and Sojka, 2010) package with window size 15 and learning rate 1c-3.
4 Experimental Results
4.1 Main Results
Table 2 presents the results for both within- and cross-hospital evaluation. In the within-hospital evaluation, single-layer CNN BiLSTM can achieve 98% F1 scores higher. This is consistent with the results of previous literature that the de- identification task can be almost perfectly solved by simple models (Dernoncourt et al., 2017; Liu et al., 2017). We add to the findings that more sophisti- cated neural models like BERT can further improve the performance, although by limited magnitudes. However, the cross-hospita setting has largely been overlooked in literature. With the help of our multi-source data, we evidence that decent neural de-identification model easily encounters noticeable performance degradation when trans- ferred across hospitals. Specifically, for CNN and BiLSTM, the F'l scores decrease to 70%-80% and 50%-60% when transferred to SY and CCKS, re- spectively. The HM-Word2Vec embeddings and CRF help to resist the performance drop, but the ef- fect is not robust. The BERT-based models also suf- fer from the cross-hospital setting, but they achieve much better results: 95%+ Fl scores on SY and
Following Cui et al. (2021), we initialize HM- BERT from the bert-base-chines checkpoint re- leased by Hugging Face. 5 We then pretrain the model with the optimizer AdamW (Loshchilov and Hutter, 2018), learning rate le-4 and batch size 384 for 20 epochs on the whole corpus. A scheduler of linear warmup in the first 20% steps followed by linear decay is applied. The masking rate is 15% for the masked language modeling (MLM) task; the maximum input length is 512. In addition, we apply the whole word masking (Cui et al., 2021) and dynamic masking (Liu et al., 2019) strategies.
Within- and Cross-Hospital Evaluation. We train the models on the training set of HM, and use the development set for hyperparameter tuning. We perform the vithin-hospital evaluation, i.e., evalu- ating the trained models on the test set of HM; and the cross-hospae pross-hospital evaluation, i.e., evaluating the models on SY and CCKS.
"All the hyperparameters have been extensively tuned and thus empirically optimal. We have tested models with larger sizes, but resulted in lower performance.
'See https://huggingface co/bert

---

HM-+HI
HM-+SY
HM HM-CCCKS
Speed
Params
FLOPs
(sents/s)
Prec.
Rec.
F1
Prec.
Rec.
F1
Prec.
Rec.
FI
CNN
97.9
98.6
98.2
70.6
76.4
73.4
51.3
57.6
54.2
576.6K
101.3M
500
+ HMM-Word2Ves
97.5
98.5
98.0
76.6
80.9
78.7
49.7
$5.0
52.2
576.6K
101.3M
500
+ CRF
95,4
95.0
95.2
70.1
66.8
68.4
64.8
59.3
61.9
577.0K
101.3M
25
BiLSTM
98.4
98.9
98.6
80.2
77.7
78.9
48.5
58.2
52.8
437.6K
51.0M
272
+ HIM-Word2Ves
98.8
99.0
98,9
82.6
82.7
82.6
44,8
55.0
49.3
437.6K
51.0M
272
+ CRF
97.6
96.8
97.2
78.5
69.3
73.6
64.2
58.4
61.0
438.0K
51.0M
25
BERT-wwm
99.3
99.5
99.4
96.2
97.8
97.0
75.9
77.5
76.7
102.6M
26.4G
22
+ BiLSTM
99.5
99.4
99.5
97.0
97.8
97.4
74.4
72.9
73.7
103.3M
26.6G
21
+ CRF
99.1
99.2
99.1
94.9
96.7
95.8
70.3
71.3
70.8
102.6M
26.4G
13
MC-BERT
99.4
99.5
99.5
96.2
97.8
97.0
75.9
77.5
76.7
102.6M
26.4G
22
+ BiLSTM
99.5
99.5
99.5
96.5
97.8
97.2
74.7
77.7
76.1
103.3M
26.6G
21
+ CRF
99.0
99.1
99.0
94.8
96.7
95.7
70.3
71.3
70.8
102.6M
26.4G
15
HM-BEE -ERT
99.6
99.8
99.7
96.2
97.2
96.7
73.6
84.0
78.5
102.6M
26.4G
22
+ BiLSTM
99.7
99.7
99.7
96.6
97.8
97.2
78.2
85.7
81.8
103.3M
26.6G
21
+ CRF
99.4
99.7
99.5
92.7
95.5
94.1
70.8
79.4
74.8
102.6M
26.4G
15
Table Results of within- and cross-hospital evaluation of de-iddentification. The models are trained on the training set of HM, and evaluated on the test set of HM, SY and CCKS, respectively.
70%-80% on CCKS. In particular, HM-BERT out- performs BERT-wwm and MC-BERT on CCKS.
ACS
FAR9S
HM-train HM-test
92.05
94,64
These results clearly reveal a noteworthy prob- lem - severe domain discrepancy exists between EMRs from different hospitals. It may greatly im- pede the cross-hospital applicability of perfectly- performing model. Empirically, using PLMs can effectively alleviate this problem. The PLMs learn universal linguistic patterns from large-scale pre- training data, which help the models generalize across hospitals.
HM-train SY
84.50
92.05
HM-train CCKS
82.51
73.03
Table ACS and FAR95 of HM training set against HM test set, SY and CCKS sets.
ilarities between them. The average cosine simi- larity (ACS) over all the test instances is used an indicator to measure the overlapping extent of training and test data.
When focused on within-hospital evaluation, we may conclude that CNN or BiLSTM models are superior to BERT-based models, because the for- mer ones achieve similar performance with better efficiency (fewer parameters and FLOPs, higher speed). However, with the awareness of cross- hospital results, we have to rethink this problem. Appendix B reports the categorical evaluation results. To avoid PHI leaks and preserve the data usability, we have carefully replaced the PHI men- tions by realistic surrogates (Stubbs et al., 2015b) in the release version. This slightly affects the evaluation results, report the corresponding results in Appendix D.
False Alarm Rate. Following Hendrycks et al. (2020), we assign the maximum softmax anomaly score for each test instance, to perform out-of- distribution (OOD) detection, and report the false alarm rate at 95% recall (FAR95).
Table 3 shows that the CCKS data present the most different distributions from the HM training set. In other words, the overlap between HM and CCKS is the lowest. Hence, the CCKS has repre- sentations of more OOD information that the mod- els fail to learn. This is the major reason for the large performance drop on CCKS. We further per- form the PCA visualizations of sentence represen- tations on the HM training/test sets, SY. and CCKS. The results are quite similar. See Appendix C for more details.
4.2 Analysis
We explore some potential reasons for the signif- icant gap between the within- and cross-hospital performance.
4.3 Domain Generalization
Cosine Similarity. Following Flangovan et al. (2021), we represent each data instance with the vector of HM-BERT and compute the cosine sim-
Given the domain shift between hospitals, we ex- plore some widely used DG methods, verifying whether they can help the models to generalize

---

CNN
HM-BERT
HM-HM
HM-SY
HM-CCKS
HM-+H
HM-SY
HM-CCKS
Baseline (w/ DG methods)
98.2
73.4
54.2
99.7
96.7
78.5
Mention Substitution
98.3 (+0.1)
56.7 (-16.7)
57.2 (+3.0)
99.4
92.2 (+4.5)
82.3 (+3.8)
Text Smoothing
99.7 (+0.0)
97.0 (+0.3)
80.7 (42.2)
Stochastic Weight Averaging
25%
98.3 (+0.1)
74.7 (+1.3)
54.3 (+0.1)
99.8 (+0.1)
(40.5)
80.5 (42.0)
50%
98.3 (+0.1)
(41.1)
55.5 (+1.3)
99.8 (+0.1)
97.3 (40.6)
79.4 (40.9)
75%
98.3 (+0.1)
74.3
56.1 (+199
99.8 (+0.1)
(40.7)
80.6 (42.1)
Dropout (def. = 0.50)
0.25
98.4 (+0.2)
75.5 (+0.2)
50.1 (4.1))
99.6 (-0.1)
96.7 (+0.0)
77.7 (-0.8)
0.75
94.8 (-3.4)
66.4 (-7.0)
48.7 (-5.5)
99.7 (+0.0)
96.7 (+0.0)
81.1 (+2.6)
L2 Regularization (def. = 0.01)
0.05
98.0 (-0.2)
74.1 (+0.7)
51.1 (-3.1)
99.6 (-0.1)
96.2 (-0.5)
79.5 (+1.0)
0.10
98.0 (-0.2)
74.3 (+0.9)
50.5 (-2.7)
99.7 (+0.0)
96.8 (+0.1)
78.4 (-0.1)
Table 4: Results of within- and cross coossthospital evaluation of domain generalization methods. The models are trained on the training set of HM, and evaluated on the test set of HM, SY and CCKS, respectiely...
across hospitals. The results shown in Table 4.
In summary, text smoothing and SWA effec- tively improve the generalization performance by small margins, while the other DG methods result in negative or unstable effects.
Mention Substitution. Data augmentation has been a common practice to prevent the models from overfitting and thus improve generalization (LeCun et al., 2015; Zhou et al., 2022). To avoid overfitting to frequent PHI mentions, we add augmentation module that randomly replaces the PHI mentions with fabricated ones of the same categories. For example, a LOCATION mention can be replaced by random addresses. The resulting effect is unstable.
Conclusion
In this paper, revisit the EMR de- de-identification task and create new dataset. It consists of EMRs from three hospitals and thus asks evaluation for both within and cross-hospital generalization. The latter poses a challenging DG task, which corre- sponds to a realistic scenario that a de-identification model is required to deploy across hospitals.
Text Smoothing. Text smoothing (Wu et al., 2022) is PLM-based text augmentation approach. It leverages the masked language modcling objec- tive of a dedicated PLM and augments each token according to the predicted probabilities over the vocabulary. Text smoothing slightly improves the generalization performance on SY and CCKS.
With this new benchmark, find significant domain shift between hospitals. model with per- feet within-hospital performance struggles when transferred across hospitals. Using PLMs or some existing DG methods can alleviate but not address this problem.
Stochastic Weight Averaging. Stochastic weight averaging (Izmailov et al., 2018, SWA) aggre- gates the model weights along the training trajec- tory. The ensemble model can achieve flatter min- ima (Hochreiter and Schnichuber, 1997a) and thus improves generalization. Specifically, we average the model checkpoints at last 15, 50%, and 75% training epochs for test. It shows that SWA brings marginal yet robust improvements for either within- coopstt performance.
Limitations
Although we have explored various existing mod- els and DG methods on our proposed task, there are still other approaches worth investigating. It may be more effective to develop new specialized meth- ods to improve the cross- cooss-hsspttall generalization medical NL. models.
In general, this study focuses on end-to-end eval- uation. It requires more in-depth analysis, either theoretically or empirically, answer some crucial questions like (1) how and what extent is the i.i.d. assumption violated between the clinical text from different hospitals? (2) how to develop invariant representations that generalize across hospitals? (3)
Dropout and L2 Regularization. Dropout (Sri- vastava al., 2014) and L2 regularization are stan- dard strategies against overfitting. We tune the dropout rate and L2 penalty to different values, but obtain lower scores.

---

how to estimate the risk of generalization failure before the models are deployed? These answers may be the key towards interpretable, robust and reliable medical NLP systems.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre-training with whole word masking for Chinese BERT. IEEF/ACM Transac- rions on Audio, Speech, and Language Processing, 29:3504-3514
7 Ethical Considerations
Hercules Dalianis and Sumithra Velupillai. 2010, De- identifying Swedish clinical text-refinement of a gold standard and experiments with conditional random fields. Journal of Biomedical Semantics, 1-0..
Our data were collected and used consistently with the terms of use. The EMRs HM and SY were obtained from the authors' affiliations, and this work was performed as a part of projects approved by the ethics review committees. CCKS was de- rived from publicly available data released shared task. We authors performed the PHI annota- tion, with full awareness of the potential impacts and risks.
Franck Demonccurt, Young Lee, Ozlem Uzuner, and Peter Szolovits. 2017. De-identification of pa- tient notes with recurrent neural networks. Journal of the American Medical Informatics Association, 24(3):596-60 606.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
The data of release version have been de- identified. Specifically, we have carefully replaced all the PHI mentions by surrogates (Stubbs et al.. 2015b), and manually verified the resulting text so that the risk of privacy leak has been minimized. particular, any text span regarded as PHI and removed if potentially reveals any identity char- acteristics, even if the risk is impossibly low. In addition, the data will be released upon a data use agreement that forbids any inappropriate use, es- pecially identification of any individuals or institu- tions.
Aparna Elangovan, Jiayuan He, and Karin Verspoor. 2021. Memorization vs. generalization: Quantify- ing data leakage in NLP performance evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisties: Main Volume, pages 1325-1335, Online. Association for Computational Linguisties.
We report data characteristics and experimental results averaged over multiple runs. We will release the data and code, ensuring the reproducibility of our work. Our experiments do not require high computational cost, relative to pretraining PLMs.
Cyril Grouin and Aurélie Névéol. 2014. De- identification of clinical notes in French: towards protocol for reference corpus development. Jour- nal of Biomedical Informatics. 50:151-161.
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace, Adam Dziedzic, Rishabh Krishnan, and Dawn Song. 2020. Pretrained transformers improve -iitt--iitti robustness. In Proceedings the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 2744-2751, Online. Association for Computa- tional Linguistics.
Acknowledgements
We thank the anonymous reviewers for their in- sightful comments and feedback. This work is supported by Zhejiang Provincial Natural Sci- ence Foundation of China (No. LQ23F020005), National Natural Science Foundation of China (No. 62106248), Ningbo Science and Technol- ogy Service Industry Demonstration Project (No. 2020F041), and Ningbo Public Service Technology Foundation (No. 2021S152).
Sepp Hochreiter and Jürgen Schmidhuber. 1997a. Flat minima. Neural Computation, 91):
Sepp Hochreiter and Jürgen Schmidhuber. 1997b. Long short-term memory. Neural Computation, 98): 1780.
P Izmailov, AG Wilson, D Podoprikhin, D Vetrov, and Garipov. 2018. Averaging weights leads to wider optima and better generalization. In 34th Conference Uncertainty in Artificial Intelligence 2018, UAI 2018, pages 876-885.
References
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12(ARTIC1LE22495-2555
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature, 521(7533)4366444

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zet,lemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint urXiv:1907.1162
various data augmentation methods on text classifica- tion tasks. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers), pages 871-875, Dublin, Ireland. Association for Computational Linguistics.
Zengjian Liu, Buzhou Tang. Xiaolong Wang, and Qing- cai Chen. 2017. De-identification of clinical notes via recurrent neural network and conditional random field. Journal of Biomedical Informatics, 75:S34- S42.
Ningyu Zhang, Qianghuai Jia, Kangping Yin, Liang Dong, Feng Gao, and Nengwei Hua. 2020. Conceptu- alized representation learning for chinese biomedical text mining, arXiv preprint arXiv:2008. 10813.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in Neural Information Processing Systems, 28.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Confer- ence on Learning Representations.
Tomas Mikolov, llya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. Advances in Neural Information Processing Systems, 26.
Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. 2022. Domain generalization: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence.
Radim Rehüřek and Petr Sojka. 2010. Software frame- work for topic modelling with large corpora. In Pro- ceedings of the LREC 2010 Workshop on New Chal- lenges for NLP Frameworks, pages 45-50, Valletta, Malta. ELRA. http://is. muni. publlicatiion/ 884893/en.
A Examples of PHI Annotations
All the EMRs are manually annotated by two native speakers. (A master's degree in computer science and an expert in both the fields of computer science and medicine.) We consider the de-identification task to be easy and straightforward, with very few disagreements during labeling and near- pear-perfect ac- curacy. We employed some post- post-processing pro- cedure to ensure the accuracy of the annotations. For example, we perform cross-validation on the whole dataset, and manually check the inconsis- tencies between the predicted PHI mentions and ground-truth.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 1919:1999-
Pontus Stenetorp. Sampo Pyysalo, Goran Topić. Tomoko Ohta. Sophia Ananiadou, and Jun' ichi Tsujii. 2012. BRAT: a web-based tool for NLP-assisted text annotation. In Proceedings of the Demonstrations Session at EACL 2072, Avignon, France. Association for Computational I Linguistics.
Table 5 shows examples of clinical text and PHI annotations. In principle, a text span is regarded as PHI if it potentially reveals any personal informa- tion, even if the risk is impossibly low.
Amber Stubbs, Michele Filannino, and Özlem Uzuner. 2017. De-identification of psychiatric intake records: Overview of 2016 CEGS N-GRID shared tasks track 1. Journal of Biomedical Informatics, 75:54-S18.
Some text spans excluded because of irrele- vance to personal information, although they seem- ingly fall into specific PHI categories. For example, some diseases are named after persons or locations, and these person or location names should not be regarded as PHI. Some clinical text may describe general medical knowledge that relates to ages or jobs, which are also excluded in our annotation.
Amber Stubbs, Christopher Kotfila, and Özlem Uzuner. 2015a. Automated systems for the de-identification of longitudinal clinical narratives: Overview of 2014 i2b2/UTHealth shared task track 1. Journal of Biomedical Informatics, 58:511-55 1-S
Amber Stubbs, Özlem Uzuner, Christopher Kotfila, Ira Goldstein, and Peter Szolovits. 2015b. Challenges in synthesizing surrogate PHI narrative emrs, Medi- cal Data Privacy Handbook, pages 717-735.
B Categorical Results
Özlem Uzuner, Yuan Luo, and Peter Szolovits. 2007. Evaluating the tfae-oo-the-ar in automatic de- identification. Journal of the American Medical In- formatics Association, 1455-555--663
Table 6 presents the evaluation results by PHI cate- gories. The cross-hospital F1 scores are lower than the corresponding within-hospita scores across all categories, while such effect is heterogeneous, Tak- ing HM-BERT as the example, it generalizes rel- atively well on PERSON, DATE and AGE categories,
Xing Wu, Chaochen Gao, Meng Lin, Liangjun Zang, and Songlin Hu. 2022. Text smoothing: Enhance

---

Category
Clinical text and PHI annotations
PERSON
IShu Jiangl. associate chief physician, ward rounds. #
#+ fif (. i (I..
The patient got [Parkinsonl's syndrome seven years ago.
LOCATION
2 4: ( 1.
DR K
The patient was born in IXinchang Countyl and grew up there.
B M Hi
[bblaa] virus disease causes fever, throat, muscle pain, and headaches. BiC 19.
HOSPITAL
#
The patient went to (Shanghai Traditional Chinese Medicine Hospital] for diagnosis. 12019 12019 1, $ 1A ABE.
DATE
[October 2, 2019), the patient was admitted because of abdominal pain. МиCT ([US00786])
ID
1E:T5 (
Chest CT ([US007866) suggests tracheal stenosis.
CONTACT
7 1 (88121834).
Outpatient phone number: (88121834).
L [120).
Emergency phone number: [120].
AGE
# (16851)) T Bi
The patient (168 years old)) was admitted for ankle fracture.
i El H. (00 BBR00//BBE...... E
High blood pressure tends to occur people over [80 years old].
PROFESSION
# #.. 5. # AUN:: [IA]].
The patient, male, graduated from junior high school, was [worker].
[Miners] and [construction workers] are vulnerable to pncumoconiosis.
Table 5: Examples of clinical text and PHI annotations. The Chinese texts are translated to English for reference. The PHI mentions are marked [*] while the non-PHI but possibly confusing mentions are marked in blue [*].
CNN
HM-BFR
HM-HH
HM--SY
HM-CCKS
HM-HHM
HM-+SY
HM--CCKS
PERSON
98.7
69.2
48.9
99.8
98.0
93.2
LOCATION
94.2
22.0
6.2
97.6
92.9
30.7
HOSPITAL
95.9
43.8
25.0
99.2
95.3
66.7
DATE
98.3
86.1
76.3
99.8
97.0
89.0
ID
89.9
13.5
24.2
97.8
78.9
65.3
CONTACT
99.7
15.5
100.0
85.7
AGE
100.0
95.7
72.8
100.0
96.4
98.4
PROFESSION
99.2
31.2
11.5
99.2
45.5
40.3
Overall
98.2
73.4
54.2
99.7
96.7
78.5
Table Categorical Fl scores of within- and cross-hospital evaluation. The models are trained on the training set of HM, and evaluated on the test set of HM, SY and CCKS, respectively.
but struggles on LOCATION, ID and PROFESSION. This plausibly attributes to that the former PHI categories are typically associated with clearer lan- guage patterns than the latter ones.
datascts in the feature space. In comparison, there small overlap between the SY dataset and the HM dataset.
Visualization of Sentence Representations
Further underlying reasons for the differences in the data distributions would be complicated. For example, different doctors may record information differently, resulting in differences in the format and content of medical records. Different medical institutions may use different electronic medical record systems.
Figure 2 displays the PCA visualizations of sen- tence representations the HM training/test sets, SY. and CCKS. The sentence representations of the HM training and test sets are most similar and overlapping. However, there are significant differ- ences in the distributions of the HM and CCKS

---

HM--IM
HM-+S
HM-+CCKS
Prec.
Rec.
Fl
Prec.
Rec.
F1
Prec.
Rec.
FI
CNN
97.8
98.5
98.1
73.4
75.0
74.2
48.5
55.3
51.7
ViM-Word2Vec
97.8
98.7
98.3
75.3
79.5
77.4
50.8
54.1
52.4
CRF
96.1
95.5
95.8
75.0
70.5
72.7
63.1
58.9
60.9
BiLSTM
98.4
98.9
98.6
78.8
77.8
78.3
45.0
56.6
50.1
Vec
98.8
99.0
98,9
80.7
81.4
81.0
49.2
57.4
52.9
CRF
97.7
97.1
97.4
83.2
75.3
79.0
57.5
55.1
56.2
BERT-wwm
99.4
99.8
99.6
95.8
97.5
96.7
76.2
80.4
78.2
BiLSTM
99.5
99.6
99.6
96.6
98.0
97.3
78.1 1
78.8
78.4
CRF
99.1
99.5
99.3
94.5
97.1
95.8
75.4
75.3
75.3
MC-BERT
99.4
99.5
99.5
95.8
97.7
96.7
74.9
78.5
76.6
BiLSTM
99.4
99.7
99.6
96.2
98.0
97.1
75.5
75.2
75.3
CRF
99.2
99.5
99.4
94.1
96.8
95.4
76.5
75.2
75.8
TIM-ERRT
99.6
99.8
99.7
95.6
97.4
96.5
75.5
84.6
79.7
BiLSTM
99.6
99.6
99.6
96.3
97.6
97.0
80.7
85.9
83.2
CRF
99.4
99.6
99.5
93.1
95.9
94.5
73.4
78.2
75.7
Table Results of within- and cross-hospital evaluation data of release version. The models trained the training set of HM, and evaluated on the test set of HM, SY and CCKS, respectively.
CCKS Set
HM Test Set HM Train
-2
-3
-2
Figure 2: The PCA visualizations of sentence represen- tations the HM training/test sets,, SY, and CCKS.
Results on Data of Release Version
We cannot release our data with the real PHI men- tions. Hence, we have carefully replaced the PHI mentions by realistic surrogates (Stubbs et al., 2015b). For example, the PERSON mentions are re- placed by combinations of randomly sampled fam- ily and given names, where the sampling accords to the frequencies reported by National Bureau of Statistics of China. The LOCATION mentions are replaced by randomly sampled addresses in China. Such process preserves the usability of our data and prevent PHI leak simultancously.
Table 7 presents the corresponding evaluation re- sults, which are highly consistent with those on the original data. Hence, experimental results reported on our release data are still indicative.


Reviews:
[{"review_id": "BJmg7VmlJz", "reviewer": "Reviewer_FbUh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We reply to your concerns as follows:\n\n> The paper does not present any relevant contribution besides the corpus they built. The techniques employed are the standard ones, their conclusion, namely that cross-hospital results are worst than in-hospital results is the expected one. \n* Thanks for your comments. This paper was submitted to the track \"Resources and Evaluation\"; and indeed, the corpus (and associated evaluation) is our primary contribution. Just like most papers previously published in EMNLP's \"Resources and Evaluation\" track, new modeling techniques are beyond the scope of our paper. \n* Our corpus has noticeable merits and contributions. As summarized in Lines 58-65, our dataset provides the first multi-source de-identification benchmark. The dataset probably interests researchers from a broader community, since DG tasks have been scarce in NLP. We will publicly release the data. \n* Our evaluation results are also valuable. As you point out, it is an expected finding that \"cross-hospital results are worst than in-hospital results\", but our evaluation provides *quantitative* instead of *qualitative* results. In practice, the performance drop from 99% to 97% (e.g., HM->SY) is probably acceptable, while drop of 99% to 81% (e.g., HM->CCKS) is probably unacceptable. In that sense, *quantitative* results are more informative and thus important. \n\n\n> The drop in F1 score is only of a 2% with PLM and increases when adding sequentiality (BiLSTM + CRF) to the system, showing that the structure of the documents probably lies behind that drop, and the generalization techniques proposed do not tackle structural differences. \n* We highly appreciate your comments. When a model trained on HM is transferred to SY, the drop in $F_1$ score is 2%; when the model is transferred to CCKS, the drop is about 18%. We agree with you that the document structures are different between datasets. From a neural modeling perspective, such differences cause distribution shifts of neural representations, and finally result in the performance drop. \n* We employ some existing domain generalization (DG) methods, and verify whether they can help the models to generalize across hospitals. Experimental results show that some DG methods can *alleviate*, but *not fully address*, the domain gap issue (Table 3). Hence, our corpus poses a new, realistic, and challenging DG task (Lines 262-265). It highlights the importance of DG research in medical NLP: even a perfectly performing model can encounter dramatic degradation when transferred across hospitals. \n* Negative findings are also implicative in practice. Our results of the ineffectiveness of some DG methods still provide a descent starting point for future research. \n\n\n> Finally, as no error analysis is done, a table with the percentage of error of each PHI category might be sufficient, it is not possible to measure to which extend the errors are very relevant because errors not identifying age or the hospital are not the same as error not identifying the names of patients or doctors.\n* Thank you for the comments. Table 5 (in Appendix C) presents the categorical $F_1$ scores of within- and cross-hospital evaluation. We also provide some analysis on the categorial results (Lines 474-483), which are exactly consistent with your understandings that the performance drop is heterogeneous across PHI categories.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Although the corpus authors built is not available yet, they claim that they will make it available as soon as the paper gets accepted.", "reasons_to_reject": "The paper does not present any relevant contribution besides the corpus they built. The techniques employed are the standard ones, their conclusion, namely that cross-hospital results are worst than in-hospital results is the expected one. The drop in F1 score is only of a 2% with PLM and increases when adding sequentiality (BiLSTM + CRF) to the system, showing that the structure of the documents probably lies behind that drop, and the generalization techniques proposed do not tackle structural differences. \nFinally, as no error analysis is done, a table with the percentage of error of each PHI category might be sufficient, it is not possible to measure to which extend the errors are very relevant because errors not identifying age or the hospital are not the same as error not identifying the names of patients or doctors.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper deals with the EHR de-identification problem in a cross-hospital context. The authors build a corpus (that they say they will make available) with EHRs with three different hospitals and build or finetune different neural models to pursue the de-identification task showing that the cross-hospital results get degraded due to the lack of generalization. They also explore several domain generalization techniques in order to prevent the mentioned problem when there is only the possibility of training with one unique hospital dataset, and they show that only Stochastic Weight Averaging and text smoothing bring slight improvements. \n\nTheir main contribution is the corpus they built.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VjiVHKz3bu", "reviewer": "Reviewer_HyJ9", "summary": "", "questions": "A) What are the specificities of the CCKS dataset? Do the authors have any hypothesis regarding the drop in performance? Could they identify any major differences between the documents?\n\nB) Details on the annotation procedure: How many annotators were there? What was their expertise? Was it a single or a double annotation (i.e. was each report annotated by one annotator, or by two annotators followed by adjudication over disagreements?)? Was inter-annotator agreement computed?\n\nC) What are the descriptive statistics of each data split in the HM corpus?\n\n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your insightful and constructive comments. We reply to your concerns as follows:\n\n> The authors chose to build their own medical BERT-based model (HM-BERT) and ignored existing pre-trained medical models such as that of Zhang et al. 2020 (see ref below). It would have been interesting to test both models.\n* We greatly appreciate your suggestion. The below table shows our evaluation results of MC-BERT vs. HM-BERT. The results are quite similar. The de-identification model encounters significant performance degradation when transferred across hospitals. \n \n|   |MC-BERT  | HM-BERT  |\n|--|--|--|\n| HM->HM   | 99.4 | 99.7 |\n| HM->SY   | 97.0 | 96.7 |\n| HM->CCKS | 76.7 | 78.5 |\n\n> The analysis of the results could have been more thorough. Specifically, investigating the reason for the much lower performance on the CCKS dataset would have been interesting (see questions below). \n\n> A) What are the specificities of the CCKS dataset? Do the authors have any hypothesis regarding the drop in performance? Could they identify any major differences between the documents?\n* Many thanks for the question. Following another reviewer's suggestion, we calculate the cosine similarities and perform out-of-distribution (OOD) detection between the HM training set and the three test sets. The below table reports the results. The results show that the CCKS data present the most different distributions from HM train set; in other words, the overlap between HM and CCKS is the lowest. Hence, the CCKS has representations of out-of-distribution information that models fail to learn. This is the major reason for the large performance drop on CCKS. \n\n|   |Elangovan et al., EACL 2021 (cosine similarity)  |Hendrycks et al., ACL 2020 (False Alarm Rate at 95% Recall)  |\n|:--|:--:|:--:|\n| HM train vs. HM test | 92.05%|  94.64% |\n| HM train vs. SY |  84.50%| 92.05% |\n| HM train vs. CCKS |  82.51% |  73.03%|\n\n* We further perform the PCA visualizations of sentence representations on the HM training/test sets, SY, and CCKS. (It seems that we cannot upload the figure to OpenReview.) The sentence representations of the HM training and test sets are most similar and overlapping. However, there are significant differences in the distributions of the HM and CCKS datasets in the feature space. In comparison, there is a small overlap between the SY dataset and the HM dataset. \n* Further underlying reasons for differences in the data distributions would be complicated. For example, different doctors may record information differently, resulting in differences in the format and content of medical records. Different medical institutions may use different electronic medical record systems. \n* We will add the above analysis in our revised manuscript. \n\n\n> B) Details on the annotation procedure: How many annotators were there? What was their expertise? Was it a single or a double annotation (i.e. was each report annotated by one annotator, or by two annotators followed by adjudication over disagreements?)? Was inter-annotator agreement computed? \n* All the EMRs are manually annotated by two native speakers (a Master’s degree in computer science and an expert in both the fields of computer science and medicine.). We consider the de-identification task to be easy and straightforward, with very few disagreements during labeling and near-perfect accuracy. Therefore, we did not compute the inter-annotator agreement. However, we employed some post-processing procedure to ensure the accuracy of the annotations. For example, we perform cross-validation on the whole dataset, and manually check the inconsistencies between the predicted PHI mentions and ground-truth. \n* We will perform another round annotation, and report inter-annotator agreement in the revised manuscript. Thanks for your suggestion. \n\n\n> C) What are the descriptive statistics of each data split in the HM corpus?\n* We sincerely appreciate your suggestion. The below table reports the descriptive statistics of HM train/dev/test splits, SY and CCKS datasets. We will replace the current Table 1 with the below table in the revised manuscript. \n\n|   | HM Train | HM Dev. | HM Test | SY | CCKS |\n|  :-- | --: | --:|  --: | --: | --:| \n| **#EMR**  | 300|  100 |100 | 100|  300 |\n| **#Sentence**|  6,005| 2,029 |1,995  | 2,505|  1,346 |\n|**#PHI Mention** |  \n| PERSON         |  2,429    | 873      | 841    | 1,923     | 85 |\n|LOCATION       | 220       |  62       | 70       |187       |  219 |\n|HOSPITAL        |  1,091    | 389      | 313     | 233      |  115 |\n|DATE                |  4,755    |  1,458   |1,420    |2,153      | 449 |\n|ID                      | 128       | 32        |31        | 22         |  9 |\n|CONTACT         |  207     |  64        |67       | 36         |  0 |\n|AGE                  |  1,087    | 401      | 369     |253        | 430 |\n|PROFESSION  |  196      | 66        | 66       |15         | 21 |\n\n\n> page 1, line 35: \"may be regarded as a easy task\" -> \"may be regarded as an easy task\"\n* Thanks for pointing this out. We will revise it accordingly.\n\n\n1. Zhang et al. 2020. Conceptualized Representation Learning for Chinese Biomedical Text Mining. *arXiv preprints arXiv*: 2008.10813.\n2. Aparna et al. 2021. Memorization vs. Generalization: Quantifying Data Leakage in NLP Performance Evaluation. In *EACL 2021*. \n3. Hendrycks et al. 2020. Pretrained Transformers Improve Out-of-Distribution Robustness. In *ACL 2020*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper tackles an important issue for medical NLP applications\n- The dataset is very valuable, as clinical corpora are not numerous, particularly cross-hospital and non-English ones.\n", "reasons_to_reject": "- The authors chose to build their own medical BERT-based model (HM-BERT) and ignored existing pre-trained medical models such as that of Zhang et al. 2020 (see ref below). It would have been interesting to test both models.\n- The analysis of the results could have been more thorough. Specifically, investigating the reason for the much lower performance on the CCKS dataset would have been interesting (see questions below).\n\nRef:\nNingyu Zhang, Qianghuai Jia, Kangping Yin, Liang Dong, Feng Gao, and Nengwei Hua. 2020. Conceptualized representation learning for Chinese biomedical text mining. arXiv preprint arXiv:2008.10813.", "questions_to_authors": "A) What are the specificities of the CCKS dataset? Do the authors have any hypothesis regarding the drop in performance? Could they identify any major differences between the documents?\n\nB) Details on the annotation procedure: How many annotators were there? What was their expertise? Was it a single or a double annotation (i.e. was each report annotated by one annotator, or by two annotators followed by adjudication over disagreements?)? Was inter-annotator agreement computed?\n\nC) What are the descriptive statistics of each data split in the HM corpus?\n\n", "paper_topic_and_main_contributions": "This paper studies de-identification of clinical notes, focusing on a cross-hospital setting. The main contribution is a manually PHI-annotated corpus of clinical notes from three Chinese hospitals. The authors also evaluate fine-tuned Chinese BERT-based models on this corpus, and compare same-hospital and cross-hospital settings. They show that performance can vary quite a lot across hospitals and that the de-identification task is not yet solved by current approaches.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "page 1, line 35: \"may be regarded as a easy task\" -> \"may be regarded as an easy task\""}, "extra_text": {}}, {"review_id": "36MgL6IisF", "reviewer": "Reviewer_iRFx", "summary": "", "questions": "- A) Are you able to include some analysis of what could possibly be some of the reasons that the performance gap is higher between hospitals \"HM\" and \"CCKS\"?  Some existing work that analyze  performance gap includes this paper for NER 1) *\"Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation (Elangovan et al., EACL 2021)\"* 2)*Pretrained Transformers Improve Out-of-Distribution Robustness (Hendrycks et al., ACL 2020)*", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive and insightful comments. We reply to your comments as follows:\n\n> A) While the paper highlights the generalization gap ( an important topic), there is limited analysis of what could be causing the generalization gap? For instance, what are the similarities and dissimilarities between the data from these hospitals that might be causing model performance to drop by upto 20 points?\n\n> A) Are you able to include some analysis of what could possibly be some of the reasons that the performance gap is higher between hospitals \"HM\" and \"CCKS\"? Some existing work that analyze performance gap includes this paper for NER 1) \"Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation (Elangovan et al., EACL 2021)\" 2)Pretrained Transformers Improve Out-of-Distribution Robustness (Hendrycks et al., ACL 2020)\n\n* Thanks for your suggestion. The below table reports the results:\n    * **Cosine Similarity**: Following Elangovan et al. (2021), we represent each train/test instance with a vector of HM-BERT and compute the cosine similarities. We then use the average similarity over all the test instances as an indicator to measure the extent of train/test overlap. \n    * **False Alarm Rate**: Following Hendrycks et al. (2020), we assign each test instance the maximum softmax anomaly score, to perform out-of-distribution (OOD) detection,  and report the False Alarm Rate at 95% Recall (FAR95).\n* The results show that the CCKS data present the most different distributions from HM train set; in other words, the overlap between HM and CCKS is the lowest. Hence, the CCKS has representations of out-of-distribution information that models fail to learn. This is the major reason for the large performance drop on CCKS. \n\n|   |Elangovan et al., EACL 2021 (cosine similarity)  |Hendrycks et al., ACL 2020 (False Alarm Rate at 95% Recall)  |\n|:--|:--:|:--:|\n| HM train vs. HM test | 92.05%|  94.64% |\n| HM train vs. SY |  84.50%| 92.05% |\n| HM train vs. CCKS |  82.51% |  73.03%|\n\n* Further underlying reasons for differences in the data distributions would be complicated. For example, different doctors may record information differently, resulting in differences in the format and content of medical records. Different medical institutions may use different electronic medical record systems. \n* We will add the above analysis in our revised manuscript. \n\n\n> Missing References: \n> Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation (Elangovan et al., EACL 2021) \n> Pretrained Transformers Improve Out-of-Distribution Robustness (Hendrycks et al., ACL 2020)\n* We will cite the papers together with the analysis on performance gap. \n\n\n> A) Please move some of the pretraining details (use of HM-BERT is based on Bert-base-chinese, and how HM-Word2Vec is created ) from the appendix to the main section. This is important for the readers to understand the context of how the custom pretrained models / word2vec embeddings were created and how they contribute to the generalization gap\n* We sincerely appreciate your suggestions. We will revise the paper accordingly. \n\n> B) Can you briefly explain \"BERT-wwm\" in line 157? and how it is different from BERT-base-chinese?\n* BERT-base-Chinese is the original Chinese BERT released by Google. \n* BERT-wwm (Cui et al., 2021) stands for \"BERT with whole word masking\", which is initialized from BERT-base-Chinese, and further pre-trained on a large Chinese corpus. During the pre-training, the whole word masking strategy is applied in the masked language modeling to enhance the model performance. BERT-wwm outperforms BERT-base-Chinese on many Chinese NLP tasks. \n* Our HM-BERT is also initialized from BERT-base-Chinese, but pre-trained on HM database. According to our experiments, HM-BERT outperforms BERT-base-Chinese and BERT-wwm on Chinese medical NLP tasks. \n\n1. Cui et al. 2021. Pre-training with whole word masking for Chinese BERT. *IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 29: 3504-3514.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well written and clear.\n- The main strength of this paper is highlighting the generalization gap (upto 20 points from in cross hospital setting ) in deep learning models.", "reasons_to_reject": "- A) While the paper highlights the generalization  gap ( an important topic), there is limited analysis of what could be causing the generalization gap? For instance, what are the similarities and dissimilarities between the data from these hospitals that might be causing model performance to drop by upto 20 points?", "questions_to_authors": "- A) Are you able to include some analysis of what could possibly be some of the reasons that the performance gap is higher between hospitals \"HM\" and \"CCKS\"?  Some existing work that analyze  performance gap includes this paper for NER 1) *\"Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation (Elangovan et al., EACL 2021)\"* 2)*Pretrained Transformers Improve Out-of-Distribution Robustness (Hendrycks et al., ACL 2020)*", "paper_topic_and_main_contributions": "This paper studies generalization gap cross hospitals for the task of PHI identification using electronic health records from 3 hospitals. This paper demonstrates that despite high test set performance for \"within-hospital\", there is upto 20 points drop in cross hospital performance even for a relatively simple task  of PHI identification of specific named entities (*location, name of the patient, hospital, patient id, date/timestamps, contact information of the patient/physician/hospital, profession/age of the patient)*.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "1) Memorization vs. Generalization : Quantifying Data Leakage in NLP Performance Evaluation (Elangovan et al., EACL 2021)\n2) Pretrained Transformers Improve Out-of-Distribution Robustness (Hendrycks et al., ACL 2020)", "Typos_Grammar_Style_and_Presentation_Improvements": "- A) Please move some of the pretraining details (use of HM-BERT is based on Bert-base-chinese, and how HM-Word2Vec is created ) from the appendix to the main section. This is important for the readers to understand the context of how the custom  pretrained models / word2vec embeddings were created and how they contribute to generalization gap\n\n- B) Can you  briefly explain \"BERT-wwm\" in line 157? and how it is different from BERT-base-chinese?"}}]

Author rebuttal:

Meta-review: **Summary:**
This paper investigates the de-identification of clinical notes, with a particular focus on a cross-hospital setting.
The authors have constructed a corpus (which they intend to make available) comprising Electronic Health Records (EHRs) from three different hospitals. They have developed or fine-tuned various neural models to tackle the de-identification task. The authors have evaluated Chinese BERT-based models that have been fine-tuned using this corpus. They have conducted comparisons between same-hospital and cross-hospital settings.
The paper illustrates that, despite achieving high test set performance for "within-hospital" scenarios, there is a significant drop in performance in cross-hospital settings, even for relatively straightforward tasks such as identifying Personal Health Information (PHI) related to specific named entities (e.g., location, patient name, hospital name, patient ID, date/timestamps, patient/physician/hospital contact information, patient age/profession).
In addition, the authors have explored various domain generalization techniques to address this issue when training with a single hospital dataset is the only option. They found that only Stochastic Weight Averaging and text smoothing yield slight improvements.

**Strengths:**
The main contribution of this work lies in the corpus constructed by the authors. While the corpus is not currently accessible, the authors have expressed their intention to release it upon the paper's acceptance.
The dataset holds significant value, given the scarcity of clinical corpora, especially those that encompass cross-hospital and non-English contexts. Furthermore, this work merits recognition for its role in highlighting the generalization gap observed in deep learning models.

**Weaknesses:**
Reviewers have identified the following weaknesses in the paper:
1. The techniques employed by the authors are standard. It is challenging to assess the significance of errors since errors in identifying age or the hospital are not equivalent to errors in identifying patient or doctor names.
2. The authors opted to create their medical BERT-based model (HM-BERT) while disregarding existing pre-trained medical models. It would have been valuable to explore alternative models for comparison.
3. The paper lacks an in-depth analysis of the aspects contributing to the generalization gap. For instance, there is insufficient exploration of the similarities and differences between the data from different hospitals that might explain the drop in model performance. It is possible that the document structure plays a role in this drop, and the proposed generalization techniques do not address structural differences.
4. The paper's conclusion, stating that cross-hospital results are worse than in-hospital results, is expected and does not provide a novel insight.

**Author-Reviewer discussion and acknowledgment:**
During the rebuttal phase, the authors have responded to the questions and concerns raised by the reviewers by providing clarifications and outlining their planned improvements to the paper.

**Conclusion:**
The paper is well-written and clear, offering valuable insights to the community, particularly in the context of medical NLP applications. However, reviewers recommend that the authors incorporate additional references and enhance the paper based on the points raised during the discussion phase. Additionally, the authors should address the identified typos.
Decision: Accept-Main