# Deep Learning Models for Automatic De-identification of Clinical Text

Ravichandra Sriram, Siva Sathya Sundaram, S. Lourdumarie Sophie

► To cite this version:

Ravichandra Sriram, Siva Sathya Sundaram, S. Lourdumarie Sophie. Deep Learning Models for Automatic De-identification of Clinical Text. 7th International Conference on Computer, Communication, and Signal Processing (ICCCSP), Jan 2023, Chennai, India. pp.116-127, 10.1007/978-3-031-39811-7_10. hal-05363237

HAL Id: hal-05363237

https://inria.hal.science/hal-05363237v1

Submitted on 13 Nov 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Deep Learning Models for Automatic
De-identification of Clinical Text

Ravichandra Sriram(✉), Siva Sathya Sundaram, and S. LourduMarie Sophie

Department of Computer Science, Pondicherry University, Pondicherry, India
{ravichandrasriram, ssivasathya, lourdumariesophie15}@pondiuni.ac.in

**Abstract.** In health care, clinical narratives play an essential part in the diagnosis and treatment of patients. In recent times, many organizations have stored these clinical narratives as electronic health records (EHR). The EHR is an organized collection of patient health information in digital format. These EHR's are an extensive collection of medical information, and they provide novel and rich data for clinical research. EHR also contains the patient's identities, such as name, address, mobile number, etc. These patient identities must be de-identified before use for clinical research to protect the patient's privacy. Clinical De-identification intends to detect and eliminate Protected Health Information (PHI) from medical data to facilitate data exchange and publication. This study aims to elucidate how the research community interprets and defines the terms like de-identification and anonymization and the steps involved in the process. A thorough review has been made to provide information on how deep learning approaches are used and how reliable they are.

**Keywords:** Clinical de-identification · Named entity recognition · Deep Learning · Natural language processing

## 1 Introduction

In recent times, the adoption of electronic health records (EHRs) in clinical research has skyrocketed, leading to the more extensive availability of textual clinical data. Nevertheless, these EHRs cannot be utilized directly for clinical studies under the Health Insurance Portability and Accountability Act (HIPAA) [29] to preserve patients' privacy since patient information is highly confidential and private to the individual. It cannot be exposed for research purposes. Hence, HIPAA mandates 18 types of PHI from EHR data (see Table 1). De-identification eliminates the PHI content from EHR data and replaces it with convincing yet realistic alternative data. Improved de-identification procedures can improve the availability of medical text for research purposes. The de-identification task can be performed in two approaches. The first approach is for human expertise to do the de-identification process manually. It is, however, time-consuming, ineffective, pricey, and inapplicable to enormous amounts of clinical data. The second approach is a computer-based automatic de-identification. It ensures privacy, high sensitivity and keeps the specificity of data.
<table><caption>**Table 1.** Protected Health Information identifiers defined by HIPAA</caption><tr><td>1.</td><td>First name, Last name of the individuals</td></tr><tr><td>2.</td><td>All the geographical units that are smaller than the state, comprising the address (county, city, district, postal codes, geocodes), excluding the first three digits of a postal code as per the Bureau of the Census's publicly accessible data:<br/>a. A geographic unit comprising more than 20,000 people is established by merging all postal codes with the same initial three digits; and<br/>b. For all such geographic areas with 20,000 or fewer inhabitants, the first three digits of postal codes are altered to 000</td></tr><tr><td>3.</td><td>All aspects of dates (excluding the year) that are linked to a person, like date of birth, date of admission, date of discharge, and date of death. Age group over 89 and all components of dates (inclusion of the year) indicative of such age, with the exception that such age group and components may be consolidated into a homogenous group of age 90 or older</td></tr><tr><td>4.</td><td>Facsimile Numbers</td></tr><tr><td>5.</td><td>Contact Information</td></tr><tr><td>6.</td><td>E-mail addresses</td></tr><tr><td>7.</td><td>Social Security Card Number</td></tr><tr><td>8.</td><td>Bank Account Number</td></tr><tr><td>9.</td><td>Health Record Information</td></tr><tr><td>10.</td><td>The beneficiary number on the Health care plan</td></tr><tr><td>11.</td><td>Certificate or License permit numbers</td></tr><tr><td>12.</td><td>Serial numbers and identifiers for vehicles, especially license plate numbers</td></tr><tr><td>13.</td><td>Device serial numbers and IDs</td></tr><tr><td>14.</td><td>IP addresses (Internet Protocol)</td></tr><tr><td>15.</td><td>Network URLs (Universal Resource Locators)</td></tr><tr><td>16.</td><td>Biometric Authentication data like Fingerprints and Voice prints</td></tr><tr><td>17.</td><td>Face photographs and any equivalent pictures</td></tr><tr><td>18.</td><td>Any other personal identification number, feature, or code</td></tr></table>

This paper concentrates on the four aspects of De-identification of patient data. First, Stages involved in Clinical De-identification process, explaining the existing models and layers in each stage. Second, The publicly accessible clinical datasets to perform the De-identification task. Third, Limitations in the existing models. Finally, observations made during the research are discussed.

Before the four aspects are discussed, it is essential to examine the HIPAA-defined protected health information to understand the challenges involved in the de-identification process. The Table 1 highlights the same.

## 2 Clinical De-identification Process

Clinical De-identification is the process of hiding or substituting the HIPAA-defined PHI with suitable placeholders or duplicate but relevant content to protect the privacy of patients' information. Though there are other algorithmic techniques, deep learning-based techniques were explored for clinical de-identification in this review. It is a task equivalent to Named Entity Recognition (NER). Ensemble methods were also employed to combine the output
<figure><img src="image_3.png" /><figcaption>Fig. 1. Overview of Clinical De-identification using Deep learning</figcaption></figure>

from rule-based and machine learning models to obtain better performance. Figure 1 depicts a high-level outline of the clinical de-identification process leveraging deep learning. The system is made up of four parts: Data pre-processing, Word embedding, Label prediction, and Label Sequence optimization. Table 2 and Table 3 present the different de-identification techniques and their performance proposed by various authors.

## 2.1 Pre-processing

The dataset is a collection of records with different formats from various sources. So, it requires a lot of cleaning and format conversion to match the input type of the model. The following section explains various pre-processing techniques used for deep learning-based de-identification methods.

**Tokenization** breaks a sentence, phrase, section, or whole text document into small fragments such as individual words or phrases. A token is a term referring to these smaller components; by examining the sequence of words, tokenization aids in deciphering the text's meaning. Depending on the language and type of model, numerous tokenization techniques can be utilized.

**Data harmonization** is a technique that overcomes structural inconsistencies in biomedical data collected from various sources. It converts the heterogeneous dataset into a standard stand-off structure with the defined criteria. Data harmonization allows these medical data to be networked and integrated in order to meet unmet requirements in a variety of medical conditions. Matching ontologies facilitates the identification of similarities between semantically related words or phrases based on their meaning.

**Sentence detection** is the problem of determining where sentences begin and end. It's a method of breaking down the given text into sentences. Because of the ambiguity of punctuation marks, it's a complex problem to solve. Detection alone with punctuation can cause issues. For instance, If “Dr. Johnson” is split by a period, then “Johnson” will become the first word in the phrase, and “Dr.” is omitted, which is a vital identifier for the DOCTOR category. The boundaries of sentences can be identified utilizing machine learning or rule-based techniques.

**Error correction** in the pre-processing stage is used to rectify the mismatch between the entity character's first and last offset values by verifying and modifying them depending on the entity's appropriate position in the text. Lack of space character preceding, succeeding, or within an entity in the text is also an issue. For instance, the term 'Discharge Report' was not correctly tokenized
<table><caption>**Table 2.** Overview of various Clinical De-identification methods using Deep learning</caption><thead><tr><th>Author</th><th>Pre-Processing</th><th>Word Embeddings</th><th>Mapping Layer</th><th>Label Prediction Layer</th><th>Datasets</th><th>F1-Score</th></tr></thead><tbody><tr><td>Ahmed et al. 2020 [1]</td><td>Tokenization</td><td>GloVe [25]</td><td>LSTM+GRU</td><td>CRF Layer</td><td>i2b2-2014<br>MIMIC-III</td><td>96.982<br>99.969</td></tr><tr><td rowspan="2">Johnson et al. 2020 [14]</td><td>Harmonization</td><td>BERT [9]</td><td></td><td>Linear</td><td>i2b2-2006</td><td>98.45</td></tr><tr><td>Tokenization</td><td>SciBERT [4]</td><td></td><td>Classification</td><td>i2b2-2014</td><td>98.62</td></tr><tr><td rowspan="2">Catelli et al. 2020 [5]</td><td>Data generation</td><td>BioBERT [18]</td><td></td><td>(Fully Connected)</td><td>PhysioNet</td><td>95.61</td></tr><tr><td>Token split</td><td></td><td></td><td>Layer</td><td>Dernoncourt-Lee</td><td>97.42</td></tr><tr><td rowspan="2">Hartman et al. 2020 [10]</td><td>entity spacer</td><td>GloVe [25]</td><td>Bi-LSTM</td><td>CRF Layer</td><td>i2b2-2014</td><td>98.32</td></tr><tr><td>tokenization<br>(NeuroNER tool)</td><td>Flair [2]</td><td></td><td></td><td></td><td></td></tr><tr><td rowspan="2">Yang et al. 2019 [34]</td><td>Tokenization</td><td>GloVe [25]</td><td>Bi-LSTM</td><td>CRF Layer</td><td>i2b2-2006</td><td>94.9</td></tr><tr><td>vTokennormilization</td><td></td><td></td><td></td><td>i2b2-2014</td><td>91.7</td></tr><tr><td rowspan="2">Liu et al. 2019 [21]</td><td>typographic error-<br>correction text</td><td>Word2vec [22]</td><td>Bi-LSTM</td><td>CRF Layer</td><td>i2b2-2014</td><td>96.46</td></tr><tr><td>normalization</td><td>fastText [16]</td><td></td><td></td><td>UF Health</td><td>95.84</td></tr><tr><td rowspan="2">Tang et al. 2019 [32]</td><td>i2b2 tools</td><td>Word2vec [22]</td><td>Bi-Capsule-<br>LSTM</td><td>CRF Layer</td><td>i2b2-2014</td><td>87.67</td></tr><tr><td>Tokenization</td><td>Word2vec [22]</td><td>Bi-LSTM</td><td>CRF Layer</td><td>i2b2-2014</td><td>98.7</td></tr><tr><td></td><td></td><td>ELMo [26] BERT [9]</td><td></td><td></td><td>N-GRID-2016</td><td>96.11</td></tr></tbody></table>

<table><caption>**Table 3.** Overview of various Clinical De-identification methods using Deep learning</caption><thead><tr><th>Author</th><th>Pre-Processing</th><th>Word Embeddings</th><th>Mapping Layer</th><th>Label Prediction Layer</th><th>Datasets</th><th>F1-Score</th></tr></thead><tbody><tr><td>Kim et al. 2018 [17]</td><td>Tokenization, Lemmatization, POS, NER</td><td>Word2Vec [22]</td><td>Ensemble (Rule-based, Machine Learning based models)</td><td></td><td>i2b2-2014</td><td>98.61</td></tr><tr><td>Zhao et al. 2018 [35]</td><td>skeleton-generation</td><td></td><td>GRU</td><td>Softmax</td><td>i2b2-2006</td><td>98.79</td></tr><tr><td rowspan="2">Liu et al. 2017 [32]</td><td>Tokenization</td><td>SENNA [7]</td><td>Ensemble (CRF, Featured Bi-LSTM, Bi-LSTM, Rule based models)</td><td>CRF Layer</td><td>i2b2-2014<br>N-GRID-2016</td><td>98.28<br>95.23</td></tr><tr><td></td><td></td><td>Bi-LSTM</td><td>Softmax</td><td>i2b2-2014</td><td>97.88</td></tr><tr><td>Dernoncourt et al. 2017 [8]</td><td>Tokenization</td><td>Word2Vec [22]</td><td></td><td></td><td>MIMIC-III</td><td>99.108</td></tr><tr><td>Jiang et al. 2017 [13]</td><td>(Stanford CoreNLPtool) Tokenization Sentence detection (openNLPsentece detector)</td><td>GloVe [25]<br>Word2Vec [22]</td><td>Bi-LSTM</td><td>Softmax</td><td>N-GRID 2016</td><td>92.18</td></tr><tr><td>Li et al. 2016 [20]</td><td>skeleton generation</td><td>random initialization</td><td>Bi-LSTM</td><td>Softmax</td><td>i2b2-2006<br>i2b2-2014</td><td>98.66<br>98.02</td></tr><tr><td>Shweta et al. 2016 [28]</td><td>Tokenization</td><td>Word2Vec [22]</td><td>Elman-RNN<br>Jordan-RNN</td><td>Softmax</td><td>i2b2-2014</td><td>93.84</td></tr></tbody></table>

because the entity 'Discharge' was attached to the word 'Report.' To resolve this issue, insert space wherever applicable, analyze the whole document, and recalculate the offset values of all entity's start and end characters inside the document. These tasks are performed using automation tools such as clinical language automation, openNLP sentence detector, NeuroNERtool, vToken normalization and i2b2 tools.
## 2.2 Word Embeddings

Word embedding or word vector is a real-valued vector used to express the words in a predefined vector space. It is a method of representing words in a two-dimensional space, finds semantic analogies between words, and calculates their similarity. Word embedding learnings are accomplished as part of a larger task-specific paradigm. Learning useful embeddings necessitates a significant quantity of text data. The model trained to understand the embedding is preserved and can be reused in subsequent models. The model is seeded with the pre-trained embedding, which is updated together during the training phase. The fundamental objective of this technique is to reduce the dimensionality of the input word vector while capturing the inter-word semantics. Currently, most of the modern NLP application models start with an embedding layer.

The word embedding layer, which accepts tokenized sentences as input, is the first layer of all Clinical de-identification deep learning models. The initial layer of practically all models is based on two types of embedding techniques: i) Direct Mapping, it directly converts a token to a vector. It is also referred to as token/word embedding; it is usually pre-trained on an enormous amount of unla-beled datasets using approaches like word2vec [23] or Glove [25]. Token embed-ding searches the entire vector space for semantically related words, resulting in a high-quality performance for a broad spectrum of applications. ii) Another technique is derived from the character-level encoder outcome.

However, data sparsity may continue to be a problem. Out-of-vocabulary words, typographical errors, and diverse verb forms or noun endings cannot be assessed using the word-based token embedding technique. A character-based token embedding technique addresses these issues; each character in a word/token creates its own vector representation, enabling the model to train from its sub-token structure like morphemes (suffixes or prefixes) and roots. As a result, the character-based technique captures various surface patterns, out-of-vocabulary terms, and other characteristics that are not included in the token embeddings. Both Liu et al. [32] and Jiang et al. [13] discovered that adding characteristics based on knowledge to word embeddings improved PHI identifi-cation.

Table 4 presents the various word embedding techniques used by different clinical de-identification frameworks based on deep learning. All models were initialized with pre-trained embeddings, and later the models were trained with the actual training data to fine-tune them. By incorporating knowledge features and word embedding techniques, the feature embedding layer based on knowl-edge increased the medical NER efficiency even more. Both Liu et al. [32] and Jiang et al. [13] discovered that adding characteristics based on knowledge to word embeddings improved PHI identification.
## 2.3 Label Prediction Layer

This layer receives a series of vectors as input from the token-embedding layer and returns the token with the label 't' likelihood. The resultant label is a PHI kind or a non-PHI kind. For instance, there would be 19 distinct labels to forecast all 18 HIPPA-defined PHI categories. During training, the label-prediction layer records the correlations between successive labels. It anticipates the label probability of input vectors by employing RNN models from Bi-LSTM or GRU.

Bi-LSTM is the frequently used deep learning technique for NLP, and it performs well in a variety of NER tasks. In the literature, several works [5, 8, 13, 20] used Bi-LSTM for clinical de-identification; it gives the probability of all potential PHI labels for every token in the series. The LSTM is a type of RNN model consisting of three gates (Input gate 'i', Forget gate 'f', Output gate 'o') and a memory cell 'c'. Word embeddings are fed as input to this layer. The LSTM layer creates a new hidden representation for the tokens by combining the obtained token embeddings with the context embeddings. At the end, these hidden representations are utilized to classify data directly.

[35] used a Bi-GRU, a modified form of the LSTM structure. In the GRU model, the input gate and the forget gate of the LSTM model are substituted by an update gate. This structure exposes the final output layer, which contains all of the previous and prospective information for every data point in the input sequence. [32] utilized ensemble learning technique to create a hybrid clinical de-identification system by combining the outcome of three independent subsystems (Conditional Random Field (CRF), Bi-LSTM, and Featured Bi-LSTM) to predict PHI label probabilities. Finally, the SVM classifier integrates the findings of rule-based subsystems with the ensemble learning-based classifier. [17] developed a hybrid de-identification model by integrating several models trained from shallow learning, deep learning, and rule-based techniques. The hybrid model utilized three kinds of ensemble methods (Stacking, Voting, and Decision Template Method (DTM)) for this integration.

## 2.4 Label Sequence Optimization

This layer accepts the label prediction layer's series of probability vectors as input. Each input token produces a series of PHI labels based on its neighbor's information as output. CRF or Softmax are the two approaches that are used to get PHI labels of the input sequence. Invariably, the CRF layer utilizes the Bi-LSTM model for named entity recognition (NER) tasks. The CRF layer adds some constraints to the final predicted labels to assure they are valid. The CRF layer may learn these restrictions from the training dataset automatically during the training stage. On the other hand, the softmax layer selects the label with the maximum probability using the most basic approach.
<table><caption>**Table 4.** Overview of various Word Embedding techniques used for De-identification</caption><thead><tr><td>Model</td><td>Model Architecture</td><td>Method</td><td>Dataset</td></tr></thead><tbody><tr><td>BERT [9]</td><td>Multi-layer Bidirectional Transformer Encoder</td><td>Language model pre-trained with two unsupervised tasks [Masked LM and Next Sentence Prediction (NSP)].</td><td>BookCorpus [36], English Wikipedia</td></tr><tr><td>SciBERT [4]</td><td>Multi-layer Bidirectional Transformer Encoder</td><td>BERT-based language model pre-trained on huge corpora of scientific writings</td><td>1.14 million publications from Semantic Scholar were randomly chosen [3]</td></tr><tr><td>BioBERT [18]</td><td>Multi-layer Bidirectional Transformer Encoder</td><td>BERT based language model pre-trained on biomedical corpus</td><td>Abstracts from PubMed database and Full-text articles from PMC database</td></tr><tr><td>GloVe [25]</td><td>log-bilinear regression model</td><td>Unsupervised learning technique: It is based on a corpus' global word-to-word co-occurrence matrix.</td><td>Wikipedia dump, Gigaword5, Common Crawl5</td></tr><tr><td>Flair [2]</td><td>Bi-LSTM</td><td>Contextual String Embedding: The language model was trained on the character level to create novel word embeddings. It depends on characteristics at the word and character level</td><td>word corpus [6]</td></tr><tr><td>ELMo [26]</td><td>Bi-LSTM</td><td>Deep Contextualized Word Representations: A bidirectional language model (biLM) generates high-quality profound context-dependent representations.</td><td>word corpus [6]</td></tr><tr><td>Word2vec [22]</td><td>Feed forward Neural Networks</td><td>CBOW: It uses context to anticipate the current word. Skip-gram: It forecasts words preceding and succeeding the present word within a specified range</td><td>Google News corpus</td></tr><tr><td>SENNA [7]</td><td>Multi-layer neural network</td><td>It is generated using discriminating and non-probabilistic models by capturing the plurality relationship on shallow syntactic features</td><td>English Wikipedia Reuters RCV1 [19]</td></tr><tr><td>MADE [12]</td><td>Bi-LSTM</td><td>Skip-gram-based language model pre-trained with biomedical datasets</td><td>PubMed open access articles, English Wikipedia, Unlabeled EHRs</td></tr><tr><td>fastText [16]</td><td>LSTM</td><td>Enriching Word Vectors with Sub-word Information: Each word is defined as a bag of n-gram characters with subword information taken into consideration</td><td>Data dumps in Arabic, Czech, English, French, German, Italian, Russian, Romanian and Spanish from Wikipedia</td></tr></tbody></table>
# 3 Datasets

Clinical notes are essential for conducting any research on medical documents. Only a few datasets are publicly accessible due to patient privacy and confidentiality concerns. This section discusses five datasets, a collection of free-text observations recorded during regular medical care. Once the associated data access agreement is signed, these datasets are made accessible to the public. Table 5 depicts the abstract view of these datasets.

<table><caption>**Table 5.** Abstract view of Datasets</caption><thead><tr><th>Statistics</th><th>2006- i2b2</th><th>2014- i2b2</th><th>2016-N -GRID</th><th>PhysioNet (MIMIC-II)</th><th>Dernoncourt -Lee (MIMIC-III)</th></tr></thead><tbody><tr><td>Vocabulary Size</td><td>20254</td><td>46803</td><td>51899</td><td>NA</td><td>69525</td></tr><tr><td>Number of notes</td><td>889</td><td>1304</td><td>1000</td><td>2434</td><td>1635</td></tr><tr><td>Number of tokens</td><td>560852</td><td>984723</td><td>2388419</td><td>334000</td><td>2945228</td></tr><tr><td>Number of PHI instances</td><td>19498</td><td>28867</td><td>34364</td><td>1779</td><td>60725</td></tr><tr><td>Number of PHI tokens</td><td>29917</td><td>41355</td><td>NA</td><td>NA</td><td>78633</td></tr></tbody></table>

- **i2b2 Corpus 2006 [33]**: The i2b2 National Center for Biomedical Computing provides a de-identification challenge dataset for the NLP Shared Tasks Challenges as part of the i2b2 initiative. This dataset comprises 889 medical summaries obtained from the Partners Healthcare system. HIPPA's 18 types of PHI identifiers have been divided into seven major categories and many sub-categories to facilitate more precise identities.

- **i2b2 Corpus 2014 [31]**: This dataset consists of clinical notes from the UF Health Integrated Data Repository (IDR) and the i2b2/UTHealth challenge, 2014. It has 1304 longitudinal clinical notes from 296 individuals collected from Partners Healthcare's Research Patient Data Repository [22]. PHI types of this corpus are the same as the i2b2 challenge, 2006.

- **CEGS N-GRID Cropus 2016 [30]**: The CEGS N-GRID NLP challenge in the year 2016 makes use of psychiatric records. This corpus contains randomly selected 1000 psychiatric intake records from Partners HealthCare. Standard questionnaires, doctor's observations, and patients' narrative summary responses make up these records. The PHI types of this corpus are the same as the i2b2 challenge.

- **MIMIC-II [27]**: It is an annotated database comprising lab reports, physiological indicators, medical history, and other information for approximately 17,000 ICU patients at Boston's Beth Israel Deaconess Medical Center. Neamatullah et al. [24] produced standard gold corpora with corresponding surrogate PHI data accessible through PhysioNet depending on a limited data use contract. The corpora were created by arbitrarily extracting a part of nursing progress observations from the MIMIC II repository. The corpora incorporate 2,434 nursing notes from 163 individuals' health records.
- **MIMIC-III** [15]: It is an upgrade of the MIMIC-II database. Compared with MIMIC-II, MIMIC-III is a massive, single-center database collecting patient information, especially those treated in CCU (Critical Care Units) at Boston's Beth Israel Deaconess Medical Center, USA. The dataset contains 53,423 ICU reports spread throughout 49,785 hospitalizations for 38,597 unique patients. Dernoncourt et. [8] created the gold standard MIMIC de-identification dataset by randomly selecting 1635 summaries of different patients containing 60,700 PHI instances. From the MIMIC-III database.

# 4 Limitations

The clinical textual documents differ from regular text documents by having their own set of characteristics. Clinical writing has several concise phrases and abbreviations when compared to conventional articles or records. Clinical text is a type of semi-structured text that attempts to address the de-identification issue by focusing on the data structure of the text. From this study, we identified the sources of errors by doing error analysis. Some of the PHI instances are abbreviations, and sometimes these are short and ambiguous; in those circumstances, detecting them can be challenging. There may be uncertainty detecting PHI labels due to standard terms or numbers like test results or dates. Data sparsity is another critical issue due to insufficient PHI instances in the training dataset.

# 5 Discussion

This work explores several Clinical De-identification techniques based on deep learning. Numerous statistical learning models like HMMs, CRFs, SVMs have already been used in the clinical de-identification process. These traditional approaches rely on various characteristics, including document-level, word-level, and list-level (i.e., dictionary-level) properties. Since 2016, Deep Learning models have started to play a significant role in Clinical De-identification processes.

The CRF is the frequently utilized technique in traditional NER applications due to its practical and conceptual efficacy. In the i2b2 de-identification challenges in the year 2006 and 2014, CRF-based systems took the lead. In the 2014 i2b2 de-identification challenge, the CRF model obtained the maximum F-measure of 92.3 by identifying the lexicon, word-token, sentence-level, contextual and orthographic characteristics. The Bi-LSTM model achieved the highest performance with an F-measure of 79.85 in the CEGS N-GRID de-identification challenge, 2016 [30]. Compared to past high results on de-identification tasks, this score is considerably lower.

First, Dernoncourt et al. [8] used a Bi-LSTM+CRF model with the 2014 i2b2 dataset and achieved an F-measure of 97.85; it was higher than the best CRF-based method in [11], which was 92.3. Later, [5,32] and others gained an F-measure of 98.32 by the Bi-LSTM-CRF model. Using the BERT [9] transformer model for word embedding and the final layer's output to a single linear fully
connected layer, [14] achieved an F-measure of 98.45 using the 2014 i2b2 dataset. We observed that a few of them confirmed the models' performance in a cross-institutional environment by interchanging the training set with the validation set of various datasets. Furthermore, this work compares the performance of several pre-trained word embedding techniques trained using medical resources, general English text, and biomedical publications.

## 6 Conclusion

The health care sector involves a massive amount of patient's personal information, due to which there is a rising interest in privacy-enhancing procedures. Various deep learning-based models and research concerns surrounding the protection of patients' privacy when using healthcare data have been explored in this work. When used in a cross-institutional environment, de-identification models must be customized by employing local clinical literature and other materials. This article can be used as a guide to shape future Clinical De-identification research.

## References

1. Ahmed, T., Aziz, M.M.A., Mohammed, N.: De-identification of electronic health record using neural network. Sci. Rep. 10, 18600 (2020). https://doi.org/10.1038/s41598-020-75544-1

2. Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., Vollgraf, R.: FLAIR: an easy-to-use framework for state-of-the-art NLP. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp. 54–59. Association for Computational Linguistics, Minneapolis, Minnesota (2019). https://doi.org/10.18653/v1/N19-4010

3. Ammar, W., et al.: Construction of the literature graph in semantic scholar. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers), pp. 84–91. Association for Computational Linguistics, New Orleans - Louisiana (2018). https://doi.org/10.18653/v1/N18-3011. https://aclanthology.org/N18-3011

4. Beltagy, I., Lo, K., Cohan, A.: SciBERT: a pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615–3620. Association for Computational Linguistics, Hong Kong, China (2019). https://doi.org/10.18653/v1/D19-1371

5. Catelli, R., Casola, V., De Pietro, G., Fujita, H., Esposito, M.: Combining contextualized word representation and sub-document level analysis through BiLSTM+CRF architecture for clinical de-identification. Knowl.-Based Syst. 213, 106649 (2021). https://doi.org/10.1016/j.knosys.2020.106649

6. Chelba, C., et al.: One billion word benchmark for measuring progress in statistical language modeling. ArXiv abs/1312.3005 (2014)

7. Collobert, R., Weston, J., Bottou, L., Karlen, M., Kavukcuoglu, K., Kuksa, P.: Natural language processing (almost) from scratch. J. Mach. Learn. Res. 999888, 2493–2537 (2011). http://dl.acm.org/citation.cfm?id=2078183.2078186
8. Dernoncourt, F., Lee, J.Y., Uzuner, Ö., Szolovits, P.: De-identification of patient notes with recurrent neural networks. J. Am. Med. Inform. Assoc. **24**, 596–606 (2017)

9. Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: NAACL (2019)

10. Hartman, T., et al.: Customization scenarios for de-identification of clinical notes. BMC Med. Inf. Decis. Mak. **20**, 14 (2020). https://doi.org/10.1186/s12911-020-1026-2

11. He, B., Guan, Y., Cheng, J., Cen, K., Hua, W.: CRFs based de-identification of medical records. J. Biomed. Inform. **58**, S39–S46 (2015). https://doi.org/10.1016/j.jbi.2015.08.012

12. Jagannatha, A., Liu, F., Liu, W., Yu, H.: Overview of the first natural language processing challenge for extracting medication, indication, and adverse drug events from electronic health record notes (made 1.0). Drug Safety **42**, 99–111 (2019). https://doi.org/10.1007/s40264-018-0762-z

13. Jiang, Z., Zhao, C., He, B., Guan, Y., Jiang, J.: De-identification of medical records using conditional random fields and long short-term memory networks. J. Biomed. Inf. **75S**, S43–S53 (2017). https://doi.org/10.1016/j.jbi.2017.10.003

14. Johnson, A.E.W., Bulgarelli, L., Pollard, T.J.: Deidentification of free-text medical records using pre-trained bidirectional transformers. In: Proceedings of the ACM Conference on Health, Inference, and Learning, pp. 214–221. CHIL 2020, Association for Computing Machinery, New York, NY, USA (2020). https://doi.org/10.1145/3368555.3384455

15. Johnson, A.E.W., et al.: Mimic-iii, a freely accessible critical care database. Sci. Data **3**, 160035 (2016). https://doi.org/10.1038/sdata.2016.35

16. Joulin, A., Grave, E., Bojanowski, P., Douze, M., Jégou, H., Mikolov, T.: Fast-text.zip: compressing text classification models (2016)

17. Kim, Y., Heider, P., Meystre, S.: Ensemble-based methods to improve de-identification of electronic health record narratives. AMIA Annual Symp. Proceed. **2018**, 663–672 (2018)

18. Lee, J., et al.: BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics **36**(4), 1234–1240 (2019). https://doi.org/10.1093/bioinformatics/btz682

19. Lewis, D.D., Yang, Y., Rose, T.G., Li, F.: RCV1: a new benchmark collection for text categorization research. J. Mach. Learn. Res. **5**(Apr), 361–397 (2004). http://www.jmlr.org/papers/volume5/lewis04a/lewis04a.pdf

20. Li, K., Chai, Y., Zhao, H., Nan, X., Zhao, Y.: Learning to recognize protected health information in electronic health records with recurrent neural network. In: NLPCC/ICCPOL (2016)

21. Liu, C., Li, J., Yuhan, L., Du, J., Tang, B., Xu, R.: Named entity recognition in clinical text based on capsule-LSTM for privacy protection, pp. 166–178 (2019). https://doi.org/10.1007/978-3-030-23367-9_12

22. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: Proceedings of Workshop at ICLR 2013 (2013)

23. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed representations of words and phrases and their compositionality. In: Advances in Neural Information Processing Systems 26 (2013)

24. Douglass, M., et al.: Automated de-identification of free-text medical records. Med. Inform. Decis. Mak. **8**, 32 (2008)

25. Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: EMNLP (2014)
26. Peters, M.E., et al.: Deep contextualized word representations. In: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 2227–2237. Association for Computational Linguistics, New Orleans, Louisiana (2018). https://doi.org/10.18653/v1/N18-1202

27. Saeed, M., et al.: Multiparameter intelligent monitoring in intensive care ii (mimic-ii): a public-access intensive care unit database. Critical Care Med. 39, 952–60 (2011). https://doi.org/10.1097/CCM.0b013e31820a92c6

28. Shweta, Kumar, A., Ekbal, A., Saha, S., Bhattacharyya, P.: A recurrent neural network architecture for de-identifying clinical records. In: Proceedings of the 13th International Conference on Natural Language Processing, pp. 188–197. NLP Association of India, Varanasi, India (2016). https://aclanthology.org/W16-6325

29. States., U.: Health insurance portability and accountability act of 1996. Public Law 104–191. US Statut Large, pp. 1936–2103 (1996)

30. Stubbs, A., Filannino, M., Uzuner, O.: De-identification of psychiatric intake records: overview of 2016 CEGS N-GRID shared tasks track 1. J. Biomed. Infor. 75, S4-S18 (2017). https://doi.org/10.1016/j.jbi.2017.06.011

31. Stubbs, A., Uzuner, Ö.: Annotating longitudinal clinical narratives for de-identification: the 2014 i2b2/uthealth corpus. J. Biomed. Inform. 58(Suppl), S20–S29 (2015)

32. Tang, B., Jiang, D., Chen, Q., Wang, X., Yan, J., Shen, Y.: De-identification of clinical text via bi-LSTM-CRF with neural language models. AMIA Annual Symp. Proceed. 2019, 857–863 (2020)

33. Uzuner, Ö., Luo, Y., Szolovits, P.: Viewpoint paper: evaluating the state-of-the-art in automatic de-identification. J. Am. Med. Inf. Assoc. JAMIA 14(5), 550–63 (2007)

34. Yang, X., Lyu, T., Li, Q., Lee, C.Y., Bian, J., Hogan, W., Wu, Y.: A study of deep learning methods for de-identification of clinical notes in cross-institute settings. BMC Med. Inf. Decis. Making 19, 232 (2019). https://doi.org/10.1186/s12911-019-0935-4

35. Zhao, Y.S., Zhang, K.L., Ma, H.C., Li, K.: Leveraging text skeleton for de-identification of electronic medical records. BMC Med. Inf. Decis. Making 18, 18 (2018). https://doi.org/10.1186/s12911-018-0598-6

36. Zhu, Y., et al.: Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 19–27 (2015)