# Efficient Approaches for the Discovery of Sensitive Information by Using Natural Language Processing Techniques

Kushal Shree Dhani, Benedikt Zundel, Doina Logofătu

► To cite this version:

Kushal Shree Dhani, Benedikt Zundel, Doina Logofătu. Efficient Approaches for the Discovery of Sensitive Information by Using Natural Language Processing Techniques. 19th IFIP International Conference on Artificial Intelligence Applications and Innovations (AIAI), Jun 2023, León, Spain. pp.399-410, 10.1007/978-3-031-34107-6_32. hal-05498069

HAL Id: hal-05498069

https://inria.hal.science/hal-05498069v1

Submitted on 6 Feb 2026

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
<figure><img src="image_2.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
# Efficient Approaches for the Discovery of Sensitive Information by Using Natural Language Processing Techniques

Kushal Shree Dhani, Benedikt Zundel, and Doina Logofătu(✉)

Frankfurt University of Applied Sciences, Frankfurt am Main, Germany
benedikt.zundel@stud.fra-uas.de, logofatu@fb2.fra-uas.de

**Abstract.** The quantity of data and documents created on a huge scale has recently increased. This signifies a rise in unstructured textual information. Extracting sensitive data is usually done manually, following certain rules, which means additional time costs and elevated chances of errors and non-performance. There is a growing need to automatically solve these types of problems, leading to the need of utilizing methodologies that are more intelligent than those previously used. The likelihood of automating this kind of assignment might significantly facilitate compliance with safety policies and imposed regulations. This work aims to highlight the status of Named Entity Recognition (NER) by evaluating models and presenting their overall presentation. It also describes the conflicts and factors that affect the perception of a given entity.

**Keywords:** Named Entity Recognition · spaCy NER Model · Flair NER Model · Comparison of NER models

## 1 Problem Description

The Named Entity Recognition (NER) process is used to recognize specific groups of words that have common semantic characteristics [8]. In reality, NER interacts with a large number of names and signifiers. The task of full named object recognition can often be split into two parts: identifying names and names being classified according to the sort of entity they relate to, such as people, organizations, or locations. Ambiguity and abbreviations as well as identifying phrases that can have more than one meaning are major challenges in recognizing named entities in language. NER is challenging because the target phrases are mainly nouns or unregistered phrases. To train the NER engine, suitable and annotated data is required. Producing training sets is the high-priced part of developing a NER engine. Annotated refers back to the reality that named entities need to be recognized within the training set, and those annotations need to be dependable to supply a well-acting engine. Therefore, we're dealing with information challenges as opposed to algorithm challenges. Algorithms will evolve and enhance over time. However, they may usually depend upon exceptional training material which is difficult and high priced to supply.
## 2 Introduction

Private Identifiable Information (PII) being derived from unstructured text is an important task in many fields. For instance, the collected information can be used for document indexing, classification, and other applications. Identifying PII items is time-consuming work that can be automated by using a NER model. The goal is to develop a NER model that reduces the cost of this task by preprocessing documents before extracting information manually. Training a NER model to extract PII requires a lot of privacy-rich text, which creates privacy issues during data annotation and model training. Named entity recognition comprises of natural language processing (NLP), which is the process by which researchers use computer systems to analyze human language and extract important metadata from the text.

To conduct reproducible research, the feasibility is presented of the suggested method on a dataset generated with a library for artificial data generation in python. Following contributions are made:

1. The comprised labeled dataset is annotated in 4 classes of personal information.

2. Creating a training set and later training a NER Model with the libraries spaCy and Flair.

3. Applying the model to new sentences and therefore testing the model on the test data.

## 3 PII Dataset

Personal Identification Information datasets are those datasets that consist of personal information of people which can be categorized by a name, address, identification number, etc. Private Identifiable Information (PII) is retrieved from unstructured text which is a critical task in many areas such as medicine [6], legal documents [11] data mining of user-generated content [7], and publication processes [2]. Our objective in this study is to put confidentiality at the forefront of building a PII extractor, from dataset development to model training [4]. There is a need to build a tool to reduce the expense of manual PII retrieval.

In this study, the focus lies more on medical data, e.g. health insurance numbers, email addresses, names, phone numbers. To create an artificial PII dataset for training a NER model, a data frame containing information about a patient and their medical record is generated and saved for later reference. It is guaranteed that each field which is created in the data frame has some unique element. The aforementioned fields are then used to generate textual data on which the model can be trained and tested. To generate textual data for the model, the unique fields from the data frame are extracted and substituted into a randomly chosen text from a collection of pre-written phrases. Those are then written to a file, which results in paragraphs of medical information for artificial patients.
# 4 NER Algorithm

The algorithm for preparing the dataset and developing the model is provided in this section. Names, email addresses, phone numbers, and health insurance numbers were all included in the new dictionaries [13]. These dictionaries are used to annotate the extracted sentences, which are then separated into training and testing datasets. The flow diagram (Fig. 1) shows the process of using the PII dataset to create the NER model and extracting the labels from the trained model.

<figure><img src="image_3.png" /><figcaption>Fig. 1. Flow Diagram of NER algorithm</figcaption></figure>

The method of creating the PII datasets is illustrated in preceding section. After generating the required PII dataset, the dataset is pre-processed using NLP tools. After applying pre-processing to the dataset, annotation is being created in which the annotated text with each entity label present in the text is represented. After generating the annotated text the goal is to train the model using a library that will create a model for our custom entities [13]. In this study, two libraries were used to create a NER model, namely Flair and spaCy. After building a model, the labels are predicted for the new sequence of texts to validate the result. In case the model fails to predict the required labeled entities, the model needs to be updated with correct labeled data and the model is rebuilt. To add new custom entities to the model, the pre-trained model needs to be updated with a new dataset that includes a new entity and the process repeats from creating annotations and training the model. This is the complete process to create the NER model.
## 4.1 Algorithm: NER [9]

**Input:** Preprocessed text documents.

**Output:** Labelled entities, **Collections:** D- Corpus, S- collection of phrases, A- collection of annotated phrases, E- collection of entities, W- collection of words, CD- created dataset.

## 4.2 Generating Dataset [9]

The resulting dataset is saved in JSON format. The saved data is put into the main python script to train the model. The suggested method has been tested with both a blank model and an existing model [5]. Using the Transfer learning principle, the current model is retrained. CNN plays a significant part in training the model. The model trained is subsequently stored with a unique name on a disk.

# 5 Building the NER Framework

The system requires two inputs from the user. Location of configuration files and training data. In JSON format, the former includes model size, model language, and model training hyperparameters. “Model size” and “Model language” indicate pre-trained spaCy models where users train their data to generate new fitted models, and “Hyperparameters” include those values. It will be used during the development of hyperparameters. This framework is generalized, because choices can be made for both the spaCy model which is trained and the hyperparameter settings which is trained, and each dataset which is generated is unique based on the configuration data provided to the framework [5]. It will be a compliant model. The main features of the framework are:

1. Loading the configuration file

2. Getting the spaCy model

3. Conversion of JSON data to spaCy training data format

4. Creating training data

5. Model Training Function

6. Model saving.

# 6 Implementation of SpaCy

## 6.1 Introduction to SpaCy Training Sets

This section details the training sets and how to use spaCy’s entity ruler to automatically generate training datasets that require manual validation. The data used to train machine learning models were provided in three formats: data for training, assessment data, and test data. All this data is in the same format. This is a list data structure where each index contains text (sentence, paragraph, or full text). The length of this paragraph is determined by your goals
for using machine learning named entity recognition. The only other components needed to train the data are the list of entities in this text and their start, end, and labels. These annotations enable the convolutional neural network (spaCy's machine learning architecture) to learn from the data and accurately identify the entity throughout the training phase. SpaCy requires training data to be in a very special format.

Since JSON training data is in list dictionary format, we will convert it using a function that converts JSON data to spaCy training data format. SpaCy, on the other hand, provides data for training in the style of string-formatted text. This annotated data consists of a start index, end index, and entity label. Therefore, it is necessary to convert the annotated JSON text file to the required spaCy format.

## 6.2 Using SpaCy to Create a NER Training Set

To produce a spaCy machine learning training dataset through entity ruler, a rule-based technique is first used to automatically generate a baseline training set. This model will be used only temporarily, so create an English model. Then all the patterns of the corresponding labels are added in this entity ruler by passing patterns to the add patterns function. Then it is going to load these patterns into the ruler which is also loaded into our NLP. The entity ruler is built and linked to the spaCy model's network, but by convention spaCy introduces a different pipe all the way to the network's ending. Using spaCy analyze pipe, the pipeline can be viewed.

In this study, the implementation is done with 4 patterns, and therefore when it finds these patterns and the tokens it is automatically going to assign the corresponding label to it. A doc object is being created with the NLP, with the entity ruler loaded in it and it also goes over the whole text and then is iterated over all of those entities as they were just a regular output from the NER. All the entities in the doc object has been taken and printed in all the text with the label in it. In this way, it is checked whether the custom entity that has been loaded in the entity ruler has been activated or not.

## 6.3 Adding a Custom NER Pipeline and Label in SpaCy

To add custom entities to the model, a custom pipeline is created into it, and thus by using a NER, a custom label is identified.

In this study, custom labels are added such as personal names, phone numbers, health insurance numbers, and email addresses. The patterns for the custom labels have been defined with the help of Regex features. The language model is created and saved on a disk. The standard things that would appear in a blank spaCy model are vocab, meta and tokenizer so that it can do some basic parsing. To add any NER pipelines, the pipes need to be added in the spaCy model. With the help of the NLP object, the NER pipeline is created. After the creation of the NER pipeline, it usually goes to factories and its NER and tries to load the corresponding classes and functions in the pipeline. When any pipe
is added with any specific name, it goes to differentiate it from other elements
in the pipeline. After adding a pipe, a spaCy model will be replaced with a new
spaCy model which will have a new entity added to it. The new pipeline can
be found in the meta-JSON file under the new spaCy model. Each of the pipes
as NER will have a customs value which will be the name of the pipe and its
values will be the list of labels corresponding to that pipe. Then custom labels
are added to the path. This is how the spaCy model and a blank NER pipeline
are created, then a custom label is added to it and that NLP model is saved.

## 6.4 Training Custom Entities into SpaCy Models

To get a model into the custom NER, the model must be trained. The dictionary with a key, that corresponds to a value is a list that contains the start and end of an entity and the corresponding label. A function is used to load the JSON data which will allow to import the training data. Once the training data gets imported, the function for training our spaCy model is created.

This function takes two arguments: one being the training data, and the
other being the iteration. If there are multiple pipes in a NER Model, then
the iteration is achieved as per the requirement to train the model. This is the
general process to create a model for custom entities.

## 6.5 Convert the Training Data to SpaCy Binary Files

By generating the spaCy binary object from the training data, a machine learn-
ing model is trained. The comprehensive goal is to convert spaCy2 train data to
binary spaCy3 training data.

To achieve this, three parameters are sent to the function:

1. Language: An empty model language.

2. Train Data: A list of lists forming of training data.

3. Output Path: The location of the spaCy binary files in the output directory.

DocBin object was built and stored to make the functionality work. After
that, they are transformed into binary objects, and files are saved to a disk.

## 6.6 Testing Our NER Model

After creating a model, the trained model is used to test the model which demon-
strates how well it has performed with some new testing data. The custom-
trained spaCy model can extract all the entities as per the requirement. Figure 2
shows the results of a NER model on a sequence of text. The model can extract
the name of a person, health insurance number, email address, and phone
number.
<figure><img src="image_4.png" /><figcaption>Fig. 2. Results Of a spaCy NER model</figcaption></figure>

## 7 Confusion Matrix for NER Models

A confusion matrix outlines a classifier's classification work concerning test inputs. This is a two-dimensional matrix indexed in the first dimension by the actual class and in the second dimension by the class assigned by the classifier [12].

The following describes a confusion matrix with given descriptions for its values.

1. Amount of correctly predicted negative outcomes

2. Amount of incorrectly predicted positive outcomes

3. Amount of incorrectly predicted negative outcomes

4. Amount of correctly predicted positive outcomes.

### 7.1 Create Target Vector

The requirement is to create two vectors: The target value, and the predicted value. In tuple lines, the first element is text. Therefore, this text can be used to get a named entity using a trained model. The second element is a list of tuples that represent named entity tags. This allows generating an expected value vector. A target class vector is created using the labeled data.

### 7.2 Generating the Confusion Matrix

The Confusion matrix is generated using the sklearn library. The parameters to generate the confusion matrix are as follows:

1. **ytrue**: These are the target values. For the NER model, this parameter is being calculated in the method create total target vector.

2. **ypred**: These are the estimated targets values. For the NER model, this parameter is being calculated in the method create total prediction vector.

The *ytrue* and *ypred* parameters are passed in the confusion matrix function.
## 7.3 Visualizing the Matrix

The confusion matrix is visualized using matplotlib. Figure 3 shows the confusion matrix. There are a total of 3699 instances of email addresses, 3898 instances of health insurance numbers, 7034 instances of a person's name, 3011 instances of the phone number, and 1239508 instances of unlabeled tokens. The degree to which the labels were predicted correctly and incorrectly can also be read from the figure. The training set is built for high precision. The desire is to have the model be able to identify the true positives, at the expense of not having incorrect entities labeled as false positives. The confusion matrix explains the correctly predicted email addresses, health insurance numbers, person names, and phone numbers.

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** Confusion Matrix plot with X-axis representing predicted label and Y-axis representing true label. Diagonal elements are correctly predicted and off diagonal elements are incorrectly predicted. Sum of the instances of each row is the total instance of a respective entity.</figcaption></figure>

# 8 Implementation of Flair

## 8.1 Introduction to Flair

Flair is an NLP platform designed to simplify training and deployment of language models, text categorization, and sequence labeling. The main goal is to
provide a simple and consistent interface for completely embedding many forms
of words and text in theory [1]. This is unique to integrations, successfully
masking all the technical complexities, allowing researchers to combine differ-
ent integrations with minimal effort. In addition to traditional model training
and hyperparameter selection methods, the platform includes a data acquisition
module for loading and converting published NLP datasets into data structures
and setting up experiments easily.

## 8.2 Using Flair to Train a Custom NER Model

The model is trained with flair to identify customized entities from the input text. Data preparation of the data provided for model training is not as easy as spaCy. By creating a training corpus first, the model can be created to find these entities in the given piece of text. In this thesis, the training for custom entities and the dataset is prepared by ourselves. The training dataset should follow the following structure.

1. A blank line divides the sentences in the corpus.

2. There are two-column: The first column contains the text and the second must have the BIO-annotated NER tag.

**Data Creation of the Flair Model.** To create corpus objects, there is a class called column corpus. The data in the three text files that belong to the train, test, and validation corpus should be in the structure discussed in the preceding section.

1. In the first step the data frame is being preprocessed where special characters are escaped, a space before the punctuation marks is being added, and white space is removed.

2. To create an annotation, the sequence matcher method is imported from difflib and is used to create the start and end index of each pattern present in the text.

3. According to the BIO Scheme, all entities in the phrase are tagged after extracting all the entities from the text. The final data would look like as shown in the Fig. 4.

After getting the data in the desired format, the data is ready to load and start the training. The tag needs to be defined and the model should be capable to anticipate and construct the tag dictionary. To create embeddings, a bunch of pre-trained models needs to be chosen. The following step was to set up the Sequence Tagger and the output of the tagger. The model is then trained with the help of a model trainer. After training, information like training logs, loss statistics, and estimates with a confidence score on the testing sample, as well as the model itself are stored in the working directory.
<figure><img src="image_6.png" /><figcaption>Fig. 4. Results obtained from creating the data for training the flair model</figcaption></figure>

## 9 Comparison Between Flair and SpaCy NER Models

Both NER models performed satisfactorily. As per the metrics measured, the Flair model has performed better than spaCy, however, both models have slight advantages and downsides. SpaCy is a free and open-source package for sophisticated NLP in Python. It is intended for production usage and aids with the development of applications that process and “understand” massive quantities of text. Whereas Flair’s framework is based directly on PyTorch, a well-known deep-learning library. Flair has given two pre-trained NER models, each of which is a bi-LSTM on top of a word embedding layer. Each classifier is trained using a distinct NER dataset. SpaCy, on the other hand, maybe utilized to create information extraction or natural language processing applications. SpaCy, on the other hand, maybe used to create information retrieval and natural language comprehension systems, as well as to pre-process text in preparation for deep learning. SpaCy’s precision was insufficient for their purposes, and Flair was sluggish. As a result, employing spaCy on languages other than English may provide poor results with pre-trained NER models, whereas Flair supports several languages. SpaCy is still faster, but the improvements they describe make Flair a better fit for their use case (Table 1).

<table><caption>Table 1. Experimental result of NER methods</caption><thead><tr><td>Metrics</td><td>Flair</td><td>spaCy</td></tr></thead><tbody><tr><td>Precision score</td><td>98.8%</td><td>99.6%</td></tr><tr><td>Recall score</td><td>98.9%</td><td>92.4%</td></tr><tr><td>F-score</td><td>98.8%</td><td>95.4%</td></tr><tr><td>Accuracy</td><td>98.6%</td><td>99.8%</td></tr></tbody></table>
# 10 Summary

Named Entity Recognition (NER) plays an essential part in the detection and classification of entities in NLP applications. In this paper, we contributed a new dataset with 10000 unique entities which included the PII Named Entity Recognition task in the medical domain including 17 fields, and proposed methods for measuring the initial performance in terms of recall, precision, and F1-score.

Initially, the model was built from scratch for custom-named entities using the spaCy library. The model performance was good enough to extract all the entities from a new sequence of text. Overall time to build the spaCy model was roughly half an hour. The F1-score for the spaCy model for overall text is 95.4%. SpaCy deals with computational linguistic tasks very well and spaCy NER methodology is very useful. The second model, built using Flair, contextualized word embeddings with the dataset attaining with competitive outcomes. Judging by the F1-score of 98%, the Flair model achieves state-of-the-art performance in the overall corpus. Time spent to build the Flair model was about three hours, which is more as compared to the spaCy model. By utilizing the information from a massive amount of unlabeled data, with language model pre-training it becomes feasible to design a high-quality NER system even with this minimal quantity of annotations.

# 11 Conclusion

Natural Language Processing encounters a variety of issues in its applications. This research provided a comprehensive assessment of the NER taking PII into account and also identified a specific set of entities. Using spaCy and Flair, two very robust and popular frameworks, this research provides a machine learning method for Named-Entity Recognition with PII datasets, a highly inflected language.

This study examined a NER approach that can find interconnections between names, health insurance numbers, phone numbers, email addresses, etc. Importantly, recent advances through deep learning in artificial intelligence have opened up a fresh viewpoint on NER. There is a huge scope of development in this area of the field.

# References

1. Akbik, A., Bergmann, T., Blythe, D., Rasul, K., Schweter, S., Vollgraf, R.: FLAIR: an easy-to-use framework for state-of-the-art NLP. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp. 54–59. Association for Computational Linguistics, Minneapolis, Minnesota, June 2019. https://doi.org/10.18653/v1/N19-4010, https://aclanthology.org/N19-4010

2. Aura, T., Kuhn, T.A., Roe, M.: Scanning electronic documents for personally identifiable information. In: Proceedings of the 5th ACM Workshop on Privacy in Electronic Society, pp. 41–50. WPES 2006, Association for Computing Machinery, New York (2006). https://doi.org/10.1145/1179601.1179608
3. Ghiasvand, O., Kate, R.J.: Learning for clinical named entity recognition without manual annotations. Inf. Med. Unlocked **13**, 122–127 (2018). https://doi.org/10.1016/j.imu.2018.10.011, https://www.sciencedirect.com/science/article/pii/S2352914818301965

4. Hathurusinghe, R., Nejadgholi, I., Bolic, M.: A privacy-preserving approach to extraction of personal information through automatic annotation and federated learning. In: Proceedings of the Third Workshop on Privacy in Natural Language Processing, pp. 36–45. Association for Computational Linguistics, Online, June 2021. https://doi.org/10.18653/v1/2021.privatenlp-1.5, https://aclanthology.org/2021.privatenlp-1.5

5. Kumar, D., Pandey, S., Patel, P., Choudhari, K., Hajare, A., Jante, S.: Generalized named entity recognition framework. In: 2021 Asian Conference on Innovation in Technology (ASIANCON), pp. 1–4 (2021). https://doi.org/10.1109/ASIANCON51346.2021.9544652

6. Kushida, C., Nichols, D., Jadrnicek, R., Miller, R., Walsh, J., Griffin, K.: Strategies for de-identification and anonymization of electronic health record data for use in multicenter research studies. Med. care **50**(Suppl), S82–S101 (2012). https://doi.org/10.1097/MLR.0b013e3182585355

7. Mosallanezhad, A., Beigi, G., Liu, H.: Deep reinforcement learning-based text anonymization against private-attribute inference. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2360–2369. Association for Computational Linguistics, Hong Kong, China, November 2019. https://doi.org/10.18653/v1/D19-1240, https://aclanthology.org/D19-1240

8. Naseer, S., et al.: Named entity recognition (NER) in NLP techniques, tools accuracy and performance (2022)

9. Ramachandran, R., Arutchelvan, K.: Named entity recognition on bio-medical literature documents using hybrid based approach. J. Ambient Intell. Humaniz. Comput., 1–10, March 2021

10. Surabhi, M.: Natural language processing future, pp. 1–3 (2013). https://doi.org/10.1109/ICOISS.2013.6678407

11. Tamper, M., Oksanen, A., Tuominen, J., Hietanen, A., Hyvönen, E.: Automatic annotation service APPI: named entity linking in legal domain. In: Harth, A., et al. (eds.) ESWC 2020. LNCS, vol. 12124, pp. 208–213. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-62327-2_36

12. Ting, K.M.: Confusion Matrix, p. 209. Springer, Boston (2010). https://doi.org/10.1007/978-0-387-30164-8_157

13. Tripathi, S.P., Rai, H.: SimNER - an accurate and faster algorithm for named entity recognition. In: 2018 Second International Conference on Advances in Computing, Control and Communication Technology (IAC3T), pp. 115–119 (2018). https://doi.org/10.1109/IAC3T.2018.8674025