# Detecting Personally Identifiable Information Through Natural Language Processing: A Step Forward
## 3 Materials and Methods
### Pre-Trained Model Selection  
As we know, the selection of machine learning and deep learning models for personal information recognition is a critical phase, as the models need to be not only accurate but also efficient in operation. After a thorough analysis of available solutions, traditional machine learning models, such as neural networks, as well as more advanced deep learning models using NLP techniques, were considered. The models' accuracy was balanced with computational efficiency to prevent the system from being too slow or resource-intensive in a production environment. Special attention was given to the models' ability to handle complex and unstructured data, such as free text containing sensitive information. Driven by the increasing demands of NLP, several pre-trained transformer models have been proposed, including BERT (Bidirectional Encoder Representations From Transformers), RoBERTa, DeBERTa, GPT, and ELECTRA, which are thoroughly described by [PERSON] et al. in [32] and by [PERSON] et al. in [33]. To maintain full control over the data generated by the PII recognition experiment and to evaluate its performance, we decided to use the pre-trained BERT model, also based on the high performance claimed by the scientific community in its use for linguistic tasks, as quantitatively reported by [PERSON] et al. in [34], where the performance of models such as ELMo, OpenAI GPT, BiLSTM + ELMO + Attn, Pre-OpenAi SOTA, CVT, and CSE were compared with BERT (BERTBASE and BERTLARGE), using key benchmarks such as GLUE and SQuAD v1.1. For example, the GLUE benchmark requires the evaluation of performance across the following eight specific tasks:  
Figure 2: PII detection software architecture. * MNLI (Multi-Genre Natural Language Inference), which evaluates the model's ability to determine the relationships between a pair of sentences;
* QQP (Quora Question Pairs), which assesses whether two questions are semantically equivalent;
* QNLI (Question Natural Language Inference), which determines if a paragraph of text contains sufficient information to answer a specific question;
* SST-2 (Stanford Sentiment TreeBank), which classifies the sentiment of sentences as positive or negative;
* CoLA (Corpus of Linguistic Acceptability), which determines the grammatical acceptability of sentences;
* MRPC (Microsoft Research Paraphrase Corpus), which recognizes paraphrased sentences;
* RTE (Recognizing Textual Entailment), which determines the entailment relationship between sentences. According to [PERSON] et al. (2018), in Table 1, we provide a summary of the GLUE benchmark results, demonstrating BERT's superior performance in text classification. We observe that the authors in (2018) focused not explicitly on the detection of PII but on text classification models that form the basis of PII recognition. During the pre-training process, BERT utilizes two key tasks: (i) the Masked Language Model (MLM), where certain words in a sentence are masked, and the model must predict the missing words; and (ii) the Next Sentence Prediction (NSP), where the model determines whether two sentences are logically connected. To effectively perform the NSP task, a bidirectional approach to text analysis is essential. This is achieved using 12 Encoders in BERTBASE or 24 Encoders in BERTLARGE. These Encoders enable bidirectional understanding via the multi-headed self-attention mechanism, which compares each word in the sentence with both preceding and succeeding words. In addition, the Encoders ensure computational efficiency and scalability by supporting parallel computation, particularly benefiting from batch processing.