# Detecting Personally Identifiable Information Through Natural Language Processing: A Step Forward
## 5 Discussion
### Comparison  
In our evaluation, we conducted a quantitative comparison with existing works in the literature, focusing on studies with open English documentation. This approach allowed us to contextualize our results and assess our model's performance relative to prior work. Specifically, [PERSON] et al. [45] evaluated the performance of several models for PII recognition tasks, including Mistral 7B, CodeLlama 7B, Long Coderbase, GraphCodeBERT, CodeT5-220m, CodeT5 Plus-770m, and CodeBERT. The dataset used in their study focuses on privacy elements from the California Consumer Protection Act (CCPA) and the General Data Protection Regulation (GDPR). It consists of records from public databases, with annotation on the following categories of PII:  
* Device unique ID;
* Account or person identifier;
* Demographic data;
* Commercial or financial information;
* Biometric data;
* Employment information;
* Education level information;
* Job positions held. \begin{table}
\begin{tabular}{l c c c c} \hline  & & **TVT** & **K-Fold 1** & **Strat. k-Fold 4** \\ \hline
**PII Name** & **Total Classes** & **False** & **False** & **False** \\  & **Predictions** & **Predictions** & **Predictions** & **Predictions** \\ \hline Full name & 10,430 & 26 (0.249\%) & 9 (0.086\%) & 17 (0.163\%) \\ \hline Surname & 6428 & 24 (0.373\%) & 8 (0.124\%) & 17 (0.264\%) \\ \hline No PII & 14,404 & 10 (0.069\%) & 7 (0.049\%) & 10 (0.069\%) \\ \hline Address & 1131 & 7 (0.619\%) & 5 (0.442\%) & 5 (0.442\%) \\ \hline Job area & 236 & 2 (0.847\%) & 4 (1.695\%) & 4 (1.695\%) \\ \hline First name & 1384 & 2 (0.145\%) & 2 (0.145\%) & 5 (0.361\%) \\ \hline Job area + Full name & 1896 & 2 (0.105\%) & 2 (0.105\%) & 2 (0.105\%) \\ \hline State & 464 & 1 (0.216\%) & 3 (0.647\%) & 2 (0.431\%) \\ \hline Surname + Address & 985 & 3 (0.305\%) & 0 (0.000\%) & 1 (0.102\%) \\ \hline Address + Full name & 1728 & 3 (0.174\%) & 0 (0.000\%) & 1 (0.058\%) \\ \hline \end{tabular}
\end{table} 
Table 6: False prediction summary for different PII sensitivity levels. [PERSON] et al. [45] primarily focused on assessing the ability of large language models (LLMs) to challenge inaccurate or misleading security advice, aiming to determine if these models can be reliable sources of guidance in real-world scenarios. Figure 9 compares the performance of the models from [45] with our BERT model across four evaluation metrics: accuracy (9a), precision (9b), recall (9c), and F1-score (9d). These metrics were computed based on the following standard definitions:  
* Accuracy = (True Positives + True Negatives)/(Total Samples);
* Precision = True Positives/(True Positives + False Positives);
* Recall = True Positives/(True Positives + False Negatives);
* F1-Score = 2
* Recall)/(Precision + Recall). In all cases, our BERT model shows clear improvements over the models in [45], achieving a 7.47-point increase in accuracy, a 6.99-point increase in precision, a 7.57-point increase in recall, and a 7.52-point increase in F1-score. The second comparison we conducted focused on the experimental work by [PERSON] et al. [46], where the GPT-4 model was used to identify students' personal information in an educational setting. The dataset analyzed consisted of student posts from various subjects, including economics, mathematics, design, gamification, business trends, poetry, mythology, probability, and immunology, written between 2012 and 2015. After applying filtering and balancing operations, the dataset was reduced to 3505 posts from 2882 unique students. In our comparison, we evaluated the precision and recall of the models presented in [PERSON] et al. [46] and compared them with those of our BERT model, which was trained using a separate training-validation-test procedure. Precision and recall were calculated using the standard formulas outlined earlier. Figure 9: _Cont._  
Figure 10 compares the precision and recall indices between the models from [PERSON] et al. [46] and our BERT-trained model.