## 4 Discussion  
For the secondary use of clinical documents, it is important to de-identify PHIs. Data augmentation and transfer learning methods can be used as effective methods to overcome the problem of a limited dataset. They can create a high-performance model when there is a problem with a small number of publicly available datasets for de-identification. A major novelty of this study lies in the process of verifying the masked words predicted by BERT through the similarity filter during the data augmentation process. PHI is a set of individual data points, such as a person's name or registration number, which is difficult to augment with the existing medical data knowledge base, such as the unified medical language system (UMLS). Therefore, it is difficult to apply knowledge-based augmentation methods. Additionally, BERT effectively predicts a masked word through context-based inference, but there is a risk that the word may be predicted as a word with a completely different meaning without being verified. We filtered words with completely opposite meanings by verifying the words for a second time through the similarity filter based on fastText embedding, which was trained in advance. When analyzing the results of the study and comparing BERT before and after augmentation, the overall performance showed significant improvements. Labels, including DOCTOR, PATIENT, and USERNAME, showed a performance improvement of more than 5%. Furthermore, in the case of labels with fewer classes such as PROFESSION, ORGANIZATION, and STREET, we could see an immense performance improvement. However, in the case of labels such as LOCATION-OTHER and DEVICE, it was observed that sufficient data for learning was not secured because the number of classes was too small. Although the absolute number of datasets affects the occurrence of this problem, the fact that the configured dataset was unbalanced had a significant impact. The data augmentation method, which increases the absolute number of datasets, was effective in improving the overall performance; however, because the number of major classes is augmented together, there may be a limit to the learning of the minor classes. The learning problem of unbalanced data was solved by the hybrid method of rule-based systems and machine learning in previous studies [8, 11, 37]. Additionally, an over \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Tags**} & \multicolumn{2}{c}{**Precision**} & \multicolumn{2}{c}{**Recall**} & \multicolumn{2}{c}{**F1-Score**} & \multicolumn{1}{c}{**Support**} \\ \cline{2-7}  & **Before** & **After** & **Before** & **After** & **Before** & **After** & \\ \hline DOCTOR & 0.73 & 0.88 & 0.76 & 0.90 & 0.75 & 0.89 & 2525 \\ PATIENT & 0.80 & 0.89 & 0.74 & 0.89 & 0.76 & 0.89 & 2275 \\ USERNAME & 0.95 & 0.98 & 0.85 & 0.91 & 0.90 & 0.95 & 167 \\ PROFESSION & 0.08 & 0.29 & 0.26 & 0.34 & 0.12 & 0.31 & 135 \\ HOSPITAL & 0.70 & 0.84 & 0.63 & 0.76 & 0.66 & 0.80 & 1665 \\ COUNTRY & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 133 \\ ORGANIZATION & 0.00 & 0.05 & 0.00 & 0.17 & 0.00 & 0.07 & 88 \\ ZIP & 0.92 & 0.98 & 0.77 & 0.99 & 0.84 & 0.99 & 416 \\ STREET & 0.28 & 0.86 & 0.32 & 0.74 & 0.30 & 0.79 & 173 \\ CTY & 0.46 & 0.69 & 0.49 & 0.52 & 0.47 & 0.59 & 404 \\ STATE & 0.60 & 0.83 & 0.94 & 0.84 & 0.73 & 0.83 & 227 \\ LOCATION-OTHER & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 16 \\ AGE & 0.92 & 0.95 & 0.86 & 0.96 & 0.89 & 0.95 & 621 \\ DATE & 0.98 & 0.99 & 0.98 & 0.98 & 0.98 & 13,024 \\ PHONE & 0.77 & 0.90 & 0.69 & 0.86 & 0.73 & 0.88 & 665 \\ MEDICALRECORD & 0.96 & 0.95 & 0.95 & 0.98 & 0.95. & 0.97 & 2046 \\ DEVICE & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 0.00 & 31 \\ IDNUM & 0.89 & 0.91 & 0.84 & 0.76 & 0.86 & 0.83 & 675 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Prediction performance by PHI entity before filtered BERT augmentation. sampling method, performed to resolve the unbalanced class, was also considered.