# Automatic Detection of Sensitive Data Using Transformer-Based Classifiers
## 8 Results  
To evaluate the effectiveness of the two architectures, we compare them on both the sensitive topic detection task and in its classification into one of the four macro-categories. To do so, we use the test data set obtained as described in Section 5.2. The flat multi-label model is based on a single neural network whose output consists of a vector of four elements, each representing a coefficient that expresses how likely the input sentence is to belong to one of the macro-categories of sensitive topics; in case one or more of the outputs exceed 0.5, the related label is assigned to the input sentence. Using this approach, if no labels can be associated to the input sentence, then the sentence topic is classified as non-sensitive. We first compared both models on the task of sensitive-topic detection, treating the union of the four macro-categories as the _Sensitive_ class. Regarding the flat multi-label model, we considered each prediction with at least one output above the threshold as the detection of a sensitive topic. Table 5 shows that both models reached similar results, with the best \(F_{1}\)-score of 0.95 obtained by the binary neural network of the hierarchical model. Next, we compared the two models on the sensitive topic recognition task, using only the samples of the test set not labeled as _Other_. The output of the flat multi-label model has been considered correct if the output corresponding to the correct prediction is greater than 0.5, thus allowing the association of the input with multiple labels at one time. The results in Table 6 show that, while the multi-class neural network of the hierarchical model has been able to obtain almost the same performance as its binary counterpart even if the task is obviously harder, the flat multi-label one obtained a lower \(F_{1}\)-score in this task: the recall value of the flat multi-label model is lower than the recall of the flat multi-class model of the \begin{table}
\begin{tabular}{l l l l l l} \hline \hline  & & **Optimizer** & **Batch Size** & **Learning Rate** & **Dropout** \\ \hline \multirow{2}{*}{**Hierarchical**} & _Binary_ & RMSprop & 32 & \(1\times 10^{-5}\) & 0.3 \\  & _Multi-class_ & Adam & 32 & \(1\times 10^{-5}\) & 0.5 \\ \hline
**Flat Multi-label** & & RMSprop & 16 & \(1\times 10^{-5}\) & 0.3 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Best hyperparameters combination for each neural network. \begin{table}
\begin{tabular}{l l l l l} \hline \hline  & **Precision** & **Recall** & **\(F_{1}\)-Score** & **Runtime (s)** \\ \hline
**Binary** & 0.97 & 0.93 & 0.95 & 0.11 (\(\pm\)0.01) \\ \hline
**Flat Multi-label** & 0.96 & 0.92 & 0.94 & 0.12 (\(\pm\)0.01) \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Comparison between the binary neural network of the hierarchical model and the flat multi-label model. hierarchical classifier on every macro-category. This might depend on the class unbalance; however, the hierarchical model seems to have better overcome this problem, possibly because it has been trained only on sensitive data according to the two-stage strategy of the hierarchical classifier. An important detail to be noticed is that out of 2400 sentences, only four have been labeled with two or more labels, and in all cases, one was the correct label, as shown in Table 7. This means that in the comparison the selectivity of the output reduces the advantage for the flat classifier to generate possible multi-class outputs almost to zero (Notice that in computing the \(F_{1}\) score, producing more than one output favors an increase of both precision and recall). Finally, we analyze both models on the complete task: detecting if a sentence contains sensitive topics and choosing the appropriate macro-category. We do so on the entire test set and according to the same result assessment policies applied in the previous comparisons.