# Using BiLSTM Networks for Context-Aware Deep Sensitivity Labelling on Conversational Data
## 7 Evaluation  
Based on the experimental results presented in Section 6 we evaluate BiLSTM\({}_{\text{BERT+WS}}\) + CRF against temporal and semantic context sensitivity labelling.  
For temporal context sensitivity labelling, the final model is evaluated on a dataset that was manually built and annotated. Manual annotation is time-consuming and thus the dataset is small compared to the synthetic one. It consists of 60 text sequences, specifically written in a way that token sensitivity is directly dependent on temporal context.  
Due to manual annotation, there were cases where stopwords were annotated as sensitive tokens but not picked up by the model, or vice versa, causing a drop in the evaluation metrics. A confusion matrix (Figure 4) on class-level (rather than token level) granularity offers a better evaluation that is invariant to annotation discrepancies. In effect this demonstrates whether the model managed to identify the sensitivity class of the text. Figure 4 illustrates that the majority of sensitivity types are classified correctly. Most of the incorrect classifications are confused with the NON_SENSITIVE class. Note that the hardest classes to classify are the \(DEMOG\_SOCECON\_INFO\) and POLITICAL_INFO. Additionally, it is important to highlight that the 76.33% and 73.07% F1 score in token-level and class-level experiments, respectively (Figure 4), support our hypothesis that a BiLSTM model can be used for temporal context sensitivity annotation.  
For our Semantic Context Evaluation, we performed a comparative evaluation against Google's DLP system ([[https://cloud.google.com/dlp/](https://cloud.google.com/dlp/)]([https://cloud.google.com/dlp/](https://cloud.google.com/dlp/))), which provides industrial-strength sensitive data annotation for over 80 sensitive data types. Google DLP is chosen for its industrial strength and was seen as a suitable benchmark for semantic evaluation since it also uses an automated methodology.  
Figure 4: Confusion matrix for temporal context evaluation.  
To perform an as accurate as possible comparison, we test the performance of the two systems (our model and DLP) solely on sensitive data types, which are common between the two. Google DLP is provided as a platform with standard functionality, allowing solely for user intervention in (i) selecting InfoTypes (equivalent to _topics_ of our Sensitivity Classes) and; (ii) creating templates to support a more structured data detection approach. To that end, another dataset was created manually, containing text sequences that include sensitive tokens affected by semantic context (as an example of a sample in this dataset, consider the sentences \"I am a male\" and \"I am a man\", they both reveal the same gender information but the wording is different). In the absence of sizeable overlap between the two systems on sensitivity topics, we select only those commonly appearing in both. These are: _Email, Credit Cards, Age, IP, SSN, Ethnic Group and Gender_.  
Results of our comparison are provided in Figure 5. For all sensitive data types, our system outperforms Google's DLP service. Particularly it was observed that, when noise that can affect the syntactic but not the semantic meaning of sensitive data was added, Google's DLP fails to annotate the sensitive tokens. On the contrary, our system exhibits resilience against such noise with a relative accuracy advantage over the Google DLP of 42.65%.