# Using BiLSTM Networks for Context-Aware Deep Sensitivity Labelling on Conversational Data
## 3 Background and Approach
### Auxiliary Features (POS, IC, WS)  
Although relying solely on word embeddings for prediction can deliver acceptable performance, we propose three additional features to enrich the learning significance of the semantic and syntactic abstraction of word embeddings: POS tags, Information Content (IC) and WordShape (WS).  
[PERSON] et al. [15] showed that specific POS n-gram sequences can be correlated with sensitive information. This provided an incentive to use POS tags as auxiliary features in this work. We extract the POS tags of the sequences using SpaCy's v2.1 ([[https://github.com/explosion/spaCy](https://github.com/explosion/spaCy)]([https://github.com/explosion/spaCy](https://github.com/explosion/spaCy))) POS tagger, preferred for its speed, industrial strength and convenience.  
Information Content (IC) offers a quantitative metric for general purpose sensitivity [19]. IC estimates the information carried by a specific token \(t\) in a given context, relying on the information-theoretic assumption that rare terms typically convey more information than general terms (e.g., \"surgeon\" vs. \"doctor\"). Thus, we expect that incorporating IC in our model can better classify sensitive tokens of particular classes. An example is labelling passwords: due to their random nature, the embeding of a password will most often result in the Out-Of-Vocabulary (OOV) embedding. Hence, apart from the surrounding context, there is nothing differentiating it from other OOV tokens. With this in mind, introducing IC features can be advantageous. As this, however, directly depends on the size and content of the corpus used to calculate the information content, a massive general corpus is required for general purpose estimations. To extract the IC of tokens we used the Bing search engine API ([[https://azure.microsoft.com/en-gb/services/cognitive-services/bing-web-search-api/](https://azure.microsoft.com/en-gb/services/cognitive-services/bing-web-search-api/)]([https://azure.microsoft.com/en-gb/services/cognitive-services/bing-web-search-api/](https://azure.microsoft.com/en-gb/services/cognitive-services/bing-web-search-api/))). Notably, Google could be more accurate, since it maintains the largest and most updated page index to date. Yet, it was not possible to use Google's search API for this project due to its API restrictions on repetitive use. As suggested by [PERSON] et al. [19] only nouns are queued for the IC extraction as the rest of the part-of-speech types have a dynamic meaning that effectively makes search engine queries unreliable for IC calculation. These tokens are assigned an IC value of 0 by default.  
Lastly, a morphological word feature, _WordShape_, was introduced in our experiments. We use the term _WordShape_ as the textual representation of a word's morphology, which is implemented through transforming words into character sequence templates. This can contribute to learning the sensitivity correlations with structured data such as credit-card numbers, national insurance numbers, and phone numbers. To generate the WordShape features, SpaCy's v2.1 parser was used (https://spacyio/usage/linguistic-features).