## 5 CSE Corpus Evaluation  
Named entity recognition (NER) [42] is a natural language processing method to extract in-context information from unstructured text data such as e-mails, web articles or chat-based dialogues. Using NER we can identify entities, like people, organizations, places etc., which exist in text. There are several well-known libraries like spaCy, Apache OpenNLP and TensorFlow with pretrained NER models that can be used to build a NER system that identifies the aforementioned entities. Furthermore, these pretrained models can be modified to recognize additional entities by training them with an annotated corpus that contains the additional tags. Such a corpus is the already presented CSE corpus, where personal, IT and enterprise information has been identified and tagged. We evaluated the CSE corpus by building and testing a NER system trained with it. This NER system takes as input preprocessed CSE dialogue sentences and identifies, in each sentence, the words or text spans that can be labelled by a tag that is associated with an entity of the CSE ontology. This kind of task is a sequence-labelling task where a token is classified as belonging to one or none of the annotation classes. We used Keras [43] and TensorFlow [44] to build, train and evaluate a standard bi-directional Long Short-Term Memory neural network (bi-LSTM) [45] for CSE attack named entity recognition. LSTMs were designed to overcome Recurrent Neural Networks (RNNs) inefficiency, since they fail to learn long dependencies due to bias toward their most recent inputs. RNNs are a family of neural networks that operate on sequential data: they take as input a sequence of vectors (x\({}_{1}\), x\({}_{2}\)..., x\({}_{N}\)) and produce as output another sequence that represents information about every step in the input sequence. LSTMs incorporate a memory cell to capture long-range dependencies. Using several gates, LSTMs control the proportion of input that is given to the memory cell, as well as the proportion from the previous state that should be forgotten. To build the NER system the following steps were taken:  
1. _Preprocessing:_ The CSE corpus has already been annotated in IOB format and thus each line of the corpus can easily be read and stored as a list of token-tag pairs. Then, each token was represented by a word embedding using a pre-trained English language model of the spaCy NLP library. 2. _Building:_ Using Keras, a bi-LSTM model was constructed, which is comprised of two compound layers: \(\circ\) _Bi-directional LSTM layer_, where the forward and backward pass were encapsulated and the input and output sequences returned were stacked. Figure 15: Five most frequent bi-grams per category. (a) Neutral, (b) Personal, (c) IT, (d) Enterprise. _Classification layer_, where classification was performed to every position of the sequences in the stack. The SoftMax activation function was used to scale the output and obtain sequences of probability distributions. 3. _Training:_ To train the model in Keras, a loss function for the model was specified to measure the distance between prediction and truth, and a batch-wise gradient descent algorithm was specified for optimization. 4. _Assessing:_ The performance assessment of the model was conducted by applying the model to the preprocessed validation data. For each sample dialogue sentence and each token in a sentence, a tensor was obtained that contained a predicted probability distribution over the tags. The tag with highest probability was chosen, and for each sentence and each token the true and predicted tags were returned. More specifically, inside the bi-LSTM neural network the following actions occurred in sequence:  
1. Each dialogue sentence was split into a sequence of token-tag pairs. 2. Each token-tag pair was represented by a word embedding vector. 3. The word embeddings were provided by spaCy NLP library and pretrained to encode semantic information. This approach is known as Transfer Learning [46]. 4.