## 4 Application and Results
### CSE Corpus Annotation
#### 4.2.5 Annotation Task  
The annotation task performed had the target to label the words of CSE corpus based on their semantic and syntactic characteristics. The two cybersecurity experts were responsible of labelling the words based on their semantic characteristics, and thus performed semantic annotation. By annotating the semantic characteristics of the words, the background information in each dialogue was linked with the CSE ontology. The syntactic characteristics of the words were labelled by the use of the specific-purpose annotation software we developed to perform syntactic annotation by extracting statistical information regarding the words used by social engineers. Table 4 presents the two different types of annotation and the specific answers that we were looking for. The syntactic characteristics that are annotated are the POS type, and the bi-grams which denote the syntactic relationship between pairs of words. By tagging the POS type and bigrams we were able to perform subsequent phrase structure analysis and extract the dependency trees of the corresponding phrases. This extra information can be used by parsers for Named Entity Recognition, n-grams identification, and Bag-of-Words representation. The labelled words or text spans were further processed in order to extract statistical information valuable for the task of CSE attack recognition. Furthermore, for each sentence, a plethora of counters was also attached as metadata. This metadata information included vocabulary words frequency, words sequence pattern, words context relativity, urgency indicators, number of URL or paths appeared, blacklisted verb and noun combinations, backlisted bi-grams (e.g., tech support, USB stick and others). Moreover, the sentences similarity was measured to be used as attacker persistence metric; i.e., if the attacker uses different sentences but the sentences are highly similar, this means that the attacker persists on his initial intent; this information counts also as metadata information for the annotation task. GATE and Prodigy handled all of the annotation steps along with WordNet [37] and LIWC [38] software. Figure 11 illustrates a tree-structured taxonomy of the syntactic and semantic annotation targets. The preferred encoding scheme to tag the existing chunks of text (word, or text spans) was based on the IOB format [39]. In the IOB encoding scheme, the \"_I-_\" prefix of a tag indicates that the tag is inside a chunk, \"_O_\" indicates that the tag does not belong to a chunk and the \"_B-_\" prefix of a tag indicates that a token is the beginning of a chunk. For example, the sentence '_Mr. [PERSON] is the Head of Financial Department, and his office is on the second floor_' using the IOB format is presented in Table 5. \begin{table}
\begin{tabular}{c c} \hline \hline
**Syntactic** & **Semantic** \\ \hline What Part-of Speech (POS) type is it? & Is it a CSE ontology entity? \\ Is it a bi-gram? & What role does it play in the CSE ontology? \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Questions per annotation type. \begin{table}
\begin{tabular}{c c c} \hline \hline
**No#** & **Sentence** & **Tag** \\ \hline
1 & You will find contact numbers on the _Intranet_ & IT \\
2 & Let me get that name again and give me your _employee number_ & Enterprise \\
3 & Our _project leader_ is [PERSON] & Enterprise \\
4 & I can cut some corners and save some time, but I’ll need your _username_ & IT \\  & and _password_ & IT \\
5 & Okay, can you tell me again your _employee ID number_. & Enterprise \\
6 & I’ll just _hit reset_ and the old one will be wiped out & IT \\
7 & For _account identification_ may you please provide your _account number_ & IT \\
8 & [PERSON], Its [PERSON]... go ahead and try your _network connection_ & IT \\
9 & May I have your _full name_ and _email address_ please & Personal \\
10 & Rest assured that they will be able to read the chat transcript as well as & - \\  & the documentation in your case & \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Important terms and tags.