# Applying BERT for Early-Stage Recognition of Persistence in Chat-Based Social Engineering Attacks
## 1 Introduction  
In a chat-based dialogue e.g., between an SME employee and a potential customer, the interlocutors exchange written sentences during their communication. The ability to identify one or more characteristics [1] that can discriminate a normal interlocutor from a malicious one can lead to sufficiently protecting the SME employee from a potential social engineering attack. To achieve her goal, a malicious interlocutor repeats her arguments many times in a conversation to convince and manipulate her partner. This was also confirmed, after processing our CSE corpus [2], where we observed that 83% of social engineering attackers tend to insist on their arguments to exfiltrate the targeted type of critical information. The persistence of an interlocutor to regurgitate the same topic that could lead to sensitive data exfiltration can be considered an enabler of a successful CSE attack. Thus, there is a need to develop a mechanism for recognizing when an interlocutor is continuously trying to lead the conversation to a specific and previously mentioned topic. To cope with such a problem, we can adopt different approaches depending on the pros and cons of each one. Traditional machine learning models require substantial amounts of labeled data to be trained for a recognition task, but labeling data is a time-consuming and expensive activity. In pursuing persistence recognition, we choose to leverage the latest trends in deep learning, especially the Transformers [3] technology. Transformers use substantial amounts of unlabeled raw text for training and take advantage of the linguistic information contained to overcome the difficulty of creating a large amount of labeled data. The Transformer models are very efficient in learning the context and relationships between sequential data, such as words in a sentence. Even if supervised learning using labeled data is an option, learning robust word vector representations in an unsupervised manner of learning can lead to a significant performance boost. In this paper, we describe a hybrid approach for the paraphrase detection task, in the context of CSE attacks, based on a combination of unsupervised pre-training and supervised fine-tuning based on the Transformer-based BERT model [4]. This approach results in our proposed model, called CSE-PersistenceBERT, which learns a universal representation that transfers knowledge with only minimal adaptation to the paraphrase recognition task. Initially, the CSE-PersistenceBERT model has access to a large corpus of unlabeled text on which BERT has been pre-trained, and later it uses the CSE-Persistence corpus, which emerged from the CSE corpus of our previous work [2]. CSE-Persistence corpus generated after manually annotating the CSE corpus and utilizing the CSE ontology [2]. This corpus which is relevant to the downstream task was used to fine-tune CSE-PersistenceBERT. The complete training procedure was performed in two stages: firstly, the BERT model parameters were learned by training the model using unlabeled data on a language modeling objective, and secondly, a part of the pre-trained parameters was fine-tuned on the paraphrase recognition task using labeled data from the CSE-Persistence corpus. The evaluation results confirmed that CSE-PersistenceBERT can recognize the persistence of an interlocutor in an early stage of a chat-based conversation. This can result in an indicator of a potential social engineering attack, given that the topic of the conversation is an entity in our CSE ontology. This work contributes to improving knowledge about utilizing existing state-of-the-art deep learning models as cyber defense mechanisms. Our study provides a better understanding of how we can tailor a BERT-based language model to expose the malicious intent of an interlocutor. Furthermore, detailed implementation details are given about fine-tuning such a model using an in-context and appropriately annotated corpus for the paraphrase recognition task.