# Leveraging Dialogue State Tracking for Zero-Shot Chat-Based Social Engineering Attack Recognition
## 6 Discussion  
Early dialogue-state tracking (DST) datasets were developed to be specific to a particular domain due to the difficulty in building models that can effectively track dialogue states for multiple domains. However, with the recent release of multi-domain datasets and the incorporation of machine learning-based methods, it has become possible to build models that can track states for multiple domains using a single set of training data. To evaluate the ability of models to generalize in zero-shot settings, we created the SG-CSE Corpus and included evaluation sets containing unseen services. To address the issue of limited dialogue resources, data augmentation can be explored as an option. Augmenting the training dataset by adding more diverse examples can improve performance. Source-based  
Figure 11: SG-CSE BERT model performance. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**System**} & \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Parameters**} & **Active** & \multirow{2}{*}{**Req Slot**} & **Agg Goal** & **Joint Goal** \\  & & & **Accuracy** & & **F1** & **Accuracy** & **Accuracy** \\ \hline \multirow{2}{*}{Seen} & BERT\({}_{\text{BASE}}\) & 110 M & 85.2 & 89.6 & 74.1 & 56.7 \\  & BERT\({}_{\text{BASE}}\) & 110 M & 53.8 & 48.3 & 31.4 & 24.9 \\ \multirow{2}{*}{All} & BERT\({}_{\text{BASE}}\) & 110 M & 69.5 & 68.9 & 52.7 & 40.8 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: SG-CSE Attack state tracker performance. augmentation generates sentences by changing a single variable value in a sample utterance, while target-based augmentation takes portions of sentences from different places in the training data and recombines them. SG-CSE BERT is a model that is built around BERT, a pre-trained transformer-based model that has been trained on a large corpus of text data. BERT has demonstrated strong generalization ability across a wide range of natural language processing tasks and domains. When fine-tuned on a specific task and domain, BERT is able to learn specific patterns and features of the task and domain, which allows it to achieve good performance. However, if BERT is not fine-tuned, it may be able to detect new unseen intents, but it would not have enough information to generate the corresponding slot values. Moreover, it may not be able to detect new unseen services or new unseen domains. Large-scale neural language models trained on massive corpora of text data have achieved state-of-the-art results on a variety of traditional NLP tasks. However, although the standard pre-trained BERT is capable of generalizing, task-specific fine-tuning is essential for achieving good performance. This is confirmed by recent research, which has shown that pre-training alone may not be sufficient to achieve high accuracy in NLP tasks. For example, the performance of a pre-trained model on a downstream task may be significantly improved by fine-tuning it on a smaller in-domain dataset. The SG-CSE Corpus is designed to test and evaluate the ability of dialogue systems to generalize in zero-shot settings. The evaluation set of the corpus contains unseen services, which is important to test the generalization ability of the model. Our evaluation set does not expose the set of all possible values for certain slots. It is impractical to have such a list for slots like IP addresses or time because they have infinitely many possible values or for slots like names or telephone numbers for which new values are periodically added. Such slots are specifically identified as non-categorical slots. In our evaluation sets, we ensured the presence of a significant number of values that were not previously seen in the training set to evaluate the performance of models on unseen values. Some slots, like hardware, software, etc., are classified as categorical, and a list of all possible values for them is provided in the schema. SG-CSE BERT is designed to handle dialogue-state tracking for a larger number of related services within the same domain.