# Development of Dialogue Management System for Banking Services
## 6 Discussion  
Main outcomes of this research can be divided into four groups: NER classification, language identification, intent classification, and dialog management. We used named entities for two purposes: as features for the intent classification and for the dialog management. In the dialog management part of the bot, achieving the highest accuracy for entity extraction is considered as one of the main goals because responses of the bot in different dialog turns highly depend on the extracted entities. In the experiments, we achieved accuracy for CRF models at approximately 95%, and DIET Classifier and the hybrid models for multi-word and single word entities showed the highest performances at about 98%. To increase the accuracy of the overall entity extraction model, we applied the hybrid model using the DIET Classifier and the multi-word entity extraction model. Authors in [21] suggested HERMIT architecture for detecting named entities in the NLU-Benchmark dataset for the English language and obtained approximately an 85% F1-score. Similar results (approximately 86%) were achieved by using DIET architecture in [23]. Considering the differences in utilized datasets, various language characteristics, and the number of entity classes, comparing our results with the research benchmark was problematic to a certain degree. \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline  & **fastText (ngram-2-5) + TF-IDF** & **fastText (ngram-2-5)** & **fastText (ngram-2-5) + CountVectorizer** \\ \hline  & **Acc.** & **F1** & **Acc.** & **F1** & **Acc.** & **F1** \\ \hline
**LR** & 0.85 & 0.84 & 0.84 & 0.84 & 0.89 & 0.88 \\ \hline
**SVM** & 0.84 & 0.83 & 0.82 & 0.81 & 0.89 & 0.88 \\ \hline
**NN** & 0.87 & 0.87 & 0.82 & 0.83 & 0.88 & 0.89 \\ \hline
**DIET** & 0.868 & 0.866 & 0.878 & 0.876 & **0.882** & 0.879 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Intent classification results with fastText embeddings. Reported numbers are accuracy and weighted average F1-scores. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & **Accuracy** & **F1-Score** & **Precision** & **Recall** & **Total Correct Conversations** \\ \hline
**Results** & 0.83 & 0.84 & 0.89 & 0.81 & 0.50 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Dialogue Manager (DM) test results. Reported numbers are accuracy and weighted average F1-scores, precision and recall. Results of the language identification method represented by the authors of [10] were 96% with a model trained on the multilingual Wikipedia dataset. Additionally, probabilistic models with evidence accumulation outperformed other methods on a multilingual Twitter dataset [11]. Although fastText language identification models showed high accuracy at 92-99% on Wikipedia, TCL, and EuroGov data, the performance of both fastText models (light and heavy) were approximately 55% applied to the text samples with spelling mistakes and 89-94% on spelling-corrected samples (Azerbaijani text data). The reason for them not performing well is that the fastText models are trained on the clean data resources with correct orthography and sentence structure. In contrast, custom ML models trained on our dataset have 98-99% accuracy for different models. To conclude, due to higher accuracy (99%) and lower complexity compared to others, we added a Language Identification component trained with Logistic Regression approach to the final chat-bot model. Despite the results of the DIET Classifier using only sparse features, it achieved 88% accuracy, and using both sparse and dense (ConveRT) features is achieved 90% on the NLU-Benchmark dataset, accuracy of DIET on our dataset changes in the range of 81-89%. It can be noted that for a noisy dataset in the agglutinative language, classic ML models such as SVM and NN almost show the same performances as the transformer base algorithm. The only work on intent classification problem in the Azerbaijan language were in [17; 18].