# Development of Dialogue Management System for Banking Services
## 4 Methods
### Tokenization and Feature Extraction  
The dataset introduced earlier has specific characteristics, such as misspellings and examples with an incomplete sentence structure. One of the main reasons of having many misspellings in Azerbaijan chat data is alphabetical differences between Azerbaijan and English. They both are Latin-script alphabets, but in the English alphabet there are 26 letters, whereas in the Azerbaijan alphabet there are 32. The Azerbaijan letters--\"c\", \"g\", \"g\", \"o\", \"u\", \"o\" do not exist in the English alphabet, and the letter \"w\" does not exist in the Azerbaijan alphabet. Thus, most of the misspellings occur due to the usage of the English keyboard layout. For example, instead of \"o\" and \"g\", customers may use \"a\", \"e\", and \"sh\", \"s\", \"w\" correspondingly, resulting in several wrong variations of the same word (e.g., correct word: \"sahor\"--misspellings: \"sheher\", \"sahar\", \"weher\", etc.). Other types of misspellings that occur in the dataset are caused by missing or adding a letter, confusing or switching the order of the letters, etc. Considering the amount of the data and the problems explained above, whitespace tokenizer and sparse features are implemented in the data processing step. Thus, tokens obtained using whitespace as a separator fed into the count vectors featurizer, which results in a BoW representation. BoW representation, known as one-hot encoding, creates sparse features for a given text dataset. In other words, the resulting representation consists of unique vectors with zeros and ones for each word with the dimension that is equal to the vocabulary size. In order to extract the internal features of words, the character n-gram model is applied with parameters min_ngram = 2 and max_ngram = 4. In other words, we implemented both word level and character n-gram level BoW representations of the dataset. Additionally, character n-gram representations are effective in handling misspelling problems encountered in the dataset. Apart from this, for machine learning models TF-IDF Vectorizer is also used. Unlike the BoW method, this creates occurrence-based representations. Here, a greater importance is given to the words that have more occurrences in the given text. In contrast, less importance is given to words that are common across all intent classes, and their impact in intent identification is very low. Figure 4: Part of natural language understanding (NLU) component. Figure 3: Architecture of the module-based chat-bot system. In addition, named entities extracted from user messages were added as features. The main goal of the named entity is to recognize important information from the raw text, for example, person names, organizations, geological places, dates, time, etc. In conversational AI, named entity recognition (NER) can be implemented to extract general entities such as those listed above, as well as domain based custom entities that are helpful to understand customer needs and respond accordingly. For instance, in the financial industry, especially within the bank domain, information like type of a credit or deposit, plastic card differences, interchange fee, and transaction percentage is considered valuable. Along with Part of Speech (PoS) tagging, NER related to the sequence labeling problems of NLP is important part of Information Extraction (IE) and Question Answering (QA) [51; 52]. For NER classification, different machine learning methods can be implemented. One of the approaches for entity recognition is using CRF. CRF can be implemented inside DIET Classifier, if the class of entity recognition is set. In this case, Entity Loss and Intent Loss are summed up to get a total loss of DIET Classifier. Both sparse and dense features first go through two transformer layers, followed by the CRF tagging layer for entity label prediction. Apart from this, CRF based entity recognition can be implemented separately by using custom entity extraction models.