## 5 Experimental Protocol  
As mentioned in the related literature outline in Section 3, there is currently no established standard for evaluating chatbots. To accomplish this task, we proposed TroubleshootingBot, a novel experimental protocol whereby it was possible to create a test dataset. It should be noted that the training data is not subject to this protocol, since it comes from the knowledge database. The test datasets were utilized to evaluate the IR and NER models in two pre-established scenarios.  
This experiment evaluates the behavior of our solution when users interact with the chatbot. One of our solution's primary challenges is accurately identifying the ATMs and providing the appropriate procedures related to this problem while extracting all relevant information to give an appropriate response. To better understand the critical points of the conversation, we divided the analysis into two scenarios. In the first scenario, we analyzed intents classification when users described the ATM's problem. In the second scenario, we evaluated the employment of the NER technique associated with IR at critical conversational flows.  
Only automated evaluation metrics were utilized to analyze the system, while human evaluation metrics [12] were not considered. Instead, we employed a confusion matrix and the metrics specified in Section 2.4 to evaluate this scenario. These metrics are particularly useful in assessing the system's ability to identify user intent and correctly fill the associated slots.  
We conducted a preliminary test utilizing phrases derived from the knowledge database. This process is described below.  
To perform this experiment, we followed the steps below:  
1. We meticulously selected critical modules to focus on to assess the effectiveness of our intent recognition (IR) and named entity recognition (NER) algorithms. We adopted this protocol to evaluate two scenarios: IR detection and NER detection. Specifically, we chose to only evaluate target intents directly related to ATM modules for our IR analysis. For the NER task, we selected entities essential for identifying potential issues with ATMs.
2. Randomly selected phrases from the knowledge database commonly utilized to reference ATM modules were selected. However, to ensure the reliability of our evaluation, we could not utilize these same phrases for testing. Therefore, we entered these phrases into a third-party paraphrasing tool, which generated a variety of completely new phrases with similar contexts. This process provided a set of phrases with unseen variations to evaluate the accuracy of our IR and NER algorithms.
3. To ensure that the generated phrases reflected the intended context, we comprehensively analyzed each phrase to identify any potential errors in interpretations. We then labeled each generated phrase according to its corresponding intent or entity by manually analyzing its underlying meaning and identifying relevant entities.
4. To further validate the reliability of our modules, we manually generated a set of phrases intentionally created to deviate from the expected context for IR. Additionally, for NER, we created training phrases that did not contain any expected entities and labeled them as no-match.
5. By employing this technique, we developed a dataset for our evaluation scenario.