## 6 Results
### First Scenario  
_IR Evaluation_  
This scenario was developed to enable the evaluation of the performance of the IR process executed in our chatbot. Table 7 presents the average result for the automated evaluation metrics (Section 2.4) that were utilized to evaluate the intent's related ATM defects.  
The results presented above demonstrate that the IR process executed by the chatbot was highly effective in identifying the user's intent with an accuracy of 0.95, indicating that a significant number of phrases from the testing set were correctly classified. The second metric, precision, with a value of 0.94, indicates that a very low rate of testing phrases was misclassified. This is crucial, as the users rely on the chatbot to correctly classify their input to receive the procedure that may solve the ATM's issue. If the chatbot misclassifies a user's intent, it could provide the wrong procedure, which may cause the banking agent to focus on other parts of the ATM unrelated to the given issue. This action may result in a waste of time and resources and a frustrating experience for the user. The recall metric result (0.92) indicates that the IR model missed a low rate of testing phrases. However, this can be a point of improvement, as a higher recall score indicates that the model could correctly identify more of the relevant input.  
The figure below illustrates the confusion matrix Figure 3, where the no-match class represents the utterances in the model that lacked confidence in its predictions to meet the threshold for accurate classification. We analyzed the confusion matrix to better understand the chatbot's performance when the user describes a defect.  
The matrix above reveals several noteworthy findings upon closer examination. First, the model accurately classified utterances related to keyboards, card readers, video monitors, and receipt printers without misclassification. However, it encountered challenges in correctly classifying phrases related to the check scanner, with a recall of 0.89 and a high frequency of confusion with the no-match class. This result indicates that predictions for utterances related to this intent were often below the threshold.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Entity Group** & **Entity** & **Number of Synonyms** & **Number of Evaluation Phrases** \\ \hline \multirow{2}{*}{Led status} & On & 7 & 20 \\  & Off & 6 & 37 \\  & paper & 3 & 20 \\ \multirow{2}{*}{Screw defect} & banknote & 5 & 19 \\  & check & 3 & 18 \\ \multirow{2}{*}{No-match} & - & - & 29 \\ \hline Total & & 24 & 143 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Overview of the entities that were analyzed during the experiments. The table below describes the number of synonyms utilized to train the NER model and the corresponding number of testing phrases utilized for the evaluation process.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Accuracy** & **Precision** & **Recall** & **F1-Score** \\ \hline
0.95 & 0.94 & 0.92 & 0.93 \\ \hline \hline \end{tabular}
\end{table}
Table 7: Macroaverage metric results.  
In addition, during the analysis, it was discovered that the recycling module intent had the highest level of confusion with the no-match class, suggesting that probabilities associated with utterances in this class were also below the threshold. The overall analysis of the confusion matrix indicates that the intent classification model exhibits commendable performance, characterized by a limited number of false positives and false negatives. Nevertheless, the analysis results also indicate certain aspects that necessitate improvement.