## 5 Results  
In this section, we present the results of our experiments regarding specific modules and the overall performance of the chatbot across different settings/modules. Furthermore, we analyze the effects of various settings and the impact of our proposed solutions on the major flaws identified during the analysis of BB3. Based on our comprehensive analysis, we have significantly improved the performance of this chatbot. Ineffective models, as identified by our analysis, have been eliminated. We have successfully increased the chatbot's speed and efficiency through various enhancements, such as the integration of APIHUB (described in Section 4.5) and the simplification of certain models. Our modifications were motivated by the famous [PERSON]'s razor principle--i.e., we simplified the chatbot and obtained better performance--our modified chatbot outperforms the original BlenderBot 3 by almost 9% in terms of overall quality and is more than three times faster. We evaluated the performance concerning search and memory decision activities through standard metrics such as accuracy and F1 score. Concerning query generation, we provide SacreBLEU scores along with two manually acquired scores: Accuracy and Weak Accuracy, as explained in Table 3, which displays the comparison of the query generation task, where we compare our two best checkpoints with the original BlenderBot 3 model. In this case, we were unable to reach a definitive conclusion on whether our new models are generally better or worse than BB3. However, considering that latency is a significant bottleneck for the chatbot, we decided to implement the FLAN-T5-base-based model in our revised 'BlenderBot' system. The performance of our classification models is shown in Table 4. The evaluation datasets, referred to as eval datasets, are obtained by merging the evaluation datasets from all the data used for each specific task by BB3. On the other hand, the test datasets are our newly designed datasets (described in Section 4.1). Figures 3-6 show confusion matrices of BB3 and the fine-tuned classification models for Memory decision and Search decision. The impact of specific pipeline settings (including our enhancements) on major flaws such as repetitions and contradictions is presented in Table 2. The results were obtained through the manual labeling of the chatbot's output utterances. We consider several settings to test the impact of long-term memory, dominant entity, external knowledge, as well as our our new proposed setting, which has superior performance in three out of the four categories. Note that 'Vanilla BB3' refers to a setting where all side modules (external knowledge, memory, and entity) are discarded, and the 3B model generates a response solely based on the user's utterance and context. Table 1 provides an overview of the overall performance of the chatbot under various settings, as evaluated by [PERSON] on a scale from 1 to 5. The reported scores include Overall Feel and Engagingness (higher score is better), as well as Repetition and Hallucination (lower score is better). We hypothesize that for some categories, especially the Overall Feel, latency could make a certain impact on the resulting scores. The results presented \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Model** & **Model Size** & **Accuracy** & **Weak Accuracy** & **SacreBLEU (EVAL)** & **Speed Up** \\ \hline BB3 & 3B & 73.6 & 96.2 & 12.1 & 1x \\ FLAN-Alpaca-large & 780M & 56.6 & 98.1 & 13.6 & 2.2x \\ FLAN-T5-base & 250M & 54.7 & 98.1 & 12.2 & 4.3x \\ \hline \hline \end{tabular}
\end{table} 
Table 3: We evaluated the performance of the BB3 model and our fine-tuned models (FLAN-T5 and FLAN-Alpaca) in query generation using a test dataset. We assessed the results based on three criteria: exact match, strict semantic match (in which the generated query is not an exact match but holds the same semantic meaning as the label), and relevance (where the generated query pertains to the topic but may not be as precise).