## 5 Results
### Detection of Response Inconsistencies
In our experiments, the 762M-parameter DialoGPT-large ([[https://github.com/microsoft/DialoGPT](https://github.com/microsoft/DialoGPT)]([https://github.com/microsoft/DialoGPT](https://github.com/microsoft/DialoGPT)) (accessed on 14 July 2023)) model and the 6.2B-parameter ChatGLM ([[https://chatglm.cn/blog](https://chatglm.cn/blog)]([https://chatglm.cn/blog](https://chatglm.cn/blog)) (accessed on 14 July 2023)) conversational model received the highest number of variations according to the annotators. The result of ChatGLM is particularly relevant since it was originally conceived as a bilingual Chinese-English dialogue model. Many of the responses obtained with this model had to be post-processed so that they did not contain Chinese or other special characters. While maintaining lexical-semantic coherence, these particularities could be the reason for the detriment in its consistency. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Chatbot** & **Avg. No. Responses** & **Av. Predicted** & **MSE** \\ \hline BlenderBot2 (400M) & 4.0 \(\pm\) 1.6 & 4.6 \(\pm\) 2.2 & 2.4 \\ BlenderBot2 (2.7B) & 3.7 \(\pm\) 1.6 & 3.3 \(\pm\) 1.9 & 2.8 \\ DialoGPT-large & 4.3 \(\pm\) 2.1 & 3.1 \(\pm\) 2.0 & 5.4 \\ Seeker & 4.0 \(\pm\) 1.7 & 4.0 \(\pm\) 1.7 & 3.1 \\ \hline Overall & 4.0 \(\pm\) 1.7 & 3.8 \(\pm\) 2.0 & 3.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 9: Original results of the different responses generated by previous chatbots and the predicted number of inconsistencies. Slimming through Table 10, the responses obtained with the GPT-4 model are those with the lowest mean square error between human annotations and predictions with an overall minimum value of 1.6. As expected, this model turned out to be the most consistent model in its responses, being the largest of all the proposed models in terms of parameters. For a better understanding of these results, the expanded questions together with the answers obtained with this model are available in Appendix A. In view of the positive results obtained with the all-mpnet-base-v2 sentence embeddings encoder, and in order to test the robustness and dependency of the prediction algorithm when using a different vector embedding model, we performed the same approach on the same conversational chatbots and their responses, but this time using the vector representations extracted with the State-of-the-Art OpenAI text-embedding-ada-002 ([[https://openai.com/blog/new-and-improved-embedding-model](https://openai.com/blog/new-and-improved-embedding-model)]([https://openai.com/blog/new-and-improved-embedding-model](https://openai.com/blog/new-and-improved-embedding-model)) (accessed on 14 July 2023)) encoder model. Table 11 provides the prediction results. A quick glance shows a slight improvement in the relationship between the annotations and the predictions on the generated responses. However, the improvements obtained with this new model are not significant. The only model whose mean square error is slightly reduced is GPT-4, by approximately 10%. As expected, this model turns out to be more consistent than the other proposed models. Next, for a better understanding of the results, Figure 7 shows the vector representations obtained when using the OpenAI encoder for randomly selected questions and answers from each conversational model. The figure includes error ellipses, with the same color, that group together the paraphrases and respective responses for the same type of question. This error ellipse allows us to represent an isocontour of the embedding distribution. In our case, the confidence interval displayed defines the region that would contain 99% (\(\sigma\) = 3.0) of all samples that could be drawn from the underlying Gaussian distribution. Considering the small number of samples (questions and answers generated), it is essential to use confidence ellipses to better see the shape of the clusters that these vectors make up. It is interesting to note some details in these visualizations.