## 4 Data Annotations
### Detection of Response Inconsistencies  
In [29], to identify inconsistencies (see Section 3.2), we manually created a set of 15 unique canonical questions along with their corresponding paraphrases. We made sure that the paraphrases retained the same semantic meaning and intent. Table 3 provides several examples of the questions and the generated paraphrases. In total, we collected 109 questions, resulting in an average of 6.3 paraphrases per canonical question.  
Then, we used four generative pre-trained chatbots to obtain their responses to the set of canonical questions and paraphrases. We selected: (a) DialoGPT-large [43] accessible via HuggingFace ([[https://huggingface.co/microsoft/DialoGPT-large](https://huggingface.co/microsoft/DialoGPT-large)]([https://huggingface.co/microsoft/DialoGPT-large](https://huggingface.co/microsoft/DialoGPT-large)) (accessed on 14 July 2023)), (b) BlenderBot v2.0 (400M), (c) BlenderBot v2.0 (2.7B) [44; 45; 46], and (d) Seeker [47] without using its retrieval search module, the three of them available in ParAI ([[https://parl.ai/projects/blenderbot2/](https://parl.ai/projects/blenderbot2/)]([https://parl.ai/projects/blenderbot2/](https://parl.ai/projects/blenderbot2/)) (accessed on 14 July 2023), [[https://parl.ai/projects/recipes/](https://parl.ai/projects/recipes/)]([https://parl.ai/projects/recipes/](https://parl.ai/projects/recipes/)) (accessed on 14 July 2023), [[https://parl.ai/projects/seeker/](https://parl.ai/projects/seeker/)]([https://parl.ai/projects/seeker/](https://parl.ai/projects/seeker/)) (accessed on 14 July 2023)).  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & \multicolumn{3}{c}{**CoBot Topics List**} \\ \hline
1. books & 2. general & 3. interactive & 4. inappropriate \\
5. movies & 6. music & 7. phatic & 8. politics \\
9. science & 10. sports & 11. other & \\ \hline \hline \end{tabular}
\end{table}
Table 2: List of labels predicted by the CoBot topic classifier.  
Following that, four expert annotators reviewed each chatbot's answers for the canonical questions and paraphrases. The annotators' task was to count the number of distinct semantic responses generated by each chatbot. Ideally, each chatbot would produce a single coherent response independently of the variability of the paraphrases. However, our findings revealed an average of four distinct answers per chatbot.  
Then, to test the performance and robustness of our proposed algorithm with more recent LLMs and more challenging paraphrases, we decided to use GPT-4 ([[https://learn.microsoft.com/en-us/azure/cognitive-services/openai/](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/)]([https://learn.microsoft.com/en-us/azure/cognitive-services/openai/](https://learn.microsoft.com/en-us/azure/cognitive-services/openai/)) (accessed on 14 July 2023)) [3]. In this case, we asked GPT-4 to automatically generate a set of 15 original questions, and 7 paraphrases each (a total of 120 unique canonical questions), that a human could ask another one when they interact for the first time and want to get to know each other (e.g., when dating). These sentences and paraphrases were manually inspected and adapted if needed. Examples of the new generated canonical questions and paraphrases are shown in Table 4.  
Finally, four State-of-the-Art pre-trained large language models were used to evaluate this new set of data. The new models tested were ChatGLM (6B) [48], BlenderBot3 (3B) [49], and GPT-4. Then, the chatbot responses were annotated in the same way as in the previous case, but this time only three experts participated. An overall improvement has been achieved with the new dialogue generation systems of up to 3% and 6% compared to the original results. These results are further discussed in Section 5.2, where the average number of different responses obtained for each new dialogue model using two different embedding encoders is shown.