# A Survey on Multi-User Conversational Interfaces
## 4 Capturing Multi-User Conversational Interfaces
### Evaluation Methods  
In the final step of the paper analysis, we reviewed the methods used for the evaluation of MUCIs. Effective assessment must capture not only the linguistic capabilities and task performance of the system but also the individual, interpersonal, and social dimensions that emerge in group interactions. Similarly to the previous sections, no standardised process was identified. However, we observed two main approaches: user-centred and performance-based metrics. User-centred metrics focus on the quality of user experience and perception. These metrics are typically obtained by human evaluations, in which participants or annotators rate system outputs according to individual factors such as empathy, engagement, appropriateness, relevance, and naturalness. In multi-user contexts, additional criteria include the system's ability to address in-group dynamics, mediate between conflicts, and maintain inclusiveness. Only a few publications were also collecting this kind of feedback on the group level. Subjective evaluation often involves questionnaires or qualitative feedback. Contrarily, performance-based and automatic metrics rely on computational measures to evaluate system outputs against reference samples or predefined criteria. Common automatic metrics include BLEU, ROUGE, and METEOR, which quantify similarity between generated and reference responses, and accuracy-based metrics for tasks such as intent recognition or emotion classification. While these metrics are efficient and scalable, they often fail to extract conversational characteristics, social appropriateness, or the evolving context of multi-user dialogues. As a result, automatic metrics may not correlate well with actual user experience, especially in complex or empathetic MU scenarios. For future MUCIs, a comprehensive evaluation requires a combination of both approaches. The categorisation of evaluation metrics used in the retrieved publications is shown in Table 4. In summary, it strongly depended on the research group which evaluation methods were selected. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Publication**} & \multirow{2}{*}{**User Study**} & \multicolumn{2}{c}{**Performance-Based Metrics**} & \multicolumn{2}{c}{**Group or User-Centred Metrics**} & \multirow{2}{*}{**Data Collection**} \\ \cline{1-1} \cline{5-5}  & & & & **Metrics** & \\ \hline
[11; 21; 28] & ✓ & ✓ & user-centred & ✓ \\ \hline
[36] & ✓ & ✓ & group-centred & ✓ \\ \hline
[12; 15; 22; 34; 37; 45; 55; 56] & ✓ & ✓ & user-centred & \\ \hline
[25; 57] & ✓ & & user-centred & ✓ \\ \hline
[35; 58] & ✓ & & user-centred & \\ \hline
[16; 33; 59] & ✓ & & group-centred & \\ \hline
[3; 39; 60] & & ✓ & group-centred & ✓ \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Characteristics of publications with regard to their evaluation metrics. To provide a clearer understanding of how evaluation has been approached in literature, we offer a more detailed description of the used evaluation methods in the following. [PERSON] et al. [37] validate the applicability of a virtual reality evaluation in clinical scenarios for MU applications. Similarly, [PERSON] et al. [40] examine the accuracy of system predictions regarding users' emotional reactions and the general emphatic state. In the domain of interactive storytelling for education, [PERSON] et al. [61] explore a MU tabletop interface designed to support social interaction. A qualitative study involving four pairs of children assessed the system through positive and negative feedback, as well as questions on social interaction. In human-robot interaction, [PERSON] et al. [62] evaluate conversation tracking and user identification in two settings: an 'easy' mode with two participants and a 'hard' mode with up to six participants. The evaluation, conducted on data recordings from 28 participants, measured face detection performance, tracking ability, and participant re-identification. [PERSON] et al. [24] investigate a virtual environment for foreign language learning with multiple AI agents.