# A Survey on Multi-User Conversational Interfaces
## 4 Capturing Multi-User Conversational Interfaces
### Evaluation Methods
In their Wizard-of-Oz study, a conversational recommender system interacts with groups using two dialogue strategies. The evaluation measures task effectiveness and efficiency, user experience, and group dynamics, revealing that while the use of proactivity accelerates decision-making and reduces the need for chatbot moderation, it may also introduce a bias towards dominant individuals, potentially impacting overall group performance. The virtual agent developed by [PERSON] et al. [48] is designed to teach metacognitive and individual- and community-level skills. For this, they conducted a pilot study in which five participants tested group interactions in English, evaluating visual signal processing and information load. A subsequent evaluation with 41 participants only considered single-user interactions. [PERSON] et al. [54] present a study on an attentive listening system generating multi-modal listening behaviours. The first study measured user perception of the robot's social presence using the Social Presence Questionnaire and assessed the focus of attention through gaze time, involving 12 participants in single-user interactions with two robots. The second study had 21 third-party observers rate videos from the previous experiment. Furthermore, [PERSON] et al. [65] investigate smart speakers engaging with users at home, conducting three studies with voice-based interfaces used by families between 3-5 members in a long-term evaluation. The research on conversational AI by [PERSON] et al. [16] assesses a chit-chat system that provides different levels of sympathetic responses. Their evaluation had participants watch a video, interact with the system for a few minutes, and subsequently fill in a questionnaire on user-centred metrics. This was repeated with a second system. They recruited 20 participants, each interacting with two agent behaviours. [PERSON] et al. [56] analyse the effects of task-related robot discrimination. For this, they gave pairs of participants a book-sorting challenge. The users had to interact with a robot assistant to complete the task and fill in self-reported questionnaires on the attribution of blame and credit of the task outcome. Moreover, [PERSON] et al. [35] use a humanoid robot acting as a social mediator using an estimator for the social state of the group in their evaluation. The first study involved single-user dialogues in which 20 participants identified system feedback messages and completed the Godspeed questionnaire. The second study introduced two-person dialogues, with the robot facilitating scenario-based conversations in four different scenarios, pairing an expert user with a real participant. System feedback was rated via a questionnaire, followed by an additional Godspeed questionnaire. Lastly, [PERSON] et al. [39] investigate multi-user empathetic dialogue generation. The evaluation was carried out through both performance-based and user assessments. The automatic metrics evaluated how model-generated responses matched human reference responses, which was subsequently extended by a human annotation of sample dialogues using subjective metrics.