## 3 Development and Results of the Multi-Party Dialogue Management
### Integration of Multi-Party Person Detection
In the left camera view, three individuals are detected; two are marked in blue, indicating that their bounding-box areas exceed the threshold and are thus considered users of the system. The third individual is marked in red, signifying a smaller area and classification as a non-user. The middle camera accurately detects both participants, and the right camera view further validates the detections, with one participant marked in red due to a smaller bounding box area, indicating background status. This feature plays a crucial role in adapting interactions to fit the social context, allowing a distinction between one-on-one and group interactions, thereby influencing the design of the conversation. The robot's person detection relies primarily on its egocentric camera for this task, which is further supported and confirmed by exocentric cameras, enabling it to adapt its conversational framework in response to the dynamics of the group. The system continuously transmits the number of detected users to our front end. This information is relayed to both the NLU and the LLM whenever a complete user utterance is detected. This concurrent data acquisition is crucial as it allows the back ends to adapt responses to the evolving dynamics of the interaction. For example, the social robot can seamlessly alternate between responses tailored for individuals and those designed for groups if another person enters the interaction space during a session. This fluidity in response is essential, as it mirrors the natural ability of humans to adapt their communication based on social context, ensuring that the robot can similarly maintain a natural flow of conversation and provide a personalized and engaging user experience. In the sample conversation provided below, the robot switches to MPI mode after recognizing multiple users. The dialogue example highlights that the concept of group-sensitive conversation design is applied not only in German but also in English. In contrast to the frequent use of personal pronouns (such as 'ihr'/'euch') in the German dialogue, the English conversation design refers to the group with, e.g., 'guys'. While in German, the group is addressed directly using personal pronouns, in English, there is a noticeable distancing effect when terms like 'children' are used. Only the formulation 'guys' again refers to the address of the group (Figure 5). Figure 4: Detection of individuals at the city hall through multiple cameras evaluates engagement by examining the area of bounding boxes. Blue boxes represent engaged individuals, while red boxes signify either non-users or background objects. Exocentric camera outputs are shown in the left and right images, and the center video is from [PERSON]’s egocentric camera. Participant faces are anonymized to safeguard their privacy. Figure 5: This is the continuation of the dialogue from Section 2.1, where a second user joins the interaction space, prompting the social robot to switch to MPI mode. The second users intents are shown in purple. Icons beside the robot indicate MPI mode and a knowledge database response. The conversation will be continued in Section 3.3. When our system was evaluated within the city administration using actual users, who naturally exhibited demographic and physical diversity, it was found that the (group) person detection system functions reliably across multiple scenarios. To optimize performance, we adjusted the interaction radius and fine-tuned the person detection thresholds based on the bounding box areas. These parameters are adaptable for each deployment environment. Using the egocentric camera (i.e., the middle one) and using the side cameras for confirmation, we improved detection accuracy. The system reliably distinguished users from non-users by analyzing the size of the detected bounding boxes and validating detections across multiple camera views. This multi-perspective analysis significantly reduced errors caused by background individuals being mistakenly counted as users.