## 2 Materials
### The Social Robot  
As detailed in Section 2.1, the social robot is required to fulfill several criteria for successful deployment in public areas. It should operate in various languages and recognize people within its interaction range, allowing it to distinguish between single and group interactions. Furthermore, the robot must provide scripted responses to specified intents and also produce impromptu reactions. We incorporated the social robot Furhat as the front end, interfacing with three largely autonomous, self-developed subsystems functioning as the back end (for a detailed view of the architecture, refer to Figure 3):  
1. Robot front end: This subsystem serves as the user interface, managing most necessary interaction functions, including automatic speech recognition (ASR), person detection, and text-to-speech (TTS) output. It also manages some dialogue management tasks, such as initiating and concluding conversations, and managing speaker turns. The front end is connected to a tablet with flag buttons that allow users to select the system language. In this work, we used a Furhat robot, which provides essential components such as ASR, person detection, and TTS. However, the Furhat robot's built-in person detection had limitations, particularly when individuals were not facing the robot directly or were looking in different directions, which often disrupted dialogue. To address this issue and improve the robot's ability to detect multiple users simultaneously, we integrated a YOLO model for more robust multi-party detection. 2. Intent-based back end: This core subsystem includes a NLU model and a related knowledge database. The database stores predefined subject areas (intents) set by example user queries, along with fixed, preformulated responses. These intents address specific user inquiries, such as identifying the right contact for an issue, providing directions to the nearest restrooms, engaging in small talk, and integrating APIs like a weather forecast. 3. LLM back end: In the event that queries are not aligned with the predefined parameters, this subsystem is engaged. The intent-based back end forwards the user's statement, dialogue history, user count, and language to the LLM server, which then generates an appropriate prompt to produce a free response to the current user query. For this, LLaMA 2 (Meta, USA) [38] in its 13 billion-parameter quantized version,  
Figure 2: This is a conversation between a user (red) and a social robot (orange) in a city administration setting. Each speech bubble is bilingual (German/English). Icons next to the robot indicate a knowledge database response and single user mode. The dialogue continues in Section 3.1.  
designed to run on consumer-grade graphics cards [22], was integrated and used as a fallback. The selection of LLaMA 2 was made according to the Open LLM Leaderboard [39], where the LLaMA 2 model was ranked the top in summer 2023, just prior to the execution of the field system assessments. The speech-to-text (STT) module of the front end transcribes spoken language into written text. This textual representation is then relayed to the intent-based back end. The intent-based back end coordinates the conversation and includes modules for NLU, dialogue management, and natural language generation (NLG). For NLG, the system has access to the knowledge database. The overall latency of the social robot is crucial for conducting real-time dialogues with users. The intent-based back end is equipped with a Google Dialogflow NLU model that deciphers the user's intent by evaluating the input against a response database. In parallel, the LLM back end (LLaMA 2) serves as a secondary back end, stepping in to provide alternative responses when the intent-based AI's confidence level does not meet a predefined threshold, ensuring that the social robot remains effective even when faced with out-of-knowledge queries. To determine whether the response from the intent-based back end or the LLM back end is chosen, the confidence level of the NLU model is decisive.