## 2 Background  
The growing demand for context-aware, domain-specific AI systems has highlighted the limitations of general-purpose LLMs in low-resource language settings and sensitive application areas [5]. While many LLMs perform well in English and high-resource languages, their effectiveness often deteriorates when applied to linguistically complex or under-represented languages such as Greek. This performance gap becomes particularly problematic in high-stakes domains like migrant support, where communication must be not only accurate but also empathetic and culturally appropriate.  
The SALLY research initiative addresses this gap by developing an intelligent chatbot designed to assist Third Country Nationals (TCNs) with accessing information and navigating bureaucratic procedures in Greece. To meet this goal, SALLY integrates generative AI with structured data, domain knowledge, and emotional sensitivity, all operating in Greek. However, the success of such a system hinges on the ability to rigorously evaluate LLM performance under real-world conditions and linguistic constraints. More specifically, deploying generative chatbots in sensitive domains entails several practical and societal risks. These systems can produce inaccurate, misleading, or overly general responses, which may be problematic in high-stakes domains, such as legal, healthcare support or minority groups, such as migrants [6; 7]. Furthermore, the data used to train such models often contain embedded social biases, potentially leading to culturally insensitive or discriminatory outputs [8]. To address these concerns, SALLY has been designed as a domain-driven model. At its core, SALLY combines traditional information retrieval with generative AI through Retrieval-Augmented Generation (RAG), pulling data from the Greek Ministry of Migration and Asylum's official website (migration.gov.gr, accessed on 21 May 2025) to craft informed replies grounded in reliable sources, while the involvement of professionals and experts working directly with migrants allows for more controlled and reliable information delivery.  
A defining feature of SALLY is its focus on Greek-language support, a necessity driven by Greece's role as a primary entry point for TCNs. While some migrants rely on widely spoken languages like English, many require assistance in Greek to navigate administrative processes, access services, and engage with local communities, key steps toward integration. In addition, sociolinguistic studies in migratory contexts have highlighted the complex, multilingual realities that migrants navigate, often relying on a mix of native, host-country, and intermediary languages depending on the setting and purpose of communication [9]. In designing SALLY, these considerations informed the emphasis on clarity, tone, and contextually appropriate host language use, aiming to reduce communicative barriers. To address this, we evaluated six LLMs to assess their ability to process and generate Greek content effectively, providing insights into their suitability for SALLY's goals.  
In the following, we situate our work within recent efforts in multilingual LLM benchmarking, domain-specific chatbot development, and LLM-based evaluation frameworks. These studies inform our methodological choices and underscore the novelty of our contribution to the field.