Title: LLM Prompting for Localization: English or Native Language in Multilingual Text Understanding
Primary area: datasets and benchmarks
Keywords: LLM Applications, Multilingual Text Understanding, Localization

Abstract: Large Language Models (LLMs) are widely used in many text understanding and related applications. However, it has been documented that not all languages are equally represented in the training data of these models. This raises important questions about the models’ performance on given tasks when inputs and prompts are provided in various languages, as well as how prompt localization should be approached to maximize task performance across languages. In this study, we focused on text data of meeting transcripts, which often contain disfluencies, transcription errors, and non-standard language use, making understanding more challenging, especially in multilingual settings. To investigate whether English prompts could effectively achieve different tasks on conversational transcripts across languages without loss of performance, we translated and open-sourced 200 real-world meeting transcripts from the Topic-Conversation Relevance (TCR) dataset into 15 non-English languages. We conducted three types of transcript-related tasks and compared the outputs of English and native language prompts using both LLM-based automatic evaluation and human judgment. This provided a robust assessment of the output quality in multilingual scenarios. Our results showed that English prompts could achieve performance comparable to native-language prompts for most languages and tasks. This suggests that instructions in English may be sufficient for scalable LLM-based localization of conversational transcripts, reducing the need for extensive language-specific adaptation.