Better to Ask in English: Evaluation of Large Language Models on English, Low-resource and Cross-Lingual Settings
Authors: Krishno Dey, Prerona Tarannum, Md. Arid Hasan, Imran Razzak, Usman Naseem


Reviewer paper summaries: This paper presented a work in multilingual LLM for South Asian languages (Hindi, Urdu, Bangla). The paper proposed 5 settings of prompts in different languages and compared the performances between these settings.
The paper's main findings are:
- Prompt in English yields better performance compared to native prompt.
- GPT-4 is better than Gemini Pro, Gemini Pro is better than LLaMA 2

Despite recent advancements in large language models (LLMs), most are developed and evaluated primarily for high-resource languages such as English. In response, this paper proposes evaluating recent LLMs on low-resource languages using a zero-shot prompting method for Natural Language Inference (NLI). Specifically, the study evaluates GPT-4, Llama 2, and Gemini on Bangla, Hindi, and Urdu, three low-resource South Asian languages. The results show that GPT-4 consistently outperforms Llama 2 and Gemini across all languages and prompt settings in NLI, with all models exhibiting superior performance in English compared to the other three languages. The study highlights the need for improved LLMs and more data resources for low-resource languages.
Summary of strengths: - The paper compared the performance of 3 LLMs using prompts in English, Hindi, Urdu, and Bangla and confirmed that using English task descriptions yields better performance than prompts in native South Asian languages.
- The paper is well-written and easy to follow.

1. This paper evaluates LLMs in low-resource languages and underlines a critical gap in their current development. This focus is important for expanding the inclusivity of NLP systems, ensuring they benefit a broader range of languages and speakers, and thus promoting fair and equitable application.
2. The paper focuses on three languages, which the authors are native speakers of, and evaluates three popular LLMs: GPT-4, Llama 2, and Gemini. The proposed evaluation using cross-lingual translation for different prompting settings is interesting. It thus shows a considerable cost, time, and manual effort to complete this work.
3. The empirical results confirm and provide some meaningful findings, such as the performance gap of LLMs between English and the three low-resource languages in NLI.
Summary of weaknesses: - The problem that the paper addresses seems to be addressed in a few works (e.g. ChatGPT beyond English.. Lai et al. (2023) https://arxiv.org/pdf/2304.05613). This paper (Table 6) has shown that prompts in English work much better than prompts in Hindi and Urdu on the same XNLI dataset.
- The LLaMA paper has clearly stated that Hindi, Bangla, and Urdu data are lower than 0.01% of 2T-token training data (Table 10, https://arxiv.org/pdf/2307.09288). The paper should have at least made some changes to make the model much stronger through vocabulary extension and additional pretraining in the target South Asian languages.
- The paper focuses too much on comparing GPT4 vs LLaMA2 vs Gemini Pro and doesn't provide any new insights. There have been Hundreds of benchmarks showing that GPT4 is better than LLaMA2 and Gemini Pro. The paper should position the paper with new insights. I suggest dropping or significantly cutting sections 4.1 and 4.2 into two short paragraphs, and leaving the space to cover new experiments and insights such as:
(i) Using a native South Asian LLM
(ii) Further fine-tuning LLaMA on more South Asian data
(iii) Testing the translation of the context (premise and hypothesis) into English and letting the model perform the task in English.

1. The evaluation is restricted to the NLI task and the selected three languages. As a result, the findings may not fully capture the capabilities and limitations of the LLMs across a broader range of NLP tasks and datasets. This limitation weakens the basis for some of the claims made and reduces the scope of the study. For example, although NLI is the only task evaluated, this is not reflected in the paper title.
2. Simply evaluating existing APIs or models and reporting the results may not provide sufficient contributions for publication at a top conference. Top venues typically seek studies that offer novel methodologies, theoretical insights, or substantial improvements over existing techniques. The paper focuses on standard evaluation methods for existing models and thus may lack the originality and novelty required in scientific research.
3. Specifically, the paper does not sufficiently discuss existing work evaluating LLMs in multilingual and cross-lingual settings, making it difficult to distinguish this work from prior research. The translation-based prompting method used is not new for evaluating LLMs [1]. Additionally, the findings are expected and align with widely acknowledged facts: GPT-4 often serves as the upper bound for most LLMs, LLMs tend to perform better in English than other languages, and in-context prompting usually provides better results.

[1] Don’t Trust ChatGPT when your Question is not in English: A Study of Multilingual Abilities and Types of LLMs (Zhang et al., EMNLP 2023)
Comments, suggestions and typos: - Tables 3 and 5: Since the performances are very different, using 100-th precision seems to be overkilled and makes it harder for the reader to digest. I suggest using the 10-th precision instead.
- The LLaMA model name is "LLaMA" not "Llama", please revise all the mentions to the correct name.

The primary issue with this work is its novelty and contributions. It is suggested to explore beyond pure evaluation. For example, is there a method to mitigate the performance gap between English and low-resource languages like Bangla? 
+ Line 082: Most LLMs are transformer-based models themselves. 
+ Section 3: This section is a mixture of methods and experimental setup. The methodology should be presented independently from the specific implementation details in the experiments. 
+ References: Provide the link and access date for Google Translator, as it has different versions and is closed-source. 
+ Proofreading: Please proofread the writing before submission. For instance, there is a missing space in line 338 and wrong quotation marks in line 335.
Overall assessment scores: 2

2