Helping Math Find Their X: Assessing the Impact of Reasoning Methods, Biases and Translation Systems in Multilingual LLMs
Authors: Marek Suppa, Ritvik Shrivastava


Reviewer paper summaries: This paper conducts a comprehensive analysis of the effects of reasoning strategies and translation models on 26 high, medium, low and extremely-low resource languages. Specifically, it provides the following findings:
- LLMs show strong logical reasoning in English but perform less effectively in other languages, particularly in low-resource languages.
- Translation and prompting methods significantly affect performance, with some methods proving more effective than others.
- A new reasoning strategy, XL-TEN, was proposed and found to improve multilingual reasoning in low-resource settings.
Summary of strengths: Several novel contributions are noted:

1. The paper emphasizes multilingual settings covering both high- and low-resource languages. It draws attention to the understudied worldwide languages in the domain of LLM reasoning, including African languages, which breaks away from the Euro-centric setting.
2. The proposal of XL-TEN as a new prompting strategy in multilingual reasoning paradigms shed lights for future research about focusing on exploiting the English-centred bias in current LLMs, in order to boost performance in other languages.
3. The work reports the first direct exploration into the impact and quality of translation models and their effects on multilingual reasoning compared to step-by-step reasoning and prompting methods.
Summary of weaknesses: The novelties and potential significances of the study are limited:
1. **Scope**: While the topic of evaluating LLMs in multilingual settings is important and worth exploring, the domain of the current study is restrained to mathematical reasoning;

2. **Datasets**: The datasets that are adopted (MGSM, AfriMGSM, MSVAMP) are a mere expansion of the datasets used in previous evaluation works (e.g., Liu et al. (2024) already evaluated MGSM in its multidomain reasoning paradigm including math). Although it should be appreciated that datasets in African languages and other extremely low-resource languages are included and this is indeed a valid contribution, it is still largely incremental work lacking potential impacts in the field.

3. **Method**: The study also lacks novelty in its evaluation method. The study mainly used six different prompting strategies to test the math reasoning performance of LLMs (Section 5.1). Among them, the first five strategies come from Liu et al. (2024) while only the last one (XL-TEN) is newly proposed.

- 3.1. The first critique here is that the only modification of XL-TEN from the previous XLT is a word change in the instructions in system messages, hence the method novelty is limited. 
- 3.2. Furthermore, the authors didn't make a strong point as to the superiority of the new method. While the motivation underlying XL-TEN sounds valid and reasonable in theory (Section 5.1), from the empirical observation of the final results in Table 4, this novel prompting strategy didn't bring a robust or substantial improvement in the three datasets across various models. When comparing all the six strategies in Table 4, XL-TEN renders the best results in only three cases out of the combination of [3 datasets * 7 models]. This makes the claims made in Section 5.3 look narrow and exaggerating.

4. **Analysis**: It looks like the objective of the paper is to assess the impact of translation, reasoning strategies, and language resource levels on the mathematical reasoning abilities of multilingual LLMs. However, the authors didn't analyze the impact of the last one (language resource levels) in the result section later. In other words, Table 4 only lists the results in the three datasets rather than based on specific languages or language resource levels. This makes the prior categorization of and emphasis on the high-, low-, and extremely low-resource languages earlier look less coherent with the entire paper.

Altogether, these aspects of lacked analysis and novelty weaken the contributions of the current paper.




References:
Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan 641 Luu, and Lidong Bing. 2024. Is translation all you 642 need? a study on solving multilingual tasks with large 643 language models. arXiv preprint arXiv:2403.10258. https://arxiv.org/pdf/2403.10258
Comments, suggestions and typos: **Content**:
Apart from tackling the weaknesses mentioned above, the current paper could be improved in the following aspects:
1. A deeper and more fine-grained exploration of possible prompting strategies (e.g., not necessarily zero-shot).
2. A more robust analysis of the significance of the novel method that was proposed.
3. More fine-grained analyses should be supplemented to support the paper's claims and objective framing.
4. Since the paper only focused on mathematical reasoning, the paper could benefit from addressing the unique challenges of this particular domain more. This will highlight the significance of the study and distinguish the paper from existing works. A section of case study/qualitative analysis in the Appendix would be much appreciated for the readers to understand the nuances in the methods and task questions.
5. It is suggested that specific settings and hyperparameter details be provided in the Appendix for reproducibility.

**Grammar and spellings:** The authors can work more on checking grammatical and format inconsistencies via further proofreading. For example:
1. In line 120, it is preferred that the cited authors are without parentheses because they function as the subject of the sentence, i.e., Liu et al. (2024).
2. The wordings or expressions of some sentences are unclear if not confusing and lack contextualization. For example, in line 77: *"Our work also serves as the first direct exploration into the intensity of translation impact, model and their quality on multi-lingual reasoning as compared to step-by-step reasoning and prompting methods."*
Overall assessment scores: 2