Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper introduces a benchmark containing 250 grade-school math problems translated from English to ten languages and also provides a deep analysis to explore multilingual reasoning abilities of two SotA large language models GPT-3 and LLM-540B in terms of both chain-of-thought and commonsense reasoning. The proposed benchmark was translated by native speakers of the target language with at least two years of professional experience. The experiments are well-designed, and the results are enough to support the claims. They find that on the MGSM benchmark, LLM-540B shows superior multilingual reasoning ability compared to GPT-3 on all languages with different settings. Similarly, for the commonsense reasoning benchmark, LLM-540B also outperforms the SotA models: RoBERTa Large and XLM-R Large on two other benchmarks: XCOPA and XL-WiC, respectively.
strength_and_weaknesses: **Strengths**
* The multilingual benchmark is translated by professional translators and verified by an additional one for a random subset.
* Experiments are well-designed with 4 prompting methods and 3 types of input exemplars.
* Intensive analysis provides new insights for multilingual reasoning abilities of large language models.
* The paper is also well-written.

**Weaknesses**
* The number of examples is not significant, with only 250 examples, compared to GMS8K with 8.5K examples, so the results may not be generalized if more examples are translated and added to the benchmark.
* The verification process is less discussed, and not convincing to guarantee the quality of the benchmark.
* The approach of translating a problem to English does not test the multilingual reasoning abilities of LLMs.
clarity,_quality,_novelty_and_reproducibility: **Clarification**
* In the "Manual translation process" passage, who are the popular MT providers?

**Questions to the authors**
* Did you check the math background of translators?
* How do you measure the quality of translated examples compared to English examples?
* How does the pre-training (i.e., datasets, model design) of LLMs affect their multilingual abilities?
summary_of_the_review: Although the findings are interesting, the contributions are not significant enough for the multilingual research since the benchmark is small and the verification process is not well described (or implemented?), which may flip the current conclusions/claims if more multilingual examples with different types of problems are added. The big plus of this paper is the intensive experiments and detailed analysis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper considers chain-of-thought (CoT) in multilingual setup and shows that despite a highly imbalanced pretraining dataset, LLM still learns strong CoT capability in non-English. It extends an existing dataset to multiple languages. It also compares different ways to construct exemplar and CoT.

strength_and_weaknesses: Strength
It introduces a new dataset and presents extensive experiments.

Weakness
- No variance is presented in the few-shot experiment.

clarity,_quality,_novelty_and_reproducibility: This paper is well written and the experiment is well designed. It considers CoT reasoning in a multilingual setup and introduces a new dataset. However, portions of the experiment is impossible to replicate due to the proprietary nature of LLM.

- Is the dataset and CoT explanation publicly available?
- Caption of table 6 has [CITE]?

summary_of_the_review: This paper considers CoT in multilingual setup and creates a new dataset, despite minor issues of the experiments.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduced a multilingual arithmetic reasoning task with a corpus named MGSM, and used the few-shot chain-of-thought (Few-Shot CoT) prompting framework to evaluate the large language models’ few-shot learning capability.  
Authors empirically found that the GPT-3 and LLM could provide reasonable performance on their constructed test set, and also provided results on two other multilingual reasoning tasks.

strength_and_weaknesses: Strength:
1.  The paper shows interesting experimental results on the multilingual arithmetic reasoning task MGSM. Large-scale language models can perform well across languages, and even on underrepresented languages like Swahili.
2. The paper provides extensive and convincing experiment results on other two multilingual tasks (common sense reasoning and word in-context semantic judgment).

Weaknesses:
1. The constructed test sample size for the benchmark is a bit small (250 samples).
2. Some experiments and additional clarification can be considered and added (see Questions).

Questions:
1. In Section 4, it would be better if the authors add the experimental results of adopting Zero-Shot CoT (Kojima, Takeshi, et al. 2022) [1], compared with the Few-Shot CoT model.
2. One major question I have in mind is, is the current sample size of MGSM sufficient as a benchmark corpus? It would be better if the test sample size can be larger than the current 250, as the original GSM8K contains 1,000 test samples.
3. It would be better if authors can add some insight or feature-based analysis on the emergent ability of cross-lingual reasoning of the tested language models GPT-3 and LLM.
4. It is a bit surprising that the language models (GPT-3 and LLM) that are not specially pre-trained on multilingual data work well on cross-lingual reasoning with few-shot settings, is it possible to compare them with some multilingual backbones?
5. I was also wondering whether the performance on MGSM (shown in Table 3) will be affected by adding some text perturbation (here the perturbation should not affect the numeral text), or adding some spans with code-switching.
6. In Table 7, why are the number of few-shot exemplars in different language settings not the same? For instance, only 1 exemplar is provided for ru, th, te, and bn, which is smaller than that of en, de, fr, and es (6 exemplars).


References:  
[1] Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. "Large Language Models are Zero-Shot Reasoners." arXiv preprint arXiv:2205.11916 (2022).

clarity,_quality,_novelty_and_reproducibility: Clarity: Very clear, the paper is well-written.  
Quality: Good.  
Novelty: This work is a multilingual version extension of the few-shot chain-of-thought reasoner. They empirically evaluated the multilingual reasoning capability of two large scale language models.
summary_of_the_review: This work is interesting and well-motivated, the technical contribution can be extended by tackling the points of questions.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper extends the prevalent GSM8K benchmark to a multilignual verison covering ten languages, and accordingly verify a recently proposed technique Chain-of-though upon it, resulting a best performance of 55 with the most competent LLM.

The authors also verify such capability on established benchmark XCOPA, achieving a new state-of-the-art.

The paper is well presented and very clear to understand.
strength_and_weaknesses: Strength:

1. The paper is well presented and easy to understand.

2. a multilingual version of the popular benchmark GSM8K is proposed and released, which might be of interest to the community.

3. new state-of-the-art on XCOPA and XLWiC is set.

Weaknesses:

1. The paper does not provide much novelty, insights and inspiration, the technique design simply follows existing works of CoT.

2. The empirical investigation is not sufficient for a solid benchmarking of the new datasets. Specifically, I believe several recently-proposed methods like ZeroShotCoT[1] and SelfConsistencyCoT[2] are also necessary and not difficult to implement.

3. The extension of the multilingual version of an existing benchmark seems providing only limited contribution for a ICLR paper to me. And the empirical verification of CoT on more datasets are also not sufficient for another new paper.

[1] Large Language Models are Zero-Shot Reasoners

[2] Self-Consistency Improves Chain of Thought Reasoning in Language Models
clarity,_quality,_novelty_and_reproducibility: Clarity:
The claim is clear and backed by experiments.

Quality:
The paper is well written. The quality is ok but not extremely solid. Experimental investigations do not cover necessary scope.

Novelty:
The novelty is limited. The technique is merely another verification of existing published paper CoT with only minimum adaptation, where a naive translation of examples provides the best performance.

Reproducibility:
The dataset is released and the code is available. Although I assume most people do not have access to GPT3 or the refered 540B LLM, which are the only keys to obtain such SoTA results.
summary_of_the_review: The paper is well presented, an multilingual extension of GSM8K benchmark is proposed, CoT is verified accordingly as well as more datasets like XCOPA, which sets a new SoTA.

But the technique is almost identical to CoT, and the results can be well expected, leaving it only a very incremental contribution. Besides, such empirical verification is also not sufficient as important baselines are missing.

Therefore, I vote for borderline reject, but I'm not absolutely sure and are open for discussion or rebuttal.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This is a very straightforward paper, with impressive and surprising - although to me sad (more on this later) - results. They will certainly motivate future work, and encourage some of the current directions.

The authors analyze the multi-lingual capabilities of language models who are trained primarily on English data, with a little of other languages as well. They combine this with recent work of chain-of-thought (CoT) prompting, showing that those the combination of large LLM + CoT provide a very decent results on challenging datasets, in a multi-lingual setting.
strength_and_weaknesses: STRENGTH
1. Translating a math-problem dataset (GSM8K) from English to 10 other diverse languages, using professional translators; and releasing the full dataset
2. Benchmarking two LLMs with CoT on that and two other multi-lingual datsets; showing that they provide competitive and sometimes state-of-the-art results compared to a fully supervised setting
3. Showing that machine-translating the CoT into English is at least as good and often better than prompting with CoT in the target language directly

WEAKNESS
1. A point which is never addressed is why translated CoT seems to perform always better than human translated CoT. Is this because some of the training data of the LLM might be translatese? Or is the difference too small to be significant? If so, it would be interesting to see the results with a different set of prompts or with a different sampling
2. Not really a weakness, but this result will not necessarily encourage more research into low-ressource languages. While the large LLM in this paper is anonymized, I picked a random other language model from all those that also have 540B (PALM in this case). PALM contained 24B French tokens, and this paper reports 46.2% when natively prompted and 55.2% when translated. Swahili only had 40M tokens for PALM, but has a comparable performance in this paper when translated (51.2%, compared to 35.2% for native Swahili prompt). Why developping a Swahili (or French?) LLM when I can just use the largest EN one and machine-translate? At the same time, 40M tokens will have much less variety than 578B tokens used for English and some idiosyncrasy of Swahili might not be captured by translation only.
Another direction which will be encouraged by this paper is scaling: Swahili had an almost 4x jump between using a 62B model (there are a few of those available these days) and a 540B (not so many available of that size).

Note that Weakness (2) is not really an issue with the paper... just with the consequences that the reported facts might produce.


clarity,_quality,_novelty_and_reproducibility: Clarity: very clear, this is a straightforward analysis but delivered in a very clear way

Quality: very good, the experiments are thoroughly done

Novelty: high. I wonder why nobody else did this until now..... </sarcasm>

Reproducibility: as soon as the 540B model will be release or an API provided this can be reproduced very easily
summary_of_the_review: Straightforward idea, very well executed. High impact
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a Multilingual Grade School Math (MGSM) which can be used to evaluate the reasoning abilities of large language models by predicting the chain of thought. The authors compare the results of different prompting strategies on two large-scale language models, and further conduct ablation studies to analyze the effect of language frequency, model scale, exemplar amount and exemplar type.
strength_and_weaknesses: Strength:
1.	The MGSM is the first multilingual dataset that can be used to evaluate the chain of thought in multilingual setting. 
2.	The paper is well-writing and easy to read.
3.	The experiments are comprehensive and well organized.
Weakness:
1.	The contribution of multilingual chain-of-thought is incremental compared to the villa chain-of-though.
2.	It is interesting to see the large gap between Translate-EN, EN-CoT and Native-CoT in MGSM. While the gaps in other benchmarks are not too much. Is it because MGSM benchmark is originated from translation?

clarity,_quality,_novelty_and_reproducibility: This paper is well-written and easy to follow. The dataset is novel while the proposed multilingual prompting strategy is incremental to villa chain-of-thought. Considering the heavy resource requirement of experiment, it might be difficult for reproduce given limited resources. 
summary_of_the_review: The contribution of the new benchmark is clear and the experiments are sufficient, while the prompting strategy of multilingual chain-of-thought is incremental.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper introduces the first multilingual benchmark for evaluating the task of arithmetic reasoning. It is built from the translation by experienced professionals of part of an English dataset. The objective is to evaluate the multilingual abilities of very large language models trained on monolingual English corpora incorporating some elements of other languages as well. The authors provide an in-depth analysis of two large language models for reasoning tasks (multilingual arithmetic and common-sense reasoning). It is found that by using recent prompting strategies, these large models perform on par or better than models fine-tuned on a target language and outperform SOTA models.

The main contributions of the paper are (i) the introduction of a multilingual dataset for evaluating language models reasoning when the existing benchmarks are in English, (ii) an in depth analysis of the reasoning abilities of language models. Their conclusion is rather surprising: when they reach a critical size, large English monolingual models do perform extremely well on multilingual reasoning, sometimes better than language specific trained models. 
All the reviewers agree that this contribution could greatly influence future work. The paper has also some limitations: the size of the translated corpus is small and the technical contribution itself is not so original as the experiments follow the recent prompting practice. However, the added value of the work lies in the conclusions made possible by the corpus and demonstrated in the analysis. During the discussion, the reviewers highlighted the potential influence of this work on future practices: this could modify the nature of research on multilingual models.

justification_for_why_not_higher_score: Nice paper but with some limitations.
justification_for_why_not_lower_score: The influential nature of the conclusions makes the paper a valuable contribution for the community.
Decision: Accept: poster