Title: Language models are multilingual chain-of-thought reasoners
Benchmark: MGSM
Summary: We introduce the Multilingual Grade School Math (MGSM) benchmark, by manually translating 250 grade-school math problems from the GSM8K dataset (Cobbe et al., 2021) into ten typologically diverse languages. We find that the ability to solve MGSM problems via chain-of-thought prompting emerges with increasing model scale, and that models have strikingly strong multilingual reasoning abilities, even in underrepresented languages such as Bengali and Swahili. 
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Multilingual chain-of-thought reasoning. 
Phenomenon defined by authors: Yes
Phenomenon definition: We evaluate the reasoning abilities of large language models in multilingual settings. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Contested
Task definition: Take a math problem and provide a numerical answer.
Task item definition: Example problem, numerical answer. 
Task source: Modified from another benchmark (e.g. translation into another language)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Dataset size: 2500
Dataset metadata available: Yes
Dataset metadata detail: Language
Sampling method: Specific criteria (items were taken from a larger set based on specified rules)
Split reviewed: Test
Response format: Short free response (e.g. single word or number)
Metric definition: Exact Match (accuracy, F1, precision, recall)
Metric subscores: Yes
Metric subscores detail: Language
Metric access: Outputs alone
Few-shot evaluation: No
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: Yes
Results comparison explanation: Yes
Results realism: The benchmark is itself realistic
Human baseline: Yes
Authors discuss validity: Yes
Author validity detail: To verify the quality of the human translations, the vendor sent a random subset of translations to an additional translator to verify the quality, and checked for n-gram overlap with popular MT providers to ensure that no machine translation toolkit has been used. We employ the translation results as gold standard translations.
Taxonomy: Reasoning / Mathematical
Alternate taxonomy: Multilinguality
New taxonomy: General Capability / Mathematical Reasoning