Language models are multilingual chain-of-thought reasoners
Authors: Freda Shi
Mirac Suzgun
Markus Freitag
Xuezhi Wang
Suraj Srivats
Soroush Vosoughi
Hyung Won Chung
Yi Tay
Sebastian Ruder
Denny Zhou
Dipanjan Das
Jason Wei
Venue: ICLR 2023 poster (2023)
Keywords: multilingual
reasoning
large language model

Abstract: We evaluate the reasoning abilities of large language models in multilingual settings. We introduce the Multilingual Grade School Math (MGSM) benchmark, by manually translating 250 grade-school math problems from the GSM8K dataset (Cobbe et al., 2021) into ten typologically diverse languages. We find that the ability to solve MGSM problems via chain-of-thought prompting emerges with increasing model scale, and that models have strikingly strong multilingual reasoning abilities, even in underrepresented languages such as Bengali and Swahili. Finally, we show that multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment. The MGSM benchmark is publicly available at AnonymousLink and the supplementary material.


Published as a conference paper at ICLR 2023
LANGUAGE MODELS ARE MULTILINGUAL CHAIN-OF-THOUGH REASONERS
Freda Shi
Mirac Suzgun 1,3,+
Markus Freitag
Xuezhi Wang'
Suraj Srivats
Soroush Vosoughi
Hyung Won Chung
Yi Tay!
Sebastian Ruder!
Denny Zhou-
Dipanjan Das
Jason Wei
GGoogle Research
Toyota Technological Institute at Chicago DDartmouth College
Stanforr University
ABSTRACT
We evaluate the reasoning abilities of large language models in multilingual settings. We introduce Multilingual Grade School Math (MGSM) bench- mark, by manually translating 250 grade-school math problems from the GSM8K dataset (Cobbe et al., 2021) into ten typologically diverse languages. We find that the ability to solve MGSM problems via chain-of-thought prompting emerges with increasing model scale, and that models have strikingly strong multilin- gual reasoning abilities, and Swahili. Finally, we show that the multilingual reasoning abilities of lan- guage models extend other tasks such as commonsense reasoning and word- in-context semantic judgment. The MGSM benchmark is publicly available at https:: //tt:///g/tt/// com/ http:/goooeeeeeeeeetteee
in underrepresented languages such Bengali
Translate English with Google Translate and solve with English intermediate steps Intermediate reasoning steps in the language of the question
Intermediate reasoning steps in English
70
60
MGSM Accuracy (%)
50
40
30
20
Underrepresented
High-resource languages
English (EN)
languages
10
(SW, BN. TE, TH) (JA, ZH, RU, ES, FR. DE)
0.01%
1%
100%
Frequency of language in pre-training dataset (token percentage)
Figure 1: Correlation between language frequency and MGSM accuracy for PaLM-540B. The accuracy is surprisingly high, even for underrepresented languages like Swahili (SW) and Bengali (BN), which account for less than 0.01% of the pre-training dataset.
"Equal contribution, Work done during internship at Gongle Research.

---

Published as a conference paper at ICLR 2023
INTRODUCTION
Recent work has shown that presenting explicit reasoning steps (i.e., chains of thought; CоT) in En- glish elicits multi-step reasoning abilities large language models such as GPT-3 and PaLM (Brown et al., 2020; Chowdhery et al., 2022; Wei et al., 2022b, inter alia). Pretrained multilingual language models have also achieved impressive performance on various NLP tasks across typologically distinct languages (Conneau et al., 2020; Xue et al., 2021; Chowdhery al., 2022; Clark et al., 2020; Hu et al., 2020; Ruder et al., 2021, inter alia). Tasks in existing multilingual benchmarks usually require only simple reasoning steps, and so it is still unclear how well language models perform on tasks that require more complex reasoning in a multilingual setting.
In this work, we introduce the MGSM benchmark to bridge the gap between the progress on English- based chain-of-thought reasoning and multilingual NLP. We extend a subset of the English-language GSM8K dataset (Cobbe et al., 2021) to ten typologically diverse languages via manual translation of problems into target languages. To the best of our knowledge, this is the first multilingual benchmark to evaluate the arithmetic reasoning abilities of language models.
We evaluate two large language models, GPT-3 (Brown et al., 2020; Ouyang et al., 2022) and PaLM (Chowdhery al., 2022), on this benchmark. While both models solve less than 20% of problems with standard prompting, the 540-billion parameter bio-llioo-pparretteer PaLM model in particular shows exceptional multilingual reasoning abilities with intermediate reasoning steps (Figure 1), solving more than 40% of the problems in any investigated language, including underrepresented languages such as Bengali and Swahili. In our best setting, PaLM achieves an average solve rate of 55% across languages. We find that intermediate reasoning steps in English consistently lead to competitive or better results than those written in the native language of the question, suggesting that English chain-of-thought prompting may be useful baseline for future multilingual reasoning work.
We further demonstrate that the multilingual reasoning abilities of pretrained models extend to common-sense reasoning (Ponti et al., 2020) and word-in-context semantic judgment (Raganato et al., 2020). By presenting the models with few-shot examples in different languages, PaLM sets a new sfate-of-the-art performance (89.9%) on XCOPA (Ponti al., 2020), outperforming the prior approaches that require thousands of training examples.
THE MGSM BENCHMARK
In this section, we describe the collection process of Multilingual Grade School Math (MGSM), to our knowledge the first multilingual arithmetic reasoning benchmark.
Source data. We used GSM8K (Cobbe al., 2021), an English-language human-annotater grade-school math problem dataset, as the base data source. For MGSM, we took the first 250 examples from the GSM8K official test example list. Each problem requires two to eight steps to solve according to the official solution (Figure 2). The answer for each question GSM8K was written as an Arabic numeral, which we kept consistent across all languages to facilitate cross-lingual prediction.'
50
78
70
60
55
Problems
30
20
13
13
10
Target language selection. We selected a typologi- cally diverse set of ten languages other than English (EN), spanning eight language families and different levels representation in standard pretraining datasets such mC4 (Xue et al., 2021): Bengali (BN), Chinese (ZH), French (FR), German (DE), Japanese (JA), Russian (RU), Spanish (ES), Swahili (sw), Telugu (TE), and Thai (TH).
4 Staps
Figure 2: MGSM problem distribution with respect to the number of reasoning steps in the standard solution.
'Certain scripts such as Devanagari employ different numerals. We restrict the data to Arabic numerals for consistency but future work may investigate tross-lingual numeracy by mapping Arabic numerals to those of the corresponding script (see Spithourakis & Riedel, 2018).
2

---

Published as a conference paper at ICLR 2023
Original Question
Frage: Roger hat 5 Tennisbälle. Er kauft noch Dosen Tennisbälle. In jeder Dose sind Tennisbälle. Wie viele Tennisbälle hat er jetzt?
DIRECT
Antwort: 11
NATIVE-COT
Schritt-für- Seaei-ffr---eeee Seeritt-üü--Shhitt---r Sew---hhhhtttworrrr Roger begann mit $ Bällen. 2 Dosen von jeweils Tennisbällen macht 6 Tennisbäll.. 3 + II. Die Antwort ist 11.
EN-COT
Stop-by by-ptyy-pppp Answer: Roger started with balls. 2 cans of 3 tennis balls cach is 6 tennis balls. 5 + 6 11. The answer is 11.
Translated English Question
tennis balls. He buys 2 more cans of tennis balls. Each can has 3 tennis balls. How many tennis balls docs he have now?
Question: Roger
TRANSLATE-EN
Step-by-Step Answer: Roger started with balls. cans 3 tennis balls each is 6 temnis balls. 5 + 6= 11. The answer is 11.
Table 1: Example solution formats ($3) for a German exemplar problem, where German-specifer components are underlined and are changed to the corresponding translations for other investigated languages. For DIRECT, NATIVE- CoT and EN-CoT, we provide the original German question as input to the model and expect an answer in the corresponding format; for TRANSLATE- ENN, we input the translated question in English, and expect step-by-step solution in English. To obtain the desirable output format, we prepend few-shot examples in the corresponding format.
Manual translation process. We enlisted the help of paid professional translators (two for Chinese and German, three for Russian, five for Thai, one for each remaining target language) for the manual translation of the 250 selected English-language examples from GSM8K, through professional translation services. ? All translators involved were native speakers of the target language and had at least years of professional experience in translating between English and the target language. All translators had signed a machine translation (MT) non-usage declaration before they started to work. To verify the quality of the human translations, the vendor sent random subset of translations an additional translator to verify the quality, and checked for n-gram overlap with popular MT providers, such as Google Translation and Bing Microsoft Translator, ensure that no machine translation toolkit has been used. We employ the translation results as gold standard translations.
MULTILINGUAL CHAIN-OF-THOUGHT PROMPTING
We provide an overview of standard prompting and chain-of-thoughh prompting, as well as their extensions to the multilingual setting, which we illustrate in Table 1 and use in experiments ($4).
In standard prompting, given a prompt in the source language, the model is asked to predict the answer (Brown al., 2020; Schick & Schütze, 2021). This can be done in a zero-shot or few-shot setting by providing exemplars following the same template as additional input to the model. We refer this setting as direct answer prediction (DIRECT) the model directly predicts the answer to the problem. This setting measures the model's 's ability to solve problems without any intermediate reasoning steps.
Chain-of- (COT; Wei et al., 2022b) prompting helps improve many few-shot reasoning tasks, by augmenting few-shot examples with intermediate reasoning steps that should be predicted by the model. In the multilingual setting, we can apply CoT to solve the problem in the native language (NATIVE- COT) by predicting the reasoning steps in the original language of the problem. This measures the model's ability to both understand and solve the problem in a specific language.
Alternatively, we can ask the model to predict the chain of thought in English (EN-CoT), regard- less of the problem language. Such approach may be useful as English is often used as the source language for cross- lonsull transfer (Hu et al., 2020) and has been found effective when used as the prompt language (Zhao& & Schütze, 2021; Winata et al., 2021; Lin et al., 2021b).
In addition, Kojima et al. (2022) find that zero-shot CoT is surprisingly effective on reasoning tasks, by prompting the model with "Let's think step by step." without any exemplar. They then concatenate
Phttps:// htttps://www.weggggt obal
3

---

Published as a conference paper at ICLR 2023
DIRECT
NATIVE-COT
EN-COT
TRANSLATE-EN
NATIVE-EXEMPLARS
ENGLISH-EXEMPLARK
N/A
N/A
MULTILINGUAL- SEITLINGUULL-EEMPPLASE EXEMPLARS
N/A
Table 2: Possible combinations between few-shot exemplar selection and solution strategies.
Model input jest soarppler promptrn)
Modellnpar pirpst (muilt rgua scompler promptig)
9acan Makure Guno wivcc are nanu
Cengai quantion
Ororcao sincc Ape ronudo octunice vero coredy
wasth daln
Russlan thought
Dovamase peuansse Rean Wesarr farmo 78
chng of thought
60ll Fist Biie (ottr meer diter aee in trom
50: 22 Oraee
Chinsse quest
coesti risaf arleFee rott eve fow eagi (V' offeter sosert (eefh aoa flx upn ate riger sen offeer foo Tay ampar path argay prggii stree rtnorh topt attrfer yi yrorest stear arla 92 Roomt ugen lefit greeons <ow's sffflw we cnto Shroter uqee??
Bongui
thouggt
80 158.8 S
Bengel
seawey Morr rite cap on mager myy aridhi part fare Fixs Sun agget anchet slow fowalh oi mempr etteter Srreti WORr)
Model outpur
Modil putput
sriften rsforiir fevd zort fit fics effor est sflfier afall (loes ofofie
Bangal chai of thought
Picz vrlive beft 90m ost stathe 199 10 3-4 20 for Flrgr womnt stedt lope in Bai zrohwe Iafh
gver trist Port fram
Figure 3: The chain-of offin-of--hhuuuht prompts and example model outputs in the MGSM experiments. The solutions written in the same language the questions of interest (NATIVE-COT)..
the question, model response and "The answer is" to prompt the model again for the final answer. In this work, evaluate zero-shot CoT by solving both in English (EN-CoT-OsHor) which uses the prompting sentences above, and in the native language of the problem LATIVE-COT-0SE which uses the translated prompting sentences in the corresponding languages with Google Translate.
Finally, we can translate the problem to English and solve it with English CoT (TRANSLATE-- EN). In this setting, use the Google Translate API to translate problems into English. This mirrors the translate- -rrinn setup (Hu et al., 2020; Xue et al., 2021; Ruder et al., 2021), the best-performing setting for fine-tuning multilingual models where the training data translated to English.
Beyond the prompting methods, there are different ways to provide few-shot examples in context for multilingual prompting:
All native question exemplars together with their solutions the few-shot prompt exemplars. This is the most natural setting when we have few examples in each investigated language.
We use a few in-language questions
All English question exemplars (ENGLISH-EXEMPLABSS When we are unable access any existing questions or solution examples in some languages, an intuitive way is to use English questions and solutions as exemplars to perform zero-shot cross-lingual transfer. Note that is unrealistic combine this exemplar selection setting with NATIVE-Con since assume access to the native language for prompting.
Generic multilingual question exemplars (MULTILINGUAL MUUTTLINGUA--EXEPPLAAA EXEMMLAAS) Similar to ENGLIISH-EEEMPLARS we assume access to questions and solutions a few languages, and test if multilingual exemplars better elicit the multilingual reasoning ability of models.
For TRANSLATE-ER EN, as all exemplar questions and solutions are English, we only experiment with the translated native question exemplars and English COT. We summarize the combinations of prompting and exemplar methods in Table 2, and present an illustration Figure Detailed prompting input for each investigated combination be found in Appendix A.2.
4

---

Published as a conference paper at ICLR 2023
AVG HRL
URL
EN
DE
Lang. Freq. (PaLM, %)
78.0
3.5
3.3
2.1
.53
.40
.38
.04
.02
.006
.005
GPT-3 (text-davinci-002,
DIRECT
11.7
15.1
5.7
16.0
14.8
16.8
17.2
12.4
18.0
11.2
8.8
0.8
4.4
8.8
NATIVE-COT
26.4
34.7
7.2
53.6
36.0
37.6
40.4
28.4
40.0
26.0
10.8
0.4
6.4
11.2
EN-CoT
31.6
39.4
13.9
53.6
44.0
46.0
44.8
28.4
40.8
32.4
19.6
5.6
9.6
20.8
NATIVE- COT-OSHOT
46.8
13.2
2.4
EN-COT-OSHOT
29.5
37.2
13.7
46.8
40.0
36.0
42.4
40.0
33.2
31.6
19.6
6.8
15.2
13.2
TRANSLATE-EN
45.6
47.5
40.7
53.6
46.4
46.4
51.6
48.8
47.2
44.8
41.2
42.8
41.2
37.6
PaLM-540B
DIRECT
18.6
19.3
16.8
22.0
18.8
19.6
20.0
22.0
19.2
16.0
16.8
17.6
17.2
15.6
NATIVE-COT
48.1
47.9
44.9
62.4
49.2
46.4
56.8
48.4
46.8
40.0
52.8
45.6
46.0
35.2
EN-CoT
51.3
52.3
46.8
62.4
53.6
51.2
58.0
55.6
46.0
49.6
49.6
46.8
46.4
44.4
NATIVE- NATIVE-COT-OSHEE COT-OSHOT
14.4
13.2
7.7
48.0
12.8
12.4
16.8
13.6
10.8
12.8
7.6
6.8
6.8
9.6
EN-COT-OSHOT
30.8
38.3
15.2
48.0
38.4
36.0
42.4
42.0
35.6
35.2
20.0
14.0
TRANSLATE-EN 55.0
56.3
51.2
62.4
57.2
55.2
60.0
59.6
55.6
50.0
50.8
49.6
53.2
Table 3: Accuracy (%) on MGSM of different models and languages with exemplar questions in native languages NNATIVEEEEEPLABSS HRL: average performance across high-resource languages with larger than O. % frequency in the training corpora; URL: average performance across underrepresented languages. We use 6 questions and solutions as the few-shot exemplar whenever possible: while the token number for 6-shot prompts in some languages may exceed the token number limit of GPT-3, we use the maximum possible number of exemplars instead for these cases. Detailed numbers of exemplars for each language in GPT-3 experiments can be found in Appendix 1. The best numbers in each column are in boldface.
EXPERIMENTS ON MGSM
In this section, we evaluate the multilingual reasoning abilities of representative state-of-the- art pretrained large language models-GPT-3 and PaLM-on our MGSM benchmark in various prompting settings using exemplars in the source language (AXTIVE-XEEMPLARS)) Throughout this paper, we generate outputs using greedy decoding (i.e., sampling with temperature r = 0).
4.1 MAIN RESULTS
We first compare the few-shot NATIVE-EXEMPLAR performance with different solution strategies (Table 3). In line with the English results reported by Wei et al. (2022h), find that intermediate reasoning steps (NATIVE-COT) and EN-COT) help both models achieve substantial reasoning perfor- mance gains across all languages, outperforming direct answer prediction with no explicit reasoning steps (DIRECT) by a significant margin. PaLM shows exceptional multilingual reasoning ability: while outperforms GPT-3 on languages with different settings, PaL.M-540B with intermediate reasoning steps (NATIVE-CoT and EN-CoTI achieves results similar to TRANSLATE-EN on all languages, even on underrepresented languages such as Bengali (BN) and Swahili (sW), which cover less than 0.01% % of the training corpora.
In addition, reasoning English COT,,,, EN EN-COT-OSHOT) consistently achieves competitive or better performance than reasoning in the native language of the question (NATIVE- C-CT, NATIVE- COT-OSHOT), suggesting that English intermediate steps can be considered as useful baseline in future work multilingual reasoning.
4.2 FURTHER ANALYSIS
Effect of language frequency in training corpora. We illustrate the main results of NATIVE- COT, EN-COT and TRANSLATE-EN with respect the language frequency PaLM training data
We focus on these two models due to their notable few-shot performance; in contrast, many other multilingual models are not competitive them in the same settings, and are generally used for finetuning with more data (Winata al., 2021).

---

Published as a conference paper at ICLR 2023
GPT-3
PaLM
en
70
70
dc
MGSM Accuracy (%)
60
60
fr
70
de
en
50
50
MGSM Accuracy (%)
60
fr
40
40
zh
50
30
ja
40
zh
20
30
ja
10
10
20
0
bn
10
te
A
C D D2
8B
62B
540B
sw
0
bn
Model Names
Model Scale
SW
# few-shot exemplars
Figure 4: MGSM accuracy with different models. Let- ters A, B, C, D1, and D2 denote text-ada-001, text- babbage-000 text-curie-001 text next-davinci-001. and text-davinci-002 in the GPT-3 family, respec- tively. While the number of parameters in each GPT-3 model is not publicly available, we order them alphabet- ically. Detailed numbers can be found in Table 8.
Figure 5: MGSM accuracy of Pal.M-540B with different numbers of few-shot exem- plars. Detailed numbers can be found in Table 8.
(Figure 1). Surprisingly, there is strong correlation between the performance and the language frequency in the training corpora: the average accuracy among the four underrepresented languages was only 3% lower than the that among the six high-resource languages (44.9% 47. 47.9%). Moreover, the performance of reasoning in Thai, Telugu, and Bengali is on par with reasoning in French, Japanese, and Chinese, despite having significantly much less data in the training corpora.
In contrast to prior work that identifies language frequency as important for complex NLU tasks with relatively smaller models (Hu al., 2020; Lauscher et al., 2020; Ahuja et al., 2022), these results thus indicate that the reasoning ability of large language models may not be primarily dependent on their presence in training data and that language models are able to transfer their knowledge from high-resource to underrepresented languages to some extent.
Effect of model scale. We analyze the effect of model scale (i.e., number of model parameters and computational resources used for training) on their multilingual arithmetic reasoning abilities (Figure 4). As the models scale up, the performance generally improves for both GPT-3 and PaLM model series on all languages. Neither model achieves a substantial solve rate until certain scale (text-davinci-o0)
for GPT-3 and PaLM-62B for PaLM), hence multilingual reasoning can be considered an emergent ability of large language models (Wei et al., 2022a). It is worth noting that the amount of training data per language is constant across language model scales for PaLM- fact that scale facilitates reasoning implies that further scaling may continue to improve the multilingual reasoning ability of large language models.
Effect of exemplar amount. We analyze how the multilingual reasoning performance of PaLM- 540B, the overall best best-performing model, is affected by the number of few-shot exemplars (Figure 5). Although not all trends are strictly increasing with the number of exemplars, Pal.M- benefits from having more examples in general for all languages.
Effect of exemplar type choice. We compare the multilingual reasoning performance of PaL.- 540B across languages with different exemplar choices (Table 4). For the MUL.TI.NGUALL- EXEMPLARS setting, concatenate one example from each of the most frequent languages (English, German, French, Spanish, Russian, and Chinese) as the generic prompt for all languages. While NATIVE-EXEMPLARE and EN-COT, MULTILINGUAL- EXEMPLARS with EN -CTT achieves competitive performance across the board, suggesting an effective approach when do not have access to any existing example in some languages.
the best choice is almost always
Most notably, with EN- COT, EXEMPLARS all -on-Egglish languages, including those not covered by the few-shot examples,
BIILINGUAL--XEMPLLCC significantly outperforms ENGLISH-

---

Published as a conference paper at ICLR 2023
AVG HRL URL
DE
NATIVE-COT
48.1 47.9
44.9
62.4
49.2
46.4
56.8
48.4
46.8
40.0
52.8
45.6
46.0
35.2
EN-CoT
51.3 52.3
46.8
62.4
53.6
51.2
58.0
55.6
46.0
49.6
49.6
46.8
46.4
44.4
MULTILINGUAL EXEMMLARS
NATIVE-COT 29.8 3188
26.3
52.0
41.6
7.2
10.4
36.0
42.8
32.8
18.0
33.6
26.8
26.8
EN-CoT 48.7 50.0 46.3
57.6
53.2
43.2
53.2
48.0
51.2
43.6
46.8
46.4
48.4
43.6
ENGLISH-EXEMPIAAR
EN-COT 34.7 39.4 26.6 62.4 46.0 37.2 50.4 23.6
26.8
17.2 30.0 34.4 24.8
Table 4: Performance MGSM with different prompt exemplar type choices: the first section is copied correspondingly from Table 3. The best numbers in each column are in boldface.
MODEL
AVG
ET
HT
ID
IT
QU
SW
TA
TH
TR
VI
ZH
HUMAN
97.6
98.2
96.4
100
97
94.8
99
98.6
98.2
96.4
98.4
96.6
MAD-X Base
61.0
61.3
53.7
65.8
63.0
52.5
56.3
61.9
61.8
60.3
66.1
67.6
XLM-R Large
68.7
71.4
(50)
79.8
72.6
(50)
59.2
72.8
74.4
73.8
mT5-XXL
74.9
77.5
72.1
81.1
75.9
54.5
74.1
75.9
78.3
78.1
76,9
79.5
RoBERTa Large (TT)
76.1
81.0
73.8
82.2
77.8
(50)
74.2
79.6
71.4
79.6
81.0
86.0
Codex (code-davinci-0002
DIRECT
73.8
55.6
88.8
95.4
51.2
56.0
54.6
70.2
88.6
80.4
91.4
EN-CoT
80.7
88.8
79.6
91.4
96.6
52.2
67.4
55.8
84.2
91.2
86.6
93.4
PaLM-540B
DIRECT
83.7
77.4
78.0
92.6
96.0
61.0
69.4
85.4
87.2
92.8
89.8
91.6
EN-CoT
89.9
91.0
89.6
94.0
97.4
66.8
85.4
90.8
90.2
94.6
94.6
94.8
Table 5: Accuracy on the XCOPA languages compared to previous work. Human evaluation (HUMAN) on XCOPA was performed by Ponti al. (2020). The MAD-X Base, XLM-R Large, and RoBERTa Large (Iranslate lest) results are from Ponti et al. (2020), whereas the mT55 results are from (Ruder et al., 2021). Applying multilingual CoT-prompting to PaLM-540B has enabled us achieve a new late-offthee-rt performance on XCOPA. The best model result in each column is in boldface.
suggesting that a multilingual few-shot prompt helps elicit the multilingual reasoning abilities of models more effectively than monolingual (English) one.
EXTENSION TO OTHER MULTILINGUAL REASONING BENCHMARKS
To better understand the multilingual reasoning abilities of large pretrained language models, we extend our experiments to two additional multilingual reasoning benchmarks, XCOPA (Ponti et al., 2020) and XL-WiC (Raganato et al., 2020). Throughout this section, we evaluate the Codex (code- davinci-002: Chen et al., 2021y* and PaLM-540B models.
5.1 XCOPA
XCOPA is multilingual evaluation dataset designed to assess the causal commonsense reasoning capabilities of language models across multiple languages." is an extension and re-annotation the English COPA dataset (Gordon al., 2012) where the validation and test set examples carefully translated and annotated
typologically diverse languages. These languages are Estonian (ET), Indonesian (ID), Italian (IT), Cusco-Collao Quechua (QU), Swahili (sw). Tamil (TA), Thai (TH),
'For both investigated tasks, find that code-davinci-002 generally produces competitive hetter results than text-davinci-002 on small set of samples. consideration of hudget, we choose to use code-davinci-002 because it supports free access at the time of our experiment. httpps: //gi thub. com/ cambri dgel 1/xcopa

---

Published as a conference paper at ICLR 2023
Model
AVG
BG
DA
DE
ET
FA
FR
HR
IT
HUMAN
87.0
74.0
97.0
78.0
75.0
76.0
85.0
XLM-R Large| 68.9 66.5
71.1 65.8 68.7 75.3 62.5 72.3 64.9 63.8 69.6
72.8 73.2
Codex (code-davinci-002
DIRECT
60.8
59.2
59.6
68.2
59.0
58.0
58.6
65.7
55.4
56.0
62.0
64.8
63.0
EN-COT
61.4 60.2 66.6
70.6 60.3
63.6
64.6
61.0
54.2
52.2
56.6
62.8 64.0
PaLM-540B
DIRECT
66.7 62.6 67.4
72.6
62.3
75.0
64.6
65.0
59.4
64.0
70.2
72.0 64.8
EN-CoT
63.2 63.4 64.6 68.6 61.5 67.2 64.6 55.9 57.4 55.6
66.4 69.4
64.0
Table 6: Accuracy on the XL-WIC languages with denotes the previous cate-of-the-art results trained with 5.4K English examples (Raganato et al., 2020). The best model result in each column is in boldface.
XLM-R Large
Turkish (TR), Vietnamese (vO), and Mandarin Chinese (ZH). The task objective is to determine the causal relationship between the premise and two options based on a question (which is either "What was the cause?" or "What happened as resulr?"). A successful model is, therefore, expected to not only perform commonsense reasoning but also generalize its reasoning capabilities to new languages. For each target language, XCOPA contains 100 annotated examples in the validation set and 500 examples in the test set. In our experiments, we focus on the examples in the test sets and use the ones in the validation set few-shot exemplars whenever needed.
We test the Codex and PaLM models under both DIRECT and EN-COT. In both settings, we include the same set of examples, randomly selected from the validation sets of TR, ZH. TA, and Qu, but for EN-COT, we additionally write brief rationales (in English) before the final answers ourselves.
Results. Table 5 presents our main results, along with per-language breakdowns for each XCOPA language. The previous state-of-the-art performance was around 76%, obtained by RoBERTa Large in the translate-test setting where the English RoBERTa Large model was first trained on the English COPA (Gordon et al., 2012) and English SIQa (Sap et al., 2019) datasets and then applied to the XCOPA test data, which was translated to English (Ponti et al., 2020). With only four multilingual chain-of-thought examples (EN-CoT), PaLM-540B outperforms RoBERTa Large by significant margin (14%), thereby setting a new high bar on XCOPA. While Codex performs better than RoBERTa Large, still falls 9% behind PaLM-540B. We also highlight that PaLM-540B performs noticeably better than all the other models on under-represented languages such as ET, IIT, and sw; this result suggests that PaLM-540B might have some internal knowledge about these languages.
5.2 XL-WIC
XL-WiC is multilingual word in-context semantic judgment benchmark covering thirteen lan- guages: Bulgarian (BG), Danish (DA), German (DE), Estonian (ET), Persian (FA), French (FR), Croatian (HR), Italian (IT), Japanese (JA), Korean (KO). Dutch (NL) and Chinese (ZH). Given two sentences in the same language and a word of interest which appears in both sentences, the model is asked whether the word is of the same sense in the sentences. In order to arrive at the correct answer, a model needs to be aware of the concept of word sense, and to infer the sense of a word based on its context. Despite its simplicity, this task is extremely challenging; PaL.M-54BB only achieves score of 64.6 on WiC (Pilehvar & Camacho-Collados, 2019), the English version of the task.
Results. We evaluate the cross-lingual word-in-conterr sense judgment performance of models (Table 6). With the supervision from only four examples. PaLM-540B achieves competitive or better results that the state-of-the Korean and Dutch) of the 12 investigated languages. However, we do not observe an improvement over direct answer prediction when using chain-of-thought prompting on this task.7
model (XIM-R Large) on (German, Persian, French, Japanese,
'ntpps: /tppppiii Lehvar git.ub..t.....o.ogo /ot./o//iiiie /wo//wicc/////cccco..ooo..cm.wi/
"One potential reason is that our prompts not necessarily optimal (Wang et al., 2022) and may benefit from broader investigation of other prompt formats. On the other hand, rationales for this task fairly
8

---

Published as a conference paper at ICLR 2023
6 RELATED WORK
Prompting. Existing work (Radford et al., 2019; Brown et al., 2020; Schick & Schütze, 2021, inter alia) has shown that prompting pre-trained large language models can lead to strong performance on various tasks such as text classification (Shin et al., 2020; Gao et al., 2021), question answering (Khashabi et al., 2020), and program synthesis (Austin et al., 2021; Nye et al., 2021: Shi al., 2022a): taking few examples of the task in a certain pattern as the prompting input, models are often able to generate accurate output following the pattern. Wei et al. (2022b) have shown that chain-of-thoughh prompting significantly improves the reasoning performance of language models, by adding explicit reasoning steps before the final answer. Ahn et al. (2022) apply chain-of-thought prompting in robotics scenarios, including a multilingual setting. In this work, we systematically analyze multilingual few-shot thain-of-thouge prompting on complicated reasoning benchmarks.
Multilingual pre-trained language models. Through masked language modeling (Devlin et al., 2019; Conneau et al., 2020), auto-regressive language modeling (Brown et al., 2020; Quyang et al., 2022) or encoder-decoder training (Liu et al., 2020; Chen al., 2021: Xue al., 2021). pre-trained Transformer-basee large language models have shown impressive performance on multiple NLP tasks across languages. Previous work (Zhao & Schütze, 2021; Winata et al., 2021; Lin al., 2021b) investigated prompting in the multilingual setting and found that using English prompts with non-English examples led to strong few-shot performance. Evaluation of multilingual models has mostly focused on general information extraction tasks such as question answering (Clark et al., 2020; Hu et al., 2020; Kassner et al., 2021; I; Ruder & Sil, 2021) as well as specific types of reasoning such commonsense reasoning (Ponti et al., 2020; Lin et al., 2021a) and temporal reasoning (Ruder et al., 2021). To the best of our knowledge, this is the first study to evaluate the multilingual multi-step reasoning abilities of large language models.
Cross-lingual transfer and generalization. Previous work has demonstrated that pre-trained multilingual models significantly help cross-lingual transfer on wide range NLP tasks such as cross-lingual named entity recognition (Pires et al., 2019; Mulcaire et al.. 2019), zero-shot cross- lingual dependency parsing (Schuster al., 2019; Shi et al., 2022b), and bilingual lexicon induction (Shi et al., 2021). In this work, we demonstrate strong cross- pooss-lingual generalization of PaLM (64.2, 55) and Codex ($5), on three tasks that require complicated reasoning.
Multilingual benchmarks. To test the multilingual NLP performance of existing models, there has been work introducing benchmarks on various multilingual tasks, including cross-lingual question answering (Liu al., 2019; Clark et al., 2020), natural language inference (Conneau et al., 2018) and bilingual lexicon induction (Lample et al., 2018), as well as collections across tasks (Hu et al., 2020; Ruder et al., 2021). The tasks in these multilingual benchmarks, to the best of our knowledge, require relatively simple reasoning processes. In this paper, we present MGSM, multilingual arithmetic reasoning benchmark, which can be used test multilingual multi-step reasoning abilities of models.
CONCLUSION
In this paper, we introduce MGSM, the first multilingual benchmark to evaluate arithmetic reasoning abilities of language models. MGSM is an extension of the GSM8K dataset (Cobbe et al., 2021) and contains 250 examples written in ten typologically diverse languages. We also present a com- prehensive analysis of the multilingual reasoning abilities of large language models such as GPT-3 and PaLM on multiple multilingual benchmarks, including our own MGSM dataset. We find that large-scale language models appear to perform complex multi-step reasoning across multiple lan- guages, including those underrepresented languages which are covered by less than 0.01% % of training corpora. Finally, we demonstrate that multilingual chain-of-thougee prompting is an empirically effective approach to multilingual commonsense reasoning, outperforming the previous best model on the challenging XCOPA dataset by 13% average.
straight-forwan forward and example-specife It is thus unclear whether the WIC task requires true reasoning that henefits from the depiction of intermediate reasoning steps. We leave further investigation for future work.
9

---

Published as a conference paper at ICLR 2023
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do as can, not as I say: Grounding language in robotic affordances. arXiv preprint arXiv:220.001691, 2022. URL https: / //aaiiiiii..///. /ttp///.i...........a../////aaaa 2204 01691.
Kabir Ahuja, Shanu Kumar, Sandipan Dandapat, and Monojit Choudhury. Multi task learning for zero shot performance prediction of multilingual models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5454-5467, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 374. URL ht ht aataaaaaaantooooooooooo org/ 2022 acl-long 374.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. 07732.
URL https: http//axvvvvv///////////.a htggg/ab/22005
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS, 2020. URL https:: Absttaabbbbb.....a.... html.
nips. cc cc/pappr/2020/
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 2021. URL https://
arxiv.org/ane abs/ 2200 03374.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, al. PaLM: Scaling language modeling with Pathways. arXiv preprint arXiv:2204.02311, 2022. URL atttps://arxt
org/ 2204. 02311.
Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. TyDi QA: benchmark for information-seeking question answering in lypologically diverse languages. Transactions of the Association for Computational Linguistics, 8:454-470, 2020. doi: 101116/tacLaa0000 URL https: wwt.tts://accatthttttt h///a/aaaaantoootoo/oooo wwwptthhooooy..o./2222202 tacl-1.30.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint arXiv:2/10. 14168, 2021. URL httptt/s///// /aaxiv.///aivvve/aa///eeae www.wwwwwww.w.......o..o.eoooott wttp://..rrrga//////ee//
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk. and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475-2485, Brusscls, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/DIS. 1-269
URL httpps://acleeeeeee
/ /awlaaatnoooogo..... org/ 1269.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning scale. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10. (0.18866 11w53//1/20200000 maadl-mminn.... 747. URL https: ptttps://aclanthologg
acl-maa...mia...... 747.
Jacob Devlin, Ming- Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. NAACL, 2019. doi: 10. 11653/v1//199 1423. URL https: htt///aaaaaaattaaaaaaaaaaaa wwwp://oooooooo//11111111
Philip Gage. A algorithm for data compression. Users Journal, 12(2):23-38, 1994.
10

---

Published as a conference paper at ICLR 2023
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre- pre-trained language models better few-shot learners. ACL, 2021. doi: 10. 1111336v1/22221111 lacl-long.299 URL https: haaaa///taaaaatog...... org/2021. acl- www.lonnn.29.. .955.
Andrew Gordon, Zornitsa Kozareva, and Mclissa Roemmele. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In "SEM 2012: The First Joint Conference on Lexical and Computational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 394-398, Montréal, Canada, 7-8 June 2012. Association for Computational Linguistics. URL https::tt///.. paclanthollgy .rg/ 21///1221110 1
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson.
Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pp. 44 1-4411. PMLR, 2020.
Nora Kassner, Philipp Dufter, and Hinrich Schütze. Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models. In Proceedings of FACI. 2021, pp. 3250-3258, 2021.
URL http: / arxiv. org/ abs 22122 00894.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896-1907, Online, Novem- ber 2020. Association for Computational Linguisties. doi: 10.16533/112020 fi2ddn. findings--mmnnnn eesgg..mmll.1111 URL https: //t:://accca /aclantooooooooooo.aoaa org/ 2020 findings -indings--ennpp ..mmnmpp...... 111.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205. 2022. URL https:
// //arxiv. org/ abs 2205. 11916.
Guillaume Lample. Alexis Conneau, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018
Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers. In Proceedings of EMNLP 2020, 2020. URL http:////// /al:://arrza /ttp/arxvvvv..//// org abs
Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, and Xiang Ren. Common sense beyond English: Evaluating and improving multilingual language models for commonsense reasoning. In Proceed- ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 1274-1287, Online, August 2021a. Association for Computational Linguistics. doi: 10.15v53/v1/ 2021 lonl...l- 102. URL https: / /acllllle anthola logy- . org/ 2021 aaclllllngn 102.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' Horo, Jeff Wang. Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyanov, and Xian Li. Few-shot Learning with Multilingual Language Models. arXiv preprint A2
2021b. URL http: / arxiv. org/ ab// 11.112..6666 10668.
Jiahua Liu, Yankai Lin, Zhiyuan Liu, and Maosong Sun. XQA: A cross-lingual open-domain question answering dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp.
Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1/P19- 1227. URL ht hww.s::pptt........m..tt..... /tt/a/aaaanncooo/ wwwp:hhhoogy..r//1111-e- wttp//ogg//////.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint arXiv:2001.088 08210, 2020. URL https: //ap:ii/////xiia// arwiv..vivv..///ii org/ pdd////////// 2001 08210. pdf.
Phoebe Mulcaire, Jungo Kasai, and Noah A. Smith. Polyglot contextual representations improve crosslingual transfer. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 3912-3918, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/NNDD 1392, URL / anthology. .rg org/Nl 9- 1392.
11

---

Published as a conference paper at ICLR 2023
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show your work: Scratchpads for intermediate computation with language models. arXiv preprint arXiv:2112.00114, 2021. URL https::///...........tt / http//eppeeeeeeeeeeeeeeeeee
www..weww.... net/ fo rum?? d 3dm0eddmm2002 Jm9200a.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02/55 2022. URL https:// arxiv. org/ abs 2203. 02155.
Mohammad Taher Pilehvar and Jose Camacho-Collados. WiC: the word-in-context dataset for evaluating context-sensitive meaning representations, NAACL, 2019. doi: 10. 1..1866/11//19-111 URL hwwps:tp/////. / / /ac anthol Segy.org/N11 9 1128.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5009, Florence, Italy, July 2019, Association for Computational Linguisties. URL https: httpe///ccaaatttooa.oee org Baagg/P10 1etp//19-11999
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. XCOPA: A multilingual dataset for causal commonsense reasoning. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2362-2376. Online, November 2020. Association for Computational Linguistics. doi: 10. 10.1866/2//2220
emalp-main.188 185..
htttt::://aaaaaaholooooo org org/2020. emn ma 185.
Alec Radford, Jeffrey Wu. Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8), 2019. URL
pdf.
Alessandro Raganato, Tommaso Pasini, Jose Camacho-Collados, and Mohammad Taher Pilehvar. XI.-WIC: A multilingual benchmark for evaluating semantic contextualization. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 7193-7206, Online, November 2020, Association for Computational Linguistics. doi: 10. 18 2020.emnlp-main.58. URL https: ac www.ahooannoooooyoooooooooooo org/ 2020. emnlp main. 584.
Sebastian Ruder and Avirup Sil. Multi-domain multilingual question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pp. 17-21, 2021.
Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. XTREME-RE R: Towards more chal- lenging and nuanced multilingual evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 10215-10222 10245, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. emnlp-main.802. URL httpsttpp.... aac//alanntt wow.hoooooog.....oooooooooooooo. org/ 2021 wwwllmmlmmmmmm 100-maa.. main 802.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence Natural Language Processing (EMNLP-JJCNLP), pp. 4463-4473, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v 1/118653/11/D1114454 URL https://aclanthoo wwtp:ly.rr./.111/o 1454.
Timo Schick and Hinrich Schützze It's not just size that matters: Small language models also few-shot learners. NAACL, June 2021. doi: 10. 10.18653/v1/201 htwp://////nnnallllaaaamaaa maacc-main.11e URL https: http:::s//ccaaatholooooo wrt.:.rr//200 naac
185.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. Cross Mooss-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1599-1613, Minneapolis,
12

---

Published as a conference paper at ICLR 2023
Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N194 1162.
URL https:
Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I Wang. Natural language to code translation with execution. arXiv preprint MrXiv:2204.1145: 2022a.
Freda Shi, Kevin Gimpel, and Karen Livescu. Substructure distribution projection for zero-shot cross-lingual dependency parsing. In Proceedings of the 60th Annual Meeting of the Association Dublin, Ireland, May
for Computational Linguistics (Volume I: Long Papers), pp. 2022b. Association for Computational Linguistics. doi: 10.1 10.188533/1/2000 aowg://aloo22....ooooo.....o URL https: / /aa///aaanthhooco
org/ 2022 aal-llllllngg 452.
Haoyue Shi, Luke Zettlemoyer, and Sida I. Wang. Bilingual lexicon induction via unsuper- vised bitext construction and word alignment. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 813-826, Online, August 2021. Association for Computational Linguisties. doi: 1/266///12022aallloooooo ht https:
URL
https://aclanthoot htt///acaaaatttooeoo/
wtgg///// acw-long..550
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. EMNLP, 2020. doi: 10.18653/v 1/2000.emmlpp---.... main.346. URL BRLLhhtss///acaattaait /aa.aaaaanatoogyyyyoooooo 2cc.laathoooy..or/22220 org/2020. emn.p-maaie ma in.i.n... 346.
Georgios Spithourakis and Sebastian Riedel. Numeracy for language models: Evaluating and improving their ability to predict numbers. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2104-2115, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 1111186///111111111
URL https: /ttp///aaattlloooroooooooo
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Rationale-augmentte ensembles in language models. arXiv preprint arXiv: 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities large language models. Transactions on Machine Learing Research (TMLR), 2022a. URL https: /aapp:://axxie orgg abs/ / 2206 07682.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou Chain of thought prompting elicits reasoning in large language models. Conference Neural Information Processing Systems (NeurIPS), 2022b. URL https:// arxiv. or.oorggaaa abs 2201222111.00 11903..
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. Language Models Few-shot Multilingual Learners. In Proceedings ofthe Ist Workshop Multilingual Representation Learning, 2021. URL http:::://axwv.or////a../oo abs/////b//// abs/2109. Ttt.///199..667.... 07684.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami AI-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. mT5: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10. (0.18653/v1/2021 //aclanthology org/ 2021 naacl-male 41.
maa.22221.aaalmmm.i111111 10.18653/11200221maacmmmmmm
URL https:
Mengjie Zhao and Hinrich Schütze. Discrete and Soft Prompting for Multilingual Models. In Proceedings of EMNLP 2021, pp. 8547-8555, 2021. URL http: aaaaiiiii.. hot..orrg/bbss. /2109. 03630.
13

---

Published as a conference paper at ICLR 2023
en
fr
es
ru
ja
th
tc bn sw
# Exemplars
6
6
5
4
6
Table Number few-shot exemplars for GPT-3 experiments in Table 3.
Native Exemplare
English Exemplers
Muttlingual Exemplars
Rragec Roger
Reger hac
Guaestor: tamna buys care tammty bach can nos kemms reveroos? Answart11 11
farms be
Fragee Ft suuan Compurer ins Senemouur Vins Mornag his Consercag
noch fien inazilller Computer prict in Senvaree eit Antwort 29
Quactiin Thane computeed
Fiss
Pragec to wananneur Cor puter in Servere Van raug Donnentin wi uden jadden Thg soch Computte toe isczalaars Computer dirdlerr Servermnn Antwort
eadiides Mornday thmmsdyy
many computers the server recent Arswen 29
Fragee Looh han ao Avainer und Schwweeter for an Wern di--5 eover ieee deis aid ragarnt noch lorght
Questcer uish nad on enoodiatann on ofeer aoster must 42, L'neyate how mury pieceu de they have latinn teain
Antwort:: 37
Anewer:
Prages Shawn forr Spiebbugge hoach hat Mama weltern Papes javei louage belicmmnus Wie Eplilouuge hat peaut?
Question:: has five soys Chrittman pos tove each from mon many toos row?
uguates
Antwort:
Respuesta:
Divvettng
gut
Sepuerry yM aleure semen roms arsparccoe
Medresddyy?
rerepan 23 vare no mude i opeay on nonepurr eage Cacraad uswall ayw rors octanocs ward expvap spripal?
Antwortt
Astwort 33
Frager Glvia 10-
für Lif
Question: hea K23, bought begalaa each, How mudioonney
Stuch yekoutt Wer heetthet eevubegg Andwort
kett
REt: R10022 Wir. MM 7:5 sieee a 913-5
aeriit Cinert..
Arswen:
BFR
Preger Jeneta Erden segen 16 Eleraro Tap Sle bust cha judke: Marger
Guastieer Jaawn drean lggen is zar pro fag. Sie aney jaoin
sum notutlcio un bubld not wie piden fag (/t thr 2on freunds den Hous vertoult are saglchroot clum dkmddeeemmeennn fon US codte pnu hocseen bitema Whfs vus Duter cenent we sigfchh otem Ansermmard
Roginn pum molattioon und becir nst wer paden tig Alothr for vie
Prager.anes Ertin 1d bie
Margen
Preands chn Murt venvautt aur tiglct uct clim dwarsmmntttte tor Gotor fricher frnona Wich Dotar airunt nightt dees Baaararr
hrurs backk mit jusker log Maritra hourds Ons Rest aerkaut di diglch ciain Bas sor 200-010--00000 Colle" pro dhicct Dreasss We vet Dofeer siteet ale rigitch sare Deconsmant afr?
Antworts
Answar
Antwort 10
Figure 6: Prompt template in the direct answer prediction setting (DIRECT), solving problem in German, Above dotted lines: few-shot exemplars; below dotted lines: the question of interest and the expected answer. The dotted lines are not included in our experiments.
A DETAILS OF MGSM EXPERIMENTS
In this section, we present details of our experiments on MGSM, including the number of exemplars used for GPT-3 ($A.1) and the detailed prompts in each setting summarized in Table 2 (SA.2).
A.1 NUMBER OF EXEMPLARS FOR EACH LANGUAGE
Given the unbalanced representation of languages in the training corpora, the byte-pair encoding (BPE; Gage, 1994) algorithm tokenizes sentences in underrepresented languages, especially those in a different alphabet from English, into more tokens. Given that the GPT-3 API supports a maximum number of 2048 tokens its input, it does support 6-shot prompting in some languages, including Russian, Chinese, Japanese, Thai, Telugu and Bengali; therefore, we use the maximum possible number of exemplars (Table 7) instead for GPT-3, while using 6-shot for all languages in PaLM experiments.
A.2 MGSM PROMPTS IN EACH SETTING
We present the prompts used in our MGSM experiments in Figures 6 to 8, where the TRANSLATE-ER experiments can be viewed English one with EN-CoT and ENGLISH-EXEMPLANN
B DETAILED MGSM PERFORMANCE
We report the detailed numbers in our analysis (Figures 4 and 5) in Table 8.
14

---

Published as a conference paper at ICLR 2023
Native Exemplars
English Exemplars
Mutlingual Exemplars
Pragee finger van Inrviisall louth soch Desen leoritu antils
Quasttinn Rogee hes tenk tee tape- more cans tewnid
Reger
buys
inder clss Tevaasulle dee Teer shalee hot jesnt Shep- dipe----ddappp.eeee Anewers Soggr cturediwih Graal: cana of tanil eachis tammu
hole Fach can fooe teoiic bain Hlow hove now!
balk hhe
the oraver
Step by Step Answer floge skarted
care harmts
Step-lay Step Answer Roggr courted with bolk of teovll ie twnsi ba - 1t The
beft sach termes beiss the enaver
Progen nove Computter Servenourr tootaagtes
Computer juizt Computar
Fregec Ea nemennee Computer in Gervarr Van his Dovnenting en jeann big woon fort Coregu instal Weo viee Conian for oir pireerrr ferverrrr)
Thursday
There That mesesin 55
Answert The dayoffrrn Moncay Thurdsay means 4-1 30
Step-lty Stoy. Aeewwnn Theie inn doee from Mlondda to Thunndayy 5 computtrr vere addeed euch day, that
the beglening 20 computees. The anaver 30.
comp added
in cotall 4 555 computee tu wene asdeed. there wene iI commuuttriin thgg new ene one i 20 compurers the enswer 2
There comouters
now there +
anower
Frogee leoh han ao Pvaiseer und thon Schwwerrer for ao Meen sin
esterr wde deii oi be geraine nech Zarig
Quastiassi eni rodidt hocoleess ane diaae wod sth t'inay pan
Questton: avait 32 choccs ara et scsur en avalt 42, slsu
Stop- ley--lii-ppp Sieepppp Anawwert lash had ab chocslome and wartt diase
26, how monr pleces do they have lehinn soralt
ond mange combem morereu reste tout Step Step Answert Leai' hes 3S choodltter en
heep Ac. ther meres the were originily by 32xx w et poclstes, 35
Stop byy Step knowert Lean had 32 chooliata und to sester
hove boen oxteon, So in tosul they stlit have Ni- at chocoiates. The enswer aie
hed 42 that menr thene wore gir 3t -it the chocolates 35
ong tally
heve boee estor So kialtthey of hevee na- 3is chroce annee
The sicwe
Progec Shawn forr Spin
We
var
Mama aetrern hapa powelh ovei tppi lomage bekirmmmes Wee tpreleagge het srist? Step- Stepp
Queetion:: Shown tive loys. Chr mitttmnes got tiv toys mach from mon many foo doen row?
Preguate Srsww riene ahcls?
Phea Roulded peiiiil
diss
(aantoe uguate
Step-ty--e
Mtep Answan Hie has tays the get trom mom, enter that hes - 2- koys then get from oo hooe has di
answer
Fragec Miches har Gaittalle Aun Bikewttngg 20 Gurtuale ver Nitrousch
Michaattiiiheeeuutt gotf bolks On Tuaesdiy love 20 goof balls On Wodrweddin he lost yoc How bualle shet
weiterr fooee, lide Go rtilii
Sequeecy W allure Buino 30 semvii aue ronk dhe aropenncos
hut Mitswoec Encie ?ager??
have
rerepen 23 Here are romuuce ii coeaszr oerromppen euar Crirueo
Shapp dheep Arewen Maci out fext with gott toatis lott itt on hos lac 64- -00- -66 after nsa miiee, hose 36-- > da bolk row. The anower aa.
Shep-ing byy -Sep Answen the with sot gol boats and lest 25 bo has bi -2 as oeher losa mooe, has - -i3
se are rome ocrsnncs waro exesagy sprca
Step- Step An Michael startad
golf
bade sos. The ower in
23. ao 58- 22 he fest more. has
belta now. snower 33
Progec Otivre hatt 2012555 US Uhtter Sie runt begeraror USi zoter pro
Queation Of hus fiz, She boughtt thee tergetarig for 12 each How
PGR NLP-51667
stuck getoort Weire of the une useeg"
muchiren only doee ahe have kett?
dollars Ciiva had the beppininng
Sheyr byy-lloo dieepppp. Arewert magyle for Sid shou cose is
doliuaiist The anawar a
ahwel 23-
Stap-ity Stop knower i toayele eihenee shoull cost iir
dofirr aroovr
dellars Olivia had the begitnaing gangaaa so nov ahe hws x
deller, bda 623 beginiigg rig now she 2a
Step-by--San Step Answart Shagel for eech chould casa
The
Regee legec
Merger
Prager lagen Elerpin
Morges
Preger.leeeb Entam togen to bler pio tag. Sie the jecten Mangan
Tap
trunstuce unur bock peoder log sve
slifch
sigfich
Al diverorant owi? Siep- byw.pppppppp.... A.e JJ The snswer
Soop-y- Anownnn J wnwwer ic
(.] the
Figure 7: Prompt template in the English CoT setting (EN-COT), solving problem in German. Above dotted lines: few-shot exemplars; below dotted lines: the question of interest and the expected answer The dotted lines are not included in our experiments.
15

---

Published as a conference paper at ICLR 2023
Native Exemplars
Multilingual Exemplars
Frege: Roger hat 5 Tenrisballe, Er keuft noch 2 Dosen Ternisbääle. In jodir Dose sind Tennishálle. Wie vieie Tennisbálle hat or jatzt?
Question: Roger has tannis balls. He buys
tannis
balls. Each can has tennis balls. How many tennis balls does he have now?
Schritt-für Schritt-Antwort Roger begarn mit Bällen. 2 Dosen van jewells 3 Tennisbällen mecht Tennisbälle, 5 6 Die Antwort ist 11.
Step-by Sttp- Answer: Roger started with 5 balls. 2 cans of tannis balls each is tennis balls, + 6 11, The answer is 11.
Frage: waren neun Computer Serverraum Montag bis fünf Computer installert. Wie Computer sind jetzt im Serverraum?
Donnerstag wurden jeden
Frage: Es woren noun Computer im Serverraum, Von Montag bis Donnerstag wurden jeden Tag noch fünd Computer installert Wie viele Computer jetzt im Serverraum?
Schritt-für- Suur-Schritt--a- Antwort: Van Montag bis Donnerstag sind Schritt-Antwortt
Tage. Jeden Tag kamen 5 neue Computer hinzu. Das macht insgesarnt 4x 5 20 Computer, die hinzugefūgt wurden. Am Antang waren es Computer, also sind os jetzt 9 20 =29 Computor. Die Antwort loutet 29.
Schritt- Ser--frrr---ttttt Sehriit-Antwortt Von Monieg Donnerstag Computer hinzu. Das mocht insgesamt xx5 5 20 Computer, die hinzugefügt wurden. Am Anfeng waren
Tago.Jedon Jodan komen
Computerr, also sind es jetzt 9 20 -29 Computer. Die Antwort lautet 29.
Frage: essen, Schritt- Sehrittt-üür-ae (ürr-chritt---- Antwort: Lesh hat Pralinan und Leahss Schwester 42, Das bedeutet, dass ursprünglich 32 =74 Pralinen waren. 35 wurden gegessen. Also haben sie insgesami noch 74 35 39 Prelinen übrig. Die Antwort lautet 39.
Prolinen
Schwester hot dann insgesemt noch abrig?
Léa avait 32 chocolats at so sosur en avait 42, cilos en ont mangé 36, combien de morcesux lour reste-t-il en tout? ?
Question:
Réponse étape par étape: Lée aveit 32 chooolats et soeur en eveil 42. Cala signifie quill avait forigine 32 42 74 chocolats.
chocolats ont été mangés. Illuu reste done total 35 chocolots. La réponse est 39.
Frage: Shawn hot fünf Spielzeuge. Zu Weihnachtte hot er von seiner Mernee und seinem Pape jewells zwel Spielzeuge bekormen. Wie viele Spisizeuge or jetzt?
Pregunta: Shawn tiene ciico jugustos. Para Novididd, recibió dos juguetes de su mamá dos de su papá. @Cuántos juguetes tiene ehora?
Schhrite-für-See Sa----hhhittttttt Antwort: het Spielzeuge. hot von er nun
Mema bekommen,
Spielzeuge Dann het
Respuesta paso a pase: Tiene juguctes, Racibió 2 do mamá, por lo que después de eso tiene 5 2 juguetes, Luego, reciblóó més del papá. asi que en total tiene 2 juguetes, respuesta es 9.
noch von seinem Papa bekammen, also hot inagesamt Spielzeuge. Die Antwort lautet
Frage: Micheel het 58 Golfbälle, Am Dienstag het 23 Golfbélle varioren. Am Mittwoch hat or weitere verforen. Wio vicle Golfbálla hat er Mittwoch am Ende des Tages?
3epaнa: Meixne Gorno 58 Narveñ jumn ronata. Bo втоpииn OH norepen 23 veve pam rondde. opeay morepan Cromaro MaHed yun romepa ocranoos Hero KOHLY oрept??
Schritt-für- Seet--für--Shet-t----e- Schritt Se---hhhtt---twrtttt Antwort: Michael hatte anfangs 58 Golfbälle und verloran, sodass or 58
Nowarrosoe poweHme: avavane Maiikna daano 58 MaNOй Ana
Nochdem weitere verforen het jetzt 35 =-3 33
roneée, 23 OH moreepen M Hero ocrencos 58 23 35. Noone eroro
Antwort leutet Stück gekauft. Wie viel Geld hat sie Giorig?
он norapan aue 2, tonepi Hero octanoow 35 2 33 Mn40. Oreer 33.
Frage: Olivia hot 23 US- Dollar, Sie hot fünn Bagels für 3 US- Dollar pro
Schritt-für- Surr-Schitt---ar Antwort:S Bagels für 3 US-Dolla pro Stück kosten x 3 15 Dollar Olivie hat anfangs 23 US- US-Doller, also hat jotzt 23 15 Dollar übrig Die Antwort buttet B.
24
1D 5 1
JU FH PA 23
Frage: Janets Enten legan 16 éler pro Tag. Sie drel jodan Morgen zum Frühstück und
Frager Jenets Enten 16 Eier Tag. Sie lest drei Morgen zum Frühstück und backt milt vier jeaion Tag Muffins für inre Fraunale. Den Rest verkauft sie täglich euf dem Bauernmankt für 2 2S- pro frischem Entenei Wie vie! Dollar nimmtt sie táglich ouf dem Bouernmantt cin?
vier jeden Tag Muffins für iive Freunde. Den Rest verkauft sie täglch dem Beuemmarkt für US- pro frischem Entenel. Wie viel Dollor nimmt sie täglich ouf dem Beuernmankt ein?
Schritt-fä für für-Schritt--e Peevit-AAntortt// [..]] Die Antwort loutet 18.
Schritt-füi Seantt-füü-Schriite Schritt-Antwortt [...] Die Antwort lautet 18.
Figure 8: Prompt template with CoT in the question language (NATiVE-CoT) solving a problem in German. Above dotted lines: shw-shot exemplars; below dotted lines: the question of interest and the expected answer. The dotted lines are not included in our experiments.
16

---

Published as a conference paper at ICLR 2023
AVG HRL
LRL
EN
DE
FR
ZH
TH
TE
BN
Lang, freq. (%)
78.0
3.5
3.3
2.1
0.53
0.40 0.38
0.04
0.02
0.006 0.005
PaI.M
Exemplar token length (avg.)
95
105
113
118
118
173
NATIVE-COT
8B 6-shot
4.0
4.1
3.1
6.4
6,8
4.4
2.4
2.8
4.0
4.4
3.2
3.6
3.2
2.4
62B 6-shot
20.0
22.7
13.2
30.4
24.0
24.0
26.0
22.8
24.8
14.8
18.0
11.6
13.6
9.6
5400 1-shot
38.9
39.7
34.8
50.8
42.8
44.8
44.8
41.2
34.8
29.6
40.0
38.0
34.0
27.2
-5408 2-shot
43.7
44.0
39.8
57.2
47.2
43.2
50.4
44.4
44.4
34.4
47.2
38.0
40.8
33.2
-540B 4-shot
45.1
45.5
41.0
58.8
44.8
49.6
47.6
46.4
46.4
38.4
46.4
41.2
44.4
32.0
-540B 6-shot
48.1
47.9
44.9
62.4
49.2
46.4
56.8
48 4
46.8
40.0
52.8
45.6
46.0
35.2
Table 8: Detailed performances corresponding to Figures 4 and
17

---

Published as a conference paper at ICLR 2023
C THE CHAIN-OF-THOUGHT PROMPTS USED IN THE PAPER
In this section, we present the details of the chain-of-thougle prompts used in our paper for the XCOPA (Figure 9) and the XL-WiC (Figures 10 and 11) tasks.
XCOPA
Given premise prompt,
meaningful the choices.
Q: What might happened result "Adam plyangayu kazand."?
Options:
"Borglandi""
"Zengin oldu."
A: Let's think step by stop.
The premise 'Adem piyangoyu kazandi," cen be trenslated from Turkish into English es "The man won the lottery." The first option 'Borglandi." can be translated as "He owes money." whereas the second option "Zengin oldu." can be translated as "He
became rich."
If the man won the lottery, then it makes sense that he became rich es result.
Therofore, the answer 'Zengin oldu."
Q: What might be the cause of 1B6 7.
Options: 10:00 78
step by step.
premise "AATAA
"can transisted Mandarin Chinese into English "Tho eyes with
The first option '1540 as "He had run out onions." It makes sense that the chef's eyes filld with tears because he chopped onions.
can be transioted as "He choppad onions.", whereas the second option "18:8499133 T. "can be translated
Therefore, the enswer is "(B4) #
What might cause of "adio de flage Ganáis www..iiiigiiuua.. amijjaarr??
Options:
"asouit gei pain safi asmpiimu dfinadllggei
"ascudt ger painuafi. u i gnei sigaugit anig @smgágnss*
step by step.
The premisse "aoss woll (lasgddldgg Gevies eudi u'ugjüuggs tusorijggnsi can be trenslated from Ternil into English as "The man felt abliged to attond the event."
The first option "Jecuskt psorusfii own gemgueu pimeellggnn" can be translated as "He refused his friend's invitation to leave.", whereas the second option lauuit get b and sngggns eundeg GenGšand" cen translated es "He had promised his friend to go." It makes sense that the man felt obliged to attend the event because he hed promised friend to go. Therefore, the answirr "sga sain peiniofh raft gadk eegsugns uning @anggtend""
Q: What might have happened result of "Warmiga wasl qhotuqwan huffulurqan.""
Options:
Warmigga wasita rantinonpaqmi yuyoychakurapnn"
'Wermige wasintam pichayta munerqan."
A: Let's think step by step.
The premise "Warmiqa wasi ghatuqwan hufukurqan. can be translated from Cusco Colleo Quechua into English as "The woman celled real estate agent"
The first option "Warmiga wosita rantinanpagmi yuyoychakurapn. can be translated as "The woman plans to buy a condo.", whereas the second option "Warmiqu wasintam pichayta munarqan." can be translated as "The woman needs to cleen her house." the woman called a real estate agent, then makes sense that the woman plans to buy a condo as a result.
Therefore, the enswer "Warmiqe wesita rantinanpagmi yuyaychakurqan."
Figure 9: The chain-of of-thuggttt prompt used in the XCOPA experiments. The four examples are randomly selected from the validation sets of Turkish (TR), Mandarin Chinese (ZH), Tamil (TA), and Cusco-Collao Quechua (QU). The rationales are written by the authors, and the task description is taken directly from (Ponti et al., 2020). Under the direct prompting setup, the answers (bolded) are given directly and rationales are entirely omitted.
18

---

Published as a conference paper at ICLR 2023
XL-WIC
"AAprrecch"
"approech" city.
"sprraach"
Options:
Yes
No
Answer: in Serieence "epproach" means deal with something: Sentence 2, "approach" meens neer something in distance. They different, thus the answer "No".
Sentence Der "Hausmeister" kam und er sah nichts als die Schatten unserer fleiöigen Körper, die putzten und wischten.
Sontence Der "Hausmeister" war hinzugokammen.
Question: the word "éclater" (marked with ") used in the semne sense in both sentences ebove? Options:
Yes
No
Answer: Sentence "Hausmoistar" moans caretaker, Sentence 2, "Housmelster" moons coretaker. They are the some, thus the answer is "Yes".
Sentence L'itteligencee "éclate" dans ses yeux,
plerreriss
toutes parts.
Question:
the 'spproach"
same sentences
Options:
Yes
Answer: Sentence "'éclate" shining someone's Sentence "éclataient" things physically shining. different,
"No".
Sentence "difectare" di denero, di coraggio, di empetia.
Sentence status é cosi reafistica, "difetta" solo della parola.
Question:
the word "difettare" (marked with used the same sense both sentences aboovs?
Options: Yes
No
Answer: In Sentence "difettare" means being lack of something: Sentence 2, 'difetta' meens be lack of something, They are the same, thus the answer is "Yes".
Figure 10: The multilingual chain-of siiin-of-hhouee thought prompt used in the XL-WiC experiments.
19

---

Published as a conference paper at ICLR 2023
XL-WIC
"AAprrecch"
"approech" city.
"sprraach"
seme
Options:
Yes
No
Answer: in Serieence "epproach" means deal with something: Sentence 2, "approach" meens neer something in distance. They different, thus the answer "No".
Sentence Der "Hausmeister" kam und er sah nichts als die Schatten unserer fleiöigen Körper, die putzten und wischten.
Sontence Der "Hausmeister" war hinzugokammen.
Question: the word "éclater" (marked with ") used in the semne sense in both sentences ebove? Options:
Yes
No
Answer: Sentence "Hausmoistar" moans caretaker, Sentence 2, "Housmelster" moons coretaker. They are the some, thus the answer is "Yes".
Sentence L'itteligencee "éclate" dans ses yeux,
plerreriss
toutes parts.
Question:
the 'spproach"
same sentences
Options:
Yes
Answer: Sentence "'éclate" shining someone's Sentence "éclataient" things physically shining. different,
"No".
Sentence "difectare" di denero, di coraggio, di empetia.
Sentence status é cosi reafistica, "difetta" solo della parola.
Question:
the word "difettare" (marked with used the same sense both sentences aboovs?
Options: Yes
No
Answer: In Sentence "difettare" means being lack of something: Sentence 2, 'difetta' meens be lack of something, They are the same, thus the answer is "Yes".
Figure 11: The English- Inglish-language chain-of- of-thuught prompt used the XL- XL-WIC experiments.
20


Reviews:
[{"review_id": "vQ93cx335j", "reviewer": "Reviewer_HXqf", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a benchmark containing 250 grade-school math problems translated from English to ten languages and also provides a deep analysis to explore multilingual reasoning abilities of two SotA large language models GPT-3 and LLM-540B in terms of both chain-of-thought and commonsense reasoning. The proposed benchmark was translated by native speakers of the target language with at least two years of professional experience. The experiments are well-designed, and the results are enough to support the claims. They find that on the MGSM benchmark, LLM-540B shows superior multilingual reasoning ability compared to GPT-3 on all languages with different settings. Similarly, for the commonsense reasoning benchmark, LLM-540B also outperforms the SotA models: RoBERTa Large and XLM-R Large on two other benchmarks: XCOPA and XL-WiC, respectively.", "summary_of_the_review": "Although the findings are interesting, the contributions are not significant enough for the multilingual research since the benchmark is small and the verification process is not well described (or implemented?), which may flip the current conclusions/claims if more multilingual examples with different types of problems are added. The big plus of this paper is the intensive experiments and detailed analysis.", "strength_and_weaknesses": "**Strengths**\n* The multilingual benchmark is translated by professional translators and verified by an additional one for a random subset.\n* Experiments are well-designed with 4 prompting methods and 3 types of input exemplars.\n* Intensive analysis provides new insights for multilingual reasoning abilities of large language models.\n* The paper is also well-written.\n\n**Weaknesses**\n* The number of examples is not significant, with only 250 examples, compared to GMS8K with 8.5K examples, so the results may not be generalized if more examples are translated and added to the benchmark.\n* The verification process is less discussed, and not convincing to guarantee the quality of the benchmark.\n* The approach of translating a problem to English does not test the multilingual reasoning abilities of LLMs.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "v7VOY8TTRq", "reviewer": "Reviewer_3ofN", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduced a multilingual arithmetic reasoning task with a corpus named MGSM, and used the few-shot chain-of-thought (Few-Shot CoT) prompting framework to evaluate the large language models’ few-shot learning capability.  \nAuthors empirically found that the GPT-3 and LLM could provide reasonable performance on their constructed test set, and also provided results on two other multilingual reasoning tasks.\n", "summary_of_the_review": "This work is interesting and well-motivated, the technical contribution can be extended by tackling the points of questions.", "strength_and_weaknesses": "Strength:\n1.  The paper shows interesting experimental results on the multilingual arithmetic reasoning task MGSM. Large-scale language models can perform well across languages, and even on underrepresented languages like Swahili.\n2. The paper provides extensive and convincing experiment results on other two multilingual tasks (common sense reasoning and word in-context semantic judgment).\n\nWeaknesses:\n1. The constructed test sample size for the benchmark is a bit small (250 samples).\n2. Some experiments and additional clarification can be considered and added (see Questions).\n\nQuestions:\n1. In Section 4, it would be better if the authors add the experimental results of adopting Zero-Shot CoT (Kojima, Takeshi, et al. 2022) [1], compared with the Few-Shot CoT model.\n2. One major question I have in mind is, is the current sample size of MGSM sufficient as a benchmark corpus? It would be better if the test sample size can be larger than the current 250, as the original GSM8K contains 1,000 test samples.\n3. It would be better if authors can add some insight or feature-based analysis on the emergent ability of cross-lingual reasoning of the tested language models GPT-3 and LLM.\n4. It is a bit surprising that the language models (GPT-3 and LLM) that are not specially pre-trained on multilingual data work well on cross-lingual reasoning with few-shot settings, is it possible to compare them with some multilingual backbones?\n5. I was also wondering whether the performance on MGSM (shown in Table 3) will be affected by adding some text perturbation (here the perturbation should not affect the numeral text), or adding some spans with code-switching.\n6. In Table 7, why are the number of few-shot exemplars in different language settings not the same? For instance, only 1 exemplar is provided for ru, th, te, and bn, which is smaller than that of en, de, fr, and es (6 exemplars).\n\n\nReferences:  \n[1] Kojima, Takeshi, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. \"Large Language Models are Zero-Shot Reasoners.\" arXiv preprint arXiv:2205.11916 (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "gx5rzjdkrZN", "reviewer": "Reviewer_aQqu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper extends the prevalent GSM8K benchmark to a multilignual verison covering ten languages, and accordingly verify a recently proposed technique Chain-of-though upon it, resulting a best performance of 55 with the most competent LLM.\n\nThe authors also verify such capability on established benchmark XCOPA, achieving a new state-of-the-art.\n\nThe paper is well presented and very clear to understand.", "summary_of_the_review": "The paper is well presented, an multilingual extension of GSM8K benchmark is proposed, CoT is verified accordingly as well as more datasets like XCOPA, which sets a new SoTA.\n\nBut the technique is almost identical to CoT, and the results can be well expected, leaving it only a very incremental contribution. Besides, such empirical verification is also not sufficient as important baselines are missing.\n\nTherefore, I vote for borderline reject, but I'm not absolutely sure and are open for discussion or rebuttal.", "strength_and_weaknesses": "Strength:\n\n1. The paper is well presented and easy to understand.\n\n2. a multilingual version of the popular benchmark GSM8K is proposed and released, which might be of interest to the community.\n\n3. new state-of-the-art on XCOPA and XLWiC is set.\n\nWeaknesses:\n\n1. The paper does not provide much novelty, insights and inspiration, the technique design simply follows existing works of CoT.\n\n2. The empirical investigation is not sufficient for a solid benchmarking of the new datasets. Specifically, I believe several recently-proposed methods like ZeroShotCoT[1] and SelfConsistencyCoT[2] are also necessary and not difficult to implement.\n\n3. The extension of the multilingual version of an existing benchmark seems providing only limited contribution for a ICLR paper to me. And the empirical verification of CoT on more datasets are also not sufficient for another new paper.\n\n[1] Large Language Models are Zero-Shot Reasoners\n\n[2] Self-Consistency Improves Chain of Thought Reasoning in Language Models", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XVqygj6pSr", "reviewer": "Reviewer_q52S", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This is a very straightforward paper, with impressive and surprising - although to me sad (more on this later) - results. They will certainly motivate future work, and encourage some of the current directions.\n\nThe authors analyze the multi-lingual capabilities of language models who are trained primarily on English data, with a little of other languages as well. They combine this with recent work of chain-of-thought (CoT) prompting, showing that those the combination of large LLM + CoT provide a very decent results on challenging datasets, in a multi-lingual setting.", "summary_of_the_review": "Straightforward idea, very well executed. High impact", "strength_and_weaknesses": "STRENGTH\n1. Translating a math-problem dataset (GSM8K) from English to 10 other diverse languages, using professional translators; and releasing the full dataset\n2. Benchmarking two LLMs with CoT on that and two other multi-lingual datsets; showing that they provide competitive and sometimes state-of-the-art results compared to a fully supervised setting\n3. Showing that machine-translating the CoT into English is at least as good and often better than prompting with CoT in the target language directly\n\nWEAKNESS\n1. A point which is never addressed is why translated CoT seems to perform always better than human translated CoT. Is this because some of the training data of the LLM might be translatese? Or is the difference too small to be significant? If so, it would be interesting to see the results with a different set of prompts or with a different sampling\n2. Not really a weakness, but this result will not necessarily encourage more research into low-ressource languages. While the large LLM in this paper is anonymized, I picked a random other language model from all those that also have 540B (PALM in this case). PALM contained 24B French tokens, and this paper reports 46.2% when natively prompted and 55.2% when translated. Swahili only had 40M tokens for PALM, but has a comparable performance in this paper when translated (51.2%, compared to 35.2% for native Swahili prompt). Why developping a Swahili (or French?) LLM when I can just use the largest EN one and machine-translate? At the same time, 40M tokens will have much less variety than 578B tokens used for English and some idiosyncrasy of Swahili might not be captured by translation only.\nAnother direction which will be encouraged by this paper is scaling: Swahili had an almost 4x jump between using a 62B model (there are a few of those available these days) and a 540B (not so many available of that size).\n\nNote that Weakness (2) is not really an issue with the paper... just with the consequences that the reported facts might produce.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "HRRzzSdJf8", "reviewer": "Reviewer_oVpU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers chain-of-thought (CoT) in multilingual setup and shows that despite a highly imbalanced pretraining dataset, LLM still learns strong CoT capability in non-English. It extends an existing dataset to multiple languages. It also compares different ways to construct exemplar and CoT.\n", "summary_of_the_review": "This paper considers CoT in multilingual setup and creates a new dataset, despite minor issues of the experiments.\n", "strength_and_weaknesses": "Strength\nIt introduces a new dataset and presents extensive experiments.\n\nWeakness\n- No variance is presented in the few-shot experiment.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8fVop6CmG_", "reviewer": "Reviewer_DTUx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a Multilingual Grade School Math (MGSM) which can be used to evaluate the reasoning abilities of large language models by predicting the chain of thought. The authors compare the results of different prompting strategies on two large-scale language models, and further conduct ablation studies to analyze the effect of language frequency, model scale, exemplar amount and exemplar type.", "summary_of_the_review": "The contribution of the new benchmark is clear and the experiments are sufficient, while the prompting strategy of multilingual chain-of-thought is incremental.", "strength_and_weaknesses": "Strength:\n1.\tThe MGSM is the first multilingual dataset that can be used to evaluate the chain of thought in multilingual setting. \n2.\tThe paper is well-writing and easy to read.\n3.\tThe experiments are comprehensive and well organized.\nWeakness:\n1.\tThe contribution of multilingual chain-of-thought is incremental compared to the villa chain-of-though.\n2.\tIt is interesting to see the large gap between Translate-EN, EN-CoT and Native-CoT in MGSM. While the gaps in other benchmarks are not too much. Is it because MGSM benchmark is originated from translation?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster