Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
Authors: Haoyang Huang
Tianyi Tang
Dongdong Zhang
Xin Zhao
Ting Song
Yan Xia
Furu Wei
Venue: EMNLP 2023 Findings (2023)
Keywords: Large language models; multilingual capability; cross-lingual-thought prompting;

Abstract: Large language models (LLMs) demonstrate impressive multilingual capability, but their performance varies substantially across different languages. In this work, we introduce a simple yet effective method, called cross-lingual-thought prompting (XLT), to systematically improve the multilingual capability of LLMs. Specifically, XLT is a generic template prompt that stimulates cross-lingual and logical reasoning skills to enhance task performance across languages. We conduct comprehensive evaluations on 7 typical benchmarks related to reasoning, understanding, and generation tasks, covering both high-resource and low-resource languages. Experimental results show that XLT not only remarkably enhances the performance of various multilingual tasks but also significantly reduces the gap between the average performance and the best performance of each task in different languages. Notably, XLT brings over 10 points of average improvement in arithmetic reasoning and open-domain question-answering tasks.

Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thouge Prompting
Haoyang Huang'; Tianyi Tang"; Dongdong Zhang', Wayne Xin Zhao
Ting Song'. Yan Xia', Furu Wei
Microsooft Research Asia, China
Gaoling School of Artificial Intelligence, Renmin University of China https:// thub. wttp://ooo//ooooieeeeoeeoooo
(%)
MGSM
100
92.6
21
FLORES
XCOPA
86.6
84.6
16
70
14
66
25
24
57
61
92.2
XL-Sum*
30
50
XNLI
81.9
82.6
69.6
38
56
MKQA
PAWS-X
MGSM XCOPA
XNU
PAWS-X MKQA
(a)
(b)
Figure 1: Comparing the effectiveness of the Cross-Lingual-Thouue prompt versus the baseline hasic prompt on 7 representative benchmarks covering 27 languages: (a) Enhancing the multilingual capability of text-davinci-000 under the zero-shot learning, and (b) Narrowing the gap between the average performance and the best performance of each task in different languages.
Abstract
Introduction
Large language models (LLMs) demonstrate impressive multilingual capability, their per- formance varies substantially across different languages. In this work, we introduce sim- ple yet effective method, called cross-lines moss-lingual- thought prompting (XLT), to systematically improve the multilingual capability of LLMs. Specifically, XLT is generic template prompt that stimulates cross-lingual and logical reason- ing skills enhance task performance across languages. We conduct comprehensive evalu- ations on 7 typical benchmarks related to rea- soning, understanding, and generation tasks, covering both high-resource and lowrresuurce languages. Experimental results show that XLT only remarkably enhances the perfor- mance of various multilingual tasks but also significantly reduces the gap between the av- erage performance and the best performance of each task in different languages. Notably, XLT brings over points of average improve- ment in arithmetic reasoning and open-domain question-answwring tasks
Large language models (LLMs) demonstrate im- pressive multilingual capability a wide range of natural language processing tasks, including language generation, knowledge utilization, and complex reasoning (Zhao et al., 2023). Their per- formance in downstream tasks has been shown to reach or mance (Brown al., 2020; Chowdhery et al., 2022; Scao et al., 2022). The capabilitics of LLMs stem from the extensive volume of training data they leveraged (Kaplan et al., 2020). The training data for current models is primarily dominated by the English language corpus, but it also encompasses data from other languages, as described in GPT- 3 (Brown et al., 2020), PaLM (Chowdhery et al.. 2022), and BL.OOM (Scao et al., 2022), etc.
surpass human-level perfor-
There over 7,000 languages worldwide, with the vast majority being low-resource extremely low-resource languages (Forkel al., 2022). De- spite the latest GPT-4 model (OpenAI, 2023) demonstrating some generalization capabilities
"Equal contribution. i Corresponding author.

---

XLT
Input
want you to act as an arithmetic reasoning expert for Chinese
Request: f 16 X B2448)) #8800000) *
Request: -
LLM
You should retell the request in English.
P 60 X. 16
You should do step-by-sten answer to obtain a number answer,
You should step step-by-step st- answer the request.
You should tell me the answer in this format 'Answer:'.
Output
Request: James decides run 3 sets 60-meter sprints three times week. How many meters does he run in total each week?
Step-by-step answer:
James runs 3 sets of 60-meter sprints, which means he runs 60 x 180 meters per sprint.
2. James runs sets 180 meters per week, which means he runs 180 x 3 540 meters per week.
Answer: James runs total of 540 meters per week.
Figure 2: Overview of our method. Given a request, its associated meta information is filled into the placeholders of
the XLT template form the language-independeet prompt, which is fed the LLM to enhance the generation of responses in the desired format.
multilingual lasks as evaluated on the MMLU benchmark (Hendrycks et al., 2021), still the case that LLMs not have equal capability to han- dle languages, leading to imbalanced capability across different languages. Furthermore, several evaluation results (Bang et al., 2023; Jiao et al., 2023; Hendy et al., 2023; Zhu et al., 2023) indicate that large models struggle with understanding and generating non-English languages, particularly in low-resource or extremely low-resource languages. Therefore, to democratize language intelligence and minimize performance gaps in different lan- guage, it is essential and meaningful to stimulate and enhance the multilingual capability of models in non-English and low-resource languages.
ral language processing tasks across different tar- get languages. Our method employs a generic and language-independent prompt, which eliminates the need to update model parameters. Depend- ing on the task input type, cross-linguallthought prompting guides the large language model to as- sume the role of expert in specific language for particular task. Given its predefined meta information, XLT directs LLMs to respond logi- cally through a process involving problem under- standing, cross-lingual thinking, task analysis, task execution, and output formatting. During this pro- cess, our method is designed to stimulate models' cross-lingual and logical reasoning skills, enabling them to respond to input requests regardless of the language. For enhanced performance, few-shot learning also be employed with method by providing an LLM-generated response output as a demonstration using cross-lingual thnuuul-thooght prompt- ing zero-shot learning.
Intuitively, LLMs improve multilingual ca- pability by augmenting data (Lin al., 2022) or fine-tuning models (Chen et al., 2021, 2022), but both are computationally expensive. Alternatively, in-context learning with prompts can also boost per- formance (Brown et al., 2020; Ahuja et al., 2023; Wei et al., 2022c) but is limited to monolingual tasks (Sanh et al., 2022).
We conduct comprehensive evaluation to ver- ify the effectiveness of XLT across seven repre- sentative multilingual benchmarks of natural lan- guage reasoning, understanding, and generation tasks. Each benchmark includes multilingual data covering both high-resouree and low-resource lan- guages. The experimental results demonstrate that our method can significantly improve the perfor-
This work explores universal in-context learn- ing approach enhance the multilingual capabil- ity of L.MMs. We introduce simple yet effective method, called cross mooss-lingual lingual-thoughht prompting (XLT), to enable models to handle various natu-

---

I want you act as task_name expert for task_ann
Lask_input
You should retell/repeat the input_tag English.
You should task_goal
You should step-by-step answer the request.
You should tell me the output__ype (output__oosstrint) in this format output__yye
Figure 3: Illustration of XLT template. Referring to Figure 2 and Appendix for instantiated examples.
mance of all benchmarks across languages under both zero-shot and few-shot learning settings. No- tably, XLT achieves an average gain of over 10 points on the MGSM and MKQA benchmarks. Fur- thermore, we observe that our prompting method significantly reduces the gap between the average performance and the best performance of each task in different languages, indicating its potential to democratize language intelligence.
training, and existing research indicates that En- glish prompting is more effective for multilingual tasks (Shi et al., 2023). In contrast to the vanilla prompt that only includes a task description, our XLT template aims to elicit multilingual capabil- ity through cross-lingual thoughts. This template comprises six logical instructions in sequence. To complete the template, only seven placeholders need to be filled in based intrinsic knowledge of the task and the request, depicted in igure 3.
2 Proos-LinguallThougee Prompting
Role Assigning First, the model receives role definition that helps establish the model's behavior, This concept akin to the system role of Chat- GPT2. To achieve this, we simply need to fulfill the task name with a known category (such as commonsense reasoning or paraphrase identifica- tion), along with the language of the task in the task language field.
Although LLMs are capable of accepting any input and generating responses, users typically structure their requests in the form of prompts to elicit the de- sired output. The design of these prompts is crucial for achieving optimal performance on downstream tasks, as LLMs are sensitive to the format of the prompts chosen (Zhao et al., 2021). Through a pro- cess called instruction tuning (Wei et al., 2022a), models can develop the ability to follow natural lan- guage instructions (Wei et al., 2022b), which can reduce their sensitivity to prompt engineering (Wei et al., 2022a). In accordance with the guidelines of the OpenAl cookbook', we propose a cross-lingual thought prompting template, denoted as the XLTT template. This generic template allows LLMs to respond to requests with cross-lingual thought and supports a wide range of multilingual tasks.
Task Inputting Second, explicitly append the request as the task input The request is ba- sically structured in terms of the task type to make sure the model can comprehend it. For example, the natural language inference task, the two sentence inputs are specified with "premise" and "hypothesis", respectively.
Cross-lingual Thinking We encourage the model to engage in cross-lingual thought by rephrasing the requested content in English, which is the dominant language used pivot lan- guage by Shi et al. (2023) and Ahuja al. (2023). Rephrasing the requested content enclosed the input tag helps the model better understand the request in its native language and knowledge. Our observations suggest that using keywords such as "retell" or "repeat" while rephrasing the content may result in better performance in practice.
Figure displays the XLT template, with the col- ored sections representing placeholders. Figure showcases an example of instantiated prompt for the Chinese request. The following section will explain the details of constructing XLT.
2.1 Construction of XLT
The XLT template is designed to emulate the pro- cess humans employ when handling multilingual tasks. Our template is written in English, as En- glish is the dominant language during LLM pre-
hnttps://platform openai.p......mmmm hawpii.coo/oocse ppp.aaa.cmm/bccs///uueeee/ee/e/e introduction
Phttps://githhin ctm http://gttb...oom/g/aa/////eeee/

---

Task Analyzing After rephrasing the task input, we need complete the task in task goal. This step comparable to the task description used in conventional prompting methods. In practice, we can get the task information from the literature or seek assistance from ChatGPT to generate effective prompts for solving the task (Jiao et al., 2023).
Requesi
Dev
XLT
LLM
Response
data
Rasponsa
Aligned?
Demonsirations
CoT Task Solving We then ask the model to follow the instructions and complete the task step by step. Since LLMs exhibit strong ability to maintain a chain-of-thought (Wei et al., 2022c), we carefully design instructions to guide the model, with the hope that it will respond to our instructions in a step-by-step manner and utilize the intermedi- ate outputs to aid in solving the task.
Figure Construction process for few-shot learning.
we can also correct or design the demonstrations for better alignment with the instruction logic.
3 Experiments
To comprehensively verify the effectiveness our method on inguagg-indeeendent generality, we evaluate our XLTT template on different LLMS cov- ering various natural language processing tasks in multiple languages.
Output Formatting Finally, we should regular- ize the output format of the model to obtain the act answer. LLMs are utilized in zero-Gu or few-shot manner, and they tend generate texts that may not conform to the format of the target answer. For- tunately, LLMs possess a strong ability to follow instructions, and we can define the output format in terms of output type and output constraint, The output type can be number, index, or text, while the output constraint is optional and determined based on the task requirements. Output constraint may include length limitations, language specifica- tions, and other relevant factors.
3.1 Experimental Setups
3.1.1 Tasks and Benchmarks
We conduct evaluations on seven typical bench- marks related to reasoning, understanding, and gen- eration tasks that can represent different capabil- ities of LLMs, encompassing both high- and low-resource languages. These benchmarks cover 27 different languages, including English (en), German (de), Russian (ru), French (fr), Chi- nese Simplified (zh), Spanish (es), Japanese (ja), Italian (it), Vietnamese (vi), Turkish (tr), Indone- sian (id), Swahili (sw), Arabic (ar), Korean (ko),
2.2
XLT for Few-shot Learning
The above construction of XLT can be directly fed LLMS to yield outputs, which performed in the zero-shot learning setting. In addition, we also explore incorporating demonstrations into XLT to enable few-shot learning. Different from previous work that just appends model outputs to the corre- sponding request (Shi al., 2023) or utilizes ver- balizer to format the output, method constructs the demonstrations with better formatted model outputs from step-by-step processing-bart based XLT. As illustrated Figure 4, we first sample few ex- amples from the development set and incorporate the requested parts into XLT. The zero-shot learn- ing is performed over LLM to collect responses that are further aligned with those of the samples. Only response-aligned requests assembled with the corresponding model responses form final demonstrations for few-shot learning. In this way, the demonstrations are constructed with rich log- ical knowledge via XLT, which will cater to the
Greek (el), Thai (th), Bulgarian (bg), Hindi (hi), Estonian (et). Bengali (bn), Tamil (ta), Galician
(gl), Urdu (ur), Telugu (te), Javanese (jv), Haitian Creole (ht), and Southern Quechua (qu). In terms of the language distribution statistics in the Com- mon Crawl Monthly Archives and the language performance of LLMs (Shi et al., 2023; Ahuja et al., 2023), we have arranged them in the or- der of language frequency from high-resource to low-resource. In particular, the frequency of some underrepresented languages is even less than 0.1% (e.g., bn, ta, gl. ur, te, jv, ht, qu).
Reasoning tasks
Arithmetic Reasoning. The MGSM (Shi et al., 2023) benchmark contains grade school mathe-
httpss:/ccomoncraill.. hithtmmmoonaaaaa..gu.u.u.... cc- cc-crawl-se crc.aal-ssaatisticcsppll///aaaaa
XLT--bsed generation of new requests. practice,

---

matical problems and asks the model to calculate the correct answer. covers 11 languages, and we utilize the accuracy score for evaluation. Commonsense Reasoning. The XCOPA (Ponti et al., 2020) benchmark contains one premise and two choices. It asks the model to choose which one is the result or cause of the premise. It covers 11 languages from 11 diverse families, and utilize the accuracy score for evaluation.
resource low resource translation (i.e. zh++ th and zh-+jv), and low resource to low resource translation (i.e, th and jv<-tth,, We utilize the SacreBLEU score (Papineni et al., 2002; Post, 2018 8) for evaluation.
Among these benchmarks, MGSM, XCOPA, XNLI, PAWS-X, and MKQA are parallel, i.e., the instances are semantics-equivaleni across each lan- guage. For all benchmarks, we report the results on the test sets using all instances (Table 5), except for XL-Sum and FLORES-200 where we only sam- ple 250 and 200 examples respectively show the trend of generation performance. In the few- shot setting, we randomly choose examples from the development set if they have, otherwise, we translate the English training set into correspond- ing languages to construct several examples.
Understanding tasks
Natural Language Inference. The XNLI (Con-
neau al., 2018) benchmark contains one premise and one hypothesis and requires the model to determine whether the hypothesis is entailed, contradicted, or neutral conditioned on the premise. It covers 15 languages, and we uti- lize the accuracy score for evaluation.
Paraphrase Identification. The PAWS-X (Yang et al., 2019) benchmark contains two sentences and requires the model judge whether they paraphrase each other or not. covers lan- guages, and we utilize the accuracy score for evaluation.
3.1.2 Baselines
Basic Prompt are the vanilla in our experiments that were proposed and suggested in previous work. After determining the prompt, we format each monolingual instance using the English ba- sie prompt. This setting is similar to the mono- lingual prompting in MEGA (Ahuja et al., 2023). The basic prompts used for the evaluation of each benchmark are listed in Table 5. Note that, we dismiss the baseline using native-language, since MEGA (Ahuja et al., 2023) reveals monolingual prompting is superior to cross lons-lin prompting.
Generation tasks
Question Answering. The MKQA (Longpre et al., 2021) benchmark contains an open-domain question and asks the model to predict a short answer. Since has unanswerable questions or long questions that do not have precise answers, we remove these questions during evaluation. It covers 25 languages, and we choose a subset of 10 languages, including de, en, es, fr, ja, ru, th, tr, vi, and zh. We utilize the token overlap F1 score for evaluation.
Chain-of- Chain-of-Thought (CoT) prompting invokes LLMs to generate series of intermediate results to solve reasoning tasks (Wei et al., 2022c), which is still effective under multilingual scenarios (Shi et al., 2023). In experiments, we append the in- struction "Let's 's think step-by-step and tell me the answer in the end" after the input to prompt LLMs.
Summarization. The XL-Sum* (Hasan et al., 2021) (250 test samples randomly sampled from XL-Sum per language) benchmark contains long news article and wants the model to summa- rize it into short text. It covers 44 languages, and we choose subset of 6 languages, including en, es, fr, tr, vi, and zh. We utilize the ROUGE-1 score (Lin, 2004) for evaluation.
Translate-Englian leverages the robust capabili- ties of LLMs in English to tackle multilingual tasks, as suggested by both Shi et al. (2023) and Ahuja et al. (2023). This approach translates instances from other languages into English beforehand. In practice, utilize the Google Translate API translate examples into English and apply the basic prompt to format them. Note that, we do not apply this method generation tasks since they require the output in respective language rather English.
Machine Translation. The FLORES (Costa- jussà et al., 2022) (200 test samples randomly sampled from FLORES- 200 per language) bench- mark contains parallel text from Wikimedia projects for 204 languages, yielding over 40,000 translation directions. We choose a subset of 12 directions, including high resource to high re- source translation (i.e., zhe ru and vi), high
XLT utilizes the proposed template consisting of multiple instructions introduced in Section 2. The

---

instantiated XLT templates for each benchmark are listed in Table 6.
may not work well in the few-shot settings. Over- all, compared to the three baseline methods, XLT achieves significant improvements over two LLMs for all tasks on both zero-shot and few-shot settings regardless of the language difference, except for slight drop on the PAWS-X X benchmark in the zero-shot setting. It is noted that XLT achieves remarkable gains of nearly 20 points on average in the MGSM benchmark for the arithmetic reason- ing task and around 10 points on average in the MKQA benchmark for the open-domain question answering task. The experiments demonstrates the effectiveness of XLT for empowering LLM with multilingual capability.
In seo-shot learning scenarios, for basic prompt, we use the same template as an additional input to the model. For XLT, we provide the exemplars with XLT template inputs and anticipate desirable step-by-step outputs as outlined in Figure 4. In the subsequent evaluation, apply the 5-shot setting, except for the XL.-Su** experiments, which use the 3-shot setting due input length constraints.
3.1.3 LLMs
We mainly evaluate two LLMs from the GPT-3.5 series models:
As for the compatibility test, list the results of LLaMA-2-Chi on the MGSM benchmark in Table 7. is notable that LLaMA 2 can also ben- efit from our cross- pouus--lnguullthhoe demonstrates the generality of our XLT template. However, the gains of LLaMA-2-Che is not good as GPT-based models. Our analysis reveals this gap can primarily be attributed to LLaMA 2's poorer multi-step instruction-followie followwng ability.
text-davinci-003f is trained using instruction tuning and reinforcement learning from human feedback (Ouyang et al., 2022). It can perform a wide range of natural language tasks with satis- factory results.
which further
3ptt3.5-turbo' is optimized for chat based on text-davinci-009 and suitable for traditional NL.P tasks. It is the most capable GPT-3.5 model.
To verify the compatibility of our XLT template, we further incorporate LLAMA-2-Chat (Touvron et al., 2023) (Llama-2 2-00a---77--catt-h1 as our base models. It is an open-source model that has been trained through supervised fine-tuning and rein- forcement learning from human feedback the base LLaMA model. In addition, also re- fer to the existing results from other LLMs, such as waw-ddiviiiieee-- when the evaluation is comparable. During inference, we employ greedy search (i.e., temperature=0) to generate the LLM re- sponses. We find LLMs have excellent instruction- following abilities to respond to our instructions in the given format. Therefore, we just extract the part after "Answer format:" as labels.
I Language Democratization. Furthermore, we try to assess the democratization degree of tasks between languages by defining "democratization score", which calculates the average percentage of performance attained by different languages rela- tive to the best performance among languages. Given the evaluation scores of 81; $2,..., corre- sponding to language on task, the democratiza- tion score is formulated as:
E/n 8i 1
max
(1)
Table 2 presents the degree of democratization for tasks across languages under both zero-shot learning and few-shot learning, and we further sum- marize it in Figure 1(b) by averaging all scores per task regardless of the setting and model differ- ences. We can observe that XLTT leads to higher democratization scores in general, particularly for XCOPA, and MKQA. As for MGSM, XNLI, and PAWS-X, our XLT can improve performance in multiple languages, where the overall performance of the baseline is consistently lower but the gap be- tween languages is smaller as shown in Tables 7, 9, and 10. In conclusion, our method can reduce the performance gap between languages and improve the language democratization of LLMs.
3.2 Experimental Results
Multilingual Capability. We comprehensively evaluate XLT's performance over seven tasks. The average score of text coeet-dvvnnii-eee -003 is summa- rized in Figure 1(a) and Table 1, and more details are listed in Appendix A. As for the CoT prompting, it can enhance reasoning tasks while becomes less effective on understanding and generation tasks. In terms of the Translate- En prompting, it can boost the performance in the zero-shot settings while
Phttps://platfot openai. cww/doccs/mmmlss/....... gpt-3-5

---

Reasoning
Understanding
Generation
Settings
MGSM
XCOPA
XNLI
PAWS-X
MKQA
XL-Sum*
FLORES
text-davinci-oee
Basic Prompt
12.5
70.1
53.3
52.0
29.0
23.7
15.4
CoT
25.7
70.9
53.0
57.8
30.9
23.8
15.8
Translate-En
15.7
68.0
54.8
55.0
XLT
23.9
73.3
62.4
57.1
40.2
25.2
17.7
7cro-shot
gpt-3.5-turke
Basic Prompt
23.3
76.9
52.6
65.5
31.6
24.7
CoT
45.5
78.3
54.8
61.0
25.4
19.7
Translate-En
27.1
75.7
52.2
66.8
XLTT
70.0
80.3
65.5
63.6
42.7
26.1
21.2
cext-davinci-00
Basic Prompt 45.5
75.6
59.1
68.7
39.1
26.8
Translate-En 46.5
77.4
56.9
68.5
XLT
55.4
81.3
67.5
72.2
49.6
27.3
Few-shot
gpt-3.5 3.-3..-turbb
Basic Prompt 63.0
80.1
61.4
66.4
43.7
25.5
Translate-En
65.1
81.9
58.3
63.7
XI.T
72.5
85.9
65.0
69.1
52.5
27.9
Table The average scores in different languages for the seven benchmarks in zero-shot and few-shot settings. We omit the results (denoted as "-") of Translate-Ei En since it is not applicable for generation tasks.
Reasoning
Understanding
Generation
Ablating the instructions. Since our XLT con-
Settings
XNLI
MKQA
sists of six logical instructions, we disable the
Zero shot setting
Role Assigning, Cross-lingual Thinking, and CoT
taxt-davinc nc--88
Basic Prompt
65.2
77.8
83.8
97.1
60.2
Task Solving instructions separately to analyze the contribution per instruction.
CoTT
89.5
Translate-En
77.2
78.7
86.0
95.3
51.6
XLTT
68.5
82.1
80.7
88.4
78.7
Reordering the instructions. Considering the logicality of our instructions, we further change the order of the instructions XLT to explore whether LLMs will handle tasks differently and lead to different results.
2pt-3, -pt-3_5-turbo 5-turbo
Busic Prompt
73.0
83.6
80.5
89.0
61.8
CoT
80.7
Translate-En
80.4
84.6
79.8
90.7
54.1
XLT
75.3
Few-shot semtig
text-davinci-See
Basic Prompt
75.4
82.0
82.5
88.2
74.3
Changing the content word. As prompts are usually sensitive to the word choice, we verify the robustness of XLT when alternating the rephras- ing keyword with "retell", "repeat", and "trans- late" in the cross-lingual thinking instruction.
Trunslate-En
77.1
82.6
79.5
87.8
68.5
XLT
84,5
85.6
85.3
91.6
82.7
S-tuurbo
Basie Prompt 76.1
84.1
83.6
94.4
82.1
Trunslate-En
78.6
86.4
79.2
95.4
71.3
84.3
The outcomes are presented in Table 3, indi- cating that XLT surpasses almost all the variants, thereby validating the effectiveness and reasonable- ness of our proposed XLT method.
Table 2: The democratization degree of tasks against languages.
3.3 Further Analysis
The effectiveness of each instruction. The re- sults from the "Instruction Ablation" row indicate that: (1) Cross Cross-lingual Thinking yields more signifi- cant gains compared to other instructions. This sug- gests that the LLM's ability of cross-lingual think- ing activated, allowing it to utilize its knowledge in English to solve tasks effectively; (2) Remov- ing Role Assigning from XLT impedes the model's
In this section, further investigate the factors that affect the performance of XLT and how they affect various multilingual benchmarks.
3.3.1 Ablation of XLT
For the XLT variants, we mainly conduct experi- ments to compare the following strategies:

---

MGSM
XNLI
FLORES#
Settings
de
hi
jv-bzh
zh jh-jj
XLT
79.8
72.6
61.3
64.8
19.0
10.5
w/o Role Assigning
76,6
69.2
57.8
63.9
16.2
8.8
Instruction Ablation
w/o Cross-lingual Thinking
75,6
62.0
56.1
62.2
13.2
8.2
w/o CoT Task Solving
77.0
68.0
62.9
65.2
16.8
9.2
Swap Role Assigning and Task leputting
77.2
71.8
54.2
61.5
19.6
11.2
Instruction Order
Swap Role Assigning and Task Analyzing
76.8
70.8
61.0
64.0
15.8
8.8
Swap Cross-lingual Thinking and Task Analyzing
79.0
71.2
59.5
63.4
16.5
9.7
retell
79.8
72.6
61.3
64.8
18.2
10.3
Rephrasing Word
repeat
77.6
60.7
64.6
10.5
translate
76.4
70.0
60.1
64.5
10.2
Table 3: Performance comparison across different variants of XLT. All the experiments are conducted using gpt-3. 5-turbo under the zero-shot setting.
Demonstration format
en
de
ru
fr
es
ja
SW
th
te
Avg.
Basic imput Basic output
84.0
79.2
78.8
78.8
70.8
81.2
68.8
70.8
68.8
65.2
44.8
71.9
Basic imput XLT output
82.4
72.4
71.2
75.2
64.4
78.8
63.2
66.8
53.6
54.8
32.4
65.0
XIT imput XLT output
84.8
81.4
80.2
79.2
71.8
81.6
72.8
71.2
69.8
64.4
40.8
72.5
Table Performance comparison actoss different few-shot variants on the MGSM benchmark. All the experiments are conducted with 5 demonstrations using 30tt3.5-turrte
understanding of the ultimate goal for diverse mul- tilingual tasks, highlighting the task transferability of XLT; and (3) the better performance of XLT can also be attributed to CoT Task Solving, which re- quires the model to respond complex instructions in step-by-step manner.
performance in their tasks.
3.3.2 Effectiveness of XLT Few-shot Learning As mentioned in Section 2.2, the construction of demonstrations for XLT few-shot learning differs from the previous method. We have compared XLT and basic prompt. Here, focus the construc- tion of the demonstration input-output pairs and compare various demonstrations that may be used to perform XLT few-shot learning. The illustrations can be found in Figure
The order of logical instructions. The perfor- mance drop is evident when the order of our de- signed logical instructions is switched. When de- signing XLT, have taken into account the pro- cess by which humans solve multilingual problems, and this experiment further confirms the optimum order of our XLTT template. Placing the Role As- signing instruction later may confuse the model initially. Additionally, conducting Cross-lingual Thinking before Task Analyzing is crucial since we rely on the English task-solving abilities of LLMs to handle multilingual tasks.
Basic prompt input + Basic prompt output:
This is the normal demonstration format used most of the previous work.
Basic prompt input XLT output: This abla- tion separate the effect of input and output formats in the demonstration.
XLT input + XLT output: This is the method that we used in this work.
The robustness of word choice for rephrasing keywords. We can find that different words in- deed affect the performance of XLT, but it is less sensitive to the other variants. Through experi- mentation, we have determined that "repeat" yields better results for text summarization and machine translation, while "retell" is more suitable for the remaining five tasks. Our aim is to provide XLT with more unified template, while still allowing users to fine-tune specific keywords for optimal
Observing the experimental results presented Table 4, we can conclude that: (1) Our XLT few- shot learning outperforms all other variants, thus confirming its effectiveness. (2) The use of normal demonstrations for XLT few-shot learning leads decrease performance. (3) Merely incorporat- ing XLTT as a demonstration input without its output does not result in any improvements. (4) Consis- tency in the demonstration for few-shot learning

---

is crucial, implying that the demonstration input- output format should align better with its zero-shot learning input-output format.
multilingual processing (Costa-jussà al., 2022), and the other is training a pre-trained model and using fine-tuning to transfer knowledge among lan- guages to achieve multilingual capability (Chen et al., 2021, 2022). However, the emergence of LLMs has made it possible to directly process multilingual tasks via in-context learning (Brown et al., 2020; Ahuja al., 2023). These LLMs, with hundreds of billions rameters, require significant amount of compu- tation resources for training, making traditional fine-tuning methods less feasible. To improve the generative ability of LLMs, researchers explore in-context learning methods that do not require up- dating model parameters, such as few-shot prompt- ing (Vilar et al., 2022), automatic prompt learn- ing (Shin et al., 2020), task-instruction prompt- ing (Ye et al., 2023), chain-of-thought prompt- ing (Wei et al., 2022c), etc. Our work builds upon these methods and proposes an optimized, generic, and baaguage-indeendern prompt to enhance the multilingual capability of L.Ms..
4 Related Work
4.1
LLM Capability Understanding
Despite the impressive capabilities of LLMs, it is crucial to determine their impact on natural lan- guage processing tasks. Liang et al. (2022) con- duct a comprehensive evaluation of LLMs from various perspectives, such as accuracy, calibration, robustness, fairness, bias, toxicity, and efficiency. Bang et al. (2023) extensively evaluate the Chat- GPT model on multiple natural language process- ing tasks and find that the model performs well in high-resource languages but exhibits certain lim- itations in low-resource and non-Latin script lan- guages. Additionally, studies by Jiao al. (2023) and Hendy et al. (2023) compare different GPT models with supervised models for machine trans- lation tasks and find that GPT models have com- petitive translation abilities in high-resource lan- guages but perform less effectively in low-resource languages. It is worth noting that achieving multi- lingual generative AI capability necessitates cross- lingual knowledge to further improve the model's performance. In this context, Ahuja et al. (2023) evaluate the multilingual task understanding abil- ity of GPT models and attempt to enhance their task processing abilities in other languages using English knowledge. Our work also focuses on eval- uating the multilingual capabilities of LLMs, in- cluding reasoning, understanding, and generative capabilities. Our evaluations indicate that LLMs ex- hibit differences in high-resource and low-resource abilities, which necessitates additional efforts to enhance their multilingual capability.
trillions of pa-
Conclusion
This work investigates the language processing ca- pabilities of large language models in multilingual settings and expects to develop a universal frame- work for handling diverse multilingual tasks. To accomplish this goal, we propose a generic prompt, referred to as XLT,, to enhance the multilingual ca-
pability and reduce the performance gaps among languages in tasks related to language understand- ing, reasoning, and generation in non-English and low-resource languages. Although our method is generally applicable across tasks and languages, discovered that prompting design factors such as instruction logic and word choice have explicit impacts on its effectiveness. Cross-language think- ing in XLT is particularly effective. Finally, we hope this work can inspire further research to pri- oritize the development of generic prompting. By doing so, large language models can encompass a wider range of modalities and languages.
4.2 Multilingual Task Processing
Multilingual knowledge has been shown be ex- ploitable and transferable between languages to im- prove model performance (Devlin et al., 2019; Con- neau et al., 2020; Raffel et al., 2020; Ouyang et al., 2021; Chi et al., 2021). While much research has been devoted to multilingual understanding tasks, multilingual generation tasks are more challeng- ing, particularly when the target language is low- resource or non-Ennglish (Ma al., 2021; Liu et al., 2020). Two methods can enable models support multilingual task processing: one is training a su- pervised model that covers multiple languages for
A Acknowledgements
Tiani Tang and Xin Zhao are supported by Na- tional Natural Science Foundation of China un- der Grant No. 62222215, Beijing Natural Sci- ence Foundation under Grant No. 4222027 and L.233008.

---

Limitations
for Computational Linguistics (Volume 1: Long Pa- pers), pages 142-157, Dublin, Ireland. Association for Computational Linguistics.
Due to limitations imposed by the evaluation bench- marks and OpenAl APT cost, we conducted tests on 27 languages, which merely seratch the surface of the vast array of languages in the world. Be- sides, our XLT template is based on English. It deserves to explore whether the template written in task language can lead to better performance and how better construct the instruction in each language. Furthermore, we only verify the effec- tiveness of our method on two GPT-based mod- els (i.e,, text-davinci-000 and gpt-3. 3pt-3.5-turdee 5-turbo) and LLaMA-2-Chat. It is worthwhile to investi- gate the generality of our template on more models, such as BLOOM and PaLM.
Zewen Chi. Li Dong, Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXIMM: An infformation-theorete framework for cross-lingual language model pre-training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 3576 -3588, On- line. Association for Computational Linguistics.
Aakanksha Chowdhery. Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint wrXiv:2204.02311.
References
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Mylc Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Kabir Ahuja, Rishav Hada. Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tamuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023. Mega: Multilingual evaluation of generative ai. arXiv preprini arXiv:2303. /2528.
Yejin Bang. Samuel Cahyawijaya, Nayeon Lee, Wen- liang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al. 2023. A multi- task, multilingual, multimodal evaluation of chatgpt on reasoning. ballucination, and interactivity, arXiv preprint 2332::230..044: 04023..
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Proceedings of
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Marta R Costa-jusså, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centerge machine translation. arXiv preprint arXiv:2207.04672
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186. Minneapolis, Minnesota. Association for Computational Linguistics.
Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, and Furu Wei. 2021. Zero-shot cross-lingual transfer of neu- ral machine translation with multilingual pretrained encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 15-26, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Robert Forkel et al. 2022. Glottocodes: Identifiers link- ing families, languages and dialects to comprehen- sive reference information. Semantic Web, 13(6):917- 924.
Guanhua Chen, Shuming Ma. Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, and Furu Wei. 2022. Towards making the most of cross-lingual transfer for zero-shot neural machine translation. In Proceed- ings of the 60th Annual Meeting of the Association
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Is- lam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang,

---

M. Sohel Rahman, and Rifat Shahriyar. 2021. XL- sum: Large-scale multilingual abstractive summariza- tion for 44 languages. In Findings of the Association for Computational Linguistics: ACL-UJCNLP 2021, pages 4693-4703, Online. Association for Computa- tional Linguistics.
Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- rics, 8:726-742.
Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. MKQA: A linguistically diverse benchmark for mul- tilingual open domain question answering. Transac- rions of the Association for Computational Linguis- fics, 9:1389-168 1406.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language under- standing. In Interaational Conference on Learning Representations.
Shuming Ma. Li Dong. Shaohan Huang. Dong- dong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021. Deltalm: Encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders. arXiv preprint arXiv:2106.13736
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Atify, and Hany Hassan Awadalla. 2023, How good are gpt models at ma- chine translation? a comprehensive evaluation. arXiv preprint 2302.0000000.00
OpenAL. 2023. Gpt-4 technical report. arXiv.
Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt good translator? yes with gpt-4 as the engine. arXiv preprint arXi::2301.0877
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730-27744 Curran Associates, Inc.
Jared Kaplan, Sam McCandlish, Tom Henighan. Tom Brown, Benjamin Chess, Rewon Child, Scout Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arxi1v2001
Viet Dac Lai. Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernnncuur, Trung Bui, and Thien Huu Nguyen. 2023. Chatgpt beyond en- glish: Towards a comprehensive evaluation of large language models in multilingual learning. arXiv preprint 2xXx:2304.05611.
Xuan Ouyang, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. ERNIE-M: Enhanced multilingual representation by aligning cross-lingual semantics with momolingual corpora. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 27-38, Online and Punta Cana, Dominican Re- public. Association for Computational Linguisties.
Percy Liang Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wa, Ananya Ku- mar, et al. 2022. Holistic evaluation of language models. arXiv preprini arXiv:2211.08 1.09110....1.....111111111111111
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-318, Philadelphia. Pennsylvania. USA. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulié, and Anna Korhonen. 2020. XCOPA: multilingual dataset for causal common- sense reasoning. In Proceedings of the 2020 Con- ference on Empiricul Methods in Natural Language Processing (EMNLP), pages 2362-2376. Online. As- sociation for Computational Linguistics.
Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Oit, Na- man Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer. Punit Singh Koura, Vishrav Chaudhary, Brian Horo, Jeff Wang, Luke Zettle- moyer, Zornitsa Kozareva, Mona Diab, Veselin Stoy- anov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9019-9059 Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Matt Post. 2018 A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference Machine Translation: Research Papers, pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi

---

Zhou, Wei Li, and Peter J. Liu, 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b. Emer- gent abilities of large language models. Transactions on Machine Learning Research. Survey Certifica- tion.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafcai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-sho shot task gener- alization. In International Conference on Learning Representations.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi. Quoc V Le. and Denny Zhou. 2022c. Chain-of-thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824-24837. Curran Associates, Inc.
Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: cross-lingual adversar- ial dataset for paraphrase identification. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-UCCNLP) pages 3687-3692, Hong Kong. China, Association for Computational Linguis- tics.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni. François Yvon. Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint 2rX05:221..051000
Seonghyeon Ye, Hyeonbin Hwang, Sohee Yang. Hyeongu Yun, Yireun Kim, and Minjoon Sco. 2023. In-context instruction learning. arXiv preprint 23i2:2302.14611
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi. Hyung Won Chung, Yi Tay. Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multi- lingual chain-of th-uuuhtt reasoners. In The Eleventh International Conference Learning Representa- tions.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint 2.Xi:233331822..
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Learning, volume 139 Proceedings of Machine Learning Research, pages 12697-12706. PMLR.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Elic- iting Knowledge from Language Models with Auto- matically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong. Jiajun Chen, Lei Li, and Shujian Huang. 2023. Multilingual machine translation with large language models: Empirical results and analy- sis. arXiv preprint arXiv:2304.0440
Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almabairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open founda- tion and fine-tune tuned chat models. arXiv preprint arXiv:2307.09288.
David Vilar, Markus Freitag, Colin Cherry, Charry.Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompt- ing palm for translation: Assessing strategies and performance. arXiv preprint arXiv:2211.09100
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022a. Finetuned language models are zero-shot learners. In International Con- serence Learning Representations.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama,

---

A Additional Experiments
languages, particularly for low-resource languages under the few-shot setting. We observe slight performance drop on average zero-shot learn- ing compared to 5-t-3.--turbo for some high- resource languages (e.g., en, de, and fr). Based on our analysis of intermediate outputs, we infer that the drop in performance may be due to cross- lingual thinking that alters the original meaning of the two sentences, leading to difficulties in judg- ment. Additionally, comparable pattern evident in a previous study (Ahuja al., 2023), where non-Latin script languages (ja, zh, and ko) exhibit significantly poorer performance than English or German in the few-shot setting. Nevertheless, by demonstrating the construction of XLT, we can guide the model on how to think across different languages and effectively address the aforemen- tioned issues.
A.1 Results on Reasoning Tasks
Table 7 presents the results of the MGSM bench- mark. XLT significantly improves the arithmetic reasoning capabiliities of both models, particularly for 3pt-3.5-turbo in the zero-shot setting. We hypothesize that gpt-3. turbo may have under- gone supervised fine-tuning (Ouyang et al., 2022) with arithmetic reasoning samples in the chain-of- thought format, which enables XLT to activate its arithmetic reasoning ability directly. For both low- resource languages (e.g., sw, th, bn, and te) and high-resource languages, XLT can further enhance the performance. Even under the few-shot setting, XLT can still significantly improve the reasoning performance of both models and reduce the perfor- mance gap for all languages. Notably, for some high-resource languages, such as de, ru, fr, and es, the performance is comparable to English.
A.3 Results on Generation Tasks
The XCOPA benchmark results are presented in Table 8. Our XLT approach significantly en- hances the performance of both models in both settings, as compared to basic prompting. In the zero-shot setting, XLT demonstrates significant im- provements for relatively low-resource languages (e.g., sw, th, ct, ta, and ht), but it underperforms the baseline for some high-resource languages such zh and it. In the few-shot setting, XLT brings enhancements for both high- and low-resource lan- guages. Our findings suggest that XLT is more effective for low-resource languages, particularly for 3.t-3.50 t-turbo on sw, th, ta, and ht, where yields improvements of over 10 accuracy points.
The MKQA benchmark outcomes are listed in Ta- ble 11. Across all languages in the zero-shot and few-shot settings, the XLT template shows sig- nificant improvement over the basic prompt. It is worth noting that text-davinci-000 performs worse than gpt-3. 3pt-3.5-turbo in this task, and we speculate that the latter is optimized for open ques- tion answering. which is common in daily chat. Additionally, our findings indicate that XLT can no- tably enhance the performance of under-resourced languages. XLT brings over 10 points of improve- ment for these languages. (e.g.. zh, ja, vi, and tr) This aligns with previous benchmarking studies and is particularly noteworthy in this evaluation. We suspect that high-resource and low-resource languages share the same cross-lingual thinking as English to greatly leverage the LLM'' ability to solve English open open-domain QA.
A.2 Results on Understanding Tasks
Table 9 presents the results of the XNLI benchmark. In the zero-shot setting, our XLT significantly out- performs the basic prompt in all languages. Addi- tionally, when using few-shot setups high- and low-resource languages, both text-davinci-e00 and gpt-3.5-turbo 5-turbo show significant improve- ments compared to the basic prompt. Specifically, for low-resource languages such as th, bg, hi, and ur. XLT achieves an average improvement of 9.4 accuracy scores for accuracy scores for gpt-3. 5-t.rb..tbbb...oo. This demon- strates that XLT is effective for both models, but text-davinci-003 has better natural language in- ference capabilities.
The results of the XL-Sum* benchmark are pre- sented in Table 12. can be observed that XLT outperforms the basic prompt in both zero- and few- shot settings across all languages. Additionally, the LLM model exhibits a significant improvement in generating summaries under the few-shot setting compared to the zero-shot setting. This suggests that providing fewer examples effectively guide the model in summarizing multilingual texts. Fur- thermore, the few-shot results revealed an interest- ing finding that text-davinci--eee performed het- ter when g.t-3.5-t-- 5-turbo and text text-davinci -003 use basic prompt. However, once XLT is enabled,
-003 and 5.3
Table 10 displays the comparisons on the PAWS- X task, where XLT outperforms basic prompt in all

---

Table 5: The basic prompt each benchmark. #Test denotes the number of instances in the test set.
Benchmark
#Test
Basic Prompt
MGSM
250
Request: {problem)
XCOPA
500
Here is premise: (premise). What is the (question)? Help me pick the more plausible
option: -choicel: {choice1), -choice2; (choice2)
XNLI
5,010
(premise) Based on previous passuge, it true that (hypothesis}? Yes, No, Maybe?
PAWS-X
2,000
Sentence I: (sentencel) } Sentence 2: (sentence2) Question: Does Sentence paraphrase Sentence 2? Yes or No?
MKQA
6,758
Answer the question in one or a few words (target_languagey) (question)?
XI.-Sum*
250
Summarize this article: (article)
FLORES
200
[source) Translate from Stoorre__langugee to (taareet_aannga
Table 6: Task meta data consisting of task name, input tag, task goal, output type, and output constraint per benchmark. Detailed examples of the input for each benchmark are listed in the following part.
Benchmork
Tack nome
Input fag
Task goal
Output type
Output constraint
MGSM
atiilteeti retson-
soquest
do step-by- step answer to obtain number answer
Ing
commonsensee
premise and up- Gous
do step by step answer to pick choice
choice number
soning
XNLI
language
hyposhesis and premise
julge wheterr the hyputheeis falso, uedlttermieed given premise. relationsiip
rolutionsip
ference
fillmni contexdiatioon nestral
PAWS-XX
jumphesse idenitfi
proside a yes
the question: Doies Sentence
choosing cither yes 6o
cation
paraphruse Senteace 2?
MKQA
qusstion
question
few words
in few wards [taggtt languge]
XI. Sum
multiliguul sum-
eniree
think by--sepp summorice entine lest in maximum
senimoor in tagget Jangungs]
FLORES
machine
Iattlaa-
souice schtence
provile the Itarget Jenguage] trunslttoon for the English
tugget translation
3-t-ubb- 5-turbo outperforms text text-davinci-003 highlighting the effectiveness of our approach.
mance of gpt-3. 3.t turbo by nearly 4.0, 2.8, and 3.3 BLEU points for th th->gl, /y-,zh, and zh -tth translations, respectively, demonstrating its effee- tiveness regardless of whether the source language is high-resource low-resource. Noticing that Hendy al. (2023) have shown that few-shot con- figurations do not yield significant improvements over the zero-shot setup for translation tasks, we do not evaluate the few-shot paradigm on FLORES# in this work and leave for future exploration.
Machine translation is special generation task where the source and target are two different lan- guages. The experiment in this part is to verify how XLT boosts machine translation tasks. Since En- glish has been specified as the pivot language in the cross-lingual thinking in XLT, we exclude English- centric tasks avoid language redundancy and fo- on 12 Enn-lsh translation directions in the FLORES# benchmark, which includes both high- resource and low-resource languages. As shown Table 13, XLT achieves impressive zero-shot re- sults for all languages compared with basic prompt. For example, it significantly improves translation quality in Chinese-to- X or to-Chinese. The re- sult emphasizes that XLT will potentially transfer the knowledge of high-resource pivot language like English to the target language. While the bene- fit of XLT may not be as obvious for high-to-high translations, it becomes more significant for high- to-low, -high, and low-to-low translations. For instance, XLT improves the translation perfor-

---

Settings (high-WWi -low
en
de
ru
fr
zh
es
ja
SW
th
bn
te
Avg.
text-davinci-081
Basic Prompt
19.2
12.8
16.4
15.2
12.8
7.2
8.8
11.6
4.4
XLT
30.0
32.4
23.6
34.8
29.2
26.8
26.0
13.6
14.8
12.8
23.9
gpt-3.5-turbo 5-tubb
Zero-shot
Basic Prompt
32.0
24.8
28.0
31.6
22.0
29.2
22.4
24.4
16.8
18.0
7.6
23.3
XLT
84.4
79.8
77.6
75.2
72.6
76.8
71.0
70.8
63.8
56.8
42.0
70.0
LLama-2-70ba 2-70b-chat-hr
Basic Prompt
58.8
48.0
47.2
45.6
39.6
50.4
39.2
10.0
13.6
17.2
5.2
34.1
XLT
60.0
52.8
52.8
48.8
42.4
52.0
39.2
16.4
18.0
17.6
10.4
37.3
code-davinci-002 (Shi et al., 2023)
53.6
46.4
48.8
46.4
47.2
51.6
44.8
37.6
41.2
41.2
42.8
45.6
text-davinci-088
Few-shot
Basic Prompt
60.4
45.6
51.6
45.6
38.8
51.6
37.6
48.8
30.4
43.6
46.8
45.5
XITT
65.6
58.0
57.6
56.8
53.2
58.0
54.4
58.8
42.4
53.2
51.8
55.4
gpt-3.5-turbe
Basic Prompt
82.8
69.2
71.6
72.4
46.8
71.2
56.0
60.0
44.0
62.4
56.6
63.0
XLT
84.8
81.4
80.2
79.2
71.8
81.6
72.8
71.2
69.8
64.4
40.8
72.5
Table 7: Accuracy scores the MGSM benchmark. Shi et al. (2023)"
utilize 6-shot learning.
Settings (high >low)
zh
it
vi
tr
SW
et
ta
ht
qu
Avg.
text-davinci-083
Basic Prompt
85.4
90.0
69.2
80.6
83.8
56.4
66.6
73.0
53.4
61.6
50.4
70.1
XLT
85.8
89.2
76.0
81.0
86.4
59.2
67.2
83.4
55.2
72.2
50.2
73.3
Zero-shol
gpt-3 3pt-3.5-turbo 5-turbo Basic Prompt XLT
90.4
92.0
83.6
86.6
88.2
77.0
70.2
84.0
57.2
65.2
51.2
76.9
87.8
89.8
87.5
90.2
89.5
82.0
78.0
88.4
64.0
74.6
51.8
80.3
code-davinci-002 al., 2023)*
96.6
86.6
91.2
84.2
88.8
55.8
79.6
text-davinci-003
Few-shol
(Ahuja al., 2023)
94.6
89.8
93.0
82.8
84.8
89.6
87.0
82.8
Basic Prompt
90.8
92.2
80.2
85.2
90.8
63.6
69.2
81.8
53.6
73.2
51.0
75.6
XLT
94.0
95.0
87.0
94.0
92.8
68.4
79.4
90.4
59.4
80.8
53.0
81.3
gpt-3.5-ture Prompt
91.0
95.2
86.2
89.0
79.2
73.6
92.0
58.6
74.2
53.0
80.1
XLT
92.8
95.8
90.6
92.2
90.2
85,2
93.0
70.8
86.0
56.2
Table Accuracy scores on the XCOPA benchmark. (Shi et al., 2023)" utilize 8-shot learning.
utilize 6-shot learning. Ahuja et al. (2023)*
Settings (high-loww)
de
v1
th
hi
text-davinci-883
Basic Prompt
shot.
XLT
77.4
67.7
64.2
68.3
64.8
69.
62.0
61.5
54.3
58.7
61.1
56.3
62.6
55.1
53.0
62.4
Not-3.5-turbo Basic Prompt
65.4
$5.5
50.6
53.2
48.8
59.8
$2.1
$4.4
49.6
50.9
$4.9
44.8
55.7
49.2
44.8
52.6
XLT
74.4
68.5
66.0
69.8
64.9
69.4
64.8
65.0
60.1
62.8
68.3
62.1
67.7
61.3
57.3
65.5
text-davinci-en
(Ahuja al.,, 2023)
55.
Basic Prompt
71.6
65.88
62.5
63.4
56.7
6416
59.4
56.9
48.2
$7.3
62.0
55.0
62.6
52.4
48.0
59.1
Fow-sbot
XLT
79.1
70.8
70.0
69.5
69.2
71.0
67.3
66.9
59.5
65.7
67.8
63.7
70.4
63.5
58.
67.5
3.t-3.55 tutuba
Basic Prompt
66.3
60.2
52.7
53.8
XLT
77.1
69.3
64.4
69.6
62.9
70.6
63.2
64.4
60.2
63.4
66.6
59.8
66.9
60.0
56.5
65.0
Table 9: Accuracy scores on the XNLI benchmark. Ahuja al. (2023)t utilize 8-shot learning.

---

Settings (high +low)
en
de
es
ja
ko
Avg.
text-davinci-600
Basic Prompt
53.3
52.9
50.8
53.6
53.0
50.3
50.3
52.0
XLT
56.0
55.3
57.4
56.0
55.9
57.1
Zero-shot
t-turbo Basic Prompt
73.6
68.3
68.4
63.4
69.6
59.7
55.7
65.5
XI.T
65.3
66.1
64.8
65.5
63.3
62.4
57.6
63.6
foxt-davinci-001
(Abuja al., 2023)*
72.5
69.8
71.3
65.2
70.1
65.4
65.8
68.6
Basic Prompt
77.8
70.6
72.5
65.0
71.7
62.5
60.5
68.7
Few-shot
XLT
76.5
78.8
77.4
61.1
78.0
65.0
68.7
72.2
gpt-3. 5- turbo
Basic Prompt XLT
65.9
70.3
66.6
64.1
68.2
65.6
64.0
66.4
73.4
69.8
68.5
70.9
67.8
66.4
66.7
69.1
Table 10: Accuracy scores on the PAWS-X benchmark. Ahuja al. (2023)
utilize 8-shot learning.
Settings (high loww
en
de
ru
zh
ja
th
Avg.
text-davinci-000
Basic Prompt
48.1
33.8
15.9
34.8
18.2
34.1
27.7
23.6
24.0
29.6
29.0
XIT
51.1
42.3
27.3
43.0
36.7
43.3
46.8
35.8
37.8
38.1
40.2
Zero-shot
8pt-3.5-turbo
5-turbo
Basic Prompt
51.1
40.6
28.4
40.1
16.5
39.3
25.9
23.3
26.9
23.7
31.6
XLT
56.7
46.0
33.9
47.6
33.0
47.9
47.5
36.5
39.1
38.6
42.7
text text-davinci-O0
Basic Prompt
52.6
42.3
21.8
42.9
33.1
42.8
45.5
35.5
37.5
36.6
39.1
XLT
57.6
49.4
42.7
50.9
51.0
50.0
60.0
46.9
46.9
40.5
49.6
Few-shot
gpt-3. 3pt-3.55turbe 5-turbo Basic Prompt
53.2
48.6
31.0
46.1
40.9
47.9
51.4
38.5
40.0
39.3
43.7
XLT
59.6
52.5
43.8
53.9
51.9
54.0
63.2
49.4
52.
44.7
52.5
Table 11: F1 scores on the MKQA benchmark. The average score is the macro average F1 score.
Settings (high-low)
fr
zh
vi
tr
Avg.
text-davinci-600
-003
Basic Prompt
22.2
26.2
30.8
25.1
22.0
15.9
23.7
XLT
24.4
28.2
32.2
26.0
22.3
17.9
25.2
Zero-shot
gpt-3.5 t.ttubbb
(Lai et al., 2023)
19.7
20.8
21.1
17.8
14.5
Basic Prompt
25.3
26.2
30.2
26.3
21.1
19.2
24.7
XLT
26.8
28.1
33.3
26.4
21.3
20.5
26.1
text-davinci -00
Basic Prompt
29.6
22.5
18.1
XLT
28.2
30.3
34.4
29.4
22.7
18.6
27.3
Few-shot
got-3.5 3.5-turbo
Basic Prompt
25.7
27.2
30.8
27.8
21.5
19.7
25.5
XLT
28.5
29.2
35.0
28.6
23.7
22.3
27.9
Table ROUGE-1 scores on the XI--Sum* benchmark.

---

High- High
High-Low
Low-Low
Settings
zh-ru
de-vi
zh-th
zh-jv
th-gl
jv-th
+
text-davinci-008
Basic Prompt
19.8
24.2
26.5
24.5
10.2
11.8
8.1
14.0
17.9
12.0
10.0
6.2
XLT
21.6
24.8
27.4
24.8
12.6
16,4
11,1
18.2
20.7
14.2
11.7
9.0
Zero-shot
got-3. 3.5- t-rurbo
Basic Prompt
23.3
25.4
34.1
29.6
16.6
18.6
9.1
16.2
18.1
18.5
13.0
7.2
XLT
25.3
25.6
33.3
31.3
19.9
19.3
10.5
19.0
22.1
21.9
15.9
10.6
Table 13: BLEU scores on the FLORES benchmark.
want to act
Prompt
want to Request: T3 A 5 1
Request: : a 5 po
You should tell the
Request:
5
should tell the
Retelling the request
5x
AM At #nswer
3 tt
Answer:
Step-by-step answer:
5
11.
$ 5
2 11.
Answer: 11
(a) Basic prompt input and output
(b) Basic prompt input and XLT output
(c) XLT input and output
Figure 5: Illustrations of different demonstration onput- pairs in the few-shot learning.

---

Input:
Request:
# P 2 PU
TE # E-
text-davinci-003 output:
Answer:
gpt-3.5-10 t-rrbb output:
Answer:
Figure 6: A Chinese example MGSM benchmark using basic prompt and the corresponding outputs under the zero-shot setting.
Input:
want you to act as an arithmetic reasoning expert for Chinese
Request: HH/ 7 2 DC OH
A #
#6 $
You should retell the request in English.
You should do step-by-step answer to obtain number answer
You should step-by-step answer the request.
You should tell me the answer in this format
Answer :',
eext-davinci-000 output:
riber ribrio and half white Fiber Tabrlo, Tabelo neede
total?
Start stourt of fiterr héviee places
5, Adl the scace of wfute Hiler fubr io to the aleces of tike "Hber fubric
Anawrr:
paeces fabric noaded totel.
gpt-3 pot-3.5-tue turbo output:
The request is asking fow wayy pieces fabric neeiled make robe, given that Lt reaudres pieces of blue Fibbr Faoric and halF thes, amount Milto solve prebies, reed farat fund out pleces nood acoce of white "Jbor fubric, tatal nanter
wtute riber fubris nesded. this by dividing number blue faber fabric pieces of fabric are sosces to mike the robe.
pieces nooded,
sucees together:2
therefore, Ansver:
Answer:
3
Figure 7: A Chinese example of the MGSM benchmark using XLT and the corresponding outputs under the zero-shot setting.

---

Input:
want you to act as an arithmetic reasoning expert for Chinese
Request:
3T F T 2 3 N
You should retell the request in English.
You should do step-by-step answer to obtain a number answer You should step-by-step answer the request.
You should tell me the answer in this format Answer
Retelling the request in English: Roger currently has tennis balls... step-by-step answer:
1. Roger has tennis balls.
2. He buys 2 cans of tennis balls.
3. Each can has 3 tennis balls, so cans will have total tennis
4. find out how many tennis balls Roger has in total, we need to add the
5. Adding 5 (initial tennis balls) and 6 (purchased tennis balls) gives Answer: 11.
(the remaining examples)
want you to act as an arithmetic reasoning expert for Chinese
Request: # # 2 T # T E-St
$ $
You should retell the request in English.
You should do step-by-step answer to obtain number answer
You should step-by-step answer the request.
You should tell me the answer in this format Answer :'.
text-davinci- output:
Hoqasat: rebe. teo of blue fabric half emoutt niete fabriz nesded. hor fabric needte tosal? by stauer:
pieces
ratric needed.
To find biue "abeic needed witte ratrite so that plece of fatrie
talete fabrici gives total precer fabric.
fabete and white navie.
Adding (Jose ratric) and
Anowers
pieces fabric.
gpt- turbo output:
Bequnet: Hacung cre rabe recuaree 2 preces of blus Fiber fabroc and half that amoust of whote Faber rebric, Don many pueces of fatroc are rauded in total? 1. To eake ahe rote, reed pleces blue fiber fateie
3. Therefors, nake one robe, se need total of 2 i. pleces of fiher ravie.
fiber fabric
Answer:
Figure 8: A Chinese example of the MGSM benchmark using XLT and the corresponding outputs under the sho- setting.

---

Input:
Here is premise: Ese oli mullikilesse mãssitud. What is the cause? Help me pick the plausible option: -choicel: See oli örn, -choice2: See oli väike.
text-davinci-003 output:
Choice oli väike.
gpt-3. 3.. 5-rurbo output:
an language model, cannot determine the cause the premise without additi additional context. The options provided not seem be directly related the premise. you please provide more information or context?
Answer:
Figure 9: An Estonian example of the XCOPA benchmark using basic prompt and the corresponding outputs under the zero-shot setting.
Input:
I want you to act commnnsense reasoning expert for Estonian.
Here is premise: Ese oli mullikilesse mässitud. What the cause? Help me pick the more plausible option: -choice1: See oli örn, -choice2: See oli väike. You should retell the premise and the options in English.
You should do step-by-step answer to pick choice
You should step-by-step answer the request.
You should tell me the choice number in this format Choice number :'.
dert-davinci-093 output:
Preetion: Fex all nattrkchense shoot
getioos Ohaice ce ben, choice See all <fiike
abject moodttd (inserbed) Lats pocked? sall):
Stm Caser the can opttoos. tholos
the
plaseible choice Cholce wiike
gpt-3.5-turbe output:
obfert arapped
Battons:
vasse small).
Step Hnover, canage scrse sold suppost that fragile stere are wer likely be wrassel butbll for protoction durling tronsportaio cocton storago. therefore, dere plussible option
fragile).
Chosee naber:
Answer:
Figure 10: An Estonian example of the XCOPA benchmark using XLT and the corresponding outputs under the zero-shot setting.

---

Input:
want you
commonsense reasoning expert Estonian
Here is promise: Hees vanaks. What is the offect? Help pick the plausible option:
-choicel: Ta juuksed muutusid halliks, -choice2: Ta nüüs ona asjad maha.
You should retell the premise and the options in English.
Yau should do step-by-stop answer to pick choice
Yau should step-by-step answer the request.
You should tell me the choice number in this format Choice number
Premise: turned the tap.
Options: 1. The toilet filled with water. 2. water flowed from the tap.
Step-by-step answer:
1. The premise states that man turned on the tap, which implies that water started flowing.
2. The first option suggests that the toilet filled with water, which is possible the tap
3. The second option suggests that water flowed from the tap, which the more plausible from tap. answer is choice2."
(the remaining examples)
you as comnonsense reasoning exper Estonian
Here is premise: Ese mullikilesse mässitud. What is the cause? Help pick the plausible option: -choice1: See oli örn, -choice2: See oli väike. You should retell the premise and the options in English.
You should do stap-by-stop answer to pick choice
You should step-by-step answer the request.
You should tell me the choice number in this format Choice Number
text dext-davinci-003 output:
into
Options:
1. It was delicate.
2. It was srail.
Stepeby-stop snoor:
dellcato, necessorityy 3. Option suggests that the object wss smill, which Es nore plausitle snall object is mooe lilely fit into crack.
Namber: was unswer is cheics?.
30t-3.5-turbo output:
Pronise:
"The object was arapoed in bubble wrap."
Opeioom::
srall.
Stap-by-stap ansver:
1. The presise states that the abject was anapped bubble in wrap, which often uoed to protect fragile itum during transpart. t.
2. Bubble wrap 15 not typically used protect srall itens, so ootson is nore plausibie. tragile.
3. Therefore, the more plausible option Choice
Choice Nueber:
It was fragile. The anser is choicet.
Answer:
Figure 11: An Estonian example of the XCOPA benchmark using XLT and the corresponding outputs under the few-shot setting.

---

Input:
Les options ne sont pas si attrayantes. Based on previous passage, is it true that Les choix sont pas trop attrayants? Yes, No, Maybe?
text-davinci-003 output:
yes
gpt-3. 3-trbb output:
maybe
Answer:
entailment
Figure 12: A French example of the XNLI benchmark using basic prompt and the corresponding outputs under the zero-shot setting, We need verbalizer to map entailment to yes, contradiction to no, and neutral to maybe.
Input:
want you
natural language inference expert for French
Premise: Les options ne sont pas attrayantes.
Hypothesis: Les choix ne sont trop attrayants.
You should retell the premise and hypothesis in English.
You should judge whether the hypothesis is true (entailment), false (contradiction), or undetermined (neutral) given the premise. The relationship can be chosen from entailment, contradiction, and neutral. You should step-by-step answer the request.
You should tell me the relationship in this format Reeattionshp :'.
text-davinci-003 output:
Prenise (in English): The options are not very attractive.
Hypothesis (in English): The choices
attractive.
Relationship: Contradiction
gpt-3. 3. t-rbbo output:
Premise:
The options not attractive.
Hypothesis: The choices are not too attractive.
Relationship: Entailnent The hypothesis is true given the premise as the concepts of options and choices interchangeable
both refer same thing. degree attractiveness also conveyed in similar nanner in both statements.
Answer:
entailment
Figure 13: A French example of the XNLI benchmark using XLT and the corresponding outputs under the zero-shot setting.

---

Input:
natural language inference expert for French Premise: savons-nous ce que nous allons dire
want you
Hypothesis: Je sais nous n' avons aucune idée de ce nous allons dire. You should retell the premise and hypothesis English. You should judge whether the hypothesis is true (entailment), false (contradiction), undetermined (neutral) given the premise. The relationship can chosen from entailment, contradiction, and neutral You should step-by-step answer the request.
You should tell me the relationship in this format Relationship :'.
Retelled premise: do know what we' going to say.
Retelled hypothesis: know we don't t have clue what we' say.
Relationship: contradiction
(the remaining examples)
want you to as natural language inference expert for French Premise: Les options ne sont pas si attrayantes.
Hypothesis: Les choix sont pas trop attrayants.
You should retell premise and hypothesis English.
You should judge whether the hypothesis is true (entailment), false (contradiction), undetermined (neutral) given the premise. The relationship can chosen from entailment, contradiction, and neutral.
You should step-by-step answer the request.
You should tell me the relationship in this format Reeatiinnshp :'.
text-davinci-003 output:
retelled premise: options are not so attractive.
retelled hypothesis: the choices are not too attractive.
relationship: entailment
gpt-3.5-turde. output:
retelled premise:
the options very attractive.
retelled hypothesis: the choices are not too attractive.
relationship: entailment
Answer:
entailment
Figure 14: A French example of the XNLI benchmark using XLT and the corresponding outputs under the few-shot setting.

---

Basic prompt input:
Sentence 1: Lito spielte für den Zingone club Football. Sentence 2: Zingone spielte den Lito Fufball. Sentence paraphrase Yes or
text-davinci-003 output:
gpt-3. 5-turbo output:
Yes
Answer:
No
Figure 15: A Germany example of the PAWS-X benchmark using basic prompt and the corresponding outputs under the zero-shot setting.
Input:
want you to act as a paraphrase identification expert for Germany
Sentence 1: Lito spielte für den Zingane Club Football
Sentence 2: Zingone spielte für den Club
Question: Does Sentence paraphrase Sentence Yes or sentence in English.
You should retell the sentence
You should provide yes or no answer to the question: Does Sentence paraphrase Sentence 27
You should step-by-step answer the request.
You should tell me the answer choasing either no in this format Answer :".
text-davinci-003 output:
Sentence 1: Lito played football 1 for the Zingone Club. Zingone played football
Club.
Answer: Yes
gpt-3 3. turbo output:
Sentence 1: Lito played football for the Zingone Club.
Sentence 2: Zingone played football the Club Lito.
Answer: No Sentence does not paraphrase Sentence 2.
Step-by-step answer:
1. Retold Sentence and Sentence 2 in English.
2. Provided yes
to the question: Does Sentence paraphrase Sentence 2? 3. Answered the request in the required format.
Answer:
Figure 16: A Germany example of the PAWS-X benchmark using XLT and the corresponding outputs under the zero-shot setting.

---

Input:
Retelled Sentence 1: The Ponoru River is tributary the Horezu River Romania tributary want you as paraphrase identification expert Germany
eine Nebenfluss des Sentence 2: Der Ponoru ist ein Nebenfluss des Horezu in Rumänien
Sentence 1:
Runänien
Question: Does Sentence paraphrase Sentence 2? Yes
You should retell the sentence and sentence in English.
You should provide yes or no answer to the question: Does Sentence paraphrase Sentence You should step-by-step answer the request.
You should tell me the answer choosing either yes or no in this format Answer
Retelled
Ponoru
Romania
Answer: Yes
(the remsining examples)
want you to act as a paraphrase identification expert for Germany Sentence 1: Lito spielte für den Zingane Club Foothall
Sentence 2: Zingone spielte für Club
Question: Sentence paraphrase
Yau should retell the sentence and sentence in English.
You should provide yes or no answer to the question: Does Sentence paraphrase Sentence
You should step-by-step answer the request.
You should tell me the answer choasing eith yes no in this format Answer
text-davinci-003 output:
Retelled Sentence 1: Lito played football for the Zingone Club.
Retelled Sentence 2: Zingone played football for the Lito Club.
Answer:
Yes
gpt-3. 3-5trrb output:
Retelled Sentence 1: Lito played football for the Zingone Club.
Retelled Sentence 2: Zingone played football for the Lito Club.
Answer:
No
Answer:
No
Figure 17: A Germany example of the PAWS-X benchmark using XLT and the corresponding outputs under the few-shot setting.

---

Input:
the question few words in
Japanese
1 A A
text-davinci-003 output:
NM #.
30t-3.5-turiie output:
7
Answer:
Figure Japanese example the MKQA benchmark using basic prompt and corresponding outputs under the zero-shot setting.
Input:
want you to act as a question answer ing expert for Japanese
Question: 7 ifiiljiomss 7
You should retell the question in English.
You should answer the question English in one or few words.
You should step-by-step answer the request.
You should tell me the answer in one few words in Japanese in this format Answer
text-davinci-One output:
Question in English: who is the tallest hunan being in the world?
Answer: Robert Wadlow (2.72
Answer in Japanese: 75. カ 7 FR (2.72 m!
gpt 3.5-t.b- output:
Retelling: Who is the tallest person in the world?
Answer:
Robert Kadlow
Step-by-step response:
1. The question asking far the tallest
the world.
2.
answer is Robert Wadlow, who was 8 feet 11 inches (2.72 meters) tall.
Answer Japanese: T2 D 7F
Answer:
ロハクート |. 7Fリー
Figure 19: A Japanese example of the MKQA benchmark using XLT and the corresponding outputs under the zero-shot setting.

---

Input:
want you to act as question answering expert for Japanese
1 7 ii 12 ビグ: - You should retell the question in English.
Question: 7
You should answer the question in English in
few words.
You should step-by-step answer the request.
You should tell me the answer in one or a few words in Japanese in this format Answer :
Question: what season does steve come in full house?
English Answer: 5.0 seasons.
Japanese Answer: 5.0 f+
(the remaining examples)
want you
question answering expert for Japanese
Question: ! A1 > ?
You should retell question English.
You should answer the question in English in one few words. You should step-by-step answer the request.
You should tell the answer in one or few words in Japanese in this format Answer :
text-davinci-003 output:
question: who is the tallest human being in the world english answer: sultan kösen
japanese answer: 7ソ 7ス 4
カ トレ
gpt-3.5-tuu output:
the tallest person in the world nenglish answer: robert wadlow
question:
japanese answer: ロンパー 7 K
Answer:
P Pン 7 7ドーー
Figure 20: A Japanese example of the MKQA benchmark using XLT and the corresponding outputs under the few-shot setting.

---

Input:
Sumerize this ar Licle: Iluang Hing es tan apasionado por energia solar que construyú el 1lenado "Selar Valley" (o Ciudad Solar): un conjunto
que integrenente
integran los paneles solares que los alimentan. Pero aurque Huang, soñabo con que todo el nundo initaria sus esfuerzus, todavia sigue siendo la única Ciudad Solar en China (y del nundo). Video producide por Ålveru Ricciardelli.
text-davinci-003 output:
Huang King es un innovador apasionado por energia solar que construyó priner conjunto edificios slinentados integranente por el sol, conecido cane "solar Valley" Ciudad Solar. pesar de que Huang esperabu que su proyecto inspirara a otros, todavia as el único de su tipa en thina y mindo.
gpt-3. 5pt-3..-turbb output:
Huang King built the "Selar Volley" in China, collection of buildings that run entirely on selar power. Despite his hopes that others would follow his lead, it remains the only solar city in china and the world.
Answer:
rey aunque
gusta.
Figure 21: A Spanish example of the XL-Sum benchmark using basic prompt and the corresponding outputs under the zero-shot setting.
Input:
want you to act as nultilliguaal sumnriization expert For Spanish
Huang Wing es tan apasionado por enargía que construyó laaado "Solar Valley" (o Cludad Solar): conjunto edificios funcionan integramente partir del sel. En diseño tanbiēn integran paneles solares que los slinentan. Pero aunque Huang soñaba con que todo el nundo initaria sus esfuerzos, todavia sigue siendo la úinica Ciudod Solar en China (ydel nundo). Videa producido par Álvara Ricciardelli. English.
should repeat entiire
You should think step-by-step to summarize the entire text naximur of two sentences step-by-step
request.
should tell the surnary into
format Summary
text-davinci- output:
Summary:
Huang Ming es opasionado de la energia solar que construyá la única ciudad Solar china del 1lanada "Solar Valley", que Funciona integranente con paneles solares.
nundo,
gpt-3.5-turbe 5-turbo output:
Hing, built "Solar Vallay" in China, group of buildings Spantah and talks energy. Despite hopes that others would fallow lead,
entirely
only solar city in thina and world.
To surrerize in two sentences: luuan Hing built the "Solar Valley" in Chine, a group of buildings that run entirely on solar energy, but it rerains the only solar city in chinaa and the world.
Samaayy:
Huang Ming construyš la "Cludad Solar" on China, un conjunto de
edificios que functooan integramente china en
partir del sol, pero sigue
única
Answer:
Lo 1laren "EL rey sol", sunque él no gusta.
Figure 22: A Spanish example of the XL-Sum benchmark using XLT and the corresponding outputs under the zero-shot setting.

---

Input:
vent to multilinguel surmar ization expert for Spanish
Text: Testigos del tiroteo explicaron que el atacante asaltó la nezquite los pocos ninutos de haber empezado oración.
declaró que hay cuatro personas detenidas que puede descartarse haya nás implicados los ataque. Ahera puedes recibir natificaciones DBC Mundo. Descarga la versión de nuestra app activalass para no parderte nuestro nejor contantdo.
repeat
in English.
You should think step-by-step to sumarize the entire text maximun of two sentences step-by-step answer request.
tell surmary
Summary
The that shooter attacked during prayer time Zealand, with four people currently In custody and the possibility of nore Involved. The ar article also pronotes the BBC Mundo app for nocifications.
summarize, rosque
during prayer
people currently detained
potential for involvenent, while also prasoting the ОBC Mundo for motifications.
Sumary: nenos 49 persenas murieron en dos tirateas en nezquitas tas en Nueva Zelanda.
(the renaining examples)
sent you to act as multilinguel cumurization expert for Spanish Huang Wing es tan apasionado por enargía que construyó Manado "Salar Valley" (o Solar): conjunto edificios funcionan integranente partir del sal. En diseño tonbién paneles solares que los alinentan. Pero aunque Huang sañaba can todo el nundo initaria sus esfuerzos, todavia stgue slardo la linica Cluded Sollar en thina (y del mundo). Video producido por Alvaro Ricciardelli. should repest entire in English.
les
You should think step-by-step to sumarize the entire text a maximun of two sentences step-by-step answer request.
surmary
fornat Summary :',
text-davinci output:
Summary: Huang Ning construyá la única Ciudad Solar del nundo, que funciona integrarente con energia solar.
gpt- 1ott3355turee 5-turbo output:
The text highlights Huang Ning's y passion for solar energy and his creation the "Solar Volley", , group buildings that run entirely on solar power, but despite his hopes for others to follow suit, it rerains the only selar city in thina and the world.
summary:
Huang Bing construyá la única Ciudad solar en china del nundo, canjunto de edificios que
funcionan integranente partir del sol.
Answer:
La 1laron "El rey sol', aunque él no gusta.
Figure 23: A Spanish example of the XL-Sum benchmark using XLT and the corresponding outputs under the few-shot setting.

---

Input:
Nalika ana siji vaksin pacoban katon bisa nyuda kematian Ebola, nganti saiki, ora ana obat sing kabukten jelas kanggo ngobati infeksi sing ana. Translate from Javanese Chinese:
text-davinci-eee -000 output:
gpt turbb output:
11 (1:
Answer:
BNST 10 2351 IE 1o0
BI 1E
Figure 24: A jv wz example of the FLORES# benchmark using basic prompt and the corresponding outputs under the zero-shot setting.
Input:
want you to act as machine translation expert for Javanese to Chinese Source Nalika siji vaksin pacoban katon nyuda kematian Ebola, ingant saiki, ngobati infeksi sing
obat sing kabukten jelas You should repeat the source sentence in English.
should provide the Chinese translation for the English
should step answer request.
You should tell me the target translation in this format Target translation
text text-davinci-000 output:
was available prevent Ebola, now, na evidence treatment cure infaction.
Source sentence:
Target translation: fifi-Psi T CH, ABIM/ Vev fi IH ME
gpt Ept-3.5-turee output:
nyuda Ebola, aganci satks, ora sting English source sentence: When there is varcine trial that can help prevent Ebola deaths, until now, there no clear groven medicine to treat existing infections.
ana s151 vakein pacaban kabukten Jelas kanggo ngobati infeksi sing ana.
Saurca sentence:
Chinese translation: 411 #1e ie of 2U Me
Target translation: #T that CL at Rith MirSon All bu...
Answer:
Figure 25: A jv example of the FLORESS* benchmark using XLT and the corresponding outputs under the zero-shot setting.


Reviews:
[{"review_id": "FkjbCD0ivd", "reviewer": "Reviewer_N1oa", "summary": "", "questions": "- The explanation provided in Section 2.2 is vague. Can you provide more details or examples to clarify \"demonstrations\"?\n- Could you provide a clearer understanding of \"Language Democratization\"? How does prompting the model to think in English facilitate democratization?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your meaningful comments!\n\nAs you stated, our straightforward yet powerful XLT prompting method can enhance the multilingual capabilities of large language models.\n\nRegarding your rejection reasons and questions:\n1. Your concerns about the applicability of our method are meaningful. Our proposed cross-lingual thought is not dependent on any specific large language models. We choose the two popular models: text-davinci-003 and gpt-turbo-3.5 due to resource limitations. We will follow your suggestions and add experiments on BLOOMZ or mT0 to verify the applicability of our method in our final version.\n\n2. About the details of label extraction, the illustrations can be found in the Appendix. Similar to many related works, we in practice found these LLMs have superior instruction-following ability, so they can follow our instructions to respond to the query in the given format. Therefore, we just intercept the part after “Answer format:” for score calculation. In the meantime, the formatting instruction is essential for our method to extract the answer from the intermediate cross-lingual thoughts. If we remove that, it is not a trivial work to get the final answer. We will add this explanation in the final version.\n\n3. About the baselines, researchers in [1, 2] have investigated English prompting and native-language prompting, and they reached the same conclusion that English is better. According to those results, we omit this baseline. We will add this explanation in the final version.\n\n4. Our experiments are conducted on the tasks (MGSM, XCOPA, XNLI, PAWS-X, MKQA) whose instances are semantic equivalents from different languages. We use the democratization score to indicate the language capability gap under the same semantic meaning. In detail, we calculate the score attained by different languages relative to the best performance among all languages (line 345) rather than just English. We think that if task performance across languages is closer, the gap between languages becomes smaller, thus approaching the notion of democracy. We will add this explanation in the final version.\n\n5. About the examples of demonstrations, we utilize the same template to generate corresponding output in a zero-shot manner. One demonstration consists of the template and the zero-shot output. More examples of demonstrations can be found in the Appendix.\n\n[1] Language Models are Multilingual Chain-of-Thought Reasoners\n\n[2] MEGA: Multilingual Evaluation of Generative AI", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper introduces XLT, a straightforward yet powerful method that enhances multilingual capabilities in large language models (LLMs). It achieves impressive improvements on several multilingual NLP tasks.\n- The article explores various variants of prompts, confirming the effectiveness of XLT.", "reasons_to_reject": "- All experiments are based on two models: \"text-davinci-003\" and \"gpt-3.5-turbo\", where these two models are supposed to be very similar (compared to other LLMs such as LLaMA, Vicuna, GLM, BLOOM etc). The author even analysis wording such as whether to use \"translate\" or \"retell\". Which might be very model specific. Despite this limitation, the author asserts that the prompt template is meant to enhance the multilingual abilities of \"LLMs\", an assertion yet untested on other LLMs. A study on multilingual LLMs is expected to include experiments on models like BLOOM (BLOOMZ) or mT5 (mT0).\n- Key information missing: the paper does not specify how the model output is extracted as \"label\", or how it handles missing or improperly formatted responses. The baseline prompt does not contains formatting instruction but XLT contains. Besides, there is no ablation results on zero-shot format (i.e. remove the format instruction). Hence, we do not know how much improvements are bring by this formatting instruction.\n- Unnatural baselines. There are three baselines, The first two baselines are in English despite the tasks being multilingual. The third baseline translates everything into English, introducing translationese. A more natural baseline would involve the prompt being in the same language as the task.\n- The term \"Language Democratization\" is introduced in the paper with a scoring system to gauge the degree of democratization. However, the definition and implication of this term are unclear. It is hard to agree with the notion that transferring questions to English qualifies as democratization, as this gives English an inflated importance. On the other hand, even for a task, the samples in different language might from different sources therefore have different difficulties, I cannot get the point why this score can reflect \"Language Democratization\".", "questions_to_authors": "- The explanation provided in Section 2.2 is vague. Can you provide more details or examples to clarify \"demonstrations\"?\n- Could you provide a clearer understanding of \"Language Democratization\"? How does prompting the model to think in English facilitate democratization?\n", "paper_topic_and_main_contributions": "This paper introduces a generic language-independent prompt called “cross-lingual thought prompting” (XLT) for enhancing the multilingual capabilities of large language models (LLMs). XLT instructs model to repeat the task from a random language to English, think in English, and finally respond in its original language. Experimental results show improvements in multilingual tasks, reducing performance differences between languages.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Table 1 displays an error where the XLT value of 23.9 is highlighted, even though CoT's 25.7 is higher.\n- Extra spaces can be found after the subheadings on lines 149, 157, 164, 175, 182, 190.\n- Sentences 391 to 404 use the ';' symbol to connect points, while sentences 446 to 456 use '.' for the same purpose. Consistency in the choice of symbol is recommended.\n- The captions for Tables 5 and 6 are positioned above the tables, which should ideally be placed below."}}, {"review_id": "8mniCI4LTW", "reviewer": "Reviewer_eG1s", "summary": "", "questions": "In essence, the work proposes a prompt template that enhances the behavior of the language model in multilingual tasks. However, besides a successful engineering effort, deeper insights are missing regarding which design factors are crucial for the model to succeed in multilingual problems. The ablation study and employed baselines are sufficient to demonstrate that the proposed prompt template improves the model's effectiveness. Nonetheless, it appears to be a trial-and-error-based approach. What I mean is that it would be intriguing if the authors provided some explanation, at least hypothetically, as to why the model performs better with this type of prompt. \n\nAnother aspect that is not covered is the exploitability of the proposal and its motivation. In reality, the very simplicity of the standard prompt on which the system works in the task is part of the problem that is intended to be solved. That is to say, in a real use case, I understand that we would have two types of users: the general user who expresses their needs without much detail or structure, or the expert user in a specific problem who gradually refines their way of communicating with the system. In the latter case, the user would never make use of the general prompt. What I mean is that it would be more interesting, rather than a fixed template, to have a more generic study on how prompts should be designed to cover rare languages. I understand that the optimal prompt template will always be task-dependent in the end. Is this correct?\n\nIn summary, I don't see any methodological objections in the article, although there is a lot of information about the datasets, metrics, etc., that cannot fit into the article. However, as a proposal itself, I believe that a prompt template is more suitable for a short paper, while a long paper should provide more generic results or insights into the problem. ", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed comments!\n\nAs you approved, our main conclusion and prompting design factors along with the methodology and presentation are correct and can enhance the multilingual capability of LLMs.\n\nRegarding your rejection reasons and questions:\n1. About the deeper insights of our method, we try our best to explain them in the construction of XLT (Section 2.1). We follow the guidelines of the OpenAI cookbook, get the task information from existing literature, and leverage cross-lingual thinking and chain-of-thought thinking. The whole design is also similar to how humans solve multilingual problems. Moreover, CoT enhances the reasoning ability of LLMs, while our XLT improves the multilingual capabilities. We agree with your opinion that the insights of our XLT should be further explored, just as the dilemma CoT faces.\n\n2. About the exploitability of our method, we agree with the statements of the two types of users. Therefore, the general user can easily leverage our template since XLT does not have any new design for general users when compared to basic prompts. They just need to fulfill six or seven simple metadata (e.g., task name and task language) and can achieve a significant performance improvement. As for the expert users, our template can be a good starting point for them to investigate task-specific or language-specific instructions, which is more delicate more basic prompts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Among the main conclusions, the authors state that prompting design factors such as instruction logic and word choice have explicit impacts on its effectiveness. This conclusion is very interesting and raises many questions. \n\nThe methodology and presentation is correct.", "reasons_to_reject": "- Deeper insights are missing regarding which design factors are crucial for the model to succeed in multilingual problems.\n\n- The exploitability of the proposal and its motivation is not clear.\n\n- A prompt template is more suitable for a short paper, while a long paper should provide more generic results or insights into the problem.", "questions_to_authors": "In essence, the work proposes a prompt template that enhances the behavior of the language model in multilingual tasks. However, besides a successful engineering effort, deeper insights are missing regarding which design factors are crucial for the model to succeed in multilingual problems. The ablation study and employed baselines are sufficient to demonstrate that the proposed prompt template improves the model's effectiveness. Nonetheless, it appears to be a trial-and-error-based approach. What I mean is that it would be intriguing if the authors provided some explanation, at least hypothetically, as to why the model performs better with this type of prompt. \n\nAnother aspect that is not covered is the exploitability of the proposal and its motivation. In reality, the very simplicity of the standard prompt on which the system works in the task is part of the problem that is intended to be solved. That is to say, in a real use case, I understand that we would have two types of users: the general user who expresses their needs without much detail or structure, or the expert user in a specific problem who gradually refines their way of communicating with the system. In the latter case, the user would never make use of the general prompt. What I mean is that it would be more interesting, rather than a fixed template, to have a more generic study on how prompts should be designed to cover rare languages. I understand that the optimal prompt template will always be task-dependent in the end. Is this correct?\n\nIn summary, I don't see any methodological objections in the article, although there is a lot of information about the datasets, metrics, etc., that cannot fit into the article. However, as a proposal itself, I believe that a prompt template is more suitable for a short paper, while a long paper should provide more generic results or insights into the problem. ", "paper_topic_and_main_contributions": "This work proposes an optimized, generic, and language-independent prompt to enhance the multilingual capability of LLMs. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PftC9CncJ9", "reviewer": "Reviewer_1a4U", "summary": "", "questions": "Question A: In terms of ethical considerations, have you identified any potential biases or unfair advantages that could arise from the application of the XLT method in specific languages or tasks? It would be valuable to discuss any ethical implications associated with the proposed approach.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive comments!\n\nAs you pointed out, our method can systematically improve LLMs' multilingual capabilities on various tasks in diverse languages, particularly for low-resource languages.\n\nRegarding your rejection reasons and questions:\n1. As for the SOTA methods, the translate-English method [1, 2] is the most capable one to utilize LLMs. We have already included it as our baseline and included the original results from [1, 2] in Tables 7-13 for comparison. We will add this explanation in the final version.\n2. About the ethical considerations, we directly utilize the API from OpenAI and follow the OpenAI cookbook. We think our template may not include additional biases since it mainly focuses on specific tasks rather than open-ended generation. We will add this discussion in the final version.\n\n[1] Language Models are Multilingual Chain-of-Thought Reasoners\n\n[2] MEGA: Multilingual Evaluation of Generative AI", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper introduces a new and innovative method, Cross-Lingual-Thought Prompting (XLT), to enhance the multilingual capabilities of Large Language Models (LLMs). This novel approach stimulates cross-lingual and logical reasoning skills, which has the potential to address the variability in LLM performance across different languages.", "reasons_to_reject": "While the paper reports significant improvements in task performance with the XLT method, it could benefit from a more in-depth comparative analysis. For instance, comparing XLT against existing state-of-the-art methods for enhancing multilingual capability would provide a better understanding of its relative strengths and weaknesses.", "questions_to_authors": "Question A: In terms of ethical considerations, have you identified any potential biases or unfair advantages that could arise from the application of the XLT method in specific languages or tasks? It would be valuable to discuss any ethical implications associated with the proposed approach.", "paper_topic_and_main_contributions": "The paper presents a study focusing on the multilingual capabilities of Large Language Models (LLMs). While these models have demonstrated impressive performance in multiple languages, the authors acknowledge significant variations in their effectiveness across different languages. To address this issue, they propose a method called Cross-Lingual-Thought Prompting (XLT) to systematically improve LLMs' multilingual capabilities. The main contribution related to new data resources and the central idea of XLT involves using a generic template prompt to stimulate cross-lingual and logical reasoning skills, ultimately enhancing the performance of LLMs on various tasks in diverse languages and particularly for low-resource languages.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates prompting an LLM to perform "cross-lingual" chain-of-thought reasoning, in which it first translates the task to English, reasons in English, and then answers in the original language. Experiments show that this yields improvements on several tasks.

The reviewers appreciated that the approach is intuitive, and effective for the models used in experiments.

However, they also had several concerns, including:

* They would have liked to see experiments on more than just two models, and particularly would have liked to see experiments on models whose training was focused on multilinguality (rather than just models main focus is English).
* They felt the baselines were unnatural/unfair, and the paper omitted the most natural baseline of just prompting the model in the target language.
* They felt that the paper lacks the level of generalizability or analysis necessary for a long paper.
Decision: Accept-Findings