Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages
Authors: Libo Qin
Qiguang Chen
Fuxuan Wei
Shijue Huang
Wanxiang Che
Venue: EMNLP 2023 Main (2023)
Keywords: Chain-of-Thought; Cross-lingual Prompting; Cross-lingual self-consistency Prompting

Abstract: Chain-of-thought (CoT) is capable of eliciting models to explicitly generate reasoning paths, thus promoting reasoning accuracy and attracting increasing attention. Specifically, zero-shot CoT achieves remarkable improvements in a wide range of reasoning tasks by simply instructing the LLM with the prompt "Let's think step by step!".  Despite the success of zero-shot CoT, the existing zero-shot prompting techniques remain limited to a single language, making it challenging to generalize to other languages and hindering global development. In this work, we introduce cross-lingual prompting (CLP), aiming to improve zero-shot CoT reasoning across languages. Specifically, CLP consists of two main components: (1) cross-lingual alignment prompting and (2) task-specific solver prompting. The cross-lingual alignment prompting is responsible for aligning representations across different languages, whereas the task-specific solver prompting is used to generate the final chain of thoughts and results for the reasoning task. In addition, we further introduce cross-lingual self-consistent prompting (CLSP) to ensemble different reasoning paths across languages. Our experimental evaluations on several benchmarks demonstrate that CLP and CLSP significantly outperform the existing prompting methods and achieve state-of-the-art performance. We hope this work will inspire further breakthroughs in cross-lingual CoT.

Cross-lingual Prompting: Improving Zero-shot Chain-of-Thought Reasoning across Languages
Libo Qin Qiguang Chen Fuxuan SShjue Huang,, Wanxiang Che
School of Computer Science and Engineering, Central South University, China
Research Center for Social Computing and Information Retrieval
Harbin Institute of Technology, China
Harbin Institute of Technology, Shenzhen, China
lbqinécsu. edu. en,
f cn, (ggchen, fxwei, car) } @ir, hit. cn joehs j3100gmail com
Abstract
English Request: Mary two years younger Joan, who is five years older than Jessa. If Jessa is 20 years old, what is the sum of the ages of the three girls? Let's think step-by-step!
ofainnof-tthougt (CoT) is capable of eliciting models to explicitly generate reasoning paths, thus promoting reasoning accuracy and attract- ing increasing attention. Specifically, zero- shot CoT achieves remarkable improvements in wide range of reasoning tasks by simply instructing the LLM with the prompt "Let's think step by step!". Despite the success of zero-shot CoT, the existing zero-shot prompt- ing techniques remain limited to a single lan- guage, making it challenging generalize to other languages and hindering global develop- ment. In this work, we introduce cross- linsullingual prompting (CLP), aiming to improve zero-shot CoT reasoning across languages. Specifically, CLP consists of two main components: (1) cross-lingual alignment prompting and (2) task- specific solver prompting. The pross-lingual alignment prompting is responsible for align- ing representations across different languages, whereas the sask-specific solver prompting is used to generate the final chain of thoughts and results for the reasoning task. In addition, we further introduce cross-lingual self-consistent prompting (CLSP) to ensemble different rea- soning paths across languages. Our experimen- tal evaluations on several benchmarks demon- strate that CLP and CLSP significantly out- perform the existing prompting methods and achieve state-of-the-art performance. We hope this work will inspire further breakthroughs in cross-lingual COT.
English CoT: Stepl: Given information: Jessa's age-20, 20, Step2: Joan's age: We know that Joan is five years older than Jessa, So, Joan's age is 20 5 25 years... Finally, the sum of the ages of the three girls is 68 years.
(a) Traditional Chain-of Thought Prompting
Chinese Request: ai HL # +tr *R 79 20 3 it A t0 # Let's think in English step-by- st-pp
English Co Stepl: Deternine age. Since Jesa is 20 years old, and Joan is five years older than Jesa, Joan's age is 20+55=55 years.
Step2: Determine M Mary's age. Mary is two thanJoan, Joan. So Mary's age 25- 2-23 years...
Y years younger
Finally, the total sum of the ages of these three girls is 68.
(b) Cross lingual Chain-of Thought
Figure I: Traditional Chain-of-Thoupg (CoT) vs. Cross- lingual CoT.
which gains increasing attention. Specifically, zero- shot chain-of-thought (CoT) (Kojima al., 2022) only needs to append the prompt "Let's think step by step!", which can elicit strong reasoning capabil- itics from large language models and demonstrate promising performance on various tasks, includ- ing arithmetic reasoning, commonsense reason- ing (Wei et al., 2022; Kojima et al., 2022) and even robotic planning(Ahr et al., 2022: Huang et al., 2022). Take traditional CoT in Figure/ (a) an example, trigger prompt "Let's think step by step!" provided along with an English request to perform step-by-step reasoning. Eventually, LLMs produce the corresponding answer "68 years".
1 Introduction
Large Language Models (LLMs) have shown re- markable success across various NLP tasks (Qin et al., 2023; Hendy et al., 2023; Pan et al., 2023; Ziyu et al., 2023). Unlike the previous pre-trained language models (PLMs) (Devlin et al., 2019; He et al., 2021), LLMs are capable of achicving zero- shot learning without the need modify the model parameters during the training and testing process,
In fact, there are over 200 countries and 7,000 languages worldwide. With the acceleration of globalization, there is an urgent need for general- izing the current CoT across different languages. Despite the remarkable success of zero-shot CoT, its reasoning abilitics still struggle to generalize to different languages. Shi et al. (2022) introduce
"quul Contribation

---

the first multi-lingus dataset evaluate the mathe- matical reasoning capabilities of language models to facilitate the research of cross-lingual COT. Un- like traditional CoT scenarios, where the language of the request and CoT output is the same, cross- lingual CoT requires the LLM to generate CoT in English for any given language by providing a trigger sentence "Let's think in English step by step!", which is illustrated in Figure (b). Unfor- tunately, little attention has been paid to zero-shot cross- mooss-lingual COT.
reveal that both CLP and CLSP are capable of improving zero-shot pross-lingual CoT effee- tively and achieving SOTA performance (with over 1.8% improvement on AVG accuracy).
We hope this work can inspire further research on cross-lingual CoT and the code are available at Cross-Lingual-Pommiingg
Background
This section describes the definition of traditional and cross-lingual chain-of-thought.
To generalize the current CoT across languages, we propose a novel cross-lingual prompting (CLP), which aims to effectively bridge the gap across dif- ferent languages. It consists of two components: (I) Cross-lingual Alignment Prompting and (2) Task- Solver Prompting. Specifically, the cross-lingual alignment prompting used align representations between different languages. In our experiments, instead of the traditional "Ler's think step by step", we use "Let's 's understand the task in English step-by-step.". The inherent intuition is that as model gradually understands the task in English, it inherently captures the relationship be- tween the source language and English. After align- ing the representations between different languages, we further utilize a lask-specific solve prompting to complete the final task by setting "Let's resolve the task you understand above step-by-step!". Such simple yet effective CLP can greatly enhance the reasoning ability of cross-lingual scenarios. Fur- thermore, inspired by the self-consistency work, we propose cross-lingual self-consistent prompting (CLSP), which enables the model to ensemble dif- ferent views of reasoning paths across languages. Experimental results reveal that CLP achieves base-
2.1 Traditional Chain-of-Thoughh
Chan-of- is powerful technique to clicit the strong reasoning ability of large language mod- els (LLM), which is capable of completing complex tasks. For the traditional chain-of- ofto-uughthhhttt (CoT) generation approach, LLM is appended as a simple prompt "Let's think step by step!" to output the specific reasoning paths, which is denoted as:
Request: [Given sentence X] Let's think step by step!
2.2 Cross-lingual Chain-of-Thougle
While traditional CoT has achieved remarkable suc- cess, it is limited generating CoT within single language and lacks effective cross-lingual trans- ferability. Therefore, cross-lingual CoT aims enable models to handle requests in any language and generate CoT specifically in the target language (i.e., English) (Shi et al., 2022).
Cross-lingual Prompting
To elicit the cross-lingual reasoning ability of LLM, introduce cross lingual prompting (CLP) as a solution. Specifically, CLP consists of two compo- nents: (1) cross mooss-lnngual alignment prompting ($3.1
the SOTA performance by outperforming lines with a gain over 1.8%. addition, CLSP can further enhance the performance by integrating knowledge across different languages. The main contributions of this work are concluded as follows:
and (2) task-specific solver prompting ($3.2).
3.1 Step 1: Cross-lingual Alignment
Prompting
We introduce cross-lingual prompting that contains cross-lingual alignment prompting and task-specific solver prompting, which jointly improve zero-shot CoT reasoning across languages;
Cross-lingual alignment is core challenge for cross-lingual transfer. Therefore, to better capture the alignment information, we first introduce cross- lingual alignment prompting (refer to Figure 2 (a)). Specifically, approach initiates the LLM with specific task of aligning information. The request is formulated follows:
We further propose cross-lingual self- consistent prompting to integrate reasoning paths across different languages;
Please act as an expert multi-lingun under- standing in [Source Language L,1
Extensive evaluations on several benchmarks

---

Cross- lingual Alignment Prompting
expert lingual underttading
Requeet 82-8--0088 2825.05 28 d0
Please act as expert in multi- multi-lingual understanding in Chinese.
Request: A9 ii 12-1-55 a5 y. 13 12: f5 & S. -b & 20 12
A-* Ak 88,45 24 fo ?
erman Prompting
Let's understand the task in English stop-by--tep!
Spanish Prompting
"28 15- 36 y/" meuns "Mary
yeurs younger than Joan."
2. "ANPAAA A 5 means "Joan is five years older than Jessic."
English Prompting
oab winl
20 5 mmaas "If' Jessie is 20 years old."
understadd Eie
Task-specific Solver Prompting
Faall mrass "Mary is maaaoo
A understanding, you should act an expert arithmetic reasoming English.
It
chere
undeounding you should ast exper
Let's resolve the task you understand above step-by-step!
Finally, you should format answer 'Answer: [num].
step step:
Accoodting sentence, Mary
Sure! Let's solve the task stop-by-step:
Accoeding
1. According to the first sentence, Mary is two years younger than
Joun. Let's suppose Joun's age J, then Mary's age is J According to second sentence,
Answer: swer: 68. swer: 48. X
five years older than Jessie. If Jessic's age x, then Joan's age 5...
Answer:
Answer: 68, v
(a) Cross-lingual Prompting (CLP)
(b) Cross-lingual Self-consistent Prompting (CLSP)
The main framework of CLP (a) and CLSP (b). CLP consists of cross-lingual alignment prompting and speci--e solver prompting, while cross- lingual self-consistent prompting (CLSP) aims to integrate various reasoning paths different languages.
Figure 2:
Request: [Given sentence X] Let's understand the task in [Target Language L,] step-by-step!
ulated by the LLM. Specifically, we design the task-specific solver prompting as:
After understanding, you should act as an expert in [Target Task T] in [Target Language L,]
Given the sentence X, we first simulate the LLM's expertise in multi-lingual comprehen- sion. Furthermore, we introduce step-by-step alignment process from source language L to
Let's resolve the task you understand above step-by-step!
target language Lt The intermediate semantic alignments are represented as {a:)²_1 where S de- notes the number of alignment steps. Overall, the formulation of our cross- pross-lingual alignment prompt- ing method can be expressed as follows:
Formally, the set of potential reasoning path R is organized into the final reasoming path R₂ for target language L,, which can be determined as:
R2 arg max pl C, La, T),
(2)
R
where C represents the dialog history, including the input variables X. L., Lt, and A. Furthermore, we provide an instruction to format the model's answer, which is defined as: Finally, you should format your answer as 'Answer: [num]'.
A = arg max maxppa.......... .... as|_ Ly, L.),
(1)
where A denotes the alignment response in step 1.
3.2 Step 2: Task-specific Solver Prompting
After achieving cross-lingual alignment, we further propose task-specific solver prompting to facilitate multi-step reasoning in multi-lingual setting. Specifically, given the target language L, and the alignment text A obtained from the previous step, we prompt the LLM to engage resolving target tast T And LLM tries to determine the final result F, along multi-step reasoning path [r:) where [R] represents the number of steps in the reasoning process, which is reg-
Formally, the answer extraction is determined as:
Fg arg max p()R,,)))).)...)
(3)
where F, represents the text of the answer, gener- ated from all potential reasoning result f.
Cross-lingual Self-consistent Prompting
In our research, we observe that LLMs show vary- ing patterns across different languages. Inspired

---

Model
bn
de
cs
fr
ja
ru
SW
te
th
AVG
GPT-3 (text-davinci-002) Di rect (Shi al., 2022)
4.4
14.8
17.2
16.8
11.2
12.4
8.8
0.8
8.8
18.0
11.3
Native-CoT (Shi et al., 2022)
6.4
36.0
40.4
37.6
26.0
28.4
11.2
0.4
10.8
40.0
23.7
Bn-Corl (Shi al., 2022)
9.6
44.0
44.8
46.0
32.4
28.4
20.8
5.6
19.6
40.8
29.2
Translate-En: (Shi et al., 2022)
41.2
46.4
51.6
46.4
44.8
48,8
37.6
42,8
41.2
47.2
44,8
PaLM-540B
Direct (Shi al., 2022)
17.2
18.8
20.0
19.6
16.0
22.0
15.6
17.6
16.8
19.2
18.3
Native-Corl (Shi et al., 2022)
46.0
49.2
56.8
46.4
40.0
48.4
35.2
45.6
52.8
46.8
48.7
En Wn-Corl (Shi et al., 2022)
46.4
53.6
58.0
51.2
49.6
55.6
44.4
46.8
49.6
46.0
50.1
Translate-En! (Shi al., 2022)
53.2
57.2
60.0
55.2
50.0
59.6
51.2
49.6
50.8
55.6
54.2
GPT3.5 (gpt-3.5trbb)
Direct
33.6
56.0
61.2
62.0
52.8
62.0
48.0
7.6
42.4
60.0
48.6
Native-CoT
26.4
70.0
70.4
64.4
52.8
62.4
54.0
10.4
40.0
59.6
51.0
En En-CoT
50.0
73.6
69.6
70.0
60.4
65.6
55.2
22.0
48.0
63.2
57.8
Translate-Er
66.4
75.6
74.4
72.4
66.0
72.8
69.6
58.0
71.6
68.4
CLP
64.8
80.0
82.4
79.2
69.2
81.6
74.8
38.8
62.0
73.6
70.6
CLSP
75.2
86.8
84.8
82.0
77.2
87.6
76.0
52.0
68.0
77.2
76.7
Table I: Main results on MGSM. rect." denotes original input request will given to model directly. "Nati iNative-tn ve-CoT" signifies that the model generates inference steps in the same language the input. "En-CoT" indicates the given non-English imput request and returned with English thiin-of-thought result. "Trans late-Ent denotes we translate non-English input requests into English by Google translation API. denotes the 6-shot results sourced from Shi et al. (2022).
by Wang et (2022), we propose cross-lingual self-consistent prompting (CLSP) to integrate rea- soning knowledge across different languages (as shown in Figure 2 (b)).
PaLM (Chowdhery et al., 2022) and GPT3,5 Fol- lowing Wei et al. (2022) and Kojima et al. (2022), we evaluate the performance using accuracy score (Acc.). The top-p parameter in all processes is set to 1. We select the temperature in Cross-lingual Alignment Prompting from [0, 2] and the tempera- ture in Task-specific Solver Prompting from [0, 1].
Specifically, for each step in the reasoning pro- cess, we require LLM to generate alignment sponses in different larget language L, and em- ploy respective reasoning steps. Finally, we retain the answers that exhibit high level of consistency in the inferred reasoning results (f) through a vot- ing mechanism. These consistently inferred an- swers are then considered as the final result, which can be formulated as:
5.2 Main Results
The main results are illustrated in Table 1. From the results, we have the following observations:
(1) GPT-3.5 exhibits notable cross-lingual rea- soning superiority. When evaluated the all scenarios mentioned in Table 1, GPT-3.5 surpasses the few-shot results of PaLM-540B and GPT-3 by significant margin (achieving improvements of 30.3%, 2.3% 7.7%, and 14.2% over PaLM-540B, respectively). As shown in Wang et al. (2023a), multi-lingual SFTT and RLHF techniques lead substantial improvement in cross-lingual reasoning performance.
L lfl
f argmax Σ M 1F
(4)
where LL represents the count of target languages, If signifies the count of potential reasoning results f across all target languages, and 1 (X) denotes 0-1 function that returns 0 when X False and returns 1 when X True.
(2) CLP achieves state-of-the-art performance. As depicted in Table 1, CLP surpasses all previ- ous baselines, specifically outperforming PALM- 540B(Trans En) with an improvement of 16.4%. This improvement cannot be solely at- tributed GPT-3.5 (CLP even achieves 2.2%
Experiments
5.1 Implementation Settings
We select three representative state-of-the-arn pre- trained large language models as baseline refer- ences for our study: GPT-3 (Brown et al., 2020),

---

Single-turn
80
prompting (STP)
65
Two-stage
50
interactive
35
prompting (CLP)
20
de
e5
ja
ru
SW
zh
AVG
STP
50.4
70.8
71.2
68.4
63.2
68.4
65.2
24.0
54.8
65.2
60.2
CLP
64.8
80.0
81.6
79.2
69.2
82.4
74.8
38.8
62.0
73.6
70.6
Figure 3: The accuracy comparison between two-stage interactive prompting and single-turn prompting.
GPT3.5 (Native-oo))
GPT3.5 (CLP)
better understood the problem statement and en- sured clear inference chain without generating irrelevant or misused information. Furthermore, we observe 2.8% and 2.5% improvements the Informativeness metries for "Step" and "Chain", respectively. It suggests that the model's reason- ing, after cross-lingual alignment, could provide more well-grounded inference steps. Additionally, CLP shows 2.8% enhancement in the Miss-step metric, indicating that the model's reasoning could encompass complete logical chain, leading to better performance.
100
95.6
Performance
91.1
#1.7
814
78.9
73.6
80
75.4
77.0
70
Faithfulncss
Informative- ness Step
Infermative- ness Chain
Missing Step
Figure 4: The analysis of Chain-of- Thouuhh quality between GPT-3.5 (Native-CoT) and CLP.
higher average accuracy than Translate-Hl These findings suggest that cross-lingual alignment prompting (CLP) goes beyond simple text trans- lation and further enhances the model's inherent cross-ling cross-lingual understanding capabilities.
5.3.2 Two-stage interactive prompting better than single turn prompting
This section explores the effectiveness of two- stage interactive prompting. Instead using two turns cross-lingual alignment prompting and task- specific solver prompting to separately perform alignment and task solving, we directly concate- nate the pross-lingual alignment prompting and task-specific solver prompting using the newline character "In" for LLM.
(3) CLSP further significantly improves perfor- mance. As illustrated in Table 1, CLSP exhibits remarkable superiority over CLP across all lan- guages (with 6.1% improvements on average ac- curacy). This observation reveals that integrating knowledge across different languages can effee- tively boost the reasoning performance on cross- lingual CoT, verifying the effectiveness of cross- lingual self-consistent prompting.
Results are illustrated in Figure Compared with two-stage interactive prompting (CLP), we observe significant average decrease of 10.4% in the single-turn prompting performance. We sup- pose that two-stage interactive prompts can better elicit the strong dialogue interactive ability of LLM,, thereby enhancing the performance.
5.3
CLP Analysis
5.3.1
CLP results better reasoning quality
To further investigate why CLP works, we employ the framework of RoscOE (Golovneva et al., 2022) to evaluate the quality of the reasoning paths in the model's Chain of Thought. The implementation details are shown in Appendix A.2.
5.3.3 CLP is not vanilla translation
As shown in Table 1, we can find that CLP even achieves 2.2% higher average accuracy than which indicates that CLP is not vanilla translation but utilizes the semantic align- ment between the languages. To further under- stand how CLP works better than translation, we randomly choose 200 samples from different lan-
As shown in Figure 4, we find that the reason- ing paths of CLP demonstrate higher faithfulness. exhibiting better consistency with key steps during the reasoning process. Specifically, the faithfulness score increased by 1.6%, indicating that the model
Translate

---

Model
ET
ID
IT
QU
SW
TA
TR
VI
AVG
mT0-XXL (Muennighofl al., 2022)
Rn-CoT
24.2
5.2
23.0
29.4
7.4
31.0
16.6
29.2
34.8
10.2
21.3
CLP
41.4
30.8
20.6
30.8
21.6
34.4
33.6
33.6
32.6
49.2
12.2
32.1
Bloomz-7B 7B (Muennighoff' 2022)
En Cor
21.8
24.2
50.6
41.6
41.4
48.6
53.8
37.6
47.0
64.2
42.7
CLP
49.0
49.6
58.0
48.8
50.6
47.6
57.8
52.0
50.2
45.2
54.2
51.2
1ama-22-3B (Touvron al., 2023)
Rn-CoT
39.6
32.5
58.4
55.8
47.2
34.6
47.4
33.2
43.0
59.6
50.4
45.6
CLP
44.8
48.2
64.4
70.2
46.6
47.0
47.8
46,4
51.2
58.4
51.4
52.4
Table 2: The Acc. comparison on smaller and open-resource LLMs.
XNLI
PAWS-X
5.3.5
Generality Analysis of CLP
(AVG: 44.9
48.0)
(AVG: 60.3 64.8)
08
In order to further study the generality of our work, verify the generality of CLP from two aspects:
dc
hi
bg
50
80
CLP works well other benchmarks. con- duct experiments other multilingual reasoning datasets, namely XNLI (Conneau et al., 2018) and PAWS-X (Yang et al., 2019). As shown in Fig- ure 5, CLP can obtain better performance across a majority of languages. In comparison to En-CoT, we observed an average improvement of 3.1% % on XNLJ and 4.5% on PAWS-X2.
50
T0
ar
ja
de
SW
th
v1
2h
GPT3.5 (En Coll
GPT3.5 (CLP)
Figure 5: The Acc. comparison on other benchmarks.
To better un- derstand the model generalization, we conduct the experiments the XCOPA with smaller LLMs. Experimental results (as shown in Table 2) demon- strate that on smaller LLMs, CLP achieves at least 6.8% improvement compared En-CoT. Those further demonstrate the effectiveness and the wide applicability of CL..
CLP works well on other LLMs.
guages for fine-gratd sine-grained exploration.
First, we find that CLP has 7 different strate- gies (as shown in Table 7), which all contribute to the final performance, which demonstrates the effectiveness of CLP. Further, find that breaking down stage 1 further can help improve. Break- ing down the actions of stage into strate- gies can significantly enhance performance (by at least 6.5%). For example, By decomposing the alignment process into "Problem Restatement" and "Preliminary Solution", better performance can be achieved, reaching 64.7% (an increase of 11.8% compared with Native-Con
5.3.6 CLP can further improved by in-context-learning
In recent years, in-context-learning (ICL) has achieved amazing results on LLMs. In order to further explore the performances of CLP within the ICL framework, a series of experiments were conducted. Subsequent analysis of the empirical findings has led the following observations:
5.3.4 How does prompt selection affect CLP?
We validate the robustness of the zero-shot cross- lingual chain-of-thought against the cross-lingual alignment prompts.
Using ICL in cross-lingual alignment prompts can significantly enhance reasoning perfor- mance. As depicted Table 5, CLP exhibits a noteworthy 6.9% improvement over the zero-shot setting on MGSM. This further underscores the ver- satility of our approach as plug-and-play solution, orthogonal to ICL methods, mutually reinforcing each other augment performance.
Table 4 illustrates the performance of different cross-lingual alignment prompts. The experimental results demonstrate that although there are some fluctuations in the AVG Acc. of alignment and rea- soning based on specific prompts (with maximum difference of over 4%), all cross-lingual alignment prompts can still improve the performance com- pared to the traditional COT. This further verifies the effectiveness of CLP.
2 Due to the cost constraint, we randomly select 200 sam- ples per language from test set.

---

CLSP Vanilla Self-consistency
2 th sne Tu tr Co be
0 828
0.821
0.821
0.821
0.824
0.821
0.18
0.820
0.822
Sourco Surpaggg
0.84
820
824
0/817
0127
0.821
0.822
0.821
0.824
0.816
0.820
0.823
1h
0,822
64.8
0.621
0.824
0.815
0.811
0.813
0.818
0.825
te
52.0
0.628
0.824
0.017
0.825
0.817
0.021
0.620
42.2
SW
76.0
76.8
0.817
0.819
0.819
0.820
624
0.823
0.020
0.831
0.81
57.6
0,618
0.816
0.817
0.025
0.827
0.824
0.825
X2.4
cn:
e0;
e0,
60g
66g
0.80
ja
Tamget languag:
The massning alignmentscorme Self Consistrrry.
fr
60.4
>h th to SAP 22 aln
0.816
0.55
154.8
0.518
0:845
0.515
0.822
0.620
0.824
0.520
86.8
0,521
3014
0.911
0.033
0.520
0/813
0.529
82200
bn
06.2
75.2
0.521
0.820
0.609
0.821
0.820
0.8
0.693
3.815
0.805
0.817
20
60
100
826
0.817
0.813
0.515
0.830
0.309
0181
0.522
Figure The accuracy comparison between Cross- lingual Self-consistent Prompting and Vanilla Self- consistency on MGSM benchmark.
zh
-.60
Target language
(b) The reesoring diguieett scorss on CLSP.
Figure 7: The reasoning alignment score comparison between Cross- longull Self-consistent Prompting and Vanilla Self-consistency.
Using ICL in task-specific solver prompting can further boost reasoning performance. As de- picted Table 5, the results reveal an additional 1.1% performance enhancement when incorporat- ing Complex-CoT (Fu et al., 2023) in task-specific solver prompting. This further solidifies the distinc- tiveness of our approach in contrast other CoT optimization methods, underscoring its adaptability and its capacity to offer more extensive support to downstream CoT inference techniques.
the alignment scores between cross-lingual CoT inference paths (including CLSP and VSC) with all correct predicted results and manually annotated CoT inference paths. As illustrated in Figure 7, the variance of alignment scores generated by CLSP is significantly higher than VSC compared with the results of Yu et al. (2023). It shows that CLSP better ensembles language knowledge to enhance the final cross-lingual CoT performance. The im- plementation details are shown in Appendix A.3.1.
For alignment, the example selection plays a piv- otal role. We conducted experiments with vari- ous combinations of Few-shot strategies. As shown in Table 6, if few-shot relies on single strategy, the model's average performance drops significantly to 63.5% even far below the effect of zero-shot. Con- versely, when a more diverse set of strategies is em- ployed within Few-shot examples, the model's per- formance shows a substantial improvement, reach- ing 75.9%. It shows that more diverse strategy samples lead to better performance enhancement.
5.4.2 More languages can not bring more improvement
A natural question that arises is, "Does integrat- ing larger number of languages in self-consistent cross-lingual prompting lead to better overall per- formance?" Rd To answer this question, we explore the relationship between performance and the num- ber of languages integrated. Some studies (Blevins and Zettlemoyer, 2022; Malkin al., 2022) suggest that the LLM's performance is highly related with the proportion of pretraining data in each language. Therefore, we examine the language distribution (refer to Figure 8) in the widely used multilingual pretraining dataset, Common Crawl 2021. Based the proportions,
5.4 CLSP Analysis
5.4.1
Cross-lingual self-consistent prompting surpasses vanilla self-consistency
To validate the effectiveness of CLSP, we con- duct experiments on vanilla self-consistency (VSC) (Wang et al., 2022) which obtains diverse CoT paths for better results. As shown in Figure CLSSP outperforms VSC about 4.5% on average, which verifies the effectiveness CLSP. Further, we try to explore why CLSP works. We evaluate
incrementally integrated languages descending and ascending order of their respective proportions. The results in Fig- ure 9 demonstrate that in high-resource settings (>4%), performance improves as more languages are added. However, when incorporating low-

---

de
2h
ft
Traditional Chain-of Thought Prompting
Roquest; S8 12 NE.. A a.y. # 20 2t
SW
te
other
6.0%
5.3%
4.4% fr
Let's think stop-hy-step
46.3%
23.1%
ru
zh
Let' break down the information
Mar two years younger than Jumn M =]=--
5,4% de ja
4.3%
4.2%
2. Joan years older than jesss: I -]] 5...
other
es
Total age 63. Therefom, the sum of the thme gins ages 63
(a) The ressoning result for traditional chain-of thughtt prompting.
Figure 8: in 2021.
The language distribution of Common Crawl
Cross lingull Alignment Prompting
Please
an expect nung-lingual understanding in Chinese Request: 14 16 d-00 R2,,, # & y. d A. 20 y, Rg. A- 2 05-h--0 is f 9-5"?
High Resource Low Resource
Low Resource High Resource
Let's understand the
English step-by-stap!
78.0
76.7
Perforaacc (%)
76.0
74,6
75.5
75.0
75.5
75.0
75.1
2. 12 A, means "Joan five years older than Jessie."
1 a8 12 12-22-1-60 d. 8 menns "Mary two years younger than Joan."
73.9
85 means Jessie years
74.0
72.4
72.2
72.4
73
73.8
74.6
Task specific Solver Prompting
72.0
71.2
understanding you ressoning in Explish
an expert artthmeti
70.4
CLSP
70.0
Let's reolvv the taok you understand above stop stsstpp Finallly you should format your answer as Waswen [uum]].
68.0
Sum! Let's solve the task stop stypp
6
7
8
9
10
11
1. A Accortiing to the listt sentence, Mary is two years younges than jann Let's suppose Joan's is then Mary's age is] -22. According the
The Number of Languages
years older Jessie. It Jessie's age is then, Joan's age is S... newer.
semtenoo, Joan
Figure 9: The impact of integrating languages on the final performance. Different colors represent different integration language sequences.
(b) The reasoning result for cruss: lingual chain-of thought prompting.
Figure 10: Case Study.
resource languages, performance decreases with an increasing number of languages. These findings emphasize that the effectiveness of language inte- gration not solely determined by the number of languages integrated. Quantity of pretraining data for each language, especially in high-resource lan- guages, play a crucial role. Bulancing multiple lan- guages considering available resources and impact is vital. This research provides valuable insights for developing multilingual models that strike balance between incorporating diverse languages and maintaining nigh-performance standards.
the information in the query. This indicates that our proposed CLP can simulate the model's abil- ity to understand the cross-lingual query clearly before attempting to solve the problem. And this capability is essential because if the misunderstood happened, the final result may also be erroneously inferred in a high probability. This observation further validates the effectiveness of CLP
5.4.4 Extension to XCOPA
To further verify the effectiveness of CLSSP, we con- duct experiments on XCOPA (Ponti et al., 2020), a widely adopted dataset for assessing commonsense reasoning skills across 1 different languages.
5.4.3 Qualitative analysis
To further understand why CLP works intuitively, we provide case study that compares the outputs generated by the traditional CoT approach and CLP. As depicted in Figure 10, we observe that the tradi- tional CoT fails to comprehend all the information present the query (missing the information about "Jessie is 20 years old"), thereby resulting in the error inference of the final result. However, our pro- posed CL. overcomes this limitation by first utiliz- ing the cross-lingual alignment prompting to ensure the model comprehensively understands the given query, which detailed aligns the source language to Then the task-specific solver prompting is implied to solve this problem step-by-ster without deviation from
As the results presented in Table 3, in compar- ison to the baselines. we observe significant av- erage improvement of 4.7% in CLP performance. And it even surpasses the results reasoning with translated requests by 1.8%. Furthermore, CLSP leads to an additional enhancement of 7.4% com- pared to CLP. These results signify that apart from excelling in mathematical reasoning, both CLP and CLSP demonstrate notable effectiveness in address- ing common-sense reasoning tasks.
6 Related Work
the target language sentence-by-sentene
Chain-of-Thought (CoT) (Wei et al., 2022; Ko- jima et al., 2022) is an effective and step- step-by-step by-

---

Model
ET
IT
QU
SW
TH
AVG
GPT-3 (fext-davinci-002)
Direct (Shi al., 2022)
73.8
55.6
88.8
95.4
51.2
56.0
54.6
70.2
88.6
80.4
91.4
73.3
En-CoI! (Shi ul.. 2022)
88.8
79.6
91.4
96.6
52.2
67.4
84.2
93.4
80.7
GPT-3.5 (gpt-3.5-turbo)
Direct (Ahuja al., 2023)
90.6
72.0
90.4
95.2
54.6
82.0
59.0
77.6
91.0
83.6"
90.4"
80.6
TransLate- (Ahuja et al., 2023)
88.2
79.4
90.8
94.4
50.0
77.6
82.2
87.8
88.4"
92.2"
83.5
CLP
89.6
79.4
94.2
92.8
63.6
84.8
73.4
87.8
91.2
90.8
91.2
85.3"
CLSP
96.8
90.6
95.2
95.8
85,8
92.8
83.2
93.2
96.8
94.2
95.8
92.7
HUMAN (Ponti et al., 2020)
98.2
96.4
100.0
97.0
94.8
99.0
98.6
98.2
96.4
98.4
96.6
97.6
Table 3: Main results of XCOPA, denotes the reproduction result.
denotes the 6-shot results.
strategy applied to Large Language Models (LLMs) for zero-shot and few-shot reasoning. CoT prompts, which can be a single instruction or a set of CoT examples, facilitate the generation of intermediate reasoning steps. Recently, series of studies (Zhou al., 2022; Wang al., 2022, 2023c; Khot et al., 2023) have proposed their respective prompting strategies, dividing the entire task into smaller sub- tasks and subsequently resolving, planning, and executing these subtasks. With the improvement in model capabilities, some works (Zelikman et al., 2022; Zhou et al., 2023; Hu et al., 2023; Gao et al., 2023) treat instructions as "programs" for further search, execution, optimization. Building upon this, considering the feedback brought by execu- tion, ReAct (Yao et al.. 2023) and Reflexion (Shinn al., 2023) further explore the interactive gen- eration of inference decisions and task execution, thereby achieving greater synergy.
Prompting (CLSP) to enhance the performance by leveraging chained cross- cross-lingual pathways devised by experts in various languages.
Conclusion
In this work, we introduced cross-lingual prompt- ing (CLP) for cross-lingual Chain-of- Thoughh.. Specifically, CLP consists of cross-lingual align- ment prompting and task-specific solver prompting to align representations across languages and gen- erate the final reasoning paths in cross-lingual set- tings. In addition, we proposed cross-lingual self- consistent prompting (CLSP) to effectively lever- age knowledge across languages, which further boosts performance over CLP. Extensive experi- ments reveal that both CLP and CLSP can attain promising performance in cross-lingual CoT.
Acknowledgements
Cross-lingual Generalization Prior studies have demonstrated the benefits of pre-trained multilin- gual models diverse downstream tasks, such as cross-lingual spoken language understanding (Qin et al., 2020, 2022; Zheng et al., 2022) and cross- lingual summarization (Wang et al., b023a,b; Bhat- tacharjee et al.. 2023). Recently, with the emer- gence of Large Language Models (LLMs), non- training-based cross-lingual learning has gained more attention (Brown al., 2020; Ahuja al., 2023; Winata al., 2023; Zeng et al., 2023; Huang et al., 2023). Additionally, in the context of cross- lingual alignment, the current common practice in- volves employing few-shot learning to guide mod- els for better alignment (Winata et al., 2021; Shi et al., 2022; Tanwar et al., 2023; Lin et al., 2022). Compared to their work, we explore the zero- shot cross-lingual alignment CoT and introduce CL. to address this problem, which does not need any additional examples to be constructed. Fur- thermore, we explore Cross-lingual Self-consistent
This work was supported by the National Natu- ralScience Foundation of China (NSFC) via grant 62306342, 62236004 and 61976072. This work was also sponsored by CCF-Baidu Open Fund. We are grateful for resources from the High Perfor- mance Computing Center Central South Univer- sity. Libo Qin is the corresponding author.
Limitations
Consistent with the findings of Kojima et al. (2022); Zhu et al. (2023), our results also indicate that CLP exhibits varying performance improvements in rea- soning based on different prompts. Although all of these prompts enhance the performance, there are still significant performance disparities, with differences exceeding 4%. Therefore, enhancing the robustness of model alignment remains an ur- gent issue that needs to be addressed in the future.

---

References
multi-step reasoning. In The Eleventh International Conference on Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. 2022. Do as can, not as i say: Ground- ing language in robotic affordances. arXiv preprint arXiv:2204.0160 01691.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Gra- ham Neubig. 2023. PAI.: program-aided language models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pages 10764-10799 PMLR.
Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, et al. 2023. Mega: Multilingual evaluation of generative ai. arXiv preprint arXiv:2303. 12528.
Olga Golovneva, Moya Chen, Spencer Poff. Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. 2022. Roscoe: A suite of metrics for scoring step-by-step reasoning. arXiv preprint arXiv:2212.07919
Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ah- mad, Suan-Fang Li. Yong-Bi Bin Kang, and Rifat Shahriyar. 2023. CrossSum: Beyond English-centric cross-lingual summarization for 1, 1,500+ language pairs. In Proceedings of the ō1st Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 2541 1341-250 -25644, Toronto, Canada, Association for Computational Linguistics.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with pradient-disentangeee embedding shar- ing, arXiv preprint 2IX.v:2111.09543
Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-linguat capa- bilities English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 3563-3574, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim. Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at ma- chine translation? comprehensive evaluation. arXiv preprint arXiv:2302.09210.
Mengkang Hu, Yao Mu, Xinmiao Yu, Mingyu Ding, Shiguang Wu, Wenqi Shao, Qiguang Chen, Bin Wang. Yu Qiao, and Ping Luo. 2023. Tree-planner: Efficient close-loop task planning with large language models.
Tom Brown Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan. Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Haoyang Huang, Tianyi Tang, Dongdong Zhang. Wayne Xin Zhao, Ting Song. Yan Xia, and Furu Wei. 2023, Not all languages are created equal in 1mss: Improving multilingual capability by cross- gooos--ingual-thouse prompting. arXiv preprint arXiv:2305.0700
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint 0rXiv:220..02311.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet. Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and brian ichter. 2022. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conference on Robot Learning.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Ad- ina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. 2018 Xnli: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computa- tional Linguistics.
Jacob Devlin, Ming-Wo- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- Pre-training of deep bidirectional transformers for language under- standing, In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-48 14716 Minneapolis, Minnesota. Association for Computational Linguistics.
Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sab- harwal. 2023. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representa- rions, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. In Advances in Neural Information Processing Systems.
Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023. Complexity-base based prompting for

---

Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Na- man Goyal, Shruti Bhosale, Jingfei Du Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' O'Horo, Jeff Wang. Luke Zettle- moyer, Zornitsa Kozareva, Mona T. Diah, Veselin Stoyanov, and Xian Li. 2022. Few-shot learning with multilingual generative language models. In Proceed- ings of the 2022 Conference on Empirical Methods Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 9019-9052. Association for Computational Linguistics.
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang. Suraj Srivats, Soroush Vosoughi, Hyung Won Chung. Yi Tay, Sebastian Ruder. Denny Zhou, et al. 2022. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.030522
Noah Shinn, Federico Cassano, Edward Berman, Ash- win Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language agents with verbal rein- forcement learning.
Dan Malkin, Tomasz Limisiewicz, and Gabriel Stanovsky. 2022. A balanced data approach for eval- uating cross-lingual transfer: Mapping the linguistic blood bank. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 4903-4915 Seattle, United States. Association for Computational Linguistics.
Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. 2023. Multilingual 1mms are better cross- pooss-lnggual in-context learners with align- ment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 6292-6307. Association for Computational Linguistics.
Niklas Muennighoft, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf. et al. 2022. Crosslingual generaliza- tion through multitask finetuning. arXiv preprint 20l02226...
Hugo Touvron, Louis Martin, Kevin Stone, Peter AL- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, An- thony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux. Thibaut Lavril, Jenya Lee, Di- ama Liskovich, Yinghai Lu, Yuning Mao, Xavier Mar- tinet, Todor Mihaylov, Pushkar Mishra, Igor Moly- bog. Yixin Nie, Andrew Poulton, Jeremy Reizen- stein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subrama- nian, Xiaoqing Ellen Tan, Binh Tang, Ross Tay- lor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan. Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Ro- driguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine- tuned chat models.
Wenbo Pan. Qiguang Chen, Xiao Xu, Wanxiang Che. and Libo Qin. 2023. A preliminary evaluation of chatgpt for zero-shot dialogue understanding.
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulié, and Anna Korhonen. 2020. XCOPA: A multilingual dataset for causal common- sense reasoning. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 2362-2376, Online. As- sociation for Computational Linguistics.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language process- ing task solver? arXiv preprint arXiv:2302.06477
Libo Qin, Qiguang Chen, Tianbao Xie, Qixin Li, Jian- Guang Lou, Wanxiang Che, and Min- Yen Kan, 2022. GI-clef: A global-local contrastive learning frame- work for cross- lingual spoken language understand- ing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2677-2686. Association for Computational Linguistics.
Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Cross-lingual summarization via chatgpt. ArXiv, abs/2302.14229,
Jiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li. Jianfeng Qu, and Jie Zhou. 2023b. Towards unifying multi-lingual and cross-lingual summarization. In Proceedings of the 61 Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 15127- 15143, Toronto, Canada. Association for Computa- tional Linguistics.
Libo Qin, Minheng Ni, Yue Zhang, and Wanxiang Che. 2020. Cosda-ml: Multi-lingual sode-switching data augmentation for zero-shot cross-lingual NLP. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020, ijcai.org.
Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023c. Plan-and-solve prompting: Improving zero- shot chain-of thouught reasoning by large language
pages
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.

---

models. In Annual Meeting of the Association for Computational Linguistics.
on Artificial Intelligence, IJCAI 2023, 19th-25th Au- gust 2023, Macao, SAR, China, pages 6290 6298. ijcai.org.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2022. Self-consistency im- proves chain of thought reasoning in language mod- els. arXiv preprint 2203.:2203..1171..
Bo Zheng, Zhouyang Li, Fuxuan Wei, Qiguang Chen, Libo Qin, and Wanxiang Che. 2022. HIT-SCIR at MMNLU-22: Consistency regularization for multilin- gual spoken language understanding. In Proceedings of the Massively Multilingual Natural Language Un- derstanding Workshop (MMNLU-22), pages 35-41. Abu Dhabi, United Arab Emirates (Hybrid). Associa- tion for Computational Linguistics.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le, and Denny Zhou. 2022. Chain thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang. Dale Schuurmans, Olivier Bousquet, Quoe Le, and Ed Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint arXiv:2205.10625.
Genta Winata, Alham Fikri Aji. Zheng Xin Yong, and Thamar Solorio. 2023. The decades progress code- switching research in NI.P: A systematic survey on trends and challenges. In Findings of the Associa- tion for Compulational Linguistics: ACL 2023, pages 9936,2978, Toronto, Canada. Association Com- putational Linguistics.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023. Large language models are human-level prompt engineers. In The Eleventh International Conference on Learning Representations. ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021. Language models are few-shot multilingual learners. CoRR, abs/2109.0768..
Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang. Linyi Yang. Wei Ye. Neil Zhenqiang Gong, Yue Zhang, et al. 2023, Promptbench: Towards evaluating the robustness of large language models on adversarial prompts. arXiv preprint vrXiv:2306.04522
Yinfei Yang. Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingua adversar- ial dataset for paraphrase identification, In Proceed- ings of the 2019 Conference Empiriaal Methods Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLP), pages 3687-3692, Hong Kong, China. Association for Computational Linguis- tics.
Zhuang Ziyu, Chen Qiguang. Ma Longxuan, Li Mingda, Han Yi, Qian Yushan, Bai Haopeng, Zhang Weinan, and Ting Liu. 2023. Through the lens of core compe- tency: Survey on evaluation of large language mod- cls. In Proceedings of the 22nd Chinese National Conference Computational Linguistics (Volume 2: Frontier Forum), pages 88-109, Harbin, China. Chinese Information Processing Society of China.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Fleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.nee
Ping Yu, Tianlu Wang. Olga Golovneva, Badr AlKhamissi, Siddharth Verma, Zhijing Jin, Gargi Ghosh. Mona Diab, and Asli Celikyilmaz. 2023.
ALERT: Adapt language models to reasoning tasks. In Proceedings of the ō1st Annual Meeting of the Association for Computational Linguistics (Volume Long Pupers), pages 1055-1081, Toronto, Canada. Association for Computational Linguistics.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Good- man. 2022. Star: Bootstrapping reasoning with rea- soning. Advances in Neural Information Processing Systems, 35: 5546-154888
Qingcheng Zeng, Lucas Garay, Peilin Zhou, Dading Chong, Yining Hua, Jiageng Wu, Yikang Pan, Han Zhou, Rob Voigt, and Jie Yang. 2023. Greenplm: Cross-lingual transfer of monolingual pre-trained lan- guage models at almost no cost. In Proceedings of the Thirty-Second International Joint Conference

---

A Appendix
Info-Step assigns higher score reasoning steps that demonstrate strong alignment with the source, thereby indicating the extent to which the generated hypothesis incorporates the information from the source. Conversely, a lower Info-Step score indicates reasoning steps that are unrelated to the source sentences or overlook the provided information in the context.
A.1 Robust Analysis Implementation
In order to further verify the robustness of CLP, conducted an analysis of the final results for vari- ous CLPs with different expressions. Specifically, we utilize GPT3.5 to generate guiding prompts synonymous with "Let's understand the task in En- glish step by step!". Our instruction is as follows:
A.2.3 Informativeness Chain
Assuming you are professional rewriter, you need to modify the following request into three different versions:.
Just like the Info-Step metric, the Informativeness- Chain (Info-Chain) metric measures the extent of concordance between the hypothesis chain and the source. The calculation as follows:
Let's think in [Target Language L+]] step by step!
Info-Chain 1 cos 8)
The final generated prompt and corresponding results are shown in Table
(8)
2
To facilitate this computation, we treat the reason- ing chain and the source context as an integrated entity.
A.2
Shain-of-Thhuughe Quality Scoring Implementation
The RoscoE framework (Golovneva et al., 2022) incorporates multiple chain-of-thought thought quality met- rics, with the reasoning alignment vector
A.2.4 Missing Step
To pinpoint any significant steps that could be lack- ing in the hypothesis, (Golovneva al., 2022) intro- duce the Missing Step (Miss-Step) metric, which examines the alignment between the reference rea- soning text r {r:) and the hypothesis h. Miss-Step needed to meticulously assess each step in the reference and verify the existence of a similar step in the hypothesis. The metric is com- puted as:
r-align(h + s) far, 0:2--.. ... .N3 E [0, 1 from the N-step hypothesis = { to the source input s of length T, where a; are defined as:
r-align(h, + +) max"" ==1 cos(h 83)
(5)
2
A.2.1 Faithfulness
The Faithfulness (F) score is calculated based on the alignment between the hypothesis steps h and the source sentences 8. It represents the average reasoning alignment score over the steps of reason- ing:
K Miss-Step min (r-agnnll + h)). (9)
A.2.5 Multi-lingual Setting Due to the limited support of the original RoscOE (Golovneva et al., 2022) framework for monolingual English, expanded RoscoE operate in cross-lingual setting to enhance the sessment of Cross-lingual CoT's inference quality. For the backbone of sentence similarity compu- tation the model, we employed a multilingual variant of MP-Net (Reimers and Gurevych, 2019).
F raaignn(h; s).
The Faithfulness score serves measure as- sess whether the model misconstrued the problem statement or if the reasoning chain is characterized by vagueness, irrelevance, or the misuse of infor- mation.
A.3 Reasoning Alignment Scoring
A.2.2 Informativeness Step
A.3.1 Metric Definition
Informativeness-- (Info-Step) measures the uti- lization of information from the source text in the reasoning steps h:
Reasoning Alignment Scoring (RAS) offers a sim- ple method to evaluate the accuracy of the hypoth- esis chain by examining the extent of overlap be- tween the hypothesis and the reference. One ap-
T
Info-Step- 1
1 -1) F (7) httpp://huuggnngfacccccoeeeeeeee
2T

---

Cross- Lingull Prompting
bn
de
8s
fr
ja
ru
SW
te
th
AVG
Let's understand the task in English step by step!
64.8
80.0
81.6
79.2 69.2
82.4
74.8
38.8
62.0
73.6
70.6
We should grasp the task English by breaking it down into steps.
50.8
69.6 84.8 82.4 77.6 80.8 73.2 36.0 61.6 73.6
69.0
Step by step, let's comprehend task English!
60.8
76.8
82.4
76.0
78.8
67.6
40.8
57.6
72.4
68.3
Our approach should involve understanding the lask in Figlish through step-by-star process.
56.4
80.8 77.6
81.6
66.4
72.8
66.4
35.6
63.2
64.0
66.5
GPT3.5 (En-Con CoT)
50.0
73.6 69.6 70.0 60.4 55.2 22.0 48.0
57.8
Table 4: Performance different prompts in CLP.
Model
bn
de
fr
ru
SW
te
th
2h
AVG
CLP
65.0
80.0
82.0
79.0
63.0
84.0
63.0
44.0
60.0
70.0
69.0
CLP(3-sho))
76.0
85.0
84.0
75.0
87.0
61.0
84.0
CLP(3-sho) +Complex-Cof (Fu et al., 2023)
71.0
89.0
85.0
81.0
86.0
88.0
72.0
50.0
61.0
87.0
77.0
Table Additional Experiment Few-shot Setting.
proach to achieving this is by quantifying the rea- soning alignment between the two, which can be calculated as:
ing from es to sw, ja to sw, and ru to te demonstrates more unique reasoning paths (S 0.805).
A.4
Strategy Definition
N N
In our deep exploration, we find that CLP not only serves as simple translation but also has seven dif- ferent strategies, which are summarized below:
RAS
a-iiinnh::-
(10)
A.3.2 Implementation Setting
Step-by-step Translation: The model divides the translation process into steps based on commas or periods and translates them step by step, as illustrated in Figure 10.
Since completely incorrect reasoning can also lead to a significant decrease in RAS, we conducted the experiments in Figure 7 by excluding all samples with prediction errors and only calculating RAS on correctly predicted samples.
Key Information Extraction: The model first extracts key terms critical conditions from the request for translation. This aids the model achieving better cross-lingual align- ment.
In Figure 7 (a), we selected English the get language and generated seven CoT reasoning results by adjusting the model's output tempera- ture. We calculated the RAS between the reason- ing step outputs of each correctly predicted sample and the standard reasoming step outputs. By av- eraging the RAS of all samples, we obtained the comprehensive RAS for source-to-English compre- hension. Similarly, in Figure 7 (b), chose high-resource language as the target language and obtained seven CoT reasoning results. We com- puted the RAS between the reasoning step outputs each correctly predicted sample and the standard reasoning step outputs, and then averaged the RAS of all samples to obtain the comprehensive RAS for source-to-tareee tortareet language comprehension.
This strategy indicates that CLP starts preliminary mathematical op- erations based on comprehension. It may even provide answers during the alignment phase. However, the model's second stage may mod- ify this answer, so it is not the final solution.
Preliminary Solution:
Complete translation: This strategy indi- cates that the model directly performs ma- chine translation of the request without sen- tence splitting or step-wise operations.
This strategy indicates that the model rephrases the request. Unlike machine translation, problem restatement quires the model to infer, add its understand- ing, and include information inferred from the request through reasoning.
Problem restatement:
Overall, the CLSP exhibits stronger diversity in reasoning paths, particularly in the original lan- guage reasoning of zh, ja, and de, which shows a higher similarity to the original reasoning paths (2 0.845). On the other hand, cross-lingual reason-

---

# Strategy
bn
de
es
fr
ja
ru
SW
te
th
zh
AVG
76.0
85.0
84.0
75.0
80.0
87.0
68.0
61.0
59.0
84.0
75.9
65.0
85.0
82.0
73.0
66.0
80.0
69.0
46.0
57.0
68.0
69.1
57.0
78.0
73.0
73.0
59.0
74.0
57.0
40.0
59.0
65.0
63.5
Table 6: Performance of different number of strategies. Strategies are adopted from large to small according to the strategy proportion in Table
Strategy
CLF Acc.
Native-Co" Acc.
Ratio (8)
Furthermore, in order to explore the impact of different examples on CLP, we further analyze the impact that examples using different alignment strategies mentioned in Section 5.3.3 can have downstream tasks. We manually annotate the dev and used multiple strategies for annotation. Ex- periments in Table show that as the diversity of strategies increases, the performance the model gradually increases.
Step by step Translation
61.11
Key Informatoo Extraction
60,00
60.00
5,00
Preliminary Solution
61.11
54.53
$4,00
Complete Transationn
58.33
50.00
24.00
Problem Restatement
$7.28
50.49
$1.50
Step Division
51.06
23.50
Uode-witching
62,50
62.50
4.00
Denial Service
50.00
42.86
7.00
Table 7: The effectiveness and distribution of the differ- ent strategies. There are more details Appendix A.4.
Step Division: This strategy encompasses two situations: (1) The model actively divides the cross-lingual alignment process into multiple steps. For example, it will divide the align- ment process into "Step Identify the context and topic", "Step Translate the sentence" and "Step 3: Analyze the sentence structure". (2) The model actively plans the next task and divides the request into several sub-questions.
This strategy indicates that the model actively replaces certain words in the text with words from the target language.
Code-switching:
Denial of Service: ChatGPT refuses to per- form cross-lingus alignment and delegates alignment directly to the second stage.
A.5 Few-shot Setting
In order to verify the effeet of CLP on ICL, we fur- ther designed experiments with few-shot settings for analysis. Specifically, samples of data from MGSM test set for testing. In the alignment stage, we immediately used some ex- amples from the dev set to construct cross-language alignment examples. The results of these examples were all generated by GPT3.5. We only keep the сorrect answers as examples.
first selected 1,000
In the problem-solving phase, we further used the example of Complex-CoT example. The results in Table 5 show that the two- stage ICL can better promote the performance of the model. This also illustrates the versatility of CLP and its ability to orthogonal to other prompt optimization solutions.
problem-solving


Reviews:
[{"review_id": "2NDjXd3v4l", "reviewer": "Reviewer_mrxR", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your acknowledgements and interests on our work! We sincerely appreciate your thorough comment of our work.\n\n**Question1**: More datasets on traditional multilingual tasks like XNLI, XTREME\n\n**Answer1**: Thanks for your constructive suggestion. In this work, we follow the same setting as Shi et al [1] to consider the MGSM and XCOPA task. We acknowledge your suggestion regarding the XNLI, XTREME task to valide the different levels of reasoning requirements and follow your suggestion to add more discussion in the next version.\n\n**Question2**: adding some open-source LLMs for better reproducibility\n\n**Answer2**: Thanks for your insightful comment. We totally agree with your point and we will add some open-source LLMs in the next version to promote the community. In addition, all codes used in our work will be made public to facilitate the research.\n\n**Question3**: Small typo\n\n**Answer3**: Thanks for your kind mention. We will follow your suggestions to polish our work in the next version.\n\n[1] Shi et al., Language Models are Multilingual Chain-of-Thought Reasoners.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper studies an important problem -- multilingual zero-shot CoT prompting and presents a novel approach to address this problem. The authors introduce two main components of cross-lingual prompting (CLP): cross-lingual alignment prompting and task-specific solver prompting, as well as cross-lingual self-consistent prompting (CLSP) to ensemble different reasoning paths across languages. The experimental evaluations on the multilingual reasoning benchmark MGSM and XCOPA demonstrate that CLP and CLSP significantly outperform existing prompting methods and achieve state-of-the-art performance. The paper provides a clear and detailed description of the proposed approach and experimental results, and the findings have important implications for improving cross-lingual reasoning in a wide range of applications. ", "reasons_to_reject": "Overall I feel the paper is good -- clearly written and the proposed method is intuitive. A few places that can be further improved include:\n1. More datasets on traditional multilingual tasks like XNLI, XTREME, to show the proposed technique can generalize to tasks with different levels of reasoning requirements.\n2. Consider adding one experiment on open-source LLM as the current GPT-series and PaLM v1 is somewhat hard to reproduce entirely from the outsider.\n3. Small typo around line 90 -- \"Let’s resolver the task ...\" to \"Let's resolve the task\"\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces a new technique called cross-lingual prompting (CLP) that aims to improve zero-shot chain-of-thought reasoning across languages. By aligning representations across different languages and generating task-specific solver prompting, CLP can help improve reasoning accuracy in a wide range of tasks. the author also introduce  cross-lingual self-consistent prompting (CLSP) to ensemble different reasoning paths across languages. The experiments show that CLP and CLSP can significantly improve reasoning accuracy across languages, and that CLSP is particularly effective in improving reasoning accuracy for low-resource languages.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NghRTmXoTn", "reviewer": "Reviewer_prAk", "summary": "", "questions": "1) The proposed method employs alignment prompting to obtain understanding in English. The aim is to compare this method with the baseline approach of first obtaining an understanding of the source language and then translating it into the target language.\n\n2) According to Table 2, the results indicate that performance is influenced by the specific wording used in prompts. It is important to know how to find the optimal wording to use.\n\n3) How well does the proposed prompting perform in the few-shot setting?\n\n4) How well does the proposed prompting perform with smaller LLMs like mT5, BLOOM, GPT-Neo, LLaMa, and ChatGLM?\n\nI will update the final score according to the authors responses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your acknowledgements and interests on our work! We sincerely appreciate your thorough comment of our work and we will address each of your concerns below:\n\n**Question1**: literature review needs improvement\n\n**Answer1**: Thanks for your kind mention. We will follow your suggestion to add more related literature work in the next version, including zero-shot CoT prompting and cross-lingual transfer learning.\n\n**Question2**: It is important to know how to find the optimal wording to use.\n\n**Answer2**: Thanks for your insightful comment. Yes, we totally agree with your perspective and this is an interesting research question to investigate. In our work, we drew inspiration from ChatGPT, a conversational LLM, and devised a multi-turn iterative prompting approach to enhance the multi-round interactive understanding capability of the LLM. Another design principle that I think to be quite important is considering task-aware prompts to better elicit the capabilities of large-scale models for solving specific task.\n\n**Question3**: How well does the proposed prompting perform in the few-shot setting?\n\n**Answer3**: Thanks for your constructive comment. Actually, we conducted the few-shot setting by providing some demonstrations in the preliminary experiments. The results demonstrate that adding demonstrations can bring improvement compared to the zero-shot setting, which is consistent with previous observations [1-3]. We will follow your suggestion to add more discussion in the next version.\n\n**Question4**: How well does the proposed prompting perform with smaller LLMs like mT5, BLOOM, GPT-Neo, LLaMa, and ChatGLM?\n\n**Answer4**: Thanks for your insightful comment. This is a very good question. Yes, we also consider the setting. When we apply mT0 in our work, we find that the results are close to 0, and there is no relevant cross-lingual CoT phenomenon observed, which is consistent with the previous observation [4,5]. We speculate that smaller LLM models might struggle with zero-shot cross-modal CoT scenarios. We will include more discussion in the next version, hoping to contribute some meaningful insights to this field.\n\n[1] Min et al., Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? EMNLP2022.\n\n[2] Wei et al., Finetuned language models are zero-shot learners. ICLR2023.\n\n[3] Li et al., In-Context Learning with Many Demonstration Examples. arxiv2023.\n\n[4] Li et al., Symbolic Chain-of-Thought Distillation: Small Models Can Also “Think” Step-by-Step. ACL2023.\n\n[5] Xue et al., RCOT: Detecting and Rectifying FactualInconsistency in Reasoning by Reversing Chain-of-Thought. arxiv2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The proposed Cross-lingual Self-consistent Prompting approach is intuitive, simple, and effective.\n\n2) The experiments conducted on multiple datasets and the comparison with various baselines demonstrate the effectiveness of the proposed approach.", "reasons_to_reject": "1) The literature review needs improvement. Numerous papers on zero-shot CoT and cross-lingual transfer learning have been published this year. The paper should incorporate more related works on zero-shot CoT prompting and cross-lingual transfer learning.\n\n2) This paper should verify the proposed prompting on more cross-lingual datasets mentioned by previous work, such as MKQA, XNLI, and XL-WiC.", "questions_to_authors": "1) The proposed method employs alignment prompting to obtain understanding in English. The aim is to compare this method with the baseline approach of first obtaining an understanding of the source language and then translating it into the target language.\n\n2) According to Table 2, the results indicate that performance is influenced by the specific wording used in prompts. It is important to know how to find the optimal wording to use.\n\n3) How well does the proposed prompting perform in the few-shot setting?\n\n4) How well does the proposed prompting perform with smaller LLMs like mT5, BLOOM, GPT-Neo, LLaMa, and ChatGLM?\n\nI will update the final score according to the authors responses.", "paper_topic_and_main_contributions": "The paper proposes a novel approach called Cross-lingual Prompting (CLP) to generalize the success of zero-shot chain-of-thought (CoT) reasoning to different languages. The paper presents two components of their approach: (1) Cross-lingual Alignment Prompting, which aligns representations between different languages using an English understanding prompt, and (2) Task-specific Solver Prompting, which completes the final task by using a resolver prompt. Additionally, this paper proposes Cross-lingual Self-consistent Prompting (CLSP) to ensemble reasoning paths across languages. The experiments are conducted on the MGSM and XCOPA datasets, and the results demonstrate that CLP achieves state-of-the-art performance with an average gain of over 5 points compared to all baselines.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Zero-Shot CoT Promptings\n\nCross-lingual Transfer Learning.", "Typos_Grammar_Style_and_Presentation_Improvements": "1) Line 52: Think -> think\n2) Line 142: Cross -> cross"}, "extra_text": {}}, {"review_id": "NrYwD3phKG", "reviewer": "Reviewer_wcFZ", "summary": "", "questions": "A. Table 1: Why are EN-CoT, Native-CoT missing for GPT3.5 backbone?\n\nB. What is the difference between \"Cross-lingual Alignment\" and \"Translating each step of the request from source language to English\"? The sample in Figure 2,9 creates the sense that the LLMs are prompted to simply conduct translation rather than understanding of the request. What is the multilingual understanding here?\n\nC. Figure 6: The difference between demonstrated alignment scores on CLSP and SC does not seem significant. What is the significance mentioned in Line 354-358? In addition, as the calculation of the score is dependent on the correctly predicted samples, will the calculation result in incomparable scores among evaluated languages?\n\nD. How does leveraging Complex-CoT [1] instead of SC-CoT affect the performance?\n\nE. Figure 8: One might argue the drop in performance when adding later languages might be due to the capacity of LLMs, leading to forgetting/ slower adaptation of LLMs. How is the performance change  when adding the languages in the reverse order (i.e. low-resource to high-resource languages)? This addition helps verify the claim \"adding low-resource languages decreases the performance with increasing number of languages\" (Line 382-384).\n\n[1] Fu et al., Complexity-Based Prompting for Multi-step Reasoning. ICLR 2023", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful and comprehensive feedback on our work. We sincerely appreciate your valuable insights and we will address concerns point to point. \n\n**Qustion1**: Table 1: Why are EN-CoT, Native-CoT missing for GPT3.5 backbone?\n\n**Answer1**: Thanks for your kind mention. Actually, we had included the EN-COT and Native CoT result for GPT3.5, which is illustrated in Table 1(in the fourth line from the bottom) and Figure 10 (in Appendix). For a more fair comparison, our Native-CoT also adopts a two round setting, with the only difference from CLP being the lack of cross language alignment. This is different from the single round settings of other baselines, so it is not placed in the main table. We will follow your suggestion and add Native-CoT to Table 1 in the next version to make the work more clearer.\n\n**Question2**: Difference between \"Cross-lingual Alignment\" and \"Translating each step of the request from source language to English\"?\n\n**Answer2**: Thanks for your insightful comment. We sincerely think that Cross-lingual Alignment is not solely a simple translation. As shown in Figure 2, Cross-lingual alignment proactively divides each step (玛丽比琼小两岁, 琼比杰莎大五岁, 如果杰莎 20 岁…) and progressively translates them (\"Mary is two years younger than Joan.\", \"Joan is five years older than Jessie. \"). The behind intuition of Cross-lingual Alignment is that when the model can gradually understand the problem correctly across languages, it can have better cross-language alignment ability. We will add more discussion in the next version.\n\n**Question3**: Figure 6: The difference between demonstrated alignment scores on CLSP and SC does not seem significant.\n\n**Answer3**:Thanks for your constructive feedback. Due to the fact that most of the gold paths for solving mathematical problems are consistent, most of the predicted correct CoT inference paths are also consistent, which leads to very high alignment scores. Therefore, the alignment scores from 0.80 to 0.84 in the figure are significant to this certain extent, which is consistent with the observation of [1].\n\n[1] Yu et al., ALERT: Adapt Language Models to Reasoning Tasks. ACL2023\n\n**Question4**: How does leveraging Complex-CoT [1] instead of SC-CoT affect the performance?\n\n**Answer4**: Thanks for your insightful comment. Complex CoT is an promising approach using a series of hard and multi-step reasoning ICL examples, which can greatly enhance the model's complex logical reasoning ability. Our work is a plug-and-play module that can be completely orthogonal to Complex CoT. We will follow your suggestion to explore the role of Complex CoT in our framework. Specifically, we will replace the simple “Let’s resolve the task step by step” trigger to the hard and multi-step examples selected from [1] for better reasoning.\n\n[1] Fu et al., Complexity-Based Prompting for Multi-step Reasoning. ICLR 2023\n\n**Question5**:How is the performance change when adding the languages in the reverse order (i.e. low-resource to high-resource languages)?\n\n**Answer5**: Thank you for your thoughtful suggestion. We have performed experiments by integrating non-English languages in reverse order, from low to high resources. We found two obvious phenomena.  First, the performance of the integration has dropped significantly, and the performance is even lower than that of pure English CLSP, which shows that the integration of low-resource languages is harmful to the performance improvement of the model. Second, we noticed that the integration performance from low resource to high resource increases gradually, but the integration performance from high resource to low resource decreases finally, which further verifies our claim that the integration of high-resource languages is more critical to the performance improvement of CLSP. We will add more analysis in the next version.\n\n**We would greatly appreciate it if you can re-consider the work considering our clarification.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well-written. The proposed method is intuitive and easy to follow.\n\n2. Extensive empirical studies are conducted. \n\n3. Promising improvements over the baselines are quite significant across different languages.", "reasons_to_reject": "1. The claim that the proposed method is plug-and-play is not sufficiently substantiated. Although 3 backbone models are presented (GPT-3, PaLM-540B and GPT3.5), the empirical study is only based on GPT3.5\n\n2. The method misses an important ablation study of comparing the proposed method with  GPT3.5-Translate-EN: Non-English input request is translated to English via GPT3.5 Machine Translation Engine (Table 1). It is possible the improvements are mostly done via the quality of MT from GPT3.5. \n\n3. Some claims are not fully substantiated by the empirical studies (See Questions).", "questions_to_authors": "A. Table 1: Why are EN-CoT, Native-CoT missing for GPT3.5 backbone?\n\nB. What is the difference between \"Cross-lingual Alignment\" and \"Translating each step of the request from source language to English\"? The sample in Figure 2,9 creates the sense that the LLMs are prompted to simply conduct translation rather than understanding of the request. What is the multilingual understanding here?\n\nC. Figure 6: The difference between demonstrated alignment scores on CLSP and SC does not seem significant. What is the significance mentioned in Line 354-358? In addition, as the calculation of the score is dependent on the correctly predicted samples, will the calculation result in incomparable scores among evaluated languages?\n\nD. How does leveraging Complex-CoT [1] instead of SC-CoT affect the performance?\n\nE. Figure 8: One might argue the drop in performance when adding later languages might be due to the capacity of LLMs, leading to forgetting/ slower adaptation of LLMs. How is the performance change  when adding the languages in the reverse order (i.e. low-resource to high-resource languages)? This addition helps verify the claim \"adding low-resource languages decreases the performance with increasing number of languages\" (Line 382-384).\n\n[1] Fu et al., Complexity-Based Prompting for Multi-step Reasoning. ICLR 2023", "paper_topic_and_main_contributions": "The authors tackle the problem of Chain of Thought (CoT) Reasoning in a multilingual setting. Specifically, the authors propose extending CoT to multiple languages via 2 major sequential prompting components (1) Cross-lingual Alignment Prompting and (2) Task-specific Solver Prompting. While the former focuses on aligning representations across different languages, the latter's goal is to generate CoT outcomes for the reasoning task.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Fu et al., Complexity-Based Prompting for Multi-step Reasoning. ICLR 2023", "Typos_Grammar_Style_and_Presentation_Improvements": "Line 289: demonstrate"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel approach called Cross-Lingual Prompting (CLP)  to improve zero-shot chain-of-thought reasoning across language. The experimental analysis show improvements over multilingual reasoning benchmarks MGSM and XCOPA.  The paper is well written and the approach and experiments are described well. 

The draft can be improved by including more related literature work, adding discussion on few-shot prompting and including the analysis discussed in the rebuttal.
Decision: Accept-Main