AUTOCAP: Towards Automatic Cross-lingual Alignment Planning for Zero-shot Chain-of-Thought (2024)

Abstract: Cross-lingual chain-of-thought can effectively complete reasoning tasks across languages, which gains increasing attention. Recently, dominant approaches in the literature improve cross-lingual alignment capabilities by integrating reasoning knowledge from different languages. Despite achieving excellent performance, current methods still have two main challenges: (1) Manual language specification: They still highly rely on manually selecting the languages to integrate, severely affecting their generalizability; (2) Static weight allocation: Current methods simply integrate all languages equally. In fact, different language reasoning paths should have different weights to achieve better complementation and integration. Motivated by this, we introduce an Automatic Crosslingual Alignment Planning (AUTOCAP) for zero-shot chain-of-thought to address the above challenges. The core of AUTOCAP consists of two components: (1) Automatic Language Selection Prompting to guide LLMs to select appropriate languages and (2) Automatic Weight Allocation Prompting to automatically allocate alignment weight scores to each reasoning path. Extensive experiments on several benchmarks reveal that AUTOCAP achieves state-of-the-art performance, surpassing previous methods that required manual effort.

heading: 1 Introduction
text: Large language models (LLMs) have achieved substantial advancements (Brown et al., 2020; Chen et al., 2021; Qin et al., 2024a). Particularly noteworthy is the emergence of the Chain-of-Thought (CoT), which has further enhanced the ability of LLMs to handle complex reasoning tasks (Wei et al., 2022; Wang et al., 2022). In addition, as globalization continues to advance, aligning representations across different languages has become an urgent issue (Pires et al., 2019; Mulcaire et al.,
∗Equal Contribution. † Corresponding Author.
2019; Qin et al., 2024b). This has motivated researchers to explore Cross-lingual CoT, aiming to break down language barriers by integrating CoT from different languages (Qin et al., 2023; Chai et al., 2024).
Specifically, Shi et al. (2022) introduce a multilingual dataset for mathematical reasoning and propose a method requiring LLMs to use English for CoT prediction and problem-solving. Qin et al. (2023) develop a two-stage framework to aid LLMs in understanding problems across languages through manual language specification and reasoning. Ranaldi and Zanzotto (2023) present a crosslingual, multi-step reasoning approach with a selfconsistent prompting mechanism to enhance reasoning in multiple languages. Huang et al. (2023) introduce cross-lingual thought prompting using a generic template to enhance the reasoning capa-
9191
bilities and performance of LLMs on multilingual tasks. In addition, recent cross-lingual CoT research has significantly improved complex reasoning across languages by aligning representations and integrating diverse linguistic reasoning paths (Shi et al., 2022; Tanwar et al., 2023; Qin et al., 2023).
Despite their improved performance, as shown in Figure 1 (a), the current approaches still face two key challenges:
(1) Manual language specification: The process continues to depend heavily on the manual selection of languages for integration, which not only expends substantial human effort but also significantly hurts its generalizability;
(2) Static weight allocation: The current methods simply integrate all languages equally, leading to sub-optimal performance. Actually, to achieve better integration of knowledge across languages, different language reasoning paths should possess different weights relative to the query.
In this paper, we introduce an automatic crosslingual alignment planning (AUTOCAP) framework to address the above challenges. Specifically, as shown in Figure 1 (b), AUTOCAP comprises two key modules: (1) Automatic Language Selection Prompting and (2) Automatic Weight Allocation Prompting. In more detail, Automatic Language Selection Prompting is applied to enable LLMs to automatically select the most accurately aligned languages for reasoning for each query. After selecting the languages, Automatic Weight Allocation Prompting is used for automatically allocating an alignment weight score to each language reasoning path. Finally, accurate reasoning answers can be obtained by integrating the CoT reasoning paths across different languages and their corresponding weight scores.
Experimental results on several benchmarks show that AUTOCAP achieves superior performance compared to previous baselines, even surpassing previous manually selected language methods. In addition, extensive analysis demonstrates the strong generalization ability of AUTOCAP.
In summary, our key contributions include:
• We introduce Automatic Cross-lingual Alignment Planning (AUTOCAP), which greatly alleviates the burden of manually selecting languages and weights.
• The core of AUTOCAP comprises Automatic Language Selection Prompting and Automatic Weight Allocation Prompting, which achieves to automatically select the most appropriate languages and weights for cross-lingual CoT.
• Extensive experiments on several benchmarks demonstrate that AUTOCAP surpassed the previous approaches, achieving state-of-theart performance and exhibiting strong generalizability.
All the code will be publicly available at https://github.com/BRZ911/AutoCAP.

heading: 2 Preliminaries
text: This section outlines the preliminaries of monolingual chain-of-thought (§2.1) and cross-lingual chain-of-thought (§2.2).

heading: 2.1 Mono-lingual Chain-of-Thought
text: Mono-lingual Chain-of-Thought (Wei et al., 2022) triggers LLMs to directly reason step-by-step in the source language to solve tasks. Formally, when presented with a query Q expressed in the source language Lsrc, the LLMs generate a reasoning path, which can be simplified and represented as follows:
Rsrc = argmax P (R|Q, Lsrc), (1) where Rsrc denotes the generated reasoning path with multiple steps in the source language Lsrc. Following this, the LLMs produce the final results Y , which are obtained by:
Y = argmax P (y|Q, Lsrc,Rsrc). (2)

heading: 2.2 Cross-lingual Chain-of-Thought
text: For better cross-lingual generalization for CoT in multilingual scenarios, Qin et al. (2023) propose Cross-lingual Chain-of-Thought methods to align multilingual representations explicitly. Formally, given a query Q in source language Lsrc, experts manually select a target language Ltgt to serve as an anchor for cross-lingual alignment. The LLMs then generate an alignment A as follows:
A = argmax P (a|Q, Lsrc → Ltgt). (3) Subsequently, the LLMs produce the result R by:
Rtgt = argmax P (R|A, Ltgt). (4) Finally, the LLMs determine the final results Y based on the reasoning path Rtgt in Ltgt and generated alignment A:
Y = argmax P (y|A, Ltgt,Rtgt). (5)

heading: 3 Automatic Cross-lingual Alignment Planning
text: In this section, we introduce an Automatic Crosslingual Alignment Planning (AUTOCAP) framework to automatically select the most appropriate languages and weights for cross-lingual CoT, which consists of two main components: (1) Automatic Language Selection Prompting and (2) Automatic Weight Allocation Prompting.

heading: 3.1 Automatic Language Selection Prompting
text: To address the significant challenge of manual language selection, as shown in Figure 2, we propose Automatic Language Selection Prompting (ALSP) to autonomously and intelligently choose the most suitable languages and further utilize the crosslingual capabilities of LLMs. Specifically, the prompt content is as follows:
Instruction: ... your task is to select [Selected Number N ] languages optimal
for cross-lingual reasoning on a given [Source Language Lsrc] sample. . . [Language Information Linfo]
Input: [Given Query Q]
Specifically, ALSP directs LLMs to select the [Selected Number N ] of languages by analyzing the [Given Query Q] , respective [Source Language Lsrc] , and a comprehensive list of potential target language information
[Language Information Linfo] . The language selection process can be formulated as:
L′tgt=argmax L
N∑
i=1
P (Litgt|Q,Lsrc,Liinfo), (6)
where L′tgt = {Li ′ tgt}Ni=1 represents the final set of chosen target languages, and Liinfo ∈ Linfo encompasses respective language family, language branch, the proportion of available pre-training data to facilitate informed decision-making.

heading: 3.2 Automatic Weight Allocation Prompting
text: After selecting the target language, as shown in Figure 2, we further introduce Automatic Weight Allocation Prompting (AWAP). Specifically, our carefully designed prompts for guiding LLMs to automatically allocate weight to each language inference path are as follows:
Instruction: After your language selection, please assign an alignment score for multilingual reasoning aggregation... [Weight Range Wrange]
Input: [Given Query Q]
Specifically, this process dynamically allocates weights from [Weight Range Wrange] to languages based on their relevance to the [Given Query Q] , enhancing performance of
LLMs by aligning the [Source Language Lsrc] to target language Li′tgt generated from last turn
more effectively. Formally, the automatic weight for each language can be obtained by:
W ′i = argmax w∈Wrange p(w|Q,Lsrc → Li ′ tgt,Liinfo), (7)
where W ′i represents the cross-lingual alignment weight for the i-th target language.

heading: 3.3 Automatic Cross-lingual Prompting Consistency
text: By automatically determining the relevant language and its associated weight, our framework further adapts Automatic Cross-lingual Prompting Consistency to more effectively merge multilingual alignments, leading to improved consistency across languages. Following Equation 4, we collect a set of generated results R. The formulation of the final integrated result R̂ is presented as follows:
R̂ = argmax R∈R
N∑
i=1
∑ r∈R W ′i · 1(R = r), (8)
where R and r both denote a reasoning outcome generated based on a specific formula in the target language Li′tgt from the generated result set R, and
W ′i represents the weight assigned to that language. Additionally, 1(X) is the indicator function, which returns 1 if X is true and 0 if it is false.

heading: 4 Experiments

heading: 4.1 Dataset and Baseline
text: Following Wei et al. (2022); Qin et al. (2023), we assess the performance of AUTOCAP on the widely utilized multilingual mathematical reasoning dataset MGSM (Shi et al., 2022) and select three representative LLMs as backbones, namely PaLM (Chowdhery et al., 2023), GPT3 (Brown et al., 2020), and GPT-3.5 (OpenAI, 2022). The top-p and temperature parameters in all processes are selected within the range of [0, 1].

heading: 4.2 Main Results
text: We follow previous work (Wei et al., 2022; Qin et al., 2023) to adapt accuracy (Acc.) as the metric to evaluate the performance. The main results are shown in Table 1.
From the results, we observe that AUTOCAP attains superior performance compared to all baseline models by achieving state-of-the-art performance, even surpassing the ensemble methods CLSP and
Cross-ToT, which manually select reasoning languages. Specifically, AUTOCAP achieves an average accuracy improvement of over 3.1%, outperforming CLSP across all tested languages. This demonstrates that AUTOCAP, while implementing the automatic selection of reasoning languages, can better elicit the cross-lingual CoT reasoning capabilities of LLMs.

heading: 4.3 Analysis
text: To gain deeper insights into our approach, we explored the following research questions: (1) Are all modules effective for AUTOCAP? (2) Can the interactive capabilities of LLMs enhance the performance of AUTOCAP? (3) Can AUTOCAP work well on fewer languages? (4) Why can AUTOCAP works? (5) Can AUTOCAP generalize to other opensource models? (6) Can AUTOCAP generalize to other benchmarks? (7) What is the intuition behind AUTOCAP?

heading: 4.3.1 Answer 1: All Modules in AUTOCAP are Effective for AUTOCAP
text: In this section, we explore whether Automatic Language Selection Prompting and Automatic Weight Allocation Prompting are effective.
Automatic Language Selection Prompting is effective. To analyze the effectiveness for Automatic Language Selection Prompting (ALSP), we removed the ALSP and randomly selected six languages for all query data instead. As indicated in Table 2 (w/o ALSP), there is a significant decline in reasoning performance across all languages. In particular, there was an average accuracy reduction of 6.6%. It indicates that ALSP can effectively select the more optimal languages for alignment, which significantly improves the process of bridging lin-
guistic gaps, directly contributing to a notable enhancement in the performance of multilingual CoT.
Automatic Weight Allocation Prompting is also crucial for performance enhancement. To verify the impact of excluding the Automatic Weight Allocation Prompting (AWAP) from our AUTOCAP. By default, we adopt the setting of Qin et al. (2023) and set the weight of all languages to 1 by default. As presented in Table 2 (w/o AWAP), it demonstrates a notable reduction in cross-lingual CoT performance, with an average decrease of 2.2%. The absence of specific weightings for each language resulted in lesser coherence when merging the outputs from multilingual reasoning, adversely affecting the reasoning process of overall effectiveness. This performance decline highlights the critical role of the AWAP. Specifically, AWAP accurately allocates weights and facilitates detailed planning to enhance the degree of cross-lingual alignment, thereby improving the precision of multilingual reasoning.
The combination of AWAP and ALSP brings a higher improvement. To validate the functionality and effectiveness of the combination of the Automatic Weight Allocation Prompting and Automatic Language Selection Prompting, we conducted an experiment where both modules were simultaneously removed. As shown in Table 2 (w/o AWAP & ALSP), we observe a significant decline in performance compared to the AUTOCAP, with a decrease of 7.5%. This decline was also evident when compared to the individual ablations of the ALSP and AWAP modules. The absence of both modules resulted in a substantial decrease in accuracy in multilingual reasoning, underscoring the importance of language selection and weighting for better cross-lingual reasoning.

heading: 4.3.2 Answer 2: The interactive feature of LLMs boosts AUTOCAP performance.
text: To investigate the influence of the interactive capabilities of LLMs on AUTOCAP, we differentiate between single-round prompting and multiround prompting. Specifically, in the single-round prompting approach, we instruct LLMs to simultaneously select reasoning languages and allocate weights. Conversely, in the multi-round prompting approach (AUTOCAP), in the first round, LLMs try to select the language, and then LLMs are required to allocate weight in the second round.
As illustrated in Figure 3, the average performance of single-round interactions exhibited a decrease of 2.9%, which indicates that leveraging the interactive capabilities of LLMs can significantly enhance the performance of cross-lingual CoT and the capability of language planning.

heading: 4.3.3 Answer 3: AUTOCAP also achieves positive results on fewer languages
text: To showcase the efficacy of our method using a limited number of languages, with a varied number of languages, specifically ranging from three to five. As shown in Figure 4, a positive trend can be observed: as the number of languages incorporated increases, there is a corresponding enhancement in model performance. This pattern underscores the scalability and robustness of our approach in processing multilingual inputs. Further, to explore
the effectiveness of AUTOCAP, we compared our results with the state-of-the-art method, CLSP (Qin et al., 2023), which uses manually selected languages and a voting mechanism to merge answers from multiple languages. In detail, as shown in Figure 4, when evaluated across linguistic settings encompassing three, four, and five languages, our model consistently outperformed the CLSP framework, registering an average performance uplift of 0.4%. Such findings demonstrate the efficiency of our approach across varying language counts, which highlights the potential of our method in leveraging cross-lingual planning automatically to improve cross-lingual CoT capabilities.

heading: 4.3.4 Answer 4: AUTOCAP selects more diverse languages for better result
text: In investigating the capacity of AUTOCAP to encompass a broader linguistic spectrum, this section counts the variety and distribution of reasoning languages selected by AUTOCAP. The statistical results are shown in the Figure 5, in its reasoning processes, AUTOCAP incorporated a minimum of seven and a maximum of ten distinct languages. This demonstrates a significant enhancement in linguistic diversity when contrasted with the conventional approach of static language selection, underscoring the superior adaptability and breadth of languages facilitated by the autonomous choices made by LLMs.

heading: 4.3.5 Answer 5: AUTOCAP also works on other LLM
text: To further validate the scalability and universality of AUTOCAP, we adapt AUTOCAP on other open-source LLMs. The experimental outcomes on LLM Mistral (Jiang et al., 2023) are depicted in Figure 6, demonstrating the optimal capability of AUTOCAP on open-source LLMs. With an average performance improvement of at least 5.2% over CLSP, these results further attest to the broad applicability of AUTOCAP.

heading: 4.3.6 Answer 6: AUTOCAP exhibits strong generalization on other benchmarks.
text: To further explore the effectiveness of AUTOCAP on other tasks, following Qin et al. (2023), we conducted experiments on two multilingual datasets, XNLI (Conneau et al., 2018) and PAWS-X (Yang et al., 2019). The results, as shown in Figure 7, indicate that AUTOCAP achieved better performance
than CLSP on both datasets, with an average improvement of 4.0% on XNLI and 2.2% on PAWS-X. And it surpasses the performance of all languages compared with CLSP. These effectively illustrate the generalization of AUTOCAP on different crosslingual COT tasks.

heading: 4.3.7 Answer 7: Qualitative analysis
text: To provide a more intuitive understanding of our method, we present a distinct case for qualitative analysis in this section. As shown in Figure 8 (a), CLSP conducts reasoning in six manually selected languages. The reasoning results are correct in English (en) and Russian (ru), but incorrect in German (de), Japanese (ja), French (fr), and Chinese (zh). CLSP treats each reasoning path equally, integrating different paths solely through a voting mechanism, which unfortunately led to the incorrect answer “14”.
Conversely, as depicted in Figure 8 (b), AUTOCAP automatically selects six reasoning languages based on the query during the first interaction round. Although reasoning in German (de), Japanese (ja), and Vietnamese (vi) lead to incorrect answers, English (en), Russian (ru), and Spanish (es) produce correct ones. In the second round, it assigns respective weight scores to each language. By aggregating these weighted scores, our method successfully circumvents the incorrect reasoning, ultimately selecting the correct answer “30”. These cases demonstrate the effectiveness and intuitiveness of our method. Specifically, our AUTOCAP is capable of performing automatic cross-lingual planning on both languages and respective weights. Such advanced planning effectively decreases the cross-lingual alignment difficulties.
Manual Language Selection
Answer: 14
(a) Cross-lingual Self-consistent Prompting (CLSP)
Automatic Language Selection Prompting
Automatic Weight Allocation Prompting
Automatically Selected Languages:
Task Instructions:…allocate weights for …
Weight Allocation:
Task Instructions:… select six languages… Query:布列塔尼和她的妈妈去博物馆…
NLP
NLP
Weight Answer: 30
Automatic Cross-lingual Prompting Consistency
(b) Automatic Cross-lingual Alignment Planning (AutoCAP)
CoT
Answer 143014 221430
11 1 1 1 1
CoT
0.9 0.9 0.7 0.6 0.5 0.8
Weight
Query: 布列塔尼和她的妈妈去博物馆… Manually Selected Languages:
en de ru ja fr zh
en de ru ja fr zh
en de ru ja vi es
Answer 143014 1430 en de ru ja vi es 30
0.9 0.9 0.7 0.6 0.5 0.8
Figure 8: Case Study. Figure (a) illustrates the CLSP process, which integrates multiple reasoning answers through the manual selection of reasoning languages and a voting mechanism. Due to the predominance of the incorrect answer “14” in the results, this incorrect answer was selected. In contrast, Figure (b) demonstrates the ability of AUTOCAP to automatically select aligned reasoning languages, with the correct inference answer “30” achieving a higher proportion than CLSP. By combining automatically allocated weights, the correct answer was selected.

heading: 5 Related Work
text: The evolution of LLMs has profoundly propelled the progress in the field of artificial intelligence (Brown et al., 2020; Schaeffer et al., 2024; Yao et al., 2024; Touvron et al., 2023; Tang et al., 2023). In particular, CoT has provided a new perspective for solving complex problems and enhancing the reasoning capabilities of models by directing LLMs through a step-by-step problem-solving process.(Wei et al., 2022; Kojima et al., 2022; Feng et al., 2024; Zhang et al., 2023, 2022).
Current research primarily focuses on English, yet given the existence of over 7,000 languages worldwide, addressing critical challenges like reasoning and generation in minority languages has become an urgent necessity (Qin et al., 2023; Shi et al., 2022; Lin et al., 2021; Huang et al., 2023). Recognizing this gap, recent research has increasingly focused on transcending vanilla CoT to explore its cross-lingual dimensions. In this vein, Shi et al. (2022) pioneer the introduction of the multilingual dataset specifically curated for mathematical reasoning. They advocate for a novel cross-lingual methodology that mandates LLMs to employ English in forecasting CoT sequences, translating the problem into English, and subsequently resolving it via English-based CoT paradigms. Ranaldi and Zanzotto (2023) unveil a self-consistent prompting mechanism within their cross-lingual, multi-step reasoning strategy, significantly augmenting reasoning capabilities across various languages. Tanwar et al. (2023) suggest the integration of exemplars showcasing semantic congruence between
source and target languages within the prompt context, aiming to facilitate seamless reasoning transitions across linguistic divides. Moreover, Qin et al. (2023) propose a self-consistent prompting, initially involving the manual selection of the reasoning language, followed by employing a voting mechanism to determine the final reasoning answer. This approach has yielded excellent results in the efficacy of cross-lingual prompting.
In comparison to the previous research, our work focuses on two key aspects. Firstly, we introduce an automatic language selection mechanism, enabling our system to choose the most accurately aligned reasoning languages for each query. Additionally, we develop an automatic weight allocation that effectively integrates the answers provided by various reasoning paths. To our knowledge, this is the first work to automatically select the reasoning languages and assign weights to reasoning paths.

heading: 6 Conclusion
text: In this paper, we present the Automatic Crosslingual Alignment Planner (AUTOCAP), a novel framework designed for enhancing zero-shot crosslingual CoT reasoning. AUTOCAP is comprised of two key components: Automatic Language Selection Prompting and Automatic Weight Allocation Prompting. AUTOCAP achieves to automatically select suitable languages and allocate weights to various reasoning paths in cross-lingual CoT, respectively. Extensive experiments demonstrate that AUTOCAP achieves superior performance, outperforming existing methods in cross-lingual CoT.

heading: 7 Limitations
text: This work achieves automatic language selection and weight allocation for cross-lingual CoT, reducing manual workload significantly and marking a meaningful first step towards automatic resolution of multilingual tasks. In the future, we can apply our framework to multi-agent systems, enabling agents to incorporate automatic selection of reasoning languages and powerful tools to tackle challenging real-world problems. In addition, we can consider the multi-lingual multi-modal CoT scenario by injecting the cross-modal CoT ability (Chen et al., 2024).

heading: 8 Acknowledgments
text: This work was supported by the National Natural Science Foundation of China (NSFC) via grants 62306342, 62236004 and 62206078. This work was also sponsored by the Excellent Young Scientists Fund in Hunan Province (2024JJ4070). We are grateful for resources from the High Performance Computing Center of Central South University.

heading: Future Work
text: In the future, we can apply our framework to multi-agent systems, enabling agents to incorporate automatic selection of reasoning languages and powerful tools to tackle challenging real-world problems. In addition, we can consider the multi-lingual multi-modal CoT scenario by injecting the cross-modal CoT ability (Chen et al. , 2024)