Eliciting Better Multilingual Structured Reasoning from LLMs through Code (2024)

Abstract: The development of large language models (LLM) has shown progress on reasoning, though studies have largely considered either English or simple reasoning tasks. To address this, we introduce a multilingual structured reasoning and explanation dataset, termed xSTREET, that covers four tasks across six languages. xSTREET exposes a gap in base LLM performance between English and non-English reasoning tasks.1 We then propose two methods to remedy this gap, building on the insight that LLMs trained on code are better reasoners. First, at training time, we augment a code dataset with multilingual comments using machine translation while keeping program code as-is. Second, at inference time, we bridge the gap between training and inference by employing a prompt structure that incorporates step-by-step code primitives to derive new facts and find a solution. Our methods show improved multilingual performance on xSTREET, most notably on the scientific commonsense reasoning subtask. Furthermore, the models show no regression on non-reasoning tasks, thus demonstrating our techniques maintain general-purpose abilities.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5154–5169 August 11-16, 2024 ©2024 Association for Computational Linguistics
We then propose two methods to remedy this gap, building on the insight that LLMs trained on code are better reasoners. First, at training time, we augment a code dataset with multilingual comments using machine translation while keeping program code as-is. Second, at inference time, we bridge the gap between training and inference by employing a prompt structure that incorporates step-by-step code primitives to derive new facts and find a solution. Our methods show improved multilingual performance on xSTREET, most notably on the scientific commonsense reasoning subtask. Furthermore, the models show no regression on non-reasoning tasks, thus demonstrating our techniques maintain general-purpose abilities.

heading: 1 Introduction
text: The ability to perform complex reasoning tasks is fundamental to human intelligence, where multiple steps of thought are required. Complex reasoning remains an open-problem for large language models (LLMs), despite some recent progress. Prior works consider complex reasoning tasks specified only in English. Such an English-centric perspective provides a limited assessment of the underlying reasoning capabilities of LLMs, given any specific language is largely a surface-form representation.
∗Work done during an internship at Amazon †corresponding author
1https://github.com/amazon-science/ xstreet released under CC-BY-4.0.
This motivates our first inquiry into the multilingual complex reasoning capabilities of LLMs.
We introduce the xSTREET reasoning and explanation dataset (as shown in Figure 2). xSTREET covers 4 tasks, and extends the English STREET benchmark (Ribeiro et al., 2022) to 5 additional diverse languages, inheriting the source’s expert annotations and structured graphs for reasoning steps (7.8 average steps/answer). The tasks cover arithmetic, logic and science commonsense problems. We perform machine translation for the training and development data splits, and also perform human post-editing to the test sets, to ensure a high quality multilingual benchmark. We use xSTREET to evaluate several LLMs, identifying the multilingual setting as significantly challenging.
To remedy the non-English reasoning gap, we turn to the widely accepted hypothesis that LLMs trained on code are better at reasoning than those trained only on text. This code and reasoning hy-
5154
pothesis has been empirically corroborated by several papers (Suzgun et al., 2022; Liang et al., 2023; Hendy et al., 2023). Our work takes a further step in investigating the extent to which this hypothesis holds for non-English tasks. We proceed with the insight that code can be leveraged as a structured framework to represent the underlying reasoning steps, regardless of the surface-form language of the task. We thus propose two techniques to elicit better multilingual complex reasoning from LLMs (as shown in Figure 1): at training time through a lightweight fine-tuning recipe on code, and at inference time using a novel code prompt format.
In the LLM literature, many capabilities have been characterized as ‘emergent’ with model scale (Wei et al., 2022a; Patel et al., 2022). Recent work on complex reasoning has thus focused on huge (175B+) and closed-source models. In our work, we instead aim to boost performance on far smaller open-source LLMs (7B). To make our findings reproducible, we release our benchmark. Our contributions are:
1. We collect and release the first dataset for multilingual structured reasoning, xSTREET, covering 6 diverse languages and 4 tasks (5.5K entries total). 2. At train time: we enhance reasoning capabilities of off-the-shelf LLMs by further training on program code data where code is interleaved with non-English comments. To this end, we augment a source code corpus through translating code comments and apply low-rank parameter-efficient fine-tuning
(LoRA (Hu et al., 2021)) to BLOOMZ (Muennighoff et al., 2022). Our method is effective yet lightweight, while preserving generalpurpose LM capabilities. 3. At inference time: we design a code-like prompting format that mimics the structure of the reasoning tasks by interweaving function calls and multilingual text. We show this format outperforms several other prompt formats used. 4. We evaluate multiple LLMs (BLOOMZ, GPT3, Falcon-40b-instruct) on our benchmark, and show improved performance — even for top-performing models — across structured reasoning tasks in different languages. As our inference and training-time techniques are orthogonal, we show that they can be used in tandem to achieve the best performance. 5. We perform qualitative and quantitative analysis to understand the roles of both the program code, and the code comments. Our findings taken together suggest that code elicits better multilingual structured reasoning by improving LLM’s adherence to the reasoning format.

heading: 2 Related Work
text: We adopt the scope of complex reasoning from Ribeiro et al. (2022), and use complex reasoning and structured reasoning interchangeably. The goal is to study the reasoning process itself, and how an LLM can structure its output in steps to improve the final performance. The tasks are selected so that the knowledge to answer a question is
contained with the input question and context. For ease of evaluation, the answers are multiple-choice selections, or numbers for arithmetic reasoning. Further details are given in §2.4.

heading: 2.1 Code & Reasoning Hypothesis for LLMs
text: This hypothesis arose from empirical evidence by several concurrent works. Suzgun et al. (2022) state, “Codex, trained on both code and text data, shows better performance in following task instructions and exploiting algorithmic patterns based on the prompt exemplars.” Liang et al. (2023) state, “for reasoning-intensive scenarios, we find that the code models, especially Codex davinci v2, consistently outperform the text models, even on synthetic reasoning scenarios posed in natural language.” Hendy et al. (2023) state that “We hypothesize that the models acquire their reasoning capabilities through training on natural language multilingual data along with programming languages data”. In summary, these works provide evidence that training LLMs on code serves as indirect supervision for complex reasoning tasks. One of our major goals is to explore the extent to which this hypothesis holds beyond English.

heading: 2.2 Code Prompts for Complex Reasoning
text: Reasoning tasks posed in natural language can be reformulated as code prompts. Using these codelike structures to interact with code-LLMs better aligns the representations seen at training time with those at inference time. Madaan et al. (2022) use few-shot prompting on the Codex LLM to convert tasks into Python graphs, deal with structured commonsense tasks. Zhang et al. (2023) proceed similarly, but for causal reasoning tasks. Chen et al. (2023) consider arithmetic reasoning tasks, and then execute the LLM-generated code on an external interpreter. The reformulation process from natural language specification to code prompts is an open-ended one, requiring manual annotation effort, creativity, and trial and error.
While these works use code prompts for complex reasoning tasks with classification or numerical outputs, as we did, code prompts can also be applied to tasks with generative outputs, such as knowledge graph construction (Bi et al., 2023) and story understanding (Dong et al., 2023). To the best of our knowledge, our work is the first to use code prompts in multiple languages.

heading: 2.3 Multilingual Reasoning for LLMs
text: The MEGA benchmark (Ahuja et al., 2023) covers 70 languages and 16 tasks. MEGA considers only simple reasoning tasks, which, as discussed earlier, limits our understanding of how well LLMs can reason across languages.
MGSM (Shi et al., 2022) is an arithmetic reasoning dataset in 10 languages, translated from GSM8K (Cobbe et al., 2021).They find that the chain-of-thought technique (CoT) (Wei et al., 2022b), by adding to the prompt few-shot examples of step-by-step reasoning, is also effective in the multilingual setting. Interestingly, they find that for non-English questions, English CoT outperforms native language CoT. They further emphasize the reasoning ability increases with model scale. Our xSTREET benchmark is a more comprehensive view of multilingual complex reasoning. xSTREET covers not only arithmetic,2 but adds logic and science tasks, has many more entries, and has ground-truth structured reasoning annotations.

heading: 2.4 STREET Complex Reasoning Benchmark
text: The STREET benchmark is a composite of several complex reasoning tasks (Ribeiro et al., 2022). The work adds expert human annotations for multipremise, multistep explanations. Each task’s explanation is structured in a reasoning graph. Reasoning graphs, as shown in Figure 2, consist of nodes which contain statements, and edges that connect nodes.
Source Tasks The tasks3 and answer formats are: • ARC science commonsense questions
(multiple-choice) • GSM8k arithmetic word problems (number) • AQUA_RAT arithmetic word problems
(multiple-choice) • AR_LSAT logic problems from a standard-
ized test (multiple-choice)
Linearized prompt format While a reasoning graph is abstract, to interface with an LLM, Ribeiro et al. (2022) use linearized prompts. This represents a graph as a sequence of tokens, as shown in Figure 3. Statements are given a number index;
2Instead of using MGSM, we perform our own translation of GSM8k given the intermediate reasoning annotations inherited from Ribeiro et al. (2022).
3STREET includes a fifth task, SCONE, which is omitted from xSTREET. SCONE is quite abstract, and involves statetracking in a toy world. This requires careful consideration beyond translation, and is thus left to future work.
output statements (i.e., reasoning steps) include a trace of the nodes leading to the new statement.
Problems with the linearized format arise in that it is task-specific, and that it diverges from LLM’s training data distribution. While in-context learning can help the model pattern-match the output format, the underlying reasoning abilities of the LLM may not be properly elicited. Following Madaan et al. (2022), we argue that interfacing with a codeLLM through code prompts is a more “intuitive” way for the LLM to reason through a task, leading to our novel code prompts format in §5.

heading: 2.5 Source Code Dataset
text: The Stack is a 3.1 TB dataset of permissively licensed source code in 30 programming languages (Kocetkov et al., 2022). In this work, we utilize the official small subset4, and consider only 3 popular programming languages: Java, JavaScript, Python (10k files each, 30k total).

heading: 3 Multilingual Complex Reasoning Benchmark: xSTREET
text: We create the xSTREET dataset by translating STREET into 5 languages: Arabic (ar), Spanish (es), Russian (ru), Chinese (zh), and Japanese (ja). These languages have linguistic and script diversity; furthermore, they are the languages used in many online programming help websites.
To create the xSTREET test split, we hire expert human translators for all 5 languages through an internal team (detailed in §9). Translators are tasked with post-editing the machine translation of one sentence at a time; for context, they can refer to the entire STREET entry the sentence comes from. After receiving the translations, we re-use the reasoning graph edges, and replace English nodes with the translations to create xSTREET. This process is shown in Figure 2. We therefore extend the 914 English entries in STREET to 5484 examples in xSTREET (914 * 6 languages).
To create the xSTREET train and development splits, we use machine translation.5 We then asked native speakers to evaluate the quality of 10 random sampled translations of each language. Annotators gave feedback that, despite some errors, the translations were of reasonable enough quality to use for training purposes.
4Available here 5We used an online translation API (anonymized here).
Dataset statistics for the xSTREET test benchmark are given in Table 1.

heading: 4 Code with Multilingual Comments as Indirect Supervision for Reasoning
text: Taking the idea of using code for reasoning, and comments for multilinguality a step further, we address the question: can multilingual code serve as indirect supervision for multilingual reasoning? In other words, we investigate whether the code & reasoning hypothesis holds multilingually. We therefore propose a lightweight fine-tuning recipe, which consists of creating a multilingually commented code dataset, then fine-tuning on it, which serves as indirect supervision for downstream reasoning tasks.

heading: 4.1 Translated Code Comments Dataset (TCC)
text: The first step of the recipe is creating a source code dataset with translated code comments, termed TCC. For each file from the source dataset, and for each target language, we perform the following. We parse the code to extract out comments, translate comments into the target language, then replace the original comments with translations. This is depicted in Appendix Figure 7.
We use two simple filters: for source code files that A) have >5 comments, and B) whose comments are over 50% in English.6 This filters 30k source code files down to 20k. After translating into 5 additional languages, TCC consists of 20k*6=120k files total. See Appendix Table 5 for dataset statistics.

heading: 4.2 Train Time: fine-tuning on TCC
text: In the second step, we leverage low-rank adaptation (LoRA) (Hu et al., 2021) to finetune instruction-
6We performed other filtering experiments, described in Appendix E.2, which had similar performance.
tuned LLMs on TCC.7 We use two methods to preserve the original model’s capabilities despite the additional finetuning. First is by using LoRA itself, as it keeps the original base model’s parameters frozen and introduces only a few learned parameters. Secondly, we replay 100k examples from the base model’s training data, xP3 (Muennighoff et al., 2022), in a multitask setup with the TCC LM task.
The recipe for a reasoning-enhanced LLM is now complete, and this is depicted in Figure 1.

heading: 5 Multilingual Complex Reasoning as a Downstream Task
text: We hypothesize that structure, when applied to reasoning problems formulated in different languages, can abstract away some of the language-specific details, better surfacing the reasoning steps needed for a model. We thus propose the SIM (Selectand-infer multilingual comments) code prompts for complex reasoning tasks.
SIM code prompts utilize several functions. We do not provide the API definitions, instead, we expect the model to learn to use them from the in-context examples. The functions are:
• select_facts(facts) • infer_new_fact(selected) • is_solved(fact, question) • make_choice(fact, choices)8
• facts.append(fact) select_facts and infer_new_fact are loosely inspired by Selection-Inference (Creswell et al., 2023). A key difference, though, is that we use a single prompt, instead of iterative prompts. We therefore include is_solved(fact, question) as a signal for the LLM to stop generation.
Each function is annotated with its return value in an inline code comment. This is inspired by prior work (Zhang et al., 2023). infer_new_fact has a string return value, i.e., the text of the new fact. We experiment with two versions of the return value of select_facts. The first, termed SIMindexed, uses variables facts[i] to reference the facts array (similar to the indices used in linearized format). The second, termed SIM-text, directly uses each fact’s text, dereferenced from facts[i]. We find that SIM-text works best for
7We used a g5.48xlarge instance from AWS, which has 8 NVIDIA A10G GPUs (24*8 GB=192GB vRAM).
8This function is not used for non-MC tasks, i.e. GSM8k.
smaller models, while SIM-indexed does for larger ones, and hence apply this going forward.
We write a rule-based Python script that converts existing structured graph annotations to SIM code prompts. SIM prompts express the exact same information as the linearized format. This property is unlike code prompts for prior work, wherein the conversion is done through in-context learning with an LLM, which can introduce errors as discussed in §2.2. The different prompting formats for LLMs are shown in Figure 3.
Multilingual code prompts We use multilingual input in SIM code prompts as follows. First, facts given in the question are listed in the language of the task in a list of strings. Second, new facts and selected facts are given as comment lines adjacent to the function calls. See Figure 3 for an example.

heading: 6 Experimental Setup
text: Models Used We primarily study open-source models, which allows for application of both train and inference-time techniques. We use BLOOMZ (Muennighoff et al., 2022) as our base LLM. This model is instruction-finetuned on prompts in 46 natural languages and 10 programming languages. For our experiments, we consider the 7.1B-parameter BLOOMZ, as well as BLOOMZ-TCC which is further finetuned on TCC.
For inference-time only, we consider two larger LLMs. We use the instruction-finetuned version of Falcon (Almazrouei et al., 2023) (40B), another open-source LLM trained on text+code. Compared to BLOOMZ, Falcon is more performant on English tasks; however, it has limited multilingual abilities.9 We also use GPT-3 (175B)10, a closedsource model that is popularly-used and powerful.
Prompting setup We use few-shot prompting, and random sample up to 5 exemplars from the train split (up to a model’s context length).11 For each inference example, the same exemplars are used for all models and prompt types. We use greedy decoding, and task the model with generating up to 682 tokens. (max context length of BLOOMZ 2048 // 3).12
9As stated in the model card for falcon-40b-instruct. 10text-davinci-002 following Ribeiro et al. (2022) 11Due to brevity, Figure 1 uses a 0-shot prompt, and only depicts the SIM prompting format. The reported results use 5-shot prompts, and are given for all prompting formats.
12We acknowledge a limitation with the max of 682 tokens, as this will truncate output for questions which require longer

heading: 7 Results
text: We report results on the xSTREET benchmark. We use the answer accuracy metric, adapting evaluation from Ribeiro et al. (2022).13
Given the extensive nature of the xSTREET benchmark and our model experimentation, we highlight our findings iteratively. We first consider only BLOOMZ and BLOOMZ-TCC, with a particular focus on ARC, where our methods are the most impactful. We then consider GPT-3 and Falcon.
The full results are given in Appendix D. Here we provide numbers for all tasks, languages, models, and prompt formats (this also includes the direct prompting format).

heading: 7.1 Results for BLOOMZ Models
text: Results for the ARC task (science commonsense reasoning) are shown in Figure 4. Several takeaways arise. We see that code prompts greatly outperform linearized prompts across all languages. For results within a single model, reasoning performance drops greatly comparing English vs. average non-English (e.g. from 76.2% to 61.1% accuracy for BLOOMZ-TCC). This provides evidence that current multilingual LLMs are still optimized for English. This underscores the usefulness of
reasoning chains. 13While STREET also measure graph similarity between linearized output and reference graphs, we did not implement them for SIM prompts. This is because for the small LLMs (7B), even the linearized format had near 0 graph similarity.
xSTREET for developing LLMs with better underlying, language-agnostic abilities.
We next turn to comparing base BLOOMZ vs. our finetuned BLOOMZ-TCC. We see that BLOOMZ-TCC outperforms BLOOMZ for all languages and both formats. More interestingly, relative multilingual gain is much larger when using code prompts vs. linearized prompts (Avg non-en, 52.6 → 61.1 vs. 33.5 → 36.9). This is evidence that the code prompt format improves multilingual reasoning, likely by the explicit separation of the reasoning task (in code) vs the multilingual understanding (in comments). Finally, looking at perlanguage trends for BLOOMZ-TCC we see that code prompts are most effective for en, es, zh, and ar, while less so for ja and ru.14
Results on GSM8K, AQUA_RAT, AR_LSAT Our results show that BLOOMZ and BLOOMZTCC struggle for the other tasks, with performance being around random chance whether using the interventions or not. We hypothesize that these tasks are “too hard” for the BLOOMZ-7B used; to reiterate, GSM8K and AQUA_RAT are arithmetic reasoning, while AR_LSAT is logical reasoning. This concurs with the common view that complex reasoning capabilities of LLMs are emergent with model scale (Wei et al., 2022a). We further discuss these results, and expand our hypotheses, in
14This is likely because the base model, BLOOM, was not trained on any ja or ru text.
Appendix §A.1.

heading: 7.2 Results for Larger LLMs
text: As code prompts are at inference time, they can be used to interface with any LLM. We report results for GPT-3 in Figure 5. We see as before that the multilingual setting poses additional challenges for reasoning, as English results are always higher than corresponding non-English tasks.
First considering ARC, GPT-3 performs strongly in English for both formats, nearly solving the task. Comparing English to multilingual ARC, linearized suffers a sharp drop (93.2 → 73.2), while code prompts remain robust (99.1 → 94.2). This underscores the effectiveness of SIM prompts in disentangling the reasoning and multilingual components of the task.
For the other tasks, SIM always outperforms linearized format. Comparing relative gains, code prompts boost performance more in English than on multilingual settings. While still a very positive result, this differs from ARC as discussed above. To discuss why this is the case, we consider the dual effects of SIM code prompts, vs. linearized: the function calls capture the reasoning structure, while the multilingual comments capture the language understanding. Because the arithmetic and logical reasoning tasks are far more symbolic than the ARC commonsense reasoning task, multilingual language understanding is less effective.

heading: 7.3 Non-Complex Reasoning Task Results
text: Recall that our fine-tuning recipe aims to improve reasoning of an LLM, while maintaining its natural language understanding (NLU) abilities. We show this is the case by reporting results on 3 multilingual tasks:
• XNLI: natural language inference • XStoryCloze: given 4 sentences from a short
story, choose between 2 possible completions • XQUAD: extractive question answering To query LLMs, we follow the specific prompting guidelines for each task from Ahuja et al. (2023). Table 2 shows that for all 3 tasks, the differences between BLOOMZ and BLOOMZ-TCC are statistically insignificant. Therefore, the mitigation strategies we used, LoRA and training data replay, have proved effective.

heading: 7.4 Effect of Code Comments on Downstream Reasoning
text: The code & reasoning hypothesis speaks to training on code improving LLM reasoning. However, an integral part of source code is comments, which have been underexplored by prior work. We study 2 ablation settings, with the same finetuning setup: TCC-en: original source code files (i.e. Englishonly comments). TCC-del: source code files without any comments (comments are deleted).
We evaluate the best prompt format (SIM) on the ARC subtask. Results are shown in Figure 6. We see that overall, finetuning on TCC is the best configuration, then TCC-en, and finally TCC-del. These trends generally hold over the 6 languages.15
This ablation study adds a new consideration to the code & reasoning hypothesis: that within code,
15Russian (ru) is an exception, where BLOOMZ-TCC-en outperforms BLOOMZ-TCC. We will investigate this further, but note this may be an artifact of the base LLM, BLOOM, not having tokenization for Cyrillic script.
even the comments are influential in downstream reasoning performance. Furthermore, we see that the diversity of code comments introduced by our proposed data augmentation of TCC further boosts performance in all languages, including English.

heading: 8 Analysis
text: To further understand wherein our techniques help, or fail to help, model reasoning, we perform some manual analysis. For brevity, we focus on 2 languages, en and ar, and 2 tasks, ARC and GSM8K. We first perform error analysis on BLOOMZ, then perform a case study for each task.
We further perform 3 additional experiments, which are detailed in Appendix §E. To highlight one interesting finding, we show that training on diverse code comments, such as from the multilingual TCC, boosts xSTREET performance in all languages including English.
Error Analysis for BLOOMZ English For this task, and with base BLOOMZ, SIM achieves 61.5
ARC accuracy, while linearized achieves 35.3. Our manual analysis of outputs reveals that the performance discrepancy is largely due to poor instruction-following when using linearized vs. using SIM. Ribeiro et al. (2022) find that for linearized (and their model), 62% of generations fail to generate a parsable answer (i.e., reasoning graph is incomplete). Our findings concur, in that linearized has 66% (223/340) invalid generations. In contrast, SIM has only ˜19% invalid. BLOOMZTCC with SIM further reduces invalid rate to 9%, and increases accuracy to 76.2. We observe that in cases where all formats output successfully, the reasoning graph and answers are nearly identical. The difference is that SIM prompts allows the model to generate a complete reasoning graph far more often. We reiterate that this behavior is a novel finding given BLOOMZ-TCC was indirectly supervised on code, rather than directly on reasoning tasks. Further discussion is found in Appendix A.
We summarize this section with the following view: our techniques elicit better instruction-
following of the proscribed reasoning format from a base LLM, leading to improved benchmark performance. Within a reasoning step, the models are making similar decisions, but at the reasoninggraph level, our methods assist in harder cases.

heading: 8.1 Case Study on GSM8k English
text: We perform a case study of one GSM8k problem, comparing 3 models (BLOOMZ, BLOOMZ-TCC, GPT-3) and 2 formats (linearized, SIM) in Appendix Table 3. We observe that only GPT-3 with SIM achieves the correct answer, and reasoning steps also concur with the gold completion. GPT-3 with linearized representation makes an erroneous first step, which propagates the error downwards. Both BLOOMZ models with linearized formatting only follow the output format, and the text statements are copied from the input instead of being new statements. BLOOMZ with SIM has repetitive output and does not output an answer. While BLOOMZ-TCC still outputs a wrong answer, it does perform 2 rounds of reasoning through selecting and inferring facts. So, we see that both interventions elicit better underlying reasoning abilities of LLMs.

heading: 8.2 Case Study on ARC Arabic
text: We look at an Arabic example from ARC in Appendix Table 4. We observe that for the linearized format, the final answer is incorrect (A), given the model makes a wrong penultimate inference. The SIM format, meanwhile, allows GPT-3 to output the correct answer (D), given it makes a correct inference step (albeit 1 step less than the gold). In fact, directly prompting GPT-3 leads to a correct answer. This again highlights the importance of aligning the prompt format, which is code here, to the training format.

heading: 9 Conclusion
text: We introduced xSTREET, a multilingual structured reasoning benchmark which covers 5 diverse languages , spans science commonsense, arithmetic and logical reasoning tasks, and includes highquality intermediate reasoning steps. We found that current multilingual LLMs underperform in the non-English setting, then proposed two methods to remedy this, based on the popular hypothesis that LLMs trained on code are better reasoners. At training, we propose translating the comments of a source code dataset, to use as indirect supervision
data for parameter-efficient fine-tuning. During inference, we leverage code structure to represent reasoning graphs. We perform extensive experimentation, and both of our methods better elicit underlying reasoning abilities of LLMs.
Our work brings together two areas of challenge for LLMs — multilinguality, and complex reasoning. In particular, our fine-tuning recipe shows that the code & reasoning hypothesis can apply multilingually. We suspect that improvements can be amplified if multilingual comments are included at the pre-training, instead of the fine-tuning stage. We hope our findings underscore the key role that code should play in the development of LLMs with better reasoning capabilities across languages.
Limitations
One limitation is that we were unable to apply our fine-tuning recipe to the stronger LLMs. “Stronger” refers to two characteristics. First and unavoidably, we can only apply the method to weaker open-source models, as closed-source models are proprietary; nevertheless, we explored them with our inference-time SIM prompts approach, and this worked well. Second, we only were able to finetune a 7B parameter model due to our resource constraints, so it is to-be-determined the effectiveness of the recipe on 70B+ models.
Between the submission and publication of this work (February to August 2024), LLM development has been brisk, and several recently released ∼7B LLMs have shown decent performance on arithmetic reasoning. In our work, we were limited to BLOOMZ-7B, which we saw was poor at math. For followup work, therefore, we are excited to try our finetuning approach on TCC while using these newer LLMs as base models.
Another limitation is for the xSTREET benchmark, we performed human translation on only the test set of the source STREET dataset. As we used machine translation for the train set, but also drew few-shot exemplars from these, the lower exemplar quality worsens performance compared to a gold standard exemplars. We also fine-tuned on machine-translated TCC.
While we tried to be inclusive with the languages chosen, studying 6 languages from different families and using different scripts, we acknowledge that more community effort will need to go into expanding the study of multilingual complex reasoning to lower-resource languages. We further
acknowledge the limits of the translation of English reasoning tasks and intermediate steps alone, in that reasoning processes may differ for speakers of different languages. So too may a multilingual LLM respond inconsistently to queries posted in different languages (Li et al., 2024), which warrants future studies into how this holds for the reasoning tasks studied in this work.
Finally, in this work, we considered only the final answer accuracy for the tasks. The original STREET tasks from Ribeiro et al. (2022) included various graph similarity metrics used to consider the intermediate reasoning steps as well – a definite strength of their structured reasoning approach vs. unstructured approaches such as CoT. We did not do this consideration due to the difficulty of reimplementing the graph similarity metric calculation for the different languages, and leave this to follow up work. Furthermore, we note that the 7B LLM we used had overall poor graph similarity (near 0 for all metrics) using the original STREET evaluation scripts and dataset.
Data Statement We provide a data statement in adherence with the ACL code of conduct and recommendations laid out in Bender and Friedman (2018). Linguists working on the Machine Translation Post Editing project for the multilingual dataset into Arabic, Chinese, Japanese, Russian, and Spanish are in-country, native speakers. They all are certified translators with more than 5 years of fulltime translation experience, according to the 17100 Translation ISO Standard. These linguists were hired through vendors and were remunerated above industry standard rates. Instructions were to postedit machine translated output and included guidelines on what to localize (artist names, city names, metric conversions), format (capitalization, punctuation) and structure (sentence level breaks). The vendor project managers made sure the instructions were adhered to. The QA process consisted of content review based on the Multidimensional Quality Metric (MQM) model that allocates different weights to 5 error severities (0-none to 5-critical) in several error topics. Total sample reviewed was 5 (5k words) of the total (100k words) source word count.

heading: 10 Acknowledgements
text: We would like to thank Danilo Neves Ribeiro for his guidance on working with the STREET benchmark, and insightful conversations on how tackle
our multilingual extension of complex reasoning. We thank several colleagues for providing annotations: Etsuko Ishii, Igor Shalyminov, Yuwei Zhang. We thank these people for discussion and feedback: Salvatore Romeo, Yi Zhang, Sam Davidson, and Sailik Sengupta.

heading: A Further Discussion
text: We find that regardless of answer correctness, BLOOMZ-based models often fail to generate new text, instead of copying text from the input. This again is likely due to the weaknesses of BLOOMZ, as this is not observed for GPT-3 with any format. Our use of both interventions, greatly reduces the incidence of this problem, which as we have discussed leads to BLOOMZ-TCCbetter eliciting the model’s underlying reasoning abilities.
A.1 BLOOM Results for GSM8K, AQUA_RAT, AR_LSAT
These results are shown in Appendix Figure 8. For all tasks, performance is around random chance. For GSM8K, random chance is 0, and the models fails to solve nearly any math problem. While all numbers are close and likely statistically insignificant, we see that BLOOMZ-TCC slightly underperforms base BLOOMZ, and linearized and code prompts perform similarly.
Our hypothesis on why this happens, as discussed before, builds on the view that truly complex
reasoning capabilities are emergent with LLM’s model scale. The 7B BLOOMZ model used has no baseline ability for these 3 tasks (while it did for ARC), and therefore our interventions, which are indirect supervision on code, cannot help elicit better reasoning.
We discuss the two interventions separately. First, we study the effectiveness of code prompts on larger LLMs in §7.2. Second, for the finetuning recipe, we draw some initial points in Appendix A, given our resource constraints on small LLMs.16
This suggests limitations to the code+reasoning hypothesis, which have not been adequately discussed in prior work. Indirectly supervising LLMs for reasoning by training on code is effective for specific types of reasoning, such as ARC’s commonsense reasoning, and less so for math problems like GSM8K, though, intuitively, code probably does not help, given code rarely includes arithmetic
16With 192 GB vRAM, we could finetune at most 7B multilingual models (which have much larger vocabulary sizes and thus larger embeddings). We leave future work to use our recipe with larger models, such as by using 4-bit quantization.
equations.17 As for the logical reasoning problems of AR_LSAT, we defer study to future work applying our finetuning recipe to larger LLMs.

heading: B Hyperparameters
text: For the TCC finetuning recipe, we set maximum sequence length to 1024, set learning rate to 1e-5 (with 0.1 weight decay), do not use warm-up, and use cosine learning rate schedule. We trained for 2 epochs using a batch size of 3 and gradient accumulation over 20 steps. We set the LoRA layers dimension to 128. The implementation is done with the DeepSpeed-Chat framework (Yao et al., 2023) and the transformers library (Wolf et al., 2020).

heading: C Dataset Statistics
text: Statistics for TCC are shown in Appendix Table 5.
17Arithmetic reasoning can be improved by directly finetuning on math – Ni et al. (2022) achieve 19.5% on GSM8k on a 2.7B LLM with this approach.

heading: D Full Results
text: We now report results of all experiments and settings studied in this work. Table 6 gives results for all models and prompt formats, on STREET and xSTREET (averaged across 5 languages). Table 7 gives per-language xSTREET results. We first discuss BLOOMZ and GPT-3, which were analyzed in the paper, then separately discuss Falcon.
2. Machine translate each comment
1. Extract comments
3. Replace comments with translation, keep code as-is
en
ru
Figure 7: The data augmentation approach used to create the TCC (translated code comments) dataset.
D.1 Direct Prompting
These tables include the direct prompting strategy, in which the model is given the same input as linearized, but needs to generate only the answer without intermediate reasoning.
For ARC, we find that, surprisingly, direct outperforms linearized (all LLMs). This is likely because the ARC questions are relatively easy already, and directly solving them given the context as well is possible. As discussed earlier, linearized format is artificial and hard to follow, which causes many reasoning graphs without valid answers. In contrast, SIM prompts are captured in code, which the models have seen, and therefore SIM results outperform direct and linearized.
For GSM8k and AQUA_RAT (using GPT-3), direct prompting fails. Using intermediate reasoning, as found by many prior works, is essential for arithmetic problem-solving. Linearized boosts performance significantly, and SIM code even further.
Regarding xSTREET, overall trends are relatively consistent with those discussed above.
D.2 Results for Falcon
Falcon is an open-source LLM that is more performant than BLOOMZ, albeit English-centric. We chose the 40B instruction-finetuned variant, intermediate between BLOOMZ 7B and GPT-3 175B.18
First, we consider STREET results. For ARC, Falcon fails with direct prompts (34.7), but does much better with linearized (76.5) and SIM (81.5). For GSM8k, now that the base model has some math ability with direct prompts (4.4), it can improve with linearized (28.9) and SIM (19.6). SIM underperforming linearized here is because of a context size issue.
Falcon has a max context length of 2048 tokens, while GPT-3 and BLOOMZ can accept up to 4096. SIM code prompts use a lot of tokens for the code structure, and therefore, Falcon will run out of tokens quickly and therefore fail to generate a full reasoning graph in more cases than when using linearized. This is more so a limitation of Falcon than our work (recall that most prior work considers complex reasoning tasks with huge closed-source models).
For AQUA_RAT, Falcon performance is near random (20) for all 3 prompt formats; i.e., it is “too hard”. For AR_LSAT, Falcon is near random (20) for direct and linearized, but achieves 34.0 with SIM prompts.
Multilingual performance Even though Falcon is an English-centric LLM, we evaluate its performance on xSTREET. We see that linearized performs the best across all tasks, with code prompts behind, and direct even further behind. Again, we attribute this to Falcon’s shorter context length of 2048 – which is especially non-optimal for 4 of 5 languages studied which do not use the Latin script. The Falcon tokenizer did not see these scripts, resulting in byte-level tokenization, which further uses up the budget.

heading: E Additional Experiments
text: We perform 2 additional ablation experiments below.
E.1 Finetuning on SIM Code Prompts
We experiment with directly finetuning on SIM code prompts (all 6 languages), so as to have a
18BLOOMZ has 7B and 176B variants, but nothing in between.
model that can perform the SIM-formatted reasoning without in-context examples. We use the same hyperparameter and configuration from §4.2, again using LoRA to fine-tune a subset of the 7B model’s parameters, but omitting the data replay to maximize performance. We train one model for all tasks, and all 6 languages. This differentiates our SIM finetuned model from the linearized finetuned model of Ribeiro et al. (2022), which finetunes a separate T5 (0.8B) model for each task with full finetuning.
As multilingual trends remain similar, we will just discuss English results (STREET).Using BLOOMZ-TCC or BLOOMZ as the base model does not make a difference. The SIM finetuned models achieves 85.9 (vs. 76.2) on ARC; the other 3 tasks are still near random chance. We suspect that performing full finetuning instead of LoRA should overcome this, and omit this experiment due to resource constraints.
E.2 Improving the Code Comment Quality of TCC
Our initial and used version of TCC, as described in the main text, simply took 30k source code files from The Stack, then filtered down to 20k files, using criteria of >5 comments, of which >50% of comments in English.
To validate the quality of the translated code files, we recruited human annotators who were proficient programmers, and native speakers of each language. While overall, translations were judged to be reasonable, the main feedback points were:
• Some files had non-useful comments, such as long copyright statements in the header, or linting messages.
• Some files had comments which were actually commented-out code (i.e. unused functions).
• Terms related to programming, or referencing function or variable names in the code, were often mistranslated, if they should have been translated at all.
We leave the last point to future work, as we used an online MT API, and programming-specific MT is out of scope. For the other two, we tried to develop a version of TCC to specifically select files which have plenty of meaningful comments. We describe this filtering experiment below.
We now consider the entire Stack dataset,19, instead of just 30k from the official small subset. As the Stack totals 6 TB, we considered only the first 3 million examples (1m each for Java, Python, JavaScript). Our scripts performed the following steps in order:
1. Delete copyrights, headers, linting comments from files.
2. Keep only those files with >1 standard deviation of number of comments:number of lines ratio. Note that 1 comment can span multiple lines.
3. Keep only those files with >5 comments.
This resulted in about 250K examples. We performed the code comment extraction and translation process, and termed the resulting dataset TCCv2. We then applied the finetuning recipe as we did with the original TCC; furthermore, we keep the data size consistent as the original, using a 67k subset of TCC-v2. The resulting BLOOMZ-TCC-v2 models had similar downstream reasoning performance as BLOOMZ-TCC, and therefore we did not use it in the main text.
We hypothesize this experiment did not improve performance because, the program code plays a much bigger role in LLM’s reasoning abilities than the comments.
19https://huggingface.co/datasets/ bigcode/the-stack-dedup

heading: Future Work
text: We leave the last point to future work, as we used an online MT API, and programming-specific MT is out of scope. For the other two, we tried to develop a version of TCC to specifically select files which have plenty of meaningful comments. We describe this filtering experiment below. We now consider the entire Stack dataset,19, instead of just 30k from the official small subset. As the Stack totals 6 TB, we considered only the first 3 million examples (1m each for Java, Python, JavaScript). Our scripts performed the following steps in order:
1. Delete copyrights, headers, linting comments from files. 2. Keep only those files with >1 standard deviation of number of comments:number of lines ratio. Note that 1 comment can span multiple lines. 3. Keep only those files with >5 comments. This resulted in about 250K examples. We performed the code comment extraction and translation process, and termed the resulting dataset TCCv2. We then applied the finetuning recipe as we did with the original TCC; furthermore, we keep the data size consistent as the original, using a 67k subset of TCC-v2. The resulting BLOOMZ-TCC-v2 models had similar downstream reasoning performance as BLOOMZ-TCC, and therefore we did not use it in the main text. We hypothesize this experiment did not improve performance because, the program code plays a much bigger role in LLM’s reasoning abilities than the comments. 19https://huggingface. co/datasets/ bigcode/the-stack-dedup