The limits of Italian in Reasoning Tasks
Authors: Leonardo Ranaldi, Federico Ranaldi, Giulia Pucci, Elena Sofia Ruzzetti, Fabio Massimo Zanzotto (University of Edinburgh, Università di Roma Tor Vergata, University of Aberdeen, 2024)
Topic: Language Models

Abstract
Earlier works have been showing the efficacy of reasoning methods in eliciting step-wise reasoning of large language models (LLMs) by operating via in-context demonstrations. These strategies, exemplified by Chain-of-Thought (CoT) and Program-Aided Language Models (PAL), have been shown to reason well in monolingual contexts, primarily in English. However, there has been limited investigation into their capabilities in other languages, especially Italian. To gain a deeper understanding of the role of reasoning methods, we propose a multidimensional analysis tailored to Italian, focusing on arithmetic and symbolic reasoning tasks. Our findings indicate that the effectiveness of reasoning methods varies significantly beyond English. Expressly, CoT, which relies on natural language demonstrations, is limited to English. Conversely, the structured nature of PAL in-context demonstrations facilitates multilingual comprehension, enabling LLMs to generate programmatic answers in Italian as well. Finally, for a more complete overview, we observe that additional alignment methods do not improve downstream performances; in contrast, in some cases, they restrict the abilities of the original models.

Introduction
Large language models (LLMs) are able to tackle tasks using prompts formed by structured patterns, a process known as in-context learning AUTHOR. This method allows the models to solve tasks without modifying their underlying parameters, relying solely on the provided inputs. The success of in-context learning has consequently heightened interest in analysing the factors that influence its effectiveness AUTHOR. Regarding reasoning methods, two effective strategies have emerged: Chain-of-Thought (CoT) AUTHOR and Program-Aided Language Models (PAL) AUTHOR. CoT decomposes a reasoning task into a series of intermediate steps using natural language, making it more general and human-understandable. In contrast, PAL employs Python functions to provide reasoning solutions, with its step-by-step programming approach leading to more systematic and structured reasoning. [t!] \centering \includegraphics[width=0.92\textwidth]{img/solo_ita new cross_lingual.pdf proposed in our analysis. We explore the impact of in-context demonstrations beyond English and the performances achieved by different LLMs (\S sec:models). *As indicated in the figure, we have translated two examples of prompts from Italian to English. Although earlier research primarily showcased the functioning of reasoning methods in English, recent studies have expanded to explore multilingual approaches. \citet{shi2022language} shown that the effectiveness of CoT rationales is limited to the languages most represented in LLMs pre-training data. \citet{huang2023languages} addressed the problem by proposing prompting mechanisms that translate the problem into English, while \citet{ranaldi-etal-2024-empowering} elicit multi- and cross-lingual alignments for enabling reasoning, or \citet{ranaldi-etal-2024-tree} self-correction mechanisms. The focus is limited to proposing performance solutions for a few languages, leaving behind the study of the role and the impacts of languages such as Italian. In this paper, we conduct an in-depth study to evaluate the role of reasoning methods in Italian. Taking previous work a step further, we study the operation of reasoning methods by analysing the effects of different types of reasoning methods on LLMs' Italian reasoning capabilities. This leads to the main research questions of this paper: (i) What role do natural language and structured in-context demonstrations play in reasoning planning in Italian? (ii) What are the impacts and limits of natural language demonstrations? (iii) Do Italian-aligned and Italian-centred models respond differently to reasoning methods? To answer these questions, we operate via CoT and PAL (shown in Table tab:example_Native_CoT and Table tab:example_Native_PAL). For multilingual CoT, we use natural language demonstrations both in English and in Italian following \citet{shi2022language}. Instead, for PAL, we propose a novel method by extending the original in English AUTHOR. We use reasoning tasks covering mathematical, commonsense reasoning, and natural language inference tasks in original versions (English) and adapted to Italian \href{https://github.com/lranaldii/italian_arithmetic_reasoning}{(resources available)}. These tasks are MGSM AUTHOR and MSVAMP AUTHOR, which consist of mathematical reasoning problems, and XCOPA AUTHOR, PAWS-X AUTHOR and XLNI AUTHOR which consist of commonsense reasoning and natural language inference. Finally, we select a range of different LLMs, we employ GPTs AUTHOR models for the results obtained in multilingual tasks, Phi-3 AUTHOR, and Mixtral AUTHOR for the results obtained in Italian benchmarks, different versions of Llama-2 and Llama-3 AUTHOR (adapted version for Italian, i.e., Llamantino-2 and -3 AUTHOR) and finally two Italian-centered LLMs for the improvements achieved by smaller-scale versions. We operate using the original models, and we propose aligned versions using state-of-the-art instruction-tuning methods based on synthetic data AUTHOR transferred for multilingual cases AUTHOR. The main contribution and findings of our paper are: \item Reasoning methods improve performance in Italian reasoning tasks as well as in English. However, although both methods bring tangible benefits, several limitations emerge in the natural language demonstrations employed in CoT. On the other side of the coin, we observe that the structured reasoning demonstrations (i.e., PAL) elicit the models to plan the solution in a more modularised way. Consequently, this benefits the final performance in both English and non-English tasks. \item We display the positive impact of structured in-context demonstrations on solution planning in Italian. We then demonstrate that since structured reasoning demonstrations are less ambiguous than natural language, they are more adaptable for math reasoning tasks and have a more noticeable impact in more articulate languages such as Italian. \item Finally, we show that the different LLMs analyzed in our contribution are able to understand problems in both English and Italian. However, performance in English is higher despite different approaches used to equate Italian and English proficiency. This reveals that the limitation is not derived from proficiency in a specific language but rather from the language's intrinsic difficulty To the best of our knowledge, this is the first work that investigates the impact of reasoning methods for the Italian and demonstrates how these strategies can consistently boost LLMs' performance, equipping them with the ability to generate step-wise explanatory reasoning for their predictions. To facilitate the reproduction, we share the data used at the following \href{https://github.com/lranaldii/italian_arithmetic_reasoning}{link}.

Conclusion
The advances of reasoning methods emerge beyond the English. Our analysis shows that properly elicited LLMs are capable of delivering reasoned answers in Italian as well. Indeed, by operating via CoT and PAL, we revealed that in-context demonstrations play a strategic role in improving performance in direct proportion to their quality and quantity. Our research highlights the need for a customised strategy for employing reasoning methods for LLMs in different languages. It supports the demand for a reasonable combination of model scale, reasoning technique, and strategic use of in-context learning to elicit the prospect of multilingual LLMs.