Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models - A Survey

Authors: philipp  mondorf
barbara  plank mainlp
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-20 13:31:05

Abstract: Large language models (LLMs) have recently shown impressive performance on tasks involving reasoning, leading to a lively debate on whether these models possess reasoning capabilities similar to humans. However, despite these successes, the depth of LLMs' reasoning abilities remains uncertain. This uncertainty partly stems from the predominant focus on task performance, measured through shallow accuracy metrics, rather than a thorough investigation of the models' reasoning behavior. This paper seeks to address this gap by providing a comprehensive review of studies that go beyond task accuracy, offering deeper insights into the models' reasoning processes. Furthermore, we survey prevalent methodologies to evaluate the reasoning behavior of LLMs, emphasizing current trends and efforts towards more nuanced reasoning analyses. Our review suggests that LLMs tend to rely on surface-level patterns and correlations in their training data, rather than on sophisticated reasoning abilities. Additionally, we identify the need for further research that delineates the key differences between human and LLM-based reasoning. Through this survey, we aim to shed light on the complex reasoning processes within LLMs.

Main paper: 
 introduction: 
 "These models are castles in the air. They have no foundations whatsoever."-Jitendra Malik (2021) Reasoning is an integral aspect of human intelligence and deliberate, rational thought (Holyoak & Morrison, 2005) . It allows individuals to draw conclusions from available information and move beyond their current knowledge (Lohman & Lakin, 2011) . As such, reasoning plays a fundamental role in problem-solving and decision-making, and has been a long-standing goal within the field of artificial intelligence (Robinson & Voronkov, 2001) .In recent years, large language models have demonstrated remarkable performance on tasks that require reasoning (Bubeck et al., 2023; Wei et al., 2022; Kojima et al., 2022) . This has sparked a vigorous debate about the extent to which these models possess reasoning abilities similar to humans (Mitchell & Krakauer, 2023; Mitchell, 2023; Borji, 2023) . While proponents argue that reasoning capabilities emerge with scale, referring to LLMs as foundation models (Bommasani et al., 2021; Kaplan et al., 2020) , skeptics contend that the models' performance primarily reflects their capacity to memorize the vast amount of data they have been trained on (Wu et al., 2024; Dziri et al., 2023; Razeghi et al., 2022; Zhang et al., 2023) . Thus, the question arises: are these models simply "castles in the air" with "no foundations whatsoever," as Malik (2021) once stated, or do they possess genuine reasoning capacities? One of the major challenges in this debate is the immense size, complexity, and closed-source nature of popular LLMs and their underlying training data. Moreover, the focus often lies on the models' performance on downstream reasoning tasks (Fu et al., 2023; Liu et al., 2023) , overshadowing in-depth analyses of their reasoning behavior. In this study, we seek to shed light on the ongoing debate by providing a comprehensive overview of research that goes large language models predominantly occurs through their engagement with designated reasoning tasks (Sun et al., 2024) . These tasks are designed to elicit the system's capability of drawing conclusions relevant to the problem at hand. In this survey, we distinguish between core and integrated reasoning tasks. Core reasoning tasks are designed to assess fundamental reasoning skills in an isolated manner. They typically aim to test a single type of reasoning, such as logical, mathematical or causal reasoning. Examples of such tasks may include syllogisms, basic arithmetic problems or structured causal-graph predictions. Conversely, integrated reasoning tasks require the concurrent use of various reasoning types, thereby assessing a combination of fundamental reasoning skills. Examples are commonsense or scientific reasoning tasks. Such problems often reflect the complex cognitive challenges humans encounter in everyday life and professional settings. A schematic representation distinguishing these task categories is provided in Figure 1 .For the purpose of this survey, our focus is confined to examining the behaviors of LLMs in the context of core reasoning tasks, specifically logical, mathematical, and causal reasoning tasks. We leave a review of the models' reasoning behaviors within the context of integrated reasoning tasks to future work. 
reasoning behavior of llms: 
 This section reviews studies that extend beyond mere task accuracy, focusing instead on evaluating the reasoning behavior of large language models.1 Through this review, we intend to address RQ1 and shed light on how these models currently behave across three core reasoning tasks: logical reasoning (Section 3.1), mathematical reasoning (Section 3.2), and causal reasoning (Section 3.3).The study of logical reasoning evolves around the question of how individuals infer valid conclusions from a set of given premises within a structured framework of logical rules and principles (Holyoak & Morrison, 2005) . Based on how conclusions are inferred, logical reasoning can be classified into three types: deductive, inductive, and abductive reasoning (Johnson-Laird, 2008; Flach & Kakas, 2000) . While in deductive reasoning, conclusions necessarily follow from the premises' truth, inductive and abductive reasoning are considered defeasible, i.e. conclusions are at most probable, but never necessary (Koons, 2022) . Inductive reasoning is concerned with deriving general conclusions from specific instances, whereas abductive reasoning entails formulating plausible hypotheses that explain the data observed.For additional details on the distinction between the three types of logical reasoning, we refer to prior surveys on reasoning performance within LLMs, such as the ones by Yu et al. (2024) or Sun et al. (2024) .Extensive research has been dedicated to examining the reasoning behavior of LLMs in the context of deductive reasoning tasks. Various studies analyze the validity and consistency of logical proofs generated by LLMs within deductive reasoning problems. For instance, Saparov & He (2023) conduct a systematic assessment of the rationales produced by various GPT-3 iterations (Brown et al., 2020) , through chain-of-thought (CoT) prompting (Wei et al., 2022) . By parsing each step of the generated reasoning trace into first-order-logic, its validity, atomicity, and utility2 is measured. Findings indicate that, in comparison to their smaller counterparts, larger models are more adept at generating both valid and atomic steps. However, the utility of these steps is often low, resulting in misleading steps from which the models struggle to recover. In a similar vein, Dziri et al. (2023) study rationales of LLMs by parsing them into computation graphs, where each node represents a partial solution of the multi-step reasoning process. Evaluations on Einstein's puzzle (Vassberg & Vassberg, 2009) indicate that GPT-3, ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI et al., 2024) solve the task by reducing multi-step reasoning into shortcut pattern matching. This shortcut behavior can yield correct answers when the model has been exposed to similar patterns during training, but falls short in generalizing to novel or more complex instances. Furthermore, theoretical findings suggest that due to the models' autoregressive nature, mistakes in early stages of the reasoning process can lead to compounding errors that exponentially diminish the likelihood of arriving at the correct solution in subsequent steps.Further studies highlight that LLMs tend to rely on superficial statistical features, rather than engaging in systematic reasoning. Chen et al. (2024b) illustrate that the premise order markedly influences the LLMs' behavior in propositional reasoning tasks. Specifically, when premises are presented in an order that does not align with the ground-truth proof, models such as ChatGPT, GPT-4, PaLM 2-L (Anil et al., 2023 ) and Gemini Pro (Team et al., 2023) encounter significant difficulties within their reasoning, even though such an ordering does not change the underlying logic. Zhang et al. (2023) show that an over-reliance on statistical features in the training data can hinder a model's reasoning and generalization capacity. By eliminating certain statistical cues from its training dataset, BERT (Devlin et al., 2019) demonstrates enhanced generalization capabilities in propositional logic. Similarly, Pirozelli et al. (2023) indicate that various fine-tuned language models show difficulties in transferring their logical reasoning ability when cross-probed on unseen deductive reasoning tasks.Additional research points to the difficulties of LLMs in understanding specific logical operators. Sanyal et al. (2022) show that GPT-3, RoBERTa (Liu et al., 2019) , and models from the T5 series (Raffel et al., 2020) exhibit deficiencies in comprehending logical negations, often failing to correctly deduce the implications of negated statements and rules. Similar findings have been reported by Truong et al. (2023) , who demonstrate that models lack a sensitivity to negations within a broad range of natural language inference tasks. Wan et al. (2024) comprehensively assess a suite of formal reasoning scenarios, subjecting GPT-3, ChatGPT, GPT-4, Bard (PaLM 2), Vicuna (Vicuna, 2023) and Guanaco (Dettmers et al., 2023) to minimum functionality tests (MFTs), serving as logical reasoning unit tests that gauge the models' inherent logic comprehension. Their analysis uncovers a common difficulty among models in identifying logical fallacies. In addition, GPT-4 appears to struggle with De Morgan's Laws, which relate conjunctions and disjunctions through logical negation.A growing body of research investigates the extent to which LLMs exhibit human-like reasoning patterns, particularly in deductive reasoning tasks. For example, Eisape et al. (2024) explore the parallels between human reasoning and that of LLMs in syllogisms. Their findings suggest that LLMs, much like their human counterparts, are susceptible to common logical fallacies and cognitive biases. Similarly, Dasgupta et al. (2022) find that LLMs, akin to humans, display content effects, indicating that the problem's semantic content can significantly influence the models' reasoning behavior. Further research corroborates the manifestation of human reasoning patterns in LLMs, as outlined in Appendix A.Mechanistic Evaluation. Additional research evaluates the reasoning behavior of LLMs by inspecting the models' internal mechanisms during the reasoning process. For example, Hou et al. (2023) analyze the models' attention patterns, specifically those of GPT-2 (Radford et al., 2019) and LLaMA (Touvron et al., 2023) , to uncover if and how these models perform multistep reasoning internally. Findings indicate a structured, step-wise information processing within the models. Furthermore, layer-wise probing reveals that LLMs tend to focus on identifying relevant information from the task in lower layers, transitioning to more intricate reasoning processes in higher layers. Pirozelli et al. (2023) similarly probe individual layers of a fine-tuned RoBERTa-large model, corroborating the pivotal role of higher layers in the reasoning process. Dutta et al. (2024) present an in-depth investigation into the internal mechanisms of LLaMA 2-7B (Touvron et al., 2023) when instructed via chain-of-thought prompting. Findings indicate a notable functional rift in the model's middle layers, where token representations in the initial half are biased towards pre-training priors, and an abrupt shift to in-context priors occurs in the latter half. As opposed to the findings by Hou et al. (2023) , the study suggests that LLaMA 2-7B employs multiple, concurrent pathways, instead of following a singular path of reasoning.In contrast to the well-examined domain of deductive reasoning, the reasoning behavior of LLMs in inductive reasoning tasks remains comparatively underexplored. Nevertheless, research exists that seeks to evaluate the capability of LLMs to infer general conclusions from specific examples. For instance, Yang et al. (2024) investigate how LLMs, such as GPT-J (Wang & Komatsuzaki, 2021) and LLaMA 7B, induce general rules from given facts. While in principle the models seem able to infer general rules from the data provided, challenges arise in ensuring that the rules are consistent with the premises, extend beyond the given information, align with real-world knowledge, and are pertinent to the given task. Similarly, Qiu et al. (2024) find that LLMs such as GPT-3.5, GPT-4, Claude 2 (Anthropic, 2023), and LLaMA 2-70B are capable of inferring rules from given data. However, the models frequently err in the application of these rules, highlighting a gap between their ability to generate and apply rules. Moreover, the rules derived often diverge significantly from those humans might produce, exhibiting a tendency towards verbosity and an inability to concentrate on the fundamental patterns for generalization. In addition, the study finds that LLMs display a pronounced sensitivity to alterations of the task descriptions. Han et al.(2024) examine the reasoning behaviors of GPT-3.5 and GPT-4 in property induction tasks, where properties common among different categories need to be induced. Findings suggest that GPT-4's behavior closely aligns with human judgments. However, challenges arise when the models need to handle premise non-monotonicity, a scenario in which adding more information to an argument can actually decrease its likelihood. For instance, Han et al. ( 2024) illustrate that the three-premise argument {crow, peacock, rabbit} → bird is considered weaker by humans than the two-premise argument {crow, peacock} → bird, a line of reasoning that both GPT-3.5 and GPT-4 seem to struggle with.Some efforts have been made to evaluate the behavior of LLMs in abductive reasoning tasks. Collins et al. ( 2022) ask both humans and GPT-3 to generate plausible explanations for unexpected counterfactual scenarios, for instance why a window did not break despite being struck by a rock. Analyses indicate that GPT-3's ability to generate plausible and coherent explanations for scenarios that require reasoning beyond established patterns is limited. In scenarios demanding inventive, coherent, and context-sensitive responses, especially when the usage of common nouns is restricted, human performance distinctly surpasses that of GPT-3, underscoring a pronounced deficiency in the model's ability to reason beyond its training distribution. Further, Rudinger et al. (2020) reveal that language models such as GPT-2, BART, and various T5 variants exhibit difficulties in identifying or generating arguments that either weaken or strengthen a given hypothesis. Even after fine-tuning, models tend to struggle with the task, often contradicting themselves by producing identical statements to strengthen and weaken the same premise-hypothesis pair. Xu et al. (2023) explore the behavior of GPT-3.5, ChatGPT, and PaLM 2 on various abductive reasoning tasks, paying particular attention to the models' rationales and errors manifested during reasoning. The paper highlights a tendency for LLMs to incorporate redundant information in their explanations and to generate content not grounded in the context, resulting in hallucinations. In comparison to deductive reasoning scenarios, findings suggest that models seem to particularly struggle with multi-hop reasoning in abductive reasoning tasks.Mathematical reasoning encompasses the structured process of arriving at conclusions based on established mathematical principles and logical deduction (Horsten, 2023) . Several studies explore the behavior of LLMs in the context mathematical reasoning tasks, especially in arithmetic and math word problems (MWPs). For instance, Srivastava et al. (2024) evaluate a set of LLMs on a functional variant of the MATH dataset (Hendrycks et al., 2021) , where the underlying mathematical principles of each problem are captured rather than a static problem formulation. This allows for evaluating models on different dataset snapshots, each comprising unique problem formulations but the same underlying reasoning process. Results reveal inconsistencies across varying snapshots, indicating a tendency of models to rely on memorization rather than reasoning. Razeghi et al. (2022) further highlight the impact of how a problem is formulated, noting a marked correlation between the frequency of terms in the models' pre-training data and their ability to solve arithmetic tasks. Similarly, Wu et al. (2024) observe that models struggle with two-digit additions expressed in mathematical bases that are less represented in the models' training data.Further studies underline the models' susceptibility to perturbations of the reasoning task. Other studies investigate whether human-like reasoning behavior in mathematical tasks manifests in LLMs. Hagendorff et al. (2023) show that, akin to humans, GPT-3 variants tend to offer intuitive yet incorrect answers to cognitive reflection tests (Frederick, 2005) . Notably, GPT-3.5 and GPT-4 provide more deliberate responses, outperforming humans in avoiding intuitive errors. At the same time, McKenzie et al. (2023) demonstrate a form of goal misgeneralization, where models, tasked with rounding numbers to a specific number of significant digits, consistently round based on the number of decimal places. This reflects a cognitive bias known as attribute substitution (Morewedge & Kahneman, 2010) , where a more challenging task is replaced with a simpler, related task. In a mechanistic evaluation, Chen et al. (2024a) conduct a layer-wise analysis of LLaMA's mathematical reasoning capabilities, discovering that higher layers exhibit superior mathematical problem-solving abilities, while lower layers seem to lack basic arithmetic and factual knowledge.Causal reasoning is the process of discerning the cause-and-effect relationships that govern the dynamics of our environment (Sloman, 2005) . Extending beyond correlation, it offers a more nuanced understanding of how changes in one variable can bring about changes in another (Pearl, 2009) . For a comprehensive introduction to causal reasoning, we strongly recommend the work of Pearl (2009) . While research on the behavior of LLMs in causal reasoning tasks is still in its early stages, the field is receiving growing attention. Various studies to date suggest that LLMs are capable of reciting causal facts from their training data, but lack an intrinsic ability to comprehend or construct causal relationships. For instance, Zečević et al. (2023) 2023)'s evaluation of GPT-3 and PaLM on the blicket detector task (Gopnik & Sobel, 2000) , where models need to infer which objects cause a light to switch on, further highlights challenges in causal reasoning with LLMs. While models can identify the correct causal structure when a set of causal hypotheses is provided, they struggle in conditions where the hypotheses are not given, underscoring a limitation in their ability to infer causal relationships from limited data. Binder (2022) , focus on the ability of LLMs to predict outcomes in hypothetical setups, identifying a significant gap between the capabilities of humans and LLMs, particularly in highly unrealistic scenarios. Li et al. (2023) further illustrate that while models like GPT-3 can produce outcomes that seem to align with counterfactual propositions, these models heavily rely on simple lexical cues within the given context, rather than demonstrating a profound grasp of the scenarios' hypothetical essence. Yu et al. (2023) evaluate LLMs on questions embedded in counterfactual presuppositions, pushing the models beyond simple fact retrieval. Their findings suggest that "closed-book" models such as GPT-3 and code-davinci-002 may fabricate facts or base their responses on flawed premises when answering counterfactual questions. Huang et al. ( 2024) explore a different angle by asking models to modify a given piece of argumentative text so that it upholds a specified logical relationship under new premises. Findings indicate that while LLMs like GPT-3.5 and GPT-4 demonstrate a capacity for solving such tasks, they struggle with modifying arguments such that a new premise makes the argument less likely to be true, an observation also made by Rudinger et al. (2020) in the context of abductive reasoning.Summary. Regarding RQ1, our review suggests that the reasoning behaviors of LLMs are nuanced and diverse, yet a significant trend emerges: while current LLMs demonstrate proficiency in reasoning problems that align with their training distribution, they frequently encounter substantial conceptual difficulties in out-of-distribution scenarios. Notably, slight alterations in the task context can markedly impair the models' reasoning capabilities. Multi-step reasoning is often reduced to shortcut pattern matching, and fundamental conceptual errors highlight the models' deficiency in understanding basic principles of logic, mathematics, and causal reasoning, particularly in counterfactual setups.  
evaluation methods: 
 So far, we have provided an overview of studies that evaluate the behavior of LLMs on three core reasoning tasks. However, to date, a standardized methodology for assessing the reasoning capabilities of large language models is lacking. To address RQ2, we review and categorize predominant evaluation frameworks for analyzing the behavior of LLMs in reasoning tasks. As depicted in Figure 2 , we categorize evaluation methodologies into four distinct groups: (i) conclusion-based, (ii) rationale-based, (iii) interactive, (iv) and mechanistic evaluations. In the following sections, we further delineate each category by discussing prevalent techniques. An overview of the advantages and disadvantages of each evaluation procedure can be found in Table 1 . Additional details are provided in Appendix B.In conclusion-based evaluation schemes, emphasis is placed on the model's final answer rather than the process by which the model arrives at its conclusion. This outcome-oriented approach, despite its limitation of overlooking the model's rationales, can nonetheless provide insights into the model's reasoning behavior. For instance, a thorough error analysis can unveil conceptual errors (Sanyal et al., 2022) , reflections of cognitive biases (Dasgupta et al., 2022) , or sensitivities to the task context (Wu et al., 2024) . Similarly, an examination of the model's output distribution may reveal inherent predispositions towards certain outcomes (Itzhak et al., 2024) , or serve as an indicator of the model's confidence in specific conclusions (Frohberg & Binder, 2022) . However, relying solely on the model's final conclusion can be less reliable than also considering how the model arrives at its conclusion. For example, candidate answers derived from first token probabilities in multiple-choice setups are often not robust (Wang et al., 2024b) , and final answers might not always align with the model's verbalized reasoning (Mondorf & Plank, 2024; Ye & Durrett, 2022) . Moreover, answers to benchmark datasets might have been compromised by data leakage during the model's training process, limiting the insights that can be drawn from a correct conclusion (Balloccu et al., 2024; Xu et al., 2024) . To address the issue of data contamination and test the model's capacity to generalize, dynamic benchmarks can be employed that update over time. For instance, functional benchmarks capture the underlying reasoning process rather than a static problem formulation, probing the model's reasoning capabilities on dataset snapshots released quarterly (Srivastava et al., 2024) .In contrast to conclusion-based evaluation schemes, rationale-based evaluation methods focus on examining the reasoning trace generated by the model, typically assessing its logical validity and coherence. While this approach allows for more nuanced insights into the model's reasoning behavior, rationale-based evaluation schemes are often more challenging to automate and scale. Given the variability in the model's reasoning, a spectrum of assessment techniques exist. For rationales following a highly consistent format-either through structured contexts or methods that guide the model's response style (Wan et al., 2024) -rationales can be parsed into more formalized representations such as first-order logic (Saparov & He, 2023) , computation graphs (Dziri et al., 2023) , or causal graphs (Willig et al., 2022) , facilitating a more granular examination. Alternatively, interpretable quantitative metrics, such as ROSCOE (Golovneva et al., 2023) or RECEVAL (Prasad et al., 2023) , may be utilized to evaluate the rationales' semantic alignment with the reasoning task, their coherence, factual consistency, and logical validity. In instances where rationales are less structured, qualitative inspections are commonly employed, either through diagnostic agents (Huang et al., 2024) , or human judgments (Mondorf & Plank, 2024) .Similar to the principle of dynamic assessment within psychology (Haywood & Tzuriel, 2002) , interactive evaluation offers a framework to engage with LLMs during the evaluation. This approach allows for more flexible assessments tailored to the model's specific reasoning behavior. Although such evaluations are often costly and challenging to scale, several techniques have been developed to automate the process. For instance, adaptive evaluations dynamically select reasoning tasks based on the model's responses, thus providing deeper insights into its capabilities and limitations beyond what static questionnaires can reveal (Zhuang et al., 2023) . Dialectic evaluation methods assess the model's reasoning in dialogue form, for example, by challenging the model's conclusions (Wang et al., 2023) , or engaging it in game-theoretical scenarios (Bertolazzi et al., 2023) . While interactive evaluations yield nuanced insights, they lack the structure of traditional methods, posing challenges in terms of standardization and reproducibility.Mechanistic evaluations of LLMs delve into the underlying processes that drive the model's responses, aiming to uncover the "how" and "why" within their reasoning processes. By analyzing the internal mechanisms such as attention patterns (Hou et al., 2023) , activation flows (Dutta et al., 2024) , and the functional attributes of individual layers (Pirozelli et al., 2023) , deeper insights into the model's operational logic can be gained, as illustrated in Section 3.1.1. Focusing on the model's intrinsic processes, this framework contrasts with previous approaches, drawing parallels to the study of human reasoning from a neuroscientific perspective (Papo, 2015) . Nonetheless, current methods remain computeintensive, and their findings may not always generalize across different models and tasks (Bereska & Gavves, 2024; Ferrando et al., 2024) . 
discussion: 
 Despite the notable performance of large language models in prominent reasoning tasks (Bubeck et al., 2023; Fu et al., 2023) , our review suggests that current models more closely resemble stochastic parrots (Bender et al., 2021) than systematic reasoners. As discussed in Section 3, we find that although many LLMs demonstrate proficiency in reasoning problems that align with their training data, the models' reasoning behavior reveals significant conceptual errors and limitations in out-of-distribution scenarios. As highlighted by Mahowald et al. (2024) , this suggests a limited functional linguistic competence in LLMs. It is likely that the apparent success of LLMs in reasoning tasks predominantly reflects their ability to memorize the extensive data they have been trained on (Wu et al., 2024; Dziri et al., 2023) .Recent studies indicate that a substantial amount of benchmark datasets has been leaked to current LLMs (Balloccu et al., 2024; Xu et al., 2024) , raising concerns about the insights derived from their performance on such benchmarks. Therefore, we advocate for more nuanced analyses of the models' reasoning behavior, particularly in novel scenarios that the models have not previously encountered.While human reasoning is not infallible (Johnson-Laird, 2010), the human capacity for robust reasoning and generalization from limited data remains unmatched by current LLMs. Various studies point to the fundamental differences between human reasoning and that of LLMs, especially the models' restrictive autoregressive pre-training objective (McCoy et al., 2023; Shanahan, 2024; Lenci, 2023) . We call for further research-particularly on reasoning behavior-within both humans and LLMs to better discern and comprehend the essential components missing in LLMs, which are crucial for robust and systematic reasoning. 
conclusion: 
 This survey provides a comprehensive review of literature that evaluates LLM-based reasoning beyond mere task accuracy, offering deeper insights into the reasoning behavior of large language models. We discuss the behavior of LLMs across three core reasoning tasks (RQ1), assessing logical, mathematical and causal reasoning. Furthermore, we outline predominant evaluation frameworks and compare their respective strengths and limitations (RQ2). Our findings indicate that although LLMs demonstrate proficiency in reasoning problems that align with their training data, they often encounter significant conceptual challenges in out-of-distribution scenarios. Considering these insights and the recent issue of benchmark dataset leakage (Balloccu et al., 2024; Xu et al., 2024) , we caution against relying solely on shallow accuracy metrics for static reasoning tasks. Instead, we advocate for more sophisticated reasoning assessments, such as those discussed in Section 4. 
 
 Appendix: Recent research has begun to explore the extent to which human reasoning behaviors are reflected in large language models, given their training on human-generated data. Various studies delve into the manifestation of human-like reasoning behaviors within LLMs in the context of deductive reasoning tasks. For instance, Eisape et al. (2024) compare the behaviors of humans and LLMs, particularly those from the PaLM 2 series, in syllogistic reasoning tasks. The findings reveal that, similar to humans, LLMs are prone to logical fallacies and cognitive biases such as ordering effects. This susceptibility persists across model sizes, though larger models tend to engage in more deliberate reasoning, showing a reduced sensitivity towards these errors. A notable finding indicates that LLMs, unlike humans, rarely produce the "nothing follows" response, even when it represents the accurate deduction. In a similar vein, Dasgupta et al. (2022) demonstrate that LLMs, akin to humans, exhibit content effects in predicate logic problems, i.e. their reasoning is influenced by the semantic content of the task. This effect is evident in various LLMs including ChatGPT, PaLM 2, and Chinchilla (Hoffmann et al., 2022) . In particular, analyses indicate that the models' reasoning in syllogisms is biased by the believability of the conclusion, a behavior known as belief bias (Klauer et al., 2000) . Further research corroborates the influence of semantic content on the models' reasoning. Ando et al. (2023) demonstrate a belief-bias in models like ChatGPT, RoBERTa and BART (Lewis et al., 2020) by comparing the models' behavior on abstract, belief-consistent, and belief-inconsistent syllogisms. Their analyses also uncover a predisposition towards conversion errors and atmosphere effects, where logical quantifiers are either misinterpreted or lead to uninformed inferences, respectively (Tversky & Kahneman, 1974) . Itzhak et al. (2024) further show that methods such as instruction tuning and reinforcement learning from human feedback (RLHF) (Ouyang et al., 2022) may amplify cognitive biases within LLMs. Mondorf & Plank (2024) delve into the inferential strategies of open-access LLMs in problems of propositional logic. Comprehensive manual evaluations of the models' rationales reveal that LLMs utilize inferential strategies similar to those employed by human reasoners. Their evaluations further underline difficulties of LLMs with logical negations and a vulnerability to logical fallacies commonly observed in human reasoning. Similarly, McKenzie et al. (2023) indicate that LLMs tend to replicate human-like logical errors when engaging with the logical principle of modus tollens, suggesting an imitation of flawed reasoning patterns from their training data. Notably, this trend becomes more pronounced with increasing model size, a phenomenon denoted as inverse scaling.In this section, we offer a more detailed overview of the evaluation frameworks commonly employed to examine the reasoning behavior of large language models. Expanding on the taxonomy depicted in Figure 2 , we present conclusion-based evaluation methods in Table 2 , provide an overview of rationale-based evaluation approaches in Table 3 We consistently specify when a particular technique, other than standard prompting, is utilized. For a review of prompting approaches designed to improve the models' reasoning performance, we recommend consulting prior surveys byHuang & Chang (2023),Yu et al. (2024), orSun et al. (2024).Validity denotes whether the proof step logically follows from preceding steps, atomicity reflects whether it can be proven with exactly one application of a deduction rule, and utility measures its direct contribution towards the derivation of the final conclusion. 


Decision: This paper provides a review of the reasoning abilities of LLMs and the prevalent approaches for evaluating these reasoning abilities.

The reviewers highlighted several strengths of this paper: it provides clear and succinct summaries of a wide range of papers, its topic is important and timely, the taxonomy of evaluation types that it provides is useful, and it provides some clear high-level conclusions and takeaways rather than simply listing papers (e.g., summarizing the strengths and weaknesses of evaluation types, and giving a big-picture view of LLM reasoning abilities).

Limitations that reviewers noted included the fact that some relevant strands of literature were not discussed (particularly prompting approaches for improving LLM reasoning, such as CoT), imprecision in some claims (particularly what is meant by brittleness and memorization), and some presentational shortcomings (i.e., insufficient highlighting of key points and takeaways). 

Despite these limitations, I believe that the paper will be a useful one that will be of interest to a large subset of the COLM audience. Therefore, I am recommending acceptance. I encourage the authors to make the following revisions to the paper (in the camera-ready version if it is accepted to COLM, or in the version that the authors submit elsewhere if it is not accepted):
- Adding the table that the authors discuss in the response to Reviewer bLrd; this would strengthen the paper and help address some of these limitations.
- Adding some discussion of prompting approaches that improve reasoning performance such as CoT, ReAct, etc. The fact that this suggestion came up in two reviews suggests that many readers will have this reaction too. Therefore, even if you view it as out-of-scope, it seems worthwhile to add some discussion of it - even if that discussion simply states that these directions are out-of-scope.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This paper is a survey of reasoning behavior in large language models. It seeks to answer 2 research questions: 1) how do current large language models behave in mathematical, causal and logical reasoning tasks, and 2) what are the evaluation methods for assessing the reasoning capabilities of large language models. The survey finds that LLMs rely on pattern matching to reasoning examples in their training distribution, leading to them making conceptual errors in reasoning particularly in out-of-distribution scenarios. To better assess true reasoning behavior, the survey categorizes evaluation methods into conclusion-based, rationale-based, interactive and mechanistic evaluations.

The paper is a well-researched summary of assessments of reasoning behavior in large language models. By using a taxonomy of logical (deductive, inductive, abductive), mathematical, and causal reasoning tasks, the paper is able to capture a good breadth of the literature, and serves as a good overview of the reasoning abilities and deficiencies of current large language models. The writing is clear and easy to follow, and is often able to capture the main ideas of cited papers in a succinct manner. This is a noteworthy contribution to a community that is contending with an open question of how much reasoning abilities truly emerge at scale, or if LLMs are just "stochastic parrots" of their data. 

That said, the paper could have gone further in its contribution to the debate. As it stands, the survey is opinionated in putting forth a coherent argument for LLMs being a reflection of their training distribution by highlighting deficiencies in reasoning behavior. There could have been further analysis on why eliciting reasoning behavior (chain-of-thought, tree-of-thought style or the ReAct framework) tend to improve reasoning outcomes, as at least at first glance this isn't immediately suggestive of pattern matching. In fact, a taxonomy on such methods to induce reasoning ability would lead to a more comprehensive survey; while the paper is more focused on evaluation methods, there seems to have been a missing component of how that reasoning behavior is obtained (via prompting, etc.) which is a first-order discussion.

Moreover, Section 4 (taxonomy of evaluation methods) could have gone further in discussing 1) the overall recent trends for the area of reasoning evaluation, 2) why the field has evolved as such, and 3) the tradeoffs using each evaluation method in the taxonomy. This would go a long way in providing a suggested direction for future work in improving our methodology for assessing reasoning capabilities.
strength_weakness: strength_weakness: 1. The paper does not synthesize trends or provide suggestions for future work in evaluating reasoning abilities of LLMs. 
2. The paper is missing some discussion on how reasoning behavior is elicited from LLMs, which seems important for such a survey paper.
questions: questions: 1. Do the authors agree that a section on how reasoning behavior is induced from LLMs (i.e. why reasoning outcomes are better upon prompting for rationales, or structuring thoughts in a particular form in prompts) is important? If not, why not?
2. Upon review of the literature surrounding evaluation methods for reasoning behaviors, for each evaluation method can the authors provide a) a synthesis of the tradeoffs for that method, and b) suggestions for future work addressing the limitations of that evaluation method
3. (comment) Section 3.1.1 includes an explanation of mechanistic evaluation as part of the review on behavior in deductive reasoning tasks, but this is later largely duplicated in Section 4.4. This seems redundant, or could be better structured to avoid duplication.
paper_summary: paper_summary: This paper provides a comprehensive review of literature that evaluates LLM-based reasoning beyond mere task accuracy, which covers three core reasoning tasks, i,e,, logical, mathematic, and causal reasoning, as well as current evaluation frameworks. The review is clear, and helpful to the researchers in this community.
strength_weakness: strength_weakness: 1) The review lacks some concrete examples to specify the advantage and disadvantage of the evaluaiton frameworks. 

2) Not sure about whether such reviews fall in the scope of the conference.
questions: questions: In the conclusion, the authors highlighted the necessity for more nuanced analyses of the models’ reasoning behavior, especially in novel scenarios, and how about the evalution of the capability of the LLMs in few-shot learning scenarios? it will be nice to provide analysis of such work.
paper_summary: paper_summary: The paper is a review of recent literature of evaluating LLM reasoning. Authors focus specifically three families of core reasoning tasks: mathematical, logical and causal reasoning. After reviewing recent work, they discuss general trends in reasoning evaluation, arguing for paradigms that go beyond measuring the correctness of the outcome of a reasoning process and that also evaluate the reasoning process itself. They suggest reasoning capabilities of modern LLMs are not robust yet.
strength_weakness: strength_weakness: 1. Some claims don't feel well-supported. 
    1. I’m not sure I’m buying the distinction between reasoning behavior vs final task performance. You can still study reasoning behavior by tracking final task performance if you design your tasks really well such that they track particular kind of error or behavior. For instance, one can measure the compositionality gap [1] (the accuracy difference between two one-hop and a single two-hop question) to study a sophisticated reasoning behavior (two-hop reasoning) while relying on metrics tracking just an outcome of a well-designed task.

   2. How do authors motivate defining core reasoning tasks a logical, mathematical or causal reasoning? Why are they non-overlapping (e.g. doesn’t math reasoning pre-supposed logical reasoning)? Why do they exhaust the space of reasoning types? Should this typology be thought as listing natural kinds, carving nature at its joints or is it just a useful albeit fuzzy set of labels that LMs might not might not respect?

   3. Authors frequently hint at the brittleness of reasoning capabilities being a result of memorization, but don’t discuss this in detail and more rigorously. The strong reading of this claim is clearly false: LLMs do generalize to unseen problems, e.g. the gsm1k paper [2] finds that only some models (Mistral) show overfitting to gsm8k that could explain their good performance. I wonder what’s the more nuanced reading of the claim authors hint at. I’d be great to see it discussed in more detail.

2. The survey seems to be missing some important thread of recent work and is somewhat shallow when discussing some others.
    1.  This is less important, but some threads feel missing to me:
         1. Performance boosts from techniques for eliciting reasoning capabilities, e.g. chain of thought, self-ask, using feedback; the gap between externalized reasoning and zero-shot reasoning performance. Some relevant papers:
            1. Do Large Language Models Latently Perform Multi-Hop Reasoning?
            2. Large Language Models Cannot Self-Correct Reasoning Yet
        2. Faithfulness of externalized reasoning: 
            1. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
            2. Measuring Faithfulness in Chain-of-Thought Reasoning
        3. The role of knowledge representation in multi-hop reasoning
            1. Physics of Language Models: Part 3.2, Knowledge Manipulation
            2. Taken out of context: On measuring situational awareness in LLMs
   2. I’m a bit worried that the authors paint their picture relying on some pre-2021 papers conducting experiments on GPT-3, RoBERTa and T5 or even GPT-2 and BART! I think modern frontier LLMs (such as Claude 3 Opus, GPT-4o) are significant more capable than GPT-3-series models and a lot of findings of prior work don’t translate to them.
   3. Section 4.4 feels very blunt: “deeper insights into the model’s operational logic can be gained” is a cliche, please tell us what we’ve learned exactly. 

Some more specific comments:

> skeptics contend that the models’ performance is a mere reflection of the extensive training data and their vast number of parameters (Razeghi et al., 2022; Wu et al., 2023; Berglund et al., 2024). 

I don’t think Berglund et al., 2024 lends evidence to this claim. The reversal curse is a failure of knowledge storage/retrieval not reasoning.

Moreover, I’d rephrase the claim to make it more precise. That reasoning capability is a result of increased scale is not a skeptical take at all. We all know that larger models and models trained on more data generally tend to get better results. Probably what the authors wanted to say is something about memorization vs generalization?

> Furthermore, theoretical findings suggest that due to the models’ autoregressive nature, mistakes in early stages of the reasoning process can lead to compounding errors that exponentially diminish the likelihood of arriving at the correct solution in subsequent steps.

Citation needed. Maybe [3], section 4?

> At the same time, McKenzie et al. (2023) demonstrate a form of goal misgeneralization

I’d avoid the term “goal misgeneralization” as it has its own technical meaning [4]

> extensive training data, rather than genuine reasoning capabilities

This strikes me as a false dichotomy. We don’t have any reason to think that genuine reasoning capabilities cannot be learned from extensive training data. What the authors mean is perhaps memorization vs generalization again. 

> Similarly, McKenzie et al. (2023) indicate that LLMs tend to replicate
human-like logical errors when engaging with the logical principle of modus tollens, suggesting an imitation of flawed reasoning patterns from their training data. Notably, this trend becomes more pronounced with increasing model size, a phenomenon denoted as *inverse scaling*.

This is too strong Some tasks show inverse scaling, but it’s a minority and tasks in McKenzie et al. (2023) were selected for exhibiting inverse scaling. For the vast majority of reasoning tasks, performance improves with scale.

[1] https://arxiv.org/abs/2210.03350

[2] https://arxiv.org/abs/2405.00332

[3] https://arxiv.org/pdf/2305.18654

[4] https://arxiv.org/abs/2210.01790
paper_summary: paper_summary: This work summarizes the current reasoning works, including logical reasoning, mathematical reasoning, and causal reasoning. Firstly, they introduce the development of reasoning works. Then, they propose a taxonomy of evaluation methods to analyze the reasoning behaviors of LLMs. However, for the discussion part, there is no deeper insight into the models' reasoning processes. Since the paper is a survey evaluating the reasoning behavior of LLMs, it should summarize the current weaknesses of LLMs and propose some future directions.
strength_weakness: strength_weakness: 1. Lack of deep discussion. There is no deeper insight into the models' reasoning processes. They should summarize the current weaknesses of LLMs and propose some future directions.
questions: questions: N/A
paper_summary: paper_summary: The paper provides a comprehensive literature review on the evaluation of reasoning behaviors in large language models (LLMs). It effectively compiles findings from various studies, categorizing them by different reasoning tasks. Furthermore, the paper examines evaluation strategies for these behaviors, organizing them into four distinct methodological categories.
strength_weakness: strength_weakness: 1. The framework for categorizing reasoning tasks is not particularly novel. Similar approaches have been detailed in existing literature, such as the HELM paper, which also evaluates models on both atomic and composite reasoning tasks. The choice to categorize reasoning behaviors according to specific tasks could be reconsidered. This is because certain erroneous behaviors manifest similarly across different tasks. For instance, an invalid entailment might represent a common type of reasoning error, yet it is characterized differently depending on the task at hand. A taxonomy based solely on tasks risks overlooking these cross-task similarities, which are crucial for deeper analytical insights.
2. The structure of the paper needs refinement to enhance its readability and impact. Key findings from the literature could be more effectively highlighted or presented in a more streamlined manner. As it currently stands, the paper's main points are somewhat obscured by overly lengthy and complex passages, making it difficult for readers to quickly grasp essential insights.
paper_summary: paper_summary: This paper performs a survey on evaluating the reasoning behavior of LLM from two aspects:
1) the behavior of LLM on reasoning tasks.
2) the evaluation method of LLM reasoning.
Overall, the paper is well-written and includes the up-to-date literatures with respect to LLM reasoning. I think the practitioners will be interested in this paper and get insights from this paper about the research progress of LLM reasoning.
strength_weakness: strength_weakness: 1) The presentation need to be substantially improved. Long paragraphs of text discussion are tiring to read. It is difficult to quickly grasp the key findings or conclusions in publication survey. I think the authors should add more figures and tables as well as highlighted text to better understand.

Comments: title: Response to Reviewer bLrd
Response to Reviewer AG6y
Response to Reviewer AG6y
comment: We appreciate the reviewer's valuable feedback and suggestions.

We agree that such a synthesis would be beneficial to this survey and are committed to incorporating it into the final version of our paper.
Thank you for the authors' response. I acknowledge that giving an overview of reasoning techniques expands the scope of the survey, but I do encourage the authors to give their taxonomy of the techniques in the appendix of the final paper if accepted.

I maintain my view that a synthesis of the tradeoffs for evaluation methods and suggestions for future work are critical for the narrative of such a survey paper. I appreciate the authors' summary of the tradeoffs, which seem sensible to me. I will maintain my score for now, but the paper would deserve a 7 if each section on evaluation techniques ends with this summary and provides the community with research ideas to pursue to address the disadvantages of the respective techniques.
Thanks for the response. Let's agree to disagree then! I think it's perfectly fine to present an interpretation of the reversal curse as a reasoning failure in the paper, but it should be clear it's a novel interpretation put forth by the authors of the present paper and it should be supported by some arguments.

> Specifically, the reversal curse illustrates “a basic inability [of LLMs] to generalize beyond the training data” (see Section 1), which will have a significant impact on their reasoning behavior.

I agree it will impact reasoning behavior but IMHO it's not justified to call it a reasoning failure. If a model training data cutoff is 2023, the model won't be able to reason about the gold and silver medalists of 2024 Summer Olympics. But we wouldn't call that a reasoning failure, would we?
We appreciate the reviewer’s engaging discussion on the reversal curse.

This phenomenon can be addressed from two perspectives: knowledge retrieval/storage or logical deduction. While we agree that the in-context learning experiments of [Berglund et al., 2024](https://doi.org/10.48550/arXiv.2309.12288) suggest that the reversal curse might stem from a failure of information retrieval, we respectfully disagree that this phenomena is unrelated to the model’s reasoning. As noted by the authors, while attributing the reversal curse solely to logical deduction may be considered "a simplification," it remains a "useful" perspective to relate it to deductive reasoning. Specifically, the reversal curse illustrates “a basic inability [of LLMs] to generalize beyond the training data” (see Section 1), which will have a significant impact on their reasoning behavior.
Thanks for the response. I will slightly improve the rating score. However, more discussion should be considered in the final version.
Thanks for the response!

> The reversal curse indicates that models struggle to capture a basic symmetry property of the identity relation, instead learning a more superficial, unidirectional relationship. As noted by the authors, this is “a basic failure of logical deduction in the LLM’s training process”, which could extend to other relationships that are essential for effective reasoning such as “logical implications [...], spatial relationships [...], or n-place relations.” (see Section 1 and 4.1 of Berglund et al. 2024)

I disagree. The reversal curse does not happen in context: models are perfectly capably of inferring "Joe Biden is US president" when **prompted** with "US president is Joe Biden"; reversing relations like this is not a problem. They just struggle to do that during in-weights learning. This suggests it's a failure of knowledge storage/retrieval (when a "A is B" is stored in one direction, models fail to retrieve A when conditioned on B) not of logical deduction. 

The first passage you cited is a hypothetical that is debunked two sentences later: "Moreover, this is not explained by the LLM not understanding logical deduction." The second cited passage is also a question, not a claim.

> The limited number of tasks in McKenzie et al. (2023) does not invalidate the authors' finding that LLMs replicate certain types of human-like logical errors present in their training data.

They *sometimes* replicate human errors. I think asserting this without a "sometimes" is a claim the McKenzie et al. don't provide sufficient evidence for.

> The term "goal misgeneralization" is used by McKenzie et al. (2023) and by us in the same context as defined in reference [4]. Could the reviewer please provide further clarification?

You're right!
Thanks for your reply. I have no further questions and maintain my score.
Thank you for the response. Please find our answer to the more specific comments below.

Missing thread of work.
1. We appreciate the reviewer's suggestions regarding interesting aspects of LLM-based reasoning. However, we have already reviewed a substantial number of recent papers and believe that incorporating the reviewer's suggestions would expand the scope of our survey beyond its intended focus.
2. We have reviewed very recent studies up to March 2024 (the month of submission) and have reported various findings for advanced models such as GPT-4, PaLM 2, and Claude 2. Additionally, we consistently specify which LLM was utilized in each respective study.

Regarding the reviewer’s more specific comments:
1. The reversal curse indicates that models struggle to capture a basic symmetry property of the identity relation, instead learning a more superficial, unidirectional relationship. As noted by the authors, this is “a basic failure of logical deduction in the LLM’s training process”, which could extend to other relationships that are essential for effective reasoning such as “logical implications [...], spatial relationships [...], or n-place relations.” (see Section 1 and 4.1 of [Berglund et al. 2024](https://doi.org/10.48550/arXiv.2309.12288))
2. The mentioned paragraph relates to the previous sentence discussing the work of [Dziri et al. (2023)](https://doi.org/10.48550/arXiv.2305.18654).
3. The term "goal misgeneralization" is used by [McKenzie et al. (2023)](https://doi.org/10.48550/arXiv.2306.09479) and by us in the same context as defined in reference [[4]](https://doi.org/10.48550/arXiv.2210.01790). Could the reviewer please provide further clarification?
4. The limited number of tasks in [McKenzie et al. (2023)](https://doi.org/10.48550/arXiv.2306.09479) does not invalidate the authors' finding that LLMs replicate certain types of human-like logical errors present in their training data.
Thanks for the response! Where can I find that separate rebuttal in which you respond to my specific comments?
Thanks for the authors' responses. I have no questions and maintain my score.