Comparing Inferential Strategies of Humans and Large Language Models in Deductive Reasoning (2024)

Abstract: Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments. It allows individuals to draw conclusions that logically follow, given the truth value of the information provided. Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks. Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior. In this study, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems. Our findings indicate that LLMs display reasoning patterns akin to those observed in humans, including strategies like supposition following or chain construction. Moreover, our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning, with more advanced models tending to adopt strategies more frequently than less sophisticated ones. Importantly, we assert that a model’s accuracy, that is the correctness of its final conclusion, does not necessarily reflect the validity of its reasoning process. This distinction underscores the necessity for more nuanced evaluation procedures in the field.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9370–9402 August 11-16, 2024 ©2024 Association for Computational Linguistics
Deductive reasoning plays a pivotal role in the formulation of sound and cohesive arguments. It allows individuals to draw conclusions that logically follow, given the truth value of the information provided. Recent progress in the domain of large language models (LLMs) has showcased their capability in executing deductive reasoning tasks. Nonetheless, a significant portion of research primarily assesses the accuracy of LLMs in solving such tasks, often overlooking a deeper analysis of their reasoning behavior. In this study, we draw upon principles from cognitive psychology to examine inferential strategies employed by LLMs, through a detailed evaluation of their responses to propositional logic problems. Our findings indicate that LLMs display reasoning patterns akin to those observed in humans, including strategies like supposition following or chain construction. Moreover, our research demonstrates that the architecture and scale of the model significantly affect its preferred method of reasoning, with more advanced models tending to adopt strategies more frequently than less sophisticated ones. Importantly, we assert that a model’s accuracy, that is the correctness of its final conclusion, does not necessarily reflect the validity of its reasoning process. This distinction underscores the necessity for more nuanced evaluation procedures in the field.

heading: 1 Introduction
text: Deductive reasoning, that is the process of drawing conclusions that logically follow from the information at hand, is an integral aspect of human cognition and plays a pivotal role in formulating sound and coherent arguments (Leighton, 2003). Take, for example, the following statements:

heading: If there is a blue marble in the box then there is a
text: green marble in the box. There is a blue marble in the box.
Even without proper training in logic, most individuals can naturally deduce the valid conclusion: Therefore, there is a green marble in the box.
This innate capability of drawing conclusions that invariably follow from the truth value of available information has been a focal point of scholarly interest for centuries (Holyoak and Morrison, 2005). Propositional logic, a subfield of deductive reasoning, focuses on constructing logical arguments based on the relationship between statements similar to those in the example previously mentioned (Hurley, 2011). Extensive research has been ded-
9370
icated to examining human reasoning behavior in contexts that involve propositional logic. For instance, Van der Henst et al. (2002) have identified five different strategies people commonly employ when navigating problems of propositional logic (see Section 2). Such behavioral studies have been crucial in shaping theories that shed light on the fundamental elements of cognitive reasoning processes (Rips, 1994; Johnson-Laird, 1986; Kahneman et al., 1982).
In parallel, recent advancements in the field of large language models have demonstrated their potential in executing tasks involving deductive reasoning (Yang et al., 2023; Yu et al., 2024; Huang and Chang, 2023). Yet, the extent to which LLMs truly possess such abilities remains a subject of ongoing debate (Mahowald et al., 2024; Mitchell and Krakauer, 2023). Unlike behavioral studies in human reasoning that are often characterized by in-depth examinations of the reasoners’ expressions, many studies on LLM-based reasoning tend to focus on task performance and accuracy metrics, offering limited insights into the underlying reasoning behavior of the models (Mitra et al., 2023; OpenAI et al., 2023; Team et al., 2023).
In this paper, we draw from the cognitive science literature (Van der Henst et al., 2002) and study inferential strategies employed by LLMs when solving propositional logic problems (see Figure 1). We analyze the reasoning behavior of three different language model families, varying in model size and fine-tuning procedure, and compare them to the behavior found in humans. To the best of our knowledge, we are the first to comprehensively compare inferential strategies employed by large language models and humans. We analyze the models’ output both quantitatively and qualitatively via manual inspection, to provide insights into the soundness of their verbalized reasoning strategies. Our findings reveal that:
• All models exhibit inferential strategies akin to those observed in human reasoning, such as supposition following and chain construction.
• The inferential strategy employed is significantly influenced by the model family, as different families favor different approaches.
• Models are often right but for the wrong reasons: the accuracy of a model, that is the number of correct final conclusions, does not reflect whether its reasoning is sound, i.e. logically follows from the statements at hand.
• The strategy employed by a model is closely related to the soundness of its reasoning, where certain strategies lead to correct reasoning and others tend to introduce errors.
• In contrast to humans, models occasionally adopt a symbolic strategy, where formal logical calculus is employed to solve the propositional logic problem at hand.
Through this work, we hope to advance the understanding of reasoning in LLMs.

heading: 2 Strategies in Propositional Reasoning
text: Propositional logic studies the relationships among statements (or propositions) and the methods for constructing logical arguments based on them (Hurley, 2011). At the core of propositional logic are simple statements that can be combined through the use of logical connectives such as "not", "and", "or", and "if... then...", thereby forming more complex compound statements. Conclusions are logically deduced, where the truth value of the propositions necessitates the truth of the conclusion. This form of logical reasoning allows us to construct sound arguments that are invariably true, given the truth value of the information provided. As such, propositional logic is fundamental to various disciplines, including science, mathematics, and philosophy, where it offers a structured approach to reasoning and argumentation.
To gain insights into the inferential processes humans employ in propositional reasoning, Van der Henst et al. (2002) conducted a series of experiments that study the behavior of participants during propositional reasoning. They formulated straightforward propositional logic problems with neutral content (the presence or absence of colored marbles in a box, similar to the problem illustrated in Figure 1) and requested participants to articulate their thought processes while engaging with these problems. Participants were permitted the use of paper and pencil for their workings. Both their verbal explanations and written responses were meticulously recorded, transcribed and analyzed thereafter. Van der Henst et al. (2002) discovered five strategies reasoners commonly utilize to navigate the problems, offering insights into their inferential mechanisms employed during propositional reasoning. In the following, we give a short description of each strategy (illustrated in Figure 2). For more details and additional examples, we refer to the original study by Van der Henst et al. (2002).

heading: Problem:

heading: Incremental Diagram:
text: Incremental Diagram. This strategy involves the creation of a comprehensive diagram that keeps track of all potential outcomes compatible with the premises of the problem. During the reasoning process, individuals progressively increment their diagrams to incorporate new information derived (see left box in Figure 2). The result is a single diagram that records a variety of possibilities compatible with the premises, often including even those that might be irrelevant to the task.1
Supposition Following. Reasoners employing this strategy start with a supposition, e.g. by assuming a marble of a certain color. Subsequently, they trace the implications of that supposition, logically following from the premises at hand, as illustrated in the upper second box from the left of Figure 2. The result is a sequence of literals (in this case, marbles of a certain color) without logical connectives.
1In contrast to Van der Henst et al. (2002), we in fact observe no single occurrence of the incremental diagram strategy in LLMs, despite the authors finding that this strategy is most frequently employed by humans. We believe that this discrepancy stems from the use of pen and paper in human assessments, implicitly encouraging diagrammatic reasoning. Exploring how this observation changes with vision-language models would be an intriguing area for future research.
The efficiency and success of supposition following strongly depends on the supposition made by the reasoner. While some suppositions lead to inferences that are relevant to the problem, others might lead to irrelevant conclusions.
Chain Construction. When employing this strategy, reasoners construct a chain of conditional statements derived either from the premises in the problem description or from intermediate deductions. An example of chain construction is displayed in the lower second box from the left of Figure 2. Premises are converted into a chain of conditional statements that are linked by their entities. A distinctive feature of this chain is the interconnection between conditionals, where the consequent of one conditional is the antecedent of the following.
Compound Strategy. Reasoners following the compound strategy combine two or more statements to derive a new compound conclusion. This process yields a series of novel conclusions, each building upon the preceding ones. An illustrative example of this strategy is given in the upper second box from the right of Figure 2. Based on the first two premises, the compound conclusion:
“If blue then not red.” is inferred, and then used
to draw another compound conclusion (“If blue then not red and not pink.”) together with the last premise of the problem statement.
Concatenation Strategy. This approach entails the concatenation of two or more statements into a single conclusion encompassing the logical implications of each combined proposition. This strategy is subtle and has only been infrequently observed by Van der Henst et al. (2002). An example of the strategy is illustrated in the lower second box from the right of Figure 2.
Symbolic Strategy. We could identify an additional strategy occasionally employed by LLMs, which has not been observed by Van der Henst et al. (2002) in human reasoners. This strategy, which we denote as symbolic strategy, is characterized by models employing formal logical calculus to solve the tasks at hand. When following this strategy, models either translate logical statements that are expressed in natural language (e.g. “If there is a white marble then there is not a red marble.”) into formal logic (W → ¬R), and then operate on those expressions, or create a truth table from which they aim to infer the validity of the conclusion. An illustration of this strategy is provided in the right box of Figure 2.

heading: 3 Experimental Setup
text: Task Overview. Our task setup aligns with the experiment conducted by Van der Henst et al. (2002) to allow for a fair comparison between the inferential strategies found in humans and those identified in LLMs.2 In particular, we evaluate each model on the 12 problems of propositional logic suggested by Van der Henst et al. (2002) (an overview of each problem can be found in Figure 5 in the appendix). For each problem, models are presented with a set of statements (or premises) and must determine whether a given conclusion logically follows (for an example, see Figure 1). Eight out of 12 problems involve three premises and a conclusion, while the remaining four problems consist of four premises leading to a conclusion. All premises, as well as the conclusions resemble either biconditionals, exclusive disjunctions or conditionals. Two problems (4 and 6) include a redundant first premise. All premises are stated such that two subsequent statements contain one proposition in common, except of two problems (11 and 12), which are arranged in
2More specifically, experiment one of Van der Henst et al. (2002).
a non-sequential manner. For half of the problems, the conclusions logically follow from the premises, whereas for the other half, they do not. To avoid the influence of external knowledge and ensure content neutrality, Van der Henst et al. (2002) framed the problems around the presence of colored marbles in a box, with colors assigned randomly to each entity within a problem.
Language Models. We aim to investigate various factors that might impact the inferential strategies displayed by LLMs. These factors include the type of model, its size, and the emphasis on alignment during training (Tunstall et al., 2023). Therefore, we assess a total of five models, consisting of three prominent open-access model types: Llama 2 (Touvron et al., 2023) with model sizes of 7B, 13B, and 70B, the recently released Mistral-7B model (Jiang et al., 2023), and Zephyr-7B (Tunstall et al., 2023), an extension of Mistral-7B with a focus on intent alignment through fine-tuning with AI Feedback (AIF). For our evaluations, we utilize the publicly accessible model weights from the HuggingFace platform, specifically Llama-2-chat-hf3(7B, 13B, and 70B), Mistral-7B-Instruct-v0.2,4 and zephyr-7b-beta.5 We consciously opt not to include proprietary models accessible via paid APIs, despite their reported superior performance in reasoning tasks (Team et al., 2023). This methodological choice reflects our commitment to promoting transparent and reproducible scientific research. Note that in this work, we refer to the above models when using abbreviations such as LLaMA-2, Mistral-7B-Instruct or Zephyr-7B-β.
Evaluation Setup. We prompt each model with a system message providing context about the task they are about to solve and the format in which they should answer (for the full prompt, see Figure 5 in the appendix). Analogous to Van der Henst et al. (2002), we inform the model of its participation in an experiment designed to explore reasoning processes, and instruct it to “think aloud” as it tackles the problem. In addition to the system message, we provide a user prompt that contains the problem description. In cases where the model does not accept system messages (such as Mistral-7B-Instruct-v0.2), we prepend the content of the system message to the user prompt. To prevent biasing the model towards a certain strat-
3https://huggingface.co/meta-llama 4https://huggingface.co/mistralai/Mistral-7B-Instruct 5https://huggingface.co/HuggingFaceH4/zephyr-7b-beta
egy, we refrain from providing few-shot examples, as done also by Leidinger et al. (2023). Instead, we elicit reasoning through zero-shot chain-of-thought prompting (“Let’s think step by step”) (Kojima et al., 2022). Answers for each model are generated with nucleus sampling using Llama-2-chat-hf’s default values (top-p = 0.9, temperature T = 0.6), as we found this configuration to work well for all models. To account for the statistical nature of language models, we ask each model to solve the set of propositional problems across 5 random seeds, resulting in a total of 60 responses per model. Our code is publicly available at: https://github.com/mainlp/inferential-strategies.
We record all answers and manually evaluate them (a total of 300 responses) for strategies employed in their reasoning (see Figure 3 for an example). For each model response, we qualitatively evaluate for strategy and soundness. That is, we manually label the inferential strategies identified, and the logical validity of the model’s reasoning. In addition, we record whether the final answer is
correct. In cases of faulty reasoning, we categorize the type of error. This comprehensive manual evaluation of model responses is independently conducted by two hired students with expertise in manual data annotation. To gauge the quality of the annotations, we report an overall Cohen’s Kappa value of κ = 0.98. For details on the interannotator agreement of each label, we refer to Table 2 in the appendix. Further annotated examples can be found in Appendix C. Following the recommendations put forward by Leidinger et al. (2023), we make all input prompts, model responses and manual annotations publicly available at: huggingface.co/datasets/mainlp/inferential_strategies.

heading: 4 Results and Analysis
text: In this section, we present the results of our evaluation. We begin with a quantitative analysis of the inferential strategies employed by LLMs, as well as the logical validity of their reasoning. This is followed by a qualitative analysis providing a more in-depth examination of the models’ reasoning.

heading: 4.1 Quantitative Analysis
text: Table 1 provides an overview of the frequencies with which large language models employ inferential strategies when navigating the problems of propositional logic described in Section 3. Our evaluation reveals that all models display strategies akin to those observed by Van der Henst et al. (2002). In particular, we find that, similar to humans, models commonly employ supposition following, chain construction and the compound strategy. In addition, we observe that models occasionally utilize the symbolic strategy, employing techniques from logical calculus to solve the tasks (see Section 2). Note that, similar to humans, models might switch from one strategy to another during a single problem, demonstrating multiple strategies within their responses (see Figure 19 in the appendix for an example). Surprisingly, we observe that distinct model families favor different inferential strategies. For instance, Zephyr-7Bβ predominantly employs supposition following, while Mistral-7B-Instruct is equally inclined towards drawing compound conclusions. In contrast, models from the Llama 2 series tend to rely on supposition following and chain construction, with negligible use of the compound strategy. Our analysis further reveals a discrepancy between the correctness of the models’ final answers and the logical soundness of their reasoning. While all models achieve an answer accuracy that approximately coincides with chance in our experimental setup, an analysis of their reasoning validity reveals a different picture: LLaMA-2-70B outperforms the other models by reasoning correctly in about 31.7% of cases, while Zephyr-7B-β and Mistral-7B-Instruct produce sound reasoning in 25% of the problems. We note that all models perform rather poorly on the propositional tasks, with LLaMA-2-7B failing entirely to construct sound arguments. Human Reasoning. Van der Henst et al. (2002) compute the percentages with which human reasoners employ inferential strategies with respect to the total number of strategies observed in their experiment, and not with respect to the total number of problems considered. Thus, their reported values mainly reflect which strategies are favored more or less by the reasoner, but do not provide information about how frequently a strategy has been observed in the overall context. To make our findings comparable to the results of Van der Henst et al. (2002), we convert our results respectively (see values in parentheses in Table 1). We note that almost all models seem to favor supposition following to a higher degree than human reasoners, who employ this strategy in only about 21% of overall use. In contrast, humans seem to draw compound conclusions more readily, except for Mistral-7B-Instruct which shows a tendency more than twice as high. Overall, both LLMs and humans hardly employ the concatenation strategy. Interestingly, Van der Henst et al. (2002) report that all reasoners successfully solve the problems of propositional logic, though not always for the correct reasons. While the study does not provide data on the number of problems where humans reasoned correctly, the high success rate of human participants contrasts sharply with the performance of the models. Effect of Model Size. Our evaluation of the Llama 2 series across three different model sizes—7B, 13B, and 70B parameters—demonstrates that
model scale significantly influences the frequency with which strategies are employed by the model. In particular, we observe that with increasing model size, Llama 2 employs strategies more readily. Furthermore, larger models within the Llama 2 framework are observed to generate a greater number of sound reasoning traces. We interpret this trend as a result of the model’s improving proficiency in strategic reasoning as its scale increases.
Effect of Alignment. The alignment of a model’s response with human preferences is crucial to emulate human-like behavior (Ouyang et al., 2022). Zephyr-7B-β is an iteration of Mistral-7B that is fine-tuned with AI Feedback (AIF) for improved intent alignment (Tunstall et al., 2023). In comparison to the observations made by Van der Henst et al. (2002), where besides the incremental diagram strategy (34%), chain construction was employed most frequently by humans, Zephyr-7Bβ demonstrates a marked preference for supposition following and significantly less engagement in chain construction. Moreover, it is noteworthy that among the evaluated models, Zephyr-7B-β most frequently adopts the symbolic strategy, an approach not reported in human reasoners.
Sound Reasoning. As previously highlighted, the accuracy of a model’s final answer does not necessarily serve as a reliable indicator of its reasoning capability. In particular, we observe that models often arrive at correct answers, but through flawed reasoning processes (refer to Figure 10 in the appendix for an illustration). Interestingly, we also find instances where models provide incorrect final answers despite reasoning correctly (for an example, see Figure 16 in the appendix). Our analysis
reveals only a moderate positive correlation between the accuracy of the models’ final answers and the logical soundness of their reasoning, with a Pearson correlation coefficient r(298) = 0.45 and a statistically significant p-value of less than 0.0001(p = 1.6× 10−16). This observation aligns with findings from previous studies (Ye and Durrett, 2022; Creswell and Shanahan, 2022) and underscores the need for more nuanced evaluation procedures, particularly in multiple-choice settings, where models might select the correct answer by chance rather than through rigorous reasoning.
In Figure 4, we explore the relationship between the inferential strategies employed by the models and the validity of their reasoning. For each strategy, we quantify the proportion of instances where the models’ reasoning is sound, compared to the overall application of that strategy. Our analysis reveals variability in the effectiveness with which different models apply various strategies. For example, Mistral-7B-Instruct tends to reason correctly when using approaches such as the chain, compound, or symbolic strategy, yet frequently encounters reasoning errors with supposition following. On the other hand, LLaMA-2-70B exhibits proficiency in supposition following, but struggles with the symbolic strategy.

heading: 4.2 Qualitative Analysis
text: We supplement our quantitative analysis by a more detailed qualitative analysis of the models’ reasoning behavior. Figure 3 depicts LLaMA-2-70B’s response to problem 1 of the task set. The response illustrates a frequently observed behavior. Initially, models tend to analyze the problem’s propositions,
often by paraphrasing each premise and the conclusion to be evaluated. They then embark on a reasoning process, typically utilizing one of the previously mentioned strategies. In the example, LLaMA-2-70B employs chain construction, creating a logical chain of conditionals that leads from the antecedent of the final conclusion to its consequent, thereby correctly affirming the conclusion’s logical validity. A notable pitfall in such reasoning chains is the models’ occasional misinterpretation of logical negations, leading to erroneous chains like: A → ¬ B; B → C; therefore A → C, where the negation in the first conditional is overlooked (for an illustrative case, refer to Figure 11 in the appendix). This behavior can be found across all models and aligns with previous work reporting difficulties of LLMs in understanding logical negations (Truong et al., 2023).
When employing supposition following, models often fail to consider all implications of their assumptions. Instead, they tend to focus only on immediate inferences, while overlooking further consequences crucial for assessing the conclusion’s validity. This leads to models prematurely concluding the inability to definitively determine the logical validity of the final conclusion: “Based on our analysis, we cannot definitively say that the conclusion logically follows from the given statements” (see Figure 7 in the appendix for a respective example). Another source of error in supposition following involves models making improper suppositions, such as conjecturing about a marble not mentioned in the final conclusion, and deriving disjointed intermediate conclusions that do not aid in solving the problem. An example of this behavior can be found in Figure 8 in the appendix.
Finally, we identify two behaviors in models that mirror logical errors seen in human reasoners (Van der Henst et al., 2002). First, models frequently attempt to prove an exclusive disjunction (A ⊕ B) by only considering a single conditional case (A → ¬ B), and second, they sometimes engage in the logical fallacy known as denial of the antecedent: A → B; therefore ¬ A → ¬ B (for illustrative examples, see Figures 12 and 13 in the appendix, respectively).

heading: 5 Related Work

heading: Human Strategies in Deductive Reasoning. A
text: considerable amount of research, especially within psychology and cognitive science, has explored
how humans approach deductive reasoning tasks (Schaeken et al., 2000). A prominent focus of these studies is on heuristics, which are cognitive shortcuts that individuals employ to arrive at satisfactory conclusions in deductive reasoning despite potential flaws in the underlying logic (Kahneman et al., 1982; Evans, 1989; Gigerenzer and Todd, 1999; Davis, 2018). For instance, Woodworth and Sells (1935) demonstrate that individuals tend to accept conclusions in syllogistic reasoning as valid when they share logical quantifiers with the premises, regardless of their actual logical validity. Nonetheless, such reliance on heuristics can result in errors and falls short of the level of strategic reasoning necessary to develop sound and coherent arguments (Kahneman, 2012). Further research has delved into more sophisticated strategies utilized by individuals in deductive reasoning. Based on the mental model theory (Johnson-Laird, 1986), Bucciarelli and Johnson-Laird (1999) identify a variety of strategies commonly employed by individuals in syllogistic reasoning. Byrne and Handley (1997) study strategies of individuals in knight-and-knave puzzles, where the truthfulness of statements made by hypothetical characters have to derived. Their experiments reveal that humans engage in both forward and backward inferences to navigate through potential solutions.
Human Reasoning Behavior in LLMs. Recent research has started to explore the extent to which LLMs mirror human-like reasoning behaviors. Dasgupta et al. (2023) demonstrate content-effects akin to those observed in human reasoning, where the deductive process is influenced by the content of the problem statement. Eisape et al. (2023) find that LLMs, similar to humans, exhibit biases such as ordering effects in syllogistic reasoning tasks. Several other studies have delved into the prevalence of biases and heuristics within LLMs (Binz and Schulz, 2023; Talboy and Fuller, 2023; Shaki et al., 2023; Suri et al., 2024). However, to the best of our knowledge, we are the first who study the presence of more sophisticated human strategies in the context of LLM-based deductive reasoning.
Faithful Reasoning with LLMs. Large language models can be instructed to explain the reasoning process by which they derive their final conclusions (Wei et al., 2022; Kojima et al., 2022). However, several studies indicate that these self-explanations might not always be faithful, i.e. accurately represent the model’s underlying reasoning process
(Jacovi and Goldberg, 2020; Agarwal et al., 2024; Lyu et al., 2024). For instance, Turpin et al. (2023) demonstrate that LLMs such as GPT-3.5 (OpenAI, 2023) and Claude 1 (Anthropic, 2023) often fail to mention biasing features in their input that significantly influence their decisions. Instead, the models produce plausible yet misleading explanations that give a false account of the underlying decision process. Lanham et al. (2023) probe the faithfulness of explanations by evaluating how the final conclusions of LLMs change when rationales are truncated or errors are introduced. Their findings reveal that the extent to which models rely on their rationales varies strongly across models and tasks. Matton et al. (2024) propose a method to quantify the faithfulness of explanations based on high-level concepts in the models’ input that influence decision-making. By measuring the difference between the set of concepts that LLMs deem influential and the set that truly are, instances of unfaithfulness could be identified, including cases where LLMs overlook the impact of social biases in their decision-making processes. Another important consideration is whether the model’s final conclusion aligns with its preceding explanation. As highlighted in Section 4.1, a correct conclusion might not always be the product of a logically sound reasoning trace, particularly in multiple-choice setups. Conversely, a sound rationale may not always lead to a logically consistent answer. Related work by Ye and Durrett (2022) indicates that in questionanswering and natural language inference tasks, explanations generated by LLMs such as OPT (Zhang et al., 2022) and GPT-3 (Brown et al., 2020) often do not entail the models’ final conclusions. Further studies aim to enhance the models’ faithfulness, for instance by enforcing causality from proof generation to entailment prediction. This can be achieved by either restricting the model’s context (Sanyal et al., 2022; Creswell and Shanahan, 2022; Radhakrishnan et al., 2023), or by utilizing deterministic tools that are inherently faithful by design (Lyu et al., 2023).

heading: 6 Conclusion
text: In this paper, we examine the inferential strategies employed by LLMs in solving problems of propositional logic. Through a comprehensive evaluation of their reasoning behavior, we demonstrate that LLMs adopt strategies akin to those observed in human reasoners. Our quantitative analysis reveals
that the frequency with which a model adopts a specific strategy strongly depends on its type, size, and fine-tuning procedure. Moreover, our analysis suggests that the accuracy of a model’s final conclusions does not adequately capture its reasoning capabilities, underscoring the importance of a more sophisticated evaluation framework that includes the model’s reasoning paths. We also provide a qualitative analysis of typical reasoning behaviors among models, pinpointing prevalent errors such as difficulties in understanding negations or recognizing all implications of a supposition.

heading: 7 Limitations
text: While our work contributes to the understanding of reasoning processes in large language models by demonstrating that these models employ inferential strategies in propositional logic similar to humans, it encompasses several limitations that could be addressed in future work.
Task setup. Our study is constrained by a limited set of problems, designed within a fixed framework that revolves around hypothesis validation based on 3-4 statements of propositional logic. We employ a constant and neutral content, disregarding potential content-effects on the models’ reasoning behavior, as shown by Dasgupta et al. (2023). Similarly, we have not yet examined factors such as the complexity of the problems, the differences between hypothesis validation and generation, and the impact of logical connectives utilized in the premises. We believe that these factors are worth investigating and leave a detailed examination to future work.
Evaluation Framework. The extent of our manual evaluation is limited by both the number of samples reviewed and the quantity of annotators involved. Despite our efforts to maximize the use of available resources, these constraints may affect the scalability and reliability of our results. Additionally, we instruct all models through zero-shot chain-ofthought prompting (“Let’s think step by step”) (Kojima et al., 2022). Exploring alternative reasoning frameworks, such as Tree of Thoughts (Yao et al., 2023) or Graph of Thoughts (Besta et al., 2024), could provide valuable insights into their influence on model behavior and the inferential strategies adopted. Based on our annotated data, we endeavored to develop a classifier capable of automatically identifying the inferential strategies employed in the models’ output, which was intended to comple-
ment our manual evaluation setup. However, due to the complexity of the task and limited size of our annotated dataset, our classifier struggled with generalization to new, unseen responses. In future endeavors, we aim to allocate more resources towards expanding our manual annotation efforts and explore this direction further. Finally, our study predominantly offers a behavioral analysis and does not delve into the mechanistic aspects that might explain the diversity in strategy usage by the models. Investigating how model-internal mechanisms might influence their choice of reasoning strategy presents a compelling direction for future research.

heading: Acknowledgements
text: We would like to thank the MaiNLP lab members for their insightful feedback, specifically Diego Frassinelli, Rob van der Goot, Siyao Peng, Robert Litschko, Jian Lan, Daniela Teodorescu, Xinpeng Wang, Verena Blaschke, Elena Senger, Elisa Bassignana, and Max Müller-Eberstein. Furthermore, we would like to express our gratitude to Huangyan Shan for her valuable work and support in data annotation. Our appreciation extends to the anonymous reviewers for their comments and suggestions. Lastly, we acknowledge the support for BP through the ERC Consolidator Grant DIALECT 101043235.

heading: A Additional Experimental Details
text: In this section, we provide additional details about the experimental setup, including supplementary information about the problem formulations and prompts utilized.

heading: A.1 Task Prompts
text: Figure 5 displays the task prompt and problem formulations employed in assessing the language models described in Section 3. Note that the prompt
Supposition Following
Chain Construction Compound Conclusion Concatenation Strategy Symbolic Strategy
Correct Answer
Sound Reasoning
Zephyr-7B-β 1.0 0.94 1.0 1.0 1.0 1.0 1.0
Mistral-7B-Instruct 1.0 0.9 1.0 1.0 1.0 1.0 1.0
LLaMA-2-7B 0.89 0.95 1.0 0.79 1.0 1.0 1.0
LLaMA-2-13B 0.88 1.0 0.85 1.0 1.0 1.0 1.0
LLaMA-2-70B 0.97 1.0 1.0 1.0 1.0 1.0 1.0

heading: A.2 Annotator Instructions
text: Our assessment of model responses involves a comprehensive independent review by two students who are specialized in the field of natural language processing and have expertise in manual data annotation. To ensure a high quality of annotations, we offer comprehensive training to both annotators. This training includes detailed explanations and extensive examples of the strategies identified by Van der Henst et al. (2002), complemented by a session dedicated to clarifying any questions that may emerge. Subsequently, the annotators are tasked with independently annotating practice examples, which serves to highlight and address any ambiguities in the annotation process. Only when both annotators are confident in their understanding of each strategy do we proceed. We instruct both annotators to independently go through each model response and mark parts where they identify a certain strategy to be employed. Each strategy is marked in a unique color code, which is afterwards converted into labels that signify the use of a particular strategy. In addition, we instruct both annotators to label whether the reasoning is sound, and the final conclusion of the model is correct. Furthermore, we ask them to classify any logical errors identi-

heading: A.4 Model Details
text: We report further details about the models used in this study in Table 3. In particular, we provide information about the number of parameters, context length and fine-tuning procedure for each model.

heading: B Additional Quantitative Results
text: In this segment, we present supplementary findings from our quantitative evaluation. Table 4 illustrates the frequencies with which the different language models employ inferential strategies when navigating the problems of propositional logic, as outlined in Section 3. Values denote percentages averaged across five distinct random seeds, accompanied by their standard deviation. Furthermore, we detail the proportions of correct final conclusions and sound reasoning. Note that all percentages are calculated relative to the overall count of tasks within the experimental framework.

heading: C Annotated Model Responses
text: Within this section, we showcase examples of model responses that exemplify each inferential
strategy identified in our study, as depicted in figures 6-19. Each figure is organized with the problem statement at the top, the model’s response on the lower left, and the annotators’ comments to the lower right. For an extensive array of model responses and annotations, we invite readers to explore our data repository at: huggingface.co/datasets/mainlp/inferential_strategies.

heading: C.1 Supposition Following
text: Figures from 6 to 8 demonstrate the application of supposition following by various models. For instance, Figure 6 presents LLaMA-2-70B’s approach to problem 7, where the model supposes the absence of a blue marble in the box and logically infers the implications of this assumption to reach the valid conclusion. On the other hand, Figure 7 depicts Mistral-7B-Instruct’s response to the same problem, where the model considers various combinations of marbles in the box, drawing immediate conclusions that follow from the premises at hand. However, it does not explore deeper ramifications of these suppositions, thereby failing to deduce the validity of the conclusion. This showcases a common behavior we observe in models that employ supposition following unsuccessfully. In Figure 8 the model approaches problem 9 by assuming the presence of an olive marble in the box, yet inferring disjointed intermediate conclusions that do not aid in solving the problem, thus failing to prove the logical validity of the problem.

heading: C.2 Chain Construction
text: Figures 9 to 13 illustrate instances where models employ chain construction to navigate the problems of propositional logic. In Figure 9, LLaMA-
2-70B adeptly forms a chain of conditional statements that bridge the antecedent of the conclusion to its consequent, effectively confirming the conclusion’s logical validity. Conversely, Figure 10 depicts a logical chain in which LLaMA-2-70B erroneously concludes the nonexistence of a white marble based on the absence of a red marble, despite an exclusive disjunction linking the two. Despite this logical misstep, the model’s final conclusion remains accurate, highlighting the discrepancy between the model’s final answer and the soundness of its reasoning. In Figure 11, LLaMA-2-13B constructs a chain correctly linking the antecedent of the final conclusion to its consequent. Nonetheless, it overlooks the negation present in one of the conditionals, resulting in a compromised reasoning chain. Figure 12 presents a scenario where the model incorrectly attempts to validate an exclusive disjunction solely through a singular conditional sequence, a reasoning error not uncommon among human reasoners (Van der Henst et al., 2002). Lastly, Figure 13 highlights LLaMA-2-70B’s engagement in the inverse fallacy, inferring ¬ W → ¬ G from the conditional W → G, mirroring a logical misjudgment frequently observed in human reasoning processes.

heading: C.3 Compound Strategy
text: The compound strategy is illustrated in Figures 14 to 16. Figure 14 presents Mistral-7B-Instruct’s approach to problem 9, where it infers a biconditional relationship between the purple and olive marble from the first two premises. On the other hand, Figure 15 shows LLaMA-2-70B’s response to the same problem, formulating a sequence of compound inferences beyond the initial bicondi-
tional deduction, culminating in the correct final answer. Additionally, Figure 16 illustrates Mistral7B-Instruct’s approach to problem 8, in which the model initially generates compound conclusions derived from the problem statements, followed by supposition following to explore the implications that the absence of an olive marble might have. However, despite the model’s sound reasoning, its final answer is incorrect.

heading: C.4 Concatenation Strategy
text: Figure 17 demonstrates the concatenation strategy, where Mistral-7B-Instruct concatenates two intermediate deductions to form a single statement. It then uses the concatenated statement to infer the invalidity of the conclusion.

heading: C.5 Symbolic Strategy
text: The symbolic strategy is exemplified in Figure 18, where LLaMA-2-70B employs a truth table to assess the conclusion’s validity, albeit with errors leading to an incorrect result. Conversely, Figure 19 shows Mistral-7B-Instruct’s application of chain construction followed by the symbolic strategy. The model makes false inferences while employing chain construction, and further errs in its validation through logical calculus.

heading: Future Work
text: While our work contributes to the understanding of reasoning processes in large language models by demonstrating that these models employ inferential strategies in propositional logic similar to humans, it encompasses several limitations that could be addressed in future work. Task setup. Our study is constrained by a limited set of problems, designed within a fixed framework that revolves around hypothesis validation based on 3-4 statements of propositional logic. We employ a constant and neutral content, disregarding potential content-effects on the models’ reasoning behavior, as shown by Dasgupta et al. (2023). Similarly, we have not yet examined factors such as the complexity of the problems, the differences between hypothesis validation and generation, and the impact of logical connectives utilized in the premises. We believe that these factors are worth investigating and leave a detailed examination to future work. Evaluation Framework. The extent of our manual evaluation is limited by both the number of samples reviewed and the quantity of annotators involved. Despite our efforts to maximize the use of available resources, these constraints may affect the scalability and reliability of our results. Additionally, we instruct all models through zero-shot chain-ofthought prompting (“Let’s think step by step”) (Kojima et al. , 2022). Exploring alternative reasoning frameworks, such as Tree of Thoughts (Yao et al. , 2023) or Graph of Thoughts (Besta et al. , 2024), could provide valuable insights into their influence on model behavior and the inferential strategies adopted. Based on our annotated data, we endeavored to develop a classifier capable of automatically identifying the inferential strategies employed in the models’ output, which was intended to comple-
ment our manual evaluation setup. However, due to the complexity of the task and limited size of our annotated dataset, our classifier struggled with generalization to new, unseen responses. In future endeavors, we aim to allocate more resources towards expanding our manual annotation efforts and explore this direction further. Finally, our study predominantly offers a behavioral analysis and does not delve into the mechanistic aspects that might explain the diversity in strategy usage by the models. Investigating how model-internal mechanisms might influence their choice of reasoning strategy presents a compelling direction for future research