RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Authors: Jason Chan
Robert J. Gaizauskas
Zhixue Zhao
Venue: ICML 2025 poster (2025)
Keywords: reasoning
logic
human-like reasoning
cognitive science
large language models
Primary area: deep_learning->large_language_models

Abstract: Formal logic enables computers to reason in natural language by representing sentences in symbolic forms and applying rules to derive conclusions. However, in what our study characterizes as "rulebreaker" scenarios, this method can lead to conclusions that are typically not inferred or accepted by humans given their common sense and factual knowledge. Inspired by works in cognitive science, we create RULEBREAKERS, the first dataset for rigorously evaluating the ability of large language models (LLMs) to recognize and respond to rulebreakers (versus non-rulebreakers) in a knowledge-informed and human-like manner. Evaluating seven LLMs, we find that most models achieve mediocre accuracy on RULEBREAKERS and exhibit some tendency to over-rigidly apply logical rules, unlike what is expected from typical human reasoners. Further analysis suggests that this apparent failure is potentially associated with the models' poor utilization of their world knowledge and their attention distribution patterns. Whilst revealing a limitation of current LLMs, our study also provides a timely counterbalance to a growing body of recent works that propose methods relying on formal logic to improve LLMs' general reasoning capabilities, highlighting their risk of further increasing divergence between LLMs and human-like reasoning.

RULEBREAKERS:
Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Jason Chan Robert Gaizauskas'
Zhixue Zhao
!
Abstract
A or B.
Not A.
Formal logic enables computers to reason in nat- ural language by representing sentences in sym- bolic forms and applying rules to derive conclu- sions. However, in what our study characterizes as "rulebreaker" scenarios, this method can lead to conclusions that are typically not inferred ac- cepted by humans given their common sense and factual knowledge. Inspired by works in cogni- tive science, we create RU BULEBREAKERS, the first dataset for rigorously evaluating the ability of large language models (LLMs) to recognize and respond to rulebreakers (versus non-rulebreakers in mhowledge-inforeet and human-like manner. Evaluating seven LLMs, we find that most models achieve mediocre accuracy on RULEBREAK ERS and exhibit some tendency to over-rigidly apply logical rules, unlike what is expected from typical human reasoners. Further analysis sug- gests that this apparent failure is potentially asso- ciated with the models' poor utilization of their world knowledge and their attention distribution patterns. Whilst revealing a limitation of current LLMs, our study also provides timely counter- balance to a growing body of recent works that propose methods relying on formal logic to im- prove LLMs' general reasoning capabilities, high- lighting their risk of further increasing divergence between L.MSs and human-like reasoning.
Not B.
Anne is either in Stockholm or somewhere in Sweden.
Anne is in Stockholm.
?
Anne is not in Sweden.
Figure I. An example adapted from Johnson-Laird (1983); rigidly applying logical rules (top) to premises (left) in natural language can result in conclusions (right) that humans would not typically draw accept, given their common sense and factual knowledge.
symbolic forms (in process known as semantic parsing) and then applying a set of pre-defined rules to evaluate or derive new conclusions (Kamath & Das, 2019; Saparov & Mitchell, 2022; Olausson et al., 2023, etc.). For example, in system using propositional logic Plummerr et al., 2011), sentences are converted into atomic propositions ("A". "B") optionally connected by
more of the five connectives ("not" (-), "and" (A), "or" (V), "if. then"(())). "...if and only if..." (++)). Given two premises, "Anne is ei- ther in Stockholm or somewhere in Germany" and "Anne is not in Germany", we can convert them symbolically into V B and -B respectively. We can then apply the disjunctive syllogism rule (for all propositions P and Q, P V Q. -Q P) to correctly conclude that A i.e. "Anne is in Stockholm",
However, as argued in some cognitive science literature (Johnson-Laird, 2010, Ragni & Johnson-Laird, 2018, Khem- lani & Johnson-Laird, 2019, etc.), this method of reasoning purely based on logical form and rules fundamentally different from how humans typically reason with natural lan- guage statements. A key problem is that this approach can lead to conclusions which factually contradict the premises and which humans typically do not draw provided that they recognize this factual contradiction. Drawing on exam- ples adapted from Johnson- suppose we are told that "Anne is either in Stockholm or somewhere Sweden" (A B) but we find out in fact that
1. Introduction
Formal logic has numerous applications in mathematics and computer science (Zach, 2024). In natural language process- ing (NLP), it enables computers to perform reasoning with natural language sentences by first converting them into
University of Sheffield, Sheffield, UK. Correspondence to: Zhixue Zhao < whw.o@@shhffficd........
(1983) such as Figure 1,
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
"Anne is not in Sweden" (-B). Simply applying the disjunc- rive syllogism rule likewise would produce conclusion that "A "Anne is in Stockholm" (A). But most readers, provided that they know Stockholm is located in Sweden, are unlikely to draw this conclusion. This is because, given our common sense and factual knowledge about the world, we recognize that the two premises are inconsistent with each other; and that the proposed conclusion ("Anne is in Stockholm") con- tradicts the second premise ("Anne is not in Sweden") in fact and should therefore be rejected. Similarly, suppose we are told that "if" Anne is in Sweden, then she is not in Stockholm" (A + -B) but we learn in fact that "Anne is in Stockholm" (B). We would clearly also not just conclude that "Anne is not in Sweden" (-A) by applying the formal rule of modus tollens (for all propositions P and Q. + Q. -Q -P)).
mains unclear how they do so and whether their reasoning processes are in any way comparable to those of humans. Our study contributes targeted insights to these questions by drawing on cognitive science expertise and assessing LLMs on an under-explored category of reasoning problems.
Moreover, a growing number of recent studies have pro- posed methods that rely on formal logic to improve LLMs' reasoning capabilities and enable them perform complex multi- reasoning. Examples include using logical rules to synthetically generate or augment training data (Wang et al., 2022; Morishita et al., 2024), incorporating logic- based training metrics (Calanzone et al., 2024) or constraints during inference (Weir et al., 2024), and using LLMs to explicitly translate natural language statements into sym- bolic forms which are then used for the models' subsequent reasoning steps (Xu et al., 2024) or processed by external solvers (Pan et al., 2023; Olausson et al., 2023). Our study provides a timely and important counterbalance to these approaches by drawing attention to their potential pitfalls and trade-off in increasing divergence between LLMs and human- like reasoning.
Our study characterizes each of these examples "rule- breaker": a set of natural language premises and conclusion whereby the conclusion can be derived by applying logical rules to the premises, but is typically not inferred or ac- cepted by humans because, given their common sense and factual knowledge, it is inconsistent with the premises.
In cognitive science, existing studies (Johnson-Laird & Byrne, 2002; Quelhas al., 2010; Quelhas & Johnson- Laird, 2017) have found that, when human participants are given the premises of rulebreakers similar to the examples above, they indeed typically do not draw what they recog- nize to be factually contradicting conclusions by rigidly ap- plying a logical rule. By contrast, most participants do draw the conclusion in what we call mnlebrrllbreakee" cases (i.e. when the conclusion is not inconsistent with the premises, given common sense and factual knowledge), even though both the rulebreakers and non-rulebreakers have the same surface structure (e.g. "A or B. Not B. Therefore, A."). In NLP, rulebreakers have received little attention, having only been acknowledged in theory (Bos& & Markert, 2005; Kor- man et al., 2018) but not systematically studied.
Our contributions are summarized as follows:
We develop RULEBREAKERS, dataset comprising 25,600 instances, along with corresponding evaluation metrics, designed to rigorously assess the reasoning ca- pabilities of LLMSs on rulebreakers, an under-explored category of reasoning problems in NLP.
We are the first to conduct a comprehensive comparison across seven state-of-the-art LLMs, including GPT-4o, on rulebreakers at scale and find that, while LLMs show some capacity to distinguish rulebreakers from non-rulebreakers when reasoning, their overall perfor- mance and responses are far from what we expect from typical human reasoners.
Our subsequent analysis indicates that the models' fail- ure to recognize rulebreakers may be associated with two factors: the models' familiarity with the relevant factual knowledge, and the relative importance they assign to different parts of the premises, opening up promising directions for future investigation.
Our study addresses this gap by creating RULEBREAK- ERS, large-scale dataset consisting of minimally differ- ing rulebreaker and non-rulebreaker pairs, to rigorously evaluate the reasoning behavior of large language models (LLMs). Specifically, we evaluate the extent to which LLMs can recognize and respond to both rulebreakers and non- rulebreakers in way that we would expect typical human reasoners to do so, by rejecting conclusions in rulebreakers but accepting those in non-rulebreakers."
2. Related Work
Problems with logic as basis of human reasoning, Exist- ing work in cognitive science, particularly by proponents of the mental model theory*, have argued against the idea that humans typically reason by identifying the logical form of
Recent studies have suggested that LLMs can perform tasks that require reasoning abilities (see Section 2). but it re-
'B is logically equivalent to -(-B) in propositional logic "Our study refers to this specific reasoning behavior as being "human-like" and "knowledge-infonee infoomeed""" interchangeably, on the basis of existing studies in cognitive science as discussed above and further in Appendix A Table 5.
We publicly release RULEBREAKERS at https:// / g thub. com/ jttp//jsonccaiiin rul ebreakers. "We refer interested readers to Johnson-Laird (1983) for the seminal text on this theory of how humans reason by constructing
2

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
natural language premises and then applying logical rules to produce a conclusion (Johnson-Laird, 2010; Khemlani & Johnson-Laird, 2017, etc.). A key reason is that logi- cal rules are defined based on connectives (such as "if", "and", "orr") that are assumed to have fixed meanings and properties in formal logic; but in everyday language, the interpretation of connectives varies significantly depending on the semantic content of sentences they connect (Johnson- Laird & Byrne, 2002; Khemlani et al., 2018).² Our study is inspired by experimental work that has provided empir- ical support to these arguments (Johnson-Laird & Byrne, 2002; Quelhas et al., 2010; Quelhas & Johnson-Laird, 2017), which find that human participants generally avoid drawing conclusions that they recognize to be factually contradict- ing the premises, even where these conclusions might be licenced by a purely formal application of logical rules to the premises." However, while these studies tested hu- man participants on dozens of handcrafted rulebreakers, our study designs rulebreaker templates by which we can sys- tematically generate minimally differing rulebreaker and non-rulebreaker pairs at scale, enabling us to rigorously as- sess LLMs and control for their world knowledge, bias and sensitivity to prompt variations.
valid purely on the basis of form. In other words, they as- sume that, given premises in the form of "if A, then 8 and "not B", the conclusion "not A" is always correct regardless of what "A" and "B" symbolize in natural language. Instead, we assess whether L.MMs can, like humans (as we argued in Section 1), take semantic content of premises into account when reasoning rather than just rigidly applying formal rules. This means using common sense and factual knowl- edge to recognize that conclusions in non-rulebreakers are correct (i.e. they are factually consistent with and follow from the premises) whereas those in rulebreakers are not, even though they might appear to have the same logical form such that the same rule seems to apply.
Comparing human and LLM reasoning. Our study con- tributes to the growing interest evidenced by similar re- cent works in comparing LLMs' reasoning behavior against how humans typically reason according to theories and find- ings in cognitive science (Suri et al., 2024; Binz & Schulz, 2023; Castello et al., 2024; Mondorf & Plank, 2024a, etc.). Lampinen et al. (2024) found that LLMs exhibit some human-like "content effect" in certain logical reasoning tasks, including the Wason selection task (Wason, 1968) and assessing whether conclusions are valid in syllogisms (arguments in the form of c.g. "All As are Bs All Bs are Cs. Therefore, all As are Cs.") In other words, models can be mistakenly affected by semantic content when rea- soning about problems where only the logical form should be considered: they can be prone to judge conclusions that are true (or plausible) in the real world as logically valid, and conclusions that are false in the real world as invalid, even where the underlying logical structure is the same in both cases. This phenomenon is corroborated by subsequent studies such as ProntoQA (Saparov & He, 2023) and Wu et al. (2024), the latter finding that models tend to perform worse at logical reasoning the more the premises deviate from real-world knowledge. Similarly, Eisape et al. (2024) compares human and LLM behavior in reasoning about syl- logisms and found that models mimie some human biases including content effect and figural effect (i.e. judgments being sensitive to the ordering of the premises and the terms within each premise), although larger models exhibit less of these biases and reason in a more logical manner.
Reasoning with LLMs. Reasoning has been characterized as an emergent ability of LLMs (Wei et al., 2022a; Suzgun et al., 2023) and a range of existing work has evaluated the ability of LLMs to perform different types of reasoning, such as logical reasoning (see below), commonsense reason- ing (Talmor et al., 2021; Bian et al., 2024) and mathematical reasoning (Meadows et al., 2024; Ahn et al., 2024). Yu et al. (2024), Qiao et al. (2023). Sun et al. (2025), and Mondorf & Plank (2024b) provide further overviews and taxonomies.
Our study is motivated in part by existing work which in- dicates that LLMs can generally perform logical reasoning using rules in propositional and first-order logic to some de- gree (Saparov et al., 2023; Han et al.. 2024; Xu al., 2025: Parmar et al., 2024), even though they may struggle with applying particular rules (Parmar et al., 2024), identifying logical fallacies (Wan et al., 2024) and constructing long inference chains (Saparov et al., 2023).
However, our study and dataset has a fundamentally dif- ferent objective from these prior works, which all assess whether LLMs can identify the underlying logical form of natural language premises, regardless of their semantic con- tent, and apply rules to derive conclusions that are logically
Our work differs from the above in that our focus is not on formal logical reasoning problems that humans might find difficult to answer correctly due to content effect or systematic biases. Rather, the rulebreakers we study are more general reasoning problems in natural language that do in fact require models to consider the semantic content of the sentences in order to answer correctly in a knowledge-informed and human-like manner.
mental models of possibilities, but note that the specific mechanics proposed by the theory itself are not integral to our current work.
"n cognitive science, this phenomenon of context-dependee interpretation is also known as the "modulation" of sentential connectives (Johnson-Laird & Byrne, 2002).
6 "Appendix A Table provides further details on these studies. In this sense, rulebreaker cases can also be considered as factual contradictions arising from over-rigid reasoning with formal logic.
We discuss additional related works in Appendix A.
3

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Table Example rulebreakers in the four templates used in our dataset, created by permuting premise-conclusion structures of logical rules: MT and DS, and two kinds of entity pairs: geographical (country, city) and categorical (type, instance). Note that we create MT rulebreakers a structure that different but logically equivalent to the paradigm form Pi then Q. Not Q. Therefore, not P.
Geographical (country, city)
Categorical (Lype, instance)
Modus tollens (MT)
Premises: If Anne is in Sweden, then she is not in Stockholm. Anne is in Stockholm.
Premises: If Anne plays some kind of brass instrument, then she does not play the trumpet. Anne plays the trumpet.
Premises: If then not Q. Q.
Conclusion: Not P.
Conclusion: Anne is not in Sweden, Premises: Anne is in either Stockholm
Conclusion: Anne does not play any kind of brass instrument. Premises: Anne plays either the trumpet or some kind hrass instrument.
Disjunctive syllogism (DS)
or somewhere in Sweden.
Premises: P or Q.
Anne is not in Sweden.
Anne does not play any kind of brass instrument.
Not Q.
Conclusion: P.
Conclusion: Anne is in Stockholm.
Conclusion: Anne plays the trumpet.
3. RULEBREAKERS Dataset
arts (8), and racket sports (5). In total, we have 91 (type, instance) pairs, listed in Appendix B.
We create dataset consisting of non-rulebreakers and rule- breakers in equal proportion. We generate rulebreakers using four templates as shown in Table 1. These templates are created in relation to two logical rules (modus tollens (MT) and disjunctive syllogism (DS)), with placeholders to be filled with entity pairs, verbs and names-pronouns.
To fill in the template for creating RULEBREAK- ERS instances, for country-city pairs, we permute each pair with six manually selected verbs ("is in", "was born in"" "died in", "will be visiting", "had studied in"", "has been to"). Similarly, we permute each type-instance pair with two manually selected verbs. Specifically, birds, insects, and fish are permuted with "saw"/"caughr", the three kinds of musical instruments with "plays"/" and martial arts and racket sports with "is good a competing in".
Verbs
3.1. Creating Rulebreakers
Logical rules RULEBREAKERS focuses on premise- conclusion structures relating to two rules in propositional logic that have been tested by prior work on human partici- pants (Johnson-Laird & Byrne, 2002; Quelhas et al., 2010; Quelhas & Johnson-Laird, 2017). See Table 1 for examples of rulebreakers with respect to these two rules.?
Names-pronouut We further permute each of these entity pair + verb combinations with five given names and corre- sponding pronouns, randomly sampled without replacement from a list of common first names worldwide".
MT: for all propositions P and Q "if P then Q" "not Q" entails "not p" (P + Q -Q.
3.2. Creating Non-rulebreakers
DS: for all propositions and Q. "P or Q" and "not Q"
We create non-rulebreaker counterpart for each rulebreaker by randomly replacing the country or type used in the premises and conclusion with a different country or type, respectively. For example, we create the following non- nulebreaker by replacing "France" with "Germany":
entails "p" (PV Q. P).
Entity pairs
We include two groups of entity pairs:
Country and city (geographical pairs): We use a list of current-day countries and their capital cities, obtained by querying WikiData (Vrandečić & Krötzsch, 2014). We filter out capitals that have the same name as their countries (e.g. Singapore) are located in more than one country, resulting in 183 (country, city) entity pairs.
Premises: If Anne is in FraneeGermany, then she is not in Paris. Anne is in Paris.
Conclusion: Anne is not in FranceGermany
In this case, the conclusion, which can be derived by apply- ing MT, follows from and is consistent with the premises, given common sense and factual knowledge about the world.
Type and instance (categorical pairs): We extract types of entities from ConceptNet (Speer et al., 2017) and manually filter to ensure correctness: birds (20), fish (13), insects (12), brass instruments (9), stringed instruments (16), woodwind instruments (8), martial
3.3. Dataset Summary
In total, RULEBREAKERS consists of 12,800 rulebreak- ers, created according to Section 3.1, with the breakdown
Appendii A Table 5 shows how the templates used in creating KUL.BREEKER dataset correspond specific examples used these prior experimental work.
8 /ntspa://github.. com/ www.ed/poppla.... names -ymms-----oontty--atteett

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
shown in Figure 2. We then create 12,800 non-rulebreakers per Section 3.2, bringing the combined number of rule- breakers and non-rulebreakers in RULEBREAKERS to 25,600.²
positive over negative answers, it would yield a score of O. Accordingly, the baseline for random guessing is 0.25.
We also measure the rulebreaker-specife and non- rulebreaker accuracies (Tp# and TDx). These are defined as the proportion of correctly answered rulebreaker and non-rulebreaker instances, respectively. Formally:
2 logical forms
183 (country, city) pairs
5 names
10,980 rulebreakers
verbs
logical forms
91 (type, instance) pairs
x
verbs
2
x
5 names
1,820 rulebreakers
TDa
D
1
Σ
[T
(2)
RE
12,800 rulebreakers
TDN
DN
1
Σ
[T(1]
(3)
Figure 2. Statistical summary of rulehreakers in our dataset.
ZEDN
Model confidence Inspired by Lampinen et al. (2024), to gain a more fine-grained understanding of a model's behav- ior, we evaluate its "confidence" in responding to both rule- breakers and non-rulebreakers. Consider scenario where model consistently outputs "yes" both non-rulebreaker and rulebreaker prompts but its confidence levels in its "yes" answers to non-rulebreakers are significantly higher com- pared to rulebreakers. In this case, despite achieving a paired accuracy of O, the discrepancy in model confidence may still indicate some underdeveloped latent ability to differentiate between rulebreakers and non-rulebreakers.
3.4. Evaluation
We measure model's ss ability to distinguish and respond to rulebreakers and non-rulebreakers in human-like manner in terms of (a) accuracy; and (b) model confidence.
In formal notation, let D represent the set of prompts used in our experiment. Specifically, D comprises all instances in RULEBREAKERS each permuted across 10 distinct phrasing variations, as detailed in Section 4.2. We define D and & as the sets of rulebreakers and non-rulebreakers respectively within D, and Deaired the subset of D x D such that for all (x >> E Dpaired, x is the non- rulebreaker counterpart to x as described in 3.2.
For each prompt x in D, compute the model's confi- dence in either the word "yes" or "true" (depending on the answer options specified in as Section 4.2 will explain) by extracting the output probabilities of tokens corresponding to that word (e.g. "Yes"""YES"""ys"""" We refer to this value as the model's confidence in a positive answer to a given prompt x: p* (z). Formally, where the answer options specified in x are "yes""/no""
Accuracy We first define paired accuracy as the propor- tion of all rulebreaker and non-rulebreaker prompt pairs (x: x) in Diaired where both parts of the pair are cor- rectly answered by a model. Pairs are defined such that the rulebreaker and non-rulebreaker counterparts differ only by country or type name. 10 Thus, a pair is answered correctly if and only if the model outputs "no" for the rulebreaker (i.e. rejecting the conclusion) and "yes" for the non-rulebreaker prompt (accepting the conclusion), or "false" and "true" respectively where we specify those as answer options per Section 4.2. Formally, paired accuracy (T) is defined as:
p² pxx) =
Σ
p())>)
(4)
SET "Ys", "yes", "YES")
where p(olx) refers to the output probability the model assigns to the token y given a prompt x. Likewise, where the specified answer options are "true"/"fasee" p= (x) is defined by the summed probabilities of "true"/"true"""" TRUEE".
Σ (R DED
[T(")) (1)
Dravred
We aim to evaluate whether the model, on average, ex- hibits higher confidence when outputting ""rue"""true for non-rulebreakers (a correct response) compared to when it outputs "yes"/"ttue" for rulebreakers (an incorrect response). To do this, we partition D) into Dves and D which rep- resent the subsets of prompts in D where the model gives positive ("yes"""truee") and negative respectively. We then compute and compare 11 and 1
where Txx) is true if and only if the model outputs the cor- rect answer to the prompt x. This paired accuracy metric is designed to reflect the core objective of our evaluation and control for any inherent biases in model responses. Specifi- cally, if a model completely fails to distinguish between rule- breakers and nolbbbebbre,,,s, e.g., if outputs "yes"" for every instance in D because of strong inherent bias for
answers,
T ExE(DMMDVe)) P (x)
AAppendix C lists the licenses of the various underlying datasets used in creating RULEBREAKERS as described above. For example, "If Anne is in France, then she is not in Paris..." (rulchreaker) and "If Anne in Germany, then she not Paris..." (non-rulebreaker) would constitute a pair.
(5)
DDN n Dues
' is account for the issue of surface form competition highlighted in Holtzman et al. (2021).
5

---

RULEBREAKERS, Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Dn DEE(DRNDOC)) p* (x) |DR 0 D
4.2. Prompting Setup
(6)
To account for LLMs' sensitivity to prompt phrasing (Zhao et al., 2021, Lu et al., 2022, Li et al., 2024), we present each instance in RULEBREAKERS in 10 different variations, by permuting five basic phrasings:
Ideally, models should have a DN as close to I, and DR as close to 0 as possible. We test for significance using Welch's 's 1-test (Welch, 1947), given that the sample sizes are likely to be unequal: model might output "yes"""true" to more non-rulebreakers than to rulebreakers, i.e. |DN Duee > | n Duee), or vice versa.
1. Does the Conclusion follow from the Premises?
2. Do the Premises entail the Conclusion?
3. Can the Conclusion be inferred from the Premises?
4. Can the Conclusion be deduced from the Premises?
5. Do the Premises support the Conclusion?
4. Experiment
with two answer-option pairs to follow the main ques- tion ("Answer Yes or No only"" or "Answer True or False only." Section 5 reports the averaged results across these 10 phrasing variations, with a breakdown shown in Ap- pendix F.
4.1. Models
We limit our model selection to those that are (a) medium- sized, due to computational constraints; (b) instruction-fine tuned, so that their outputs are more likely to follow instruc- tions in our prompts; and (c) apeen-surred, to ensure access to their internal weights and hidden states for further analy- sis. Based on these criteria, we select six models from the top of the Open LLM Leaderboard on Hugging Face at the time of writing 13 Microsoft- Phi-oftt-------iiii 1ewiiii----k---- Instruutt and Microsoft--hii-3-oe 1et-----mmiium--22----- 22k--Instrrttt (Abdin et al., 2024), Meta-Llamaa3-883 Baa-LLlama-3-8-IInsttee Instruct and Meeaa-Llamaa--70--Innnte (Dubey et al., 2024), -a-iral--7--Insuutt---O- (Jiang et al., 2023), and Gemma-2-27b-it (Gemma Team et al., 2024). 14
We apply the chat template specific each model, without including any system prompts. Example prompts used are in Appendix G. Using the default configurations, we perform one forward pass on each prompt to extract the most proba- ble output token and the output probability distribution.
For generalizability, we also conduct further experiments with variations of this setup. These include adding instruc- tions and further modifying prompt/question phrasings (see Appendix H); and also prompting LLMS to generate a con- clusion from the premises themselves instead of simply choosing whether or not to accept a given conclusion (see Appendix I).
For comparison, we also conduct a limited evaluation of a popular close-sourced model, GPT-4o (gpt-4o-2024-) 11-20) (OpenAI, 2024) on RULEBREAKERS not have access to its full output probability distribution and internal states.
given that we do
4.3. Implementation
To ensure the model's failure to recognize rulebreakers is not due to a lack of knowledge about the entities involved, we filter out those entity pairs which the model lacks rela- tional knowledge of. For example, an LLM needs to know that the city is situated within its corresponding country to recognize when the conclusion is factually inconsistent with the premises. Therefore, we prompt each of the seven LLMs with "Complete the sentence: [city] is in" for each (country, city) pair and "Complete the sentence: [instance] is a type of for each (type, instance) pair. Using pattern-matching matching, we identified 20 city names for which at least one model did not produce the expected answer (e.g.. some LLMs inter- pret "Georgetown" to refer to the university rather than the capital city of Guyana). We exclude the subset of prompts in D containing any of these 20 city names. We perform a similar check for (type, instance) pairs (see Appendix E) but did not identify any instances that need filtering.
For all open-sourced LLMs, we use the model version hosted on Hugging Face (Wolf et al., 2020). All experi- ments are performed on NVIDIA A100 GPUs, with further details in Appendix J. We access GPT-4o via API.
5. Results
5.1. Accuracy
As shown in Figure 3, while a majority of LLMs achieve an overall paired accuracy above the baseline (0.25), only Meta- 1amma-3-8--Inttrree achieves a paired accuracy greater than 0.6, indicating substantial room for improvement for all LLMs' reasoning capabilities. Notably, GPT-4o underper- forms Phi-3-minn 1-i-3--mii--- Instructt in paired accuracy and even scores the lowest 0.0022 on the MT categorical-entity pairs subset compared to all other models.
We rephrase true/falsc questions into this form: "Is it True or False that [the Conclusion follows from the Premises]? !?" Unless specified otherwise, experiments with GPT-4o were conducted in January 2025.
Where perfect model answers all rulebreakers correctly with "nos""false", _ n Dues| 0 and IT DR
is undefined.
"https: / /huggii aac.....ee.cc. co. spaces/ pppacce//ppen111111 cppen-1mm Teaderboarde //d//ppnn cpen_ 1 1mm_ 1eaderboard
Further details on model size are provided Appendix D.
6

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
(country, city) DS, Eype, instance)
MT, [yyp, instunce)
Rulebreakers
Non-ralebreakers
DS, (country city)
0.9
0.8
0.8
0.7
Paired avcuracy
0.7
Accumey
0.5
0.5
0.4
0.4
0.2
0.2
0.1
0.1
Meta-Ilam
Mate-llams
Mittal-
gommun-
spi-4o-
Phi-3-
medium
Ptt---
Mett-Llama
Meta-LLlama 3-70B
Misrrl- 7B-
genma- 3- 27b-11 (27B)
BDi-le- (sinc unreperted)
mini 1286-
instrctt
medium-
128t-
Intrrct
-588-
Insinact
Indradt
7B-
27bitt (278)
2024-1 1-10
(elec
(4B)
instuct
Instrcc
(SB)
Inxiradi (70B)
v0.3
(41)
(143)
(100)
(BB
enrepetted
(148)
(7R)
Figure 4. Accuracy each LLM achieves on all rulebreakers (TDR) and non-rulebreakers (TpN) in D (95% CI).
Figure 3. Paired accuracy achieved by each LLM (95% CT). The overall paired accuracy achieved over all prompts is shown in red, while the paired accuracy with respect different subsets of prompts are shown other colours. Subsets of prompts are defined with respect the logical rule (MT DS) and kind of entity pair used in each prompt, as set out earlier in Table 1.
rather than geographical entity pairs. As we will further explore in Section 6, we conjecture that a model's ability to recognize rulebreakers may influenced by its "familiarity" with entities mentioned in the prompts (this might be due to e.g. specific city and country names having appeared less frequently in the models' pre-training data than generic nouns, e.g. "wasp"). Nonetheless, the fact that these trends are far from universal suggests that model performance on RULEBREAKERS is driven by multiple competing factors, which highlights challenge for further analysis.
Figure 4 shows that the por-to-mmeiocrr paired accuracy results can be attributed primarily to the models' poor per- formance with respect to rulebreakers. On the one hand, all models perform well on non-nulebreakers, correctly pre- dicting that the conclusion follows from the premises in these cases. Gemma-2-2766 and GPT-4o, for example, even achieve perfect scores. On the other hand, all models perform worse on rulebreaker prompts to various degrees. In particular, three models that perform particularly well on non-rulebreakers, Phi-3-mini- 1ew-1188kkknnttte Instruct, Gemma-2- 27b-it and GPT--o,, also perform the worst on rulebreakers. This suggests that these models may indeed be exhibiting some degree of Dover-generalizing" the two logical rules in study, MT and DS, rigidly applying these rules unlike what is typically expected of wwwwledge-iffornee human reasoners. In other words, while these models exhibit reasoning behavior that appears highly logical and ro- bust against variations in semantic content, this same trait also appears undermine their ability recog- nize when semantic content should in fact be taken into account when reasoning.
5.2. Model Confidence
As shown in Table 2, the value of D is greater than II for all models (p < 0.0001). That is, all models are on aver- age more confident in their positive answer ("yes"""truee") to non-rulebreakers (which is the correct answer) than in their positive answer to rulebreakers (which is the incor- rect answer). This suggests that all models in our study possess a latent ability to distinguish rulebreakers from non- rulebreakers, even where this is not evident from evaluating models only based on their final output tokens. GPT-4o is excluded from this analysis since our methodology (see Sec- tion 3.4) requires access the probability of specific tokens regardless of where they rank in the model's predictions.
In light of these accuracy results, we also conduct further experiments to investigate how models that have been "en- hanced" using various logic-related methods (e.g. fine- tuning on a logic dataset, integrating the model with an exter- nal symbolic solver) would perform on RULEBREAKERS (see Appendix K).
6. Analysis
This section investigates potential factors that contribute to suboptimal performance of the LLMs. That is, we are interested why distinguishing between rulebreakers and non-rulebreakers poses such challenge to these models.
Furthermore, we see high degree of variability in terms of each model's performance on different prompt subsets. Generally, we observe that models achieve higher paired accuracy on the DS subsets compared to MT. and achieve higher paired accuracy on prompts containing categorical
To correctly distinguish rulebreakers and non-rulebreakers. intuitively. a model needs to meet two necessary but not sufficient conditions: (1) accurate and confident knowledge of entities mentioned in the prompts, such as correctly asso- ciating a city with its country, and (2) sufficient attention

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Table 2. For each model, we compare the model's mean confidence (%) in a positive answer ("yes"?"tnuee) between (a) the subset of rulebreaker prompts to which it outputs positive answer nn): and (b) the subset of non-rulebreaker prompts to which it outputs a positive answer DN ). Std dev in brackets.
Table 3. For each model, we extract its familiarity with the [city] entity mentioned in each prompt pair in D and compute the mean (and std dev) corresponding to those prompt pairs the model answers incorrectly. against those it answers correctly.
CORRECT
INCORRECT
MODEL.
IT DR
I
PAIRS
PAIRS
ON
THI-3-MINI-128K
92.055 (8.857)
96.224 (2.738)
PHI-3-MINI-12880
0.906 (0.128)
0.872 (0.172)
INSTRUCT
INSTRUCT
PHI-3-MEDIUM-
93.955 (11.699)
97.158 (8.763)
PHI-3-MEDIUM- 128K-
0.922 (0.089)
0.912 (0.
128K-INSTRUCT
INSTRUCT
META-LLAMA-
77.202 (15.011)
90.457 (12.095)
LLAMA-3-8B-
0.837 (0.187)
0.844 (0.189)
8B-INSTRUCT
INSTRUCT
META-LLAMM-3-
96.336 (9.812)
99.950 (0.366)
LLAMA-3-70B-
0.959 (0.074)
0.949 (0.081)
70B-INSTRUCT
INSTRUCT
MISTRAL-7B-
92.553 (12.863)
98.106 (6.925)
MISTRAL-7B-
0.977 (0.072)
0.969 (0.085)
INSTRUCT-VO.3
INSTRUCT-VO.3
GEMMA-2-27B-IT
97.992 (6.074)
99.996 (0.057)
GEMMA-2-27B-IT
0.997 (0.032)
0.992 (0.048)
GPT-40
0.909 (0. 134)
0.914 (0.136)
the "factual" part of the premises (e.g. "Anne is in Paris") that implicitly contradicts the conclusion (e.g. "Anne is not in France") in case of rulebreakers.
If our hypothesis (1) is correct, we would expect each model to have a higher "familiarity" with respect to prompt pairs in its "recognized" group, i.e. those that the model has answered correctly, as compared to its "failed" group. As shown in Table 3, this is the case for most LLMs, except for P < 0.001). While -aa---llaaa------nnnnie- from the predominant pattern might be explained by its relatively high sensitivity to prompt variations (as further discussed in Appendix F), we recognize however that this not the case for GPT-4o, which is one of the models least sensitive to prompt variations.
Accordingly, we propose two hypotheses: (1) failure to rec- ognize rulebreakers is associated with the model's s lack of confidence in its knowledge of the entities involved? and (2) this failure is associated with insufficient attention given to the factual premise. The intuition is that when a model is not sufficiently "familiar" with the entities mentioned in the prompt (i.e. sufficiently confident in its knowledge about the entities), then it will tend to treat these entities symbol- ically and focus more on the surface form rather than the semantic content of the prompt, thereby overlooking the sig- nificance of the factual premise which implicitly contradicts the conclusion in case of rulebreakers." 18
and GPT-4o (Welch's 's /-test deviation
As an additional observation, when comparing overall fa- miliarity levels across models, we see that simply being familiar with entities does not guarantee that a model will perform well on RULEBREAKERS. A particular example is Gemma-2-277 2-27biit, which is on average more familiar with entities in our dataset compared all other models (in terms of both its "failed" and "recognized" groups) but is among the werst-performing models in terms of paired accuracy. This is also despite the perfect accuracy it has achieved on the non-rulebreaker subset of the prompts (see Section 5.1). When combined, the two observations aptly characterize the blind spot that our study has highlighted with LLMs' reasoning: a model may excel at both retrieving factual knowledge and applying rules in formal logic, yet fail to recognize where rigidly applying these logical rules would produce conclusions that factually contradict the premises.
Is failure associated with the model's 's lack of confidence in its knowledge?
To answer this, we prompt each LLM with the instruction: "Complete the sentence: [city] is in" for each city in RULE- BREAKERS that all LLMs have answered correctly as part of the filtering process in Section 4.1. We extract the output probability of the first token of the country name, which we refer to as the model's "familiarity" with the particular (country, city). We compare each LLM's mean "familiarity" between prompt pairs in Draired the model recognizes
against pairs the model fails as shown in Table 3.
1TTii inspired in part by Neeman al. (2023) investigating how models resolve conflict between in-context information and parametric knowledge.
'FProviding support to this intuition, Johnson- -aird & Byrne (2002) found that human participants are also more likely not to draw factually inconsistent conclusions when given rulebreakers with familiar entities opposed to unfamiliar ones.
We conduct a qualitative analysis on the most common failure instances across models, observing further evidence
breaker, non-rulebreaker) pairs (permuting 6 verbs, 5 first names, logical forms and 10 prompt phrasing variations).
19 As per Section 3.1, each [city] mentioned in 600 (nule-
8

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Halman-like Reasoning
our hypothesis (I), the general though not universal pattern observed in attention distribution suggests that it is one of multiple competing factors influencing model performance on RULEBREAKERS, warranting further investigation in future work.
Table Mean normalized score ratios (with sd) of each I.I.M Paired accuracy (r) from Section 5.1 is displayed for analysis. The greater the ratio, the more the model attends to the factual context.
SCORE RATIO
SCORE RATIO
(INPUT'S GRADIENT)
(ATTENTION)
PHI-3-MINI-12 1EAN--12K-INNSTRCC
0.738 (0,204)
1,037 (0.124) 0.208
PHI-3-MEDIUME IN8K-INSTRUCT
0.657 (0.156)
1,065 (0.534) 0.292
We conduct a further analysis experiment with respect to neuron activations and discuss our findings in Appendix N.
METX-LLAMA-3- --BNNSRUUT
0.814 (0.149)
1,299 (0.200) 0.609
0.827 (0. (0.201)
1.618 (02) 0.497
0.740 (0.163)
1.397 (0.145) 0.476
GEXMA---227--1
0.787 (0.193)
1,558 (0.274) 0.071
7. Conclusion
of our hypothesis. Detailed discussion and examples are provided in Appendix L.
We present RULEBREAKERS, a large-scale dataset for evaluating LLMs' ability to reason with natural language in a knowledge-informed informed and human-like manner by utilizzing common sense and factual knowledge rather than rigidly applying rules prescribed by formal logic. In particular, our study of seven LLMS reveals significant blind spot in their ability to recognize and reject conclusions that can be trivially derived using certain rules in propositional logic but are nonetheless factually inconsistent with the premises. Whilst robust logical reasoning may be desirable in certain applications, our findings highlight the gap between this ap- proach to reasoning and how humans typically reason with natural language in knowledge-infornet attention to important trade-off in relying logic-based methods and constraints to improve LLMs" general reason- ing capabilities.
failure associated with the model overlooking the factual information in the second premise?
To answer this, we apply feature attribution methods (inpul x gradient (Simonyan et al., 2014) and attention (Bahdanau et al., 2015))²0 to assign an importance score to each token in a prompt. We then sum the importance score of tokens within the second premise, and the first premise, separately. To ensure a fair comparison between the two, we normalize the importance score by the number of tokens aggregated, since the second premise (e.g. "Anne is in Paris") is always shorter than the first premise (e.g. "If Anne is in France, then she is not in Paris."). We then compute a "score ru- tio" by dividing the normalized importance score of the second premise by the normalized importance score of the first. Therefore, the lower the ratio, the less importance is assigned to the second premise (factual context) by the model. For each model, we compute a score ratio averaged across all prompts. GPT-4o is excluded since we do not have access to the model's ' internal states or gradient information.
manner, drawing
A Acknowledgement
This work was supported by the UKRI AI Centre for Doe- toral Training in Speech and Language Technologies (SLT) and their Applications funded by UK Research and Innova- tion [grant number EP/S023062/1J. For the purpose of open access, the author has applied a Creative Commons Attribu- tion (CC BY) licence to any Author Accepted Manuscript version arising. We acknowledge IT Services at The Uni- versity of Sheffield for the provision of services for High Performance Computing.
Our second hypothesis is that low paired accuracy is associ- ated with insufficient attention a model assigns to the factual information in the second premise. As shown in Table 4, this appears to be the case when comparing Phi-3-medium- 128k-instruct, Dessral-7B-IInsruct-vvo
and Meta-Llama-3- 70B-Instruct: the greater the т, the higher the score ratio, indicating the greater attention to the second premise. The two models with the highest t, Satta-Llama-388 -aaee-------ns-tuu Instruct also have the two highest score ratios. However, Gemma-2-27b-it notably has the third highest score ratio (whether computed by input gra- dient or attention) but exhibits the lowest by a substantial margin.21 21 These findings suggest that the relation between failure and attention distribution is less clear-cut than ex- pected. Similar to the "familiarity" factor investigated in
Impact Statement
and Meta-Llama-3-77BBInne
This work advances our understanding of language models' reasoning capabilities, revealing critical implications for AI safety and system robustness. Our findings directly in- form the development of more reliable AI systems through concrete metrics and benchmarks. These contributions are crucial as AI increasingly drives high-stake decisions where alignment with human reasoning processes and judgments is essential.
Input x gradient has been shown to be reasonably faithful across models and datasets (Zhao et al., 2022, Zhao & Aletras, 2023). To compare, we also compute a separate score ratio using attention weights (Bahdanau et al., 2015). We implement both methods with inseq v0.7.0 (Sarti et al., 2023).
We also analyzed unnormalized score ratios (see Appendix M) but did not find any substantive deviation from these observations.
9

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
References
lations, 2024, URL https: httpp:://penreview..n Forum?id=GPKTTIE
Abdin, M. I., Ade Jacobs, S., Awan, A. A., Aneja, J., Awadallah, A., Hassan Awadalla, H., Bach, N., Bahree,
Bian, N., Han, X., Sun, L., Lin, Jiang. S., and Dong, B. ChatGPT is a knowledgeable but inexperienced solver: An investigation of common- sense problem large language models. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint Interna- tional Conference on Computational Linguistics, Lan- guage Resources and Evaluation (LREC-COLING 2024), pp. 3098-31 10, Torino, Italia, May 2024. ELRA and ICCL. URL https: httpatttsss//aaattttstteeeeeeeee /tt///aaaaaattcgoeeeeeee org/2024. 2024. Irec-ma i n 276/.
Y. He. B.,
A., Bakhtiari, A.. Behl, H.. Benhaim, A., Bilenko, M., Bjorck, J., Bubeck, S., Cai, M., Mendes, C. C. T., Chen, W., Chaudhary, V., Chopra, P., Giorno, A. D., de Rosa, G.. Dixon, M., Eldan, R., Iter, D., Goswami, A., Gunasekar, S., Haider, E., Hao, J., Hewett, R. J., Huynh, J., Javaheripi, M., Jin, X., Kauffmann, P., Karampatziakis, N., Kim, D., Khademi, M., Kurilenko, L., Lee, J. R., Lec, Y. T. Li, Y.. Liang. C., Liu, W. Lin, X. E.,, Lin, Z., Madan, P. Mitra, A., Modi, H., Nguyen, A., Norick, B. Patra, B., Beecer, D., Portet, T., Pryzant, R., Qin, H., Radmilac, M., Rosset, C.,, Roy, S., Saarikivi, O., Saied, A., Salim, A., Santacroce, M., Shah, S., Shang, N., Sharma, H., Song, X., Ruwase, O., Wang, X., Ward, R., Wang, G., Witte, P., Wyatt, M., Xu, C., Xu, J., Xu, W., Yadav, S., Yang, F., Yang, Z., Yu, D., Zhang, C., Zhang, C., Zhang, J., Zhang, L. L., Zhang, Y., Zhang, Y., and Zhou, X. Phi-3 technical report: A highly capable language model locally on your phone. Technical Report MSR-TR-2024-12, Microsoft, August 2024. URL https://www.mii
Binz, M. and Schulz, E. Using cognitive psychology to understand gpt-3, Proceedings of the National Academy of Sciences, 12066):c2118223122 pnas 221 22as:2218523121 URL https:/ /ttps://wwwww// /ww.. www.wwww.....wp..www............ www.ppaaas..o.o.. org/ doi/ doi/abs/ae /10. 1073 1000//pa....... 2218523120.
2023. doi: 10.1073/
Bos, J. and Markert, K. Recognising textual entailment with logical inference. In Mooney, R., Brew, C.,, Chien, I..F., and Kirchhoff, K. (eds.), Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp. 238-635, Vancouver, British Columbia, Canada, October 2005. Association for Computational Linguistics. URL https: http://aaaaaathhocceooo org/ H00--10077
sepott-a--highhl--aaabbbell----e on your -yroner.pooee.....
Ahn, J. Verma, R., Lou, R., Liu, D., Zhang, R., and Yin, W. Large language models for mathematical reason- ing: Progresses and challenges. In Falk, N., Papi. S., and Zhang, M. (eds.), Proceedings of the 18th Confer- ence of the European Chapter of the Association for Computational Linguisties: Student Research Workshop, pp. 225-237, St. Julian's, s, Malta, March 2024. Asso- ciation for Computational Linguistics. URL https: //aclanthologe wow.lnnthoooyy..oooooooo owg//o////2//. eac44-eacll-cm srw. 17/.
Calanzone, D., Vergari, A., and Teso, S. Towards logi- cally consistent language models via probabilistic rea- soning. In ICLR 2024 Workshop on Reliable and Re- sponsible Foundation Models, 2024. URL https: / //openreview.... netp://ee/tteeeeeeeeeeeeeeeeeeee www.fforrmmiccooo 2ixxumiidgg3SGaf1111.
Castello, M., Pantana, G., and Torre I. Examining cog- nitive biases in ChatGPT 3.5 and ChatGPT 4 through human evaluation and linguistic comparison. In Knowles, R., Eriguchi. A., and Goel, S. (eds.), Proceedings of the 16th Conference of the Association for Ma- chine Translation in the Americas (Volume I: Re- search Track), pp. 250-260, Chicago, USA, September 2024. Association for Machine Translation in the Amer- icas. URL https: / /aa//claaattt aclanthelogy hwwpchhhllgoooooooe.oeeeee org/oo/2/. am anta-ressarch.22
Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In 3rd International Confer- ence on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceed- ings, 2015. URL http: /tattp://axxv... www.aaiwwwr./a.aaaeaaaoaaeooeeee wwtp://orgg/b//////////oo 0473.
Bengio, Y. and LeCun, Y. (eds.), 3rd
D., Barwise, J., and Etchemendy, J. Lan guage, Proof, and Logic: Second Edition. Center for the Study of Language and Information/SRI, 2nd edition, 2011. ISBN 1575866323.
Chen, X., Chi, R. A., Wang, X., and Zhou, D. Premise order matters in reasoning with large language models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A.. Oliver, N., Scarlett, J., and Berkenkamp. F. (eds.), Pro- ceedings of the 41st International Conference on Mu- chine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 6596 66200. PMLR, 21-27 Jul 2024. URL https: wttpttsss/ppoceeehieeee http://poceeeinngee/ee mlr press/ v235/ chen241. html.
Berglund, L., Tong, M., Kaufmann, M., Balesni, M., Stick- land, A. C., Korbak, T., and Evans, O. The reversal curse: LLMs trained on "a is b" fail to learn "b is a". In The Twelfth International Conference on Learning Represen-
10

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Yang, A., Fan, A.,
Yazdan, A., James, B., Maurer, B., Leonhardi, B., Huang. B., Loyd, B., Paola, B. D., Paranjape, B., Liu, B., Wu, B., Ni, B., Hancock, B., Wasti, B., Spence, B., Stojkovic, B., Gamido, B., Montalvo, B., Parker, C., Burton, C., Mejia, C., Wang, C.,, Kim, C.,, Zhou, C.,, Hu, C.,, Chu, C.-H., Cai, C., Tindal, C., Feichtenhofer, C., Civin, D., Beaty, D.,, Kreymer, D., Li, D., Wyatt, D., Adkins, D., Xu, D., Testuggine, D., David, D., Parikh, D., Liskovich, D., Foss, D., Wang. D., Le, D., Holland, D., Dowling, E.,, Jamil, E., Montgomery, E., Presani, E., Hahn, E., Wood, E., Brinkman, E., Arcaute E., Dunbar, E.,, Smothers, E.,, Sun, F. Kreuk, F., Tian, F. Ozgenel, F., Caggioni, F. Guzmán, F., Kanayet, F., Seide, F., Florez, G. M., Schwarz, G., Badeer, G., Swee, G., Halpern, G., Thattai, G., Herman, G., Sizov, G., Guangyi, Zhang, Lakshminarayanan, G., Shojanazeri, H. Zou, H., Wang, H., Zha, H., Habeeb, H., Rudolph, H., Suk, H.,, Aspegren, H.,, Goldman, H., Damlaj, I., Molybog, I., Tufanov, I., Veliche, I.-E.,, Gat, I. Weissman, J., Geboski, J., Kohli, J., Asher, J., Gaya, J.-B., Marcus, J., Tang. J., Chan, J., Zhen, J.. Reizenstein, J., Teboul, J., Zhong, J., Jin. J., Yang. J., Cummings, J., Carvill, J., Shepard, J., McPhie, J., Torres, J., Ginsburg, J.. Wang. J., Wu, K., U. K. H., Saxena, K., Prasad, K., Khandelwal, K. Zand, K., Matosich, K., Vecraragha- van, K., Michelena, K., Li, K. Huang, K., Chawla, K., Lakhotia, K., Huang, K., Chen, L., Garg. L., A, L., Silva, L.., Bell, L.,, Zhang, L.,, Guo, L., Yu, I.., Moshkovich, L.. Wehrstedt, L., Khabsa, M., Avalani, M., Bhatt, M., Tsimpoukelli, M., Mankus, M., Hasson, M., Lennie, M., Reso, M., Groshev, M., Naumov, M., Lathi, M.. Ke- neally, M., Seltzer, M. I.., Valko, M., Restrepo, M., Patel, M., Vyatskov, M., Samvelyan, M., Clark, M., Macey, M., Wang, M., Hermoso, M. J., Metanat, M., Rastegari, M., Bansal, M., Santhanam, N., Parks, N., White, N., Bawa, N., Singhal, N., Egebo, N., Usunier, N., Laptev, N. P., Dong. N., Zhang, N., Cheng, N., Chernoguz, O., Hart, O., Salpekar. O., Kalinli, O., Kent, P., Parekh, P., Saab, P., Balaji, P., Rittner, P., Bontrager, P., Roux, P., Dollar, P., Zvyagina, P., Ratanchandani, P., Yuvraj. P., Liang, Q. Alao, R., Rodriguez, R., Ayub, R., Murthy, R., Nayani, R., Mitra, R., Li, R., Hogan, R., Battey, R., Wang. R., Maheswari, R., Howes, R., Rinott, R., Bondu, S. J., Datta, S., Chugh, S., Hunt, S., Dhillon, S., Sidorov, S., Pan, S., Verma, S., Yamamoto, S., Ramaswamy, S., Lindsay, S., Lindsay, S., Feng, S., Lin, S., Zha, S. C., Shankar, S., Zhang, S., Zhang, S., Wang, S., Agarwal, S., Sajuyighe, S., Chintala, S., Max, S., Chen, S., Kehoe, S. Satterfield, S., Govindaprasad, S., Gupta, S., Cho, S., Virk, S., Subramanian, S., Choudhury, S., Goldman, S., Remez, T., Glaser, T., Best, T., Kohler, T., Robinson, T., Li, T., Zhang, T., Matthews, T., Chou, T., Shaked, T., Vontimitta, V., Ajayí, V., Montanez, V., Mohan, V., Kumar, V. S., Mangla, V., Albiero, V., lonescu, V., Poe- naru, V., Mihailescu, V. T., Ivanov, V., Li, W., Wang, W.,
Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravanku-
mar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gregerson, A., Spataru, A., Roziere, B., Biron, B., Tang, B., Chern, B., Caucheteux, C., Nayak, C., Bi, C., Marra, C., McConnell, C., Keller, C., Touret, C., Wu, C., Wong, C.. Ferrer, C. C., Nikolaidis, C., Al- lonsius, D., Song, D., Pintz, D., Livshits, D., Esiobu, D., Choudhary, D., Mahajan, D., Garcia-Olane D., Perino, D., Hupkes, D., Lakomkin, E., AlBadawy, E.,, Lobanova, E., Dinan, E., Smith, E. M., Radenovic, F., Zhang, F., Syn- naeve, G., Lee. G., Anderson, G. L., Nail, G., Mialon, G., Pang, G., Cucurell, G., Nguyen, H., Korevaar, H., Xu, H., Touvron, H.. Zarov, I., Ibarra, I. A., Kloumann, I., Misra, I. Evtimov, I., Copet, J., Lee, J., Geffert, J., Vranes, J.. Park, J., Mahadeokar, J., Shah, J., van der Linde, J., Billock, J., Hong. J., Lee, J., Fu, J., Chi, J., Huang, J., Liu, J., Wang, J., Yu, J., Bitton, J., Spisak, J., Park, J., Rocca, J., Johnstun, J., Saxe, J., Jia, J., Alwala, K. V., Upasami. K., Plawiak, K., Li, K., Heafield, K., Stone, K., El-Arini, K., lyer, K., Malik, K., Chiu, K., Bhalla, K., Rantala-Yeary, L., van der Maaten, L.., Chen, L.., Tan, L.. Jenkins, L., Martin. L., Madaan, L., Malo. L., Blecher, L., Landzaat, I.., de Oliveira, I.., Muzzi, M., Pasupuleti, M., Singh, M., Paluri, M., Kardas, M., Oldham, M., Rita, M., Pavlova, M., Kambadur, M., Lewis, M., Si, M., Singh, M. K., Hassan, M., Goyal, N., Torabi, N., Bashlykov. N., Bogoychev, N., Chatterji, N., Duchenne, O., Çelebi, O., Alrassy, P., Zhang, P., Li, Vasic, P., Weng, P., Bhargava, P., Dubal, P., Krishnan, P., Koura, P. S., Xu, P., He, Q., Dong, Q. Srinivasan, R., Ganapathy, R., Calderer, R., Cabral, R. S., Stojnic, R., Raileanu, R., Girdhar, R., Patel, R., Sauvestre, R., Polidoro, R., Sumbaly, R., Taylor, R., Silva, R., Hou, R., Wang, R., Hosseini, S., Chennabas- appa, S., Singh, S., Bell, S., Kim, S. S., Edunov, S., Nie, S., Narang, S. Raparthy, S., Shen, S., Wan, S., Bhosale, S., Zhang. S., Vandenhende, S., Batra, S., Whitman, S., Sootla, S., Collot, S., Gururangan, S., Borodinsky, S., Her- man, T., Fowler, T., Sheasha, T., Georgiou, T., Scialom, T., Speckbacher, T., Mihaylov, T., Xiao, T., Karn, U., Goswami, V., Gupta, V., Ramanathan, V., Kerkez, V., Gonguet, V., Do, V., Vogeti, V., Petrovic, V., Chu, W., Xiong, W., Fu, W., Meers, W., Martinet, X., Wang, X., Tan, X. E., Xie, X., Jia, X., Wang, X., Goldschlag, Y., Gaur, Y., Babaei, Y., Wen, Y., Song, Y., Zhang, Y., Li, Y. Mao, Y., Coudert, 7. D., Yan, 7.,, Chen, 7., Papakipos, 7 Singh, A., Grattafiori, A., Jain, A., Kelsey, A., Shajnfeld, A., Gangidi, A., Victoria, A., Goldstand, A., Menon, A., Sharma, A., Bocsenberg, A., Vaughan, A., Baevski, A., Feinstein, A., Kallet, A., Sangani, A., Yunus, A., Lupu, A., Alvarado, A., Caples, A., Gu, A., Ho, A., Poulton, A., Ryan, A., Ramchandani, A., Franco, A., Saraf, A., Chowdhury, A., Gabriel, A., Bharambe, A., Eisenman, A.,

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Jiang, W., Bouaziz, W., Constable, W., Tang, X., Wang,
Eccles, T., Hennigan, T., Kocisky, T., Doshi, T., Jain, V., Yadav, V., Meshram, V., Dharmadhikari, V., Barkley, W., Wei, W., Ye, W., Han, W., Kwon, W. Xu, X., Shen, Z. Gong, Z., Wei, Z., Cotruta, V., Kirk, P., Rao, A., Giang, M., Peran, L.,, Warkentin, T., Collins, E., Bar- ral, J., Ghahramani, Z., Hadsell, R., Sculley, D., Banks, J., Dragan, A., Petrov, S., Vinyals, O., Dean, J., Has- sabis, D., Kavukcuoglu, K., Farabet, C., Buchatskaya, E., Borgeaud, S., Fiedel, N., Joulin, A., Kenealy, K., Dadashi, R., and Andreev, A. Gemma 2: Improving open language models at practical size, 2024. URL https://azzi org/ abs/ 20002 2001.00188......
X., Wu, X., Wang, X., Xia, X., Wu, X., Gao, X., Chen, Y., Hu, Y., Jia, Y., Qi, Y., Li, Y., Zhang, Y., Zhang, Y., Adi, Y., Nam, Y. Yu, Wang, Hao, Y., Qian, Y., He, Y., Rait, Z., Vito, Z., Rosnbrick, Z., Wen, Z., Yang, Z., and Zhao, Z. The Ilama 3 herd of models, 2024. URL https: http//aaw.r.aaaaaaa////////a btggabs/////oooooo////oo////beba abs 2407. 21783.
Eisape. T., Tessler, M., Dasgupta. I., Sha, F., Steenkiste, S., and Linzen. T. A systematic comparison of syllogistic reasoning in humans and language models. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies (Volume I: Long Papers), pp. 8425- 8444, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 10.18653/w1/222..maad long.466. URL https: http://aaaannntooo.ooooo org/ 2024. www.aaaanccnnee--00011
00118.
Han, S., Schoelkopf, H., Zhao, Y., Qi, Z., Riddell, M., Zhou, W., Coady, J., Peng. D., Qiao, Y., Benson, L., Sun, L., Wardle-Solanc, A., Szabó, H., Zubova, E., Burtell, M., Fan, J., Liu, Y., Wong, B., Sailor, M., Ni, A., Nan, L.. Kasai, J., Yu, T., Zhang, R., Fabbri, A., Kryscinski, W. M., Yavuz, S., Liu, Y., Lin, X. V., Joty, S., Zhou, Y., Xiong, C., Ying, R., Cohan, A., and Radev, D. FO- LIO: Natural language reasoning with first-order logic. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22017-22031, Miami, Florida, USA, November 2024. Association for Com- putational Linguistics. doi: 10. 10.18653/11/2244.om main. 1229. URL https: / //a/ccaanttt aaclanooooooyyy.... org/ 2024 emnlp main. 1229/.
Gemma Team, Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L.., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P. Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., Girgin. S., Momchev, N., Hoffman, M., Thakoor, S., Grill, J.-B., Neyshabur, B., Bachem, O., Walton, A., Severyn, A., Par- rish, A., Ahmad, A., Hutchison, A., Abdagie, A., Carl, A., Shen, A., Brock, A., Coenen, A., Laforge, A., Pater- son, A., Bastian, B., Piot, B., Wu, B., Royal, B., Chen, C. Kumar, C., Perry, C., Welty, C., Choquette-Choo, C. A., Sinopalnikov, D.. Weinberger, D., Vijaykumar, D., Rogozinska, D., Herbison, D., Bandy, E., Wang. E., Noland, E., Moreira, E., Senter, E., Eltyshev, E., Visin, F., Rasskin, G., Wei, G., Cameron, G., Martins, G., Hashemi, H.,
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus), 2023. URL https://arxiv.orr abb// 1606. 1606.08415. 08415.
Holliday, W. H., Mandelkern, M., and Zhang, C. E. Condi- tional and modal reasoning in large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods Natural I Lnguuge Processing, pp. 3800-3821, Miami, Florida, USA, November 2024. Association for Com- putational Linguistics. doi: 10.18653/v 1/2234///002. main.222. URL. https:/ / declanthologg org/ 2024. emnlp-main at0-main.22//.
H., Batra, H., Dhand, H., Nar- dini, I, Mein, J., Zhou, J., Svensson, J., Stanway, J., Chan, J., Zhou, J. P., Carrasqueira, J., Iljazi, J., Becker, J., Fernandez, J., van Amersfoort, J.. Gordon, J., Lipschultz. J. Newlan, J.. yeong Ji, J.. Mohamed, K., Badola, K., Black, K. Millican, K., McDonell, K. Nguyen. K., Sod- hia, K., Greene, K., Sjoesund, I.. I.., Usui, I.., Sifre, I.., Heuermann, L., Lago, L., McNealus, L., Soares, L. B., Kilpatrick, L., Dixon, L., Martins, L., Reid, M., Singh, M. Iverson, M., Görner, M., Velloso, M., Wirth, M., Davidow, M., Miller, M., Rahtz, M., Watson, M., Ris- dal, M., Kazemi, M., Moynihan, M., Zhang, M., Kahng, M., Park, M., Rahman, M., Khatwani, M.. Dao, N., Bar- doliwalla, N., Devanathan, N., Dumai, N., Chauhan, N., Wahltinez, O., Botarda, P., Barnes, P., Barham, P., Michel, P., Jin, P. Georgiev, P., Culliton, P., Kuppala, P., Co- manescu, R., Merhej. Jana, R., Rokni, R. A., Agarwal, R., Mullins, R., Saadat, S., Carthy, S. M., Cogan, S., Perrin, S., Amold, S. M. R., Krause, S., Dai, S., Garg, S., Sheth, S., Ronstrom, S., Chan, S., Jordan, T., Yu, T.,
Holtzman, A., West, P., Shwartz, V. Choi, Y. and Zettlemoyer, L. Surface form competition: Why the highest probability answer isn't 't always right. In Moens, M.-F., Huang, X., Specia, I.,, and Yih, S. W.-t (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7038-7051, Online and Punta Cana, Dominican Republic, November 2021. Association for Computa- tional Linguisties. doi: 10. 10.18663/v112221.cmmmmmm 564. URL emnlp-man ma..a.in..6... 564/.
Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C.,
12

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L.,, Lavaud, I.. R., Lachaux, M.- A., Stock, P., Scao, T. L.,, Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023, URL https: //arxiv. org/ abaa 6825.
Li, Y., Zhao, Z., and Scarton, C. Label set optimization via activation distribution kurtosis for zero-shot classification with generative models. arXiv preprint arXi::4410.1199 2024.
Lu, Y., Bartolo, M., Moore, A., Riedel, S., and Stene- torp, P. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sen- sitivity. In Muresan, S., Nakov, P., and Villavicen- cio, A. (eds.), Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ire- land, May 2022. Association for Computational Linguis- ties. doi: 10. 10.18655ww//20222.clllooozg wwwp://gg.22......o woiecllllggg 556/.
Johnson-Laird, P. N. Menial models. Cambridge University Press, 1983.
Johnson-Laird, P. N. Against logical form. Psychologica Belgica, Oct 2010. doi: 00.5334/bb-503344-
Johnson-Laird, P. N. and Byme, R. M. J. Conditionals: A theory of meaning, pragmatics, and inference. Psycho- logical Review, 109(4):646-678, 2002. doi: 10.1037/
URL https:
(033-295X.109 URL https: / /ttp:/oo.o.oooo/////////o 1037/0033 2958. 109.4. . 646.
Meadows, J., Valentino, M., Teney, D., and Freitas, A. A symbolic framework for evaluating mathematical rea- soning and generalisation with transformers. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers). pp- 1505-1523, Mexico City, Mexico, June 2024. Associa- tion for Computational Linguistics. doi: 10.18653/v1/ 2024 moaclloncl- 84. URL https: h/////aaaantoooryoo/oooo org/2024 naacl -
Kamath, A. and Das, R. A survey semantic parsing. In Automated Knowledge Base Construction (AKBC), 2019. URL https: http//pppprrevieeeeee wwt.eviiiiiieeeoounnn. id=HylaEWeTT7.
Khemlani, S. and Soanson-Laird P. Why machines don't (yet) reason like people. KI- - Künnstliche Intelligenz, 33
(3):219 228, Sep 2019. ISSN 1610-1987. doi: 10.1007/ s13218-019
URL https: /ttp://o....o.../////// 1007/s13218 01177/1122180019000333
84/.
Khemlani, S. S. and Johnson-Laird, P. N. Illusions in reasoning. Minds and Machines, 27(1):11-35, Mar 2017. ISSN 1572-8641. doi: 10. 10.1007//11023--17-9402 x. URL https: / /tt.///oiioo//o 10. 1007 1000//010..000/110020. 017 9421-x.
Mondorf, P. and Plank, B. Comparing inferential strate- gies of humans and large language models in deductive reasoning. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9370-9402, Bangkok, Thailand, August 2024a. Association for Computational Linguis- tics. doi: 10. (0.18653/v120244 aow.o/o......-.oo......00. URL https: / //aclanthology org / 2024
Khemlani, S. S., Byme, R. M. J., and Johnson-Laird, P. N. Facts and possibilities: A model-based the- ory of sentential reasoning. Cognitive Science, 42(6): 1887-1924, 2018. doi: https///bi.org//1.1111// 12634. URL https:/ onl wwwoonneennnbrrrrrr wi ley. com/do /o/ abs//b//////// / 10. /1/11///oos/ 12634.
508/.
Mondorf, P. and Plank, B. Beyond accuracy: Evaluat- ing the reasoning behavior of large language models survey. In First Conference Language Modeling, 2024b. URL https: htttp:://pperreeeeeeeee /wwpreeeeeeeeeeeeeeeeeeeeeeeeeee nwt. forum? id=Imjg12nllu.
Korman, D. Z., Mack, E., Jett, J., and Renear, A. H. Defining textual entailment. Journal of the Association for Information Science and Technology, 69(6))663-77 2018. doi: https://doi.org//0000000000 100p:/as...0000//.....2000000000 bttps::/asiite wtt.sissttt....l.lleeierr.r.aaaa wiley. com/do! /abs//b///// 1002 224007 24007.
URL
Morishita, T., Morio, G., Yamaguchi, A., and Sogawa, Y. Enhancing reasoning capabilities of LLMs via princi- pled synthetic logic corpus. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: http:::///ppeeee/eeee wwtpp/ewnneeeeeeeeeppppee net forum? id=m jDUa@pIn.
Lampinen, A. K., Dasgupta, I., Chan, S. C. Y., Sheahan, H. R., Creswell, A., Kumaran, D., McClelland, J. L., and Hill, F. Language models, like humans, show content effects on reasoning tasks. PNAS Nexus, 3(7):paae23,, 07 2024. ISSN 2752-6542, doi: 10. 10..199/pnassexuus/pgee22 URL https: / /oiioiioooo org 10. 1093 ppwps//xaa/ass///ea//meaa pgae233.
Neeman, E., Aharoni, R., Honovich, O., Choshen, L., Szpektor, I., and Abend, O. DisentQA: Disentangling parametric and contextual knowledge with counterfac- tual question answering. In Rogers, A., Boyd-Graber,
13

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
J., and Okazaki, N. (eds.), Proceedings of the 61 An- nual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pp. 10056-10070, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 1/116666/11/22222.ccc-oogg URL https: http:://aaaaattoolgo.... org/:/23.a222////// long. 559/.
Quelhas, A. C. and Johnson-Laird, P. The modulation of disjunctive assertions. The Quarterly Journal of Experi- mental Psychology, 70(4):703-717, 2017. doi: 10.1080/ 714700218.2016.1154000 URL https: www.dooog....oo.... 10.1080/17470000 .21 6. 1154079. PMID: 26878158.
Quelhas, A. C., Johnson-Laird, P. N., and Juhos, C. The modulation of conditional assertions and its ef- fects reasoning. Quarterly Journal of Experimen- tal Psychology, 63(9):1716-1739) 2010. doi: 10.1080/ 174702 URL https: /ttttts:::/ooo/ /ttp://o/oo//////////// 1 (080/17470210900300
Olausson, T., Gu, A., Lipkin, B., Zhang, C., Solar- Lezama, A.. Tenenbaum, J., and Levy, R. LINC: A neurosymbolic approach for logical reasoning by com- bining language models with first-order logic provers. In Bouamor, H., Pino, J., and Bali, K. (eds.), Pro- ceedings of the 2023 Conference Empirical Meth- ods in Natural Language Processing, pp. 5153-5176, Singapore, December 2023. Association for Computa- tional Linguistics. doi: 10.18653/v1/0023eemmllp main. 313. URL https: htc//claann htt/a/aaaattooooo..ooooo wrwp:///ooloo..//223333333 emnlp-main. 313/.
PMID: 20204920.
Ragni, M. and Johnson-Laird, P. Reasoning about pos- human reasoning violates all normal modal logics. In Proceedings of the Annual Meeting of the Cog- nitive Science Society, volume 40, 2018. URL https: / //escholarship wwwg/c/geaeeeooooooooooeee wttp:///ce//////////e
sibilities:
Saparov, A. and He, H. Language models are greedy rea- soners: A systematic formal analysis of chain-of-thought. In The Eleventh International Conference on Learning Representations, 2023. URL https: / openreviee. net (wt.ffrrmmmmi-FFFFBxxxee-e
OpenAl. Hello gpt-4o. https://openati com/ teadex/hellogpptttoe May 2024.
Pan, L., Albalak, A., Wang, X., and Wang, W. Logic- LM: Empowering large language models with symbolic solvers for faithful logical reasoning. In Bouamor, H., Pino, and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3806- 3824, Singapore, December 2023. Association for Com- putational Linguistics. doi: 10.18653/v1/2023.n findings- emnlp.248. URL https:: : / ac anthology........ 2023. wwwndinnseeenlp......ee 248/.
Saparov, A. and Mitchell, T. M. Towards general natural language understanding with probabilistic worldbuilding. Transactions of the Association for Computational Lin- guistics, 10:325-342, 04 2022. ISSN 2307-387X. doi: 2..1162/aacl..00446. URL https: /ttp::///do..ooo org/00.........oooooo 1 1166:/tacl_a_0040
Saparov, A., Pang, R. Y., Padmakumar, V., Joshi, N., Kazemi, M., Kim, N., and He, H. Testing the general deductive reasoning capacity of large language models us- ing OOD examples. In Thirty-seventh Conference on Neu- ral Information Processing Systems, 2023. URL https: HgPO.
Parmar, M., Patel, N., Varshney, N.. Nakamura, M., Luo, M., Mashetty, S., Mitra, A., and Baral, C. LogicBench: Towards systematic evaluation of logical reasoning abil- ity of large language models. In Ku, L.-.., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers). pp. 13679-13707, Bangkok, Thailand, August 2024. Association for Com- putational Linguistics. doi: 10. 10.18653////0024.ddloogg 739. URL https: https://caaa http///aaaantooolooooo.r//////// ac 1oll-oog..20 739/.
/ /ppenreview net/
Sarti, G., Feldhus, N., Sickert, L., and van der Wal, O. Inseq: An interpretability toolkit for sequence genera- tion models. In Bollegala, D., Huang, R., and Ritter, A. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 421-435, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https:
Qiao, S., Ou, Y., Zhang, N., Chen, X., Yao, Y., Deng, S., Tan, C., Huang, F., and Chen, H. Reasoning with lan- guage model prompting: A survey, In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5368-5399 Toronto, Canada, July 2023. Association for Computa- acll URL https: / / //t///acllna aclanthologyy long.294/.
doi: 10. / //aclanthology wwgg acl demo 40/.
Simonyan, K., Vedaldi, A., and Zisserman, A. Deep in- side convolutional networks: Visualising image clas- sification models and saliency maps. In Workshop at International Conference on Learning Representations, 2014. URL https: openrevieww net/ forum? 1d=c04ycnpaxKc55.
tional Linguistics. doi: wwtpollloy..oo.//////ee
wwt.ooo..........oooo -acl-
14

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Speer, R., Chin, J., and Havasi, C. Conceptnet 5.5: open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelli- gence, AAAT 17, pp. 4444 4451. AAAI Press, 2017.
Thailand, August 2024. Association for Computa- tional Linguistics. doi: 10. 10.18653/11/2004.fnnnim 75. URL https:/ https////aaaaaanllgg..oooo otg. 2024. Findings-acl. 75/.
Sun, J., Zheng, C., Xie, E., Liu, Z., Chu, R., Qiu, J., Xu, J., Ding. M., Li, H., Geng, M., Wu, Y., Wang, W., Chen, J., Yin. Z., Ren, X., Fu, J., Hc, J.. Wu, Y., Liu, Q., Liu, X., Li, Y., Dong, H., Cheng, Y., Zhang, M., Heng, A.. Dai, J., Luo, P., Wang. J., Wen, J.-R., Qiu, X., Guo, Y., Xiong, H., Liu, Q., and Li, Z. A survey of reasoning with foundation models: Concepts, methodologies, and outlook. ACM Comput. Surv., April 2025. ISSN 0360- 0300. doi: 10.1145/3729218. URL https: /doi.. arg/10.1 1
Vrandežić, D. and Krötzsch, M. Wikidata: a free collabora- tive knowledgebase. Commun. ACM, 57(10):78-85 sep 2014. ISSN 0001-0782. doi: 10.1145/2629489 URL https:/ httpss//ddoi.ooo org/ 11 /1/1444//2228...
Wan, Y., Wang, W., Yang, Y., Yuan, Y., Huang, J.-t. He, P., Jiao, W., and Lyu, M. LogicAsker: Evaluating and improving the logical reasoning ability of large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Proceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pp. 2124- 2155, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 118633/1122222 emnlp-main. 128. URL https: acc aal.oooooooooyyy.... org / 2024. emnlp main.128/.
Just Accepted.
Suri, G., Slater, L. R., Ziaee, A., and Nguyen, M. Do large language models show decision heuristics similar to humans? a case study using gpt-3.5. Journal of Exper- imental Psychology: General, 153(4): 1066- 2024.
doi: 10. 10770//00000
Wang. S., Zhong, W., Tang, D., Wei, Z., Fan, Z., Jiang, D., Zhou, M., and Duan, N. Logie-driven context ex- tension and data augmentation for logical reasoning of text. In Muresan, S., Nakov, P., and Villavicen- cio, A. (eds.), Findings of the Association for Com- putational Linguistics: ACL 2022, pp. 1619-1629, Dublin, Ireland, May 2022. Association for Computa- tional Linguistics. doi: 10.18633 127. URL http:////aaaaaatogooooooooee f Findings-acl.122
Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung. H. W., Chowdhery, A.., Le, Q., Chi, E., Zhou, D., and Wei, J. Challenging BIGibench tasks and whether chain-of-thought can solve them. In Rogers, A., Boyd- Graber, J., and Okazaki, N. (eds.). Findings of the As- sociation for Computational Linguisties: ACI. 2023, pp- 13003-13051, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/0033 findings-acl.824. URL https: / /aclaaaaannooo..... org/2023. wwwfffinnnggnn
8 Talmor, A., Yoran, O., Bras, R. I.., Bhagavatula, C., Gold- berg, Y., Choi, Y., and Berant, J. CommonsenseQA 2.0: Exposing the limits of AI through gamification. In Thirty-fifth Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 1), net/ forum? 2021. URL http:p::///ppeeevviieeee id=cF7F1UT5cks
Wason, C. Reasoning about a rule. Quarterly Journal of Experimental Psychology, 20(3);273-281, 1968. doi: 404.0000016604444040000000000
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Met- zler, D., Chi, E. H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J., and Fedus, W. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022a. ISSN 2835-8856. URL https: / openreview. hwt.//ee///ae. htt///oormmmii///// foorrmmiddyyksussssii Sur- vey Certification.
Tian, Li, Y., Chen, W., Xiao, L., He, H., and Jin, Y. Di- agnosing the first-order logical reasoning ability through LogicNLI. In Moens, M.-F., Huang, X., Specia, L.,, and Yih, S. W.-t (eds.), Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pp. 3738-3747, Online and Punta Cana, Do- minican Republic, November 2021. Association for Com- putational Linguisties. doi: 10.18653/1122011emmmi) main.303. URL https: ttitt:::/acaaatthooooooo org/ 2021.emn 1121..mnlppmm. wwwpe-mainnnnn....eeeeemmammmaie 303/.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q. V., and Zhou, D. Chain-of- thought prompting elicits reasoning in large language models. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.), Ad- vances in Neural Information Processing Systems, volume 35, pp. 24824-24837. Curran Associates, Inc., 2022b. URL https::/prroceeding www.iingggnneuurpeee cc/paper_ htaapppr_fieeesppper/22//////// aa50991332eccfff5sffff133aaa...
Voita, E., Ferrando, J., and Nalmpantis, C. Neu- rons in large language models: Dead, n-gram, po- sitional. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Findings of the Association for Computa- tional Linguistics: ACI. 2024, pp. 1288-1301. Bangkok,
Paper-Conference.ppds
15

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Weir, N., Clark, P., and Van Durme, B. Nellie: A neuro- symbolic inference engine for grounded, compositional, and explainable reasoning. In Larson, K. (ed.), Pro- ceedings of the Thirdy-Third International Joint Con- ference on Artificial Intelligence, IJCAI-24, pp. 3602- 3612. International Joint Conferences on Artificial Intel- ligence Organization, 8 2024. doi: 10.24963/jjcii.2224 399. URL https: /tw.coiiio//// wtgg10..4963/1je.. 2024 / 399. Main Track.
tics. doi: 10.18653W//2222:.ccooogg. URL https: //aclantholoooeoeoooo hww.:/atwttoolll.oo...oeeeme attp://..............1..a.a 720/-
Yu, F., Zhang, H., Tiwari, P., and Wang, B. Natural language reasoning. a survey. ACM Comput. Surv, may 2024. ISSN 0360-0300. doi: 10.1 145/666419, URL https: / //doi aeooioorg/11 10..0/11133360 466446411116 194
Zach, R. Logic in mathematics and computer science, 2024.
URL https::tp//......... http//axxvvvvv/////v////////etvl wor.iioorrrrreeee..e... abs/ 2044 09033.
Welch, B. L. The generalization of 'Student's' problem when several different population variances are involved. Biometrika, 34(1-2):28-35, 35, 01 1947. ISSN 0006-3444. doi: 10. 10.1093/biommet3341-22222 URL https: /doii org http:/..1993/biometteo 1.2.2881........
Zhao, 7. and Aletras, N. Incorporating attribution impor- tance for improving faithfulness metrics. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61 Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pp. 4732- 4745, Toronto, Canada, July 2023. Association for Com- putational Linguistics. doi: 10. 10.18663/w1/2022.aclllooomm 261. URL https:: /ttp:s://alaatttt ogy. ogg.og//2233 acl 2aa-long.2200 261/
Wolf, T., Debut, L., Sanh, V., Chaumond, J.. Delangue, C., Moi, A., Cistac, P. Rault, T., Louf, R., Funtow- icz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C.,, Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the- art natural language processing. In Liu, Q. and Schlangen, D. (eds.), Proceedings of the 2020 Confer- on Empirical Methods in Natural Language Process- ing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: URL https:// acl aclanthology org wwn 6/.
Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. Calibrate before use: Improving few-shot performance of language models. In Meila, M. and Zhang, T. (eds.), Proceedings of the 38th International Conference on Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12697-12706. PMLR, 18-24 Jul 2021. htt. http://///coooooeeeeeeccoee ml press/ v139/ ht////haoll.....
Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang. B., Kim, N., Andreas, J., and Kim, Y. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Duh, K., Gomez, H., and Bethard, S. (eds.), Proceedings of the 2024 Con- ference of the North American Chapter of the Associa- tion for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1819-1862, Mexico City, Mexico, June 2024. Association for Compu- tational Linguistics. doi: 10. 10.18655/ 102. URL. https: : naacl long.o....... 102/.
Zhao, Z., Chrysostomou, G., Bontcheva, K., and Aletras, N. On the impact of temporal concept drift on model explana- tions. Goldberg, Y., Kozareva, Z., and Zhang, Y. (eds.), Findings of the Association for Computational Linguis- tics: EMNLP 2022, pp. 4039 4054, Abu Dhabi, United Arab Emirates, December 2022. Association for Com- putational Linguistics. doi: 10. 10.1865311//2222.mmg emnlp.298. URL https: http//aaaaaatloooo..o.ooo 2022. f
ac antho wwwphhollloooo..g.////o//. org/2024.
Xu, F., Lin, Q., Han, J., Zhao, T., Liu, J., and Cambria, E. Are large language models really good logical rea- soners? comprehensive evaluation and beyond. IEEE Transactions on Knowledge and Data Engineering, 37(4): 1620-1634, 2025. doi: 10. 1009/TKDD..202555556660
Xu, J.. Fei, H., Pan, L., Liu, Q. Lee, M.-L,, and Hsu, W. Faithful logical reasoning via symbolic chain-of- thought. In Ku, L.-.., Martins, A., and Srikumar, V. (eds.). Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 13326-13365, Bangkok, Thailand, August 2024. Association for Computational Linguis-
16

---

RULEBREAKERS, Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
A. Additional Related Work
Further comparisons between RULEBREAKERS and existing datasets in logical reasoning. On the surface, RULEBREAKERS seems similar to logical reasoning datasets introduced in Parmar al. (2024) and Holliday et al. (2024) in that they also focus on single-step inference instead of long reasoning chains. It also seems similar other logical reasoning datasets such as LogieNLI (Tian et al., 2021) and ProntoQA (Saparov & He, 2023) in that sentences within the dataset are also syntactically straightforward and easy to represent in symbolic forms in formal logic.
However, as Section 2 has explained in detail, RULEBREAKERS has a fundamentally different nature and objectives from these logical reasoning datasets. It assesses LLMs' ability to utilize common sense and factual knowledge about the real world when reasoning, not just how rigidly they can apply logical rules. For this reason, our examples also do not introduce any fictional ontology, nonsensical entities or deliberately false premises (e.g. "All mammals are cats") unlike some logical reasoning datasets such as ProntoQA (Saparov & He, 2023) or Wu al. (2024).
Failure modes of LLMs. LLMs have exhibited surprisingly simple failure modes, such as being sensitive to premise order (Chen et al., 2024) and unable to infer that "B is A" upon learning that "A is 8" (Berglund et al., 2024). By examining LLMs' reasoning behavior in the context of rulebreakers, our work also sheds light on another potential limitation of these models: their propensity to "over-genee" overrgeneralize" certain logical rules learned from patterns in their training data, potentially applying them incorrectly or too rigidly in reasoning tasks.
Concurrent our work, Holliday et al. (2024) evaluated LLMs on reasoning problems in conditional and modal logic, including some "controversial" cases which we would also consider to be rulebreakers under our definition. However, there are key differences between their contributions and ours. Firstly from theoretical perspective, while they consider these cases as specific anomalies that are subject to competing theories in logic and philosophy, we draw on expertise in cognitive science to characterize these as part of a larger category of examples that demonstrate the general gap between formal logic and human-like reasoning. Accordingly, we draw attention to scenarios where even basic rules in propositional logic which they do not consider to be "controversial" (e.g. disjunctive syllogism) can produce problematic inferences when applied rigidly in reasoning with natural language. Moreover, while Holliday al. (2024) gathered data instances for each logical rule by handerafting one question and then generating dozens more in the same form with the help of LLMs, our study's novel design and use of rulebreaker templates enables us to generate instances at scale that are carefully controlled - this allows for more in-depth and rigorous analysis into LLMs' reasoning behavior with respect to these problems.
Table 5 provides an overview of the relevant studies conducted in cognitive science relating to what we have termed as "rulebreakers". We show how templates introduced by our RULEBREAKERS are abstracted from specific examples that were manually written and tested on human participants these studies.? 23
B. Categorical Entity Pairs in the RULEBREAKERS Dataset
Type
Instances
finch, robin, ostrich, parakeet, swan, pheasant, crow, falcon, goose, nightingale, pelican, hawk, puffin, peacock, flamingo, quail, gull, toucan, partridge, kingfisher
bird
fish
trout, crappie, pike, tuna, catfish, flounder, mullet, salmon, haddock, hake, eel, goldfish, cod fly, butterfly, ant, earwig, beetle, flea, firefly, stonefly, termite, wasp, bee, mayfly banjo, sitar, koto, samisen, zither, guitar, dulcimer, psaltery, violin, gusli, lute, harp, balalaika, cello, mandolin, viola
insect
stringed instrument
brass instrument
tuba, trumpet, euphonium, trombone, sousaphone, French horn, cornet, bugle, flugelhorn
woodwind instrument
clarinet, oboe, flute, piccolo, bassoon, bagpipe, shakuhachi, English horn tai chi, karate, aikido, kung fu, judo, tae kwon do, jiu jitsu, muay thai tennis, badminton, table tennis, squash, racquetball
martial art
racket sport
For example, "If Anne was born in Sweden, then she was not born in Stockholm" does not imply that Stockholm is not in Sweden rather, it only implies that Anne might be born somewhere else in Sweden, or in another country. 2eWe also carried out an informal inspection/amootaiion on subset of samples generated with our RULEBREAKERS templates, validate that they mirror the semantic patterns manually written examples used these prior studies.
17

---

Johnson-Laie Laird & Byrne, 2002 RULEBREAKERS Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Table Key studies rulebreakers in cognitive science and their connection to various templates we have introduced in RULE- BREAKERS dataset.
Study
Relevant examples from their dataset
No. of human sponding participants rulebreaker included in template(s) study MT (type, in- 28
Our corre-
Relevant finding
Quelhas et al., 2010
"If Ana drinks a juice, then she doesn't I drink an orange juice"; Manuel plays a game, then he doesn't football"
Participants generally avoid conclu- sions that factually contradict the premises, even where they can be de- rived by applying a logical rule (e.g. modus tollens). Instead, they con- clude that "nothing follows" from the premises.
stance)
Quelhas Johnson-Laird,
& "Andre is in Lisbon or he is in Portugal"; "Luis is cooking bass or he cook- ing fish"; "Luis is eating chicken or he eating meat"
DS (country, 80 city), DS (type, instance)
As above, participants generally avoid conclusions that factually con- tradict the premises, even where they can derived by applying a logical rule (disjunctive syllogism).
2017
"If Bill is in Brazil then he is not in Rio de Janeiro"; "If Ann is in the Hotel LaBlanc then she is not in the Champagne Suite"
MT (country, 41 city)
When participants are familiar with the entities mentioned in the premises, they are more likely to recognize and avoid factually contra- dicting conclusions, as compared to when they are unfamiliar with the entities.
C. Licenses of Datasets Used in Creating RULEBREAKERS
We list in Table 6 existing datasets that we used in creating RULEBREAKERS, and their respective licenses.
Table 6. Datasets used in creating RULEBREAKERS and their corresponding licenses
Dataset
Usage in RULEBREAKERS
License
Source of geographical entity pairs
WikiData
CCO-1. 1.0 Universal
(list of current-day countries and capital cities)
ConceptNet
Source of categorical entity pairs
CC BY-SA 4.0
Popular Names by
List of most common first names and
CCO-1 1.0 Universal
Country Dataset
corresponding pronouns around the world
D. Model Sizes
Table 7 lists the six open-sourced LLMs included in our study, and their respective number parameters.
E. Checking Models' Knowledge with Respect to Categorical Entity Pairs
Compared to the check for geographical (country, city) pairs, the check for categorical (type, instance) pairs elicited more varied responses from the models. While some of these responses did not precisely match the ground truth (i.c. the specific type that an entity belongs to according to our dataset), they are technically correct and demonstrate that the model has the relevant knowledge. For example, when prompted with "Complete the sentence: goose is a type of", some models responded not with "bird" (which is the ground truth type our dataset) but with "waterfowl", which is nonetheless correct,
18

---

RULEBREAKERS:
Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Table 7. Number of parameters of each open-sourced I.L.M used in study.
MODEL NAME
No. OF PARAMETERS
PHI-3-MINI- 1AAMMINI--8--IIII 28K-INSTRUCT
3.8 x 10"
PHI-3-MEDIUM- -ISTTRCCT
1.4 X 10th
3-8B-INSTRUCT
8.0 x 10
META-LLAMA-3-70R -
7.1 X 10"0
MISTRALL7B-IISTTRUCVV....3
7.3 x 10
GEMMA-2-27B-IT
2.7 x 10td
since waterfowl is a subspecies of birds.
Consequently, we perform an additional manual check and, on this basis, did not exclude any type-instance pairs as part of this filtering process.
F. Results Breakdown by Prompt Phrasing
As shown in Figure 5, the seven LLMs exhibit varying degrees of sensitivity to input prompt phrasings used in our study, We find that Sesta-lamaa--8B--nnstee I---ttrutttttt is the most sensitive this regard, with paired accuracy varying by over 0.4. On the other end the spectrum, we find that Phi-3-medium- (aretiuumm--k-iiiniie 128k-instru GPT-4o and Gemma-2-27b-it are the most robust to the prompt phrasing variations included in our study, although the latter are also the worst performing models in terms of paired accuracy.
An interesting observation is that we did not identify any particular correlation between model performance and the different verbs used in the phrasings. As shown in Figure 6, verbs that elicit the best (or worst) accuracy on rulebreakers and non-rulebreakers differ among the seven LLMs. This contrasts with an intuitive expectation that e.g. the word "support" might be eliciting softer reasoning by model or that, conversely, the words "entail" and "deduced", which are more often used in the context of logical reasoning, might be motivating the model to adhere more strictly to logical rules when reasoning (and therefore perform worse on rulebreakers). We leave a more detailed investigation of this phenomenon future work.
19

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human- like Reasoning
Model: Phi-3-mini- 128k- instruct
All
Question keyword, Answer options
MT, (country, city)
MT. (type, instance)
entail, Yes-No
DS, (country, city)
entail, True-False
DS, (lype, instance)
followFrom, Yes-No
Model: Phi-3-medium- 128k-instruct
followFrom, True-False
inferred, Yes-No
MT, (country, city)
inferred, True-False
MT, (type, instance)
DS, (country, city)
deduced, Yes-No
DS, (type, instance)
deduced, True-False
Model: Meta-l 1aaaa-3--B--nntmtee
support, Yes-No
support, True-False
MT. (country, city)
MT, (type, instance)
DS, (country, city)
DS. (type, instance)
Model: Meta--limaa3-700-nnsttr
MT, (country, city)
MT, (type, instance)
DS, (country, city)
DS, (lype, instance)
Model: Mistral-7B- 1--nstuct...
MT, (country, city)
MT, (type, instance)
DS, (country, city)
DS, (type, instance)
Model: 2-2m--t--27-it
MT. (country, city)
MT. (type, instance)
DS, (country, city)
DS, (lype, instance)
Model: gpl-4o-2024-114
MT. (country, city)
MT, (type, instance)
DS, (country, city)
000
DS, (type, instance)
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Paired accuracy
Figure 5. Breakdown of results with respect paired accuracy, by different question phrasings.
20

---

RULEBREAKERS:
Challenging LLMs at the Crossroads between Formal Logic and Human- like Reasoning
Model: Phi-3-mini- 128kiiniii12kkkktttrutttttt
non-rulebreakers
Question keyword, Answer options
rulebreakers
entail, Yes-No
Model: 1iit---mediumn118k-inntttt
entail, True-False
non-rulebreakers
followFrom, Yes-No
rulebreakers
*
followFrom, Truc-Fulse
Model: Meta-Llama- ----aamm-------s-treeee---
inferred, Yes-No
non-rulebreakers
inferred, True-False
rulebreakers.
deducod, Yes-No
Model:
non-rulebreakers
*
deduced, True-False
rulebreakers
support, Yes-No
support, Truc-False
Model: Mistral- 1B-nsrruuctt-
non-rulebreakers
rulebreakers
Model: gemma-2-27b-
non-rulebreakers
rulebreakers:
Model: gpt-4o-2024-11-20
non-rulebreakers
rulebreakers
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Accuracy
Figure 6. Breakdown of results with respect to rulebreaker and non non-rulebreaker accuracy, by different question phrasings.
21

---

RULEBREAKERS:
Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
G. Example Prompts
list example prompts used in experiment that are produced by applying the chat template specific to each open-weight LLM accessed on Hugging Face. Chat templates are applied using the apply chat template function documented in attps://huggingee ww/docs/coooooooocccoooo.....cco https//ttaaafoommmemoo meain/e//// htt/////cchhtteeeeeeeeeeeeaeaa hthhhhhtteeppaainnnnn
Phi-3-mini-128k-innsntud and Phi-3-mediuun-128k-iisss
<seer Premises: If Lív caught some kind of bird, then she did not catch a goose. Liv caught goose. \n
Conclusion: Liv did not catch any kind of bird.An
Do the Premises entail the Conclusion? Answer Yes No only. 
> < <lassistant
Meta-Llama-33 --a-a---------sss I-St-unttt and Meta-Llama-33 3-70B------------
id henuserrieen_hhhde___diiddd
Premises: Liv caught some kind of bird, then she did not catch goose. Lív caught goose. \n
Conclusion: Lív did not catch any kind of bird.An Do the Premises entail the Conclusion? Answer Yes or No <loot
Mistral-7B-inttructv0..33

 (INST] Premises: If Liv caught some kind of bird, then she did not catch a goose. Liv caught goose. \n Conclusion: Liv did not catch any kind of bird.An Do the Premi ses entail the Conclusion? Answer Yes or No only, 

Gemma-2-27b-it:
<bos><start_of beexxsttattoof_uunnuseett turn>user\n
Premises: If Liv caught some kind of bird, then she did not catch goose. Liv caught goose. An
Conclusion: Lív did not catch any kind of bird.An An
Do the Premises entail the Conclusion? Answer Yes No only <end. of of_nur>> <ftartt_o__tummnnddell \n
22

---

RULEBREAKERS, Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
H. Additional Instructions and Variation in Prompt Phrasing
In light of our results Section I, we further investigate whether prompting instructions that remind LLMs to utilize common sense and factual knowledge when reasoning improve their mediocre performance on RULEBREAKERS.
We do this in two different ways:
1. Baseline phrasing additional instructions. We include the bold instructions into each of the same 10 prompt phrasings described in Section 4.2:
[Does the Conclusion follow from the Premises?] When reasoning, you should take into account your factual knowledge about the real world and assess whether any relevant logical rules should be applied. Answer [Yes] or [No] only.
2. Alternative phrasing. Instead of including additional instructions, we use this alternative question phrasing:
Based on the provided context and commonsense knowledge, is the Conclusion correct? Answer Yes or No only.
Additionally, while using the alternative phrasing, we also prefix the two premises in each prompt with the bold wording, to test whether this improves models' ability to recognize the factual contradiction between the two premises (we refer to this condition as alternative phrasing + prefixed premises), i.e.:
Premises: Suppose we are told that [premise 1]. However, as a matter of fact, [premise 2].
Conclusion: [conclusion].
Based on the provided context and commonsense knowledge, is the Conclusion correct? Answer Yes or No only.
As shown in Table 8, using the alternative question phrasing in combination with adding prefixes to the premises improves the paired accuracy achieved by most mo with the most substantial gain being +24.13 (%) over the baseline condition for gemma-2-27b-4a However, paired accuracy of all models remain mediocre, ranging from 31.27 69.12 (Meta-Llama-3. ---lllma---B--Insttcc)l
to
Importantly, for the majority of the models, we often observe a trade-off in rulebreaker versus non-rulebreaker accuracy under different conditions. For example, while by +5.58 in non-rulebreaker accuracy compared to the baseline condition, the model suffers a drop of -13.62 in rulebreaker accuracy. Conversely, under the baseline phrasing + additional instructions condition, by +10.51 in rulebreaker accuracy but drops by -18.21 in non-rulebreaker accuracy. We find this "trading off" pattern with Phi-3-medium- waiieediiummmk---mmmme 128k-Instruct and Meta-LLlma---700-Innstt
under the alternative phrasing condition improves
Baaa-llama-B--Insttee
improves
respect to 111-3-mmni-122k-Insttee the challenge that RULEBREAKERS poses to current LLMs, these results also indicate the inherent tension between prompting models to reason according to what formal logic demands (i.e. robustly applying rules regardless of the semantic content of the propositions) and prompting them to reason in a more human-like and knowledge-in infforrde manner by taking semantic content, factual knowledge and common sense into account.
Whilst highlighting
I. Task Variation: Conclusion Generation
Inspired by similar experiment setups in Johnson-L & Byrne (2002) and Quelhas & Johnson-Laird (2017), instead of presenting both the premises and conclusions to LLMs and asking them to either accept or reject the conclusions (as we had done in our main experiment), we present only the premises and ask the LLMs to generate the conclusions themselves for both rulebreaker and non-rulebreaker scenarios. For example:
Premises: Lív caught some kind of [if non-rulebreaker: insect if rulebreaker: bird], then she did not catch a goose. Lív caught goose.
What conclusion, if any, follows from the Premises? If you think nothing follows from the Premises, answer 'Nothing follows'. Keep your response concise.
The evaluation of GPT-4o on the alternative phrasing prefixed premises condition was carried out March 2025.
23

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human- like Reasoning
Table Comparison models' accuracy results (paired accuracy (T), rulebreaker accuracy a), non non-rulebreaker accuracy (Tp*) in %) across four conditions: (a) baseline question phrasings (as described in Scction 4.2); (b) baseline question phrasings with additional instructions appended: (c) alternative question phrasing: and (d) alternative question phrasing with prefixed premises. Highest score in each row highlighted in bold. Accuracy gain/loss of each condition (compared against the baseline) provided in brackets.
ALTERNATIVE
BASELINE PHRASING
BASELINE
ALTERNATIVE
PHRASING
+ ADDITIONAL
PHRASING
PHRASIIG
PREFIXED
INSTRUCTIONS
PREMISES
20.75
22.30 (+1.55)
24.56 (+3.81)
28.72 (+7.97)
PHI-3-MINI- 128K-INSTRUCT
TDR
21.07
22.70 (+1.63)
25.12 (44.05)
29.34 (+8.27)
TDx
99.68
99.59 (-0.09)
99.44 (-0.24)
99.37 (-0.31)
29.19
32.61 (+3.42)
29.14 (-0.05)
44.66 (+15,47)
PHI-3-MEDIUM- 2IS-INSTRUCT
TD
38.19
42.49 (+4.30)
35.03 (-3.16)
44.87 (+6.68)
TDN
89.63
89.11 (-0.52)
92.97 (+3.34)
99.73 (+10.1)
60.92
52.89 (-8.03)
65.94 (+5.02)
69.12 (+8.20)
META-LLAMA-3-
TD
73.58
84.09 (+10.51)
78.32 (+4.74)
88.22 (+14.64)
8B IB-INSTRUCT
TDx
86.46
68.25 (-18.21)
87.36 (+0.90)
80.53 (-5.93)
49.71
51.95 (+2.24)
59.39 (49.68)
56.19 (+6.48)
META-LLAMA-3- TOB-INSTRUCT
TDR
49.73
52.03 (+2.30)
59.53 (49.80)
56.40 (+6.67)
Tpx
99.98
99.91 (-0.07)
99.86 (-0.12)
99.79 (-0.19)
47.64
47.08 (-0.56)
39.53 (-8.11)
40.09 (-7.55)
MISTRAL-7H
TDR
56.28
57.13 (+0.85)
42.66 (-13.62)
55.12 (-1.16)
INSTRUCT-VO.3
TDN
90.99
89.57 (-1.42)
96.57 (+5.58)
84.96 (-6.03)
7.14
6.99 (-0.15)
8.65 (+1.51)
31.27 (+24.13)
GFMMA-2-27B-IT
TD
7.14
6.99 (-0.15)
8.65 (+1.51)
31.28 (+24.14)
TDiN
100.00
100.00 (+/-0)
100.00 (+/-0)
99.99 (-0.01)
15.06
13.91 (-1.15)
19.78 (+4.72)
31.92 (+16.86)
GPT-40
Tn
15.06
13.91 (-1.15)
19.78 (+4.72)
31.92 (+16.86)
TDN
100.00
99.99 (-0.01)
100.00 (+/-0)
100.00 (+/-0)
24

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like like Reasoning
In the non-rulebreaker case, the answer which we consider correct and expected of typical human reasoners would be "Lív did not catch ar insect", since it is uncontentious application of modus tollens; by contrast, we consider the answer "nothing follows" as incorrect.
In the rulebreaker case, the conclusion we consider correct would be "nothing follows". as the premises are factually inconsistent with each other (since goose is in fact a type of bird); by contrast, we consider the answer "Liv did not catch a bird" as incorrect, since it factually contradicts the second premise. As our main experiment, this again reflects the views and findings in existing cognitive science literature (Quelhas et al., 2010; Khemlani & Johnson- -aird, 2019, etc.) that, when faced with what they recognize to be inconsistent premises. typical human reasoners are not expected just conclude that any conclusion whatsoever follows. By contrast, in classical logic, any conclusion follows validly from set of contradictory premises.
Similarly, for the disjunctive syllogism instances, e.g. "Lív caught either a goose or some kind of [insect/bird]. Lív did not catch [an insect/a birdj.", , the correct conclusion would be "Lív caught a goose" in the non-rulebreaker case, and "nothing follows" in the rulebreaker case.
We prompt each model generate using greedy decoding and setting a maximum generated token limit of 50 tokens. We then pattern- pattern-match their outputs against expected correct and incorrect answers. Where mattern-matching fails to map an output to either a correct or incorrect answer, we categorize it as "unparsed".
We compute the frequency of each LLM's correct, incorrect and unparsed responses to all rulebreakers and non-rulebreakers in RULEBREAKERS Additionally, we apply the same approach in calculating paired accuracy to compute the frequency of correct, incorrect and unparsed responses to rulebreaker and non-rulebreaker pairs (see Section 3.4). If a model's response to both the rulebreaker and non-rulebreaker prompt a pair are correct, we categorize the paired response as correct. If at least one response is unparsed, we categorize the paired response as unparsed. In all other cases, we categorize the paired response as incorrect.
As shown in Figure 7, the results of this additional experiment generally conform to the patterns we had identified in the results of our main experiment in Section 5.1. Most models still perform better with respect to non-rulebreakers than rulebreakers. As in our main experiment, Phi-3-mini- 1iii-122k-iinn -ask-instuuct, gemma-2-27b-it and apt-44--024-111 1-20 exhibit behavior that suggests the models' over-rigidly applying logical rules in generating conclusions, thereby performing particularly well on non-uulebssskers but particularly poor on rulebreakers.
Notable exceptions, however, are Phi-3-medium- 128k-instruct and Mistral-7B-Innstuct-00..33 Unlike their behavior in main experiment, both these models perform better on rulebreakers than non-rulebreakers in this setup. In particular, we observe that Mistral-77 77-IIntruct--O.. exhibits a strong bias towards outputting followss which results in near perfect accuracy on rulebreakers but to the detriment of its performance on non-rulebreakers. As a result, its proportion of correct paired responses is even lower than the paired accuracy it achieved in our main experiment. Whilst highlighting an interesting phenomenon that merits future investigation, these results also demonstrate the robustness of our paired accuracy metric against any abrupt shifts in models' preferences due variations in task and prompt phrasing.
We also conduct follow-up experiment sampling as opposed to greedy decoding during inference with different temperature settings. As shown in Table 9, the results obtained from using sampling with varying temperatures do not substantially deviate from our baseline. Increasing temperature exhibits mixed effects with respect to different models, harming the performance of 1amma----B-Instruuee and Miistral7B-Insttreee Instruct-v0.3 but improving the performance of Phi-3-mini-122 228innstruct (which has performed the worst under the baseline condition compared to the other two models).
subset of models to validate that our findings above are robust to the use of
J. Further Implementation Details
All open-sourced models used in our study were accessed and downloaded through Hugging Face: https: huggingfar acce. co co/mmooe mo.els.
We ran each model on a single NVIDIA A100 GPU, except for Meta-Llama-3-700 3--00--Insttuue which we ran on three.
Running inference on the full RULEBREAKERS dataset permuted by 10 prompt phrasing variations took approximately
In extracting the LLMs' answers by pattern-matching matching, we also allow for the article in this position to be substituted with "any", "any kind of" or "some kind of", since these produce the same meaning.
25

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like like Reasoning
Correct
Unparsed
Incorrect
Model: Phi-3-mini- 1aaad-11kkkiintttuu
Rulebreakers
Paired
Non-rulebreakers
Model: Phi-3-medium-- 128k-instruct
Rulebreukers
Paired
Non-rulebrenkers
Model: Mctaa-llama-3-8B 8B-Instrct
Rulebrenkers
Paired
Non-rulebreakers
Model: 3---laam------maee
Rulebreakers
Paired
Non-rulebreakers
Model: Mistral-7B-Ins 1B-Instruct--0.3
Rulebreakers
Puired
Non-rulebreukerss
Model: gemma-2-2272 2--2-it
Rulebrenkers
Paired
Non-rulebreakers
Model: gpt-4o-2024-1 11-20
Rulehreakers
Paired
Non-rulebreakers
0.2
0.4
0.6
0.8
Proportion of responses
Figure 7. Proportion of correct, unparsed and incorrect responses extracted from LLMs' outputs when they provided with only the premises and instructed to generate conclusions.
Table Percentage of corrected paired responses in conclusion generation by using random sampling with different temperatures
PHI-3-MINI- 128K-INSTRUCT
LLAMA-3-
MISTRAL-7B- INSTRUCT-VO.3
TEMPERATURE
8B- SB-INSTRUCT
N/A GREEDY DECODING
1.23
32.44
19.50
(BASELINE)
0.1
1.47
32.34
19.26
0.5
4.24
31.76
18.68
1.0
7.59
28.51
17.12
26

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
one hour per model with batch size of 128, except for Gemma-2-27b-it which took four hours with a batch size of 128, and Metta-llamaa--70--nnnuu which took 10 hours with batch size of 64.
K. Further Experiments and Discussion regarding Logic-enhanced Models
While revealing a reasoning blind spot in current general-purpose models, our findings also point to the potential trade-off in methods that rely on formal logic to enhance these models' general reasoning abilities.
For example, intuitively, we would expect that models specifically fine-tuned on logical reasoning datasets would exhibit an even more pronounced tendency to over-rigidly apply formal logic. In addition, some hybrid systems are explicitly guided or constrained to reason according to formal logic. This is done, for example, by using LLMs to parse natural language statements into symbolic logical forms, which are then either processed by an external symbolic solver (as in the case of, e.g., LogicLM (Pan et al., 2023)) or fed back into the model in the subsequent stages as part of a multi-turn prompting strategy (e.g. in SymbolicCoT (Xu et al., 2024)). However, the problem RULEBREAKERS poses to these systems is precisely that each rulebreaker and non-rulebreaker a pair share the same surface logical form, such that the conclusion logically valid given the premises in propositional logic. Thus, even if the parsing is correct, the system would theoretically accept the conclusion in both non-rulebreakers (which would be correct) and non-rulebreakers (which would be incorrect). hence leading to low paired accuracy.
To validate these theoretical arguments, we evaluate an existing Llama-3. 1.taa-..1-BB-.... been fine-tuned on dataset for propositional logic reasoning, and compare its performance on RULEBREAKERS against the baseline model before fine-tuning. We also evaluate the performance gpt-3,5-turbo on RULEBREAKERS across four conditions: (a) directly prompting the model for an answer (baseline); (b) using chain-of-thoughh (CoT) prompting (Wei al., 2022b); (c) using LogicLM; and (d) using SymbolicCoT.
model on Hugging Face that has
We prompt the models to answer the question under the alternative phrasing prefixed premises condition introduced in Appendix H, using greedy decoding. With respect to -ttrro, we use the LogicLM authors' implementation (including their default settings)* for conditions (a), (b) and (c); and that of the SymbolicCoT authors 28 for condition (d). As in the LogicI.M authors' implementation, where the model fails to output valid logical forms that can be successfully executed, it falls back to the answer outputs under the CoT condition.
As shown in Table 10, the fine-tuned Llama-3.1 1.---...-----tt--e model outperforms the baseline model in accepting and generating the correct conclusions in non-rulebreaker cases. However, we expected, its performance on rulebreakers is substantially lower than the baseline model, indicating that the fine-tuned model is over-rigidly applying logical rules without regard to the semantic content of the premises.
With respect to gpt-3.5-turbo Table 10 shows that using LogicI. and SymbolicCoTT boosts the model's performance on non-rulebreakers at the expense of its performance on rulebreakers, leading as expected to worse paired accuracy than the baseline. We also observe that using CoT has a similar trade-off in boosting non-rulebreaker performance (to a level comparable to using LogicLM and SymbolicCoT), but the drop in rulebreaker performance much less drastic. Nonetheless, these results show that the inherent challenge remains in prompting models to reason robustly in knowledge-informed manner.
L. Qualitative Analysis
With respect to our hypothesis (1), we observe that the most frequent country-city pairs about which LLMSs answer incorrectly tend to be those that are considered less well-known or likely to appear less frequently in the models' training corpora. These include, for example, capital cities of relatively small island countries (such as Funafuti in Tuvalu and Ngerulmud in the Republic of Palau), and those of smaller African countries relative to their neighbours (such Gitega in Burundi, Bissau in Guinea-Bissau and Maseru in Lesotho). Conversely, the most frequent country-city pairs that these models answer correctly about tend to be those that are relatively better known or larger in size. Examples include New Delhi in India, Madrid in Spain and Tokyo in Japan.
>https: http://////gggggaaaaeeeeeeeeeeee peatts::/githhue http:///eeeeeeeeeeeeeaaaaaaggooa prop prop_logic_tt
#https:/ // con http:/......eeen//////////oo///
27

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Table 10. Results of logic-enhanced models. "Fxecutable subset" refers to the subset of RULEBREAKERS instances where gpt-3.5-turbe has parsed the premises and conclusion into valid logical forms that were successfully evaluated by the symbolic solver. Paired accuracy (7) for the executable subset is not computed since the model can output valid logical forms non-rulebreaker not the rulebreaker, or vice versa, within prompt pair.
T
TDR
TDX
LLAMA-3. BASELINE MODEL
60.68
65.68
94.67
WITH
19.03
20.67
97.15
LOGIC DATASET
GPT-3.5-TURBO
BASELINE MODEL
50.80
90.88
59.92
COT (2-SHOT)
65.36
71.78
91.75
LOGICLM
30.81
32.56
94.59
(EXECUTABLE SUBSET)
5.98
95.07
SYMBOLICCOT
9.53
10.47
91.33
Interestingly, LLMs seem to do particularly well on cities that also contain their country names (e.g. Mexico City in Mexico, Kuwait City in Kuwait). We speculate that this might be because these entity pairs make it easier for models to recognize the implicit contradiction between the factual premise and the conclusion in rulebreakers on a symbolic level (i.e. between "Anne is in [XYZ] City" and "Anne is not in (XYZ]")-
Furthermore, this "symbolic overlap" factor may also hint at a possible explanation for the trend we observed in Section 5.1 that models generally perform better on the DS subset rather than the MT subset of RULEBREAKERS. Intuitively, for model to apply logical rule, it needs recognize where segments of the natural language premises negations of one another and abstract these segments accordingly e.g. into "B" and "not D", This mapping is arguably more obvious and straightforward in MT cases than in DS ones. For example, in MT (country, city) cases, the bold and underlined parts form a clear negation pair:
"If Anne is in Sweden, then she is not in Stockholm. Anne is in Stockholm." (I then not B. B.)
By comparison, the negation pair is less obvious in DS (country, city) cases:
"Anne is either in Stockholm or somewhere in Sweden. Anne is not in Sweden." (A or B. Not B.)
We conjecture that this may make it more difficult for models to map DS premises and conclusions to their logical forms, compared to MT ones. As a result, models may be less likely to rigidly apply logical rules, hence avoiding incorrect conclusions in DS rulebreaker cases. Nevertheless, as with other factors such as entity familiarity, the precise impact of this simillrrrtssssliittttty.... effect" on model behavior may vary across models, potentially contributing to the performance variation we observe.
M. Unnormalized Score Ratios
Table 1l shows variation of Table 4 that lists the importance score ratios computed without normalizing by the number of tokens aggregated in each of the two premises. As expected, since the second premise always contains fewer tokens than the first, the score ratios listed in Table 11 are all lower than their counterparts in Table 4. Nonetheless, we do not observe any substantive deviations from the patterns we have identified with respect to the normalized score ratios Table This indicates that our findings are not substantially affected by variations in how prompts in our experiment are tokenized for each model.
N. Counting and Comparing Activated Neurons in Rulebreakers versus Non-rulebreakers
To better understand how LLMs internally distinguish rulebreakers from non-rulebreakers, we perform an additional experiment inspired by Voita et al. (2024), focusing on individual neurons' activation values between the two linear layers of the feedforward neural network (FFN) within each model layer's Transformer block.
28

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human-like Reasoning
Table 11. Mean unnormalized ratios (with sd) of each L.M.. The paired accuracy (7) from Section 5.1 is provided for the analysis. The greater the ratio, the more the model attends to the factual context.
SCORE RATIO
SCORE RATIO (ATTENTION)
(INPUT X GRADIENT)
PHI-3-MINI- M-MIIIII12K-ISTRRIII
0,412 (0.152)
0.572 (0.125)
0.208
PHI-3-MEDIUM- INS-IUTTT
0.358 (0.090)
0.589 (0.320)
0.292
0,441 (0.109)
0.991 (0.184)
0.609
META-LLAMA-3-700B INNSRUCTT
0.445 (0.119)
0.877 (0.165)
0,497
MIITTRAL--B-INSTRUCCT-NN
0.405 (0.111)
0.771 (0.169)
0.476
GEMMA-2-27B-IT
0.412 (0.095)
0.815 (0.116)
0.071
Specifically, given imput prompt, count the number of activation values that are greater than zero summed over all token positions, with respect to each layer within a model.
We divide all prompts used in our main experiment described in Section 4.2 into four categories: rulebreakers versus non-rulebreakers that an LLM answered correctly versus incorrectly. We then compute the average number of activated neurons across all prompts within each of the four categories.
As the results in Figure show, models differ widely in their activation counts with respect each of the four categories. On one end of the spectrum, we see near perfect overlap in activated counts across all layers in Gemma-2-27b-ir in other words, there is no substantial difference in the average number of neurons activated by rulebreaker or non-rulebreaker prompts (regardless of whether or not the model has answered the prompt correctly). On the other end, with respect to Balllllaaa-------nttttttt
we observe that the four categories of prompts are clearly (and consistently across all layers) distinguishable in terms of activated counts. Somewhat surprisingly, in three out of six models (Phi-3-mini- innk-inttruct Llama-3-70B-Insse
Meta-LLamm----8 3-nt-tsrutt and activate more neurons than correctly answered ones. These results suggest non-trivial relationship between model's neuron activations and its responses to rulebreakers and non-rulebreakers, presenting an avenue for future investigation.
incorrectly answered rulebreakers and runrrbebrraasrss appear
We select threshold for consistency (whilst recognizing that these values also be less than zero), since Gemma-2-22h-in
uses GeL.U activation function estimated with tanh, whereas other L.MSs our study use SiL.U (Hendrycks & Gimpel, 2023).
29

---

RULEBREAKERS: Challenging LLMs at the Crossroads between Formal Logic and Human- like Reasoning
Carractly meverod ruldreaken
Insemelly moweral niddveakens
Cermally anewenl noo-ruldreaaer
Inmmeelly ansuree noneuldseakke
Phi-3-mini- 1aiini--2kkinstruue
Phi-3-medium- 128k-instruct
300
450
200
400
Meta-Llama-3 Baatmna----B-maan I-Strnnst
Avg bo. of neurons activate ('000)
450
1150
1100
1050
350
900
850
300
300
250
600
200
150
10
20
25
10
15
20
25
40
45
60
70
75
Mistral-7B Bestall-B-Instteee T-Insruucct--O.
semma-2-276-24
350
550
500
450
400
359
250
300
250
200
150
150
755
layer
Figure 8. Average number of neurons activated per model layer, with respect prompts belonging each of four categories (rulchreaker vs ron-bbeebrea,er. correctly vs incorrectly answered by each model).
30


Reviews:
[{"review_id": "4RNqkQA8i9", "reviewer": "Reviewer_twqU", "summary": "The authors introduce RULEBREAKERS, a dataset designed to assess LLMs' ability to reason using common sense and factual knowledge rather than strictly following formal logic. The experimental evaluation proposed in the paper spans over seven LLMs, and its findings uncover a notable weakness in these models' ability to identify and reject conclusions derived from propositional logic that are factually inconsistent with the premises. The paper is primarily well-written, easy to understand, and follow (excluding a few paragraphs). The results are interesting and highlight an open challenge that LLMs need to face. Meanwhile, the experimental settings could be extended to take into account large LLMs and hybrid approaches to test the impact of model complexity/size on its reasoning capability. Overall, the paper feels relevant. Thus, I consider the paper slightly above the ICML conference's acceptance threshold.\n\n\n## update after rebuttal\n\nI thank the authors for the clear and detailed rebuttal they provided. Overall, I think the authors covered some of my doubts, while leaving also some open questions. Therefore, I am considering the paper to be generally solid and my feeling concerning it is still positive. Therefore, I am keeping my original review score (which was already positive).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer twqU for commending that our “**methods are reliable**”, our “**dataset represents a valid addition to the LLM community**”, our experiments are “**sound**”, and results “**highlight an open challenge**”. We are glad they found our paper “**primarily well-written**”, echoing Reviewer fmAM.\n\n**Suggestion 1 (S1): “mention how failure could impact LLM real-world applications”**\n\nLLMs’ ability to detect and handle knowledge conflicts is crucial to ensure robustness and guard against misinformation. Suppose a model is given this context, like DS-type rulebreakers in our dataset:``“The patient was informed that the treatment for their illness is available either in Berlin or somewhere in Germany. However, as it turns out, the treatment is not available anywhere in Germany.”``\n\nModels rigidly applying logic would incorrectly conclude that “the treatment is available in Berlin”, contradicting the fact in context. Instead, an ideal model should recognize that, since Berlin is in Germany, the treatment is not available in Berlin either.\n\nTo demonstrate, we add the **bold** phrases to premises in our prompts (\"**Suppose we are told that** Anne is either in Stockholm or somewhere in Sweden. **However, as a matter of fact,** Anne is not in Sweden\"), but found that models' performances on RULEBREAKERS remain mediocre (paired accuracy ranging 0.29 to 0.69). Due to space, we will add and discuss these results in the final version.\n\n**S2/Weakness 1 (W1): “Some findings not clearly presented: [Section 5.2, Section 6, Fig 5]”**\n\nWe will replace Fig 5 with a table and revise Section 5.2 to improve clarity. Specifically, we will make clear that we use a model’s output probabilities as a measure of its confidence. We will also add to Section 6 the clarification we set out in our Q2 response to Reviewer QkHy.\n\n**W2: “selected only small/medium-sized LLMs”/Question 1 (Q1): “what would authors expect evaluating larger models?”**\n\nWe included a 70B model and the large model GPT-4o. We do not observe a correlation between model size and performance on RULEBREAKERS: GPT-4o underperforms most models, and Llama-3-70B-Instruct underperforms its 8B variant. \nThus, we consider that our results with selected model sizes **already make a strong point regarding limitations of current LLMs, including state-of-the-art GPT-4o**.\n\n**Q2: “Did authors consider impact of temperature when prompting?”/S3: “add experiments [re] temperature”**\n\nIn our main experiment, we run only one forward pass to extract the most probable token without randomly sampling, so temperature is not used.\n\nIn Appendix G, we prompt LLMs to generate conclusions by greedy decoding. Our results already make a strong case, but we follow the Reviewer’s suggestion and test a subset of models. As Table 1 shows, random sampling with temperature yielded mixed effects for different models but did not alter our findings. We will add these results in the final version.\n\n| **Temperature** | **Phi-3-mini-128k-Instruct** | **Llama-3-8B-Instruct** | **Mistral-7B-Instruct-v0.3** |\n|---|---|---|---|\n| Not set - greedy decoding (baseline) | 1.23 | **32.44** | **19.50** |\n| 0.1 | 1.47 | 32.34 | 19.26 |\n| 0.5 | 4.24 | 31.76 | 18.68 |\n| 1.0 | **7.59** | 28.51 | 17.12 |\nTable 1. % of correct paired responses\n\n**Q3: “performance of same LLM varies across rulebreaker types...no correlation between performance of different models on different types...any insight on why?”/S4: “add more insights on why these behaviours emerge”**\n\nWe appreciate and agree with these observations. We expect models may differ in specific traits: some may be better at reasoning with geographical entities; others worse at recognizing some forms of negated premises. However, the fact that performance varies unpredictably shows that models do not handle these reasoning problems robustly or consistently. Rather, they may be affected by biases or quirks in the training data.\n\n**Q4: “additional experiments implementing hybrid LLM approaches?”/W3: “do not check how these hybrid models perform”**\n\nOur study aimed to expose a blind spot even in general-purpose models. We add a model specifically fine-tuned for logical reasoning (see our Q3 response to Reviewer QkHy), which enhances our findings and reflects our motivation for introducing “rulebreakers”.\n\nHybrid systems like LogicLM [1] use LLMs to parse natural language statements into logical form. However, **each rulebreaker and non-rulebreaker in a pair share the same surface form and are both valid in propositional logic**. Thus, even if the parsing is correct, the system would theoretically accept the conclusion in both rulebreakers and non-rulebreakers, hence scoring 0 in paired accuracy. We will add this discussion to the final version.\n\n**S5: “better representation of Fig 6”**\n\nPer the Reviewer's suggestion, we will replace Fig 6 with a table of mean familiarity values (and std) for clarity.\n\n[1] Pan et al. https://aclanthology.org/2023.findings-emnlp.248", "claims_and_evidence": "The authors’ claims are largely supported by empirical evidence. Although the experimental evaluation could be extended to take into account larger LLMs or hybrid approaches, the performance of the tested models seems to back the authors’ claims in most experiments. Therefore, I believe that the authors' claims are sound.", "methods_and_evaluation_criteria": "The methodology proposed by the authors to test the performance of LLMs on rule-breakers is reasonable. The evaluation metrics considered highlight different aspects of the LLM's capabilities, allowing the authors to identify a few different relevant challenges related to rule-breakers and LLMs. Overall, the methods and evaluation criteria are reliable.", "theoretical_claims": "The authors did not provide any theoretical claims given that the paper focuses on experimental analysis only. Therefore, this question is not applicable.", "experimental_designs_or_analyses": "The evaluation of the experiments proposed by the authors is sound and extensive enough to back up their claims. The rule-breakers are tested over a few different LLMs, thus giving an overview of general LLM limitations. However, the experimental evaluation could still be extended to larger LLMs to identify if the limitations highlighted by the authors are connected to model size or if an intrinsic issue exists with how the LLM thought process is brought about.", "relation_to_broader_scientific_literature": "The authors did not mention how the failure of LLMs to recognize and handle rule-breakers could impact their application from a real-world perspective.", "essential_references_not_discussed": "I don’t believe there are any essential references missing. However, given that I’m not an expert of the topic, I might be wrong.", "other_strengths_and_weaknesses": "STRENGTHS:\n- The paper tackles an interesting research field, analyzing the ability of LLM to reason in a human-like manner\n- The rue-breakers proposed by the paper seem to be constructed in a sound manner\n- The obtained dataset is extensive and represents a valid addition to the LLM community\n- The experimental evaluation highlights some interesting findings on LLM’s behaviour on such rule-breakers\n\nWEAKNESSES:\n- Some findings are not clearly presented by the authors. Such as section 5.2 or the claim on the familiarity effect in section 6.\n- The authors selected only small/medium-sized LLMs. Thus, the validity of the paper’s findings may be limited.\n- The authors mention that approaches exist that incorporate logic-based training metrics or constraints during inference in LLMs. Still, they do not check how these hybrid models perform in the rule-breaker settings.", "other_comments_or_suggestions": "- Although I understand that the model selection process was bounded by resource limitations, leveraging only small/medium-sized LLMs for the experimental evaluation represents a relevant issue for the paper. Indeed, it would be interesting to understand if larger, more complex models are capable of performing better on rule-breakers. As it stands, the experimental evaluation is a bit undersized with respect to the relevance of the task considered.\n- The temperature parameter used by the LLM represents a very relevant hyper-parameter to study for validating the output. However, the authors fail to mention anything about such a parameter in the paper (unless I failed to read it properly). The behaviour of an LLM when varying the temperature parameter may largely vary, thus altering the performance of the obtained output. Therefore, I would suggest the authors add some other experiments to show what happens at the variation of the temperature.\n- Looking at Figure 3, it seems that the performance of the same LLM largely varies across the different types of rule-breakers. Also, there does not seem to be a precise correlation between the performance of the different models on the different types of rule breakers. For example, Phi3 medium performs much better on MT (type, instance) than on DS (country, city). Meanwhile, Llama3 70B is the exact opposite. However, the authors seem to fail to mention anything concerning this aspect in their discussion of the results. Therefore, I suggest the authors add more insights concerning why these behaviours emerge.\n- Figure 5 is not presented clearly. In my opinion, section 5.2 is a bit tricky to follow as the authors confused the readers when presenting the results. I suggest the authors rephrase it a bit and be more careful in their writing.\n- The authors mention, “If our hypothesis (1) is correct, we would expect models to have a higher “familiarity” with respect to prompt pairs in their “recognized” group, i.e. those that the model has answered correctly. As shown in Figure 6, this holds true for all LLMs except for Meta-Llama-3-8B-Instruct.”. However, from Figure 6, it seems like the opposite is true. This may be due to the authors plotting the results in the form of a box plot, which does not allow us to distinguish the nuanced variations among the red and green boxes well. From an outside perspective, it seems like the boxes are very much overlapping, thus invalidating the authors' statement. Assuming that the authors’ statement is indeed true, I would suggest that the authors find a better representation of the results in Figure 6. A simple table with a mean and standard deviation of the familiarity would be enough.\n- Since the authors mentioned that there exist approaches incorporating logic-based training metrics or constraints during inference in LLMs, it would be interesting to check how these hybrid models perform in the rule-breakers dataset.", "questions_for_authors": "- Could the authors discuss what they would expect to find out when applying the same experimental evaluation to larger models?\n- Did the authors consider the possible impact of the temperature parameter when prompting the LLMs in their experimental evaluation?\n- Looking at Figure 3, the performance of the same LLM largely varies across the different types of rule-breakers. Do the authors have any insight on why this happens? Also, there does not seem to be a precise correlation between the performance of the different models on the different types of rule breakers. For example, Phi3 medium performs much better on MT (type, instance) than on DS (country, city). Meanwhile, Llama3 70B is the exact opposite. Do the authors have any insight on why this is happening?\n- Could the authors provide additional experiments implementing hybrid LLM approaches incorporating logic-based training metrics or constraints during inference to test their performance on the rule-breakers dataset?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xCvnnWkaor", "reviewer": "Reviewer_QkHy", "summary": "This paper introduces RULEBREAKERS, a dataset specifically created to assess LLMs on reasoning scenarios that emphasize \"human-like reasoning\" over logic reasoning. The study demonstrates that state-of-the-art LLMs, including GPT-4o, frequently apply logical rules, which is inconsistent with human reasoning.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer QkHy for their helpful feedback and recognizing that our study was “**extensive**” in having “**compared multiple advanced LLMs**”.\n\n**Weakness 1 (W1): “human reasoning is undefined”/Question 1 (Q1): “how to define human-reasoning and why?”**\n\nWe will **replace references to “human reasoning” with “knowledge-informed reasoning”**. \n\nWe define reasoning to be “knowledge-informed” if a reasoner incorporates factual and commonsense knowledge in their reasoning process, and does not draw or accept conclusions that factually contradict any premises. Accordingly, a model that accepts the conclusions in our rulebreaker cases would fail to be “knowledge-informed”.\n\nOur definition is motivated by arguments in existing cognitive science literature that humans typically incorporate factual and commonsense knowledge to interpret natural language premises, and avoid drawing conclusions that, to their knowledge, factually contradict any premises. We discussed these in **lines 45-63 and 96-116** and will add more justifications for our definition.\n\nThis term also highlights our findings' implications for LLM use in knowledge-intensive tasks, as discussed in our S1 response to Reviewer twqU.\n\n**W2: “example in Fig 1” is not counterintuitive**\n\nThe example is commonly used in cognitive science works (e.g. [1], [2]) to demonstrate reasoning problems with “relevance conditional” statements. We will replace this with an example from our own dataset to better represent our rulebreakers.\n\n**W3: “dataset...was neither created nor validated by humans”** \n\nWe kindly refer to our S1/Q2 response to Reviewer fmAM.\n\n**W4: “Limited exploration into internal model mechanisms that cause failure”**\n\nPlease see **Appendix M**: we analyzed neuron activations in feedforward layers, complementing two potential causes diagnosed in Section 6. We welcome further suggestions on specific investigation methods.\n\n**Suggestion 1 (S1): “Potential effects of prompt phrasing…not deeply explored.”**\n\nWe refer to **Appendices D and F**: we analyzed the breakdown of our results by different phrasings, and tested more re-phrasing and additions to confirm that our findings are robust against prompt variations.\n\n**S2: “hypothesis regarding model over-generalization …could benefit from further empirical validation”**\n\nSee **Appendices F and G**: we validated our main results with further experiments, in addition to existing controls for models’ factual knowledge and prompt sensitivity (as above). \n\n**Q2: “Can you provide additional details on how model familiarity with entities impacts performance?”**\n\nTo clarify **lines 396-416**, we found that a model generally performs worse on prompts containing unfamiliar entities, as compared to prompts containing familiar ones. This is supported by our qualitative analysis in **Appendix I**. **This trend mirrors findings from cognitive science [3]** that when participants are familiar with entities in the premises, they are more likely to recognize and avoid factually contradicting conclusions, as compared to when they are not. \n\nSeparately, when comparing across models, we found that simply being familiar with entities does not guarantee good performance on RULEBREAKERS: while Gemma-2-27b-it is overall highly familiar with entities in our dataset, it scores among the poorest in paired accuracy. This shows that **a model can excel at recalling factual knowledge yet still struggle to apply that knowledge effectively in reasoning**. We will add this expanded explanation in the final version.\n\n**Q3: “Have experiments been considered with models specifically fine-tuned for logical reasoning tasks?”**\n\nWe did not do so, as our work initially aimed to highlight a blind spot that exists even in general-purpose models. We expect the behaviour of over-rigidly applying logic to be even more pronounced in such fine-tuned models. \n\nTo test this, we **evaluate a Llama-3.1-8B-Instruct** on Hugging Face [4] **fine-tuned on a dataset for propositional logic reasoning**, and compare its performance on RULEBREAKERS against the baseline model before fine-tuning.\n\n|  | **Main experiment setup (% accuracy)** |  |  | **Appendix G setup** |  |\n|---|---|---|---|---|---|\n|  | Paired  | Rulebreakers | Non-rulebreakers | % of correct conclusions generated in rulebreaker cases | % of correct conclusions generated in non-rulebreaker cases |\n| Baseline | **50.45** | **91.46** | 58.60 | **74.20** | 47.91 |\n| Fine-tuned | 11.42 | 32.93 | **75.48** | 0.40 | **65.75** |\n\nAs expected, the fine-tuned model performs better on non-rulebreakers but substantially worse on rulebreakers compared to the baseline. We will include a detailed discussion of these results in the final version.\n\n[1] Johnson-Laird. Mental models. Cambridge University Press, 1983.\n\n[2] Quelhas et al. https://doi.org/10.1080/17470210903536902\n\n[3] Johnson-Laird and Byrne. https://doi.org/10.1037/0033-295X.109.4.646\n\n[4] huggingface.co/ergotts/llama_3.1_8b_prop_logic_ft", "claims_and_evidence": "No. \"Human reasoning\" is undefined and highly ambiguous. A person with even basic logic training would not find the example in Figure 1 counterintuitive. Moreover, while the dataset is claimed to evaluate human-like reasoning, it was neither created nor validated by humans, making the claim unsupported.", "methods_and_evaluation_criteria": "No. See above.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "This work tries to connect cognitive science and LLM reasoning.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\nExtensive compared multiple advanced LLMs.\n\nWeaknesses:\n1. \"Human reasoning\" is undefined and highly ambiguous. A person with even basic logic training would not find the example in Figure 1 counterintuitive. Moreover, while the dataset is claimed to evaluate human-like reasoning, it was neither created nor validated by humans, making the claim unsupported.\n2. Limited exploration into internal model mechanisms that cause failure.", "other_comments_or_suggestions": "1. Potential effects of prompt phrasing on model performance are recognized but not deeply explored.\n2. The hypothesis regarding model over-generalization of logic rules is plausible but could benefit from further empirical validation.", "questions_for_authors": "1. How to define \"human-reasoning\" and why?\n2. Can you provide additional details on how model familiarity with entities impacts performance?\n3. Have experiments been considered with models specifically fine-tuned for logical reasoning tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YhFMjaMJLs", "reviewer": "Reviewer_fmAM", "summary": "The authors propose a new dataset for single step reasoning, which consists of pairs of premises and a conclusion, which are answered in a binary way. The premise and the conclusion always are true if one only follows the logical reasoning. However the pair is divided into \"rulebreakers\", where the reasoning contradicts common knowledge, and \"non-rulebreakers\", where the reasoning is consistent with such knowledge. The dataset consists of 12,800 of such pairs, which generated based on a template, that the authors define in their paper.\n\nThe authors then proceed to evaluate six open source large language model as well as GPT-4o. They evaluate their accuracy on getting the pairs completely correct as well as their performance on the individual sets. They also conduct experiments with two different types of logical reasoning and the model confidence in their outputs for the open source models.\n\nThe authors conduct an analysis in two areas. Firstly, whether the performance of the models depends to a certain degree on their familiarity with the respective area of knowledge required for a given question, where they find the key insight that a model might be excellent at knowledge retrieval and/or logically reasoning, but can not recognize that a conclusion is inconsistent with factual knowledge. They also investigate whether models pay attention to the right information in the premises and here the insights are not conclusive, but in general the models that pay more attention to the factual information in the second premise also produce better results overall.\n\n## update after rebuttal\n\nAfter a cursory reading of the two papers that reviewer 8kkH suggested, I agree with him, that they reduce the novelty of the presented approach and therefore also reduce my score to 3. However the authors have written very clearly how they construct their dataset, which is the main focus of their work. They also use a templated dataset, which makes it harder for language models to just learn the answers and also more extensible. Their evaluation shows the benefits of their dataset. The other papers in comparison rely on small datasets and/or manually written datapoints. I therefore think their paper deserves a fair chance and disagree with the negative view of reviewer 8kkH.\n\nIn my opinion, the main focus of the article is the dataset, which the authors describe in detail and their description is well-written and easy to follow. So even if the authors can not fully explain the trends, as reviewer QkHy remarks, seen in their analysis, the evaluation shows the necessity for such a dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer fmAM for their helpful feedback. We appreciate their positive comments that our evaluation is “**well executed and broad**”, our analyses are “**convincing**” and that our paper is “**clearly written, well presented and easy to follow**”.\n\n**Q1: “How can GPT-4o be in Fig 6, if calculation relies on token probability?”**\n\nWe can only access probabilities of the top 20 most likely tokens predicted by GPT-4o. The Reviewer is correct that Fig 6 is based on probability of the first token in the country name. By design, our experiment’s control for factual knowledge (**lines 257-278**) **guarantees** that the first token in the country name is **the most likely token** predicted by the model at that timestep. We can therefore access its predicted probability. By contrast, our method for model confidence (**lines 251-267**) requires access to probability of tokens which may not be in the top 20 predictions.\n\n**Suggestion 1 (S1): “claim that humans recognize 'rulebreakers', but no evaluation data provided”**/**Q2: “Did you consider providing a baseline based on humans solving these questions?”**\n\nOur dataset is based on rulebreakers used in prior cognitive science studies (see table below), which were all manually **crafted by expert psychologists and tested on human participants under controlled settings**. These studies already establish a baseline, finding that people generally handle these rulebreakers as we expected. Our key contribution is in systematically scaling these examples by identifying recurring patterns, creating templates and generating controlled variations that preserve the structure of these initial examples.\n\nAlso, **the ability to recognize rulebreakers is contingent on reasoners having the relevant knowledge (e.g. that “Stockholm is in Sweden”)**. Since LLMs have often been observed to possess broader factual knowledge, directly comparing their performance on RULEBREAKERS against humans would not be valid. As such, we consider that an additional human baseline would add limited value. We are happy to conduct a small-scale human annotation but, given limited time to hire annotators, will include it in the final version.\n\n| **Study** | **Examples from dataset** | **Our corresponding rulebreaker template(s)** | **No. of participants in study** | **Relevant finding** |\n|---|---|---|---|---|\n| Quelhas et al. (2010) [1] | “If Manuel plays a game, then he doesn’t play football” | MT (type, instance) | 28 | Participants generally avoid conclusions that factually contradict the premises, even where they can be derived by applying a logical rule (modus tollens). Instead, they conclude that “nothing follows” from the premises. |\n| Quelhas and Johnson-Laird (2017) [2] | “Andre is in Lisbon or he is in Portugal”; “Luis is eating chicken or he is eating meat” | DS (country, city), DS (type, instance) | 80 | Same as above, with respect to the logical rule of disjunctive syllogism. |\n| Johnson-Laird and Byrne (2002) [3] | “If Bill is in Brazil then he is not in Rio de Janeiro”; “If Ann is in the Hotel LaBlanc then she is not in the Champagne Suite” | MT (country, city) | 41 | When participants are familiar with the entities mentioned in the premises, they are more likely to recognize and avoid factually contradicting conclusions, as compared to when they are not. |\n \n**Weakness 1 (W1): “idea is rather simple and only based on a few templates”**\n \nOur approach is intentionally grounded on existing cognitive science studies: we ensure the **four** templates (from which we generated **25,600 instances**) are directly based on examples carefully crafted and validated in prior work with human participants.\n \nExpanding the set of templates is a promising direction for future work but, as the Reviewer rightly implies, such new templates will also need to be based on examples that are validated empirically with human participants. We hope our novel methodology and findings will inspire efforts in this direction from both NLP and cognitive science communities.\n \n**W2: “only single step reasoning”**\n \nAs the Reviewer rightly points out, models exhibit poor to mediocre performance even in single-step settings. We believe that a targeted and rigorous study of single-step inferences is critical in exposing blind spots that lead to failures in multi-step settings.\n\nWhile we specifically focus on inherent patterns of a model’s reasoning process independent of any prompt engineering, future work could design prompting techniques to steer model behaviour and address the blind spot we identified of LLMs over-rigidly applying logical rules to accept and draw factually inconsistent conclusions. \n \n**S2: typos**\n\nThank you for spotting these. We will incorporate the corrections in our final version.\n\n[1] Quelhas et al. https://doi.org/10.1080/17470210903536902\n\n[2] Quelhas and Johnson-Laird. https://doi.org/10.1080/17470218.2016.1154079\n\n[3] Johnson-Laird and Byrne. https://doi.org/10.1037/0033-295X.109.4.646", "claims_and_evidence": "Mostly yes.\n\nIf I understand Figure 6 correctly, it is based on token probability of the first token in the country name. However the results include GPT-4o, a closed source model, where the authors previously claimed that they cannot access such information.\n\nThe authors claim that most humans would recognize these \"rulebreakers\", however there is no evaluation data provided in the paper itself.", "methods_and_evaluation_criteria": "Yes, the authors created their own dataset and evaluate various aspects of that dataset for seven different models.", "theoretical_claims": "There are no theoretical claims. The authors define the metrics for their evaluation with formulas, which look reasonable.", "experimental_designs_or_analyses": "Evaluation methodology seems simple enough and reasonable. Prompts and samples are documented well enough. The conducted analyses seems convincing, albeit sometimes inclusive.", "relation_to_broader_scientific_literature": "The authors propose a new dataset for testing, whether language models can conduct reasoning, but also realizes if the reasoning conflicts with factual knowledge. While the idea for the dataset is simple, that idea is presented clearly and the description is easy to follow, which should make the use of the dataset fairly straightforward.\nThe authors intend to contribute to a growing body of literature, that cautions the increased use of reasoning by language models.", "essential_references_not_discussed": "No, not to my knowledge.", "other_strengths_and_weaknesses": "strengths:\n* paper was clearly written, well presented and easy to follow\n* well executed and broad evaluation\n\nweaknesses:\n* idea is rather simple and only based on a few templates, which begs the question\n* only single step reasoning, but even so models show only poor or mediocre performance", "other_comments_or_suggestions": "typos:\n* line 395: extra white space between end of the sentence and the footnote number\n* line 832: \"all As and Cs\" - typo: \"and\" -> \"are\"", "questions_for_authors": "* How can GPT-4o be in Figure 6, if the calculation relies on the token probability?\n* Did you consider providing a baseline based on humans solving these questions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ab5qCjrPr", "reviewer": "Reviewer_8kkH", "summary": "The paper RULEBREAKERS: Challenging Large Language Models at the Crossroads between Formal Logic and Human-like Reasoning introduces RULEBREAKERS, a dataset designed to evaluate large language models' (LLMs) ability to distinguish between logical rule-based conclusions and conclusions that align with human reasoning, which incorporates commonsense and factual knowledge. The study defines \"rulebreakers\" as scenarios where conclusions derived using formal logic contradict human expectations. Evaluating seven state-of-the-art LLMs, including GPT-4o, the paper finds that most models perform poorly on recognizing rulebreakers, often over-applying formal logic in a rigid manner. The authors identify two possible reasons for this failure: (1) models' poor utilization of world knowledge, and (2) suboptimal attention allocation in reasoning. Their findings highlight a crucial limitation of LLMs and provide a counterpoint to recent works that integrate formal logic to improve LLM reasoning, warning against potential divergences from human-like reasoning.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank Reviewer 8kkH for commending our “**methodology is well-designed**”, our claims are “**largely supported by empirical evidence**”, and that our experimental design and analyses are “**thoughtfully constructed, with several mechanisms to ensure validity**”. We further appreciate and agree with their assessment that our paper is “**well-situated within the broader scientific literature**” and “**contributes to multiple strands of prior research**”.\n\n**Weakness/Question: “finding well-studied by previous works” [1, 2]**\n\nThe only weakness raised by the Reviewer may stem from a misunderstanding as the objective, methodology, and findings of these two works are fundamentally different from ours. In light of the clarifications to follow and the Reviewer’s positive comments above, we respectfully ask the Reviewer to consider whether an adjustment to their initial score may be appropriate. \n\nWe kindly refer to **Appendix A (lines 829-833), where we differentiate our work from Lampinen et al. (2024) [1]** and the comparison table below. We did not initially include Wu et al. (2024) [2] because their methodology (applying a counterfactual ontology to logical templates) and findings have largely been established in **Saparov and He (2023) [3]**, which we also already distinguished in **Appendix A (lines 846-852)**. We are grateful for the suggestion and will add more discussion in the final version to make it clear. \n\n|  | **Example instance in dataset** | **Dataset uses a counterfactual ontology?** | **Conclusion is valid according to logic?** | **Premises are not true in the real world?** | **Conclusion is not true in the real world?** | **Assuming the premises are true, does the conclusion contradict any of the Premises?** | **Relevant finding** |\n|---|---|---|---|---|---|---|---|\n| Lampinen et al. (2024) [1] | Premises: All Swedish cities are French cities. All French cities are in Poland. Conclusion: All Swedish cities are in Poland. | Yes to some extent. Examples that violate real-world knowledge were manually written. | Yes | Yes. Sweden, France and Poland are three separate countries, with different cities. | Yes | No | When performing **logical reasoning**, LLMs tend to be biased in judging conclusions that are true in the real world as logically valid, and conclusions that are false in the real world as invalid, even where the underlying logical structure is the same in both cases. |\n| Wu et al. (2024) [2] | Premises: Swedish cities are French cities. French cities are American cities. Conclusion: Swedish cities are American cities. | Yes | Yes | Yes. As above, Sweden, France and America are three separate countries, with different cities. | Yes | No | In a **logical reasoning** task, the more the premises deviate from real-world knowledge, the worse LLMs tend to perform in correctly applying logical rules to derive a valid conclusion. |\n| Our work | Premises: Anne is either in Stockholm or somewhere in Sweden. Anne is not in Sweden. Conclusion: Anne is in Stockholm. | No | Yes | Undetermined | Undetermined | Yes. In the example rulebreaker, the conclusion contradicts the second premise because it is factually impossible for Anne to be in Stockholm if she is not in Sweden, given that Stockholm is located in Sweden in the real world. | When **reasoning with natural language _in general_**, LLMs tend to accept conclusions that can be derived by rigidly applying logical rules, even when these conclusions contradict the premises in fact. |\n\nAs shown in the table above, our work differs significantly from [1] and [2] in that (a) we do not use a counterfactual ontology, (b) we do not introduce premises or conclusions that are untrue in the real world; and (c) our findings are not concerned with models’ performance in purely logical reasoning tasks, but with their ability to reason in general with natural language.\n\nAs the same two papers are mentioned in the Reviewer's question, we avoid repeating our response here for clarity.\n\n**Suggestion: alternative terminology**\n\nWe thank the Reviewer for suggesting alternative terminology to connect our paper to other related work. We will incorporate this by describing rulebreaker cases as “factual contradictions arising from over-rigid reasoning with formal logic” in the final version. Additionally, we will make clear the connection and implications of our findings for LLM applications in knowledge-intensive tasks as we discuss in our S1 response to Reviewer twqU.\n\n[1] Wu et al. https://aclanthology.org/2024.naacl-long.102\n\n[2] Lampinen et al. https://doi.org/10.1093/pnasnexus/pgae233 \n\n[3] Saparov and He. https://openreview.net/pdf?id=qFVVBzXxR2V", "claims_and_evidence": "The claims in RULEBREAKERS: Challenging Large Language Models at the Crossroads between Formal Logic and Human-like Reasoning are largely supported by empirical evidence.", "methods_and_evaluation_criteria": "Yes, the methodology is well-designed for assessing how LLMs handle a crucial failure mode: rigidly applying logical rules without considering common sense or world knowledge. The dataset ensures control over semantic content while allowing systematic testing across models. The evaluation metrics provide both direct (accuracy) and indirect (confidence, attention) insights into model reasoning.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "The experimental design and analyses in RULEBREAKERS appear to be thoughtfully constructed, with several mechanisms to ensure validity.", "relation_to_broader_scientific_literature": "The RULEBREAKERS paper is well-situated within the broader scientific literature at the intersection of formal logic, human-like reasoning, and LLM evaluation. It contributes to multiple strands of prior research, including cognitive science, natural language inference, logic-based AI methods, and LLM reasoning evaluation.", "essential_references_not_discussed": "https://arxiv.org/pdf/2307.02477: Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks\n\nhttps://arxiv.org/pdf/2207.07051: Language models show human-like content effects on reasoning tasks", "other_strengths_and_weaknesses": "The main finding of this paper seems to have already been well-studied by previous works:\n\nhttps://arxiv.org/pdf/2307.02477: Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks\n\nhttps://arxiv.org/pdf/2207.07051: Language models show human-like content effects on reasoning tasks", "other_comments_or_suggestions": "The naming of RULEBREAKERS vs. non-rulebreakers is clear, but a brief mention of alternative terminology (e.g., \"rigid logic contradictions\") could help connect the paper to related work in logic and commonsense reasoning.", "questions_for_authors": "Are the authors aware of these two well-established works that study very similar topics and has come up with very similar conclusions?\n\nhttps://arxiv.org/pdf/2307.02477: Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks\n\nhttps://arxiv.org/pdf/2207.07051: Language models show human-like content effects on reasoning tasks", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper contributes a dataset designed to assess LLMs' ability to reason using common sense and factual knowledge rather than strictly following formal logic. Most of the reviewers agree that the work is of high quality, and the dataset + evaluation is well executed, though there is a large score discrepancy. This is because the focus of the work has strong similarities to prior work by Lampinen et al. and by Wu et al., which test a different but closely related hypothesis. The author do discuss this prior work and highlight differences, but the reviewers can not agree whether this is reason to recommend rejection or not. After reading the reviews, I am inclined to agree with the summary of Reviewer fmAM who highlights the merits of this work even though there are similarities: 

> the authors have written very clearly how they construct their dataset, which is the main focus of their work. They also use a templated dataset, which makes it harder for language models to just learn the answers and also more extensible. Their evaluation shows the benefits of their dataset. The other papers in comparison rely on small datasets and/or manually written datapoints. I therefore think their paper deserves a fair chance and disagree with the negative view of reviewer 8kkH. In my opinion, the main focus of the article is the dataset, which the authors describe in detail and their description is well-written and easy to follow. So even if the authors can not fully explain the trends, as reviewer QkHy remarks, seen in their analysis, the evaluation shows the necessity for such a dataset.

Considering the above, together with the broad agreement (except Reviewer QkHy) about the quality of this work (ignoring the prior work issue) I will mark this work as leaning accept.
Decision: Accept (poster)