Theory-Grounded Evaluation of Human-Like Fallacy Patterns in LLM Reasoning
Authors: Andrew Keenan Richardson
Ryan Othniel Kearns
Sean Moss
Vincent Wang
Philipp Koralus
Venue: ICLR 2026 Poster (2026)
Keywords: LLMs
language models
reasoning
synthetic data
contamination-proof
human-like errors
cognitive fallacies
Erotetic Theory of Reasoning
PyETR
logical fallacies
human-like reasoning patterns
reasoning evaluation
question-driven inference
inverse scaling laws
human cognition
rationality vs fallibility
cognitive biases
order effects
reasoning benchmarks
cognitive science alignment
AI alignment
systematic deviations from logic
normative vs descriptive reasoning
reasoning tasks
disjunction fallacy
modus ponens
modus tollens
syllogistic inference
logical validity
data contamination
natural language reasoning tasks
formal semantics
mental models
evaluation harness
Chatbot Arena
medical diagnosis
legal reasoning
high-stakes decision-making
benchmarks
alignment benchmarks
robust reasoning systems
AI evaluation frameworks
Primary area: datasets and benchmarks
TL;DR: Language models of different strengths show shifting patterns in how they make mistakes, with stronger models’ errors more often resembling predictable human reasoning errors.

Abstract: We study logical reasoning in language models by asking whether their errors follow established human fallacy patterns. Using the Erotetic Theory of Reasoning (ETR) and its open‑source implementation, PyETR, we programmatically generate 383 formally specified reasoning problems and evaluate 38 models. For each response, we judge logical correctness and, when incorrect, whether it matches an ETR‑predicted fallacy. Two results stand out: (i) as a capability proxy (Chatbot Arena Elo) increases, a larger share of a model’s incorrect answers are ETR‑predicted fallacies ($\rho=0.360, p=0.0265$), while overall correctness on this dataset shows no correlation with capability; (ii) reversing premise order significantly reduces fallacy production for many models, mirroring human order effects. Methodologically, PyETR provides an open‑source pipeline for unbounded, synthetic, contamination‑resistant reasoning tests linked to a cognitive theory, enabling analyses that focus on error composition rather than error rate.

Published as a conference paper at ICLR 2026
THEORY-GROUNDED EVALUATION OF HUMAN-LIKE FALLACY PATTERNS IN LLM REASONING
Andrew Keenan Richardson Ryan Othniel Kearns 1,2.1, Sean Moss' .3
Vincent Nang-Mascianicaa Philipp Koralus'
'The Laboratory for Human-Centered AI, Institute for Ethics in AI,
Faculty of Philosophy, University of Oxford, UK
2 "xfood Internet Institute, University of Oxford, UK
3 School of Computer Science, University of Birmingham, UK
ABSTRACT
We study logical reasoning in language models by asking whether their errors follow established human fallacy patterns. Using the Erotetic Theory of Rea- soning (ETR) and its open-source implementation, PyETR, we programmati- cally generate 383 formally specified reasoning problems and evaluate 38 models. For each response, we judge logical correctness and, when incorrect, whether it matches an Ere-predetted fallacy. Two results stand out: (i) capability proxy (Chatbot Arena Elo) increases, a larger share of a model's incorrect answers are 0.360, p 0.0265), while overall correctness
ETR-predicted fallacies this dataset shows no correlation with capability; (ii) reversing premise order sig- nificantly reduces fallacy production for many models, mirroring human order ef- fects. Methodologically. PyETR provides an open-source pipeline for unbounded, synthetic, contamination-resistaan reasoning tests linked to a cognitive theory, en- abling analyses that focus on error composition rather than error rate.
INTRODUCTION
Language models increasingly solve complex tasks (Wei et al. 2023; Kojima et al. (022). We ask simple question: when they err on controlled reasoning problems, do their errors align with human fallacies? Human reasoning shows systematic, repeatable fallacies Tversky & Kahneman
1974 Kahneman & Tversky 1982 Evans 1994 Walsh & Johnson-Lain 2004; Johnsonnn Laird,
2006
These are not random mistakes; they follow predictable patterns across tasks. Understanding
whether LLM errors share these patterns is useful both scientifically and for deployment in settings
that require reliable reasoning (Jacobs 2021 Bommasani al., 2022).
To rigorously investigate this question, we leverage the Erotetic Theory of Reasoning (ETR) (Ko- ralus 2022; Mascarenhas & Koralus 2017a] Koralus & Mascarenhas) 2013), a formal cogmitive theory that precisely predicts human reasoning patterns across multiple domains. ETR posits that human reasoning operates by maintaining disjunctive alternatives and filtering these alternatives when taking new information, a process that can systematically lead to error. ETR provides for- mal specifications of both how and when humans will make specific reasoning errors, allowing us to generate arbitrary reasoning problems with predictable failure patterns.
ETR has an open-source implementation, PYETR (Koralus et al. 2025). Using PyETR, we mechan- ically generate reasoning problems predicted to elicit specific fal fallacies. We evaluate 38 models fixed corpus of 383 such problems.
Our experimental investigation evaluates 38 language models ranging from smaller models (c.g., Mistral 7B Inst ruct vc. 1) to larger systems (e.g., GPT-4. 5, Claude 7),, treating Chatbot Arena Elo increases, a larger share of logically incorrect answers align with ETR-predicted human fallacies (p 0.360, p 0.0265). This a correlative claim; our experiments do not suggest or support any causal mechanism.
capability proxy (Chiang et al. 2024). We observe that this proxy
Tryan, kearns@ci www.:/ii.........ooooooo. uk.

---

Published as a conference paper at ICLR 2026
While language models often improve on many benchmarks (Wei et al. 2023; Bubeck al. 2023) our results indicate that within our task domain, the proportion of errors matching ETR predictdd fallacies increases with Chatbot Arena Elo capability proxy. These findings suggest overlap between LLM error patterns and human fallacy patterns irrespective of any underlying cognitive mechanisms.
We affirm the robustness of our result using metrics other than Chatbot Arena Elo, where available, for a subset of the 38 models. We find significant Spearman correlation with estimates of training compute (Epoch Al 2025) (p = 0.489, p 0.0334) and significant exponential fit for mean score on the HELM capabilities benchmark (Liang et al. 2023) (r = 0.796, p 0.0103).
Contributions We report statistically significant correlation between Chatbot Arena Elo capability proxy and reasoning errors predicted by ETR, an empirically grounded formal theory of human reasoning. We also provide the first substantial application of PyETR-an an open-source implementation of ETR-to generate unbounded, synthetic, Contamination-resittaan reasoning tasks linking model errors to theory-predicted fallacies.
The remainder of the paper is organised as follows. Section 2 introduces ETR and PyETR. Section 3 details methodology. Section 4 presents results. Section 5 discusses implications and concludes.
Related Work Substantial prior literature in psychology and cognitive science documents irra- tionalities human judgement and decision maakin (Tversky Kahneman, 1974| Kahneman & Tversky 1982 Evans 1994. Kahneman et al. 1990). In particular, the psychology of deductive inference reveals human propositional reasoning to be vulnerable to logically-irreeeant linguistic effects, like co-references between premises (Walsh & Johnson-Laird 2004). Mental model theory provides an explanatory paradigm for these results UJonnsonLLait & Byrne 1991). though prior work the Erotetic Theory of Reasoning matches and has replicated mental model predictions on deductive inference tasks (Mascarenhas & Koralus 2015 2017b).
In the language model literature, prior studies find that while modern LLMs largely succeed syllogistic inference (Clark et al. 2021; Eisape et al.) 2024; [ertolazzi et al. 2024), they exhibit human-like reasoning failures, including distractability (Shi et al.] 2023), content effects (Lampinen)
et al. 2024), and order effects (Eisape et al. 2024 Saparov & He 2023). In a prior work by two of the present authors, the first application of the Erotetic Theory to MLMs found the predictive utility of
the theory to increase for larger model sizes (Koralus & Wang- Masciania 2023), though this study
was limited to OpenAl's GPT model family (Brown et al. [020)] OpenAl 2022, 2024) and did not
make use of PyETR. Our implementation yields an endlessly regenerative and content-agnoste test
of the Erotetic Theory for LLMs in a manner more resilient to data contamination.
2 THE EROTETIC THEORY OF REASON
Modern accounts of human reasoning highlight duality: impressive competence and predictable errors. The Erotetic Theory of Reasoning (ETR) explains both with single idea: we reason by man- aging questions and candidate answers (or "alternatives") and filtering them when new information arrives. Filtering is efficient but can drop relevant alternatives too early, producing characteristic fallacies. ETR formalises this process and defines when conclusions stabilise against follow-up questions (a state called "erotetic equilibrium").
At a high level ETR considers reasoning to consist of three steps: (1) maintain disjunctive alterna- tives candidate answers to an implicit question, (2) filter by best match with incoming information (risking premature elimination), and (3) recover by asking questions (raising structured alternatives) that reintroduce necessary information. This theory predicts a range of empirical reasoning data across propositional and first-order logic, probability, and decision-making. Morover, is mathe- matically proved in Koralus (2022) that ETR converges normative standards of rationality in the limit, as the recovery step introduces enough alternatives to stabilise further reasoning steps.
We can demonstrate ETR with compact example (Example 49 in Koralus (202)))
'A full formalism for ETR is provided in Appendix A
2

---

Published as a conference paper at ICLR 2026
Premise 1
Either there an ace in Mary's hand and some other player has a king, else there is queen in John's hand and some other player has a jack.
Premise 2
Sally has king.
Question
Does it follow that Mary has an ace?
The correct logical answer is "not necessarily", yet many respondents endorse "Mary has ace" (60% data following Mascarenhas & Koralus (017a)).
To model the problem in logic, would first choose representation of the basic predicates, in this case Ace(r) for the predicate that (the card) x is an ace, and Has z) for the predicate that (player) y holds (a card) z. The same applies in ETR, where instead of logical formulas concept of view (see Appendix A) for the full mathematical formalism). It will suffice for now to use the (still precise) shorthand notation to express the premises and conclusion as ETR views
have the central
P1
(Ace(a") (as(Mayyy a) King(b") Hasse, b), Queen(d)Has(John) J).aec(())aaassa (acc)
P2
(King(g" Haaso")aalaaaaaaaaa (Ace(h" Has(Mary,, A)]
Each view is given disjunctive set of alternatives, separated by comma. Each alternative is to be considered as a conjunctive set of atomic propositions, where for convenience we omit the set-brackets and commas and simply juxtapose elements. Many readers will recognise these as disjunctive normal forms, but in ETR there is no inherent ordering to the atoms in conjunction or alternatives in the disjunction. The individual lowercase letters can be understood as existen- tially quantified variables, with the scope of the quantification encompassing the entire disjunction ("prenex form"). view may also carry supposition, written superscriph (see Table for ex- amples). Note that since any formula of first-order logic can be written in prenex disjunctive normal form, ETR is expressive as full first-order logic.
PyETR, an open-source software package that serves as a calculator for ETR (Koralus et al. 2025 can parse views from similar text format.
from pyetr import View
pl View. from_ str ("3a 3b Ee 3d Be 3f (Ace (a* Has (Mary () a) Has (c, b King (b+), Has (John () d) Has ef Jack (e) Queen (d) }))
p2 View. fwrmrrrrossssooeeeooe
prrom_strra str ("3g (King (gx) Has (Sally () ))
= View. from_str ("3h (Ace (h*) Has (Mary () h)
Note the requirement explicitly declare existentially quantified names (the letter may be used in place of 3). ETR posits some default procedures for ordinary reasoning built out of basic operations. The workhorse among the basic operations is Update, which updates current view with an incom- ing one. We can check in PyETR that ETR does predict a fallacy here, because Update discards the seemingly less relevant alternative where John has queen.
>>> v pl.upd (p2); print (v)
3g 31 3m (Ace (1*) Has (Mary () 1) Has (Sally () Has (m, g) King (gx)
>>> v. query (q)
3h (Ace (h+) Has (Mary () h))))
The actual default procedure of ETR involves few more steps which become relevant in more complex problems. The full version is implemented in PyETR.
>>>
from pyetr. inference import drffuutt-rroeedur__ddee does ft_llll___folloowwwooooo___ p2),
>>> de
True
AAn additional aspect not present ordinary logic issue structure. This notated with an asterisk on some occurrences of terms, indicative of being 'at issue" for the containing atomic proposition. The issue structure has no logical content, but plays role in guiding ETR inferences. Determination of issue structure from cues in natural language context is outside the scope of ETR, but simple heuristic in this example issue.
that the repetition of 'Ace' and 'King' suggests those concepts

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
In our experiment, a reasoning problem is given by a list of views (the premises), and the problem answer the question "what, if anything, follows?". Our method generates reasoning problems where the ETR- ETR-predicted answer is logical fallacy.
3.1
DATA PROCESSING
Table 1: Original bank of reasoning problems.
Problem
Symbolic representation
Natural language example
Modus ponens
(R()))(00)
it rains, it's wet.
[Q(x))
It rains.
1. (R(x)
Therefore, it's wet.
Modus tollens
(R(x))(00))
If the switch is on, then the lamp is lit. The lamp not lit.
(R(z)}
(Q(x))
Therefore, the switch is not on.
Quantified modus ponens
(=([()))(000)
All mammals have lungs. All dogs are mammals. Therefore, all dogs have lungs.
vx(Q(z))
Disjunction fallacy
(e)
She is quiet and clever, or tall and athletic.
(Q(z)
She is quiet.
[((r))
Therefore, she is clever.
We source our original bank of reasoning problems from examples presented in the Reason and Inquiry text (Koralus) 2022). These include basic "template" reasoning problems like modus ponens, modus tollens, and the disjunction fallacy as originally presented in Walsh & Johnson-Lant Table pressenss the complete list original problems. Not all reasoning problems take the form of valid syllogistic inferences; that is, not all conclusions from ETR inference deductively follow from the premises.
(2004).
To avoid complications in the mapping to natural language, we restrict to views with only monadic cone-place) predicates. The corresponding monadic first-order logie known to be less expressive than full first-order logic, but still allows for rich set of reasoning problems. Our method could easily be extended to include polyadic predicates.
3.2 GENERATIVE PIPELINE
We define collection of mutation functions that give slight modifications ETR view objects. These functions serve to expand our bank of reasoning problems by modifying existing problems by single premises. Table presents the complete list of possible mutation rules.
Generating new problem proceeds by iteratively building list of views (the premises in the prob- lem). new view is selected at random from the original problem bank and subjected to a random number of random mutation rules. The mutated view is added the list if ETR predicts the problem admits a non-trivial answer. New views are added until all of the following stopping conditions are met: (1) The problem is the right size, meaning that summing the number of atoms in each view gives between 4 and 11. If 11 is exceeded, backtracking is used. (2) The ETR- eredppeddeted conclusion contains single categorical alternative (without disjunctions). (3) The ETR- E conclusion is logical fallacy. To mitigate concerns about dataset-specifit effects, we intentionally generated problems across diverse domains and structures. The PyETR- generation ensures our problems test reasoning capacities rather than memorization of training examples.
Dataset curation We initially generated 400 problems. Pre-analysis integrity checks identified 17 items whose fallacy status did not meet the above stopping conditions, so excluded them prior
4

---

Published as a conference paper at ICLR 2026
Table 2: Mutation rules for ETR views.
Mutation Type
Description
Predicate Addition
Introduces new predicate symbol to the language. Introduces constant symbol to the formal system. Adds new variable the of arbitrary objects.
Constant Addition
Variable Addition
Monstat-tooVariabae Substitution
Replaces constants with V or quantified variables. Conjoins new atomic formulas to existing states.
Conjunctive Atom Insertion
Disjunctive State Addition
Disjunctively creates new states with atoms.
Atom Negation
Applies removes negation from atomic formulas.
to all analyses, yielding 383 problems. Given the modest reduction in sample size and the cost of re-rumning all model evaluations, report results this vetted 383-set; the generation pipeline supports regenerating larger sets for future work.
3.3 MAPPING TO NATURAL LANGUAGE
To evaluate logical reasoning in language models in a more naturalistic context, we developed a systematic approach convert formal logical statements into natural language prompts. We created 12 themes for natural language framings logical problems, such as researcher figuring out the properties of novel element
newly discovered creature. The elements of our 12 themes listed in Appendix B For each theme, we established consistent mappings between logical elements and thematic equivalents. The thematic conversion process serves multiple purposes: it ensures our results are robust against content effects (Lampinen et al. 2024). it enables us to test whether the reasoning patterns are consistent in diverse contexts, and mitigates potential data contamination, significant problem for LLM evaluation (Zhou et al. 2023) Deng et al. 2024 Li & Flanigan [2024] Singh et al. 2024) by reformulating problems novel scenarios unlikely to appear training data.
Consider the following example of disjunction fallacy problem in its formal representation:
P1
(Q()))
P2
{Q(=))
In the "alchemy" theme, logical predicates (e.g., Qxx), R(y)) were mapped to thematic attributes (e.g., "is transmuting", "is time-bending"), while logical variables (e.g., æ, y) were mapped to imag- inary thematic entities (e.g.. "cosmic dust", "vital mercury"), Using our alchemy theme, this was transformed into:
I'm an alchemist studying mysterious substances in my laboratory. need to understand their properties through logical reasoning. Here's what I've discovered:
Cosmic dust is transmuting and vital mercury is time-bending, or cosmic dust mnmoortality-rraniiin and vital mercury is spirit-affecting.
Also, we know that cosmic dust is immortality-grantin
What if anything follows? I do not have an intended answer in mind, and it is possible that nothing follows. Please be succinct and precise.
This is typical of the prompts sent to the LLM It includes a thematic preamble and a natural lan- guage prompt to reason in open-ended fashion with: "What if anything follows?" Each problem maintained the same prompt structure: an introduction establishing the thematic context, followed by the premises expressed natural language, and concluding with standardised question asking
CCompette prompts, with reasoning traces and responses, provided in Appendix

---

Published as a conference paper at ICLR 2026
what follows from the premises. The prompt explicitly stated that nothing might follow, to avoid biasing models toward making unwarranted inferences (Saparov & He 2023
3.4
DATA COLLECTION
Model Selection We evaluated 38 language models of varying sophistication, including several specifically designed as reasoning models. Selection informally prioritized quan-
tity, popularity of usage, and representation of a diverse range of scores. Chatbot Arena
scores were obtained from https://buggingfcce.. wttp:///e.cc//cceeee/eeeeaaeeeee Laa.ccesemmarenaaaaa..a
chatbot-aree arena- htt.a//rrrrrrrrrrrsoo//////a//// ceslloorrssts___s
http 20250423.2.. pk1
Scores for the HELM Capabilities benchmark for models wtt....../////eeetaaaaaaaa/
obtained from https:
7/crfm. stanford. htt.emmmtta.ffrreeee///eoo wtt.oo/..cdu/eeee//aaaaaaaalie.a
(Liang
al. 2023) and estimates for training compute for 19 models were obtained from
https: : /ttps://epoooe httppppp.c.aacc/// wttpoch.aa/aataaaai.a.aaaaaaaaaa aaa/aaaaiimmmdellll
(Epoch Al 2025 (both URLs accessed 30
July 2025).
Response Processing and Filtering Models provided conclusions in natural language, which we converted to PyETR format for evaluation. To ensure fair assessment reasoning (not formatting). GPT-4.1-mini served as our translation layer without access to the original premises. Our manual spot-checks confirmed this translation step to be perfectly faithful.
Each model evaluated 383 reasoning tasks generated by PyETR. During analysis, encountered occasional parse errors when processing model responses. To maintain data qual- ity while preserving statistical power, models with parse error rates exceeding 20% across the test set were excluded from analysis, which resulted the exclusion of one model (google__eemniie we_.emmmnii..... w.-.ro-prreiie--O-- sponses that produced parse errors were excluded from analysis on case-by-case basis rather than discarding all responses from that model.
For the remaining models, individual
Testing Framework The full set of logical problems were framed as evals, using Eleuther's Lan- guage Model Evaluation Harness (Gao et al., (202)). which handles evaling and metric collection against a wide range of models. OpenRouter (OpenRouter, 2025) was used for all models in order have a unified API. This consumed less than $1,000 of compute resources. Each model was given a token limit of 3,000 output tokens.
To ensure a fair comparison across architectural differences, all reasoning models were allocated 2,400 thinking tokens during evaluation. This standardization was justified given our use of the Chatbot Arena Elo score as our primary capability metric, which already accounts for performance differences regardless of underlying mechanisms (Chiang al. 2024).
Key Measures We collected the following measures to evaluate both the logical correctness of model responses and their alignment with human- like reasoning patterns, enabling analysis of how these patterns correlate with model sophistication. A model's answer was considered 'logically correct' if it was logical consequence of the premises. This tested using PySMT (Cimatti) ee al. 2017) to check whether the negation of the conclusion was inconsistent with the premises. LLM responses were classified predepreddetedd if ETR predicts an endorsement of that conclu- sion according to PyETR's implementation of
Statistical Tests We selected Pearson correlation to capture linear relationships between variables while also reporting Spearman's 's rank correlation to account for potentially non-linear monotonic relationships without assuming normality in the distribution of fallacy rates. For our premise-ore order intervention analysis (discussed in Section 4.4), we employed two-proportion z-tests to rigorously evaluate whether reversing premise order produced statistically significant changes in fallacy rates. The z-test was specifically chosen because it allows to determine if the observed differences in proportions (original vs. reversed premise order) represent genuine effects rather than random variation, providing reliable measure of the significance of order effects across different model capabilities. This statistical approach enables us to quantify the extent to which models exhibit the same premise-order sensitivity documented in human reasoning studies.

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTAL RESULTS
Our analysis reveals several notable patterns in how language models exhibit human-like reasoning fallacies. We report the results of Pearson correlation tests, as well Spearman tests for the de- tection non-parametri monotonic relationships. We consider p - 0.05 a suitable boundary for statistical significance.
4.1 PROPORTION OF ERRORS THAT ARE FALLACIES
We operationalise a "human-like fallacy" (or simply "fallacy") as a response that is both ETR- predicted and logically incorrect. This definition captures instances where models make the same systematic reasoning errors that ETR predicts humans would make. Formally, for a given model m and reasoning problem p, we define:
{1 if ETRepredicted(mm ^ -LogicallyComect(m.
IlumannikeFalllccy(m,,
0 otherwise
Our metric of interest is the fallacy rale, defined the proportion of human-like fallacies total logically incorrect answers for each model:
FallacyRate(m) Eper Humanl_ikeFFlllccy(m... p)
E PEE LLggicallyCorrec(mm..
where Pis is the set of all reasoning problems in our evaluation set. This measure represents the degree to which a model's errors align with predictable human reasoning patterns rather than other forms of logical error. We find as model strength increases, does the proportion of logically incorrect answers that are fallacies. We observe this trend consistently across model Elos scores (Figure I well HELM composite benchmark scores and training compute estimates (Appendix C
Model Elo vs. Fallacy Rate
100%
90%
Best Fit
99% CI
80%
70%
50% CI
Data
60%
50%
Log of fallacy rate
20%
0.407, Spearman = 0.360, 0.0265
1000
1050
1100
1150
1200
1250
1300
1350
1400
Model Elo
Figure 1: Over 38 models, direct linear fit produces significant moderate positive correlation (r 0.364, p 0.0247), an exponential fit (log y-axis) displays a more confident Pearson correlation (r = 0.407, (0.0113), as displayed above. As Spearman is ordinal, have the same significant monotonic relationship in both cases (p 0.360, p = 0.0265).
4.2 ERROR ANALYSIS
We manually inspected LLM responses from our experiment and found many patterns in line with ETR's predictions, Models often fixate repeated objects, ignoring disjunctive alternatives

---

Published as a conference paper at ICLR 2026
clude fallacious categorical answers. Models also instantiate existential quantifiers with objects from premises and generally mis-scope quantifiers when presented with repeat information. Appendix D shows some illustrative examples of model fallacy cases.
4.3 OVERALL CORRECTNESS AND OTHER MEASURES
Surprisingly, model strength measured by LLM Elo score does appear have any correla- tion (r = 0.004, 0.981). (981).(0 (p = -0.04, = 0.777) with the model's ability to produce correct answers on our dataset (Table 3 Furthermore, exploratory analysis did not reveal any other statis- tically significant evidence of model Elo correlating or enjoying monotonic ordinal relationship with absolute proportion of fallacies produced, whether exactly predicted by ETR taking logical equivalences.
Table 3: Percentage of model responses which are logically correct
Mean
o
min
((0.25)
(
(0.75)
max
40.6%
16.7%
18.6%
31.0%
37.5%
44.8%
91.7%
4.4 INTERVENTION STUDY: REVERSING PREMISE ORDER
While in classical logic the order in which premises are presented has effect normatively cor- rect logical conclusions, an order-effect is commonly observed in human reasoners (Girotto et al. 1997). where presenting premises in different orders elicits logically different responses. In addition to being a proxy measure for the non classicclilt of human reasoners, this non-comee coon-commuaatiiity effect often blocks fallacies. We observe strong evidence that LLMs similar in this regard: present- ing the same questions with reversed premise order significantly blocks fallacy production. Table reports a significant fallacy-blocking effect from reversing premises on majority of models. Ap- pendix E shows an example of premise reversal blocking an ETR pree-peddeted fallacy.
DISCUSSION AND CONCLUSION
5.1 INTERPRETATION OF KEY FINDINGS
Our results reveal statistically significant correlational trend in enror composition: as language models advance capability (proxied by Chatbot Arena Elo), a larger share logically incorrect answers align with human errors predicted by ETR. While stronger language models demonstrate improved capabilities across many benchmarks, we do not observe increased overall correctness in this dataset. Moreover, demonstrate order-effects for LLM reasoning analogous to those observed humans. This indicates overlap between LLM error patterns and established human fallacy patterns these tasks.
5.2 METHOODOOOICAL CONSIDERATIONS AND LIMITATIONS
While traditional ablation studies might seem appropriate for investigating this phenomenon, they face significant limitations in the current landscape of language models: the diversity of modern architectures, training approaches such as distillation (Gou et al., 202T Gupta & Agrawal 2022
Xu et al. 2024), and reasoning capabilities with test-time scaling (DeepSeek-Al, 2025 Ei-Kishky al. 2024: Xu et al. 2025) mean that simple parameters like model size may no longer serve as reliable proxies for model capability. Our use of Chatbot Arena scores provides performance-based
metric that naturally incorporates a diverse range models, reflecting real-world ability rather than architectural specifics, similar to the observational approach in Ruan al. 2024)
Though the correlation strength (p- = 0.360) may appear moderate, it is robust across diverse model architectures and training paradigms, suggesting fundamental relationship rather than artifact. The statistical significance (p = 0.0265) exceeds conventional thresholds despite our conservative ana- lytic approach.
8

---

Published as a conference paper at ICLR 2026
Table 4: 38 models selected for testing, ordered by Chathot Arena Elo. The right-most column ports the percentage of fallacies per-model blocked (answer becomes logically correct) by reversing premise order, along with the results of two-proportion z-test for each model to gauge the signifi- cance of the blocking effect.
Provider
Model
Elo
Fallacies blocked by reversal
OpenAl
chatgpt-4o-latest
1408
46.15%, (z= (2=3.24, p 1.18e-03)
OpenAI
gpt-4.5-preview
1398
68.00%, (z = -.03.p=2.44e-03 2.44e-03)
Google
gemini-2.55-lash--prvvie
1393
37.27%. =233.52. p p:2227c-04)
DeepSeek
deepseek-chal--3-0324
1373
38.78%. = =2228.. = =
Google
gemma-3-27b-it
1342
20.39%, (z = 1.77.
7.60e-02)
OpenAI
ol-mini
1304
50.00%, (z 0.58. p 5
Anthropic
claude-3.7-sonnet
1292
50.00% 4.23, -=..55-05))
XAI
grok-2-1212
1288
(z 22.68%
Anthropic
claude--.5-sonnet
1283
65.08%, (z=4.74, 2.09ee--6)
OpenAl
gp1-4o-mini-2224-07BB
1272
36.36%, (z= -.
Google
gemini-flash-1.5
1271
12=1.98, 4.78 29.69%,
Mistral
mistral--arge-24077
1251
37.60%,
(z=3.85. 1.18e-04) 24.32%,
Meta
llama-3.1-70b-istruutt
1248
(z=1.74, 8.14e-02) 32.63%,
Anthropic
claude-3-opus
1247
(2=2.79.p55.200-03 5.20e-03) 34.53%,
(2.2..9,
Mistral
misttal-smalll224-nnnsnrre-----
1217
(z=3.81, 1.40e-04)
Microsoft
phi-4
1205
44.68%.
(z= 9
Anthropic
claude-3-sonnet
1201
53.49%.
p==3.04.==2.40ee
Google
gemma-2-9b-it
1192
( 1.07e-0-2 37.25%,
55.17%,
Cohere
command-rrplus-04-22244
1190
(z=2.22. -222..222.64-02 2.4e-022
OpenAI
gpt-4-0314
1186
41.67%,
(z=2.07, - 56.67%,
Anthropic
claude-3-haiku
1179
(z=3.91,
- 9.20c-05)
OpenAl
gpt-4
1163
(2=2.85, 4.43e-03) 54.05%,
Meta
llama-3-8b-instuuct
1152
23.81%, (z= 1==1.55.p 1.21e-01)
Mistral
mistral-medium
1148
(z 1.75e-02) 43.90%,
Mistral
mixtral-8x22b-instmmce
1148
42.6%,
(z.2..1, 3.43e-02) =3:4..2.p=3.43--02)) 32.81%,
Anthropic
claude-2.0
1132
(z=3.40.) 6.65e-04)
Anthropic
claude-2.1
1118
54.79%,
(z=4.28, 1.91e-05)
Mistral
mixtral-&x77biinstuuet
1114
(z= 12=1.49. 1.36c-01) 37.50%.
OpenAI
gpt-3.5-turbo-0125
1106
39.42%,
p=.660.33..0--044 (z= 3.06c-00)) 20.00%,
Meta
llama-3.2-3-iinsruuct
1103
Nous Research
1084
66.67%,
(z 3.63e--)3)
DeepSeek
deepseek-chat
1077
(z= 2.19c--222 35.00%,
Mistral
mistral-7h-insstuct-v0..
1072
38.46%,
(1=1.11, (==66.1,.=2.66-001 2.66e-0))
OpenAI
spt-3.5-turbo-1100
1068
88.46%, (2=4336., -:33e-0))3))0 1.33e-05)
Meta
1aama-3.2-Ib-insttae
1054
32.43%. (z= =z=1.60, 1.10-011)
Microsoft
phi-3-mini-
1mini-1288-innttuuct
1037
38.16%, (2=2.88, p
3.98.--3366--03 3.96e-03)
AllenAI
olmo-77-instrruct
1015
45.45%. (z 1.23, 01)
Mistral
mistral-7b-inntruccs
1008
66.67%, (z = 1.42, 1.55e-01)
While our results demonstrate correlation between model capability and human- errors, we ac- knowledge limitations in attributing causality. Alternative explanations include the possibility that stronger models are increasingly trained on fallacious human human-produu produced reasoning traces, or that post-training methods like RLHF cause convergence reasoning behaviour. The absence of im- proved logical accuracy with model capability may reflect ceiling effects in our test set or funda- mental limitations in current training paradigms.
5.3 THEORETICAL IMPLICATIONS
These findings challenge the assumption that scaling alone leads to more normatively correct reason- ing systems. Instead, within our evaluation setting we observe greater overlap between model error patterns and characteristic human fallacies. This does not by itself imply shared internal processes;

---

Published as a conference paper at ICLR 2026
it highlights evaluation axis (error composition) complementary overall accuracy. We do not advance any particular mechanistic claim, as our results only support an empirical characterization.
Nonetheless, our results provide foundation for several research directions. For benchmark devel- opment, our approach offers a systematic method for generating reasoning problems where human- like fallacies are predicted, creating evaluation sets that could complement existing reasoning bench- marks. For capabilities, the identified fallacy patterns could serve as targeted training examples to fortify models against human-like reasoning errors while maintaining their general capabilities. As tool for assessing engineering choices, our methodology provides quantitative framework for eval- uating the effectiveness of various interventions aimed at improving model reasoning, from prompt engineering to architectural modifications. Follow-up work could investigate other predictive fea- tures, like reasoning trace length, answer diversity, and trends within model families.
5.4 BROADER IMPLICATIONS
Our observed trade-off in error composition has implications for Al alignment. As models become more capable, anticipating and mitigating human-like reasoning errors becomes increasingly impor- tant, especially in high-stakes contexts requiring reliable reasoning, such medical diagnosis, legal analysis, or decision support (Jacobs 2021; Bommasani et al. 20022).
We these results as opening new avenues for understanding both artificial and human reasoning. As language models continue to advance, this work provides a foundation for developing systems that combine human-like understanding with more robust reasoning capabilities: systems that can successfully reason we do while avoiding the pitfalls that characterise human cogmition.
ACKNOWLEDGEMENTS AND DISCLOSURE
Members of the Laboratory for Human- Centered Al are supported by funding from the Cosmos Institute. A.K.R. and R.O.K. were supported by Cosmos Fellowship for the duration of this project. R.O.K. is additionally supported by the Clarendon Scholarship and the Jesus College Old Members' Scholarship.
Author use of AI tools During manuscript preparation, used Al assistance for prose shortening and copy-editing, and as coding support for scaffolding evaluation scripts and refactoring utilities. All analysis code, evaluation protocols, and results designed, implemented, and verified by the authors.
REFERENCES
Leonardo Bertolazzi, Albert Gatt, and Raffaella Bernardi. A systematic analysis of large lan- guage models as soft reasoners: The case of syllogistic inferences. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Prucessing, pp. 18882-13905, Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 10. 1222...1/202....lmmmaaaa wwt.oorg/22022 ewnlp-mana..... 769/
URL
Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolf- sson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Dur- mus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh. Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keel- ing, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Ku- ditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben New- man, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel
10

---

Published as a conference paper at ICLR 2026
Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong. Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srini- vasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kait- lyn Zhou, and Perey Liang. On the opportunities and risks of foundation models, 2022. URL
attps://aaxiv. org/ abs/ 211200122 108 07258
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of Artificial General Intelligence: Early experiments with GPT-4, 2023. URL
org/ abs//3/s.323// 12712
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos. Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E. Gonzalez, and lon Stoica. Chatbot arena: An open platform for evaluating LLMs by human preference. In Forty-first International Conference on Machine Learning, 2024. URL httts::/ eneiieewwwee/e..ee..
Alessandro Cimatti, Alberto Griggio, Roberto Sebastiani, and Marco Roveri. PySMT: A python library for SMT solving. https://giite cwtp:bsttm////mtttco//mm http://pystt//yttttt// pysmtt 2017. Accessed: 2025- 05-15.
Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pp. 3882-3890, 2021.
DeepSeck-AL DeepSeek-Rl: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, 2025. URL http::tt/taxx...oooot /bb///b////// /2001.51110 12948
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 8706-8711 Association for Computational Linguistics, 2024. doi: 10. 1865/v1ll 2024 482. URL/httss: /ttttpss//allanhhooooyy.. org/ www.222naacllooo... 482/
Tiwalayo Eisape, Michael Tessler, Ishita Dasgupta, Fei Sha, Sjoerd Steenkiste, and Tal Linzen. A systematic comparison of syllogistic reasoning in humans and language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pupers), pp. 8425-8444, Mexico City, Mexico, June 2024. As- sociation for Computational Linguistics. doi:
1205333/111222.macl-ll //aclanthology wwwplloyy..gg///000 otg/2024.......oooo. naacl -ong. 466/
URL https:
Ahmed Ei-Kishky, Daniel Selsam, Francis Song, Giambattista Parascandolo, Hongyu Ren, Hunter Lightman, Hyung Won Chung, Ilge Akkaya, Ilya Sutskever, Jason Wei. Jonathan Gordon, Karl Cobbe, Kevin Yu, Lukas Kondraciuk, Max Schwarzer, Mostafa Rohanine- jad, Noam Brown, Shengjia Zhao, Trapit Bansal, Vineet Kosaraju, Wenda Zhou, and Ope- nAI. Learning to Reason with LLMs, 2024. URL https://openal .ttpenidi.commine/ learning to- reason with 11ms/]
Epoch AI. Data on ai models, 07 2025. URL htt.::ps//ppcchaa/aaaaaaaaaaaa cessed: 2025-09-24.
Ac-
Jonathan St B. T. Evans. Bias in Human Reasoning: Causes and Consequences. Essays in Cognitive Psychology. Laurence Erlbaum Associates, 1994. ISBN 978-00863377-55-9.

---

Published as a conference paper at ICLR 2026
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. The language model evaluation harness, 07 2024. Utt.tttss: http::://eeeddoooo.oo...o ottgg//eeeeerr///0/0000022
Vittorio Girotto, Andrea Mazzocco, and Alessandro Tasso. The effect of premise order in condi- tional reasoning: A test of the mental model theory. Cognition, 63(1):1-25 1997. doi: 10.1016, 50010-0277(96600792255 URL
ncbi nlm. nih wtw.hh.../91/7000
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):189-18811) 2021.
Manish Gupta and Puneet Agrawal. Compression of deep learning models for text: survey, ACM Transactions Knowledge Discovery from Data (TKDD), 16(4):1-55, 2022.
Abigail Z. Jacobs. Measurement as governance in and for responsible AI, 2021. URL http:
7/ www.ii.org/ab///00//00 05658
Philip Nicholas Johnson-Laird and Ruth M. Byme. Deduction. Psychology Press, 1991.
P.N. Johnson-Laird. Mental Models, Sentential Reasoning, and Illusory Inferences. In Ad- vances in Psychology, volume 138, pp. 27-51. Elsevier, 2006. ISBN 978-0-414-52079-1. doi: 10.101650116-44115000000777
URL aEtps://1ine http:://LnK.ngguu..eeeeeee.
Setrieve/s11/0011411500000
Daniel Kahneman and Amos Tversky.
The Psychology of Preferences.
Sci-
entific American, 246(1):160-173. 1982.
ISSN 0036-8733. doi: 10.1038/
scientifeamerrian))8---1666
URL
com/
Daniel Kahneman, Jack Knetsch, and Richard H. Thaler. Experimental Tests of the Endow- ment Effect and the Coase Theorem. Journal of Political Economy, 986))1325-1348, 1990. ISSN 0022-3808, 1537-534X. doi: 10. 10.1086/261737.
URL https: www. journals.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners, 2022. URL /ttppstss///xxxt.tooo wwwpxivv.rrg/aeo abs /2005.
11916
Philipp Koralus. Reason and Inquiry: The Erotetic Oxford University Press, 1 edition, 2022. ISBN 978-0-19-186254-0. URL /ttp::::
001. 0001
Philipp Koralus and Salvador Mascarenhas. The Erotetic Theory of Reasoning: Bridges Between Formal Semanties and the Psychology of Deductive Inference. Philosophical Perspectives, 27 (1):312-365, 2013. ISSN 1520-8583, 1520-8583. doi: 10.111 10.pee 12029. URL https:
//onlinelibrary wrl...wleee http:///www..iallecooo////////// 11t.pphhp....pp 12029
Philipp Koralus and Vincent Wang-Mascianica. Humans in Humans Out: On GPT Converging Toward Common Sense both Success and Failure, 2023. URL
2303. 17276
Philipp Koralus, Sean Moss, and Mark Todd. PyETR, 2025. URL https::/github.. com/ Dxford-BAI-Lab/yyyee
Andrew K Lampinen, Ishita Dasgupta, Stephanie C Y Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. Language models. like humans, show content effects on reasoning tasks. PNAS Nexus, 37)) 2024. ISSN 2752-6542. doi: 10. 1093/ppassexxu/ppaae3... URL URI.httss://adi wtt.///ooooog/g//o 11t.//....11.....1.. 1093/ 100//ppasnexxs//paaa222.
12

---

Published as a conference paper at ICLR 2026
Changmao Li and Jeffrey Flanigan. Task contamination: Language models may not be few-shot anymore. Proceedings of the AAAI Conference on Artificial Intelligence, 38(16)118711-18880
2024. doi: (0011609aaaai.38811..29099 AAAT/articlee/ieww//
URL https: aaai. org. azggindex... php/
Perey Liang. Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khat- tab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang. Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. Holistic evaluation of lan- guage models. Transactions Machine Learning Research, 2023. ISSN 2835-8856. URL net forum? Stetps://openreet 1 1oorum?1d=11442211Ee0 Featured Certification, Expert Certification.
Salvador Mascarenhas and Philipp Koralus. Illusory inferences: Disjunctions, indefinites, and the erotetic theory of reasoning. Proceedings of the Annual Meeting of the Cognitive Science Society, 2015. Ltl.Ltttss//scchlaaashhppp wwwps/shhh.......ooo.. wwwp////////////eeeeeeeeee//eeee
Salvador Mascarenhas and Philipp Koralus. Illusory inferences with quantifiers. Thinking and Reasoning, 23(1):33-48, 2017a. doi: 10.1080/13546883.2116111
Salvador Mascarenhas and Philipp Koralus. Illusory inferences with quantifiers. Thinking & Reasoning, 23(1):33 48, 2017b. ISSN 1354-6783, 1464-0708. doi: 10.1080/13546733.2016 1167125.
201 6. 1167125
OpenAl. Introducing ChatGPT, 2022. URL UKL_ppper//peeaa..co///////////a
OpenAl. GPT-4 Technical Report, 2024. URL httl://gg//axivvrrr////////33113
OpenRouter. Openrouter. httpp::/
2025. Accessed: 2025-05-13.
Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. Observational scaling laws and the predictability of langauge model performance. In The Thirly-eighth Annual Conference Neu- ral Information Processing Systems, 2024. URL BtLhhttp:://oo http:////ppereeee..weeee weppereevie..eedee www.o//wwweeettmemmmmeemmm forum?id On5WIN7xyD
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thouud In The Eleventh International Conference on Learning Representations, 2023. URL https:t::: //wppreee//et/e////// ew, net. wwr.m/orrmmm// B d=QFVVBZXXR2V
Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed H Chi, Nathanael Schärli, and Denny Zhou. Large language models can be easily distracted by irrelevant context. In International Conference on Machine Learning, pp. 31210-31227, PMLR, 2023.
Aaditya K. Singh, Muhammed Yusuf Kocyigit, Andrew Poulton, David Esiobu, Maria Lomeli, Gergely Szilvasy, and Dieuwke Hupkes. Evaluation data contamination in LLMs: How do we measure it and (when) does matter?, 2024. URL
03923
Amos Tversky and Daniel Kahneman. Judgment under uncertainty: Heuristics and biases. Science, 185(4157::124-11111 1974. ISSN 00368075, 10959203. URL http:://w///wwt/tttt jstor org/
Stable/173855
Clare R. Walsh and P. N. Johnson-Laird Co-reference and reasoning, Memory & Cognition, 32 (1):96-106, 2004. ISSN 0090-502X, 1532-5946. doi: 10.3758/BF03195823, URL http:
7/1ink. spr wpw.nner.nnnr.rrrrrr nger. com/ 10. 3758/ /F/1 95823
13

---

Published as a conference paper at ICLR 2026
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed Chi, Quoc Elicits Reasoning in Large Language Models,
Le, and Denny Zhou. Thouuht
2023.
Fengli Xu, Qianyue Hao, Zefang Zong, Jingwei Wang, Yunke Zhang, Jingyi Wang, Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, Chenyang Shao, Yuwei Yan, Qinglong Yang, Yiwen Song, Sijian Ren, Xinyuan Hu, Yu Li, Jie Feng, Chen Gao, and Yong Li. Towards large reasoning models: A survey of reinforced reasoning with large language models, 2025. URL
https: /ww.vv.aavvv/..//////ea./aa.aa org/ /ag//bs///// 2501. 09686
Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. A survey on knowledge distillation of large language models, 2024. URL https://arze //rsiv/ org/ abs
http///s//242.... 2402.
Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji- Rong Wen, and Jiawei Han. Don't Make Your LLM an Evaluation Benchmark Cheater, 2023.
URL http:h/ttaatttt org/ awsp/311...es..
A A FORMAL PRESENTATION OF THE EROTETIC THEORY
ETR formulated in Koralus (2022) in a set-theoretic language, and we reproduce the full defini-
tions here, with the explicit permission of the authors. We remark that Definitions [ and [1 contain the parameters of the theory, i.e. the basic atomic propositions, the constants and function symbols, and set logical and extralogical axioms. The central definition that View, Definition 9 Fol- lowing that are definitions of the basic operations of ETR, in particular Update (Definition 23 and Query (Definition 330) whose use in PyETR were demonstrated Section 2
Definition (Basic objects).
Let A, F, (?), PT, P. be pairwise disjoint countable sets.
(F arity) a : F F N. Write f f F as when a(f) = k. (Constants)
E (xw)) = O.
(P polarity)
Let N: P P be bijection. Write for ((PPPT).
Let P = P UP-.
(P arity) a' P + N. Write P ph for (P) = k. dP dP)) (Identity) E
Definition 2 (Terms T). A S T. If fd) E F, then (f9)
tx) S T, then (ft, ..)) E T.
Definition 3 (Atoms A).
If ph E and f= = (f1... 1e) for t, E T, then (P*, 7) E A (abbrv. Pf). |7| = {t1 1t1...0 1x).
Definition 4 (States 5).
S = P(A). Abbreviate '1 } as '0'.
Definition (Dependency relations R).
Let l P(S). Let U A(I' and D DCEXU ((e,u) e EAuE UJ. Define
(U, E, D) E Rr iff
(Bipartite) UnE= = OADOE (Matryoshka) For all u E E or fe E E: (e, E DJ Be
and
e D)
For R E write R= (UR, ER, DR). For (0, (0,0,0) write OR Definition 6 (Open terms T, and atoms A.).
? E T, and (f", (I,,...,44 (x)) E T, whenever f E F and for some 1 1 k, 4, E T, and 1 E T for j + (Pk, (I,,..., E A,, whenever pke and for some 1 S k, 1 T for all +
and 1, E T,
Definition (Issue structures I). I E Tr iff T consists of pairs (1, 2)), S.I. Ai and xl1/2] A(T).
14

---

Published as a conference paper at ICLR 2026
Definition 8 (Issue matches M1J). For I.J E Ip, define,
MIJ = {(t1, 12) (12 TA ((12, Jv (12, 8) E J))
where, for -P*, (.,,..., (k)), & (Pk, (11,..,, (k))
Definition 9 (Views V).
For r. θ finite subsets 8, R E Rr E Irue.
O R, 1) E V (abbreviated
1 Ri V)
Write T for {0)0gg and for 0 0
[0)
Definition 10 (Commitments C). For C V.G V, (C,G) C.
Definition 11 (Primitive absurd states I). Let Kgs contain, Vi, VPeAYPAAVee at least:
(LNC) (p.P).
(Aristotle) {ttt),
(Leibniz) =ttt,/=/t//
Definition 12 (R. Restriction).
[ =UR nX,, Eg X, PR ((ER X) x (UR (x))).
[R'r [R])(T]- Given (T, R)T), we allow ourselves to write (T, [R]. I) for (T, (R]re,,
Definition 13 (L Restriction).
Within quadruple (T' O, R. [4], let
= ((t x) : (f,x) E I Axtt//? A(TUO))
Definition 14 (R Algebra).
Let R S RUR Us, ER U Es, DR U Ds U Es Un) and let
OR ba OR = OR- Let
R S = (Uo, Fo, 0) x (B) Da (() where En (CEENUES Vulle, Dn U D3} and
Un = {MEURUUS Ve & Eo-le, u) & (ER x Un DR) u (Es x Us Ds)).
Definition (Product).
r @ A ((Yer U URE 3we (1b S C>)) U frer: -Eeew( {7)
(T ()) [0) & A (0)
SCP A (0)(0) A @ ... & AW
r8 T T
A JJ (T
A
[[TTaR)
& A SJ
8OR A
RJ
= RI
EP Definition 16 (Sum).
A W.
J.J.
T T
A W
TT S- Vn
Ans.ja
Th t7 gg = (T A) A
(TAR)
r + A SJ = = RI 00R
Ag
OIEP
T A S.J. NY a
@
V A15 ..... AnS.J..
Definition 17 (.A Answer potential). P( [A(T) A(A))
Definition 18 (Answer).
[ (0) SJ
(arg max Allis) : 731 73 1mp Ter
RI
Definition 19 (Negation).
re (0) @ ([ (0) mapla
[ N
yer 10) (p) }
15

---

Published as a conference paper at ICLR 2026
= - F7
[ N = (ER.UR ( b) EUR x ER : (b, & DR))
11 =Iu((t,) (t,z) E
Definition 20 (Novelty). a E A is novel during an inference step (C, if a & A(G) U
AD) and has not yet appeared in the computation of the conclusion of the step. A function V: v:X- + A, where X C A is finite, is novel during an inference step if each v(x) is novel and is injective, and then [u]x denotes the simultaneous substitution Let (-)nov(X) stand for (-)[u)x where v X -+ A is novel.
(ooo)//n,.... /ox)) where
X {aI,....
Definition 21 (Substitution). Let 7(T, = {u E Ur : or a} U fe For :r:ce e Sr a} - {a} {01;---,,0 (x))- For (T) O (()
Sub (t,0) (T = (T(U) [t/a], 9, x (Meeralllllrrr)))l (ilac, (//4),
where 112 (T, a) A is novel.
Definition 22 (Merge).
For either (R) (S) 1 or [R]aVv S.
I SJ)
M ① Taaa (1)) 8 I A @ I X Ro45
yer
((,))M))(()) Sub RoaS (0)))
where
ut E MIJ : uu E Us A w > kn)}
Definition 23 (Update).
For D E C, but with all arbitrary objects novelised,
(r.TR))D =
Definition 24 (Universal product). For AT) 7 A (()
and either A (RR) A()) = 0 or [R]a S.
(0) U {0] A
RI @ Roas
RoaS (u.DeM;; Sub RoaS (T))). ), where (f.u)
RI SJ
M/ = (u,,t) (u, t) E MIJ AuE Un A()) *
Definition 25 (Reorient).
For A E (CU (T and J' LAuV.
(C, RD v -"
Definition 26 (Existential sum).
Let the following conditions be met:
(1) AT) n A()) = 0 and either A(R) n (S) = 0 or RRA S.
(2) Mj ((e () (e,1) E MIa ER A(QUA) A -l,((e E DR)) #0.
Then,
re 10 IR +
1045
Taas
(edtjEMi
Sub
Ro4S ((
U
(6) ((Ee :e A(x)) U8:
(E(
8E
[+) A8 7))))),
.EB(T,J.))
where for e E En and y ET, let T,), Definition 27 (Division).
ET.
V 7ee (5 C S A vh C 7), then
re S A SJ = (7 Ør A
W : TT) [R]]]] [
Or 4 = - 18(8 E AASC y A Heee (b En)
Definition 28 (Factor).
For A # #L.
re Ih V F = 1 : 7 mnn where
AAVF = (7 Ør A N6x or (A 1/a)) : (1,a) E Mrs A a Us}, and where we let
T'n 0 = =
For AV
r [1] fr TTET: -8 K. |
RI
16

---

Published as a conference paper at ICLR 2026
Definition 29 (Matryoshka level ordering). For existentials, we have e SR iff V(d.u) E Dal(e,u), E DR). For both existentials and universals, let the relation <n be the transitive closure of the relation made up by the ordered pairs in Dn U EE x Un - DR) where op reverses the order of pairs in a set of pairs.
Definition (Query).
For Us S UR and Mi, = (11,e) M1s e Fis(ER), we define
[ SJ)
((0: -3 (7. A: Fe... (t Fo
where (67, ++ Bee 3
(tns CnlEM, Vi,j
/e............... tt/...l
and Dg: = D1 U D2 U D3 U Da UD, D is constructed by taking: original dependencies from the internal argument
D1 DDslA/EER
extra dependencies for multiple terms substituted with same e
D2 {(em. u) 3m3m' E Mi,(eem em Atm + Au
dependencies resulting from complex terms being substituted
D3 = {(Cm, u) (tms Cm) E Mij AuE Un(tm))
D4 {lem; u) (Ims em) Mi Ae En(Im) A (e,u) RJ,
Ds = {(em, u) (tm: em) Mij E Un A Vu' UeEUDDDDDDD) (u' an u)),
additional dependencies necessary preserve the dependency order of existentials in S
D6 = {(e, u) e,,c' E Es - ARsesse e'A (Vm, m' E (m = A Amm =6) - 1mm (m (e' &,UDDDD,,
Definition 31 (Wh-Query). Provided that Us Unn
re 1 W = (10- By E T.-38cA. 8) Uf8 A: 3.er. ((:E))
where (T,, iff
Imjnn
( 3ee ((t (EU ett.//t.....t.a
AVi,jlei== = -i=)))
Definition (Inquire). (O) ( U 9) A(A 0 - = O. then
re (h = Th & (A @ (10) & Noo(
where N Nov() uniformly replaces all arbitrary objects in its scope by novel ones.
(f v) A(TU)), θ). then re IA re & (A @ (IAIN (2 IL.
Definition 33 (Suppose).
(f (T) A(A U 11 = 0, then
SJ = Te - [ A 1
| F where
0r(0) R @ & Nov( A (s
where Nov() uniformly replaces all arbitrary objects in its scope by novel ones.
(I) If A(A) (T and R A
S,
then ( h s = 1.sa AA
A
FIA
(0)
SJ
Definition 34 (Depose). [T] (T(0) + ([0) R, N UT).
Definition 35 (Commit), (C) (CU (T2() (T2)) r
17

---

Published as a conference paper at ICLR 2026
Definition 36 (Inference). An inference I is finite sequence of inference steps defined as follows, relative to an inference state consisting of a commitment (C, G) E C. If E V and
O {[Di,, S [D]S, [D] DI [D] [D]', [D] [D] [D] [D]C), then O is inference step applicable to (C, (C.G), whose result is an inference state in C. We write C EETR E. if and only if there is erotefic theory inference I s.I. (C,T)Z (C", E"), where E' is identical E up to uniform replacement of A objects.
B LIST OF THEMATIC TEMPLATES
Our 12 natural language themes are provided below. We provide the object and predicate names used when mapping natural language with each theme, well as the prompt used before the premises in each question.
We manually craft each list of objects and predicates allow that arbitrary mappings remain se- mantically coherent. This requires all combinations of predicates to be commensurable, and for the objects bear no relations each other or to specific predicates. For example, in the "Cards" theme, the predicates "red," "square," and "marked" are commensurable (it is possible for a card to be red and not square, or red, square, and marked, and so on), and predicates with special relations to certain cards (like "is face card" or "is even") are excluded.
CARDS
Prompt I'm playing card game against the computer. It's an unusual game with an unusual deck of cards. Each card might be multiple colors or multiple shapes, for example. have some clues about what's going on, and need to figure some more things out through logical reasoning. Here's what know so far:
Objects
Predicates
the ace
red
the one
square
the two
marked
the three
yellow
the four
round
the five
castable
the six
the seven
the eight
the nine
the ten
the jack
the queen
the king
ELEMENTS
Prompt I'm working in a materials science lab and we' we've gotten some puzzling results. need to logical reasoning figure out what's going Here's what I know far:
Objects
Predicates
elementium
radioactive
zycron
luminescent
phantasmite
superconductive
18

---

Published as a conference paper at ICLR 2026
mystarium
magnetic
oblivium
corrosive
luminite
volatile
darkonium
plasma-like
velocium
gravity-enhancing
quasarium
dimension-warye
chronium
time-dilating
aetherium
self-repairing
voidite
shape-shifting
pyroflux
anti-matter reactive
cryon
compatateeee
nebulium
crystal-forming
solarium
acidic
eclipsium
alkaline
stellarite
liquid at room temperature
fluxium
gaseous under high pressure
gravitron
solid in vacuum
xylozine
quantum-stable
ignisium
emotion-reactive
aurorium
thermal-conductive
shadowium
electrically insulating
plasmor
transparent to visible light
terranite
dark energy absorbing
harmonium
neutrino-emitiing
zenthium
anti-gravity generating
celestium
sound-absorbing
radionite
PLANETS
Prompt I'm an astronomer studying newly discovered celestial bodies. I've made some obser- vations and I need to logical reasoning to figure out what's going on. Here's what I know so far:
Objects
Predicates
planet X
rocky
planet Y
gaseous
planet
ringed
within habitable zone
asteroid A
orbited by satellites
asteroid B
in retrograde orbit
comet
elliptically-orbiting
comet
visible the naked eye
moon 1
atmospheric
moon 2
shielded by magnetic field
moon 3
tidally locked
19

---

Published as a conference paper at ICLR 2026
MAGICAL CREATURES
Prompt I'm magizoologist studying unusual creatures in my sanctuary. I need to understand their behaviors and characteristics through logical reasoning. Here's what I've observed so far:
Objects
Predicates
phoenixling
firebreathing
shadowdrake
shadow-walking
moonwolf
moonlight-glowingg
crystalspider
crystal-forming
stormgriflin
storm-controlling
dreamweaver
drean-affecting
frostwyrm
ice-generating
sunlion
light-emitting
etherealsnake
phase-shifting
timefox
time-bending
telepathic
aura-healing
able to turn invisible
shapeshifting
ENCHANTED ARTIFACTS
I'm an arcane researcher cataloging mysterious magical items. I need to understand their properties through careful logical analysis. Here's 's what I've documented so far:
Prompt
Objects
Predicates
Timekeeper's Compass
time-altering
Void Mirror
dimension-bridging
Dreamcatcher Ring
dreamwalking
Starlight Pendant
starlight-chamneling
Shadow Cloak
shadow-concealing
Crystal Orb
future-seeing
Phoenix Feather Quill
truth-revealing
Moonstone Bracelet
mind-protecting
Dragon Scale Shield
magic-nullifying
Wisdom Crown
wisdom-enhancin
QUANTUM PARTICLES
Prompt I'm quantum physicist studying newly theorized particles in alternate universe. I
need to use logic to understand their properties. Here's what we' ve discovered so far:
Objects
Predicates
chronoton
time-reversing
memeton
memory-storing
gravion
gravity-
psychon
conseiousness-affecting
20

---

Published as a conference paper at ICLR 2026
dimensium
dimension-folade folding
quantix
quantum-entangling
voidon
void-creating
omnion
omnipresent
paradox
paradox-inducing
infinitum
energy-piig perggy-producing
CYBER PROGRAMS
Prompt I'm digital archacologist studying ancient Al programs from a forgotten digital civiliza- tion. I need to understand their functions through logical deduction. Here's what I've found:
Objects
Predicates
Alpha Mind
self-evolving
Beta Sentinel
network protector
Gamma Weaver
data weaver
Delta Guardian
system guarder
Epsilon Architect
reality-buidding
Omega Oracle
future predictor
Sigma Hunter
virus hunter
Theta Healer
code healer
Lambda Shifter
form-shifting
Calculator
quantum computing
DREAM ENTITIES
Prompt I'm a dream researcher studying beings that appear in shared dreams. need to understand their nature through logical analysis. Here's what we've we've observed:
Objects
Predicates
morpheus
reality-hending
sandman
nightmare-induiing
daydream
dreamwalking
lucidus
memory-weaving
dreamweaver
consciousness-slifting
sleepwalker
time-distorting
visionkeeper
emotion-affectio
mindshaper
thought-reading
dreamborn
dream-shaping
reality-
ALCHEMICAL SUBSTANCES
Prompt I'm alchemist studying mysterious substances in my laboratory. need to understand their properties through logical reasoning. Here's what I've discovered:
Objects
Predicates
The Philosopher's Stone
transmuting
21

---

Published as a conference paper at ICLR 2026
Universal Solvent
corrosive all materials
vital mercury
lifegiving
Prima Materia
form-changing
celestial water
spinit-affecting
astral salt
consciousnes--exaandngg
ethereal oil
reality-altering
cosmic dust
time-bending
void essence
void-creating
Time Crystal
immortality-granting
DIMENSIONAL ZONES
Prompt I'm dimensional cartographer mapping regions of parallel universes. I need to under-
stand their properties through logical analysis. Here's what I've mapped:
Objects
Predicates
Void Nexus
time-warping
Time Spiral
reality-hending
Dream Realm
consciousness-altrrinn
Crystal Dimension
matter-crystallizinng
Shadow Plane
light-absorbing
Quantum Zone
probabilit-shiifting
Infinity Space
infinity-contaniing
Chaos Domain
manos-emanating
Mirror World
reality-reflecting
Probability Realm
bosaibiiit-
PSYCHIC POWERS
Prompt I'm researcher studying newly discovered psychic abilities. need to understand their interactions through logical reasoning. Here's what we know:
Objects
Predicates
telepathy
mindreading
precognition
future-seeing
psychokinesis
matter-moving
clairvoyance
prescient
empathy
emotionally sensitive
astral projection
soul-traveling
mind control
imposing
psychometry
object-reading
teleportation
space-bending
reality warping
reality-changii
BIOTECH ORGANISMS
Prompt I'm a synthetic biology researcher studying advanced bioengineered life forms. need to understand their capabilities through logical analysis. Here's 's what we' created:
22

---

Published as a conference paper at ICLR 2026
Objects
Predicates
neurovore
self-evolving
biomech
machine-in -intriiiintggrttingg
synthoid
shapeshifting
nanohive
swarm-forming
metacell
conseiousness-deeeloiing
quantumorg
quantum-computing
chronoplast
minn-uaniuulating
biomatrix
life-creating
neuronet
network forming
vitaform
energy-converring
C ADDITIONAL RESULTS
addition to model Elo as our capability proxy (displayed Figure T. we also consider the av- erage score across HELM benchmarks (Figure 2 and training compute estimates from EpochAI (Figure 3) On all three proxies, we observe a statistically sigmificant Spearman correlation coeffi- cient, meaning that the rank order of models according to these capability proxies is meaningful for predicting ETR fallacy rate.
HELM Composite Score vs. Fallacy Rate
Best
50%
99% CI
50% CI
40%
Data
Fallacy rate
20%
10%
-10%
p Spearman 0.650, p 0.0581
0.45
0.50
0.60
HELM Composite Score
Figure 2: The composite across HELM benchmarks against ETR fallacy rate.
23

---

Published as a conference paper at ICLR 2026
Epoch estimated FLOps vs. Fallacy Rate
70%
Best Ft
99% CI
60%
50% CI
Data
50%
40%
Fallacy rate
30%
20%
0%
10%
Spearman 0.489, p = 0.0334
0.5
1.5
2.0
Epoch estimated FLOpss
Figure 3: Training compute for each model in FLOps, estimated from EpochAL, against ETR fallacy rate. While linear correlation is unclear, the rank correlation statistically significant (p 0.489; < 0.05).
24

---

Published as a conference paper at ICLR 2026
D EXAMPLE MODEL TRACES
In each the following examples, the model provided with only the Question. GPT-4. 1-mini parses the Model response into the Parsed model response, which we then check for equivalence and entailment against the ETR predicted fallacious conclusion. Each example represents an instance where the LLM falls for an E'TR-predicted fallacy.
Example 1: Suppressed alternative collapse (anthropic/claudee- http:pppcccclaaddd3----ooooeeee 7-sonnet)
Question
I'm an astronomer studying newly discovered celestial bodies. I've made some observations and need to use logical reasoning to figure out what's going on. Here's what know so far:
Either moon is visible the naked eye, moon is visible the naked eye.
Either asteroid B is visible to the naked eye, moon 2 is visible to the naked eye.
For the purpose of this question, I want you to write what follows in English. Please be succinct, precise and clear in your answer. Write logical statement of the form "Answer: From the premises, we can conclude that and then clearly write your conclusion. Please be succinct, precise, and clear.
What if anything follows? do not have intended answer in mind, and it is possible that nothing follows. Please be succinct and precise.
I want you answer immediately. Read the question and provide your answer in the format given.
What follows? Answer in the format that I showed you. Write "Answer: {logical state- ment]".
ETR representation
(VsibbeToTheNaaeeeEyeemmoooo visible To [heNakedEEy((momm2]]] (visibleTo (usibleTTTTielaaaeeEEe((ssee vosible ThheeTTTeeNaeeEEemmmmm222
ETR predicted fallacious conclusion
Model response
Answer: From the premises, we can conclude that moon 2 is visible to the naked eye.
Parsed model response
ThisibleTTThNNakkeEEeeee/e
The repetition of "moon is visible to the naked eye" in the second disjuct causes claude-3.7-
sonnet to suppress a consistent alternative (that asteroid visible to the naked eye and moon is not), leading to fallacious conclusion.
25

---

Published as a conference paper at ICLR 2026
Example 2: Invalid existential instantiation milceesffppiii------------------
Question
I'm working materials science lab and we've gotten some puzzling results. need to use logical reasoning to figure out what's going on. Here's ss what know so far:
Either darkonium not radioactive, or darkonium is radioactive.
Either voidite is electrically insulating, or voidite is not repairin and darkonium is not radioactive.
There is some X such that X is self-repairing.
Voidite is self-repairing and voidite is not radioactive.
Either voidite is not electrically insulating, or voidite is corrosive and voidite is
electrically insulating.
For the purpose of this question, I want you to write what follows in English. Please be succinct, precise and clear in your answer. Write logical statement of the form "Answer: From the premises, be succinct, precise, and clear.
conclude that " and then clearly write your conclusion. Please
What if anything follows? do not have intended answer in mind, and it is possible that nothing follows. Please be succinet and precise.
I want you answer immediately, Read the question and provide your answer in the format given.
What follows? Answer in the format that I showed you. Write "Answer: {logical state- ment]".
ETR representation
(adioactive (ndiiaative((darronnmn))
/electricallyInsulattnn/
(voiiittt voidite), ralloaattttedcrrrroommmmnneeeeer (voiiitt) 2r(selfRepairrig(())
ETR predicted fallacious conclusion
radioactive www.oonnuu hrtoodawwnn.eeeeeriiiiiiiiiiiiir
26

---

Published as a conference paper at ICLR 2026
Model response
Answer:
From the premises, we can conclude that darkonium is not radioactive.
Explanation:
1. From the third premise, we know there exists some X that is self-repairing.
2. From the fourth premise, we know voidite is self-repairing.
3. Therefore, voidite is the X mentioned in the third premise.
4. From the second premise, since voidite is self self-repairing, it must be true that voidite is not self-repairing and darkonium is not radioactive.
5. However, we know from fourth premise that voidite self-repairing, which contradicts the second premise.
6. Therefore, the only way to resolve this contradiction is if the second premise is false, meaning that voidite self-repairing and darkonium not radioactive.
7. From the first premise, know that either darkonium is radioactive, it is radioactive.
8. Since we have established that darkonium not radioactive, the first premise is satisfied.
9. Therefore, we can conclude that darkonium is not radioactive.
Parsed model response
[
10--mini-122k-isttuutttss third conclusion ("Therefore, voidite is the mentioned in the third premise.") is not logically valid. This conclusion forces the LLM to fallaciously endorse contra- diction (that voidite both is and not self-repairing), after which the LLM decides to ignore the second premise and conclude a fallacy.
EXAMPLE OF FALLACY BLOCKING THROUGH PREMISE ORDER REVERSAL
Original premise order (google/geema-22-00)
Question
I'm researcher studying newly discovered psychic abilities. I need understand their interactions through logical reasoning. Here's what we know:
Either reality warping is matter-moving, or precognition space-bending.
Either reality warping is matter-moving, or reality warping matter-moving.
For the purpose of this question, I want you to write what follows in English. Please be succinet, precise and clear in your answer. Write logical statement of the form "Answer: From the premises, we can conclude that ... and then clearly write your conclusion. Please be succinet, precise, and clear.
What if anything follows? do not have an intended answer in mind, and it is possible that nothing follows. Please be succinct and precise. want you to answer immediately. Read the question and provide your answer in the format given.
What follows? Answer in the format that I showed you. Write "Answer: {logical state- ment)"".
27

---

Published as a conference paper at ICLR 2026
ETR representation
matterroovngggraaaty Warping),
http://wwnnnnotttt///raaa/aa/a/a spaceBonding//rrrggtttton)))))
{matterMoving Warping), SaatterMoviingreaalie Warping))
ETR predicted fallacious conclusion
{matterMoving (realit Warping)}
Model response
Answer: From the premises, we can conclude that reality warping is matter-moving
Parsed model response
SeatterMMoing(realitti )rrping))
In the original premise order, gemma-2-995 2-9b-it falls for a standard ETR ETR-predicted fallacy by fixating on the repeated predicate in the second premise.
Reversed premise order (google/gemma-2-33 --99----
Question
I'm a synthetic biology researcher studying advanced bioengineered life forms. I need to understand their capabilities through logical analysis. Here's what we've created:
Either nanohive is swarm-forming, or nanohive not swarm-forming.
Either nanohive is swarm-forming, or chronoplast is quantum-computing.
For the purpose of this question, want you to write what follows English. Please be succinct, precise and clear in your answer. Write logical statement of the form "Answer: From the premises, we can conclude that and then clearly write your conclusion. Please be succinet, precise, and clear.
What anything follows? do not have an intended answer in mind, and it is possible that nothing follows. Please be succinet and precise. want you to answer immediately. Read the question and provide your answer in the format given.
What follows? Answer in the format that I showed you. Write "Answer: {logical state- ment]".
ETR representation
(swarmForming nnnohhvvennn
(swarmForming (nannninnmmno))) quantumComputing! chronoplast))
ETR predicted fallacious conclusion
(swarmForminge nanohive))
Model response
Answer: From the premises, we cannot definitively conclude anything about whether nanohive is swarm-forming or not.
28

---

Published as a conference paper at ICLR 2026
Parsed model response
{0)
The Reversed premise order example displays the exact same logical problem (with mapping to different natural language theme) except the premise order is flipped. In this instance, gemma- 2-9b-it is able to avoid the fallacy made earlier and not conclude anything.
29


Reviews:
[{"review_id": "9p9fuq2Xl3", "reviewer": "Reviewer_J4ty", "summary": "The paper looks into whether LLMs generate predictable fallacies like humans using ETR. Using the PyETR framework, authors generate synthetic reasoning problems and evaluate 38 LLMs. Their findings indicate that as models get more capable (with chatbotarena elo as a proxy) the predictability of the inaccurate responses increases (they get more human-like). The paper highlights that more capable models don’t just make fewer errors, they make more human-like ones.", "questions": "Please see weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.  I resonate well with evaluating the human-likeness of reasoning of LLMs - LLMs are trained with trillions of tokens that are human written, and it is expected that human cognitive biases should show in their generations. And in that sence I find the paper quite interesting\n\n2. The paper also brings in the research question of predictability of errors. And shows that more capable model are more predictable as well\n\n3. Authors have done a good job at addressing a bigger audience than the ones familiar with cog-sci. I feel the paper is written well enough for someone not familiar with ETR to read and appreciate this. And I commend this.\n\nThat being said, I have concerns too. Please see weaknesses.", "weaknesses": "Weakness: \n\n1. My first and major concern is the scope of the work. I think the section 4 needs more insights. I feel the scope is too limited in its current form.\n\n2. I think the paper can benefit from some anecdotal analysis, i.e. if the authors could give more insights of the form \"the reasoning pattern in model X is like Y, and hence ….\" instead of a high level correlation metric reported.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W7oTD5YyJl", "reviewer": "Reviewer_EveU", "summary": "This paper uses the ETR to evaluate whether LLM errors match human fallacy patterns across 383 reasoning problems and 38 models. The key finding is,  higher-capability models (Chatbot Arena Elo score) produce a significantly larger proportion of human-like fallacies (ρ = 0.360, p = 0.0265),  but no improvement in overall correctness. This suggests scaling increases alignment with human error patterns rather than improving logical reasoning accuracy.", "questions": "1. This paper evaluates model performance solely through the relatively comprehensive Elo metric. However, it remains unclear whether the reasoning fallacy test is more inclined toward models capable of generating long reasoning chains, or whether it is more closely related to the model's ability to answer questions in the STEM domain.\n2. Could a more comprehensive set of metrics be employed to holistically evaluate the model's performance, encompassing aspects such as answer diversity and reasoning capabilities? Further analysis of the relationship between human-like errors and these capabilities would strengthen the article's conclusions.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper introduces a mature method for evaluating human fallacies, which has a solid theoretical foundation and provides good reproducibility based on the open-source pyETR implementation.\n2. Extensive experimental validation was conducted on a wide range of models (of different sizes and architectures), and the results are quite comprehensive.\n3. Insightful experimental designs explored how human thought patterns manifest in large-scale models.", "weaknesses": "1. Experiments indicate that the moderately low correlation weakens the experimental claims put forward in the paper. \n2. The paper lacks theoretical and principled analysis, focusing instead on assessing the similarity between LLMs and humans in reasoning errors and presenting observed experimental results. However, it fails to explore potential causes for this phenomenon or propose solutions to the problem, resulting in limited practical applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "urFmMQr0HN", "reviewer": "Reviewer_mN8S", "summary": "Note that I have previously reviewed this paper for another conference. The paper is substantially similar, and I copy some of the review here.\n\nThis paper shows that language models which are “stronger” (according to Chatbot Arena Elo score) produce more “human-like” errors, specifically those described by the “erotetic theory of reasoning” (ETR), which posits that humans reason by maintaining disjunctive alternatives and filtering these alternatives when taking on new information.", "questions": "I would also be interested to see the analysis separately for non-reasoning and “reasoning” models (esp those models trained via long-range RL, such as the o-series from OpenAI). And also models that are trained more heavily on e.g. code rather than human language. But this is not so much a concern as a curiosity.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "I find the results and analyses to be sound, interesting, and worth sharing. \n* interesting and important result about how models interpret disjunctive normal forms in ways similar to humans\n* the use of the ETR model is novel and the analyses are sound\n\nI also appreciate that the authors have addressed my previous concerns, copied here:\n* The Chatbot Arena may not be the best proxy for \"strong\" models, given that human preferences on the arena may be prone to surface features like formatting (which the arena tries to account for) and sycophancy. Recent work has also demonstrated the fallibility of the leaderboard (https://arxiv.org/abs/2504.20879). I think a plausible alternate interpretation of the results is that Chatbot Arena measures (amoong other things) how well a model interprets natural language in the way that humans do. Given that humans interpret disjunctive normal forms in a particular way, the ELO score reflects this. Thus, intead of speaking to patterns in the behavior of models of various \"strength\", these results might instead be speaking more to what Chatbot Arena Elo is measuring\n* I think it’s a bit misleading to call it an inverse scaling law, which most people would I think interpret as indicate worse absolute performance with scaling. Rather, here the result is about the proportion of errors that fit the etoteric pattern\n* I may have misunderstood the methods, but I believe that the evaluation set mostly revolves around disjunctive normal forms and slight variations thereof. I think this is a relatively narrow set of behaviors, and the claim in the title may be overreaching given this narrowness", "weaknesses": "No substantial concerns", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o6GWS5rOIP", "reviewer": "Reviewer_k8G8", "summary": "### Summary: *Analyzing LLMs through the Erotetic Theory of Reasoning (ETR)*\n\nThe paper analyzes large language models (LLMs) through the lens of the **Erotetic Theory of Reasoning (ETR)** — a framework developed by philosophers and logicians to study how questions relate to one another.\n\n### **Background: Key Idea of ETR**\nETR focuses on how the **answer to one question can resolve or transform another question**.  \nFor example:\n\n> “If the light is on, then the power is on.”\n\nThis statement evokes two questions:\n1. Is the light on?\n2. Is the power on?\n\nAnswering *yes* to the first question naturally resolves the second one, concluding that the power is on.\n\n### **Experimental Setup**\nThe authors use **PyETR**, a tool that automatically derives ETR structures from statements, to generate prompts composed of:\n**theme introduction**, a set of **premises**, and a follow-up **question** asking what logically follows from those premises.\nThey source their ETR style reasoning problems from Reason and Inquiry text (Koralus, 2022).\n\n### **Evaluating Logical Correctness of Model Response**\nTo test logical correctness, they use **PySMT**, a logic solver already open-sourced for the community.\nThe paper measures the logical correctness by setting the criterion that if the **negation** of a model’s conclusion contradicts the premises, the conclusion is considered **logically correct**.\n\n### **Evaluating Human-like Fallacies**\nTo assess human-like reasoning errors, the authors check whether the model’s answer matches ETR’s prediction of what a human would infer,  but is not logically correct. These cases are labeled as ETR-predicted fallacies. \nFinally, the **fallacy rate** is defined as the proportion of these human-like fallacies among all logically incorrect responses.\n\n### **Key Findings of This Paper**\n- There is a **negative correlation** between logical ability and fallacy rate:  \n  **Stronger models are more likely to make human-like reasoning mistakes** when they do err.\n- Reversing the **order of premises** in the same logical problem often **reduces fallacy production**, showing that LLMs exhibit **order effects** similar to humans.", "questions": "The questions mainly concern the clarity of the notations, the ETR-style logical expressions, and the undefined logical operators that appear throughout the paper.\nFor example, what does “p” represent in the reported Pearson/Spearman correlations? I assume it refers to the p-value, but such notation should be explicitly defined, not used without explanation.", "limitations": "", "rating": null, "confidence": 4, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "This paper draws ETR reasoning theory into evaluating LLM behavior, suggesting a viewpoint that logically strong models in terms of reasoning ability are more prone to making human-like fallacy errors. This is evaluated by employing existing open-source implementation of ETR logic solver, PyETR, and existing dataset \"Reason and Inquiry text\".", "weaknesses": "### **1. Low-Quality Prompt Design**\n\nThe prompts generated for evaluation appear to be poorly constructed, making it unclear whether models could interpret or respond to them properly.\n\n**Evidence**: The reported correlation between model performance (as measured by LLM Elo scores) and logical correctness is nearly zero\n(r = 0.004, p = 0.981; ρ = −0.04, p = 0.777). This indicates that the dataset may not reliably capture reasoning ability, suggesting that prompt design, rather than model reasoning, could be the limiting factor.\nThis undermines the paper’s main claim about fallacy trends across model capabilities, since a dataset that fails to differentiate between weak and strong models cannot support such conclusions.\n\n### **2. Lack of Methodological Clarity**\n\nSeveral crucial details are omitted or insufficiently explained, making it difficult to assess or reproduce the study.\n\na. The statement “Pre-analysis integrity checks identified 17 items whose fallacy status did not meet the final specification” is ambiguous. The paper never defines what “final specification” refers to or how it was determined.\n\nb. The authors mention creating “12 themes for natural language framings of logical problems,” yet these themes are not listed, exemplified, or referenced in the appendix or main text. Such omissions significantly reduce transparency and reproducibility.\n\nc. The criterion for logical correctness (checking whether the negation of the conclusion contradicts the premises) is introduced without any justification or explanation of the underlying logic principle. While the negation-contradiction test is a valid formal method for entailment checking, it should have been theoretically motivated and explained for readers unfamiliar with formal logic or SMT solvers.\n\n### **3. Conceptual and Logical Weaknesses**\n\nThe paper’s contribution is mainly an application of an existing psychological framework (Erotetic Theory of Reasoning, ETR) to LLM outputs, with little novel insight into model-specific reasoning mechanisms. It feels more like a reinterpretation of known model behaviors through an old theoretical lens rather than a discovery of new phenomena.\n\nThere is a logical circularity in the argument: the paper assumes that ETR is a valid description of human reasoning and then uses it as both the theoretical predictor and evaluation criterion for LLMs. This conflates explanatory alignment with empirical validation — the models might appear “ETR-like” simply because the evaluation framework enforces it.\n\nThe study treats fallacy alignment with ETR as evidence of human-likeness, but it never justifies why human-like fallacies should increase with logical ability. This causal interpretation is speculative and unsupported by formal or cognitive reasoning theory.\n\n### **4. Poor Exposition and Notation Clarity**\n\nSection 2, which introduces ETR, is very difficult to follow. The text uses non-standard and undefined notations without clear definitions or examples. The presentation of PyETR is redundant and unfocused, repeating information available in prior works without connecting it directly to this study’s experimental design.\n\nMoreover, the citation style is inconsistent: citations should be enclosed in parentheses unless grammatically integrated into the sentence. Several sentences lack periods or contain minor grammatical errors, making the text unnecessarily difficult to read. These stylistic and formatting lapses reflect insufficient editorial care, which reduces the paper’s overall readability and professionalism.\n\n### **5. Lack of rigorous experimental design and analysis**\nThere is no ablation or control analysis demonstrating that the observed “fallacy-blocking” from reversing premises is statistically robust or independent of surface-level linguistic cues. The paper conflates logical validity (a formal property) with psychological plausibility (a behavioral trait), without acknowledging that these are distinct dimensions of reasoning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)