LogiConBench: Benchmarking Logical Consistencies of LLMs
Authors: Zheng CHEN
Chuan Zhou
Fengxiang Cheng
Yip Tin Po
Fenrong Liu
Yisen Wang
Jiajun Chai
Xiaohan Wang
Guojun Yin
Wei Lin
Bo Li
Haoxuan Li
Zhouchen Lin
Venue: ICLR 2026 Poster (2026)
Keywords: consistency
llm reasoning
symbolic reasoning
Primary area: datasets and benchmarks

Abstract: Logical consistency, the requirement that statements remain non-contradictory under logical rules, is fundamental for trustworthy reasoning, yet current LLMs often fail to maintain it even on simple inference tasks. Existing benchmarks for LLM logical consistency are not scalable, not diverse, and not challenging, with state-of-the-art models already surpassing 95\% accuracy. LogiConBench is the first benchmark that (1) generates unlimited logical rule combinations with precise labels, (2) provides controllable-depth graphs with explicit reasoning paths, and (3) remains challenging for state-of-the-art LLMs. To achieve this, LogiConBench automatically generates logical graphs where nodes represent symbolic propositions and edges denote reasoning relations. From these graphs, it samples lists of propositions, extracts reasoning paths, determines all consistent label lists, and translates them into diverse natural language expressions. While we release a 280K-sample corpus in this work, the framework can be scaled to generate unlimited data. To strengthen its evaluative significance, we evaluate 14 frontier LLMs on three tasks with varying difficulty levels, and find that the Enumerative task remains extremely challenging, with the best exact accuracy as only 34\%. Our code and data are available at https://github.com/Bellafc/LogiConBench.git.

Published as a conference paper at ICLR 2026
LOGICONBENCH: BENCHMARKING LOGICAL CON- SISTENCIES OF LLMS
Zheng Chen Chuan Zhou?.* Fengxiang Cheng'
Yip Tin Po' Fenrong Liu
Yisen Wang Jiajun Chai Xiaohan Wang Guojun Yin Wei Lin
Bo Li Haoxuan Li Zhouchen Lin
'Computer Science and Engineering, Hong Kong University Science and Technology State Key Lab of General AI, School of Intelligence Science and Technology, Peking University Institute for Logic, Language and Computation, University of Amsterdam "The Tsinghua-UvA JRC for Logic, Department of Philosophy, Tsinghua University 'Institute for Artificial Intelligence, Peking University 'Meituan "Center for Data Science, Peking University zchenin@connect wwwccconnnttuus...i.. .us.uutt bli0cse. ust .ı, hxli0stu, pku edu, en, zliněpku. edu. en
ABSTRACT
Logical consistency, the requirement that statements remain non-contradictory un- der logical rules, is fundamental for trustworthy reasoning, yet current LLMs often fail to maintain it even simple inference tasks. Existing benchmarks for LLM logical consistency are not scalable, not diverse, and not challeng- ing, with state-of-the-arr models already surpassing 95% accuracy. LogiCon- Bench is the first benchmark that (1) generates unlimited logical rule combina- tions with precise labels, (2) provides controllable-deptt graphs with explicit rea- soning paths, and (3) remains challenging for state-of-the-art LLMs. To achieve this, LogiConBench automatically generates logical graphs where nodes repre- sent symbolic propositions and edges denote reasoning relations. From these graphs, it samples lists of propositions, extracts reasoning paths, determines all consistent label lists, and translates them into diverse natural language ex- pressions. While we release a 280K-sample corpus in this work, the frame- work can be scaled to generate unlimited data. To strengthen its evaluative significance, we evaluate 14 frontier LLMs on three tasks with varying diffi- culty levels, and find that the Enumerative task remains extremely challenging, with the best exact accuracy as only 34%. Our code and data are available at
INTRODUCTION
Logical reasoning is category of capabilities essential to trustworthy Al (uu et al. 2024 2025a)
Among these, logical consistency refers to the property that a set of statements does not contain contradictions under logical rules (Huang & Chang 2023 Liu et al. 2025). Maintaining model consistency central for reliable conclusions and reasoning (Cheng et al. 2025). However, recent studies show that LLMs frequently generate self-contradictte
reasoning outputs, even for simple inference tasks (Calanzone et al. 2025 Ghosh et al. 2025) Paleka al, 2025 Song et al. 2025). For example, suppose sentence entails H. and H entails Z: by transitivity, one should infer that P entails z. However, models may produce inconsistent judgments which violate logical principles (Li et al.) 2019). Such inconsistencies worsen the local inference and also propagate through reasoning chains, which ultimately disrupt the overall reasoning process.
Existing efforts benchmarking the logical consistency of LLMs can be summarized as follows. Be- liefBank (Kassner et al. 2021) generates constraints over entities based on a commonsense knowl- edge base. EntailmentBank (Dalvi et 202T) provides multi-step entailment trees, where answers
"Equal contribution.
Bo Li, Haoxuan Li, and Zhouchen Lin the corresponding authors.

---

Published as a conference paper at ICLR 2026
Table I: Comparison of logical consistency datasets of size, depth, operators, reasoning
path availability, scalability, and rule count. The detailed explanations can be found in Appendix B
Dataset
Size
Depth
Operators
Reasoning Path Scalahility # of Rule
BelicfBank
12,525
No No
EntailmentBank
1,840 avg.
Yes
No
LFC
~2,000
up to A, V, 7,
No
No
Set-LConVQA & Set-SNLI
13,779
up to 5 A, V. -, 7,
No
No
51
LogiconBench
280K
32
V. 2,76
Yes
Yes
280K
Accuracy of Frontier LLMs Across Logical Consistency Benchmarks
Question
Given three possible lists of Boolean Labels which the three
all
LegiConBeoch EntalllmentBank
BeliefBank
consistent.
Sel-LGumVQA and SeI-SNLI
99959697
9699
95979695
98969799
Answer
Answered by the LLM
Either oor not ogen. (pV mu)
the
T
T
Accuricy (%)
The light is on. (N)
T
F
34
The rings and light is not on. [s A-)))
7.7
Consistent
Inconsistent
806-5
grok-4-Gust
gumini-25-pro dapssenk-1-05226
Possible
(a) Accuracy comparison
(b) Enumerative task
Figure 1: (a) Accuracy of frontier LLMs on LogiConBench vs. existing benchmarks. Existing benchmarks are saturated, while LogiConBench remains challenging and discriminative. (b) Il- lustration of the enumerative task. Given three statements, the LLM must list consistent label assignments, but often outputs incomplete incorrect lists.
are supported by explicit reasoning steps. LFC datasets (Ghosh et al. 2025) are built by transform- ing knowledge graphs into logical fact-checking queries. Set-I.ConVQr VQA and Set-SNLI (Song
al. 2025) extend Visual Question Answering (VQA) dataset and Natural Language Inference (NLI) tasks into set-level format and check for consistency across multiple sentences.
Despite their contributions, these datasets have several limitations. As shown in Table I first, their sizes and rule counts remain relatively small, which becomes inadequate for realistic logical reason- ing scenarios. Secondly, their depths are shallow and the reasoning paths are absent, preventing the models from fully capturing multi-step reasoning chains. Thirdly, all of them either human- written or derived from existing datasets, highly limiting their scalability. Finally, our empirical results in Figure show that the most advanced L.MMs such gpt-5 and grok 4-fast already achieve above 95% accuracy on these datasets, which suggests that they are no longer sufficiently challenging be used as logical consistency benchmarks for frontier LLMs.
To address the aforementioned limitations, we propose a framework named LogiConBench that automatically constructs complex and large-scale datasets for logical consistency evaluation. Logi- ConBench first generates logical graphs, which can expand indefinitely and record reasoming re- lations, where nodes represent symbolic propositions and edges denote reasoning relations. From the graph, we randomly sample lists of propositions, extract their shortest reasoning paths, and propagate Boolean labels along the edges according to logical rules, by which we collect all truth- value assignments that keep the sampled propositions consistent. To enhance structural variety, further apply symbolic rewriting techniques to produce logically equivalent formulas. Finally, propositions are translated into natural language through templates and lexical substitutions.
Through this construction, LogiConBench directly overcomes the above limitations. First, it can generate unlimited logical rule combinations automatically with precise consistency labels, which supports scaling up and covering a wider variety of logical rules. Second, graphs with control- lable depths provide explicit reasoning paths for multi-step inference. Finally, benchmark
2

---

Published as a conference paper at ICLR 2026
remains challenging for otate-of-the-art LLMs, as shown in Figure 1 which shows its significance for benchmarking the consistencies of current frontier models.
To systematically evaluate logical consistency reasoning.
design two primary benchmarking tasks: Discriminative task, determining whether a given Boolean Label list can lead to contra- diction for the given statements, and Enumerative task, enumerating all consistent Boolean Label assignments for the given statements. To capture performance across different levels of difficulty, further introduce variants for the three tasks. Our large-scale experiments with 14 frontier closed and open-source models reveal several consistent findings: Results show that frontier models (e.g.. gpt-5, -rok-4-ao-----aast deepseek-r1-0522
gemini- 5-pro) achieve 85-95% accuracy in Discriminatiiv task, but Enumerative task remains extremely difficult, where only gpt-5 reaches averagely 34% exact accuracy on the subset, while most other models stay below 1%, and the best model only reaches 42% consistency rate for the Generative task. In difficulty-basse found that the accuracy of Task on hard samples drops to around 80% for frontier models and below 40% for smaller ones, while in Task 2 on Easy samples, the best model (gpt-5) improves average exact accuracy to 58%. Moreover, whether the natural-language statements are common- sense, counterfactual, or human-like has little impact on the results. We summarize the contributions as follows:
analysis, we
LogiConBench Framework. We propose novel framework that automatically constructs diverse and scalable logical consistency data through newly designed logical graphs, where the nodes represent propositions and logical relations
the edges.
LogiConBench Dataset. We produce a large-scale corpus of 280K samples with varying difficulty levels, which covers diverse and important logical reasoning rules.
Evaluation and Analysis. We conduct experiments across varying levels of difficulty on 14 sfute-of-the-att LLMs, which shows that even strong models fail on more than half of the tasks, confirming that logical consistency reasoning remains highly challenging.
PRELIMINARIES
Our benchmark is grounded in standard natural deduction rules (Liu & Stokhof
2024 In particular, ( - v)
we adopt the introduction rules for five logical operators: implication 7-, (4
disjunction Tv: 4p (4p V v)) eb = (p v)); conjunction TA 4, 1 (40 1) negation 7
1) (-p)); and biconditional L.:: (y - wb) A (v + p) (p + b). These rules serve as
the fundamental logical primitives, which allow atomic propositions be systematically combined and deduced into complex reasoning forms and axiomatic structures (Chiswell & Hodges, 2007]
Westerstahl 2022). hence, they serve as the foundation for our benehmark. detailed illustration
given in Appendix
3 CONSTRUCTION OF LOGICONBENCH CORPUS
LogiConBench is constructed through a pipeline that generates datasets with k: = 2,4,4,5, where k denotes the number of propositions in single sample, and, detailed in Appendix [ the data exhibit diverse distributions of atoms and logical operators.
3.1 THE LOGICAL GRAPH
Granularity of the logical graph. Constructing multi-step reasoning trees or graphs an ef- fective way capture the reasoning process and to enable scaling across diverse corpora. Instead of relying on pre-defined axioms or costly natural language explanations, our approach deliberately starts from the level of atomic propositions, the finest granularity of reasoning. By composing these atomic elements through basic logical rules, we construct Logical Graphs that naturally re- flect complex reasoning structures.
Logical Graph Construction from Basic Rules. Section 2 introduces five fundamental induction logical reasoning rules, which be composed into infinitely many deductive forms, including axioms. Specifically, Implication (-) and Biconditional (+) are expressed only on graph edges, while Disjunction (V) and Conjunction (A) are expressed only nodes. Negation (-) can present
3

---

Published as a conference paper at ICLR 2026
Randantly Dijanction ateor's orgicinn déjoin
V
Inslication can Imply p
Starting Inom the zoom
Negation pare contradictury
proportinn
pvu
Implication Implied.
pVuvs
p
Conjunction Randomily negacion conjotn
S
Logical Graph Generation Mechanism
Logical Graph
1tep1: Generate the Logical Graph
Step 2: Select K Nodes
Step 3: Label
Step4: Translate
S1
(
(-p)
Uither the light
PV mM
se the tlorr
$2
S3
rriggs
AAA -p
SA-pp
The overall pipeline of Logicon including steps: logical graph generation, node selection, truth labeling, and natural language translation.
Figure 2:
both on nodes and edges, where an edge with contradiction represented "x". For example, we begin from fixed atomic proposition and generate its negation -p, the conjunctive expansion p A q (where is a randomly sampled atom from a pre-defined atom list with length 8), and disjunctive expansion p q that implies p. These yield the initial structure:
PVq - p PAq.
p x >p
Subsequently, each newly added node can be further expanded in the same manner, which allows the Logical Graph to grow indefinitely through iterative application of these basic rules.
3.2 RANDOM SAMPLING AND LABELING ALONG REASONING PATHS
Random sampling. To stratify difficulty, we sample, for each {2,3, 5). exactly 10,000 exam- ples where the target set contains distinct nodes. Since require every example to admit at least one inconsistent label list, we avoid picking nodes that are too distant in the global graph, which weakly constrained and satisfy all the truth labels. So we ensure they remain within bounded graph distance set to so that every example preserves sufficient local constraints. We also expect the formulas within example have diverse complexity, nodes S = (11, .,, 2 with a uniformly distributed number of atoms in their formulas.
randomly sample k distinct
Path extraction. After sampling from the logical graph, obtain the target set S. We then extract a small subgraph that connects all targets by solving the Steiner tree problem (Hwang & Richards) 1992) with permutation-basel shortest-walk search, which results in ordered edge list & [u, (1, U1)..., (um, fm, Um)] that we use for labeling. The details can found Appendix
Constraint semanties. Edges encode pairwise truth truth-compatibilityy of their endpoints via the rule set
+: ((T,)) (F)
((T,T), (T, (F, (FT,T) F)},
LOCIC
(ITF
Labeling via DFS propagation. We perform a depth-first search (DFS) over & that incre- mentally assigns Boolean values to nodes according to LOGIC.RULES The projections all rule-consistent assignments onto the target nodes collected as consistent. each target list of nodes has 2* possible Boolean assignments in total, the remaining assign- ments form the inconsistent.list.. consistent lists and inconsistent lists for downstream evaluation. An example of labeling is shown in Appendix
Since
We ensure that every sample contains both nonempty

---

Published as a conference paper at ICLR 2026
Setting Determination
Setting 2: Enumeration
Setting Generation
Instruction
Instruction
Statements Generation
Given three statettettss deteeming whether a list Boolean Labels lcods to coucradiction Each
Gives three statements, mummerate all possible lists of Boolean Labels witch make the three staiements consistent
Sppllit the g global premises set into groups.
2 Generalte 10 natural-language statements
labol curnespondd du to one alaiement onder
promises per group connectvess
show-shotEBs Example
Few-shot Example
3. Convert each generated statements into computable logical form.
Statements
Statements
Fither the light bacccsuss the dour open (rV val The babcc sts (P)
Bither Usclkght thedese spt
The light on.
iearind testion GrA-pp)
The rap
(kp
Consistency Checking
Beatoning Parth
(.P)
Reasoning Path
Enuuserate cross-group statements somhinations with one statement per group 2 Evaluate each combination using the locical graph.
(e mpp) (--03 (P) 1000-0p qe mM)
(+ -p) (-03000 V-)
Yes, Boolean
Figure 3: Illustration of the three task tasks in LogiConBench.
3.3 REWRITING
After sampling k = 2/3/4/5 target nodes (10,000 examples for each k) in the logical graph gener- ated by the five induction rules, we apply the set of elimination rewrite rules as shown in Appendix G every node in the sampled subgraphs (Liu Stokhof 2024). For each node, a rewrite rule pro- duces a valid transformed formula, the rewritten form is retained as an additional node. Several rewrite rules (e.g., equivalence elimination, conversion to conjunctive normal form (CNF) or dis- junctive normal form (DNF)) are applied at the formula level, since nodes may contain multiple atoms, thereby producing multi-atom rewrites.
3.4 SYMBOLIC-TO-NATURE LANGUAGE TRANSLATION
Following Morishita et al. (2023), we convert symbolic formulas into natural language sentences. For each atomic proposition, we randomly select NOUNs, ADJs, and VERBs drawn from Word- Net (Fellbaum) 2005), where we obtain a large variety of words to prevent models from overfitting to fixed wordings and better approximate real-world language diversity. We apply negation rules to cover both affirmative and negative variants of atomic statements. For composite formulas, we structural templates that map logical connectives (e.g., A, V, -) into corresponding natural language operators. The full set rules and templates is provided in Appendix]]
4 BENCHMARKING LOGICAL CONSISTENCY
Structure. As shown in Figure 3 in this chapter, we introduce three primary benchmark tasks: Task (Discriminative task), Task 2 (Enumerative task), and Task 3 (Generative task). To capture more diverse performance patterns, hard samples. where consistent and inconsistent label lists differ by only one element, and we also evaluate samples with short reasoning paths include label completion variant. For Task fine-graine -rained assessment of model reasoning. We and short statement length, which enable a more fine- also evaluated task on languale statements in common-sense, counterfactual, and human-like types, which shows little impact on the model performance.
further refine them in Chapter For Task 1, we evaluate on
Testing Models. We evaluate diverse collection of 14 models on the LogiCon- Bench, covering state-of-the-ant proprietary systems open-source models: grok-4-fast (xAl 2025). gwen3-235b-a222
well as large and small size
(Yang al. al. 2024). gpt-5 (OpenAl (025a). (Yang ibinnsructt Diang et al. 2024)) o3-mini (OpenAT] (0255). mixtral
2025).
phi reasoning-p-uu (Abdin et al. 2025). 1lama-3 1-8b 1a----innss iinntruut (Grattationi
ct al. 2024). 11.ma-3.1- 3-05b instruct Grrattafiori et al. 2024). gpt-4o 40 (Hurst et al,
2024 gemin1-2.5-pm (C (Comanici al. 2025). Prepseek-r1-0521 (Guo al.) 2025 sauude-ccnnet4 ( (Anthropic 2025). ). and claude-3. 5-hakk (Anthropic 2024).
For each experiment, consider three evaluation settings: zero-shot (without examples in prompt), few-shot (with three examples in a prompt), and few-shot with reasoning paths (with three examples plus corresponding ground-truth reasoning paths, as shown in Figure Each setting is
5

---

Published as a conference paper at ICLR 2026
Table 2: Performance on Task 1 (Discriminative Task) across different numbers of statements (2-5). Results are shown for consistent samples, inconsistent samples, and their overall average accuracy under three evaluation setups: zero-shot, 3-shot, and 3-shot with reasoning path. Back- ground cell colors range from light to dark, indicating increasing values within each column.
Model
Accurey on Statements
Accuracy on 3 Streemnts
Accuracy 4 Stasements
Accunay on Statements
Coo.
Incoes.
Overall
Con.
Incon.
Overall
Con.
Incon.
Overall
Con.
Lacon.
Overall
grok-4-fast
95.30%
40%
94.35%
70%%
97.30%
15..01%
97.21%
74.90%
96.90%
85.300
spi-3
91.70%
00.05
92.50%
12.80%
90.05%
78.40%
60%
deeppeck-rl--n
94.70%
89.00 60%
91.60%
05.70%
40%
94.90%
64.65%
63.70%
95. 1..0%
79.50%
de loannnng
clande sonnet-1
quent-2350-a220
61.70%
49.00%
56. 10%
57.90%
69.45%
73.90%
53.80%
6010055
84.30%%
85.00%
72.15%
69.40%
39. 20/6
54 20%
54.70%6
91.40%
65.30%
60.45%
54.30%
34.10%
58. 905
91.30%
56.5542
63.95%
gemini--25 2.5 pro
58,0056
59.300%
58.05%
58.90%6
07.
63.007%
51.60%
67. 30%
59.4556
47.660%
66. 10.550%
57,0516
Blama-1. inniimii
64.30%
43.900
54.01%
68. 37%
36.40%
52. 305
72. 60%
37. 1.99999
55.25%
65. 3053
36, 5055
$1,05%
queen2 5-7b-instract
54.65%
43.601%
42 80%
43.20%%
phi-4 reasoring plus
52.85%
50%
57.20%
73.00%
65.40%
45.35%
mixtral ix7b-inetrett
39.2011
51,90016
89.3056
25. 25.301
57.55%
51.50%
22.2016
51,851
79,201
14.30%
46.7516
03 mini
58.10%
43.30%
89.80%
56.60 6076
35.40%
46.00%
61.30%
27.10%
44.20%%
64.00%
16.40%
40.201
clande-3 3.5-hakk
40.50%
44 60.
42.55%
82. 8096
15. 80.%
49. 3006
93.40%
8.805
$1.1098
95.10%
2.40%
48.7548
Baaa-3. 1-18 ins-intrrctt
36,00%
32.00%%
34.000%
52.70%
18.401
35.55%
54. 71.7
10.10%
$2,4058
56.90%
6.70%
31.80%
gre-4o
36,909
20.005
32.955%
24. 000%
19. 10.4
31. 11.70%
58 50%
11.80 8.90
$5.15%
47.40%
14 14.00%
31.15%
03.70%
30%
30%
96.30%
76. 30%
70.%
gpe-5
92.70%
85.40%
57.50%
$6
96,10
23.90%
95. 10%
54.50%
deepeesk-r1--S22
89.00%
94.90%
51 91.9555
95 D05
94.5%%
89 89.00%
96,
95 8055
79.95%6
clande sonnet-4
61.80%
54.40%
75. 10%6
52.40%
89.8 89.8018
37.00
90.8066
38-2055
06.20%
3-shdd loaning
quent-238b-a220
58. 70%
56.00%
72.35
95.00%
86, $6.505
91 21.300
99.20%
6- 1000
53 70/5
78. 50%
gemini-2 S-pro
80.809
59 50%
85, 155
57.50%
86, 1098
86 S05S
SS 90%
47.4
911558
76.20%
97.30%
86.75%
Ilama-3. in0----instuuc
66.87%
47.50%
57.30%
72.00 72.00%
18.00%
55.00%
50.5%%
37,909 77.90%
62.50%
38. 50%
50.50%
gwenz 5-7b
54,25%
54,10%
52.50%
08.701
46. 00%%
47.6516
phi-4 reasoning plus
55.55% 96.90%
19.90%6
$4.50%
52.30%
70.20%
32.70%
60.
48.15%
Siixtal-Sx7b-innenne
65.40%
44.0
2015
4056
23, 70%
57,05%
60%
29, 205
55.55
87-0055
14. 5056
50.755
03-mini
95.60%
10.10%
75,8556
95.00%
46.30%
70,055
94.50%
572209
94.200%
0 16.700%
25.4556
cland-3 S-haika
41 ..0%
52. 80%
46 0019
48%4
20, 40%
$4.400%
01 60%
6.70%
S01 555
2402
3. 5.10%
48 685
Ilama Sb instruct
40.10%
34.20%
37.15%
55.90%%
16. 10%
36.00%
53.90%
10.10%
$2.000
58.00%
6.70%
32.35%
gpi-lo
60.60%
54.60%
57.80'
77.40%
26.00%
51,70%
30%
34,70%
$4.50%
62.70%
20.50%
41.60%
3elas learning wi reasoning palk
grok-4-fast
gpo.5
96.10%
92.90%
53.500%
20%
93.201 2015
93.00%%
98. 30%
96.501
95.05%
92.55%
99,50%
97.70%
93.00%
79 60.6
50.50%
100.00%
80%
89.70%
deepwekk-rl-099
90.60%
50%
92.55%
93.705
06, 60%
95.
90%
40
69.60%
96 80%
2015
clande sconess4
67.50%
92. 30%
90.0155
$4, 10%%
91. 91.000%
72.55%
46 10%
05.000%
50.555
40.20%
95. 005
67. 60.6
quen3-235b-a22 2356-222b
73.80%
99.78%
86: 75%
98.70%
89, 900%
94. 30%
100.00%
76.30%
S5. 15%
99.40%
59. 31/
79.35%
gemini-22 2.5 pro
84.00%
30.90%
92.45%
10%
99.40%
93.75%
$2.800
99.70%
91.25%
50:901%
95.500%
24,7096
Ilama 1--505----aa instruct
71,907%
51.80%
61.85%
35.3
27,80%
62.307%
77.60%
41.30%
59.15%
70.80%
45. 45.50%
58.15%
quie22 5-7b-instractt
95. 25..00%
600.55%
72.201%
59.45%
61.
57,40%
59.50%
40%
52.35%
phi-4-reasoning plus mixtral-Sx mixtral-&x7h-iinemme 7h-ineemett o3 mini
14.90%
90.30%
35. 6011
34.00
38.801%
61.70%
34.10%
75.00%
54.55%
36.20%
75.00%
55.9096
50.50%
70%5
60.009%
709%
31. 11.90%
15.55%%
40%
22.00%
2095
.0.00%
16 50%
79.75%
58.70%
44.70%
51.20%
60.80%
36.20%
48.50%
61.40%
32.30%
46.85%
62. 10%
12.90%
37.50%
clande-3 5- hatku
49.40%
62. 40%
$5. 90%
89. 2008
21. 40%
55. 30%
93. 30%
9.50S
$1.40%
96. 70%
3.70%
50.20%
Haama-. 1-B8 instruct
55.70%
45.30%
30,,50%
$4.00%
27. 6016
55. 900%
$9.705 70%
12.209
54.554
85. 99799
1335905
49.9006
gpe 4e
34.30%
59.30%
46.80%
20.20%
38.55%
54.80%
35.00%
44.90%
45.50%
32.30%
38,9015
applied across datasets of statement size k = 2, 5. All evaluations conducted in a single- round format with the temperature fixed at O, unless otherwise specified in the model card. For every configuration, we randomly sample 1,000 instances from the dataset for evaluation. The evaluation metrics details can be found in Appendix]] !
4.1 TASK 1: DISCRIMINATIVE TASK
Formulation of Task 1. Task I focuses on determining whether a given list of Boolean labels assigned to l logical statements leads to contradiction, for statements size k = 2,3.4.5.
Findings of Task
The experimental results for Task 1 is presented in Table deepseek-r1-0522 and
The most advanced LLMs. including gpt-5, grok-4- Brok-4-fasse gemini-2. 5 5-pro, consistently achieve accuracies in the 85-95% range. (2) Models show stable bias patterns across settings and number of statements k: some (e.g., 2laude-3.5-ha.1. 5-hhaiku) per- form better on consistent than inconsistent statements, others (e.g.. 1iii---reaooing show the opposite trend, while third group (e.g.. gpt- 5) consistently favors inconsistent cases, with the gap widening as task size grows. (3) Prompting improves performance: average accuracy rises from 59.21% % (zero-shot) to 65.70% (few-shot) and 67.58% (few-path), confirming the value of reasoning- path supervision. (4) Increasing k substantially raises difficulty: accuracy drops from 66.87% at k 2 to only 59.59% at k 5.

---

Published as a conference paper at ICLR 2026
Table 3: Performance in Task 2 (Enumerative Task) across models and prompting settings. We report Format (Executable rate), Exact (Exact accuracy of all consistent lists), and F/ (partial cor- rectness) under three evaluation setups: zero-shot, 3-shot, and 3-shot with reasoning path. The best model for each metric and setup is shown in bold, and the second-best is underlined.
Model (mode)
Accuracy on Statements
Accuracy on Statements
Accuracy on Statements
Accuracy
Statements
Format
Exact
F1
Format
Exact
F1
Format
Exact
F1
Format
Exact
Fl
grast
0.280
0.072
0.459
0.433
0.167
0.301
0.517
0.034
0.230
0.517
0.000
0.194
gpt-5
0.982
0.383
0.751
0.972
0.439
0.775
0.962
0.250
0.736
0.971
0.073
0.664
dbepseek-rl-0528
0.892
0.270
0.099
0.724
0.000
0.068
0.565
0.032
0.041
0.528
0.009
0.400
claude claude-somne-44
0.880
0.043
0.494
0.843
0.317
0.813
0.000
0.193
0.754
0.000
0.123
Zero shot leaniin
qwen33-235b-al a22b
0.959
0.021
0.523
0.875
0.031
0.437
1,000
0.000
0.207
1,000
0,900
0.135
gemini-2.55pm
0.977
0.058
0.522
0.950
0.120
0.409
0.914
0.000
0.099
0.840
0,080
0.040
Ilamma 3.1-u.3..-4555 3.1-405bbbb instruct
0.530
0.017
0.267
0.604
0.002
0.487
0.706
0.005
0.489
0.612
0.003
0.218
1wen--2.5-700 7b-insstttt
1,000
0.009
0.455
1,000
0.006
0.452
1,000
0.000
0.367
1,000
0.000
0.240
phi reai----esnningg--- plus
0.947
0.017
0.196
0.937
0.000
0.093
0.918
0.055
0.937
0.000
0.069
mixtral-Sxx 8x7b instruct
1.000
0.003
0.109
0.972
0.000
0.097
0.964
0.000
0.062
1,000
0.000
0.019
03-mini
1,000
0.082
0.014
0.067
1,000
0.000
0.010
claude-3.5 1.5-hiiku
1.000
0.027
0.406
0.940
0.000
0.263
0.957
0.000
0.141
0.890
0,000
0.059
1aamna-,--S--iistrwe
0.351
0.002
0.287
0.430
0.000
0.326
0.333
0,000
0.074
0.435
0.000
0.058
spt-4o
1,000
0.018
0.111
0.980
0.000
0.065
1,000
0.000
0.025
1,000
0.000
0.024
grok-4-fast
0.233
0.067
0.147
0,448
0.034
0.255
0.533
0.100
0.298
0,300
0,000
0.174
gpt-5
0.997
0.392
0.768
0,990
0,443
0.759
0.971
0.267
0.776
0.985
0.124
0.678
0.833
0.069
0,108
0.681
0.034
0.069
0.590
0.004
0.047
0.599
0.008
0. 0.468
claude-sonict-4
0.050
0.494
0.315
0.218
0,720
0.000
0.124
3-bbtt loarning
qwen3-2350-a22b
0.897
0.006
0,564
0.914
0.006
0.470
1,000
0.000
0.179
1,000
0.000
0.122
gemini-2.5
0,990
0.045
0.511
0,949
0.012
0.127
0,939
0,030
0.085
1,000
0.037
0.036
1atma-3.1-400--imm
0.652
0.023
0,257
0.680
0.005
0.255
0.667
0.006
0.181
0,640
0.000
0.436
qwen-2.55 2-2----eee
1.000
0.028
0.518
0.995
0.000
0.478
0.995
0.005
0.370
1.000
0.000
0.257
phi-t-rasoonnngplls
0.962
0.261
0.949
0.000
0.057
mixtral-8x 11--8xx-h---tet
1.000
0.000
0,233
0.983
0.000
0.079
1.000
0.000
0.018
1,000
0.000
0.074
03-mini
0.963
0.037
0.070
1,000
0.000
0.048
0.949
0.017
0.065
1,000
0.000
0.006
claude-3 5-haiku
0.943
0.042
0.489
0.953
0.003
0.301
0.984
0.000
0.144
0.907
0.000
0.063
Ilama-3. 3.1-hhinnstmacc
0.293
0.010
0.229
0.458
0.000
0.327
0.300
0.000
0.044
0.400
0.000
0.029
gp
0.106
0.955
0.000
0.136
1,000
0.000
0.006
0.973
0.000
0.031
3-shot learning wf reasoning path
grok-4-fast
spt-5
0.133
0,980
0.204
0,290
0.815
0.533
0.167
0.812
0.381
0.533
0.299
0.100
0.834
0.249
0.582
0.155
0.000
0.709
0.293
despseck-r1-0528
0,800
0.120
0.112
0.784
0.054
0.094
0.625
0.036
0.407
0.625
0.010
0.619
slaude-somned
0.900
0.053
0.507
0.800
0.110
0.333
0.760
0.000
0.229
0.790
0.000
0.225
qwcn3-2350-a22
0.963
0.009
0.460
0.915
0.017
0.492
0.964
0.000
0.278
1,000
0.000
0.170
gemini-22.5pp
0.980
0.124
0,711
0.962
0.085
0.235
0.972
0.100
0.090
1,000
0.018
0.045
Ilama- 3.1 1-405 instruct
0.567
0.325
0.221
0.642
0.008
0.526
0.000
0.481
qwen-2.5. 2.5-innnttt
1.000
0.032
0,539
0.991
0.012
0.479
1,000
0.009
0.394
1,000
0.004
0.263
phi 4hii4-ree wew.---ssonnnggblee plus
0.953
0.030
0.221
0.963
0.000
0.096
0:957
0.000
0.068
0.927
0.000
0.063
mixtral 8x7b insiruct
1.000
0.018
0.046
1,000
0.000
0.127
1,000
0.000
0.000
0.94]
0.000
0,020
03-mini
1,000
0.054
0.087
0.818
0.000
0.015
1,000
0.000
0.085
0.925
0.019
0.047
claude-3. huiku
0.938
0.047
0.547
0.957
0.003
0.302
0.947
0.000
0.152
0,903
0.000
0.075
Iaama-3.1 3.1-Sb-imstruct
0,334
0.023
0,283
0.410
0.006
0.342
0.241
0.000
0.052
0.335
0.000
0.045
spt-4o
0.970
0,037
1,000
0.000
0,042
1,000
0.000
0.019
1,000
0.000
0.001
4.2 TASK 2: ENEMERATIVE TASK
Formulation of Task 2. Task 2 focuses on the task of enumeration. Given set of logical state- ments, the model is required to enumerate all possible lists of Boolean label assignments that remain logically consistent. Unlike Task which only verifies whether a specific label set leads to a con- tradiction, Task 2 demands a complete search over the label space. Therefore, LLMS must account for all logical consistency constraints, thereby eliminating the potential for shorteuts in consistency evaluation. As result, Task 2 poses more challenges to the logical consistency reasoning of LLMs.
Findings Task 2. As shown in Table 3 Task is considerably more challenging than Task with exact accuracy for most models below 1%. Only the same top models identified in Task I perform competitively: g under the 3-shot learning setting with reasoning paths achieves the best re- sults (F1 ~ 0.83, Exact accuracy x 0.51), followed by grok-4 trok-4-fast (F1 ~ 0.29, Exact accuracy x 0.20), gemini- 2. --rro (FI Rs 0.71, Exact accuracy & 0.1 and deepseek r1 r1-0528 (Fl x 0.11, Exact accuracy R 0.12). All other systems remain far below 1% Exact accuracy, which highlights the challenge of exhaustive enumeration. Consistently, the models perform best under the 3-shot learning setting with reasoning paths, followed by 3-shot learning without reasoning paths, while the zero-shot setting sees the worst model performances. This observation confirms that reasoning-pan supervision is especially critical for this task.
7

---

Published as a conference paper at ICLR 2026
Table Performance on Task (Generative Task) across different group statement numbers (2-5).
Results are shown for Executive Rate and Consistency under the zero-shot setup.
Model
2 statements
statements
4 statements
statements
Exec
Cons
Exec
Cons
Exec
Cons
Exec
Cons
grok-4-fast
1
0.836
0.95
0.710
0.74
0.458
0.67
0.427
gpt-S
0.826
0.94
0.686
0.77
0.436
0.66
0.381
dcepseck-r1-0528
1
0.802
0.88
0.649
0.63
0.372
0.46
0.283
claude-sonnet-4
0.94
0.776
0.93
0.608
0.60
0.324
0.53
0.172
gwen3-235b-a228
0.93
0.593
0.77
0.471
0.48
0.141
0.30
0.019
gemini-2.5-pro
1
0.817
0.93
0.663
0.76
0.416
0.65
0.348
llam--31--44
0,92
0.696
0.91
0.639
0.67
0.369
0.54
0.247
qwen2.5-7b
0.88
0.575
0.82
0.452
0.53
0.101
0.38
0.120
phi-4-reasoning-plus
0.93
0.687
0.90
0.625
0.67
0.349
0.53
0.219
mixtral-8x7b
0.84
0.587
0.77
0.466
0.55
0.128
0.33
0.106
03-mini
0.790
0.94
0.713
0.87
0.478
0.67
0.358
claude-3,5-hand
0.94
0.696
0.91
0.638
0.68
0.369
0.43
0.257
11ama-3.1-8b
0.89
0.668
0.88
0.596
0.66
0.308
0.33
0.138
gpt-40
0.98
0.703
0.649
0.68
0.385
0.55
0.287
4.3 TASK 3: GENERATIVE TASK
Formulation of Task 3. This task evaluates whether, given n mutually consistent premises, LLMs can generate new statements that remain logically consistent. The process begins by partition- ing the global set of atomic propositions P and their truth assignments into n disjoint groups {G:) For each group, set of ten natural-language facts {Prem;} is generated under strict logical and linguistic constraints, including the mandatory use of multiple logical connectives and rich vocabulary. Each generated fact is then translated into symbolic logical form fey The core of the task lies in the exhaustive consistency evaluation of all cross-group combinations Ca = - Jn,a, ) for a E {1...., 10)": a combination is deemed consistent if and only every fact within it evaluates to true under the global assignment L. Further implementation details are provided the Appendix.
Formulation of Task 3. As shown in Table 4 Execution rate, which means format correctness across statements in a group, remains high for top models such as grok-4 4ffas, gpt-5, gemini-2. 5-pro, and o3-mini. even when mixtral-8x7b) show notable degradation as n increases. Furthermore, performance follows clear trend that higher leads to worse performance, which reflect the difficulty still exists even for non-enumerative tasks.
5. Smaller models (e.g., qwen2 5-7b,
5 DIFFICULTY-BASER ANALYSIS
design three benchmark tasks and extend them with variants to capture a deeper assessment of performance. For Task 1, we evaluate on hard samples, where consistent and inconsistent labels differ by only one position, and also introduce a label comple- tion variant. For Task 2, we evaluate on short reasoning edge samples, short statement length samples, and commonsense, counterfactual, and human-like natural language statements. This design provides finer view of when models succeed and when they fail.
As introduced in Section
5.1 TASK ON HARD SAMPLES AND THE LABEL COMPLETION VARIANT
To better probe model limitations, we formulate two harder task variants using hard samples and label completion, as detailed in the Appendix K Compared with the aggregate Task re- sults, model performances on hard samples reveal sharper contrasts. On hard samples (dark- colored bars in Figure 5a). all model performances drop remarkably. While on the comple- tion variant (Figure 5bb, model performances differ more significantly: strong models (gpt-5, gemini-2. 12-2..2 5 5-pro) plateau around 80-86%, while smaller models (e.g., claude-3 5 5-haiku) collapse below 40%, near random guessing.
8

---

Published as a conference paper at ICLR 2026
Table Performance on three tasks and downstream benchmark correlations.
Task 1
Task 2
Task 3
Benchmark
Pearson
Spearman
Pearson
Spearman
Pearson
Spearman
livecodebench
0.675
0.636
0.651
0.696
0.623
0.689
infinite
0.762
0.763
0.735
0.779
0.744
0.749
aime
0.786
0.643
0.653
0.678
0.690
0.652
aa-ler
0.624
0.598
0.614
0.695
0.603
0.665
acebench
0.632
0.640
0,526
0.688
0.714
0.702
5.2 TASK 2 ON SAMPLES WITH SHORT REASONING PATH AND STATEMENT LENGTH
Performance shown in Figure improves modestly but remains low overall. On the Short Path subset, under the 3-shot learning setting with reasoning paths, gpt-5 averagely increases its Exact accuracy from about 34% to 58%, while grok-4-fast Exact accuracy averagely rises from 12% to %. On the Short Length subset, gains are smaller: for instance, gpt 5 averagely reaches only 34% Exact accuracy, which shows that shorter statements provide limited benefit. Across both subsets, few-shot and especially few-path prompting remain the most effective strategies, with best models retaining clear advantage.
5.3 TASK 2 ON COMMONSENSE AND COUNTERFACTUAL SAMPLES
To address potential reasoning shorteuts from generated statements coinciding with contra- dicting real-world facts, we conducted a controlled experiment. We used gpt-5.1 to generate 100 commonsense atomic propositions and 100 counterfactual ones. For both sets, randomly sub- stituted these atomic propositions into previously generated natural-language statement sets while preserving the sets' original logical labels, since the propositional structure remains unchanged. We then evaluated models on these modified statement sets. Our qualitative analysis reveals that the core reasoning strategy remains fundamentally unchanged between commonsense and counterfac- tual conditions (Appendix L Table 14 and Table 15). Models consistently translate sentences into symbolic representations, leading to highly similar performance patterns despite surface-level dif- ferences.
5.4 TASK 2 ON HUMANIZED NATURAL-LLAGGAAN STATEMENTS
To address the lack of diversity and naturalness in templated text generation, created 1000 "human-style" paraphrased statement sets. We sampled from each (t)-statement sets and used gpt- 4o rewrite the sentences into more natural English (see Appendix [M for the prompt). As shown in [apeeeix[[TTabeeTITe [6] large models exhibited almost no performance drop, in contrast, small models suffered clear degradation, which confirms that the natural languuge understanding component be- comes significantly more challenging once the strong surface regularities of templated sentences are removed. The finding that only semantically capable large LLMs succeed when linguistic cues removed provides strong evidence that LogiConBench genuinely measures the different reasoning capabilities.
REAL-WORLD SIGNIFICANCE
To clarify the practical value of LogiConBench, we evaluated the same set of models on several widely-used real-wor world downstream benchmarks, covering code generation (LiveCodeBench (Jain) et al., 2025)), long-conten context writing (InfiniteBench (Zhang et al. 2024)). mathematical reasoning (aime MMaxwellJjal 2025)), long-horizon logical reasoning (AA-LCR (Artificial Analysis Team 2025 and agent collaboration (ACE Bench (Chen al.) 2025 as shown in Appendix NN Ta- ble 17 We additionally computed Pearson and Spearman correlations between LogiConBenchh 's three tasks, and each downstream benchmark (as shown in Table 5). The correlations consis- tently moderate to strong across all domains, showing that LogiConBench performance is tightly aligned with capabilities that models actually rely real-world use.

---

Published as a conference paper at ICLR 2026
Key observation. (1) All three tasks correlate strongly with long-context reasoning and agent- style collaborative tasks, which indicates that LogiConBench captures a model's ability for agent planning, multi-step tool use, and delegated workflows. (2) LogiConBench also correlates with math and code benchmarks, which demonstrates that stable logical consistency is predictive of models' reliability in symbolic and algorithmic domains. Importantly, (3) Task 2 correlates more strongly with Long-horizon and multi-context benchmarks, which reflects that cross-premiss consistency (the core of Task 2) aligns with the demands of real agent systems, which often must reason coherently across multiple partial states or instructions.
UNDERSTANDING MODEL BEHAVIORS
7.1 QUANTITATIVE ERROR ANALYSIS
Across all tasks' quantitative performance, we observe three recurring failure modes. (1) Enumer- ation breakdown: in Task 2, models often omit some consistent label lists, produce duplicates, or output in the wrong format, which reflects a lack of systematic coverage. (2) Error propa- gation: as k increases, small local mistakes compound along reasoning chains, which explains the sharp accuracy drop from k 2 to k 5 in Task 1 and the plateauing of frontier mod- els in the Hard samples. (3) Bias asymmetry: many models exhibit skewed sensitivity, either over-predicting consistencies (e.g.. phi- phi-4-reason ing - ss)), while a few frontier systems (e.g., 5pt-5) lean toward inconsis- tency more systematically. Together, these trends suggest that contradiction detection (Task 1) is still manageable for frontier models, but exhaustive enumeration (Task 2) exposes deeper weak- nesses in structured reasoning and systematic search, with large room for improvement.
or over-detecting inconsistencies (c.g.,
7.2 QUALITATIVE ERROR ANALYSIS
(1) Different-sized models have a clear difference. Large models (e.g.. GPT-4, Claude-Sonnet) first translate problems into symbolic form, enabling effective short-step logic. In contrast, smaller models (e.g., (--mB)3.1-8B) tend to paraphrase the problem and jump to a conclusion, explaining their low accuracy despite moderate F1 scores. Shared Error Patterns in Large Models. Even advanced models exhibit critical flaws. Their reasoning is often sigrt-sighted, leading to three recur- ring errors: Incomplete Emumeration (checking too few cases), One- One-Way Simplification (failing to map symbolic results back to the original problem), and Lost Goals (losing sight of the main objec- tive during reasoning).GG Task-Specific Failure Modes. The two tasks expose distinct weaknesses. In Task 1, models struggle with counterfactual exploration, failing to systematically consider "what- if" scenarios. In Task 2, semantic drift occurs, where models output correct intermediate reasoning symbols the final answer, confusing the tool with the solution.
8 CONCLUSIONS, LIMITATIONS AND FUTURE WORKS
LogiConBench advances the study of logical consistency in LLMs by addressing three key lim- itations of existing benchmarks: lack of scalability, absence of explicit reasoning structures, and insufficient difficulty. Our framework automatically generates unlimited logical rule combinations with precise labels, constructs graphs with controllable depths that provide explicit reasoning paths, and remains challenging for state-of-the-art LLMs. Large-scale experiments with 14 frontier mod- els show that while Discriminative Task can reach 85-95% accuracy, Enumerative Task remains extremely difficult, with the best average exact accuracy only 34%, and Generative Task consis- tency drops sharply as group size increases. with even the best model falling from 83.6% to 42.7%. Moreover, the introduction of difficulty- task variants reveals stable relative performance rank- ings across models, which highlights the benchmark's evaluative value. Despite its contributions, LogiConBench has certain limitations. First, the natural language generation process still relies on templates and lexical substitution, which may limit linguistic diversity compared to fully human- authored datasets. Second, our experiments mainly evaluate single-turn consistency reasoning. Future directions include enhancing the diversity of language formulations beyond templates to interactive multi-turn, multi-agent multi multi-modal tasks where logical consistency plays central role (Xu et al., 20256). We also foresee applying LogiConBench for model training and alignment to strengthen consistency reasoning in frontier L.MMs.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work does not involve human subjects or sensitive personal data. All datasets are automatically generated using symbolic rules and publicly available lexical resources, with no privacy or security risks. The study does not raise foreseeable ethical concerns related to fairness, discrimination, or potential harmful use.
REPRODUCIBILITY STATEMENT
All datasets and preprocessing procedures, and evaluation metrics are described in detail in the main and appendix. Complete implementation details are provided in an anonymized repos- itory containing the code and reproduction instructions at /ttp/btttt.../.../ cow Be 1lafc/
LegiConBench.co git
ACKNOWLEDGMENTS
Zhouchen Lin was supported by the Beijing Natural Science Foundation (L257007), the NSF China (62276004), and the Beijing Major Science and Technology Project (225110000422006) Bo Li was supported in part by an NSFC grant 62432008, RGC RIF grant R6021-20, an RGC TRS grant T43-513/23N-2, RGC CRF grants C7004-22G, C1029-22G and C6015-23G, NSFC/RGC grant CRS HKUST601/24 and RGC GRF grants 16207922, 16207423 and 16203824. Fenrong Liu is sup- ported by the Beijing Natural Science Foundation (L2257007) and Tsinghua University's 's Initiative for Advancing First-Class and World-Leading Disciplines in the Humanities and Social Sciences. Yisen Wang is supported by the Beijing Natural Science Foundation (L.257007, the Beijing Major Science and Technology Project (Z251 100008425006), the National Natural Science Foundation of China (92370129, 62376010), the Beijing Nova Program (2023048434,, 20240484642), and State Key Laboratory of General Artificial Intelligence.
REFERENCES
Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, et al. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.2133 2025.
Anthropic.
Introducing claude 3.5 haiku, 2024.
URL http:://ww.anthroplc..coo
claude/haikul
Anthropic. Introducing claude 4, 2025.
URL http::/ww..nntirgiic.comm
claude-4]
Artificial Analysis Team. Artificial analysis long context reasoning benchmark, 2025.
Akari Asai and Hannaneh Hajishirzi. Logic-guided data augmentation and regularization for consis- tent question answering. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp- 5642-5650, Online, July 2020. Association for Computational Linguistics. doi: 10. 18653/vll 2020.acl- mai---9aiinn...... URL http:: /tatps://aacannthg ogy wtrg/2220..om aal.0aaclmaaa 4997
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In C.J. Burges, L. Boutou, M. Welling, Z. Ghahramani, and K.Q. Weinberger (eds.), Ad- vances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., files/paper/2013/
2013. URL / http:://pppoeei..s...erioo.../ htl1:/eccca7772ee333332222888222
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large anno- tated corpus for learning natural language inference. Lluís Märquez, Chris Ballison-Burch and Jian Su (eds.), Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 632-642, Lisbon, Portugal, September 2015. Association for Computational Lin- guisties. doi: 10.18653/v 1/D15-1075 URL /wtp:httt///// antho anthol anthollogy.orr/D155-1035

---

Published as a conference paper at ICLR 2026
Diego Calanzone, Stefano Teso, and Antonio Vergari. Logically consistent language models via neuro-symbolic integration. The Thirteenth International Conference on Learning Represen- tations, 2025. (RRptttt:///eeeeeeee http:////....oee.e...e.../e/aaa.
net/
Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam Hruschka, and Tom Mitchell. Toward architecture for never-ending language learning. In Proceedings of the AAAI conference on artificial intelligence, volume 24, pp. 1306-1313, 2010.
Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li. Shuai Wang, Weinan Gan, Yuefeng Huang, et al. Acehench: Who wins the match point in tool usage? arXiv preprint arXiv:2501. 1285, 2025.
Fengxiang Cheng, Haoxuan Li, Fenrong Liu, Robert Van Rooij, Kun Zhang, and Zhouchen Lin. Empowering Ilms with logical reasoning: A comprehensive survey. In Proceedings of the Thirty- Fourth International Joint Conference on Artificial Intelligence, 2025.
Ian Chiswell and Wilfrid Hodges. Mathematical logic. OUP Oxford, 2007.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities, arXiv preprint arXiv:2507.06261. 2025..
Bhavana Dalvi, Peter Jansen, Oyvind Tafjord, Zhengnan Xie, Hannah Smith, Leighanna Pi- patanangkura, and Peter Clark. Explaining answers with entailment trees. In Franciee Moens, Xuanjing Huang, Lucia Specia, and Scout Wen-tau Yih (eds.), Proceedings of the 2021 Conference Empirical Methody in Natural Language Processing, pp. 7358-7370, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10. 10.18653/1/2021.mmmll http:/2022.mmmmmmmm.mmaeaa.a. m8in. htt htt.ltttss///claathooooo/// orgp:/hoooooooo.o0222211111 org/2021. main. emn.p-main..e 585/
Christiane Fellbaum. Wordnet and wordnets. In Keith Brown (ed.), Encyclopedia of Language and Linguistics. Elsevier, Oxford, 2 edition, 2005.
Bishwamittra Ghosh, Sarah Hasan, Naheed Anjum Arafat, and Arijit Khan. Logical consistency of large language models in fact-checking. In The Thirteenth International Conference on Learning Representationss 2025. URL https:://peerreiieeee herperrreviee..ee///oruutt www.:eeweeeetttooeeeeeeeeee Sim--i11im--G0
Aaron Grattafiori, Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, al. The llama 3 herd of models. arXiv preprint arXiv:240.221783, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in 1lms via reinforcement learning. arXiv preprint arXiv:2501.1294/ 2025.
Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi. Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu, Rui Zhang, Alexander Fabbri, Wo- jeiech Maciej Kryscinski, Semih Yavuz, Ye Liu, Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying, Arman Cohan, and Dragomir Radev. FOLIO: Natural language rea- soning with first-order logic. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 22017-22031, Miami, Florida, USA, November 2024. Association for Computational Lin- guistics. doi: 10. 11/////////22.....// 1/w111/22222222 /22224.mmmmmmmaiiii 1229. URL https: http://aaaaaatooooooooooooo org/ 2024. www.eenwwnmmiimeem 1229/
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata. Yuxiao Dong, and Jure Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs. In Thirty-fiftl Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
12

---

Published as a conference paper at ICLR 2026
Jie Huang and Kevin Chen-Chuai Chang. Towards reasoning in large language models: A survey. In ACL (Findings), 2023.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21277 2024.
Frank K Hwang and Dana S Richards. Steiner tree problems. Networks, 22(1):55-89, 1992.
Naman Jain et al. Livecodebench: Holistic and contamination free evaluation of large language models for code. In Proceedings of the International Conference on Learning Representations (ICLR), 2025.
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.040085 2024.
Huichi Zhou Xin Quan Qijun Huang Shenggiong Wu William Yang Wang Mong-Li Lee Wynne Hsu Jundong Xu, Hao Fei. Logicreward: Incentivizing Ilm reasoning via siee logical supervision. arXiv preprini 25 18.1966,, 2026. URL httphtttst//axii.cooom wwwpaarwwwwwvoooaa/a//a/aaaaa/ / 181
Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Yoav Goldberg. Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1266-1266 1279, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguisties. doi: 10. (0.18653/v122) 1/2022. emnlp-main.82. URL https: 200 e...022.emmmll. emnlp-ma i.. 827
ogy
Nora Kassner, Oyvind Tafjord, Hinrich Schüitze, and Peter Clark. BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief. In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.). Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, pp. 8849-8861, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/2011 00.86633//22221emmmmmmm.../
eemmlp ma main. 697/
Nora Kassner, Oyvind Tafjord, Ashish Sabharwal, Kyle Richardson, Hinrich Schuetze, and Pe- ter Clark. Language models with rationality. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL
6 F4
Tao Li, Vivek Gupta, Maitrey Mehta, and Vivek Srikumar. A logic-driven framework for consistency of neural models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019.
Fenrong Liu and Martin Stokhof. Logic, Language and Philosophy: An Introduction Standard Logic. CSLI Publications, 2024.
Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vulié, and Nigel Collier. Aligning with logic: Measuring, evaluating and improving logical preference consistency in large language models. In Forty-second International Conference on Machine Learning, 2025. URL https: //openzeview.
Maxwell-Jia. Aime: All 30 problems from the 2025 american invitational mathematics examination. Technical report, AIME, 2025.
Eric Mitchell, Joseph Noh, Siyan Li. Will Armstrong, Ananth Agarwal, Patrick Liu, Chelsea Finn, and Christopher Manning. Enhancing self-consistency and performance of pre-trained language models through natural language inference. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pp. 1764, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653// 1,0663//112022..mm-pmmmiiil
URL
https://aci antho ogy. og.. wwm.22022222mm222. emn p-nmllmma- n. 1..111 1157
13

---

Published as a conference paper at ICLR 2026
Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Learning deductive reasoning from synthetic corpus based on formal logic. In Proceedings of the 40th International Conference on Machine Learning, pp. 19254-25272 2023.
Yasumasa Onoe, Michael J.
Q. Zhang. Eunsol Choi, and Greg Durrett. Creak: A dataset for
commonsense reasoning over entity knowledge. In NeurIPS Datasets and Benchmarks, 2021.
neur www..us....ppppp.rr......c 2021 20t.////11777770000/11aaaaaaaaa0
OpenAl.
Introducing gpt-5,
2025a.
URL http:///penaai.com/nnooo
introducing-gre gptt 57
OpenAl. Openai 03-mini,
2025b.
URL https://openai.commmna
Openai -0-mmii---min3/
Daniel Paleka, Abhimanyu Pallavi Sudhir. Alejandro Alvarez, Vineeth Bhat, Adam Shen, Evan
Wang, and Florian Tramèr. Consistency checks for language model forecasters. The Thirteenth International Conference on Learning Representations, 2025.
pott/forumiidd=IIIB
Arijit Ray, Karan Sikka, Ajay Divakaran, Stefan Lec, and Giedrius Burachas. Sunny and dark outside?! improving answer consistency in VQA through entailed question generation. In Ken- taro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNL.--JCNLPP, pp. 5860-5865, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v//D19-15500 URL https: antho ogy. www.yoorg/D11-111
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Shikhar Singh, Nuan Wen, Yu Hou, Pegah Alipoormolabashi, Te-lin Wu, Xuezhe Ma, and Nanyun Peng. COM2SENSE: A commonsense reasoning benchmark with complementary sentences. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 883-898, Online, August 2021. Asso- ciation for Computational Linguistics. doi: 10. 101118855512222 .indings-acl.77. URL https:
// //aclanthology.o wrg.
wttpaa/wwwwooop.g./22.......... wwt.2221....nnnnio. findings 781
Mooho Song, Hye Ryung Son, and Jay- Lee. Introducing verification task of set consistency with set-consisteney energy networks. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 53346-33366, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89177-251-19 doi: 18.1533//v 2025.acI 2025.acl-long.. 1599. URL https:: /tttss:/aaclanthoooog.oooo 2tt.:/2022....111010 ac1-
Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen-tau Yih, and Ashish Sabharwal. Quarel: dataset and models for answering questions about qualitative relationships. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33. pp. 7063-7071, 2019.
Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Peter Clark, and Antoine Bosselut. WIQA: A dataset for "what if..." reasoning over procedural text. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing DEMNLP-IJCNLAN pp. 6076-6085, Hong Kong, China. November 2019. Association for Com- putational Linguistics. doi: 10.18653/v 1/D19- 1629. URL https: http:///laaaannnccc.ooooooooool. orzg
D1 019-16297
14

---

Published as a conference paper at ICLR 2026
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. TRL: Transformers Reinforce- ment Learning, 2020. URLhttpa /tte:://ggttttt github..b.....o............... com. wwt./hugginggffee///t
/
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupata, and Afra Alishahi (eds.), Proceedings of the 2018 EMNLP Workshop Black- boxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353-355, Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/11/W1--5555
URL
wrg. 00t.:/hhhooooooo0008-5555000
Dag Westerstahl. Foundations of logic: completeness, incompleteness, computability. 2022.
xAL Grok 4 fast, 2025.
Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logical reasoning via symbolic chain-of-thougle Computational Linguistics, 2024. URL htttps://aaxiiiii .rgg abs 24C5.18357
The 62nd Annual Meeting of the Association for
Jundong Xu, Hao Fei, Meng Luo, Qian Liu, Liangming Pan, William Yang Wang, Preslav Nakov, Mong-Li Lee, and Wynne Hsu. Aristotle: Mastering logical reasoning with A logic-complete framework. In Proceedings the 63rd Annual Meeting of the Associa- arxiv. .owg//oggoooooo.o.o abs 2412. 16933
cesoomoos--eercc-reeeee fionfor Computational Linguistics, 2025a. URL.ttpss
Jundong Xu, Hao Fei, Yuhui Zhang, Liangming Pan, Qijun Huang, Qian Liu, Preslav Nakov, Min- Yen Kan, William Yang Wang, Mong-Li Lee, and Wynne Hsu. Musir: Multimodal symbolic logical reasoning. In Proceedings of the Annual Conference on Neural Information Processing Systems, 2025b. URL /tLh:tts:://// //wps////ppp///ee cct/vittal//225/////cccc /ttp:////////////9990
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09338 2025.
Qwen An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxin Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue. Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yi-Chao Zhang, Yunyang Wan, Yuqi Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu, Shanghaoran Quan, and Zekun Wang. Qwen2.5 technical report. ArXiv, abs/2412. 151115 2024.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. BeBenhh Extending long context evaluation beyond 100K tokens. Lun-Wei Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15262-15277, Bangkok, Thailand, August 2024. Association for Computational Lin- URL StLhttps: http::::s//aaaaaahleeoooooo www.ooog......o..1
guistics. doi: 10. 1202555311/2222211111111lo1 acl-long.88 8147
USAGE OF AI
In this work, we made limited use of LLMs as an assistive writing tool. Specifically, we used LLMs to replace synonyms, restructure sentences, and brainstorm alternative ways of expressing ideas within paragraphs. All conceptual contributions, research design, experiments. analyses, and final writing decisions were made by the authors. The authors take full responsibility for the accuracy and originality of the content.
15

---

Published as a conference paper at ICLR 2026
A RELATED WORK
A.1 DATASETS FOR LOGICAL CONSISTRNCY
Several datasets have been introduced to study consistency, but they mostly fall into three categories: (I) constraint-graph and knowledge-grung resources, which encode logical relations over structured triples or constraint graphs; (2) QA-style datasets, which induce logical dependencies between an- swers to paired or sequential questions: and (3) NLI-style corpora, which frame consistency in terms of entailment, contradiction, or neutrality. While each line provides valuable insights, they are gen- erally tied to specific domains, limited rule templates, or narrow label spaces. Importantly, many of these resources conflate factual consistency (whether statements are true with respect to exter- nal world knowledge) with logical consistency (whether statements are mutually non-contradictory under formal rules), whereas our work isolates and directly evaluates the latter.
Constraint-graph and knowledge-graph datasets. BeliefBank (Kassner et al. 2021) builds an explicit constraint graph from ConceptNet (Speer et al.] 2017) and WordNet (Fellbaum 2005), where structural rules define positive implications and mutual exclusivities. By instantiating enti- ties into this graph, 12,525 "silver" truth-labeled facts are automatically propagated, and additional lubeenlabeled calibration facts are introduced to refine consistency. The recently introduced Log- ical Fact-checking Datasets (Ghosh et al. (025) (FreebaseLFC. NELLLFC, and WikiLFC) are derived from large knowledge graphs (Bordes et al. 2013 Carlson et al. 2010 Hu et al. 202T),
where triplets are transformed into (Fact, Context) pairs. These benchmarks explicitly support propositional logic queries with negation, conjunction, and disjunction, thus enabling large-scale fact-checking with logical operators. EntailmentBank (Dalvi et al., (02I)) provides the first dataset of multistep and full derivation entailment trees, graph-structured form of explanation that links atomic facts hypotheses through multipremise entailment steps. However, these graph-based re- sources are tied to specific domains, such as facts, and to specific rule templates, which make them less general for evaluating logical consistency in diverse open-domain settings.
QA-style consistency datasets. ConVQA (Ray al. 2019) consists of visual question-answer pairs that are logically related, which naturally induces logical constraints between answers, mak- ing the dataset well suited for evaluating consistency in visual reasoning. Several commonsense and scientific QA datasets follow similar design by introducing logically related statements or questions. Com2Sense (Singh al. 2021) presents paired statements where only one is logically consistent with commonsense knowledge, requiring models to distinguish true from false assertions.
CREAK (Onoe et al. 202T) further targets commonsense abduction, consisting of fact-verification questions that require linking claims to implicit commonsense knowledge. Likewise, OBQA and QuaRTz provide science and quantitative reasoning questions with built-in logical dependencies between answers. Beyond pairwise consistency, datasets such as WIQA (Tandon al., 2019), QuaRel (Tafford et al., 2019), and HotpotQA (Yang et al. (018)) emphasize causal reasoning. For example, WIQA asks about the effects of perturbations requiring the model to trace causal chains and determine whether changes lead to positive, negative, or neutral outcomes. Nonetheless, these QA datasets typically test specific reasoning phenomena and do not provide systematic coverage of logical consistency rules across diverse contexts.
processes described in procedural texts,
Natural Language inference (NLI) datasets. Standard NLI corpora such as SNLI (Bowman al. 2015) and MultiNLI (Wang et al., 2018) can be adapted for logical consistency evaluation, as they contain
pairs annotated with entailment, contradiction, or neutrality. Recent extensions have moved beyond pairs to sets of statements: Set-SNL. and Set (Song et al., 2025) require detecting whether an entire set of sentences is mutually consistent and identifying the specific statements that introduce conflict. FOLIO (Han et al.) 2024) is the first expert-written dataset for first-order logic (FOL) reasoning, where each example pairs set of natu- ral language premises with a conclusion derived from NLI symbols, together with a parallel formal- ization in FOL. Yet these NLI-style resources are limited in scale and coverage of logical operators, which are contradiction, neutral, and entailment, and thus cannot fully capture the broad range of logical consistency phenomena.
16

---

Published as a conference paper at ICLR 2026
A.2 METHODS FOR LOGICAL CONSISTENCY REASONING
Fact-checking tasks. BeliefBank (Kassner et al. 2021) embeds a pretrained language model in system with evolving symbolic memory, using weighted MaxSAT solver to reason over dependencies and a feedback mechanism to query the model with known beliefs as context. Ghosh al. (2025) introduces logical fact-checking datasets over knowledge graphs. proposes measures of logical consisteney on propositional logic queries, and applies supervised fine-tuning to improve performance. Calanzone et al. (2025) introduce neuro-symbolic loss that enforces consistency with an external set of facts and rules, allowing multiple constraints to be combined in principled way and improving generalization to unseen but semantically similar knowledge. Paleka et al. (2025) define consistency metrics for LLM forecasters based on arbitrage opportunities, generate logically related question sets, and demonstrate that instantaneous consistency metrics correlate with ground-trutt forecasting performance.
NLI-based tasks. Li ct al. 2019) present a framework that compiles knowledge stated in first- order logic into loss functions, reducing inconsistency in neural models. ConCoRD constructs factor graph combining model predictions and NLI-based pairwise relations, then applies weighted MaxSAT select globally consistent answers, boosting performance QA and VQA bench- marks (Mitchell et al., 2022). Maieutic Prompting recursively generates trees of abductive expla- nations and frames inference as satisfiability problem over these explanations and their logical relations, achieving improvements on commonsense reasoning benchmarks (Jung et al. 2022). RE- FLEX adds rational, self-reflecting layer on top of LLMs: it builds belief graphs through backward chaining and uses a constraint reasoner to minimize contradictions, significantly improving consis- tency without harming accuracy (Kassner et al.. 2023). LogicReward (Jundong Xu (026) constructs reward function integrating step-level logic validity with outcome correctness to train LLMs using reinforcement learning, significantly enhancing the model's logical reasoning capability.
Comparison QA approaches. Asai & Hajishirzi (2020) propose logic-guided data augmenta-
tion and regularization, leveraging logical and linguistic knowledge to augment training data and constrain predictions, thereby improving global consistency across multiple QA tasks. REPAIR in- troduces a framework to quantify logical consistency via proxies such as transitivity, commutativity, and negation invariance, evaluates LLMs across multiple comparison tasks, and enhances consis- tency through data refinement and augmentation (Liu et al. 2025).
B TABLE 1 EXPLANATION
In this section, we explain how the statisties in Table T were obtained.
BeliefBank. Section 5.3 Kassner et al. 1021 mentions that the dataset contains 12,525 "silver" truth-labeled facts. The constraints used are only of two types: Positive Implications and Mutual Exclusivities. Thus, we consider the rule count as 2. Since all constraints are directly instantiated without multi-step reasoning, the depth set to and the operators involved are implication (-+), negation (-), and bidirectional implication (++). The dataset does not explicitly provide reasoning paths. As is developed from ConceptNet, its scalability is considered limited.
EntailmentBank.
The dataset contains 1.840 QA pairs. Table Dalvi et al. 1 reports that the average number of edges per inference is 6, which we use as the average reasoning depth. Only the implication operator (-+) is present, and reasoning paths are explicitly included. However, since all examples were annotated by experts, scalability remains limited.
LFC. According to Ghosh et al. (2025). the Logical Fact-checking datasets (FreebaseL.FC. NEL- LLFC, WikiLFC) consist of around 2,000 examples. Rules are listed in Tables 13, 19, and 20 of the paper, totaling 9 distinct logical rules. The maximum reasoning depth among them is 4. Since the dataset is constructed from existing sources (Freebase, NELL, and Wiki), scalability is limited, and reasoning paths are provided.
Set-LConVVQA Set-SNLI Section 4 of [Song et al. (025) states that the dataset contains 6, 754 6, 225 + 200 13, 779 instances. From Tables 6-9, we identify the largest rea-
17

---

Published as a conference paper at ICLR 2026
Table 6: Natural deduction rules for the five logical operators.
Operator Implication (-)
Introduction Rule
Elimination Rule
If assuming leads (possibly through several steps) to th, then in- fer yp + & (discharge assumption, make part the conclusion).
E_+ (Modus Ponens): From yp and
4 + v. infer eb.
Disjunction (V)
ly: From infer yp V it; or from v infer yp (introduce by either side).
Ev: From & and the two deriva- tions yp F X and X, infer X-
Conjunction (A)
LA From yp and v,, infer yp & (in-
Ex: From şp A 1h, infer either yp or t (eliminate A from the premise).
troduce in the conclusion).
Negation (-)
L.: From assuming leads to con-
F..: From yp and mp, infer contra- diction 1.
tradiction 1, infer mip.
Biconditional (++)
Tep From yp + to and v >> y,, infer q ++ vb.
Ess: From yp + 4/, infer either yp 1 th + 4.
soning depth 5. The total number of rules is 36 + 6 51. Operators involved include
conjunction (A), disjunction (V), implication (-+), negation (-), and bidirectional implication (++). Since the dataset derived from SNLI and LConVQA, scalability limited, and reasoning paths are not included.
LogiconBench. Our benchmark contains 280k logical graphs, with maximum depth 32, involving all five standard operators. Each graph explicitly records reasoning paths, and the dataset con- struction process can be scaled indefinitely. Therefore, scalability is considered unlimited, and the number of rules grows with the dataset size.
C DETAILS OF THE PRELIMINARIES
Table summarizes the introduction and elimination rules in the Natural Deduction for the five log- ical operators (Liu & Stokhof 2024) considered in work: implication, disjunction, conjunction, negation, and biconditional. We used introduction rules to construct the logical graph, and we in- cluded the elimination rules for rewriting. These rules specify how propositions can be derived or simplified in proof system, and thus provide the basis for generating and evaluating logical graphs. By grounding our benchmark in these standard rules, we ensure that the reasoning tasks both formally precise and interpretable.
D DATASETS STATISTICS
To better characterize our dataset, we report the distribution of logical operators within the con- structed nodes in Figure 4 Specifically, we count the occurrences of Not, And, and Or, since these operators are directly involved in forming node-level expressions. In contrast, Implication (-) and Equivalence (++) are naturally reflected in the directed edges between nodes, and thus not included in the node-level statistics. The results in Table [ show that the distributions vary with k, but consistently exhibit wide coverage across different counts, ensuring diversity in logical complexity.
E
PATH EXTRACTION
The details of extracting the shortest path described as follows:
For each permutation æ S, concatenate undirected shortest paths between consecutive pairs (my, T++1) to form walk covering all targets.
18

---

Published as a conference paper at ICLR 2026
Atom Count Distribution
"Not" Count Distribution
3.29%
7 0.85%
= 0.03%
4 11.55%
5 3.21%
17.26%
15.25%
2 34.69%
1.22%
7 9.18%
Others (8 to 18) 3.533
3 27.39%
2= 10.63%
20.23%
18.91%
20.84%
1 5.25%
"Or" V Count Distribution
"And" Count Distribution
7 = 1.52%
7 0.77%
8 0.45%
8 0.08%
3626600%
19.69%
3 26.29%
2
11.37%
= 11.78%
13.25%
21.1%
Others (9 to 16) 1.18%
21.17%
Others
13)
4.69%
4.12%
Figure 4: The distribution of logical operators within the constructed nodes.
Table 7: Distribution of operator counts
Operator
4
8
9
10
11
12
13
14
15
16
17
18
2743
5302
10284
12524
12007
10230
61.6
1366
0
40529
67757
K-5
28089
52362
84020
99497
96025
73891
43030
18548
203
9833
18196
18351
3460
2719
1166
758
432
390
88
80
16
40
28
22
5
11
14
NOT
43066
75087
112364
70093
54378
37269
24771
6539
3972
2910
2438
2110
1777
1128
871
791
373
455
195
155
264
23
35
3
11356
15350
1479
300
253
1]
OR
1=1
28244
89910
71836
37021
51723
85635
73407
42465
32373
20075
11421
12697
2660
4489
1417
832
670
794
1177
374
565
62
213
128
0
0
0
k-5
57077
10048
130061
110012
59731
23579
7642
1763
800
522
373
233
254
137
0
0
7174
12532
16139
2821
82740
71965
386
K-5
71365
119122
131309
101983
53451
15553
1928
248
261
42
343
0
2. Among all permutations, pick the walk with the fewest distinct edges (ties broken by fewer nodes).
3. Recover edge labels from the original directed graph: if we traverse - along stored edge, use its type; we traverse against direction v - u, flip ->/-- and keep x or <> unchanged.
This yields an ordered edge list E [[u1, (i,,,,, vi).... (Um &ms vm) that for labeling.
F LABEL EXAMPLE
Consider three target nodes S ((p V -u), p. (s A -p)}. connected by the path
(p -u) p p V -p).
By labeling nodes according to RULES, the consistent assignments over all nodes on the path in order are
T,T.F,F; T,F,T,;; F,T,F,F; F,F,T,F; F.F.T.T.
19

---

Published as a conference paper at ICLR 2026
Projecting these assignments onto the target list S gives the consistent.lists:
FI, [T. F), [F, [.].
Since S has 2 possible Boolean assignments, the other three, [T. T) [T, F. FI, T. Anconsistent.liste
G
REWRITE RULES
The rewrite rules are shown in Table g
Table Rewrite rules of the symbolic languages.
ID
Rewrite Rule
Description
Example
A
- simpli fu(()
Simplification
(pAT) V ~e pVq
ip NNF(})
Negation Normal Form
1pp + 4) Aud 4
C
(o b) (~a V b)
Implication elimination
p 7 ~ V
DI
(a ++ b) ~ (a + b) A (b a)
Equivalence elimination
pt+ q ~n (p- ②) n (q p)
D2
(a ++ b) (a ^ (-a -)
Bquivalence elimination
ps99 (p^g) (DDA 7
(-a b) ~M (a b)
Implication introduction
PVV ~ p
ba V (-a b) (a
Equivalence introduction
(p^q) V (-PA -g) pto
G1
(NF
Conjunctive Normal Form
(pAg) Vr (~9V)) (pVg)Ar (PAT)
so DNF(())
Disjunctive Normal Form
H SYMBOL LANGUAGE TO NATURAL LANGUAGE STRUCTURES
This appendix summarizes the templates used in data construction. First, we present the negation rules, which ensure coverage of both affirmative and negative variants of atomic statements.
The NOUN ADJ" + The NOUN is not ADJ".
The NOUN occurs" A The NOUN does not occur".
The NOUN VERBs" " H The NOUN does not VERB".
The NOUN has..." The NOUN does not have ...
Second, we define composite formula templates, where structural operators correspond to logi- cal connectives (e.g., conjunction, disjunction, negation), enabling natural language rendering of complex formulas.
AA B;g++; First, A. Second, B."
AVBVC C + Either (i) A, (ii) B, or (ii)) C." AA33;;; "It is not the case that the following holds: A."
EVALUATION METRICS
Task and Task on Hard Samples. The evaluation metric accuracy, defined as the percent- age of correctly predicted labels. We report: (1) accuracy consistent lists (i.e., given a consistent set, the model outputs "yes"), (2) accuracy on inconsistent lists (i.c., given an inconsistent set, the model outputs "no"), and (3) overall accuracy. Since experiments include 500 consistent and 500 inconsistent lists, the overall accuracy is computed as the average of the two.
Task Variant: Label Completion. The evaluation metric is overall accuracy, which measures the correctness of label completion, i.c., whether the predicted Boolean assignment correctly com- pletes all statements without contradiction.
20

---

Published as a conference paper at ICLR 2026
Task 2 and Task 2 on Easy Samples. The evaluation metrics include Format accuracy, Exact accuracy, Precision, Recall, and F1.
Format accuracy: the percentage of outputs that follow the required enumeration format.
Exact accuracy: a strict metric that measures the proportion of samples where the model enumerates exactly all the consistent label lists in the dataset.
TP, TN, FP, FN: we treat the unlisted assignments as the model's predicted inconsistent lists. Given the ground truth partition of assignments into consistent vs. inconsistent:
TP (True Positive)= = percentage assignments that are consistent in the ground truth and also predicted as consistent.
EN (False Negative) = percentage assignments that are consistent in the ground truth but predicted inconsistent.
TN (True Negative) = percentage of assignments that are inconsistent in the ground truth and also predicted as inconsistent.
FP (False Positive) = percentage assignments that are inconsistent in the ground truth but predicted consistent.
Precision: intuitively, how many of the lists predicted consistent are truly consistent.
Recall: how many of the truly consistent lists are successfully identified by the model.
F1 score: the harmonic mean of precision and recall, reflecting the balance between the two.
J EXPERIMENTAL RESULTS
J.1 TASK 1 ON HARD SAMPLES
The performance of Task on Hard Samples are detailed in Table
J.2 TASK I VARIANT: LABEL COMPLETION
The performance of Task Variant is shown in Table 10
J.3 TASK 2 PRECISION AND RECALL.
The Precision and Recall Score of Task shown in Table
I
J.4 TASK 2 ON EASY SAMPLES (SHORT REASONING PATH SAMPLES AND SHORT STATEMENT LENGTII SAMPLES)
The Performance of Task on Easy Samples is shown in Table 12 and Table 13
K DIFFICULTY ONE AND TWO
Formulation of Task hard samples. In the original task, closed-source models such as prok-4-2 f-st and gpt-5 perform well, which suggests that the task may not fully expose their limitations. To probe this, we design a hard-sample variant where provided lists of Boolean labels come from consistent and inconsistent lists differing in only one element, which makes them more challenging distinguish. For example, choosing [T. T, T] in consistent.lise incons stent. l sists the label list to discriminate.
[T, T. F] in
Formulation of Task 1 variant: Label Completion. Label Completion is harder variant of the Discriminative task. Instead of verifying a full label set, the model must recover the hidden label so that the full label list remains logically consistent, which increases reasoning difficulty.
21

---

Published as a conference paper at ICLR 2026
Table 9: Performance decomposition across zero-shot, 3-shot, and 3-shot with path. Each block reports consistent (con.), inconsistent (incon.), and overall accuracy for k = 2, 4, 5.
Model (mode)
Accuracy on 2 Statements
Accuracy on 3 Statements
Accuricy on Statements
Accuracy on Statements
incem.
overall
incon.
overall
overall
incun.
overall
zero-shot learning
claude-3.5 3.5-hiik (acro-huut)
31.57
38.58
35.07
31.61
38. 64
35.
31,66
38. 38.69
35.17
31.70
38.75
35.22
clunde--onne-d (zero-shot)
$1.63
63.10
57.37
$1.68
63.16
57.42
$1.72
61.21
$7.47
$1.77
63.27
57. $7.52
despweek-r1 1-08 (zero-dhoo)
67.86
82.93
75.40
67.90
82.99
75.45
67.95
83.04
75.50
67.99
83.10
75.55
gemini-2.55 2.5, pro (2eo----- (zero-shor)
35.0 68
43.61
39.64
35.73
43.66
39,69
35.77
43.72
39.74
35.82
43.77
39.79
spt-4o (acto
17.39
21.25
19.32
17.43
21.30
19.37
17.48
21.36
19.42
17.52
21.41
19.47
1lama-3. 3.1 105b instruct (zcro- shot)-hot)
29.98
36.65
33.32
30.03
36.70
33.37
30.07
36.76
33.02
30.12
36.81
33:67
Ilama-3. 3.1 1-- instrrctutt (zemv-shol)
21.78
26.62
24.20
21.82
26.67
24.25
21,87
26.73
24,30
21.91
26.78
24.35
thi-4-rssoninn-pll (rcro-dhon)
24.80
30.31
27.56
24.85
30.57
27.6
24.89
30.42
27.66
24.94
30.48
27.71
mixtral-8<7h- Diktral-87h--instruz (per0-skor)
18.42
22.91
18.46
22.56
20:51
18:51
22.
18.55
20.61
o3-mini (zern-chot)
19:53
23.87
21.70
19.58
23.93
11.25
19.62
21.80
19.67
21.85
gp-- (2cro-shot)
79:57
97.25
$3.41
79.61
97.
88.
79.66
97 36
88.51
79. 70
97.41
88.
quen-2.5-7h-- 2.---2.--7--nniiist (sero-shoo)
27.85
34.04
50.95
90
34. 10
31. 100
27.94
34.11 15
31.05
27.99
34.21
51.10
quen-3.225h--. (2eno-shot)
43.91
53.67
48. 48.79
43.95
53.72
48. 84
44,00
53.78
48.89
44.04
53.83
48.94
grok-4-fae 4-st (zcro-shot)
09.43
84.86
77.15
69.48
84.92
73.20
69.52
84.97
77.25
09.57
85,03
77.30
3-dhot learning
Fande-1.5-baikiu (3-4hod)
31.14
38.06
34,60
31.19
58. 18.12
34.65
31.25
38.17
34.70
31.28
18.23
34.75
clande-sonnet-44 (3-dboo)
52.85
64.60
58.73
52.90
64.65
58,
52.94
64.71
58.83
52.99
(4.76
58,
deepicek-e1 1-0228 (3-thot)
77.23
94,39
85.81
77.27
94.44
85.86
77.32
50
85,91
71.36
94.55
85.96
gemins 2.5 5-pro (3-shot)
82.72
101.10
91.91
$2.76
101.15
91.96
82.81
101.21
92.01
$2.85
101.26
92.06
gpt-Io (3-sbo))
28. 10
34.71
31
28.15
34.77
31.61
28.09
34.82
31.66
28.54
34.88
31.71
flama-3.1 3.1 1-40sb-insttuer (3-shot)
34.55
42.22
38.
34.59
42:28
38.43
34.64
42.35
38.
34. 0S
42.39
38.53
Ilama- 3.1 Sb instruct (2 (3-shet)
19:72
24.10
21.91
19.76
24.16
21.96
19.81
24.21
22.01
19.85
24.27
22.06
phi 1-rcasoning plus (3-shot)
28.87
35.29
32.08
28.92
35.35
32.13
28.96
35.
32.16
29.01
35. 16.16
32.25
miixtral-&76--inettn (3-shot)
47.01
$7.45
52.23
47.05
52.28
47.10
57.56
52.35
03-mini (3-shot)
47.87
58.51
53.
47.91
58, 56
51.24
47.96
58 62
53.29
48,00
58.67
53.34
gp--5 (3-ahot)
82,87
101.29
92.08
82.92
101. 34
92.13
82.96
101 40
92.18
83.01
101.45
92.25
2eeen225-7-iinstrrcr (3-shot)
41.12
50.26
45.0
41.17
50.31
45.74
41.21
50.37
45.79
41.26
50.42
45.84
quen quen:3-238 3-235b-22-222 a22b (3-sbot)
68.57
83.81
76.19
68.62
83.87
76.24
68.66
83.92
76.29
68.71
83.98
76.34
grok-4 (3-hhot
82.19
100.82
91.65
82.53
100.97
91.70
82.58
100.93
91.75
$2.62
100.98
91.80
3-shot learning w/ reasoning
path
cluude-3,5 3,5- (3-shot with pubh)
16.76
57.15
51.95
16.88)
$7.20
$2.00
46.85
57.26
$2.05
46.89
$7.31
52.10
clande--onne-4 4 (3-boo with path)
63.79
77.96
70.87
63.83
78.02
70.92
63.88
78.07
70.97
63.92
78.13
71.02
deepeeek-r1 1-0588 (3-shot with path)
80.06
108.86
89.11
108
99.01
89.15
10R
99.06
89.20
109.02
99.11
gemini-2.5. pro (3-ahot
path)
101.02
91
92.70
101
91.
82.75
101.13
91.94
82.79
101.19
91.99
gpt 4o (3-shot with path)
35.13
42.93
39.03
35.17
42.99
39.08
35,22
43.04
39.13
35.26
43.10
39.18
Ilama- 3.1 405b instrcet (2- with path)
53.58
65.49
59.53
53.63
65.54
59.55 19
$3.87
65. 60
59.63
53.72
65.65
59.68
Ilama-3, 3.1 Sb- instruct (3-shot with puih)
42.62
$2.09
47.36
12.67
52.15
47.41
42.71
52. 20
17.46
42.76
$2.26
47.51
shi-4-reawmnng-pll (3-shot with puil))
46.89
$7.32
52.11
$7.37
52.21
52.26
Diktral-877-inddrr (3-ahot with puth)
43.34
52.85
45.05
43.29
52.91
48. 10
43.35
48.15
$3.02
48:20
a3-mini (3-shot with path)
49.12
60.04
54.58
49.17
60.09
54.63
49.21
60.15
54.68
49.26
60.20
54.75
gpt- (3 (3-dhot with pazh)
$5.81
108.55
98. 68
$8.86
108.61
98. 28.73
$8.90
108. 06
98.78
$8.95
108.72
98.83
guen 2.5-7b-instract (3 shot with puth)
47.49
58.04
52.76
47.53
58.09
52.81
47.58
58, 15
$2.86
47.62
58,20
52.91
quen 3-235h-22 a22b (3-shot puil))
83.61
102.20
83.66
102.25
92.95
83.70
102.31
93.00
83.75
102.
93:05
grok-4ffist (3-shut with puth)
$5.41
108.06
98.3 04.24
38.46
108.12
98.33 29
88.50
108.17
98.34
88.35
108.23
98.39
Table 10: Performance on Task variant: label completion of accuracy (%).
Model
2 statements
statements
4 statements
5 statements
Z2ro
few
low.path
2er0
Fow
Tew.path
2CTO
Few
low.ppat
zero
Fow
few.poth
claude-3.5-haiku
0.112
0.341
0.765
0.116
0.171
0.196
0.259
0.303
0.323
0.378
0.419
0.470
claude-sonnet
0.312
0,587
0.672
0.518
0.617
0.654
0.413
0.534
0.713
0.319
0.576
0.654
deepscok-11-0528
0.582
0.595
0.608
0.646
0.667
0.747
0.570
0.598
0.646
0.523
0.622
google-gemini-22 2.5 pro
0,890
0.912
0.966
0.828
0,867
0.878
0.758
0.759
0.814
0.765
0.790
0.792
gpt-4o
0.276
0.291
0.363
0.260
0.274
0.294
0.243
0.307
0.341
0.302
0.366
Mama--31 3311-405b-isstuu
0,280
0,335
0.490
0.227
0,263
0.327
0.205
0.262
0.275
0.158
0.162
0,189
llama 3.1 instruct
0.546
0.194
0.264
D.138
0.216
0.290
0.150
phi-4-reasoning plus
0.423
0,532
0.654
0.432
0,546
0.672
0.598
0.657
0.677
0.243
0,542
0.564
mixtral 8x7b instruct
0.122
0.143
0.294
0.297
D.102
0.178
0.274
0.143
o3-mini gpt-5
0.527
0,582
0.615
0.702
0.740
0.780
0.610
0.670
0.688
0.532
0.632
0.668
0.876
0.924
0,883
0.891
D.789
0.809
0.778
0.790
qwen2.5-7b 7-b-in--bb
0.469
0.497
0.501
0.552
0.553
0.638
0.489
0.601
0.619
0.473
0.513
0,526
gwen3-235h.oo
0.512
0.522
0.460
0.503
0.538
0.392
0.395
0.483
0.168
grok-4ffsst
0,613
0.637
0.640
0.665
0
0.823
0.720
0.728
0.813
0.686
0,708
0,776
Formulation of Task 2: Easy Samples. In the original Task 2, smaller open-source models such as 1 11aaam3..... 1..ma-3..-8b---- 1e..1--biine mob-instruct and phi-4-reasoning--eeeerrr
struggle with complex depen- dency structures. To provide controlled easier variant, construct test sets by selecting the 1,000 samples with the shortest reasoning paths and the 1,000 samples with the shortest natural language statements to reduce logical and linguistic complexity, respectively.
22

---

Published as a conference paper at ICLR 2026
Table 1I: Precision and Recall in Task
2 Statements
3 Statements
4 Statements
Statements
Model
Precision
Recall
Precision
Recall
Precision
Recall
Precision
Recall
Haude-3.5-haiku_zee
0.648
0.396
0.436
0.196
0.414
0.095
0.278
0.034
Maude-3.5-haiku.ff
0.451
0.444
0,505
0.219
0.398
0.087
0,316
0.044
claude-3.5-haid he-hhaiiuuwwaalll
0,588
0.528
0.522
0.227
0.385
0.091
0,302
0.036
claude-sonnet
0.591
0.476
0.523
0.263
0,547
0.168
0,531
0.076
claude-sonnet--lfoww
0.587
0.479
0.50M
0.264
0.531
0.161
0.499
0.076
clande-sonne--Lfeepaahh
0.615
0.486
0.534
0.283
0.487
0.136
0.482
0.066
deepseek-r1
--pppsee---1052888002 zero
0.109
0.110
0,081
0.069
0.064
0.035
0,429
0.375
deepseek-r1
-0528_
0.111
0.133
0.084
0.072
0,504
0,341
0,483
0,455
0.110
0.123
0.109
0.100
0,074
0.043
0.716
0,546
gemini-2.5-prro.zen
0.609
0.495
0.556
0.388
0.181
0.077
0.077
0.033
gemini-22.5ppro.fet
0.584
0.489
0.172
0.116
0.158
0.071
0.084
0.027
gemini-2.5-pp -.so...eaat
0.595
0.481
0.318
0.220
0.178
0.072
0.012
0.004
gpt-4o_zero
0.133
0.104
0.109
0,049
0.054
0.016
0.091
0.014
gpt-4o_few
0.127
0.101
0.221
0.103
0.016
0.004
0.117
0.018
gpt-4o_fewpath
0.044
0.034
0.071
0.031
0.041
0.013
0.003
0.000
gpteS_
0.852
0.702
0.875
0.739
0.886
0.667
0.913
0.566
gpt-5.few
0.859
0.730
0.856
0.720
0.912
0.716
0.894
0.590
gpt-
0.782
0.688
0.897
0.777
0.894
0.669
0.926
0.626
grok-4-fastzzero
0.000
0.000
0.309
0.299
0.364
0.187
0.340
0.157
grot-4-fast_few
0.178
0.133
0.355
0.214
0,434
0.259
0,255
0.143
fask--tfa
0.108
0.078
0.417
0.382
0.369
0,213
0,421
0.241
1lama-3.1-405b
0.517
0.386
0.576
0,421
0,520
0,461
0.427
0,444
1lama-3. 1.m5-..-405bbfee
0.592
0.403
0.623
0.453
0.523
0.453
0.447
0.466
1lama-3.1 3.1-3..514_beeeee fe0Sahfe
0.611
0.430
0.633
0.445
0.574
0.485
0.467
0.497
Ilama 1i-----binttuuttzeee
0.287
0.002
0.326
0.000
0.117
0.056
0.186
0.036
3. 1.tua-3.--bbiinn few
0.229
0.010
0.316
0.000
0.083
0.032
0.101
0.017
11ama-3.1- -t---iittruttffwppattt
0.283
0.000
0.342
0.006
0.107
0.038
0.097
0.029
mixtral-8x7b_ee
0.110
0.121
0.139
0.076
0.128
0.046
0.083
0.010
Nextral-xx7b.B5
0.180
0.333
0.131
0.060
0.033
0.013
0.102
0.058
mixtral-8x7b Sewwpah
0.037
0.065
0.178
0.104
0.000
0.000
0.074
0.012
03-mini_zero
0.099
0.076
0.023
0.011
0.176
0.046
0.040
0.006
03-mini_few
0.085
0.063
0.083
0.035
0.165
0.044
0.024
0.003
03-mini fewpath
0.100
0.081
0.021
0.012
0.013
0.003
0.152
0.031
0.229
0.206
0.140
0.082
0.167
0.037
0.249
0.044
rell---rasooingpllus...o wlw.-orruuning-puu..e.ewa
0.296
0.266
0.134
0.074
0.135
0.053
0.171
0.038
0.251
0.246
0.151
0.085
0.142
0.052
0.214
0.039
qwen-2.5 2.5- abbiintruutt_ee zero
0.544
0.391
0.504
0.410
0.364
0,371
0.204
0.292
(wen-2.5-7b-ing tol-iinstuutfee few
0.637
0.436
0.521
0.441
0.364
0.376
0.221
0.307
(wen-2.5-77binnse instruct.fewpathh
0.670
0.451
0.543
0.429
0.398
0.389
0.227
0,314
qwen3-235b-122b-22 zeroo
0.667
0.430
0.529
0.372
0.480
0.163
0.570
0.078
(wen3-235b-a222 few
0.758
0.450
0.599
0.386
0.459
0.113
0.601
0.069
fewpath
0.592
0.376
0.632
0.402
0,500
0.121
0.533
0.066
L COMMONSENSE CONDITION RESULTS
M HUMANIZED PROMPTS
We use the following prompt to humanize the statements.
Please rewrite the follewing sentences into natural human-styy English. Requirements.
Do NOT keep any numeric labels (no "1)", "2)", "First,..." etc.).
Do NOT place "no:" or any negation operator at the beginning of a sentence. Negation must appear inside the clause in natural way.
23

---

Published as a conference paper at ICLR 2026
Table 12: Performance in Task 2 with short Reasoning Paths.
Model (mode)
Precision
Accuracy
Recall
Statements
Fl
Accuracy Preeision
Statements Revall
Procision
Accuracy on 4 Statements
Rocall
F1
Precision
Accuracy
Recall
Statements
F1
cback-3,,5-a 3.5..-haiikk.eee zero
0,609
0.471
0.531
0.345
0.435
0.384
0.246
0,420
0.310
0.198
0.420
0.269
0.643
0.489
0.556
0.370
0.406
0.244
0,419
0.312
0.255
0.474
0.332
chande 3.5 h.5-butk poth
0.753
0.511
0.630
0.378
0.460
0.415
0.291
0,170
0.362
0.250
0,500
0.359
chonk-soenct- 1osstt---aa -usero
0,725
0.544
0.622
0.507
0.533
0.519
0.459
0.583
0.514
0,708
0.741
0.724
clande-somiett44jc
0.543
0.623
0.54%
0.543
0.478
0.594
0.530
0.752
chande-soenct- 1_.ew path
0.745
0.550
0.633
0.537
0.557
0.516
0.187
0.609
0.511
0.797
0.770
0.783
deepseck-r1-05255 zero
0.613
0.439
0.512
0.258
0.257
0.257
0.523
0.444
0.480
0.499
0.430
0.462
dexpseck-1-00222
0.535
0.573
0.957
0.854
0.902
0.450
doxpseck-r1-0528.0 path
0.834
0.620
0.708
0.628
0.580
0.603
0.976
0.858
0.913
0.601
0.568
0.634
gemini-
Semini-2.5-pennzee zeropen_zero
0.746
0.550
0.639
0.457
0.468
0.452
0.390
0.525
0.448
0.551
0.872
0.676
gemini-2.5-pro._
0.744
0.564
0.642
0.464
0.508
0.485
0.431
0.585
0.497
0,993
0.949
0.970
gemmini-2.-ppo.few path
0.765
0.558
0.645
0.626
0.566
0.594
0.488
0.611
0.542
0.997
0.952
0.974
gpt-4o zero
0:450
0,430
0.439
0.184
0.260
0.216
0.251
0,451
0.323
0.2335
0.457
0.311
gpt-4o_few
0.459
0.423
0,441
0.325
0.357
0.278
0.453
0.344
0.343
0.461
0.318
spt-fo__eww poth
0:481
0.436
0.458
0.396
0.436
0.415
0.321
0.509
0.393
0.364
0.571
0.445
Iaama instractttttttttt zero
0.498
0.499
0.498
0.236
0.456
0.311
0.742
0.812
0.775
0.307
0,703
0.427
11ama-3.1 3.1-:111---0--mttract
0.499
0.499
0.501
03450
0.750
0.811
0.77%
0.333
0.461
Ilama-31 3.11-0511-bbmmmee 10000-insttractfeee path
1,000
0.794
0.885
0.454
0.719
0.557
0.881
0.816
0.847
0.355
0.771
0.486
Iaama-3.1 1.b..nnnttssntctttta zero
0.526
0.484
0.505
0.147
0.234
0.178
0.145
0.308
0.197
0.070
0.308
0.114
-1ama-31--00-mnns few
0,488
0.402
0.413
0,202
0.431
0.276
0.358
0,135
Iaama-311 3.16-.3.1--0-- 3.1--00-nsmmmtt..ff few path
0.999
1.000
1.000
0.495
0.503
0,490
0.315
0.521
0.392
0.008
0.391
0.157
phi-4-reasoeinngphue zero
0,230
0.251
0.240
0.167
0.253
0.201
0.158
0.234
0.185
0.152
0.218
0.179
phi---reasooing- flas
0.287
0.279
0.283
0.204
0.291
0.240
0.176
0.243
0.204
0,167
0.246
0.199
htt-4-eaooiing-luu__Ss path
0.325
0.346
0.335
0.283
0.365
0.319
0.259
0.335
0.292
0.239
0.319
0.273
mixtral-Sss 8x77 instruct zero
0,000
0.000
0.000
0.004
0.008
0.005
0.046
0.122
0.066
0.000
0,000
0.000
miixtral&x77h-nnnttn
0.045
0.068
0.084
0.059
0.112
0.077
0.056
0.153
0,082
0.003
0.020
0.005
mixtral-Sxt hx7Db-isttuut.feee few path
0.050
0.073
0.059
0.169
0,206
0.149
0.343
0.208
0.346
0.130
o3-mini, zero
0.633
0.461
0.534
0.455
0.469
0.462
0.625
0.689
0.655
0.644
0.840
0.729
oš-mini few
0.481
0.506
0.501
0.671
0.713
0.691
0.844
0.749
a3-mini, few path
0.530
0.611
0.536
0.523
0.993
0.821
0.899
0.786
0.768
gpt-5 .eerro
0.905
0.664
0.766
0.954
0.954
0.954
0.886
0.892
0.889
0.914
0.951
0.932
gpt-5 few
0.872
0.686
0.768
0.980
0,950
0.965
0.893
0.902
0.898
0,968
0.936
0.952
gpt-5 path
0.689
0.816
0.999
0.997
0.912
0.899
0.905
1.000
0.992
0.996
qwen2.5-77 10225-7b--sstucc zero
0.597
0.428
0.198
0.601
0.566
0.583
0.182
0.365
0.243
0.096
0.363
0.152
Sren2...7biinteee few
0.612
0.433
0.507
0.646
0.578
0.610
0.245
0.501
0.329
0.352
0.766
0.482
qwee22557h-inntmuct few path
0.647
0.446
0.528
0.693
0.606
0.646
0.295
0,.49
0.370
0.405
0.802
0.539
qwen3-235b-222
204-233bb2222 -22b zero
0,952
0,940
0.946
0,342
0.358
0.340
0.367
0.6466
0.468
0,961
0.844
0.899
qwent-235h-a22h... few
0.958
0.949
0.954
0.782
0.723
0.751
0.511
0.605
0.554
0.960
0.877
0.916
qwen3-235b-a222 few path
0.962
0.955
0.813
0.715
0.761
0.514
0.671
0.582
0.974
0.884
0.927
grok faatt.zeee
0.835
0.805
0.820
0.752
0.776
0.764
0.718
0.825
0.768
0.839
0.987
0.907
grok-I-fast few
0,977
0.956
0.966
0.781
0.797
0.789
0.886
0.826
0.855
0,990
0.841
0.909
gros-4-fastffew path
0.979
0.965
0.972
0.814
0.808
0.986
0.897
0.940
0.978
0.967
0.973
Preserve all logical relations exactly.
N REAL-WORLD BENCHMARK
O TASK 1 AND TASK 2 CORRELATION ANALYSIS
As shown in Figure across all modes, Pearson correlation coefficients between Task I and Task 2 remain high (mostly > 0.6), which indicates that models that perform well in harder variants also tend rank highly in easier ones. This suggests that difficulty calibration mainly shifts the absolute performance levels while preserving the relative ordering of models. In other words, all tasks probe shared underlying reasoning capability at different levels of difficulty, which validates the robustness and coherence of our benchmark design. The correlation analysis results are shown in Figure 7
P REINFORCEMENT LEARNING
Q REINFORCEMENT LEARNING EXPERIMENT ON TASK 1
Despite the fine-tuning experiment, we also conducted one RL experiment. We applied TRL yon Werra et al. 2020) with GRPO|Shao al. (2024) on Task 1 using two base models (LLaMA-3.1-8)) 1-8h
24

---

Published as a conference paper at ICLR 2026
Table 13: Performance in Task with short Statement Length.
Model (mode)
Accuracy on Statements [1
Accuracy on 3 Statements Fl
Accuracy on Statements Fl
Accuracy on Statements
Prec.
Rec.
Prec.
Rec.
Prec.
Rec.
Prec:
Rec.
0.501
0.331
0.399
0.566
0.523
0.544
0.489
0.493
0,491
0.384
0.548
0.452
S.sule-.--huikuffes
0,499
0.333
0.400
0.589
0.534
0,560
0.507
0.499
0.503
0.452
0.59%
0.513
claude-355 1oude--.5-haiku... few palh:
0.504
0,333
0.401
0.630
0.561
0,593
0.603
0,549
0.574
0.497
0.622
0.552
claudc-somnct-4..
0.651
0.788
0.610
0.688
0.513
0.525
0,500
0.603
0.547
clunde-somnet1 few
0.814
0.546
0.654
0.520
0.633
0.714
0.516
0.548
0.531
0.753
0.707
0.729
chande-sonet- f.wew path
0.830
0.546
0.639
0.829
0.657
0.733
0.555
0.557
0.556
1,000
0.756
0.861
deepseekk-t zero
0.827
0.739
0.591
0.773
0.691
0.730
0.947
0.611
0.743
deeppock-+110022..com Jow
0.841
0.603
0.702
0.788
0.710
0.747
0.768
0.798
0.783
0.839
0.721
0.775
despscck-1-0528..m fow path
0.889
0.817
0.852
0.784
0.730
0.756
0.789
0.500
0.795
0,913
0.780
0.841
gemini-2255ppon
0.788
0.542
0.642
0.751
0.586
0.658
0.333
0.496
0.399
0.335
0.529
0,410
gemini 1omini-2.55ppp 2.5-pro few
0.879
0.615
0.723
1.000
0.713
0.833
0.532
0.517
0.524
0.672
0.717
0.694
gemini-2.5ppoo.eew path
0.941
0.640
0.762
1.000
0.715
0.834
0.537
0.633
0.581
0.676
0.740
0.706
gPt-40_zere
0.641
0.469
0.541
0.606
0,532
0.567
0.480
0.500
0.490
0.368
0.540
0.438
get-to_eww
0.644
0.485
0.554
0.636
0.561
0.596
0.493
0.516
0.504
0,383
0.544
0,450
gpt-lo few path
0.922
0.627
0.747
0.689
0,566
0.621
0.602
0.568
0.585
0.459
0.606
0.523
Iaama-3.1 3.tt-...---b-istttue zero
0.697
0,545
0.612
0.537
0.522
0.529
0.285
0.358
0.318
0.479
0.604
0.534
I1ama-3.1 1-005b instruct__www
0.745
0.583
0.654
0.745
0:572
0.647
0.503
0.537
0.519
0.585
0.668
0.624
Naaa--311 1-400binntrrut_feww pach
1.000
0.747
0.855
0.996
0.666
0.798
0.504
0.544
0.524
0.664
0.090
0.677
Baam-33 3. 1-8h-innttrect,eee
0.325
0.389
0.354
0.350
0.408
0.377
0.980
0.625
0.766
0,503
0.436
D.467
1amm-3.1-50b-nnse fow
0.499
0.479
1.000
0.725
0.693
0.610
0.649
Ilama-3.1 3.1-hb instract few path
0.497
0.429
0.461
0,518
0.484
0.500
1.000
0.735
0.848
0.868
0.756
0.808
phi-4-reasoning plus zero
0.512
0.427
0,466
0.509
0.494
0,501
0.332
0.401
0.363
0.292
0.492
0.367
0.527
0,440
0.480
0.564
0.516
0,539
0.362
0.418
0.388
0.332
0.505
0.401
pli--4-easoning--i Jew path
0,540
0,438
0.484
0.582
0.531
0,555
0.379
0.446
0.410
0,408
0,582
0.480
mixtral-8x770 707----sttttn..ce
0.374
0,368
0.369
0.412
0.137
0.224
0,097
0.240
0.138
mixtral-B Br--Bx-7Tsnnaa few
0:120
0.434
0.427
0.448
0,461
0,154
0.155
0.211
0.189
0.098
0.236
0.138
mixtral-8x77b cerul--x---nstmmeett few path
0.750
1,000
0.857
0.502
0.487
0.494
0.168
0.260
0.204
0.190
0.383
0.254
o3-mini zero
0.516
0.618
0.752
0.587
0.634
0.583
0.716
0.720
0.718
o3-mini_few
0.782
0.533
0.634
0.800
0.627
0.703
0.069
0.581
0.622
0.803
0.753
0.777
o3-mini few path
0,907
0.611
0.730
0,848
0.865
0.856
0.671
0.600
0.634
0.837
0.759
0.796
gpt-5.zcrr gpt-5 few
0.830
0.586
0.687
0.897
0.777
0.833
0.738
0.694
0.715
0.747
0.731
0.739
0.835
0.597
0.696
0.945
0.746
0.834
0.884
0.811
0.846
0.745
0.734
0.739
[pe-5_few path
0.949
0.798
0.867
0.937
0.796
0.861
0.866
0.844
0.855
0.887
0.839
0.862
qwen2.5 5-t.--nsttetttt....mmmmmmmm zero
0.751
0.529
0.621
0.502
0.465
0.483
0.201
0.296
0.239
0.206
0.384
0.268
qwen2. S.w.-------nttt... Jew
0.757
0.533
0.626
0.552
0.498
0,523
0.198
0,302
0.239
0,246
0.431
0.313
qwen2.5-7b
few path
0.539
0.643
0.600
0,521
0.219
0.353
0.292
0.254
0.441
0.322
qwen3-225 2313--33b-222 aoro
0.905
0.613
0.731
0.876
0.622
0.727
0.199
0.573
0.533
0.876
0.767
0,818
qwen3. 201-2355--222 a22h few
1.000
0.717
0.835
0.884
0.754
0.814
0.579
0.641
0.808
0.882
0.766
0.820
qwen't (-25hh-33 Jew path
0.999
0.725
0.840
0.999
0.717
0,835
0.633
0.702
0.666
0.995
0.793
0
grok gros-4-fast,zee
0.748
0.508
0.605
0.857
0.727
0.787
0.093
0.600
0.643
0.886
0.796
D.839
grok-4-Gastfew
0.641
0.907
0.715
0.772
0.643
0.702
0.899
0.828
0,862
grok-4-fast fast few path
0.850
0.560
0.675
0.921
0.709
0.801
0.769
0.658
0.709
0.891
0.847
0,868
Table 14: Results on the commonsense version.
k
k=5
Model
Fmtt
Ex
Prec
Rec
Fl
Fmm
Ex
Prec
Rec
Fmm
Ex
Prec
Rec
Fl
Fmz
Ex
Prec
Rec
Fi
fastt
0.62
0.13
0.83
0.73
0.78
0.71
0.10
0.79
0.82
0.81
0.54
0.03
0.65
0.70
0.67
0.49
0.00
0.78
0.63
0.69
2pt-5
093
0.23
088
0.78
0.30
0.72
0.78
0.93
0.72
0.76
0.74
0.15
0.60
0.64
doepseck-11-995 110022
0.83
0.26
0.72
0.61
0.56
0.71
0.14
0.81
0.51
0.63
0.46
0.05
0.80
0.54
0.64
0.58
0.03
0.81
0.55
0.66
claude-some
0.3
0.04
0.54
0.63
0.44
0.72
0.03
0.42
0.61
0.50
0.89
0.00
0.53
0.62
0.57
0.63
0.00
0.67
0.74
0.70
quenk.235b-a2bb
0.63 0.15
0.56
0.63
0.45
0.13
0.65
0.43
0.68
0.83
0.64
0.72
0.03
0.57
0.55
gemini-2.5-pro
0.98 0.04
0.64
0.56
0.46
0.33
0.02
0.90
0.73
0.81
0.92
0.09
0.81
0.8
B83
0.83
0.05
0.86
0.52
D.65
1lama-3.1 3.1-1050-inntttee
0.63 0.04
0.76
0.76
0.74
0.73
0.00
0.53
0.63
0.58
0.65
0.00
0.51
0.67
0.58
0.59
0,00
0.60
0.62
0.61
gwen2.5.7b-inne in-inntuett
0.98
0.66
0.64
0.81
0.65
0.99
0.64
0.54
0.72
0.00
0.59
0.63
phi-44ranooingg- pluus
0.44 0.54
0.69
0.62
0.55
0.55
0.02
0.42
0.42
0.42
0.96
0.00
0.51
0.64
0:57
0.87
0.00
0.69
0.75
D.72
mixtral-Sx77-innee Sx7b-isstret
1,00 0.01
0.71
0.71
0.65
0.97
0.00
0.71
0.53
0.61
0.93
0.00
0.68
0.86
0.76
1.00
0,00
0.51
0.62
0.58
o3-mini
1.00
0.74
0.01
0.53
0.97
0.53
0.57
0.00
0.61
0.66
claude-3 Saude--.S-hailkn
1.00 0.02
0.53
0.55
0.38
0.3
0.00
0.51
0:35
0.42
0.92
0.00
0.55
0.64
0.59
0.83
0.00
0.63
0.76
0.69
Iaana-3. 3.---miisttt
0.53 0.01
0.85
0.53
0.58
0.42
0.00
0.52
0.33
0.4]
0.38
0.00
0.55
0.64
0.59
0.43
0.00
0.80
0.56
0.66
gpt-lo
0.52
0.02
0.57
0.52
1.00
0,00
and Qwen-2.5-7b) and found that even small-scale RL using Task data can have modest improve- ment on Task 1, Task 2, and other logical-related tasks, which demonstrates that LogiConBench can be used as a meaningful reward source, but the inherent reasoning ability cannot be easily mitigated through data augmentation.
25

---

Published as a conference paper at ICLR 2026
zero-shot
3-shot
3-shot with path
cle sddeeee Schallo
coepsook'1 ( (a28
Model
90
owan 2. (0A
100
Overall Accuracy 180
Overall Accuracy
Overall Accuracy
(a) Task I (on hard/all samples)
zero-shot
3-shot
3-shot with path
claudee- S-hwitu tu
21
30
45
despscok-r2-0b22
27
Model
Dama- 3.1 1-aofh inttrct
grok-dTast
21
B1
25
leme-1 Jodbstnstrue
19
36
pill4 pous
42
56
64
5
owen3 2355-4228
37 Accuracy (%)
20
00
40 Accuracy
Accuracy
(b) Task 1 Variant (label completion)
Figure 5: (a) Task 1 (on hard/all samples). Hard samples are cases where the provided lists of Boolean labels come from consistent and inconsistent lists differing in only one element. (b) Task 1 Variant (label completion). One label is missing, and the model must recover it to make the full label list consistent. Light color indicates results on all samples, and dark color is for hard samples.
zero-shot
3-shot
3-shot with path
Chuds scenet e 3
gemaniza
Mostel
55
lamee 31 Beeeetts
gli 4 reasoriing Dlu 200rd-2300--222
14
20
10
20
40
50
6000
10
20
30
40
50
20
30
40
50
50
(a) Task on Short Reasoning Path Samples
zero-shot
3-shot
3-shot with path
clausbe- S. Sulakku ku
2
Mustel
gemiit
gel-ioih m1
30
Fomo-d 1.153
phu- de
17
21
25
20
15
20
23
30
20
15
20
30
10
25
40
(%)
(b) Task on Short Statement Length Samples
Figure 6: Performance on Task 2 under two easy conditions: (a) short reasoning paths and (b) short statement lengths. The lighter bars indicate the average Exact accuracy score on Task 2.
Q.1 REWARD MODEL
We define rule-based reward function that assigns scores based the alignment between the model's 's response and the ground-trutt truth label y E (correct, incorrect, unknown]. Specif- ically, let a denote the model's response and y [correct, incorrect, unknown). The reward is computed as:
1.0 if conttainn(a.)))
r(a, v)
0.5 if e (v), contains(a./"),
(1)
0.2
otherwise.
26

---

Published as a conference paper at ICLR 2026
Table 15: Results on the counterfactual version.
Model
Fint
Ex
Prec
Rec
F1
Pint
Ex
Prec
Rec
F1
Fint
Ex
Prec
Rec
F1
Pmt
Ex
Prec
Rec
P1
grok-4-fast
0.59
0.19
0.85
0.75
0.80
0.69
0.10
0.39
0:42
0.41
0.51
0.02
0.25
0.31
0.28
0.49
0.00
0.13
0.21
0.16
gpt-5
0.90
0.75
0.85
0.78
0.89
0.27
0.72
0.78
0.93
0.24
0.72
0.76
0.74
0.3
0.65
0.68
0.66
deepseck-r1-0528
0.89
0.23
0.04
0.33
0.01
0.69
0108%
0.02
0.51
9.04
0.43
0.02
0.05
0.05
0.03
0.5%
0.03
0.34
0:35
0.35
cludd--onnet-4
0.89
0.02
0.56
0.65
0.44
0.70
0.01
0:42
0:41
0.43
0.88
0.00
0.53
0.32
0:40
0.63
0.00
0.14
0.24
0.18
gwee332335h 235)
0.58
0.00
0.66
gemini-2.41 pm5
0.95
0.04
0.66
0.58
0.46
0.91
0.10
0.40
0.43
0.41
0.89
0.04
0.02
0.05
0.03
0.3
0.05
0.06
0.52
0.11
Naama-3.1 311-455b-isstuuc
0.50 0.01
0.38
0.28
0.12
0.71
0.00
0.53
0.63
0.58
0.63
0.00
0.53
0.36
0:4
0.59
0.00
0.35
0.32
0.32
quee22.5-7h-ae 5-7-inss-t
0.80
0.01
0.21
0.44
0.10
0.93
0.04
0.01
0.12
0.06
093
0.00
0.02
0.34
0.04
0.87
0.00
021
0.05
0.08
mixtral- 8c7b insttrctttt
0.92 0.00
0.33
0.33
0.12
0.95
0.00
0.03
0.53
0.06
0.99
0.00
0.07
0.35
0.11
1.00
0.00
0.04
0.02
0.03
as-mini
0.05
0.76
0.03
0.05
0.53
0.08
0.2
001
claude-3.. 3.5.haikk
0.55
0.27
0.00
0.21
0.92
0.14
0.06
0.04
1oma--.11 188-insttrcc
0.50
0.01
0.27
0.55
0.17
0.40
0.00
0.52
0.33
0.41
0:35
0.00
0.06
0.03
0.04
0.43
0.00
0.0
0.06
0.06
gpt-to
0.87
0.01
0.55
0.51
0.36
0.96
0.03
0.05
0.01
1.00
0.00
0.32
0.06
0.05
0.05
0.01
Table 16: Results on the humanized version.
km2
km 3
k
k=5 5
Model
Prec
Fint
Prec
Rec
grok-I-fast
0.82
0.23
0.75
0.72
0.74
0.73
0.11
0.69
0.62
0.65
0.62
0.09
0.70
0.61
0.65
0.53
0.01
0.70
0.61
0.65
gpt-5
0.98
0.29
0.65
0.63
0.64
0.91
0.20
0.73
0.83
0.78
0.93
0.13
0.68
0.83
0.75
0.90
0.68
0.83
0.75
dcrpscek-r1-0528
082
0.96
0.41
0.04
8.90
0.08
0.83
0.76
0.8
0.04
0.69
0.76
0.87
0.86
0.69
0.76
clande-sonee--4
0.87
0.04
0.46
0.53
0.38
0.82
0.06
0.63
0.62
0.62
082
0.08
0.58
0.63
0.00
0.83
0.00
0.58
0.63
0.60
qwea3-23550 226
0.62
0.20
0.48
0.63
0.47
0.70
0.04
0.64
0.62
0.63
0.73
0.03
0.65
0.63
0.64
0.73
0.01
0.5
0.6
0.64
gomini-2.5-po0 p.5-p
1.ama-3.1 405b-iinsrrtt
0.51
0.00
0.33
0.28
0.30
0.70
0.00
0.54
0.63
0.58
0.79
0.00
0.49
0.63
0.55
0.81
0.00
0.49
0.6
0.55
qwen2.5 7022555hh---neee instrrctt
0.89
0.02
0.39
0.36
0.37
0.95
0.00
0.75
0.64
0.69
0.93
0.00
0.77
0.63
0.69
0.89
0.00
0.77
0.63
0.69
phi Shil---renoeing-plun
0.28
0.44
0.05
0.84
0.8
081
mixtmi-8s777 8c7b-instuc
0.92 0.03
0.33
0.33
0.33
0.96
0.00
0.82
0.54
0.65
0.2
0.00
0.78
0.55
0.64
0.94
0.00
0.78
0.55
0.64
oó mini
0.97 0.05
0.05
0.76
0.09
0.97
0.07
0.65
0.69
0.67
0.88
0.03
0.65
0.68
0.66
0.90
0.00
0.65
0.68
0.66
Glaude-35--hiike
0.89
0.00
0.55
0.27
0.36
0.90
0.02
0.82
0.55
0.66
0.89
0.00
0.87
0.55
0.68
0.57
0.00
0,87
0.55
0.68
Ilama- 3.1 -.---iinstrrcct
0.27
0.58
0.06
0.53
0.42
0.47
0.42
gpt-40
0.87
0.04
0.55
0.54
0.54
0.98
0.04
0.72
0.1
0.76
0.92
0.02
0.76
0.80
0.78
0.93
0.00
0.76
0.00
0.78
where contains(a,) v) indicates that the response contains the label y. Intuitively, full reward of 1.0 is assigned when the response matches the ground-truth label, a partial reward 0.5 given when the response contains plausible but incorrect label, and a default reward of 0.2 is assigned to all other responses.
Q.2 EVALUATION RESULTS
Table 1 reports Task I accuracy for both LLaMA and Qwen across different values of k. measured on consistent, inconsistent, and overall subsets.
Tables 2 and 3 present Task 2 results for LLaMA and Qwen after RL training, evaluated on format accuracy, exact match, precision, recall, and FI.
Table 4 compares general downstream performance before and after RL fine-tuning.
Q.3 OVERALL ANALYSIS
RL training yields modest but consistent gains on Task across most values of kt while Task 2 ben- efits only marginally, suggesting limited transfer to enumerative reasoning. On broader downstream benchmarks including LogiQA, LogicNLI, MathQA, HumanEval, and MMLU, improvements re- main minimal or inconsistent, indicating little generalization beyond the training distribution. We attribute these limited gains to the relatively low task complexity and the tendency of RL to exploit surface-level reward patterns rather than develop deeper reasoning capabilities. Nevertheless, the fact that both RL and fine-tuning produce improvements on our benchmark suggests that LogiCon- Bench can serve useful training resource for enhancing logical reasoning in language models.
27

---

Published as a conference paper at ICLR 2026
Table 17: Performance on downstream benchmarks.
Model
LiveCodeBench
Infinite
AIME
AA-LCR
ACEBench (agent)
grok-4-fast
79.00
65.80
89.70
64.70
73.00
gpt-5
84.00
86.50
91.70
72.80
78.00
deepseek-r1-0528
64.30
36.50
68.00
52.30
64.00
claude-sonnet-4
55,90
64.60
74.30
64.70
53.00
(wen3-235b-a22b
79.00
53.20
91.00
67.00
51.00
gemini-2.5-pp
69.00
54.10
87.70
66.00
63.00
30.50
19.00
33.00
24.30
41,00
qwenn.55-7binnstruet
12.60
21.00
9.00
16.00
12.00
phi-4-reasoning-ple
23.10
32.00
21.00
20.00
15.00
mixtral-8x77-instruct
6.60
13.00
3.00
8.00
6.00
03-mini
71.70
28.70
25.00
30.00
65.00
claude-3.5-haikk
20.20
28.00
21.00
43.00
35.00
llama-3.1-88 1-0---isr-t
11.60
16.40
4.50
15.70
4.00
gpt-4o
42.50
25.10
25.70
53.00
71.50
Correlation across settings by model
Correlation across settings (mode zero)
1.00
0.84
0.75
0.66
0.62
setiing hadd
0.59
1.00
0.78
0.75
0.68
0.67
0.55
sutting wwwnpp_Jer..orr Jard_anr
xetingt
0.84
0.78
1.00
0.91
3.77
0.76
Jelf LLuee
0.35
3.65
1.00
0.59
0.67
0.67
3.20
0.46
0.73
0.59
1.00
0.90
0.82
sering
0.75
0.75
0.91
1.00
3.79
0.77
0.90
witeee wdoe
0.00
0.68
0.77
0.79
1.00
0.62
-3.5
Hetog2_ Sengh
0.67
0.67
0.76
0.77
0.62
1.00
veting wroh
Correlation across settings (mode few)
Correaation across settings (mode = texpath)
1.0
aetting, sany Jrapoth
1.00
0.62
0.52
0.64
sott hond,
0.50
1.00
0.74
0.64
0.67
0.59
sertept Jund Feeatth
0.62
1.00
0.63
0.56
0.55
0.43
0.
0.73
0.63
1.00
0.64
0.72
0.66
setting Jempati
0.52
0.56
0.64
1.00
0.88
0.73
0.54
0.64
0.62
1.00
0.91
0.78
0.64
0.55
0.00
1.00
-36
1.00
0.6
werting
0.55
0.59
0.73
0.78
0.91
3.00
Sespot
peisp Jen
Figure 7: Correlations between different tasks.
R FINE-TUNE
R.1 FINE-TUNING IMPROVES PERFORMANCE BUT THE GAINS ARE LIMITED
We conducted controlled fine-tuning experiment and found that although fine-tuning yields mea- surable improvements on the benchmark, the gains are fundamentally limited. Mcanwhile, the stor- age and computation required for fine-tuning substantial. These results suggest that the perfor-
28

---

Published as a conference paper at ICLR 2026
Table 18: Accuracy on Task I (Consistent Inconsistent Overall).
Model
k
Con
Incon
Overall
2
0.58
0.59
0.585
0.51
0.52
0.515
LLaMA
4
0.32
0.48
0.40
5
0.31
0.33
0.32
2
0.42
0.5436
0.4818
3
0.47
0.5308
0.5004
Qwen
0.33
0.4673
0.3987
5
0.30
0,3406
0,3203
Table 19: Results on Task 2 after RL training.
Model
k
Format
Exact
Precision
Recall
F1
2
0.691
0.285
0.7384
0.7962
0.7662
0.704
0.176
0.3894
0.8189
0.5278
LLaMA
0.613
0.104
0.5843
0.7320
0.6499
0.642
0.050
0.4830
0,5718
0.5237
0.989
0.183
0.4829
0.4899
0.4860
1.000
0.145
0.6348
0.4138
0,5010
Qwen
0.977
0.058
0.4342
0.4280
0.4312
5
0.992
0.030
0,2143
0.2038
0,2090
mance bottleneck primarily lies in the models' inherent reasoning limitations rather than insufficient supervision.
R.1.1 FINE-TUNING ON TASK 2 PROVIDES IMPROVEMENTS, BUT FAR FROM SOLVING THE TASK
We fine-tuned three small open-source models (Llama-3-8B, Qwen-2.5-7B, and Mistral-7B) on 1,000 synthetic Task 2 training samples generated by pipeline (100 for k=2, 200 for k=3, 300 for k=4, and 400 for k=5). Each training instance contains both the full reasoning path and the final answer, providing complete in in-domain supervision perfectly aligned with the evaluation format.
Across all models, lne-tuning improves Task performance, but the gains remain limited (Table 21 Crucially, even after fine-tuning none of the small models approach frontier- frontier-model performance.
1.1.2 FINE-TUNING DOES NOT OVERFIT: IT GENERALIZES TO TASK AND INDEPENDENT BENCHMARKS
Despite being trained solely on Task 2, all models exhibit improvements on Task (Table 22 Since Task involves distinct output format and only partially overlapping reasoning skills, this demonstrates genuine reasoning transfer rather than overfitting.
Fine-tuned models also show improvements multiple independent logical reasoning benchmarks (Table 23), including LogicNLI, LogiQA, MathQA, HumanEval, and MMLU-non none of which ap- pear in the training data. This demonstrates that synthetic data strengthens models' broader logical competence.
R.2 SUBSTANTIAL STORAGE AND COMPUTATIONAL COSTS OF FINE-TUNING
Fine-tuning not only limited in performance gains, but also extremely expensive in storage and computation.
Storage Cost. Each training instance requires storing its (i) symbolic representation, (ii) reasoning edges, (iii) atom expressions, (iv) consistency and non-consistency sets, and (v) the full logical graph. Since the logical graph grows with O complexity as the number atoms increases, the storage footprint escalates rapidly.
29

---

Published as a conference paper at ICLR 2026
Table 20: General Downstream Evaluation (Before vs. After RL)
Benchmark
LLaMA
LLaMA-FT
Qwen
Qwen-FT
LogiQA
39.6
42.1
33.0
32.5
LogieNLI
28.5
30.4
24.0
24.4
MathQA
42.8
38.2
36.0
33.6
HumanEval
22.6
25.3
32.9
33.8
MMLU
65.3
60.9
55.0
57.9
Table 21: Fine-tuning results on Task across three models.
Model
Format
Exact
Precision
Recall
0.783
0.327
0.5626
0,9362
0.7028
0.802
0.190
0.4457
0.9749
0.6117
Llama-3
0.736
0.132
0.4349
0,5556
0,4879
0.669
0.080
0.4247
0.4419
0.4331
1.00
0.213
0.6723
0.5773
0.6212
3
1.00
0.161
0.7587
0,4635
0.5754
Qwen-2.5
1.00
0.066
0.4294
0.4593
0.4438
5
1.00
0.063
0.3277
0.3393
0.3340
1.00
0.122
0.4307
0.8447
0.5705
1.00
0.113
0.3326
0.2312
0.2728
Mistral-7B
1.00
0.047
0,2626
0.2362
0.2487
5 1.00
0.000
0.1457
0.2749
0.1905
Computational Cost. Even with only 1,000 training examples, fine-tuning an 8B-parameter model requires an A800 GPU for approximately 4 hours. Achieving stronger performance would require orders of magnitude more computation, making fine-tuning impractical at scale.
30

---

Published as a conference paper at ICLR 2026
Table 22: Fine-tuning Task and evaluating on Task
Model
k
Cons
Incons
Overall
2
0.69
0.47
0.58
3
0.46
0.53
0.495
Llama-3
0.52
0.35
0.435
5
0.61
0.33
0.47
2
0.43
0.52
0.48
0.63
0.62
0.63
Qwen-2.5
4
0.67
0.79
0.73
5
0.64
0.78
0.71
2
0.81
0.61
0.71
0.74
0.48
0.61
Mistral-7B
4
0.85
0.38
0.615
0.73
0.42
0.575
Table 23: Performance on general benchmarks with and without fine-tuning.
Benchmark
Llama
Llama-FT
Qwen
Qwen-FT
Mistral
Mistral-FT
LogiQA
39.6
45.2
33.0
35.2
34.0
33,0
LogicNLI
28.5
47.5
24.0
30.6
26.0
26.0
MathQA
42.8
49.6
36.0
28.0
33.0
36.2
HumanEval
22.6
29.3
32.9
41.0
28.7
32.0
MMLU
65.3
69.3
55.0
58.0
64.2
67.0
31


Reviews:
[{"review_id": "c3XFm89jYR", "reviewer": "Reviewer_qtmJ", "summary": "This paper introduces LogiConBench, a scalable benchmark for testing LLMs’ logical consistency through automatically generated logical graphs with controllable reasoning depth. It supports unlimited rule combinations and diverse natural language formulations. Evaluations on 14 leading LLMs show that despite high accuracy on prior benchmarks, models still struggle on LogiConBench’s Enumerative task.", "questions": "please refer to weakness", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a scalable and systematic data generation pipeline capable of synthesizing a large number of well-labeled evaluation samples. This framework allows for controlled complexity, diverse logical structures, and extensibility to new reasoning settings.\n2. The authors conduct comprehensive and fair evaluations across a wide range of baseline models, covering both open-source and proprietary models.", "weaknesses": "1. There are concerns regarding the true difficulty of the proposed benchmark. In the Discriminative Task (Task 1), models such as GPT-5 and Grok-4-Fast already achieve near-perfect performance, while in the Enumerative Task (Task 2), their F1 scores reach around 70. This raises the question of whether frontier models have largely mastered the benchmark, suggesting that the observed gap for open-source models might simply reflect scaling effects rather than a fundamentally unsolved challenge.\n2. The paper does not sufficiently articulate the real-world significance of LogiConBench. It remains unclear which practical abilities the benchmark aligns with. Establishing stronger connections between LogiConBench performance and real-world downstream tasks (e.g. agents) would help clarify its broader impact and necessity.\n3. Since the benchmark’s data generation pipeline is fully scalable, one might argue that it is also straightforward to construct large volumes of in-domain synthetic data for targeted finetuning. This raises doubts about whether the reported difficulty truly reflects inherent reasoning limitations or can be easily mitigated through data augmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wzlX4EaOuQ", "reviewer": "Reviewer_cu2W", "summary": "The paper proposes LogiConBench, a benchmark that evaluates LLMs on logical consistency. It addresses three key limitations of existing benchmarks by generating scalable, diverse, and challenging problem instances. These instances are created through logical graph generation, reasoning path labeling, and translation into natural language. The benchmark is tested on 14 frontier models. The discriminative tasks are relatively easy, while the enumerative tasks remain highly challenging. The paper also introduces several task variants with intermediate levels of difficulty.", "questions": "- How are model outputs extracted for scoring? The process seems straightforward for the discriminative task (as it produces a binary answer), but it is less clear for the enumerative task, where answer formatting and extraction can be challenging. More details or examples of output parsing and scoring would be needed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and clear.\n- The motivation is strong, and the proposed benchmark directly addresses the main shortcomings of existing logical consistency benchmarks in terms of scalability, diversity, and difficulty.\n- The evaluation is extensive, covering both instruction-tuned and reasoning models, including closed- and open-source models, under three prompting setups (zero-shot, few-shot, and few-shot with reasoning paths).", "weaknesses": "- The benchmark includes only two main tasks (discriminative and enumerative). Although a few variants are explored, they remain closely related to the original tasks. The benchmark could benefit from more diverse task types to better capture different dimensions of logical consistency.\n- The qualitative analysis is somewhat limited. While Section 5.3 briefly discusses three main failure modes, a deeper qualitative examination of the reasoning paths behind these errors could significantly strengthen the paper and yield more actionable insights for researchers and practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XxrYmuBbqo", "reviewer": "Reviewer_jijm", "summary": "This paper introduces LogiConBench, a new benchmark designed to evaluate the logical consistency of Large Language Models (LLMs). The authors argue that existing benchmarks for logical consistency are no longer sufficient, as they lack scalability, diversity, and challenge; SOTA models, for instance, already achieve over 95% accuracy on them.\nTo address this, LogiConBench provides a framework including:\n1. Automatically constructing graphs where nodes represent symbolic propositions and edges represent reasoning relations.\n2. It samples sets of propositions (nodes) from these graphs, extracts the reasoning paths connecting them, and determines all possible Boolean (True/False) assignments that are logically consistent.\n3. These symbolic propositions are then translated into diverse natural language sentences using templates and lexical substitution from WordNet.\n\nThe paper releases a 280K-sample corpus and evaluates 14 frontier LLMs on Discriminative(The model must determine if a given list of Boolean labels for a set of statements leads to a contradiction) and Enumerative(The model must enumerate all possible lists of Boolean labels that make a set of statements consistent) tasks.", "questions": "Refer to weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper convincingly argues that existing benchmarks for logical consistency are \"saturated\". The results shown in Figure 1a demonstrate that SOTA models easily pass older benchmarks (like BeliefBank and LFC) but are significantly challenged by LogiConBench, proving its necessity.\n2. Unlike static, human-written datasets, the LogiConBench framework is automatic and can generate \"unlimited logical rule combinations\". This graph-based generation pipeline is a novel approach that ensures a large-scale, diverse, and renewable source of evaluation data.\n3. A key feature is the benchmark's ability to provide \"controllable-depth graphs with explicit reasoning paths\". The authors show that providing these paths as prompts (\"few-shot with reasoning paths\") improves model performance, especially on the difficult Enumerative task. This highlights the value of explainability and chain-of-thought data.\n4. The study is thorough. It tests 14 frontier closed and open-source models across multiple evaluation settings (zero-shot, few-shot, and few-shot with path).", "weaknesses": "1. The natural language generation relies on templates and lexical substitution. While this allows for scale, the resulting sentences may lack the nuance, ambiguity, and diversity of \"fully human-authored datasets\". This could make it easier for models to map the text back to symbolic logic, potentially simplifying the natural language aspect of the reasoning. At the same time, it is hard not to suspect that the constructed natural language data contains content that is consistent with or contrary to the real world, such as \"the sky is blue\". This situation may cause problems such as reasoning shortcuts or knowledge contradictions, affecting the evaluation results.\n2. Meanwhile, I believe that the task form of Enumerative task cannot well reflect the inherent logical ability of the model, because it is well known that exhaustive search has always been a serious problem for LLMs, but this does not completely mean that the model's logical level is low.\n3. The lack of detailed and profound analysis of the different performances of the model on various tasks makes it difficult to reflect the necessity and value of designing these two types of tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rKiRfIC7vi", "reviewer": "Reviewer_VUDU", "summary": "The paper introduces LogiConBench, a scalable benchmark for testing logical consistency in LLMs. A four‑stage pipeline first generates logical graphs from natural‑deduction rules, then samples some target nodes under bounded distance, propagates truth values along extracted reasoning paths to derive all consistent/inconsistent label lists, and finally verbalizes symbolic formulas into natural language via templates and WordNet substitutions. The authors evaluate 14 models, empirically finding that frontier models reach 85–95% on Task 1, whereas Task 2 remains difficult.", "questions": "* A key advantage of your framework is that it can scale to generate unlimited data. Given this, is it possible to leverage large‑scale RL on synthetically generated splits to train, rather than benchmarking, reasoning capabilities?\n* If I understand the tasks correctly: If we had a reliable Task 1 verifier, then in principle Task 2 could be solved by evaluating all 2^k assignments with Task 1 and collecting the consistent ones. Why do LLMs still struggle so much with Task 2? Is the main bottleneck long‑horizon search/planning, cumulative local errors, output‑formatting limits, or something else?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "* Data‑generation procedure appears methodologically sound: edge‑level truth‑compatibility for each operator is explicitly specified; Steiner‑tree extraction bounds depth while preserving connectivity among targets; the DFS enumerator yields exact, reproducible labels; and NL verbalization is decoupled from labeling, isolating the resulting problems from commonsense knowledge.\n* Very comprehensive evaluation across many models and many difficulty slices: 14 LLMs (open/proprietary), 2–5 statements, with rich metrics.\n* Even frontier models remain far from solved on Task 2: most models have Exact < 1%; best is gpt‑5 Exact≈0.51 with reasoning paths, while prior datasets are near‑ceiling (Figure 1a). This makes LogiConBench a timely and progress‑sensitive benchmark in an era when many others have saturated", "weaknesses": "* It is hard to gauge how diverse the generated logical problem patterns are. The study used the introduction rules for five logical operators but did not use elimination rules, which are also crucial in symbolic logic. Is it OK?, how diverse are the resulting patterns? Although we understand that defining and measuring the diversity of problem patterns is subtle and challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)