HardcoreLogic: Challenging Large Reasoning Models with Long-tail Logic Puzzle Games
Authors: Jingcong Liang
Shijun Wan
Xuehai Wu
Yitong Li
Qianglong Chen
Duyu Tang
Siyuan Wang
zhongyu wei
Venue: ICLR 2026 Poster (2026)
Keywords: long-tail benchmark
logic puzzle games
large reasoning model
Primary area: datasets and benchmarks
TL;DR: We propose HardcoreLogic, a logic puzzle game benchmark with non-canonical long-tail puzzles that evaluates the reasoning capability robustness of LLM/LRMs.

Abstract: Large Reasoning Models (LRMs) have demonstrated impressive performance on complex tasks, including logical puzzle games that require deriving solutions satisfying all constraints. However, whether they can flexibly apply appropriate rules to varying conditions, particularly when faced with non-canonical game variants, remains an open question. Existing corpora focus on popular puzzles like 9x9 Sudoku, risking overfitting to canonical formats and memorization of solution patterns, which can mask deficiencies in understanding novel rules or adapting strategies to new variants. To address this, we introduce **HardcoreLogic**, a challenging benchmark of over 5,000 puzzles across 10 games, designed to test the robustness of LRMs on the "long-tail" of logical games. HardcoreLogic systematically transforms canonical puzzles through three dimensions: **Increased Complexity (IC)**, **Uncommon Elements (UE)**, and **Unsolvable Puzzles (UP)**, reducing reliance on shortcut memorization. Evaluations on a diverse set of LRMs reveal significant performance drops, even for models achieving top scores on existing benchmarks, indicating heavy reliance on memorized stereotypes. While increased complexity is the dominant source of difficulty, models also struggle with subtle rule variations that do not necessarily increase puzzle difficulty. Our systematic error analysis on solvable and unsolvable puzzles further highlights gaps in genuine reasoning. Overall, HardcoreLogic exposes the limitations of current LRMs and establishes a benchmark for advancing high-level logical reasoning.

Published as a conference paper at ICLR 2026
HARDCORELOGIC: CHALLENGING LARGE REASONING MODELS WITH LONG-TAIL LOGIC PUZZLE GAMES
Jingcong Liang; Shijun Wan" & Xuehai Wu" Fudan University (jcliang22, sjwan25, (ahwu25)@m. fudan.edu.com
Siyuan Wang
University of Southern California sw_ edu
Yitong Li, Qianglong Chen & Duyu Tang
Zhongyu Wei! Fudan University & Shanghai Innovation Institute zywe zywei@fudan edu.
Huawei Technologies Ltd.
[liyitong3 chenqianglong, tangduyu
ABSTRACT
Large Reasoning Models (LRMs) have demonstrated impressive performance on complex tasks, including logical puzzle games that require deriving solutions satisfying all constraints. However, whether they can flexibly apply appropriate rules to varying conditions, particularly when faced with non-canonical game variants, remains an open question. Existing corpora focus on popular puzzles like 9x9 Sudoku, risking overfitting to canonical formats and memorization of solution patterns, which can mask deficiencies understanding novel rules or adapting strategies to variants. To address this, introduce HardcoreLogic, a challenging benchmark of over 5,000 puzzles across 10 games, designed to test the robustness of LRMs the "long-tail" of logical games. HardcoreLoggi systematically transforms canonical puzzles through three dimensions: Increased Complexity (IC), Uncommon Elements (UE), and Unsolvable Puzzles (UP), reducing reliance on shortcut memorization. Evaluations on diverse set of LRMs reveal significant performance drops, even for models achieving top scores on existing benchmarks, indicating heavy reliance on memorized stereotypes. While increased complexity is the dominant source difficulty, models also struggle with subtle rule variations that do not necessarily increase puzzle difficulty. Our systematic error analysis on solvable and unsolvable puzzles further highlights gaps in genuine reasoning. Overall, HardcoreLogic exposes the limitations of current LRMs and establishes a benchmark for advancing high-level logical reasoning.
Leaderboard:
ht http:///wwwgggggee..///////////r
Code: http::/ggthh..ccoo//////e/rrrrrr
INTRODUCTION
Recent large reasoning models (LRMss) (Lin al. 2025b) have demonstrated remarkable performance across tasks requiring complex reasoning. Among them, logical puzzle games have emerged as particularly prominent benchmark where models need deduce search for solutions achieve specific goals under logical rules and constraints. Such puzzles probe diverse reasoning skills, including logical deduction (Lin et al.) (025a), pattern recognition (Chollet al.) (202)). and rule induction (Li et al. 2025), while featuring well-defined rules and objectives that enable systematic difficulty control and straightforward evaluation. These characteristics make logical puzzle games an ideal testbed for assessing and advancing LRMss.
"Equal contributors. "orresponning authors.

---

Published as a conference paper at ICLR 2026
Despite recent successes on benchmarks such Enigmata (Chen al. 2025a) and ZebraLogic Lin et al. 2025a), whether LRMs are genuinely capable of true logical reasoning, ie., flexibly apply approprate rules to relevant conditions to derive correct conclusions, remains important question. Take Sudoku as an example: while most real-world puzzles follow the canonical 9x9 format with nine 3x3 zones, variants with alternative constraints or irregular subgrids often prove challenging even for humans. Similarly, existing corpora exhibit a severe imbalance between canonical and non-canonical logic puzzles, making models prone overfitting to canonical puzzles (Inger et al. 2025), leading to difficulties in solving non-canonical variants that fall into the long-tail of the distribution. This limitation manifests in two specific ways: (1) Models recognize only the canonical form of logical puzzles; when given a variant, they either struggle in understanding the new rules or ignore them, leading to faulty reasoning. (2) Models develop fixed solution strategies and reasoning patterns to solve canonical puzzles; even when they successfully understand the variant, they still apply mismatched solution strategy, eventually producing errors or being confused.
100
Bardnotadd
Irregular Subgrid
Long-bil Tramsformation
4
2
60
Incrome pasale seale
40
Add challinlng comiraines Alerr legitll rales
20
A
aptaato Da0
0
speae sost Mlekdd
minlmat-ol
getvente Jals
greved svret
Mlae-i
minimet
Letter Encoding and Diagonal Constraint
Figure 1: Left: Illustrative examples of two long-tail transformed Sudoku. The right top shows an iregular-subgrie Sudoku, replacing standard 2x2 subgrids with irregular subgrids. The bottom right shows a letter-encoded Sudoku with diagonal constraints, where each diagonal must contain all unique symbols. Right: Overview of our long-tail transformation applied logic puzzle games, shows that LRMs have consistent and significant degradation on HardcoreLogic.
Existing logic puzzle benchmarks mainly focus on canonical game forms and fail to expose the aforementioned deficiencies. To address this limitation and provide detailed inspection of LRMs' reasoming robustness. we introduce Hardcore a logic puzzle game benchmark that challenges models with long-tail variants of puzzles. Hardcorelogge transform common puzzle games along three dimensions: (1) Increased Complexity (IC) through larger search spaces and more entan- gled constraints; (2) Uncommon Elements (UE) involving novel rules and altered puzzle forms; (3) Unsolvable Puzzles (UP) generated from previously solvable puzzles. The left panel of Figure illustrates two examples of Sudoku transformations designed to increase puzzle difficulty. These enhancements reduce the likelihood that puzzles in HardcoreLogic appear in training corpora, thereby limiting gains from memorizing canonical forms fixed reasoning patterns.
HardcoreLogia comprises over 5, 000 puzzles spanning 10 logical puzzle games, covering logical deduction, pattern recognition, and sequence searching. Each game is transformed in multiple ways among the three aforementioned dimensions. Comparing with existing datasets of the same games, our puzzles exhibit higher theoretical complexity (for IC puzzles and UE puzzles with novel rules) and higher model perplexity (for UE puzzles with altered forms). Furthermore, our UP puzzles address the absence of unsolvable logical reasoning tasks in mainstream benchmarks.
We evaluate HardcoreLogic across multiple popular and state-of-the-art (SOTA) LRMs, ranging from models (The right panel of Figure compares
small distilled models large open/closed-souue the performance of multiple LRMs on the Original and HardeoreLogic). All models, including SOTA models that achieve top performance on baseline benchmarks (e.g., GPT-5), suffer significant performance degradation on HardcoreLogic. Models with stronger reasoning abilities generally exhibit smaller relative drops; however, we also observe large-parameter models that score moderately on the baseline but perform poorly on HardcoreLogic, suggesting the presence of puzzle-gand stereotypes in these models. The primary source of difficulty HardcoreLogic stems from increased complexity, yet also identified cases where puzzles with novel rules (without added complexity or perplexity) still misled many models. For unsolvable puzzles, models often failed to detect unsolvability and instead produced "partial solutions" that were clearly incorrect.
2

---

Published as a conference paper at ICLR 2026
We further conduct a systematic enror analysis to probe the underlying causes of model failures HardcoreLogic. For solvable puzzles, we classify erroneous responses into six categories, and find that factual errors dominate across models, while more powerful models tend to exhibit brute-force errors, attempting exhaustive searches rather than strategic reasoning. Besides, models' misunderstanding of problem constraints and misapplication of rigid rules lead to significant performance drops. For unsolvable puzzles, our analysis reveals that models performing well on solvable problems genuinely recognize unsolvability better. However, weaker models like Minimax-MI may output "unsolvable" simply when they fail to find an answer, rather than through true recognition of logical unsatisfiability. When models fail to recognize unsolvability, we observe that stronger models mainly fail due to erroneous reasoning or inability to output answers within token budgets, while weaker models tend to force out solutions even without successfully deriving them. These highlight the need to improve models' deep reasoning capabilities and robustness against degenerate behaviors. Overall, our contributions are threefold:
We introduce HardcoreLogic, enhanced benchmark spanning 10 types of logic puzzle games, designed to challenge LRMs with long-tail variants of common puzzle games, featuring higher complexity, novel elements, and unsolvable options.
We evaluate HardcoreLogic on mainstream and SOTA LRMs, uncovering the limitations of their reasoning abilities. All models, including the latest SOTA models, show substantial performance degradation on puzzles with increased complexity or unfamiliar forms, and exhibit varying behaviors on unsolvable puzzles.
We conduct systematic error analysis of LRMs on HardcoreLogic, revealing diverse failure modes and suggesting directions for improving the model's deep reasoning abilities and robustness. In addition, our automatic data construction pipeline provides scalable protocol for building model training data and environments.
2 HARDCORELOGIC
In this section, introduce the HardcoreLogie benchmark, describing the covered logic puzzle types, the long-tail transformation process with statistical analysis, and a detailed complexity analysis of long-tail transformations.
2.1 PRELIMINARY: LOGICAL PUZZLE GAMES
In HardcoreLogic, we focus on 10 types of logic games spanning 6 puzzle categories, including challenging subtasks sourced from Enigmata (Chen et al.) 2025a), the Zebralogic game from the ZebraLogicc dataset (Lin et al.] 2025a), and a classic Hanoi game synthesized by ourselves following its standard rules. All these three sources constitute the Original data used for comparison with HardcoreLogic. Specifically, HardcoreLogic covers the following categories: (1) logic puzzle, (2) grid puzzle, (3) search puzzle, (4) pattern puzzle, (5) graph puzzle and (6) sequential puzzle. The 10 specific games are Zebralogic. Sudoku, Skyseraper, Kakurasu, Crypto. Navigation, Binario, Minesweeper, Hanoi and Hitori. See Appendix B. for a more detailed introduction.
2.2 LONG-TAIL TRANSFORMATION
Standard logic puzzles are constrained in size, form diversity, and rule design, and thus fail to capture the irregularity and scale of real-world reasoning. To systematically construct more challenging evaluation data, we introduce a set of long-tail transformations that extend puzzles along three distinct dimensions: Increased Complexity, Uncommon Element, and Unsolvable Puzzle.
Taxonomy
We categorize transformations into five types from three families:
Increased Complexity (IC) enhances difficulty by expanding the search space and depth of reasoning. Search space expansion (IC1) enlarges the number of candidate states by reducing the number of initial givens or scaling the puzzle size. For example, removing as many digits as possible while ensuring unique solution in Binario. Constraint strengthening (IC2) increases entanglement among constraints to demand longer reasoning chains. For example, in Zebralogic, instead of Pet-dog Sport-football + 1, we use looser condition like Pet-dog > Sport-foothall.
3

---

Published as a conference paper at ICLR 2026
Uncommon Element (UE) modifies question forms rules, often inducing out-of-distribution generalization. Form variation (UE1) introduces new types of question forms, such as applying constraints onto irregular subgirds and replacing digits with letters in the Sudoku. Rule variation (UE2) alters or hybridizing the governing principles. For example, in Sudoku, we introduce a diagonal constraint requiring that digits both main diagonals must also be distinct.
Unsolvable Puzzle (UP) deliberately lacks a valid solution, distinguishing them from harder-but- solvable cases. They are used examine whether large language models can detect inconsistency or insufficiency of information, rather than hallucinate plausible but incorrect answers.
Basic statistics The different logie puzzles in HardcoreLLogie have different ways of long-tail transformation types. Table details the aspects of long-tail transformation types that each logic puzzle has. The rules for each logic puzzle and their more specific long-tail transformation details can be found in the Table B and each task may correspond to multiple long-tail transformation types.
Table I: Statistical details of Original and HardcoreLogic on different games and transformations, with the second and last column respectively representing the total sample size of Original and HardcoreLogic different games. Note that some puzzles belong multiple transformation categories, so row sums may exceed the overall total.
Long-tail Transformation
Game
Original
Overall
Complexity
Element
Unsolvable
IC1
IC2
UEI
UE2
Zebralogic
100
x
X
V
400
Sudoku
100
V
V
V
550
Skyscraper
200
V
X
V
800
Kakurasu
V
V
300
Crypto
300
V
X
X
V
400
Navigation
100
X
X
V
300
Binario
150
V
X
V
450
Hanoi
140
800
Hitori
100
Minesweeper
150
V
V
750
Overall
1389
1350
1150
1400
850
1350
5250
2.3 COMPLEXITY ANALYSIS
To systematically evaluate the hardness introduced by our long-tail transformations, we conceptualize complexity as four-dimensional construct: Search Space Expansion (IC1), Constraint Strengthening (IC2), Form Variation (UEI), and Rule Variation (UE2). Each transformation associated with dedicated quantitative metrics, and we compare all generated puzzles against the original bench- mark. In the following, these four transformation types to demonstrate how each contributes to increased puzzle difficulty.
50-
Original HardoorsLogic
40-
Search Space(log
Search
present quantitative analyses of
10
Bimario
Sudeke
Crypts
Hiori Mineswrper
Puzzle
Search space expansion (IC1) This dimension captures the growth of candidate assignments induced by empty cells. Closed-form formulas are derived for each puzzle family (See Appendix B.4). For instance, in Binario, empty cells result in a search space of |S| = 2 Figure shows the average log-scale search space across five puzzle families, confirming that HardcoreLLogic systematically enlarges the combinatorial space in all five games.
Figure 2: Average search space size (in
log scale) across five puzzle families. See Appendix B.4 for detailed results.
Constraint strengthening (IC2) This transformation increases puzzle hardness by introducing denser logical entanglement.
4

---

Published as a conference paper at ICLR 2026
For CSP-based puzzles (e.g., Zebralogie, Binario), we encode instances into 7 and collect: (i) Decisions:
Explicit branching steps made by the solver; (ii) Conflicts: Backtracking events where partial assignments lead to contradictions. Larger counts indicate more complex search spaces and stronger constraint interactions, reflecting higher difficulty.
For graph-based puzzles (Navigation), we apply Dijkstra's algorithm and record: (i) Generated Nodes: the number of candidate states created; (ii) Expanded Nodes: the number of states fully explored. Increases in both values reflect higher search effort.
As shown in Figure 3 (light green background), HardcoreLogic instances consistently show higher complexity than originals.
Zahra
Brads
Kakem
Slaurager
Nevigticn
1590
340
Chittel
1990
200
1303
2003
31
100
353
3330
4000
TF Sedrkb Add diigenal bods
Scaicka
Kakeran
tixes
tiiler
Ceatlee
2.0
Origien Hantsant.eg ago
Driginal Hetteestg
Origenl Hentoott.gg
Oeiged 15 Eisrl.gg
500g
3864
Sadsla-405 Shose Mai
Sudika ans adonn
Syurapun dan
Xuripalion-dd mmrtr eatees
Codtes
100e
Coattess
200
280
160
Duciiies
Ducidoos
Declfccss
Expanded Noces
Decktoat
Figure 3: Quantitative comparison of transformation-induee complexity. Panels with light green, light orange, and light purple backgrounds correspond to IC2, UE1, and UE2, respectively. In each panel, dashed lines indicate the mean value of the corresponding metric.
Form variation (UE1) Form variation introduces novel symbols or forms that preserve puzzle validity but complicate comprehension. Since symbolic solvers cannot capture this representational difficulty, we measure it using perplexity, the inverse probability assigned by a pretrained LRM,, which quantifies how surprising an instance appears. Higher perplexity values indicate that mutated forms impose greater representational complexity for LRMs. Figure 3 (light orange background) presents boxplots comparing the perplexity distributions of Original and HardcoreLogic instances; form variation consistently results in higher perplexity and thus greater representational difficulty.
Rule variation (UE2) Rule variation modifies or extends the logical rules governing puzzles, forcing solvers to adapt to new structural constraints.
For CSP-based puzzles, we again use Z3 to measure decisions and conflicts.
* For graph-based puzzles, we evaluate expanded and generated nodes with Dijkstra's 's algorithm.
As shown in Figure 3 (light purple background), mutated-rule puzzles consistently yield higher solver statistics, indicating rule changes intensify reasoning complexity. A notable exception is the minesweeper dataset with "landmine clusters": numerical clues now represent adjacent landmine clusters, and more clues are added to ensure unique colutions-this reduces the search space, making required decisions lower than Original. Yet large models show lower accuracy on this modified dataset: unlike counting individual mines, models must continuously track landmine cluster connectivity (e.g.. judging cluster membership) for reasoning. This exceeds their simple pattern-matching capabilities, causing performance drops even for powerful models.
"73 refers to the Satisfiability Modulo Theories (SMT) solver developed by Microsoft Research. de Moura
Bjomer 2008
5

---

Published as a conference paper at ICLR 2026
3 EXPERIMENT AND RESULTS
3.1 EXPERIMENT SETTINGS
Benchmark models We evaluate HardcoreLLogic multiple open-source and closed-source LRMs, a full list available in Appendix [.I]] All models except Kimi-K2-Instruc are native LRMs, that is, they support generating separated reasoning part (usually surrounded by special tokens) before generating the final output. For hybrid reasoning models that can also generate non-reasoning responses (e.g., Qwen3 and DeepSeek-v3.. I). we always enable reasoning. For Kimi-K2-Instrucc we guide the model to perform a chain-of teain-oof-hhugeee of-thought (CoT) reasoning. Appendix C.1 also provides details of various model configurations.
Generation configuration On open-source models, we limit the reasoning budget to 32, 768 tokens before generating the final answer, regardless of their actual context window limitation. More specifically, we first input the prompt to the model to generate the reasoning part. If the model finishes reasoning within the budget, we then guide the model to generate the final answer that strictly follows the predefined JSON schema eliminate presentation errors. A generation run is considered correct if and only if the model successfully finishes reasoning and produces a correct answer. We repeat 4 runs each sample with decoding temperature 7 0.6. Closed-source models do not support hard reasoning budget limits, hence we simply limit their total output budget to 32, 768 tokens. Furthermore, we sample 600 cases across all games (5 per transformation type per game) due to expenditure constraints, while remaining repeating 4 runs each extracted sample. The prompt templates, including corresponding JSON schema for each game, are listed in Appendix C.2
3.2
MAIN RESULTS
Overall results Figure 4 iilustratte the overall models performance on HardcoreLogic, compared with Original. Kimi-K2-Instren showed the greatest decrease in accuracy compared to Original on HardcoreLogic. Among open-source models, gpt-oss- 120b exhibited the highest accuracy on both datasets, while GPT-5 performed the best the source models. Minimax-Mi performs the worst among all models.
80
Original Mean
HardcoreLogic
70
Original
60
HardcoreLogic
Percenlage (%)
50
40
30
20
10
gpt-oss-120b
gwen3-235b
gpt-5
deepseek-v3.1
qwen3-30 30b-a3b
got-5-mini
qwen3-nexx
ad-minl
seed-oss
deepsookrt
gemini-2.54 2-prr
grok4
grok- 3-mini
claude sonne--onne--- -
kimi-k2
gim-4.5
qwan3-32
gemini-2.5. nomini-2.5-4aaal s-llash
qwen3-8b
loapsoek-qver qwon
minimax-mri
Figure 4: Overall models performance on Original and HardcoreLogic. Dashed lines represent the average values each model on the corresponding dataset.
Per-game results Figure 5 shows the comparison of the accuracy of each puzzle on both Original and the HardcoreLogic across all open-source models The overall performance of all puzzles and
Due to the limited number of subtasks in some puzzles and the small sample size for testing such puzzles on cosedd-ouuree models, all analysis of per game mainly focuses on open-source models.

---

Published as a conference paper at ICLR 2026
models shows continuous downward trend. For open source models, Binario has the largest average performance degradation on the HardcoreLogic and Original. Skyscraper has the smallest decrease, followed by Navigation. These two puzzles are extremely difficult and extremely simple, which is why they have the smallest decrease.
Binario
Sudoku
Hitori
Zebralogic
Minesw ceper
Accoracy (%)
Original Larg
Original HardcoreLagle Kakurasu
Original Hadge Hanoi
Original HardcoreLagle Navigation
Original HardcoreLLogk Skyscraper
Cryplo
100
Accurne
50
Original Hardoc
Original Hardeorel.ogte
Original HardeoreLagie
Original HardcoreLLogi
Original Hardsorel.ogk
Original
HardooreLoglc
Original
HardeoreLLoglc mean
Figure 5: Performance of each puzzle on open-source models.
ANALYSIS AND DISCUSSION
4.1 DIFFERENT LONG- LONG-TAIL TRANSFORMATION
In Section 2 we introduce four methods of long-tail trans- formation, including Search Space Expansion (IC1), Con- straint Strengthening (IC2), Form Variation (UEI), and Rule Variation (UE2). Puzzles may also have two different long-tail transformation attributes the same time. To quantify the impact of different long-tail transformations on puzzle difficulty, we fit weighted multiple linear regression for each puzzle. The dependent variable rep- resents the accuracy of the puzzle after undergoing four different long-tail transformations (IC1, IC2, UE1, UE2). We weight the number of samples in the data, and for mixed transformations, the predictive performance of the model is the sum its coefficients; the intercept represents the model-predicted baseline accuracy when long-tail transformation is applied.
Binario -74.09
-2.45
0.00
0.00
Crypto -26.58
0.00
0.00
-10
Hanol-
0.00
0.00
-22.51
0.00
Hitori
0.00
-4.39
Kakurasu
-25.90
-11.96
-7.55
0.00
Mineswepper
-0.09
Navigation
0.00
-10.00
0.00
-5.54
-50
Skyscraper
0.00
Sudola
-39.86
0.00
-23.11
-13.95
Zebralogic
I0:2
UKI
Basic Long tail Transformation
Figure 6: Effects of long-tail transforma- tions on puzzle accuracy.
Figure 6 shows the coefficients of four long-tail transformations in the regression model, which trained on data from spen-source models and reflects their impact on puzzle difficulty. We can observe that IC1 has the greatest comprehensive impact the models, space directly requires the improvement of the models' memory and reasoning ability. UE1 requires the models to recognize its highest value for Sudoku puzzles, mainly due to the need to recognize irregular nine-grid patterns, indicating that the models struggles in this scenario. The parameters of the minesweeper puzzle in UE2 also show that the "landmine cluster" rule has significant impact on the models, which is consistent with our hypothesis Section 2
the increase in search
uncommon elements. It is worth noting that the parameter UE1 reaches
4.2 ERROR ANALYSIS
To probe the underlying causes of LRM failures HardcoreLogic, we conduct systematic er- ror analysis. Based on the comparison between the puzzle, the correct answer, and the model's complete responses, identify six emror categories: (1) Misunderstanding of the Logic Puzzle, (2) Misapplied Solution Framework, (3) Brute-Force with Excessive Complexity, (4) Factual Errors, (5) Over Verfication, and (6) Infinite Repetition. This enables us move beyond aggregate
7

---

Published as a conference paper at ICLR 2026
accuracy in how different models fail. We randomly sample 50 erroneous cases from each of four representative models: gpt-oss- 120b, the best performing closed-source model on HardcoreLogic; Qwen3-235B, representative of the Qwen series that we extensively evaluated; Kimi K2 Instruct, which experienced the largest performance drop from Original to Hardcorel Logi:; and Minimax-M1, the worst-performing model on HardcoreLLggic We employ GPT-5 [OpenAT] 2025b) as a secondary annotator to classify each case into one of the six categories. Detailed explanation of each category was shown in Appendix
We employ GPT-5 (OpenAl 2025b), Gemini-2.5- (ee--ProoGemmil Team 2025), and Claude-Sonnet- 4.5 (Anthropic 2025) as secondary annotators classify each case into one of the six categories. The final label is determined through a majority-vote scheme. In situations where the three models produce three distinct labels (i.e., no majority), we conduct manual verification. detailed consistency analysis of this voting-and seeiingand-ddjiine adjjdiicatiion scheme is provided in the Appendix D.3
Figure 7 shows the error distribution for each model. Overall,
Misunderstanding and Misapplied errors are particularly prominent in Kimi-K3 12-Instruct, ac- counting for roughly 50% of its errors. Notably, Kimi-K22Instruet also exhibits the largest performance drop from Original to Hardcorel ciated with its frequent misunderstanding of puzzles and misapplication solution frameworks. This indicates that the model struggles to correctly interpret problem structures and select appro- priate reasoning strategies in more challenging or structurally novel logic problems. Optimization could involve enhancing problem understanding through structured prompts and step-by-step reasoning training, as well as guiding the model to identify problem types and adopt suitable solution frameworks, combined with symbolic or constraint-based verification.
suggesting that this decline is closely asso-
Factual errors are the most prevalent, suggesting that during extended reasoning, LRMs often fabricate facts fill missing steps, compromising truthfulness and consistency. Mitigation may involve stronger penalties for factual deviations during fine-tuning or reinforcement learning, and mechanisms for intermediate reasoning verification. Meanwhile, during manual review, was found that the Kimi-K' Limi-K2-Innsiin Instrucus'''s s significant skipping of steps during reasoning makes it more prone introducing information that is not present in the problem or cannot be directly obtained during the reasoning process, thus making factual type errors more pronounced.
Brute-Force errors in stronger models (gpt-oss-1200 Qwen3-235B) indicate that their generative power can lead inefficient, enumerative strategies. Performance could be improved by training models to identify problem types and adopt optimal solution frameworks, or by integrating LRM reasoning with symbolic/constraint solvers to guide search.
16.0%
12.0%
38.0%
10.0%
2.0%
34.0%
2.0%
58.0%
20%
Maundentanding
Error Type
Mhopplled
8.0%
22.0%
2.0%
Brute-Force
12.0%
32.0%
Factual- Errors
Vver-verfiation
28.0%
14.0%
4.0%
24.0%
Infinite Repetition
2pt-os--120b
qwen3-235b
kimi-A2
minimax-m1
Figure 7: The distribution of error types in HardcoreLogic across the four models.
To compare how model error patterns shift between Original and HardcoreLogic. we also uniformly sample 50 erroncous Original cases and classify them using the identical model-voting and human- verification pipeline. This provides a matched error-type distribution for Baseline errors, enabling a direct comparison against HardcoreLogic results. Figure 8 shows the percentage distribution of six enror categories across the two benchmarks.
Rule Perturbation Raises Understanding-RRlltee Failures HardcoreLogie introduces greater rule diversity, non-canonical puzzle structures, and more complex constraint dependencies. These perturbations substantially weaken models' robustness in understanding and applying task rules. As a result, both Misunderstanding and Misapplied errors increase markedly across models. Increased Complexity Reduces
Reasoning Over- Verification errors decline under HardcoreLogic, indicating that models are less able to generate coherent but incorrect explanations when faced with more complex logical dependencies. Instead of producing confident
8

---

Published as a conference paper at ICLR 2026
and polished but unfaithful rationales, models tend to break earlier in the reasoning process. yielding errors that stem from misunderstanding or rule misapplication.
gpt-oss-120b
qwen3-235h
Redienlagy
Peruantngg (90)
Pereninge
Miaepeild
Brots
Parmal grrep
Owe- Pwe-Hofkatto
Isbeln Rypeituve
Masaal retteen
Atteppiie
Farmal-Rran
Ove Hotfutte
Istadk Ropelttre
Breer
kimi-k2
minimax-mi
In Lagk
Baibenlage
Pencrtngg (k)
Pepcertage (2)
16
Mauppha
Ielabio Ryprttie
M retenaaoo
Misgpii
Aran-For
Factosl
Ovn Varkerrin
Figure 8: Comparison of error-type distributions between Original and HardcoreLogie across four large language models.
4.3 UNSOLVABLE GAMES
Overall results To explore the model's ability to han- dle contradictory puzzles, we constructed a batch of un-
solvable puzzles based each puzzle. Figure ? shows the performance of each model in a puzzle-free scenario, where the overall performance of closed-source models is better than that of open-source models. To investigate this phenomenon, we take several open-source models examples to analyze the possible problems that models may encounter when encountering unsolvable puzzles.
Accuracy (%)
Models
Sufficiency analysis To investigate how LRMs han- dle unsolvable logic puzzles, we analyzed cases where LRMs correctly labeled puzzles unsolvable deter- mine whether the judgment was a genuine understanding (Justified Unsolvability) a heuristic claim due to failure to solve (Unjustified Unsolvability)
Figure Overall model performance on unsolvable puzzles.
We sampled 50 responses from four models and classified each accordingly, revealing differences in their reasoning behavior, shown in Figure TO Stronger models (gpt-oss-120) and Qwen3-235B) typically provide justified explanations, while weaker models (Minimax-M1) more often output unjustified "unsolvable" claims. This suggests that the ability to maintain deeper and more consistent reasoning chains is crucial for producing sufficient unsolvability explanations.
Error analysis We further analyzed the LRMs' incorrect responses unsolvable logic puzzles, categorizing the eπrоrs into four types: (1) Erroneous Reasoning. (2) Mandatory Response. (3) Unable to Deduce, and (4) Infinite Repetition. Detailed explanation of each category given in Appendix providing a fine-grained view of how LRMs fail on unsolvable puzzles. As shown in Figure I0 error distributions vary significantly across models. Stronger models (gpt-oss- 120b and Qwen3- 235B) mainly fail through Erroneous Reasoning or being Unable to Deduce, indicating limitations in sustaining reasoning depth. By contrast, weaker models (Kimi-K3 K2-iiK2---- Instruct and Minimax-MI exhibit higher rates Mandatory Responses and especially Infinite Repetition, reflecting brittle control over output structure. These results suggest that future model updates should not only enhance logical
C.3 Each incorrect response from the four models was classified accordingly.

---

Published as a conference paper at ICLR 2026
consistency and depth of reasoning but also incorporate stronger mechanisms to prevent degenerate behaviors such as repetitive loops or forced answers.
UP Sufficient Analysis
UP Error Analysis
Type
1yp
Jestfled Lmnavability
0.8
Tal uotted Ulaoivaiitt
Mandoney
Unabée delues
Percentage
Poreentags
teflels Hapettinn
11.4
0.4
0.2
0.0
221006-1206 qweni-215h
kimi-22
minimas-mi
0.0
gpt-oss-1200
quen3-215p
kimi-k2
minima 1-01
Figure 10: Analysis LRMs' responses on unsolvable puzzles: the left panel shows correct responses and the right panel shows incorrect responses.
RELATED WORK
Reasoning benchmarks Logic puzzles aim to test the logical reasoning ability of a model. Re- searchers have proposed different benchmarks test the reasoning ability of models in puzzles. including deductive reasoning (Wang et al.) (2022). inductive reasoning, causal reasoning (Yang et al. 2024), and mixed reasoning (Luo et al.) 2024). The datasets used include synthetic datasets (Chen investigators proposed different benchmarks to test and evaluate data sets. For example, Logiegame (Gui et al., 2025) grades the difficulty of tasks by evaluating the number of reasoning steps and achieves dual evaluation of the process and the results. Multi-LogiEvall (Patel et al., 2024) systematically evaluated the impact of inference depth on LRM. However, there still deficiencies in data diversity, with limited difficulty limits for puzzles and a lack of high-difficulty reasoning tasks. We introduced various puzzles, increased the difficulty limit of logical puzzles, performed long-tail transformation puzzles from multiple aspects, and evaluated the impact of these changes on model performance.
et al. 2025a) and collected
Long-tail benchmarks Several studies have shown that large language models often excel at memorization but struggle generalize tasks requiring systematic reasoning or complex com- binatorial problem-solving For example, Anil et al. (022) and Wold et al. (2024)iighlight that Transformers can fail to generalize longer sequences or novel compositional structures. These findings suggest that LRMs' apparent reasoning ability may rely heavily on pattern recognition from training data rather than true algorithmic generalization. several benchmarks have been proposed that target "long-tail" or challenging reasoning instances. For example, JustLogic (Chen et al. 2025b), LINT (Li et al., 2024). and SATbench (Wei et al. 2025) enrich traditional tasks with harder problem instances, extended reasoning chains, or compositional variations, revealing LRMs' difficulty in tackling distribbutinn or rare configurations. Further- more, |Wan et al. (2025) dynamically generate adversarial questions against LRMs. Building upon these insights, we introduce a new benchmark suite that systematically generates a wide range of logic puzzles under diverse long-tail transformations. Our dataset provides richer structural variations and increased reasoning complexity, allowing more comprehensive evaluation of LRMs' generalization and problem problem-solving solving capacity beyond what prior benchmarks offer.
systematically evaluate these limitations,
CONCLUSION
In this paper, we introduce HardcoreLogic, a challenging logic puzzle benchmark comprising over 5,000 puzzles spanning 10 different puzzle games. Our experiments show that LRMs exhibit substantial performance drop HardcoreLogic compared to the Original datasets. This highlights that current models still struggle in less conventional, long-tail scenarios and often rely on pattern recognition memorized experience rather than genuine reasoning. At the same time, HardcoreLogic provides a valuable benchmark for future research, offering a platform to systematically evaluate and improve the reasoning capabilities of LRMs in diverse and challenging logical contexts.
10

---

Published as a conference paper at ICLR 2026
Acknowledgments The work is supported by Al for Science Program, Shanghai Municipal Com- mission of Economy and Informatization (2025-GZL-RGZN- BEAGGZN-BBBX-02220 BTBX-02028). The project's 's computa- resources partially supported by CFFF platform of Fudan University.
Ethics statement Original contains samples from existing published datasets including Enigmata and ZebraLogic, of which we strictly follow the corresponding licenses in data use. Meanwhile HardcoreLogic, we only cover the same logic games but have all puzzles generated independently; we guarantee the transparency and reproducibility of the generation of HardcoreLogic.
Reproduction statement We publish both Original and HardcoreLogic the public for reproduc- tion and future research. We also publish the data generation and evaluation code for reproduction of our datasets and evaluation results. We make our best effort to ensure deterministic outcomes, and guarantee so open-source LRMs; however, due to the black-box, stochastic nature of closed-source LRMs, we cannot guarantee any precise reproduction on these closed-source models.
REFERENCES
Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz. Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari. Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems, volume 35, pp. 38546 38556 Curran Associates, Inc., 2022. URL /ttp: meureedddiinnnsneiiieeee wttpeeerrrss.ccppppppppeeeeee
Anthropic. Introducing claude 4 (sonnet 4). httpp://ww..athrhppccc.ccoo////s 2025.
ByteDance Seed Team. Seed-oss open-source models.
oss 2025.
Jiangjie Chen, Qianyu He, Siyu Yuan, Aili Chen, Zhicheng Cai, Weinan Dai, Hongli Yu, Jiaze Chen, Xuefeng Li, Qiying Yu, Hao Zhou, and Mingxuan Wang. Enigmata: Scaling logical reasoning in large language models with synthetic verifiable puzzles. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a. URL htttps://operreviee. net/forumiid fmmxunacr4
Michael K. Chen, Xikun Zhang, and Dacheng Tao. Justlogie: A comprehensive benchmark for evaluating deductive reasoning in large language models. arXiv preprint, 2025b. doi: 10.48550/A RXIV 2501. 14851. URL
abwgg///////...../ 14851
Francois Chollet, Mike Knoop, Gregory Kamradt, Bryan Landers, and Henry Pinkard. Arc-agi-2: A new challenge for frontier reasoning systems. arXiv preprint, May 2025. doi: 10.48550/ARXIV 21805.1181..
Leonardo de Moura and Nikolaj Bjørner. Z3: An efficient smt solver. In C. R. Ramakrishnan and Jakob Rehof (eds.), Tools and Algorithms for the Construction and Analysis of Systems, pp. 337-340, Berlin, Heidelberg. 2008. Springer Berlin Heidelberg. ISBN 178-3-5401 003-3--40-77800-3.
DeepSeek-Al, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu. Haocheng Wang. Haowei Zhang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J. L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang, Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Mingming Li, Ning Tian, Panpan Huang, Peiyi Wang, Peng Zhang, Qiancheng Wang, Qihao Zhu, Qinyu Chen, Qiushi Du, R. J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang, Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao
11

---

Published as a conference paper at ICLR 2026
Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang. Wenjun Gao, Wenqin Yu, Wentao Zhang. X. Q. Li, Xiangyue Jin, Xianzu Wang. Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song. Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang. Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li, Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang. Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang. Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu. Zhongyu Zhang. Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report. arXiv preprint, December 2024.
doi: 10.48550/ARXI..2212..
AI, Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang. Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen. Dongjie Ji. Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang. Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen. Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai. Jiaqi Ni. Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang, Mingming Li, Ning Tian, panpan Huang, Peng Zhang. Qiancheng Wang, Qinyu Chen, Qiushi Du Ruiqi Ge. Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye. Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye. Tao Yun, Tian Pei, Tianyu Sun, Wang. Wangding Zeng, Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun Gao, Wengin Yu, Wentao Zhang, W. L. Xiao, Wei An, Xiaodong Liu, Xiaohan Wang, Xiaokang Chen, Xiaotao Nie, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li, Xiangyue Jin, Xiaojin Shen, Xiaosha Chen, Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li. Yao Zhao, Yaofeng Sun, Yaohui Wang, Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong. Yuheng Zou, Yujia He, Yunfan Xiong, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhicheng Ma. Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in iims via reinforcement learning. arXiv preprint, January 2025. doi: 10.48550/ARRIV 22w11.51111.. 12948.
Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next gemeration agentic capabilities. June 2025. URL httpttttts///o/agttttt/s/ googleapis. com/d eepminndmeeiiaa/emiiii/ wttp://iieiiiiii.____eeeeee wtt___________________ pdd
GLM-4.5 Team, Aohan Zeng. Xin Lv, Qinkai Zheng, Zhenyu Hou, Bin Chen, Chengxing Xie, Cunxiang Wang, Da Yin, Hao Zeng, Jiajie Zhang. Kedong Wang, Lucen Zhong, Mingdao Liu, Rui Lu, Shulin Cao, Xiaohan Zhang, Xuancheng Huang, Yao Wei, Yean Cheng, Yifan An, Yilin Niu, Yuanhao Wen, Yushi Bai, Zhengxiao Du, Zihan Wang. Zilin Zhu, Bohan Zhang, Bosi Wen, Bowen Wu, Bowen Xu, Can Huang, Casey Zhao, Changpeng Cai, Chao Yu, Chen Li, Chendi Ge. Chenghua Huang, Chenhui Zhang, Chenxi Xu, Chenzheng Zhu, Chuang Li, Congfeng Yin, Daoyan Lin, Dayong Yang, Dazhi Jiang, Ding Ai, Erle Zhu, Fei Wang. Gengzheng Pan, Guo Wang. Hailong Sun, Haitao Li, Haiyang Li, Haiyi Hu, Hanyu Zhang, Hao Peng, Hao Tai, Haoke Zhang, Haoran Wang, Haoyu Yang, He Liu, He Zhao, Hongwei Liu, Hongxi Yan, Huan Liu, Huilong Chen, Ji Li, Jiajing Zhao, Jiamin Ren, Jian Jiao, Jiani Zhao, Jianyang Yan, Jiaqi Wang, Jiayi Gui,
12

---

Published as a conference paper at ICLR 2026
Jiayue Zhao, Jie Liu, Jijie Li, Jing Li, Jing Lu, Jingsen Wang, Jingwei Yuan, Jingxuan Li, Jingzhao Du, Jinhua Du, Jinxin Liu, Junkai Zhi, Junli Gao, Ke Wang, Lekang Yang, Liang Xu, Lin Fan, Lindong Wu, Lintao Ding, Lu Wang, Man Zhang, Minghao Li, Minghuan Xu, Mingming Zhao, Mingshu Zhai, Pengfan Du, Qian Dong, Shangde Lei, Shangqing Tu, Shangtong Yang. Shaoyou Lu, Shijie Li, Shuang Li, Shuang-Li, Shuxun Yang. Sibo Yi, Tianshu Yu, Wei Tian, Weihan Wang, Wenbo Yu, Weng Lam Tam, Wenjie Liang, Wentao Liu, Xiao Wang, Xiaoban Jia, Xiaotao Gu, Xiaoying Ling, Xin Wang, Xing Fan, Xingru Pan, Xinyuan Zhang, Xinze Zhang, Xiuqing Fu, Xunkai Zhang, Yabo Xu, Yandong Wu, Yida Lu, Yidong Wang, Yilin Zhou, Yiming Pan, Ying Zhang, Yingli Wang, Yingru Li, Yinpei Su, Yipeng Geng, Yitong Zhu, Yongkun Yang, Yuhang Li, Yuhao Wu, Yujiang Li, Yunan Liu, Yunqing Wang, Yuntao Li, Yuxuan Zhang. Zezhen Liu, Zhen Yang. Zhengda Zhou, Zhongpei Qiao, Zhuoer Feng. Zhuorui Liu, Zichen Zhang. Zihan Wang, Zijun Yao, Zikang Wang, Ziqiang Liu, Ziwei Chai, Zixuan Li, Zuodong Zhao, Wenguang Chen, Jidong Zhai, Bin Xu, Minlic Huang, Hongning Wang, Juanzi Li, Yuxiao Dong, and Jie Tang. Glm-4.5: Agentic, reasoning, and coding (arc) foundation models. arXiv preprint, August 2025. doi: 10.48550/ARXIV 06671..
Jiayi Gui, Yiming Liu, Jiale Cheng, Xiaotao Gu, Xiao Liu, Hongning Wang. Yuxiao Dong, Jie Tang. and Minlie Huang. LogieGame: Benchmarking sule-based reasoning abilities of large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACI. 2025, pp. 1491, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN findings-a doi: 18.53vv3/v 1/2025. findings-acl.77. URL hhtp:: httpps:/aacannnhoooo...rrrr 202p c1.777
Nurit Cohen Inger, Yehonatan Elisha, Bracha Shapira, Lior Rokach, and Seffi Cohen. Forget what you know about LLMs evaluations- LLMS are like a chameleon. In Christos Christodoollppolls, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.). Proceedings the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 2 21664-21677, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 7-77799-81771911 176-332-6. doi: 10. 18653/v11200 25. memee--nmiin..... 1098. https: / http///ccaaattoooocoooo wwwpoooooooo.oog////// ottp://// emnlp Ip main. 1098/
Chunyang Li. Weiqi Wang, Tianshi Zheng, and Yangqiu Song. Patterns over principles: The fragility of inductive reasoning in LLMs under noisy observations. In Wanxiang Che. Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational I Linguisties: ACL. 2025, pp. 19608-19626 Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979- 891 76-256-5 doi: 10. N.L18863/1/2225 http:///2222...inns-neeee 006. URL
org/ w findings- acl. 1006/
Huihan Li, Yuting Ning, Zeyi Liao, Siyuan Wang, Xiang Lorraine Li, Ximing Lu, Wenting Zhao, Faeze Brahman, Yejin Choi, and Xiang Ren. In search of the long-tail: Systematic generation of long-tai tail inferential knowledge via logical rule guided search. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 2348-2222 2370, Miami, Florida, USA, November 2024. Association for Computational L Linguistics. doi: 10. 11166///1/222242mmmmmmmmm gy.org/ 2024. emnlp-main 140/
URL httt::://c/ccatteet
Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi. Zebral On the scaling limits of L.MSs for logical reasoning. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj. Kiri Wagstaff, and Jerry Zhu (eds.), Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 37889-37905. PMLR, httppiiis..n...erres//////////// prwss//267/lin2222 htmml
July 2025a. URL https: http://proceein....... mlr
Haowei Lin, Xiangyu Wang, Ruilin Yan, Baizhou Huang, Haotian Ye, Jianhua Zhu, Zihao Wang, James Zou, Jianzhu Ma, and Yitao Liang. Generative evaluation of complex reasoning in large language models. arXiv preprint, 2025b. doi: 10.48550/ARXIV, 22w.0..20000001100 URL |htpps
Man Luo, Shrinidhi Kumbhar, Ming Shen, Mihir Parmar, Neeraj Varshney, Pratyay Banerjee, Somak Aditya, and Chitta Baral. Towards logiglue: A brief survey and benchmark for analyzing logical reasoning capabilities of language models. arXiv preprint, 2024. doi: 00048500/rXii.231000003
URL
08836
13

---

Published as a conference paper at ICLR 2026
MiniMax, Aili Chen, Aonian Li. Bangwei Gong, Binyang Jiang, Bo Fei, Bo Yang. Boji Shan, Changqing Yu, Chao Wang, Cheng Zhu, Chengjun Xiao, Chengyu Du, Chi Zhang. Chu Qiao, Chunhao Zhang, Chunhui Du, Congchao Guo, Da Chen, Deming Ding, Dianjun Sun, Dong Li, Enwei Jiao, Haigang Zhou, Haimo Zhang, Han Ding, Haohai Sun, Haoyu Feng. Huaiguang Cai, Haichao Zhu, Jian Sun, Jiaqi Zhuang. Jiaren Cai, Jiayuan Song. Jin Zhu, Jingyang Li, Jinhao Tian, Jinli Liu, Junhao Xu, Junjie Yan, Junteng Liu, Junxian He, Kaiyi Feng, Ke Yang, Kecheng Xiao, Le Han, Leyang Wang, Lianfei Yu, Liheng Feng, Lin Li, Lin Zheng, Linge Du, Lingyu Yang, Lunbin Zeng, Minghui Yu, Mingliang Tao, Mingyuan Chi, Mozhi Zhang, Mujie Lin. Nan Hu, Nongyu Di, Peng Gao. Pengfei Li, Pengyu Zhao, Qibing Ren, Qidi Xu, Qile Li, Qin Wang, Rong Tian, Ruitao Leng, Shaoxiang Chen, Shaoyu Chen, Shengmin Shi, Shitong Weng. Shuchang Guan, Shuqi Yu, Sichen Li, Songquan Zhu, Tengfei Li, Tianchi Cai, Tianrun Liang, Weiyu Cheng, Weize Kong, Wenkai Li, Xiancai Chen, Xiangjun Song, Xiao Luo, Xiao Su, Xiaobo Li, Xiaodong Han, Xinzhu Hou, Xuan Lu, Xun Zou, Xuyang Shen, Yan Gong, Yan Ma, Yang Wang, Yiqi Shi, Yiran Zhong, Yonghong Duan, Yongxiang Fu, Yongyi Hu, Yu Gao, Yuanxiang Fan, Yufeng Yang, Yuhao Li, Yulin Hu, Yunan Huang, Yunji Li, Yunzhi Xu, Yuxin Mao, Yuxuan Shi, Yuze Wenren, Zehan Li, Zelin Li, Zhanxu Tian, Zhengmao Zhu, Zhenhua Fan, Zhenzhen Wu, Zhichao Xu, Zhihang Yu, Zhiheng Lyu, Zhuo Jiang, Zibo Gao, Zijia Wu, Zijian Song, and Zijun Sun. Minimax-ml: Scaling test-time compute efficiently with lightning attention. arXiv preprint, June 2025. doi: 10,48550/ARXIV 2506612111
Moonshot AL. Kimi- 1imi-K2-IIntruee 2025.
OpenAL. gpt-oss-120b & gpt-oss-20b model card, 2025a. URL/ 0925
OpenAl. Introducing gpt-5.
OpenAl. Introducing gpt-4.1 I in the April 2025c. URL http://sss//pppaaa..ccoo/nne///s
OpenAI. Introducing openai o3 and o4-mini, April 2025d, URL http:tpss//popaa..cc/////ee/////
roducing- 03-- o3-and-o4-mini//
Nisarg Patel, Mohith Kulkarni, Mihir Parmar, Aashna Budhiraja, Mutsumi Nakamura, Neeraj Varshney, and Chitta Baral. Lulti-LogiFyval: 'Toards evaluating multi-step logical reasoning ability of large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 20856 50856-20879, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.com hwtp2222..mmmmmmmaa.. main. 1160. URL https::///tcaaaaee https://acllanthology org/ www...mnllp........mmmmmmmmmmll. emnlp-manlp mai n. 1160/
Siyuan Wang, Zhongkun Liu, Wanjun Zhong, Ming Zhou, Zhongyu Wei, Zhumin Chen, and Nan Duan. From Isat: The progress and challenges of complex reasoning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:2201-2216, 2022. ISSN 2329-9304. doi: 10.1 A..10//AASLPP.02.31622222
Siyuan Wang, Zhuohan Long. Zhihao Fan, Xuanjing Huang, and Zhongyu Wei. Benchmark self- evolving: A multi-agent framework for dynamic LLM evaluation. In Owen Rambow, Leo Wanner, Marianna Apidianaki, Hend Al-Khalifa, Barbara Di Eugenio, and Steven Schockaert (eds.), Proceedings of the 31 International Conference on Computational Linguistics. pp. 3310- 3328, Abu Dhabi, UAE, January 2025. Association for Computational Linguistics. URL https: //aclanthology.ooo org/ www..//....../..////.///.. 20250 col ing 1og--main.2 main.223/
Anjiang Wei, Yuheng Wu, Yingjia Wan, Tarun Suresh, Huanmi Tan, Zhanke Zhou, Sanmi Koyejo, Ke Wang, and Alex Aiken. SATBench: Benchmarking LLMs' logical reasoning via automated puzzle generation from SAT formulas. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 33832-33849 Suzhou, China, November 2025. Association for
Computational Linguistics. ISBN 979-8- 879-8-89111 191--99377666666
doi: 10. 10.11853/w1/2023 1/2032/ wwwp:/222.eem..mmme.ee http:/5.mmmmmmmi..1177 main. hwwp:/looyoorgggge/g org 2025. www.::2......mmaammmmmmmm emnlp-main. 1716/
URL
Sondre Wold, Étienne Simon, Lucas Charpentier, Egor Kostylev, Erik Velldal, and Lilja Øvrelid. Compositional generalization with grounded language models. In Jun-Wei Ku, Andre Martins,
14

---

Published as a conference paper at ICLR 2026
and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, PP. 3447-3460, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: ac1.205. URL httpps///aaclaantoooooooor////// Findiiigaa...
057
XAI. Grok beta - the age of reasoning agents, February 2025a. URL https/t/..a.a/eess////////
xAI. Grok 4. https://x.aa/nnww////.
2025b.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang. Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su. Yichang Zhang, Yinger Zhang. Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report. arXiv preprint, May 2025. doi: 10.48550/ARXIV Wtt..2555509.88.
Zonglin Yang, Xinya Du, Rui Mao, Jinjie Ni, and Erik Cambria. Logical reasoning over natural language as knowledge representation: survey, arXiv preprint, 2024. doi: 10.48550/ARXIV .20 3.12023. URL HtLLhttps::// / wwwperiiiv.ogog/boaa www.org///bbaa.ee 2303. 12023
A USE OF LLMs
We use LLMs at several parts of our work: (1) To construct natural-language versions of Zebralogic, used GPT-4o-mini (OpenAl) 2025c) for clue translation from formal mathematical expressions, with GPT-o4-mini (OpenAl 2025c) verifying semantic consistency between the mathematical and translated forms. (2) For response annotation, GPT-5 (OpenAl 2025b), Semmini-25--rroGemmmn
Team 2025). and C.aude-Sonnet-4.5 (Anthropic 2025 as secondary annotators to categorize model responses. (3) During paper preparation, we leveraged GPT-5 refine our writing by compressing redundant descriptions and improving narrative conciseness. (4) LLMs were also employed to assist literature search, helping us identify and locate relevant prior work.
B BENCHMARK DETAILS
B.1
CATEGORY DEFINITION
In Section 2 mention that the HardcoreLogic contains puzzles for 6 different categories. Table provides an introduction to their specific definitions.
B.2 PUZZLE DEFINITION AND TRANSFORMATIONS
To provide a clearer view of how we constructed the HardcoreLogic, we include additional details on each puzzle type. For every puzzle, we present the original rules alongside the applied transfor- mations. The original rules specify the standard constraints of the puzzle, while the transformations describe the modifications we introduced to increase reasoning difficulty or adapt the puzzles to our evaluation framework. [ublee]ssmmmrri reference for reproducibility and further analysis. Figure] provides examples of some long-tail transformations for each puzzle.
these rules and transformations, offering a comprehensive
15

---

Published as a conference paper at ICLR 2026
Table 2: Logic game categories in Hardcorel.ogic.
Category
Definition
Logic puzzle
This type of puzzle provides with multiple related logical clues, requiring us to integrate each clue. Zebralogic belong to this category.
Grid puzzle
This type of puzzle provides a grid of different sizes, where the cells may be blank cells cells with numbers. Need fill in the numbers in blank cells through puzzle rules. Sudoku, skyscraper, and Binario belong this category.
Search puzzle
This type puzzle requires searching for the required cells through puzzle rules. Minesweeper, Hitori, and Kakurasu be- long to this category.
Pattern puzzle
This type of puzzle will provide specific pattern and require to understand, extract, and apply that pattern. Crypto belong to this category.
Graph puzzle
This type of puzzle provides some graphic clues that need understand and model to answer questions. Navigation belongs to this category.
Sequential puzzle
This type of puzzle requires us to solve a multi-step puzzle in a specific order. Hanoi belongs this category.
Table 3: Rules and transformations of each game in HardcoreLogic.
Puzzle
Rule
Description
Sudoku
Original
(1) Puzzle categories: Grid puzzle
(2) Puzzle rules:
1. The Sudoku board is a 9 x grid, divided into smaller
3 3 subgrids, which includes known cells (numbers 1-9)
and unknown cells.
2. Row constraint: Each row must include all numbers from without duplication.
3. Column constraint: Each column must include all numbers
from I to without duplication.
4. Subgrid constraint: Each 3 subgrid must include all
numbers from to without duplication.
(3) Puzzle task: Fill numbers into unknown cells satisfy row, column, and subgrid constraints.
16

---

Published as a conference paper at ICLR 2026
Puzzle
Rule
Description
Transformation
(1) More empty cells and larger grid: Increase the number of unknown cells or 16 x 16 grid (subgrid 4 x 4).
(2) Irregular subgrid: Divide the 9 x grid into nine irreg-
ular regions; the three constraints remain unchanged.
(3) Additional constraints:
Each main and secondary diagonal must include all numbers from I to 9 without repetition.
Diagonal constraint:
AAdaaceny constraint: The difference between each cell and its orthogonal neighbors (up, down, left, right) cannot be 1. Maximize box constraint: Divide the
grid into nine 3 3 subgrids indexed 1-9. Each puzzle requires sub- grid to maximize its score (score = sum of cell index x cell value).
When generating puzzles with additional constraints, we en- sure that they have multiple solutions under the original con- straints and only one solution under these constraints. This requires more empty cells to ensure this situation. When comparing the difficulty of such puzzles with the Original dataset, we compare the difficulty changes between different constraints and the entire Original dataset. However, their search space did not show a significant improvement com- pared to the data of IC1 in HardcoreLogic. Therefore, we categorize it as UE2.
(4) Letter version: Replace numbers 1-9 with letters A-I, also applied to irregular, diagonal, and adjacency variants. Unsolvvablle puzzle:In Sudoku puzzle, blank cell en- ters "no valid number can be filled" state, meaning that the union of its row, column, and range already contains all numbers from 1 to 9.
Kakurasu
Original
(1) Puzzle categories: Search puzzle
(2) Puzzle rules:
1. The Kakurasu board is a 6 x 6 grid. Numbers at the top (columns) and on the left (rows) are constraints.
2. Row sum sum of column indices (1-based) of black cells in that row.
3. Column sum sum of row indices (1-based) of black cells
in that column.
(3) Puzzle task: Blacken cells in a 6 grid so that row/-
column sums equal the given constraints.
Transformation
(1) Add blocked cells: Some cells cannot be blackened; grid sizes include 6 x and x
(2) Hide partial clues: Some row/column constraints
hidden (denoted by -1); grid sizes include 6 x 6 and 7 x
(3) Unsolvable puzzle:First, we generate a solvable puzzle. Then, we modify some of the clues to make the puzzle un- solvable-for example, by swapping certain row or column clues and then verifying that the resulting clues indeed render the puzzle unsolvable.
17

---

Published as a conference paper at ICLR 2026
Puzzle
Rule
Description
Hitori
Original
(1) Puzzle categories:
Search puzzle
(2) Puzzle rules:
1. The Hitori board is 4x4(5x5) grid. Each cell in the grid
has number in the range of 1-4(1-5).
2. Connectivity constraint:
All cells that have not been blackened are secercoonected(4ccooneeee
3. Blacked cell constraint: All blackened cells cannot be adjacent(4-connected)
4. Unique constraint: The numbers in each row and column cannot be repeated.
(3) Puzzle task: Black some cells in the grid to meet the connectivity constraint, blacked cell constraint, and unique constraint mentioned above.
Transformation
(1) Larger grid: Upgrade the grid specifications 6x6 and
7x7.
(2) Encrypted: Encrypt numbers into letters using the fol- lowing encryption method: In the i-th row (from 1 to grid size), the cell numbered now becomes "'A'+ +(i+k22) grid size".
(3) Unsolvable puzzle: Generate a Hitori puzzle that cannot satisfy all of the original puzzle constraints simultaneously.
Skyscraper
Original
(1) Puzzle categories: Grid puzzle
(2) Puzzle rules:
1. The puzzle will be provided with an n n sized grid with blank cells inside. We need to use clues (located around the grid) to fill the numbers.
2. Definition and constraints for filling in numbers: Each row and column can only be filled with numbers to n (the size of the puzzle), the numbers filled in cannot be repeated, and each number in 1 to n needs to be filled at least once. The number filled in represents the building height that location.
3. Clues outside the grid: The clues for each puzzle can be one of the following two ways.
-The count hint: The numbers outside the grid indicate how many buildings can be seen from that direction. Tall build- ings will block shorter buildings, and the height of the build- ing is represented by the number filled in the blank cell. At this point, the numbers the clue indicate how many build- ings be seen from that direction towards the other end (the viewing direction is along the row or column).
-The sum hint: The numbers outside the grid indicate the height of the building visible from that direction. Tall build- ings will block shorter buildings, and the height of the build- ing is represented by the number filled in the blank cell. At this point, the numbers in the clue represent the sum of the heights of the buildings that can be seen from that direction towards the other end (the viewing direction along the row or column).
(3) Puzzle task: Fill numbers in the grid to satisfy Clues outside the grid.
18

---

Published as a conference paper at ICLR 2026
Puzzle
Rule
Description
Transformation
(1) Add diagonal constraint: Add Visibility Clues to the top left, top right, bottom left, and bottom right corners of the grid. The numbers filled in the table need to satisfy additional diagonal constraints in addition the original four directions of Visibility Clues. But there is no constraint on the diagonal that 1-n cannot be repeatedly filled in. (This constraint only occurs when using the count hint)
(2) Hide partial clues: Hide clues for certain rows and columns (represented by -1).
(3) Unsolvable puzzle: First, we generate a set clues that define a solvable puzzle. Then, we intentionally modify some of the clues to make the puzzle unsolvable. For example, consider contradiction introduced between the maximum- value clue and the non-minimum clue a column: the top clue is set to the maximum value (n), while the bottom clue is set to a value between 2 and n - 1. This is inconsistent because a maximum top clue implies the column must be strictly increasing from 1 to n, in which case the bottom clue should necessarily be
Minesweeper
Original
(1) Puzzle categories: Search puzzle
(2) Puzzle rules:
1. The Minesweeper board is 9x9 grid. Each cell may be a number 0-8 or a hidden cell (represented by ".", which may contain landmines or not).
2. The number in number cell represents the number of landmines around (8-connected with) it.
(3) Puzzle task: Search for all cells that can be determined to be landmines through numerical cell clues.
Transformation
(1) More mines and larger grid: Increase the number of landmines in the answer or use a 12x12 grid.
(2) Cluster hint: Convert the meaning of a number cell the number of landmine clusters surrounding (8-connected with) it. Among the 8 neighbors around a certain grid, any landmines that can be reached by connecting them in 8 direc- tions (up. down, left, right, or diagonal) belong to the same landmine cluster.
Since the calculation formula for the original search space of Minesweeper is derived from the rules of the standard Minesweeper, we measure the scarch space of regional Minesweeper Puzzles as 2 where N denotes the number of unknown cells. The search space of regional Minesweeper Puzzles in Lordc is slightly smaller than that of the Original. Therefore, we categorize it as UE2.
(3) Letter version: Use letters A-H represent numbers 1-8 and to represent O.
(4) Unsolvable puzzle: First, we generate a puzzle. We then randomly select a non-mine tile and modify its displayed value so that it no longer matches the actual number of sur- rounding mines. After that, we verify whether this altered value indeed makes the puzzle unsolvable.
19

---

Published as a conference paper at ICLR 2026
Puzzle
Rule
Description
Binario
Original
(1) Puzzle categories: Grid puzzle
(2) Puzzle rules:
1. The Binario board is an n-times-n grid. Each cell may be
number cell (0 or 1) or an empty cell (represented by ".").
2. Non-adjacent constraint: Each row or column should not have more than two adjacent identical numbers (4- connected).
3. Quantity constraint: The number of O's and I's each row and column is the same.
(3) Puzzle task: Fill in the empty cells with numbers
0 and 1 to satisfy the non-adjacent constraint and quantity constraint.
Transformation
(1) More empty cells and larger grid: Increase the number of empty cells or use larger grid.
(2) Extra constraint: Each question will add some unique additional constraints based on the original constraints, which will be described in the specific question.
(3) Unsolvable puzzle: First, generate binario puzzle with only one solution. Then, add numbered square opposite to the solution, thus making it unsolvable.
Navigation
Original
(1) Puzzle categories: Graph puzzle
(2) Puzzle rules:
Each question contains some road signs (such as schools, banks), given some letters, each letter corresponds road sign, and one road sign may correspond to several letters. The description the question provides some one-way paths between letters, well as starting letter and an ending landmark.
(3) Puzzle task: Find the shortest path from the starting letter to the endpoint landmark.
Transformation
(1) More complex paths: Add more complex paths to make
the path from the starting point to the endpoint longer.
(2) Add multi-hop: Add an intermediate target landmark.
(3) Unsolvable puzzle: Destroy the connectivity of the graph
so that the starting point cannot reach the ending point.
Hanoi
Original
(1) Puzzle categories: Sequential puzzle
(2) Puzzle rules:
1. Hanoi contains m pegs and disks. Disks are represented by numbers (indicating the size of the disk), pegs are rep- resented by letters, and disks stand above the pegs. Each puzzle will provide the initial state of the peg and disk, as well the target state.
2. Each step moves a disk on top of peg to another peg that is either empty or whose current top disk is larger than the
moved disk. Pegs cannot be moved.
(3) Puzzle task: Move the disks on the Pegs from their initial state to the target state.
20

---

Published as a conference paper at ICLR 2026
Puzzle
Rule
Description
Transformation
(1) Random start: Initially, the disks are randomly dis- tributed the cylinder under the condition of solvability.
(2) Custom target pegs: The target pillar is not necessarily
the last one, but is randomly assigned.
(3) Custom disk order: The order of the disks is not neces- sarily from small to large, but specified order
(4) Unsolvable puzzle: The disks are restricted to moving only to the right, and the problem verified using the BFS algorithm to filter out unsolvable problems.
Crypto
Original
(1) Puzzle categories: Pattern puzzle
(2) Puzzle rules:
1. For the KPA puzzle: Given set of plaintext and eipher- text pairs, observe the encryption method to decrypt another ciphertext.
2. For the KKA puzzle: Given an encryption method, de-
crypt the ciphertext
(3) Puzzle task: Decrypt the ciphertext to get the plaintext
Transformation
(1) More random text: Randomly generate more diffieult
ciphertext
(2) Two-layer with two samples: For the KPA puzzle, given two sets of plaintext and ciphertext pairs with the same en- cryption rules, decrypt the double-encrypted ciphertext (3) Multiple layers or Multiple segments: Multi-layer ci- phertext encryption or the text will divided into several parts, and each part will be encrypted using different en- cryption method.
(4) Unsolvable Puzzle: Regarding the KPA decryption issue, two samples are provided, each encrypted using a different method, leading to unsolvable problem.
Zebralogic
Original
(1) Puzzle categories:
Logic Puzzle
(2) Puzzle rules:
1. Problem scenario: Each problem will describe scenario that includes a specific number of houses.
2. Characteristics: Specific quantity characteristics (e.g.. name, pet, etc.), each feature has unique item equal to the number of houses.
3. Clues: Each question will provide some clues, including direct correspondence, positional relationships, and other clues.
4. Constraints: No repetition in the same dimension; each house uniquely matches one item from each dimension; rea- soning only based on clues
(3) Puzzle task: Deduce the complete correspondence be- tween houses and all dimensional characteristies based on clues.
Transformation
(1) Create harder rules: For example, instead Per-dog Sport-football + 1, we looser condition like Pet-dog Sport-football. We also add more clue types like "1 of 3" and "imply".
(2) Unsolvable puzzle: Constructing contradictory con-
straints that render the problem unsolvable.
21

---

Published as a conference paper at ICLR 2026
Irregular
Original
HardcorsLogic
Peb
P4p
Aeg
Pej
Ped
Diegonal
Sudoku
Extra
lnarapie Clees Zearalogi Drig and Hanscorbaagi
Hoses
Houes
kafeehyy
More empty calls
Originsl Bog
Hardoors Class
2.
not tos dat
Binario
10
Onginal
More complex pakho
Add mulk hop
Traditional Minsawwaper (Number Adjacent Mines)
Regional Milrosweeper (Number = Adjacent Mine Clusters)
M
Diagonal constraint
M Taplooe Gne Douil wwriicee
M
Gomeersc tine Sagastre Cotke
Skyscraper
Encrypted
KPA:
YYNSLF
YYNSLF
Decryption Exampie:
Decryption Example) ABC MMC--6FF EFC
Original
HardcoreLogic
Hittor
Figure 11: Examples some long-tail transformations for each puzzle. Left column: Examples of Sudoku, Binairo, Kakurasu, Skyscraper, and Hitori. Right column: Examples of Hanoi, Zebralogic, Navigation, Minesweeper and Crypto.
22

---

Published as a conference paper at ICLR 2026
B.3 TRANSFORMATION TAXONOMY
To systematically characterize the transformations applied in HardcoreLogic, we organize them into a taxonomy spanning three major categories: increased complexity, uncommon element, and unsolvable puzzle. Each category is further divided into subcategories, with representative examples drawn from different puzzle types. This taxonomy (Table illustrates how diverse transformations reshape the puzzles, either by enlarging the search space, strengthening or mutating rules, or deliberately creating contradictions render puzzles unsolvable.
Table 4: Detailed taxonomy of longtail transformations across puzzles.
Family
Type
Examples
Increased
IC1. Search Space
Sudoku: More empty cells and larger grid
Complexity
Expansion
Binario: More empty cells and larger grid
Crypto: More random text
Crypto: Multiple layers and segements
Hitori: Large grid
Minesweeper: More mines and larger grid
IC2. Constraint Strengthening
Zebralogic: Create harder rules
Kakurasu: Partial hint
Binario: Extra constraint
Skyscraper: Partial hint
Navigation: More complex paths
Uncommon Element
UE1. Form Variation
Sudoku: Letter encoding
Sudoku: Irregular subgrid
Kakurasu: Add block cells
Hanoi: Custom target pegs and disk order
Hitori: Encrypted
Minesweeper: Letter encoding
UE2. Rule Variation
Sudoku: Add diagonal constraint
Sudoku: Add Adjacency constraint
Sudoku: Maximize box constraint Skyscraper: Add diag hint Minesweeper: Cluster hint Navigation: Add multi-hop
Unsolvable
Unsolvable
Zebralogic: Add conflicting constraint Sudoku: Add conflicting hint Skyscraper: Add conflicting constraint Kakurasu: Add conflicting constraint Crypto: Two sample with different encryption method
Puzzle
Minesweeper: Add conflicting hint Navigation: Destroy the connectivity of the graph Binario: Add confficting hints
Hanoi: Limit the direction of disk movemen
Hitori: Generate an initial solution that does not satisfy the rules
B.4 COMPLEXITY ANALYSIS DETAILS
quantify the difficulty of logic puzzles by calculating the search space. Table lists specific expressions for calculating the search space of a logic puzzle, and Table provides detailed results for Figure
In Section 2.3
23

---

Published as a conference paper at ICLR 2026
Table 5: Definition and formula of search space in logic games.
Puzzle
Key Parameters
||]
Binario
N: The number of empty cells
2
N: The number of empty cells
Sudoku
M
M: Grid size
Crypto
L: Ciphertext length
264
04 Number in digital grid i
Minesweeper
II. CN
Nj Number of empty cells around digital grid i
Hitori
M: Grid size
2 2
Table Search space sizes of puzzles Table from Original and HardcoreLogic respectively.
Puzzle
Dataset
Mean
Median
Original
1.18 x 102
6.40 x 10!
Binario
HardcoreLogic
5.43 10
1.41 10
Original
4.08 x 1026
6.46 x 102t
Sudoku
HardcoreLogic
2.10 1048
3.04 1052
Original
4.97 x 10
3.29 x 1020
Crypto
HardcoreLogic
1.31
1033
1.35 x 1031
Original
1.48 x 106
1.48 x 106
Hitori
Hardcorel.ogic
6.22 1012
6.22 x 1012
Original
2,00 x 108
1.30 x 108
Minesweeper
HardcoreLogic
3.03 1028
8.18 x 106
24

---

Published as a conference paper at ICLR 2026
C EXPERIMENT DETAILS
C.I MODEL AND CONFIGURATION
We categorize the LLMs that we use into three types: open-source large models, open-source small models, and closed-source models. Table lists all candidate models with their parameter sizes.
Table 7: Candidate LLMs for experiments. A "closed" size indicates a closed-source model.
Family
Model
Size
GPT
GPT-5 (OpenAl 2025b)
closed
GPT-5 mini OpenA 2025b
closed
04 mini (OpenAl] 2025d
closed
Grok
Grok 4 (XAT] 2025b)
closed
Grok mini XAI 2025a)
closed
Gemini
Gemini 2.5 Pro Gemmn Team 2025
closed
Gemini 2.5 Flash (Gemin Team
2025
closed
Claude
Claude Sonnet 4 (Anthropic 2025
closed
DeepSeek
DeepSeek- V3.1 (DeepSeck- Al et al.
2024
671B
DeepSeek- - 1-0528 (DeepSeck-Al et
2025
671B
Qwen
Qwen3-235B-AM A22B-Thnkmgg-250
(Yang et al.
2025
235B
MiniMax
MaiiMax-M1144k (MiniMax et al.
2025
456B
GLM
GL.M-4.5 GLMM-45 Team et al.
2025
358B
Kimi
Kimi-K2-Instriet (Moonshot A1
2025
IT
GPT
gpt-oss- (OpenAl
2025a)
120B
DeepSeek
DeepSeek-R1-0528--0weeee
(DeepSeek-Al) et al.
2025
8B
Qwen
Owen3-Next-80B-.. 80---0--------- -
Yang et al. 2025
80B
Qwen3-32B (Yang al.
2025)
32B
Yang et al.
2025
30B
Qwen3-8B (Yang et al.
2025
8B
Seed
Seed-OOSS-SoB--nsiinn (Bytel Dance Dance Seed Team
2025
36B
few notes:
We observe in experiments that GPT-5, GPT-5 mini, and 04 mini tend exceed the 32, 768 token budget more often when choosing the "high" reasoning level. Therefore, we select the "medium" reasoning level encourage generating valid responses within the limit.
For gpt-oss-1200 120b, we keep enabling the "high" reasoning level as this model is not prone to the above issue. Following OpenAl's official guidance, we inject this setting by system prompt. Limi-K2-Instrune is not an LRM, hence we ask the model to perform CoT in the system prompt. More specifically, we adopt two-step generation approach: first, generate reasoning output wrapped between pair of special tokens, and then final answer based on the original prompt and generated COT.
C.2 PROMPT TEMPLATE
To ensure consistency and reproducibility across all puzzle types, we constructed prompt templates using a structured format. Each template specifies the puzzle description. the task instruction, and standardized JSON output schema. We adopted Jinja2-style template language so that puzzle instances can be instantiated automatically by substituting parameters such as grid size n and puzzle content. Below, we present the detailed templates for each puzzle family.
Sudoku Prompt Template
# Puzzle to Solve
{x set (subs Length) x)
25

---

of all cells in Published as a conference paper at ICLR 2026
C( })x{{ n }) sudoku puzzle a cell grid with { n }} rows and {( }) columns.
The grid is divided into {{ }} zones, each with {( n }} cells, outlined with "0
Each cell contains exactly one of the {{ }} candidate elements: for с subs[1:] 8) (( }) (x if not loop. last %), (% endif *)* endfor x). The goal is to fill all empty cells (denoted as ".") with one of these elements.
Each candidate element must appear exactly once in every row. Each candidate element must appear exactly once in every column.
Each candidate element must appear exactly once every zone. (% if diag EXTRA: Each candidate element must appear exactly once in the two diagonals. (% endif x)X if discon 8}
EXTRA: Adjacent cells cannot have adjacent elements, e.g., "{( subs[2] }}" and "(( subs[3] }}" cannot be next each other endif %JX if irzone %)
WARNING: Zones are NOT regular squares! Pay attention to their outlines! (% elif mc_box 0 x3
EXTRA: The score of zone is the sum of cell_index*cell the zone,
where cells are indexed
to { n from left right, from top to bottom; the complete puzzle should satisfy that zone {( mo_ }} has the highest
score,
where zones are also indexed from to {( n }} from left to right, from top to bottom. endif
#1 Puzzle to Solve
{{ puzzle }}
Instruction
Now please solve the above sudoku puzzle. If the puzzle is unsolvable, output "null" as the solution in the following json format:
"solvable"; false,
"solution"; null
Otherwise, present your solution in the following json format:
"solvable": true,
"solution":
{% for in range(n) x){{% for c in range(n) xy"_"(x if 8), (% endif (% endfor x)](% if r < n %), endif %)
(% endfor x)]
where each represents the final element in the corresponding cell.
Kakurasu Prompt Template
Puzzle to Solve
A C( n_row }}x[{ n_col }} kakurasu puzzle is cell grid with {( n_row }} rows and {( n_col }} columns.
Rows are numbered to {{ nLrow }}, and columns numbered (( n_col }}. The goal is to mark cells to satisfy the following column and row constraints, On top of the puzzle, a row of (( n_col }} numbers give the **column** constraints --- the row index of
26

---

Published as a conference paper at ICLR 2026
each column; a --1" indicates that column At the beginning of each row, number gives the **row** constraint the column index sum of
cells **marked as no constraint.
all cells **marked *0"** in the row; "-1" indicates that the has constraint.
The initial grid consists of and "x" cells, and only cells can be marked as "0"; "X" cells **cannot** be marked as "
#t Puzzle to Solve { puzzle }}
Instruction
Now please solve the above kakurasu puzzle. If the puzzle is unsolvable, output "null" as the solution in the following json format:
"solvable": false,
"solution"; null
Otherwise, present your solution in the following json format:
"solvable": true,
"solution";
(% for in range(n_row) x)[(X for e in range(n_col) x)_(% if e < n_col 8),
(% endif (x endfor x)](% if n_row x)) (% endif x)
(x endfor x)]
where each represents whether the corresponding cell is *xmarked as 00 ** ("tree") or not (" falsee).
Hitori Prompt Template
Puzzle to Solve
A C( }}x(( }} hitori puzzle is a cell grid with (( }} rows and (( }} columns.
goal is erase certain cells so the cells left in each row and each column are unique.
Erased cells cannot be 4-adjacent, and **all** non-erased cells must be 4- connected.
than 3 of its 8-adjacent WARNING: The puzzle is encrypted into letters!
braced cell ('(x)") cannot be erased, and cells can erased. (% encrypted %)
In row (from 1 to {( })), cell with number k now becomes ''A' (i + k 2)
% n })"
example, in row }} becomes "A"
becomes "A", row 2 "1" becomes "B" and [( n
Decrypt the puzzle back to numbers before solving it.
{% endif x)
## Puzzle Solve (( puzzle })
Instruction
27

---

Published as a conference paper at ICLR 2026
Now please solve the above hitori puzzle. If the puzzle is unsolvable, output "null" as the solution in the following json format:
"solvable": false,
"solution"; null
Otherwise, present your solution in the following json format:
"solvable": true,
"solution":
(% for in range(n) x)[(X for с in range(n) x)_(X if < 1 8), (% endif x)(x endfor x)](% if - x),(% endif x}
{% endfor x)]
where each represents whether the corresponding cell is **erasedxx ("true") not false').
Skyscraper Prompt Template
Puzzle to Solve
A (( })x(( n }} skyscraper puzzle is cell grid with [( }} rows and {( }} columns.
Each cell contains exactly of the numbers CE },, representing the height" of the cell.
Each number must appear exactly once in every row and every column. Looking from a side, cell in the front blocks x*all** cells sxbehindæx it that **not taller*x.
hint of now/collum/diagome looking from side the [[ vv }} cells in the row//colum/diaggnal that are not blocked; a number of "-1" means no constraint.
On top of the puzzle, there is the first number the hint of the main diagonal looking from top left; the next [[ }} numbers are the hints the last number is the hint of the sub diagonal looking from top right. Then, at the beginning of each grid row is the hint of that row looking from the left;
of {( }} numbers:
columns looking from the top;
at the end of that row the hint Finally, below the puzzle, there is row of {( n }} numbers: the first number is the hint of the sub diagonal looking from bottom left; the next {{ }} numbers are the hints of the columns looking from the bottom; the last number is the hint of the main diagonal looking from bottom right.
row looking from right.
## Puzzle to Solve {l puzzle })
Instruction
Now please solve the above skyscraper puzzle.
If the puzzle is unsolvable, output "null" the solution in the following json
format:
"solvable": false,
"solution"; null
28

---

Published as a conference paper at ICLR 2026
Otherwise, present your solution in the following json format:
"solvable": true,
"solution":
(% for in range(n) x)[(X for in range(n) 8)_{% if 8), (x endif *)(x endfor x)](% if n 83, endif %}
(% endfor x}]
where each represents the final number in the corresponding cell.
Minesweeper Prompt Template
# Puzzle to Solve
A (( row }x{{ col }} minesweeper puzzle is a cell grid with {( row }} rows and [[ col }} columns.
Each cell has either one mine (mine cell) no mine (safe cell). Some safe cells opened beforehand, showing the number of (% regional *)x88-connected components** (% endif x)mine cells in their adjacent cells. (% if regional x}
For example, if an opened safe cell has three 8-adjacent mine cells, but all three mine cells are 8-connected with each other, then the opened safe sell will show '1' instead The goal is to find out all closed cells that must be mine cells. The puzzle is unsolvable if and only if the current numbers lead to contradiction. (% if no_adj %]
"3" .(% endif %)
EXTRA: It is also guaranteed that mines 8-adjacent to each other endif x)% letter %)
EXTRA: The puzzle is encrypted into letters, where Z represents 0 and A-H represents 1-8. (% endif x)
## Puzzle Solve {( puzzle }}
# Instruction
please solve the above minesweeper puzzle.
If the puzzle is unsolvable, output "null" as the solution in the following json format:
"solvable"; false,
"solution": null
}
Otherwise, present your solution in the following json format:
"solvable": true,
"solution":
(% for in range(row) x)[(% for с range x)_(% endif x)(% endfor x)](% if r row x), (% endif %)
8), (%
{% endfor x)]
where each represents whether the corresponding cell **must be mine cell** ("true") or safe/undetermined (" (aaase"").
29

---

Published as a conference paper at ICLR 2026
Binario Prompt Template
Puzzle to Solve
A [( }}{{( }} binario puzzle is cell grid with { }} rows and (( columns.
Each cell can either "0" '
The goal is to fill all empty cells (denoted as -.") with -o- or - Each row must have the same number of "0's and "1's. Each column must have the same number of "0's and "1's. Furthermore, no more than two identical digits are adjacent.
## Puzzle to Solve {( puzzle }}
Instruction
Now please solve the above star battle puzzle. If the puzzle is unsolvable, output "null" the solution in the following json format:
"solvable": false,
"solution"; null
Otherwise, present your solution in the following json format:
"solvable"; true, "solution":
(% for in range(n) x)[(% for in range(n) 8)_(% if 8), (% endif x) endfor *)](% if 83, endif x}
{% endfor %}]
where each represents the final element in the corresponding cell.
Hanoi Prompt Template
Puzzle to Solve
A (( n_peg }}x(( n_disk }} hanoi puzzle has {( n_peg }} pegs and {( n_disk }} disks.
disks, in the order size, are: (smallest) (% for in order x) c }) (%
not last x), (% endif XXX endfor %} (largest).
The goal is to transform the start state to the goal state in minimum number of steps.
Each step
disk top of peg another peg that is either empty, or whose current top disk larger than the moved disk. (% right_only %) Furthermore, the target peg must be to the right of the source peg. (% endif x)
## Puzzle Solve {( puzzle }}
Instruction
Now please solve the above hanoi puzzle. If the puzzle is unsolvable, output "null" as solution the following json format:
30

---

where each encryption method consist of mul multiple sub-methods concatenated with Published as a conference paper at ICLR 2026
"solvable"; false,
"solution"; null
}
Otherwise, present your solution in the following json format:
"solvable": true,
"solution":
[" ["_", " "_",, ["_", "_"], ["_", "_"]...
where each "["_" "_"]" pair represents the peg and the target peg of disk-moving step.
Crypto Prompt Template
Puzzle to Solve
An uppercase ASCII text is encrypted into a cipher.
The goal is meanings.
the plain text, which may may not have semantic
A 1ist of candidate encryption methods may be provided, one method per line, in which case the encryption is done by applying each method once sequentially {% if ordered X} in the given order(% else %}, but NOT necessarily in the given order(X endif %).
Sample plain text-cipher pairs that use the same encryption procedure may also be
given as a hint.
When "I" appears in the cipher, the encryption is segmented,
"+" one line,
each applied to the corresponding cipher segment separated by "I".(% if prompt_example x)
**IMPORTANT: The encryption method NOT be the same in the examples!**
**Use the information below (NOT the examples) to find out the actual encryption method! **(x endif %)
## Cipher to Solve
{{ puzzle }}
# Instruction
Now please recover the above cipher. the cipher cannot recovered, e.g. there a contradiction the clues, output "null" as the solution in the following json format:
"solvable"; false, "solution"; null }
Otherwise, present your solution in the following json format:
"solvable": true, "solution": "
where represents the plain text string in uppercase.
31

---

Published as a conference paper at ICLR 2026
Zebralogic Prompt Template
Puzzle to Solve
{ puzzle }}
Instruction
Now please solve the above puzzle. If the puzzle is unsolvable, output "null" the solution in the following json format:
"solvable": false,
"solution"; null
Otherwise, present your solution in the following json format:
"solvable": true,
"solution":
(% for id in house_ide ids x)"(( house_alias }} {( id })": {
{% for key in keys x}"({ key })"; "_"(x if not last 8), endif %)
{% endfor *)K* if not last 23.(% endif %)
{% endfor x)}
where each ""_" represents attribute lowercase.
Navigation Prompt Template
Puzzle to Solve
{( puzzle }}
Instruction
Now please solve the above puzzle.
If there is no path, output "null" as the solution in the following json format:
"solvable"; false, "solution": null
}
Otherwise, present your solution in the following json format:
"solvable": true, "solution"; ["_" "
where each represents a point on the path (an uppercase letter), including the start point and the end point.
C.3 ERROR TYPES
In Sections 4.2 and 4.3 we specifically classified the types errors returned by the model. Tables and give specific definitions of each category.
32

---

Published as a conference paper at ICLR 2026
Table 8: Error types in error analysis.
Category
Definition
Misunderstanding
The model does not truly understand the logical puzzle, there is a deviation in its understanding.
Misapplied
The problem was correctly understood, but an inappropriate and often more common thinking framework was applied when select- ing solution.
Brute-Force with Excessive Complexity
Large language models attempt to solve problems through brute force search, but the search space is too large, making it difficult find a solution.
Factual/Hallucinatory
the intermediate steps of reasoning, large language models fabricate non-existent facts, data, logical relationships, leading to erroneous conclusions.
Over Verification
The correct answer appeared during the reasoning process, but was not ultimately obtained.
Infinite Repetition
The model keeps repeating certain segment during reasoning, resulting in the inability obtain results or output answers in the specified format.
Table 9: Error types UP error analysis.
Category
Definition
Erroneous reasoning
The model genuinely believes, through reasoning, that there a solution to the problem.
Mandatory response
The model did not obtain an effective solution through logical rea- soning, but was forced to answer that the problem had solution in the end.
Unable to deduce
The model cannot derive an answer within the maximum token limit (whether halfway through).
has deduced that the problem unsolvable
Infinite repetition
The model keeps repeating certain segment during reasoning, resulting in the inability to obtain results or output answers in the specified format.
33

---

Published as a conference paper at ICLR 2026
D ADDITIONAL ANALYSIS
D.1 CORRELATION BETWEEN COMPLEXITY AND MODEL ACCURACY
In Section 2.3
analyze the complexity of HardcoreLogge from algorithmic perspective. For IC1, we quantity difficulty through the expansion of the search space; for IC2 and UE2, we evaluate solver- level metrics such as conflicts, decisions, generated nodes, and expanded nodes. These indicators provide principled way to assess puzzle hardness under classical algorithmic or constraint-solvinn paradigms.
However, whether these transformations indeed increase difficulty for LRMs remains an empirical question. To align algorithmic hardness with LRM performance, we conduct an additional analysis in this part
For each puzzle instance, our evaluation adopts an Rsampling 4 protocol, where a model is queried four times and the instance-level success rate is computed the proportion of error-free outputs. To examine how solver-based complexity measures relate to LRM performance, we correlate these success rates with classical complexity.
Figure 12 Figure 13 and Figure 14 summarize how LRM success rates vary with different complexity indicators under ICI, IC2, and UE2. To quantitatively validate these relationships, Table 10 [9 report the corresponding significance tests, showing the statistical strength these complexity -ectlly-prrorrccccc correlations across all models.
Bibario
Cayyte
Miluo
2
Figure 12: Correlation between IC1 complexity indicators and LRM success rates
kini
Zebra coafficts
coaffiets
coaflicts
Skyscraper
Nasigstion expunded
Siaccen S414
Succcn Bate
2
60.
capende_tool
Skyseraper
Navigation generated
Stecen Iaae
Saccese Bide
Sunces Mare
08
40
decitives
decidonc
decicions
peurated
Figure 13: Correlation between solver- based IC2 complexity indicators and LRM success rates.
D.2 KEY CELLS VS. COMPLEXITY
Among 10 puzzles, the Search puzzles include Hitori, Minesweeper, and Kakurasu. They all have one thing in common: searching for (or deleting) certain key cells. However, we observe that increasing these key cells does not necessarily make the puzzle harder:
34

---

Published as a conference paper at ICLR 2026
43
minbnax-mi quead
qwen3-2356
deqpeck-r1
alu-45
encaa-323
ques-ocit
Sudoku confliets
Skyseraper conflicts
Navigation expanded node:
14
1.0
Success Rate
Success Rate
0.6-
0.6
0.4
6.4
0.2
6.2
2009
4009
2000
4000
10
20
conflicts
flicts
expanded modes
Sudoku decisions
Skyseraper decisions
Navigation generated node
1.41
1.0
Sueces Rute
0.8
Success Ratee
8.8
0.6
0.6
8.6
0.--
20003
decisions
generated
Figure 14: Correlation between UE2 complexity indicators and LRM success rates.
Table 10: P-values from significance tests evaluating the relationship between ICI search space complexity and LRM success rates (Part 1).
Game
deepseek qwen deepseek rl deepseek v3.1
glm 4.5
gpt-oss- 120b
kimi-k2
Binario
4.24c-83
2.21e-119
1.35e-114
2,45c-81
1.88e-106
6.73e-96
Crypto 3.20e-15
3.32c-04
2.22e-17 1.16e-37
1.64e-16
1.82e-50
Hitori
6.25e-29
5.58e-35
1.51e-25
8.19e-24
1.69e-19
4.81e-29
Minesweeper
3.95e-10
1.79e-35
2.94e-32 3,41e-12
1.05e-36
5.71e-20
Sudoku
2.97e-09
3.64e-22
7.88e-22
5.66e-17
2.46e-54
1.23e-13
Table 11: P-values from significance tests evaluating the relationship between ICI search space complexity and LRM success rates (Part 2).
Game
minimax-m1
qwen3-235b qwen3-30b-aßb
qwen3-32h
qwen3-8b
qwen-next
Binario 3.99e-75
6.76e-111
1.32e-111
4.82e-101
1.02e-91
6.90e-113
Crypto 2.18e-10
2.16e-07
1.63e-31
4.90e-32
1.35e-39
6.58e-35
Hitori
1.39e-27
2.35e-34
7.54e-36
8.29e-37
2.20e-32
1.01e-27
Minesweeper 5.34e-01
1.10e-32
1.12e-34
4.20e-05
4.05e-02
3,45e-31
Sudoku
3.67e-07
7.08e-27
2.77e-27
1.14e-16
4.58e-15
4.52e-27
Table 12: P-values from significance tests evaluating the relationship between IC2 conflicts com- plexity and LRM success rates (Part 1). For the Navigation task, the solver metric used is expanded nodes
Game
deepseck qwen deepseck rl deepseck v3.1
glm 4.5
gpt-oss-120b
kimi-k2
Binario
5.13e-51
3.84e-50
4.86c-60 2.36c-41
5.29e-49
1.67e-68
Kakurasu
8.01e-03
1.78e-01
7.90e-01
4.55e-01
5.34e-01
5.01e-01
Skyscraper
2.24e-01
7.67e-07
1.77e-06
2.55e-08
2.43e-16
2.78e-01
Zebra
1.53e-16
1.00e-21
1.41e-14
1.74e-13
6.88e-23
2.15e-08
Navigation
7.38e-20
1.55e-01
7.53e-03
1.82e-06
2.28e-04
5.65e-26
35

---

Published as a conference paper at ICLR 2026
Table 13: P-values from significance tests evaluating the relationship between IC2 conflicts com- plexity and LRM success rates (Part 2). For the Navigation task, the solver metric used is expanded nodes
Game
minimax-m1
qwen3-235b qwen3-30b-a3b qwen3-32b
qwen3-8h
qwen-next
Bínario
1.71e-53
5.88e-47
7.36e-64
8.48e-58
7.98e-59
4.73e-61
Kakurasu
1.66e-01
5.58e-01
7.72e-01
5.38e-05
2.28e-02
1.51e-01
Skyscraper
9.27e-02
6.08e-15
9.09c-07
3.70e-07
2.31e-02
9.18c-14
Zebra 9.80c-09
9.10e-21
1.65e-20
2.71e-17
1.85e-12
2.25e-23
Navigation
1.16e-01
3.56e-01
3.74e-04
2.70c-07
4.37e-23
3.25e-04
Table 14: P-values from significance tests evaluating the relationship between IC2 decisions complexity and LRM success rates (Part 1). For the Navigation task, the solver metric used is generated nodes
Game
deepseck qwen deepseek deepseck v3.1
glm 4.5
gpt-oss-120b kimi-k2
Binario
2.35e-32
4.02e-63
1.76e-56
6.37e-35
1.03e-64
2.79e-41
Kakurasu
1.29e-03
3.00e-01
8.34e-01
1.55e-01
5.27e-01
1.71e-01
Skyscraper
2.11e-01
1.58e-07
1.49e-07
1.13e-09
6.72e-19
2.12e-01
Zebra 3.62e-22
4.10e-29
9.19e-18
4.51e-18
3.54e-28
7.82e-11
Navigation
6.69e-22
1.30c-01
3.41e-02 9.42e-07
1.01e-04
2.45e-28
Table 15: P-values from significance tests evaluating the relationship between UE2 decisions com- plexity and LRM success rates (Part 2). For the Navigation task, the solver metric used is generated nodes
Game
minimax-m1
qwen3-235b qwen3-30b-a3b
qwen3-32b
qwen3-8b
qwen-next
Binario
6.33e-34
6.30e-57
2.10e-53
6.56e-47
7.50e-38
3.77e-61
Kakurasu
6.17e-02
4.03e-01
5.57e-01
1.29e-05
1.11e-02
3.27e-02
Skyscraper
6.59e-02
1.11e-17
1.25e-07
1.63e-08
1.21e-02
7.64e-16
Zebra
3.59e-11
2.03e-27
2.51e-27
2.94e-23
3.67e-16
5.83e-32
Navigation
1.13e-01
1.83e-01
3.38e-05
6.02e-07
6.15e-25
9.91e-04
Table 16: P-yalues from significance tests evaluating the relationship between UE2 conflicts complexity and LRM success rates (Part 1). For the Navigation task, the solver metric used is expanded nodes
Game
deepseek qwen deepseek rl deepseek v3.1
glm 4.5
gpt-oss-120b kimi-k2
Skyscraper
3.77e-01
1.20e-03
4.05e-03
3.43e-04
3.33e-10
5.60e-01
Sudoku
5.74e-08
4.48e-27
1.23e-24
1.13e-14
1.60e-38
5.49e-12
Navigation 2.60e-15 15
2.98c-02
1.47e-04
1.51e-06
1.79e-03
3.57e-18
Table 17: P-values from significance tests evaluating the relationship between UE2 conflicts com-
plexity and LRM success rates (Part 2). For the Navigation task, the solver metric used is expanded nodes
Game
minimax-m1
qwen3-235b
qwen3-30b-a3b
qwen3-32b
qwen3-8b
qwen-next
Skyscraper
2.98e-01
2.99e-07
1.12e-03
7.90e-04
5.36e-02
4.92e-09
Sudoku
2,88e-06
6.85e-31
4.06e-33
1.52e-15
1.32e-12
1.56e-29
Navigation
6.89e-09
6.79e-02
2.56e-03
2.29e-05
1.89e-14
1.78c-04
36

---

Published as a conference paper at ICLR 2026
Table 18: P-values from significance tests evaluating the relationship between UE2 decisions complexity and LRM success rates (Part 1). For the Navigation task, the solver metric used is generated nodes
Game
deepseck qwen deepseck rl deepseck v3.1
glm 4.5
spt-oss-120b kimi-k2
Skyscraper
2.60e-01
2.16e-04
7.46e-04
2.96e-05
2.27e-13
4.65e-01
Sudoku
1.61e-10
2.42e-33
4.10c-30 6.54e-19
1.53e-37
2.16e-15
Navigation
7.58c-17
8.82c-03
1.87e-04
8.31e-07
5.55e-04
4.99e-21
Table 19: P-values from significance tests evaluating the relationship between UE2 decisions com-
plexity and LRM success rates (Part 2). For the Navigation task, the solver metric used generated nodes
Game
minimax-ml
qwen3-235b qwen3-30b-aßb
qwen3-32b
qwen3-8b qwen-next
Skyscraper
2.19e-01
1.31e-09
2.48e-04
3.61e-05
2.11e-02
4.64e-12
Sudoku
3.14c-08
1.47e-37
3.6le-41
1,48c-19
2.83e-16
2.91e-36 1.13e-18
Navigation 2.02c-09
3.49e-02
9.99c-04
5.25c-06
3.83e-04
Under the same grid size, increasing the number of cells to be erased in Hitori does not make more difficult according to the CSP solver.
Minesweeper from Original are leveled according the number of landmines; however, the search space does not vary much.
On Kakurasu, increasing the number of marked cells also increases the conflicts of decisions. which is the sole positive case.
We tested the performance of Hitori when only increasing the number of cells to searched(results shown in Table and found that there was no significant difference in performance compared to the Original data when the model was large, but there was significant difference when the model was small. The grading of Minesweeper also indicates this conclusion that increasing the number of cells to be searched is more difficult for smaller models. For models with insufficient reasoning ability. it is not possible to think about multiple cells in mixed manner, and it is necessary to think about each cell. Whenever they determine whether a cell is the one they need to find, the probability of errors increases, and increasing the number of cells that need to be found makes it difficult. Due the unclear impact of this factor, we did not consider it as an independent long-tail transformation. HardcoreLogie has an average of more cells to find for on search puzzles of the same size than the Original dataset.
Table 20: Performance Hitori of the same size. Compared with the data from Original, the data from HardcoreLogic requires more cells to be searched.
Data type
gpt-oss- 120b
qwen3-235b
qwen3-8b
Original-4 x 4
91.00
88.00
68.00
Original-5 x 5
81.50
55.00
29.50
HardcoreLogic-4 X
90.50
85.50
62.50
Hardcorel.ogic-B
84.00
47.00
21.50
D.3 ERROR TYPE ANNOTATION CONSISTENCY ANALYSIS
In Sections 4.2 and we conducted detailed error analyses for both regular reasoning failures and UP cases, covering UP-error and UP-sufficient categories. For all sampled instances, the final labels were obtained through voting-based annotation scheme involving three annotator LLMs (Gemini-2.5 Pro, Claude Sonnet 4.5, and GPT-5), followed by manual resolution when no majority vote reached. Table 21 and Table 22 report the consistency analysis of these annotations. We use Fleiss' Kappa to measure agreement among the three annotator models, and Cohen's 's Kappa to quantify the agreement between each individual annotator and the final (three MLMshuuman hybrid)
37

---

Published as a conference paper at ICLR 2026
labels. The results show generally high agreement, especially the consistently strong alignment between GPT-5 and the final annotations, indicating the reliability the labeling process.
Table 21: Inter-annotator agreement for error-type labels across three annotator LLMs (Gemini,
Claude, GPT-5) on both Original and Hardcorel Logic datasets. The table reports Fleiss' Kappa for multi-rater agreement and pairwise Cohen's Kappa between each annotator and the final voted label.
gpt-oss- -20b kimi-k2 minimax-ml qwen3-235b
overall
0.32
0.51
0.55
0.36
0.50
Gemini final
0.35
0.75
0.78
0.55
0.64
Original
Claude vs final
0.63
0.54
0.87
0.54
0.68
GPT5 vs final
0.82
0.87
0.59
0.83
0.81
Gemini-Claude GPT5
0.37
0.29
0.52
0.34
0.43
Gemini final
0.50
0.62
0.74
0.66
0.66
Hardcore
Claude vs final
0.51
0.38
0.77
0.34
0.54
GPT5 final
0.92
0.67
0.72
0.87
0.81
Gemini-Claude Mlaude-GPTI
0.36
0.41
0.55
0.36
0.47
Gemini final
0.43
0.69
0.76
0.61
0.65
Both
Claude vs final
0.57
0.46
0.82
0.44
0.61
GPT5 final
0.87
0.78
0.67
0.85
0.81
Table 22: Inter-annotator agreement for UP-error and UP-sufficient cases. Similar Table 2] the
table includes Fleiss' Kappa across the three annotator LLMs and pairwise Cohen's Kappa with the final voted label, reflecting the reliability of annotations in the unsolvable-puzzle setting.
gpt-oss- 120b kimi-k2 minimax-mi qwen3-235b overall
GPT5
0.55
0.25
0.45
0.35
0.54
Gemini vs final
0,48
0.58
0.65
0.43
0.62
error
Claude final
0.88
0.21
0.48
0.60
0.69
GPT5 vs final
1.00
0.91
0.92
0.93
0.96
Gemini-Claude
0.51
0.36
0.019
-0.02
0.27
Gemini vs final
0.54
1.00
0.00
0.00
0.39
sufficient
Claude final
1.00
0.65
0.37
0.00
0.58
GPT5 vs final
0.67
0.43
0.71
0.85
0.66
D.4 SKYSCRAPER SOLUTION COUNT
We found that almost all models performed poorly in solving Skyscraper, due to the difficulty of the problem itself. We found that the number of solutions to such difficult puzzles may affect the performance of the model. We performed two different long-tail transformations on Skyscraper: add diagonal constraints and hide partial clues. These two types of long-tail transformations referred to as diag and partial. These two types of long-tail transformations show improvements in both decisions and conflicts compared the Original dataset the same size. However, found that some well-performing models, the accuracy of large-sized (6 x 6 and above) partial transformations (without guaranteed unique solutions) partially increased, while diagonal transformations and 5 partial transformations (with guaranteed unique solutions) showed significant downward trend model performance
shown Table 23 Large-sized partial transformations result in an increase in the number of solutions due to hidden clues, which affects the performance of the model. The partial transformation and 5 5 diagonal transformation ensure that the solution does not increase compared to the Original dataset, and with the increase of decisions and confficts, even in some diagonal transformation data that can be solved with the original constraints, the performance of the model still decreases significantly. So when the puzzle is difficult and the model does not have enough clues to analyze, may tend to guess the answer, and the number solutions becomes factor affecting the difficulty of the game.
38

---

Published as a conference paper at ICLR 2026
Table 23: The performance of some models Skyscraper with sizes of 5 x and using the count hint.
Data type
gpt-oss- 120b
deepseek-v3.1
qwen3-235b
Original-5 5
41.30
14.13
30.43
Original-6 x
1.85
0.00
0.00
HardeoreLogiiodiag- x 6
19.50
1.50
8.50
HardcoreeLogieediag--6 x 6
0.50
0.00
0.00
HardcorelLogic-parttee
24.50
7.00
22.00
HardeoreLLogi--partta-- x
5.00
0.00
0,00
D.5
OTHER ANALYSES OF WEIGHTED MULTIPLE LINEAR REGRESSION
In Section 4.1
performed weighted multiple linear regression to examine the effects of four different long-tail transformations on puzzle difficulty. Concretely, we fit the following model:
y = kICI l1CI + lic2 1rc2 + kUEi JUEI + kuEa lUE2 +
where is the observed accuracy for specific puzzle variant, LIC1 is a binary indicator (1 if transformation ICI is applied, otherwise), KIC1 quantifies the marginal accuracy change attributable to ICI under the assumption of additive effects, 6 is the expected accuracy predicted by the model when all dummy variables are zero, and weights w;=N; (sample sizes) give greater influence to observations with larger sample sizes when caleulating the loss function.Weighted linear regression isolates the marginal effect individual transformations through two mechanisms: (1) the additive linear model with dummy variables statistically disentangles combined transformation effects by estimating each factor's 's contribution relative to the baseline configuration, (2) sample-sized based weighting assigns greater influence high-reliability observations during coefficient estimation. ensuring parameters reflect dominant patterns in robust data.
To complement the results presented in Section 4.1 of the main text, this appendix provides additional details of the weighted multiple linear regression analysis. First, we refitted the model using data that contained only a factor, excluding all data points that included multiple factors. Second, based on the original multivariate model, we computed the corresponding 95% confidence intervals (and corresponding p-values) of the regression coefficients.
Binario
-74.09
0.00
0.00
0.00
95% CI of Puzzle Transformations (Al Variants)
Cryptn-
-26.58
0.00
0.00
0.00
-10
Raaris
Hunoi
0.00
0.00
-22.11
0.00
-20
Osyta
Hitori
46.06
0.00
-7.00
Howis
-30
Puzzle
Kakurasu
0.00
-9.87
-9.64
0.00
Minesweeper
-25,38
0.00
0.00
-29.67
-40
Navigution
0.00
-10.00
0.00
0.00
Missmaga
-50
Navigation
Skyscraper
0.00
-2.04
0.00
-3.70
-60
Skynergurr
Sudoku
Sadika
K2
Zebrulogic-
0.00
-33.58
0.00
0.00
-70
IC1
IC2
UÉ1
46
Dasic Long tail Trans (Singae Factor)
Figure 15: Left: Effects long-tail transformations on puzzle csiuracy.(single factor) Right: 95%
confidence intervals for puzzle difficulty coefficients.
The left side of Figure 15 shows the impact of long-tail transformations on puzzle accuracy when only considering single factor data. For most puzzles, the coefficients obtained from this simulation closely match those from the full multiple regression model. The only notable deviation occurs in Sudoku. This is because the UE1 category for Sudoku actually contains two heterogeneous subtypes letter version and irregular subgrid. The letter version variant has only a minor standalone effect and appears only in combination with other variants, whereas the irregular subgrid variant never co-occurs
letter
39

---

Published as a conference paper at ICLR 2026
Table 24: p-values of the fitted weighted linear regression.
Puzzle
IC1
IC2
UE1
UE2
Puzzle
IC1
IC2
UE1
UE2
oobc
.000
Minesweeper
.000
.982
.000
Sudoku
.000
.000
.006
Navigation
.132
.402
Skyscraper
.098
.010
Binario
.000
.661
Kakurasu
.000
.188
.403
Hanoi
.000
Crypto
.000
Hitori
.000
.283
with any other factors. As result, when simulating Sudoku using single-facte factor data, the model's s intercept becomes shifted, which in leads changes in the estimated parameters.
The right panel of Figure 13 presents the 95% confidence intervals of the with the corresponding p-values reported in Table 24 Most of the confidence interval bounds are negative, and the overall conclusions are consistent with those in Section 4.1 The figure further
coefficients,
shows that, even after accounting for estimation uncertainty, ICI still exhibits the largest effect size
in our data. Moreover, all p-values associated with ICI are below 0.001, confirming that its influence on puzzle difficulty is statistically significant.
40


Reviews:
[{"review_id": "SnS7TJtOM4", "reviewer": "Reviewer_na2W", "summary": "This paper builds a new benchmark, HardcoreLogic, to test whether large reasoning models (LRMs) can truly reason or are just memorizing common puzzle patterns. The authors build a dataset of over 5,000 puzzles across 10 different games such as Sudoku, ZebraLogic, and Minesweeper. These are drawn from or aligned with earlier logic-reasoning datasets (like Enigmata, ZebraLogic), but then systematically transformed. Unlike existing datasets, HardcoreLogic includes three types of harder variants: Increased Complexity (IC) – bigger puzzles or denser constraints; Uncommon Elements (UE) – unusual formats or new rules, and Unsolvable Puzzles (UP) – puzzles with no valid answer. When tested on this benchmark, even top-performing models that excel on standard puzzles showed large drops in accuracy, especially when the puzzles were slightly altered or unsolvable.", "questions": "Please see Weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper clearly identifies a real hole in current LRM evaluations: most logic-puzzle benchmarks stay in the “canonical” zone, so they don’t reveal whether models can handle odd, rare, or slightly rule-tweaked versions.\n\n- The benchmark spans 10 games from 6 categories (logic, grid, search, pattern, graph, sequential), so the paper’s claims aren’t tied to a single puzzle formalism. That’s good evidence that the problem is about LRMs’ robustness, not about one idiosyncratic game. It  totals >5,000 puzzles. \n\n-  The experiments directly compare Original vs. HardcoreLogic and show that all models, including SOTA, lose accuracy when puzzles are made longer-tail.", "weaknesses": "- Section 2 claims that puzzles are extended along three “orthogonal” dimensions — Increased Complexity (IC: IC1, IC2), Uncommon Elements (UE: UE1, UE2), and Unsolvable Puzzles (UP). **Why are they orthogonal?**\nHowever, Section 4 immediately notes that “puzzles may also have two different long-tail transformation attributes at the same time.”\nThis means the core experimental factor is conflated: when accuracy drops, we cannot uniquely attribute the drop to IC1 vs. UE1, because several knobs were turned at once. Nevertheless, Section 4.1 fits a linear model and makes a fairly strong statement that IC1 “has the greatest comprehensive impact.” That claim seems stronger than the actual design supports. \nOn top of that, the paper doesn’t report confidence intervals or any regularization that would stabilize the estimates, so the single set of coefficients they show looks more definite than it really is.\n\n\n- In Sec. 3, “a generation run is considered correct if and only if the model successfully finishes reasoning and produces a correct answer” under a 32,768-token budget. This policy conflates two different failures: the model logically failed, and the model found the answer but over-generated verification and hit the budget.\nGiven that many LRMs today use self-critique / multi-step CoT, the current metric likely underestimates the real reasoning capability on the hardest instances. A secondary metric (“first correct step,” “correct intermediate state”) would make Section 3 more credible.\n\n- The error analysis in Sec. 4.2 is a nice idea (six failure types), but it is built on a limited set of erroneous responses and on automatic judgments. With such a small pool and no human inter-annotator agreement, fine-grained statements like “stronger models show more brute-force errors” should be treated as hypotheses, not as solid findings. There is no human inter-annotator agreement reported.\n\n- Sec. 2 says UPs “deliberately lack a valid solution” to test whether models detect inconsistency. But the main text does not spell out whether UPs are generated via contradiction, under-specification, or conflicting hybrids. In Sec. 4, however, models are evaluated on whether they “justify” unsolvability. Without a clear taxonomy of UP generation in the main paper, it is hard to know whether model mistakes are due to weak inconsistency detection or to task underspecification.\n\n\n\n\n- Section 3 explicitly says that, due to cost, closed-source models are tested on a small number of cases and that per-game analysis “mainly focuses on open-source models.” Yet the intro-level narrative is still all models, including SOTA, degrade. With such small samples, long-tail variance can dominate; strong statements about closed models should be toned down or supported with stratified sampling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wyCd96xfGB", "reviewer": "Reviewer_izsk", "summary": "The paper introduces HardcoreLogic, a new benchmark with 5k puzzles spanning over 10 logic puzzle games designed to probe reasoning models on long-tail of logical. The benchmark is constructed by systematically applying long-tail transformations on exiting logical games/puzzles in primarily three ways: Increased Complexity, Uncommon Elements, and Unsolvable Puzzles, and evaluates a broad set of open and closed models. The study reports substantial accuracy drops relative to “Original” datasets, analyses transformation-induced difficulty and provides analysis on error types and model behaviours for solvable/unsolvable puzzle cases.", "questions": "- The paper lack head-to-head comparison of error types between the HardcoreLogic and the original problems. Does factual errors increase in HardcoreLogic vs original? Does Brute-force errors exists in LRMs when solving the original problems?\n- What is Z3? not defined anywhere in the paper.\n- Figure 7 missing label for orange error type.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces HardcoreLogic, a valuable new resource for the community. By evaluating on proposed long-tailed transformed logic puzzles with increased difficulty, unknown element, and addition of unsolvable puzzles, the benchmark highlights through their evaluations and error analysis, that a lot of the time the reasoning models rely on their memorised experiences rather than genuine reasoning over the given problem. \n- The inclusion of unsolvable instances to evaluate ability of LLMs reason and comprehend insufficiency of information is uncommon and valuable.\n- The analysis is thorough, investigating performance across different game types, long-tail transformations and models (open/closed), with insightful error analysis on models and why they perform poorly on HardcoreLogic.\n- The paper is well-written, the methodology is sound, and the results are clear and impactful.", "weaknesses": "- The qualitative proxies for hardness (e.g., search-space scale, Z3 decisions/conflicts, generated/expanded nodes) are informative, but the paper does not show whether they correlate with what humans perceive hard or LLM error rates. Without such correlations, it is hard to conclude the transformations reliably push puzzles into the “long-tail”.\n- Error analysis lacks reliability and confidence reporting. Although GPT-5 is used as an annotator, the resulting error analysis is hard to trust without inter-annotator agreement or multiple human/LLM annotators. The claims of the benchmark can be vastly improved with improving the error analysis and using human experts (or use of multiple LLM-as-judge models), and proving confidence scores.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EkrJDFEZ3y", "reviewer": "Reviewer_xWwM", "summary": "This work introduce HardcoreLogic, a challenging benchmark comprising over 5,000 long-tail variants of 10 logical puzzle games, designed to test the robustness and genuine high-level logical reasoning of Large Reasoning Models . To bypass model reliance on memorized canonical patterns, the benchmark systematically transforms puzzles using Increased Complexity, Uncommon Elements, and Unsolvable Puzzles. Evaluations on a diverse set of LRMs, including SOTA models like GPT-5, revealed a significant performance degradation, indicating that current models struggle with less conventional scenarios and often rely on memorization rather than flexible reasoning. The dominant source of difficulty is the Increased Complexity, which tests the model’s ability to handle larger search spaces and greater constraint entanglement.", "questions": "When curating the benchmark, it would be great for show more examples so that the readers can understand the details of the data construction process.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The work introduce a challenging benchmark, which helps evaluate the reasoning and generalization capabilities of LRMs.\n2. The statistics and the analysis on the model responses are informative, which reveals the weaknesses of the LRMs.", "weaknesses": "1. The data construction process is not so clear. It is not clear whether datasets are currates manually, rule-based, or by LLMs. How do you guarantee the validity of the transformed questions and answers.\n2. Some important experimental and analysis are also missing. How do you evaluate the correctness of the responses? \n3. For the error analysis, GPT-5 was used to do the classification. It is not clear how robust is such setting. Manual analysis may be necessary for more reliable analysis. In section 4.3, are the analysis conducted by humans or LLMs? How many samples do you use for the analysis?\n4. One important concern of this work is the novelty. The main contribution of this is a transformed benchmark that is more difficult and diverse but still similar to the original benchmark, which limits its novelty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1hAOSDVZ0p", "reviewer": "Reviewer_eXAq", "summary": "In this paper, the authors introduce HardcoreLogic, a new benchmark of puzzle and algorithmic reasoning problems that systematically expands the complexity of several well-known puzzle datasets (e.g., Zebralogic, Enigmata, Hanoi game and the others shown in Table 1). In particular, they use the taxonomy of transformations detailed in Section 2.2, which includes increasing problem complexity (via increases to the search space and depth of reasoning), constraint strengthening, modificaitons to question forms, rule variation, and introduce an unsolvable puzzle setting that are not (as I understand) native to any of the existing datasets.\n\nGiven that most datasets are carefully contructed using algorithmic tools like Z3, creating such modifications is not an altogether straightforward task. Figure 2 shows the increase in the search space size of the resulting puzzles, and Figure 3 very nicely illustrates the relative differences (in the green and purple plots) in the numbers of SAT conflicts and decisions (often a proxy for empirical hardness of a problem) between the old and new datasets. Computationally, these problems are indeed considerably more difficult (I'm naturally wondering, however, how such added complexity increases the raw size of the input problem. Perhaps I missed this detail somewhere?)\n\nThey experiment with a comprehensive set of both commerical and open weight large reasoning models (GPT-5, GPT-oss, deepseek, qwen3, etc.) and a single non-reasoning model kimi-k2. Figure 4 shows the aggregate performance of different models across old and new puzzle tasks, where all models suffer significant reductions in performance (some of the particular details here are interesting, such as the relative robustness of the GPT models and Grok). Dataset-level results are shown in Figure 5, which does indicate that the degree of difference is quite mixed across these different tasks. \n\nTo better understand the effects of the particular puzzle transformations on model performance, they fit a linear regression to each puzzle task, the results of which are reported in Figure 6. I find this analysis to be very interesting and unique, as it could provide a recipe for how to create an even harder set of tasks.  They also report some interesting error analysis.", "questions": "-- Did you experiment any anthropic models?\n\n-- My question from above: how does the increase complexity affect the input size of each problem? I worry that making problems too complex might result in puzzles that are very cumbersome and far too verbose to be realistic.  \n\n-- Why did the transformations have such a small effect on the skyscaper tasks (reported in Figure 5)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-- A **new set of hard puzzle datasets** that systematically improve the complexity and difficulty of several well benchmarks. I could imagine this dataset being the new standard resource for these class of problems. \n\n-- **Compelling empirical evidence** about the hardness of the tasks, with very interesting analysis concerning the factors that contribute to the increased difficulty and current error cases. \n\n-- A very clear and easy to read paper that will be approachable for those not working directly in this area.", "weaknesses": "-- (**minor**) Whenever complexity is increased, especially for transformations involving novel rules (i.e., their *uncommon elements*), one runs the risk of introducing ambiguous or impossible to understand constraints. I'd be curious to know if anything was done to mitigate such issues (e.g., manual analysis of some kind or human evaluation) and would like to have the authors address this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)