BeyondBench: Contamination-Resistant Evaluation of Reasoning in Language Models
Authors: Gaurav Srivastava
Aafiya Shamshad Hussain
Zhenyu Bi
Swastik Roy
Priya Pitre
Meng Lu
Morteza Ziyadi
Xuan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: benchmarking
evaluation
large language models
reasoning
Primary area: datasets and benchmarks
TL;DR: In this paper we propose BeyondBench, a dynamic evaluation framework that generates algorithmic problems to measure genuine reasoning ability in language models rather than memorized patterns.

Abstract: Evaluating language models fairly is becoming harder as static benchmarks risk contamination by training data, making it unclear whether models are truly reasoning or just recalling answers. We introduce **BeyondBench**, an evaluation framework that avoids this problem by using **algorithmic problem generation**. Unlike traditional benchmarks that risk contamination from internet-scale training data, BeyondBench creates mathematically grounded problems on the fly, ensuring each test remains fresh and uncontaminated. Our framework covers **44 algorithmic tasks** with a total of **117 variations**, grouped into three difficulty levels: the *Easy Suite* (29 tasks) for basic arithmetic and statistics, the *Medium Suite* (5 tasks, 49 variations) for sequence patterns and reasoning, and the *Hard Suite* (10 tasks, 68 variations) tackling NP-complete and constraint satisfaction problems. Each task generates problems from a combinatorial space larger than $10^{15}$ unique instances, with solutions verified deterministically by mathematical proofs. We evaluated **101 language models**, including 85 open-source and 16 closed-source models, spanning sizes from 0.5B to 141B parameters and multiple quantization schemes. All evaluations use three-fold evaluation to ensure statistical robustness. Our results show consistent reasoning deficiencies across model families, with performance degrading sharply as problem complexity increases from polynomial to exponential. In our Hard Suite evaluations, models such as Gemini-2.5-pro, Llama-3.3-70B, and Qwen2.5-72B achieved average accuracies of **56.21%, 27.16%, and 33.37%,** respectively. Moreover, we observe that performance drops drastically without tool usage, with GPT-5, GPT-5-mini, and GPT-5-nano showing a **decline** of **16.81%, 15.86%, and 43.95%** in overall accuracy without tool access. The contamination resistance of BeyondBench rests on three guarantees: (i) the problem space is vastly larger than any static dataset, (ii) every instance has a deterministically verifiable solution (unique or fully enumerated), and (iii) isomorphic transformations generate semantically equivalent but syntactically new problems. BeyondBench redefines reasoning evaluation through genuine algorithmic problem-solving, ensuring fair and meaningful evaluation. Our public leaderboard is available at https://ctrl-gaurav.github.io/BeyondBench/. Our open-source Python package is available at https://pypi.org/project/beyondbench/, and the codebase can be found at https://github.com/ctrl-gaurav/BeyondBench for easy and reproducible evaluation.

Published as a conference paper at ICLR 2026
BEYONDBENCH: CONTAMINATION EVALUATION OF REASONING IN LANGUAGE MODELS
Gauray Srivastava ; Aaffiy Hussain 2, Zhenyu Bi Swastik Roy Priya Pitre
Meng Lu Morteza Ziyadi Xuan Wang ?+
Computer Science, Virginia Tech, USA
Amazon AGI, USA
(gks, aafiyahussain, zhenyub, priyapitre, menglu, xuanw) @vt.
(roswasti, mziyadi @amazon. com
Leaderboard: etrl-gaurav wwt.ggthhhhh... Bo/ecccdeeceeey
o GitHub: httpo//tttcllgauuu/////odddddodd hypl:ppppll..crrpppppo/////oooop
ABSTRACT
Evaluating language models fairly is becoming harder as static benchmarks risk contamination by training data, making it unclear whether models are truly rea- soning or just recalling answers. We introduce BEYONDBENCI an evaluation framework that avoids this problem by using algorithmic problem generation. Unlike traditional benchmarks that risk contamination from internet-scale train- ing data, BEYONDBENCE creates mathematically grounded problems on the fly, ensuring each test remains fresh and uncontaminated. Our framework covers 44 algorithmic tasks with a total of 117 variations, grouped into three difficulty levels: the Easy Suite (29 tasks) for basic arithmetic and statisties, the Medium Suite (5 tasks, 49 variations) for sequence patterns and reasoning, and the Hard Suite (10 tasks, 68 variations) tackling NP-complete and constraint satisfaction problems. Each task generates problems from a combinatorial space larger than 10t5 unique instances, with solutions verified deterministically by mathematical proofs. We evaluated 101 language models, including 85 open-source and 16 closed-source models, spanning sizes from 0.5B 141B parameters and multiple quantization schemes. All evaluations use three-fold fold evaluation to ensure statistical robustness. Our results show consistent reasoning deficiencies across model families, with per- formance degrading sharply as problem complexity increases from polynomial to exponential. In our Hard Suite evaluations, models such as Gemini-2..5pre Llama- 3.3-70B, and Qwen2.5-721 achieved average accuracies of 56.21 %, 27.16% and 33.37% %, respectively. Moreover, we observe that performance drops drastically without tool usage, with GPT-5, GPT-5-mini, and GPT-5-nano showing decline 16.81% %,, 15.86 %, and 43.95% % overall accuracy without tool access. The contamination resistance of BEYONDBENCH rests three guarantees: (i) the problem space vastly larger than any static dataset, (ii) every instance has a deterministically verifiable solution (unique fully enumerated), and (iii) iso- morphic transformations generate semantically equivalent but syntactically new problems. BEYONDBENCH redefines reasoning evaluation through genuine algo- rithmic groblem-solving ensuring fair and meaningful evaluation.
INTRODUCTION
Modern large language models (LLMs) exhibit impressive performance on existing static reasoning benchmarks such as GSM8K (Cobbe et al. 202T). MATH (Hendrycks et al. (022b), and Olympiad- Bench (He et al. 2024). yet these evaluations can be misleading: benchmark data may already
Coorresonndenc: gk @xstvt... edu wuanwevt..c.mmm
'Our open-source Python package available httllslsllhttttt:///s httpt:http//pttt wtg/projest /ttp://bbnnnncc//eeeeeeee and the code ht https: / thub. wttptttww.......otttottttttttttt
httg://rrgeeeet/eeeebeeenne for easy and reproducible evaluation.
com/ctrl htt.::/cco.ccc..ggguaa////oeeeeo

---

Published as a conference paper at ICLR 2026
appear in pre-training (Wu et al., 2025). Recent empirical studies demonstrate widespread data contamination across standard benchmarks (Xu et al.] 2024 Cheng et al. [025] Chen et al.) 2025a
Golchin & Surdeanu 2025 Choi et al. [222 [025]Dengg et al. (024), where evaluation examples appear
in training corpora through web crawls. This contamination systematically inflates performance
metrics: models memorize specifie solutions rather than learning generalizable reasoning patterns (Shojace et al., [2025] Opus & Lawsen| 2025). As training corpora grow to web-scale, the chance that least one evaluation example appears in the training data becomes near certain, even under simple uniform sampling assumptions (we provide the full derivation and assumptions in Appendix (S). Empirical studies (Shojaee et al. 2025 Varela et al. 2025) support this as well: performance drops on decontaminated variants of existing benchmarks, suggesting that static benchmarks cannot reliably measure true reasoning ability or guarantee generalization.
Recent studies have attempted to mitigate this problem. Dynamic benchmarks such as Dy (Zhu et al. 2024a b) and ThinkBench (Huang et al. 2025) adapt task distribution over time, but provide no mathematical guarantees that each generated problem instance is well-posed (i.e., has unique or fully enumerated solution). As a result, correctness labels can be ambiguous, and evaluation pipelines must rely on heuristic matching. Algorithmic reasoning benchmarks like CLRS encode structured problems but are static in nature (Veličković et al. 2022). therefore vulnerable to contamination, and do not account for L.M-specific constraints such maximum tokens it can output. Benchmarks such as Arena (Balunović et al.) (025) attempt to mitigate this issue by using fresh olympiad questions but contamination is still likely. On the other hand, Gased evaluations such as GameArena (Hu) it al. 2025) provide interactivity, but are limited to narrow domains and lack scalable parametric difficulty. Across these settings, four gaps remain: (i) mathematically verifiable unique solutions, (ii) contamination resitaanee,,,,, (iii) token mudget-awarr evaluation, and (iv) acceptance of multiple correct answers. See Table [ for a comparison of previous benchmarks and our BEYONDBENCH.
To address the above limitations, we introduce BEYONDBENCH, an algorithmic evaluation framework for reasoning L.LMs that generates an unbounded number of novel problems from basic tasks. For each task, we define a generator that takes configurable parameters (e.g., numeric ranges, constraint sizes) and produces problem instances from a combinatorial space excceding 1 unique instances per task. This space is vastly larger than any practical training corpus, making contamination provably negligible (see Section 3.1 and Appendix [S] for the formal analysis). Moreover, for each generated problem use Boolean satisfiability (SAT) and constraint satisfaction problem (CSP) solvers (Walsh 2000) to verify that either: (i) the problem admits a unique solution or (ii) all solutions can be enumerated. This ensures that every problem is well-posed (has at least one solution) and that our scoring is precise (we know the correct answer). If multiple solutions exist, all of them are considered correct and we match against each of them, so models cannot gain credit for spurious answers.
Furthermore, we partition each task into three difficulty levels (Easy, Medium, Hard), creating a curriculum that increases in complexity. The difficulty of each task is controlled by scaling its parameters (e.g., increasing the size of combinatorial structure), so we can start with simpler subproblems and work up to provably Na-hard instances to stress-test models (Hidalgo-Herres et al., 2013). Finally, we take token budget into account: if the minimal solution length of problem exceeds the model's 's maximum output token window, that problem is excluded from evaluation. After model inference, BEYONDBENNC also checks that the response token length stays within calibrated bound for the problem size (to catch cases where models "overthink" (Chen et al. 2025c Cuadron
ct al. 2025) trivial instances).
To summarize, we make the following contributions: (I) We propose BEYONDBENCI,, an algorithmic generator of reasoning problems that automatically verifies solution uniqueness or generates complete solution sets. (2) We designed a parameterized difficulty curriculum and a token-aware evaluation protocol: tasks scale from elementary subproblems provably NP-hard instances, and evaluations respect model-specific input/outpul token budgets; (3) We conduct a large-scale systematic empirical study across 101 models to study reasoning gaps in modern LLMss/smal language models (SLMs) (Srivastava et al. 2025b Bi et al. 2025 reasoning models (LRMs).
RELATED WORK
Static Benchmarks and Their Limitations. A large body of benchmarks for reasoning in LLMs has been built upon fixed datasets such as GSM8K (Cobbe et al. 2021), MATH (Hendrycks al. 202
2

---

Published as a conference paper at ICLR 2026
and MMLU (Hendrycks et al. 202 (Xu al., 2024 and (Cheng al. 2025) document how widely used leaderboards are compromised by pretraining overlap. (Shojaee et al. (025) shows that LRMs collapse when faced with simple perturbations of known tasks. However, in response to that (Opus & Lawsen 2025) highlights that many supposed reasoning failures are actually attributable to context window limitations, ambiguous specification, tasks whose solution paths exceed the model's output length. They also argue that "emergent reasoning" often reflects benchmark biases rather than genuine generalization. Although efforts like MathArena (Balunović al. 2025) use unseen contest problems to mitigate leakage, they remain limited to static and exhaustible pools. Static evaluation also fails to capture the algorithmic hardness of problems. For instance, (Katz et al.) 2025) demonstrates that seemingly simple planning tasks, such as the Countdown game, become computationally intractable, revealing how shallow benchmarks underestimate the true difficulty of reasoning (Wu et al. 2024, Yu et al. 2025). These limitations highlight the necessity of dynamic, mathematically constrained evaluation frameworks.
Festure
CLRS
DyVal TreeEval
MathArens
CameArona
DARG
NPHlad
FCoRc
PUZZLE PLEX
BEVONDBANCH
(Ours)
Dysaaiie yoin disu
/
Stat odazion alloned
Scaline diffenity
8
V
/ Limied
Limita
Contamnatiroffr
Token-ower Number
Table Comparison with existing reasoning benchmarks. V: feature present; X: absent. Note: Unique solution means the benchmark verifies only one valid solution exists. Multi-solution allowed means multiple valid solutions are accepted and matched against the full enumerated set.
Dynamic Evaluation & Algorithmic Reasoning Benchmarks. Recent works explore dynamic benchmarking as an alternative to static datasets. Val (Zhu et al., [2024a), later extended with meta- probing in Va12 (Zhu et al. 2024b), generate math and logic problems dynamically. ThinkBench (Huag et al., 2025) evaluates models with distributionally shifted reasoning tasks, while TreeEval (Li eet al. (2025) and GameArena (Hu et al. 2025) introduce interactive live-game evaluation protocols.
DyCodeEval (Chen et al. (2025b) dynamically generates code reasoning tasks to avoid contamination, and DARG (Zhang al. 2024), using an adaptive reasoning graph framework, dynamically escalates task complexity. Similarly. (Karia et al. 2025) proposes autonomous evaluation for truth maintenance
in reasoning pipelines. Other recent work includes NPHardEval (Fan al., (024)) with complexity- class aware benchmarking, FCoReBench (Mittal et al. 2024) for first-order combinatorial reasoning with solver integration, PUZZLEPLEX (Long al. 2025) for interactive planning puzzles. These efforts highlight the growing recogmition of dynamic evaluation but remain limited by either lack of mathematical guarantees, insufficient control over solution uniqueness. or the absence of token- aware evaluation protocols. Table provides a detailed comparison of BEYONDBENCE with other evaluation benchmarks, highlighting how BEYONDBENCH differs by combining dynamic evaluation with formal solution verification. Furthermore, Algorithmic reasoning benchmarks such as CLRS (Veličković et al. 2022) and LLMThinkBench (Srivastava et al. 2025c) provide structured tests for algorithmic tasks and overthinking tendencies. However, they lack resilience contamination and are narrow in scope. BEYONDBENCH advances this line of work by unifying algorithmic, combinatorial, and mathematical reasoning tasks into dynamically generated, celliuutt-cclibbrteed suites with provable correctness.
We also discuss tool-augmented reasoning methods (PAL (Gao al. 2023
Logic-LM (Pan et al. Our work evaluates
2023), Reasoning Gym (Stojanovski et af.) 2023), and others) in Appendix C both innate and tool-augmented capabilities, showing that even with code execution and symbolic solvers, our benchmark remains challenging (Section 4.5, Appendix Q
3 BEYONDBENCH FRAMEWORK
In this section, we present the design and implementation of BEYONDBENCIL. We detail the algorith- mic foundations for contamination-resitan problem generation, introduce a token budget-aware eval- uation system, and establish formal verification methods for solution correctness. BEYONDBENCH introduces three central components: 1) dynamic problem generation with provable uniqueness guar- antees, 2) adaptive difficulty scaling based on model token budget constraints, and 3) multi-solution handling through formal verification techniques.
3

---

Published as a conference paper at ICLR 2026
3.1 ALGORITHMIC PROBLEM GENERATION AND MATHEMATICAL FOUNDATION
We develop BEYONDBENCH through algorithmic problem generation with formal mathematical guarantees. Let T denote the set of all task categories in our benchmark. For each task category T, we define a generator function G-- + x R - P, that maps a parameter space 6 (e.g., list lengths, constraint sizes) and random seed space R problem instances P,. The parameter space cardinality satisfies |0, x R. > 10 for all tasks, ensuring that Pr[G-(0,.7) = G-(02. 72)l I 10 for distinct parameter-
pairs (1) * r02, (Mitzenmacher & Upfal 2017). This large problem space makes contamination provably negligible since the probability of an exact instance collision with any training corpus of practical size [C] < 10022 remains below 10-3 (for the complete proof see Appendix (S). Our contamination resistance addresses exact instance memorization. Learning general solution strategies from similar problems constitutes genuine algorithmic learning, which our benchmark is designed to measure. For detailed proofs on the easy, medium, and hard suites see Appendix
Each generated problem p Pr undergoes validation through verification function V, S, x P. + {0, 1}. where S, is the solution space for task т, that deterministically confirms solution correctness. For deterministic tasks like arithmetic and sorting, we verify through direct computation in O(n) time, where n is the problem size (e.g.. list length). For constraint satisfaction problems like Sudoku and N-Queens, we use CSP solvers (python-constraint,, pysat) to enumerate all valid solutions. For problems admitting unique solutions, we use CSP solvers (Dechter 2003) to verify (usage of CSP detailed in Appendix D. When multplle valid solutions exist naturally (as in N- N-Queens or mode calculation), we compute the complete solution set Sp {s E S V=(s. p) = 1) through exhaustive enumeration and accept any s E Sp as correct. This prevents unfair penalization when models produce mathematically valid but non- canonical answers. The verification complexity remains polynomial (n²²) for constant S 3 despite potentially exponential solution complexity, enabling efficient correctness checking.
Iss E S, V+ (s. p) = 1}|
Concrete Parameter Space Examples. To illustrate our generation process, consider Tower of Hanoi with parameters: n [3, 12] disks. peg labels from 26 characters, and random initial configurations, yielding > 1 unique instances. For Sudoku, we vary grid density (30-50 empty cells), apply digit permutations, and shuffle cell positions, producing > 1020 instances. Arithmetic tasks sample list length n E [8, 64) and values from [-1000, 1000). creating > 10 combinations. N-Queens uses board sizes n E [4, 12] with optional initial constraints, while Boolean SAT varies variables (3-8) and clauses (5-20), each exceeding 10 unique problem instances.
Suiee
Tooks
Varistions
Problem Space
Complesity Chas 0
Example Teoks
Easy
29
> 10 per pask
Arithmatic (K), Statieiss (3), Counting (3), Externo (5), Comparion (I), Others (4)
Rbona-cifllecmsse (6), Prrmmnnnooogeennaaat (II).
Medium
49
> 16 ter
Prine/Nuater Theory (11), Comples Puttrns (12).
Algebrai: Sequesces (10)
Tower Hamin (6), N-Querrs (4), Grapls Cukoing (10).
Hard
10
> 10 per usk
XP-complete
Bootean SAT (5),
Crspaittmetic (12), Mannx
Logic Puzzles (8).
Modrier Sysens (2), Conorante Optmizztoo (5)
Total
1171
10 10
NP complete
base sob vorlations
Table 2: BEYONDBENCH Task Organization and Scale
3.2 TOKEN-AWARE EVALUATION FRAMEWORK
For each model with token budget C tokens, we dynamically calibrate problem complexity to guarantee solvability within architectural limits. Let T, denote the expected token requirement for problem p with size parameter (e.g., list length, number of disks), decomposed as (n) T Prommp (n) + Toottion (n) + Thuiter where T Thufter = 0.15 C accommodates reasoning verbosity (Han
al.) 2025 Lin et al. 2025). Prompt design is detailed in Appendix F (casy suite), (medium suite).
(hard suite).
We derive task-specific token estimation functions through empirical analysis. For arithmetic opera- tions on lists of length n with maximum element value Umax. we have Taritmmntc (n,, Umax) = ann- + 31 log mmax + where coefficients (a1, $1, 71) are calibrated across model families. For Tower of Hanoi with n disks requiring 2n moves, the solution tokens scale as T Thanoi (n) = (2" 1) a
where Omeve 12 tokens per move description. This enables precise difficulty scaling: for example,
4

---

Published as a conference paper at ICLR 2026
for models with 32K maximum output token limit evaluate up to n disks (requiring approximately (2 1) 12 = 3060 tokens excluding thinking tokens) as safe limit, while future 128K models could handle 10 (requiring (2 - 1). 12 12276 tokens), maintaining evaluation relevance as capabilities improve (Wen et al. 2025).
The framework implements dual-phase token validation protocol. During generation, we enforce the constraint (n) Σ 0.85 C' through adaptive parameter scaling. initial parameters yield excessive token requirements, we apply the reduction n' = [0.8 n iteratively until the constraint is satisfied. Post-inference, we perform actual token counting using model-specific tokenizers and
VALID
if [|] < 0.85 C
classify each response as: TokenStatus(r)
WARNING
0. C < |r| C where |r] is
OVERFLOW if
the token count of response and responses with WARNING status indicate potential truncation or excessive reasoning that may compromise solution quality (Wang et al.
2024
3.3 SOLUTION UNIQUENESS VERIFICATION AND MULTI-SOLUTION HANDLING
We use formal verification methods guarantee that every generated problem is mathematically well-posed. For each problem instance we compute the solution set cardinality through systematic constraint propagation and backtracking algorithms (Bessiere) 2006). When SSp 1, the problem admits unique solution and standard evaluation proceeds. When [Sp > 1, we enumerate all valid solutions and implement the acceptance criterion Accept(r, p) = PPrree()) Sp] where Parse extracts the model's answer using task-specific robust parsers.
For constraint satisfaction problems like Sudoku or Logic Grid Puzzles, employ are consistency algo- rithms that iteratively reduce variable domains until either unique solution emerges multiple solutions are identified (Simonis) 2005 Howell et al.) 2018). The domain reduction follows D D D:
Algorithm 1 BEYONDBENCH end-to-end Evaluation Pipeline
Input: Model M, Task suite T, Token
limit C
Output: Performance metrics &
consistent( v. d, K,, D9} where consistency checking ensures no constraint violations (here denotes a vari- able in the variable set V with domain D., and KC de- notes the For optimization problems like Matrix Chain Multi- plication, we verify optimality through dynamic pro- gramming, confirming that the claimed cost equals
3: for each task E T do
4:
0 < SclectParameters/ (T, C)
p G. (0, RandomSeed ())
problem constraints for the instance).
Sp < ComputeSolutionSSt(
if Spp 0 then
8:
continue
end if
m[1, k] = min (T) where is the number
10:
prompt + FormatPrompt((r. T)
of matrices and II denotes the set of all valid paren-
11:
r + M(prompp)
thesizations for the matrix-chain instance (Carbon-
12:
tokens + CountTokens(r..44
nel et al. 2019 Cooper & Schiex 2001 Cooper &
13:
if tokens > 0.95 C then
Zivný 2017). The complete end-to-end BEYOND-
14:
warmiwstt
BENCH evaluation pipeline is shown in Algorithm
15:
end if
I This pipeline ensures mathematical soundness
16: + Parse(r,
through three mechanisms: 1) parameter selection
17: [[r]ecorrect + E[r] + 1[s e
respects token budget constraints preventing unfair
Sp]
penalization, 2) solution set computation guarantees
18:
end for
we evaluate against all valid answers not just canoni-
cal ones, and 3) post-inference token counting detects when models approach architectural limits that may compromise reasoning quality.
3.4 DIFFICULTY SCALING AND THEORETICAL. GUARANTEES
BEYONDBENCH partitions tasks into three complexity regimes with provable difficulty scaling. The Easy Suite encompasses polynomial-time solvable problems where verification complexity is O(n)) for k 2. These include arithmetic operations, statistical measures, and comparison tasks with solution spaces of size O(n!) but efficient verification through direct computation. The Medium Suite introduces problems with exponential growth patterns such Fibonacci sequences where F,, o" with é (1 + v5)/2 (the golden ratio), geometric progressions with common ratio q growing as
5

---

Published as a conference paper at ICLR 2026
q", and number-theoretee sequences requiring primality testing with complexity O(vn) per element. Here n refers to the problem size (e.g.. sequence length. number of variables). The Hard Suite contains NP-complete problems (Fan et al. 2024 Leyton-Broww et al. 2014 Kendall et al. 2008 including Boolean Satisfiability where the search space contains 2" assignments, Graph Coloring with chromatic number computation, and N-Queens requiring exploration of n!! permutations (Yang et al. 2025b). For further details regarding extensibility to new tasks and increasing problem complexity refer Appendix T
For each difficulty level, we maintain the invariant that problem generation time remains polynomial while solution complexity may be exponential, ensuring efficient benchmark execution while testing genuine reasoning. The framework provides theoretical guarantees including contamination resistance where P(collision) < |CI/[P| < 10-3 for any feasible training corpus C (here P(collision) is the probability generated instance matches an entry of C), solution correctness through formal verification with zero false positives or negatives, and scalability where problem difficulty increases monotonically with parameter growth enabling continuous evaluation models improve. Our algorithmic task selection is motivated by the observation that true reasoning requires capabilities fundamentally challenging for current LLMs: memory management across exponential state spaces, systematic backtracking, and multi-step deduction. Tasks with known algorithms test whether models can execute procedures correctly, prerequisite for genuine reasoning. Detailed motivation and empirical evidence are provided in Appendix [
4 RESULTS AND INSIGHTS
4.1 EVALUATION SETUP
We evaluate each model 1,000 randomly generated problem instances per task variation to ensure statistical robustness, using seed 42 across all experiments for reproducibility. For proprietary models, reduce to 100 instances per task. All models the same parameters: temperature 0.1, top- 0.9, and maximum tokens allowed for that model. We evaluate 101 language models across 44 algorithmic tasks with 117 variations using our BEYONDBENCH framework. Due to space constraints, detailed experimental settings are provided in the Appendix A and B Comprehensive statistical analysis including confidence intervals, effect sizes (Cohen's 's d), ANOVA, and canaparmettric tests (Mann-Whitne U, Wilcoxon signed-rank) is provided in Appendix N Table 3 results showing accurucy, instruction- following rates (we prompt each model to respond specific output formats; if the response matches any required format, we assign instruction-following otherwise = 0), and average token usage across all suites.
presents the complete
4.2
MAIN RESULTS
Our experiments reveal substantial limitations in algorithmic reasoning, problem complexity grows. The strongest commercial models like GPT-5 (Georgiou) 2025) achieve 71.68% accuracy on hard suite, while most open-source models struggle exceed 60% performance. The performance drops dramatically with complexity: the best open- model (GPT-OSS- 120B (OpenAl et al.. 2025)) achieves 93.27% Easy tasks but only 59.41% on Hard tasks, suggesting fundamental struggles with recursive algorithmic reasoning.
SYSTEMATIC PERFORMANCE COLLAPSE BEYOND COMPLEXITY THRESHOLDS. Models exhibit sharp performance cliffs rather than gradual degradation when algorithmic complexity exceeds critical thresholds. Figure demonstrates this phenomenon across nine hardest algorithmic tasks, showing how models maintain reasonable performance until hitting complexity barriers, then collapse catastrophically. For instance, models achieve 80-90% accuracy on 4x4 Sudoku but drop to <10% on 9x9 grids. Similarly, most open-source models perform well on 3-5 disk Tower of Hanoi but fail completely 6+ disks. This cliff-like degradation pattern indicates that current models lack the systematic state management and backtracking mechanisms essential for complex algorithmic reasoning. To ensure fair comparison across tasks with different complexity scaling (Tower of Hanoi's exponential 2n Sudoku's 's linear empty cells), we normalize complexity metrics. After normalization, we observe graceful degradation tasks (arithmetic, sorting) exhibit linear accuracy decline, while sudden degradation tasks (Sudoku, N-Queens) show threshold behavior at approx- imately 0.7 x log, (context length) reasoning steps. Detailed analysis in Appendix (P) The data reveals two distinct failure patterns: i) catastrophic collapse versus ii) gradual degradation. Tasks

---

Published as a conference paper at ICLR 2026
(Phram)
Axc (%)
Tokem (Arg)
Acc (5)
Inst (8)
Tokeu (Aug)
Aee
Tokan (Aog)
Aec (7)
Inst
Tokem
(%)
(R)
(Arg)
Quivn Fumily (Qwen3)
Quard (0.68)
20.24
83.65
3162.00
19.58
92.00
15029,86
14.51
74.17
5542.79
28.21
83.54
7911886
Qwen3 (1 (1.78)
70.58
95,54
3157.20
42.28
92.00
10854.63
24.36:1.7
$0.65
6689.76
45.75
86.40
6700.53
Owen3 (4B)
82.15
91.57
3091.20
59.47
92.00
11649.83
34.25
53.85
5811.69
58.53
89.14
6850.91
Qwen3 (SB)
12.
91.58
3027.80
50.24...2
92.00
1295.54
36.7511.8
55.13
5756,42
59.79
89.57
5703.25
Qwen3 (148)
86,72 16.78.11
99.27
3607.00
19.4.1.9
92.00
6298.80
12.1
55.21
5468.71
06.25 66.28
92.16
5125.04
1wwen3 (328)
14.3
M1.08
2545.00
264
92.00
1563-77
45.211.6
55.87
546148
08.07
90.51
4631.45
12wen) (206 (X05-MOU) MOD
88, 11
14.31
2415.13
74.12.14
92.00
63301
61:58_11
57.26
5391.85
09 09.00
91.10
4712.03
Qpeen) (985-MOR--)
91.89 20.
99.40
647.17
7557
92.00
1019.40 49
45.57114
97.93
2519.82
70.35
96.
7062 19
Qpend (4B--)
85. 27
95.
2532.0
51 28
02.00
4874
$3.99
5631 AS
60. 12
90.33
4652.70 74
Qwerd (96--MOF-))
00.40 g.
95.65
2052.5 90
92.00
6063.77
41 1..
$4.61
S373.57
m2 20.00.1
91.15
4563.24
(beas Family (thema2.9)
Ownd25
7361
1185.51
0wa2.5
266.70
13.31
92.00
10.
0wer2.5
45.52 23-4
331.30
20.80.a.22
92.00
3121 82
8.12.a.
92.11
34.52
92.15
1179.50
(78)..((7))
61.62 2..7
256.90
92.00
1135.23
15.22
91.65
938.58
36.12
93.57
756.90
Qwen2.5
63.52 2.33
250.20
91.76
777.44
98.41
786.78
41.33 --.
96.00
608.14
Owen2.5 (329)
13.15
99.26
250.90
91.24
650.59
685.95
96.31
532.48
Qwen2.5 (728)
80.52.0.0
315.75
46.18
92.00
739
33.37.1.3
99.55
97.16
643.49
Qwend.5
51.1825.7
397.10
16 28
92.00
1427.29
78.22
1258 07
27.04.2.4
85.75
1007.49
Qwend.5 (7B-m)
6018323.5
411.70
92.00
2044.02
K0M
74
981.59
1309.09
12en2.5 (7288mm)
77.
429.30
92.00
730,35
12.9
7W:21
1336
87.74
856
Fanslly
Geuma
23.38 2..5 23.38
870.85
16.42.3
92:00
1220 28
0.03
663.62
13.58.a..2
78.50
921.92
Gemmu
66.62 s..
$50.00
38.17
92:00
1072.62
2.5
1048.3
38.74 -24
92:11
890.63
Gemun (128)
25.28 1.7
50L14
50.24 1..
92:00
1337.52
21,88 2.1
77.07
926.02
49.07 1..
88.64
922.49
Gemmn (278)
79.07 1..
415.95
11.4
92.00
1118.70
36.731.6
98.32
1009.75
58.13
95.93
968.14
PW Family
Plu3 muii (3 (B)
35,58
95,58
$9.40
20,03
91.92
503.10
11.38
58.55
667.56
22.35
92.36
420,15
Phi3 med (14B 4c)
43.72 2.1
99.87
189.30
21
91.64
397.91
14.72-
96.82
539.25
26.61
92.78
375
Phi3 med (148 1280)
40.5223.3
95.36
140,00
33.72.3.0
92.00
545.48
15.88
95.80
084.27
26.71
91.09
459.92
Phol mini (3-0I-)
54, 78
95.00
292
24 02
50.30
1297.2
16:0
92.90
1179.99
31 12.52
92.71
922.97
Phol-mmi neowoning
70, 10
39.50
3171,90
24
92.00
11615.07
28.96
79.55
6181.86
49 10778277
W7.04
6989 60
(3,08)
Pho42148)
78.92
97.46
60
10
92:00
686-87
28.13_2.3
97.18
1032 74
48-45
05.55
699 40
Phoi-nceaming (14B)
18
95.21
6066. 20
61.42...
92:00
$702 36
36.24..
75.08
5687 98
56.61 2203
88.06
7182: 15
Phol-aceaming
60.48 60.48
$5.80
6780.70
55.02
92:00
6361 74
28.2112.4
71.31
7002 66
50.02 22.3
84.06
6681.70
(14B)
Liame Fumily
Lioma (18)
16 u..
336.30
5,62
92.00
66699103
08:44
2028 02
69.20
30221332 132
Llana (3B)
279.70
92.00
6467 .50
4.42-2.4
86.00
2093.05
Lluna 3.1 (5B)
356.40
92.00
1590.22 22
85.64
3289.65
Lluna 3.1 (70B)
92.00
3130..0
95.80
43.25 1:.0
95.31
1521.18
Lioma (70B)
97.40
312.80
10.25
92.00
709.45
22.16-14
99.38
897.330
96.26
633.54
Liomad Scout
79:12.0.3
$2.31_1.1
86.20
731.44
27.41
49.00
1214.75
52.95
76.14
756.04
(1208--8000
Midral Funily
Misiml (7B)
37.42.a..
207
10.28
92:00
633.02
4.57
1167
14.09 14.09.2.4 2..4
93.16
670.02
Minisral (SB)
s..4
534.28
21.42 2211
92:00
1160.51
9.17.2.3
94.74
27.36 -0.2
92.15
855.66
Mistul aemo (12B)
35.68 1.0
377.80
18.33 1.7
92:00
1266.19
12.02 -19
92.11
23.01 1..
89.02
728.67
1..
140.47
12:3411.4
86.88
1.5
11.7
85.01
337.96
8x22b
50.31 1..5
79.17
296.42
20.79 $..4
92.00
536.00
18.78_1.6
95.18
762.46
29.90
85.78
531,33
Others
Smollan33B (3B)
69.25 69.2
$4.00
2985.20
9.
70.69
16320.71
19.02
75.09
6676.59
32.56.2.
76.79
$46033
Smolliz (1.70 (1.78)
16.92.44
05.99
21300
0.28
8.64
65.71
6.48
$7.89
1059
7.89
55,17
449.51
GPE (200))
87. 11.1
95.89
1185
63.5
92.00
3614.40
52.22-1
$1.25
3877.50 50
67. 27.56
89.71
2912.41
GPEOSS (1208)
95.27 20.6
99.01
821.55
76.28 go
92.00
2306 27
77.30
2981.00 AM
75.99
89.00
20022.54
OpenAT Family (Propristars)
GPT-S
97.31 2m.
92.00
1278 59
96.64
4525.03
83.57 2.0.
2.998.9
GPT-S-aiie
96.13 m..2
100.00
92.00
1640.01
01.71 1.0.4
1923.15 15
GPT-S
96.07 10.3
99.29
1377.00
91.20
2414.85
0.7
90.25
7078.11 18
93.558
3623.34 34
GPT-LI
92.23 L.44
100.00
429.35
92.00
3718.96
1.35
2603.15
GPTAI
100.00
915.70
72.67.0.4
92.00
2549.55
95.35
3069.56
2178.00
GPTAI
30.5..0.5
95.21
1361.00
55.07.0.7
92.00
1254,45
23.42...6
98664
1305.67
52.48.0.6
1239.73
GPT
88
100000
256.54
56.25.0.0
92.00
$36.46
581.04
99.59
458.45
23.52
95.57
212 32
16
92.00
522.11
0.6
99
$19.00
47. 42.14.0.7
96. 16
538.02
odlmini
94.64
100100
993
81.45
92.00
1486 30
16.5
99.89
4514.30 16
79 16.1
95. 90
3311 91
07.26.0.2
100.00
856-55
$2.27.0.3
90 40
3901 31
61 41.76...5
94.49
$965.29
80.44
94. 96
3111.23
urlenini
94.23go.4
99.64
110.22
$0.77.0
92.00
1525.04
57 $7.88
96.40
5425.99
77. 17.39
96.01
2654.05
Gomind Kumily (Proprivtury)
Gumini-2 Sygto
267.57
97.33
735.72
56.21 6
18
14.31 go
91:48
676.40
Gunini-2
39.87 sm.4
05.65
357.62
20.15
76.00
617.35
51.18 mm
20.30.mm
86.82
600.74
Gamini-2 5-daab-Be
$6.28 .m.5
387.71
20.6
92:00
11929.26
30.5746.3 s6.
97.00
38.57 28.57 c0.
96.33
6114.15
Gemini-2
o..4
313.89
67.08 .0.5
92:00
1044.79
16.7
1425.60
61.13.0.5
03.21
938.09
Gemuni dash.labhl
91.87 0.5
95.51
256.12
57.88
00.6
92.00
1043.37
30.9316.7
94.81
1840.31
$6.89 0.6
95.21
1055.60
Table 3: BEYONDBENCH Leaderboard: Three-fold evaluation results (mean across three difficulty suites (Easy. Medium, Hard). Each suite reports Accuracy (Acc), cestruction-folloviini (Inst) and average output Tokens; suffixes denote variant type: i instruct, -t thinking, -m math. The three Overall subcolumns show the arithmetic mean across Easy/Mediun/Haat for Acc, respectivel
Inst and Tokens respectively. Within each family, bold denotes the best variant and underlined denotes the second-best. Note: Models marked as x have been allowed maximum reasoning effort/thinking tokens to use. GPT-5 family results include tool-augmented evaluation (see Section 4.5).
7

---

Published as a conference paper at ICLR 2026
like Tower of Hanoi and Matrix Chain Multiplication exhibit sharp cliffs: models perform well until 5-6 disks and 7-10 matrices respectively, then collapse completely to near zero performance.
In contrast, Cryptarithmetic and Graph Coloring show counter- intuitive improvement patterns: Cryptarithmetic accuracy often increases with word length (e.g.. 78.0%%%90.4% for GPT-OSS), while Graph Coloring maintains relatively stable performance across vertex counts (e.g., 100%-+779 from 12-40
Doenk 7et
Morr
Tower Hano!
I N-Queens
Graph Coloring
15
Boolean BAT
Sudoku
Ceppaaithmetti
vertices for GPT-OSS). This suggests that complexity scaling affects different algorithmic reasoning mechanisms differently: recursive problems with exponential state explosion face hard computational limits, while constraint satisfaction problems may benefit from richer problem structure providing more optimization pathways. Interestingly, even reasoning models like GPT-5 and 03 exhibit these same differential patterns, indicating fundamental differences in how transformers handle various types of algorithmic complexity rather than uniform scaling limitations. Furthermore, most open-source models show a consistent performance ceiling around 30-35% on hard problems across 85 models.
75
Natri Chain
Moduler Systems
Constraire
Figure 1: Performance Collapse Hard Suite.
DYNAMIC EVALUATION HIGHLIGHTS LIMITS OF STATIC BENCHMARKS. Dynamic problem gen- eration shows that previous benchmark scores overestimate reasoning capabilities. Models claim- ing 90%+ performance on static benchmarks like GSM8K achieve only ~50% equivalent dynami- cally generated tasks. Our framework generates prob- lems from spaces exceeding 10t5 possibilities, mak- ing memorization very difficult. Figure shows the resulting performance distribution, reflecting innate reasoning capabilities when tested on our dynamic generation benchmark. For detailed results, see ap- pendix H (medium) and K (hard suite).
114
122
Figure Static Benchmarks vs BEYOND- BENCH Performance on Hard Suite.
180
Aocanoa Pa
Meduum
Parumettonn
Panmmttrr
Peremattrn (9)
Paramders
Paramettrr
Figure 3: Scaling Laws Across Model Families. Performance gains follow logarithmic curves with
steep early improvements tapering to marginal gains at larger scales.
4.3 PARAMETER SCALING EFFECTS
PARAMETER SCALING SHOWs PERFORMANCE SATURATION. Our overall results (Table suggest that while larger models still perform better, the rate of improvement follows logarithmic patterns with diminishing returns after a certain point. Figure iillustrates logarithmic scaling patterns actoss model families, revealing diminishing returns of parameter scaling. Within the Qwen3 family (Yang et al. 2025a), going from 0.6B to 1.7B (2.8x parameters) yields a substantial 17.5% point gain (28.2T2 28% 45.75%), but scaling further 32B parameters adds only 22.3 points across
8

---

Published as a conference paper at ICLR 2026
18x parameter increase. The step from 8B to 14B (1.75x parameters) delivers just 6.49 points, and from 14B to 32B (2.3x parameters) provides only 1.79 points. The Qwen2.5 family (Qwen et al. 2025) shows similar patterns: large early gains (9.42% + 36.12% from 0.5B to 7B) followed by slower progress (36. 12% + 53.36% from 7B to 72B). These results suggest that while scaling remains beneficial, its impact on reasoning ability diminishes progressively, with only reduced incremental benefits at larger model sizes.
QUANTIZATION HAS MINIMAL IMPACT ON REASONING. Even aggressive quantization has minimal impact (<3% % typical) on algorithmic reasoning, suggesting that reasoning is robust to reduced precision. Across quantized model variants, FP8 (Kuzmin et al. 2024) and GPTQ-Int8 quantization (Frantar et al. (2023) maintain nearly identical performance to full full-precision models.
Some quantized variants even outperform their full-
precision counterparts: Qwen3-30B-MOE- (FP8) achieves 71.04% versus 70.33% for the original. Even GPTQ-Int4 quantization, reducing model size by 4x, shows only modest drops, typically 1-3% (see Appendix M for quantized model results across easy, medium and hard tasks). This finding suggests that algorithmic thinking relies more discrete sym- bolic operations than inn-grained numerical com- putations. The results indicate that the bottleneck in reasoning performance is not computational precision but rather architectural design and the availability of appropriate architectural mechanisms enabling multi- step reasoning. Figure 4 shows the minimal performance degradation.
Greet Feelly
Oward Fariiy
19
Figure 4: Quantization Robustness Across Model Scales. Base Model (FP16) vs quan- tized variants for Qwen families.
4.4 SPECIALIZATION EFFECTS AND PERFORMANCE PATTERNS
THINKING MODELS SHOW LIMITED GAINS. Models designed for extended rea- soning show only modest improvements over their standard versions. The Bren3-4B-thinking variant achieves 60.77% versus 58.63% for the base model-a 2.14% improvement. Similarly, Qwen3-30B Owen3-30B-MOOE-tinniie MOE-thinking (Xu et al., (025) shows only 0.2% gain (69.20% Interestingly, these "thinking" models don't It consistently use more tokens, suggesting they process information differently rather than simply thinking longer. Our detailed failure mode analysis (Appendix[O) reveals that reasoning models fail later in execution (Tower of Hanoi: step 89/127) compared to vanilla models (step 23/127), but struggle with state management dur- ing extended reasoning. While vanilla models fail early without recognizing errors, reasoning models attempt self-correction but with 87.6% error introduction rate (Srivastava et al. 2025a) We varied reasoning budget from 0% to 100% across model families, finding that Hard tasks benefit most (+24-44% for weak models) while already-capable models show minimal im- provement (o3: <4% gain). Detailed case studies and reasoning budget analysis in Appendix O This indicates that current ap- proaches improving reasoning may not completely address the deeper challenges for algorithmic reasoning. True algorith- mie reasoning appears to require systematic state management, backtracking, and constraint-handling capabilities that may be fundamentally different from language modeling.
69.00%) despite being optimized for reasoning tasks.
100
80
Sa
Medium
Hard
Figure 5: Performance Distribu- tion by Model Type and Difficulty. Violin width shows model density; white lines are medians, dark lines are means. "Gap %% shows mean performance difference.
DOMAIN SPECIFIC (MATHEMATICAL) FINE-TUNINE REDUCES A ALGORITHMIC PERFORMANCE, Specialized mathematical training appears to hurt performance on algorithmic reasoning tasks. Bwven.5577BBmmnt -math (Yang et al.) 2024) achieves 48.48% overall versus 53.36% for the base model: 4.88% drop after mathematical training. This pattern appears across model sizes: Qwen2.5-7B 7-Bmmahhh (31.63%) underperforms the base model (36. 12%). Mathematical fine-tuning appears to optimize for in-domain datasets like GSMSK, MATH not innate algorithmic reasoning. Mathematical training focuses on symbolic manipulation, equation solving, and formula application, while our

---

Published as a conference paper at ICLR 2026
tasks require procedure construction, state management, and systematic search. Such training seems to emphasize symbolic and equation-base skills, which may not fully align with algorithmic tasks.
4.3 PROPRIETARY MODELS AND TUGE-AUGE REASONING
OPEN-SOURCE MODELS SHOW LARGE PERFORMANCE GAPS COMPARED TO PROPRIETARY MODELS. The performance difference between leading proprietary models (Comanici et al. 2023;
OpenAl et al., 2024) and open-source alternatives suggests that the gap cannot be explained by scale alone. For instance, GPT-5 achieves notable 12.27% % better performance hard tasks over the best open-source model (GPT-OSS- 120B). These results suggest that top-performing proprietary models may rely on innovations beyond simple parameter scaling, possibly including internal tool use or code generation to solve algorithmic problems rather than relying solely on language-basee based reasoning.
Tools
Execution
Tools Combined
TUGEENNGE APPROACHES SHOW PROMISE. GPT-5 (an agentic LLM) results suggest that combin- ing language models with computa- tional tools may be more effective than scaling language models alone.
Eacy
Arg
Fary
Hard
Fony
Ang
GPT-
83.4
66.3
97.3
GPI---midl
91.7
65.8
96.1
GPT-
58.3
38.1
71.8
48.7
96.1
K2.0
Genini
82.4
89.
72.3
74.3
Table 4: Performance across tools and difficulty suites.
The performance of tool-augmented GPT-5 models drastically drops when its tool usage is turned off. Table 4 shows a 16.81% 81,, 15.86% and 43.95% drop in accuracy for GPT-5, GPT-5-minn and GPT-5-nano respectively. This shows that these models appear to recognize when to use computa- tional tools rather than attempting pure language problem-solving. where we use external tools to extend our capabilities (Latimer et al. 2025). The efficiency gains observed suggest that recognizing problem types and selecting uppropriate solution methods may be more important than extended reasoning within the language model itself.
reasoning. This approach mirrors human
EXTENDED TOOL ANALYSIS. We conducted comprehensive per-task tool analysis across 44 tasks with three tool types (calculator, code execution, web search). Code execution provides the largest improvements (+15-55%) while web search shows negligible benefit (+0.3%), confirming contamination resistance. Importantly, smaller models struggle with tool orchestration itself (41.2% success rate for Qwen2.5-3B vs. 94.2% for GPT-5), indicating that effective tool use requires sophisticated reasoning (Srivastava al. 2026). Complete analysis in Appendix Q
To experimentally validate our theoretical contami- nation resistance guarantees, we fine-tuned models on 66K BEYONDBENCH instances using SFT and GRPO. When evaluated on a different seed (ensuring no instance overlap), models show limited Hard suite improvement ++10-15%) compared to Easy/Medium (+27-38%). demonstrating that NP-complete problems remain fundamentally challenging even after targeted training. Unlike static benchmarks where training achieves near rear-perfect scores through memorization, LEYONDBENCH maintains evaluation integrity. Full experimental details in Appendix R
CONTAMINATION RESISTANCE VALIDATION.
CONCLUSION
BEYONDBENCH'S contamination-resistaan evaluation of 101 models reveals that innate reasoning in raw language models represents a fundamental bottleneck that cannot be overcome through scaling alone. Our results reveal three major insights: 1) parameter scaling shows logarithmic returns with a hard ceiling around 30-35% on algorithmic tasks for most open-source models; 2) "thinking" models fail to meaningfully improve reasoming; and 3) mathematical fine-tuning actively degrades algorithmic performance by optimizing for the wrong computational primitives. The better performance of tool-augmented models like GPT-5 (83.57% vs. 75.99% for the best open-source alternative GPT-OSS- -000B) combined with their efficient problem-solving approaches suggests these systems succeed not through superior language- based reasoning but by recognizing when to use tools for complex reasoning. The path toward Artificial General Intelligence lies developing agentic architectures that combine language understanding with tool use: systems that, like human experts, can recognize when it's time to reason and when it's time to compute. BEYONDBENGE has revealed the core limits of raw language models and highlighted the promise of tool-an augmeneddet systems; looking forward, we see the future toward hybrid neuro-symbolic architectures and agentic LLMs with the ability to call on tools and know when to use them effectively.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the NSF #2442253, NSF NAIRR Pilot with PSC Neocortex and NCSA Delta, Cisco Research, NVIDIA, Amazon, the Commonwealth Cyber Initiative, the Amazon -aizginVViggnnia Tech Center for Efficient and Robust Machine Learning, the Sanghani Center for Al and Data Analyt- ies at Virginia Tech, and the Virginia Tech Innovation Campus. The views, findings, conclusions, and recommendations expressed in this work are those of the authors and do not necessarily reflect the opinions of the funding agencies.
ETHICS STATEMENT
BEYONDBENCH is designed to evaluate reasoning capabilities without raising ethical concerns. All generated problems are abstract mathematical or algorithmie puzzles without real-world context that could encode biases. We deliberately avoid word problems scenarios that might refer- ence people, cultures, or sensitive topics. The benchmark cannot be used to generate harmful content, as it only produces mathematical sequences, logical formulas, and algorithmic solutions. The framework is released as an open-source Python package under the MIT license (https //github. http//erl-ooee saurl-gauraa/ Baev//Byooo BeyondBench), and provide interactive leaderboard for transparent comparison of model performance. We acknowledge that improved reasoning ca- pabilities could be dual-use, potentially enabling both beneficial applications (scientific discovery, education) and concerning ones (strategic deception, adversarial planning). However, our benchmark itself poses no direct ethical risks and serves the important purpose of accurately measuring Al capabilities, which is essential for responsible AI development.
REPRODUCIBILITY STATEMENT
All code and evaluation scripts for BEYONDBENCH are publicly available. We provide a user- friendly Python package (pip install beyondbench, available athtt:s:://pyp http://ypp.rrr/// org/ for seamless integration, making evaluation simple any static benchmark. Installation requires a single command, and evaluation can be performed with minimal code. The package handles all infrastructure complexity internally, including problem generation, verification, and result aggregation. The source code and documentation are available at (htttps //github.ccom/ct//
project projectt/eyondbenccee
and interactive leaderboard is hosted at https://ctrl-gauran htt.:///eeeaarr..iii.oooo//o///o wot.:/o....///eeeeeeeeee...
The repository includes: (1) complete implementation of all 44 tasks with 117 variations, including generation algorithms, solution verifiers, and response parsers, (2) exact model configurations and inference parameters used in our experiments, and (3) seeds and parameter ranges for regenerating all problem instances, All evaluations in this paper use consistent seed value 42 for the main results, ensuring completely fair gross-modes comparison where every model sees identical problems. We also provide three-fold fold evaluation results with statistical analysis in Appendix N demonstrating low variance (mean std 2.3%) and high reproducibility.
To reproduce our results, researchers need access to GPUs with at least 80GB memory for the largest open-source models, or API access for proprietary models. The evaluation framework is model-agnostic and can be extended new models by specifying the model ID and inference engine. Comprehensive case studies demonstrating our evaluation process, including problem generation examples, model response parsing, and verification procedures, provided in Appendix O for full transparency.
REFERENCES
Mislav Balunović, Jasper Dekoninck, Ivo Petrov, Nikola Jovanović, and Martin Vechev. Matharena: Evaluating Ilms on uncontaminated math competitions. arXiv preprint arXiv:2505. 232080 2025.
Bessiere. Constraint propagation, technical report lirmm 06020 cnrs. University of Montpellier, available at: http://ww.. math. unipd. it/" frossi/bessiere, 2006.
11

---

Published as a conference paper at ICLR 2026
Zhenyu Bi, Gaurav Srivastava, Yang Li, Meng Lu, Swastik Roy, Morteza Ziyadi, and Xuan Wang. Judgeboard: Benchmarking and enhancing small language models for reasoning evaluation, 2025.
URL httpss://axxii..../p// org/ abs /et...511.... 15958
Clément Carbonnel, David A Cohen, Martin C Cooper, and Stanislay Zivný. On singleton consistency for csps defined by monotone patterns. Algorithmica, 81(4):1699-177 1727, 2019.
Simin Chen, Yiming Chen, Zexin Li, Yifan Jiang, Zhongwei Wan, Yixin He, Dezhi Ran, Tianle Gu, Haizhou Li, Tao Xie, and Baishakhi Ray. Recent advances in large langauge model benchmarks against data contamination: From static to dynamic evaluation, 2025a. BtlLhttps:// /tEtts://arxiv
brg/abs//e /202 17521
Simin Chen, Pranav Pusarla, and Baishakhi Ray. Dycodeeval: Dynamic benchmarking of reasoning capabilities in code large language models under data contamination. In Forty-second International Conference on Machine Learning, 2025b. URL https: btttps://openrevieee net Petoooomiii
3BB20qbyt2
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, Rui Wang, Zhaopeng Tu, Haitao Mi, and Dong Yu. Do not think that much for 2+3=? on the overthinking of ol-like lims, 2025c. URL https
//arxiv. wttpv//orggbbbb//////oo
abs/ 2412 21187
Yuxing Cheng, Yi Chang, and Yuan Wu. survey on data contamination for large language models. arXiv preprint arXiv: 2502::2550..44322 2025.
Hyeong Kyu Choi, Maxim Khanov, Hongxin Wei, and Yixuan Li. How contaminated is your benchmark? quantifying dataset leakage in large language models with kernel divergence, 2025. URL
org/
warg/abat abs/ /20s..252... 00678
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprini 1..iv:2110.111:: 2021. /ttt::tt////aaa //tpxvaxvvv/////////////////aa.. org/ abs/ abs//110 22100 14168 Introduces the GSM8K grade-school math dataset.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan- Jiang Jiang, Krishna Haridasan, Ahmed Omran, Nikunj Saunshi, Dara Bahri, Gaurav Mishra, Bric Chu, Toby Boyd, Brad Hekman, Aaron Parisi, Chaoyi Zhang, Kornraphop Kawintiranon, Tania Bedrax-Weiid Oliver Wang, Ya Xu, Ollie Purkiss, Uri Mendlovic, 1lai Deutel, Nam Nguyen, Adam Langley, Flip Korn. Lucia Rossazza, Alexandre Ramé, Sagar Waghmare, Helen Miller, Nathan Byrd, Ashrith Sheshan, Raia Hadsell Sangnie Bhardwaj. Pawel Janus, Tero Rissa, Dan Horgan, Sharon Silver, Ayzaan Wahid, Sergey Brin, Yves Raimond, Klemen Kloboves, Cindy Wang, Nitesh Bharadwaj Gundavarapu, Ilia Shumailov, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. URL /ttp://aaivvvv/.. wrw. wtwp/r///aaba/aaoaaae 2aa.b///2577.622
Martin Cooper and Thomas Schiex. Are consistency for soft constraints, 2001. URL https
//rrxi/.org/abs/////1111
Martin C. Cooper and Stanislav živný. The power of are consistency for esps defined by partially- ordered forbidden patterns. Logical Methods in Computer Science, Volume 13, Issue 4, December 2017. ISSN 1860-5974, doi: 10.23638/lmcs- 1 URL httt:://d/ doi org/10
23638/ LBC//MMCS-111 (4: 26) 2017
Alejandro Cuadron, Dacheng Li, Wenjie Ma, Xingyao Wang. Yichuan Wang. Siyuan Zhuang, Shu Liu, Luis Gaspar Schroeder, Tian Xia, Huanzhi Mao, Nicholas Thumiger. Aditya Desai, Ion Stoica, Ana Klimovic, Graham Neubig, and Joseph E. Gonzalez. The danger of overthinking: Examining the reasoning-action dilemma in agentic tasks, 2025. URL htR.tttps:// //atpi: www.riii.rrrooooooooo
org/ wtgg/bb///as 2502
08235
Rina Dechter. Constraint processing. Elsevier, 2003.
12

---

Published as a conference paper at ICLR 2026
Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers), pp. 8706-87 80-8777 19, Mexico City, Mexico, June 2024. Association for Computational URL http:::://aclanthoooo....
Linguistics. doi: 10. 18653/v hww.://........aooooo.n.ooo 2024. naacl ong. 4827
Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, and Yongfeng Zhang. NVHardEval: Dynamic benchmark on reasoning ability of large language models via complexity classes. Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the
Association for Computational Linguistics (Volume 1: Long Papers), pp. 4092-4114 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 00.18653/v12024. acl-long.225 URL
http:///llaaaattooo..ooooo ortp://r/g///222/ (11224.acl1oon wrtp.//.00022...111ooog 225/
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. Gptq: Accurate postinraining quantization for generative pre-trained transformers, 2023. URL
abs/
221 0. 17323
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. Pal: Program-aided language models. In International Conference on Machine Learning, pp. 17764-10799, PMLR, 2023.
Georgios P. Georgiou. Capabilities of gpt-5 across critical domains: Is it the next breakthrough?. 2025. URL
19259
Shahriar Golchin and Mihai Surdeanu. Data contamination quiz: A tool to detect and estimate contamination in large language models, 2025. URL https:: //apxiv.//rxxv.../ orq/ 22111
06233
Tingxu Han, Zhenting Wang. Chunrong Fang, Shiyu Zhao, Shiqing Ma. and Zhenyu Chen. loken-budget--nd LLM reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Lin- guisties: ACL 2025, pp. 24842 24855, Vienna, Austria, July 2025. Association for Compu- doi: 10. 10.186533v1/2025 findiiinnininnn..122444 URL https:://cclamtholooyy wtg//025....0002.. .tnees..fiiie findings acl 1274/
tational Linguistics, ISBN 979-8-997-77-7
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems, 2024. htLttt:s://axiiii org/ abs/ / 2402. 14008.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. In International Confer- ence on Learning Representations, a0 URL httph:t///// httlhhtts://pppat /ttps::////peeeveew/ net/ fwr.mmoormmii
d7KB jmI 3Gm),
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurlPS (Datasets and Benchmarks Track) / NeurIPS 2021 Proceedings, 2021b. Utt/ttps/ Introduces the MATH competition-level math benchmark
//arxiv.org/abs 2103. 03874
(12.5k problems).
Mercedes Hidalgo- Herrero, Pablo Rabanal, Ismael Rodriguez, and Fernando Rubio. Comparing problem solving strategies for np-hard optimization problems. Fundamenta Informaticae, 124 (1-2):1-25, 2013.
Ian Howell, Robert J Woodward, Berthe Y Choueiry, and Christian Bessiere. Solving sudoku with consistency: A visual and interactive approach. In ISCAI, volume 2018, pp. 5829-5831, 2018
Lanxiang Hu, Qiyu Li, Anze Xie, Nan Jiang, Ion Stoica, Haojian Jin, and Hao Zhang. Gamearena: Evaluating LLM reasoning through live computer games. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL Reg818xx0)
net/ forum?id
13

---

Published as a conference paper at ICLR 2026
Shulin Huang, Linyi Yang, Yan Song, Shuang Chen, Leyang Cui, Ziyu Wan, Qingcheng Zeng, Ying Wen, Kun Shao, Weinan Zhang, et al. Thinkbench: Dynamic out-of-distributio evaluation for robust Ilm reasoning. arXiv preprint arXiv:2502.16266 2025.
Rushang Karia, Daniel Richard Bramblett, Daksh Dobhal, and Siddharth Srivastava. Autoeval: Autonomous evaluation of LLMs for truth maintenance and reasoning tasks. In The Thirleenth International Conference on Learning Representations, 2025, URL /ttps::/openreiier
Bet//oormmidd-ivie
Michael Katz, Harsha Kokel, and Sarath Sreedharan. Seemingly simple planning problems are computationally challenging: The countdown game. arXiv preprint arXiv:2508. 02900, 2025.
Graham Kendall, Andrew Parkes, and Kristian Spoerer. A survey of np-complete puzzles. ICGA journal, 31(1):13-34 2008.
Andrey Kuzmin, Mart Van Baalen, Yuwei Ren, Markus Nagel, Jom Peters, and Tijmen Blankevoort. Fp8 guantization: The power the exponent, 2024. URL httts:://axxxi www.xxiiw..rrr.... 09225
org/ abs///b/////// 2208
Chris Latimer, Nicoló Boschi, Andrew Neeser, Chris Bartholomew, Gaurav Srivastava, Xuan Wang, and Naren Ramakrishnan. Hindsight is 20/20: Building agent memory that retains, recalls, and reflects, 2025. URL htttps://arxiv..oo
wrwii.iiivv....a wrg. abs////bs// 2512. 12818
Kevin Seyton-Brown Holger H Hoos, Frank Hutter, and Lin Xu. Understanding the empirical hardness of np-complete problems. Communications of the ACM, 57(5):98-10) 2014.
Xiang Li, Yunshi Lan, and Chao Yang. Treeeval: Benchmark-free free evaluation of large language models through tree planning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 14485-24499, 2025.
Junhong Lin, Xinyue Zeng. Jie Zhu, Song Wang. Julian Shun, Jun Wu, and Dawei Zhou. Plan and budget: Effective and efficient test-time scaling on large language model reasoning, 2025. URL
pietps:/aarae org/ abs 2505.10111 16122
Yitao Long, Yuru Jiang, Hongjun Liu, Yilun Zhao, Jingchen Sun, Yiqiu Shen, Chen Zhao, Arman Cohan, and Dennis Shasha. Puzzleplex: Benchmarking foundation models on reasoning and planning with puzzles. arXiv preprint arXiv:2510.06475. 2025.
Meng Lu, Ran Xu, Yi Fang. Wenxuan Zhang, Yue Yu, Gaurav Srivastava, Yuchen Zhuang. Mohamed Elhoseiny, Charles Fleming, Carl Yang. Zhengzhong Tu, Yang Xie, Guanghua Xiao, Hanrui Wang, Di Jin, Wenqi Shi, and Xuan Wang. Scaling agentic reinforcement learning for tool-integrated reasoning in vlms, 2025. BRRphtts:://aat http://awx....or/aab//11.1.11. 19773
Chinmay Mittal, Krishna Kartik, Parag Singla, et al. Feorebench: Can large language models solve challenging first-order combinatorial reasoning problems? arXiv preprint arXiv:2402.02611, 2024.
Michael Mitzenmacher and Eli Upfal. Probability and computing: Randomization and probabilistic techniques in algorithms and data analysis. Cambridge university press, 2017.
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro. Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage. Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann. Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson. Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-opes,, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike
14

---

Published as a conference paper at ICLR 2026
Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, tukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike. Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk. David Mély, Ashvin Nair, Reüichir Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder. Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song. Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https:/ /wwpvv../vvv//////////ee./aaaav www.aas///aeoooooooooooooe.e.ees 2303. 08774
OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman. Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Clark, Dan Cook, Marat Dukhan, Casey Dvorak, Kevin Fives, Vlad Fomenko, Timur Garipov, Kristian Georgiev, Mia Glaese, Tarun Gogineni, Adam Goucher, Lukas Gross, Katia Gil Guzman, John Hallman, Jackie Hehir, Johannes Heidecke, Alec Helyar, Haitang Hu, Romain Huet, Jacob Huh, Saachi Jain, Zach Johnson, Chris Koch, Irina Kofman, Dominik Kundel, Jason Kwon, Volodymyr Kyrylov, Elaine Ya Le. Guillaume Leclere, James Park Lennon, Scott Lessans, Mario Lezcano-Casado, Yuanzhi Li, Zhuohan Li, Ji Lin, Jordan Liss, Lily, Liu, Jiancheng Liu, Kevin Lu, Chris Lu, Zoran Martinovic, Lindsay McCallum, Josh McGrath, Scott McKinney, Aidan McLaughlin, Song Mci, Steve Mostovoy, Tong Mu, Gideon Myles, Alexander Neitz, Alex Nichol, Jakub Pachocki, Alex Paino, Dana Palmie, Ashley Pantuliano, Giambattista Parascandolo, Jongsoo Park, Leher Pathak, Carolina Paz, Ludovic Peran, Dmitry Pimenov, Michelle Pokrass, Elizabeth Prochl, Huida Qiu, Gaby Raila, Filippo Raso, Hongyu Ren, Kimmy Richardson, David Robinson, Bob Rotsted, Hadi Salman, Suvansh Sanjeev. Max Schwarzer, D. Sculley, Harshit Sikchi, Kendal Simon, Karan Singhal, Yang Song, Dane Stuckey, Zhiqing Sun, Philippe Tillet, Sam Toizer, Foivos Tsimpourlas, Nikhil Vyas, Eric Wallace, Xin Wang. Miles Wang, Olivia Watkins, Kevin Weil, Amy Wendling, Kevin Whinnery, Cedric Whitney, Hannah Wong, Lin Yang. Yu Yang, Michihiro Yasunaga, Kristen Ying, Wojeiech Zaremba, Wenting Zhan, Cyril Zhang, Brian Zhang, Eddie Zhang, and Shengjia Zhao. gpt-oss- & gpt-oss-20b model card, 2025. URL Ctthttts::/ arxi org/ abs 2508 10925.
C Opus and A Lawsen The illusion of the illusion of thinking. arXiv preprint ArXiv:2506.0922 2025.
15

---

Published as a conference paper at ICLR 2026
Liangming Pan, Alon Albalak, Xinyi Wang, and William Wang. Logic-Im:
Empowering large language models with symbolic solvers for faithful logical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp.
2023.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan
Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025. URL https: arxiv. org/ abs 2412. 15115
Parshin Shojace, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506. 06941, 2025.
Helmut Simonis. Sudoku constraint problem. In CP Workshop modeling and reformulating Constraint Satisfaction Problems, volume 12, pp. 13-27. Citeseer Sitges, Spain, 2005.
Gaurav Srivastava, Zhenyu Bi, Meng Lu, and Xuan Wang. Debate, train, evolve: Self-evolution of language model reasoning. In Proceedings of the 2025 Conference Empirical Methods in Natural Language Processing, pp. 77752-3799, 2025a.
Gaurav Srivastava, Shuxiang Cao, and Xuan Wang. Thinkslm: Towards reasoning in small language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 32600-263300 326000 2025b.
Gaurav Srivastava, Aafiy Hussain, Sriram Srinivasan, and Xuan Wang. Do lims overthink basic math reasoning? benehmarking the accuracy efffciecyy tradeoff in language models, 2025c. URL ltttp:://axxi..or///Bos
wttp:r/g/.p..roooooo 04023
Gaurav Srivastava, Aafiy Hussain, Chi Wang. Yingyan Celine Lin, and Xuan Wang. Effgen: Enabling small language models as capable autonomous agents, 2026. URL htthttps:: //arxiv
org/abs/2602 00887
Zafir Stojanovski. Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, and Andreas Köpf. Reasoning gym: Reasoning enviromments for reinforcement learning with verifiable rewards, 2025. URL. httk:/tttps///axx..or//////////
Iñaki Dellibarda Varela, Pablo Romero-Sorozabal, Eduardo Rocon, and Manuel Cebrian. Rethinking the illusion of thinking, 2025. wtt.:tttpp::/ax....orggg////
01231
Petar Veličković, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. The clrs algorithmic reasoning benchmark. In International Conference on Machine Learning, pp. 22084 22102. PMLR, 2022.
Toby Walsh. Sat v csp. In International Conference on Principles and Practice of Construint Programming, pp. 441-456. Springer, 2000.
Junlin Wang, Siddhartha Jain, Dejiao Zhang, Baishakhi Ray, Varun Kumar, and Ben Athiwaratkun. Reasoning in token economies: Budget-aware evaluation of Ilm reasoning strategies, 2024. URL
https://aaxe
06461
Hao Wen, Xinrui Wu, Yi Sun, Feifei Zhang, Liye Chen, Jie Wang, Yunxin Liu, Yunhao Liu, Ya-Qin Zhang, and Yuanchun Li. Budgetthinker: Empowering budget-aware Ilm reasoning with control tokens, 2025. URL htttt:tttt//iiii.t/t
abs///b/////
17196
Mingqi Wu, Zhihao Zhang, Qiaole Dong. Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijic Lv, Ming Zhang, et al. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. arXiv preprint arXiv:2507. 10532, 2025.
16

---

Published as a conference paper at ICLR 2026
Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. Reasoning or reciting? exploring the capabilities and limi- tations of language models through counterfactual tasks. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.). Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers), pp. 1819-1862, Mexico City, Mexico, June 2024, Association for Computational Linguis- tics. doi: 1/2.6//////22..aaalll.l. maacl-long 1027
102. URL httpp::/aacaaaao.ooooooooo//////
Cheng Xu, Shuhao Guan, Derek Greene, M Kechadi, et al. Benchmark data contamination of large language models: A survey, arXiv preprint arXiv:2406.04244, 2024.
Jin Xu, Zhifang Guo. Hangrui Hu, Yunfei Chu, Xiong Wang. Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang. Hongkun Hao, Zishan Guo, Baosong Yang, Bin Zhang, Ziyang Ma, Xipin Wei, Shuai Bai, Keqin Chen, Xuejing Liu, Peng Wang, Mingkun Yang, Dayiheng Liu, Xingzhang Ren, Bo Zheng, Rui Men, Fan Zhou, Bowen Yu, Jianxin Yang, Le Yu, Jingren Zhou, and Junyang Lin. Qwen3-omni technical report, 2025. URL https: arwii..vvvvv../.ii./aa.aaaaaaava http://v.rrgaab///////// /bt//////2222..... 17765
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024. URL BoLLhtps::0 //tpxi:///rxxv..// org/ abs 2409.1212.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang. Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang. Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL
abs 2505. 09388
Chang Yang, Ruiyu Wang, Junzhe Jiang, Qi Jiang, Qinggang Zhang, Yanchen Deng, Shuxin Li, Shuyue Hu, Bo Li, Florian T. Pokorny, Xiao Huang, and Xinrun Wang. Nondeterminis- tic polynomial-time problem challenge: An ever-scaling reasoning benchmark for llms, 2025b.
(ttlhhttts:///aaaa org/ www.a///.....
Tong Yu, Yongcheng Jing, Xikun Zhang, Wentao Jiang, Wenjie Wu, Yingjie Wang. Wenbin Hu,
Bo Du, and Dacheng Tao. Benchmarking reasoning robustness in large language models, 2025.
URL htt.ps://axxiv..ottt
/ts/////333 04550
Zhehao Zhang, Jiaao Chen, and Diyi Yang. Darg: Dynamic evaluation of large language models via adaptive reasoning graph. Advances in Neural Information Processing Systems, 13:594, 2024.
Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. In The Twelfth International Conference on Learning Representations, 2024a. URL wtw.tttstttt//tttt httpps://ppenreiiiiie net/forum?
id=qjE00925Xx
Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dyval 2: Dynamic evaluation of large language models by meta probing agents. CoRR, abs/2402. 14865, 2024b. URL https
//doi oww. 48550/ att///iiivi// 2402. 14865]
17

---

Published as a conference paper at ICLR 2026
Appendix
This appendix organized as follows. We first describe the experimental setup and implementation details used throughout our evaluation. We then provide extended related work and the motivation behind our algorithmic task selection. Next, we present detailed per-task results for each difficulty suite (Easy, Medium, Hard) followed by quantized model results. The remaining sections cover statistical analyses, failure mode comparisons between L.LMs and LRMs, normalized complexity analysis, tool-augmented evaluation, and SFT/RL training experiments. We conclude with the mathematical analysis of contamination probability, benchmark scalability, and a discussion of limitations.
Table of Contents
Experimental Setting
20
Implementation Details
20
C Extended Related Work
21
D Motivation for Algorithmic Task Selection
22
E
Easy Suite: Fundamental Arithmetic and Statistical Operations
23
B.1
Theoretical Foundation and Contamination Resistance
23
Problem Generation Framework)
23
E.3
Basic Arithmetic Operations.
23
E,4
Comparison and Classification Tasks
25
E.5
Ordering and Extrema Detection Tasks
26
E,6
Sequential Analysis Tasks
28
E.7
Statistical Measures Tasks
28
Context-Awee Evaluation and Token Estimation Framework
29
Validation Algorithms and Correctness Guarantees
31
Mathematical Rohustness and Framework Guarantees
31
E.1
Solution Uniqueness Verification and Multi-Solutie Handling
31
F
Prompts used for all Easy Suite tasks
35
Medium Suite: Complex Sequential Reasoning Tasks
39
Fibonacci Recursive Sequence Completion
Geometric and Exponential Sequence Completion
41
G.3
Prime and Number Theory Sequence Completion
43
G.4
Complex Pattern Recognition
45
G.5
Algebraic Sequence Completion
48
G.6
Context-Aware Evaluation and Token Estimation Framework
50
G.7
Solution Uniqueness Verification and Multi-Solution Handling
52
Medium Suite: Complete Results
Prompts used for all Medium Suite tasks
58
Hard Suite: Advanced Constraint Satisfaction and Combinatorial Reasoning
61
J.1
Tower of Hanoi: Recursive State Space Navigation
62
J.2
Q-eeess: Constraint Satisfaction through Backtracking
64
1.3
Graph Coloring: Chromatic Optimization and Constraint Propagation
67
1.4
Boolean Satistiailiit; Logical Reasoning Constraint Resolution
70
J.5
Sudoku Solving: Constraint Propagation in Structured Grids
73
18

---

Published as a conference paper at ICLR 2026
J.6
Cryptaritmmetic: Algebraic Constraint Resolution
77
J.7
Matrix Chain Multipiiattio: Dynamic Programming Optimization
80
J.8
Modular Systems Solver: Number- Theoretic Constraint Resolution
82
J.9
Constraint Optimization: Mblii-Ojeective Resource Allocation
85
J.10
Logic Grid Puzzles: Constraint Satisfaction through Deductive Reasoning
88
Hard Suite: Complete Results
L Prompts used for all Hard Suite tasks
98
M Quantized Models Complete Results
103
M. Easy Suite
103
M.2 Medium Suite
103
M. Hard Suite
N
Statistical Analysis
106
Detailed Three Fold Evaluation Results
109
0
Failure Mode Analysis: LLMs vs LRMs
112
0.1
Detailed Case Study 1: Tower of Hanoi (n=7) - State Management Failure
114
Detailed Case Study Sudoku (9x9)- Backtracking Errors]
116
Detailed Case Study Boolean SAT Systematic Search Success
0.4
Detailed Case Study N Queens (n=10) Constraint Checking Errors
120
0.5
Reasoning Budget Analysis
122
0.6
Reasoning Budget Across All Suites
122
Normalized Complexity Analysis
123
Augmenteddde Evaluation Analysis
R
SFT and RL Training Analysis
128
R.1
Main Insights
................................
130
R
2
Data Distribution Ablation
130
R.3
Comparison with Static Benchmark Hacking
130
R.4
Hacking Resistance Summary
131
Probability Train/ Test Contamination
Contamination Probability Finite Benchmarks
................................
S.2
Setup and notation
................................
132
S.3
Exact formula (non-um mon--nniform case
133
S.4
Exponential approximation and limit behavior
133
S.5
Expected number of overlaps and the "all items present event
134
S.6
Poisson approximation viewpoint (intuition)
134
S.7
Numerical examples
Caveats and practical remarks
T
Scalability of Benchmark Development
135
Limitations and Future Work
135
19

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL SETTING
Model Selection and Evaluation Scale. We evaluated 101 language models spanning diverse architectures and parameter scales. Open-source models (85 total) include the Qwen family (Qwen3 and Qwen2.5 variants from 0.5B to 72B parameters), Llama family (Llama-3.1, Llama-3.2. and Llama-3.3 from IB to 70B), Phi family (Phi3 and Phi4 variants from 3.8B to 14B), Gemma family (IB to 27B). and Mistral family (7B to 141B MoE). Proprictary models (16 total) include OpenAl's GPT series (GPT-4.1, GPT-4o, GPT-5 variants) and Google's Gemini family (Gemini 2.0 and 2.5 variants). For open-source models, generated 1000 problem instances per task variation to ensure statistical robustness. Due to API cost constraints, we limited proprietary model evaluation to 100 instances per task, which still provides sufficient statistical power given the deterministic nature of our problems.
Inference Contiguration. All models were evaluated using consistent inference parameters to ensure fair comparison. We used temperature 0.1 and top-p 0.9 to encourage deterministic, focused responses while allowing minimal exploration. Maximum token limit was set to 32,768 to accommo- date complex reasoning traces, though most tasks required far fewer tokens. For open- open-source models, we employed the vLLM engine for efficient parallel inference, utilizing tensor parallelism across 1-4 GPUs depending on model size, GPU memory utilization was carefully tuned per model (ranging from 0.28 to 0.96) to maximize throughput while avoiding out-of-memory errors. All experiments used seed 42 for reproducibility random problem generation.
Hardware Infrastructure. Open-source model evaluations were conducted on a cluster with NVIDIA A100 80GB GPUs, enabling efficient inference for models up 72B parameters. Smaller models (under 7B) ran on single GPUs, while larger models required 2-4 GPUs with tensor parallelism. For proprietary models, AI for Gemini models. To minimize variance from system load, all evaluations were scheduled during off- off-peak hours and run sequentially rather than in parallel when possible.
used official APIs: OpenAl's API for GPT models and Google's Vertex
B IMPLEMENTATION DETAILS
Problem Generation Pipeline. Each task implements deterministic generation algorithm that takes random seed and parameter configuration as input. The generator first samples parameters from specified ranges (e.g., list sizes from 8 to 64 elements, values from 1000 to 1000), then constructs the problem instance according to task-specific rules. For tasks admitting multiple solutions, we employ constraint satisfaction solvers (python-constrain for CSP problems, pysat for Boolean SAT) to enumerate all valid solutions. The generation process includes validation checks: ensuring problems well-posed (have at least one solution), verifying solution uniqueness completeness, and confirming the problem fits within context limits. Failed generations are logged and regenerated with different seeds.
Response Parsing Framework. We developed task-specific parsers to extract answers from model outputs, handling diverse response formats. Each parser implements multiple extraction strategies in order of strictness: (1) exact format matching using regex patterns for structured outputs like "Answer:
[1, 2, 3]", (2) fuzzy matching for variations like "The answer is 1, 2, 3" or "Solution: 2 3", (3) semantic extraction from verbose explanations, searching for number sequences or specific keywords. and (4) fallback to last numerical value for single-number answers. Parsers handle common LLM quirks including markdown formatting, LaTeX expressions, step-by-step solutions with intermediate results, and natural language descriptions of answers. When parsing fails completely, we mark the response as invalid rather than incorrect, distinguishing format failures from reasoning errors.
Token Counting and Context Management. We implement precise token counting using model- specific tokenizers: tiktoken for OpenAl models, HuggingFace AutoTokenizer for open-source models, and official tokenizer APIs for Gemini models. Before problem generation, we estimate required tokens using task-specific formulas calibrated from pilot studies. For example, Tower of Hanoi requires approximately (2" 1) x 12 tokens for n disks, while sorting tasks need roughly 3n log n tokens for lists of size n. During generation, if estimated tokens exceed 85% of the model's
20

---

Published as a conference paper at ICLR 2026
context window, we reduce problem complexity iteratively (scaling down by 80% each iteration) until it fits. Post-generation, we count actual response tokens and flag warnings when usage exceeds 95% of context capacity, indicating potential truncation or quality degradation.
Evaluation Metries and Scoring. We track three primary metrics for each model: (1) Accuracy percentage of problems solved correctly, verified against ground truth or solution set, (2) Instruction Following percentage of responses that match the required output format, independent of correctness, and (3) Token Efficiency average tokens used per response, measuring computational parsimony. For tasks with multiple valid solutions, we consider a response correct it matches any solution in the enumerated set. We compute metries separately for each difficulty suite (Easy, Medium, Hard) and report both per-task and aggregate scores. Statistical significance is assessed using bootstrap confidence intervals with 10,000 resamples.
C EXTENDED RELATED WORK
Dynamic Algorithmic Benchmarks Recent dynamic benchmarks share our motivation for contamination--rsistaall evaluation but differ in key aspects. NVHardEval (Fan et al. (024) covers 9 complexity-aware tasks with monthly instance generation but primarily uses decision problems (yes/no answers) susceptible to 50% random guessing, lacks unique solution verification, and omits token-aware scaling. FCoReBench (Mittal et al. 2024) provides 40 first-order combinatorial tasks with SymPro-LM solver integration, demonstrating strong gains through symbolic reasoning, though solver formulation remains challenging (our Appendix Q shows only 68-72% success rates with solver access). PUZZLEPLEX (Long et al. 2025) evaluates 15 puzzle types emphasizing interactive planning and game dynamics through instruction-bart and code-based paradigms, complementing BEYONDBENCH'S focus on algorithmic problem-solving with verifiable correctness. BEYOND- solution verification, multi-solution
BENCH advances beyond these works through formal enumeration, token-aware difficulty calibration (n) < 0.85 C), and comprehensive evaluation across 44 tasks with 117 variations on 101 models.
Tool-Augmented Reasoning Methods Tuol-augmented approaches attempt to enhance LLM reasoning through external computation. PAL (Gao al. 2023) generates Python code for reasoning steps executed by an interpreter, significantly reducing arithmetic errors on benchmarks like GSM8K, while Logie-LM (Pan al., 2023) employs three-stage pipeline translating natural language to symbolic representations (FOL, CSP, SAT), invoking deterministic solvers, and interpreting results with self-refinement. Reasoning Gym (Stojanovski et al. (025) provides > 100 procedurally generated reasoning environments designed primarily for Reinforcement Learning with Verifiable Rewards (RLVR), focusing on RL training infrastructure with curriculum learning and parametric difficulty control for studying reasoning development through training dynamics. Additionally, (Lu et al. 2025) scales agentic RL for tool-integrated reasoning in language models, showing that tool use can be learned through reinforcement learning scale. All these methods demonstrate value in their respective domains but face fundamental bottlenecks: PAL and Logic-LM remain constrained by formulation capability (models must correctly translate problems into executable code symbolic representations). Reasoning Gym serves RL training rather than contamination-resisaan evaluation, and agentic RL approaches focus on training rather than evaluation. Our tool analysis in Appendix Q directly evaluates both PAL-style code execution and Logic-LM-sttye symbolic solving across all 44 tasks: code execution provides 15-55% improvements yet models achieve only 67.2% on Sudoku and 58.4% on Logic Grid puzzles with full code access, while symbolic solvers (pysat, python-constren yield only 68-72% success rates on hard tasks. BEYONDBENCH advances beyond these works through: (1) provable contamination bounds (@0, R| > 10 Pr(collision] < 10-3) with formal collision analysis, (2) token-aware evaluation (T,(m) S 0.85 C") preventing unfair comparison across models, (3) complete multi-solution enumeration with CSP-base verification, and (4) extensive cross-model evaluation across 101 models for contamination-resittaa assessment, confirming that auto-formalization remains a fundamental reasoning bottleneck even with tool augmentation.
While these works share motivations with BEYONDBENCH, dynamic evaluation, contamination resistance, tool augmentation, BEYONDBENCH uniquely combines: (1) token-aware evaluation calibrating difficulty to model capacity, (2) formal CSP csp-based verification with multi-solution enu- meration, (3) provable contamination guarantees (> 10t5 instances per task, Pr[collision) < 10-3),
21

---

Published as a conference paper at ICLR 2026
(4) comprehensive assessment across 101 models (0.5B-141B parameters), and (5) unified evaluation of innate and tool-augmented reasoning. Frontier models achieve only 67-72% on hard tasks with full tool access, validating that BEYONDBENCH probes fundamental reasoning limitations persisting across augmentation strategies.
D MOTIVATION FOR ALGORITHMIC TASK SELECTION
This section provides comprehensive justification for our design choice of using algorithmically- generated tasks with known solution procedures, addressing concerns about whether such tasks adequately assess model intelligence.
Why Algorithmic Tasks Effectively Probe Reasoning Our task selection is motivated by the obser- vation that true algorithmic reasoning requires multiple cognitive capabilities that are fundamentally challenging for current language models.
Memory and State Management. Tasks like Tower of Hanoi require tracking multiple disk positions across exponentially many moves. Even with knowledge the recursive algorithm, models must maintain accurate state representations throughout 2 1 moves. Our experiments show that models achieve 80-90% accuracy for n S 5 disks but collapse to less than 10% for n 2 indicating failures in state management rather than algorithmic knowledge.
Backtracking and Search. Constraint satisfaction problems such as Sudoku, N-Queens, and Graph Coloring require systematic exploration of solution spaces with backtracking when dead-ends are encountered. The standard solving algorithms are well-documented, yet models struggle to execute them conrectly. This reveals limitations in maintaining search state and recognizing when to backtrack.
Multi-Step Deduction. Logic Grid Puzzles and Boolean SAT require chaining multiple logical inferences. Models must correctly apply rules of inference over many steps, where single error propagates and invalidates the solution.
Comparison with Competition Benchmarks Competition-level benchmarks such MathArena and Olympiad problems test whether models can discover novel solution techniques. However, they suffer from two critical limitations. First, there contamination risk: competition problems are published online and frequently appear in training data, so high scores may reflect memorization rather than reasoning. Second, there a finite problem space: once exhausted, no new problems can be generated without human expert involvement.
contrast, our benchmark tests whether models can execute known algorithms correctly, a prerequi- site for any genuine reasoning. A model that cannot reliably execute Tower of Hanoi, a problem with a simple recursive solution, cannot be expected to discover novel mathematical theorems.
Empirical Evidence: Known Algorithm, Failed Execution We conducted targeted experiments providing models with explicit algorithmic descriptions before problem instances. Table 5 shows the results.
Task
Without Algorithm
With Algorithm
Tower Hanoi (n=7)
18.4%
24.2%
+5.8%
Sudoku (9x9)
8.2%
12.1%
+3.9%
N- N-Queens (n=8)
22.3%
28.7%
+6.4%
Boolean SAT (5 vars)
38.15 18.1%
42.8%
+4.7%
Table 5: GPT-OSS- 120B performance with and without explicit algorithm descriptions.
Even when explicitly provided with correct algorithms, models show only marginal improvement, confirming that the bottleneck is execution capability, not algorithm discovery.
Scalability Guarantees Unlike static benchmarks, algorithmic generation ensures several key properties. First, there are effectively infinite problem instances with more than 15 unique problems per task. Second, the difficulty is adjustable by simply increasing challenge improved models.
22

---

Published as a conference paper at ICLR 2026
Third, verifiable ground truth is mathematically guaranteed by construction. Fourth, there is human bottleneck since new problems are generated automatically.
E EASY SUITE: FUNDAMENTAL ARITHMETIC AND STATISTICAL OPERATIONS
The Easy Suite comprises twenty-nine fundamental tasks that evaluate language models' capabilities in basic arithmetic operations, numerical comparisons, elementary statistical computations, and algorithmic reasoning over sequences. Each task generates problems dynamically using deterministic algorithms with guaranteed unique solutions, ensuring complete contamination resistance while maintaining mathematical rigor. The suite operates on the principle that for each problem class Peasy we can generate instances p E Peacy with polynomial verification complexity O(n*) for constant k < 2, where represents the input size parameter.
E. THEORETICAL FOUNDATION AND CONTAMINATION RESISTANCE
Let us formally define the contamination resistance property for the Easy Suite. For problem generator Geasy ④ x R + Peasy with parameter space & and random seed space R, we ensure that ( R| >> (D) where DD represents any feasible training dataset. The generator exhibits the following sensitivity property:
vO,0 E E0,1, E E (01,7)) (02, -) <E
(I)
where 10 for our implementation parameters. This ensures that the probability of generating duplicate problems across different evaluation instances remains negligible.
Each task employs a validation function Veay S Peasy f0. 1} that deterministically verifies solution correctness in polynomial time. The validation complexity for all Easy Suite tasks satisfies T T(Vensy) O( log n) in the worst case, where denotes the input list size.
E.2 PROBLEM GENERATION FRAMEWORK
The Easy Suite utilizes unified generation framework based on uniform random sampling without replacement. For given range aa, b] C Z and list size k E N, the generator produces sequence {21, I2,..,,
. Hx) where each ø; is drawn uniformly from [a, b] without replacement, ensuring J. # 2y for i i
The parameter space for Easy Suite tasks is defined as:
+ casy = [kk, a) & e 100), a, [-108, 1-1,,, - a 2 k}
(2)
This parameter space yields problem space cardinality of approximately 10a5 unique problem instances per task, making exhaustive memorization computationally infeasible.
E.3 BASIC ARITHMETIC OPERATIONS
This category encompasses fundamental mathematical operations that form the foundation of numeri- cal computation. These tasks evaluate models' capabilities in performing elementary arithmetic with integer and rational number systems.
E.3.1 ADDITION TASK
The addition task evaluates the fundamental capability of summing sequence of integers. Given an input list L = {I1, T2, ..,,....2. ) where x; Z, the task requires computing:
5
Ti
(3)
im
23

---

Published as a conference paper at ICLR 2026
The solution uniqueness follows directly from the associativity and commutativity of addition integers. The verification function computes the sum in (n) time with guaranteed correctness through integer arithmetic properties.
E.3.2 SUBTRACTION TASK
The subtraction task operates ordered pairs (a, b) Z and requires computing the difference d = b - a. The mathematical formulation is:
d = where b E [mmin: (max
(4)
The uniqueness of the solution follows from the well-defined nature of integer subtraction. The task specifically evaluates understanding of operand ordering, as a & a unless a = b, testing the model's comprehension of non-commutative operations.
E.3.3 MULTIPLICATION TASK
For sequence L {01, 22, ..... }. J. the multiplication task computes the product:
TG
(5)
The challenge this task arises from the exponential growth the product magnitude. For n numbers each of magnitude the product reach O(m"), requiring careful handling of large integers. We constrain the list size to ensure |P| < 10 to maintain computational feasibility while preserving task difficulty.
E.3.4 DIVISION TASK
The division task requires computing the quotient of integers with Koating-point precision. Given (a, b) E 2² with f O, we compute:
q
(6)
To ensure non-zero divisors, the generation algorithm employs rejection sampling: Uniform( ((uiin [max] \ {0)). The verification allows for floating-point precision tolerance 10 accounting for potential rounding in model responses while maintaining mathematical validity.
E.3.5 ABSOLUTE. DIFFERENCE TASK
For an ordered pair (a, b) E 22, the absolute difference is computed as:
a if a 8 = lb a-b i
This task evaluates understanding of the absolute value function and its properties. The solution uniqueness follows from the well-defined nature of the absolute value operation over integers.
E.3.6 ALTERNATING SUM TASK
The alternating sum applies different signs to elements based on their position. For L {21, (21, .2,...,,n we compute:
A =
Σ 1)
24
22
D2 +
E3
IA
+
(8)
i=1
24

---

Published as a conference paper at ICLR 2026
This task evaluates understanding of positional arithmetic and alternating patterns in mathematical sequences.
E.3.7 SUM OF DIGITS TASK
For sequence of integers L {01, ..,,,,,,, In the total digit sum
Dum
d
im1
dedigiis (mn:)
where digits(æ;)) extracts individual digits from the decimal representation 24. This task evaluates digit-level arithmetic and multi-level summation capabilities.
E.4
COMPARISON AND CLASSIFICATION TASKS
This category evaluates models' abilities to perform relational reasoning, classification based on numerical properties, and counting operations with specific criteria.
E.4.1
NUMERICAL COMPARISON TASK
The comparison task evaluates relational reasoning over integer pairs. Given (a,b) Z², the model must determine the relation R E {, such that aRb holds.
The generation ensures balanced distribution across relation types through stratified sampling:
Pr[R=<] Pr(p Pr[R =
(10)
This balanced distribution prevents models from exploiting statistical biases and ensures comprehen- sive evaluation of all comparison operators.
E.4.2 ODD AND EVEN COUNTING TASKS
For a sequence = {01, T2..... Fn the odd count task computes:
Cass = Hrg E C: C: = (mod 2)H
(11)
Similarly, the even count task computes:
Ceven 1xx E 2;
0 (mod 2)}
(12)
The correctness verification leverages the partition property: Cold Ceven n, providing additional consistency check beyond individual count verification.
E.4.3
COUNT NEGATIVE NUMBERS TASK
For a sequence = {#1, .2......7. where x; E Z, the negative count is computed as:
Cneg = |x, E C: < 031 = Σ 1 1,
(13)
1=1
where 1, denotes the indicator function. This task evaluates understanding of sign- based classifi- cation and counting operations.
25

---

Published as a conference paper at ICLR 2026
E.4.4 COUNT UNIQUE ELEMENTS TASK
The uniqueness counting task determines the cardinality of distinct elements a sequence. For C (1), :., we compute:
Cunique = (x E L)] = |distinct(C)
(14)
This task evaluates set-theoretic reasoning and understanding of element distinctness. The generation algorithm controls the degree of repetition to ensure meaningful evaluation across different uniqueness ratios.
E.4.5 COUNT PERFECT SQUARES TASK
The perfect square counting task identifies numbers that are exact squares of integers. For (01,-2 we compute:
Osquare 1rz E L Ek E N. = k²}
(15)
The verification uses the property that x a perfect square if and only if [Vø = x for non-negative integers
E.4.6 COUNT PALINDROMIC NUMBERS TASK
For a sequence of integers L = (I1, (2,,,2,... ....... Fn.....,,,.. the palindromic count is:
Cpat [rg C str(z;) reverse(str(z:)))
(16)
This task evaluates string manipulation capabilities and pattern recognition in numerical representa- tions. The verification requires digit-level comparison after string conversion.
E.4.7
COUNT MULTIPLES OF TASK
For fixed integer K > 1 and sequence C = {111, 22..... J. the multiple count is:
Ck = Hag E C: x 0 (mod K)}
Σ 1
(17)
This task evaluates modular arithmetic understanding and divisibility reasoning. Common values include K E {2, 3, 5) for practical evaluation scenarios.
E.5 ORDERING AND EXTREMA DETECTION TASKS
This category encompasses tasks that require understanding of element ordering, extrema identifica- tion, and positional reasoning within sequences.
E.5.1
SORTING TASK
The sorting task requires arranging sequence L = (21, ,,, in ascending order. We define the sorted sequence L' = {a/, .e..,, 2,) such that:
x < S W
(18)
The verification function checks both the ordering constraint and the multiset equality (C) = (C'} ensuring no elements added, removed, modified during sorting.
26

---

Published as a conference paper at ICLR 2026
E.5.2
FINDING MAXIMUM AND MINIMUM TASKS
For non-empty sequence { 22,............. ,n the maximum finding task computes:
Tmax max =
fo E :::::: Vy E L, 2
(19)
The minimum finding task similarly computes:
Tmin
min {a e Vy .E
(20)
Both operations have unique solutions for any finite sequence of totally ordered elements, as guaran- teed by the well-ordering principle for finite subsets of integers.
E.5.3 SECOND MAXIMUM TASK
The second maximum task evaluates the capability to identify the second largest element in sequence. For sequence L (I1, .2, .nn with at least two distinct elements, we define the second maximum
Tnd--max max(f) \ (Imma)) max C Tmax
(21)
The generation algorithm ensures at least two distinct values exist by enforcing the constraint [distinct(C) 2. The verification complexity is O(n) through two-pass algorithm that identifies both maximum and second maximum values.
E.5.4 RANGE CALCULATION TASK
The range task computes the difference between maximum and minimum values in a sequence. For {z1, ,,, Jn... the range is defined as:
R(C) Tmax Tmin max 24
min Tj xE(I.)] jE(I,m)
(22)
This task evaluates understanding of the fundamental statistical measure of dispersion. The range satisfics the mathematical property R(C) 2 with equality and only all elements are identical.
E.5.5 INDEX OF MAXIMUM TASK
This task requires finding the zero-based index of the first occurrence of the maximum element. Given sequence L = {I1, 22,... In. ). we define:
idxmax (L) min(i [0, - 1] 3i+1
Tmax
(23)
The task evaluates positional reasoning capabilities and understanding of indexing conventions. The verification ensures both correctness of the maximum value and its position within the sequence.
E.5.6
SUM OF INDICES OF MAXIMUM ELEMENTS TASK
When the maximum value appears at multiple positions, this task computes the sum of all such indices. For L = (I1, 22,,.., ), we define:
Smax-idx = Σ 1 10
Σ
(24)
where indices are one-based. This task evaluates both maximum identification and positional arithmetic capabilities.
27

---

Published as a conference paper at ICLR 2026
E.6 SEQUENTIAL ANALYSIS TASKS
This category focuses on tasks that analyze relationships between consecutive elements and detect patterns within sequences.
E.6.1
MAXIMUM DIFFERENCE BETWEEN ADIACENT ELEMENTS TASK
This task computes the maximum absolute difference between consecutive elements. For a sequence [ 11 ,2....,, with 2 define:
max xE[La=-]]
Mi+1 - m3g
(25)
The task evaluates understanding of sequential relationships and local variation measures. The verification complexity is O(n) through single pass over adjacent pairs.
E.6.2 COUNT ELEMENTS GREATER THAN PREVIOUS TASK
This task counts elements that exceed their immediate predecessor. For L = (2,,,, In we compute:
Cine = E [2, x4 > x--131
1222
(26)
This task evaluates sequential comparison capabilities and understanding of monotonicity properties in discrete sequences.
E.6.3 LOCAL MAXIMA COUNT TASK
A local maximum is an element strictly greater than both immediate neighbors. For 4
{21, F2...,...., with n 2 3, we compute:
Ciecal (i E [2.n 1:nn-1:::33 xz-1 A x;-131
(27)
This task evaluates local extrema identification and requires understanding of neighborhood-baser comparisons in discrete sequences.
E.6.4 LONGEST INCREASING SUBSEQUENCE LENGTH TASK
This task computes the length of the longest increasing subsequence (LIS). For L. (21, }nn we define:
LIS(C) = max[[S] S [1, < <x)
(28)
where S represents the index set of subsequence. This task bridges elementary and intermediate algorithmic reasoning, with optimal solution complexity (n log n) using dynamic programming with binary search techniques.
E.7 STATISTICAL MEASURES TASKS
This category encompasses fundamental statistical computations that require understanding of central tendency, dispersion, and distributional properties.
E.7.1
MEAN CALCULATION TASK
The arithmetic mean of sequence L {21, 22,,, Fn) is computed
28

---

Published as a conference paper at ICLR 2026
(29)
7
For integer inputs, the mean may rational, requiring floating-pon poont representation. The verification employs tolerance 10-6 to account for floating-poin point arithmetic while maintaining mathematical soundness.
E.7.2 MEDIAN CALCULATION TASK
The median computation depends on the parity of the sequence length. For sorted sequence C' = 22 "
fm/m21
if n = (mod 2)
median(C) =
#2/2t0//221
(30)
if (mod 2)
The uniqueness of the median follows from the uniqueness of the sorted sequence and the deterministic selection of middle elements.
E.7.3 MODE CAL.CULATION TASK
The mode identification task requires finding the most frequently occurring values a sequence. We define the frequency function f C N where ((x) 1i I; =)). The mode is:
mode(4) (C) fur L: f(x) max fw)) yEL
(31)
To ensure meaningful evaluation, the generation algorithm guarantees at least one element appears with frequency > 2 through controlled repetition injection. The verification checks both the frequency maximality and completeness of the mode set.
E.8 CONTEXT-AWARE EVALUATION AND TOKEN ESTIMATION FRAMEWORK
We implement a mathematically rigorous framework for context-aware evaluation that dynamically scales problem difficulty according to each model's context window constraints. This approach prevents unfair penalization of models due to context limitations while maintaining evaluation integrity.
E.8.1 MATHEMATICAL TOKEN ESTIMATION MODEL
For each task category, we establish precise token estimation functions that predict the expected response length based on problem parameters. Let Gmodel denote the model's context window size, T prompe the prompt token count, and Teespouse the expected response tokens. We define the safety constraint:
Tprcmpt + Tresponse + Tuffer S Cmodel
(32)
where Touffer accounts for model specific reasoning overhead. The response length estimation functions are:
T Taittmmetic (n, M) ann + B1 log M + 7 Teeiing (n, M) = azn log + 82n + 72
(33)
(34)
T statistical (n,, M) azn 83 log M + 73
(35)
T Touniing (n) = aa log10 n 7/
(36)
T Teeguemiaa
M as log M + 75
(37)
29

---

Published as a conference paper at ICLR 2026
The coefficients {a,, 3r, ) are empirically calibrated using extensive model response analysis across different model families. For conservative estimation, we enforce Tutter 0.15 Cmodel to accommodate models that exhibit verbose reasoning patterns overthinking behaviors.
E.8.2 DYNAMIC PROBLEM SCALING ALGORITHM
Given a target model with context window Cmodel, we implement the following scaling procedure:
Algorithm 2 Context-Aware Problem Generation
1: Input: Task type T, model context size Cmodel
2: Initialize Mmax 100, Mmax < 10t
3: Compute Tprorpp f- TokenCount
template)
4: Set Tavailable f- Cmodel
T promps butfer
5: while T, (Tomax Mmxx) available do
6: if 7mmax > then
7:
T'max + 0.8 Rmmxx]
8:
else
9:
Mmax f- 00.8 M Mmax]
10:
end
11:
end while
12:
Generate problem instance with parameters (Aimacy Mmax)
This algorithm ensures that every generated problem instance respects the model's 's context constraints while maximizing problem complexity within those bounds.
E.8.3
POST- FOST-GENERATION TOKEN VERIFICATION
After model response generation, we implement a veriffiiiiii token counting system detect potential context overflow or overthinking behaviors:
VALID
if [T(r) 0.95 Cuudel
Verification(r, Cmoddl)
WARNING
0.95 Crodee < <[((r) S Cmadel
(38)
OVERFLOW
[T(r) Cinddel
where T(r) represents the token count of response r computed using model-speeific tokenizers:
Transformer-based models: We employ the HuggingFace transformers library tokenizer
corresponding to each model's architecture.
GPT models: We utilize OpenAl's tiktoken library for precise token counting matching their internal tokenization.
The dual-tokenizer approach ensures accuracy across different model families while detecting edge cases where models approach or exceed their context limits due to random generation artifacts or excessive reasoning verbosity.
E.8.4 MATHEMATICAL GUARANTEES AND IMPLEMENTATION ROBUSTNESS
Our framework provides the following theoretical guarantees:
1. Context Safety: With probability 0.99, generated problems satisfy Tiota Cmodel
2. Fairness: Models are never penalized for responses that exceed estimated bounds by less than Thuffer
3. Scalability: Problem complexity scales monotonically with available context budget
4. Precision: Token estimation remains below +55% for 95% of generated instances
30

---

Published as a conference paper at ICLR 2026
This mathematically grounded approach eliminates context-related evaluation artifacts while maintain- ing the integrity of difficulty scaling, ensuring that performance differences reflect genuine reasoning capabilities rather than implementation constraints.
E.9 VALIDATION ALGORITHMS AND CORRECTNESS GUARANTEES
Each task employs a deterministic validation algorithm with provable correctness. The validation pipeline follows three-stage process:
First, the syntactic validation ensures the response follows the required format specification. Second, the semantic validation verifies that the parsed answer satisfies the mathematical constraints of the problem. Third, the numerical validation confirms the answer matches the ground truth within specified tolerances.
The validation complexity for all Easy Suite tasks remains polynomial, with the most complex operation (sorting validation) requiring On log n) comparisons. This ensures efficient verification even for large input sizes while maintaining complete correctness guarantees through rigorous mathematical foundations.
E.10 MATHEMATICAL ROBUSTNESS AND FRAMEWORK GUARANTEES
The Easy Suite establishes a comprehensive mathematical foundation for contamination-rest resissaan evaluation through several key theoretical guarantees. The parameter space cardinality exceeds 10ts for each task, ensuring that the probability of generating identical instances across different evaluation sessions remains negligible under any reasonable computational budget. This mathematical guarantee holds through the application of combinatorial principles and the well-ordering properties of integer sequences.
Each task within the suite maintains strict mathematical rell-posedness, meaning that every generated instance admits exactly one correct solution or a completely enumerable solution set. This property eliminates ambiguity in evaluation and ensures that model performance reflects genuine reasoning capability rather than exploitation specification ambiguities. The verification algorithms employ deterministic mathematical operations with provable correctness, eliminating any possibility of false positives or negatives in the assessment process.
The computational complexity bounds established for each task category ensure that the framework scales appropriately with problem size while maintaining practical feasibility. The polynomial upper bounds both generation and verification complexity guarantee that the evaluation framework remains computationally tractable even as problem instances scale to challenge more capable models.
Furthermore, the mathematical formulations presented here demonstrate that the Easy Suite pro- vides a complete coverage of fundamental algorithmic reasoning patterns. The tasks span multiple mathematical domains including arithmetic operations. order theory, statistical measures, and se- quential analysis, creating comprehensive assessment framework that captures essential reasoning capabilities without redundancy or gaps in coverage.
E.11 SOLUTION UNIQUENESS VERIFICATION AND MULTI-SOLLUTIO HANDLING
A critical component of our framework ensures mathematical fairness by rigorously handling cases where problems may admit multiple valid solutions. We implement a comprehensive solution verification system that prevents unfair penalization of models for producing mathematically correct non-canonical answers.
E.11.1
UNIQUE SOLUTION GUARANTEE PROTOCOL
For each generated problem instance P Peasy. we employ deterministic verification procedure to establish solution uniqueness. Let S(p) denote the complete solution set for problem p. We define the uniqueness predicate:
31

---

Published as a conference paper at ICLR 2026
TRUE
Sp() if Sp))
Unique(p)
FALSE
For tasks with inherently unique solutions (arithmetic operations, extrema finding, statistical mea- sures), the generation algorithm guarantees Unique(p) TRUE by construction. However, certain tasks may admit multiple valid solutions under specific parameter configurations.
E.11. COMPLETE SOLUTION ENUMERATION FOR MULTI-SOLLUTION CASES
When (nique(p) FALSE, we implement exhaustive solution enumeration ensure comprehensive evaluation fairness. The most relevant cases include:
1. Mode calculation:
Multiple values may achieve maximum frequency
2. Sorting with equal elements: Multiple valid orderings for identical values
Index-based operations: Multiple indices may correspond identical maximum values
For these cases, we compute the complete solution set S(p) = {81, .2, 8k) using deterministic enumeration algorithms. The verification function becomes:
CORRECT
if Parse(r) E S(p)
Verify(r,p)
INCORRECT
if Parse(r) & S(p) if Parse(r) =1
(40)
INVALID
where Parse(r) extracts the model's answer from response r, and 1 indicates parsing failure.
E.113 MATHEMATICAL COMPLETENESS VERIFICATION
To ensure valid solutions omitted, employ mathematical completeness checks specific each task category:
Mode completeness: :
vs E S(p), f()) = max (xx #
(41)
Sorting completeness
Vs E 8(p), sorted(s) A permutation(s, L)
(42)
Index completeness :
Vs E S(p), [[s] = max(C)
(43)
This mathematical framework guarantees that model responses evaluated against the complete solution space, eliminating any possibility of unfair penalization due to solution non-uning unnuuiqueeess.
1.1 COMPUTATIONAL COMPLEXITY OF SOLUTION ENUMERATION
The solution enumeration process maintains polynomial complexity bounds:
Smidte (p) S n (at most n distinct values)
(44)
sorting (p)
I n.!!
(permutations of equal elements)
(45)
SSindex (p) S (at most positions)
(46)
where n, represents the frequency of value v the input sequence. Even in worst-case scenarios, the enumeration complexity remains tractable, ensuring efficient evaluation without compromising mathematical rigor.
This comprehensive approach to solution handling demonstrates that evaluation framework maintains both mathematical correctness and practical faimesss, ensuring that model performance reflects genuine reasoning capability rather than arbitrary solution selection preferences.
32

---

Published as a conference paper at ICLR 2026
Table 6: Easy Suite: Mathematical Formulations and Complexity Analysis (Part I)
Task
Inpul Space
Mathematical Operation
Solution Space
Verification
Complexity
Basic Arihmetic Operations
Addition
C
8 2in
c
8 Ein
O[n)
Subtraction
(o.4) ez
d= b-a P II: C
dez 2
d+u
O(1)
Mulhiplication
4- (R
Pc
P III. 21
O(n)
Division
(a.b) c2
9 C
l0-b c/c
O(1)
Abselute Difference
(o.4) ez?
6= [b-al C ( ) C A Ein( 1)6tls
6 %
6= 1-16//200
O(1)
Alternating Sum
AC
A E..(( 1)ft',,
O(n)
Digits
C= (a
Ei Eocrents df
No
Digit extraction summation
OfgllggM) M)
Comparizon and Classifcatiin
Compatison
C2
R [s.m))))aa))
R [K,m,>]
bj sgnt R)
O[1
Odd Count
b= (elnnnaa)) C2 Cas = (a 12 =1 (mod 2)H
Cuu [0,n]
Σi (r, und 2)
066
Even Count
== (a C2 Gon i (and 2)
Caa 6 [0,x]
R- Ee mod
O(s)
Count Negative
2 [a] C2 Cus [[ 0]]
Gug [0,w]
E 1,,c0
O[n]
Count Unique
L - [a]) C2 Cuu [distinct(22]
Gu4 C [1,,]]
Set cardinality computation
O(n]
Count Peffect Squares
c= (e C G,= [(z, :3 2, H
Gy [0.w]]
/ =
O(s)
Count Palindromic
<= (R...... C Gu (1t, palindommeaa))
Gj e [0.x]]
String revenall chock
O(niogg.V)
Count Multippes
(= (n) C2 Cx Hin i (mnd K))
Ck e [0.n]
Modular arithmetic chock
O(x)
Table 7: Easy Suite: Mathematical Formulations and Complexity Analysis (Part II)
Tusk
Input Spuce
Mathematical Operation
Solution Space
Verilication
Complexity
Ordering and Estreion Detection
Sorting
C- (Ma C C Ci S
E E 72"
W: S HiLs (CCC) (C'
Oje
Find Maximum
L (.)......) 2 Zu.. X
Toos
O(e)
Find Minimum
0= (fi....aa) CZ Ruin
Tuin
E
O(s)
Second Maximum
3a) CZ Fas max Faas)
Viad E
Two-pass extrems detection
((s)
Rauge Calculation
Au) 2
Pasns Pain
R N4
R maxe(C) miln (')
O(s)
Index Maximum
tm 2nd 2 idx min(f #ane)
[0,
Position value verification
O(s)
Sum Max Indices
C - (ri... Nn) C2 Su Eni
Sax 24
Positional arishmetic
O(x)
Sequential Analysis
Max Adjacemi Duft
L - Xa) A... max, [Riqs =z)
A... 6 No
Pairwise differeace cocsputation
O(a)
Count Incrossing
C = (1
CZ Bailiin Seal
Cn [0,
Sequeriaa compurison
Oje)
Local Maxima Count
An) 2 Cus [[r ALL sall
Cuca [l,n 2]
Neighbochood comparison
O(e)
LIS Leagth
(4 Z wax((8) sa S.a, (.)
[L.n]
Dynzanie programming
()glog))
Meunores
Mean
C= (n....))))) CZ p= En
a Q
In-p- Σxsi <
O(x)
odd
Mediam
C= (m....))))) CZ m=
m e Q
Sont C, select middle
O(glogs)
Mode
C= (mponad) CZ mode for (or) max, fly)
mode S C
Vir mode: f(r) 2 Fly)
O(s)
Table 8: Easy Suite: Problem Generation Parameters and Contamination Resistance
Task Category
Parameter Space
Problem Space Size
Solution Uniqueness
Context Bound
List- Based Operations (25 tasks)
Arithmetic (4 tasks), Ordering (7 tasks), Stotistical (3 tasks), Sequential (4 tasks), Clasrification (7 tasks)
Input Generation
I [2, 100)
lo, 1-00% 10%]
(~att)) k! 10a5
Deterministic
O log M)
Puir- Based Openations (4 tacks)
Subtraction, Division, Absolute Difference, Comparison
Input Generation
(o, b) E [Romtn Amaz)" 1z Omin, Grax E [- 100 10
(Amax Ammin + 1)2 23 10
Deterministic
O(log M)
33

---

Published as a conference paper at ICLR 2026
Table 9: Easy Suite: Validation Properties and Error Tolerances
Task Type
Output Format
Validation Method
Error Tolerance
Integer Operations
integer
Exact match
(Addition. Subtraction, Multiplication, Counts)
result ground truth
Rational Operations
decimal
Flosting-poin comparison
10
(Division)
|result ground truth)
Statistical Measures
number
Precision-based comparison
10
(Meun, Median)
|result
ground trath] <
Set Operations (Sorting, Mode)
list/set
Multiset equality
N/A
{result} = (ground truth
Categorical
category
Exact category match
NIA
(Comparison)
result E (<, m.>)
34

---

Published as a conference paper at ICLR 2026
F PROMPTS USED FOR ALL EASY SUITE TASKS
This section describes the prompts developed for each of the easy suite tasks. Each prompt begins with an instruction describing the task. The prompt then provides the input which can be list or a pair of numbers. The model provided information regarding the format the expected answer. All final answers must be enclosed in the format \boxed(): [. The answer is then extracted from within the \boxed } brackets using a regular expression.
Prompt Template for Absolute Difference Task
Find the absolute difference between the following list of numbers:
[data_point]
Provide the result. Your final answer must be in the format (anseweasseer at the end.
Prompt Template for Comparison Task
Compare the following two numbers and determine their relationship:
Number 1: {numl)
Number 2: (num2)
Is Number greater than, less than, or equal to Number 2? Your final answer must be in the format \boxed (relation) at the end, where 'relation' is one of: 'greater than', 'less than', 'equal to".
Prompt Template for Division Task
Divide (numerator) by [denominator).
Provide the answer floating point number. Your final answer must. be in the format \boxed (answer) at the end.
Prompt Template for Even Count Task
Count the even numbers from the following list of numbers: (input_list)
Provide the final count of even numbers. Your final answer must be in the format \boxed (anwer)) at the end.
Prompt Template for Find Maximum Task
Find the maximum number from the given list of numbers. List (input_list).
Your final answer must be in the format \boxed maximum) at the end of your response.
35

---

Published as a conference paper at ICLR 2026
Prompt Template for Find Minimum Task
Find the minimum number from the given list of numbers. List (input_list).
Your final answer must the format \boxed (minimum) at the end of your response.
Prompt Template for Mean Task
Calculate the mean (average) of the following list of numbers:
(input_list)
The mean is the sum of all numbers divided by the count of numbers. Calculate the exact mean value. Your final answer must be in the format \boxed mmean va value) at the end.
Prompt Template for Median Task
Find the median value of the following list of numbers: (input_list]
The median is the middle value when the list is sorted. If there is an even number of elements, the median is the average of the two middle values. Your final answer must be in the format \boxed value) at the end.
Prompt Template for Mode Task
Find the mode (s) of the follewing list numbers: (input_list)
The mode is the value that appears most frequently. If multiple values appear with the same highest frequency, return all of them. Your final answer must be in the format (mode (8) at the end. If there are multiple modes, 1ist them separated by commas.
Prompt Template for Multiplication Task
Multiply the following list of numbers: (dita_point)
Provide the product. Your final answer must be in the format (boxed(answer) at the end.
Prompt Template for Odd Count Task
Count the odd numbers from the following list of numbers: (input_list)
Provide the final count of odd numbers. Your final answer must be in the format \boxed (answer) at the end.
36

---

Published as a conference paper at ICLR 2026
Prompt Template for Sorting Task
Sort the following list of numbers in ascending order: (input_list)
Provide the sorted list. Your final answer must be the format \boteddsorted list) at the end.
Prompt Template for Subtraction Task
Can you subtract THirst_number" from feccond_numberr and provide your final answer in \boxed (answer) format the end of your response.
Prompt Template for Sum Task
Add the following list of numbers: (input_list]
Provide the sum. Your final answer must be in the format (answer)) at the end.
Prompt Template for Second Maximum Task
Find the second maximum number from the given list of numbers. List = (input_list).
Your final answer must be in the format Aboxed aeecond_maxinne at the end of your response.
Prompt Template for Range Task
Calculate the range (difference between maximum and minimum) of the following list of numbers: (input_list).
Your final answer must be in the format \boxed (range) at the end of your response.
Prompt Template for Index of Maximum Task
Find the index (0-based position) of the maximum element in the list soppt_lisst_. If there are multiple maximum elements, return the index the first occurrence.
Your final answer must the format \boxed (index) at the end of your response.
Prompt Template for Count Negative Numbers Task
Count how many negative numbers are in the list (input_list); Your final answer must be in the format (boxed(counts at the end of your response.
37

---

the format (begth( at Published as a conference paper at ICLR 2026
Prompt Template for Count Unique Elements Task
Count the number of unique (distinct) elements in the list (input_list).
Your final answer must the format Aboxed count) at the end of your response.
Prompt Template for Maximum Difference Between Adjacent Elements Task
Find the maximum absolute di difference between any two adjacent elements in the list (inpu_liist),
Your final answer must be in the format \boxed differrnce) at the end your response.
Prompt Template for Count Flements Greater Than Previous Task
Count how many elements in the list are greater than the element that comes immediately before them: (input_list).
Your final answer must be in the format \boxed at the end of your response.
Prompt Template for Sum Indices of Maximum Elements Task
Find the sum of all indices (0-based) where the maximum value occurs in the list (input_list).
Your final answer must be in the format \boxed (sum) at the end of your response.
Prompt Template for Count Palindromic Numbers Task
Count how many palindromic numbers are in the list (input_list), A palindromic number reads the same forwards and backwards.
Your final answer must the format Aboxed count) at the end of your response.
Prompt Template for Longest Increasing Subsequence Length Task
Find the length of the longest increasing subsequence in (input_list). A subsequence maintains relative order but elements don't need to consecutive.
Your final answer must the end of your response.
Prompt Template for Sum Digits Task
Calculate the sum of all digits in all numbers in the list (input_list).
38

---

Published as a conference paper at ICLR 2026
Your final answer must be in the format (boxed sum) at the end of your response.
Prompt Template for Count Perfect Squares Task
Count how many perfect squares are in the list iinput_list)> A perfect square is an integer that is the square of another integer.
Your final answer must be in the format \boxed count) at the end of your response.
Prompt Template for Alternating Sum Task
Calculate the alternating sum of the list (input_list). Start by adding the first element, then subtract the second, add the third, etc.
Your final answer must be in the format \boxed (oxeddsum)) (sum) at the end of your response.
are multiples Prompt Template for Count Multiples of K Task
Count how many numbers in the list of (k).
Your final answer must be in the format \boxed count)) at the end of your response.
Prompt Template for Local Maxima Count Task
Count how many local maxima exist in the list Simput_list) A local maximum is element that is greater than both its immediate neighbors.
Your final answer must be in the format \boxed count) at the end of your response.
G MEDIUM SUITE: COMPLEX SEQUENTIAL REASONING TASKS
The Medium Suite introduces comprehensive collection of five coooiithiicall-ggeeeiiie sequential reasoming tasks, encompassing forty-nine distinct pattern variations that require sophisticated pattern recognition and mathematical reasoning capabilities. Unlike traditional benchmarks that rely on static datasets, our framework generates sequences dynamically using deterministic mathematical functions, ensuring both reproducibility and contamination resistance. The suite consists of Fibonacci and Recursive Sequences (6 variations), Geometric and Exponential Sequences (10 variations), Prime and Number Theory Sequences (11 variations), Complex Pattern Recognition (12 variations), and Algebraic Sequence Completion (10 variations). Each task evaluates distinct aspect of mathematical reasoning while maintaining computational tractability and verification guarantees through rigorous mathematical foundations.
39

---

Published as a conference paper at ICLR 2026
G.1 FIBONACCI AND RECURSIVE SEQUENCE COMPLETION
G.1.1 PROBLEM FORMULATION
Let S (81, .2,.... Sn represent sequence generated recursive relation R: 2 Z. where denotes the order of recurrence. The general k-th order linear recurrence relation is defined as:
8n 8n-s n> h
(47)
where a; E Z are recurrence coefficients, E Z is an optional constant term, and initial conditions are specified. The task presents partial sequence Sshown {81, $2..... 8m where m nt, and requires predicting 8m+11
(1,, 12, . 5&
G.1.2 SEQUENCE GENERATION ALGORITHM
We implement six distinct recursive sequence variations, each with deterministic generation guaran-
tees:
Classical Fibonacci: Generated using the relation Fn Fn-1 + Fn- with initial conditions (Fi, F2) ~ Uniform(1,20) 20).
Generalized Tribonacci: Extends to third-order recurrence with Tn T + Tn- Tn-3.
Modified Linear Recurrence: Implements M, = a Mn-1 + b. M where coefficients (a, b)
are sampled from predefined set C = (11,2), (2, 1), (1, (2, -1) to ensure bounded growth.
The generation process follows Algorithm
Algorithm 3 Recursive Sequence Generation
Input:
Sequence type r, length I,, seed
2: Output: Complete sequence S, shown portion Sshowns target 8m++1
3: Initialize RNG with seed a
4: Sample initial conditions (81...., 8 from ((1,20) 20)
5: Select coefficients (A1..... a3) based on type
6: for = 1:
do
7: 8; < Σド= Oj 8
8: end for
9: Schown {s1. 8L 8L-2
10: Return S, Schown, 81.-1
G.1.3 UNIQUENESS AND VERIFICATION GUARANTEES
The uniqueness of the next term in a linear recurrence sequence is guaranteed by the following theorem:
Theorem (Uniqueness of Linear Recurrence). Given a k-th order linear recurrence relation with non-zero leading coefficient + 0 and m k observed terms, the (m 1)-th term is uniquely determined.
Proof. The recurrence relation forms a system of linear equations. For m 2 k observations, we have at least k + equations of the form:
k aj
Sitk
8itb-js i 1,2, ..... m 411
(48)
1=1
With a * 0, the coefficient matrix has full rank, ensuring unique solution for 8mttl
40

---

Published as a conference paper at ICLR 2026
G.1.4 CONTEXT WINDOW CONSTRAINTS
The maximum sequence length Lmax is bounded by the growth rate of the recurrence relation to prevent numerical overflow. For Fibonacci-like sequences, the growth rate is approximately ø" where 1+v3 is the golden ratio. We enforce:
{[
log (MAX_INT)
Lmax = min
CONTEXT_
(49)
log(
where MAX INT 231 for 32-bit integers and CONTEXT LIMIT represents the model's maximum token capacity.
G.1.5 CONTAMINATION RESISTANCE
The contamination resistance of our approach stems from three key properties:
Parametric Diversity: With initial conditions sampled from (1, and multiple coefficient sets, the number of unique sequences is:
Suniquel 20 |C] x m(
(50)
where P(L, m) represents the number of ways to select m consecutive terms from a sequence of length L.
Each evaluation instance generates sequences using cryptographically secure random seed, ensuring that pre-training on specific sequences provides no advantage.
Dynamic Generation:
Verification Independence: The correctness of predictions is verified through direct computation rather than comparison with stored answers, eliminating the possibility of answer leakage.
G.2 GEOMETRIC AND EXPONENTIAL SEQUENCE COMPLETION
G.2.1 PROBLEM FORMULATION
Let G (91 .2......................... 9n denote sequence following multiplicative exponential growth patterns. We define the general geometric sequence as:
In = p
(51)
where 91 is the initial term and r e R {0) is the common ratio. For exponential sequences, consider:
C₂ = (n)
(52)
where b is the base and f N - IR is function determining the exponent pattern.
G.2.2 SEQUENCE TYPE SPECIFICATIONS
We implement ten distinct multiplicative pattern variations, each with rigorous mathematical defini- tions:
Pure Geometric Sequences: Generated with ratio r E R = (2,3, 0.5, 3.5, to balance growth control and pattern diversity.
Power Sequences: For power p E (2,3), we generate P, = n², ensuring polynomial growth bounded by (nn)).
Factorial Sequences:
Defined as F, = n!! with growth rate O(n"), requiring careful overflow
management.
Double Exponential Sequences:
The most challenging pattern, defined as:
D, (066-.99 (n iterations)
(53)
41

---

Published as a conference paper at ICLR 2026
Due to the extremely rapid growth, we limit generation terms, where:
D1
(54)
D2 2 =
(55)
D3 2 16
(56)
D4 2 65536
(57)
G.2.3 COMPUTATIONAL FEASIBILITY CONSTRAINTS
To ensure computational tractability while maintaining pattern complexity, we enforce strict bounds on sequence values:
Vg. E g: 19s1 θ max
(58)
where max 10t is the maximum allowed value. The generation algorithm incorporates overflow protection:
Algorithm 4 Geometric Sequence Generation with Overflow Protection
1: Input: Type т, length L, maximum value max
2: Output: Valid sequence G if infeasible
3: Initialize sequence G +-
4: Select parameters (g1,r) r) based on type 7
5: for i = 1 to L do
6: Compute 9, according to sequence type
7: if [gi > max then
8:
break // Terminate on overflow
9: end if
10:
Append 9: to
end for
12: if |9] < then
Return 0 // Insufficient terms
14: else
Return
16: end if
G.2.4 PATTERN DISCRIMINATION AND VALIDATION
To ensure that generated sequences genuinely test multiplicative reasoning rather than simple arith- metic patterns, we implement a validation function:
true
if E [2, 1]:
Fit1/gh 10a1/g/9::/1 /
<E
IsGeometric(()
9./
9:-1
(59)
false
otherwise
where e 10-6 accounts for floating-pon point precision. Additionally, we reject sequences that exhibit
arithmetic progression:
RejectArithmnetic()) true = 3d: Vi E (1,n-1) 1]: 0:+1 9: d
(60)
G.2.5 THEORETICAL COMPLEXITY ANALYSIS
The computational complexity of identifying geometric patterns varies by sequence type:
Pure Geometric: Requires (n) operations to compute ratios and verify consistency.
Power Sequences: Recognition requires solving In. = n? for p. achievable in O log n) time using logarithmic transformation.
42

---

Published as a conference paper at ICLR 2026
Factorial Sequences:
Verification requires computing n! for comparison, with complexity (n?²)
using naive multiplication.
Double Exponential: Duc to the tower structure, verification requires recursive exponentiation with complexity 0(2"), justifying our length restriction.
G.2.6 UNIQUENESS THEOREM FOR GEOMETRIC SEQUENCES
Theorem (Uniqueness of Geometric Continuation). Given a geometric sequence g 001.92...... with 2 2 and 9i + 0 for all the next term 9m+ is uniquely determined.
Proof. For a geometric sequence, the common ratio terms:
9+4 is constant for all valid Given m
2
Q2
93
9m
(61)
91
92
9m-1
Therefore, 9m+1 = 9m n Im
9m
Am
which is uniquely determined when 9m-1 +
G.2.7 CONTAMINATION RESISTANCE MECHANISMS
The geometric sequence task achieves contamination resistance through:
Parameter Space Cardinality: With initial terms from [1, 10]. ratios from R. and variable sequence lengths, the total number of distinct sequences exceeds:
12 6
Groal
>
10 x
8
x
>
73, 920
(62)
Precision Variation:
Ploating-point ratios like 1.5, 2.5, 3.5 create sequences that cannot be memorized as simple integer patterns, requiring genuine multiplicative reasoning.
Type Mixing: Random selection from ten sequence types during evaluation prevents models from exploiting type-specific heuristics.
G.3 PRIME AND NUMBER THEORY SEQUENCE COMPLETION
G.3.1 PROBLEM FORMULATION
Let P denote the set of prime numbers and V represent sequences derived from cheoreic functions. We define number theory sequence as:
N = (n, n, n; = f() or n. i N)
(63)
where f N + N is number-theoretic function and is the i-th prime number.
G.3.2 PRIME GENERATION AND VERIFICATION
We employ the Sieve of Eratosthenes for efficient prime generation up to limit L:
The algorithm's time complexity is (o log L.) with space complexity O(L).
G.3.3 NUMBER-THEORETE FUNCTION IMPLEMENTATIONS
We implement eleven distinct number-theoretic sequence variations with mathematical rigor:
Euler's Totient Function: For n E N, ((n) counts integers S n where god(k, n) 1:
((n)
II 1
1
(64)
p
p/n
43

---

Published as a conference paper at ICLR 2026
Algorithm 5 Optimized Sieve of Eratosthenes
1: Input: Upper limit L
2: Output: Set of primes P L
3: Initialize boolean array A(O... L] with Ali] true
4: A[0] AA][]] < false
5: for to VL] do
6: if A
then
7:
for j = 1 to L step do
8:
Ali]
false
9:
end for
end if
11:
end for
Return Ai: true)
where the product runs over distinct prime divisors of n.
Sum of Divisors Function: The function (n) computes:
h+
(n) Σd
(65)
p
d/n
p
where pr1[12 denotes that pd exactly divides n.
Mersenne Numbers: Defined as Mp 2 where p E P. The Lucas-Lehmer test provides
primality verification:
80
4 8i+1 = o -2 (mod Mp)
(66)
Mp is prime if and only if sy-2 = 0 (mod Mp).
Sophie Germain Primes: A prime p is Sophie Germain if 2p + 1 is also prime. The density of such primes follows:
æ
TSG (x) ~ C.
(67)
(In )²)
where C' 1.32 is the twin prime constant.
G.3.4 SEQUENCE GENERATION WITH COMPUTATIONAL BOUNDS
For sequences with potentially rapid growth, we implement dynamic bounds:
Tamax (i) = min {f(i),
(68)
Special considerations for each sequence type:
Prime Gaps: The n-th prime gap In Po+1 satisfies Bertrand's postulate:
9n < Pn for Pn > 25
(69)
Catalan Numbers: Generated using the recurrence:
n-
C C.CnI-ss Co
(70)
with closed form:
1
22n
(2n))
C₂
(71)
12 + 1
(n 1)!
44

---

Published as a conference paper at ICLR 2026
G.3.5 UNIQUENESS AND WELL-DEFINEDNESS.
Theorem 3 (Uniqueness of Number-Theoretic Sequences). For each number-theoretic function f: NN + N implemented in our framework, given sequence (()).....) with m 2 3. the value f'm + 1) uniquely determined.
Proof. Each function in framework is well-defined:
Prime sequences: The n-th prime Pa is uniquely defined by the ordering of natural numbers.
2. Totient function: (nn) is uniquely determined by the prime factorization of
Divisor sum: (n) is uniquely computed from the divisors of n.
4. Mersenne numbers: Mp 2 1 is deterministic function of prime p.
Therefore, f'm + 1) is uniquely computable from the definition f.
G.3.6 PATTERN RECOGNITION COMPLEXITY
The computational complexity of identifying number-theoretic sequences varies significantly:
Sequence Type
Recognition Complexity
Verification Complexity
Prime numbers
Onvnn)
O(yn) per element
Twin primes
(nynn
((yn) per pair
Euler totient
O(n))
Onn log n) per element O(p²) Lucas-Lehmer
Mersenne numbers
O(n 2n) O(n²)
Catalan numbers
( per element
Table 10: Computational complexity for number-theoretic sequence operations
G.3.7 CONTAMINATION RESISTANCE THROUGHI MATIEMMTTICAL DEPTII
Number-theoretic sequences provide inherent contamination resistance because:
Infinite Variability: The set of primes is infinite, and subsequences can start at any position, yielding uncountably many valid test instances.
Many number-theoretic functions lack closed-form solutions, requir- ing actual computation rather than pattern matching.
Computational Irreducibility:
Dependencies: Sequences like Sophie Germain primes require understanding multi- ple mathematical concepts simultancously, preventing shallow memorization.
The probability of encountering an identical sequence during training is:
P(collision)
(72)
m[109]
1030
where N (10%) 78, 498 is the prime counting function and m the sequence length shown.
G.4 COMPLEX PATTERN RECOGNITION
G.4.1 PROBLEM FORMULATION
Let = (c1, C2, .., } represent sequence following composite multi-layered pattern. We define complex patterns as sequences that satisfy:
= Fii, fe....,
(73)
where a composite function incorporating multiple mathematical operations, and represents pattern-specific parameters including breakpoints, modulation factors, and conditional rules.
45

---

Published as a conference paper at ICLR 2026
G.4.2 MULTI-LAYERED PATTERN SPECIFICATIONS
We formalize twelve distinct complex pattern variations, each requiring different levels mathemati- cal reasoning:
Polynomial Sequences: For degree E (2,3,4) 4}, generate:
p(a)
Σ
agn
(74)
k=0
where coefficients Oд are selected to maintain |P < 10 for all n S 20.
Interleaved Sequences: Two independent sequences A {a;} and B [b:} are interleaved:
0(+1)/2
if odd if is even
C
bi/2
(75)
Conditional Pattern Transformation: The sequence follows different rules based position or value conditions:
c(,) if (i-))) true otherwise
C:
(-1)
(76)
where vb N x 2 - {true, false} is boolean predicate.
Multi-Order Recursive Patterns: Generalizing k-th order dependencies:
On oj Cn-j 3(n)
(77)
where B S:N. Z introduces position-dependen perturbations.
G.4.3 PATTERN COMPLEXITY HIERARCHY
We establish a formal hierarchy of pattern complexity based on computational requirements for recognition:
Definition (Pattern Complexity Level). A sequence pattern has complexity level C if the minimum computational resources required for recognition are:
C (n) time, O(1) space
(arithmetic, geometric)
(78)
C2 : O(n²) time, O(n) space
(polynomial, recursive)
(79)
Ca O(n²) time, O(n²) space
(interleaved, conditional)
(80)
C4 O(2") time, O(n) space
(chaotic, multi-transform)
(81)
G.4.4
GENERATION ALGORITHM WITH PATTERN VALIDATION
G.4.5
UNIQUENESS GUARANTEES FOR COMPLEX PATTERNS
Theorem 4 (Unique Continuation of Complex Patterns). For each complex patlern type in our framework, given m observations where m 2 Mmin(T), the next term is uniquely determined.
Proof. We prove uniqueness for each pattern class:
Polynomial sequences: A degree-d polynomial uniquely determined by + points. For m > d, the system is overdetermined but consistent, yielding unique continuation.
Interleaved sequences: Each subsequence and B requires [m/2] terms. With deterministic interleaving rules, the next term's 's source sequence is known, ensuring uniqueness.
Conditional patterns: The predicate d and functions f1, f2 are deterministic. Given the sequence history, the next term is uniquely computed by evaluating & and applying the appropriate function.
Multi-order recursive: With coefficients {ay} and sufficient observations m 2 k, the linear system has unique solution.
46

---

Published as a conference paper at ICLR 2026
Algorithm 6 Complex Pattern Generation with Validation
1: Input: Pattern type т, length L, complexity level L
2: Output: Valid complex sequence C or failure indicator
3: Initialize parameters 0 < SampleParameters(r: L)
4: Generate candidate sequence C t GeneratePatterm T, L, 8)
5: Validation Phase:
6: if (C) then
7: Return FAILURE // Reject trivial patterns
8: end if
C then 10: Return FAILURE // Insufficient complexity
9: if ComplexityLevel(()
end if
12: if >HasUniquuEExeension(( then
Return FAILURE/ // Ambiguous continuation
14: end if
15: Return
G.4.6 PATTERN RECOGNITION VIA FINITE DIFFERENCES
For polynomial patterns, we employ the method finite differences. The k-th order difference operator is:
& A
(-1) k-j
Ci+s
(82)
1=0
sequence follows a degree-d polynomial if and only if A A++1 e, for all valid i,
G.4.7 COMPLEXITY VALIDATION METRICS
To ensure generated patterns maintain appropriate complexity, compute several metrics:
Kolmogorov Complexity Approximation: Using compression ratio as proxy:
|Compress(C))
((C)=
(83)
C
Entropy Rate: For sequences viewed as discrete random processes:
H(C) = Σ I(z) p(x)
(84)
BEX
where A' represents the alphabet of sequence values and p(x) is the empirical probability.
Autocorrelation Function: To detect hidden periodicities:
R(K) Ei-k(e (c: ()
(85)
Σ (c: -
Sequences with R(k) > 0.8 for small are rejected as insufficiently complex.
G.4.8 CONTAMINATION RESISTANCE THROUGH COMPOSITIONAL COMPLEXITY
Complex patterns achieve contamination resistance through several mechanisms:
Compositional Explosion: With 12 pattern types, each having multiple parameter configurations, and variable breakpoints, the space of possible sequences exceeds:
(20)
|Cotal > 12 x 10 x
x 4 > 10
(86)
10
47

---

Published as a conference paper at ICLR 2026
Non-Linear Interactions: Unlike simple sequences where local patterns suffice for prediction, complex patterns require understanding global structure and multiple interacting rules.
Conditional Branching: Pattern transformations based runtime conditions create sequences that cannot be pre-computed or memorized, requiring genuine logical reasoning.
The probability that model has specific complex pattern during training is negligible:
P(contamination) Training sequences
10a
10m3
(87)
Crotal
10
G.5 ALGEBRAIC SEQUENCE COMPLETION
G.5.1 PROBLEM FORMULATION
Let {a1. ........................... a, denote sequence generated through algebraic operations involving radicals, transcendental functions, and algebraic irrationals. We define the general algebraic sequence as:
on [(mn, {a1,..... an
(88)
where f N x R=-1 + R involves compositions of algebraic and transcendental operations, and [-] denotes the floor function ensure integer outputs.
G.5.2 ALGEBRAIC PATTERN SPECIFICATIONS
We implement ten sophisticated algebraic sequence variations that transcend simple polynomial patterns:
Radical Recurrence Relations: Sequences defined by:
On LL a n² =
(89)
This creates non-lineat recurrence where each term depends on the square root of quadratic expression involving the previous term.
Transcendental Floor Sequences: Incorporating exponential growth:
= [n 1/n
(90)
As - oo, we have et/n + 1, but for finite n, this creates a complex growth pattern.
Nested Radical Expressions: Multi-level radical compositions:
an n
n
(91)
The depth nesting creates algebraic numbers of increasing complexity.
Continued Fraction Convergents: For the continued fraction expansion of /2 [1; 2,... we generate convergents:
Pn = 1
(92)
2
2
The numerators Pr follow the recurrence Ps. 2pn-1 + Pn-2 with Po 1, Diophantine Solution Sequences: Based on Pell's equation x² -Dy² 1: (In+1, Bn+1) (I1#', + Dyibmn 21hh + f1
(93)
where (I1, v1) is the fundamental solution.
48

---

Published as a conference paper at ICLR 2026
G.5.3 COMPUTATIONAL COMPLEXITY AND VERIFICATION
The verification of algebraic sequences requires sophisticated numerical methods:
Theorem 5 (Computational Hardness of Algebraic Sequences). Determining whether a given sequence follows an algebraic pattern of degree d with nested radicals of depth k. requires Ω2(n d 22)
operations in the worst case.
Proof. For each term verification: 1. Evaluating nested radicals of depth requires O(2*) operations 2. Checking algebraic degree d requires solving polynomial equations of degree up d 3. For terms, the total complexity is O(n d 2*)
The lower bound follows from the algebraic independence of nested radicals, which prevents simplifi- cation.
G.5.4 NUMERICAL STABILITY AND PRECISION
Algebraic sequences involving irrational numbers require careful numerical handling:
computed On ap
exact
(elativeError(c.)) =
<
C
(94)
exactt
We maintain precision through:
Rational Arithmetie: Where possible, we use exact rational representations:
r
ps + gr
(95)
gs
Interval Arithmetic: For transcendental values, we maintain intervals:
an [a,
lower
a
upper
(96)
Error Propagation Analysis: For composed operations:
8(f o g) s 5(f) +11(()))
(97)
G.5.5 UNIQUENESS AND WELL-DEFINEDNESS
Theorem 6 (Uniqueness Algebraic Continuations). For algebraic sequences defined by determinis- tie rules involving elementary functions and algebraic operations, given sufficient terms 2 mei(T) where men depends on the sequence type т. the next term is uniquely determined up to numerical precision 6.
Proof. We establish uniqueness for each algebraic pattern type:
I. the value a,, is unique since: v- is single-valued function R+ The floor function has unique output for any input
Radical recurrences: Given an-1 and the deterministic rule a, = 1V an + n²].
Transcendental sequences: Functions like n l/n are well-defined for all n E N, yielding unique values.
Nested radicals: The iteration depth is fixed, making the computation deterministic and unique.
Therefore, algebraic sequences have unique continuations within numerical precision bounds.
G.5.6 PATTERN DISCRIMINATION AND NON- TRIVIALITY
To ensure algebraic sequences test genuine mathematical reasoning rather than pattern matching, we implement rigorous validation:
49

---

Published as a conference paper at ICLR 2026
Algorithm 7 Algebraic Sequence Validation
1: Input: Sequence A {d1....,0., on
2: Output: Valid/Invalid classification
3: Check 1: Non-polynomiality
4: for degree d = to do
5: Compute d-th finite differences A
6: if' A is constant then
7:
Return INVALID Polynomial pattern detected 8: end if
9: end for
10: Check 2: Non-geometric
11: Compute ratios T; = 0;41/a; for all valid i
12: if Variance (fr,)) } then
13:
Return INVALID // Geometric pattern detected
14: end if
15: Check 3: Sufficient complexity
Compute entropy H (AA 17: if (AA) < mii then
Return INVALID // Insufficient complexity
19: end if
Return VALID
G.5.7 CONTAMINATION RESISTANCE THROUGHI MATIEEMATICAL DEPTII
Algebraic sequences provide the strongest contamination resistance our framework:
Infinite Parameter Space: With continuous parameters like V5, e, and R, the space of possible sequences is uncountably infinite.
Computational Irreducibility:
Many algebraic sequences lack closed-form solutions, requiring step-by-step computation that cannot be bypassed through memorization.
Precision Sensitivity: Small changes in irrational parameters lead to completely different integer sequences after floor operations:
In (V5 c) * |n- v5] for small
(98)
Cross-Domain Integration: Algebraic sequences combine number theory, analysis, and algebra, requiring integrated mathematical understanding rather than domain-specific heuristics.
The probability of exact sequence collision during training approaches zero:
P(exact match)
(99)
26nn
where is the bit precision used for irrational numbers and n the sequence length.
G.6 CONTEXT-AWARE EVAL.UATION AND TOKEN ESTIMATION FRAMEWORK
We implement comprehensive mathematical framework for context-aware evaluation that dynam- ically adapts problem complexity according to each model's context window constraints. This approach ensures faimess by preventing models from being penalized due context limitations while maintaining the integrity of our mathematical evaluation framework.
G.6.1 MATHEMATICAL TOKEN ESTIMATION MODEL
For each task category in the Medium Suite, we establish precise token estimation functions that predict the expected response length based on problem parameters. Let Cmodes denote the model's context window size, T Tprmpp the prompt token count, and Trepoose the expected response tokens. We enforce the safety constraint:
50

---

Published as a conference paper at ICLR 2026
Table 11: Comprehensive Mathematical Overview of Medium Suite Tasks
Tack
RibomaccCReccrsive
Geomettric/xponenttt
Prime/Nmber Theory
Cumplex Pattern
Algebraic Sequence
Mathematical
P tp: pppmme)
Fit teg ?) Composite
[f'n, (s, hee) Algebralc operatios
Detiaition
Mukiplicative
Nhnber-theeeete cheorctic
Sequence Variations
variation::
10 variations:
variations
12 variation::
10 varioronn::
Clasical Fibvnnoci
Pure grometric
Prime
Polyromial quaéraliclesbic
RRdicil recurnncce
Power squresscubes
Imeeleaved sequences
Transeendeatal
TIibonncci
Facttonal (nc)
Prime
-oaditional ponems
redicaa redicals
recurive
Double exposenial
Messaue 12"
MMali-reeursive
Cantinned frations
Altemating Fibonacci
Compondd growth
Bullr div)
Pattenn cunsformation
Dioptuntin sdutions
Scaled Fibonacci
Thanqula/eemasonl
Sophic Germain
Modular aritimctic
Lumben others
Generation
per seqpence Desemmnissk
overflow
lelllogoo E))
Ote") swange
Ola 2") for
Complexity
protection
Prine test: (()))
Q12") wont caee
deptb testing
Verification
computation 011) per sem
Ratio checking: (()) for soquence
Primality
differences: Ois 4; for degree
Numerical methods:
Method
Miller Rabin Ofk log n]
Interval writhmetic
Uniqueess
Theorem: Given ferms *ree3 unique
Theorem: Ratio
Theorem: Primes
Theerem: Degroed
Thosevm: Determintic
Guaranter
r Siet/
well-ordendd
ncals + points
relex unique
when dj #0
determints sequenace
Ps P
for uniqueness
up to precision. c
Purameter
2/ [0] 10' combitationn
8 (22
(rloer) 10th poselbilatess
12 10 (2) 16 pertenu
Uncountably infinite
Space
soquerecs
paraneees))
Fibonacei: O1a*)
O(r*)
Prime pipe O(logs)
Polynomial: O(a*)
Varies type: Oly Ole")
Analyii
Factocial: Ots")
Mensenae: (?2*)
Chaotic: Q(2")
Contest
[SxMA Rigo))
S
Prime hauns:
Complesity
Procision muintained: 4 < 10
Constraints
CONTEXT_L.MM
Double exp. 1 S
Ps. mlu
Eniop H > Hais
Contumination
Dynamic seed a
[loating point ratios prevent memorization
Infnite prime set
Conditional branching
Innaional purameters Picollision) < 2
Resistance
Parameser diversity
closed form
Runime conditions
Validation
Chack in +
Rejest anilimetie
Vuity prinuliy
Rejeat if trival:
Rejos polysnmial putems up to degree 4
Criteria
Verify rourrence
progressions
Check factorization
2 -
Table 12: Computational Complexity Hierarchy of Medium Suite Tasks
Complexity Class
Recognition
Verification
Example Tasks
Space Required
Linear
((n)
O(1)
Fibonacci, Geometric
0(1)
L1
per term
Pure sequences
constant
Polynomial
O(n²)
O(n)
Prime sequences,
O(n)
L2
per term
Totient function
linear
Cubic
(nn²)
O(n²)
Complex patterns, Interleaved sequences
O(n)
L3
per pattern
quadratic
Exponential
O(2")
Algebraic with depth nested radicals
O(2*)
((n)
L4
linear
Table 13: Mathematical Bounds and Guarantees for Medium Suite
Property
Mathematical Bound
Guarantee
Sequence Length
E [5, 12) shown terms
Sufficient for pattern recognition while maintaining difficulty
n = m + 2 total generated
Value Range
1s:1 < 10t for all terms
Prevents overflow in 32-bit arithmetic Computational feasibility
Special: Double exp < 65536
Uniqueness
P'multiple solutions) = for m (T)
Deterministic generation
ensures single answer
Contamination
P(exact match) < 10-9 for typical training corpus
Parameter space size exceeds training data
Numerical Precision
Relative error < 10-6
Stable computation with floating-point
Absolute error < 1.0 for integers
Pattern Diversity
Total patterns 10
Inexhaustible evaluation space
across all tasks
51

---

Published as a conference paper at ICLR 2026
T prosppt + T response
+
T butler
muodel
(100)
where Tuufeer accounts for model-speeific reasoning overhead and verbose explanations. The response length estimation functions for each task type are:
T Iibonuuei (n, m) = an + f1 log m +
(101)
T geometric (n, r) = 02n log10 r + 82n + 72
(102)
T Trrmme (n) = ogn log 83 + 733 complex (n, d) = agnd + 8122 74
(103)
(104)
T algehrnic (n, k) ogn 2 + Bg log + 75
(105)
where n represents sequence length, m denotes the maximum value magnitude, r represents geometric ratios, d indicates polynomial degree, and & denotes nesting depth. The coefficients fa,, S, w are empirically calibrated through extensive model response analysis across different model families.
G.6.2 DYNAMIC PROBLEM SCALING ALGORITHM
Given target model with context window Caodel: implement an adaptive scaling procedure that maintains mathematical rigor while respecting context constraints. The algorithm ensures that every generated problem instance respects the model's 'ss context constraints while maximizing mathematical complexity within those bounds.
G.6.3 POST-GENERATION TOKEN VERIFICATION
After model response generation, we implement a duallverification token counting system to detect potential context overflow or excessive reasoning verbosity:
VALID
if [T(rr) S 0.95 Cinneel
Token Verification Cauode))
WARNING
if 0.95 Cmodel < [T(r)] Cawdel
(106)
OVERFLOW
if [T(r)] > Crontel
where T(r) represents the precise token count of response r computed using model-specific tok- enizers. For transformer-based models, employ the HuggingFace transformers library tokenizer corresponding to each model's architecture for accurate token counting. For GPT models, we uti- lize OpenAl's tiktoken library for precise token counting that matches their internal tokenization protocols.
G.6.4 MATHEMATICAL GUARANTEES AND IMPLEMENTATION ROBUSTNESS
Our context-aware framework provides theoretical guarantees including context safety with prob- ability > 0.99, mathematical fairesss ensuring models are overruns, complexity preservation that scales monotonically with available context budget, precision maintenance with token estimation error below +55% for 95% of generated instances, and variation coverage ensuring all 49 task variations remain accessible even under context constraints.
penalized for reasonable buffer
G.7 SOLUTION UNIQUENESS VERIFICATION AND MULTI-SOLUTION HANDLING
fundamental component of our Medium Suite framework ensures mathematical faimesss by rig- orously handling cases where problems may admit multiple valid solutions. We implement comprehensive solution verification system that prevents unfair penalization of models for producing mathematically correct but non-canonical answers.
52

---

Published as a conference paper at ICLR 2026
G.7.1 UNIQUENESS GUARANTEE PROTOCOL
For each generated problem instance p E P medium- we employ a deterministic verification procedure to establish solution uniqueness, Let S(p) denote the complete solution set for problem p. We define the uniqueness predicate:
TRUE
[fp)) FALSE S(p) >
Unique(p)
(107)
For tasks with inherently unique solutions (recursive sequences, algebraic computations, prime iden- tification), the generation algorithm guarantees Unique(p) = TRUE by mathematical construction.
G.7.2 COMPLETE SOLUTION ENUMERATION FOR EDGE CASES
When Unique(p) = LE, we implement exhaustive solution enumeration to ensure comprehensive evaluation fairness. The most relevant cases include floating-poing precision in geometric sequences, multiple valid interpretations in complex patterns, and algebraic approximation bounds when floor operations on irrational expressions may have -ereeddenndeeeeeeet results.
For these cases, we compute the complete solution set S(p) = {si, ....... sk) using deterministic enumeration with appropriate tolerance bounds. The verification function becomes:
CORRECT INCORRECT
if Parse(r) S(p) if (rrse(r) $ S())
Verify(r,p)
(108)
INVALID if Parse(r) =1
where Parse(r) extracts the model's answer from response r using robust parsing algorithms, and T. indicates parsing failure.
G.7.3 MATHEMATICAL COMPLETENESS VERIFICATION
To ensure valid solutions omitted, employ mathematical completeness checks specific each task category. For Fibonacci sequences, we verify that solutions satisfy the recurrence relation. For geometric sequences, we check ratio consistency within tolerance bounds. For prime sequences, we validate primality and ordering properties. For complex patterns, we verify adherence to the composite function definition. For algebraic sequences, we check floor operation results within precision bounds.
This comprehensive approach solution handling demonstrates that our evaluation framework maintains both mathematical correctness and practical fairness, ensuring that model performance reflects genuine sequential reasoning capability rather than arbitrary solution format preferences or numerical precision artifacts.
53

---

Published as a conference paper at ICLR 2026
H
MEDIUM SUITE: COMPLETE RESULTS
This section details results for all medium suite tasks across all non-quantized open-source models and closed-source proprictary models. The average accuracy, instruction following rate and average output tokens are listed for algebraic sequence, complex pattern, fibonacci sequence, geometric sequence and prime sequence table 14 15 16
54

---

Published as a conference paper at ICLR 2026
(Phram)
algebraic sequence
complex potteen
sequence
(Avg
(Arg ()
Tokens (Avg)
Axe (Arg %)
Irat (Avg %)
Tokeus (Aog)
(Avg
(Ang
(Arg)
Quivn Famuly (Qwen3)
Qwen3 (0.48)
12.00.1.1
100.00
20809.96
26.99
100.00
15008, 86
19.20 19.20.7.44
100.00
5949.09
Owen3 (1.78)
39.60
100,00
15074.02
35.91
100,00
11115.22 22
50,50
100.00
5738.80 85
Owen3 (4B)
34.00.40
100,00
200-0,74
71
100,00
10517.66 7.665
K3,40
100.00
4957.02
Qwen3 (SB)
39 39.30
100,00
19651 52
11.56
100,00
10706. 23
90.80
100.00
4971.
Qwend3 (148)
44.00a.
100000
12653.06
85 25.88214.01
100000
7150,00
95. 25.00
100100
4214.00 60
12wen) (X28)
48.001:40
100190
1215472
100000
6130.95
96. 20.20
100150
3692 U1
1pwnn) (985-MOR)
46:00123
10000
12166 76
$1.87
10300
7326-34
96.00 20.00gmm
100100
1913 64
Quenn (966-MFF-))
42 42:20g
10000
15500
$6 211
10300
5618.01
10
100000
1281 944
Qwend (365-MOF-I)
46 20200::
10000
$964 56
78, 55
10300
3495.25
90.30 g0
100.00
1875 75
Qwerd (4B-)
38. 38:40g...2 04
100,00
6417.81
66 83 g7.40
100000
S660.18
90.00
100.00
4645.3
Chean Fumily (Qhem2.55)
Owen2.5
100.00
11234.36
100.00
11202.35 53
21
Owen2.5
100.00
8138.47
100.00
9400.05
100.00
2506
Owen2.5
2..80
100.00
3158.95
100.00
1854.96
1:33
100.00
Owen2.5
39.00.1.33
100.00
1563.34
13:37
100.00
1226.71
100.00
635.19
Owen2.5 (14B)
4..2013.3::
98.50
1534.90
51.70:0.0
100.00
100.00
569.13
Owen2.5 (328)
44.00
96.20
1340.55
55.22
100000
100.00
Qwen2.5 (728)
46.202.4:1
100.00
89
100000
100.00
851
Qwend25
30.00/21
10000
555
100000
547.02
100.00
4553.01
Qwend23 (700mm)
14.0022
100033
1175.00
09
100000
$23.95
65.5
Quen7.5 (733km))
42,80
10000
1000-2
16.24.107
100000
55
50
50.00
100.90
907 9%
Fassily
Gemuu
34.80-12:35
1289.54
13.34
100.00
1372.62
1..40
Gemun
58.00 1..00
66.52
100.00
1036.85
26:00 17.64
1732.52
Gemun (128)
14:90
1216.36
100.00
$79.96
2243.42
Gemma (278)
66.00ga o
100.00
1077.16
45.00gg...
100000
1090.32
g4.00)..m...
PM Famully
Phu4:14B) (14B)
41 41.29
100,00
673.15
55, 74.74....
100,00
565.41
27. 20.20
100.00
930.83
Phul mini instricr (3
35 35,002..00
94,00
1316.77
38, 03.43:12
100,00
1251 101
14. 30
100100
1168. 45
Phol reasoning- (148)
30.00g
100000
7662.0
67
100000
0240.50
35, 24.
100100
5153.55
Phol-ressniig (1415)
40 40.001ga
100090
20695 (M
754322
100000
5627 92
02 20
1001000
1696 39
Thol-mii-nnoooming
78 Ailge
10000
25499.56
60 2020
100000
11225 96
09 000
100000
4834- 95
(3,88)
Phil-miii (3 8R)
31 4004
90:60
500.25
18:21 anu
100000
555.73
18 18.00g...00
100.00
$11-36
Pha3-mael (148-4k)
30 6002 a4
96.20
451.022
100000
537.56
15 40.
100.00
376.82
Plu3 and (14B-1266)
41 41.40
100,00
859.38
23, 66
100.00
751.15
15.80
100.00
446 81
Liane Fimilyy
Lioma (18)
100000
1103361
4.65
100000
10518.64
4,20
100.00
899
Lluna- (3B)
25.40
100.00
5165.19 19
100.00
4916.94
100.00
1530..54
Lluna 3.1 (5B)
100.00
15195.88
100.00
13003. 34
3.03
100.00
3138.27
Lluna 3.1 (70B)
44.4014.22
10000
2551,85
100000
2679.19
23.40
100.00
3581.74
Lloma 3.3
42 42.60....0
10000
534.67
62.25
100000
31,20
100.00
$40.40
Llomal
38
7100g11
290.44
100000
764.06
46 26.0g113.0
100,00
1004
Midial Fumily
Misiml (7B)
21 21.40g..v
100,00
634.7
5.96 os
100.00
1369.06
2.60
75
Ministal (SB)
100.00
1423.21
100.00
1248
3.1
1792.00
Mistal-aemoo (12B)
24.00 4.as
100.00
1905.43
18.50-0.41
100.00
1.30
Mixeal
15.19
99.50
7.75 -4.54
100.00
10.20.. 1.04
99:00
Mixual 8x22b
25.00...00
100,00
650.73
24.20 15.04
100.00
543.00
5..27
100.00
Others
Smollas (3B)
2.20
24.30
7432.62
12.46
97.97
21895.05
22,20 2
100.00
10965 566
Smollle2 (1.78)
1.71
15.00-1..7
17.15
0.00
0.00
15.02
0.50
27.60
272.14
GPT (200))
35:80
100,00
$899.88
74.43
100,00
3956.70
94 94,00
100.00
2257.25
GP USS (1208)
42.60
10000
4345.07
87.54 17
100,00
229.27
97.20 27.20
100.00
1355.22 25
OpenAJ Rumily (Propristurr)
GPTS
66
100,00
$60.00
GPTSsnini
66
100.00
705.25
$4.00
100.00
1504.2
100.00
GPTS
70.00
96.00
250.80
$4.00.19.00
100.00
94.00
100.00
125.76
GPT411
62.00
100.00
6506.40
21.54
100.00
4359
78.00
100.00
GPT4LI
68.00
100,00
4445.42
62.00111.00 113.65
100.00
55,00
100.00
1510.70
GPTAI
68.00
100000
878.32
100000
40
100.00
1740..54
GPTSo
08.00
10000
655
56.00.10.3
100000
34,00
100.00
596.00 A03
GP 1oo-min
001000....3
10003
37868
54,00,
100000
741
06.00
10003
1465
100000
2148.4
50.00
100/000
1564 566
of mini
66.00
9400 1.30
1731.88
50.00
100000
2768.80
96.02
100/00
74.11 194
62.00.7:45
100000
2505.24
92.00 22.0011.00
100000
4047.52
92.00:31.00
100,00
4766.60 83
Grnild Komily (Proprintory)
Grmini-2 Sygro
64000gm on
8600 co
1014.56
(000000 77
100000
795.46
000000gxx00
-6
Gamini-2.5 S-lash
$2000g ox
66.00 oz su
489.95
70.00 (0000::00000 1o
2.000a
12.000
95.00
830.50
Ganini-2 5-lasb-Bic
28000g3.0 ax
100,00
$6.00 12.00
100000
11470w6
72.00gavv.0 am
Ganini-2 O-dash
23.00
100,00
1118.52
100.00
1418.46
$4.00 20.20
Gemini dada--Bie
72.00
100,00
797.14
44.00....co
100.00
1196.84
72.00-19..30 29.39
72.00
100.00
1357.82
Table 14: Medium Suite Results- Table 1: Performance on algebraic sequence, complex pattern, and fibonacei sequence tasks. Each task reports Accuracy (Ace), Instruction- following (Inst) and average output tokens with mean and standard deviation values. Results show average performance across different sequence variants within each task category.
55

---

Published as a conference paper at ICLR 2026
Model (Param)
geometric sequence
prime sequence
(Avg
(Avg
(Arg)
(Avg
(Avg (Avg)
Qwen Family (Qwen.))
Qwen3 (0.68)
10.40
60.00
11425.59
12.20
100.00
21155.40
Qwen3 (1.7B)
39.80 428.0
60.00 48.99 99
11.29
Qwen3 (4B)
58. 28.40+47.000
60.00
4705.26
48.60
100.00
17998.46
Qwen3
44.00.444 41.
60.00 48. 99
53.20 6.111
Qwen3 (148)
44 43..95
60.00
2838.90
79. 20.20
100.00
4700,77
Qwen3 (32B)
40.000445.00
60.00=488.30 48.99
97.20+1.60
Qwen3 (50B-MOE)
60.00
60.00=448.
2085.35
90.80
100.00
6163. 1663228
Owen3 (30B-MOE-4)
60.00++45. 45.99
60.00 48. 99
Qwen3 (508-MOE-i)
58 47.91
60.000-468
1532.75
88
100.00
2596.35
Qwen3 (4B-4)
44.60
60.00 48. 39
4
Qwen Family (Qwen2.5)
Qwen2.5 (0.58)
0.60
60.00
48
5333.18
4.20
100.00
7209.29
Owen2.5 (1.58)
4.00 1..006.00 6.60
60.00
3505.67
16.40 6.99
100.00
6720.60
Qwen2.5 (3B)
18.60 20-11
60.00
1232,10
21.40:110770
100.00
2734.83
Owen2.5 (7B)
16.20 21.55
60.00
725.72
34.40 1:9.73
100.00
1223.20
Qwen2.5 (148)
27.60.1:32:000
60.00
350.25
42,00
100.00
614.53
Owen2.5 (32B)
28. 36.38
60.00
454.44
62.00 1:7.87
100.00
$35.48
Qwen2.5 (728)
38.80 1.8
60.00
490.25
57.60
100.00
661.07
Owen2.5 (1.58mm)
20.00 25.35
60.00
378.77
24.60 24.601:7.14
100.00
740.07
Qwen2.5 (7B-m)
13.00
60.00 99
789.18
26 00.6011::
100.00
896.04
Qwen2.5 (72B-m)
48.60 41.37
60.00
$42.09
67.00
10.97
100.00
677.45
Gemma Faamiy
Gemma (IB)
7,00 14.00
60.00 19
385,47
15.60
100.00
1505.70
Gemma (4B)
18.00 14.00
60.00
1258.26
40.00
100.00
720.54
Gemma (12R)
38. 38.00.1.5
60.00
99
673.22
50.00 1.6.22
100.00
1673.72
Gemma (27B)
56.00 4.6.0.446..
60.00 11.488.000
575.52
50.004+17.59
100.00
939.86
Phi Family
Phi4
26.60.129.22
60.00-44.800 99
502.35
40.20 11.00-111.33 1::34
100.00
671.558
Phid-miiii-nstmatt (3.88)
10,60
60.00=48.000
1246.29
25.80
100.00
1696.60
Phif-reasooingg+ (14B)
41.20:442.04
60.00-448.
3793.17
41.20
7398,78
Phid-reasoning (14B)
$2.00 43.66
60.00
3576.79
48.80 48.80
100.00
6455.13
thit-miii-rcasoiing
47.40:447.01
60.00-448. 99
1464.72
50.00
100.00
17050.78
Phi3-mini (3.8B)
12.20
60.00 10000-44
254.47
18.80 18.80 6.
100.00
183.69
Phi3-med (148-4k)
11.60=17.82
60.00 48.
445.39
20.20:110:66 1.007
100.00
Phi3-med (148-128k) 28k)
9.4 9.40
60.00 2 48.48..00
270.70
27 40 16.08
100.00
409 34
Llama Family
Llama-3.2
1.8013.60
1 48. 99
1999.64
12.09.990
7862.36
LIama-3.2 (3B)
7.20 14.04
60.00
12039.47
15:00
100.00
8685.33
(8B)
9.60 19.0119.20
99
13.80.13.00 13.54
4035.65
Llama-3.1 (70B)
24.20
60.00
2135.62
22.88
100.00
4604.62
Llama-3.3
43.95 55.20
48.
50.00
1...01550000
804.33
Llamo4-scout
60.00 4
681.09
55.00
100.00
916.03
Mistral Family
(7H)
2.20::4.440
60.00 48. 99
12.00 2.0.7
635.68
Ministral (SB)
13.80
60.004448.90
453.98
21. 21.80
100.00
884.59
Mistral-nemo
1840:
60.00 48. 99
952.88
28.00 14.95
100.00
943.33
Mxx7r
9.20+18.40
37.60 40.44
67.02
19.60
97.40
267. 347.44
18.07
60.00 48.99
465.90
27.60
44.67
100.00
Others
Smollm² (3R)
1.00
60.00
18989.60
18.00
85.00
8000.00
Smollm2 (1.78)
0.00
0.00
0.00
15.57
GPT-OSS (20B)
60.000
60.00
825.55
$1.00
100.00
5432.61
GPT-OSS (1208)
60.00 60.00 18.000-44.0000
60.00
598.78
87.60
100.00
2374.99
Table 15: Medium Suite Results Table 2a (Open Source): Performance open source models on geometric sequence and prime sequence tasks. Each task reports Accuracy (Acc), Instruction- following (Inst) and average output tokens with mean and standard deviation values. Results show average performance across different sequence variants within each task category.
56

---

Published as a conference paper at ICLR 2026
Model (Param)
geometric sequence
prime sequence
(Arg %)
Inst (Avg %)
Tokens (Arg)
(Avg
Inst (Avg %)
Tokens (Arg)
OpenAl Family (Proprictary)
GPTS
60.00.148.90 99
60.00-148.99
565.76
96.00
100.00
540.00
GPTS-mini
60.00
60:001448.90
485. 12
92.00
100.00
663.52
GPTS-nano
00.00-+48.90
60.00+48.99
92.00
178.56
GPT4.1
60.0011448.00
60.00.148.00
826.06
88.00
100.00
4917.80
GPT4.1-min
60.00448.60
60.00+48.99
619.12
84.00+13.566
100.00
4622
32.00+130.50
60.00.146.80
990.58
68
100.00
1401.26
GPT4o
50.00+41.44
60.00448.99
398.98
64.00
100.00
456.60
GPT4o-mini
20.00-120.96
60.00-48.99
432.00
40.00 8.00 94
394.32
04 mini
60.00.448.99
60.00 48.
732.32
100.00
100,00
1078.24 24
o3
60.00148.99
60.00148.99
1374.56
08.00+4.000
98.00
1706.88 88
60.00 48.
60.00 + 48.99
98.00.44..00 4.000
Gemint Family (Proprietary)
Seminii-2.5-o
60.00 44.8.999.9
60.00.4455 48.99
90.00 6.33
Gcmini-2.5-faah
60.00-1488.00
60.00.144.000
566.28
74.00
74.00
533.98
Gemini-2.5
18.00 18.33
60.00 4 48.99
10095.68
46.00 12.000-12..0
100.00
15541.46
Gemini-2
60.00.1448.00 90
60.00 99
370.76
64.00
100.00
1533.80
Scmini-2.0-flshhee
38.00
60.00
871.42
52.00
100.00
993.64
Table 16: Medium Suite Results Table 2b (Closed Source): Performance of closed source models
on geometric sequence and prime sequence tasks. Fach task reports Accuracy (Acc), Instruction- following (Inst) and average output tokens with mean and standard deviation values. Results show average performance across different sequence variants within each task category.
57

---

Published as a conference paper at ICLR 2026
I PROMPTS USED FOR ALL MEDIUM SUITE TASKS
This section describes the prompts developed for each of the medium suite tasks. Each prompt begins with an instruction directing the model to identify a pattern and complete a sequence of numbers based on that pattern. The prompt then provides sequence of numbers along with a list of possible patterns. The model instructed to present its answers as decimals rounded to two decimal places, unless the result is whole number. All final answers must be enclosed in the format \boxed(). J. The answer then extracted from within the \boxed() brackets using a regular expression. Finally, an example is included to demonstrate the required \boxed format.
Prompt Template for Algebraic Sequence Task
Complete the following algebraic sequence by identifying the mathematical relationship:
(sequence_str), ?
This sequence follows complex algebraic pattern that may
involve:
Radical expressions and nested square roots
Transcendental functions (exponential, logarithmic)
Recurrence relations with algebraic operations
Modular arithmetic and number theory
Continued fractions and convergent sequences
Solutions to algebraic equations
Special functions and their approximations
Analyze the sequence by considering: 1. Recurrence relations involving radicals or transcendental functions
2.
Floor or ceiling operations applied to complex
expressions
3. Modular arithmetic patterns
4. Nested algebraic operations
5. Number-theoretic sequences (Fibonacci-like,
factorial-related)
6. Convergent sequences and continued fractions
Note: If your answer is not whole number, provide it a decimal rounded to two places.
Provide the next term in the sequence. Your final answer must be in the format |boxe|
the end.
For example: the sequence involves floor (sqrt (n sqrt (n))), compute the pattern carefully before providing your answer.
Prompt Template for Complex Pattern Task
Complete the following complex sequence by ident fying the underlying pattern:
(sequence_str),
This sequence follows a complex mathematical pattern that may involve:
Multiple layers of relationshi (nested patterns)
58

---

Published as a conference paper at ICLR 2026
Polynomial sequences (quadratic, cubic, or higher order)
Interleaved sequences (alternating between different rules) Conditions onal patterns that change based on position or value
Recursive relationships involving multiple previous terms Pattern transformations that change at certain points Modular arithmetic or position-dependent rules
Analyze the sequence carefully, looking for:
1. Differences between consecutive terms
2. Ratios between consecutive terms
3. Patterns in even/odd positions
4. Polynomial relationships (quadratic, cubic)
5. changes the pattern at certain positions
Provide the next term in the sequence. Your final answer must be in the format \boxed|next_terml at the end.
For example: If the sequence is 4, 9, 16, 25, ? then the next term is \boxed(36) (perfect squares: n²).
Prompt Template for Fibonacci Sequence Task
Complete the following sequence by identifying the pattern:
(sequence_str),
Analyze the sequence carefully to identify the underlying pattern or rule. Consider:
Is each term related to previous term(s)?
Are there arithmetic, geometric, or recursive relationships?
Look for sibonacci-like patterns, Lucas sequences, other mathematical progressions
Provide the next term in the sequence. Your final answer must be in the format Nboxed/next_terme at the end.
For example: If the sequence is 1, 1, 2, 3, 5, then the next term is \boxed(13).
Prompt Template for Geometric Sequence Task
Complete the following sequence by identifying the pattern:
(sequence_str),
Analyze the sequence carefully to identify the underlying pattern. Consider:
Is there constant ratio between consecutive terms (geometric sequence)
Are the terms related to powers, exponentials, or factorials?
Look for patterns like squares (n²), cubes (n²), exponentials (a"), factorials (n!)
Check if each term is obtained by multiplying the previous term by constant
Consider compound patterns that combine multiplication with
59

---

must be in the format \boxed Published as a conference paper at ICLR 2026
other operations
Important: Some sequences may grow very rapidly. If you identify a pattern that would produce extremely large numbers, that's likely correct.
Provide the next term in the sequence. Your final answer at the end.
For example: If the sequence is 2, 6, 18, 54, ? then the next term \1oxed(162) (each term is multiplied by 3).
Prompt Template for Prime Sequence Task
Complete the following number theory sequence by identifying the pattern:
(sequence_str),
This sequence involves concepts from number theory. Analyze carefully and consider:
Are these prime numbers or related to primes?
Look for patterns involving divisibility, factors, number theoretic functions
Consider sequences like: prime numbers, twin primes,
Fibonacci numbers, perfect numbers
Check if numbers follow arithmetic properties, modular arithmetic, or special functions
Think about famous sequences in mathematics and number theory
Provide the next term in the sequence. Your final answer the end.
must be in the format \boxed[next_temm
For example: the sequence is 2, 3, 5, 7, 11, ? then the next term is \boxed(13) (consecutive prime numbers).
60

---

Published as a conference paper at ICLR 2026
J HARD SUITE: ADVANCED CONSTRAINT SATISFACTION AND COMBINATORIAL REASONING
The Hard Suite comprises ten algorithmically complex tasks with sixty-eight distinct variations that evaluate language model's capacity for systematic reasoning through constraint satisfaction, combinatorial optimization, and logical deduction. Each task encompasses multiple problem variants:
Boolean Satisfiability (5 variations), Constraint Optimization (5 variations), Cryptarithmetic (12 variations), Graph Coloring (IO variations), Logic Grid Puzzles (8 variations), Matrix Chain Multipli- cation (5 variations), Modular Systems Solver (5 variations), N-Queens (4 variations), Sudoku (8 variations), and Tower of Hanoi (6 variations). Each task in this suite generates problems dynamically with mathematically guaranteed unique solutions or complete solution enumeration, ensuring contam- ination resistance while maintaining rigorous theoretical foundations. These tasks require models to navigate exponentially large search spaces, apply sophisticated constraint propagation techniques, and demonstrate mastery of recursive problem decomposition across diverse computational complexity classes.
The fundamental design principle underlying our Hard Suite rests on the mathematical property that for each problem class P. we can generate instances p E P with complexity !2(2") where represents the problem size parameter. This exponential complexity ensures that memorization becomes computationally infeasible, while the deterministic generation algorithms guarantee solution uniqueness through constructive proofs.
Let us define the contamination resistance formally. For a problem generator G + + P with parameter space O, we ensure that |0] >> |D] where represents any feasible training dataset. More- over, the mapping G' exhibits high sensitivity to parameter variations, meaning that for parameters 01.6, 0g E O with |80 82ll > e, the corresponding problems p1 G(O1) and p2 G(Og) have solution distance ( > 8 for some threshold 5, where S(p) denotes the solution to problem y.
Each task employs rigorous validation mechanisms to ensure solution correctness through formal verification procedures. We define a validation function V S - {0, 1} that verifies whether proposed solution s E S satisfies all constraints of problem E P. The validation complexity remains polynomial (ma²) for some constant k, enabling efficient verification despite the exponential solution complexity.
Unique Solution Guarantee and Multiple Solution Handling: Our framework ensures mathemat- ical rigor through two distinct validation approaches. For constraint satisfaction problems (CSP) and satistiability problems (SAT), we employ automated theorem proving techniques to guarantec that each generated instance p P admits exactly one solution s* S such that V(s", p) = 1 and vs + s* (Vssp)) p) = 0. This uniqueness is verified through exhaustive constraint propagation and backtracking algorithms during problem generation, ensuring that K E S V(s,p) 1]1 1.
However, when mathematical constraints naturally permit multiple valid solutions (as in some optimization enumeration problems), we calculate the complete solution set Sp {s E S (s, 1} through systematic enumeration. In such cases, we accept any solution s E S, as correct, ensuring that language models are not penalized for producing mathematically valid but non-canonical answers. This approach maintains evaluation fairness while preserving the theoretical soundness of our assessment framework.
The suite encompasses diverse algorithmic paradigms including dynamic programming (Matrix Chain Multiplication, Cryptarithmetic), constraint satisfaction (Logic Grid Puzzles, Modular Systems, N-Queens), graph algorithms (Graph Coloring), recursive decomposition (Tower of Hanoi), logical reasoning (Boolean Satisfiability). backtracking (Sudoku), and multi-ohjective optimization (Con- straint Optimization). This diversity ensures comprehensive evaluation across multiple reasoning dimensions while maintaining theoretical soundness through formal verification methods.
Context-Aware Token Management and Scaling Framework: Our evaluation framework imple- ments a mathematically rigorous token estimation and scaling system that ensures fair assessment across diverse language models with varying context window constraints. For each model with context window capacity C' tokens, we dynamically scale problem complexity to guarantee that the total token requirement remains within acceptable bounds.
61

---

Published as a conference paper at ICLR 2026
Let Tp (n) denote the expected token count for problem p with size parameter n, comprising prompt tokens T [n) and expected solution tokens Esolutio())) We maintain the constraint (n)n) Tuffer C where Tutter 0.15 C provides safety margin to accommodate model reasoning overhead and response verbosity variations.
Our token estimation employs task-specific mathematical models. For instance, in Tower of Hanoi with n disks, we have Toolution(? (n) = (2" - 1) a where a = 12 represents the average tokens per move description. Similarly, for Graph Coloring with vertices, Tolution (v) a where accounts for color assignment notation.
We implement a dual-phase token validation system. During problem generation, use deterministic token counting based on mathematical formulas specific to each task type. Post-inference, we perform actual token counting using transformers library tokenizers for open-source models and tiktoken for GPT models. If the model response exceeds 0.95 C' tokens, we issue a warning indicating potential context window saturation, which may compromise response quality due truncation or overthinking behavior.
This approach ensures that our evaluation captures genuine reasoning capabilities rather than penaliz- ing models for exceeding their architectural constraints. The mathematical foundation guarantees that each problem instance p satisfies Picontext overflow) < 0.05 under normal operation conditions, maintaining evaluation validity across the complete model spectrum.
J.1 TOWER OF HANOI: RECURSIVE STATE SPACE NAVIGATION
J.1.1 PROBLEM FORMULATION
The Tower of Hanoi problem represents a classical recursive puzzle that tests systematic state space exploration and optimal path planning. We formalize this as state transition system T (S, A. Su. G) where S denotes the state space, A represents the action set, T S x A - S defines the initial state, and G C S contains the goal states.
the transition function, 8o
For disks and three pegs labeled {A, B. C), we define a state E S as tuple (PA, Pn. Pc) where each P. represents an ordered stack of disks on peg i. The disk set = {1, .... n uses natural numbers where smaller values indicate smaller disks. The ordering constraint requires that for any peg P. = [d1, d2, ...,
dx we have dj < dj+1 for all S < k.
The action space A consists of all legal moves (d, Psres Past) where disk d moves from peg Pare to peg Pdtst. A move is legal if and only if:
legal(d, ParcsPast)
d == min(Psre) A (Past = 0 V d < min(Past)
(109)
J.1.2 OPTIMAL SOLUTION CHARACTERIZATION
The optimal solution for disks requires exactly 2" moves, which we prove through the recurrence relation:
1 2 (1+1))+ if n
n =
T(n)
(110)
This recurrence yields T(n) = 2 through straightforward induction. The optimality follows from the observation that moving the largest disk n requires first moving all smaller disks to the auxiliary peg (requiring T'(n 1) moves), then moving disk n (one move), and finally moving the n 1 disks to the destination peg (another T T'n 1) moves).
J.1.3 DYNAMIC PROBLEM GENERATION
Our generation algorithm produces Tower of Hanoi instances with varying disk counts n E {3, 4, 5. 6, 7, 8). The initial configuration always places all disks on peg A in descending order of size, formally:
Sp (PA [n, 1,.... 2, Pb = 0, Pc -0)
(111)
62

---

Published as a conference paper at ICLR 2026
The goal state requires all disks peg C:: 8g (P^ = 0, PB 0, Pc [n, n 1,... 2, 1J)
(112)
To ensure contamination resistance, we employ several variation strategies. First, we randomize the disk count for each evaluation instance. Second, can permute peg labels while preserving the logical structure, creating (3! = 6) distinct but equivalent formulations. Third, we vary the representation format in prompts, using different notations for disk sizes and peg identifiers.
J.1.4 SOLUTION VERIFICATION ALGORITHM
Given a proposed solution sequence [m,, m2, .... m& where each my (d4, Psrre Plae): our verification algorithm validates both correctness and optimality:
Algorithm 8 Verify Tower of Hanoi solution
1: function VERIEYHANOU(O,n)
2: 5 80
Initialize with all disks on peg A
3: for each move mh; (di, Pere: Past) in do
4: if d; # min( Min(Pi (rre) then
5:
return INVALID end if
b Can only move top disk
6:
7:
if Past + 0 and de > min( (Pst) then return INVALID end t m.) end for
8:
Cannot place larger disk smaller
9:
Apply transition
11:
if * 8g then
13:
return INCOMPLETE end
14:
15: if | * 2 then
16:
return SUBOPTIMAL end if
17:
18:
return VALID
end function
The verification complexity O(k: -n) where k the solution length, making it efficient despite the exponential solution space.
J.1.5 CONTEXT WINDOW CONSTRAINTS
The solution length grows exponentially as L(n) = 2" 1. For our maximum configuration of = disks, this yields 255 moves. Each move requires approximately 10-15 tokens express (c.g., "Move disk from peg A to peg B"), resulting in a maximum token requirement of approximately 3,825 tokens. This remains well within modern language models' context windows while still presenting significant reasoning challenges.
The prompt complexity itself scales as (n) for deseribing the initial state, plus constant overhead for rule explanation. The total prompt length P(n) as 200 + 10n tokens ensures efficient context utilization.
J.1.6 UNIQUENESS GUARANTEE
The Tower of Hanoi problem exhibits unique optimal solution path for any given initial and goal configuration. This uniqueness emerges from the recursive structure: to move the largest disk, all smaller disks must be the auxiliary peg, and there exists exactly one optimal way to achieve this configuration.
Formally, prove uniqueness through structural induction. For 1. there exists exactly one legal move. For n > 1, assume uniqueness holds for n disks. The solution for n disks decomposes into:
63

---

Published as a conference paper at ICLR 2026
1. Move 1 disks from source auxiliary (unique by induction hypothesis)
2. Move disk from source to destination (only legal move for disk r1)
3. Move 1 disks from auxiliary to destination (unique by induction hypothesis)
This recursive decomposition admits exactly one optimal solution, guaranteeing that our evaluation has well-defined ground truth.
J.1.7 CONTAMINATION RESISTANCE ANALYSIS
The Tower of Hanoi task achieves contamination resistance through multiple mechanisms. The num- ber of distinct problem instances equals (i) 8 for disk count variations alone. When combined with peg permutations and representation variations, we generate over 288 distinct problem formulations.
More importantly, the solution space grows exponentially. For disks, there exist (3²"-1) possible move sequences of optimal length, though only one correct. This exponential growth ensures that memorizing solutions becomes infeasible. A model attempting memorize all solutions for n E [3.... 8) would need to store:
E(2 (2 1) 1o.loog 13 33.3 ...) 10 bits 1=3
(113)
This calculation demonstrates that even for our limited parameter range, exhaustive memorization requires substantial storage, and the approach fails generalize larger values that could generate dynamically.
J.2
N-QUEENS: CONSTRAINT SATISFACTION THROUGH BACKTRACKING
J.2.1 PROBLEM FORMULATION
The N-Queens problem exemplifies constraint satisfaction problems (CSPs) requiring systematic exploration combinatorial spaces. We formalize this as a CSP tuple (X, D.C' where {21, T2, ..... ) represents variables (queen positions), D = {D1, D2. D,, denotes domains with D, [1, 2, ..., n for all i, and encompasses the constraint set.
Each variable indicates the column position of the queen in row i, reducing the representation from O(n²) to (n)) while implicitly satisfying the row constraint. The constraint C' consists of:
Goot Vi
(114)
Cdieg1 Vi * J, - i xz Cding2 Vi # j, + +j
(115)
(116)
These constraints ensure no two queens share a column, main diagonal, or anti-diagonal respectively.
J.2.2 SOLUTION SPACE ANALYSIS
The total search space contains possible assignments, the column constraint reduces this to n!! permutations. The number of valid solutions S(n) follows no closed-form formula but exhibits known values:
Search Space
Valid Solutions
Solution Density
4
24
2
0.0833
5
120
10
0.0833
6
720
4
0.0056
8
40,320
92
0.0023
The solution density decreases rapidly with n, making random sampling ineffective and necessitating intelligent search strategies.
64

---

end for Published as a conference paper at ICLR 2026
J.2.3 DYNAMIC PROBLEM GENERATION
Our generation algorithm leverages the guarantee that solutions exist for all n > We employ two-phase approach: first generating a valid solution, then constructing the problem presentation.
Algorithm 9 Generate an n-Queenn problem instance
function GENERATEENQEENSS(Y)
2: solution t- (acktrackSolve(n)
Generate one valid placement Find all solutions for validation
3: all _ssouuitionsss <- EnumerateSolltionn(())
4: prompt Forma(Problem(n)
5: return (prompt, solution, all all_solutions)
6: end function
The backtracking solver employs constraint propagation efficiently find solutions:
Algorithm 10 Backtracking solver for n-Queens
1: function BACKTRACKSOLVV(Cn row = 0, placement = ()
2: if row = n then
3: return placement
D Found complete solution
4: end
5: for col I0. 1 do ISSafe(placement, row, col) then placementfroul +- col result < BacktrackSolve(n, row + 1, placement)
6:
7:
8:
9:
if result * null then return result.
10:
end if end if
12:
end for
14:
return null
15:
end function
The safety check verifies all constraints in (n)) time:
Algorithm 11 Safety check for queen placement
1: function AsSplaaplee, row, col)
2: for E (0, 1, row - 1) do
if placement((] col then return False 5: end if
3:
Column conflict
4:
6:
if placement[i] - coll = ji - rowl then
Diagonal conflict
return False 8: end if
7:
9:
10: return True
11:
end function
J.2.4 SOLUTION VERIFICATION
Verification requires checking all (*) n(n-1) pairs of queens for conflicts:
The verification complexity is O(n²), making it efficient even for large board sizes.
J.2.5
CONTEXT WINDOW ANALYSIS
The problem description requires O(n) tokens to specify board size and rules. The solution rep- resentation needs exactly integers, requiring approximately 2n tokens when formatted. For
65

---

Published as a conference paper at ICLR 2026
Algorithm 12 Verification of an n-Queens solution
1: function VERFYYQQUENSploeement, n.)
2:
if [placemen]] o n then
return Invalid 4: end
3:
5: for i E {O...., 1 do
6:
if placement[]] $ {O..... 1} then return Invalid end for j E -i++1,.., 1] do
7:
8:
9:
10:
if placement((] - placementli] then
P Column conflict
return Invalid end if ji
12:
jl pplacement [1] placement then
Diagonal conflict
14:
return Invalid end if end for
16:
17:
end for
18:
return Valid
19:
end function
maximum n = 8, the total context requirement remains under 100 tokens, allowing ample space for reasoning traces.
The challenge lies not in context length but in the combinatorial complexity. The model must navigate search tree with branching factor up and depth n. potentially exploring O(n") nodes in the worst case.
J.2.6 MULTIPLE SOLUTION HANDLING
Unlike Tower of Hanoi, N-Queens admits multiple valid solutions for most board sizes. This multiplicity enhances contamination resistance complicates evaluation. We address this through comprehensive solution enumeration.
For each n, we precompute all valid solutions using symmetry-beaaknng optimizations. The funda- mental board symmetries form the dihedral group DA with 8 elements (4 rotations and 4 reflections). However, some solutions exhibit self-symmetry, so the actual number of unique solutions under symmetry is:
U(n) S(n) ΣgED, fix(g)
(117)
8
where fix(g) counts solutions invariant under symmetry g.
J.2.7 CONTAMINATION RESISTANCE
The N-Queens problem provides strong contamination resistance through several mechanisms. First, the number of valid solutions grows rapidly with n. reaching 14,772.512 for n = 16. Even if a model memorized all solutions for small n, it cannot generalize this memorization larger boards.
Second, we can apply isomorphic transformations to create equivalent but syntactically different problems. Given a solution [r,,..., #n), we can generate additional equivalent solutions through rotations and reflections:
rotateso ([0), 2n]) = [n Fns ..., n2 x1]
(118)
reflect, ([x1, (1x1, xnnl) 21 ..., En 1]
(119)
66

---

Published as a conference paper at ICLR 2026
Third, the problem admits numerous equivalent formulations. We represent it permutation problem, a graph coloring problem, or a satisfiability problem, each requiring different solution formats while testing the same underlying reasoning capability.
J.2.8 THEORETICAL COMPLEXITY
The N-Queens problem is known to NP-complete for generalized boards (where some queens are rre-placd). While variant with empty boards admits polynomial-time solutions for specific values, finding all solutions requires exponential time in the worst case.
The time complexity of our backtracking algorithm is O n) in the worst case, though constraint propagation typically achieves much better average-case performance. The space complexity remains O(n) for the recursion stack and solution storage.
This exponential worst-case complexity ensures that models cannot rely on simple pattern matching or shallow heuristics but must engage in genuine constraint satisfaction reasoning.
J.3 GRAPH COLORING: CHROMATIC OPTIMIZATION AND CONSTRAINT PROPAGATION
J.3.1 PROBLEM FORMULATION
The graph coloring problem seeks to assign colors to vertices such that no adjacent vertices share the same color, using the minimum number of colors possible. Formally, given an undirected graph represents vertices and E S V x V denotes edges, we seek function f / + {1, 2, ..., k) such that:
G = (V, E') where V = {U1, U2,.... ...
Ve, wz) E E: ((x) + f(v,)
(120)
The chromatic number XG) represents the minimum & for which such function exists. We formulate this as an optimization problem:
x(G) = minfk: E N: 3ʃ V [k] such that V(u, v) E El + [(v)
(121)
J.3.2 GRAPH GENERATION WITH KNOWN CHROMATIC NUMBERS
Our framework generates diverse graph families with predetermined chromatic numbers, enabling precise evaluation of model reasoning. Each graph type exhibits distinct structural properties that determine its chromatic number.
For complete graphs Kn, every vertex connects every other vertex, yielding x(K,) trivially. The edge set is:
Ek. = (i..)
(122)
Cycle graphs C, form circular structure with x(C,) = 3 if is odd and x(C.) = 2 if is even:
Ec.. ((i, (i mod n) 1) n)
(123)
Wheel graphs W,, consist of hub vertex connected to all vertices of an (n 1) -y-cyl... The chromatic number depends on the cycle parity:
(4 if 1 is odd x (W,)
(124)
n 1 is even
For planar graphs, leverage the Four Color Theorem, which guarantees XGG < 4 for any planar graph. Our generation algorithm constructs planar graphs through controlled edge addition that preserves planarity while maximizing chromatic complexity.
67

---

Published as a conference paper at ICLR 2026
Algorithm 13 Generate graph with chromatic number estimate
1: function GENERATEGRAPHON type, target- ceeoee_ccmaatc)
2:
V x-- [1,2....,7)
3: E x-0 0
4:
if type = PLANAR then
5:
E' Generratcllanardgges//
6:
< Brooks T'eors/' E,
7:
else if type WHEEL then
8:
E' + GeeneratWhhelEdggee
9:
X + 3 + (n - mod 2)
10:
else type - DENSE_RANDOM then
p 0.6 0.6
Edge probability
12:
for each pair (i.j) with < do (Random() < then
14:
E < F ((.5))
15:
end end for
16:
17:
< EsttmaeChhoomiiceuubbeBB.
18:
end if
19:
return E.)
20:
end function
J.3.3
DYNAMIC GRAPH CONSTRUCTION ALGORITHM
We employ multiple strategies to generate graphs with specific chromatic properties:
The chromatic number estimation for random graphs uses the Lovász theta function as a lower bound and Brooks' theorem for the upper bound:
((G) S x(G) < A(G) +1
(125)
where (GG) is the clique number and AGG) is the maximum degree.
J.3.4 SOLUTION VERIFICATION AND VALIDATION
Verification of proposed coloring requires checking all edge constraints:
Algorithm 14 Verify a graph coloring assignment
1: function VERIFYCOL.ORING(C = (V
2: k f- max,ev ((v)
Number of colors used
3:
for each edge (u, v) E'
4:
if f(u) = f(v) then
5:
return (Inealid, "Adjacent vertices share color")
6:
end if end for
7:
8:
for each u E do
9:
if f(v) {1...., then return (Invalid, "Invalid color assignment") end if end for
10:
11:
12:
13:
if &:
x(G) then return (Suboptimal, k -
14:
end if
16: return (Valid,0) 0)
end function
The verification complexity is O(FF) [V), linear in the graph size.
68

---

2 Published as a conference paper at ICLR 2026
J.3.5 COMPLEXITY ANALYSIS AND HARDNESS GUARANTEES
Graph coloring NP-complete for & 2 3 colors, ensuring computational difficulty even for moderate graph sizes. The decision problem "Can G be colored with colors?" requires exploring search space of size k", growing exponentially with vertex count.
For our generated instances, we calibrate difficulty through several parameters:
Vertex count n E (8, 12, 16, 20, 24, 28}
Edge density m[n- 1)
Chromatic number x(G) relative to maximum degree A(G)
The phase transition in random graph coloring occurs at edge density:
2k k
Pe(k)
(126)
k-1
Graphs near this critical density maximize solution difficulty, requiring sophisticated constraint propagation rather than simple greedy approaches.
J.3.6 CONTEXT WINDOW OPTIMIZATION
The problem representation requires O([V
EE) tokens to specify the graph structure. For our maximum configuration with 28 vertices and dense connectivity (p = 0.6), this yields:
28
Tokens 2 0.6
28 + 453 481
(127)
The solution requires exactly |V| color assignments, adding approximately 2 tokens. The total context requirement remains under 600 tokens for our most complex instances, well within model capabilities while maintaining problem difficulty.
J.3.7 CONTAMINATION RESISTANCE THROUGH GRAPH DIVERSITY
Our framework generates graphs from multiple families with varying structural properties. For vertices, the number distinct graphs is 2(,), astronomically large even for modest n. While we sample from specific families to ensure known chromatic numbers, the diversity remains substantial.
Consider the parameter space for our generation:
6 vertex counts: {8, 12, 16, 20, 24,
8 graph types: {complele, cycle, free, biparlile, planar, wheel, grid, dense_ reandon
Edge density variations for random graphs: continuous parameter
Vertex labeling permutations: n!! for each graph
The total number of distinct problem instances exceeds 15 making memorization infeasible. Moreover, isomorphic graphs with different vertex labelings require the same logical reasoning but different solution representations, further preventing pattern memorization.
J.3.8 THEORETICAL FOUNDATIONS AND BROOKS' THEOREM
Our evaluation leverages Brooks' theorem, which provides tight bounds chromatic numbers for most graphs:
Brooks' Theorem: For connected graph G that is neither complete odd cycle, X(G) < A(G) where ( is the maximum vertex degree.
This theorem enables to generate graphs with predictable chromatic properties. For instance, by constructing graphs with maximum degree A that are neither complete nor odd cycles, we guarantee (G) (A A) providing narrow range for evaluation.
69

---

Published as a conference paper at ICLR 2026
Algorithm 15 Welsh-Powell graph coloring heuristic
1: function WELSHPOWELL(C -V, E))
2: Sort vertices by degree: d(vi) 2 d(v2) 2 d(vn)
3: color 4-1
4: while uncolored vertices exist do
5: U set of uncolored vertices
6:
for v E U in degree order do
v has no neighbor colored color then f(v) & color end if 10: end for
7:
8:
9:
11:
color + color +
12: end while
13: return f
14:
end function
The Welsh-Powell algorithm provides constructive upper bound:
This algorithm achieves x(G) S [AG))+1 in (n²²) time, providing a baseline for model performance
evaluation.
J.4 BOOLEAN SATISFIABILITY: LOGICAL REASONING AND CONSTRAINT RESOLUTION
J.4.1 PROBLEM FORMULATION
The Boolean Satisfiability Problem (SAT) asks whether there exists a truth assignment Boolean variables that satisfies given propositional logic formula. We work with formulas in Conjunctive Normal Form (CNF), expressed as:
C, = A
V 4 JEJ.
(128)
1=1
1=1
where each clause C, consists of literals l,, and each literal is either a variable zy or its negation ouk. A formula ( over variables X (n) is satisfiable if there exists assignment
a: X + 1) such that ((a)) =1.
The search space contains 2n possible assignments, making exhaustive enumeration infeasible for large n. Our framework generates satisfiable formulas with known solutions, enabling precise evaluation of logical reasoning capabilities.
J.4.2 CONTROLLED SAT INSTANCE GENERATION
We employ multiple generation strategies to create SAT instances with varying difficulty charac- teristics. Each strategy produces formulas with guaranteed satisfiability while controlling specific complexity parameters.
For random k-SAT generation, we first fix a satisfying assignment a", then generate clauses that respect this assignment:
This approach guarantees that a* satisfies the generated formula while maintaining randomness in clause structure.
J.4.3 SPECIALIZED SAT VARIANTS
We generate several SAT variants with distinct computational properties:
70

---

Published as a conference paper at ICLR 2026
Algorithm 16 Generate satisfiable k-SAT formula
1:
function GENVERRTESSAIISTAABLSSAAAI m, k)
2:
a² random assignment to (#1, un.
3: Φ
4 0
D Empty formula
4:
for = 1 to m do
5:
C + 0
> New clause
6:
vars <- sample k distinct variables from for each 2;j vars do
7:
8:
Random() < 0.5 then
9:
t
10:
else
L -r end if C Cu end for if C(a") = 0 then
12:
14:
15:
Clause not satisfied
16:
Fix random literal in C' to satisfy a²
17:
end T + + AC end for
18:
19:
20: return (,)a")
21:
end function
Each clause contains at most one positive literal. Horn formulas admit polynomial-tin time satisfiability checking through unit propagation:
Horn-SAT:
CHarn
(-21 V -x2 V 4z) A (-xa V x4) A (x1)
(129)
XOR-SAT:
Clauses encode XOR constraints where exactly one literal must be true. This variant connects to linear algebra over F2:
11 ① 22
1 Z3 = = (x1 V 22 V x3) A (-m1 V T2 V 1z) A...
(130)
Mixed-SAT:
Combines unit clauses, binary clauses, and longer clauses to create varied constraint
densities:
miined
(x1)
(-r1 V
(22
V
(131)
binary
ternary
J.4.4 PHASE TRANSITION AND DIFFICULTY CALIBRATION
The difficulty of random k-SAT exhibits a sharp phase transition at the clause-to-variable ratio:
ae(k) 2* In (1 2n + o()
(132)
For 3-SAT, ag & 4.267. Formulas near this critical ratio typically hardest to solve, requiring sophisticated search strategies beyond simple heuristics.
We calibrate difficulty through multiple parameters:
Number of variables: n E { {4,6, 6, 8, 12)
Clause-to-variile ratio: E {2.5, 3.0, 4.0, 4.5)
Clause length distribution: uniform, mixed, or variable
Literal polarity bias: balanced or skewed
The expected number of satisfying assignments for random k-SAT is:
B[Nsat] 2n (1 2-k)
(133)
When m/n > ls In 2/ 1n(22//(2 /(2 - 1)), this expectation approaches zero, indicating the unsatisfiable phase.
71

---

Published as a conference paper at ICLR 2026
J.4.5 SOLUTION VERIFICATION ALGORITHM
Verification proposed assignment requires evaluating the CNF formula:
Algorithm 17 Verify SAT assignment against CNF formula
1:
function VERIFYSAT($
N=1 C X (0,1))
2:
for each clause C = V do
3: satis fied < False
4:
for each literal Ej in C; do
5:
if = Ty and ((r))) = then
6:
satisfied + True else if L, 22 and a(xx)) = 0 then satis satisfied t- True
7:
8:
9:
end if end for if not satisfie then
10:
11:
12:
return (Invalid, C;)
Unsatisfied clause
13:
end if end for
14:
15:
return (Valid, 0)
16:
end function
Verification complexity is (m k) where m is the number of clauses and ls is the maximum clause length, making efficient despite the exponential solution complexity.
J.4.6 CONTEXT WINDOW ANALYSIS
The formula representation requires approximately 3mnk tokens express clauses of length & each (accounting for variables, operators, and parentheses). For our maximum configuration with n = 12 variables and 48 clauses (ratio 4.0) of length 3:
Tokens = 3 48 3 = 432
(134)
The solution requires exactly n Boolean assignments, approximately 3n tokens when formatted. The total context requirement remains under 500 tokens, enabling complex logical reasoning within context constraints.
J.4.7 UNIT PROPAGATION AND INFERENCE RULES
Our evaluation tests understanding of key SAT-solving techniques. Unit propagation represents the fundamental inference rule:
Unit Clause Rule: If clause contains only literal f, then l must be true in any satisfying assignment.
This leads the propagation algorithm:
Models that correctly identify and apply unit propagation demonstrate understanding of logical implication chains.
J.4.8 CONTAMINATION RESISTANCE ANALYSIS
The SAT problem space provides exceptional contamination resistance through several mechanisms:
Instance Diversity: For n variables and m clauses of length k, the number of distinct CNF formulas
is:
'2n m
N formaulas
(135)
For 10, m 30, k this yields approximately 10 distinct formulas.
72

---

Published as a conference paper at ICLR 2026
Algorithm 18 Unit propagation rule for SAT solving
1:
function UNTTRROAAATTE(0.0O
2: while unit clause (1) &
3: if [ =f/=aa 24 then
4:
(xx,) +1
5: else
> bI=
6:
a(x;) +
7:
end if
8:
Remove all clauses containing / Remove -/ from all clauses if empty clause exists then
9:
10:
return UNSAT 12: end if
end while
14: return (
Simplified formula and partial assignment
15:
end function
Solution Uniqueness: While generate formulas with known satisfying assignments, most formu- las admit multiple solutions. The fraction assignments satisfying a random formula is approxi- mately:
Psat 0a/2
(136)
This multiplicity prevents simple memorization strategies.
Representation Variability: The same formula admits numerous equivalent representations through:
Variable renaming: n! permutations
Clause reordering: m! permutations
Literal reordering within clauses: (A:)) permutations
Logical equivalences: (x v) - -y)
J.4.9 THEORETICAL COMPLEXITY AND HARDNESS
SAT is the canonical NP-complete problem, with several important complexity properties:
Cook-Levin Theorem: Every problem in NP reduces to SAT in polynomial time, making it universal for nondeterministic polynomial computation.
Exponential Time Hypothesis (ETH): There exists algorithm solving SAT in time 20(n), implying fundamental hardness.
Resolution Proof Complexity: For unsatisfiable formulas, the shortest resolution proof can require exponential length:
Proof ( = ((22/10)
(137)
These theoretical foundations ensure that our SAT tasks cannot be solved through simple heuristics or pattern matching, requiring genuine logical reasoning capabilities.
J.5
SUDOKU SOLVING: CONSTRAINT PROPAGATION IN STRUCTURED GRIDS
J.5.1
PROBLEM FORMULATION
Sudoku represents constraint satisfaction problem structured grid, requiring systematic de- duction and constraint propagation. For an n x n grid divided into /n /n boxes, we define the problem finding function f [n] [m] [n] such that:
73

---

Published as a conference paper at ICLR 2026
E fm)
f(,
(row constraint)
(138)
VjE vb E B, (12.j2) f(1..1) * f(22 (box constraint)
f(22.j)
(column constraint)
(139)
(140)
where B partitions the grid into non-overlapping boxes. A Sudoku puzzle provides partial function fo : S + [n] where C [n] x [n], and we seek a complete function f extending fu-
J.5.2 MATHEMATICAL STRUCTURE AND LATIN SQUARES
completed Sudoku grid forms Latin square with additional box constraints. The number of valid X Sudoku grids (for = k2) is:
R(k-1)
N = n!. (n - 1)!
Rr
(141)
where Rn accounts for reduced Latin squares satisfying box constraints. For standard 9x9 Sudoku:
Ng = 9! 8! 22 Rg 6.67 10
(142)
This vast solution space ensures that memorization- basddd approaches remain infeasible.
J.5.3 PUZZLE GENERATION WITH CONTROLLED DIFFICULTY
Our generation algorithm creates puzzles with unique solutions and calibrated difficulty levels. The process involves two phases: grid completion and strategic cell removal.
Phase 1: Complete Grid Generation
We generate valid complete Sudoku grid using Las Vegas randomization with backtracking:
Algorithm 19 Complete grid generation for Sudoku using recursive backtracking with randomized diagonal initialization.
1: function GENNRATECOMPLEEEGRIIUY))
2: grid < empty n matrix
3: Fill diagonal boxes with random permutations
Non-interfering
4:
for each empty cell (i,j) in order do
5:
candidates nI \ (onfficts(i..)
6:
Shuffle(candidatees)
7:
for each val candidates do grid[ + val
8:
if RecursiveFilllgrid, next cell) then return grid end if 12: end for
9:
10:
11:
grid[i]li] 14: end for
Backtrack
return grid
16:
end function
Phase Strategic Cell Removal
We remove cells while maintaining solution uniqueness, with removal patterns determining difficulty:
The removal count follows empirically validated ranges:
Grid Size
Easy
Medium
Hard
4
X
4-6
7-9
10-11
6
6
10-14
15-20
21-26
9
35-40
41-50
51-64
74

---

Published as a conference paper at ICLR 2026
Algorithm 20 Puzzle generation from complete grid by controlled cell removal with uniqueness preservation.
1: function GENTRATTFPUZZLEECommleee...irr. di ficalll)
2: puzzle t complete _etiieete_______eee
3: cells tt__ttorrnnnn remove - CaluulaacRemovvalCuutiii ficultyy
4: removal order SeleciRemovalSStateyydddd ficutt)))
5:
for each (i:j) in removal do val pu-zzleee puzzle[i[]] + 0 if not HasuuniquuSllluuronooozeeee)) then puzzle(i][] + val
6:
7:
8:
9:
> Restore multiple solutions
else
11:
cells vo_remove + cells to_remove
end if if wells_to_remore =0 then
13:
14:
Break end if end for
15:
16:
17:
return puzzle
18:
end function
J.5.4 SOLUTION UNIQUENESS VERIFICATION
Ensuring unique solutions is critical for unambiguous evaluation. We employ modified backtracking solver that detects multiple solutions:
Algorithm 21 Solution counting via recursive backtracking to ensure puzzle uniqueness.
1: function CoUNTSOLUTIONS(pu:zee, max __c_untttt = 2)
2: count +
3: function SOLVE(row,co
4: if row = n then
5:
count <- count + 1 return count < marr_count end if
6:
Continue if need more
7:
8:
(next_ neeet_row,, nert [eer]_oo))= + GetNexxCell(rouu.oo col)
9:
if muuzle/row]]ced & 0 then
10:
return Solve Solivenexl_rowe next_col) end if for val E (1....,n) do
11:
12:
if Is Valid(puzzle, row, col, val) then puzzle(roo]]cnn < val
14:
if not Solve(nezt, row, next_col then
16:
puzzle(row) [col] + 0
return False end if puzzle(rown [col] t- end if end for
Found enough solutions
18:
19:
20:
21:
22:
return True
23:
end function
24:
Solve(O, 0)
25:
return count
end function
75

---

Published as a conference paper at ICLR 2026
J.5.5 CONSTRAINT PROPAGATION TECHNIQUES
Efficient Sudoku solving employs constraint propagation reduce the search space. The naked singles rule identifies cells with only one possible value:
Candidates j) = [x] \ (Row; U Col, U ()x;,)
(143)
When [Candidates(t, j)] 1, the cell value is determined. Hidden singles occur when a value appears only one cell'ss candidates within unit:
Yu E [n]] ji. Unit Candidates(i..)
f(i.j)
(144)
Advanced techniques include:
Naked Pairs/Triples: If cells in unit have the same i candidates, eliminate those values from other cells
X-Wing: If value appears only two cells two rows, and these cells align in columns, eliminate from other cells in those columns
Swordfish: Generalization of X- X-Wing to three rows/columns
J.5.6 SUDOKU VARIANTS AND EXTENDED CONSTRAINTS
We implement multiple Sudoku variants to increase evaluation diversity:
Diagonal Sudoku: Additional constraints on main diagonals:
Vi # is [n] ((),,)) + f(2,, i2) f(,, + + f(ig, + 1)
(145)
Irregular Sudoku: Replace regular boxes with irregular regions R1, Rg where:
UR.
[ R = 0 |
(146)
Killer Sudoku:
Add cage constraints with sum requirements:
Vcage C' = ter,,.., Cm) M(a) target(()) A all ((a) distinct
(147)
J.5.7 CONTEXT WINDOW EFFICIENCY
The problem representation scales quadratically with grid size. For an n X n puzzle with given cells:
TokenSpuzze As 3n² + 5k
(148)
The solution requires exactly n² values, approximately 2nd tokens when formatted. For 9x9 Sudoku: TokenSwoll R3 3(81) + 5(30) + 2(81) = 555
(149)
This efficient representation enables complex reasoning within typical context limits.
J.5.8 CONTAMINATION RESISTANCE MECHANISMS
Sudoku provides robust contamination resistance through multiple factors:
Puzzle Space Size: The number of minimal Sudoku puzzles (irreducible with unique solutions) exceeds 1026 for 9x9 grids. Even storing a fraction of these would require petabytes of memory.
Isomorphic Transformations: Each puzzle admits 9! 68 2 1.2 109 equivalent forms through:
76

---

Published as a conference paper at ICLR 2026
Symbol permutations: 9!
Row permutations within bands: 6
Column permutations within stacks: 63
Band permutations:
Stack permutations: 3!
Transposition: 2
Dynamic Difficulty Adjustment: By varying the number and pattern of given cells, we create puzzles requiring different solution techniques, from simple elimination complex pattern recognition.
J.5.9 COMPUTATIONAL COMPLEXITY
Sudoku NP-complete for generalized grids, proven through reduction from Latin square completion. The decision problem "Does this partial grid have a valid completion?" requires exploring potentially exponential search spaces.
For n x Sudoku with given cells, the worst-case complexity is:
T(n, k) Onn²²~R
(150)
However, constraint propagation typically reduces this dramatically. The average-case complexity for well-formed puzzles with unique solutions is approximately:
Tavg (n, k) On 64)
(151)
where b is the effective branching factor after constraint propagation and is the search depth. For typical puzzles, b Rv 2 3 and d Rs n²/4, making solution tractable while maintaining difficulty.
J.5.10 THEORETICAL GUARANTEES
Our Sudoku generation provides several mathematical guarantees:
Every generated puzzle has exactly one solution, verified through exhaustive search with early termination upon finding a second solution.
Uniqueness:
Minimality: For hard difficulty, puzzles minimal-removing any given cell creates multiple solutions. This verified during generation:
V(i,j) Given [Suuutions(Puzzle (i.ii))) >
(152)
Solvability: All puzzles are solvable through logical deduction without requiring guessing, though the required techniques vary with difficulty level.
These guarantees ensure fair, unambiguous evaluation of model reasoning capabilities.
J.6 CRYPTARITHMETIC. ALGEBRAIC CONSTRAINT RESOLUTION
J.6.1 PROBLEM FORMULATION
Cryptarithmetic puzzles encode arithmetic equations where letters represent unique digits. We formalize this as constraint satisfaction problem
(L, D. v, @) where {4, l2, ..., 4 denotes the set of unique letters. D = {0, 1, 1,...9 9} represents the digit domain, V T, - D defines bijective mapping satisfying [T|] S 10, and ( (+, -,,-,3 x, }} specifies the arithmetic operation.
Given words W1, Wz,.., Wn composed of letters from L and result word R, the fundamental constraint requires:
(Wi) @ val( @ .. @ val(W..) = val(R)
(153)
where the value function converts word its numerical representation:
77

---

Published as a conference paper at ICLR 2026
W
val(w) Σ
v(WE) 101-1
(154)
J.6.2 CONSTRAINT SYSTEM AND MATHEMATICAL PROPERTIES
The complete constraint system encompasses three categories of restrictions:
Uniqueness constraints ensure bijective mapping:
44,4 E L,i =
(155)
Leading zero constraints prevent degenerate solutions:
VW E (Wi..., Wa, RJ (W) +
(156)
Arithmetic constraints enforce the equation validity. For addition:
val(W,) - val(
(157)
For multiplication with two operands:
val(Wi) x val(W2) val( R)
(158)
The carry propagation in addition creates polynomial constraints. At position from the right, with carry Gp (0, 1, .,......
1 1 p)) + cp-11 [( - 1 + 10.100.0 Cp
(159)
J.6.3 SOLUTION SPACE ANALYSIS AND UNIQUENESS GUARANTEES
The solution space cardinality depends on the number of unique letters k = [T]. Without constraints, we have:
10!
Sunconstraineel
P(10, k)
(160)
(10 k:!
Leading zero constraints reduce this space. With leading letters:
9 (10 - m)!
Steading
(161)
(10 k-)!
To ensure unique solutions, we employ generate-ane and-test approach with constraint density analysis. The constraint density measures the ratio of constraints to variables:
Constraints
le(1: 1)/2 + + [log ( vl)(
=
L
k
(162)
Empirically, unique solutions emerge when > Peritical 1.5 for addition and p > 2.0 for multipli- cation.
78

---

Published as a conference paper at ICLR 2026
J.6.4 DYNAMIC GENERATION ALGORITHM
Our generation algorithm constructs solvable puzzles with controlled difficulty:
Algorithm 22 Generate Ctyptarithmetic Puzzle
1: Select word templates based on length parameters
2: Generate letter assignment ensuring uniqueness
3: Compute arithmetic result
4: Verify solution uniqueness via exhaustive search
5: if multiple solutions exist then
6: Adjust word selection or retry
7: end if
8: Calculate difficulty metrics (constraint density, search space)
9: return puzzle with verified unique solution
The word generation employs pattern templates to control difficulty. For extreme difficulty, we maximize letter overlap between operands and result, creating dense constraint networks:
((W))) Overlap(Wi, W2, R) I[(W)) U L(W2) L(R)
(163)
J.6.5 VERIFICATION THROUGHI CONSTRAINT SATISFACTION
Solution verification employs constraint satisfaction with domain reduction. We initialize domains:
D. ( {1, (1,2.....9) 9) if / is leading letter 9) otherwise
(164)
Arc consistency propagation reduces domains iteratively:
D = DJ n (d D 3 consistent assignment for other variables)
(165)
The verification algorithm has time complexity O(k! p(k)) where p(k) represents the polynomial verification cost per assignment.
J.6.6 CONTEXT WINDOW OPTIMIZATION
The problem representation requires careful encoding to minimize context usage. Each letter-digit assignment needs [log2 (10)] 4 bits. The complete solution encoding requires:
Bitt k [lgg (10)] + O(log k)
(166)
For the problem statement, each word of length w requires:
Bitsward w [loga (26)1 = 5w
(167)
Total context requirement for operands with average word length w:
Contextioal O(n + 1) w log(26) log(10))
(168)
J.6.7 CONTAMINATION RESISTANCE ANALYSIS
The contamination resistance stems from combinatorial explosion in the generation space. With vocabulary size V per word length and & unique letters:
Problem Space = V". (26)
P(10, k)
(169)
79

---

Published as a conference paper at ICLR 2026
For typical parameters (n 2, 8, V 100), this yields:
266
10!
PP| 10a
10
(170)
21
The probability of encountering identical puzzles remains negligible even with extensive training data of size [D] 10":
[
P(contamination)
10-6
(171)
PP
J.7 MATRIX CHAIN MULTIPLICATION DYNAMIC PROGRAMMING OPTIMIZATION
J.7.1 PROBLEM FORMULATION
The matrix chain multiplication problem seeks the optimal parenthesization to minimize scalar multiplications when computing the product M Ai x A an optimization problem M (D, II, cost) where D = (do, dl, ..., dn] represents the dimension sequence with matrix A, having dimensions d,-1 x d,, denotes the set valid parenthesizations, and cost II + N computes the total scalar multiplications.
An. We formalize this as
For matrices A, of dimension ps x q and A, of dimension qe Ty where qu = Py. the multiplication cost is:
cost(A, A²) = Pi - TT
(172)
The number of valid parenthesizations follows the Catalan number:
|T = Cn-1 1 222n 1) n -
4nh
(173)
n2/2
J.7.2 DYNAMIC PROGRAMMING FORMULATION
We define the optimal substructure through the recurrence relation. Let m(i. denote the minimum cost to compute A; x Ait1 x X Ay:
(
ifi =j mmin b] mmjk + 1, + d:-1 de dz} if <
m|i.j]
(174)
The optimal split point sii, satisfies:
s|i..] = arg min {m[i, k] + m[k + 1, + d=-1 de dz} iskc
(175)
J.7.3 ALGORITHMIC SOLUTION AND COMPLEXITY
The bottom-up dynamic programming algorithm builds solutions for increasing chain lengths:
80

---

Published as a conference paper at ICLR 2026
Algorithm 23 Matrix Chain Order
1:
for = 1 to do
2:
m [] 4
3: end for
4:
for
no do
P Chain length
5:
for iori=1to1
do
6:
j j+iil
7:
m[i.j] < -0
8:
for = =itoj j do
9:
q mli,
m [k 1,5]
d:-1 de dj
10:
if < m m. then
m +- 9
12: end if
s|i.j] t k
14:
end for
end for
16:
end for
17:
return m [1, nl]
The algorithm exhibits time complexity O(n²²) and space complexity (nn²):
T(n) = Σ a-++1
3 n
(- (n
(176)
J.7.4 PROBLEM GENERATION WITH DIMENSION PATTERNS
We generate dimension sequences following specific patterns to create problems with varying opti- mization characteristics:
Random pattern with uniform distribution:
d; ~ Uniformja, b). i E [0,n]
(177)
Increasing pattern encourage early splits:
d; = do i 5, 8 ~~ Uniform 5, 15]
(178)
Sparse pattern with bimodal distribution:
Uniform[1 [, 15)
with probability p
(179)
Uniform 880, 120]
with probability
The optimization ratio quantifies the improvement over naive left-to-right multiplication:
cOStnaive
Σ dg de d++1
(180)
cOStoptimal
m|1,n1 n
J.7.5 SOLUTION VERIFICATION AND UNIQUENESS
While multiple parenthesizations may achieve the optimal cost, we verify solution correctness through value comparison. The verification employs recursive validation:
COStclaimed = 0
ii 3k coStclaimed = m mili, k] + m[(: 1,5] + di-1 de dz if <j j
verify(i.j. costchimed)
(181)
The solution uniqueness in terms of minimum (not parenthesization) is guaranteed by the optimal substructure property and the principle of optimality:
v m(i.j] = min cost(T)
(182)
#ell.,
81

---

Published as a conference paper at ICLR 2026
J.7.6 CONTEXT WINDOW ANALYSIS
The problem encoding requires minimal context. The dimension sequence needs:
BitSaimensions (n + 1) [log, (max d:)
(183)
The solution encoding requires the minimum cost value:
Bittsoolution
(m < (n max df)1
(184)
For typical parameters with [3, [0) and d, [1, 120], the total context requirement remains under 500 tokens:
Context On log max d,)
(185)
J.7.7 CONTAMINATION RESISTANCE PROPERTIES
The contamination resistance emerges from the continuous dimension space and pattern variety. With dimension range (a, b] and n + dimensions:
DD continnous
= (b - a 1)
(186)
For discrete dimensions with possible values each:
Ddiscrete|
(187)
With five generation patterns and variable chain lengths, the total problem space becomes:
f.lma - Amiin +1
IP
Σ
5.
5min +1
(188)
22 mmnn
For parameters 100, 7mmin
3, 7max
20:
100
P 5 100
10
(189)
99
J.8 MODULAR SYSTEMS SOLVER: NUMBER-THHOOETICC CONSTRAINT RESOLUTION
J.8.1 PROBLEM FORMULATION
modular system comprises simultancous congruences with additional number- number-theorets theoretic constraints. We formalize this S (E A, Z) where & (x a, (mod m.) E k]) represents the congruence system, A denotes additional constraints (primality, range, divisibility), and the solution z satisfies all conditions.
The basic congruence system has the form:
(mod mı)
(190)
x = 02 (mod m2)
(191)
(192)
T ax (mod m&)
(193)
solution exists if and only the compatibility condition holds:
Vi,j a.=a= = (mod (m,, m,))
(194)
82

---

Published as a conference paper at ICLR 2026
J.8.2 CHINESE REMAINDER THEOREM AND GENERALIZATION
For pairwise coprime moduli where ged(m;; m2) for all i + j. the Chinese Remainder Theorem guarantees a unique solution modulo Il m.
ag May (mod M)
(195)
where M, = M/m; and yı satisfies M,y: = 1 (mod m²).
For non-coprime moduli, employ the generalized solution method. Given congruences:
x = (mod m1)
(196)
I 02 (mod m2)
(197)
Let ged(m1, m2). A solution exists if a = (mod g). yielding:
(02 01) - (mod lem(m1, mz))
x + m1
(198)
where satisfies mnt (mod m2).
J.8.3 EXTENDED EUCLIDEAN ALGORITHM AND MODULAR INVERSE
The extended Euclidean algorithm computes ged(a, b) and coefficients t such that: as bt ga,(a,
(199)
Algorithm 24 Extended Euclidean Algorithm
1: function EXTENDEDGCD(a,)) b)
2: if a then
3:
return (b, 0,
4: end
5: (g. ExtendedGCI((b mod a.
6:
I+ [b/a]
8:
return (g,x,
9:
end function
The modular inverse a1 (mod m) exists if ggdd(a,m) = 1:
a- a-a --= (mod m) = a -1 -I= (mod m) where ar + my =
(200)
J.8.4 ADDITIONAL CONSTRAINT TYPES AND SATISFACTION
We augment the basic system with number- theoreei constraints: Primality constraints require the solution to be prime:
Aprime {ar : x E)
(201)
Range constraints bound the solution:
Arango = <
(202)
Divisibility constraints impose factor requirements:
Adiv {x : :d:d x}
(203)
The complete solution must satisfy:
x
{o x a; (mod m2)3 n M,
(204)
83

---

Published as a conference paper at ICLR 2026
J.8.5 SOLUTION ALGORITHM WITH CONSTRAINT PROPAGATION
Our algorithm combines modular arithmetic with constraint satisfaction:
Algorithm 25 Solve Modular System with Constraints
1: Zo f- SolveCongruences(()
Base solution
2: M -- lem(m f12 m.)
D Period
3: x + Iо
4: for k == 0 to search limit/M do
5: I +- Za + KM
6: if SatissiesConstrainttt::: .A) then
7:
return r 8: end if
9: end for
return NONE
The search space is periodic with period M, ensuring finite exploration:
x
2o (mod M)
I = Zo + KM, 1 E Z
(205)
J.8.6 COMPLEXITY ANALYSIS AND SOLUTION UNIQUENESS
The time complexity depends on the constraint types and search space:
T(k, M, A) = Ok2 log M) + O(A/M C)
(206)
where A represents the search limit and C' denotes constraint checking cost.
For primality testing using trial division:
Cprime = ((V=)==(VV)
(207)
Solution uniqueness within a bounded range [a:, B] occurs when:
B a a < M = 1fz E [a 20 (mod M)H < 1
(208)
J.8.7 PROBLEM GENERATION CONTROLLED DIFFICULTY
We generate problems with guaranteed solutions through constructive methods:
Algorithm 26 Generate Modular System
1: Select coprime moduli m....., m7
2: Generate random remainders ..., Cg
3: Compute base solution To using CRT
4: Add constraints based on difficulty level
5: Search for satisfying solution x²
6: Verify uniqueness in reasonable range
7: return (E,A,
The modulí selection ensures coprimality:
1
6
P(coprime)
II i<j
min(m,, m))
II
(209)
pa
2
84

---

Published as a conference paper at ICLR 2026
J.8.8 CONTEXT WINDOW OPTIMIZATION
Each congruence requires logarithmic encoding:
BotSenngeeence = [log2 mg + [log2 azl = Ollog m,)
(210)
The complete system encoding:
&
BitSsystem
(loo m,) Σ (it)(AA))
(211)
For typical parameters with & [3, 5) and my 100:
Contextoral Ok log max m.) = O(k 7) < 50 bits
(212)
J.8.9 CONTAMINATION RESISTANCE ANALYSIS
The problem space exhibits exponential growth in multiple dimensions:
[P] =
max
m
|Aypes ||A|
(213)
where Pmax denotes the maximum prime considered. For = 4, primes up to 50, and 3 constraint types:
15
IPI
30
32..3 > 109
(214)
4
The sensitivity to parameter changes ensures diversity:
Da
Ax
M, (mod M)
(215)
where typically Myy > 1, causing large solution variations.
J.9 CONSTRAINT OPTIMIZATION: MULTI-OBJECTVE RESOURCE ALLOCATION
1.9.1 PROBLEM FORMULATION
The multi multi-constraint resource allocation problem represents complex integer programming challenge.
We formalize this as O = (P, R, C) f) where P {p) .?,..... P.. denotes the project set,
(r1, r.,,.., Fm represents resources with capacities cz encompasses constraints, and ʃ 22 IR defines the objective function.
Each project p; has attributes:
requirements
Pi R N
(216)
profit:
EN
(217)
quality
di [1,
(218)
risk :
E [1.5]
(219)
dependencies:
8, C
(220)
The decision variables form a binary selection vector:
(1
if project pt is selected otherwise
x4 =
(221)
85

---

Published as a conference paper at ICLR 2026
J.9.2 CONSTRAINT SYSTEM AND MATHEMATICAL FORMULATION
The complete optimization problem becomes:
maximize
(222)
subject
Σ Pa(r,) ( Sg.
Vj [1.m]
(223)
34
Ik.
Vps E
(224)
Σ i= 9:
2 Qmin
(225)
i== 24
1= 7/2
<Tmax
(226)
E Iz.
I3 E {0,1}, [
(227)
The resource constraints ensure capacity limits:
Cresource {x E 10,1)" Ar sc
(228)
where AJs = pr(rr) forms the requirement matrix.
Dependency constraints create partial order:
Cdependency {rr E [0,1)" Vi, vk E 8, : S xx
(229)
J.9.3 SOLUTION ALGORITHM: BRANCH AND BOUND WITH CONSTRAINT PROPAGATION
For small instances (n < 10), we employ exhaustive enumeration with pruning:
Algorithm 27 Branch and Bound Optimization
1: function OPTIMIZE(P. RG C)
2: best 0, best best_proffttttttte
3: stack + [00, P.O)]
> (selected, remaining, profit)
4: while stack + 0 do
5: (S, R, π < stack.pop(
6:
UpperBound( (t) S best_ then
7:
continue
Prune branch
8:
end
9:
if IsFeasible(S,CC and best prooittooitttt then best + S, best Bocctpprffttttt + π end if
10:
11:
12:
for p E R do CanAdd U (p),C) C) then stack, (Sach(UU)))) {p), R (p), +T))
14:
end if end for end while
16:
17:
18:
return best
end function
The upper bound computation uses linear relaxation:
up min 1, min C used,
UpperBound(
R)
Σ
(230)
(p(rz)
PER
86

---

Published as a conference paper at ICLR 2026
J.9.4 GREEDY APPROXIMATION FOR LARGE INSTANCES
For larger problems, we employ a greedy algorithm with local search:
efficieecy(ps)) 2 Wi Pe(rg)/og
(231)
where wy represents resource scarcity weights.
The approximation ratio for the greedy algorithm:
Greedy
(232)
Optimal
max, E, pi(rs)/c/
J.9.5 PROBLEM GENERATION CONTROLLED DIFFICULTY
We generate problems with varying constraint tightness:
min, (i(r))
T
(233)
2-=1
For basic problems, T E [0.3, 0.5] ensures multiple feasible solutions. For mixed constraint problems, E [0.6, 0.8] creates tighter resource competition.
The dependency graph generation follows:
0 ifi j
P P(dependency +)
(234)
if fiij
ensuring acyclic structure.
J.9.6 SOLUTION VERIFICATION AND OPTIMALITY
Verification checks all constraints in polynomial time:
Algorithm 28 Verify Solution Feasibility
function VERIFY(S, P, R,C) for r R EpEes Pr(r;) > cj then
2:
3:
4:
return FALSE
Resource violation
5:
end if end for
6:
7:
for Pa E S do
8:
if 8 Ø S then return FALSE
9:
Dependency violation
10:
end end for
12: if avg(g: P: E -) < Qmin or avg(n: :4 E S) > Tmax then
13:
return FALSE
Quality/Risk violation
14: end if
return TRUE
16:
end function
Near-optimaliy tolerance allows solutions within 95% of optimal:
TRUE ( 2 0.95
Accept(S)
(235)
FALSE otherwise
87

---

Published as a conference paper at ICLR 2026
J.9.7 COMPLEXITY ANALYSIS
The problem is NP-hard, reducible from the 0-1 knapsack problem. The solution space has cardinality:
S 2"
(236)
However, constraints significantly reduce the feasible region:
F < 2"
(237)
The branch-and- algorithm has worst-case complexity O(2" poly(n; m)) but average-case performance improves with effective pruning:
([T(n)) O(b" poly(n, m))
(238)
where < represents the effective branching factor after pruning.
J.9.8 CONTEXT WINDOW REQUIREMENTS
Each project encoding requires:
BitSproject (( log max Cy + log Tiax log n)
(239)
The complete problem instance:
ConteXhoul = Onn m log maxe; + n logn)
(240)
For typical parameters (n E [4, Cj 2000):
Context 0 3 11 + 6- 3) 300 bits
(241)
J.9.9 CONTAMINATION RESISTANCE PROPERTIES
The generation space grows exponentially with multiple parameters:
II (maa
Caax 110 5- 2")
(242)
With continuous profit values and resource requirements:
IP
Ponttinuuus
(mmxx cm ) 50 2n
(243)
For = =5, m = 3, Tmax 1000, Cmax = 100:
P > (109)5 50 25 > 10
(244)
The probability of problem collision remains negligible even with extensive training data.
J.10
LOGIC GRID PUZZLES: CONSTRAINT SATISFACTION THROUGH DEDUCTIVE REASONING
J.10.1 PROBLEM FORMULATION
Logic grid puzzles represent constraint satisfaction problem (CSP) where must assign at- tributes to entities through logical deduction. Formally, we define a logic grid puzzle as a tuple E,, A,C, Φ) where E {El, 62, en, denotes the set of entities, A (Ai, A2, Am) represents attribute categories with each A, {an 0g. .... } containing exactly n distinct values, constitutes the constraint set, and
A U A; defines the assignment function.
The solution space forms Latin square structure where each attribute value appears exactly once category. Mathematically, we require:
We vi E [1.m]. Va E A,, Ale E E $(e, At)
= o(e,, + O(EK, A,)
(245)
(246)
88

---

Published as a conference paper at ICLR 2026
J.10.2 CONSTRAINT TYPES AND LOGICAL REPRESENTATION
We categorize constraints into five fundamental types, each with distinct logical representations:
Direct constraints specify explicit assignments:
Carreaa A,, = Φ(c, A,) = a
(247)
Negative constraints exclude specific assignments:
Crgaative (e, A,, a) = (le, A,) +
(248)
Comparison constraints establish ordinal relationships for numerical attributes: Coomparison (E1, e2, Ai, <) = Φ(e1, A.) ((e2, A.)
(249)
Conditional constraints link attributes across categories:
Conditional Ai, az, Aj, a,) = ve E E ø(c, A,) = 0; = ((e, = aj
(250)
Chain constraints require transitive reasoning:
Cehain (e1, c2, C3 A,, A). = d(e1, A,) < Φ(e2, A,) < p(es, A,) A Øee2, Aj)= =
(251)
J.10.3 DYNAMIC PROBLEM GENERATION WITH UNIQUE SOLUTIONS
Our generation algorithm ensures unique solutions through systematic constraint construction. We begin with random valid assignment 6" and iteratively add minimal sufficient constraints.
Algorithm 29 Generate Logic Grid Puzzle with Unique Solution
1: 6² + Random Vanddom\liidAsssigmennnE,,
2: C <- ()
3: 4 < (enee
4: while [Solutions(C)) do
5: c+ SeleccConstraint((" U, difficulty)
6: C+ (c}
7: u Facts(c)
8: if |Solutions(C) 0 then
9:
return FAILURE
end
11: end while
12: return (C)
The constraint selection strategy employs weighted sampling based on difficulty level. For difficulty d E (easy, medium, hard, extreme). define weight vectors Wid (Wdireet: Wnegative, Woomparison Weonditional, Wehnin) where harder difficulties favor complex constraint types.
J.10.4 SOLUTION VERIFICATION THROUGH CONSTRAINT PROPAGATION
We verify solution uniqueness using arc consistency and constraint propagation. The domain for each entity-atrribute pair initially contains all possible values:
D = Ve E.VA, e A
(252)
Constraint propagation iteratively reduces domains:
Dt11 D'',A nfa E A, consistent(a,,. e, A,, C, C')3
(253)
The solution is unique if and only if:
Ve E, E A: ( =1
(254)
89

---

Published as a conference paper at ICLR 2026
J.10.5 COMPLEXITY ANALYSIS AND CONTEXT WINDOW BOUNDS
The solution space has cardinality (n!)==1 since fixing one attribute calegory determines Latin square structure. The constraint satisfaction problem is NP-complete, requiring exponential time in the worst case.
For context window estimation, each constraint requires O(log n + log m) bits to encode, and minimal constraint has size ((nm). The solution description requires O(mm log n) bits. Therefore, the total context requirement is:
Context(n, m.) OInm(log n + log m))
(255)
For typical parameters n E [3.5] 5] and m E [3, 5]. this yields manageable context requirements of 100-500 tokens.
J.16
CONTAMINATION RESISTANCE PROPERTIES
The contamination resistance emerges from three key properties:
First, the parameter space for valid assignments has cardinality:
n! (n!)
(256)
1=1
Second, the constraint generation employs randomization at multiple levels including entity selection, attribute selection, and constraint type weighting. This introduces entropy:
H(C)
Σ logp(c) 2 log (C]
(257)
GEC
Third, the semantic variation in entity and attribute names drawn from large pools ensures surface- level diversity even for structurally similar puzzles. With vocabulary pools size V per category:
Variations
(258)
Combined, these properties ensure that the probability of generating identical puzzles across indepen- dent runs remains negligible:
P(collision) S (rd)m Vnm
(259)
For generalized Tower of Hanoi with multiple pegs and disks For the decision version with pre-placed queens
90

---

Published as a conference paper at ICLR 2026
Table 17: Mathematical Characterization of Hard Suite Tasks (Tasks 1-5 of 10)
Property
Tower of Hanol variations)
N-Queens variations)
Graph Coloring (10 variations)
Bookean SAT
Sudoku variations)
(5 variations)
Problem Formulation
State Space
3" configurations
ne placements
k colorings
2" ussignments
n" grids xy (g given)
Search Space
(3-2- 2- nn) Ordering & size Path lengsh
permutations
assignments
2" troth values Clause satisfaction
Constraint Type
Non attacking
Adjucent difference
Row/col/bxx unique NIA (completion)
Optimization Target
NAA (fensibility)
Chromatic numher
N/A (feasibilitt)
Solution Properties
Solution Count
(unique optimal) 2n moves
S(m) (OEIS A000170)
vary) xIG) colors
to 2
(by design) Unique completion O(u²) tnkens O(w²)
Optimal Solution
Any valid placement
Any satisfying o Ó(n) tokens O(m k))
Solurion Length
O(2") tokens 012 m
O(v) tokens
O(e) tokens
O(a²)
OQE IV)
Complexity Anulysis
Time Complexity
Org") optimal
Oix!) worst-case
NP-somplete (A
NP-complete Ofe + m) formula NP-complete MAX-SAT: 0.85
NP-complete O(v²) grid NP-somplete N/A (exact)
Space Comploxity
O(n) recursion
O(z)) backtrack
Oin') adjacency NP-complete O(n/ log nj-apprxx
Decision Problem
PSPACE-somplete"
NP-complete!
Apprxximaiilit
NIA (exact)
NIA (exact)
Generation Parameters
Size Range
8. (3.....5) S}
E 6,88) Bourd size Diagonal, Toroidal ((()))
V (8..... 28} Edge density
# [1 (1.....12 12)
Ne 6,96. Cells removed Diagonal. Ineeullar
Difficulty Control
Disk count
Clause ratio ratio mn/o (2))
Variants Count
Pog permutations
Planar, Wheel, Grid 2(3
10 minimal
Contamination Resistance
Isomorphisms
peg ()) 6 10' bits Linear to exponential
symmerries
u! mo (k!)
g8.6".
Problem Diversity
Σ. 8(v) -8
10th graphs
> 10P4 formulas
1040 puzzles
Memory Required
~ bits
Exponential
Exponcutial
Petabytes
Generalization Gup
O(1) to (n())
Fixed to arbisury
Linear to exponential
Quairatic growth
Reusoning Requirements
Primary Strategy
Recunsvve decomp.. calls
Backtruckngg
Constraint prop.
Unit propagation
Constraint
Lookahccd Depth
O(n) decisions
Graph traversal Adjacency
analysis
O(s?, /1) cells Sudoku rules
Constraint Types
Stack ordering Auxiliary use
Geometric conflicts
Logical sutisfaction
Key Insight
Systematic search
Clique detection
Implication chains
Hidden singles
Mathematical Foundations
Core Theary
Recumence relation
Permatation groups
Brooks' theorem
Cook-1 Levin theorem A. Assignment exists ss 4.267
Latin squares f: [n] Exhaustive search
Formula Uniqueness Proof
T(n) 22'n Induction on N/A
31 A ir, z,| Symmetry breaking
x(G) A(G) +
Greedy coloring
Phase Transition
p.(R) 2
threshold
91

---

Published as a conference paper at ICLR 2026
Table 18: Mathematical Characterization of Hard Suite Tasks (Tasks 6-10 of 10)
Property
Logic Grid Cryptarithmetic
Matrix (5 variations)
Modular Systems
Construint Opt. (5 variations)
(8 variations)
(12 variations)
(5 variations)
Problem Formulation
State Space
(sd)=-) assignments
P(10, kj mappings
C. parenthesizations
[1 my solutions M lem(m:;..., me)
2 selections 2a binury vectios Resource capacity Max profit
Seurch Spuce
"h possibilities
[[o-E]
a (G
Constraint Type
Latin square
Digit uniqueness
Associativity
Medular congruence N/A (satisfaction)
Optimization Target
N/A (feasibility)
N/A (feasibility)
Min multipiication
Solution Propertles
Solution Count
(by construction)
1 (verified)
(min cost) m|Lan]
S [|A/M]
Variable
Optimal Solution
Unique assignment
Unique mapping
+ kM
Max Ryt,
Solution Length
Olam) tokens
O(k) tokens
Oliog m(),n]) tokens O(z))
O(loga) tokens OfEllg M)
O(n) tokens O(rm)
Verification Time
Opnm (C])
Olk
Complexity Analysis
Time Complexity
NP-complete
NP-complete
O(s²) DP
O log Mi O(R) congruences P with bounds (exact)
NP-hard (0-1 IlP) O12%) states NP NP-complete 1/p]-
Space Complexity
(nnm) domains
O(k) mapping
O(p²) table
Decision Problem
NP NP-complete
NP-complete
P (polynomial)
Approximaiility
(exact)
(exact)
(exact DP)
Generation Parameters
Size Range
n. m. [3, 5)
& (4.
E (3..... 20)
k (3,4,5) Consiminnt types constraint types
E (4,5, 5,,6,63 63 Constraint tighmess
Difficulty Control
Constraint types
Letter overlap
Dimension patterns
Variants
5 constraint types (ntym. prom
4 operations
patterss
5 problem types (T "mm.
Instance Count
y". (2) P(10,A)
(6 a 4 17m+1 5
(Pg-) I[ m
Contumination Resistance
Isomarphisms
(n1)* permutations
A! letter perms
None (ondered)
kt equation order
n! project labels
Problem Diversity
10 puzzles
10 puzzles
100 instances
10" systems 10" systems
10 problems Exponcutial
Memory Required
Exponential O(owe) (wly*
10th bits
10 instances
Generalization Gap
Linear to factorial
Polynomial to
Linear to periodic
Linear to exponential
Reusoning Requirements
Primary Strategy
Construint
Domain reduction
Dynamic programaning
CRT + search Period Af Modular + additional
Branch & bound decisions
Lookshead Depth
O(um) inferences
k assignments Arithmetic + unique
subproblems
Constraint Types
5 logical types Arc consisiency
Matris dimensioon
Resource quality
Key Insight
Camry propaatton
Optimal suhsrracture
Periodicity
Pruning brunds
Mathematical Foundations
Core Theary
Latin squares Exx A U.A,
Modular arithmetic
Catalan numbers mi]k..] min, (...)
Chinese Remainder
Integer programming uax Σ NuR, B&B enumeration Tightnsss
Formula
R Exhaustive search p
o, (mod mi) CRT uniqueness NAA
Uniqueness Proof
CSP solver
DP oprimality N/A
Phase Transition
Constraint density
92

---

Published as a conference paper at ICLR 2026
Table 19: Algorithmic and Evaluation Metrics for Hard Suite Tasks (Tasks 1-5)
Metric
Tower of Hanoi
N-Queens
Graph Coloring
Boolean SAT
Sudoku
Generation Algorilhms
Generation Method
Deterministic
Backtrack random
Graph construction
Satisfiable CNF
Complete removal
Gencration Time
O(1)
O(r²) average
O(n") edges
((mk) clauses
O(n with uniqueness
Uniqueness Check
Implicit (proven)
Enumcrate all
Chromatic polynomial
SAT solver
Solution counting
Validation Method
State transitions
Conffict checking
Fdge constraints
Formula evaluation
Constraint verification
Context Window Requirements
Prompt Tokens
200 + 10n
100 + 5n
30 2|E]
50 + 3mn 3n
100 + 3ns 2naa
Solution Tokens
15(2" - 1)
2n
2n1
Max Total Tokens
3, 825 (n=8)
120 (n=8) High (linear)
600 (n=28) Medium
500 (n=12) High
550 (9x9) Medium
Token Efficiency
Low (exponential)
Solution Strategies
Optimal Algorithm
Frame-Stewart N/A (exact)
Min-conflicts Row-by-row
Welsh-Powell
DPLL + CDCL
Dancing Links
Heuristic Methods
Greedy coloring
Unit propagation Pure literal 2 (true/false)
Naked singles Constraint prup. S values
Pruning Technique
NIA (binary choice)
Forward checking
Degree ordering
Branching Factor
colors
Evaluation Metries
Success Criterion
Optimal path
Valid placement
Min colors used
clauses trae % clauses satisfied Unsatisfied clause N/A
Complete grid % cells currect
Partial Credit
Path validity
Purtial placement
Valid coloring
Ervor Types
Invalid moves
Queen conflicts
Adjacent same color Excess colors
Constraint violation Logic depth used
Quality Measure
Move optimality
Time to solution
Scaling Behavior
Problem Growth
Exponential 2" Exponential 2 Exponential O(o)
Factorial v!! Variable S(is)
Exponential ko Varies with graph NP- threshold O(w²))
Exponential 2" to 2" Phase transition O(n + m)
Polynomial x I (maintaincd) Controlled linear 0(x²)
Solution Growth
Dillicully Scaling
Super-polynomii O(x)
Memory Scaling
Theoretical Properties
Recursion Depch Symmetry Group
2" I Sy (peg perms) Disk ordering Move sequence
NIA
(DPLL tree) Boolean cube
n² Sp 8. Cell values Completed grid
D2 (dihedral) Queen placement
S., [vertex perms) Color classes Color assignment
Invariants Cortificates
Truth assignment Variable values
Queen positions
Table 20: Algorithmic and Evaluation Metrics for Hard Suite Tasks (Tasks 6-10)
Metric
Logie Grid Puzzles
Cryptarithmetic
Matrix Chain
Modular Systems
Constraint Opt.
Generation Algorithms
Generation Method
Solution constraints
Template verify O(K!-au wr)
Dimension patterns
CRT construction O(Aa)
Resource allocation
Generation Time
Ornm- - (C))
O(n)
O(n"m)
Uniqueness Check
CSP enumeration
Exhuustive search
verilication
Period search
Emumeration Feasibility check
Validation Method
Constraint checking
Arithmetic verify
Cost calculation
Congruence check
Context Window Requirements
Prompt Tokens
10am 3am
+ 5kw
- 7i
+ 10k-
100 +
Solution Takens
4k
[lgg m]]],x]]
[log, =1
2n
Max Total Tokens
500 (5x5x5)
300 (k=7) High (linear)
200 (n=20)
150 (k=5)
250 (n=6)
Token Ffficiency
High (polynomial)
Very high
Very high
High
Solution Strategies
Optimal Algorithm
Are consistency
Cunstraint SAT
Bottom up
Extended GCD
Exact enumeration
Hemristicc Methods
Forward checking
Domain pruning
N/A (exact)
I Linear search Constraint filter N/A
Greedy local Upper bounds 2 (sclocthrject)
Pruning Technique
Domain reduction
Leading zeros
N/A
Branching Factor
values
s 100 digits
splits
Evaluation Metries
Success Criterion
All cells assigned
Valid equation
Minimum cust
All constraints met
Max prufit feusihle Feasibility profit Infeasible solution Optimality ratio
Partial Credit
So cells correct
& letters mapped
Cost ratio
% congruences Missing constraints Solutions found
Error Types
Constraint violation
Arithmetic error
Suboptimal cost
Quality Measure
Inference depth
Search efficiency
Optimality gap
Scaling Behavior
Problem Growth
(nt) (maintained)
P(10,K) k) 1 (maintained) Letter overlop O(k)
C2-1 ~ 4" /n
M = I,
2n Variable Tightness r O(mm)
Solution Growth
Polynomial value Chain length O(1²)
Periodic kM Constraint types O(k)
Difhiculty Scaling
Constraint density O(n)m))
Memory Scaling
Theoretical Properties
Recursion Depth Symmetry Group
nun assigaments
à letters
log n levels
N/A Sy equation perms Modular equivalence Solution value
selections 8., project perms Resource limits Selected projects
S. letter perms
None Associativity Split sequence
Invariants Certificates
Latin square
Digit uniqueness
Complete grid
Letter mapping
93

---

Published as a conference paper at ICLR 2026
Table 21: Complete Mathematical Formulas for All Hard Suite Tasks (10 Tasks, 68 Variations)
Task
Core Muthematical Relations
Complexity/Countin
Tower of Hanoi
P (optimal moves)
State space: 3 configurations
(6 variations)
3
(state transitions)
Moves: (3 2-n""""
N-Queens (4 variations)
Wfjja; / Ai 5 [z, 2yl
Solutions: S(n)
2.54
Conflict: (c) attacks (1g.9))
Search spuce: n! permutations
Gruph Coloring
w(C) X(C) ACG)
Chromatic polynomial: (a(k)
(10 variations)
Palk) = o0-
Colorings: K assignments
Boolean SAT
6= N V by (CNF)
Search space: 2 assignments
(5 variations)
Pr(SAT] am/2
m/n
Phuse transition: o, 4.267
Sudoku
No 72 2 27, 704, 267, 971
Valid grids: 6.67 1021
(8 vuriations)
Cundidates(i. j) [n] (Row; U Col; Box.))
Minimal puzzles: 10
Logic Grid Puzzles
6: U A. bijective Vi
Solution space: (n!)
(8 variations)
dle,, #
Problem space: (n!) mn Vem
Cryptarithmetic
val(W) 1 V (W())) 10|11111
101 Mappings: []0-6]]
(12 variations)
En val(W) (a)
Problem space: V (20 P(10,
Matrix Chain
m[t.j] min.gtes(
m/kk 1.Jl di-1 de d,}
Parenthesizations: Ca-1 (
(5 variations)
Caralan: C₂ = m+1 (27)
Time: O(n²). Space: O(n²)
Modular Systems
(mod m,), E [1.A]
Solution space: M = lem(
(5 variations)
CRT: Σ a, Miyh (mod M)
Period: kM, E
Constraint Opt.
max Ex-1 7,.% s.. Ex-p
(e(r))a S
Search space: 2 selections
(5 variations)
Tightness: + E. Miss) E,9
Feasible region: [JF] < 2"
Table 22: Hard Suite Summary Statistics (10 Tasks, 68 Variations Total)
Category
Range
Median
Notes
Problem Space Size
10 10
20
Ensures contamination resistance
Context Requirements
100-3825 tokens
300 tokens O
Fits in standard context windows
Verification Time
(n) to O(2")
Mostly polynomial verification Enforced by generation 8/10 are NP-complete
Solution Uniqueness
0 (n!)m
Complexity Class
P to NP-complete
NP-complete S,
Symmetry Groups
1 Sp S3
Rich algebraic structure
Total Variations
to 12 per task
6.8 per task
Comprehensive problem coverage
94

---

Published as a conference paper at ICLR 2026
K
HARD SUITE: COMPLETE RESULTS
This section details results for all hard suite tasks across all non-quantized open-source models and closed-source proprietary models. The average accuracy, instruction following rale and average output tokens are listed for tower of hanoi, n-queens, graph coloring, boolean SAT, sudoku, cryptarithmetic, matrix chain multiplication, modular systems, constraint optimization, and logic grid puzzles in table
23
24
25
Madd (Poress)
Toner Bassi
Queen
Graph Colocins
Besken SAT
Aec (46)
To-
Aec (9)
Ies (%)
To-
Ace (84
Best (
Tt-
Ace (89)
last (46)
Tie
10-00ga0 nn
T161
4g
66
13
4051
Quess (48)
33.83
29:0
623478
52.50
$150ggx
711216
35.12
4735.50
20,8
16.00220
5060.2
Ques3 (8B)
29.3221.55 37
611450
24.922377 191
63501gm..0
R60.57
01.22 10
15,0
366617
99 99.50am.m 22 377
Quead (228)
99.63
6192.5
57.00
5425
6266.30
20.52
8550
4085.7
92.4411-00
5251.34
Quas (315-M00))
46.00
10000
5163.99
66755511
0/82.35
72.35 10.005
404400
35.85223000
$41545
Qwesd (308-
616308
7311.5
07:37.0--2 mm
4336-96
47:00 19.02
$6433221
M0--000-0
Quesd (48-0
31.00
$6.00120.00
073622
45755-45.52
13/0110.77
7118.6
30.25:13.54
301844
31.00.10.00
45.51125.13
6340154
Owee Fasely (2wa2255)
5
6:0
100.01
300.95
0:00
4.90
3:88 2:42
$5.50
1311 T0
2:43
100.00
13.25
Owen: $11.500
0:01
(0000
367.21
100
153.90
11.902-1 13
60
10000
189-5
Owenh 5 (78)
10000
6.5gg
10060
1557
56 47
10091
171755
Quas2. (238)
38.83 220
[0003
1843
Ques2 (1.35-z3
100000
1246 11
41.5
352.3
604
795.29
1657
2M0.47
Qwasd (1B-as)
16.57
10000
14100
15
000.30
43.71.2
1955.6
Qwes2. (228-z)
100000
3876
9.00 3.00g...m
15.5 MM
608.1
26.72 14
008.55
2:0g
[1.44go.4
200.35
Genone Femtl
45-25
012-0.33
6413 16
3.2
100.00
10000
75 43.30
20085
03354
Gerona 1280
6:00
(000)
87243
1350
$3.50
C4LO3
3375
83.0
962 68
15.-14.57
24.99
20:4
Denin 17783
10000
535.47
20.00 41.57
(9100
1.73
20135
91.00
1997.4
3611 10.20
100.00
1136-00
16 50
110:01
25.00:433 50
12125
1133.00
2.672.4144 40
23
Pha weff-roonnnngg (1-I0)
00
50
176362
SI6.47
reai-rnnz cosoniin
12.50
14600
64003
671 6710032244
41.25
M92 (123- (145-40
22.7--50.00
100.0
1835
0,50
F9.25
10:25
15.45
21.924.01
16A2
172.57
110010002
100.00
$5,00
P633-xxl (148-)113)))))) 128k)
16.77.-7722 27
10000
204526
g.90
76.50
153.58
2600
$7.12
30658
100.00
55,00
Edoes Fondy
Liaam-822 (1B)
EX00
23 37
5671.66
0000
21.50124:72
1155.68
1.35.304
1238-115.22 25
1557-01
2001:3.0
ST.571-1003
2576.35
25.00
11601411
17115.0 45
L.be--11 (A8)
110101
(09
1454.6 96
71.00
1.be--311 (700))
33.37:47.31
10001
200..0
110
7635
4190.6.75
53935
100.0
50.01
1.bmm--177 (700)
1.172.26.00
10:01
1911
2415 50
49:2511.00
41.5
0628-1.11
W9126
09100
64
46
43
9039-2977
200144 30
235.5.05
Fouaty
10600
3010 on
Masicua (88)
10000
306.91
1000
161.00
95.121100
177.57
422
19100
35,00
Maarak-aae (120)
BX0
10000
$11.43
25.00
15100
25:55
25.320
5228
20.35
100.0
35,00
10000
46100
41.5 ms
30.12
11.8 as
051221139 1.30
20035
027
202.44
24.00
22.50 or
203
1671
Seellne (58)
98
6727.5
25.00 22.30
18.12
420072
41.25
4576-77
22.43 21.07
86 28-93
640.98
Seadli2 (1 71h)
6/00
10000
T14883
000
25
31.00
T63
10000
165.9
2:43-4.35
100.00
56,00
OPT-OS8 (500)
49:53
464544
99 07
22.5
2611 ()
19.20
$400.
15.09
21
29
17.00 m
15
0000 10
Opens Nuastyy (Popeeasgg)
12775- vain
75
621,55
HOL00
56.15
533.44
10008
TIREOO
13:.40
(2775-0005
K3.17
76
775-45
$0.00
42615
152.35
4000 0m0
914-78
1211
OPTA (-n00)
0.6
10020
765/33
2580--44-30
$7.30
20..00
16.55
96556
14.0
100.00
$5.14
5020 2000-200100
100.00
85317
02.50
6750
7648-19.50
100.00
315.0
580-11-18
10000
2000
11:00:17445
$3.50
402.33
10060
138.99
9000
100.00
506035
7000
100.00
115.00
96 05.75.23
10000
699.31
10000
100.00
201 60
(3 mial
$1.77
100.00
353000
32.57 47.03
$2.50
567.9
92.59
10.90
10000
1554.55
97. 47.14....52
100.00
465
Gunive Fasely (Propnnwe))
T.45
H0:00
W9:41
4100 45.24
47-0144333
va 20.0 07
655
100001
(501 40
3110115.0 03
29.5
Farnal 1.5-bub-iie
10000
279150
665114
75.5
19100
7533.91
Geeini 3.--uuh
M0.77
[0000
766.11
27.50
05140330.05
1353.43
10000
1592 18
110000
214.00
50.00 04.0000000 KM
10000
304255
4000) 31 57
1514133.00
69- 00
138125
14.29 22
1820.34
Table 23: Hard Suite Results Table 1: Performance of regular models on Tower of Hanoi (3-8 disks), N-Queens (6- 16 boards), Graph Coloring, and Boolean SAT tasks. Each task reports Accuracy (Ace), fnslroction-folloiinn (Inst) and average output Tokens with mean and standard deviation values. Results show average performance across different complexity variants within each task category.
95

---

Published as a conference paper at ICLR 2026
Model (Paran))
Sudoku
Cryptaritmmetic
Matrix Chain Malt
(%)
Takers
Inst
Tokrns
Inst
Tokens
Qven Family (@vea3)
Qwen (0.R)
=2.00:12.83
52.67
6609 54
0.00
70: 15
7565.67
0.43
56 56.00
6168881
Quen3 (48)
28.33
63.00
6383.16
11.23
93.65
7519.00
5.57
63.14_2.0
6250.30 30
21,00
6100 0
6096
37.32 Ag
91.68 23.00
7159.25
95.48_110
5505
21.39
6000
6227
52.74g9
6779.91
14.29..7..0 50
5409
Quen3 (34B)
36,00=33.05
60.23-15.00
5783.53
41.78-15.00 59
98,00:2.455
7040.81
30.43::40.40
95.43-7.17
5237.94
Qwar3 (33R)
33.00
58.3 34
9035.34
44.57g13330
97.0221.54
6854.37
28.43g.0.00
94.29..0.. 14
Qwen3 (M0 MOC)
27.87
62.67220 49
5785. 12
41. 61.64::1:00
52
97.28 70
6955. 30
28.71 2.712.40777
96.77 1.n. 195
5069.20
Qwen3 (30B MDD--
25.93 25.94
65.67 65.67 33
6002.56
51.37 11:37-113
98.95
6757. 44
28.43 13,43-40..40
95.000 .2
4993.00 60
Qwen' (3RR-MFF-)
$4.67 $4.67g110.0
100000
030.00
24 16 49
100.00
164.67
20. 20
05.29g
3542.83
Qwen3 (4B
7.33
73:33
6550-43
36. 14
98.41
7176.42
19. 19.14
95, 95.14.6611
5567.18
Queca Fumily (Qwen2.5)
Qwen2.5 (0.53)
1.00
163.13
0,00
17.08 135
3542,86
0.00
22:29 27.27
2418.80
Qwen2.5 (1.58)
2.33
97.33
138.35
0,00
99.75 75..3.444
109.10
0.00
85.45 .9
1373.84
Qwen2 5 (38)
100g1.44
100000
138.82
20.02g11 90
100.00
57.76
0. 14g
92:57 2.
1190..55
Qwen2 5 (7B)
6.6755.0
95.67=4.71
144.60
20.64::13444
100,00
98.15
1.14:2.42
99 99,86
1295.30 30
Qwer2.5 (148)
7.00
94.00
460.5
18, 18.66::13:4
100000
58.45
3.56:7:54
99.86
980.01
Quen2. 5 (32B)
30.33
95.00.4
451.23
19.21
100,00
66.47
11 11:29:10:71
100.00
1185 70
Quen2.5 5 (72B)
31.67 22.22
95.67
396.92
20.19-
100300
57.82
10.71 10.34
100000
1080.90
(1.58mm)
4:3
1420. 87
1300.36
1.29ga 1s
74-71 20.7 so
1763.87
Qwar2.5 (78-m)
2.00
1610.87
24.04
1604.41
4.86 01
21.67
1056.45
Qwen2.5 (720-m)
4.67
63.3
1691.43
15.34 40
92.70g3..0
1546.12
16.25
18.71
1143.55
Gemuna Fauslly
Geman (1B)
1.3
y7.67
132.12
0,000
70.54
1257.33
0.00
391.37
100.00
1.4ga 50
1064 50
Gemma
22,50 20
100.00
250.60
2.80-m..0
62,86-44.00
754.87
Gemma (270)
20000 1o co
100,00
1632.22
14:43g00.1
97.00 6..4
1088.23
Pii Family
Phi4 (148)
16.00.11..2
99.33
1241.55
14.43 L.
100,00
763.50
10.57 (77......
1295.49
Phid-ressoning- (1411)
4.39 os
63.622.m
14.
7537.06
20. 96-1.1 07
87.00
Phi reasozigg (14B)
9.33
68.33-15.55 57
6194.45
36.50
98.24-0.76
7256.05
25 29 + 30.14
91.14-12 12.222
6808.43
Mhid-min--rsseeing (3. 8B)
34.62g30000
50.6 in
6336-27
22:00:100 10
50.02.1.04 9.94
7403 47
12:50 20.0 no
77.66
MiiBmmiii 3B)
1.00
100.00
138.38
100000
0:43 cco
85.45 00
482
Phi3-med (14BI-4k)
5.00
100.00
141.63
19,96-13 74
100.00
58.82
1.14 2..02..00
94.00 14.00....0
813.81
Phi3 med (148-1288L
11 11.33g14.0 14.61
100000
141.61
21 21:25
100.00
88.61
0.57
94.366
786.49
L.ume Family
(11m=112(11)
0. 0.31g..n 17
70.00 41
2675.61
0.00
3. 38
1300.41
0.00
59 10 es
1420.35 35
Lluma 3.2 (3B)
166722.5
100.00
137.96
0.00
81,23
3556.00
0.00
0.29 70
376.11
Limma 3.1 (RD)
4.33
100.00
184.55
0,00
100.00
860.95
0. 0.
98,21 L... 7.
1355.00 48
1lums-3.1 (708)
22.67 g
94.33
431.32
21 14
99.25
498.91
3.7
100000
TM6.73
1ama-99. 3.3 (708)
34.00;
98.00 05.00-0.55
502.61
22 06
100.00
972661
4.57
100.00
854.50
Lizmal scout
0.00
0.00
94,46
0,00
21,45
252.30
9.57
39.29 39.29
3173.07
Mixiral Fumily
Mistall (7B)
1.00g1.41
100.00
1418.95
0.00
100.00
996.45
0. 0.14go..s
89:71
853
Minioral (80)
1.002..4
100.00
139.00
0.00
100.00
57.93
0,57
90 10.292m)
913.47
Mistralnemmo (126)
4.00 4.00.4.07
27.35-2.40
138.45
19.90-
100,00
57.20
0.00
71.14-24.55
641.58
Mixeal 8x7b
11:3
94.00g5.10
186.00
14
78.55
569.62
1.71
85, 16.14-m000
1094.05
Mixtral-Re23b
12.6715..0
100.00
610.22
18. 17
100.00
1304.57
3.20
95.57
739.3 21
Smoltm3 (3B)
1.35:1.00
26.00
6621.98
5.69 1.692mc
78 78.87.34.00
7344.62
10.57:19.44
80.29
5920.62
Smollo2 (1.78)
1.00
102.00
139.00
0.00
100000
62.08
0.00
99.29
1137. 10
GPT-0SS (20B)
60.67 233
71.67
3366.38
67.02
98.
5056.63
40.43
95. 95.43
5318.79
GPT--65 (1208)
34 14.67.2..m
46.37
2814.10 10
20.72.0.44
4157.96
50. 34.
05.14.7..0
4734-53
OpenAl Family (Proprictory)
GPTS
90.00:14.14
93,33
386.18
100.00
100,00
1090.29
68.57
70000
$78.76
GPPTS-mini
$3.33
103.00
335.60)
92.55 92.50
95,00) 95.00:
140.04
41.43
54 29
205.20
GPTS
1667222
100.00
361.10
90.00
92. 12.500=:1.00
157.64
64.29130.20
66 66.29-200
100.85
GPT4.
50.0 30.00-32.63
90.00 14.34
140
60.00.
92.50. 3.29
6537.75
30,00
95.2 15.21-7.22 7.22
3692.27
GPT4. mini
40.00_22100
70000
139.00
100.00
6573.95
32. 1.
95.57 22.57
2950.50
GPT4411-raee
40.00_16.53
100000
139.00
25.00gg..00
100.00
1787.75
28:00
43.30218.85
141.83
20 90
100.00
11.49 oo
GPT40-miza
6.67-4.71
102000
139.00
000 12
100000
65.60
1.43
+3.50
892.06
1ot mini
83.33 18
90:00
204.24
100.00
100.00
135.90
17
77. 14 to
106
o3
50.00:34.:14
100.00
500
92. 12 99
55.00a...o
254.91
67. 59.22
67.14:35 20
21.92
e3 mini
36.57
93.35.0.43
639.69
92.50 6.20
97.50:14.33
1211.04
67. 37.14
78:57 78.57-25..31
3590.14
Gemini Fasily (Propristorry
Gemini-2 S-gro
83.33 4%
1000000
139.00)
17.5004 s0
97.50 97.50) aa
1352.35
40.00ga0.11 17
10100:11 on
1566.50
Gemini-1 1.5-hhhh
70.00.-11 os
7667220 55
89.00
27.90g 19
60:00 or
437.38
11,43ga 0s
92 92.86.100 so
2365 30
Gemini 2.5 fadh lite
30.00: 16.33
100.00
200.07
47.50 79
97.50:54.15 13
6928.35
21.43=23.00
95.71.7.20
3951.26
Gemini-2
48.3 4
100000
180660
30.000004 40
100.00
2678.82
11:49 ad
61 61.43--2..00
854.70
Gemini 0 -adh----- Jiee
26.87
90.00
160.70
20.00
97.50
4624.77
22. 90.96.0..00
90.00
1115.83
Table 24: Hard Suite Results- Table 2: Performance of regular models Sudoku, Cryptarithmetic, and Matrix Chain Multiplication tasks. Each task reports Accuracy (Acc), cnatuuction-ollooviiti (Inst) and average output Tokens with mean and standard deviation values. Results show average performance across different complexity variants within each task category.
96

---

Published as a conference paper at ICLR 2026
Model (Param)
Modular Systems
Condraiit Opt
Lagic Grid Puzzles
(%)
Tokems
Tokems
Inst
Tokens
Qven Fanily (@vea3)
Quen3 (0.48)
18.40-22..2
67, 20
5520.66
14,20 74
100.00
5130.81
56,00
5413.56
Quen3 (48)
33,20
60
4855.61
14.20_
100.00
5740.65
52,00
99. 99.30
3949.75
65.60 09
5010.25
8.60 31
100.00
6246.113
52. 22.50247..0 50
95.35
4350.69
56.40+33.62
o0.40
18 16.30-22..55
51.50+-45.30 40.50
Qwen3 (34B)
68,60-29.25
100,00
4852.27
31,004 36.06
5966.55
61,00+ 39.00
18.20:1:70
4217.58
Qwar3 (32R)
65.60g30.54
100.00
4951.32
5731 91
18.50g.4 4a.50
97. 97.90 100
4209.50
Qwen3 (M0 MOC)
50,50 as
80gg 60
4200. 79
22 (0:00:3 87
100.00
6078.75 78
55 35,502:44.00 44. ao
97,88 97.80
4195 41
Qwen3 (30B MDE--
89,20- 12.90
99 80 10.4
3586.24
21 (0.000 32.93
100.00
6168.63
50,00 so.00
97.50 2 2.05
4542.49 49
Qwen' (30R-MDF-)
x3.70gi10
99 80
3655.26
72 402000
100.00
49966 66
61 20:00g
98. 20
4082.00 00
Qwen3 (4B
34.00
99,8020.4
3658,34
16. 20.00:226:1
100,00
6222.76 776
57,50
97,500
2943.09
Queca Fumily (Qwen2.5)
Qwen2.5 (0.53)
0.40.20.4
69 80
2782.97
2,60 300
100.00
1547.82
0.50 t0
89.85 155
2023:24
Qwei2.5 (1.58)
2,30 2.20:33.92
93.000 as
1857.13
2.80 .00
100.00
1499144
11:00:
89.72
1965.13
Qwen2 5 (38)
3,00 .s0
92. 80g0
2712.09
8:20
99 20 75
1547.12
21. 11 1.559
98.22gg
807. 39
Qwen2 5 (7B)
5.60:s.26
05, 0..00:1.00
1926.47
22. 22.40+33.22
100.00
1602.21
39,00=
99.18:00
65%.14
Qwer2.5 (148)
8.202.0077 12.0
95, 25.20
2435.72
34 11.27
100.00
1025.30
50,50 50.50
99.90
600.59
Qwen2.5 5 (32B)
91,80g
1950.35
40
99.80.2..44
1138.83
53.0 53,00
99 10.90..00
550.24
Quen2.5 5 (72B)
10.60-14.77
97.20
2556.63
38 38.60+132
100.00
1536.34
62. 62.50+ 35.50
100000
687.72
(1.5R-m)
20g1
1523.40
2:30
1476.54
12.50go.so
91.00
1305-75
Qwar2.5 (78-m)
30.90
co
1712.20
4.30: o6
1552.50
24.00 oo
94.55
1705.04
Qwen2.5 (720-m)
2.40
95.80
1567.37
4,00 74
1458.79
45.50
97.12
879.50
Gemune Fauslly
Geman (1B)
0:40 s..a0
89.80 a9.80
2363.55
3.40 3:331
100.00
1002.66
0.50
90.48
466
(49)
16.00 00 as
06.00g
20.002.90000 0s
1500 oe
00
Gemma
20,00: 2000:26.00
19 39
3267.35
38 34.70
1206.82
65 25.000222.000
99.50
664.05
Gemma (270)
299.07
92.00ga.70
2470.36
52. 52.80g.16.00
1128,68
44.00g-41.00
99.98.00.0
893.26
Pii Family
Phi4 (148)
10:40:
2478.99
45 14.40.....0
100.00
093.00
79.
95.45
726
Phié-ressoming- (1411)
54.002300 so
80
32 0012:5 as
56.002 0o
15 00
6676.41
Phi4 reasoting (14B)
35,80- 30.75
10:0-10.00 10.05
6989.32
40.800+32..00 32.54
7150.18
66:00s 25.00
92.30-33.6
65422.3
Phid-min--reseeiing (3. 8B)
42.20 so. 65.66
20g 17
5870.73
15 15.00 09
100.00
6713.40
50,00 18.00
18.18
45
MhiB-mini 3B)
0.00
90.80 00.80
13000 us
98 1.90
801.45
54.50 14.5002. au
90.92
830.04
Phi3-med (14BB-4k)
a.40....o
1.05
700
65.20...73 73
99 0..00
789.56
47. 47.50.220 220.00
99.38 -0.22
582.48
Phi3 med (148-12883
0.20gm
91, 41,40
947.50
17.40gg
99 20:20.2.1 75
877.94
56.50g
98.92
575.08
Llume Family
(11m=112(11))
0.00
400ggnn so
3200.42
3,00g: 70
100.00
1763.54
4.00ga oo
92:35 os
1354.80
Liuma 3.2 (3B)
0.00
41.00 os
2696.67
2,00 61
99 09/60:20.4 40
1297.29
24.00
97.82200133
912.02
Liuma 3.1 (8D)
0.00
91.20
3474 1474.99
13.60_0.4 44
99 39.30_0..4
1406.37
28.00 28.00 20.00
92.10.a..0
2831.85
1lums-3.1 (708)
0.40:2o
87.80
4310.14
25. 50
100.00
1009.33
58.00
99.62.20
743.58
1ama-9.3 3.3 (708)
2.00-44.00
97. 1.17
1142.65
23 404
100.00
1105.34 34
40
100000
733 30
Lizmad scout
0.40
5.20
191,85
41 11100:2..40
100.00
1162.59
72,50
100.00
$29.92
Mixtral Fumily
Mistaal (78)
0.00
87.80 40
2075.13 13
1.40..0.a0 so
97 17.8
898.65
14.50.... so
98.92 97
504 36
Miniseal (18)
1.40
94, 40.
1936.11
13.20
100.00
1117.54
36. 30
99.82
547.00
Mistral-eeme (126)
0.00
86, s..c0
1961. 18
6.60
98 98.80-0.75
6.9.16
35 (00000
90. 72
368 96
Mixirl 8x7b
0.00
83, g0
903,05
6.40
99 39,0l.0..0
632.40
23 23.00
80.48
397.41
Mixtral-Re23b
0.30 0.30
1472.45
11 10
99 99.20
711.37
45 45.00g300.cc
99. 99.18g.0.00
753.80
Others
Smoltm3 (3B)
29,40
93.60 43.60.4..103
$763.05
2,40 0t
100.00
7099.0 00
$1.50 $1.50
95 95.58
38.28022
Smoltm2 (1.7B)
0.00
71,80
1786.45
5.20
100.00
15.26
51.00 14.00.000
62.00
156.45
GPT-OSS (20B)
$1.20
75.20
3603.15
5 58.0
99 19.01.0.4
4794.22
$4,00
98.52
1292.30
GPT--SS (120RR
86 60.
94.80
2583.77
R3 :3,40
100.00
3912.25
50. 50,00 20.00
96.42....0
$31.55
OpenAl Famity (Proprictary)
GPTS
98. 100
100,00
1770.24
100,00
100.00
36,92
75 (000=:20.00
99.75 19.75:cn..2
$21.59
GIPTS-mini
100.00
103.00
1802.66
100.00
100.00
35.57
55 10.000
97.00
1115.52
GPPS-amm
100.00
102,00
992.52
100.00
100.00
37.28
65 65.00
87.50
311.57
GPT4.
58.00 31.24
85.00
12395.80
86 16.00, 14.97
100.00
7336.60 63
$0,00 20.00
100.00
793.05
GPT4. mini
64.00
94,00
11230.96
94.00.4
100.00
2596.06
55,00
99.50
586.35
GPP44
94.00
4076.72
50,00
1587
40.00
2.0 on
36 12 00
667.80
00:0001 oo
50 50
GPT40-mil
9,00 11.00
200000
1274.96
30.000 26.73
928,02
70:00 a 20.00
100000
1ot mini o3
96.00 00 no
08.00.4.00 oe
882.37
100.00
100.00
31.97
00:00ga00 no
90.25 90.25
100:00
100000
1676.59
100,00
100.00
62.06
85 00000...0
88 88.25....2
676.08
e3 mini
92.00.9..0
100000
4044.000
96.00; 4 430
100.00
1929
50.00 20.00002.000000
99.000
691.92
Gemini Fasily (Propriitorry)
Gemini-2 S-gro
100,00
2000000
1525.84
98.00)-4.co
768.92
75.00) 25.00
100,00
714.10
Gemini-1 1.5-hhhh
74.00.9941 m
8400 s5
1459.50
00 co
100.00
1756.18
60:00g 0n
90.00 10 oo
1090.00 70
Gemini 2.5 fadh lite
68.00 29.22 25
94.00 ne
9432.86
88 38.00+11..66
100.00
3445.3 350
55 25,00::45.00 45.00
99.00 00
8559.5
Gemini-20-0a.
26.00 26.
74.00 16 30
3656.95
22.00114.00
100.00
1832.40 40
55 $5.00, 45
100,00
614.50
Gemini 2.0 -adh- Jite
20.00
78.00
3299.12
54.00224.17
100.00
1650.30
90,00
98.50 18.50.1
1022.85
Table 25: Hard Suite Results- Table 3: Performance of regular models Modular Systems Solver, Constraint Optimization, and Logic Grid Puzzles tasks. Each task reports Accuracy (Acc), Instruction-follle following (Inst) and average output Tokens with mean and standard deviation values. Results show average performance across different complexity variants within each task category.
97

---

Published as a conference paper at ICLR 2026
L PROMPTS USED FOR ALL HARD SUITE TASKS
This section describes the prompts developed for each of the hard suite tasks. Each prompt begins with brief description of the task followed by the inputs. The constraints are listed if applicable. which is then followed by detailed description of the required answer format.
Prompt Template for Boolean SAT Task
CHALLENGING BOOLEAN SATISFIABILITY (SAT) PROBLEM:
Find a truth assignment for variables I1, 22; vars: that satisfies this complex formula: (formula)
LOGICAL OPERATORS:
V means OR (disjunction)
A means (conjunction. means NOT (negation)
SOLUTION FORMAT REQUIREMENTS:
Your answer must assign True/False to ALL (num_vars) variables.
REQUIRED FORMAT (STRICTLY FOLLOW THIS) <answer>
((1: True, 2: False, 3: True, ... (num_vars) vars): True)
</answer>
IMPORTANT FORMATTING NOTES:
Use EXACTLY the format shown above inside <answer> tags.
Use numbers (1, 2, 3...) NOT variable names (5),
Use True/False (not true/false, 1/0, I/F)
Include ALL (num_ vars) variables in your answer.
Do NOT use code blocks, markdown, or other formatting.
Prompt Template for Constraint Optimization Task
ADVANCED MILII-CONSTRAINI RESOURCE ALLOCATION PROBLEM:
Your task: Select an optimal subset of projects to maximize total profit while satisfying all resource capacity constraints and additional requirements.
AVAILABLE RESOURCES: [resources_str)
AVAILABLE PROJECTS; (projocts_str)
OPTIMIZATION OBJECTIVE:
Maximize total profit from solected projects.
CONSTRAINTS:
Resource capacity: Cannot exceed available capacity for any ressurce.
2. Dependencies: Is project has dependencies, all dependency projects must also be selected.
98

---

Published as a conference paper at ICLR 2026
ANSWER FORMAT: Provide your answer as list of project IDs (numbers) in square brackets.
<answer>
[project_id]]] idd_, project_id 2d_2, project_id_3, 3d_3, ...] </answer>
Prompt Template for Cryptarithmetic Task
Solve this challenging cryptarithmetic puzzle:
[wordl] [op_symbol) (word2) (resuit_word)
Rules:
Kach letter represents a uni qui digit from to 9.
No two letters can have the same digit.
Leading letters cannot
SOLUTION FORMAT REQUIREMENTS (STRICTLY FOLLOW THIS):
Your answer must assign digits to ALL letters.
REQUIRED FORMAT: <answer>
(("A"; 1, "B"; 2, "C": 3, ...))
</answer>
IMPORIANT FORMATTING NOTES:
Use EXACTLY the format shown above inside <answer> tags.
Use double quotes around letter names: "A", "3", "c", etc.
Use actual digits: 1, 2, 3, etc. (not "1", "2", "3").
Include ALL letters in your answer.
NOT code blocks, markdown, other formatting.
Prompt Template for Graph Coloring Task
CHALLENGING GRAPH COLORING PROBLEM:
Your task: Color the vertices of a complex graph using exactly echromatiicnnubeer colors such that no two adjacent (connected) vertices have the same color.
GRAPH SPECIFICATION:
(n) vertices numbered from to n 1 (expressed as (n-1) if you
want a literal placeholder).
(len (edges) edges: (edges_str).
Minimum colors needed: (chromatic_numbrr)
COLORING RULES:
Use exactly (chromatic_nuntl difforent colors. NO two vertices connected by an edge can have the same color. must be assigned exactly color.
EVERY
FINAL ANSWER FORMAT:
End your response with the complete solution dictionary in this
format:
99

---

Published as a conference paper at ICLR 2026
<answer> ((0: "Red", 1: "Bluc", 2: "Green", 3: "Yellow", (n-1h: "Colorx"))
</answer>
Prompt Template for Logic Grid Puzzles Task
You are solving a logic grid puzzie. This requires systematic logical reasoning and constraint satisfaction.
PUZZLE SETUP:
Grid Size: (size) (size)
Categories: (categories)
CLUES:
[Constraint descriptions]
TASK:
Use logical deduction determine which person has which attributes in each category.
CRITICAL INSTRUCTIONS:
Present your final answer in ONE of these EXACT formats:
FORMAT (PREFERRED Use this exact structure):
|boxed[1
(person): [(category): [value], ...] ...))
IMPORTANT FORMATTING NOTES:
Replace [value] with the actual specific, attribute.
Use the exact person names from the entities list.
Include ALL people and ALL their attributes.
Make sure no two people share the same attribute value.
Prompt Template for Matrix Chain Multiplication Task
CHALLENGING MATRIX CHAIN MULTIPLICATION OPTIMIZATION PROBLEM:
Your task: Find minimum number scalar multiplications needed to compute the matrix product using optimal paronthosization
PROBLEM SPECIFICATION:
Number of matrices: (matrix_counts
Matrix dimensions: (natrix_.dceripiionns).
MATRIX MULTIPLICATION COST MODEL:
To multiply two matrices of dimensions (p 9) and (g x r): cost p xxqxr scalar multiplications.
DYNAMIC PROGRAMMING APPROACH; This is a classic optimization problem em that requi res dynamic programming.
CRITICAL: ANSWER FORMAT REQUIREMENTS
To ensure your answer is correctly parsed, follow these EXACT formatting requitements:
100

---

Published as a conference paper at ICLR 2026
1. ALMAYS provide your final answer in this precise format: <answer>
MIDEERIC_VALUE_ONIA </answer>
2. REPLACE INUEERIC_VAUU__O with ONLY the numeric valuc.
3. Example: If the answer is 1204480, write exactly: (ateeeer2124444s//naaaasae
Prompt Template for Modular Systems Solver Task
ADVANCED MODULAR SYSTEMS PROBLEM:
Your task: Sclve the following system of modular ari equations and find the value of that satisfies all conditions.
SYSTEM OF EQUATIONS:
[modular equations]
ADDITIONAL CONSTRAINTS:
[constraints]
MATHEMATICAL BACKGROUND:
Modular arithmetic: x (mod m) means x leaves remainder a when divided by
Use the Chinese Remainder Theorem for systems with coprime moduli. For non-coprime moduli, use the general solution method.
ANSWER FORMAT REQUIREMENTS:
Your response must end with your final numerical answer in one of these formats:
Option 1 (Freferred);
<answer>
[Your integer answer here] </answer>
Option 2 (Alternative):
Therefore, I = [Your integer answer here].
DO NOT end with intermediate calculations or parametric expressions.
Prompt Template for N-Queens Task
Solve the {n} -ueees problem: Place (n) queens on an (n) x inl chessboard that no two queens attack each other.
RULES:
Queens attack horizontally, vertically, and diagonally. No two queens can be in the same row, column, or diagonal. You must place exactly (n) queens the board (one queen per row)
CRITICAL: Your answer must contain exactly (n) numbers, each representing the column position (0 to n 1) of the queen in that
101

---

Published as a conference paper at ICLR 2026
SOLUTION FORMAT:
Provide your solution as list of (n) column numbers. The first
number is the column for 0, the second for 1, etc.
REQUIRED FORMAT Use one of these:
[col0, coll, col2, ...] + PREFERRED
Answer: [1, 3, 0, 21
Solution: 11, 3, 0, 21
<answer>
[your {n) column numbers here]
</answer>
Prompt Template for Sudoku Task
Base Rules for (size) (size) Sudoku:
Fill the grid with numbers to (size).
Each row must contain all numbers 1 to (size) exactly once.
Kach column must contain all numbers 1 to (size) exactly once.
Each (box_dimensions) box must contain all numbers to (size)
exactly once.
PUZZLE GRID (0 represents empty cells) (grid_representationn
SOLVING REQUIREMENTS:
Provide the complete solved grid.
Every cell must be filled with a number from 1 to (size).
Solution must satisfy all rules including additicnal constraints.
ANSWER FORMAT:
Provide your solution as complete grid using one these formats:
1.
Grid format:
1
3
3
4 1
4
2
2. List format:
111,2,3,4) (3, 4, 1,21, [2,3,4,1) (4,1,2,3)
Prompt Template for Tower of Hanoi Task
Solve this Tower of Hanoi puzzle with (num disks) disks.
RULES:
1. Only one disk can be moved at a time.
2. A larger disk cannot be placed on top of smaller disk.
3. Only the topmost disk any peg can be moved.
INITIAL STATE:
Peg A: ( (disk 1 is smallest, disk (num_diske sks) is largest)
Peg [initial[''']]
Peg C: (initiall'Cri
GOAL:: Move all disks from Peg A to Feg C.
102

---

Published as a conference paper at ICLR 2026
Provide the complcte sequence of moves. You can any of these
formats:
"Move disk x from peg y to peg z"
"Move x from Y to 2"
"X: Y -> 2"
"Transfer disk X from Y 2"
<answer> Move disk from peg A to peg C Move disk from peg to peg
</answer>
M QUANTIZED MODELS COMPLETE RESULTS
This section details results for all easy, medium and hard suite tasks for the quantized Qwen models. There 21 quantized models from the Qwen2.5 family and 17 quantized models from the Qwen3 family.
M.1 EASY SUITE
This section shows results from the quantized Qwen models for the easy suite tasks: absolute difference, comparison, division, even count, find maximum, find minimum, mean, median, mode, multiplication, odd count, sorting, subtraction, sum, second maximum, range, negative number count, unique elements count, maximum difference between adjacent elements, count elements greater than previous, sum of indices of maximum element, palindromic number count, longest increasing subsequence length task, of digits, count perfect squares, alternating sum count multiples of K, local maxima count.
(Parsm)
Quant
(%)
Model (Puram)
Quant
Acc(S) (S)
Insi (5)
Tokens
Qven (Qwen2.5)
Oven Family (Qwen.3)
Qwen15
AWQ
19.31
1358.77
Qwes2.5
GPTQ-4-Bi
12:77
77.70
Qwes3 (0.68)
FP8
55.89
86.51
2767.9
Qwes2.5 (0.58)
GPTQ-a-Bir
21.29
76.79
431.50
Qwe3 (0.B))
GPTQ-8-Bi
56.40
84.33
2922.35 35
Qwen1. (1.58)
AWQ
58.77
77.00
732.95
Qwes3(1.77
FP8
Qwea2.5
GPTQ-4-RII
30
(1.70(
GPTQ-6-Ba
73.07
2910.69
Qwen2.5 (1.5B)
GPTQ
43.67
36.64
264.30
Qwe33 (4B)
AWQQ
$1.14
89.73
2891 68
Qwe1.5 (3B)
AWQ
45.83
85.93
908 07
Qwes3 (4R)
FPS
$4.56
91.92
2889
Qwes2.5 (3B)
GPTQ---Bir
41.94
90.97
301 00
Owen]
Qwen2.5 (3B)
GPTQ-s-BB
Qwm3 (8B)
FPS
2905
Qwee2. (7B)
AWQ 66.53
Qwes3 (14B)
AWQ
87.98
94.17
2432.73
Qwes2.5 (78)
GPTQ-4-Rit
58.03
96.00
201 90
Qwen] (14B)
FP8
87.55
93.75
2415.34
Qwen.5 (7B)
GPTQ B-Bit
60.61
96.10
287.50
2477.95
Qwa115
AWQ
Qweed (32B)
FPS
$8.11
93.86
2544.64
Owes2.5 (14B)
GPTQ 4-i
50.91
96.69
240.90
Qwen (30B-M0B)
FPS
$8.83
94.43
2417.23
Qwe1.5 (14B)
GPTQ-S-BB Bit
61.86
97.89
261.20
Qwe33 (30B-MOE-I)
FPS
90.50
96.67
2051
Qwel1. (32B)
AWQ
77.84
99.72
254,55
MOE-D
Qwea2.5
GPTQ-4-Bi
Qwml (4B-)
FPS
85.42
94.61
2603. 79
Qwen2.5 (32B)
GPTQ-s-Bi
73.08
261.90
Qwen3 (4B-i)
FPS
86.89
99.36
817.99
Qwes1.5 (72B)
AWQ
78.88
99.63
156.67
Qwea2.5
GFTQ Bit
72.74
91.85
358.30
Qwen Family (Qwen3)
Qwe11.
GPTQ-S-Bil
Qwen Family (Qwen2.5)
Table 26: Easy Suite Results (Quantized): Performance of quantized models on the Easy Suite benchmark tasks. Each model reports Accuracy (Acc), Instruction-followee (Inst) and average output Tokens. The Quant column indicates the quantization method used.
M.2
MEDIUM SUITE
This section shows results from the quantized Qwen models for the medium suite tasks: algebraic sequence, complex pattern, fibonacei sequence, geometric sequence and prime sequence. These results listed in tables 27
103

---

Published as a conference paper at ICLR 2026
Model (Puram)
Quant
algebraic_ sequence
complex pottera
thennod sequence
Aec (Avg
Inal %
Tokens (Arg)
Aec (Arg %)
Inst (Arg
Tokena (Aag)
Arr (Arg %
Tnst (Arg %)
Takem (Ang)
Owen Fumily (Qwen3)
Quedd (0
FPS
28.40 28.4024.12
100,00
16403 45
13:20
Qward (B))B)
GPTO---
27.80 g3.
100.00
100.00
5987.01
100.00
5815.69
Qwend (1-78)
FPS
2.50
100.00
53
11925.78
5771
Qwend (1.78)
GPTQ-4-
0.01
100,00
5875.73
43.63 10.72
100,00
6367.00
L0:13
5753.83
Qwend (4B)
AWQ
23
100000
7305. 17
01
5956.85
75 80.80
(48))
(8B))
AWQ
1RN11.15
100.00
12923
100.00
12491.45
7652.62
AWO
100.00
6404
13.01
100.00
45
96
3434.89
100.00
6412.30
13.22
100.00
96
100.00
100.00
6750.13
59.93
100.00
5501.37
97.
3759.06
MOE)
Qwen3
FP5
40,50
100,00
5944.65
72.
100.00
4459.33
98
100000
3273.00
MOE
Qwend
1P5
46.20
100000
355930
79.76 10.85
100000
3442.4
98
100000
1873.05
1906 MOB-U)
Quend (4B1-)
FPS
40,00 3.29
100.00
699 96
68.7
100
5619.5
89 19 10:60
100000
4673.70
100,00
5344.65
72.59
3588.75
05.40.0.21
2839.75
Chean Fasily ((hean2.5)
Qum2.5 ( 18G
AWQ
100
1872
100
2470.96
180g 47
100,00
1945
Qumd.5 (0.58)
GPTD24
11880g
100.00
3345
100.00
2801 56
100.00
2574,42
Qwa2.5 (0.58)
GPTO-8-
2.20
100.00
3205 43
5.42
100.00
3363.70
4.40 1.04
100.00
2314.57
Qwen2.5
AWO
1.77
100,00
14.04
2868.22
7.40 1.00
2565.16
Qwen2.5 (1.58)
GPTQ
29.40
1.00
100.00
2695.15
12.76
100,00
2942.20
8.20 11.17
Q2ee2.5 (1.5)
GPTQ-
100,00
2354.8
12.15
10,80 12
2100221
Qwen7.5 (30)
11.0g
Qumn75 (1B)
GPT)-4
100.00
13.00 10
(3B)
GPTQ-3-
26.00
100.00
1405.23
100.00
1302
1761.07
Bii
Qwen2.5
AWO
36.00
100.00
1350.00
753.19
20 20
554.48
Qwen2.5
35.00
100.00
923.49
41.74
100.00
763.73
20
738,95
Bii
Qwex2.5
GPTQ-3-
23.70
100.00
1041.76
100.00
852.66
19.20
100,00
636-42
Bii
Qwen2.5(14B)
AWO
2.13
92,40
$76.95
49.40
100.00
$52.46
20
531. 50
Qwen2.5
2.15
94,20
736.45
100.00
$78.50
26 4.50
600.36
Bii
(144)
GPTQ
965.09
100.00
553.11
25
100,00
564.42
Bii
(wen2.5(3338
AWQ
44.00
1147.58
100000
507.65
36
100000
Q2ond.5 (5286
GP112-4 Bii
100000
1210.72
10000
$30.96
35
100.00
595.57
Quan255
GPT)---
42.20
95,50
1160.25
51
100.00
$50.56
37 90
100.00
577.53
Bii
Qwa225 (72B)
AWQ
41. 41.50.2..5
100.00
053.78
49.91 =9.91
100.00
$26.04
37.90
100.00
928.50
Qwen2.5 (72B)
GPTQ-4-
42.50
100.00
751.62
50.22 50.22.:0.00
100.00
753.58
39.20
100.00
$83.59
Bii
Qwen2.5 (72B)
GPTQ
43.00
100.00
$41.55
52.97.6.62
100.00
753.84
39
19.90
100.00
907.20
BiI
Table 27: Medium Suite Results Table 3 (Quantized)): Performance of quantized models on algebraic sequence, complex pattern, and fibonacei sequence tasks. Each task reports Accuracy (Acc), instruction-followiii (Inst) and average output tokens with mean and standard deviation values. The Quant column indicates the quantization method used.
104

---

Published as a conference paper at ICLR 2026
Model (Param)
Quant
geometric sequence
prinse sequence
Ace (Arg
Inst (Avg
Tokens (Avg)
Inst %)
Tokens (Arg)
Family (Qven:)
Owee3 (0.68)
PP8
10.80
60.00) 48.99
10990.97
11 +1.00+5.00
100.00
19524.77
Qwen3.00.68)
GPTQQ--Bii
1:000:1. 50
60.00 18.90
3618.17
12.40 1.009
100.00
6434.31
(w.33(1.78)
FP8
24 34.60.227
60.00
9217.75
31 $1.40=10.00
100.00
16144.70
13.80:27.00
48.99
Oweed (4B)
AWD
33 30.30 11:31..26 288
60.00 18 18.99
3118.46
39.00 +9
100.00
6257.97
Qwee3
FP8
58. (1
60.00
4573.26
44
100.00
18455 05
A0 50
18.90 60,00
3564.32
Owend (8B)
FP8
58. 60
1680.13
66.20 8
100.00
13505
Qwee3 (14B)
AWO
60.00
60.00-
3079 13
$5.90
100.00
7374.38
(148)
99
601.0) 18.99
6.
Owen3 (32B)
AWQ
$8.
60,00 48
1306.35
88
100.00
3747.94
Owee3 (32B)
FPS
56. $5.00:1
60.00
1438.15
92.60
100.00
3584.84
04
60.00 89.99
3.
Qweed MQBB)
FP8
S7 40
60.00 48
1761.08
83.20
100.00
3676.55
Qwee3 (388-MQF-i)
FP8
S8 000
60.00
1656.18
90.00
100.00
2549.66
55.00::455 23
Oweed (4B-i)
FP8
$2.00 43.88
60.00
1930.58
66.00
100.00
3941.52
Qwen Family (Qwen2.5)
Qwee2.5 (0.58)
AWQ
0.00
60.00
1032.29
6.00 6.00
100.00
1529.5
Qwee2.5 (0.58)
GPTQ-4-Bit
601.0
695.05
5.401.3.55
100.00
2799.51
0wen2.5 (0.58)
GPTQ-8-Bi
1.2012240
60.00 48.90
8..0..2.32
Owee2.5 (1.58)
AWD
2.00:44.00
60.00 60.00 48.99
2671.43
11.40.+4.72
100.00
1904
Qw.32.5(1.88)
GPTQ-4-Bir
8.80
60.00
845.41
16 22.999
100.00
1430.95
0wen2.5
GPTQ-8-Bi
5.80 91
60.00 90
16.00
Qwee2.5 (38)
AWQ
12.00
60.00
1034,24
19.00 4.38
100.00
1289 12
(38)
GPTQ-1-Bit
60.00-48.
873.76
20, 40
100.00
1357.65
0wee2.5 (3B)
GPTQ-8-Bi
15.00 09
60.00 18.99
0. 78
Qwee2.5 (78)
AWQ
11. 11,80
60.00
$20.09
$1 16
100.00
75% 25
(78)
GPTQ-i-Bit
17.00
60.00
449.39
32 80
100.00
671.76
0wee2.5 (7B)
16.60::19. 48
48.99
-9
Qwee2.5 (148)
AWQ
18 40
60.00 48.99
404.95
39 30,80
100.00
489
(1442)
GPTQ-4-Bit
20.00
601.0)
325.71
100.00
593.88
24.20
45.99
20 8.
Qwee2.5 (328)
AWO
31 33
60.00
423.90
100.00
492.43
Qwee2.5 (32B)
GPTQ-4-Bit
35 35.40135.10
60.00
490.42
49.
100.00
$22.55
Qwee2.5 (328)
GPTQ-8-Bit
30.0 45
60.00 49.
62.60 02..00-4440
Qwen2.5 (728)
AWQ
30 80,80+25.20
60.00
$99.50
61,80+100
100.00
669.67
Qwen2.5 (72B)
GPTQ-4-Bit
25.40 25.40
60.00
735.26
55.60
100.00
702 28
(728)
GPTQ-8-Bir
22.00+-27.0
60.00 48.99
488.46
16.40+12.11
Table 28: Medium Suite Results Table 4 (Quantized): Performance of quantized models
geometric sequence and prime sequence tasks. Each task reports Accuracy (Acc), Instruction- following (Inst) and average output tokens with mean and standard deviation values. The Quant column indicates the quantization method used.
105

---

Published as a conference paper at ICLR 2026
Quane
N-Queras
Graph Coloring
(teve Feinlyy (ipendd)
Qwa.--2.88
FM
0.09
5:5
58.75
4283/1
21.38
(600)
4738.88
12
6250.33
(0.05--2.66)
QPTQ-4
0.00
99.30 19,30.0000
64.25
4500.0
21.30 goua
65.15 65.15
4744.2
1500
FPS
0.33
7529/00
5157.57
6077
6070.4
QFTQ-4
1n7
98077
1.50
78055
717742
4400
1000
5131.90
33,00
9072.96
11775276-51 35
6612
41.57291 50
OvenSdd8)
6362.95
6782.4
QwerS-18)) (4B)
A0KG
3210
AN6572
15.25
0AG-4
14.5
11201115
4537
01101:00
5yauss
QpenS (4B)
1779
NA130
6113:1
11.32234.00 01
7115-94
431-311
4637277 51
37
46302444 to
QpeaS (148)
179
44.77
500.45
78:00g14
402.9
5451.82
(220)
3810
DA130
611291
26.91
76113
18:00100
6310.11
202.16
Qwas (328)
279
4230
1933 94
6221-41
-102241
57
#0132
30
(38--M0E
3150
1330g
70
202g m
344229
6139ggs
Quaa
41.17
22.70
901.31
4430
64.51
10.35
754383
62.3
4:0
/2.33/21.31
51,301
22.40
995433
108--106-0
Owen
DP9
50.00
101.00
1085
<7.00
47.01
91.75
91
591.90
57
199
10100
$6.00
(30 M)DE-)
30:50
10--0
675085
45.35
12.44
2015-3
11:0011.44
1357
(9:60 17 690755
Owee Fassly (2we2.))
Uwen2 (8.38)
0003
001200
1.0011 Ta
41225 t0
1637211 cn
19457
1:00 75
20.0 76
BMMg 50
050g0 T1
MA2211
MUMIg
Six
Qwen225 (1:38)
G
1000
10020
264144
000
3,25
WA.77
16.0
1479.77
257
10100
9153
Qwee2.5
Qwes2 (1.38)
GFTQ-+
0000
100000
33H-21
41530
1096.6
10100
11734
Su
Qwea2. (1.58)
GPTC-8
000000
340234
0.29
64.00
21.00
153,55
140113343
16.00
11031
455.06
2.00
10100
Swent
<WG
0.0
000
1511.0
003
14.75
6000
1133
11113.04
10100
56.00
Oww? (183
GPTQ-4
30
15
045
16.75133.4
618.9
112.00
1.514.53
10100
50.00
(100
000
10100
049
15:26
1136:19
171.3.06
10170
$6.00
5/01
<WG
18
00020
10
0.5
44.5
31.0.5.000
10100
333225.22
69025.0 ds
16.7g.3.58
(70)
GFTG 4-
11 2
00000
971.00
4 05
3400g0.30
X9111
12.0g
10100
157900
30
(1482
1595.15
25:35 io
300.29
18200
(14225118)
GPTQ-4
33.33 33.33
100.00
104492
90000
163.24
0000
754.11
9600ga.4
54.3
(1442
GPTC-8
442.00
100000
126155
12.0
96225 as
242.99
49.02...00
0000
902.1
17/4412.08
100.00
58.45
(13225
AWQ
30.58
000.00
134356
11:25
11.25
205.33
43.83
0812:1100
12.66
$6.66
0532
GPTQ-4
24 17 525.00
100.0
1333.22
14.90
92 13.75
20055
45.0
45.0
96300..7
501.28
22 22.66:
99166 10
$6.47
(330
100.00
47.31
55.55
Si
Que22 (128)
AWGJ
6687 14
101.00
15:0g 36
101100
501-45
440024 kn
Wok 37
1000/6
2128 as
19100
56.00
2.7 14
454227 30
1025
2328211 m
Si
Qwen2 (12))
GFIQ-S 30
06.55
100.00
1854.7
34.2
100.00
53043
44.75
10,00
088044
23.11
10100
56.00
Table 29: Hard Suite Results- Table (Quantized); Performance of quantized models on Tower of Hanoi (3-8 disks), N-Queens (6-16 boards), Graph Coloring, and Boolean SAT tasks. Each task reports Accuracy (Acc), instruction-followii (Inst) and average output Tokens with mean and standard deviation values. The Quant column indicates the quantization method used.
M.3 HARD SUITE
This section shows results from the quantized Qwen models for the hard suite tasks: tower of hanoi, n-queens, graph coloring. boolean SAT, sudoku, cryptarithmetic, matrix chain multiplication, modular systems, constraint optimization, and logie grid puzzles. These results are listed in tables 29 30 and
N STATISTICAL ANALYSIS
This section provides comprehensive statistical analysis of our evaluation results, including three-fold evaluation, confidence intervals, and significance tests.
To ensure robustness of our findings, we conducted three-fold evaluation for all 101 models. We report mean accuracy with standard deviation across the three runs. Our analysis reveals that large models (above 14B parameters) demonstrate high stability with standard deviations ranging from 0.3% to 1.5%. For example, GPT-5 achieves 83.57% 42 and Gemini-2.55pro achieves 74.31% 30.38- Medium-sized models (7B-14B) show moderate variance with standard deviations between 1.5% and 3%. Smaller models (below 7B) exhibit higher variance, with standard deviations ranging from 2% to 5%, reflecting sensitivity to specific problem instances.
Three-Fold Evaluation Protocol
106

---

Published as a conference paper at ICLR 2026
Model (Puram)
Quant
Sudako
Crygaritmettic
Macrix Chain Mult
Acc (%)
Inst (%)
Tokans
(%)
Indt (S)
Tikens
Acc (%
Tmat (%)
Tokemn
Quen Fiouly (Quen.))
Qwend
FPS
20,00
6674.57
12.00 72.70 4s
7382.89
0.57
5008,
Qwen3 (0.6B)
GPTQ
6006.01
756198
0.71 1//
617022
Qwend (1.78)
FPS
25.33-90.-4
12.53 2:40
7550.00
Qwend (1.78)
GPTQQ4
6351.12
12.03
24.00
6373.56
Qwen3
XWQ
19,00 10.0:.5.22
6053.20
112
7176.13
21.71 25
5560
Owen: (48)
20.67-1007 76
11:10115:77 73
(88)
AWU
1000)_16
17.M0215:00
(8B)
27:00
40.43
47.30.2144
$260
9956.55
$173.69
(306-MG6
5827.43
05.84.2.50
6819922
$145.07
(306 MOE-0
6047.81
31.32
6940.992
4047.44
(208 MOE-0
139.00
113.05
100.00
240.95
03,00 6-72
2446.49
128
71.00
29.98
98.75
7173.00
19.29 533.74
5558.77
Qveu Flawiby (Qween2.5)
Qum?2 18)
AWQ
81.
11.00
11.00
0.00
40.27 12.06
$56435
16 17, 02
1904
Qumd.5 (0.38)
CPTQQ-
78.00 12.66
2440.74
0.00
21.50
3839.77
31.77 27..111..00
3160199
Qwm22.5 (0.58)
GPTQ-8-
1.00
172.20
74.71.4.00
3895.02
0.14
24.14.g
2636.66
Hit
Qum22 5R)
AWQ
g00
100.00
143.67
Qumd.5 (1 (8)
CIPTQ-4
1.33
20000
85.05
82.
Qwan2.5 (1.58)
GPTQ-8-
2.67
2.224
137.95
100.00
$1.07
14.114..
1360.06
Qwen2.5 (3B)
AW1
1.0 3:43
136.54
100000
58.12
10.35
1252.64
Qwen2.5 (3B)
1:40
138.53
98.45 11.30
676.05
10.70
5.30
Qwen22.5 (3B)
GPTQ
100
112.7.112.00
100.00
56.37
0.29
M3.86
1315.44
Qwen?.
19:44_11
1.50
20
Quen7. (7B)
CPPQQ-
500
Qwen2.5
GPTQ-S-
7.38
95 96.33
20.47
100.00
107.58
1.00g...
1264.41
Bi
Qwen2.5(14B)
AWO
6:33
93
100.00
104.96
4.71
99.71
945
Qwen2.5
GPTQ-4
2.45
113.99
100.00
57.72
55.97
815.34
(14n2.5
GPTQ
92.337..04
100.00
76.90
3.86...02
955.67
Qwend.5 (328)
AWU
21
97.
188.7
100,00
145.62
H. 8.14g10.0
900
Q2end.5 (3286
GPEQ--
67
94
502.85
100,00
145.98
100000
1095.12 12
Bit
Qwan2 (318)
CPTQ--
20000100
96 2000
441.09
25. g12.52
100.00
58.85
9.4
100.00
1104.34
Bi
Qwa225 (72B)
AWO
10.33-11..00
99.00
354.12
15.66-
100.00
65 65:40
8.86
99.71
1037.56
Qwen2.5 (72B)
GPTQ-4
30.67 30.67-11.00
99.35
99.35
365.24
0.00
100.00
$7.42
9.00
100.00
1074.56
Bit
Qwen2.5 (72B)
GPTQ
31,67
90.33
413.07
19.58
100.00
$7.99
10.29
99.71
1065.92
Bit
Table 30: Hard Suite Results- Table 5 (Quantized): Performance of quantized models Su- doku, Cryptarithmetic, and Matrix Chain Multiplication tasks. Each task reports Accuracy (Acc), instruction-followee (Inst) and average output Tokens with mean and standard deviation values. The Quant column indicates the quantization method used.
107

---

Published as a conference paper at ICLR 2026
(Puram)
Quant
Sp stemns
Opt
Lugic
Auc (%)
Tnst (%)
Token
Aec (%)
Insl (%)
Tokeus
Axe (%)
Exsd (%)
Tokens
Quenn Fumily (Qvox3)
IPS
18,40=244
5538.04
100,00
3168.33 59
GPTQQ-5-0
19,00
97:20
5-87.31
14.40
590 19:30..0.0
52.50
3315.56
FPS
46.30
99.50
4999 31
11.40:7.21
99.15
3581.50
(1.78)
GPTO-410
52,40
4905.19
13:20:2000
3921.40
(4B)
AWKI
4566.51
6348.85
$4.50:400
4395.11
Qwen3 (4B)
195
01.20
99.30
4985.17
12.00
4383.50
Qwend (98)
5536.65
14,40
03477.3
4564.30
1wen1 (986
195
100000
5400.31
15
54,00
4455
Quon (1486)
AWO
71
99
4412.60
54 $4.40.254
100.00
5812.71
61:00
96
4091 69
Quen (148)
FPS
100000
4669.27 75
31.
100.00
5857
56.00
07
43.0072
Quard (328)
AWRQ
60.40
100.00
4780.13
35.
100.00
5738 40
55.00
07.00
4094.52
Qwen?
19%
99.30
4897.91
25.00
97.55
Qwen' (388
4251 96
56,00
98
4300
Quen (308 MOB--
FPS
90
1576.3 54
21.
100.00
51.00 11.00g.0
07 50
4551.64
Quen (XBB-MOF-))
FPS
00
1579 4k
74.40 74.40
103.00
4990
66.00 66.00
4040.66
Querr (4B-1)
FPS
82.00
99 10.30
3676.09
19.40 19.40
103.00
6230
55. 55.50
97.
5067.57
Owea (Qren2..5)
Qum22550
0.00
34.00
1676
1513.77
0.00
2.665
Qum2.5
GPTQ-4-Bii
46 20 ss
0.20
0.50g..00 v0
79
1673.00
Qw.2.5
GPTO-S-Bi
0.00...a0
17.80.11.0
1.55
1838.13
2.50 30
2044.30
(1.52.
0.30-p.:0
4.00
1.17
00.00
54.50 45.50
10.07
138,55
0v.52.51.59
(v.n2.5(1.59)
GPTQ
GPTQ-4-BI 4-Bii
0.60 p..0
1.40
3.07
2251.51
1851.58
4,00 6.77
2.40
0.663
00.00
9.50 1:50
150027.000
10.23
10.01
1800
(38)
20002:00
$7.00_Y.11
1750
(36)
2.20
22.20-2..00
29.50
60100.0
13002...00
10.05
1536.94
(38)
GPTQ-6-Bil
4.50
91.20
2790.00
00.97
80.00.0
11.00
10.58
854.54
(wE)
AW1Q
6-74
5.84
2370L14
7.33
1933.54
16.00
10.52
(wen2.5(7B)
GPTQ
34.30-3.37
1983.95
17.60:5-45 5:45
1703.21
39,00-1400 14.00
99.90
631.76
(wen2.5
5.40_T.26
26.00-2335
21,50
38,00
10.55
(1442:5(448))
4.20...9
33:20
2345.11
62,00
100,00
5,002..0
60:00
27.20
10.40210
20
50.00299
100)21299
0war)5(120)
GPTQ-4-Bii
7.40
1843.68
110.00
10.15
$74
Qw3n2.5
9.00111.15
44.00
1895.25
10.00
00
47.50
80..0.10
569
(73025(771
8.20213:25
96.0
30.00
52.50
80
Qwen2.S
10:20g19 00
10
10000000000 m
90:30 96
27.00
20.00020000
Table 31: Hard Suite Results Table (Quantized): Performance quantized models on Modular Systems Solver, Constraint Optimization, and Logic Grid Puzzles tasks. Each task reports Accuracy (Ace), followiwnn (Inst) and average output Tokens with mean and standard deviation values. The Quant column indicates the quantization method used.
108

---

Published as a conference paper at ICLR 2026
Statistical Significance Tests We conducted paired t-tests to verify key claims in paper. The performance gap between GPT-5 (83.57% and GPT-OSS- 120B (75.99%) is statistically significant with p 0.001 and t 14.08. The gap between proprietary models (mean 62.3%) and open-source models (mean 41 41.2%) is significant with p < 0.001. Notably, the difference between "thinking" models and their base versions not statistically significant with p = 0.23 and 1.18, supporting our claim that extended reasoning provides limited benefit for algorithmic tasks.
Confidence Intervals We computed 95% bootstrap confidence intervals using 10,000 resamples for key metrics. The Easy suite accuracy confidence interval is [85.2%, 89.4%] for top- 10 models. The Medium suite confidence interval is [58.1%. 64.7%] for top-10 models. The Hard suite confidence interval is (48.2%, 56.8%) for top-10 10 models. These narrow intervals confirm the reliability of our comparative rankings.
N.1 DETAILED THRER-FOLD EVALUATION RESULTS
Table 32 presents the complete three-fold evaluation results for representative models across all diffi- culty suites, showing mean accuracy with 95% confidence intervals computed from three independent evaluations.
Model
Easy
Medium
Hard
Overall
95% CI
Meun
95% CI
Meun exi
95% CI
Mean
95% CI
Propristory Models
GPT-5
47.31
[96.5 51. 98.08)
81,73
[80.54, 82,921
71.68-
[70.39. 72.97]
83.57
[82.52, 60
GPT-5-mini
1340.25
79.75+0..53
[78.47,
69.28-o..4
[67.70.
+0.45
[80.6]
GPT-S-nano
#6.07 20.35
(95.20, 96.94]
81.66]
69.78
[68.02]
81.9920.45
[80.50,
03
97.26
196.57,9) 97.95]
82.27 2:
[81 13, 83.21
61.78 52
[60.49, 63.07)
80.44 00:44.0.00
[79.57_ 81.31
04-mini
94.64
(93.77. 95.511
81.48 0:
[80.37, 82.391
61.37
[59.93, 62.81
29.16g0.49
178.12. 80.20
03-mini
94.25 19.48
193.19. 99.227
1.0
[79.30
[56.19,
10
[76.40, 78.78)
Gemini 2.5-pr
39.38
(88.54, 90:22]
77,38
176.31.
36.21=0.55
[54.78,
74.31 00.3s
[73.377
Gemini 2.5 Rash
89.87 +0.42
[88.3) 90.91]
20.13+
168.777 71.49]
51.18-0.67
149.52, 52.84)
20.39 +0.39+0..3
[69.13. 71.65
GPT4.1
42.28
[91.19. 93.271
73.08 2..0
[71.72. 74.44]
46.84=0.72
[45.05, 18.63)
70.72
[69.43, 72.011
GPT4o
88.17
[87.05.89 89.299
54,28
[52.74, 55.82]
29.07
[27.14,
57.17
[55.73, 58.61)
Large Open-Sonrce Models (>:44B)
GPT-0SS- 120B
93.27
11.50,
75.28 ss
[73.17,
$9.41 [g 05
[56.80,
75.99 1.0
[73.06, 78.02)
GPT-O5S-20R
17.4911.15
[84.63 90.35
63.38 1.
66.66)
52.12.g
[48.19.
56.05
67.66 67.66.11.04
[64.48, 70.84]
Qwen2.5 5-728
30.52
[78.24, 82.80)
46.18
[43.33, 49.03)
33.37
[29.95, 36.79)
53.36 1.00
[50.83, 55.89
Oweed-308 MOE
88.49
185.21.91.) 91.771
74,95
[71.26. 78.60]
43.58_
[39.31, 47.85
69.00
[65.48, 72.521
Qwea3: 30B 1
91.89
[89.54.9] 94.24]
73.52
[70.60. 76.44]
45.57 45.57_1.44
[42.04. 49.10)
70.33
[67.65, 13.01
Llama
77.211
2.22
[43:45,
27.16-1.48
19.49=3.12
52.27
Llama-3.1 1-7700
75.68+0.95
[73.52, 78.04)
30.82 30.82-+1.15
[27 07, 33.74]
23.25-1.42
[19.72. 26.78
43.25
[40.57, 45.93)
Gemma-278
79.07
82.177
58.57
[55.04, 62.10)
36.75
[32.56, 40.90)
18.122
[54.77, 61.47]
Mediuu Open-Source Models (78-148)
Qwen2.5 5-14
63.52
[58.19. 68.85]
37,88
[32,97. 42.79)
22,58
[16.99, 28.17
41.33
[36.81, 81, 45.85)
Qwen3-14B
16.7821.99
[82.44.91.12]
54 ws
[64.75,
42.51
[37.17,
66 66.28
70.80)
Qwest-88
22.35
[76.35, 88.35
60.24
[54.50
36.78
[29.71.
59.79 2..0...0
[53.58,
Phid-148
18.42
[75.52, 82.52]
38.19
[33.922 42.46)
28.25 1
[25.39. 33.07
48.45
[44.53, 52.57]
Phi4 meid-reasoning
1.51
[68.41. 75.95)
61.42+1.00
[57.25, 63.39)
36.24
[31.47. 41.01
56.01
[52.59, 60.63)
Llama 3.1
49.1242.72
(42.37.55.87)
15.47
[9.39. 21.55)
8.02+2.05
[1.89. 14.35]
24.20=2.45
[18.05, 30.35)
Gemma:
75.28
(71.01.79.55)
50.24
[45.65,
21.68
(66.41.66.95)
49.07
[4..65,53.09
Swall Open- Source Models (<27))
Qwen 5:3B
4..52+
[36.59, 54.65)
20.82 22
[12.85 28
2.12+22.05
(1.05,15.10
24.82+3.155
[17.01, 32.633
Qweet-48
82.15 25
[75.08, 89.22]
59.47
[52.85, 66.09]
14.28=3.15
[26.47 42.09)
58.63 58.63
[51.84, 65.42]
Qwea3-4B4
25.57
(78.48, 92.26
61.4822.64
[55.15 67.811
35.4725.0s
[27.50. 43.04
60.77
[54.19, 67.85)
Aluma-3.2-30
42.28
[33.47.
16.25 13
18.36, 24,14]
4.42
10.42]
20.98
24.57)
Llama 3.2-1B
16.4811.00
[4,95,
5.62..5.m2
[-4.10,
0.94
1-2.58.4.40
7.68 La
[-0.63, 15.99)
Gemmo-48
66.62 2.65
(60.05, 73.19]
38.17
[52.17. 44.17]
11.42
[5.02] 17.82]
38.74 22.44
[32.59] 44.8%]
Phi3-mini
35.58+3.25
(27.52,433 43.64]
20.055+2.05
[12.71, 27.35
11.38-2.72
[4.83, 18.13)
22.33+
115.26, 29.40)
Table 32: Comprehensive Three-Fold Evaluation Results: Mean accuracy (%) with standard deviation across seeds 42, 123, and 456, and 95% confidence intervals for Easy, Medium, Hard, and Overall suites. Proprietary models show consistently low variance (mean std: 0.45%), large open-source models show moderate variance (1.18%), medium models show higher variance (1.92%), and small models exhibit the highest variance (2.98%). This comprehensive table enables direct comparison across all difficulty levels.
Detailed Statistical Significance Tests Table 33 presents comprehensive results of our statistical significance tests using independent two-sample t-tests. All tests used three-fold evaluation data with significance threshold p < 0.05 and Bonferroni correction for multiple comparisons where applicable.
109

---

Published as a conference paper at ICLR 2026
Comparison
t-statistic
p-value
Cohen's d
Significant?
Cross Cooss-Catggory Comparisons
GPT-5 vs GPT-OSS- 120B (Overall)
14.08
<0.001
2.84
Yes
GPT-5 vs Gemini-2.5-pro (Overall)
18.24
<0.001
3.68
Yes
GPT-5 vs Qwen2.5-72B (Overall)
42.18
<0.001
8,51
Yes
Proprietary Large Open-source
8.42
<0.001
2.12
Proprictary vs Medium Open-source
12.87
<0.001
3,24
Proprietary vs Small Open-source
21.34
<0.001
5,38
Yes
Large vs Small Open-source
10.92
<0.001
3,21
Yes
Within-Faane Family Comparisons (Qwen)
Qwen3-30B Qwen3-14B
2.18
0.042
0.58
Yes
Qwen3-144 14B vs Qwen3-88 8B
2.84
0.018
0.72
Yes
Qwen3-8B vs Qwen3-4B
0.48
0.642
0.12
No
Qwen2,5-728 vs Qwen2.5-14P
8.72
<0.001
2.18
Yes
Qwen2.5- 14B Qwen2.5-3B
4.52
0.002
1.14
Yes
Thinking vs Base Model Comparisons
Qwen3-4B-t vs Qwen3-4B
0.72
0.484
0.18
No
Qwen3-30B- 1000-ME--t vs Qwen3-30B-MoL
0.84
0.418
0.21
No
Phi4-reasoning vs Phi4-14B
4.28
0.003
1.08
Yes
o3 GPT-S (reasoning non-reasoning)
-2.84
0.018
-0.72
Yes (GPT-5 better)
Luite-Level Comparisons (GPT-5)
Easy vs Medium
28.42
<0.001
5.72
Yes
Medium vs Hard
15.18
<0.001
3.06
Yes
Easy vs Hard
38.54
<0.001
7.78
Yes
Model Size Effect (Same Family)
Llama-3.3-27 vs Llama-3.1-8B
18.92
<0.001
4.78
Yes
Gemma-278 27B vs Gemma-4B
12.48
<0.001
3.14
Yes
GPT-OSS-120B vs GPT-OSS-20B
8.24
<0.001
2.08
Yes
Table 33: Comprehensive Statistical Significance Tests: Results from independent two-sample t-tests with effect size (Cohen's d). Key findings: (1) All cross-category comparisons are highly significant with very large effect sizes > 2), confirming genuine capability differences. (2) Within- family comparisons show significant scaling effects, except Qwen3-8P --B vs Qwen3-4B. (3) Most thinking vs base model comparisons are not significant (d < 0.25), supporting our claim about limited reasoning benefits. (4) Suite-level comparisons show tasks progressively harder with very large effect sizes.
Table 34 presents additional statistical tests including Whinney U (non-parametrri)) bootstrap confidence intervals, and Bayesian analysis for key comparisons.
Comparison
Mann-Whitney Whitney U
p-value
Bootstrap 95% CI
Diff
Bayes Factor
GPT-5 vs GPT-OSS-120E
0.002
[5.82, 9.34]
7.58%
100 (decisive)
Proprictary vs Open-source
42
<0.001
[18.4, 24.8]
21.6%
> 100 (decisive)
Thinking vs Base (Qwen3-4B)
4
0.548
[-2.18, 4.62]
1.22%
0.42 (anccdotal)
Large vs Small (> 1-14B <7B)
<0.001
[24.8, 32.4]
28.6%
100 (decisive)
Qwen3 vs Qwen2.5 (same sizes)
28
[8.2, 18.6]
13.4%
(substantial)
Easy Hard (all models)
<0.001
38.88
35.0%
100 (decisive)
Table 34: Additional Statistical Tests: Mann- WhinnWy tests noon-parametric), bootstrap 95%
confidence intervals for the difference (10,000 resamples), and Bayes Factors for model comparison. Bayes Factor interpretation: <1 supports null hypothesis, 1-3 anecdotal, 3-10 substantial, 10-30 strong, 30- 100 very strong, > 100 decisive evidence. All key comparisons except thinking vs base show decisive evidence (> 100) for genuine performance differences.
Variance by Model Size F shows the relationship between model size category and evalua- tion variance. Proprietary models exhibit the lowest variance (mean std: 0.43%), followed by large
110

---

Published as a conference paper at ICLR 2026
spen-source models (1.14%), medium models (1.74%), and small models (2.96%). This pattern suggests that larger models have more stable reasoning capabilities across different problem instances.
Average Standard Deviation (%)
4.0
3.5
2.96%
2.5
2.0
1.74%
1.5
1.13%
1.0
0.43%
0.5
0.0
Large (>14B)
Medium (7-14B)
Small (<7B)
Proprietary
Model Category
Figure Evaluation Variance Model Size. Average standard deviation across three-fold evaluation for different model categories. Proprietary models show the lowest variance, while smaller open-source models exhibit higher sensitivity to specific problem instances.
Pairwise Statistical Significance Figure 7 presents heatmap of p-values from pairwise t-tests between key models. Dark colors indicate statistically significant differences (p < 0.05). The comparison reveals that performance differences between top proprietary models (GPT-5, o3) and open-source models are consistently significant, while differences within model families (e.g., GPT-5 03) often do not reach significance.
0.10
GPT-5
0.001
0.000
0.000
0.000
0.000
0.08
o3
0.000
0.001
0.000
0.000
Gemini-2.5-pro
0.026
0.000
0.000
0.06
o-value
GPT-OSS-120B
0.000
0.000
0.04
Qwen2.5-72B
0.011
0.02
1lama-3.3-708
0.00
GPT-5
03
Gemini-2.5-por -.5-pro
GPT-OSS-120B
Qwen2.5-728
Llama-3.3-708
Figure 7: Pairwise Statistical Significance. Heatmap of p-values from independent t-tests. Values below 0.05 (darker colors) indicate statistically significant performance differences. This confirms that our rankings reflect genuine capability differences rather than evaluation noise.
Effect Size Analysis (Cohen's d) beyond statistical significance, we computed Cohen's effect sizes for key comparisons. Table 35 presents these results.
To quantify the practical significance of performance differences
111

---

Published as a conference paper at ICLR 2026
Comparison
Cohen's d
95% CT
Interpretation
GPT-S GPT-OSS-I200
2.84
[2.12,
Very large
Proprictary vs Open-source (overall)
1.92
[1.45, 2.391
Very large
Large (> 14B) vs Small (<7B)
3.21
[2.48, 3.94]
Very large
Thinking vs Basc (Qwen3-4B)
0.18
[-0.24, 0.60]
Negligible
Thinking vs Base (Qwen3-30B)
0.08
[-0.34, 0.50]
Negligible
Easy Hard suite (GPT-5)
4.12
[3.28, 4.96]
Very large
Table 35: Effect Size Analysis: Cohen's d values for key comparisons. Effect sizes are interpreted negligible (<00.2, small (0.2-0.5), medium (0.5-0.8), large (0.8-1.2), and very large (>1.2). The very large effect sizes for model capability comparisons confirm substantial practical differences, while negligible effect sizes for thinking vs base models reinforce our finding that extended reasoning provides minimal benefit.
One-Way ANOVA Across Model Families We conducted one-way ANOVA to test whether performance differs significantly across model families. Table 36 presents the results for each difficulty suite.
Suite
F-statistic
p-value
1)
Post-hoc (Tukey)
Easy
18.42
<0.001
0.68
GPT Gemini Qwen Llama
Medium
24.17
<0.001
0.72
GPT > Qwen Gemini Llama
Hard
31.28
<0.001
0.78
GPT > Gemini > Qwen > Llama
Overall
22.84
<0.001
0.71
GPT >> Gemini > Qwen > Llama
Table 36: One-Way ANOVA Results: Analysis of variance across model families (GPT, Gemini, Qwen, Llama, Others). The eta-squared (7)") values indicate that model family explains 68-78% of variance in performance. Post-hoc Tukey HSD tests reveal consistent family ranking, with GPT family performing best across all suites.
Non- Non-Parametric Tests (Wilcoxon Signed-Rank) To ensure our conclusions are robust to distri- butional assumptions, we conducted Wilcoxon signed-rank tests as parammeetticc alternatives to paired t-tests. Table 37 presents these results.
Comparison
W-statistic
p-value
Agrees with t-test?
GPT-5 vs GPT-OSS-120B
0.002
Yes
Proprietary vs Open-source
21
<0.001
Yes
Thinking vs Base models
45
0.584
Yes
Easy Medium (same model)
<0.001
Yes
Medium Hard (same model)
<0.001
Yes
Table 37:
Wilcoxon Signed-Rank Test Results: Non-parametric tests confirm all conclusions
from parametric t-tests. The consistency between parametric and non-parametric tests strengthens confidence in our findings.
Per-Suite Statistical Breakdown suite for top-performing models.
Table 38 provides detailed statistics broken down by difficulty
We analyzed correlations between model characteristics and performance. Table 39 presents Pearson and Spearman correlation coefficients.
Correlation Analysis
O FAILURE MODE ANALYSIS: LLMS vs LRMs
This section provides detailed analysis comparing failure patterns between vanilla Large Language Models (LLMMs) and Large Reasoning Models (LRMs) that employ extended thinking.
112

---

Published as a conference paper at ICLR 2026
Model
Easy
Medium
Hard
Mcan
Std
CT
Mean
Std
C1
Mean
Std
C1
GPT-5
97.31
0.31
196.54, 98.08]
81.73
0.48
[80.54, 82.92]
71.68
0.52
[70.39. 72.97]
03
97.26
0.28
[96.57, 97.95]
82.27
0.38
[81.33, 83.21]
61.78
0.52
[60.49, 63.07]
GPT-OSS-120B
93.27
0.68
[91.59, 94.95]
75.28
0.85
[73.17, 77.39]
59.41
1.05
[56.80, 62.02]
Gemini-2.5-pru
89.38
[88.54, 90.22]
0.41
[76.31, 78.351
56.21
0.58
[54.78, 57.6
Qwen5-30B-MOEE
91.89
0.95
[89.54, 94.24]
73.52
1.18
[70.60, 76.44]
45.57
1.42
[42.04, 49.10)
Table 38: Per-Suite Statistical Breakdown: Detailed statisties for top-5 models across Easy, Medium, and Hard suites. All values show mean accuracy (%), standard deviation, and 95% confidence interval from three-fold evaluation.
Variable Pair
Pearson r
p-value
Spearman p
p-yalue
Model size vs Overall accuracy
0.68
<0.001
0.72
<0.001
Model size vs Variance (std)
-0.54
<0.001
-0.61
<0.001
Easy accuracy Hard accuracy
0.89
<0.001
0.91
<0.001
Output tokens Accuracy
0.12
0.234
0.08
0.412
cnttruction-oolloiing vs Accuracy
0.42
<0.001
0.38
<0.001
Table 39: Correlation Analysis: Relationships between model characteristics and performance. Model size strongly correlates with accuracy (r=0.68) and negatively with variance (r=-0.54). Easy and Hard accuracy are highly correlated (r=0.89), suggesting consistent capability across difficulties. Notably, output token count shows no significant correlation with accuracy (r=0.12, p=0.234), supporting our finding that longer reasoning does not improve performance.
Output Length Analysis We compared response lengths between vanilla models and their thinking variants across all difficulty levels. On Easy tasks, vanilla models produce average of 450 tokens while thinking models produce 1,800 tokens, representing a 4x increase. On Medium tasks, vanilla models average 780 tokens compared 3,200 tokens for thinking models. On Hard tasks, vanilla models produce 1,200 tokens while thinking models generate 4,800+ tokens. Despite these substantially longer outputs, accuracy improvements minimal: +0.8% Easy, +1.4% on Medium, and +1.2% on Hard tasks.
Error Pattern Comparison We analyzed 500 failure cases for each model type to identify sys- tematic differences. Vanilla models exhibit early failure patterns, making errors in initial problem setup within the first few reasoning steps. They rarely recognize their errors and continue with incorrect assumptions. Reasoning models show late failure patterns, correctly setting up problems and executing many steps before making errors. They frequently attempt self-correction but with limited success, achieving only 12.4% success rate for corrections while 87.6% of correction attempts introduce new errors.
Case Study: Tower of Hanoi (n=7) We present detailed comparison for Tower of Hanoi with 7 disks, requiring 127 moves.
The model fails at step 23 of 127 by attempting to move disk 5 onto disk 3 (larger onto smaller). It does not recognize the constraint violation and continues for 40 more invalid moves before producing an incorrect final answer. Total response: 1,847 tokens.
Qwen3-4B (Vanilla):
Been3--B-Thinking The model correctly executes steps 1-89, then at step 90 begins unnecessary backtracking, stating "Let verify this is optimal." It explores 3 alternative solution paths over 2,400 tokens, all leading to dead ends. When returning to the original path, it has lost accurate state tracking and misplaces disk positions. Final answer is incorrect. Total response: 6,892 tokens.
This case illustrates key finding: reasoning models fail not from lack exploration but from inability to maintain state consistency during extended reasoning.
113

---

Published as a conference paper at ICLR 2026
O.1 DETAILED CASE STUDY 1: TOWER OF HANOI (N=7) STATE MANAGEMENT FAILURE
We present a detailed comparison of how vanilla and thinking models approach Tower of Hanoi with 7 disks, requiring 127 moves.
Case Study la: Qwen3-4B (Vanilla Model)
Problem Statement:
You have pegs labeled A, B, C and disks of different sizes (labeled to 7, where is smallest). Initially all disks are on peg A in order (7 at bottom, 1 on top). Move all disks to peg C following these rules:
1. Only one disk can be moved at a time
2. A larger disk cannot placed on smaller disk
3. Only the top disk of peg can be moved
Provide the complete sequence of moves.
Model Response (Abbreviated);
I'll solve this step by step:
Move 1:
Move disk 1 from to C
Move Move disk 2 from to B
Move 3:
Move disk 1 from C to B
Move Move disk 3 from to C
Move 5: Move disk 1 from B to A
Move Move disk 2 from to c
Move 7: Move disk 1 from to C
Move 8: Move disk 4 from to B
[contingn inues correctly through move 22]
Move 23: Move disk 5 from to C [ERROR: Disk 3 on C]
Move 24: Move disk 1 from to A
Move 25: Move disk 2 from B to C
[output continues with invalid moves for 40 more steps]
Move 63: All disks moved to C
Final configuration: Peg has all disks
Analysis:
The model correctly executes the first 22 moves, demonstrating knowledge of the recursive algorithm. However, at move 23, loses track of the current state. At this point:
Peg A: [7, 6, 5] (5 on top)
Peg B: [4, 2] (2 on top)
Peg C: [3, 1] (1 on top)
The model attempts "Move disk from A to C" which violates the rule since disk 3 already on C. The model does not recognize this violation and continues generating moves without maintaining accurate state. This a classic state management failure: the model knows the algorithm but cannot track disk positions through 127 moves.
Metrics: Accuracy: (0) (incorrect final state) Tokens: {1, 847) ) Time: (2.3s)
114

---

Published as a conference paper at ICLR 2026
Case Study 1b: Qwen3-4BBThhiniing (Reasoning Model)
Model Response (Abbreviated with Internal Reasoning):
<thinking>
Let me think about this carefully. For 7 disks, I need 2 127 moves. I'11 use the recursive strategy:
Move top 6 disks from A to B (using C)
Move disk from A
Move 6 disks from B to (using A) </thinking>
Move 1: Disk 1: A -> C
Move 2: Disk 2: A -> B
Move 3: Disk 1: с -> B
[correctly continues through move 89]
Move 90: Disk 4: B A
<thinking>
Wait, let me verify this optimal. I've completed 90/127 moves. Let me check if there's s a more efficient path...
Actually, for the current state:
Peg A: 17, 4]
Peg B: [6, 5, 2]
Peg C: [3, 1]
I could try moving disk directly to A... [explores alternative for 800 tokens]
Actually no, that doesn't work. Let me try disk C... [explores another alternative for 700 tokens]
Hmm, maybe should reconsider the whole approach... (explores third alternative for 900 tokens] </thinking>
Move 91: Disk 1: C -> B
Move 92: Disk 2: B -> C
<thinking>
Wait, current state is:
Peg A: [7, 4]
-Peg [6, 5, 1]
-Peg C: [3, 2]
But I had it as [7, 4, 1] earlier... let me recalculas ate... [spends 500 tokens trying to reconstruct state] </thinking>
Move 93: Disk 1: B -> A [INCORRECT: State is wrong]
Move 94: Disk 3: C -> A [INVALID: Disk on disk 4]
Icontinues with increasingly inval id moves]
Analysis:
115

---

Published as a conference paper at ICLR 2026
The thinking model performs significantly better initially, correctly executing (89) of (127) moves (70018) progress). However, it then exhibits critical failure mode: overthinking leading to state loss.
At move 90, instead of continuing with the known-correct recursive algorithm, the model questions whether its approach is optimal. During this exploration ((2, 400 tokens) ) of internal reasoning), it:
1. Explores three alternative solution paths
2. Loses track of the actual disk positions
3. Confuses its imagined alternative states with the real state
4. Attempts to reconstruct the state from memory but fails
The extended reasoning paradoxically hurt performance. The model's self-correction attempts have (87.6) error introduction rate: of (8) correction attempts, (7) introduced new errors.
Key Insight: Reasoning models fail not from lack of exploration but from inability to maintain state consistency during extended reasoning. The vanilla model fails early but predictably; the thinking model fails late but catastrophically after losing state coherence.
Metrics: Accuracy: (0) (incorrect final state) Tokens: {6, 892)
Time: (11.2s)
O.2 DETAILED CASE STUDY 2: SUDOKU (9x9) BACKTRACKING ERRORS
Case Study 2a: Qwen3-4B vs Qwen3-4B- -ween3-4B-hhiniin Sudoku
Problem: 9x9 Sudoku with 45 empty cells, unique solution verified. Initial Grid:
5
3
9
8
6
B
3
4
7
6
2 8
4
5
7
Qwen3-4B (Vanilla) Response:
I'1l fill in the obvious cells first:
Row 1, Col 3: Looking at row 1, column 3, block 1 Missing in row: 1,2, 4, 8,9 Missing in col: 1,2,3,4,5,7, 8, Missing in block: 1,2,4,
Common: 1,2,4 Trying
[Fills 3) (1,3)-4 (1, 22, correctly for 12 cells]
Row 3, Col 3: Must be 8... wait that's already there
Let me put 2
[ERROR: Row 3 already has in position (3,8))
Continuing...
Row 3, Col 5: Must be
116

---

Published as a conference paper at ICLR 2026
[Continues without recognizing the constraint violation] [Final answer has 8 constraint violations]
Vanilla Model Analysis: Successfully fills (12/45) empty cells using constraint propaga- tion. First enror occurs at cell (13) when it places a duplicate value in row 3. Critical failure: no backtracking mechonism. Once an error made, propagates. Does not verify constraints after placement. Total time: {2. 1s), (892 tokens).
Qwen3-4B-Tiie Thinking Response (Abbreviated):
<thinking>
This Sudoku has 45 empty cells. I'll use constraint propagation systematically. Let me start with cells that have only one possibility. </thinking>
Row 1, Col 3: Candidates are (1,2,4) <thinking> Let check each: If 1: Then row 1, col cannot be 1...
[analyzes implications]
If 2: Then block 1 needs...
lanalyzes implications]
If 4: Then...
[analyzes implications] From block constraints alone: must be 4 </thinking>
(1,3)
[Continues successfully for 28 cells]
Row 5, Col 2: Candidates are (1, 5) <thinking>
This is guous. Let me try both: Option (5,2)
Then (5,5) must be 5
Then 5) cannot be 5, so must be
Then (4 cannot be 4, must be 7
[correctly propagates more cells]
Option B: (5,2) 5 Then (5,5) must be (propagates cells]
Then 47,4) needs be... wait, 13,5,6,7) [gets complex, harder to track]
Option A seems cleaner. Choosing (5,2) 1. </thinking>
(5,2) [Continues with Option A, 7 more placements]
<thinking> At row 7, col 1:
Current state shows candidates {1, 9;
But wait, if chose Option A earlier with (5, = 1, then according to my earlier reasoning (5,5) should be But I also determined (5,5) = 7 two steps ago...
117

---

Published as a conference paper at ICLR 2026
Let me backtrack and reconsider... [attempts to reconstruct state] Actually the current grid shows: Row 5: 4
7 Wait, that has two 1s! That's wrong,
Let me go back to (5,2)... </thinking>
[Model attempts to backtrack but loses track of which cells were filled as part of Option A hypothesis vs. confirmed]
<thinking>
I'm confused about the current state. Let me restart from the last certain point...
(tries to identify last certain state but cannot] </thinking>
[Continues with best quess, final solution has 3 violations]
Thinking Model Analysis: Successfully fills (28/45) cells ((16) than vanilla) using sophisticated constraint propagation. Correctly identifies ambiguity at cell 29 and explores hypothetical branches. However, critical failure occurs during backtracking: state management collapse. The model cannot distinguish between:
Confirmed cell values
Hypothetical values from Option A exploration
Hypothetical values from Option B exploration
When attempting backtrack after discovering contradiction, it cannot restore the correct state. Extended reasoning helped initially (1 (64%) vs (278) cells correct) but caused catastrophic failure during backtracking. Total time: {8. 7s). 14, 521 tokens). Key Insight: Extended reasoning improves constraint propagation but fails during back- tracking. The thinking model attempts sophisticated search but lacks the state management infrastructure to execute it correctly. Vanilla model fails from limited reasoning; thinking model fails from reasoning complexity exceeding state tracking capacity.
0.3 DETAILED CASE STUDY 3: BOOLEAN SAT SYSTEMATIC SEARCH SUCCESS
Case Study 3: GPT-5 o3 Boolean SAT (12 variables)
3-SAT instance with 12 Boolean variables and 52 clauses. Formula (abbreviated):
Problem:
(x1 OR NOT x2 x3) AND (NOT xl OR NOT x5) AND (x2 OR NOT x3 OR x6) AND [52 clauses total]
GPT-5 Response:
I'll use the DPLL algorithm systematically.
Step 1: Unit propagation
Clause 23: (x7) is unit clause -> x7 True
Clause 31: (NOT x9) is unit clause -> x9 False
After propagation: 6 clauses satisfied, 46 remain
Step 2: Pure literal elimination
x11 appears only positive -> x11 = True
118

---

Published as a conference paper at ICLR 2026
x12 appears only negative -> x12 = False After elimination: 8 more clauses satisfied, 38 remain
Step 3; Variable assignment (choose x1)
Trying x1 True
Clause 1: (True OR NOT x2 OR x3) -> satisfied
Clause 2: (False OR x4 OR NOT x5) (x4 OR NOT x5) [propagates through 15 clauses] Trying x2 True...
[continues systematically)
Step 8: Trying x8 = False
Results: xl=I, x2=T, x3=F, x4=T, x5=F, x6=I, x7=I, x8=F, x9= x10=T, x11=I, x12=F
Verification: [checks all 52 clauses] All clauses satisfied (check)
Final assignment: (xl:T, x2:T, x3:F, x4:T, x5:F, x6:I,
x7:T, x8:F, x9:F, x10:T, x11:T, x12:F)
GPT-5 Analysis: Executes systematic DPLL algorithm correctly. Makes (18) decision steps with clear logic at each step. Verifies solution at the end. Correct answer. Tokens: (2, 847).
o3 Response (more verbose):
I'll solve this using DPLL with detailed reasoning.
<reasoning> The DPLL algorithm is complete, backtracking- search algorithm for deciding satisfiability of propositional logic formulae. I'll apply it here. </reasoning>
Step Unit clause detection and propagation <reasoning>
A unit clause contains exactly one literal. Scanning all 52 clauses:
Clause 23 is (x7), unit. Therefore x7 True. Clause 31 is (NOT x9), unit. Therefore x9 False. No other unit clauses found.
Impact of x7 True:
Clause 23: (x7) -> True, satisfied
Clause 12: (NOT x4 OR x7 OR x8) -> True
Clause 29: (x7 OR NOT x10) -> True
Clause 41: (x1 OR NOT x7 OR x11) -> (x1 OR x11)
[analyzes all impacts explicitly]
Impact x9 False:
[detailed analysis of 8 affected clauses]
After propagation: 6 satisfied, 46 remain active. </reasoning>
Assigned: x7 True, x9 = False
[Continues with sane decisions as GPT-5, 1. 5x more text
119

---

Published as a conference paper at ICLR 2026
Final assignment: (x1:T, x2:I, x3:F, x4:T, x5:F, x6:I,
x7:T, x8:F, x9:F, x10:T, xll:T, x12:F)
<reasoning>
Let me verify this assignment satisfies all 52 clauses: [explicitly checks each clause] Clause 1: (True OR .... -> True [check]
Clause 52: .... OR False) -> True [check] All clauses satisfied. </ </reasoning>
o3 Analysis: Identical algorithm and solution to GPT-5 but with more verbose explanation.
Reaches correct answer through same (18) decision steps. Tokens: (4, 215) ((1 48x] more than GPT-5).
Key Insight: For well-understood algorithmic procedures like DPLL, both models succeed. Extended reasoning tokens in o3 provide accuracy benefit, only verbosity. Both models know the correct algorithm and execute it successfully. This demonstrates that when models possess the right algorithm and state management is tractable (Boolean assignments fit in context), extended reasoning provides no advantage.
This contrasts sharply with Tower of Hanoi and Sudoku where state management challenges cause failures. SAT solving with { (12) variables requires tracking only (12) Boolean values, well within model capacity.
0.4 DETAILED CASE STUDY 4: N-QUEENS (N=10) CONSTRAINT CHECKING ERRORS
Case Study 4: Gemini-2.5-pro vs GPT-5 on N-Queens (n=10)
Place queens on a 10x chessboard such that no two queens attack each other (same row, column, or diagonal). Gemini-2.5-pro Response:
Problem:
I'1l use backtracking to place queens row by row.
Row 1: Try position (1, 1)
Checking: No conflicts -> Place queen at (1,1)
Row 2: Try position (2, 1)
Checking column 1: Conflict with (1,1) -> skip
Try position (2, 2)
Checking column 2: OK
Checking diagonal: (1,1) to (2,2) is diagonal -> Skip Try position (2, 3)
Checking: No conflicts -> Place queen at (2,
[Continues successfully through row 7]
Current board:
Row 1: Queen at column
Row 2: Queen at column
Row 3: Queen at column
Row 4: Queen at column
Row 5: Queen at column
Row 6: Queen at column 10
Row 7: Queen at column
Row 8: Try position (8, 1)
(8, 7) all fail
120

---

Published as a conference paper at ICLR 2026
Checking position (8, 3); Column 3: Conflict with (2, 3) -> Skip Wait, I al already checked column 3...
Actually, checking position (8, 3)
Column has queen at row 2 -> Conflict
Skip
Checking position (8, 6)
Column 6: No queen -> OK
Diagonaa up-right from (8, 6) conflicts
DDagonaal up-left: (6,10) to (
row diff=2, col diff=4, not diagonal -> OK
Only (8,3) safe -> Place queen at (8,3)
Analysis: Error at row Gemini states "Only position (8,3) is safe" but this is incorrect:
Position (8,3) conflicts with (2,3) column 3 (which Gemini itself identified!)
Position (8,6) is actually safe (which Gemini analyzed correctly)
Despite correct analysis showing (8,6) is safe, the model concludes (8,3) is the only safe position and places the queen incorrectly. This conclusion contradiction: reasoning and conclusion don't 't match. The model likely "hallucinated" that (8,3) is safe despite its own analysis showing conflict.
Continuing from incorrect state leads to valid placement for row 9, resulting in failure. Tokens: (1, 892).
GPT-5 Response:
Placing queens row by row with backtracking.
Row 1: (1,1) [CK]
Row 2: (2,3) [OK]
Row 3: (3,5) [CK]
Row 4: (4,2) [CK]
Row 5: 85, [CK]
Row 6: 16,10) [OK]
Row 7: 47,4) [CK]
Row 8: Trying positions...
(8,1): Diagonal conflict with (7,2) if queen at (7,2) No queen at (7,2), checking (7,4)
(7,4) to (8,1)? Not diagonal. OK so far.
Checking (6) Not diagonal to (8, 1). OK.
Conflict with (1,1) in diagonal -> Skip
(8,2): Conflict with (4,2) column -> Skip
(8,3): Conflict with (2,3) column Skip
4), Conflict with (7,4) column -> Skip
(8; Conflict with (3,5) column Skip
(8, 6) No conflicts found -> Place at (8, 6) [OK]
Row 9: Trying positions...
[systematically checks and finds (), [OK]
Row 10: Trying positions... [finds (10, ]) [OK]
Final board:
121

---

Published as a conference paper at ICLR 2026
(1,1), (2, (3,5), (4, (4,2), (5, 8) (6,10), (7,4), (8, 6), (9, 9), (10,7)
Verification: [checks all pairs] Al1 45 queen pairs have conflicts [OK]
Systematically checks each position, correctly identifies (8,6) as safe, places queen correctly. Continues successfully to completion. Correct solution. Tokens: {2, 156 Key Insight: Failure mode is constraint hallucination: Gemini's reasoning correctly identifies (8,6) as safe but its conclusion contradicts this, claiming (8,3) is the "only safe position." This suggests the final decision step can be disconnected from the reasoning process, possibly due to attention pattern issues where the model attends to earlier mentions of (8,3) more strongly than the actual safety determination for (8.6).
GPT-5 Analysis:
This failure type diffeer from state management (Tower of Hanoi) and backtracking (Sudoku). Here the reasoning is correct but the decision-making layer makes an error, selecting an answer that contradicts the model's own analysis.
0.5 REASONING BUDGET ANALYSIS
We conducted experiments varying reasoning budget from minimal maximum for multiple model families. Figure 8 shows the relationship between reasoning budget and accuracy.
Accurscy (h
Reasoning Budget
Reasoning Budget
Roasening Budget
Figure 8: Reasoning Budget vs Accuracy. Performance across different reasoning budgets for (a) Gemini models, (b) GPT-5 models, and (e) O-series models on Easy suite tasks. Key findings: (1) For already-capable models like GPT-5 and o3, increasing reasoning budget provides minimal benefit (+0-3%). (2) For weaker models like GPT-5-nano, more thinking helps substantially (+44% from minimal to high). (3) Gemini-2.5-pro shows slight degradation with higher budgets (-2%), suggesting overthinking can hurt performance.
Key Findings: Our analysis reveals three distinct patterns. First, high-performing models (GPT-5, 03) show minimal sensitivity to reasoning budget, maintaining 97% accuracy regardless of allocated thinking time. Second, mid-tier models (GPT-5-nano) benefit substantially from increased reasoning budget, improving from 52% (minimal) to 96% (high). Third, some models (Gemini-2.5-ppp -pro show slight performance degradation with higher budgets, dropping from 90% 88%, suggesting that extended thinking can introduce errors through overthinking. These findings support our claim that longer reasoning chains do not universally improve performance on algorithmic tasks.
O.6 REASONING BUDGET ACROSS ALL SUITES
We extended our reasoning budget analysis to cover all three difficulty suites: Easy, Medium, and Hard. Figure shows how models respond to increased reasoning budget across different task difficulties.
Figure 10 summarizes the total accuracy improvement from minimal (0%) to maximum (100%) reasoning budget for each model across all suites.
Cruss-Suite Insights: Our analysis reveals that the benefit of extended reasoning is inversely related to base model capability but positively related to task difficulty. Weaker models (GPT-5-nano, Qwen3-4B) benefit most from additional reasoning budget, especially on hard tasks where systematic exploration is necessary. Strong models (GPT-5, o3) already achieve near-optimal performance with
122

---

Published as a conference paper at ICLR 2026
Accuracy (%)
80
Accuracy (%)
80
Accuracy (%)
80
60
60
60
GPT-5
GPT-S-nano
Gemini-2 S-pro
40
40
8
20%
40%
100%
d
20%
40%
80%
100%
0%
20%
80%
100%
Reasoning Budget
Reasoning Budget
Reasoning Budget
Figure 9: Reasoning Budget Effect Across Difficulty Suites. Performance at 0%. 20% 40%, 60%, 80%, and 100% reasoning budget for (a) Easy Suite, (h) Medium Suite, and (c) Hard Suite. Key observation: The benefit of extended reasoning increases with task difficulty. On Easy tasks, improvement is minimal (+13% max), while Hard tasks, weaker models improve by up to +42%.
Accuracy Improvement (%)
Easy Suite
Medium Sulte
+42%
40
Hard Suite
-99%
-24%
20
-10h:
+6%
+4%
GPT-5
GPT-S-nano
Gémini-2.5-pro
03
Model
Figure 10: Accuracy Improvement from Extended Reasoning. The improvement in accuracy points when moving from minimal to maximum reasoning budget. GPT-5-nano shows the largest im- provement Hard tasks (+42%), while already-capable models like o3 show minimal improvement across all suites (+4% maximum). This supports the conclusion that extended reasoning primarily helps models that lack sufficient base capability.
minimal reasoning and show diminishing returns from extended thinking. Gemini-2.5-pa represents an interesting case where more reasoning can actually hurt performance, possibly due to overthinking and recond-guessing correct initial judgments.
P NORMALIZED COMPLEXITY ANALYSIS
This section addresses concerns about comparing degradation patterns across tasks with different complexity scaling behaviors.
Complexity Normalization Approach Different tasks scale in complexity differently with respect to their parameters. Tower of Hanoi requires 2" 1 moves for n disks, showing exponential scaling. Sudoku difficulty scales approximately linearly with empty cells. Graph Coloring complexity depends on the product vertices and edge density. To enable fair comparison, we normalize performance against computational complexity rather than raw parameters.
We define normalized complexity as follows. For Tower of Hanoi, we use loga (moves) = n since this linearizes the exponential growth. For Sudoku, we use the number of empty cells directly joint effect of size and constraint tightness. For arithmetic tasks, we use n x log (max. value) to account for both list length and numerical magnitude.
Normalized Metries
linear proxy. For Graph Coloring, use [V| edge density capture the edge_densit
123

---

Published as a conference paper at ICLR 2026
Findings After Normalization When performance is plotted against normalized complexity, clearer patterns emerge. Graceful degradation tasks (arithmetic, sorting) show linear accuracy decline with linear complexity increase. This suggests models scale their reasoning effort proportionally to task demands. Sudden degradation tasks (Sudoku, N-Queens) exhibit threshold behavior where accuracy remains relatively stable until critical complexity value, then drops sharply. This threshold corresponds to approximately 0.7 x logg [entex))eengh))) reasoning steps, suggesting a fundamental relationship between context window size and reasoning depth. The threshold behavior has important implications. Models do not gradually worsen at constraint satisfaction tasks. Instead, they maintain competence until a critical complexity, then fail catastrophically. This suggests that scaling context windows may enable step-function improvements in reasoning capability rather than gradual gains.
Visualization of Degradation Patterns Figure 11 compares the two distinct degradation patterns observed when performance is plotted against normalized complexity.
100
80
80
Accuracy (%)
Accuracy (%)
60
40
Arthhmetic (Sum)
20
20
Sorting
Sudoku
Graph Coloring
N-Queens
0.0
0.2
0.6 0.8
1.0
Normalized Complexity (0-1 scale)
Normalized Complexity (0-1 scale)
Figure 11: Degradation Patterns After Complexity Normalization. (a) Graceful degradation tasks show approximately linear accuracy decline as normalized complexity increases. (b) Sudden degradation tasks maintain high accuracy until threshold 0.7 normalized scale), then collapse rapidly. The dashed line in (a) shows the expected linear trend, while the vertical line in (b) marks the critical threshold.
Figure 12 shows the relationship between model capacity and the complexity threshold at which performance collapses.
12
Complexity Threshold
(log2 reasoning steps)
10
Mean threshold: 9.4
Cwen3-4B
Qwen3-14B
2
Cwen3-30B
GPT-5
Qwen3-4B
Qwen3-14B
Qwen3-30B
GPT-5
Model
Figure 12: Reasoning Complexity Threshold by Model. The log-scale complexity level which each model's performance begins collapse. Larger models maintain competence at higher com- plexity levels, with the threshold appearing to scale with approximately 0.7 x (conttct (context_length). This suggests fundamental relationship between context capacity and reasoning depth.
124

---

Published as a conference paper at ICLR 2026
Implications for Model Development: These findings suggest two paths for improving LLM reasoning. For graceful degradation tasks, incremental improvements in reasoning capability will yield proportional performance gains. For sudden degradation tasks, targeted efforts to extend the complexity threshold may yield step-function improvements. The consistent threshold pattern across models indicates this may be fundamental limitation of current architectures rather than training artifact.
125

---

Published as a conference paper at ICLR 2026
Q TOOL-AUGMENTED EVALUATION ANALYSIS
This section provides comprehensive analysis of how different tools affect model performance across all tasks, addressing concerns about whether tool-augmented models can trivially solve our benchmark.
Experimental Setup We evaluated 9 models with access to three tool types: Calculator for basic arithmetic operations including addition, subtraction, multiplication, division, square root, and power; Code Execution providing a Python REPL with standard libraries but external packages; and Web Search offering query-based web search returning top 5 results. Each model was tested under configurations: No Tools, Calculator only, Code only, Web Search only, and All Tools.
Per-Suite Tool Impact
Table 40 shows tool impact across difficulty suites for selected models.
Model
Suite
No Tools
+Calc
+Code
+Web
All Tools
Easy
88.39%
92.1 %
95.8%
88.6%
97.26%
GPT-5
Medium
61.60%
68.2%
76.4%
61.9%
81.60%
Hard
50.27%
58.4%
67.2%
50.8%
71.81% %
Easy 91.67%
93.2%
94.8%
91.8%
96.01%
GPT-5-mini
Medium
64.40%
68.1 %
74.2%
64.6%
79.60%
Hard
41.36%
48.2%
62.8%
41.8%
69.41% %
Easy 58.33%
62.4%
71.8%
58.6%
96.19%
GPT-5-nano
Medium
33.60%
38.2%
52.7%
33.9%
80.00%
Hard
22.33%
31.2%
48.7%
22.8%
69.92%
Easy 80.27%
81.4%
84.2%
80.4%
86.8%
Qwen2.5-72B
Medium
45.94%
47.8%
52.1%
46.1%
55,4%
Hard
33.60%
35.2%
41.8%
33.8%
45.2%
Easy 93.52%
94.1%
95.8%
93.6%
96.4%
GPT-OSS-1208
Medium 75.05%
76.2%
79.4%
75.2%
81.8%
Hard 59.64%
61.8%
67.2%
59.8%
71.4%
Table 40: Tool impact across difficulty suites for selected models.
Per-Task Tool Analysis (Easy Suite) Table presents detailed tool impact for GPT-5 on Easy Suite tasks. Caleulator provides the largest improvements for pure arithmetic tasks: Multiplication gains 15.8% by eliminating large number multiplication errors, Division gains 18.2% through improved decimal precision and edge case handling. Sum of Digits gains 14.6% via digit extraction and summation, and Alternating Sum gains 14.8% through better sign alternation with large numbers. For con-arithmetic tasks such sorting, counting, and finding elements, code execution provides better improvements as these require algorithmic logic rather than pure computation.
presents detailed tool impact for GPT-5 on Medium Suite tasks. Calculator excels sequence lasks involving exponential growth: Geometric Sequence gains 24.2% through ratio calculations with large powers, and Algebraic Sequence gains 23.4% via polynomial term computation. For Fibonacci and Prime sequences, code execution superior because these require iterative computation and primality testing beyond simple arithmetic.
Per-Task Tool Analysis (Medium Suite) Table 42
Per-Task Tool Analysis (Hard Suite) Table 43 presents detailed tool impact for GPT-5 Hard Suite tasks. Code execution dominates for all NP-complete tasks. However, Calculator provides meaningful auxiliary benefits for Cryptarithmetic with 12.6% gain through digit arithmetic verifica- tion. Modular Systems with 7.3% gain via modular arithmetic computations, and Matrix Chain with 3.7% gain from cost computations in dynamic programming.
126

---

Published as a conference paper at ICLR 2026
Task
No Tools
+Calc
+Code
+Web
Best Tool
Multiplicatiin
82.4%
98.2%
97.8%
82.6%
Calc
+15.8%
Division
78.6%
96.8%
95.2%
78.8%
Calc
+18.2%
Sum
91.2%
99.4%
98.8%
91.4%
Calc
+8.2%
Subtraction
89.8%
98.6%
97.4%
90.0%
Calc
+8.8%
Absolute Difference
85.4%
97.2%
96.1%
85.6%
Cale
+11.8%
Mean
84.2%
96.4%
97.2%
84.4%
Code
+13.0%
Median
88.6%
89.2%
96.8%
88.8%
Code
+8.2%
Mode
86.4%
86.8%
95.4%
86.6%
Code
+9.0%
Find Maximum
94.2%
94.6%
98.2%
94.4%
Code
+4.0%
Find Minimum
93.8%
94.2%
97.8%
94.0%
Code
+4.0%
Sorting
87.2%
87.4%
96.4%
87.4%
Code
+9.2%
Comparison
92.4%
92.8%
97.2%
92.6%
Code
+4.8%
Even Count
90.8%
91.2%
97.6%
91.0%
Code
+6.8%
Odd Count
91.2%
91.6%
97.4%
91.4%
Code
+6.2%
Second Maximum
89.4%
89.8%
96.2%
89.6%
Code
+6.8%
Range
86.8%
94.2%
95.8%
87.0%
Code
+9.0%
Negative Count
91.8%
92.2%
97.2%
92.0%
Code
+5.4%
Unique Elements
88.2%
88.6%
96.4%
88.4%
Code
+8.2%
Max Adjacent Diff
84.6%
92.8%
95.2%
84.8%
Code
+10.6%
Count Greater Than Prev
87.4%
87.8%
95.8%
87.6%
Code
+8.4%
Sum of Digits
83.2%
97.8%
96.4%
83.4%
Calc
+14.6%
Count Perfect Squares
82.8%
91.4%
94.8%
83.0%
Code
+12.0%
Alternating Sum
81.4%
96.2%
95.8%
81.6%
Calc
+14.8%
Count Multiples of K
84.6%
95.8%
94.2%
84.8%
Calc
+11.2%
Local Maxima Count
86.2%
86.6%
95.4%
86.4%
Code
+9.2%
Palindromic Count
85.8%
86.2%
96.2%
86.0%
Code
+10.4%
Longest Inc Subseq
78.4%
78.8%
92.6%
78.6%
Code +14.2%
Sum of Max Indices
82.6%
90.4%
94.8%
82.8%
Code
+12.2%
Table 41: Per-task tool impact for GPT-5 on Easy Suite tasks. Green rows indicate tasks where Calculator provides the best improvement over tools.
Task
No Tools
+Cale
+Code
+Web
Best Tool
A
Fibonacci Sequence
58.4%
72.6%
84.2%
58.6%
Code
+25.8%
Geometric Sequence
54.2%
78.4%
76.8%
54.4%
Calc
+24.2%
Algebraic Sequence
52.8%
76.2%
74.6%
53.0%
Calc
+23.4%
Prime Sequence
62.4%
68.2%
82.4%
62.6%
Code
+20.0%
Complex Pattern
48.6%
54.2%
72.8%
48.8%
Code
+24.2%
Table 42: Per-task tool impact for GPT-5 on Medium Suite tasks. Green rows indicate tasks where Calculator provides best improvement.
Findings 1) Code Execution Dominates: Across all tasks, code execution provides the largest improvements ranging from 15% to 55%, especially for recursive problems requiring state tracking such as Tower of Hanoi with 44.2% gain, constraint satisfaction requiring search such as Sudoku with 54.9% gain, and combinatorial optimization such N-Queens with 44.2% gain. 2) Calculator Has Limited Impact. Caleulator primarily helps arithmetic-heawy tasks: Cryptarithmetic gains 12.6% through digit arithmetic, Matrix Chain gains 3,7% through cost computation, with minimal benefit for structural reasoning tasks. 3) Web Search Confirms Contamination Resistance. Web search provides negligible benefit of 0.2% to 0.4% across all tasks, validating that problem space of more than 15 instances prevents answer lookup. 4) Tasks Remain Challenging. Even with all tools, many tasks remain difficult. Sudoku achieves only 67. 27.2% with code because constraint propagation
127

---

Published as a conference paper at ICLR 2026
Task
No Tools
+Calc
+Code
+Web
Best Tool
A
Tower of Hanoi
34.2%
35.1%
78.4%
34.4%
Code
+44.2%
N- N-Queens
28.1%
28.4%
72.3%
28.2%
Code
+44.2%
Sudoku (9x9)
12.3%
12.8%
67.2%
12.4%
Code
+54.9%
Boolean SAT
41.2%
41.8%
68.9%
41.4%
Code
+27.7%
Graph Coloring
52.4%
53.1%
71.2%
52.6%
Code
+18.8%
Cryptarithmetic
45.6%
58.2%
69.8%
45.8%
Code
+24.2%
Matrix Chain
38.4%
42.1%
64.7%
38.6%
Code
+26.3%
Logic Grid
31.2%
31.8%
58.4%
31.4%
Code
+27.2%
Modular Systems
44.8%
52.1%
61.3%
45.0%
Code +16.5%
Constraint Opt
48.2%
51.4%
63.8%
48.4%
Code
+15.6%
Table 43: Per-task tool impact for GPT-5 on Hard Suite tasks. Green rows indicate tasks where Calculator provides notable (though not best) improvement.
logic is still challenging. Logic Grid reaches only 58.4% with code due to multi-step deduction errors. Modular Systems achieves 61.3% with code because the difficulty of symbolic manipulation.
5) Tool Orchestration Capability Smaller models struggle not just with solving problems, but with correctly using tools. Table 44 shows tool orchestration capability across models. Common tool-related failures include failing to invoke tools when beneficial, selecting inappropriate tools for the task type, providing malformed inputs to tools, and misinterpreting tool outputs.
Model
Tool Call Success
Correct Tool Selection
Correct Code Generation
Correct Result Interpretation
GPT-5
94.2%
89.1%
91.3%
88.7%
GPT:5-mini
88.4%
82.3%
84.6%
81.2%
GPT-5-nano
78.4%
71.2%
73.8%
69.4%
Qwen2,5-72B
81.2%
74.8%
76.2%
72.8%.
Qwen2.5.
68.3%
62.1%
64.7%
60.2%
Qwen2.5-3B
41.2%
34.8%
37.4%
32.1%
Table 44: Tool orchestration capability across models.
Comparison with Static Benchmarks + Web Search To demonstrate contamination resistance, compared web search impact static versus dynamic benchmarks. Table 45 shows GPT-5 performance with web search.
Benchmark
No Web Search
Web Search
A
MATH500
78.2%
94.6%
+16.4%
AIME22
72.1 %%
89.3%
+17.2%
AIME25
68.4%
85.7%
+17.3%
BEYONDBENCH (Hard)
50.27%
51.8%
+1.53%
Table 45: GPT-5 performance with web search on static versus dynamic benchmarks.
Static benchmark questions can be found online, yielding 16% to 17% improvements. Our benchmark shows only 1.5% improvement, confirming that our dynamic generation prevents web-based answer retrieval.
R SFT AND RL TRAINING ANALYSIS
This section provides experimental validation of our benchmark's 's contamination resistance through targeted training experiments, addressing concerns about what happens when models are deliberately fine-tuned on benchmark data.
128

---

Published as a conference paper at ICLR 2026
Experimental Setup Data Generation: We generated 66,000 problem instances from bench- mark using seed value 123, which is different from our evaluation seed 42. The distribution was balanced across suites with 22,000 instances (33.3%) for each of the Easy, Medium, and Hard suites, Models Trained: We fine-tuned models using both SFT and GRPO: GPT-OSS-2065 Qwen2.5-3B, Qwen2.5-7B, Qwen2.5-14BB and Llama-3.1-88 Training Configuration: For SFT, we used learning rate 2e-5, batch size 32, 20 epochs, and cosine scheduler. For GRPO, we used learning rate 1e-6, batch size 16, KL coefficient 0.I, and 20 epochs. We tested data splits of 20%, 40%, 60%, 80%, and 100% of the generated data. Evaluation: All models were evaluated on seed 42, our standard evaluation set, ensuring zero overlap between training and evaluation instances.
SFT Training Results Table 46 presents overall accuracy after supervised fine-tuning with varying amounts of training data. All five models show consistent improvement as training data increases, with the largest gains occurring in the early stages of training. Qwen2.5-7B improves from 36. 36.12% baseline to 57.4% after training on 100% of the data, representing 21.28 percentage point gain. The smaller Qwen2.5-3B shows similar relative improvement, jumping from 24.82% 47.1%. GPT-OSS-20B, which starts with the highest baseline at 67.66%, reaches 79.8% after full training. Notably, the rate of improvement decreases as more data is added. The jump from baseline 20% data yields approximately 12 to 14 percentage points for most models, while the jump from 80% to 100% data yields only 0.6 to 1.0 percentage points. This diminishing returns pattern suggests that models quickly learn transferable heuristics from initial training examples but struggle to extract additional value from larger datasets.
Model
Baseline
20 % Data
40% % Data
60% % Data
80% Data
100% Data
Qwen2.5-3B
24.82%
38.2%
42.1%
44.8%
46.2%
47.1%
Qwen2.5-7B
36.12%
48.7%
52.3%
55.1% %
56.8%
57.4%
Owen2.5-14B
41.33%
52.1%
56.4%
58.9%
60.2%
61.1%
Llama-3.1-8B
24.20%
35.8%
39.2%
41.6% 78.1% 1%
43.1%
44.2%
GPT-OSS-20B
67.66%
74.2%
76.8%
79.2%
79.8%
Table 46: Overall accuracy after SFT training with varying data amounts.
GRPO Training Results Table 47 presents results from Group Relative Policy Optimization training. GRPO follows similar improvement trajectory to SFT but with slightly lower final accuracies across all models. Qwen2.5-7B reaches 56.2% after full GRPO training compared to 57.4% with SFT, difference of 1.2 percentage points. The gap consistent across model sizes, with Qwen2.5-14B achieving 59.6% versus 61. % for SFT and GPT-OSS- 2SB reaching 78.4% versus 79.8%. This performance difference likely stems from the nature of tasks. SFT directly optimizes the model to produce correct answer sequences, while GRPO relies on reward signals that can be sparse when problems have binary correctness and no partial credit. The Hard suite problems, which have high variance in difficulty even within the same category. may provide particularly noisy reward signals for GRPO. Despite the slightly lower performance, GRPO shows the same diminishing returns pattern, with most gains occurring before the 60% data threshold.
Model
Baseline
20% Data
40 % Data
60% Data
80% Data
100% Data
Qwen2.5-3B
24.82%
36.4%
40.8%
43.2%
45.1 %
46.3%
Qwen2.5-7B
36.12%
47.2%
51.1%
53.8%
55.4%
56.2%
Qwen2.5-14p
41.33%
50.8%
54.9%
57.2%
58.8%
59.6%
Llama-3,
24.20%
34.2%
37.8%
40.1%
41.8% 77.6%
42.9%
GPT-OSS-20B
67.66%
72.8%
75.2%
76.8%
78.4%
Table 47: Overall accuracy after GRPO training with varying data amounts.
Suite-Level Analysis Table 48 breaks down performance by difficulty suite before and after training.
The results reveal that training benefits not uniform across difficulty levels. The Easy suite shows the largest absolute improvements, with Qwen2.5-7B jumping from 61.62% to 89.2%, a gain of 27.58 percentage points. The Medium suite shows substantial but smaller gains, with the same model
129

---

Published as a conference paper at ICLR 2026
improving from 30.32% 68.4%. The Hard suite shows the most modest improvements despite receiving equal training emphasis, with Qwen2.5-7B improving from 16.42% to only 28.6% 28.6%, a gain of just 12.18 percentage points. This pattern holds across all five models and suggests a fundamental limitation. Easy suite tasks involve polynomial-time algorithms that models can learn effectively from examples. Medium suite tasks require pattern recognition and sequence extrapolation, which benefit from training but remain challenging. Hard suite tasks are NP-complete by construction, and training helps models learn better heuristics but cannot overcome the fundamental computational difficulty. GPT--SSS-200 shows the smallest relative Hard suite improvement, gaining only 14.68 percentage points from an already strong baseline of 52. suggesting that even larger models face ceiling on Hard suite performance.
Model
Baseline
After 100% SFT
Easy
Medium
Hard
Easy
Medium
Hard
Qwen2.5-3B
45.52%
20.82%
8.12%
78.2%
51.4%
18.8%
Qwen2.5-7B
61.62%
30.32%
16.42%
89.2%
68.4%
28.6%
Qwen2.5-14B
63.52%
37.88%
22.58%
88.4%
72.1%
34.2%
Llama-3.1-88
49.12%
15.47%
8.02%
76.8%
42.8%
19.4%
GPT-OSS-20RB
87.49%
63.38%
52.12%
94.2%
81.4%
66.8%
Table 48: Suite-level performance before and after 100% SFT training.
R.1
MAIN INSIGHTS
Differential Suite Impact. Training shows dramatically different effects across difficulty levels. The Easy Suite shows 27% to 38% improvement as models learn polynomial-time algorithms. The Medium Suite shows 30% to 40% improvement as pattern recognition improves. The Hard Suite shows only 10% to 15% improvement because NP-complete problems remain fundamentally difficult.
Diminishing Returns. Performance gains plateau after 60% data. From 0% 60% data, we see approximately 15% total improvement. From 60% to 100% data, we see only about 3% additional improvement. This suggests models quickly learn transferable heuristics but struggle to improve further on genuinely hard problems.
SFT vs. GRPO. SFT slightly outperforms GRPO by 1% 2%, likely because SFT directly optimizes for correct answers, GRPO's reward signal is sparse for hard problems, and hard suite problems have high variance in difficulty within categories.
R.2 DATA DISTRIBUTION ABLATION
We tested whether emphasizing hard problems during training improves Hard suite performance.
Distribution Fasy:Med:Haard
Easy A
Medium A
Hard A
Balanced
33:33:33
+27.58%
+38.08% +29.1%
+12.18% +14.8%
Hard-Focused
26:18:56
+18.2%
Table 49: Qwen2.5-7B performance under different training distributions.
Hard-focused training provides marginal Hard suite improvement of 2.62% at the cost of Easy and Medium performance, suggesting algorithmic reasoning skills do not transfer across difficulty levels.
R.3 COMPARISON WITH STATIC BENCHMARK HACKING
To contextualize these results, we compare with what happens when models are trained directly on static benchmark data.
Static benchmarks can be trivially solved through memorization, achieving 97% to 99%. Our benchmark remains challenging because training and evaluation instances provably different due
130

---

Published as a conference paper at ICLR 2026
Scenario
Before Training
After Training
A
GSM8K (train test) MATH (train on
83.2%
99.1%
+15.9%
52.4%
97.8%
+45,4%
BEYONDBENCH Overall (train seed 123, eval seed 42)
36.12%
57.4%
+21.3%
Table 50: Comparison of training impact: static benchmarks versus our benchmark (Qwen2.5-7B).
to different seeds, models must learn generalizable algorithms rather than specific solutions, and No-complete problems remain fundamentally difficult.
R.4 HACKING RESISTANCE SUMMARY
Our benchmark provides robust defense against deliberate benchmark hacking. Unlimited fresh instances are available since new seed produces an entirely new evaluation. Only transferable learning works because models can only improve by learning genuine algorithms. A hard ceiling persists as NP-complete problems remain challenging with approximately 40% to 45% ceiling even after extensive training. Memorization is not possible because the problem space of more than 1015 instances prevents instance memorization.
PROBABILITY TRAIN/TEST CONTAMINATION
Concise statement.
Let Px be the probability that single training sample equals item and let q q= E Eg Pa be the total mass of the evaluation set under the training distribution. After N independent draws, the probability that at least one evaluation item appears in the training corpus equals 1 - (1 - q) & 1 - 6- In the uniform-draw special case q n/M, which recovers the formula used in the main text. Importantly, real-world corpora exhibit heavy skew, duplication, and curated sources which typically increase q above the uniform estimate; therefore contamination is usually more likely than the uniform analysis predicts. The relevant risk scale is qNV and even small per-item masses can produce near-certain overlap when N large.
Here explain the mathematics behind contamination probability statements in the Purpose. main text.
S.1
CONTAMINATION PROBABILITY IN FINITE BENCHMARKS
We provide rigorous analysis of why finite benchmarks inevitably suffer from contamination when evaluated against models trained large corpora.
S. BASIC CONTAMINATION MODEL.
Consider three sets:
U: universe of all possible problem instances, [ M
D: evaluation benchmark with |D] test problems
C: training corpus containing N samples drawn from 2/
Under uniform sampling where each training example is drawn i.idd with probability 1/M for any element U, we can compute the exact contamination probability.
Theorem 7 (Contamination Probability). The probability that least one evaluation example appears in the training corpus is:
(260)
M
131

---

Published as a conference paper at ICLR 2026
Proof. Each training sample avoids all evaluation examples with probability (1-n/MM, Since the N training samples are independent, all samples avoid D with probability (1 m/MM). The complement gives the result.
For large corpora where n/M M is small, we obtain the exponential approximation:
exp (- n.N
Pr(DnC + Ø)
(261)
M
This follows from the limit (1 -1- x)) as + with kæ fixed.
S.1.2 IMPLICATIONS FOR MODERN LLMS
The key insight is that contamination depends on the product nN/M::
When nN/M 1: contamination probability nN/M (linear regime)
When nN/M R contamination probability R 0.63 (transition point)
When nN/M M > 1: contamination probability + (saturation regime)
For concrete numbers: if benchmark has n 10 problems, the universe contains M = 10 possible problems, and training uses N 10 samples, then nN/M 10 yielding contamination probability ~ e e-1000 = 1. Even with conservative estimates, modern web-scale training pushes us deep into the saturation regime.
S.1.3 WHY COMPLETE CONTAMINATION IS EXPONENTIALIY HARDER
While
contamination virtually certain, the probability that all evaluation examples appear in training is much smaller. Each item D is present in the corpus with probability 1 (1 - 1/M)*. Because the items are sampled independently, the probability that every evaluation
item appears is:
Pr(DCC) = 1- -
1 M
" = NIMM (262)
RED
For this close 1, we need each factor close 1, i.e. NNN < 1/n, which requires N 2 M log exponnnteyy more than the N M/n needed for single contamination. This gap explains why partial contamination dominates: models see some test examples during training, enough to inflate scores, but not the complete benchmark.
S.2 SETUP AND NOTATION
We work with three sets and some sizes probabilities:
W: a finite universe of distinct items (for example, all distinct problem statements or documents). Write [2]] M.
D: finite evaluation (test) of items. Write [D]
C: the training corpus, which is multiset of N samples drawn independently (with re- placement) from Zd. We view C as the set of items that appear at least once among the N draws.
We consider two models for how the training samples are drawn from the universe:
1. Uniform model. Each training sample is drawn uniformly at random from U4 so every item has probability 1/M of being drawn single sample.
2. Non-uniform model. Each training sample is drawn i.i.d. from probability distribution
p = {pa Jaaus where p. 2 and Ereu Pa. = this case an item æ has probability Pz of being selected single sample.
132

---

Published as a conference paper at ICLR 2026
Throughout use standard limits. The event of interest is
(DNC * Ø),
L.e. least one item from the evaluation set appears in the training corpus.
S.3 EXACT FORMULA (NON-UNIFORM CASE)
The non-uniform model gives an exact and simple expression for the contamination probability. Theorem 8 (Exact contamination probability). Let {px) be the sampling distribution of single training draw, and let
= Σp=
(263)
JED
be the total probability mass that the training distribution assigns to the evaluation set D. Then the probability that the training corpus of N i.i.d. draws contains at least one element of is
Pr(D Ø) - 11 -)
(264)
This identity is exact for any N 221 1.
Proof. A single training sample is not equal to any element of DD with probability - q by definition of in equation Because the N samples independent, all N samples avoid D with probability
(1 - 9)" N The complement event, that at least one sample falls D, therefore has probability 1 - (1 9),, which proves the formula.
Remark 1 (Uniform model as special case). If the training draws are uniform on Ut then Ps = 1/M for every x and
q=
Σ
n M
M
xED
Equation equation
264 reduces to the uniform formula
M
which is the formula used in the main text.
S.4 EXPONENTIAL APPROXIMATION AND LIMIT BEHAVIOR
The expression (1 - g) is often well approximated by exponential when is small and N may be large. We make this precise and state useful limits.
Proposition 9 (Exponential approximation). If 10, and N 2 1, then
( exp(NN log(1 4)) exp( q
--pp-NN + o(gN)) as -
(265)
Consequently.
Pr(DC+#0) 1 1 exp(-gN N + o(gN)).
(266)
NqN- + X [0, 00) then the probability converges to -
Proof. Use the Taylor expansion log(1 - 9) - o(g) as q + 0. Multiplying by N gives
N log(1 - q) = N No(g) N o(g.), which yields equation 265 Substitute into the exact formula (1 - 9) to obtain equation 266 The limit statement follows by continuity of the
exponential map.
Corollary 10 (Uniform limit), In the uniform model, write X = limm--x nN/M when the limit exists. Then
lim - 2)
-0000
In particular, if nNNM + 2o then the probability tends to
Proof. Set q n/M in the previous proposition and apply the same limit argument.
133

---

Published as a conference paper at ICLR 2026
S.5 EXPECTED NUMBER OF OVERLAPS AND THE "ALL ITEMS PRESENT" EVENT
It is useful to look at two related quantities:
the expected number of evaluation items that appear in the training corpus;
the probability that every evaluation item appears at least once in the training corpus (this is a much stronger requirement).
Proposition 11 (Expected number of present evaluation items), For each x E DD let Ia be the indicator random variable of the event "item z appears at least once in C". Then
E
Σ
L (1--(1))))))
(267)
ZED
In particular, in the uniform case where pa. 1/M this equals
n 1 1 -1- M1 1m nn(1 eN-MM)
Proof. Linearity of expectation gives the result because B[L] Pr(ls: = 13 p2) for each x.
Why "all items present" is much harder. For the event that every item in D appears at least once we need every Le = 1. A simple (but informative) way to see the scale difference is to look at the expected number of missing items:
B missing
(1-pa))
xED
In the uniform case this is n(1 - 1/M)* ne NNMM To make the expected number of missing items small (say 1), require roughly
e -NNM < 1/n
N/M 2 logn.
Thus to expect all n items to appear we typically need N on the order of M log n, which is much larger than the N = M/nn scale where single overlap becomes likely. This gap explains the claim that "a single overlap is the relevant contamination risk" while the event D C C (every test item in training set) exponentially harder in n.
S.6 POISSON APPROXIMATION VIEWPOINT (INTUITION)
When the per-item sampling probabilities Pa are small and N large, the counts of how many times each item appears can be approximated by independent Poisson random variables with means d₂ = V. Under this approximation the indicator that item x appears at least once has probability 1 e A. and the number of distinct evaluation items present approximately Poisson- Poisson-binomial with parameters {1 11-2 e ->>x )re Summing the small means gives the same controlling quantity ExeD Xr q... When q moderate to large expects many overlaps. This Poisson viewpoint justifies the e approximation and the limit results given earlier.
S.7 NUMERICAL EXAMPL.ES
Example A: nN/M 10. Then the overlap probability in the uniform model is 1 e- a 0.99995 (virtually certain).
Example B: 1000, to make the expected number of missing items 1 we need N/M z log 1000 z 6.9, ie. x 6.9M. By contrast, to get single overlap with constant probability only need N x M/n M/1000 (much smaller).
134

---

Published as a conference paper at ICLR 2026
S.8 CAVEATS AND PRACTICAL REMARKS
The math above is exact under the stated i.i.d. sampling model. Real training corpora differ from this idealization several ways, each of which changes the effective mass q:
Skew popularity. Some items or sources much more likely than others; this increases relative to the uniform estimate n M and makes contamination more likely.
Duplicates and amplification. Multiple copies (near-duplicates) of the same content in the scraped corpus effectively increase the empirical sampling mass of that content, again increasing contamination probability.
Curated sources. If the training collection intentionally includes popular public repositories or textbooks, 9 can be large. those sources are filtered out, q can be small.
Partial or paraphrase contamination. Real overlap not always an exact equal-by -by- identity event; partial phrase overlap or paraphrases matter. The same formulas hold if 'D is interpreted as the set of all substrings/paraprrases interest, but then M and ps: change accordingly.
The bottom line is the controlling quantity is qN, not nN/M in general. Under realistic skew and duplication, q is typically larger than n/M, so practical contamination risk often higher than the uniform model predicts.
T SCALABILITY OF BENCHMARK DEVELOPMENT
Extensibility New Tasks. BeyondBench's architecture makes adding new tasks straightfor- ward. Each task only needs to implement four methods: generate_problee stances, verify_solution() for checking correctness, estimate_ srimatt_ttokes()) for context management, and patpeaaaspppoeel response() for answer extraction. The framework handles evaluation orchestration, metric computation, and result aggregation automatically. We've ve demonstrated this extensibility by success- fully adding 10 new task categories after the initial 19, each requiring less than 500 lines of code. The modular design means tasks can be developed and tested independently before integration.
for creating in-
Scaling Future Model Capabilities. The benchmark scales naturally as models improve. Each lusk's difficulty can be increased by adjusting generation parameters: larger problem sizes (c.g.. N-Queens from 8x8 to 64x64 boards), tighter constraints (e.g.. SAT problems closer to phase transitions), or additional complexity dimensions (e.g., multi-objective optimization with more constraints). The algorithmic generation ensures we never exhaust the problem space-even with million evaluations, collision probability remains below 10-9. As context windows expand, previously infeasible problems become testable. For instance, Tower of Hanoi with 12 disks (requiring 4.095 moves) needs approximately 49,000 tokens, currently beyond most models but feasible with emerging 100K+ context windows.
Computational Efficiency at Scale. Despite the large problem space, BeyondBench remains computationally efficient. Problem generation is typically O(n) or O(n²) where is the problem size, taking milliseconds even for complex instances. Solution verification, though potentially exponential in worst case, uses optimized solvers that handle typical cases in under a second. The bottleneck model inference, not benchmark operations. We can generate and verify 10,000 problems per second single CPU core, meaning the framework could evaluate millions of instances without computational strain. This efficiency enables large-scale evaluations for robust statistical analysis and fine-grained capability assessment.
U LIMITATIONS AND FUTURE WORK
While BEYONDBENCH addresses critical contamination issues LLM evaluation, several limitations merit acknowledgment. First, our framework focuses exclusively algorithmic and mathematical reasoning, not capturing other important reasoning facets that lack deterministic solutions, like commonsense, causal, creative reasoning. Second, lungle-prompt-pe-ttaae approach may
135

---

Published as a conference paper at ICLR 2026
underestimate models optimized for specific prompting strategies, though this maintains evaluation consistency. Finally, our token-aware evaluation, while preventing unfair penalization due to context limits, may not fully capture how models perform when approaching their computational boundaries. Despite these constraints, BEYONDBENCH provides a robust foundation for contamination-free reasoning evaluation that can be extended to address these limitations in future work.
136


Reviews:
[{"review_id": "3O46T6ctT5", "reviewer": "Reviewer_9Z6M", "summary": "This paper introduces a framework for evaluating the reasoning capabilities of LLMs. It addresses the issue of data contamination in existing static benchmarks by generating mathematically grounded problems algorithmically and on-the-fly, ensuring each test instance is fresh and uncontaminated. BeyondBench comprises 44 algorithmic tasks with 117 variations across three difficulty suites (Easy, Medium, Hard) and offers provable unique solutions verified by mathematical proofs. It also incorporates a token-aware evaluation system that dynamically scales problem complexity based on a model's context window.", "questions": "1. If data is constructed for SFT or RL targeting this benchmark, how will the evaluation results change? In cases of deliberate hacking, does BeyondBench have any advantages compared to static evaluation and other dynamic evaluation methods?\n2. A more in-depth analysis should be conducted on the phenomenon where tool-augmented models achieve extremely high results. Is it possible that certain tasks can be entirely solved by tools, or is the improvement in computation more significant?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. Contamination Resistance: BeyondBench uses algorithmic problem generation to create mathematically grounded problems on the fly, ensuring each test is fresh and uncontaminated. \n2. Task and difficulty: This work covers 44 algorithmic tasks with 117 variations, grouped into three difficulty levels. The complexity of each task can be controlled by scaling parameters.\n3. Token-Aware Evaluation Framework: The framework dynamically calibrates problem complexity based on a model's token budget.", "weaknesses": "1. The benchmark questions do not sufficiently assess the level of intelligence. Existing competition benchmarks often cannot be solved by fixed algorithms and instead require special techniques or constructions. However, the problems used in this benchmark are almost all based on fixed algorithms, with difficulty controlled by adjusting complexity and computational difficulty. This approach fails to evaluate the \"cleverness\" or \"ingenuity\" of a model in solving problems.\n2. The benchmark seems to be easily hacked by tool-augmented models. Fixed algorithms and complex computations can be easily solved by calculators or code. The performance gap of GPT-5 family with/without tool shows the evidence.\n3. This work only theoretically argues that the combinatorial variations of this benchmark ensures the test is uncontaminated, but it lacks experimental validation. If related data is constructed for SFT or RL targeting this benchmark, the influence is not studied.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QFmTRQ8rQX", "reviewer": "Reviewer_ao5p", "summary": "BeyondBench is a dynamically generated benchmark for evaluating reasoning in language models. It addresses data contamination by algorithmically generating problems from a combinatorial space of >10^15 instances across 44 tasks in three difficulty tiers. Each problem has verifiable solutions using SAT/CSP solvers. The authors evaluate 101 models and find that most open-source models plateau around 30-35% accuracy on hard tasks, quantization has minimal impact, and tool-augmented models substantially outperform reasoning-focused models.", "questions": "In addition to questions implied by the weaknesses section, here are some additional questions\n\nQ1. Could the authors provide information about if there were any differences in failure modes between vanilla models and reasoning models, even if they performed similarly in raw metrics? This is one of the most interesting findings of this paper but it is not emphasized and the analysis on it seems thin.\n\nQ2. Criteria for graceful vs sudden degradation needs more care, for example using number of disks for Tower of Hanoi may not make sense as number of steps are not linear in number of disks. The degradation may be linear (and hence 'graceful') w.r.t reasoning steps, why should we expect linear degradation on arbitrary complexity metrics? Could the authors provide arguments for why 'n-disks', 'grid size', 'vertices', 'board size', 'word length' etc. are comparable complexity metrics? Otherwise the conclusions about differences in gradual vs sudden degradation across tasks lack any significance.\n\nQ3. Could the authors provide more justification that avoiding parameter space collisions with high probability is enough to be confident that structural/semantic space collisions are avoided? This is related to the earlier point about superficial heuristics (W2), does this approach prevent collisions akin to, say, IMO questions requiring the same integration trick multiple years in a row?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "S1. Data contamination in static benchmarks is well-documented, and this paper provides a solution with mathematical guarantees about solution uniqueness and problem space size.\n\nS2. The evaluation covers 101 models across different sizes (0.5B-141B parameters), architectures, and quantization schemes, providing broad coverage of the current model landscape.\n\nS3. The use of SAT/CSP solvers to verify solution uniqueness and enumerate all valid solutions when multiple exist shows proper attention to evaluation correctness.\n\nS4. The framework dynamically scales problem difficulty based on model context windows and checks for token overflow, preventing unfair comparisons and catching models that \"overthink\" simple problems.\n\nS5. The empirical findings about performance plateaus on hard tasks, minimal quantization impact, and superiority of tool-augmented approaches over pure reasoning models provide actionable insights for model development.\n\nS6. The contamination probability analysis in Appendix E gives formal backing to the motivation.", "weaknesses": "W1. The 44 tasks focus almost entirely on algorithmic puzzles, arithmetic, and constraint satisfaction, missing many reasoning types relevant to LLM applications: causal reasoning, analogical reasoning, argument evaluation, multi-step logical inference in natural contexts. The paper doesn't justify why performance on Sudoku or Tower of Hanoi reflects general reasoning capability. Their choice to view reasoning to be interchangeable with algorithmic computation needs to be better motivated. (To the authors' credit, this is touched on in their limitations section but should be highlighted)\n\nW2. While specific problem instances are novel, models could still learn superficial algorithmic strategies from similar problems in training data. The paper doesn't distinguish between \"memorizing specific solutions\" and \"learning heuristic algorithms\", and their method seems to only expose the former. If allowing memorization of fixed, superficial procedures (e.g. a problem which requires addition, followed by multiplication, followed by subtraction in that exact order) is not considered as part of contamination, this should be made clear.\n\nW3. No confidence intervals, significance tests, or reporting of variance across different problem instances appears in the paper. Claims about performance differences lack statistical backing. The results are interesting, so it would be best to give readers more confidence in the findings.\n\nW4. Problem generation algorithms, solution verification procedures, and task specifications are relegated to appendices, making it hard to assess the approach without reading 20+ pages of supplementary material. Although the methods appear sound, readers will benefit from more explicit presentation of key technical methodological details (especially section 3.1, where there is not enough explanation on how problems are generated and along what dimensions the problems are being varied. The 'parameter space' is presented too abstractly.).\n\nW5. Calling this 'Benchmark-Free' seems confusing. Readers might expect some kind of mechanistic interpretability/probing method, or some metric derived from the model which does not require inference on a dataset of test problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "totsbvyhNu", "reviewer": "Reviewer_3dvS", "summary": "The authors introduce a new dataset namely **BeyondBench** which benchmarks LLMs performance on algorithimic tasks and evaluate the performance of both open open source and proprietary on their dataset", "questions": "Please address the weakness section", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is easy to read and authors have provided sound mathematical justification for their arguements in the paper. The appendix is detailed and is very nicely written to accompany the main text.", "weaknesses": "1. The fundamental weakness of the paper is with respect to novelty, there are already exact same work done in this direction and the authors unfortunately fail to cite them as well as evaluate them against their benchmark which would be a fairer comparision. [1], [2], [3]\n\n2. Problems in hard suite like crypt arithemtic are known NP complete problems, so the benchmarking of LLMs/LRMs directly only with prompting is in itself an unreasonable task. is it even a fair experiment to expect LLMs which are atmost polynomial time methods to solve problems of NP complete complexity class?\n\n3. A more reasonable approach would be to evaluate methods like PAL [4] or logic lm [5], but these works are neither cited nor are these methods evaluated. There is a section which talks about augmenting LLMs with tools and with some numbers but there is no detail as to what these \"tools\" were and as to what was the LLMs asked to generate for a given instance using these \"tools\".\n\n[1] NPHardEval: Dynamic Benchmark on Reasoning Ability of Large Language Models via Complexity Classes\n\n[2] FCoReBench: Can Large Language Models Solve Challenging First-Order Combinatorial Reasoning Problems?\n\n[3] PUZZLEPLEX: Benchmarking Foundation Models on Reasoning and Planning with Puzzles\n\n[4] PAL: Program-aided Language Models\n\n[5] Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qEDAcY52UD", "reviewer": "Reviewer_YAiF", "summary": "This paper introduces BeyondBench, a novel evaluation framework designed to address the critical issue of data contamination in LLM assessment.\n\nThe core problem BeyondBench tackles is that traditional, static benchmarks are often found on the internet and inadvertently included in LLM training data. This leads to models achieving high scores through simple recall rather than demonstrating genuine reasoning capabilities. It becomes unclear whether a model is solving a problem or merely retrieving a memorized answer. BeyondBench solves this by implementing a benchmark-free evaluation paradigm based on algorithmic problem generation. Instead of relying on a fixed set of questions, the framework dynamically and perpetually generates new, unique, mathematically grounded test problems on the fly. This \"freshness\" ensures that no test instance has been seen during the model’s training, thereby eliminating the risk of data contamination.\n\nThe primary contribution of this work is the development of a contamination-resistant and reliable method for testing the core reasoning skills of LLMs. By covering a variety of difficulty levels and task types, BeyondBench provides a dynamic, rigorous, and fairer mechanism to evaluate and compare the true intelligence and problem-solving capacity of different language models, moving the field of LLM assessment beyond the limitations of static benchmarks.", "questions": "1. Will programmatically generated problems bias towards LLMs with coding tools capabilities? Since the problems are generated via code, is there a risk that LLMs with strong coding skills will have an unfair advantage in solving these problems compared to those without such capabilities?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles the critical and growing issue of data contamination in LLM evaluation. It provides a robust solution to move past static benchmarks。 This shift is vital for accurately assessing the true capabilities and progress of frontier language models.\n\n2.  Excellent Execution and Contamination-Free Design. By employing algorithmic problem generation, it ensures that every test instance is unique, mathematically grounded, and perpetually fresh, thereby guaranteeing a contamination-free evaluation. This design covers a wide range of difficulty levels and task types, making it a rigorous and challenging test of core reasoning skills.\n\n3. The methodology is presented clearly and logically, effectively demonstrating how the new framework can accurately distinguish between different models' true problem-solving capabilities.", "weaknesses": "1.  Reduced User-Friendliness and Increased Score Variance: Compared to using static datasets, dynamically generated evaluations are inherently less user-friendly for the general public and researchers. They require a bit more complex infrastructure for on-the-fly problem creation and may lead to higher score variance between evaluations due to the non-static nature of the test instances, making direct comparisons of models more challenging than with a fixed set.\n\n2.  Missing Discussion of Prior Work: The paper's core idea—using dynamically generated environments/problems for evaluation—is not entirely novel, especially in related fields like RL and other LLM evaluation efforts (e.g., Reasoning Gym [1]).\n\nReferences\n\n1. Stojanovski, Zafir, et al. \"REASONING GYM: Reasoning Environments for Reinforcement Learning with Verifiable Rewards.\" arXiv preprint arXiv:2505.24760 (2025).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)