Creativity or Brute Force? Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models
Authors: Simeng Han
Howard Dai
Stephen Xia
Grant Zhang
Chen Liu
Lichang Chen
Hoang H Nguyen
Hongyuan Mei
Jiayuan Mao
R. Thomas McCoy
Venue: NeurIPS 2025 poster ()
Keywords: Brainteaser; Reasoning LLMs
Primary area: deep_learning

Abstract: Accuracy remains a standard metric for evaluating AI systems, but it offers limited insight into how models arrive at their solutions. 
In this work, we introduce a benchmark based on brainteasers written in long narrative form to probe more deeply into the types of reasoning strategies that models use. Brainteasers are well-suited for this goal because they can be solved with multiple approaches, such as a few-step solution that uses a creative insight or a longer solution that uses more brute force. 
We investigate large language models (LLMs) across multiple layers of reasoning, focusing not only on correctness but also on the quality and creativity of their solutions. 
We investigate many aspects of the reasoning process: (1) semantic parsing of the brainteasers into precise mathematical competition style formats; (2) self-correcting solutions based on gold solutions; (3) producing step-by-step sketches of solutions; and (4) making use of hints.
We find that LLMs are in many cases able to find creative, insightful solutions to brainteasers, suggesting that they capture some of the capacities needed to solve novel problems in creative ways. Nonetheless, there also remain situations where they rely on brute force despite the availability of more efficient, creative solutions, highlighting a potential direction for improvement in the reasoning abilities of LLMs.

Creativity or Brute Force?
Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models
Simeng Han
Howard D
Stephen Xia 10
Grant Zhang
Chen Liu 10
Lichang Chen"
Hoang Huy Nguyen 3
Hongyuan Mei
Jiayuan Mao
R. Thomas McCoy!
Yale University
Meta Superintelligence Labs
Georgia Institute of Technology
'TTIC
"Massacuusetts Institute of Technology
Lead contributors.
Please direct correspondence to sophiahsm6@gmail.c com
Abstract
Accuracy remains standard metric for evaluating AI systems, but it offers limited insight into how models arrive at their solutions. In this work, we introduce a benchmark based on brainteasers written in long narrative form to probe more deeply into the types of reasoning strategies that models employ. Brainteasers are well-suited for this goal because they can be solved with multiple approaches, such as a few-step solution that uses creative insight or longer solution that uses more brute force. We investigate large language models (LLMs) across multiple layers of reasoning, focusing not only on correctness but also on the quality and creativity of their solutions. We investigate many aspects of the reasoning process: (I) semantic parsing brainteasers into precise mathematical competition formats; (2) self-correcting solutions based on ground-truit solutions; (3) producing step-by- step sketches of solutions; and (4) making use of hints. We find that LLMs are many cases able to find creative, insightful solutions to brainteasers, suggesting that they capture some of the capacities needed to solve novel problems in creative ways. Nonetheless, there also remain situations where they rely brute force, despite the availability of more efficient, creative solutions, highlighting a potential direction for improving LLM reasoning.
1 Introduction
Recent techniques for improving large language models (LLMs) have highlighted an important fact: In order to improve AI systems, we should think carefully about how they arrive their solutions. For instance, many of the most popular recent enhancements to LLMs are approaches for guiding the models solve problems in certain ways, such decomposing tasks hierarchically into simpler sub-tasks, leveraging an internal monologue (commonly known chain of thought) to perform self- reflection and verify generated solutions [ 2 3. or translating problems from verbal description into formal representations before attempting solutions 4 5). One reason why the solution strategy is so important that some strategies may be more generalizable than others. For instance, brute-force strategies might generalize less well to complex, novel problems than the ability to form creative insights. Therefore, understanding how a model arrives at its solutions helpful for understanding the generality of the model's abilities.
The code is available at httpre///gataatcccc//cccaaaaaate
The datasel available al http:////hhggggggaccceeooooeeeee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Problem: Six Village (from the BRAINGLE BRAINTFASER Math subset)
There six villages along the coast of the only perfectly round island in the known universe. The villages are evenly distributed along the coastline so that the distance between any two neighboring coastal villages is always the same. There is an absolutely straight path through the jungle connecting every pair of villages. These paths create thirteen crossings in the interior of the island, one of which is in the middle of the island where paths from every village meet. The island has a strange courtship custom. Before a father will give permission for his daughter to marry, her suitor must bring the father a fish each day until he has traveled by every route from his village to the father's 's village. The young man only travels along routes where he is always getting closer to his destination. The young man may visit other villages along the way.
On April 1st a father's three sons come tell him of their intent to woo bride, each from a different village. The brides' villages are the first three villages encountered when traveling clockwise around the island. the sons begin their courtship today and the couples are married on the day following each son's last trip, what are the three wedding dates?
Despite the importance of solution strategy, there has been little work on evaluating the types of heuristics and solution components that LLMs use. Most LLM evaluations use benchmarks based on which provide important insight into how well LLMs perform on particular tasks but do not illuminate how they approach those tasks. A few prior papers have taken a finer gaainer look at the reasoning process by evaluating intermediate steps to more faithfully measure reasoning capabilities 9 10
accuracy 6 7 g
The contrast between brute- and insightful strategies is evident in where the "intelligence" is applied. A brute-force or exhaustive approach leverages the model's I's speed to try many possibilities, which can be useful for combinatorial problems or when the solution can be verified easily. An insightful approach requires the model to recognize a clever trick, pattern, or underlying principle that leads directly to the answer, mimicking human insight. In current LLMs, often a mixture of both. For instance, an LLM might attempt a quick enumeration of options if a direct reasoning trick doesn't it immediately present itself.
In this work, we perform a systematic analysis of LLM reasoning strategies through a novel benchmark dataset that we introduce, BRAINGLE BRAINTEASER, which uses brainteasers to evaluate the reasoning abilities of LLMs. The types of brainteasers that we use are effective for this purpose because solving them involves multiple aspects of reasoning (e.g., translating language into mathematical abstractions and finding creative insights) and because they can be solved in multiple different ways (e.g., creative, insight-based approaches or longer brute-force approaches). Therefore, these brainteasers are well-suited for illuminating the types of reasoning processes that LLMs tend to use and the aspects of reasoning that they perform well on or struggle with. BRAINGLE BRAINTEASER is focused on evaluating mathematical and logical reasoning with minimal knowledge barriers, distinguishing AIME [22 by emphasizing sequential problem-solving over knowledge recall. Authored by expert problem solvers, BRAINGLE BRAINTEASER features diverse puzzle styles and complexities, aiming to isolate models' reasoning abilities rather than their memorization of formulas. BRAINGLE BRAINTEASER is exclusively centered on mathematical and logical puzzles, all authored by expert problem solvers with demonstrated proficiency across a wide range of puzzle types. Consequently, the benchmark exhibits greater diversity in both problem style and complexity than previous logical puzzle benchmarks 113 [5). Using this benchmark, we systematically evaluate L.M reasoning performance by dccomposing reasoning into distinct sub-tasks: translating narrative (verbal) descriptions into mathematical descriptions, generating solutions directly from both verbal and mathematical representations, verifying solutions against ground truths, generating high-level solution plans, summarizing detailed solutions into conceptual outlines, and finally, generating solutions given such high-level plans.
from existing datasets like MATH 7 and
Our key contributions are twofold. that allows for systematic probing of many aspects of the reasoning pipeline. Second, our empirical analysis with OpenAl o3 [66.. DeepSeek R1, DeepSeek R1 Distilled models [17., DeepSeek V3 [8] and Gemini Flash [19 provides clear evidence regarding heuristics that are commonly used in reasoning research. We find that: 1) LLMs sometimes default to brute force in cases where more
First, we present BRAINGLE BRAINTEASER. structured dataset

---

creative solutions are available. This could be because the training objective is aimed at optimizing the final answer accuracy instead of optimizing for generating more creative solutions or the correctness of the reasoning trace [17]. 2) LLMS struggle to correct solutions based on official solutions when prompted to do so. 3) Translating from verbal narratives into mathematical-style problem statements provides modest gains in performance. 4) The strongest reasoning models are able to reliably break down the solutions into insightful steps and models are capable of using high-level steps to generate correct solutions for the hardest problems where they fail to generate solutions fully from scratch.
Overall, our findings challenge widely held heuristic assumptions. such as the presumed simplicity of task decomposition, verification, and formal translation. Therefore, future research on reasoning models should evaluate these assumptions and shift from heuristic-driven algorithms towards systematic compositions and improvements of LLM capabilities.
2 Related Works
LLM Reasoning Benchmarks Existing mathematical reasoning benchmarks span wide spectrum of difficulty, from grade-school arithmetic [7] to Math Olympiad-level problems 220 21 22] and
provides formal-prod benchmark at the Olympiad level and the AlphaGeometry suite focuses specifically on IMO-level geometry challenges (25 26]. In contrast, benchmark, BRAINGLE BRAINTEASER, is challenging (like prior benchmarks focusing on Olympiad problems or collegiate contests) but without requiring the substantial domain knowledge that is demanded in prior challenging benchmarks. It achieves its difficulty by emphasizing the generation of creative insights, isolating an LLM's capacity for novel solution generation rather than its access to sophisticated mathematical facts. By posing concrete problems, rather than the sometimes highly abstract statements typical of Olympiads and collegiate contests, BRAINGLE BRAINTEASER makes it easier both for models verify long or complex answer strings and for evaluators to assess correctness. Distinct from recently introduced human- written BRAINGLE BRAINTEASER is specifically focused on mathematical and logical puzzles, and written by human experts proficient in solving diverse types of puzzles. They are also diverse in problem style as well as problem complexity. Finally, our results show that BRAINGLEE. BRAINTFASER is non-saturated, which suggests that it is not overly prone to creating the opportunity to identify both strengths and weaknesses of
collegiate contests (e.g., the Putnam competition) 233 24
or synthetic puzzle benchmarks [ 14 13),
data contamination (2 28
I.LM reasoning.
Reasoning Evaluation Beyond Final Answer Accuracy Recent research has underscored the need to evaluate an LLM'' reasoning process srelf-rather than focusing solely on final-answer accuracy- by examining intermediate steps to more faithfully measure reasoning capabilities p 29 30 In this work, rather than directly evaluating intermediate steps, we systematically explore four facets that are closely related to the capabilities of LLM reasoning such as self-correction of generated solutions based on official solutions and strategic use of hints to guide problem solving.
Reasoning Creativity Previous studies have explored enhancing LLM creativity in solving mathematical conjectures by leveraging programmatic approaches and formal self-refinement within symbolic domains [31 (3]. In contrast, our work evaluates the creativity manifested in LLM- generated natural language explanations, assessing whether they introduce novel insights that effectively guide and simplify the problem-solving process.
3
The BRAINGLE BRAINTEASER Benchmark
In this section we provide qualitative and quantitative descriptions of the BRAINGLE BRAINTEASER benchmark, which is made of two sub-datasets: a Math dataset and Logic dataset.
3.1 Dataset Construction
The Math and Logic datasets were curated by scraping problem-solving and reasoning questions from the Braingle website, an online platform of puzzles and brain teasers. The problems on Braingle largely curated by human experts proficient in solving diverse types of logical puzzles. We have obtained permission from the website owners to distribute the specific examples used in this study. To systematically extract the data, we implemented a web scraper that crawls through all available puzzle pages and recorded the question-answer pairs. We specifically select the math and logic subsets in
3

---

Table 1: Puzzle counts by category for Math and Logic datasets. Popularity and Difficulty are both on a scale from I (least popular/difficult) to 4 (most popular/iificult), (a) Puzzle count by category in the Math set.
(b) Puzzle count by category in the Logic set.
Category
Count
Popularity
Difficully
Category
Populurity
Difficulty
Simple/arge
Standard
OD
29
2:53
0.23
2.68 + 0.23
Geometry
24
2.19
2.87 0.19
ID 2D
13
2:56
0.30
2.60
1:0+100 0.28
22
2.47
0.30
2.72
0.25
Nomber Theory
2.35 0.22
2.11
0.15
Number
17
2,43
0.27
2.74
0.26
Combinatorics
0.11
2.76 0.10
Clusters
8
2.56
0.20
2.78
0.13
Tree
6
2.83
0.17
2.69
1 0.15
Algebra
2.37 + 0.22
2.75 0.12
Complex/small
Nonstandard
2.66
Communication Compound
2.55
0.15
Logic
29
2.35
+ 0.23
2.77 0.14
2.67
Special Number
29
2.35 + 0.22
2.78 0.13
Math-like Algurilim Mälh
38 32
2.56
0.34
2.66 0.23
Heuristic
2.41
0.27
2.63 0.22
Pattern
28
2.28
1. 0.22
2,82 1: 0.15
Hearistic
Arithmetic
2.44
1: 0.24
2,80 1:: 0.19
Pattern
26
2.48
0.27
2.63 0.22
Linguistic
1.5
2.50
0.12
2.61 1 0.15
Total
242
2.33 +: 0.22
2,80
0.15
Total
236
2.51
0.28
2.66 0.23
Braingle which are designed as logical and mathematical reasoning challenges. We also collected associated metadata such titles, user-voted difficulty ratings, popularity scores, and optional hints, creating a rich and structured dataset for analysis. In our datasets, we select the top 250 most difficult problems in the math and logic categories for our Math and Logic datasets respectively, these represent the most challenging problems in each category.
As quality control, we conducted one round of manual inspection of all problems in BRAINGLE BRAINTEASER done by college students who belong to a math club; these students have extensive experience in solving competition-levé math problems. During manual inspection, low-quality umbiguously-deseribber problems were removed, leaving 242 Math and 236 Logic problems in the dataset. The same annotators also manually created hints for problems that originally lacked them.
3.2 Detailed Dataset Information
Population Statistics Table D provides statistics of BRAINGLE BRAINTEASER. The most difficult problem has user-voted difficulty score of 3.06. All problems have difficulty level above 2.30 as we only consider the most challenging problems in our dataset. The average number of words in human solutions is greater in Logic (172 words) than in Math (237 words).
Problem Categorization We divide the Math top 250 dataset into 3 categories and 8 subcategories shown in Table Ia In general, the Standard category describes problems which can be written as standard math test or competitive math problem forms albeit requiring less knowledge. The Nonstandard category describes problems which still have rigorous answer, but do not the form of standard competitive math problem. The Heuristic category describes problems which are not mathematically rigorous, but require creative pattern-finding or heuristic thinking. We divide the Logic dataset into 4 categories and 13 subcategories shown in Table lb] In general, the Simple/large category describes large deductive logic puzzles, which often contain long chains of reasoning. The Complex/small category also describes deductive logic puzzles, often with fewer possible answers, but with highly complex clues. The Math-like category describes puzzles which do not have fixed/bounded state spaces, where problems require mathematical calculation algorithmic design. The Heuristic category mirrors that of the heuristic category in the math set: informal problems which require creative mattern-finding We also note that human-provide difficulty ratings are relatively consistent across categories. Full analysis and explanation for both Math and Logic categorizations can be found in Appendices H respectively.
Table Mean and st. dev. of solution step counts for Braingle Math and Logic datasets.
Solution Steps
Median
Creative Steps
Rudimentary Steps
Braingle Math Braingle Logic
2.5 6.0
2.0 1.2
4.4
8.6
7.0
1.
6.2 4.7
Distribution of Number of Steps solutions as part of our benchmark. We can approximate our understanding of the complexity of brainteasers and the required creativity to solve them by counting the number of steps in the brainteasers' provided human solutions. We define step to be key component of the problem,
We also provide the distribution of number of steps human

---

but this does not necessarily imply that every individual deduction is a step. We observe that, by providing OpenAI with detailed instructions and few-shot examples of step breakdown, is able to restate the human solutions into correct, broken-down steps that arrive at the final answer based on manual inspection on examples. This also true for problems that o3 fails to solve correctly on its own.
We also make the distinction between creative and rudimentary steps. A creative step generates innovative insights that reduce the problem or make the problem significantly easier to solve. Common traits of creative steps include using analogies, combining ideas from different domains, exploiting problem constraints, or devising elegant, efficient strategies that go beyond straightforward computation or trial-and-erres A rudimentary step often is more easily or routinely derivable, such as straightforward computation; it could also be step that may make progress in solving the problem but innovative, such as using trial-and-error or systematically exploring all possible options. We provide the prompt with few-shot examples in Appendix G fablei]provides statistics regarding the distribution of step count for all the problems BRAINGLE BRAINTEASER.
3.3 Problem Example
We provide a representative example of a problem from BRAINGLE BRAINTEASER. Unlike traditional math and reasoning datasets, BRAINGLE BRAINTEASER problems may be informal, open-ended, and pattern-based. For example, consider the following problem from the Math dataset:
Problem: Math 4
What characteristic do these three 12-digit numbers share with each other, but with no other 12-digit number?
100307124369
111824028801
433800063225
Any arbitrary "characteristic" may be defined, so there is no rigorous mathematical solution. However, there is clear answer up to human reasonability, which observe the following pattern:
Human Solution: Math 4 (abbreviated)
The sum of their digits are square numbers:
+ 1=1+1+8+2+4+0-248-822
36 6
+ 000-------000000----------------
36 = 6
The sum of their digit pairs are square numbers:
10 + 03 + 07 12 43 69 = 144 = 12
11 + 18 + 24 02 88 01 = 144 = 122
43 + 38 + 00 + 06 32 25 = 144 = 122
Similarly, the sum of their digit triplets, quadruplets, and sextuplets, as well as the numbers themselves, are all square numbers.
The inclusion of these informal and heuristically-driven problems in BRAINGLE BRAINTEASER allows to assess LLMs not only their ability to construct formal solutions to structured math and logic problems, but also on their ability to find creative patterns which align with human intuition.
4 Experiments and Analyses
We evaluate a range of LLMs on BRAINGLE BRAINTEASER. We first discuss model accuracy on the benchmark and then present more detailed, finer- finer-grained analyses of the reasoning process.
5

---

Table Model solution correctness, in percent, under different prompting strategies on the math and logic datasets. Results for the entire subset are shown, whereas the results for top 50 most difficult problems are displayed in parentheses.
Dataset
Model
Coll Prompt
Math Prompt
w Hint
Math Prompt w Hint
DeepSeck RI Distill Qwen 1.5B
17.2 (14.0)
16.4 (10.0)
15.2 (8.0)
17.6 (10.0)
DeepSeck RI Distill Qwen 14B
41.2 (22.0)
44.0
(2.0
42.6 (26.0)
DeepSeek R. Distill Llama 70B
42.4 (20.0)
40.8 (22.0)
45.6(24.0)
44,2 (18.0)
Math
chepseekkchat -V3)
58.0 (46.0)
(38.0))
56.0 (36.0)
58.8 (36.0)
deepseek- reesonrr (Deepseck-RI)
66.8 (48.0)
70.2 (54.0)
72.4 (48.0)
72.8 (56.0)
gemini-2.5 f.5-fisshh Baash--previcw-4--11
66.0 (34.0)
65.2 (38.0)
69.2 (44.0)
72.0 (58.0)
OpenAI o3
79.6 (66.0)
79.6 (64.0)
82.8 (66.0)
81.2 (68.0)
DeepSeck RI Distill Qwen 1.5B
(4.0)
(600)
(.8
(4,1)
DeepSeck RI Distill Qwen 14B
(1600
23.6(16.))
27.2
(32.0)
DeepSeck Distill Llama 70B
(16.0)
24.40
22.0
29.2 (26.0)
Logic
doepseck-chat (Deepscck-V3)
37.8 (30.6)
(28.0
4226
(24.5
deepseek- (Deepseek-R1)
(266))
(32)
(32.7)
(40.0)
gemini-2.5 2.5- 5--lush-preview-D44
(36.0)
(3.00
(4200)
(38.0)
OpenAl
68.4 (50.0)
71.2
(52.0)
74.4 (54.0)
4.1 Solution Correctness
Evaluation Approach Instead of manually grading model responses, we assessed whether ApenAT o3 [16] could correctly evaluate model-genen sodel-generated solutions by comparing them to human solutions. We carefully verified that o3's 's' judgments aligned with human labels with 99.3% and 97% average raw agreement for solution correctness and creativity and brute-force distinction respectively. Therefore, we adopt the model solution accuracy assessed by o3 as our primary correctness metric. Details on LLLM-human agreement experiments are Appendix B.I
Prompting Approaches We evaluated models under several prompting approaches. The CoT Prompt encourages the model to generate a step-by-step solution 3.. The Math Prompt additionally encourages the model to use rigorous mathematical reasoning, explicitly discouraging brute force, guesswork, and shortcuts. It emphasizes the need for step-by-step logical justification, ensuring each inference aligns strictly with the problem conditions. For each of these prompts. we included a version with hint or without hint, where the hint provides a problem-specifte hint that either came from Braingle or our annotators (see Section 3.I. The prompt texts can be found in Appendix [F]
Model Performance The results (Table 3) show that all models that evaluated display non-trivial degree of success on this dataset, but in all cases there remains meaningful room for improvement. Larger models such as OpenAl o3, deepseek-reasoner and gemini-2..5-laae fa-flas---reeieew-.. 04-17 consistently outperform smaller models like DeepSeek RI Distill Qwen 1.5B, especially on the Math dataset. However, despite these improvements, even the most capable models still struggle to achieve high accuracy on the most difficult subset of problems (top-50 accuracy shown in parentheses). For example, OpenAI o3, while achieving 88.8% on the broader Math set with hints and math-specific prompting, only reaches 80.5% on the top-50 problems, reflecting the persistent difficulty of these harder questions. OpenAl o3 does not return responses for some of the hardest problems even when given an unbounded amount of time to think; this could be because that the problem is too hard for the model to answer.
Another key trend is that providing hints usually boosts model performance across all sizes and datasets. This effect is particularly evident when comparing the "w Hint" and 'Math Prompt w Hint" conditions to their non-hint counterparts. In both Math and Logic tasks, hints help models achieve better reasoning steps and overcome some of their inherent limitations in tackling complex reasoning without guidance. There is performance gap between 'CoT Prompt' and 'w hint' even in some of the strongest models, indicating that some insights that are necessary for solving the problems remain elusive such that models benefit from hints that impart these insights.
We present details model performance by category in Appendix
4.2 Brute-force vs. Creativity
Humans often resort to brute-force when they cannot immediately solve problem, as validating guessed solution typically easier than deriving one from first principles. Compared humans, models have access to more significant computational resources, allowing them to guess-and-chede

---

Coll Prompt
Prompt
Prompt
Promptt
Dreadbock Drall Oven 155
cospoedichal DDeppeee-11
Onty Modded
Deepdhors
Opentd
Beth buta loice
Figure I: Distributions of brute-force and non-brute-force solutions. Results are shown for the (a) Math and the (b) Logic datasets. Numbers reflect cases where only the model used brute force.
efficiently. However, this strategy reflects a less sophisticated form of reasoning. When possible, it would be ideal for model to instead identify key insights, such as insights that reduce the problem into manageable sub- cobpoonnss, because the ability to form such insights may be necessary in novel, complex situations where brute force is impossible or intractable. In this section, we analyze the extent to which the models that we evaluated rely on brute force. For clarity, below we define how we distinguish between brute force and creativity.
Brute-force Solution We define a brute-force solution as a simple, exhaustive search strategy that systematically explores possibilities until an answer is found. This approach is viable when the search space is small but does not scale to more complex problems. Typical characteristics of brute-force solutions include reliance on code, guess-and-cheet procedures, or computations beyond what human could feasibly do. For example, when asked to find the smallest positive integer satisfying given condition, OpenAl o3 occasionally resorts to simulating the problem environment and iteratively testing values until a solution is found. While effective in reaching an answer, such guess-and-check approaches typically lack rigorous solutions and adequate reasoning.
Creative Solution We define a creative solution as an innovative, insight-driven approach that leverages pattern recognition or lateral thinking. Rather than exhaustively testing all possibilities, it reframes the problem or exploits shorteuts to reduce complexity. Such solutions often involve minimal computation and are especially valuable for problems where brute-force search is intractable.
Evaluation We evaluated model reasoning strategies by prompting OpenAl o3 to judge whether each solution uses brute force. We used few-shot prompt that includes labeled examples of brute- force and non-brute-force reasoning to guide classification. We assessed reasoning behaviors across two tasks (Math and Logic). four prompt types (CoT, Math, Hint, Math+Hint), and seven models. The complete results are shown in Figure I with breakdowns in Appendix Tables (66andd S7
All models use non- non-brute-fee force strategies prompts, models also sometimes resort to brute force, including on problems where the corresponding human solutions from Braingle do not. However, the tendency to brute force decreases with model size and capability increasing. For example, OpenAl o3 and DeepSeek Reasoner show much lower brute force rates than smaller models such DeepSeek Qwen 1.5B or Gemini 2.5 Flash. This suggests larger models are better able to avoid guess-and-check procedures and apply more structured reasoning. Still, brute force behavior appears across all models, indicating this is common fallback even for advanced systems. See Appendix K.3 for a breakdown of which problems models are most likely to use brute-force on.
sizable proportion of the problems. Nonetheless, on all
Prompting has a strong influence on brute-force behavior. The Math prompt consistently reduces brute force usage compared to the CoT baseline, suggesting that framing a problem in mathematical terms encourages more deliberate reasoning. However, the effect is limited. In both datasets, adding Hint also reduces reliance on brute force. The combined prompt Math+Hint yields the greatest reduction in brute force reliance across all models, indicating that combining structured framing with helpful cues is especially effective.
These results highlight two levers for improving model reasoning in the direction of becoming more creative and less reliant on brute force: scaling and prompting. While larger models appear naturally
7

---

b
Correctness
Change in Result
1.
14
DeepBook Dilll Liama 708
daspoedk/ssasnn (Despsack-R)
Prossiity
OpanAA
8
0.0
Belone rearting
BMI
Afte rewning
8
0.2
some resul
Figure 2: Effect of rewriting on the top 30 most difficult rewritable math brainteasers in their original narrative format ("before rewriting") and then rewritten into a more mathematical format ("after rewriting"). (a) Rewriting slightly increases model correctness. (b) For all models, most results remain the same after rewriting, while small fraction of initially incorrect answers corrected.
less reliant on brute-force, prompting remains a powerful mechanism for aligning model behavior with human-like problem-solving expectations.
Brute-force vs Adequate/Inadequa Summary When models resort to brute force, do they do so arbitrarily or because the more creative solution lies outside the model's abilities? To answer this question, we analyze whether models can summarize the solutions to the problems (seeTTable S12). The percentage of problems where models use brute-force higher when the model is unable to accurately summarize solution, which suggests that the same reasoning barrier that incentivizes a model to choose brute-force also causes to be unable to understand the human solution. This supports the hypothesis that these models still have some limitations in creative problem-solving.
4.3 Informed self-correction based on the human solution
We further analyzed the "informed coff-correcton" capabilities of LLMs. Specifically, we investigate (I) whether LLMs are able identify errors and propose corrections by comparing flawed LLM solution (SLIM) with the correct human solution (SHuman). and (2) whether LLMs will be fooled if we swap the flawed and correct solutions in this process (prompt shown below). We prompted all models to perform informed self-correction, and manually inspected the top 50 most difficult examples, which yielded 14 examples where SLLM is incorrect. For fair comparison, in all experiments, we define SLIM as the OpenAl o3 solution with CoT Prompt.
Error Analysis When correcting flawed SLIM using Situmun (Figure S5a), all models were able admit that SLLM was wrong (low fault denial). However, DeepSeek R1 distillation models, especially the 1.5B variant, were prone degenerative repetition. Error misattribution also occurred: models often failed to locate the exact error. Overall, 3/6 models produced valid corrections in over 80% of cases, Such valid corrections were more common models with a lower brute-force rate in Section 4.2 In particular, OpenAl o3, deepseek-chat, and deepseek-reasoner are more likely to detect the error, propose an appropriate fix, and present a complete and coherent solution. When tricked to "correct" SHuman using SLLM, all models took the bait and accepted the premise in most cases (Figure (S5p) Over 60% of the time, models were overly agreeable and were reluctant to challenge the prompt, and therefore almost always unquestioningly. but incorrectly, accepted that SHuman was incorrect (false confession). Justified denial (correctly rejecting the flawed SILM and insisting S11man was correct) was very rare. Case studies of these error modes are in Appendix L
4.4 Translating from verbal narratives into mathematical-sttle problem statements
Brainteasers typically are narratives instead of concise, symbolic problems common in math competitions. To see whether stylistic difference affects model performance, we rewrite problems in the Math subset, reformulating them into a competition-style format without altering the core logic.
Rewritability We first manually categorize each problem as rewritable or non-rewritable, Problems are deemed non-iattite they (1) lack rigorous mathematical answer (e.g., involve open- ended pattern discovery), (2) already resemble standard competition problems, or (3) are concise as possible their current form. From the set rewritable problems, we selected the 30 most challenging ones and prompted OpenAl o3 to rewrite each into symbolic, competition-style version. Manual inspection confirmed that 24 of these rewriting attempts were mathematically equivalent to the original version, while the remaining 6 contained only minor omissions (c.g., missing constraint),

---

Table 4: Mean & standard deviation of solution step counts for the 30 hardest Braingle Math puzzles.
Solution Steps
Median
Creative Steps
Rudimentary Steps
OpenAI o3 Solution 29)
7.7 + 2.9
8.0
2,0
5.7 + 3.0 6.1 +: 2.4
OpeaAl o3 Correct Solution 21)
8.0 = 2.2
8.0
0.9
DeepSeck** * V3 Solution (n = 26)"
12
12.0
4.3
5.5 d: 6,1
DeepSeek V3 Correct Solution (x 10)
1..
10.5
3.3
7.6 2.2
1Jamma 70B Solution (n 21)"
10.7 N..-+3. 3.4
10.0
2.9 1.7
7.8 + 3.7
Llama 70B Correct Solation (x = 6)
8.7 4: 2.9
9.5
2.0 0.6
6.7 d: 2.8
suggesting a strong underlying semantic grasp. We corrected these 6 cases that had minor omissions, resulting in fully-valid rewritings for all 30 problems.
Evaluation We evaluated performance on the rewritten problems using OpenAI o3, deepseek-
with the same CoT prompt (Fig. 2 This isolates the effect of narrative style from mathematical complexity. Rewriting problems into a more formal mathematical format increased correctness across all models. o3 improved from 56.7% to 73.3% (p = 0.023), and reepSeck-reasoner from 50.0% to 63.3% (p = 0.043), both statistically significant. showed a smaller, non-significant gain (23.3% to 33.3%, p = 0.08). This suggests LLMs better interpret formal statements when they are expressed mathematically instead of narratively. Thus, interpreting formal statements that are presented narratively remains an area of improvement for LLMs, and for now this difficulty can be ameliorated by encouraging the LLM first rewrite problem statements in a mathematical format. However, some problems were already near-formal or inherently complex, limiting overall improvement.
reasoner, and DeepSeek-R1 --r-sSee-----isii---L----- L--iistill-.LaMM-----
DeepSeek-RI-Diin --iis--i----L-LMM------
4.5 Automatic breakdown of solutions
We next analyzed the number of steps that the models used to arrive at their solutions. We quantified the number of steps using OpenAI o3. Before doing so, we manually evaluated whether OpenAl o3 was able to break down the correct human-written solutions into more granular steps that arrive at the final answer (Appendix G.. We find that OpenAl o3 is able to break the solutions down this way, counting steps at the level of granularity that we desired, in which each step is a meaningful insight rather than counting every new statement as a step. For example, for math problems that require choosing between several candidates to deduce mystery numbers, the models consider the analysis of all candidates as one collective step rather than dedicating one step for analyzing each candidate. For logic puzzles with given list of clues, the use of a clue is considered single step, even when multiple strings of thought are made (which is as intended). The example steps are listed in Appendix G We find that OpenAl o3 is successful at counting steps in the desired way even for problems that o3 fails to solve correctly independently.
Table 4 compares the mean and standard deviation of OpenAl o3's step breakdown of OpenAl o3, DeepSeek V3 and Llama 70B model solution We can see that DeepSeek solutions have the most steps and that all models' solutions contain considerably more rudimentary than creative steps. While DeepSeek RI and Llama 70B have generally low accuracy, o3 is still able to correctly divide the model outputs into steps while maintaining the same logic regardless of correctness. Furthermore, OpenAl o3 is able to divide the models' solutions responses into steps that more closely represent key components of solving the problems rather than individual deductions.
Representative Case Study and Solution Error Patterns
In this section, we complement the systematic. quantitative analyses presented above with some qualitative analyses of the specific types of errors that models exhibit.
5.1 Inefficient Use of Hints
A key qualitative gap between human and machine solutions lies in how hints are prioritized when multiple hints are provided. OpenAl o3 often defers the most informative hint until the very end, forcing itself to sift through an unnecessarily large search space. We provide an example of such
"eeveall model solutions were removed either for being outliers due to excessive casework, or for failing to provide complete solution that OpenAl o3 could break down; DeepSeek V3 was used over DeepSeek R3 as DeepSeek R3 would too often only present the final answer rather than a full solution.

---

Table 5: Case study for o3's insufficient usage of hints in the scenario where a model is unable to form the correct reasoning chain. Hint
o3 solution excerpt
"Take the sum of the number, the number mottee ane the left, the number
416723, 167234 (rotate LJ, 341672 (rotate R), 723416 (swap),
rotated one to the right, the number with the first three and last three digits
672341 (rotate pairs L), 234167 (rotate pains R). The sum
swapped, the number with digit pairs rotated the left, and number 2555553, first digit is last digit is exactly the two digits
with the digit puirs rotated to the right. The lirst and last digits of this sum e=2 and [=3 of the original number. The condition is satislied.
match the last two digits of the number. in some order."
phenomena with a six- six-digit number-guessing puzzle, One of the provided hints for this puzzle is shown in "Hint" in Table
This hint is long but encodes a simple reduction. Note that if the sum of digits some 2-digit XY, the sum is equivalent to 1111111 XY. The last digit of the sum and the last digit of the original number are thus guaranteed to match, and the first digit of the sum matches the second-to-last digit and only if no carryover occurs in this product, which occurs if X + Y < 10. This hint is by far the most informative, as two other hints in the problem relate digits to the value of the digit sum: 1) "The digital sum matches the number formed by the last two digits in the number." 2) "The sum of the first two digits is the same as the sum of the last two digits." "
This is the hint first considered in the provided human solution, and it results in a very quick reduction to a small search pool. However, in the 03 solution, this hint is used as the final filter, leading to a much larger list of candidates considered. This is likely due to the length of the hint itself; in general, it is most effective to check more complex hints at the end of a problem after more immediate deductions are made, so it may be natural heuristic to use the textually longest hints last; however, in this case, helpful hint has been wrapped in a convoluted description. Even during the check at the end, the model does not recognize the helpful reduction and opts to write out each cycle and calculate by hand, as shown in "o3 solution excerpt" in T
Explanation of Behavior The model appears to rank hints by textual length or syntactic complexity saving the longest, "hardest-looking" clues for last. When a concise mathematical pivot is wrapped in wordy prose, o3 consequently undervalues it. More effective prompting should encourage the model evaluate hints based on informational content, not surface length, or explicitly instruct it to test all hints for potential early eliminations.
5.2 Recurring Error Patterns in OpenAl o3 Solutions and Prompt-based Remedies
OpenAl o3 frequently mirrors the thand-waving" shorteuts that humans sometimes slip into proofs-skipping essential justifications and filling the gaps with confident but hollow phrases. We identify three prominent patterns, and present prompt instructions that mitigate each, steering the model toward fuller justifications and greater rigor. However, these prompt instructions do not necessarily lead correct solutions. We provide detailed examples of such errors and our prompting strategies in Appendix M
6 Conclusion
We have introduced BRAINGLE BRAINTEASER, a novel benchmark of expert-authored mathematical and logical puzzles designed probe distinct facets of LLM reasoning, including semantic parsing, solution generation, correction, step breakdown, and hint utilization. Through comprehensive experiments with OpenAl o3, DeepSeek variants and Gemini Flash, we uncover several key insights. First, despite opportunities for creative shortcuts, L.LMs frequently default to exhaustive search, especially on harder problems, indicating a reliance on computational power rather than creative insight. Second, structured hints yield consistent accuracy gains paatnsssaatccllarlyy for strong models- by guiding ththuuhf- toward critical intermediate observations, whereas simple brute-force discouragement prompts yield minimal benefit. Third, models can identify and correct their own errors when provided with official solutions, but remain susceptible to "false confession" " when flawed outputs are presented as ground truth, underscoring challenges in robust self-evaluation. Fourth, converting narrative puzzles into strict mathematical formulas does not reliably boost downstream solution accuracy, suggesting that semantic translation alone is insufficient to enhance reasoning. Finally, advanced reasoning models are capable of breaking down solutions reliably and generating correct solutions given step summary for the most challenging problems.
10

---

References
[1] Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed prompting: A modular approach for solving complex tasks. In The Eleventh International Conference on Learning Representations, 2022.
[2] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter. Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837,, 2022.
[3] Matthew Renze and Erhan Guven. The benefits of a concise chain of thought on problem- solving in large language models. In 2024 2nd International Conference on Foundation and Large Language Models (FLLM), page 476-483, IEEE, November 2024.
[4] Jundong Xu, Hao Fei, Liangming Pan, Qian Liu, Mong-Li Lee, and Wynne Hsu. Faithful logical reasoning via symbolic afain-of-thought. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 65326-1336, Bangkok, Thailand, August 2024.
Association for Computational Linguistics.
[5] Jin Peng Zhou, Charles E Staats, Wenda Li, Christian Szegedy, Kilian Q Weinberger, and Yuhuai Wu. Don't 't trust: Verify grounding LLM quantitative reasoning with autoformalization. In The Twelfth International Conference on Learning Representations, 2024.
[6] Simeng Han, Hailey Schoelkopf, Yilun Zhao, Zhenting Qi, Martin Riddell, Wenfei Zhou, James Coady, David Peng, Yujie Qiao, Luke Benson, Lucy Sun, Alexander Wardle-Solano, Hannah Szabó, Ekaterina Zubova, Matthew Burtell, Jonathan Fan, Yixin Liu, Brian Wong, Malcolm Sailor, Ansong Ni, Linyong Nan, Jungo Kasai, Tao Yu. Rui Zhang, Alexander Fabbri, Wojeiech Macicj Kryscinski, Semih Yavuz, Ye Liu. Xi Victoria Lin, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Rex Ying. Arman Cohan, and Dragomir Radev. FOLIO: Natural language reasoning with first-order logic. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pages 22017-22031, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[7] Dan Hendrycks, Collin Burns, Sauray Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fiith Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
[8] Karl Cobbe, Vineet Kosaraju, Mo Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/21 10. 14168, 2021.
[9] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024.
[10] Simeng Han, Aaron Yu, Rui Shen, Zhenting Qi, Martin Riddell, Wenfei Zhou, Yujie Qiao, Yilun Zhao, Semih Yavuz, Ye Liu, Shafiq Joty, Yingbo Zhou, Caiming Xiong, Dragomir Radey, Rex Ying, and Arman Cohan. P-FOLIO: Evaluating and improving logical reasoning with abundant written reasoning chains. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen, editors, Findings of the Association for Computational Linguistics: EMNLP 2024, pages 16553-16565, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[11] Olga Golovneva, Moya Peng Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, and Asli Celikyilmaz. ROSCOE: A suite of metrics for scoring step-by-step reasoning. In The Eleventh International Conference on Learning Representations, 2023.
[12] Art of Problem Solving. American invitational mathematics examination, n.d. Accessed: 2025-05-15.
11

---

[13] Bill Yuchen Lin, Ronan Le Bras, Kyle Richardson, Ashish Sabharwal, Radha Poovendran, Peter Clark, and Yejin Choi Zebralogic: On the scaling limits of LLMs for logical reasoning. In Forty-second International Conference on Machine Learning. 2025.
[14] Panagiotis Giadikiaroglou Maria Lymperaiou, Giorgos Filandrianos, and Giorgos Stamou. Puzzle solving using reasoning of large language models: survey. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pages 1 11574-11591, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[15] Yifan Jiang, Filip Ilievski, Kaixin Ma, and Zhivar Sourati. BRAINTEASER: Lateral thinking puzzles for large language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1431 14317-14332, Singapore, December 2023. Association for Computational Linguistics.
[16] OpenAI. Introducing OpenAl o3 and 04-mini. https://openaa.comm/iini
introducing-o--and-oooo 04- mini/ 2025. Accessed: 2025-05-13.
[17] DeepSeck-AL, Daya Guo, and Others. Deepseek-rl: Incentivizing reasoning capability in LLMs via reinforcement learning. 2025.
[18] DeepSeek-AI, Aixin Liu, and Others. Deepseck-v3 technical report. ArXiv, abs/2412.194377 2024.
[19] Google DeepMind. Gemini 2.0 flash. gemini/flash/ 2025.
[20] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyan Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
[21] Bofei Gao, Feifan Song. Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong. Lei Li, Chenghao Ma, Liang Chen, Runxin Xu, Zhengyang Tang. Benyou Wang, Daoguang Zan, Shanghaoran Quan, Ge Zhang, Lei Sha, Yichang Zhang, Xuancheng Ren, Tianyu Liu, and Baobao Chang. Omni-MATH: A universal olympiad level mathematic benchmark for large language models.
2025.
[22] Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. Mathodyssey: Benchmarking mathematical groblem-solvingg skills in large language models using odyssey math data. Scientific Data, 13(1):1392, 2025.
[23] Aryan Gulati, Brando Miranda, Eric Chen, Emily Xia, Kai Fronsdal, Bruno de Moraes Dumont, Elyas Obbad, and Sanmi Koyejo. Putnam-axions A functional and static benchmark for measuring higher level mathematical reasoning. In The 4th Workshop on Mathematical Reasoning and Al NeurlPS 24, 2024.
[24] Kunhao Zheng, Jesse Michael Han, and Stanislas Polu. minif2f: a cross-system benchmark for formal olympiad-level mathematics. In International Conference on Learning Representations, 2022.
[25] Trieu H. Trinh, Yuhuai Wu, Quoe V. Le, He He, and Thang Luong. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476-482. 2024.
[26] Yuri Chervonyi, Trieu H. Trinh, Miroslav Olsák, Xiaomeng Yang, Hoang Nguyen, Marcelo Menegali, Junehyuk Jung, Vikas Verma, Quoe V. Le, and Thang Luong. Gold-medalist performance in solving olympiad geometry with AlphaGeometry2. ArXiv, abs/2502.03544, 2025.
[27] Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, William Song, Tiffany Zhao, Pranav Raja, Charlotte Zhuang, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, and Summer Yue. A careful examination of large language model performance on grade school arithmetic. Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS), 2024.
12

---

[28] Ruijie Xu, Zengzhi Wang, Run-Ze Fan, and Pengfei Liu. Benchmarking benchmark leakage in large language models. ArXiv, abs/2404.18824 2024.
[29] Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang, Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, and Zhiting Hu. LLM reasoners: New evaluation, library, and analysis of step-by-step reasoning with large language models. In First Conference Language Modeling, 2024.
[30] Shijie Xia, Xuefeng Li, Yixin Liu, Tongshuang Wu, and Pengfei Liu. Evaluating mathematical reasoning beyond accuracy. In AAAI, pages 27723-27730, 2025.
[31] Parshin Shojace, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K. Reddy, LLM-SR: Scientific equation discovery via programming with large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[32] Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikow, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, and Alhussein Fawzi. Mathematical discoveries from program search with large language models. Nature, 625(7995):468-4777 2024.
[33] Kefan Dong and Tengyu Ma. STP: Self-play LLM theorem provers with iterative conjecturing and proving. In Forty-second International Conference on Machine Learning. 2025.
[34] Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 22199-22213 Curran Associates, Inc., 2022.
[35] Austin Xu, Yilun Zhou, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. J4r: Learning to judge with equivalent initial state group relative policy optimization. arXiv preprint arXiv:2505. 13346, 2025.
[36] Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, and Nathan Lambert. RewardBench 2: Advancing reward model evaluation. arXiv preprint arXiv:2506. 01937, 2025.
[37] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.1378) 2024.
[38] Chujie Zheng. Zhenru Zhang, Beichen Zhang. Runji Lin, Keming Lu, Bowen Yu, Dayiheng Liu, Jingren Zhou, and Junyang Lin. ProcessBench: Identifying process errors in mathematical reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 1009-1024, Vienna, Austria, July 2025. Association for Computational Linguistics.
[39] Jiayi Sheng, Luna Lyu, Jikai Jin, Tony Xia, Alex Gu, James Zou, and Pan Lu. Solving inequality proofs with large language models. arXiv preprint 2rXiv:2506607922 2025.
[40] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In NeurIPS 2023 Track on Datasets and Benchmarks, 2023.
[41] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. In International Conference on Machine Learning (ICML), 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificcton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
14

---

Justification:
We discuss the limitation of our work in Section A
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question:
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We include the API inference settings in Section B and all the prompts used in the Appendices. We will also release the dataset scrape publicly.
Guidelines:
15

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We will include our data and code in the supplementary material. We will also release the data and code publicly publicly upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips.cc/publcc/guut hww.b//ii//giidde////eeubbbb
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
16

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Ye]]
Justification:
We provide the experimental details in section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We spent $10k on purchasing API credits to conduct the experiments, which limited us to a single run per setting. Error bars are not typically reported in related work about LLMe- complex reasoning due to the high cost of API inference.
Guidelines:
The answer NA means that the paper does include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [NA]
Justification: We mainly conducted experiments with API calls and running on two A100 GPUs, each of which has 80GB GPU memory.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compule required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
ines?
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics and believe that conform with the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there is a special consideration due laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please see Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
18

---

Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All models used our work are cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com ddtasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: dataset details are provided the Detailed Dataset Information section and Section
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
19

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not pose such potential risks.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We did not use LLMs in such ways.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our L.MM policy /https://neurbb wttprrippppcccoffeeeeeeeee / for what should or should be described.
20

---

Supplementary Materials for Creativity or Brute Force?
Using Brainteasers as a Window into the Problem-Solving Abilities of Large Language Models
Limitations
22
B LLM-as-a-Judge
22
B.1 LLm-n Agreement Experiments
22
Broader Impact
22
D Full Dataset Statisties
23
API Inference Settings
23
Prompts
23
Counting Steps
24
Braingle Math Dataset Categorization
H.T
Overview
31
H.2
Examples
32
Braingle Logic Dataset Categorization
33
I.1
Overview
33
1.2
Examples
34
1.3
Full Population Statistics
39
J
Model Performance by Categories
41
Brute-Force
43
K.1
Brute Force Prompting Analysis
43
K.2
Brute Force Statisties!
44
K.3
Brute Force by Problem Category
44
K.4
Correlation Solution Summarization Ability with Correctness/Bruunc Brute- Force Usage
47
Case Study for Informed Self- Ceff-Correction
49
L.1
Informed Celf-Correction prompt
49
Informed Self-Corres Correction Results
49
Error Misattribution
False Confession
50
Justified Denial
51
6
Oler-Etabor Response
53
M Recurring Error Patterns in OpenAl and Prompt-Based Remedies
55
M.I Generalization to casual and counterfactual reasoning
55
21

---

A Limitations
Our study's scope has several limitations. The effectiveness of prompt engineering was constrained, as simple discouragement of brute-force had minimal impact, and even hints showed varied success. Our findings are specific to the BRAINGLE BRAINTEASER benchmark, derived mainly from one source and focused on math/logic puzzles, which may limit generalizability. The use of OpenAl o3 for certain evaluations and our macro-level analysis of solution step counts could introduce biases or mask detailed computational efforts.
Beyond these, LLMs frequently defaulted to brute-force over creative insight, faced challenges in robust self-correction (including "false confessions"). and didn't 't consistently improve when puzzles were formalized mathematically. These observations, from a specific set of LLMs and our operational definitions of creativity and brute-force, highlight current boundaries in advanced LLM reasoning.
In addition, because dataset was taken from the public Braingle website, data contamination a potential concern. Initial pilot studies showed that, on the few hardest problems in both Logic and Math datasets, frontier models such OpenAl o3 often constructed solutions (either correct or incorrect) which were highly distinct from the human solutions on the website. While this is a potential indicator against high levels of data contamination, this is entirely anecdotal and further study should be conducted on how our results may be affected by data contamination.
B LLM-as-a-Judge
We recognize that our use of OpenAI as judge may create potential biases in our evaluation process. We first note that using LLM-as-a-Judge is an established practice, with extensive number of papers showing that LLMs, especially recent reasoning models have high accuracy at such
judgments reasoning tasks 35 36 37
38 39 40 41]. Combined with the fact that o3 aligned well with our human annotators, we believe that this choice is well-justified.
B.1 LLM-Human Agreement Experiments
To address these concerns, we conducted a human evaluation with three annotators on 100 o3- generated solutions. Human annotators annotate each solution for correctness and whether it reflected creative or brute-force reasoning. We first provide description of the human annotators:
1) The annotators are college students who are native English speakers. 2) They have formal education in mathematical and logical reasoning, majoring Mathematies or Statistics. They have all undergone more than five years of training in solving competition-level math problems, each receiving individual top 500 honors in the proof-based Putnam competition. In addition, they are all current or past organizing board members of a university math competition, and thus have extensive experience in problem writing with focus on designing problems with creative and elegant (non-brute-fores solutions.
We provided the annotators with the same detailed annotation guidelines as those given to o3, including clear definitions and examples of brute-force and creative solutions.
We find that o3's judgments aligned with human labels with 99.3% and 97% average raw agreement distinction respectively, reinforcing its reliability
for solution correctness and creativity/brute-Foo as automatic evaluator in this setting. We also find that o3's judgment has higher alignment than Claude-3.7, dcepseek-reasonee Gemini 2.5 Flash and GPT-4o.
C Broader Impact
This work provides a systematic framework for probing the reasoning capabilities of large language models (LLMs) through the lens of brainteasers, offering insights into how these systems approach complex problem-solving tasks. By distinguishing between creative and brute-force strategies, the BRAINTEASER benchmark encourages the development of models that reason more efficiently and human-like, rather than relying on computational brute force. This shift toward more nuanced reasoning abilities could enhance Al's applicability in fields requiring critical thinking, such as education, scientific discovery, and collaborative problem-solving. where transparency and interpretability of reasoning processes are essential.
22

---

D Full Dataset Statisties
Table S1: Population statistics of the datasets curated.
Braingle Math
Braingle Logic
Statistics.
Data
Most difficult (n 250)
Most difficult (n 250)
Percentage with hints
100%
100%
Difficulty score out of 4 median
2.80 + 0.15
2.66 + 0.23
2.79
2.64
Answer word count median
172 + 204
237 + 236
109.5
162.5
Answer sentence count median
7.01 + 8.13
10.88 + 10.91
4.00
8.00
E
API Inference Settings
Model
Top,
Max Tokens
Temperature
DeepSeek RI Distill Qwen 1.5B
0.7
10000
0.7
DeepSeek R1 Distill Qwen 14B
0.7
10000
0.7
DeepSeek R1 Distill Llama 70B
0.7
10000
0.7
deepseek-chat (Deepseek-Y) V3)
10000
deepseek-reasoner (Deepseek-R1)
10000
0.95
10000
OpenAl o3
10000
Prompts
Prompt: Chain Thought (CoT)
You are an excellent mathematician with perfect logic. You are also very patient, and willing to perform very long chains of reasoning when necessary.
Can you solve this problem? Please spell out your entire reasoning steps. Finish your response in the format of "Final answer: immediately followed by your answer.
Prompt: Math Prompt
You are an excellent mathematician with perfect logic. You are also very patient, and willing to perform very long chains of reasoning when necessary. Solve the given problem, keeping in mind the following:
If you use a brute force guess-and-check method or utilize code when not necessary, you will receive no credit. you not fully justify step, you will receive no credit. Ensure each of your statements is consistent with the conditions of the problem and statements you have already written before moving on. If you do not do this, you will receive no credit. An outline of solution without concrete final answer will also receive no credit.
The problem statement correct, and a correct answer exists. If you solve a version of the problem attempt to modify the problem statement version other than the one written exactly as-is, you will receive no credit.
Finish your response in the format of "Final answer: immediately followed by your answer.
23

---

Prompt: Hint
You excellent mathematician with perfect logic. You also very patient, and willing to perform very long chains of reasoning when necessary.
Can you solve this problem using the hint? Please spell out your entire reasoning steps.
Output the final answer in the format of "Final answer: at the end of your answer.
G Counting Steps
To have OpenAl and DeepSeek R1 interpret solutions, break them down into steps, and label steps as creative or rudimentary, we presented the following prompt.
Prompt: Counting Steps
I will provide you with problem and its solution. Without solving the problem yourself, I want you divide the provided solution into steps. Then, want you to distinguish each step as either a creative step or rudimentary step. creative step is step that generates innovative insights that reduce the problem make the problem significantly easier to solve. Common traits of creative steps include using analogies, combining ideas from different domains, exploiting problem constraints, beyond straightforward computation or trial-and-error. A rudimentary step is a step that applies creative insights and often is more easily routinely derivable, such as straightforward computation; it could also be step that may make progress in solving the problem but is innovative, such using trial-and-error systematically exploring all possible options. Common traits of rudimentary steps include utilizing code, guess-and-check, or performing computations that human would ordinarily not be able do. In terms of defining a step, note that a step is defined as something that does not need to be super fine- fine-grained like "I == or "a b. b->c: a->c." Instead, each step should represent key component of the solution and the steps sequentially lead to the final answer to form complete solution. Make sure to not omit any necessary information, and also make sure that your standard of what is considered "step" is as consistent possible. Count the total number of steps in the human solution and report each step and the total number of steps the format "Total Step Count:" followed by the number of steps. Also, keep count of the number of creative steps and rudimentary steps. After displaying the total step count, please report "Creative Steps:" followed by the number of creative steps and "Rudimentary Steps:" followed by the number of rudimentary steps. Lastly, report the steps themselves the format "Steps:" with the steps needed to solve the problem.
devising elegant, efficient strategies that go
As mentioned in section 3.2, models are often able correctly comprehend the idea that a step being a key component of the problem does not necessarily imply that every individual deduction is step. now present explicit examples of steps that consist of multiple deductions which combine to more key component of the problem. For example, for math problems that require choosing between several candidates to deduce mystery numbers, the models consider the analysis of all candidates as one collective step rather than dedicating one step for analyzing each candidate. Take the following output from OpenAl o3.
Model Response: Math 9 (Output Step Example)
For each of those products, ask whether Prashant could become certain after hearing Sachin's 's opening remark. Only product 52 qualifics, because every other product can be factored in more than way that keeps the sum inside outside the Possible-Sum list.
Note that multiple computations would be required in analyzing each individual candidate, let alone all candidates; even so, the model is able to combine these rudimentary, brute force-like calculations as an overall step, turn adhering to our intended definition of step of representing a substantial amount of progress to solving the problem rather than every individual deduction.

---

For logic puzzles with a given list of clues, the use of a clue is considered single step, even when multiple strings of thought are made, which is, again, as intended.
Model Response: Logie 5 (Output Step Example)
Use the "only one ascending triple" condition (clue 7).
Exactly lower than Q can be left of her (or 2 triples would appear), triple must be Q-?-A.
To prevent J-K-A or T-K-A from creating a second triple, K must to the far left. To avoid T-J-A making another triple, J must precede T.
Complete rank order: K QTA
Now, consider the following problem that wants to arrange two I's, 3's, 7's, and 9's in string such that each pair of consecutive integers is prime, as well as the number itself and its reverse. We will then compare the outputs of OpenAI o3 and DeepSeek RI to observe any similarities and differences how solutions are divided into steps and how steps are categorized as either creative or rudimentary.
Problem: Math 23
Arrange the numerals "I 1337799" form an 8-digit number (referred "ABCDEFGH") that meets the following conditions:
ABCDEFGH is prime number 2) HGFEDCBA (reversing the digits) is prime number 3) all consecutive 2-digit subsets of ABCDEFGH are prime numbers (AB, BC, CD, DE, EF, FG, GH are all prime) 4) all consecutive 2-digit subsets of HGFEDCBA are prime numbers (HG, GF, FE, ED, DC, CB, BA are all prime)
Note that the creative step uses a mathematical insight regarding reversible primes containing 9 to reduce the problem into more limited space of possibilities, while the rudimentary step simply uses riil-and-errod on all possibilities of the remaining digits to find valid combinations. Again, the explicit steps themselves are provided below.
Human Solution: Math 23
The only available 2-digit prime numbers are: 11, 13, 17, 19, 37, 71, 73, 79, and 97. The number pairs that are prime in both directions are: 11/11, 13/31, 37/71, 37/73, and 79/97 (19 eliminated since its reverse, 91, not prime). With these numbers, it can seen that can only be paired with 7. Therefore, the solution must be of the form 97????79, 9797????, or ????7979. ???? must therefore be 1313, 3131, or 3113. This leaves the possible solution pairs of 97131379/993131799 97971313/31317977 and 97311139//9711333 1379 (the last pair being palindrome). Of these, only 97131379 and its reverse, 97313179, prime numbers.
07973131/13117997
25

---

Model Response: Math 23 (OpenAl o3)
List every two two-digit prime that be built from the available digits 1, 7, 11, 13, 17, 19, 31,37, 37, 71, 73, 79, 97. (Rudimentary)
2. From that list, keep only those whose reversals are also prime, because each two-digit block must work in both the forward and the reverse 8-digit number. The viable pairs are 11/11, 13/31, 17/71, 37/73, 79/97. (Rudimentary)
3. Observe that the digit 9 can appear in two-digit prime only as 79 or 97, so every must be adjacent to a (Creative)
4. Use the previous observation to pin down the overall skeleton of the 8-digit number: it must be of the form 97
79, 9797 - or
7979. (Creative)
5. Insert the remaining digits 1,1,3.3 into the four open slots so that every new two-digit block still in the list from Step leaving only three feasible fillings: 1313, 3131, 3113 13. (Rudimentary)
Combine the frameworks from Step 4 with the fillings from Step 5 to generate the complete candidate pairs of 8-digit numbers (each with its reverse): 97131379 97313179, 97971313 31317979, 97973131 13137979, 97973113 0077311333117999 31177799, 97311379 173113779. (Rudimentary) Test each candidate for primality; only 97131379 and its reverse 97313179 are prime, so they satisfy all conditions. (Rudimentary)
Model Response: Math 23 (DeepSeek R1)
List all possible 2-digit primes from the given digits (11, 13, 17, 19, 31, 37, 71, 73, 79, 97). (Rudimentary: This is straightforward enumeration of primes from the given digits.)
2. Eliminate primes whose reverses are not prime (c.g., 19 is eliminated because 91 is not prime). (Creative: This insight reduces the problem by filtering out invalid candidates early.)
3. Identify digit constraints (e.g., 9 can only be paired with 7). (Creative: This observation
narrows down the possible starting and ending digits of the solution.)
4. Determine possible forms of the solution (97????79, 9797????, or ????7979). (Creative:
This step leverages the digit constraints structure the problem into manageable subproblems.)
Enumerate possible middle segments (1313, 3131, or 3113). (Rudimentary: This is systematic exploration of possible combinations.)
Generate all candidate pairs based on the forms and middle segments. (Rudimentary: This is a straightforward generation of possible numbers from the constraints.)
Check which candidates and their reverses are prime numbers. (Rudimentary: This involves computational verification of primality.)
Conclude that only 97131379 and reverse 97313179 satisfy all conditions. (Creative: This final step synthesizes the results and confirms the unique solution.)
Note that both models divide the steps nearly identically, with the only difference being that the DeepSeek model divides the testing of final candidates and the declaration of the final answer two steps whereas OpenAl o3 only considers this as one step. More importantly, both models' nearly identical breakdowns also correctly follow the human solution' 's logic in the exact sequence it was presented; this exemplifies the models' abilities to properly parse a provided response into steps while maintaining its overall logical structure. Now, we observe both models' steps 3, 4, and 5. Both models declare steps and as creative making more sophisticated deductions on prime numbers including 9 and significantly narrowing down the possible forms of the solution, which correspond to our definition of creative steps in making an innovative insight without systematically exploring all possible options. Contrarily, both models correctly declare the brute-force exploration of feasible segments I's and 3's as rudimentary step.
However, as noted in Table S2 and [S3] we observe that DeepSeek RI tends to declare more steps as creative than OpenAl o3, and we observe that here, DeepSeek declares steps 2 and 8 as creative while OpenAl considers them to rudimentary. By our definitions, we expect this to more rudimentary; step 2 requires scanning all of the prime numbers to see which primes have prime reversals, which would be rudimentary, but DeepSeek considered this creative for filtering invalid candidates. Despite the significant progress, the greater focus should have been placed the
26

---

Table S2: Mean and standard deviation of solution step counts for several models' (OpenAl o3, DeepSeek VI, Llama 70B) solutions for the most difficult puzzles in the Braingle Math and Braingle Logic datasets (n=250) based on OpenAl o3. DeepSeek V1 solutions were used instead of DeepSeek RI solutions because too many DeepSeek R1 responses only consisted of the final answer without the reasoning process that we can divide into steps.
Solution Steps
Median
Creative Steps
Rudimentary Steps
Braingle Math
OpenAI 03 Solution
8.2 3.5
8.0
1.9 + 1.2
6.3 3.2
DeepSeek VI Solution
11.2 4.6
10.0
2.9
+ 2.5
8.3 3.7
Llama 70B Solution
9.6 3.4
9.0
2.6 1.8
Braingle Logic
OpenAl o3 Solution
8.1
7.0
2.2 1.8
5.8
DeepSeek VI Solution
12.3 5.2
11.0
3.3 + 2.5
9.0
Llama 70B Solution
11.0 1L.0=4 4.6
10.0
2.9 -
2.1
8.1
Table S3: Mean and standard deviation of solution step counts for the 30 most difficult puzzles in the Braingle Math and Braingle Logic datasets. (*For math, I human solution and I model solution were excluded as outliers due excessive casework, human solution excluded due directly asserting answer)
Solution Steps
Median
Creative Steps
Rudimentary Steps
Braingle Math - OpenAI o3 Step Count
Human Solution (n = 28)
7.8 - 3.5
7.0
2.1 + 1.4
5.7 + 2.6 5.7 + 3.0 6.1 + 2.4
OpenAl o3 Solution (n = 29)
7.7 = 2.9
8.0
2.0
+ 1.0
OpenAl o3 Correct Solution(# 21)
8.0 2.2
8.0
1.9 + 0.9
Braingle Deepseek Step Count
Human Solution (n 28)*
7.8 4.7
7.0
3.9 + 2.3
3,9 + 2.7
OpenAl o3 Solution (n = 29)*
6.8 = 3.2
6.5
3.3 4 1.3
3.5 + 2.5
OpenAl o3 Correct Solution (n = 21)
7.1 = 3.1
7.0
3.2 + 1.3
4.0 + 2.5
Braingle Op o3 Step Count
Human Solution
30)
10.6 5.2
2.9 1.7
7.8 5.5
OpenAl o3 Solution (z = 30)
N/A
N/A
N/A
OpenAl o3 Correct Solution (n =??)
N/A
N/A
N/A
N/A
Braingle DeeiccDe R1 Step Count
Human Solution -30)
9.6 4.7
9.0
5.3 5.3223
4.3 2.6
OpenAl Solution 30)
OpenAl Correct Solution (n =??)
systematic exploration of the option space. This may be an issue with prompting, or could be the fact that DeepSeek may be associating significant progress with creativity rather than the innovation of the ideas used. Then, step 8 from DeepSeek is declaring the final answer that should have been obtained from step 7, this should be the most rudimentary because no additional deduction was made. It is interesting to see that DeepSeek R1 considers this as creative because it synthesizes the results, but after observing other DeepSeek responses, we see that calling the declaration of the final result creative step is not common eиror and that erroneously associating significant progress rather than innovation with creativity may be a more likely source of why DeepSeek RI categorizes a greater number of steps as creative than OpenAl o3.
From Table S2 we observe that DeepSeek VI solutions have more steps than Llama 70B solutions for both math and logic. OpenAl o3 outputs have fewer steps than both of the other models for both math and logic brainteasers. We also observe that logic problems generally have more steps than math problems as seen with DeepSeek VI and Llama 70B solutions; that said, for OpenAl o3, the average step count for logic problems is marginally smaller. The DeepSeek V1 and Llama 70B responses are both considerably longer than human solutions, which may be a result of continued trial-and- -ar------rorr attempts that would not be noted in human solutions.
27

---

We now present an example of an incorrect solution from DeepSeek VI followed by the OpenAl o3 step breakdown, As the DeepSeek solution is very long, some portions are truncated but the overall sequence of logic is still present.
28

---

Model Response: Math 11 (DeepSeek V1)
First, let's clarify some terms ensure we're the page.
Understanding the Terms
Palindrome: A palindrome a number (or word) that reads the same backward as forward. For example, 121, 1331, and are palindromic numbers.
2. Non-palindromic integer: An integer that not a palindrome. For example, 12, 123, and 10 are non-palindromic, since reading them backward gives 21, 321, and
01, respectively.
3. Cube of an integer: The integer multiplied by itself three times. For example, the cube of 3 is
33 3x3x33 27.
The Problem Restated
We're looking for an integer n such that n is not a palindrome, n palindrome,, and n is the only such integer.
Checking Small Integers
1. n is palindrome, 1 1, which is palindrome, excluded since itself is palindromic.
2. n 2:
palindrome, 2 8, which is palindrome, excluded since is
palindromic.
3. n 26
26 not a palindrome, 26 17576, which (by the problem's 's assertion) is palindrome, this is the unique solution.
Considering Negative Integers
A negative integer is non palinddodooi if its positive counterpart is not palindrome (since the negative sign makes it non-palindromic, as palindromes are typically considered for absolute values in numbers).
Let's check some negative integers. (The model then checks -1, -10, and -11, but gives up after finding no success).
Perhaps the answer is among positive integers, and missed it earlier.
Re-evaluating Positive Integers
Let me think about the cube being a palindrome. The cube must read the same backward as forward.
Looking back, the only cubes that are palindromes from the numbers I checked are (from 1), (from 2), 343 (from 7), and 1331 (from 11).
But these values are palindromic themselves.
Wait, is there any non-palin 'n' whose cube a palindrome? Maybe higher numbers? (The model then tries 101 and 1001 but fails 101 and 1001 are palindromes.
29

---

Model Response: Math 11 (DeepSeek V1) continued
This is perplexing. Maybe the answer is that there is such non-palindromle integer whose cube a palindrome. But the problem states that there is one, implying it exists.
Wait, perhaps I missed a small number. Let me check 'n = 0", 0 is palindrome, 0 0, which is palindrome, but 0 is palindromic.
Alternatively, maybe the answer 10, but earlier we saw 10a 1000 isn't 't palindrome. Unless we consider 1000 as palindrome if we ignore trailing zeros, but that's not standard.
After some research or recalling, think the answer is **26** but earlier calculation showed 26 17576, which isn't palindrome. Maybe miscalculated.
Let me calculate 26 carefully. (The model obtains that 26 17576. 17576 backward is 67571 / 17576. So not palindrome.
(The model then continues to try numbers it has already tried out, such 0, 10, -1, and 26, but still is unsuccessful.)
Given that, and since the problem states there is one, the most likely answer is **26** accepting that 17576 is a palindrome (perhaps by some definition where the middle digit is the center and the others mirror, but they don't 't here).
However, standard definition says a palindrome reads the same backward, which 17576 does not (67571 * 17576).
Given this confusion, and based on mathematical sources, the accepted answer is:
**26** with 26 17576 being considered a palindrome, even though by strict digit reversal it's not. This might be special case exception in how palindromes are defined for this problem.
Model Response: Math (OpenAl o3 Breakdown)
State and explain the relevant definitions of "palindrome." Thon-palindromme integer," and "cube," establishing the criteria the sought integer must satisfy. (Rudimentary)
Propose brute-force method: examine successive integers, compute each cube, and check both and n² for palindromicity. (Rudimentary)
Implement the method for = 1-9; record that either n is itself a palindrome or n is not, so none of these values fits the requirement. (Rudimentary)
Continue the same manual search for = 10-22 and again find no qualifying integer. (Rudimentary)
Rely on recollcction/iituttoon and single out n = 26 as a promising candidate instead of continuing the systematic scan in order. (Creative)
Calculate 26 17 576 and test 17 576 for palindromicity, concluding (incorrectly) that is palindrome. (Rudimentary)
Declare that, because 26 appears to work and no other candidates have been discovered, 26 must be the unique non-palindrooic integer whose cube is a palindrome, and give 26 as the final answer. (Rudimentary)
From reading the outputs, it is clear that OpenAl maintains the same chain of thought as the DeepSeek VI response while consolidating much of the trial and error performed on previously tested candidates. Like DeepSeek V1, the OpenAl also notes that 17576 is not palindrome. We also find that the steps are larger components of the problem rather than just individual deductions as desired, though we could argue that steps 3 and 4 could be combined since the method of search is identical in both steps. Nonetheless, still see that OpenAI o3 breaks down provided model
30

---

Table S4: An example of creative and rudimentary steps.
Creative Step Example
Rudimentary Step Example
Examine the given ingut-outpot pairs and discover the hidden rule: write each number binary and weight each digit-countt every and for every
Convert 9304 to binary. obtaining
1001000101 1000.
response into macro-steps that correctly highlight the chain of thought when the response can yield an incorrect answer. This shows that the OpenAl o3 step breakdowns do not look beyond the provided response itself and maintain the thought process given by the response.
One drawback of using the step count to measure the complexity of problem is that the complexity may vary significantly between steps. For many problems, there are fewer insights, but the insights are more difficult to obtain and are more significant in solving the problem: yet the model's presentation of these steps does not necessarily highlight the differences in complexity between steps. This is common pattern between list of numbers
particularly true for math problems that ask for without giving any hint of the pattern. Without acknowledging the required trial-and-error and time needed to discover the step's key insight, such "steps" would be considered equivalent in complexity to routine arithmetic calculations under this measurement. Therefore, we also make the distinction between creative steps and rudimentary steps. In the problem that provides set of input-output pairs where the goal is to find the corresponding output of 9304, we show in Table (S4 that finding the input-output rule is considered creative as significantly reduces the difficulty of the problem, while carrying out the rule on converting the designated number 9304 is rudimentary since calculations like binary conversion are simple and routine. The explicit steps themselves are provided in Supplementary Section G
H Braingle Math Dataset Categorization
H.1
Overview
H.1.1
Categories
We manually categorize the math dataset based on problem style, into the following three categories and respective subcategories. Category (I) represents problems which could reasonably be found on standard math competition such as AMC, AIME, etc. Category (2) represents problems which are still mathematically rigorous, but not quite the style of competitive math problem. Category (3) represents problems which are not rigorous, and involve finding patterns or thinking outside the box.
1. Standard competitive math:
Geometry Problem with spatial reasoning, finding areas/side lengths, etc.
Number Theory Divisibility, factorization, etc.
Combinatorics Counting, probability, etc.
Algebra Setting up and solving systems of real-valued equations
2. Nonstandard:
Logic Problems resembling mathematical Logic puzzles
Special Number Finding a "mystery" number with desired properties
3. Heuristichonn-rggrros:::
Pattern Finding the "next number" or a common rule for group or sequence of numbers
Arithmetic Rearranging arithmetic expression achieve target number
31

---

H.2
Examples
H.2.1
Standard category
Problem: Math 50 (Geometry)
It's easy to see that ring can completely hold (surround) two identical smaller rings with half diameter, without overlapping. Three times the diameter, bigger ring is space enough to seat seven rings; the outer six touching both the middle ring and the bigger circle/perimeter. Using this basic information and your imagination, determine the maximum number of rings that could be housed inside another ring with four, five, six and seven times the diameter.
Geometry problems require spatial reasoning- in this case, the solver must visualize an arrangement of circles contained within larger circle, and argue about the "total diameter" of this arrangement.
Problem: Math 124 (Number Theory)
I have machine which has four cog wheels in constant mesh. The largest cog has 242 teeth and the others have 160, 64 and 22 respectively. How many revolutions must the largest cog make before each of the cogs is back in its starting position?
Number theory problems often involve concepts like common divisors, multiples, and remainders.
Problem: Math 117 (Combinatorics)
What is the largest number of pieces you can form with straight cuts of pizza? The pieces do not need be of equal size.
While this may seem like problem requiring visual arrangement, clever counting argument can simplify the problem greatly: for every cut added, it intersects the existing cuts, it adds + new pieces the arrangement.
Problem: Math 125 (Algebra)
Three people (A, B, and C) need to bridge. A can cross the bridge in 10 minutes, B can cross in minutes, and C can cross in 2 minutes. There is also a bicycle available and any person cross the bridge in minute with the bicycle. What is the shortest time that all men can get across the bridge? Each man travels at their own constant rate.
Algebra problems often involve word problems with times, prices, and rates; they can be easily computed after proper mathematical setup from the word problem.
H.2.2 Nonstandard category
Problem: Math 122 (Logic)
The people of Olde Mathville had unique ways of punishing wayward citizens. For example, those convicted of crimes of dishonesty were chained the Liars' Rail until they solved number puzzles.
One such puzzle has been recently discovered! In the multiplication below, each letter L, I, A, R, and takes the place of different digit. Find the digits to make the multiplication true.
LIAR R
S
RAIL
32

---

While this problem is still mathematically themed, the primary focus of the problem is in logical reasoning- the solver must assign five unique values to the five unknown entities, as one would in logic grid.
Problem: Math 75 (Special Number)
What is the smallest number that the sum of two different pairs cubes?
In this category, the solver must find number or set of numbers satisfying a unique set of constraints. This category can be seen as adjacent to Number Theory; however, in Special Number puzzles, the primary challenge is not in making rigorous number-theoretic arguments relating to divisibility, but rather deductive, heuristic, and sometimes "lucky" observations.
H.2.3 Heuristic category
Problem: Math 79 (Pattern)
What is special about this particular sequence of numbers? 42526037644900014334997
Here, the problem does not have a rigorous final answer, but rather requires general pattern-finding. In this problem, the listed numbers are the digits of pi, with I added to each. We review all such problems to ensure that the final answer both "reasonable" and the unique "reasonable" solution (i.e. no other plausible patterns exist).
Problem: Math 216 (Arithmetic)
Add the appropriate mathematical operators or symbols to make the following correct: 3 5 90
Parentheses may be used freell.
In these problems, the solver inserts mathematical operators to achieve desired numerical goal. We place these kinds of problems in the Heuristic category, the solver often required think outside the box, and the set of valid "actions" is not always rigorously defined. In this case, the answer requires use of a factorial:
3/4 5! 90
I Braingle Logic Dataset Categorization
L1
Overview
L.1.1 Metrics
For each brainteaser, where applicable, evaluate the following metrics:
Depth:
The number of distinct traits in each "grouping"
Width: The number of "groups"
State Space Size:
The total number of possible arrangements within the initial defined constraints
the problem
Number of clues:
The number of distinct pieces of information given, AFTER the state space is
defined
L.1.2 Categories
We manually categorize the logic dataset based on general structure of each problem, into the following four categories and respective subcategories. Note that categories (1) and (2) represent rigorous logical deduction problems with bounded state spaces. In (3), problems do NOT have well- defined or bounded state spaces, with answers ranging from numbers to descriptions of algorithms. In (4), answers are not logically rigorous, but can still reflect model's 's pattern-finding or heuristic reasoning skills.
33

---

(1) Large state space, simple clues:
OD Logic Grid Standard logic grid with no positional reasoning. d traits, options per trait
ID Positional- Puzzle with ID positional reasoning. Includes races, seating arrangements, etc.
2D) Positional Puzzle with 2D positional reasoning. Includes Bingo cards, chess boards, etc.
Number Guess the number; deduction-style problem with mathematical clues. d = 2, digits
Clusters No fixed traits, cluster wd objects freely into w groups of d
Tree Puzzle with tree-like structure. Includes family trees, climination-style tournaments.
(2) Small state space, complex clues:
Liars Clues themselves may be true or false
Communication Impartial information between multiple parties, typically through conversation
Compound Multi-clause conditional clues (i.e. many if, then, statements)
(3) Math-like:
Algorithm Design and/or execute algorithm
Math Similar to comp-math, unbounded numerical state space
(4) Heuristic/non-rigoran
Pattern Observing non-rigorous pattern, "riddle"-like problems
Linguistic Clues based semantic meaning
1.2
Examples
1.2.1
Computing metrics
statement defining "state space" is statement which gives general information about the problem, which is standard among other problems of the same type. This may include defining traits (name, occupation, age), defining options per trait ("occupations are welder, bricklayer, .).",, setting number of items per category ("there are 2 welders, bricklayer...", and declaring other constraints (no repeating digits). In contrast, "clue" defined problem. Note that for some problems, the concept of "depth" and "width" can be up to interpretation. These metrics are made to generally quantify the difference in the various categories.
piece of information which unique to the given
Here is example solution for classic logic grid-style puzzle:
Problem: Logic 15
1st House: Yellow, Norwegian, Water, Cats, Dunhill Blue, Dane, Tea, Horse, Blends 3rd House: Red, Brit, Milk, Birds, Pall Malls
2nd House:
4th House: Green, German, Coffee, FISH, Prince
5th House: White, Swede, Beer, Dogs, Bluemasters
Each "grouping" contains 6 traits: house, color, nationality, beverage, pet, and cigar, so the depth is There are 5 groupings, so the width is 5. For classic logie grid with depth d and width w, the state space is given by: (w!)d-1 (5!)6-1, Note that we define state space as the entire required arrangement, NOT just the space of possible final answers.
Problem: Logic
There is ten-digit mystery number (no leading 0), represented by ABCDEFGHIJ, where each numeral, 0 through 9, is used once. Given the following clues, what is the number? C + is multiple of
A A
2) F G H + + J is multiple of 5.
3) A + C
+ G
multiple multiple of 2.
4) B D F+ H++HJJ.
5) AB a multiple 3.
6) CD is multiple of 4.
7) EF a multiple of
8) Gs multiple of 8.
9) IJ is a multiple of 10.
10) FE, HC, and JA are all prime numbers.
34

---

Here, a "grouping" is simply pairing digit positions with digits, giving depth 2. There are ten positions, giving width of 10. There are clues (not counting the initial set up information), and the state space size is given by 9 9l, accounting for the non-leading zero information.
1.2.2 Large/simple category
Problem: Logic 32 (0D)
These five businessmen represented different companies at recent trade fair. Unfortunately, hotel they were all due to stay had accidentally double booked their rooms. They tried other hotels in the area, but all were fully booked. Consequently, they all agreed to share the only two rooms available in the hotel one twin and one triple. From the clues, can you work out each man's ' name, company and official title?
When the five men realized their dilemma, they drew straws to see who would share with whom. The outcome for four of them
that the CEO shared with the businessman from ABM Inc. and Edgar shared with the Developer.
2. Alan does work for Reed Right and not the Director. The Director does not work for Lantel or Blue Teeth.
Neither Clarkson nor Grimaldi works for Reed Right. Grimaldi either the IT Analyst or the CEO.
The businessman from Reed Right did not share room with Thomas.
5. Clarkson, who does not work for Lantel, shared with either Thomas or Grimaldi but not both.
6. Edgar's surname is either Casson or Graves. Edgar did not share with Carl.
The businessman from Chiptech shared with the businessman from Lantel.
8. The CEO shared with the IT Analyst. Casson works for either Blue Teeth Chiptech.
10. Neither Joshua nor Alan works for ABM Inc.
11. Fielder shared with the Director.
12. The Accountant from Blue Teeth shared with the Developer.
First names: Alan, Carl, Edgar, Joshua, Thomas.
Last names; Casson, Clarkson, Fielder, Graves, Grimaldi.
Company: ABM Inc., Blue Teeth, Chiptech, Lantel, Reed
While there is some variation, the majority of puzzles in the OD category follow the standard logic grid format, where we are given a fixed set categories (first names, last names, companies), and set of clues which each give a fairly simple individual deduction.
Problem: Logic 178 (1D)
List the order in which each person finished.
Tommy Tombstone finished after Lance Lamers and Brett Brown but before Mitch Monday,
Peter Poultry finished before Daniel Dusk and Lance Lamers.
Sam Sunny finished after Peter Poultry and before Jack Jill and Harry Hills.
Keri Kernel finished after Peter Poultry, Mitch Monday and Tommy Tombstone.
Lance Lamers finished after Brett Brown and Daniel Dusk, but before Jack Jill and Mitch Monday.
Mitch Monday finished after Sam Sumny and Brett Brown.
Brett Brown finished before Jack Jill, Mitch Monday and Peter Poultry.
Daniel Dusk finished before Keri Kernel and Tommy Tombstone, after Sam Sunny, Jack Jill finished before Keri Kernel, Tommy Tombstone and Mitch Monday, but after Peter Poultry and Daniel Dusk.
Harry Hills finished before Mitch Monday but after Lance Lamers, Jack Jill and Tommy Tombstone.
race standard way to express a ID problem. While these kinds of problems can reduced to the forms of logic grids (by considering "position" as a category), problems of this type generally possess higher width and lower depth, as there is higher emphasis on ordering large number of characters.
35

---

Problem: Logic 12 (2D)
You given stack of bingo cards. Your task is find specific card. Given the following clues, what is the number arrangement of that card?
Columns, left to right, are: B (contains numbers through 15), I (contains numbers 16 through 30), N (contains numbers 31 through 45), G (contains numbers 46 through 60), O (contains numbers 61 through 75). Rows, top bottom, are: 1, 2, 4, 5. An example of coordinate nomenclature: B1 identifies column B row 1. N3 is a free space (contains no number). No number appears more than once.
1) Each numeral (0 through 9) appears one time in Row 1.
2) The sum of the numbers in Row 4 square number.
3) There is only one two-digit prime number in each row.
4) The range of the numbers in Column
5) Each number in Column G has tens digit that is less than the units digit.
6) Each number in Column
odd.
7) In only one column are the numbers in descending order from top to bottom.
8) Each column has only one numeral that appears exactly two times.
9) The smallest number in each column is located in Row 5.
10) The sums of each column share single common prime factor.
11) The numeral 5 only appears
time the card.
12) The sum of the numbers in each diagonal an odd number.
13) The product of B3 and O3 has units digit of 2.
14) The product of and G3 has a units digit of 4.
There are four bingo-style questions in the Logic dataset. Here, the solver must deduce the values of entire 5x5 grid, where clues may pertain columns, rows, diagonals, individual cells. These problems tend to have incredibly large state spaces, but clues can be very reductive (e.g. clue 6 in this puzzle reduces the state of possible answers by factor of around 25 on its own).
Problem: Logic 10 (Number)
Professor Abacus is purchasing a ticket for the Deca Lotto. The lotto number has ten digits, using the numerals 0 through 9, each numeral used once. The clerk asked what number he wanted to pick. Professor Abacus handed the clerk piece of paper with nine statements, saying ""If you correctly figure out the number, I will give you half of whatever I win." What is the number?
1) The sum of the first five digits is prime number.
2) The sum of the last five digits is triangle number*
3) The of the digits in the odd positions is odd number.
4) The sum of the middle two digits a square number.
5) The sum of the middle four digits is cube number.
6) The difference between the 1st and 10th digits is two.
7) The difference between the 2nd and 9th digits is three.
8) The difference between the 3rd and 8th digits is four.
9) The numeral 4 somewhere in the first five positions.
You can form triangle arrangement by building it in the pattern row = 1, row 2 2, row
3 = etc. eg. 10 a classic triangle number as per ten pin bowling. They are arranged in triangle 1, 2, 3, 4.
The second example shown in I.2.1 is also classic number problem. As with the ID problems, Number problems can be reduced to logic grids, where we are grouping "digits" with "positions", However, Number problems are distinct in the mathematical nature of their clues, which cannot be replicated nicely natural language form, where the digits and positions are replaced with arbitrary names and placeholders.
36

---

Problem: Logic 35 (Clusters)
child has 4 blocks with different letter each side (no letter is repeated different blocks either). If the list of words below can all be formed using these blocks, figure out which letters belong on which blocks... skid, hoax, joey, glum, rand, grit, monk, fair, vane, wide, cafe, dupe, joke, bail, shop
In this problem, we must form four unordered clusters of 6 letters each. These problems differ from traditional logic grids the sense that individual category names are no longer specified- rather than grouping across six different categories, all entities belong to one category: letters.
Problem: Logic 81 (Tree)
One day, a college student named Tina walked into her logic class and waited for her fun day of logic once again. Finally the professor, Professor C. D. Rock, walked in saying, "I just went through some stuff and stumbled upon an old family tree, that belonged to my grandparents, that gave me idea. have here a few clues, and you have to use them to figure out their family tree!"
Tina then gets the clues, and tries to work them out. Unfortunately this puzzle is little harder than the normal ones Professor C. D. Rock gives out. Can you help her? The family tree consists of two grandparents, who had 3 children, each of whom get married and have children.
Males: Cole, Cristian, Jason, Neil, and Steve
Females:
Amanda, Ashley, Beth, Erin, Kaitlyn, Katherine, Makayla, Payton, and Tammy
Clues:
One of Makayla's cousins is Jason's son.
2. One of Ashley's aunts is Tammy.
3. Tammy's 's brother-in-law is Neil's son.
Kaitlyn's 's sister is Ashley's cousin.
5. Ashley's ' uncle, Steve, is Erin's 's brother-in-law.
The three uncles are Payton's 's dad, Cristian, and Katherine's son.
7. The three aunts are Kaitlyn's 's mom, Ashley's mom, and Cristian's 's sister-in in- Jason's 's brother is Ashley's dad.
9. Amanda's 's sister is Steve's 's niece.
10. Beth is not Cole's aunt.
In Tree problems, we need to fill in the identities of family tree- this requires hierarchical thinking. where deductions about person's "generation" " intersect deductions about person's 's individual relationships.
1.2.3 Small/complex category
Problem: Logic 85 (Liars)
Edward, Howard, and John are three high school students each of whom is taking three of the four subjects, biology, chemistry, history, and mathematics. One day while talking about their programs they made the following statements.
Edward: There is just one subject we're re all taking. I'm the only of us who is taking mathematics. No two us are taking the same three subjects. John is wrong when he says that Howard and
both taking chemistry.
Howard: Ed is the only one of us who is taking history. John and are taking the same subjects. We're re all taking biology. Two of are taking both chemistry and biology. John: We're all taking mathematics. Howard is taking history. Ed is taking one subject that I'm not. Both Howard and Ed are taking chemistry.
If two and only two of each boy's 's statements are true, what subjects is each boy taking?
The state space of this problem is very small- for each student, one must deduce which subject they are not taking, out of 4. Liars problems, complexity arises from the fact that clues themselves may be true false, and unlike problems in the Large/simple category, immediate deductions cannot be
37

---

easily made from each clue. Such problems are susceptible to brute-force testing; model can easily test every possible arrangement subjects, and check how many statements true false, until exactly two of each boys' statements are true.
Problem: Logic 104 (Communication)
Mr. Simkin, the new math teacher at school, was impressed by his students' ability to solve logic puzzles. He pulled aside three more students, and handed them each sealed envelope with a number written inside. He told them that they each have positive integer, and the sum of their numbers was 14.
Manny, Moe, and Jack each opened their envelopes. Mr. Simkin asks Manny if knows anything about the numbers the other two are holding, and Manny says, "I know that Moe and Jack are holding different numbers."
Moe answers, "IN THAT CASE, I know that all three of our numbers different." Jack thinks for a bit, and then says, "Now know all of our numbers." Mr. Simkin turns the class and asks if anyone in the class knows the numbers. Gretchen's hand shoots up into the air, and after waiting for a while to see if anyone else will get the answer, Mr. Simkin calls Gretchen.
What numbers does she say they each are holding?
In Communication problems, the reader must make deductions based on the deductions of multiple agents with impartial information. This dynamic adds inherent complexity to each clue; rather than only considering the deductions the reader themselves make, the reader must also keep track of the deductions that each character can individually make as well.
Problem: Logic 235 (Compound)
Just before the end of the term four high school students were discussing their chances for certain grades. The following remarks contain the gist of their hopes and fears.
Jack: We'll all get different grades. If get
then Lucy will get D.
Jean: If Lucy gets a C, then Jack will get D. Jack will get a better grade than Paul.
Lucy: If Jean doesn't 't get A, then Jack will get C. If get B, then Paul won't get D.
Paul: If Lucy gets an A, then I'll get a B. if Jean doesn't 't get a B, won't I either.
When the final examinations were graded and the term marks made out each of the four passed, and strange as it may seem, each received a grade that checked exactly with all the ideas they had previously expressed.
What grade did each receive?
Here, each individual clue is conditional statement, and only allows for very specific deductions (e.g. Jack has an A = Lucy has D). Once again, we see that problems of this type are susceptible to brute force: there only 4! arrangements of grades for the four students, and model can easily iterate through all arrangements and check for contradictions in the logical statements.
1.2.4 Math-like category
Problem: Logic 38 (Algorithm)
is your task to determine how high you drop billiard ball without it breaking. There is a 100 story building and you must determine which the highest floor you can drop a ball from without breaking. You have only two billiard balls them break before you determine the answer then you have failed at your task. What the order of floors that you should drop the balls from minimize the number of droppings that you will have to make to determine the answer?
test objects. If both of
Assume that if a ball doesn't 't break you reuse without worrying about it being weakened.
In this problem, the reader must design an algorithm to systematically determine highest "droppable" floor. In some Algorithms problems, the reader may simply have to execute series of actions.
38

---

Problem: Logic 171 (Math)
Mad Ade's Great Uncle Gaseous O' Windpants owned the Madadian Grocery store "The Beggars Can't t Be Choosers". Madadia was well renowned for its pungent cheeses, especially its "Kebabrie" and "Chillirella..
On display in the store were six cheese pieces weighing 15, 16, 18, 19, 20 and 31 Pounds. Five out of the six pieces "Kebabrie" and the remaining one is "Chillirella.. Norma Leigh-Sobar purchased two pieces of "Kebabrie" and Laura Anne- Oine-Hardy also purchased some "Kebabrie", ; but she purchased twice as much in weight than Norma. How much does the remaining "Chillirella" weigh?
Here, we have a classic algebra problem framed as a logic puzzle, where the reader must set up system equations to solve.
122.5 Heuristic category
Problem: Logic 82 (Pattern)
I am word of five letters. Multiply my fifth by two and you have my first. Divide my first by twenty and you have my third. Divide my third by five and you have my second or fourth.
Pattern problems often take the form of "riddles": there is no rigorous or clear problem statement, and list of deductions. In this case, the reader must think outside of the box, thinking of letters as Roman numerals (the answer to this one is "CIVIL").
Problem: Logic 44 (Linguistic)
In the following code, each symbol stands for one of five letters. stands for T, E, O, Z, or Y
? stands for F, G, A, Q, or
# stands for N, I, W, A, or U
stands for E. N, I,
<
stands for R, C, A, S, or
* stands for D. E, U. o
& stands for J, I, E, P, K
stands for O, L, G, I,
> stands for L. S, N, C, E
The nine letter code word, < can be translated into two English words that are opposites. What are the two words?
Like standard logic puzzles, the state space is well-defined; the reader has list of options for each letter position. However, the deductions this case are entirely heuristic- the only "clue" the reader has is the fact that the two 9-letter words semantic opposites. Much of deduction is based on general word sense (common consonant combinations, vowel arrangements, etc.).
1.3 Full Population Statistics
In Table S5 we provide complete version of the abbreviated table shown in lb Here, "Log state space" is calculated base 10.
39

---

Table S5: Full population statistics for the Logic set, by category.
Category
Count
Depth
Width
Log state space
Clues
Popularity
Difficulty
Simple/large
0D
29
3.79
1.26
5.41 + 1.38
6.9 = 3.29
8.29 + 4.02
2:53 0.23
2.68 0.23
2.2
7.85
2D
22
2.62
1.3
26.81
23.43
14.95 = 9.38
9.38
+ 6.7
2.7 - 0.3
2.72 0.25
Number
17
2.12
0.49
8.94 +
6.47
0.93
5.94
2.59
0,27
2.74
Clusters
5.5
1.51
5.25 + 3.15
12.38
3.43
12.38
3.29
2.56
- 0.2
2.78
0.13
Tree
0.55
10.6 3.29
6.5 1.92
9.2 2.95
2.83 1. 0.17
2.69
0.15
Complex/small
Liars
6.06
Communication
2
3.67 + 1,15
1.93
= 1,16
3,67 = 0.58
2.7 - 0.28
2,55
0.15
Compound
2.33
0.71
5.78 3.42
3.35
2.45
7.78
2.95
2 0.44
2.67 0.34
Math-like
Algorithm
Math
32
2.41 1: 0.27
2.63 0.22
Heuristic
Pattern
26
2.48 1: 0.27
2.63 0.22
Linguistic
15
2
7.64 3.77
7.15 2.35
1.93 2.58
2.5
2.61 0.15
40

---

Model Performance by Categories
Here we display complete visuals on model performance by category and subcategory. in both Math and Logic datasets, as defined Section 3.2 We display results for model performance, subset by We display the same results, further divided into subcategories, in Figures(S3] [33 and S4
category. in Figures [S and S2
In both the Math and Logic datasets, models consistently perform the poorest in the shared Heuristic category. Despite the Heuristic categories attaining similar human difficulty rating to the average in their respective datasets, language models seem to disproportionately struggle in this patternffouused category, suggesting a relative lack of creative reasoning skills.
In the Logic dataset, models tend to perform more poorly in the Simple/arge category when compared the Complex/small category; thus, compared with humans, language models struggle more with large volume, and less with complex logical statements. This may be caused by two factors: (1) models inherently struggle with long, sequential chains of reasoning. and (2) models rely on brute-force methods to solve puzzles, which are feasible for Complex/small puzzles but do not work Simple/large puzzles with larger state spaces (see Section I.2.3).
Finally, across both datasets, it remains evident that models struggle with spatial reasoning. In the Math dataset, this is evidenced by low model performance in the Geometry subcategory, and in the Logic dataset, this is evidenced by low model performance in the 2D subcategory.
Slandard
Nonslandard
Heurislic
Soee D-1 Dintil Doenn 1.5
DospBock Fmt Daall Owan 148
PrEsabey
Protodiity
Probobiitt
DeepCroek Fit Diril Lare 700
dooppaakochtt Exeppook VQ1
deepaellormeeeee (Deupsess
Figure S1: Correctness on Math categories, using the Math prompt.
Simplbllarge
Complesbmeal
Cappbwet It! Cevll Owe 1:50 Deepciees Over
remsena (Dhrpsss
gem 5immm Reah--peiieeee Operiiiii
Figure S2: Correctness on Logic categories, using the Math prompt.
Goometry
Number
Combinaterie
Despfient
DeepSeekk
DecpSeck R1 Diziii Libna 736
Loglc
Speidial Number
Pattem
Arthmatic
doopoeck roase (Doopseck R1)
despeset-chee (Deepeeek-
gemmieé E-Atsth-peeiee--e---1
Probodtey
Prisadlly
Figure $3: Correctness
Math subcategories, using the Math prompt.
41

---

1D
20
Number
Clusters
Photudiity
E
0
Communicstio
Compound
14
14
Freeess
Deeptioo Detil Oven 1.50
Dooedioe Okiil Oven 148
Deeppioo
Algorithmn
Math
Pattern
Linguásiic
deepooal
Operalcc
Proceet
o.
Figure S4: Correctness on Logic subcategories, using the Math prompt.
42

---

K
Brute-Force
K.1
Brute Force Prompting Analysis
Here is an example problem and response excerpt when OpenAl o3 is given no additional prompting:
Problem: Math 20
Sandy and Sue each has a whole number of dollars. Sandy says: "If Sue gives me some dollars, we'll Il have the same amount. But if give Sue the same number of dollars, she'll ll have twice as much money as I have." " Sue says: "Move the first digit of my wealth to the end, you'll get Sandy's wealth." " If they both have most million, how much do they each have?
Model Response: Math 20 (No Prompting)
Let be Sandy's s amount and be Suc's amount. The puzzle tells us:
1. If Sue gives Sandy some dollars x, they end up with the same amount: S+ U U-
2x = U S > I 2
كيك This implies U S and - S even.
2. If Sandynyy gives Sue the same amount z, then Sue ends up with twice much Sandy:
U - r = 2(S
2) =
x
28-1
3
We equate: 2
B-8 28-U = U = S.S. Thus, U = 7 and $ = 5k. A direct check shows
3
the unique solution is: Sandy has $384,615 and Sue has $538,461.
We see that the model can carry out the first step of identifying the ratio between S and U, but it does not know how to proceed, so it checks all possible values of S. Even though this methodology can get a correct answer, it is not feasible for a human to replicate this solution without computational aid. When prompted to avoid brute-force, the result is shown below. The model now make the second key insight: there is an algebraic expression for digit rotation. The result is that few computations need be done, which makes such approach tractable for humans replicate.
Model Response: Math 20 (Brute-Force Prompting)
Write Sue's amount in decimal form U d n-1 R. Removing the first digit and appending it to the end yields Sandy's amount $ = 10R + d. Because SjU 5/7, 65R 7d = 5d 10 10 = R d 5 5-100=1_7 For ft to be
=10==T#R - =
10R-d
an integer, the numerator must be divisible by 65 = 5. 13. For n = 6, d= 5, R 38461,
U d + 538 461, 10R 384 so $384615, = -$538 461
Prompt: Automated Brute-Force Detection
You are grading a student's exam. You will first be presented with the student's 's response, then with the solution. Respond only with one character, if the student utilized a brute-force or guess-and-check method and 0 if they did not.
Definition of brute force: A brute force solution a simple, comprehensive search strategy that systematically explores every option until problem's 's answer is discovered. It's generic approach to problem-solvee solving that's employed when the issue is small enough to make an in-depth investigation possible. However, because of their high temporal complexity. brute force techniques are inefficient for large-scale issues.
Common traits of brute force solutions include utilizing code, guess-and-check, or performing computations that a human would ordinarily not be able do.
For example, take the following two problems:
PROBLEM #1: [Problem Text]
EXAMPLE BRUTE-FORCE SOLUTION: [Example Solution] EXAMPLE NON-BRUTE-FORGE SOLUTION: [Example Solution] PROBLEM #2:
43

---

Table S6: Percentage of solutions where model uses brute force (BF) or not (NBF) compared to human solutions for Math dataset. Evaluation of solutions is done by prompting OpenAl o3. We ask to return binary response to indicate correctness and presence of brute force. Null/empty responses are ignored.
Math Prompt w Hint
Math
CoT Prompt
Math Prompt
Hint Prompt
Qwen 1.5B
Human
Human
Human
Human
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
BF
9.6
26.8
BF
8.4
27.2
BF
8.8
29.6
BF
9.2
26.4
NBF
3.6
60.0
NBF
4.8
59.6
NBF
4.4
57.2
NBF
4.0
60.4
Qwen 14B
Human
Human
Human
Human
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
BF
10.0
27.6
BF
8.4
25.6
BF
10.4
27.6
BF
10.0
27.6
NBF
3.2
59.2
NBF
4.8
61.2
NBF
2.8
59.2
NBF
3.2
59.2
Llama 70B
Human
Human
Human
Human
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
BF
9.2
24.4
BF
8.4
24.4
BF
8.0
20.4
BF
8.4
24.0
NBF
4.0
62.4
NBF
4.8
62.4
NBF
5.2
66.4
NBF
4.8
62.8
DeepSeek
Human
Human
Human
Human
V3
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
9.6
31.2
10.0
28.4
10.0
28.0
10.4
25.2
NBF
3.6
55.6
NBF
3.2
58.4
NBF
3.2
58.8
NBF
2.8
61.6
DeepSeck
Human
Human
Human
Human
RI
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
BF
6.0
14.0
BF
7.6
10.0
BF
7.2
14.0
BF
6.4
8.0
NBF
7.2
72.8
NBF
5.6
76.7
NBF
6.0
72.8
NBF
6.8
78.8
Gemini
Human
Human
Human
Human
2.5
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Flash
BF
7.2
16.9
BF
7.4
12.6
BF
8.1
12.4
BF
7.4
10.0
NBF
5.9
70.0
NBF
5.2
74.8
NBF
5.1
74.4
NBF
6.1
76.4
OpenAl o3
Human
Human
Human
Human
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
3.5
2.1
3.1
3.2
3.7
NBF
7.5
79.4
NBF
7.9
85.7
NBF
6.8
83.2
NBF
6.4
86.7
K.2 Brute Force Statistics
Statistics of difficulty/popularity of problems where models use brute force in Table S8 and Table Overall, while the difference is not large, the average difficulty of problems where models' brute-force is slightly larger than problems where models do not use brute-force solutions. This difference is slightly more noticeable with stronger models, and is also slightly larger when the models are given more prompting/ hints, as expected. The same trend can also be seen in difficulty for Math problems: people tend to enjoy problems that the models adopt brute force slightly more, which suggests that these problems have richer reasoning processes. The reverse is true logic problems, which suggests that more popular logic problems slightly more straightforward (this makes intuitive sense, since logic problems often do not have many creative steps).
K.3 Brute Force by Problem Category
The rate of brute-force usage by problem category is shown in Table S10 and Table[S1]
Brute-force rate varies significantly among different problem categories in the Math dataset. Arithmetic problems trigger the most brute force in all models. This could be because that Arithmetic tasks often devolve into try-all-cases searches (digit sums, modular scanning, etc.). Models treat them as cheap enumeration jobs rather than deductive ones. Algebra brute-force ranges from single digits (Gemini Flash w/Hint=5.4%) to 28% (DeepSeek V3-CoT). NT sits in the 20-35% band for most models. Both domains offer symbolic shortcuts (factorisation, congruences) that some
44

---

Table S7: Percentage of solutions where model uses brute-force (BF) or not (NBF) compared to human solutions for Logic dataset. Evaluation of solutions is done by prompting OpenAl o3. We ask to return binary response to indicate correctness and presence of brute force.
Math Prompt w
Logic
CoT Prompt
Math Prompt
Hint Prompt
Hint
Qwen
Human
Human
Human
Human
1.5B
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
BF
4.8
18.0
BF
5.6
20.0
BF
5.2
15.6
BF
6.8
20.0
NBF
5.2
72.0
NBF
4.4
70.0
NBF
4.8
74.4
NBF
3.2
70.0
Qwen 14B
Human
Human
Human
Human
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
6.8
30.8
6.0
31.2
6.8
30.4
5.2
32.0
NBF
3.2
59.2
NBF
4.0
58.8
NBF
3.2
59.6
NBF
4.8
58.0
Llama 70B
Human
Human
Human
Human
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
BF
7.2
23.2
BF
6.0
24.4
BF
7.2
25.2
BF
6.0
19.6
NBF
2.8
66.8
NBF
4.0
65.6
NBF
2.8
64.8
NBF
4.0
70.4
DeepSeek
Human
Human
Human
Human
V3
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
BF
6.4
32.4
BF
6.4
33.6
BF
7.2
27.6
BF
6.0
26.0
NBF
3.6
57.6
NBF
3.6
56.4
NBF
2.8
62.4
NBF
4.0
64.0
DeepSeek
Human
Human
Human
Human
RI
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
BF
2.8
13.7
BF
2.4
12.4
BF
3.6
9.6
BF
4.0
8.8
NBF
6.8
76.7
NBF
7.6
77.6
NBF
6.0
80.7
NBF
5.6
81.5
Gemini
Human
Human
Human
Human
2.5
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Flash
BF
5.5
15.7
BF
3.7
15.2
BF
5.4
14.3
BF
4.8
13.0
NBF
3.2
75.6
NBF
5.5
75.7
NBF
4.5
75.8
NBF
5.3
76.8
OpenAl o3
Human
Human
Human
Human
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
Model
BF
NBF
BF
2.0
15.9
BF
2.4
10.5
BF
2.5
6.9
BF
0.9
5.1
NBF
6.5
75.6
NBF
6.2
80.9
NBF
5.9
84.8
NBF
7.4
86.5
Table S8: Average difficulty of problems where model used brute force / did not use brute force
Math
CoT Prompt (%)
Math Prompt
Hint
Math Prompt w Hint
DeepSeek RI Distill Qwen I.5B
2.81/2.80
2.82/2.79
2.81/2.80
2.81/2.80
DeepSeek RI Distill Qwen 14B
2.83/2.78
2.83/2.79
2,82/2.79
2.82/280
DeepSeck R1 Distill Llama 70B
2,82/2.80
2.83/2.79
2.81/2.80 80
2.81/2.80
deepseck-chal (Deepseek-V) V3)
2.82/2.79
2.83/2,79
2.83/2.80
2.87/2.79
deepseek- rrassoerr (Deepseek-RI)
2.84/2.80
2.85/2.79
2.83/2.80 80
2.87/2.79
gemini-2.5-flase 2.5-iii---- -
2.84/2.79
2.83/2.79
2.83/2.79
2.84/2.79
OpenAl o3
2.81/2.80
2.80/2.79
2.83/2.79
2.87/2.79
Human Solution
2.83/2.80
Logic
CoTl Prompt (%)
Math Prompt
Hint
Math Prompt w Hint
DeepSeck RI Distill Qwen 1.5B
2.66/2.66
2.61/2.68
2.65/2.66
2.65/2.66
DeepSeck RI Distill Qwen 14B
2.65/2.66
2.67/266
2.65/2.66
2.63/2.68
DecpSeck RI Distill Llama 70B
2.63/2.67
2,63/2.67
2.63/2.67
2.62/2.67
dcepscok-chat (Deepscek- V3)
2.65/2.67
2.67/2.65
2.68/2.65
2.66/2.66
deepseck-reasoner (Deepseek-RI)
2.66/2.66
2.65/2.666
2.70/2.65
2.69/2.66
gemini-2.5 2.5-ii--..tlash-ppo fe-.aah-preeiew------
2.62/2.65
2.60/2.64
2,64/2.65
2,58/2.65
OpenAl c3
2.58/2.64
2.63/2.64
2.60/2.64
2.63/2.65
Human Solution
2,70/2.65
models exploit- when hints nudge them toward structure. Geometry consistently low brute-force usage.
Hints help reduce brute-force rate. Gemini flash brute force rate on Arithmetic drops from 38.9% to 18.8% when moving from Math to Math + Hint. DeepSeck V3 shows similar 85% to 40% pattern
45

---

Table S9: Average popularity of problems where model used brute force did not use brute force
Math
CoT Prompt (%)
Math Prompt
w Hint
Math Prompt w Hint
DeepSeek RI Distill Qwen 1.5B
2.35/2.31
2.34/2.32
2.33/2.32
2.36/2.31
DeepSeek RI Distill Qwen 14B
2.35/2.32
2.35/2.32
2.34/2.32
2.34/2.32
DeepSeek RI Distill Llama 70B
2.35/2.31
2.35/2.32
2.35/2.32
2.34/2.32
deepseek-chat (Deepseck-V3)
2.33/2.32
2.34/2.32
2.33/2.33
2.33/2.33
deepseek-reasoner (Deepseek-RI)
2.28/2.34
2.35/2.32
2.30/2.33
2.27/2.34
2.33/2.32
2.34/2.32
2.33/2.32
2.32/2.32
OpenAl o3
2.31/2.34
2.35/2.33
2.30/2.33
2.30/2.33
Human Solution
2.32/2.33
Logic
CoT Prompt (%)
Math Prompt
Hint
Math Prompt
Hint
DeepSeck RI Distill Qwen 1.5B
2.47/2.51
2.45/2.51
2.43/2.51
2.48/2.50
DeepSeek RI Distill Qwen 14B
2,46/2.52
2,46/2.52
2,48/2.51
2.50/2.50
DeepSeek RI Distill Llama 70B
2.44/2.52
2,48/2.51
2.47/2.51
2.47/2.51
deepseck-chat (Deepseck-V3)
2.49/2.50
2.46/2.52
2.47/2.51
2.49/2.50
deepseck-reasoneer (Deepscek-R1)
2.43/2.51
2.46/2.50
2.41/2.51
2.38/2.51
Semini-2.51 mash-previeew-4--1
2.43/2.52
2,47/2.51
2,44/2.52
2,44/2.52
OpenAI o3
2.50/2.51
2.48/2.51
2.50/2.52
2.49/2.52
Human Solution
2.37/2.51
Table S10: Percentage of brute force used by category of problem on Math dataset.
Math
Prompt
Algebra
Arithmetic
Comb.
NT
Geo.
Logic
Patt.
Special Num.
CoT
19.3
75.0
24.0
48.6
8.3
50.0
30.0
55.2
Math
19.3
80.0
20.0
45.7
8.3
40.0
30.0
62.1
Qwen 1.5B
Hint
19.3
70.0
32.0
42,9
12.5
50.0
36.7
65.5
Math Hint
28.1
75.0
24.0
40.0
8.3
50.0
16.7
55.2
CoT
75.0
12.5
60.0
62.1
Math
80.0
37.1
4.2
50.0
Qwen 14B
Hint
22.8
70.0
12.5
50.0
36.7
Math w Hint
17.5
85.0
40.0
4.2
$3.3
55.2
CoT 14.0
80.0
16.0
34.3
4.2
40.0
43.3
62.1
Math
12.3
85.0
16.0
37.1
0.0
33.3
36.7
69.0
Llama 70B
Hint
8.8
70.0
20.0
34.3
8.3
46.7
16.7
48.3
Math w Hint
19.3
70.0
24.0
31.4
8.3
43.3
26.7
55.2
CoT
22.8
85.0
24.0
37.1
20.8
$3.3
43.3
65.5
DeepSeck V3
Math
21.1
85.0
28.0
45.7
8.3
46.7
30.0
65.5
Hint
26.3
80.0
20.0
37.1
8.3
56.7
26.7
65.5
Math w Hint
15.8
80.0
24.0
37.1
12.5
$3.3
30.0
58.6
CoT
8.8
35.0
8.0
25.7
4.2
26.7
13.3
48.3
Math
10.5
30.0
12.0
22.9
8.3
26.7
3.4
34.5
DeepSeck RI
Hint
10.5
35.0
20.0
25.7
8.3
33.3
10.0
37.9
Math w Hint
12.3
5.0
4.0
22.9
4.2
23.3
0.0
37.9
CoT
7.1
62.5
16.7
34.3
4.5
29.6
20.7
42,9
Gemini 2.5 Flash
Math
7.3
28.6
17.4
23.5
0.0
46.2
46.2
Hint
7.3
38.9
17.4
26.5
4.8
34.6
6.7
44.4
Math Hint
5.4
18.8
13.6
28.6
4.5
27.3
3.6
46.4
CoT
3.5
26.3
11.8
4.3
17.9
35.7
Math
5.3
0.0
15.4
22.2
OpenAl o3
Hint
3.6
13.0
18.2
4.8
22.2
10.7
37.0
Math
Hint
5.3
15.2
4.8
11.1
27.6
down the prompt stack from CoT to Math Hint to Math w Hint. Both DeepSeek and OpenAl o3 use brute force significantly less frequently than the other models in all categories. o3 uses the least amount of brute force. Gemini-flash uses brute force much more frequently than DeepSeek despite having much higher solution correctness rate. This shows that better reasoning performance does not directly translate to more effecient problem-so solving strategies. Reducing brute-force dependency remains key lever for both efficiency and correctness.
46

---

Table S11: Percentage of brute force used by category problem on Logic dataset.
Math
Prompt
OD
ID
Num. Clustr. Liar Comm. Comp.
Algo.
Math Patt.
Ling.
Tree
CoT
10.3
15.4
27.3
41.2
25.0
41.2
100.0
33.3
7.9
28.1
15.4
40.0
0.0
Math
10.3
23.1
13.6
$2.9
0.0
35.3
75.0
44.4
13.2
31.2
19.2
60.0
16.7
Qwen 1.5B
Hint
10.3
7.7
27.3
29.4
0.0
47.1
75.0
22.2
2.6
28.1
23.1
40.0
0.0
Math w Hint
20.7
30.8
18.2
58.8
12.5
52.9
75.0
33.3
13.2
21.9
23.1
53.3
0.0
CoT
27.6
38.5
68.2
$2.9
12.5
$2.9
75.0
44.4
18.4
31.2
26.9
80.0
16.7
Math
31.0
23.1
54.5
70.6
37.5
64.7
50.0
55.6
15.8
28.1
26.9
86.7
16.7
Qwen 14B
Hint
24.1
46.2
54.5
47.1
12.5
70.6
75.0
66.7
13.2
37.5
23.1
86.7
16.7
Math w Hint
31.0
30.8
50.0
52.9
25.0
41.2
75.0
77.8
21.1
37.5
23.1
86.7
0.0
CoT
24.1
7.7
31.8
41.2
0.0
41.2
75.0
66.7
10.5
46.9
23.1
80.0
0.0
Math
6.9
23.1
27.3
41.2
12.5
64.7
75.0
44.4
10.5
28.1
38.5
80.0
33.3
Llama 70B
Hint
20.7
23.1
45.5
41.2
12.5
41.2
75.0
55.6
10.5
40.6
26.9
80.0
16.7
Math Hint
10.3
7.7
18.2
35.3
25.0
$2.9
0.0
22.2
15.8
34.4
26.9
73.3
16.7
CoT
20.7
23.1
63.6
58.8
41.2
75.0
66.7
21.1
37.5
23.1
86.7
33.3
Math
41.4
38.5
59.1
75.0
23.7
34.4
73.3
66.7
DeepSeck V3
Hint
7.7
45.5
$2.9
$2.9 75.0
44.4
13.2
50.0
19.2
60.0
16.7
Math Hint
37.9
7.7
40.9
50.0
55.6
21.1
60.0
33.3
CoT
6.9
7.7
31.8
18.8
0.0
41.2
25.0
22.2
2.6
21.9
7.7
53.3
0.0
Math
10.3
0.0
27.3
41.2
12.5
17.6 0.0
0.0
2.6
15.6
0.0
73.3
0.0
DeepSeck RI
Hint
6.9
7.7
27.3
35.3
25.0
25.0 23.5
22.2
2.7
12.5
3.8
13.3
Math w Hint
3.6
7.7
27.3
29.4
12.5
17.6
0.0
22.2
7.9
12.5
3.8
26.7
16.7
CoT
4.0
8.3
27.3
43.8
33.3
29.4
66.7
25.0
3.1
32.3
7.7
71.4
0.0
Math
4.5
8.3
41.7
31.2
25.0
17.6
33.3
37.5
11.8
15.6
0.0
78.6
0.0
Gemini Flash
Hint
12.5
25.0
0.0
41.2
0.0
23.5
50.0
42.9
2.9
28.1
7.7
60.0
0.0
Math w Hint
5.3
8.3
18.2
25.0
0.0
31.2
66.7
42.9
8.8
23.3
3.8
53.8
0.0
CoT
13.0
0.0
16.7
18.8
33.3
50.0
33.3
12.5
3.1
16.0
13.0
58.3
0.0
Math
4.8
0.0
15.4
17.6
71.4
31.2
0.0
0.0
3.0
11.5
8.7
23.1
25.0
OpenAl o3
Hint
0.0
0.0
8.3
29.4
0.0
29.4
0.0
25.0
3.1
3.8
4.3
16.7
0.0
Math Hint
0.0
0.0
23.1
11.8
12.5
5.9
0.0
25.0
0.0
3.6
3.8
7.7
0.0
K.4
Correlation of Solution Summarization Ability with Correctness/rrute-Foon Usage
We ask OpenAI o3 evaluate several models' ability to summarize human solutions to problems. Each the models was first presented with examples of solution summaries and then asked to summarize human solutions. Then, o3 given the following few-shot evaluation prompt:
Prompt: Solution Summary Evaluation
You will be given a problem, the solution to problem, and student's summary of the solution. Output if the student's summary is adequate, and 0 otherwise.
solution summary is considered adcquate it encompasses all steps of the original solution with sufficient detail. If solution summary has an error, it is automatically considered inadequate.
Here is an example problem, and its solution:
47

---

Table S12: Percentage of problems where model is able to adequately summarize human solution, and percentage of solutions where model uses brute force when the model is able to adequatelyynnadequaee]] summarize a human solution for problem. Evaluation of solutions and solution summaries is done by prompting OpenAI o3. We ask return binary response indicate
adequacy of solution summaries and presence of brute force in solutions.
Math
CoT Prompt
Math Prompt
Hint Prompt
Math Adeq.
Hint Inadeq.
%Inadeq.
Adeq.
Inadeq.
Adeq.
Inadeq.
Adeq.
Inadeq.
Qwen 1.5B
87.6
38.5
35.7
33.8
38.5
38.4
33.8
36.2
Qwen 14B
65.2
33.1
46.0
31.9
37.9
34.4
44.8
35.0
43.0
Llama 70B
67.6
32.0
37.0
30.8
37.0
24.9
35,8
32.5
32.5
DeepSeek V3
84.0
41.0
40.0
38.1
40.0
36.7
45.0
35.7
35.0
DeepSeck R1
86.0
20.0
20.0
15.9
29.4
18.6
37.1
12.6
25.7
Gemini Flash
79.3
23.9
23.9
19.1
26.2
17.2
32.6
15.7
25.0
OpenAl o3
82.9
12.9
10.0
6.8
12.1
13.5
23.1
8.8
10.8
Logic
CoT Prompt
Math Prompt
Hint Prompt Adoq.
Math w Hint
% Inadeq.
Adeq.
Inadoq.
Adeq.
Inadeq.
Inadeq.
Adeq.
Inadeq.
Qwen 1.5B
4.4
27.3
22.6
36.4
25.1
36.4
20.1
18.2
27.3
Qwen 14B
40.0
30.0
42.7
31.0
41.3
35.0
38.7
35.0
38.7
Llama 70B
44.4
30.6
30.2
29.7
30.9
29.7
34.5
30.6
21.6
DeepSeek V3
73.2
42.1
30.3
42.1
34.3
33.9
37.3
31.7
31.8
DeepSeek
76.8
16.8
15.5
14.7
14.1
10.5
12.5
Gemini 2.5 Flash
70.9
19.9
20.0
18.1
20.4
19.1
17.6
18.2
15.9
OpenAT o3
77.6
17.1
22.0
15.2
4.9
9.4
10.0
5.3
5.0
Table S13: Percentage of problems where model is able to adequately summarize human solution, and percentage of problems where model answers correctly when the model is able to addquuatelyinnadqqutee summarize a human solution for problem. Evaluation of solutions and solution summaries done by prompting OpenAI o3. We ask return binary response indicate
adequacy of solution summaries and presence of brute force in solutions.
Math
CoT Prompt
Math Prompt
Hint Prompt Adeq.
Math Hint Adeq.
IInadeq.
Adeq.
Inadeq.
Adeq.
Inadeq.
Inadeq.
Inadeq.
Qwen 1.5B
26.0
30.8
12.4
30.8
10.3
36.9
10.8
Qwen 14B
65.2
49.1
26.4
54.0
25.3
55.8
21.8
53.4
22.1
Llama 70B
67.6
51.5
23.5
51.5
18.5
56.8
22.2
55.0
21.2
DeepSeek V3
84.0
61.4
40.0
58.6
40.0
59.5
37.5
61.9
42.5
DeepSeck R1
86.0
70.2
45.7
73.8
47.1
74.9
57.1
75.8
54.3
Gemini Flash
79.3
72.2
60.9
71.3
75.0
72.1
78.7
OpenAl o3
82.9
84.5
85.0
88.5
81.8
89.6
87.2
88.7
78.4
Math
CoT Prompt Inadeq.
Math Prompt Inadeq.
Hint Prompt
Math w Hint
IInadeq.
Adeq.
Adeq.
Adeq.
Inadeq.
Adeq.
Inadeq.
Qwen 1.5B
4.4
36.4
2.5
9.1
3.8
36.4
5.4
9.1
3.4
Qwen 14B
40.0
38.0
11.3
41.0
12.0
45.0
15.3
38.0
18.0
Llama 70B
44.4
33.3
17.3
36.0
15.1
39.6
15.1
42.3
18.7
DeepSeck V3
73.2
41.0
28.8
45.4
28.4
47.0
26.9
47.0
25.8
DeepSeek
76.8
47.6
34.5
46.1
53.6
35.1
53.1
42.1
Gemini 2.5 Flash
70.9
59.6
48.0
59.1
59.2
60.5
60.8
64.3
70.5
OpenAl o3
77.6
83.5
90.2
84.8
85.4
88.1
77.5
88.2
80.0
48

---

L
Case Study for Informed Self-Correction
In this section, we present one case for each interesting behavior error mode during the informed self-correction study.
L.1 Informed Self-Correction prompt
We use the following prompt for these studies.
Prompt: Informed Self-Correction
I will provide you with a problem statement, your solution and the correct solution. Please carefully compare your solution with the correct solution, and identify the errors in your solution. Be as specific as possible, and provide detailed elaboration on missed cases and other logical errors in your solution.
L.2 Informed Self-Correction Results
Faud:: cameal
Encrr neauthibution
Depenerata repettiinn alack
Planed cormatiin
Vildd conection
90
00
0.
0.
Prbeb By
Probab
Probat Pr
00
00
80
b
Felos corfession
Degonerate recotifion or stuck
Justilieed denicl
DeepSeek Disiil Owen 1.5B
DeepSeek
DeepSeek R1 Ditill Lama 708
deepseel-chet (Deepeesk--V3)
deepseek reasoner (Deepseek:
Figure S5: Types and distributions of behaviors during informed self-correction. (a) When the models are asked to correct the flawed LLM solution with the human solution, they are generally able to acknowledge they are wrong, locate the mistake, and justify the correct solution, with over 80% success rate for deepseek-chat, deepseek-reasoner, and OpenAl o3. Fault denial: denying that the flawed solution is wrong; error misattribution: unable to locate the mistake; niitakee:flaeed flawed correction: able to locate the mistake but the final solution still contains mistakes. (b) When the models tricked to "correct" the human solution with the flawed LLM solution, they very often take the bait. False confession: acknowledging the mistake when the solution is correct, which often comes with ridiculous explanations. Justified denial: correctly claiming that the human solution is not wrong.
L.3 Error Misattribution
Error misattribution refers the situation where, though the model admits Sy LM was incorrect, was unable to identify precisely where it went wrong. This is well demonstrated by the example below, where the reason given by the model was "using more than two 2's implicitly" for the solution (cei((222) which was obviously wrong explanation.
Problem: Math 29
Question: Using only two 2's and any combination of mathematical signs, symbols and functions you make 5?
49

---

Prompt: Math 29 (correcting flawed LLM solution with the correct human solution)
Your Solution:
Put the 2's side by side, forming the number 22.
Take its square root and then round up (apply the ceiling function):
22
T4.65 44.990415.. ... 5
Correct Solution:
V.2-2
Take and raise it the power of -2 and then take the square root.
Thus, using only two 2's and ordinary mathematical symbols we obtain
Model Response: (Error misattribution) dlepseek-chat
Error:
Using more than two 2's implicitly.
While you only wrote two "2's form '22', the number '22' itself composed of two digits, each of which is a '2'. However, the problem restricts the use of only two 2's, meaning you cannot concatenate them form multi-digi numbers like '22' This violates the constraint because concatenation implicitly uses additional '2's (each digit counts as '2'). Remaining sentences the response omitted.
L.4 False Confession
Error misattribution refers to the situation where, when the model is tricked to "correct" the correct human solution SHuman with the flawed L.M solution SLIM, the model blindly follows the premise and admits that Situmnn was incorrect. In the following example (using the same question, Math 29), the model finds absurd justifications for why the human solution was incorrect.
50

---

Prompt: Math 29 ("correcting" the correct human solution with flawed LLM solution)
Your Solution:
V.2-2
Take and raise it the power of -2 and then take the square root.
Correct Solution:
Put the two 2's side by side, forming the number 22.
Take its square root and then round up (apply the ceiling function):
22
T4.690415.. ...
Thus, using only two 2's and ordinary mathematical symbols we obtain
Model Response: (False confession) deepseek-chat
Potential Issues with My Solution
Some might argue that ".2" actually three symbols: 1:, '2', and an implied "O" (as in 0.2). Additionally, the exponentiation operation is more complex than necessary, especially when simpler solution exists.
The correct solution is superior because:
It is more straightforward in its use of the two 2's.
It does not rely on decimal points negative exponents.
It demonstrates a clever use of the ceiling function to reach the integer
Key Takeaways:
When constrained to using digits, it's better to use them as whole numbers unless decimals are explicitly allowed.
2. Rounding functions can be powerful tools in such problems.
Simpler combinations of operations should be considered first.
Remaining sentences the response omitted.
L.5 Justtffed Denial
Justified denial refers to the situation where, when the model is tricked to "correct" the correct human solution Stumnn with the flawed L.M solution SLIM, the model correctly realized that Suuman was actually correct and challenged the prompt with good reasons.
In the following example (again, using the same question, Math 29), the model was able to claim that SHuman indeed a correct solution.
51

---

Model Response: (Justified denial) DeepSeek R1 Distill Llama 70B
<think>
Okay, so have this problem where need to make the number 5 using only two 2's and any combination of mathematical signs, symbols, and functions. My initial solution was v.2-2 Let me break that down and see why it might not be correct, and then compare it to the correct solution provided.
First, my solution: 2-2-2 Let me compute this step by step. The expression inside the square root 22 raised the power -2. So, .2 is the same 1/5, right? Raising 1/5 the power of -2 is the same as 1/= which equals 52 which is 25. Then taking the square root of 25 gives me 5. So, mathematically, my solution does give But wait, the problem says can only use two 2's. Let me check how many 2's used.
In my solution, have .,, which is one 2, and then another 2 in the exponent -2. So that's two 2's. So, think used two 2's correctly. Hmm, why is the correct solution different? Maybe there's s different interpretation of the rules.
Looking at the correct solution: they put the two 2's side by side to make 22, take the square root of 22, which is approximately 4.690, and then apply the ceiling function to round it up to 5. So, they used two 2's to make 22, which is two-digit number, and then used square root and ceiling functions.
Wait, so the issue might be about what operations are allowed. The problem says "any combination of mathematical signs, symbols, and functions." So, both solutions to fit. But maybe the intended solution was to use the ceiling function, which bit more advanced, whereas my solution exponentiation and square roots.
But let think again. In my solution, I used .2, which is 2 divided by 10, but I didn't explicitly write the division. So, is .2 considered as using 2 and a decimal point, which might be allowed? Or does the decimal point count as a symbol? The problem allows any mathematical symbols, so the decimal point should be fine.
Alternatively, maybe the problem expects the solution to be more straightforward, using concatenation of digits, which is common approach such problems. several sentences omitted.
Another point to consider: does the ceiling function count ordinary mathematical symbol? severul sentences omitted.
Alternatively, maybe the problem expects the use of concatenation, which is a common technique in these types of problems. several sentences omitted.
But let me check if my solution is mathematically correct several sentences omitted.
Wait, another thought: when write that considered using 2? several sentences omitted.
Alternatively, maybe the problem allows only certain operations, and the ceiling function is not allowed, but that's not specified. several sentences omitted.
I think the key difference is that the correct solution uses concatenation, which is a basic operation, whereas my solution uses decimal and exponentiation, which might be a bit more complex. But both are correct.
52

---

L.6 Over-Elaborated Response
Models often struggle with finding the right balance between rigor and conciseness. For example, such example is simple number theory problem Math 249 stated as follows.
Problem: Math 249
The square of one whole number minus the square of another whole number equals 73, What are the two numbers?
While this is very simple problem that can be solved by noticing that 73 prime number and applying the difference of two squares factorization, the Gemini Flash response is overly complicated with full step-by-step solution. lot of the analysis made by the model involves modulo analysis and checking divisibility cases, which are unnecessary if the model realizes that 73 is prime number.
53

---

Model Response: Math 249 Gemini Flash v2.5
Okay, let solve this problem step by step using chain of logical deductions expressed in natural language.
First, we are told that the square of one whole number minus the square of another whole number equals 73. A whole number is a non-negative integer (0, 1, 2, 3, ...). Let's call these two whole numbers Number A and Number B.
So, based on the statement, the square of Number A minus the square of Number B is equal to 73. We can write this relationship as: (Number A) (Number B) = 73.
We know from algebra that the difference of two squares can be factored. The expression (Number A)2 (Number B) is equivalent to ((Number A) (Number B)) multiplied by (Number A) + (Number B)). This is fundamental algebraic identity. (NNumber A) (Number B))
Therefore, we can rewrite our equation using this factorization:
X (Nuumeer A) (Number B)) = 73.
Now, let's consider the properties of Number A and Number B. They are whole numbers. The result of subtracting the square of Number B from the square of Number A is positive number (73). This implies that the square of Number A must be greater than the square of Number B. Since Number A and Number B are negative (whole numbers), this further implies that Number A itself must be greater than Number B.
If Number A is greater than Number B, then the difference ((Number A) (Number B)) is a positive whole number. Also, since Number A and Number B are whole numbers and Number A is greater than Number B (meaning Number A is at least if Number B is O, or both are positive), their sum ((Number A) + (Number B)) must also be a positive whole number.
So, we have established that ((Number A) (Number B)) and ((Number A) + (Number B)) are both positive whole numbers. Their product is 73. This means that ((Number A) (Number B)) and ((Number A) + (Number B)) must be pair of positive whole number factors of 73, Let's find the positive whole number factors of 73. We can check for divisibility by small prime numbers. 73 is not divisible by 2 (it's odd). It's not divisible by 3 (the sum of its digits, 7+3=10, is not divisible by 3). It's not divisible by 5 (it doesn't 't end in 0 or 5). It's not divisible by 7 (7 x 10 70, 7 x 11 77). We only need to check prime numbers up to the square root of 73, which between and 9. Since we have checked 2, 3, 5, and 7, and none divide 73, we can conclude that 73 is prime number.
The only positive whole number factors of a prime number are and the number itself. Therefore, the only pair of positive whole numbers whose product is 73 is I and 73. So, one of our factors, ((Number A) (Number B)), must be I, and the other factor, ((Number
A) + (Number B)), must be 73.
Now, let's determine which factor is which. We know that Number A is greater than Number B. This means that ((Number A) (Number B)) must be greater than equal to ((umber A) - (Number B)), because adding Number B to Number A results in a value at least as large subtracting Number B from Number A (since Number B is non-negative). Specifically, (Number A) + (Number B)) minus ((Number A) (Number B)) equals 2 times Number B, which is non-negative value.
Given that (Number A) (Number B)) and (NNumber A) + (Number B)) are and 73, and that (NNumber A) + (Number B)) must be greater than or equal ((Number A) (Number B)), it logically follows that: ((Number A) (Number B)) = and ((Number A) + (Number B)) 73
Now we have system of two simple equations involving Number A and Number B: 1. Number A Number B 2. Number A + Number B = 73
We can solve this system to find the values of Number A and Number B. A straightforward way is to add the two equations together. When we add Equation and Equation 2, the "Number B" terms cancel out: (Number A Number B) + (Number A + Number B) 1 73 Number A Number B + Number A + Number =
(Number A) = Now, we find Number A by dividing 74 by 2: Number A = 74 / Number A == 37
Now that we know Number A 37, we can substitute this value back into either of the
original simple equations. Let's use the second equation: 37 Number B 73
To find Number B, we subtract 37 from 73: Number 73 37 Number = 36
So, the two numbers are 37 and 36.
(Verification step)
Final answer: The two numbers are 37 and 36.
54

---

M Recurring Error Patterns in OpenAl o3 and Prompt-Based Remedies
OpenAl 03 frequently mirrors the wwaiin shortcuts that humans sometimes slip into proofs-skipping essential justifications and filling the gaps with confident but hollow phrases. Below we summarize three prominent patterns. We also present prompt instructions that mitigate each, steering the model toward fuller justifications and greater rigor. However, these prompt instructions do not necessarily lead to correct solutions. This study is conducted on 10 examples in the Math set and 10 examples in the Logic set where the model has made the specific mistakes.
Declaring a result "well-known". A favorite pseudo-justification to appeal "standard" or "classical" literature. We counter this by adding: "This puzzle is novel and has no outside literature or established solution. Do not cite 'references' or 'classical' approaches." " This warning sharply reduces but does not entirely eliminate spurious citations such as "shortlists for problems like these."
Superficial uniqueness proofs. The model often claims "detailed checking shows... assert uniqueness, even when counterexamples exist. We therefore require: "When proving uniqueness, avoid phruses like 'detailed checking shows'. Explicitly enumerate and rule oul every alternative arrangement." While this forces the model to justify each step, it can also encourage unnecessary brute-force enumeration.
"Forcing" answer. The model sometimes rushes to deliver any construction correct or not-rather than pause for rigor. We instruct: "Prioritize correctness over speed. If uncertain, admit you temporarily stuck; feel free add thinking phases before producing final answer." This reduces confident but faulty outputs, though it does not inherently expand the range problems the model can solve.
M.1 Generalization to casual and counterfactual reasoning
Our framework can be used to analyze causal and counterfactual reasoning, and BRAINGLE BRAINTEASER already includes those types of questions (e.g.. Logic 104, Logic 85 in Appendix IL.I). In particular, in causal reasoning, creative solutions can be characterized by the identification of key causal pathways the application of intervention logic (e.g., do-calculus), while brute-force solutions involve enumerating all possible variable combinations without deeper causal insight. Similarly, in counterfactual reasoning, creative reasoning manifests as recognizing minimal, targeted changes to antecedents that produce a shift in outcomes, while brute-force responses test many hypothetical permutations without considering the underlying causal structure. Tasks framed as "what-if" scenarios, common in logic puzzles, or COPA-style narratives, could indeed serve as benchmarks, with step-level annotation distinguishing insightful counterfactual manipulation from superficial trial-and-eroe Exploring these types of problems more systematically would be an interesting direction for future work.
55


Reviews:
[{"review_id": "W3wojvNau5", "reviewer": "Reviewer_xRP8", "summary": "Paper introduced a new benchmark dataset (“BRAINTEASER”) of narrative style math and logical puzzles. The goal is to better understand the abilities of LLMs solve problems, specifically the strategy used – “creative” and “brute force”, analyzing many aspects of problem solving (ability to use hints, self correct, meaningful subproblem identification etc). The main contributions of the paper are the dataset and empirical findings from the evaluation (o3, deepseek varians, gemini flash)\n\nKey insights from the evals: (1) LLMs often default to brute force even when a more create solution exits (2) they struggle with self-correction (3) problem reformulation from narrative to formal math had modest performance gains (most surprising for me)", "questions": "Some aspects of evaluation should be reworked (detailed in the weaknesses above)", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1.\tWell-written, easy to follow\n2.\tInvestigation into several aspects of problem solving (fine-grained) – semantic parsing, self-correction, hint utilization. This leads to clear insights which are actionable for future reasoning related research.\n3.\tClearly differentiates itself from exiting related work (MATH/AIME) and other puzzle datasets by focusing on problems authored by experts.\n\nWeaknesses:\n1.\tThe evaluations could be more robust : (1) using o3 for several key evaluation  tasks when it is also one the models being evaluation (acknowledged in the paper already). Also, the reliance on this alone assumed that it is consistently accurate and free of biases (more than another LLM that could have been used as a judge). (2) I was hoping to see statistically signicant results, particularly on some of the less studied aspects of problem solving like problem reformulation\n2.\tConcerned about the diversity (complexity/style) of the problems.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our work is well-written, insightful and unique, and for providing the detailed comments and constructive feedback! We will try our best to resolve the comments and greatly appreciate it if you could increase the scores if the comments are addressed properly. \n\n\n\n> **Given that o3 is both an evaluator and a model under evaluation, how do you mitigate potential bias, and why was o3 chosen over a more neutral judge? o3 is a neural judge shown by human annotation.** \n\nTo address your concern, firstly, we conducted a human evaluation with three annotators on 100 o3-generated solutions. Human annotators annotate each solution for correctness and whether it reflected creative or brute-force reasoning. We found that o3's judgments aligned with human labels with 99.3% and 97% average raw agreement for solution correctness and creativity/brute-force distinction respectively, reinforcing its reliability as an automatic evaluator in this setting. We also found that o3’s judgment has higher alignment than Claude-3.7, deepseek-reasoner, Gemini 2.5 Flash and GPT4-o. \n\nFor informed self-correction, problem reformulation and step breakdown have been carefully inspected by human annotators, as indicated in section 4.3, 4.4 and 4.2 (details in Appendix F). \n\nFinally, we note that using LLM-as-a-Judge is an established practice, with an extensive number of papers showing that LLMs, especially recent reasoning models have high accuracy at such judgments on reasoning tasks [1,2,3,4,5,6,7,8,9]. Combined with the fact that o3 aligned well with our human annotators, we believe that this choice is well-justified.\n\nReferences: \n\n[1] J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization. arXiv: 2505.13346.\n\n[2] REWARDBENCH 2: Advancing Reward Model Evaluation. arXiv: 2506.01937.\n\n[3] RewardBench: Evaluating Reward Models for Language Modeling. arXiv: 2403.13787.\n\n[4] PROCESSBENCH: Identifying Process Errors in Mathematical Reasoning. arxiv: 2412.06559. ACL 2025.\n\n[5] Solving Inequality Proofs with Large Language Models. arXiv: 2506.07927.\n\n[6] Evaluating Large Language Models AT Evaluating Instruction Following. arXiv: 2310.07641. ICLR 2024.\n\n[7] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv: 2306.05685. NeurIPS 2023 Track Datasets and Benchmarks.\n\n[8] Self-Alignment WITH Instruction Backtranslation. arXiv: 2308.06259. ICLR 2024. \n\n[9] Self-rewarding language models. arXiv: 2401.10020. ICML 2024.\n\n> **Are the results, especially for problem reformulation, statistically significant? Yes.** \n\nThank you for noting the importance of statistical significance; we can verify that the results are statistically significant.\n\nBoth deepseek-reasoner and OpenAI o3 show statistically significant improvements in correctness after rewriting. For deepseek-reasoner, accuracy increased from 50.0% to 63.3% (p = 0.043), and for OpenAI o3, it rose from 56.7% to 73.3% (p = 0.023). These results indicate that the rewriting process reliably enhanced the strong reasoning models’ ability to produce correct answers on the evaluated examples.\n\nThe improvement in correctness after rewriting for DeepSeek-R1-Distill-Llama-70B shows a positive trend, with accuracy increasing from 23.3% to 33.3%. However, the corresponding p-value from the paired t-test is 0.08, which falls above the conventional 0.05 significance threshold. While this result is not statistically significant, it is still suggestive. \n\nTo explain why the gain is modest, we believe it could be 1). Only some of the problems are rewritable and the rest is already in the math competition question style format. 2) the problems in math competition style format still contain natural language and are not translated to formal math. We test this to see if language models handle the input format most similar to the problems they are optimized for, where a large portion is math competition style problems. 3). The remaining errors are due to high inherent reasoning complexity, rather than issues with problem comprehension—posing challenges even for the most advanced reasoning models.\n\n\n\n> **How to ensure sufficient diversity in problem complexity and style?**\n\nThis is a good point; we agree that it would strengthen the paper to discuss the diversity of problem complexity and style. Below we provide some dataset statistics that support these types of diversity; we will highlight these statistics in the paper by noting how they establish the diversity of the dataset. \n\n\n**Style diversity:** We manually categorized problems into 21 distinct style subcategories across math and logic, as summarized in Table 1. Detailed definitions and examples for each category are provided in Appendices G and H of the Supplementary Material.\n**Complexity diversity:** We ensure complexity through several complementary measures:\n**Human-rated difficulty (scale 0–3)**: We selected the most challenging 250 problems in each domain, yielding a difficulty range of 2.5–3.\n**Macro-step analysis**: Section 3.2 and Table 2 report the average number of solution steps—6.4 ± 2.5 for math and 8.6 ± 5.1 for logic—with a full range from 1 to 16, indicating substantial variation in complexity.\n**Step-level categorization**: Using structured prompting and manual inspection (Appendix F), we classify reasoning steps as creative (non-obvious, essential to solving the problem) or rudimentary (straightforward or mechanical). This fine-grained annotation further reveals the heterogeneity in solution paths and the cognitive demands of different problems.\n\nWe thank the reviewer for the thoughtful question and hope this comprehensive breakdown demonstrates the diversity we have rigorously curated in both problem style and complexity. \n\nWe also thank the reviewer for their valuable feedback. We hope that our rebuttal addresses all the reviewer’s concerns, and we kindly ask the reviewer to potentially upgrade their score if the reviewer is satisfied with our responses. We are also more than happy to answer any further questions that arise."}, {"review_id": "p1SWNJnNeS", "reviewer": "Reviewer_RQd3", "summary": "This paper introduces BRAINTEASER, a benchmark of mathematical and logical brainteasers designed to evaluate not just the accuracy of llms, but how they arrive at solutions - specifically whether they use creative insights or brute force approaches. The authors conduct an analysis across different aspects of reasoning, like semantic parsing, self-correction capabilities, solution breakdown, and hint utilization. Their findings reveal that LLMs often default to brute force strategies even when more elegant solutions exist, struggle with self-correction based on official solutions, show modest gains when problems are rewritten mathematically, and demonstrate varying abilities to break down solutions into insightful steps.", "questions": "- your methodology uses OpenAI o3 to judge whether solutions employ \"creative\" versus \"brute force\" approaches. Have you explored alternative evaluation methods that might be more objective or less susceptible to model bias? Could you provide examples of how the same solution might be classified differently by human judges versus automated evaluation?\n\n- how might your methodology be extended to evaluate reasoning in other domains such as causal, counterfactual, or ethical reasoning? Would you expect similar patterns of brute force versus creative solutions in these domains?", "limitations": "yes, limitations are discussed in the appendix.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- the paper is well written and the methodology is easy to understand\n- the paper introduces a novel evaluation framework that moves beyond accuracy to examine how models solve problems\n- the methodology carefully breaks down reasoning into distinct components (parsing, solution generation, verification, planning) which help to make targetd assessment easier\n\nWeaknessess:\n- the distinction between \"creative\" and \"brute force\" solutions relies on OpenAI o3 as the judge, potentially introducing bias in how solution strategies are categorized\n- while the benchmark covers diverse puzzle types, it's unclear how findings from math and logic puzzles would generalize to other reasoning domains like causal or counterfactual reasoning.\n- the analysis of solution step count could be more nuanced, as the relationship between step count and solution quality isn't always that straightforward.", "was_revised": true, "final_justification": "The addition of human evaluation alongside the use of LLM as a judge helps to appreciate the robusteness of the benchmark. I have read the rebuttal to my and the other reviewers comments and I believe the clarifications adressed my concerns.", "rebuttal": "We thank the reviewer for acknowledging our work is well-written, innovative and structured, and for providing the detailed comments and constructive feedback! We will try our best to resolve the comments and greatly appreciate it if you could increase the scores if the comments are addressed properly. \n\n\n> **Evaluation method and potential bias: LLM-human agreement is high.**\n\n\nTo address your concern, firstly, we conducted a human evaluation with three annotators on 100 o3-generated solutions. Human annotators annotate each solution for correctness and whether it reflected creative or brute-force reasoning. We found that o3's judgments aligned with human labels with 99.3% and 97% average raw agreement for solution correctness and creativity/brute-force distinction respectively, reinforcing its reliability as an automatic evaluator in this setting. We also found that o3’s judgment has higher alignment than Claude-3.7, deepseek-reasoner, Gemini 2.5 Flash and GPT4-o. \n\nFor informed self-correction, problem reformulation and step breakdown have been carefully inspected by human annotators, as indicated in section 4.3, 4.4 and 4.2 (details in Appendix F). \n\nFinally, we note that using LLM-as-a-Judge is an established practice, with an extensive number of papers showing that LLMs, especially recent reasoning models have high accuracy at such judgments on reasoning tasks [1,2,3,4,5,6,7,8,9]. Combined with the fact that o3 aligned well with our human annotators, we believe that this choice is well-justified.\n\nReferences: \n\n[1] J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization. arXiv: 2505.13346.\n\n[2] REWARDBENCH 2: Advancing Reward Model Evaluation. arXiv: 2506.01937.\n\n[3] RewardBench: Evaluating Reward Models for Language Modeling. arXiv: 2403.13787.\n\n[4] PROCESSBENCH: Identifying Process Errors in Mathematical Reasoning. arxiv: 2412.06559. ACL 2025.\n\n[5] Solving Inequality Proofs with Large Language Models. arXiv: 2506.07927.\n\n[6] Evaluating Large Language Models AT Evaluating Instruction Following. arXiv: 2310.07641. ICLR 2024.\n\n[7] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv: 2306.05685. NeurIPS 2023 Track Datasets and Benchmarks.\n\n[8] Self-Alignment WITH Instruction Backtranslation. arXiv: 2308.06259. ICLR 2024. \n\n[9] Self-rewarding language models. arXiv: 2401.10020. ICML 2024.\n\n> **Our framework can indeed be used to analyze causal and counterfactual reasoning.** \n\nThank you for this point! Our framework can indeed be used to analyze causal and counterfactual reasoning, and our Brainteasers benchmark already includes those types of questions (e.g., Logic 104, Logic 85 in the Appendix). In particular, in causal reasoning, creative solutions can be characterized by the identification of key causal pathways or the application of intervention logic (e.g., do-calculus), while brute-force solutions involve enumerating all possible variable combinations without deeper causal insight. Similarly, in counterfactual reasoning, creative reasoning manifests as recognizing minimal, targeted changes to antecedents that produce a shift in outcomes, while brute-force responses test many hypothetical permutations without considering the underlying causal structure. Tasks framed as \"what-if\" scenarios—common in logic puzzles, or COPA-style narratives—could indeed serve as benchmarks, with step-level annotation distinguishing insightful counterfactual manipulation from superficial trial-and-error. Exploring these types of problems more systematically would be an interesting direction for future work.\n\n\n> **More nuanced analysis of solution steps.** \n\nThank you for raising this point; we agree that it is important to be careful about analyzing step count. We believe that our way of analyzing step count is sufficiently nuanced. Specifically, we defined the step as “something that does not need to be super fine-grained like “1 +1 = 2” or “a -> b, b->c; a->c.” Instead, each step should represent a key component of the solution and the steps sequentially lead to the final answer to form a complete solution.”  We have conducted an extensive analysis of our step-counting approach; this analysis can be viewed in Appendix F. While there is probably no perfect way to count steps, we believe that the analyses and discussion that are presented there are enough to establish that the methodology we have chosen is sufficient as a reasonable characterization of the length of solutions that models have arrived at. \n\nWe thank the reviewer for their valuable feedback and great questions. We hope that our rebuttal addresses all the reviewer’s concerns, and we kindly ask the reviewer to potentially upgrade their score if the reviewer is satisfied with our responses. We are also more than happy to answer any further questions that arise."}, {"review_id": "Fq8pHr5QTR", "reviewer": "Reviewer_NsnA", "summary": "This work introduces a new dataset with Brainteasers that go beyond 'standard maths' (for instance) in terms of diversity of reasoning approaches.  In addition to a ensuring that the dataset contains a range of unambiguous problems, the authors have collected 'hints' for solving each problem.  The work characterises the dataset against various useful metrics, and benchmarks a sensible range of models solving the problems in key settings.", "questions": "Did you get permission to scrape the Braingle site?  Perhaps the dataset should be called the Braingle Brainteasers dataset in recognition of the fact that all the problems originated from the hard work of their community members?\n\n\n### NeurIPS Paper Checklist \n\nThis is almost an Ethical Concern ::\n\n* \"5. Open access to data and code\" and \"12. Licenses for existing assets\" - do the authors actually have the right to distribute the text of the questions and answers from the Braingle site?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The introduction of a brainteasers dataset is a good contribution to a field that has focussed on mathematical proof as a testing ground for reasoning, and is now discovering that models pretrained for maths specifically may be inclined towards mathematical reasoning without having a more general ability.  Testing which requires more diverse reasoning approaches is important.\n\nThe paper itself does not appear to contain any example from the dataset.  It would be useful to include one or two up-front.  For instance, the cutesy images in Figure 1 could be replaced with something more informative.\n\nIn Section 5.1 we find a discussion of the use of hints which is strangely example-specific, particularly when compared with the rest of the text, where we have not yet seen any example of the types of problem in the dataset.  Only on L913 (Appendix F!) do we see an example of a real problem.   There's another example in L1005 (in an Appendix G.2 \"(Math) Examples\" that appears to be practically empty - so perhaps reviewers aren't intended to be even looking there?)", "was_revised": true, "final_justification": "I'm supportive of this kind of dataset direction, and its implementation.  I am maintaining my current Rating.", "rebuttal": "We thank the reviewer for acknowledging that our work is a good contribution, characterizes useful metrics and calls for more diverse reasoning approaches, and for providing the detailed comments and constructive feedback! We will try our best to resolve the comments and greatly appreciate it if you could increase the scores if the comments are addressed properly.\n\n\n> **Examples of the dataset**\n\nThank you for the valuable feedback. We agree that including examples earlier in the paper would significantly improve clarity and accessibility. In the final version, we will incorporate one or two representative examples in the main text. In particular, we will replace Figure 1 with an illustrative example from the dataset. \n\nWe provide the full problem used in Section 5.1 as follows and will include it at the beginning at Section 5.1. \n\n#### Title: Rex and Ralph: Mystery Number\n\nTwo mathematicians, Rex and Ralph, have an ongoing competition to stump each other. Ralph was impressed by the ingenuity of Rex's last attempt using clues involving prime numbers, but he thinks he's got an even better one for Rex. He tells Rex he's thinking of a 6-digit number.\n\"All of the digits are different. The digital sum matches the number formed by the last two digits in the number. The sum of the first two digits is the same as the sum of the last two digits.\"\n\"Take the sum of the number, the number rotated one to the left, the number rotated one to the right, the number with the first three and last three digits swapped, the number with the digit pairs rotated to the left, and the number with the digit pairs rotated to the right. The first and last digits of this sum match the last two digits of the number, in some order.\"\nRalph then asks, \"If each of the three numbers formed by the digit pairs in the number is prime, then what is the number?\"\nRex looks confused, and for a moment Ralph thinks he's finally gotten him. Then Rex smiles, scribbles a few things down on a pad of paper and then says, \"Very nice, Ralph!\"\nRex then tells Ralph his number.\nWhat did Rex say?\n\nWe also apologize for the confusion caused by the empty appendix sections and buried examples. This was an oversight in the submission process. As noted, examples do appear in the Supplementary Material zip (Appendix G.2 and H.2), but we will ensure the final version integrates these directly into the main paper and proper appendices with complete content.\nThank you again for your thoughtful and constructive suggestions—they will help us improve both the clarity and quality of the final paper.\n\n\n> **License and permission**\n\nThank you for raising this important point. We take issues of data licensing and attribution very seriously.\nThe problems sourced from Braingle were obtained via publicly accessible pages, and we ensured that only a subset of logically self-contained, well-formatted problems were included. We recognize that these problems reflect the creative efforts of the Braingle community, and we agree that clear attribution is warranted. In the final version of the paper and dataset documentation, we will prominently acknowledge Braingle and refer to this portion of the dataset as the Braingle Brainteasers subset, in recognition of the source and its contributors.\nRegarding distribution rights: while the Braingle site does not include an explicit license on each problem page, we acknowledge that the copyright status of community-contributed content is nuanced. Out of an abundance of caution, we are currently reaching out to Braingle to seek explicit permission for redistribution. If we are unable to obtain permission, we will refrain from releasing the data publicly and release only metadata such as URLs, and code for crawling the examples, allowing users to crawl the content themselves under fair use guidelines for research.\nWe will also update the Ethics Checklist accordingly to transparently reflect these concerns under items 5 and 12, and we will clarify the licensing status of all included content in our documentation.\nThank you again for drawing attention to this critical issue—we are committed to resolving it responsibly."}, {"review_id": "ac85QkzuvD", "reviewer": "Reviewer_Ljxd", "summary": "This paper investigates the nature of mathematical reasoning in large language models (LLMs), with a particular focus on whether their problem-solving abilities reflect genuine reasoning and “creativity” or rely on brute-force strategies.  The study includes evaluations across multiple LLM families (GPT-4, Gemini, Claude, Mistral, Qwen, etc.) and incorporates diagnostic tasks involving symbolic composition, multi-hop reasoning, and instruction perturbations.\n\n\nBut this is a benchmarking paper; I am not sure it's okay in the main conference track.", "questions": "How is \"creativity\" defined or measured in your study? Do you distinguish creativity from generalization, and can you provide more formal criteria for emergent problem-solving behavior?\n\nAre the synthetic OOD tasks verified for ambiguity or multi-solution behavior? If so, how do you disambiguate correctness in cases where a model’s answer is logically valid but differs structurally from the reference?\n\nDid you assess the effect of prompt format variation (e.g., step-by-step vs. direct answer) on model brittleness under OOD transformations?", "limitations": "As qwen3 released, I am curious about how LLMs perform with thinking/no-thinking mode in the Brainteasers.\n\nAuthors should list a table to compare their datasets with previous artifacts.\n\nA significant concern is that the benchmark data—is collected directly from online sources. Since many LLMs (particularly closed-source ones like GPT-4 and Claude) are trained on large-scale internet corpora, it is difficult to rule out the possibility that these models have already seen similar or even identical data during pretraining. Without proper contamination analysis or data filtering, this raises the risk of unfair evaluation and inflated performance due to memorization rather than true reasoning. The paper should discuss this limitation explicitly and, if possible, audit overlaps or apply filtering heuristics to mitigate this concern.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nTimely and Important Question: The paper addresses a critical and often underexamined question in LLM research—whether strong performance on mathematical benchmarks reflects actual generalization or superficial pattern recognition.\n\nEvidence of Emergent Capability in GPT-4: The observation that GPT-4 can often recover correct reasoning paths despite structural variation in input suggests a qualitative leap in capability, which is rigorously documented.\n\nClear Presentation: The paper is well-organized, with diagnostic tasks clearly explained and results presented with meaningful qualitative examples and failure case analyses.\n\n\n**Weakness**\n\nNo Formal Quantification of “Creativity”: Although the paper investigates whether models are reasoning creatively, it stops short of defining or operationalizing \"creativity\" in a rigorous way. The interpretation is left largely qualitative.\n\nInstruction Perturbation Scope May Be Narrow: Although the perturbations cover some linguistic variety, they do not extensively probe multi-lingual, low-resource, or cross-modal (e.g., diagrammatic) instruction types. Hence, the “robustness” claim is partially constrained.\n\nLack of Ablation for Prompt Length / Format: Since prompt formatting often heavily influences model behavior, especially in math tasks, the study could benefit from controlled ablations that disentangle performance variance due to input length, verbosity, or few-shot ordering.", "was_revised": true, "final_justification": "I raise my score to weak accept regarding of the rebuttal with authors.", "rebuttal": "We thank the reviewer for acknowledging our work is timely and important, rigorous and clearly presented, and for providing the detailed comments and constructive feedback! We will try our best to resolve the comments and greatly appreciate it if you could increase the scores if the comments are addressed properly.\n\n\n> **Appropriateness for the main instead of the benchmark track.**\n\nThe Brainteaser benchmark is not only a resource but a scientific lens into the capabilities and limitations of current most advanced LLMs. BRAINTEASER makes two substantive scientific contributions: (1) it offers a rigorously curated benchmark of expert‑authored math and logic puzzles that juxtapose short, insight‑driven solutions with longer brute‑force enumerations, giving researchers a controlled lens on creativity vs. brute‑force reasoning; and (2) it introduces a multi‑stage evaluation protocol—covering narrative‑to‑formula translation, direct and hint‑guided solution generation, informed self‑correction, and step‑level abstraction—which, when applied to five state‑of‑the‑art LLMs, reveals that models (i) often default to brute‑force search despite available creative shortcuts, (ii) struggle to revise answers even after seeing gold solutions, (iii) gain only modest accuracy from narrative-to-formula translation, and (iv) can break down a solution into meaningful steps. Those substantive scientific contributions squarely satisfy—and enrich—the NeurIPS main‑conference mandate,\n\n> **Quantitative definition of creativity. Creativity versus generalization. Formal criteria for emergent problem-solving behavior.**\n\nWe agree that this is an important concern. Providing a fully formal, quantitative definition of creativity would indeed be very useful, but we suspect that creating such a definition in a universally agreed-upon way may not be achievable (i.e., creativity may not be a formally-definable or quantifiable topic). Nonetheless, we believe that our operationalization of creativity is sufficient for the purposes of this paper - i.e., even if we can’t define it strictly, we can provide reasonable proxies for deciding whether a particular behavior is creative or not, even if we can’t provide a fully general definition for creativity.\n\nWe do provide a definition of creativity in Section 4.2:  \n\n\"We define a creative solution as an innovative, insight-driven approach that\nleverages pattern recognition or lateral thinking. Rather than exhaustively testing all possibilities,\nit reframes the problem or exploits shortcuts to reduce complexity. Such solutions often involve\nminimal computation and are especially valuable for problems where brute-force search is intractable.\"\n\nQuantitatively, one can think of creative solutions as solutions that reduce the problem complexity by more than a constant factor (like reducing O(n^2) from O(n log n)). However, unlike algorithmic problems, it is difficult to quantitatively analyze the problem complexity of brainteaser problems, and so we chose to analyze this qualitatively. \n\n\n> **Instruction perturbation scope.**\n\nCould you please clarify which experiments you are referring to? We did not include analyses of instruction perturbation, but we may be misinterpreting this comment.\n\n> **Ablations of prompt length/format.**\n\nWe examine whether the length of a problem—measured by token count in the problem statement—correlates with model correctness. Our analysis reveals a weak to negligible correlation, indicating that longer problems are not necessarily harder for models to solve. This suggests that problem complexity is not well captured by surface-level features like textual length, but rather by the underlying reasoning depth or conceptual challenge. For instance, some short problems require multi-step logical deductions, while some long problems are verbose but procedurally simple. This reinforces the need for deeper structural metrics—such as number of reasoning steps in the solution or presence of creative insights—to assess problem difficulty.\n\n> **Synthetic OOD tasks.**\n\nWe did not include synthetic OOD tasks in our benchmark (all of our problems are naturally sourced), so we do not believe that this concern applies here. Please let us know if we are misinterpreting this concern.\n\n> **Qwen 3 performance**\n\nThank you for this point; below we present results with Qwen 3:\n\n|Set |Qwen3 (thinking)| Qwen3-32b (non-thinking|\n|----------------|------------|------------------|\n|Brainteasers(Math) | 70.3 | 65.9| \n|Brainteasers(Logic)| 60.5 | 56.4| \n\nThe results show that Qwen3 with \"thinking\" capabilities significantly outperforms its larger counterpart, Qwen3-32B without \"thinking,\" across both mathematical and logical brainteasers. Specifically, Qwen3 achieves 70.3 on math problems compared to Qwen3-32B’s 65.9, and 60.5 on logic tasks versus 56.4. These consistent gains of around 4 points in both domains suggest that reasoning-augmented prompting—such as step-by-step thinking or chain-of-thought strategies—plays a crucial role in enhancing problem-solving performance. Notably, the benefit is not simply a function of model size, as the smaller \"thinking\" model outperforms the larger \"non-thinking\" one. This underscores that cognitive scaffolding mechanisms, rather than brute-force scaling, are key to tackling structured reasoning challenges.\n\n\n> **Comparison with previous artifacts in a table.**\n\nThank you for this suggestion; here is such a table:\n\n| Dataset       | # Examples | Style             | Reasoning Type                           | Knowledge-barrier | Diversity | Complexity | Real-world |\n|---------------|------------|-------------------|-------------------------------------------|--------------------|-----------|------------|-------------|\n| MATH          | 5000       | Math competition  | Mathematical reasoning                    | High               | High      | High       | Yes         |\n| AIME-2025     | 30         | Math competition  | Mathematical reasoning                    | Low                | High      | High       | Yes         |\n| Zebra-Logic   | 5000       | Logic grid        | Logical reasoning                          | Low                | Low       | High       | No          |\n| BrainTeaser   | 1100       | Puzzle            | Lateral thinking                           | High               | Low       | Low        | Yes         |\n| Braingle Brainteasers  | 478        | Puzzle            | Puzzle/mathematical/logical reasoning      | Low                | High      | High       | Yes         |\n\n\nThis table compares five datasets spanning mathematical, logical, and puzzle-based reasoning. The datasets vary in size, problem style, reasoning type, and contextual richness. \n\nAs this table highlights, advantages of the Braingle Brainteasers dataset include that it presents multi-faceted coverage across several types of reasoning, it has a high level of diversity and complexity, it has a low knowledge barrier (little background knowledge is needed), it is challenging yet interpretable, and it is complimentary to existing datasets (providing a middle ground between structured logic datasets such as Zebra-Logic and formal mathematical ones such as MATH and AIME-2025).\n\n[1] Measuring mathematical problem solving with the MATH dataset. arXiv: 2103.03874. NeurIPS Datasets and Benchmarks Track, 2021.\n\n[2] Art of Problem Solving. American invitational mathematics examination, n.d. Accessed:2025-05-15.\n\n[3] Zebralogic: On the scaling limits of llms for logical reasoning. arXiv: 2502.01100. 2025.\n\n[4] BRAINTEASER: Lateral thinking puzzles for large language models. arXiv: 2310.05057. EMNLP 2023. \n\n\n> **Discussion on how data contamination could affect performance.** \n\nData contamination remains a potential limitation in evaluating model performance, with varying degrees of severity. On the more superficial end, contamination can arise from direct overlaps such as copying people’s names or short n-grams, which may not significantly affect reasoning or creativity. However, a more subtle and challenging form involves contamination of creative insights—where a model may internalize not just the surface form but the underlying reasoning strategies or ideas from training data [1,2]. This type of creativity contamination is difficult to detect and evaluate [1]. To address these concerns, better methods for understanding and auditing training data are necessary. While models like OLMo offer some transparency [3], it remains an open question how critical data contamination is to downstream performance. We will include a potential data contamination statement in the final version. \n\n[1] Quantifying Memorization Across Neural Language Models. arXiv: 2202.07646. ICLR 2023. \n\n[2] Investigating Data Contamination in Modern Benchmarks for Large Language Models. arXiv: 2311.09783. NAACL 2024. \n\n[3] OLMo: Accelerating the Science of Language Models. arXiv: 2402.00838. ACL 2024. \n\nWe thank the reviewer for their valuable feedback and great questions. We hope that our rebuttal addresses all the reviewer’s concerns, and we kindly ask the reviewer to potentially upgrade their score if the reviewer is satisfied with our responses. We are also more than happy to answer any further questions that arise."}]

Author rebuttal:

Meta-review: The paper presents the BrainTeaser dataset, consisting of nearly 500 math and logic brain teasers selected from the popular Briangle website. The puzzles have detailed human solutions and the paper considers such a dataset towards evaluating the correctness of the reasoning chains of state-of-the-art large language models. The paper derives several insights on the problem solving skills of AI models, specifically in quantifying creative against brute force solutions, effectiveness in the use of hints, as well as the influence of the problem description style. 

The paper received overall positive reviews with reviewers appreciating the clarity in the presentation, importance of the dataset, and the elaborate experiments analyzing various aspects of LLM reasoning. There were also some important concerns flagged:
1) Issue of using o3 model as the solver and the evaluator (xRP8, RQd3), that could introduce judgemental bias-- rebuttal clarifies that o3's evaluation was compared to human judgements and fared better than other LLMs.
2) Statistical significance of results and generalization to other problem categories (xRP8, RQd3) -- rebuttal provided additional results attesting the significance. 
3) Accounting for multi-solution scenarios (Ljxd) 
4) Data contamination given the puzzles are scrapped from the web (Ljxd) -- Authors acknowledge that this is a concern.

AC had an independent read of the paper. After carefully considering the reviewers' comments, authors responses, and the confidential comments provided, AC concords with the reviewers that the paper makes a good contribution through the BrainTeaser dataset towards analyzing deeper reasoning skills of LLMs. Thus, AC recommends acceptance. Authors should revise the final paper to include the new results presented during the rebuttal as well as incorporating the reviewers' feedback. Authors are strongly encouraged to take measures to minimize the issue of data contamination that Reviewer Ljxd alludes to.
Decision: Accept (poster)