PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts
Authors: Hengzhi Li
Justin Zhang
Brendon Jiang
Alexander Naehu
Regan Song
Megan Tjandrasuwita
Chanakya Ekbote
Steven-Shine Chen
Adithya Balachandran
Wei Dai
Rebecca Chang
Paul Pu Liang
Venue: ICLR 2026 Poster (2026)
Keywords: Benchmarking
Foundation Models
Multimodal Reasoning
Primary area: datasets and benchmarks
TL;DR: We introduce PuzzleWorld, a benchmark of 667 puzzlehunt problems to test open-ended, multimodal AI reasoning, and find that models perform poorly, highlighting gaps in their creative problem-solving abilities in non-constrained environments.

Abstract: Puzzlehunts are a genre of complex, multi-step puzzles lacking well-defined problem definitions. In contrast to conventional reasoning benchmarks consisting of tasks with clear instructions and constrained environments, puzzlehunts requires discovering the underlying problem structure from multimodal evidence and iterative reasoning, mirroring real-world domains such as scientific discovery, exploratory data analysis, or investigative problem-solving. Despite progress in foundation models, their performance on open-ended settings remains largely untested. We introduce PuzzleWorld, a comprehensive benchmark of 667 puzzlehunt-style problems designed to assess step-by-step, open-ended, and creative multimodal reasoning. Each puzzle is annotated with the final solution, detailed reasoning traces, and cognitive skill labels, enabling holistic benchmarking and fine-grained diagnostic analysis. Most state-of-the-art models achieve only 1-4\% final answer accuracy. On PuzzleWorld, the best model solves only 18\% of puzzles and reaches 40\% stepwise accuracy, matching human puzzle novices but falling significantly behind puzzle enthusiasts. To demonstrate the value of our reasoning annotations, we show that fine-tuning a small model on reasoning traces boosts stepwise accuracy from 4\% to 11\%, which translates to improvements in downstream visual reasoning tasks. Our detailed error analysis reveals that current models exhibit myopic reasoning, are bottlenecked by the limitations of language-based inference, and lack sketching capabilities crucial for visual and spatial reasoning. We release PuzzleWorld at https://github.com/MIT-MI/PuzzleWorld to support future work on building more general, open-ended, and creative reasoning systems.

Published as a conference paper at ICLR 2026
PUZZLEWORLD: A BENCHMARK FOR MULTIMODAL, OPEN-ENDED REASONING IN PUZZLEHUNTS
Hengzhi Li Justin Zhang" Brendon Jiang" Alexander Naehu' Regan Song"
Megan Tjandrasuwita'
Chanakya Ekbote' Steven-Shine Chen 1.2 Adithya Balachandran'
Wei Dai Rebecca Chang' Paul Pu Liang'
MIT Media Lab and MIT EECS
Imperial College London
https:/ https://githbb.o. com
MI MI/PuzzleWorll
ABSTRACT
Puzzlehunts genre of complex, multi-step puzzles lacking well-defined prob- lem definitions. In contrast to conventional reasoning benchmarks consisting of tasks with clear instructions and constrained environments, puzzlchunts re- quires discovering the underlying problem structure from multimodal evidence and iterative reasoning, mirroring real-world domains such as scientific discov- ery, exploratory data analysis, or investigative problem-solving. Despite progress in foundation models, their performance on open-ended settings remains largely untested. We introduce WIZZLEWORLD. puzzlehunt-style problems designed to assess step-by-step, open-ended, and cre- ative multimodal reasoning. Each puzzle is annotated with the final solution, de- tailed reasoning traces, and cognitive skill labels, enabling holistic benchmark- ing and fine-grained diagnostic analysis. Most state-of-the-art models achieve only 1-4% final answer accuracy. On PUZZLEWORLD the best model solves only 18% of puzzles and reaches 40% stepwise accuracy, matching human puz- zle novices but falling significantly behind puzzle enthusiasts. To demonstrate the value of our reasoning annotations, we show that fine- men-tuning a small model on reasoning traces boosts stepwise accuracy from 4% to 11%, which trans- lates improvements in downstream visual reasoning tasks. Our detailed er- ror analysis reveals that current models exhibit myopic reasoning, are bottle- necked by the limitations of language-base pabilities crucial for visual and spatial reasoning. We release PUZZLEWORLD at hhtpp:://// : /tt////ittui www.co///////// /MIT MI MI/Puzzleeorleee to support future work on building more general, open-ended, and creative reasoning systems.
comprehensive benchmark of 667
inference, and lack sketching ca-
INTRODUCTION
Recent advances in language and multimodal reasoning (Liang et al. 2024b) have enabled sig- nificant progress in step-by-step problem-solving (Wei al.) [2022] Yao al. 2023), transparent reasoning (Creswell & Shanahan 2022; Luo et al. 2003). and enhanced human-AI Al collaboration (Wu et al. 2022] Chen et al.) 2025b). Such progress has been fuelled by and evaluated on com- prehensive benchmarks, particularly in domains like mathematics (Lu et al. (022)) and code (Jiang al. 2024). However, these benchmarks are largely confined to narrow, dell-ee environments. In coding, tasks are meticulously specified and validated within executable environments (Jimenez al. 2024). In geometry, models often rely on domain-specific languages to structure their rea- soning (Chervonyi al. 2025). While valuable, these benchmarks primarily test model's ability within a pre-defined problem space, rather than its ability to discover the problem itself.
In contrast, human reasoning excels in open-ended environments, where the rules are unstated and the objectives ambiguous. We dynamically form hypotheses, adapt to implicit structures, and reason creatively across modalities to solve problems ranging from deciphering an escape room to novel scientific discovery. To build more generalist AI, we argue that the next frontier for evalu- ation lies beyond the current constrained settings. It demands benchmarks that challenge models to operate in less structured, sicovery-driver environments that require more flexible and holistic reasoning (Mondorf & Plank 2024).
"fhhese authors contributed equally.

---

Published as a conference paper at ICLR 2026
Puzzle Input
Human Annotation
Evaluation
MAXI
First, well need to answer the clues place these words spiral, now, pull the first letter of each world_looking closely.
Solve each clue determine word.
Place each answer in the circuler diegram.
Answer: GFT TO BOTTOM OF
circle..
Read the letters..t get ROLL WT
Step correct.
Step correct.
Answer: ROLL WITH IT
Incorrect.
Inout Modallit
Reasoning Skill
Final Answer Correct:
False
Teet
Visuaal
Atructured
lngin
Inowdedge
spanal
cypricc CommonoeeEE wordpay
Stapwise Accuracy:
Figure 1: Overview of PUZZLEWORLD: PUZZLEWORLD is a dataset of complex puzzles that lack explicit instructions, requiring solvers to deduce the final answer from nuanced, multimodal cues from the puzzle content as well as external domain-specific knowledge. The raw puzzles and solutions are sourced from Puzzled Pint, and the solutions, which are PNG images, are transcribed into a sequence of reasoning steps by human annotators. These annotations enable us to measure the accuracy of the final answer and the step-by-step progress made towards the solution. Best-viewed zoomed in and in color, high-resolution puzzles are in Appendix E.3
Puzzles are designed precisely test these human-like generalist reasoning abilities. While some popular puzzles such as Sudokus and crosswords are rigidly formatted, others, like puzzlehunts, are intentionally open-ended. In a puzzlehunt, puzzle solvers are not given clear task; they must first infer the nature of the problem from ambiguous clues embedded in text, images, or cultural references before devising and executing solution.
Beyond their entertainment value, puzzlehunts model the essential challenges of real-world dis- covery and analysis. They demand compositional thinking, lateral reasoning, and the resilience to pursue leads, backtrack from dead ends, and manage uncertainty. Unlike current Al benchmarks that present well-specified tasks, puzzlehunts compel solvers to discover both what the problem is and how to solve it. This dual challenge makes them uniquely suited for evaluating general-purposs reasoning systems under conditions that more closely resemble open-ended scenarios like scientific investigation, intelligence analysis, or exploratory design.
To bridge this gap, we introduce PUZZLEWORLD a benchmark of 667 real-world puzzlehunt prob- lems curated from Puzzled Pint (Puzzled Pint 2025), monthly puzzlehunt event with content released under Creative Commons license. These puzzles offer an open-ended, compositional challenge beyond prior benchmarks focused
mnstuctiionfolloviin or task completion, and will grow with new puzzle releases. For each puzzle, we provide fine-grained annotation of its solution, input modalities, cognitive reasoning skills it exercises, and manually curated step-by-sted solution trace. These rich annotations support diagnostic analysis, model training, and detailed evaluation of models' reasoning capabilities. An overview of PUZZLEWORLD is provided in Figure
FUZZLEWORLDE enables us systematically study the multimodal and multi-step reasoning capa- bilities of today's 's best foundation models. Most state-of-the-art models achieve only 1-4% final answer accuracy. with the best model solving only 14% of puzzles and reaching 40% stepwise accuracy. We additionally find that detailed annotations are important, as fine-tuning a model on annotated reasoning traces significantly improves small model's performance, both within Puz- ZLEWORLD and on other visual reasoning datasets. We also conduct detailed error analysis on models' performance on PUZZLEWORLD yielding tangible directions for future work in improv- ing multimodal open-ended reasoning in AI. Together, these elements position PUZZLEWORD as a rigorous resource for evaluating and improving ceneral-purpos multimodal reasoning in AI sys- tems. In the long run, believe PUZZLEWORLE mathematical and logical reasoning, open-ended scientific discovery, and assistive agents. PUZZLE- WORLD is publicly available at wtttttss//gggiee thub. com/ MTT /Puzz eWorld
catalyze more general and adaptable AI for
2

---

Published as a conference paper at ICLR 2026
Puzzle Taxonomy
Difficulty: Easy
Difficulty: Medium
Difficulty: Hard
Sulair TTV
Input Modality
VILL sthuchured
Reasoning Skills
edgo wordslay
Answer: aeteeca PULL
Answort THEY SAY CROWD
Answer: MOLUEE
Trgua Mossity
Text
leguk Moduriy
Text
Viael
Input
Toxt
Visual
Stractare
Reconiing Skils
Wordylay
Knowledgy
Rossoring Skill
loggl
Spurtel
Reseaning Biils
Logie
Oplatal
We Sigiyyyy
Manciplay
Onyaric
Figure 2: Overview of samples from PUZZLEWORLD. Left: To gain a deeper understanding of model performance PUZZLEWORLD, each puzzle is annotated with the input modalities of the puzzle content, the reasoning skills required to solve the puzzle, and step-by-step reasoning steps. Right: Example modality and reasoning skill annotations on three puzzles. High-resolution puzzle images are included in Appendix E.3
2 RELATED WORK
Large Language Model (LLM) Reasoning. LLMs have demonstrated remarkable emergent ca- pabilities, often matching or even surpassing human performance across wide range of tasks (Street et al., 2024). Notably, models such as GPT-4 (Achiam et al., 2023) and Claude (Anthropic) 2025) have achieved strong results not only on traditional NLP benchmarks-
question answering
summarization, and translation (Widyassari et al. 2022 Soares & Parreiras 2020 Singh et al.,
2017, but also in more complex domains such as mathematical reasoning, programming, and log- ical deduction (Ahn et al. 2024| Jiang et al. 2024 Lam et al.] 2024). These abilities suggest that LLMs are beginning to exhibit general-purpose reasoning skills, making them increasingly relevant to both academic research and practical applications. However, despite these impressive capabili- ties, understanding the full extent and limitations of LLM reasoning remains a crucial open question, underscoring the need for benchmarks that rigorously assess their capability for flexible, holistic rea- soning (Mondorí & Plank| 2024; Chang et al. 2024).
Reasoning Benchmarks. Numerous reasoning benchmarks have been proposed to evaluate vari- ous cognitive skills, including visual mathematical reasoning (Lu et 2024), spatial understanding
(Wang et al.) 2024a). analogical reasoning (Yiu et al. 2024). and social reasoning (Li et al., 2025 Mathur et al. 2025). ). However, few have addressed abstract, open-ended problems that demand
holistic reasoning. HEMM (Liang et al. 2024a), SciBench (Wang et al. 2024B). MMMU (Yue) et al. 2024a), MMMU-Pro (Yue et al. 2024b), MMT-Bench (Ying et al. 2024). and Olympiad- Bench (He et al. 2024) test multimodal reasoning across various disciplines in academic and real- world contexts. While these tasks are broad and challenging, they typically involve well-defined questions that closely resemble the training distributions of large models. As such, they primarily assess in-distribution reasoning rather than creativity or adaptability. ARC-AGI (Chollet (2019) tests the ability to reason and adapt to new situations through abstract visual pattern recognition tasks that require minimal prior knowledge, yet lacks the open open-ended, exploratory nature of real-world prob- lem solving. In contrast, PUZZLEWORE ORLD targets open-ended reasoning through puzzlehunts that lack explicit instructions. Solving these tasks requires creatively piecing together subtle hints, often across many modalities, into coherent multi-step reasoning chains.
Puzzle Benchmarks. A growing line of work has explored the use of puzzles to test the reasoning capabilities of AI systems. PUzzleVQA (Chia et al. 2024) consists of 2,000 puzzles that require abstracting patterns from visual puzzles to answer multiple-choice questions. AlgoVQA (Ghosal ct al. 2024) is a visual puzzle benchmark requiring algorithmic reasoning. PUZZLES (Estermann et al. 2024 tests the ability of RL agents to perform algorithmic reasoning on a set of 40 puzzles. While valuable for evaluating specific skills, these benchmarks focus on narrow domains with con- strained task formats, and modern models generally perform well on these benchmarks (Chia al.) 2024] Moskvvchev et al.) 2023) Yue et al., (024a). On the other hand, the unstructured nature of the puzzlehunt problems in PUZZLEWORRE WORLD requires models to interpret ambiguous cues, explore
3

---

Published as a conference paper at ICLR 2026
Statistic
Value
Puzzle Phil
Toral puzzles
Avg. 4ol Reasoiing Steps
Percent of Visual Reasoning Steps
12.
Raw Puzzles
Human Annotation
Automatic
Manual Data Cloaning
Avg. Word Count per Ressoning Step
22.5
Solutions
Verification
Conceition and of Reasening Steps
Dilliculty
First, we source raw puzzles and solutions from Puzzled Pint. As the Puzzled Pint solutions are often not correctly parsed by OCR, each puzzle's s metadata and reasoning steps are human-annotated. flag ambiguous and inconsistent annotations. Finally, two human verifiers perform manual data cleaning on the flagged puzzles to ensure a consistent annotation format. Right: We summarize the statistics of our dataset. The average number of reasoning steps is high, and the steps relatively complex, as shown by the high average word count.
Figure 3: Dataset construction procedure and statistics: Left:
use GPT-4o to automatically
creative strategies, and integrate information across diverse modalities and knowledge areas. con- trast to previous benchmarks that isolate strictly structured vertical reasoning (Chen et al. 2025a) or narrative-basee lateral thinking (Huang et al., 2024], puzzlehunt puzzles require an integrated blend of lateral thinking, symbolic abstraction, and visual-spatiaal reasoning. The closest to benchmark is EnigmaEval (Wang et al. 2025). which also evaluates Al's reasoning capabilities on puzzlehunts. However, EnigmaEval is closed-source evaluation-only dataset and does not include manually annotated step-by-step solutions. The open-sourced puzzles and rich annotations in PUz- ZLEWORLD support fine-gres sree-graiedd analysis of intermediate reasoning and failure modes, facilitating the development and evaluation of more robust,
reasoning models.
TAXONOMIZING MULTIMODAL REASONING IN PUZZLEHUNTS
To understand how solving puzzlehunts engages reasoning capabilities evaluated separately in benchmarks like MMMU (Yue et al. 2024b), we analyze puzzle solutions and classify them along two dimensions: input modality and reasoning mechanism. This taxonomy provides a comprehen- sive evaluation framework that captures both the form in which information is presented and the cognitive strategies required for reasoning.
3.1 PUZZLE INPUT MODALITIES
We consider three puzzle input modalities: Text, encompassing textual information such as instruc- tions, narratives, word puzzles, testing the model's ability to extract relevant linguistic informa- tion; Visual, which includes unstructured visuals like images, icons, and typography, challenging the models to interpret visual semantics and patterns; and Structured, which refers to systematically organized visual information, such as tables, graphs, grids, matrices, and charts. Table [ shows the distribution of puzzles across modality and difficulty.
3.2 PUZZLE REASONING MECHANISMS
Table 1: Count of puzzles across modalities and difficulties. Across all modalities, the distribution of difficulties is similar.
We identify six core cognitive abilities essen- tial for effective puzzle-solving in PUZZLE- WORLD. These include logic, which cov- ers inferential reasoning such as deduction and causal inference; wordplay, involving flexi- ble linguistic interpretation through puns, ana- grams, and homophones; spatial reasoning.
Easy
Medium
Hard
Text
131
322
151
Visual
90
226
111
Structured
181
108
which tests an Al's ability to mentally manip- ulate objects and navigate structures; and cryptic decoding, which requires recognizing and apply- ing transformations like ciphers and hidden encodings. In addition, knowledge-based reasoning leverages domain-specinf facts from areas such as science or history, while commonsense reason- ing draws implicit real-world expectations. This taxonomic approach enables targeted evaluation and analysis of Al reasoning capabilities across different cognitive dimensions. By mapping specific puzzles and reasoning tasks to combinations of modalities and mechanisms, we can identify areas of strength and weakness in Al systems, track progress over time, and guide future development efforts toward more balanced reasoning capabilities.
4

---

Published as a conference paper at ICLR 2026
4 CREATING PUZZLEWORLD
4.1 DATA COLLECTION AND PRE-PROCESSING
We collected our puzzle corpus from Puzzled Pint (2025), an organization that publishes puzzles under Creative Commons (CC BY-NC-SA Intl. 4.0). Their repository contains monthly puzzles designed for collaborative solving, covering diverse range of puzzle types and difficulties. This allowed us to obtain more than 700 raw puzzles spanning from 2010 to 2025.
Each puzzle in our dataset consists of original PDF containing the puzzle content, a single-phrase answer, and solution document. Unlike Wang et al. (2025). we deliberately preserved the origi- nal puzzle format rather than transcribing content into separate text and images. This decision was motivated by the importance of spatial relationships in puzzle layouts to the solving process. Fur- thermore, Wang et al. (2025) showed that the best foundation models are not primarily constrained by OCR capabilities. Instead, devote our manual effort to construct fine- mne-grained annotations of puzzle reasoning steps, ensuring that the annotations accurately capture the intended solution pathways while maintaining the integrity of the original puzzle presentation.
4.2 DATA ANNOTATION
To facilitate AI's reasoning capabilities, we designed a comprehensive annotation structure for Puz- ZLEWORLD. Each puzzle represented by standardized metadata and visual assets. To prevent ambiguity, we discard puzzles that have incomplete solutions, multiple ground truth answers, or require physical activity solve the puzzles. This leaves us with 667 annotated puzzles.
4.2.1 METADATA SCHEMA
Each puzzle is annotated using JSON schema comprising several fields: descriptive title; flavor text providing narrative context; dif- ficulty label (easy, medium, or hard); solution representing the canonical answer; reasoning field of an ordered sequence of steps leading to the solution; modality tag specifying the in- put types involved; list of skills capturing the cognitive abilities required for solving; and source field attributing the data origin. Figure illustrates an example annotation.
content.png
metadatta.jon
Pardel
<8055 ARRONS
Thil
FRICEI ARROWS
Fovor Text
Dras, the god love, hae...
Dificully
Medium
Spirier
AMOUR
Reasoring
Hecogiaaze
lifff
dots.. (Ngure UnK
Dake
Combine
Modality
text, vievel, structured
Kary 600
Jamaao
enypli,
4.2.2 REASONING ANNOTATION
Sourte
retpecippertteeppr ntoegg....
A key contribution of our annotations is the de- composition of puzzle-solving into reasoning steps. Each step is formalized as tuple (e, f where e represents the text explanation and f denotes an optional figure illustrating the rea- soning. To ensure consistency, we loosely re- quire each step to begin with atomic opera- tion, such as pattern discovery or sketching, followed by the operation's 's intermediate output. This annotation structure enables fine-grained reasoning trajectory analysis on Al solutions.
Figure 4: Illustration of metadata schema: All puzzles are annotated with accompanying meta- data, which includes the title, flavor text, diffi- culty, final answer, reasoning steps, input modali- ties, reasoning skills, and the link the puzzle.
4.3 VERIFICATION OF ANNOTATIONS AND DATA CONTAMINATION
To ensure annotation quality and integrity, we implemented a two-stage verification protocol. First, we used GPT-4o to flag each puzzle annotation for correctness and reasoning coherence. This auto- mated screening identified reasoning steps exhibiting ambiguity or logical discontinuities that might impede systematic analysis, which has flagged 12.1 1 % of the dataset. Subsequently, two human ver- ifiers independently reviewed all flagged annotations, applying corrections where necessary. This verification process resulted in modifications to 10.93% of the initially annotated puzzles. As an additional quality assurance measure, prising 5% of the dataset. In this evaluation, 96.5% of the verified annotations are marked as correct by the verifiers, demonstrating the high reliability of our annotation methodology. Finally, we verify whether frontier models has memorized any of the puzzles in PUZZLEWORLD. We describe our procedure in E.1 where we find no evidence of data contamination.
conducted manual verification of a random subset com-
5

---

Published as a conference paper at ICLR 2026
100%
Difficu
Percent
50%
60%
40%
Frequency
200
Easy
Hard
21.2%
8%
20%
100
Rort
Visual
Structured
Logic
Spatial
Wordplay
Commensense
Cryptic
Know
53.0%
20 Number Reasoning Steps
Modality
Reasoning Skill
Figure 5: balanced. While the majority of puzzles are of medium difficulty, there is significant number of easy and hard puzzles. The number of reasoning steps follows long-tail tail distribution, with many solutions requiring more than 5 steps and some hard puzzles requiring up to 30 steps of reasoning.
PUZZLEWORLD dataset statistics. Distributions of modalities and reasoning skills are
4.4 DATASET STATISTICS
We summarize key statistics in Figure 3 (right). The average number of reasoning steps is above 5, and the average word count per reasoning step is above 20, demonstrating the complexity of the reasoning traces. Additionally, 12.3% of the steps have visual intermediate output, highlighting the importance of sketching and spatial reasoning to solve puzzles. The correlation between puzzle difficulty and number of reasoning steps is 0.24. While we expect difficulty and number of reasoning steps to be positively correlated, the magnitude of the correlation is relatively low, as the difficulty of the puzzles also stems from their open-ended nature. Figure shows the distribution of puzzles by modalities, reasoning skills, number of reasoning steps, and difficulty.
5 EXPERIMENTS
In this section, we evaluate frontier closed and open-source multimodal LLMs on the PUZZLE- WORLD dataset. We detail the evaluation setup, present quantitative results, and conduct qualitative error analysis to understand model behavior in open-ended, multimodal puzzle reasoning.
5.1 EXPERIMENTAL SETUP
We evaluate frontier flosed-source reasoning models on PUZZLE WIOZLEWORLD including GPT-o3 (Ope-
nAI|[2055] GPT-4o (Achiam al., 2002)), Claude Opus 4 (Anthropic (202)). Gemini-2.5-Pro (Co-
manici et al. 2025), Gemini-3-Pro Google 2025) and Grok 4 (x.AI, 2025). ) We also evaluate open-
source models Qwen QVQ @wen 2024), Intern VL3 (Zhu et al.) 2025). and Kimi VL A3B (Team)
et 2025). We prompt each model with comprehensive prompt as in Wang et al. (2025), followed by the puzzle images and transcribed flavor text. See Appendix for the evaluation prompt.
We also provide human baseline PUZZLEWORLD considering three tiers of puzzlehunter ex- pertise: Novice, with no prior puzzlehunt experience; Enthusiasts, who showed interest have oc- casionally participated (1-2 sessions) in puzzlchunts, and Experts among the top teams at monthly Puzzled Pint meetings. We we gathered 9 Novices and 9 Enthusiasts across high school and college ages. We sampled 5% puzzles from Pad Dand assigned each participant to solve four puzzles. Participants were given an hour to solve each puzzle, matching the usual expected time at live session, and were asked to provide paragraph explanations for their solution. For Experts, use statistics from Puzzled Pint sessions in Syracuse, New York, and Bangalore, India, dating from January 2023 June 2025. The statistics suggest that expert puzzlehunters consistently solve all five puzzles within one to two hours, which is on average less than the time prescribed to our human par- ticipants. We thus assume that human Experts achieve perfect accuracy on PUZZLEWORLD. WORLD.
5.1.1 AUTOMATIC EVALUATION METRICS
Beyond final answer accuracy, we additionally evaluate the models' stepwise accuracy by comparing their solution with the annotated ground truth reasoning steps. Since puzzles can have multiple solution pathways, we define the stepwise accuracy score of candidate solution to be the last annotated reasoning step it successfully executed out of all the reasoning steps. We implement an LLM judge (Zheng et al., 2023) with GPT-4o to determine the stepwise score each candidate solution. For each reasoning step in the reference solution, the LLM judge determines if the step met by the candidate response. To evaluate LLM judge's reliability. we compared its stepwise evaluations 20 random puzzles against human evaluations. The LLM judge achieved Pearson correlation r = 0.829 (p = 6.3 x 10 and a mean absolute error (MAE) of 0.083 with respect to human scores, indicating strong alignment with human judgment.

---

Published as a conference paper at ICLR 2026
Table 2: Model performance. Accuracy (Acc) and stepwise accuracy (Step) are reported over- all and per modality. Models struggle significantly on PUZZLEWORLD, most achieve only 1-4% answer accuracy, with closed-source models generally outperforming open-source ones. The best model, Gemini 3 Pro, solves only 18% of puzzles and reaches 40% stepwise accuracy, matching human Novice performance but significantly falling behind Enthusiasts.
Overall
Text
Visual
Structured
Model
Acc
Step
Acc
Step
Acc
Step
Acc
Step
Open
QVQ-72B-Prevve
1.36
30.23
1.33
29.25
0.63
27.96
1.18
32.40
Intern VL.3-78
0.89
15.49
0.83
14.80
0.47
14.48
1.15
17.97
Kimi VL.
1.33
19.10
1.16
17.91
0.94
18.84
1.72
21.41
GPT-o3
14.22
39.81
15.16
39.92
8.96
33.38
13.53
41.28
Closed
GPT-4o
1.83
22.09
1.92
20.00
0.73
20.20
2.77
28.09
Claude Opus 4
4.50
24.56
4.20
23.77
4.04
22.60
4.37
26.93
Gemini 2.5 Pro
7.65
31.61
8.07
31.09
4.99
29.06
6.71
32.34
Gemini Pro
18.00
39.99
20.30
39.34
14.71
38.81
20.25
39.99
Grok 4
3.33
13.79
3.85
13.64
3.70
14.19
1.56
11.22
Human
Human Novice
13.89
23.10
16.98
25.32
11.00
22.70
16.67
24.92
Human Enthusiast
44.44
51.70
44.14
52.58
44.00
52.20
54.17
57.81
Human Expert
100.0
100.0
100.0
100.0
100.0
100.0
100.0
100.0
5.2 RESULTS
5.2.1 OVERALL PERFORMANCE OF FRONTIER MODELS
We report the models' performance in Table 2 All models exhibit extremely low final answer accuracy on PUZZLEWORLD with most achieving close 1-4%. Gemini 3 Pro attains the highest overall accuracy at 18.00%, matching human Novice performance, while the best best-performing open- source model, QVQ-72B-Previeeee Prevvew, reaches just 1.36%. All models perform significantly worse than human Enthusiasts and Experts. Although the uniformly low accuracies underscore our benchmark's ss difficulty, offers limited insight into the models' reasoning capabilities.
To address this, our stepwise evaluation metrics provide a more nuanced view of models' reasoning performance. These metrics reveal that models with poor final answer accuracy, such as Intern VI.3, still demonstrate good intermediate reasoning, achieving up to 15.49% stepwise accuracy. Simi- larly, while BVQ-72B-Previce lags behind all closed-source models in final answer accuracy, outperforms many of them in stepwise accuracy (30.2%), reflecting coherent and promising rea- soning capabilities despite not reaching the correct final output. The combination of final answer and stepwise metries enable PUZZLEW WORLD remain highly challenging while offering detailed diagnostics for model evaluation and development.
In terms of input modalities, models generally perform best text-based puzzles, with significantly lower accuracy on puzzles involving unstructured visual inputs. Interestingly, most models achieve better performance on structured puzzles, such as crosswords where the spatial format constrained, over unstructured visual puzzles. In contrast, puzzles involving free-form visuals remain difficult, with models often achieving less than half their text puzzle accuracy on these inputs. Additionally, in Appendix D.2 we find that these modality-leva performance discrepancies are not systematically driven by optical character recognition (OCR) limitations. These trends highlight current models' persistent weaknesses in visual grounding and spatial reasoning. In contrast, we observe that visual and structured modalities do not pose additional difficulty for human solvers regardless of their puzzle solving experience.
We further analyze model performance by puzzle difficulty in Appendix We find that, as ex- pected, model accuracy decreases difficulty increases. Interestingly, GPT-o3 performs best easy, structured puzzles likely because their clean, regular layouts whereas even harder struc- tured puzzles remain challenging they become more irregular and visually complex.

---

Published as a conference paper at ICLR 2026
5.3 IMPROVING REASONING ON DOWNSTREAM TASKS WITH PUZZLEWORLD
To explore whether PUZZLEWORLD
support improvement in model reasoning, fine-tuned an 8B Vnte model with supervised fine-tuning on annotated reasoning traces from 80% of the dataset, and evaluated performance on the 20% held-out test set. As control, fine-tuned the same model using only the final answers, without access to reasoning traces. Full training details are provided in Appendix F2
Our results in Table highlight the value of PUZZLEWORLD stepwise annotations. Fine- tuning on reasoning traces doubles the model's stepwise accuracy - from the base model's 4.78% to 11.00%. In contrast, fine-tuning on fi- nal answers alone impairs performance, reduc- ing stepwise accuracy 2.96% and driving an- swer accuracy to zero. Despite the improved stepwise accuracy, the fine-tuned model's an- swer accuracy remained at 0.76%. This sug- gests that while fine-tuning enhanced model's intermediate reasoning, it was insufficient to solve additional puzzles completely. This result underscores both the difficulty of PUZZLEW ORLD and the limitations of straightforward fine-tuning approaches in addressing such complex reasoning challenges in our dataset.
Table Fine-tuning on PUZZLEWORLD. Fine- tuning InternVL3 on PUZZLEWORLD reasoning traces significantly improves its stepwise aceu- racy, while fine-tuning final answer only causes model reasoning to collapse.
Model
Acc.
Step.
Base
0.76%
4.78%
Fine-tuned (Answer-only)
0.00%
2.96%
Fine-tuned (Reasoning)
0.76 %
11.00 %
Table 4: PUZZLEWORLA fine-tuned model performance on downstream reasoning tasks. Fine- tuning InternVL3 on PUZZLEWORLE leads to performance gains visually- tasks, such as Rebus puzzles, geometry, and visual question answering, but slightly reducing performance on problems dependent external knowledge, such textbook question answering.
Dataset
Task
Base Model
Fine-tuned on WORLD
Rebus Puzzles
Puzzle reasoning
3.2%
5.1 %
65.87% Textbook question answering
Geometry problem solving
66.35
63.92% %
60. 13%
Math Vista
Math word problem
62.37 %
59.14%
Visual question answering
32.40%
39.11% %
We then explore whether PUZZLEWORIEE detailed stepwise annotation can improve models downstream reasoning tasks. We finetuned a model on 80% PUZZLEWORLD and evaluated on
two benchmarks: Rebus puzzles dataset (Lee et al. 2025) involving visual metaphors without ex- plicit instructions, and the Math Vista dataset (Lu et al. 2024), ranging from general visual question answering to domain-speent -specffc geometry questions. Our results are shown in Table
Finetuning on PUZZLEWORLD yields notable performance gains on Rebus puzzles, where the model's accuracy increased from 3.2% 5.1%. On Vista, the fine-tuned model shows signifi- cant improvement in geometry problem solving and visual question answering, but its performance slightly decreased on tasks that are outside of PUZZLEWORLD 's reasoning skills and require exter- nal knowledge, such as textbook question answering and math word problems. This performance improvement suggests that the skills learned from PUZZLE WORLD are not merely task- aas--specific. They represent transferable, ceneral-purposs reasoning capabilities, making our dataset reflection of diverse reasoning expertises and a valuable tool for enhancing models' capabilities.
5.4 DETAILED ERROR ANALYSIS
In the following section, we highlight the main sources of errors by frontier reasoning multimodal L.LMs on PUZZLE capabilities. We focus our analysis on GPT-o3, and we provide qualitative examples for each error category Figure In Appendix D.2 we provide less common "visual forgetting" error pattern, where model deviates from visual puzzle content during extended reasoning processes.
myopic reasoning, limitations language, and lack of visual sketching
8

---

Published as a conference paper at ICLR 2026
Myopic Reasoning
Limitations of Language
Lack Sketching Capabilities
77p Ventie Ze Vahomit
Pazzled
A Dolorous Din
BESM
Parl
Solution
coda thind semaphore..
GPT-03
the it the tleour tad, and the fact thet the
stave shona onl dfferntt benptt "boundd mensure Mome code.
circke words, which good you
Solution
read bettors slong the BEST MANN TO ROMEE HAMED NES THIIR CAPULET NORMONTAOEE
lettrrs order apal i.stachia sttl rersenw..
GPT-a3
ling the an dh alant the diggua bokow
rings]
GPT-03
kifeers
FRUAR LAURENCE...
Figure 6: Example puzzle errors. Left: (myopic reasoning) The model is unable backtrack when it hits a dead end. Middle: (language bottleneck) The model misrepresents the visual contents due to inherent limitations of texts. Right: (sketching errors) The model fails to execute the visual sketching steps to obtain correct intermediate outputs. High-resolution images are in Appendix E.3
Myopic reasoning. Despite strong perfor- mance on conventional benchmarks, frontier models often exhibit myopic commitment in their reasoning. Rather than exploring alterna- tives or revisiting prior steps, models tend to fixate on early, surface-leve hypotheses, result- ing in reasoning trajectories that are locally co- herent but globally misaligned with the puzzle. For example, in Figure solving the puzzle re- quires interpreting musical notes using a mix of binary, Morse code, and flag semaphores. Instead, GPT-o3 identifies Morse code ref- erence early on and rigidly adheres iv-n as contradictions arise demonssttntt a lack of backtracking and verification.
Distribution of Stepwise Accuracy
40.0%
percentage
3.3000%
20.0%
10.0%
0.0%
0.0
0.2
0.4
0.6
0.8
1.0
Stepwise Accuracy
Figure 7: Stepwise accuracy distribution of GPT-o3. GPT-o3 receives stepwise accuracy o for most puzzles, highlighting the model's 's myopic reasoning tendencies and its inability to backtrack after committing to an incorrect first step.
To further examine this behavior, analyze the stepwise accuracy distribution of GPT-o3 (Fig- ure 7 We find that, on most puzzles, the model receive a score of O, meaning the model often fails to correctly identify even the first step of the solution. Once committed to an incorrect path, the model rarely recovers, highlighting its brittle reasoning and a lack of dynamic self-correction, especially when it cannot rely on external environments for verification.
Limitations of language. Modern multi- modal models rely heavily on language-based reasoning strategies, such as chain-of-thought and code generation. However, this dependence becomes bottleneck in puzzles with complex visual structure. In Figure 6 the puzzle is com- posed of four interlocking loops arranged in clover-like pattern. This layout is visually intu- itive, but difficult to represent in text.
Plpprssentaioon
ROTAL
coubre
Lonez
FINs
While GPT-o3 correctly solves the word clues, fails to capture the layout when converting the puzzle into text, as shown in Figure 8 This ul- timately leads the model to derive an incorrect answer. This example highlights broader lim- itation: when faced with highly complex struc- tured inputs, models that default textual reasoning often lose critical spatial information. This inherent mismatch between visual intuition and language-centric inference poses a fundamental challenge to models, especially those that depend on textual or code-based reasoning chains.
LAMES WITHTEAM
KOPSOW
Figure 8: Limitations of text. An example failure case where GPT-o3 fails represent complex structured puzzle into text.

---

Published as a conference paper at ICLR 2026
Multimodal reasoning needs sketching. While frontier models have made notable progress in logical deduction and arithmetic reasoning, they consistently underperform on spatial tasks that require sketching, drawing, and manipulating visual structure, such decoding based on spatial arrangements or tracing paths through grids and mazes, In F the model correctly solves the individual clues in grid-based puzzle but fails trace the intended path, resulting in an incorrect final answer. Humans naturally rely sketching or mental imagery to reason through such spatial challenges, using external or internal visualiza- tions to keep track evolving structure. The absence of such capabilities in current models reveals critical gap: without the ability to sketch and update a persis- tent visual representation, models prone failure in tasks that depend on spatial coherence.
Reasoning Skills of Failed Steps Wordplay
Commonsense
3.3%
23.
53.3%
20.0%
Spatial
Logic
Figure 9: Reasoning skills of failed steps. We annotated the bottleneck steps with their reasoning skills, and we identify spa- tial reasoning as primary bottleneck.
To understand the impact of sketching model performance, manually analyzed 30 puzzles where GPT-o3 produced incorrect answers. For each failure, we annotated the reasoning step re- sponsible for the егтог with its corresponding reasoning skill. As shown in Figure ! we found that 53.33% of these bottleneck steps involved spatial reasoning or sketching-related capabilities. This highlights a gap in models' ability to manipulate visual structure during inference. Incorporating sketch-like like visual memory and reasoning (Wu et al. 2024 Hu et al. 2024 Chen et al. 2025b) may offer a promising direction toward more robust and spatially grounded reasoning Al.
CONCLUSION
This paper introduced PUZZLEWORLD a benchmark of 667 real-world puzzlehunt-style problems for evaluating open-ended multimodal reasoning under ambiguity. PUZZLEWORLD preserves puz- zles in their original formats and provides rich annotations iictiionn-incudinn including modality and skill labels, well as human- written step-by-step solution traces with intermediate states enatle-nnabligg both final- answer benchmarking and fine-grained diagnosis of where and how models fail on complex, gener- alist reasoning tasks.
Our experiments show that, despite strong performance on conventional reasoning benchmarks, frontier models still struggle in this discovery-drived setting: accuracy drops sharply with difficulty and modality complexity, and stepwise evaluation reveals frequent early mistakes and limited back- tracking. We further show that training on annotated reasoning traces can substantially improve reasoning behavior, and that these gains transfer to other visually grounded reasoning tasks. In con- clusion, PUZZLE WORLD offers both challenging benchmark and valuable data resource. We hope that PUZZLEWORLD can foster building more robust, general-purposs multimodal reason- ing models that can excel open-ended scenarios in the real-world, such as scientific discovery, exploratory data analysis, investigative problem-solving
ETHICS AND REPRODUCIBILITY STATEMENT
This research focuses on developing benchmark support the creation of models with robust open- ended, multistep. multimodal reasoning. All data sources are cited and employed within the scope of their intended use and applicable copyright licenses. To promote transparency and reproducibility, we provide detailed data collection and annotation process in Section 4, evaluation setup in Section 5.1 and compute details in Section of the appendix. We publicly released the PUZZLEWORLD benchmark and code to facilitate reproducibility and further research.
ACKNOWLEDGEMENT
MT is supported by the National Science Foundation (NSF) under Grant No. 2141064. Any opinion, findings, and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of the National Science Foundation. We thank the MIT Office of Research Computing and Data (ORCD) for support through ORCD Seed Fund Grants. We also thank the NVIDIA Academic Grant Program for GPU support.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Mit mystery hunt.
1981
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08877. 2023.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wempeng Yin. Large language models for mathematical reasoning: Progresses and challenges, 2024. URL httppsss//aww...orggg//o///////e
00157
Anthropic. Introducing claude 2025. URL htt.:pppe/www..tttrooooocccoocea 2atpm///www//auueea
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang. Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology. 15(3): 45, 2024.
Guizhen Chen, Weiwen Xu, Hao Zhang. Hou Pong Chan, Chaoqun Liu, Lidong Bing. Deli Zhao, Anh Tuan Luu, and Yu Rong. FineReason: Evaluating and improving LLMs' deliberate reasoning through reflective puzzle solving. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 6685-6715, Vienna, Austria, July 2025a. Association for Computational Linguistics. /
ISBN 979-8-891766251-- doi: 1t..1865/112225..aeel wtt.//20................. 2025 long. 333/]
Steven-Shine Chen, Jimin Lee, and Paul Liang. Interactive sketchpad: multimodal tutoring system for collaborative, visual problem-solving. arXiv preprint arXiv:2503. 16434, 2025b.
Yuri Chervonyi, Tricu Trinh, Miroslav Olšák, Xiaomeng Yang, Hoang Nguyen, Marcelo Menegali, June- hyuk Jung, Vikas Verma, Quoc V Le, and Thang Luong. Gold-medalist performance in solving olympiad geometry with alphageometry2. arXiv preprint arXiv:2502.033. 2025.
Nathan A Chi, Teodor Malchev, Riley Kong, Ryan A Chi, Lucas Huang. Ethan A Chi, R Thomas McCoy, and Dragomir Radev. Modeling: A novel dataset for testing linguistic reasoning in language models. arXiv preprint arXiv:2406.1700 2024.
Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. PuzzleVQA: Diagmosing Multimodal Reasoning Challenges of Language Models with Abstract Visual Patterns, 2024. /
13315
François Chollet. On the measure of intelligence. arXiv preprint 1 2019.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon. Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
Antonia Creswell and Murray Shanahan. Faithful reasoning using large language models. arXiv preprint arXiv:2208.1 14271, 2022.
Benjamin Estermann, Luca A. Lanzendörfer, Yannick Niedermayr, and Roger Wattenhofer. PUZZLES: Benchmark for Neural Algorithmic Reasoning. Advances in Neural Information Processing Systems, 127059-127098, December 2024.
Deepanway Ghosal, Vernon Toh Yan Han, Chia Yew Ken, and Soujanya Poria. Are Language Models Puzzle Prodigies? Algorithmic Puzzles Unveil Serious Challenges in Multimodal Reasoning, 2024. URL kttp:
// //aaxiv.crg//aeee...... hbs///b///a 00066036644)
Google. Gemini Developer Guide. httpo://44..o000../0/////ha11111 aptpp/ccc/gcmmlmlaccppppp gemini-i 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-leve bilingual multimodal scientific problems, 2024.
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as visual chain of thought for multimodal language models. arXiv preprint arXiv:2406.00 09403, 2024.
11

---

Published as a conference paper at ICLR 2026
Shulin Huang, Shirong Ma, Yinghui Li, Mengzuo Huang. Wuhe Zou, Weidong Zhang, and Haitao Zheng. Late- val: interactive llms evaluation benchmark with incomplete information from lateral thinking puzzles. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources 2024.
and Evaluation ELREC-COLING 2024), pp.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation, 2024. URL htw.httpss//// /ariivv/aaivvvv/iaa org/ alss 2406 00515
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve worldworld github issues? In The Twelfth International Conference on Learning Representations, 2024.
Long Hei Matthew Lam, Ramya Keerthy Thatikonda, and Ehsan Shareghi. A closer look logical reasoning with Ilms: The choice of tool matters, 2024. CtRL:htp::////
arg/abb/2446.. 00284
Heekyung Lee, Jiaxin Ge, Tsung-Hay Han Wu, Minwoo Kang, Trevor Darrell, and David Chan. Puzzled by puzzles: When dision-language models can't take a hint. arXiv preprint M.XXi:2500.237555 2025.
Hengzhi Li, Megan Tjandrasuwita, Yi R Fung, Armando socially-intelligent nonverbal foundation models. arXiv preprint arXiv:2502 16671, 2025.
and Paul Liang. Mimeqa: Towards
Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Ruslan Salakhutdinov, and Louis- Philippe Morency. Hemm: Holistic evaluation of multimodal foundation models. In The Thirty-eight Con- ference Neural Information Processing Systems Datasets and Benchmarks Track, 2024a.
Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal machinc learn- ing: Principles, challenges, and open questions. ACM Computing Surveys, 56 10): 2024b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning foundation models in visual contexts. In International Conference on Learning Reprexentations (ICLR), 2024.
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and inter- pretable large language model reasoning. arXiv preprint arXiv:2310.01061 2023.
Leena Mathur, Marian Qian, Paul Pu Liang, and Louis-Philippe Moreney. Social genome: Grounded social reasoning abilities of multimodal models. arXiv preprint 255: 1590 2025.
Seyed Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Fara- jtabar. Gsm-symbolie: Understanding the limitations of mathematical reasoning large language models. In The Thirteenth International Conference on Learning Representations, 2025.
Philipp Mondorf and Barbara Plank. Beyond accuracy: Evaluating the reasoning hehavior of large language models-a a survey. arXiv preprint arXiv:2404.0/866 2024.
Arseny Moskvichev, Victor Vikram Odouard, Melanie Mitchell. The ConceptARC Benchmark: Evaluating Understanding and Generalization in the ARC Domain. 2023. doi: 10.48500/AXX1123300071111
OpenAl. Openai o3 and 04-mini system card. https://ch..peeaai.co ppt///
2221 1221c8755 c875-8755-00 02dc - 4 78 000033880000777777222722820 /0 1el3--nndo4--innne a
2025. Accessed: 2025-05-16.
Puzzled Pint. Puzzled pint. https::/ppzzleedpiin
2025. CCBY-NC-SA Intl. 4.0.
Qwen. Qvg: To see the world with wisdom, December 2024. URL htttps://wwenl githenmlmggittbbb.....m ii/
blog/ qvg- 72b /2b-preview/
Shashi Pal Singh, Ajai Kumar, Hemant Darbari, Lenali Singh, Anshika Rastogi, and Shikha Jain. Machine translation using deep learning: overview. In 2017 international conference computer, communica- tions and electronics (comptelix), pp. 162-167. IEFE, 2017.
Marco Antonio Calijorne Soares Fernando Silva Parreiras. A literature review on question answering techniques, paradigms and systems. Journul of King Saud University- and Information Sciences, 646,, 2020.
Winnie Street, John Oliver Siy, Geoff Keeling, Adrien Baranes. Benjamin Barnett, Michael McKibben, Tatenda Kanyere, Alison Lentz, Blaise Aguera Arcas, and Robin I. M. Dunbar. Llms achieve adult human perfor- mance higher-order theory of mind tasks, 2024. URL httpt::ttt//tttt orr http://gggaaaaaaaa/ee abs/2455. 18870
12

---

Published as a conference paper at ICLR 2026
Hai-Long Sun, Zhun Sun, Houwen Peng. and Han-Jia Ye. Mitigating visual forgetting via take-along visual conditioning for multi-modal long CoT reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), pp. 5158 5171, Vienna, Austria, July 2025. Association for Computational Linguisties. ISBN
doi: 2118663/11/2025.00 10.16653///022..cclloo....... URL htttss://allanthol..gg org/ 2325 acl.-aaclll. lngg 257/
Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. A benchmark for learning to translate a new language from one grammar book. arXiv preprint urXiv:2309.100 165755 2023.
Kimi Team, Angang Du, Bohong Yin, Bowei Xing, Bowen Qu, Bowen Wang. Cheng Chen, Chenlin Zhang, Chenzhuang Du, Chu Wei, et al. Kimi-vl technical report. arXiv preprint arXiv:250.007911, 2025.
Clinton J Wang. Dean Lee, Cristina Menghini, Johannes Mols, Jack Doughty, Adam Khoja, Jayson Lynch, Sean Hendryx. Summer Yue, and Dan Hendrycks, Enigmaeval: A benchmark of long multimodal reasoning challenges. arXiv preprint arXiv:2502. 08859, 2025.
Jiayu Wang. Yifei Ming. Zhenmei Shi, Vibhav Vineet, Xin Wang, Yixuan Li, and Neel Joshi. Is a picture worth thousand words? delving into spatial reasoning for vision language models. The Thirty-Eighth Annual Conference on Neural Information Processing Systems, 2024a.
Xiaoxuan Wang. Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam. Arjun R. Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. SciBench: Evaluating College-Level Scientific Problem- Solving Abilities of Large Language Models. In Proceedings of the Forty-Firs First International Conference on Machine Learning, 2024b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quac Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Adhika Pramita Widyassari, Supriadi Rustad, Guruh Fajar Shidik, Edi Noersasongko, Abdul Syukur, Affandy Affandy, and De Rosal Ignatius Moses Setiadi. Review of automatic text summarization techniques & methods. Journal of King Saud University Coimere and Information Sciences, 34(4):1029-104, 2022.
Tongshuang Wu, Michael Terry, and Carrie Jun Cai. Ai chains: Transparent and controllable human-ai inter- action by chaining large language model prompts. In Proceedings of the 2022 CHI conference on human factors in computing systems, pp. 1-22, 2022.
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind's eye of llms: Visualization- of of-thought clicits spatial reasoning in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
XAI. Grok-4. https://x.ainneww//ookk4 2025. Accessed: 2025-09-23.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths. Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information process- ing systems, 36: 18:9 11822, 2023.
Kaining Ying. Fanqing Meng. Jin Wang. Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang. Yuqi Lin, Shuo Liu, Jiayi Lei, Quanfeng Lu, Runjian Chen, Peng Xu, Renrui Zhang, Haozhe Zhang, Peng Gao, Yali Wang, Yu Qiao, Ping Luo, Kaipeng Zhang, and Wenqi Shao. Mmt-hench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi, 2024.
Eunice Yiu, Maan Qraitem, Anisa Noor Majhi, Charlie Wong, Yutong Bai, Shiry Ginosar, Alison Gopnik, and Kate Saenko. Kiva: Kid-inspired visual analogies for testing large multimodal models. arXiv preprint arXiv:2407. 17773, 2024.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu. Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: massive multi- discipline multimodal undcrstanding and reasoning benchmark for expert agi. In Proceedings of CVPR. 2024a.
Xiang Yue, Yuansheng Ni, Kai Zhang. Tianyu Zheng, Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi- cittidiscipline multimodal understanding and rea- soning benchmark for expert agi. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pottern Recognition, pp. 9556 2024b.
13

---

Published as a conference paper at ICLR 2026
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuo- han Li, Dacheng Li, Eric Xing, Judging llm-as-a-judge with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 36:46595 46623, 2023.
Yaowei Zheng, Richong Zhang, Junhao Zhang. Yanhan Ye. Zheyan Luo, Zhangchi Feng, Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguisties. URL http:///axx.v.oott/bb//2223
139/2
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, arXiv preprint 2r0v222004. 10479, 2025.
14

---

Published as a conference paper at ICLR 2026
A LIMITATIONS AND BROADER IMPACT
To ensure consisteney and standardization across the dataset, we excluded puzzles involving under- explored or toorepeettt modalities such as audio, video, or interactive file-based inputs. As result, PUZZLEWORLE may not fully capture the breadth of sensory and interaction- based reason- ing found in some real-world, more challenging puzzlehunts. We discuss the potential incorporation of more challenging puzzles Appendix C Additionally, unlike unlike/Wang et al. (202)) that uses human annotators to transcribe textual and visual components separately, we preserve the puzzle content in its original image format and focus annotation efforts on intermediate reasoning traces. While this allows PUZZLEW ORLD to provide richer annotation of the solution reasoning process, it may also introduce variability in model performance depending on the quality of their OCR capabilities. We discuss whether OCR is limiting capability for frontier models in Appendix [.2]
Finally, our evaluation pipeline relies on L.L--bseed judges automatically assess generated rea- soning traces. To address this, we adopted careful prompting and cross-checking. For example, we enforce that each individual annotated step is evaluated separately by the LLM judge to determine whether it matches the model generated solution. We tested the alternative approach, where the LLM judge is prompted with the full candidate solution and outputs the latest ground truth step that the candidate response achieved. However this approach judge sometimes outputs a stepwise accuracy greater than 1. As such, our approach of rumning the LLM judge to output boolean on each ground truth step helps mitigate potential hallucination. Nevertheless, we acknowledge that the use of M.-M-aased evaluations may be subject to instability or bias, and the metries should be taken with caution.
more prone to hallucinations, LLM
One possible concern with our grading scheme is that model might "hallucinate" the correct final answer without engaging in proper reasoning. However, such case is extremely rare in Puz- ZLEWORLD. The high-quality, human-designed puzzles are deliberately constructed to discourage superficial guessing, and even experienced human solvers cannot easily infer the answer without following the intended logic. In a thorough manual inspection of models' puzzle responses, we did not find any case where the model arrived at the final answer without demonstrating the necessary reasoning. In Appendix D.3. we discuss whether our stepwise scoring mechanisms successfully credit alternative solution paths.
Our goal in releasing PUZZLEWORLD is to advance research in general-purpose, multimodal rea- soning systems. However, we recognize that increasingly capable AI models, especially those skilled at complex reasoning, carry risks of misuse. These include the potential for externalizing replacing human reasoning in settings where authenticity creativity is essential, such as ed- ucation, scientific authorship, or collaborative problem-solving. While our dataset does not pose direct risks its own, support future work that includes safeguards mitigate misuse and encourages the responsible deployment of ceasalee AI systems in alignment with human values.
B USE OF LARGE LANGUAGE MODELS
This project used Large Language Models (LLMs) to assist dataset verification and model evalu- ations. Details are described in the Sections 4.3 and 5.1 We also acknowledge the use of LLMs to assist with correcting grammatical errors and improving clarity of the writing. This assistance was limited to language refinement and did not affect the core methodology, scientific rigour, or originality of the research. We confirm that no Al-generated content has been presented as our own intellectual contribution.
C FUTURE DIRECTIONS
We note that some prior benchmarks (Chen et al., 2025a) adopt a structured reflective reasoning paradigm, centered on structured, text-based puzzles with explicit rules, enabling precise state verifi- cation and surgical analysis of multi-step logical reasoning. In contrast, PUZZLE WORLDemphasize multimodal, discovery-drive driven reasoning in which the puzzle rules are implicit and must be inferred by the solver. This setting offers more holistic evaluation of generalist reasoning capabilities that requires a broader range of cognitive skills, though at the cost of less precise, -Lp-apprxedmtate intermediate state verification due to the absence of explicit nule structures.
15

---

Published as a conference paper at ICLR 2026
An exciting direction for future work is to extend PUZZLEWORIE with structured, verifiable reflective-reasoning frameworks akin to those in Chen et al. (025a). However, we caution that PUZZLE WORLD'S open-ended and multimodal puzzles often involve diverse, creative reasoning steps that are difficult to formalize as explicit rules, making it challenging to achieve the same level of rule-based granularity as in more structured puzzle domains in Chen al. (2025a). That said, integrating reflective reasoning paradigms with PUZZLEWORLD could yield richer diagnostic tools and may ultimately benefit the training and evaluation of frontier reasoning models.
We acknowledge that, beyond PuzzledPint, extensive puzzlehunt-style puzzle sources could be inte- grated expand TUZZLEWORLE such as MIT Mystery Hunt (MIT) 1981-). While MIT Mystery Hunt is an attractive source of challenging, high-quality puzzles, we note several considerations that make it challenging to integrate it to WORLD's initial release.
Unlike PuzzledPint (released under Creative Commons), MIT Mystery Hunt puzzles do not have centralized copyright policy. Permissions would need to be obtained from individual authors, which adds substantial overhead.
Mystery Hunt puzzles frequently involve audio, video, interactive web tools, custom code, and even physical components. These are exciting modalities for future generalist AI sys- tems, but they are difficult standardize under PUZZLEWORLD'S stepwise annotation format and are not well supported by current frontier models.
Mystery Hunt puzzles intentionally designed to challenge top teams and often require multiple people to solve each puzzle. Given that frontier models already struggle on the entry-level PuzzledPint puzzles, including much harder Mystery Hunt puzzles would likely provide limited practical diagnostic value at this stage.
Therefore, for the present, we prioritize the more accessible PuzzledPint puzzles, as they are both suitably challenging and sufficiently structured to support reliable stepwise annotation and informa- tive intermediate diagnosties. That said, as models and annotation tools improve, future work that extends WORLD to incorporate more advanced modalities and higher-difficlly sources potentially including Mystery Hunt puzzles would offer a richer evaluation suite.
D ADDITIONAL DISCUSSION
D.1 ON THE ABSTRACT NATURE OF PUZZLEHUNTS
We note that PUZZLEWORLD like prior puzzle datasets (Chia et al., 2024 Ghosal al., 2024 Estermann et al. 2024), is abstract in nature rather than drawn from real-world tasks such as math- ematics or physics. While this means that PUZZLEWORLD does not directly evaluate model's ability on any specific application, this abstraction is intentionally designed to assess generalist, open-ended reasoning while reducing pattern memorization. For example, prior work has shown that symbolic variants of math datasets can reveal brittle reasoning behavior in models that oth- erwise appear strong (Mirzadeh et al., 2025). PUZZLEW ORLD's D's abstraction thus helps isolate and measure model's generalist and genuine reasoning ability beyond domain-specific patterns.
Existing reasoning benchmarks ranging from math and coding tasks - typiccalyy operate in closed- ended, well-defined environments. These datasets assess correctness under fixed rules, but they do not require models to navigate underspecified or open-ended problem spaces. In contrast, many real-world tasks (e.g., exploratory data analysis, investigative research, science discovery) involve ambiguous signals, multiple possible solutions, and iterative hypothesis testing.
PUZZLEWORLD fills this gap in two ways that previous abstract reasoning benchmarks do not. First, unlike traditional puzzle datasets (e.g. Sudoku) that evaluate within limited rule set, our puzzle- hunt puzzles require leveraging diverse reasoning competencies to dynamically form and evaluate hypotheses from nuanced multimodal signals. Second, our manually annotated reasoning traces enable systematic analysis of model behavior in these open-ended settings, revealing characteristic failure modes of frontier models such as "myopic commitment" (Section 5.4). Our transfer experi- ments in Table has also shown that capabilities exercised in PUZZLE W ORLD transfer to real-world benchmarks, suggesting that puzzlehunt-styl reasoning is a useful proxy for generalist reasoning. Nonetheless, acknowledge that no single benchmark can represent all forms of reasoning.
16

---

Published as a conference paper at ICLR 2026
Visual Forgetting
GPT-o3 with transoribed text
+ A
Puzzled Pint
Just Ach Hall'ss Scom
toarmi lora shalls ofliie statmm tee las ie sovee ree slownomm
dacaasa auri hes anguuain
Add elthor or
Clstai FAMINE, CONTAINMENT, SPICE..
From The San thes eary parties beren
Index into the transformed words with the situation to get
f map enrl
transcribed
Stmake35
Wins saily aulaggair to po dhuuu
Add either "SC" or "AM" to each inalicized word.
Stmake41
Woln derrut uataraaaaat lane wulk help ins bolondd
5tmake33
acunto puni
Obtain FAMINE, CONTAINMENT, SPICE..
Wionred io au un wnbbrakk alaisa alosa vou your
saual your bite deesing waades (3)
Reading
gives SCSCSC..
themite the beit being lnv iwalitiandl
Based on the situation order this gives 12345..
Whemee dloneting tymm yore growns yoursolff
The handbook tells apprentice art-thief improve. 12345667 spells out PROTEGE.
Figure 10: Example error of visual forgetting. GPT-o3 failed to correctly reference the text clues in the puzzle image, but it solved the puzzle once the clues were provided in transcribed text form.
D.2 ARE REASONING MODELS BOTTLENECKED TEXT RECOGNITION?
We acknowledge that multimodal large language models may be bottlenecked by text recognition in images. In the construction of PUZZLEWORLD since Wang et al. (2025) has shown that today's 's best multimodal reasoning models capabilities, we devoted our manual effort to construct fine-grained annotations. Although each puzzle includes a munan--ranscribed "flavor text", this may not capture all textual content.
primarily constrained by optical character recognition (OCR)
We thus provide an analysis on whether OCR is limiting the performance of frontier models on PUZZLEWORI.D. We selected 5% subset of strictly text-only puzzles, manually transcribed all textual content, and compared GPT-o3's performance with and without the full transcription. A paired t-test showed no significant difference (t 0.52. p > 0.1), suggesting that text recognition limitations are not major bottleneck for the frontier multimodal models.
That said, we did observe isolated cases where GPT-o3 improved from partial to full solutions transcription
provided. Upon examination, these instances appear stem from model hallucination in later stages of reasoning when relying only on images, whereas the transcription helps anchor its reasoning more reliably. For example, in Figure I0 GPT-o3 solves the puzzle correctly when given the transcribed text. When relying solely on the image, GPT-o3 can solve the intermediate text clues but fails during extraction: misreferences the visual text, as shown by its inability to recover the added letters against the italicized words in the image, and hallucinates a final answer only tangentially related to the flavor text. This aligns with patterns of visual forgetting observed in prior works (Sun et al. 2025).
D.3 CAN PUZZLEWORLD STEPWISE SCORING CREDIT CREATIVE SOLUTIONS?
Due to the PUZZLEWORLD. open-ended nature, legitimate concern is whether PUZZLEW ORLD's stepwise scoring mechanism can fairly credit creative, alternative reasoning paths, rather than being limited the annotated reference chain. In puzzlehunts, the solution steps are intentionally designed to be interlocking and sequentially dependent. It is thus hard to reach the correct answer while following completely different solution path, and correct partial chains must partially converge with the annotated -ound-truth, even if the solver temporarily deviates or makes logic leaps.
That said, human puzzlehunters do commonly make educated guesses that skip over intermediate steps. For example, after identifying three of the four final answer letters L solver might correctly infer "ICLR" without fully completing the preceding clue. PUZZLEW WIrZLEWWRLLD 's step-level scoring explicitly accounts for this behavior: identify the latest ground-truth step that appears in the candidate solution. In this scenario, because the solver correctly arrives at the final answer, our scoring grants full credit, if several intermediate steps were omitted or approximated.
17

---

Published as a conference paper at ICLR 2026
Table GPT-o3's performance on PUZZLEWORLD by difficulty.. GPT-o3's accuracy monoton- ically decreases as difficulty increases.
Difficulty
Accuracy
Stepwise
Easy
18.46%
40.58%
Medium
14.76%
39.38%
Hard
9.64%
39.71%
Table 6: GPT-o3's accuracy on PUZZLEWORLD by difficulty and modality. Within each modal- ity, GPT-o3's accuracy consistency decreases with increased difficulty, with relatively well perfor- mance on easy structured puzzles.
Difficulty
Text
Visual
Structured
Easy
19.5
11.9%
20.4%
Medium
15.3%
8.9%
13.5%
Hard
10.6%
7.6%
9.6%
As a proxy for evaluating how robust our step-level crediting is to alternative partial chains, we randomly sampled 10% of PUZZLEWORLD and automatically paraphrased GPT-o3's solutions ing GPT-5-mini. We then compared the LLM-judge's step-level scores on the original and para- phrased solutions. We observe strong positive correlation between the sets of scores (r 0.885, < 0.001) with a mean absolute eπтоr of 0.066, indicating high degree of score stability across semantically equivalent but structurally varied solutions. Although this does exhaustively cover all possible reasoning paths, it indicates that the LLM-judge can reliably credit alternative solu- tions.
E PUZZLEWORLD DETAILS
E. CHECKING FOR DATASET CONTAMINATION
To assess the possibility of data contamination, we test whether GPT-o3 (OpenAI 2025) has memo- rized any of the puzzles in our dataset. Specifically, inspired by prior work Tanzer al. (023); Chi et al. 2024), we prompt the model to reconstruct the flavor text for 40 randomly sampled puzzles out of the 84 that
answered correctly. We then use GPT-4o (Achiam al., 2023 to automatically evaluate the similarity between the reconstructed and original flavor texts. We find reconstruction evidence of data leakage. Furthermore, since Puzzled Pint (Puzzled Pint 2025) publishes new puzzles on monthly basis, our dataset can be continuously updated mitigate the risk of model overfitting on released content.
accuracy of 0%, suggesting little
E.2 DIFFICULTY LABEL ANALYSIS
We acknowledge that the difficulty labels in PUZZLEWORLD are obtained from the original Puz- zledPint sources, without manual calibration. According to the PuzzledPint website (Puzzled Pint, 2025), all submitted puzzles undergo internal playtesting by an editorial team, where the difficulty tags are revised and finalized by expert editors and reflect community-standard difficulty assess- ment. We thus treat them as expert annotations rather than ad-hoe metadata.
To understand whether the human-labelled difficulty labels meaningfully reflect task difficulty, inspect GPT-o3's performance broken down by difficulty in Table As expected, we observe that GPT-o3's accuracy decreases as difficulty level increases. Another valid concern is that human- perceived difficulty and Al-perceived difficulty might not align. For example, certain diagram- heavy puzzles may be straightffrward for humans but require nontrivial spatial reasoning for Als. To investigate this, we further break down GPT-o3's performance by modality and difficulty.
As observed Table 6 within each modality, accuracy consistently decreases with difficulty. No- tably, GPT-o3 performs relatively well on easy structured puzzles - likely due to their clean, regular layouts- but its performance drops sharply on harder structured puzzles where the diagrams be- come more irregular and visually complex. Overall, both aggregated and modality-leve results show that PuzzledPint's 's difficulty labels provide a meaningful and consistent difficulty metric for AI models.
18

---

Published as a conference paper at ICLR 2026
E.3 PUZZLEWORLD IMAGE SAMPLES
We provide high-resolution images of puzzle samples used in this paper.
Puzzle Pinl JUNE 3054
MAKI
Location rucer Stephanee muntioned to her recemmate fodl that ahe dich't know heariig dfftreet
there muttal rype
to
At sushi rastaren, Madaaarrt,,,nn Katik contimued Stuphari''s cresh cours sush,
"Wi'll basies" Katte uaid. "You mandioned have only
Califernia rolls burtinn Thois limeo the casegory of mail or sudii rolls There's
seaweed and rice on the outside the maln susti, and the filing componennts
conter. make chefl la
other that they awarlap uxtent an then high
"Wow, seee" Stephonie said "This purticular piecoe enomooss. Here, let's spiit down lange?"
"Ii not sung," Karta replied. "Bue wirl
Former bride
Sworm aleglnnc
fcoling more inorse than
Prychic communicator, 4g.
Consibrate
"Pan"" folover
One three famous
12
Fox
13. c.g. Sun combuttion,
more Enss
15. For jes an ilstration
creatree
Maryeen''sslee neptle
20. Shorter aturmative to signature
Sephare (tustar,
Puzzled Pint
QUESABIRRIA TYYYY
MAY 2023
quesodilias cocos, quesabirii stewed, goat, jackinuit together together. hardsgressed long-ámmeredt welf-reduced
eearty
consomme dipping
Assertive recidess
Assistance (3)
Attempt overfill (4)
Bountiful celebratory meal
Dlasitication of vinegar or dirrs juice
Currency in London (5)
Euphemism for
covering that greener where you aren't
High fat dairy liquid (5)
Kind and polite (4)
Launderer's
(5)
Man who wrote about a raven (3)
Noted Vatican resident
Quick (4)
Russian dictator (6)
celebration Orleans
Sibling's s daughter (5)
Site for restaurantrevieew reviews (4)
Suppart garments
19

---

Published as a conference paper at ICLR 2026
Puzzli
Pint DE
Arthimedes and Borjamia Barnelor are dfting on bonch, hovting a discamiio.
Avcible friend,
drewing don't
willing subject of kolnapping another dinenaio??
Berple: Wow, Artion, leewer thought of that way |ust knsw (hey always seens make Dary foclings
hopelees
angleal
takesa tueee poizrs to define angle, and I've always Lited Broes Thrce mullertocss theoe loggod races, theee eytlotble weeds.
Burpie: angles makk many
ahapes
Archae: know,
Friedrich Gazas obtuae, angles have always aeemsd ide!
and tries join
Casl: Hay drawing
Auchie: USa eamuse you This was an A convecaation, to you can your way out
Y
D
W
Sn TEI
A
Y
2004 CC BYANG 4. Rodtett Becker MOJ
Puzzied PinlAAAA
M in en
thtee forcetully inlocated of ereplovinggte his new home. To further com plicate matters, the mandion has foue lewels, and his perppettive
rdionn
will ano soop shifting with such new lovell Alorng the way Miltins will discover shorteata puth that heartss which huve make way way up
S
f
9
q
9
Start
X
d
3
b
20002 CE IVNGGA Int, den Zoon (Betse DX
20

---

Published as a conference paper at ICLR 2026
Puzzled Pint APRIL 2021
EROS'S ARROWS
seszion. Each person this speed-dating session in ansigned sumber. Fros used his arcows make each person love with eomeane on other side, everyone loved by love be reciprocated.
only five different arrowheads; each arrowhead does something different. Addinonalit, Eros may sboot up to three arrows from each person get
Eros
even ropeating type requized.
Can you figure out who fell in love with whom?
Posatble arrowhead effects:
20400
number lelttera the person's
Divide by 3
Muliiply
-Subtract 2
-Subtract
1. Blake
A
66. Leslie
2. Charlie 8O
F
77. Jesse
G
H
Z
3. Avery 440
I
L
8. Emersoncv
K
O
N
4. Kerry 4OA .J
W
99. Jamie
5. Parker 80
P
R
10. Alex O
(C) 2021 OO BYANC-- SA tnt. by Siephnii Yog (lockelle, MD)
Puzzkd
A Dolorous Din
Pint
loped whal meaer
21

---

Published as a conference paper at ICLR 2026
Puzzled Pint
MISSING LINKS
tounned seaeffee codaed' jour longin
Every magician good par comning irida my inthought ling ginmick thi insead metal liiked words unoo
artwina
Hightgghoe stantart a
night into
fatteerr (imaltaneoua
-Mourta with
WWariars who
gift
Barbie Raggedy Ane
Appellations
-Femou Giaa structure
feelnn grief Flyes acample uggah mone
16
-Seeppng Lighty
12
Can you Egune out who Mage namesake was?
Puzzld
*** The Venute in Verona
Pint go0)
MyMondee 2017
block their terttory,
Mortagues
chiimed have diviced each
will well, corner corner. makes gesting the weeking traversing maze. Help couple find path that crosses the city, from side to the other, to find out who awarts them there.
like
opposing connect that Unnumbered poi nts can have number of lines meeting them.
every squere, possible directions,
numbered cirile shows esectly how
No erea can be completely walled off. The entire grid must be reschable from loast one of the sides (Lo. do not create any complete squares or rectangles with your line).
between lines you draw, making a
A
W
5
M
R
K
S
K
M
0
E
B
3
2
M
M
EEE
R
M
D
1 O
V
Q
A
X
T
A
C
HTP
0
O
A
M
$
W
R
M
E
N
2
TG
FAH
M
AAR
R
N
EER
IIS
LTO
PTO
O
L
G
U
A
TE R
U
M
0
0
2
2
M
RAN
D
K
C
A
N
R
R
U
NNN
8
W
E
T
W
Muzzle by Nest lnbrwadia (Puntin
22

---

Published as a conference paper at ICLR 2026
F BENCHMARKING DETAILS
F.1 COMPUTE RESOURCES
All evaluations and experiments in this paper were conducted on a remote cluster equipped with two NVIDIA H200 GPUs (each with 141 GB HBM3 memory). Runtime for each model evaluation varied between 10-48 hours depending on the model size and architecture.
F.2 SUPERVISED FINETUNING
All fine-tuning experiments were conducted using the LLaMA Factory framework (Zheng et al, 2024). For fine-tuning the 8B InternVL3 model, used LoRA fine-tuning with rank of 8, learning rate of x 10-6, and trained for epochs on PUZZLE
For the transfer experiments, fine tune Qwen2.5-VIL 11--255-VVL--B----- -Ln--B-Intttt model, using full parameter finetuning with learning rate of 1 x 10-5 for 5 epochs on PUZZLEWORLD We did not perform extensive hyperparameter tuning for any of these experiments.
F.3 HUMAN BASELINE DETAILS
We provide additional context for the human baseline in Table 2 While the human baseline only covers a subset of PUZZLEWORLD the resulting baselines reveal clear separations between human tiers and current models-foe example, >30% gap between human enthusiasts and GPT-o3, This gap is substantially larger than the uncertainty introduced by the sample size, suggesting that the current human results provide reliable directional reference point.
Given our resource constraints, we prioritized depth and quality in the human evaluation rather than exhaustiveness. To reduce individual variability in task as open-ended as PuzzleWorld, each sampled puzzle was solved by least two independent participants, each allotted up one hour and asked to produce detailed reasoning traces. Since we assign multiple participants to each puzzle to ensure consistency, each puzzle modality has more than 40 human attempts, which consider sufficient for modality-level human baseline.
Finally, we note that the same LLM-judge protocol is applied to both human and model solutions, ensuring that the comparison is fair and step-symmetric. We observe that, unlike AI models, visual and structured modalities do not pose additional difficulty for human solvers, further highlighting the multimodal limitations of current foundation models.
F.4 ROBUSTNESS OF LLM-JUDGE TO IMPERFECT ANNOTATIONS
PUZZLEWORLD contains extensive manual annotation to provide fine-grained analysis on models' performance. In this section, we include an ablation on the impact of potential human annotation noise on our step-level grading accuracy. To simulate annotation noise, we select 10% of Puz- ZLEWORLD and use GPT-5-mini evaluate GPT-o3's output on both the original and paraphrased ground truth. We observe strong pos- itive correlation between the two sets of scores (r = 0.898, p < 0.001) with a mean absolute error of 0.056. This indicates that PUZZLEW WORLD's stepwise evaluation pipeline is robust to moderate noise in the stepwise annotations.
paraphrase each step of the ground truth annotation. Then, we
F.5 PROMPT FOR BENCHMARKING
Below is the system prompt template for benchmarking models on PUZZLEWORLD, which adapted from Wang et al. (2025).
You will be presented with pezzle to solve. The puzzle not have spacific Inatructions, snswer
you
paszle
phrese rarely, number).
Do not ask any quastions abous how to proceed, just do your best to selve the pussle. Hare some tips for solving pursles of this types
Ganaral Tips:
Puzeles will often have milsiple steps to get to the answer wordh You can cauelly sell you the zight Lrack if the in-eendiaat ansvers ageve with the Lille, Clavor, or chane
puzsle.
You can uyuelly find hints she rnductayyyy text, For example references to the dack or "sight" are often hiata sonething is uncuded with braille.
Puxrles often Incorporate acrostics: clue where the first letter, syllsble, word of nessage.
recarring you and up with garbled "alghabet soup", chan look for clue on how to order then. Indexing is one of the nost common pozsle recharisns. Try indexing when you have Hist of words or ghrases and cocrespooding list of nurbers. Count into the word phraze by the letter that position. For exarpLe: Cake, Padding,
line, paragraph,
spells
given nuncer and record Shortening ng* gi ves you *
23

---

Published as a conference paper at ICLR 2026
Alpha nunerie codea are vesy common Ic you up
nunbers czy zeplacing
the numbers with the corraspanting lettars like chis: 1 A, 3 C... 26 2.
occasisnallyy, these types of codes will "weap cround", so doz despair it you cze In
26 atc. you try
127-25
d,
docan't work, try other numaric codes
ruch as ASCII.
Often puzzle zepeata stratecy multiple Limes.
likely need backtracks lyequenlly, so make ause wail out your alupa as you go. If you get stuck, txy to think of a now way to approach the puzzle. Tey:
Screading the title and the flavor text. These are the nast Laportant hinze about what type rnfurences night be Chacking pop cuiture saferences
sisalegies,
Chocking for references to a sorg/ haana//ocn//oo/to www.//book//eteee book/// show
night try include:
Alghabet lxing
Using leftover lettera to spell sonething
Bearsanq-ng the lettors (aka anagrana or "crenspesine";
Sewing
any acronyns Diagonalizing Itaking
fiee: lettez
anaver, secund lester aecond
answes, etc.)
looking for unusuel letter Crequnniiss
bonephenea
Shifting from letters cunbers
For numbera, tzy:
lesters
Using it as phone namber
Treating nunbera as datea
Treating numbers ASCIT numbers
any alzange sequences involved
Beeing prino
Fox images, Lay:
booking St
Squinting ing at from ewey
filling Lt
looking it upsidde down
looking through
Tsanseribing ibing meally
We additionally append the user prompt:
Your cask so selve the tollowing puzzle. The attached imaqes are presented the ordar refereneed
text.
The puzzla'a sitle in: I)
The puzzic's flaver text is:
Write sut a step-by-stop solution to the puzzlo. At the end of your sclution, write your snewer in the tol lowing tornat:
hnswer:
Below prompt for LLM judge:
Answms Bquivalence Instructions::
Bsing the puzzle and the refarencs solution, grade the candidace solution as follows.
every reasening alep of the refecmnow
candidate solution
the step and echieves the same intermediate rexult of the step, ocherwise False.
Explain why the candidaters solution did did not get the ressoning step cocrec.. solution evaleate every step
Da not in the refwrence solut ian,
steps
Theze is a exception fon in sporing for the last ceasoning step. Idencify the cardidace output salus valution
IE the candiate outpat sol ion is the
the zeferwnow solut ion
(pazziee_eo eeuoteet_olooo ion]
then output El Final step as true.
G ANNOTATOR DETAILS
We employ university undergraduates to assist the human annotation process in PUZZLEWORLD. All annotators are compensated at a rate of $16.00 per hour. Prior annotation, annotators receive detailed guidelines and participate in training sessions to ensure annotation consistency and task understanding.
G. ANNOTATOR INSTRUCTIONS
We provide the instructions given to annotators below:
ruct for Submitting
24

---

Published as a conference paper at ICLR 2026
submit puzzle, this sepository and create naw branch. Then, create folder "jpuzsio_name)" in the "data/pezzles" folder, and place the following files in it:
"metadata, jeen's
file containing netadata the puzzle puzzia content
"content..pag" inace "tiqure_;K] :paag" (optisoal) Fiqures illastreting the reasening stops
example pezzic,
"data/perzies//nneee exammple foldor.
sugueat awage your branch anto the nain regusilary.
Rota, please replace any spaces in the puzzie name with shen creating the new rolder!
Mecadata
The "nctadets..pow tile should concain a JSeN object with the following riclds:
Field
Description:
Lille ateing The ti:le of the pozzle
flaver text strine The clavor text of the pezzle, possinly enety
diffieulty stzing solction
difficulty level of che pusxle teasy, medium,
solution atzing
puzsle
reasoning solution
Stopht M Ac. ordered 11st of reasoning
treassonttttt bowards the
nodality
atzinght
1iat inpat [nodalities] (ea 1el--lss------pp t- modd-iioosiii
contains
akills puzzle
ateinght M A 11at of [skilial (ta List of reasesing skills) segulted to solve Lhe
1H1 Reasening Step
The "ressoning" Field should concain 1ist of 'Step' objects, which regresenced diet illioneriee iss
fellowing fielda: Fiold Type Dascriptien
in string The sex.tal explanet iun of Uhe alep
Figure
path (Optional)
pash Eigure illusraatin step
Each of the explanat lon should begin with one of the following atunic aulions: Pattern sosvery: discover pacterns insights from current information
R.g- inovering
patterns senaphores
Skatchings sketching on or interacting with visual elenenta
Rig. traversing through mare
B.g. connecting ing the dota
Manipolation: manipulating
azranging sequerce elemence R.g. sorting el phabets in order
Rig. applying cryplic ancoding! decoding
combining Casining: combining chaining multiple pieces observations
Rig- natching patterns images
segments
Extract lons excting information from are pattern observation
Big. extracting letters tron sensphore patterns
wording or actior important long resembles the above categories!
Note:
Each explanation should consist actior the intermeciato cutcome action Identify the pattern (..,,
Go.d
1t A tist et Input Modalitios
Keyword Description
'text' "visuel muctured visuel information e-g. inages, icons, fonce, etc.
Textual intormation
'structured' Structurod informatica tables, graphe, crosswords,
1t A tist of Reasoning Skills Keyword Description
'logie" intornation
Logie ressening e.g. rulc decuction or inferriag conclusion given partisl
'wordplay"
Manipulating words based Haquistic proparties
homoptones,
'spatial"
Spacial visual unserstanding, manipulation and ravigation e.g. nazes, connecting
dote, etc.
'aryotiz" Brcoding and decoding information c.g. ciphers, iadexing, etc.
Severarging semair-specitic knowledge histery, eciencs, "commnnsnnse" Applying common reascaing e.g. paysics. laxs, noms, usool_use* Searching through an excernal detabaso for information uniikely in model's tssining
'knowledge"
data, such Google Haps
25


Reviews:
[{"review_id": "YhRW0AKjyq", "reviewer": "Reviewer_8chG", "summary": "The paper introduces PUZZLEWORLD, a benchmark compiles 667 real puzzlehunt problems to test open-ended multimodal reasoning, adding stepwise solution traces, modality and skill tags, and an LLM-judge for intermediate scoring; frontier models perform poorly overall (best ≈14% answer accuracy, ≈40% stepwise), fine-tuning on reasoning steps improves stepwise scores but not final accuracy.", "questions": "I will recommend the authors to increase the rigor of the benchmark, as discussed above.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- open-ended puzzles obtained from real world, filling a current research gap. \n\n- the paper demonstrated a clear performance gap that stresses present models; best model only has ~14% answer accuracy", "weaknesses": "- the work is very interesting, but unfortunately not very thoroughly done, for examples:\n    - The “easy/medium/hard” tags come from original puzzle metadata rather than a benchmark-defined rubric. There is no formal difficulty calibration, no solver-time distribution analysis, and no additional validation along the annotation. \n\n   - Humans are grouped into novice/enthusiast/expert tiers, but there is no modality-level breakdown and no step-level scoring symmetry with models. In addition, the human evaluation is only for 5% of the samples, thus the comparison in table 2 is not as informative as it appears. \n\n- The benchmark comes with one gold standard chain per puzzle , yet there is a chance the puzzlehunts can have multiple legitimate solve paths. There is no mechanism for crediting alternative correct partial chains or heuristic leaps, thus the evaluation of the reasoning paths can be quite limited.\n\n- the significance of the benchmark may be limited in the sense that the problems and solutions are quite scoped with the culture of puzzlehunt. These specialized cognitive challenges might not well generalize to a broader scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GSw7EGIiUt", "reviewer": "Reviewer_NwBy", "summary": "This paper presents PUZZLEWORLD, a benchmark of 667 real-world puzzlehunt problems for evaluating multimodal, open-ended, multi-step reasoning. Each puzzle includes detailed reasoning traces, modality and cognitive-skill labels, enabling fine-grained diagnostic analysis. Experiments show that even frontier models achieve only 1–14% final accuracy, while fine-tuning on reasoning traces improves stepwise accuracy from 4.8% to 11%. The benchmark is clear, rigorous, and highly relevant to general reasoning research.", "questions": "– Please add a short discussion contrasting PUZZLEWORLD with FINEREASON, emphasizing that PUZZLEWORLD extends structured reflective reasoning to multimodal, discovery-driven contexts.\n\n– Evaluate robustness to imperfect annotations and potential cross-benchmark transfer.\n\n– Future work could merge PUZZLEWORLD’s multimodal puzzles with reflective reasoning frameworks for richer diagnostics.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) Proposes the first large-scale open-ended puzzlehunt benchmark, transforming real Puzzled Pint puzzles into machine-readable tasks that test discovery-driven reasoning.\n\n(2) A two-stage GPT-4o + human verification pipeline ensures 96.5% correctness and no contamination.\n\n(3) Evaluates eight major models with both final and stepwise metrics, identifying systematic weaknesses (myopic reasoning, language bottleneck, lack of sketching).\n\n(4) Reasoning-trace supervision improves intermediate reasoning and transfers to Rebus and MathVista tasks.", "weaknesses": "(1) The paper omits discussion of FINEREASON (Chen et al., 2025; arXiv:2502.20238), which also studies reflective puzzle reasoning through step decomposition. Comparing PUZZLEWORLD’s open-ended, multimodal puzzles with FINEREASON’s structured logic ones would clarify its unique scope.\n\n(2) Limited ablations on annotation noise and prompting baselines; fine-tuning details appear only in the appendix.\n\n(3) Dataset excludes audio/video modalities and depends on OCR, limiting breadth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tl2ZPvx1Xe", "reviewer": "Reviewer_vb72", "summary": "The authors introduce a dataset of 667 \"puzzle hunt\"-style puzzles, complete with detailed annotations of reasoning for how to properly solve the puzzle. The authors discuss how this dataset is constructed, and how the annotations are carefully done to ensure no ambiguity in puzzle answers. They evaluate frontier models on the puzzles, and report the performance of both closed- and open-source models. The paper then investigates the effect of fine-tuning on reasoning traces on better performance on other puzzles, and expounds a detailed error analysis of current-generation VLMs.", "questions": "Would the MIT Mystery Hunt archives be a good source to expand a dataset like this?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "I really like this dataset, both in terms of the intrinsic open-endedness of the puzzles, and the fact that the puzzles have an unambiguous and easy-to-evaluate final answer; I think this is a strong contribution to the field. I also appreciate the attention paid to making the dataset correct and unambiguous. The paper itself is well-written, and the figures are informative. I like the detailed analysis in section 5.4, and appreciate the contamination check in C.1. In all, this is a valuable dataset which was constructed carefully, and an excellent paper analyzing current performance.", "weaknesses": "No obvious weaknesses beyond the classic advice that more data points would be better :) Also, it would be nice to re-evaluate on all the new models that came out since the submission deadline, if that's not too hard!", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G5xYzcYoSO", "reviewer": "Reviewer_s7Za", "summary": "PUZZLEWORLD introduces a benchmark of 667 puzzlehunt-style problems designed to evaluate multimodal, open-ended reasoning in AI systems. Unlike conventional benchmarks with well-defined tasks, puzzlehunts require discovering problem structures from ambiguous, multimodal clues, mirroring real-world challenges like scientific discovery or investigative problem-solving. The dataset includes detailed annotations of solutions, step-by-step reasoning traces, and cognitive skill labels. Evaluations show state-of-the-art models achieve only 1–14% final answer accuracy, with the best model (GPT-o3) matching novice human performance but lagging behind enthusiasts and experts. The benchmark also enables diagnostic analysis, revealing model limitations in backtracking, visual reasoning, and sketching capabilities. Fine-tuning on reasoning traces improves stepwise accuracy, demonstrating the utility of PUZZLEWORLD for advancing general-purpose reasoning.", "questions": "See the above weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. PUZZLEWORLD fills a critical gap by focusing on open-ended, discovery-driven problems rather than constrained tasks. Its emphasis on multimodal clues and unstructured problem-solving aligns with real-world scenarios, providing a more holistic evaluation of reasoning capabilities.\n2. The dataset includes meticulously curated step-by-step reasoning traces, cognitive skill labels, and modality tags. These annotations support fine-grained diagnostics, error analysis, and model training, surpassing existing benchmarks in depth and utility.\n3. The study comprehensively tests frontier models (e.g., GPT-o3, Claude Opus) and human baselines, highlighting significant performance gaps. Stepwise accuracy metrics offer nuanced insights beyond final answers, revealing intermediate reasoning failures.", "weaknesses": "1. I think the experimental analyses are not sufficient. For example, we see that almost all MLLMs perform poorly, but the reasons behind this are worth further investigation. Is the model's poor performance due to poor performance in the visual modality or poor performance in the text modality? In my experience, most MLLMs are unable to perform well in downstream tasks due to their insufficient ability to recognize text in images.\n2. The article lacks discussion of more related work, such as [1].\n3. I look forward to seeing more case studies.\n\n[1] LatEval: An Interactive LLMs Evaluation Benchmark with Incomplete Information from Lateral Thinking Puzzles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)