CrossWordBench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generation
Authors: Jixuan Leng
Chengsong Huang
Langlin Huang
Bill Yuchen Lin
William W. Cohen
Haohan Wang
Jiaxin Huang
Venue: COLM 2025 (2025)
Keywords: LLMs
LVLMs
Evaluation
Benchmark
TL;DR: We propose CrossWordBench, a benchmark to evaluate the reasoning capabilities of both LLMs and LVLMs though the medium of crossword puzzles.

Abstract: Existing reasoning evaluation frameworks for Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) predominantly assess either text-based reasoning or vision-language understanding capabilities, with limited dynamic interplay between textual and visual constraints. To address this limitation, we introduce CrossWordBench, a benchmark designed to evaluate the reasoning capabilities of both LLMs and LVLMs through the medium of crossword puzzles—a task requiring multimodal adherence to semantic constraints from $\textbf{text-based clues}$ and intersectional constraints from $\textbf{visual grid structures}$. CrossWordBench leverages a controllable puzzle generation framework that produces puzzles in two formats ($\textit{text}$ and $\textit{image}$), supports adjustable difficulty through prefill ratio control, and offers different evaluation strategies, ranging from direct puzzle solving to interactive modes. Our extensive evaluation of over 20 models reveals that reasoning LLMs substantially outperform non-reasoning models by effectively leveraging crossing-letter constraints. We further demonstrate that LVLMs struggle with the task, showing a strong correlation between their puzzle-solving performance and grid-parsing accuracy. Our findings highlight limitations of the reasoning capabilities of current LLMs and LVLMs, and provide an effective approach for creating multimodal constrained tasks for future evaluations.

Published as a conference paper at COLM 2025
Cross CrossWord Bench: Evaluating the Reasoning Capabilities of LLMs and LVLMs with Controllable Puzzle Generation
Jixuan Leng', Chengsong Huang', Langlin Huang,, Bill Yuchen Lin', William W. Cohen', Haohan Wang", Jiaxin Huang
CCMU, WUSTL, UW, +UIUC
jixuanlêcs. cmu.edu ji jiaxinh@wustl.ed .du
Code: httpp: thu.
Dataset: http::///hhgiiiiie...eaaaatttttr
Abstract
Existing reasoning evaluation frameworks for Large Language Models (LLMs) and Large Vision-Language Models (LVLMs) predominantly assess either text-based reasoning or vision-language understanding capabilities, with limited dynamic interplay between textual and visual constraints. To address this limitation, we introduce designed evaluate the reasoning capabilities of both LLMs and LVLMs through the medium of crossword puzzles task requiring multimodal adherence semantic constraints from text-based clues and intersectional constraints from visual grid structures. CrossWordBenn trollable puzzle generation framework that produces puzzles in two for- mats (text and image), supports adjustable difficulty through prefill ratio control, and offers different evaluation strategies, ranging from direct puz- zle solving to interactive modes suitable for agentic evaluation. Our exten- sive evaluation of over 20 models reveals that reasoning LLMs substantially outperform non-reasoning models by effectively leveraging crossing-letter constraints. We further demonstrate that LVLMs struggle with the task, showing strong correlation between their puzzle-solving performance and grid-parsing accuracy. Our findings highlight the limitations of the rea- soning capabilities of current LLMs and LVLMs, and provide an effective approach for creating multimodal constrained tasks for future evaluations.
benchmark
leverages con-
1
Introduction
Correctly Filled Words (%)
Consistency with Crossing Letters (%)
100
100
Relative 89%
Relative increase 34%
0
0
7x7
14x14
7x7
14x14
Best Reasoning LLM
Best Non-reasoming LLM
Best LVLM
Figure 1: Performance gap between top-performing reasoning LLMs and non-reasoning LLMs//LV LVLMs on CrossWordBencc. Reasoning LLMs achieve better overall performance and adhere more effectively to crossing-letter constraints than non-reasoning LLMs/LVV LLMMs..LM..
Large reasoning models (e.g., OpenAl-ol (Jaech et al., 2024) and DeepSeek-R1 (Guo et al., 2025)) have made exceptional progress on reasoning benchmarks including math problem solving (Veeraboina, 2023), coding (Austin et al., 2021) and commonsense reasoning (Clark et al., 2018). However, existing evaluation benchmarks for Large Language Models (LLMs) and Large Vision VinguaeL Models (LVLMs) (e.g., Visual Question Answering (Yue et al., 2023)) mostly focus on text-based reasoning or vision-language understanding, lacking dynamic interplay between textual and visual constraints that characterizes real-world

---

Published as a conference paper at COLM 2025
problem solving. Consequently, evaluating multimodal reasoning -ataliiiiesspatuuuulllllll Shanenniingg..y on reasoning tasks requiring both textual and visual constraints remains challenging.
Crossword puzzles, classic grid-based task in which horizontal (" ("Across") words and vertical ("Down") words must be filled in based on text-based clues, provide a unique testbed for such evaluations. They pose two distinct challenges: (1) question answering for each text-based clue, which may admit multiple correct solutions, and (2) visual constraint satisfaction, which requires precise letter alignment the intersections of Across and Down entries. Prior crossword datasets (Efrat et al., 2021; Rozner et al., 2021; Kulshreshtha et al., 2022; Chen et al., 2025) often suffer from their reliance on static and copyrighted online news sources and adopt a text-centric formulation, thereby neglecting the visual structure.
In this paper, we introduce Berd,nh,,,,,,, a controllable and scalable benchmark for evaluating the reasoning capabilities of both LLMs and LVLMs. WordeWWrd collects data and generates puzzles from three sources: (1) multilingual word-clue pairs from public repositories, (2) dictionary-based definitions, and (3) adapted question-answer pairs from existing benchmarks (e.g., CAmmonsenseQA (Talmor et al., 2018)) where the answers are open-ended or unconstrained. By representing grids and clues in different formats, as shown in Figure 2, WordseWo facilitates the evaluation of both model types. It supports two evaluation modes: direct puzzle-solving mode, which generates one-round responses using zero-shot Chain-of- of-houhh (CoT) prompts, and an interactive mode for step-by-step puzzle-solving, where grid update functions can provide intermediate visual outputs for follow-ups, thereby serving as a foundation for evaluating agents using function calling.
We evaluate over 20 state-of-the-art models, including both proprietary models (e.g., GPT- 4o (Hurst et al., 2024) and Claude 3.7 Sonnet (Anthropic, 2024)) and open-weight models (eg,, DeepSeek-R1 (Guo al., 2025) and Pixtral-Large-Instruct (Agrawal et al., 2024)). Our evaluation yields several notable findings: (1) LVLMs perform significantly worse than LLMs on BordBehc (as shown in Figure 1), and they struggle in OCR in vertical ("Down") word extraction. In fact, their puzzle-solving performance strongly correlates with their grid-parsing accuracy (r = 0.94). (2) Reasoning LLMs outperform non-reasoning constraints. (3) Even puzzles derived from saturated benchmarks (e.g., Commonsense(A) remain challenging, highlighting the significance of structural constraints in reasoning evaluation.
models, and benefit from both test-time scaling and increased
2 Related Work
LLMs and LVLMs Reasoning. Recent research on the reasoning capabilities of LLMs (Ahn et al., 2024; Huang & Chang, 2022; Kojima et al., 2022; Plaat et al., 2024; Jaech et al., 2024; Guo et al., 2025; OpenAl, 2025; Huang et al., 2022) has led to the development of various approaches, including prompting-bel 2024; Chen et al., 2022) that guide LLMs through intermediate reasoning steps for solving complex problems, fine-tuning methods that train LLMs on long reasoning chains (Ye et al., 2025; Muennighoff et al., 2025; Zhao et al., 2025), and test-time scaling via self-refinement or the use of a verifier (Setlur et al., 2024; Feng et al., 2023; Wang et al., 2023; Zhang et al., 2024a; Uesato al., 2022; Huang et al., 2025a). Moreover, a recent study from Deepseek-Ri (Guo et al., 2025) demonstrates that reinforcement learning (RL) with verifiable rewards facilitates the emergence of complex thinking processes in LLMs. Several studies have explored reasoning in the multimodal domain by constructing CoT data for fine-tuning (Xu et al., 2024) and developing verifiable problems for RL (Yang et al., 2025; Huang et al., 2025b). While these approaches have demonstrated success, existing evaluation datasets remain largely restricted to math problems (Wang et al., 2024; Lu et al., 2023; Zhang et al., 2024b).
methods (Wei et al., 2022; Yao et al., 2023; Besta et al.,
Crossword Puzzles in Language Model Evaluation. Crossword puzzles have long been focus of research in natural language processing (NLP), particularly before the advent of LLMs. Early approaches typically employed constraint satisfaction algorithms augmented by external knowledge bases. Notable examples include systems such as Proverb (Littman et al., 2002), Dr. Fill (Ginsberg, 2011), and specialized models such as the Berkeley Crossword Solver (Wallace et al., 2022), which incorporate a fine-tuned BERT (Devlin et al., 2019) and belief propagation. More recent studies have leveraged LLMs to add crossword puzzles
2

---

Published as a conference paper at COLM 2025
through techniques including fine-tuning (Efrat et al., 2021; Rozner et al., 2021; Sadallah et al., 2024), prompting strategies such as Tree-of-Thoughts (Yao et al., 2023), or integration with search algorithms (Saha et al., 2024), demonstrating the potential of LLMs for crosswords.
Several datasets for crossword puzzles have been proposed, covering both English (Efrat et al., 2021; Rozner et al., 2021; Kulshreshtha et al., 2022; Chen et al., 2025) and Ara- bic (Zeinalipour et al., 2025). However, one significant limitation of these datasets and approaches is that they rely on data from online news sources (Efrat et al., 2021; Rozner et al., 2021; Kulshreshtha et al., 2022; Chen et al., 2025) and often formulate crossword solving as question-answering (QA) task (Sadallah et al., 2024; Efrat et al., 2021; Rozner al., 2021; Yao et al., 2023), thereby overlooking the fundamental constraint-baset nature of the problem. Moreover, all of them treat crossword solving as a text-based task, despite the inherently visual nature of crossword grids, leaving gap in extending crossword puzzles for evaluat- ing LVLMs. In this work, we address these limitations by introducing CrossWordBench, framework that features controllable puzzle generation and extends evaluation to LVLMs.
3 Benchmark Curation
(a)
Word-Clue Pairs
Text Inputs to LLMS
Img Inputs to LVLMS
Public Repos
ABLITY prsaemiin
qualities jespos
empoy 10.0.0.10..0000000
Across:
Dictionaries
Q
Abbr on every
original Bartles song
CARN-AGE Bloodshod
comething done
notthing
Clese: Eachclue concains. Duue Direction Number s. Ac Dowen
0,0.0. D. 0.0.1.0
Whenn baffe
"The heaade"
AbBr.
mallooown
Ownes sllniigg whees
Detherr
Benchmarks
of ssuit?
Down:
eosts contery
UPl hows service inits.
Doon 1.00 de Not in cookery
SART Trad feeture
YANKEE New En Englander
C
VLMs LLMs
Response
clues for solving
Empty grid with
(b)
Response Model response in
OOR D
salve clue ata time.
VLMs Prompt VLMS
(c)
Extract the clue direction, number, answer
Zero-Shot CoT
Interactive Mode
CrossWrrd
Evaluation
Word Coverage Rate Coverage
Answer Parser
Filled Grid
Plot Generator piet to answer and wnteraateion imns
follow prompts with ings of filled grids
Interaction Consistency
direction, number, and answer
Rate
Figure 2: Framework of CrossWordBench. (a) Dataset curation process and input templates for LLMs and LVLMs; (b) Zero-shot CoT evaluation; (c) Interactive Mode Evaluation.
Online sources, including The New York Times and The Los Angeles Times, provide complete crossword puzzles. Nevertheless, directly utilizing these puzzles presents several disadvan- tages. (1) Copyright restrictions may limit their usage, and their online availability increases the probability that they have been incorporated into the pretraining datasets of contem- porary models.
(2) Online puzzles are typically static-with predefined words, themes, not only limits their
grid sizes, and strict formatting (e.g., black square " adaptability for diverse benchmark generation, but also imposes arbitrary formatting con- straints that do not yield meaningful benefits for evaluation. Prior research (Mirzadeh et al., 2024) has shown that even minor modifications to questions can significantly affect model performance, particularly when performance on the original version is near saturation. (3) The static characteristics these puzzles restrict the range of potential evaluation strategies. To overcome these limitations, we propose two-stage benchmark construction strategy:
The New York Times Crossword Requirements.
A Dataset of Cryptic Crossword Clues
3Chinese Crosswords
3

---

Published as a conference paper at COLM 2025
Word-Clue Pairs Curation. We compile word-clue pairs from three source categories: (1) Public repositories: We selectively extract individual, often cryptic, word-clue pairs from public online repositories with samples in both English and Chinese (2) Dictionary-
Pairs: We collect standard English words and use their dictionary definitions from NLTK WordNet clues, referring to this category English Simple. (3) Adapted Benchmark Data: We demonstrate that existing LLM benchmarks, which are typically designed for open-ended or multiple-choice questions without strict formatting constraints, can be transformed into word-clue pairs. In our study, we filter the CommonsenseQA (Talmor et al., 2018) training set for single-word answers and use the associated questions as clues.
While the possibility of data contamination may still exist for these word-clue pairs, the variations in grid design can still yield distinct crossword puzzles, and as demonstrated in Section 4.5, even when incorporating extensively tuned data such as CommonsenseQA training set, the resulting puzzles remain remarkably challenging for both LLMs and LVLMs.
Table 1: Crossword puzzle statistics for different subjects and grid sizes. Statistics are presented separately for each category, as distinct word and clue pairs are used for their construction. Additionally, the aggregated statistics for all English puzzles are included.
English
Chinese
English Simple
CommonsenseeAA
Stats.
7x7
14x14
7x7
7x7
7x7
Total of puzzles
100
100
100
100
Total # of words
1,193
3,472
1,327
1,139
543
Unique words (% total)
83.82%
80.76%
92.92%
36.70%
59.85%
Unique clues (% total)
100%
100%
100%
100%
100%
Aggregated (English 7x7 & 14x14): 200 puzzles, 74.02% unique words, 100% unique clues
Words per Puzzle
Minimum
11
22
11
11
Maximum
16
44
18
13
13
Mean
11.93
34.72
13.27
11.39
10.86
Word Length (Letters)
Minimum
3
Maximum
5
12
5
5
Mean
3.59
4.31
3,02
3,63
3.77
Avg blocked cells (%)
39.51 %
45.22%
43.37%
39.12%
38.37%
Puzzle Generation. After collecting word-clue pairs, we implement an automatic pipeline to generate puzzles in both text and image formats, as illustrated in Figure 2. The pipeline enables (1) puzzle difficulty control by adjusting grid sizes, and (2) incremental word place- ment functions, which facilitate the interactive evaluation mode described in Section 4.4.
The generation framework leverages a heuristic scoring function that encourages desirable properties such as high number of horizontal and vertical word intersections. Post- processing filters are applied to discard puzzles with low clue counts or high blocked cell ratios, thereby enhancing overall quality and structural coherence. These heuristics and filters, in combination with grid size adjustments, help control puzzle complexity. To preserve the category integrity, we ensure that clues do not overlap within each category
Puzzle Quality. While CrossWord does not include human-created puzzles, its design statistiss-inchuding
inspired by them. Table reports overall dataset statistics iincuuding clue count, average word length, and blocked cell which serve as proxies for puzzle quality and difficulty,
Compared human-created puzzles- for example, those used LR²Bh (Chen et al., 2025), which were sourced from outlets such as The Los Angeles Times and Vulture- our generated puzzles exhibit comparable difficulty when evaluated using similar word-level metrics. Our focus on automatically generated puzzles is motivated by the need for scalable, controllable, and diverse evaluation across subjects and formats. This enables systematic variation in
4NLIK Wordnet
Splease check out our code for more implementation details.
4

---

Published as a conference paper at COLM 2025
difficulty through grid size, prefill ratio, and clue selection; supports step-by-step reasoning via an interaction mode; and facilitates fine-grained metric analysis beyond overall accuracy.
4
Experiment
In the following sections, we present an extensive empirical evaluation of across multiple different model architectures and analyze their performance characteristics.
4.1
Experimental Setup
Evaluation Data and Metrics. For the main experiments, we evaluate models on the English set with three metrics to assess answer accuracy and adherence to crossword constraints,
Word Coverage Rate (WCR): word-level accuracy, percentage of correctly solved words.
Letter Coverage Rate (LCR): letter-level accuracy, percentage of correct letter placements.
Intersection Consistency Rate (ICR): the internal consistency of the model's answers at intersections where across and down words overlap, defined as:
1
ICR
Σ (aA.)A)E
1[alf = =d]]]
(1)
where I denotes the set of all intersections, where each tuple (a,d, indicates the jth letter of the across word overlaps with the kth letter of the down word d. This metric reflects whether models correctly adhere to the grid structural constraints of puzzle.
Evaluated Models. We evaluate collection of proprietary and open-weight models, including both LVLMs and LLMs. For proprietary models, we consider state-of-the-art models such as GPT-4o (Hurst et al., 2024) and Claude 3.7 Sonnet (Anthropic, 2024) (with and without thinking mode). For open-weight models, our selections range from 3B to 124B parameters, such as the Qwen Series (Team, 2024; Bai et al., 2025) and Fixtral-Large-Instruee 2411 (Agrawal et al., 2024) for LVLMs. For LLMs, we include both reasoning models such as the Deepseek Series (Liu et al., 2024; Guo et al., 2025) and non-reasoning models such as Llama series (Dubey et al., 2024). The full list of models is shown in Table 2. We set decoding temperature to 0 for non-reasoning models for consistency, 0.6 for reasoning models based on commonly recommended settings, and 1.0 for certain proprietary models (e.g., used by Claude 3.7-Sonnet with Thinking). Further generation details are provided in Appendix D.3.
Input Prompt Templates and Output Response Parsing. For the main evaluation, we adopt the zero-shot Chain-of-Thought (CoT) (Wei et al., 2022) prompting strategy. In LVLM evaluation, the clues and the grid are both embedded within an image. In LLM evaluation, the grid is represented as 2D binary array, with 1 indicating blocked cell and representing an unfilled cell, and prepended to text clues in the prompt. To extract answers from responses, we leverage the structured output capabilities of a3-mini to convert raw model responses into JSON format by generating dynamic Pydantic models. Detailed prompt templates and implementation details are listed in Appendix D.4 and D.2.
4.2 Main Results
Reasoning LLMs substantially outperform conventional ones across metrics, with notable improvements in ICR, as shown in Table 2. In particular, among reasoning LLMs, o3-mini achieves an ICR of 0.891 on 7x7 grids, demonstrating a strong ability to interpret and enforce grid constraints. Although DeepSeek-R1 attains the highest WCR and LCR on 7x7 grids, its ICR is slightly lower than that of 03-mini. Other reasoning models, such show moderate performance, with WCRs of
as QwQ-32B and RI-Distilled-Llama-l approximately 0.347 and 0.387, respectively. One notable outlier is Reaeen-Rasoner-Zeee which performs poorly across all metrics, indicating that does not effectively leverage grid constraints for reasoning. This suggests that its thining-Priiarily, focused on mathematical reasoning-dee does not generalize well to tasks requiring spatial and linguistic integration, thereby highlighting key limitation in the training strategies of these reasoning models.
6https://opena wwwpdiiopeeaiio---noe
5

---

Published as a conference paper at COLM 2025
Table 2: Comparison of various LLMs and LVLMs on CrossWordBench English set across two difficulty levels using zero-shot COT. We report the mean and standard error over 100 samples for both 7x7 and 14x14 grids. ? indicates that the model is reasoning model. We use the Fireworks API for DeepSeek V3 and Llama-3.1-405B, while offical API for R1.
7x7
14x14
Models
WCR
LCR
ICR
WCR
LCR
ICR
Proprictary LVLMs
Claude-3-7-Sonnet
0.479+0.014
0.52840.013
0.366+0.016
0.416+0.009
0.44940.009
0.272+0.010
Claude-3-7-Sonnet ?
0.365 0.011
0.448
0.448
0.0014
0.330 00.015
0..02
0.428 0.0007
0.228
GPT-44-2022-111-2
0.348 1.001
0.403
0.234
0.350
0..93
0.190.10.008
Gemini 2.0 Pro Exp
0.351 =00014
0.368
0.339 20.015
0.303 0.273
o.007
0.221 +0.007
Gemini 2.0 Flash
0.015
0.300
0.225+0.013
0.260
0.204 0.190.10.007
Open-Weight LVLMs
Pixtal-Large-Imstrrrt211111
0.297
0.338
0.19820.014
0.200 0.284
0.13440:007
Intern WL.
0..01
0.164400.09
0.099=0.011
0.119+0.007
0.159=0.006
0.073++0.00
NVLM-D-72B
0.134
0.179
0.076
0.085
0.120
0.053
Owem2.5-VL-72B-Insttue
0.207 10.013
0.245
0.133
0.19440.007
0.227 #0.006
0.110440006
QVQ-72B-Preview
0.197+a.012
0.218+0.010
0.091
0.195+0.007
0.215
0..08
llaa-onneviiio-7bbbovccaatt
0.141 10.012
0.165
0.097 10.009
0.102
0.141 0.000
0.005
gemma-3-27b-it
10.01
0.21840.011
0.124=000100
0.106g0.000
0.160=0.009
0.075
Aria
0.061:1.000
0.101
0.051 10.006
0.035
0.070...006
0.04640.004
MiniCPM-V-2.6
0.043
0.085
0.004
0.0 0.023
0.057
0.04010.005
Brenn..5-VL-3B-nnstrue
0.013
0.040
0.038_0.006
0.014
0.034
0.023+4.008
Proprietary LLMs
03-mini-high ?
0.587+0.023
0.684
0..01
0.445+0.011
0.520=0.011
0.512+0.007
Claude-3-7-Sonnet
0.712 0.617
0.754 00021
0.542 0.492
0.401
Claude-3-7-Sonnet
0.482
0.574+0.014
0.472=0.019
0.44640.011
0.485+0.011
0.321+00011
GPT-44-2024-111
0.410
0.472
0.288 0.001
0.338
0.369
0.196+0.010
Gemini 2.0 Pro Exp
0.460
0.525 0.388
0.425
0.457
10.00
0.28910.010
Gemini 2.0 Flash
0.301
0.31840.012 0.255
0.280 0.298
0.198
Opin-Weigh! LLMs
laama-31-4055D-Inttruct
0.161 10.13
0.35910.m1
0.243
0.355
0.390.1.009
0.202.10.008
DeepSeek-R1 ?
0.646
0.707++0.17
0.678=0.023
0.472+0.011
0.507 20.011
0.35640.001
DeepSeek-V3t
0.303
0.369400014
0.186=00013
0.290+00009
0.335
0.145
RI-Distill-Llam--08 ?
0.387
0.448
0.347 +0.017
0.285
0.319
0.161
Llama-3.3-70B-Instnu
0.303
0.371
0.206
0.280
0,340.20.009
0.173++0008
QwQ-32B
0.347
0.44540.018
0.518
0.25440.009
0.307 +0.009
0.189
Open-RRessoner-Zero:288
?
0.139 10,010
0.204
0.184
10.012
0.146
0.199 10.007
0.095
Phi-4
0..12 0.19440.010
0.113
0.140
0.200
0.085

---

Published as a conference paper at COLM 2025
Non-reasoning LLMs show limitations in ICR, and performance declines further with increasing grid size. Among non-reasoning LLMs, Claude 3.7 Sonnet and Gemini 2.0 Pro Exp yield the best results, with WCRs of 0.482 and 0.460 on the 7x7 grid, respectively; how- ever, their relatively lower ICR indicates model limitations on explicit reasoning constraints for crossword puzzles. Notably, thinking mode improves Claude 3.7 Sonnet on all three metrics, highlighting the importance of reasoning and reflection in solving constraints-based crossword tasks. Additionally, larger grids lead to decreasing performance, demonstrating the increased complexity of maintaining constraint adherence over a larger search space.
LVLMs currently lag behind LLMs in performance, with minimal adherence grid constraints. With image inputs, Claude 3.7 Sonnet achieves the highest performance among L.V.Ms, but underperforms its own text inputs version. Models such as GPT-4o and Gemini 2.0 Pro Exp exhibit similar trends, with WCRs below 0.35 on larger grids. All LVLMs demonstrate low ICRs, suggesting that they struggle to maintain reasoning consistency. Notably, performance declines when thinking mode is enabled on Claude 3.7 Sonnet with image input, which contradicts the improvements observed with text inputs; we explore this phenomenon in the next section. Among open-weight LVLMs, Pixtral-Large- Instruct achieves the best WCR 0.297 on 7x7 grid, while still lags behind most proprietary LVLMs.
4.3 Positive Correlation: LVLMs' Grid Parsing & Puzzle-Solving Performance
Model performance on crossword puzzles exhibits strong dependence on grid pars- ing capabilities, with systematic biases in word orientations. We evaluate grid pars- ing ability by prompting models to parse completed puzzle grids and associated clues (prompt details are in Appendix D.4). Successful grid parsing requires: (1) iden- tifying grid indexing numbers, (2) map- ping numbers to clues to determine word orientation, and (3) extracting words with boundary recognition. The grid parsing ability reflects an LVLM's ability to inter- pret both spatial and textual information.
Grid Parsing and Performance Correlation
(Metrics Word Coverage Rate)
Direct Corr (e) 0,9558
clande----sonnett
p-value 0.0000003
144
Boutstmp 95% (0.9158, 0.9478)
da-de-----oonnet q
Angeminge
WCR
0.3
Fistral-Largo-nssrrue
gemini--20-Blahh
QVQ-72B-Ptvien Dwen255VVL772--lsttee
Tlava-onevisignngveett qreeniz-- 3-emma---27b-it
a
1atemmL22--75BBMMP
MiniCPMM-Vv2. S-renn25-VVL-
Linear Fit ty 0.4672x 0.0091)
Geid
Figure 3: Grid Parsing vs. Puzzle-Solving 7x7 English puzzles, measured with WCR.
We reveal two significant patterns: (1) As shown in Table 3, LVLMs extract Across words more accurately than Down words, highlighting OCR limitations. (2) Figure 3
Table 3: WCR on Grid Parsing.
Models
Across
Down
demonstrates strong positive correlation between grid parsing and overall puzzle- solving performance (both measured by WCR). Notably, when enabling thinking mode, Claude 3.7 Sonnet exhibits lower grid parsing performance, which aligns with its lower puzzle-solving performance. We hypothesize that the additional token generation in thinking mode diverts atten- tion from image processing. These findings highlight the critical role of precise spatial- textual interprelation for future LVL.M design. For full results, please refer Appendix C.2.
Proprietary VLMs
Claude-3-7-Sonnet
0.760 0..04.0.009
Claude-3-7-Sonnet ?
00.010
0.654 0.654 10002
GPT-40-2024-11-20
0.88610.014
0.448.0.0
Weight VLMs
Pixtral-Large-Instrue 0.703
0.361 20.022
QVQ-72B-Preview 0.717 10.022
0.139
4.4 Agentic Evaluation Setting: Interactive Mode for LVLMs
Motivated by recent research on visual-of-thoughts (Wu et al., 2024; Li et al., 2025), introduce a new evaluation setting for LVLMs, referred to as Interactive Mode. This setting leverages the ability of Bench to automatically generate updated grid images.
7

---

Published as a conference paper at COLM 2025
Interactive Mode requires step-by-step puzzle solving rather than completing the tire puzzle a single pass. Specifically, the implementation of the controllable genera-
tion framework allows for updating grid images each time model provides an answer. We maintain a four-round conversation his-
of Interactive Success Slep Model
tory due to context window limitations. We introduce Interactive Success Step (ISS)
to quantify how many words model cor-
Model
rectly solves before making its first error. Figure 4 shows the cumulative distribution of ISS for each moodel-mov models fail at the initial solution step on most puzzles.
"gemini--2.-ppr
-GPT-lo
0.7
-gemiini-..--leeh
claude-3-7-senee 3-seemee--
0.6
Qwen2.55VL 5-V-5-7----7--sstttuu
InternVL2 5-788-PPO
Unlike the visual-of-thoughts approach, which focuses on generating intermediate reasoning plots, we employ external func- tions to update the grid. This establishes foundation for evaluating LVLMs in agentic settings, where the word placement functions can serve as callable tools for the model and provide error feedback to guide the model in refining its responses for example, by indicating whether proposed word matches the expected length and whether its intersecting characters align with previously filled cells.
Step
Figure 4: CDF of ISS on 7x7 English Puzzles.
4.5
Beyond English: Evaluations on Multilingual, Dictionary-based, and Adapted Data
Table 4: WCR and ICR for Chinese, English Simple, and CommonSenseQA puzzle sets.
Chinese
English Simple
CommonSenseQA
Models
WCR
ICR
WCR
ICR
WCR
ICR
Proprietary LVLMs
GPT-44-2024-11-20
0.366 0.1
0.335 0.227 +0.01
0.392+0.026 0.246
Gemini 2.0 Flash
0.208 10.031 0.233 1.001
0.229 10.014 0.216...011
0.327 10.021
0.206
Weight LVLMs
Fixtnal-aarge-Isstuut221111
0.252 2.00
0.21620.116 0.187
0.439
0.200
Oween25-VL-72B-nnstrrt
0.391 +0.25 0.282
0.239
0.183
0.418 +0.022
0.252
Proprietary LLMs
GPT-44-2022-111-0 03-mini-high ?
0.543 10,019
0.448 1.021
0.438 10.20 0.306_1..22
0.524 10.22
0.326 10.029
0.774 0.953
0.782 0.94610012
0.812 0.001
Opin-Weight LLMs
DeepSeek-R1 ?
0.907 0.898
0.759 0.787+a.m2
0.752 0.8294
QwQ-32B ?
0.701 0.654_0.022
10.02 0.734 0.6991a.226 0.766.1.027
We extend zero-shot CoT prompting evaluation to Chinese, dictionary-basee and benchnark-adapted word-clue pairs in CrossWordBench,, with results shown in Table 4.
Reasoning models outperform conventional ones across metrics, and the performance gap between open-weight and proprietary models is reduced. 03-mini maintains high ICR across all three datasets, consistent with the main evaluation results. Furthermore, the performance gap between open-weight and proprietary reasoning models is reduced, possibly due to the simplicity of these tasks and differences training data. Interestingly, when solving Chinese puzzles, observe that QwQ-32B consistently reasons in Chinese.
Effectiveness of grid construction: puzzles constructed with CommonsenseQA training data remain challenging. Despite CommonsenseQA having saturated performance most
8

---

Published as a conference paper at COLM 2025
LLMs, its training set continues to pose significant challenges for reasoning models. For example, o3-mini achieves the highest WCR 0.812, yet it still falls short of perfection. In contrast, the best-performing non-reasoning LLMs such as GPT-4o and open-weight LVLMs Fixtral-Large-Instrue obtain WCR of 0.524 and 0.439, respectively. These results highlight the effectiveness of our grid construction strategy in repurposing existing benchmark data.
5 Analysis
In this section, we provide an analysis of model behavior on Cross through the impact of structural constraints and reasoning mechanisms, and discuss future applications.
5.1 Grid Format Ablations: LLMs Robustness Textual Grid Representations
LLMs exhibit robustness to variations in grid text representation. In the text-only evaluation setting, the empty crossword puzzle grid is represented as a 2D binary
Table 5: Grid Format Ablations.
Models
Array
Markdown
array, where denotes blocked cell and o denotes an unfilled cell. This array is then prepended to the prompt containing clues. To evaluate the impact of different formatting choices, we test an alternative markdown-style representation in this section, with indicates unfilled cells and indicates blocked cells. An example of the two formats is provided in Appendix D.5 and Figure 19.
Claude-3-7-Sonnet
0.482
0..60
GPT-40-2024-11-20
0.410
0.398
Gemini 2.0 Flash
0.301 0.301 =0.014
0.309
DeepSeek-VS.
0.303
0.006
03-mini
0,587 0.0023
1..02
Due to space constraints in the main text, defer additional ablation studies LVLM input formats and few-shot prompting to Appendix C.6 and C.8, respectively. We also demonstrate how controlling prefill ratio be used to adjust task difficulty. We encourage readers to refer to these appendices for a comprehensive analysis and further discussion.
5.2 Crossing Letters: Reasoning LLMs Improve with More Intersections
Average word accuracy on reasoning mod- els increases with crossing letter count.
Average WCR vs. Crossing Letter Count
09
Five Rezening 1LMs
(esclade Qpe--traauran-ZZmm-ttt
Figure 5 presents the average WCR for each range of crossing letter counts, divided into three groups, across five reasoning and nine non-reasoning LLMs. We observe that rea- soning models exhibit increasing accuracy with a greater number of letter intersections, whereas this trend is not observed for non- reasoning LLMs. This observation aligns with main experiment results, in which reasoning models tend to exhibit higher ICRs, suggesting that they benefit from effectively utilizing grid constraints for solution space reduction. However, do not observe the same pattern on 14x14 puzzles-likely because puzzles with larger grids are substantially more difficult even for reasoning models. We provide detailed results in Appendix C.7.
Nine Non resoring LLM
Avergg
29- 05
low (1)
medium (2)
high (3+)
Crossing: Letter Count Range
Figure 5: Crossing letter counts and average WCR 7x7 English puzzles across L.Mss.
Table 6: Average WCR results by crossing let- ter count and density on 7x7 English puzzles.
To further isolate the effect of crossing-letter density within a constrained grid, we per- form a quasi-ablation study on 7x7 puzzles by stratifying puzzles into low, medium, and high-density groups based on the av- erage number of crossing letters per word, as shown in Table 6. Overall, within differ- ent densities, reasoning LLMs demonstrate improved WCR with greater number of crossing letters, whereas non-reasoning L.LMs show relatively stable performance.
Density Crossing Letter Reason. Non-Reason.
Low(1)
0.4341
0.2960
Low
Med(2)
0.5275
0.3355
High(3+)
0.6165
0,2794
Low(1)
0.4271
0.2754
Medium
Med(2)
0.5477
0.3089
High(3+)
0.6029
0,4193
Low(1)
0.5288
0.3097
High
Med(2)
0.5151
0.3217
High(3+)
0.5928
0.3030

---

Published as a conference paper at COLM 2025
5.3 Self-Reflection: Limited Impact Crossword Puzzle Solving
Backtracking and verifying previously filled answers essential components of effective crossword puzzle solving. To eval- uate the effect of self-reflection, we include manually crafted follow-up query that prompts models revisit their previous zero-shot CoT responses (please see Ap- pendix D.4 for details). As shown Fig- ure 6, observe measurable perfor- mance improvement for reasoning LLMs or non-reasoning LVLMs. This find suggests that additional interactions with manual prompting alone are insufficient enhance reasoning capabilities for puzzle solving.
Deepseck-RI (LLM)
QwQ-32B (LLM)
WCR
claude-3-7ssonnet (LVLM)
Virtral-arge- (LVLM)
LCR
WCR
WCR With Self-Reflection
Without Self-Refiection
Figure Self-reflection improvements on 7x7 English puzzles across metrics. Top: reason- ing LLMs. Bottom:
non-reasoning LVLMs.
5.4 Test-Time Scaling: Diminishing Returns on Puzzle Performance
To examine the effect of test-time scaling on CrossWordBench, evaluate 03-mini 7x7 English puzzles by varying its reason- ing effort, which controls the number of rea- soning tokens generated during inference.
03-mini Reasoning Effort vs. Puzzle-Solving
High Effort (49,865 tokens)
Medium Eifort (23,723 takens)
Affort (4,770 tokens)
0.6
As shown in Figure 7, increasing the effort from low to medium yields substantial performance improvement across all three metrics. However, further doubling the rea- soning tokens provides no significant addi- tional gains, indicating diminishing returns.
0.4
0.2
WCR
LCR
ICR
Figure 7: o3-mini performance on 7x7 English puzzles under three levels of reasoning efforts.
5.5
Discussion: Verifiable Crossword for Future Multimodal RL Training
Table 2 shows that reasoning models such as Open-Reassone--eeno3333 which are primarily trained on mathematical problems, exhibit limited generalization to CrossWordBennh. This highlights the limitations of relying predominantly on math-based tasks with verifiable answers for reinforcement learning in more complex reasoning settings. As multimodal rea- soning advances, significant challenge persists: the lack of multimodal environments with verifiable answers suitable for rule-based reinforcement learning. We propose crossword puzzles as compelling alternative, owing to their unique combinations of verifiability, multimodal structure, and goal-eriented task design. Crossword puzzles are also suitable for multi-turn training, where each word fill represent discrete, observable decision.
Our framework supports this setting by providing word placement functions that dynami- cally update the puzzle grid in both text and image formats, allowing seamless integration with multimodal agents and enabling tool-use training. We leave the exploration of training multimodal agents on crossword puzzles with real-time function feedback for future work.
6 Conclusion
This paper introduces BrossWord benchmark designed to evaluate the multimodal reasoning capabilities of both LLMs and LVLMs using crossword puzzles, which uniquely integrate text-based clues and visual constraints. Our extensive evaluation of over 20 models shows that reasoning models substantially outperform non-reasoning counterparts and can benefit from increased crossing-letter constraints. Additionally, we find strong correlation between puzzle-solving performance and grid-parsing accuracy in LVLMs. Even puzzles derived from saturated benchmarks remain challenging, emphasizing the necessity of structural complexity in rigorous reasoning evaluation. This work paves the way for improving future multimodal RL training where interplay between modalities essential.
10

---

Published as a conference paper at COLM 2025
References
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al. Phi-4 technical report. Xro preprint ar axXt::2412.08905, 2024.
Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. Pixtral 12b. arviv preprint ar arXio:2410.07073 2024.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large lan- guage models for mathematical reasoning: Progresses and challenges. arxiv preprint arXiv:2402.00157, 2024.
Anthoopcc. Claude 3.5 sonnet. httpss//ww..aattrrrrccccc//////s 2024.
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang. Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program synthesis with large language models. ArXiv, Abs/2218.07732, 2021. URL https://api. semanticscholss org :Ctp///oppp::::: 2371 42385.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. ar preprint arXio:2502.13923, 2025.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings of the AAAI Conference on Artificial Infelligence, volume 38, pp. 17682-17690, 2024.
Jianghao Chen, Zhenlin Wei, Zhenjiang Ren, Ziyong Li, and Jiajun Zhang. Lr²bench: Evaluating long-chain reflective reasoning capabilities of large language models via constraint satisfaction problems, 2025. URL https: //tp::/.oxvvvooo htt.://org//b/////// /
Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. Xiv preprint ar Xiv:22211. 1.18888, 2022.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. ArXiv preprint, abs/1880300553 2018
Wenliang Dai, Nayeon Lee, Boxin Wang, Zhuolin Yang, Zihan Liu, Jon Barker, Tuomas Rintamaki, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. Nvlm: Open frontier- class multimodal 1lms. ar Xiv preprini, 2024.
Google DeepMind. Gemini 2.0. http::///eeemnnn.ggggo////coooo
2024.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171-4186, 2019.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama herd of models. Xiv preprint ar arxi:22077.21788 21783, 2024.
11

---

Published as a conference paper at COLM 2025
Avia Efrat, Uri Shaham, Dan Kilman, and Omer Levy. Cryptonite: A cryptic crossword benchmark for extreme ambiguity in language. arviv preprint arXiv:2103.01242, 2021.
Xidong Feng, Ziyu Wan, Muning Wen, Stephen Marcus McAleer, Ying Wen, Weinan Zhang, and Jun Wang. Alphazero- tree-search can guide large language model decoding and training. arXiv preprint ar arXiv:2309.17179 2023.
Matthew L Ginsberg. Dr. fill: Crosswords and an implemented solver for singly weighted csps. Journal of Arlificial Intelligence Research, 42:851-884 2011.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint (ix::::5011124444 2025.
Chengsong Huang, Langlin Huang, Jixuan Leng, Jiacheng Liu, and Jiaxin Huang. Efficient test-time scaling via self-calibration. ar arxiv preprint ar arXio:2503.0003 2025a.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. ar ar preprint ar arXi::2210.11610, 2022.
Jie Huang and Kevin Chen-Chuan Chang- Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212.10403, 2022.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. ar Xrviv preprint ar arXi::2503.06749, 2025b.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. Xiv preprint arXiv:2410.21277 2024.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. aiv preprint ar arXim:2412.16721 2024.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systemss, 11-22199-22213 2022.
Saurabh Kulshreshtha, Olga Kovaleva, Namrata Shivagunde, and Anna Rumshisky. Down and across: Introducing crossword-solving as new nlp benchmark. arXiv preprint ar arXie:2205.00442 2022.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. ar Xiv preprint Xrx:::2408.055: 033326, 2024a.
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulić, and Furu Wei. Imagine while reasoning in space: Multimodal cssullzation--f-thhughh arXiv preprint ar XrXxv22501.0552. 2025.
Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen, Bowen Qu, Xinyao Niu, Guoyin Wang, Bei Chen, and Junnan Li. Aria: An open multimodal native mixture-of- experts model. Xiv preprint arXio:2410.05993, 2024b.
Michael L Littman, Greg A Keim, and Noam Shazeer. A probabilistic approach to solving crossword puzzles. Artificial Intelligence, 134((-2):23333 2002.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXio preprint ar Xiv:2444 19437, 2024.
12

---

Published as a conference paper at COLM 2025
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and
Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models. arXiv preprint ar XrXi::2410.05229, 2024.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. Xiv preprint ar axXiv:2501.19399 2025.
OpenAL. Openai 03-mini. httpss///peeaal.cm//ieneee//////
2025.
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Back. Reasoning with large language models, a survey. arXiv preprint arXit24007.11511 2024.
Josh Rozner, Christopher Potts, and Kyle Mahowald. Decrypting cryptic crosswords: Semantically complex wordplay puzzles as a target for nlp. Advances in Neural Information Processing Systems, 34:11409-11421 2021.
Abdelrahman Sadallah, Daria Kotova, and Ekaterina Kochmar. Are llms good cryptic crossword solvers? arviv preprint ar arXiv:2403.12093 2024.
Soumadeep Saha, Sutanoya Chakraborty, Saptarshi Saha, and Utpal Garain. Language models are crossword solvers. arXiv preprint xrXiv:2406.09043, 2024.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for Ilm reasoning. arXiv preprint XfXo:2440008816 2024.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsensega:: A question answering challenge targeting commonsense knowledge. arXiv preprint ar arXie:1811.00937 2018.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma technical report. arXiv preprint arXit:2503.19786, 2025.
Qwen Team. Qvq: To see the world with wisdom, December 2024. URL https:s/////:nnnm github. io/blog////ooooo qvq- 7ei--2bb-ppa paabbbpreeeww....
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https://qwenIn. gi thub.
Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXio preprint ar arXiv:2211.14275 2022.
Hemish Veeraboina. Aime problem set 1983-2024, 2023. URL hittps://ww/ httt:://www..kaggee boggle.com com/
datasets/hemi shveer aiwa-imee-potro wwwpppe--et--98222022
Eric Wallace, Nicholas Tomlin, Albert Xu, Kevin Yang, Eshaan Pathak, Matthew Ginsberg, and Dan Klein. Automated crossword solving. arXiv preprint arXic:2205.09665, 2022.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95169, 2024.
Peiyi Wang, Lei Li, Zhihong Shao, RX Xu, Damai Dai, Yifei Li, Deli Chen, Y Wu, and Zhifang Sui. Math-shepherd: A label-fres free step-by-step verifier for llms in mathematical reasoning. ar Xiv preprint ar Xix::2312.00 089355, 2023.
13

---

Published as a conference paper at COLM 2025
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Adoances in neural information processing systems,, 35:24824-24837, 2022.
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, and Furu Wei. Mind's eye of Ilms: Visualization-o-tthouuhh elicits spatial reasoning in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. arv preprint ar Xio:2411.10040, 2024.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. Rl-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint ar Xiv:25008.10615, 2025.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36:11809-11822, 2023.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint ar Xiv::2408.000 0180000000000,.... 2024.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. Xiv preprint ar arXiv:2502.03387 2025.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. org/ org/CorpusID: 265466525.
2023. URL
semanticscholar.
Kamyar Zeinalipour, Mohamed Zaky Saad, Marco Maggini, and Marco Gori. From arabic text to puzzles: Llm-driven development of arabic educational crosswords. arv preprint ar axXiv:2501.11035, 2025.
Dan Zhang, Sining Zhoubian, Ziniu Hu, Yisong Yue, Yuxiao Dong, and Jie Tang. Rest-mets": Lim self-training via process reward guided tree search. Advances in Neural Information Processing Systems, 37:64735-64772, 2024a.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal 11m truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169-186. Springer, 2024b.
Han Zhao, Haotian Wang, Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li. 1.4 million open-source distilled reasoning dataset empower large language model training, 2025. URL httpp:://arxt wwtpv//..rg//bb////eee/ 19633.
14

---

Published as a conference paper at COLM 2025
A Acknowledgment
We thank all reviewers for their feedback, which contributed to improving this work.
B
Limitations
We propose Brossh to evaluate the reasoning capabilities of both LLMs and LVLMs. Unlike prior work, CrossWordBend does not incorporate any human-authored puzzles from online sources. Although our generation process is carefully designed to preserve key structural properties-suee such as clue distribution, word length, and blocked cell ratios-as sunmarized in Table 1, and the generated puzzles exhibit difficulty levels comparable to those of human-created puzzles when evaluated using standard word-level metrics, manual verification of puzzle quality could be helpful. Although our framework is also capable generating crossword puzzles for potential use in training, this paper focuses solely on evaluation and leaves the exploration of reinforcement learning for future work.
C More Results and Analysis
C.1 Behavior Analysis
To better analyze the errors made by models in puzzle-solving, we define two metrics: global length error and local length error. The global length error metric compares the number of words produced by the model with those in the reference answer list, assessing whether the model supplies an answer for every clue in the puzzle. In contrast, the local length error metric compares the length of each individual word to its corresponding reference, thereby quantifying the model's 's adherence to the grid constraints. Table 7 shows that even the pest-performing reasoning models, such as Claude 3.7 Sonnet with thinking mode and DeepSeek R1, exhibit global length errors two puzzles. Almost all models-with the exception of o3-mini, which demonstrates the highest ICR-commit a significant number of local errors. Moreover, we observe that all global length errors arise from models providing fewer answers than required, i.e. failing to address some clues. In contrast, most local length errors are due to models generating answers that exceed the expected word length.
C.2 Complete Results on Grid Parsing Performance
In Section 4.3, we demonstrate positive correlation between LVLM's grid-parsing accuracy and their puzzle-solving performance. We also provide several examples illustrating the limitations of LVLMs in extracting horizontal words. Here, we present a complete table of all models shown in Table 3, as summarized in Table 8, where similar trend is observed.
C.3 Complete Results on Extended Data
In this section, we present additional results for evaluations on the extended data. As shown in Table 9, LLMs generally perform better across these three categories than on the English puzzles; however, even for puzzles from CommonSenseQA, performance remains far from perfect. In contrast, LVLMs do not exhibit a significant performance improvement over English puzzles. Based on our observation of a strong positive correlation between grid- parsing and puzzle-solving performance (See Figure 3 for more details), we hypothesize that the reasoning capabilities of LVLMs are constrained by their visual processing abilities.
C.4 More on Test-Time Scaling
In Section 5.4, with three distinct reasoning efforts for o3-mini on 7x7 English puzzles. Here, we provide a table detailing the specific numerical values for each metric as supplement to Figure
bar chart to demonstrate the performance differences associated
15

---

Published as a conference paper at COLM 2025
Table 7: Global and Local Length Errors across models 7x7 English puzzles. "Long" and "Short" indicate words that are longer or shorter than the corresponding reference answer.
Global
Length Error
Local Length Error
Models
Tot. Long
Short
Tot. Long
Short
Proprietary LVLMs
Claude-3-7-Sonnet
0
0
0
363
205
158
Claude-3-7-Sonnet ?
0
0
454
237
217
GPT-44-2024-11-20
10
581
255
Gemini 2.0 Pro Exp
565
98
Gemini 2.0 Flash
0
0
665
568
97
Open-Weight LVLMs
Pittal-Large-Instrut--1111
3
0
3
623
481
142
Interm
0
0
0
834
682
152
NVLM-D-72B
8
0
8
791
620
171
Qwen225-VL-77B-nnsttud
0
744
600
144
QVQ-72B-Preview
20
0
765
525
240
llave-onevision//2bbo-cchht
3
0
829
715
114
gemma-3-27b-it
18
0
18
781
645
136
Aria
16
0
16
894
720
174
MiniCPM-V-2.6
16
16
918
688
230
Owen2.5-VL-3B-Instuul
1034
613
421
Proprietary LLMs
03-mini ?
2
Claude-3-7-Sonnet ?
2
0
124
44
80
Claude-3-7-Sonnet
1
0
274
74
200
GPT-44-2024-11-20
17
0
17
399
183
216
Gemini 2.0 Pro Exp
0
0
0
378
254
124
Gemini 2.0 Flash
0
0
633
545
88
Open- Weight LLMs
Lamma33114055-Insttruet
8
0
8
835
741
94
DeepSeek-R1 ?
2
0
2
25
14
11
DeepSeek-V3'
11
0
11
513
206
307
RI-Distill-Llama-708 ?
203
93
Llama-3.a-70B-Instrruet
22
598
272
QwQ-32B ?
9
0
65
34
31
Open-ReasonnerZeen-2238
10
0
10
697
473
224
Phi-4
2
0
2
709
447
262
16

---

Published as a conference paper at COLM 2025
Table WCR of Grid Parsing for all models.
Models
Size
Across
Down
All
Proprietary LVLMs
Claude-3-7-Sonnet Claude-3-7-oonnel ?
0.954=0.009
1.7601
0.855
0,949
0.654
0.800
GPT--1-2024-11-20
0.806
0.448++0.024
0.668+0.015
Gemini 2.0 Pro Exp
0.902
0.603
0.826+0.011
Gemini 2.0 Flash
0.954=00000
00.044
0.667
Open-Weight LVLMs
Fittal-Large-Instruut-1111
124B
0.753
0.361
0.556+0.016
NVLM-D-72B
78.4B
0.429 +0.024
0.009
0.261
InterVL22 5-78B-MPO
78.4B
0.744
0.202
0.501
Owen2.5-V1-722-IIstrruc
73.4B
0.730=0.017
0.554 0..08110.03
QVQ-72B-Preview
73,4B
0.717 10.022
1:0.019 0.139 0..09
0,428
73.2B
0.382 10.021
0.185 0.000
0.281 10.014
gemma-3-27b-it
27.4B
0.746 00021
0,250 1.0021
0.499 a.011
Aria
25.3B
0.258
0.004
0.165
MiniCPM-V-2.6
8.1B
0.091
0.018
0.054
Qwen2.5-V1-3B-Insttue
3.75B
0.023
0.003
0.013
Table 9: WCR and ICR for Chinese, English Simple, and CommonSenseQQ puzzle sets.
Chinese
English Simple
CommonSenseQA
Models
WCR
ICR
WCR
ICR
WCR
ICR
Proprietary LVLMs
GPT-4o-2024-11-20
0.366 0.170
0.335 0.227+0.017
0.39220.006 0.247+0.026
Claude 3.7 sonnet
0.339 +0.023 0.267 00.018 +0.018
0.408 0.408 0.288
0.540 0.38610.028
Gemini 2.0 Flash
0.208 0.233_0.018
0.229 0.216400013
0.327 #0.021
0.216440018
Open-Weight LVLMs
Fistal-large--Isttrut22111
0.252+0015 0.101
0.216+0.016 0.187+0.0
0.439 +0.023
0.270++0.02
Qwen255-VL77B-Insttrut
0.391 100025 0.282 10.018
0.239 10.016
0.183
0.418 10.22
0.252...025
Proprictary LLMs
GPT-40-2024-1114
0.593 0.448 0.001
0.438=0.200
0.306+0.002
0.524
0.326+:.029
Claude 3.7 sonnet
0.478
0.487 0.529 0.4
40.02
0.583 0.024
0..18
03-mini-high !
0.774
0.913
0.782 0.94610.012
0.812 0.97110.01
Open-Weight LLMs
DeepSeek-R1 ?
0.907
0.759 0.898_0.1s 0.787 =0.020
0.752
0.82910.02
QwQ-32B ?
0.701 10.000 0..24
0.647 +0.02 0.734 10.20
0.699 10.026 0.766 10.027
Table 10: 03-mini performance on 7x7 English puzzles across three distinct reasoning efforts.
Reasoning Effort
WCR
LCR
ICR
Avg. Tokens
High
0.587 +-00223
0.684
10.01
49865
Medium
0.534
0.634
0.777
23723
Low
0.295
0.392
0.363
4770
17

---

Published as a conference paper at COLM 2025
C.5
Token Usage
In this section, we report the token usage for all evaluated models on both the 7x7 and 14x14 English puzzles. Notably, we include non-reasoning models in this analysis, defining token usage as the total number of completion tokens. For reasoning models, token usage is calculated as the sum of reasoning tokens and response tokens. As shown in Figure token usage increases with grid size across all models, with reasoning models generating more.
Token Usage Per LVLM for 7x7 and 14x14 Puzzles 77x7 14x14
18k
Token Usage
16k
14k
12k
10k
8k
6k
4k
2k
0
Aria gemini-2.0- 2-rprr pro
MinCPM- V-2 claude-3-7 1oa-dde--7--nnn
gemma-3- 3--mmm----21- 3-------ttit
Qwen2 5- VL- IL-33--nsttrut Instructt
llava-one celva-onevisse inevision-2
claude-3- -soonnet GPf-40
Semini-2.00 --llash2.0--lash flfash
QVO-72B
Pixtral Large- -ar----rrg---tee Instruct-244 2411 CVV--2B--0
Owen2.52 VL-V2LL-BB2-- 1-72--1B-- NVL NVLM-D-72
Intern VL2 5-788-77880 MPO
3-7---net- -sonnet
------B------ Prev-erevveeee review
(a) Token Usage of LVLMs.
Token Usage Per LLM for 7x7 and 14x14 Puzzles 7x7 14x14
60k
Token Usage
50k
40k
30k
20k
10k
0
Dieepeeee R1
claude daude-3-7 3-7 3-7-sonnee -soneet ?
03-min mini (high)
phi-4
03-m --mi R
Open- Reasoner-Zero-331 32ro-32B
GPT GPT-4o deepseek-n
Ilama -lama-405b-1--
Llama-3.3 3-0------Bss-ttcc 4055-1n-17-- Instruct
gemini-2 01-flash
claude daude-3-7 -sonnet
OwQ-32B
03-mini (low)
R1- RI-Distill Distill-Llama-700 Llama-70B
O-pro
(b) Token Usage of LLMs.
Figure 8: Token usage 7x7 and 14x14 English puzzles. Reasoning models in red.
C.6 Ablations: LVLM Input Format and Modality Effects
To isolate modality-specific effects, we explore feeding LLMs the OCR-extracted output from grid images. Our experiments show that while both OCR models and LVLMs can accurately extract clue text from images, they struggle to follow instructions and to construct spatially coherent representation of the empty grid. For example, when applying stepfun-ai/ GOT OCR-2.0-hf to image containing both the grid and clues, it generates "Across:) Canon
18

---

Published as a conference paper at COLM 2025
product, for short n4. Defense n16' Certain fraud protector, for short Abbr. before founding
date' n110 Philosopher's 's study',n2 m12. May honoree' n13. nn.. Fraternity nDown: ml. Like white
panthers),n2 n2. [Not my mistakel n3. Beta dog's n1. Gridiron abbr. (18. One of the muskrats in the 1976 hit "Muskrat Love" "19. Slow--wittd n111. Going rate?: Abbr. \n"... While the clue text is correctly extracted, the output lacks any representation of the grid structure, rendering unsuitable as input for LLMs due to the absence of critical structural constraints. On the other hand, the clue content extracted by OCR models and LVLMs is nearly identical to what we explicitly feed to LLMs in the text input setting. In fact, we augment the clue text with explicit positional hints to compensate for the lack of visual spatial information- that LVLMs might infer from the image but LLMs cannot derive from plain text alone.
In our main evaluations, both the clues and the grid are embedded within single image as inputs for LVLMs. In this section, we investigate the effect of isolating the clues and providing them to the LVLMs as separate text inputs, while the image contains only an empty grid. The prompt used for this experimental setting is shown in Figure 11 in Appendix D.
The results, shown in Figure and measured by WCR on 7x7 English puzzles, reveal no significant performance differences between the two input formats. This indicates that the input format not the primary cause of the suboptimal performance observed for LVLMs on Wordeen suggesting that LVLMs are robust to slight variations in input format.
Image Grid + Text Clue vs. Image Only
Image Grid + Text Clue Image Only
Word Coverage Rate
0.4
-0.004
0.35
+0.016
0.3
0.25
-0.031
+0.065
0.2
0.15
0.1
0.05
0
Intern IntermVL2,5578B-MMP 5
Qwen2.5-VL-72B-Inntte
GPT-40
Pixtral-Large-Instree
Figure 9: WCR difference on two inputs formats for LVLMs.
C.7 More on Crossing Letter Count
Table 11: Average WCR results by crossing
letter count on 7x7 and 14x14 English puzzles.
Table 11 presents the results shown in Fig-
ure 5, along with additional results 14x14 English puzzles. On these larger puz-
Grid
Crossing Letter
Reason.
Non-Reason.
7x7
Low
0.4539
0.2922
zles, even reasoning LLMs struggle to effec-
7x7
Med
0.5308
0.3230
tively leverage crossing-letter constraints.
7x7
High
0.6183
0.3259
We hypothesize that this is due to the in-
14x14
Low
0.4115
0.3469
creased complexity of larger puzzles, which remains challenging for reasoning models.
14x14
Med
0.4186
0.3514
14x14
High
0.2306
0.1438
C.8 More Few-Shot Prompting and Prefill Ratio Control
In our main evaluation, we treat each puzzle whole rather than set of independent question-answer (QA) pairs, in order to preserve the structural constraints inherent to crossword solving. This setup is not well-suited to few-shot prompting, as providing full puzzle solution as a demonstration example would lengthy and potentially distracting for the model. Nevertheless, we experiment with one-shot prompting on 7x7 English puzzles,
19

---

Published as a conference paper at COLM 2025
using completed example generated by DeepSeek-RI on the Simble subset as the demonstration. As shown in Figure 12a, the performance difference not significant.
As discussed in Section 5.1, our controllable generation framework supports difficulty adjustment via the prefill ratio. In Table 12b, we compare performance on 7x7 English puzzles under two prefill conditions: 0% and 50%. To ensure fair comparison, we prevent any word from being fully revealed. As expected, a higher prefill ratio leads to improved performance, as more letters are made available to the model. These results suggest that prefill ratio is more effective method for puzzle difficulty control than few-shot prompting.
Table 12: WCR under different prompting and prefill ratio settings on 7x7 English puzzles.
(b) WCR with 0% and 50% prefill.
(a) WCR with 0-shot and 1-shot prompting.
Model
0%
50%
Model
0-Shot
1-Shot
GPT-4o (Text)
0.410
0.483
GPT-4o (Text)
0.410
0.394
GPT-4o (Img)
0.348
0.383
Claude-Sonnet (Text)
0.482
0.519
Claude-Sonnet (Text)
0.482
0.604
Gemini-Flash (Text)
0.301
0.313
Claude-Sonnet (Img)
0.479
0.568
Llama-3.3-70B (Text)
0.303
0.269
Gemini-Flass (Text)
0.301
0.362
DeepSeek-V3 (Text)
0.303
0.307
Gemini-Flash (Img)
0.277
0.346
o3-mini (Text)
0.587
0.864
D Implementation Details
D.1 Evaluation Metrics
Here, we provide a more complete and formal description of the three metrics used.
Word Coverage Rate (WCR): WCR measures word-level accuracy by calculating the percentage of correctly solved words in the crossword puzzle, defined as:
1
WCR
Σ (r mw) WEWA
Σ 1 [ mw] DEW,
(2)
[WAl+ + [WDl
where WA and Wp denote the set of across and down words, respectively. For each word w, Tw represents the reference answer, while my represents the model answer.
Letter Coverage Rate (LCR): LCR evaluates letter-level accuracy, providing partial credit for correct letter placements. For each word w, Let:
min(Iral mwr)
C#
1 [r mw [i]} and L max (Irwl, I |
where Cp counts the correctly matched letters and Lw is the total number of positions considered, the overall letter accuracy is defined as:
Evew Cw
LCR
(3)
Ewew Lw
Intersection Consistency Rate (ICR): the internal consistency of the model's answers at intersections where across and down words overlap, defined as:
1
ICR
Σ
where I denotes the set of all intersections, where each tuple (a, j,k, indicates the jth letter of the across word a overlaps with the kth letter of the down word d. This metric reflects whether models correctly adhere to the grid structural constraints of puzzle.
20

---

Published as a conference paper at COLM 2025
D.2 Parsing Details
Answers extracted from model responses converted into JSON format by leveraging the structured output capabilities of o3-mini and dynamic Pydantic models that adhere to the reference answer structure. Algorithm provides the pseudocode for creating these models.
Algorithm Create Dynamic Pydantic Model
1: function CREATE DYNAMIC.PYDANTIICMMM
MoDEL(reference PTDAATIC.MODDD((eeeeennn answerss)
2: fields <
> Initialize an empty dictionary
3: pattern +- (across \sdd)
\s+\d+\$
for all answer E reference.anee waw.eerrssssssssseeeeoee do
5: key + answer -nnswer""direct ("direcioon"""
6:
if key does not match pattern then
7:
raise ValueErroor (Refferee key does not match expected format")
8:
end if
9: fields[key] f- ((ptioonal[Chuekksseer) Field with description "Answer for clue at key")
10:
end for
11:
SonamicPPydanticMoon + create
"fieees **feelds)
12: return DynamiccPPoantii.MMoeelll
13:
end function
D.3
Generation Configuration
Table 13 lists all the models evaluated along with their corresponding generation configs.
Model Name
Max Tokens (7x7) Max Tokens (14x14) Temperature
Claude 3.7 Sonnet (Anthropic, 2024)
8192
8192
0.0
Claude 3.7 Sonnet (Thinking) (Anthropic, 2024)
64000
64000
1.0
GPTT-4-2000 100-2004--1-2- (Hurst et al., 2024)
16384
16384
0.0
Gemini 2.0 Pro Exp (DeepMind, 2024)
20480
20480
0.0
Gemini 2.0 Flash (DeepMind, 2024)
20480
20480
0.0
03-mini-high (OpenAI, 2025)
100000
100000
0.6
Pixtra-Large-lnstrrut-21111 (Agrawal et al., 2024)
20480
20480
0.0
NVLM-D-72B (Dai et al., 2024)
20480
20480
0.0
MttemVL25-78B-MPP (Chen et al., 2024)
20480
20480
0.0
Owen2.5-VL-72B-nnstruu (Bai al., 2025)
20480
20480
QVQ-72B-Preview (Team, (024)
100000
100000
llvaa-onersionn72ooovv-hh (Li et al., 2024a)
20480
20480
0.0
gemma-3-27b-it (Team et al., 2025)
8192
8192
0.0
Aria (t et 2U24b)
20480
20480
0.0
MiniCPM-V-2..6 (Yao et al., 2024)
20480
20480
0.0
Qwen2.5-VL-3B-Insttee (Bai et al., 2025)
20480
20480
0.0
1lama-31-4055--Instmme (Dubey et al., 2024)
100000
100000
0.0
DeepSeek-R1 (Guo al., 2025)
100000
100000
0.6
DeepSeek-V3 (Liu et al., 2024)
100000
100000
0.0
RI-Distil--Lamm--0B (Guo et al., 2025)
100000
100000
0.6
Llama-3.3-70B-Instrreed (Dubey et al., 2024)
20480
20480
0.0
QwQ-32B (Team, 2025)
100000
100000
0.6
Phi-4 (Abdin et al., 2024)
10000
10000
0.0
Table 13: Generation Configurations for Beddshh..
21

---

Published as a conference paper at COLM 2025
D.4 Prompts
Figures 10, 11, 12, 13, 14, 15, 17, and 18 present all the prompts employed in this compricnn image prompts for LVLMs and text prompts for LLMs. We observe that Claude 3.7 Sonnet sometimes produces partial outputs and requests confirmation to continue. To mitigate this issue, we incorporate an additional system prompt (see Figure 18 for the system prompt); note that this modification applies only to the Claude 3.7 Sonnet.
Image Zero-Shot CoT Prompt
You are given crossword puzzle image containing clues grid. Your task to solve puzzle accurately, ensuring that all answers fit both the given clues and the grid structure, including intersecting words.
For each clue, provide a step-by-step explanation: by EXACT sequential (e.g., Across clues night be numbered 1, 4, and while Down clues night be 2, 3, 8).
Identify
DIRECTION
inage. numbers may
Deternine word length fron available grid spaces.
Check for any pre-filled letters from intersecting words that have already been solved and explain how
they constrain passible
Analyze the clue (definition, wordplay, cryptic hint).
Explain your reasoning process.
Confirm alignnent with crossing letters.
Solving tips:
Answers must single word with no spaces (conbine phrases if needed). Abbreviations clues typically indicate abbreviated
Natch the clue's tense, singular/plural forn, and part of speech.
(?) puns cryptic Down words are filled from top to botton, Across words from left to right. Always confirm that intersecting words renain valid after placing each answer.
wordplay signals,
question
Present your final solution as:
DNumber shown in inagel: [Answer]
Down:
[Nunber as shown in image]: [Answer]
IMPORTANT:
list clues sequential numerical order. match nunbering pattern from the
image.
NOT for confirmation or stop midway. Always provide complete solution all clues.
Figure 10: Image Zero-Shot CoT Prompt.
22

---

Published as a conference paper at COLM 2025
Image Zero-Shot CoT with Grid Only Prompt
You are given crossword puzzle image containing only the grid, while the clues are provided separately solve puzzle accurately, ensuring given clues and the grid structure, including intersecting words.
Clues:
Each clue contains:
Clue Direction and Nunber "Across 1", "Down 2").
Start Position (row, column) for the first letter of the answer
<clues>
For each clue, provide a step-by-step explanation:
Identify the clue by its EXACT NIMBER AND DIRECTION shown the clue description. The nunbers may
not be sequential (e.g., Across clues might be numbered 1, 4, 7, and 9, while Down clues night be 2,
3, 6, 8).
Determine word length from available grid spaces.
Check for any pre-filled letters from intersecting words that have already been solved explain how
they constrain possible
Analyze the clue (definition, wordplay, cryptic hint)..
Explain your
process.
Confirm alignnent with crossing letters.
Solving tips:
Answers nust single word with no spaces (combine phrases if needed).
Abbreviations clues typically indicate abbrevisted
Match the clue's tense, singular/plural forn, and part of speech.
Looo for wordplay signals,
question marks for puns cryptic hints.
Down words are filled from top to botton, Across words from left to right. that intersecting words remain after placing answer.
Always
Present your final solution as:
Across:
DNumber as shown in clues]: [Answer]
Down:
[Nunber shown in clues]: [Answer]
IMPORTANT:
DO NOT list clues in sequential numerical order. MUST match the exact nunbering pattern from the
- given clues.
confirmation stop midway. Always provide complete solution all clues.
Figure 11: Image Zero-Shot CoT with Grid Only Prompt.
23

---

Published as a conference paper at COLM 2025
Text Shro-Shot CoTT Prompt
You are given crossword puzzle grid and set of clues. Your task is to solve the puzzle accurately, ensuring
and structure, including intersecting fit both the
words.
Grid Representation:
The crossward grid is represented as array where: represents black (blocked) cell
represents an empty (unfilled) cell
<grid>
Clues:
Each clue contains:
Clue Direction and Number *Across "Down 2").
Start Position (row, colunn) for the first letter of the answer
For each clue, provide a step-by-step explanation:
Identify clue by its EXACT NUMBER DIRECTION shown clue description. nunbers may not be sequential (e.g., Across clues might be numbered 1, 7, and 9, while Down clues night be 2, 3, 6, 8).
Determine word length from available grid spaces.
Check any pre-filled letters intersecting words that have already been explain how they constrain possible answers.
Analyze the clue (definition, wordplay, cryptic hint).
Explain your reasoning process.
Confirm alignnent with crossing letters.
Solving tips:
Answers nust single word with no spaces (combine phrases needed),
Abbreviations in clues typically indicate abbreviated
Match the clue's tense, singular/plural forn, and part of speech.
Look wordplay signals,
puns cryptic hints.
Down words are filled from top to botton, Across words from left to right. intersecting
Always
placing answer.
Present your final solution as:
Across:
[Nunber as shown in clues]: [Answer]
Down:
clues]: [Answer]
IMPORTANT:
DO NOT list clues in sequential nuterical order. You MUST match the exact nunbering pattern from the
given clues.
confirmation stop midway. Always provide complete solution clues.
Figure 12: Text Zero-Shot CoT Prompt.
24

---

Published as a conference paper at COLM 2025
Interactive Mode Prompt
You are given a crossword puzzle inage containing clues and grid. Your task to solve the puzzle accurately, ensuring that all answers fit both the clues and the grid structure, including intersecting words.
Pick ONE clue, provide step-by-step explanation: Identify the clue by its EXACT MUMBER AND DIRECTION shown inage. The numbers may not be sequential
Across clues night numbered 1,
while Down clues night be 3,
+> and 8).
word length from available grid spaces.
Check for any pre-filled letters from intersecting words that have already been solved and explain how they constrain possible answers.
Analyze the clue (definition, wordplay, cryptic hint).
Explain your reasoning process.
Confirm alignment with crossing letters.
Solving
Answers nust be single word with no spaces (conbine phrases if needed).
Abbreviations clues typically indicate abbreviated answers.
Match the clue's tense, singular/plural forn, and
of speech.
Look for wordplay signals, such as question marks (?) for puns or cryptic hints.
Down words filled from top botton, Across words from left to right.
Always confirn that intersecting words remain valid after placing each answer.
Only solve ONE clue at a time and wait for confirmation before proceeding to the next round.
Figure 13: Interactive Mode Prompt.
Interactive Mode Follow-up Prompt
For the following round:
Using the confirmed answers far:
clue, provide step-by-step explanation:
Identify the clue by its EXACT NIMBER AND DIRECTION shown in the image. The numbers may not be sequential (e.g., Across clues night numbered 1, 4, 9, while Down clues night 2, 3, 5, and 8).
Determine word length from available grid spaces.
Check for any pre-filled letters from intersecting words that have already been salved and explain how - they constrain possible answers.
Analyze
(definition, wordplay, cryptic hint).
Explain your reasoning process.
Confirm alignnent with crossing letters.
Solving tips:
Prioritize clues that intersect with confirmed
Answers must be single word with no spaces (conbine phrases if needed).
Abbreviations
typically
abbreviated answers.
Match the clue's tense, singular/plural forn, and part of speech.
Look for wordplay signals, such as question marks (?) for puns or cryptic hints.
Down words are filled from top to botton, Across words from left to right.
Always confirn that intersecting words remain valid after placing each answer.
Only solve ONE clue at a time and walt for confirmation before proceeding to the next round and do not clues.
repeat previously
Figure 14: Interactive Mode Follow-up Prompt.
25

---

Published as a conference paper at COLM 2025
Grid-Fansing lrrupt
task is extract and natch all words from puzzle grid with their respective clues.
image consists of two sections:
Section:
Contains two lists: "Acrass" and "Down."
Each
nunbered corresponds starting position grid.
The Grid Section:
crossword grid containing letters, empty cells, and numbered starting positions for words.
extend either
(left right) down (top botton).
Grid Identify all clues under the "Across" and "Down" sections, preserving their numbers. Identify nunbered cells grid.
Step 2: Extract Words fron the Grid
For each numbered cell:
If the word extends ACROSS:
right reaching an empty Start at the numbered cell and read consecutive letters top to botton until reaching empty cell
numbered
consecutive
- grid boundary.
If the word extends DOWN:
- or grid boundary.
Step 3: Natch Words to Clues
grid corresponding in Down
Ensure extracted words are correctly assigned to their respective clues.
Output Format:
ACROSS:
imagel: [Clue Text]
Extracted Word: [Nlord fron Grid]
DOWN:
DNunber as shown in imagel: [Clue Text] Extracted Word: [Nlord fron Grid]
accuracy natching their clues,
fully without cnitting any.
Figure 15: Grid-Parsing Prompt.
Self-Reflection Prompt
Your previous solution contains incorrect answers. Take step back, carefully re-examine your entries, and systenatically verify each word complete consistency and correctness within the crossword puzzle.
Provide step-by-step verification:
Cross-Check Letters: List every intersection explicitly, noting clues steet.
Across Down Consistency Check: Verify that each intersection natches perfectly. Identify and highlight any conflicting letters imediately.
3. Clue Validation: Revisit each clue thoroughly, confirming that each answer fully aligns with its clue description adheres strictly length constraints.
Grid Integrity: Confirm that your corrected entries naintain the integrity of the puzzle grid, leaving no unresolved conflicts or empty cells.
After completing these steps, present your revised and verified solutions in the following format:
[Clue Number]: [Corrected Answer]
Down:
[Clue Number]: [Corrected Answer]
IMPORTANT:
nunbering Do NOT restate previous incorrect answers. Provide only fully corrected solutions after reflection.
1ist clues sequential nuterical order.
Figure 16: Self-Reflection Prompt.
26

---

Published as a conference paper at COLM 2025
Ansswer-Parsing System Prompt
You are a crossword puzzle answer extractor. Extract only valid answers from text response containing crossword solutions.
If an answer contains spaces multiple words, combine then into single word.
not shift reorder answers. For example, the expected keys are "Down 12" and "Down and only the for "Down 13" provided in the text, then "Down 12" should be null and "Down 13" should +> contain the answer.
Output your response given explicitly stated the
invent infer
Figure 17: Answer-Parsing Prompt.
Claude System Prompt
You are a helpful assistant who completes tasks fully without seeking confirmation. Your role is to deliver comprehensive responses one go. Never if user you or more must provide the complete response.
Figure 18: Claude System Prompt.
D.5
Textual Grid Representations
0
1
2
3
4
0
1
0
0
0
0
0
0
0
0
1
0
0
0
2
1
0
1
0
0
0
0
0
1
0
4
1
1
0
0
0
(a) 2D array format (ARC-style).
(b) Markdown-stylle grid with symbols.
Figure 19: Two text representations of the puzzle grid: array (left) and markdown (right).
27


Reviews:
[{"review_id": "cgBHKEu7O7", "reviewer": "Reviewer_kmED", "summary": "The paper proposes a new benchmark for evaluating multimodal and text based reasoning and non-reasoning models on crossword tasks. The authors dont rely on public benchmarks but rather on novel curated crosswords through puzzles, dictionaries, etc. The authors also compare the performance on different models and find interesting insights like (1) vision models are worse than language models; (2) reasoning models are always better.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The crossword problem is interesting and a good task to evaluate the models on reasoning and visual understanding.\n- The evaluation is thorough and offers interesting insights.\n- The crossword construction can be extended to specific domains and more sophisticated benchmarking.\n- The proposed metric (performance on intersection) is interesting.", "reasons_to_reject": "- The vision models are not given the text input also which is a big threat to validity for the hypothesis (vision models are worse). It would be more interesting to see if the vision models are given additional input and how that helps.\n- The 1 shot setup seems very restrictive and not sure if the evaluation in 1 shot is fair.\n- Was any qualitative study done on the crosswords? stats like number of instructions will validate the quality of the data.\n- All vision models are worse than their text only counterparts, which indicates that the problem is not equivalent for vision and non visison.", "questions_to_authors": "- Why was the text input not given to vision models. Otherwise this makes it an OCR task.\n- Why not GPT-4o non vision model?\n- In multi shot setting, which clue does the model choose to answer? the first one or any one it chooses?\n- In multi shot, if the model runs into an error where a prediction cannot be added to the grid, what happens? does the turn end?\n- Were any pairs filtered? for non obvious dictionary words for example?", "extra_scores": {}}, {"review_id": "5J1fAbjA6G", "reviewer": "Reviewer_XYjy", "summary": "This paper presents CrossWordBench, a novel benchmark designed to evaluate the reasoning capabilities of LLMs and LVLMs through crossword puzzles that integrate both semantic (text-based clues) and structural (grid-based) constraints. The authors develop a controllable puzzle generation pipeline that supports both image and text formats, enabling direct and interactive evaluation modes. By evaluating over 20 state-of-the-art models, the study finds that reasoning LLMs outperform non-reasoning models, especially in leveraging crossing-letter constraints, while LVLMs underperform significantly due to weak grid parsing abilities. The benchmark reveals substantial gaps in multimodal reasoning, offering a useful testbed for future research on constrained, verifiable tasks in both LLMs and LVLMs.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* **The dataset is useful and may have potential to serve as a standardized testbed** for evaluating multimodal reasoning under structural constraints, which is currently underexplored in existing benchmarks. By leveraging crossword puzzles that integrate both textual clues and spatial grid constraints, the benchmark opens up new directions for studying verifiable and compositional reasoning in both LLMs and LVLMs.\n\n* **The evaluation is comprehensive, covering a wide range of models** including both proprietary and open-weight LLMs and LVLMs, reasoning and non-reasoning variants, and multilingual settings. The authors also explore different puzzle formats, grid sizes, reasoning strategies, and interactive solving setups, providing a thorough analysis of model performance under varied conditions.\n\n* **The paper is well-written and clearly presented**, with intuitive figures, detailed methodology, and insightful analysis. The empirical findings are clearly tied back to the motivation and demonstrate the practical value of the benchmark.", "reasons_to_reject": "* **The novelty and scope of the benchmark may be limited** in terms of its ability to generalize beyond crossword-style tasks. While the integration of textual and visual constraints is interesting, the setting may not fully capture the breadth of reasoning challenges faced in real-world applications of LLMs and LVLMs.\n\n* **The findings offer limited new insights** beyond confirming that reasoning models perform better and that LVLMs struggle with visual parsing. Many of the conclusions align with existing understanding, and there is little discussion on deeper failure modes or unexpected behaviors.\n\n* **The paper does not sufficiently explore different training or inference methods**, such as supervised fine-tuning, reinforcement learning tunning, test-time scalling, or more advanced prompting strategies (e.g., multi-step or multi-agent prompting). This limits the utility of the benchmark as a tool for driving methodological innovation.", "questions_to_authors": "N/A", "extra_scores": {}}, {"review_id": "zhe5eyFR5i", "reviewer": "Reviewer_7ztd", "summary": "This paper introduces a benchmark for evaluating the reasoning capabilities of  Large Vision-Language Models (LVLMs) using crossword puzzles. The benchmark leverages a controllable puzzle generation framework that produces puzzles in text and image formats from various sources (public repositories, dictionaries, adapted benchmarks). The authors evaluate over 20 models using metrics assessing word/letter accuracy and consistency with crossing constraints (ICR). Key findings include: reasoning LLMs significantly outperform non-reasoning LLMs and LVLMs, particularly in leveraging constraints; LVLMs struggle significantly, with performance correlating strongly with grid-parsing accuracy; even puzzles from saturated benchmarks remain challenging. The paper also proposes an interactive evaluation mode and analyzes performance variations based on crossing letters, self-reflection, and test-time scaling.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Focus on Constraint Satisfaction: The emphasis on intersectional constraints (and the ICR metric) highlights a crucial aspect of reasoning that is often under-explored in standard benchmarks – the ability to satisfy multiple, interlocking hard constraints.\n\n2. Controllable and Scalable Benchmark Generation: The idea of a framework that can generate diverse puzzles from multiple sources (public repos, dictionaries, adapted QA) and in different formats is a significant strength if well-executed. This offers flexibility and potential for creating varied and challenging evaluation sets.\n\n3. Evaluation of Both LLMs and LVLMs: Systematically comparing both types of models on a task that can be framed bimodally provides a broader understanding of current model capabilities and their respective weaknesses.", "reasons_to_reject": "1. Limited Novelty and Contribution: The use of crosswords in NLP is not new. The paper should show its contribution from prior work beyond citing limitations of existing datasets. Crucially, it fails to articulate why this specific task, compared to existing complex multimodal reasoning benchmarks, offers a more significant or fundamentally different evaluation of the claimed \"dynamic interplay\" between modalities. \n\n2. Opaque Puzzle Generation: The core \"controllable puzzle generation framework\" is presented as a black box. Critical technical details regarding the algorithm for word placement, constraint management, black square insertion, and difficulty control (beyond grid size) are missing. It would help the readers if the authors could provide them. \n\n3. Lack of Validation: There is no validation demonstrating that the generated puzzles possess comparable quality, difficulty, or structural properties to standard human-created crosswords. Can there be a comparison between the same. This could strengthen this work. Human baselines could also help. \n\n4. Claims about the benefits of crossing constraints (Sec 5.1) are based on observed correlations rather than controlled variation. If authors could do an ablation study regarding isolating the effects of grid size, clue difficulty/ambiguity, or crossing letter density on performance, it would help the reader. \n\n5. The evaluation does not control for or isolate the impact of image resolution, grid clarity, font variations, or underlying OCR capabilities, making it difficult to disentangle visual perception failures from reasoning failures. A control experiment feeding LLMs the OCR'd output from the grid images could assist to isolate modality-specific effects.\n\n6. Assertions that better ICR indicates superior \"reasoning capability\" are unsubstantiated; it may merely reflect better pattern matching or constraint satisfaction heuristics specific to this grid format.", "questions_to_authors": "1. A model could achieve high ICR by being consistently wrong across intersections, providing a false sense of constraint adherence accuracy. Is ICR the best suited metric to be chosen here ?\n\n2. The \"Agentic Evaluation Setting\" (Sec 4.4) lacks detail on error feedback mechanisms and most models fail immediately (ISS mostly 0 or 1). Can the authors provide more details for the same. \n\n3. Could it be that the strong correlation found between LVLM grid-parsing accuracy and puzzle-solving performance (Fig 3, Table 3), along with the Across/Down bias, suggests that LVLM failures are dominated by fundamental limitations in OCR and fine-grained spatial layout understanding, rather than deficits in higher-level multimodal reasoning about constraints ?", "extra_scores": {}}, {"review_id": "E1UYCMoFFH", "reviewer": "Reviewer_KRbZ", "summary": "The authors propose \"CrossWordBench\" which is intended to evaluate VLMs and LMs reasoning abilities over crossword puzzles. They produce a crossword generator which can vary difficulty level and other characteristics, sidestepping various issues in prior work using static crossword puzzle sets.\n\nThis is another task that reasoning models clearly outperform LMs on, and one that VLMs struggle at slightly compared to LMs.\n\nThey collect existing clue, word pairs and can generate text or visual puzzles from this. They evaluate models both in zero-shot and multi-turn settings, in English and Chinese.\n\nThey introduce metrics to score partially completed crosswords including word accuracy, character accuracy, and intersection coherence.\n\nIn addition to simple zero-shot eval, they evaluate VLMs in an agentic, interactive mode.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Useful and sensible benchmark design. Major pluses include being dynamic, which is novel relative to other crossword benchmarks.\n\nThere are a lot of interesting findings in here, including the tendency of VLM-based agents to succeed conditioned on not failing on the first few guesses which could start future analysis.\n\nFairly comprehensive set of models evaluated.", "reasons_to_reject": "Some method details are vague in the main text and require too many trips to the Appendix. For example, the description of the interactive mode agent is not given, but can be kind of inferred from reading the example prompts in the appendices. They don't even tell you in the main text that these prompts are in the appendices, though. I'm assuming the implementation is a simple while loop that starts with the initial prompt and then \n\nIn the text-only mode, how is the input puzzle formatted? Were ablations on this format performed like the VLM format ablations?\n\n**Please make some of these implementation details a little easier to comprehend in the main text if accepted.**", "questions_to_authors": "Elements of your evaluation like intersection accuracy strictly apply to how an LM would solve a task like this---no human will fill in, or even think to fill in, a crossword with intersecting words that don't match with their intersecting letter. This feels like a more \"catastrophic\" failure that renders their understanding of the puzzle as a whole mistaken, as opposed to word or character-level errors in an otherwise valid puzzle. So, I am at a loss for how to interpret cases where WCR is significantly higher than ICR; is the model just ignoring intersection consistencies and filling in a bunch of words with coherent, but wrong examples? Does the model have a chance to \n\n**I would like to see a discussion of the interplay between these metrics that maybe addresses some of these questions in the CR if accepted.**", "extra_scores": {}}, {"review_id": "5iUZHPplRW", "reviewer": "Reviewer_TLs7", "summary": "This paper presents an evaluation framework for LLMs in the context of crossword puzzles.  Specifically, the authors present a novel framework for generating small (7x7 and 14x14) English and Chinese crossword puzzles.  Via prompting, they explore the performance of both language-only and language and vision LLMs on dataset of generated crosswords.   The contributions include the methods for generating puzzles, the evaluation metrics and framework and an analysis of the performance of a suite of modern LLMs and LVLMs.  The evaluation includes both reasoning and non-reasoning models.  Results indicate that large reasoning models outperform standard zero-shot methods and that visual language models struggle with this task. \n\nOverall the paper is well-structured and clear with appropriate treatment of the early literature on this task.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper represents an interesting take on a task that has fascinated NLP researchers since Littman's work on Proverb. It serves as an interesting benchmark combining traditional word-based reasoning with constraint-based visual processing.", "reasons_to_reject": "There are 2 limiting possibilities for this benchmark:  1) that the results will be too specific to crosswords and not really tell us much about LLMs ability to employ constraint-based reasoning, or 2) that the word-level results are really just an odd form of Q/A and results on this benchmark don't tell us much more than what we already know with more standard QA benchmarks.  \n\nAnother minor criticism is the disconnect in the evaluation with earlier work involving actual crosswords. Such puzzles likely involve aspects not captured in the generation scheme used here.  While the issues of data contamination and copyright are real, it would nevertheless to present a small set of results based on real crosswords that have been employed all the way back to the Proverb system.  If only to demonstrate the gap between the kind of data presented in CrossWordBench and \"real\" crosswords.  Another avenue of extrinsic evaluation would be the NY Times \"mini\" crossword.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KWyQNKgGCJ", "reviewer": "Reviewer_wJrT", "summary": "This paper introduces CrossWordBench, a novel benchmark designed to evaluate the multimodal reasoning capabilities of both LLMs and LVLMs through the task of solving crossword puzzles. Unlike existing benchmarks that separately assess either textual reasoning or vision-language understanding, CrossWordBench requires models to integrate semantic constraints from textual clues and structural constraints from crossword grids. The benchmark introduces a controllable puzzle generation framework that supports multiple formats (text and image) and includes both direct and interactive evaluation modes. Experiments on 20+ models show that reasoning-capable LLMs significantly outperform non-reasoning models, while LVLM performance strongly correlates to their ability to parse visual grids. These results highlight current limitations in multimodal reasoning and suggest directions for future evaluations.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Novel benchmark design: The paper presents a controllable crossword puzzle generation framework that uniquely combines semantic reasoning from textual clues with structural reasoning from crossword grid constraints. The benchmark supports diverse formats (text and image) and includes both direct and interactive evaluation modes, enabling a broad assessment of model capabilities.\n* Thorough evaluation and insightful analysis: The experiments offer a comprehensive evaluation across a wide range of LLMs and LVLMs, revealing clear limitations in current models’ multimodal reasoning abilities and offering insights such as LVLM performance is strongly tied to their ability to parse visual grids.", "reasons_to_reject": "* Lack of proposed solutions: While the benchmark identifies current limitations of LLMs and LVLMs in multimodal reasoning, the paper does not explore training strategies or data augmentation methods to address these issues, resulting in its limited technical contribution.", "questions_to_authors": "Could the authors suggest potential training strategies or data augmentation techniques that might improve the performance of open-weight models on this benchmark?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a a novel benchmark evaluating LLMs and LVLMs on constraint-based reasoning using dynamically generated crossword puzzles. It is a fun paper, testing another type of reasoning capabilities of foundation models. The authors conduct a thorough evaluation across 20+ models and introduce metrics capturing both accuracy and structural consistency. While some concerns about generalizability and missing methodological details were raised, the authors provided detailed and convincing responses. The benchmark offers a unique testbed for multimodal reasoning and agentic evaluation, and is likely to stimulate further research. The authors should incorporate their responses and address reviewers' comment in the final cam-ready version of this paper.  I recommend acceptance.
Decision: Accept