VisuRiddles: Fine-grained Perception is a Primary Bottleneck for Multimodal Large Language Models in Abstract Visual Reasoning
Authors: Hao Yan
Xingchen Liu
Hao Wang
Zhenbiao Cao
Handong Zheng
Liang Yin
Xinxing Su
Zihao Chen
Jihao Wu
Minghui Liao
CHAO WENG
Wei Chen
Yuliang Liu
Xiang Bai
Venue: ICLR 2026 Poster (2026)
Keywords: MLLMs
AVR
Fine-grained Perception
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent strides in multimodal large language models (MLLMs) have demonstrated significant progress in many reasoning tasks, but they still fail in Abstract Visual Reasoning (AVR) tasks. Our experimental findings indicate that the core bottleneck lies not only in the reasoning capabilities of MLLMs but more critically in their absence of fine-grained perception. To address this issue, we present VisuRiddles, a dedicated resource for AVR research. It consists of (i) a benchmark, collected from real-world data, for the systematic evaluation of MLLMs' AVR capabilities, and (ii) a synthesizer, which automatically generates AVR instances enriched with perceptual descriptions and reasoning chains, enabling supervised training and deeper investigation. Building on VisuRiddles, we propose a two-stage training paradigm that progressively enhances perceptual ability and strengthens reasoning, producing the Perception-Augmented Visual Reasoner (PAVR). Experiments demonstrate that PAVR unifies perception and reasoning, substantially outperforming both open-source and commercial MLLMs, thereby underscoring fine-grained perception as the primary bottleneck in AVR.

Published as a conference paper at ICLR 2026
VISURIDDLES: FINE-GRAINED PERCEPTION IS A PRI- MARY BOTTLENECK FOR MULTIMODAL LARGE LAN- GUAGE MODELS IN ABSTRACT VISUAL REASONING
Hao Yan', Xingchen Liu', Hao Wang", Zhenbiao Cao', Handong Zheng', Liang Yin- Xinxing Su², Zihao Chen², Jihao Wu², Minghui Liao²; Chao Weng"
Wei Chen', Yuliang Liu Xiang Bai
Huazhong University of Science and Technology, "Huawei Inc.
(haoyan, xcliu04, caozhenbiao, hdzheng, liangyin, lemuria_chen, ylliu, xbai jehust.. (wanghao. ai, xinxingsu, mh1iao)@foxxail.com com, (chenzihao23, wujihao]@huawel. com, chao. fhaa.weng@gmail.com com
ABSTRACT
Recent strides in multimodal large language models (MLLMs) have demonstrated significant progress in many reasoning tasks, but they still fail in Abstract Visual Reasoning (AVR) tasks. Our experimental findings indicate that the core bottleneck lies not only in the reasoning capabilities of MILMs but more critically in their absence of fine-grained perception. To address this issue, we present VisuRiddles, a dedicated resource for AVR research. It consists of (i) a benchmark, collected from real-world data, for the systematic evaluation of MLLMs' AVR capabilities, and (ii) a synthesizer, which automatically generates AVR instances enriched with perceptual descriptions and reasoning chains, enabling supervised training and deeper investigation. Building on VisuRiddles, we propose two-stage training paradigm that progressively enhances perceptual ability and strengthens reason- ing, producing the Augmented Visual Reasoner (PAVR). Experiments demonstrate that PAVR unifies perception and reasoning, substantially outperform- ing both open-source and commercial MLLMs, thereby underscoring fine- fine-grained perception as the primary bottleneck in AVR. Code and data are available at httpss//jitubb.com//hh-hu//////s
INTRODUCTION
Enhancing this capability in Multimodal Large Language Models (MLLMs) is key bridging the gap to human-level performance (Schulze Buschoff et al., 2025 Caffagni et al. (2024). Recent advancements (Bai et al., Chen al.]| 2024aj Wu et al.) 2024 Yu et al., 20244
Reasoning is a core component of human intelligence (Johnson-Laird 2010).
et al., 2024 in MLLMs have made significant progress in various general visual tasks, but still struggle with Abstract Visual Reasoning (AVR) tasks (Mitchell et al.). As shown in Fig. Ma), even advanced models such as Gemini2.5 Pro continue to face significant challenges in this task, showing substantial gap compared to human performance. While such AVR tasks are relatively easy for humans, even the most advanced MLLMs struggle to solve them. It unexpected that models with strong performance in general understanding and reasoning tasks still fall short in handling such reasoning challenges. In fact, the challenges of AVR primarily stem from two aspects: fine-grained perception and reasoning. Compared to the recent advances in enhancing reasoning capabilities, the ability to perceive subtle visual structures, such as position, style, and attribute, has been largely overlooked in existing studies (Xu et al.) 2025b Jiang ct al. 2024). Unlike humans, which naturally process such fine-graines visual cues, MLLMS often
2025
lack the perceptual ability required for abstract graphics understanding (Tong et al. 2024 Cao et al. 2024). As illustrated in Fig. [(b,, once abstract graphics are reformulated into structured perceptual descriptions, the model is able to generate accurate responses. This highlights that strengthening perceptual capability is crucial factor for advancing AVR. Therefore, genuine progress in enabling MLLMs to master AVR requires addressing both perception and reasoning, with special emphasis on enhancing the often-overlooked perceptual capability.
"Corresooniigg authors.

---

Published as a conference paper at ICLR 2026
63.0
Ruaken
23,6
25.4
25.5
29,2
38,3
Oreak5
GFT-4e
Ge
Cladd-33 -Taasing
Cloade3.7
Does
Geneniló
Hoaso
-Thnking
(a) Performance comparison the VisuRiddles.
Image Inpot
Percepiual Istractured Input
Ploor dignn froe be tu the gpettor mark gastonn
Plose sdlct a Syym fron Doveptoes io the quedion inthe gastoo
The greetion contrins 2 groege imagle nach grosp with imagee
The fins group onanages are image...
sccond proup of imague one image....
dhe opsioea followa....
Oweed SV
Identify the Pattern:
Anelysis of First
The seguence in the pmbblem
Groupt
finst The fifth shape
Pattern Identification:
Shape: ... Pattern with Optiame:.. ...
Determining the Misinn Shapet
X
ond lusion: Therofors, the coerett optian Option..
C Cunclusion: Option monches the shope the second group
Option
[Wrmm
(Riati
oakko Bffecivv Parception Reconing Fellur
Perceptial Modiling Rescoring Secois
(b) Comparison between abstract graphics and perceptual description input.
Figure I: Performance and analysis of MILMs AVR. (a) Most advanced MLLMs achieve limited accuracy on VisuRiddles, often close to random choice and far below human performance. (b) Model responses to abstract graphies and their perceptual descriptions show that once equipped with perceptual capability, MLLMs can succeed in AVR task.
To advance the study AVR tasks, we introduce the VisuRiddles benchmark, which is primarily derived from real riddles and enables objective evaluation of MLLMs' performance. Since existing datasets lack fine-grained perceptual annotations, we further design the VisuRiddles Synthesizer, which automatically generates AVR instances enriched with structured perceptual descriptions. Leveraging this synthesized data, we perform Supervised Fine-Ture Tuning (SFT) to enhance the model's ability to perceive fine-grained visual cues, thereby equipping with the perceptual foundation necessary for AVR. Although SFT substantially improves the perceptual capability of MLLMs, they still face persistent bottlenecks in AVR, including errors in perceptual strategy (e.g., difficulty in determining whether the appropriate interpretive cue lies in the figure's symmetry axis or in its right- angle structures) and limited reasoning ability on more difficult instances. Therefore, we introduce Reinforcement Learning (RI.,, which leverages the enhanced perceptual capability to guide models toward more reliable perceptual grounding and further strengthen their reasoning ability.
Building upon these components, develop the which unifies enhanced perceptual capability with improved reasoning ability to address the dual chal- lenges of AVR. Experimental results show that PAVR significantly outperforms advanced commercial models on AVR tasks, highlighting a systematic solution to AVR tasks.
Visual Reasoner (PAVR),
Our contributions can be summarized as follows:
(I) We introduce VisuRiddles, which comprises a benchmark derived from real riddles for objectively evaluating MILMs AVR tasks, and synthesizer that automatically generates AVR instances enriched with structured perceptual descriptions.
(2) We leverage synthesized data for SFT to enhance fine- grained perceptual capability for AVR tasks, and further adopt RL to improve reasoning ability and stabilize perceptual strategy selection, resulting in the Perception-Augmente Visual Reasoner.
2

---

Published as a conference paper at ICLR 2026
(3) We conduct extensive experiments to demonstrate key findings, including the reasoning limitations of current MLLMs, the effectiveness of our synthesized riddles, and the effective approach to addressing abstract visual problems.
2
RELATED WORK
2.1 MULTIMODAL BENCHMARKS
High-quality benchmarks are essential for driving progress in MILMs. Earlier evaluation effort predominantly centered on basic visual comprehension tasks, leading to the development of extraction- oriented benchmarks (Mathew et al. 2021 Masry et al. [022 Liu et al. 2024a). These benchmarks primarily assess a model's ability to identily and align explicit visual elements. However, they depend heavily on surface-level cues, offering little assessment of an MLLM's capability for abstraction and logical reasoning. Therefore, caneral-purposee benchmarks (Xu et al. 2023] Bitton et al. 2023 Yang et al. 2024 Yue et al., 2024 2025) are widely focused on. These benchmarks span multiple tasks, such visual understanding and mathematical reasoning, and are designed to assess the generalist capabilities of MLLMs. However, they tend to emphasize knowledge-baid visual logical reasoning. Moreover, due to their strong semantic dominance, MLLMs may directly rely on their inherent knowledge, making it difficult to disentangle and evaluate models' true visual reasoning capabilities. Therefore, some studies have shifted toward evaluating visual logical reasoning, leading to logic-oriented benchmarks such as RAVEN (Zhang et al. 2019), I-RAVEN (Hu ct al. 22021) RAVEN-FAIR (Benny et al., 2021), CVR (Zerroug al. [2022). RPMs (Zhang et al. 2024b). MaRs-VQA (Cao et al.) 2024). MathVerse (Zhang et al. 20024a) MARVEL (Jiang et al. (2024). VisCogBench (Cao
evaluation
(2024), VisuLogic (Xu al. 2025b). Puzull (Song et al. 2025), VisualSphinx (Feng et al. (025) and MV-MATH (Wang et al. 2025a). These benehmarks use structured diagrams, mathematical forms, and spatial layouts to assess analogical reasoning, compositionality, and pattern induction, signaling a move toward higher-level AVR. However, these benchmarks still exhibit notable limitations, particularly in their partial reliance on external knowledge and lack breadth in reasoning coverage.
2024). PuZzleVQA (Chia
2.2 MULTIMODAL LARGE LANGUAGE MODELS FOR VISUAL REASONING
Advanced commercial LLMs, including the GPT (OpenAl 2022). Gemini (Team et al.) 2023), and Claude (Anthropic] 2024) series, have exhibited impressive capabilities in multimodal understanding and generation ( YYn et al. 2024 Yan al. 2023). These advances have spurred systematic research on MLLMS in both academia and industry. Early representative general-purpose MILMMs, such as LLaVA (Liu et al. 2023). Instruct-BLii (Dai et al.) 2023), Qwen-VI (Bai et al., 2023), and VL (Chen et al. 2024b), perform well general multimodal tasks but exhibit limitations in fine-grained visual perception. Subsequently, DeepSeek- VL (Lu et al. 2024). Monkey (Li et al.
2024). TextMonkey (Liu at. 2024b), InternL.-X WWw.MMXXooooosse2242225
(Dong et al. 2024). Qwen2-
VL (Wag et al. 2024a) have introduced high-resolution visual perception mechanisms, establishing more robust foundation for fine-grained visual understanding and reasoning. However, such improvements primarily expand the perceptual scope of MILMS without substantially enhancing their reasoning capabilities. Therefore, inference-time scaling methods including MMVP (Tong
et al. 2024). LLaVA-CoT (Xu et al.) (224). BBA (Zhao et al. (2024). R-CoT (Deng et al.) 2024p. and RedStar (Xu et al., (025aa have introduced and advanced inference-time scaling techniques such as chain-of-thought (CoT), significantly enhancing the visual reasoning capabilities of MLLMs.
More recently, following the advances Kimi K1.5 (Team et al.) 2025) and DeepSeck-RI (Guo) et al. 2025), series of studies, such as R R1-One Vision (Yang et al. 2025), LMM-RI (Peng et al.
2025). MM- MM-EURRKA (Meng et al. 2025), R1-V (Chen et al. 2025). -isua--FFT (Liu et al.) 2025b). PRM (Wang et al.) 2025b), OThink-MR1 (Liu et al. (025a)) and VLM-RI (Shen et al.)
2025), have employed reinforcement learning to develop adaptive reasoning policies and improve
generalization in complex visual reasoning tasks.
3

---

Published as a conference paper at ICLR 2026
Table 1:
The statistics of VisuRiddles, including the number questions, average question length (in tokens), number of associated images, and the corresponding answer format.
Type
Questions Number
Question length
Answer Formats
Sudoku RAVEN
100
48.3
Constraint- based grid
100
70.7
Single-choice Single-choice
Numerical
250
26.3
Stylistic
117
26.3
Single-choice
Attribute
97
27.0
Single-choice
Positional
111
25,5
Single-choice
Spatial
156
27.2
Single-choie
Other
69
30.3
Single-choice
Total
1000
32.5
VISURIDDLES
In this section, introduce VisuRiddles, innovative and comprchensive resource for AVR research. consists of the VisuRiddles Benchmark (Sec.3.II. built from real-world problems to objectively evaluate MILMs' AVR performance, and the VisuRiddles Synthesizer (Sec. 3.2) designed to generate diverse perceptually annotated AVR instances for model training.
3.1 VISURIDDLES BENCHMARK
Unlike RAVEN (hhng et al. 2019) and PuzzleVQA (Chia et al. 2024), which focus on general visual reasoning tasks, we target the most challenging part of AVR tasks. Thus, the VisuRiddles benchmark has been constructed to evaluate MLLMs. It explicitly evaluates models across five key dimensions: a) Numerosity assesses the model's ability to perceive and reason about quantity and distribution; b) Attribute evaluates the understanding of intrinsic visual features that determine structural semantics; c) Style tests the capability to identify and generalize transformation based visual patterns: d) Position reflects the ability reason the relative positions and layout of visual elements; e) Spatiality examines the understanding of three-dimensional structures, shape variations, and spatial transformations abstract graphics. To extend the evaluation beyond low-level perceptual reasoning and address limitations in task diversity, structural complexity, and the inherent uncertainty introduced by single-choice formats, we additionally incorporate two high-level reasoning tasks that require analogical abstract reasoning and consistency- logical reasoning, as represented by RAVEN and Sudoku, respectively. Finally, VisuRiddles also includes a subset, namely Other of AVR tasks involving planar shape composition and character-based semantic patterns. iddues provides a unified benchmark spanning basic to high-level reasoning. enabling comprehensive assessment of AVR, with representative samples shown in Fig. [5] We construct the VisuRiddles benchmark through a structured pipeline comprising Collection, Cleaning, and Consolidation. We provide the representative examples and construction details in Appendix A.2
We initially collected 1,275 AVR problems, each accompanied by expert-level analysis and answer. After careful filtering and quality control, construct VisuRiddles with 800 samples in the basic categories and 200 samples in the high-level categories. Key statistics of VisuRiddles are summarized in Tab. 1 For the basic AVR tasks, each question is formulated single-choice problem, with answer option distributions of (25.75%) ), B (24.75%), C (24.375%) and D (25. 125%). In the high-level categories, RAVEN Reasoning and Sudoku Reasoning each account for 10% of the total data, and require models to generate exact symbolic outputs to be considered correct.
3.2 VISURIDDLES SYNTHESIZER
Recent MILMs demonstrate strong general reasoning, but their limited performance on AVR is largely attributed to insufficient capability of fine-graine fine-grained perception for abstract graphics. Most existing datasets lack intermediate perceptual description, providing only question-anae ansser pairs (Xu Jiang et al. 2024 Song et al. 2025). This prevents explicit modeling of the perception-to- reasoning process, leading to black-box inference, weak inductive capability, and poor generalization.
let al. 20256] Jiang
4

---

Published as a conference paper at ICLR 2026
Rutel: Positional
Rule Pool
Imagees
Images
Riddles
SubRale: Rutaiion
Ioon Gallery
Construction
IDON
Structured Deseriptions
Labeling
.ON
CoT
Solutions
Perceptiaal Description
Stype velaing
tollww
arrow, 1906, (rod)
Riddles Construction
API Labeling
Col Satutiens
Rule Selection
SubRakk
Quastion
Purceptiza Deseripiin
oecortaa rotuttio..
Please sduct
?
mas aprepriac
Cyplent becees
Subitale
-Figan
Rule:: Styliviie
Layont
Sclection
postion
Layout
Ioun
Background
Protefieed boyas Aste-pmerttd Ioeas
Prodefined hamplase
Predicted
CaT Solution
Subitales OR
Aaswer
Rendering
Percegtnal Descriztion
(3) Instance Constraction
API label
fldtttinii Starfied
Del
Parseettr Salectee
Visnn Bentkring
Purcpiteal Docripiine
Narcher
Tigan
CUT Solation:
Scope eneccinds
Optiee
Ground Truth
dader by
(a) Unified Pipeline for Riddles Synthesis
Instantiated Examples
Figure 2: Overview of the VisuRiddles Synthesizer. (a) A unified pipeline for generating abstract graphics with fine-grained perceptual descriptions. (b) Visualization of synthesized riddles based on positional rule and stylistic rule.
More critically, existing commercial models struggle annotate perceptual processes effectively, and manual labeling requires significant expert effort, both of which further exacerbate the issue. To address this, we introduce VisuRiddles Synthesizer, a Riddles Synthesis Framework to provides abstract graphics with aligned fine- grained perceptual description.
Figure 2 illustrates the pipeline of our VisuRiddles Synthesizer for AVR task. The VisuRiddles Synthesizer pipeline comprises two stages: Riddles Construction and API Labeling. The former generates abstract visual instances with aligned perceptual deseriptions, while the latter generates reasoning chains based on these descriptions. Details and representative examples are provided in Appendix B.2 and Appendix B.3
The VisuRiddles Synthesizer generates data that spans five core perceptual reasoning types and two high-level reasoning tasks. The detailed configurations of VisuRiddles Synthesizer are provided in Table The VisuRiddles Synthesizer not only provides riddles' visual instances but also structured perceptual descriptions, thereby enhancing the model's fine-grained perceptual ability for AVR tasks. It is noteworthy that the riddles instances generated by the VisuRiddles Synthesizer focus improving fine-grained perception, rather than reasoning. Therefore, their reasoning difficulty is deliberately kept lower than that of real-world riddles.
PERCEPTION-AUGMENTI VISUAL REASONER
To tackle the challenges of AVR, we introduce the Deccption-Augmentte Visual Reasoner (PAVR). As illustrated Fig. 3 PAVR developed through two-stage training paradigm. In the first stage, SFT equips the model with the ability to capture fine-grained visual cues in abstract graphics. Building upon this, the second stage employs RI. effectively integrate these visual cues into stable reasoning strategies for solving complex tasks, resulting in PAVR. These two stages are complementary: SFT enables the model to see clearly, while RL guides it to reason reliably.
Perceptual Augment via Supervised Fine- Sine-Tuning. The lack of fine-grained perceptual ability remains a major bottleneck for MLLMs on AVR tasks, yet existing real-world datasets typically provide only single-choice answer and rarely provide explicit perceptual annotations to supply sufficient supervision. To address this limitation, we employ the VisuRiddles Synthesizer described in Sec. 3.2 to generate AVR instances across seven categories, each enriched with structured perceptual
5

---

Published as a conference paper at ICLR 2026
Riddle information
Explanation
Question
Analyyiis
[Logical reasoning]
openaton (same=1, deffemett Each me satidfis Col =al XXOR Col
From the four given options, choose the most suitable one the question mark (?).
Row 3 (1 right, top to bottom, filled D):
Options
Cal 1011000111111111110110 112.6.277.11111
Col (10011000000001111111111111
Thenfore.
Col 3 100000000000011110111010 1,10 ic., 6.9
and fillod. Henoe the answer
B
Model Response
Baseline
PAVR-SFT
PAVR
<thimk
<think>
<think>
Grid shapes. hessgoons
squar Starting from the c'occk divection clockwiie
trangular,
Row 1, aquare divided into trongulor,
Row 1: Albermatnng bisck/whste besagons....
Starting from the 12 af o'clock dinectoo clock 2nd, 3ed, 6th, and 8tth are illled...
2nd, 3rd, 6th, and 8th are dilled...
Row 3:
Black hocagons with white trimmpes...
Option Blank hetagon with white trangles, ahterating position..
3. 3: regulor hesogan drvided into 12 mangula regions...
Row Col 1: rogular hestagon divided into
triangullr ingions...
Option
5th, 8th, 10th, 11th tninguler filled
The third row fallows the same pattorn as the secood row but milaled 90 degiees clockiiiee Therofore, the cornact option <ithink>>
Optico A ...haaagn diviled into The 5th, 8th, 9th, 10U, and I1th triangulat are iilled
notion convisunt rale: Col - XNOR
Itpppoars them's consistent skcpping poétion.. Thus, Option B <tthink>
(-2256677.8..11
(1,31. 12).
CLXNOR C2m 6.911 1,100111) comesponds Oplion
</think>
canower> </nnwer>>
cansuer>>
<anower>
Figure 3: Overview of PAVR. (i) Baseline: Incorrect perception leads to incorrect results. (ii) PAVR- SFT, which is trained synthesized data to enhance fine-graine grained perceptual abiliiyy, can accurately understand visual content in riddles but fails in pattern identification. (iii) PAVR, which builds upon PAVR-SFT with reinforcement learning, can effectively recognize patterns and derive the correct answer.
descriptions. These synthesized examples serve training data for SFT, enhancing the model's perceptual ability and enabling to capture fine-grained cues abstract graphics. This establishes a reliable foundation for subsequent reasoning optimization through reinforcement learning.
Reasoning Optimization via Reinforcement Learning. Although SFT on synthesized data improves fine-grained perceptual ability, it still faces limitations AVR, including unstable perceptual strategy selection (e.g., axis vs. angles) and insufficient reasoning on complex tasks. Therefore, we apply Group Relative Policy Optimization (GRPO) (Guo et al. 2025) to endow the model with more reliable perceptual grounding and stronger reasoning ability. The reward design includes: (i) Answer Reward, giving for correct answers and otherwise, and (ii) Format Reward, incentivizing outputs that match the required template: <think> </think> <answer> </answer>. The training data for GRPO likewise generated by the VisuRiddles Synthesizer. This process enables the model to achieve more reliable perceptual grounding and stronger reasoning capabilities.
EXPERIMENT
5.1 EXPERIMENTAL SETUP
We use Qwen2.5-Ve 5-VBBB Bai et al. 2025) as the baseline model develop PAVR. In the supervised fine-tuning stage, the model is trained for 20 epochs using 20K synthesized AVR instances generated by the VisuRiddles Synthesizer. For this stage, we employed the AdamW Optimizer, with a Batch Size of 16, and Learning Rate set at 5e-7. Subsequently, in the GRPO stage, optimization is performed using 4K synthesized instances over 40 epochs. For this stage, the Learning Rate was set to 1e-6, the Rollout Number was 5, and the KI. Loss Coefficient and CL.P Ratio were 0.01 and

---

Published as a conference paper at ICLR 2026
Table Evaluation result on VisuRiddles benchmark. The superseript number indicates the number of answer options (e.g., "4" means a 4-choice question with one correct option; represents vast combinatorial solution space). The best results are marked bold and the second results are underlined.
Model
Param
Num Styl" Attr"
Posit Spat* Sudo"
Rav Other
Human
61.3
60.9
67.5
67.9
58.8
61.9
Open-Source MLLMs
Minicpm-V-2.6
8B
21.6
26.5
23.7
31.5
23.1
0.0
10.0
24.6
20.6
IntemVL2.5-8B
8B
21.2
13.7
30.9
23.4
23.1
0.0
0.0
29.0
18.1
LncmVLL2.588BMMPP
8B
22.0
23.9
27.8
25.2
21.8
0.0
3,0
27.5
19.4
Deepseekv12
27B
21.2
27.4
18.6
18.0
17.3
0.0
15.0
13.0
17.4
InternV1.2.5-388
38B
22.0
19.7
29.9
26.1
29.5
1.0
19.0
30.4
22.3
IntemnV1.2.5338B-MM
38B
26.0
20.5
19.6
27.0
27.6
0.0
18.0
31.9
22.1
Qwen2.5VL-32B
32B
23.2
20.5
30.9
27.0
29.5
0.0
40.0
24.6
24.5
1n157VL2..577B
78B
26.0
25.6
27.8
29.7
23.7
0.0
16.0
27.5
22.7
IntemVL22.5778-MMM
78B
25.6
30.8
24.7
27.9
26.3
0.0
12.0
20.3
22.2
IncmnVL2.5578B-MPOOL
78B
23.2
23.1
24.7
28.8
22.4
0.0
11.0
23.2
20.3
Qwen2.5VI.-2BB
72B
23.6
23.1
19.6
30.2
26.9
0.0
62.0
23.9
25.9
(wen2.5VVI-72(coot)
72B
27.2
26.5
24.7
19.8
28.2
55.0
23.2
26.0
Swen3-VL-235B-Instue
235B
31.2
28.2
36.1
34.2
26.3
29.0
53.0
27.5
32.6
Boon3-VL-235B-Thee
235B
31.2
29.9
44.3
33.3
30.1
33.0
49.0
39.1
34.9
Dots.vim!
671B
30.4
31.6
38.1
38.7
35.3
1.0
20.0
33.3
29.2
Commercial products
GPT-4o
23.2
21.4
29.9
27.9
28.2
0.0
14.0
24.6
21.8
GPT-4o(cot)
22.4
23.9
32.0
29.7
30.1
22.0
29.0
23.7
o3
28.4
40.2
41.2
27.0
21.8
0.0
25.0
33.3
27.0
Blaude-----sonnet
24.0
27.4
27.8
29.7
28.2
4.0
43.0
31.9
26.5
claude-3-7-sonned
24.4
36.8
27.8
29.7
25.6
3.0
36.0
27.5
26.2
claude-3-7
24.0
24.8
33.0
23.4
25.6
3.0
24.0
26.1
23.2
21.2
30.8
27.8
21.6
16.7
17.0
16.0
23.2
21.5
Gemini2.5-pro
31.6
31.6
48.5
26.1
30.1
39.0
30.0
44.9
33.9
GPT-5
30.8
30.8
38.1
32.4
30.8
2.0
29.0
31.9
28.7
Ours
Baseline(Qwen2.5VV-777 PAVR-SFT
7B
24.4
28.2
23.7
22.5
25.0
0.0
48.0
24.6
24.6
7B
31.2
31.6
44.3
31.5
45.5
43.0
61.0
39.1
39.5
PAVR
7B
39.6
39.3
50.5
39.6
51.9
46.0
65.0
55.1
46.8
1.0, respectively. All experiments are conducted on setup comprising 8 NVIDIA A800 80G GPUs. Evaluation settings are summarized in Appendix
5.2 MAIN RESULTS
Table 2
presents performance of various MILMs on VisuRiddles benchmark. We summarize the key observations follows:
Performance of Open-Source MILMs. Most open-source MLLMs demonstrate limited perfor- mance VisuRiddles, with accuracies across core categories often close to random choice and consistently failing on the two high-level reasoning tasks, which reveals a substantial gap between current model capabilities and human performance in AVR.
Performance of Commercial Models with Thinking Mode. Several commercial MLLMs with 'thinking' mode achieve comparatively improved performance VisuRiddles. Their advantage is particularly notable in structured tasks like Sudoku and Raven. These results indicate that AVR can benefit from advanced reasoning capabilities. However, the overall performance remains limited.
Effectiveness of Model Scaling and CoT Prompting. Strategies such as increasing model size and applying CoT prompting, though commonly used to enhance reasoning, ineffective for AVR tasks.
7

---

Published as a conference paper at ICLR 2026
Riddle Information
Explanation
Question:
Analysis
O no
A=
?
[l.ogical Reasoning] grvonn optons, chocar the most
The key insight of this question is Numerical rile.
Common Features of Each Grid:
sultable one to till in the question
Composed of four distinct elements.
+
X
that demonstrates crrtain
2. Contains a pentagram.
polteen or regularity
Thus, only Option A satisfies both conditions.
Answer: A
(a) QVQ- Pre-ire
(b) Qwen2.5 Co
(c) PAVR
Output:
Output:
Output:
<think>
(Step-by-see analysis:
<thin>>
soguemce and and
We number the boxes from left
The question presents four grds and four options: the first grid has four distinct elements (pentagram...) the second ... the third ... the fourth
look Zor the pattecs.
right as 1 to 5. with the 6th box being
the question mark.
We observe pottern: each ynd cuntann four distinct elements, and
Further analysis is needed
four dements.... Box 2... Box J... Box 4...
includes pentagramh...
Conrect Perception Pathern
After careful observation...
Box St.dd
Let look at optins: option contains pentagoe, squam, heragran, a hesagram, which meens...
Envoneous thinkknng
look aprions
EmmER p
</think>
Optionn Option .:... Xoption Co Option D: ..:
Upon re-avamining the images, we observe that each grid consistenlly comprises four distintt elements, including pentagram...
Okay, let's ty to figure pattom.
[ hemeoor,, the 6th box follow a pottemn "sycle" the corret answer es: B.
Let us reconsider the options upton costains four distinct elements and includes a pentagram: Option B.... Option C...; and Option D..:: Thersfom,, bacad on the above analysi, O pption A is correct.
Thorefor, the most likely is option
Emerass proceptionn
<thinkk
Rethinking and Conrection
Canswer>C <lanswer>
canswer B-sanswer>>
canswer er >À <(answer>
Stack in a Flawed
Flawed Percoption and R Reasoning
Correct Reasoning with Re-thinking
Thinking Loop
Figure 4: Case study comparing different reasoning strategies VisuRiddles example. (a) QVQ- 72B reflects a flawed loop under the "thinking" mode. (b) Qwen2.5-VL-72B with CoT prompting exhibits incorrect perceptual understanding. (c) PAVR exhibits accurate perception and coherent reasoning, ultimately arriving at the correct answer.
Models with larger parameter scales do not consistently outperform smaller ones, and CoT yields only limited improvements on select tasks. These findings indicate that scaling in model parameter or inference time is insufficient to address the core challenges of AVR.
Effectiveness of PAVR. PAVR significantly outperforms other MLLMs across all categories of VisuRiddles, especially in high-level reasoning tasks, demonstrating the model's ability handle complex visual patterns. Notably, compared to the baseline, the significant improvement achieved by training on data with perceptual descriptions through SFT highlights the critical role of fine- grained perceptual ability in AVR tasks. On the other hand, the application of RL further enhances performance across various dimensions on VisuRiddles, with the improvements primarily stemming from enhanced reasoning capabilities through strategic guidance.
We also provide the results of PAVR on the VisuLogic (Xu al.. 2025b) benchmark in D experimental results further support our conclusions.
The
5.3 VISUALIZATION RESULTS
To further analyze the reason that MLLMs underperform on AVR task, we examine two widely adopted reasoning enhancement strategies: inference-time scaling and CoT prompting. We conduct experiments on (---72B--Previee (equipped with a "think" mode), Qwen2.5-VLL-72B (with CoT prompting). and PAVR. Fig. visualizes a representative case of broadly observed phenomenon uncovered during our experiments. The results reveal clear differences in the reasoning behaviors across these models. The inference-time scaling approach, which leverages "think" prompting without any perceptual grounding, frequently leads to verbose yet logically inconsistent reasoning, often ending in incorrect answers. The CoT prompting strategy often suffers from perceptual errors, which cascade into flawed reasoning and ultimately lead to incorrect answers. In contrast, by grounding reasoning in fine-gme fine-grained perceptual capability, PAVR correctly interprets the riddle and identifies the underlying pattern, ultimately producing the correct answer. Notably, while PAVR occasionally exhibits erroneous perception, we observe a 'rethink' phenomenon, where the model re-examines and corrects them, realigning the reasoning trajectory reach the correct solution.
8

---

Published as a conference paper at ICLR 2026
These results underscore the importance of perception in AVR. Inference-time scaling and CoT prompting alone cannot compensate for the lack of perceptual grounding, which often leads to incorrect inconsistent reasoning. Conversely, the incorporation of fine-grained perceptual capability enables the model to capture subtle visual cues and sustain reliable performance on AVR tasks.
5.4 ABLATION STUDY
Ablation Study Bottlenecks in AVR. To investigate whether the performance gap in AVR stems from perceptual limitations rather than reasoning ability, we conduct an ablation study using our perception annotated peneeption-amnottted synthetic data. We evaluate two representative MLLMs, GPT-4o (closed-source) and (0wen2.5VL-722B (open-source), on a subset of our synthetic dataset under two input settings: (i) raw abstract graphies (V), and (ii) structured perceptual descriptions (P).
The experimental results in Tab. 3 show that both models perform poorly when given only raw visual inputs. Taking Sudoku reasoning as example. this task requires recognizing digit values and their positions within grid, and inferring a globally consistent solution. Under the Visual setting, both GPT-4o and Qwen2.5VL-72B fail handle such constraint constraint-based reasoning, while in the perceptual descriptions setting, accuracy increases from near-zero to 15% and 65%, respectively. This improvement primarily stems from structured perceptual descriptions compensating for the MILMs' limited perceptual abstraction capabilities. The difficulty stems from the nature of Sudoku images: small digits, dense layout, and absent semantic cues make them hard for MLLMs to interpret, though humans process such structure effortlessly through rapid visual scanning.
Table 3: Impact of Descriptions vs. Visual Inputs in MLLMs. Where "V" denotes inputting abstract graphics directly, while "P" refers to replacing graphics with structured perceptual descriptions.
Model
Num.
Styl.
Attr.
Posit.
Spat.
Sudo.
Rav.
Avg
GPT-4o(V)
35.0
32.0
38.0
36.0
32.0
0.0
20.0
27.6
GPT-4o(P)
(2.0
(v10
(0.00)
(s5.m
100.0 (vea.)
( (e1.00
25.0
60.1
Owen2.5VL(V)
41.0
43.0
50.0
32.0
40.0
0.0
10.0
30.9
Qwen2.5VI.(P)
73.0
83.0(-4am)
80.0
79.0
100.0
65.0
35.0u-25.0)
73.6
Ablation Study on Reasoning Enhancements in AVR. To examine the contribution of reasoning- oriented enhancements, we conduct an ablation study on three components: Caption, CoT, and GRPO, shown in Table 4 We can draw the following conclusions form the experiments: (i) perception serves as the essential foundation, as reasoning enhancements alone contribute only marginal improvements; (ii) relying solely on structured perceptual descriptions for SFT yields limited gains and weak generalization, whereas augmenting SFT with CoT annotations generated by MLLMs alleviates this issue, endowing the model with fine-grained perceptual ability and partial reasoning capability; and (iii) Building on perception -rrunnddd model, the incorporation of GRPO further unifies perception and reasoning, ultimately delivering substantial performance improvements.
Table 4: Ablation study on Reasoning Enhancements. Where Caption denotes training with fine- grained perceptual descriptions, CoT denotes augmenting training with cf reasoning traces that incorporate structured perceptual descriptions generated by MLLMs, and GRPO refers to applying Group Relative Policy Optimization to strength model's reasoning capability.
Model
SFT
RL
Avg
Baseline (Qwen2.5-VL)
X
X
24.6
Baseline Caption
V
X
33.3 (48.7)
Baseline GRPO
X
29.4(+4.8)
Baseline CoT (PAVR-SFT)
39.5(+14.9)
Baseline CoT + GRPO (PAVR)
V
V
46.8 (+22.2)
6 CONCLUSION
In this work, we observe that the primary bottleneck MLLMs in AVR lies their overlooked fine-grained perceptual capability. To address this issue, introduce VisuRiddles, unified resource

---

Published as a conference paper at ICLR 2026
that comprises benchmark for objectively evaluating MLLMs AVR tasks and synthesizer for generating training instances with structured perceptual descriptions. Building on this resource, we propose PAVR. which unifies perception and reasoning through supervised fine-tuning for perceptual enhancement and GRPO-based optimization for reasoning improvement for AVR tasks. Experimental results demonstrate that current MILLMs perform poorly AVR tasks. Moreover, scaling model parameters, CoT prompting, and inference-time scaling fail to effectively address this challenge. These findings highlight that fine-graines perception constitutes the fundamental bottleneck AVR, while reasoning enhancements applied based on perception -rreppidongg model can further improve performance.
In short, our work not only offers systematic resource for both evaluation and training, but also provides a practical framework for addressing the AVR challenge.
Limitation: Due to limitations in time and human labor, the scale of the resources (e.g., rule design, icon library) used for data synthesis is restricted, which somewhat limits the diversity and richness of the synthetized data. In future work, we plan to expand the resource pool to a larger scale with greater complexity, further enriching the synthesized riddles.
ACKNOWLEDGMENT
This work was done during the research internship of Hao Yan, Zhenbiao Cao and Liang Yin at Huawei.
This work was supported by the National Natural Science Foundation of China (NSFC) under Grants No.62576147 and No. 66.62225603.
REFERENCES
Anthropic. Claude 3 models, 2024. htt.l///ss///...aatrooooocc////c Accessed: 2025- 05-06.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 1(2):3, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song. Kai Dang, Peng Wang. Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Yaniv Benny, Niv Pekar, and Lior Wolf. Sccll-loe abstract reasoning. In Proceedings of the IEEE/CVF Conference on Compuder Vision and Pattern Recognition, pp. 12557-12565, 2021.
Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A benehmark for vision-language instruction following inspired by real-world use. arXiv preprint 230i:::20000059955 2023.
Davide Caffagni, Federico Cocchi, Luca Barsellotti, Nicholas Moratelli, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, and Rita Cucchiara. The revolution of multimodal large language models: a survey. arXiv preprint 24X.v:2402.1245/ 2024.
Xu Cao, Yifan Shen, Bolin Lai, Wengian Ye, Yunsheng Ma, Joerg Heintz, Jintai Chen, Meihuan Huang, Jianguo Cao, Aidong Zhang, et al. What is the visual cognition gap between humans and multimodal 1lms? arXiv preprint a arXiv:2406. 10424, 2024.
Liang Chen, Lei Li, Haozhe Zhao, Yifan Song, and Vinci. R1-v: Reinforcing super generalization ability in 2025-02-02.
models with less than $3.
2025. Accessed:
Zhe Chen. Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXtv:2412.0527,, 2024a.
10

---

Published as a conference paper at ICLR 2026
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185 24198, 2024b.
Yew Ken Chia. Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. Puzzlevqa: Diagnosing multimodal reasoning skills of language models with abstract visual patterns. ACI, 2024.
Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi, Instructblip: Towards general- purposs vision- vision-language models with instruction tuning. Advances in neural information processing systems, 36:49250 49267, 2023.
Linger Deng, Yuliang Liu, Bohan Li, Dongliang Luo, Liang Wu, Chengquan Zhang, Pengyuan Lyu, Ziyang Zhang, Gang Zhang, Errui Ding, et al. R-cot: Reverse chain-of thought problem generation for geometric reasoning in large multimodal models. arXiv preprint arXiv:2410. 17885, 2024.
Xiaoyi Dong, Pan Zhang, Yuhang Zang. Yuhang Cao, Bin Wang. Linke Ouyang, Songyang Zhang. Haodong Duan, Wenwei Zhang, Yining Li, al. feemim-xcomposer22 model handling resolutions from 336 pixels to 4k hd. Advances in Neural Information Processing Systems, 37:42566 42592, 2024.
pioncering large vision-language
Yichen Feng. Zhangehen Xu, Fengqing Jiang. Yuetai Li, Bhaskar Ramasubramanian, Luyao Niu, Bill Yuchen Lin, and Radha Poovendran. Visualsphinx: Large-scale synthetic vision logic puzzles for rl. arXiv preprint Nal::2505.23777 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint axXiv2250111944 2025.
Sheng Hu, Yuqing Ma, Xianglong Liu. Yanlu Wei, and Shihao Bai. Stratified rwle-aware network for abstract visual reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 15577-174, 2021.
Yifan Jiang, Jiarui Zhang, Kexuan Sun, Zhivar Sourati, Kian Ahrabian, Kaixin Ma, Filip llievski, and Jay Pujara. Marvel: Multidimensional abstraction and reasoning through visual evaluation and learning. Advances in Neural Information Processing Systems, 37:46567 46592, 2024.
Philip N Johnson-Laird, Mental models and human reasoning. Proceedings of the National Academy of Sciences. 107443):1244-118250 2010.
Zhang Li, Biao Yang, Qiang Liu,, Zhiyin Ma, Shuo Zhang, Jingxu Yang. Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large modtil models. proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26763 26773, 2024.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systemy. 36:34892-3400 34916, 2023.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: the hidden mystery of ocr in large multimodal models. Science China Information Sciences, (1112222010 2024a.
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint arXiv:2403.04473 2024b.
Zhiyuan Liu, Yuting Zhang, Feng Liu, Changwang Zhang, Ying Sun, and Jun Wang. Othink-mrl: Stimulating multimodal generalized reasoning capabilities through dynamic reinforcement learning. arXiv preprint 20X3 2025a.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.0178; 2025b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint 14Xxv::2033055222 2024.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203, 10244, 2022.
11

---

Published as a conference paper at ICLR 2026
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: dataset for vqa document images. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2200 2209,, 2021.
Fanging Meng, Lingxiao Du Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang. Junjun He, al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforccment learning. arXiv preprint (xXiv:2503.07366 2025.
M Mitchell, AB Palmarini, and A Moskvichev. Comparing humans, spt-4, and gpt-4v on abstraction and reasoning tasks. arxiv 2023. arKiv preprint arX.v:2311.09227..
OpenAL. ChatGPT. https://openai:
com/ wtt. wwtp:/og///.tatppp//ooooo/ 2022.
Yingzhe Peng, Gongrui Zhang, Miaosen Zhang, Zhiyuan You, Jie Liu, Qipeng Zhu, Kai Yang. Xingzhong Xu, Xin Geng, and Xu Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint 2rXiv:2503.07536 2025.
Luca M Schulze Buschoff, Elif Akata, Matthias Bethge, and Eric Schulz. Visual cognition in multimodal large language models. Nature Machine Intelligence, pp. 1-11, 2025.
Haozhan Shen, Peng Liu, Jingeheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang. et al. VIm-rl: A stable and generalizable rl-style large vision-language model. arXiv preprint arXiv:2504. 07615, 2025.
Yueqi Song. Tianyue Ou, Yibo Kong, Zecheng Li, Graham Neubig, and Xiang Yue. Visualpuzzles: Decoupling multimodal reasoning evaluation from domain knowledge. arXiv preprint Sidi:22004103322 2025.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 2r 2023.
Kimi Team, Angang Du Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12599 2025.
Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llss. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9568-9578, 2024.
Peijie Wang, Zhong- -Zi Li, Fei Yin, Dekang Ran, and Cheng-Lin Liu. Mv-math: Evaluating multimodal math reasoning in multi nistil contexts. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 19541-199999 19551, 2025a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. Yang Fan, Kai Dang. Mengfei Du Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-lanuuee model's perception of the world any resolution, 2024a. http://sss//aai..orr//////////// hbtrgg/as//2/ 2409. 12191
Weiyun Wang. Zhe Chen, Wenhai Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jiteng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization, arXiv preprint arXiv:2411. 10442, 2024b.
Weiyun Wang, Zhangwei Gao, Lianjie Chen, Zhe Chen, Jinguo Zhu, Xiangyu Zhao, Yangzhou Liu, Yue Cao, Shenglong Ye, Xizhou Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint arXiv:2503. 10291, 2025b.
Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang. al. Deepseck-v12: Mfxture-of- vision-language models for 2024.
advanced multimodal understanding, arXiv preprint arXiv:2412.
Cheng Xu, Xiaofeng Hou, Jiacheng Liu, Chao Li, Tianhao Huang, Xiaozhi Zhu, Mo Niu, Lingyu Sun, Peng Tang, Tongqiao Xu, et al. Mmbench: Benchmarking end-to-end multi-modal dnns and understanding their hardware-soffiee implications. In 2023 IEEE International Symposium on Workload Characterization (IISW), pp. 154-166. IEEE, 2023.
Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step, 2024. URL, htttts:://ariii.or//aa///////// 10440
Haotian Xu, Xing Wu, Weinong Wang. Zhongzhi Li, Da Zheng, Boyuan Chen, Yi Hu, Shijia Kang, Jiaming Ji, Yingying Zhang, al. Redstar: Does scaling long-cot data unlock better slow-rcasoning systems? arXiv preprint arXiv:2011.1 /284, 2025a.
12

---

Published as a conference paper at ICLR 2026
Weiye Xu, Jiahao Wang. Weiyun Wang. Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li. Xiaohua Wang, Xizhou Zhu, et al. Visulogic: benchmark for evaluating visual reasoning in multi-modal large language models. arXiv preprint arXiv:2504. 15277 2025b.
Hao Yan. Yuliang Liu, Lianwen Jin, and Xiang Bai. The development. application, and future of Ilm similar to chatgpt. Journal of Image and Graphics, 28(9):2749-22 2762, 2023.
Yi Yang. Xiaoxuan He, Hongkun Pan, Xiyan Jiang. Yan Deng, Xingtao Yang. Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. RI onevvissio: Advancing generalized multimodal reasoning through cross- modal formalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2025.
Zhen Yang, Jinhao Chen, Zhengxiao Du, Wenmeng Yu, Weihan Wang, Wenyi Hong, Zhihuan Jiang, Bin Xu, and Jie Tang. Mathglm-vistin vision: solving mathematical problems with multi- large language model. arXiv preprint AXiv:2409.133299 2024.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junho Cui, Hongji Zhu, Tianchi Cai, Chi Chen, Haoyu Li, Weilin Zhao, Efficient gpt-Av level multimodal large language model for deployment on edge devices. Nature Communications, 16(1):5500 ):500, 2025.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. survey on multimodal large language models. National Science Review, (112)n 2024.
Ya-Qi Yu, Minghui Liao, Jihao Wu, Yongxin Liao, Xiaoyu Zheng, and Wei Zeng. Texthawk: Exploring efficient fine- groo-graind perception multimodal large language models. arXiv preprint arXiv:2404.09204 2024a.
Ya-Qi Yu, Minghui Liao, Jiwen Zhang, and Jihao Wu. Texthawk2: A large vision-language model excels in bilingual ocr and grounding with 16x fewer tokens. arXiv preprint arXtv:2410.05261, 2024b.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi- disciplisei multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9556-9567, 2024.
Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, el al. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark. In Proceedings the 63rd Annual Meeting the Association for Computational Linguistics ( (Volume Long Pupers), pp. 15134-15186, 2025.
Aimen Zerroug, Mohit Vaishnav, Julien Colin, Sebastian Musslick, and Thomas Serre. A benchmark for compositional visual reasoning. Advances in neural information procexsing systems, 35:29776-299 297888, 2022.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5317-5327, 2019.
Renrui Zhang, Dongzhi Jiang. Yichi Zhang, Haokun Lin, Ziyu Guo. Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal I1m truly see the diagrams in visual math problems? In European Conference Computer Vision, pp. 169-186. Springer, 2024a.
Yizhe Zhang, He Bai, Ruixiang Zhang, Jiatao Gu, Shuangfei Zhai, Josh Susskind, and Navdeep Jaitly, How far are we from intelligent visual deductive reasoning? In COLM, 2024b.
Xueliang Zhao Xinting Huang, Tingchen Fu, Qintong Li, Shansan Gong, Lemao Liu, Wei Bi, and Lingpeng Kong. Bba: mo-al behavioral aligument for reasoning with large vision- language models. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 7255-7279, 2024.
13

---

Published as a conference paper at ICLR 2026
A DETAILS OF VISURIDDLES BENCHMARK
A.1 REPRESENTATIVE EXAMPLES OF VISURIDDLES BENCHMARK
Attribute Reasoning
Positional Reasoning
Stylistic Remsoning
C
glvan options choose
sahable queston mall,, that preseras
Quentton: Among the fnllowing optione, winch one best the gives patten of fiurrs?
appropritt glvcn optione,
cettan putien rrgionty
canale
Answer:
Amsrers
Numerical Reasoning
Sparial Reasoning
Other
+
Quesiin:: From fhz foor sprien. proriaal choose the mos appnepries one the question mark dus censis pallen.
Quesiion: Choos the most oqgropeiose
oution Sioss the lou. given choices to the posilion marked qucrsion mark, dest sequenes lngical
F
F
Questien: Froo the foor givce options choose the moer szitable queation (T)a0 fallows canan patians
H
P
O
T
Amewar:
Amswer:
RAVEN Reasoning
Sudoku Reasoning
Ausmer:
57689111
Sadoloa Each
cell sboul be iitled with sumber from so
sach that every ses, colurnn and eich Jo eahgrd the same color
7:5
approprione grophic option question
contatee all digis from to wilh repeitiinn
4-0 9-3.2.1
Anenver
1324566997
Figure 5: Representative examples from VisuRiddles. The benchmark includes eight reasoning categories, designed comprehensively evaluate diverse reasoning capabilities of MLLMS.
A.2 CONSTRUCTION DETAILS OF VISURIDDLES BENCHMARK
We construct the VisuRiddles benchmark through structured pipeline comprising Collection, Cleaning, and Consolidation:
Collection. We manually collect diverse set of visual reasoning problems from Chinese National Civil Service Examination, which form the five core categories of the benchmark. RAVEN and Sudoku samples generated following the method! and the protocols?.
Cleaning, After initial collection, we recruited twelve trained annotators to verify and refine the dataset. This process involved identifying and removing duplicated questions, correcting answer key errors, and filtering out incomplete or noisy samples to ensure data consistency and reliability. Quality control included annotator training, multi-pass review, dispute resolution, and cross-validation on overlapping samples.
Consolidation. All cleaned samples were manually translated into English, covering question and answer. Each item was then categorized into its corresponding reasoning type based on expert explanations provided by the Fenbi educational platform. All data were standardized into unified ISON format, supporting structured access to image content, metadata, and reasoning annotations.
14

---

Published as a conference paper at ICLR 2026
B
DETAILS OF VISURIDDLES S SYNTHESIZER
B.1 CONFIGURATION THE VISURIDDLES SYNTIEESIZER
Table 5: Configuration details of the VisuRiddles Synthesizer across seven reasoning categories.
Rule
Sub-rule
Layout
Template
Icon
Numerical
Line, Curve, Angle, Cart, Space, Parts
1
5k
Stylistic
AND. OR, XOR, XNOR
Attribute
Element, Group
1
5k
Positional
Translate, Rotate, Flip
4
7
5k
Unfolding. Three- View, 3D-R 3D-Reconstruutton
Spatial
13
5k
Cow-Coosittency. Multiple-Vicet Views
Sudoku
Sudoku
RAVEN
Raven
7
10k
B.2 SYNTHESIS PIPELINE RIDDLES INSTANCES
The synthesis pipeline for generating riddle instances with structured perceptual descriptions consists of two main stages: Instance Construction and API Labeling.
Instance Construction
Instance synthesis is responsible for generating the visual instances and corresponding perceptual descriptions. The detailed steps follows:
Rule Selection:
The synthesis process begins with rule selection. We first extract representative reasoning patterns from real-world standardized tests, such as civil service visual logic exams. These high-level rules are categorized into positional, stylistic, att.rrcclllatrrbuut.spaaaallllla decomposed into finer-grained sub-rules, each representing a distinct visual transformation or logical operation (e.g., "simultaneous rotation and revolution", or "black-white XOR operation"". This decomposition enables modular and interpretable synthesis, allowing each instance to target specific reasoning skill.
and sudoku types, and further
2. Element Configuration:
Once a sub-rule selected, we proceed to define the visual components in structured manner. This includes: (i) choosing layout template (e.g., 3 x 3 matrix, progression row, etc.), (ii) selecting or auto-generating icon elements with specific visual properties (e.g., shape, color, direction), and (ii)) assigning background grid. Elements are configured in rule-consistent way: for example, a "positional" sub-rule may require translation across rows, while a "stylistic" sub-rule may apply logical operations to shape fill. Each configuration ensures the pattern remains visually interpretable and solvable.
3. Rendering and Annotating:
Each question instance is constructed by applying parameterized transformations under defined constraints: single correct answer, consistent application of one rule, and plausible distractors. The resulting sequence of visual instances is rendered into final image grids. In parallel, the framework generates structured perceptual deseription in JSON format, including layout metadata, element- level properties (e.g., object shape, position, state), and reasoning traces. These annotations bridge perception and reasoning, enabling models explicitly learn visual abstraction rather than relying on black-box pattern matching.
APT Labeling
Due to the poor performance of existing commercial models in abstract visual reasoning (AVR) tasks, especially when processing images, these models often suffer from significant hallucination issues and fail to generate correct reasoning annotations. Thus, directly feeding images into the model does not produce accurate results. To overcome this, we introduce a new step: first, we replace the images with corresponding perceptual descriptions and input these deseriptions into a large model. This allows the model to generate CoT based on the perceptual descriptions. The detailed steps are as follows:
15

---

Published as a conference paper at ICLR 2026
1. CoT Generation:
Perceptual descriptions are first extracted from the synthesized visual instances to supply fine- grained visual information. These descriptions are then provided to advanced commercial model (Gemini Flash-Think this work), which generates CoT reasoning process and produces a corresponding answer.
2. Validation:
The generated answer is compared the correct answer of the synthesized riddle. If the model's output does not align with the correct answer, the data point is discarded, ensuring that only valid and accurate samples are included in the final dataset.
B.3 EXAMPLES OF SYNTHESIZED RIDDLES INSTANCES
Figure 6: Examples of synthesized riddle instances in Numerical, Positional, and Attribute
Numerical
Positional
(a) inne
(b) Curve
Translate
?
?
?
O
Analy sist This ridile Cocuses on the number of Amalyiis This riddle locuses the number of lines. The sequence lines
The sequenne of curves the vocrect answer is B, which cuntains curves.
Analysis This sddtle Locuses on Irnnlation trnsformations. In this sequence, the fiest row sacond now remain unchanged. Therefom, the cornet answer C.
the coreett answer C, which cunlains line.
(c) Space
(d) Angle
right,
3
?
?
00 3
(b) Rotate
This nádlle focuses on the number enclosed. reçtons. The sequence enclosed regions
Analysis
A na nysise This nddle focuses on the namber of angles. The sequence angles 2, 1. 6, and 8, womeeci auswer which contains angles..
answer is B, which contains enclosed regious.
(e) Cart
i Part
?
?
Amalysise
This nddil focuses rotation transformation. this sequeace, each black trangular aree.
relates 45* wouleetlookwisee Therefone, the
Analysie
Analysis: This riddle focuses on the number parts Each figure consists of two separate parts, the currct B, which contaiss two separale pats..
right angles. The sequenon right angles is 1, 5. and 7, the nomest answer is which contains right angles.
(c) Flip
ttribute
Element
Group
?
A Analysis
This nádia focuses transfomations
The Listt colums is rolated 90* to ferm the second column, and then flip is appled Thessdurs tird
nódie focused elemnnt attribute. Each of figure contains anis of symmetry, cormect C.
Analysis
on group The elements in both groups need to be symmetical
Analysi:
correct answer C..
16

---

Published as a conference paper at ICLR 2026
Figure 7: Examples of synthesized riddle instances in Spatial, Stylistic, Sudoku, and RAVEN
Spatial
Stylistic
(4) Unnoolding
Gb) AND
Analysks This roddle focuses the Option relatie positions the pattems.
Analysis
This mide focuses un the
AND opensica of pallets where
Three View
(c) Re-oononsttaation
Thersfor, is conrect cnswer
(h) OR
Analysis
This nidle focuses on the OR operatio patems.
This ridéle focuses structurels theee views. Option D iepresents its view.
Aralyske
Aralydde This nádle focuses Option side view match the gaven ones in the problem
Col
Col.
Therefom, is the commot
(d) Viess-onnistenry
(e) Multiple-Viess
answer
3
?
XOR
Analysic
This didiee focuses on XOR operation of poltects whem
Focuses view- consistency, Uption side view aligns the given sequemoe.
Azalysis
This
nddde
Amalyals Views.
multiple
Option
represents different
perspectivess the object given in the problem.
XORC Col.
Therefore, the comeect arswer.
Sudoku
Raven
XNOR
Annilyniez
Anelyses
The goal empty cells so that every now, column, and subgrid contains the digits 1-9 esactly
The task to infer the undectying zules goveming the arangement and select optin. completes the makti..
This nédle foosses the opecation patterss. where
Col. XNOHOO 2 -Col. 3. Therfoo, is the corret
17

---

Published as a conference paper at ICLR 2026
C DETAILS OF EVALUATED MODELS
We evaluate seventeen MLLMs, including cleven open-source models and six advanced proprictary models, as shown in Table Human reference answers obtained from the Fenbi Educational Platform, widely used source for civil service exam preparation. Notably, GPT-o1, Claude-3-7- Sonnet-Thinking, and Gemini-2.5-Pro are equipped with specialized "thinking" modes aimed enhancing multi-step reasoning. To further explore reasoning capabilities, CoT prompting is applied selected for their reasoning capacity,
GPT-4o, Qwen2.5-VL-72B. and Intem model scale, or lack of built-in "thinking" mode. Inference for all models is performed using the vL..M framework with unified decoding configurations. The specific evaluation prompts are detailed as follows; the non-CoT variant is shown in Figures X and the CoT variants are presented in Figures 9 Figures 10 and Figures
Table 6: Overview of evaluated MLLMs.
Model Name
Params
Think
CoT
Provider
Version
Open-Source MLLMs
Qwen2,5-VL Bai al 2025
7B
X
X
Alibaba
Qwen2.5-VL-32B
32B
x
Alibaba
Qwen2.5-VL-72B
x
Alibaba
Qwen2.5-VL-722B(Co]
72B
X
Alibaba
IntemVLL2.58BB Chen et al. 2024a
Chenn
8B
X
Shanghai AI Lab
IntemWVL2.5-MPO033 Wang
2024b)
8B
X
Shanghai AI Lab
Vnte3m\V.22.5-888B
38B
Shanghai AI Lab
Intem-V12.55888BMPP
X
Shanghai AI
IntemmV12..578BB
78B
X
Shanghai AT Lab
L 3PM
78B
X
X
Shanghai Al Lab
IntertVL2..-778--MPOOCC
78B
X
Shanghai AI Lab
DeepSeek-VL2 VL22 Wu (2024
DeepSeek
MiniCPM-V 2.6 Yao (2025
X
ModelBest
Dots.vim1
67
X
X
Rednote hi lab
Commercial Products
GPT-4o
OpenAI
2024.11.20
GPT-4o(CoT)
OpenAI
2024.11.20
o3
OpenAl 2025.04.16
Claude-3.7-Sonect
X
Anthropic
2025,02.19
Claude-3.7-SS
Sonadd-3.--Snnne--Thnneee
Anthropic
2025.02.19
Gemini-2.4 Flash-TTinkkinn
Google DeepMind
2025.01.21
S.iimii-.--Flas--Thinnee
V
Google DeepMind
2025.06.21
2e5ini-2.5-Pro
V
X
Google DeepMind
2025.03.22
GPT-5
OpenAl
2025.08.07
18

---

Published as a conference paper at ICLR 2026
Multiple-choice
Task Definition:
Please select the correct answer based on the question and the provided image, and output the answer.
Note:
The final answer should be provided as follows:
<answer>X</answer The final answer, where X should be one of the answer choices A, B, o.
The final answer must be in <answer> </answer>.
<answer>X<<answer>> The final answer, where should be A, B, C, D.
Sudoku
Task Definition:
Please solve the following Sudoku puzzle. The basic requirement of Sudoku is to fill each empty space with any Arabic numeral from to such that in each row, each column, and each adjacent 3x3 small square of the same color, the numbers filled in must be from to without any repetition.
Note:
The final answer must be provided in the format <answer>X</answerr> where X is the solved Sudoku grid.
The solution should be in the form of a 9x9 grid, where each row contains exactly 9 digits (from 1 to 9) with spaces between them, and rows separated by newline character (An).
The final answer must be enclosed within <answer> </answer>.
Raven
Task Definition:
Please solve the following raven puzzle.
Note:
The final answer must be in <answer> </answer>.
Sanswer>X</answers The final answer, where X should be exactly one of the option labels A-H.
Figure 8: Evaluation prompt for responses to various types riddles.
19

---

Published as a conference paper at ICLR 2026
Prompt for CoT response to multiple-choice riddles
Task Definition:
Please select the correct answer based on the question and the provided image, and output the answer. You must output reasoning process step-by-step.
Follow these steps to solve the problem:
Analyze the Image: Look at the image and identify key details.
2. Summarize the Pattern: Find any patterns relationships in the image.
3. Evaluate the Options: Review the answer choices and see which matches the pattern.
4. Choose the Best Answer: Select the answer that best fits the pattern.
Note:
The final answer must be in <answer> </answer>.
<answer>X<<answer>> The final answer, where X should be A, B, C, or D.
Figure 9: Evaluation Prompt for CoT Response Multiple-choice riddles.
Prompt for CoT response to raven riddles
Task Definition:
Please solve the following raven puzzle. You must output reasoning process step-by-step Follow these steps:
1. Analyze Elements: Look at all the shapes and their characteristics (e.g., size, color, position, quantity).
2. Identify Patterns:
Find any relationships or patterns between the shapes (e.g.. changes in size, position, or color).
3. Form Hypothesis: Based on the pattern, guess what the next shape should be.
4. Derive the Answer: Use your hypothesis find the correct answer, making sure matches the pattern.
Note:
<answer>X</answer> The final answer, where X should be exactly one of the option labels A-H.
the final answer must be <answer> </answer>
Figure 10: Evaluation Prompt for CoT Response Raven riddles.
20

---

Published as a conference paper at ICLR 2026
Prompt for CoT response sukudo riddles
Task Definition:
Please solve the following Sudoku puzzle. The basic requirement of Sudoku is to fill each empty space with any Arabic numeral from to 9 such that in each row, each column, and each adjacent 3x3 small square of the same color, the numbers filled in must be from to 9 without any repetition. You must output reasoning process step-by-step.
Follow these steps:
Look at the given Sudoku puzzle. Identify which cells are already filled with numbers, and which ones are empty. Also, notice the numbers in each row, column, and 3x3 subgrid.
Analyze the Initial Grid:
2. Identify the Constraints: Each row, each column, and each 3x3 subgrid must contain the numbers through 9 without repetition. Based on this rule, identify which numbers are missing from the rows, columns, or subgrids.
3. Start with Obvious Placements:
Look for rows, columns, subgrids where there is only one possible place for a missing number. Fill these cells in first.
4. Use Logical Deduction: If you cannot immediately fill in a number, use logical reasoning to eliminate possibilities. For each empty cell, consider the numbers already present in the same row, column, and 3x3 subgrid, and narrow down the potential candidates for that cell.
5. Ittrate and Fill in More Numbers: As you fill in more numbers, you will uncover more obvious placements. Continue using logical deduction to fill in the grid.
6. Complete the Puzzle: Continue applying these steps until the Sudoku grid fully solved.
Note:
The final answer must be provided the format "<answer>X</answeer>" where is the solved Sudoku grid.
The solution should be in the form of a 9x9 grid, where each row contains exactly 9 digits (from to 9) with no spaces between them, and rows are separated by newline character (* n').
The final answer must be enclosed within <answer> </answer>'
Figure I1: Evaluation Prompt for CoT Response to Sudoku riddles.
21

---

Published as a conference paper at ICLR 2026
D EXPERIMENTAL RESULT ON VISULOGIC
Table 7: Experimental result VisuLogic Benchmark. The best results marked bold and the
second results are underlined.
Models
Param.
Quantity
Spatiality
Position
Attribute
Style
Other
Overall
$2.7
50.0
51.4
MILM Description - -LM
Deepscok-RI
670B
27.7
23.5
24.0
27.8
23.0
35.0
26.6
Qwen2.5.728-Bnn b-rrt-nstrrt
72B
24.4
27.5
26.5
30.8
28.0
Clande-3.7 3.7-emnett (20250219)
26.6
22.5
25.0
28.0
25.6
25.9
Doubao- 1,5 1.5-Pro-2kk (20250115)
30.0
22.5
25.0
25.6
30.0
24.1
26.6
Close MLLMS
GPE-4ommini (20240718)
GPT-4o (20240806)
28.6
24.7
27.2
26.8
20.0
25.9
26.3
Kimi-
Doubao-1 1,5- 3-Vision-Pto-32 (20250115)
28.1
23.8
29.1
25.1
32.1
35.0
28.1
Gemini-2 0-Prr (20250205)
Clande 3.7-eennet (20250219)
22.7
27.3
27.9
28.0
22.2
22.2
24.8
Open Source MLLMs
LLaVA-v15
7B
26.1
24.2
23.5
17.1
31.1
22.2
24.6
Maa (SI)
22.4
27.3
33.1
23.2
25.6
22.2
25.3
Ovic2
8B
26.1
23.8
27.2
28.0
25.6
24.1
25.6
72B
25.2
23.8
27.2
25.6
25.6
34.3
26.2
V12.531...53388
Inter V1...71.
78B
26.6
26.0
26.5
268
31.1
30.6
21.3
Intem VL33388
38B
28.7
27.6
26.1
21.4
23.9
28.5
27.1
VL3-77L
78B
27.7
26.1
31.6
26.3
21.3
32.3
27.7
Qwen2 -VLL-77-Innttoxt-FFT (VisuLogic)
78
24.4
26.4
27.2
23.2
25.6
26.9
25.5
5-1.7---1.....nneeer-- (Vice)
23.2
28.0
Intem SPT oVislLogic)
38B
30.6
29.4
20.6
25.6
30.0
25.0
27.9
lmtmmVL2.5538B. (Viulogic)
Ours
PAVK
78
28.9
30.3
31.6
34.1
30.0
33.3
30.6
E USAGE OF LARGE LANGUAGE MODELS
We thank DeepSeek for assisting in polishing the language and improving the overall clarity of this manuscript.
22


Reviews:
[{"review_id": "XYlv5ewN9W", "reviewer": "Reviewer_emm3", "summary": "This work presents VisuRiddle, a benchmark + data synthesis framework that is designed to improve MLLMs’ capability on Abstract Visual Reasoning (e.g. RAVEN-style IQ Tests and Sodoku). The authors also provide a baseline method, PAVR, for VisuRiddle. PAVR involves a 2-stage training - first using SFT for more accurate perception, then using GRPO for more optimized reasoning. The proposed PVAR baseline model, built on top of Qwen2.5VL-7B, tops the VisuRiddle leaderboard over many larger open source MLLMs as well as common proprietary models.", "questions": "Since the SFT and GRPO stages to tune PVAR involve using synthesized instances created by the VisuRiddle pipeline, I would like to confirm that none of these instances used in training overlap with those in the actual VisuRiddle benchmark during evaluation.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I believe the authors have presented a project in high completeness. This manuscript offers clear descriptions about the motivations, the method designs, and the overall contributions.", "weaknesses": "My only (and perhaps relatively trivial) concern is regarding the generalizability of the new PAVR baseline method. Although PAVR is shown to offer a high bar for the self-made VisuRiddle benchmark, how does PAVR perform on existing AVR benchmarks? So far, the authors have only shown its performance on VisuLogic in Appendix D. But I believe PAVR needs to be further tested on alternative benchmarks regarding its consistency, such as LogicVista and/or VOILA for rigorous logic reasoning, and MathVerse and SeePhys for abstract perception.\n\n**References**\n\n- LogicVista: https://github.com/Yijia-Xiao/LogicVista\n\n- VOILA: https://huggingface.co/datasets/nlylmz/VOILA\n\n- MathVerse: https://github.com/ZrrSkywalker/MathVerse\n\n- SeePhys: https://huggingface.co/datasets/SeePhys/SeePhys", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tD4D4o5nA1", "reviewer": "Reviewer_TJrV", "summary": "This paper investigates the limitations of Multimodal Large Language Models (MLLMs) in Abstract Visual Reasoning (AVR) which are tasks such as Raven’s Progressive Matrices and visual riddles that require structural and relational perception. The authors hypothesize that fine-grained perception, rather than reasoning itself, is the main bottleneck in these models.\nTo address this, they propose VisuRiddles, which consists of \n1) A benchmark of 1000 diverse AVR problems collected and curated from real-world riddles, covering categories such as numerical, positional, stylistic, and spatial reasoning.\n2) A synthesizer, which programmatically generates abstract visual puzzles paired with structured perceptual descriptions and chain-of-thought reasoning traces.\n\nWith VisuRiddles, they rain a model called the Perception-Augmented Visual Reasoner (PAVR) through a two-stage process:\n1) Supervised Fine-Tuning to strengthen fine-grained perceptual understanding using synthetic perceptual annotations.\n2) Reinforcement Learning via Group Relative Policy Optimization (GRPO) to improve reasoning reliability and perceptual grounding.\n\nExperiments on the VisuRiddles benchmark show that PAVR significantly outperforms open-source and commercial MLLMs (including GPT-4o, Gemini 2.5, Claude 3.7, and Qwen2.5VL), achieving 46.8% accuracy compared to ~25–33% for top baselines.", "questions": "1) What measures ensure that the perceptual descriptions generated by the API labeling stage are accurate and consistent? Could noisy annotations harm training?\n2) Is there quantitative evidence that PAVR’s improvements are more from perceptual enhancements than reasoning policy optimization (e.g., via perceptual accuracy metrics)?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper presents rigorous empirical analysis, including comprehensive comparisons with both open-source and commercial MLLMs. Ablation studies are detailed and support the key claims (e.g., perception annotations yield +42.7% improvement, RL adds +7.3% further gains).\n2) The findings have broad implications for MLLM development, suggesting that improving perceptual resolution and grounding may be more impactful than simply enlarging models or adding reasoning traces.", "weaknesses": "1) While the synthesizer is innovative, the paper admits that generated riddles are “deliberately easier” and may lack the richness and noise of real-world abstract reasoning tasks. This could limit transferability.\n2) Although VisuRiddles is comprehensive, stronger evidence of generalization would come from evaluating PAVR on unseen external datasets (e.g., MARVEL, PuzzleVQA, or VisLogic)\n3) The “rethinking” phenomenon observed in PAVR is intriguing but underexplored. Quantitative metrics on perceptual correction or reasoning consistency would strengthen the argument.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nuW9RKnu3V", "reviewer": "Reviewer_FgFP", "summary": "This paper identifies fine-grained visual perception as a key bottleneck limiting the reasoning abilities of multimodal large language models (MLLMs). To study this limitation, the authors introduce VisuRiddles, a benchmark designed to assess Abstract Visual Reasoning (AVR) across multiple dimensions such as numerosity, attributeness, style, position, and spatiality. They also propose a new method, Perception-Augmented Visual Reasoner (PAVR), which applies supervised fine-tuning (SFT) and reinforcement learning (GRPO) to train Qwen2.5-VL-7B on VisuRiddles. Empirical results show that PAVR outperforms several zero-shot MLLMs, suggesting that perception-oriented fine-tuning can partially mitigate AVR limitations.", "questions": "1. How exactly was the data from each source processed and filtered? Could you include pseudo-code or a more detailed benchmark construction pipeline?\n2. How are the train, validation, and test splits organized for PAVR training?\n3. Why was Gemini 2.5-Flash-Think used for generating reasoning chains, and how sensitive is the system to this model choice?\n4. Did you consider evaluating on additional MLLMs (e.g., LLaVA, InstructBLIP) or on supervised/few-shot models?\n5. How do you interpret the below-random results observed in Fig. 1a?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles an important and timely problem - the limited fine-grained perception of MLLMs - which is of clear relevance to the ICLR community.\n2. Introduction of a new benchmark (VisuRiddles) focused specifically on AVR is valuable.\n3. The benchmark covers diverse aspects of perceptual reasoning (numerosity, attributeness, style, position, spatiality).\n4. The paper demonstrates clear limitations of several open and proprietary model families on the benchmark.\n5. The proposed method achieves improved results over zero-shot baselines, showing the benefit of perception-focused training.\n6. The literature review is comprehensive and the writing is generally clear and accessible.", "weaknesses": "1. Limited novelty of the benchmark: The benchmark largely repackages existing datasets (Chinese National Civil Service Examination, RAVEN, and Sudoku) with limited methodological innovation. Details of dataset construction are missing. For instance:\n    1. Were questions from the Civil Service dataset used as-is or modified?\n    2. How were 100 RAVEN questions selected from the 70k instances?\n    3. A pseudo-code description of the “Synthesis Algorithm” (Fig. 2a) would improve reproducibility.\n2. Limited methodological novelty: The approach (SFT + GRPO on Qwen2.5-VL-7B) applies standard fine-tuning methods rather than introducing new learning techniques. The contribution is mainly empirical.\n3. Unfair evaluation setup: The trained PAVR model is evaluated on the same benchmark it was fine-tuned on, whereas competing models are evaluated zero-shot. This limits the fairness and interpretability of the comparison.\n4. Missing training details: The paper does not specify the train/validation/test splits used for fine-tuning PAVR, leaving open the possibility of data leakage or overfitting to the benchmark distribution. The claimed improvements might be in-distribution effects rather than true reasoning gains.\n5. Limited analysis: While Appendix D attempts to evaluate transfer to other tasks, details are insufficient. Stronger evidence is needed to show that perceptual training benefits general reasoning.\n6. The discussion stops short of exploring why models fail across AVR dimensions. A qualitative error analysis would be helpful.\n7. The structured perceptual descriptions and reasoning chains are generated using only Gemini 2.5-Flash-Think. The paper should justify this choice and assess how dependent results are on this specific model.\n8. Missing baselines: The evaluation omits several relevant MLLMs (e.g., LLaVA, InstructBLIP) and non-MLLM baselines such as supervised CNNs or few-shot methods (e.g., Prototypical Networks, SNAIL, MetaBaseline).\n9. Figure 1a shows some models performing below random-guess levels without discussion or explanation.\n\n\n\nMinor comments:\n1. Describing VisuRiddles as “real-world data” is misleading since the benchmark primarily consists of synthetic, 2D, geometry-based tests.\n2. Clarify the distinction between Position and Spatiality dimensions.\n3. Specify whether few-shot evaluation was used for baseline models.\n4. The description of prior benchmark limitations in Section 2.1 (“partial reliance on external knowledge and lack breadth in reasoning coverage”) is too vague—consider providing examples.\n5. Section 2.2 should explicitly highlight how PAVR differs from prior approaches.\n6. The claim “we target the most challenging part of AVR tasks” (Section 3.1) is unsupported—please justify what makes these dimensions the most difficult.\n7. Clarify how “question length” is computed in Table 1 for image-based data, given that tokenization methods differ across models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hhQioH7BeR", "reviewer": "Reviewer_VP3M", "summary": "Current Multimodal Large Language Models (MLLMs) still exhibit significant limitations in Abstract Visual Reasoning (AVR) tasks. To address this issue, this paper introduces the VisuRiddles benchmark. To tackle the dual challenges of perception and reasoning inherent in AVR, the authors develop the Perception-Augmented Visual Reasoner (PAVR), which integrates enhanced perception capabilities with refined reasoning abilities. Through a novel SFT+RL paradigm, the model's fine-grained perception for AVR tasks is substantially improved, and the effectiveness of this approach is demonstrated through rigorous experiments.", "questions": "1. The patterns in VisuRiddles are relatively simple and clean geometric shapes. If these shapes are replaced with more complex or \"noisy\" objects (such as small icons of real-world objects like cats or cars, or complex fractal patterns), will the model fail in the abstract reasoning task due to being disturbed by the semantic content of the complex objects? For a given abstract rule (such as \"the number of elements increases one by one\"), if the elements undergo a simple style change (for example, from black squares to blue triangles), will the model's accuracy be affected? This will test to what extent the model generalizes the abstract rule itself rather than merely memorizing specific visual features.\n2. The paper convincingly demonstrates that fine-grained perception is the key bottleneck. However, Table 3 shows that even when perception descriptions (denoted as \"P\") are provided, powerful models like GPT-4V still perform poorly (with an accuracy rate of approximately 30%). What is the essence of these failures? When perception prompts are provided, do the models fail to understand the prompts, or do they understand the prompts but make fundamental logical errors? If a better perception description is provided, will the models be able to correctly arrive at the answer? If not, does this prove that reasoning ability remains the core limitation?\n3. The SFT+RL mentioned in this article is a training paradigm specifically for the VisuRiddles domain. Has the final PAVR model been evaluated on standard and general multi-modal large model benchmark tests (such as MMBench, etc.)? Will such training weaken the original basic capabilities of the model?\n4. The experiments in the paper were conducted only on the Qwen2.5-VL-7B model. Have you ever attempted to apply this SFT+RL training paradigm to other models with different parameter sizes or to other MLLM architectures?\n5. The paper states that the benefits brought by the Chain-of-Thought (CoT) hinting mechanism to the baseline model are limited. Could you provide the detailed information for this part?\n6. There are many MLLM models that enhance reasoning capabilities. They can also be compared with PAVR on the VisuRiddles benchmark.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The structure of this paper is extremely clear, presenting a definite logical flow.\n2. The paper thoroughly discusses the deficiencies of existing general-purpose MLLMs in Abstract Visual Reasoning and validates this observation by introducing the VisuRiddles Benchmark, a comprehensive testbed covering multiple reasoning dimensions.\n3. It innovatively proposes the VisuRiddles Synthesizer, an ingenious data synthesizer that generates a diverse set of AVR instances with detailed perception annotations. These instances are highly interpretable and reusable, making this a valuable contribution for future research aimed at enhancing the AVR capabilities of MLLMs.\n4. The paper employs a systematic two-stage training paradigm (SFT+RL) that effectively enhances the model's abstract visual reasoning capabilities.\n5. The experimental section comprehensively presents the performance of current open-source and closed-source MLLMs on the VisuRiddles Benchmark, showcasing the remarkable performance achieved by the PAVR model after the two-stage training. Furthermore, well-designed ablation studies are conducted to demonstrate the rationale and contribution of each component in the training process.", "weaknesses": "1. Detailed experimental settings and configurations should be included in the appendix to ensure the reproducibility of the reported results.\n2.  A central claim of the paper is that deficient fine-grained perception is the primary bottleneck for MLLMs on AVR tasks. While the experiment in Table 3 supports this, it also shows that commercial MLLMs achieve only around 30% accuracy on Raven-like tasks even when provided with perception descriptions ('P') as input. This raises a question: is this poor performance due to the perception descriptions still being insufficiently comprehensive, or is it due to the inherent limitations of the models' reasoning capabilities? Further experiments are required to more conclusively disentangle these factors.\n3. The trained model should also be evaluated on a suite of standard MLLM benchmarks. This would verify that the model's original, general-purpose capabilities have not been compromised by the specialized training (i.e., to check for catastrophic forgetting).\n4. The paper states that common reasoning-enhancement strategies, such as increasing model parameter scale or applying Chain-of-Thought (CoT) prompting, yield limited benefits on AVR tasks. However, it does not provide sufficient detail on the specific implementation of the CoT prompts used for evaluating the baseline models.\n5. The experiments are conducted exclusively on the Qwen-VL-7B model. To validate the generalizability of the proposed training paradigm, it should be tested on a wider range of MLLMs with varying architectures and parameter scales.\n6. The paper would be more persuasive，if it could be directly compared with other MLLMs specifically designed to enhance reasoning abilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)