VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
Authors: Weiye Xu
Jiahao Wang
Weiyun Wang
Zhe Chen
Wengang Zhou
Aijun Yang
Lewei Lu
Houqiang Li
Xiaohua Wang
Xizhou Zhu
Wenhai Wang
Jifeng Dai
Jinguo Zhu
Venue: ICLR 2026 Poster (2026)
Keywords: Multi-modal Large Language Models
Benchmark
Visual Reasoning
Primary area: datasets and benchmarks

Abstract: Visual reasoning is a core component of human intelligence and a critical capability for advanced multimodal models. Yet current reasoning evaluations of multimodal large language models (MLLMs) often rely on text descriptions and allow language-based reasoning shortcuts, failing to measure genuine vision-centric reasoning. To address this, we introduce VisuLogic: a benchmark of 1,000 human-verified problems across six  categories (e.g., quantitative shifts, spatial relations, attribute comparisons). These various types of questions can be evaluated  to assess the visual reasoning capabilities of MLLMs from multiple perspectives.  We evaluate leading MLLMs on this benchmark and analyze their results to identify common failure modes. Most models score below 30\% accuracy—only slightly above the 25\% random baseline and far below the 51.4\% achieved by humans—revealing significant gaps in visual reasoning.

Published as a conference paper at ICLR 2026
VISULOGIC: A BENCHMARK FOR EVALUATING VISUAL REASONING IN MULTI-MODAL LARGE LANGUAGE MODELS
Weiye Xul.3.,,Jialll Jiahao Wang2.3+1, Weiyun Wang Zhe Chen Wengang Zhou', Aijun Yang", Lewei Lu", Hougiang Li', Xiaohua Wang", Xizhou Zhu Wenhai Wang", Jifeng Dai Jinguo Zhu? 'University Science and Technology of China, 2X an Jiaotong University, Shanghai Artifcial Intelligence Laboratory, *SenseTime Research, TTinnhua University
ABSTRACT
Visual reasoning is a core component of human intelligence and a critical capability for advanced multimodal models. Yet current reasoning evaluations of multimodal large language models (MLLMs) often rely on text descriptions and allow language- based reasoning shortcuts, failing to measure genuine vision-centric reasoning. To address this, we introduce VisuLogic: a benchmark of 1,000 duman-verificn problems across six categories (e-g., quantitative shifts, spatial relations, attribute comparisons). These various types of questions can be evaluated to assess the visual reasoning capabilities of MLLMs from multiple perspectives. We evaluate leading MLLMs this benchmark and analyze their results to identify common failure modes. Most models score below 30% accuracy-one slightly above the 25% ran- dom baseline and far below the 51.4% achieved by humans- revealing significant gaps in visual reasoning. Furthermore, we provide a supplementary training dataset and reinforcement-leanit baselines are available at htttpss/iisulggi--eeccnnaatk htwp:lgggi...beccmm..iiiubbo.ooo Cattub.oo/Visu...oo Vogic
baseline support further progress. Code, data, and
INTRODUCTION
Remsoning
Ode:
Amriaae
Quartty
Orber
12.0%
Spatialt
Stylidiz Reasmning
Postiornal Resoringg
dbun
100
Alsiuue Reasoning
Figure 1: Composition of the VisuLogic benchmark and performance of representative MLLMs. The left figure shows the distribution of the categories and their subcategories in VisuLogic. The right figure shows accuracies (%) achieved by MLLMS and by human on each category of VisuLogic.
Reasoning, as fundamental component of human intelligence, has become a critical criterion in evaluating progress toward Artificial General Intelligence (AGI) [28] 78). Recent advancements in Large Language Models (LLMs) have demonstrated substamtial improvements in reasoning
capabilities across complex domains such mathematics 64 86 85 61 logical reasoning 722 83

---

Published as a conference paper at ICLR 2026
Question in MMMU quadrialcrall BC-4, CD-13, AD-12,
MLLM Description (GPTT-40)
LLM Solution
shown. What the quadriation!?
A quadrilateral ABCD with right angle at vertex B. The segment BC is perpendicular to AB, and the figure appears to be an irregular quadrilateral extending to point D.
Preserve Sufficient Information
(a) Pipeline "MLLM description -LmiptttonnLLM"" for Question in MMMU 93 It trivial that SOTA MLLMs extract key visual details, thereby enabling the LLM to answer questions solely based on language reasoning.
Question in VisuLogic
Reference Solution
From given options, choose one question
?
B
B
Single symmetric axis with adjacent shapes rotated by 45 degrees.
? DXI
MLLM Description (GPT-40)
LLM Solution
① tog-Jeft corner
forming
A 4x4 grid of squares with upper-right comer square missing, consists of smaller squares
bearkrrboadd third columnn some
areas deviate from the clockerboard patiem.
organized into three full rows and one incomplere
creating irregular gaps.
row, crcating shape resembling an "L"
@ The squares the and
44) figure
grid resembling a cross, with three white squares in the middle column, black squares in the center of the loft and right columns, and white spaces ebswheeee
and remaining
squares are white. The overal effect is
black contrust in the upper right corner.
Lost Symmetric Information
(b) Pipeline of "MLLM description LLLM" for Question in VisuLogic. Even SOTA MLLMs struggle to describe images precisely, leading to ambiguous interpretations.
Figure 2: Comparison of the "MLLM description->l -LLM" pipeline on two benchmarks. In
MMMU, detailed descriptions lead to correct solutions, while in VisuLogic, critical visual cues (e.g.. symmetry, rotation) can be easily lost, causing the LLM misinterpret the image. This highlights that textual reasoning alone is insufficient, underscoring the benchmark's demand for robust and in-depth visual reasoning.
25 50
and coding 37 44 34
Techniques like Chain-of-Thought (CoT)
77 prompting and
test-time compute scaling (e.g., OpenAl ol 36 and Deepseek-RI (20) have significantly enhanced
the reasoning performance of LLMs 20
28 78 Along with the rapid development of language
reasoning research for LLMs, considerable progress [8 64 61 13
53
76
[4 66 77
6, 47 has
been made in improving multimodal reasoning capability of Multimodal Large Language Models (MLLMs).
These methods, which often incorporate reinforcement learning techniques 13 53
64
to enhance
the reasoning capabilities of MLLMs, have achieved some early successes 88 64
61
53
54
666.
However, they typically rely on existing multi-modal benchmarks that struggle to accurately capture [66] assesses "visual reasoning" with referring expression comprehension tasks [92] 58 40). yet these tasks primarily focus on object localization, demanding only basic perceptual skills rather than more advanced visual cognitive processes. Meanwhile, several works [6] 64 adopt mathematical problem-solvine benchmarks that include Stagrams-sucl visual reasoning. In practice, however, as 995 observes, many MLLMs translate these visual clues into textual descriptions and then rely standard language reasoning. This approach can incorrectly attribute language-drivg driven results to visual reasoning, resulting in a misleading assessment of the model's 's visual reasoning capabilities (95 B2] Consequently, designing new benchmarks that explic- itly focus on vision-centric reasoning rather than conflating with text- based reasoning remains critical for advancing MI.LMs² visual reasoning capacities.
a model's core visual reasoning ability. For example, VLM-R1
Math Vista [55 Math Verse (95). and Math Vision 73 to evaluate
2

---

Published as a conference paper at ICLR 2026
Typical Question Our Benchmark
Typical Question in MATH-Vision
Typicul Question in MMMU
From the four options given, choose the most suitable one to fill in the "?" so that it shows certain pattern:
The region bounded by the graph shown above. Choose an integral expression that can be used to find the area of R.
Two rectangles form the angles 40 and 30 respectively, with straight line (see diagram). How big is angle a
(A) abx)]dx (B) 100) lox
009
Typical Question Math Vista
?
(0 - [x)]xxx (D) (D)1/1100> - (c)
What time is shown? Answer by typing time word, not number. It is ( after eight.
800
'0000
00g0
8
C
D
Figure 3: Comparison of questions from different Benchmarks. Compared to MathVista [55] Math Vision [7]. and MMMU (93), VisuLogic focuses more explicitly on assessing pure visual reasoning capabilities.
To address this limitation, we propose VisuLogic, novel benchmark specifically designed to evaluate visual reasoning abilities in multimodal models without mixing them with purely text- based reasoning (see Figure 3p. VisuLogic comprises carefully constructed tasks that span multiple reasoning categories (see Figure T. As shown Figure these tasks are classified into six key types, such as Quantitative Reasoning, which requires understanding and deducing shifts in the quantity of certain elements within an image. In contrast to cxisting benchmarks, as demonstrated in Figure 2] state-of-the-aut (SOTA) MLLMs often omit crucial visual details when describing VisuLogic problems, making it difficult for them to rely solely on a text-based inference shortcut. Indeed, even humans would find it challenging to capture every essential visual cue in single description, so effectively tackling VisuLogic demands more robust, vision-centric reasoning. By reducing reliance on textual inference shorteuts, VisuLogic thus provides stringent evaluation of MLLMs' genuine visual reasoning capabilities.
We conducted comprehensive evaluation and systematic analysis reasoning capabilities. When leading text-only LLMs were supplied with detailed descriptions in place of raw images, their -aarracc--ouuao11.5555
current models' visual
(26.6%), Claude-3.7-Sonme (25.9% and 1reen..572B-Instruue [8]] (28.0%) ( barely exceeded the candom-chame baseline of 24.9%. This clearly demonstrates that textual reasoning alone are insufficient for solving our VisuLogic tasks. Even state-of-the-art multimodal large language models
OpenAl-o3, GPT- 4o 35 2..mini-200-Pro-Exx (68 and Intern 78B 98 achieve only 29.5%, 26.3%, 28.0% and 27.7%, respectively, whereas human participants reached 51 ..4%. The substantial gap between these results and human performance underscores the challenge of robust visual reasoning in current MLLMs. To probe the limits of these models further, ran "hint" experiments in which explicit problem-solving cues are provided. Under such conditions, human accuracy rose to 83.6%, yet MLLMs still failed to surpass 50.0%.
RELATED WORK
Multi-modal Large Language Models. Recent years have witnessed substantial advancements in
Multi-modal Large Language Models (MLLMs). Early works like BLIP 43 42 and Flamingo (5)
introduce lightweight parameters between vision transformer (23 (ViT) and LLMs, laying the groundwork for multimodal perception. Subsequent efforts, such as LLa VA 48 and MiniGPT-4 (9]]
integrate instruction tuning, further enhancing the performance MILMss. Proprietary models like GPT-4o 35 and Gemini-Pro (8] have advanced MILMM performance on complex multimodal tasks, while open-source models such as Qwen- VL series g 74 and InternVL series [17
26. 16 98)
achieve competitive results through optimized architectural design, dataset expansion and training paradigm improvements. Meanwhile, some related studies further advance the ability of large models
by incorporating new modalities (e.g., audio [24 21 81], point clouds 229 [1]. video [96] [4]
and by supporting more tasks (c.g., grounding 84 75,, computer usage 63 7l). Notably, limited
research attempts to enhance the reasoning capabilities of MIL.Ms. Some pioneering works, such as
3

---

Published as a conference paper at ICLR 2026
Image Verification
Category
[RegExj
given options, most
Quantitaive Reasoning
Stracturing
2
Duplicate Removal
Spatial Reasoning
Fesching!
Pusitioaal Reasening
LICENSE O
TXT
Attribute Reesoning
HTML
Cleaning
Manual Inspection
Fasovu
Reasoning
Data Collection
Quality Control
Data Taxonomy
Figure 4: Data curation pipeline of Visuloogie. The pipeline includes Data Collection, Quality
Control and Data Taxonomy.
R1-Onevision LMM-RI
64
MM-EUREKA
RI-V Visual-rft 53 Visualprm 76
OThink-MR1 54
VIM-R1
666 and Open-rl- Video 7 have explored the visual reasoning
capabilities of MLLMs through Reinforcement Learning (RL), but they are still in the nascent stage.
Multimodal Benchmarks. With the development of MLLMs, multimodal benchmarks have also evolved significantly [45). Early benchmarks primarily address visual perception tasks through
simple tasks like visual question answering (VQA) [15 46 38 82), image captioning 662 39)
and referring expression comprehension [2]] Subsequent works expand the capability coverage of benchmarks into more specialized domains; OCRBench [52 Chartqa (59] and Voc [60]
assess textual content extraction; AgentBench 51 and ToolEyes [0] test tool usage capabilities;
and egocentric perception benchmarks [57 19 quantify first-person scene interpretation. Despite
the progress, they ignore the evaluation of visual reasoning abilities 194, (93]. Recently, some bench- marks have made explorations examining MLLMs' visual reasoning abilities, but methodological deficiencies still cause limitations to assess the intrinsic visual reasoning capabilities [32 4 80
InfiMM-Eve [3] test reasoning abilities around daily life, lacking deep-level reasoning scenarios. MMMU [93] and Emma 32 provide benchmarks demanding advanced reasoning abilities in fields such as chemistry and physies, they ignore questions around the images' fundamental visual
components (e-g., shapes, elements). While mathematical benchmarks [73 55
33
65
|95
30 eval-
uate mathematical reasoning with geometric and diagram problems included, they focus on math capabilities but disregard logical analysis about the vision information. Logic Vista [80] provides a multimodal logical reasoning benchmark, its visual questions lack analytical aoith-dominated by single-hop, superficial queries in limited data scope. Unlike previous works, introduce challenging benchmark focused specifically on the domain of visual logical reasoning.
3 VISULOGIC
In this section, we first describe the VisuLogic data-curation pipeline, which comprises three key stages: data collection, quality control, and the detailed taxonomy. We then report the benchmark ss construction statistics, including total size, answer-option distributions, and category- proportions. Finally, we introduce supplementary training dataset- conssstin of questions analogous to those in VisuLogic VisuLogic-designee bolster future research and facilitate community engagement.
3.1 DATA CURATION PIPELINE
Data Collection. We construct the VisuLogic dataset by sourcing all questions from publicly available online resources in compliance with relevant licenses and regulations. As shown in Figure 4 our automated data processing pipeline comprises three stages: 1) Fetching: We employ Playwright to systematically scrape raw web content, supplemented by custom parsing scripts that extract question-ans answer pairs. 2) Cleaning: We remove noise, irrelevant content, and extraneous HTML markup (e.g., <div>) to ensure the integrity of the textual data. 3) Structuring: We standardize the cleaned text and images by structuring all information ISONL format.
4

---

Published as a conference paper at ICLR 2026
Quality Control. To ensure the reliability of the benchmark dataset, employ three-stage data validation procedure: 1) Image Verification: Each image referenced in the questions checked for existence and correct formatting; any item that fails to meet the criteria is removed following human review. 2) Duplicate Removal: We eliminate redundant entries at both the text and image levels by (i) detecting lexical overlap among text segments and (ii) applying perceptual hashing (pHash) to identify visually similar images. 3) Manual Checking: After automated filtering, we perform thorough human-led review of every remaining entry confirm its validity and ensure reliability.
Data Taxonomy. We categorize all collected data into a taxonomy of six primary classes based on expert human annotation of the reasoning skills each question requires. Annotators first tag questions according to the targeted reasoning competency; these annotated tags are then analyzed and merged into five primary categories. A subsequent human review ensures that every question is accurately classified, with any ambiguous instances consolidated under the "Other" category. Specifically, define each category follows. Quantitative Reasoning focuses changes in the number or count of graphical elements (for example, points, lines and angles) and on arithmetic relationships among shapes. Spatial Reasoning requires mentally reconstructing ciree-dimensional shapes from two-dimensional figures, folding or unfolding surfaces, and integrating three-dimensional structures. Positional Reasoning examines transformations such as translation, rotation and reflection of objects while preserving their fundamental elements. Attribute Reasoning involves intrinsic properties of shapes, including symmetry (axial central), curvature and measures of openness or closedness. Stylistic Reasoning entails alterations in stylistic features such as overlay, subtraction and assessments of shape similarity or difference. Other includes questions that fall outside the preceding categories, including those involving letters, alphanumeric symbols or specialized characters.
3.2 BENCHMARK DATASET STATISTICS
The VisuLogic benchmark comprises 1,000 rigorously validated single-choice visual-reasoning ques- tions spanning six categories- Quantittiviee (35.3%), Spatial (23.1 %), Positional (13.6%), Attribute (8.2%), Stylistic (9.0%), and Other (10.8%) with correct answers evenly balanced across options ABCD (23.1%, 26.7%, 25. 2.,2%, 25.0%).
3.3 SUPPLEMENTARY TRAINING DATASET
To facilitate further investigation of visual reasoning, we provide an auxiliary training set of 4,296 question-anae answer pairs drawn from the same domains and subjected identical validation procedures to prevent overlap with the benchmark. The training split mirrors the primary taxonomy, with category proportions of Quantitative Reasoning (30.7%), Spatial Reasoning (25.5%), Positional Reasoning (13.0%), Attribute Reasoning (8.8%), Stylistic Reasoning (9.9%), and Other (12.1%).
EXPERIMENTS
In this section, we present a comprehensive evaluation of the VisuLogic benchmark. We first describe the experimental setup in Section 4.1 followed by overall performance results in Section 4.2 We
then analyze systematic errors in Section 4.3 and provide qualitative insights in Section 4.4
4.1 EXPERIMENT SETUP
To fully investigate models' performance, we establish two reference points: 1) Human Performance: We invited 100 graduate students majoring in science and engineer- ing to solve 10 randomly sampled VosuLogie questions each, allowing 2-5 minutes per question. The aggregate accuracy over all participants constitutes the human benchmark. 2) Random Selection: We simulate random guessing by sampling answers uniformly over 10 independent runs and report the average accuracy the random baseline.
References Performance.
Evaluated Models. We evaluate total of 31 models VisuLogic, comprising large language models (LLMs) and 23 multimodal large language models (MLLMs). Appendix provides more details.
5

---

Published as a conference paper at ICLR 2026
Quantitative Reasoning
Spatial Reasoning
Positional Reasoning
Choose the most suitable option to fill question mark, pattern is presented.
On the left is net the surface of cube. Which option the right can be formed by folding it?
The option that matches the given pattern
?
Attribute Reasoning
Stylistic Reasoning
Other
Select the most appropriate one to a fill in the question mark to maintain certain pattern.
Choose the most suitable option to fill the question mark, so that it presents certain regularity.
Choose the most suitable one to fill in the question mark, so that il presents certain regularity.
?
?
?
THIS
HELLO
HAIR
FAIR
BOOK
EYE
BYE
D
Figure 5: Question examples of different categories our VisuLogie Benchmark. VisuLogic contains 6 categories of questions, which require models' abilities in visual logic reasoning.
?
S
A A.1.2.C.B
2
Question:
Question:
Question:
From the given options, select the most suitable one to fill the question mark, so that certain
Among the following options, which one conforats
Divide the following six figures anto two eategorios, ensuring each category figures
change given
petien presented:
figure
conution charactes or puttems.
Hint:
Hlin::
Hint:
To solve this pantom innsfoomttinn puzzle, observe how the lines each figure change, In rotating clackoise incressing mumber, slaming to the right are alsu robsing cluckwise but decrensing number Applyy this pottern figure focusing on the number and oriemnation the lines to determine the corract configaration.
Consider how the elements of each figure are the
Observe the given figures and pay attention to their symmetry properties. The figures in symmetrical figures, their symmetry types are diffeerent Try figures into two categuries: controsymment and axisymmetry help stentiff the regularity figures. Throug con find the atriburion of each figure and get the connect answer.
cumposed, Observe that the
step,
thind figure combination too figures. Specifically, bottom half of the third figure ideniccal to the bestom half the first figure, and the top half top half
lines
cnnsition from
Apply this pattern the suqueooe on right. Identir the parts of figinss en the right first.
Figure 6: Hint prompts visualization. Hint prompts examples, which supply solution guidance for MLLMs, are shown in the image, with solution-critical elements highlighted in red.
LLM Evaluation Protocol. For language models, we generate an image description using GPT-4o and prepend it to the question. Specifically, each question is formatted as "Following a detailed caption describing an image: [DESCRIPTION] Based on the provided description, select the best answer from the four options.". ". This combined prompt is fed directly into LLMs for inference.
Prompts Setting. We apply three distinct prompting paradigms to investigate model reasoning capabilities:
1) Non-CoT prompt evaluation:
"Answer the question using a single word phrase, following this format: Answer: Vboxed/ SLETTER]"
Models receive a concise instruction:
2) CoT prompt evaluation: We prompt models to articulate intermediate reasoning steps: "Solve the complex visual logical reasoning problem through step-by-step reasoning. Think about the reasoning process first and answer the question following this format: Answer: Boxed(SLETTER:
3) Hint prompts evaluation: Leveraging GPT-4o, generate question-specific hints derived from the reference solutions. Solution -elutie hints are provided alongside the CoT prompt to guide reasoning without revealing the final answer directly.

---

Published as a conference paper at ICLR 2026
Problem
soed skenify GPT-4p
From the four options givn, choose most appoprate une to fill quodioo mork so
"o deterning the crreet
the sequetce of images Upen axalyang soparior, cach figure shows progresion where Madk and white dos clerass postions is specifif and regalar way.
Saloeee cortain regulbrity
Obsuraation::
La cach skas dac back doa skifl posiicea, crasitg spstetuiic patere. The wher back dore alumree in way fur sprars unitorniy wnose the gral By anilysing lae prograsice sequenoe of grds Option the coroed chosce sigga with Di stozree rogabiiit
V..55 12B-Instruet
X
fhie grd bask circko wistz
carlex
Reference Solution
that the nember back ctrlls decroess by the namber of circke nemsoo by in sabooqune ani Tharefore, the fou gnd doold lave 7 Mack circlo and wiite cirdle..
we
priedit The figans
Ih elerscro ard the same, so the posiize putten is grven gueetce all have squres, and
New, ler check the optine
nomber of book Macis the widd same Sa lenrr cate ciecko are comilarred separeds sunoitioo. posttion
Ar this gri 7 back orsiox and 9 whize cadde.
This grld has 6 Madk clrcke and white ctrcles.
C: lhi and hes
circko whsts circlo.
sencrrrre (erie) cach abe the pastitonn the thick blddes alzmrating on the left and right sides, and the cetemost black Neck moros
aquaros cledosive (coariecledtvo
block Hooos
Thre, the appograat
and that matches
wlach back cirdlr walk fill the quouior mark Ilrod)))).
Blaude-37-Sonme
sqgare clodovise ouch line. Only option Cia conrei
LL2.5 38B X
need is ideasify the pattans to detemiis option snuald replaze the quoiios viriiss undlyzz the fianciio
uppeurs
black cag cigezaFaa
fioed
griauily
both
goll
docresingg wlich aillows
opiinn Bee last
addninaal black dem seem roser ormindeedbee anord rigie- the sack doa in the oppeor neate chokkiteee Eadi Mack de mises are paslion clokwwise oach axceesing fianeiiaaesiing this pateeria Itll fellow tae cnablikee sctasicn ralo fiom the tis irame so prodict the odh frime oqaciiass marktes to- The inner black cata
the frod centery trane, thene black dees
Let's
poutinn
"The number of black cirdex optices A and mare
then is the pervkoas graphs which is sbviirayyy mocesitzat
with the matern.
sheeld mome one pesiiion for ber The orer black doss dhvsl move are position clodlostel Prariiiig optices and D cacially apainss these nioo, cas sos optioe continnes cotatio pazerra correctly portices frome
There we more black cácies is option C Jazn befiee,
which is ircencitere with the presizes parurn
chrosco-
locror portiicns frome Parskmawer (broaddCC
asstber circlo is optios comistent will
Thercboo,
the sahalee oplion.
Image Deseription (GPT-40)
03-mini
Sinattrre the Puole:
"The top mos concints of fho grids, each with fon
1.*st Otserve the Parsen
parnla, lefs eup-ityddep Fox Grllst
sppeaach
eopaesoe gride
srongemeen of baak and whire cirrles
cioecly cramine black and while cindis each gnd, payinz steation how the Nack circlas evalre
gnd miking crpprermtd a "?'s
optir
The goil deieniie pallens leloct coreect
The fivst line gris fellww a specifie patten, whik the Gilk
crnwer chelos (A, D) shows Soqaeaee
2. ""idriii Powitle Translenatiees""
ooross each gritt
mino skilling), o besgggennnnng
circles proditable mannees Cuoide clarges like. Moving Gidd",, Adding inunsforaaionn "Poddel TTink about the posiliuss number of circes, grid..
The inneverraton cruld he haurd the momemmr of blek cincree change in the arongoment p. memoon
Fook the numher of Maak rirkes, noir posiiine, and any shifs or tra mnneemmmttio from one grld to the rest
waste cirdes.
The amargrmen of blok clecke come to fellow trazs besstiin fiots one and to the sod.
Flach
supprnce continn mik back
shool
from the firs giól, lepoliasioo whut
or
fith shodd
laaod the
By cnulocing the fistt for grdb, sale patten proemning change cun skemitied.
"CCoppor the Assuer Choioee**
Oncs you lave a hypolaao Su sissin arid, compare is aquinss the foxe staver dasico (A, B. C, DI.
Anseet Choios (A, Dj:
coretally srultpre the firs four grids and intented clear parter in movement and smmagement of the Nack cirller
Classs dae oue dist auticees the prodistel forsointaion Baaed os this reawesing approcch, lnt's mounse that you'
secuaner fellowe the ahorvel the grids
d
Amseer: (Bo.
h
Figure 7: Solution examples generated by different models. Reference solution and outputs generated by GPT-4o 35), (ween2.5VL-72B-Insstee (9) VnternV (18), and Claude-3.7- Sonnet. Additionally, the image description and solution from LLMS are also illustrated.
Notably, unless otherwise specified, CoT prompt evaluation is employed by default for assessing model performance. And hint prompt cases can be found in Figure 6
4.2 OVERALL RESULTS
LLM Performance. Table Ireporss that all evaluated LLMs attain rather low accuracy on VisuLogic.
The best-performing LLM, 1iven2.5-72B-Instruue reaches only 28.0%, while GPT-4 and Deepseek- RI achieve 23.6% and 26.6%, respectively. These findings underscore that reasoning based solely on textual deseriptions is insufficient to capture the rich visual information required by our benchmark, causing failures to resolve visual logical reasoning problems.
MILM Performance. As shown in Table [ current multimodal LLMs also perform poorly on VisulLogic. The highest score is 29.5% by OpenAl-o3, which remains substantial 21.9 points below human performance. Advanced models such as GPT-4o and Gemini-2.0- and 28.0%, respectively, revealing marked gap between existing MLLMs and human-level visual reasoning. Overall, these results indicate that current MLLMS have serious deficiencies in visual reasoning and that significant advances are still required.
attain only 26.3%
Effectiveness of CoT Prompts. Contrary to expectations, chain-of daain-of-thon tf-thot (CoT) prompting yields minimal improvements in visual reasoning. As detailed in Table 2 GPT-4o-mini benefits most, with only a 1. 1.2-point gain under CoT compared to direct-answer prompts; all other models exhibit gains below 1.0 point. We speculate that this limited effect likely stems from current CoT training being
7

---

Published as a conference paper at ICLR 2026
Table 1: Cross-Modal performance with CoT prompts on VisuLogic. The table shows the evaluation scores of baseline references. LLMs, and MLLMs. Top performers per category are bolded, second place ones underlined. The first column shows the model name. The second shows the total score of the Hint prompt with details in Table The third shows the total score with the CoT prompt. And the remaining columns show the category-wiss scores with the CoT prompt.
Models
Hint
Overall
Quantity
Spatiality
Position
Attribute
Style
Other
References
Human Random
83.6
51.4
45.3
52.7
71.1
50.0
47.5
44,2
25.4
Open Source LLM (MLLM Description +LLM)
Deepeekk-B RI
20
26.6
27.7
23.5
24.0
27.8
23.0
35.0
Qwen2.5-7228 128-tssuuut QwQ-32B 71
28.0
30.2
24.4
27.5
26.5
30.8
22.8
24.5
20.1
25.1
19.0
20.7
24.0
Description
GPT-4 (20240613) IT
23.6
21.2
22.5
21.3
25.6
23.3
35.2
(20250131)
18.8
28.4
Gemini-2.00 1a..0-FFass--iee Flush-Thinking (20250121) 68
23.4
23.2
26.0
16.9
17.1
21.1
33.3
Claude-3.7-Sonne (20250219)
22.5
30.6
Domhan- 1,5 P.5- (20250115)
26.6
30.0
22.5
25.0
25.6
30.0
24.1
Close Source MILMSs
OpenAL-o3 (20250417)
GPT-4o-mini (20240718)
27.3
24.3
27.2
23,4
23.5
18.3
31.1
16.7
(20240806)
Kimi-Intest2022S0M
27.8
25.9
24.9
29.4
26.5
28.0
16.7
26.9
Gemini-22 Somiini-2.000 2.0.P (20250205)
08
36.5
28.0
29.7
24.2
27.9
30.5
22.2
33.3
Claude (20250219)
Open Source MLLMs
LLVVA.w1.5-BB 4v
25.3
24.6
26.1
24.2
23.5
17.1
31.1
22.2
MaVA-OneVisioon--
41
26.8
25.3
22.4
27.3
A3.1
23.2
25.6
22.2
ShareGPT4V 12
26.8
23.4
24.9
22.1
23.5
19.5
28.9
19.4
MiniCPM-n-2.6 69
28.8
25.3
25.6
23.0
27.3
21.9
24.5
29.9
GLM-4v-9B 27
29.1
24.3
22.4
23.7
28.3
26.0
24.1
25.3
Ovis2-8B
27.7
25.6
26.1
23,8
27.2
28.0
25.6
24.1
mPLUGGOOWIB-7B-22
25.6
21.5
15.2
20.7
20.4
Skywork-R1V3-388
67
31.2
27.9
26.5
29,6
24.6
21.2
26.6
39,3
Emiee445.TTuro-VV
10
31,0
27.1
27.8
25.0
24.5
35,7
30.0
24.4
Owen2.5.VL-7B-Icstte
30.1
26.0
27.5
20.9
25.2
23.2
37.8
25.0
Qwen2.5VVL-72B-Imm Instrrcct
32.2
26.2
25.2
23,8
27.2
25.6
25.6
34.3
Q:Q-72B-Previow
10
29.8
23.0
24.2
17.0
24.1
21.0
24.4
30.6
IntemmL.....38B [6
33.1
25.5
24.4
26.4
27.2
23.2
25.6
26.9
InternLL2..-78B
16
30.7
27.3
26.5
26.0
26.5
26.8
31.1
30.6
IntemVLL3.388
33.2
27.1
28.7
27.6
26.1
21.4
23.9
28.5
InterVLL3-78B
33.6
27.7
27.7
26.1
31.6
26.3
21.3
32.3
based only pure-text corpora; future works should explore CoT techniques tailored multimodal data to better support visual reasoning tasks.
Effectiveness of Hint Prompts. Table shows that hint prompts can boost model performance-Clandee 7-Sonetet, Gemini-2.0- 2-rroo.. and Doubao-1.5- 1.5-Vision-Pro--22 all improve by over 8 points, reaching accuracies above 35%. However, even with explicit guidance, models still fail to construct coherent, reliable reasoning chains. This suggests that simply augmenting training data with similar tasks insufficient (which can help MLLMs come up with specific directions for solving the problem); future efforts must focus on enhancing the reliability and correctness of reasoning procedures of MLLMs to achieve more accurate reasoning inference. The complete results and analysis for all MILMs can be found in Appendix C
we observe a positive correlation between parameter size and model performance. With in the same model series, Qwen2.5-VL-- outperforming Qwen2.5VL-77 -B--V----B--ntcct (26.0%) (%) by 0.2%. Furthermore, IntermVL2.5-788. (27.3%)
Impact of Model Scaling. In Table I
achieves 26.2 %
surpasses
surpasses Intern VL2.5-38B (25.5%) by margin of 1.8%.
Open-Source vs Close-Source. Table further compares open- and closed-source models. The top open-source MILM,, Intern VI.3-888. attains 27.7%, trailing the closed-source leader (OpenAl-o3,
8

---

Published as a conference paper at ICLR 2026
Table 2: Influence of Chain-of-Thoughh on model performance. Positive value changes are highlighted in red, negative changes in green, and statistically insignificant variations (delta 1) are denoted in gray. With CoT prompts, MLLMs only exhibit tiny improvements in visual reasoning.
Models
CoT
Overall
Quantity
Spatiality
Posttion
Attribute
Style
Other
4o (20240806)
24.7
25.9
26.0, 0.3)
26.9; 11.7)
24.2( 0.53
26.5, 26-5;-0.77 0.73
23. 2
24.0
29. 29.0
Kini lasest
X
25.9
25.1 (--8(-
22.9 (2.00
24,9
22-5 -6..
29.4
26.5
25.0,- 1.50
19.5;
167
55.00-18.00
26.9
24-1(-2.0)
GPT 4o mini (20240718)
x
24.3
23-1 (-19))
27.2
-3.4)
23.4
(-0.5)
23,5
24.3
17:1
18.3
$1.1
90-00-1.11
16.7
(8.5
Qwend. S-VI-lmstme--78
27.6
20.9
25.2
25.0
25.9 (00.3
20.3
-0.3)-2.3)
(2.8
26.4,41.23
25.3 25.3(-2.1)
20.6 20.6
(4:33))
24.1
26.4
27.2
25.6
22.2
25.0 (10.)
(4.6)
25.5(-00) 00.9)
22.1
22.0 1.20
26.7( 26.70-1.1)
29.6(47.4)
Table 3: Influence of hint prompts on model performance. MLLMs exhibit measurable perfor- mance enhancements with hint integration, yet retain significant gaps against human performance. In comparison, humans achieve task mastery on VisuLogic with hints. Value changes are color-coded with red indicating positive shifts and green denoting negative variations.
Mineb
Hint
Quaatity
Spatialit
Postiin
Attribute
Slyle
Other
Haras
$1.4
45.3
$2.7
71.1
NL.0
41.
44.2
63.0
$0.1
68.0
100.0
0G.7
78.6: 1)
00,5;
GPT-4o (20040800)
25.3
28.5
24.7
10.2
26.8
20.0
25.9
30
25.
25-4
28.3 28.2:42.0)
26.00 26.0
30
42 12.9:+17700
Clande 7-5ome (20750019)
24.8
22.7
27.3
27.9
28.0
22.3
22.2
SS. (18.8)
$35.3
23.5
10. 10.4)-6..0
28. 3 2:435.000
Gemmi-2 8-hhu (2025)000))
78.0
96.
79.7
44.8
98.3
22.9
26.0
88. (447
20.3
16.
42 12.9
Desbas-1 1.5 Viicon (20290115)
28.8
37. (19.)-00))
46 16
25. 0.
54:2
54:2
33.3
23. 3
28 28:66-000
29. 99.5% by only 1.8% points and outperforming other proprictary competitors such as GPT-4o and Slaude-3.--oomme Overall, both open- and closed-source models exhibit uniformly low performance, highlighting widespread neglect of visual reasoning objectives in current multimodal model training and data collection.
4.3 FINE-GRAINED COMPARISON
We systematically analyze model capabilities by examining error distributions across reasoning categories for different models. Figure 8 presents the eтrоr rates of LLMs, MLLMs, and human participants over six distinct reasoning categories.
Figure Sa reveals that LLMs struggle most with Spatial Reasoning questions, indicating that text- only deseriptions are insufficient infer tnree-dimensiona structures or spatial transformations. In contrast, their performance on Quantitative Reasoning tasks is comparatively stronger, suggesting that quantitative relationships are more readily conveyed through language.
As shown in Figure 8b. Stylistic Reasoning presents the greatest difficulty for MILMs, with error rates exceeding 75% worse than random guessing (25% accuracy). This result underscores a fundamental limitation of current MLLM architectures in capturing subtle visual cues such overlays, contours, and shape variations.
Figure 8c reveals that human error patterns form distinct cluster, separate from LLMs and MLLMs. Human participants maintain error rates below 30% on Positional Reasoning tasks, reflecting robust position-basee based visual inference. In contrast, LLMs and MLLMs face significant challenges with posi- tional reasoning, underscoring a fundamental divergence between human and model visual-cognitive processes and revealing limitations in how these models interpret positional information.

---

Published as a conference paper at ICLR 2026
70
75
10
85
70
40
10
09.
08.
s0y
Sty.
AB
Art
Spa.
Qu0
63-mini
Qwen2.5-728 Clando-3.7SSonect
Kimi-tatest
Gemini-220-Pro
GPT-4o
Human
(a) LLMs' emor distribution.
(b) MLLMs emor distribution.
(c) Humans' distribution.
Figure 8: Error distribution analysis. The figure demonstrates distinct error type allocations across Humans, LLMs and MLLMs, revealing differences among their cognition patterns.
4.4 QUALITATIVE ANALYSIS
LLM Failures. As shown in Figure 7b), LLMs that rely on externally generated image captions often omit critical visual details required for multi-step logical deduction- -uch as the counts, shapes, and progression patterns of the black and white dots in Figure Za). As result, their reasoning deviates from the correct solution, often producing hallucinations irrelevant responses.
MLLM Failures. Figure also presents cases in which MILMs correctly describe static visual content yet fail infer the evolving relationships among shapes, instead resorting superficial cues like object counts. While these models can recognize individual shapes and tally items, they struggle reason over inter-element relations, which limits their ability solve visual-logic problems. Besides, we provide failure mode analysis of each categories in VisuLogic in Appendix C.5
5 CONCLUSION
In this paper, we present VisuLogic, novel benchmark designed to evaluate the visual reasoning capabilities of Multi-modal Large Language Models (MLLMs). The benchmark consists of 1,000 vision-centric reasoning tasks distributed across six distinct categories. We conduct comprehensive evaluation several advanced I.LMs and MI.LMS on this benchmark and provide an in-depth analysis of their performance. Our findings reveal that even the most advanced models fall short of human performance, highlighting substantial opportunities for advancement in visual logical reasoning. To promote further research and innovation, we'll open-source the evaluation code and datasets associated with this work. We hope this work serves as an important research in visual reasoning and contributes the broader progress of MLLMs.
Reproducibility Statement. We provide detailed descriptions of dataset construction, preprocessing steps and quality controlling methods in Section 3 and Appendix B Experimental setups, model configurations, and hyperparameters are reported in Section and Appendix C An anonymous link download our dataset is included in the supplementary materials.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, 1. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[2] W. U. Ahmad, S. Narenthiran, S. Majumdar, A. Ficek, S. Jain, J. Huang, V. Noroozi, and B. Ginsburg. Opencoderraasonnng: Advancing data distillation for competitive coding. arXiv preprint arXiv:2504.01194, 2025.
[3] A. Ahmadian, C. Cremer, M. Gallé, M. Fadace, J. Kreutzer, O. Pietquin, A. Üstün, and S. Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in Ilms, 2024.
[4] N. Akter, Lee, Chang, Y. Bisk, and E. Nyberg, Visreas: Complex visual reasoning with unanswerable questions, 2024.
[5] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 0-23776-237333 2022.
[6] R. An, S. Yang, M. Lu, R. Zhang, K. Zeng, Y. Luo, J. Cao, H. Liang, Y. Chen, Q. She, et al. Mc-llava: Multi-concept personalized vision-language model. arXiv preprint arXiv:2411.11700 2024.
[7] H. Bai, Y. Zhou, E. Li, Levine, and A. Kumar. Digi-q: Learning q- q-value functions for training device-control agents, arXiv preprint arXiv:2502.1511 2025.
[8] J. Bai, S. Bai, S. Yang, S. Wang. S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint ar0iv:2308. /2966, 2023.
[9] S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Cheng, H. Zhang, Yang, H. Xu, and Lin. Qwen2.5-vl technical report. arXiv preprint 25X2::2502.13923,, 2025.
[10] Baidu-ERNIE- Team. Ernie 4.5 technical report, 2025.
[11] G. Chen, M. Wang, Y. Yang, K. Yu, L. Yuan, and Y. Yue. Pointgpt: Auto-regressively generative pre-training from point clouds. Advances in Neural Information Processing Systems, 36:29667- 29679, 2023.
[12] L. Chen, J. Li, X. Dong, P. Zhang, C. He, Wang, F. Zhao, and D. Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023.
[13] L. Chen, L. Li, H. Zhao, Y. Song, and Vinci. R1-v: Reinforcing super generalization ability in vision-language models with less than $3. 2025. Accessed: 2025-02-02.
[14] S. Chen, H. Li. Q. Wang, Z. Zhao, M. Sun, X. Zhu, J. Vast: A vision-audio-subiile--ee omni-modality foundation model and dataset. Advances in Neural Information Processing Systems, 11772242-728666 2023.
[15] X.
Chen, H. Fang. T.-Y. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015.
[16] Z. Chen, W. Wang. Y. Cao, Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time 2024.
scaling. arXiv preprint arXiv:2412.00527
[17] Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu. J. Luo, Z. Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. /682/ 2024.
11

---

Published as a conference paper at ICLR 2026
[18] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguiss tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185-24198, 2024.
[19] S. Cheng, Z. Guo, J. Wu, K. Fang, P. Li, H. Liu, and Y. Liu. Egothink: Evaluating first-person perspective thinking capability of rision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14291-14302, 2024.
[20] DeepSeek-Al. Al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. 2025.
[21] A. Défossez, L.. Mazaré, M. Orsini, Royer, Pérez, H. Jégou, E. Grave, and N. Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv preprint arXiv:2410.00037, 2024.
[22] H. Dong, Li, B. J., Wang, Zhang, and H. Guo. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405.19092, 2024.
[23] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
[24] Q. Fang, S. Guo, Y. Zhou, Z. Ma, S. Zhang, and Y. Feng. Llama-omni: Seamless speech interaction with large language models. arXiv preprini 240900409900666 2024.
[25] J. Feng, R. Xu, J. Hao, H. Sharma, Y. Shen, D. Zhao, and W. Chen. Language models can be logical solvers. arXiv preprint arXiv:2311.06152 2023.
[26] Z. Gao, Z. Chen, E. Cui, Y. Ren, W. Wang, J. Zhu, H. Tian, S. Ye, J. He, X. Zhu, al. Mini-internvl: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):1-17, 2024.
[27] T. GLM, Zeng, B. Xu, B. Wang, C. Zhang, D. Yin. D. Rojas, G. Feng, H. Zhao, H. Lai, H. Yu, H. Wang. J. Sun, J. Zhang. J. Cheng. J. Gui, J. Tang, J. Zhang. J. Li, L. Zhao, L. Wu,
I.. Zhong, M. Liu, Huang, Zhang, Q. Zheng, R. Lu, Duan, Zhang, S. Cao, S. Yang, W. L. Tam, W. Zhao, X. Liu, X. Xia, X. Zhang, X. Gu, X. Lv, X. Liu, X. Liu, X. Yang. X. Song,
X. Zhang. Y. An, Y. Xu, Y. Niu, Yang, Y. Li, Y. Bai, Y. Dong, Z. Qi, Z. Wang. Z. Yang. Z. Du,
Z. Hou, and Z. Wang. Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024.
[28] B. Goertzel and C. Pennachin. Artificial general intelligence, volume 2. Springer, 2007.
[29] Z. Guo, R. Zhang, X. Zhu, Y. Tang, X. Ma, J. Han, K. Chen, P. Gao, X. Li, H. Li, et al. Point- bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.00615. 2023.
[30] H. Gupta, S. Verma, U. Anantheswaran, K. Scaria, M. Parmar, S. Mishra, and C. Baral, Polymath: A challenging modal mathematical reasoning benchmark, 2024.
[31] X. Han, Q You, Y. Liu, W. Chen, H. Zheng, K. Mrini, X. Lin, Y. Wang, B. Zhai, J. Yuan, et al. Infimm-eval: Complex open-ended reasoning evaluation for multi-modal large language models. arXiv preprint arX15:2311.11567, 2023.
[32] Y. Hao, J. Gu, H. W. Wang, L. Li, Z. Yang, L. Wang, and Y. Cheng. Can milms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.055 05444444 2025.
[33] C. He. R. Luo, Y. Bai, S. Hu, L. Thai, J. Shen, J. Hu, X. Han, Y. Huang, Y. Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402. 14008, 2024.
[34] D. Huang, Q. Bu, Y. Qing, and H. Cui. Codecot: Tackling code syntax errors in cot reasoning for code generation. arXiv preprint arXiv:2308.0888 087884 2023.
12

---

Published as a conference paper at ICLR 2026
[35] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda,
A. Hayes, A. Radford, et al. Gp system card. arXiv preprint arXiv:2410.21276, 2024.
[36] A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry,
A. Beutel, A. Carney, et al. Openai ol system card. arXiv preprint (exXi:2412..17222 2024.
[37] X. Jiang, Y. Dong. L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao. Self-planning code generation with large language models. ACM Transactions on Software Engineering and Methodology, 33(7):1-30, 2024.
[38] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan. F. Viola, T. Green, Back, P. Natsev, M. Suleyman, and A. Zisserman. The kinetics human action video dataset, 2017.
[39] I.. Ke, W. Pei, Li, Shen, and Y.W. Tai. Reflective decoding network for image captioning. In Proceedings of the IEEE/CVF intermational conference on computer vision, pages 8888-8897, 2019.
[40] X. Lai, Z. Tian, Y. Chen, Y. Li, Y. Yuan, S. Liu, and J. Jia. Lisa: Reasoning segmentation via large language model, 2024.
[41] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu, and C. Li. Llava-onevision: Easy visual task transfer, 2024.
[42] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[43] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unitied vision-language understanding and generation. In International conference on machine learning. pages 12888 12900. PMLR, 2022.
[44] Li, G. Li, Y. Li, and Jin. Structured thain-of-thouglle prompting for code generation. ACM Transactions on Software Engineering and Methodology, 34(2):1-23, 202..
[45] Li, W. Lu, H. Fei, M. Luo, M. Dai, M. Xia, Y. Jin, Z. Gan, D. Qi, C. Fu, Tai, W. Yang, Y. Wang, and C. Wang. A survey benchmarks of multimodal large language models, 2024.
[46] T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, Perona, D. Ramanan,
C. L. Ziticck, and P. Dollár. Microsoft coco: Common objects in context, 2015.
[47] W. Lin, X. Wei, R. An, P. Gao, B. Zou, Y. Luo, S. Huang, S. Zhang, and H. Li. Draw-and- understand: Leveraging visual prompts to enable milmss comprehend what you want. arXiv preprint arXiv:2403.2027/ 2024.
[48] H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in neural information processing systems, 06:34492-34916. 2023.
[49] H. Liu, C. Li, Q. Wu, and Lee. Visual instruction tuning, 2023.
[50] H. Liu, Z. Teng, L. Cui, C. Zhang, Q. Zhou, and Y. Zhang. Logicot: Logical chain-of of-i--of-nnn offthought instruction-tuning arXiv preprint arXiv:2305. 12147, 2023.
[51] X. Liu, H. Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. Agentbench: Evaluating Imms as agents. arXiv preprint arXiv:2308. 03688, 2023.
[52] Y. Liu, Z. Li, M. Huang, B. Yang, W. Yu, C. Li, X.-C. Yin, C.-L. Liu, L. Jin, and X. Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 61(12)220102, 2024.
[53] Z. Liu, Z. Sun, Y. Zang, X. Dong, Cao, H. Duan, D. Lin, and J. Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arXiv:2503.01785, 2025.
13

---

Published as a conference paper at ICLR 2026
[54] Z. Liu. Y. Zhang, F. Liu, C. Zhang, Y. Sun, and J. Wang. Othink-mrl. Stimulating multimodal generalized reasoning capabilities through dynamic reinforcement learning. arXiv preprint arXiv:2503.16081, 2025.
[55] Lu, H. Bansal, T. Xia, J. Liu, C. Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255, 2023.
[56] Lu, Y. Li, Q.-G. Chen, Z. Xu, W. Luo, Zhang, and H.-J. Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv:2405.20799) 2024.
[57] K. Mangalam, R. Akshulakov, and J. Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36:46212 46244, 2023.
[58] Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11-20, 2016.
[59] A. Masry, D. X. Long, J. Q. Tan, Joty, and Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
[60] M. Mathew, D. Karatzas, and C. Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF winter conference applications of computer vision, pages 2200-2209, 2021.
[61] F. Meng, L. Du, Z. Liu, Z. Zhou, Q. Lu, D. Fu, B. Shi, W. Wang, J. He, K. Zhang, et al. Mm-eureka: Exploring visual aha moment with rule-based large-scale reinforcement learning. arXiv preprini arXiv:2503.07365. 2025.
[62] T. Nguyen, S. Y. Gadre, G. Ilharco, S. Oh, and L. Schmidt. Improving multimodal datasets with image captioning. Advances in Neural Information Processing Systems, 36:22047-22069, 2023.
[63] R. Niu, J. Li, S. Wang, Fu, X. Hu, X. Leng, H. Kong, Y. Chang, and Q. Wang. Screenagent: vision language model-driven computer control agent. arXiv preprint arXi::4402.07945, 2024.
[64] Y. Peng, G. Zhang, M. Zhang. Z. You, J. Liu, Q. Zhu, K. Yang, X. Xu, X. Geng, and X. Yang. Lmm-rl: Empowering 3b Imms with strong reasoning abilities through two-stage rule-based rl. arXiv preprint N.Xiv:25033055388 2025.
[65] R. Qiao, Q. Tan, G. Dong. M. Wu, Sun, X. Song. Z. GongQue, S. Lei, Z. Wei. M. Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint
2024.
[66] H. Shen, Z. Zhang, K. Zhao, Q. Zhang, R. Xu, and T. Zhao. Vlm-rl: A stable and generalizable rl-style large vision-language model. httpp:://githut www.co//oooooooo om ai-lab/ /VMM-MM--- R1 2025. Accessed: 2025-02-15.
[67] W. Shen, J. Pei, Y. Peng, X. Song, Y. Liu, J. Peng, H. Sun, Y. Hao, P. Wang, J. Zhang, and Y. Zhou. Skywork-rlv3 technical report, 2025.
[68] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, Yu. Soricut, Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. 2023.
[69] K. Team, A. Du, B. Gao, B. Xing, C. Jiang, C. Chen, C. Li, Xiao, C. Du, C. Liao, et al. Kimi k1. 5: Scaling reinforcement learning with 1lms. arXiv preprint arXiv:2501.12599 2025.
[70] Q. Team.
Qvq: To see the world with wisdom, December 2024.
[71] Q. Team.
Qwq-32b: Embracing the power of reinforcement learning, March 2025.
14

---

Published as a conference paper at ICLR 2026
[72] Y. Wan, W. Wang, Y. Yang, Y. Yuan, J.-t. Huang, P. He, W. Jiao, and M. R. Lyu. Logicasker: Evaluating and improving the logical reasoning ability of large language models. arXiv preprins a7Xiv:2401.00757, 2024.
[73] K. Wang, J. Pan, W. Shi, Z. Lu, H. Ren, A. Zhou, M. Zhan, and H. Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095-95166 169, 2024.
[74] Wang, S. Bai, S. Tan, Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12 12191, 2024.
[75] S. Wang, D. Kim, A. Taalimi, Sun, and W. Kuo. Learning visual grounding from generative vision and language model. In 2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 8057-8067. IEEE, 2025.
[76] W. Wang, 7. Gao, L. Chen, Z. Chen, J. Zhu, X. Zhao, Liu, Cao, S. Ye, X. Zhu, et al. Visualprm: An effective process reward model for multimodal reasoning. arXiv preprint S.tI::2503.1099,,, 2025.
[77] X. Wang and P. Peng. Open-rl-vided Open R1 Video 2025.
[78] Y. Wang, W. Chen, X. Han, X. Lin, H. Zhao, Y. Liu, B. Zhai, J. Yuan, Q. You, and H. Yang. Ex- ploring the reasoning abilities of multimodal large language models (mllms); A comprehensive survey on emerging trends in multimodal reasoning, 2024.
[79] Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q Le, D. Zhou, al. Chain-of- thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824 24837, 2022.
[80] Y. Xiao, E. Sun, T. Liu, and W. Wang. Logievista: Multimodal IIm logical reasoning benchmark in visual contexts. arXiv preprint
2024.
[81] 7.. Xie and C. Wu. Mini-omni: Language models can hear, talk while thinking in streaming, 2024. URL. https://arxiv. org/abs/2408.. 16725, 2024.
[82] D. Xu, Z. Zhao, J. Xiao, F. Wu, H. Zhang, X. He, and Y. Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653, 2017.
[83] F. Xu, Z. Wu, Q. Sun, S. Ren, F. Yuan, S. Yuan, Q. Lin, Y. Qiao, and J. Liu. Symbol-Ilm: Towards foundational symbol-centric interface for large language models. arXiv preprint arXiv:2311 1.09278, 2023.
[84] R. Xu, Z. Huang, T. Wang, Y. Chen, J. Pang, and D. Lin. VIm-grounder: A vim agent for zero-shot 3d visual grounding. arXiv preprint arXiv:2410.13860 2024.
[85] Y. Xu, X. Liu, X. Liu, Z. Hou, Y. Li, X. Zhang, Z. Wang. A. Zeng. Z. Du, W. Zhao, al. Chatglm-math: Improving math problem-solving in large language models with self-critique pipeline. arXiv preprint arXiv:2404. 244.22404.0 2024.
[86] A. Yang. B. Yang. B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, et al. 15/15, 2024.
Qwen2. 5 technical report. arXiv preprint
[87] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, Yang, J. Zhou, Lin, K. Dang, K. Lu, K. Bao, K. Yang, I. Yu, M. Li, M. Xue, P. Zhang, Q Zhu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Su, Y. Zhang, Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Z. Qiu. Qwen2.5 technical report. arXiv preprint 24Xv:22412.1511.. 2024.
[88] Y. Yang, X. He, H. Pan, X. Jiang, Y. Deng, X. Yang, H. Lu, D. Yin, F. Rao, M. Zhu, B. Zhang, and W. Chen. Rl-onevision: Advancing generalized multimodal reasoning through cross-modal formalization. arXiv preprint arXiv:2503.106 2025.
15

---

Published as a conference paper at ICLR 2026
[89] Y. Yao, T. Yu, A. Zhang, C. Wang, J. Cui, H. Zhu, T. Cai, H. Li, W. Zhao, Z. He, al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408.0 01000, 2024.
[90] J. Ye, G. Li, S. Gao, C. Huang, Y. Wu, S. Li, X. Fan, S. Dou, Q. Zhang, T. Gui, et al. Tooleyes: fine-grained evaluation for tool learning capabilities of large language models in real-world scenarios. arXiv preprini 3xXix:2401.00744: 2024.
[91] J. Ye, H. Xu, H. Liu, A. Hu, M. Yan, Q. Qian, J. Zhang, F. Huang, and J. Zhou. mplug-ow13: Towards long image-sequence understanding multi-modal large language models, 2024.
[92] L. Yu, P. Poirson, S. Yang. A. C. Berg, and L. Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[93] X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren. Y. Sun, C. Wei, B. Yu, Yuan, Sun, M. Yin, B. Zheng. Z. Yang, Y. Liu, W. Huang, H. Sun, Y. Su, and W. Chen. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of CVPR, 2024.
[94] C. Zhang, F. Gao, B. Jia, Y. Zhu, and S.-C. Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[95] R. Zhang, D. Jiang, Y. Zhang, H. Lin, Z. Guo, P. Qiu, A. Zhou, P. Lu, K.-W. Chang, Y. Qiao, al. Mathverse: Does your multi-modal Ilm truly the diagrams in visual math problems? In European Conference on Computer Vision, pages 169-186. Springer, 2024.
[96] Q. Zhao, S. Wang, C. Zhang, C. Fu, M. Q. Do, N. Agarwal, K. Lee, and C. Sun. Antgpt: Can large language models help long-term action anticipation from videos? arXiv preprint arXiv:2307. 16368, 2023.
[97] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[98] J. Zhu, W. Wang. Z. Chen, Z. Liu, S. Ye, L. Gu, Y. Duan, H. Tian, W. Su, J. Shao, Z. Gao, E. Cui, Y. Cao, Y. Liu, X. Wei, H. Zhang, H. Wang, W. Xu, H. Li, Wang, D. Chen, S. Li, Y. He, T. Jiang, J. Luo, Y. Wang, C. He, B. Shi, X. Zhang, W. Shao, J. He, Y. Xiong, W. Qu, Sun, Jiao, H. Lv, L. Wu, K. Zhang, H. Deng. J. Ge, K. Chen, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025.
16

---

Published as a conference paper at ICLR 2026
APPENDIX
The Use of LLMs. We only used LLMs to polish the paper and did not involve them the core contributions of this work.
A OVERVIEW OF THE APPENDIX
In the appendix, we provide additional details and supplementary information to further elaborate on sections mentioned above. In Section B we analyze the statistical features of the dataset, meanwhile providing examples of questions ranging from different categories. Section Contains experiments details, including the evaluation of LLMs, the evaluation hint prompts and RL experiments. Some examples of model outputs are also illustrated.
Due to file size limitations, we place all supplementary materials related to the paper in an anonymous link https://anonymmuu wwwp:/owwoooooooooooooooooo 4ww.n..pppnnnneeoe repository contains the benchmark data used in the paper, along with an additional training dataset.
12055467c554d6655 The anonymout
B BENCHMARK ANALYSIS
B. STATISTICAL ANALYSIS
As shown in Figure T0 the text length of questions in VisuLogic is mostly concentrated around 40 tokens (calculated by Llama-3.13s and Intern VL.2.''s tokenizer). We also analyze the distribution of image sizes, shown in Figure 9 The image widths range from 200 to 700 pixels, with an average of 592.3 pixels, while the heights range from 90 to 825 pixels, with an average of 327.9 pixels.
200
100
100
205
320
603
505
600
700
600
Figure Image size distribution. The size of images limited to within the same order of magnitude.
B.2 MORE EXAMPLES OF VISULOGIIC
To provide thoroughly presentation of our benchmark, we include more examples questions from different categories in the Figure and Figure 12
B.3
TRAINING DATASET
To facilitate further investigation of visual reasoning, we provide an auxiliary training set of 4,296 question-answer pairs drawn from the same domains and subjected identical validation procedures
17

---

Published as a conference paper at ICLR 2026
Distribution Lengths Llama-3.1-70B
Median: 42.0
Total semples: 1000
Moan:
Max length: 123
250
Min longth 25
Frequescy
1.50
50
40
60
80
100
120
Token Length
Distribution Tuken Lengths IntervVL2 5-78B
Median: 42.0
Total samples: 1000
Mean: 53.8
Max length: 123
250
Min longth
Trequesty
200
150
100
Token Length
Figure 10: Distribution of text token length in VisuLogic.
to prevent overlap with the benchmark. The training split mirrors the primary taxonomy, with category proportions of Quantitative Reasoning (30.7%), Spatial Reasoning (25.5%). Positional Reasoning (13.0%), Attribute Reasoning (8.8%), Stylistic Reasoning (9.9%), and Other (12.1%).
B.4 ETHICAL CONSIDERATIONS
Data Source. Our data sourced from https://www.w..... wwnbbi.wwwweee.......//.eo.....i wwt.cooo//so....o wtt.c///o/.sptt..ccoo The website's robots.txt https:t///// h/t:w///wwwwwww www. fenbi. wtt.:/c...omaae/ http:/rooooos...o file permits crawling.
Ethics of Human Data Annotation.
1. Transparency. All potential annotators received detailed briefing outlining the specific nature, expected outcomes, and necessary skills for the annotation tasks prior to commencing work. We provided all annotators with information regarding the usage of the data, as well as whether or not it will be publicly released.
2. Fair Compensation. We implemented an equitable compensation scheme-clearpe defining all payment terms upfront, ensuring annotators were paid well above local minimum-wi wwage standards, and issuing verified payments promptly based on tracked task-completion times.
3. Privacy Protection. We strictly anonymized annotators' PII at recruitment, presented only fully screened and de-identified source data during annotation, and restricted all collected annotation data and metadata to the core research team to ensure security and confidentiality.
This study is designed to be non-invasive, presenting no physical risk to the subjects. The tasks performed are benign [e.g., viewing stimulus material, answering multiple-choice questions, or interacting with a standard user interface]. We have reviewed the content to ensure it is not offensive, will not cause emotional distress, and will not induce stress anxiety. Therefore, we confirm that this study poses no adverse mental or physical effects on the subjects. Hence, we conclude that this study is exempt from IRB review.
18

---

Published as a conference paper at ICLR 2026
Quantitative Reasoning
From four given options, select suitable fill the question mark.
From the given four options, choose suitable fill question mark.
Divide the following figures into two calegories, ensuring that each calegory shares common patterns.
11888 286
2
P----300 240
From the given four options, select the most suitable one to fill in the question mark.
From the given four options, select the mest suitable fill question mark.
From given four options, select the appropriate fill the question mark.
?
Spatial Reasoning
From any angle, which on right view of the three- dimensional shape given the left?
following four options is the front elevation view of the figure the question.
image on net of cube. Which of the options the right can formed by folding it?
The diagram on left shows the net of a cube's outer surface. Please identify the incorrect option.
Choose the option from the four given choices that cannol assemble into 3D shape shown in the question.
The left shows the net of cube's outer surface. Which option on right can folded
::
Positional Reasoning
From the four given options, select the most suitable one to fill in the question
From the four given options, choose the most suitable one to fill in the question certain pattern.
From the four given options, choose the most appropriste one to fill in the question mark.
?
The option that best matches the given pattern
From the given four options, choose the most suitable to fill in the question mark.
From the given four options, choose the most suitable one present certain rogularity.
?
N
X
Figure 11: More examples in VisuLogic of Quantitative Reasoning, Spatial Reasoning, Positional Reasoning.
19

---

Published as a conference paper at ICLR 2026
Attribute Reasoning
Divide the figures below two calegories, that each calegory of figures has its own common patterns.
From the given options, choose the most suitable fill the question mark.
Divide the following figures into calegories, ensuring that each category shares common patterns.
Ae 1223.-9
?
S A- GIAIR
Bi D.2 Biale
2:1
C-0)3 2000 IX 21315
8.0224.866
Divide the figures below into categories, that each category of figures has its own common patterns.
From the given four options, select the most suitable to fill in the question mark.
From given options, select the most appropriate one to fill in the question mark.
4
Fq
?
%
An A.11//3..645 8/50
D:-02206 GHIS
8
A
D
Stylistic Reasoning
From the four given options, choose the most suitable one fill in the question mark.
Select the suitable option from the four given choices to fill in the question mark.
Choose most suitable option from the four given choices to fill in the question mark.
?
From the four given options, choose the most suitable one to fill in the question mark.
From the four given options, choose most suitable one to the question mark.
From four given options, choose most appropriate one the question
64 F4
Other
From the four given options, choose the most suitable one to fill in the question mark: B: C: 3.
From the given four options, choose the most appropriate one to fill in the question mark.
Select the most suitable option from the four given choices to fill in the question mark.
C
Q
10
K
14
<
P O
?
Choose the most suitable option from the four given choices to fill in the question mark.
From the four given options, choose the most suitable one to fill in the question mark.
From the four given options, choose the most suitable one question mark.
g
*
?
+
56
O9
G5
FAIR
BOOK
BYE
HO
D8
Figure 12: More examples in VisuLogic of Attribute Reasoning, Stylistic Reasoning, and Other.
20

---

Published as a conference paper at ICLR 2026
C EVALUATION & EXPERIMENT
C.I COST OF HUMAN EVALCATION
Because VosuLogic relies on heavily automated sistency filters, and rule-based unit tests tests-only only single, lightweight human pass is needed the end. Concretely, the entire benchmark (4296 train + 1000 test items) was reviewed by five annotators working hours each (around 10 person-hours total). At typical crowd-platfor rate of USD 120, or $0.02 per item.
checks, image- con-
C.2 EVALUATION OF LLMs
Caption Generation for LLMs Evaluation. In our experiment, we employ large language models (LLMs) for comparative analysis. Specifically, when setting up the LLM-based experiment, we initially utilize GPT-4o to generate captions for images with the following prompt: Please describe the fine-gred fie-grained content of the image or figure based on this question, including scenes, objects, relationships, and any text present. Please note that you do not need to answer this question directly, just describe the information of this picture. Additional examples of generated image captions are presented in Figure 14 and Figure 15
More Examples of Captions. We provide additional image captions for six categories, as illustrated in Figures [4 and [5| Even SOTA MLLM (GPT-4o) encounters difficulties in accurately describing the details of images from Vogul.ogic.
Distribution of Token Lengths Hermes 3-L.ama-3,
Total samples: 1000 length: length:
Meen: 5014
70
60
50
10
800
1000
Length
Distribution of Token Lengths InternVL.2 5-78B
80
Median: 502.0
samples:
Meen: 508.1
longth: Min
70
254
60
Proquenc
50
40
30
20
10
400
600
800
1000
Token Length
Figure 13: Distribution of tokens length in LLM evaluation settings, including image description.
Evaluation of Caption Quality Generated by MILMS.. We conducted human evaluation of four MILMS (GPT-4o, Claude Sonnet 3.5, Gemini 2.5 Flash, and Qwen2.5- 2--22555LL-72B- VL-72B) across six dimensions of caption quality. Using 100 image-pair questions, we collected total of 400 captions and invited 50 human evaluators assess them. Each caption annotators, with tasks distributed evenly to ensure balanced coverage across models. On average, each annotator evaluated 32 captions, corresponding to eight captions from each of the four models. Caption assignments were randomized to minimize systematic bias. The final evaluation results were obtained by averaging the scores for each model on each dimension.
independently evaluated by four
21

---

Published as a conference paper at ICLR 2026
Table 4: The average standard deviation of scores given by different evaluators for GPT-4o.
Integrity 0.266
Granularity
Hierarchy 0.324
Task Alignment
Clarity 0.326
Reasoning
mean
0.307
0.328
0.582
Table 5: Detailed scores of captions from different models across various evaluation dimensions.
Model
Integrity
Granularity
Hierarchy
Task Alignment
Clarity
Reasoning
GPT-40
1.78 (0.41)
1.75 (0.43)
1.73 (0.44)
1.74 (0.44)
1.71 (0.45)
0.99 (0.72)
3.n
1.00 (0.74)
1.01 (0.69)
1.78 (0.41)
0.95 (0.69)
1.74 (0.44)
0.26 (0.44)
Gemini-2.5-Flash
1.04 (0.70)
0.24 (0.43)
0.96 (0.71)
0.24 (0.43)
0.98 (0.67)
0.25 (0.43)
Qwen2.5-VL-72B
1.74 (0.44)
0.91 (0.67)
1.76 (0.43)
1.04 (0.70)
1.00 (0.72)
1.70 (0.46)
Table pressenss the results. We adopt three-level scoring scheme (high = 2, mid = 1, low 0) and report both the mean score and the variance to reflect annotator consistency. The table lists unrounded mean scores, with standard deviations shown in parentheses.
The six evaluation dimensions are defined as follows: 1) Integrity: whether the description compre- hensively covers the main image content without omitting key elements. 2) Granularity: the level of detail in the description, capturing how specific and fine-grained the information is. 3) Hierarchy: the organization and structure of the description, including clarity of main points and subordinate details. 4) Task Alignment: how well the description aligns with the given task requirements or instructions. 5) Clarity: the readability and understandability of the language. emphasizing conciseness and avoidance of ambiguity. 6) Reasoning: whether the description demonstrates logical reasoning and causal interpretation of the image content.
Besides, we also conduct experiments measure inter- antortan agreement among evaluators. For GPT-4o's "inter-annotator agreement among evaluators", we also use the standard deviation (std) as the metric. Specifically, we calculate the std of evaluator scores on each caption for every dimension, and then compute the average of these std values to quantify agreement. Notably, the standard deviation (std) here differs from the previously mentioned std of scores. this case, the std calculated across different evaluators' ratings for the same caption and the same evaluation dimension.
As Table 4 shows, the average standard deviation of scores given by different evaluators remains low, illustrating high inter-annotator agreement for our evaluation.
C.3 MORE SOLUTIONS FROM MODELS
We provide more solutions generated from different L.Ms/MLLMs on our benchmark, as shown in Figure [6] Figure and Figure 18 For the majority of questions, almost all models fail provide accurate solutions. Sometimes even when the final answer correct, methodological wrong may persist.
C.4 VERIFICATION OF BENCHMARK SUFFICIENCY
In our evaluation of GPT-4o on the VisuLogic benchmark, we tested three sets of problems at each step. As increased the number of benchmark questions from 10 to 1280, the standard deviation of the accuracy scores (corresponding to 68.27% confidence interval) changed as follows:
The result shows that with around 1000 benchmark questions, the standard deviation has reached approximately 0.006, which sufficient to reliably evaluate the model's capability in this aspect.
C.5 MORE FAILURE-MODE ANALYSIS
In Section 4 analyze some failure issues. In this part provide more detailed analysis for each categories. For commonly used models such as GPT-4o and Gemini 2.0 Pro, we have summarized the issues in Table
22

---

Published as a conference paper at ICLR 2026
Table 6: Stability of GPT-4o accuracy on the VisuLogic benchmark as sample size grows.
Index
Number of Questions
Std. Dev.
0
10
0.135
20
0.094
2
40
0.065
3
80
0.037
4
160
0.036
5
320
0.029
6
640
0.010
7
1000
0.0064
8
1280
0.0052
Table 7: Failure-mode analysis of MLLMs across question categories on the VisuLogic benchmark.
Categories
Analysis
Position Reasoning
(I) The pattern descriptions are often vague, using phrases like "in some way" or "following certain rule," which hinders further reasoning. (2) Even when identifying correct patterns, the explanations lack specificity. For instance, saying "the cirele moves up by one grid" doesn't 't capture the full logic when the grid itself may carry structural or relational significance.
Spatial Reasoning
(1) The contents on each face are only roughly described, and the model often treats distinct faces similar. It struggles with complex shapes individual faces. (2) When shapes are slanted or irregular, the model fails to match the unfolded and folded views of 3D object. (3) Spatial reasoning techniques are mostly limited to face adjacency, lacking more advanced spatial transformation abilities.
Attribute Reasoning
(1) Uses imprecise descriptors like "more edges," "more protruding parts," or "more complex shapes," which not clearly defined measurable, making verification and comparison difficult. (2) Shows little evidence of tracking dynamic changes across sequence; instead, it tends only compare two figures at time. (3) Fails to recognize changes in symmetry axes, showing limited sensitivity to symmetry transformations.
Quantitative Reasoning
(1) Uses abstract phrases like "more lines" or "becomes rester/symmetrrcal," without quantifying these changes (e.g.. how many lines were added or what structure was formed).
(2) Some conclusions, like 'complex - simple complex,' are based on subjective impressions rather than measurable criteria. (3) Unable to analyze figures that do not consist of basic elements; often resorts to describing only vague or similar shapes.
Stylistic Reasoning
(1) In tasks involving rotation, might state that "rotation is involved," but fails to specify the rotation angle, whether there is central symmetry, or which figures related through rotation. (2) When dealing with complex visual styles, the reasoning process becomes disjointed and lacks clear correspondence between described features and actual image elements. (3) Multiple distinct patterns
task hinder the model's 's ability to generalize consistent stylistic rules.
Other Reasoning
(1) For letter or character-basee tasks, the model is often misled by the semantic meaning of the symbols, which interferes with pattern recognition and hinders reflective correction. (2) When facing mixed-type symbol patterns, the model has difficulty synthesizing rules across different symbol categories.
C.6 HINT PROMPTS EVALUATION DETAILS
We first generate hint prompts with GPT-4o, combining reference solutions with question data as inputs (see Figure 6). All outputs undergo manual validation to prevent solution leakage. More examples are shown in Figure 19 After that, we input the hint prompts along with the same CoT prompt in CoT experiments S"Solve the complex visual logical reasoning problem through step-by- step reasoning. Think about the reasoning process first and answer the question following this format: Answer: (oxed(SLEETTER]]
MILMS.
23

---

Published as a conference paper at ICLR 2026
Table 8: Results of hint prompts for all models. Value changes are color-coded with red indicating positive shifts and green denoting negative variations.
Modets
His
Overall
Quantity
Spatiality
Position
Astribute
Style
Other
Numan
X
$1.4
40-7
47.5
44.2
B3. Bra
SG. (4-30.0))
G6.5 (5
100. O
35.7(u-40.77 77
75.0(4 4:50
OpenA (30251417)
24.5
24.3
$1.0
34.0
21.0
25.0
41.9
40. (--30--300)
43.4
355-0
53.3;
54.6
(6-0
377 5
(20247118)
34.3
13.5
16.7
24.7 (- 2.50
24.0 (4:13.
27.9 (77.9-44.0
22.0(=0
24.4
(-0 73
GPT to (20040600)
X
36.3
24.7
27.2
20.0
25.9
(/+-3.17 93
25.4
25:4(--3.0) (-3.9)
(
2
(++)
20:48 (++
Kimi lutest (202504)
X
25.9
24.9
20,4
26.5
25,0
16.7
26,9
27-0 42
277
93
(4 00
Doabao 1.6 Visios (250815)
34.9
32.9
34,6
30.0
31.7
30,0
43.5
43. 9...5.55.00
29.8
41.9
(0.0()11
(4.4(+4.4)
G1.9(ce
Comini-2.0-ee (10150705)
24.3
22.3
2G. 52
(4.00))
33.3
(5.0
15.4(-0.0)
42. (4/90))
Chande 7.-Sone (20250119)
X
24.8
22.7
27.3
27.9
76.0
22.2
22.2
(-.8
(3.3
(7.5(
23.6
15:4 (1.-0
(4:30 38.00
34.0
24.2
13.5
51.1
22.2
25. 3,+0077
26,0 (6-0
23:4 53
20.3,+0.00)
18.3
37. (+-0.)
21. 3
LLaVA QseVision
25.3
27.3
35.1
25.6
22.2
( 50
27-5 92
29.4 29.4(+-2.13
25.0 (5.0)-).)
19.0 (-3 93
28, (+-..0)
ShaeGPPTV
X
13.4
24.9
22.1
23.5
19.5
28,9
19.4
70.776550 80
27.8 (1- 92
26.7 26.7--2.0
MaiCPM 2.26
25.3
25.5
23.0
27.5
21.9
24.5
29.9
28.5 (c..8))
28.1
31.6 G
26.5
34.2 (1-12-3)
24.4 (-000
27.5 9
CLM--199
24.3
23,7
18.3
25,3
29.1 (44.8)
(
30.3 (35.0)
39. 2
29.3(4
26,0 26.0
34.5 3 (4.000
Ov62.68
X
25.6
23.3
27.2
25.6
24.1
((+233
(1-8) 32
(9.9(4.6.))
(-.5)
29.3
23.3 (0.30-2200)
(.-.80
18.9
21,5
15.2
162
20.7
18.9
20.4
25.6 35.0(45.73
27.8(-6.33
27.7 (7-7(4-12.))
(0.0(444.))
25-6(1-4:9)
16.0(-3.55
20.0(+
Skywodi-R1V3..5
27.9
26.5
20.6
24.6
21.2
26.6
30.3
81. 91.2
:0.9:-4.4)
80.7(+1.13
25 25.4(40.85)
36.7 (--1.55)
(8.9
86.9
Eriic-4--TTute--VL
27.1
34.5
24.4
$1. 31.00-43.0
28.6
$2.0(47.0)
27.2
$5.4
52 (2.2()
07.0
Qwen2.5 VL-7B Anstuct
26.0
20,9
25.2
57.8
25.0
(44433
34.0 4:
20.4(4-5.5)
25.0 -00.2)
20.7|-2.00
22 (-1.0)
Qwen2.5-VL ML----L--BBBBBB Instruct
X
26.2
25.2
23,3
27.2
25,6
25.6
34.3
$2.3 31
45.164-33.33
-5.0
26.1(-2.53
34.4
37.00+2.77
QvQ-72B-Be Preview
23.0
24.2
17.0
24.4
21.0
24.4
30.6
(00.8,+6.88
$0.1 10.11-5..00 91
39.7
19.6
(4.3
16.7 (-7.7-77)
(4.0(+334)
IstemWL2..530B
25.5
26.4
27.2
25.6
26.9
33.5 3,
$2.9,
36.8
$1.56
31.7(-9.55
31 (4+5.5)
2rl (+0.0)
IntemVL2..5788
X
27.3
26.0
265
31.1
30,6
30.2 ((+344
30.3
24.6(49.0)
34.3
26-7(-4-4)
8 -..00
IitmmVL11-888
X
27.1
26.7
27.6
26
21.4
23,9
28.5
(23.3 13
35.4 73
40-0(4-11-4)
20.3(-1.9)
25.0
24.4(+0.55
(++.5)
IntmVV1.1758
22.7
27.7
26.1
31.6
26.3
21.3
52.3
62 03
(008 03
35. (40.8)
32.9
27. ((400)))5)
Table 8 presents the complete results of the hint experiments.
Overall Trends. Most MILMS improve with hints, but gains are modest (R-3-10 points) and substan- tially smaller than for humans. Several strong positives include Doubao-1.66 Vision (34.9-+33.8, +8.9), Claude-3.7 3.7oont (24.8+33.5; +8.7), and the InternVL family (e.g.. Intern InternVL2.5-355 VL2.5-38B: 25.5-+3..3; +7.8). In contrast, Kimi-latest shows limited net change (25.9 8: + +1.9). This indicates that models differ markedly in their reasoning abilities.
Dimension-wise Effects. Gains are uneven across skills. Attribute and Other often show the largest improvements (e.g., Doubao: +18.3 on Attribute; Claude: + +15.9 on Other). Quantity also improves notably for several models (e.g., Gemini: 115.1; Claude: +14.6: InternVL lines: +6 9). However, we also observe regressions: Position degrades for some models (e.g.. Kimi: -6.6; GPT-4o-mini: +4.4 but with other dips), and Style can suffer negative transfer (e.g., Qwen2.5-VI.-BB: -15.6; Gemini:
-6.8; Claude: -6.8). The results indicate that models vary in their specializations, with certain models showing deficiencies in specific domains.
Model-Specific Analysis. GPT-4o and GPT-4o-mini exhibit small overall gains (+3.7 and +3.0, respectively), with mixed per-dimension changes (e.g.. improvements on Spatiality but dips on Quantity and Style). OpenAl-03 shows moderate overall lift (+10.6) driven by Quantity and Attribute, yet small declines on Position and Other. Larger Intern models (e.g., IntemVLL-788)) improve more consistently across Quantity, Spatiality and Attribute.
24

---

Published as a conference paper at ICLR 2026
Concluding Remarks. (1) Although hints provide substantial help for humans, they yield only limited and inconsistent gains for current MLLMs, leaving a large human human-model gap. (2) Improvements concentrate Attribute, Other, and Quantity, while Position and Style regress, suggesting incomplete exploitation of models' abilities.
C.7 RL EXPERIMENTS
We further include two reinforeement-lcrming baselines built on Qwen2.5-VL-7B WLL7BB-Instructt [9] and InternVL2.553888 [18] respectively, trained via rule-based RL procedure training dataset. Fully supervised fine-tuning (SFT) experiments on the same datasets serve as controls to isolate the effect of RL optimization. Comparative SFT Experiments. To verify the effectiveness of RL method, we arrange the comparative SFT experiments on the same dataset as RL experiments. The instruction consists of questions and Non-CoT prompts, and the responses formatted direct answers.
supplementary
RL Algorithm. We employ REINFORCE Leave-One-On (RLOO) our reinforcement learning training phase. As a critic-model algorithm,rloo is at a low computational cost while maintaining more robustness noise and KL constraints.
Reward Modeling, Inspired by Deepseek-Ri RI
20 we design our rule-based reward system that
mainly consists of two types of rewards:
1. Format rewards: To clarify model's outputs, design format rule that forces model to put its thinking process between '<think>' and '</think>" tags and put its final answer between "<answer>' and "</answer>' tags. Regular expression is applied to judge whether outputs conform to the format rule.
2. Accuracy rewards:
The accuracy reward decided by the response's correctness. The model should generate the response in right format, then the answer is extracted and judged whether it is matched to the correct option.
Hyperparameter settings. Our two RL models are trained with the hyperparameter configuration detailed in Table TO And the hyperparameters used in SFT training stage listed in Table
Table 9: Hyperparameter Settings for SFT Training Stage.
Qwenn25-VVL-7--nnstuee--
Intern VL2.5-38B-SFT
pretrain model
Qwen2.5-VLL-7B-Instte
Intern V1.2.5-38B
learning rate batch size
0.5e=5
2e-5
64
128
optimizer Ir scheduler
Adam
AdamW
cosine
cosine
image strategy
image_man _ax__max______________________ htpp_maa_________ pixels=262144
max 6astdmc_mcc___hh=-66
warmup ratio
0.1
0.03
max epochs
1
bf16
True
True
Other Details. The training environment consists of CentOS Linux release 7.6.1810 operating system with CUDA 12.1. 1. For
train for 80 steps 1x8 A800 GPUs and for Intern V.2.53L..-3BBBB we train for 100 steps on 6x 8 A800 GPUs.
C.8 RL MODELS EVALUATION DETAILS
As mentioned above, we apply format rewards RL experiments. Thus, to fully investigate the models' latent reasoning abilities, utilize implement training-aligned prompts during evaluation in
25

---

Published as a conference paper at ICLR 2026
Table 10: Hyperparameter Settings for RL Training Stage.
Dwen225-v1-77-Inssuut-llL.
Intern V1.2.5538-8-
pretrain model
Dwen2.5-VV-7B--Imsse Instrut
InterV
VL2.5-38B
RL Algorithm
rloo
rloo
train batch size
128
64
rollout batch_size size
256
128
temperature
1
1
samples per prompt prompt max generate max len 6 actor learning rate init kl coef
16
8
1024
4096
3000
3000
True
True
le-6
1e-6
0
0
VisuLogic, which is shown as follows: "Solve the complex visual logical reasoning problem through step-by-step reasoning. Think about the reasoning process first and answer the question following this format: <think> THINKING </think> ANSWER </answer>".
C.9 RESULTS OF RL EXPERIMENTS
As shown in Table D MLLMs with reinforcement learning optimization can yield obvious improve- ments in visual reasoning performance. Breen2.--V--7--nnttrrct attains 28.0%, a 2.0 percentage point boost over its non-RL counterpart. More strikingly, passing the original non-RL model by 5.6% and establishing a new state-of-the ofate-of-the-att on VisuLogic. Furthermore, compared to supervised fine-tuning (SFT) identical datasets, RL-enhanced models demonstrate substantially larger performance gains, underscoring the promise of targeted RL methods for advancing multimodal visual reasoning.
reaches 31.1%, %. sur-
For qualitative analysis of the effectiveness of RL, as illustrated in Figure [g), reinforcement learning (RL) encourages deeper, stepwise logical reasoning. The RL-enhanced model successfully captures state transitions (e.g., the movements of chess pieces in Figure aa)) and accurately predicts subsequent configurations. Moreover, it learns iteratively revise intermediate hypotheses-akin to trial-and-erron until coherent deduction emerges (see additional examples in the Appendix). These findings highlight the potential of RL methods to bolster performance visual reasoning tasks.
24 and 25 demonstrate qualitative differences in model outputs between base- line and RL optimized models. It illustrates reinforcement learning (RL) training enables the model to perform fundamental analysis of reasoning tasks embedded in graphical representations.
Figures 20
26

---

Published as a conference paper at ICLR 2026
The image presents a visual patter recognition puzzle consisting sequence geometric shapes organized rows. The objective determing the missing shape the sequence from the gisen options 'niskie Top Row (Sequence of Shupesyoint. *Fiistt Stape" 'n- Two identical trinngles, one upright and one upside down, overlap each other perfectly forn star-like shape with total of ourward-pointigg
IMAGE DESCRIPTION:
?
tringulla tips.'n- in The outlines of the trimngles intersect, farming an internal bexagon/aan2.. SSSoon Sn Two overlapping triangles, one upright and one upside down, slightly offict and not perfectly aligned, creating a symmetric 1X"" formation center. in The overlap forms six pointed star more distinct enhanced intersoction lines visible inside, iinn. ""Third Shupe**:n Two overlapping triangles, upright and upsido down, arranged symmetrically to create six- pointed star. in The intenal lines are more connected, forming central hesagon similar in style the first shape, with additional internal lines from the triangle ovarlaps iine4. "oourth Shupe""/'n Two overlapping parallclograms forming a shape that looks like bexagon outlined in bold, overlapping
in The parallelograms intersect to pattern with shapp angles and straight lines defining internal quadrilateral shapes. tainS. "*Fithh Sn A centralized actagon with an inner pattern of lines creating star shape, with the comees of the star connecting each side of the octagon from the center. "Misssng Shape (Queation Marky* "/-- This the space where the next shape in the sequence would logically fil, represented by question mark. 'alushii Betion Rew (Anser Options/anin- "Option A An
vertical rectangles central pointed trisngle merging al the form arrowhead shape/nn The basc of the rediangle is flat, with an internll vertical line separating them. inie- *Oytionn H 'n Two trangles placed side by side with a small gap between them - The triangles are upright and identical and shape/ inin- **Option C** Two parallelograns overlupping each other slightly, tilted sideways. The overlapping creates interesting central intersection with slanting lines, inin- **Option D**:n- octagon with central pattern formed by crossing diagonal lines. in The star symmetrical and fits perfectly within the octagon's outline.innatéé inintté Taskinin The objective to determine which of the four options (A, B, C, or D) logically completes the sequence in the top row.
The image is 3x3 grid puzzle with the task of identiffing the missing symbol in the bottom right cell. Below the grid are four answer options labeled A, B, C, and
IMAGE DESCRIPTION:
X
D/vlatéee Grid Description'zaattt First Rowial, **Cell I** A circle with diagonal line ruzning the top
right in2, *CCell 2 ideutical circle diagonal line, the top left botiom right'a3, **Cell 3*a: A square with YXX crossing through it. lllneww Sccond Row/nl, *Cell 4**; combination of'a partial square missing the right side, with triangle pointing upwards In2, **Cell 5TT; A partial square missing the left side **Cell 6**; An upward- pointing triangle with horizontal line intersecting about one-third from the top imantttee Third Rowinl, **Cell 7**; A mirrored combination of partial circle on the left and an (*1:"" shaped line on the right **eel g+e; Similar to Cell 7, mirrored partial circle on the left with an extended VLL" shape on the right.
B
2
?
gea Contains question mark, indicating the missing symbol this cell. ininaa Anewer Optionsinnnkaze Option A:'e- A combination o minored portial circle the left with a small V"L"" shaped line extending from the right side ininteas Option Biin- Similar to Option A, but the partial cirole has an additional barizontal line inindtee Option Cri-- vertical and herizontal mirroed partial circle the left side, without any additional lines. The task involves identifying the pattern logic that govems the symbols in the grid to select the correct option that completes it
A
D
short parallel lines resembling an incomplete rectangle, lninttw Option Dous- I"Li" shaped with
IMAGE DESCRIPTION: image consists of two rows, each containing boxes
?
geometric shapes. indicating puzzle pattem. in'nttt Top Rowinl. **First Bax** Contains black equilateral triangle centered within a white square In2. **Scoond Box**, Features two black trimnlee, one inverted and ene upright, forming an hourglass shape, centered insid' a pentagon.aa 'n3, **Third harizonial while ovall Fourth Box Displays black reciangle and larger black semicircl,, ereating shape like the letter IC,," inside
Shows black trisngle pointing right, inside
A
cirole/inS. **Fitth Hox**, Contains a black diamond inside pentagan.'ve In6, **Sixth Box"*, Contains a question mark, suggesting the missing piece the pattern tnincht Rottom RowitaThis row presents possible answer choices labeled A DJnnis. *OOption ***; Depicts horizontal black oval inside white heart shape/n- **Option Be+: Includes black shape resembling curved segment inside pentagon **Option C*+, Features a black diagonal stripe dissecting a white cross ip. **Option D**; Displays an L-shuped black figure inside a pentagon. Putters in the top row suggest a sequence or rule, and the task is to determine which option (A. B. C, or D) should replace the question mark based on the continuation of the sequence patter..
Figure 14: Part of image caption in LLM evaluation.
27

---

Published as a conference paper at ICLR 2026
IMAGE DESCRIPTION, The image displays a cube-based puzzle with one reference figure on the left and four options labeled A, B, C, and D to the right. Each option is within its roctangular boundary. lll Reference Figure: * Consists of three cubes configured in L-shape. - The base cube the bottom layer.'n Another cube placed directly on top the base cube.'n in The third cube adjacent the top cube the right, forming the I-shapee'n The front face
is filled with black color.inin2. **Option A: A:* Contains six cubes forning stepped structure. in The base consists two rows, where the left row two cubes and the right row three cubes,in- in The top cube of the left row and the middle cube of the right row are stacked on, forming a second layer with two cubes, in The left cube in the second layer matches the position of the black face from the reference, so it is filled with black.nnnn ** Option B:* Composed of six cubes arranged vertical stack. - The bottom layer has three cubes forming L-shapo.in- Two cubes are the left forming column, with on the right adjacent to the top the first cube the column.in The top layer a single cube directly over the central bottom cube, curving upwards. One of
fices bottom left cube is filled in black. *Oppion Cn* "m Fealures five cubes in a salf-stair configuration. The configuration starts with two vertical cubes on the left and three cubes forming a step on the right A vertical column of two cubes stands on the right edge, and another cabe is beside the topmosi cube of this columni'n 'n The black face appears on the buttom left cube in front-facing position wins. **Option D: Displays symmetrical arrangement with central cube surrounded by four cubes,in The configuration resembles plus sign when viewed from the front. A straightforward 3x3 cube grid with single black on the bottom layer, located at the front of the central cube.in'nFach option offers different combination cubes only one mirors the black face placerment from the initial reference ligure.
IMAGE DESCRIPTION: The image is a visual puzzle asking which shape four be folded make given 3D object. is presented single rectangular fame with two sections inlllWe Left Section''n- *3DD Object** 'n- Depicts a transparccal cylinder with rectangular top and bottom, rescmblling drum with squared faces. in The cylindrical shape thin and widc/n There solid roctangular face shown lop, with dashee lines indicating back and botion redanggl edges inside transparcnt surfacc. Section'nn- in- Fouu 2D Shapes Labeled A to Dr 'n- ** rM; 'n- Shape resembling horseshoc or a U, oriented vertically 'n- Two long, curved edges form the sides conmetting a wider curved top and narrow open bottom 'nin- *eger 'n - Two identical, seporate tall rectingles placed circle encompassing smaller square exactly the center/nn The square's sides are panallel the cirole's diamneter, touching the circle's lain **g**;'n in Two identical trapezoids with longer vertical sides the left and shorter the right, slanted inwardly lowards each other. In inEachh stups option is shown as a potential development that could fold into the 3D cylinder on the loll.
Right
vertically side by side inin **(**;
IMAGE DESCRIPTION: image presents visual pattem recognition problem involving genmetric shapes, divided into two main parts: the problem set on top and four possible answer choices below:'nizlit Problem Set (Top Party'n- *"First Row (Left to Right):** 1. **Cylinda:" firee-dimensional shape with two parallel hases connected by a curved surface, depicted with dashed lines indicating hidden edges/n =*Truncated Cone (F (Frustum):** A shape with a circular base and smaller circular top, connected by curved surface. The top is parallel and smaller the also shown with dached for edges. in 3, **Cahe*** represented with dashed lines to suggest depth'n'a *SSecond Row (Left to Right) ** I. **Cirll:*** flat, two dimensional shape with only the perimeter shown. la 2. **oone:** A three dimeenioonal shape with circular basc tapering point (apex) the top. depicted with dashed lines to show hidkkn parts of the busc. in 3. *uQustioon Mark.** A placcholder indicating the missing shape, suggesting the need to find a shope that logically follows the presented sequenee Irins Answer Choices (Botiom Part) option labeled a capital letter below each slape from A Diiipp- *h O"Otiion A y *Suuue Pyramid.** ** A three-dimensional shape with square hase and four triangular taees that mest single apex. Dushed lines indicate the hidden part of the base'n in- *Opption B.**g "Hexaggnal Prism:** A solid with two parallel hexagonal faces and Dashed lines represent the hicden edges of hexagons. "*Option C:**ig- ** "Spbore: perfect
?
A
B
C
shape with equal square faces. The hidden edges
coctangular faces connecting
dimensioral round shope. Equstor and meridian lincs are shown dashed formal suggest depth and roundness. iimn "*Option D.*
Triangular Pyramid (Tetrahedron):** r* A three- prro-dimecsiooaal shape with four triangular faces, one of which is the base. Dushad lines indicate the hidden edges ininent Task:/nThe The task is to identify the shape from the answer choices (A, B, C, D) that logically completes the sequence the problem ses, based the pattern from the first row where dime-diinnsioaall shapes were involved, and consistent with the exisring sequence pattern.
Figure 15: Part of image caption in LLM evaluation.
28

---

Published as a conference paper at ICLR 2026
Predexx
GPT-4s
ofick fillesing four optins is far fivel clenation the figare the seeaatttoo
dossciaall Clicck bduual Mook dilele aprices mig.. mUpptin tn Sheres vi left A the exidile, and D the beason loft The an ongenee meadee in. *roysion Bt Shows on the left in far makkli, and on the niltt Taa decs use andck dee fiom size up is not viable direcly fiom the hann "*Oriie cine Strne ar the top,
Toodee this pritle ne need detaniie cleutiin viww given figare (mikiie), r-lil
the best stnn. **atacs the
the mistie This vecomee the posticuing doos nee in. ""Opption top, is mitill and D buttes but uimina the cutre laok vhitke from the fountstidd only foor
hettee,
Stanes
rxpeseoted by eaannty veiitryy alganenttlllllleeeeee the blade lerdkwweer (boosd)A1" wilh
cermecty repecamee
that conceely
dones ralarve
OneoVT3
To desemnie the comeet from elcotio coopeetvviira itall * "Mdanii the stratur froes the foxt ffa- The lapensst loyer las Modd Labelel 'B The rikkle boyer has the hlncke landle 'A and ta The bottor layer hax thrne Mocoe bheidd t2 werd "cCorpune with the given optine. Haa. *OOyekos A. Shows B on spx i' the midilee and the bolten, Boveses, does not show is the ruakie ayer "Noytum showe WI top, is the midda, 'IP betsen, Horevvrr does not show the loyerinn "1oysos Shows top and the foot alouaion lehhour Option Chatedeeaarr
the
need
onetee from the firet
D
Refenmece Seluttion
The fost vicw oft figae in the spcotion shoeld be inc one eota threr equane the and lener vp two leters
the meildle
soose 'C7 the mildle layer intalald theoralyds (looud)))
comorpende optize
Deelree-1 & V-Viiin Pro 250-1111115
Sky Geniii-22 Eegras-cg-- $5-6
sndeneand postiir the olyers when viravdl fives
fhom ore I minerly below them cate and the nightor of (ri the rebative proilinta B. and D ure oxsed. B the by axsd paikin, is direcly below B.Cia to tight A the same level A, and is boloww Als Ogiion to the right of A, wiille in tiis optios, C is bellw Buadd to left of in Opton C In this cytiee the overll smoctare and the relutiic positces the ratee we noe in Yee with the fooe edroviee vie of given 1- 73 farm The pasltin of and is oveual lasee ure anegi la- Optice D: la the optioe, the of of is isonect C stoud be at the sane lovel i but the right of wlike is thing C to the wase level ao and de siggt of Saakaaween (Aos
figireee looking the
that
noe the circcive or -rSuc 2. Disemiis aeeon ayer bonara Thar aorker blick ianudassly tidl rigl cfD, ans rubhle foo vaide tee Hlckk aeeiiie waree sheve back E' meelee block inreckenty eige it
leeel to aree Ciln Thllw thnee atee Dast Analype each opaonta- Optin Ie this
Os option, the ofC is inorect C Coloodllo be De some level A but
from widdi
Biternttr tighos dook
ras'
cilee ay faym tollee and
Cloode 3 Sosrec Thisking
dovarta on viow they Top sunc block i Naldd soe Block [A] wish it phoo onticas tng Usiics makioo cia sracans muety: twe in the lask
"othiak ei stalear the cohic bhhiled with kenis A, C. and with botton pur chose four difieree 301 stease lateled woppree BI, C. and D sond need teumeize whtch of thone ID viewe expearne the tree elevaton vire ofthz 30 amingeneet otail and, le me underntand the smrngemere of other in the 30 figects Caps A pootonee in midil ot the amregemeniis Cube B is podioooe on of Ain Cuke is prolieted dhe right sik of top lodfo Cube D poslooed the lower level, the of Konnooo lots consder what fhor electtion diew mourctih fioe dlevation wlew 2D
produces carcially miil
imnge darenn 1D anargemene
it wi Cheenel deniiii Oprice tos only and the bebere anclatalezes thaualis Ao
reprosetciine wokking ar atjer from the from showing only wha would he sichte from the perpertiio wir resmine ouch optorr ppptto lnShoocc yiew wabion the replette- the midiee reseddngg to the fae betion-effsiaQqiui
Lasaze Caption
IMAGT DESc RIPMON trange candic two mole suctios The par deoas sttr-diinnnirol stractin
Coaflowes Optii Bicchooo e 3D sice with
view with willin top-rightn in nibeergght i botces bisten-ighttetr
the lop milM-ie
the ap-llfii mibllelefft extcnding the righttini D'vdlanno
figirr
cormectd other while the heken part proeerms four flr armagemmene landled et
fieet conueriag
""Cubee*
comparbeee Top Section
up ofright iddittlsl
dociption.
Optin cloudy wiihh
Foom loll TTy Bowe Cube tieeld 'B"" B,C.
Arongemees
fou
labokel
wruch sice Oyten Akliiiai Eleetion Andyyiisaaee selve puileon, tesl the clestoon Undenttad the amagenentain 1D figarte- Cabe pooiineed be mokie lasers Cabe paincced of Nin Cute C pootional at the top-iiht Cake ia prodiceed the botte: liltaaat Sup 2 Desaminc vint. more diectton vios sheeld clawink the dirctton view choor what you or when looking direlly st objer fioe the mhee in the casr, ntll accme the from the boo when wa can are aaee D. Danter Step v. Aralyos euch optariet coking the opuine iie- *Mrystee A shewe at mpsiett A in misdk, and D ecunding so tte right 2a the bateerts.
alown the
stelt i:
left to- **atadde Ro Caas lahlldd oor
inabll I- cafilln- masutees Rowen Caae lasced 'Dr the lowa lnfils DDDes and Hei There ndiiioaal cale divetlly behind the cuts labeled ''A'"
"Opiioo B Shows B at tepelel C the nillle-ighs and D botonsiel wit cabes oxtceding (glas "Oppiinn C++, Shows B tgg modee midde right and the banontarenn Stay Maxch with the 3D wngerreeeoteen looknng at the fggur from the thver (where it webieylnn desild he at botten aboue Am The posrr onnnnning in Optize moscher thes amangenent parkectly errdectyerakkooee erakere "boonddkg"
pootring dephinn in--a Aneterr cabe is obone the the leuiled (oA os iop of cobe (n. cree tomtn Secionn That Armmgmeentiio o eeper Optons (A, c. bym lg. Eack optas represett a potertal fatunetion on 20 confiaeratin the 30 atrcaue whMl Optin Alla- A concal new oaee with
the mndde lel, and D enconding right hhe tomorrs- "Cyrio D-* Shows at koge right
shold shose and to the right of Il shoeld he
the top, lolondd the bettn to- stggl cote amuched the agat of A moseess Optinn thro- vertiod tine sinike Option
by and then
Teoahve the postkan we seed to casuidll snabeer de 3D anstaa and corptac to foc ID smasgemaraa provieed Lets hoss doar the exppciii **orrity the froer Vira Amerrnantt a The five vise the 1D emettr shows cahes bowles it C. A and The amaggemmn frue to aye and topo heron ioin Top now Bia- Midde now C (riht), (letiee aetren 13 cattered "Arayys the Depeh tinght There addmanal cate dinetty sehind cato labcled me- Anstur cate atove cate labeled as orpeetie of aad. *N 21 Attangeeers "Myyeer A Verical imegresse
top.
follwed by
single
5 tap. by Cyrios
(xg). (mWMF)) thettent in Single
vertiel line wilhod side the Beleo "hat ITA" Inlineed by UU" same level sinal NCT stucked ri ot arasill Pappncillle ask matci foe 3D souctaccss tiibk and insplial aanggneet with the 3D Datieedd oppcoorationn aleial A B. C. D.
right
deyth
berioonal lise
with
brige comeaty "Opprion
pop),
Benoon)' Single
rith
trore smagemen ""Opaar Catey
amengeaeddide "Oppikn D** Fociootal line of tlece B.Ckell,, oriddll). Drightt Stelle atuchee fh
Caniklee, Oliich, D Bodiennla dece madd
Tiis manils
flver emagresee acovums adition dyth height comeetly wie. "Coolltsoon* The 1D amagenoee motches 1D anerral frwm siew depth and Opuion brnawowee (Dhaadd(D)
Figure 16: Solution examples generated by different models. Reference solution and outputs generated by GPT-4o, Qwen2.5VL-7225 - 2emmini-.0-pro-exp--000 Doubao- 1.5-iiiin Vision- Pro-32K and mruude-.7-sonnet-tee (Qwen2.5-72BB Innstruct are also illustrated.
Additionally, the image caption and solution from LLMs
29

---

Published as a conference paper at ICLR 2026
Prebica
Wikdrol the folicwiiz optrop fieet clvation
To solve the problen Ie's asalsze the sequetes of fisso oos "Idesif Putterss in Givee Figuzz ** Break dows eais fguae is the squeense to slanify parenss or trss trassocantions in- The firt image contains lagg square disided iato sere al tiargula secioss in The secend image saows as olge seciing and Runiing sow slaye thid mage inmoduees mar Exgmented pane in- The foorth image repaits emiar patters with now teral in Impremative in2.
?
at the Specifi Iramffrmmatime "aa Notize peuible sctations reflections, comsinen aratin- There is paters divizing the square incemtntiss
basafomrations dast occur fiom one figue biasstier rectangala patiion is cash stes *Fredkct eligttly chregre align gamenic
Transfornationn ""... cractising
figne,,
"AAplyyng
D
drsighsforear charge optioms depalin Optio dispraa
Pattem==*ing
Reficeace Setustion
basuforrationn Option sugges
Ir frund the the clood obsiew: Consideving the of boins, the wamher of tues in the figare is dis quaskai den and iss upeaos flance 6, it impoosiil chease anwes Consdkiing velieeneni of facss spui
peoppction own
slape disiio tsinar/eaaS, **odect the Anooerthin Camoder all ovslener the pasl for trreffemmtio mutching (ibosed (I))
contitues simple
asalyse the claracoratiss Beetoo- dd- Pntte-Prr-33 27-3222-5-155
munegttr thess. L'ansdortag namber wrapaa ormnnggtar hore the figies is the quanter eers is 1.2, a red in eccerace, oo wanther mrnnggi for bucs the figrn at shelld be rigarement
numther
aquare divided such
anall asgled binzgles We consider the selasoosiiip lise
only
perpadicclar relciouships Obezrve the number right angled mangies each figuee The five figer right anglod tranglos the sccoed figure las right unpled vingles the thin Egure has ride- anded tiangles the fourth figare has 2 right angled trarglos and the fith fagus has right- snplod trinule So, the sumber of right txpplld trinnges in the fiaures in the cuctioe set is decocasing by I succssiveylla. in. Thes, asalyze the options in Option A. The Figir has right engled mrnngiie which dock not nest the repirement of the number of ligh- anglee trinnglce dhanosing I easoasenncyy in Ogiaan Dt The figure has right- angled tranglie Awcrding to the rale that the number of night angled thangles the figues in the quotion- set 5,4 3, 1, and than i this option most the ruke in. Optiae The fipac hae 4 right snuled trannds which daas not meei the niein Optizn D. The figue has i ripht- uxplol trangls whics does not neet the rulo lenseeer Wicsed
betweze the lines
sobre this prbbenn sequence the first five Sequeece+t.
whick of
C,
this unall "OObseve
that figut the
divited the smaller
Nodice
two disporala that intrrcct at the cener of aquare ii.
*1Iccstiy the Puttze" Oboarve how the trinnics ure iladed in cack figure in- The fast flause has the top- left and bottons riglt. triangles staded a The secand figure has the top right aod bottom-leh left trangio shunka in- The diidd figure has the lop-lcf.
and botom--iall righ triangle staced assin in The fouthh finue Iso the tap right and bottons left tiaudes deaded asiin in- The filt fague las the top- Icft and betoes right tranglcs staseed sice.
**Arahyzz the soquencet Obeeve fire prinided the and look fo a palte or relatonshg beween the to detemnise the sodh figare (Poja"" "Eaaiine stracardd features in Courr. line segnerss, anilyne dlspes, check symmeryy look incsectiors, en. Each figure concistently has interal thne segneers The ships and gymnekks iss vary sgrifliantty 'n" *1t Look for dtrple andtirratinae Raration rellectoons do not sren to map consecativ imagns direclly (rg 1'1
(let' call them F2, F3, F4, PS)
**Detennie the Patioua*, The pattiit altouates beteet. shading the top- left and seton-iggtt t longls and stading the
top-right botiom--est mnanglcs wisi. Applyy the Patter"
seeated refleceed not P2) t enç ceustir shorssing panars or sydox-**g Saybs onfonimbencl figime (F1, I', ESy follow one pancrn, and con--umbennt figims (22, 14 16) follow smother in Mache therc's cycle of specifie leagth (ea. 4) lrde cysle 5-4, then E5 sholk ielice to F1, and Fo should relite to F2.
Since the fifth fixue las top-kst
button righs inazgiics
duddedin- hiving sep-rghtt icuniiyy
and botton- left tiangles shaded and. **aacch Optixeo
sbaded, sest figuse the secsense should have the top-right
A
constomatioo Tuch that Tor-k- - T(Fee) some Let's my le-S in eeCheck Fl central stretton heard connecting midpoints (a dianced chape's plan vertics line segiennts connecting diansodd cente point scar Lets denore surcture IMD
Vissally, quite differcet from un "Hepohesize a reltioashyy with fined offert tn Let's nost
minngle
Top
Option
Top
top riglt "usiittd tnnsfomraioon changes smacure (MD complex strecton
(soppe
centaal soucture based on two moin degosals (an shape)) plus
the Let' deoote this strettuee
disporal segrners
dhaded/anins **oclcct the Corret Optiar+* Ioeh sprans and have the lop-ight and botiom-ieet triegloe shaded. Howeyer. we need to eneare that the patiers cznsisrri with the prsiics figuess Since dhe sequeace alteratts and the fifts Sgarc the top- luet and bomom-nghh right insegles shaded, the now figrt dhrild inckral have the sep-right and bottor-tzit -eftt minglos dusted Bolt and this pattern hat we acad to chasse onc Croen the spttoms, we can schoct sitter or D. For the sdlt of thes selution. we will chouse option winlharcoon the mos scitable option to in the qucation mark is (Bused(3).
skodediaa
Top right
left
"_shitedd semooms
odded
disporal (7 shape)
Li
Chanae Change
Addiions Prdddd F3
botizuetal and velioal midliees
stape) plus
left
rigit
the
Let' this
+0 Apply mondoratton Charge Bese
Additios the
Following palice in T104 poteniially 23-26 twhere the base also becornes XL
base (+) changes (23, Weat hapsers the added lisz ( discendat ia TI clarged to i. 13-85,
changes (Ta ITe' changes to (30 and changes (3 the roulling fagze F5 would be (X- it Tais ereass on X shope fiom the main digguaal plas vertial line seaments Let's check the optionsia in
Cycoo A: OC* Diggonal cross plus benizoral comeotinnta Option (f- compley) Slash diggosal plus other lines Oyton Co compless - Stack diagonsl plus other tneeco Option D: (X- Dingoral cose plua veriaal connectoon is The peodiced stractare (X makiica Ontion exaclly iti"
Irange Coption
"Cusclusion." The patiea Fur-3)- - Transform holds consiteatly, whare the tanniomation invoives chassing the lese stucture and Be oygegostiion of added lass, Applyias lii trrssfomntioon to sugaris F6 shoud be Cytion D Diiiaaooeer (Boxee)())
lscras guenatti The
foe
follwwal ganaie spareis pateen agane
la
Secied heatl right Condies lego agga
audee
Top
To selve this cinzal lugical nasoting peohten lnt' aralys the panams and inraformurans oecurring from one square to the sest in the top nos. We will look for conenical dhungge or rulos that can help peodic the nzat patien ielttii Skep by-Stp Analysa /dial, "SSqare to Scurce 24% **Inital Patter Lame diagooal from boton left to spp right in- "rarsffemmeton" The diggral flis to go from top left botter right, and auditinnl brtoors line is stted craatig mor seprers dest **Squmm" so Sejore
Qnee3,67718 Eutrrnt
Phoaue lat Wilhs
right
peus, Senting andler chingio and snall irpcesed ros. 20: paters
ladaroonl betionaal deginal lives sopiii fill
righ
olde
or riciagle in the if enl Then who
you
*iniaal Patterc Lans dipiaal from upp left betomn right win heriomial lire "raraformtiion" Addiconal vertiaal and boroamtd lince are added, croring a gubble stretore with diagrnal intereasing "Squure Square d""og *lnnial Pattrn** Griblike struture disgonalin- sipplifees cantaa diggraal
opuer
five
Foes and suap vata ** Faatro ranglocee
kegs
right
cosuting mringles rapeoaide "*Spurr Sqare
Iesposede eercaur got Caraea vatted liz dnading de qqasc] I
fonter right disting sqare half
The gatura simplitio
I foceera transoaing criugddrr far rug on ina iequerbe Ma A Ii plocccddee reporriing I The partess ppde I staras suptem lna. PCpppao
gis half with berounad the up comsingg
thetoy
dapora
*Daggrall Fipping" mid-top. diagoral
diggnal
srch aqurs
segnoems bus then square
Additin Sinplifiaion Addiiioal lines added
smpl and then deerass, erggniing opelical "Symmctry Complexity" ** The complexity inivel Prodicting
dingiaal
lo dis
Pattere rutteee-nGivee vrGvee altemoing
the ryclcaa romec coppecttyy the
should-s Tbme
fe coggon and quatiilae lawe sheps, lowing seunch
diggonal Seee bottonn left to top right (flepiig bock Be reliivlly sinple, possiily with fewer additional Lues suuntt Evalaating the Options in "Opllon A Muliill inte secting digonaal crentin stx like offect This is too comples and does orr tollow the sinplicty besd.n *Opikon ger, Large diggoaal thom kift to botton right with adtionaa lires foming mik of depe. This a good cancituee in fallows the disgona flip and has marknit lnve of complettty in *Cogrur C* Insoracting verticl horioernl, and diggrna lines This is aloo eoeplic and doss ror ii the cimplificennn tnnd in- *TTgtionn pes, Multpple intssecting digonns eexingg unliiple trirgala scattents This is sinils to Optict A and too comples. antth Careltsion 2Option B the nos sutable choice as Bollows the dicgotal flip and mamain a moderne lovel of comppoooyyy voyry fiting the observed pottm windoserr (Doood (B)
prlples Contaio insmucting
bara and jet fingeeent peltoe til enllpiinn pao Sr 1
peadage
opiion I luich of fur (o C. (1) leghtily the
with tp rin that
Figure 17: Solution examples generated by different models. Reference solution and outputs generated by GPT-4o, Kimi-latest, 2ammini-.0--rr-exp--222220 and Doubao-1.5. --s-vvio---Po-322 Additionally, the image caption and solution from LLMs 1rren2.5-72--nnstruee are also illustrated.
30

---

Published as a conference paper at ICLR 2026
Problem
Which the fillowing figures differest from the other three:
To solve this problem, need to discem the parter sequence of symbolssaaii 19%, Happexx 'div Viddos... Mfainle''s analyze each anctninl. Wwi' is symbol for equality ln, "Me" denutes pereentage nš. the"" is the symaol for approximation int I"+1" represents diviston, ins I"M signifies factorial no. Iis implies ""theredoee"" ("t.. ins generally denotes consider the sequence typeslala The symbols seem be mathematical logical operatoes Each symbel could represent seties or group classiscation in math math Now for pattems or celanooshhipe sial, Equiliyy percentige, agproximation, divisio:: These are basic mathematical operations relations, 12. Factorial mathematical operation with numbers (n3, "Teerfore"" and ratio bodi have lopical or presentazional mecaiing in winNow match optious inA 1987. Hash, quite fiting into the matherstical uperation 'eB. (quite fitting) IoC. Mat Addition fits mathematically but not panicullrrl logical sequence. InD: ('x!" Multpplication, mathematical uperasee but nut sequence- linked innGivvm that "i'l" means '"therefore." next logical sequence relatonships would 1"2"" aseaning ("becasse)" often appears a subsequen: explanation 's'a Tha,s the patter aligns more with logical progression from "ta'" to Pr.""alnAmwerr (Bowed;B;
the symbuls inlol ookkooingg
%
presence something ialaT steunify the
food that sember Reforeece Sctutte
prrpl qpesion in 3, 3. 3.2.3, 2.2 Therefor, the srass slaudd be dire-ssat grapt. C, D all cec-ous paspla, Only axtin swed the reqaismen. canract stavect
"Beecasee"" lugically follows "therefund"
Quenn SVL-215
solve this problens analyne sequence symbool and by to iduniiy any pattems relationslips that night help detenming the cornect symbal for the eight hox widee Step-hy-Step Acalysia utaal
Doshbe Vidia Pau- 1o--33--240 28--50015
(Equal signt/n- 2nd how ("56"" (Parcntt signiin 3ed hot Po1"
""Soquence of Symbbls in-
Autlyye the nature of the givei symiollsu syniool soquetes %/.
special - purpone symbols used mathematies, logic, general nosation, not common seithmetic operation symbols like "+" and "x" Among the oprions in- Opiion C C*44" and Oprion D3 "sx" are busic ani arifhmmtiic openation symbols for alditoo and miltipliaation respectively. They do fit the pontern of artihmetic operation spacial symbols the given seguenee Option "s" no direct relation
(Approximatell equil sign/at 4th box ""l" (Division sign dats/nn 5th box: 191 (Exclumation marklnn- box: 1"61" (Therefore symbolya 7th fh"" (Colastiinz *CChserving Pattoms** in- includes - "Mattematiccl Symbolg** The thatt imo0", ttat", mothematical symbals
geseral symboo ofhen used progretmiigg lagging, etc. but symnhols related muthematical and logical notations in the given sequeiee in Option B """"" a logical symbol used to mor "because" in logical and mathamarical ressoning, which is line with "therefore").nz Conclusion.nn The most suitable symbol to in the blank to maintain the pottern "," belongs to the same catggory of legical and muthenucical notation symbols as the ones in the given sequence in innnnseer [B)
17677, and ("M. These symbols cocusionly used in mutheraticall expression and logic in- **1 **Logical Symbbole* ee. The 1ta'" (therefore) and 1h1" (colon) logical cessoning
given symbolc (such ":" which
"Progression Relationshig** nead to determine there is logical progression or relationdiip between thes symbols iain3, "Poosibbe Relatinnhipp +hin- "*Mathematical Operations The symbols
Gentai 82
Detailed step- by step deriaaton." al. Analyye the sequence: The sequence consists of seven symbols followed by a questio mark:
represent a sequence of opentions logical aepa in-
11" 22 '? Tinn.. **Identify
L"Logical Hlow* The 1Xl" (tiecefoos efor suggests conclusiinn or romil, and the 1-15 (colon) could be used to introdice condition explenation the Oprimms */nn "Opiimn A ''8' (Tinhing ur number sign) - This symbal not typically usad in standard mathematical or logical notation OOpion T+*a; ("ro" (Becuue symbol)- This symisol logical notation denote "'ecmssd" or **Option Ca in (* (Plus sign) This is common muthematicall operation "Oppionn D* t'x'" (Maliipllcctioon sign)
potential pattems: ** Lock for visual pattenss, manerical pattares, or logical relationships between the sycabols. Une noticeable feature the presence and number of dots in each symbol a3.
** *Amilyzing
*oCunnt date ** Count mumber dots (or small circles) in each symbok'n
deten dotsin
datinn 0
dotii "T: dotin
dotila
Unknowale4. **Formulate the doi sequenoe:** The sequence of the number of dots is 2. 0, 2, 2, Pan5. L*ookk for puttems within the der sequenoo: x* The sequenes II, 2, D, 2, 1, docan't immodiately suggest simple arithmetic goomatric progression. Let's
altemnating
I'n-
positions Odd positions (1st, 5th, 7ta:: O. 1. This sequence appears
son-
decreasing (0) 0, I, 3) Even position (2nd, 4th, 6th, 88): 2, 2, 3, This sequenes starts with two 25, then 3. A plausible contonation would be another Sorming the pattem (2. 2, 3),n6. *DDttrmiine the requieed number dotc. Based the altemoting pattem hypothesis, the 8th systbol (an even position) should
moother
mathematical operation nins. *Concianion** to Giver the presence ofs's (therffrr) and the use of logicil symbols, symbol that represents resson cause in- (bocause) fits well in this context, is complement the t'ei" (therffone) by providing the reason or cause for the conclusion inniiiv Final Answer [B)
the puttem (2. 2, 3, 3) mesning ahould
dots in7.
ve "'vallutte the options. Cheek the mumber of does each of the given options in A. :
logical complete the sequence with
doetaa * U. :3 dots'n * C. 4" dosii. * D. 'xx dotsing. **Select the correct option;**
symbol
has the required dotslu9. **Coufinn pattern." ** If complete dat sequence is 2, 0, 3, 2,
option
pantras for (U, O, 2) and even (2, 2, 3,
positions hold innAneer The final nnswer is (1))
Basage Caption
image puzzle soquence symbols arranged
I borizuntal row within individual square buses. Below this sequence.
To salve the purzle, let's analyse the sequerce of symbols and iekenify underlying pattem/zinl. **Sequrnce Analysis* + (Equal Sign/**, Rupresents equality.n- in 4*% (Percent Sign)**; Represents natio proportion ** (Appresimtelly Fquil Sign)**: Indiestes approsimation in (Division
there are four miltiple choice oprions lnbeled A, B, C, and D. each with I
Ia distinct symbel, ninl. **Symbel Sequence Boxes**/y *n There seven square boces aliiged hurizontally''s 'n- Each
contains single
symbol from left right'n The first box contains in equil sign
Sign with Dtt: Represents division. in **| (Ixchmattion ** Often used to denute facturial in mathematics or emphisis in language in- 44,
(ha in 2. The second box cumsins a percent sign (e 3. The third
box cumaain appensimately equil sign Whe 1n 4. The fourth box
(Therefone Symbbll** Indicstos conclusion in. ** (Collen)* Used to introduce list explamation innn.. "*deetifyingg ing Pattern **op sequence tramsiions from muthermtical symbols to logical and punctuation symbols 'n- Afler the "therefore'" symbal (6), which signifiss conclasion, the colon (1) used introduce explanation reason Following this logical progression. the symbal should lasically represent "bucuuee"" i which denoted (Recuss Symbolj* Inie3, "FFallating Options" hin *ppion
dor above and below the slash -1 The fifth box contains an exclamation mark lay 6. The sixth box cunmains symbul of three dots artinged in a triangalar formation, with two dots the botiom and one tep "'oi" (silo known seventh righth bux is marked with a questim mark ("9))) minn (992 MMlliple
division sign
""terefune""" (n
contains colon tn The
Choose Sppiins"*: Below the sequence. there are four options labeled A, C. Dan "Oppion s* hashingg number sign symbol Mi"" in "OOpiion gaa symbol
(A)**; laahhag or number sign, unrehted to the patiemm in- "ppiio D (:-y*+. Combines the oolon with the "'ecmmse" symbol, fining the logical continaation **Option C (+)** Plu sign, unrelated
vertically aligned
cursent logical flow Opption D (=) Multiplication sign, also unrelsedd the curnent inie "Conclusinn" "*in- *Option B (:-)++ best fiss the established putter introducing the '"because!" symbol which logically follows the "thoreffone" symbel and colon. in'n "ncweer" Vibseed
dots with a color between them, similar to the tringular dol puttern (rott ** Option C* A plus sign 1"1 in "Opiion D=*, A miltiplication sign ( The task is likely ibeitify puttern relationship among symbols the sequence determine which the four symbols (A, B, C, or D) belongs in the eighth box marked with Ia question mark.
Figure 18: Solution examples generated by different models. Reference solution and outputs generated by GPT-4o, Qwen2.5VL-72B, Gemini-2.0-pro-exp-- and Doubao- w.w.-Viiioioo-oo------ Additionally, the image caption and solution from LLMs (o3-mini) are also illustrated.
31

---

Published as a conference paper at ICLR 2026
X
?
?
M
X
Question:
Question:
Question:
Proms given options, select most suitable one fill in the question mack, so that certain pattem is presented:
the giver uptions, choose moot suinble one to fill the question mark, so that åt presents coctinn regulriit:
Choose the must appropriate option from the given
in question mark as to present certain regularity
Hiat:
Hint:
Hint:
solve potteen transfamationn puzzle. ocserve how the lizes in each figuce change in each step, lines slanting to the left are rotating clockwise and increasing number, while lines slanting the right are also rotating clocksise but decreasing in number. Apply this patters to transition from the shird figure to the next. foousing the sumber and orientation tho lines to deternine correct configuration.
Think about the pertern formed by the intersecting shapes the first set figures. Nocice how the number of sides of the intersection increeses sequentially with sccond
Examine how the black squares within the grid move. Focus
puttern of movement
the inner and outer parts of the grid Note that in the inner 3x3 grid, except for the center square which remains fxod, the other black squared shift positiioss in specific direction sequence. In the outer grid, observe how the black squares stuft consistently in particular direction. Which option cominues this pattem?
each figure, Apply this pattorn set of figures to determins the shape of the intersection at the question mark. W Which option continees this sequence?
886, W3S
B: BX 260
C: C:
D:. 0:003 356
Question:
Question:
Question:
Among the following options, which one conforss puttern change the gives figure
From the given foor options, seleet the most appropriate fill question mark that a certain pattem formed:
Divide the six figures below into ewe categocies. making each category figures has common features arpattems
Hint:
Ilint:
Hint:
Consider how the slements of each figure are the left, figure combination of parts from the first two figures Specifically, the botteen half of the third figure identical to the bottoen half of the fiest figurs, and the top half as figore. Apply this pattern the sequence on the right. Ideatify the ports of liguree on the right first.
Focus counting specilie slapes
Consider the problem as one involving grouping of
composed Obsurve that in the
figure Pay attention
many of particular
and classibianion. I Focus on the characterittiss
geometric shape appears concistently scross the options. Remember may designed distrect you, concentrate on count the shapes thanschess.
provided Notice that all slapes divided distinct closed regions, While the number of these regions varies, more subtle criteion is play. Pay astention to the largest region each shape. Examine symmetry: exhibit symmetry?
elements like shading
identical the tup half of
reflective
S
A 2GS
X
H+
3D
?
?
B: 00S, 233
C002. 982
W
D: D: 330
Question:
Question:
Question:
Divide the sis Figures below into two calegorics,
From the four given options, choose the most
Choosec the most appropriate inte figure from the
making sure each category of liguros has its own
appropriate rill the question
four given options fill in the question murk, that certain pattern formed.
common features putters: Hint:
that presents certain patten:
Hint:
Hint:
Obsarve the given figures andray allection symmetry properties
Consider the arrangement and symmetry shupes
Consider the symmetry of the curves in the given problem for asis symmetry and check curves the options shure this samee property. Additionally observe whe the curves pres through the origin, au this could differemiitoor Focus these sitributes to identify the correct option.
figunes the tilk sce all symmetrical figuros, their
claments Focus on properties
and their composition In the first group. the shages exhibit axial symmerry and are composed letters, Antlyse letters both gi groups, noting the symmeiiiy and repetition. When examining the optices, identity the one that maintaise the same symmetrrcal properties compoctioon the exumplea given.
symmetry types are different. Try to divide the figures into categorics centrosymmety and axisymmetty help stentify regularity belwern figures. Through this classifiaation, we can find the attibution of each Gigute and get the corcet answer.
Figure 19: Examples of hint prompts. Hint prompts are provided to guide reasoning without revealing the final answer directly.
32

---

Published as a conference paper at ICLR 2026
?
From the four given options, select the most suitable one to fill in the place of the question mark, so that it presents certain regularity:
A
B
solve this problem, we need to identify the pallem in the given sequence of circles and upply it find the correct option for the question mark. 'n'nLet's analyze the sequence step by step/'nin1.. **iisst Row:**ln The first circle has a triangle pointing left, 'n The second circle has triangle pointing down. in - The third circle has a triangle pointing right 1n'm2. "Secoond Row:**'n - The first circle has square pointing up. In The second circle has square pointing down. ln We need to determine the orientation of the square in the third circle. Inin3, *PPttoom Recognition: *n'n In the first row, the triungles pointing squares are also pointing
cyclic order (left, down, right). IIn the second row, the cyclic order (up, down). 1n'n4, ** Applying the Pattern: * 'n squares continue the same cyclic pattern as triangles, next direction after down should be to the right. in'mS. **Checking the Options:** Opption A: The square is pointing up. im Option B: The square is pointing down.'n. in Option C: The square is pointing right. in -Option D: The square is pointing left. in'nSince the pattern suggests the square should point to the right, the correct option is: 'n'nnnswwr: Whoxed((C)
<think>> To solve this problem, we need to identify the pattern in the sequence of shapes. Let's analyze the given images step by step. 'n'nl. The first row of images shows sequenee of circles with triangles inside them, in The first circle has triangle pointing left.'n The second circle has triangle pointing down. 'n The third circle has triangle pointing right.'n The pattern the triangles alternating directions (left, down, right).'nia.. The second row images shows sequence of circles with squares inside them. In The first circle has square on the top right comer. in The second circle has a square on the bottom right corner. in The third circle has a square on the top left corner la - The puttern of the squares is alternating positions (top right, bottom right, top Inin3. We need to determine the position of the square in the third citcle of the second row.'n - The pattern alternates between right, bottom right, and lop left. 'n Therefore, the next position
The previous positions should be top left. 'n'nNow, let's look the options provided:'nn /n A: A (This option does match the pattern the squares.) 'n- B (This option does not match the pattern of squares.) y'n- (This option does match the pattern (This option matches the pattern of the squares, as the square on the top left corner)) Thus, the most suitable option to fill in the question mark is D. 'n </think= iannse-- D </answer>>
top right and bottom right.
squares.) ) In-
Figure 20: Comparison of model outputs before and after RL training stage for Qwen2.5-VL-7B.
33

---

Published as a conference paper at ICLR 2026
Among the options below, which can be obtained by flipping the given figure vertically:
QwenVL2.S. -E--st.-----ttsst
determine which option can obtained by flipping given figure vertically, we need follow these steps: 'n' Identify original figure. 'm2. Visualize the vertical flip original figure. 'n3. Compare the flipped figure the given
options. In'n start with the original figure: The original figure shown in the leftmost image (labeled ('D""). It has letters VDDI", V'A", and VKY arranged in specific pattern on the surface of dodecahedron. When we flip this figure vertically, the arrangement of the letters will change, Specifically, the letter VD! will move to the bottom, the letter V"A"" will move to the top, and the letter VKY will move to the right side. 'nw let's compare this with the given options: inin- Option A: The letters are arranged as (KW, VVV", VDD", In- Option B: The letters are arranged as V'D!", VVV", VKK". Option C: The letters are arranged as VDD", VVV", VKK".n Option D: The letters are arranged as (KY", VVV", VDDV" IninFrom the description of the vertical flip, we see that the correct arrangement after the flip should be VDDI,, \"V", I'K". This matches with Option B. Thherrooorr, the correct answer is: is:'n'nn is/oo//
<think> To determine which option be obtained by flipping given figure vertically, need to analyze spatial relationships
orientation of the within the figure, in'nl. **Identify the Original Figure**; original figure has the letters V'DV", \"K", VVV" arranged specific pattern on surface of polyhedron. The letter \"Di" the left side, VKY" on right side, ** Understand Vertical Flipping ** When we flip figure vertically, the top part figure will become the bottom part, and the bottom part will become the top part. The letters will also change their positions relative to each other. 'n'n3. *Analyee Each Option"*:ln **Option A In this option, the letter D'D!" is on the left side, VKY on the right side, and ("V)" is in the middle. This is the same as the original figure, but is not vertical flip. in **Option B**; this option, the letter WDI" is on the left side, VKY is on the right side, and ""V"" is in the middle. This is the same as the original figure, but it is not vertical flip. in **Option C**; In this option, the letter VDD" is on the left side, V"K" is on the right side, and \"VV is in the middle. This is the sume as the original figure, but is not verticul flip. ***Option D**; In this option, the letter \"D'" is on the left side, VKY is on the right side, and \"V\" in the middle. This the same the original figure, but it is vertical flip. inin4. *CConclusion** * The only option that shows the figure after vertical flip Option
middle, inin2.
</answer>
Figure 21: Comparison of model outputs before and after RL training stage for Qwen2.5-VL-7B.
34

---

Published as a conference paper at ICLR 2026
?
Choose most appropriate option from the given four options fill in question mark, so that presents certain regularity:
QwenV1 2.5.V1...-------1 2.5-778-nnstrrcct
To solve this problem, we need to identify the pattern or rule that governs the sequence of images in the top row and then apply that same rule to the correct image for the question mark. Ln'ne''s analyze the sequenee of images in the top row: iniat. The first image half-cirele. In2. The second image is lollipop. In3. The third image flag. 'm4. The fourth image fish. In5. fifth image an atom symbol. 'n'nNow, let's look images in the bottom lww:at'n:: An cream cone. on: half-circle. InC: A slice watermelon. 'nD; slice cake, 'n'n need to determine which these images fits the pallern established by the top row. Let's consider shapes
second image (lollipop) transformations: spiral shape.
first image (half-circle) simple. The third image (flag) wavy line. The fourth image (fish) has curved body. 'n- The fifth image (atom symbol) complex structure with multiple parts. 'n'n pattern be the transformation of basic shapes into complex forms. The half -iirele bocomes lollipop. which The atom symbol is complex with multiple parts, 'onnoooing the bottom row, the image that fits this pattern is the that transforms simple shape half-cirele (imuge B) matches this description corresponds to the simplest shape in the bottom 'n'n' Thereffore the correct answer Answer: (B)
spiral shape. The flag has wavy line, and the fish has curved body.
complex form. The the simplest shape in the top und
Qwen 2.5-7B-lnstrcc
<think>> To solve this problem, noed to identify the pattern in the sequence of images. Let's analyze the images step by step: 'n'nl. The first image is half-eircle.nn. The second image lollipop. ln3. The third image is flag. 'm4, The fourth image fish. in5. The fifth image flower. In6. The sixth image determine, 'n'nNow let's look at the options provided: u'nA: An ice cream cone. 'nB: A half- circle. InC: A watermelon slice. InD: A piece of cake. 'n'nWe need to identify which of these options fits the puttern established by the previous images. Let's analyze the pattern: 'nin- The first image half-cirele. In- The second image lollipup, which has circular shupe but with stick. in The third image is flag, which has a distinct shape with vertical and horizontal section. in- The fourth image is fish, which has distinct shape with tail and fins. in- The fifth image is flower, which has distinct shape with potals and a center center/'n'nTh The pattern seems to be that cach image has a distinct and rocognizable shapc. Now, let's look at the options: in'n- Option A (ice cream conc) has a distinct shape with cone and a scoop. in- Option (half-circle) is ulready seen the first image. ln- Option C (waternelon slice) has distinct shape with triangular section. 'n- Option D (piece of cake) has distinct shape with triangular section. in'nSince the pattern is that each image has distinct and recognizable shape, and the options A, C, and D have distinct shupes, we need to choose the one that fits the pattern consistently. The most consistent and distinct shape among the options is the cream cone. Thenhoreeore, the most appropriate option the question mark is: <answer> A <lanswer>
question mark, which we need to
Figure 22: Comparison of model outputs before and after RL training stage for Qwen2.5-VL-7B.
35

---

Published as a conference paper at ICLR 2026
From the four given options, choose the most suitable one to fill in the question mark, that it presents a certain pattern:
To solve this visual logical reasoning problem, let's analyze the patterns in the series. In'n1. Observing the first three shapes: in The first shape shows segment (clockwise) from upper position going left. 'n Thee second shape shows segment from the top center going left. 'n The third shape shows segments from the bottom left and top left going left. 'n'n2. The fourth shape shows segment from the upper left position going down. In'nNow, let's identify the progression: in- The segments appear subsoquent shape adds an additional segment or changes the position of existing segments. innnGiven this sequenee and the changes between shapes: in- The next in the series should logically follow the pattemn of adding changing segments around shape a clockwise dircction. 'ninInspecting the options: Option A shows shape where segments are from the upper left, with onc part of the right bottom, which looks like continuation of the previous pattern. \n'nHenee, option A fits the pattern best, An'snr:::::: Wboxed
moving around the shape clockwise manner. 'n- Each
InteraVI2 5.38B-RI
<think> 'nTo solve this problem, need to identify the pattern in the given sequence shapes and apply the next shape. 'ninl. The first shape line segment positioned the top-right corner of the circle, In2. The second shape has the line segment rotated 90 degrees counterclockwise to the top -left corner 1n3. The third shape has the line segment rotated 90 degrees counterclockwise the bottom -left corner The fourth shape has the segment rotated degrees
counterclockwise to the bottom-right corner The pattern here is a 90-degree counterclockwis rotation of the line segment in each subsequent shape. 'n'nNow, let's examine the options: 'n- Option A: The line segment the top-right corner. Option The line segment at top- corner Option C: The line segment at the bottom-left corner. in- Option D: The line segment is at the bottom- corner in in'nSince the pattern involves 90-degree counterclockwise rotation, the next position should be the top-right corner, which corresponds Option A. TnvnTeerefore correct answer D. wt/tthink""<<snsweer </answer>
Figure 23: Comparison of model outputs before and after RL training stage for Intern VL-2.5-38B.
36

---

Published as a conference paper at ICLR 2026
Divide the following six figures into two categories, ensuring each category figures hus its commen
cberactoristies or patterns. The correctly classified option is:
InternVLa 5-38B
classification of the given figures, let's analyze the patterns and characteristicy of each figure: 'n'nl. **Figure (1**; A four-sided polygon with concuve shupe the lower part having five vertices you include internal points where it turns, 'm2. **Figure 2 Another four-sided figure but with
determine
shape two sides, making it pentagon like consists of five vertices, In3, **Figure (3 (3**, ; A four sided polygon with bends. 1n4, ** *Fiigure 4:**; A six-sided polygon (hexagon- shape with sections leading to lotal vertices. 'm.. **Figure 55** Similar to Figure (2) four- sided figure with concave sides summing to five vertices, 'n6. **Figure (6**, * A four sided polygon but with vertices. in'mUsing the characteristics: in'-- ** Vertices Count**:'n 1), (3, (, @ have five vertices. (2), 4 huve five vertices well, but they look different. 'n'nSince there isn't It obvious way distinguish based solely vertices, try looking for pattern:'n'a ** Patteem of Concave and Convex sections** 'n Figures ( (5) and ( four-sided (quadrilateral) and have similar concavelconvex patterns that slightly complicate their but still make them consistent shape complexity. lnkn With this consideration mind, specific concave/convex putters: Four-sided shapes with somewhut complex putterns (44), (5) ()), Shapes differing in vertex count and distinct concave sections (1), (2, (3)mm hnmn Thus the classificattin is:'n *Option 8**, (1)36), 22.4)))ammm Answer Wboxed
part leading five vertices total when counting internal
section and protrusion,
totaling five
subtle
classify based quadrilateral liiee churacteristics with
<thinks solve this problem, need to identify common characteristics or patterms among figures. in'ml. Figure 1: shupe with un "LL turn curve. 'n2. Figure A shape with curve line. 1n4. Figure A shape with an
V"S"" turn tnmn./n.. Figure 3: shape with an \"L!" turn und straight turn and a curve. in5. Figure 5: shape with an '"LA" turn and straight line. Figure 6; shape have V"L" turn curve. Figures
curve and
turn
can observe following: in- Figures 1, 5, have curve and VSS"" tumn./nnn Therrffor,,,,, the common churacteristic for first category (13344) V"LA"
curve, and for second category (2x5 and VSV Thus, the classification is:AnA: (1344, ( ins:think--o </answer>
Figure 24: Comparison of model outputs before and after RI. training stage for Intern VI.-2.5-38B.
37

---

Published as a conference paper at ICLR 2026
?
Which of the following options conforms the given pattern change in the figures:
InternVL2 5-38B
To determine which option conforms to the given pattern of change in the figures, let's analyze the sequence step step. 'n'n **First Figure:**in The figure contains square from inin- **cono Figure:**'n The square rotated slightly clockwise, in'n **Third Figure:**in The square cut rotated slightly more clockwise. inin- Fourth Figure:**nn The square cut out is rotated again, continuing the clockwise rolation pallern each time, in'n'To identify the next logical transformation: in- **iith Figure:** ** in Following the established pattern, the square cul oul should continue to rotale cluckwise to the next logical
appeurs
position'''nNNw, let's evaluate cach of the options: in'n- **Option A: This figure retains the unrotated square ** *Optioon B:** The square following, additional rotations on each step. This scems correct for continued rotation pattern. 'n- **Option C:** This figure's square cut out matches no existing rolations patterm, and follows diffirentt path rotation. 'n- **Option D The square cut out is rotated very different from the sequence. 'n'nBy verifying the patterns provided and comparing each option, Option B fits the rotation pattern exactly as continued clockwise, In'nAnswer: \boxxed(B)
Therefore, it does follow the pattern. In- the same way the previous figures
Intern 5-38B-RL
<think> 'nToo solve this problem, we need to identify the pattern of change the given sequence of figures. 1n'nl. The first figure shows an octagon with a right angle inside it In2. The second figure shows the same octagon with the right angle rolated 45 degrees clockwise, 'n3. The third figure shows the right angle rotated another 45 degrees clockwise. 1n4. The fourth figure shows the right angle rotated another 45 degrees clockwise. In5. The fifth figure shows right angle rotated another 45 degrees clockwise. la'aFrom inside the octagon is rotated 45 degrees clockwise in each subsequent figure. Now, we need to identify which of the given options matches this pattern. 'n'nOption A: The right angle is in the same position as the first figure.'n 'nOptionn B: The right angle is rotated 45 degrees clockwise from the first figure,\nOption \nOpionn C: The right angle is rotated 90 degrees clockwise from the first AnOption D: The right angle rotated
above observations, we can that the right angle
degrees clockwise from the first -degree clockwise rolation in each step, the next figure should have right angle rotatod 135 degrees clockwise from first figure. In'n Therefore, the vorrect option is /.nn/////nnn/a www/thinnknnnnnssne <ansserr>> D <janswer>>
figure, 'n'nSince pattern involves
Figure 25: Comparison of model outputs before and after RL training stage for Intern VL-2.5-38B.
38


Reviews:
[{"review_id": "DHSNccbNI1", "reviewer": "Reviewer_Npe8", "summary": "The paper introduces VisuLogic, a benchmark for evaluating visual reasoning in MLLMs without relying on textual shortcuts. It contains 1000 human-verfied logic puzzles across six categories [ quantative, spatial, positional, attribute, sytlistic and other]. The examples in this benchmark require extensive visual centric pattern reasoning. They also did extensive evaluations on multiple models on this benchmark.", "questions": "- Humans score 51.4% suggesting it's a very hard benchmark even for humans. It would be helpful if authors could report per per-category score performance of humans. Is it the same as MLLMs?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Addresses a gap in visual evaluation on MLLM, so that the model must look at images instead of language reasoning for solving some task.\n- Clear data collection, data cleaning pipeline.\n- The paper is also easy to read, though as a human, the questions are more of IQ test questions, hard and require a lot of thinking to solve.\n- Evaluates 31 models, including open and closed MLLM. Extensive evaluation.", "weaknesses": "- The paper says the models fail at those task; however, it's unclear whether the low performance comes from visual encoder limitation [ trying different vision foundation models in the same architecture], alignment of MLLM, or prompting limitations.\n- Models may help if in-context prompting is given. Ablation Study of such is needed. \n- A more broader motivation of where these puzzles will be needed in real life to solve would help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Z9NXPionkC", "reviewer": "Reviewer_Fnrw", "summary": "This work presents a novel benchmark, VisuLogic, to examine how and where MLLMs fail at abstract visual reasoning (AVR). VisuLogic includes a 1000-instance human-verified challenges in terms of multiple-choice RAVEN-style abstract pattern recognition questions. Each question is also equipped with a non-revealing hint that provides additional solving cues. VisuLogic’s leaderboard shows that most of the state-of-the-art MLLMs do not surpass 30%, only slightly over the random chance of 25%, while the human baseline is at around 51%. This large gap indicates that VisuLogic possesses a genuine challenge to existing MLLMs.", "questions": "Please find my main concern in the Weaknesses section. \n\nAs a minor suggestion, I believe it’d be more helpful to at least add a reference link to Table 8 within the caption of Table 1 - readers would love to see how the baseline models compare ‘with Hint vs. without Hint’ across all the individual categories in detail, rather than just seeing the ‘Hint’ averages under a single aggregated column at Table 1 in its current state.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "I applaud the authors for presenting a highly comprehensive work. I generally could not find anything to nitpick about when it comes to the design of the benchmark and the documentation of the experiments. Moreover, I greatly appreciate the authors’ extra effort on shortcut prevention when it comes to curating the Hints in VisuLogic, showing they pay extra attention on benchmark data integrity and ensure that the hints and cues do not give away the solutions from the get-go, according to Appendix C. Such a level of meticulousness is a rare sight.", "weaknesses": "I only have one thing to worry about regarding the data quality of VisuLogic. Judging from Table 8, it seems to me that even with hints(solution cues) given, several categories (esp. Spatiality and Style) still pose great difficulty to human annotators. Could this indicate that **a decent portion of VisuLogic’s current instances, under such ‘difficult’ categories, are indeed unsolvable to start with**? If so, we would risk seeing inflated benchmarking performances and obtaining non-robust models in return.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jHlMfHty8b", "reviewer": "Reviewer_igUC", "summary": "This paper introduces VisuLogic, a vision-language reasoning benchmark designed to minimize text-based shortcuts and emphasize genuine visual reasoning. The benchmark includes 1,000 human-verified problems across six categories. The authors evaluate a range of open and proprietary multimodal large language models (MLLMs), revealing significant limitations in their ability to perform vision-centric reasoning. While humans achieve around 51% accuracy, most models remain below 30%, underscoring a clear gap between current MLLMs and human reasoning ability.", "questions": "1. The reported 51.4% average human performance appears unexpectedly low. Could the authors clarify the reasons behind this result? Even with hints, average human accuracy rises to 83.6%, but substantial gaps remain in certain categories, such as Spatiality (68.5% with hints). Could the authors provide a problem difficulty analysis, particularly for the Spatiality category? Are there problems that no human solved, or conversely, did any annotator solve nearly all assigned problems?\n2. Since the dataset was scraped from the internet, how do the authors ensure that these samples are not already included in the training data of existing MLLMs?\n3. The paper does not describe how the Reference Solutions were obtained. Were these available in the scraped material, or were they manually created by the authors?\n4. The hints were generated by GPT-4o based on the Reference Solutions. Were these hints verified by human annotators to ensure they do not reveal the correct answer directly?\n5. The paper identifies gaps in MLLMs’ visual reasoning. Could the authors elaborate on how these insights might guide future model design or training strategies?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper addresses an important and timely problem - evaluating whether MLLMs can solve genuinely vision-based reasoning tasks where textual descriptions alone are insufficient. This topic is clearly relevant to the ICLR community.\n2. The VisuLogic benchmark comprises 1,000 human-verified samples across six task categories, providing a well-structured and challenging evaluation setting. The tasks are demanding not only for MLLMs but also for humans, yet the inclusion of hints demonstrates that they remain solvable and meaningful.\n3. The inclusion of problem hints is particularly valuable. They illustrate both the intrinsic difficulty of the tasks (as seen from improved human performance with hints) and offer a complementary lens for analyzing model reasoning behavior\n4. The evaluation is thorough, encompassing a diverse set of both open-source and proprietary models.\n5. The paper is clearly written, well-organized, and easy to follow.", "weaknesses": "1. (Section 3.1, Data Collection) The benchmark is collected from various internet sources via a scraping pipeline. While the authors state that they verified licenses and regulations, the paper should include a list of data sources for each problem. This is important to (1) properly credit original authors, (2) verify licensing and usage compliance, and (3) allow others to check whether the benchmark overlaps with data used to train evaluated MLLMs. Without this information, reproducibility and ethical transparency are limited.\n2. (Section 3.1, Quality Control) The authors mention conducting a “thorough human-led review,” but methodological details are insufficient. Appendix C.1 indicates that five annotators spent two hours each reviewing data - an effort that appears limited given the dataset’s size and difficulty. It remains unclear how many annotators reviewed each problem, how disagreements were resolved, and what the inter-annotator agreement was during the “Manual Inspection” phase.\n3. The benchmark aims to address the lack of vision-focused reasoning evaluations for MLLMs. However, this issue has been recognized in prior research. The importance of emphasizing the visual component was highlighted as early as [Goyal, 2017]. Several vision-centric benchmarks already exist, including Raven’s Progressive Matrices [Barrett, 2018; Zhang, 2019], Bongard Problems [Małkiński, 2025; Wüst, 2025], and ARC [Chollet, 2019; Moskvichev, 2023]. Spatial reasoning was studied in SPACE [Ramakrishnan, 2025]. MMMU-Pro [Yue, 2024] explicitly eliminates text-only solutions to MMMU. As a result, the novelty of VisuLogic is limited.\n4. Consequently, the paper shows that current MLLMs struggle with visual reasoning, but this limitation has already been observed in earlier work (including several cited by the authors). Consequently, the paper’s impact is somewhat limited by reiterating well-known weaknesses rather than uncovering a new research direction.\n\nBarrett, David, et al. \"Measuring abstract reasoning in neural networks.\" International conference on machine learning. 2018.\n\nChollet, François. \"On the measure of intelligence.\" arXiv:1911.01547. 2019.\n\nGoyal, Yash, et al. \"Making the V in VQA matter: Elevating the role of image understanding in visual question answering.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\nMałkiński, Mikołaj, et al. \"Reasoning limitations of multimodal large language models. A case study of bongard problems.\" International conference on machine learning. 2025.\n\nMoskvichev, Arseny, et al. \"The ConceptARC benchmark: Evaluating understanding and generalization in the ARC domain.\" Transactions on machine learning research. 2023.\n\nRamakrishnan, Santhosh Kumar, et al. \"Does spatial cognition emerge in frontier models?\" International conference on learning representations. 2025.\n\nWüst, Antonia, et al. \"Bongard in Wonderland: Visual puzzles that still make AI go mad?\" International conference on machine learning. 2025.\n\nYue, Xiang, et al. \"MMMU-Pro: A more robust multi-discipline multimodal understanding benchmark.\" Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics. 2025.\n\nZhang, Chi, et al. \"RAVEN: A dataset for relational and analogical visual reasoning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DEnR48Q07A", "reviewer": "Reviewer_7QFv", "summary": "This paper proposes VisuLogic, a 1,000-item benchmark purpose-built to evaluate vision-centric reasoning in multimodal large language models (MLLMs) without permitting language-only shortcuts. The dataset spans six carefully annotated categories—Quantitative (35.3%), Spatial (23.1%), Positional (13.6%), Attribute (8.2%), Stylistic (9.0%), and Other (10.8%)—and is curated via an automated pipeline (web fetching, cleaning, structuring) followed by duplicate filtering (text overlap and pHash) and full human verification. \n\nThe authors argue that many prior evaluations (e.g., math-with-diagrams or caption-to-LLM pipelines) conflate vision with language reasoning; they empirically validate this by showing that text-only LLMs fed detailed GPT-4o image captions perform near random (e.g., Qwen2.5-72B 28.0%, DeepSeek-R1 26.6%) on VisuLogic, indicating that genuine visual cues are not recoverable through textual description alone. Comprehensive experiments across 31 models reveal large gaps to humans: the random baseline is 24.9%, humans achieve 51.4% (83.6% with hints), while strong MLLMs—OpenAI-o3 (29.5%), GPT-4o (26.3%), Gemini-2.0-Pro (28.0%), InternVL3-78B (27.7%)—cluster around 25–30%. Chain-of-thought prompting offers negligible gains for visual reasoning, suggesting current CoT training is misaligned with multimodal needs. Hint prompts deliver modest improvements (typically +3–10 points) yet still leave models far from human performance, reflecting brittle and inconsistent reasoning procedures. \n\nFine-grained analyses show distinctive weaknesses: LLMs struggle most on Spatial tasks; MLLMs are particularly poor on Stylistic reasoning (often worse than random); humans maintain comparatively low error on Positional reasoning, underscoring a human–model cognitive gap in interpreting spatial relations and transformations. Scaling within families helps only slightly (e.g., InternVL2.5 38B→78B), and closed-source models do not markedly outperform open-source counterparts. The authors further explore reinforcement learning with RLOO using format and accuracy rewards, observing improvements over supervised fine-tuning (e.g., InternVL2.5-38B-RL to 31.1%), hinting that RL may better shape stepwise visual reasoning. \n\nOverall, the paper contributes a carefully controlled benchmark that suppresses text-only shortcuts, a transparent curation and taxonomy, multi-prompt evaluation protocols (non-CoT, CoT, hints), and diagnostic analyses that map where current MLLMs fail and how hints/RL partially mitigate those failures; code and data will be open-sourced to facilitate reproducibility and future work.", "questions": "1. Contamination: Can you run a near‑duplicate search against common pretraining data and provide detailed source distributions and licensing to assess training leakage and bias?\n2. Hints: Will you ablate hint specificity/strength and add category‑specific templates (e.g., face adjacency lists for folding) to separate perception vs. reasoning deficits?\n3. Human baselines: Can you add confidence intervals and per‑category variance, recruit a more diverse participant pool, and report item ambiguity checks (e.g., inter‑annotator agreement)?\n4. Reliability: Will you provide item difficulty/discrimination (e.g., IRT), prune ambiguous/low‑discrimination items?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "VisuLogic contributes something genuinely new by testing models on vision-first reasoning instead of letting them “cheat” via text. It deliberately blocks caption-only shortcuts and evaluates six well-defined reasoning types (quantity, spatial, positional, attribute, stylistic, other). This taxonomy is broader and more structured than many prior datasets. The dataset is built with a transparent pipeline (crawl → clean → structure → pHash dedup → human checks), answers are evenly balanced across choices, and the evaluation covers 31 models under consistent settings (No CoT, CoT, and hints), with per-category analyses and instructive hint studies. The paper shares prompts, hyperparameters, and even RL baselines (using RLOO), which makes the work reproducible and goes beyond just releasing a benchmark. Writing and figures are clear. Most importantly, results expose a big gap: top models are near chance (about 25–30%), while humans reach 51.4%, and 83.6% with hints. This shows VisuLogic is a strong diagnostic target for advancing multimodal reasoning methods (e.g., multimodal CoT, PRM/RFT, RL-style test-time compute).", "weaknesses": "Some tasks may still be solvable by pattern matching rather than true multi-step reasoning. The paper doesn’t formally map each item to “atomic” visual operations (like rotate by k degrees, reflect, count, fold adjacency), nor does it run ablations that remove superficial cues to force deeper reasoning. Data provenance is under-detailed: there’s no near-duplicate audit against common pretraining datasets or a breakdown of sources, which weakens claims that tasks are “new” to models. The human study uses a relatively uniform STEM cohort and reports limited stats; hints derived from solutions might subtly bias phrasing. Methodologically, the caption→LLM control depends on GPT‑4o’s caption quality, but there’s no human-captioned upper bound; CoT conclusions come from text‑centric prompts without multimodal/process variants; and the RL setup mainly rewards final answers/format, with little process supervision or overfitting checks. Finally, statistical reporting is light (few confidence intervals or significance tests), and there’s no item difficulty/discrimination analysis—even though some categories are near chance—so reliability and calibration remain open questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)