Zebra-CoT: A Dataset for Interleaved Vision-Language Reasoning
Authors: Ang Li
Charles L. Wang
Deqing Fu
Kaiyu Yue
Zikui Cai
Wang Bill Zhu
Ollie Liu
Peng Guo
Willie Neiswanger
Furong Huang
Tom Goldstein
Micah Goldblum
Venue: ICLR 2026 Poster (2026)
Keywords: visual chain of thought
interleaved text and image generation
multimodal reasoning
Primary area: datasets and benchmarks

Abstract: Humans often rely on visual aids, such as diagrams or sketches, when tackling complex problems. Teaching multimodal models to adopt similar strategies, a process known as Visual Chain of Thought (visual CoT), is much more difficult. The main challenges are: (1) weak performance of off-the-shelf visual CoT, which hinders reinforcement learning, and (2) the lack of high-quality visual CoT training data. We introduce **Zebra-CoT** a diverse large-scale interleaved text-image reasoning dataset with 182,384 reasoning traces across 18 domains with over 50 distinct tasks.
This dataset is specifically designed to train models to natively perform visual CoT. 
We emphasize four categories of tasks where sketching or visual reasoning is especially natural, spanning (a) *scientific questions* such as geometry, physics, and algorithms; 
(b) *2D visual reasoning tasks* like visual search and jigsaw puzzles; 
(c) *3D reasoning tasks* including 3D multi-hop inference, embodied and robot planning; 
and (d) *visual logic problems and strategic games* like chess. 
Fine-tuning Anole‑7B model on Zebra-CoT yields a +12\% improvement in our test‑set accuracy and up to +13\% performance gains on standard VLM benchmarks. 
Similarly, fine-tuning Bagel‑7B produces models capable of generating high-quality interleaved visual reasoning chains, underscoring Zebra-CoT's effectiveness in advancing multimodal reasoning.

Published as a conference paper at ICLR 2026
ZEBRA-COT: A DATASET FOR INTERLEAVED VISION-LANGUAGE REASONING
Ang Li
Charles L. Wang" Deging Fu
Kaiyu Yue
Zikui Cai
Wang Bill Zhu *.
Ollie Liu *.
Peng Guo S+
Willie Neiswanger
Furong Huang
Tom Goldstein
Micah Goldblum"
Columbia University
New York University
Uniieessity of Southern California
University of Maryland
Equal contribution
ABSTRACT
Humans often rely on visual aids. such as diagrams or sketches, when tackling complex problems. Teaching multimodal models to adopt similar strategies, a process known as Visual Chain of Thought (visual CoTT), is much more difficult. The main challenges are: (1) weak performance of off-the-sheif visual CoT, which hinders reinforcement learning, and (2) the lack of high-quality visual CoT train- ing data. We introduce ZEBRA-COT: a diverse large-scale interleaved text-image reasoning dataset with 182,384 reasoning traces across 18 domains with over 50 distinct tasks. This dataset is specifically designed to train models to natively per- form visual COT. We emphasize four categories of tasks where sketching or visual reasoning is especially natural, spanning (a) scientific questions such as geometry, physics, and algorithms: (b) 21) visual reasoning tasks like visual search and jig- saw puzzles; (c) 3D reasoning tasks including 3D multi-hop inference, embodied and robot planning; and (d) visual logic problems and strategic games like chess. Fine-tuning Anole-7B model on ZEBRA-COT yields + +12% improvement our test-set accuracy and up to +13% performance gains on standard VLM bench- marks. Similarly, fine-tuning Bagel-7B produces models capable of generating high-quality interleaved visual reasoning chains, underscoring ZEBRA-COT's ef- fectiveness in advancing multimodal reasoning.
Datasets: muulimmod--rrassnnnnaaa///iiii Cab
Anole-Zebra-Cof Model: mwl.inodl--ccoonnn--b//ooeeeeeee
Bagel-Zebra-Cot Model: muttinnddarraaannnnnn--aaaeeeeee
GitHub Repository: ghtuacoowmmniiinddddoeeeiiii/aan
INTRODUCTION
Human cognition naturally integrates multimodal thought processes when solving complex prob- lems. For example, a high school student sketches diagrams to solve geometry or physics problems, an engineer creates diagrams to design and debug workflows, and a data scientist generates plots to better understand data. These visual aids are central to effective problem solving. While recent vision-language models (VLMs) have shown strong performance on multimodal tasks like visual question answering. their reasoning traces remain predominantly textual. Enabling models to explic- itly reason in the visual space, Visual Chain of Thought (visual CoT), remains fundamental open challenge. Unlocking visual CoT may improve reasoning performance in domains where visual in- tuition is relevant and may make the reasoning patterns expressed by models more interpretable to humans.
Recent advances frontier multimodal models (Team et al. 2023 Hurst et al. 2024 Bai et al. 2025 2025) have made visual CoT feasible primarily through agentic pipelines that leverage external tools (e.g.. Python such generating
OpenAl 2025a: Team 2024 Chern et al.] 2024; Sun et al. 2024 Deng al.
functions, expert vision models) for visual programming (Surís et al.
2023).
sketches for geometry, algorithms, and spatial reasoning tasks (Hu et al.
2024 OpenAl 20256), or
bounding boxes for fine-grained visual tasks (Shao et al. 2024a: Wu and Xie 2024 Zheng al.

---

Published as a conference paper at ICLR 2026
Distribution Images Per Trace
Visual Logic& Strategic Games
Scientific Reasoning
Viad
0.8
Wiaad Resertie
AscAdi LAS
Chastey 26%
Checkr 1.%%
Conpetthe Prggaening
Vauel Cratege
1122%
Geeadry 0x.s
0.6
Proportion
More 11.0%
RPM
5.
0.4
Visual Reasoning
Zebra-CoT
Visual
Reasoning
Fndaadee Bay OMgp:
Plassing 18%
Viaad Sturch Reiriton Rasseiig 5.55
0.0
Voad Starch Geseral vgga 4.%
Number of Reasoning Images
Figure 1: We curate large-scale multimodal dataset by sourcing and cleaning raw traces from real- world domains, and generating synthetic examples using templated reasoning filled by VLMs. ZEBRA-COT comprises 4 major categories and 18 subcategories, encompassing over 182K in- stances in total. detailed breakdown of the data statistics appears Table
Scientific Reasoning
2D Visual Reasoning
3D Visual Reasoning
Games Puzzles
Guantion Teurs
ucsccales 302 shows Desveruce of
Question: What aequence
groing
as IFG
Thought 7: unaharr tningls 822 detemmew helgen will abttack noin netion 7 buse XXC
Thought to idedlyy the animnall
Thought 1:gna
reed esante fhom more aleeyy
Thought b5 Skac
This wwwaabb noos
Thaughe 2 The alggran Phow attado withi 2 X12 Shase A X12 berowss with XT m2 17 base X2 the ataade 201 busca ze. Thor Ii = Riz X212 = 2012 ib.
Thought rounced shows rot wocky deeaddest whoud lege Upooi shaaps
Thoght conadde conter dana loiigg Slanpp to
Thauggh
Figure 2: Visual CoT helps answer complex visual reasoning questions, as illustrated by examples from ZEBRA-COT.
2025
An emerging possibility is innate visual reasoning, where models directly generate explicit
visual tokens during their thinking process (Li et al., 2023 Chern al.) 2025 Xu al. 2025b).
However, current VLMs with interleaved text and image generation capabilities (Team 2024 Chern al. 2024) either fail generate useful visual aids for reasoning or are not inherently trained for such multimodal generation during the reasoning process (Deng et al. 2025), making reinforcement learning approaches to reasoning infeasible. Li et al. (2025) demonstrate visual CoT synthetic mazes by training specialist models, but remain far from foundation models capable of general high-quality visual CoT, largely due to the lack of large-scale diverse interleaved text and image reasoning training datasets.
To support the development of next generation vision language models that can explicitly reason with both text and visual modalities, we present ZEBRA-COT. high quality dataset of interleaved text and image reasoning traces. Our dataset covers four main categories: scientific questions, 2D visual reasoning, 3D visual reasoning, and visual logic and strategic games, each containing multiple
2

---

Published as a conference paper at ICLR 2026
Dataset
Primary Task
CoT Modality
Suitability for visual CoT Training
GQA
Compositional visual QA
Text
No visual CaT
ScienceQA
Multimodal QA
Text
CaT
CLEVVR
Synthetic compositional visual QA
Text
No visual CaT
VCR
Visual commonserse QA with rationale
Texx
No visual CoT
VideoCoT
Video QA
Text
No visual CoT
EgoCOT
Embodied planning
Text
No CoT
LLaVA-CoT
Multimodal reasoning QA
Text
No CoT
MAmmoTH-VL
Large scale multimodal instruction tuning
Text
No visual CoTT
MM-Verify
Multimodal reasoning with verification
Text
No visual CoT
RI-Onevision
A SPT and RL multimodial reasoning dataset
Text
No visual CaT
CuT
Visual QA with CoT
Image,
Limited to visual tasks
MM PMAAA
Physics visual CoT
Image, Text
Physies data only. not open sourced
COT VLA
Robotics visual Col
Image. Action
No text reasoning
OmniCorpus
B-level interleaved corpus
None
Noisy pretraining data without Col
MINT-IT
web-scale interleaved
Noisy pretraining
Col
ZEBRA-COT
Diverse and high quality visual CoT
Image. TTet
Diverse interleaved vision- Maaionn CoTl
Table I: ZEBRA-COT introduces broader set of high quality visual CoT traces compared with prior datasets and pipelines.
subdomains and task types, as exemplified in Figure To the best of our knowledge, ZEBRA-COTI is first dataset to provide diverse and logically coherent multimodal reasoning traces across such a wide range of domains. Unlike prior large-scale interleaved datasets that are primarily composed of web-scraped image-text pairs with weak semantic alignment and no explicit reasoning structure (Li et al. 2024b Awadalla et al. 2024 Zhu et al. 2023), LEBRA-COT is carefully curated as training resource in the spirit of hugh-quality text-based reasoning datasets. At the same time, compared to the only existing open-source interleaved text visual reasoning dataset
aware of, VISUAL- COT (Shao et al. 2024a), which focuses on a single task of visual search, ZEBRA-COG introduces a much broader and more diverse of tasks with richer reasoning trajectories. We provide detailed comparison with other datasets below in T
Our contributions summarized follows:
1. We release ZEBRA-COR -OT, a high quality and diverse dataset with interleaved text and visual CoT that contains 182,384 samples for training models to natively perform visual CoT for problem solving. Details regarding the dataset are shown in Section 3
2. We evaluate three frontier LLMs, including GPT-5, Claude Sonnet 4, and Gemini 2.5 Pro, on the tasks in ZEBRA-COT in Section 4 Despite their advanced multimodal reasoning capabilities, these models perform poorly on those challenging tasks, with an average 31.51%. 1. Moreover, to demonstrate the effectiveness and value of visual CoT, construct scaffolding experiment that provides the first one or two multimodal CoT steps in context. Accuracy rises to 47.99% after one step (+16.48 pts) and 56.70% after two steps (+25.19 pts) overall, with gains of up to +43.77 pts in specific domains. These findings highlight the challenging nature of our dataset, the quality of our reasoning traces, and the value of visual COT.
3. After fine-tuning ANOLE-7B BB (Cher et al. 2024 on our in-distribution test set from 4.2% to 16.9%. When evaluating the resulting model on benchmarks requiring visual reasoning, our ANOLF-ZERRA-COT-TT model achieves aver- age improvement of 4.9% across seven challenging datasets, with a maximum gain of 13.1 % visual logic benchmark. as shown in Table
training set, improved the accuracy
4.
We fine-tune BAGEL-7B (Deng al. 2025). tively generate interleaved text and images on our dataset. After fine-tuning, the model is able to inherently generate high-quality visual CoT during its own reasoning process, making it well- suited for future RL training, as shown qualitatively in the examples in Figure and Appendix B
high-quality multimodal model that cannot
3

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Visual chain of thought. The community has predominantly been tackling visual CoT by using
visual programming to generate images (Surís et al.] 2023] Zhang et al. 2023 Mitra et al. 2024
Yang* et al. 2023; Wu and Xie 2024 Hu et al., 2024; Menon et al. 2024] OpenAl 2025b Zheng al. 2025). In particular, VISUAL SKETCHPAD (Hu et al. 2024) presents the most versatile open-
source visual reasoning agents among existing works, handling a wide range of tasks. Another line of work explores model-generatee images: for example, Rose al. (2023) uses diffusion model to bridge gaps in storytelling, and CCern et al. (2025) generates intermediate images to improve image generation tasks: Zhao et al. (2025) generates intermediate images as subgoal predictions and derives actions based on them for robotic planning: [i et al. (2025) and Xu et al.] (2025b) explore spatial reasoning tasks like mazes by visualizing each temporal step. However, these model-generated image approaches are mostly specialists, and developments are still primitive compared to visual programming methods that leverage external tools.
Visual reasoning datasets. Many multimodal visual reasoning datasets have been proposed, such as GQA (Hudson and Manning. 2019h, SCIENCEQA (Lu al. (2022), VIDEOCOT (Wang et al.,
2024c), EGOCOT (Mu et al.) 2023). ALAVA-COT (Xu et al. 2024), MAMMOTH-V -V. (Guo et al., 20241. ME-VERIFY (Sun et al. 2025), RI -ONEVISION Yang et al. 2025), CLEVR (Johnson et al. 2017), VCR (Zellers et al. 2019), although most focus on multi- multi-modality only in the in- put question, leaving the reasoning traces purely textual. Among them, VISUAL-COT (Shao et al.) 2024a) stands out as the only open-source dataset featuring interleaved text and image reasoning. MM-PHYQA (Anand et al.) 2024) on the other hand, introduces a paradigm for incorporating im- ages into the reasoning process for physics problems, though the dataset is not publicly available. Several vision-centric benchmarks (Fu et al. 2024b Hao et al. 2025a) present diverse and challeng-
ing tasks, but they lack annotated reasoning traces.
Large-scale corpora with interleaved text and images have become essential for pretraining VLMs with reasoning capabilities (Alayrac et al. 2022 Chen and Wang 2022; Sun et al. [2024] Wang et al. 22024b Hurst et al. [2024] Li et al. 2024 Bai et al. 2025 Team et al. 2025). However, in most existing interleaved text and image datasets MULTIMODAL C4 (Zhu et al. (023), OBELICS (Laurençon et al. 2023), OMNICORPUS (Li et al. 2024b), images are primarily used for recognition, captioning, or as supplementary context in bext-based reasoning, rather than serving as explicit visual aids that contribute meaningfully to the reasoning process. While MINT-IT (Awadalla et al., (024)) includes some scientific content from arXiv where images may aid reasoning. both the text traces and visual content are often noisy and not well-suited for post-training or fine-grained reasoming tasks. Instead, our ZEBRA-COT introduces a broader and higher-quality set of visual CoT examples, enabling effective training for visual reasoning.
Interleaved text and image datasets.
3 DATA CURATION DETAILS AND COMPOSITIONS
3.1 CURATING A DIVERSE AND HIGH QUALITY INTERLEAVED VISION AND LANGUAGE REASONING DATASET
A key challenge in training multimodal generation models to output visual CoT natively is the lack of datasets with strong logical coherence between text and visual modalities, and diverse categories of such visual COT. Existing interleaved datasets often fail to provide clear, meaningful connections that demonstrate when and why visual reasoning is necessary for problem-solving, while current visual CoT datasets are confined to few domains, limiting the model's ability to learn generalizable visual CoT capabilities when faced with out-of -iito---istibuttion problems.
To address these gaps, we developed a comprehensive data curation pipeline that bridges logical connections across modalities, as shown in Figure 5 For logical coherence across modalities, we leverage frontier vision-language models (Gemini-2.5 Pro) to enrich reasoning traces and ensure clear logical flow between textual reasoning and visual aids. For diversity, we combine real-world problems from multiple domains (mathematics, physics, chemistry, coding, chess, visual question answering, robotics) with synthetic examples generated through computer programming, simula- tion, and graphic rendering. This pipeline enabled us to curate over 182 K high-quality interleaved text and visual reasoning traces spanning four major categories: scientific reasoning, 2D visual rea- soning, 3D visual reasoning, and visual logic and strategic games. Unlike existing limited datasets
4

---

Published as a conference paper at ICLR 2026
that focus primarily visual search spatial reasoning, curated dataset provides the breadth and diversity necessary for training models that can generalize across domains. For details regard- ing data curation pipeline, please refer Appendix [A.2] In the following sections, provide a brief introduction to the tasks of each broad category. And for the details regarding the subcat- egory and domains, please refer Appendices A.3 to A.6 For prompt templates, please refer to Appendix F
3.2
SCIENTIFIC QUESTIONS
Geometry
B
othints Thought To better wouwin whe expecialy ManglellC lers deggonal RV our daggrmm Lot the interscction of segrentteeltzdd to AC can considered median stanglee BCO. ezhint
ethiiee Thought updated d'vgram shows the square ABCD intersecting point segment BM intericcss tringlle <oHHenno
Question eqpace ARC3)
AD conametess, and M
be she intossaction XC
idennty
ITT
M
Image
Reseoning Image
Visual reasoning is particularly valuable in STEM domains, it enables the visualization of abstract concepts such as auxiliary lines, free-body diagrams, and sketches, which clarify ideas that are hard describe in language and support step-by-step problem solving in ways that mirror human cog- nition. In ZEBRA-COT, this category spans subdomains including geometry, physics, chemistry, algorithmic problem solving, and graph problems. For geometry, physics, and chemistry, we lever- age openly licensed datasets and textbooks, using Gemini-2.5 (Comanici et al. (025) to denoise and parse them into clean, logically structured visual COT. For graph problems, we employ com- puter programs to generate images and text templates, which are then diversified using Gemini-2.5. For algorithmic problems, we use a GPT-4. agent built upon Hu et al. (2024) to produce detailed traces for sol solving competitive programming tasks. For details regarding all tasks in this domain, see Appendix A.3
3.3 2D VISUAL REASONING
Jigsaw
Question:
Thought orighal image undendland mixsing suctioe and the aurounsing detais. Thought 1: To complete image the originl source imge, vithenk
ethiil
smage cenoers prosidedd
of panzies
a of plenee matches the consent and posaton of the
mevcca maxing prices indicated by areas. Idorafy
anoas the original puzale.
Thought 3: Option contwins
pieces options
image neecied
prowaeed will comeetly complets alo puzzio
mixingg sections
Probiom
Reasoning Image
the fal maagl.s/bllkke.
In 2D visual reasoning, visual aids support the manipulation and interpretation of 2D visual in- formation, enabling tasks that involve spatial arrangement, pattern recognition, and fine-grained inspection. For this category, we include tasks such as visual search and visual jigsaw. For visual search, we adapt datasets from Shao et al. (2024a) and incorporate two types of visual aids: draw- ing bounding boxes and zooming into focal regions. We apply those visual CoT broadly across data categories, such as charts, documents, relations, and general VQA. For visual jigsaw tasks, we crop images from ImageNet (Deng al. 2009) to create puzzles with random number of missing pieces in diverse shapes. The visual Col is either iteratively filling in the pieces or reconstructing the original image directly. Further details provided in Appendix A.4
5

---

Published as a conference paper at ICLR 2026
Robot Planning
Question:
<think Thought move the robot's arm towards the bveed. </think>>
<chink> Thought 2: use the robot' arm grab bread </ttiink>
sthink> Thought move the Eread towerds table. <othinks>
descrine
sequence of robot actions from first imnege to the
goal atate (the image).
Problem Image 1
Reasoning Image
Reasoning Image
Reasoning Image
3.4 3D VISUAL REASONING
For 3D visual reasoning tasks, we focus on two domains: (1) embodied reasoning and robotic planning in the physical world, and (2) understanding 3D) transformations from different view- points. For the first domain, prior work has shown that generating visual predictions of the physical world and extracting inverse dynamics can improve performance in long-horizon decision-making in robotics (Zhao et al., 2025 Yang et al., 2024). To capture this, we reformulate the ALFRED benchmark (Shridhar et al.J 2020) into an image goal-conditioned planning task in which models generate detailed step- seep-by-stte stes plans to transition from an initial state to goal state. We also adapt RoboMIND (Wu al. 2024) for real-war weal-world robot planning, where models receive initial and goal images, along with descriptions of robot embodiment, and must produce precise high-level action plans. For reasoning about 3D transformations, we design multi-hop object counting tasks inspired by CLEVR (Johnson et al. (017), where scenes undergo sequential modifications, such adding or removing objects, requiring models to visually reason through each transformation step. For details, Appendix A.5
3D Multi Hop Object Counting
Question iiemou zeco
othinll-Tought Procecokig instrution to "Wlemow ("FRMM oijecte upon reaiiing the cureent Here
omhinko cthink-Thuught Thought 3: for Step dlwectiv showe "Remove FURPLE object. i, mis operation specilis that puple objees are to be removed the objects remaiinng the scena alter Step are v/dikkk>
CAPSUL ocjects Elmmmate
PYRAMI
gynamed Tais object
obacets Discar af CUBE
objects, Count how mony blue blue-colere objects
znangoment upaitco this removal dispcted in the following Image c/tinks
Probiom Image
Remoning Image
3.5
VISUAL LOGIC AND STRATEGIC GAMES
Maze
Question Decermine the stralest number mosements
Thought
cthrit-Thought Fallwwng the jountyy loads lotor partoular segment ouite leigthy, requing 41 moues clue the winding of Righligghthtttt
ongment sysimal pach
on from the minl grewn marbee to
witlous repcttionn starting green morker (7 stop banstions
the letter This
of 16 the
four oivections anound abstucled) Optiins. (A) 88, 99,
the
For visual logic puzzles (IQ matrices, Tetris, ciphers, ARC-AGI (Chollet et al. 2024)), previous VLMS tended to solve problems primarily using text reasoning. They first verbalize visual inputs

---

Published as a conference paper at ICLR 2026
Chees
<thinio Thought The intiis postion shonee Blook's king paterdaily ailtaaalle, and tnight on oi by White's bistop or gs, Black needs find a move that improves she posttio or clefenchs aqaiist posential theeats Lat'ss Tals
<iiick Though- kniggtt longer planeel still attacked by the queen ed and also blocks she queen's cielense of the promn Thit improvee p signilaadly. <ithinss
Question:
move for Black to clay?
queen pienedd
Nez
Knl
C: G-O
die candiaute broakss the pin kright.--thin
Problam Image?
Ressoning Image
into text, which causes information loss and makes visually salient patterns, such spatial re- lationships, difficult to capture. In contrast, humans solve these directly and efficiently via visual imagination and manipulation, even for babies who have not yet acquired language capabilities (Zhu jet al. 2020). To bridge the gap, we construct visual CoT traces that include explicit intermediate visual transformations to encourage models solve these problems visually. Similarly, for strategic games (chess, checkers, Connect Four), decision making typically involves searching and generat- ing counterfactual rollouts. While LLMs can simulate this by symbolizing board states into text, much of the spatial structure is lost, and rollouts in text space are difficult for problems with large visual information. Thus, we render those search and simulation steps into images so that models trained on this data can perform long-horizon planning in the visual space inherently. Finally, we generate a diverse suite of maze tasks and visual CoT traces that require a combination of capabil- ities, including high-level symbolic search and low-level perception. For details of those tasks, see Appendix A.6
ANALYSIS OF ZEBRA-COT AND THE VALUE OF VISUAL COT
Proprictary frontier models (GPT-5 (OpenAl 2025c), Gemini-2.5 Pro (Comanici et al.] 2025). Claude-4 Sonnet (Anthropic 2025) have achieved state-of- state-of-the-art performance on multimodal rea- soning benchmarks. Despite their advanced multimodal capabilities, we show that they struggle significantly with the tasks in ZEBRA-COil To explore these limitations and demonstrate the chal- lenging nature of our dataset alongside the effectiveness of visual reasoning traces, we design a scaf- folding experiment. Specifically, our dataset consists of structured reasoning chains: <question> + <text-reasoning-
<text-rreasonngg-11 + <answer>.
In the zero-shot setting, we provide models only with the <question: (containing both image and text). For scaffolding experiments, incrementally provide the first i multimodal reasoning steps as context:
www.:aaoooo-.ttttaaaaeeeeaooooao 2MT (k= = 2): <question>> + <text rtext--ressonnng-- <visual reea-rrinssnnnn1
1MT (k 1): <question> +
<text frext-reasoning nissonnnn--2> =
Importantly, most tasks in ZEBRA-COT require various multimodal reasoning steps (which can involve as many as 20 images) to reach the final answer. By providing only the first two steps as scaffolding, we ensure that models must still perform substantial reasoning to solve the task. We can safely assume that the provided steps serve guidance rather than revealing the solution. Since our dataset comprises diverse tasks, some of which extend beyond traditional QA formats (e.g., robotic planning and embodied CoT) that are not suitable for evaluation, we select the most challenging and representative examples for evaluation: graph questions for scientific reasoning, visual jigsaw for 2D spatial reasoning, multihop object counting for 3D reasoning, and maze/chess/tetris for visual logic and strategic games.
We plot the results for three evaluation settings across each task domain Figure 3 We observe that frontier models achieve poor zero-shot performance: GPT-5 reaches 41.98% accuracy, while Claude-4 Sonnet and Gemini-2.5 Pro achieve only 27.61% 11% and 24.93% respectively. However, with multimodal CoT scaffolding, observe substantial improvements: average accuracy across the three models increases to 47.9 9..9% (+16.48%) %) with one reasoning step and 56.70% (+25.19 % with two steps.
7

---

Published as a conference paper at ICLR 2026
Scaffolding With Multimodal CoT Prompting
Overall
Scientific
2D Visual
3D Visual
Logic & Games
80%
80%
Accurscy
60%
40%
60%
60%
80%
40%
40%
40%
40%
20%
20%
20%
20%
1MT
2MT
0%
1MT
2MT
0%
1MT
2MT
0%
Q
1MT
2MT
0%
1MT
2MT
GPT-5
Claude Sonnet 4
Gemini 2.5 Pro
Figure 3: Scaffolding experiment with frontier models. represents zero-shot question-only evalu- ation, 1MT denotes question with the first multimodal reasoning step provided, and 2MT indicates a question with the first two multimodal reasoning steps. We show that even frontier models with the best multimodal reasoning capabilities perform poorly overall on tasks in ZEBRA-COI. However, as we provide the first one or two multimodal steps to those models, the accuracy im- proves significantly.
Performance gains vary across task types, but generally improvement trend. Maze tasks show the most dramatic improvements, which jump from 52.59% to 76.60% (+24.01 %) and to 96.36% (+43.77%) on average, while challenging tasks such as graph reasoning improve from 3.92% to 22.03% (+18.11%) with two multimodal reasoning steps on average. Even tasks with higher baseline performance, such as multihop object counting (with an initial accuracy 59,40%) benefit from visual CoT, eventually reaching 67. 67.65% accuracy on average. Detailed statistics are shown Table R
To isolate the contribution of visual reasoning aids from text CoT in our traces, we conduct an ablation task where we remove all visual aids from the reasoning traces and retain only the textual steps. We observe that text-only CoT yields substantially smaller performance gains compared to full visual COT, and in some cases even degrades performance. This is expected: in our dataset, the visual and textual components are highly complementary. Many reasoning steps reference visual elements that, once removed, leave the text chain logically incomplete or incoherent. Model even requests for the missing visual aids that are referred in the text cot. These results indicate that the majority of the performance improvements stem from the visual reasoning steps, or the combined visual + text reasoning, rather than from textual CoT alone. The statisties for text only results shown here:
TRAINING MODELS ON ZEBRA-COT
Model
Visions
MathVista"
VisuLogic
EMMA
MMVP
Blink
Vstar
Anole with CoT prompting Anole-Zebra- (Ours)
13.80
22.80
8.50
12.80
10.00
26.46
23.60
16.45
25.30
21.80
15.02
15.33
31.25
27.20
Table 2: Overall performance (%) across eight datasets for the base Anole model with chain-of- thought prompting the same Anole model further trained on ZEBRA-COT. "We evaluate on the mini versions Math Vision and Math because interleaved generation is time consuming. A full breakdown of each evaluation set is presented in Appendix
Anole-Zebra-CoT., We fine-tune Anole (Chern et al. 2024) our dataset, which builds on Chamelcon (Team) 2024). using the codebase from Chern et al. (2025). We finetune the model fully end-to-end
node with 8 H200 GPUs for 12 hours, with learning rate of 1 10-5 cosine decay, a batch size of 8, and a max token length 12288. We train the model for 10k steps. To evaluate trained model,
the maximum generation length to 16384. After fine-tuning Anole on our ZEBRA-CA -CTT corpus, the accuracy increased from 4.2% to 16.9%, delivering a 4 times relative performance improvement and 12% gain in accuracy.
8

---

Published as a conference paper at ICLR 2026
Example Interleaved Visual Reasoning Chains Bagel-Zebra-CoT
Cenerated by
Question: Subtract all cylinders. red
<think>
<think>
Remove all the cylinders. </think>
Add red sphere. </think>
sphere. How many objects inft?
Final Answer: </answer>
Problem Image 1
Reasoning Image?
Reasoning Image 2
<think>> The problem asks for the number of missing small squares in 5x5 grid, We can see the complete 5x5 grid structure. To find the missing squares, we need to see the full pattem. The first image shows the complete grddl </think>>
<thinks Camparing the complete grid in the previous
Question: A big square is made from 25 small squares put together, A few of the small squares have beon lost. How many have been fost?
image with the given puzzle imagi,, we count number squares present the complete grid but missing in the puzzle. By counting the missing squares the complete grid, we find number of last squares. </thinka-
<answer>
Final Answer: 10
</answer>
Problem Image
Reasoning Image?
Reasoning Image 2
Figure 4: Example interleaved reasoning chains generated by finetuned on ZEBRA-COT. These traces demonstrate ZEBRA-COT's for instilling intrinsic visual reasoning capability complex multimodal models.
Bagel-7B model
Furthermore, evaluate seven challenging benchmarks that require visual reasoning, including MathVision Wang et al. 2024a), MathVise sta (Lu et al. (2024). Vi VisuLogic (Xu al. (025a)). EMMA (Hao al. 2025b), MMVP (Tong al. 2024), BT. ST.INE- (Fu al. 2024b). and Vstar (Wang) et al. 2023). All the evaluations are done using VLMEva K (uan et al. 2024). To ensure fair comparison, we use chain-of-thought prompting (Wei et al. 2022) when evaluating the base Anole model. As shown in Table 2
training with ZEBRA-COT significantly improves the Anole model all benchmarks. Most notably, it could improve the Anole model's 's visual logical reasoning capabilities by 13.3 points.
Ceb To further whether ZEBRA- CoT can enhance stronger backbone, we fine- tune the BAGEL-7B model (Deng et al.) (025) end-to-end on node with 8x H200 GPUs for 1,000 steps using packed sequences with 60,000 tokens, a learning rate of 10-5 and cosine decay. We cap all images at resolution of 512 on the minimum side, resulting in approximately 1, 0244 visual tokens per image. Because the original Bagel implementation cannot natively generate interleaved text-image outputs, revise the training loop to include a loss term the vvssionssarrt token, enabling seamless visual token generation. We additionally wrap text reasoning tokens be- tween <think> and </think>, and the final answer within <answer> and < /aanswer>, At inference time, when encountering <im end>, we sample one additional token to check whether the next token is vivis s start |>; if so, the model itself seamlessly switches image gener-

---

Published as a conference paper at ICLR 2026
ation mode to generate visual aids. The entire interleaved generation process only stops if the model generates the <answer> token.
We observe that our trained model inherently generate visual CoT when solving problems, on tasks outside its training distribution. This suggests its potential as a strong initialization for future reinforcement learning fine-tuning. In Figure we include representative reasoning traces produced by the model. We further include more reasoning traces in Appendix [ as well as a model performance analysis in Appendix D
6 CONCLUSION & FUTURE DIRECTIONS
In this paper, we introduced ZEBRA-COT, a large-scale dataset of 182K interleaved text-image reasoning traces spanning 4 major categories across 18 domains with over 50 distinct tasks. Fine- tuning experiments demonstrate substantial improvements: Anole-7B achieves an average 4.9 % gain across seven challenging benchmarks, with up 13.1% on visual logic tasks, while Bagel- 7B learns to inherently generate visual aids during problem solving, a capability absent in the base model.
This work opens several exciting avenues for future research. Most immediately, models trained on ZEBRA-COT, particularly our Bagel variant that natively generates visual thoughts, provide strong initializations for reinforcement learning. Just text-based reasoning models have benefited from RL fine-tuning to improve logical consistency and correctness, we envision similar gains for visual reasoning through RL with verifiable rewards (Shao et al. 2024b Guo et al. 2025) or fine-grained
rewards (Zeng et al., 2024 Fu et al. 2025).
We believe ZEBRA-COT represents crucial step toward Al systems that think visually naturally as humans sketch diagrams, generate graphs, and use spatial reasoning to solve complex problems. With our dataset and fine-tuned model, we hope accelerate progress toward this goal.
7 LLM USAGE DISCLOSURE
We used LLM for two purposes. The first one is for improving grammar and wording when writing the paper. The second usage is synthetic data generation, where details can be found in Section and Appendix A.2
ACKNOWLEDGMENTS
MG and AL supported by a Research Award from the Columbia Center of AI Technology in collaboration with Amazon and by Google. KY, ZC, FH, and TG were supported by DARPA Trans- fer from Imprecise and Abstract Models to Autonomous Technologies (TIAMAT) 80321, National Science Foundation NSF- 2147276 FAL, and DOD-AFOSR-Ai Force Office of Scientific Re- search under award number FA9550-233110000
WN was supported in part by the National Science Foundation under Grant No. CMMI-2427856. OL, DF, and BZ. would like to thank Center for Ad- vanced Research Computing (CARC) and NLP Group at USC for providing compute resources. In addition, Compute resources were generously provided by the NVIDIA Academic Grant program. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
REFERENCES
AgileX Robotics.
Cobot
magic.
https://gglobe aggi www.lle.....po.....oee wtwpouucp/oosssss/////oo
cobot-maçic, 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. 2022.
org/ abs 2200 14198
10

---

Published as a conference paper at ICLR 2026
Avinash Anand, Janak Kapuriya, Apoorv Singh, Jay Saraf, Naman Lal, Astha Verma, Rushali Gupta, and Rajiv Shah. Mm-phyqa: Multimodal physics answering with multi-image cot prompting. In Pacific-Asia Conference Knowledge Discovery and Data Mining, pages 53-64. Springer, 2024.
Anthropic. System card:
Claude
opus 4 & claude sonnet 4.
System card,
URL. https://www http://wwwwdddddddcee anthhnnthopicccco com. 6ba2a7c285....... changelog entries July 16, 2025 and Sept 2, 2025.
Anthropic, May 2025.
6cd8a855522000771
Original May 2025;
Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Guha, Sheng Shen, Mohamed Awadalla, Silvio Savarese, Caiming Xiong, et al. Mint-lt: Scaling open-source multimodal data by 10x: A multimodal dataset with one trillion tokens. Advances Neural Information Process- ing Systems, 37:36805-36822 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 2016.
Xi Chen and Xiao Wang. Pali: Scaling language-image learning in 100+ languages. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large mul- timodal models for interleaved image-text generation. arXiv preprint arXiv:2407.06135. 2024.
Ethan Chern, Zhulin Hu, Steffi Chern, Siqi Kou, Jiadi Su, Yan Ma, Zhijie Deng, and Pengfei Liu. Thinking with generated images. arXiv preprint 2.Xiv:2505.22555 2025.
François Chollet. On the measure of intelligence. arXiv preprint arXiv: 1911199147,,,,111
Francois Chollet, Mike Knoop, Gregory Kamradt, and Bryan Landers. Are prize 2024: Technical report. arXiv preprint 24t.::22222046600 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.06261 2025.
Joost CF de Winter, Dimitra Dodou, and Yke Bauke Eisma. Responses to raven matrices: Governed by visual complexity and centrality. Perception, 52())645-661. 2023.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. 14683, 2025.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 258..55.22 Iece, 2009.
Haodong Duan, Junming Yang. Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong. Yuhang Zang, Pan Zhang. Jiaqi Wang, et al. VImevalkit: An open-source toolkit for evaluat- ing large multi modaittyd models. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 11 198--12011 2024.
Franka Emika GmbH. Franka Emika Panda Robot Arm, 2018 https: www. Stttp::/www.Franaa. franka .de
Deqing Fu, Ruohao Guo, Ghazal Khalighinejad, Ollie Liu, Bhuwan Dhingra, Dani Yogatama, Robin Jia, and Willie Neiswanger. IsoBench: Benchmarking multimodal foundation models on isomor- phic representations. In First Conference on Language Modeling (COLM), 2024a.

---

Published as a conference paper at ICLR 2026
Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, and Lawrence Chen. TLDR: Token-level detective reward model for large vision language mod- els. In The Thirteenth International Conference Learning Representations, 2025. URL
https: http::///peeeeeeee... net fow
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. In European Conference on Computer Vision, pages 148- 118-166. Springer, 2024b.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint urXiv2250..12445 2025.
Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, and Xiang Yue. Mammoth-vl: Eliciting multimodal reasoning with instruction tuning at scale. arXiv preprint arXiv:2412.052277 2024.
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng- Can milms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.08 1.04444, 2025a.
Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang. Lijuan Wang, and Yu Cheng. Can milms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.0524 .05444, 2025b.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL httph:ttp///// http://pppeeeiiiieeeeee wttp:/iwwwwwwweeeeeeeeeeeeeeeeee 7 2mQsCel
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as visual chain of thought for multimodal language models. arXiv preprint arXiv:2406.094003 2024.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Michael Igorevich Ivanitskiy, Rusheb Shah, Alex F. Spies, Tilman Räuker, Dan Valentine, Can Rager, Lucia Quirke, Chris Mathwin, Guillaume Corlouer, Cecilia Diniz Behn, and Samy Wu Fung. A configurable library for generating and manipulating maze datasets, 2023. URL https:
//aaxiv.orq/aaat abs/ 2200/3399202 10498
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2901-2910, 2017.
Alex Lau-Zhu, Emily A Holmes, Sally Butterfield, and Joni Holmes. Selective association between tetris game play and visuospatial working memory: A preliminary investigation. Applied cogni- tive psychology, 31(4):438-445, 2017.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander M. Rush, Douwe Kiela, Matthieu Cord, and Victor Sanh. Obelics: An open web-scale filtered dataset of interleaved image-text documents, 2023.
Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulié, and Furu Wei. Imagine while reasoning in space: Multimodal visualization- of-thouuht. arXiv preprint arXiv:2501.07555 2025.
12

---

Published as a conference paper at ICLR 2026
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Lava--exx-interleavee Tackling multi -maat, video, and 3d in large multimodal models. arXiv preprint arXhv:2407.0789) 2024a.
Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhangwei Gao, Erfei Cui, et al. Omnicorpus: A unified multimodal corpus of 10 billion-level images interleaved with text. arXiv preprini arXiv:2406.08418, 2024b.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang. Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hanmaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations (ICLR), 2024.
Sachit Menon, Richard Zemel, and Carl Vondrick. Whiteboard-of-thougee Thinking step-by-step across modalities. arXiv preprint arXiv:2406. 14562, 2024.
MIT Open [Course Title]. https://ooat http:::///ww..mt.ed/////
mit. eet/ 2022. MIT OpenCourse- Ware: Massachusetts Institute Technology. License: Creative Commons BY-NC-SA.
Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. Compositional chain-of- thought prompting for large multimodal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14420 14431, 2024.
William Moebs, Samuel J. Ling, and Jeff Sanny. OpenStax, Houston, Texas, 2016.
University Physics Volume 1. URL http:://oeestaax.org/bokk
unn.versstt--piyscss-ooooo BY B.0.0.
Licensed under
Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng
Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36:25081-25099 2023.
OpenAl. Openai o3 and 04-mini system card, Technical report, OpenAI, April 2025a. URI
03 http:r/r.o....ooo.c..cco///////r
OpenAl.
Thinking with images.
https://openai.coo//t
April 2025b. Accessed: 2025-07-21.
OpenAl. Gpt-5 system card, August 2025c. URL https://openai.. wwt.c//mmm/n/////ooo
gpt 5- --ssem--cardd card/ System card for GPT-5.
Bharath Ramsundar, Peter Eastman, Patrick Walters, Vijay Pande, Karl Leswing, and Zhenqin Wu. Deep Learning for the Life Sciences. O' O'Reilly Media, 2019. https::/ amazon. com/
Deep Learning Life Sciences (alcroscopy//p/1442200
Daniel Rose, Vaishnavi Himakunthala, Andy Ouyang. Ryan He, Alex Mei, Yujie Lu, Michael Saxon, Chinmay Sonar, Diba Mirza, and William Yang Wang. Visual chain thought: bridging logical gaps with multimodal infillings. arXiv preprint arXiv:2305.00 02310.2.7.........000000111.1.10 2023.
Hao Shao, Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong, Letian Wang, Yu Liu, and Hong- sheng Li. Visual cot: Advancing multi modlt language models with comprehensive dataset and benchmark for chain-of th-tuughtttt reasoning, Advances in Neural Information Processing Systems, 37:8612-8642 2024a.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024b. https:/::: /ttps:///rxxvv org/ abs/
2402. 0.330
13

---

Published as a conference paper at ICLR 2026
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. ALFRED: A Benchmark for Interpreting Grounded Instruc- tions for Everyday Tasks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020. URL Stt.lhhtps://// hatp::///axvv.orroo.p htt//bbs//99// /09.///19991011114111111
Linzhuang Sun, Hao Liang, Jingxuan Wei, Bihui Yu, Tianpeng Li, Fan Yang. Zenan Zhou, and Wentao Zhang. Mm-verify: Enhancing multimodal reasoning with Shain-of-thougee verification. arXiv preprint arXiv:2502. 2025.
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 14398-14409, 2024.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888-11898, 2023.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, lean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.1/803 2023.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, al. Gemma 3 technical report. arXiv preprint 2rXiv: 2025.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal Ilms, 2024.
Universal Robots A/S.
UR5e Collaborative Robot Arm, 2018.
https://www
universa --roorts.s.... com haom/prouccteo s/ure//
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024a. URL
ew www.ne///e... www.:rett/orrmmmmmooooooooom
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprini arXiv:2409, 109.86999, 2024b.
Yan Wang, Yawen Zeng, Jingsheng Zheng, Xiaofen Xing, Jin Xu, and Xiangmin Xu. Videocot: A video chain-of of-thought dataset with active annotation tool. arXiv preprint arXiv:2407.05355 2024c.
Yuxuan Wang, Zilong Zheng, Xueliang Zhao, Jinpeng Li, Yueqian Wang, and Dongyan Zhao. VS- TAR: A video-grad grounde dialogue dataset for situated semantic understanding with scene and topic transitions. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 5048, Toronto, Canada, July 2023. Association for Computational Linguistics. URI. https:
ogy. / 2023 acl long.
276
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. thaouht prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24883 2022.
Kun Wu, Chengkai Hou, Jiaming Liu, Zhengping Che, Xiaozhu Ju. Zhuqin Yang, Meng Li, Yinuo Zhao, Zhiyuan Xu, Guang Yang, Zhen Zhao, Guangyu Li, Zhao Jin, Lecheng Wang. Jilei Mao, Xinhua Wang. Shichao Fan, Ning Liu, Pei Ren, Qiang Zhang, Yaoxu Lyu, Mengzhen Liu, Jingyang He, Yulin Luo, Zeyu Gao, Chenxuan Li, Chenyang Gu, Yankai Fu, Di Wu, Xingyu
14

---

Published as a conference paper at ICLR 2026
Wang, Sixiang Chen, Zhenyu Wang, Pengju An, Siyuan Qian, Shanghang Zhang, and Jian Tang. Robomind: Benchmark on mubii-embodiment intelligence normative data for robot manipulation. arXiv preprint arXiv:2412.13877 2024.
Penghao Wu and Saining Xie. V*: Guided visual search as a core mechanism in multimodal llms. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattem Recognition, pages 13084-13094, 2024.
Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-ol: Let vision language models reason step-by-step. arXiv preprini arXiv:241 1. 10440, 2024.
Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, Wenhai Wang, Jifeng Dai, and Jinguo Zhu. Visulogic: A benchmark for evaluating visual reasoning in multi-modal large language models, 2025a. URL
https://arxin
abs 15279
Yi Xu, Chengzu Li, Han Zhou, Xingchen Wan, Caiqi Zhang, Anna Korhonen, and Ivan Vulić. Visual planning: Let's think only with images. arXiv preprint arXiv:2505. 11409, 2025b.
Sherry Yang, Jacob Walker, Jack Parker-Holder, Yilun Du, Jake Bruce. Andre Barreto, Pieter Abbeel, and Dale Schuurmans. Video as the new language for real-world decision making. arXiv preprint arxi::2402. 171.999...99., 17139, 2024.
Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. RI-onevision: Advancing generalized multimodal rea- soning through cross-modal formalization. arXiv preprint arXiv:2503. (tt.::2533..011.... 2025.
Zhengyuan Yang*, Linjie Li*, Jianfeng Wang*, Kevin Lin Ehsan Azarnasab*. Faisal Ahmed* Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multi- modal reasoning and action. 2023.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In Proceedings of the JEEE/CVF conference on computer vision and pattem recognition, pages 6720-6731, 2019.
Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang. Haifeng Zhang, and Jun Wang. Token-level direct preference optimization. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learn- ing Research, pages 58348-58366 PMLR, 21-27 Jul 2024. URL cestps://proceediin m1 A.r.press/w2335 wewg24c. html
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chin Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5317-5327, 2019.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thouge thought reasoning in language models. arXiv preprint arXiv:2302.00923, 2023.
Qingqing Zhao, Yao Lu, Moo Jin Kim, Zipeng Fu, Zhuoyang Zhang, Yecheng Wu, Zhaoshuo Li, Qianli Ma, Song Han, Chelsea Finn, et al. Cot-vla: Visual tffin-of-thought reasoning for vision- language-action models. arXiv preprint arXiv:2503.2200 2025.
Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505. 14362, 2025.
Wang Zhu, Hexiang Hu, Jiacheng Chen, Zhiwei Deng, Vihan Jain, Eugene Ie, and Fei Sha. Baby- Walk: Going farther in vision-and languuge navigation by taking baby steps. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2539-2556. Association for Computational Linguisties, 2020.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion- scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36:8958-8974 2023.
15

---

Published as a conference paper at ICLR 2026
A DATASET DETAILS
A. DATA STATISTICS.
Here show detailed statistics about ZEBRA-COT's categories.
Table 3: Statistics of ZEBRA-COR
General Category
Sub Category
Count
Percentage (%)
Visual Jigsaw
21,899
12.0
2D Visual Reasoning
Visual Search
30,000
16.4
Subtotal
51,899
28.5
Embodied Cot
22,666
12.4
Multi-Hop Objects Counting
10,000
5.5
3D Visual Reasoning
Robot Planning
6,944
3.8
Subtotal
39,610
21.7
Chemistry
4,666
2.6
Competitive Programming
1,207
0.7
Geometry
1,058
0.6
Scientific Reasoning
Graph Algorithms
10,000
5,5
Physics
7,090
3,9
Subtotal
24,021
13.2
Arc-Agi
2,000
1.1
Checkers
2,753
1.5
Chess
20,483
11.2
Ciphers
6,589
3,6
Visual Logic Strategic Games
Connect Four
2,029
1.1
Maze
20,000
11.0
RPM
3,000
1.6
Tetris
10,000
5.5
Subtotal
66,854
36.7
Total
182,384
100.0
16

---

Published as a conference paper at ICLR 2026
A.2 CURATING DIVERSE AND HIGH QUALITY VISUAL COT
Bridging logical connections across modalities. A key challenge in training multimodal gen- eration models to output visual CoT natively is the lack of datasets with strong logical coherence between text and visual modalities, and diverse categories of such visual CoT. Existing interleaved datasets often fail provide clear, meaningful connections that demonstrate when and why visual reasoning is necessary for problem-solving, while current visual CoT datasets are confined to a few domains, limiting the model's s ability to learn generalizable visual CoT capabilities when faced with out-o--distribution problems.
High quality Reasoning Trace
stolo
Program Generated Raw Real World Data
Sending Text Trace and Images to VLMs
Image Holders
What force between the two blocks?
Q: What the force block A?
C: What is the force of contact between
<image starts [eroblerr__ng imagle_ <image_end-
The total external forca, F. FLextt, on system accalaration is
This Problem image
Reasoning Trace:
Thought We need to first construct body clagram for each the blocks.
Reasoning image
simage_stari staartstttt [reasoning image 1) <imege_enii ends <image_sstrre start>> freasoning_image. <inege ends
external force, ext, the system 3.2 N. The acceleration is ...
This Reesoning image
Thought Based on the sketches, the acceieration is
Figure 5: An overview of our data curation pipeline.
To address these requirements, we first source a diverse range of question types and domains. For real world data, we source high-quality problems from online resources such as math, physics, coding, and chess competition datasets. We then extract and clean the available raw reasoning traces that contain text and images. However, even from high quality sources, traces can still lack clear logical connections between modalities, as well as clear references to the images for automatic parsing into interleaved text and image data ready for training. For example, most geometry data uses reference labels such "Figure x²", which makes it hard to find the mapping between the actual image and the text reference. For synthetic data, we create our own examples by generating images utilizing real images from online sources, then crafting corresponding reasoning templates. This procedure raises a clear issue, namely that we lack diversity and expressiveness in textual reasoning regarding templated data. For instance, in visual search tasks, it is erucial to elucidate the rationale behind drawing specific bounding boxes, and in chess, generating reflections and descriptions of move visualizations is key.
We address both of these issues using frontier VLMs (Gemini-2.5 and GPT-4.1) to fill in the template placeholders, enhance the reasoning traces, and complete the textual reasoning narrative. We feed both images and raw text reasoning traces into the language model and ask the language model to output pure text traces with image placeholders. We further filter out invalid cases, such multiple image placeholders referring to the same image and unreferenced image placeholders, to ensure that the data can be automatically parsed into training dataset.
Broadening breadth and diversity of interleaved visual language reasoning dataset. Further- more, existing multimodal rationale datasets are also limited their breadth. The only available datasets focus on either visual search (Wu and Xie 2024 Shao et al. 2024a) spatial reasoning, such as maze navigation (Li et al., 2025). Such limited datasets are unlikely to enable training visual reasoning models that generalize across domains more broadly. Visual Sketchpad (Hu) (et al. 2024) offers a diverse range of VLM agents to tackle a wider variety questions. Though 'Sketchpad offers powerful and significant contribution to generating visual aids, the pipeline is
17

---

Published as a conference paper at ICLR 2026
not designed for collecting post-training datasets. First, the reasoning traces generated by agentic pipelines often involve tool call errors and debug information, which degrade their quality. Second, the scalability and diversity of the dataset
fundamentally constrained by the limited number of agent tool designs and the high cost, as each reasoning trace may require many API calls. To tackle these issues, curate total of over 182K high-quality interleaved text and visual reasoning traces, spanning four major categories: scientific reasoning, 2D visual reasoning, 3D visual reasoning, and visual logic and strategic games. We provide the details in the section below and example traces from our dataset.
18

---

Published as a conference paper at ICLR 2026
A.3 SCIENTIFIC QUESTIONS
Geometry. Geometric understanding is a core ability for multimodal models to ground reasoning over complicated mathematical tasks. Many datasets have been proposed evaluate mathematical capabilities, including geometry. The MATH dataset (Hendrycks et al., (022]) is widely used for evaluating the mathematical performance LLMs. Although the MATH dataset includes numerous geometry competition problems, their geometric elements are provided as plotting code rather than rendered images (see Figure
Here, we provide example code for geometry sketch generation.
MATH/GHOMETRYY44
[asy]
import three;
(ize(2.
cuzrentprojection
otthograpaic(112,,11.114
triple A = (0, (0,0,6);
triple[] base = new [r]ple[4];
base[0] (-4, -4, 0);
basell] (4, -4, 0);
base[2] (4, 4, 0);
base[3] (-4, 4, 0);
triple[] mid = new triple[4];
for i=0;
4; ++i)
mid[i] = (. (.6xxpart (base (i]) +
.4xpart (A), .5ypart (base [i])
4+ypprt (A), .6+ppart [baseli])
4*zpart (A));
for i=0; i < 4; ++i)
draw (A--base [I]);
draw (base baseballaaasllllee (asel
draw mid) (i+1)84), dashed);
10 units
label ( '8/ units" base [0]= bb]ss
label units", base [0] --, 2+W); [/asy]
8/2 units
(a) Geometric Example ZEBRA- CoT
(b) Geometric Example in MATH Datasct (Hendrycks et al. 2021
Figure 6: Comparison of the same geometric figure our ZEBRA-COT dataset and the MATH dataset. Ours focus on multimodal reasoning and explicitly plot the geometry problem than using the text-only plotting codes.
In ZEBRA-COT, we convert every piece of plotting code into figure renderings, producing both the problem diagram and its solution illustration serve explicit visual reasoning chain for model training.
In total, we collect 1,061 samples from the MATH dataset's train split. Our data provides only rendered images for both the problem and solution reasoning chains, with no plotting code included. Solving these problems requires generating images assist. The problems not restricted to the geometry subcategory but also include some problems from counting and probability, pre-algebra, pre-ca leu us, etc.
19

---

Published as a conference paper at ICLR 2026
Physies. A variety of physies problems benefit from sketches, such free body diagrams for force analysis, motion diagrams for kinematics, circuit diagrams for electricity, and ray diagrams in optics. We construct samples of classical mechanics problems programmatically. Problem instances are generated from parametric Python templates (e.g., Atwood machines, inclined planes, elastic collisions, pendulums), with physically plausible parameters sampled from predefined ranges. For each sample, we render free-body diagrams, kinematic visuals, and structured CoT traces capturing the full solution process.
We also leverage openly licensed resources such as OpenStax (MIT OpenCourseWare: 2022) and MIT OCW (Moebs et al. 2016) to generate more diverse and complex physics problems, ultimately achieving scalable and legally clear dataset generation while ensuring diverse, high-quality exam- ples.
Chemistry. Organic reaction prediction is a classic multimodal reasoning task, typically framed as symbolic input and structural output. We include a chemistry subset 4,700 two-to-one reactions from the USPTO-50K dataset (Ramsundar et al., 2019). filtered for distinct reactants and single products. Each reaction trace includes three visual artifacts: individual molecular depictions each reactant, combined schematic of both reactants side-by-side, and the resulting product structure. Molecules are rendered with RDKit, , and names are retrieved from PubChem when available. Text prompts use randomized templates (c.g.. "What is formed by combining acetic acid and ethanol?"), and PubChem names are included when available. This visual progression helps models learn com- positional chemical structure without SMILES reaction templates.
Chemistry
othalo <tholo-Thuuhh Thought The firt dietch shows the diemine structure groups (MM))
Question: Wieen diyyiin vlaxyborzene diamine
etthit-Thought undbostand
reaction, structure the olamine reactont 4schiovn-d S-dayritinck ploiyoonpone 1,2 olamine Thgs
bergene cha along ciloro and poridieylory subvatuents New, visuatiz second roccontt pyrihiie carbory orthenge
undergoes meacion with mynidie---on Dwwbboyylii fenmee??
isenity gerupe
Reasoring image
Reasoning Image
Algorithmic problem solving. Humans naturally create visual diagrams when solving complex problems, transforming abstract concepts into spatial representations for deeper reasoning. We for- malize this by interpreting coding problems through compact visual scaffolds: one or two diagrams depicting graph structure, edge weights, etc. To build traces, run iterative "visual sketchpad" loop: GPT-4.11 receives a prompt and returns THOUGHT statements plus VIS SPEC blocks when sketches needed; we render specs with networkx/mat otlib, feed images back to the model, and repeat until complete, then clean transcripts with post post-processing.
Problem samples come from competitive programming, prioritizing real-world abstractions like lo- gistics, network routing, and flow optimization. The orchestrator produces simple visual structures emphasizing clarity over style. Each trace contains the problem prompt, 1-3 reference diagrams, and polished explanations, supporting grounded reasoning in discrete structures while mirroring how al- gorithms are taught. The final corpus comprises 1,200 diverse algorithm- based problems spanning competitive programming.
Graph problems. Graph algorithms are useful for large language model applications because they efficiently organize and traverse structured relationships. for example in search and retrieval applications. Methods like shortest-path and subgraph matching enable multi-step reasoning by connecting relevant concepts across knowledge graphs. Recent work by Fu et al. (2024a) shows that although LLMs can solve graph problems such as comnectivity and maximum flow to some extent when textual description of the graph given, multimodal LLMs suffer when solving graph problems. This finding suggests potential for improving multimodal models' graph-underssandinn abilities by guiding their reasoning over images.
20

---

Published as a conference paper at ICLR 2026
We create 10,000 graph problems with full reasoning traces spanning four tasks: graph nectivity, shortest path. minimum spanning tree, and topological sort. Each task has about 2,500 samples, with one problem image and at most 19 reasoning images per sample. Each reasoning image is coupled with an explanation for the underlying algorithms, for example, Dijkstra for the shortest path, BFS for connectivity, etc.
A.4
2D VISUAL REASONING
Visual search. Previous research has shown that drawing bounding boxes and zooming can im- prove accuracy on visual search tasks (Wu and Xie 2024, [hao et al. (024a). We follow such tasks by creating two types of traces, one for drawing bounding boxes one for zooming. We use data from Shao et al. (2024a) to generate our traces covering four categories of visual search tasks: chart, text/doc, relation study, and general VQA.
Visual Search
<thikk <thys-TThougt 0 To the dosely examine afton the prowided image Thought wil soom on the bey get ciaaee vew o''is
cthint Thought 2: mommediin
image cleatly
tyne eloshing Ery in the foreground wearing while plasing on fiel??
shor-dleeved
black sharts, white
stripe,, typical of socner uniform
Thought Banrd on these distaiis
suitable
(ogerax
playing tmiiike
Visual jigsaw. Visual jigsaw refers to filling in missing pieces of an image, as in jigsaw puzzle. Each puzzle is constructed from an ImageNet (Deng et al. 2009) image, with I to 4 missing pieces of varying shapes, including rectangles and irregular regions. Each puzzle includes four multiple- choice options, where each option presents a set of candidate missing pieces. Only one set correctly matches the pieces removed from the original ImageNet image. We generate
types of visual CoT traces for solving each puzzle. In the first type, we iteratively fill in the missing patches using the pieces from each multiple-choice option and identify the one that produces coherent image. In the second type, we imagine what the original image would look like and then select the option whose pieces best match the imagined reconstruction.
A.5 3D VISUAL REASONING
Embodied planning. For embodied planning tasks, agents must ground high-level decisions the evolving visual context of the environment. We reformulate the ALFRED (Shridhar et al., (0200) benchmark, an interactive 3D simulation environment where agents perform complex tasks based on human instructions, into an image coal-condii planning task.
Question:
Alhikko Thought tamn lwft take step, sanrightt malk the
<thinko Thought 2: pici the cancle that the courter
Thought
Stiik Thought put the canale on the buck of the codle.
fam aroundd walt the sinwee, turn left, watk
the sequence of
robot actions from the inttal ithe
first image) to the goel stete (the sacond Image).
Image
Reasoning Image
Reasoning Image
Reasoning Image
Reasoning Image
21

---

Published as a conference paper at ICLR 2026
In this new task, the model receives two images: the initial and goal states. Then the model is tasked with generating a textual description of the high-level planning steps required to transition from the initial to the goal state. To emphasize the role of visual reasoning, we require the generated descriptions to be detailed and step-by-step (e.g. "turn and go to the TV; pick up the bowl that is on the TV stand in front of the TV; with the bowl in hand... ") rather than brief summaries (e.g.. "move bowl to coffee table") "). which can often be produced through shortcut reasoning without capturing intermediate visual steps.
We compile the entire training set, as well as the seen and unseen validation sets from ALFRED, resulting in total of 7,080 examples spanning diverse visual reasoning trajectories. When multiple textual reasoning annotations exist for a single visual trajectory, we include all of them, resulting in 22,666 textual reasoning traces.
Robot planning. While low-level manipulation may rely on reactive control, continuous plan- ning for complex tasks often requires high-level visual guidance, making visual CoT essential for bridging perception and long-horizon decision-making in robot planning, Similarly, we reformulate RoboMIND (Wu et al.) 2024), a multi-embodiment dataset of real-world robot manipulation, into an image goal-conditioned planning task. In this setting, model is provided with the initial and goal states images, along with a textual description of the robot setup (c.g.. AgileX (AgileX Robotics, 2023). Franka (Franka Emika GmbHl 2018), or UR5e (Universal Robots A/S 2018 ) and is tasked with generating a detailed textual plan outlining the high-level steps required to transition from the initial to the goal state.
Unlike embodied planning tasks that often involve partial observability and require agents to infer unobserved states, this robot planning task is fully observable. Therefore, the challenge lies not in imagining the visual trajectory but in articulating precise movements for each arm or gripper to accomplish the task (e.g., "[left] move towards the oven door and [right] grab the corn.").
To control degrees of freedom, we exclude the humanoid robot examples from the original Robo- MIND dataset, focusing solely on tasks involving robotic arms. This results in curated subset of 6,945 robot planning tasks, each annotated with human-generated high-level actions that serve as visual reasoning trajectories.
3D multi-hop objects counting. A core aspect of human transformations and imagining scenes from different viewpoints. For this task, our setup follows a structure similar to that of Johnson et al.] (2017), using 10 predefined shape types (e.g., sphere, cylinder, donut) in various colors. At each step, we randomly apply one of three operations: remove all instances of attribute (e.g., all red objects), remove subset (c.g., 5 red objects), or add new objects (e.g., 2 blue prisms, red sphere). We then create questions that ask about the quantity of specific attributes are rendered from varying viewpoints (front, back, left, right), where some objects may be partially occluded by those in front. The first visual reasoning step involves generating top-down 45 view to reconstruct the full scene, allowing the model to see potentially blocked objects. The subsequent visual sketches correspond to each transformation step in the instruction. We also improve upon the data from Johnson et al. (2017) by adding in different materials, backgrounds, and floor designs.
reasoning is understanding
what objects
left in the field. To increase difficulty, the initial scenes
A.6 VISUAL LOGIC AND STRATEGIC GAMES
Visual logic puzzles. Humans approach logic puzzles such as Tetris, Raven's Progressive Matrices (RPM, Zhang et al. 2019). and the Abstraction & Reasoning Corpus (ARC-AGI, Chollet 2019 Chollet et al. 2024) primarily through visuospatial reasoning: we see how pieces combine, trans- form, or complete a pattern before committing to an answer. These logic games rely heavily on visuospatial working memory, which is correlated with general intelligence level (Lau-Zhu et al.)
2017 de Winter et al. 2023).
To enhance models with such cognitive ability, we include the following tasks. For Tetris, we collect three types of tasks: a) shape assembly: given silhouette and candidate tetromino sets, select the one that perfectly tiles the shape; b) grid completion: fill a partially occupied grid using a specified set of tetrominoes; c) spatial transformation: apply a sequence of geometric operations (translate, rotate, mirror, scale) to an irregular shape in the grids. The visual CoT involves visualizing each transformation step. For RPM (IQ matrix), we include three types from Zhang al. (2019) that
22

---

Published as a conference paper at ICLR 2026
Tetris
Question:
chirks Thought f. Apply firs mamioomatioon scain the shape by 2 aomes thes the shape cithoks
ahiek Thought fonnaformation rotate roaled shape 90 degraws clockoesn </ttinns
ohhiko Thought uirg
Apply the following eequeere
fransfarmanion: framslate rotased shape 1 unit </thini
monioomations to the Exwe shapsi scalo X then notate bistes, then translate up, then iolookaiioo. Choose the option that shows the reautting
mi
Problem Image
Rewsaning Image
Reasoning Image
Reasoning Image
involve compositional reasoning. The reasoning trace identifies visual patterns for each composi- tional component across rows columns. For ARC-AGI, while prior models often rely on textual reasoning, humans typically solve these tasks through visual pattern recogmition. To better align with human strategies, we construct sentations of the training examples and test input; the reasoning trace first visualizes the training examples, the test input, and finally the predicted output. The second type directly uses visual repre- sentations in the task instruction, thus the model only has to generate visual sketch of the predicted output as part of its reasoning process. For all data, we use VLM generate accompanying textual descriptions to enrich interleaved text-image rationales.
types of visual CoT. The first begins with matrix repre-
Mazes. Mazes serve as a canonical testbed for visual CoT reasoning, bridging low-level perception with high-level symbolic search. Unlike purely pixel-based 2D visual tasks such as visual search and visual jigsaw, mazes possess explicit graph structure yet remain visually intuitive, letting us disentangle vision errors from planning errors.
We adopt the maze-dataset library to procedurally generate thousands of grid mazes with di- verse topologies (lattice type, branch factor, loop density) > Each instance is exported in two com- plementary formats: a) m. as as_pisiiiee pixeee s () an RGB raster that encodes walls, free cells, start m, and goal suitable for visual perception: b) MazePlot, vector overlay that superimpose solution paths, candidate trajectories, heat-maps, or landmark nodes for numan-readable walk-throuugh. To increase maze diversity, also use OpenAl Gym's FrozenLake- environment (Brockman et al. 2016)
We evaluate broad spectrum of spatial reasoning skills across multiple question types: topo- logical analysis (e.g., counting isolated regions, identifying connected components under or 8- connectivity, finding the largest connected area), II. pathfinding (e.g., determining reachable end- points, computing shortest paths, enumerating all optimal routes), IlI. navigation planning (e.g., se- lecting correct paths from alternatives, calculating minimal moves reach targets), and IV. coverage problems (c.g., visiting all marked locations, identifying the farthest reachable position). This di- verse task suite goes beyond simple start-to-goa -t-ggal navigation, encompassing the full range of spatial reasoning strategies that humans use to interpret complex environments. We also introduce vary- ing complexity of the matrix, including different maze side lengths ranging from (5, 15), different branching factors 6, loop probability f, and number of distractor endpoints A. Larger exponen- tially increases search space, while higher b and degrade heuristic admissibility. Both those require genuine planning rather than rote memorization.
Chess.
Strategic planning in chess involves simulating multiple futures and selecting moves that maximize long-term advantage. To support counterfactual reasoning, we construct a dataset of mid-game posi- tions from rated Lichess games? each with structured visual traces. Given position, Stockfish
ma ze -oee--aae dat. supports Scursive-hacktracker randomized Prim, Wilson, and Kruskal generators; see [vanitskiy et al. 2023). https: https://ichess.oo
23

---

Published as a conference paper at ICLR 2026
What's White's s best move? Options: A: Ba2, B: Na4, C: Q15, D: Bb3.
(A) Ba2: Safe position (B) Na4: Poorly placed (C) Q15: Exposed queen (D) Bb3: Vulnerable position supports central pawns weak attack b6
vulnerable to g6
weaker than Ba2
Figure 7: Traces showing reasoning for each move option. Option A (Ba2) is evaluated as strongest, providing safe bishop placement while supporting potential central pawn advances.
identifies the optimal move, and three alternates are sampled randomly from legal moves. Each can- didate is visualized independently for comparative evaluation. By rendering possibilities in isolation, move consequences, tempo gain, structural weakening, and tactical motifs become legible, enabling better strategic reasoning. Traces are formatted as multiple-choice tasks with visual sketches, en- couraging tactical awareness and pattern recognition. Postprocessing with Gemini 2.5 Flash refines traces into coherent multimodal reasoning sequences for short and long-horizon planning, yielding 11,500 chess traces across diverse positions and hypothetical move states.
Connect 4. We also include 2,100 Connect-Fouu traces that teach short-horizon grid planning. Each starts from mid- game position obtained after 4-20 random moves; a depth-4 a-3 minimax (center- control, open-three heuristic) then plays the forced win. Every drop is rendered 7 6 board with a glowing overlay, and an LLM condenses the raw log into a clear, step-by-step "How does Red win?" narrative pairing text and visuals.
Checkers. Our checkers traces, initiated from randomized mid-game boards and continued through depth-4 minimax, designed capture key tactical motifs such forced captures, multi-jump chains, and king-row advancement, providing dense supervision for spatially grounded decision making. To ensure data quality, we apply rule-based validators that discard invalid traces. Final traces are passed through language model to synthesize coherent visual reasoning aligned with turn-based planning, ending with 2,800 full traces.
Ciphers. Classical encryption schemes convert symbolic rules into spatial transformations, enabling visual reasoning over grids and coordinates. We generate examples of Scytale, Polybius, Rail Fence and Playfair ciphers, each with multimodal traces showing encoding steps. Visual sketches highlight relevant regions such as column positions, grid lookups, or digraph geometry, while text describes applied rules (e.g., "Letter 'p' goes on the second rail," " "'A' is al coordinates (1, 1).") Early steps use visuals, later steps use language, mirroring real cryptanalysis. The 6,600 traces align symbolic manipulation with visual structure.
24

---

Published as a conference paper at ICLR 2026
B EXTRA BAGEL ZEBRA COT GENERATED VISUAL CoT
Visual CoT Example: EMMA
Prompt:
There are 81 grid points (uniformly spaced) in the square shown in the diagram below, including the points the edges. Point P is the center of the square. Given that point Q randomly chosen from among the other 80 points, what is the probability that line PQ is a line of symme- try for the square?
Visual CoT:
<think>
The problem asks for the probability that randomly chosen point Q from the 80 points other than the center P results in the line PQ being a line of symmetry for the square. We need to determine the number of such points Q and divide by the total number of possible points (80).
First, let's visualize the grid and the center point P. The grid has 9 points along each side, so there are 80 points in total (excluding P). A line symmetry for a square passes through the center P' and divides the square into two con- gruent halves. For PQ to be a line of symmetry, Q must be the reflection of P across this line. This means Q must be on the grid line that forms a line of symmetry with P. Let's consider the lines of symmetry for the square. square has 4 lines of symmetry: 2 diagonals, the vertical line through the center, and the horizontal line through the center. Let's visualize these lines. </think>
[Visual Thought]
25

---

Published as a conference paper at ICLR 2026
Visual CoT Example: Visulogic
Prompt:
Solve the complex visual logical reasoning problem through seep-by-step reasoning.
?
A
ABCD
B
D
Visual CoT:
<think>
Observe the grid to identify patterns in the shapes and their components. The grid is divided into 3x3 cells. Each cell contains two distinct shapes: line segment and dot. I will analyze the patterns for the line segment and the dot separately. For the line segment, the pattern is: (Top Left, Top Right) (Bottom Left, Bottom Right). For the dot, the pattern is: (Top Left, Top Right) (Bottom Left, Bottom Right). Now, I will apply these patterns to the third row and third column find the missing shape. The missing shape in the third row and third column should have a line segment like the one in the third row, third column (which is the same as the third row, second column line segment). The missing shape in the third row and third column should also have a dot like the one in the third row, third column (which is the same the third row, second column dot). Combining these, the missing shape should be line segment from the third row, second column and a dot from the third row, second column. </think>
A
B
D
[visual Thought:
26

---

Published as a conference paper at ICLR 2026
Visual CoT
Example: EMMA
Prompt:
The picture shows cube with four marked angles: < <WXY, XY, 2 XYV, < zYWW, and < ZWX X. What is the sum of these angles?
Z
Visual CoT:
<think>
The problem asks for the sum of four specific angles in cube. The angles are < WWXP XY, <XYZ, <YZW> and < X.WX. These angles are formed by the edges of the cube meeting at vertices X,Y, Y, 2, W.
In cube, each vertex is formed by three edges that are tually perpendicular. This means that at each vertex, the three edges meet form right angles.
Let's consider vertex X. The edges WX, XY, and X meet at vertex X. Since the edges are perpendicular, the angle < WWWP XY = =00.
Similarly, we can analyze the other vertices:
At vertex Y: edges XY, YZ perpendicular, XXY Z = 90
[Visual Thought]
At vertex Z: edges YZ, ZW are perpendicular, so < <YWW 90"
At vertex W: edges ZW, WX are perpendicular,
so ZW 90"
We have identified the measures of all four angles: <WXY- 90% <XYZ 900. < Y ZW 90%, < ZWW X 90" To find the sum of these angles, we add their measures:
90a 90" 90 90 360
The sum of the angles zWXY, XYZ, <YWW, and < ZWX is 360 </think>
27

---

Published as a conference paper at ICLR 2026
Visual CoT Example: Math Vista
Prompt:
Question: As shown in the figure, AB is diameter of 60, and point C' is on eO. If < < 40.0, then the degree of B is?
A
B
Visual CoT:
<think>
First, let's analyze the geometry of the problem. Since AB the diameter the circle, any angle inscribed in the semicircle formed by AB will be right angle. Thus, angle degrees. Now, let's draw an auxiliary line connect-
ACB is right angle, measuring ing point O (the center of the circle) to point C. This line segment OC is crucial part of our reasoning. The line segment OC can labeled as shown in the image.
C
Since AB is the diameter of the circle, O is the center: There- fore, AO and BO are radii of the circle, making them equal in length. By the properties of isosceles triangle (AO = BO), and having an angle of 90 degrees at C (i.e., angle ACB), we deduce that triangle ABC is isoseeles right-angled tri- angle. This means that angle A is equal to angle CBA, and since angle ACB 90 degrees, the other angles must add up to 90 degrees. Given that angle A is 40 degrees, an- gle B must be 50 degrees. Therefore, angle 50 degrees. </think>
B
[Visual Thought]
28

---

Published as a conference paper at ICLR 2026
Visual CoT Example: BLINK
Prompt:
How many kids are sitting on the bench in the image?
Visual CoT:
<think>
Let's count the number of kids sitting on the bench in the image. To get clearer view of the kids on the bench, will zoom in the relevant clearly shows four children sitting on the bench. Based on this visual evidence, the number of kids on the bench is four. //think>
The zoomed-in image
[Visual Thought]
29

---

Published as a conference paper at ICLR 2026
C
ANOLE-ZEBRA PERFORMANCE BREAKDOWN
Split
Anole
Anole-Zebra-CoT (Ours)
Overall 12.80
15.03
Chemistry
12.84
15.48
Coding
9.75
16.31
Math
13.12
14.35
Physics
21.79
10.90
Table 4: EMMA: breakdown by subject (%)).
Subtask
Anole
Anole-Zebra-Cof (Ours)
Overall
22.80
24.90
Scientific reasoning
30.33
32.79
Textbook question answering
36.08
29.75
Numeric commonsense
16.67
17.36
Arithmetic reasoning
15.58
18.98
Visual question answering
24.58
29.61
Geometry reasoning
20.50
23.01
Algebraic reasoning
25.27
24.56
Geometry problem solving
21.15
24.04
Math word problem
9.14
12.37
Logical reasoning
29.73
10.81
Figure question answering
24.54
28.25
Statistical reasoning
20.27
26.58
Table 5: MathVista breakdown by subtask for base vs. our model (%).
Subtask
Anole
Anole Mhole-Zebra-CoT (Ours)
Overall
8.50
21.80
Quantitative reasoning
8.78
21.81
Spatial reasoning
8.23
22.08
Positional reasoning
8.82
19.85
Attribute reasoning
9.76
25.61
Stylistic reasoning
10.00
24.44
Other
5.56
18.52
Table 6: Visual Logic: breakdown by subtask (%).
30

---

Published as a conference paper at ICLR 2026
Category
Anole
Anole-Zebra-Con (Ours)
Overall
26.46
31.25
Art Style
19.66
35.04
Counting
19.17
15.00
Forensic detection
0.00
20.45
Functional correspondence
17.69
22.31
IQ test
26.67
23.33
Jigsaw
11.33
39.33
Multi-view reasoning
48.12
21.05
Object localization
50.82
45.90
Relative depth
38.71
41.94
Relative reflectance
29.10
27.61
Semantic correspondence
19.42
17.99
Spatial relation
41.26
57.34
Visual correspondence
21.51
26.16
Visual similarity
30.37
44.44
Table 7: Blink: breakdown by category (%).
31

---

Published as a conference paper at ICLR 2026
D BAGEL PERFORMANCE ANALYSIS
We evaluate our Bagel model trained on ZEBRA A-OT across several benchmarks but did not observe substantial improvements over the original model, where the original generates pure text responses. In fact, we even saw slight performance drops on some tasks such as Math Vista. A detailed analysis revealed likely cause of this decline. The Bagel model employs two visual encoders: ViT-based understanding encoder and VAE-based generation encoder. For generated images, the model often produces hallucinations. For example, when instructed remove all red balls from scene, the generated image may also remove yellow balls. When this corrupted image is passed back into the VIT encoder, the representation correctly reflects that both red and yellow balls are missing, lead- ing the model to reason over inaccurate visual information, ultimately reducing accuracy. Instead generating pure text responses avoids such image generation hallucinations.
32

---

Published as a conference paper at ICLR 2026
E
SCAFFOLDING RESULTS BREAKDOWN
Chess
Model
(%)
1MT (%)
2MT (%)
AMMT (%)
22MT (%)
Claude-4 Sonnet
32.95
57.95
67.05
25.00
34.09
Gemini-2.5 Pro
15.07
39.73
39.73
24.66
24.66
GPT-5
45.78
62.65
61.45
16.87
15.66
Graph
Model
(%)
IMT (%)
2MT (%)
A AIMT (%)
2 (%)
Claude-4 Sonnet
8.11
20.72
22.52
12.61
14.41
Gemini-2.5 Pro
1.90
11.43
20.95
9.52
19.05
GPT-5
1.74
14.78
22.61
13.04
20.87
2D Visual Jigsaw
Model
Q (%)
1MT (%)
2MT (%)
A AMMT (%)
22MT (%)
Claude-4 Sonnet
21.74
36.23
62.32
14.49
40.58
Gemini-2.5 Pro
34.38
56.25
59.38
21.88
25.00
GPT-5
62.86
77.14
85.71
14.29
22.86
Maze
Model
(%)
IMT (%)
2MT (%)
AIMT (%)
A2MT (%)
Claude-4 Sonnet
35.06
58.44
94.81
23.38
59.74
Gemini-2.5 Pro
59.70
85.07
97.01
25.37
37.31
GPT-5
63.01
86.30
97.26
23.29
34.25
3D Multi-Hop Counting
Model
Q (%)
IMT (%)
2MT (%)
A AMT (%)
22MT (%)
Claude-4 Sonnet
59.68
67.74
74.19
8.06
14.52
Gemini-2.5 Pro
45.95
48.65
51.35
2.70
5.41
GPT-5
72.58
74.19
77.42
1.61
4.84
Tetris
Model
(%)
1MT (%)
2MT (%)
A AIMT (%)
A2MT (%)
Claude-4 Sonnet
18.87
39.62
45.28
20.75
26.42
Gemini-2.5 Pro
8.57
25.71
31.43
17.14
22.86
GPT-5
30.77
50.00
59.62
19.23
28.85
Table 8: Scaffolding evaluation results across task domains. Q: zero-shot question-only; 1MT: question with first multimodal reasoning step; 2MT: question with first two multimodal reasoning steps. A columns show absolute improvement baseline (Q).
33

---

Published as a conference paper at ICLR 2026
Chess
Model
(%)
1TT (%)
2TT (%)
AITT (%)
22TT (%
Claude-4 Sonnet
32.95
42.05
40,91
9.10
7.96
Gemini-2.5 Pro
15.07
13.70
19.86
-1.37
4.79
GPT-5
45.78
51.81
62.65
6.03
16.87
Graph
Model
Q (%)
1TT (%)
2TT (%)
AITT (%)
ATTT (%)
Claude-4 Sonnet
8.11
20.72
15.32
12.61
7.21
Gemini-2.5 Pro
1.90
5.71
5.71
3.81
3.81
GPT-5
1.74
11.30
19.13
9.56
17.39
2D Visual Jigsaw
Model
(%)
1TT (%)
2TT (%)
AITT (%)
22TT (%
Claude-4 Sonnet
21.74
39.13
30.43
17.39
8.69
Gemini-2.5 Pro
34.38
24.22
25.00
-10.16
-9.38
GPT-5
62.86
68.57
71.43
5.71
8.57
Maze
Model
Q (%)
1TT (%)
2TT (%)
AITT (%)
2TT (%
Claude-4 Sonnet
35.06
49.35
55.84
14.29
20.78
Gemini-2.5 Pro
59.70
23.29
39.73
-36.41
-19.97
GPT-5
63.01
63.01
75.34
0.00
12.33
3D Multi-Hop Counting
Model
(%)
1TT (%)
2TT (%)
AITT (%)
A2TT (%)
Claude-4 Sonnet
59.68
69.35
69.35
9.67
9.67
Gemini-2.5 Pro
45.95
54.84
56.45
8.89
10.50
GPT-5
72.58
72.58
74.19
0.00
1.61
Tetris
Model
(%)
1TT (%)
2TT (%)
AITT (%)
ATTT (%)
Claude-4 Sonnet
18.87
26.42
32.08
7.55
13.21
Gemini-2.5 Pro
8.57
11.54
7.69
2.97
-0.88
GPT-5
30.77
28.85
42.31
-1.92
11.54
Table Text-only CoT' evaluation results across task domains. Q: zero-shot question; 1TT:: first text reasoning step; 2TT: first two reasoning steps. A columns show absolute improvement over Q.
34

---

Published as a conference paper at ICLR 2026
F PROMPT TEMPLATES
F.1 PROMPT FOR ENHANCING RAw REASONING TRACES FOR ONLINE AND AGENTIC DATA
Prompt Template
You are an expert in creating clean and logically coherent multimodal chain of thought traces. Your task is to analyze
and comprehend raw reasoning trace with interleaved text and images, then transform it into clean, step-by-step multimodal
reasoning trace that correctly solves the original problem.
INPUT
1. Problem & Noisy Trace: A raw interleaved text and image reasoning trace. Images this trace are represented by placeholders: [problem Baciilem_mmaee____ xJ for original problem images (e.g.,
[problem_image_1]] [problem_image_]] [reesoning__mage__X for images generated during
reasoning (e.g., [reasoning_
[reasoning__mage_22]
2. Image Data: The actual image data corresponding the placeholders, provided separately.
Your Task
Generate clean, logical multimodal reasoning trace **plain text** that represents the *ideal* reasoning process solve the problem.
OUTPUT FORMAT
You MUST generate the formatted reasoning trace with the following structure:
QUESTION:
<The original problem. statement with text and image
placeholders:
<image_ddd_,, <image_start> 2aaaam__mmmaae___mmee___eee
etc. Stay close to the original problem statement as possible but remove noise ensure clarity>
REASONING TRACE:
THOUGHT 0: <Clear description of initial reasoning step that identifies key elements the problem>
THOUGHT 1: <Next reasoning step, often explaining why an image will be created> Himage_start>|eee (raasssssrreoonnnmmeeeeee [almaag__liimaaeeeeeee_e <image_end> THOUGHT 2: <Further reasoning step based on the image, explaining insights gained>
< [rg_:arre_eeeeooonnnnmeeee______
// Additional thoughts and images as needed <image_start> (reasooingggggiggggee iimageeeenz THOUGHT <Final reasoning step before the answer, summarizing key insights>
FINAL ANSWER:
35

---

Published as a conference paper at ICLR 2026
<The final calculated answer based on the reasoning>
Cuidelines
1. Enhancing Original Trace Rather than Generating New Trace: Instead of generating new trace, your task is
enhance the original trace (which is correct trace but rather concise and lacks coherent multimodal
reasoning) by adding more details and explanations, see the following sections guidelines for more details.
You MUST use all the images provided in the original
trace.
You should use the original trace reference rather than copying it verbatim.
2. Multimodal Reasoning Flow:
Develop coherent, step-by-step chain of thought that seamlessly integrates textual and visual reasoning.
Clearly explain the necessity of generating a sketch visual thought / image before introducing its placeholder.
After each image placeholder, describe the insights gained from the sketch visual thought / image, how it contributes to advancing the solution.
Ensure each step logically builds on the previous ones, especially between text reasoning and visual reasoning steps.
3. Image Placeholders and References:
Use placeholder tags ONLY when you want actually insert/shoow/geeertt an image in your trace. when doing so, write the correspending placeholder tag exactly as shown, including the <image_start> and <image_enee end>> tags.
Each unique image in the original problem and the reasoning trace should be represented by a unique placeholder tag, and each unique placeholder tag should only show up once in the trace.
When referring to images in your explanations,
natural language descriptions (e.g., "the diagram in the question", "the first sketch", "the visual thought x I created") instead of using placeholdes der tags. This to parse into interleaved text and image sequences.
is important because it helps
For images from the original problem, use:
simage_start> (problem_image x]<xi]]]_immaa_a__a For sketches or visuals generated during reasoning, use:
<image_start> [reasoning_image <image_end>
4. Narrative Style:
Remove irrelevant technical details such as debugging info, code snippets, and LaTeX package imports. Eliminate verbose language that do not contribute to solving the problem.
Focus on the essential reasoning path that leads to the correct solution, using concise and clear language describe the overall reasoning process.
36

---

Published as a conference paper at ICLR 2026
F.2 PROMPT FOR ENHANCING PROGRAM GENERATED TEMPLATE DATA
Prompt Template
You are an expert in enhancing multimodal reasoning traces. Your task to transform template reasoning trace into a diverse multimodal reasoning trace that correctly solves the problem, while staying close to the original template and final answer.
INPUT
1. Problem & Template Trace: template with interleaved text and image placcholders:
[problem_image "] for original problem images (e.g.r [probllen_image_ll")
"[reasonin__imaee_""" for images generated during
reasoning (e.g., [eeasoning_image_]l]
2. Image Data: The actual image data corresponding to the placeholders, provided separately.
Your Task
Generate concise multimodal reasoning trace as *plaiin text**.
OUTPUT FORMAT
You MUST generate the formatted reasoning trace with the following structure:
QUESTION:
<Rewrite the problem statement in your own words while maintaining all key information. Do not change key information. Include image placeholders: <image_start> [ simageeeee <image_end>, <image_start> (problem_image.ee. <image_end>, etc.>
REASONING TRACE:
THOUGHT 0: <Identify key elements of the problem>
THOUGHT 1: <Explain reasoning step, often why an image
sketch / visual thought is needed> Simage_start>(rett (bbart>(raasoning_imaaa
<image_end> THOUGHT 2: <Explain insights from the image>
atreessttttttrrreateeee [osl__maae_2___ [em_m]]m_eeenda // Additional thoughts and images as needed <image_ste sttreesttttttttte (reasoning ii___mmgee_ee_ X] <]]miage_eeme end> THOUGHT <Summarizg key insights before answer>
FINAL ANSWER:
<The original final answer in the template, do not change it>
Guidelines
1. Diversifying the Template:
Rewrite the problem statement and reasoning steps in
your own words while preserving all key information. Avoid deviating from the original template reasoning structure. Your job to diversify the text of the original trace, not the logic.
37

---

Published as a conference paper at ICLR 2026
Vazy the language and phrasing to avoid repetitive
patterns.
You MUST use all the images provided in the original
trace.
You MUST keep the original final answer.
Maintain the original template's core reasoning
structure and rationale while introducing textual reasoning refinements rather than substantial changes to the logical flow.
2. Multimodal Reasoning Flow:
Develop coherent, step-by. stee chain of thought that seamlessly integrates textual and visual reascning. Clearly explain the necessity of generating a sketch visual thought / image before introducing its placeholder.
After each image placeholder, describe the insights gained from the sketch / visual thought / image, and how it contributes to advancing the solution.
Ensure each step logically builds on the previous ones, especially between text reasoning and visual reasoning steps.
3. Image Placeholders and References:
Use placeholder tags ONLY when you want to actually insert/show/genertee
image in your trace. When doing so, write the correspending placeholder tag exactly shown, including the <image_start> and Simage_ends tags.
Each unique image in the original problem and the reasoning trace should be represented by a unique placeholder tag, and each unique placeholder tag should only show up once in the trace.
When referring to images your explanations, use natural language descriptions (e.g., "the diagram in the question", "the first sketch", "the visual thought X I created") instead of using placeholder tags. This is important because it helps us to parse into interleaved text and image sequences.
For images from the original problem, use: [problem, aatroblemmmmageeee <immge__end? For sketches or visuals generated during reasoning, use: <image_start> [reasoning_imagee <image_end>
F.3
PROMPT FOR ALGGRRITHMIC PROBLEMS
Prompt Template 2
You are an expert in mathematical problem solving, algorithmic reasoning, visual explanation, and creating multimodal reasoning traces.
1. STRICT VISUALIZATION POLICY (IMPORTANT)
38

---

Published as a conference paper at ICLR 2026
You are only allowed to produce at most visualizations, and they must all appear at the very beginning of your reasoning (within the first 3--4 thoughts). You may only use the following types for these visualizations:
graph
flow_network
tree_from _ict
tree_from_ root grid
After these initial visualizations, you must do all further reasoning purely mentally/textually
with pseudocode- MORE [VIS_SPEC] blocks are allowed after the first 3. Any attempt to include more than 3 visualizations or use a disallowed type will be ignored. The visual reasoning should only be used to understand the setup of the question humans visualize at the beginning to sset the board." The actual problem solving is done purely textually.
General Rules:**
Interleave THOUGHT steps and [VIS_SPEC] image requests.
Your final reasoned solution must match the logic of the given solution code.
Prefix THOUGHT 0 with REASONING TRACE in the previcus line. Prefix each reasoning step with "THOOGGHT n: (n starts at 0, less than 50 words each)
Max 3 [VIS_SPEC] blocks, all within the first 3--4 thoughts.
Diagram #1: raw structural sketch (graph topology, blank grid, etc.).
Diagram #2--3: showcase pivotal elements if helpful. IIIneraaa self-check (no output) *** "Would human scribble this as a quick setup sketch?'! If the answer is no, **do not** emit a VIS_SPEC.
Strictly do not regenerate the same image simply refer to the previous images in text if needed.
Max of 10 thoughts.
Every visualization request **mus*** use minimal [VIS] SPEC] block with the correct type specified. Do not use any other format.
[VIS
**not** include file names, imports, The orchestrator will handle image generation. If you cannot meaningfully visualize or correctly visualize a thought using the provided tcols and inputs, then do not generate an image.
drawing code.
Images are meant to be simple and visually cohesive do not make grandiose images with titles and axis it's simply for baseline understanding of the question.
The first line of the trace should be QUESTION: followed by detailed in depth recap of the problem, specifying all the important aspects, without mentioning the solution.
2. Validity Rules:
39

---

Published as a conference paper at ICLR 2026
A11 [VIS_SPEC] parameters must be valid, fully- formed Python literals.
For [VIS_SPEC] type "grid", the values must be a valid Python 1ist of lists with exactly rows and cols columns (or flat list of length rows cols), and each value should be a number string.
For type graph, tree_from_dict, tree_from_roots and similar, node and edge labels may be stringe or integers, but all structures must be valid Python literals. Never output incomplete or empty Hists/rraassdiccss in [VIS_SPEC] blocks. All lists must be fully closed and contain at least one value, unless an empty structure is explicitly required by the problem.
Do not use variable names, symbolic labels, ellipses, or placeholders (e.g., al, x, \ldots, an) anywhere in the
** [VIS_SPECC] Reference Examples: Your blocks must follow the same format as these.**
[VIS_SPEC)
type: graph
nodes: [,
edges: [ (A, B), (B, C)
[/VIS_SPEC]
[VIS_SPEC] type: flow_network
nodes: [A,
edges: [A, (B,
flows : { (A, 2, (B, 1)
capacities (optional): B), 3, (B,C): 2) [/VIS_5PEC]
s \ldots
\ldots
3. Reflection step inmediately after each VIS SPEC Write new THOJGHT that:
a. Describes what you in the previous generated Lrrtooiinn_iaae__aa.oo Explains how it informs your next reasoning move.
4. FINAL ANSWER
After all reasoning, output "FINAL ANSWER: and your
~ concise solution (pseudocode is sufficient)
5. Formatting and Output Requirements
Everything must be plain text with only the full QUESTION (just the problem itself, not the name of the problem), FINAL ANSWER, REASONING TRACE marker, THOUGHT lines and VIS_SPEC markers.
40

---

Published as a conference paper at ICLR 2026
G IMPACT STATEMENT
All data sourced in this work were either publicly available under open licenses or generated syn- thetically. We ensured that all original content and assets used the dataset creation process respect copyright and licensing terms. No human subjects were involved, and we do not foresee any di- rect harm to individuals communities as result of this work. The dataset is intended solely for academic research to improve multimodal reasoning capabilities in Al systems.
H LICENSES
We list the licenses involved in this work as follows.
Anole-7B model is under Chameleon Research License.
BAGRL--7B-Mo model is licensed under the Apache 2.0 license. finetuned from 1ern2..-7B-Instruee and
model, and uses the FLUX.1-
schnell VAE model, all under Apache 2.0.
ImageNet dataset in under BSD 3 license.
Vi Visull CoT dataset is licensed under CC BY 4.0
MATH dataset (Hendrycks et al. 2021 is under MIT License.
OpenStax Physics books are license under CC BY 4.0.
MIT OCW Physics lecture notes under CC BY 4.0.
Maze datasets licensed under CC BY 4.0.
41


Reviews:
[{"review_id": "GXkcGGfl77", "reviewer": "Reviewer_hQr7", "summary": "The paper proposes a new vision-language reasoning dataset Zebra. It contains 182k reasoning chains featured with interleaved visual and textual reasoning steps. Experiments on two VLM backbones show substaintial performance gains after fine-tuning on the proposed Zebra dataset.", "questions": "See the weaknesses above.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "+ It proposes a high-quality dataset featured with interleaved visual reasoning chains on multiple domains, fill in the gap of previous visual cot datasets\n+ The pilot scalffording experiments and the fine-tuning experiment shows that the dataset could improve downstream visual reasoning performance.\n+ The paper is well written and easy to follow.", "weaknesses": "+ Some details of the dataset curation process is missing. From Appendix A.2 we know that the dataset contains both real-world data and synthetic data, but the author does not reveal the detailed source of the real-world data or the specific process of how the synthetic data is generatd. This is important because we don't know if the evaluation dataset like MathVista or MathVision is contaminated or not. Testing the high-similarity overlap between the proposed dataset and the evaluation benchmark would make the performance gains more convincing.\n\n+ I would suggest that In section 5 the fine-tuning experiment to include fine-tuning on previous CoT dataset for a more direct comaprision of your data quality with previous ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8vNx69v6wH", "reviewer": "Reviewer_8hSM", "summary": "The paper introduces ZEBRA-CoT, a large-scale dataset of 182K interleaved text–image reasoning traces designed to teach models to perform Visual Chain-of-Thought (visual CoT) reasoning. It contains 18 domains across four categories, e.g., scientific reasoning, 2D and 3D visual reasoning, and visual logic games.\nExperiments show that frontier multimodal models (including GPT-5, Claude-4 Sonnet, Gemini 2.5 Pro) perform poorly on ZEBRA-CoT , but their performance improves when given the first one or two multimodal reasoning steps as scaffolding.\nOverall, ZEBRA-CoT provides the first established dataset enabling models to think with images, representing a significant step toward interpretable and general multimodal reasoning.", "questions": "- How is visual coherence between text and generated sketches quantitatively verified?\n- How does model scale affect the emergence of visual CoT?\n- What are the major limitations of the proposed dataset? Is it ensured that it does not contain any noise or mislabeled examples?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper addresses an important research question to enable models to reason visually rather than purely through text. It proposes a novel curated dataset (ZEBRA-CoT) that directly facilitates this capability beyond simple captioning or visual QA.\n\n- The four categories (scientific, 2D, 3D, strategic) demonstrate careful task design and strong diversity, enabling generalization across reasoning types rather than a single narrow task.\n\n-  The scaffolding experiments convincingly show that models benefit from intermediate multimodal reasoning cues, quantifying the value of explicit visual CoT supervision.\n\n- Fine-tuning both Anole-7B and Bagel-7B demonstrates the dataset’s practicality and shows real gains on external benchmarks such as MathVista, VisuLogic, and MMVP.\n\n- Establishing visual CoT as a training paradigm parallels the importance of textual CoT in LLM reasoning, with potential long-term implications for interpretable and grounded AI.", "weaknesses": "- While diversity is emphasized, there is little quantitative or human evaluation of logical coherence, correctness, or visual–text alignment within reasoning traces. Including human consistency checks or inter-annotator agreement would strengthen credibility.\n\n- The paper does not isolate how different types of visual reasoning steps (e.g., sketches vs. crops vs. bounding boxes) contribute to performance. A finer-grained analysis would clarify which modalities drive improvement.\n\n- Many examples rely on synthetic visual reasoning (mazes, jigsaws, diagrams).  Demonstrations on real-world, noisy visual data would better illustrate applicability.\n\n- The limitation of the proposed dataset is not explicitly discussed. Elaborated discussions on the limitations or possible improvements would enhance the clarity of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wa1dktE2ws", "reviewer": "Reviewer_VPmE", "summary": "This paper introduces Zebra-CoT, a large-scale, multi-domain interleaved text-image reasoning dataset designed to address the core issue of scarce high-quality visual chain-of-thought training data. The dataset covers four major categories of natural tasks particularly suited for visual reasoning: scientific problems, 2D/3D visual reasoning, and logical strategy tasks. Models fine-tuned on this dataset achieved significant performance improvements.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Scale and Diversity: The dataset is extensive (over 180K reasoning traces) and covers a wide range of domains (18 domains, 50+ distinct tasks), providing rich and comprehensive learning material for visual reasoning.\n\nRelevance and Naturalness: The dataset is specifically constructed for task types where visual aids naturally enhance reasoning, ensuring high relevance and learning efficiency.\n\nEmpirically Validated Effectiveness: Experiments on multiple mainstream models demonstrate that the dataset significantly improves visual reasoning accuracy and the ability to generate high-quality visual reasoning chains.\n\nThe main contribution is the training dataset for the community.", "weaknesses": "1. After fine-tuning, the model demonstrates improved overall performance. However, to pinpoint which specific subcategories contribute to its gains on benchmarks like MathVision, MathVista, VisuLogic, EMMA, MMVP, Blink, and Vstar, a more detailed analysis is required due to the limited baseline comparisons currently available in these areas.\n\n2. A detailed comparative analysis with prior benchmarks should be provided in the main paper, quantitatively examining aspects such as dataset scale and evaluation dimensions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u7g2E9hnZB", "reviewer": "Reviewer_UEnu", "summary": "The authors propose a visual chain-of-thought (CoT) collection methods that can provide visual CoT trajectories with strong correlation between the text and the images. They use the methods to collect 182k trajectories across 18 domains. They authors also fine-tune two models to support its usefulness of their collected dataset, where the fine-tuned models shows better visual CoT capacity or even get interleaved visual-text reasoning ability from scratch.", "questions": "1. I am not sure whether the analysis in section 4 can support the claim the visual CoT is important. In the experiments, the 1MT and 2MT provides additional finished reasoning steps for the models. It should significantly reduce the difficulty of the reasoning tasks. So the performance gain may come from the provided finished reasoning steps, instead of the visual information itself. I guess it should be compared to a baseline where the test-reasoning steps are also provided as text-only CoT.\n2. I believe a static dataset is useful but not enough because the VLM or LLM with both visual understanding and generation ability are rapidly developing. In my opinion, a better way is to collect data in a dynamic way. For example, the model can generate useful visual CoT by itself and then collect them and finetune on them in an iterative way. Do you believe your data collection method can be extended to such dynamic way?\n3. How to create the visual CoT is an important problem. I believe the images in your section 3 should also contain some information that how to create them. The figure 5 is not directly related to the specific tasks.\n4. A minor suggestion: There are too many related works in section 2 and it is hard to align them to the table 1 because in your table 1, the dataset use their abbreviated name while in section 2, you use the authors' name. I believe it is better to also provide the dataset name in section 2 for better alignment. And I also wonder whether all mentioned datasets are included in Table 1? If not, I suggest to provide a more complete table in the appendix.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The dataset focus on an important problem: how to collect visual chain-of-thought (CoT) data. Their collected dataset provide valuable resources for training better visual reasoning models.\n2. The dataset is large-scale and diverse. It contains 182k traces across 18 domains.\n3. The authors also finetune two models to support the usefulness of the dataset.", "weaknesses": "1. In the data curation process, the images are first prepared and them the text reasoning traces are generated based on the images. It means that the reasoning steps must be pre-decided and cannot easily modified and extended. For these synthetic problems, it also means it heavily depends on the human prior knowledge (especially on some authors manual design?)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)