PixelCraft: A Multi-Agent system for High-Fidelity Visual Reasoning on Structured Images
Authors: Shuoshuo Zhang
Zijian Li
Yizhen Zhang
Jingjing Fu
Lei Song
Jiang Bian
Jun Zhang
Yujiu Yang
Rui Wang
Venue: ICLR 2026 Poster (2026)
Keywords: chart understanding
multi-agent system
visual reasoning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose PixelCraft, a novel multi-agent system that unifies high-fidelity tool agents with a flexible reasoning workflow for structured images such as charts.

Abstract: Structured images (e.g., charts and geometric diagrams) remain challenging for multimodal large language models (MLLMs), as perceptual slips can cascade into erroneous conclusions. Intermediate visual cues can steer reasoning; however, existing cue-based methods are constrained with low-fidelity image processing and linear, rigid reasoning patterns, limiting their effectiveness on complex structured-image tasks. In this paper, we propose PixelCraft, a novel multi-agent system for high-fidelity image processing and flexible visual reasoning on structured images. The system comprises a dispatcher, a planner, a reasoner, critics, and a set of visual tool agents. To achieve high-fidelity processing, we construct a high-quality corpus and fine-tune an MLLM into a grounding model, whose pixel-level localizations are integrated with traditional computer vision (CV) algorithms in tool agents. 
Building on this foundation, PixelCraft facilitates flexible visual reasoning through a dynamic three-stage workflow of tool selection, agent discussion, and self-criticism. 
Moreover, unlike prior linear reasoning patterns that simply append historical images, PixelCraft maintains an image memory to allow the planner to adaptively revisit earlier visual steps, explore alternative reasoning branches, and dynamically adjust the reasoning trajectory during discussion. Extensive experiments on challenging chart and geometry benchmarks demonstrate that PixelCraft significantly improves visual reasoning performance for advanced MLLMs, setting a new standard for structured image reasoning.

Published as a conference paper at ICLR 2026
FIXELCRAFT A MULTI-AGENT SYSTEM FOR HIGH-FIDELITY VISUAL REASONING ON STRUCTURED IMAGES
Shuoshuo Zhang Zijian Li Yizhen Zhang Jingjing Fu Lei Song
Jiang Bian Jun Zhanget Yujiu Yang"i Rui Wang
Tsinghua University "Hong Kong University of Science and Technology
Microsooft Research Asia
[zss24, changyizan 2hee24
www.llsssstinnuuu..c.o cn, zijian 1i@connect. ust. hk
{jifu, lesong, jiabia, ruiwa wwwwaw/miwceeeeee.....ee www.oommt.oooooooooooooooooo ee jejzhanguust hk, yang. wwwpyiess.ssiinnnnnuu edu.cc
ABSTRACT
Structured images (e.g., charts and geometric diagrams) remain challenging for multimodal large language models (MLLMs), as perceptual slips cascade into erroneous conclusions. Intermediate visual cues can steer reasoning; however, ex- isting cue-based methods constrained with low-fidelity image processing and linear, rigid reasoning patterns, limiting their effectiveness on complex structured- image tasks. In this paper, we propose PixelCraft, novel multi-agent system for high-fidelity image processing and flexible visual reasoning on structured images. The system comprises dispatcher, planner, reasoner, critics, and set of vi- sual tool agents. To achieve high-fidelity processing, we construct high-quality corpus and fine-tune an MLLM into grounding model, whose pixel-level lo- calizations are integrated with traditional computer vision (CV) algorithms in tool agents. Building on this foundation, PixelCraft facilitates flexible visual reasoning through a dynamic three- stage workflow of tool selection, agent discussion, and self-criticism. Moreover, unlike prior linear reasoning patterns that simply append historical images, PixelCraft maintains image memory to allow the planner to adaptively revisit earlier visual steps, explore alternative reasoning branches, and dynamically adjust the reasoning trajectory during discussion. Extensive exper- iments on challenging chart and geometry henchmarks demonstrate that Pixel- Craft significantly improves visual reasoning performance for advanced MILMs, setting new standard for structured image reasoning. Our code is available at https::/githuut http://www..cooooooooooooo//oeaa
INTRODUCTION
Structured images, such as charts and geometric diagrams, pose formidable challenges to current multimodal large language models (MLLMs) (Wang et al. 2025 Yin et al. 2024; Li et al. 2025a Xia al. 2025). Whereas natural images are typically characterized by features such as objects, textures, and local visual patterns, which can be effectively captured by existing vision models, structured images encode symbolic and structural elements such as coordinates, data points, line connections, and numerical annotations. Interpreting these structural representations requires pre- cise symbolic abstraction rather than mere pattern recognition. Moreover, reasoning over structured images requires much higher granularity and accuracy. While coarse visual features may suffice for understanding natural images, even subtle differences in structured images (e.g., slightly misreading the height of single bar) dramatically alter the interpretation and downstream reasoning.
To enhance the understanding and reasoning ability of structured images, initial strategies, repre- sented by chain of thought (CoT) (Wei et al. (022). focused fine-tuning with specialized textual
"Equal contribution, author order does not indicate contribution. Work done during internship at Microsoft. Corresponning authors.

---

Published as a conference paper at ICLR 2026
What the difeeenne betwoen the statsloend decline HOVz in the tep-right ponel (e-80 K-5) and Be middle-kett parel (0,20, K-tup (roudal 0.3)
COT
Ressowng:
Visual CoT
Resecring
Topatiht pore (1-01 K-4);
Top-iaat pascl to 60, K-5t
Rie-dy (o in
Raoed, 100v 1.0.
Rho HDV2
L.HOV2.s
1.0-
Midile pond 20,
Mideeleft
Emprrea Deargeo will
by umel Oomngg wi))
Rho HON 8.0
Rac-t, HOV
Ria 10, B0V 0.4.
At Racnt, B0V2
Dedine -1 0-4m 0.55
Dedine 0.95 025--06-00 0.5..
Diffronee 55 0075 M
Dafference 0.35 022 D.5.
Mesking invelevant data Romdal
PixelCraft
Empiical Covarage rid0
Emppil Cowerage,
Searoring
Tup-hatt pand candi Kod):
Rhoei, MMPO in Faadlie -1 9.00
cle Mesking legand
B.ubFigure cropping
Ressoning
Mibll-ki panol (m=30) K-ilit
Rie HOVe uSS
At Rie-l, HOV U.3. Dedine D.9- 0.3 -i
Subfiggre crogping
Masking data with loyond
Ditfernnce 066
Figure Comparison between CoT, visual CoT, and our proposed PixelCraft. Compared with existing methods, PixelCraft enables high-fidelity image processing and flexible visual reasoning.
reasoning paths Masry et al., 2024b Huang et al., 2023 Xu et al. 2024). However, relying solely on textual reasoning often leads to the loss of tineigr spatial and structural information, mak- ing difficult to capture patterns such as subtle visual differences or geometric constraints. Recent studies have attempted to address this limitation by constructing intermediate visual clues to support visual CoT (Meng et al. [023] Hu et al. 2024, Fu et al.) (025). Despite this innovation, existing methods either depend on underlying source codes, which often unavailable, provide only low- fidelity image processing. Consequently, their applicability is restricted to a specific set of structured images, and their performance remains limited on increasingly complex and realistic benchmarks, such as CharXiv (Wang et al. 2024) and ChartQAPro (Masry et al. 2025).
Besides high-fidelity image processing, structured image understanding also requires accurate and flexible multi-step visual reasoning. However, most existing studies focus on one-step editing or adopt chain-like linear paradigm, where each intermediate image is derived solely from its prede-
cessor (Hu et al. 2024 Fu et al. 2025 Kumar et al. 2024). This linear approach imposes a cognitive rigidity, forcing the model into monologue of one-way reasoning. Sophisticated visual analysis, however, is inherently a non-linear process of hypothesis testing, formulating premise, exploring its implications visually, and, upon encountering contradictions, backtracking to revise earlier as- sumptions. While few works on natural images have hinted at non-linearity through zoom-in or multi-region marking (Yang et al., 2023a [Su al. (025). they cannot support the rich, recursive exploration needed for structured images. As the example illustrated by Fig. [ flexible visual rea- soning is needed for structured image understanding, which requires the ability to recall historical images, explore multiple reasoning branches, and dynamically adjust the trajectory.
To fill this research gap, we propose PixeICraft, a multi-agent system designed for high-fidelity image processing and flexible visual reasoning of structured images. The system comprises dis- patcher, a planner, a reasoner, a planning critic, a visual critic, and a suite of visual tool agents. To dismantle the barrier of low-fidelity processing,
architect a synergistic approach: a compact MLLM (Qwen2.5-VL-3B (Bai et al.] (2055), fine-tuned on our synthetic corpus for precise pixel- level grounding, acts as a "smart eye" to map textual references coordinates. These coordinates then drive classical computer-vision (CV) operators within our tool agents, which act as "robotic hands" to perform precise image edits. To break free from the cognitive straightjacket of linear reasoning, we propose three-stage, planner-centric workflow: 1) the dispatcher performs query- aware agent selection; 2) the planner coordinates role-driven discussion among agents to assemble the visual reasoning process; and 3) planning critic inspects the trace for errors, triggering re- reasoning. Crucially, we introduce a maaneer-mnagge image memory that functions as a "cognitive whiteboard." Instead of indiscriminately feeding all historical images into ever-expanding con- text. the planner stores intermediate clues and selectively recalls them for subsequent steps. This enables flexible branching and backtracking, departing from the ephemeral, one- one-way nature of prior visual CoT methods while simultancously reducing long-contex. overhead. The main contributions of our work summarized as follows:
2

---

Published as a conference paper at ICLR 2026
We propose PixelCraft, novel multi-agent system for structured image reasoning that inte- grates query-aware agent selection, agent discussion, and iterative corf-correttion with a planner- managed image memory. It features planner-manage image memory that selectively recalls prior visual states instead of streaming all images, enabling explicit branching and backtracking while keeping context compact. This non-linear, discussion-centric workflow departs from elassic visual-interleaved chains and delivers flexible, high-fidelity reasoning on structured images with reduced long-context overhead and fewer visual errors.
PixelCraft enables high-fidelity, pixel-level image processing to enhance the visual reasoning per- formance structured images. We construct a high-quality corpus finetune compact MILM (Qwen2.5-VL..) VL-3B) for high-fidelity, pixel-level grounding: its precise coordinates drive classical CV operators inside tool agents, yielding precise, robust and faithful edits.
PixelCraft shows strong empirical gains on structured- -maar benchmarks with component-wise evidence. The gains consistent and significant across widely used and challenging benchmarks (CharXiv, ChartQAPro, EvoChart) and hold across diverse advanced backbones (GPT-40, GPT- 4. 4.1-mini, and Claude 3.7 Sonnet).
2 RELATED WORKS
Visual understanding of structured images. The visual understanding of structured images like charts remains significant challenge for multimodal models (Wang et al. al.] [2024] Masry al., [025] Xia et al. 2025)] Yang et al., 2024). Pioneering works adopted a two-stage approach, using OCR for data extraction followed by LLM reasoning (Liu et al.] 2022 Lee et al. 2023). To enhance
the reasoning capabilities, subsequent research leveraged Chain of Thought (CoT) data to fine-tune
models (Masry et al., 2022 Han et al., 2023 Masry et al., 2024b a Meng et al. 2024 Li et al,
2025b), with recent works introducing reinforcement learning (RL) (Chen et al. 2025). Despite
these advances, performance remains unsatisfactory on challenging benchmarks, particularly when relying solely models' intrinsic capabilities.
Tool use for visual reasoning, Recent advancements in MLLMs have been significantly propelled
by their integration with external tools like OCR and search engines (Liu al.) 2025 Yin et al., 2024 Wang et al. [0225] Li et al. 2025a). Early tool-augmented methods, such as MM-REACT Yang al. (023b) and ViperGPT (Surís et al., (003)), primarily generate textual or code-based outputs, orchestrating calls to vision models or generating Python scripts to solve visual queries. To address more complex tasks, subsequent line of research has focused on generating intermediate visual evidence. A prominent example, Markkf (SoM) (Yang et al.] (223a),, overlays explicit markers onto image regions to provide the model a clear visual vocabulary. a paradigm successfully extended to tasks like visual grounding (Jia et al. 2024 Wu et al.) 2024] Gao et al. 2024 Liu al., 2024) and object detection (Zhou et al. 2024 Hu et al. 2024). Our work diverges by focusing on visual reasoning for structured images, domain demanding high-fidelin processing. Recent works Refocus (Fu et al.) 2025) and ThenkIin (Su et al.] (025)) also employ visual tools for chart understanding. However, their toolsets often specialized for example, relying contour line detection- which limits generalizability. Moreover, their linear, chain-like reasoning is inflexible: it lacks backtracking and branching while overlooking recent findings that MLLM performance degrades on multi-image inputs (Tian et al. 2025: Zhang et al. 2025d).
Multi-agent collaboration. The multi-agent collaboration paradigm enhances reasoning beyond single-agent systems. A prominent approach is "multi-agent debate", where agents independently generate solutions and then converge on final answer through aggregation or discussion, thereby improving reasoning quality (Liang et al.) 2023] Du et al.) 2023). This principle of leveraging solution diversity is also evident in frameworks for voting (Wang al., 2022), medical decision- making (Kim et al.] (2024). and group discussion (e.g., ReConcile (Chen et 2024)). ). In contrast to these "debate--add-sleec" methods, a distinct paradigm focuses on structured collaboration among specialized agents (Wu et al. 2023: Li et al. 2023). A popular methodology is role-playing (Shao) jet al. 2023), , where a task is decomposed and collaboratvelly addressed by agents in specific roles, like a planner or reasoner (Wu et al. 2023). Our work builds upon this role-playing concept by defining a team of agents with synergistic capabilities for planning, image processing, reasoning, and criticizing. While recent work explores dynamic tool generation (Zhang al. 2025b), creating
3

---

Published as a conference paper at ICLR 2026
Query-aware Agent Selection
Role driven Agent Discussion
What is and decline of HOVz in the lop-ightt panel (n-60, K-5) and the middl-seff ponel (n-30, K-10y, (rounded 0.1)
between statolo-
Planner
Reasener and Selected egents
Visual Critic
Naabyii quessonn wabligur crepprng agret
Tosl Agents ehhigur copping...
@ TTool opents:
The correct... @Recstonr::
a. Laee to dacass with thcrooeee
Roctonert Topright paned (1-00) 1e-5 Docline 129 00.4 -13.
amwwer
Manory
The figit can real lb asoe the subquery....
Toal Agents
The selected tool Sabfigure coopiig and Making dea with Iegead.
IEl Iterctive Refinement and Self-Correction
Dispatcher
Trijettrics
preezes The final inswer 0.1 iscorre:
Oritic
Figure 2: An illustration of the PixelCraft workflow. The process begins with Agent Selection, where the dispatcher chooses the appropriate tools. Next, during Agent Discussion, the planner coordinates tool agents process the image (e.g., cropping and masking) and the reasoner to perform analysis, with the visual critic providing real-time validation. Finally, the planning critic performs a post-hoc review of the entire process, confirming its correctness.
high-fidelity visual tools automatically remains challenging. Our framework equips dedicated tool agents based on the LLM-generated tools.
3 PIXELCRAFT
We consider the complex visual reasoning tasks of chart and geometric understanding, which re- quire step-by-step intermediate visual clues to obtain the final answer. To achieve this, we propose high-fidelity agents (PixelCraft), a multi-modal multi-agent system providing high-fidelity and flex- ible visual reasoning on structured images. Its design is built on two core principles: synergizing MILM intelligence with CV algorithms for high-fidelity tool use, and enabling flexible, non-linear reasoning through planner-centric, discussion-based workflow. As illustrated in Fig. 2 MLLMs serve as the dispatcher, planner, reasoner, and critics, collaborating with specialized tool agents to exccute three-stage process: 1) query-aware agent selection, 2) role-driven agent discussion, and 3) iterative refinement and self-correction.
3.1 AGENT ROLES
Dispatcher. The dispatcher serves the initial entry point. It analyzes the query's 's requirements and intelligently activates only the most relevant tool agents for the reasoning process, improving the efficiency and response quality.
Planner. Functioning as the "conductor" of the agent orchestra, the planner orchestrates the entire reasoning process. It decomposes the complex query into manageable sub-tasks, selects the next agent to act, and manages the flow of image and text information. A critical feature here is the introduction an image memory. Conventional approaches that feed all historical images into the context suffer from severe long-context overhead and are restricted to a linear, chain-like reasoning pattern. The planner's image memory stores all intermediate visual outputs, allowing it to adap- tively recall any historical image. This enables the exploration of alternative reasoning branches and facilitates more flexible and effective visual reasoning process for complex queries.
Tool Agents. To address the nuanced requirements of visual analysis, we initially attempted to prompt L.MSs to generate specialized tool agents based on the ArxivQA training set (Li al. 2024). However, we found that these auto anto-genneated tools were often ineffective, primarily due to the lack of precise grounding coordinates, or were simply invalid, suffering from incorrect code execution and producing erroneous visual outputs. To overcome these challenges, we adopt two-pronged approach: first, we finetume a grounding model on a meticulously curated dataset (Section 4 to provide accurate pixel-level references. Second, we manually refine the invalid tools to ensure their functionality and correctness. This semi-automated approach, combining LLM-bbsed generation with expert-driven validation, proved crucial for building reliable toolset. Notably, this data-driven
4

---

Published as a conference paper at ICLR 2026
pipeline significantly reduces the manual burden and reliance domain expertise compared to manual design. The detailed process of tool generation and refinement is available in Appendix B
For chart reasoning, develop four visual tool agents: 1) Subfigure cropping crops single sub- figure from a multi-chart image using a textual description (e.g., "subplot at row 2. column 1"). 2) Region magnification zooms into specified region to highlight local details with æ/y-axis ticks. 3) Adding auxiliary lines adds a reference line with s/y-axis ticks. 4) Masking Data with Legend: masks irrelevant data series by identifying the color associated with a specified legend item.
For geometric reasoning, we obtain the tool agents based on symbolic entities (points and lines) and their relations: Point Connection draws dashed line segment between two specified points to vi- sualize their geometric relation. 2) Perpendicular Line Construction constructs line perpendicular reference line and passing through given point. 3) Parallel Line Construction constructs a line parallel to reference line and passing through given point. We also include the code execution tool perform numerical computation for the above two tasks.
The reasoner functions as the system's 's dedicated analysis expert. Given input images and a query, it applies logical reasoning to interpret structured data within charts and geometric diagrams. While various sophisticated prompting strategies could be employed, we deliberately use a brief generic instruction to demonstrate the inherent power of our high-fidelity tool agents and the overall framework, rather than attributing performance gains to complex prompt engineering.
Reasoner.
Planning Critic and Visual Critic. To form a robust, two-layer error correction mechanism, we introduce two distinct critics operating at different stages of the workflow: 1) A visual critic for in-loop verification: Visual tools, unlike deterministic coding, can introduce errors. To ensure the fidelity of the reasoning chain, the visual critic operates in-loop goal satisfaction of the processed image (e.g.. verifying a crop was successful) and answerability of an image before is passed to the reasoner. (2) A planning critic for post-hoc refinement: It scrutinizes the sequence of tool usage and logical steps for inefficiencies or errors, such as using a suboptimal tool or a flawed reasoning path. Its feedback is then used to guide the self-correction stage.
3.2 WORKFLOW OF PIXELCRAFT
The workflow of PixelCraft is query-aware, role-driven, and visual-critic, which enables high- fidelity image processing and flexible visual reasoning. The system's workflow unfolds across the following three stages:
Query-aware Agent Selection. The workflow begins with the dispatcher, which analyzes the in- coming query to determine the necessary modalities and processing requirements. Based on this initial triage, the dispatcher selects relevant subset of tool agents to perform visual reasoning, together with the reasoner. This careful selection ensures that only the most pertinent agents are activated, thereby optimizing both computational efficiency and the relevance of the subsequent process.
Role-driven Agent Discussion. Following agent selection, the planner orchestrates the reasoning process. It decomposes the primary query into manageable subqueries, sequences agent activations, and coordinates all inter-agent communication. At each step, the planner dynamically activates the appropriate agent (either a tool agent or the reasoner) and selects an image from the image memory (which contains all historical visual clues and their corresponding descriptions). To ensure execution, the planner assigns a specific goal to tool agents (e.g., "crop the subfigure row 1 column 1") or poses precise subquery to the reasoner. Throughout this process, both processed images and textual information- such as goals, subqueries, and analytical outputs- are flexibly exchanged among agents via the planner, enabling each subquery to be tackled with the most suitable capabilities.
To prevent visual errors from propagating, we introduce critical inspection steps. After a tool agent processes image, the visual critic evaluates its goal satisfaction, i.e., whether the visual output successfully fulfills the goal assigned by the planner. Furthermore, when a processed image and its associated subquery are sent to the reasoner, the visual critic assesses the image's 's answerability. When an error is detected, the error alert will be returned to the planner for subsequent reasoning, ensuring the reasoning's 's robustness.
5

---

Published as a conference paper at ICLR 2026
Iterative Refinement and Coff-Correction Once initial answer is generated, the planning critic performs a final review of the entire reasoning process, serutinizing each step for accuracy, logical consistency, and completeness. By identifying misused tools erroneous conclusions, the planning critic proposes corrections to the tool list (e.g., adding or removing tools) and provides valuable sug- gestions to refine the reasoning process, such as proposing better subqueries tool usage strategies. This feedback serves as additional input for second attempt where PixeICraft re-answers the query. This procedure enables cycle of self-improvement, enhancing the system's decision- and tool selection over time. The self-correction example can be found in Appendix D.1
Through this structured and collaborative workflow, PixelCraft integrates flexible tool selection, adaptive image memory, robust visual verification, and a self-correcting reasoning process. This synergy achieves accurate, flexible, and interpretable visual reasoning for complex queries.
4 GROUNDING FOR HIGH-FIDELITY IMAGE EDITING
Precise visual grounding essential for downstream image editing tools, such as extracting a spe- cific subplot for further analysis. However, grounding elements within structured visual inputs like charts, scientific plots, and geometric diagrams poses unique challenges due to their abstract and compositional layouts. While the prior work Refocus (Fu et al. 2025) has approached this by locat- ing objects based on specific visual primitives (e.g., lines, squares), its reliance on these predefined structures limits its applicability introduce a hybrid dataset of synthesized charts and geometric diagrams, leveraging it to fine-tune Owen2.5-VL-33 (Bai et al., 2025) for high-fidelity grounding these challenging domains. We curate a hybrid dataset from two complementary sources; (1) programmatically synthesized charts and (2) annotated geometric diagrams.
narrow range of chart types. To overcome these limitations,
Programmatically Synthesized Charts. To generate a large-scale dataset with diverse layouts and precise ground-truth annotations, we propose a multi-stage synthesis pipeline. The initial stage fo- cuses on single-panel charts and involves two primary steps: content specification and diversified rendering. First, we employ GPT-4o (Hurst et al. 2024) to produce structured JSON objects that specify textual elements (e.g., titles, legends, axis labels). Second, we build upon a set of prede- fined base code templates, which are then programmatically augmented and rewritten by GPT-4o to enhance visual and structural diversity. These modified templates are subsequently rendered into images via Matplotlib. The rendering process is instrumented to extract the precise position coordi- nates of all visual elements, thereby generating accurate ground- truth for grounding tasks.
To introduce greater structural complexity and align with common subfigure grounding tasks, the second stage of our pipeline composes multi-panel chart figures. We randomly sample, duplicate, and arrange the generated single-panel charts into complex layouts, creating composite images that feature between and 16 subfigures with randomized spatial margins. In total, our pipeline yielded 53k ground-truth annotation pairs, with 43k extracted from our synthesized single-panel charts and the remaining 10k from multi-panel compositions.
Geometry Annotation. To enhance our model's geometric grounding capabilities, we augment our dataset with 2,000 samples from the Inter-GPS benchmark (Lu et al., 2021). Each sample in this benchmark consists of geometric diagram accompanied by rich annotations of its constituent elements and properties. We process these source annotations to extract the position coordinates of geometric points along with their corresponding textual labels. This curated data is specifically utilized for point-level geometric tools.
We formulate structured image grounding as an autoregressive sequence prediction task. Specifically, given an input image and textual prompt P. the model generates a sequence Y = ........ yr) that jointly encodes the textual answer and the corresponding bounding boxes. To this end, we finetune Qwen2.5-VL-33 VL-3BB (Bai et al. (2025) with our curated dataset, where spatial locations are represented by absolute coordinates to align with the model's 's native grounding format. Full training details are provided in Appendix C.I
Supervised Fine- Tuning.
The finetuned grounding model provides accurate grounding coordinates with the elements in struc- tured images, enabling high-fidee fidelity image processing for visual reasoning. The quantitative evalu- ation and example comparison of the model's grounding accuracy are provided in Section 5.3

---

Published as a conference paper at ICLR 2026
Table I: Performance comparison across different models and evaluation methods
GPT-4a
GPT-4. I-mini
Clando-3.7 3.7-sonnet
Method
CharXiv
CharQQAro
EvoChart
CharXiv
ChanQAPre
EvoChart
CharXiv
CharQQAPro
EvoChart
Direct answer
19.6
52.51
62.61
58.6
37.85
71.28
67.1
62.83
74.16
COT
51.1
56.52
68.64
63.5
62.21
76.64
68.3
66.07
77.92
Debate
50.7
49.38
65.52
62.4
57.75
72.72
67.7
60.58
78.08
Reconcile
52.4
54.36
65.20
63.5
59.49
72.64
68.5
65.97
78.56
Refocus
47.2
16.30
50.88
00.7
57.24
58.96
62.4
58.42
77.06
Ours
58.83
70.24
68.1
65.56
79.44
73.9
A
+5.6
+6.32
+7.60
+9.5
+7.71
+3.16
+6.8
+6,99
+6.32
Table 2: Accuracy on the Geometry3K (Lu et auxiliary- line subset.
2021
Table 3:
Results comparing visual CoT paradigm with ours framework.
Model
Direct Answer
CoT
Debate
Reconcile
Ours
CharXiv ChartQAPro
GPT-40
21.09
17.19
25.00
26.56
26.56
Visual CoT
65.0
61.04
GPT-4,
24.22
26.56
29.69
32.81
34.38
Claude-3. 3omnde-3.7-sone 7-sonnet
30.47
28.13
28.13
32.03
33.59
Ours
68.1
65.56
5 EXPERIMENTS
In this section, we first evaluate PixelCraft chart and geometric reasoning tasks. Ablation stud- ies demonstrate the effectiveness of each component of PixelCraft. Furthermore, the analysis and discussion demonstrate why PixelCraft improves visual reasoning capabilities in the aspects of grounding accuracy, high-fidelity image processing, ficxible visual reasoning, tool usage, and self- correction.
5.1 RESULTS ON CHART REASONING
Benchmarks and Models.
Our evaluation is conducted on three recent and challenging chart rea- soning benchmarks: CharXiv (Wang et al. 2024). ChartQAPro (Masry et al. 2025), and EvoChart (Huang et al. 2025). We utilize the reasoning-focused question set for CharXiv commonly used and the full test sets for both ChartQAPro and EvoChart. Following the protocol from CharXiv to
(Wang et al. 2024). we employ an LLM-as----udge approach using GPT-4. 1il--mmmii220000 evaluate the correctness of the final answers. To ensure fair comparison across all methods, par- ticularly given the long outputs generated by agentic and CoT approaches, we omit the 5% margin typically used for ChartQAPro and apply this unified evaluation standard consistently. We demon- strate the effectiveness of PixelCraft by integrating it with several powerful MLLMs, including GPT-40-20240806. GPT-4.1-mini-2001 1--mmni-022504144 and Claude-3.7-sonnet.
Baselines. We take Direct answer and chain of thought (CoT) methods as the basic baselines. Additionally, we include more advanced baselines in terms of tool using and multi-agent collabo- ration: Tool-using method: Although tool-augmented visual reasoning has been widely explored, chart-specific tools remain limited. To knowledge, Refocus (Fu et al. (025) is representative chart-tool approach, and we use it as our baseline. Multi-agent methods: we select two popular multi-agent methods: Debate (Du et al. 2023) and Reconcile (Chen et al.) 2024). We set up Debate and Reconcile with two reasoners corresponding to the two agents (planner and reasoner) in our work, where these two agents discuss to converge on a final answer.
Main results. Table that our proposed PixelCraft significantly outperforms all the baselines on these three chart-reasoning benchmarks across GPT-4o, GPT-4.1-mini and Claude-3.7-Somal Moving from the naive "Direct answer" setting to chain-of-thought (CoT) prompting already pro- vides a boost (nearly 3-6 %), confirming the general value of explicit reasoning. The multi-agent collaboration methods, including Debate and Reconcile, show little benefit, indicating that multi- ple agents without specialized visual tools struggle to perform visual reasoning on charts. Refocus couples LLM with chart-related tools, performs inconsistently and in several cases lags even the CoT baseline, suggesting that its visual tools are insufficient for complex structured image process- ing. In contrast, PixelCraft, equipped with high-fideliy tools powered by fine-tuned grounding model. consistently achieves the highest accuracy across all benchmarks, demonstrating the clear superiority of our approach.
7

---

Published as a conference paper at ICLR 2026
COT
Owen
GPT-4.
Qwen
Refocus
70
68.1
Accuracy (%)
65.6
65.0
64
63.8
64.1
63.8
62.2
Q: In panel D. which AGe value corresponds to the peak high specificity when TáS, kT? Qwen: -20
60
CharXiv
CharQQA-Pro
Refocus: X
Ours:
Dataset
(a) Comparison of grounding methods.
(b) Illustrative grounding example.
Figure 3: Grounding case and method comparison. (a) Comparison different grounding methods within agent framework, including our fine-tuned model, Qwen2.5- VL--BB and GPT-4.1-ming (b) Grounding case showing the performance Qwen2.5-VL VL-77, Refocus and Ours, with colored boxes indicating grounded regions.
5.2 RESULTS ON GEOMETRIC REASONING
Complex geometric reasoning often requires the visual clues (i.e., auxiliary lines) to solve, where auxiliary lines need to be sketched in the geometric image to help visual reasoning. To construct such a specific geometric benchmark with requirements of visual clues, filter 128 complex test samples that require the intermediate visual clues to answer from the test set of Geometry3K (Lu al.) 2021). Since the tools from Refocus (Fu et al. 2025) and Visual Sketchpad (Hu et al. 2024) do not support geometric reasoning tasks or require the source code to perform image processing, which is commonly unavailable, we omit them in the evaluation.
Main results. The results in Table 2 demonstrate a clear hierarchy of performance among differ- reasoning strategies the auxiliary-line subset of Geometry3K. The effectiveness of baseline methods progresses logically. Compared to a simple "Direct Answer", incorporating a CoT proves beneficial for most models, underscoring the value of explicit reasoning. Furthermore, the multi- agent strategies, Debate and Reconcile, consistently outperform these single-agent approaches. This highlights the power of collaborative frameworks verify logic and reduce errors complex tasks. Building upon these insights, our proposed PixelCraft method consistently surpasses all baselines across every model. It achieves the highest accuracy in all test cases, establishing a new state-of-the- art this challenging benchmark. This consistent and significant advantage validates the superior design and robustness our method for solving complex geometric problems. Detailed case studies and visual illustrations provided in Appendix D.2
5.3 ANALYSIS AND DISCUSSION
To further demonstrate the effectiveness of PixeICraft, we first conduct ablation studies for each component and then explore the question 'Why can PixelCraft improve the performance?". Addi- tionally, we extend evaluation with comprehensive analysis of computational efficiency and generalization to real-world infographics in Appendix E and Appendix respectively.
Ablation studies. To evaluate the effectiveness of each component of our agent, we conduct a role- wise ablation study. As shown in Table 4 introducing the Tool Agents (TA) provides the largest average performance gain relative to the no-component CoT baseline, underscoring the critical need for specialized tools in visual analysis. Adding the Dispatcher (Disp) provides further improve- ments, as filtering irrelevant tools leads to more accurate tool usage. The Visual Critic (VC) also contributes positively by filtering invalid processed images and avoiding erroneous answers. When the three are combined, their synergistic effect brings performance to 67.5% on CharXiv and 64.89% on ChartQAPro. Finally, incorporating the Planning Critic (PC) yields the best overall results of 68.1% 1% and 65.56%, respectively. This systematic analysis validates that each component contributes positively and cumulatively to the agent's overall reasoning capabilities.
8

---

Published as a conference paper at ICLR 2026
(4) Performance
(b) Usage Frequeney
Geometry Tool Performance
(d) Geometry Usage Frequeney
Eapio Naant Satilay
Sorkad
Conolbale
Figure& 4: Effectiveness and prevalence of tool usage with GPT--4.1miie on CharXiv WWag et al., 2024) and Geometry3K (Lu et al. 202I). The performance gain is calculated specifically on the subset of queries where each tool was activated.
TA
Disp
VC
PC
CharXiv
ChartQAPro
Rediom aransyy (9)
65.4
63.8
62.21
x
X
X
65.0
63.66
Number
65.0
V
X
X
65.9
64.43
V
X
66.0
63.96
67.5
10
X
67.5
64.89
V
V
68.1
65.56
Refinement round
Table 4: Role-wise ablation over Tool Agents (TA), Dis- patcher (Disp), Visual Critic (VC), and Planning Critic (PC).
Figure 5: Critic-based erroneous query identification (TP/FP counts) and post- refinement accuracy.
Comparison with visual CoT. To validate the architectural benefits of our agent framework, we conducted controlled comparison against simplified Visual Chain-of-Thought (CoT) baseline, which simply includes all the historical images for reasoning. This baseline utilized the exact same prompts and tools but employed single planner handle both task planning and visual reasoning monolithically. As shown in Table 3 our full agent framework significantly outperforms this simpli- fied approach. This result underscores the superiority of our design of the setting of image memory and image selection beyond the visual tools, which allows flexible visual reasoning by allowing image selection and image recall for alternative branches of reasoning.
High-fidelity image edits benefit visual reasoning. Precise localization of chart components is critical prerequisite for accurate data extraction, directly contributing to superior performance on downstream chart analysis benchmarks. To achieve this, we finetuned grounding model that sub- stantially boosts localization accuracy, increasing the overall IoU from 0.26 to 0.93 compared to the down- stream benchmark. As shown in Fig. 3a our proposed agent system with the finetuned grounding model significantly outperforms that with the base model. Fig. 3b provides visual explanation for this performance gap: our model accurately localizes the queried subfigure while both the base model and the Refocus method fail, thus enabling more precise subsequent processing.
base model (details in Appendix D.3). The impact of this enhancement is validated
Analysis on visual tool usage frequency and performance gain. We analyze the tool usage fre- quency and the performance gain of each visual tool. As shown in Fig. 4 LLMs achieve an im- balanced tool calling frequency on CharXiv and Geometry3K, with more preference on Subfigure Cropping and Point Connection. This is because the tool calling query-driven and image-driven. Most of the images on CharXiv are multi-chart images, with queries about deep analysis on a sin- gle subfigure or comparison between subfigures. The queries on Geometry3K mainly require point connection for mathematical analysis. All of the visual tools improve the performance over the base CoT method without visual tools, with significant improvement on some tools. For instance, Mak- ing Data with Legend improves 18.4% accuracy on CharXiv and Parallel Line Construction holds an accuracy improvement of 50% on Geometry3k. This demonstrates the necessity and effectiveness of our proposed visual tool agents.
PixelCraft can identify and refine the erroneous answering. After the answer generation, Pixel- Craft will review the tool usage and the whole reasoning process identify the erroneous answers for the next-round reanswering. We conduct three-round reviewing and reanswering to evaluate the identification and the reanswering accuracy. Fig. 5 shows that most of the identified queries

---

Published as a conference paper at ICLR 2026
incorrect-answenti (true positive), and the identification number significantly drops 0 after 2 rounds. Furthermore, after reanswering with the updated tool and suggestions, PixelCraft refines some incorrect answers, resulting in higher accuracy.
CONCLUSION
In this paper, we proposed PixelCraft, novel multi-agent system that unifies high-fidelity, pixel- level tool agents with flexible reasoning workflow for structured image reasoning. Tool agents, powered by fine-tuned pixel-level grounding model and classical CV operators, deliver precise image edits that preserve critical visual evidence. Building on these trusted tool agents, coor- dinated planner, reasoner, and critic workflows, supported by image memory, enable non-linear, multi-branch exploration and targeted re-reasoning. Across challenging chart and geometry bench- marks, PixelCraft consistently elevates strong MLLMs, establishing standard for reliable and sophisticated multimodal reasoning.
ACKNOWLEDGEMENTS
This work was partly supported by the National Natural Science Foundation of China (Grant No. 62576191) and the Shenzhen Science and Technology Program ZIDCY2025001110530001
ETHICS STATEMENT
This work adheres to the ICLR Code of Ethics. No human subjects or animal experiments were involved in this study. All datasets and language models employed are publicly available, widely adopted in the research community, and contain personally identifiable information (PII) or sen- sitive content. We have taken steps to identify and mitigate potential biases in data selection, model prompting, system design, and evaluation to ensure fairness and avoid harmful or discriminatory out- comes. The agent system developed and evaluated solely for research purposes, with safeguards in place to minimize misuse and unintended consequences. No personal identities were collected, used, or disclosed in this research. All procedures comply with relevant legal, privacy, and data pro- tection requirements, and the study was conducted in accordance with established research ethics standards.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide detailed information regarding our mod- els, prompts, and code. Appendix of this paper includes the training details for our grounding model and the core prompts utilized by the agent framework. Furthermore, the source code for framework is accessible at the following anonymous repository: https: /aannnymous. 4open. sereenssciinnee/re wwwaPeeeerraaaaaaaaa CCraft DB52
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint 13923,, 2025.
Justin Chen, Swarnadeep Saha, and Mohit Bansal. Reconcile: Round-table conference improves reasoning via consensus among diverse llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7066-7085, 2024.
Lei Chen, Xuanle Zhao, Zhixiong Zeng, Jing Huang, Yufeng Zhong, and Lin Ma. Chart-rl: Chain-of-thought supervision and reinforcement for advanced chart reasoner. arXiv preprint arXiv:2507.15509 2025.
Yi Ding and Ruqi Zhang. Sherlock: Self-correctigg reasoning in languaee models. arXiv preprint arXiv:2505.220 22651, 2025.
10

---

Published as a conference paper at ICLR 2026
Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving fac- tuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2023.
Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Florencio, and Cha Zhang. Refocus: Visual editing chain of thought for structured image understanding. arXiv preprint arXiv:2501.05552, 2025.
Timin Gao, Peixian Chen, Mengdan Zhang, Chaoyou Fu, Yunhang Shen, Yan Zhang, Shengchuan Zhang, Xiawu Zheng, Xing Sun, Liujuan Cao, et al. Cantor: Inspiring multimodal chain-of- thought of milm. In Proceedings of the 32nd ACM International Conference on Multimedia, pp- 9096-9105, 2024.
Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A multimodal Ilm for chart understanding and generation. arXiv preprint arXiv:2311.16483, 2023.
Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as visual chain of thought for multimodal language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Muye Huang,
Han Lai, Xinyu Zhang, Wenjun Wu, Jie Ma, Lingling Zhang, and Jun Liu. Evochart: A benchmark and self-training approach towards real-world chart understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 3680-3688, 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 2024.
Zixi Jia, Jiqiang Liu, Hexiao Li, Qinghua Liu, and Hongbin Gao. Dcot: Dual chain-of-thought prompting for large multimodal models. In The 16th Asian Conference on Machine Learning (Conference Track), 2024.
Yubin Kim, Chanwoo Park, Hyewon Jeong, Yik S Chan, Xuhai Xu, Daniel McDuff, Hyeonhoon Lee, Marzyeh Ghassemi, Cynthia Breazeal, and Hae W Park. Mdagents: An adaptive collabora- tion of Iims for medical Docision-making Advances in Neural Information Processing Systems, 37:79410-79452, 2024.
Somnath Kumar, Yash Gadhia, Tanuja Ganu, and Akshay Nambi. Mmetagent: Multi-modal critical thinking agent framework for complex visual reasoning. arXiv preprint arXiv:2405. /8358, 2024.
Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screen- shot parsing as pretraining for visual language understanding. In International Conference on Machine Learning. pp. 18893-18912 PMLR, 2023.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society, Advances in Neural Information Processing Systems, 36:5 195511999-2009 2023.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multi- modal arxiv: A dataset for improving scientific comprehension of large vision-language models. 2024.
arXiv preprini arX3::2403.0022
Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang. Zhenran Xu, Xinyu Chen, Haoyuan Shi. Shenyuan Jiang, Xintong Wang, Jifang Wang, et al. Perception, reason, think, and plan: survey large multimodal reasoning models. arXiv preprini arXiv:2505.049) 2025a.
Zijian Li, Jingjing Fu, Lei Song, Jiang Bian, Jun Zhang, and Rui Wang. Chain of functions: A programmatic pipeline for fine-graine ine-grained chart reasoning data. arXiv preprint arXiv:2503. 16260, 2025b.
11

---

Published as a conference paper at ICLR 2026
Tian Liang, Zhiwei He. Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi- agent debate. arXiv preprint rrXi:2335..99118 2023.
Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, et al. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv preprini arXiv:2504.0199) 2025.
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Eisenschlos. Matcha: Enhancing visual lan- guage pretraining with math reasoning and chart derendering, arXiv preprint arXiv:2212. 12.0:2221,.0966,, 2022.
Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive rea- soning improves large vision-language models. arXiv preprint arXiv:2403. 12966, 2024.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint wrXiv:2155.0416. 2021.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-2276 2022.
Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartinstruct: Instruction tuning for chart comprehension and reasoning. arXiv preprint 0rXiv:2403.09008, 2024a.
Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, and Shafiq Joty. Chartgemma: Visual instruction-tumde tuunonntttnnnggg for chart reasoning in the wild. arXiv preprint arXiv:2407.04172, 2024b.
Ahmed Masry, Mohammed Saidul Islam, Mahir Ahmed, Aayush Bajaj. Firoz Kabir, Aaryaman Kartha, Md Tahmid Rahman Laskar, Mizanur Rahman, Shadikur Rahman, Mehrad Shahmoham- madi, et al. Chartqapro: A more diverse and challenging benchmark for chart question answering. arXiv preprint arXiv:2504.05506 2025.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In Proceedings the IEEF/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. Chartassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning. arXiv preprint arXiv:2401.0233 2024.
Fanxu Meng, Haotong Yang, Yiding Wang, and Muhan Zhang. Chain of images for intuitively reasoning. arXiv preprint arXiv:2311.0924/ 2023.
Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. Character-Ilm: A trainable agent for role- playing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13153-13188 2023.
Zhaochen Su, Linjie Li. Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjic Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, et al. Openthinkimg: Learning to think with images via visual tool reinforcement learning. arXiv preprint arXiv:2505.08617, 2025.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python exccution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11888-1 11998, 2023.
Xinyu Tian, Shu Zou, Zhaoyuan Yang, and Jing Zhang. Identifying and mitigating position bias of multi-image vision- Languale models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 10599-10609, 2025.
12

---

Published as a conference paper at ICLR 2026
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou, consistnny improves chain of thought reasoning in language models. arXiv preprint arXhv:2233.11171, 2022.
Yaoting Wang, Shengqiong Wu, Yuecheng Zhang, Shuicheng Yan, Ziwei Liu, Jiebo Luo, and Hao Fei. Multimodal chain-of-thought reasoning; A comprehensive survey. arXiv preprint arXiv:2503.12605. 2025.
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. Charxiv: Charting gaps in realistic chart understanding in multimodal Ilms. Advances in Neural Information Processing Systems, 37:113599-113697, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 25:24824-24837, 2022.
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen 11m applications via multi- agent conversation. arXiv preprint arXiv:2308.0m 205555508.08155, 2023.
Yixuan Wu, Yizhou Wang, Shixiang Tang, Wenhao Wu, Tong He, Wanli Ouyang. Philip Torr, and Jian Wu. Dettoolchain: A new prompting paradigm to unleash detection ability of milm. In European Conference on Computer Vision, pp. 164-182. Springer, 2024.
Renqiu Xia, Mingsheng Li, Hancheng Ye, Wenjie Wu, Hongbin Zhou, Jiakang Yuan, Tianshuo Peng, Xinyu Cai, Xiangchao Yan, Bin Wang. et al. Geox: Geometric problem solving through unified formalized vision-language pre-training, In ICLR, 2025.
Zhengzhuo Xu, Bowen Qu, Yiyan Qi, Sinan Du, Chengjin Xu, Chun Yuan, and Jian Guo. Chart- moe: Mixture of diversely aligned expert connector for chart understanding. arXiv preprint arXiv:2409.03277, 2024.
Cheng Yang, Chufan Shi, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, et al. Chartmimic: Evaluating Imm's 's cross- cross-modal reasoning capability via chart-to-code generation. arXiv preprint arXiv:2406.0990
2024.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310. 11441, 2023a.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11388 2023b.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11 11(12):wwa4403 2024.
Di Zhang, Jingdi Lei, Junxian Li, Xunzhi Wang. Yujic Liu, Zonglin Yang. Jiatong Li, Weida Wang, Suorong Yang, Jianbo Wu, et al. Critic-v: Vim critics help catch vim errors in multimodal reason- ing. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9050-9061, 2025a.
Jayy Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hus Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong. Jinlin Wang, et al. Aflow: Automating agentic workflow genera- tion. In The Thirteenth International Conference on Learning Representations, 2025b.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. Qwen3 embedding: Advanc- ing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05174 2025c.
13

---

Published as a conference paper at ICLR 2026
Yizhen Zhang, Yang Ding, Shuoshuo Zhang, Xinchen Zhang. Haoling Li, Zhong-zhi zzi Li, Peijie Wang, Jie Wu, Lei Ji. Yelong Shen, et al. Perl: Pemutatiion-chaance reinforcement learning for interleaved vision- lsionnlanguage reasoning. arXiv preprint arXiv:2506.14907 2025d.
Qiji Zhou, Ruochen Zhou, Zike Hu, Panzhong Lu, Siyang Gao, and Yue Zhang. Image-of-thought prompting for visual reasoning refinement in multimodal large language models. arXiv preprint arXiv:2405.13872, 2024.
14

---

Published as a conference paper at ICLR 2026
A USE OF LARGE LANGUAGE MODELS
In adherence to ICLR 2026 policy, we disclose the assistive use of Large Language Models (LLMs) in the preparation of this paper. The models were employed to refine grammar and improve the clarity of the text. Furthermore, an LLM was utilized to synthetically generate a portion of our dataset and tool codes. The authors have reviewed all LLM-generated contributions and take full responsibility for the content and integrity of this work.
B AUTOMATED GENERATION OF VISUAL TOOLS
To automate the discovery and creation of visual tools, we leveraged GPT-4.1-ming programmat- ically generate potential tool candidates. We initiated this process by first randomly sampling 500 questions from ArxivQA (Li et al. 2024), which comprises diverse set of charts from arXiv papers. Following this, we engineered detailed prompt designed to instruct the model to generate visual tool codes and corresponding tool descriptions.
In total, this process generated 468 tool candidates. To systematically organize the output, we first encoded the name and description of each tool into a vector representation using Qwen3- Embedding-8B (Zhang al.) 2025c). We then performed clustering analysis these em- beddings consolidate duplicates and identify key functional categories. An illustrative sample of the resulting clusters is provided in Fig. 6a and Table 5 Our analysis of these clusters vealed that the vast majority of the generated tools fell into a few recurring functional categories, primarily those demanding strong visual grounding capabilities like Subf Subfigure Cropping and Adding Aidding-Aurilaa Awwiliarry...... LaiiirrrrrTnn..... Based on this finding, we consolidated the most promising tools from these recurring categories. To improve their robustness and generality, these selected tools were first processed by the GPT-o3 for automated rewriting, followed by a final phase of manual tuning to ensure their correctness and practical usability. This curated of high-fidelity visual tools formed the core of our agent's toolkit.
Table 5:
Cluster statistics and representative tool descriptions.
Cluster
Cluster Size
Tool Definition (examples)
Summary Tools
www.vertical..pe .gu.d.llg.edddeeeeeeee de_line
148
dra..bori.omtal.gnide...... mark.chreshold
Adding Auxiliary Lines
highliggt..subfigrrs
1
128
crop.subfiguze
Subfigure Cropping
isolate subplot
bighliggtt.srries.aas..
101
highlightt.g..ourve
Masking Data with Legend
bighlight..dtt..piints
mamniEy.eggon..rommddx
3
64
crop and magnify region crop_region
Region Magnification
find.peaks.in.dda bighlight.max www.mma..pppo. max.point crop beta vs time panel hi igghlighttnttrsee intpontt draw spectral type guides
Others
27
Useless tools
To probe the breadth GPP-4.1-minif generative capabilities beyond this initial set, we conducted second experiment where we explicitly constrained the model from producing the known grounding- based tools. This constraint prompted shift towards generating significantly more complex tools, whose procedures were heavily reliant on classical computer vision primitives for precise object detection and data extraction, with representative examples including detecting all data points in a scatter plot to compute their mean, locating the intersection point of two plotted lines. However, our empirical evaluation revealed that these novel computational tools consistently exhibited low accuracy, even after being subjected to the same GPT-o3 rewriting process (as depicted in Fig. 6b).
15

---

Published as a conference paper at ICLR 2026
t-SNE of Generated
Casro Cass Casoa Olber
RegDG
20
CHMAM
ReCoSa
E-NN cim 2
1OT7 100 RAd
NEXUS
VHRED+BOW
-10
Seq2Seq+Att
20
t-SN dim
0.00
0.02
Perturhation (O)
0.54
0.08
0.00
0.10
(a) Clustering visualization
(b) Line-intersection case
Figure 6: Left: a t-SNE projection of the generated tool embeddings. Right: representative case in which classical computer-visie vision utilities detect the intersection of two lines (red dots) on a chart.
Given the critical impact of tool fidelity on the agent's downstream reasoning performance- and the high risk of error propagation from unreliable tools made the strategic decision to exclude this latter set from our agent's 's final toolkit during evaluation.
C IMPLEMENTATION DETAILS
C.I TRAINING DETAILS
We performed full-parameter supervised fine-tuning on Qwen2.5-VL-3B-Inss (Bai et 2025 with our custom-collected dataset of 53,000 samples. The model was trained for single epoch on four NVIDIA A100 GPUs using a learning rate of 10-5, representative training prompt shown below:
Example Prompt
You are a helpful assistant specialized chart analysis. Examine the provided chart image and return the pixel coordinates of the requested element, or return "Not found" if the element does not exist in the image.
System:
User: (example pppgm This is chart image. Please locate the pixel coordinates of >>e subfigure row 2, column 2< object.ref.ende >
< </object.ref.ster
in the image.
Assistant: <lbox.start >> [x1, yl, x2, <2]<|box.ent
C.2 VISUAL TOOL IMPLEMENTATION DETAILS
This section provides a more detailed breakdown of the implementation for the visual tools intro- duced in Section 3.1 A core component across these tools is specialized grounding model, which localizes the elements specified in the planner's query into precise pixel coordinates within the image.. The operational flow for each tool is follows:
Subfigure Cropping: This tool first employs our grounding model to localize the bounding boxes of both the target subfigure and any relevant legends (e.g., "the legend for plot A") based on the textual description. If multiple elements are identified, such as a chart and its corresponding but spatially separate legend, the tool programmatically crops these regions and composes them into single, coherent image for analysis.
Region Magnification: For this operation. the planner provides the start and end tick values on the æ and y axes that define the region of interest. The grounding model is then tasked with identifying the precise pixel coordinates corresponding these tick marks.
16

---

Published as a conference paper at ICLR 2026
1.0
0.8
0.8
04
cooperation
(a) Subfigure Cropping
100
1147
10
20
(b) Region Magnification
Masking Data with Legend
CAOE CANE 3017 (000))
CAND AA rolrgg
ACAME
AGANE Phan-res
Toaus
DVDS
moaee" oovcl Teaus 1V00 OSA
oovol
Reaffr
(d) Adding Auxiliary Lines
Figure 7: Examples of visual tools for structured images: (a) cropping subfigures, (b) magnify- ing regions, (c) masking elements by legend, and (d) adding auxiliary lines.
Subsequently, the tool performs crop operation based these coordinates and resizes the resulting region to achieve magnification, preserving local details.
Similarly to region magnification, this tool leverages the ground- ing model determine the pixel locations of specified tick values on an axis. It then renders a horizontal or vertical line at this position to serve as visual reference for comparison or thresholding tasks.
Adding Auxiliary Lines:
Masking Data with Legend: This tool operates in two-stage process. First, the ground- ing model localizes the specified legend item (c.g., "the line representing Group A") within the chart. Second, the tool extracts the dominant color from the localized legend's 's icon. This extracted color value is then used create binary mask for the corresponding data series the plot area, effectively isolating it from other visual elements.
Furthermore, our Geometric Tools share a unified implementation pipeline where the grounding model is first invoked detect the key points (e.g., line endings, vertices) specified by the query. The tool then programmatically renders the desired line based on these geometric constraints. Illustrative case studies for these tool implementations presented in Fig.
17

---

Published as a conference paper at ICLR 2026
C.3 PROMPT TEMPLATES
Planner
You are an expert on chart understanding to analyze the question step by step until an- swering the final answer. You are given the chart figure < immaee >> with the image path being < image.gappp. pe..peeppa....... and need to solve the following question: quest i on > The question can be solved via the intermediate processed image using tools. To solve the complex task, you decompose it into
simple subquestions and use tools generate intermediate reasoning processes until getting the final answer.
Available Tools
Use the following tools to analyze and process images:
@ttool.eeccrppte
Image Pool < image_pool >>
Instructions
1. Workflow:
Actions should be logical and tool-driven rather than directly outputting the answer.
Each THOUGHT analyzes progress; if the final answer is reached, output it in FINAL ANSWER with TERMINATE in the ACTION section.
Select tools based on their descriptions and constraints; if one fails, try alternatives.
Always use image paths from the image pool for tool inputs.
2. Final Answer:
Provide concise summary when the solution is found, then end with TERMINATE.
3. Format: Follow the structure strictly:
THOUGHT N: [Analysis]
ACTION tool_name (key=value)
OBSERVATION N: [Result]
Response History: < history|
Current Step Template:
THOUGHT (i): [Your analysis]
ACTION (i):
(key=value)
OBSERVATION {i): [Result or observation]
Reasoner
System: You expert in chart analysis and data visualization. Please analyze the chart carefully and provide accurate answers based on the visual data presented.
User: (example png) {question}
Visual Critic
Please check whether the given image contains enough information answer the question: questiio inssiiinn >
Examples:
If the question requires value extraction from chart, the image should contain the chart with clear axis and data points.
If the question about the trend of chart, need not contain the chart title or axis labels.
If the question is about one subchart compared to another, it is acceptable to have only one subchart in the image.
18

---

Published as a conference paper at ICLR 2026
If the image is extremely incomplete the question is not related to the image, return
f False.
If the image is complete and the question is suitable for answering with the image, return true.
Planning Critic
You are helpful Al assistant. You are given question and chart image. You need to reason with the question and the chart image, and then evolve the plan to answer the question by adjusting the tools and the corresponding parameters. If the current plan is perfect, you should maintain the current plan.
Available Tools: tttollldee ttto:://ooooooooooitaooa
Question: < | question
Reasoning Process:
< current plan|>
Instructions:
1. Analyze the question and the answer to assess the correctness the final answer based on the image.
2. Some intermediate outputs from tools may be incorrect; analyze the correctness of the extracted information.
3. If the final answer is correct, mark ADJUSTMENT: False; otherwise, mark ADJUSTMENT: True.
4. If the final answer incorrect, analyze the flawed reasoning process and extracted in- formation, with ADJUSTMENT: True.
5. If some tools failed them from the tool list. Output the updated tool list as:
incorrectly applied during the reasoning process, remove
tools: [tooll, tool2, tool3]
6. If ADUUSTEEN:: True, provide detailed suggestions for the incorrect parts.
Dispatcher
You are an expert chart analyst specializing in automated tool selection for chart analysis tasks.
Task Overview Given chart image and a related question, you must analyze both inputs and select all possible tools from the list that can help answer the question. Available Tools < tool descripts ons Selection Guidelines
1. Priority Order: Consider tools in order of their relevance to the question type.
2. Empty Selection: Return [] if no tools are suitable.
3. Context Awareness: Consider the description and constraints of each tool carefully.
Analysis Framework
1. Question Analysis: Identify the core task (counting, comparison, extraction, calcula- tion).
2. Chart Assessment: Determine chart structure (single/multi-chaal complexity).
Tool Matching: Map identified needs to available tools.
4. Validation: Ensure selected tools satisfy constraints and requirements.
19

---

Published as a conference paper at ICLR 2026
the Distance-[Fe/t] (Fere, what the label of the data which has larger average [Fe/H] value?
Original Solution
Planner:
The question will solve this question direetly with
reasoner...
Reasoner: Step I: Identify relevant subplot the question... The data
471
labeled black points have larger average [Fc/H] value Distance- [Fc/H] plot.
10
474
Critic
Suggestion: 1. The analysia should focus only on the red and blue points as per question. Use Subfigure Cropping tool zoom on bottom: left subplot (Distance [Fe/lI))
saa "2.0
[Fe/H] < -2 [Fe/H] 22225
(ongrn))
Final Solution
Planner: The question asks... To analyae this clearly, I will zoom in on this specific subplot
Reasoner: The extracted information cleatly states that the red points labeled as "[Fo/H] a lurger average [Fe/H] points.
Acouth tasn
Figure Illustration of celf-correction.
Q: In O G. LAGB 30 and CG GD. Find BCD
Ours
Planner:
To find the measure of are BCD To better
visualize and confirm the relationships, I will connect points
GPT-4.1-min:: Given LAGB 30 CG GD, so 4CGD = 90" If circle arranged symmetrically (as is often the case in these problems), are BC may also measure 30".
We are given circle with center G, and the angle... we can find BGCb subtracting from ACC ... the angle BGC is 60"
Reasoner:
Final Answer: 120"
Final Answer: 150"
Figure 9: Illustration Geometric Reasoning.
D ADDITIONAL RESULTS
D.I SELF-CORRET EXAMPLES
We present illustrative case demonstrate the system's self-correction capabilities. In Fig. g the critic identified that the initial reasoning omitted the Subfigure Croppiin tool. Based on this feedback, the planner refined its process, leading the correct answer.
D.2 GEOMETRIC REASONING EXAMPLES
As shown in Fig. we present a representative case to illustrate our system's 's ability in geometric reasoning. Our visual tools further enable the model to faithfully capture point-line relationships in geometric diagrams, thereby enhancing the accuracy and robustness of its reasoning performance.
D.3 ADDITIONAL GROUNDING RESULTS
To demonstrate the effectiveness of our finetuned grounding model, we compare its performance with the base model test set of 500 samples. This test set annotated using the same pipeline as our synthetic training data to ensure consistency. As summarized in the main text and detailed in Table 6 our finetuned model achieves dramatic improvement in grounding accuracy. The overall
20

---

Published as a conference paper at ICLR 2026
Table Grounding accuracy on structured chart elements. 1oU columns evaluate bounding-box overlap for subplots, legend regions, and textual labels (titles and axis labels). PCK measures point localization accuracy for axis tick marks using a threshold of 0.01 max(height, width).
Model
Subplot regions (IoU)
Legend regions (IoU)
Text labels (Iot)
Axis ticks (PCK@0.01)
Overall
Qwen2.5-VL-33
0.27
0.04
0.05
0.04
0.10
Qwen2.5-VL. VLL77B
0.52
0.17
0.21
0.15
0.26
GPT--41-miin
0.78
0.18
0.11
0.17
0.31
Ours (3B)
0.99
0.89
0.90
0.93
0.93
donn marid
B Rocazon anarias musts
Lesiograppic aeatic
Lesiographic goodsies
The subfigure row 2, col 2
The subfigure at row 3, col
Log masis Reaate hauus
50
Overall Arcitl)
50
Deccialy crin Portils
30
60
20
Sessions
(4) S-way 10-shet
S-way full-shot
20.0
2015
INCNS
Danw4l
Oun-AL-N
Legend item "Joint-CNN"
The whole legend
Throughput per GPU (TFLOPS)
ZeRO-Offload w/o DPU
ZeRO-Offload w/ DPU
40
30
Cse Fu NNC
20
Sparse ULA
10
2B
6B
8B
10B
Model size (billion parameter)
(00p)
y=
6=
Figure 10: Additional grounding examples.
accuracy for our model is 0.93, substantial leap from the base model's 0.26. Additional visual examples can found Fig. [0 This precise localization of chart components the foundation upon which system's downstream chart analysis capabilities built.
21

---

Published as a conference paper at ICLR 2026
E COMPREHENSIVE ANALYSIS OF COMPUTATIONAL COST AND EFFICIENCY
In this section, we provide a detailed analysis of PixelCraft's computational overhead. We first compare our method with existing visual tool and multi-agent baselines terms of latency and API calls. Subsequently, we compare PixelCraft against Self- Beli-Refine strategies (Test-Time Scaling) demonstrate that our performance gains stem from the proposed architecture rather than merely increased computational budget.
E.1 COMPARISON WITH VISUAL TOOL AND MULTI-AGEN, BASELINES
There is generally a trade-off where agentic frameworks achieve better performance by incurring higher computational costs due iterative planning and tool execution. To demonstrate PixelCraft relative efficiency, we compare it with both Visual Tool Baselines (Refocus and Visual CoT) and Multi-Agent Baselines (Debate and Reconcile) on the CharXiv benchmark.
We report the average inference latency (seconds per query), the average number of API calls, and the corresponding accuracy in T Note that "Visual CoT defined under the same setting as in Section 5.3 and refers to linear agent baseline equipped with visual tools and indiscriminately includes historical images in the context.
Table 7: Computational cost analysis comparing PixelCraft with visual tool and ageti base- lines on CharXiv. Latency is measured in seconds per query.
Method
Latency (s)
API Calls
Accuracy (%)
CoT
3.75
1.0
63.8
Refocus
14.31
2.0
60.7
Visual CoT
19.31
4.4
65.0
Debate
27.65
7.0
62.4
Reconcile
25.90
6.0
63.5
Ours
16.45
5.7
68.1
Scalability vs. Multi-Agent Systems. As shown in Table 7 PixelCraft demonstrates superior efficiency compared existing multi-agent frameworks, reducing latency by approximately 36% to 40% relative to Reconcile and Debate. This efficiency stems from PixelCraft's 's adaptive workflow. Unlike Debate or Reconcile, which typically enforce fixed workflow (e.g., fixed debate rounds) regardless of query complexity, our Planner can terminate the process early for simpler queries. This adaptability avoids the computational overhead inherent in rigid multi-agent interactions.
Comparison with Linear Agent (Visual CoT). Crucially, PixelCraft achieves lower latency than the Visual CoT baseline (16.45s compared to 19.31s). This validates the computational advantage of Image Memory design. Unlike the linear Visual CoT approach, which stacks the entire history of generated images into the context window (increasing token count and processing time), our planner selectively recalls only the necessary visual clues. This ensures more compact context and faster inference without sacrificing reasoning depth.
Trade-off Analysis. While PixelCraft incurs a slight increase in latency compared to the simpler Refocus method 142), it yields substantial performance gain (+7.4% CharXiv). Regarding API calls, our design introduces a dispatcher, image memory, and critics to reduce the planner's decision burden. Although this yields more API calls than single-turn methods, the context per call is much shorter, resulting in superior latency efficiency compared to baseline systems.
E.2 COMPARISON WITH SELF-REFINE BASELINES (TEST-TIME SCALING)
To rigorously validate that our performance gains are driven by our specific multi-agent design rather baseline.
than simply increased test-time compute, we compare PixelCraft against
22

---

Published as a conference paper at ICLR 2026
Experimental Setup. We implemented Belf-Refine baseline following the workflow proposed in Critic-V (Zhang et al. 2025a). The workflow involves a Reasoner generating an answer and a Critic providing feedback to refine it. To ensure fair, cost-aligned comparison, we configured the Self-Refine baseline to run for rounds (approx. 5 calls) and rounds (approx. calls), which brackets the average API calls of PixelCraft (5.7).
Table 8: Comparison with Self-Refine baselines (Test-Time Scaling) on CharXiv and ChartQAAPo.
Method
Cost (Approx. Calls)
CharXiv
ChartQAPro
CoT
63.8
62.21
Self-Refine (1 round)
3
61.4
63.14
Self-Refine rounds)
(Similar Ours)
62.0
62.47
Self-Refine (3 rounds)
7
61.0
61.60
Ours
55.
68.1
65.56
Analysis. As shown in Table simply applying self-refinement strategies did not yield consistent performance gains and even caused degradation on CharXiv (63.8 - 62.0). This observation aligns with recent findings in Sherlock (Ding & Zhang 2025), which reported that VIMM self-refinement often degrades performance reasoning benchmarks due to hallucinated critiques.
The significant gap between PixelCraft (68.1%) 1%) and Self-Refine (62.0%) under the same compute budget demonstrates that access to high-fidelity tools and flexible reasoning paradigm rather than just more reasoning turns-is is the key to overcoming perceptual bottlenecks in structured image reasoning.
F ADDITIONAL EVALUATION ON REAL-WORLDD INFOGRAPHICVQA
To evaluate the generalization capability of PixelCraft on open-domain, real-world structured im- ages beyond standard chart benchmarks, we conducted additional experiments on the Infograph- ieVQA (Mathew et al. 2022) benchmark. We reuse the same configuration as in the chart settings, without introducing any additional tools.
F.1 EXPERIMENTAL. SETUP
Dataset Filtering. Infographic VQA contains a mix of document images, some of which require only optical character recognition (OCR) on dense text, while others involve complex visual reason- ing over infographics, charts, and tables. Since our work focuses on structured image reasoning, we curated relevant subset from the Infographic VQA validation set.
We employed GPT-5-mini as filter to identify questions specifically requiring reasoning over charts, tables, graphic layouts, excluding queries that rely solely on reading text blocks. This process yielded a challenging subset of 947 samples. The prompt used for this filtering process provided below ensure reproducibility:
Filtering Prompt for InfographicVQA Subset
You are a helpful Al assistant. Given an image (which may contain charts, tables, diagrams, screenshots, plain text) and a question about that image, decide whether answering the question truly requires understanding chart/table data, or can be answered only by reading/extracting text. Your goal is to select subset of chart/table-related examples.
System:
Follow these rules:
1. Chart/Table-related (select true) Set "select": true only if the question mainly requires understanding the data structure quantitative information in chart or table. Typical signs: 1) Interpreting axes, scales, bars, lines, points, or cells. 2) Com- paring values (higher/lower, increase/decrease, trends, patterns). Aggregating or
23

---

Published as a conference paper at ICLR 2026
combining numbers (totals, differences, ratios, averages, etc.). Understanding the layout/structure of a table (row/column relationships).
2. Text-only (select false) Set "select": false if the question can be answered by sim- ply reading text in the image, without using any chart/table structure. Examples: Asking about title,, captions, legend labels, axis labels. Asking for a single printed number that requires comparison or trend analysis. 3) Asking about UI elements (buttons, menus, logos). 4) Anything solvable via OCR-style text extrac- tion only.
3. Special cases the image does not contain chart or table - ""select" false'. 2) If the question is unrelated to the chart/table the image content - "select": false. 3) uncertain, be conservative - "select"; false.
User: (example le png) Please decide whether the above image should be selected given the question: {question}
F.2 QUANTITATIVE RESULTS
We compared PixelCraft against standard Chain-of- (CoT) and multi-agent baselines (De- bate, Reconcile) on this subset. To handle the diverse visual styles and ensure better generalization on these real-world images, we employed (0wn2.5-VVL-77. grounding model for this experiment, which was fine-tuned on the exact same training data as the 3B model used in our main evaluation. As shown in Table 9 PixeICraft achieves consistent performance gains across different backbones.
Table 9: Performance comparison on the structured reasoning subset of Infographic VQA. PixelCraft consistently outperforms baselines, demonstrating strong generalization to real-world infographics.
Method
GPT-4.1-minl
GPT-4o
CoT
77.3
69.9
Debate
74.0
67.6
Reconcile
73.7
67.3
Ours
79.8
71.6
Unlike structured charts, real-world infographics often contain noisy layouts and diverse visual styles. The superior performance of PixelCraft suggests that our grounding-based tool agents effec- tively handle these complexities, reducing perceptual errors where standard VLM reasoning fails.
F.3 QUALITATIVE ANALYSIS
To better understand the performance gains shown in Table 9 visualize representative reason- ing trajectories in Figure TI These examples highlight how PixelCraft overcomes the perceptual bottlenecks inherent standard Chain-of-Thouge (CoT).
G LIMITATIONS AND FUTURE WORKS
While PixelCraft significantly improves performance across several challenging benchmarks, we identify two key limitations that highlight promising directions for future work.
One limitation stems from the current inability of MILMs to autonomously generate reliable, high- fidelity visual tools. Our initial explorations found that purely L..M-generated tools were often ineffective, suffering from incorrect code execution for visual editing, which requires slight manual validation and refinement (see details in Appendix B). Future work on fully automated tool genera- tion and verification is thus critical to improve adaptability and reduce this curation overhead.
Additionally, PixelCraft requires strong backbone MLLMS for task decomposition and tool orches- tration. Weaker ones may constrain the framework's effectiveness with incorrect planning and tool calling. Future work could investigate mitigation strategies, such as developing more robust agent communication protocols lightweight, specialized models for planning and criticism.
24

---

Published as a conference paper at ICLR 2026
Q: Which is the major city in Province 5?
CoT
Step I: Identify color Answer: The major city in
eBC
Provinec 5 is Pokhara Lekhnath.
Ours
POPULATION DISTRIBUTION BY SIX
To find the major city
in Province 5, need to focus on ... Masking
Data with Legend Final Answer:
The major city
Province 5 is Ghorahi.
Q: How many companies constitute the Disney Theatrical group?
COT
The Disney Theatrical Group consists of the following companics ... there are companics/entitie
Beqquis Ban fas
Ours
The question asks for the
number of companies
Subfigure Cropping
Final Answer:
The Disney Theatrical Group consists of 8 companies.
HEMM SHUDER
Q: By what colour is IPHONE 3G in pie chart represented-yellow, green or purple?
Ours
The question asks about the color Subfigure Cropping
PAST HISTOR OF MOBILE
2009
CoT
Let's analyze the pic chart ... Therefore, the IPHONE 3G the pie chart is represented by the
BILLION MOBILE PHONES WERE SOLD
WAGUMIIR
110110
50%
Final Answer: IPHONE 3G is represented by the purple color the pic chart.
yellow color.
Qualitative Examples Infographic PixelCraft effectively selects and utilizes tools (e.g., Subfigure Cropping, Masking Data with Legend) to answer questions requiring detailed inspection of complex infographies, even when visual inputs are less structured.
Figure 11:
Progress in these complementary directions will be critical for evolving PixelCraft into more gen- cral and adaptable framework for structured image reasoning.
25


Reviews:
[{"review_id": "GxEvrxJ6LS", "reviewer": "Reviewer_k4VN", "summary": "This paper aims at solving visual reasoning on structured images, eg. Charts and geometric diagrams. The authors propose PixelCraft, a multi-agent system comprising of three main componets: query-aware agent selection, agent discussion, and iterative self-correction with a planner managed image memory. To achieve accurate visual grounding and image editing, this work also finetunes Qwen2.5-VL-3B into a visual grounding model based on a synthesized training set, and uses it as a reliable Tool Agent. Experiments on multiple recent datasets demonstrated the superior performance of PixelCraft against competitors.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Visual reasoning on structured images is important and has a wide range of applications. This paper proposes an effective multi-agent system for performing visual reasoning on structured images, achieving strong performance across multiple datasets.\n- This paper identifies visual grounding as a key obstacle in performing visual reasoning tasks on structured images, and proposes an effective approach to constructing a high-precision visual grounding model based on synthetic training data and MLLM fine-tuning.\n- This paper provides detailed experimental results that sufficiently demonstrate the effectiveness of each proposed component.", "weaknesses": "- It appears that in the current implementation and evaluation, PixelCraft requires pre-specified tool sets for different task types, such as chart reasoning and geometric reasoning, which limits the assessment of PixelCraft’s generalization capability.\n- The paper does not report the computational cost of PixelCraft, particularly in comparison with direct answering and other test-time scaling methods such as Chain-of-Thought (CoT) and Multi-Agent Systems. Comparing both accuracy and computational cost would enable a more comprehensive evaluation of the proposed method’s advantages.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HKyKjoW6PE", "reviewer": "Reviewer_yBc8", "summary": "The paper presents a new framework designed to improve how multimodal models reason about structured visual content such as charts, diagrams, and geometry problems. It introduces a multi-agent architecture where specialized agents, like a planner, reasoner, and visual tool users, collaborate to analyze images, exchange feedback, and iteratively refine their reasoning. A key feature is a pixel-level grounding module, which allows the system to interact precisely with image regions through actions like zooming, masking, and highlighting. The framework also incorporates an image memory and planning mechanism that enables non-linear reasoning and self-correction. Experiments show that PixelCraft achieves strong performance across visual reasoning benchmarks, outperforming previous multi-agent and single-model approaches.", "questions": "I am confused about the Figure. 4. Why is the usage frequency of Legend Masking and Adding Auxiliary very low, but the performance gain is giant? Most of the time, these tools will not be called.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality: This paper integrates multi-agent collaboration with fine-grained visual grounding for structured image reasoning. While the idea of multi-agent reasoning is not entirely new, combining it with pixel-level visual tools and planner-managed image memory adds a modest but meaningful improvement in coordination and interpretability.\n\n2. Quality: The work is technically sound and experimentally thorough. The system design is coherent, and the experiments cover diverse benchmarks with reasonable comparisons and ablations.\n\n3. Clarity: The paper is generally well organized, with clear descriptions of each agent’s role and effective figures that illustrate the workflow.\n\n4. Significance: The contributions are incremental but relevant. The framework addresses practical challenges in visual reasoning and could be adapted to other structured domains, offering a small yet useful step forward for multimodal systems.", "weaknesses": "1. Unclear Cost and Scalability\n\nThe multi-agent architecture (planner, critics, tool agents, etc.) implies high computational and communication overhead. The paper doesn’t report inference latency, average agent calls, or cost per sample — critical factors for assessing practical deployability. Since they use a strong vision-language backbone (Qwen2.5-VL), much of the pipeline’s gain might come at the cost of efficiency. Report computation and cost metrics, average number of reasoning turns, total API calls, and latency per image — to give a fair sense of scalability.\n\n2. Overreliance on Synthetic or Narrow Benchmarks\n\nMost experiments use structured-image datasets (e.g., ChartQA, Geometry3K), which are quite specialized. This limits generalization to open-domain or real-world structured reasoning tasks like document or diagram understanding. It’s not shown whether PixelCraft scales to noisy or natural images where structure is less explicit. I suggest adding a small evaluation on broader multimodal datasets (e.g., DocVQA, InfographicsVQA) or including a discussion on how the system might handle unstructured visuals.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PkELlxh2Jk", "reviewer": "Reviewer_tt9L", "summary": "This paper proposes PixelCraft, a multi-agent system to perform complex reasoning over charts and geometric figures. The multi-agent system features a tool selector, a tool planner, visual critic, plan critic and multiple tool agents. Experiments on chart reasoning benchmarks like ChartXiv and ChartQAPro, and geometry reasoning benchmarks shows substantial improvement over other framework or vaniila visual CoT.", "questions": "See the weaknesses above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "+ The paper proposes a novel multi-agent system to improve visual reasoning performance on structured image.\n+ Achieves high-fidelity image process with a fine-tuned grounding model\n+ The paper is well written and easy to follow.", "weaknesses": "+ My major concern is the cost of the proposed multi-agent system compared with vanilla visual CoT. As shown in Appendix D.4, the response time of the multi-agent system is approximately three or four times longer than baseline methods. Therefore, it would be helpful to reveal the cost (or the number of generated tokens) of PixelCraft in comparision with baseline method. \n\n+ Moreover, it is not clear if the compared baseline methods also use a critic to iteratively refine and improve the final result. If not, I would suggest equipping the baseline methods with a critic to do test-time scaling, and draw a comparison when the respond time/cost is similar.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pjol7tflOl", "reviewer": "Reviewer_hJXi", "summary": "This paper presents PixelCraft, a novel multi-agent system designed to address significant failings in MLLM-based visual reasoning for structured images like charts and geometric diagrams. The authors identify two primary weaknesses in existing methods: low-fidelity image processing, which leads to perceptual errors, and rigid, linear reasoning patterns (like visual CoT) that cannot correct mistakes. PixelCraft's core contribution is a dual-pronged solution. First, it achieves high-fidelity processing by synergizing a compact MLLM, fine-tuned on a new synthetic corpus for precise pixel-level grounding, with classical CV algorithms that act as tool agents. Second, it enables\nflexible, non-linear reasoning through a dynamic workflow involving a planner, reasoner, and critics. A key innovation is the \"image memory,\" which allows the planner to adaptively revisit, recall, and branch from earlier visual steps, facilitating backtracking and exploration of alternative reasoning paths. The authors demonstrate through extensive experiments on challenging chart and geometry benchmarks that their system significantly improves reasoning accuracy for advanced MLLMs.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Well-motivated and significant problem. The paper addresses the challenging problem of visual reasoning on structured images, a well-known weak point for current MLLMs. This task is difficult because it requires precise perception of details (like axis values or geometric points) and multi-step logical deduction, where small perceptual errors can cascade into completely wrong answers. The paper clearly identifies these failings as a critical and practical area for improvement.\n\n2. Comprehensive experimental setup. The authors validate their system on a strong suite of recent and difficult benchmarks, including CharXiv, ChartQAPro, and the auxiliary-line subset of Geometry3K. Testing against these diverse datasets demonstrates the system's robustness and generalizability. Furthermore, the comparison is not limited to simple baselines; it includes advanced agentic methods like Debate and Reconcile, which makes the consistent and significant performance gains of PixelCraft more convincing.\n\n3. High-quality presentation and readability. The paper is very easy to read and well-organized, making the complex multi-agent architecture and reasoning process accessible to the reader. The figures are a standout component for presentation; Figure 1 provides an excellent, intuitive comparison of why standard CoT and Visual CoT fail while PixelCraft succeeds, and Figure 2 clearly illustrates the entire agentic workflow, from selection to discussion and correction.", "weaknesses": "1. Significant efficiency and latency issues. The multi-agent workflow, which involves a dispatcher, planner, reasoner, and multiple critics, inherently requires numerous sequential LLM inference calls for a single query. This leads to a substantial increase in latency and computational cost compared to simpler CoT methods, as confirmed by the paper's own analysis (e.g., 16.45s vs. 3.75s on CharXiv), which may limit its practical applicability.\n\n2. Limited methodological novelty in the agentic framework. While the application to structured images is effective, the core agentic workflow (combining a planner, tool-use, and a reasoner) is a widely adapted paradigm in recent LLM research. The novelty of the agent framework itself is somewhat incremental, with the main contributions lying in the specialized toolset and its application rather than a fundamentally new agentic reasoning process.\n\n3. Constrained generalizability and flexibility due to a fixed toolset. The system's success is heavily reliant on its manually-curated set of visual tools, which were specifically designed for charts and geometric diagrams. This presents a generalization bottleneck; the system would likely fail in scenarios with novel image types or tasks requiring tools not in its predefined set. This lack of flexibility to adapt or generate new tools on the fly is a key limitation, which the authors also acknowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)