ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
Authors: Muye Huang
Lingling Zhang
Jie Ma
Han Lai
Fangzhi Xu
Yifei Li
Wenjun Wu
Yaqiang Wu
Jun Liu
Venue: NeurIPS 2025 poster ()
Keywords: Charts
VLMs
CV
Primary area: deep_learning
TL;DR: ChartSketcher improves multimodal chart understanding by enabling models to sketch intermediate reasoning steps directly onto charts using a programmatic library and iterative visual feedback, enhancing reasoning and reflection.

Abstract: Charts are high-density visualization carriers for complex data, serving as a crucial medium for information extraction and analysis. Automated chart understanding poses significant challenges to existing multimodal large language models (MLLMs) due to the need for precise and complex visual reasoning. Current step-by-step reasoning models primarily focus on text-based logical reasoning for chart understanding. However, they struggle to refine or correct their reasoning when errors stem from flawed visual understanding, as they lack the ability to leverage multimodal interaction for deeper comprehension. Inspired by human cognitive behavior, we propose ChartSketcher, a multimodal feedback-driven step-by-step reasoning method designed to address these limitations. ChartSketcher is a chart understanding model that employs Sketch-CoT, enabling MLLMs to annotate intermediate reasoning steps directly onto charts using a programmatic sketching library, iteratively feeding these visual annotations back into the reasoning process. This mechanism enables the model to visually ground its reasoning and refine its understanding over multiple steps. We employ a two-stage training strategy: a cold start phase to learn sketch-based reasoning patterns, followed by off-policy reinforcement learning to enhance reflection and generalization. Experiments demonstrate that ChartSketcher achieves promising performance on chart understanding benchmarks and general vision tasks, providing an interactive and interpretable approach to chart comprehension.

ChartSketcher: Reasoning with Multimodal Feedback and Reflection for Chart Understanding
Muye Huang 12,4 1.24 Lingling Zhang Jie Jie Ma², Han Lai Fangzhi Xu',
Yifei Li4 Wenjun Wu Yaqiang Wu Liu
'Scooo of Computer Science and Technology. Xi an Jiaotong University, Xi'an, China
Shannxi Province Key Laboratory of Big Data Knowledge Engineering, Xi' Xi'an, China MOO KLINNS Lab, Xi'an, China
*Zhongguancun Academy, Beijing, China 'Lenovo Research Thuangmuye, fangzhixu98, 1iyxffei6115544922)ettt. xut................ eeuuu.dduu... edu, cn (zhanglling, jiema, liukeen]@xxj su.ttu.eedhh edu. cn vuyqe@lenovo.com
Abstract
Charts are high-density visualization carriers for complex data, serving as a crucial medium for information extraction and analysis. Automated chart understand- ing poses significant challenges to existing multimodal large language models (MLLMs) due the need for precise and complex visual reasoning. Current step-by-step reasoning models primarily focus on text-based logical reasoning for chart understanding. However, they struggle to refine or correct their reasoning when errors stem from flawed visual understanding, as they lack the ability to leverage multimodal interaction for deeper comprehension. Inspired by human cognitive behavior, we propose ChartSketcher, multimodal feedback-drives step- by-step reasoning method designed to address these limitations. ChartSketcher is a chart understanding model that employs Sketch-CoT, enabling MLLMs annotate intermediate reasoning steps directly onto charts using a programmatic sketching library, iteratively feeding these visual annotations back into the reason- ing process. This mechanism enables the model to visually ground its reasoning and refine its understanding over multiple steps. We employ two-stage training strategy: a cold start phase to learn sketch-based reasoning patterns, followed by off-policy reinforcement learning enhance reflection and generalization. Experi- ments demonstrate that ChartSketcher achieves promising performance on chart understanding benchmarks and general vision tasks, providing interactive and interpretable approach to chart comprehension.
Introduction
Charts are widely used as data visualization methods in scientific papers and business reports. Automated chart understanding a key step in achieving automated data analysis. Recent advances in MLLMs [ 24 38
45 50 have shown substantial progress in chart understanding tasks. These include proprietary models like GPT-4o (88 and Gemini-2.0 [4], as well as open-source models such Qwen-2VL (50) and InterVV--2.5
4 The use of MLLMs has become mainstream approach for
chart understanding.
However, existing MILMs face significant challenges in chart understanding, which involves the systematic interpretation and analysis of visual data representations. Chart understanding requires
'Corresponding authors. Emails: zhanglling @ xjtu.edu.en; wuyge @emovooenoooo
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Question Ingut
Final Answer
Can you tell me what the difference is between Bad and Good for the cconomic conditions in 2015?
Okay, we've successfully marked all the data points. Their values are now displayed, with Good and Bad being 57 and 42 respectively, so their difference 57-42=15.
ChartSketcher
Sketch Solution
Check & Reflection
Continue Reasoning
Alright, solve this problem, think
Let me check my drawing... Wait,
Ah, looks much better. correctly marked the objects. Now, I'll continue, marking the positions for Good and Bad...
I need to first determine where 2015
think mislabeled the position on the x-axis. This is clearly 2014, not 2015.
is... can be inferred that 2015 is
between 2014 and 2016...
Let mc correct that...
Poe ecceomic sendiment iecuuniing inde 2015
Peetfre ecosomic sentrrent incundte sinee 2018 co
Poetiine acceamic sontimant incweding siace 2015
3073
2003
204
30
2014
264
30.8
2003
26.4
25R
Figure 1: The overview of the proposed ChartSketcher. Dashed lines indicate intermediate reasoning and reflection processes, with corresponding sketch outputs shown for each step.
high-precision visual reasoning capabilities to process complex elements such as overlapping data points, multiple intersecting trend lines, and dense numerical information, demanding simultaneous comprehension of both spatial relationships and their semantic meanings. For example. in Figure [ answering "What is the value of 'Good' in 2015?" requires identifying that 2015 lies between the marked years 2014 4 and 2016. Models must precisely locate and identify these visual elements while understanding their quantitative relationships to correctly determine that the 'Good' value in 2015 is 57. This visual reasoning process requires analysis of visual dependencies and precise numerical understanding each step. Such complex visual reasoning tasks pose significant challenges to exist- ing approaches. MLLMs 5 47 have attempted to achieve fine-graine grained visual reasoning through long chains of thought. For example, multimodal reasoning models like QvQ (77 demonstrate the capability to generate long-chain reasoning text. However, their effectiveness remains limited in visually intensive scenarios like charts. This limitation stems from their predominant focus on textual logical processes rather than visual information processing, causing reduced interpretability for users and an inability to correct errors originating from flawed multimodal understanding. Recent attempts, such as VisualCoT [4]), Refocus [10] and SketchPad [1], propose image cropping techniques to enhance visual understanding by focusing on key regions. However, the inherent constraints of cropping mechanisms prevent simultaneous analysis of multiple regions, thereby limiting the model's capacity for complex visual reasoning. As illustrated in Appendix [A] this represents a challenging case in existing MLLMs. The development of visual reasoning models that focus processing complex elements requires urgent exploration.
Interestingly, when humans encounter complex visual information, they often create sketches to mark and organize key details. This process helps them break down problems and focus on critical areas within the image. For example, when determining the value of 'Good', humans typically start by locating the relevant position on the x-axis, then trace vertically upward to identify the corresponding value on the colored line: natural way of decomposing the visual reasoning process. This behavior reflects a subconscious strategy humans use to enhance visual focus and understanding.
Drawing inspiration from natural human behaviors, we propose ChartSketcher, a multimodal feedback-driven step-by-step reasoning method that addresses these visual reasoning limitations in chart understanding. Specifically, ChartSketcher employs Sketch-CoT, enabling MLLMs to explicitly annotate their intermediate reasoning processes on images and feed the visual annotations back to themselves, achieving stable step-by-step multimodal reasoning. Morcover, by incorporating

---

reflection processes between steps and leveraging reinforcement learning, we endow MLLMs with human-like reflection capabilities. The model not only marks the visual reasoning process on images but can also identify reasoning errors and promptly correct mistakes from previous steps. As illus- trated in Figure [ our approach demonstrates powerful visual reasoning capabilities across diverse scenarios. The implementation of ChartSketcher follows a two-stage training pipeline: a cold start phase and an RL phase. In the cold start phase, we transfer reasoning and reflection patterns from LLM to MLLM through cross-modal distillation, creating 300K fine-grained annotated chart under- standing samples. The subsequent RL phase employs MCTS and diverse data sampling techniques with over 50K step-by-step reasoning examples to enhance the model's ' capabilities through off- off-policy reinforcement learning.
Our main contributions can be summarized in three aspects:
We propose ChartSketcher, novel multimodal feedback reasoning approach that enhances MLLMs visual reasoning capabilities through iterative Sketch-CoT and self-reflection mech- anisms. Code and data available at https: //gi thub. coltuu..oom//// Muwn/MayyHHHg//////aa Charr cSktchrrr
We construct a comprehensive dataset of 300K annotated samples for cold start training and 50K curated samples for reinforcement learning. The dataset is designed to support chart step-by-step reasoning.
We conduct extensive experiments across multiple datasets to demonstrate the effectiveness of ChartSketcher. Through comprehensive ablation studies, we investigate the importance of each training stage and validate the contribution of key components in our work.
2 Related Work
Chart Understanding. Chart Understanding aims to comprehend the visual context of charts to address specific tasks, such as QA or summarization. FigureQA [18] stands as a pioneering work, introducing a chart understanding pipeline capable of handling binary classification tasks for chart-
further enhanced chart understanding capabilities by employing multi-component pipelines. For instance, DePlot [63] leveraged multiple components combined with the mathematical capabilities of LLMS to achieve performance improvements on PlotQA [36). With the advent of MLLMs, approaches utilizing MLLMs as the primary component
related questions. Subsequent works [29] 22 26 23
[5] [33 52). ChartLlama [12,, through clever data construction and fine-tuning of LLaVA 24], built a robust chart-expert model. Leveraging the powerful language capabilities inherent in MLLMs, recent studies have employed multi-task training methodologies to bolster chart understanding across a variety of tasks. ChartAssistant (35] utilized unified multi-task training improve overall performance. TinyChart [62) utilizes the Program-of- Thoughts technique to enhance numerical reasoning capabilities in chart QA tasks. ChartMoE (57) employed Mixture of Experts approach to model different chart types effectively, thereby enabling understanding across diverse chart categories.
have become mainstream in the field [31 32 2
Multimodal Reasoning. Models such as OpenAl-ol the strong reasoning capabilities of LLMs [8 However, the reasoning capabilities of MLLMs remain an of active investigation. Current
337 and Deepseek-R1 have demonstrated
40 54
48
55 28), often enhanced through RL.
approaches often focus on training MLLMS using CoT techniques 5 59 to generate step-by-step reasoning sequences across diverse tasks. These methods g 25
27
6 predominantly concentrate
on CoT techniques within the textual modality, relying heavily on the MLLM's underlying LLM backbone to perform multi-step inference. VisualCoT [41] introduced a method involving cropping critical regions to aid the model in focusing on pertinent visual areas. While prior work predominantly focuses text-based CoT methods or region-limited approaches, these techniques struggle with scenarios requiring attention to multiple distinct visual elements. Our work addresses this limitation by integrating ooffprromped visual markers and multimodal feedback into the CoT process, enabling more comprehensive and robust multimodal reasoning.
3 ChartSketcher
We propose ChartSketcher, which enables step-by-step reasoning in multimodal chart understanding by sketching directly on chart images. the following sections, we will present the implementation details of ChartSketcher, including its architecture and training specifics.
3

---

Cold Start
Stend-Cof Datn Synthesis
First, should ...
Reflective Keasoning Dam Synchets
should carefiilly review
code. 0z
Sketch-CoT Rendering
<code_DD D>
Oh, I think I made mistake
Alright, will proceed step by step to...
<code__ D>
<code_12
Wwn 2.5
Ah, thar's
Next,
Finally, the
Intentionally Chaatal Errurs
<code_13 I>
Finally, the answer is
RL Optimization
Skitc-MCCSS
caueflly
reles: coade
mekes soode to
Secodl T Taliid cacdully
revien Soade, I
Maybe the answer
maaee munke Spole
Alniti, the anowerria..
Pist Faborid candilly
ChartSketcher
Alnigti, snawer
serien... 2.00
cucciall
Multiple
Training
Construction of RL Dara
Figure 2: Overview of ChartSketcher Training Process. The upper part illustrates the cold start phase, focusing knowledge distillation and pattern learning. The lower part shows the offline RL optimization process, which is conducted on diverse datasets. In the figure, <code> indicates that ChartSketcher is calling the Programmatic Sketching Library draw. When ChartSketcher no longer outputs <code>, it indicates that the reasoning process has ended.
3.1 Architecture
Enabling the model to reason while sketching, much like human, is the core objective of ChartS- ketcher. To enable the MLLM to perform Sketch-CoT reasoning, we designed two integrated modules: programmatic sketching library and sketching reasoning pipeline. The details of these two modules are introduced as follows:
Programmatic Sketching Library. To equip MLLMs with image sketching capabilities, design a simple drawing language library. The library provides basic operations to create and manipulate geometric shapes (points, lines, circles, arrows, and their combinations) through simple command syntax. During the reasoning process, the MLLM can insert drawing commands at any position create visual elements modify existing ones through operations like translation, rotation, and deletion. The detailed command specifications, library guide, and supported operations listed in Appendix B
2) Sketching Reasoning Pipeline. the process Sketch-CoT reasoning, it is necessary to view the draft content in real time ensure the continuity of reasoning. We have designed visual feedback pipeline that parses the output of the MLLM, generates sketches, and feeds the newly rendered image back the MILM. This new image is then fully re-encoded to generate for the next reasoning step, forming a closed loop of action, perception, and reflection. This process operates "reflection and draw-feedback" loop, illustrated in Figure which terminates upon the completion of reasoning, where no further sketching code is produced, and the pipeline exits the loop automatically.
visual tokens
3.2 Training Process
ChartSketcher implements Sketch-CoT reasoning through the multi-turn dialogue mechanism of MLLMs. Therefore, ChartSketcher primarily focuses on understanding the differences between sequential images in a dialogue. Existing MLLMs lack capabilities for such serialized visual understanding; therefore, we introduce a two-stage process comprising cold and RL optimization. Cold start focuses on learning the reasoning patterns for multi-tur visual feedback, RL optimization leverages an off off-policy RI. approach to further enhance reasoning capabilities. The following sections will detail the construction of training data and the specifics of the training process for these steps.

---

3.2.1
Cold Start
Cold start is designed to learn the Sketch-CoT reasoning patterns with visual feedback. It is divided into two steps: the first step trains the model to understand and generate sequential visual reasoning patterns, enabling it to process multi-turn visual information coherently, while the second step builds upon this foundation by developing reflective reasoning capabilities that allow the model to evaluate and improve its solutions based visual feedback.
Sketch-Cor Data Synthesis. This step aims to synthesize rich Sketch-CoT data, which is used to train the model's reasoning patterns and its ability to read visual feedback. The detailed process is illustrated in Appendix C Specifically, the construction process is divided into the following three steps:
1) Question Construction: We used the EvoChhrt-CCrpus [15), a dataset with high-quality synthe- sized chart images. While it provides chart images, its QA pairs are -epplatt-geerrtedd which may limit the diversity of reasoning chains. Therefore, we used all the images and part of the QA pairs from EyoChart-Corpus. We then developed a seed-based method create more diverse questions. We used QA pairs from existing ChartQA datasets as seeds to prompt the LLM to generate similar new questions based on the EvoChart- Corpus image and annotations. This approach helped create many diverse and meaningful questions.
With annotations available, multimodal questions can be converted into simpler text- based questions. Similarly. visual reasoning chains can also be converted into textual reasoning chains. Inspired by this, distilled the reasoning capabilities of the LLM into the MILM. Specifically, we input the questions and detailed annotations of the EvoChart- CCrpuus into the LLM and prompted the model to output reasoning chains. We enforced rule that the LLM must output sketching code to justify its conclusions before providing any final or intermediate conclusions. This ensures that the constructed multimodal reasoning chains are factually grounded.
2) Reasoning Process Construction:
3) Rendering:
We used the sketching code in the reasoning chains as a boundary to segment the reasoning process into multiple steps, adding visual feedback between these steps. All prompts can be found in the Appendix C Using the above methods, we constructed over 300k Sketch-CoT samples.
Reflective Reasoning Data Synthesis. This step aims to synthesize Sketch-CoT data with reflective reasoning processes, training the model to identify errors from visual feedback and correct them in timely manner. To enable reflection, we manually construct erroneous reasoning processes. For simplicity, the reflective reasoning data is built based on the previously generated correct Sketch-CoT data. Specifically, the process involves the following steps:
1) Reflective Construction: We prompt the LLM to introduce an error in a specific step by providing incorrect drawing coordinates. The error types vary, such as using coordinates from other points, coordinate drift, and more. Subsequently, the LLM reflects on and corrects these coordinates, providing new drawing code. During the reflection process, the LLM uses conversational expressions and human-like interjections, such as "Oh" or "Hmm," to mimic natural reasoning. Detailed examples can be found in Figure
Through the above steps, each Sketch-CoT generates two versions: one with reflection and one without. These two versions share the same CoT prefix. At the final step of the prefix, if erroneous data is selected, it constitutes reflective data; otherwise, it is non-reflective data. We mix reflective and non-reflective data at a 1:1 ratio, encouraging the model to reflect only when errors occur, with greater focus on visual feedback information.
2) Data Mixing:
3.2.2 RL Optimization
After undergoing a cold start phase, ChartSketcher learns patterns from annotated synthetic data but has not been trained on real-world unlabeled datasets. To enhance the generalization ability of ChartSketcher, we incorporate off- off-po RL optimization inspired by works on natural language. We designed an off- off-policy RL strategy based on a variant of MCTS, sketch-MCTS, which can collect high-quality RL data on datasets without bbox annotations. Our approach identifies optimal paths by evaluating the average Q/N value of each potential solution path, selecting nodes along the optimal trajectory as positive samples while designating low-value siblings, nodes with rendering errors, and duplicate nodes negative samples. To maintain sample quality, we exclude siblings with
5

---

positive values above zero from the negative sample pool. This strategic sampling mechanism enables effective learning from unlabeled data while preserving the model's discriminative capabilities. The following sections detail the sketch-MCTS algorithm that underpins this RI. optimization framework.
Sketch-MCTS Algorithm. MCTS is multi-step reasoning algorithm with single-step action output, which implicitly considers the consequences of multi-step decision making, Our proposed sketch-MCTS collects all implicit processes and identifies the potentially optimal answers. The formal representation is shown in Appendix [D] Sketch-MCTS modifies the original MCTS algorithm while retaining its core principles, enabling the generation of a complete search tree in a single run. Specifically, we made the following modifications to the MCTS algorithm:
1) Modifications to the Expansion step: We set rules to control the behavior of the Expansion step in order to obtain more diverse paths. Expansion generates as many potential next steps as possible by using high temperature. To limit redundant paths, we set deduplication mechanism: if two nodes generate identical drawing codes, the duplicate node is directly removed. Additionally, to prevent invalid reflection, if the drawing code of child node a subset of its parent node's drawing code, the child node is considered an invalid reflection node, as it results in ineffective changes.
2) Handling of leaf nodes: In our method, the criteria for determining leaf nodes are more stringent and explicit. node is only marked as a leaf node if its drawing code contains errors or if it fails to generate any drawing code (indicating the end of the solution). Furthermore, once node is identified as a leaf node, its correctness is immediately evaluated, and the reward is backpropagated.
3) Conditions for terminating the search: To prevent infinite searches for complex problems and excessive searches for simple problems, we designed dual termination conditions: 1) The search ends when certain number of correct answers are found in the search tree. 2) The search also terminates when the number of simulations exceeds a predefined threshold. This dual condition adapts to problems of varying difficulty; for simple problems, the algorithm converges quickly, while for complex problems, the algorithm can perform sufficient exploration within a reasonable range.
Experiments
4.1 Settings
Data Construction. During the cold start phase, our base dataset images were sourced from EvoChart- Corpus, with seed questions from ChartQA (30) and EvoChart-QA [15). To ensure general capability, we incorporated 20% of VisualCoT [41 data into the training mix. For the RL phase, we conducted
training across multiple datasets, including ChartQA, ChartBench [56). and VisualCoT. For model selection, we employed Qwen2.5-320 5-BB (40] to construct QA pairs and distill multimodal reasoning and reflection data. We used DeepSeek-Distill-Weee evaluating the correctness of final answers. We also trained a smaller, 2B version ChartSketcher- 2B facilitate its use in scenarios with limited computational resources. ChartSketcher-721 and ChartSketcher-228 2B were initialized with Qwen2VL-72B (50] and Qwen2VL-2B weights, respectively. All prompts used in data construction are detailed in the Appendix C We tested chart understanding capabilities on ChartQA and other datasets [36]. and evaluated general performance on Openimages and other datasets
(7] as the value network for SKetch-MCTS,
43
The detailed composition of our training data is
presented in Table
Table 1: Detailed breakdown of the training data used in each phase of Charsketcher's development.
Phase
Method
Data Source
Data Type
Quantity
EvoChart Synthetic Chart Data VisualCoT and its Annotations
Correct Reasoning Path
155,203 (87.3%) 22,510 (12.7%)
SFT
Cold Start
Correct Reasoning Path
SFT Subtotal
177,713
DPO
EvoChart Synthetic Chart Data
Reflection Reasoning Path
147,955
ChartQA and ChartBench General QA-
MCTS Sampled Paths
41,196 (81.6%)
RL
KTO
MCTS Sampled Paths
9,259 (18.4%)
RL Subtotal
50,455

---

And Cil Winss
Skotzh--Cal
Chartq CharigA-lf BeaChart Chartleach
PleigA
DecVQA
Emetti:
Fille Kirtok
GDA
Openimagas VInual7W
Figure 3: Analysis of CoT length and the number of rethink iterations for both correctly and incorrectly answered questions across all listed datasets. Datasets listed on the left (blue font) are chart-specific benchmarks, while those on the right (brown font) represent general image datasets.
Evaluation Metrics. To accurately evaluate model performance, we employ DeepSeek-Distill-OUee 32B 77 to assess the alignment between MILM outputs and the QA dataset answers. To mitigate model variance, we adopt a voting mechanism where each question is evaluated 3 times, and correct answer is determined by a majority vote threshold of 2. To ensure fair comparison, all experimental results reported in this paper are based our local reproduction of baseline methods.
Training Settings. During the cold start phase, we trained ChartSketcher for 4 epochs on data without reflection, followed by epoch using RPO 660 loss on reflection data. To reduce computational costs, we employed LoRA [13] training in the cold phase, with a LoRA rank of 16, Alpha of 32, batch size of 64, and learning rate of le-4. The RPO ratio was set to 1.0. In the RL phase, we conducted KTO [9] training for epoch, maintaining a LoRA rank 16 and Alpha of 32, while adjusting the batch size to 32 and reducing the learning rate to le-5, For the key parameters of MCTS, the maximum tree depth 8, the maximum number child nodes is 3, CPUCT 3.0, the simulation count limit is 15, on two machines: an Atlas 800T A2 and # A800-40G GPUs. For more training details, see supplementary materials.
and the search exits after successfully finding 3 answers. All experiments
4.2 Performance Comparison
Table 2 presents the complete results for the chart-specific benchmarks alongside selected results from the general datasets. Compared the baseline Qwen2VL-72B, our proposed ChartSketcher- 72B exhibits significant improvements across both chart-specific and general-purpos datasets. Notably, even when compared to PrQ-Prev and GPT-4o, our method still maintains an advantage on the chart-specific datasets. Mcanwhile, Figure 4 demonstrates that our method offers richer interactivity. Furthermore, ChartSketcher-25 achieves substantial improvements over its baseline Qwen2V1--2B and chart expert model ChartGemma 32 across nearly all evaluated datasets. This demonstrates that our approach effectively enhances performance in the specialized chart domain without significantly compromising its general-purpose capabilities. Moreover, as shown in Figure our method demonstrates better user- are-friendlsines and greater interpretability compared other approaches. The complete evaluation results on all 18 datasets can be found in Appendix E
4.3 Ablation Study
Our ablation study is conducted based on the ChartSketcher-22B model. Specifically, we investigate the following configurations:
w/o Rethink RL: This setting omits the Rethink learning step during the cold start phase. The model proceeds directly with MCTS sampling followed by SFT.
w/ Rethink w/o RI: This setting includes the complete cold start phase (with Rethink learning), but replaces the subsequent RL phase with SFT.
7

---

Table 2: Experimental results on chart and vision benchmarks. PlotQA reports sampled results, and VisualCoT shows composite score across multiple datasets.
Model
ChariQA-H
ChariQA-A
EvoChart-QA
ChartBench
PlotQA
VisualCoT
Proprictary models
GPT-4o
84.32
88.48
52.80
61.47
42.96
78.45
Gemini-2.0
84.00
88.24
64.64
55.63
63.36
77.90
Claude-3.5
85.04
90.72
56.96
56.96
57.63
75.93
Open-source models and chart expert models
Qwen2VI.-72B
82.48
88.56
54.00
54.77
73.76
72.14 14
BVQ-Preview-72B
83.20
89.76
54.32
42.40
69.04
76.52
Intern IntemVL22.5788B
78.48
89.44
57.44
65.57
57.20
78.93
ChartGemma-2B
53.44
86.64
36,08
23.87
25.76
55.62
Qwen2VL-2B
50.48
75.84
23.84
20.27
38.80
58.33
ChartSketcher-2B
55.60
80.88
26.72
30.10
41.12
66.86
ChartSketcher-722 85.20
92.64
63.28
68.33
76.72
76.59
Ablation study based on ChartSketcher-72B
w/o Rethink RL
77.76
91.12
51.12
50.40
67.76
72.58
w/ Rethink w/o RL
76.64
90.56
51.36
52.93
67.68
70.95
w/o RI.
77.12
88.80
39.84
52.73
67.84
68.89
w/o Feedback
81.52
91.04
57.76
56.13
72.24
75.18
w/o CoT
75.12
90.08
55.36
47.43
68.16
76.12
Model
Openimages
Flickr30k
DacVQA
Visual7W
GQA
Emotic
Proprietary models
GPT-4o
52.49
79.04
94.93
77.60
68.30
53.81
Gemini-2.0
57.78
79.34
95.27
77.20
68.51
41.22
Claude-3.5
62.50
75.68
97.64
73.70
60.63
35.42
Open-source models and chart expert models
Qwen2VL-72B
51.75
61.64
93.36
73.90
57.06
43.14
QvQ-Previcw-722
60.21
72.96
92.68
69.90
63.91
65.55
60.85
76.39
95.16
74.40
72.19
61.59
ChartGemma-2B
49.21
57.37
57.32
57.30
51.33
53.20
Qwen2VL-2B
53.97
34.48
79.50
60.40
23.31
50.15
ChartSketcher-25
64.23
68.95
69.82
64.90
61.25
59.45
ChartSketcher- 72B
68.68
72.19
92.68
73.00
65.85
67.16
Ablation study based on ChartSketcher-72B
Rethink RL
62.33
66.04
89.08
65,80
62.17
$8.54
w/ Rethink w/o RL
59.05
66.17
88.40
66.00
61.96
58.38
w/o RL
57.57
66.43
86.94
62.50
57.87
53.51
w/o Feedback
67.94
70.96
92.57
70.80
65.54
54.88
w/o CoT
62.43
69.53
92.23
67.20
62.88
64.70
w/o RL: This represents the ChartSketcher model after completing only the cold start phase, without undergoing the RL phase.
w/o Feedback: In this setting, the multimodal image feedback mechanism is disabled during inference. An empty string used placeholder for the multimodal feedback input.
w/o Co7: This baseline setting does not apply the ChartSketcher methodology. Instead, the model is fine-tuned using SFT on the identical dataset used for training ChartSketcher.
Based on the results presented Table 2 we can draw the following conclusions:
Sketch-CoT is effective. Observing the w/o CoT setting reveals significant performance decline compared to the baseline when Sketch-CoT is not employed. This highlights the crucial role of our CoT approach guided by sketches and multimodal feedback.

---

Queeiee: Whatis diftormace value tiihh bined ragar and High Elood prosund?
rikk Ne
Vinea majer UK poiticcll parties hake
Party?
Chantketchen
ChariShetcher:
I. Fass, locaee the High slood legar dets
I. Fist, locate
Consevattee Parry's dats point on chart
Wast might have made raiaks... The crendeaam usst were Cbeaity initnd lood
2. Wai, bald on. That docsti seem right. The coordinaees used weme for the abour Party insiced of the Conservttvve Party mer cornect that.
atrigt, the moried the liigh bood egu valae label alown das paize. Accorcing de land, the sdae Hugh flood segarie
CharOPT-Ar des
high preares
To Rnd the dffieence values High Hood sagar
3. Airiht, Z've marked the Conerrvtive Party's point correctly 'e color associated with tis olivedrab.
rape
2 2.
locar
blood
Citos muriied the titgs Nood
High bkead pessaae: sagir
d7t. Submutt the wolee Figh bland greseare from High birod
Mosd
labd shone des poine, Aucuninng labd. the wala for stahh blaad prosure High
cagar
Hood
CharGPT-6e
Findl Answer
2015
The colur of the lahel for the Conservative Party the chart is gram.
The difiresce de valac High blood rapar and High
Quertae: What namber mnaieed
hamisdor 2012
Quertio:: Where car deeping baily op?
Chersknsher Homsides ini
Drsaked
ChartSkecchrr
boate whax
Maai Lougle has made rritale. Lita intral of 2014, Let ce earnec and
2. Airight Tve Eechialighaal lae uoe wibce checer sleping on infermation dieeping
Aingt: I've eecemmeet poreiirr
ChuntiFT-le:
2.344 Jadla
sloging floee bring cound pook
nocc vake ihewn cau prins.
The sacieral sandedd wwwsddd 2014 DI.
silsed Rowa 2014 111
Figure 4: Four cases for ChartSketcher. The drawing code associated with each step is omitted for clarity. Arrows indicate the visual outputs generated by specific reasoning steps. Semi-transparent elements represent outputs that were subsequently corrected or erased by later steps.
2) The Rethink and RL phases are critical components of ChartSketcher. Removing the Rethink step or replacing RL with SFT, as seen in the w/ Rethink w/o RL setting, results in comprehensive drop in performance across the board. Notably, substituting RL with SFT leads substantial decrease in general-purposs capabilities, evidenced by the VisualCoT aggregate score dropping to 70.95%, which below the baseline performance.
3) The two-stage ColdStart-RI. training methodology effectively enhances model capabilities. For instance, the model after only the cold start phase (w/o RI.) performs slightly below the baseline. This is expected, as the cold start phase primarily utilizes Out-Of-Distributee (OOD) synthetic data. However, the subsequent RL phase rapidly elevates the model's performance, surpassing not only the baseline but even the QvQ-Preview model.
4) Multimodal feedback plays a significant role in multimodal reasoning. As indicated by the w/o Feedback setting, the absence of feedback has a relatively minor impact on datasets that emphasize di- rect information extraction with less need for complex multimodal reasoning. such as ChartQA-A and the overall VisualCoT score. However, for datasets demanding intricate multimodal reasoning, like EvoChart and ChartQA-H, removing feedhack leads to more pronounced performance degradation.
4.4 Analysis of Reasoning Steps
To quantitatively analyze the general patterns regarding the length of reasoning and the frequency of rethinking employed by ChartSketcher when addressing different questions, we examined the average CoT length and the average number of rethink iterations for both correctly and incorrectly answered questions within the evaluation sets. The results are depicted in Figure We derive the following findings:
More challenging questions elicit longer CoTs. As observed in Figure 3, the average CoT length for the more demanding datasets, EvoChart-QA and ChartBench, reaches 6-7 steps; both require complex chart reasoning. The CoT length for correctly answered questions is consistently shorter than that for incorrectly answered questions across all datasets. This indicates that the model tends to employ more reasoning steps for difficult problems and attempts to refine answers through multiple rethink iterations when facing challenges.

---

2) The model utilizes rethinking to identify potential errors. Across all datasets shown in Figure the average number of rethink iterations higher for incorrect answers than for correct ones. This suggests that when faced with difficult problems, the model not only extends the CoT through multi-step reasoning but also actively employs the rethink mechanism in an attempt to revise its solution. This occurs even if the model ultimately fails to provide the correct answer, demonstrating persistent attempts at self-correction.
3) Chart-specific datasets demand more complex reasoning processes compared to general- purpose datasets. Within the chart-specific benchmarks, even CharQQA-H, which has the highest accuracy among them (implying relative simplicity), exhibits an average CoT length greater than that of GQA, the general-purpose dataset with the longest average COT. This demonstrates ChartSketcher's capability to engage in complex, multi-step CoT reasoning, potentially involving multiple rethink iterations, to achieve precise inference specifically within the demanding chart domain.
4.5 Case Visualization
We select four representative examples to illustrate the visual reasoning capabilities of ChartSketcher, as depicted in Figure These cases demonstrate ChartSketcher's ability identify errors within its own reasoning steps and implement timely corrections. The example presented in the top-right indicates that ChartSketcher can rectify single-step mistakes while concurrently executing multi- step numerical extraction and computation tasks. Interestingly, the bottom-right example reveals that despite being model specialized for charts, ChartSketcher retains significant capacity for understanding natural images. This finding broadens the potential application scope and highlights the versatility of ChartSketcher. For more detailed case visualizations, please refer Appendix
Conclusion
We presented ChartSketcher, a novel multimodal feedback-driven approach for chart understanding. By enabling MLLMs to visually sketch charts during their reasoning process through a programmatic sketching library, method more closely mirrors human cognitive behavior in visual analysis tasks. The two-stage training strategy combines cross-modal distillation and reinforcement learning with Sketch-MCTS, which allows the model effectively learn and refine Eketch-based reasoning chains. Experimental results demonstrate that ChartSketcher's integration of visual feedback and iterative refinement outperforms existing methods various chart understanding benchmarks. While our empirical results are strong, a formal theoretical analysis of the convergence and robustness of the Keecch-feedback loop remains an important avenue for future research. Future work could also explore expanding the sketching capabilities and feedback mechanisms to tackle even more complex visual reasoning scenarios, such as 3D or dynamic charts.
Acknowledgments and Disclosure of Funding
This work was supported by the National Key Research and Development Program of China (2022FC33036000), National Natural Science Foundation of China (No. 62137002, 62293550, 62293553, 62293554, 62450005, 62437002, 62477036. 62477037, 62176209, 62192781, 62306229), 'LENOVO-XITU" Intelligent Industry Joint Laboratory Project, the Shaanxi Provincial Social Science Foundation Project (No. 2024P041), the Natural Science Basic Research Program of Shaanxi (No. -B--5993,, the Youth Innovation Team Shaanxi Universities 'Multi modal Data Mining and Fusion', Project of China Knowledge Center for Engineering Science and Technology, the Zhongguancun Academy Project (No. 20240103), the Youth Al Talents Fund China Association of Automation (Grant No. HIBRCJJKYZD-2044.2
10

---

References
[1] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang. Sinan Tan, Peng Wang. Junyang Lin, Chang Zhou, and Jingren Zhou, Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
[2] Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui Ham, and Xiangyu Zhang. Onechart: Purify the chart structural extraction via one auxiliary token. arXiv preprint 4011:2240.09987, 2024.
[3] Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. cot: A novel benchmark for multi-domain multi-step multi-modal chain-of-thought. In Lun- Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics ( (Volume J: Long Papers), ACL 2024, Bangkok, Thailand, August 1 11-16, 2024, pages 8199-8221. Association for Computational Linguistics, 2024.
[4] Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, and Zhaoyang Liu. Expanding performance boundaries of open open-source multimodal models with model, data, and test-time scaling, 2025.
[5] Kanzhi Cheng. Yantao Li, Fangzhi Xu, Jianbing Zhang. Hao Zhou, and Yang Liu. Vision laiiunla models can self-improve reasoning via reflection. In Luis Chiruzzo. Alan Ritter, and Lu Wang, editors. P Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL 2025 Volume Long Papers, Albuquerque, New Mexico, USA, April 29- May 4, 2025, pages 8876-88868888888888 8892. Association for Computational Linguistics, 2025.
[6] Kanzhi Cheng, Wenpo Song. Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, and Jiajun Chen. Caparena: Benchmarking and analyzing detailed image captioning in the LLM era. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 August I, 2025, pages 14077-14094 Association for Computational Linguistics. 2025.
[7] DeepSeek-AL, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang. Xiao Bi, Xiaokang Zhang, Xingkai Yu, and Zhen Zhang. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, 2025.
[8] Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. CoRR, abs/2411.14433 2024.
[9] Kawin Ethayarajh, Winnic Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kicla. KTO: model alignment as prospect theoretic optimization. CoRR, abs/2402.01 01300, 2024.
[10] Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei A. F. Florêncio. and Cha Zhang. Refocus: Visual editing as chain of thought for structured image understanding. CoRR, abs/2501.05455 2025.
[11] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama 3 herd of models, 2024.
[12] Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. Chartllama: A multimodal LLM for chart understanding and generation. arXiv preprint 1114333, 2023.
[13] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. 2022.
[14] Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zeutlemoyrr Noah A. Smith, and Ranjay Krishna. Visual sketchpad: Sketching visual chain of thought for multimodal language models. In Amir Globersons. Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak. and Cheng Zhang, editors, Advances in Neural Information Processing Systems 38: Annual Conference on Newral Information Processing Systems 2024, NewrIPS 2024, Vancouver. BC, Canada, December 10 15. 2024, 2024.

---

[15] Muye Huang, Han Lai, Xinyu Zhang, Wenjun Wu, Jie Ma. Lingling Zhang, and Jun Liu. Evochart: A benchmark and self-training approach towards real-world chart understanding. In Toby Walsh, Julie Shah, and Zico Kolter, editors, AAAI-25. Sponsored by the Association for the Advancement of Artificial Intelligence, February 25 March 4, 2025, Philadelphia, PA, USA, pages 5680-3688, AAAI Press, 2025.
[16] Zheng Huang, Kai Chen, Jianhua Hc, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. ICDAR2019 competition on scanned receipt OCR and information extraction. In 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20-25, 2019, pages 1516-1520. IEEE, 2019.
[17] Drew A. Hudson and Christopher D. Manning. GQA: new dataset for weal-world visual reasoning and compositional question answering. In IEFE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 6700-6709. Computer Vision Foundation! / IEEE, 2019.
[18] Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figurequ: An annotated figure dataset for visual reasoning. In ICLR, 2018.
[19] Ronak Kosti, José M. Álvarez, Adrià Recasens, and Agata Lapedriza. Context based emotion recognition using EMOTIC dataset. IEEE Trans. Pattern Anal. Mach. Intell., 42(11))2755-2766, 2020.
[20] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Ducrig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. CoRR. abs/1811 1.0922, 2018
[21] Jordy Van I Landeghem, Rafal Powalski, Rubèn Tito, Dawid Jurkiewiez, Matthew B. Blaschko, Lukasz Borchmann, Mickaël Coustaty, Sien Moens, Michal Pietruszka, Bertrand Anckaert, Tomasz Stanislawek, Pawel J6ziak, and Emesst Valveny. Document understanding dataset and evaluation (DUDE). In IEFF/CVF International Conference on Computer Vision, ICCV 2023, Paris, Frunce, October 1-6, 2023, pages 19471-19483, 2023.
[22] Kenton Lee, Mandar Joshi, Tulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Fisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In ICMI,, pages 202: 18893-18912, 2023.
[23] Matan Levy, Rami Ben-Ari, and Dani Lischinski. caasifiiattion-rgresssiso for chart comprehension. In ECCV, pages 13696; 469 484, 2022.
[24] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma. and Chunyuan Li. Llava-next-interleaae:: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895, 2024.
[25] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In IEEL/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 26753-26763. IEEE, 2024.
[26] Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier, and Julian Martin Fisenschlos. Matcha: Enhancing visual language pretraining with math reasoning and chart derendering. In ACL. pages 12756-12770. 2023.
[27] Zuyan Liu, Yuhao Dong. Yongming Rao, Jie Zhou, and Jiwen Lu, Chain-of sp-ttt Interactive reasoning improves large vision-language models. CoRR, ahs/2403. 1 1266,, 2024.
[28] Zihan Ma, Taolin Zhang, Maosong Cao, Junnan Liu, Wenwei Zhang, Minnan Luo, Songyang Zhang, and Kai Chen. Rethinking verification for Ilm code generation: From generation to testing, 2025.
[29] Ahmed Masry, Parsa Kavehzadeh, Do Xuan Long, Enamul Hoque, and Shafiq Joty. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. In EMNLP, pages 14662-14684, 2023.
[30] Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq R. Joty, and Enamul Hoque. Chartqu: A benchmark for question answering about charts with visual and logical reasoning. In Findings of ACI, pages 2263-2279, 2022.
[31] Ahmed Masry, Mehrad Shahmohammadi, Md. Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartin- struct: Instruction tuning for chart comprehension and reasoning. arXiv preprint 20X0v22403.09022, 2024.
12

---

[32] Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, and Shafiq Joty. Chart- gemma:: Visual instruction-tuning for chart reasoning in the wild, 2024.
[33] Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and C. V. Jawahar. Infographicvqa. In IEER/CVF Winter Conference Applications of Computer Vision, WACV 2022, Walkoloo, HI, USA, January 3-8, 2022, pages 2582-2591. IEEF, 2022.
[34] Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docyqa: A dataset for VQA document images. In IEEE Winter Conference on Applications of Computer Vision, WACV 2021, Waikoloa, HI, USA, Jamuary 3-8, 2021, pages
IEEE, 2021.
[35] Fanqing Meng, Wengi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang. Yu Qiao, and Ping Luo. Char- tassisstant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning. arXiv preprint arXiv: 2401.02384, 2024.
[36] Nitesh Methami, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. WACV. pages 1516-1525, 2020.
[37] OpenAl, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, and Alex Tachard Passos Zhuohan L.. Openai ol system card, 2024.
[38] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, and Florencia Leoni Aleman et Gpt-4 technical report, 2024.
[39] Bryan A. Plummer. Liwei Wang. Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier. and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer amage-to-sentence models. 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2641-2649, IEEE Computer Society, 2015.
[40] Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, and Zihan Qiu. Qwen2.5 technical report, 2025.
[41] Hao Shao. Shengju Qian, Han Xiao, Guanglu Song, Zhuofan Zong. Letian Wang, Yu Liu, and Hongsheng Li. Visual cot: Advancing multi- language models with a comprehensive dataset and benchmark for chain-of thought reasoning. In Amir Globersons. Lester Mackey, Danielle Belgrave, Angela Fan, Urrich Paquet, Jakub M. Tomezak, and Cheng Zhang, editors, Advances in Neural Information Processing Systems Annual Conference Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, 2024.
38:
[42] Oleksii Sidorov, Ronghang Hu, Mareus Rohrbach, and Amanpreet Singh. Textcaps: dataset for image captioning with reading comprehension. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan ancMhh Frahm, editors, Computer Vision ECCV 2020 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II, volume 12347 of Lecture Notes in Computer Science, pages 742-758. Springer. 2020.
[43] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards VQA models that can read. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 8317-8326. Computer Vision Foundation IEEE, 2019.
[44] Gemini Team, Rohan Anil, Sebastian Borgcaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, and Anja Hauth. Gemini: A family of highly capable multimodal models, 2024.
[45] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, and Soroosh Mariooryad al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
[46] Kimi Team, Angang Du, Bofei Gao, Bowei Xing. Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with Ilms, 2025.
[47] Qwen Team. Qvq: To see the world with wisdom, December 2024.
[48] Qwen Team. Qwq-32b: Embracing the power of reinforcement learning. March 2025.
[49] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech uesd 2idds-200-001 dataset. Technical Report ENS-TR-2011- California Institute of Technology, 2011.
13

---

[50] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge. Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-langee saiin-la model's perception of the world at any resolution, 2024.
[51] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma. Brian lchter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Ohain-of-thoughu prompting elicits reasoning in large language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA, USA, November 28- December 2022, 2022.
[52] Rengiu Xia, Bo Zhang, Haoyang Peng. Ning Liao, Peng Ye, Botian Shi, Junchi Yan, and Yu Qiao. Struetchart: Perception, structuring, reasoning for visual chart understanding. CoRR, abs/2309. 11268, 2023.
[53] Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, and Yu Qiao. Chartx chartvim: versatile benchmark and foundation model for complicated chart reasoning. CoRR, abs/2402. 12185, 2024.
[54] Fangzhi Xu, Qiushi Sun, Kanzhi Cheng, Jun Liu, Yu Qiao, and Zhiyong Wu. Interactive evolution: A ncural-symbolic self-training framework for large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27- August 1, 2025, pages 12975-12993. Association for Computational Linguistics. 2025.
[55] Fangzhi Xu, Hang Yan, Chang Ma, Haiteng Zhao, Qiushi Sun, Kanzhi Cheng, Junxian He, Jun Liu, and Zhiyong Wu. Genius: generalizable and purely unsupervised self-training framework for advanced reasoning. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Popers), ACL 2025, Vienna, Austria, July 27- - August 1, 2025, pages 13153-13167. Association for Computational Linguisties, 2025.
[56] Zhengzhuo Xu. Sinan Du, Yiyan Qi. Chengjin Xu, Chun Yuan, and Jian Guo. Chartbench: A benchmark for complex visual reasoning charts. CoRR, abs/2312.1599) 2023.
[57] Zhengzhuo Xu, Bowen Qu, Yiyan Qi. Sinan Du, Chengjin Xu, Chun Yuan, and Jian Guo. Chartmoe: Mixture of diversely aligned expert connector for chart understanding, 2025.
[58] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, and Fei Huang. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
[59] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shatran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurlPS 2023, New Orleans, IA, USA, December 10 16, 2023, 2023.
[60] Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, and Mingyuan Zhou. Relative preference optimization: Enhancing LLM alignment through contrasting responses across identical and diverse prompts. CoRR, abs/2402. 10958, 2024.
[61] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event deseriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[62] Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yiehen Xu, Qin Jin, Zhang, and Fei Huang. Tinychart: Efficient chart understanding with visual token merging and program-of-thougee learning. arXiv preprint arXiv: 2404.16635, 2024.
[63] Mingyang Zhou, Long Chen Yi R. Fung, Christopher Thomas, Heng Ji, and Shih-Fu Chang. Enhanced chart understanding in vision and language task via cross-modal pre-training of ACL, 2023.
pre-traing on plot table pairs. In Findings
[64] Yuke Zhu, Oliver Groth, Michael S. Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages
IEEE Computer Society, 2016.
14

---

Appendix Contents
A. Challenging Case in Current MLLMs
B. Programmatic Sketching Library
C. Detailed Process of Sketch-CoT Data Synthesis
Detailed Skeich-MCTS Algorithm
E, Detailed Experimental Results
20
Additional Cases of ChartSketcheri
A Challenging Case in Current MILMS
We conducted comprchensive testing multiplc multimodal language models, including QvQ-Preview, OpenAl-o3-mini, GPT-4o, and Kimi-1.5-pro [46]. As illustrated in Figure our analysis revealed that despite possessing text reasoning capabilities, both QvQ and OpenAl-o3-mini produced erroneous responses even after extended deliberative processes. Given that o3's reasoning mechanisms are not externally observable, we focused our analytical efforts on QvQ's reasoning trajectory. This examination demonstrated that despite generating elaborate chains of reasoning. QvQ consistently failed to overcome fundamental visual perception limitations specifically, the inability verify the presence "2015" within the image. Additionally, our findings indicate that the performance of more advanced models such as GPT-4o and Kimi-1.5 1.ii-1.5-pp remains sub- optimal in this context, as they demonstrate significant deficiencies in comprchending eritical visual information essential for accurate task completion.
QrQ-Preview:
be sakwing The ailewes Pestio pvemann a0 an ahi von iiee 200 TH1 fiee was permme oe te 18 Exeting ar"0 The wirch hard pighs
Posltive aconomic sentiment Increasing 2016
anvatoe
The griph poines years (2010,
20% determand
masked
Anower The valoxied of "podd 2015 dinetty provided in the The closnd 39% in 2014 and 29% in 2016.
05
57
QgenAl-a3-ntn
54
The proces
84
devemine the value "Geud" in the 20155
L.Lostts the 2015 the burizuntal acts (c-aoda) of the chart.
Good
regresenting "Good" and cheds veriial position relative the percentuge en the isft (-aaaa).
2.30eetl the grees
3 3.Fr the chust, the grece loc inserects the your 2015 42%.
This, the value "Good" 2015 47%
2010
2018
Soarce Survey of Mighrin canductae Jue 25- 2018 02.
Kimi- 1.5mm (Cal Prompeingh:
Seontyy image
in on legred
represente
find
for 2015 We can dara 2014 3006, which
70% respccivrll
Question:
extmatz the value 2015, we simply take the averuge of the values for 2014 and 2016. follows.
The calouacton
GT: 57
(39%, 20%) 2 30%
So, based on the chem data, we can estimane that the percentage people who rosed the camomic situaion as "Cood" in 2015 arproscmanly 50%.
Figure 5: An illustrative example of challenge in current MILMs
B Programmatic Sketching Library
To equip MLLMs with advanced image sketching capabilities, designed lightweight and versatile drawing language library. This library supports operations create and manipulate basic geometric shapes-such points, lines, circles, arrows, and their combinations through simple and intuitive command syntax. During the reasoning process, MLLMs can dynamically insert drawing commands to generate new visual elements or modify existing ones using operations like translation, rotation, and deletion. Below, we detail the pseudocode structure, supported commands, and their usage.
15

---

B.1 Pseudocode Overview
The pseudocode serves as a structured and concise language for defining geometric shapes and applying transformations. It operates within normalized coordinate system: The top-left corner of the canvas corresponds to (0, 0), and the bottom-right comer corresponds to (1, 1). The horizontal axis is denoted by and the vertical axis by y-
The pseudocode executes commands line-by-line, starting with the BEGIN keyword and terminating at END, Any commands written after END are ignored.
Key Features of the Pseudocode
Execution Blocks: Commands are executed between BEGIN and END. Lines outside this block are ignored.
Normalized Coordinates: The canvas is scaled to unit square with (0,0) at the top-left and (1, 1) at the bottom-right.
Dynamic Operations: Shapes be created, modified, and deleted in real-time through simple commands.
Geometric Flexibility: Supports points, lines, circles, rectangles, and arrows, covering wide range of visual elements.
B.2 Supported Commands
B.2.1 Shape Creation
The library allows for the creation of several geometric shapes. Below are the commands and their specific syntaxes.
Shape Creation Commands
create_point entity_id color Creates point at coordinate (a, with the specified color. Example: create_point p1 0.2 0.2 red
Point:
Line: create_ line entity_id y1 y2 color Creates line connecting (x11 y1 and U2) with specified color. Example: create_line 0.2 0.2 0.8 0.8 blue
create rreat_cirrle entity_id cy radius color Creates circle centered (cz, cy) radius radius and the specified color. Example: create_circle 0.5
Circle:
0.5 0.1 green
Rectangle: preate_rectangl. entity_id y1 x2 y2 color Creates rectangle with the top-left comer (x1, v1) and bottom-right corner (æ2, v2). Example: create_rectangle r1 0.1 0.1 0.4 0.4 black
Arrow: create _arrww entity id x1 x2 y2 color Creates arrow from (f1, (tail) (22, U2) (head). Example: create, arrow 0.3 0.3 0.7 0.7 purple
B.2.2 Transformation Operations
library supports following operations to manipulate existing shapes:
Transformation Commands
Translation:
translate entity_id dx dy Moves the shape identified by entity_id by (dæ, dy). Example: translate 0.1 0.1
Rotation: rotate entity_id _id angle cy Rotates the shape identified by entity_id
around the point (cz,cy) ey) by angle degrees. Example: rotate 11 45 0.5 0.5
Deletion: delete entity_ie Deletes the shape identified by entity_id. Example: delete 11
16

---

B.2.3 Program Control
Special commands control the execution of the pscudocode:
Begin Command: BEGIN Marks the start of pseudocode exccution. commands following this are
executed until END is encountered. Example: BEGIN
End Command: END Terminates pseudocode execution. Commands after END ignored. Example: END
B.3 Example Pseudocode
The following example illustrates the creation and transformation of shapes using the pscudocode:
Example Pscudocode
BEGIN
create_point 0.2 0.2 red
create_line 11
0.8 0.8 blue
create_circle c1 0.5 0.5 0.1 green
create_arroz a1 0.3 0.3 0.7 0.7 purple
translate 11 0.1 0.1
rotate 11 45 0.5 0.5
END
reetanlee r1 0.1 0.1 0.4 0.4 black
Explanation: The above pseudocode performs the following steps:
Creates a red point, blue line, green circle, and purple arrow.
Translates the line by (0.1, and rotates 45 around the center (0.5, 0.5).
The rectangle defined after END is ignored.
B.4 Frequently Asked Questions
1. How can change the color of a shape? Specify the color in the creation command. For example: create_point p1 0.2 0.2 red
2. What does END do? The END command stops the pseudocode exccution. Commands after END ignored.
3. What happens if forget END? If END is missing, the parser continues parsing until the last line. Always include BEGIN and END.
How do translation and rotation work? Translation: Moves the shape by (dæ, dy). Rotation:
Rotates the shape around specified center (cr, cy) by a given angle.
5. How do delete shape? Use the delete command with the shape's identifier. For example: delete 11
C Detailed Process of Sketch-CoT Data Synthesis
As shown in Figure we present the data synthesis process during the cold start stage. The process begins with the synthesis of data without reflection, as the synthesis of reflection-base data relies on the initial non-reflective data.
In the non non-reffesn reflective data synthesis process, we use seed techniques to augment QA pairs. Subsequently. the reasoning process is distilled using Qwen2.5-32B, The distillation prompt is formally defined in Prompt C below.
17

---

Cold Strat
Charts & Seeds
Query Set
Distilled Reasoning Path
Intentionally Created Errors
Natural Sperch Particles
First, I should...
First, should....
<code>>
code>
Distill
Altight, will...
Oh, I think made mistake....
5wen2.5
<code"
<code>
Finally,
Ah, that's better. Next, I'll...
Figure 6: The synthesis process of Sketch-CoT. The left side illustrates the CoT synthesis process without reflection, while the right side demonstrates the synthesis process of CoT with reflection.
Prompt for Distilling Non-Reflective Sketch-CoT
Your task is to output simulated human-like reasoning dialogue. Since you cannot draw directly, all drawing operations must be expressed in pseudocode enclosed between the keywords BEGIN and END.
The following the requirements for the simulated dialogue:
not reveal the final answer before completing the reasoning process. The answer must derived from the reasoning steps. The BEGIN and END markers should be embedded within the text, and will parse them automatically to generate the drawings. Your output should be conversational, interspersed with brief drawing instructions. Avoid drawing too much at once. To solve any given problem, you must draw least twice. If solving problem involving X Y axes, you must draw auxiliary lines series label shown, you not need align the value to the coordinate axes to obtain the number. the label is not shown, you can align to the coordinate axis or infer the value from other evidence. The output format should be similar to: "Fürst, I will circle BEGIN.... END. Hmm, it looks like I have drawn... Then will... BEGIN ... END. " After each drawing, act as if you can visually interpret the content you have drawn to make the explanation vivid. You must not use exhaustive methods for drawing. Draw only what is relevant the question. For instance, if the X-axis line is missing, you need to infer the content. Partial conclusions must be derived through drawing.
on the
axis locate the target.
Specific instructions series label show information. series label show=True, there no need to align the data points to the numerical axis; simply state the values based on the series label position. series label shos-False, draw auxiliary lines to align the data points to the numerical axis for accurate evaluation.
follows: 1. Check and output whether all the data points involved have
There is legend area annotation, do colors describe legend. 3. Do not reveal that you can see annotations or metadata. Your output should he string that simulates human reasoning dialogue, with no additional content.
rectangle to draw the legend area. Instead,
For reflective Sketch-CoT, have built upon the non-reflective Sketch-CoT by introducing modifications. Through carefully designed prompt Qwen2.5-32B, we enable the model to randomly introduce errors during the reasoning process and subsequently self-correct them. The prompt ensures that errors are systematically generated and resolved, fostering reflective mooblem-solvin approach. The full content of the prompt can be found in Prompt
18

---

Prompt for Reflective Sketch-CoT
I need you to modify and refine the following dialogue by injecting reflective processes, replacing the originally completely correct solution process.
The method is follows: Using BEGIN and END boundaries, everything between and including
END and what comes before is called the "former part," and everything after END is called the "latter part." To create reflection, you first need to introduce an error.
The error type should be: coordinate errors between BEGIN and END in the former part, replacing them with coordinates of other points. Subsequently, you should correct the error by discovering and fixing the latter part, outputting new BEGIN and END commands to complete the dialogue.
Keep the language fluent and conversational. Let me further explain: The content between BEGIN and END is an operation, and you will "see" the result of the operation after END. Therefore, your reflection process must occur immediately after the END output, not after the erroneous reasoning concludes. Note that there no legend area; if you one, you should remove it and colors describe the legend instead. Do not include any comments in the instructions, as instruction code does not support comments. Do not include any hints that you are deliberately making errors.
The reflection must be brief and accurate, keeping the dialogue concise and organized. You should directly modify the dialogue rather than reflecting after erroncous reasoning ends. For example: ...BEGIN instruction END Wait/Hold on/Oh/Hmm, might have made mistake... (explain the reason for the mistake).. ... redraw now. BEGIN instruction END...
Here is the dialogue you need to modify:
D Detailed Sketch-MCTT Algorithm
Algorithm demonstrates detailed workflow of Sketch-MCTS, along with comprehensive descriptions of its speeific parameters. Our proposed Sketch-MCTS represents a variant of the Monte Carlo Tree Search methodology. We have modified the termination condition of MCTS to conclude when either SUCCmim is satisfied or the number of simulations exceeds Mimm Additionally, we have retained the -w-temperattuee rollout approach evaluate the value of the current node.
Compared methods that randomly generate N correct solution paths, Sketch-MCTS offers the following advantages: First, we can evaluate value the current step in real-time, enabling stable expansion of reasoning tree, whereas random rejection sampling methods of N paths are entirely stochastic. Second, we can derive multiple correct and incorrect nodes from high-value nodes, creating step-level contrasts between correct and incorrect samples, which is crucial for off aff-policy reinforcement learning. In contrast, rejection sampling methods, with their completely independent reasoning paths between samples, cannot achieve step- level comparative analysis. Third, we can control the length of reasoning paths during the dynamic sampling tree process and ensure that the computational overhcad of the sampling process remains manageable. Rejection sampling methods are static processes and cannot effectively control computational costs or reasoning path lengths.
Symbol
Meaning
Default value
textual query posed by the user chart image provided the model
N/A
N/A
y
gold (reference) answer
N/A
C
multimodal large language model queried during search
N/A
R
difforentiable sketch renderer (ChartSketeher back-end)
N/A
u.Q n../u.N
cumulative reward visit count of node u exploration constant in the PUCT formulation weighting factor for the depth-based penalty small constant preventing division by zero maximal number of MCTS iterations carly-stop threshold on successful terminal nodes
0/0
OPUCT
1.9
Alen
0.05
x 10
SIM_lim
25
SUCC_lim
3
MAX_DEPM
maximal dialogue depth (assistant-user turns) maximal expansions sampled per node maximal non-virtual children per node sampling temperatures for expansion / rollout
Omax
MAX CHILD
3
Thish Thih/TTow Tho
0.9 0.9/0.4
19

---

Algorithm Sketch-MCTS
Require:
visual query chart image I, ground-truth answer multimodal LLM L. sketch executor R 1: hyper-parameters @ {SIM lim, SUCC. lim, MAX DEPTH, Cmax, MAX CHILD, Thigh, Tlow CrUCT, E
function UCB(u)
b depth-aware upper confidence bound
return
u.Q u.N CPUCT
Inku.parent.Nv +1)
Xue 0.01 u.depth + 0.3 0.7 max((, depth length penalty
u.N +e
explotcation
exploration
function
6: root <- NOD(..NII((user:: D)
7: successes <- 0
8: for <- 1 to SIM lim while successes < SUCC lim do
Selection
+ descendant root maximising UCB(-)
Expansion
10:
if -v.temnnaal A fullul A w.depth MAX DEPTH then
11:
sample Cmax replies (rv} L(T Thigh
for all do append r; to dialogue; extract SKETCH- COT program if no code then add terminal child u, u.reward ISRIGHT((r: y)
13:
14:
15:
16:
else
parse & render via R; on failure virtual child (r = 0) continue persist hitmap; send as user visual fcedback
18:
19:
duplicate/subset detection - virtualise redundant children
20:
end if
21:
u.ternnnal A u.reward then successes+
22:
end if end for
23:
24:
v.full + (#children MAX_CHILD)
25:
end if
Rollout
26:
if -v.terminal A -o.rolled A -e.viruual then
27:
simulate assistant - render + feedback with T = Tiw terminate no-code, render-fail, or depth limit; set v.reward v.rolled <-true end if
28:
29:
30:
Backpropagation
31:
for u E ancestors(r) with virtuul do
32:
u.N +- u.N +1; u.Q u.Q e.reward
33:
end for
34: end for
35:
return non-virtual child of root with maximal mean value Q/N
E Detailed experimental results
We conducted comprehensive evaluation of ChartSketcher's chart comprehension ability and overall per- formance on 18 datasets. shown in Table S VisCoT represents the weighted average of remaining 12 general-purpose datasets, serving as an aggregated metric to assess the model's general understanding capabilities.
To further validate the robustness and generalization capabilities of our proposed ChartSketcher method, we conducted additional experiments the challenging ChartX dataset. ChartX serves diverse benchmark for chart understanding, featuring more complex chart types reasoning tasks. presented in Table we compared CherhSeethher against another advanced chart understanding model, ChartVL...
E.1 Comparison with ReFocus
In addition to evaluating on diverse benchmarks, we also conduct a direct performance comparison with ReFocus [10]. To ensure fair and direct comparison, we evaluated our method on the test set provided by the ReFocus paper.
20

---

Table 3: Performance comparison on the ChartX benchmark. This experiment is conducted to further validate the generalization ability of our method on more diverse and complex chart datasets. denotes Structural Extraction with different tolerance levels.
SE (Structural Extraction)
Model
QA
Strict
Slight
High
ChartVLM
23.18
30.68
38.30
43.84
ChartSketcher
35.65
41.74
47.06
56.94
Table 4: Performance comparison with ReFocus on its own test set (subsets of ChartQA). All models were evaluated under the same conditions to ensure a fair comparison.
Model
chartqa-vbar
chartqa-hbar
Models based Qwen2VL-2B
Qwen2VL-2B (Base)
51.31 59.69
48.65 54.73
ChartSketcher-2B
Models based on phi-3.5-4.2B
phi-3.5-4.2B (Base)
63.10
60.10
ReFocus--pi3.554222
72.20
71.00
ChartSicecche--hhi3.5--2222
74.35
72.30
E.2 Ablation on RL Sampling Strategy
To isolate and quantify the contribution of our proposed Sketch-MCTS algorithm, we conducted an additional ablation study. A key question raised during the review process was how sophisticated Monte Carlo Tree Search-basse based sampling strategy compares against simpler, more "naive" reinforeement learning approach. To address this, we designed baseline that uses simple rejection sampling method for collecting training data for the RL phase. This bascline, termed "w/ naive RL", embodies a less guided exploration strategy.
We trained ChartSketcher- 2B model with this naive RL data and compared its performance against our full model, which leverages Sketch-MCTS. The results are presented in Table
Table Ablation study on the RL sampling strategy. We compare the performance of ChartSketcher- 2B trained with our proposed Sketch-MCTS against version trained with a naive rejection sampling method ("w/ naive RL"))
Method
ChartQA-II
ChartQA-A
VisualCoT
ChartSketcher-2B (with Aketch-MCTS) w/ naive RI.
55.60
80.88 78.24
66.86
52.08
62.12
E.3 Analysis of Inference Efficiency
practical consideration for any reasoning model is its inference efficiency and handling of finite context windows. section, we provide quantitative analysis ChartSketcher's inference throughput.
Inference Throughput. Each step in the Sketch-CoT process requires the model re- re-process the visual feedback, which makes its inference profile distinct from models that perform purely textual reasoning. To quantify this, we measured the inference throughput of ChartSketcher-725 and compared it against QvQ Preview-72B and the base Qwen2VL-72B model. All experiments were conducted using the vLLM framework on server with 8x A800 40G GPUs.
E.4 Comparison with Chart-Expert Models
To provide comprehensive view of ChartSketcher's performance within the specialized field of chart under- standing. compare model with several prominent chart-expert models. These models specifically designed and trained chart-related tasks, making them highly relevant baselines. For this comparison, we use the recognzeee ChartQ benchmark, which includes both machine -agaminee (ChartQA-A) and human-ann annotated (ChartQA-H) -nswweinn sets. The results presented in Table 7 offer a nuanced perspective ChartSketcher's capabilities.
21

---

Table 6: Inference throughput comparison. We report the average tokens per second for both the input processing and output generation stages, The overall speed is normalized relative to ChartSketcher.
Model
Input Throughput (tokeu//s)
Output Throughput (token/s)
Overall Speed (Normalized)
ChartSketcher-222
2995
329
1.00
QvQ Preview-72B
.70
1705
1.33
Qwen2V1--7BB
1926
685
9.11
Table 7: Performance comparison with specialized chart-expert models on the ChartQA benchmark.
Model
ChartQA-Augnentee
ChartQA-Humen
OnaClait
85.30
ChantAs
93.90
ChurtSketcber-71B
Table 8: Comprehensive results 18 benchmarks.
(e) Chart understanding Expert Benchmarks
Model
CharQA 11 10
ChanQA-A
EvoChari- QA
1S]
ChanBench
56
PlxQA 56
VisCoT 31
Progrietary models
GPI-4o
$4.32 $4-00
98.48
52,
61.47
42.96
78.45
Gemini-2.0 Claude 3.5
88.24
64.64
$5.63
63.36
77.00
$5.04
90.72
36.96
$7.63
00.64
75,93
Open source experr
Qwen2VLL-72B
$2.48
88.56
$4.00
54.77
73.76
72.14
Q+Q- Preview-T2 720
$3.33
89. 76
54.32
42.
69.00
76.
Intern 28.25-V....
4M
89,
57.
63.57
57.20
28.93
OwenCVL-22
Chartkettcher
26.72
30.10
Clechhetth 73B
85.30
92.64
33.28
68.33
76,72
76.50
Attation (ChrtSieerhher 728)
w/o Rethink& RLL
77.76
91.12
51.12
50.40
67.76
72.58
Rethank wio Rt.
76.64
90.55
51.
$2.93
67.68
70.95
w/o RL
77.12
$8.80
39.81
$2.75
67.84
68.80
wio Foedback
$1.52
91.04
47.76
56.13
72.24
25.15
wio
75.12
50.08
47.43
68.16
Generie/ Dacument Scene Benchmarks
Model
Opentragee
Flickr301
3 DoeVQA 21
CUB
DUDE 21
GQA 1x
Propeietary
GPT-4o
$2.80 89
19.04
91.99 05.27
$4.75
83.25
68,
Geminii22.
$7.78
34
82.72
82.09
69.51
Clande
62.50
97.64
70.93
34,40
60.63
sonrce models
OwenEVL-7728
31.75
61.64
93.36
21.14
82.75
$7.06
Oytt-Previog-228
83.25
ChantGemaaa
Qwen2V1-2RR
$3.07
34.48
79.550
50.20
71:31
23.31
ChanSketcheer 28
64.23
68.95
69 82.82
$2.65
60.20
61.25
ChartStetcceer-335
68.68
72.19
92.68
68.09
79.10
65.85
Addattio (Char1tWknnebs-772)))
w/o Rethink&&L
62.33
66.04
89
71.14
82.75
62.17
wi Rethink wio RL.
49.05
66.17
841 04.40
60.57
72.04
61
wio RT.
$7.57
66.43
$7.87
62.45
92.25
77.64
Generie/ Document: Benchmarks
Modal
TeXiVQA 43
TeuCap 4
Infographic 35
Vissa7WW
Proprietory moddlfs
GPT-4o
93.7 73
94.25
82,22
$3.81
77 60
Gemini220
04.30
78.26
35.42
Open-woree/ expert models
Qwan2V1.
QvQ Previgw: 729
39.35
94.61
84.72
65.55
69.50
ImaenVV ImarnV1.....725 2.5.78
00.30
88 88.75
93.88
81:11
611
74.40
ChanGermma 20
74,33
70.81
$1.17
37.22
$3.20
$7.30
Owen2VL-288
S3.97
80.77
04.122
47.78
$0.15
60.40
ChurSSettceer22B
$4.03
38.05
61.90 90
ChartSkettteer-72Z
Ailtaroo (ChurtNeetshoo-222
Rethink
RL Wio Faudback
$4.23
85.42
68.33
53.51
01.63
85.81
89 94
73.80
54.85
70.80
wio Col
90.19
85.70
91.90
78.89
63.70
67.20
22

---

Question: yoar difference Tabor Force (in %g of Femaal labourers and Labur Force (in %) of Male Jabourers
GT: 17.4
ChartSketcher:
points www.eiiiaacteeeeeeeeeeeeo p1 0.4519 0.2635 sandybrowa increate_ pount p2 0,4519 0.075 orange taEND
Wait, might
coorcinases
lahorers' point... Let check again acoms like woordinate IIIl cornect it now NnInDDGGN IcBBBGINNmmenn mureane point p1 0:4519 2635 sendybrown increste point 0,4519 0.078 VnEND
wrong, y-
Alright, I've marked both points correctly now. The Feraae laburers' point is at (0 4519, 0.2635) and the Male laboreri point is at (0. (0.4519, 0.078). Nn'nNow, Illl check there looks like align them the y-axds determine their values laborers point, I'll draw a horisontal line to the y-axia. BRGrNanercate. line
value labe's shown for these points.
for either point. So, aeed
the Female
Il o 0.4319 263555 0.01 0,2635 sandybrown' anE
laboress' point, do same. vncreate 0.459 0.075 0.01 0.078 orangeNn NEEND
Based scale the axiss, value for Female laborers 1993 approximately 58.4%, hh difference beween the two 76.1% 38.4% 17.7%.
approtimatcly 76.
What the difference between Area of total land Brunci Darussalam 2007 Area (% of tocal land area) St. Vincent and the Grenadines in 2009 ?
Question:
ChartSketcher:
First, locate the data points Brunei Derasallm 2007 and
Vineent and the Grenadines in 2009. BEGINN increate point pl 0. 7.7744 0.5486 darigreem incrente
0. 0.7294 0,2639 darkolivegreem INDND
Wait, might have
mistake
coordinates
Darussalam in 2007 Let me check again. It scems I used the wrong coordinate I'll cornect that now NnNNIEEGNN CTNeeneeee point pl 0 7774 darkgreen
point 7204
Airight, I've marked both points correctly now the value for Brunei Darussalam 2007 is 72.945, and for St. Vincent and the Grenadimes in 2009, 68 46% differeace
approsimately 4.48%.
Figure 7: Visualization of an additional case study PlotQA.
F
Additional Cases of ChartSketcher
appendix, we provide a detailed account of cases involving ChartSketcher across multiple datasets. As shown in Figure Figure and Figure D our approach offers rich user interactivity and explicit interpretability. Through the reinforcement learning process, ChartSketcher acquires reasoning capahilities that are ahsent during the cold start phase, demonstrating more conversational expressions compared to those in the cold start phase.
It is worth noting that, like many reasoning models, approach faces common limitations such ineffective self-reflection, overthinking, and infinite loops of reflection. To mitigate these issues, lower the temperature during reasoning and cap the maximum chain length at 12, Addressing these limitations will be a key focus our future work.
23

---

Question: states, highest
GT: 59.66
ChartSketcher: locase
BEGIN wncreateeeeeeoiiiiieeeii pl
1819 (nEND
Alright, Tve marked the United States the x-axis. Now, Iil check values Moi Modian Median valur 53, Comparing thrse two, the highest bar valae for the United States 39.66. Waita hold on. think there might be an issue with my initial point placement. Let me double check the coordinates. Oh, placed the point wrong coordinates
Moan
should have used
REGIN RFGINNancetttt nncrosto point p1 0 1820 1827 black (FFN
conrected the placement. checking again,
So, highest
United States is indeed 59.66.
Bipartisan willingness to force to defend allres in confrict China
Question: Look at Democrats, Find the vatio between the two options?
nouffor lus Chilc, 3oc we finos to diffedd
ChartSketcher:
Finz, I'll locate the section for Democrats on the chazt. BEGIN mnreate rectangle rl 0.349 0.529 0.78d 0.5e2 blueknEED
nemaow
Indooondert
35
Alright, highlighted Democrate' section. associated with "Should" and "Should nox options. It looks like value for "Should" is 52% and for "Should ooc" is 39%, The ratio between "Should" and 'Sholld not" is approximately 1,33:1.
values
Seputlcan
Question: When does the line reach the penk?
ChartSketcher:
locate the data points England on the
1.00
BEGIN vncreate, point pl 0.0927 0, 0,4155 steelblue sacreateppintt p2 0.2622 0. 0,3069 steetbluc' increae_ point 0.4316 0.2754 steelbluch nereanee point 0.6011 0,2503 steelblue' stechbluennceeae steclblue
0,7705 0.2352
point
steelllue
Hmm, I might have made mistake with the coordinates for the last puint. Let double check that. Oh right,
coorcinate should 0.15
CGGGNNnnccrate poist pl 0.0927 0.4155 steelblue's vacreate point p2 0.2022 0. 3060 stechbluchneeee increate point 4316 0.2754 stechbluehnnroo 0.8011 0,2503 steelblue's steelblueh point
0,7705 0.2352
steelblue InEND
Alright, I'v marked all the daza points for England. New, I'll check if thene' value label showing the points looks the label shown, values VakaFor point Incceste line 94 0.15 152 0.01 0. 152 steefblae InEND
horizontal the y-axis.
Based about y-axis,
highest amoog data points.
indicating that the line reaches its peak 2014.
Figure 8: Visualization of an additional case study ChariQA
24

---

Question: around
GT: trees
ChartSketcher:
Firs, locade pectangleeeee
the the BEGIN green
END
Alright, I've highlighted the around the looks like there this region. visual trees.
surrounded
Question:
GT: elephant
ChartSketcher:
rectangle END
Atright, high region animmal located. Rased the visual information,
elephant
Question: Where the deep grey cat?
GT: In the cage
ChartSketchers
Firs, I'll locate the area where the deep grey cat might be. BEGIN create rectange 0.169 0.306 0.387 0.200 blue
END
Atright, Tvve highlighted the area where the deep grey cat located. Based the visual information, tooks like the cat sitting inside
Question: Which kind of furniture not wooden?
GT:: couch
ChartSketcher:
Fins, I'll locate the where the farmiture is positioned the image BEGIN
create rectingle
END
hightighted
where flamirure is positioned,
check there's showing type furniture label showing the value inside the sector. According the label, the value asonciated with this furniture is sofa
Figure Visualization of an additional case study on general image.
25

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The proposed ChartSketcher, with its multimodal feedback-driven reasoning and Sketch- CoT approach, directly addresses the challenges outlined and demonstrates its effectiveness through experiments chart understanding and general vision tasks.
Guidelincs:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Due to space constraints in main text, limitations of ChartSketcher and future directions are discussed in Appendix FF
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings. model well- sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice
what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Yes, the paper provides experimental evidence support the discussion of ChartSketcher.
Guidelines:
The answer NA means that the paper does not include theoretical results.
26

---

All the theorems, formulas, and proofs in the paper should be numbered and cross cross-reference..
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can cither appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, all experiments in the paper are reproducible and will open-sourced GitHub. The code is partial provided in the supplementary materials due to anonymity considerations.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility. which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g.. a large language model), then there should either be a way to access this model for reproducing the results a way to reproduce the model (e.g.. with open- dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed source models, it may be that access to the model is limited in some way (e.g.. registered users), hut it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Yes, all model weights and code will he open-soureed.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines for more details.
http:////ipps.ccpppppe
guides/Cool https///ooooouuoiiiiioooooooo
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
27

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://nipa.cc/ppun
Buzdee//DodeSubmiissiooooPP
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, all training hyperparameters presented in the paper, and those that could fit in the main text are provided in the appendix.
Guidelines:
The answer means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experimental results are the averages obtained after running three trials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanicd by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap. etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state they have 96% CI, if the hypothesis of Normality errors not verified.
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Yes, we have provided detailed disclosures about the computational resources. Guidelines:
The answer NA means that the paper does not include experiments.
28

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethhie httph:tttt//t
Answer: [Yes]
Justification: This paper does not involve ethical issues. After carefully reviewing the Ethics Guidelines, it is confirmed that it fully complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Codc of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make
preserve anonymity (e.g.. if there is special consideration duc laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper does involve societal issues; therefore, it does generate require discussion of societal impacts.
Guidelines:
The answer NA means that there is no socictal impact of the work performed. If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considcrations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
conference expects that many papers be foundational research not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase models, providing defenses in addition attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, we can confirm that the data released in this paper does not contain any risky data.
Guidelines:
The answer NA means that the paper poses such risks.
29

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the data used in this paper is properly licensed and credited. Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version the asset is used and, possible, include a URL.
name of license (e.g.. CC-BY 4.0) should be included each asset. scraped data from particular source (c.g., website), the copyright and terms of service of that source should provided.
assets are released, license, copyright information, and terms of use the package should be provided. For popular datasets, pappreaii thoode.code.... www.cco//caa/.c com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, all assets will be publicly available after the review process, and some will be provided in the supplementary materials.
Guidelines:
The answer means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
answer NA means that paper does not involve crowdsoureing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of paper involves human subjects, then as much detail as possible should he included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
30

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMSs if it an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is required.
Question:
Answer: [Yes]
Justification: LLMs are core components our work, and our method hased on LLMs and MILMMs For all I.I.ss and MI.LMs used, we have described their usage in detail and provided citations.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy (https: htta::///euu...eeeeeee should or should not be described.
http::::/aarrrppeec////foereeeee ccc/Cooofeeeccse222555555555 LLM) for what
31


Reviews:
[{"review_id": "MDLibBAVIP", "reviewer": "Reviewer_QsY7", "summary": "This paper introduces ChartSketcher, a method for improving multimodal large language models (MLLMs) in chart understanding tasks. The core idea is to enable step-by-step visual reasoning by equipping MLLMs with a programmatic sketching interface (Sketch-CoT), allowing them to annotate, reflect, and correct their reasoning on chart images through iterative multimodal feedback. ChartSketcher employs a two-stage training scheme: first, a 'cold start' phase with cross-modal distillation and stepwise reasoning pattern acquisition from synthetic annotated data, followed by off-policy reinforcement learning (RL) with a Monte Carlo Tree Search (MCTS) variant, to refine reflection and error correction capabilities. Empirical evaluations show that ChartSketcher improves performance over competitive baselines on a range of chart-specific and general vision-language benchmarks, and ablation studies clarify the importance of its feedback, reflection, and RL components.", "questions": "See the weaknesses", "limitations": "No", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "*Strengths*\n\n1. ChartSketcher presents a mechanism for visual sketching during multimodal CoT reasoning, providing a clear methodological advance over text-only or region-cropping based approaches. The method grounds abstract reasoning into explicit visual steps, as well-illustrated in Figure 1, which shows the circular process of solution, reflection, correction, and continued multimodal feedback.\n\n2. Extensive benchmarks are reported in Table 1, demonstrating tangible improvements versus strong open-source and proprietary models (e.g., Qwen2VL-72B, QvQ-Preview, and even GPT-4o) on chart-specific datasets, with good transfer to general vision-language QA tasks. The inclusion of ablation studies provides further insight into the importance of each architectural and training component.\n\n3. This work describes the creation of a substantial annotated multimodal reasoning dataset, which is a significant contribution for future research in this area.\n\n*Weaknesses*\n\n1. While the empirical advances are clear, the paper lacks a deep theoretical analysis or guarantees about the convergence/robustness of the multimodal sketch-feedback loop or the off-policy RL setup beyond empirical success. Readers might expect some formal theoretical justification or deeper analysis of why Sketch-CoT and visual feedback provide improvements, versus being primarily empirically driven innovations.\n\n2. The feasibility of scaling the programmatic sketching framework and feedback pipeline to more complex, real-world chart styles (e.g., 3D charts, interactive/time-varying charts, annotated infographics) is not systematically explored. More benchmarks can be evaluated, such as ChartX and CharXiv.\n\n3. More baselines can be supplemented to increase the impact in the field of Chart Understanding, e.g., StructChart,ChartVLM, ChartAst, OneChart", "was_revised": true, "final_justification": "After careful review of the author's revisions and responses, I confirm that **all concerns raised in my initial assessment have been adequately addressed**. The rebuttal significantly strengthened the manuscript's methodology, clarified key ambiguities, and improved the overall coherence of the argument. I recommend this paper for acceptance.", "rebuttal": "## overall:\nThank you very much for your detailed review of our paper, and for your valuable feedback. We are pleased that you recognized the novelty of our method, our extensive experiments, and the value of our contribution of a high-quality annotated dataset to the community.\n\nWe have carefully considered each of the weaknesses you pointed out. In our response, we will address your concerns regarding theoretical depth, benchmark diversity, and the inclusion of additional baseline models one by one.\n\n## weakness 1:\n## reply:\nThank you for your valuable suggestion. Our work is situated within a significant method that draws inspiration from complex human cognitive processes to address current challenges in chart understanding. \n\nIndeed, in the domain of cognitively-inspired AI, many pioneering works establish their contributions primarily through robust empirical validation. This is often because a formal theoretical analysis of such complex, interactive systems—especially those involving Large Language Models—presents a formidable open challenge. Prominent recent examples, such as \"Mind's Eye of LLMs\", \"Enhancing LLM Reasoning via Vision-Augmented Prompting\", and \"Goal Reduction with Loop-Removal...\", all exemplify this approach. They introduce novel frameworks inspired by human cognition and demonstrate their efficacy and potential through extensive empirical success, which constitutes their primary contribution.\n\nWhile a formal theoretical analysis is exceptionally challenging for such complex, cognitively-inspired systems, our comprehensive experiments robustly demonstrate the effectiveness of the proposed ChartSketcher method. We believe that establishing this empirical success is a crucial and indispensable first step that validates the potential of this paradigm, paving the way for future theoretical investigation. In our revised manuscript, we will explicitly state this as a limitation in the conclusion and highlight it as a key avenue for future research.\n\n[1]Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models [NeurIPS 2024]\n\n[2]Enhancing LLM Reasoning via Vision-Augmented Prompting [NeurIPS 2024]\n\n[3]Goal Reduction with Loop-Removal Accelerates RL and Models Human Brain Activity in Goal-Directed Learning [NeurIPS 2024]\n\n## weakness 2:\n## reply:\nThis is a great suggestion. We agree that validating the generalization capability of our framework on more diverse and complex charts is crucial. To address this and further test the robustness of our method, we commit to adding citation and experimental results on the ChartX dataset in the revised version of our paper. We will place the supplementary experimental results table at the end of this paragraph.\n\n| Model | SE (strict) | SE (slight) | SE (high) | QA |\n| :--- | :--- | :--- | :--- | :--- |\n| **ChartSketcher** | 35.65 | 41.74 | 47.06 | 56.94 |\n| **ChartVLM** | 23.18 | 30.68 | 38.30 | 43.84 |\n\nRegarding more advanced types such as 3D or interactive charts, we believe this would require dedicated extensions to our programmatic sketching library and represents a valuable direction for future research. We will discuss this in depth in the Appendix of our paper.\n\n## weakness3:\n## reply:\nThank you for your valuable suggestion. We will cite and discuss in detail the important chart understanding baselines you mentioned in both the \"Related Work\" section and the appendix of our revised paper.\nRegarding a direct performance comparison, we would like to clarify one point: since the aforementioned models are chart expert models, we will discuss and compare their performance specifically on chart-related tasks in the appendix. We will not evaluate their general-purpose VQA capabilities, as this would constitute an unfair comparison for these baselines, which were not designed for such tasks.\n\nTo provide an intuitive reference directly in this response, here is a comparison of ChartSketcher with these works on a representative chart dataset:\n\n| Model | ChartQA (aug) | ChartQA (hum) |\n| :--- | :--- | :--- |\n| **ChartSketcher** | **92.64** | **85.20** |\n| OneChart | 85.30 | 49.10 |\n| ChartAst | 93.90 | 65.90 |\n\nA complete discussion and detailed results on more datasets will be added to the appendix of the paper. We believe this approach satisfies the need for rigor while providing readers with the clearest and fairest performance reference."}, {"review_id": "C3X5h1FUsH", "reviewer": "Reviewer_hekG", "summary": "This paper focuses on multi-step chart visual understanding. They propose ChartSketcher, a multimodal feedback-driven step-by-step reasoning method. ChartSketcher is a chart understanding model that employs Sketch-CoT, enabling MLLMs to annotate intermediate reasoning steps directly onto charts using a programmatic sketching library, iteratively feeding these visual annotations back into the reason14 ing process. This mechanism enables the model to visually ground its reasoning and refine its understanding over multiple steps. They employ a two-stage training strategy: a cold start phase to learn sketch-based reasoning patterns, followed by off-policy reinforcement learning to enhance reflection and generalization. Experiments demonstrate that ChartSketcher achieves promising performance.", "questions": "1. Could you add citation and comparison to the papers: [ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding] from ICML 2025 and [Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models] from Neurips 2024?\n2. What's the training data exact size for each different source at each stage?\n3. What's the performance without using Visual-CoT at all?\n3. What's the performance with only using Visual-CoT?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Provides a huge high-quality chart reasoning dataset and could be helpful.\n2. Shows efficacy of RL on chart reasoning problems.\n\nWeakness:\n1. Limited Novelty. There is a similar paper: [ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding] from ICML 2025. The idea seems same, and the experiment result of ReFocus (on Phi3.5vision) with ChartSketcher-2B seems similar. Please add comparison to it. \n2. Lack Citation: [Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models] from Neurips 2024. Also lack comparison to it.\n3. The experiment needs to done more clearly, especially if you want to draw the conclusion that ChartSketcher proposes a chart reasoning dataset that helps chart and also helps general VQA tasks. In this case, First, the training data exact size for each different source at each stage should be listed. Second, since the Visual-CoT data was also used, it's hard to say that the model improves on general VQA tasks from Chart data. The paper should do the experiment with and without using Visual-CoT data to make more precise claims. Right now, if the proposed dataset is the biggest contribution, but the experiments are done with Visual-Cot, it's hard to tell whether the experiment conclusions can hold. \n4. Continue from 3, the paper should also train a version with only Visual-CoT to compare with on general VQA tasks as a baseline.", "was_revised": true, "final_justification": "The rebuttal has answered all my questions and added the required experiments / details. I have increased my score correspondingly.", "rebuttal": "## overall\nThank you very much for your insightful and critical feedback on our paper. Your points regarding comparison with recent related work and clarification of the experimental data composition are valuable for improving the rigor and clarity of our paper. We will add citations to the works you mentioned and improve our work in the revised version. In the following response, we will elaborate on the uniqueness of our method and explain and clarify the data details and ablation studies you requested.\n\n## weakness1,2 & question1:\nThank you for pointing out these two valuable related works. We will add detailed citations and discussions in the revised version.\n\nTo provide a direct comparison between Refocus and ChartSketcher, we present the performance comparison between ChartSketcher-2B and Refocus-phi3.5 on the Refocus test set. It can be noted that ChartSketcher's base model (Qwen2VL-2B) is significantly weaker than the base model of Refocus, phi-3.5 (4.2B). Therefore, to ensure a fair comparison, we have retrained a version of ChartSketcher based on phi-3.5 (which has now been open-sourced). \n\n| Model            | chartqa-vbar | chartqa-hbar |\n|----|--------------|--------------|\n| Qwen2VL-2B       | 51.31        | 48.65        |\n| ChartSketcher-2B | 59.69        | 54.73        |\n| ChartSketcher-phi3.5-4.2B | 74.35      | 72.30        |\n| phi3.5-4.2B           | 63.1         | 60.1         |\n| Refocus-phi3.5-4.2B   | 72.2         | 71.0         |\n\nThe comparative experiments are also shown in the table above. This demonstrates the effectiveness and competitiveness of our method. All experiments were run on four 8 x A800 (40GB) servers.\n\n### **Regarding the comparison to Sketchpad and Refocus**, we would like to elaborate on how our work builds upon and diverges from these approaches.\n\nFirst, ChartSketcher differs from the works you mentioned in its domain. ChartSketcher primarily focuses on charts—such as bar, line, and pie charts—and our solution can also be applied to natural images or structured images, like documents. The two works you mentioned, such as Refocus, mainly focus on structured data like tables and charts(only include bar chart in train/test set). \n\nSecond, ChartSketcher differs from the works you mentioned in its methodology. Although these works all explore the common theme of \"reasoning on images,\" our approach to achieving this goal is entirely new. Our method is centered on the inherent grounding and reflection capabilities of Multimodal Large Language Models, and we have built a novel core mechanism and learning method to support it.\n\nThe main differences can be summarized below:\n\n### **1.Comparison with ReFocus:**\nThe primary distinction of ChartSketcher is that it leverages the end-to-end visual grounding capability of MLLM itself to draw sketches, and it also provides image feedback and reflection capabilities, offering integration and interactivity that are not present in the other two works. Refocus are semi-\"training-free\" solutions that rely on external tools for grounding and do not possess reflection capabilities. The main differences can be summarized in the following four aspects:\n\n1.1 **Difference in Idea**: ChartSketcher utilizes its own learned grounding ability to externalize its reasoning process through \"addition\" (drawing sketches), whereas ReFocus relies on external tools to acquire coordinates and simplifies the input through \"subtraction\" (masking information).\n\n1.2 **Difference in Capability**: ChartSketcher's own grounding and reasoning capabilities enable it to \"inspect, reflect, and correct\" its own drawings, forming a closed-loop for error correction. In contrast, ReFocus's reliance on external tools makes its process linear, focusing on preventing errors, reducing information redundancy, and focusing on content.\n\n1.3 **Difference in Technical Route**: ChartSketcher builds the model's own end-to-end grounding capability from scratch through a two-stage training process to provide robust reasoning. In contrast, ReFocus uses prompting or fine-tuning to teach the model how to use an external tool to achieve clear reasoning.\n\n1.4 **Difference in Implementation Cost**: ChartSketcher constructs its reasoning data using locally deployed open-source models, while ReFocus uses proprietary APIs to build its data.\n\n### **2.Comparison with Visual Sketchpad:**\n\nOverall, ChartSketcher internalizes visual reasoning and self-correction capabilities as the model's own core skills through a specialized training pipeline, whereas Visual Sketchpad positions the model as a zero-shot planner whose capability is demonstrated in the effective orchestration of external specialist tools.\n\n2.1 **Difference in Idea:** ChartSketcher uses internal cognitive emulation - teaching models to generate primitive drawing commands for intrinsic visual reasoning and self-correction. Visual Sketchpad uses external tool orchestration - using MLLMs as planners to invoke specialist models.\n\n2.2 **Difference in Capability:** ChartSketcher's core capability is self-correction through training to inspect visual outputs and generate corrective actions. Visual Sketchpad excels at task delegation, decomposing problems and assigning them to appropriate external tools.\n\n2.3 **Difference in Technical Route:** ChartSketcher employs a novel two-stage training method with cross-modal distillation and RL. Visual Sketchpad uses zero-shot prompting without fine-tuning, leveraging pre-existing tool-use abilities.\n\n## question2:\n## reply:\nThank you for your suggestions; it will help us improve the transparency of our dataset. In line 74, we stated that 300k data points were used for cold start and 50k for RL, and in line 219, we mentioned using 20% VisualCoT data, but this was not precise enough. We will add a new section to the Appendix detailing the sources of our training data. Specifically, we used a total of 325,668 data points during the cold start phase and 50,455 data points during the RL phase. The specific breakdown of the datasets is fully listed below.\n\n| Training Phase | Method | Data Source                 | Data Type                 | Quantity         |\n| :-- | :----- | :----- | :----- | :- |\n| Cold Start     | SFT    | EvoChart Synthetic Chart Data | Correct Reasoning Path    | 155203 (87.3%)   |\n|                |        | VisualCoT and its Annotations | Correct Reasoning Path    | 22510 (12.7%)    |\n|                |        | Total                       |                           | 177713           |\n|                | DPO    | EvoChart Synthetic Chart Data | Reflection Reasoning Path | 147955           |\n| RL             | KTO    | ChartQA and ChartBench      | MCTS Sampled Paths        | 41196 (81.6%)    |\n|                |        | General QA-Pairs \\*          | MCTS Sampled Paths        | 9259 (18.4%)     |\n|                |        | Total                       |                           | 50455            |\n\n\\* 18.4% of the KTO training data was derived from general vision-language QA-pairs. These were sourced from datasets aggregated by VisualCoT (*TextVQA, TextCaps, DocVQA, DUDE, SROIE, CUB-200-2011, Flickr30k, Visual7W, InfographicsVQA, VSR, GQA, and OpenImages*). For these samples, we only used their image and QA-pair without adopting the original annotations from VisualCoT, which is effectively equivalent to using the datasets listed above. In the main text, this collection was abbreviated as 'VisualCoT' to save space, and we provide individual citations for each of these datasets in the appendix.\n\n## weakness 3,4 & question 3,4:\nThank you again for your in-depth analysis of our work and your valuable suggestions for revision.\n\nWhile the high-quality dataset we built is a valuable outcome of our research, the central scientific claim of our paper—as outlined in Lines 71-79—is exclusively about the **“ChartSketcher method”**. The effectiveness of each component of ChartSketcher has been proven through ablation experiments. Our experiments involving general VQA data were designed with a specific purpose: to test the hypothesis that this method generalizes across domains, **not to suggest** that the “chart dataset” itself “helps general VQA tasks”. We will revise the manuscript to make this distinction even more explicit.\n\nNevertheless, you raise a valuable point, as it helps to precisely clarify the role and contribution of our dataset. To strongly address your concerns and to more precisely quantify the generalization ability of our method, we have strictly followed your recommendations and completed these critical ablation and baseline experiments on a small-scale ChartSketcher-2B model.\n\n| method            | ChartQA-H | ChartQA-A | visCoT  |\n|-|-|-|-|\n| Qwen2vl 2B        | 50.48     | 75.84     | 58.33   |\n| ChartSketcher-2B  | 55.60     | 80.88     | 66.86   |\n| w/o VisCoT        | 56.72     | 81.76     | 56.10   |\n| w/o Chart Data    | 49.60     | 74.48     | 64.81   |\n\nThe results of these experiments clearly reveal the following points:\n\n1.\tWhen the model is trained only with our chart data, it still maintains strong performance on chart tasks. This demonstrates the effectiveness of our method and data for chart tasks.\n\n2.\tWhen the model is trained only with Visual-CoT data, due to the small amount of data, we observed that the Chart capability did not decrease significantly compared to Qwen2vl, and the VisualCoT capability showed a slight improvement.\n\n3.\tOur full model outperforms the aforementioned baseline model in general tasks. This strongly proves that the ChartSketcher method can be used for natural images. And it proves that there might be potential synergistic effects between data from different domains.\n\nIn Appendix F, Figure 9, we also demonstrate the intuitive results of applying the ChartSketcher method to natural images."}, {"review_id": "a43EWm8XaO", "reviewer": "Reviewer_z7tZ", "summary": "In this paper, the authors introduce ChartSketcher, a novel multimodal reasoning framework that enhances chart understanding by enabling large language models to iteratively annotate and refine their reasoning steps directly on chart images. Drawing inspiration from human sketching behavior, the authors propose a Sketch-CoT approach where the model generates visual annotations (e.g., arrows, lines) to ground its reasoning, then uses programmatic sketching libraries to refine these annotations based on visual feedback. A two-stage training strategy combines cross-modal knowledge distillation (cold start) with off-policy reinforcement learning (RL) optimized via a customized MCTS algorithm (sketch-MCTS). Experiments show strong performance on chart-specific benchmarks (e.g., ChartQA, PlotQA) and general vision tasks (e.g., OpenImages), demonstrating improved accuracy and interpretability compared to baseline models like GPT-4o and Qwen2VL. While the technical contributions are solid and the experimental setup is thorough, the evaluation could be strengthened by broader dataset coverage (e.g., more diverse chart types/visual tasks) and deeper analysis of failure modes.", "questions": "1. Experiments focus heavily on chart-specific benchmarks (ChartQA, PlotQA) but lack diversity in visual task types (e.g., infographics, mixed-modal charts, 3D plots). How do you justify this scope, and what steps will you take to validate ChartSketcher’s robustness across diverse visual domains?\n\n2. Ablation studies focus on component removal (e.g., \"w/o Rethink&RL\") but do not quantify the marginal benefit of key innovations like sketch-MCTS vs. naive RL. Can you redesign ablations to isolate the impact of iterative sketching/refinement vs. baseline MCTS?\n\n3. ChartSketcher struggles with charts requiring 3D perspective (e.g., financial candlesticks) or dynamic updates (e.g., live dashboards). What architectural changes would be needed to handle such cases, and how would your training strategy adapt?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "​​Strengths​​:\nFor strength, I would like to say that this paper presents a technically sound and innovative approach to chart understanding by introducing ChartSketcher, a multimodal reasoning framework that bridges human-like sketching behavior with programmatic visual feedback:\n(1) Sketch-CoT method enables iterative annotation and refinement of reasoning steps directly on chart images, addresses a critical gap in existing MLLMs that struggle with complex visual reasoning. \n(2) Two-stage training strategy (cold-start knowledge distillation + RL optimization via sketch-MCTS) is well-designed, and the experimental results demonstrate strong performance gains on both chart-specific benchmarks (e.g., ChartQA, PlotQA) and general vision tasks (e.g., OpenImages). \n(3) The work’s emphasis on interpretability and self-correction through visual grounding is particularly impactful, offering a practical pathway for improving automated data analysis tools.\n\n​​Weaknesses​​:\nContributions are significant, however, the evaluation has limitations that temper the paper’s impact:\n(1) Experiments prioritize chart-specific datasets (e.g., ChartQA) but lack diversity in visual task types (e.g., fewer tests on infographics or mixed-modal charts). \n(2) Additionally, the theoretical analysis of why iterative sketching improves reasoning—beyond empirical validation—is sparse.", "was_revised": false, "final_justification": "", "rebuttal": "## overall\nWe sincerely thank Reviewer for the insightful feedback on our work. We are encouraged that the reviewer recognizes the innovation of our ChartSketcher method, the well-designed two-stage training strategy, and the practical value of our work in enhancing model interpretability. We will address the identified weaknesses and questions below.\n\n## Weakness1:\n## reply:\n### 1: Thank you for your valuable feedback. We are pleased that you recognize the value of our work. Regarding the diversity of our evaluation, we would like to clarify that we did make a conscious effort to validate ChartSketcher's robustness on a range of visual tasks beyond charts.\n\nAs you can see in the appendix, our evaluation across 18 benchmarks already includes challenging datasets such as InfographicVQA and DocVQA. These datasets are characterized by complex, mixed-modal layouts with dense text and graphics, demanding a different and often more comprehensive reasoning capability than standard charts. Our strong performance on these benchmarks demonstrates that the core principles of ChartSketcher--iterative visual grounding and self-correction—can indeed generalize to more unstructured and diverse visual domains. To further illustrate the effectiveness of our method, we additionally tested the ChartX dataset, which we will add to the Appendix for further comparison.\n\n| Model | SE (strict) | SE (slight) | SE (high) | QA |\n| :--- | :--- | :--- | :--- | :--- |\n| **ChartSketcher** | 35.65 | 41.74 | 47.06 | 56.94 |\n| **ChartVLM** | 23.18 | 30.68 | 38.30 | 43.84 |\n\n### 2:Thank you for this excellent and thought-provoking point. Let us address your concerns regarding theoretical analysis.\n\nAs we replied to reviewer QsY7, theoretical analysis is indeed very important. Currently, many works[1][2][3] are inspired by human behavior. These works have empirically demonstrated effectiveness, but a theoretical analysis of human cognitive behavior is quite challenging. Our primary contribution in this work is ChartSketcher, which successfully operationalizes this human-like strategy and, in doing so, provides a highly effective and interactive reasoning experience. We see our work as the foundational step of validating that this approach works and works well in practice. We believe that a formal theoretical validation is necessary, but also a grand undertaking that could constitute a significant research program in its own right. Such an analysis might involve frameworks from information theory or formal methods.\n\nWhile a full theoretical treatment is beyond the scope of this paper, we can hypothesize about the underlying principles: \n\nCognitive Offloading: Each sketch acts as an external memory, offloading the burden of holding complex spatial relationships and intermediate conclusions in the model's \"working memory\". This allows for more focused, incremental processing. \n\nError Correction via Grounded Feedback: The visual feedback loop provides a powerful mechanism for error detection. By comparing its intended action with the actual outcome, the model gets a grounded signal to correct flawed internal states, which is much more direct than purely text-based reflection. \n\nWe are grateful you raised this point. In our revised manuscript, we will add a discussion in the conclusion to better frame our contribution in this context and explicitly highlight the need for this future theoretical work, suggesting it as an exciting and important path forward for the community.\n\n[1]Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models [NeurIPS 2024]\n\n[2]Enhancing LLM Reasoning via Vision-Augmented Prompting [NeurIPS 2024]\n\n[3]Goal Reduction with Loop-Removal Accelerates RL and Models Human Brain Activity in Goal-Directed Learning [NeurIPS 2024]\n\n## question1:\n## reply:\nThank you for this insightful question about the scope of our work. \n\nTo clarify our evaluation scope, we did test ChartSketcher on 18 diverse benchmarks, which do include complex mixed-modal datasets like InfographicVQA. However, it is correct to point out that our evaluation has its limitations. We have not explicitly tested on 3D plots or dynamic video-based charts. We acknowledge this as a limitation of our current work's scope.\n\nWe can demonstrate a clear and feasible path for ChartSketcher to address these domains, grounded in its core principles:\n\n1.For any 2D visual representation, including mixed-modal charts and 2D projections of 3D plots: The fundamental act of sketching to deconstruct a problem remains viable. This mirrors the intuitive, tool-agnostic way humans approach complex visual information. For a 3D plot, for instance, our model could sketch on its 2D projection to highlight points, axes, or surfaces. The primary work would involve extending our programmatic library with primitives that are aware of the 3D context.\n\n2.For dynamic content, videos: We can adapt our method by sampling keyframes. By treating a keyframe as a static image, Sketch-CoT can be directly applied to analyze a specific moment in time. This approach could even be extended to reason about changes by applying sketches to consecutive frames and prompting the model to compare them.\n\nIn essence, while our current work focused on establishing a strong foundation on a wide array of static benchmarks, the ChartSketcher framework is inherently extensible due to its mimicry of a fundamental human problem-solving process. \n\n## question2:\n## reply:\nThank you for your suggestion. We completely agree with your advice regarding our ablation studies. Comparing Sketch-MCTS with a general naive sampling method is necessary to demonstrate the effectiveness of Sketch-MCTS. \n\nTherefore, we have added an ablation experiment to prove the effectiveness of Sketch-MCTS. We used a simple rejection sampling method to obtain data, embodying the characteristics of \"naive RL.\" The experimental results are as follows. It can be seen that, compared to Sketch-MCTS, the simple rejection sampling method does not perform well. This is reasonable because simple rejection sampling algorithms cannot constrain the reasoning process within a concise and efficient range, leading to excessively lengthy reasoning chains.\n\n| method            | ChartQA-H | ChartQA-A | visCoT  |\n|-|-|-|-|\n| ChartSketcher-2B  | 55.60     | 80.88     | 66.86   |\n| w/ naive RL        | 52.08      | 78.24      | 62.12   |\n\n## question3:\n## reply:\n**1. Handling 3D Perspective Charts**\n\nRegarding 3D views, we concur with the implicit premise of your question: current MLLMs generally struggle with true 3D spatial understanding and depth perception. In practice, 3D information is almost always presented and processed as a 2D projection on a screen.\n\nGiven this reality, our Sketch-CoT methodology remains directly applicable and highly relevant. The model would operate on the 2D flattened image of the 3D chart, using sketches to deconstruct the visual information presented within that projection. The core challenge is not about rendering in 3D, but about teaching the model to correctly interpret the visual cues of depth and perspective present in the 2D image.\n\n**2. Handling Interactive Info (Live Dashboards)**\n\nFor interactive elements like dashboards, we believe the solution lies beyond simple sketching and requires evolving the model's action space. We envision extending our framework by incorporating key \"point-and-click\" and \"drag-and-drop\" operations.\n\nThis evolution would position the resulting model as a powerful hybrid of a Sketching Reasoner and a GUI Agent. It would not only analyze static visual content through sketches but also actively interact with interface elements to reveal new information or trigger updates. For example, it could \"click\" on a dropdown menu to filter data or \"drag\" a time-slider to see how a chart evolves, then apply Sketch-CoT to the resulting view. This represents a fascinating and powerful fusion of passive reasoning and active interaction.\n\n**3. Adaptation of the Training Strategy**\n\nTo train a model capable of such complex, multi-step interactions, a shift in our training strategy would indeed be necessary.\n\nWe agree that moving from our current off-policy RL to an on-policy RL approach (like GRPO) would be more suitable. On-policy methods are better at exploring and learning in environments where the consequences of an action are complex and path-dependent.\n\nFurthermore, we would need to implement a step-level reward mechanism. Instead of only rewarding the final correct answer, we would provide intermediate rewards for successfully completing sub-tasks, for example, a positive reward for correctly clicking a filter button that reveals the target data. This dense reward signal is crucial for guiding the model through the long and complex action sequences required for effective Sketch navigation and analysis."}, {"review_id": "Riz0oOB9n2", "reviewer": "Reviewer_6CLL", "summary": "This paper introduces ChartSketcher, a multimodal feedback-driven stepwise reasoning method. Through Sketch-CoT, the model annotates intermediate reasoning steps on charts using a programmatic plotting library, and iteratively feeds visual annotations back into the reasoning process to achieve visual anchoring and progressive optimization. The model employs a two-phase training strategy of \"cold-start learning + offline reinforcement learning.\" Experiments show that it excels in chart comprehension benchmarks and general visual tasks, providing an interactive and interpretable solution for chart understanding.", "questions": "1. How reliant is the proposed method on the model's coordinate localization ability? If the baseline's localization or coordinate output is weak, will the method still work?\n2. What does the reasoning process look like for cases with 6-7 steps? Could the model response fall into a loop?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Pros:**\n\n1. The motivation is sound, using tools to mimic human sketch-based understanding is an interesting direction in VLM RL.\n2. The authors provide detailed descriptions of implementation, including prompts, commands, and algorithms.\n3. Experiments show that ChartSketcher performs exceptionally well.\n\n**Cons:**\n\n1. The core idea is unclear (Line 124-129). Reviewer is hard to intuitively understand the intermediate steps of reasoning. How visualisations interact with the model during cold-start and RL training? Are only instructions command code used, or do visual tokens get filtered and re-encoded via ViT for the token sequence?\n2. There is no analysis of inference speed or generative token length. Will the model's maximum context length limit the rethink mechanism?\n3. There is a lack of training data visualization, raising concerns about data quality and potential homogeneity.", "was_revised": true, "final_justification": "Thanks to the authors for their response, which addressed most of my concerns. I would like to give a positive score if the authors can improve their paper writing about the SFT and RL workflow.", "rebuttal": "## overall\nWe would like to express our gratitude for your thorough and insightful review. Your feedback has been invaluable for improving our explanations and clarifying our contributions. We have carefully considered each of your points and appreciate the opportunity to provide a more detailed clarification of our work.\n\n## cons1:\n## reply:\nWe apologize if our description was not clear enough. \n\nTo be explicit, first, at every reasoning step, the generated sketch visualization is fully re-encoded by the model's ViT.\n\nChartSketcher's interaction is a multi-turn, multi-modal dialogue. For a single reasoning step, the precise data flow is as follows:\n\n* **1.\tInput (Turn $t$):** The MLLM receives the current image $Image_t$ and text prompt $Text_t$ (which includes the dialogue history).\n* **2.\tGeneration:** The model's language decoder generates a response containing reasoning text and a `<code>` block with sketch instructions.\n* **3.\tRendering:** Our system's backend renderer receives $Image_t$ and the sketch instructions from `<code>`, and generates a new image, $Image_{t+1}$, with the corresponding sketch already drawn on it.\n* **4.\tFeedback ($Turn  t+1$):** This new image, $Image_{t+1}$, will become the visual input for the next turn. It is fed into the model's ViT to generate new visual tokens. These visual tokens, along with the updated text prompt, will form the input for the next reasoning step.\n\nWhen the model believes the reasoning is complete, it no longer outputs `<code>`, and thus the reasoning loop ends. This ultimately forms a sequence of `<image><query> <think1><code> <feedback image 1> <think2><code> <feedback image 2> ...... <final think>`.\n\nTherefore, the model is not merely \"reading\" the code it generated. It is visually perceiving the consequences of its actions by processing the newly rendered image at each step. We will revise Section 3.1 of the paper to highlight this iterative \"image-ViT-feedback\" loop more clearly.\n\n## cons2:\n## reply:\nThis is a very practical and valuable question. Similar to reasoning models like DeepSeek-R1, the maximum context length indeed limits the number of reflections the model can perform, and we have made trade-offs in our practical engineering implementation.\n\n**Inference Speed:** Each step in the Sketch-CoT chain requires reading the feedback image, which makes the inference speed slightly slower compared to general reasoning models. On average, across all datasets involved in testing, our inference speed per query is 75% of QvQ's. The specific inference throughput is shown in the table below. All experiments were conducted with vLLM on a machine with 8x A800 40G GPUs. \n\nThis table clearly shows the model's throughput and also reveals interesting results. ChartSketcher, due to the need to check whether previous reasoning steps were correct, requires higher performance for input token prefill during inference. QvQ primarily focuses on text reasoning, thus requiring higher decode throughput. Qwen2VL-72B, without \"Thinking,\" has significantly shorter output tokens than input tokens, so its input and output performance are relatively balanced, with the main bottleneck being input prefill.\n\n| Model | Inference Speed (Input) Average token/s | Inference Speed (Output) Average token/s | Overall Inference Speed (Average across all test sets) Normalized |\n|-|-|-|-|\n| ChartSketcher-72B | 2995 | 329 | 1 |\n| QvQ-72B | 570 | 1705 | 1.334 |\n| Qwen2VL-72B | 1926 | 685 | 9.107 |\n\n**Context Length Limit:** Yes, the model's maximum context length is indeed a hard constraint on the reflection mechanism. The entire dialogue history, including all text and <code> blocks, accumulates with each turn. For models with limited context windows, overly long reasoning chains can indeed lead to content truncation. To manage these limitations, we have implemented two key safeguards, which we will emphasize more clearly in the paper:\n\n* **1.Maximum Dialogue Depth:** As described in Appendix F, we set an upper limit for the maximum length of the reasoning chain (typically 8-12 steps) to prevent infinite loops and manage context length.\n\n* **2.Efficient Search via Sketch-MCTS:** Our Sketch-MCTS algorithm is designed to find the correct solution more efficiently than naive exploration. By pruning redundant and unpromising paths, it helps keep the reasoning chain concise, thereby saving both time and context window space. This RL strategy implicitly saves token count during the inference time.\n\n## cons3:\n## reply:\nWe agree that visualizing training data is crucial for the transparency of our work. To address this, we will add a new dedicated section in the appendix for visualizing our synthetic data.\n\nHere, we detail the scale of the data we used. All data has been rigorously checked and will be open-sourced to reduce the potential for data homogeneity. As you can see, we used synthetic chart data along with VisualCoT and its annotations in the first ColdStart phase (without reflection). In the second ColdStart phase (with reflection), we exclusively used synthetic chart data. For the RL phase, we utilized real-world Chart datasets and VisualCoT's QA pairs. All data will be publicly accessed and open-sourced.\n\n| Training Phase | Method | Data Source                 | Data Type                 | Quantity         |\n| :- | :- | :- | :- | :- |\n| Cold Start     | SFT    | EvoChart Synthetic Chart Data | Correct Reasoning Path    | 155203 (87.3%)   |\n|                |        | VisualCoT and its Annotations | Correct Reasoning Path    | 22510 (12.7%)    |\n|                |        | Total                       |                           | 177713           |\n|                | DPO    | EvoChart Synthetic Chart Data | Reflection Reasoning Path | 147955           |\n| RL             | KTO    | ChartQA and ChartBench      | MCTS Sampled Paths        | 41196 (81.6%)    |\n|                |        | General QA-Pairs *          | MCTS Sampled Paths        | 9259 (18.4%)     |\n|                |        | Total                       |                           | 50455            |\n\n\n\\*The datasets for the RL phase include TextVQA, TextCaps, DocVQA, DUDE, SROIE, CUB-200-2011, Flickr30k, Visual7W, InfographicsVQA, VSR, GQA, and OpenImages. The term \"VisualCoT dataset\" is, in fact, an aggregated name for the aforementioned datasets.\n\n## question1:\n## reply:\nThank you for your question regarding the robustness of our work. While the strong initial localization capability of the base model is undoubtedly beneficial, ChartSketcher is specifically designed to be robust to, and even correct, its own localization errors.\n\nHere's why:\n\n* If the model's initial localization capability is weak, it might generate a sketch with incorrect coordinates.\n* However, in the very next step, the model will \"see\" its own mistake, because that incorrectly rendered sketch now becomes part of its visual input.\n* During the reflective training phase, the model learns to identify this inconsistency between its textual reasoning and the visual evidence. It learns to generate corrective dialogue.\n* This self-correction loop allows the model to iteratively refine its understanding and eventually converge to the correct answer.\n\nTherefore, our method does not solely rely on the base model's capabilities; it actively enhances and corrects them. In fact, the 2B version of ChartSketcher, trained on the Qwen2VL-2B which has weaker localization capabilities than 72B version, also achieved significant improvements. To further illustrate the correlation between the model's capabilities and our method, we have also trained a medium-sized 4.2B version of the model, which will be open-sourced soon and its performance added to the article's Appendix.\n\n## question2:\n## reply:\nThis question is similar to cons 2, and it's a very practical one. \n\nFirst, allow me to answer: a 6-7 step reasoning process typically occurs for complex problems that require 2-3 sub-steps and may involve self-correction. An example might be as follows:\n\nStep 1: \"First, I need to understand what the blue and orange lines represent. I'll highlight the legend.\" (Draws a box around the legend).\n\nStep 2: \"Okay, blue is 'Good' and orange is 'Bad'. Now I need to find the value for 'Good' in 2015.\" \"2015 is between 2014 and 2016. I will mark this position on the x-axis.\" (Draws a point on the x-axis).\n\nStep 3: \"Now I will draw a vertical line until it intersects the blue line.\" (Draws a vertical line).\n\nStep 4 (Self-correction): \"Wait, I accidentally drew the line to the orange line. Let me erase it and redraw it to the blue line.\" (Deletes the old line, draws a new correct line).\n\nStep 5: \"Now I'll draw a horizontal line from this point to the y-axis to read the value.\" (Draws a horizontal line).\n\nStep 6: \"This line intersects the y-axis at approximately 57. The answer is 57.\" (No Sketch, reasoning ends).\n\nSecond, we will address your concern about infinite loops.\n\nThis question is very insightful for Reasoning Models. Honestly, for all existing Reasoning Models, infinite loop reasoning is possible, which can lead to uncontrolled context. Our model is no exception, but we have designed a series of steps to prevent such occurrences.\n\nAdditionally, we have implemented two mechanisms to prevent this from happening:\n\n**Maximum Depth Limit:** We enforce a hard upper limit on the number of reasoning steps.\n\n**MCTS - Shaping:** The reinforcement learning process implicitly discourages overly long or circular reasoning, as these paths are less likely to efficiently reach the correct answer and thus receive lower cumulative rewards.\n\n## Paper Formatting Concerns:\nWe apologize for the confusion we caused. The detailed algorithmic workflow of our work is presented in Appendix D. We will revise the main text to provide a more explicit and formalized description of the ChartSketcher workflow."}]

Author rebuttal:

Meta-review: This paper introduces a method to encourage multimodal large language models to perform feedback-driven, step-by-step reasoning. The main idea is a programmatic sketching interface (Sketch-CoT), which allows models to annotate intermediate reasoning steps directly onto charts using a programmatic sketching library. The authors also design a two-stage training procedure: a cold-start phase and RL tuning. Experimental results show promising improvements.

Some reviewers raised concerns about novelty and requested further comparison to existing work, specifically “ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding” (ICML 2025). The authors provided additional results and explanations during the rebuttal, and the reviewers seemed satisfied. There were also minor concerns about implementation details and conceptual clarifications, which the authors addressed adequately.

Overall, there are no critical drawbacks to this paper. However, while the reviewers appreciated the contributions, they did not find them particularly impressive. The AC considers this a borderline paper, leaning slightly toward acceptance.
Decision: Accept (poster)