BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning
Authors: Ahmed Masry
Abhay Puri
Masoud Hashemi
Juan A. Rodriguez
Megh Thakkar
Khyati Mahajan
Vikas Yadav
Sathwik Tejaswi Madhusudhan
Alexandre Piché
Dzmitry Bahdanau
Christopher Pal
David Vazquez
Enamul Hoque
Perouz Taslakian
Sai Rajeswar
Spandana Gella
Venue: COLM 2025 (2025)
Keywords: charts
chartqa
vision language models
multimodal
TL;DR: We propose BigCharts, a diverse chart dataset, and a training framework that improves VLMs’ chart comprehension.

Abstract: Charts are essential to data analysis, transforming raw data into clear visual representations that support human decision-making. 
Although current vision-language models (VLMs) have made significant progress, they continue to struggle with chart comprehension due to training on datasets that lack diversity and real-world authenticity, or on automatically extracted underlying data tables of charts, which can contain numerous estimation errors. Furthermore, existing models only rely on supervised fine-tuning using these low-quality datasets, severely limiting their effectiveness. To address these issues, we first propose BigCharts, a dataset creation pipeline that generates visually diverse chart images by conditioning the rendering process on real-world charts sourced from multiple online platforms. 
Unlike purely synthetic datasets, BigCharts incorporates real-world data, ensuring authenticity and visual diversity, while still retaining accurate underlying data due to our proposed replotting process. Additionally, we introduce a comprehensive training framework that integrates supervised fine-tuning with Group Relative Policy Optimization (GRPO)-based reinforcement learning.  By introducing novel reward signals specifically designed for chart reasoning, our approach enhances model robustness and generalization across diverse chart styles and domains, resulting in a state-of-the-art chart reasoning model, BigCharts-R1. Extensive experiments demonstrate that our models surpass existing methods on multiple chart question-answering benchmarks compared to even larger open-source and closed-source models.

Published as a conference paper at COLM 2025
BIGCHARTS-RI: Enhanced Chart Reasoning with Visual Reinforcement Finetuning
Ahmed Masry1,2 Abhay Puri', Masoud Hashemi', Juan A. Rodriguez Megh Thakkar', Khyati Mahajan', Vikas Yadav', Sathwik Tejaswi Madhusudhan', Alexandre Piché", Dzmitry Bahdanau 1.77, Christopher Pa]1.3677 David Vazquez! 134,
Enamul Hoque", Perouz Taslakian', Sai Rajeswar 1,3,, Spandana Gella'
"ServiceNow Research York University Mila Universstéé de Montréal
SÉTS Montréal "Polytechnique Montréal
"CIFAR AI Chair
Abstract
Charts are essential data analysis, transforming raw data into clear vi- sual representations that support human decision-making: Although cur- rent vision-language models (VLMs) have made significant progress, they continue to struggle with chart comprehension due to training datasets that lack diversity and real-world authenticity, or on automatically ex- tracted underlying data tables of charts, which contain numerous es- timation errors. Furthermore, existing models only rely on supervised fine-tuning using these low-quality datasets, severely limiting their effec- tiveness. To address these issues, we first propose BIGCHARTS, a dataset creation pipeline that generates visually diverse chart images by condi- tioning the rendering process on real-world charts sourced from multiple online platforms. Unlike purely synthetic datasets, BIGCHARTS incorpo- rates real-world data, ensuring authenticity and visual diversity, while still retaining accurate underlying data due to our proposed replotting pro- cess. Additionally, we introduce comprehensive training framework that integrates supervised fine-tuning with Group Relative Policy Optimiza- tion (GRPO)-based reinforcement learning. By introducing novel reward signals specifically designed for chart reasoning, our approach enhances model robustness and generalization across diverse chart styles and do- mains, resulting in state-of-the-art chart reasoning model, BIGCHARTS- R1. Extensive experiments demonstrate that our models surpass existing methods on multiple chart question-answering benchmarks compared to even larger open-source and closed-source models.
1 Introduction
Charts are essential tools for transforming raw data into clear and visually intuitive for- mats, significantly aiding data analysis, interpretation, and decision-making across nu- merous domains such as scientific publications, business reporting, and media presenta- tions (Kim al., 2020). Given their ubiquity, there have been increasing efforts into devel- oping models capable of understanding and reasoning about charts (Hoque et al., 2022).
Existing methods aimed at training these vision-language models (VLMs) capable of chart comprehension predominantly use supervised finetuning (SFT) by utilizing synthetic datasets generated by teacher models, primarily due to the high cost of manual annota- tion (Liu et al., 2024b; Meng et al., 2024; Zhang et al., 2024; Masry et al., 2023). These methods typically follow a two-step process: (i) generate synthetic chart images, and (ii) generate corresponding question-answer (QA) pairs from language models such as Gem- ini (Georgiev et al., 2024) or GPT4 (OpenAl et al., 2024). Despite their practicality, such
"Correpondance ahmed Aameddmasy@seerieeno..cm and http::/www...oo.a.aaaaaaeeeeeee.

---

Published as a conference paper at COLM 2025
Dafsset Ceneration
Chart Image Collection
Chart Replotting
Chart QA Generation
Extraction
Datasets
Search
Crant PDFs
cnarthar-ohhat
Collected Charts
Reproduced Code
Repletted Charis
Repelatted
GA para with Theughtt
colection
Visual Instruction Tuning (SFT)
Reinforcement Learning (GRPO)
Model Training
Symthesic
Input Image Question
Limited Humen Verifled Data
Image- Question
Ventien/Reaard
Gemini
Supervised Finetuning
Reinforcement Learning
Chart Errer Hake Rowers
Unventfee
Aluundant
Oulput Col
Answer
Limited
Output Answer
Response
Figure 1: BIGCHARTS construction and BIGCHARTS-RI training pipeline. We begin by extracting high-quality corpus from open chart datasets, Google Search, and Common Craw1($3.1.1), We then generate and execute the code responsible for producing these charts to replot them ($3.1.2), and then derive question-answer pairs with chain-of-thought reasoning ($3.1.3). For training BIGCHARTS-RI, we use a two-stage approach: (i) visual instruction tuning via SFT on large-scale synthetic data($4.1), and (b) RL (GRPO) with verifiable rewards and human-labeled data to enhance chart reasoning (64.2).
methods have significant drawbacks. First, synthetic charts created purely with plotting tools (e.g. matplotlib) using a narrow set of controlled parameters are visually homoge- neous and fail to replicate the rich diversity observed in real-world charts. Second, syn- thetic QA pairs generated from plotting scripts often accurately reflect underlying data but neglect crucial visual details of the chart images (see Figure 2b) (Liu et al., 2024b; Han et al., 2023; Meng et al., 2024; Deitke et al., 2024; Yang et al., 2025). This can lead to models over- fitting to certain types of charts and chart components, affecting their generalization and effectiveness when encountering visually different charts (Wang et al., 2024).
An alternative line of research has sought to leverage chart images directly crawled from real-world sources to capture visual variety (Masry et al., 2024a;b). However, these images typically lack access to accurate underlying data (e.g., plotting code), necessitating reliance on external models estimate data values directly from the chart images. This leads to substantial inaccuracies and estimation errors (Wang et al., 2024) (see Figure 2a) in the dataset, subsequently affecting the model training and downstream performance.
To summarize, these iinitations-visull homogeneity of the dataset, estimation errors in underlying chart data, and reliance on simple SFT prone to hinder model performance on in-domain tasks and impair generalization to out-of- distribution scenarios for chart understanding and reasoning (Kumar et al., 2025; Trung et al., 2024). To overcome these limitations, we present two novel contributions: a pipeline to curate a diverse and accurate dataset for training chart reasoning models, and a training strategy utilizing the curated dataset. Firstly, we introduce BIGCHARTS, novel dataset and pipeline explicitly designed for enhanced chart reasoning as shown in Figure 1. Our innovative method combines the strengths of real-world and synthetic datasets by col- lecting visually diverse real-world charts and conditionally replotting them using VLMs. While initial VLM--geneattd code for the charts may have inaccuracies, we mitigate this by re-rendering these codes into new, improved chart images and discarding the original charts, ensuring visual authenticity coupled with accurate underlying data. Furthermore, in the subsequent QA generation stage, we provide both the rendered charts and their as- sociated data in the form of code, enabling VLMs to create logically consistent QA pairs accurately reflecting both visual and numerical characteristics (Figure 2c).
We also propose a robust training strategy that integrates SFT with reinforcement learn- ing (RL), leveraging Group Relative Policy Optimization (GRPO) (Shao et al., 2024) as it has proven effective in improving reasoning in textual contexts. We introduce novel reward functions specifically designed to encourage models to leverage visual chart prop- erties for accurate numerical estimation and mathematical and visual reasoning. Our ex- tensive experiments demonstrate that the models trained with BIGCHARTS and method,
2

---

Published as a conference paper at COLM 2025
BIGCHARTS-RI, significantly surpass existing state-of-the-art approaches across multiple chart reasoning benchmarks and exhibit better generalization to -ut-of-distribution tasks.
Our main contributions are: (1) BIGCHARTS: A visually diverse, high-quality chart train- ing dataset possessing precise ground-truth data alongside realistic visual variety; (2) BIGCHARTS-R1: Advanced chart reasoning models that combine SFT and GRPO using novel chart-specific rewards, achieving state-of-the-ars performance and robust general- ization in both 3B and 7B model categories; and (3) Extensive empirical evaluations con- firming the effectiveness and robust generalization advantages of our dataset and training methodology. We release our code and artifacts at bigcharts.githu..io
2 Related Work
Instruction-tuning for Chart Understanding. There have been development of models exclusively for chart understanding (Masry et al., 2023; Liu et al., 2023; Zhou et al., 2023; Masry et al., 2025), well as models that are obtained by finetuning existing VLMs on chart understanding datasets (Masry et al., 2024a; Han et al., 2023; Meng et al., 2024; Masry et al., 2024b; Rodriguez et al., 2025). In the second set of works, datasets used to fine- tune VLMs such as ChartLlama (Han et al., 2023), ChartAssistant (Meng et al., 2024), and ChartInstruct (Masry et al., 2024a) rely on underlying datatables of the charts either ob- tained from the original dataset source or extracted from the charts, failing to capture the nuances of the chart images. Methods like ChartGemma (Masry et al., 2024b) utilize the chart images directly and use closed-source models to generate the datatable for curating instruction-tuning samples. This leads to data estimation errors introduced by the multi- modal models, impairing the quality of the training dataset. Our dataset addresses these critical concerns as consists of a large variety of charts that have faithful underlying datatables, enabling effective and accurate learning.
Post-training for Reasoning in LLMs and its extension to VLMs. Numerous recent post- training methods for LLMs have focused ties (Kumar et al., 2025). This is achieved by leveraging reasoning traces in the dataset and sophisticated extensions of RL or SFT methods. Initial RL-based approaches, such as REINFORCE (Williams, 1992), treated token generation as sequential decision-making but struggled with high variance. Newer methods like Proximal Policy Optimization (PPO; Schulman et al., 2017a) introduced clipped policy updates, significantly improving training stability for extended reasoning tasks. Further refinements include Direct Preference Opti- mization (DPO; Rafailov et al., 2024) and Group Relative Policy Optimization (GRPO; Shao et al., 2024), which directly optimize outputs human preferences through binary com- parisons or relative rankings, respectively, thereby enhancing reasoning coherence. In par- allel, of-huuhttt finetuning (Chung et al., 2022) explicitly trains models to generate intermediate reasoning steps, improving interpretability and logical consistency. Comple- menting these approaches, recent studies have also leveraged synthetic reasoning datasets to augment training sets, further improving generalization on reasoning- intennive tasks (Kim et al., 2023). These developments in LLMs have recently been adapted for finetun- ing VLMs (Liu et al., 2025a), but their effectiveness in tasks such as chart comprehension, which requires strong mathematical and visual reasoning abilities, is underexplored.
enhancing their multi-step reasoning abili-
Chart Benchmarks. A variety of tasks and benchmarks have been designed to assess VLMs' chart understanding capabilities, including question answering (Masry et al., 2022; Wang et al., 2024), chart summarization (Kantharaj et al., 2022b), fact-checking (Akhtar et al., 2023a;b), explanation and caption generation (Kantharaj et al., 2022a; Tang et al., 2023), and visualization recommendation systems (Hu et al., 2019). Early benchmarks like FigureQA (Kahou et al., 2018), DVQA (Kafle et al., 2018), Leaf-QA (Chaudhry et al., 2019), and PlotQA (Methani et al., 2020) were based on synthetically generated charts and templated questions, and had limited visual diversity. More recent benchmarks, such as ChartQA (Masry et al., 2022) and CharXiv (Wang et al., 2024) incorporate real-world charts and introduce more com complex questions, requiring advanced visual reasoning.
3

---

Published as a conference paper at COLM 2025
Generated Q/A on Chart Image Only
Generated Q/A Based on Underlying Data Only
Generated Based Chart Image & Data (Ours)
A
B
1YQ
What is the lotal Internationt student atrolment indiconed by the lefirnstt fight blue bar?
What international student
Whot is the sotar Insernational studert arrokment inticated by the lefimeat Aghht Mis hor?
acrolment 7V/22 academic year?
Approximately 500,000
764,08
764,486
Captures Visual Fealures?
Captures Visual Features?
Captures Visual Features? Accurate Values?
Accurate Data Values?
Accurata Values?
Figure 2: Different Chart Generation Approaches. Generating QA pairs automatically can be achieved with three approaches: using only images (image-only), leveraging raw data values from the charts (data-only), or combining both (our proposed approach). Our method integrates semantic and visual features while ensuring accuracy in underlying data values.
3 BIGCHARTS Dataset
3.1 Generation Pipeline
As illustrated in Figure 1, our BIGCHARIS pipeline for curating the dataset comprises of three stages: (1) Chart Collection, (2) Chart Re-Plotting, and (3) Question and CoT Generation.
3.1.1 Chart Collection
We initially collect 245,414 chart images from three diverse sources to ensure comprehen- sive visual and topical coverage:
Existing Datasets. We aggregate images from existing datasets ChartGeema (Masry et al., 2024b), FigureQA (Kahou et al., 2018), DVQA (Kafle et al., 2018), PlotQA (Methani et al., 2020), and ArXivQA (Li et al., 2024b), resulting in a total of 174,277 images. While these datasets form solid base, visual diversity varies significantly among them.
2. Common Crawl. To enhance visual diversity, we utilize the Mint-1T dataset (Awadalla et al., 2024), which includes extensive PDF files covering numerous topics from Com- mon Crawl!. We extract images from these documents and employ rigorous two-step filtering approach to obtain charts. We first train a high-recall ResNet-50 (He et al., 2015) binary classifier using labeled chart images (positive class) and natural images from CC- 12M (Changpinyo et al., 2021) and ImageNet (Russakovsky et al., 2015) (negative class). Then, we use this classifier to select potential chart images. In the second stage, we enhance the classifier by manually labeling 5,000 predicted charts from the first stage and retrain the classifier aiming for higher precision. Overall, this process yields 57, 17, filtered charts. Despite broad topical coverage, prevalent types remain bar, line, and pie charts. Detailed methodology used for filtering Mint-1T is available in Appendix A.2.
Google Search. To further diversify chart styles, such as heatmaps, scatter plots, dash- boards, and infographics, we conduct targeted Google searches using 200 curated key- words. We collect the top search results for each keyword, acquiring an additional 13,941 visually diverse charts. The full keyword list is provided in Appendix A.3.
3.1.2 Chart Re-Plotting
A critical challenge in chart reasoning datasets derived from real-world images is the ab- sence of underlying data (e.g., plotting code or datatables). To address this, we introduce
http:://oommoncrawlorg//
4

---

Published as a conference paper at COLM 2025
Chart Types
Chart Topics
Chart Sources
Bar
2,984 126.8 6%)
Finance
7,06
arXiv
Mint-TT
ChartGemma 73K (53.7%)
Google Search 12k (8.8%)
Mulb Bar
875 (78%)
Economy
(39.0%)
24k (177%)
27k (20.0%)
Stacked Bar
345 (3.1%)
Data Science
5,001
Line
2,585 (22 8%%)
STEM
5,385
Question Types
Answer Types
Visuel Featuree
745k (49.5%)
2459
Physics
13.5%
Not Answerable
208k (12.4)
Aera & 1ido,, Wile Lebel, Tick Diftrance
792 (711%)
Chemistry
8.7%
Simple Operations
273k (18.5%)
Answerable
1,46K (878%)
Biology
Naaverical Visual Numwrica Appregation
Numerical
51.7%
Chart
(2.0%)
Mathematies
2.8%
Counting
179k 111 9%%
Textual
20.4%
Line Bar
Donut Chart
188 (1.8%)
Time Series
880 (3.7%)
Reasoning
Data Retrieval
185c (11.0%)
(9:196)
Yas/No
8.4%
Others
84 (0.7%)
Others
(0.2%)
Nareeroot Vicual oal, Compiee Smaath stepp
Muttiple Cholce
Figure 3:
BIGCHSRTS Dataset Statistics. Type and topic distribution of the charts, chart sources, and type distribution of questions and answers.
novel chart re-plotting strategy that recovers both visual styles and underlying data. We uti- lize Gemini Flash 2.0 (Georgiev et al., 2024) to generate code replicating the design and con- tent of each chart image. Recognizing potential inaccuracies in data estimation by vision- language models (VLMs) (Wang et al., 2024), we subsequently render new chart images directly from this generated code, ensuring visual authenticity and data consistency.
We predominantly employ Python-based plotting libraries, matplotlib and plotly, for most collected charts. However, for more stylistically complex charts from ChartGemma and Google Search, we leverage the Chart.js² library which captures their intricate designs more accurately. Charts resulting in rendering failures due to erroneous code are excluded, ultimately resulting in refined set of 134,950 charts complete with accurate underlying data. Detailed prompts for code generation are described in Appendix A.4. We provide samples of original and replotted charts in Figure 19 in A.8.
3.1.3 Question and Chain-of-Thought (CoT) Generation
After curating our chart images, we generate corresponding questions enriched with de- tailed step-by-step reasoning. Unlike prior methods reliant solely on either underlying data (missing visual context) visual chart images alone (prone to data inaccuracies), our approach integrates both chart images and their accurate underlying data for question generation (see Figure 2). This combined approach enables VLMs to produce QA pairs that accurately reflect both visual and numerical chart characteristics, significantly reducing er- rors and improving overall dataset quality.
For each chart, we use Gemini Flash 2.0) to generate sixteen questions targeting direct data retrieval, visual analysis, and mathematical reasoning accompanied by compre- hensive chain-of-thought reasoning process leading to final answers. Detailed prompt in- structions and examples are provided in A.4 and A.6, respectively.
3.2 Dataset Analysis
Figure 3 provides statistics of the BIGCHARTS dataset, presenting the distribution of chart sources, chart topics, and chart question and answer types. We provide the prompts used to obtain our statistics in Appendix A.5.
Chart Types and Topics We use Gemini Flash 2.0 to classify the charts into various types and observe that the distribution demonstrates variety of visual representations. While bar charts, line charts, and multi-line charts are the most common, the dataset also includes specialized visualizations such scatter plots, stacked bar charts, and area charts. The "Other" category includes less frequent but important visual types like heatmaps, radar charts, waterfall charts, and violin plots.
Phttps:/wwww.caari..org
5

---

Published as a conference paper at COLM 2025
For the topic distribution, initially prompt Gemini Flash 2.0 using the chart images to obtain topics, and then use GPT-4o cluster them into 20 representative groups based on their similarity. Finance & economy (39%), STEM (29.7%), and Data Science (27.5%) consti- tute the largest portion of the dataset, encompassing a broad range of subtopics. Finance includes subtopics like GDP, stock market trends, and inflation. Data Science covers sta- tistical methods, machine learning techniques, and time series analysis, which are widely used across multiple domains for data-driven decision-making. STEM topics like Physics and Chemistry reflect the dataset's applicability to scientific research.
Questions and Answers in Dataset Samples Each dataset sample contains three ele- ments: 1) question, 2) answer, and 3) chain-of-thought. As shown in Figure 3, BIGCHARTS consists of a total of around 1.8 million questions across various question types. The major- ity of questions are about visual features, asking the title, information in specific positions, axes, labels, etc. Other question types include retrieving numerical data from the images, doing mathematical computations, and performing single-step or multi-step reasoning. BIGCHARTS also has unanswerable questions with "Not Appli sobblc""l"" as the ground truth to test a model's ss ability to abstain and recognize unanswerable questions. Answers also have different types: numerical, textual, multiple-choice options, and abstention for unanswer- able questions. Among the answerable questions, numeric answers are the most prevalent, accounting for over half of the dataset. This is followed by 342k textual answers and 141k Yes/No questions. Multiple-choice questions with options make up almost 118k samples.
Each question has generated chain-of-thoughh (CoT), which is used for model finetuning. These CoTs vary in length, with the longest containing 1,933 tokens and a median length of 39 tokens (and an average of 49.65 tokens), indicating that the majority of CoTs are relatively concise. We present the distribution of the length of the CoT in Figure 5.
4 Methodology
We employ the Qwen2.5-VL-Instruct (Bai et al., 2025) models with 3B and 7B parameters for our experiments, chosen due their state-of-the-art capabilities in vision-language un- derstanding tasks. Our training methodology comprises two primary stages: supervised finetuning (SFT), followed by reinforcement learning (RL) using Group Relative Policy Op- timization (GRPO). We present an overview of our training pipeline in Figure 1.
4.1 Supervised Finetuning (SFT)
We perform supervised finetuning using the high-quality reasoning chain-of-thought data from our BIGCHARTS dataset. This improves the model's ability to extract precise data values from chart images while encouraging detailed, step-by-step the model's overall capability in solving complex chart-based problems. We provide the prompts used for finetuning our models in Appendix A.4.
btep-by-stee reasoning, enhancing
4.2 Reinforcement Learning (RL)
To mitigate potential errors arising during the SFT phase where inaccuracies from teacher models can propagate and degrade model performance, we employ RL. This step is de- signed to rectify deviations resulting from imperfect training data, consequently enhanc- ing the model's chart understanding and reasoning capabilities. Our RL approach follows the Reinforcement Learning with Verifiable Rewards (RLVR) framework (Lambert et al., 2025), specifically utilizing Group Relative Policy Optimization (GRPO) (Shao et al., 2024) given its recently proven effectiveness in enhancing reasoning in LLMs.
Reinforcement Learning with Verifiable Rewards (RLVR) RLVR has recently been ap- plied for training large language models through reinforcement learning on tasks with verifiable outcomes, such mathematics and code generation (Zelikman et al., 2022; Liu et al., 2025b). Unlike traditional methods like Reinforcement Learning from Human Feed- back (RLHF) (Ouyang et al., 2022; Christiano et al., 2023), which rely learned reward

---

Published as a conference paper at COLM 2025
models, RLVR directly employs verifiable reward functions (R) that objectively assess out- comes. The RLVR training objective is defined as follows:
max max-Ey-mpl (yxxm((x) Th
[R(x,y) y {x - BKL
(1)
where R is the reward, 8 is scaling factor, and KL represents the KL divergence between the current model policy Ty and the reference model policy Tee--
Group Relative Policy Optimization (GRPO) A primary advantage of GRPO is its in- dependence on a separate value or critic model, unlike PPO (Schulman et al., 2017b). By doing so, GRPO substantially reduces memory usage and improves training efficiency by estimating values directly from sampled response groups.
Given an input query 9, GRPO samples a set of responses 01, 02,...,0G from the current policy model aa Each response is evaluated by our reward functions to yield individual rewards 22..,...., rC- The advantage A; for each response then computed as:
mean(ri, TT.
A;
(2)
Reward Function Formulation We propose two complementary reward signals: Chart Error Rate Reward (CERM) and Response Format Reward, which are combined to guide the reinforcement learning process.
The Chart Error Rale Reward (CERM) is smooth, error-based dense reward specifically designed for numeric answers. We first calculate an error rate (ER) based on the difference between the predicted value and the ground-truth value Then, we apply homographic function to map this error rate into continuous reward between (0,1), incentivizing the model to minimize numerical prediction errors. For non-numeric answers, we utilize straightforward exact-match criterion:
if both ý and y are numeric, if non-numeric and y, otherwise.
ER(g.y) = 19 yl lyl
+ ER(Ú,))
STERM(0.V)
1,
(3)
Additionally, inspired by Shen et al. (2025), we define a Response Format Reward (R to encourage adherence to the required answer format:
R = {
the model follows the required response structure, otherwise.
(4)
The final combined scalar reward is computed the sum of these two signals:
Rotal = RCERM + R
(5)
5 Experiments and Results
5.1 Experimental Setup
Training Setup. For our SFT experiments, we use LlamaFactory (Zheng et al., 2024). Both the 3B and 7B models are trained for one epoch on BIGCHARTS with SFT using a learning rate of 2e-5, batch size of 32, 0.1 warmup ratio, and a cosine scheduler.
For GRPO training, we train for one epoch with a batch size of 8, learning rate of 1e-6, and generate 8 candidate outputs per sample. All experiments are conducted on a single node with 8xH100-80GB GPUs. To construct our RL data, we utilize combination of high-quality human-labeled and verified datasets such as the ChartQA training split and 1K randomly sampled instances from the training subsets of the template-based datasets: PlotQA (vl & v2), DVQA, and FigureQA. Overall, our RL dataset contains 22,297 chart images and 32,297 questions with verifiable answers.
7

---

Published as a conference paper at COLM 2025
Model
FigureQA-Sub
DVQA-Sub
PlotQA-Sub
ChariQA
CharXiv
Avg
Valt
Va1z
ValF
ValH
ang
hum
avg
Reas.
Des.
Clased-Source Models
GPT-4o (OpenAl al, 2024)
65.70
69.10
57.50
61.20
59.50
19.90
85.07
50.50
82.58
61.22
Gemini-Flaze Flaniii-- (Georgiev al. 2024)
56.90
54.50
60.60
59.50
60.40
32.70
65.40
50.30
75.10
59 99.26
Claude Sonnet 3.5 (Anthropic, 2024)
43.30
44.70
56.90
56.60
49,20
90.80
60.20
54.30
57.65
Open-Source Models
Intern-VL2.-118 (Chen et al., 2025)
59.1
60.0
93.2
52.2
61.70
21.30
75,9
19.00
38:10
58.29
Intern-VL2. (Chen et al,, 2022)
64.3
64.3
97.5
95.7
71.10
38.20
79.2
21.30
49.70
64.59
Phi 3.V3..n-isonn-88 (Abdin et al., 2024)
64.9
66.8
84.9
84.1
48.6
11.90
81.8
32.70
55.02
58.97
Open-Source Models 7-32B
Intern-VL.2.-888
LLaVA-Nert-Miirri al., 2025)
(2024a)
58.1
57.7
72.1
71.2
41.7
13:90
33.40
45.54
Ilama 3.2 Vision 13B (Grattaflori et al., 2024)
0.0
0.0
3.5
3.2
0.0
83.40
31.20
59.35
20.07
Chart-Specific LVLMs
ChartGemma- 3B (Masry et 2024b
38,90
37.00
37.90
37.00 00
35.60
20.70
90.80
69
80.16
12.50
21.30
35.67
TinyChart 30 (Zhang 2024)
48.00
46.10
61.90
93.86
83.60
8.30
16.15
Our Qwen2
Qwe-2.5-VLL388 (CoT)
58.10
57.00
76.20
75.60
58.80
13.30
86, 86.10
63.84
75.12
32.60
59.77
59.17
Qwen225-VL-38 SFT
76.10
75.71
76.30
73.80
79.60
58, 05.40
90.00
70
84.60
36.
62.85
68.71
HigCharts-R1333
80.10
81.00
81.20
80.60
78.50
59.90
94.32
$2.00
88.16
37.40
62.38
72.14
Owen2.5-VLL22 VL-7B (CoT)
80.70
79.30
78.30
78.30
73.40
50.40
61.68
71
76.48
41.30
66.85
69:45
Owen2.5-VL-22 SFT
79.10
75.90
79.80
77.70
69.00
BigCharts R1-7B
81.20
81,20
83.80
83.60
80.90
61.90
41.30
66.38
Table 1: Comparison of BIGCHARTS-RI and its variants with open-source and closed- source baselines on chart question answering benchmarks ($5.2). Color coding for compar-
ing categories: closed-source models, open-source models below 7B paramaters, and
open-source models between 7-12B parameters sented in bold.
Best scores in each category are pre-
Evaluation Benchmarks. We evaluate our method and models on a diverse set of chart QA benchmarks which encompasses both real-world and synthetic datasets. For real- world evaluations,
use ChartQA (Masry et al., 2022), and CharXiv (Wang et al., 2024). We also include three synthetic et al., 2018), and PlotQA (Methani et al., (020)-which also serve as effective measures of reasoning abilities. These synthetic datasets often have extraordinarily large test and val- idation splits (for instance, PlotQA'ss Testl set contains 1.1M Q/A pairs derived from just 74 templates), making full-scale evaluations overly expensive. To alleviate this, we create FigureQA-Sus DVQA-Sub, and PloiQA-Sub by sampling 1K chart QA pairs from the origi- nal splits. These smaller subsets preserve the diversity of both visual layouts and question types while significantly reducing the computational burden. We plan to publicly release these subsets, offering the research community practical and representative suite of chart QA evaluation benchmarks to evaluate wide range of chart understanding abilities.
(Kahou et al., 2018), DVQA (Kafle
Evaluation Metrics. For our evaluations, we use exact accuracy as the metric for FigureQA-Sub and DVQA-Sub, while we use relaxed accuracy (Methani et al., 2020; Masry et al., 2022) for PlotQA-Sub and ChartQA. For CharXiv, we use GPT4o in conjunction with the prompt proposed in the original work (Wang et al., 2024) for evaluation.
5.2 Performance Comparison on Chart Question Answering Benchmarks
Synthetic Benchmarks: We compare the performance of various open-source and closed- source baselines with our models across different chart question answering benchmarks in Table 1. Performance of baseline models on synthetic benchmarks such as FigureQA-Sule DVQA-Sub, and PloiQA-Sub relatively low. This is mainly because these datasets con- tain charts that lack explicit numerical labels on their visual elements (Figures 15, 16, 17), forcing models to interpolate numerical values visually. In contrast, charts in the ChartQA dataset typically include clear numerical labels (Figure 14), simplifying the model's recog- nition task. Previous model evaluations on ChartQA have not sufficiently addressed this issue, which we highlight here to encourage further research.
Our results show that fine-tuning the Qwen2.5-VL. models with BIGCHARTS improves per- formance on the synthetic benchmarks, particularly when combined with the reinforce-
8

---

Published as a conference paper at COLM 2025
FigureQ4-Sub
DVQA-Sub
PlotQA-Sub
ChartQA
CharXiv
Avg
Model (SFT Dataset)
Vall
Val2
ValE
ValH
T2
avg
Reas.
Des.
BIGCHARIS
Existing Datasets
Qwen2.5-VI-3B (TinyChart)
47.30
48.80
51.40
48.19
57.09
50.00
70.28
24.60
41.62
48.80
Qwen2.5-VL-ah (ChartGemma)
78.30
79.10
73.30
74.30
40.50
58.60
68.48
10.70
39.25
58.05
Qwen225-VLL3B (BIGCHARTS)
76.10
75.70
76.30
73.80
74.60
58.40
84.60
36.00
62.85
68.70
Original Charts vs. Replotted Charts & Code
Qwen2.5-VI-3B (Original Charts)
73.70
73.10
75.20
72.50
72.40
52.40
82.24
30.90
63.62
66.22
Qwen2,5-VL-3B (Replotted Charts)
76.10
73.70
76.30
73.80
74.60
58.40
84.60
36.00
62.85
68.70
Table 2: Ablation results comparing (i) Qwen2.5-VI-33 fine-tuned BIGCHARTS vs. exist- ing datasets (TinyChart and ChartGemma), and (ii) training on Q A pairs generated from original charts vs. our replotted charts with code. BICCHARTS consistently yields stronger performance, particularly on challenging benchmarks like ChartQA, PlotQA, and CharXiv.
ment learning step. We attribute this the fact that our novel approach of integrating chart images and corresponding data tables during the SFT data generation enables accu- rate numerical retrieval and visual interpolation in our models (illustrated in Figure 2).
BigCharts-R1: Supervised finetuning (SFT) significantly enhances the model's ability to accurately interpret chart images and retrieve precise numerical data, while also strength- ening its reasoning capabilities. However, SFT alone can propagate errors introduced by the teacher models used during curating the finetuning data. Reinforcement Learning (RL), specifically through our novel reward function (Chart Error Rate Reward, CERM), mainly boosts reasoning performance rather than simple data retrieval. This distinction is evident in the descriptive (Des.) subset of the CharXiv benchmark, which emphasizes simple infor- mation retrieval without complex reasoning. Here, RL does not enhance, but infact slightly decreases the performance, unlike other benchmarks which include numerous reasoning- intensive questions benefiting from our RL approach.
Our results show that combining SFT and RL effectively balances improvements in chart comprehension, understanding, and advanced reasoning capabilities. Notably, improve- ments are more pronounced for the 3B model compared to the 7B model, suggesting poten- tial saturation in larger models and highlighting the need for stronger vision capabilities in backbone models through improved pretraining strategies. Overall, BIGCHARTS-RI sets new state-of-the-arr standard for chart reasoning across multiple established chart question-answering benchmarks.
5.3 Ablation Studies
BigCharts Existing Chart Datasets. To ensure fair comparison between BIGCHARTS and prior datasets such as TinyChart (Zhang et al., 2024) and ChartGemma (Masry et al., 2024b), we fine-tuned the same backbone model (0Wen2.5-VL-3B (Bai et al., 2025)) these datasets using identical hyperparameters (see Section 5.1). We also adopted the of- ficial codebases and "program-of-thought" formats from TinyChart and ChartGemma to ensure their optimal performance. As shown in the upper part of Table 2, fine-tuning on BIGCHARIS consistently yields superior performance, especially on challenging bench- marks like ChartQA, PlotQA, and CharXiv. These results highlight the effectiveness and broader potential of BIGCHARTS for advancing chart reasoning research.
Replotted Charts vs. Original Charts. We further compared models trained on Q/A pairs generated from original real-world charts with those trained on our synthetic replot- ted charts (which include both the chart image and underlying code). For fairness, Q/A pairs for the original charts were generated using the same teacher model, Gemini Flash 2.0 (Georgiev et al., 2024), with similar prompts. Both setups used identical training hy- perparameters (see Section 5.1). As shown in the lower part of Table training with our replotted charts leads to consistent performance gains, primarily due to the improved ac- curacy of Q/A A generation enabled by explicit metadata we have shown in Figure 2.

---

Published as a conference paper at COLM 2025
ChartQA
PlotQA-Sub
DVQA-Sub
Relexed Acameyy (ho
Relaxce Acouracy Ph0
Relaved accuracy (N6)
730-
SAWIINN Frntarry 3F1)
Seariadd Fintarig 5mhh
puperieed Fintarrn BIT)
20
125
24
33%
125
170
24
589
2009
125
190 175
Training Stapo
Training Staps
Training Stopp
Figure 4: Comparison between SFT and RL (GRPO) learning curves on in-distribution benchmark (ChartQA) and out-of-distribution benchmarks (PlotQA-Sule A-Sb and DVQA-Sub). The RL model shows significant improvements over the SFT model in all cases.
5.4 Comparing RL vs SFT for Out-of-Distribution Generalization
We compare the generalization capabilities of supervised finetuning (SFT) and reinforce- ment learning (RL), specifically GRPO, on out-of-distribution (OOD) tasks. To effectively evaluate OOD generalization, we exclusively train on the ChartQA (Masry et al., 2022) dataset and treat benchmarks such as PlotQA-Sub, and DVQA-Sub, and FigureQA-Sub as OOD test sets. Since ChartQA lacks step-by-step reasoning annotations, we leverage the Gemini Flash 2.0 (Georgiev et al., 2024) model as a teacher to generate reasoning steps. Specifically, we prompt Gemini Flash 2.0 with the chart image, question, and gold answer to obtain these reasoning steps for the SFT training dataset.
We fine-tune the Qwen2.5-VV. 3B-Instruce model for one epoch CharlQA using both SFT and RL approaches, with batch size of 8 and evaluations every 100 steps. The results are i
presented in Figure 4. For the in-distribution ChartQA test set, both SFT and RL exhibit similar performance trends, though RL consistently shows superior performance under this small-batch training setup. More notably, we observe substantial differences between RL and SFT performances on all OOD benchmarks, clearly indicating that RL generalizes significantly better and reduces overfitting issues inherent SFT. This also supports our decision of utilizing RL for second stage of our training pipeline compared to existing works that only performssperriise models (Masry et al., 2024a;b).
supervised finetuning using data generated from stronger teacher
Conclusion
We introduce BIGCHARTS, novel dataset designed to significantly enhance chart rea- soning capabilities in VLMs. BIGCHARTS uniquely combines the visual authenticity of real-world chart images with the accuracy of synthetic datasets through conditional re- plotting process. This approach addresses the longstanding issues of visual homogeneity and data estimation errors prevalent in existing datasets. We further utilize BIGCHARTS in novel training strategy for chart reasoning, leveraging supervised finetuning and Group Relative Policy Optimization (GRPO)- based reinforcement learning with specially designed reward signals resulting in state-of-the-art the-art chart reasoning model, BIGCHARTS- R1. We believe that BIGCHARTS and BIGCHARTS-RI will serve as valuable resources for fostering continued progress in the field of chart understanding and reasoning.
As future work, we plan to extend our methodology to even broader visual reasoning domains such tables and geometric figures, and designing more diverse reward signals for tackling chart understanding tasks like chart summarization and fact-checking.
Acknowledgments
We thank the reviewers for their valuable suggestions, which helped improve the quality of our work. We are also grateful to Ghazwa Darwiche for technical support and assistance with compute resources. This research was partially supported by the MITACS program.
10

---

Published as a conference paper at COLM 2025
References
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorek, Sébastien Bubeck, Martin Cai et al. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL abs/2404. 14219.
.rg/
Mubashara Akhtar, Oana Cocarascu and Elena Simperl. Reading and reasoning over chart images for evidence-based automated fact-checking. In Andreas Vlachos and Isabelle Augenstein (eds.), Findings of the Association for Computational Linguistics: EACL 2023, pp. 399-414, Dubrovnik, Croatia, May 2023a. Association for Computational Linguis- tics. doi: 10.18653/ h/////11/222222 findiiiinninns-ecc..... URL findings-eacl.
org/2023.
Mubashara Akhtar, Nikesh Subedi, Vivek Gupta, Sahar Tahmasebi, Oana Cocarascu and Elena Simperl. Chartcheck: An evidence-based fact-checking dataset over real-world chart images. arXiv preprint arXiv:2311.07 1.7453, 2023b.
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorski Yejin Choi and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation- based formalisms, 2019, URL https:://arxi http://aaaai..ogaabb/////s/a /1995/ 13319.
Anthropic. Introducing the next generation of claude, 2024. URL http://wwww.tth..pc..cc com/news/cclaude 3- sonnet.
Anas Awadalla, Le Xue, Oscar Lo, Manli Shu, Hannah Lee, Etash Kumar Guha, Matt Jor- dan, Sheng Shen, Mohamed Awadalla, Silvio Savarese, Caiming Xiong, Ran Xu, Yejin Choi and Ludwig Schmidt. Mint-lt: Scaling open-source multinodal data by 10x: A multimodal dataset with one trillion tokens, 2024. URL https://araa www.vv.org/bbb/////// 11271.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan et al. Qwen2.5-vl technical report, 2025. URL /ttps://arxivv.i org/abs/ 2502. 13923.
Soravit Changpinyo, Piyush Sharma, Nan Ding and Radu Soricut. Conceptual 12m: Push- ing web-scale image-text pre-training to recognize long-tail visual concepts, 2021. URL https://arxiv.ct wwtprriwvvogg/abb//r//ee/ wrtp:/s////////////// 08981.
Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal and Ajay Joshi. Leaf-qa: Locate, encode & attend for figure question answering, 2019, URL https://arxivv
https://aarxiv.orr//b/////// wrgg/bb/////// http//bbb.199...11..... 12861.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, Lixin Gu, Xuehui Wang, Qingyun Li, Yimin Ren, Zixuan Chen et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, 2025. URL https://arxiv, org/abs/ 2412.05271.
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian- Guang Lou and Dongmei Zhang. HiTab: A hierarchical table dataset for question an- swering and natural language generation. In Smaranda Muresan, Preslav Nakov and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 1094-1110, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 22.16633/1120222acdhe https://aclantholl.
URL
org//022 acl -lng.
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg and Dario Amodei. Deep reinforcement learning from human preferences, 2023. URL https:/ http://axxvvvoooo abs/1706. 03741.
11

---

Published as a conference paper at COLM 2025
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunx- uan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixi- ang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen et al. Scaling instruction- finetuned language models, 2022. URL
http://ariii.oorrab///21ee 11416.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini and others. Molmo and pixmo: Open weights and open data for state-of-the multimodal models. arxiv preprint ar Kiv:2409..71444 2024.
Petko Georgiev Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, Xinyang Geng, Fred Alcober, Roy Frostig, Mark Omernick et al. Gemini 1.5: Unlocking multi- modal understanding across millions of tokens of context, 2024. URL https://arxinn org/abs/2403. 05530.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jahini Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang et al. The llama herd of models, 2024. URL /ttps::/arxiv, org/ /b//
Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu and Han- wang Zhang. Chartllama: A multimodal Ilm for chart understanding and generation, 2023. URL
16483.
Kaiming He, Xiangyu Zhang, Shaoqing Ren and Jian Sun. Deep residual learning for image 03385.
recognition, 2015. URL
Enamul Hoque, Parsa Kavehzadeh and Ahmed Masry. Chart question answering: State of the art and future directions, 2022. URL http://aaaiivvooaaa/////s// 03966.
Kevin Hu, Michiel A Bakker, Stephen Li, Tim Kraska and César Hidalgo. Vizml: A ma- chine learning approach visualization recommendation. In Proceedings of the 2019 CHI conference human factors in computing systems, pp. 1-12, 2019.
Kushal Kafle, Brian Price, Scott Cohen and Christopher Kanan. Dvqa: Understanding data visualizations via question answering, 2018. URL https: org/ abs/ 08163.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning, 2018.
URL https::/arxxv..og//aaaaaa htarg/abs/1711 1711.07300.0
Shankar Kantharaj, Xuan Long Do, Rixie Tiffany Ko Leong, Jia Qing Tan, Enamul Hoque and Shafiq Joty. Opencqa: Open-ended question answering with charts. arXiv preprint ar MrXiv:2210006622 2022a.
Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Ena- mul Hoque and Shafiq Joty. Chart-to-text: A large-scale benchmark for chart summa- rization, 2022b. URL
064866.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi and Ali Farhadi. A diagram is worth dozen images, 2016. URL https://arxiv.org/abll 07396.
Dae Hyun Kim, Enamul Hoque and Maneesh Agrawala. Answering questions about charts and generating visual explanations. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Syslems, pp. 1-13, 2020.
Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin and Minjoon Seo. The cot collection: Improving zero-shot and few-shot learning of lan- guage models via chain-of-thought fine-tuning, 2023. URL https: //arsvv./.iiv///// org/abs/ 2305. 14045.
12

---

Published as a conference paper at COLM 2025
Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip H. S. Torr, Fahad Shahbaz Khan and Salman Khan. Lim post-training A deep dive into reasoning large language models, 2025. URL https: / /aaxivv. htg//bbs/202... 21321.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjang Yang et al. Tulu 3: Push- ing frontiers in open language model post-training, 2025. URL https: /ttps:///rxivv org/abs/ 2411.15124 15124.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu and Chunyuan Li. Llava-onevision: Easy visual task transfer. Xiv preprint ar
2024a.
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong and Qi Liu. Multimodal ArXiv: A dataset for improving scientific comprehension of large vision- language models. In Lun-Wei Ku, Andre Martins and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 14369-14387, Bangkok, Thailand, August 2024b. Association for Computa- tional Linguistics. doi: 10.18653/v1 wwtp///////22.//looooo//// a0224ancllong 775/.
URL https: http///alaaaaooo.....
Fangyu Liu, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Yasemin Altun, Nigel Collier and Julian Eisenschlos. MatCha: Enhancing vi- sual language pretraining with math reasoning and chart derendering. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12756- 12770, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/ 1ttp///w/2222aee// 2023..cl--oog..71 URL https: // /aaass//llaaanooogyyyy org/2023. acl- -oog.llon..... 714/.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning, 2024a. URL
https://arxiv.. org/abs//2/ 2306. 14565.
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob and Dong Yu. MMC: Advancing multimodal chart understanding with large-scale instruction tuning. In Kevin Duh, Helena Gomez and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Compu- fational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 1287-1310, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 10. 18653/ / vv naa.loo.all- URL https::///cllae https://aclanthhoogyy org/2024. naacl-long.70 70/.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint ar XXXx:2503.01785 2025a.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning, 2025b. URL https: //arxiv. wwtllv..rggbb////eee 2albb//5558..8
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang and Song- Chun Zhu. Inter-GPS: Interpretable geometry problem solving with formal language and symbolic reasoning. In Chengqing Zong, Fei Xia, Wenjie Li and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pracessing (Volume 1: Long Papers), pp. 6774-6786, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1 202Lacl-loog..555 URL https: https/:///caaahhholl/oo org/2021.. .2a.1........ogg acl-long. 528/.
Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty and Enamul Hoque ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In
13

---

Published as a conference paper at COLM 2025
Smaranda Muresan, Preslav Nakov and Aline Villavicencio (eds.), Findings of the Associ- ation for Computational Linguistics: ACL 2022, Pp. 2263-2279, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ http:///ww//222..niinnnssa/e/ee/ URL htt.ttss//aalannhollloooooo wwwphhholog..oggg///00e org/2022. http2/222....nnnsa..eeeea aindings-ac..1111
Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque and Shafiq Joty.
UniChart: A universal vision-language pretrained model for chart comprehension and reasoning- In Houda Bouamor, Juan Pino and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 14662-14684, Singa- pore, December 2023. Association for Computational Linguistics. doi: 10.18653 v1/ 2023.emnlp-main.95 URL htts:: h/tpaa/thlaallgyy..////.. http:/loggooogg////0/ org/2023. emnlp-main. 906/.
Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque and Shafiq Joty. Chartinstruct: Instruction tuning for chart comprehension and reasoning, 2024a.
Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque and Shafiq Joty. Chartgemma: Visual instructiontuunnnn tuniing for chart reasoning in the wild, 2024b. URL https://arxiv. www.w/r///bbb//0...// 04172.
Ahmed Masry, Juan A. Rodriguez, Tianyu Zhang, Suyuchen Wang, Chao Wang, Aarash Feizi, Akshay Kalkunte Suresh, Abhay Puri, Xiangru Jian, Pierre-André Noël, Sath- wik Tejaswi Madhusudhan, Marco Pedersoli, Bang Liu, Nicolas Chapados, Yoshua Ben- gio et al. Alignvlm: Bridging vision and language latent spaces for multimodal under- standing, 2025. URL httptt //axxiv www.ooggg////./..oooo... /atpabb//22221111
Minesh Mathew, Viraj Bagal, Rubèn Pérez Tito, Dimosthenis Karatzas, Ernest Valveny and 12756.
C. V Jawahar. Infographicvqa, 2021. URL https://arxiv.org/abss.
Fanqing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao and Ping Luo. ChartAssistant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning. In Lun-Wei Ku, Andre Martins and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp- 7775-7803, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10.18653/v/ 211/2224.indinngs.oe.. findings-acl. 463/.
URL https://aclanthee hclthooooy .wc.oog..orrgg// org/2024.
Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra and Pratyush Kumar. Plotqa: Reason- ing over scientific plots, 2020. URL httphtss///riii.rrr/aab//////
80997.
OPenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anad- kat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom al. Gpt-4 technical re- port, 2024. URL h/tpps://axxv..or http:/axxiv..ogggb/aasss/ wtt.org///bb//////e 08774.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens et al. Training language models to fol- low instructions with human feedback, 2022. URL https: /tttp:://axxvv...o wttp:v/w.oo///b///2..3/ 02155.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning and Chelsea Finn. Direct preference optimization: Your language model is secretly reward 18290
model, 2024. URL.
Juan Rodriguez, Xiangru Jian, Siba Smarak Panigrahi, Tianyu Zhang, Aarash Feizi, Abhay Puri, Akshay Kalkunte, François Savard, Ahmed Masry, Shravan Nayak, Rabiul Awal, Mahsa Massoud, Amirhossein Abaskohi, Zichao Li, Suyuchen Wang et al. Bigdocs: An open dataset for training multimodal models document and code tasks, 2025, URL https://arxiv.org// www./bs/////aaooooooooooosesrs/s 2412. 04626.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhi- heng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg and Li Fei-Fei. Imagenet large scale visual recognition challenge, 2015. URL https: //arxiv. wwwpvvv.org/bb//400009 /t/p///.b//49..55.... 0575.
14

---

Published as a conference paper at COLM 2025
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017a.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov. Proximal policy optimization algorithms, 2017b. URL https://arxiv.orrg arggabs/1700 .63477.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL
org.gbs.
03300.
Haozhan Shen, Zilun Zhang, Qianqian Zhang, Ruochen Xu and Tiancheng Zhao. Vlm-rl: A stable and generalizable rl-style large vision-language model. https://gi thub. com/ 2025. Accessed: 2025-02-15.
om-aii-ab/VLM-RR)
Ryota Tanaka, Kyosuke Nishida and Sen Yoshida. Visualmre: Machine reading compre- hension on document images, 2021. URL
Benny Tang, Angie Boggust and Arvind Satyanarayan. Vistext: A benchmark for semanti- cally rich chart captioning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7268-7298, 2023.
Luong Trung, Xinbo Zhang, Zhanming Jie, Peng Sun, Xiaoran Jin and Hang Li. ReFT: Reasoning with reinforced fine-tuning- In Lun-Wei Ku, Andre Martins and Vivek Sriku- mar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7601-7614, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1 10565///1/224..acooog.1l URL https://acllatthol.gg org/ acl llog..oo0.
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning, 2021.
URL U
Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, Alexis Chevalier, Sanjeev Arora and Danqi Charting gaps in realistic chart understanding in multimodal Ilms, 2024. 18521.
Chen. Charxiv:
URL https::/aaxxiv.org/ab/////
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist rein- forcement learning. Machine learning, 8:229-256 1992.
Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Bulc, Ranjay Krishna, Aniruddha Kembhavi and others. Scaling text-rich image understanding via code-guided synthetic multimodal data generation. ar preprint ar Xixx:2502.14844 2025.
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Jun- feng Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Alex Lin and Fei Huang. Ureader: Universal ocr-free visually-situated language understanding with multimodal large lan- guage model, 2023. URL https: //arxiv.///xv...////.//
ing with reasoning, 2022. URL http://aaw....o////31111111 /tgg/////////////// 051226.
Eric Zelikman, Yuhuai Wu, Jesse Mu and Noah D. Goodman. Star: Bootstrapping reason- 14465.
Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang and Fei Huang. Tinychart: Efficient chart understanding with visual token merging and of learning, 2024. URL httttp::/aaxxivorrr/aas////////
16635.
Yilun Zhao, Chen Zhao, Linyong Nan, Zhenting Qi, Wenlin Zhang, Xiangru Tang, Boyu Mi and Dragomir Radev. RobuT: A systematic study of table QA robustness against human-annotated adversarial perturbations. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 6064-6081, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 v1/ htt///222ccllllla acl-long..34. URL https://aclanthologgy wrwpoloooooor//222333 acl- -ong-ooo..3333 334/.
15

---

Published as a conference paper at COLM 2025
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language mod- els. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrationns) Bangkok, Thailand, 2024. Association for Computa- 13372.
tional Linguistics. URL http:
Mingyang Zhou, Yi Fung, Long Chen, Christopher Thomas, Heng and Shih-Fu Chang. Enhanced chart understanding via visual language pre-training on plot table pairs. In Anna Rogers, Jordan Boyd-Graber and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 1314-1326, Toronto, Canada, July 2023. As- v1//023/..2220////// aindings-acl.85. URL
sociation for Computational Linguistics. doi: 10.18653/ 10.18653/v1/200 https://aclanthollog org/202/ 2023. findings-ache acl.
A Appendix
A.1 CoT token distribution
Figure 5 presents the distribution of the number tokens in the CoTs generated by Gemini- Flash-2.0 for the supervised finetuning stage. More details about the dataset is provided in $3.2.
Frequency (log scale)
20a
10
102
10a
250
500
750
1000
1250
1500
1750
2000
Chain-of-Thought Length
Figure 5: Distribution of the number of tokens in Stlain-of-Thoughe
A.2 Mint-1T Filtering
We adopt a rigorous two-step filtering approach to collect chart images. In the first stage, train a high-recall ResNet-50 (He et al., 2015) binary classifier using chart images (pos- itive class) and natural images (negative class). We compile the chart images from the following public datasets: ChartQA (Masry et al., 2022), Chart-to-Text (Kantharaj et al., 2022b), DVQA (Kafle et al., 2018), FigureQA (Kahou et al., 2018), LRV-Chart (Liu et al., 2024a), InfoVQA (Mathew et al., 2021), AI2D (Kembhavi et al., 2016), Geo3K (Lu et al., 2021), GeoQA+ (Lu et al., 2021), Hitab (Cheng et al., 2022), robut_sqa, wikis,, robut_wtq (Zhao et al., 2023), Mathqa (Amini et al., 2019), screen2words (Wang et al., 2021), visualmrc (Tanaka et al., 2021), and ureader (Ye et al., 2023). For the natural im- al., 2021) and ImageNet (Russakovsky et al., 2015).
ages, we use CC-12M (Changpinyo We trained the classifier for one epoch and evaluated it on 100 random validation samples using a 0.95 threshold. Manual inspection showed that 88 out of 100 samples were chart- related (e.g,, charts, infographics, dashboards, documents). We then applied this classifier to identify potential charts from the original image pool. In the second stage, we manually labeled 5,000 of these predicted charts to retrain the classifier for higher precision. The fined model, with a stricter threshold 0.98, was used to filter chart images, resulting in 57,196 validated charts.
16

---

Published as a conference paper at COLM 2025
A.3 Google Search Keywords
List of 210 Google Search Keywords for Dataset Collection (Part 1 of 2)
inferactive chunopleth map
clinical trials visualization
goverument board
transparency dash-
real-time traffic dashboard
pharmaceutical data dashboard
demographic data visualization
mental health data visualization
factory efficiency dashboand
election result map
vaccine distribution dashboard
machine performance visualiza- tion
geospatial heatmap
diseuse spread huatmap
production line analytics dash- board
weather forecast dashboard
healthcare KPI dashboard
urban planning visualization
student performance dashhoard
robotics monitoring dashboard
transportation data dashboard
e-learming analytics dashboard
ToT sensor data visualization
university admissions dashboard
satellite imagery visualization
predictive board
maintenance
dash-
completion visualization
historical data map visualization
learning outcomes dashboard
social media sentiment analysis dashboard
supply chain risk dashboard
education spending visualization
warehouse management visual- ization
engagement metries dashboard
teacher effectiveness dashboard
customer retention dashboard
MOOC analytic dashboard
logistics data dashboard
influencer analytics dashboard
school district performance visual- ization
quality control dashboard
brand reputation analysis dash- board
global literacy dashboard
stacked bar chart
public policy impact visualization
grouped bar chart
advertising dashboard
crime statisties dashboard
horizontal bar chart
content performance dashboard
tax revenue visualization
3D bar chart
SBO dashboard
product analytics visualization
city budget dashhoard
clustered chart
sales dashboard
employment rate dashboard
vertical bar chart
welfare program analytics
patient monitoring dashboard
comparative bar chart
environmental impact dashboard
public health analytics dashboard
segmented bar chart
legislation analysis dashboard
hospital performance dashboard
interactive bar chart
transport infrastructure dash- board
epidemic tracking dashboard
animated bar chart
17

---

Published as a conference paper at COLM 2025
List of 210 Google Search Keywords for Dataset Collection (Part 2 of 2)
multiple line chart
boxen plot
statistical infographic
stacked line chart
ridge plot
goographic infographic
smoothed line chart
beuswarm plot
list infographic
trend line chart
heatmap
flowchart infographic
spline chart
correlation heatmap
hierarchical infographic
ansa chart
clustered heatmap
pyramid infographic
stacked area chart
density heatmap
business analytics dashboard
stepped area chart
logarithmic heatmap
sales performance dashboard
streamgraph
calendar heatmap
marketing analytics dashboard
filled line chart
sequential hestmap
financial dashboard
exploded pie chart
divergent heatmap
social media dashboard
multi-level pie chart
matrix heatmap
customer insights dashboard
3D pie chart
geographical heatmap
HR analytics dashboard
semi-circle pie chart
bubble chart
real-time data dashboard
proportional pie chart
radial bar chart
project management dashboard
nested pie chart
polar area chart
executive KPI dashboard
doughnut chart
treemap
side charts
half donut chart
sankey diagram
stacked vs grouped bar chart
radial pie chart
chord diagram
multiple line charts
percentage pie chart
sunburst chart
comparative scatter plots
colored scatter plot
waterfall chart
linked histograms
parallel coordinates plot
3D scatter plot
small multiples visualization
spider chart
chart
clustered scatter plot
parallel coordinate plot
candlestick chart
jitter plot
multiple pie charts
regression scatter plot
OHLC chart
subplots of charts
funnel chart
time-series scatter plot
facet grid visualization
gantt chart
outlier scatter plot
stuck market dashboard
pyramid chart
labeled scatter plot
cryptocurrency dashboard
marimekko chart
denuity scatter plot
financial report visualization
mosaic plot
variable size scatter plot
supply chain dashbuard
tree diagram
frequency histogram
budget allocation dashboard
network graph
comparative histogram
corporate performance dashboard
word cloud
cumulative histogram
time-series line chart
ROI visualization
relative frequency histogram
stacked time-series chart
business growth dashboard
overlaid histogram
seasonal trend
decomposition market segmentation visualiza- tion
logarithmic histogram
chart
histogram with KDE
moving average chart
economic data visualization
animated histogram
exponential smoothing chart
genomic data visualization
probability density plot
trend decomposition chart
climate change visualization
normalized histogram
forecasting chart
population density visualization
box plot
growth curve chart
energy consumption dushboard
notched box plot
rolling mean chart
earthquake data visualization
grouped box plot
autocorrelation plot
epidemiology data dashboard
overlapping box plot
business infographic
model performance dashboard
violin plot
timeline infographic
scientific workflow visualization
split violin plot
process infographic
astronomical data visualization
half violin plot
comparison infographic
medical imaging dashboard
18

---

Published as a conference paper at COLM 2025
BIGCHARTS CoT Data Generation Prompt
Generate nunerical and visual question-anawe answer pairs for an LUM that we are trying ture for Chart Nunerical and Visual Reasoning. should
fornat
each example has four rialds:
input: which only asks nunerical/vissal question,
chain of thought: step-by-step solution that leads to the final ansier,
rinal answer: which is the riral answer to the Inout question based on the chart irage, and question
type:
of cuestion.
access underlying data and context, hovever your questions should be based on the information in the chart irage.
have attached the underlying
that used render the image that you
For the final ansier follow the folloxing instructions: a X should contain as fes words possible.
in image. the Final answer has two more itens, provide it the format like [1, 21-
paraohrase reformat
when asked to give ratio, give out the decinal value like 9.25 instead of 1:4,
when asked to give percentage, give out the whole value like 17 Instead of decixal 1ike 8.17% the Try include the full label the graph when asked about an entity.
Don't include any
Generate questions that contain sone numerical operations such but not linited to, nax, rin, sun, average, dif difference, ratio, redian, rode,
Ganarate qaestions that not only have nusorical operations,
but also some visual aspects such as lef trost, rightnest, top, botton, niddle, peak, colors, etc.
Generate simple data retrieval queations that ask about values, x- e-labels, Tegend Tabels From the chart. Ganerate yes/no nunerical reasoning
oither Yes No.
Generate questions that ask count some elesents in the chart (e.g., the number bars/ple slices/colors/eeleeel Generate unnswerabbe question lon which carsot be arswared based on the viaual inforsation in the figure. The answer to this question should be "Inansserabll""
5, 4, 5, The option labela can different types: al alphabet, Arabic nuserala, or Roran nuserals. The answer should be the option label only.
sultiple-choice question
options.
Generate conversation whose history contains
3 more turns (questions and their answers) in addition to the finel question history and
be
The whole conversation should a single string in the input field.
The reasoning steps to the firal question only should be in the chain_of_thoug field.
The Final answer to the final question should be in the Final_ _anser field.
Resember that your questions should based on the chart image (the code is just helper!), and the chein_of_thhugh should solve the question step by step and shows the answer in the and in this Format:
<thinkings step by step here </thinkings conswer> final answer here </answers.
Figure 6: Prompt to SFT data using Gemini Flash-2.0.
Charts Underlying Python Code Generation Prompt
Recreate the rollowing visunitzation as Python notplotiib code. Ensure it precisely natches the original in terss of color schene, layout, data, text elements, axis labels, title, and overall visual agpearance.
Maintain sase structure original image, and make sure is matching it. Renerber return the python only.
Figure 7: Charts Underlying Python Code Generation Prompt.
A.4 Prompts
We provide the prompt for generating the underlying codes of the charts in Figure 7 and Figure 8. Also, we provide the prompt used for generating CoT data in Figure 6.
A.5 Data Analysis Prompts
We provide the prompts used for data analysis in Figure 9, 10, and 11. For topic identifica- tion, we first use the prompt in Figure 10 to generate a list of topics, and then we group the topics into 20 clusters using the prompt in Figure 11.
A.6 Samples from BIGCHARTS
We provide sample rendered charts from our dataset in Figure 12. In addition, we also provide sample questions and step-by-step solutions in Figure 13.
19

---

Published as a conference paper at COLM 2025
Charts Underlying React Code Generation Prompt
write seript
provided lmage using the Chart. ja 1ibrary.
Ensure code runs seanlassly with simple comsand, such rpe babel-node file_xith_code, ja. The script should rander the chart and save it as shart.png. You may refer to the example below for guidance. Resember should return the code only! Do not return any additional text or explanation!
Figure 8: Charts Underlying React Code Generation Prompt.
Chart Type Classification Prompt
Analyze the provided image and identify the chart type(s) present. Return the detected chart type(s) as a 3S0N array of strings,
Consider the Following chart types and their patential variattions, inclusinn edge cases:
*eSingle Linerss line graph showing data points comected by a single line.
Edge cases: may have rarkers, may be part of larger multi-chart layout, may have a very short time series.
*nLine:ee Similar Single Line, but imply general line graph without specific constraints the number of lines.
Edge
Single Line.
xxbonut: circular chart with a non-solid center, showing proportions of whole.
Edge cases: may have very thin slices, ma have text labels within the slives, ray be partially obscured.
+sStacked Bar & bar chart where bars are divided into segrents represencing subeatsgories, stacked top of each other.
Edge cases: nay have very snall segrents, ray be horizontal, nay have overlapping labels.
*nLine Barras corbinstion ion chart with both line and bar elements.
Edge cases: ray have rultiple lines and bars, ray have different scales for lines and bars, may have overlapping elements.
sMrra Chart:ss line chart with the area between the line and the axis filled in.
overlapping areas, may have gradient *#5catterplot: sA chart showing data points as individual markers on coordinate plare.
Edge cases: ray
any be partially transparent.
Edge cases: may have overlapping points, may have different marker sizes or colors, ma have trend lines.
*nGroup Barcss bar chart where bers grouped category, showing rultiple values for each category. cases: nay have very snall groups, nay have overlapping labels, may be horizontal.
xxPierax A circular chart divided into slices representing proportions of whole.
Edge cases: nay have very thin siices, have text labels within the slices, nay partially ooscured. KABar:Ak chart with rectargular bars representing
values.
Edge cases: may horizontal, may have overlapping labels, ma verying bar widths.
AxMalti Lineua line grash showing muitiple lines regresenting different data series.
Edge cases: nay have large number of lines, ma have overlapping lines, ray have different line styles. chart showing multiple bars representing different data series.
seMulti Darow
Edge cases: nay have large number of bars, may have overlasping labels, may horizontal.
sollistogran: ss chart with verticol bars representing the frequency of data volues in a range. Edge cases: may have overlapping bars, ma have gradient fill. ray be horizontal.
vSBopplotiww chart with box whiskers reoresenting the distribution of values.
Edge cases: nay have overlapping whiskers, ray have grodient fill. may be horizontal.
**Hostrap::e A chart with a grid of calls representing the values of ratrix.
Edge cases: have overlapping cells, have gradient fill, may be horizontal.
saViolin Plotiss chart with violin representing the distribution of data volues. Edge cases: nay have overlapping violins, nay have gradient fill, nay be horizontal.
the chart type is arbiguous not explicttly listed
return "Other'.
miltiple chart types are present, include all then the array-
If the chart is not one of the above, report the most appropriate chart type for it. If no chart type can be idartified, return an empty array: "3"".
Output Forrat:
**"ison
"chart, types": ("Single Line", "Fle")
}
Figure 9: The prompt used for identifying the chart type.
20

---

Published as a conference paper at COLM 2025
Chart Topic Identification Prompt
Analyee the provided chart image and report the topic of the information it represents.
Focus the following aspects detersine the char harrtss care tople:
Types:
being represented (e-g., nuericall, categorical, terporal)?
Variables: [centify the key variables shown in chart (e.g., axes labels, legend entries)..
Relationships: How on the var lablea ralate to each other? what patterra or trends are being ahown?
chart it (e-g., bar chart, line graph, purpose Contextual Clues: Are there any titles, labels, or annotations within the chart that provide direct informatio about the topic?
Chart Type: what
chart)?
provides
presentation.
Image filenane: The image filenare is: "(irage, "(Iroge_filenaa filenane)". This may provide additional context about the chart's 's content and
the image filenane is not neaningful, ignore it. The sain source of information is the title of the chart and the axis labels.
precise description of 3 words or fever. Focus on the core subject natter being presented, not the specific data values.
your analysis, provide
Fewer, subcopic
Always return
of topies.
multiple topice, return
arrey.
there is no clear topic, return an expty array '[3".
"*ssonn
"topic main': ("Toopie"", "Topic2", "Tooic3") "topic ("Topto"", "Topic5", "Topscs")
Figure 10: The prompt used for identifying the chart topics.
Chart Topic Clustering Prompt
You are an expert clustering and topic analysis. You are given a list topics and their subtopics. You nust cluster thes into 28 nain clusters. Your resporse mus be in the fors of 3S0N object, the key is the cluster nase, and the value is list of topics and subtoples from the feput 1ist. Hake sure to put all the given toples in least one cluster; the number topics in each cluster tant ne.
Output Format: (cluster name: E( waann'
sain toples in this cluster, 'sub's sub topics in this cluster)])
Figure 11: The prompt used for clustering the chart topics.
21

---

Published as a conference paper at COLM 2025
Lifetiarnn
IH
1Herrte
Figure 12: Examples of different rendered charts from our BIGCHARTS dataset.
A.7 Sample Outputs from BigCharts-R1 Model
Sample Predictions from BICCHARTS-RI on the ChartQA benchmark are presented in Figure 14. Sample Predictions from BIGCHARTS-RI the PlotQA benchmark are pre- sented in Figure 14. Sample Predictions from BIGCHARTS-RI on the DVQA benchmark are presented in Figure 16. Sample Predictions from BIGCHARTS-RI on the Figureqa bench- mark are presented in Figure 17. Sample Predictions from BIGCHARTS-RRI on the Figureqa benchmark presented in Figure 18.
22

---

Published as a conference paper at COLM 2025
Question: the comp lirk precisio for A2 single precision AS7
Question: How many athletes have total greater than 50007
Queetion:: in 212011, wha the dffieence betwwen Minor
Step step
Step by step
Step
Thhinking-The quection noking precaton greater than ainle Irk precasion
completts
attates having total greater than Altietea reapective are yane marquee 5252, grant 5160, nane agare 5160, mariene 5000, kala meatguez
<thestngsl need
chineinga fixd minor crime
values
fnd the diference between these
As. complete likk predition untae aparostrnnl for The stngle precttion vatue approdmantly for Since greater than d.t,, the answer uthascnng- cwnper--Yeschoowere
pleatle
aci's mimmn
crtee
appeaematayy 600.
Visent
lakya
appocinatay
diftrence betwnen y aue
garea 4226, garca 4402, shells toornine Tolas
between asie mistrum and Vistet difeeracee
micimu and Winer crina 6000
4112, pabrictu
parets
graaer 5060 5484, 2266, ses2, 2508, 5160, There are E.otthreingg conperooortnneer
Then, thd the dfference: aarserr:53lllssee
Si0-ttiniing-
Question: the first cheri, what is the average fraction backstops for the 'sriginal category acrosa dalasets?
Qusstion:
What the difernce between
Question: many values the axis than 3 letters?
masirun and valuos Pox Proft Growh Tron??
Step by step solution:
Step by stop solutions
Step by slup solution:
<trivting- calcutato the average, need to sun the original data points for all datanets in the firtt chart and stride by the number of catancta. The "orggina veluea are 1.55. 8.2 9.4,, 9.22, 8.50 054, 0.20 and 9.29. Burninng theee valune get 6.a-- 42va desa 10000000..0000 2avd 2.5 Now divdde by the number datzeets, which is So. 2.5 6 84375 54175-2thenkng- www.aan---7755-aaaea
<thisiing- Fiut, retilve the values of Pre- Tox Potlt Bren Trand for all years PY50té en, F10077 106. FVaasa a 185, Froars 115, PVa0ad 130. Second find wat which 138 and the wakur which Tird, sutmrst the from
<thinaing-Fiitt have not wakess
adis wiie, kasby,
onn ideal supect. Then, hawe so count haw mony have more than
lattrr Wife hax i letters noboy haw one has ideal haw
Wile, hobtyy adeal and pxpect more sistters eetters-itiiknng conveendoieneeen
120 90 cn.
dthiraing vanoserr classwers
v.0
Question: Calculate the rato of Ricll GDP Growth in PY 2005-07 to that 2011119.
Question: How many ounees are plotted on the chart?
Question: chort (o), what the Injection Rato [3] when the Electron Energy [Gev] is 2?
Step
Step by stop solution:
Step
(he) in 9064477
<thining-Thhe There fire ploted
sthanknng-Lokkngg chan (6), the Election Energy
Grawth (he) abridrs Retto 6.0 1800kthhnkkigg
Each comruponds difurent
[Gav] ls the injectoo Rati [S] about 60 Rittiigingg camoeer Bevttnoee
1800-22meeer>
Figure 13: Examples of different questions and their step-by-step solutions from our BIGCHARTS dataset.
A.8 Charts Replotting
Examples of original charts and their replotted variants in BIGCHARTS in Figure 19.
23

---

Published as a conference paper at COLM 2025
leggst gerder
diferncee wage and salaay?
Quastion: For how many months did the price decresss compared with the previous month?
Question: populaton Abice, America
Model Guiput:
Model
Bouth America more Europe the statement correct?
<thricrg-The gender dffermncss are Maragement bualness trancia operations: 17.1-183 -MI. Proteosional and masse 20.311 - 1088 Service: 5- -L4, Saise
count number mentha where the price decrosedd from January Feteoary. price incremedd From hebraary March, the price decressed.
<ltrkingo
Output:
thining-ffra,,
popeiation Atrics, and South America Aftic'ss popslation 227.79
bigpesl gender diference is 16.1 in Natural rescurewa.
admimstrative supprt 641665 12.2 Natural ressuroos.
osnatrucion, and naireeanee: 17.1-1 - 16.1. Promustion.
trnnsprration and meterial moving: 1255633 122 The
massd: 2544 0.1 Office and
incressed From September to October, the price inoreased.
the piie inrressed. From Augusi to Septersber, the price
Rrom Jure Joly, the price incraasd. From Juy to Augunt,
from March Apri, price incressed. from Apri to May,
price decrassed. From May June, the price decrassed.
South Amerie's populasion 113.77 aetioon sun 113 Next. need lind the populitton of Earops, whrch la 540.35 nelioon Pirelly I compare the lotal population of Alrica. North America and South America with the popuration Europe 514.16
millon, North America's popuration
these populatione 227.76
miltion and
rillion.
conetnuction, and maimersnce chtiniing cinsveroiaturr rescurcss, consttuction and maintrrancacceneeer.
so, the price decreased for morths.chhiking- -anserr-shnmeee
lesa
shan 540.32-0ttining-
Extracted
Extracted Answer:
and maiieenince
Extracted Answer: No
wtte efficii and hroe lcccen the United Stater'
Question: Which category has the highest calistaction ratin??
Question: Who't the merrag whlt of cruiled ted ber and tongettnnbaa
number of male lemsle across age groups?
Output:
Model Dutput:
Model Output:
<he estataction nelage ars Pull workers in your communiy (66) Officisle who na alactinn in Your
<thiscingTThe anallwe two ban are Brasti with 0.2 and Leset Deseloped Countiie with 0.4. The longast teo bart are Land Looked Developing Countrios with 0. and Low inoome Feed Dulictt Countries with o.5.. The average unaliet two (0.23 9-40/
cthinalnguFleet need to calcutoss the sote number main
intabitante sumning the valuse for each age group: 17.00 51.0 8.00 395. Then need to calostate the total number female intatitants by surresing the values
orea (88), Oliouds who ran dections in Your state (95), and Across the United States (21). The leghost setistaction rating is 63. which commupords to both 'Pal workers in your communty and Officutaa who run diections in Your local arss: Since Pall worter in your community appears fret. is the cornec ansoer-tteedigg caneaer-Poll workers your commuriyylwwseer Extracted Poil your communiy
The
each
15.05 08.45 5.21 92.31. Finally subtinct the total number female intubntants from the total
longest (0.8 8.381 2 8.59. ovenage these
boo
is (0.36 0.59)
0.475 ummmingg
main Intabitante: $2.31 00.0 2.71 2.1.cthrrdigg 21:ttan2.7cttreeee
cansverrd ATS-hen
Extracted Answer: 0.475
Extracted Answen: 2.71
Figure 14: Examples of different outputs from BIGCHARTS-RI on the CharlQA benchmark.
Quastiio:: in how manyy years. is the anciment ralio of male students in tersery schools in Lao PDR grailer the evarago onvolment
Quertian: What
Iro
marchandis
100 country?
Model Output:
Question: In how many years, the percentage of the who smoke taken over years? Model Output:
tertary schools Lac
all years
What is the average amount samed from macchandee toports
males smoke
Output:
"modd, supott: Toteningo and the overage anount carned from merchardise se esports 1980, need
1940 par countr 7, yound mate: vesscces e:
percentage
athenking-Pest, need calculate average ensotment nato maie studett tartory schoole
the curinga
POR.
each
1900 and than divide the sumber country 1960 are Nattivas:
<thindingTTe The parcentnges of motee who smois 33A5
G... 7.1, 17.8, and 197. The sum those values 87.9 There are 2 years. So the average is 574)7 7 New. need count how many yours have value greater than 5.27. The valuea are S- 1.O. 1.3. 7.1. 17.5, and 13.7. Only 17.3 and 15.7 are groate than 6.27. So there ane years cotiningg conawersdcteneeee..
valuee are
courtris. The carrings
32.8 31 and 31.8 The everaee percentage 0SS 32
aosccon. Mai: 13000000 Natt: 10100566, Mauroonic a0s0ec0. Samming
31.9 31.8) /44 22 57t, Now seed count hew manr
se60003 13050000
years have percentage greater than 32.575. 33.5 32.575
32.5 32.575 31.5 32.55, 31.8 32.75, So there are
27750000
yeare.cthinking cnnverr/
Extracted Answer:
Extracted Answer: 6940000
Extracted Answer:
Figure 15: Examples of different outputs from BICCHARTS-RI on the PlotQA benchmark.
24

---

Published as a conference paper at COLM 2025
preferred objects
diffrrem calagorics
SEC000
FOLD
Queetion: Uiiic sigsthen los. the oraded aocuery suroned pooss the (Maxc3?
Gueed eni Whah pobered by the romer pocple surtre scrau caageleor
Model Output:
Output:
ee no 30xx 201 Vitair
rt 30 a THE
Since ia was ttan 13, pot proferee mast number paoplk.cttio sing- -rwwnerppoo-eeeeee
8. the etject shoek preferet by live people then trat
prefer S. Sinee
cormeeoond to saae ounge www.eeriiiiiiiitaaeeeeo
Extructed Anower
Extructed Arsenon.po.
Exiradedd Arswerr You
Figure 16: Examples of different outputs from BIGCHARTS-RI on the DVQA benchmark.
Quaetion Cyan lesn than Dare
Ouetoo Cows Cour hav lighal rilav?
Queatio:: Perwenda medias?
Nodel Output: chhiking-Tee Oyan'a aix
Output:
taan suss check marimum recinar vas The
35, Perwent Owep thes 60 Yellow eo Serting thee vakue gine
figh redlav/ttiiiig www.rtt--leeeeee
Figure 17: Examples of different outputs from BIGCHARTS-RI on the Figureqa benchmark.
349
315
Quastion: the highoot avorago
in
stltutons with degrs of tokip neth so with
ctee spud 2009 sobplet
Model Outpot:
grea mango casseerceerr cileneeer
the snectter tameatoo von
Modal Outpet:
dugree 59 ageradnaedy aukk The propertinn of papera
chiidd indingg-- the praserion of pasers fow top tnstaat wha
Yeverd seate Yeward snaee and inewert
E.. Therkoos was secoed fitkeet everge
n.i. naw noe srrgrr with begire valaen
Figure 18: Examples of different outputs from BIGCHARIS-RI on the CharXiv benchmark.
25

---

Published as a conference paper at COLM 2025
scams Australia
2012
Froert
contr COVID-I
New confirmed COWID-15 cases VICTORIA
[921
Figure 19: Examples of original charts (the left) and their replotted variants (the right) in BIGCHARTS.
26


Reviews:
[{"review_id": "QvEEQb9YlU", "reviewer": "Reviewer_wCKY", "summary": "This work propose BigCharts-R1, where the author introduces two novel contributions: (1) A dataset curation pipeline to collect a large scale diverse chart plots from real-world resource and obtain accurate data-chart pairs via replot. (2) introduce a post-training pipeline SFT plus RL to augment general-purpose VLMs (Qwen-3B Vision and Qwen-7B Vision). Here the author introduces two verifiable novel reward functions designed toward chart understanding and reasoning tasks. Finally, the author has compared Big-R1 with a large set of Open-Source and Close-Source VLMs, including chart-based VLMs as well on ChartQA tasks. Big-R1 has achieved competitive performance across multiple benchmarks and further abalation study demonstrate that the proposed Reward Function plus GRPO leads to consistent performance gain on out-of-distribution dataset, which proves Reinformance Learning's capability to enhance model's generalization capability.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.The author propose a very useful dataset resouce to augment VLM's capability for chart question answering tasks, which includes a combination of both synthetic charts and real-world resourced charts to assure a good divsersity. Meanwhile, it also propose a useful replot technique to address the inaccurate data extraction issue from real-world plots. \n2. To my knowledge, this is the first post-training pipeline to include reinforcement learning for chart understanding tasks. The reinforcement learning stage significantly improve model's capability on numerical reasoning type chart understanding questions and the generalization to out-of-distribution tasks. \n3. The details of data collection and model training are clearly introduced, which will help the community to follow up this line of research.", "reasons_to_reject": "1. The experiment does not include a comprehensive study on the effectiveness of different components: (1) What is the usefulness of synthetic charts and real-world charts? (2) What is the effectiveness of the reploted charts vs simply using the original charts and the noisy underlined data prediction? (3) What is the effectiveness of the two reward function? How much gain or loss will I get if I remove one of them. \n2. When compare to previous chart-specific LVLMs (ChartGemma-3B, TinyChart-3B), it would be more fair if those model are further fine-tuned on the BIgCharts datasets instead of simply comparing the original pre-trained weight.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EY3GRXMA0D", "reviewer": "Reviewer_Lpvk", "summary": "This paper addresses the challenge of poor generalization in chart reasoning by current vision-language models (VLMs), largely due to limited visual diversity and inaccurate chart data in training sets. To overcome this, the authors introduce BIGCHARTS, a dataset pipeline that re-generates real-world chart images with accurate underlying data, and BIGCHARTS-R1, a model trained using supervised finetuning and GRPO-based reinforcement learning with chart-specific rewards. Experiments on five benchmarks demonstrate that BIGCHARTS-R1 achieves state-of-the-art performance and significantly improves out-of-distribution generalization.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper presents a moderately new approach by combining real-world chart images with accurate data through a conditional replotting process.  The integration of supervised finetuning and GRPO-based reinforcement learning is appropriate for addressing generalization issues in chart reasoning.  The methodology is clearly described, with a well-structured dataset pipeline and targeted reward design. Empirical results on multiple benchmarks support the effectiveness of the approach, and the framework shows a few potential applicability to related tasks in visual-language understanding.", "reasons_to_reject": "1. **Justification of GRPO**: The choice of GRPO as the reinforcement learning algorithm is not sufficiently justified. The paper does not compare GRPO with other standard RL methods (e.g., PPO), leaving unclear whether GRPO is necessary or advantageous for this task.\n\n2. **Experimental Instability and Setup Issues**: In Section 5.3, the paper claims stable performance for SFT on in-distribution data, yet Figure 4 shows a consistent early peak followed by a sharp drop across multiple datasets. This pattern suggests potential issues in the experimental setup, such as small batch sizes or unshuffled training data, which are not discussed.\n\n3. **Limited Analysis of Reward Function Design**:  The proposed CERM is a central component, but the paper provides little analysis or ablation to assess its specific contribution. Moreover, the final reward is a simple sum of two components (Eq. 5), with no discussion of other possible combination strategies or weighting schemes.\n\n4. **Insufficient Dataset Construction Details**:  The dataset collection process, particularly from heterogeneous sources such as ChartGemma, FigureQA, and Common Crawl, lacks detail regarding quality control, filtering, and normalization. This omission hinders reproducibility and raises concerns about dataset consistency.\n\n5. **Presentation and Structural Issues**: The paper has presentation inconsistencies, such as unnecessary sub-subsections (e.g., only 4.2.1 under 4.2) and inconsistent abbreviation usage (e.g., \"reinforcement learning\", \"Group Relative Policy Optimization\" vs. \"RL\", \"GRPO\") in the paper. These issues affect clarity and readability.", "questions_to_authors": "See \"Reasons To Reject\"", "extra_scores": {}}, {"review_id": "ak1JALrv7y", "reviewer": "Reviewer_P2Yy", "summary": "This is a benchmarking paper. This work introduces a dataset for the chart question-answering task, incorporating existing synthetic datasets and real-world chart data, and a training framework integrating supervised fine-tuning with Group Relative Policy Optimization. The experiment results show improvements using their proposed training framework.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work introduces a large dataset for the chart question-answering task, incorporating existing synthetic datasets and real-world charts data.\n- The proposed training framework shows the improvements in the in-domain and out-of-domain datasets.\n- The writing is overall clear and easy to follow.", "reasons_to_reject": "C1: The technical contribution is limited. The training framework is a combination of technologies in other works, and the proposed Chart Error Rate Reward is impractical for non-numeric answers, i.e., exact-matching criterion.\n\nC2: It may need more studies to support that their proposed BigCharts dataset solves the limitations of existing benchmarks.\n\nC3: It still may need some clarification.", "questions_to_authors": "Q1: For the replotting process, what if the replotted chart mismatches the original chart? How to check whether the plot is reproduced by the generated code, including style and content? Figure 19 shows the figures that are mismatched.\n\nQ2: For Common Crawl processing(Line 138-143), where are labeled chart images from in the first stage? What’s the accuracy after training in the first stage? You have labeled chart images in the first stage and also manually labeled data in the second stage, so why not directly label 5000 charts of data for training a classifier? Furthermore, if you have labeled chart images, why not directly using in-context learning with LLM? The binary classification of identifying whether it is a chart image or not is quite a simple task.\n\nQ3: As Table 3 shows, the BigCharts is an imbalanced dataset regarding chart types, chart topics, question types and answer types. I think it should be a similar situation in the baseline benchmarks, so the sole exact accuracy metric may not be suitable for the evaluation. The additional metrics might be required.\n\nQ4: Do the sampled baseline benchmarks follow the same distribution as their original benchmark?\n\nQ5: It might require additional studies to show whether BigCharts solves the limitations of existing benchmarks, i.e., visual homogeneity of the dataset, estimation errors in underlying chart data(Line 51-52). Suggesting that (1)evaluating Closed/Open-source Models on the BigCharts dataset and (2) training Chart-Specific LVLMs with the BigCharts and then evaluating them on their dataset. See the performance changes in these two studies.", "extra_scores": {}}, {"review_id": "YsV58yXhK2", "reviewer": "Reviewer_8ZTe", "summary": "The paper introduces BIGCHARTS-R1, a state-of-the-art framework for chart reasoning in vision-language models (VLMs). It addresses key limitations in existing models—namely, low visual diversity, reliance on inaccurate chart data, and overfitting from supervised fine-tuning alone—by proposing two main innovations: (1) BIGCHARTS, a visually diverse and accurate chart dataset curated through a novel replotting process that combines real-world chart images with synthesized data; and (2) a two-stage training approach combining supervised fine-tuning (SFT) with Group Relative Policy Optimization (GRPO)-based reinforcement learning. Extensive experiments across synthetic and real-world benchmarks demonstrate that BIGCHARTS-R1 outperforms both open-source and closed-source models in chart question answering and generalization tasks.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors propose a novel dataset curation strategy that combines real-world chart visual diversity with accurate underlying data through a replotting process. This addresses the common issue of synthetic chart homogeneity and real-chart data inaccuracy, leading to a more robust and authentic training dataset.\n\n2. By integrating supervised fine-tuning with Group Relative Policy Optimization (GRPO)-based reinforcement learning, the paper improves both chart comprehension and reasoning abilities. The use of verifiable, task-specific reward functions (e.g., Chart Error Rate Reward) further strengthens the model’s performance and generalization.\n\n3. The model, BIGCHARTS-R1, achieves state-of-the-art results across multiple chart QA benchmarks, outperforming existing open-source and closed-source models, including larger ones. Its performance gain on out-of-distribution datasets underscores the robustness and effectiveness of the proposed methodology.", "reasons_to_reject": "1. While the paper focuses heavily on chart question answering, it provides limited exploration of other important chart-related tasks like summarization, captioning, or fact-checking. This narrows the scope of its generalizability across diverse chart understanding needs.\n\n2. The data generation pipeline relies on vision-language models (e.g., Gemini Flash 2.0) to generate plotting code and question-answer pairs. Although followed by re-rendering and filtering, this introduces a risk of propagating model biases or subtle inaccuracies in both visual and semantic content.\n\n3. The proposed training pipeline, especially the reinforcement learning stage using GRPO with verifiable rewards, demands substantial computational resources (e.g., 8× H100 GPUs). This might limit reproducibility and accessibility for smaller labs or practitioners with limited hardware.", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates chart reasoning in VLMs and propose BigCharts, a dataset pipeline that re-generates real-world chart images with accurate underlying data, and BigCharts-R1, a model trained using supervised finetuning and GRPO-based reinforcement learning with chart-specific rewards. Experiments on five benchmarks demonstrate that BigCharts-R1 achieves state-of-the-art performance and significantly improves out-of-distribution generalization.

The paper received scores of 7, 4, 5, 6 before the rebuttal. After the authors’ response, both reviewers whom scored 4 and 5 raised their score, while the other reviewers maintained their original scores. There was also sufficient discussion between reviewers and authors, and the AC gave additional time for reviewers to further discuss.

Among reviewers and this AC, we acknowledge that this work proposes a solid "dataset curation strategy that combines real-world chart visual diversity with accurate underlying data through a replotting process", and that the supervised fine-tuning with GRPO-based RL is appropriate for addressing generalization issues in chart reasoning. The methodology is clearly described, along with a comprehensive empirical study, shows that the framework has "potential applicability to related tasks in visual-language understanding."

Some minor concerns remain, including:
- Evaluation centers on VQA; however, this AC considers VQA to be a complete and complex task itself.
- Some questions regarding the reward design for text answers; which can be further explored in future works.
- GRPO vs. PPO: authors are encouraged to include PPO experiments, but the presented GRPO-based approach is effective and well-motivated.

Based on the positive feedback after rebuttal and discussions, this AC recommends to accept this work and encourage the authors to incorporate the response and additional results to the main paper or supplementary materials.
Decision: Accept