Do MLLMs Really Understand the Charts?

## Abstract

Abstract Although Multimodal Large Language Models (MLLMs) have demonstrated increasingly impressive performance in chart understanding, most of them exhibit alarming hallucinations and significant performance degradation when handling non-annotated charts 1 1 1 The non-annotated charts are those that require viewers to estimate values using the vertical/horizontal axis scale. . Therefore, a question arises: Do MLLMs really understand the charts? Since a human is capable of understanding charts and estimating the values by visual reasoning, we first carefully establish a comprehensive Chart Reasoning Benchmark ( CRBench ) to rigorously evaluate the visual reasoning abilities of MLLMs on non-annotated charts. We argue that MLLMs are primarily relying on recognition rather than reasoning to interpret the charts. To steer MLLMs to reasonable chart understanding, we propose ChartReasoner that mimics human behavior by grounding their estimation in chart understanding. Extensive results on the proposed CRBench show that ChartReasnoner-3B/7B achieves superior performance in chart reasoning, even compared to GPT-4o and Gemini-2.5-Flash. More importantly, ChartReasnoner also demonstrates the visual reasoning abilities in general chart comprehension on public benchmarks, leading to significant performance gains and enabling MLLMs to rationally understand the charts. The code and dataset will be publicly available upon publication.

## Introduction

Multimodal Large Language Models (MLLMs) (Bai et al. 2025; Comanici et al. 2025; OpenAI et al. 2024a; Lu et al. 2024) now play a pivotal role in the field of Artificial Intelligence, particularly for understanding complex visual data. These models have demonstrated a remarkable ability to process charts, analyze their content, provide insightful explanations, and achieve competitive performance on many existing chart benchmarks (Wang et al. 2024; Masry et al. 2022; Xu et al. 2024b; Masry et al. 2025a; Xia et al. 2025).

However, we discover an intriguing phenomenon: When specific numerical annotations are missing in the charts, in other words, when handling non-annotated charts, the MLLMs’ propensity for hallucination increases dramatically—-a failure exemplified in Figure [1](https://arxiv.org/html/2509.04457v1#Sx1.F1), where a model defaults to hallucination in non-annotated charts while our proposed method succeeds through multi-step reasoning. This leads us to a fundamental question: Do MLLMs really understand the charts?

Figure: Figure 1: illustration of the visual reasoning deficit in MLLMs when processing non-annotated charts. A “standard Initial Model”, limited by its underdeveloped visual reasoning capacity, often resorts to guessing and fails. In contrast, our “Optimized Model” (ChartReasoner) executes a deliberate, human-like reasoning chain—identifying the target, locating data based on visual scales, and forming a conclusion—to successfully estimate the value.
Refer to caption: x1.png

To investigate this issue, we elaborately introduce a Chart Reasoning Benchmark (CRBench), a novel benchmark that specifically requires visual reasoning for chart understanding, focused on the task of numerical estimation on non-annotated charts. Our goal was to rigorously evaluate the visual reasoning capabilities of current MLLMs on chart understanding and explore whether they truly comprehend the visual clues in the charts. The evaluation reveals that not only open-source MLLMs (Bai et al. 2025; Zhu et al. 2025; Lu et al. 2024) face performance degradation, but even powerful close-source MLLMs, such as GPT-4o (OpenAI et al. 2024a) and Gemini-2.5-Flash (Comanici et al. 2025), also struggle significantly with CRBench. Consequently, we argue that most MLLMs are much more relying on visual recognition capabilities, e.g., Optical Character Recognition (OCR), to read the charts, rather than employing genuine visual reasoning to truly understand them.

Inspired by the success of Reinforcement Learning (RL) in enhancing textual reasoning in domains like mathematics and coding (DeepSeek-AI et al. 2025; OpenAI et al. 2024b), we introduce ChartReasoner, a new model forged through a novel Reinforcement Finetuning (RFT) strategy. This two-stage framework is designed not just to recognize the charts, but also understand the charts via visual reasoning. It begins with a “cold start” through Chain-of-Thought Supervised Fine-Tuning (CoT-SFT) to activate a structured thinking paradigm within the MLLM. This initial ability is then dramatically amplified in the second stage, where the Group Relative Policy Optimization (GPRO) (Shao et al. 2024) training method selectively reinforces correct reasoning pathways and penalizes flawed logic.
The experimental results demonstrate that this strategy significantly enhances the visual reasoning capabilities of ChartReasoner, achieving superior performance on the CRbench. Besides, we discovered that the foundational visual reasoning skills cultivated by our method generalize broadly, leading to substantial performance gains across a diverse suite of public chart understanding benchmarks. CharReasoner exhibits highly potential in moving towards a more genuine form of chart comprehension.

The main contributions are summarized as follows:

- •
We introduce CRBench, a distinctive benchmark designed to isolate and evaluate genuine visual reasoning capability in chart understanding. It overcomes the limitations of prior work by focusing exclusively on numerical estimation tasks, thus disentangling reasoning from text recognition.
- •
We propose ChartReasoner, a powerful MLLM with significantly enhanced visual reasoning capabilities for chart understanding. It achieves excellent performance on our challenging CRBench benchmark, compared with chart-specific and general MLLMs, even surpassing powerful proprietary models like GPT-4o and Gemini-2.5-Flash.
- •
We demonstrate that the visual reasoning ability cultivated by our method is foundational and highly generalizable. ChartReasoner is not confined to the specific numerical estimation task, but achieves substantial performance gains across a diverse suite of public, multi-task chart understanding benchmarks.

## Related Work

### Chart Understanding

The advent of MLLMs, encompassing both powerful open-source and close-source (Comanici et al. 2025; OpenAI et al. 2024a; Team et al. 2025), has laid a foundational groundwork for automated chart understanding. Driven by large-scale visual instruction tuning (Liu et al. 2023), this progress has spurred the development of numerous chart-specific fine-tuned models (Han et al. 2023; Zhang et al. 2024; Masry et al. 2024, 2025b) tailored to enhance performance on these specialized tasks. However, the predominant training paradigm for these models relies on SFT (Liu et al. 2023; Wei et al. 2022), which, as we argue, can lead to superficial mimicry rather than genuine reasoning. In contrast, our work explores a paradigm based on RL (Ouyang et al. 2022) to cultivate a deeper, more robust form of chart understanding.

Figure: Figure 2: The training paradigm of ChartReasoner and the data distribution of CRBench. A: ChartReasoner leverages a two-stage RFT strategy. Stage 1 activates the model’s reasoning abilities via SFT on CoT data, while Stage 2 uses GRPO with a multi-component reward system to reinforce correct chart understanding. B: The composition of CRBench, detailing the distribution of seven chart types across both synthetic and real data sources.
Refer to caption: x2.png

**Table 1: Performance evaluation on CRBench with direct answering, covering a range of open-source, close-source, and chart-specific models. This table shows the results when models are prompted to directly output the final answer without intermediate reasoning steps.**
| Methods | Synthetic Charts | Real Charts | Overall |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Box | Area | Radar | Scatter | Bar | Line | Combo | Bar | Line | Combo |  |  |
| Human | 74.72 | 56.70 | 83.97 | 77.90 | 90.10 | 93.71 | 77.27 | 80.36 | 65.22 | 52.00 | 78.64 |
| Open-source Models |  |  |  |  |  |  |  |  |  |  |  |
| InternVL3-2B (Zhu et al. 2025) | 31.87 | 7.32 | 5.77 | 23.19 | 51.40 | 45.70 | 32.98 | 39.29 | 33.04 | 20.80 | 31.59 |
| Qwen2.5-VL-3B (Bai et al. 2025) | 49.45 | 15.24 | 19.23 | 52.74 | 74.58 | 82.45 | 62.48 | 67.86 | 54.78 | 48.80 | 57.68 |
| Ovis1.6-llama3.2-3B (Lu et al. 2024) | 16.48 | 2.44 | 10.90 | 14.22 | 17.04 | 12.25 | 11.52 | 12.50 | 13.04 | 9.60 | 12.47 |
| Gemma-3-4B (Team et al. 2025) | 15.38 | 3.05 | 10.90 | 10.28 | 8.38 | 8.94 | 6.28 | 5.36 | 9.57 | 8.80 | 8.32 |
| Qwen2.5-VL-7B (Bai et al. 2025) | 72.53 | 28.05 | 26.92 | 66.08 | 84.08 | 86.42 | 71.03 | 63.39 | 60.87 | 47.20 | 66.25 |
| InternVL3-8B (Zhu et al. 2025) | 53.85 | 19.51 | 14.10 | 43.33 | 62.29 | 61.92 | 48.52 | 58.93 | 55.65 | 42.40 | 42.40 |
| Close-source Models |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o (OpenAI et al. 2024a) | 32.97 | 15.24 | 12.82 | 25.82 | 22.35 | 25.17 | 19.90 | 15.18 | 26.96 | 16.80 | 21.69 |
| Gemini-2.5-Flash (Comanici et al. 2025) | 72.53 | 32.93 | 11.54 | 59.96 | 70.11 | 71.19 | 55.50 | 53.57 | 46.96 | 37.60 | 55.32 |
| Chart-Specific Models |  |  |  |  |  |  |  |  |  |  |  |
| ChartGemma-3B (Masry et al. 2025b) | 10.99 | 10.98 | 7.05 | 21.44 | 42.74 | 32.78 | 24.43 | 37.50 | 42.61 | 28.80 | 26.74 |
| TinyChart-3B (Zhang et al. 2024) | 13.19 | 7.93 | 7.69 | 25.16 | 56.15 | 54.30 | 36.65 | 57.14 | 40.87 | 34.40 | 35.83 |
| ChartInstruct-7B (Masry et al. 2024) | 10.99 | 1.22 | 4.49 | 16.63 | 35.47 | 16.56 | 18.85 | 51.79 | 45.22 | 18.40 | 20.91 |
| ChartVLM-7.3B (Xia et al. 2025) | 9.89 | 10.37 | 7.69 | 10.28 | 70.39 | 45.36 | 32.81 | 50.00 | 54.78 | 35.20 | 33.63 |
| ChartLlama-13B (Han et al. 2023) | 10.99 | 1.83 | 5.77 | 5.25 | 3.35 | 4.30 | 4.01 | 8.04 | 9.57 | 7.20 | 5.01 |

A suite of chart benchmarks, including ChartQA (Masry et al. 2022), PlotQA (Methani et al. 2020), and others (Xu et al. 2024b; Han et al. 2023; Wang et al. 2024), have been proposed to evaluate MLLMs on diverse chart tasks. While instrumental, their task design often prioritizes language-level reasoning (Lewkowycz et al. 2022) (e.g., calculating averages or summarizing) over fundamental visual reasoning (Johnson et al. 2017; Hudson and Manning 2019). The core skill of grounding reasoning in a chart’s geometry, such as estimating values from axes on non-annotated charts, remains largely untested. Our CRBench is specifically designed to isolate and evaluate this crucial visual reasoning capability.

### Visual Reasoning

RL has been successfully employed to significantly enhance the reasoning abilities of Large Language Models (LLMs) (DeepSeek-AI et al. 2025), particularly in complex domains like mathematics and coding (Shao et al. 2024). Inspired by this success in language-only models, several works have begun to apply similar RL-based paradigms to MLLMs (Feng et al. 2025; Tan et al. 2025; Huang et al. 2025), successfully activating their visual reasoning capabilities on various challenging tasks (e.g., visual counting, spatial transformation). Building on these advancements, our work adapts and applies an RL-based framework to the specialized domain of chart understanding.

## Chart Reasoning Benchmark

We introduce Chart Reasoning Benchmark (CRBench), a comprehensive, multi-domain, and reasoning-centric benchmark designed to rigorously assess the visual interpretation capabilities of MLLMs on charts that lack explicit numerical annotations. Engineered to move beyond simple OCR-dependent tasks, the benchmark comprises a total of 2,453 question-answer pairs. It features a majority (2,101 pairs) of synthetically generated charts to ensure controlled complexity and a significant portion (352 pairs) sourced from real-world examples to guarantee practical relevance. The benchmark provides extensive coverage across seven primary chart types, including bar, line, scatter, and combo charts, with a detailed breakdown of the data distribution shown in Figure [2](https://arxiv.org/html/2509.04457v1#Sx2.F2). This structural diversity is complemented by thematic breadth, with data spanning 38 distinct topics, including finance, healthcare, and technology, etc. This dual emphasis on structural and thematic variety ensures a rigorous evaluation, mitigating the risk of models overfitting to specific chart formats or familiar domains.

### Data Curation

Our data generation process integrates automated synthesis with manual curation to ensure both scale and quality.

#### Synthetic Chart Generation.

Our synthetic chart generation process is partially adapted from the Code-as-Intermediary Translation (CIT) methodology proposed by (He et al. 2024), where executable plotting code serves as the ground truth for each chart. The process begins with a curated set of seed scripts, which are then programmatically diversified using Self-Instruct (Wang et al. 2023) and Evol-Instruct (Xu et al. 2024a) techniques to generate a vast library of visually complex charts. A critical constraint is the deliberate omission of numerical labels on data points, ensuring that every chart necessitates visual estimation. To maximize yield, a self-repair mechanism leverages an LLM to debug and correct any code that fails during execution. Following an automated visual fidelity check by a multimodal language model, the entire collection of synthesized data underwent a final, rigorous human review. This manual verification step served to confirm the high quality of the chart images and the accuracy of their corresponding question-answer pairs. This code-centric approach, combined with multiple stages of validation, provides an unimpeachable ground truth, allowing us to generate verifiably correct Q&A pairs.

#### Real Chart Collection.

To anchor our benchmark in real-world applications, we sourced charts from reputable data repositories such as Statista and Our World in Data.
Each chart was manually vetted by human annotators to meet strict criteria: high visual quality, data integrity, and a complete absence of explicit numerical annotations. Following selection, an MLLM was used to generate candidate question-answer pairs for each chart.
Every MLLM-generated pair then underwent a final round of human verification and refinement to guarantee the accuracy and relevance of both the question and its ground-truth answer.

### Evaluation Protocol

Standard exact-match accuracy is ill-suited for value estimation from non-annotated charts, as it fails to account for the slight perceptual ambiguity inherent in the task, even for human observers. To address this, we employ a relaxed accuracy metric, which judges a prediction correct if its relative error from the ground-truth value falls within a tolerance threshold, denoted as $\tau$. To align this threshold with human performance, we conducted an empirical study and found that human estimations consistently fall within a 2% error margin. Accordingly, we empirically set $\tau=0.02$.
Formally, a model’s predicted value, $A_{pred}$, is deemed correct if and only if it satisfies the following condition relative to the ground truth, $A_{gt}$:

$$ $A_{pred}\in[(1-\tau)\times A_{gt},(1+\tau)\times A_{gt}]$ $$

This protocol ensures that our evaluation is both rigorous and fairly aligned with human-level interpretive capabilities, rewarding models for precise visual reasoning rather than penalizing them for minor, human-like estimation variance.

**Table 2: Comparison of ChartReasoner with representative MLLMs on the porposed CRBench, using Chain-of-Thought (CoT) prompting to elicit the models’ intermediate reasoning steps. The best and second-best scores in each column are highlighted using bold and underline formatting, respectively.**
| Methods | Synthetic Charts | Real Charts | Overall |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Box | Area | Radar | Scatter | Bar | Line | Combo | Bar | Line | Combo |  |  |
| Open-source Models |  |  |  |  |  |  |  |  |  |  |  |
| InternVL3-2B (Zhu et al. 2025) | 25.27 | 8.54 | 9.62 | 25.16 | 51.68 | 43.05 | 34.55 | 43.75 | 34.78 | 32.00 | 32.98 |
| Qwen2.5-vl-3B (Bai et al. 2025) | 46.15 | 14.63 | 17.95 | 52.30 | 72.35 | 80.79 | 62.65 | 66.96 | 53.04 | 46.40 | 56.62 |
| Ovis1.6-llama3.2-3B (Lu et al. 2024) | 12.09 | 3.66 | 7.69 | 14.66 | 13.13 | 10.93 | 11.69 | 8.04 | 16.52 | 12.00 | 11.66 |
| Gemma-3-4B (Team et al. 2025) | 9.89 | 3.05 | 12.82 | 11.82 | 9.22 | 12.25 | 6.81 | 8.04 | 12.17 | 7.20 | 9.34 |
| Qwen2.5-vl-7B (Bai et al. 2025) | 69.23 | 20.12 | 19.87 | 61.93 | 74.02 | 83.77 | 67.71 | 49.11 | 53.04 | 46.40 | 60.74 |
| InternVL3-8B (Zhu et al. 2025) | 36.73 | 12.80 | 12.18 | 39.17 | 43.58 | 47.68 | 36.82 | 38.39 | 46.09 | 34.40 | 36.73 |
| Close-source Models |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o (OpenAI et al. 2024a) | 28.57 | 12.20 | 11.54 | 25.61 | 21.23 | 27.15 | 18.15 | 13.39 | 26.96 | 18.40 | 20.87 |
| Gemini-2.5-Flash (Comanici et al. 2025) | 68.13 | 25.61 | 7.69 | 61.93 | 72.07 | 75.17 | 55.85 | 49.11 | 52.17 | 39.20 | 55.77 |
| Chart-Specific Models |  |  |  |  |  |  |  |  |  |  |  |
| ChartReasoner-3B (Ours) | 58.24 | 21.95 | 22.44 | 56.24 | 79.61 | 86.42 | 66.14 | 65.18 | 54.78 | 48.80 | 61.27 |
| ChartReasoner-7B (Ours) | 72.53 | 31.71 | 32.05 | 65.43 | 82.96 | 84.44 | 71.55 | 65.18 | 66.09 | 53.60 | 67.06 |

## ChartReasoner

We propose ChartReasoner, a MLLM designed to perform visual reasoning for better visual understanding on non-annotated charts. We formally define this task as follows: given a chart image $I$, and a corresponding textual question $Q$, the goal is to derive a numerical answer $A$ with a reasoning procedure $R$. This process can be represented as a mapping function $\mathcal{F}$:

$$ $\mathcal{F}:(I,Q)\rightarrow(R,A)$ $$

where $I$ is the chart image, $Q$ is the question in text, $R$ is the step-wise reasoning procedure in text, and $A\in\mathbb{R}$ is the numerical answer. The fundamental challenge lies in interpreting non-annotated charts, which requires the model to reason about geometric structures (e.g., axes, scales, positions) to infer values, rather than simply extracting them via text recognition. While the mapping $\mathcal{F}$ inherently performs this reasoning, producing an explicit reasoning chain $R$ that explains the derivation is an optional, secondary objective.

To address this challenge, we propose a two-stage Reinforcement Finetuning (RFT) strategy. This approach is designed to first instill a robust, human-like reasoning framework and then meticulously refine the model’s numerical precision. As illustrated in Figure [2](https://arxiv.org/html/2509.04457v1#Sx2.F2), the RFT pipeline consists of two sequential stages: (1) Reasoning Activation via SFT, which uses supervised fine-tuning to teach the model the structure of reasoning, followed by (2) Reasoning Generalization via RL, which uses reinforcement learning to improve the accuracy and generalizability.

### Stage 1: Reasoning Activation via SFT

The initial stage of our pipeline aims to establish a foundational reasoning paradigm. Instead of having the model directly guess an answer, we teach it to adopt a structured, step-by-step thought process that mirrors human analysis.
To achieve this, we first generate a high-quality dataset by distilling detailed Chain-of-Thought (CoT) processes from a MLLM with emerging reasoning abilities. Fine-tuning our base MLLM on these 68,500 synthesized samples teaches the model to emulate a human’s analytical approach, guiding it to identify and utilize critical chart components—such as axes, scales, and legends—before arriving at a final answer.

Formally, we employ SFT on this dataset. Each data instance is a tuple $(x,q,r,a)$, where $x$ is the chart image, $q$ is the question, $r$ is the intermediate reasoning chain, and $a$ is the final answer. The training objective is to maximize the log-likelihood of the model generating the complete sequence $y$ (the concatenation of $r$ and $a$) given the image $x$ and question $q$:

$$ $\mathcal{L}_{\text{SFT}}=-\mathbb{E}_{(x,q,r,a)\sim\mathcal{D}}\sum_{t=1}^{|y|}\log\pi_{\theta}(y_{t}|x,q,y_{<t})$ (1) $$

where $\mathcal{D}$ is our CoT dataset and $\pi_{\theta}$ is the policy of the model with parameters $\theta$. The resulting fine-tuned model, denoted as $\pi_{\text{SFT}}$, learns a robust template for visual reasoning and serves as the starting point for the next stage.

### Stage 2: Reasoning Generalization via RL

Building on the reasoning foundation from Stage 1, the second stage focuses on enhancing the model’s precision and reliability for the numerical estimation task. For this, we use a smaller, high-signal dataset of 3,400 samples curated to target the model’s specific weaknesses. These samples are identified by selecting problems where the SFT-tuned model exhibits ”stochastic correctness”—that is, problems it can solve but not consistently. By training on these borderline cases with higher-resolution images, we force the model to refine its visual interpretation skills.

We employ GRPO (Shao et al. 2024), an efficient and scalable reinforcement learning algorithm, to fine-tune the policy $\pi_{\text{SFT}}$. Unlike traditional algorithms like PPO (Schulman et al. 2017), GRPO forgoes a computationally expensive value network and instead calculates relative advantages by comparing rewards within a group of sampled responses. For each input $(x,q)$, we sample a group of $G$ candidate answers $\{a_{1},a_{2},\dots,a_{G}\}$ from the current policy $\pi_{\beta}$. Each answer $a_{i}$ receives a reward $R(a_{i})$, and these rewards are used to compute a normalized relative advantage $A_{i}$ for each sample:

$$ $A_{i}=\frac{r_{i}-\text{mean}\{r_{1},\dots,r_{G}\}}{\text{std}\{r_{1},\dots,r_{G}\}}$ (2) $$

The policy is then updated to increase the probability of actions with positive advantages, while a KL-divergence penalty against the reference model $\pi_{\text{SFT}}$ ensures stable training.

### Reward Function Design

The effectiveness of our RL stage hinges on a well-designed reward function. Our function $R(a_{i})$ is a composite of two components, targeting both response structure and numerical accuracy:

$$ $R(a_{i})=R_{\text{format}}(a_{i})+R_{\text{acc}}(a_{i})$ (3) $$

#### Format Reward.

To encourage interpretable and well-structured outputs, we provide a binary format reward, $R_{\text{format}}$. The model receives a reward of 1 if its response strictly adheres to our predefined template—reasoning enclosed in <think></think> tags and the final answer in <answer></answer> tags—and 0 otherwise.

#### Accuracy Reward.

To overcome the sparse signal from a simple correct/incorrect binary reward, we introduce a continuous accuracy reward, $R_{\text{acc}}$. This reward provides a fine-grained signal that recognizes ”nearly correct” answers. For a predicted answer $A_{\text{pred}}$ and a non-zero ground truth $A_{\text{gt}}$, we first calculate the relative error:

$$ $d_{\text{rel}}=\frac{\lvert A_{\text{pred}}-A_{\text{gt}}\rvert}{\lvert A_{\text{gt}}\rvert}$ (4) $$

Based on a human-calibrated tolerance threshold of $\varepsilon=0.02$, we define the reward using a piecewise quadratic function that smoothly decays from 1 to 0:

$$ $R_{\text{acc}}(d_{\text{rel}})=\begin{cases}\left(1-\frac{d_{\text{rel}}}{\varepsilon}\right)^{2}&\text{if }d_{\text{rel}}<\varepsilon\\ 0&\text{if }d_{\text{rel}}\geq\varepsilon\end{cases}$ (5) $$

We employ the quadratic formulation for two critical reasons. First, this design provides a clear, bounded, and intuitive reward range. It yields a reward of 1 for a perfect answer ($d_{\text{rel}}=0$) and smoothly decay to 0 as the relative error hits the 2% tolerance boundary. Second, the quadratic shape creates a desirable non-linear decay. It has a gentle slope for subtile errors, granting substantial partial credit for close answers, while the penalty accelerates as the error approaches the tolerance threshold. This behavior encourages the model to make fine-grained improvements when it is already close to the correct answer, while strongly penalizing larger, unacceptable deviations.

Figure: Figure 3: Qualitative analysis on CRBench. GPT-4o and Qwen2.5-VL-7B exhibits hallucination in the answer and reasoning procedure, respectively. In contrast, our ChartReasoner-7B is able to produce a coherent and correct step-by-step reasoning process, leading to an accurate answer.
Refer to caption: x3.png

**Table 3: Performance of ChartReasoner compared to other chart-specific models on various public chart understanding benchmarks. The results of the compared methods are from the corresponding benchmarks. The best and second-best scores in each column are highlighted using bold and underline formatting, respectively.**
| Methods | CRBench | CharXiv (R) | CharXiv (D) | ChartBench | ChartQAPro |
| --- | --- | --- | --- | --- | --- |
| ChartGemma-3B (Masry et al. 2025b) | 26.74 | 12.50 | 21.30 | - | 6.84 |
| TinyChart-3B (Zhang et al. 2024) | 35.83 | 8.30 | 16.15 | - | 13.25 |
| ChartInstruct-7B (Masry et al. 2024) | 20.91 | 8.80 | 21.40 | - | 4.88 |
| ChartVLM-7.3B (Xia et al. 2025) | 33.63 | - | - | 12.06 | - |
| ChartLlama-13B (Han et al. 2023) | 5.01 | 14.20 | 19.23 | 21.30 | - |
| ChartReasoner-7B (Ours) | 67.06 | 43.40 | 63.90 | 45.34 | 36.06 |

## Experiments

### Experimental Setups

#### Training Setting

The implementation was built upon the ModelScope SWIFT framework (Zhao et al. 2025) for all training procedures. We initialize ChartReasoner with Qwen2.5-VL (Bai et al. 2025) and tune it with the dataset described in the previous section. All experiments were carried out on a server with 8 NVIDIA H800 GPUs. The methods evaluated on CRBench follow their default inference settings. All results were obtained from a single forward pass with a fixed random seed of 42. Please refer to the supplementary materials for more details.

#### Training Paradigms and Baselines

To systematically evaluate our training strategies, we conduct a comparative study in four distinct paradigms.

- •
CoT-SFT: Provides a reasoning “cold start” by fine-tuning the model on CoT dataset, with the reasoning data distilled from Qwen2.5-VL-32B (Bai et al. 2025).
- •
GRPO-Zero: An ablation study in which our GRPO algorithm is applied directly to the base model to assess the efficacy of reinforcement learning in isolation.
- •
RFT: Our proposed final paradigm starts from the CoT-SFT checkpoint and uses GRPO to further refine and generalize the model’s reasoning capabilities.

To further validate the generalization and effectiveness of our training paradigms, we conduct experiments across a diverse set of base models. Specifically, we evaluate our methods on both the Qwen2.5-VL-Instruct (Bai et al. 2025) series, comparing the 3B and 7B variants to assess scalability.

**Table 4: Ablation study showing the individual contribution of each strategy (CoT-SFT, GRPO-Zero, and RFT) to the final performance of the ChartReasoner models. The best and second-best scores in each column are highlighted using bold and underline formatting, respectively.**
| Methods | Accuracy |  |  |
| --- | --- | --- | --- |
| Synthetic | Real | Overall |  |
| ChartReasoner-3B |  |  |  |
| Zero-Shot | 46.41 | 42.90 | 45.90 |
| +CoT-SFT | 51.02 | 45.45 | 50.22 |
| +GRPO-Zero | 56.59 | 56.82 | 56.62 |
| +RFT | 60.92 | 54.83 | 60.05 |
| ChartReasoner-7B |  |  |  |
| Zero-Shot | 60.54 | 45.17 | 58.34 |
| +CoT-SFT | 63.59 | 51.70 | 61.88 |
| +GRPO-Zero | 66.44 | 54.83 | 64.78 |
| +RFT | 66.97 | 60.23 | 66.00 |

### Experimental Results

#### Evaluation on CRBench

The results in Table [1](https://arxiv.org/html/2509.04457v1#Sx2.T1) underscore the significant challenge our CRBench presents to current models. The generally low scores across the board reveal a critical weakness: the widespread inability to reason over non-annotated charts. This difficulty stems from our benchmark’s emphasis on genuine visual reasoning—inferring values from axes and scales—rather than relying on annotation-based shortcuts common in other benchmarks. Furthermore, while most methods obtains comparable results on synthetic and real charts, the overall scores are still unsatisfying. To reveal the failure stems, we employed CoT prompting to compel models to externalize their reasoning process. The results in Table [1](https://arxiv.org/html/2509.04457v1#Sx2.T1) and Table [2](https://arxiv.org/html/2509.04457v1#Sx3.T2) show that most methods do not benefit from CoT prompting. It can be attributed to the lack of visual reasoning ability, as shown in Figure [3](https://arxiv.org/html/2509.04457v1#Sx4.F3).

#### Evaluation of ChartReasoner

Our comprehensive evaluation confirms that ChartReasoner-7B decisively outperforms not only existing chart-specific models but also top-tier proprietary models like GPT-4o and Gemini-2.5-Flash, as shown in Table [2](https://arxiv.org/html/2509.04457v1#Sx3.T2). This success stems from its ability to perform genuine visual reasoning, a skill that other models lack, as evidenced by their reliance on taking annotation shortcuts. This advanced capability is particularly evident in its handling of complex chart typologies. For instance, on challenging radar charts, where many models falter, ChartReasoner demonstrates a significant performance leap, proving it can interpret complex visual structures.

#### Ablation Analysis of the Training Paradigm

Our experimental results, summarized in Table [4](https://arxiv.org/html/2509.04457v1#Sx5.T4), strongly validate the effectiveness of the RFT strategy for enhancing visual reasoning. As discussed in the previous section, simply prompting a model with a CoT format often leads to performance degradation, as it forces the model to articulate a reasoning process it does not possess. Our training paradigms were designed to address this fundamental issue. The result clearly indicates that while a naive CoT-SFT offers only a marginal improvement, it is the synergistic combination within the RFT model that yields the most significant performance gains, consistently outperforming all other configurations.

#### Case Study of Visual Reasoning

In Figure [3](https://arxiv.org/html/2509.04457v1#Sx4.F3), powerful models like GPT-4o and Qwen2.5-VL-7B either misinterpret the query or resort to factual hallucination. In contrast, our ChartReasoner, sculpted by the RFT strategy, demonstrates a flawless, step-by-step reasoning process.
This case study provides a compelling visual proof of our quantitative findings: ChartReasoner enhanced by the RFT strategy is transformed from a system prone to errors and hallucination into a reliable and structured visual reasoner that moves beyond merely optimizing for the answer.

#### Evaluation of ChartReasoner on the Public Benchmarks

The robust generalization of ChartReasoner is confirmed by its strong performance across a suite of public benchmarks, as detailed in Table [3](https://arxiv.org/html/2509.04457v1#Sx4.T3). This success stems from our model’s core visual reasoning capability, honed through our training strategy, which is highly transferable to other reasoning-intensive tasks. This approach contrasts with that of other chart-specific models relying on SFT. Although SFT can improve performance on specific tasks, it often fails to instill a generalizable reasoning faculty, limiting extensibility. In contrast, ChartReasoner, enhanced by the RFT strategy, is specifically designed to cultivate this fundamental reasoning, validating the superiority of our method.

## Conclusion

In this paper, we investigate a compelling yet significant question: ”Do MLLMs really understand the charts?” By establishing the CRBench, we extensively evaluated open-source, close-source, and chart-specific MLLMs. The results shows a significant degradation in the performance of these models, and, through chain-of-thought reasoning, revealed their inability to estimate numerical values through visual reasoning, similar to human behavior. To address this issue, we propose ChartReasoner, which enhances the visual reasoning ability of MLLMs via an RFT strategy. This strategy first activates reasoning capabilities through SFT, and then generalizes reasoning abilities through RL. Extensive experiments conducted on the proposed CRBench and public chart reasoning datasets demonstrate the effectiveness of ChartReasoner. This work paves the way for empowering MLLMs to really understand the charts in a human-like manner.

## References

- Bai et al. (2025)
Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025.
Qwen2.5-VL Technical Report.
arXiv:2502.13923.
- Comanici et al. (2025)
Comanici, G.; Bieber, E.; Schaekermann, M.; Pasupat, I.; Sachdeva, N.; Dhillon, I.; Blistein, M.; Ram, O.; Zhang, D.; Rosen, E.; et al. 2025.
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities.
arXiv:2507.06261.
- DeepSeek-AI et al. (2025)
DeepSeek-AI; Guo, D.; Yang, D.; Zhang, H.; Song, J.; Zhang, R.; Xu, R.; Zhu, Q.; Ma, S.; Wang, P.; et al. 2025.
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning.
arXiv:2501.12948.
- Feng et al. (2025)
Feng, K.; Gong, K.; Li, B.; Guo, Z.; Wang, Y.; Peng, T.; Wu, J.; Zhang, X.; Wang, B.; and Yue, X. 2025.
Video-R1: Reinforcing Video Reasoning in MLLMs.
arXiv:2503.21776.
- Han et al. (2023)
Han, Y.; Zhang, C.; Chen, X.; Yang, X.; Wang, Z.; Yu, G.; Fu, B.; and Zhang, H. 2023.
ChartLlama: A Multimodal LLM for Chart Understanding and Generation.
arXiv:2311.16483.
- He et al. (2024)
He, W.; Xi, Z.; Zhao, W.; Fan, X.; Ding, Y.; Shan, Z.; Gui, T.; Zhang, Q.; and Huang, X. 2024.
Distill Visual Chart Reasoning Ability from LLMs to MLLMs.
arXiv:2410.18798.
- Huang et al. (2025)
Huang, W.; Jia, B.; Zhai, Z.; Cao, S.; Ye, Z.; Zhao, F.; Xu, Z.; Hu, Y.; and Lin, S. 2025.
Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models.
arXiv:2503.06749.
- Hudson and Manning (2019)
Hudson, D. A.; and Manning, C. D. 2019.
GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering.
In *2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 6693–6702.
- Johnson et al. (2017)
Johnson, J.; Hariharan, B.; van der Maaten, L.; Fei-Fei, L.; Zitnick, C. L.; and Girshick, R. 2017.
CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning.
In *2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*, 1988–1997.
- Lewkowycz et al. (2022)
Lewkowycz, A.; Andreassen, A. J.; Dohan, D.; Dyer, E.; Michalewski, H.; Ramasesh, V. V.; Slone, A.; Anil, C.; Schlag, I.; Gutman-Solo, T.; Wu, Y.; Neyshabur, B.; Gur-Ari, G.; and Misra, V. 2022.
Solving Quantitative Reasoning Problems with Language Models.
In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., *Advances in Neural Information Processing Systems*.
- Liu et al. (2023)
Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023.
Visual Instruction Tuning.
In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., *Advances in Neural Information Processing Systems*, volume 36, 34892–34916. Curran Associates, Inc.
- Lu et al. (2024)
Lu, S.; Li, Y.; Chen, Q.-G.; Xu, Z.; Luo, W.; Zhang, K.; and Ye, H.-J. 2024.
Ovis: Structural Embedding Alignment for Multimodal Large Language Model.
arXiv:2405.20797.
- Masry et al. (2025a)
Masry, A.; Islam, M. S.; Ahmed, M.; Bajaj, A.; Kabir, F.; Kartha, A.; Laskar, M. T. R.; Rahman, M.; Rahman, S.; Shahmohammadi, M.; Thakkar, M.; Parvez, M. R.; Hoque, E.; and Joty, S. 2025a.
ChartQAPro: A More Diverse and Challenging Benchmark for Chart Question Answering.
In Che, W.; Nabende, J.; Shutova, E.; and Pilehvar, M. T., eds., *Findings of the Association for Computational Linguistics: ACL 2025*, 19123–19151. Vienna, Austria: Association for Computational Linguistics.
ISBN 979-8-89176-256-5.
- Masry et al. (2022)
Masry, A.; Long, D. X.; Tan, J. Q.; Joty, S.; and Hoque, E. 2022.
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning.
In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., *Findings of the Association for Computational Linguistics: ACL 2022*, 2263–2279. Dublin, Ireland: Association for Computational Linguistics.
- Masry et al. (2024)
Masry, A.; Shahmohammadi, M.; Parvez, M. R.; Hoque, E.; and Joty, S. 2024.
ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning.
In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., *Findings of the Association for Computational Linguistics: ACL 2024*, 10387–10409. Bangkok, Thailand: Association for Computational Linguistics.
- Masry et al. (2025b)
Masry, A.; Thakkar, M.; Bajaj, A.; Kartha, A.; Hoque, E.; and Joty, S. 2025b.
ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild.
In Rambow, O.; Wanner, L.; Apidianaki, M.; Al-Khalifa, H.; Eugenio, B. D.; Schockaert, S.; Darwish, K.; and Agarwal, A., eds., *Proceedings of the 31st International Conference on Computational Linguistics: Industry Track*, 625–643. Abu Dhabi, UAE: Association for Computational Linguistics.
- Methani et al. (2020)
Methani, N.; Ganguly, P.; Khapra, M. M.; and Kumar, P. 2020.
PlotQA: Reasoning over Scientific Plots.
In *2020 IEEE Winter Conference on Applications of Computer Vision (WACV)*, 1516–1525.
- OpenAI et al. (2024a)
OpenAI; :; Hurst, A.; Lerer, A.; Goucher, A. P.; Perelman, A.; Ramesh, A.; Clark, A.; Ostrow, A.; Welihinda, A.; Hayes, A.; et al. 2024a.
GPT-4o System Card.
arXiv:2410.21276.
- OpenAI et al. (2024b)
OpenAI; :; Jaech, A.; Kalai, A.; Lerer, A.; Richardson, A.; El-Kishky, A.; Low, A.; Helyar, A.; Madry, A.; Beutel, A.; et al. 2024b.
OpenAI o1 System Card.
arXiv:2412.16720.
- Ouyang et al. (2022)
Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Gray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022.
Training language models to follow instructions with human feedback.
In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., *Advances in Neural Information Processing Systems*.
- Schulman et al. (2017)
Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017.
Proximal Policy Optimization Algorithms.
arXiv:1707.06347.
- Shao et al. (2024)
Shao, Z.; Wang, P.; Zhu, Q.; Xu, R.; Song, J.; Bi, X.; Zhang, H.; Zhang, M.; Li, Y. K.; Wu, Y.; and Guo, D. 2024.
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.
arXiv:2402.03300.
- Tan et al. (2025)
Tan, H.; Ji, Y.; Hao, X.; Lin, M.; Wang, P.; Wang, Z.; and Zhang, S. 2025.
Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning.
arXiv:2503.20752.
- Team et al. (2025)
Team, G.; Kamath, A.; Ferret, J.; Pathak, S.; Vieillard, N.; Merhej, R.; Perrin, S.; Matejovicova, T.; Ramé, A.; Rivière, M.; et al. 2025.
Gemma 3 Technical Report.
arXiv:2503.19786.
- Wang et al. (2023)
Wang, Y.; Kordi, Y.; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H. 2023.
Self-Instruct: Aligning Language Models with Self-Generated Instructions.
In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, 13484–13508. Toronto, Canada: Association for Computational Linguistics.
- Wang et al. (2024)
Wang, Z.; Xia, M.; He, L.; Chen, H.; Liu, Y.; Zhu, R.; Liang, K.; Wu, X.; Liu, H.; Malladi, S.; Chevalier, A.; Arora, S.; and Chen, D. 2024.
CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMs.
In *The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track*.
- Wei et al. (2022)
Wei, J.; Bosma, M.; Zhao, V.; Guu, K.; Yu, A. W.; Lester, B.; Du, N.; Dai, A. M.; and Le, Q. V. 2022.
Finetuned Language Models are Zero-Shot Learners.
In *International Conference on Learning Representations*.
- Xia et al. (2025)
Xia, R.; Zhang, B.; Ye, H.; Yan, X.; Liu, Q.; Zhou, H.; Chen, Z.; Ye, P.; Dou, M.; Shi, B.; Yan, J.; and Qiao, Y. 2025.
ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning.
arXiv:2402.12185.
- Xu et al. (2024a)
Xu, C.; Sun, Q.; Zheng, K.; Geng, X.; Zhao, P.; Feng, J.; Tao, C.; Lin, Q.; and Jiang, D. 2024a.
WizardLM: Empowering large pre-trained language models to follow complex instructions.
In *The Twelfth International Conference on Learning Representations*.
- Xu et al. (2024b)
Xu, Z.; Du, S.; Qi, Y.; Xu, C.; Yuan, C.; and Guo, J. 2024b.
ChartBench: A Benchmark for Complex Visual Reasoning in Charts.
arXiv:2312.15915.
- Zhang et al. (2024)
Zhang, L.; Hu, A.; Xu, H.; Yan, M.; Xu, Y.; Jin, Q.; Zhang, J.; and Huang, F. 2024.
TinyChart: Efficient Chart Understanding with Program-of-Thoughts Learning and Visual Token Merging.
In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, 1882–1898. Miami, Florida, USA: Association for Computational Linguistics.
- Zhao et al. (2025)
Zhao, Y.; Huang, J.; Hu, J.; Wang, X.; Mao, Y.; Zhang, D.; Jiang, Z.; Wu, Z.; Ai, B.; Wang, A.; et al. 2025.
Swift: a scalable lightweight infrastructure for fine-tuning.
In *Proceedings of the AAAI Conference on Artificial Intelligence*, volume 39, 29733–29735.
- Zhu et al. (2025)
Zhu, J.; Wang, W.; Chen, Z.; Liu, Z.; Ye, S.; Gu, L.; Tian, H.; Duan, Y.; Su, W.; Shao, J.; et al. 2025.
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models.
arXiv:2504.10479.

## Appendix

- •
Sec. A provides details of the proposed benchmark, CRBench, including the task definition, the data construction pipeline, and data sources, topics, and chart types.
- •
Sec. B describes the training strategy of the ChartReasoner, detailing the construction of the SFT and RL datasets.
- •
Sec. C presents further details on the inference.
- •
Sec. D includes other relevant details and extended results to supplement the main findings.

## Appendix A A. CRBench Benchmark Details

### Problem Definition

The primary task addressed by our benchmark, CRBench, is numerical value estimation on non-annotated charts.
Formally, given a chart image $C$ and a query $Q$ that specifies a target data point within the chart, the goal is to produce a numerical answer $A$ that accurately estimates the value of that data point.
Crucially, the chart image $C$ is non-annotated, meaning that the numerical values corresponding to graphical elements (e.g., the height of a bar, a point on a line) are not present as explicit text labels.

This task is fundamentally a visual reasoning problem, rather than a simple recognition or textual reasoning task.
To arrive at the correct answer $A$, a model cannot only rely on Optical Character Recognition (OCR).
Instead, it must perform a multi-step cognitive process grounded in the visual geometry of the chart:

- 1.
Semantic Understanding & Grounding: The model must first parse the query $Q$ and correctly associate the textual description with the corresponding graphical elements in the chart image $C$ (e.g., a specific bar, a specific colored line).
- 2.
Structural and Scale Interpretation: The model must identify and interpret the chart’s structural components, particularly the relevant axes (e.g., the y-axis) and their numerical scales, including the range and the value represented by tick marks and grid lines.
- 3.
Spatial and Proportional Reasoning: Finally, the model must perform spatial reasoning by comparing the target graphical element’s dimension (e.g., its height or position) against the interpreted scale of the axis. This often requires proportional estimation or interpolation between labeled tick marks to infer the final numerical value.

By designing a task that necessitates this entire reasoning chain, we directly evaluate a model’s ability to not just recognize a chart, but to truly understand its underlying quantitative information.

Figure: Figure 4: The automated data generation pipeline for CRBench.
Refer to caption: x4.png

### Data Generation Pipeline

The data generation pipeline for CRBench and training datasets, visualized in Figure [4](https://arxiv.org/html/2509.04457v1#A1.F4), is an automated, multi-stage process designed to programmatically create a large and diverse set of non-annotated charts. The entire workflow is built upon generating and refining executable Python plotting scripts, ensuring that each data sample is verifiably correct.

- 1.
Chart Generation by Large Language Model (LLM): The process begins with a curated seed pool of high-quality plotting scripts that serve as few-shot demonstrations. Using a Self-Instruct methodology, an LLM is then prompted to programmatically generate a vast and diverse set of new, non-annotated chart plotting scripts across 38 distinct topics.
- 2.
Validation and Evolutionary Enhancement: Each generated script is rigorously validated via sandboxed execution to ensure it runs error-free. A key feature of the pipeline is an AI-driven repair loop, where the language model automatically debugs and corrects any code that fails execution. Subsequently, a portion of these validated scripts undergo an optional Evol-Instruct phase to further increase their visual and structural complexity.
- 3.
Final Curation: Only scripts that successfully pass the entire validation and enhancement pipeline are accepted into the final datasets. These validated scripts are then executed to generate the final, high-fidelity chart images used in our benchmark and datasets.

#### Ground-Truth Q&A Generation and Verification.

After generating the chart images, we create the corresponding ground-truth Q&A pairs. We first use a LLM to generate questions and answers based on the chart’s source code. Each generated Q&A pair then undergoes a rigorous, multi-tiered verification: it is first cross-checked for accuracy by another LLM instance, and subsequently undergoes a final manual review by human annotators to guarantee its unequivocal correctness.

### Data topics and Chart Examples

To ensure the breadth relevance of CRBench, the data generation process samples from a diverse set of 38 distinct topics, span a wide range of domains including finance, healthcare, technology, and social sciences, guaranteeing that the charts cover a variety of contexts and narratives.
Furthermore, to robustly evaluate a model’s visual reasoning capabilities across different graphical representations, CRBench incorporates seven primary chart types. Figure [6](https://arxiv.org/html/2509.04457v1#A4.F6) provides a representative example for each of these types, showcasing the visual diversity and complexity present in our benchmark.

### Real-World Chart Collection

To complement our synthetic dataset and ensure CRBench reflects the challenges of real-world applications, we curate a substantial collection of charts from public online sources. This process involves careful sourcing from key data providers, a rigorous manual filtering stage, and a hybrid human-AI approach to generate high-quality question-answer pairs.

#### Statista

A significant portion of the real-world charts is sourced from Statista(^2^22https://www.statista.com/), a leading global platform specializing in market and consumer data. Statista provides professional, data-driven visualizations for a wide array of industries, covering topics from economic indicators and market forecasts to technology trends and consumer behavior. The charts on this platform are typically clean, well-structured, and designed for clear communication in business and academic contexts.

#### The World in Data

The second major source is Our World in Data(^3^33https://ourworldindata.org/), a renowned scientific online publication based at the University of Oxford. Its mission is to make data and research on the world’s most significant challenges, such as global health, economic development, and environmental change, accessible and understandable. The visualizations from the World in Data are often more complex and data-rich, including long-term time-series line charts, multi-country scatter plots, and stacked area charts.

In addition to these two primary repositories, the collection is supplemented by charts from various other miscellaneous public reports and online publications. All data collection is conducted in strict adherence to the copyright policies, terms of service, and licensing agreements of each source to ensure full ethical compliance.

#### Manual Filtering and Vetting.

A large pool of charts is initially collected. Each chart then undergoes a meticulous manual vetting process conducted by our team of annotators to ensure it met the stringent criteria for inclusion in CRBench. A chart is only accepted if it satisfied all of the following conditions:

- 1.
High Visual Quality: The image has to be of sufficient resolution, clear, and free of significant compression artifacts or other visual noise that could impede interpretation.
- 2.
Data Integrity: The chart has to be coherent and visually consistent, with clearly defined axes, legends, and graphical elements.
- 3.
Absence of Annotations: We exclusively select charts where numerical values are not explicitly printed on the graphical elements (e.g., no numbers on top of bars or next to points on a line). This constraint is fundamental to our benchmark, as it forces a model to perform genuine visual reasoning rather than relying on OCR shortcuts.

Charts that fails to meet any of these standards are discarded from the final dataset.

#### Question-Answer Pair Generation.

Once a real-world chart is approved, we employ a two-stage, human-in-the-loop process to generate its corresponding question-answer pair:

- 1.
MLLM-based Candidate Generation: We use a capable Multimodal Large Language Model (MLLM) to generate an initial set of candidate question-answer pairs for each chart. The model is prompted to ask a specific numerical estimation question based on the visual information in the chart.
- 2.
Human Verification and Refinement: Every MLLM-generated pair is subsequently subjected to rigorous human review. Annotators would first verify the question’s clarity and relevance. Then, they would carefully perform the visual estimation themselves to validate the accuracy of the answer. If necessary, they would refine the question’s phrasing or correct the ground-truth answer to ensure the final Q&A pair was unambiguous and factually correct.

This careful, multi-stage curation process ensures that the real-world portion of CRBench is not only challenging and relevant but also contains reliable and accurate ground-truth data. Figure [7](https://arxiv.org/html/2509.04457v1#A4.F7) showcases several examples of the final curated real-world charts from our collection, along with their corresponding question-answer pairs.

## Appendix B B. Details of Datasets and Training

### CoT Data Construction for Reasoning Activation

The dataset used for the Supervised Fine-Tuning (SFT) ”cold start” phase is meticulously constructed through a knowledge distillation process.
The goal is to generate high-quality Chain-of-Thought (CoT) data that can effectively activate the reasoning paradigm of our base models.
This process leverages a powerful teacher model(In our experiment, we use the Qwen2.5-VL-32B-Instruct as the teacher mdoel) to generate reasoning traces for the question-answer pairs in our CRBench dataset.

The data construction pipeline, implemented in the generation scripts, involves several key stages to ensure the quality and validity of the final CoT samples:

- 1.
Prompting for Chain-of-Thought Generation: For each (C,Q,A) triplet from original data, we prompt the teacher model to generate a step-by-step reasoning process.
Critically, the prompt(in Figure[8](https://arxiv.org/html/2509.04457v1#A4.F8)) is designed to guide the model to derive the provided ground-truth answer, forcing it to articulate a logical pathway that leads to the correct result.
The model is required to structure its output using specific tags, separating the reasoning steps (<think>...</think>) from the final answer (<answer>...</answer>).
- 2.
Validation and Filtering: Each generated CoT sample undergoes a rigorous, multi-step validation process to filter out low-quality or incorrect reasoning:
•
Structural Check: The generated text is first parsed to ensure that both the reasoning and answer tags are present. Samples with missing tags are discarded.
•
Answer Verification: The final answer extracted from the <answer> tag is programmatically compared against the ground-truth answer. We employ a robust evaluation function that checks for both exact string matches and numerical equivalence within a tolerance threshold to ensure correctness.
•
Leakage Detection: The generated reasoning trace within the <think> tags is scanned for any mention of the ”original answer.” This crucial step prevents the model from ”cheating” by simply copying the ground-truth answer into its reasoning, ensuring that the generated thought process is genuine.

This meticulous, multi-stage distillation and verification process—with examples showcased in Figure [9](https://arxiv.org/html/2509.04457v1#A4.F9) and Figure [10](https://arxiv.org/html/2509.04457v1#A4.F10)—ensures that the data used for the reasoning activation phase is both correct and logically sound, providing a strong foundation for the subsequent reinforcement learning stage.

### Dataset Construction for GRPO

The dataset for the Generative Rejection-based Policy Optimization (GRPO) stage is meticulously curated to focus the training on the most informative examples. Instead of using a random subsample of data, we construct a specialized dataset of moderately difficult samples. These are question-answer pairs that the base model can sometimes answer correctly but frequently gets wrong, representing the frontier of its reasoning capabilities and thus offering the richest signal for reinforcement learning.

The curation process inspired by the principles of rejection sampling and active learning, involves a multi-round, varied-prompting inference pipeline designed to probe the model’s knowledge boundaries.
We first run inference on the entire CRBench dataset multiple times using the same base model.
To elicit diverse responses, we systematically vary the prompts between rounds (e.g., alternating between direct answering and Chain-of-Thought prompts) and increase the model’s sampling temperature to 0.9.
The correctness of every generated response is logged. After the evaluation is complete, we filter this log to isolate the key samples: we select only those question-answer pairs that the model answered correctly in at least one round but incorrectly in at least one other round.

The rationale behind this selective filtering strategy is to maximize the efficiency and impact of the subsequent RL training.
By focusing on these ”boundary” cases, we avoid wasting computational resources on problems the model has already mastered (always correct) or those that are currently far beyond its reach (always incorrect).
Training on these ”boundary” samples forces the policy to learn to distinguish between successful and flawed reasoning pathways for the exact same problem. This targeted approach ensures that the GRPO stage is dedicated to resolving ambiguity and reinforcing robust reasoning where it is most needed, leading to more significant and generalizable improvements in the model’s core reasoning abilities.

### Detailed Formulation of the Accuracy Reward

A core component of the GRPO framework is the Accuracy Reward ($R_{\text{acc}}$), which is designed to provide a dense, fine-grained signal for optimizing the model’s numerical estimation capabilities. A simple binary reward (correct/incorrect) is often too sparse for reinforcement learning, as it fails to differentiate between a near-miss and a completely wrong answer. To overcome this, we designed a continuous reward function that recognizes and rewards ”nearly correct” answers, thereby creating a smoother optimization landscape. The formulation of this reward is visualized in Figure [5](https://arxiv.org/html/2509.04457v1#A2.F5).

Figure: Figure 5: Visualization of our quadratic accuracy reward function ($R_{\text{acc}}$) with a tolerance threshold of $\varepsilon=0.02$. The reward smoothly decays from 1 to 0, with a gentle slope for small errors and an accelerating penalty for larger deviations.
Refer to caption: reward.png

For a predicted numerical answer $A_{\text{pred}}$ and a non-zero ground-truth answer $A_{\text{gt}}$, we first calculate the relative error:

$$ $d_{\text{rel}}=\frac{\lvert A_{\text{pred}}-A_{\text{gt}}\rvert}{\lvert A_{\text{gt}}\rvert}$ (6) $$

Based on a tolerance threshold of $\varepsilon=0.02$, we define the reward using a piecewise quadratic function that smoothly decays from a maximum of 1 to 0:

$$ $R_{\text{acc}}(d_{\text{rel}})=\begin{cases}\left(1-\frac{d_{\text{rel}}}{\varepsilon}\right)^{2}&\text{if }d_{\text{rel}}<\varepsilon\\ 0&\text{if }d_{\text{rel}}\geq\varepsilon\end{cases}$ (7) $$

Our accuracy reward function provides a dense, informative, and well-behaved signal that is ideally suited for guiding our reinforcement learning process towards generating highly accurate and reliable numerical estimations.

## Appendix C C. Inference Details

This section provides additional details regarding our inference and evaluation setup to ensure full reproducibility.

### Inference Hyperparameters and Framework

All experiments reported in this paper were conducted using the default hyperparameters of each respective model, with no model-specific tuning performed at inference time. This ensures a fair and consistent comparison across all models. To guarantee reproducibility, the random seed for all generation processes was fixed to 42. Our inference pipeline is built upon the vLLM framework, which provides efficient, high-throughput serving for Large Language Models.

### Prompting Strategies for Evaluation

To thoroughly investigate the models’ reasoning capabilities and behaviors, we employed three distinct prompting strategies during evaluation, designed to control the externalization of the model’s thought process.
The first is Direct Answering, a baseline prompt that instructs the model to provide only the final answer.
The second, Optional Chain-of-Thought (Optional-CoT), encourages but does not require the model to generate a reasoning trace, allowing us to observe its natural tendency to reason.
The third, Forced Chain-of-Thought (Forced-CoT), strictly requires a step-by-step reasoning process before the final answer, compelling the model to articulate its logic.
As illustrated in Figure [14](https://arxiv.org/html/2509.04457v1#A4.F14) and Figure [15](https://arxiv.org/html/2509.04457v1#A4.F15), which showcases two comparative examples, these varied strategies allow us to gain deeper insights into not just what the models answer, but how their performance is affected by the explicit demand for reasoning.

## Appendix D D. Evaluation Protocols for Specialized Models and Benchmarks

This section outlines the precise met hodologies used to evaluate specialized, chart-specific models and to benchmark our ChartReasoner on existing public datasets. Our protocols were designed to ensure maximum fairness and comparability by strictly adhering to the author-recommended settings and official evaluation scripts for each model and benchmark.

### Evaluation of Chart-Specific Models on CRBench

Evaluating the performance of existing chart-specific models on our proposed CRBench required a careful approach to ensure a fair comparison. These models, including ChartGemma, TinyChart, ChartInstruct, ChartVLM, and ChartLlama, are often fine-tuned on specific data formats and are highly optimized for a particular style of prompting.

For this reason, we did not use our generalized prompting templates (as shown in Figures [12](https://arxiv.org/html/2509.04457v1#A4.F12) and [13](https://arxiv.org/html/2509.04457v1#A4.F13)) for these models. Instead, to elicit the best possible performance from each model and establish the strongest baseline, we followed this procedure:

- 1.
We cloned the official public repository associated with each chart-specific model.
- 2.
We utilized their provided, out-of-the-box inference scripts and default model weights without modification.
- 3.
We fed the images and questions from our CRBench test set into these scripts.

This methodology ensures that the results reported in the main body of the manuscript represent each model performing under its own optimal, author-intended conditions. This prevents any potential performance degradation that might arise from imposing a foreign prompt structure and guarantees that we are comparing ChartReasoner against the most capable version of each specialized model.

### Evaluation of ChartReasoner on Public Benchmarks

To validate the generalization of ChartReasoner’s enhanced reasoning capabilities, we evaluated it against several standard public benchmarks. The following subsections detail the exact procedure used for each.

#### CharXiv

For the CharXiv benchmark, we utilized the official code and evaluation scripts available in their repository. Our evaluation process involved adapting their provided inference pipeline. We integrated our locally-deployed ChartReasoner (served via vLLM) as the model backend, replacing the default model call. All other components, including data loading, pre-processing, and the final scoring scripts for both the reasoning (R) and descriptive (D) tasks, were kept identical to the original setup.

#### ChartBench

Our evaluation on ChartBench followed a similar protocol. We used the complete evaluation pipeline provided in the official ChartBench repository. Predictions for each test instance were generated by passing the questions to our ChartReasoner vLLM endpoint. The resulting outputs were then fed directly into the official scoring script, which computes the overall accuracy based on their defined metrics and question categories.

#### ChartQAPro

The official repository for ChartQAPro provides a standalone evaluation script but does not include a full inference pipeline. Therefore, to benchmark ChartReasoner, we implemented a two-step process that carefully reconstructs the intended evaluation conditions:
1. Prediction Generation: We developed a custom script to generate predictions for the entire ChartQAPro test set. For each question, this script formatted a query for ChartReasoner that precisely replicated the direct prompt template described in the official ChartQAPro paper.
2. Scoring: The complete set of generated predictions was saved to a file. This file was then used as input for the official evaluation script provided by the ChartQAPro authors. The script automatically compares our model’s predictions against the ground-truth annotations to compute the final accuracy score.

Figure: Figure 6: An overview of sample question-answer pairs for various synthetic chart types within the CRBench dataset.
Refer to caption: x5.png

Figure: Figure 7: An overview of sample question-answer pairs for various real chart types within the CRBench dataset.
Refer to caption: x6.png

Figure: Figure 8: The Prompt in Distilling the CoT Data
Refer to caption: x7.png

Figure: Figure 9: Example 1 of CoT Dataset
Refer to caption: x8.png

Figure: Figure 10: Example 2 of CoT Dataset
Refer to caption: x9.png

Figure: Figure 11: Direct Prompting in CRBench Evaluation
Refer to caption: x10.png

Figure: Figure 12: Prompting for Optional Output of Reasoning Steps in CRBench Evaluation
Refer to caption: x11.png

Figure: Figure 13: Prompting for Forced Output of Reasoning Steps in CRBench Evaluation
Refer to caption: x12.png

Figure: Figure 14: Example 1 from CRBench comparing the CoT outputs of Qwen2.5-VL-7B and ChartReasoner-7B.
Refer to caption: x13.png

Figure: Figure 15: Example 2 from CRBench comparing the CoT outputs of Qwen2.5-VL-7B and ChartReasoner-7B.
Refer to caption: x14.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 2
presentation: 3
questions: - How are the intermediate steps of the reasoning evaluated on ChartVRBench? I cannot find this in Section 3.2 of the evaluation protocol.
soundness: 2
strengths: - The paper is well-written and easy to follow, with a clear structure.
- The motivation behind the work is interesting and well-presented.
- The proposed method (ChartVR) is technically sound and clearly described.
summary: This paper introduces ChartVRBench, a benchmark designed to isolate and evaluate visual reasoning capability in chart understanding. The authors proposed ChartVR, a series of LVLM with significantly enhanced visual reasoning capabilities for chart understanding. The paper tackles an important problem at the intersection of LVLM and data visualization.
weaknesses: 1. The paper is missing a discussion and analysis of the proposed benchmark. This paper claims the benchmark, ChartVRBench, is part of its contribution. However, there is limited analysis of the benchmark's quality or comparison with existing benchmarks.
2. The chart data in the benchmark are mostly synthesized by an LVLM. Without assessing its quality, the performance on the benchmark cannot be supported. Evaluation, such as using human evaluators to check the correctness of the generated answers and the diversity of the generated questions, is highly recommended. Also, using an LVLM to generate data can introduce its own biases; for example, generated charts tend to have monotonic trends (only going upward or downward). One can observe in Figure 5 that the charts seem to have only simple trends, which is largely different from the real-world distribution. How is this addressed in the data generation pipeline? Is there any post-filtering or evaluation to ensure correctness and diversity?
3. The benchmark features charts without annotations. However, existing benchmarks (such as ChartXiv, ChartBench, and EvoChart) also contain charts without annotations. What is the difference between the proposed benchmark and these existing works? What are the novel benefits that this benchmark brings to the community? Is the data higher quality? Does the data cover more chart types? Or is the data aligned more closely with the real-world distribution?
4. A head-to-head comparison is needed to show the effectiveness of the proposed solution. The training of ChartVR involves an RL algorithm (GRPO), which is different from existing supervised finetuning techniques. A head-to-head comparison with previous methods on published benchmarks is necessary to support the effectiveness of the proposed method. For example, the comparison in the table is unfair: ChartVR-7B is built on Qwen 2.5VL, which is a much stronger backbone model than those used in existing approaches. A fairer comparison would involve keeping the backbone models the same as in previous work to showcase the effectiveness of both the proposed dataset and the training algorithm. Additionally, you should fix the model and data for both previous works and your model, then train using the previous SFT algorithm and your RL training to showcase the effectiveness of the proposed training strategies. Without these head-to-head comparisons, I cannot be convinced of the approach's effectiveness.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See the weaknesses.
soundness: 3
strengths: 1. The paper is well-organized, clearly introducing the background and motivation.
2. The motivation and problem studied in this paper are significant, highlighting current MLLMs could not understand charts without annoations accurately.
3. The benchmark is necessary for the field of chart understanding, which measures the ability of MLLMs in chart understanding in a specific scenario.
4. The performance of ChartVR on both ChartVRBench and public general chart understanding benchmark is great.
summary: This paper studies the problem that MLLMs fails to undertand charts without annotations. A specific benchmark ChartVRBench is developed to comprehensively evaluate the ablites of MLLMs in such a scenario. Besides, to boost the performance of MLLMs in undertanding annotation-free charts, ChartVR is proposed.
weaknesses: 1. In Table 2, since MLLMs with size of 3B are compared, the performance of ChartVR-3B on public chart understanding benchmarks is supposed to be compared.
2. The performance of MLLMs on Radar charts is poor and far from human baseline, does it mean the reasoning pattern could not generalize to this type?

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: See Cons.
soundness: 2
strengths: **Important Topic**: Chart understanding is a crucial area for advancing MLLM applications in real-world data analysis, such as scientific papers and financial reports.

**Comprehensive Training Approach**: The VR-RFT strategy effectively combines SFT and RL, with the continuous quadratic accuracy reward delivering a dense, fine-grained signal to refine visual estimation precision.

**Strong Empirical Results and Generalization**: ChartVR demonstrates superior performance on the new benchmark and transfers well to diverse public datasets, suggesting the cultivated visual reasoning is foundational and not task-specific. The paper includes qualitative case studies highlighting improved reasoning chains.
summary: The paper examines whether Multimodal Large Language Models (MLLMs) truly comprehend charts, especially non-annotated ones requiring visual estimation of numerical values. It introduces ChartVRBench, a benchmark with 2,453 Q&A pairs across seven chart types, using synthetic and real-world data to test visual reasoning. The authors propose ChartVR-3B/7B, trained with a two-stage Visual Reasoning Reinforcement Finetuning (VR-RFT) method—Chain-of-Thought Supervised Finetuning (CoT-SFT) followed by Group Relative Policy Optimization (GRPO) with a continuous accuracy reward. Experiments show ChartVR outperforming models like Gemini-2.5-Flash on ChartVRBench and generalizing well to other benchmarks.
weaknesses: **Limited Novelty**:  I think this is the main weakness of this paper. The core idea of visual reasoning on non-annotated charts is not entirely new; benchmarks like PlotQA (cited in the paper, but several years before) already include such data without explicit annotations. Also, the synthetic data generation relies on established methods like Code-as-Intermediary Translation and Self-Instruct, reducing the perceived innovation in dataset curation.

**Insufficient Comparison to Recent SOTA Models**: The evaluation lacks direct comparisons with stronger open-source chart-specific models like Chart-R1, which employs similar CoT supervision and RL strategies to achieve state-of-the-art results on chart understanding benchmarks—potentially undermining claims of superiority.

---

confidence: 5
contribution: 1
initial_rating: 2
presentation: 1
questions: 1. Can you please provide an unambiguous confirmation that a strict and held-out test set from ChartVRBench was used for evaluation, and that no part of this test set (images, questions, or answers) was used in any capacity during the SFT or RL data construction phases?

2. Given that VR-RFT is a pipeline of existing techniques (CoT-SFT, GRPO), could you clarify what you see as the core novel algorithmic contribution beyond the application to this specific task and the design of the reward function?

3. How can you substantiate the claim that the model has learned a "foundational" reasoning ability, rather than just a set of chart-specific interpretation skills? Why were no out-of-domain reasoning benchmarks (e.g., math, general VQA) included to test this claim of generalizability?
soundness: 2
strengths: 1. Excellent Problem Formulation: The paper's core premise is strong, clear, and important. It correctly identifies a key weakness in current MLLMs and frames a well-defined research question: are models reasoning or just recognizing? The focus on non-annotated charts is a methodologically sound way to isolate this specific capability.

2. Purpose-Built Benchmark: The creation of ChartVRBench, a benchmark designed specifically to test a single, well-defined skill (visual numerical estimation), is a good scientific practice. The inclusion of both synthetic and real-world data is a positive design choice.
summary: This paper investigates a critical failure mode in Multimodal Large Language Models (MLLMs): their inability to perform visual reasoning on charts that lack explicit numerical annotations. The authors argue that MLLMs excel at text recognition but fail at genuine visual reasoning, which they define as the ability to estimate numerical values from a chart's visual geometry (e.g., axes, scales, and positions).

To address this, the paper makes two primary contributions. First, it introduces ChartVRBench, a new benchmark specifically designed to isolate and evaluate this numerical estimation skill. The benchmark contains both synthetic and real-world charts, all deliberately stripped of explicit data labels. Second, it proposes ChartVR, a series of MLLMs trained with a novel two-stage Visual Reasoning Reinforcement Finetuning (VR-RFT) strategy. Stage 1 ("Activation") uses Supervised Fine-Tuning on distilled Chain-of-Thought (CoT) data to teach the model a structured reasoning process. Stage 2 ("Generalization") uses Group Relative Policy Optimization (GRPO), a reinforcement learning algorithm, to refine the model's precision, guided by a continuous quadratic accuracy reward function.
weaknesses: 1. High Risk of Data Contamination and Invalid Evaluation: The paper states that the training data for both the SFT and RL stages is derived directly from the ChartVRBench dataset itself. The SFT data consists of reasoning traces distilled from ChartVRBench's question-answer pairs, and the RL data consists of samples from ChartVRBench where the SFT model was inconsistent. The paper never explicitly states that it maintained a strict separation between the subset of ChartVRBench used for training data generation and the subset used for final evaluation. This strongly suggests that the model was trained on the same data distribution it was tested on, making the impressive results on ChartVRBench a measure of memorization or overfitting, not generalization. This potential train-test overlap invalidates the paper's primary empirical results.
2. Incremental Methodological Contribution: All the techniques are from prior works.
3. Narrow Evaluation Scope and Unconvincing Generalization Claims: The paper's claim that it imparts a "foundational and highly generalizable" reasoning ability is not sufficiently supported. The evaluation of generalization is restricted only to other chart-specific benchmarks (CharXiv, ChartBench, ChartQAPro). This demonstrates in-domain transfer, but it does not prove that the model has learned a general reasoning skill. To validate a claim about foundational reasoning, the evaluation should have included out-of-domain benchmarks that require similar geometric or logical skills, such as mathematical reasoning.
4. Concerns with Distilled Data Quality and Utility: The SFT stage relies on reasoning data distilled from a powerful teacher model (Qwen2.5-VL-32B). This introduces two issues: i) The student model's reasoning quality is capped by the teacher's ability. It is learning to mimic one specific, potentially flawed or suboptimal, reasoning style. ii) The very fact that a 32B model can generate these reasoning traces (even post-hoc) raises questions about the dataset's utility for training even more powerful models in the future. It creates a potential "ceiling effect" for the task itself.


Decision: Reject