ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation
Authors: Xuanle Zhao, Xianzhen Luo, Qi Shi, Chi Chen, Shuo Wang, Wanxiang Che, Zhiyuan Liu, Maosong Sun
Accepted at: acl25

ChartCoder: Advancing Multimodal Large Language Model for Chart-to-Code Generation

Xuanle Zhao1,*, Xianzhen Luo2,*, Qi Shi1,†, Chi Chen1,†,
Shuo Wang1, Zhiyuan Liu1, Maosong Sun1
1 Tsinghua University, Beijing, China
2 Harbin Institute of Technology, Harbin, China
2429527z@gmail.com, xzluo@ir.hit.edu.cn

Abstract
Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in chart understanding tasks. However, interpreting charts with textual descriptions often leads to information loss, as it fails to fully capture the dense information embedded in charts. In contrast, parsing charts into code provides lossless representations that can effectively contain all critical details. Although existing open-source MLLMs have achieved success in chart understanding tasks, they still face two major challenges when applied to chart-to-code tasks: (1) Low executability and poor restoration of chart details in the generated code and (2) Lack of large-scale and diverse training data. To address these challenges, we propose ChartCoder, the first dedicated chart-to-code MLLM, which leverages Code LLMs as the language backbone to enhance the executability of the generated code. Furthermore, we introduce Chart2Code-160k, the first large-scale and diverse dataset for chart-to-code generation, and propose the Snippet-of-Thought (SoT) method, which transforms direct chart-to-code generation data into step-by-step generation. Experiments demonstrate that ChartCoder, with only 7B parameters, surpasses existing open-source MLLMs on chart-to-code benchmarks, achieving superior chart restoration and code excitability. Our code is available at https://github.com/thunlp/ChartCoder.

1 Introduction
Recently, Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in addressing a wide range of visual tasks, such as captioning and question answering (Zhang et al., 2024c; Wang et al., 2024b; Bi et al., 2024b; Zhang et al., 2025). However, current models still face significant challenges in understanding and analyzing the dense visual information present in complex and informative images. As a significant form of information-intensive images, charts contain complex information such as data and structures, playing a pivotal role in effectively presenting details. The automation of chart comprehension and summarization has garnered significant attention from the research community. To advance chart understanding tasks, current studies leverage existing MLLMs and perform supervised fine-tuning (SFT) on various large-scale datasets, such as chart question answering (Methani et al., 2020) and chart-to-text generation (Kantharaj et al., 2022), achieving state-of-the-art performance on existing chart understanding benchmarks.

However, existing works generally treat charts as natural images and fine-tune models by generating natural language descriptions (Zhang et al.,

![Comparison of existing MLLMs performance on ChartQA and ChartMimic benchmarks. In the chart-to-code task, open-source MLLMs struggle with mismatches in chart types and sizes, whereas ChartCoder generates accurate code.](page_823_370_563_312.png)

Figure 1: Comparison of existing MLLMs performance on ChartQA and ChartMimic benchmarks. In the chart-to-code task, open-source MLLMs struggle with mismatches in chart types and sizes, whereas ChartCoder generates accurate code.
2024b; Han et al., 2023; Meng et al., 2024; Bi et al., 2025). This inevitably overlooks the dense information embedded within the charts, resulting in inefficient analysis and comprehension. On the other hand, parsing a chart into code offers a lossless representation, providing a more efficient and comprehensive approach to understanding the chart by accurately capturing and summarizing all its information. Recent works (Shi et al., 2024; Wu et al., 2024; Xia et al., 2024) have proposed various chart-to-code benchmarks, aiming to evaluate the chart reasoning abilities through code. However, current open-source MLLMs are not well-aligned with code generation tasks (Zhang et al., 2024a), resulting in poor performance in parsing charts into corresponding code and limited execution rate of the generated code. Figure 1 shows that the InternVL2-8B suffers from chart type errors and coordinate size mismatches when converting boxplots to code.

To overcome the above challenges in chart-to-code generation, we first conduct an exploratory attempt by leveraging Code LLMs as the language backbone of the MLLMs and propose ChartCoder, the first dedicated chart-to-code MLLM, which incorporates a two-stage training paradigm that contains chart-to-text alignment and chart-to-code instruction tuning. However, compared to chart-to-text, the available chart-to-code dataset is significantly smaller in scale, making it insufficient to support effectively supervised fine-tuning. Therefore, to address the scarcity of data for the chart-to-code domain and train our proposed ChartCoder, we propose the first large-scale, diverse and high-quality chart-to-code dataset named Chart2Code-160k along with the model, which contains 160k diverse chart-code pairs with 27 chart types. To enhance the model’s capacity to capture critical information, such as chart types and data values, and strengthen its reasoning ability, we propose the Snippet-of-Thought (SoT) method, which emphasizes critical information and optimizes the chart-to-code reasoning process. Specifically, we sample 50k chart-code pairs from the Chart2Code-160k, then utilize Chain-of-Thought (CoT) (Wei et al., 2022) method to extend direct generation to step-by-step generation, which aims to emphasize critical information in each step. Experimental results show that by utilizing our proposed Chart2Code-160k with the SoT method, ChartCoder, which, with only 7B parameters, outperforms all open-source MLLMs across various chart-to-code benchmarks. As shown in Figure 1, ChartCoder demonstrates a significantly higher ability to generate correct and executable code.

In summary, the main contributions of this work are as follows:

• We propose ChartCoder, the first chart-to-code MLLM, which leverages Code LLMs as language backbones. With only 7B parameters, ChartCoder outperforms existing open-source MLLMs on chart-to-code benchmarks.

• We introduce Chart2Code-160k, the first large-scale and diverse chart-to-code dataset, consisting of 160k chart-code pairs across 27 chart types.

• We propose Snippet-of-Thought (SoT), transforming direct generation to step-by-step generation to emphasize critical information and enhance reasoning capabilities.

2 Related Works

2.1 Chart Understanding

Chart understanding is a crucial area of research that encompasses both low-level and high-level tasks. Previous approaches (Singh et al., 2019; Methani et al., 2020) have typically relied on pipeline-based methods. However, these pipeline approaches often struggle with error accumulation across different stages, which limits their overall effectiveness and flexibility. Recent works have led to the development of end-to-end MLLMs (Liu et al., 2023b,c; Yu et al., 2025) specifically designed for chart-related tasks. Trained on extensive chart-specific datasets, these chart-domain MLLMs (Xia et al., 2024; Zhang et al., 2024b) have achieved superior performance across various chart-related tasks. However, existing studies typically describe charts in natural language, which inevitably overlooks the dense information embedded within them, leading to inefficiencies in analysis and understanding. In contrast, code serves as a lossless representation of charts, offering a more effective and expressive approach to capturing chart information, thereby facilitating the solution of various chart-related tasks.

2.2 MLLMs For Code

Multimodal code generation has recently garnered much more attention. Several works, such as MMCode (Li et al., 2024b) and HumanEval-V (Zhang
et al., 2024a), have been developed to evaluate the capability of MLLMs in solving code problems that incorporate visual elements. Design2Code (Si et al., 2024) and Web2Code (Yun et al., 2024) evaluate the performance of MLLMs by focusing on code generation for HTML web page creation. Among the emerging tasks in this domain, chart-to-code generation has attracted significant interest as the visual elements of charts are more complex. This task challenges MLLMs to generate code that accurately reproduces a given chart or visual representation. Recent works (Zhang et al., 2024d; Zhao et al., 2025) like ChartMimic (Shi et al., 2024) evaluate the reasoning ability of MLLMs in this context. Similarly, Plot2Code (Wu et al., 2024) and ChartX (Xia et al., 2024) also evaluate MLLMs code generation ability, especially for text and data reproducibility. To the best of our knowledge, no dedicated research has focused on solving the chart-to-code generation problem. Our work is the first to attempt to address this challenge.

3 Chart2Code-160k Dataset

3.1 Direct Chart-to-code Generation

Despite the availability of many chart-reasoning instruction-tuning datasets, there is a notable lack of datasets specifically for chart-to-code tasks. Compared to chart reasoning data, chart-to-code data have the following distinct characteristics: (1) One-to-One Mapping: Unlike chart reasoning datasets, which could derive multiple question-answer pairs from a single chart, chart-to-code datasets require a one-to-one correspondence, demanding a large number of chart images for training. (2) Diversity Reflect on Charts: Unlike the diversity of chart reasoning data, which can be reflected in instructions, the diversity of chart-to-code data primarily lies in the variety of chart types and structures. (3) Syntax Constraints: Unlike the flexible natural language answers in chart reasoning tasks, the output code must strictly adhere to programming syntax to ensure executability.

Therefore, collecting a large number of chart-code pairs that meet the above requirements is challenging. Recent studies have demonstrated the feasibility of generating code with LLMs (Xu et al., 2023; Zhang et al., 2024c). Leveraging the one-to-one mapping property of chart-to-code data, we generate code first and execute it to produce the corresponding charts. In this way, we construct the first large-scale and diverse chart-to-code dataset, named Chart2Code-160k.

<table>
  <tr>
    <th>Dataset</th>
    <th>Train/Eval</th>
    <th>Chart Type</th>
    <th>Number</th>
  </tr>
  <tr>
    <td>ChartX</td>
    <td>Eval</td>
    <td>18</td>
    <td>6k</td>
  </tr>
  <tr>
    <td>Plot2Code</td>
    <td>Eval</td>
    <td>6</td>
    <td>132</td>
  </tr>
  <tr>
    <td>ChartMimic</td>
    <td>Eval</td>
    <td>22</td>
    <td>2.4k</td>
  </tr>
  <tr>
    <td>ChartLlama</td>
    <td>Train</td>
    <td>10</td>
    <td>7.8k</td>
  </tr>
  <tr>
    <td>Chart2Code-160k</td>
    <td>Train</td>
    <td>27</td>
    <td>160k</td>
  </tr>
</table>

Table 1: Comparisons of existing chart-to-code datasets.

Specifically, we generate chart-to-code data through the following steps: First, we prompt the LLM to generate keywords within a specific domain and guide it to generate simulated data related to these keywords. Then, to ensure the diversity of chart types, we identify 27 commonly used chart types and manually write 79 template codes for each as in-context demonstrations. These template codes contain almost all common chart formats. We further provide available functions such as plt.text() and parameters such as hatch='/' to encourage the generation of more diverse functions and parameters, resulting in the chart structures more diversely. To enhance the generality of generated code, LLMs are encouraged to use standard libraries such as Matplotlib and Seaborn. Additionally, we explicitly define all parameters within the code itself, eliminating the need for external files such as CSVs. This ensures that the code can be executed directly and accurately to represent the chart details. The final step involved executing the generated code to produce the corresponding chart. We utilize the above process to yield 200k code snippets for charts. After executing the code and filtering out problematic charts, such as those with excessive pixels or ticks, we construct a high-quality dataset of 160k diverse chart-to-code pairs. These pairs are formatted as multimodal instruction-tuning samples in the unified structure of <chart, instruction, code>.

3.2 Step-by-step Chart-to-code Generation

Although the dataset described above includes various chart types and structures, most of the generated code follows a similar template, with only certain details (such as colors and values) providing the essential distinguishing information. This can cause chart-to-code generation models to overlook these critical details and thus produce incomplete or incorrect results. To address the above challenge and further improve the reasoning ability of MLLMs, we propose the Snippet-of-Thought (SoT) method to expand direct chart-to-code generation
Figure 2: Illustration of Chat2Code dataset generation process and the ChartCoder training process. The dataset generation process is divided into two stages: direct generation and step-by-step generation. In the step-by-step generation, the code processed by the Snippet-of-Thought method is sampled from the Chart2Code-160k generated in the direct generation process. The training process of the ChartCoder also consists of two stages: alignment and instruction tuning.

into step-by-step generation formats, which has demonstrated effectiveness in text-to-code generation tasks (Zheng et al., 2023; Luo et al., 2024).

Specifically, we adopt the SoT to imitate the human reasoning process, deriving the final code step by step. This process is divided into four steps: Step 1: Generate the chart type and layout, such as plt.bar() and plt.subplot(). Step 2: Generate the data and corresponding colors used in the chart, such as data=[10, 15] and colors=['#FF0000', '#00FF00']. Step 3: Generate critical details of the chart, such as hatch='/' and loc='upper left'. Step 4: Generate the complete and final code. Different from CoT and PoT, we incorporate textual explanations and code snippets for each step to emphasize key information enhance the reasoning process and produce comprehensive outputs.

However, directly instructing the LLM to generate step-by-step code may lead to hallucinations, causing inconsistencies between intermediate code snippets and the final executable code. To maintain consistency among code snippets, we reformulated the step-by-step code data generation into a two-step process involving code generation and decomposition. We sample 50k chart-code pairs from the previously generated 160k data pairs and encourage the LLM to decompose the original code into the required textual explanations and code snippets of Steps 1–3, then concatenate the complete code in Step 4. To further mitigate hallucinations, such as undefined values or parameters in Steps 1 and 2, we used placeholder or default parameters during code decomposition to ensure the construction of consistent and reliable step-by-step code.

3.3 Dataset Analysis

Chart2Code-160k dataset provides three key advantages: (1) The First Large-Scale Dataset: It contains 160k data pairs for instruction tuning, significantly surpassing the size of previous datasets. (2) Diverse Chart Structures and Types: It includes 27 different chart types, with diverse structures enabled by a wide variety of functions and parameters in the code. (3) Syntactically Correct and Executable Code: All corresponding code is syn-
<table>
  <tr>
    <th>Dataset</th>
    <th>Source</th>
    <th colspan="2">Chart Quality</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Mean \( \mu \)</th>
    <th>SD \( \sigma \)</th>
  </tr>
  <tr>
    <td>Chart2Code-160k sample</td>
    <td>Generated</td>
    <td>77.32</td>
    <td>4.04</td>
  </tr>
  <tr>
    <td>ChartMimic testmini</td>
    <td>Real-world</td>
    <td>78.96</td>
    <td>3.96</td>
  </tr>
</table>

Table 2: Quantitative evaluation of the chart quality, comparing with real-world charts. SD is the abbreviation version for standard deviation.

tactically correct and executable, with explicitly defined parameters that ensure precise alignment between chart structures and code representations.

The comparisons of Chart2Code-160k with relevant chart-to-code datasets are listed in Table 1. To ensure data quality, we randomly sample around 1k instances and evaluate the quality of the chart images manually during the dataset construction period. Given the strong generation capabilities of LLM, we reckon the generated charts are suitable for training purposes. Furthermore, to quantitatively evaluate the chart quality, we also sample 8k data pairs (5% of the total) from Chart2Code-160k and utilize gpt-4o-2024-08-06 to evaluate them on four criteria: Aesthetics, Readability, Reproducibility, and Data Presentation Simplicity. The results in Table 2 show that the overall scores are broadly the same as real-world charts in Chart-Mimic. The detailed prompt is in the Figure 8. Our proposed Chart2Code-160k fills the gap between chart and code, equipping the model with advanced capabilities for downstream chart tasks.

4 ChartCoder Model

After constructing the Chart2Code-160k, we aimed to leverage the data to enhance the capacities of MLLMs to generate code from charts. Unlike previous methods that rely on general LLMs with a low proportion of code in their training corpus, we pioneer the use of Code LLMs to enhance the coding abilities of MLLMs from scratch.

4.1 Model Architecture

Following the standard architecture of MLLMs, ChartCoder consists of three modules: a pre-trained vision encoder (SigLIP-384 (Zhai et al., 2023a)), a vision-language connector (two-layer MLP) and a Code LLM backbone (DeepSeek Coder 6.7B (Guo et al., 2024)). The vision encoder extracts the input image into visual features, and the connector projects it into the word embedding space. LLM backbone then combines visual and textual features to generate responses.

Previous works emphasize the importance of high-resolution input for chart understanding (Liu et al., 2024; Guo et al., 2025), as details like textual words may lost in low-resolution images. However, vision transformers (ViTs) like CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023b) are constrained to resolutions of 224^2 and 384^2 respectively, which limits their capacities to encode chart images with sufficient detail. To address this, we utilize the Any Resolution strategy (Liu et al., 2024) to resize and patchify chart images to ensure ChartCoder processes high-resolution chart images effectively. Specifically, the input chart image is first resized to a pre-defined optimal aspect ratio, whose height and width are integer multiples of the image resolution. The resized image is then divided into patches of standard resolution and concatenated with a directly downsampled version of the image. This approach preserves both general and detailed information without requiring the original high-resolution image to be resized into a standard square, thereby avoiding the loss of fine details. Details are shown in Figure 2.

4.2 Model Training

Since we propose to use Code LLMs as the language backbone to enhance the code abilities of MLLMs, existing models do not meet our requirements as their backbones are general LLMs. Thus, to align charts with text and perform supervised fine-tuning for chart-to-code tasks, we adopt the following two-stage training process.

Chart-to-text Alignment. The alignment process aims to endow the model with chart structure perception capability. In this stage, we freeze the language and vision encoder models and pre-train the vision-language connector (Liu et al., 2023c). We collect and filter public chart corpora for alignment, which contains multiple tasks like chart caption and chart-to-table. Specifically, we use the following corpora: (1) UniChart (Masry et al., 2023), (2) Chart-to-Text (Kantharaj et al., 2022), (3) SciCap (Hsu et al., 2021), and (4) SciCap+ (Yang et al., 2024). Additionally, we incorporate the LLaVA pre-training dataset (Liu et al., 2023c) and our proposed Chart2Code-160k to achieve a more balanced coverage of concepts.

Chart-to-code Instruction-tuning. The second stage focuses on enhancing the model’s capabilities in chart-to-code tasks. In this stage, all three modules are jointly fine-tuned with our proposed Chart2Code-160k, and additional code-related data,
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Params</th>
    <th colspan="3">ChartMimic</th>
    <th colspan="3">Plot2Code</th>
    <th rowspan="2">ChartX</th>
  </tr>
  <tr>
    <th>Exec.Rate</th>
    <th>Low-Level</th>
    <th>High-Level</th>
    <th>Pass Rate</th>
    <th>Text-Match</th>
    <th>Rating</th>
    <th>GPT-score</th>
  </tr>
  <tr>
    <td>Full score</td>
    <td>-</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>10</td>
    <td>5</td>
  </tr>
  <tr>
    <th colspan="9">Proprietary</th>
  </tr>
  <tr>
    <td>GeminiProVision</td>
    <td>-</td>
    <td>68.2</td>
    <td>53.8</td>
    <td>53.3</td>
    <td>68.2</td>
    <td>53.6</td>
    <td>3.69</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Claude-3-opus</td>
    <td>-</td>
    <td>83.3</td>
    <td>60.5</td>
    <td>60.1</td>
    <td>84.1</td>
    <td>57.5</td>
    <td>3.80</td>
    <td>-</td>
  </tr>
  <tr>
    <td>GPT-4V</td>
    <td>-</td>
    <td>91.2</td>
    <td>76.4</td>
    <td>78.9</td>
    <td>84.1</td>
    <td>57.7</td>
    <td>5.58</td>
    <td>2.63</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>-</td>
    <td>93.2</td>
    <td>79.0</td>
    <td>83.5</td>
    <td>88.6</td>
    <td>56.5</td>
    <td>5.71</td>
    <td>-</td>
  </tr>
  <tr>
    <th colspan="9">Open-Source General-Domain</th>
  </tr>
  <tr>
    <td>DeepSeek-VL-7B</td>
    <td>7.3B</td>
    <td>41.3</td>
    <td>19.0</td>
    <td>17.6</td>
    <td>64.4</td>
    <td>32.6</td>
    <td>2.26</td>
    <td>-</td>
  </tr>
  <tr>
    <td>LLaVA-Next-Mistral-7B</td>
    <td>7.6B</td>
    <td>59.7</td>
    <td>20.7</td>
    <td>21.3</td>
    <td>72.0</td>
    <td>38.7</td>
    <td>2.87</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>7.0B</td>
    <td>67.0</td>
    <td>32.9</td>
    <td>35.0</td>
    <td>68.2</td>
    <td>33.8</td>
    <td>3.10</td>
    <td>1.50</td>
  </tr>
  <tr>
    <td>InternVL2-4B</td>
    <td>4.2B</td>
    <td>66.2</td>
    <td>33.8</td>
    <td>38.4</td>
    <td>66.3</td>
    <td>33.4</td>
    <td>2.52</td>
    <td>1.57</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>8.1B</td>
    <td>61.8</td>
    <td>34.4</td>
    <td>38.9</td>
    <td>77.3</td>
    <td>37.1</td>
    <td>2.78</td>
    <td>1.63</td>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V2.5</td>
    <td>8.4B</td>
    <td>80.3</td>
    <td>36.6</td>
    <td>42.1</td>
    <td>76.3</td>
    <td>37.3</td>
    <td>2.61</td>
    <td>1.66</td>
  </tr>
  <tr>
    <td>InternVL2-26B</td>
    <td>26.0B</td>
    <td>69.3</td>
    <td>41.4</td>
    <td>47.4</td>
    <td>81.3</td>
    <td>43.1</td>
    <td>3.42</td>
    <td>1.70</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td>
    <td>72.0B</td>
    <td>73.3</td>
    <td>54.4</td>
    <td>50.9</td>
    <td>72.0</td>
    <td>53.4</td>
    <td>4.26</td>
    <td>1.69</td>
  </tr>
  <tr>
    <td>InternVL2-Llama3-76B</td>
    <td>76.0B</td>
    <td>83.2</td>
    <td>54.8</td>
    <td>62.2</td>
    <td>85.6</td>
    <td>46.6</td>
    <td>3.89</td>
    <td>1.74</td>
  </tr>
  <tr>
    <th colspan="9">Open-Source Chart-Domain</th>
  </tr>
  <tr>
    <td>ChartLlama</td>
    <td>13B</td>
    <td>57.5</td>
    <td>24.8</td>
    <td>28.1</td>
    <td>58.4</td>
    <td>40.3</td>
    <td>2.32</td>
    <td>0.94</td>
  </tr>
  <tr>
    <td>ChartAssistant</td>
    <td>13B</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>TinyChart</td>
    <td>3B</td>
    <td>42.5</td>
    <td>26.3</td>
    <td>25.9</td>
    <td>43.2</td>
    <td>44.6</td>
    <td>2.19</td>
    <td>1.89</td>
  </tr>
  <tr>
    <td>ChartVLM-L</td>
    <td>14.3B</td>
    <td>19.5</td>
    <td>15.8</td>
    <td>13.9</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
    <td>1.58</td>
  </tr>
  <tr>
    <td>ChartCoder (Ours)</td>
    <td>7.0B</td>
    <td><b>91.4</b></td>
    <td><b>77.4</b></td>
    <td><b>74.0</b></td>
    <td><b>87.9</b></td>
    <td><b>54.5</b></td>
    <td><b>4.50</b></td>
    <td><b>2.09</b></td>
  </tr>
</table>

Table 3: Evaluation results of various baseline models. Unless otherwise specified, we directly use the results in the relevant benchmarks. We evaluate models that are not reported in those benchmarks. The best performances of open-source MLLMs are indicated in bold.

<table>
  <tr>
    <th>Model</th>
    <th>Chart Types</th>
    <th>Layout</th>
    <th>Text Content</th>
    <th>Data</th>
    <th>Style</th>
    <th>Clarity</th>
  </tr>
  <tr>
    <td>Full score</td>
    <td>20</td>
    <td>10</td>
    <td>20</td>
    <td>20</td>
    <td>20</td>
    <td>10</td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>18.96</td>
    <td>9.59</td>
    <td>17.16</td>
    <td>15.68</td>
    <td>14.66</td>
    <td>8.84</td>
  </tr>
  <tr>
    <td>InternVL2-Llama3-76B</td>
    <td>13.06</td>
    <td>8.44</td>
    <td>12.59</td>
    <td>10.51</td>
    <td>8.74</td>
    <td>7.87</td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td>
    <td>10.45</td>
    <td>7.83</td>
    <td>9.92</td>
    <td>8.14</td>
    <td>7.10</td>
    <td>7.47</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>7.20</td>
    <td>6.82</td>
    <td>8.81</td>
    <td>5.74</td>
    <td>5.42</td>
    <td>6.64</td>
  </tr>
  <tr>
    <td>TinyChart</td>
    <td>4.16</td>
    <td>5.06</td>
    <td>5.22</td>
    <td>2.74</td>
    <td>3.21</td>
    <td>5.58</td>
  </tr>
  <tr>
    <td>ChartVLM-L</td>
    <td>0.97</td>
    <td>3.53</td>
    <td>2.48</td>
    <td>0.81</td>
    <td>0.90</td>
    <td>5.25</td>
  </tr>
  <tr>
    <td>ChartCoder (Ours)</td>
    <td><b>16.83</b></td>
    <td><b>9.13</b></td>
    <td><b>14.77</b></td>
    <td><b>12.41</b></td>
    <td><b>12.68</b></td>
    <td><b>8.29</b></td>
  </tr>
</table>

Table 4: Detailed results of high-level scores on ChartMimic Direct Mimic task. All the subscores of ChartCoder are close to GPT-4o.

such as ChartQA PoT (Zhang et al., 2024b) and ChartLlama chart-to-chart (Han et al., 2023).

5 Experiments

5.1 Baselines and Benchmarks

We compare ChartCoder with existing models in three setups (1) General-domain open-source MLLMs including InternVL2(4B, 8B, 26B, 76B) (Chen et al., 2024), Qwen2-VL(7B, 72B) (Wang et al., 2024a), DeepSeek-VL-7B (Lu et al., 2024), LLaVA-Next(7B) (Li et al., 2024a) and MiniCPM-Llama3-V2.5 (Yao et al., 2024). (2) Proprietary models include GeminiProVision (Team et al., 2023), Claude-3-opus (Anthropic, 2024), GPT-4V (OpenAI, 2023), and GPT-4o (OpenAI, 2024). (3) Chart-domain MLLMs including ChartLlama (Han et al., 2023), ChartAssisstant (Meng et al., 2024), Tinychart (Zhang et al., 2024b) and ChartVLM (Xia et al., 2024). All the methods are evaluated on the benchmarks ChartMimic (Shi et al., 2024), Plot2Code (Wu et al., 2024) and ChartX (Xia et al., 2024). We revise the Rating calculation in Plot2Code. The original evaluation only considers charts corresponding to executable code, which leads to higher ratings for only generating simple charts. We calculate all the results, which better reflect the impact of complex charts. For all methods, the zero-shot setting was adopted during the evaluation. Details about these benchmarks are shown in the Appendix A.2.
<table>
  <tr>
    <th>Model</th>
    <th colspan="4">Text</th>
    <th colspan="4">Layout</th>
    <th colspan="4">Type</th>
    <th colspan="4">Color</th>
  </tr>
  <tr>
    <td></td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Full score</td>
    <td>81.5</td>
    <td>89.8</td>
    <td>77.3</td>
    <td>67.2</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>GPT-4o</td>
    <td>81.5</td>
    <td>89.8</td>
    <td>77.3</td>
    <td>67.2</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>InternVL2-Llama3-76B</td>
    <td>54.1</td>
    <td>74.5</td>
    <td>49.2</td>
    <td>41.5</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2-VL-72B</td>
    <td>43.2</td>
    <td>80.5</td>
    <td>54.6</td>
    <td>39.4</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>31.5</td>
    <td>51.1</td>
    <td>28.6</td>
    <td>26.2</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>TinyChart</td>
    <td>9.8</td>
    <td>48.2</td>
    <td>32.9</td>
    <td>14.2</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>ChartVLM-L</td>
    <td>7.7</td>
    <td>33.7</td>
    <td>17.6</td>
    <td>5.2</td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td><b>ChartCoder (Ours)</b></td>
    <td><b>67.2</b></td>
    <td><b>95.0</b></td>
    <td><b>78.5</b></td>
    <td><b>69.0</b></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 5: Detailed results of low-level scores on ChartMimic Direct Mimic task. Three out of four subscores of ChartCoder are even higher than GPT-4o.

5.2 Main Results

As indicated in Table 3 ChartCoder achieves the best performance among open-source MLLMs in all the chart-to-code tasks and even better than some proprietary models. Notably, on the most challenging ChartMimic task, ChartCoder surpasses leading small-scale general-domain MLLMs (<20B) such as MiniCPM-Llama3-V2.5 and InternVL2-8B with average scores of 26.7 and 34.6 respectively. The improvement achieved by ChartCoder highlights the effectiveness of our proposed Code LLM as the language backbone, combined with the Chart2Code-160k dataset, in enabling MLLMs to excel in chart understanding and code generation tasks. In addition, ChartCoder also performs better than existing state-of-the-art large-scale MLLMs such as InternVL2-Llama3-76B and chart-domain MLLMs such as TinyChart.

We further illustrate the detailed high-level and low-level scores for the ChartMimic benchmark. The high-level score utilizes GPT-4o to evaluate the detailed similarity between the ground truth and generated chart images in six aspects: chart types, layout, text content, data, style, and clarity. The low-level score is calculated based on a comparison between the ground truth and the generated code, focusing on the code similarities in four aspects: text, layout, type, and color.

Table 4 denotes the high-level results. ChartCoder is the model most comparable to GPT-4o, as the evaluations were conducted by GPT-4o itself, suggesting the actual performance gap may not be as pronounced as it appears. Notably, ChartCoder shows the largest gap with GPT-4o in the data category, which highlights the complexity of extracting numerical values from charts, aligning with conclusions from existing chart understanding benchmarks: current MLLMs struggle to directly and accurately extract complete data from complex charts (Wang et al., 2024c; Zhang et al., 2024b).

Table 5 shows the low-level results. ChartCoder even slightly outperforms GPT-4o in layout, type and color, highlighting the diversity of our proposed Chart2Code-160k dataset. However, the text score of the ChartCoder is lower than GPT-4o, which is similar to the results of high-level scores. We believe this is due to the lack of specialized chart OCR-oriented training for our model. Nevertheless, our text accuracy still surpasses that of open-source models, indicating the effectiveness of our proposed ChartCoder model and Chart2Code-160k dataset. We further present some case studies on ChartMimic and compare ChartCoder with existing MLLMs. The results are shown in Figure 3, the outputs of ChartCoder are much more similar to the ground truth chart than open-source models.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">ChartMimic</th>
  </tr>
  <tr>
    <th>Exec.Rate</th>
    <th>Low-Level</th>
    <th>High-Level</th>
  </tr>
  <tr>
    <td>ChartCoder</td>
    <td>91.4</td>
    <td>77.4</td>
    <td>74.0</td>
  </tr>
  <tr>
    <th colspan="4">Code LLM → General LLM</th>
  </tr>
  <tr>
    <td>DeepSeek LLM</td>
    <td>80.6</td>
    <td>61.4</td>
    <td>63.4</td>
  </tr>
  <tr>
    <td>△</td>
    <td>-10.8</td>
    <td>-16.0</td>
    <td>-10.6</td>
  </tr>
  <tr>
    <th colspan="4">Different Visual Encoders</th>
  </tr>
  <tr>
    <td>CLIP-336</td>
    <td>91.6</td>
    <td>77.3</td>
    <td>70.3</td>
  </tr>
  <tr>
    <td>△</td>
    <td>+0.2</td>
    <td>-0.1</td>
    <td>-3.7</td>
  </tr>
  <tr>
    <th colspan="4">Without Step-by-step Generation</th>
  </tr>
  <tr>
    <td>w/o SoT</td>
    <td>89.2</td>
    <td>70.1</td>
    <td>65.4</td>
  </tr>
  <tr>
    <td>△</td>
    <td>-2.2</td>
    <td>-7.3</td>
    <td>-8.6</td>
  </tr>
  <tr>
    <th colspan="4">Open-source MLLM Finetuned on Chart2Code-160k</th>
  </tr>
  <tr>
    <td>Qwen2-VL-7B</td>
    <td>67.0</td>
    <td>32.9</td>
    <td>35.0</td>
  </tr>
  <tr>
    <td>Finetuned Model</td>
    <td>83.6</td>
    <td>73.4</td>
    <td>68.2</td>
  </tr>
  <tr>
    <td>△</td>
    <td>+16.7</td>
    <td>+40.5</td>
    <td>+33.2</td>
  </tr>
</table>

Table 6: The ablation studies on model architecture and data. The results show that the effectiveness of our proposed code LLM backbone and dataset.

5.3 Ablation Study

We perform extensive ablation experiments to validate the effectiveness of our proposed model and dataset. We divide the ablation study into three parts, and the results are shown in Table 6. (1) <i>Code or general LLMs.</i> To investigate whether employing Code LLMs as language backbone provides specific advantages in chart-to-code tasks and identify the nature of these potential benefits, we replace the Code LLM, DeepSeek LLM 6.7B, with general LLM, DeepSeek LLM 7B (Bi et al., 2024a), maintaining the same two-stage training procedures. The result shows that compared with general LLM, utilizing code LLM as the language
Figure 3: Generated charts of different model outputs after code execution. Our proposed ChartCoder performs significantly better than InternVL2-8B of a similar model scale.

<table>
  <tr>
    <th>Model</th>
    <th>Image</th>
    <th>Image+Code</th>
    <th>△</th>
  </tr>
  <tr>
    <td>MiniCPM-Llama3-V2.5</td>
    <td>0.76</td>
    <td>0.81</td>
    <td>6.5%</td>
  </tr>
  <tr>
    <td>InternVL2-8B</td>
    <td>0.79</td>
    <td>0.82</td>
    <td>3.8%</td>
  </tr>
</table>

Table 7: Comparison of the impact of using code as auxiliary contexts on the MMC True/False task.

backbone could significantly improve the execution rate, as well as the low-level and high-level scores. We further analyze the types of errors in the code that failed to execute and find that utilizing code LLMs significantly reduces syntax errors like missing closing quotation marks and type errors like incorrect argument type. (2) Resolution of vision encoders. Previous studies have indicated that performance on chart understanding tasks is resolution-dependent, with lower resolutions negatively impacting model performance (Liu et al., 2024). To verify whether resolution affects chart-to-code tasks, we replace SigLIP-384 with CLIP-336 and maintain the other setting. The result shows that the resolution of the vision encoder generally does not affect the output code execution rate but slightly influences the high-level chart similarity. Through our analysis, we find that, similar to the challenges in chart understanding, this issue is caused by the negative impact of low resolution on the recognition of text and special symbols. However, as we utilize the Any Resolution strategy, this impact has been reduced significantly.

(3) Dataset effectiveness. We design two scenarios to illustrate our proposed Chart2Code-160k dataset. Firstly, to evaluate our proposed SoT method to emphasize the critical information in the chart, we remove the 50k step-by-step generation data and train the model using only the direct generation data. The result shows it influences the low-level and high-level scores notably, especially in text content and data, which shows the role of emphasising critical information. Secondly, we select Qwen2-VL-7B as the baseline of open-source MLLM and directly fine-tune it on our proposed Chart2Code-160k datasets. The result illustrates that after fine-tuning, the performance improves significantly on all the metrics, demonstrating the effectiveness of Chart2Code-160k.

5.4 Analysis

We further evaluate the role of code in the chart understanding task. We use two MLLMs to evaluate two input forms, Image only and Image with Code, on the MMC True/False benchmark (Liu et al., 2023a). The result in Table 7 shows that using code helps the model better understand chart details, especially the chart types and the data they contain. A case study is shown in Figure 5. Also, we utilize LLM to evaluate the readability of ChartCoder output code, and details are in the Appendix A.4.
6 Conclusion

This work aims to tackle the challenge of chart-to-code tasks with MLLMs. First, we propose the ChartCoder, which utilizes Code LLM as the language backbone dedicated to chart-to-code tasks. Second, to solve the scarcity of chart-to-code data, we present the first large-scale and diverse chart-to-code dataset, Chart2Code-160k. Finally, to emphasize the key information, we propose the Snippet-of-Thought (SoT) method to generate step-by-step data. Experiments show that ChartCoder outperforms existing open-source MLLMs.

Limitation

Our study is comprehensive but has certain limitations we aim to address in future research. Due to constraints in computational resources, we only trained ChartCoder with 7B parameters, which has demonstrated sufficiently good results for now. A larger model could potentially achieve even better performance. Future work may explore more complex and diverse charts and codes while experimenting with other image types like HTML to develop a comprehensive multi-modal code large language model.

Ethical Statement

Our research employs publicly available models and datasets with proper citations. This approach minimizes the risk of generating toxic content, leveraging the widely used and non-toxic nature of our datasets and prompts.

Acknowledgment

The work is initiated and supported by the AI9Stars Team.

References

Anthropic. 2024. Introducing the next generation of claude.

DeepSeek-Al Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Dama Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qushui Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wen-Hui Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, Aixin Liu, Bo Liu (Benjamin Liu), Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xi-aotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Jun-Mei Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Min Tang, Bing-Li Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Yu Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yi Xiong, Hanwei Xu, Ronald X Xu, Yanhong Xu, Dejian Yang, Yu mei You, Shuiying Yu, Xin yuan Yu, Bo Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghu Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qiha Zhu, and Yuheng Zou. 2024a. Deepseek llm: Scaling open-source language models with longtermism. ArXiv, abs/2401.02954.

Jinhe Bi, Yifan Wang, Danqi Yan, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2025. Prism: Self-pruning intrinsic selection method for training-free multimodal data selection. arXiv preprint arXiv:2502.12119.

Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. 2024b. Visual instruction tuning with 500x fewer parameters through modality linear representation-steering. arXiv preprint arXiv:2412.12359.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024. Intervl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198.

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, et al. 2024. Deepseek-coder: When the large language model meets programming—the rise of code intelligence. arXiv preprint arXiv:2401.14196.

Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. 2025. Llava-uhd: an lmm perceiving any aspect ratio and high-resolution images. In European Conference on Computer Vision, pages 390–406. Springer.

Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. 2023. Chartlama: A multimodal llm for chart understanding and generation. arXiv preprint arXiv:2311.16483.

Ting-Yao Hsu, C Lee Giles, and Ting-Hao’Kenneth’ Huang. 2021. Sicap: Generating captions for scientific figures. arXiv preprint arXiv:2110.11624.

Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. 2022. Chart-to-text: A large-scale benchmark for chart summarization. arXiv preprint arXiv:2203.06486.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. 2024a. Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895.

Kaixin Li, Yuchen Tian, Qisheng Hu, Ziyang Luo, Zhizhong Huang, and Jing Ma. 2024b. Mmcode: Benchmarking multimodal large language models for code generation with visually rich programming problems. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 736–783.

Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. 2023a. Mmc: Advancing multimodal chart understanding with large-scale instruction tuning. arXiv preprint arXiv:2311.10774.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023b. Improved baselines with visual instruction tuning.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024. Llava-next: Improved reasoning, ocr, and world knowledge.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023c. Visual instruction tuning.

Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. 2024. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.05525.

Xianzhen Luo, Qingfu Zhu, Zhiming Zhang, Libo Qin, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. 2024. Python is not always the best choice: Embracing multilingual program of thoughts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7185–7212.

Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Enamul Hoque, and Shafiq Joty. 2023. Unichart: A universal vision-language pretrained model for chart comprehension and reasoning. arXiv preprint arXiv:2305.14761.

Fanjing Meng, Wenqi Shao, Quanfeng Lu, Peng Gao, Kaipeng Zhang, Yu Qiao, and Ping Luo. 2024. Char-tassistant: A universal chart multimodal language model via chart-to-table pre-training and multitask instruction tuning. arXiv preprint arXiv:2401.02384.

Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1527–1536.

OpenAI. 2023. Gpt-4v(ision) system card.

OpenAI. 2024. Gpt-4o. Accessed: 2024-05-13.

Qwen Team. 2024. Qwen2.5: A party of foundation models.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR.

Chufan Shi, Cheng Yang, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, et al. 2024. Chartmimic: Evaluating lmm’s cross-modal reasoning capability via chart-to-code generation. arXiv preprint arXiv:2406.09961.

Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Difyi Yang. 2024. Design2code: How far are we from automating front-end engineering? arXiv preprint arXiv:2403.03163.

Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. 2019. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhi-hao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024a. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191.

Wenhai Wang, Zhe Chen, XiaoKang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. 2024b. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36.

Zirui Wang, Mengzhou Xia, Luxi He, Howard Chen, Yitao Liu, Richard Zhu, Kaiqu Liang, Xindi Wu, Haotian Liu, Sadhika Malladi, et al. 2024c. Charxiv: Charting gaps in realistic chart understanding in multimodal llms. arXiv preprint arXiv:2406.18521.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. 2024. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots. arXiv preprint arXiv:2405.07990.
Renjiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, et al. 2024. Chartx & chartvilm: A versatile benchmark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244.

Zhishen Yang, Raj Dabre, Hideki Tanaka, and Naoaki Okazaki. 2024. Scicap+: A knowledge augmented dataset to study the challenges of scientific figure captioning. Journal of Natural Language Processing, 31(3):1140–1165.

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.

Yahan Yu, Duzhen Zhang, Yong Ren, Xuanle Zhao, Xiuyi Chen, and Chenhui Chu. 2025. Progressive lora for multimodal continual instruction tuning. In Findings of the Association for Computational Linguistics ACL 2025.

Sukmin Yun, Haokun Lin, Rusiru Thushara, Mohammad Qazim Bhat, Yongxin Wang, Zutao Jiang, Mingkai Deng, Jinzhong Wang, Tianhua Tao, Junbo Li, et al. 2024. Web2code: A large-scale webpage-to-code dataset and evaluation framework for multi-modal llms. arXiv preprint arXiv:2406.20098.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023a. Sigmoid loss for language image pre-training. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11941–11952.

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023b. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986.

Duzhen Zhang, Yong Ren, Zhong-Zhi Li, Yahan Yu, Jiuhua Dong, Chenxing Li, Zhilong Ji, and Jinfeng Bai. 2025. Enhancing multimodal continual instruction tuning with branchlora. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

Fengji Zhang, Linquan Wu, Huiyu Bai, Guancheng Lin, Xiao Li, Xiao Yu, Yue Wang, Bei Chen, and Jacky Keung. 2024a. Humaneval-v: Evaluating visual understanding and reasoning abilities of large multi-modal models through coding tasks. arXiv preprint arXiv:2410.12381.

Liang Zhang, Anwen Hu, Haiyang Xu, Ming Yan, Yichen Xu, Qin Jin, Ji Zhang, and Fei Huang. 2024b. Tinychart: Efficient chart understanding with visual token merging and program-of-thoughts learning. arXiv preprint arXiv:2404.16635.

Wenqi Zhang, Zhenglin Cheng, Yuanyu He, Mengna Wang, Yongliang Shen, Zeqi Tan, Guiyang Hou, Mingqian He, Yanna Ma, Weiming Lu, et al. 2024c. Multimodal self-instruct: Synthetic abstract image and visual reasoning instruction using language model. arXiv preprint arXiv:2407.07053.

Zhehao Zhang, Weicheng Ma, and Soroush Vosoughi. 2024d. Is gpt-4v (ision) all you need for automating academic data visualization? exploring vision-language models’ capability in reproducing academic charts. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8271–8288.

Xuanle Zhao, Xuexin Liu, Haoyue Yang, Xianzhen Luo, Fanhu Zeng, Jianling Li, Qi Shi, and Chi Chen. 2025. Chartedit: How far are mllms from automating chart analysis? evaluating mllms’ capability via chart editing. arXiv preprint arXiv:2505.11935.

Wenqing Zheng, S P Sharan, Ajay Jaiswal, Kevin Wang, Yihan Xi, Dejia Xu, and Zhangyang Wang. 2023. Outline, then details: Syntactically guided coarse-to-fine code generation. In International Conference on Machine Learning.
A Appendix

A.1 Implementation Details

In the data generation stage, we utilize gpt-4o-2024-08-06 as the LLM for both direct and step-by-step generation processes.

In the training stage, ChartCoder is initialized with SigLIP-384 (Radford et al., 2021) as the vision encoder and DeepSeek Coder 6.7B (Guo et al., 2024) as the large language model. The whole training process is divided into alignment and instruction tuning. During the alignment stage, we only train the vision-language connector with the chart-to-text alignment data. The learning rate is set to 1e-3. In the instruction tuning stage, we train the entire model for 1 epoch with a batchsize of 128. The learning rate of SigLIP and other modules are 5e-6 and 1e-5 respectively, with a warmup at the beginning of 3%, then decays to 0 at the end of training. The alignment and instruction tuning processes cost 12 and 5 hours on 32 Tesla A100 GPUs with 80 GB VRAMs.

A.2 Benchmark Details

ChartMimic (Shi et al., 2024) focuses on evaluating the ability of MLLMs to redraw charts from ArXiv papers, emphasizing the preservation of the original style and appearance. It consists of two subsets: testmini and test. Following the settings in the original paper, we adopt the Direct Mimic task on the testmini subset as the default evaluation standard, reporting execution success rates alongside low-level and high-level scores.

Plot2Code (Wu et al., 2024) aims to evaluate models’ abilities to generate code corresponding to charts from the available Matplotlib galleries, with a focus on textual similarity. We evaluate models on its Direct Asking task using three metrics: Pass Rate, Text-Match, and Rating.

ChartX (Xia et al., 2024) contains various tasks with synthesis chart images, including Question Answering, Summarization, Description and Redrawing. We choose the Redrawing task and report the GPT score as the metrics in ChartX.

A.3 More Ablation Studies

We also perform more ablation studies on the language backbone and further choose Qwen2.5-7B and Qwen2.5 Coder-7B (Qwen Team, 2024) for comparison. The results also show that using Code LLM as the language backbone is better than using general LLM. However, we find that using the

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="3">ChartMimic</th>
  </tr>
  <tr>
    <th>Exec.Rate</th>
    <th>Low-Level</th>
    <th>High-Level</th>
  </tr>
  <tr>
    <td>ChartCoder</td>
    <td>91.4</td>
    <td>77.4</td>
    <td>74.0</td>
  </tr>
  <tr>
    <th colspan="4">Replace Language Backbone</th>
  </tr>
  <tr>
    <td>Qwen2.5</td>
    <td>88.1</td>
    <td>73.4</td>
    <td>67.9</td>
  </tr>
  <tr>
    <td>△</td>
    <td>-3.3</td>
    <td>-4.0</td>
    <td>-6.1</td>
  </tr>
  <tr>
    <td>Qwen2.5 Coder</td>
    <td>90.3</td>
    <td>76.8</td>
    <td>69.7</td>
  </tr>
  <tr>
    <td>△</td>
    <td>-1.1</td>
    <td>-0.6</td>
    <td>-4.3</td>
  </tr>
</table>

Table 8: The ablation studies on model architecture and data. The results show that the effectiveness of our proposed model architecture and dataset.

<table>
  <tr>
    <th>Dataset</th>
    <th>Source</th>
    <th>Code Readability</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th>Mean \( \mu \) SD \( \sigma \)</th>
  </tr>
  <tr>
    <td>Qwen2-VL-7B Output</td>
    <td>Model generated</td>
    <td>82.48 6.81</td>
  </tr>
  <tr>
    <td>ChartCoder Output</td>
    <td>Model generated</td>
    <td>85.22 6.78</td>
  </tr>
  <tr>
    <td>ChartMimic Source</td>
    <td>Human written</td>
    <td>87.66 4.30</td>
  </tr>
</table>

Table 9: Performance Comparison of model outputs and human-written sources. SD is the abbreviation ddfor standard deviation.

Qwen2.5 Coder as the backbone does not perform as well as using the DeepSeek Coder. This observation seems counterintuitive, as the official evaluation suggests that the performance of the Qwen2.5 Coder is better than the DeepSeek Coder. We analyze experimental results and find that the code generated by Qwen2.5 is more standardized. For instance, the DeepSeek Coder backbone tends to use ax[0], ax[1], while the Qwen2.5 Coder backbone prefers a more standardized approach, such as using for i in range(2): ax[i]. However, in some complex scenarios, using a for loop may lead to errors, such as ax[0] and ax[1] do not have same number of bars.

A.4 Output Code Analysis

To evaluate the output code readability, we conduct an ablation experiment, utilizing gpt-4o-2024-08-06 to evaluate the output code readability. We evaluate four aspects of the generated code, including Naming Conventions, Code Structure, Comments, and Logical Clarity, with a total score of 100. We choose the generated code from the ChartMimic task (ChartCoder output) and the ground truth code (human-annotated) in the ChartMimic dataset. The results are as shown in Table 9. We also evaluate the error types on ChartMimic direct generation tasks with code and general LLMs as the language backbone. The results are shown in Figure 4.
Figure 4: Comparison of error types on ChartMimic direct generation tasks with code and general LLMs as language backbone, respectively.

<table>
  <tr>
    <th>Type</th>
    <th>pie</th>
    <th>line</th>
    <th>bar</th>
    <th>bar_num</th>
  </tr>
  <tr>
    <td>Percent</td>
    <td>8.0%</td>
    <td>9.7%</td>
    <td>8.3%</td>
    <td>3.3%</td>
  </tr>
  <tr>
    <th>Type</th>
    <th>3d</th>
    <th>area</th>
    <th>box</th>
    <th>bubble</th>
  </tr>
  <tr>
    <td>Percent</td>
    <td>5.6%</td>
    <td>3.9%</td>
    <td>4.4%</td>
    <td>2.8%</td>
  </tr>
  <tr>
    <th>Type</th>
    <th>candlestick</th>
    <th>funnel</th>
    <th>heatmap</th>
    <th>multi-axes</th>
  </tr>
  <tr>
    <td>Percent</td>
    <td>2.8%</td>
    <td>2.7%</td>
    <td>3.9%</td>
    <td>3.8%</td>
  </tr>
  <tr>
    <th>Type</th>
    <th>rader</th>
    <th>ring</th>
    <th>pie</th>
    <th>rose</th>
  </tr>
  <tr>
    <td>Percent</td>
    <td>3.8%</td>
    <td>2.7%</td>
    <td>2.8%</td>
    <td>3.9%</td>
  </tr>
  <tr>
    <th>Type</th>
    <th>treemap</th>
    <th>violin</th>
    <th>scatter</th>
    <th>quiver</th>
  </tr>
  <tr>
    <td>Percent</td>
    <td>3.9%</td>
    <td>3.9%</td>
    <td>3.8%</td>
    <td>5.2%</td>
  </tr>
  <tr>
    <th>Type</th>
    <th>inset</th>
    <th>histogram</th>
    <th>graph</th>
    <th>error bar</th>
  </tr>
  <tr>
    <td>Percent</td>
    <td>1.2%</td>
    <td>1.2%</td>
    <td>1.2%</td>
    <td>1.6%</td>
  </tr>
  <tr>
    <th>Type</th>
    <th>error point</th>
    <th>density</th>
    <th>Combination</th>
    <th>Total</th>
  </tr>
  <tr>
    <td>Percent</td>
    <td>1.6%</td>
    <td>1.2%</td>
    <td>2.8%</td>
    <td>100%</td>
  </tr>
</table>

Table 10: Type distributions of the Chart2Code-160k instruction-tuning dataset.

A.5 Chart2Code-160k Analysis

We count the proportion of different charts in the Chart2Code-160k dataset in Table 10. Also, we utilize gpt -4o-2024-08-06 to evaluate the quality of the charts in the Chart2Code-160k and compare them with the real-world chart. The prompts are shown in Figure 8.
ChartCoder Output
< Code >
"python3 import matplotlib.pyplot as plt
# data
models = ['GPT4', 'GPT-few-shot', 'GPT3.5', 'StarCoder', 'Code Llama', 'MCL']
accuracy = [3.5, 3.8, 3.6, 3.8, 3.5, 2.9]
completeness = [3.5, 3.8, 3.6, 3.8, 3.5, 2.9]
consensus = [3.5, 3.8, 3.6, 3.8, 3.5, 2.9]
readability = [3.5, 3.8, 3.6, 3.8, 3.5, 2.9]
# Plotting
fig, ax = plt.subplots(figsize=(8.0, 5.0))
ax.barh(1, width=0.8, color='blue', label='Accuracy')
ax.barh(2, width=0.8, color='orange', label='Completeness')
ax.barh(3, width=0.8, color='red', label='Consensus')
ax.barh(4, width=0.8, color='green', label='Readability')
ax.set_xlabel('Ratings of Language Models on a Scale from 1-4')
ax.set_title('Ratings of Language Models on a Scale from 1-4')
ax.set_yticks([1, 2, 3, 4])
ax.set_yticklabels(['Accuracy', 'Completeness', 'Consensus', 'Readability'])
ax.legend()
plt.tight_layout()"
</ Code >

<table>
  <tr>
    <th>Model</th>
    <th>Accuracy</th>
    <th>Completeness</th>
    <th>Consensus</th>
    <th>Readability</th>
  </tr>
  <tr>
    <td>GPT4</td>
    <td>3.5</td>
    <td>3.5</td>
    <td>3.5</td>
    <td>3.5</td>
  </tr>
  <tr>
    <td>GPT-few-shot</td>
    <td>3.8</td>
    <td>3.8</td>
    <td>3.8</td>
    <td>3.8</td>
  </tr>
  <tr>
    <td>GPT3.5</td>
    <td>3.6</td>
    <td>3.6</td>
    <td>3.6</td>
    <td>3.6</td>
  </tr>
  <tr>
    <td>StarCoder</td>
    <td>3.8</td>
    <td>3.8</td>
    <td>3.8</td>
    <td>3.8</td>
  </tr>
  <tr>
    <td>Code Llama</td>
    <td>3.5</td>
    <td>3.5</td>
    <td>3.5</td>
    <td>3.5</td>
  </tr>
  <tr>
    <td>MCL</td>
    <td>2.9</td>
    <td>2.9</td>
    <td>2.9</td>
    <td>2.9</td>
  </tr>
</table>

Figure 5: A case study comparing the outputs of utilizing image and image+code as the inputs

<html>
  <body>
    <p>You are an expert Python developer who specializes in writing matplotlib code based on a given picture. I found a very nice picture in a STEM paper, but there is no corresponding source code available. I need your help to generate the Python code that can reproduce the picture based on the picture I provide. Note that it is necessary to use figsize=(9.0, 5.0) to set the image size to match the original size. Now, please give me the matplotlib code that reproduces the picture below.</p>
    <img src="page_1012_1012_728_384.png" alt="Bar chart showing ratings of language models">
  </body>
</html>

GPT-4o Output
import matplotlib.pyplot as plt
# Data
data = {
    'GPT4': {'Accuracy': 3.5, 'Completeness': 3.5, 'Consensus': 3.5, 'Readability': 3.5},
    'GPT-few-shot': {'Accuracy': 3.8, 'Completeness': 3.8, 'Consensus': 3.8, 'Readability': 3.8},
    'GPT3.5': {'Accuracy': 3.6, 'Completeness': 3.6, 'Consensus': 3.6, 'Readability': 3.6},
    'StarCoder': {'Accuracy': 3.8, 'Completeness': 3.8, 'Consensus': 3.8, 'Readability': 3.8},
    'Code Llama': {'Accuracy': 3.5, 'Completeness': 3.5, 'Consensus': 3.5, 'Readability': 3.5},
    'MCL': {'Accuracy': 2.9, 'Completeness': 2.9, 'Consensus': 2.9, 'Readability': 2.9}
}
# Plotting
fig, ax = plt.subplots(figsize=(8.0, 5.0))
for model, ratings in data.items():
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='blue', label='Accuracy')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='orange', label='Completeness')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='red', label='Consensus')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='green', label='Readability')
ax.set_xlabel('Ratings of Language Models on a Scale from 1-4')
ax.set_title('Ratings of Language Models on a Scale from 1-4')
ax.set_yticks([1, 2, 3, 4])
ax.set_yticklabels(['Accuracy', 'Completeness', 'Consensus', 'Readability'])
ax.legend()
plt.tight_layout()

ChartCoder Output
import matplotlib.pyplot as plt
# Data
data = {
    'GPT4': {'Accuracy': 3.5, 'Completeness': 3.5, 'Consensus': 3.5, 'Readability': 3.5},
    'GPT-few-shot': {'Accuracy': 3.8, 'Completeness': 3.8, 'Consensus': 3.8, 'Readability': 3.8},
    'GPT3.5': {'Accuracy': 3.6, 'Completeness': 3.6, 'Consensus': 3.6, 'Readability': 3.6},
    'StarCoder': {'Accuracy': 3.8, 'Completeness': 3.8, 'Consensus': 3.8, 'Readability': 3.8},
    'Code Llama': {'Accuracy': 3.5, 'Completeness': 3.5, 'Consensus': 3.5, 'Readability': 3.5},
    'MCL': {'Accuracy': 2.9, 'Completeness': 2.9, 'Consensus': 2.9, 'Readability': 2.9}
}
# Plotting
fig, ax = plt.subplots(figsize=(8.0, 5.0))
for model, ratings in data.items():
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='blue', label='Accuracy')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='orange', label='Completeness')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='red', label='Consensus')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='green', label='Readability')
ax.set_xlabel('Ratings of Language Models on a Scale from 1-4')
ax.set_title('Ratings of Language Models on a Scale from 1-4')
ax.set_yticks([1, 2, 3, 4])
ax.set_yticklabels(['Accuracy', 'Completeness', 'Consensus', 'Readability'])
ax.legend()
plt.tight_layout()

InternVL2-Llama3-76B Output
import matplotlib.pyplot as plt
# Data
data = {
    'GPT4': {'Accuracy': 3.5, 'Completeness': 3.5, 'Consensus': 3.5, 'Readability': 3.5},
    'GPT-few-shot': {'Accuracy': 3.8, 'Completeness': 3.8, 'Consensus': 3.8, 'Readability': 3.8},
    'GPT3.5': {'Accuracy': 3.6, 'Completeness': 3.6, 'Consensus': 3.6, 'Readability': 3.6},
    'StarCoder': {'Accuracy': 3.8, 'Completeness': 3.8, 'Consensus': 3.8, 'Readability': 3.8},
    'Code Llama': {'Accuracy': 3.5, 'Completeness': 3.5, 'Consensus': 3.5, 'Readability': 3.5},
    'MCL': {'Accuracy': 2.9, 'Completeness': 2.9, 'Consensus': 2.9, 'Readability': 2.9}
}
# Plotting
fig, ax = plt.subplots(figsize=(8.0, 5.0))
for model, ratings in data.items():
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='blue', label='Accuracy')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='orange', label='Completeness')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='red', label='Consensus')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='green', label='Readability')
ax.set_xlabel('Ratings of Language Models on a Scale from 1-4')
ax.set_title('Ratings of Language Models on a Scale from 1-4')
ax.set_yticks([1, 2, 3, 4])
ax.set_yticklabels(['Accuracy', 'Completeness', 'Consensus', 'Readability'])
ax.legend()
plt.tight_layout()

InternVL2-B Output
import matplotlib.pyplot as plt
# Data for the bar chart
data = {
    'GPT4': {'Accuracy': 3.5, 'Completeness': 3.5, 'Consensus': 3.5, 'Readability': 3.5},
    'GPT-few-shot': {'Accuracy': 3.8, 'Completeness': 3.8, 'Consensus': 3.8, 'Readability': 3.8},
    'GPT3.5': {'Accuracy': 3.6, 'Completeness': 3.6, 'Consensus': 3.6, 'Readability': 3.6},
    'StarCoder': {'Accuracy': 3.8, 'Completeness': 3.8, 'Consensus': 3.8, 'Readability': 3.8},
    'Code Llama': {'Accuracy': 3.5, 'Completeness': 3.5, 'Consensus': 3.5, 'Readability': 3.5},
    'MCL': {'Accuracy': 2.9, 'Completeness': 2.9, 'Consensus': 2.9, 'Readability': 2.9}
}
# Set up the figure and axes
fig, ax = plt.subplots(figsize=(8.0, 5.0))
for model, ratings in data.items():
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='blue', label='Accuracy')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='orange', label='Completeness')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='red', label='Consensus')
    ax.barh(model, [ratings['Accuracy'], ratings['Completeness'], ratings['Consensus'], ratings['Readability']], color='green', label='Readability')
ax.set_xlabel('Ratings of Language Models on a Scale from 1-4')
ax.set_title('Ratings of Language Models on a Scale from 1-4')
ax.set_yticks([1, 2, 3, 4])
ax.set_yticklabels(['Accuracy', 'Completeness', 'Consensus', 'Readability'])
ax.legend()
plt.tight_layout()

Figure 6: A example of comparing the code corresponding to the bar chart generated by different models.
Prompt for Code Readability Evaluation

Please score the code’s readability based on the following four aspects. Each aspect is worth 25 points, for a total of 100 points.
Naming Conventions (25 points)
Score: [X]/25
Explanation: [Provide a brief explanation of how well the variable, function, and class names convey their purpose and whether the naming style is consistent across the codebase.]
Code Structure (25 points)
Score: [X]/25
Explanation: [Explain whether functions are concise, whether the code uses indentation and blank lines appropriately, and whether the code is modularized effectively.]
Comments (25 points)
Score: [X]/25
Explanation: [Discuss the clarity and appropriateness of the comments, and whether functions/methods have proper documentation comments explaining inputs, outputs, and functionality.]
Logical Clarity (25 points)
Score: [X]/25
Explanation: [Evaluate the intuitiveness of the code, whether it’s easy to understand, and whether the control flow is simple and avoids unnecessary complexity.]
Total Score: [X]/100
Summary: [Provide a brief overall assessment of the code’s readability, pointing out strengths and potential areas for improvement.]

Figure 7: Prompt for dataset quality evaluation.
Prompt for Chart Quality Evaluation

You are a professional chart analyser. Please evaluate the image based on the following four criteria: aesthetics, readability, reproducibility, and data presentation simplicity. Provide a score for each criterion and include an overall score along with a brief evaluation.

Scoring Criteria and Requirements:
Aesthetics (25 points)
Requirements:
The chart design should be simple and clear, avoiding complex decorations, and should effectively communicate information.
Colors should be harmonious and have high contrast, making it easy to differentiate between different data groups.
Legends and labels should be clear, with appropriately sized fonts, avoiding visual clutter.
Scoring: [X]/25
Readability (30 points)
Requirements:
The chart should have clear titles, axis labels, and legends, enabling quick communication of the main message.
Data curves or point annotations should avoid being overly dense or overlapping, maintaining good readability.
The overall layout should follow a logical structure without any confusing elements.
Scoring: [X]/30
Reproducibility (30 points)
Requirements:
The chart design should be easy to replicate using common tools.
Data availability is critical: even if the design is simple, missing context or data should result in point deductions.
Data should be provided with clear sampling methods, units, and formats, enabling others to recreate the chart from scratch.
The presentation of data should align logically with the chart design, avoiding overly customized or complex elements.
Scoring: [X]/30
Data Presentation Simplicity (15 points)
Requirements:
Data presentation should be concise, avoiding redundant information.
Data should be displayed in an intuitive way, without excessive curves, points, or annotations.
High-scoring charts should focus on the main data being presented, avoiding decorative or unrelated information.
Scoring: [X]/15

Figure 8: Prompt for dataset quality evaluation.

Reviewer paper summaries: The paper introduces ChartCoder, a dedicated  multimodal LLM designed specifically for generating executable code from chart images. ChartCoder leverages a Code LLM as its backbone for improved code generation capabilities and introduces a new large-scale dataset, Chart2Code-160k, containing 160,000 chart-code pairs covering 27 chart types. The authors also propose a Snippet-of-Thought (SoT) method to decompose chart-to-code tasks into structured, step-by-step processes, enhancing the model's reasoning and execution accuracy. Experiments demonstrate that ChartCoder, despite having only 7B parameters, outperforms state-of-the-art open-source MLLMs across multiple benchmarks, including ChartMimic and Plot2Code, with significant gains in code executability and chart fidelity.

This paper proposes a multimodal LLM (ChartCoder) for chart-to-code generation with higher quality and executability. In addition, this paper introduces Chart2Code-160k, a large-scale training set for chart-to-code generation, and Snippet-of-Thought (SoT), a method to convert chart-to-code data into different generation steps.

This paper presents ChartCoder, a specialized multimodal large language model designed for the task of chart-to-code generation. The model is built upon SigLIP and DeepSeek Coder. To enhance its chart code generation capabilities, the authors develop a comprehensive pipeline to construct a dataset of 160K chart-code pairs for model training. Additionally, they introduce Snippet-of-Thought (SoT), a step-by-step generation framework, which proves effective in improving performance in the chart-to-code generation task.

The paper proposes a novel large-scale synthetic data that includes chart-code pairs. The authors also propose a model for chart to code generation that consists of three components: (1) vision encode, vision-language connector, and a code-LLM backbone. The model finetuned with the proposed synthetic data outperforms baselines, and achieves comparable performance with GPT-4 on the chart-code generation tasks.
Summary of strengths: 1. First MLLM specifically designed for chart-to-code generation.
2. A Large-scale Chart2Code-160k dataset with 160,000 chart-code pairs.
3. The proposed step-by-step reasoning strategy is Interesting.
4. The proposed MLLM achieves better performance compared to existing models.

1. The method is well-elaborated and the paper is well-written.
2. The proposed dataset (Chart2Code-160k) is much larger than previous chart-code datasets, which could serve as a useful model training source. The proposed Snippet-of-Thought (SoT) method further makes the dataset helpful in a step-to-step generation/reasoning setting.
3. Experimental results on three chart benchmarks show that the proposed model consistently outperforms other open-source baseline models. The ablation study further verifies the effectiveness of the method.

1. The paper is well-organized and clearly written, with the methods and contributions effectively articulated.
2. The constructed 160K dataset is a valuable resource that can benefit future research in this domain.
3. The experiments are comprehensive, incorporating appropriate baselines and evaluation benchmarks. Additionally, the ablation studies provide insightful findings that are likely to capture the interest of the research community.
4. The proposed Snippet-of-Thought (SoT) approach demonstrates effectiveness in enhancing chart-to-code generation performance.

1. The paper presents an efficient way to generate synthetic data chart-code data, and demonstrate that smaller llm finetuend on large-scale synthetic data outperforms much larger llms. The proposed approach is insightful and might be adopted to other tasks.
Summary of weaknesses: 1. The dataset and evaluation focus on 27 predefined chart types. However, the main paper lacks the detailed explanation of these different types.
2. Lack of quality control for generating snippet of thoughts.
3. Lack of User Study or Interpretability Analysis. While the model’s outputs are described as interpretable, the paper does not present a formal user study or qualitative analysis to validate how understandable or usable the generated code is.

See "Comments, Suggestions And Typos"

1. **Limited Diversity of Chart2Code-160K.** The dataset primarily consists of charts generated through random permutations of attributes and plotted using Matplotlib. This raises concerns about its diversity and the extent to which it represents real-world chart distributions. 
2. **Limited Technical Contribution.** The Snippet-of-Thought (SoT) method is a straightforward adaptation of Chain-of-Thought (CoT), a widely used and well-established approach.
3. **Trade-offs in Model Capability.** While the proposed model achieves state-of-the-art performance in the code generation task, its specialized design makes it less suitable for generic chart comprehension tasks (e.g., question answering), which baseline models are better equipped to handle.

1. The proposed dataset is generated by LLMs; however, the authors do not specify any methods employed to ensure the quality of the synthetic data (both the original code and decomposed code). It appears that the sole criterion considered is executability. The absence of data quality validation, such as human evaluation, may limit the dataset's practical utility.
2. The authors propose that training on chart-to-code tasks allows large language models (LLMs) to better capture chart information. However, the work does not include experiments to evaluate the model's performance on other chart-related tasks. While the analysis in Section 5.4 provides some insights, it may not be sufficiently convincing. 
Firstly, the results are based on a single LLM backbone, which limits the generalizability of the findings. Secondly, the analysis does not directly demonstrate the model's performance on chart understanding tasks. If the model is primarily intended to serve as auxiliary context, might this approach be considered too resource-intensive for the benefits it provides? Further exploration and evidence in these areas would greatly enhance the study's impact.
Comments, suggestions and typos: See weakness.


**Comments after Author Discussion: **

Thank the authors for the reply. It addresses most of my concerns. I raised my ratings and  suggest authors to add an user study to validate how understandable or usable the generated code is in the final version.

**Comments**:

1. Despite the proposed dataset being large enough for model training, is there any quality control over this large-scale dataset?
2. Are there natural language comments in the code snippets? If so, what is the effect of these comments? Presumably, they are helpful to the model provided they are correct.

**Typos**:

* Line 13: "tasks.: (1)" $\to$ tasks: (1)

Chart2Code-160k --> Chart2Code-160**K**?

N/A
Overall assessment scores: 3.5

4

3

3
Metareview score: 4
Metareview: This paper presents a novel approach to chart understanding by converting charts directly into executable code, addressing limitations in current MLLMs. The authors identify two primary challenges: low executability and poor restoration of chart details in generated code, and the lack of large-scale, diverse training data. To tackle these issues, they propose ChartCoder, which leverages Code LLMs as the language backbone, and introduce Chart2Code-160k, a comprehensive dataset for chart-to-code generation. Additionally, they present the Snippet-of-Thought (SoT) method to enhance step-by-step code generation. Experimental results demonstrate that ChartCoder, with 7 billion parameters, outperforms existing open-source MLLMs on chart-to-code benchmarks, achieving superior chart restoration and code executability.
Summary of reasons to publish: - The creation of the Chart2Code-160k dataset addresses the scarcity of large-scale, diverse training data in this domain, offering a valuable resource for future research.
- ChartCoder outperforms existing open-source MLLMs on various benchmarks, demonstrating superior performance in chart-to-code generation.
- The proposed model architecture, combining a vision encoder, vision-language connector, and a Code LLM backbone, is well-designed and tailored for chart-to-code tasks. The Snippet-of-Thought (SoT) enhances the model's reasoning capabilities by emphasizing critical information in a step-by-step manner.