MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization (2024)

Abstract: Scientific data visualization plays a crucial role in research by enabling the direct display of complex information and assisting researchers in identifying implicit patterns. Despite its importance, the use of Large Language Models (LLMs) for scientific data visualization remains rather unexplored. In this study, we introduce MatPlotAgent, an efficient modelagnostic LLM agent framework designed to automate scientific data visualization tasks. Leveraging the capabilities of both code LLMs and multi-modal LLMs, MatPlotAgent consists of three core modules: query understanding, code generation with iterative debugging, and a visual feedback mechanism for error correction. To address the lack of benchmarks in this field, we present MatPlotBench, a high-quality benchmark consisting of 100 human-verified test cases. Additionally, we introduce a scoring approach that utilizes GPT-4V for automatic evaluation. Experimental results demonstrate that MatPlotAgent can improve the performance of various LLMs, including both commercial and open-source models. Furthermore, the proposed evaluation method shows a strong correlation with human-annotated scores.1

heading: 1 Introduction
text: A picture is worth a thousand words. Data visualization is an essential process in scientific research, facilitating a more direct conveyance of complex information and aiding researchers in uncovering implicit patterns. There are many advanced toolkits, such as Matplotlib2 and Origin3, that can help researchers plot various types of figures for complex data distributions. However, transforming raw data into informative and easy-to-understand
* Equal contribution. † Corresponding authors. 1 MatPlotAgent and MatPlotBench are be publicly avail-
able at https://github.com/thunlp/MatPlotAgent. 2https://matplotlib.org 3https://www.originlab.com
visualizations is still time-consuming and laborintensive. Before the invention of large language models (LLMs) (OpenAI, 2023), automating this process with AI models is almost impossible.
With large-scale parameters and extensive training data, LLMs have demonstrated remarkable capabilities in a wide range of complex tasks, including reasoning (Wei et al., 2022; Kojima et al., 2022a; Yao et al., 2023a), mathematics (Yu et al., 2024; Luo et al., 2023a; Azerbayev et al., 2024; Shao et al., 2024) and coding (Rozière et al., 2024; Luo et al., 2023b; Guo et al., 2024; Wei et al., 2023). This breakthrough has unlocked new opportunities for utilizing LLMs as autonomous agents in a diverse range of practical scenarios, such as web browsing (Nakano et al., 2021; Yao et al., 2022; Qin et al., 2023; Zhou et al., 2023; Deng et al., 2023; Yao et al., 2023b; Xie et al., 2023), social simulations (Park et al., 2023; Xu et al., 2023; Chen et al., 2024a; Wang et al., 2023), tool utilization (Qin et al., 2024; Schick et al., 2023; Liu et al., 2024; Li et al., 2023a; Lu et al., 2023; Qian et al., 2023b; Shinn et al., 2023), and software development (Qian et al., 2023a). Using LLMs to enhance human productivity in specialized areas is now a key research focus with great potential.
Recent advancements in LLM-based agents inspire us to explore the utilization of LLMs for scientific data visualization, a realm that remains rather unexplored in existing studies. A closely related line of research is text-to-image generation (Ramesh et al., 2021; Saharia et al., 2022), where diffusion models (Rombach et al., 2022) have shown great potential in generating various types of images. However, existing text-to-image generation methods predominantly focus on artistic expression, potentially misaligning with the needs of scientific data visualization, where clarity and precision in conveying information are the most important principles. This work aims to automatically generate figures with precise information.
11789
We propose leveraging modern code LLMs and multi-modal LLMs to develop scientific data visualization agents that can significantly enhance human efficiency. The resulting MatPlotAgent4 is comprised of three modules: (1) the query understanding module that can thoroughly understand user-provided requirements; (2) the code generation module with iterative debugging capabilities that use code to precisely preprocess raw data and generate figures; and (3) the visual feedback module that possesses visual perceptual abilities to find errors in the plotted draft and provide visual feedback to the code generation module to rectify the errors. Our method is model-agnostic, which can be driven with any code LLMs and multi-modal LLMs. Through experiments, we find that MatPlotAgent can work with both closed-source LLMs (e.g., GPT-4 (OpenAI, 2023)) and open-source LLMs (e.g., Magicoder (Wei et al., 2023)).
Another critical challenge in the field of automatic scientific data visualization is the absence of benchmarks for evaluation purposes. To address this issue, we introduce a meticulously crafted benchmark called MatPlotBench to quantitatively evaluate the approaches involved. Specifically, MatPlotBench contains 100 carefully hand-crafted test examples, each of which contains a user query, the corresponding input data, and a ground-truth figure verified by human experts. We believe that
4This name is in homage to the well-known Matplotlib.
high-quality test sets play a crucial role in driving advancements in the field.
To facilitate automatic quantitative evaluation, we also design a scoring mechanism based on GPT4V (OpenAI, 2023), which is one of the strongest multi-modal LLMs that can effectively understand text and figures. Specifically, GPT-4V is prompted to produce a score between 0 and 100 based on the ground-truth figure and the one generated by AI models. Additionally, we conduct human evaluation and estimate the correlation coefficient between human-annotated scores and the automatically calculated scores. The results reveal a strong correlation between the automatic score and the human-annotated score, thus affirming the reliability of the scoring mechanism. In summary, our contribution can be listed as follows:
• We introduce MatPlotBench to enable automatic quantitative evaluation of AI methods designed for scientific data visualization. Through comparison with human evaluation, we observe that MatPlotBench can effectively capture the performance of AI approaches in this cutting-edge task.
• We propose an effective and generalizable LLM agent framework, MatPlotAgent, that can improve the performance of a wide range of LLMs based on the newly proposed visual feedback mechanism.

heading: 2 Task Description
text: We first introduce the scientific data visualization task investigated in this work. Given a user query x described in text and the corresponding data D, the AI system is expected to output a figure V that can satisfy the user’s demand:
V = f(x,D), (1)
where f denotes the involved AI system that can be either an LLM or an LLM-based agent.
Specifically, x specifies the visualization requirements, encompassing the visualization type, data to plot, structural or spatial requirements for individual elements or the entire plot, and aesthetic preferences. D represents the data, a collection of data points {d1, · · · , dn} whether specified by the user or stored in the external data file. Figure 1 provides some examples for this task.

heading: 3 MatPlotBench
text: Automatic evaluation is important in AI tasks as it enables researchers to efficiently assess the performance of various methods, thereby guiding the development of the field. While the DS-1000 benchmark (Lai et al., 2023) includes coding problems about Matplotlib, the solutions’ average length is merely three lines, rendering them too simplistic to gauge the proficiency of contemporary AI agents in tackling practical challenges. Therefore, we propose to construct MatPlotBench with complex data visualization problems that are more close to real-world scenarios. We will illustrate the data collection process in Section 3.1 and then explain the scoring mechanism in Section 3.2.

heading: 3.1 Data Collection
text: Principles To enhance the quality of MatPlotBench, we adhere to the following principles for data collection: (1) Covering diverse types: encompassing a broad range of plot types, including not only the most commonly used but also rare but useful ones; (2) Containing representative instances: ensuring that the test examples reflect the representative features of scientific data visualization, such as varying data complexity; and (3) Balancing easy and challenging problems: including problems of varying levels of difficulty in the benchmark.
Selecting Original Examples In accordance with the principles outlined above, we first select
some original examples from reputable online scientific data visualization forums. These examples are carefully selected from the Matplotlib Gallery and OriginLab GraphGallery, encompassing diverse and representative instances with varying levels of difficulty. Specifically, we select 1 or 2 examples from every section in the Matplotlib Gallery, covering bars, lines, markers, pie charts, polar plots, contour plots, statistics plots, 3D plots, text annotations, radar charts, shapes, scales, axes, spines, subplots, and so on. We also seek more advanced test examples from the OriginLab GraphGallery, focusing on those that are more aesthetically appealing or complex, such as Sankey diagrams, sunburst charts, radial plots, chord diagrams, streamplots, and others. Finally, 75 original examples come from the Matplotlib Gallery and the 25 other original examples come from the OriginLab GraphGallery. Subsequently, these examples undergo several modifications to become the final test cases in MatPlotBench.
Preliminary Query Generation Based on the selected original examples, we use LLMs to generate preliminary queries, which are then revised by humans. For original examples from the Matplotlib Gallery, we use GPT-4 to convert the code in each original example into preliminary queries. For the examples from the OriginLab GraphGallery, there are only images. We thus use GPT-4V to convert each image into a preliminary query.
Data Replacement Based on these preliminary queries, we begin data replacement for examples from the Matplotlib Gallery due to the observed phenomenon of memorization by GPT-4. In this process, we replace the original data points with newly generated ones, while keeping other factors such as the plot type unchanged. For examples from OriginLab, we find that the data is inherently complex, and even GPT-4 does not exhibit memorization with these examples. As a result, we only perform data replacement for Matplotlib examples.
Human Modification After completing the data replacement process, we engage human annotators to refine the preliminary queries. These annotators are tasked with correcting errors, eliminating ambiguity, and adding any omitted essential information. Each annotator involved has a minimum of three years of experience in coding and NLP. Furthermore, each query undergoes refinement by two independent human annotators.
Updating Ground-Truth Figures After obtaining the human-annotated queries, as the data in Matplotlib examples are altered, we cannot directly use the images in the original example as the ground truth. To this end, we manually wrote code to plot the ground truth for the Matplotlib examples. For examples from OriginLab, we employ human experts to prompt GPT-4 to generate code and subsequently produce its corresponding plots as the ground truth. If GPT-4 fails to generate the plots, we extract the images from their website to serve as the ground truth.
Human Verification After obtaining the queries and their corresponding ground truths, we performed a final round of manual verification. Three NLP researchers were asked to conduct this verification. In this turn, the focus is mainly on checking whether the user queries and the ground truths are well aligned. The researchers meticulously checked each element in the ground truth images and looked for their corresponding descriptions in the user query. Ill-described elements and those missing clarifications are corrected. Redundant and incorrect descriptions are removed. This process results in 100 high-quality (query, raw data, ground-truth figure) triplets, which comprise our final benchmark.

heading: 3.2 Automatic Quantitative Evaluation
text: To ease the burden of manual evaluation and broaden the applicability of our benchmark for research purposes, we suggest employing GPT-4V, a cutting-edge multi-modal LLM, to conduct automatic evaluations on our proposed benchmark. We carefully prompt GPT-4V to give a score from 0 to 100 on model-generated visualizations using the corresponding ground truths as reference. The prompt is shown in Figure 6 in Appendix.
Correlation with Human Evaluation To assess the reliability of GPT-4V as an automatic evaluator for scientific visualizations, we calculate the correlation between the automatic scores and humanevaluated scores. Specifically, we employ GPT-3.5 and GPT-4 to generate figures on MatPlotBench, and then conduct both automatic and human evaluation for the generated figures. For each model, we iteratively sample a subset that consists of n examples from the total benchmark, and then calculate the average score of both automatic and human evaluation. This process repeats k times and we get k data points for each type of evaluation,
which can be represented by A = {a1, · · · , ak} and H = {h1, · · · , hk}. ai denotes the average automatic score on the i-th randomly sampled subset, and hi represents the average human-evaluated score in the same subset. n and k are set to 25 and 100, respectively.
We utilize the statistical functions provided by scipy5 to compute the Pearson correlation coefficient r and the corresponding p-value p. For GPT4, we obtain r=0.876 and p=7.41e-33, while for GPT-3.5, the values are r=0.836 and p=2.67e-27. Figure 2 shows the data points for GPT-4. Given that r > 0.8 and p <0.05, we conclude that the automatic evaluation scores are strongly correlated with human evaluation results. This demonstrates the reliability of the proposed scoring mechanism in assessing the quality of model-generated figures on MatPlotBench.

heading: 4 MatPlotAgent
text: To improve the capabilities of LLMs for scientific data visualization, we propose an agentic framework that mimics the plotting process of human experts. The proposed MatPlotAgent is comprised of three modules, including the query expansion module, the code agent, and the visual agent. Figure 3 illustrates the workflow of MatPlotAgent.

heading: 4.1 Query Expansion
text: The query expansion module interprets and refines the user query, converting the high-level requirements into a sequence of explicit and detailed instructions that are easy for LLMs to follow. This module can also be viewed as a planning module, creating an overall plan before generating the
5https://docs.scipy.org/doc/scipy/reference/ stats.html
figure. Specifically, this module is based on the involved code LLM, which is prompted to give detailed instructions on how to use code to fulfill the requirement specified by the user, including what libraries to import, what library functions to call, how to set the parameters in each function correctly, how to prepare the data, how to manipulate the data, and so on.

heading: 4.2 Code Agent
text: The code agent is the core component in MatPlotAgent, responsible for generating the code to plot figures. Given detailed instructions from the query expansion module, the code agent first generates the code using appropriate libraries and functions. To improve the success rate of the generated code, we also employ the self-debugging mechanism (Chen et al., 2024b), which helps the involved code LLM iteratively identify and correct bugs in the code. To prevent an infinite loop, we set the maximum iterations of self-debugging to 3.
Similar to humans, who need to repeatedly refine the figure based on current drafts, we also introduce a visual feedback mechanism. This mechanism employs multi-modal LLMs to provide suggestions to improve the figure and better fulfill the user’s queries. These suggestions, which we call visual feedback, are then provided to the code agent to
further improve the code. Our experiments in Section 5.2 demonstrate that MatPlotAgent is compatible with several modern code LLMs, including both some well-known closed-source models and some open-source models.

heading: 4.3 Visual Agent
text: The major difference between MatPlotAgent and previous LLM-based coding agents (Qian et al., 2023a; Chen et al., 2024b) is that we take the visual signal into account, which is important in scientific data visualization. Some errors or weaknesses may be difficult to identify in the code but become apparent when observing the output figure through “eyes”. The visual agent is the “eyes” for MatPlotAgent, while the aforementioned code agent acts as the “hands” for MatPlotAgent.
Specifically, the visual agent is powered by multi-modal LLMs. We introduce several guiding principles for the visual agent, including verifying whether the figure aligns with the provided data, and enhancing the colors or labels to improve the figure’s informativeness. Based on the principles, the user query, and the current draft of the figure, the visual agent generates some suggestions to refine to figure. These suggestions serve as feedback for the code agent to refine the code. Experimental results in Section 5.4 show that our visual feedback
Model Direct Zero-Shot MatPlotAgentDecod. CoT w/ GPT-4V
Model Direct MatPlotAgentDecod. w/ Gemini Pro Vision
mechanism can significantly improve the quality of the plotted figures.

heading: 5 Experiments

heading: 5.1 Setup
text: Models Since the proposed MatPlotAgent is model-agnostic, we can employ various LLMs in this framework. The code LLMs we use in our experiments include GPT-4, GPT-3.5, Magicoder-SDS-6.7B (Wei et al., 2023), Deepseek-coder-6.7Binstruct (Guo et al., 2024), Deepseek-coder-33Binstruct (Guo et al., 2024), WizardCoder-Python33B-V1.1 (Luo et al., 2023b), and CodeLlama-34BInstruct (Rozière et al., 2024). The decoding temperature is set to 0.0 for all involved code LLMs. For GPT-4 and GPT-3.5, we use the API provided by OpenAI6. For the other five open-source LLMs, we use vLLM (Kwon et al., 2023) for model inference. For the visual agent, we utilize GPT-4V (OpenAI, 2023) and Gemini Pro Vision (Google, 2023), two representative multi-modal LLMs. We leave the exploration of using open-source multi-modal LLMs to power the visual agent for future work.
Evaluation We evaluate the involved methods on MatPlotBench, using the proposed automatic scoring mechanism that is shown reliable in Section 3.2.
6https://openai.com/product
For each code LLM, we evaluate its performance in three ways:
• Direct decoding: given the query, the model directly generates the plotting code.
• Zero-Shot Chain-of-thought (Kojima et al., 2022b): the model is prompted to inference with the zero-shot CoT mechanism.
• MatPlotAgent: the model is equipped with the proposed MatPlotAgent framework, driving the query expansion module and the code agent, as illustrated in Section 4.

heading: 5.2 Main Results
text: Table 1 presents the results of different methods on the scientific data visualization task. In the direct decoding setting, GPT-4 achieves the highest score of 48.86. Surprisingly, the open-source model Magicoder-S-DS-6.7B (Wei et al., 2023) achieves the second-best performance, surpassing models with substantially larger parameter sizes, such as WizardCoder-Python-33B-V1.1.
The results also suggest that the zero-shot CoT mechanism does not effectively enhance the performance of many recent code LLMs. Zero-shot CoT only improves the results of Deepseek-coder33B-instruct (Guo et al., 2024) from 30.88 to 36.10. Conversely, for other models, implementing zeroshot CoT results in poorer performance. For example, when zero-shot CoT is applied, the performance of GPT-4 drops to 45.42, which is lower than the direct decoding result of 48.86.
From Table 1, we find the proposed MatPlotAgent can improve the plotting capabilities of several models. For GPT-4 and GPT-3.5, MatPlotAgent leads to significant improvements of 12.30 and
9.48, respectively. For the other five open-source LLMs, MatPlotAgent improves the performance of four models. With MatPlotAgent, the open-source Magicoder-S-DS-6.7B model even surpasses GPT4 with direct decoding (51.70 vs. 48.86), showcasing the effectiveness of our method.
To investigate the generalizability of MatPlotAgent across various multi-modal LLMs, we present the results of employing Gemini Pro Vision as the visual agent in Table 2. We observe considerable improvements of 7.87 and 5.45, respectively, over the direct decoding baseline. This evidence further demonstrates the model-agnostic characteristic of our approach, leveraging various multi-modal LLMs to achieve enhanced performance.

heading: 5.3 Results on Qwen-Agent Code Interpreter Benchmark
text: In Table 3, we detail the performance of MatPlotAgent on the visualization subset of the Qwen-Agent Code Interpreter Benchmark7, which was recently published. According to their GitHub repository, GPT-4 achieved scores of 66.7 and 60.8 on the Visualization-Hard and Visualization-Easy subsets, respectively. Utilizing MatPlotAgent, we attained higher scores of 72.62 and 68.35 on these subsets. When the visual feedback mechanism is disabled, MatPlotAgent reached scores of 66.67 and 65.82, reconfirming the necessity of visual feedback.

heading: 5.4 Ablation Study
text: Compared to previous LLM-based coding agents (Qian et al., 2023a; Chen et al., 2024b), the major contribution of our work lies in the newly proposed visual feedback mechanism, which is expected to leverage visual signals to enhance the quality of the output figure. To gain a deeper understanding of the impact of the visual feedback mechanism, we conduct both qualitative and quantitative analyses in this section.
7https://github.com/QwenLM/Qwen-Agent/tree/ main/benchmark
Figure 4 presents examples plotted by LLMs both with and without the visual feedback mechanism. We observe a clear improvement in the quality of the output figure with the visual feedback. For example, in case C, the text in the figure is jumbled, but this issue is resolved with the assistance of visual feedback. It is important to note that the visual agent does not reference the ground-truth figure when generating feedback; it only examines the draft plotted by the model. Table 4 also presents quantitative results of the visual feedback mechanism, indicating that the absence of visual feedback would result in significantly poorer outcomes for both GPT-4 and GPT-3.5. This reaffirms the importance of visual signals in the task of scientific data visualization.

heading: 5.5 Case Study
text: We present output figures in Figure 5. The first example is relatively simple, correctly plotted by GPT-4 augmented with MatPlotAgent. The second example is more challenging; while GPT-4 and Magicoder-S-DS-6.7B can generate a draft, both omit some elements. The third example is the most difficult, where none of the three models can produce the correct result. These results indicate that the proposed MatPlotBench poses a significant challenge for current LLMs. Even the state-of-the-art LLM, GPT-4, equipped with MatPlotAgent, fails in some cases. We believe this benchmark will be effective not only for evaluating AI systems in scientific data visualization but also for assessing general capabilities such as coding
and visual perception.

heading: 6 Related Work
text: Code LLMs Since the release of Codex (Chen et al., 2021), many closed- and open-source code LLMs have been published, pushing the boundaries of LLMs’ capabilities to write functional code. Early open-source efforts include SantaCoder (Allal et al., 2023) and StarCoder (Li et al., 2023b). More recently, the Code Llama (Rozière et al., 2024) series is released, including models of varying sizes. DeepSeekCoder (Guo et al., 2024), a series of open-source code models ranging in size from 1.3B to 33B, has also garnered significant attention for its impressive performance on general coding benchmarks. Wei et al. (2023) introduce a novel data augmentation method for automatically creating high-quality fine-tuning data. The resulting Magicoder model surpasses a wide array of open-source code LLMs in performance.
LLM Agents Recently, a wide range of LLMbased agent frameworks is proposed to explore LLMs’ potential in real-world scenarios (Nakano et al., 2021; Yao et al., 2022; Qin et al., 2023; Zhou et al., 2023). OpenAgents (Xie et al., 2023) pro-
posed an open platform that leverages LLM agents in everyday situation by employing a Data Agent, a Plugins Agent, and a Web Agent. Park et al. (2023) proposed an interactive simulation of human behavior in which software agents emulate realistic human actions and interactions through computation. Voyager (Wang et al., 2023) introduced the fisrt LLM model-driven autonomous agent in Minecraft, designed to perpetually explore the environment, master various skills, and uncover new insights independently, without any human guidance. ChatDev (Qian et al., 2023a) proposed creating a virtual, chat-driven software development enterprise that follows the traditional waterfall methodology. In this study, we explore the capabilities of LLMbased agents in the task of scientific data visualization, a critical and practical area for contemporary researchers.
Automatic Data Visualization Dibia and Çağatay Demiralp (2018) first used LSTM networks to turn JSON data into Vega-Lite visualizations, marking an early innovation in automatic data visualization. Unlike them, MatPlotAgent uses LLMs to generate Python code from user queries and data. Dibia (2023) used a
modular approach that contains four components to generate visualization and infographics. Cheng et al. (2023) introduced an end-to-end data analysis framework using GPT-4. Distinctively, MatPlotAgent employs a novel visual agent to enhance plot quality based on visual feedback.

heading: 7 Conclusion
text: We propose to assess and enhance the capabilities of modern LLMs for scientific data visualization, a multifaceted task demanding coding and visual skills. We begin with the creation of MatPlotBench, a rigorous benchmark supporting automated quantitative evaluation that strongly aligns with human assessment. Additionally, we introduce MatPlotAgent, a model-agnostic mechanism employing visual feedback to enhance LLMs’ plotting abilities. Experimental results demonstrate that MatPlotAgent enhances the performance of various LLMs.

heading: 8 Limitations
text: In this paper, we introduce MatPlotBench, a benchmark designed for scientific data visualization. However, the demands of scientific data visualization can vary significantly across disciplines. Since MatPlotBench is developed for general scientific data visualization, it may not encompass all
domain-specific requirements, potentially restricting its applicability to certain fields. In the future, the data construction and evaluation approaches can be customized for specific domains if necessary.

heading: Acknowledgements
text: We thank all anonymous reviewers for their valuable comments and suggestions on this work. This work is supported by the National Key R&D Program of China (No.2022ZD0116312), the National Natural Science Foundation of China (No. 62236011), and a grant from the Guoqiang Institute, Tsinghua University.

heading: A Detailed Prompts
text: To better understand MatPlotBench and MatPlotAgent, we list the prompts for automatic evaluation and the three modules in MatPlotAgent, including the query expansion module, the code agent, and the visual agent.
A.1 Evaluation Prompts The automatic evaluation prompt primarily requires GPT-4V to provide a score between 0 and 100 for the model-generated plot, with reference to the ground truth plot.
A.2 Prompts for MatPlotAgent The query expansion prompt mainly requires LLMs to generate step-by-step, detailed instructions on how to use Python code to fulfill the requirements specified by users, as shown in Figure 7.
For the code agent, there are two prompts for the code generation process and the self-debugging mechanism. The code generation prompt mainly requires LLMs to generate executable code according to the user query to plot and save the output figure, as shown in Figure 8. The self-debugging prompt mainly requires LLMs to correct the buggy code according to the error message from a Python interpreter, as displayed in Figure 9.
The visual agent prompt mainly requires multimodal LLMs to firstly understand the user query and analyze the draft plot, then generate the visual feedback to refine the draft, as shown in Figure 10.

heading: B Human Evaluation Details
text: We engage human annotators from computer science departments at various universities via social media. They are compensated for their work at a rate slightly higher than the prevailing market rate. All human annotators involved are informed that the collected data will be used solely for academic research purposes, and their personal information will not be disclosed.
B.1 Evaluation Guide for Human Annotators Figure 11 gives detailed instructions for human annotators when scoring the model-generated plots.

heading: Future Work
text: In the future, the data construction and evaluation approaches can be customized for specific domains if necessary