Venue: NAACL 2025

# C\({}^{2}\): Scalable Auto-Feedback for LLM-based Chart Generation

Woosung Koh1, , Jang Han Yoon1, MinHyung Lee1, Youngjin Song1,

**Joegwan Cho1**, Jaehyun Kang1, Taheyeon Kim2, Se-Young Yun3,

**Youngjae Yu14**, Bongshin Lee14

Footnote 1: footnotemark:

Footnote 1: footnotemark:

Footnote 2: footnotemark:

###### Abstract

Generating high-quality charts with Large Language Models (LLMs) presents significant challenges due to limited data and the high cost of scaling through human curation. \(\langle\text{instruction},\text{data},\text{code}\rangle\) triplets are scarce and expensive to manually curate as their creation demands technical expertise. To address this scalability challenge, we introduce a _reference-free_ automatic feedback generator, which eliminates the need for costly human intervention. Our novel framework, **C\({}^{2}\)**, consists of (1) an automatic feedback provider (ChartAF) and (2) a diverse, reference-free dataset (ChartUIE-8K). The results are compelling: in our first experiment, 74% of respondents strongly preferred, and 10% preferred, the results after feedback. The second post-feedback experiment demonstrates that ChartAF outperform nine baselines. Moreover, ChartUIE-8K significantly improves data diversity by increasing queries, datasets, and chart types by 5982%, 1936%, and 91%, respectively, over benchmarks. Finally, a study of LLM users revealed that 94% of participants preferred ChartUIE-8K's queries, with 93% deeming them aligned with real-world use cases. Core contributions are available as open-source at chartsquared.github.io, with ample qualitative examples.

+
Footnote †: Corresponding authors

+
Footnote †: Corresponding Authors

## 1 Introduction

Charts are a powerful means to convey information in diverse fields, including journalism, business, and scientific research Fox and Hendler (2011); Rodriguez et al. (2015); Islam and Jin (2019). With the success of foundation models Kaplan et al. (2020); Roziere et al. (2023), there has been an increasing demand for generating charts using Large Language Models (LLMs). For instance, LIDA Dibia (2023) uses LLMs to automatically generate visualizations and infographics, and Chat2VIS Madigan and Susnjak (2023) incorporates LLMs to create charts from natural language queries. Moreover, LLM-generated charts empower humans by helping non-experts generate high-quality charts Maddigan and Susnjak (2023) and improving accessibility to those with special needs Gorniak et al. (2023); Moured et al. (2024). Despite the rising interest, two key challenges persist: **(i)** the difficulty in evaluating LLM-generated charts and **(ii)** the limited availability of training data.

**(i)** Chart generation lacks straightforward evaluation methods, making it difficult to assess and improve the quality of LLM-generated charts. Unlike tasks with clear-cut answers, such as mathematical problem-solving where verifiers can automatically assess correctness Uesato et al. (2022); Wang et al. (2024), chart evaluation is inherently subjective. Multiple correct designs may exist for a task (or goal), and quality often aligns with human aesthetic and functional preferences. Consequently, current evaluation systems Yang et al. (2024); Wu et al. (2024); Xia et al. (2024) rely on reference-based approaches, necessitating labor-intensive \(\langle\text{instruction},\text{data},\text{code}^{1}\rangle\) triplets as a gold reference for evaluation and thus limiting their scalability.

**(ii)** Furthermore, in contrast to image generation, which typically requires only \(\langle\text{instruction},\text{image}\rangle\) pairs Radford et al. (2021), chart generation demands more complex \(\langle\text{instruction},\text{data},\text{code}\rangle\) triplets. This significantly increases the costs associated with data collection and annotation. The limited number and diversity of available data restrict the variety of charts users can generate, making chart generation expensive and labor-intensive even for common applications Vazquez (2024).



To effectively tackle both **(i)** and **(ii)**, it is essential to first address the primary bottleneck--reference-based evaluation--which also opens the door to significantly improving data diversity and scale. To this end, we introduce \({\bf C}^{2}\), a scalable framework, composed of the following two synergistic components:

* ChartAF: A pipeline for Chart AutoFeedback, comprising ChartAF-S for evaluation scores and ChartAF-G for granular feedback in natural language.
* ChartUIE-8K: A large-scale (over 8,000 instances) Chart User Interaction Emulation dataset.

Fig. 1 illustrates a schematic overview of \({\bf C}^{2}\). ChartAF empowers automatic (i.e., human-annotation-free) chart generation improvements (Sec. 2). ChartAF works exceptionally well reference-free, enabling the cost-effective curation of a large-scale chart generation evaluation set ChartUIE-8K (Sec. 3).

The quantitative and qualitative results of \({\bf C}^{2}\) is overwhelmingly positive. First, by leveraging ChartAF's scalar evaluation scores for a simple test-time scaling scheme, 84% of respondents preferred the post-feedback results, with 74% strongly preferring them and 10% preferring them (Sec. 4.2). Second, employing ChartAF's granular feedback to in-context tune, ChartAF's post-feedback preference scores beat 9 baselines, that are alternatives to ChartAF (Sec. 4.3). The qualitative improvements for both test-time scaling and in-context tuning can be viewed on our open-source project site.

Finally, ChartAF's reference-free nature allows us to curate, ChartUE-8K, dramatically raising data diversity via number of queries, underlying datasets, and chart types by 5982%, 1936%, and 91%, respectively, against existing evaluation sets (Tab. 1). We also demonstrate that ChartUIE-8K closely aligns with real-world human requests via a study of LLM users (Sec. 4.4). The study highlights that ChartUIE-8K's evaluation set distribution closely aligns with real-world users, and 94% and 93%, prefer, and think is realistic, respectively.

### Related Work

Chart GenerationMaddigan and Susnjak (2023) offer prompt-engineered, LLM-based chart generation--however, they only provide qualitative case studies to verify their contribution. Dibia (2023) presents an LLM-based infographic visualization tool, which includes interactive charts. However, Dibia (2023) does not include a human study, making it challenging to assess its effectiveness. Sah et al. (2024) propose a natural language-to-chart recommendation approach based on the visualization language Vega-Lite. Therefore, their task deviates from the LLM-based chart generation we tackle. Tian et al. (2024) recently proposes a chart generation work that generates charts from "abstract user utterances" (as stated in the paper), which diverges from the instruction-based queries we address. An example of an "abstract user utterance" they provide is "What kind of movies earn the most recently?"--this diverges from the example instructions provided in our study of LLM users (App. B.2). This difference is understandable as

Figure 1: Schematic overview of \({\bf C}^{2}\) illustrating the synergy between ChartUIE-8K and ChartAF. The scale is made possible by ChartAF’s capability to provide _reference-free_ feedback. An end-to-end example is available on our project site’s github README.

 

their chart generation is based on proprietary user-interaction software, not a common LLM chatbot.

Tuning with Feedback.Feedback-based LLM tuning is commonly done via three methods: parameter tuning (Ouyang et al., 2022), in-context tuning (Liu et al., 2022), and test-time scaling ( TTS; OpenAI (2024)). Parameter tuning occurs through (implicit or explicit) rewards, requiring large amounts of reward data. For instance, Kirstain et al. (2023) and Xu et al. (2024b) collected over 500,000 and 137,000 annotations, respectively. In-context tuning allows LLMs to improve via \(n\)-shot generation (Chen et al., 2023), which requires \(n\) additional prompts. TTS refers to the case where a verifier with an ordinal output (typically of scalar value) can help improve the final generation Snell et al. (2024). Each tuning method requires an external feedback-provider, such as a reward model, additional prompt, or verifier.

## 2 C\({}^{2}\): ChartAF

In **C\({}^{2}\)**, a feedback-provider, ChartAF enables TTS and in-context tuning. We first describe the shortcomings of existing approaches, and introduce two versions of ChartAF: ChartAF-S and ChartAF-G.

### Towards High-performing Feedback

Two feedback types have been explored in the LLM research community: **(1)** scalar score-based (\(s_{\texttt{Nl}}\in\mathbb{N}_{\cup 0}\)) and **(2)** natural language-based (\(s_{\mathcal{N}}\in\mathcal{N}\), where \(\mathcal{N}\) is the natural language set space) feedback. Instead of competing, **(1)** and **(2)** serve different purposes: **(1)** is suited for parameter tuning with rewards, and TTS, while **(2)** is suited for context tuning. We summarize prior efforts of the two approaches in chart generation below.

**(1)** Yang et al. (2024), Wu et al. (2024), and Xia et al. (2024) provide \([0,100]\), \([1,10]\), \([0,5]\), scores, respectively. However, as these methods were developed under a reference-based regime, their performance sharply deprecates when applied reference-free. While there are other similar works, they are closed-source without clear details for replication Han et al. (2023); Xu et al. (2023).

**(2)** A naive approach uses a zero-shot LLM to provide feedback Yang et al. (2024), which we call Naive Feedback (NF). This method, as our experiments later show, is ineffective (Tab. 3). Furthermore, Yang et al. (2024) does not provide human studies of their NF against baselines so there is no evidence on its efficacy.

To overcome the limitations of past works, we present ChartAF (Fig. 2). As ChartaF-S is a subset of ChartAF-G, we first introduce ChartAF-S and then the additional component corresponding to ChartAF-G. Fine-grained pseudocode and prompts are provided in App. A.

### ChartAF-S (\(\tilde{f}_{Af}\))

Module 1.The user query, \(q\in\mathcal{Q}\in\mathcal{N}\), is first decomposed into three essential factors in a chart generation query: Task, Purpose, and Audience Choe et al. (2017); Lee et al. (2020); Narechania et al. (2020); Wang et al. (2020); Parsons (2021). By explicitly decomposing the \(q\) into Task, Purpose, and Audience, ChartAF can better infer the user intention Quadri et al. (2024); Bressa et al. (2024). Since user queries are often brief (Fig. 6, App. B.2), the intention may not always be clearly stated. Nevertheless, ChartAF utilizes the \(q\) to induce underlying intentions.

This information is then fused with the Basic Criteria--a general, high-level criteria applicable to all chart evaluations. The Basic Criteria ensures that ChartAF comprehensively considers chart elements: Chart Type, Visual Embellishment, Text, Color, Annotation, Aesthetics, and Visual Clutter. This criteria is inspired by the rich literature in visualization research. We document the research that corresponds to each element in App. A.1.

This first module is the domain grounding module. ChartAF replaces gathering costly human annotations with existing scholarly research. Not only is this cheaper, this approach closely aligns with how human-made chart generations would be evaluated. We would grade human students' chart generations with a domain expert (lecturer) that has learned the principles of chart generation, grounded in scholarly literature Bach et al. (2023).

Module 2.Considering the decomposed \(q\), and Basic Criteria, ChartAF generates \(q\)-Specific Criteria--specializing the general Basic Criteria. Specialization is key to generating feedback that is customized to the specific \(q\)Kim et al. (2024). The criteria is then transformed to binary (i.e., yes or no) questions Hu et al. (2023) as we find that LLMs are more reliable when reasoning with binary rather than open-ended questions. This also allows ChartAF to explicitly associate each criterion with one question. Otherwise, LLMs tend 

to lump numerous criterion together, resulting in duplicate related criteria.

Then, these questions are evaluated considering the LLM-generated chart (generated by executing the code output \(o\in{\cal O}\in{\cal N}\)). This code is generated by the chart generating LLM, \(f:{\cal Q}\mapsto{\cal O}\). The final scalar, \(s_{\text{Nl}}\), is derived by equal-weighting each binary answer, yes (1) and no (0). If researchers only require a scalar score, for downstream applications, the process can be terminated.

### ChartAF-G (\(f_{Af}\))

Module3For granular feedback, the process continues by associating each answered question with Retain, Edit, Discard, and Add. This association helps to decompose the evaluation result of each criterion to _actionable_ feedback. Based on this association, code-centric text feedback is provided Bi et al. (2024): the LLM is prompted to provide fine-grained feedback considering the downstream application, i.e., chart generation via code generation. Code-centric feedback encourages the LLM to provide explicit feedback that can be directly applied to the downstream \(f\).

## 3 C\({}^{2}\): ChartUIE-8K

Leveraging the reference-free nature of ChartAF, we curate ChartUIE-8K, a comprehensive chart generation evaluation set. Since no gold references are required, we can significantly scale the dataset. Qualitative examples of ChartUIE-8K can be viewed in Figs. 1, 2, App. C, and our project site.

Curation MethodAn overview of ChartUIE-8K's curation process is illustrated in Fig. 3. First, for diverse chart topics, we semi-automatically crawl diverse datasets online with appropriate licenses. We include only the datasets that have been used by humans for chart generation purposes, as not all datasets are suitable for visualization. Next, to ensure diverse chart types and annotations, we adopt a comprehensive list of chart types Hess (2022) and annotations Ren et al. (2017). Then, we emulate two types of users: [**U1**] lay users and [**U2**] detailed users.

To this end, we synthetically (LLM-assisted) generate initial instructions with two configurations: approx. [**U1**] <50, and [**U2**] <100 words. Considering the salience of multi-turn benchmarking Wang et al. (2024c), we further emulate a single QA cycle. I.e., the LLM asks for clarifying questions to the user, then, the user emulator responds to [**U1**] 25% or [**U2**] 50% of the questions. The pseudocode is provided in App. B.1.

Statistical Summary.To systematically understand the evaluation set, we compare key statistics against relevant benchmarks (see Tab. 1). We exclude chart topic counts for benchmarks, as they are

Figure 2: Schematic diagram of ChartAF, including a qualitative example indicated by dashed containers. The process starts in the top-left with the user query, which is processed by the chart-generating LLM and either ChartAF-S or ChartAF-G . Notably, ChartAF-S and ChartAF-G both share the first two modules (red and green). The final output is a scalar evaluation score or granular feedback, depending on the chosen path.

 

not provided in MatPlotBench Yang et al. (2024) and Plot2Code Wu et al. (2024). We do not manually count their chart topics as this a subjective task. On the other hand, we count MatPlotBench's and Plot2Code's chart types using the same taxonomy used for ChartUIE-8K (App. B.1). We leave the original number of coarse chart types provided by Plot2Code in parentheses for documentation. Finally, the distribution of chart topics and types of ChartUIE-8K are presented in Fig. 4.

## 4 Empirical Study

### Preliminary and Notations

Denote the chart code-generating LLM, \(f:{\cal Q}\mapsto{\cal O}\). Let a feedback-provider take the output of \(f\), including the executed chart image, then \(\tilde{f}_{AF}:{\cal O}\mapsto{\cal S}_{\mathbb{N}}\in\mathbb{N}_{\cup 0}\), \(f_{AF}:{\cal O}\mapsto{\cal S}_{\mathcal{N}}\in\mathcal{N}\), for TTS, and in-context tuning feedback, respectively. Following Liang et al. (2024), \(h:{\cal O}_{pre}\times{\cal O}_{post}\mapsto\{-2,-1,0,1,2\}\) represents the human post-feedback preference score function. The co-domain refers to \(\succ\): strongly prefer pre-feedback (-2), \(\succeq\): prefer pre-feedback (-1), \(\sim\): indifferent (0), \(\preceq\): prefer post-feedback (1), \(\prec\): strongly prefer post-feedback (2). To demonstrate ChartAF-S (\(\tilde{f}_{AF}\)) and ChartAF-G (\(f_{AF}\))'s utility, we empirically study TTS with ChartAF-S **(Sec. 4.2)** and in-context tuning with ChartAF-G **(Sec. 4.3)**. Fine-grained details are reported in App. D.

### Test-time Scaling with ChartAF

Experiment Set-up.We demonstrate that ChartAF-S is an effective verifier for TTS. Following Snell et al. (2024)'s parallel best-of-\(N\), we generate \(N\in\mathbb{N}\) independent samples and choose the one with the highest score (\(s_{\mathbb{N}}\in{\cal S}_{\mathbb{N}}:=[0,100]\)). Here, the unit of inference budget is \(N\)s. Let memory set \({\cal M}:=\{o_{1},\cdots,o_{N}\}\), \(o\in{\cal O}\), hold the \(N\) independent outputs of \(f\). Then, we experimentally show that

\[\hat{o}:=\underset{o~{}\in~{}{\cal M}}{\text{arg max}}~{}\tilde{ f}_{AF}(o), \tag{1}\] \[\quad\text{if}~{}\hat{o}_{i}>\hat{o}_{j},~{}i\neq j\in\mathbb{ N},\] \[\Rightarrow h(\hat{o}_{j:=1}:=f(q),\hat{o}_{i})>0, \tag{2}\]

across \(q\in{\cal Q}\).

We employ a double-blind 120 human study (of which 77 pass the rigorous sanity check) to compare pre-TTS (\(N:=1\)) and post-TTS (\(N:=4\)) preference scores. Each participant is presented with a random TTS sample. For this experiment we set both \(f\) and \(\tilde{f}_{AF}\) backbone as GPT 4o to emphasize that \(\tilde{f}_{AF}\) does not have to be a superior model for ChartAF-S to be useful.

Results.First, the scaling curve is depicted in Fig. 5. The positive slope highlights that ChartAF can be effectively used as a TTS verifier. To rigorously verify this claim, the distribution of the human study is presented in Tab. 2. Going from a median \(s_{\mathbb{N}}\) of 40.47\(\rightarrow\)62.75 (pre-\(\rightarrow\)post-TTS) leads to 74% strongly preferring the post-TTS and 10.4% preferring post-TTS. This is a strong indication that \(s_{\mathbb{N}}\) is closely proxying human preferences.

### In-context Tuning with ChartAF

Experiment Set-up.For a comprehensive empirical study we consider four LLMs: **(i)** GPT 4o (Achiam et al., 2023), **(ii)** Claude 3.5 Sonnet Anthropic (2024), **(iii)** llama 3.1 70b Dubey et al. (2024), and **(iv)** Gemma 2 27B Team et al. (2019).

Figure 3: ChartUIE-8K curation schematic diagram.

 

2024)--of which **(i)** and **(ii)** are closed-source and **(iii)** and **(iv)** are open-source. The two closed-source models are used as \(f_{AF}\) backbones, while all four models are used as \(f\). Detailed LLM configurations are provided in App. E.

Our empirical study employs a double-blind human study of 60 queries with 120 participants, 77 of whom pass the rigorous sanity check. 120 individuals are required for 60 queries as we test the two \(f_{AF}\) independently. Within the 60 queries, 15 are designated for each of the four \(f\). For representative sampling, we random sample the 15 queries with two constraints: (1) each of the 15 queries ask for different chart types, (2) 1:1 ratio of [**U1**] and [**U2**] queries. Concretely, our experiments show

\[h(f(q),f(q\oplus f_{AF}(q)))>\]

\[h(f(q),f(q\oplus f^{baseline}_{AF}(q))) \tag{3}\]

across \(q\in\mathcal{Q}\), where \(\oplus\) denotes string concatenation.

Baselines.We identify four baselines: **(1)** ChartX Xia et al. (2024), **(2)** MatPlotBench Yang et al. (2024), **(3)** Plot2Code Wu et al. (2024), and **(4)** ChatEval Chan et al. (2024) that can provide feedback for chart generation. **(1)****(2)****(3)** are chart generation specific feedback providers, while **(4)** is a generalist. One advantage of **(1)****(2)****(3)** is its token cost light nature. Therefore, for a fair comparison vis-a-vis token cost, we enhance these baselines with Auto-Chain-of-Thought (A-CoT) Zhang et al. (2023) and Self-Consistency (SC) Wang et al. (2023). Beforehand, we ran preliminary studies with A-CoT, SC, and Self-Refine Madaan et al. (2023), and found that A-CoT+SC performed best. To solidify the utility of CharTAF we include two additional baselines. The first is Naive Feedback (NF) which is zero-shot asking another LLM to give feedback Yang et al. (2024). The second is skipping the feedback stage, and instead directly adding A-CoT to \(f\) (Skip+A-CoT).

Results.We present our findings in Tab. 3 including (input plus output) token costs. ChartaF ranks **first** 13 (out of 16) times and second for the remaining three. This level of consistency across four different models accentuates ChartaF's utility across large and smaller \(f\). Furthermore, regardless of \(f_{AF}\) backbone, post-feedback always (8 out of 8) results in improvement, \(\mu>0\), spotlighting ChartAF's universality. Detailed qualitative examples are provided on the project site.

It is important to note that ChartAF's distinguished performance cannot be trivially matched by enhancing baselines with state-of-the-art prompting methods. While the three enhanced baselines use **128%**ChartX+A-CoT+SC), **85%**MatPlotBench+A-CoT+SC), **70%**Plot2Code+A-CoT+SC) more tokens on average than ChartAF, they fail to be competitive. At the time of ex

Figure 4: ChartUIE-8K distribution. Top 10 topics and types are explicitly depicted while the remaining is classified as others. (\(n\)) is the number of samples out of 8028.

Figure 5: Test-time scaling with ChartAF as a verifier. Raising \(N\) leads to improved generations.

 

perimentation, an evaluation for a single query costed $0.1 and $0.12 on average using GPT4o and Claude3.5 Sonnet, respectively.

### ChartUIE-8K Experiments

Experiment Set-up.We provide evidence than our novel evaluation set is of high-quality, and potentially more realistic than past benchmarks. We employ a double-blind human study of 130 participants (89 of whom pass the rigorous sanity check). Each participant is given a randomized chart image they are imagining, and asked to prompt an LLM their initial instructions. As the factors that comprise query realism are qualitative in nature, we analyze three dimensions that can be quantitatively captured: (i) the word count distribution, (ii) % of respondants preferring ChartUIE-8K's interaction, and (iii) % of respondants who think ChartUIE-8K's user emulation is realistic. Lastly, the participant is asked whether the extra QA cycle is desirable and realistic when interacting with an LLM. Details of the human study is provided in App. D.4.

Results.We visualize the results in Fig. 6. In terms of word count, ChartUIE-8K (green) most closely matches the ground truth distribution of the study of LLM users (purple). Please see some qualitative examples of the instructions the users gave in App. B.2. Notably, 94% of respondents prefer the extra QA cycle, and 93% believe ChartUIE's user emulation is realistic. We also qualitatively observe, as presented in App. C, existing evaluation sets' queries are too technical to be realistic.

## 5 Discussion and Impact of C\({}^{2}\)

### ChartAF Enables Scalable Feedback

Scalability.As described in Sec. 1, the lack of training data, \(\langle\text{instruction},\text{data},\text{code}\rangle\), can be mitigated by an effective reference-free feedback provider that only requires \(\langle\text{instruction},\text{data}\rangle\). However, as indicated by the predominance of red in Tab. 3, existing feedback providers perform poorly under this reference-free regime. Furthermore, increasing token usage via current methods fails to resolve this issue. ChartAF addresses this scalability barrier, as demonstrated by the experimental results in in-context tuning and TTS.

\begin{table}
\begin{tabular}{|l|r r|r r|r r|r r|c|} \hline \hline \(f\) & \multicolumn{2}{c|}{GPT 4o} & \multicolumn{2}{c|}{Claude 3.5 Sonnet} & \multicolumn{2}{c|}{Llama 3.170b} & \multicolumn{2}{c|}{Geema 2 27B} & \multicolumn{1}{c|}{\multirow{2}{*}{\(f_{AF}\)}} \\ \cline{1-1} \cline{6-11} Statistic & \(\mu\) & \(\tilde{\mu}\) & \(\mu\) & \(\tilde{\mu}\) & \(\mu\) & \(\tilde{\mu}\) & \(\mu\) & \(\tilde{\mu}\) & Token Cost \\ \hline \(f_{AF}\) Backbone & \multicolumn{11}{c}{} & \multicolumn{11}{c}{} & \multicolumn{11}{c}{} & GPT 4o & & & & & \\ \hline CharTAF (**ours**) & 0.5 & **1** & **0.5** & **1.5** & **1.308** & **2** & **0.333** & **1** & 14013\(\pm\)1155 \\ ChatEval & -0.375 & -1 & -0.583 & -0.5 & -0.615 & -1 & -1.222 & -2 & 70195\(\pm\)8121 \\ ChartX+A-CoT+SC & -0.25 & -1 & -0.167 & 0 & -1.154 & -2 & 0.111 & **1** & 31791\(\pm\)4604 \\ MatPlotBench+A-CoT+SC & -0.25 & 0 & -0.5 & -1 & -0.231 & 0 & -0.556 & -1 & 24773\(\pm\)2149 \\ PlotZCode+A-CoT+SC & -0.75 & -1.5 & -0.25 & 0 & -0.385 & 0 & -1 & -1 & -22519\(\pm\)2122 \\ Chartx & 0.5 & 0.5 & 0.083 & 1 & -0.385 & 0 & -1.111 & -2 & 1665\(\pm\)235 \\ MatPlotBench & 0.25 & 0.5 & -0.5 & -1 & -0.385 & 0 & -0.444 & 0 & 1653\(\pm\)284 \\ PlotZCode & -0.5 & -0.5 & -0.167 & -0.5 & -0.077 & 0 & -0.667 & -1 & 1604\(\pm\)277 \\ NF & **0.75** & **1** & -0.583 & -1 & -0.692 & -2 & -0.333 & -2 & 2739\(\pm\)385 \\ Skip+A-CoT & -1.375 & -2 & -1 & -1 & -1.692 & -2 & -1.667 & -2 & — \\ \hline \(f_{AF}\) Backbone & \multicolumn{11}{c}{} & \multicolumn{11}{c}{} & \multicolumn{11}{c}{} & \multicolumn{11}{c}{} & \multicolumn{11}{c}{} & \multicolumn{11}{c}{} & \multicolumn{11}{c}{} & \multicolumn{11}{c}{} & \multicolumn{11}{c}{} \\ \hline \hline ChARTAE (**ours**) & **0.167** & 0 & 0.273 & **1** & **1.273** & **2** & **1** & **2** & 14927\(\pm\)981 \\ ChatEval & 0 & **0.5** & -1 & -1 & 0 & 0 & 0.143 & 1 & 74369\(\pm\)8397 \\ ChartX+A-CoT+SC & -0.667 & -1 & -0.727 & -2 & 0.818 & 1 & -0.571 & -1 & 34210\(\pm\)4084 \\ MatPlotBench+A-CoT+SC & -1.333 & -1.5 & 0.273 & **1** & -0.455 & -1 & -1 & -2 & 28980\(\pm\)2608 \\ PlotZCode+A-CoT+SC & -0.667 & -1.5 & -0.727 & -1 & 0.273 & 1 & -0.286 & 0 & 26849\(\pm\)2520 \\ ChartX & -0.167 & 0 & -0.364 & 0 & -0.545 & -1 & -1.143 & -1 & 1753\(\pm\)277 \\ MapItQBench & -0.5 & -0.5 & **0.364** & 0 & -0.545 & -1 & 0 & 0 & 2040\(\pm\)407 \\ PlotZCode & -1.167 & -1.5 & -0.273 & 0 & 0.182 & 0 & -0.143 & 0 & 1851\(\pm\)412 \\ NF & -1.167 & -1.5 & -1.727 & -2 & 0 & 1 & -1.571 & -2 & 2739\(\pm\)562 \\ Skip+A-CoT & -1 & -1 & -1.182 & -2 & -1.636 & -2 & -1.571 & -2 & — \\ \hline Color-coded: \(\tilde{\mu}\)/\(\xi=1.5\): \(=1.5<\mu,\tilde{\mu}<0\): \(\mu,\tilde{\mu}=0\) : \(0\leq\mu,\tilde{\mu}<1.5\): \([\mu,\tilde{\mu}>1.5]\) & \multicolumn{1}{c}{} & & & & \\ \hline \end{tabular}
\end{table}
Table 3: Empirical study preference scores as presented in Sec. 4.1. \(\mu\) represents mean, \(\hat{\mu}\) represents median. Bold is the best result across the column, and underlined is the second-best. Total (input plus output) token costs per query evaluation are reported as \(\mu\pm\sigma\).

 

In-context Tuning with ChartAF.ChartAF performs best among existing methods for nearly all \(f\) and \(f_{AF}\) (Tab. 3). This is particularly notable, as in-context tuning is often challenging for smaller models. Even when the feedback is helpful, the intrinsic limitations of \(f\)--such as small model size or insufficient tuning to follow instructions--may result in \(f\) inaccurately reflecting the feedback. Despite these challenges, the granular feedback of ChartAF effectively improves smaller models (Llama 3.170b, Gemma 2 27b). In fact, Llama 3.170b and Gemma 2 27b on average experience greatest performance improvements after incontext tuning with ChartAF, likely due to their weaker base performance that allows for larger post-tuning gains.

Towards Complex Task Verification.Additionally, ChartAF's reference-free TTS performance shows significant strength. TTS has been a paradigm-shifting development as it introduces a novel neural scaling axis. While previous scaling laws focused on data and training compute, recent works show that similar scaling laws apply to the inference compute axis. However, current TTS is limited to tasks with reliable and cheap verifiers [45, 8], emphasizing the salience of fast, reliable, and cost-effective verifiers [69, 70]. ChartAF is the first effective demonstration of a chart generation verifier within the TTS framework.

We encourage future works to build upon our approach to advance TTS for chart generation and other similar tasks. We used the simplest approach for TTS to empirically prove the verifier's effectiveness. More advanced TTS for chart generation is an open problem.

Scaling Frontier Models with ChartAF.Notably, ChartAF is not a distillation method transferring knowledge from larger\(\rightarrow\)smaller models. In Tab. 3, ChartAF remains effective even for \(\langle f,f_{AF}\rangle\) pairs where \(f\) and \(f_{AF}\) are similarly performing models. Furthermore, TTS is demonstrated using the same backbone LLM. Such self-improving LLMs are indispensable in advancing frontier models [11].

Scaling without Parameter Updates.We demonstrate the effectiveness of \(\mathbf{C}^{2}\) with no parameter updates. This is notable as parameter updating LLMs incur a large memory and training throughput cost. Therefore, \(\mathbf{C}^{2}\) is orthogonal to Zadeh et al. zadeh2024, where they focus on automating the instruction (parameter) tuning process for chart generations.

Unlocking Large-scale Data with \(\mathbf{C}^{2}\)Comprehensive Evaluation and Generation.The strong reference-free performance of ChartAF (Sec. 5.1) enables the curation of a cost-effective and diverse query set, ChartUIE-8K. This approach contrasts with the reference-based nature of existing query sets, which suffer from limited diversity (Tab. 1). By pairing ChartAF with ChartUIE-8K, the combined framework of \(\mathbf{C}^{2}\) supports a _broadly_ inclusive evaluation set (Tab. 1, Fig. 4). This enables researchers to evaluate models on a more comprehensive and diverse set of queries. Moreover, through ChartAF, \(\mathbf{C}^{2}\) can generate large-scale, high-quality outputs that significantly improve over previous methods.

Realistic Evaluation.Lastly, it is crucial to create evaluation sets that closely reflect real-world use-cases. As shown by the distributions in Fig. 6, existing query sets often diverge significantly from common user queries, reducing their practical utility. We recommend that future work proposing evaluation sets include rigorous studies (e.g. Sec. 4.4) to ensure their assets are pragmatically aligned with real-world use-cases.

 



## 6 Limitations

We discuss the limitations of this work to ensure full transparency. To our knowledge, we disclose all reasonable information throughout the paper, project site, and github repository.

Code Execution Error.The feedbacks, \(s_{\text{Nl}}\), \(s_{\mathcal{N}}\), presented in this paper is conditioned on the fact that the chart image has been successfully generated from the code. Occasionally, the code fails to execute or executes with an error. Under such circumstances, we allow a maximum of 5 re-generations for the initial \(f\) inference, and a maximum of 3 re-generations for post-feedback \(f\) inference. We document the initial \(f\) inference error rate for each \(f\) in App. F.

Coverage.The coverage of ChartUIE-8K and ChartAF is limited to the English language. Additionally, this study is not conducted with smaller models, e.g. 8B parameter size LLMs. We leave investigating expanded coverage to future works.

## 7 Human Study Ethical Consideration

To our knowledge, we follow best practices in computer science human studies Muller et al. (2014); Muller and Sedley (2015). First, to guarantee the privacy of both researchers and surveyors the study is double-blind, and do not collect any unnecessary data. Second, we clearly indicate at the very start that this is an academic survey for an academic paper. We do not upload the raw data on the public domain to avoid any potential unethical usage. Third, surveyors voluntarily conduct the surveys and can choose to leave at any time. Fourth, we ensure that the surveyors are compensated fairly. While we do not include any geographic restrictions, we pay \(\$14\) per hour, well above the U.S. federal minimum wage of \(\$7.25\) Henderson (2024) as of the surveys. We pay surveyors within 48 hours of completing the survey. Finally, our surveys do not contain explicit or triggering content.

### Acknowledgement

We would like to thank Yun Ho Ro for his assistance with the figures and Jian Kim for her help in preparing the human study survey. This work was supported in part by the Yonsei University Research Fund of 2024-22-0058, the National Research Foundation of Korea (NRF) Grant funded by the Korea government (MSIT) (RS-2024-00353125), and the Institute of Information and Communications Technology Planning and Evaluation (IITP) Grant, Artificial Intelligence Graduate School Program, Yongei University (RS-2020-II201361) and KAIST (RS-2019-II190075).





## References

* Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. _arXiv preprint arXiv:2303.08774_.
* Ajani et al. (2022) Kiran Ajani, Elsie Lee, Cindy Xiong, Cole Nussbaumer Knaflic, William Kemper, and Steven Franconeri. 2022. Declutter and focus: Empirically evaluating design guidelines for effective data communication. _IEEE Transactions on Visualization and Computer Graphics_, 28(10):351-364.
* Andry et al. (2021) Tiffany Andry, Christophe Hurter, Francois Lambotte, Pierre Fastrez, and Alexandru Telea. 2021. Interpreting the effect of embellishment on chart visualizations. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_, pages 1-15.
* Anthropic (2024) Anthropic. 2024. The claude 3 model family: Opus, sonnet, haiku.
* Bach et al. (2023) Benjamin Bach, Mandy Keck, Fateme Rajabiyazdi, Taitana Losev, Isabel Meirelles, Jason Dykes, Robert S Laramee, Mashael Alkadi, Christina Stoiber, Samuel Huron, et al. 2023. Challenges and opportunities in data visualization education: A call to action. _IEEE Transactions on visualization and computer graphics_.
* Bateman et al. (2010) Scott Bateman, Regan L Mandryk, Carl Gutwin, Aaron Genest, David McDine, and Christopher Brooks. 2010. Useful junk? the effects of visual embellishment on comprehension and memorability of charts. In _Proceedings of the SIGCHI conference on human factors in computing systems_, pages 2573-2582.
* Bi et al. (2024) Zhangqian Bi, Yao Wan, Zheng Wang, Hongyu Zhang, Batu Guan, Fangxin Lu, Zili Zhang, Yulei Sui, Hai Jin, and Xuanhua Shi. 2024. Iterative refinement of project-level code context for precise code generation with compiler feedback. _Preprint_, arXiv:2403.16792.
* Borgo et al. (2012) Rita Borgo, Alfie Abdul-Rahman, Farhan Mohamed, Philip W Grant, Irene Reppa, Luciano Floridi, and Min Chen. 2012. An empirical study on using visual embellishments in visualization. _IEEE Transactions on Visualization and Computer Graphics_, 18(12):2759-2768.
* Bressa et al. (2024) Nathalia Bressa, Jordan Louis, Wesley Willett, and Samuel Huron. 2024. Input visualization: Collecting and modifying data with visual representations. In _Proceedings of the CHI Conference on Human Factors in Computing Systems_, pages 1-18.
* Brown et al. (2024) Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V Le, Christopher Re, and Azalia Mirhoiseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. _arXiv preprint arXiv:2407.21787_.
* Cavthon and Moore (2007) Nick Cavthon and Andrew Vande Moere. 2007. The effect of aesthetic on the usability of data visualization. In _2007 11th International Conference Information Visualization (IV'07)_, pages 637-648. IEEE.
* Chan et al. (2024) Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2024. Chateval: Towards better LLM-based evaluators through multi-agent debate. In _The Twelfth International Conference on Learning Representations_.
* Chen et al. (2018) Siming Chen, Jie Li, Gennady Andrienko, Natalia Andrienko, Yun Wang, Phong H Nguyen, and Cagatay Turkay. 2018. Supporting story synthesis: Bridging the gap between visual analytics and storytelling. _IEEE transactions on visualization and computer graphics_, 26(7):2499-2516.
* Chen et al. (2023) Yongrui Chen, Shenyu Zhang, Guilin Qi, and Xinnan Guo. 2023. Parameterizing context: Unleashing the power of parameter-efficient fine-tuning and in-context tuning for continual table semantic parsing. In _Advances in Neural Information Processing Systems_, volume 36, pages 17795-17810. Curran Associates, Inc.
* Chi et al. (2015) Ming-Te Chi, Shih-Syun Lin, Chiang-Yi Chen, Chao-Hung Lin, and Tong-Yee Lee. 2015. Morphable word clouds for time-varying text data visualization. _IEEE transactions on visualization and computer graphics_, 21(12):1415-1426.
* Choe et al. (2017) Eun Kyoung Choe, Bongshin Lee, Haining Zhu, Nathalie Henry Riche, and Dominikus Baur. 2017. Understanding self-reflection: how people reflect on personal data through visual data exploration. In _Proceedings of the 11th EAI International Conference on Pervasive Computing Technologies for Healthcare_, PervasiveHealth '17, page 173-182, New York, NY, USA. Association for Computing Machinery.
* Dibia (2023) Victor Dibia. 2023. Lida: A tool for automatic generation of grammar-agnostic visualizations and infographics using large language models. In _The 61st Annual Meeting Of The Association For Computational Linguistics_.
* Dubey et al. (2024) Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishekd Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. _arXiv preprint arXiv:2407.21783_.
* Ellis and Dix (2007) Geoffrey Ellis and Alan Dix. 2007. A taxonomy of clutter reduction for information visualisation. _IEEE transactions on visualization and computer graphics_, 13(6):1216-1223.
* Figueiras (2013) Ana Figueiras. 2013. A typology for data visualization on the web. In _2013 17th International conference on information visualisation_, pages 351-358. IEEE.
* Fox and Hendler (2011) Peter Fox and James Hendler. 2011. Changing the equation on scientific data visualization. _Science_, 331(6018):705-708

Joshua Gorniak, Jacob Ottiger, Donglai Wei, and Nam Wook Kim. 2023. Vizability: Multimodal accessible data visualization with keyboard navigation and conversational interaction. In _Adjunct Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology_, pages 1-3.
* Gudibande et al. (2024) Arnav Gudibande, Eric Wallace, Charlie Victor Snell, Xinyang Gang, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2024. The false promise of imitating proprietary language models. In _The Twelfth International Conference on Learning Representations_.
* Han et al. (2023) Yucheng Han, Chi Zhang, Xin Chen, Xu Yang, Zhibin Wang, Gang Yu, Bin Fu, and Hanwang Zhang. 2023. Chartllama: A multimodal llm for chart understanding and generation. _arXiv preprint arXiv:2311.16483_.
* Harrison et al. (2015) Lane Harrison, Katharina Reinecke, and Remco Chang. 2015. Infographic aesthetics: Designing for the first impression. In _Proceedings of the 33rd Annual ACM conference on human factors in computing systems_, pages 1187-1190.
* Healey (1996) Christopher G Healey. 1996. Choosing effective colours for data visualization. In _Proceedings of Seventh Annual IEEE Visualization'96_, pages 263-270. IEEE.
* Henderson (2024) Kaitlyn Henderson. 2024. The crisis of low wages: Who earns less than S17 an hour in the us in 2024?
* Hess (2022) Kosma Hess. 2022. 80 types of charts & graphs for data visualization (with examples). In _darylon_.
* Hu et al. (2023) Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A Smith. 2023. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 20406-20417.
* Huang et al. (2023) Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2023. Large language models can self-improve. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 1051-1068, Singapore. Association for Computational Linguistics.
* Islam and Jin (2019a) Mohaiminul Islam and Shangzhu Jin. 2019a. An overview of data visualization. In _2019 International Conference on Information Science and Communications Technologies (ICISCI)_, pages 1-7.
* Islam and Jin (2019b) Mohaiminul Islam and Shangzhu Jin. 2019b. An overview of data visualization. In _2019 International Conference on Information Science and Communications Technologies (ICSICT)_, pages 1-7.
* Jung et al. (2017) Daekyoung Jung, Wonjae Kim, Hyunjoo Song, Jeong-in Hwang, Bongshin Lee, Bohyoung Kim, and Jinwook Seo. 2017. ChartSense: Interactive data extraction from chart images. In _Proceedings of the 2017 chi conference on human factors in computing systems_, pages 6706-6717.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. _arXiv preprint arXiv:2001.08361_.
* Kim et al. (2021) Dae Hyun Kim, Vidya Settlur, and Maneesh Agrawala. 2021. Towards understanding how readers integrate charts and captions: A case study with line charts. In _Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems_, pages 1-11.
* Kim and Park (2013) Hoyoung Kim and Jin Wan Park. 2013. Topics on aesthetic data visualization: viewpoints, interpretation, and alternative senses. In _SIGGRAPH Asia 2013 Art Gallery_, pages 1-7.
* Kim et al. (2024) Tae Soo Kim, Yoonjoo Lee, Jamin Shin, Young-Ho Kim, and Juho Kim. 2024. Evalm: Interactive evaluation of large language model prompts on user-defined criteria. In _Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems_, CHI '24, New York, NY, USA. Association for Computing Machinery.
* Kirstain et al. (2023) Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. 2023. Pick-a-pic: An open dataset of user preferences for text-to-image generation. _Advances in Neural Information Processing Systems_, 36:36652-36663.
* Lavric et al. (2017) Primoz Lavric, Ciril Bohak, and Mutija Marolt. 2017. Collaborative view-aligned annotations in web-based 3d medical data visualization. In _2017 40th International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)_, pages 259-263. IEEE.
* Lee et al. (2020) Bongshin Lee, Eun Kyoung Choe, Petra Isenberg, Kim Marriott, and John Stasko. 2020. Reaching broader audiences with data visualization. _IEEE computer graphics and applications_, 40(2):82-90.
* Lee et al. (2015) Bongshin Lee, Nathalie Henry Riche, Petra Isenberg, and Sheelaph Carpendale. 2015. More than telling a story: Transforming data into visually shared stories. _IEEE computer graphics and applications_, 35(5):84-90.
* Lee et al. (2012) Sungkil Lee, Mike Sips, and Hans-Peter Seidel. 2012. Perceptually driven visibility optimization for categorical data visualization. _IEEE Transactions on visualization and computer graphics_, 19(10):1746-1757.
* Liang et al. (2024) Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, et al. 2024. Rich human feedback for text-to-image generation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 19401-19411.
* Liu et al. (2022) Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning 

across scales and tasks. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 61-68.
* Madaan et al. [2023] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-refine: Iterative refinement with self-feedback. In _Thirty-seventh Conference on Neural Information Processing Systems_.
* Maddigan and Susnjak [2023] Paula Maddigan and Teo Susnjak. 2023. Chat2VIS: Generating data visualizations via natural language using chatpot, codex and gpt-3 large language models. _IEEE Access_, 11:45181-45193.
* Midway [2020] Stephen R Midway. 2020. Principles of effective data visualization. _Patterns_, 1(9).
* Moured et al. [2024] Omar Moured, Sara Alzalabny, Anas Osman, Thorsten Schwarz, Karin Muller, and Rainer Stiefelhagen. 2024. Chartformer: A large vision language model for converting chart images into tactile accessible svgs. In _International Conference on Computers Helping People with Special Needs_, pages 299-305. Springer.
* Muller et al. [2014] Hendrik Muller, Aaron Sedley, and Elizabeth FerralNunge. 2014. Survey research in hci. _Ways of Knowing in HCI_, pages 229-266.
* Muller and Sedley [2015] Hendrik Muller and Aaron Sedley. 2015. Designing surveys for hci research. In _CHI '15 Extended Abstracts on Human Factors in Computing Systems_, pages 2485-2486, New York, NY, USA.
* Narechania et al. [2020] Arpit Narechania, Arjun Srinivasan, and John Stasko. 2020. NL4DV: A toolkit for generating analytic specifications for data visualization from natural language queries. _IEEE Transactions on Visualization and Computer Graphics_, 27(2):369-379.
* OpenAI [2024] OpenAI. 2024. Learning to reason with llms.
* Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in neural information processing systems_, 35:27730-27744.
* Parsons [2021] Paul Parsons. 2021. Understanding data visualization design practice. _IEEE Transactions on Visualization and Computer Graphics_, 28(1):665-675.
* Quadri et al. [2024] Ghulam Jilani Quadri, Arran Zeyu Wang, Zhehao Wang, Jennifer Adorno, Paul Rosen, and Danielle Albers Szafir. 2024. Do you see what i see? a qualitative study eliciting high-level visualization comprehension. In _Proceedings of the CHI Conference on Human Factors in Computing Systems_, pages 1-26.
* Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sasry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In _International conference on machine learning_, pages 8748-8763. PMLR.
* Ren et al. [2017] Donghao Ren, Matthew Brehmer, Bongshin Lee, Tobias Hollerer, and Eun Kyoung Choe. 2017. ChartAccent: Annotation for data-driven storytelling. In _2017 IEEE Pacific Visualization Symposium (PacificVis)_, pages 230-239. Ieee.
* Rhyne [2017] Theresa-Marie Rhyne. 2017. Applying color theory to digital media and visualization. In _Proceedings of the 2017 CHI conference extended abstracts on human factors in computing systems_, pages 1264-1267.
* Rodriguez et al. [2015] Maria Teresa Rodriguez, Sergio Nunes, and Tiago Devezas. 2015. Telling stories with data visualization. In _Proceedings of the 2015 Workshop on Narrative & Hypertext_, pages 7-11.
* Rosenholtz et al. [2005] Ruth Rosenholtz, Yuanzhen Li, Jonathan Mansfield, and Zhenlan Jin. 2005. Feature congestion: a measure of display clutter. In _Proceedings of the SIGCHI conference on Human factors in computing systems_, pages 761-770.
* Roziere et al. [2023] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Got, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. 2023. Code llama: Open foundation models for code. _arXiv preprint arXiv:2308.12950_.
* Sah et al. [2024] Subham Sah, Rishab Mitra, Arpit Narechania, Alex Endert, John Stasko, and Wenwen Dou. 2024. Generating analytic specifications for data visualization from natural language queries using large language models. _Preprint_, arXiv:2408.13391.
* Skog et al. [2003] Tobias Skog, Sara Ljungblad, and Lars Eriks Holmquist. 2003. Between aesthetics and utility: designing ambient information visualizations. In _IEEE Symposium on Information Visualization 2003 (IEEE Cat. No. 03TH8714)_, pages 233-240. IEEE.
* Snell et al. [2024] Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. 2024. Scaling llvm test-time compute optimally can be more effective than scaling model parameters. _arXiv preprint arXiv:2408.03314_.
* Stokes et al. [2023] Chase Stokes, Cindy Xiong Bearfield, and Marti A Hearst. 2023. The role of text in visualizations: How annotations shape perceptions of bias and influence predictions. _IEEE Transactions on Visualization and Computer Graphics_.
* Stokes et al. [2022] Chase Stokes, Vidya Setlur, Bridget Cogley, Arvind Satyanarayan, and Marti A Hearst. 2022. Striking a balance: Reader takeaways and preferences when integrating text and charts. _IEEE Transactions on Visualization and Computer Graphics_, 29(1):1233-1243 

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Leonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Rame, et al. 2024. Gemma 2: Improving open language models at a practical size. _arXiv preprint arXiv:2408.00118_.
* Tennekes and de Jonge (2014) Martijn Tennekes and Edwin de Jonge. 2014. Tree colors: color schemes for tree-structured data. _IEEE transactions on visualization and computer graphics_, 20(12):2072-2081.
* Tian et al. (2024) Yuan Tian, Weiwei Cui, Dazhen Deng, Xinjing Yi, Yurun Yang, Haidong Zhang, and Yingcai Wu. 2024. Charfept: Leveraging llms to generate charts from abstract natural language. _IEEE Transactions on Visualization and Computer Graphics_, page 1-15.
* Uesato et al. (2022) Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. _arXiv preprint arXiv:2211.14275_.
* Vazquez (2024) Pere-Pau Vazquez. 2024. Are llms ready for visualization? _arXiv preprint arXiv:2403.06158_.
* Wang et al. (2024a) Ante Wang, Linfeng Song, Ye Tian, Baolin Peng, Dian Yu, Haitao Mi, Jinsong Su, and Dong Yu. 2024a. Litesearch: Efficacious tree search for llvm. _arXiv preprint arXiv:2407.00320_.
* Wang et al. (2024b) Chaojie Wang, Yanchen Deng, Zhiyi Lv, Shuicheng Yan, and An Bo. 2024b. Q*: Improving multi-step reasoning for llms with deliberative planning. _arXiv preprint arXiv:2406.14283_.
* Wang et al. (2024c) Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2024c. MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. In _The Twelfth International Conference on Learning Representations_.
* Wang et al. (2023)Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In _The Eleventh International Conference on Learning Representations_.
* Wang et al. (2020) Zezhong Wang, Lovisa Sundin, Dave Murray-Rust, and Benjamin Bach. 2020. Cheat sheets for data visualization techniques. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_, pages 1-13.
* Wu et al. (2024) Chengyue Wu, Yixiao Ge, Qiushan Guo, Jiahao Wang, Zhixuan Liang, Zeyu Lu, Ying Shan, and Ping Luo. 2024. Plot2code: A comprehensive benchmark for evaluating multi-modal large language models in code generation from scientific plots. _arXiv preprint arXiv:2405.07990_.
* Xia et al. (2024) Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, et al. 2024. Chartx & chartvlm: A versatile benchmark and foundation model for complicated chart reasoning. _arXiv preprint arXiv:2402.12185_.
* Xu et al. (2024a)Anran Xu, Shitao Fang, Huan Yang, Simo Hosio, and Koji Yatani. 2024a. Examining human perception of generative content replacement in image privacy protection. In _Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems_, CHI '24, New York, NY, USA. Association for Computing Machinery.
* Xu et al. (2024b)Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qingkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. 2024b. Imagreeward: Learning and evaluating human preferences for text-to-image generation. _Advances in Neural Information Processing Systems_, 36.
* Xu et al. (2023) Zhengzhuo Xu, Sinan Du, YYyan Qi, Chengjin Xu, Chun Yuan, and Jian Guo. 2023. Chartbench: A benchmark for complex visual reasoning in charts. _arXiv preprint arXiv:2312.15915_.
* Yang et al. (2023) Fumeng Yang, Yuxin Ma, Lane Harrison, James Tompkin, and David H. Laidlaw. 2023. How can deep neural networks aid visualization perception research? three studies on correlation judgments in scatterplots. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_, CHI '23, New York, NY, USA. Association for Computing Machinery.
* Yang et al. (2024) Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, et al. 2024. Matplotagent: Method and evaluation for llvm-based agentic scientific data visualization. _arXiv preprint arXiv:2402.11453_.
* Zadeh et al. (2024) Fatemeh Pesaran Zadeh, Juyeon Kim, Jin-Hwa Kim, and Gunhee Kim. 2024. Text2chart31: Instruction tuning for chart generation with automatic feedback. In _Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing_, pages 11459-11480.
* Zhang et al. (2023) Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. Automatic chain of thought prompting in large language models. In _The Eleventh International Conference on Learning Representations_.
* Zhao et al. (2021) Jian Zhao, Shenyu Xu, Senthil Chandrasegaran, Chris Bryan, Fan Du, Aditi Mishra, Xin Qian, Yiran Li, and Kwan-Liu Ma. 2021. Charlstory: Automated partitioning, layout, and captioning of charts into comic-style narratives. _IEEE transactions on visualization and computer graphics_, 29(2):1384-1399 



###### Contents

* 1 Introduction
	* 1.1 Related Work
* 2 C\({}^{2}\): ChartAF
	* 2.1 Towards High-performing Feedback
	* 2.2 ChartAF-S (\(f_{AF}\))
	* 2.3 ChartAF-G (\(f_{AF}\))
* 3 C\({}^{2}\): ChartUE-8K
* 4 Empirical Study
	* 4.1 Preliminary and Notations
	* 4.2 Test-time Scaling with ChartAF
	* 4.3 In-context Tuning with ChartAF
	* 4.4 ChartUIE-8K Experiments
* 5 Discussion and Impact of C\({}^{2}\)
	* 5.1 ChartAF Enables Scalable Feedback
	* 5.2 Unlocking Large-scale Data with C\({}^{2}\)
* 6 Limitations
* 7 Human Study Ethical Consideration
* A ChartAF Details
* A.1 Basic Criteria References
* A.2 ChartAF Pseudocode
* B ChartUE-8K Details
* B.1 ChartUIE-8K pseudocode
* B.2 Example User Study Initial Instructions
* C Chart Generation User Query Examples
* D Human Study Details
* D.1 Compensation and Qualification
* D.2 Sanity Check
* D.3 ChartAF Human Study
* D.4 ChartUIE-8K Human Study
* E LLM Configurations
* F Code Error Rate
* G Icon Attribution



## Appendix A ChartAF Details

### Basic Criteria References

See Table 4.

### ChartaF Pseudocode

We present the pseudocode of ChartAF in Alg. 1, along with the prompts used within it. These prompts can be accessed by clicking on the highlighted phrases in the pseudocode.

We first introduce the notations. Let \(inst\) represent the initial instruction, \(qst\) the follow-up questions, and \(ans\) the answers to \(qst\). Let \(d\) represent the dataset, and \(d_{attr}\) its attributes. \(f_{AF}\) denotes the backbone LLM for feedback generation. Let \(mode\) be one of two values: "Scalar" or "Granular." Finally, let \(code_{gen}\) be the code that generates a chart. This code is produced by the chart-generating LLM, using the prompt \(Generate\). The arguments in the prompt--data_path, data, file_index, initial_instruction, questions, and answers--should be set to the data path for the dataset, \(d\), the index for the resulting image, \(inst\), \(qst\), and \(ans\), respectively. Denote \(img\) as the chart generated by executing \(code_{gen}\). The procedure returns either a single scalar value, \(s_{\mathbb{N}}\), or fine-grained natural language feedback, \(s_{\mathcal{N}}\), depending on the value of \(mode\).

```
1:procedureChartAF(\(inst,qst,ans,d_{a},d\), \(f_{AF},mode,code_{gen},img\))
2:\(p_{tpa}\gets TPA_{AF}\)
3:\(p_{tpa}\).format(
4:initial_instruction:=\(inst\),
5:tasks:=\(task\),
6:data:=\(d\)
7:)
8:\(\langle tsk,prps,aud\rangle\gets f_{AF}(p_{tpa})\)
9:\(p_{ctr}\gets Criteria\)
10:\(p_{crt}\).format(
11:initial_instruction:=\(inst\),
12:questions:=\(qst\),
13:answers:=\(ans\),
14:tasks:=\(tsk\),
15:purpose:=\(prps\),
16: audience:=\(aud\)
17:)
18:\(cnt\gets f_{AF}(p_{crt})\)
19:\(p_{qst}\gets Criteria_{Q}\)
```

**Algorithm 1** ChartAF

\begin{table}
\begin{tabular}{l l} \hline \hline
**Criteria Category** & **References** \\ \hline Chart Type & Figueiras (2013); Jung et al. (2017); Islam and Jin (2019b); Midway (2020) \\ Visual Embellishment & Bateman et al. (2010); Borgo et al. (2012); Andry et al. (2021) \\ Text & Chi et al. (2015); Stokes et al. (2022, 2023) \\ Color & Healey (1996); Lee et al. (2012); Tennekes and de Jonge (2014); Rhyne (2017) \\ Annotation & Lee et al. (2015); Ren et al. (2017); Lavivé et al. (2017); Chen et al. (2018); Kim et al. (2021); Zhao et al. (2021) \\ Aesthetics & Skog et al. (2003); Cawthon and Moore (2007); Kim and Park (2013); Harrison et al. (2015) \\ Visual Clutter & Rosenholtz et al. (2005); Ellis and Dix (2007); Ajani et al. (2022) \\ \hline \hline \end{tabular}
\end{table}
Table 4: References that form the theoretical background of the Basic Criteria

```
1:\(p_{qst}\).format(
2:task:=\(tsk\),
3:purpose:=\(prps\),
4:audience:=\(aud\),
5:criteria:=\(crt\)
6:)
7:\(crt_{qst}\gets f_{AF}(p_{qst})\)
8:\(p_{eval}\gets Evaluate\)
9:\(p_{eval}\).format(
10:evaluation_questions:=\(crt_{qst}\),
11:initial_instruction:=\(inst\)
12:)
13:\(s\leftarrow f_{AF}(img,p_{eval})\)
14:\(s_{\mathbb{N}}\leftarrow\text{Ratio of "yes" responses in $s$}\)
15:if\(mode\) is "Scalar" then
16:return\(s_{\mathbb{N}}\)
17:endif
18:\(s_{\mathcal{N}}\leftarrow\text{Feedback in $s$}\)
19:\(\langle trtn,dsc,edit,add\rangle\leftarrow\text{Classification of}\)
20:feedback according to the tag in \(s_{\mathcal{N}}\)
21:\(p_{cf}\gets CF\)
22:\(p_{cf}\).format(
23:initial_instruction:=\(inst\),
24:code:=\(code_{gen}\),
25:attributes:=\(d_{attr}\),
26: retain:=\(rtn\),
27:discard:=\(dsc\),
28:edit:=\(edt\),
29:add:=\(add\)
30:)
31:\(s_{\mathcal{N}}\gets f_{AF}(p_{cf})\)
32:return\(s_{\mathcal{N}}\)
33:endprocedure
```

**Algorithm 1** ChartAF (Continued)

```
1:GenerateGret(Generate)
```
2:You are an expert data visualizer. The following instruction asks you to generate code for data visualization of the underlying data file that we have attached. I will give you the data, but you can ignore some parts from the data if it is not necessary and unrelated to the instruction. Assume that the data file that has been attached in the path "[data_path]" in the generated code. The file format of the data is **f".json or .csv".** Your code should include loading the data file, and check and verify the data type and representation of the data to avoid errors while executing.
3:<start of data format> {data} <end of data format>
4:Your code should also automatically download the final visualization in a lower level directory (contained within the current directory) named "plots_d2c". You MUST name your final generated 

visualization as "[file_index].png". You can freely choose package(s) that work best to make the visualization.

Here is the instruction set:

<start of initial instruction> {initial_instruction} <end of initial instruction>

<start of further instruction> Questions: {questions} Answers: {answers} <end of further instruction> Ensure you use this code format in order to avoid errors, and only give the executable Python Code.

CHARY TRal Purpose and Audience Interest (7/24/10)

You are a data visualization expert. Given the data and user request, your task is to analyze the user request to (1) select the most suitable task that the user is expecting from the list of various tasks in data visualization, (2) specifically figure out the purpose of the user's request in data visualization, and (3) prospective audience of the data visualization.

<start of user request> {initial_instruction} <end of user request>

<start of data format> {data} <end of data format>

<start of various task types> {tasks} <end of various task types>

We use data visualization tasks presented in Choe et al. (2017).

List of Tasks (task)

Show External Context Uncaptured data provided by the self-tracker to understand and explain a phenomenon shown in the data.

Show confirmation Collected data confirms existing knowledge.

Show Contradiction Collected data contradicts existing knowledge.

 

* Focus on Identifying value Explicitly specify the measured value, its range for one or more clearly identified data points, or the difference between two measured values.
* Focus on Identifying extreme Explicitly state the identities of the data points possessing extreme values of the measure variable.
* Focus on Identifying reference Explicitly state the values of categorical variables, labels from the axes, or legends.
* Comparison by Time Segmentation Compare measured values segmented by time.
* Comparison by Multiple services Compare the same data type from two or more services.
* Comparison against external data Bringing in external data for comparison.
* Comparison by Factor Compare measured values by a factor (other than time).
* Comparison by Instances Compare two specific instances.
* Show Trend Describe changes over time.
* Value judgement Convex positive or negative connotations about the data.
* Distribution with variability Explicitly state the variability of measured values.
* Distribution By Category Explicitly describe the variation of measured values across all or most of the values of a categorical variable.
* Correlation Specify the direct relationship between two variables (but not as comparison).
* Outlier Explicitly point out outliers or state the effect of outliers.
* Summarization of data Summary of collected data (such as number of data points, duration of tracking, and averages).
* Prediction/Forecasting Predict the future based on the collected data.

[TABLE:S3.T1][ENDT 

basic criteria, to evaluate a data visualization (chart). These criteria should be based on the user instruction, the user request through questions and answers, tasks at hand, intended purpose, and the prospective audience.

<start of basic criteria>
* Chart Type Choose a chart type that aligns with the given purpose, task, and audience. The chart type should effectively convey the intended message; for example, bar charts are ideal for comparing quantities for limited number of categorical data, while line charts show trends over time. The choice must consider the inherent spacing requirements and the context in which the chart will be used, ensuring clarity and comprehension.
* Visual Embellishment Use embellishments to enhance understanding without overwhelming the data. Visual embellishments, like icons, patterns, or textures, should be used sparingly and purposefully to make the chart memorable and engaging while maintaining a balance that does not distract from the core data.
* Text Prioritize legibility and adhere to consistent textual criteria. Text elements, such as legends, titles, and labels, should be legible and easy to read, with sufficient contrast against the background. Consistent font size, style, and placement should be maintained to create a cohesive visual narrative that guides the audience's understanding.
* Color Use color purposefully and sparingly to convey meaning. Choose a limited palette that enhances readability and highlights key data points, considering color statistics and opponent processing principles (contrasting colors for clarity). This helps ensure accessibility for viewers with color vision deficiencies.
* Annotation Emphasize critical data while minimizing irrelevant details. Use annotations strategically to draw attention to important insights, trends, or outliers, and smooth over or de-emphasize less significant data points, ensuring the chart communicates its key message effectively.
* Aesthetics Tailor aesthetics to the chart's purpose, audience, and context. Consider the chart's purpose, the target audience, and the presentation environment when designing aesthetics, including compact spacing and visual hierarchy. This ensures the chart is both functional and appealing, maximizing its impact and effectiveness
* Visual Clutter Optimize the chart size to fit its content and context, balancing data and available space to prevent clutter or excessive white space while maintaining readability. Manage visual elements by minimizing overcrowding and overlapping, adequately spacing text, data points, and annotations, removing unnecessary details, and maintaining a clean layout to enhance clarity. Segmentation of complex charts or data visualizations can also be employed if the visual complexity is high, breaking down the data into smaller, more manageable parts for easier interpretation. It is important to emphasize key data by using size, color, and opacity to highlight critical insights while downplaying less relevant information for a focused presentation.

<end of basic criteria> 

<start of user instruction> {initial_instruction} <end of user instruction>

<start of user request through QA> {questions} {answers} <end of user request through QA>

<start of tasks> {tasks} <end of tasks>

<start of purpose> {purpose} <end of purpose>

<start of prospective audience> {audience} <end of prospective audience> Note that the interactivity of the chart, file format, credibility and integrity of data source, summary statistics do not need to be considered. The quality of the data visualization to general audience is the only subject to be considered. Think about the essential chart component requirements that align with the task, purpose, and user request.

GIARYAT Generate Grider(Ouestions)

You are an expert critic. You will be given wanted tasks, intended purpose, prospective audience, and established criteria for the chart that you gave in the previous prompt. Your task is to create a list of Yes/No questions that checks if the generated chart satisfies the established criteria. Use the established criteria as a reference, but avoid applying them directly when crafting questions to evaluate the chart.

<start of task> {task} <end of task>

<start of purpose> {purpose} <end of purpose>

<start of prospective audience> {audience} <end of prospective audience>

<start of established criteria> {criteria} <end of established criteria> 

"Yes" should be treated as satisfaction, while "No" should be a dissatisfaction. Here is a detailed protocol for making questions: First, create questions according to the criteria, tasks, purpose, and audience. Extra questions that the criteria do not cover can be generated, yet it must help judge evaluating the chart. Lastly, summarize similar questions and rank the questions so that the first question is the most important and the last question is the least important. Your output should follow the format below:

"" Question 1 : [Question] Question 2 : [Question] ... ""

Citeria-A Evaluation (Evaluating) You are an expert evaluator (judge, critic) of the attached data visualization image.

<start of evaluation questions> {evaluation_questions} <end of evaluation questions> The evaluation questions consist of YES/NO questions; the answer for each question MUST be either YES or NO. Don't give anything else like N/A. With the answers, you need to give feedback. When answering the questions, follow the step-by-step protocol below:

1. Determine and tag whether the question is subjective or fact-checking * Fact-checking Verify if the chart image meets the criteria directly based on the visual content. If the image shows any deviation from the criteria, answer NO. If the image meets the criteria, answer YES. * Subjective Consider whether the image meets the criteria based on visual appeal, clarity, and other subjective measures. Provide reasons for both YES and NO answers. If there is clear evidence to support a YES and no substantial reasons to support a NO, answer YES. Answer NO otherwise.
2. Answer the questions and provide feedback After answering each question, provide feedback explaining your evaluation. List potential improvements categorized as RETAIN, DISCARD, EDIT, or ADD if necessary. Feedback Classification: * RETAIN Identify and specify any elements that should be retained even after the improvement. * DISCARD Identify and specify any elements that should be discarded for better visualization. * EDIT Specify edits needed in the image to satisfy the user's request. Provide examples if applicable. * ADD Identify and specify elements that should be added for better visualization of the user's initial prompt 

To help your task, here is the user's initial prompt.

<start of initial prompt> {initial_instruction} <end of initial prompt>

CITARAT Generate Code Feedback(CH)

You are an expert software engineer on the Quality Assurance team Your task is to provide feedback on the code based on the critic's feedback on the result of the code. The code's goal is to successfully draw a chart, fulfilling the user's needs. You will be given the user's needs, the original code, the critic's feedback, the data attributes, and the resulting image of the code.

Here is the user's needs.

<start of needs> {initial_instruction} <end of needs> Here is the original code.

<start of the code> {code} <end of the code> Here are the data attributes. <start of the attributes> {attributes} <end of the attributes> Here is the critic's feedback.

<start of feedback> Elements to RETAIN -- {retain} Elements to DISCARD -- {discard} Elements to EDIT -- {edit} Elements to ADD -- {add} <end of feedback> 

Your task is to provide feedback on the code for debugging and offering better data visualization. Specifically, focus on cases where the image does not correctly reflect the intended output, even though the code appears correct. Follow these steps:

1. Review the Evaluation Feedback Examine the feedback, especially noting where the image does not align with the expected results despite the code being correct.
2. Analyze the Feedback Determine what changes are necessary in the code to correct errors and enhance the output based on the feedback. If there are potential errors that may occur, feel free to provide feedback on those lines. Again, your task is not only to offer better data visualization but also to debug the code.
3. List your feedback on the code, and make sure such modifications help generate the executable code. Explain the modification, log the lines of code that should be modified, and log lines of new code that can be implemented. When logging the code, log the line number as well, where the original code lies, and where the new code should be put.

 



## Appendix B ChartUIE-8K Details

### ChartUE-8K Pseudocode

We present the pseudocode of ChartUIE-8K in Alg. 2, along with the prompts used within it. These prompts can be accessed by clicking on the highlighted phrases in the pseudocode.

We first introduce the notations. Let \(d\) represent the underlying dataset. If it exists, let \(d_{ttl}\) denote the title of \(d\); otherwise, set \(d_{ttl}\) to "unknown." Similarly, if the topic of \(d\) is provided, let \(d_{lpc}\) represent it. Finally, let \(f_{uie}\) refer to the backbone LLM for ChartUIE-8K. Here, \(f_{uie}:=\)gpt 4o.

The procedure returns a list of queries, \(Q\), generated by the algorithm. Each tuple in \(Q\) includes the initial instruction (\(inst\)), column labels for visualization (data attributes, \(d_{attr}\)), a selected task (\(tsk\)), a visualization purpose (\(prps\)), a text description of the target audience (\(aud\)), follow-up questions (\(qst\)), and answers (\(ans\)). The model \(f_{uite}\) infers \(d_{attr}\), \(tsk\), \(prps\), and \(aud\) from \(inst\) and follow-up questions to clarify user preferences, of which only \(q_{\%}\) are answered.

```
1:procedureUIE-8K(\(d,d_{ttl},d_{lpc},f_{uie}\))
2:\(p_{ct}\gets Select_{ct}\)
3:\(p_{ct}\).format(data:= \(d\))
4:\(C_{type}\gets f_{uie}(p_{ct})\)
5:\(p_{annot}\gets Select_{annot}\)
6:\(p_{annot}\).format(
7: data:=\(d\),
8: data_title:=\(d_{ttl}\),
9: topic:=\(d_{lpc}\)
10: )
11:\(annot\gets f_{uie}(p_{annot})\)
12:\(j_{max}\leftarrow\min\{15,\text{the length of $C_{type}$}\}\)
13:\(Q\leftarrow[]\)
14:for\(j\in\{0,1,...,j_{max}-1\}\)do
15:for\(word\in\{50,100\}\)do
16:if\(j\leq 2\)then
17:\(p_{trg}\gets Trigger_{annot}\)
18:\(p_{trg}\).format(
19: data:=\(d\),
20: annotations:=\(annot\),
21: data_title:=\(d_{ttl}\),
22: topic:=\(d_{lpc}\),
23: chart_type:=\(C_{type}[j]\)
24: word_count:=\(word\)
25: )
```

**Algorithm 2** ChartUIE-8K

```
26:else
27:\(p_{rpg}\gets Trigger_{annot^{\prime}}\)
28:\(p_{trg.}\)format(
29:data:=d,
30:data_title:=\(d_{ttl}\),
31:topic:=\(a_{tpc}\),
32:chart_type:=\(C_{type}[j]\)
33:word_count:=\(word\)
34:)
35:endif
36:\(inst\gets f_{uie}(p_{trg})\)
37:\(p_{tpa}\gets TPA_{uie}\)
38:\(p_{tpa}\)format(
39:initial_instruction:=\(inst\),
40:data:=\(d\),
41:task:=\(task\)
42:)
43:\(\langle d_{a},tsk,prps,aud\rangle\gets f_{uie}(p_{tpa})\)
44:\(p_{qst}\gets Question_{uie}\)
45:\(p_{qq}\)std::format(
46:data:=\(d\),
47:initial_instruction:=\(inst\),
48:attributes:=\(d_{attr}\),
49:audience:=\(aud\),
50:tasks:=\(tsk\),
51:purpose:=\(prps\)
52:)
53:\(qst\gets fuie(p_{qt})\)
54:\(p_{ans}\gets Answer_{uie}\)
55:if\(word==50\)then
56:\(q_{\%}\gets 25\)
57:else
58:\(q_{\%}\gets 50\)
59:end if
60:\(p_{ans}\).format(
61:initial_instruction:=\(inst\),
62:q_percent:=\(q_{\%}\),
63:purpose:=\(prps\),
64:\(f\)response:=\(qst\)
65:)
66:\(ans\gets f_{uie}(p_{ans})\)
67:\(Q.append(\)
68:\(\langle\\ inst,d_{a},tsk,prps,\\ aut,qst,ans\rangle\)
69:)
70:end for
71:end for
72:return\(Q\)
73:end procedure 

We use chart types presented in Hess (2022).

``` Start Type Selection (\(Coded_{A}\)) <start of data example format> {data} <end of data example format> Look at the following list and select one or more chart types appropriate for the data. Try to choose as many different charts as possible. Consider the purpose and chart type that can visualize the data well. But Chart types that are not reasonable for data visualization of the data example format I attached must be excluded. Respond with the chart types that are compatible with the data. Please include only one chart type that is similar to each other. Comparison, Correlation, Part-to-whole & hierarchical, Data over time (temporal), Distribution, Geospatial & other are the purposes, and the chart types below are suitable for the above purpose. Look closely at the characteristics of the data, and the annotation should be one that produces as little clutter as possible. <start of the chart type list> 1. Comparison * Bar chart * Column chart * Grouped bar/column chart * Lollipop chart * Bullet chart * Dot plot * Dumbbell * Pictogram * Icon chart * Range chart * Radial bar chart * Parallel coordinates * Radar chart * Nightingale chart * Waterfall chart * Matrix chart * Small multiples * Word cloud * Slope chart * Table chart * Categorical scatter plot * Quadrant chart
2. Correlation * Heatmap * Bubble chart * Scatter plot 

* Connected scatter plot
* Hexagonal binning
* Contour plot
3. Part-to-whole & hierarchical
* Stacked bar/column chart
* Diverging bar chart
* Population pyramid
* Icon array
* Waffle chart
* Pie chart
* Donut chart
* Semi-circle donut chart
* Marimekko chart
* Treemap
* Circular treemap
* Convex treemap
* Dendrogram
* Venn diagram
* Euler diagram
* Circular gauge
* Sunburst chart
* Funnel & pyramid chart
4. Data over time (temporal)

* Area chart
* Stacked area chart
* Stream graph
* Bump chart
* Bump area chart
* Line chart
* Spline chart
* Step line chart
* Candlestick chart
* Gantt chart
* Barcode chart
* OHLC chart
5. Distribution
* Density plot
* Ridgeline plot
* Horizon chart
* Histogram
* Radial histogram
* Strip plot
* Jitter plot
* One-dimensional heatmap 

* Beeswarm chart
* Box chart
* Violin plot
6. Geospatial & other
* Geographic heatmap
* Choropleth map
* Tile map
* Chord diagram
* Arc diagram
* Sankey
* Network diagram
* Flowchart

<end of the chart type list>

Your response should ONLY contain the chart types. Do not include anything else.

**A Annotation Selection (_foracle\({}_{nmm}\)_)****

``` <start of data example format> {data} <end of data example format> <start of data details format> {data_title} {topic} <end of data details format> Look at the following list and select one or more annotations appropriate for the data. Choose two annotations. <start of the annotation list>
1. Text Annotations: Description: Data-driven text annotations display values linked to chart elements, such as data points in a scatterplot. They draw attention to specific elements by highlighting their values. Purpose: When only some elements are annotated, the intent is to focus the viewer's attention on those before examining others. Other Uses: Non-data-driven annotations can provide context, orientation, or editorial comments.
2. Shapes: Description: Shape annotations include lines, arrows, rectangles, and other shapes. They can highlight or enclose specific chart elements to emphasize or compare them. Data-Driven Use: Some shapes, like trend lines, are calculated from the underlying data.
3. Highlights: Description: Highlights modify the appearance of chart elements (e.g., size, color) to emphasize or reduce their importance. Purpose: Used to distinguish certain elements from others, making them stand out visually <end of the annotation list> 

Look closely at the characteristics of the data, and the annotation should be one that produces as little clutter as possible. Also, refer to the data details to create as practical and realistic instructions as possible. Your response should ONLY contain the annotations, description, purpose, and uses. Do not include anything else.

```
\begin{tabular}{|l|} \hline Enulting with annotations (_Truggent_) \\ \hline You are an expert user emulator. \\  <start of data example format> \\   {data} \\  <end of data example format> \\  <start of annotations format> \\  {annotations} \\  <end of annotations format> \\  <start of data details format> \\  {data_title} \\  {topic} \\  <end of data details format> \\  Given a data format, imagine a chart that visualizes this data as the final output you want from the service provider. It MUST be a chart that can be created using only data columns. Consider what purpose the data has and the practical purpose of visualization and include it in the instructions. You need to imagine a chart with {chart_type} and given annotations that utilizes the data format. If there are multiple given data formats, imagine a chart with {chart_type} and given annotations that utilizes all the data formats. Since you are an amateur user, your instruction will be partially SUBJECTIVE and NOT DETAILED. Also, refer to the data details to create as practical and realistic instructions as possible. Instructions must reflect the context of the data. To emulate a real-world user your instruction should be {word_count} in size (word count). Do not include data path in the instruction. Your response should ONLY contain the user emulated instruction. Do not include anything else. \\ \hline \end{tabular}
``` 
\begin{tabular}{|l|} \hline Emulating without annotations (_Truggent_) \\ \hline You are an expert user emulator. \\  <start of data example format> \\  {data} \\  <end of data example format> \\  <start of data details format> \\  {data_title} \\  {topic} \\  <end of data details format> \\  Given a data format, imagine a chart that visualizes this data as the final output you want from the service provider. It MUST be a chart that can be created using only data columns. You need to imagine a chart with {chart_type} that utilizes the data format. If there are multiple given data formats, imagine a chart with {chart_type} that utilizes all the data formats. Since you are an 

amateur user, your instruction will be partially SUBJECTIVE and NOT DETAILED. Also, refer to the data details to create as practical and realistic instructions as possible. Instructions must reflect the context of the data. To emulate a real-world user your instruction should be {word_count} in size (word count). Your response should ONLY contain the user emulated instruction. Do not include anything else.

[frame=lineskip] Task Purpose and Audience Inference for ULE (\(\textbf{1/P4}\),...) You are a data visualization expert. Given the data and user request, your task is to analyze the user request to figure out the (1) data attributes needed for data visualization, (2) select the most suitable task that the user is expecting from the list of various tasks in data visualization, (3) specifically figure out the purpose of the user's request in data visualization, and (4) prospective audience of the data visualization.

<start of user reques> {initial_instruction} <end of user request>

<start of data format> {data} <end of data format>

<start of various task types> {task} <end of various task types> For data attributes needed in data visualization, store them in query['Data Attribute']. Data attributes MUST match exactly the column names of the data. Store the selected task from the task types in query['Task']. Store the purpose of visualization in query['Purpose']. Store the prospective audience in query['Audience']. Please reply in the same format without altering the key value. {"Data Attribute": None, "Task": None, "Purpose": None, "Audience": None} But, please make sure there is no ' in each keys and values. Use only " for the response. But when you write a value sentence or each data attribute's title, you only can use '. Unless you are writing a sentence or each data attribute's title, you should never include ' in response. If there are multiple pieces of data, there is no need to reveal which file each is. Please consider that JSON conversion must be done properly.

[frame=lineskip] List Preference Questions (\(\textbf{0}\))

You are an expert data visualization analyst. Given data, data attribute, tasks, prospective audience, purposes of the chart(data visualization) from the initial instruction (user request) of the user, you have a 2-step task to do. (1) First, your task is to figure out the essential chart attribute requirements that the chart must have in order to satisfy such tasks and purposes (2) Then, create a list of questions to the user if the user have specific chart attribute preference for effective data visualization.

Do NOT include anything else in your response other than the list of questions. Your questions should be primarily focused on retrieving the user's preferences. Do NOT include any questions related to (1) interactivity of the chart, and (2) the file format of the chart.

<start of data format> 

{data} <end of data format>

 <start of data visualization instruction>  {initial_instruction}  <end of data visualization instruction>

 <start of data attributes>  {attributes}  <end of data attributes>

 <start of prospective audience>  {audience}  <end of prospective audience>

 <start of type of tasks>  {tasks}  <end of type of tasks>

 <start of purpose>  {purpose}  <end of purpose>

 ```

Answer Reference Questions (Ansumer)

You are an expert user emulator.

 <start of user request>  {initial_instruction}  <end of user request>

 Imagine the best result that can be achieved based on the given instructions and is the final output you want from the service provider with your purpose given below. Based on your imagination you are to respond to the service provider. Since you are an amateur user you should concretely answer {q_percent}% of the questions you think are most important, and the remaining questions should be answer with uncertainty, e.g. "I am not sure", "I do not know", "I have no specific preference". Finally, remember that you are looking for an image, not an interactive data visualization.

 <start of purpose>  {purpose}  <end of purpose>

 <start service provider message>  {\(f\_response\)}  <end service provider message> Also, remember that the service provider can not see the image you have access to. Your response should ONLY contain the user emulated response. Do not include anything else.

### Example User Study Initial Instructions

We provide user study initial instruction examples curated by the LLM user participants below 

Use Initial Interaction Example1

Please make a bar chart of R&D investment (in billions of USD) according to country. Make each bar green, except for Japan and India, which you should make blue. Title the chart, "R&D Investment by Country." Include a red line that connects the maximum of each bar. And add a blue dash line where the average is across the whole chart (including the average value in Billions out to two decimal places). User Infital Instruction Example2

Chart grades over 12 months. Have different colors for each student and label by names. Y axis will be percentage and x axis months. Title will be Students Grades Over the Year.

 



## Appendix C Chart Generation User Query Examples

```
Plot2Code Example
``` The figure generated by the provided Python code consists of four subplots, each displaying a different transformation of the same image. The image is created using a mathematical function that generates a grid of values. The first subplot shows the image rotated by 30 degrees. The second subplot shows the image skewed by 30 degrees along the x-axis and 15 degrees along the y-axis. The third subplot shows the image reflected along the x-axis and scaled by 0.5 along the y-axis. The fourth subplot combines all these transformations: rotation, skewing, scaling, and translation. Each subplot has a yellow dashed line indicating the intended extent of the image. The x-axis limits are set from -5 to 5, and the y-axis limits are set from -4 to 4 for all subplots. To recreate this figure, you would need to generate the same image and apply the same transformations in each subplot. The transformations are applied using an affine transformation matrix, which allows for rotation, skewing, scaling, and translation. The specific parameters for each transformation are mentioned above. The figure is displayed using the matplotlib library in Python, which would need to be installed and imported to recreate the figure. The numpy library is also used to generate the image and would need to be installed and imported as well. The image is generated using a mathematical function that creates a grid of values between -3.0 and 3.0 with a step of 0.25. This function uses the numpy exp function to calculate the exponential of the negative square of each value in the grid. The result is a 2D array of values, which is used as the image in the subplots. The transformations applied to the image in each subplot are done using the Affine2D function from the matplotlib.transforms module. This function creates a 2D affine transformation matrix that can be used to apply various transformations to the image. The specific transformations and their parameters are as follows:

* Rotation: 30 degrees
* Skew: 30 degrees along the x-axis and 15 degrees along the y-axis
* Scale: -1 along the x-axis (which reflects the image) and 0.5 along the y-axis
* Translation: 0.5 along the x-axis and -1 along the y-axis

The extent of the image in each subplot is set to [-2, 4, -3, 2], and the image is clipped to these bounds. The yellow dashed line indicating the intended extent of the image is drawn using the plot function with these bounds. Finally, the figure is displayed using the show function from the matplotlib.pyplot module.

``` Matplotlib's Simple Instruction Bemple Create a Python script using matplotlib to generate a specific plot with the following detailed parameters: Initialize a figure with a custom size of 7.5 by 7.5 inches. Add a single axis to the figure with a custom aspect ratio and specified position. Define X as a linear space from 0.5 to 3.5 with 100 elements. Calculate Y1 as 3 plus the cosine of X, Y2 as 1 plus the cosine of 1+X/0.75 divided by 2, and Y3 as random values uniformly distributed between Y1 and Y2. Set major and minor locators for both x and y axes with major intervals of 1 and minor intervals of 4. Set minor formatter for the x-axis to display values with two decimal places. Limit the x and y axes to a range from 0 to 4.



For major ticks, set the width to 1.0, length to 10, and label size to 14. For minor ticks, set the width to 1.0, length to 5, label size to 10, and label color to '0.25'. Add a grid with these specific attributes:

* Linestyle set to "-" (dashed).
* Linewidth of 0.5.
* Color set to '.25' (a shade of gray).
* Z-order set to -10.

Plot three lines with distinct characteristics:

* The first line (Blue signal) should use color 'C0', linewidth of 2.5, and be placed at z-order 10.
* The second line (Orange signal) should use color 'C1' and linewidth of 2.5.
* The third line should consist of scatter markers at every third point, with no linewidth, markersize of 9, marker style 's' (square), marker face color 'none', marker edge color 'C4', and marker edge width of 2.5.

Set the title "Anatomy of a figure" and axis labels "x Axis label" and "y Axis label" with specific font sizes:

* Title font size should be 20.
* Axis label font sizes should be 14.

Add a legend with these specifications:

* Positioned at the "upper right".
* Font size set to 14.

Annotate the figure, including tick labels, axes labels, grid, etc., using circles, text, and code snippets at specified coordinates. The circles should have a radius of 0.15, a border color defined by the royal_blue variable with an alpha of 0.6, and a white, non-filled center. Text annotations should be in both royal blue and black colors, with bold and italic styles.

[title=Chargline]Crate U1-SX Example Create a bar chart showing the GDP compositions of agriculture, industry, and services for the top 5 countries by GDP. Highlight the country with the highest percentage of each sector. Add annotations to display exact values for each bar segment, focusing particularly on the leading country in each sector.

 

Human Study Details

### Compensation and Qualification

We use paid crowdsource surveyors to conduct our human study. We use the same platform as Gudibande et al. (2024). To ensure quality responses, all voluntary participants are paid \(\$14\) per hour, well above the U.S. federal minimum wage of \(\$7.25\)Henderson (2024). We ensure that surveys feel safe to express freely by clearly indicating: "feel free to be wholly honest". Moreover, we enforce five requirements to participate in our study: **(i)** Masters2, **(ii)**\(>80\%\) Approval Rate, **(iii)**\(>50\) Approved, **(iv)** Casual experience using LLMs, **(v)** Survey with a large screen (\(\geq 10\) inches, \(\geq 25.4\) cm). For privacy, we do not gather any other unnecessary data.

Footnote 2: Masters are “Workers who have demonstrated excellence across a wide range of tasks are awarded the Masters Qualification. Masters must continue to pass our statistical monitoring to retain the Masters Qualification.”

### Sanity Check

Following Yang et al. (2023) and Xu et al. (2024a), we include multiple sanity check questions within the survey for reliability. These sanity checks are solidified _a priori_, and are _never_ changed.

ChartUIE-8K.In ChartUIE-8K, we manually remove LLM initial query that clearly suggests that the surveyor did not understand the survey instructions or put in virtually zero time. For transparency, we report _all_ the sanity check fail cases on our open-source github repository.

ChartaF.For the ChartAF human study, two sanity checks filter very poor quality surveyor responses. First, we remove surveys that were completed in 50% of the lower-bound completion time. We time the duration it takes to complete a survey, and create a lower- and upper-bound to signal to potential surveys how long our study takes. For example, 10 to 16 minutes. In this case, surveys that were completed within 5 minutes would be removed as this would indicate that _very low effort_ was put into the survey. Second, a sanity check question was included, presenting an image option intentionally unrelated to the user instructions. Survey responses that prefer (or strongly prefer) this option is removed as it indicates that the surveyor does _not_ understand the task or is putting in very little effort.

### ChartaF Human Study

We provide the precise instructions provided to the surveyors below. We further provide exact screenshots in Fig. 7 and 8.

 

or post-Plot2Code+A-CoT+SC-feedback or post-Skip+A-CoT-feedback] [insert preference score options]

```
CHARTUE Human Study (Part 2) User Instruction USER REQUEST [insert initial instruction] QA WITHUSER [insert QA] Referring to the User Instruction, carefully select the better chart. [insert generated image by pre-TTS] [insert generated image by post-TTS] [insert preference score options]

### ChartUIE-8K Human Study

We provide the precise instructions provided to the surveyors below. We further provide screenshots in Fig. 9 and 10.
``` CHARTUIE-8K Human Study (Part I) ```
* Looking at the following sample image, imagine you want to ask Large Language Model (like ChatGPT) to draw a chart like this image: [insert example chart image]
* Please write how you would prompt (instruct) an LLM
* (You can assume that you have already provided the dataset to the LLM.) [insert text form field]
``` CHARTUIE-8K Human Study (Part II) ```
* After the user has given an initial set of instructions, the LLM may be inclined to ask some questions to make sure it clarifies on any lack of detail or subjective instructions.
* The following is an example of the LLM's response to the user's initial instructions.
_[insert sample initial instruction and questions]_
* If you were the user, would you prefer the LLM respond to you with, for example, the clarification questions above?
_[insert binary option (Yes or No)]_
* After the user has given an initial set of instructions, the LLM may be inclined to ask some questions to make sure it clarifies on any lack of detail or subjective instructions.
* The following is an example of the LLM's response to the user's initial instructions.
_[insert sample initial instruction, questions, and answers for the questions]_
* Do you think that the user response above is realistic?
_[insert binary option (Yes or No)] 

* [26] M. C. Bader, A. K. Goyal, and R. S. H. Smith, "A new approach to the design of a smart device for smart devices using smart devices in smart devices: A survey on smart devices with applications," _IEEE Transactions on Systems and Information Theory_**53**(1), no. 4, pp. 78-90, Jan. 2006.

[MISSING_PAGE_POST]

 



## User Instruction

**User REQUUST**

Create a radar chart to compare different factors affecting students' adaptability in online education, include variables like "Gender," Age," Education Level," Institution Type," IT Student," Location, Financial Condition," Internet Type, etc.

**QA WITH USEB****Q:** Should the radar chart include a legend to identify each variable?

**A:** Yes, please include a legend.

**Q:** Would you prefer shaded areas or lines to represent the different factors on the radar chart?

**A:** would prefer shaded areas.

* 12. Choose better Chart considering _User Instruction_ carefully.

Figure 8: ChartAF Human Study (Part 2) Example

\({}^{*}\) I. Looking at the following sample image, imagine you want to ask Large Language Model (like ChatGPT) to draw a chart like this image:

**R&D Investment by Country**

Figure 9: ChartUIE-8K Human Study (Part 1) Example



* 2.
* After the user has given an initial set of instructions, the LLM may be inclined to ask some questions to make sure it clarifies on any lack of detail or subjective instructions.
* The following is an example of the LLMs response to the user's initial instructions.

Figure 10: ChartUIE-8K Human Study (Part 2) Example



LLM Configurations

Table 5 shows the LLM configurations we used. These configurations are determined _a priori_ to all experiments, and never changed.

## Appendix F Code Error Rate

Fig. 11 shows the mean execution error rates of the four tested LLMs. This is an inherent limitation of the original chart-generating LLM and does not relate to our method.

## Appendix G Icon Attribution

All icons are from flaticon.com except Retain(cycle-100) is from icons8.com. The icons and their respective authors are in Table 6. The license to these icons belongs to their respective owners. We use them for non-profit academic use, with attribution as requested by the distributors.

Figure 11: Execution error rates of the tested LLMs





\begin{table}
\begin{tabular}{|l|p{142.3pt}|} \hline
**Category (Icon)** & **Author(s)** \\ \hline Retain (cycle-100) & Icons8 \\ \hline Discard (remove) & Pixel perfect \\ \hline Add (add) & Arkansasi \\ \hline Task (checklist) & juicy\_fish \\ \hline Purpose (target) & Freepik \\ \hline Audience (users-avatar) & SeyDesigner \\ \hline Basic Criteria (list) & Smashicons \\ \hline Addition (add) & Digby Garrett \\ \hline Query (question) & Md Tanvirul Haque \\ \hline Binarize (file-management) & iconsmind \\ \hline Social Science (network) & Freepik \\ \hline Biology (book) & Freepik \\ \hline Box Chart (chart) & Freepik \\ \hline Stacked Chart (column-chart) & Freepik \\ \hline LLM (big-data) & LAFS \\ \hline AI (ai) & Freepik \\ \hline Number 1 (circle-1) & iconographics \\ \hline Number 2 (circle-2) & iconographics \\ \hline Number 3 (number-3) & BS Editing \\ \hline LLM to Evaluate (dashboard) & xnimrodx \\ \hline Data Set Filter (database-structure) & Uniconlabs \\ \hline Chart Annotations (notes) & Freepik \\ \hline Data Set Crawl (bug) & Freepik \\ \hline Coding (code-review) & Freepik \\ \hline Evaluate (criteria) & cah nggunug \\ \hline Edit (modified) & Uniconlabs \\ \hline \end{tabular}
\end{table}
Table 6: Icons and Respective Authors