VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations
Authors: Yupeng Xie
Zhiyang Zhang
Yifan Wu
Sirong Lu
Jiayi Zhang
Zhaoyang Yu
Jinlin Wang
Sirui Hong
Bang Liu
Chenglin Wu
Yuyu Luo
Venue: ICLR 2026 Poster (2026)
Keywords: Multimodal Large Language Models
Visualization Assessment
Data Visualization
Primary area: datasets and benchmarks

Abstract: Visualization, a domain-specific yet widely used form of imagery, is an effective way to turn complex datasets into intuitive insights, and its value depends on whether data are faithfully represented, clearly communicated, and aesthetically designed. However, evaluating visualization quality is challenging: unlike natural images, it requires simultaneous judgment across data encoding accuracy, information expressiveness, and visual aesthetics. Although multimodal large language models (MLLMs) have shown promising performance in aesthetic assessment of natural images, no systematic benchmark exists for measuring their capabilities in evaluating visualizations. To address this, we propose VisJudge-Bench, the first comprehensive benchmark for evaluating MLLMs' performance in assessing visualization aesthetics and quality. It contains 3,090 expert-annotated samples from real-world scenarios, covering single visualizations, multiple visualizations, and dashboards across 32 chart types. Systematic testing on this benchmark reveals that even the most advanced MLLMs (such as GPT-5) still exhibit significant gaps compared to human experts in judgment, with a Mean Absolute Error (MAE) of 0.553 and a correlation with human ratings of only 0.428. To address this issue, we propose VisJudge, a model specifically designed for visualization aesthetics and quality assessment. Experimental results demonstrate that VisJudge significantly narrows the gap with human judgment, reducing the MAE to 0.421 (a 23.9% reduction) and increasing the consistency with human experts to 0.687 (a 60.5% improvement) compared to GPT-5. The benchmark is available at https://github.com/HKUSTDial/VisJudgeBench.

Published as a conference paper at ICLR 2026
VISJUDGE-BENCE: AESTHETICS AND QUALITY As- SESSMENT OF VISUALIZATIONS
Yupeng Xie', Zhiyang Zhang's Yifan WirnSi Lu', Jiayi Zhang',
Zhaoyang Yu Jiinin Wang". Sirui Hong", Bang Liu², Chenglin Wu Yuyu Luoi
Thhe Hong Kong University of Science and Technology (Guangzhou), 2 DDeeWisdom,, Université de Montréal & Mila
ABSTRACT
Visualization, a domain-specific yet widely used form of imagery, is an effec- tive way to turn complex datasets into intuitive insights, and its value depends on whether data are faithfully represented, clearly communicated, and aesthetically designed. However, evaluating visualization quality is challenging: unlike natural images, it requires simultaneous judgment across data encoding accuracy, infor- mation expressiveness, and visual aesthetics. Although multimodal large language models (MLLMs) have shown promising performance in aesthetic assessment natural images, systematic benchmark exists for measuring their capabilities in evaluating visualizations. To address this, we propose VISJUDGE- BAJJJDGGE-BNCCH,,, the first comprehensive benchmark for evaluating MILMs' performance in assessing visualization aesthetics and quality. It contains 3,090 expert-annotated samples from real-world scenarios, covering single visualizations, multiple visualizations, and dashboards across 32 chart types. Systematic testing on this benchmark re- veals that even the most advanced MLLMs (such as GPT-5) still exhibit signifi- cant gaps compared to human experts in judgment, with Mean Absolute Error (MAF) of 0.553 and correlation with human ratings of only 0.428. To address this issue, propose VISJUDGE, model specifically designed for visualization aesthetics and quality assessment. Experimental results demonstrate that VIS- JUDGE significantly narrows the gap with human judgment, reducing the MAE to 0.421 (a 23.9% reduction) and increasing the consistency with human experts to 0.687 (a 60.5% improvement) compared to GPT-5. The benchmark is available at https:// /ttbb cwm HKUUTD a a sVdiieuedeeeeee
INTRODUCTION
Visualization serves as an effective approach for transforming complex datasets into intuitive insights (Shen et al. 2023; Qin et al. 2020 Ye et al.
Fidelity Data Fidelity
Expressiveness Information Clarity
Aesthetics Visual Aesthetics
Representation
Communication
Professional Design
2024 Li et al. 2024a
Zhu et al.
2025b Li et al. 2025b a) The value
of high-quality visualization depends on whether its data is faithfully pre- sented (Fidelity), whether information is clearly communicated (Expressive- ness), and whether the design aes- thetically well-presented (Aesthetics), as shown in Figure [ These three dimensions are closely interconnected and indispensable, posing challenges for visualization quality assessment.
Misleading
Visual Design
25.5%
74.5%
Figure 1: The "Fidelity, Expressiveness, and "estheties" evaluation framework.
"Corresponding author; Yuyu Luo (F-mail: yuyuluo@ Saliuuuuuuuuuuttgg...mm

---

Published as a conference paper at ICLR 2026
Artistic Assessment
Visualization Aesthetics
Assessment (Mutt) Dimensional Evaluztion)
(General Aesthetics)
Muuti-dimensional Assessment Required:
Fidelity Expresshveness + Aesthelics
Ground
Fidelity:
UD
Perpe Pupfe-green gradimn creates fahse bounderics
Oritcca temporal information absant
Expert
Aesthetios
Viually appoaling funationaly
MLLM (w/o Domain Knowledge)
Visludge (with Domain Knowledge)
Assessment Requirement:
Aesthatics
Eflective represontation
Fidwity:
Gradinn distorts reading
Expresslon:
Time usta labe's mixaing
MLLM Assthetios:
veli striling andprefessiona
VisJudge
Assthatics:
app
boanthul ertstic compasitio
Falls Fidelity Expression
Accurately identifies
MLLM
Only saNs surface meschetics
Domain rxpartise assential
Figure 2: From natural images to visualization: the need for specialized visualization assessment. Green and red denote positive and negative assessments, respectively, highlighting the contrast be- tween MILMs' capabilities in general aesthetics versus visualization-specife evaluation.
Although Multimodal Large Language Models (MLLMs) have shown potential in aesthetic eval- uation of natural images (Murray al. 2012 Cao et al. 2025). applying them visualization evaluation faces unique challenges. Unlike natural images, visualization evaluation requires simul- taneous judgment of data encoding accuracy, information communication effectiveness, and visual design appropriateness, as shown in Figure However, existing MILM benchmarks are insufficient for such comprehensive evaluation, as detailed in Table T First, chart question answering bench- marks (c.g.. ChartInsights (Wu et al. (2024)) evaluate models' ability to understand chart informa- tion, rather than their overall design quality, Second, natural image aesthetic evaluation benchmarks (e.g., ArtiMuse (Cao et al.] (0255) ) focus on assessing aesthetics, but ignore the core purpose of vi- sualization effectively communicate data. Finally, existing visualization evaluation benchmarks (e.g.. VisEval (Chen et al., 2024b)) mainly evaluate natural language to visualization (NL2VIS) tasks (Luo et al. 2021c). with the focus on assessing whether generated visualizations accurately reflect natural language queries, rather than the aesthetics and quality of visualizations. This leads to a critical research gap: we lack a systematic framework to measure MLLMs' comprehensive capabilities in evaluating visualization aesthetics and quality.
To address this challenge, we construct VISJUDGE-BENCE the first comprehensive benchmark based the "Fidelity, Expressiveness, and Aesthetics" principles assess MILMs' capabilities in visualization aesthetics and quality evaluation. It contains 3,090 expert-scored samples from real-world scenarios, covering single visualizations, multiple visualizations, and dashboards across 32 chart types. Using this benchmark, we conduct extensive testing on 12 representative MILMs, including GPT-5, finding that even the most advanced models show significant differences from hu- man experts (MAE as high as 0.553, correlation only 0.428). This finding clearly demonstrates that general MLLMs cannot automatically acquire specialized evaluation capabilities in the visualization domain, making the development of specialized optimization models necessary.
Based on this, we propose VISJUDGE, a model specifically designed for visualization aesthetics and quality assessment, aimed at improving the consistency between general MLLMs and human expert evaluation standards. Experimental results prove the effectiveness of this approach: VISJUDGE significantly improves consistency with human experts, achieving 23.9% reduction in MAE (to 0.421) and a 60.5% improvement in correlation (to 0.687) compared to GPT-5, performing best among all tested models.
In summary, our main contributions are: (1) We construct VISJUDGE-BENCI,, a comprehensive benchmark based "Fidelity, Expressiveness, and Aesthetics" principles to evaluate MLLMs' pabilities visualization assessment. (2) We systematically evaluate representative MLLMs, re- vealing notable gaps with human expert standards. (3) We propose VISJUDGE, optimized model that significantly outperforms existing models and better aligns with human expert judgment.
2 RELATED WORK
Data Visualization Quality Assessment. Assessing the quality of data visualizations is a core problem in visualization generation and recommendation tasks.
2

---

Published as a conference paper at ICLR 2026
Table 1: Comparison of related benchmarks across key evaluation dimensions.
Types
Benchmark
Input
Types
Evaluation Dimensions
Fidelity
Expressiveness
Aesthetics
AVA (Mumy al. 2012
General Images
Aestheric Evaluation
ArtiMuse [Cao et al. 2125
Images
General Images
x
x
CharlQA Maary al
2022
Chart. Question
Single Vis
x
x
Chart Understanding
PlatQA (Memani & Mim Charlnsight
20A0
Chart, Question
Single Vis
x
/
8
Question
Single
VisEval (Chep di
202
Single
x
X
Visualization Evalustion
VIS Stspphip Pan cl
Chart, NL, Data
Single Vis
x
x
Single Vis, Multi Vis,
VisJudge-Bench (Ours)
Chart
Dashboard
In visualization recommendation tasks, the goal is to enumerate and recommend the best (top-k) visualizations for given dataset. To achieve this, existing methods fall into two main categories. The first is rule-based approaches, such as Voyager (Wongsuphasawat et al. 2017). Draco (Moritz et al. 2019). and CoInsight (Li et al. 2024b), which use heuristic scoring based on established design principles. However, their rules are often hard-coded and lack flexibility. The second category is learning-based methods, like VizML (Hu et al., 2019). DeepEye (Luo et al.] 2018 2022). and HAIChart (Xie et al. 2024). These methods train models on large annotated datasets to predict user preferences but are limited by simplistic evaluation dimensions and expensive annotated data.
In NL2VIS tasks, the goal is to generate corresponding visualizations based user user-provided natural language queries (Luo et al. 2021c). Representative works include ncNet (Luo et al. (222cc), Deep- VIS (Shuai et al. 2025), CharGGPT TTan et al. 2023), and LLM4Vis (Wang al. (023). To assess how accurately these methods translate natural language into visualizations, several benchmarks have been proposed, including nvBench (Luo et al. (021bba), a), nyBench 2.0 (Luo al. (023), ) and
MatPlotAgent (Yang et al. 2024). However, these methods and their related evaluations primarily focus on the model's ability "write" code rather than "judge" the quality of visualizations.
MLLM as a Judge. Recently, MILMS have shown significant potential in emulating human expert judgment, a paradigm known "WLLM-as-a-Judge" (Zheng et al., 2023 Chen et al. [2024a] Bian
et al. (025). As summarized in Table [ these works fall into three categories: (1) general visual
aesthetics assessment (e.g., AVA (Murray al., 2012), ArtiMuse (Cao al. 2025)), which evalu- ates the artistic quality of photographs but overlooks information communication efficiency in data visualization: (2) chart understanding tasks (e.g., ChartQA (Masry et al., (022), ChartInsights (Wu)
et al. 2024)). which focus on understanding and interpreting chart content but neglect overall de- sign quality; and (3) visualization evaluation (e.g., VisEval (Chen et al.) 20246). VIS-Shepherd (Pan) et al. 2025)), which evaluates whether generated visualizations accurately reflect natural language queries, but lacks a comprehensive assessment of intrinsic design quality. To address this gap, we in- troduce BEJJUDGE-BBNNCC the first comprehensive benchmark designed to systematically evaluate the capabilities of MILMS as "visualization quality judges".
VISJUDGE-BENCH DESIGN AND CONSTRUCTION
To systematically evaluate the capability boundaries of MLLMs in visualization evaluation, we de- sign VISJUDGE-BENCE As shown in Figure its construction follows three-stage methodology: (1) data collection and processing; (2) adaptive question generation; and (3) expert annotation and quality control. We detail the specific implementation of each stage below.
3.1 BENCHMARK CONSTRUCTION PIPELINE
3.1.1 DATA COLLECTION AND PREPROCESSING
Corpus Construction. To evaluate the performance of MLLMs across different visualization types, we construct a corpus covering three main categories: single visualizations, multiple visualizations, and dashboards. To ensure the authenticity and diversity of corpus, collect visualization samples from search engines using web crawling methods with diverse query keywords (see Ap- pendix A. for detailed crawling architecture and keyword generation strategy).
3

---

Published as a conference paper at ICLR 2026
Data Collection and Processing
Adaptive Question Generation
Original
Generic Evaluation Criterla
Aielity-xprresshvone
Aesthetios
Data Fieelty
Readabilty
Insight Discovery
Web Crawling
Image Filtering
Classification
Chart Metadata Extraction
Question Rewriting
Customized Chart Information Evaluation Questions Scoring
Chart Metadata:
Data Fidelity:
Single 122
Multi (typae)
red codtnng
Hor rontal ber chart, Gmenn and
Doss this horizental bar chart sccurataty represent Injury counts
Scoring Critarta
Major d tions.. Noticcable issues..
Description:
proportional customized questions....
baseline
Total: 3090 Real World Samples
showingg grouped
data...
Three-Stage Stage Annotation Workflow
Stage 1 Initial Annotation
Evaluation Questions Scoring
3 Annotators Carefully Screened
Crowdsourced Annotators
Relevant- Expertise Filter
Ann.1
Ann.2
Ann.3
Stage 2
Intia IFBE Scar
Consalency
Candidate Strategies
Quality Control
Comozrnncy
Remove total outlier
Q Disagreement Analysis Candidate Resolution Strategies
Filter
Candidate
Remove aub-dim outtier
Check and Sort
Resclution
Fitter malicious rating
Strategles
Stage 3
Candidate Strategies
Expert Validation
Remove outler
SONETT
Score
Professional Annotators
Quality Assurance
Filter natitiuss rating
Apply Strategy Adjust
Expert
Export Review
Figure VISJUDGE: BENGH construction framework.
Table 2: VISJUDGE-BENCI statistical information (Dash. Dashboard).
Vis Type
Count
4-Subtype
Subtype Details (Count)
Bar Chart
176
Pic Chart
129
Line Chart
100
Area Chart
75
Heatmap
55
Scatter Plot
49
Single Vis
1,041
22
Histogram
48
Donut Chart
47
Funnel Chart
45
Treemap
62
Sankey Diagram
61
Bubble Chart
29
10 more subcategories
Comparison Views
670
Small Multiples
195
Coordinated Views
97
Multi Vis
1,024
5
Other Multi View
Overview Detail
3
Analytical Dash.
743
Operational Dash.
122
Interactive Dash.
91
Dashboard 1,025
Strategic Dash.
62
Other Dash.
7
Data Preprocessing Pipeline. We design a three-stage data filtering process to curate the bench- mark from over 300,000 initial images. (1) Initial Filtering: We employ automated scripts and perceptual hash algorithms to eliminate non non-visualization content and duplicates, yielding 80,210 candidate images (detailed algorithms in Appendix [..]). [.T). (2) Automated Classification: We lever- age GPT-4o for visualization type classification and quality filtering, resulting in 13,220 valid visu- alization samples after human verification (classification prompts and criteria in Appendix[A.I] (3) Stratified Sampling: We apply stratified random sampling to select the final 3,090 samples, ensuring balanced distribution across categories. As shown in Table 2 the final corpus contains 1,041 single visualizations, 1,024 multiple visualizations, and 1,025 dashboards, covering 32 distinct subtypes. Detailed dataset statistics and distribution analyses are provided in Appendix A.2
3.1.2 THE "FIDELITY, EXPRESSIVENESS, AND AESTHETICS" EVALUATION FRAMEWORK
To enable fine-grained visualization assessment, this stage first establishes multi- coltii-immesionall eval- uation framework, then implements an adaptive question generation process based on this frame- work (as illustrated in the upper-right panel of Figure
4

---

Published as a conference paper at ICLR 2026
The "Fidelity, Expressiveness, and Aesthetics" Framework Design. To systematically evalu- ate visualization quality, we construct a multi-dimennsonal evaluation framework. This framework draws inspiration from classical translation theory principles of "Fidelity, Expressiveness, and Aes- thetics" (illustrated with nositive-negative examples in Figure combined with established theories in graphical perception (Cleveland & McGill 1984), information visualization design (Munzner) 2025), and aesthetic evaluation (Cao et al., 2025). We operationalize this core concept into six
measurable evaluation dimensions (as shown in Figure D
Fidelity focuses on Data Fidelity. This dimension draws from Tufte's design principles (Tufte) 1983) for avoiding "graphical lies" and recent research on visualization misleadingness is- sues Nguyen et al., 2013) Szafir 2018 Lan & Liu 2024 McNutt et al. (020). Since source data for web-collected visualizations is typically unavailable, this dimension evaluates data pre- sentation accuracy at the visual level. It assesses whether visual encodings accurately reflect the displayed values, examining visually detectable distortions such as improper axis settings, scale distortions, truncated baselines, disproportional encodings, or other misleading design patterns.
Expressiveness focuses on the effectiveness of information communication. This dimension evaluates how effectively visualizations convey information users. It includes two progressive sub-dimensions: First, (1) Semantic Readability evaluates the clarity of basic information en- coding, assessing whether users can unambiguously decode visual elements in charts (Pan al.)
2025 Tang et al. 2026 Waag et al., 2026). Building on chart readability, (2) Insight Discovery further evaluates the analytical value in revealing deep data patterns, trends, or outliers, helping users transition from "reading information" to "gaining insights" (Chen et 2024bb) (025).
Aesthetics focuses on Aesthetic Quality of visual design, integrating visualization perception theory (Ware 2019) with design practice. This dimension consists of three sub-dimensions that collectively influence the overall visual experience:
(1) Design Style evaluates the inno- vation and uniqueness of design, measuring the degree of novel visual elements and distinctive style (Dibial 2023 Brath & Banissi 2016): (2) Visual Composition focuses on the rationality of spatial layout, evaluating the balance and order of element positioning, size proportions, and spacing arrangements (Wu et al. 2023); and (3) Color Harmony evaluates the coordination and functionality of color combinations, ensuring color palette choices balance aesthetics with effective information communication (Harrower & Brewer 2003 Gramazio et al. 2017).
In addition, this evaluation framework offers flexibility, with evaluation questions and scoring crite- ria adaptively tailored to each visualization's 's type (such single visualizations (Chen et al. 2024b), multiple visualizations (Chen et al., 2020), and dashboards (Bach al. (023)) and its specific visual characteristics. Complete evaluation rules and customization details are provided in Appendix C
Adaptive Question Generation Mechanism. Based on the evaluation framework, we have devised an adaptive question generation process (detailed workflow shown in Figure 3) This process be- gins by leveraging GPT-40 to extract metadata from the chart, such as its type and visual elements. Subsequently, it rewrites questions by populating predefined templates based on this metadata, gen- crating highly customized questions and their corresponding five-point scoring criteria for the six evaluation sub-dimensions. For instance, under the data fidelity dimension, the generated question specifically targets the chart's data presentation accuracy, asking whether the "horizontal bar chart accurately represents injury counts with a proper zero baseline and proportional bar lengths." The scoring criteria focus on visually detectable issues: a score of I identifies "major distortions, such as truncated axes or misleading scales that exaggerate differences." " whereas a score of 5 confirms a "highly faithful representation where bar lengths are strictly proportional to the displayed values." This approach ensures that the evaluation questions are closely aligned with the specific visualiza- tion content. For more detailed examples, please refer to Appendix
3.1.3 EXPERT ANNOTATION AND QUALITY CONTROL
To build reliable human ground truth, VISJUDGE-BERN BENCH adopts a rigorous three-stage annotation and quality control workflow (bottom panel of Figure informed by benchmark construction ap- proaches (Rein et al.] 2024| Liu et al., 2025) Zhu et al. 2024) Shen et al.) (026). This systematic process ensures fidh-iitlityy and consistent scoring through careful review and expert judgment.
Stage 1: Initial Annotation. We recruited 603 highly qualified crowdsoureing workers through the CleudResearch platform (Hartman et al. 2023) To ensure annotation quality, we set strict
5

---

Published as a conference paper at ICLR 2026
screening criteria (see Appendix A.3.1 for details) and designed dedicated annotation interface (Appendix[A.3.3), Crucially, we embedded validation checks to identify and filter out inattentive re- sponses (examples in Appendix/A.344 Each of the 3,090 samples was scored by three independent annotators across six evaluation dimensions (task design details in Appendix A.3.2). generating an initial scoring matrix.
Quality Control. To address scoring disagreements among annotators, we designed a sys- tematic conflict identification and resolution mechanism based established crowdsourcing quality control and statistical evaluation theory (Gadiraju et al. 2015 Rousseeuw & Leroy 20033 [rennna 1992). The system first identifies high-disagreement samples by analyzing score variance, then al- gorithmically generates candidate resolution strategies including outlier removal. malicious scoring detection, and sub-dimensional bias correction. These gesorithm-geeerane suggestions are pro- cessed and ranked before being submitted to the expert team for final review (complete algorithmic details and parameters in Appendix A..3...
Stage 2:
Stage 3: Expert Validation. Three experts with visualization analysis experience independently reviewed all samples using a dedicated interface (Appendix (A.35), selecting, modifying, or rejecting algorithm--eneraaed candidate solutions. For complex cases, the team reached consensus through discussion. Through this rigorous process, we built a high-quality human scoring benchmark for all 3,090 samples as the gold standard for model evaluation. Detailed annotation quality analysis is provided in Appendix A.4
4 VISJUDGE: A SPECIALIZED MODEL FOR VISUALIZATION EVALUATION
The primary goal of VISJUDGE-BENCH is systematically evaluate MLLMs' capabilities in visu- alization aesthetics and quality assessment. Building on this benchmark, we explore a key question: can domain-specifie fine-tuning improve models' visualization evaluation capabilities? To answer this question, we develop VISJUDGE by fine-tuning multiple open-source multimodal architectures different parameter scales. In Section 5, conduct unified evaluation of these fine-tuned models alongside closed-source and sper-source baselines on the same test set.
Training Setup. We use VISJUDGE-BENCHH human- annotated data with 70%/10%%/20% train/- validation/test split (2.163/779/648 samples) via stratified sampling to maintain consistent visual- ization type distribution across all splits. Training data is kept separate from baseline evaluation to prevent contamination.
Model Training. We fine-tuned four representative open- source multimodal models as base models: Qwen2.5-Ve VL (13B/BB-Instrnn (Bai et al. (025). Intern VL3-8B (Zhu et al. (2025a), and Llava-v1.66 mistral-7B (Liu et al. 2024). These models span different architectures and parameter scales (3B to 8B), enabling comprehensive evaluation of our training approach's 's arestterche generaliza- tion (Lin et al., (025). All models are trained to generate quality scores (1.0-5.0) and rationales aligned with human expert judgments. We employ reinforeement learning with the GRPO algo- rithm (Shao et al. 2024| Liang et al., [2026] Wu et al.] 2026), using a composite reward function combining accuracy reward (minimizing prediction error) and format reward (ensuring structured outputs) (Shi et al. 2025 Wu et al. 20225. ). Formal reward definitions are detailed in Appendix D.2 For parameter-efficien fine-tuning, we adopted Low-Rank Adaptation (LoRA) (Hu al. 2022) All models were trained using consistent hyperparameter configurations across architectures. Training used 5 epochs with learning rate of le-5, Detailed configurations are in Appendix D.3
EXPERIMENTS
5.1 EXPERIMENTAL SETTINGS
To evaluate existing MILMS in visualization quality assessment and validate our VISJUDGE, we conduct comprchensive experiments VISJUDGE-BENN
Evaluation Setup. We evaluate 12 representative MLLMs: GPT-5, GPT-4o, Claude-4- Sonnet, 3.dde-3.5-Soneett Gemini-2.0-Flaan Gemini-2.5-Pra Qwen2.5-VL (3B/7B/32B/22B- Instruct) (Bai et al. 2025). VLter8VL.3-8B (Zhu et al. 2025a). 2024), and their corresponding fine-tuned variants, which collectively refer to VISJUDGE, on balanced test set 648 samples (see Appendix A....7 for distribution details). Each model provides 1-t0-5 scores with justifications based evaluation framework. Following human
(Liu et al.

---

Published as a conference paper at ICLR 2026
Table 3: Overall performance of MILMs and the VISJUDGE VISJUDGE-BENCE across different
evaluation metrics and dimensions.
Metric
Model
Overall
Fidelity
Expresiiveness
Acsthetics
Readabillit
Insight
Design Style
Composition
Color
Clunde 3.5 Somnet
0.824
0.978
0.904
L.154
0.783
0.9:0
0.862
Clande-4-Son SSnnner
0.622
0.841
0.757
0.832
0.679
0.734
0.785
Closed source
Gemini P.0.Hluch
0,829
0.820
0.638
0.729
0.79%
Gemini Pro
0.900
0.918
GFT-4o GPT5
0.610
0.983
0.806
0.744
0.009
0.695
0.657
0.553
0
0.781
0.778
0.649
0.699
0.682
Qwen2, Qwen2.5-VIL.Bee --V1-338-n
0.211
1,055
1,258
1,08
0.723
0.727
0.808
MAE (1)
Qwen2255 VL----L--------
02477
1.171
1,296
0.858
0.356
0.122
0,772
Open-
Qwan2.5-VV Peom2.--V1-32---mme ---i--22--nntteet
source
0811
0.735
0.717
0.793
1.254
1,195
0,870
0.679
0.759
0.753
1 1ama-1 1.oms--l.--miitaal-777
0.734
0.929
1,124
0,939
0.801
0.14
0,818
I1-emV113-88
0.541
0.797
0.769
0.691
0.608
0.645
0.616
Vislulge
Llag-s Banrg------mmmstt 166misttal--B
0.496
0,767
0.695
0.775
0.643
0.574
0,598
Qwen2.5.VI 1...1---------srru
0.571
Qwen2.5 V-nn.- Instrcctt
0.421
0.661
0.648
0.677
0.580
0.545
0,604
Clmade-3.--Sonett
1,577
1,999
0.994
1.467
Clande 4 Sonnet
0.603
1,182
0.973
1.147
0.774
0.934
1.037
Closed-
Gemini-2.0-
0.720
1,182
1,326
1,120
01673
0.925
Gemini-2.5.PPP 2..5ii-
1,371
1.112
1.325
1,400
GFT-4o
GPT-5
0.486
1,219
1,562
0.989
0.921
0.970
0.720
0.627
0823
0.862
02200
0IR0
Owen2.5-VL-3B--Une
10388
2,400
L.852
0.868
0.903
1046
MSE
Qwen2.5.VV 5.111-17-----s
1045
2.051
2.413
1.177
0,938
1.093
n,96
Open-
Qwen2.5.V (1..2---VL.22--nsnnn
0.756
1,361
1,528
1,067
0.728
0.988
0380
Owan2.5-VV VL-7-2--stttt
1,426
1.092
V1..m/11.388
2,340
2,088
1.219
0.747
0.981
0,988
Llavs-1..--ttsttn f--isstal-78
0.17
1.513
1,960
1,408
1.007
1.071
1.076
IntemVLL3-88
0.477
1,068
0.951
0.822
0.638
0.736
0.655
Vishulge
Laag-s 6-misstmll77
0.383
0.919
0,752
0.952
0.642
0.541
0.619
(Ouns)
Qwen2.5-VL V-LL-B Instrct
0.377
0.855
0,850
0,822
0.620
0.539
0.603
Qwen2.5-VL-70 M-L02.5-LL7---Imttn
0.756
0.545
0.498
Claade S.5memnett
Sonnet
0.395
0.325
0.492
0.365
0.455
0.137
0.259
Cland-4--onnet
0.452
0.168
Closed-
Gemini-2 D-Hash
0.395
0.572
0.489
0.417
0.459
0.157
0.209
source
Gemini 2.5 Pro
0.265
0.175
0,379
0.353
0.445
0.193
0.208
GFT-4o GFL-
0.442
0.277
0,382
0.276
Qwen2.5.V1 -.v....-i--Btt
0.275
0.338
0.130
Corn. (1)
0.281
0.357
0.149
Open-
Qwan2 S-L.2-5-V--33----1 Ins-trrettt
0.435
0.348
0.468
0:408
0.449
0,200
0.268
sonrce
Qwan2.5VV1 2ol.2.5-VI-7BB-Bmme Insttrettt
0.440
0.331
D.479
0446
0.435
0.165
0.251
0.342
0.216
Laaa-1 5..mm-srral---
0.180
0.201
0.137
0.160
0.188
0.064
0.076
Intern V1.3-88
0.660
0.545
VisJulge (Ouns)
Llava v 6--mistral-7 7B
0.605
0.226
0.536
0.443
0.432
0.406
0,403
Qwen2,5-Vi 1-2.225--------mtteee
0.648
0.533
0.581
0,579
0.504
0.490
0.402
Qwen2.5 VL-7B 7B Instruct
0.687
0,574
0.628
0.576
0.568
0,513
0.385
annotation procedures, we each model three times and average the results. All inference uses vLLM on four NVIDIA A6000 (48GB) GPUs with bfloat precision and temperature of 0.8.
Evaluation Metrics. We assess model performance through correlation analysis using the Pear- son coefficient and error metrics (MAE and MSE) compared to human scores. We also analyze score distributions to identify systematic biases. Metries are computed for each sub-dimension and aggregated across the three main evaluation dimensions.
5.2
EXPERIMENTAL RESULTS AND ANALYSIS
5.2. 1
CAN MLLMS ASSESS VISUALIZATION AESTHETICS AND QUALITY LIKE HUMANS?
Table presents comprehensive performance comparison of 12 representative models including the latest GPT-5 across our evaluation framework, revealing significant capability differences and systematic limitations in current MILMs for visualization assessment.
Hierarchical Capability Structure. Current MLLMs exhibit a hierarchical performance pattern: "Fidelity" is relatively strong, "Expressiveness" is moderate, and "Aesthetics" is the most challeng- ing, with average MAE around 0.755 and correlations of only 0.177-0.40S across the three aesthetic sub-dimensions, reflecting the difficulty of abstract design principles and cultural context for cur-
7

---

Published as a conference paper at ICLR 2026
Human (j=33133
Gemini-2.0 Flash (j--
Visjudge (u=3.11)
Gemini-2.5-PPP (u=3.02)
C Claude-d -Sonn-n
Owenn 15--5---77
Denslyy
Claude-3.5-Somce (u=3.877
Dwon2.5-VL-78 (.89)
GPT-40 (1-3.533)
(w=3.36)
GPT5-6
Claude-33 (.S-Senneet
2.0
Claude 4.Sonnet 3.56)
GPP:-
0.5
Gemini-2.5-PP 2.5PPr
1.0
5.0
Score
Figure 4: Distribution and bias analysis of MLLM scores. Score distribution density curves showing the rating patterns of different models compared human experts the 1-5 scale.
rent models. Models also exhibit distinct evaluation characteristies: GPT-5 performs consistently across dimensions; GPT-4o is relatively strong in Color Harmony (MAE 0.657); Claude-4-Sonnet excels in Semantic Readability (MAE 0.757); and Gemini-2.0 2.0-Faah leads in Data Fidelity (MAE 0.829). Among open-source models,
1-0.-255----22-Intttte achieves the best overall perfor- mance (MAE 007 Corr. 0.440), approaching commercial model capabilities.
Domain-Specific Fine-tuning Effectiveness. VISJUDGE (based on Qwen2.5-VL-77B-Instne achieves the best overall performance with MAE 0,421 and correlation 0.687, representing 23.9% MAE reduction over GPT-5 and a 42.5% correlation improvement over GPT-4o. Domain-specifer fine-tuning consistently yields 30-40% error reductions across different backbone architectures: Qwen2.5-VL-38 -L-3B-Istuutt (0.793 to 0,541), and Llava-vl 1.6-mistral-777 31.5% (0.724 to 0.496), demonstrating robust gen- eralization across diverse model families. In subsequent analyses, VISJUDGE refers to this best- performing variant unless otherwise specified. For analysis of how training data scale affects these results, see Appendix E.1
Instruct achieves 40.2% reduction (from 0.821 0.491). Intern VL3-8B 31.8%
5.2.2 Do MILMS EXHIBIT HUMAN-LIKE SCORING BEHAVIORS?
To analyze systematic biases in model evaluation behavior, examine score distribution patterns across representative baseline models. Figure reveals significant bias issues in current MILMs compared to human experts (μ+ = 3.13).
Systematic Biases in Current Models. Most models exhibit score inflation with rightward-shiffeed distributions. Qwen2.5-VL-7BB-mme
and Claude-3.-Sonmet show the most severe inflation (μ ===33 3.89 and 3.87), while Gemini-2.0-Flash GPT-4o, Claude-4-Sonnet, and GPT-5 demon- strate moderate inflation (p 3.64, μ 3.53, μ 3.56, and μ 3.36 respectively). Notably, GPT-5 shows relatively better control compared to other inflated models. Conversely, Gemini-2.5- Pro exhibits overly conservative behavior (μ+ 3.02). Additionally, models like Qwen2.5-VL- IB- Haude-3.5-Sonmee and Gemini-2.0-Flast exhibit sharp peaks around 4.0, indicating excessive score concentration that limits discriminative capability.
Effective Bias Correction through Fine-tuning. Our VISJUDGE achieves near-perfect alignment with human scoring patterns (μ 3.11) and maintains a broader, more balanced distribution. This demonstrates that domain-specific fine-tuning effectively corrects both inflation and concentration issues, achieving human-likk like evaluation behaviors.
5.2.3 How DOES VISUALIZATION COMPLEXITY AFFECT MODEL PERFORMANCE?
To understand model robustness across varying complexity, we analyze representative baseline mod- els three visualization types: single visualizations, multiple visualizations, and dashboards. Fig- ure 5 shows the main trends.
Performance Degradation with Complexity. All models show consistent performance degrada- tion: single visualizations > multiple visualizations > dashboards. VISJUDGE achieves the best performance across all types with correlations of 0.577 (single visualizations). 0.565 (multiple vi- sualizations), and 0.375 (dashboards), significantly outperforming baselines. This demonstrates the effectiveness of domain-specific fine-tuning for complex multi-element interactions.
Stability in Complex Scenarios. Baseline models show significant instability in complex sce- narios. For dashboards, most baselines experience substantial correlation drops, with Claude-3.5-
8

---

Published as a conference paper at ICLR 2026
Single le Vis
Multi Vis
Dashboards
Insight Discovery
Semantic Readability
Insight Discovery
Somantic
Insight Discovery
Semantic Readability
Readabliity
ru r
Design Sty
Data Fidelity
Design Slyle
Data Fidelity
Design Style
Data Fide ity
Visual Composition
Color Hormany
Visual Compasition
Color Harmony
Visual Composition
Color Harmony
Visludge
Gemini-2.5-PP
Gerini 2.0-Fashh
Qwen2.5 VL 78 Instruct
Claude 4 Sonnet
GPT
GPT-4o
Figure 5: Model Human rating correlation across visualization types.
Human Rating: 1.67
Gemini-2 2.0 Flash
Score: 3.33 X The trocmap represents market cap and 1-day change.
Human Rating: 176
Score: 3.50 X The U.S. map legind for 'pills, so readable.
Gemini-2.--flahh
cart is generslly accurste..
The lifted ur dedertaadingg a hinorance thouge..
Qwen2 5- VL 70 Instract
Score: 3.67 X The clant represents data with color gidleett and
Gwen226-VL-7B B-nnn..----B--nstuuee
Score: 3.67 X
The bet chatt... and chorap with map sccunstely reflact the data, each view using a cletr encoding method...
(2205. The logend is clear...
Claude-4: Sonnet
Score: 5.06 X
Claude-4
The diuallzation effectively reveals relationchips The
leyou demonstrates excellent spatial organization...
standard projections...
GPT-5
Score: 3.17 X
DPT-6
Score: 3.17
color trends, yielding basic insights..
octection
reveals basie infer meanings..
VisJudgee
Vis Judge
1,94 ti
The treemaps have a chaotic layout with dipprocrtionately stzed elements, making interpret the visuals. Over harmony
and accurately represent data. defined, leadingg to confusion... these clearty
hampered by denstty..
Figure 6: Model evaluation examples on low-quality visualizations.
2. Human Ratingg 4.17
Gemini22 G-Pro Scorw: 2.54 The waterbbel cherl under Stateement funddamentill hoowed, iox contradict cts the Nett Pholit KM card
Human Rating: 3.56
Gamini 2.5 Pro
The chart uses two difinent Vaasss
distortion making
Imponsible to accumbely compane the mugnritude bettowen two ranges.
Vullue of 535.
GPT-5
Score: 3.5e E
300
baselinos proportional scaling. and gouge slgns with the KP
The legend clearly mane contioents colons aves basio understandable..
11.1% 18854407/ubbess and
Claude-4 Sarnet
Score: 3.83
Cloude-4- Sonnet
Scoee: 388
frinncial groper basclines cerrectly scaled gouge charts showing ng goresnaage...
legend elearty displayed the top, making easy Identry which color continent.
VisJudge
The cuuaiiattion appaor uscuratety represent the underlying dats. The numel values match the test sbots, and the bar charts show propertions offfenecces..
The uses uniqus colors for continents and sep entass the data into may be sightt verietions the soect values..
klows. Thotte
Figure 7: Case study highlighting the conservative bias of Gemini-2.5-Prr
Sonnet and GPT-5 even showing negative correlations in Data Fidelity (-0.031 and -0.013), while VISJUDGE maintains consistency (0.224-0.482). Functional dimensions (Data Fidelity, Semantic Readability) remain stable across types, but aesthetic dimensions struggle with complex layouts, par- ticularly Visual Composition in dashboards (most models <0.2). These findings highlight the critical importance of specialized training for robust visualization evaluation across diverse complexity lev- els. For detailed error analysis of multi villtiiviuualizationn and dashboard cases, including systematic bias patterns and failure modes across different models, see Appendix B.6
5.2.4 How Do MODEL EVALUATION BEHAVIORS DIFFER IN PRACTICE?
To qualitatively analyze model evaluation behaviors, we conduct case studies on representative base- line models to reveal two common biases: "score inflation" and "overly conservative" assessments.
Figure 6 illustrates score inflation low-quality visualizations. For chaotic treemap (human rating: I.67), baseline models give inflated scores. For instance, Qwen2.5-VLL-7--nnss
(3.67) praises its "clear legend" while ignoring the confusing layout, and Claude-4-Sonne (3.08) incor- rectly highlights "excellent spatial organization". In contrast, VISJUDGE'S 's score of 2.00 aligns with human judgment, correctly identifying the "chaotic layout" that impairs interpretation.
Conversely, Figure highlights the overly conservative bias of Gemini-2.5-Pra For a high- dashboard rated 4.17 by humans, Gemini-2.5-Por gives a disproportionatell low score of 2.94, fo-

---

Published as a conference paper at ICLR 2026
Table 4: MatPlotAgent quality improvement with different feedback models.
Generation Model
Direct Decoding (Baseline)
Qwen2.5-VL-7B as Feedback
Qwen2.5-VL-72B as Feedback
VISJUDGE as Feedback
GPT-5
67.15
66.49
69.80
72.07
GPT-4o
62.71
58.49
63.53
71.62
68, 18.10
65.04
66.12
72.20
Claude-4-Soe --uude--Soonet
64,50
67.25
65.20
72.23
Claude-3.5 Sruue-3.5-Somee 3.5-Somnet
63.89
59.23
63. 14
69.04
Qwen2.5- ------t--st
61.18
58.70
60.30
67.32
Breen.5--LL-B--nsttee
49.76
44.94 94
55.29
cusing on single data inconsistency while overlooking the chart's overall effectiveness. Similarly, for another chart (human rating: 3.56), it scores only 2.33 due to the use of dual Y-axes. While other models like GPT-5 and Claude-4-Sonnet provide scores closer human ratings, VISJUDGE also demonstrates more balanced evaluations (3.83 and 3.00, respectively). Additional case studies across different quality levels and comprehensive model error analysis are provided in Appendix
5.2.5 CAN VISJUDGE GENERALIZE TO REAL-WORLD APPLICATIONS?
To validate practical generalization, we integrated VISJUDGE into two real-world visualization sys- tems with significant distribution shifts from our training data (detailed setup in Appendix E.2
Visualization Generation. VISJUDGE provides feedback to MatPlotAgent (Yang et al. 2024 for iterative quality improvement. Table 4 shows consistent improvements across seven generation models (+6.07 points average). Notably, base models without domain fine-tuning often degrade performance (7B: -2.39, 72B: -0.61 average). This degradation stems from their systematic eval- uation biases (Section (5.2.2): score inflation and poor discriminative capability lead to misleading feedback that misdirects generation models toward suboptimal outputs. In contrast, VISJUDGE'S accurate quality assessment consistently improves all generators. Even state-of-the-art models like GPT-5 and Claude-4-Sonee achieve +4.92 and +7.73 improvements, demonstrating that domain- specific fine-tuning is essential for providing effective feedback.
Visualization Recommendation. VISJUDGE as reward model in HAIChart (iie al. 2024 improves recommendation accuracy on the VizML dataset: +4.40% (Data Queries Hit 1), +2.50% (Overall @it@1 1), -5.30% (Overall H). While 7B and 72B base models show minimal mod- erate improvements, VISJUDGE achieves consistently larger gains across all metrics, outperforming both base models and the baseline (detailed results in Appendix E.2). This validates VISJUDGE'S broad applicability across generation and recommendation tasks.
CONCLUSION
This paper constructs VISJUDGE-BENC and fine-tunes VISJUDGE to validate the effectiveness of domain-speciff training. Our research finds that existing MLLMS (including GPT-5) show sig- nificant gaps with human experts in visualization evaluation, exhibiting issues like scoring bias. VISJUDGE effectively mitigates these problems, achieving 23.9% MAE reduction and 60.5% cor- relation improvement over GPT-5. for the community, while VISJUDGE'S success demonstrates that domain-speciio training is a viable approach for improving MLLMs' evaluation capabilities, supporting future work on finer evaluation and qughlit-uualtyy visualization generation.
BAJJDGGE-BENCCE provides standardized evaluation platform
LIMITATIONS AND FUTURE WORK
VISJUDGE- BENCH has several limitations: (1) it focuses static visualizations, with limited cov- erage dynamic and interactive content; (2) web-collected samples typically lack raw data, so eval- uation relies on visual presentation alone, limiting data fidelity verification; and (3) point-estimate scoring may not fully capture the diversity of human preferences. Future work includes expand- ing dynamic visualization samples with corresponding evaluation criteria, exploring ways to obtain samples with raw data for more thorough fidelity assessment, and investigating distribution based evaluation methods better reflect human preference diversity.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
The VisJudge-Bor -engh framework presented in this work aims improve multimodal large language models' capabilities in visualization quality assessment and promote the development of automated visualization evaluation technology. We believe this work will not produce direct negative social impacts, but recognize that the framework should be used with caution and ethical oversight when applied to sensitive domains or potentially harmful models. Although jectively assess visualization quality, the base models it relies on (such as Qwen2.5-VL-7B) or the datasets used to construct the benchmark may inadvertently reflect biases. Future work could in- vestigate the fairness implications of these evaluation features across different populations, cultural backgrounds, and visualization styles. We particularly focus on the following ethical considerations: (I) strict compliance with copyright and usage terms during data collection; (2) ensuring fair com- pensation and voluntary participation for expert annotators; (3) avoiding content that may reinforce stereotypes or biases in benchmark design; (4) open-source release aimed promoting community development rather than commercial monopoly. All research was conducted strict compliance with ICLR ethics guidelines.
SaaJudge-Bennce aims to ob-
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide comprehensive documentation and resources. The com- plete VisJudge-Bendi construction process is detailed in Appendix [A] covering data collection, filtering, and expert annotation protocols. The six-dimensional evaluation framework and Vis- Judge implementation are described in Appendices C and D respectively. We release the complete dataset with 3,090 expert-annotated samples across three categories (single__y _vis,,,, mult i_viiiiii, dashboard) rd) at https: / gighub..bb.... com/ HKUS TDial/Vi Iiiiaaaa ple includes visualization images, dimennnnsa scores, and evaluation prompts. All experimental configurations, evaluation metrics (MAE, MSE, Pearson correlation). and human consistency anal- ysis methods are fully documented enable result reproduction.
where each sam-
ACKNOWLEDGMENTS
This paper was supported by the NSF of China (62402409); Youth S&T Talent Support Programme of Guangdong Provincial Association for Science and Technology (SKXRC2025461) the Young Talent Support Project of Guangzhou Association for Science and Technology (QT-2025-001); Guangdong Basic and Applied Basic Research Foundation (2023A1515110545) Guangzhou Ba- sie and Applied Basic Research Foundation (2025A0413935); and Guangzzhou-HKKUSS((ZZ)) Joint
Funding Program (2025A03J3714),
REFERENCES
Benjamin Bach, Euan Freeman, Alfie Abdul-Rahman, Cagatay Turkay, Saiful Khan, Yulei Fan, and Min Chen. Dashboard design patterns. IEEE Transactions on Visualization and Computer Graphics, 29(1):342-352, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392 13923, 2025.
Scott Bateman, Regan L. Mandryk, Carl Gutwin, Aaron Genest, David McDine, and Christopher Brooks. Useful junk? the effects of visual embellishment on comprehension and memorability of charts, In Proceedings of the SIGCHI conference on human factors in computing systems, pp. 2573-2582, 2010.
Yutong Bian, Xianhao Lin, Yupeng Xie, Tianyang Liu, Mingchen Zhuge, Siyuan Lu, Haoming Tang, Jinlin Wang, Jiayi Zhang, Jiaqi Chen, et al. You don't know until you click: Automated gui testing for production-ready software evaluation. arXiv preprint a50i 14104, 2025.
11

---

Published as a conference paper at ICLR 2026
Michelle A Borkin, Azalea A Vo. Zoya Bylinskii, Phillip Isola, Shashank Sunkavalli, Aude Oliva, and Hanspeter Pfister. What makes a visualization memorable? IEEE transactions on visualiza- tion and computer graphics, 19(12)2330-22 2315, 2013.
Richard Brath and Ebad Banissi. Using typography to expand the design space of data visualization. She Ji: The Journal of Design, Economics, and Innovation, 2(1):59-87, 2016.
Robert L Brennan, Generalizability theory. Educational Measurement: Issues and Practice, 11(4): 27-34, 1992.
Shuo Cao, Nan Ma, Jiayang Li, Xiaohui Li, Lihao Shao, Kaiwen Zhu, Yu Zhou, Yuandong Pu, Jiarui Wu, Jiaquan Wang, et al. Artimuse: Fine-grained image aesthetics assessment with joint scoring and expert-level understanding. arXiv preprint arXiv:2507. 14533, 2025.
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Milm-as-a-judge- Assessing multimodal Ilm-as-a-june a-a-a-----ude with vision-language benchmark. In Forty-first International Conference on Machine Learning. 2024a.
Nan Chen, Yuge Zhang, Jiahang Xu, Kan Ren, and Yuqing Yang. Viseval: A benchmark for data visualization in the era of large language models. IEEE Transactions on Visualization and Com- puter Graphics, (1():1301-1311, 2024b.
Xi Chen, Wei Zeng, Yanna Lin, Hayder Mahdi Ai- Ai-Maneea, Jonathan Roberts, and Remco Chang- Composition and configuration patterns in multiple-view visualizations. IEEE Transactions on Visualization and Computer Graphics, 27(2):1514-1522 2020.
Yiyu Chen, Yifan Wu, Shuyu Shen, Yupeng Xie, Leixian Shen, Hui Xiong, and Yuyu Luo. Chart- mark: A structured grammar for chart annotation. CoRR, abs/2507.21810, 2025, doi: 10.48550/ ARXIV.2507.2188 21010. URL
William Cleveland and Robert McGill. Graphical perception: Theory, experimentation, and appli- cation to the development of graphical methods. Journal of the American Statistical Association,
79(387):531-45 554, 1984.
Victor Dibia. Lida: tool for automatic generation of grammar-agnostic visualizations and info- graphics using large language models. arXiv preprint arXiv:2303.00292 2023.
Ujwal Gadiraju, Ricardo Kawase, Stefan Dietze, and Gianluca Demartini. Understanding malicious behavior in crowdsourcing platforms: The case of online surveys. In Proceedings of the 33rd annual ACM conference human factors in computing systems, pp. 1631-1640, 2015.
Connor C Gramazio, David H Laidlaw, and Karen B Schloss. Colorgorical: Creating discriminable and preferable color palettes for information visualization. JEEE Transactions on Visualization and Computer Graphics, 23(1):521-530) 2017.
Mark Harrower and Cynthia A Brewer. Colorbrewer.org: An online tool for selecting colour schemes for maps. The Cartographic Journal, 40(1):27-37, 2003.
R. Hartman, A. J. Moss, S. N. Jaffe, C. Rosenzweig, L. Litman, and J. Robinson. Introducing connect by cloudresearch: Advancing online participant recruitment in the digital age, 2023. URL URL.stpps///0...orr//... 31113.33311001111333333333 ///3////f..../ wtte4//ff..ooo// 1o/ksgyr Retrieved from https: org 00g/10.312344 20g/10003234//$f.1///os5
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, (22)3,2022,
Kevin Hu, Michiel A Bakker, Stephen Li, Tim Kraska, and César Hidalgo. Vizml: A machine learning approach to visualization recommendation. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1-12, 2019.
Xingyu Lan and Yu Liu. "i came across junk": Understanding design flaws of data visualization from the public's perspective. IEEE transactions on visualization and computer graphics, 31(1): 393-403, 2024.
12

---

Published as a conference paper at ICLR 2026
Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. The dawn of natural language to SQL: are we fully ready? [experiment, analysis & benchmark J. Proc. VLDB Endow., 17 (11):3318-33311 2024a. doi: 111.4777//681954438200000 URL httpp://wwwtt vldb. org/
pvl 2tllb/vo117p333 1t/////33111uu....11 luo.
Boyan Li, Chong Chen, Zhujun Xue, Yinan Mei, and Yuyu Luo. Deepeye-sql: A software- engineering-inspired text-to-sql framework. arXiv preprint arXiv:2510. 17586, 2025a.
Boyan Li, Jiayi Zhang, Ju Fan, Yanwei Xu, Chong Chen, Nan Tang, and Yuyu Luo. Alpha-sql: Zero-shot text-to-sql using monte carlo tree search. arXiv preprini arXiv:2502. 1222:22002./248, 2025b.
Guozheng Li, Runfei Li, Yunshan Feng. Yu Zhang, Yuyu Luo, and Chi Harold Liu. Coinsight: Visual storytelling for hierarchical tables with connected insights. IEEE Transactions on Visual- ization and Computer Graphics, 30(6):3049-3000 -061,, 2024b.
Zhen Li, Duan Li, Yukai Guo, Xinyuan Guo, Bowen Li, Lanxi Xiao, Shenyu Qiao, Jiashu Chen, Zijian Wu, Hui Zhang, Xinhuan Shu, and Shixia Liu. Chartgalaxy: dataset for infographic chart understanding and generation, 2025c. URL URLhttps: wtt.vviivv... org/ ats///bs/22/000 11155555.11666668 18668
Zhuowen Liang, Xiaotian Lin, Zhengxuan Zhang, Yuyu Luo, Haixun Wang, and Nan Tang. Long- caaff-ssruuuurd-thooeeee and fine-tuned slms. In The Fourteenth Interna-
document qa with fional Conference on Learning Representations, 2026.
Xiaotian Lin, Yanlin Qi Yizhang Zhu, Themis Palpanas, Chengliang Chai, Nan Tang, and Yuyu Luo. Lead: iterative data selection for efficient Ilm instruction tuning. arXiv preprint arXiv:2505.07437, 2025.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024.
Xinyu Liu, Shuyu Shen, Boyan Li, Nan Tang, and Yuyu Luo. N12sql-bugs: A benchmark for detect- ing semantic errors in nl2sql translation. In Proceedings of the 31 ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pp. 5662-5673, 2025.
Tianqi Luo, Chuhan Huang, Leixian Shen, Boyan Li, Shuyu Shen, Wei Zeng, Nan Tang, and Yuyu Luo. nvbench 2.0: Resolving ambiguity in visttt-visuullzzation through stepwise reasoning. arXiv preprint arXiv:2503. 2025.
Yuyu Luo, Xuedong Qin, Nan Tang, and Guoliang Li. Deepeye: Towards automatic data visualiza- tion. In 2018 IEEE 34th International Conference on Data Engineering (ICDE), pp. 101-112. IEEE, 2018.
Yuyu Luo, Jiawei Tang, and Guoliang Li. nvbench: A large-scale synthesized dataset for cross- domain natural language visualization task. arXiv preprint arXiv:2112.12926, 2021a.
Yuyu Luo, Nan Tang, Guoliang Li. Chengliang Chai, Wenbo Li, and Xuedi Qin. Synthesizing natural language to visualization (n12vis) benchmarks from n/2sql benchmarks. In Proceedings of the 202/ International Conference on Management of Data, pp. 1235-1247, 2021b.
Yuyu Luo, Nan Tang, Guoliang Li, Jintao Tang, Chengliang Chai, and Xuedong Qin. Natural language visualization by neural machine translation. IEEE Transactions Visualization and Computer Graphics, 28(1):217-266, 2021c.
Yuyu Luo, Xuedi Qin, Chengliang Chai, Nan Tang, Guoliang Li, and Wenbo Li. Steerable self- driving data visualization. IEEE Trans. Knowl. Data Eng., 34(1)4455449) 2022.
Ahmed Masry, Xuan Long Do, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2263-22777 2022.
Andrew McNutt, Gordon Kindlmann, and Michael Correl. Surfacing visualization mirages. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1-16, 2020.
13

---

Published as a conference paper at ICLR 2026
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Anupam Kumar. Plotqa: Reasoning scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1527-1536, 2020.
Dominik Moritz, Chenglong Wang, Gregory Nelson, Halden Lin, Adam M. Smith, Bill Howe, and Jeffrey Heer. Formalizing visualization design knowledge as constraints: Actionable and exten- sible models in draco. IEEE Trans. Visualization Comp. Graphies (Proc. InfoVis), 2019.
Tamara Munzner. Visualization analysis and design. Proceedings of the Special Interest Group on Computer Graphics and Interactive Techniques Conference Courses, pp. 1-2, 2025.
Naila Murray, Luca Marchesotti, and Florent Perronnin. Ava: A large-scale database for aesthetic visual analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 2408 2415, 2012.
Quan Nguyen, Peter Fades, and Seok-Hee Hong. On the faithfulness of graph visualizations. In 2013 IEEE Pacific Visualization Symposium (PacificVis) Vis), pp. 209-216. IEEE, 2013.
Bo Pan, Yixiao Fu, Ke Wang, Junyu Lu, Lunke Pan, Ziyang Qian, Yuhan Chen, Guoliang Wang, Yitao Zhou, and Li Zheng. Vis-shepherd: Constructing critic for Ilm-based data visualization generation. arXiv preprint 2r0iv:2506. 13326, 2025.
Xuedi Qin, Yuyu Luo, Nan Tang, and Guoliang Li. Making data visualization more efficient and effective: a survey. VLDB J., 29(1):93-117, 2020.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-pros q&a bench- mark. In First Conference on Language Modeling, 2024.
Peter J Rousseeuw and Annick M Leroy, Robust regression and outlier detection. John wiley & sons, 2003.
Alper Sarikaya, Michael Correll, Lyn Bartram, Melanie Tory, and Danyel Fisher. What do we talk about when we talk about dashboards? IEEE Transactions on Visualization and Computer Graphics, 25(1):682-692, 2019. doi: 2C.1009/TTCC.221188888869
Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi. Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemati- cal reasoning in open language models. arXiv preprint arXiv:2402.03322 2024.
Leixian Shen, Enya Shen, Yuyu Luo, Xiaocong Yang. Xuming Hu, Xiongshuai Zhang, Zhiwei Tai, and Jianmin Wang. Towards natural language interfaces for data visualization: A survey. IEEE Trans. Vis. Comput. Graph. 29(6):3 1444 2023.
Shuyu Shen, Sirong Lu, Leixian Shen, and Yuyu Luo. Debugging defective visualizations: Empir- ical insights informing human-ai co-debugging system, 2026. URL.hhttss///rrxiv.oor
abs 2413 2. 07673
Jingze Shi, Yifan Wu, Bingheng Wu, Yiran Peng, Liangdong Wang, Guang Liu, and Yuyu Luo. Trainable dynamic mask sparse attention. CoRR, abs/2508.02 122 2025.
Zhihao Shuai, Boyan Li, Siyu Yan, Yuyu Luo, and Weikai Yang. Deepvis: Bridging natural language and data visualization through step-wise reasoning. arXiv preprint arX00 2025.
Danielle Albers Szafir. The good, the bad, and the biased: Five ways visualizations can mislead (and how to fix them). Interactions, 25(4):26-33, 2018.
Yinghao Tang, Xueding Liu, Boyuan Zhang. Tingfeng Lan, Yupeng Xie, Jiale Lao, Yiyao Wang, Haoxuan Li, Tingting Gao, Pan, et al. Igenbench: Benchmarking the reliability of text-to- infographic generation. arXiv preprint urXiv:2601.04490 2026.
Yuan Tian, Weiwei Cui, Dazhen Deng, Xinjing Yi, Yurun Yang, Haidong Zhang, and Yingcai Wu. Chartgpt: Leveraging Ilms to generate charts from abstract natural language. arXiv preprint 1oXiv:2311.01920 2023.
14

---

Published as a conference paper at ICLR 2026
Edward R Tufte. The visual display of quantitative information. Graphies Press, 1983.
Lei Wang, Songheng Zhang. Yun Wang, Ee-Peng Lim, and Yong Wang. LIm4vis: Explainable visualization recommendation using chatgpt. In Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 675-692, 2023.
Liangwei Wang, Zhengxuan Zhang, Yifan Cao, Fugee Tsung, and Yuyu Luo. Tabletale: Reviving the narrative interplay between data tables and text in scientific papers, 2026. URL https:
//arx/v.orr//as//2522.. 22908
Colin Ware. Information visualization: perception for design. Morgan Kaufmann, 2019.
Kanit Wongsuphasawaa, Zening Qu, Dominik Moritz, Riley Chang, Felix Ouk, Anushka Anand, Jock Mackinlay, Bill Howe, and Jeffrey Heer. Voyager 2: Augmenting visual analysis with partial view specifications. In Proceedings the 2017 CHI Conference on Human Factors in Computing Systems, pp. 2648-26535 2017.
Bingheng Wu, Jingze Shi, Yifan Wu, Nan Tang, and Yuyu Luo. Transxssm: A hybrid transformer state space model with unified rotary position embedding. CoRR, abs/2506.09507, 2025.
John Wu, John Joon Young Chung, and Eytan Adar. viz2viz: Prompt-driven stylized visualization generation using a diffusion model. arXiv preprint arXiv:2304.019/3 2023.
Yifan Wu, Lutao Yan, Leixian Shen, Yunhai Wang, Nan Tang, and Yuyu Luo. Chartinsights: Eval- uating multimodal large language models for low-level chart question answering. arXiv preprint MXXv::2405.0701,, 2024.
Yifan Wu, Yiran Peng, Yiyu Chen, Jianhao Ruan, Zijie Zhuang, Cheng Yang, Jiayi Zhang, Man Chen, Yenchi Tseng, Zhaoyang Yu, et al. Autowebworld: Synthesizing infinite verifiable web environments via finite state machines. arXiv preprint arXiv:2602.1 14296, 2026.
Yupeng Xie, Yuyu Luo, Guoliang Li, and Nan Tang. Haichart: Human and Al paired visualization system. Proc. VLDB Endow., 17(11):318-3311 2024. doi: 10. 14.14778/368 011133661199443333 1992. URL https: www. vldb. otg. httpddd/olllllc
Zhiyu Yang. Zihan Zhou, Shuo Wang. Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, et al. Matplotagent: Method and evaluation for Ilm-based agentic scientific data visualization. arXiv preprint arXiv:2402.1 1453, 2024.
Yilin Ye, Jianing Hao, Yihan Hou, Zhan Wang, Shishi Xiao, Yuyu Luo, and Wei Zeng. Generative ai for visualization: State of the art and future directions. Visual Informatics, 8(2):43666, 2024. ISSN 2468-502X.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yong Zhuang, Zhuo- han Lin, Dacheng Li, Eric Xing, Hao Zhang, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2023.
Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint a50iv:2504. 10479, 2025a.
Yizhang Zhu, Shiyin Du, Boyan Li, Yuyu Luo, and Nan Tang. Are large language models good statisticians? Advances in Neural Information Processing Systems, 37:62697-62731 2024.
Yizhang Zhu, Liangwei Wang, Chenyu Yang, Xiaotian Lin, Boyan Li, Wei Zhou, Xinyu Liu, Zhangyang Peng, Tianqi Luo, Yu Li, al. A survey of data agents: Emerging paradigm or overstated hype? arXiv preprint arXiv:2510.23587 2025b.
15

---

Published as a conference paper at ICLR 2026
Appendix Contents
LLM Usage Statement
Appendix A. Dataset Construction Details and Statistical Analysis
A.1
Data Collection and Construction Pipeline
A.2
Dataset Statisties and Distribution Analysis
20
A.3
Annotation Process
26
A.4
Annotation Quality Analysis
20
Appendix B. Case Studies
B.1
High-Score Case Studies: Human- Alignment
34
B.2
Medium-Score Case Studies: Human-Model Alignment
B.3
Low-Score Case Studies: Human-Model Alignment
B.4
Dimension-Specific Case Studies: Validating Evaluation Criteria
B.5
Model Error Analysis Cases
B.6
Error Analysis of Multiple Visualizations and Dashboards
Appendix C. Evaluation Framework and Criteria
C.I
Detailed Evaluation Questions and Scoring Criteria
46
C.2 Single Visualization Evaluation Criteria
48
C.3 Multiple Visualization Evaluation Criteria
C.4 Dashboard Evaluation Criteria
54
C.5 Evaluation Prompt Templates
57
Appendix D. Model Implementation and Training Details
D.1
Software Environment
59
D.2
Reward Function
D.3
Hyperparameter Settings
59
Appendix E. Extended Experiments
E.1 How Does Training Data Scale Affect Model Performance?
60
E.2
Generalization to World Applications
E.3
Cross-Architecture Generalization
16

---

Published as a conference paper at ICLR 2026
LLM USAGE STATEMENT
We used Claude-4-Sonnet for English grammar polishing and consulted Claude-4-Sonnet for sug- gestions figure layout and color design. During dataset construction, used GPT-4o for auto- mated adaptive question generation and chart metadata extraction. All code was written, reviewed, and verified by the authors. All prompts contained no private sensitive data. Large language models did not provide any novel algorithmic ideas or academic claims; the authors take full re- sponsibility for the content. Large language models are not authors of this paper.
A DATASET CONSTRUCTION DETAILS AND STATISTICAL ANALYSIS
A.1 DATA COLLECTION AND CONSTRUCTION PIPELINE
To build a large-scale and diverse visualization dataset, we designed and implemented systematic web crawling and data filtering pipeline. This process aims to collect wide range of visualizations from the web, spanning from poorly designed examples professional exemplars, while ensuring that all collected data is of high relevance and quality. The entire pipeline consists of three core stages: keyword generation, a high-throughuut crawling architecture, and multi-stage filtering.
Keyword Generation Strategy. The foundation of our data collection is a meticulously designed keyword generation strategy to ensure broad coverage across visualization types, quality levels, and application domains.
Base Keyword Lexicon: We first established base lexicon of over 200 professional vi- sualization terms, such as "professional bar chart design," "clean line graph visualization," and "business intelligence dashboard."
Visualization Type Expansion: Building on this, we systematically incorporated over 30 different chart types, covering basic charts (e.g., bar, line, pie charts), advanced visu- alizations (e.g., Sankey diagrams, treemaps, radar charts), and interactive systems (e.g., interactive dashboards, animated charts).
To intentionally capture charts of varying quality lev- els, we programmatically combined chart types with high-quality modifiers (e.g., "pro- fessional," "clean," "effective." "well-designed") and low-quality modifiers (e.g.. "poor," "cluttri " "misleading").
Quality Modifier Combination:
"confusing," " "cluttered,"
Domain-Specifie Terminology: We also integrated professional terminology from 20 application domains (e.g.. business, finance, healthcare, education) to generate context- specific search queries, such "financial dashboard," " "sales performance chart," and "COVID cases chart."
This automated strategy ultimately generated over 2,000 unique, high-quality search keywords, lay- ing a solid foundation for our large-scale data crawling efforts.
Tiig--Throughut Crawling and Preliminary Filtering To efficiently collect vast number of candidate images from the web, we developed a high-throughput crawling architecture based on Bing Image Search. This architecture utilizes multi-threaded, asynchronous requests to fetch up to 10 pages of search results for each keyword, maximizing data recall. High-resolution image URLs were reliably extracted by parsing JSON data embedded within the web pages. During the crawling phase, we implemented an initial round of automated preliminary filtering:
We strictly filtered images by size, requiring minimum width of 400 pixels, a minimum height of 300 pixels, and a total area of at least 150,000 pixels. This effectively eliminated low-resolution thumbnails and icons.
Size Filtering:
Heuristic Content Pre-screening: We conducted a rapid pre-assessment of image content using programmatic analysis techniques. By employing edge detection algorithm and color complexity analysis (counting the number of unique colors), we discarded a signif- icant number of images that were either too simple (e.g., solid-color backgrounds, blank images) or too complex (e.g., real-world photographs), as these typically do not represent data visualizations.
17

---

Published as a conference paper at ICLR 2026
This stage yielded a large-scale preliminary dataset containing tens of thousands of candidate im- ages, laying the groundwork for subsequent fine-grained refinement.
Fine-Grained Filtering and Hierarchical Classification via Multimodal LLMs To precisely filter high-quality, relevant visualizations from the preliminary dataset and organize them into a structured classification, designed a fine-grained filtering pipeline centered around an MLLM.
Perceptual Hash Deduplication: Before semantic analysis, we first employed a Percep- tual Hashing (pHash) algorithm deduplicate all candidate images. This technique iden- tifies visually identical or highly similar images, regardless of differences in size, format, compression. By setting strict similarity threshold (Hamming distance <), effec- tively ensured the diversity of the final dataset and eliminated redundancy.
Prompt-Based AI Semantic Filtering: We utilized an advanced MLLM (e.g., GPT-4o) as our core classifier. We engineered highly restrictive system prompt that defined the model's primary task as that of a strict filter rather than a simple classifier. This prompt compelled the model to adhere to the following top-priority rules:
1. Reject Non-Sereenshot Images: Any image appearing to be a photograph, con- taining tilted perspectives distortions, including real-world environments (e.g.. monitor bezels, keyboards, desks) was immediately classified as non-compliant mton_visualizat
Reject Images with People: Any image containing human figures (including car- toons) or body parts (e.g., hands, fingers) was strictly filtered out.
3. Reject Work-in-Progress and Development Interfaces: We mandated that only "finished" visualizations retained. Any screenshot depicting the visualization cre- ation process- such as those including software UI elements (menus, toolbars, prop- crty panels), code editors (like Jupyter Notebooks), or configuration windows-wer also classified non-compliant.
The full content of this prompt detailed the "Prompt Template" box below.
Hierarchical Content Classification: Only images that passed all the stringent screening criteria and were identified "clean, front-facing, person-free visualization screenshots" proceeded the classification stage. The model then categorized them into hierarchical system based on their structure and function, primarily including: single visualizations, multiple visualizations, and dashboards.
Prompt Template: Fine-Grained Filtering and Classification via Multimodal LLM
You are a professional data visualization analysis expert. Your core mission is to strictly filter and accurately classify data visualization images.
WARNING Highest Priority Principle: Absolutely reject all photographs and any images containing people.
Your primary duty is must first assess its form.
rigorous filter. Before evaluating the content of image, you
Is it a photograph? If yes, immediately classify as non _ao_ visualization.
Does it contain people? yes, immediately classify as non_ non_visualization.
Is it tilted in perspective? If yes, immediately classify non_ visualization.
Only when an image perfectly meets the standard of "clean, front-facing, person-free screen- shot" can you proceed to analyze its content.
Strict Filtering Criteria (Classify as non_visualization if any condition is met):
1. Reject ALL Photographs:
Characteristic:
Reject any image that appears to be taken with a camera rather than direct screenshot.
Clues:
Tilted Angle/Perspective Distortion: The image is not flat and front- facing.
18

---

Published as a conference paper at ICLR 2026
Physical borders of a laptop, monitor, phone, or tablet are visible.
Device Bezels:
Real-World Environment: Backgrounds like desks, offices, conference rooms, keyboards, or mice are visible.
People or Body Parts: Any presence of people, hands, or fingers.
Reflections or Screen Moire: Reflections of ambient light the screen.
2. Reject ALL Images with People:
Characteristic:
Absolutely forbidden. Any image containing people in any form (full body, portrait, cartoon) body parts (hands, fingers) must be rejected.
3. Reject Marketing/Concepl Images:
Characteristic:
Images that look like stock photos, promotional materials, web- site banners, stylized concept designs. These often have artistic effects, tilted perspectives, or non-data elements and are not genuine analytical tool interfaces.
Content Classification Criteria (Applicable only clean screenshots that pass the above filters):
1. Single Visualization (single_viss view): A pure, single, complete data chart.
Ultra-Strict Prerequisites (All must be met):
(a) Pure Chart: The main subject of the image must be the chart itself, with no external UI elements.
(b) No Editing Controls: must absolutely not contain any UI elements for configuring editing the chart (e.g.. toolbars, property panels, formatting panes, pop-up menus). such elements are present, must be classified as von_visualization.
(c) Front-facing, person-free, freee non con-phooograph screenshot.
2. Multiple Visualizations (multi :iet): A pure composition of multiple data charts for analysis.
Ultra-Strict Prerequisites:
(a) Pure Chart Composition: The image must only contain data charts, with ab- solutely no other types elements.
(b) Multiple Charts: must contain 2 or more independent data charts.
(c) Front-facing, person-free, non-photograph, non-editing interface screenshot.
3. Dashboard (dashboard): An end-user-facing interactive interface for data exploration and monitoring.
Ultra-Strict Prerequisites: Must be front-facing, clean screenshot, absolutely free of any people or device bezels.
Core Features:
-Composed of multiple, coordinated data charts and KPI metrics. (A single chart does not constitute a dashboard).
IInteractive elements are end-user-facing and intended for data consumption (e.g., filtering, drilling down, switching views), not for chart creation or edit- ing.
4. Non-Compliant Image (non_ visualization): Any image that is not pure, finished data visualization product.
This category serves as a "catch-all" filter out all visualizations that do not meet the strict criteria.
Core Judgment: Is this image showing something "in progress" is it "finished product"? Anything "in progress" non-compliant.
Docision-Making Process (Strictly Adhered To):
1. Step 1: Check for "Non-Compliant Image" viouaiiuuaiiztion). (This is the highest- priority filter).
19

---

Published as a conference paper at ICLR 2026
2. Step 2: Classify the "pure visualization products" that pass the first step.
3. Step Differentiate between Multiple Visualizations and Dashboard.
Return Format: JSON object with the following structure:
"category"; "primary category in Engli ish (single_ dashboa paco__visuliiaaee "cype": "Sub-type in English", "confidence": "Confidence score (0-100)",
"reasoning": "Provide the core reason for the judqment, e.g., 'The image is a mootograph/contatie people/is not front-facing, and is therefore a nen-compliant image.'"
This multi-stage pipeline, combining heuristic pre-screening, perceptual hash deduplication, and AI- driven semantic refinement, enables the fully automated construction of high-quality, structured visualization dataset from vast web-scale data. It significantly reduces the manual annotation burden while ensuring the relevance and quality of the collected data.
A.2 DATASET STATISTICS AND DISTRIBUTION ANALYSIS
A.2.1 OVERALL DATASET STATISTICS AND DISTRIBUTION
VISJUDGE- BENCH consists of 3,090 professionally assessed visualizations covering the full range of modern visualization design. was constructed to ensure broad coverage across visualization types, evaluation dimensions, and quality levels, reflecting practices in business intelligence, aca- demic research, and data journalism. The collected quality scores approximately follow normal distribution (mean 3.13, std 0.72, range from poor to exemplary designs. Figure presents representative visualization examples across dif- ferent quality score ranges (from 1-2 to 4-5), showcasing the diversity of visualization types and the clear quality distinctions captured by our evaluation framework. All samples include complete six- dimensional annotations, enabling users study visualization quality holistically as well as across specific types, subtypes, and evaluation dimensions.
Figure Sa), capturing a broad range
A.2.2 VISUALIZATION TYPE AND SUBTYPE ANALYSIS
Visualization Classification. A hierarchical taxonomy organizes visualizations by structural com- plexity and functional purpose. includes three major categories: single visualizations (1,041 samples, 33.7%). multiple visualizations (1,024 samples, 33.1% and dashboards (1,025 samples, 33.2%). These categories further expand into 22, 5, and 5 subtypes respectively, ensuring represen- tation from basic charts (e.g., bar, pie, line) advanced analytical dashboards. This classification system allows users to study quality differences across visualization types and subtypes. Detailed information can be found in Table (S and Figure illustraaees the quality score distributions across these three major categories, revealing distinct distribution patterns for each visualization type.
For the single visualizations (1,041 samples, 22 subtypes), we observe clear head-tail distribution. Four basic chart types-bar, bar, pie, line, and area charts charts-account for 46.1% (480 samples), reflect- ing their central role in practical visualization design. second tier of commonly used analytic charts (e.g., treemap, Sankey, heatmap, scatter, histogram, funnel, bubble) contributes 38.0% (396 samples). The remaining 15.9% (165 samples) consists of more specialized forms, where network graphs make up about 2.2% and map-based views (choropleth and point maps) around 3.6%, with the rest spread across other rare, domain-specificc encodings. This skewed distribution in line with (Borkin et al., 2013) empirical studies of real-world visualization usage.
Reflecting real-world usage patterns, the distribution of our multi visualizations samples (1,024 samples, subtypes) is heavily skewed towards Comparison views, which constitute the majority (65.4%, 670 samples). Small multiples (19.0%) and coordinated views (9.5%) follow as key pat- terns, with the remaining 6.1 1 covering sverview-detal layouts and other forms. This composition
20

---

Published as a conference paper at ICLR 2026
250
95
Probat By Densty
Meon 2.9
200
76
Medon: 2.940
150
57
Count
100
38
50
19
2.5
3.5
1.0
1.5
3.0
3.5
4.5
Quality Score
Quality Score
Overall Score Distribution
(b) Single Visualization Score Distribution
97
143
Probeeiity by Denetty
Pictabily Dereit
Meon
Newdin::
114
Count
39
57
19
28
2.5
Quality
Quality
(c) Multiple Visualization Score Distribution
(d) Dashboard Score Distribution
Figure 8: Quality score distributions of the dataset across different visualization categories.
Comprehensive Visualization
Comprehensive Visualization Evaluation Score 2-3
Comprehensive Visualization Evaluation Score 3-4
AOANG ARLUINN IBAISIROONGG BAR CWAYS
Dispicced Aves
PAA
Before
After
mnmesse
Comprehensive Visualization Evaluation Score 4-5
Figure 9: Representative samples from VISJUDGE-BEL BENCH
aligns with established characterizations (e.g., (Chen al. 2020 ensuring our benchmark captures representative cross-section of common multi-view configurations.
21

---

Published as a conference paper at ICLR 2026
Table 5: VISJUDGE- statistical detailed information.
Vis Type
Count
Proportion
#-Subtype
Subtype
Details
Bar Chart
176
Bubble Chart
29
Pie Chart
129
Choropleth Map
25
Line Chart
100
Radar Chart
24
Area Chart
75
Network Graph
23
Treemap
Candlestick Chart
20
Single Vis
1,041
33.7%
22
Sankey Diagram
61
Gauge Chart
20
Heatmap
Box Plot
17
Seatter Plot
49
Point Map
12
Histogram
48
Word Cloud
Donut Chart
Violin Plot
Funnel Chart
45
Other Single View
22
Comparison Views
670
Overview Detail
Multi Vis
1,024
33.1%
5
Small Multiples Coordinated Views
195
Other Multi View
59
97
Analytical Dashboard
743
Strategic Dashboard
62
Dashboard 1,025
33.2%
5
Operational Dashboard
122
Other Dashboard
7
Interactive Dashboard
91
Among the dashboard samples (1,025 samples, 5 subtypes), Analytical dashboards are overwhelm-
ingly dominant, constituting 72.5% (743 samples) of the dataset. The remainder consists of Op- erational (11.9%) Interactive (8) and Strategic (6.0%) types, with a negligible fraction (0.7%) classified other forms. This distribution, heavily weighted towards analysis- and decision- tools, mirrors the real-world prevalence observed in genre analyses by (Sarikaya et al. 2019) and design surveys by (Bach et 2023).
A.2.3 DESIGN STYLE DISTRIBUTION ANALYSIS
From a quantitative perspective, Figure 10 shows the score distribution for the design style dimen- sion: this dimension has a mean score of 2.97, showing relatively uniform distribution that indi- cates the dataset covers diverse design styles ranging from low to high innovation.
To further understand the style diversity in dataset, we refer to related work chart de- sign styles (Borkin et al.] 2013] Bateman et al.) 2010). Prior research has shown that visualiza- tion design exhibits different orientations ranging from function-oriented expression-focusee proaches (Borkin et al. 2013), and that different visual styles can influence the aesthetic perception and memory effects of visualizations (Bateman al., 2010). Based on these studies, we conducted manual inspection of all 3,090 samples in the dataset and identified the following main style cate- gories: (1) Professional/Standdrd Style (53.3%), adopting clean and standardized designs commonly found in business intelligence tools and academic publications, corresponding to imnction-oriented design approaches; (2) Freative/Infographic Style (28.4%), demonstrating more creative design ap- proaches prevalent in data journalism and marketing materials, reflecting sspression-focusee design orientations; (3) Minimalist Style (18.3%), following minimalist design principles by reducing vi- sual elements and using generous whitespace emphasize data clarity.
ap-
We acknowledge that this style classification is coarse- coarse-grained and somewhat subjective, as design style itself is multi-dimensionall and context-dependent. A more systematic style classification sys- tem would require dedicated annotation work, which beyond the scope of this work. Nevertheless, this preliminary analysis demonstrates that VisJudge-Bert Bench covers diverse design approaches com- monly found in real-world practice, ranging from function-oriented standardized designs to visually engaging creative expressions.
A.2.4 APPLICATION DOMAIN DISTRIBUTION ANALYSIS
We annotated the dataset using high-level categories from the IPTC taxonomy (Li et al. 2025c).
About 6.2% of samples grouped "other" due to limited context. Since the dataset
22

---

Published as a conference paper at ICLR 2026
Data Fidelity
Sermantic Readability
Insight Discovery
3:138
800
800
Median: 3.000
Median: 3.
Modian: 3,000
500
600
600
400
Count
Count
300
400
400
200
200
200
100
Quality Score
Quality Score
Quality Score
Design Style
Visual Composition
Color Harmony
800
Mean: 2.972
1000
Mean: 3.314
1000
Mean: 3.369
Median: 3.030
Modian: 3.500
Median: 3.500
600
800
800
Count
Count
600
400
400
400
200
200
200
5
0
1
2
Quality Score
Quality Score
Quality Score
Figure 10: Quality score distributions across six evaluation dimensions.
structed from real-world web sources through our systematic crawling pipeline (see Appendix I). its domain distribution naturally reflects actual visualization demand and usage patterns. The an- notated subset shows a clear concentration: economy. business and finance (26.3%), science and technology (20.3%), and labor (15.7%) together account for 62.3% of all labels, aligning with high practical demand in these fields. A middle tier includes society (6.6%) and domains like lifestyle and leisure, health, and politics and government (ranging from 2.7% to 4.9%), while topics like crime, law and justice and religion are rare (<1%). This concentration is even more pronounced in dashboards, where the top three categories comprise 78.5% of the data, reflecting the dominant role of business analytics in dashboard applications.
We further analyze the domain co-occurrence matrix understand how topics are combined within the same visualization or dashboard. Economy, business and finance plays a central role and most often appears together with labor, science and technology, and society, reflecting typical themes such as employment, productivity, and general economic conditions. Science and technology often co- occurs with health, environment, and education, corresponding to topics such as medical research, climate and energy, and science education. In addition, politics and government frequently appears with economy, business and finance and society, while environment is closely linked to both sci- ence and technology and weather. These co-occurrence patterns show that many visualizations in our dataset address cross-domain topics rather than a single isolated domain, capturing common interdisciplinary scenarios in practice.
A.2.5 QUALITY SCORE DISTRIBUTION
Quality Grade Distribution. To examine the overall quality of the dataset, we analyze the distri- bution of quality scores across different visualization categories. Figure X presents the histograms with fitted density curves, highlighting both the mean and median values for each category. This analysis allows to compare quality differences between single visualizations, multiple visualiza- tions, and dashboards.
Quality Distribution Across Visualization Categories. The overall quality score distribution (Figure Ba) exhibits a indicating balanced representation of both lower and higher quality samples.
distribution with scores predominantly ranging from 2.0 to 4.0,
23

---

Published as a conference paper at ICLR 2026
Individual visualization categories show distinct patterns. Single visualizations (Figure 8b) and multiple visualizations (Figure [c) exhibit similar quality levels with means of 2.910 and 2.917 re- spectively, both displaying broad distributions across the quality spectrum. In contrast, dashboards (Figure (d) show notably higher scores (mean: 3,555, median: 3,610) with a right-skewed dis- tribution concentrated in higher quality ranges. This difference reflects that published dashboards typically undergo more rigorous design review as polished, production-ready tools, while single and multiple visualizations include more experimental designs with varying execution quality.
Quality Distribution Across Evaluation Dimensions. Figure 10 reveals distinct patterns across the six evaluation dimensions, which align with our three-tier framework of Fidelity, Expressiveness, and Aesthetics.
At the Fidelity level, data fidelity (mean: 3.138, median: 3.000) exhibits balanced near-normal
distribution, indicating varied success in truthful data representation-a fundamental requirement that shows substantial room for improvement across the dataset.
At the Expressiveness level, both semantic readability (mean: 2.983, median: 3.000) and insight discovery (mean: 2.983, median: 3.000) center around 3.0 with broad distributions, reflecting the persistent challenge of effective communication and analytical support. These similar patterns sug- gest that clarity and insight facilitation remain equally difficult aspects of visualization design.
At the Aesthetics level, we observe gradient in achievement: color harmony (mean: 3.369, me- dian: 3,500) and visual composition (mean: 3.314, median: 3,500) show the highest scores with right-skewed distributions, benefiting from well-established design guidelines and tool support. In contrast, design style (mean: 2.972, median: 3.000) shows the lowest average with broader spread, reflecting its subjective nature and the varying emphasis placed on stylistic sophistication versus functional priorities.
This hierarchical distribution pattern-froe foundational data accuracy, through communicative ef- fectiveness, to aesthetic tetnemennt-nnsreee that our benchmark evaluates models across the com- plete spectrum of visualization quality assessment.
A.2.6 LOW-QUALITY SAMPLE DISTRIBUTION AND DESIGN FLAW ANALYSIS
We observe distinet correlation between visualization gramularity and quality stability. The pro- portion of low-quality samples (score < 2) decreases markedly as complexity increases: from 12.5% in single views (130/1041), to 9.2% in multi-views (94/1024), and down to just 1.3% in dashboards (13/1025). likely driver dashboards often represent mature, engineered products, whereas the single-view dataset contains more exploratory, or structurally complex charts (e.g., network dia- grams, treemaps, heatmaps) prone to design failures.
Single visualizations. Quality issues in this subset are unevenly distributed, clustering heavily in complex or less common categories. Five specific categories, Treemaps, Scatter Plots, Network Graphs, Heatmaps, and Pie Charts, collectively account for 57.7% of all low low-quality instances. Across dimensions, design style emerges as the primary bottleneck (mean 2.623, 34% low-score rate), followed by semantic readability (2.831, 31%) and insight discovery (2.779, 32%). Our analy- sis reveals that the mechanisms of failure diverge significantly based on chart complexity. For com- plex chart types, quality problems are typically multi-ddmmensinnl rather than isolated to a single criterion. In the Candlestick Chart, 55% of such samples fail on at least three dimensions simul- taneously, as semantic illegibility triggers a chain reaction that degrades data fidelity and insight discovery. Conversely, standard formats like Bar and Line charts generally maintain robust fidelity and readability (mean approx 3.000) while suffering primarily from isolated lacks in aesthetic re- finement rather than fundamental communicative breakdowns.
Multiple visualizations. While Comparison Views have a low-score rate (7.2%), their predom- inance in the dataset means they contribute over half (51.1%) ()) of all low-quality multi-views. A critical finding here is the divergence between visual and semantic performance. Visual composi- tion and color harmony achieve high stability (mean 3.00 low-score rates < 12%). yet semantic readability and insight discovery lag significantly (mean < 2.700, low-score rates > 33%). This indicates that the core challenge in multi-view design is not layout coloring, but semantic integra- tion-users struggle to bridge information across views. Furthermore, these semantic failures are strongly coupled with Data Fidelity: in Comparison Views, approximately 20% of cases fail data
24

---

Published as a conference paper at ICLR 2026
Quality Eooee
2.0
N0rua Temmp
CMA
Sagl Vey single Subnype
CMA
The
5.0
4.0
Qualtty Store
2.5
30
13
Mall Pow
Cold
Beshward
Multi Subtype
Dashiooard
Figure 11: Quality score distributions across visualization subtypes.
fidelity, semantic readability, and insight discovery simultaneously, pointing to systemic inability to convey trustworthy information.
Dashboards. Representing the most stable layer, dashboards exhibit high baseline scores, particu- larly visual dimensions (visual composition and color harmony mean > 3.650, low-score rates 3.5%). The few remaining low-quality cases are concentrated in Analytical and Interactive subtypes (38.5% of total failures). We observe significant error coupling in these high-stakes scenarios: for instance, 12.1% of Interactive Dashboards suffer from joint failures in data fidelity and semantic readability and 6.1% % of Analytical Dashboards are simultancously low on semantic readability and insight discovery. This suggests that dashboard failures are rarely aesthetic but functional, mani- festing in high-stakes picision-making scenarios as coupled deficits in semantic explanation, metric summarization, and insight extraction.
To illustrate these design flaw patterns, we provide detailed case studies in Appendix B Ap-
pendix B.T B.2 and [3. present representative cases across high-score (>4), medium-score (2 224), and low-score (<2) ranges for different visualization types; Appendix [B.4] provides typical low- score examples and problem analyses for the three core dimensions data fidelity, expressiveness, and aestheties. These cases clearly demonstrate common quality issues and failure modes for each visualization type.
A.2.7 TEST SET DISTRIBUTION ANALYSIS
To ensure reliable and comprehensive evaluation of model performance, we partitioned the dataset into training (70%, 2,163 samples), validation (10% 279 samples), and test (20%, 648 samples) sets using stratified sampling based on visualization types. 6 of the test set across visualization types and subtypes, demonstrating that the stratified sampling successfully maintains proportional representation consistent with the overall dataset.
the detailed distribution
The test comprises 231 single visualizations (35.6%), 209 multiple visualizations (32.3%), and 208 dashboards (32.1% 16), closely mirroring the overall dataset distribution. Within single visual- izations. the test set covers 20 distinct chart types, ranging from common charts like bar charts (37 samples) and pie charts (27 samples) to specialized visualizations such as Sankey diagrams (14 samples) and network graphs (5 samples). Multiple visualizations include 135 comparison views, 41 small multiples, and 20 coordinated views, while dashboards predominantly feature analytical dashboards (150 samples) alongside operational and interactive dashboards. The test set also pre- serves domain diversity, covering all major application domains to ensure comprehensive evaluation across different real-world contexts.
The test set maintains quality distribution characteristies similar to the full dataset, with mean of 3.13 and standard deviation of 0.72, ranging from 1.11 to 4.89. Score distribution across quality ranges shows 7.4% low-quality samples (1.0-2.0), 31.5% below-average samples (2.0-3.0), 49.5%
25

---

Published as a conference paper at ICLR 2026
Table 6: Test statistical detailed information (N=648, 20% of "ISJUDGE-BENCH)
Vis Type
Count
Proportion
#-Subtype
Subtype Details
Bar Chart
Bubble Chart
Pie Chart
27
Other Single View
6
Line Chart
Choropleth Map
Area Chart
15
Radar Chart
6
Treemap
Candlestick Chart
Single Vis
231
35.6%
20
Sankey Diagram
14
Gauge Chart
Heatmap
12
Network Graph
Histogram
Point Map
Scatter Plot
Box Plot
Donut Chart
Funnel Chart
10
Comparison Views
135
Other Multi View
13
Multi Vis
209
32.3%
4
Small Multiples Coordinated Views
41
20
Analytical Dashboard
150
Other Dashboard
Dashboard 208
32.1%
5
Operational Dashboard
25
Interactive Dashboard
Strategic Dashboard
13
Total
648 samples
above-average samples (3.0-4.0), and 11.6% high-quality samples (4.0-5.0). This balanced distri- bution ensures comprehensive evaluation across the full quality range, enabling robust assessment of model performance on both challenging low-quality and high-quality visualizations.
A.3 ANNOTATION PROCESS
A.3.1 ANNOTATOR RECRUITMENT STANDARDS
To ensure high-quality responses and reduce the risk of careless or malicious submissions, we im- plemented strict screening criteria during the annotator recruitment process.
To maintain annotation quality, we applied the following recruitment criteria:
Education: Participants were required to have completed at least a Bachelor's degree. Preference was given to those with Master's, professional, or doctoral degree ensure familiarity with analytical and design tasks.
Approval Rating:
Only individuals with historical approval rate between 97% and 100% were permitted to participate, reflecting a track record of reliable and consistent task com- pletion the platform.
Approved Projects Count: Annotators were selected from those who had completed be- tween 100 and 10,000 approved projects, ensuring adequate experience with crowdsourcing workflows.
English Language: All participants were required be native English speakers to guar- antee accurate comprehension of visualization-related terminology and rubric-based ques- tions.
We targeted professionals working in relevant domains such arts, business, education, finance, STEM, public administration, and product design, to match the content and context of visual analysis tasks.
Occupation Field:
Participants were drawn from white-collar, creative, and IT-related professions, including developers, designers, analysts, and content creators, all of whom typically interact with visual content in their daily work.
Job Classification:
26

---

Published as a conference paper at ICLR 2026
Last Project Completed: Annotators required to have completed project within the past 180 days, ensuring recent and active engagement with the platform.
Age: To maintain a cognitively active and professionally engaged participant pool, we limited participation to those aged between 20 and 50 years.
Technical Skills: We prioritized individuals proficient in data science, product design, front-end development, computer science, and other related technical fields that support informed and thoughtful visual reasoning.
A.3.2 ANNOTATION TASK DESIGN
Our BAALUGGE-BENNCC contains 3,090 visualizations, each requiring evaluation across dimen- sions. To ensure annotation quality and allow annotators familiarize themselves with the eval- uation criteria, we organized the annotations into batches of 15 images per task, with each batch carefully balanced include single visualizations, multiple visualizations, and 5 dashboards. Before starting each task, annotators were presented with detailed explanations of the evaluation framework, including the meaning and significance of each dimension (Fidelity, Expressiveness, and Aesthetics), enabling them to quickly understand the task requirements and evaluation stan- dards. With questions per image, each annotation task comprised questions and typically took 30-60 minutes, depending annotator familiarity with the criteria and visualization complexity. Each task was independently annotated by three qualified participants to ensure reliability through majority voting and enable inter-annotator agreement analysis. Annotators estimated hourly rate of $10 USD, which competitive relative to the platform average and helped attract and retain qualified workers able to devote adequate attention to the task. Together with our strict screening criteria and embedded validation checks, this compensation structure supported the collection of high-quality annotations.
compensated at
Each visualization is evaluated across six dimensions derived from our "Fidelity, Expressiveness, and Aesthetics" framework: (1) Data Fidelity, which assesses whether the visual representation accurately reflects the underlying data; (2) Semantic Readability, which evaluates whether infor- mation clearly conveyed; (3) Insight Discovery. which measures whether meaningful patterns are discoverable; (4) Design Style, which assesses aesthetic innovation and uniqueness; (5) Visual Composition, which evaluates spatial layout and balance; and (6) Color Harmony, which mea- sures color coordination and effectiveness. For each dimension, annotators provide ratings on a 1-5 scale, where each rating level is accompanied by clear descriptive criteria to ensure consistent in- terpretation across annotators (see Appendix C for detailed evaluation questions and scoring criteria for each dimension).
A.3.3 ANNOTATION INTERFACE AND WORKFLOW
We designed a dedicated crowdsoureing interface to ensure annotators clearly understood the task, followed structured workflow, and submitted high-quality responses. Before beginning the eval- uation, participants were presented with both a brief and an extended task introduction. The short version stated:
Evaluate 15 data visualizations with 90 simple multiple-choice questions (6 per chart) covering Fidelity (data accuracy). Expressiveness (information clarity), and Aesthetics (visual aesthetics). Each question has clear 1-5 rating descriptions to make evaluation straightforward. Please only participate if you can provide thoughtful responses-wy 'vve designed this to be as simple as possible for you!
The extended version was shown in full on the task interface:
Welcome! Thank you for joining our study on data visualization quality. You will evaluate 15 data visualizations with 90 simple multiple-choice questions based three classical design principles:
Fidelity: Data Fidelity- whether the visual representation accurately reflects the underlying data.
27

---

Published as a conference paper at ICLR 2026
Semantic Readability, Insight Discovery whether information is clearly conveyed and meaningful patterns are discoverable.
Expressiveness:
Aesthetics: Design Style, Visual Composition, Color Harmony whether the visu- alization has aesthetic appeal and professional design quality.
For each chart:
View the image and its description
Answer 6 straightforward questions (1 for Fidelity, for Expressiveness, 3 for Aes- thetics)
Simply select your rating from I (Poor) to 5 (Excellent) each option has clear descriptions to guide your choice
We' 'e designed this minimize your effort while ensuring quality feedback. Please take your time to carefully consider each visualization before making your selections. Please consider the time commitment carefully and only proceed if you can provide thoughtful. quality responses. If you' re not ready to participate seriously, feel free to skip this task. We will check for quality and may reject careless random responses. Your thoughtful and careful feedback is important-thank you!
As illustrated in Figure 12 the annotation interface presented one chart at time, along with textual description. Below the visualization, the six evaluation questions were displayed, customized based on the chart content. To proceed, participants were required to complete all six questions. At the end of each chart, annotators also rated their overall confidence in their answers and were optionally allowed to flag any uncertain responses via free-text input box. This structured flow encouraged serious participation while enabling us to monitor annotation quality and filter out unreliable data.
A.3.4 CROWDSOURCING QUALITY CONTROL AND CANDIDATE STRATEGY GENERATION
Ensuring the reliability of collected annotations requires two-stage quality control design.
Stage I: Crowdsourcing Quality Control. During the crowdsourcing phase, we embedded vali- dation checks into the annotation interface to identify inattentive or careless responses. Specifically, a small number of chart-pair questions were designed where the superior or inferior chart was visu- ally and functionally obvious. For instance, one pair compared a clean and readable pie chart against an overly cluttered line chart. Annotators failing such checks were highly likely to be engaging in random or inattentive behavior. These responses were flagged and either discarded or subjected to further scrutiny, thereby improving the reliability of the collected scores. Figure 13 illustrates examples of these validation questions.
Stage 2: Candidate Strategy Generation for Expert Review, In the expert adjudication stage, we further designed systematic conflict identification and resolution mechanism based on represen- tative studies in crowdsourcing quality control, statistical outlier detection, and multi-dimensional evaluation theory (Gadiraju et al. 2015) Rousseeuw & Leroy] 2003; Brennan 1992). This mecha- nism provides algorithmic candidate strategies serve reference signals for expert review, with- out replacing expert judgment.
High- Bighh-Diaggremmett Sample Identification. The system first calculates the standard deviation of each sample across all three annotators. Samples with standard deviation 1.0 are
initial scores for automatically identified as "high-disagreement samples" requiring further algorithmic analysis and expert attention.
Algorithmic Candidate Strategy Generation. For high-disagreement samples, the system gen- erates three types of candidate resolution strategies:
Outlier Removal Strategy: When two annotators' scores are close (absolute difference
< 2.0) but the third annotator's 's score differs significantly from both (absolute difference > 1.5 from each), the system suggests removing the anomalous score and averaging the remaining two scores. This strategy addresses cases where one annotator may have misun- derstood the task or made systematic errors.
28

---

Published as a conference paper at ICLR 2026
Malicious Scoring Filter Strategy: The system identifies and flags abnormal rating be- haviors where annotators assign identical scores across all six evaluation dimensions. Such patterns statistically unlikely for genuine evaluation and may indicate inattentive or gaming behavior. Flagged annotations undergo additional scrutiny or removal.
Sub-dimension Bias Correction Strategy: To address potential systematic biases in spe- cific evaluation dimensions, the system independently applies dual-threshold mechanism (threshold = 2.0) to each of the six evaluation dimensions. When an annotator's score in any dimension deviates by more than 2.0 points from the other two annotators' average, the system flags this as a potential dimensional bias and suggests score normalization expert review.
Strategy Integration and Ranking. All candidate strategies are processed through a score inte- gration and ranking module that evaluates their statistical validity and consistency with the overall dataset distribution. The ranked strategies are then presented to the expert team as structured rec- ommendations, along with confidence scores and rationale explanations.
The four complementary strategies mentioned include: (I) the standard evaluation process, which provides baseline scores; (2) nub-dimension major deviation detection, which highlights dimen- sional inconsistencies; (3) malicious or abnormal score filtering, which identifies problematic re- sponses; and (4) major deviation detection, which flags overall inconsistencies.
Together, these two complementary mechanisms -aaiiiittis-aiiiiiiiinnnnnno checks during crowdsourcing and can- didate strategies during expert adjudication-form multi-layered quality control pipeline, ensuring that the final dataset reflects trustworthy and rigorously validated quality scores.
A.3.5 EXPERT INTERFACE AND ANNOTATION
To streamline post-crowdsoureing adjudication, we developed an expert review interface that aggre- gates all 3,090 tasks and presents them in a prioritized queue. Samples are ranked by their score divergence (a, the standard deviation across candidate scores), from high to low, enabling experts to resolve highly contentious cases first.
For each task, the interface displays chart preview, metadata (type, subtype, and modification status), and the task description, which provide essential context for assigning a fair and informed score. The interface also presents per-dimension evaluation scores together with the outputs of the four candidate strategies namely the standard evaluation process, sub-dimension major deviation detection, malicious abnormal score filtering, and major deviation detection. These auxiliary sig- nals do not override expert judgment; rather, they support experts in detecting anomalies, validating consistency, and ultimately determining the most reasonable final score. screenshot of the expert review interface is shown in Figure 14
A.4 ANNOTATION QUALITY ANALYSIS
To validate the reliability and consistency of our annotation process, conducted comprehensive quality analysis across multiple dimensions the collected annotations.
A.4.1 OVERALL ANNOTATION QUALITY ASSESSMENT
We used the average rating from all crowd annotators as the base- line and calculated the MAE between each annotator and this baseline to measure consistency and noise levels among annotators. Results show that across the three visualization types, the internal MAE was lowest for dashboards (0.6441), followed by single views (0.6842), and highest for multi- views (0.6963). This indicates that under our task and experimental settings, annotators provided more concentrated ratings with higher consistency for dashboards, while showing relatively larger disagreements for multi-views.
Crowd-Crowd MAE Analysis:
Crowd-Expert MAE Analysis: We used expert ratings as the gold standard and calculated the MAE between each crowd annotator and the expert to assess the deviation between crowd annotations and expert standards. Results showed consistent pattern: the crowd-expert MAE was lowest for dashboards (0.5451), slightly higher for single views (0.5534), and highest for multi-views (0.5854),
29

---

Published as a conference paper at ICLR 2026
meaning crowd annotations were closest to expert ratings for dashboards and showed relatively larger deviations for views.
Notably, the crowd-cxpert MAE was lower than the corresponding crowd-crowd MAE for all three visualization types. This indicates that the deviation between each annotator and the expert standard was actually smaller than the deviations among annotators themselves, suggesting that expert ratings have good representativeness and can effectively balance different annotator perspectives. For the 1-5 rating scale used in this study, our observed MAE values (crowd-crowd: 0.64-0.70, crowd- expert: 0.54-0.59) are comparable to acceptable levels reported in prior crowdsourced visualization studies (Lan & Liu 2024) and subjective evaluation task studies (Li et al. (025c), indicating that annotation quality falls within a reasonable range.
A.4.2 ANNOTATION QUALITY ACROSS ANNOTATOR BACKGROUNDS
Crowdsourcing Platform and Geographic Distribution: Our crowdsourcing experiment was conducted through CloudResearch, a leading online participant recruitment platform. Since the CloudResearch platform primarily targets users in the United States and our experiment used En- glish as the interface language, the majority of participants were from the U.S. In total, 603 partici- pants were recruited for the annotation tasks. Specifically, the U.S. contributed the most participants with 535 individuals (88. (88.7%), followed by Canada (30 participants, 5.0%). New Zealand (24 par- ticipants, 4.0%), the UK (11 participants. 1.8%). Ireland (2 participants. 0.3%). and Australia (1 participant, 0.2%) We acknowledge the limited geographic diversity and discuss this as limitation in the paper. Building on the overall MAE analysis, we further grouped annotators by background information to examine quality and consistency across different backgrounds.
Professional Background Analysis: We divided annotators into professionals and non- professionals based on whether they work in data visualization or data analysis-related fields, with professionals accounting for 41 % and oo-professionals 59%. Results showed that the professional group had an average within- group MAE of 0.4245, while the non -ro-esffessional group had 0.4881, a difference of approximately 0.0637. This indicates that under our task and experimental settings, the professional group provided more concentrated ratings with slightly higher within-group consis- tency than the non-professional group.
Educational Background Analysis: We divided annotators into undergraduate, master's, 's, and doc- toral groups, accounting for 70%, 23%, and 7% respectively. The undergraduate group had an average within-group MAE of 0.4825, the master's group 0.4762, and the doctoral group 0.3944. Overall, the graduate groups (especially doctoral) showed better within-group consistency than the undergraduate group, with more concentrated ratings. However, due the relatively small sample size of the doctoral group, we take a cautious and conservative approach to interpreting this result in the paper.
Combining both professional background and education level dimensions, we can draw three main conclusions. First. MAE levels do differ somewhat across different background groups, with anno- tators having relevant professional backgrounds and higher education showing certain advantages in within-group consistency and alignment with expert standards. Second, the magnitude of these differences is relatively limited overall, and after appropriate quality control measures and result aggregation, the overall annotation results from professfess or lower-education annotators still maintain good reliability. Finally, and most critically, observed consistent error patterns across all background subgroups that matched the overall analysis (e.g., relative performance across different visualization types remained consistent). This indicates that main conclusions about crowd- sourced annotation quality and effects under different visualization conditions are relatively robust across different annotator backgrounds.
A.4.3 EXPERT QUALITY CONTROL IMPACT ANALYSIS
Experts reviewed all 3,090 samples individually. During the review process, approximately 11% of samples underwent adjustments at the overall evaluation level, with 7% using the Malicious Scor- ing Filter Strategy and 4% using the Outlier Removal Strategy-ther ratings before recalculating the base scores. Additionally, 16% of samples used the Sub-dimension Bias Correction Strategy, removing outlier ratings for certain tub-dimensions with large disagree- ments, particularly those where the standard deviation exceeded 1. Nearly 73% of samples retained
strategies remove abnormal
30

---

Published as a conference paper at ICLR 2026
all original ratings and underwent further expert review based on these ratings. These results demon- strate that our quality control process is rigorous and reliable, while also indicating that the overall quality of crowd annotations is high, with the vast majority of samples not requiring quality control through removal of abnormal ratings.
A.4.4 DIMENSION- SPECIIIC QUALITY ANALYSIS
We further analyzed expert adjustment patterns across different dimensions and found that adjust- ment frequency is closely related to the objective/subjective characteristics of dimensions.
Overall Adjustment Patterns: Among the 511 samples requiring sub-dimension adjustments, the most frequently adjusted dimensions were semantic readability (19.20%), data fidelity (18.04%), and insight discovery (16.79%) which have strong objective characteristies. Semantic readability concerns whether text and legends clear and easy read, data fidelity focuses whether there misleading information or data distortion, and insight discovery examines whether visualizations effectively convey key information. Because these aspects have relatively clear evaluation standards, experts tend to apply stricter control over these dimensions to ensure annotation accuracy, resulting in higher adjustment rates. In contrast, dimensions with lower adjustment frequencies include de- sign style (1 (17.05%), visual composition (14.73%), and color harmony (14.20%), which involve more aesthetic judgments with strong subjective characteristies. Different annotators may provide differ- ent ratings based on personal aesthetic preferences, and these differences often reflect the diverse perspectives of real user populations. Experts made fewer adjustments to these dimensions, indicat- ing that quality control process ensures annotation accuracy while respecting and preserving certain degree of subjectivity and diversity, avoiding sver-standardization of aesthetic criteria.
Adjustment Characteristics Across Visualization Types: Further analysis by visualization type revealed different adjustment patterns:
Single Visualizations (159 samples. 31.1%): Data fidelity was adjusted most frequently (20.06%), followed by semantic readability (18.58%), while acsthetics-related dimensions such as color har- mony (14.75%), design style (15.34%), and insight discovery (15.04%) had relatively lower ad- justment rates. This indicates that single visualizations are most sensitive to data truthfulness and semantic interpretation.
Multiple Visualizations (153 samples, 29.9%): Semantic readability had the highest adjustment rate (20.46%), followed by design style (19.02%) and data fidelity (18.73%), while color harmony had the lowest adjustment rate (1 (11.24%). This suggests that multiple visualizations show the largest dis- agreements semantic communication and overall design, but relatively fewer color coordination issues.
Dashboard (199 samples, 38.9%): Adjustment rates across dimensions were relatively balanced, with semantic readability (18.66%), insight discovery (18.20%) and data fidelity (17.05%) slightly higher, while aesthetic dimensions such as design style (16.82%), color harmony (16.13%) and visual composition (14.29%) had relatively lower adjustment rates. This indicates that dashboards show certain disagreements across all dimensions but are overall more balanced.
This finding validates the rationality of our quality control strategy: applying strict control over objectively verifiable dimensions such as data fidelity, semantic readability, and insight discovery, while maintaining appropriate tolerance for subjective aesthetic dimensions such as color harmony and visual composition, thereby achieving balance between annotation consistency and perspective diversity. This balance is crucial for building visualization quality assessment dataset that reflects the diverse preferences of real users.
In summary, annotation process produced reliable and consistent evaluation data suitable for training and evaluating visualization quality assessment models. To facilitate reproducible re- search and further academic exploration, will publicly release all annotation data, quality control records, and related statistical analysis results to enable the research community to verify and extend our work.
31

---

Published as a conference paper at ICLR 2026
Visualization Quality Assessment (15 Charts)
Prertrutin krevers Auintted be Worters
x
Youre neat competinng the ros boters and deo "Ca-rar orc precitm Ooman ot subiniied reeots
Chart 14 of 15
IPPORTANN HEMrINDeiid
Heup iemmer dbaird wonning
Quality Contrel descaa
Sype desl
pruss sehve to reprossnt Ceneuree Coporais, and Home Offec catogenter reggocheey. The
prog mus insiatio camentt
o1 Data Fidelly
Category: Ficelly
Qusation: dembbaad beit
salis dito with epproe kute oves wales lubolo fox ber chan and progess Indiato Please provice 1-3 booee lasud sookig unkens
charce progees caizazee hace dign ricart issard lie happonp czals miscing nume
chart indeators reaty oocuntt, with maae
dole Hore
the bar chan and prograss ioliators are acca and standrdd with app up take carns and intand lbell.
componers
Five other sub sub-dimensions omitted...
Pevoe
Hed
IMPORTANT REMINOORS
Qualte Connect sytere devere reoi olding and under come the boneer
cannte moview Plenee wilk arale herk proaic cuaade tiigs kalle giaare
Question: Your Overall Confidence
Pleae
gl wnseess
gef aostuatann
Hot shoun ou all mus enull gard ue
Bighte suetionn fies amour alout reprode
Medonatddy cor fofont are vecooablee sur about arovers
Hight contii tarn ou sure arovens securntt.
Mery sue start tann zetaan about munommet.
Optonal: Quessiions any)
you ancatain aoou spaciiic ona slaae Dara belica spperated spacas Ecamppee ancerai sbout openticnss va
Figure 12: Crowdsourcing interface for expert annotation process.
32

---

Published as a conference paper at ICLR 2026
Validation
Figure 13: Examples of validation checks embedded in the crowdsourcing interface.
Crars
Papiane
mochont Uil, www.yrreroooneccooooooooo
sedgh
Desiue
eocution
Pori:
Some tasks are omitted...
affactvv
MaK
soree
2.55
Ovaral Coors
2
Crees Stoe
4
Strye
140
CHORES
525
Figure 14: Expert review interface.
33

---

Published as a conference paper at ICLR 2026
B CASE STUDIES
B. HIGH-SCORE CASE STUDIES: HUMAN-MODEL ALIGNMENT
We define High-Score visualizations those that achieve strong performance dimensions of fidelity, expressiveness, and aesthetics. Such visualizations demonstrate professional design principles and convey information in manner that is both accurate and visually engaging.
the three
As illustrated in Figure 15 we present a representative high-score case along with the correspond- ing human ratings and VISJUDGE output. This example demonstrates that, when the visualization adheres to established best practices, the model's evaluation is largely consistent with human judg- ment.
B.2 MEDIUM-SCORE CASE STUDIES: HUMAN-MODEL ALIGNMENT
We define Medium-Score visualizations as those that perform adequately across fidelity, expres- siveness, and aesthetics, fall short excellence in least one of these dimensions. Such visu- alizations generally succeed in conveying information correctly and remain interpretable, yet they may exhibit shortcomings in specific aspects, most often in visual aesthetics or design refinement.
present representative medium- case, where the visualization fulfills its communicative purpose but does not achieve high-quality standards in every dimension.
As shown in Figure T6
B.3 LOW-SCORE CASE STUDIES: HUMAN-MODEL ALIGNMENT
We define Low-Score visualizations those that exhibit clear deficiencies across the three dimensions fidelity, expressiveness, and aesthetics. Such visualizations often distort or obscure the underlying data, employ ineffective misleading encodings, or suffer from poor design choices that hinder interpretability.
of
As illustrated in Figure 17 we present representative low-score case, where both human ratings and VISJUDGE outputs highlight significant problems that severely compromise the effectiveness of the visualization.
B.4 DIMENSION- SPECIFIC CASE STUDIES: VALIDATING EVALUATION CRITERIA
TIMENSION-PPECIFIC
To facilitate a clearer understanding of the three evaluation dimensions-fidedt expressiveness, and aesthetics-we provide representative low-score case studies for each dimension. Specifically, Figure 18 illustraasss case with low fidelity, Figure 19 presents case with low expressiveness, and Figure 20 shows a case with low aesthetics. These targeted examples highlight how deficiencies in individual dimensions manifest in practice and demonstrate that VISJUDGE'S evaluations align with human ratings along the intended criteria.
B.5 MODEL ERROR ANALYSIS CASES
While evaluating visualizations along the three dimensions of fidelity, expressiveness, and aesthetics, we observe that certain base models fail to correctly identify the deficiencies of low-quality charts and consequently assign them undeservedly high scores. To illustrate these issues, we present two complementary types of error analysis:
Overview Analysis: Figure 21 provides a high-level overview of typical failure patterns across different models, summarizing where and how the evaluations deviate from expert judgment.
Detailed Case Studies: Figures 22
Figure 23
Figure 24 and Figure 25 present represen-
tative detailed cases, each showing the full output of a model and highlighting the specific reasons for misalignment with human evaluations.
Together, these analyses reveal both the systematic error modes of existing models and the necessity of explicitly evaluating visualizations along the dimensions fidelity, expressiveness, and aesthet- ics.
34

---

Published as a conference paper at ICLR 2026
B.6 ERROR ANALYSIS OF MULTIPLE VISUALIZATIONS AND DASHBOARDS
Our evaluation identifies a distinct negative correlation between visualization complexity and model performance. As visualization complexity increases, from single charts multi-view compositions and dashboards, the efficacy of current multimodal models notably declines. To conduct an in-depth analysis of model error patterns in complex visualizations, we filtered from the test set those samples where at least one model's 's predicted score deviated substantially from the human annotation (abso- lute deviation > 1.0), yielding a subset of 139 multi-visualization sets and 94 dashboard cases. In this subset, quality scores on visual composition and color harmony drop sharply. In dashboards, this degradation extends beyond aesthetic metrics, significantly impacting insight discovery and seman- tic readability. Based on our analysis, we attribute this degradation to the lack of cross-view visual understanding. The models tend to process each view as an isolated visualization and rely heavily on local visual cues (e.g., grid alignment or color consistency) while failing to construct the cross-view semantic connections and analytical pathways characteristic of human expert interpretation.
Visual Composition and Color Harmony Evaluation. Regarding aesthetic dimensions, the mod- els exhibit a problematic dual bias. They tend to overestimate layout regularity, assigning high scores as long as views are arranged in standard grid with consistent styling, while disregarding redundancy or the absence of meaningful cross-view relationships (see Figure left). For instance, in the mulii-visualization subset, this positive bias is widespread: Intern Claude-3.5-Sonnet, and Qwen2.5-VE VL rated visual composition higher than human experts in roughly 90% of the cases (89.2%-922 1)), inflating scores by over 1.1 points on average. This tendency persists in dashboards, where Claude-3.5-Somee continued to overestimate Color Harmony in 75.5% of cases, inflating the metric by 1.09 points. Simultaneously, they are overly striet about practical design trade-offs. The models tend to enforce strict encoding rules, penalizing visualizations that make necessary compro- mises, such as minor inconsistencies in color encoding or redundant legends (see Figure 7 right). Cemini-2.5-Pro for example, strictly penalized color harmony, underestimating scores in 44 multi- visualization cases and 29 dashboard cases, with negative deviations nearing point. This indicates that models perform surface-level pattern matching rather than comprehending the functional intent behind design decisions.
Data Fidelity Evaluation. In terms of data fidelity, we find that the models often confuse struc- tural completeness with content validity. Charts containing only axes and frames, but no actual data marks, often receive high fidelity scores merely because they adhere to visualization design stan- dards (Figure 25 This hallucination is pervasive across both open-source and proprictary models. In the multi-visualization subset, Qwen2.5-VL. and Intern overestimated data fidelity in 95.0% and 96.4% of cases. respectively. Proprietary models like GPT-4o and Claude-3.5 3.0ee,,,---nnett followed a similar pattern, inflating fidelity scores in 87.8% of cases with an average increase exceeding 1.50 points. In contrast, simplified schematic views, such as those omitting detailed axis labels to em- phasize trend comparison, are frequently deemed unreliable by GPT-5 and Gemini-2.5- due to their deviation from standard forms. This suggests current models prioritize structural conformity to standard charts over the accurate conveyance of underlying data.
Dashboard Insight Discovery and Semantic Readability Evaluation. In dashboards, we ob- serve that models tend to equate interface completeness with analytical depth. Dashboards fea- turing diverse chart types within a unified visual style often receive high insight discovery and semantic readability scores, even when lacking logical coherence or narrative structure (c.g.. Fig- ure 19 top). Qwen2.5-VL. exheiiie the most extreme positive bias in semantic readability, inflating scores in 90.4% of the dashboard cases with an average increase of 1 1.28. Conversely, complex and information-rich dashboards are frequently unfairly penalized by GPT-5 and Gemini-2.5-PP which misinterpret their high density as "visual clutter" or "data distortion".. GPT-5 underestimated seman- tie readability in 33 cases (average decrease of 0.78 points), while Gemini-2.5-Pra showed an even stronger negative bias. lowering scores in 67 cases by an average of 1.15 points. This underscores a critical limitation: current models lack a robust understanding of narrative logic within visualization interfaces and struggle to distinguish between visually polished artifacts and functionally effective analytical tools.
35

---

Published as a conference paper at ICLR 2026
Sigh-Score Case
Single Visualization Chart
Fidelity
Books Books
Dota Fidellty: 4.50
The bor chart accuratety represents the of U.S. adults who resa printed book versus e-bcoks ocross the years 2011, 2015, and 2019. The chort uses appropriate boselines and consistent encodings.
Expressiveness
Semantic Readability: 4.00
clear
percentoges, allewing users reaning proferences the years.
meoning of
information
Discovery. 4.00
The chort effectively the This signffccnt with prectical volue.
higher proference printed books compored e-books
Aesthetics
Design Style: 3.00
Humon baseline
The design is straightfoword using a bosic bar chart format with no unique elements, compromsing some onginolity.
Dota Fidellyy
Semontic Reasability
4.00
4.50
s. Visuel Composition: 4.00
The leyout is well-ezerueed with aporopriately placed elements and bolenced proportions enhancing clarity and case of understonding
Insight Discovery
Visaal Composiion
Design Style
4.00
4:00
4.00
The choice of two shades of biue to represent printed ond e-books is well courdinated, with clear dustnction highlightin the subject motter.
Color Harmony. 4.00
Color Hormony
4.00
Overall Score
4.08
Visjudge Overall Score: 3.92
High-Score Case
Multiple Visualization Charts
Fidelity
Dota Fidellty: 4,00
The Wefore and Aftr bor charts have acturate lobels and volues, clearty represeming the data on the number of people lxing with major illeesses by age group. The doto appeors consistent and thre are no aignfficont distortions.
Melkoa bost endert amogle my red panperieeedd
Expressiveness
Semantic Readability: 3.33
delere
The color encudings uge groups (ourple for 20-69 years Mus for 70 years oider) are cleat and the percentage change indicutors ayter view are informative, out might small font size.
some contusion
Insight Discovery. 3.33
Wefore' After
allow identfication of trends data,
of people Bving mator illeesses across age groups 2010 2010, percontage change indicators the view furmer highlight significant growtn.
Aesthetics
Style: 3.00
The dasign style is informative with 'Before' and Wher
but it does not stond out particulrrty innovativ. (he balance benieen the two views could be improved to ensure the everall design style is more calesive.
Human baseline
4.00
Visuel Composition: 3.67
Semontic Scadabiltt
3.50
The visual compositon reasonable, with well proportioned 'äefore' and aitter sections. However the visual flow could be improved by enhancing the sporing and digemeet between the vurious elements.
Dircovery
4.00
Design Style
3,50
Viwei Compostion
4,50
Color Harmany 4.00
Color Hormony
3,50
The purple and color encodings for age groups are aperopriate and easy diffenentiate. color harmony consistent across the 'Before' and lihter sections, ensuring each age group visuoily distinct.
Overall Score
3.83
Visjudge Overall Score:
High-Score Case
Dashéoard
Fidelity
Data Fidelity: 4.00
The doshboard accurately represents dusiness data os provides and diraaleed with appropricts calor highlights indicating pooniv or neganive changes. appropriately, and the gouge indicates percentage volues with reasonable eccurary.
Expressiveness
Readability:
6Pts bor graphs, understandable within business context. The aigtt incress//decreeee indicators further enhonce readabilit.
doshboard
their meanings
Insight Divcovery: 4.00
The doshboard effecturly highights key imiihts by prominently displaying XP% and the trends in the ber chart. The audience and engagement rate dearly identfedd altwinng business decision.
Aesthetics
Design Style: 4.00
The dark theme with green hightights creates a modere and professional lock, However, the repeated use of green in differett elements could be considered slighlly overwheiming.
Humen baseline
Visuel Composition: 4.00
Data Fidelity
4,67
The spatia arrangement of the components well organized, with appropriate proportions and spacing, The key performance indicators are easily visble, and the nodfications are disinct and prominent
Semaratc Readability
4.67
insight Olecovery
4.53
Design style
4.57
Color Harmony 4.00
Visaal Compasiton
4.33
The color scheme primanly dark green creates a conesive look and offective in highlighntng importent elements, Howerec the use black for the entire background might be stark for prolonged wiewing.
Color Hormony
4.67
Overall Score
4.56
visjudge Overall Score:
Figure 15: High score cases.
36

---

Published as a conference paper at ICLR 2026
Medium-Scooe Score Case
Single Visualization Chart
Hestrogram the Overall Rariing of the Exrm
Fidelity
Fidellyy:
accurately represent the frequency responses with
Expressiveness
Semantic Readability:
Oor colors lobels clet
cotegories and their frequency.
Insight Discovery: 3.67
The chort revelts shaves distribution with 'ood being the most frequent response, but less insight into the total somple sze.
Reguoe
Aesthetics
BIRIKESS
with unique patentilly engoging Design Style:
design
Visual
3.33
baseline
The ordenly with appropristely aligned bors lobet. there's
Data Fidellyy
4.00
Color Harmony: 4.00
Semontic Readability
4.00
Singie blue color is not hormonious patentialy ovenwhelming the viewer out con improved with caditional visual hierarchy.
Insight Discovery
3,50
Design Style
2.00
Visual Compostion
4.00
Visjudge Overall Score: 3.50
Hormony
3.00
Overall Score
3.42
Medium-Score Case
Multiple Visualization Charts
Fidelity
The sires clesely tabeled percentages, siiht vartetions, maintaining level
Expressiveness
Readability:
provide percentages, o encoding
Insight Discovery. 3.67
While
distibution, always the lock of comparative line legend, expecially nuclear
understond these
other
Aesthetics
Design Style: 2.33
Human baseline
The design simple and fuectional but locks vonovstive elementy the pe straighrfowwrr without much creatrutty
Visuel Composition: 4.00
4.00
The charts are well-ttgred and spared evenly, with clear distinction between the two years, which oids in comporstive onalysi.
Somontic Scanabilty
Dircovery
4.00
Color Harmony: 3.00
Design Style
2.00
The colors are distnnt, contributing to increased readobiliy, though the palette seems a bit monotone and could be improved for more engaging look.
Viwei Composiion
4.00
Color Rormony
3.33
Overall Score
3.61
Visjudge Overall Score:
Medium-Score Case
Dashéoard
Fidelity
Data Fidelity: 3.67
The bor and line chorts the dashboard display clear data segments, with distinct colors used consistently chroughout Hovever there is no indication of units for the oxes, which could obscure the of doto.
Expressiveness
Readability:
dashboard display clear segments, of units oses,
used
consistently throughout, the ratto.
Discovery:
The chorts provide for buying and limited chorted volues.
trends
clear increased older mothers, the data depth due
Aesthetics
Design Style: 3.33
The dashboard foltnws a simple FieThirryighht design style with appropriate graphing elemants.
Humen baseline
Howerrr the design style could be more innovative create uniquety strong impression.
Visual Composition: 3.33
Data Fidelity
3.33
The loyout orgonized with balonced and afigner arrungement of doshhourds. There is good use of space to separate vach chart, but some more con be done with space handing information densiy.
Semantic Readability
4.00
insight Oiscovery
4.00
Color Harmony- 4.00
Visoal Compasition
Design style
3.67
3.67
The color polette's is adequate for distngushhng different doto segments, with good controst betveen but resl green, ond yellow. Howerer the color combinations for line charts lock aesthetic depiit
Color Hormony
4.00
Overall Score
3.78
Visjudge Overall Score: 3.50
Figure 16: Medium score cases.
37

---

Published as a conference paper at ICLR 2026
Low-Score Case
Single Visualization Chart
Fidelity
Fidellty: ony
chart
moking impussible to iffrmmtion.
accurotely intepreel
Expressiveness
Semantic Readobility: 1.67
The chart uses single color gradient without any dotingushable morkert or legends, moking hard to understond the different data series trends
Insight Discovery. 1.67
The fall falls to reveal any significott insights or trends due the lock of axs labels unclear data seress representation, and failure to highlight any specthic pottern over the time period
Aesthatics
deiign ordinary
monochramatic purple originalty.
any distnngsshing
baseline
fectures, overall look does offer
Visuel Compesition: 2.33
Dota Fideliyy
1.00
The spoliai layout the stocked areas is somewhot cluttered, there overiapuinng loyers and no clsar duzinction between different series, although it lnn't the most chootic leyout.
Semontic Readability
1,53
Insight Discovery
1,33
Color Harmony: 2.67
Design Style
2,00
The color scheme of multille shades of purple does not provide much controst, making it hard to differendate betveen the different layers and the exect values each represents.
Visual Compostion
2.00
Hormony
2.33
Overall Score
1.67
Visjudge Overall Score: 1.89
Low-Score Case
Multiple Visualizatio Charts
Fidelity
Data Fidelity: 1.67
The stocked hittgram and box placs are difficuit to interpret due the small size and lock of detovled jobeling, which makes hard to understond the evoct distnbution ocross lattended categories.
Expressiveness
Semantic Readability:
histogrom disticclly different,
viewer
distiegushh between cotegories
readubillt.
Insight Discovery: hattorrm and box minimal the meaningful
The synergy
docem
insights from the visualiztiion due the cluttered and poorly lobeled
nature
Aesthetics
Design Style: 1.33
Human baseline
The design 20/t does not show any innovation, using common chart types in bask layout without any unique elements.
Visuel Composition: 2,00
1.00
The spatla layout is not well orgonized, The Nistgrrmm and bax plats are separated and have on awlnward arrangement, making the overall composthon look choone and unbalanced.
Somontic Scanabilty
2.00
Dircovery
1.00
Color Harmöny: 3.00
Design Style
1.00
The color choices are somewbot apgraprinte for the two cotegories, but the poor legibilly and ocross vieus.
Viwei Composiion
1.67
suing make harmony difficult
Color Hormony
1,50
Overall Score
1.36
Visjudge Overall Score: 1.83
Low-Score Case
Dashéoará
Fidelity
Fidelity:
reatonably well, may piot appeors represent glabal
comparson feature but not the model the dota point itelf.
occurately
Expressiveness
Semancic Readability:
meanings
scotter
adsitional or legends.
droprisirrs
toggle cleonyy exploined, confusing.
Insight Divcovery.
Insights clearly hightighted. Whth limitdd points
detalled legend, difficuit derive meoningful trend'
Aesthetics
Design Style:
The dashboard
dropdowms and radio buttons without any unique Innovative overoll design locks vissel fintt.
Humen baseline
Visuaf Composition: 2.33
Data Fidelity
1.33
The loyout samewhat dtumered The scomtr and bar plats are indicated beside each other but placement could be improved for better integration. The spacing between the elements could be optimized.
Semantic Readability
1.67
insight Oiscovery
Visoal Composition
Color Hormony
Design style
1.67
1.67
1,67
1,57
terms of their frequency and contrust. doesn't appcor
The color hormany somewhot lonking as the use of blue and rnd could be more balonced in appcorperfcctly perfectly harmanious.
Color Hormory: 3.00
Overall Score
1.51
Visjudge Overall Score: 2.28
Figure 17: Low score cases.
38

---

Published as a conference paper at ICLR 2026
Low Fidelity Example
Single Visualization Chart
Fidelity
Dete Fidelity: 1.00
Without oves, lobels, and tites, impossible to conffrm the true data values and understond the undertying data.
Expressiveness
Semontic Readability: The histggrom and curve symbots undeecpherable without fobels, making challenging understond the chert.
Insight Discovery. 2.67 The lnck uses and lobels prevents users from idemlfying specific trends onomalies.
Aesthetics
Human baseline
Design Style: 2.67
The gradient color scheme locks Innovation and does not provide unique design elemont that enhances visuci appeal.
Octo Fidelit
1.00
Semondic Readobalty
1.50
Wseel Compositio:: 2.67
insight Discovery
1.50
The barz and curve are Improperly aligned, offecting readonilty and making hard to interpret.
Design Stylt
2.00
Color Harmony: 3.00
Viuel Composition
3.00
background shades slighlly overshadow the gradient effect, offecting overall clarity.
Color Hormany
3.50
Overall Score
2.08
Visjudge Overalf Score: 2.33
Low Fidelity Example
Single Visunfization Chart
Fidelity
Date Fidelity:
The chart hos non-rera beselines for her segments, which exaggerate differences and make exect volues hard read accuratol.
Expressiveness
Semontic Readobility: 2.67
The meonings of the colors are baskr, but without claar labels legends, they become less disingushablr..
Insight Discovery: 2.33
chart does not have sufficient detat's or clariyy for users to discover significon insights about sotes trens.
Aesthetics
Human beseline
The design ztyee is quite ondhnory, not innovative and does not stond eut
Design Style:
Visel Composition: 2.67
Ridelly
The bars are reasonably speced, but the Irregulor placoment of the labels for the years offects readocalit,
Readobitty
imsight Diccvery
1.33
Color Harmony:
Design Slyle
2.00
mhugh distngushhbll, have hormonious combination some might percerved
Visuol Composttin
1.53
Cofor Hormmny
1.50
Overall Score
1.42
Visjudge Overalf Score: 2.50
Figure Low fidelity cases.
39

---

Published as a conference paper at ICLR 2026
Exprassion Example
Dushlwwrd
Dato Aldelty: 2.00
The mop styled and may not occuvatey repebern peopnaphite dito, and ths inalcooars ars dfficut intepper undenwing oxcwerety.
dearly Vabelot
Somorcic Readrolliy: 2.60
The deshoserd year very compler and clttered desien with mony stements, making c atpest for sears vamly undiriiond meoning maps gruge, indiatoor.
The dechócord does noc promitser indiotorr to hyhigtt ke ttsghts and the everal design moves it dijectt to quitly dercower sigepicatt busteesr trends.
Aesthetics
Derige Style: 4,00
The desge julunasi tcnovative, with profecsional cesthetic that conves o sense
Humomn besellne
visual Composition: 3.00
Ribeity
The sponel kyost is genersty ressonatie win es atceptatie omongement our mayy afficst foou sage dipact of
Semontic Readebiity
Inight Durovery
2,57
Color Hermeny. The caler schecce, prnoelly tske doep piyples and firy aranges, relutivety oppraprions for tont unly strong.
Visel
profesiena
Color Normory
2.07
Overatl Score
Visuude Overall Score: 2,78
Expression Example
Stogle Vasalization Chart
Fidelity
The lack hadek lagendz moles impossible to detormise aie exect dots proportions Its r2lotenshhpp is the underlying oato es undetermined if chese are not prowieed
Expressiveness
Readebilty: The lock label moses it inpousiblee to dacenn ohe mosnings of the gray shootss make sense grche mort wittout adamrona infermation Insight
Dos die lac cfiees,, u difccut identfy mesningfsl froms the chost, ihe olort choock whtiuus ubet to provide contort
Aesthetics
Human beseline
Denige Style: 2.00
gray
fretures making andiary
Filetty
lacking intovation
Semonne Readebilty
Visunl Compostrion: 3.00
inight Ducowery
1.00
Culor Hormany improperly algned rffectng readsaiity making
Visal
The dark beckground shedls alghtly werslorio the gradione offect, affecting overt cirdly
Color Normort
1.30
Overall Score
Visudge Overall Score: 1.89
Figure 19: Low expressiveness cases.
40

---

Published as a conference paper at ICLR 2026
Low Aesthetics Example
Multiple Vounlization Chorls
Fidelity
Deno Aidvity: 1.67
and of loeel, oves, andiear comospandiene
Expressivenesy
Sementic Seadrallty: 1.77
exp laliing the Aestnups calors and dendoggross.
Insight Discover: 1.47
moger
proper czbelop
Austimapi dkniisgromm, and Isslagrass making dfficsl deew costingia muigit..
Aesthetics
(hs design siyle cruinary there is innovation in me comminatio of dlements. lhe
Hussom
heconap, dtndrograms, and Mistgron are bust and fuc conerive novel soyt.
Dasu Adeiiy
2,50
aeth choetss proportions Aucemup ainarooram,, and hastgrans, loading to cortiing ovrnngemens.
(he compostiio
spacing ontween
Semore: Readebiity
1.3
Inuget
The Herweny: someahet
style
1,30
gradient
Wauel Compositoon
1.00
heeimup and sorigrom, but owest coler marching us not nel--corrdnntted
Overail
Visfadge
Low Aesthetics Example
Mullipie
Gharts
Fidefity
(he scoter old lss cieer lseek and the geographeaa and not descrioed the isl making it elficol to vanfy the dats fuokly occanately.
Expressiveness
Semontic Readeaility 2.50
elenees clearty labwied and the spasiic consingg beiind acosueed posnts, color
desoriedt
Insight Dircovary: 2.00
dffical
hnighigul informttion obous the relonirrhhps between the dato repeesented in me oifferene whvks
Asscherics
Desige Sigll: 2,67
Humon besellne
comtiinetten of
but overdll map, glote heatmans
innovotte and offers ungted.
Semore: Readebiity
Denu Rdsity
2,00
2,00
Visue' compesition: 2.67
otrangsmers wiws appears somewtex jurolled and uebulanced propartions
hermonious
Inught Ducowery
Color Horeenyy: 2.33
Viuci Composition
2.00
2.23
mezier teaimopu and the glóoe somoshot untyy but the color lighting
Overall
Vistadge Score:
Figure 20: Low aesthetics cases.
Human Rating: 178
2 Human Ratting: 1.95
information clearly...
obvlous useful insights
Owen2 5--ssr7--hstuut
Scare:417 X
Qwan2 5-LL-7B-tnntttee
Score: 3.39
The cotar renging from lights yellow dark red and liaals provite necensery context...
The color guadiant and lin plot reprosent a time serinc of data. charl sommatma clear using color..
Cleade-d Sonnet
Score: hestmap allrow and column lnbels are present and
Cloude 4-Sornet
Score: 3.44 X
The stales sponer condetentt follows logical price
movements from around 89 250 000.
Score:
3.44
Clusteling and heatmip clear patterns, users
price path properly aligned heatmup
VisJudge
Score:
Visludge
meanings colora are not clear without color acale, making clear insights patterns dua the misxing scale..
visusi encoding accurately grodlent and thne plat appear chaotic, makingg it Impossible to deelpher the relationship between the data values and the color intensites...
The color
Figure 21: Overview of error cases.
41

---

Published as a conference paper at ICLR 2026
MODEL ERROR ANALYSIS
Qwen22.5 VL- Instruct
Fidelity
Fidellty:
chart accuratety reprecents date using color grodients and rectongle sizes reflect the underlying volues for different sectors. legend provides cleor information about the color scoles, enhoncing foikfuines..
Expression
Reedability: veriatons makes visualization reedable Interpretable. iegend elso clear alding users understanding the dota representatios.
usage
Insight Discovery:
treemop allows for some bosic insights, such as comporing the relotive sizes and colors, but it' less effective in revenling complex correlations or refining deeper dota potterns.
Aesthetics
4. Design Style: 3.67
The chort uses consictent color scheme and layout, which contributes modern and idettilablee design. However, does not odopt highl Innovative approach.
Human baseline
S. Visual Composition: 4.00 &
Data Fideity
2,00
The treemops ore satsfactorlyy arranged with reasonable proportions and spacing, ensuring that the averail layout harmoninus and junctional.
Semontic Reariablity
1.67
Insight Discovery
1.67
Color Harmony: 4,00
Design Style
1.67
The red and blue creates controst visuplly uppealing and helps distinguishing different sectors. The color choices consistent and enhonce readobility.
Visual Compostiion
1.67
Color Hormony
1.33
Overall Score
1.67
Overall Score: 3.67
MODEL ERROR ANALYSIS CASES
GPT-5
Fidelity
Data Fidellty:
theer for magnitudes, step with dashed begins (e.g. foltnfully ospect-ratic distorton misleading
Expression
Semantic Readability: X
Positive data of legend ornatation exploining
volues
stepping distinguished and and reinforced by signed generic siightly
reduce
Insight Discovery:
key insights easy provides largest increase, decrease, informative, not
cumulative peok desper relationships beyand strelghfforoord observations.
followed
Aesthetics
4. Design Style: 2.00
baseline
The design follows conventioonl waterfall template with standard colored bars and lobels.
There are no natable innovative elements or unique stylstic touches beyond basic conventions.
Dato Fideiity
Semontiz Readiobiltt
3.00
1.00
Bars are eventy spaced and property nligned with cleor stepping and adequate margins; tabe's are legible und not cluttered. The layout orderly and supports reading, though not exceptionally refired.
s. Visuet Composition: 4.00
Insight Discovery
1.50
Color Hormany
Design Style
Visual Compostion
3,50
2.00
3,50
Teal and red provide clear, balanced controst for positive and negotive contribtions without rausing vitual fatigue. The palett harmanious ageinst the neútral background, though not exceptional nuonced.
6. Color Harmony: 4.00
Overall Score
2.42
Overall Score: 3.83
Figure 22: Qwen2.5-VL-7B II-nrutuct and GPT-5 error cases.
42

---

Published as a conference paper at ICLR 2026
MODEL ERROR ANALYSIS CASES
Gemini-2.0- Flashh
Fidelity
1. Data Fidelity. 4,00
The map appears to have accurate representation with appropriate scaling and no Walble distortions, The map boundarles align well with the scotter points.
Expression
Semantic Readability: 3,00 X
meaning basically clear in representing population, dota points. However, lack of regend makes precise understanding difficult
Insight Discovery: 0.00 C
conceniration of people are clearly identifibble the casterp and some major alles are discernbbee This map effectivly shows population density relative to geography.
Obvious population insights
Aesthetics
4, Design Style: 3.00
The design some innovctive eiements by using minimalist approach, but overall pretty conventional, presenting a basic scotter plot.
5. Visuel Composition: 4.00 X3
Semontic Readabillty
Dato Fidelity
2,00
1,50
The spocal layout is quite reasonoble the dete points are reflective of octual geography and populated areas, Spocing and balonce are generally good.
Insight Discovery
Design Style
1,00
1.67
The Color choice relatvely coordinated, with shade of biue that stands out enough to nighlight data points without cousing confusion.
6. Color Harmony: 4.00
Visual Composition
1.50
Color riarmony
3,00
Score
3,67
MODEL ERROR ANALYSIS CASES
Glaude 3.5 Sonnet
Fidelity
Date Fidelity:
redlr visuoliation maintains accurate spotial relationships with clearly posttioned dots representing bocked objects. The concentric circles and radial lines provide consistent roordingle system for precise postion tracking without any opparent distartion.
Expression
2. Semontic Readabllity: 5.00 X
The visuaizztioo uses universlly recognired rodar conventions white dots for torgets, concentric dirles for distonce, and Sweeping sector for sconning elements are ummediately interpretbble without need for explonatio.
3. Insight Discovery: 3.67 Ø
chart effectiely reveals the spatial dotribution and relative posiions of tracked objects. beom helcs highlight temporal changes, though with stotic image full dynamic insigtts
Aesthetics
Design Style: 4.00
design shows strong innovation with clean, modern interpretation of rodor oesthetics. sconning beom effect and grid system are well-exeruted, though still following established rodor visolization conventions.
Visual Composition: 5.0
Dato Pideitty
2.00
The spottal orrangement excellent with perfectly concentric circles, evenly spoced radial lines, and well balonced overali compasttion The sconning sector and tracked objects are positioned with precise geometric ofigmment.
Semontic Readobilly
1.67
Insight Discovery
2,00
Design Styte
2.67
6. Color Harmony: 4.67
Visuel Composition
3.00
The color schemn uses corefully controlled polette of dork green background with brighter green gccents, creating clear visual hierorchy without cousing fotigue. white dots provide perfect contrest for tracked objects.
Color Harmany
267
Figure 23: Cemini-2.0-Fllath and Claude-3 B.eede-3.--oooeee 3.5-Sonnet error cases.
43

---

Published as a conference paper at ICLR 2026
MODEL ERROR ANALYSIS CASES
Claude Sonnet
Fidelity
Fidelity: X
visuolization appeors mointain fideitty ocross all four bar chort uses proper zero bosslines, maps appear to standars projections, Howeves without legends clear lobels, it's difficutt verity complete consistency across wevs, prerenting perfect score.
encadings
seem consistent
Expression
Semantic Readability: 2,00 +
While visual elements technically clear, the visualization lacks essential explanatory like legends, titles, latels Users determine what the yellwbluee segments represent
map, matrics the maps. sevwney hampers understanding. Insight Discovery:
the signly
approoch potentol for revealing insights different geogrophic scales types,
proper labeling legends, users emilly meoningful not
patterns or understond relationship between vievs. synergistic potental reolized lock of contest.
Aesthetics
4. Design Style: 3.33
Human baseline
The viuolization demonstrates good innovation by combining different chart types (bor chart, detoiee mop, global heat map, choropleth map) in a coordinated multiview appronch, Each view uses oppropriate visualiation techniques for its dota type, and the overall style shows originality the multi stale geographic onalass approach.
Oata Fidelity
1.50
Semontic Reauabiltt
1,50
Insight Discovery
2.00
5. Visual Compesition: 4.00 (
Design Style
2,50
The four- ponel layout wel organized with apprapriate size praportions for each view tpe. The spacing between panels adequete, and the orrangement creates legical flow from temporal dete (bar chart) incressingly detailed geographic views, The composition effectively vtilces the ovailable space.
Vitual Composition
Hormany
Overall Score
1.83
Color Harmony: 2.67
While each indiviual could be better. The yellow blue arange, pink coordination
appropriate color schemes for purpose, the overall color
schemes
strongly unifies visual though they don't clash severaly More coordination would enhonce overall oesthetic unty..
2.94
MODEL ERROR ANALYSIS CASES
GPT 40
Fidelity
1. Data Fidelity: 5.00
The chart acturately represents the data with a clear zero beseline and appropriate ernor bors, reflecting data voriabilty precisely. The oxes are correctly lobeled, ensuring fideity the undertying dato.
Expression
most Readability: 3.00
might the purpose scate the bors requiring adeitional contest fully understond their significacce
are
Insight Discovery: C
chort allows users to idencity clear upward trend, and the error bors provide insight into dato variobiliy, offering significant ond proctical insights.
Aesthetics
Design Style: 2.00
baseline
design quite standard for bor chart, with blue bars and block error bars, locking
innovetive elements that would distingishh it from conventionol designs.
Dato Fideitty
Semontic Readiobilty
2.00
3,00
The spotial leyaut well argonized with appropriate sporing and alignment of bars and error bars, contributing to overall reosabillty the chart.
Visual Compesition: 4.00
Insight Discovery
Visual Compostion
Design Style
3.50
1.00
1,50
The color scheme is hasic yet effective, with blue bers and block error bars providing sufficient controst, but locking in worancy or uniqueress.
Color Hormony: 3.67
Color Hormany
2.50
Overall Score
2.25
Overall Score: 3.56
Figure 24: Claude-4-Sonnet and GPT-4o error cases.
44

---

Published as a conference paper at ICLR 2026
MODEL ERROR ANALYSIS CASES
Fidelity
Gemini 2.5 Pro
1. Data Fidelity: 5.00
The visuolzattonn consits of four identical Cartesian gnis that are perfectly formed. mes are cleorly labeled and consistently oriented across all four vows. The grid lines imply a uniform and consistent scalt, bath within each grid and between the grids. There no worping. distortion, or error in the representation of these coordinate systems. This represents a nighly failfuu template for plotting date.
and
Expression
Semantic Readability: 2 meaning
Y-azis, grid lines) very of perfect potential use for platting Howerer provide detgiled explonation any
remove any patential ambiguity for expieny detoiled.
purpose broodly inferable
Insight Discovery.
Although the grids empty thus offer direct insights, structure meelf high potentiol, use
muttipies loyout with identcal, perfectly olignedi grids powerful design pottern for comporotive anolyss, different dutosets or scenorios were piotted on each grid. uniform fromework would make it easy for user to compare them and discover patterns, trends, or anomaiees. The potential for dieor insight discovery is therefore high.
Aesthetics
Design Style: 1.00
design completely standard and conventiona. It basic, unadorned representation of Cartesian coordinate syztem, replicated four times. There are no unique styliticc eiements, innovztive treatments or any attempt originaiiyy it purety functional template noheres strictly to the most common and generic form, thus scoring the lowest for design style and innovation.
Human baseline
5. Visual Composition: 5.00 X
Data Fideiity
1.00
The visuel composhion escellent. The four grids are arronged perfectl symmetrical 2x2 lnyout. The spacing (qutters) between the gnds appears unform, and the alignment prenie both horizontally and verticolly This creates a highly ordered, bolanced, and visaaly harmonious presentation that is easy soon and oesthetially pleasing. 6. Color Harmony: 5.00 "
Semontic Reauibillty
1.00
Insight Discovery
1.00
Design Style
2.00
Visui
Hormany
The color cholces highly offectie and harmonious. The use of block for the prinory axes creates clear structural lines, The muted, desaturated blue for the secondory grid lines allws them to
Overall 1.83
guitee without visunily competing with dato that might be plotted. This minimolst, professional, and consistent resulting excellent unity readobilty.
creates vhuol hierorchy. The
al
Overall Score: 3.89
Figure 25: Gemini-2.5 -25-in error case.
45

---

Published as a conference paper at ICLR 2026
C EVALUATION FRAMEWORK AND CRITERIA
C.I
DETAILED EVALUATION QUESTIONS AND SCORING CRITERIA
This appendix provides comprehensive details on the evaluation framework underlying VISJUDGE- BENCH, including specific evaluation questions and scoring criteria for each visualization type and evaluation dimension.
Evaluation Framework Overview Our evaluation framework is inspired by the classical Fidelity, Expressiveness, and Aesthetics principles, operationalized into six orthogonal sub-dimensions:
Fidelity:
Data Fidelity: Evaluating visual-level data presentation accuracy and detecting mis- leading design patterns.
Expressiveness:
Semantic Readability: Clarity of information communication.
Insight Discovery: Effectiveness revealing meaningful patterns.
Aesthetics:
Design Style: Innovation and uniqueness.
Visual Composition: Spatial layout and organization. Color Harmony: Color coordination and visual appeal.
To support context-aware evaluation tailored to specific visualizations, design evaluation ques- tions and scoring criteria rewriting prompt template that transforms generic evaluation criteria into more concrete versions. An example rewriting is shown in Figure 26 which illustrates a set of customized sub-dimension evaluation questions and scoring criteria tailored to a specific single visualization chart.
This rewriting process incorporates: sub sttphddmmmioon___eeee.ee _ame__mamee_e_ (the sub-dimension name under three classical principles), sub_di mens sionnneee (the standard evaluation question and scor- ing criteria definitions). The resulting prompt enables the generation of chart metadata and adapted evaluation rubries that are grounded in the specific context of the visualization being evaluated.
Prompt Template: Evaluation Questions and Scoring Criteria Rewriting
You are professional data visualization evaluation expert. You need to generate targeted evalu- ation questions and specific scoring criteria for each metric based on the provided visualization chart and (sub wwwrbbdimessiiii (aciensssiinnnmeeee name) evaluation metrics.
btt__dddmeesiion_namma evaluation metrics information: isub_dimens wwwp:dddmesiionneeee text]
Task Requirements:
1. Carefully observe the provided { (image_type) type visualization chart
2. Based on the specific content of the chart (such as chart type, data content, visual elements, design features, etc.)
3. Combine the requirements and scoring criteria of each evaluation metric above
4. For each metric, generate:
A specific, targeted evaluation question
Custom scoring criteria (1-5 scale) that is specifically adapted to this chart's 's features
Question Format Requirements:
Questions should clearly point specific features of the chart (e.g., "This bur chart uses blue and red contrast...")
Questions should relate to the core points of the evaluation metric
46

---

Published as a conference paper at ICLR 2026
how oftan do you use these charts to Visualize data in your repor??
Chart Metadata
chart metadata extraction
('hart deseriptiin:
horizontal stacked
representing how
drfferent strates true indicate frequercy catugories from Very often' 'Nevar: The chart includes legend at the top to decode the colons".
orien different chart tyoes used reports.
'ey, swe.enttt Poorizomta stacked bar chartz, "Shades of dlue for different frequenddes", "Legend indicating trequency categorles", "Peroentage axis from DS to 10012, "Labala for each chart sypa?]]
question rewriting
Generated Customized Questions
questions (
Gale. Fdailly
"qeestoon:" *bocs this horizonaal stacked bar chart sccuratol rop sprrosnnt the dota with approprtate ancs, scales, and lobels whthou any distortion or miccing eloments?"
tustom, scoing"? 14
"Severe ripeppesenttoon,
missing axes or ubb s, or misleading scales
impossiblee Judge the cata accur atoly".
makinn inapproeaate percentage slighty
Acourate repretantation with appropriate scales and
"Sasicaly accessable
and visible distortion..
Fidelity
"5" Highly fanniul representaiion with clear and acourato 2x0s, scales, and lobals, ensuring precise data incorpretation
sermanicc maddbilityy (
"qastlorr MAre the mearings of the colors and lasels in this chart clear and wasy un understand, sllowing usars to accurataly sopp the data distriutiinn"" puston_scootng"
ulament meanings are completely unclear, and users canmot under stand what the colora or labels repr s
categories. "Mannings vague and unctesr inuutitent explanations, couitig difficurty understancting nurprntationn.
definito 15" "Meenrigs completely with bed socur explenttions
"Maanings
explanations
visual encoding lsbels".
usens to full anaadd the chert without any berrira""),
Insight disoovrryy:
"qaectior: "Dues this foo vontal stacked bae chart fac Stare the easy icentification of sightiiant dats insigtts, euch as trends or pattirms in chart type unage frequercy""
"custom_ scoring:: 1
11: "Completely unatie provide any valuable mrights, with cheotic presantation that fass to grlight any signfficatt patterme..
22: "offfcuut identily meaningtu ins gnos, most information being abvious?.
petteens
data ideroihable,
insights sig
average"
dantifcation
Expressiveness
"g" Frofound dignl
insights
revelee hightghtingg key
with importer lidiin value ').
"dssign :ityles
"question:" "eastinn""Dorr "Does this chart with ios use of grasient blues introduce any innovative or unique des gn ements that make it stand out from typicalionss ta of this kind?,
"suaton, acoring:: 8
"The chart design is night conventional with no unique elemenns, resembling many typical stacked bar charts without any distinct feanures."
Thee chart dasign is somewhat standard, minor estempts uniqueres emeas de not sigrificantly cilieren ate common designs.
"The chart noludes some innovative elements such color gradlent usage, but overal maintains conventioal layour typical stacked bar charts?.
highy Innovativo, incorporating gradients legouf, elemants distinttvve
"The chort
standard rsuuiizations
Visuul, composttinn's
"question: "Now wall doos the spatial layout cloments nts in this stacked bar chart facilitato ordenty and comprohonsiblc presentation?"
"oustom, scoring: 34
"tt "The spatial layout chaotic with misel goed bars and poo fy placed legend, making comprehension diffccutt.
"The spatisl taynut has che iesues, some missiignns nnd etements and subeptimal legend placement sthecting mladabilly..
"g". "The spotial layour mostly ressonable, with acceptable algnment and legend placement, though some minor imp ovemmnnns could be made?,
"The
grgiieedd with approprstey wilneed elements cxamplary porfectly aligned
y placed legend, enturing readabilityy
"9".
intogr
space afficiecyy:..
"ootor hww.mhmmmm
"quealkor:;" *Exaluata the many of blue gradiant color used d diferenttatt frequency
ina Dona the cofor prevent visual faligue wlile guiding ig understandingg.
"custom_ 31
"T: Thee color schems of chaotic with poorly chosen contrasting dlars, caus na signfficnnt visual fat que and confusion:;
'2" "The color scheme lacks sufficlent harmony, with too many simller blues causing some visual farigue and kindering cotegory distinction.,
"The color schama moatly hamenizus, though the range buus right cause sight complisity by and minor comprehent ne soums:
Aesthetics
'The color scheme harmonious with wel-chosen gradentt effectively diferrnttating categories minimizing visual contusion..
Figure 26: Rewriting result of customized evaluation questions and scoring criteria for a single visualization chart.
Questions should guide evaluators to think about the specific performance of that met-
ric
Questions should end with: "Please provide 1-5 score based on the scoring criteria." "
Scoring Criteria Format Requirements:
For each question, provide a custom 1-5 rating scale
Each score description should specifically reference elements in this chart
Score 1 should describe the worst-case scenario for this chart
Score 5 should describe the ideal implementation for this chart
Scores 2-4 should provide meaningful intermediate levels
Return Format: JSON object with the following structure:
"chart_descripti "Brief description of the chart (chart type, main features, etc.)", " ["Observed element 1", "Observed element 2", " ").."),
47

---

Published as a conference paper at ICLR 2026
"questions":
@aub__imensicn_nne sariiioonnamme_:__
"question"; "Specific question text...", "eusttonsscoring"
"1": "Score 1 description specific to this chart...",
"2": "Score 2 description....
"3": "Score 3 description..."
"4": "Score 4 description..."
"5": "Score 5 description..
b
C.1.1 SINGLE VISUALIZATION EVALUATION CRITERIA
Data fidelity evaluation questions and scoring criteria: The following prompt is designed to guide the generation of scoring rubrics for the Data Fidelity dimension, using 1-5 scale. It focuses on ensuring that visual representations truthfully and accurately reflect the underlying data.
Prompt: Single Visualization Data Fidelity Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Evaluates whether the visual encodings faithfully represent the data without misleading distor- tions. Focus appropriate axes and baselines (c.g., zero baseline for bar charts), reasonable scale ranges and tick intervals, consistency between encodings (postion//ength/anlll/aaad color) and numeric labels, and absence of aspect-ratio deformation, cropping/stretching. 3D distortion, or improper broken axes.
Scoring criteria:
S 1. Severe misrepresentation or impossible to judge due missing/unreadable key ele- ments (titles, legends, axcs/ticks/lablls, units, data labels, baselines, etc.), or clear dis- tortions (e.g., non-zero baseline in bars exaggerating differences, 3D effects causing misread, obvious aspect-ratio deformation).
S 2. Important issues likely to mislead: inappropriate baseline/scale range, inconsistent en- coding vs labels, selective ranges that exaggerate differences, or partial key elements missing causing notable uncertainty.
S 3. Basically acceptable with minor uncertainties: some elements may be missing or hard read but obvious distortion; scales and encodings are mostly reasonable and roughly consistent with labels.
S 4. Accurate and standardized: axes/baselines are appropriate, scale ranges and ticks reasonable, encodings match labels, and visible distortion manipulation is present.
S. Highly faithful representation: all relevant elements are present and clear; axes/base- lines and scale choices are well-justified (e.g., proper zero baseline for bars, appro- priate linear/log choice), encodings and labels are fully consistent, with no cropping. deformation, 3D misuse.
Semantic readability evaluation questions and scoring criteria: The following prompt guides the generation of scoring rubrics for the Semantic Readability dimension. It evaluates how clearly the chart communicates information through its visual encodings and annotations.
48

---

Published as a conference paper at ICLR 2026
Prompt:
Single Visualization Semantic Readability Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Based on complete and clear chart elements, evaluates whether users can understand the mean- ing of these elements and the information the chart conveys, including clarity of visual encoding (whether meanings of colors, shapes, sizes are clear) and clarity information communication (whether users accurately understand chart content).
Scoring criteria:
S 1. Chart element meanings are completely unclear, cannot understand what visual encod- ing represents, users cannot obtain meaningful information.
S 2. Chart element meanings are vague and unclear, insufficient visual encoding explana- tions, users have difficulty understanding and need extensive guessing to understand chart content.
S 3. Chart element meanings are basically clear, users can understand main information, but still have understanding barriers in some encoding or labeling aspects.
S 4. Chart element meanings are clear and definite, visual encoding has appropriate expla- nations, users can smoothly understand the information conveyed by the chart.
5. Chart element meanings are completely clear, visual encoding explanations are de- tailed and accurate, users understand all chart information without any barriers.
Insight discovery evaluation questions and scoring criteria: The following prompt guides the generation scoring rubries for the Insight Discovery dimension. It assesses the chart's ability to reveal meaningful patterns, trends, non-obvious findings.
Prompt: Single Visualization Insight Discovery Evaluation Questions and Scoring Cri- teria (1-5 Scale)
Description:
Evaluates whether the chart can easily identify significant and meaningful data insights, trends, patterns, or anomalies that must have actual value and guiding significance for users, rather than trivial or obvious general information.
Scoring criteria:
S 1. Completely unable to provide any valuable insights, chaotic information presentation, significant meaningful discoveries, insight delivery completely failed.
S 2. Difficult to identify meaningful insights, most presented information is trivial obvi- ous, lacking practical value, insight discovery difficult.
S 3. Can identify some basic data patterns or trends, the significance and practicality of insights are average, with limited guiding value for users.
S 4. Can clearly identify obvious and useful data insights and trends, discovered patterns or anomalies have certain practical significance and reference value.
S. Very intuitively reveals profound and highly significant data insights, can discover significant trend changes, important anomalies, or key patterns that have important guiding value for decision-maaiie making or understanding.
Design style evaluation questions and scoring criteria: The following prompt guides the gen- cration of scoring rubrics for the Design Style dimension. It reflects the level of visual creativity, uniqueness, and design innovation present in the chart.
49

---

Published as a conference paper at ICLR 2026
Prompt:
Single Visualization Design Style Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Evaluates whether the chart design has innovation and uniqueness, whether can stand out from many visualization works through novel design elements and unique style characteristics.
Scoring criteria:
S 1. Design completely lacks innovation, style is outdated or shows obvious imitation, no uniqueness originality.
S 2. Design lacks innovation, style is quite ordinary, basically uses common design tech- niques, insufficient uniqueness.
3. Design has some innovative elements, but overall is relatively conventional, uniqueness not prominent enough, innovation level is average.
S 4. Design has strong innovation, style is quite unique, has novel design elements, with certain originality.
S5. Design is extremely innovative, style is unique and novel, uses innovative design ele- ments or expression techniques, with strong originality and recognizability.
Visual composition evaluation questions and scoring criteria: The following prompt guides the generation of scoring rubrics for the Visual Composition dimension. It focuses on the spatial ar- rangement and organization visual elements for effective communication.
Prompt:
Single Visualization Visual Composition Evaluation Questions and Scoring Cri-
teria (1-5 Scale)
Description:
Evaluates whether the spatial layout of chart elements is reasonable, including whether element positions, size proportions, spacing distribution and other spatial relationships appropriately and orderly arranged.
Scoring eriteria:
S 1. Spatial layout seriously unreasonable, element positions chaotic, proportions severely unbalanced, spacing distribution is disorderly.
S 2. Spatial layout has obvious unreasonable aspects, improper element positions, unbal- anced proportions, uneven spacing distribution.
S 3. Spatial layout is basically reasonable, element positions are acceptable, proportional relationships are basically coordinated, but may have small layout issues.
S 4. Spatial layout is quite reasonable, element positions appropriate, good proportional relationships, reasonable spacing distribution, proper space utilization.
S. Spatial layout is very reasonable, element positions are perfect, proportions are coor- dinated, spacing distribution is even, space utilization efficiency is extremely high.
Color harmony evaluation questions and scoring criteria: The following prompt guides the gen- eration of scoring rubrics for the Color Harmony dimension. evaluates how effectively the color scheme supports readability, aesthetic appeal, and visual coherence.
Prompt: Single Visualization Color Harmony Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Evaluates whether chart color choices are coordinated, contrast is appropriate, color quantity is reasonable, and whether it avoids too many colors causing visual fatigue.
50

---

Published as a conference paper at ICLR 2026
Scoring criteria:
$ 1. Color choices are chaotic, extensive use uncoordinated colors, serious visual fatigue, completely unable to effectively guide user understanding, color use severely affects information delivery.
S 2. Color choices are not coordinated enough, using too many colors (5 or more), inappro- priate contrast, causing obvious visual fatigue and understanding difficulties.
S 3. Color choices are basically coordinated, but color quantity excessive (4-5 colors), may cause slight visual complexity, somewhat affecting understanding.
S 4. Color choices are relatively coordinated, appropriate contrast, reasonable color quan- tity control (3-4 colors), basically avoiding visual confusion, well guiding user under- standing.
S 5. Color choices are very coordinated, appropriate contrast, limited color scheme (usually 1-3 main colors), colors are simple and unified, completely avoiding visual fatigue, effectively maintaining user attention focus.
C.1.2 MULTIPLE VISUALIZATION EVALUATION CRITERIA
Data fidelity evaluation questions and scoring criteria: The following prompt is designed to guide the generation of scoring rubrics for the Data Fidelity dimension, using 1-5 scale. It focuses on ensuring that visual representations truthfully and accurately reflect the underlying data.
Prompt: Multiple Visualization Data Fidelity Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Evaluates whether multiple coordinated views faithfully represent data without distortion indi- vidually and in combination. Check appropriate axes/baselines, reasonable scale ranges, consis- tency between encodings and labels, and cross-view consistency (c.g., comparable scales where appropriate), avoiding aspect-ratio deformation, cropping, 3D misuse, improper broken axes.
Scoring criteria:
S 1. Severe misrepresentation or impossible to judge due to missing/unreadable key ele- ments in one or more views, or clear distortions (non-zero bar baseline where required, 3D effects, obvious deformation), leading to unreliable reading.
S 2. Important issues exist in one or more views (inappropriate baseline/scale, inconsistent encoding vs labels, selective ranges), partial key elements missing causing notable cross-view uncertainty.
S 3. Basically acceptable overall: minor uncertainties or small issues in some views, but no obvious distortion; scales and encodings mostly reasonable across views.
S 4. Accurate and standardized across views: appropriate axes/baselines and scales; encod- ings match labels; no visible distortion; reasonable cross-view scale alignment where needed.
S 5. Highly faithful multi-vis representation: all views present complete, clear elements; well-justified axes/baselines and scale choices; full consistency between encodings and labels; no cropping/deformationn3)) misuse; coherent cross-view comparability.
Semantic readability evaluation questions and scoring criteria: the generation of scoring rubrics for the Semantic Readabiliry dimension. It evaluates how clearly the chart communicates information through its visual encodings and annotations.
The following prompt guides
51

---

Published as a conference paper at ICLR 2026
Prompt:
Multiple Visualization Semantic Readability Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Based on complete and clear chart elements, evaluates whether users can understand the mean- ing of these elements and the information the chart conveys, including clarity of visual encoding (whether meanings of colors, shapes, sizes are clear) and clarity information communication (whether users accurately understand chart content).
Scoring criteria:
S 1. Chart element meanings are completely unclear, cannot understand what visual encod- ing represents, users cannot obtain meaningful information.
S 2. Chart element meanings are vague and unclear, insufficient visual encoding explana- tions, users have difficulty understanding and need extensive guessing to understand chart content.
S 3. Chart element meanings are basically clear, users can understand main information, but still have understanding barriers in some encoding or labeling aspects.
S 4. Chart element meanings are clear and definite, visual encoding has appropriate expla- nations, users can smoothly understand the information conveyed by the chart.
5. Chart element meanings are completely clear, visual encoding explanations are de- tailed and accurate, users understand all chart information without any barriers.
Insight discovery evaluation questions and scoring criteria: The following prompt guides the generation scoring rubries for the Insight Discovery dimension. It assesses the chart's ability to reveal meaningful patterns, trends, or non-obvious findings.
Prompt: Multiple Visualization Insight Discovery Evaluation Questions and Scoring Cri- teria (1-5 Scale)
Description:
Evaluates whether the entire multi-vis visualization can easily identify significant and mean- ingful data insights, and whether multiple views synergize to reveal deeper and more valuable comprehensive insights than single views.
Scoring criteria:
S 1. Completely unable to provide any valuable comprehensive insights, lack of coordina- tion between views, no significant meaningful discoveries, insight delivery com- pletely failed.
S 2. Difficult to identify meaningful comprehensive insights, most information presented by multiple views trivial or independent, lacking synergistic value, insight discovery difficult.
S 3. Can identify some basic multi-vis insights, but the significance and practicality of in- sights average, limited cooperation between views, guiding value not prominent enough.
$ 4. Can clearly identify obvious and useful multi-dimennional insights, good cooperation between views. discovered comprehensive patterns have certain practical significance and reference value.
5. Very intuitively reveals profound and highly significant comprehensive insights, per- fect synergy between multiple views, can discover significant cross-dimensional pat- terns, important correlations, or key anomalies with important guiding value.
Design style evaluation questions and scoring criteria: The following prompt guides the gen- eration of scoring rubrics for the Design Style dimension. It reflects the level of visual creativity, uniqueness, and design innovation present in the chart.
52

---

Published as a conference paper at ICLR 2026
Prompt: Multiple Visualization Design Style Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Evaluates whether the overall design of multi-vis visualization has innovation and uniqueness, whether it can stand out from many visualization works through novel design elements, unique style characteristics, and coordinated style language.
Scoring criteria:
S 1. Multi-vis design completely lacks innovation, style is outdated or views have chaotic styles, no uniqueness or originality.
$ 2. Multi-vis design lacks innovation, style quite ordinary, basically uses common de- sign techniques, insufficient uniqueness.
S 3. Multi-vis design has some innovative elements, but overall is relatively conventional, style characteristics not prominent enough, innovation level average.
S 4. Multi-vis design has strong innovation, style is quite unique, coordinated style among views with certain originality.
S5. Multi-vis design extremely innovative, style unique and novel, views maintain unified style while showing strong originality and recognizability.
Visual composition evaluation questions and scoring criteria: generation of scoring rubrics for the Visual Composition dimension. It focuses on the spatial ar- rangement and organization of visual elements for effective communication.
The following prompt guides the
Prompt:
Multiple Visualization Visual Composition Evaluation Questions and Scoring
Criteria (1-5 Seale)
Description:
Evaluates whether the spatial layout of individual sub- views and the overall arrangement of multiple views are reasonable, including element composition within individual views, size proportions between multiple views, arrangement methods, spacing distribution, etc., forming a harmonious and orderly visual whole.
Scoring criteria:
S 1. Sub-view layouts chaotic overall multi-vis arrangement is seriously unreason- able, size proportions severely unbalanced, spacing distribution is disorderly.
$ 2. Sub-view layouts or multi-vis arrangement have obvious unreasonable aspects, inap- propriate size proportions, uneven spacing distribution.
S 3. Sub-view layouts are basically reasonable, overall multi-vis arrangement is acceptable, but may have small issues proportional relationships or spacing handling.
S 4. Sub-view layouts are good, overall multi-vis arrangement is reasonable, appropriate size proportions, proper spacing distribution, good space utilization.
S Sub-view layouts are perfect, overall multi-vis arrangement is extremely reasonable, coordinated size proportions, even spacing distribution, extremely high space utiliza- tion efficiency.
Color harmony evaluation questions and scoring criteria: eration of scoring rubrics for the Color Harmony dimension. It evaluates how effectively the color scheme supports readability, aesthetic appeal, and visual coherence.
The following prompt guides the gen-
53

---

Published as a conference paper at ICLR 2026
Prompt:
Multiple Visualization Color Harmony Evaluation Questions and Scoring Cri- teria (1-5 Scale)
Description:
Evaluates whether color choices in multi-vis are appropriate, including color combinations within each sub-view, coordination of color schemes between multiple views, unity of over- all tone, and appropriate control of color quantity and saturation.
Scoring criteria:
S 1. Sub-view color choices very inappropriate, overall color matching among multiple views is chaotic, tone conflicts or color use is seriously inappropriate.
$ 2. Sub-view color choices are not appropriate enough, overall color matching among multiple views is not coordinated enough, tone not unified or color use inappropriate.
S 3. Sub-view color choices are basically appropriate, overall color matching among mul- tiple views acceptable, but may have small issues in tone unity or saturation.
S 4. Sub-view color choices relatively appropriate, overall color schemes among multi- ple views are quite coordinated, tone basically unified, appropriate color use.
S view color choices are very appropriate, overall color schemes among multiple views are highly coordinated and unified, perfect tone matching, appropriate control of color quantity and saturation.
C.1.3 DASHBOARD EVALUATION CRITERIA
Data fidelity evaluation questions and scoring criteria: The following prompt is designed to guide the generation of scoring rubrics for the Data Fidelity dimension, using 1-5 scale. It focuses on ensuring that visual representations truthfully and accurately reflect the underlying data.
Prompt: Dashboard- Data Fidelity Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Evaluates whether the dashboard faithfully represents business data in each component and in the overall composition. Focus on appropriate axes/baselines, reasonable scale ranges, con- sistency between encodings and numeric labels, avoidance of aspect-ratio deformation, crop- ping/stretching, 3D distortion, or improper broken axes; also consider consistency of compara- ble metries across components.
Scoring criteria:
S 1. Severe misrepresentation or impossible to judge due to missing/unreadable key el- ements in important components, clear distortions (e.g., improper bar baselines, distorted 3D, obvious deformation), undermining data fidelity.
S 2. Important issues likely to mislead one or more components: inappropriate baseli- nes/scales, inconsistent encoding labels, selective ranges, missing key elements causing notable uncertainty at the dashboard level.
S 3. Basically acceptable overall: minor issues or uncertainties in parts, but no obvious distortion; most components reasonable scales/encodings consistent with labels.
S 4. Accurate and standardized: components and overall dashboard use appropriate ax- es/baselines and reasonable scales; encodings match labels; no visible distortion.
S. Highly faithful: all components present complete and clear elements; axes/baselines and scales are well-justified; encodings and labels fully consistent; no cropping/defor- mation/3D misuse; consistent comparability across related components.
Semantic readability evaluation questions and scoring criteria: The following prompt guides the generation of scoring rubrics for the Semantic Readability dimension. evaluates how clearly the chart communicates information through its visual encodings and annotations.
54

---

Published as a conference paper at ICLR 2026
Prompt: Dashboard Semantic Readability Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Based on complete and clear dashboard elements, evaluates whether users can understand the meaning of these elements and the information the dashboard conveys, including clarity of visual encoding (whether meanings of colors, shapes, sizes, gauge pointers, status indicators clear) and clarity of information communication (whether users accurately understand various parts and overall business meaning).
Scoring criteria:
S 1. Dashboard element meanings completely unclear, cannot understand what charts, indicators, color encoding represent in business terms, users cannot obtain meaningful information.
S 2. Dashboard element meanings are vague and unclear, insufficient business indicator explanations, unclear color encoding and status indication, users have difficulty under- standing and need extensive guessing to understand content.
S 3. Dashboard element meanings are basically clear, users can understand main business information and indicator meanings, but still have understanding barriers some en- coding or labeling aspects.
S 4. Dashboard element meanings are clear and definite, business indicators have appro- priate explanations, clear visual encoding, users can smoothly understand the business information conveyed by the dashboard.
S 5. Dashboard element meanings are completely clear, business indicator explanations are detailed and accurate, all visual encoding has clear interpretation, users can understand all business information in the dashboard without any barriers.
The following prompt guides the generation of scoring rubries for the Insight Discovery dimension. It assesses the chart's ability to reveal meaningful patterns, trends, or non-obvious findings.
Insight discovery evaluation questions and scoring criteria:
Prompt: Dashboard Insight Discovery Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Evaluates whether the entire dashboard can easily identify significant and business- valuualle key insights, whether key business indicators (KPIs) prominently display important information, and whether it can quickly discover business issues or opportunities that require attention.
Scoring criteria:
S 1. Completely unable to provide any valuable business insights, key indicators not promi- nent, no significant or meaningful business discoveries, insight delivery completely failed.
S 2. Difficult to identify meaningful business insights, insufficient highlighting of key in- dicators, most presented business information trivial obvious, lacking decision value.
S 3. Can identify some basic business insights, key indicators have some prominence, but the significance and business value of insights average, with limited guiding effects.
S 4. Can clearly identify obvious and useful business insights, key indicators prominently displayed, discovered business patterns or anomalies have certain practical value and decision reference significance.
$ 5. Very intuitively reveals profound and highly business-valuable insights, key indica- extremely prominent, quickly discover significant business trends, important anomalies, or key opportunities with important guiding significance for business deci- sions.
55

---

Published as a conference paper at ICLR 2026
Design style evaluation questions and scoring criteria: The following prompt guides the gen- eration of scoring rubrics for the Design Style dimension. It reflects the level of visual creativity, uniqueness, and design innovation present in the chart.
Prompt: Dashboard
Design Style Evaluation Questions and Scoring Criteria (1-5 Scale)
Description:
Evaluates whether the overall design of the dashboard has innovation and uniqueness, whether it can show attractive visual effects and business professionalism through novel design elements, unique style characteristics, and professional design language.
Scoring criteria:
$ 1. Dashboard design completely lacks innovation, style is outdated or chaotic, no unique- ness, seriously insufficient professionalism.
S 2. Dashboard design lacks innovation, style is quite ordinary, basically uses common design techniques, insufficient uniqueness and professionalism.
S 3. Dashboard design has some innovative elements, but overall is relatively conventional, style characteristics not prominent enough, average professionalism.
S 4. Dashboard design has strong innovation, style is quite unique and professional, with certain originality and good business sense.
S 5. Dashboard design is extremely innovative, style is unique, novel, and professional, overall presents strong originality and high business aesthetics, leaving deep impres- sions.
Visual composition evaluation questions and scoring criteria: The following prompt guides the generation scoring rubrics for the Visual Composition dimension. focuses on the spatial ar- rangement and organization of visual elements for effective communication.
Prompt:
Dashboard- Visual Composition Evaluation Questions and Scoring Criteria (1-5
Scale)
Description:
Evaluates whether the spatial layout of dashboard components and overall arrangement reasonable and beautiful, including size proportions of various blocks, alignment relationships between components, spacing distribution, information density control, etc., forming harmo- nious, orderly, and beautiful visual whole.
Scoring criteria:
S 1. Component layouts are chaotic or overall arrangement is seriously unreasonable, size proportions severely unbalanced, spacing distribution is disorderly, poor information density handling.
S 2. Component layouts overall arrangement have obvious unreasonable aspects, inap- propriate size proportions, uneven spacing distribution, or information too crowded.
S 3. Component layouts are basically reasonable, overall arrangement acceptable, but may have small issues in proportional relationships, spacing handling, or information density.
S 4. Component layouts are good, overall arrangement is reasonable and beautiful, appro- priate size proportions, proper spacing distribution, good information density control.
S 5. Component layouts are perfect, overall arrangement is extremely reasonable and beau- tiful, coordinated size proportions, aesthetic spacing distribution, appropriate informa- tion density control, extremely high space utilization efficiency.
Color harmony evaluation questions and scoring criteria: The following prompt guides the gen- eration of scoring rubrics for the Color Harmony dimension. evaluates how effectively the color scheme supports readability, aesthetic appeal, and visual coherence.
56

---

Published as a conference paper at ICLR 2026
Prompt:
Dashboard Color Harmony Evaluation Questions and Scoring Criteria (1-5
Scale)
Description:
Evaluates whether color choices the dashboard are appropriate and beautiful, including color combinations within various components, coordination of overall color schemes, unity of tone, and appropriate control of color quantity and saturation, ensuring both beauty and compliance with business professional requirements.
Scoring criteria:
$ 1. Component color choices are very inappropriate, overall color matching is chaotic, conflicts color use is seriously inappropriate, very poor aesthetic effect.
S 2. Component color choices are not appropriate enough, overall color matching is not coordinated enough, tone not unified or color use inappropriate, affecting aesthetic effeect.
S 3. Component color choices are basically appropriate, overall color matching is accept- able, but may have small issues tone unity, saturation, or business sense.
S 4. Component color choices are relatively appropriate, overall color schemes are quite coordinated, tone basically unified, appropriate color use, good business sense.
S 5. Component color choices are very appropriate, overall color schemes are highly coor- dinated and unified, perfect tone matching, appropriate control of color quantity and saturation, presenting excellent business aesthetics.
C.2 EVALUATION PROMPT TEMPLATES
To guide consistent evaluation visualizations, we present a structured prompt template built upon the classical Fidelity, Expressiveness, and Aesthetics principles, operationalized into six orthogonal sub-dimensions. The prompt is generated based on the rewritten evaluation questions and scoring criteria from Appendix C.I
The following prompt template outlines how evaluators are instructed to conduct evaluation using these customized inputs. The prompt includes the following components:
The
} field specifies the total number of evaluation criteria distributed across the three main dimensions.
The
field indicates how many of these criteria adopt customized scor- ing guidelines tailored to the chart.
The (chart deserii cnterriptoon field provides metadata about the visualization, such as chart type and design structure.
The (fidelity ietiitttsseetnnntttteeee field includes rewritten evaluation questions and scoring criteria aligned with the data fidelity sub-dimension.
The texpressiveesss
field covers the semantic readability and insight
discovery sub-dimensions.
The (aesthetis ceteiissssssiiioeeeeeeeeeeeeieeo sectiennnn field captures design-related sub-dimensions, including style, spatial composition, and color harmony.
Prompt Template: Visualization Evaluation
You are rigorous data visualization evaluation expert. You must strictly judge each visual- ization based on the "Fidelity, Expressiveness, and Aesthetics" framework and the 1-5 scoring criteria for each metric.
Chart description: (chart_descriptice
57

---

Published as a conference paper at ICLR 2026
Note:
This chart has (total_count) evaluation metrics across three dimensions, which fcustoom_cuntt] use custom scoring criteria.
The evaluation follows the "Fidelity, Expressiveness, and Aesthetics" principle:
Fidelity: Data accuracy and truthfulness
Expressiveness: Information clarity and understandability
Aesthetics: Visual aesthetics and refinement
For each evaluation question, provide score from to 5 and reasoning based on the scoring criteria.
(fidelity_section]
[exprressiveness_secctooo)
faesthetics_section)
Return Format: JSON object with the following structure:
"data_fidelity": ("score": 1-5, "reascning": "Your explanation here. I,
"semantic reacabiiityaa;; :"score": 1-5, "reasoning": "Your
explanation here.") ",,
" "insight_ddisooeery" ("score"; 1-5, "reascning": "Your explanation
here. "), "design_style": ("score"; 1-5, "reasoning"; "Your explanation here."
}, "visual_composition" ("score": 1-5, "reascning": "Your explanation here."), "color_harmony": ("score"; 1-5, "reasening"; "Your explanation here. "I, laverage_score", "the average of the abeve six scores, rounded to 2 decimals"
Where for each metric, score should be an integer from 1 to 5 based on the above metric de- scriptions and the 1-5 scoring criteria, and reasoning should explain your choice. average_ score is the average of all six scores rounded to 2 decimal places. Do not include any additional text, only the ISON object.
58

---

Published as a conference paper at ICLR 2026
D MODEL IMPLEMENTATION AND TRAINING DETAILS
D.1 SOFTWARE ENVIRONMENT
The training framework based the open-source library SWIFT (Scalable lightWeight Infrastruc- ture for Fine- TimenT
[ utilizing PyTorch and DeepSpeed (ZeRO Stage 2) for distributed training and memory optimization.
D.2 REWARD FUNCTION
As described in the main text, composite reward function, Reomposites is weighted combina- tion of an accuracy reward (Race) and format reward (Rtormat), with weights of 0.9 and 0.1, respectively.
Accuracy Reward (Race) This component measures the proximity between the model's predicted scores (across the six dimensions and the overall average score) and the human-annotated ground- truth values. We employ a smooth exponential decay function to calculate the reward for each individual score:
[scorepredieeed scoreground-ruthh
Race_single = exp
0.5
The final accuracy reward the average of the rewards calculated for all dimensional scores and the overall average score.
Format Reward (Biormat) This component ensures the model produces a complete and parsable JSON structure. The reward is 1.0 if the model's output contains all required fields (i.e., the score and reasoning for each the six dimensions, plus the average_score score) otherwise, the re- ward is O.
D.3 HYPERPARAMETER SETTINGS
We fine-tuned four representative open-source multimodal models: WL-.---VL-B---Iiiii Qwen2.5--V--3B-nnstuee InterVVL3-8B. and Rank Adaptation (LoRA) for parameter-efficient fine-tuning with consistent hyperparameter con- figurations across architectures. Specifically, we set LoRA rank and alpha to 128 and applied it to all linear layers. For reinforcement learning, we used the Group Relative Policy Optimization (GRPO) algorithm with beta parameter of 0.01. The models were trained for 5 epochs with a learning rate of 1c-5, using Cosine Annealing scheduler with warmup ratio of 0.1. We used the Adam W optimizer with weight decay of 0.01. The global batch size was 16 (per-device batch size of 1 with 4 gradient accumulation steps, across 4 GPUs). For computational efficiency, utilized bf bfloat 1 6 mixed-precision training.
All models employed Low-
59

---

Published as a conference paper at ICLR 2026
Correlation with Huma
1.25
0.6
Prediction Error
Mean Absolute Error (MAE)
1.00
Squared (MSE)
0.4
(R*=0.964)
MSE Trend (R*-0.981)
Correlation
Log Trend (R'=0.678)
0.25
0.0
500
1000
1500
2000
0.00
500
1500 2000
2500
Training Data Scale (samples)
Training Data Scale (samples)
(a) Training data scale vs. human-model correlation
(b) Training data scale vs. prediction error
Figure 27: Impact training data scale on VISJUDGE model performance.
E EXTENDED EXPERIMENTS
E.1 How DOES TRAINING DATA SCALE AFFECT MODEL PERFORMANCE?
To evaluate data scaling effects and guide deployment strategies, we analyze VISJUDGE perfor- mance across different training data scales with a single training epoch. To ensure fairness, data samples are proportionally extracted based on visualization types and score distributions. Figure reveals clear mathematical patterns as data scale increases.
Predictable Scaling Laws. Model performance follows well-defined trends: human Houman-model model correla- tion shows logarithmic growth (R²=0.678) from 0.341 0.639 the training data scale increases to 2,442 samples, while prediction errors exhibit exponential decay with MAE decreasing from 0.847 0.440 (R2=0.964) and MSE from 1.045 to 0.309 (R3=0.981). The 500-1,000 sample range pro- vides the most efficient improvement, contributing 41.7% of total correlation gains, while beyond 1,000 samples, marginal returns diminish but remain valuable for continued enhancement.
E.2 GENERALIZATION TO WORLD APPLICATIONS
To validate the practical generalization capability of VISJUDGE, we integrate it into two real-world visualization systems: MatPlotAgent (visualization generation) and HAIChart (visualization rec- ommendation). These scenarios differ significantly from our training set, providing rigorous OOD evaluation.
E.2.1 EXPERIMENTAL SETUP
MatPlotAgent (Visualization Generation). We integrate VISJUDGE into MatPlotAgent (Yang al. 2024) as feedback mechanism for iterative quality improvement. We seven generation models (GPT-5, GPT-4o, Claude-4-Sonnet, Claude-3.5-Sonnet, Gemini-2.5-Pro. Swen2.5-VL-72BB Instruct, Qwen2.5. V VL-7B- VL-Tte-nstrrut), comparing baseline (no feedback) against feedback from 7B base model, 72B base model, and VISJUDGE. Final visualizations are automatically evaluated using GPT-4o on 0-100 scale, following the original MatPlotAgent evaluation setup, where higher scores indicate better overall visualization quality.
HAIChart (Visualization Recommendation) We integrate VISJUDGE into HAIChart (Xie al,, 2024 as reward model to re-rank candidate visualizations. We evaluate on the VizML dataset con- taining real user-created Plotly visualizations, which represents a significant distribution shift from training data. We compare the original system against versions using 7B, 72B, and VISJUDGE as reward models, measuring Hit and Hit@3 accuracy on data queries, design choices, and overall recommendation tasks, where higher values indicate better top-k recommendation accuracy.
E.2.2 RESULTS AND ANALYSIS
MatPlotAgent Results. Table 7 shows that VISJUDGE achieves consistent quality improvements across all seven generation models, with average gain of +6.07 points (range: +5.53 to +8.91). )). In stark contrast, base models without domain-speciinn fine-tuning often degrade performance: the 7B base model averages -2.39 points while the 72B base model averages -0.61 points. This degradation attributable to their systematic evaluation biases documented in the main paper: score inflation (7B model μ 3.89 human f 3.13) and sharp score concentration around 4.0 limit dis-
60

---

Published as a conference paper at ICLR 2026
Table 7: MatPlotAgent quality improvement with different feedback models.
Generation Model
Direct Decoding (Baseline)
Qwen2.5-VL-7B as Feedback
Qwen2.5-VV--2BB as Feedback
VISJUDGE as Feedback
GPT-5
67.15
66.49
69.80
72.07
GPT-4o
62.71
58.49
63.53
71.62
68, 68,10
65.04
66.12
72.20
Claude-4-Son 1laude4-Sonnet
64,50
67.25
65.20
72.23
Claude-3.5 3.0e-33.-Sonneee
63.89
59.23
63.
69.04
Qwen2.5-VV
61.18
58.70
67.32
Qwen2.5 Bwee2.5-VL-7B-Inssie
49.76
94.94
55.29
Table 8: Visualization recommendation performance in HAIChart integration VizML dataset. VISJUDGE as reward model improves recommendation accuracy across different tasks, outperform- ing larger base models.
Task
Metric
HAIChart (Baseline)
Qwen2.5-VL-7B a Reward
Qwen2.5-VL-72B as Reward
VISJUDGE a Reward
@it@1 Hit@3
79.30%
80.30%
82.80% 92.10%
83.70%
Data Queries
91.90%
90.70%
93.20% %
@it@1
48.70%
45.90%
46.50%
48.80 82.70% %
Design Choices
Hit@3
81.50%
80.70%
79.80%
@it@1
36.90%
36.30%
39.10%
39.40% %
Overall
Hit
67.40%
69.10% 10%
70.20%
72.70 %
criminative capability, leading them to praise flawed visualizations while failing to identify genuine quality issues. Such biased feedback actively misleads generation models, directing iterative refine- ment toward suboptimal outputs. This demonstrates that domain-speciff fine-tuning is not merely beneficial but essential-sinee scaling model size without domain expertise harm rather than help. Even state-of-the-art commercial models benefit significantly from VISJUDGE'S feedback: GPT-5 +4.92, Claude-4-Sonnet +7.73, Gemini-2.5-Pro +4.10. This suggests that current generation models can effectively improve their outputs when provided with accurate quality assessments, and VISJUDGE provides this critical evaluation signal.
HAIChart Results. Table 8 shows VISJUDGE significantly improves recommendation accuracy across all metries: Data Queries Hit@1 from 79.30% to 83.70% (+4.40%), Overall Hit@ from 36.90% to 39.40% (+2.50%). and Overall Hit@3 from 67. 67.40% to 72.70% (+5.30%). The 7B and 72B base models show minimal to moderate improvements, while VISJUDGE achieves larger gains. Notably, the VizMI. dataset represents a significant distribution shift from our training data (real Plotly community visualizations), yet VISJUDGE generalizes effectively, validating that learns generalizable quality assessment principles rather than memorizing training patterns.
E.2.3 CROSS-ARCHITECCTRE CENERALIZATION
Beyond application scenarios, validate that our fine-tuning methodology generalizes across di- verse model architectures. We fine-tuned four different base models using the same training pro- cedure: Owenn25--V--7B-nntmmic (correlation improves from 0.341 1 to 0.687, +101.5%), Qwen2.5- -L-3B-Instruct (from 0.272 to 0.648, + +13,,2%) VL3e (from 0.409 to 0.660, +61 (61.4%), and LLaVA-I.6-mistral-77. (from 0.180 to 0.605, +236.1 1).. All models show substantial improvements after fine-tuning, with an average correlation increase of +0.334 (corresponding to + +34.3% relative improvement), validating the robustness of our training methodology across diverse architectures.
The improvements are observed across models with different architectural designs (Qwen, In- ternVL, LLaVA), parameter seales (3B to 8B), and base capabilities, demonstrating that VISJUDGE- BENCH provides effective training signals that benefit diverse model families. Interestingly, mod- els with weaker initial performance (e.g..
1.-.-..-mistral-7777 with 0. 180 baseline correlation) achieve larger relative improvements (+236.19) (%) compared to stronger base models, suggesting that VISJUDGE- -BECH is particularly valuable for enhancing models with limited initial visualization assessment capabilities.
61

---

Published as a conference paper at ICLR 2026
E.2.4 DISCUSSION
Our generalization experiments reveal key insights. VISJUDGE substantially outperforms both 7B and 72B base models across applications, demonstrating that domain-specific fine-tuning is more effective than model scaling. The model shows strong OOD performance on MatPlotAgent's gen- erated visualizations and HAIChart's 's Plotly community data, validating that it learns generalizable quality assessment principles rather than memorizing training patterns.
The consistent improvements across diverse architectures and two distinct applications (generation and recommendation) validate VISJUDGE'S practical utility and readiness for deployment produc- tion visualization systems. These findings collectively demonstrate the value of VISJUDGE-BENC for developing robust, deployable visualization assessment capabilities.
62


Reviews:
[{"review_id": "Y3YCEATs9d", "reviewer": "Reviewer_iwcV", "summary": "This paper introduces VISJUDGE-BENCH, a comprehensive benchmark for evaluating the capabilities of Multimodal Large Language Models (MLLMs) in assessing the quality and aesthetics of data visualizations. The authors propose a \"Fidelity-Expressiveness-Aesthetics\" framework, operationalized into six measurable dimensions, to create a dataset of 3,090 expert-annotated samples. The work systematically benchmarks several state-of-the-art MLLMs, revealing significant gaps compared to human expert judgment, particularly in subjective aesthetic evaluations. To address this, the paper also presents VISJUDGE, a fine-tuned model that demonstrates substantially improved alignment with human ratings. The paper is well-written, the experiments are thorough, and the dataset is a valuable contribution to both the machine learning and visualization communities. However, the work rests on the critical assumption that a singular ground truth for visualization quality can be established, a premise that may oversimplify the inherent subjectivity and context-dependency of human perception, which is the main point of concern.", "questions": "- **Modeling Disagreement vs. Consensus:**  \n  The benchmark uses a single consolidated ground truth score. Given the known subjectivity in aesthetic and design judgments, could the authors share whether inter-annotator disagreement was analyzed before expert consolidation? It may be helpful to discuss whether such disagreement could carry meaningful signal rather than being treated purely as noise.\n\n- **Potential for Distribution-Based Evaluation:**  \n  Related to the above, have the authors considered representing human ratings as a distribution instead of a single averaged score? Capturing variance or multimodal preferences might provide a more realistic target for model alignment with diverse human judgments.\n\n- **Role of Domain Expertise:**  \n  Some visualizations (e.g., financial heatmaps with candlesticks) may be more interpretable to domain experts than to general audiences. Were annotators screened for domain familiarity, and do the authors foresee value in domain-specific subsets of the benchmark? It would be interesting to know how “ground truth” ratings might vary under domain-aware annotation.\n\n- **Deeper Analysis of Multi-View Challenges:**  \n  The performance drop on multi-view and dashboard visualizations is an important observation. A brief breakdown of typical failure modes—e.g., difficulty assessing inter-chart relationships vs. cumulative single-chart errors—would provide useful insight into where models struggle most and help guide future work.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Timely problem:**  \n   The paper addresses a relevant and underexplored challenge—evaluating MLLMs on data visualizations, which differ significantly from general images and remain difficult for current models.\n\n2. **Well-motivated framework:**  \n   The Fidelity–Expressiveness–Aesthetics framework is grounded in visualization theory and provides a clearer structure than using a single quality score.\n\n3. **Solid experiments:**  \n   The benchmarking results across multiple MLLMs offer useful insights into current model strengths, weaknesses, and issues such as score inflation.\n\n4. **Reproducibility efforts:**  \n   The release of data, code, and detailed documentation supports replicability and future work.\n\n5. **Clear presentation:**  \n   The paper is well-written, logically structured, and supported by clear visuals and appendices.", "weaknesses": "1. Fundamental Assumption of a Singular \"Ground Truth\": The primary and most significant weakness of this work stems from a fundamental assumption that a singular, objective \"ground truth\" for visualization quality exists and can be established. This is particularly contentious for subjective dimensions like Aesthetics and even Expressiveness, where human preferences are known to be diverse, context-dependent, and often lack a single consensus.\nThe paper's methodology—collecting scores from three crowdworkers and then using expert review to resolve disagreements—effectively enforces a consensus rather than acknowledging or modeling the inherent diversity in human perception. The authors do not sufficiently discuss the possibility that disagreement among annotators may not be a sign of low-quality annotation but rather a reflection of genuine, valid differences in interpretation and preference within the human population. This core assumption is critical, as its potential invalidity could undermine the significance of the benchmark's \"ground truth\" and, consequently, the meaning of the entire evaluation framework. Regrettably, because the potential invalidity of this core assumption would render all subsequent experiments meaningless, this is the main reason I cannot give a higher score, despite the authors' thorough and professional work.\n2. Domain-Specificity and Audience Dependence: A compelling, concrete example illustrates the issue above. In Figure 2 and Figure 21, the authors present a \"layered heatmap with candlesticks\" as an example of a poor visualization that MLLMs incorrectly rate highly. The authors claim it \"does not accurately present the data,\" is \"chaotic,\" and \"impossible to decipher.\" However, this visualization is a standard and highly effective tool in the cryptocurrency trading domain, known as a \"liquidation heatmap,\" which displays price action (candlesticks) alongside market liquidation levels (heatmap) (see, for example, https://www.coinglass.com/pro/futures/LiquidationHeatMap). To a domain expert (e.g., a trader), the chart is professional, intuitive, and information-dense. To a generalist audience, it is indeed confusing.\nThis example powerfully illustrates the central weakness: visualization quality is often domain-specific and audience-dependent. A \"ground truth\" established by generalist annotators may not be valid for specialized audiences. I do not criticize the authors for their lack of specific investment knowledge; rather, I use this to emphasize that finding a universal consensus on visualization quality is difficult, if not impossible. A deeper discussion of this challenge is needed.\n3. Insufficient Analysis of Complex Visualizations: The results clearly show that model performance degrades significantly on more complex visualizations like multi-view charts and dashboards compared to single-view charts. While the paper notes this drop, it lacks a deep, quantitative analysis of why this occurs. A more detailed investigation into the specific failure modes (e.g., assessing inter-chart relationships, handling information density) would have been highly valuable, especially since evaluating such complex artifacts is a major research gap that this work laudably aims to address. This feels like a missed opportunity to generate deeper insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q0VATNrCOH", "reviewer": "Reviewer_w8Z6", "summary": "VisJudge-Bench introduces a large-scale benchmark for evaluating multimodal large language models (MLLMs) on visualization aesthetics and quality assessment. It contains expert-annotated samples across diverse chart types and visualization settings. Experimental results show that existing MLLMs, including GPT-5, still fall short of human performance. To mitigate this, the authors propose VisJudge, a model trained via the GRPO method, which achieves substantial improvements in accuracy and alignment with human judgments.", "questions": "1. Annotators are mainly presented with specific questions and asked to do scoring tasks. Are they asked to do enough free-form critique? It may be beneficial to diversity and generalizability.\n\n2. What is the purpose of the experiment part?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The topic is interesting and the task is practical in real-world applications.\n2. The authors constructed VISJUDGE-BENCH, a comprehensive benchmark based on the principles of Fidelity, Expressiveness, and Aesthetics to evaluate MLLMs’ visualization assessment capabilities.\n3. They systematically evaluated representative MLLMs and found significant gaps compared with human expert standards.", "weaknesses": "1. Annotators are mainly presented with specific questions and asked to do scoring tasks. Are they asked to do enough free-form critique? It may be beneficial to diversity and generalizability.\n\n2. While the construction of the benchmark dataset is well illustrated, I am very confused about the use of it. In Section 4, authors claimed: \"To validate VisJudge-Bench as an effective training resource...\". I am confused with the authors' purpose.\n\n(1) Given that its called 'VisJudge-Bench' and referred as a benchmark in prior sections, as well as its tiny data size (only ~3k images), it is naturally a benchmark. However, authors further split it into train/test set and fintuned a MLLM on it. What is the purpose of the experiment?\n\n(2) Training a MLLM with such a tiny dataset and testing it on the dataset from the same distribution does not make much sense. The model can easily overfit and achieve good results with the certain data distribution, therefore the results are not convincing. Generalizability is a big concern. \n\nIn summary, the experiment part is confusing. As a benchmark paper, we may expect a comprehensive evaluation of MLLMs on all of the\ncollected data, instead of an experiment where the benchmark data is split into two parts for training and testing, which does not convey much useful information.\n\nOverall the topic is interesting and the construction of dataset can contribute to related studies. However, the organization of the paper is confusing. The whole experiment section is a bit off. I may accept it as a benchmark paper but not as a dataset one. The current version is not a solid ICLR submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wR1gwp9EHx", "reviewer": "Reviewer_dX6V", "summary": "This paper proposes a MLLM benchmarking framework called VISJUDGE-BENCH for visualization quality assessment, aiming to address the notable shortcomings of existing models in evaluating data fidelity, information expressiveness, and aesthetic quality. The research team constructed a benchmark dataset containing 3,090 expert-annotated samples, covering three major categories (single visualizations, multiple visualizations, and dashboards) across 32 chart types. Through systematic testing, it was found that the current state-of-the-art MLLMs exhibit significant biases when assessing visualization quality. To bridge this gap, the research team developed the VISJUDGE model, which reduces the Mean Absolute Error (MAE) to 0.442 (a 19.8% reduction) and increases the correlation with human expert ratings to 0.681 (a 58.7% improvement). This study provides a standardized framework and optimization pathway for the development of automated visualization evaluation technology.", "questions": "The specific scoring criteria are only presented in the appendix, and some representative scoring criteria could be selected and incorporated into the main text.\nAll 3,090 samples of VISJUDGE-BENCH are static visualizations, and it is suggested to include dynamic samples to support the measurement of MLLM evaluation capabilities in dynamic scenarios.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality: It proposes VISJUDGE-BENCH, the first MLLM visualization evaluation benchmark covering the three dimensions of \"Data Fidelity-Expressiveness-Aesthetics\", filling the gap where existing benchmarks only focus on a single dimension. Through GRPO reinforcement learning and LoRA fine-tuning, it achieves significant alignment between MLLMs and human experts in visualization evaluation for the first time.\nQuality: The benchmark construction undergoes rigorous multi-stage screening and multi-level quality control; the experimental design adopts a unified environment, uses multiple metrics, and covers scenarios of different complexities, ensuring data reliability.\nClarity: The paper uses figures and diagrams to support its arguments, with clear logic and presentatio.\nSignificance: It addresses the issue of the lack of standardized benchmarks for MLLM visualization evaluation, providing a benchmark for subsequent model optimization.", "weaknesses": "The paper only uses 3 annotators, which may pose a risk of concentrated subjective bias, and the elimination of abnormal scores may lead to insufficient valid data volume.\nThe 7 MLLMs tested in the experiment do not cover gradient comparison of models with different parameter scales, and the only open-source model included is Qwen2.5-VL-7B, making it difficult to verify the adaptability of VISJUDGE's fine-tuning strategy and the universality of the benchmark in the open-source ecosystem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zEjgH47iD0", "reviewer": "Reviewer_jQs1", "summary": "This paper presents VISJUDGE-BENCH, a large-scale, expert-annotated benchmark for evaluating the aesthetics and quality of data visualizations using multimodal large language models (MLLMs). The dataset covers 3,090 samples across a diverse set of visualization types and is accompanied by a multi-dimensional evaluation framework (“Fidelity-Expressiveness-Aesthetics”). The authors systematically benchmark several MLLMs and introduce VISJUDGE, a fine-tuned model that shows improved alignment with human expert ratings.", "questions": "How is data fidelity assessed without access to the source data? Are there proxies or heuristics used, and what are their limitations?\nCan you provide a more systematic benchmarking of the dataset’s coverage, strengths, and weaknesses? For example, are certain chart types, styles, or domains underrepresented?\nDid you analyze annotation quality and agreement across different annotator backgrounds or regions?\nWhat are the practical limitations of using VISJUDGE-BENCH for training or evaluation in real-world visualization systems?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Addresses an Important Gap: The paper tackles the underexplored problem of automated visualization quality assessment, providing a resource that could catalyze further research in this area.\nComprehensive Dataset: VISJUDGE-BENCH is notable for its scale, diversity (single/multi/dashboard visualizations), and rigorous annotation pipeline.\nSystematic Evaluation: The authors benchmark multiple MLLMs and provide detailed analysis of their strengths and weaknesses across different evaluation dimensions.\nOpen Resources: The dataset and benchmark are made available to the community, supporting reproducibility and future work.", "weaknesses": "Data Fidelity Unclear: The paper emphasizes “fidelity” as a core evaluation dimension, but it is not clear how this is assessed given that the source data for the visualizations is not available. Without access to the underlying data, it is difficult to objectively judge whether a visualization is faithful or misleading.\nLimited Dataset Benchmarking: As a dataset paper, there is insufficient analysis of the final dataset’s strengths and limitations. There is little discussion of what types of visualizations, design flaws, or domains are well-represented or underrepresented, and how annotation quality varies across the dataset.\nScope of Contributions: Beyond dataset curation and MLLM fine-tuning, the paper’s contributions are somewhat incremental. The model fine-tuning, while useful, is marginal given the rapid progress in adapting MLLMs to new domains.\nGeneralization and Utility: The paper does not sufficiently characterize what the dataset enables (and what it does not), nor does it discuss how VISJUDGE or the benchmark would perform in real-world or out-of-distribution scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)