VisJudge-Bench: Aesthetics and Quality Assessment of Visualizations
Authors: name: Yupeng Xie; name: Zhiyang Zhang; name: Yifan Wu; name: Sirong Lu; name: Jiayi Zhang; name: Zhaoyang Yu; name: Jinlin Wang; name: Sirui Hong; name: Bang Liu; name: Chenglin Wu; name: Yuyu Luo
Published: 2025-10-25 17:31:02

Abstract: Visualization, a domain-specific yet widely used form of imagery, is an
effective way to turn complex datasets into intuitive insights, and its value
depends on whether data are faithfully represented, clearly communicated, and
aesthetically designed. However, evaluating visualization quality is
challenging: unlike natural images, it requires simultaneous judgment across
data encoding accuracy, information expressiveness, and visual aesthetics.
Although multimodal large language models (MLLMs) have shown promising
performance in aesthetic assessment of natural images, no systematic benchmark
exists for measuring their capabilities in evaluating visualizations. To
address this, we propose VisJudge-Bench, the first comprehensive benchmark for
evaluating MLLMs' performance in assessing visualization aesthetics and
quality. It contains 3,090 expert-annotated samples from real-world scenarios,
covering single visualizations, multiple visualizations, and dashboards across
32 chart types. Systematic testing on this benchmark reveals that even the most
advanced MLLMs (such as GPT-5) still exhibit significant gaps compared to human
experts in judgment, with a Mean Absolute Error (MAE) of 0.551 and a
correlation with human ratings of only 0.429. To address this issue, we propose
VisJudge, a model specifically designed for visualization aesthetics and
quality assessment. Experimental results demonstrate that VisJudge
significantly narrows the gap with human judgment, reducing the MAE to 0.442 (a
19.8% reduction) and increasing the consistency with human experts to 0.681 (a
58.7% improvement) compared to GPT-5. The benchmark is available at
https://github.com/HKUSTDial/VisJudgeBench.