ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code
  Generation Evaluation
Authors: name: Chenchen Zhang; name: Yuhang Li; name: Can Xu; name: Jiaheng Liu; name: Ao Liu; name: Shihui Hu; name: Dengpeng Wu; name: Guanhua Huang; name: Kejiao Li; name: Qi Yi; name: Ruibin Xiong; name: Haotian Zhu; name: Yuanxing Zhang; name: Yuhao Jiang; name: Yue Zhang; name: Zenan Xu; name: Bohui Zhai; name: Guoxiang He; name: Hebin Li; name: Jie Zhao; name: Le Zhang; name: Lingyun Tan; name: Pengyu Guo; name: Xianshu Pang; name: Yang Ruan; name: Zhifeng Zhang; name: Zhonghu Wang; name: Ziyan Xu; name: Zuopu Yin; name: Wiggin Zhou; name: Chayse Zhou; name: Fengzong Lian
Published: 2025-07-07 12:53:00

Abstract: The generative capabilities of Large Language Models (LLMs) are rapidly
expanding from static code to dynamic, interactive visual artifacts. This
progress is bottlenecked by a critical evaluation gap: established benchmarks
focus on algorithmic correctness and are blind to the visual fidelity and
interactive integrity that define modern user experiences. To bridge this gap,
we introduce ArtifactsBench, a new benchmark and paradigm for the automated,
multimodal evaluation of visual code generation. Our framework programmatically
renders each generated artifact and captures its dynamic behavior through
temporal screenshots. This visual evidence, alongside the source code, is then
assessed by a Multimodal LLM (MLLM)-as-Judge, which is rigorously guided by a
fine-grained, per-task checklist to ensure holistic and reproducible scoring.
We construct a new benchmark of 1,825 diverse tasks and evaluate over 30
leading LLMs. Our automated evaluation achieves a striking 94.4% ranking
consistency with WebDev Arena, the gold-standard for human preference in web
development, and over 90% pairwise agreement with human experts. This
establishes ArtifactsBench as the first framework to reliably automate the
assessment of human-perceived quality at scale. Our analysis provides a
high-resolution map of the current SOTA, revealing that generalist models often
outperform domain-specific ones. We open-source ArtifactsBench, including the
benchmark, evaluation harness, and baseline results at
https://artifactsbenchmark.github.io/, to provide the community with a scalable
and accurate tool to accelerate the development of user-centric generative
models.