Evaluation Agent: Efficient and Promptable Evaluation Framework for
  Visual Generative Models
Authors: name: Fan Zhang; name: Shulin Tian; name: Ziqi Huang; name: Yu Qiao; name: Ziwei Liu
Published: 2024-12-10 18:52:39

Abstract: Recent advancements in visual generative models have enabled high-quality
image and video generation, opening diverse applications. However, evaluating
these models often demands sampling hundreds or thousands of images or videos,
making the process computationally expensive, especially for diffusion-based
models with inherently slow sampling. Moreover, existing evaluation methods
rely on rigid pipelines that overlook specific user needs and provide numerical
results without clear explanations. In contrast, humans can quickly form
impressions of a model's capabilities by observing only a few samples. To mimic
this, we propose the Evaluation Agent framework, which employs human-like
strategies for efficient, dynamic, multi-round evaluations using only a few
samples per round, while offering detailed, user-tailored analyses. It offers
four key advantages: 1) efficiency, 2) promptable evaluation tailored to
diverse user needs, 3) explainability beyond single numerical scores, and 4)
scalability across various models and tools. Experiments show that Evaluation
Agent reduces evaluation time to 10% of traditional methods while delivering
comparable results. The Evaluation Agent framework is fully open-sourced to
advance research in visual generative models and their efficient evaluation.