EdiVal-Agent: An Object-Centric Framework for Automated, Scalable,
  Fine-Grained Evaluation of Multi-Turn Editing
Authors: name: Tianyu Chen; name: Yasi Zhang; name: Zhi Zhang; name: Peiyu Yu; name: Shu Wang; name: Zhendong Wang; name: Kevin Lin; name: Xiaofei Wang; name: Zhengyuan Yang; name: Linjie Li; name: Chung-Ching Lin; name: Jianwen Xie; name: Oscar Leong; name: Lijuan Wang; name: Ying Nian Wu; name: Mingyuan Zhou
Published: 2025-09-16 17:45:39

Abstract: Instruction-based image editing has advanced rapidly, yet reliable and
interpretable evaluation remains a bottleneck. Current protocols either (i)
depend on paired reference images -- resulting in limited coverage and
inheriting biases from prior generative models -- or (ii) rely solely on
zero-shot vision-language models (VLMs), whose prompt-based assessments of
instruction following, content consistency, and visual quality are often
imprecise.
  To address this, we introduce EdiVal-Agent, an automated, scalable, and
fine-grained evaluation framework for multi-turn instruction-based editing from
an object-centric perspective, supported by a suite of expert tools. Given an
image, EdiVal-Agent first decomposes it into semantically meaningful objects,
then synthesizes diverse, context-aware editing instructions. For evaluation,
it integrates VLMs with open-vocabulary object detectors to assess instruction
following, uses semantic-level feature extractors to evaluate content
consistency, and leverages human preference models to judge visual quality. We
show that combining VLMs with object detectors yields stronger agreement with
human judgments in instruction-following evaluation compared to using VLMs
alone and CLIP-based metrics. Furthermore, the pipeline's modular design allows
future tools to be seamlessly integrated, enhancing evaluation accuracy over
time.
  Instantiating this pipeline, we build EdiVal-Bench, a multi-turn editing
benchmark covering 9 instruction types and 11 state-of-the-art editing models
spanning autoregressive (AR) (including Nano Banana, GPT-Image-1),
flow-matching, and diffusion paradigms. We demonstrate that EdiVal-Agent can be
used to identify existing failure modes, thereby informing the development of
the next generation of editing models. Project page:
https://tianyucodings.github.io/EdiVAL-page/.