Title: Co-EditBench: Human-Aligned Benchmark for Instruction-Based Image Editing with Multi-Dimensional Assessment
Primary area: datasets and benchmarks
Keywords: Image Editing; Benchmark

Abstract: Multimodal large language models (MLLMs) have made significant progress in instruction-guided image editing; however, comprehensively evaluating them in a way that aligns with human judgment remains a considerable challenge. Existing benchmarks often exhibit obvious limitations, including restricted editing types, limited evaluation dimensions, coarse perception of image details, and systematic deviation from subjective aesthetics. To overcome these issues, we proposed a more comprehensive evaluation benchmark, Co-EditBench, for human-aligned evaluation. First, we constructed a diagnostic dataset by crowd-sourcing, to obtain high-resolution, real-world image-instruction pairs covering 16 editing types. Then, to enable a fine-grained and consistent assessment, we define 11 novel evaluation dimensions that dissect “AI artifacts” into traceable visual pathologies. Additionally, we propose a comprehensive automated evaluation pipeline Co-EditEval that leverages multi-dimensional evaluators and a meticulously designed Chain of Thought for contextualized visual reasoning. Extensive experiments demonstrate that Co-EditBench provides a more reliable and nuanced evaluation than existing benchmarks, achieving a significant correlation with human judgments.

TL;DR: We present Co-EditBench, a benchmark for  image editing with over 1,000 image-instruction pairs, 11 evaluation dimensions, and a novel automatic evaluation pipeline.