IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment
Authors: Yinan Chen
Jiangning Zhang
Teng Hu
Yuxiang Zeng
Zhucun Xue
Qingdong He
Chengjie Wang
Yong Liu
Xiaobin Hu
Shuicheng YAN
Venue: ICLR 2026 Poster (2026)
Keywords: Instruction-guided video editing
Benchmark suite
Multimodal large language models
Evaluation metrics
Primary area: datasets and benchmarks
TL;DR: IVEBench is a large-scale benchmark for instruction-guided video editing with 600 videos, 7 dimensions, 8 task categories, and 35 subcategories, offering a 3D evaluation (quality, compliance, fidelity) with human-aligned, reproducible metrics.

Abstract: Instruction-guided video editing has emerged as a rapidly advancing research direction, offering new opportunities for intuitive content transformation while also posing significant challenges for systematic evaluation. Existing video editing benchmarks fail to support the evaluation of instruction-guided video editing adequately and further suffer from limited source diversity, narrow task coverage and incomplete evaluation metrics. To address above limitations, we introduce IVEBench, a modern benchmark suite specifically designed for instruction-guided video editing assessment. IVEBench comprises a diverse database of 600 high-quality source videos, spanning seven semantic dimensions, and covering video lengths ranging from 32 to 1,024 frames. It further includes 8 categories of editing tasks with 35 subcategories, whose prompts are generated and refined through large language models and expert review. Crucially, IVEBench establishes a three-dimensional evaluation protocol encompassing video quality, instruction compliance and video fidelity, integrating both traditional metrics and multimodal large language model-based assessments. Extensive experiments demonstrate the effectiveness of IVEBench in benchmarking state-of-the-art instruction-guided video editing methods, showing its ability to provide comprehensive and human-aligned evaluation outcomes. All data and code will be made publicly available.

Published as a conference paper at ICLR 2026
IVEBENCH: MODERN BENCHMARK SUITE FOR INSTRUCTION-GUDDED VIDEO EDITING ASSESSMENT
Yinan Chen Jiangning Zhangl2; Teng Hu²,
Yuxiang Zeng', Zhucun Xue! Xiaobin Hu?, Shuicheng Yan
Qingdong H Chengjie Wang Yong Liu
'Zhejiang University, Tencent Youtu Lab, "Shanghai Jiao Tong University *University of Auckland, *National University of Singapore
ABSTRACT
Instruction-guided video editing has emerged a rapidly advancing research direction, offering new opportunities for intuitive content transformation while also posing significant challenges for systematic evaluation. Existing video editing benchmarks fail to support the evaluation of instruction-guided video editing adequately and further suffer from limited source diversity, narrow task coverage and incomplete evaluation metrics. To address the above limitations, we introduce IVEBench, modern benchmark suite specifically designed for instruction-gun video editing assessment. IVEBench comprises a diverse database of 600 high- quality source videos. spanning seven semantic dimensions, and covering video lengths ranging from 32 to 1,024 frames. It further includes 8 categories of editing tasks with 35 subcategories, whose prompts are generated and refined through large language models and expert review. Crucially, IVEBench establishes a dinee-dimensioal evaluation protocol encompassing video quality, instruction compliance and video fidelity, integrating both traditional metrics and multimodal large language model-based assessments. Extensive experiments demonstrate the effectiveness of IVEBench in benchmarking state-of-the-a art cnrrucctionguidder video editing methods, showing its ability to provide comprehensive and human- aligned evaluation outcomes. All data and code will be made publicly available.
INTRODUCTION
Video editing, which aims to transform source videos to satisfy user-specified editing requirements, has emerged crucial capability in both creative industries and practical applications. As the field of generative modeling and multimodal understanding advances (Yang et al. 2024 Bai et al. 2025b), cioruction-guiden Video Editing (abbreviated as IVE that edits directed by natural language instruction) has attracted significant research interest (Cheng et al. (202)). This paradigm promises intuitive and fine-grained control over video content, unlocking new possibilities for content creation, entertainment, and human-compune interaction.
Despite rapid progress, current video editing benchmarks still present notable limitations. Existing benchmarks (Sun et al., 2025b) suffer from three major challenges: i) Insufficient diversity in video sources: The coverage semantic categories, scenes, and editing instructions remains limited, constraining the generalizability of evaluation results (Chen et al. 2025 Li et al. 2025b). ii) Restricted editing prompts: Editing instructions are often narrowly ly defined or lack granularity, failing to reflect the diverse and complex requirements of real-world editing scenarios (Wang et al., 2025a). ùi) Fragile evaluation metrics: Current evaluation protocols are frequently restricted to basic quality alignment measures, lacking comprehensive, multidimensional assessment, especially those leveraging advances in Multimodal Large Language Models (MLLMs) for semantic understanding (Sun al., 2025b) Chen al. (025). Besides, existing benchmarks primarily designed for video editing methods based on source-target prompts. However, due to their poor user-friendliness and unclear editing requirements, mainstream video editing methods have now
"qqual contribution. Corresponding author.

---

Published as a conference paper at ICLR 2026
shifted toward instruction guided approaches (Cheng et al. 2023), mirroring a similar trend in image editing (Brooks et al. 2023). Therefore, there is an urgent need for a comprehensive benchmark that fully supports IVE.
IVEEBenc Database
IVE Models
VE-Bench
EditBoard
IVEBench (Ours)
Subject
Theme
Style Editing
Subject Editing
Source
Edit Prompt
Target
Attribute Editing
Quantity Editing
Video
Video
Motion
Perspective
Time
IVEBench Metrics
Subject Motion
Visual Effect
Editing
Editing
Video Quality
5 Metrics
Emiotion
Scene
Camera Motion
Camera Angle
Instruction
Metrics
Editing
Editing
Compliance
600 Videos
7 Dimensions
600 Prompts
Editing Tasks
Video Fidelity
Metrics
Figure 1: Overview of our proposed IVEBench. 1) We construct diverse video corpus consisting of 600 high-quality source videos systematically organized across 7 semantic dimensions. 2) For source videos, we design carefully crafted edit prompts, covering 8 major editing task categories with 35 subcategories. 3) We establish a comprehensive three-dimensional evaluation protocol comprising 12 metries, enabling human- aliineed benchmarking of state-of of-hc----the-rtt art IVE methods.
In this paper, we propose modern benchmark suite termed IVEBench for IVE assessment, which tackles the aforementioned challenges through three key innovations: 1) Diverse video corpus: We construct a highly diverse dataset of 600 source videos, systematically collected and filtered to cover a wide range of topics across 7 semantic dimensions (see Fig. D 2) Comprehensive editing prompts: Editing tasks are designed to cover 8 categories, with prompts generated and refined via Large Language Models (LLMs) and expert review. 3) Robust evaluation metrics: We introduce a ctree-dimensional evaluation protocol encompassing video quality, instruction compliance, and video fidelity, incorporating both traditional metrics and MLLM-based assessments for richer, more objective evaluation.
We systematically demonstrate that evaluation suite exhibits high degree of alignment with human perception across all metrics. Through both qualitative and quantitative analyses of mainstream IVE methods, we provide valuable insights for the field of video editing. We will open-source the code, release the dataset, and keep track of the latest IVE methods.
2 RELATED WORK
eestruction-guidee video editing. In recent years, the rapid advancement of image editing tech- nologies has laid a solid foundation for video editing tasks. As the demand for understanding and generating higher-dimensional content increases, research focus has gradually shifted from static image editing to dynamic video editing (Wu et al. (023). Early video editing methods are initially influenced by inversion techniques in the image editing domain (mainly DDIM Inversion (Song et al. 2021)), leading to the development of numerous source-target prompt- basep based editing approaches (Qi al. 2023 Ceylan al. 2023 Li et al. 2024). Although these approaches
accurately preserve
object locations and poses during the inversion process (Geyer et al. 2023 Jeong & Ye 2023 Cong
al. 2023), they are inherently limited when it comes to editing tasks involving subject movement or camera motion (Yatim et al. 2024 Kara et al.) 2024). Furthermore, rather than providing detailed target prompts, users tend to express their editing requirements through instructions (Cheng et al.) 2023). Given these limitations, IVE methods have gained burgeoning attention in the industry due to their greater uue-friendllness and adaptability to diverse editing needs (Cheng al. 2023 Main- stream approaches typically combine InstructPix2Pix (Brooks et al. 2023) for first-frame editing and then leverage generative models to propagate the modifications across the entire video (Khachatryan)
al. 2023 Ku et al. 2024 Liu et al.] 2024a). To overcome the quality bottleneck in such pipelines, Ditto (Bai et al. 2025a) further introduces a scalable synthetic data generation pipeline, leveraging high-quality image editors and in- cn-context video generators construct massive high-fidelity editing pairs. In contrast to these paradigms, Ins V2V (Cheng et al., (2023) retrains the model on synthetic triplets of input video, editing instruction, and target video, enabling direct learning of instruction-
2

---

Published as a conference paper at ICLR 2026
Part I Diverse Video Collection
Part II Comprehensive Editing Prompt
Dimensions
Butject
High Quality Video Source
Videos
Source Prompts
Motion
Emotion
Pron TEOP
Manual fiterin
Diversified Editing Tasks
LUM
Manual correction
Perspactive
Time
Theme
OpenHumanvi
Ultatiide BI
Source Videos
Attributo Editing
Style Editing
Quantity Editing
Editing
Target pramppt: The sidea dhoeccior the sejosic Anch of Paxce is Mills, lats ander
precpp
Thesitr devecaco mejor Arch e hats clor The enriggan and maaus by vedi-nanaacd grace lncs
Subject Motion
Visual Effect
dle bhacoky sods gread stracaaee
with
Editing
Editing
musteeand grom There lii Badi
High Quality Video Caption [Source Prompts)
MLLM
Editing
Editing
Angle
Minds Syteraabese the arck. The scate
Lurrst planme w--.
Iarnet squa
Figure 2: Data acquisition and processing pipeline of IVEBench includes: 1) Curation process to 600 high-quality diverse videos. 2) Well-designed pipeline for comprehensive editing prompts.
driven video modification for consistent long video editing. Building this, InsV (Wu et al.
2025) further adopts multi-stage training on Cog CogVideoX-2B ( (Yang et al. 2024) and supports static
video editing tasks involving camera motion. Lucy-Edit (Team 2025) employs rectified-llow
framework with channel concatenation to achieve high-fidelity editing without requiring masks or auxiliary inputs. Subsequently, ICVE (Liao et al. 2025) introduces low-cost pretraining strategy leveraging in-context learning from unpaired video clips to learn general editing concepts. Recently, there is a growing trend towards unifying video understanding, generation, and editing within a single framework by integrating MLLMs with Diffusion Transformers (DiTs). Omni-Video (Tan) al. 2025) and Uni Video (Wei et al., 2025) propose dual-stream or unified architectures where the MLLM handles complex multimodal instruction understanding and guides the DiT for consistent video generation and editing. Based on this, InstructX (Mou et al., 2025) further utilizing mixed image-video training to transfer robust editing capabilities to the video domain.
Video editing benchmarks. With the introduction of benchmarks such as VBench (Huang et al. 2024) and T2V-CompBench (Sun al. 2025a), the evaluation systems in the field of video generation have become increasingly comprehensive. Concurrently, video editing has also garnered sigmificant attention. leading the recent emergence of dedicated benehmarks for text-driven video editing. Among them, VE-Bench (Sun et al.] 2025b) and EditBoard (Chen et al. (025) introduce dedicated datasets and evaluation systems for text-driven video editing, partially covering editing tasks of subject, style and attribute editing. Building on these foundations, FiVE (Li et al.) (0255) and TDVE-Assessor (Wang et al., (202aa) further push evaluation by proposing MLLM-based metrics that enhance the objectivity of evaluation. However, significant limitation of these existing benchmarks is that they are designed to support source-target prompt- based editing methods, while offering no or only partial support for IVE methods (Sun et al. 2025b Chen et al. (025). Furthermore, these benchmarks are constrained by limited dataset sizes, narrow content coverage, and include only a small subset of editing task types (Li et al. (025b; Wang et al. (022a)). To address these issues, propose IVEBench, a thorough benchmark suite specifically designed for IVE methods.
3 IVEBENCH DATABASE
3.1 DIVERSE VIDEO COLLECTION FOR IVE
Video data source. To ensure the comprehensiveness of our benchmark for video editing evaluation, we first expand the semantic coverage of source videos. Specifically, we define seven semantic dimensions and further subdivide each dimension into multiple fine-grained topics, resulting total of 30 topics. These subdivisions form diverse set of semantic requirements for source videos, as illustrated in Fig. (b). Based on these requirements, we manually collect high-quality video samples (22K) on Pexels (Ingo et al., 2014) and Mixkit (Ta eed & Assi, 2019), well some from open- source VitraVideo (Xue et al. 2025). In addition, we incorporate subset from OpenHuman Vid (Li al. a) dataset to further enhance the quantity and diversity of human-centric vidcos (see Fig, 2
Hybrid automated and manual filtering. All candidate videos undergo a two-stage processing pipeline. In the automatic preprocessing stage, black borders, subtitles, and low-quality content are removed. Subsequently, during the manual screening stage, we further ensure that the video content is suitable for editing and capable of covering a wide spectrum of tasks ranging from simple to complex. Ultimately, we construct a source video dataset comprising 600 videos with comprehensive semantic coverage, high resolution, and varied frame lengths. The dataset is organized into two
3

---

Published as a conference paper at ICLR 2026
edferenee
devecntion elick
orpeie caading
whelt
eroe
Distribution
Shore
Belice Strne
Oaler Thumes
Suppess
Emolode
Stewats
ang Nofforraiie
151
Su Bagens
(c) Video Distribution
(d) Video Resolution Distribution
ACTOTTE
add camera a angle
Canere alevonnenns
Human AcCo
convert
an untma
Time
transform
RACAAA StarpiEE avia
Cavess Angien
Orient maaaan
Abtinal
qrrencnn Demil
show appl's
remove
réplace
rigHt side
Serposd
tooe Vaw
black white
change low mixel
scale
Mallsall- you
Video style
13
15
[b) Video Topic Distribution
[e] Edit Prompt Word Length
(1) Edit Prompt Word Cloud
Figure Statistical distributions of IVEBench.
subsets according to frame count: i) the short subset contains 400 videos ranging from 32 128 frames. ii) the long subset includes 200 videos ranging from 129 to 1,024 frames, representing higher standard for long-sequence evaluation.
Structural video caption. After obtaining the high-quality source videos, we employ Qwen2.5- VL-72B (Bai et al.] 2025b) to generate captions of appropriate length for each video, capturing key aspects such as subjects, backgrounds, subject actions, emotional atmosphere, visual styles, as well as camera perspectives and movements. These annotated attributes are designed form a structured vocabulary of editable elements, establishing robust foundation for subsequent user-driven modification requests.
3.2 COMPREHENSIVE IVE PROMPT GENERATION
Diversified editing objectives. To ensure comprehensive coverage of task types in our benchmark for video editing evaluation, categorize the editing prompts into eight major classes. Each of these main categories is further subdivided into more fine-grained subcategories, resulting in a total of 35 subcategories, illustrated in Fig. 3 (a). Together, these eight categories encompass the full range of current requirements for IVE tasks and effectively address the limitations of existing benchmarks in terms of task coverage.
LLM-assisted prompt generation and selection. For each source video, we employ Doubao-1.55 pro (Seed al. 2025). together with previously obtained detailed captions, automatically select the most suitable editing category and generate corresponding editing prompt. In addition, the system simultaneously produces the associated target prompt and target phrase, which serve as references for subsequent evaluation metrics. This design ensures that our benchmark can also accommodate text-driven video editing methods. All editing categories and prompts are further manually reviewed and refined to guarantee balanced category distribution well as clear and reasonable prompts.
COMPREHENSIVE METRICS OF IVEBENCH
In the context of IVE tasks, we define a video editing instance as comprising three data elements: the source video, the edit prompt (i.e., the editing instruction expressed in natural language), and the target video. Based on the relationships among these elements, we evaluate the target video along three dimensions: 1) Video Quality focuses on the quality of the target video itself; 2) Instruction Compliance focuses on the alignment between the edit prompt and the target video; 3) Fidelity focuses the consistency between the source video and the target video. Notably, the dimensions
4

---

Published as a conference paper at ICLR 2026
of Video Quality and Instruction Compliance also applicable as evaluation criteria in video generation tasks, whereas Fidelity is a dimension specific to video editing.
4.1 VIDEO QUALITY
Since a video is essentially composed of a sequence of image frames arranged in chronological order, video quality can be subdivided into two aspects: temporal quality and spatial quality. Temporal quality focuses the consistency and continuity between consecutive video frames, while spatial quality emphasizes aspects such as aesthetic value, image sharpness and the naturalness of the content.
Subject Consistency (SC). For the subjects in video, we assess whether their appearance remains consistent throughout the sequence by computing the cross-frame similarity of DINO (Caron et al. 2021 feature, which serves to evaluate different models' capability in maintaining subject consistency,
Background Consistency (BC). For the video's 's overall background, we evaluate the temporal consistency of the background scene by computing the cross-frame similarity of the CLIP (Radford) et al. 2021 feature.
Temporal Flickering (TF). We observe that videos produced by many editing models exhibit temporal flickering. Accordingly, we quantify temporal flicker by sampling frames and computing the mean absolute difference across frames.
Motion Smoothness (MS). Motion smoothness is utilized to evaluate the continuity and naturalness of subject or camera movements. Under normal cireumstances, video should be free from jitter and unnatural acceleration variations. We adopt the motion priors from the video frame interpolation model (Li et al., 2023) assess the smoothness of motion the edited videos.
Video Training Suitability Score (VTSS) (Wang et al. 2025b) is the output of supervised model trained on muman-annotated data. It integrates indicators such as compositional coherence, aesthetic quality, image sharpness, color saturation, content naturalness, and motion stability, thereby enabling a comprehensive assessment of a video's 's spatial quality.
4.2 INSTRUCTION COMPLIANCE
Instruction compliance is used to evaluate whether the generated target video correctly fulfills the requirements specified in the editing prompt, and whether it is semantically aligned with the target prompt. In addition to general metrics and task-specific criteria for different editing tasks, we further employ MI.LM to assist in assessing the semantic consistency hetween the video content and the editing instructions, thereby enhancing the comprehensiveness and objectivity of the evaluation.
Overall Semantic Consistency (OSC). Global semantic consistency is used to holistically evaluate the semantic correspondence between the target video's content and the instruction's intent, with an emphasis on the overall scene. Therefore, we employ VideoCLIP-XL2 (Wang et al. 2024) to compute the semantic similarity between the target video and the target prompt.
Phrase Semantic Consistency (PSC). Phrase-level editing adherence is used to assess whether the specific phrases or operations in the instruction are accurately reflected in the target video, with greater emphasis on the edited subject. Accordingly, we employ VideoCLIP-XL2 (Wang et al. 2024) to compute the semantic similarity between the target video and the target phrase.
Instruction Satisfaction (IS). Since tasks such as subject motion editing, camera motion editing and camera angle editing are difficult to evaluate accurately using traditional methods, we employ Qwen2.5-VI (Bai et al. 2025b) to assist in determining whether the target video has faithfully executed the edit prompt. Specifically, we input both the edit prompt and the target video into the model, instructing it to assign score on five-point scale indicate the accuracy of execution. Furthermore, we provide detailed descriptions for each score level to ensure that the model maintains consistent evaluation criteria across multiple rounds of assessment.
Quantity Accuracy (QA). Quantity correctness is a metric specifically designed for quality editing tasks. This metric uses the target span as input to Grounding DINO (Lu et al.) 2024b), compares the number of detected bounding boxes with the quantity specified in the edit prompt, and assigns a score of for correctness and () for incorrectness.
5

---

Published as a conference paper at ICLR 2026
Table I: Attributes comparison with open-source video editing benchmarks. Our proposed
IVEBench boasts distinct advantages across various key dimensions.
Method
Video Collection
Prompt Type
Evaluation Metrics
Year
Video Count
Prompi Count
Quantity Editiing
Subject Motion
Edlting
Camera Editing (Motion and Angle)
Visual Elliet Editing
Instruction Compliance
Video Fidelity
MILMM
Bench
1788
2025
EditBoard
.0
2025
VACE: Benchmark
240
480
x
2025
FIVE
420
2025
TDVE-Asesoer
2025
IV KeHehch
600
600
2025
4.3
VIDEO FIDELITY
Fidelity is utilized to assess whether the target video retains the unedited portions of the source video, thereby ensuring that the editing process does not introduce irrelevant alterations. In addition to devising conventional metrics from the perspectives of motion and semantics, we further leverage MLLM to assess the content fidelity of the target video, enhancing the robustness of the metric on more challenging tasks.
Semantic Fidelity (SF). To quantify the degree semantic preservation in the target video, we employ VideoCLIP-XIL (Wang al. 2024) to compute the feature similarity between the source and target videos.
Motion Fidelity (MF), Existing video motion detection often relies optical flow, but it struggles with occlusions. Therefore, we employ Cotracker3 (Karaev et al. 2024), which is capable of handling occlusions, for extracting reliable motion trajectories. The details of the trajectory similarity computation are provided in Appendix
Content Fidelity (CF). For tasks such camera movement editing, camera angle editing and transition editing, the same subject may display different orientations due to variations in perspective, which makes difficult for traditional metrics adequately capture content preservation. To address this limitation, we use Qwen2.5-VL (Bai et al. 2025b) to assist in evaluating whether the target video correctly retains those elements that should remain unedited. Specifically, we input the source prompt, the edit prompt, and the target video into the model, instructing it to assign a score on five-point scale reflecting the fidelity of the unedited content. In addition, we provide detailed descriptions for each score level to ensure that the model adheres to consistent evaluation standards across multiple rounds of assessment.
Among these 12 metrics, SC, BC, TF, and MS are adopted from VBench (Huang et al. 2024). We conducted independence tests on the remaining 8 metrics (detailed Appendix G). demonstrating that the IVEBench metrics exhibit a high degree of independence.
4.4 HUMAN ALIGNMENT FOR BENCHMARK VALIDATION
We select three video editing models {A, B, C} and provide 30 source videos with corresponding editing instructions. For a given source video v; and its editing instruction ps, each selected video editing model produces an edited video, resulting in a set G; (VLA, Vi,,, Vi.c.. Within each set, the generated videos are compared in pairs, yielding C² = 3 pairwise comparisons. For each evaluation dimension, we prepare detailed guidelines and illustrative examples, and participants receive prior training to ensure clear understanding of the dimension definitions. In every pairwise comparison, human annotators are instructed to evaluate the videos exclusively with respect to the specified metric (see Fig. 4. and to subjectively judge which video performs better in that dimension, or to mark the pair as "hard to distinguish." We recruit 30 participants to conduct the human annotation. The conclusions of this experiment will be presented in Sec. 6.2] with further details provided in Appendix
4.5 UNIFIED SCORING FOR BENCHMARK ASSESSMENT
Each evaluation dimension comprises multiple metrics. To assess their relative importance, trained annotators rate the contribution of each metric and dimension. The average ratings rounded to the nearest integer and used as weights in the scoring formulas. Detailed formulas for dimensions and total score provided in Appendix

---

Published as a conference paper at ICLR 2026
Table 2: Performance comparison of different video editing methods on our benchmark..Hiherr values indicate better performance. denotes that certain high-frame videos fail during inference due to out-of out-of-memory issues. t denotes that the method has fixed maximum frame number, which is lower than the maximum length of the source videos.
Dimension Performance
Metric Performance
Dutabase
Method
Total Score
Video Quality
Instruction Compliance
Video Fidelity
SC
TF
OSC
PSC
Ins2V
0.67
0.80
0.39
0.82
0.91
0.96
0.97
0.97
0.045
0.24
0.23
3.10
0.30
0.95
0.86
4.05
AnyV2V
0.42
0.59
0.89
0.22
2.75
SuaieVVV
0.51
0.69
0.11
0.41
0.85
0.92
0.96
0.96
0.019
0.20
0.24
3.56
0.20
0.70
0.75
1.79
VACE:
0.50
0.95
0.23
0.97
4.03
Short
Lucy-Edit-De Edit-Dev
0.64
0.82
0.31
0.75
0.95
0.96
0.98
0.99
0.051
0.24
0.22
2,84
0.20
0.93
0.68
3.83
Omni- Video*
0.96
0.81
2.85
ICVE:
0.60
0.71
0.15
0.61
0.95
0.97
0.99
1.00
0.017
0.23
0.23
3.62
0.30
0.85
0.16
3.55
Dinaf
0.96
InsV2V
0.66
0.50
0.37
0,79
0,90
0.94
0.98
0.98
0.048
0.24
0.23
3.10
0.20
0.95
0.68
4.13
V2yt
0.72
0.36
0.57
0.97
0.80
2.65
0.51
0..99
0.42
0.41
0.83
0.91
0.96
0.96
0.021
0.23
0.23
3.45
0.25
0.70
0.77
1.79
Long
VACET
0.27
0.78
0.92
0.96
0.24
0.96
0.96
3.74
Lucy Edit Dev
0.65
0.82
0.32
0.81
0.91
0.95
0.98
0.99
0.053
0.24
0.22
2.65
0.20
0.97
0.73
4.13
Omni- Videa*
0.42
0.51
0.94
0.97
0.22
0.81
2.59
ICVE
0.59
10.72
0.40
1164
0.95
0.97
0.99
1,00
0.019
0.23
0.23
3.63
0.00
0.86
0.48
3.48
Dintf
0.78
0.48
0.72
0.96
0.97
0.23
0.24
3.93
0.86
3.69
DISCUSSION WITH RECENT VIDEO EDITING BENCHMARKS
Existing video editing benchmarks are primarily designed for source-target prompt-based methods, and they either fail to support or can only minimally accommodate IVE methods (Li et al., (025b). As summarized in Tab. [] these benchmarks exhibit clear limitations dataset scale and coverage. More critically, their prompt design largely remains confined to image editing types (subject editing, attribute editing, or style editing) without dedicated task formulations that address the temporal nature of video. In contrast, IVEBench provides comprchensive and instruction-cent centeree evaluation suite that introduces three substantial advances: 1) A large-scale dataset of 600 videos, covering 35 topics across 7 dimensions, with lengths ranging from 32 to 1024 frames, organized into short and long subsets to enhance source diversity and semantic coverage. 2) Full coverage of eight major categories and thirty-five subcategories of editing tasks, including those that explicitly leverage the unique properties of video, spanning different levels of granularity as well as tasks involving both single and multiple subjects. 3) MLLM based metrics specifically designed for Instruction Compliance and Video Fidelity, coupled with annottaed weightings and dimensions scoring formulas. These innovations enable IVEBench surpass existing benchmarks in video collection, task coverage, and evaluation methodology, thereby establishing a systematic and practically relevant standard for IVE.
BENCHMARKING VIDEO EDITING METHOD IN IVEBENCH
6.1 EXPERIMENTAL SETUP
We evaluate state-of-the-an art IVE models Ins V2V (Cheng et al. 2023), Vny (Ku et al. 2024)
and V2V (Liu et al. 2024a), Lucy-Edit-Des Dev (Team) 2025), Omni-Video (Tan et al.) 2025).
ICVE (Liao et al.] 2025). Ditto (Bai et al. 2025a) as well as the multi-conditional video editing model VACE (Jiang et al. 2025) using IVEBench, all employed with their official implementations and pretrained weights. Evaluations are conducted on the IVEBench Database. Model-specific configurations, hardware requirements, treatment of failure cases, and evaluation details are described in Appendix
6.2 BENCIMARKING STATE-OF-TIEE-ART METHOOSS ON IVEBENCH
Quantitative analysis. From the numerical results in Tab. 2 and Tab. 3 as well as the visualizations in Fig. 4 it can be observed that eight evaluated methods demonstrate relatively good frame-to to frame consistency. However, the frrmframe image quality remains unsatisfactory, which consequently leads to low Video Fidelity scores. Moreover, these methods achieve very limited performance in instruction adherence, primarily due to the narrow range of task types they support. Among
7

---

Published as a conference paper at ICLR 2026
Smaodhress
Smastiness
Plckering
cooking
Semariec
SondiRency
Semarie
Canciwny
Cerren
SLiNEtt Cormiitency
Sudject
Intrudior
Centent
Hutrudarr
Contert
Quantly
Moben
Tidel
Fiielly
Fise
(a) Metric Score
Short Subset
Laty--di DDvv
VACE
Omni Video
StabweVVV
(b) Metric Score in Long Subset
0.50
0,00
0000
Quality
Complance
Fidely
Total Score
0.00
Quality
Compliane
Fadcliy
Total Soore
(c) Dimension Score Subset
(d) Dimension Score in Long Subset
Figure 4: IVEBench Evaluation Results Video Editing Models. visualize the evaluation results of eight IVE models in 12 IVEBench metrics. We normalize the results per dimension for clearer comparisons. For comprehensive numerical results, please refer to Tab.
exhibits the best performance in editing speed. Ditto (Bai et al. 2025a and Ins V2V (Cheng al.) 2023) demonstrate the best performance in terms of editing capability, as they can execute editing prompts while preserving visual integrity. Furthermore, since Ditto successfully handles a larger number of cases, achieves the best performance Instruction Compliance, Nevertheless, these models achieve Total Score of no more than 0.7 and an Instruction Compliance score of no more than 0.5, indicating that existing IVE methods still have substantial room for improvement in overall editing capability, particularly in Instruction Compliance.
them, Lucy-Edit-Dev Team 2025)
Qualitative analysis. As illustrated in Fig. 5 the outputs of different models reveal consistent weaknesses across multiple editing scenarios. First, all models tend to introduce inaccurate lo- calization of the desired edit, leading visible artifacts such as geometric distortion, semantic bleeding, semantic collapsing, boundary blur- ring, and texture flickering. These artifacts sig- nificantly compromise the per-frame visual qual- ity of edited videos, which in turn diminishes their overall fidelity. Second, when observing more challenging editing types, such as subject motion editing and camera angle editing, we find that the editing capability of current models is particularly limited, underscoring an urgent need for broader task coverage in future devel- opment. Moreover, the models show distinct behavioral patterns: StableV2V (Liu et al., (2024a) often applies overly aggressive modifications that satisfy the editing prompt but neglect the preservation of unedited content; Ins (Cheng et al.) 2023), in contrast, tends to adopt a conservative strategy, retaining much of the source content when dealing with unfamiliar instructions; while VACE (Jiang al. 2025). not being native IVE model, frequently fails to properly execute the given edits, resulting in weak compliance with the prompts. These qualitative findings highlight that improving fer-frame image fidelity and expanding editing
Table 3: Inference efficiency and resolution.
Time per Framej
Max Memory:
Video
Resolution
InsV2V
3.96s
1288GBB
512x512
V2V
11666s
27.37GB
512x512
StableV2V
3,90s
28.31GB
512x512
Short
VACE#
27.03s
122,18GB
1280x
Lacy- Sddy--iit-De
Omni-Videoo
1,80s
36377G
640x 352
ICVE#
5.05s
88.33GB
384x 240
Diuto
InsV2V
4.05c
13448GB
512x512
Any V2y!
11.47s
63.15GB
512x512
StableV2y!
3.72s
49,82GB
512x512
Long
VACE#
51,00s
132.90GB
1280 720
Lacy-Bdit-Dev
2.23s
Omni- Video!
2.04s
36.37GB
640x: 352
ICVE#
5.28s
91.67GB
384x
Dintol
20.15s
38.86GB
832xx480
8

---

Published as a conference paper at ICLR 2026
Camera Angle High angle
Change the view to high angle.
Replace the water bottle with newspaper.
Subject Replace subjject
Style Low-poly
Convert video to low poly style
Make the parents and Tive children clap their hands.
Subject Motion Multi
Category
Edit Prompt
Source Video
Ins
AnyV2V
StableV2V
VACE
Figure 5: Qualitative comparison of selected IVE methods.
versatility are essential directions for advancing IVE models. More detailed qualitative comparisons and analyses can be found in Appendix E
Table 4: Spearman's Rho (p) across different metrics. These scores show that IVEBench metrics
are highly aligned with human judgments.
Video Quality
Instruction Compliance
Video Fidelity
SC
BC
MS
VTSS
OSC
PSC
QA
SF
0
0.9536
0.9503
0.8842
0.9774
0.9985
0.7210
0.8316
0.9834
0.8104
0.9453
0.9371
0.9892
Human alignment results. To validate that our evaluation metrics align with human perception, described in Sec. 4.4 we conduct human annotations for each metric. In pairwise model comparisons, the preferred model assigned score of 1, while the other receives 0. If annotators express no preference, both models are assigned 0.5. For each metric, a model's final human score is computed as the total score divided by the number of comparisons. We then calculate Spearman's 's rank correlation coefficient between these human scores and the automatic evaluation metric scores. The results in Tab. demonstrate that our proposed evaluation metrics exhibit a high degree of consistency with human preferences. Furthermore, to ensure the inter-rater reliability of the evaluation, we calculated Fleiss' Kappa for our human annotations and obtained score of 0.78, which indicates 'substantial agreement' according Landis and Koch (Landis & Koch 1977). This demonstrates that our rigorous calibration process effectively aligned the standards different annotators.
6.3 INSIGHTS AND DISCUSSIONS
High frame-to-frane consistency, weak single-frame quality. Across models, Trame-to-frame consistency is generally well preserved, with limited temporal flickering. However, the quality of individual frames often shows frequent visible artifacts such semantic bleeding, boundary blurring and texture flickering. These issues also lead to noticeable degradation in Video Fidelity, highlighting the necessity for future work to develop effective strategies to mitigate such artifacts.
Limited support for diverse editing prompt types. Models perform poorly in the Compliance dimension mainly because they only handle few basic editing types reasonably well, namely subject editing, style editing and attribution editing. In contrast, they lack the capacity execute more advanced editing types such as quantity editing, subject motion editing, visual effect editing, camera motion editing and camera angle editing. This leads to consistently low scores across all compliance-relant metrics. Future video editing models should therefore place greater emphasis on broadening the range of supported editing prompts.
Intrinsic limitations of frst-frame-baseed editing models. Video editing, unlike image editing, requires maintaining temporal coherence, which introduces the need modify middle later frames

---

Published as a conference paper at ICLR 2026
of video. For example, to handle transitions to insert intermediate events. These tasks do not originate from modifications the initial frames but instead focus on transformations that occur later in the sequence. First-frame-baste models, however, propagate changes from the beginning throughout the entire video, making them inadequate for such editing requirements.
Scalability to long video sequences. A critical challenge in IVE lies in handling long sequences with hundreds or even thousands of frames. Most existing methods, especially those relying on frame-wise diffusion first-frame propagation, exhibit near-linear growth in GPU memory consumption and latency as sequence length increases, making them impractical for videos beyond 128 frames. In contrast, InsV2V (Cheng et al. 2023) demonstrates superior scalability by adopting a chunked inference strategy with latent overlap, where only a limited set of reference frames is preserved across segments. This design effectively constrains memory growth while maintaining temporal continuity.
Resolution limitations. Existing IVE methods, typically operate at 512x512 or 832x480 resolution, which is far below the standard of real-world user content. The multi-conditional video editing model VACE (Jiang et al. (2025) can support 720P outputs; however, this still falls short of the practical demand, as user videos are commonly recorded in 1080P or higher resolutions, and the expectation is that edited outputs should preserve this level of detail. The low-resolution setting limits visual fidelity, which results in artifacts such as blurred textures and edge degradation, and also reduces usability in professional media workflows.
CONCLUSION
With the rapid progress of IVE, how to systematically and comprehensively evaluate these methods become a central challenge in the field. Existing benchmarks exhibit clear limitations in terms of video source diversity, task coverage, and evaluation dimensions, making them insufficient to reliably reflect the true capabilities of current approaches or to provide meaningful guidance for subsequent rescarch. To address these issues, propose IVEBench, modern benchmarking suite designed for IVE models. IVEBench integrates a large-scale and diverse dataset, a broad range editing tasks, and multi-dimensionall evaluation protocol that leverages MLLMs and aligns closely with human perception. We expect IVEBench to play key role in the evaluation of video editing models and in advancing the development of the field.
Limitation and future work. As more IVE models are released as open source, we plan to incorporate them into IVEBench for further benchmarking and comparison. In the future, with the increase of computational power, we will also expand the scale of data used for evaluation.
ETHICS STATEMENT
This work complies with the ICLR Code Ethics. Our dataset is built exclusively from publicly available, open-licensed video sources, and all annotations were conducted with careful respect for privacy and fairness. No personally identifiable or sensitive information is included. We release both data and code promote transparency, reproducibility, and responsible stewardship of research in instruction-guide video editing. The dataset and code are released solely for the purpose of advancing academic research.
REPRODUCIBILITY STATEMENT
We have already elaborated all the models algorithms proposed, experimental configurations, and benchmarks used in the experiments in the main body or appendix of this paper. Furthermore, declare that the entire code used in this work will be released after acceptance.
REFERENCES
Qingyan Bai, Qiuyu Wang. Hao Ouyang, Yue Yu, Hanlin Wang, Wen Wang, Ka Leong Cheng, Shuailei Ma, Yanhong Zeng, Zichen Liu, et al. Scaling instruction- based video editing with high-quality synthetic dataset. arXiv preprint 2xXiv22510.15744 2025a.
10

---

Published as a conference paper at ICLR 2026
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 2025b.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Duygu Ceylan, Chun-Har P Huang, and Niloy J Mitra. Pix2video: diffusion. In ICCV, 2023.
Video editing using image
Yupeng Chen, Penglin Chen, Xiaoyu Zhang, Yixian Huang, and Qian Xie. Editboard: Towards comprehensive evaluation benchmark for text-based video editing models. In AAAI, 2025.
Jiaxin Cheng, Tianjun Xiao, and Tong He. Consistent video-to-video transfer using synthetic dataset. arXiv preprint arXiv:2311.0022 2023.
Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manuel Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flatten: optical flow-guided attention for consistent text-to-video editing. arXiv preprint arXiv:2310.05922, 2023.
Carsten F Dormann, Jane Elith, Sven Bacher, Carsten Buchmann, Gudrun Carl, Gabriel Carré, Jaime R García Marquéz, Bernd Gruber, Bruno Lafourcade, Pedro J Leitão, et al. Collinearity: review of methods to deal with and a simulation study evaluating their performance. Ecography, 36(1): 27-46. 2013.
Michal Geyer, Omer Bar- Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint 0att23377.1373. 2023.
Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, al. Vbench: Comprehensive benchmark suite for video generative models. In CVPR, 2024.
Ingo, Bruno Joseph, and Daniel Frese. Pexels videos. httpa://ww.exxes.ccom wwwp:/ppeews..ccc..ccoeooe 2014. Accessed: 2025-04-06.
Hyeonho Jeong and Jong Chul Ye. Ground-a-video: Zero-shot grounded video editing using text-to- image diffusion models. arXiv preprint arXiv:2310.01107, 2023.
Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.07598 2025.
Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M Rehg. and Pinar Yanardag. Rave: Ran- domized noise shuffling for fast and consistent video editing with diffusion models. CVPR, 2024.
Nikita Karaev, Iurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.1 1831, 2024.
Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zene zero: Text-to-image diffusion models are zero-shot video generators. In ICCV, 2023.
Max Ku, Cong Wei, Weiming Ren, Harry Yang, and Wenhu Chen. Anyv2v: A tuning-fre free framework for any video-to-video editing tasks. arXiv preprint 2xXv:24403.1466. 2024.
J Richard Landis and Gary Koch. The measurement of observer agreement for categorical data. biometrics, pp. 159-174, 1977.
Hui Li, Mingwang Xu, Yun Zhan, Shan Mu, Jiaye Li, Kaihui Cheng, Yuxuan Chen, Tan Chen, Mao Ye, Jingdong Wang, and Siyu Zhu. Openhumanvid: A large-scale high-quality dataset for enhancing human-centric video generation. In CVPR, 2025a. 2025a.
11

---

Published as a conference paper at ICLR 2026
Minghan Li, Chenxi Xie, Yichen Wu, Lei Zhang, and Mengyu Wang. Five: A fine-ge fine-grained video editing benchmark for evaluating emerging diffusion and rectified flow models. arXiv preprint arXiv:2503313664, 2025b. 2025b..
Xirui Li, Chao Ma, Xiaokang Yang, and Ming- Hsug- Yang. Vidtome: Video token merging for zero-shot video editing. CVPR, 2024.
Zhen Li, Zuo-Liang Zhu, Ling- Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. Amt: All-pairs multi-field transforms for efficient frame interpolation. In CVPR, 2023.
Xinyao Liao, Xianfang Zeng, Ziye Song, Zhoujie Fu, Gang Yu, and Guosheng Lin. In-context learning with unpaired clips for instruction-based video editing. arXiv preprint arXiv:2510. 14648, 2025.
Chang Liu, Rui Li, Kaidong Zhang. Yunwei Lan, and Dong Liu. Stablev2v: Stablizing shape consistency in video-to rideo-to-video editing. arXiv preprint arXiv:2411.11455 2024a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: marrying DINO with grounded pre-training for open-set object detection. ECCV, 2024b.
Chong Mou, Qichao Sun, Yanze Wu, Pengze Zhang, Xinghui Li, Fulong Ye, Songtao Zhao, and Qian He. Instructx: Towards unified visual editing with mllm guidance. arXiv preprinl 1rXiv:2510.08485, 2025.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang. Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In ICCV, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seedl. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXhv:2504. 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR. 2021.
Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2-compbench: A comprehensive benchmark for compositional text-to- video generation. In CVPR, 2025a.
Shangkun Sun, Xiaoyu Liang, Songlin Fan, Wenxu Gao, and Wei Gao. Ve-bench: Subjective-aligned benchmark suite for text-driven video editing quality assessment. In AAAI, 2025b.
Collis Ta ced and Hichame Assi. Mixkit. https: htt////ixxiitttt hww.:kkiit.coofeeeeeeeeeeee www.co/..eeeeeeeeeee.. www.kreee-scck--ee viiee// 2019. Accessed: 2025-04-06.
Zhiyu Tan, Hao Yang, Luozheng Qin, Jia Gong, Mengping Yang, and Hao Li. Omni-video: Democ- ratizing unified video understanding and generation. arXiv preprint arXiv:2507.06/19 2025.
DecartAl Team.
Lucy edit: Open-weight text-guided video editing.
https:
/ fpddrrjpin cloud ront. net Let/LLcyy_aaae Edit_ High__ggghhh_h High__Fidlityy_eex Text
Video_d_Vide___aaeee Editing pdf 2025. Technical Report.
Jiapeng Wang, Chengyu Wang, Kunzhe Huang, Jun Huang, and Lianwen Jin. Videoclip-xl: Advanc- ing long description understanding for video CLIP models. In EMNLP, 2024.
Juntong Wang, Jiarui Wang, Huiyu Duan, Guangtao Zhai, and Xiongkuo Min. Tdve-assessor: Benchmarking and evaluating the quality of text-driven video editing with lmms. arXiv preprint arXiv:2505. 19535, 2025a.
12

---

Published as a conference paper at ICLR 2026
Qiuheng Wang, Yukai Shi, Jiarong Ou, Rui Chen, Ke Lin, Jiahao Wang, Boyuan Jiang, Haotian Yang, Mingwu Zheng, Xin Tao, Fei Yang, Pengfei Wan, and Di Zhang. Koala-36m: A large-scale video dataset improving consisteney between fine ine-ggrined conditions and video content. In CVPR, 2025b.
Cong Wei, Quande Liu, Zixuan Ye, Qiulin Wang, Xintao Wang, Pengfei Wan, Kun Gai, and Wenhu Chen. Univideo: Unified understanding, generation, and editing for videos. arXiv preprint arXiv:2510.08377, 2025.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, 2023.
Yuhui Wu, Liyi Chen, Ruibin Li, Shihao Wang, Chenxi Xie, and Lei Zhang. fective instruction-based video editing with elaborate dataset construction. arXiv preprint 1oXiv:2503.2028: 2025.
Insvie-Im: Ef-
Zhueun Xue, Jiangning Zhang, Teng Hu, Haoyang He, Yinan Chen, Yuxuan Cai, Yabiao Wang. Chengjie Wang, Yong Liu, Xiangtai Li, et al. Ulravideo: High-quality uhd video dataset with comprehensive captions. arXiv preprint arXiv:2506. 13691, 2025.
Zhuoyi Yang. Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint Xiv:22408.06072 2024.
Danah Yatim, Rafail Fridman, Omer Bar- Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. In CVPR, 2024.
13

---

Published as a conference paper at ICLR 2026
APPENDIX
OVERVIEW
The supplementary material presents more comprehensive results of IVEBench facilitate the comparison of subsequent benchmarks:
prrovidee more detailed descriptions of edit prompt subcategories, accompanied
by concrete examples.
Appendix provides the detailed procedure for computing motion fidelity score
Appendix provides the unified scoring formulation and detailed explanations of the weighting strategy across metries and dimensions
Appendix D provides experimental details, including hardware configurations, dataset
partitioning for evaluation, model implementations and failed video IDs.
Appendix provides a detailed comparison model performance.
Appendix provides human alignment details, including annotator guideline design and annotation interface.
Appendix provides independence analysis for the metrics introduced in IVEBench.
Appendix provides information on the use LLMs.
A DESCRIPTIONS OF VARIOUS CATEGORIES OF EDIT PROMPTS
In this section, we provide detailed descriptions of all 35 subcategories of editing prompts included in IVEBench. Each subcategory is defined with its specific editing operation and supported by representative example to illustrate how the editing request expressed. The purpose of this collection ensure clarity, reproducibilityy, and comprehensive coverage of diverse instruction-guicae video editing tasks. Tab. A1 summarizes the categories, subcategories, descriptions, and corresponding examples for of reference.
Table A1: Description and example for each subcategory. We provide detailed descriptions of 35 subcategories along with corresponding examples to facilitate understanding.
Category
Subcategory
Description
Example
Apply watercolor
Convert the video to watercolor style
Style Editing
watercolor
painting style to video
Convert video to retro pixel art style
Style Editing
pixel
Convert the video to a pixel-style
Render video in anime style
Change the style of the video to anime
Style Editing
anime
American
Apply American
Transform the video into a Ameri- comic style
Style Editing
comie style
comic book style
Render video in Japanese ukiyo-e style Convert video to black-and-white tones Apply oil painting ef- fect to video
Style Editing
ukiyo-e
Convert the video style ukiyo-e
black and
Style Editing
Convert the video to black and white
white
Transform the video into an oil paint- ing style
Style Editing
oil painting
Render video in neon futuristic cyberpunk style
Convert the video to a cyberpunk style
Style Editing
cyberpunk
Apply Studio Ghibli- inspired animation style
Change the video style Ghibli style
Style Editing
Ghibli
14

---

Published as a conference paper at ICLR 2026
Category
Subeategory
Description
Example
Convert video sim- plified low-poly visu- als
Transform the video into low-poly style
Style Editing
low-poly
Change weather condi- tions in the video
Change the weather to a torrential downpour
Style Editing
weather shift
Subject Edit-
add new sub- ject
Add new subject into the video
Add a heron standing among the reeds
ing
Subject Edit-
remove exist- ing subject
Remove subject from the video
Remove the young child from the video
ing
Subject Edit-
replace exist- ing subject color adjust- ment
Replace one subject with another
Replace the grotesque creatures with friendly fairy-like beings
ing
Attribute Edit- ing
Adjust colors of video or subjects
Change the sky to deep red color
Attribute Edit-
subject scal- ing
Resize subject in the video
Scale up the man dressed in ancient Egyptian attire
ing
Attribute Edit-
position
Change subject posi- tion in the scene
Move the girl to the left side of the stone steps
ing
change
Subject Mo-
single subject
Animate or adjust one subject's motion
Make the man in the black leather jacket stand up and stretch
tion Editing
motion
Subject Mo-
multiple sub-
Animate or adjust mul- tiple subjects' motions
Make the woman and the man cry and wipe their tears with their hands
tion Editing
ject motion
Camera Mo-
Simulate camera mov- ing forward
Move the camera closer to the man in the black shirt
dolly in
tion Editing
Camera Mo-
Simulate camera mov- ing backward
Gradually move the camera away from the group of men
dolly out
tion Editing
Camera Mo-
Simulate camera fol- lowing a subject
Track the movement of the red pow- der as it falls into the bottle
tracking
tion Editing
Camera Mo-
Simulate camera mov- ing upward
Perform boom up shot on the white Toyota SUV driving up the dirt hill
boom up
tion Editing
Camera Mo-
Simulate camera cir-
Perform an arc shot around the tram as arrives the station
are shot
tion Editing
cling around subject Zoom on the video subject
Camera Mo-
Zoom in on the slice of yellow cake being lifted
zoom
tion Editing
Camera Mo-
Zoom out to show more
Gradually move the camera away to the ancient temple
zoom out
tion Editing
Camera Angle
View subject from high angle
high angle
Change the view high angle
Editing
Camera Angle
View subject from low angle
low angle
Change the view to low angle
Editing
Camera Angle
Show subject from the front
front view
Change the view to front view
Editing
Camera Angle
Show subject from the side
side view
Change the view to side view
Editing
Quantity Edit-
Increase number of subjects
Increase the number of A woman with a tattoo to 2
increase
ing
Quantity Edit-
Decrease number of subjects
decrease
Decrease the number of flowers to 1
ing
Visual Effect
Add transition be- tween video contents
A particle effect transition, the man wearing sunglasses and smiling
transition
Editing
Visual Effect
decoration ef- fect
Add decorative visual overlays
Add flame effect to the metal file
Editing
15

---

Published as a conference paper at ICLR 2026
Category
Subeategory
Description
Example
Visual Effect Editing
Add event-based ef- fects
The man turns into sand and is blown away
event effect
B MOTION FIDELITY COMPUTATION DETAILS
We describe the computation of motion fidelity between a source video and target video. Given video sequence, we sample query points
uniform grid of size g. For each query point with x E R together with visibility vector v [ef U. ..... oy) where u [0, 1] indicates whether p is visible at frame L. To compare two videos of different lengths, all trajectories are interpolated to synchronized length 7 min(Ti, T2) using linear interpolation based visible frames.
CoTracker3 (Karaev et al. 2024) outputs a trajectory x" = (x" x². ,,....,,y, xy)
Given two synchronized tracks (sP, (P) and (y9, w we compute the frame-wise position distance
de |ز v2l12.
and velocity distance
dx (1 (57 99-1)11 for t 1,
with diet dzdt. Both distances are normalized by the average spatial span of the tracks
a = (1 max min x;I12 + max 57 min mins?? 12).
with a 2 10-6. We then define normalized distances at detht /a, di di /a and convert them into similarities sPPe = 1/ + ). siat 1/(1 + d The frame-wise similarity is obtained by weighted combination
8t 0.7s 0.3s
and further weighted by visibility w; = min w The overall track similarity is
Σ sime Dim we 0,
if E w/ > 0,
S(p.9)
otherwise.
Let N1 and N2 be the numbers of valid tracks in the source and target videos. We construct similarity matrix M E XNN with M,₃ = S(p, qj). establish correspondence, we apply the Hungarian algorithm to maximize E, M,) with one-to-one mapping We discard pairs with Mc,(()) S 0.3 and compute the final motion fidelity between videos V and V/2 as
MF(V, Vz)
1 PP|
Σ M, ((()) VEP
where P = (i M, > 0.3) is the set of valid correspondences. Finally, given K video pairs, the dataset-level motion fidelity score is
1 MF
Σ (F)) (R), v(()), ).
K
igt
Here, the number of synchronized frames, x" E R2 is the 2D position of track p at time t, vg is its visibility, ded and del frame-wise distances, E [0, 1] is the frame-wise similarity, S(p, q) is the similarity two tracks, Miy the similarity matrix, and π the matching permutation given by the Hungarian algorithm.
16

---

Published as a conference paper at ICLR 2026
C
UNIFIED SCORING FORMULATION AND DETAILS
For a given evaluation dimension D, let the set of metrics be {m, ....... Mapp with corresponding weights (W1, W2; ..... Wpp J. The score for dimension D is defined as:
RD W; m,
SD
i=
In our study, the three dimensions are computed as:
Video Quality = Eiey Wz my Liev wi
Σiez W/ mi Σier Wi
Instruction Compliance
ΣiEr W; m
Video Fidelity
Σier
where V, I. and F denote the sets of metrics belonging to Video Quality, Instruction Compliance, and Video Fidelity, respectively. The overall score is obtained by treating the three dimension scores as higher-level metrics. Let the set of dimensions be D, with scores {Sy} and corresponding weights {a,}. The total score is given by:
ΣjED Oj a5. SJ ΣjED ag
Total Score
Both metric-level weights w; and dimension-level weights ay are determined from the user study: each participant provided importance ratings (0-5) for metrics and dimensions. The ratings were averaged across participants, rounded the nearest integer, and applied directly in the above formulas. According to the participants' ratings, the weight of VTSS is 5, the weights of IS and CF are 3, while the weights of other metrics are 1. Since the weights of the three dimensions are all 4, they are normalized to 1. The resulting scores for different models are reported in Tab, 2
D EXPERIMENT DETAILS
All experiments were performed on NVIDIA H20 GPUs: Ins, Any and StableV2V were single GPU, while VACE required two GPUs for 720P inputs. Each model's output video resolution followed its officially recommended setting, and the number of generated frames was matched the input sequence. When certain videos could not be processed due out-of-memory errors, the corresponding results were excluded, and the indices of these failed videos are provided in Tab. A2 Moreover, due to VACE's fixed maximum frame limit of 81 frames (Jiang al.) 2025), source videos exceeding this length were uniformly sampled to 81 frames specifically for VACE editing. The maximum frame count for ICVE (Liao et al.) 2025) is set to the officially recommended 81 frames, as exceeding this limit results in excessive GPU memory usage. Similarly, Omni-Video (Tan al. (0255) configured with maximum 17 frames prevent output noise, while Ditto (Bai et al. 2025a) is set to the officially recommended 73 frames. Both short and long subsets of the IVEBench Database were used to assess editing capability across video lengths. For each model and subset, we also recorded the average runtime per frame and the peak GPU memory consumption. Evaluation was carried out using the twelve indicators of IVEBench Metrics, organized into three dimensions, where indicator scores were first computed per task, then averaged across videos. with irrelevant indicators omitted depending on the editing type; finally, all scores were normalized before visualization.
17

---

Published as a conference paper at ICLR 2026
Table A2: Failed video count and IDs of IVE methods. We present the methods that cause GPU memory usage to exceed the capacity of single H20 card due to excessively long frame sequences in certain videos.
Method
Failed videos count
Failed video IDs
long 0001, long 0004, long 0005, long 0007,
long_0008, long_0009, long_0010, long_0011,
long_0013, long-0014, long-0015, long-0016.
long 0018, long 0020, long 0021, long 0022.
long- 0023, long-0022 0025, long 0028, long- 0029.
long_0030, long_0031, long_0032, long_0033,
long-0034, long-0035, long_0037, long-0039,
long-0042. long-0043 long 2053, long-0055
Vny
65
long-0058, long-0059, long_0060, long-0061,
long-0064, long-0068, long-0070, long-0074,
long 0075, long 0082, long 0088, long 0091,
long 00994, long 0095, long 0096, long 0098,
long_0100, long_0104, long_0113, long_0114 long-0115, long-0117, long Long_015,, long-0150,
long_0152, long_0153, long-0155, long-0159,
long-0169, long-0179, long 0000, long-0186
long_0200
long_0001, long_0004, long_0005, long-0007,
long_0008. long_0009, long_0010, long_0011,
long-0012, long-0013, long_0014, 1ong-0015,
long_0016, long-0018, long-0020, long_0021,
long_0022, long-0023, long-0025, long-0028,
long 0029, long 0030, long 0031, long 0032,
long 0033, long 0034, long 0035, long 0036,
long_0037, long_0038, long_0039, long_0040,
long-0041, long-0042, long_0043, long-0044,
long-0045 long-0047, long long long-0053,
long long-0055, long-0057, long 1059-0059,
long-0060) long-0061, long long-0067,
long long
long
Vta
102
long 0074, long 0075, long 0077, long-0079,
long_0082, long_0083, long_0088, long_0089,
long-0091, long-0094, long-0095, long-0096.
long_0097, long-0098, long-0100, long-0102,
long 0104, long 0106, long-00000 long-0110,
long-0111, long-0112, long long 14
long_0155, long.0117 long 0123, long_0124
long-01255 long-0128, 0130, long long-0150,
long-0152, long_0155, long_0159, long-0160,
long-01644 long-0168, long-0169, long-0171,
long 0173, long 0177, long 0179, long 0180,
long 0186, long 0188, long 0195, long 0197,
long_0198, long_0200
E DETAILED QUANTITATIVE COMPARISON AND ANALYSIS
In this section, we provide more detailed quantitative comparison of the evaluated models across different categories of instruction guided video editing. While the main results are summarized in Table 2 and Figure 4 of the main paper, here extend the analysis to highlight model behaviors under specific editing tasks and frame lengths. We further complement the numerical results with Fig. AT which illustrates representative editing scenarios by concatenating the first, middle, and last
18

---

Published as a conference paper at ICLR 2026
Subject Replace subject
Replace the rocky beach with candy beach
Camera Angle Front View
Change the view to front vie
Change the style of the viceo anime
Subject Motion Single subject motion
Make the stati spider-man mural dynamic swing
Camera Motion Dolly
Mova thes commena closer the man the black shirt
Quantity Decrease
Decresson the number medieval krights to
Visual Effect Transition
After wave-foam transition the small fishing boot is eaten by giant whole.
Style Anime
Transform the video into watersolor style
Source Video
InsVVV
AnyV2V
StableV2V
VACE
Source Video
InsV/V
AnyV/V
StableVZV
VACE
Figure A1: Visualization of Model Output Comparison. We concatenate the first, middle, and last frames of the video facilitate comparison of the temporal performance across different models.
frames of each generated video. This visualization helps reveal temporal dynamics and qualitative differences that may not always be fully captured by scalar metrics.
Specifically, InsV2V demonstrates relatively balanced performance across most categories, main- taining higher semantic fidelity and motion fidelity even in longer sequences. However, its conserva- tive strategy sometimes leads to under-editing, resulting in lower scores in instruction satisfaction. V2V exhibits strong Instruction Compliance in simpler style and attribute editing tasks, yet struggles under difficult editing tasks. The aggressive editing strategy of stableV2V leads to higher instruction satisfaction score, but visual inspections clearly show severe semantic bleeding and boundary artifacts when dealing with complex prompts. Finally, VACE, though not originally designed for IVE, achieves reasonable temporal smoothness and high resolution outputs; nevertheless, its restricted maximum frame length limits its applicability. and its overall performance in instruction compliance remains unsatisfactory compared to native IVE models.
19

---

Published as a conference paper at ICLR 2026
Study Control Ponel
Video Comparison 1/90
Video Information
appdee Dity
Leagory conter snpp acors
Und datio nomton
Edit Prompt
Original
EE
sat/aa
120 223
smoothness,,
Figure A2: Human Annotation Interface for Benchmark Validation. The interface presents the source video, the editing instruction, and the outputs of different models under specified evaluation dimension, enabling annotators to conduct pairwise comparisons and judge which video better satisfies the given criterion.
Taken together, these detailed results and the examples in Fig. A1 confirm that current models, while capable of maintaining frame-to-frame coherence, still fall short in faithfully executing diverse instructions and preserving high per-frame fidelity. This underscores the necessity of IVEBench in identifying fine-grained weaknesses and providing clear guidance for future methodological improvements.
F HUMAN ALIGNMENT DETAILS
To validate the alignment of IVEBench metrics, we first provided each annotator with a detailed explanation of the meaning of each metric along with illustrative examples good and poor cases, followed by additional case-based tests to ensure that the annotators fully understood the intended interpretation of the metric. Moreover, we conducted further tests confirm that the annotators focused exclusively on the designated metric during comparisons, rather than being influenced by the overall quality the videos. For ease of experimentation. we designed a dedicated annotation interface for human evaluators. The interface displays the source video, editing instruction, and the outputs of different models for direct comparison under a specified evaluation dimension. Annotators instructed to make pairwise comparisons between outputs, choosing the video that better satisfies the designated metric or marking them as indistinguishable when necessary. The design of the interface is illustrated in Fig. A2
20

---

Published as a conference paper at ICLR 2026
1.00
OSC
1.00
0.16
0.22
0.16
0.09
0.19
0.52
0.75
IS-
0.16
1.00
-0.06
0.29
-0.03
-0.13
-0.06
0.21
VTSS
0.22
-0.06
1.00
0.08
0.07
0.42
0.27
0.32
0.25
PSC
0.16
0.29
0.08
1.00
-0.28
0.02
0.06
0.05
-0.00
QA
0.09
-0.03
0.07
-0.28
1.00
0.14
0.02
-0.05
-0.25
-0.13
0.42
0.50
MF
0.19
-0.06
0.27
0.06
0.02
1.00
0.25
CF
0.52
0.21
0.32
0.05
-0.05
0.63
0.25
1.00
1.00
OSC
VTSS
PSC
MF
Figure A3: Heatmap of metric independence coefficients. We calculated the Spearman correlation coefficient matrix across the results of all the eight IVE models.
G INDEPENDENCE OF METRICS
Metric independence is crucial for a benchmark, as it helps avoid redundancy and better reveals the trade-offs among the evaluated models. To quantitatively verify the independence the 8 metrics proposed in IVEBench (distinct from SC, BC, TF, and MS, which are adopted from VBench (Huang) ctt al. 2024 we calculated the Spearman correlation coefficient matrix across the results of all models, as shown in Fig. A3 The results demonstrate that the Spearman correlation coefficients among all metrics do not exceed 0.65. This indicates the absence of collinearity (Dormann et al. 2013), supporting the independence and distinctiveness of our metric design. Additionally, negative correlations observed among several metrics suggest that IVEBench metrics can effectively reveal the trade-offs within the evaluated models.
H THE USE LARGE LANGUAGE MODELS
We large language models solely for polishing writing, and have conducted careful check, taking full responsibility for all content in this work.
21


Reviews:
[{"review_id": "h1GReWKVYT", "reviewer": "Reviewer_2aZn", "summary": "This paper presents IVEBench, a modern benchmark suite specifically designed for Instruction-guided Video Editing (IVE), aiming to address the limitations of existing benchmarks, including insufficient diversity of video sources, narrow task coverage, and incomplete evaluation metrics. The suite contains 600 high-quality source videos covering 7 semantic dimensions, with frame lengths ranging from 32 to 1024 frames, divided into a short-sequence subset (400 videos) and a long-sequence subset (200 videos). It includes 8 major categories and 35 subcategories of editing tasks, generated by large language models (LLMs) and refined by experts.\n\nIVEBench establishes a three-dimensional evaluation protocol—comprising Video Quality, Instruction Compliance, and Video Fidelity—integrating 12 metrics, including both traditional measures and MLLM-assisted assessments. In tests on four state-of-the-art IVE models, such as InsV2V and AnyV2V, IVEBench demonstrates high alignment with human perception (e.g., the VTSS metric achieves a Spearman correlation of 0.9982), and reveals existing model limitations in handling long videos (frame lengths exceeding 128 frames may cause memory overflow) and executing complex instructions (e.g., camera angle editing), providing a comprehensive standard for IVE evaluation.", "questions": "1. **On evaluation efficiency:**  \n   Given the large number of proposed evaluation metrics and assessment procedures, is there potential redundancy among them? Could the evaluation framework be simplified while maintaining its effectiveness? It would also be helpful to know the average time required to evaluate a single video under the current protocol.\n2. **On the validity of the human rating process:**  \nHow do the authors ensure that subjective human ratings align with objective quantitative metrics? Are there measures such as inter-rater reliability or calibration steps to minimize inconsistency among annotators?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. **Comprehensive and well-constructed video and prompt design:**  \n   The paper collects a dataset of 600 videos across 7 dimensions, with each video paired with a corresponding prompt covering 8 different editing tasks. The dataset is large-scale and well-curated, with quality ensured through a combination of automatic preprocessing and manual screening. This provides a reliable foundation for evaluating video editing models.\n\n2. **Thorough evaluation framework:**  \n   The study employs three primary evaluation dimensions and 12 fine-grained metrics, assessing video quality from multiple perspectives. This comprehensive approach ensures that different aspects of video editing performance are systematically measured.\n\n3. **Provides insightful analysis:**  \n   Using the proposed evaluation framework, the authors analyze existing video editing models and reveal meaningful observations, such as high frame-to-frame consistency, weak single-frame quality, and limited support for diverse editing prompt types. These insights highlight both the strengths and current limitations of state-of-the-art video editing models.", "weaknesses": "1. **Ambiguity and overlap among evaluation dimensions:**  \nAlthough the paper proposes a comprehensive set of evaluation metrics, there exists noticeable overlap among certain dimensions. For example, both *Overall Semantic Consistency (OSC)* and *Instruction Satisfaction (IS)* aim to assess semantic alignment between the edited video and the given prompt, focusing on similar aspects of overall semantic coherence. This overlap raises concerns about the independence and discriminative validity of each metric.\n\n2. **Limited reliability of human alignment experiments:**  \nThe *human alignment* evaluation is conducted using only ten source videos, which is a relatively small sample size and may limit the representativeness of the results. Furthermore, the approach of directly using human rating distributions introduces subjectivity and potential bias, since human perceptual judgments are difficult to map linearly onto quantitative scales. The authors should clarify how they ensure consistency and statistical reliability in this human evaluation process.\n3.**Limited diversity of evaluated models:**  \nThe paper reports results from only four video editing models. This limited coverage weakens the generality of the analysis and makes it difficult to assess the broader applicability of the proposed benchmark. Evaluating more diverse or state-of-the-art models would significantly strengthen the conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "THMYVkPNBb", "reviewer": "Reviewer_W9Es", "summary": "This paper presents IVEBench, a benchmark for evaluating instruction-guided video editing. It includes 600 videos with diverse content and multiple editing categories, along with a three-dimensional evaluation protocol covering video quality, instruction compliance, and video fidelity. The authors also combine traditional metrics with multimodal large language model evaluations to better capture semantic alignment between edits and instructions.", "questions": "I just wanted to ask the authors if they’ve thought about reweighting the metrics so that instruction compliance has a bigger impact on the final score?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper provides a well-structured benchmark in evaluating instruction-based video editing with diverse videos and a clear evaluation framework. The idea of combining automatic metrics with MLLM-based assessments is creative and reflects current research trends. The presentation is clear, the examples are easy to follow, and the benchmark has the potential to become a useful resource for future studies in this area.", "weaknesses": "I believe in video editing, the most crucial factor should be instruction compliance (whether the model actually does what the user asks for). From the qualitative examples, VACE achieve relatively high overall scores despite barely changing the video content (mostly just color tone adjustments). A model can still perform well even if it doesn’t really edit the content (even using the original video). This makes the benchmark a bit pointless. A weighted scoring system that puts more emphasis on instruction compliance would probably give a fairer picture of real editing performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kNiJkcDods", "reviewer": "Reviewer_84vm", "summary": "This paper introduces IVEBench, a new and modern benchmark designed to properly test today's instruction video editing models. The authors point out that existing benchmarks are falling behind; they use a limited variety of videos, cover only basic editing tasks (like changing style or swapping an object), and rely on outdated metrics that don't capture the full picture.\n\nTo tackle this, IVEBench brings three big upgrades. First, it features a diverse collection of 600 high-quality videos, spanning different themes, scenes, and lengths. Second, it includes a massive range of editing tasks: 8 categories with 35 sub-types, from simple attribute changes to complex camera movements—with prompts generated by LLMs and polished by experts. Finally, it introduces a smarter, three-dimensional evaluation system: Video Quality, Instruction Compliance, and Video Fidelity. Crucially, it leverages Multimodal LLMs to score complex edits that traditional metrics can't handle, ensuring the results are much closer to what a human would think.", "questions": "1.  The MLLM-based metrics are fantastic for evaluating complex instructions. But how do you plan to prevent the benchmark from becoming an \"echo chamber,\" where editing models are trained to please a specific MLLM evaluator rather than genuine human users?\n\n2. Edit instructions are often messy and combine multiple steps (e.g., \"Make the cat look like it's made of fire and have it jump onto the table\"). Does IVEBench have a plan to assess these kinds of compositional, multi-part edits in the future?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. A Massive and Diverse Dataset: Unlike older benchmarks with a handful of videos, IVEBench provides 600 clips, including both short and long ones (up to 1,024 frames!). This variety really pushes models to prove they can work on all kinds of content, not just cherry-picked examples.\n2. Goes Beyond Basic Edits: The benchmark covers editing tasks that are specific to video, like adding camera motion or changing subject motion. This is a huge step up from benchmarks that just treat video as a series of images.\n3. Smart, MLLM-Powered Evaluation: Using MLLMs to judge \"Instruction Satisfaction\" and \"Content Fidelity\". It allows the benchmark to automatically score tricky instructions (e.g., \"change the camera to a high angle\") that are almost impossible to measure with old-school pixel-based metrics.\n4. Strong Alignment with Human Judgment: This work show that their automated metrics are highly correlated with human preferences (with Spearman's Rho scores often above 0.9). This means the benchmark isn't just spitting out numbers—it's reflecting what people actually see as a \"good\" or \"bad\" edit.", "weaknesses": "1. The initial evaluation only tests four models. While these are representative, the benchmark's true power will be seen when it's used to compare a much wider range of open-source and commercial models over time.\n2. Potential Bias in MLLM-Based Evaluation: Relying on a specific MLLM (Qwen2.5-VL) for scoring could introduce subtle biases. For instance, if a new video editing model is built on a similar architecture to Qwen, the evaluator might unintentionally favor it. Diversifying the MLLM evaluators in the future could make the results even more robust.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l6z7zYfBBX", "reviewer": "Reviewer_xwi2", "summary": "This paper introduces IVEBench, a comprehensive benchmark suite designed to address the shortcomings of existing evaluation methods for instruction-guided video editing. The authors identify three limitations in current benchmarks: insufficient source video diversity, limited scope of editing tasks, and incomplete evaluation metrics. To overcome these challenges, this paper proposed to source more diverse videos, enriching the prompt types and introduce better metrics for evaluation, resulting in the IVEBench benchmark which contains 600 videos.", "questions": "N/A", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Main contributions:\n\n1.  A new dataset of 600 high-quality, high-resolution source videos, categorized across seven semantic dimensions and split into short (32-128 frames) and long (129-1,024 frames) subsets to test model scalability. The IVEBench includes a set of 600 editing instructions spanning eight major task categories and 35 subcategories, which go beyond simple style/subject edits to include complex temporal tasks like camera motion and subject motion editing.\n2.  This paper proposed a three-dimensional evaluation framework assessing **Video Quality**, **Instruction Compliance**, and **Video Fidelity**. This eval framework integrates traditional metrics with MLLMs-based assessments for a more fine-grained and human-aligned evaluation. The Spearman's Rho correlations (e.g., 0.98 for IS, 0.99 for VTSS, 0.94 for SF) provide strong evidence that the metrics are aligned with human perception.", "weaknesses": "1. First, the benchmark's construction creates an idealized environment that may not reflect the challenges of real-world video editing. The dataset is composed of high-quality, professional video, which does not represent the noisy in-the-wild videos that users commonly edit. Similarly, the editing prompts are not complex enough (too short and too \"clean\"), failing to test a model's ability to handle the messy prompts/ This gap between the benchmark's sanitized conditions and real-world complexity limits the generalizability of the performance scores, as models that excel on IVEBench may still falter on more realistic user-generated content and commands.\n\n2. Second, the evaluation protocol, while interesting, has weaknesses related to its reproducibility and objectivity due to its heavy reliance on specific MLLMs - Qwen2.5-VL.  The use of MLLMs in eval, while has some merits as I mentioned, still fall short in lack of explanability, creating a risk of hidden biases. I don't see the novelties of evaluating such a VE task with MLLMs.\n\n3. The benchmark's scope does not fully encompass the complexity and scale of advanced video editing tasks. Especially in terms of the quantity - 600 videos with 35 topics are not enough to cover the complex and dynamic nature of real-world videos. The authors need to clearly explain why only source 600 videos like the cost of creating such benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)