VEditBench: Holistic Benchmark for Text-Guided Video Editing

Authors: Jay Zhangjie Wu, Guian Fang, Dongrong Joe Fu, Vijay Anand Raghava Kanakagiri, Forrest Iandola, Kurt Keutzer, Wynne Hsu, Zhen Dong, Mike Zheng Shou
Primary area: datasets and benchmarks
Keywords: Benchmark, Generative Models, Video Editing

Abstract:
Video editing usually requires substantial human expertise and effort. However, recent advances in generative models have democratized this process, enabling video edits to be made using simple textual instructions. Despite this progress, the absence of a standardized and comprehensive benchmark has made it difficult to compare different methods within a common framework. To address this gap, we introduce VEditBench, a comprehensive benchmark for text-guided video editing (TGVE). VEditBench offers several key features: (1) 420 real-world videos spanning diverse categories and durations, including 300 short videos (2-4 seconds) and 120 longer videos (10-20 seconds); (2) 6 editing tasks that capture a broad range of practical editing challenges: object insertion, object removal, object swap, scene replacement, motion change, and style translation; (3) 9 evaluation dimensions to assess the semantic fidelity and visual quality of edits. We evaluate ten state-of-the-art video editing models using VEditBench, offering an in-depth analysis of their performance across metrics, tasks, and models. We hope VEditBench will provide valuable insights to the community and serve as the standard benchmark for TGVE models following its open-sourcing.

Decision: reject
Reviewer scores: 5, 5, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730261302189

thread_type:
official_review

<thread_object>
creator: reviewer_ddax

<content>
summary:
This paper presents VEditBench, a new benchmark for text-guided video editing (TGVE). VEditBench enjoys several key features, including (1) 420 real-world videos spanning diverse categories and durations, consisting of 300 short videos (2-4 seconds) and 120 longer videos (10-20 seconds); (2) defining six editing tasks that capture a broad range of practical editing challenges like object insertion, object removal, object swap, and style translation; (3) suggesting nine evaluation dimensions to assess the semantic fidelity and visual quality of edits. Furthermore, the authors evaluate ten state-of-the-art video editing models using VEditBench, and conduct an in-depth analysis of their performance across metrics, tasks, and models.

strengths:
The main strength of this paper is that the authors introduce a new benchmark for text-guided video editing by collecting 420 short and long videos, which includes more videos than previous benchmarks. The authors also suggest some metrics for evaluation and some editing tasks for experiments. This paper is also well-written, making the main text easy to follow and easy to read. The authors also conduct numerous expeirments of recent state-of-the-art methods using VEditBench, offering a comprehensive comparison and experiments.

weaknesses:
As a benchmark paper, I would expect that the authors provide open-source datasets or links to the dataset/website/codes, etc. Unfortunately, I cannot find them in this paper. I believe this is necessary and the authors should elaborate more on that.

Another concern is that this paper seems a bit incremental. Many of the metrics and tasks defined in this paper are actually come from prior works or have already being used in the community. Furthermore, the categories of the gathered videos are only restricted to six categories. The authors could consider further expanding them and including categories like clothes, plants, buildings, etc.

Furthermore, it seems that the tasks introduced by VEditBench is a bit simple. For example, as shown in Table 2 and Table 3, existing methods can achieve quite strong performance under many tasks and metrics in VEditBench.

questions:
- what do you think are the most key and novel components of your benchmark compared to previous works?
- why do you not include some long videos (say 200s) like V2VBench? Would the editing tasks and metrics still be reliable for long videos (e.g., 200s)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730445766572

thread_type:
official_review

<thread_object>
creator: reviewer_4m8x

<content>
summary:
This paper introduces a new benchmark, VEdit Bench, for evaluating text-guided video editing methods. The benchmark comprises 420 videos, including both short and long formats, and covers six distinct editing tasks. It provides a standardized evaluation framework with nine metrics that assess performance in terms of **semantic fidelity** and **visual quality**.  Furthermore, it conducts a comprehensive evaluation of existing video editing methods, providing insights into their performance across various tasks and metrics.

strengths:
1. The new VEdit benchmark includes a significantly larger dataset, with 420 videos compared to the previous LOVEU-TGVE23’s 76 videos.
2. The benchmark addresses long video editing (10-40 seconds), which is essential for real-world applications.
3. VEdit Bench introduces a novel motion similarity metric that considers both positional and directional costs, distinguishing it from the motion fidelity metric in DMT.
4. The VEdit benchmark provides a comprehensive evaluation of most existing video editing methods, primarily those based on Stable Diffusion T2I models, with the exception of DMT, which is based on a T2V model. It also offers new insights, such as the importance of balancing semantic fidelity and visual quality as a direction for future research.

weaknesses:
1. **Metric Weakness 1 (Coarse Semantic Alignment)**: The metric for semantic fidelity focuses primarily on text alignment and video alignment. However, relying solely on CLIP’s text and visual embeddings may not fully capture text alignment, as CLIP’s global representation is often coarse. The authors could enhance this by incorporating object masks, such as calculating visual-text alignment only within the foreground object’s mask region.

2. **Metric Weakness 2 (Editing Accuracy)**: For video editing methods based on T2I models, an essential issue is the model’s ability to accurately follow the instruction prompt. Therefore, **editing accuracy** should be a key metric to evaluate whether edits align with the prompt. Additionally, some methods unintentionally alter the background or other video elements even when only the object should be edited. Accurately assessing whether edits are limited to the specified target is essential.

3. **Task Weakness 1 (Multi-Object Editing)**: The benchmark lacks tasks sensitive to multiple object editing, such as multi-object video editing. For instance, VBench [1] includes specific metrics for multi-object editing, which could provide a more nuanced assessment of the model’s performance in handling complex scenes with multiple subjects.

4. **Task Weakness 2 (Subject Consistency)**: The benchmark does not adequately address subject consistency as a key quality indicator. Calculating overall motion consistency without distinguishing between foreground and background may be inadequate. Similar to VBench[1], separately evaluating subject consistency and background consistency could offer a more fine-grained metric for realistic video editing quality.

5. **Task Weakness 3 (Inadequate for T2I Models)**: The object addition and removal tasks fall under inpainting or outpainting. Current T2I models may lack the capability to perform object addition in a time-consistent manner, meaning these tasks likely require video generation models with temporal priors.

6. **Task Weakness 4 (Multi-Attribute Editing)**: The object swap task may involve simultaneous scene replacement, making it a challenging task. Editing multiple subjects or instances, also known as **multi-attribute editing**, should be considered a distinct task to evaluate complex editing capabilities.

7. **Task Weakness 5 (T2I Limitations on Motion)**: Tasks like motion change and object addition are challenging for T2I-based editing methods due to the lack of video generative priors. T2I models excel in visual quality but often lack temporal consistency, whereas T2V models offer better temporal consistency but lower aesthetic quality. Balancing the evaluation for both T2I and T2V models’ editing abilities is an important consideration.

8. **Visualization Weakness (Lack of Qualitative Demos)**: A comprehensive benchmark should include qualitative visualizations, such as video or GIF demos. VBench, for instance, visually demonstrates what a high or low temporal consistency score looks like, helping readers intuitively understand the metric. VEdit Bench, however, focuses heavily on quantitative metrics, lacking qualitative demonstrations that could help readers better grasp how current metrics align with human perception and what quality aspects they evaluate.

[1] Huang, Z. et al., "VBench: Comprehensive Benchmark Suite for Video Generative Models," CVPR, 2024.

questions:
please see weaknesses above
</content>
</thread_object>

</thread>
<thread>
creation date:
1730577499918

thread_type:
official_review

<thread_object>
creator: reviewer_edse

<content>
summary:
The authors introduce a text-guided video editing benchmark that includes a large-scale collection of videos across diverse categories, varying durations, and editing tasks grouped into six categories. Additionally, they define nine evaluation metrics to assess outputs across multiple dimensions. By addressing the limitations of previous methods, which relied on private or unreleased datasets and non-standardized evaluation pipelines, this paper seeks to standardize the evaluation of text-guided video editing. The authors also apply their benchmark to evaluate prior approaches, providing a consistent framework for comparison.

strengths:
- I find this benchmark highly valuable for the field, as I agree with the authors' observation that each method currently introduces a unique evaluation pipeline, leading to non-standardized assessments. This work provides a standardized framework for evaluating new methods, which will benefit future research.

- The categorization based on video duration enhances the benchmark, as some methods cannot handle longer videos. This categorization will be useful in differentiating capabilities among approaches.

- The benchmark also covers a wide range of evaluation types. I find the inclusion of motion similarity particularly innovative; if this metric is novel, please cite the relevant sources, as I have not encountered previous works using this measure. Motion similarity offers a meaningful assessment of how well the resulting video maintains motion consistency.

- The paper is very well presented and written.

weaknesses:
- A potential enhancement could be to further categorize object-swapping tasks into two subcategories: one for swapping objects of similar size and another for cases that necessitate substantial motion adjustments (e.g., swapping a car with a bike). These scenarios involve distinct challenges, and such categorization would allow for a more precise assessment of method capabilities.

- Additionally, incorporating GPU requirements and runtime as evaluation metrics would improve the benchmark’s comprehensiveness by enabling comparisons on computational efficiency and scalability.

- I have a question regarding the evaluation of methods with different prompt requirements. Methods such as InstructVid2Vid rely on instructional prompts, while others like TokenFlow and RAVE use target-based prompts. How does the benchmark account for these differences in prompt style? The instructional format used for all editing prompts, as indicated in Supplementary A.1, may not align with some methods that were not trained with instructional prompt styles, potentially impacting their performance.

- In Table 1, it would be beneficial to include the dataset used in RAVE, as it features longer videos categorized by frame count, providing a clear basis for evaluating duration-dependent performance.

- When discussing video duration (e.g., in lines 205 or 207), it would be more informative to also specify the frame rate (FPS), as duration in seconds alone does not provide a complete measure of the content length without this context.

- In line 402, the figure number is missing within the parentheses, which should be corrected for clarity.

questions:
Could you provide qualitative video results for each method in a downloadable format, such as a PDF or a hosted HTML link? Additionally, example prompts from the dataset would be helpful, as they are currently not visible.

Please refer to my questions in the weaknesses section, as addressing these points would allow me to reconsider and potentially increase my score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708190400

thread_type:
official_review

<thread_object>
creator: reviewer_cj9a

<content>
summary:
This paper proposes VEditBench, a comprehensive benchmark designed for the text-guided video editing (TGVE) task. It contains 420 real-world videos spanning six categories, which are further divided into short (2-4 seconds) and long (10-40 seconds) clips. VEditBench encompasses a wide range of editing tasks, such as object insertion, removal, swapping, etc. It evaluates existing state-of-the-art approaches from multiple dimensions, including semantic fidelity and visual quality, leading to insightful discussions/conclusions in TGVE.

strengths:
- The benchmark for text-guided video editing is both crucial and captivating. The work notably bridges a substantial gap, laying an important foundation for future studies of video editing.
- The paper is crafted with a coherent structure and logical flow, making it accessible and comprehensible.
- The proposed benchmark reveals the capabilities and limits of existing methods, demonstrating the trade-offs in performance across various tasks and dimensions.

weaknesses:
- The absence of visual results makes it difficult to fully assess whether the quantitative metrics align with the intended objectives or functionality.
- [Motion] Motion plays a vital role in video generation and editing, yet the proposed benchmark doesn't fully address this aspect. Is the CLIP score sensitive to the video motion (both object and camera motion) and text description? How to measure the performance of different methods in editing videos with varying levels of motion?
- [Diversity in video] The video collection lacks diversity. While it includes various categories, it's unclear if there is style diversity, such as cartoons and paintings (different levels of abstraction).
- [Diversity in task] Simultaneous multi-element editing with text isn't addressed in VEditBench. Typically, the editing requirements can be combinatorial and complicated. 
- [Metric] The structure preservation is not considered, which is common and important. Imagine that users intend to edit the motion of the main character while preserving the overall layout.
- [Metric] The efficiency of different editing methods is not considered in the benchmark.


Typo:
- Lack of figure index in L402.

questions:
- How are the starting points selected for the tracking of motion similarity metric? How significantly does this impact the evaluation?
- How can different methods be measured and compared if they only support editing videos with fixed-length frames (and distinct from each other)?
- Why do most metrics in VEditBench-long show better results compared to VEditBench-short?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730792759799

thread_type:
official_review

<thread_object>
creator: reviewer_duef

<content>
summary:
This paper introduces VEditBench, a new benchmark dataset for text-to-video editing tasks. The benchmark contains 420 real-world videos across six editing tasks. Additionally, the authors propose nine evaluation metrics to assess the semantic fidelity and visual quality of edits. They also provide a quantitative analysis of the performance of ten state-of-the-art text-to-video editing models on the newly introduced benchmark and evaluation metrics.

strengths:
+ The proposed benchmark could be valuable for advancing research in text-to-video editing.
+ The quantitative analysis provided is strong.
+ The paper is generally well-written.

weaknesses:
* The dataset collection and annotation process is not very clear 
    - How many videos were collected per category?
    - How was GPT-4o used to caption the videos? What prompts were used? It would be helpful if the authors provided a sample video with corresponding captions obtained through this process.
    - How many people participated in the manual review and revision process?
    - No sample dataset is provided as a reference.
    - Concerns regarding the copyright and continued accessibility of the curated dataset are not addressed.

* The proposed tasks and evaluation metrics are not strongly motivated

    - The proposed editing tasks are not novel, as they have been utilized in prior works. What is the purpose of presenting them as new? Why were these six tasks selected? It is important to discuss how the proposed dataset and tasks differentiate themselves from existing datasets.
    - The inclusion of all nine evaluation metrics seems redundant. Why is each metric necessary? For instance, what is the rationale for a spatio-temporal metric when individual spatial and temporal metrics are already provided?
    - The evaluation process relies heavily on numerous pretrained models, which may introduce errors and impact the reliability of the proposed metrics. Did the authors implement any measures to mitigate potential error propagation?
    - Different editing cases should ideally weigh metrics differently. For instance, spatial alignment is nearly irrelevant for a motion change task unless the caption specifically references motion (which would be rare). This is a critical consideration in designing evaluation metrics and appears to be overlooked in the paper.
    - Did the authors consider using optical flow for the motion similarity metric?


* The experimental analysis in the paper is limited and lacks depth
    - Beyond the qualitative results, what insights can we draw about the models? Why do some models perform well on one metric but poorly on another?
    - Given the limited technical contribution, the submission would benefit significantly from a detailed analysis from the perspectives of model architecture, dataset composition, and training settings.
    - The paper lacks a video analysis. Including a demo video to showcase the different quantitative results would provide a clearer understanding of the outcomes discussed.
    - The quantitative charts do not yield any meaningful insights, as the models seem to behave inconsistently across metrics. To what extent does this issue stem from the evaluation metric design itself?
    - What is the takeaway message of the paper? What are the key research directions that remain unexplored? The paper briefly mentions the need for “specialized architectures and training strategies tailored to the specific challenges of long video editing,” but this statement is vague and lacks depth.

questions:
Please refer to the questions (or issues) mentioned in the Weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733277392206

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle Reminder: Upcoming Paper Discussion Deadline

comment:
As the paper discussion deadline is quickly approaching, we would like to kindly remind reviewers who have not yet had the chance to respond to our rebuttal. We would greatly appreciate it if you could let us know whether our response has sufficiently addressed your concerns or if there are any remaining questions. Should you have any unresolved issues, we would be more than happy to actively address them!
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper discusses a new benchmark for the text-guided video editing task. Reviewers acknowledged the importance and the difficulty of the problem. Yet, all of them place the paper around or below the borderline. They listed a number of weaknesses, such as heavy reliance on existing 3rd-party models, lack of experimental analysis, lack of certain types of editing, lack of diversity. They further mention that the used metrics have been previously discussed and analyzed. The AC believes that the problem is a very nuanced and multifaceted, making it hard to be addressed by one single manuscript. Since the paper didn't get the necessary level of support from the reviewers the decision is to reject the manuscript.