Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing

Authors: Mingce Guo, Jingxuan He, Shengeng Tang, Zhangye Wang, Lechao Cheng
Primary area: generative models
Keywords: Text-Guided Video Editing

Abstract:
Text-driven video editing utilizing generative diffusion models has garnered significant attention due to their potential applications. However, existing approaches are constrained by the limited word embeddings provided in pre-training, which hinders nuanced editing targeting open concepts with specific attributes. Directly altering the keywords in target prompts often results in unintended disruptions to the attention mechanisms. To achieve more flexible editing easily, this work proposes an improved concept-augmented video editing approach that generates diverse and stable target videos flexibly by devising abstract conceptual pairs. Specifically, the framework involves concept-augmented textual inversion and a dual prior supervision mechanism. The former enables plug-and-play guidance of stable diffusion for video editing, effectively capturing target attributes for more stylized results. The dual prior supervision mechanism significantly enhances video stability and fidelity. Comprehensive evaluations demonstrate that our approach generates more stable and lifelike videos, outperforming state-of-the-art methods. The anonymous code is available at \href{https://anonymous.4open.science/w/STIVE-PAGE-B4D4/}{https://anonymous.4open.science/w/STIVE-PAGE-B4D4/}.

Decision: reject
Reviewer scores: 5, 3, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1729513463029

thread_type:
official_review

<thread_object>
creator: reviewer_zgwc

<content>
summary:
This paper addresses limitations in text-driven video editing using generative diffusion models, particularly the challenge of nuanced editing when constrained by pre-trained word embeddings. The authors propose a concept-augmented video editing approach that allows for flexible and stable video generation through the use of abstract conceptual pairs. The key components include concept-augmented textual inversion, which provides plug-and-play guidance for stable diffusion to better capture target attributes, and a dual prior supervision mechanism that enhances video stability and fidelity. By integrating these elements, the approach overcomes the disruptions caused by direct keyword alterations and allows for more nuanced, stylized editing. Experiments demonstrate that this method outperforms existing state-of-the-art techniques in generating stable and lifelike videos.

strengths:
1. The paper is easy to follow.
2. The clarity of the writing.
3. SCAM loss and TCAM loss sound novel.
4. Convincing metric for qualitative evaluation: Masked Peek-Signal-Noise Ratio.

weaknesses:
1. Diffusion based video editing has developed quickly and Tune-A-Video and Fatezero are not the strong baselines at this point. Plus, how is this paper different from DreamVideo [1] and MotionBooth [2] paper? I think these works could serve as better baselines?
2. In L36, Zeroscope is not a video editing method but an open-sourced video diffusion model.
3. Is there a reason why image diffusion (stable diffusion) is used as a backbone? I don't see reasons that the method can't be applied to video diffusion models and compared to video diffusion based methods.
4. L242-245, can the authors show results?
5. CLP consistency fails to reflect temporal consistency. Using other metrics like DINOv2 (like the ones from VBench[3]) or LIPIPS would be better.
6. In terms of temporal consistency, the resulting videos of MotionDirector look better.
7. The concept augmented textual inversion lacks technical contribution.

[1] DreamVideo: Composing Your Dream Videos with Customized Subject and Motion, CVPR 2024

[2] MotionBooth: Motion-Aware Customized Text-to-Video Generation, Arxiv 2024

[3] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024

questions:
Please refer to the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730435825858

thread_type:
official_review

<thread_object>
creator: reviewer_g4m5

<content>
summary:
This paper introduces a novel pipeline for video editing that leverages spatial-temporal attention and dual prior supervision. The proposed method demonstrates potential through its structure and method of concept-augmented textual inversion and self-attention blending, albeit with certain limitations in innovation and effectiveness. The ablation study provides some insights, particularly around car color changes, but overall performance falls short of expectations.

strengths:
- Comprehensive experimental setup with both ablation and comparative studies.
- Clear writing that outlines the approach and methodology effectively.

weaknesses:
- Limited innovation; the first proposed method (concept-augmented textual inversion) is not significantly novel, and the second (dual prior supervision mechanism) shows only moderate innovation.
- The quality of generated videos is subpar, with short video length and lackluster editing effects.
- Missing details on the training set used in the experiments.
- Evaluation metrics and comparison methods are unconventional and somewhat outdated, potentially impacting the rigor of performance comparisons.
- Ablation study results indicate some changes in attributes (e.g., car color) but are unimpressive in terms of visual quality and effect.

questions:
See question in weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711676157

thread_type:
official_review

<thread_object>
creator: reviewer_hlfd

<content>
summary:
This paper present a novel concept-based video editing framework. The framework includes a Concept-Augmented Textual Inversion (CATI) to extract the target concept from given video, and then use a Dual Prior Supervision (DPS) to manipulate the attention matrices to ensure the stability of output videos. For the CATI part, optimization is performed on both textual embeddings and LoRA modules added to the cross-attention layers. And for the DPS part, loss on both irrelevant parts and edited subject are used to prevent unwanted changes.

Results are conducted over several with or without concept video pairs and was compared with several methods. The overall results are good.

strengths:
1. The motivation for introducing the concept of video editing is clear and good. 
2. The results are impressive, the concepts injected are consistent, and the backgrounds are generally consistent with the target videos.
3. The paper reads well and is easy to follow.

weaknesses:
1. How authors obtain the results from comparing methods is not explained in the paper. Some of those don’t accept concept video/images input, and if not given the concept video, then direct comparison is a bit unfair. 
2. The quantitative comparison table seems to be evaluated on both with or without concept videos. And the ablation for using or not using concept videos is missing. I suggest the author having separate dataset for using or not using concept videos, and this can further strengthen the contribution.
3. Missing references and some for comparisons:
    
    *[1] AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks
    [2] GenVideo: One-shot target-image and shape aware video editing using T2I diffusion models*
    
    *[3] Video Editing via Factorized Diffusion Distillation*
    
    *[4] VidToMe: Video Token Merging for Zero-Shot Video Editing*
    
    *[5] Pix2video: Video editing using image diffusion*
    
    *[6] Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices*
    
    *[7] TokenFlow: Consistent Diffusion Features for Consistent Video Editing*
    
    *[8] Video-P2P: Video Editing with Cross-attention Control*

questions:
1. The pipeline for editing without concept video is not very clear for me, and in the code repository provided, a pretrained lora/concept model is required for all the cases including car to porsche. Can authors provide a more detailed explanation?
2. It would be interesting to know the range of concepts. More examples of diverse concepts or attributes could clarify the extent to which CATI handles generalization across different object classes and scenarios.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730908910973

thread_type:
official_review

<thread_object>
creator: reviewer_yehn

<content>
summary:
This paper proposes a text-driven video editing method using Stable Diffusion. It involves concept-augmented textual inversion and a dual prior supervision mechanism. The results show that the proposed method can outperform the baselines.

strengths:
1. The overall presentation of the paper is easy to follow. Most of the details are clear and well-documented. 

2. Based on the quantitative metrics, the model can outperform the previous baselines.

weaknesses:
1. My main concern is that the overall quality of the edited videos is not satisfactory. Mainly, there are obvious changes (shape and color) between frames, so the videos don’t look consistent. I think this may be the inherent weakness of the approach of inflating Stable Diffusion with temporal layers. The base model doesn’t have video prior knowledge compared to a video diffusion model which has been trained on video datasets. Possible ways to improve can be adding motion prior such as adopting optical flow [1] or propagating features among frames [2]. 

2. Besides the quality, the methodology is also not strong to me. Textual inversion comes with several variations, and one of them can be finetuning the model parameters instead of only finetuning the textual embedding. It seems concept-augmented textual inversion belongs to this variation by additionally finetuning V. The scam and tcam loss, although improves the results, does not seem novel or strong enough to boost the consistency of the videos. 

3. I appreciate that the authors also provide details of efficiency. I think the efficiency is also not optimal enough. More than 30 minutes (training + inference) to edit a 6-frame video seems inefficient and not practical. 

[1] Cong, Yuren, et al. "Flatten: optical flow-guided attention for consistent text-to-video editing." ICLR 2024.

[2] Geyer, Michal, et al. "Tokenflow: Consistent diffusion features for consistent video editing." ICLR 2024.

questions:
1. What are the parameters that are finetuned during the scam and tcam loss optimization? 

2. What possible reasons do you think that the edited videos are not looking consistent enough? What possible solutions could you propose?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730914389893

thread_type:
official_review

<thread_object>
creator: reviewer_ouvi

<content>
summary:
This paper introduces an improved text-driven video editing approach using generative diffusion models. It overcomes limitations of traditional methods by incorporating a concept-augmented textual inversion technique and a dual prior supervision mechanism. For the concept-augmented textual inversion, LoRA is adopted, and for the dual prior supervision mechanism, a cross-attention-based masking technique is utilized.

strengths:
* Quantitative/qualitative evaluation shows relatively improved performance.
* The two proposed methods intuitively make sense.

weaknesses:
* Writing needs improvement. It is difficult to distinguish what is being proposed and what are existing components in the method section.

* Lack of technical novelty. I think the use of LoRA with Textual Inversion (or other inversion/personalization technique) is already widely used as open-source, and compared to these, the use of LoRA in the proposed Concept-Augmented Textual Inversion does not appear to be significantly different. Also, although it is minor, it is not convincing why this technique is called Concept-Augmented Textual Inversion.

* In Dual Prior Supervision, a mask is obtained by thresholding the activated attention part corresponding to the text via cross-attention, and this is applied to the loss. This approach itself has been widely used in segmentation and image editing (e.g., masactrl) and does not appear technically novel. Even setting aside the issue of technical novelty, I have concerns about the approach of specifying areas corresponding to certain words through hard thresholding, which seems heuristic and may have a marginal effect outside situations where the object is explicitly referenced.

questions:
Please see the weaknesses.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper receives reviews of 5,3,6,3,3. The AC follows the decisions of the reviewers to reject the paper. The main concerns of this paper are: 1) The need to improve on the writing to make the technical contributions clearer. 2) The lack of novelty since the use of LoRA with Textual Inversion is already widely used as open-source. 3)  The overall quality of the edited videos is not satisfactory. There are obvious inconsistencies between the frames of the videos. 4) Evaluation metrics and comparison methods are unconventional and somewhat outdated. On the other hand, the reviewer who gave a 6 did not give any strong reasons to overrule the majority of the other reviewers who reject the paper.