Shaping a Stabilized Video by Mitigating Unintended Changes for Concept-Augmented Video Editing

## Abstract

Abstract Text-driven video editing utilizing generative diffusion models has garnered significant attention due to their potential applications. However, existing approaches are constrained by the limited word embeddings provided in pre-training, which hinders nuanced editing targeting open concepts with specific attributes. Directly altering the keywords in target prompts often results in unintended disruptions to the attention mechanisms. To achieve more flexible editing easily, this work proposes an improved concept-augmented video editing approach that generates diverse and stable target videos flexibly by devising abstract conceptual pairs.
Specifically, the framework involves concept-augmented textual inversion and a dual prior supervision mechanism. The former enables plug-and-play guidance of stable diffusion for video editing, effectively capturing target attributes for more stylized results. The dual prior supervision mechanism significantly enhances video stability and fidelity. Comprehensive evaluations demonstrate that our approach generates more stable and lifelike videos, outperforming state-of-the-art methods.

## 1 introduction

Text-driven video editing using generative diffusion models (Ho et al., ; Song et al., ; Rombach et al., ) has garnered significant attention due to its potential applications in various fields, including film production, art, and advertising (Ho et al., ; Hong et al., ; Blattmann et al., ).

Existing text-driven video editing methods based on diffusion models, such as Tune-A-Video (Wu et al., ), FateZero (Qi et al., ), Zeroscope (Sterling, ), and VideoComposer (Wang et al., ), have significantly improved the ability to edit objects, backgrounds, and styles in video scenes while maintaining overall scene consistency through the optimization of attention mechanisms and spatiotemporal continuity. These approaches have demonstrated notable success in video generation. However, they are often limited by the restricted word embeddings provided by CLIP (Radford et al., ) during the text-driven encoding process, which restricts their ability to perform diverse and nuanced edits on targets with specific attributes. Modifying words in the target prompt can disrupt the attention mechanisms, leading to inconsistencies in non-target areas before and after the editing process.

To address these limitations, recent methods such as (Bar-Tal et al., ; Lee et al., ; Chai et al., ) have explored the use of Neural Layered Atlases (NLA) (Kasten et al., ). These methods primarily focus on extracting layered atlases from video frames, editing these atlases using text-driven image-editing diffusion models (Rombach et al., ; Zhang et al., ), and then synthesizing the final edited video through post-processing. While this approach is effective at preserving non-edited background areas, it exhibits poor performance in maintaining spatio-temporal continuity. Moreover, the processing of individual image frames makes the generation of neural atlases extremely time-consuming.

To achieve more diverse editing results easily, one feasible approach is to draw inspiration from the Textual Inversion (Gal et al., ) method used in image generation by incorporating external concept word embeddings. In text-to-image generation, external word embeddings (referred to as ”concepts”) are optimized within the CLIP text encoder (Radford et al., ) while freezing the diffusion model’s parameters. This allows the model to address the need for user-provided custom images to guide image editing. However, using a pre-trained diffusion model for self-supervised textual inversion presents limitations in the expressive power of these external embeddings, which are constrained by the size of the latent space and the number of training iterations, often leading to under-fitting and restricted expressive capacity.

In this paper, we propose an improved concept-augmented video editing method. This approach flexibly generates diverse and stable target videos by defining abstract conceptual pairs (concept prompt and concept video) that describe the target scene. Specifically, we propose the Concept-Augmented Textual Inversion method, which reliably and accurately captures the target attributes in the user’s custom video. In addition, we also introduce a Dual Prior Supervision mechanism that stabilizes the generated video by crossing the attention between the source and target, preventing attention dispersion caused by modifications to the target prompt. This mechanism effectively improves the consistency of non-target areas before and after video editing, while also enriching the fidelity of the concepts in the edited results. Our key contributions are as follows:

- •
We orchestrate a framework that allows users to extract concepts from custom videos to efficiently generate diverse edited videos through concept templates.
- •
We propose a concept-augmented textual inversion method, which efficiently and stably extracts detailed attributes of the target in the user’s custom concept video and supports plug-and-play guiding of stable diffusion for video editing.
- •
We present a dual prior supervision mechanism, which effectively improves the consistency and stability of video editing results.

## 2 related work

Text-Driven Video Synthesis.
A series of works based on diffusion models (Ho et al., ; Song et al., ; Rombach et al., ) has made significant progress in text-driven image generation.
Subsequent efforts (Esser et al., ; Wang et al., ; Sterling, ; Blattmann et al., ) further achieve text-driven video generation by extending existing image generation diffusion models.
These works commonly inherit the spatial parameters of UNet and fine-tune the newly added temporal modules with large-scale video-text pair datasets to improve inter-frame stability for video synthesis.
These works laid a good foundation for video editing with textual descriptions.

Text-Driven Video Editing.
Current approaches for text-driven video editing mainly fall into three categories: fine-tuning video generation models (Zhao et al., ; Wang et al., ), fine-tuning image generation models extended with temporal modules (Wu et al., ; Qi et al., ), and combining NLA (Kasten et al., ) with pre-trained image generation models (Bar-Tal et al., ; Lee et al., ; Chai et al., ).
MotionDirector (Zhao et al., ) improves the ability of editing camera and object motions by adding LoRA (Hu et al., ) to the attention modules of the pre-trained Zeroscope (Sterling, ), strengthening the connection between texts and motions in edited videos.
VideoComposer (Wang et al., ) enhances inter-frame consistency by introducing a condition fusion module with spatial and temporal conditions such as motion vectors, depth maps, and sketches.
(Bar-Tal et al., ; Lee et al., ; Chai et al., ) extract layered neural atlases from video frames to edit atlases which are further processed to synthesize videos; however, generating a neural atlas demands considerable computational time.

Recently, Tune-A-Video (Wu et al., ) achieves one-shot video editing with improved inter-frame coherency by updating self-attention with sparse causal attention.
FateZero (Qi et al., ) further proposes self-attention blending and incorporates attention control (Hertz et al., ) to enhance the ability of editing objects, background, and styles while maintaining scene consistency.
However, these methods are constrained by the finite word embeddings of CLIP (Radford et al., ).
For editing targets with specific attributes, it becomes necessary to introduce external word embeddings.
Our method supports the incorporation of external concept word embeddings. Furthermore, inspired by Tune-A-Video (Wu et al., ) and FateZero (Qi et al., ), we introduce a dual prior supervision mechanism between video frame latents and word embeddings to enhance scene consistency before and after video editing based on the prompt-to-prompt attention control method.

Textual Inversion.
(Gal et al., ) proposes a textual inversion method that optimizes newly added concept word embeddings in the CLIP (Radford et al., ) text encoder, supervised by the latent variable distribution of specific images in the diffusion model. However, using a pre-trained diffusion model for self-supervised text inversion may lead to under-fitting for some specific images due to the finite latent variable space. Although it’s feasible to optimize concept word embeddings with a smaller learning rate simultaneously, or to train the diffusion model with frozen concept embeddings in the next stage, this process faces issues of easy over-fitting and high storage costs. Our method, building upon textual inversion (Gal et al., ), attempts to add LoRA (Hu et al., ) modules to the diffusion model, optimizing them simultaneously with concept words at a smaller learning rate, to enhance the text editing capabilities of concept words.

Cross Attention Control and Supervision.
Prompt-to-Prompt (Hertz et al., ) proposes three attention control methods for stable text-driven image editing based on diffusion models: word swap, refinement, and reweighting. By applying the cross-attention probability map recorded from the original text and original image latent variables to the denoising process of edited text and original image latent variables, it has achieved significant success in stable text-driven image editing (Avrahami et al., ; ). Additionally, (Qi et al., ) proposed self-attention blend effectively transfers the stability of text-driven image editing to the video editing domain. Our method, building upon this foundation, introduces external concept words to support editing with higher degrees of freedom. However, when performing text-driven editing, whether using existing word embeddings or external concept word embeddings as editing words, there exists a problem of attention dispersion. This means that editing words have non-negligible effects on latent variables other than the editing target. Inspired by the work of (Yang & Tang, ), we introduce an attention supervision mechanism to address the issue of dispersed attention in editing words.

## 3 Method

### 3.1 Preliminaries

Textual Inversion.
Textual inversion (Gal et al., ) learns new embeddings that represent user-provided visual concepts within the textual embedding space.
These learned embeddings are then associated with pseudo-words that can be incorporated into novel sentences to achieve text-to-vision editing.
The learning process of textual inversion relies on a latent variable diffusion model, which typically consists of an autoencoder and a noise prediction network.
For an image $x$, the autoencoder is pretrained such that the encoder $\mathcal{E}$ maps the image to a latent variable $z=\mathcal{E}(x)$, and the decoder $\mathcal{D}$ reconstructs the original image from the latent variable $x\approx\mathcal{D}(z)$.
Particularly, textual inversion leverages a CLIP (Radford et al., ) text encoder $c_{\theta}$ with additional concept words to encode conditional text input $y$.
The optimization objective is defined as:

$$ $\mathcal{L}_{noise}=\mathbb{E}_{z\sim\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0,1),t}[\|\epsilon-\epsilon_{\theta}(z_{t},t,c_{\theta}(y))\|_{2}^{2}],$ (1) $$

where $z_{t}$ is the noised latent at time step $t$, $\epsilon$ is the noise, and $\epsilon_{\theta}$ is the noise prediction network.

Low-Rank Adaption.
(Hu et al., ) proposes an efficient fine-tuning scheme based on matrix low-rank decomposition. For the pre-trained weight ${\bm{W}}_{0}\in\mathbb{R}^{d\times k}$ in the original model, it applies low-rank decomposition to update the weight as ${\bm{W}}={\bm{W}}_{0}+\Delta{\bm{W}}$, where $\Delta{\bm{W}}={\bm{B}}{\bm{A}}$, ${\bm{B}}\in\mathbb{R}^{d\times r}$, ${\bm{A}}\in\mathbb{R}^{r\times k}$, and $r\ll min(d,k)$. During the fine-tuning process, the pre-trained weight ${\bm{W}}_{0}$ is frozen, while ${\bm{A}}$ and ${\bm{B}}$ are trainable parameters. For the forward computation of the original weight ${\bm{h}}={\bm{W}}_{0}{\bm{x}}$, the updated forward computation becomes:

$$ $LoRA({\bm{h}})={\bm{W}}_{0}{\bm{x}}+\Delta{\bm{W}}{\bm{x}}.$ (2) $$

Video Diffusion Models with Temporal Extensions.
Tune-A-Video (Wu et al., ) introduces Spatio-Temporal Attention (ST-Attn) to replace the original Self-Attention (Vaswani, ) in the 2D UNet.
When calculating the keys ${\bm{K}}$ and values ${\bm{V}}$, ST-Attn concatenates latent variables of the first and former frames of the video, leading to the attention result where the current frame attends to both the first and former frames.
The specific operations for replacing ${\bm{K}},{\bm{V}}$ in Self-Attention are as follows:

$$ ${\bm{K}}={\bm{W}}^{K}[{\bm{z}}_{v_{1}};{\bm{z}}_{v_{i-1}}],{\bm{V}}={\bm{W}}^{V}[{\bm{z}}_{v_{1}};{\bm{z}}_{v_{i-1}}],$ (3) $$

where $W^{K}$ and $W^{V}$ are projection matrices for key and value respectively, $z_{v_{i}}$ denotes the latent variable of the $i$-th frame of the video to the current attention layer, and $[\cdot]$ denotes concatenation.

### 3.2 Stabilized Text-Driven Video Editing

Figure: Figure 1: Overview of our training and inference pipelines. During the training stage, we first adapt the diffusion model to new visual concepts using our introduced Concept-Augmented Textual Inversion (CATI), and then we tune the temporally extended diffusion model with our proposed Dual Prior Supervision (DPS) mechanism to prevent unintended changes in edited videos. During the inference stage, we blend self-attention matrices to retain semantic layout (Self-Attention Blending) and swap cross-attention matrices to achieve text-driven video editing (Cross-Attention Swap).
Refer to caption: /html/2410.12526/assets/x1.png

Our training and inference pipelines are visualized in Fig. [1](#S3.F1).
We adopt a UNet that inherits the pre-trained 2D UNet parameters from Stable Diffusion (Rombach et al., ) as the noise prediction network.
The original 2D UNet consists of a series of spatial convolution layers and cross-attention layers.
To adapt it for 3D video inputs, we replace the original spatial self-attention layers with spatio-temporal attention as explained in Eq. ([3](#S3.E3)).
Following FateZero (Qi et al., ), we also incorporate LoRA-structured temporal convolution layers after the spatial convolution layers, and temporal self-attention layers with zero-initialized linear output layers after the cross-attention layers.
The outputs of these newly added modules are residually connected to the outputs of the original modules.

Our approach for stabilized text-driven video editing has two learning phases.
In the first phase, we introduce Concept-Augmented Textual Inversion (CATI) to adapt the diffusion model to new visual concepts.
In the second phase, we tune partial parameters of the temporally extended diffusion model to suppress unintended changes in edited videos by calibrating cross-attention results.

Concept-Augmented Textual Inversion.
Textual inversion (Gal et al., ) learns to represent a specific set of user-provided images with pseudo-words in the latent space, offering an intuitive way for natural language-guided image editing.
We incorporate this technique into our framework to facilitate video editing.
However, due to the self-supervised nature within the limited latent space of the pre-trained diffusion model, the vanilla textual inversion often results in varied performance in terms of quality and efficiency for different image sets, requiring meticulous adjustments for learning rates and iteration counts.

To alleviate this issue, we draw inspiration from existing parameter-efficient fine-tuning techniques and propose adding LoRA modules (Hu et al., ) to the value projection parameters in the cross-attention layers of the UNet.
Consequently, the values ${\bm{V}}$ are updated to $LoRA({\bm{V}})$ according to Eq. ([2](#S3.E2)).
The rationale behind our approach is that we aim to enhance the expressiveness of the pre-trained diffusion model by slightly adjusting its capacity to accommodate new visual concepts while preserving its original generation capability.
Besides, inserting LoRA modules not only augments textual inversion with low storage overhead but also maintains a plug-and-play characteristic during inference.
We train the concept-word embeddings of textual inversion and the weight parameters of LoRA modules in an end-to-end manner (see orange blocks in Fig. [1](#S3.F1)), where the learning rate for LoRA parameters is relatively smaller than that for concept-word embeddings to avoid over-fitting.
Denote the noise prediction network with LoRA modules loaded on value projection parameters as $\epsilon_{\theta_{L}}$, the optimization objective of concept-augmented textual inversion is then updated from Eq. ([1](#S3.E1)) to the following:

$$ $\mathcal{L}_{noise}=\mathbb{E}_{z\sim\mathcal{E}(x),y,\epsilon\sim\mathcal{N}(0,1),t}[\|\epsilon-\epsilon_{\theta_{L}}(z_{t},t,c_{\theta}(y))\|_{2}^{2}].$ (4) $$

Figure: Figure 2: Visualization of the dual prior supervision mechanism. Each row displays a video frame, a set of cross-attention maps between this video frame and prompt words, and a pseudo ground truth mask. The scam loss and tcam loss are computed between relevant words and pseudo masks to prevent unintended changes for stabilized video editing.
Refer to caption: /html/2410.12526/assets/x2.png

Model Tuning with Dual Prior Supervision.
After learning concept-augmented textual inversion, we adapt and tune the video diffusion model for text-driven video editing in line with the paradigm of few-shot learning.
Specifically, we learn the LoRA-structured temporal convolution layers, the query projection weights within spatio-temporal attention layers and cross-attention layers, and the temporal self-attention layers (see red blocks in Fig. [1](#S3.F1)).
These parameters are selected for updates during training due to their strong relevance to the temporal modeling of 3D videos.
To attain more stable and higher quality editing results, we tried directly integrating existing attention control techniques (Hertz et al., ) in an early attempt; however, we found that when applying text-driven video editing types such as word swap, the dispersion phenomenon of cross-attention between text embeddings and video latents leads to reduced stability in editing results.
To address this challenge, we propose a dual prior supervision mechanism, which includes a source cross-attention mask (scam) loss and a target cross-attention mask (tcam) loss.

The scam loss is designed to reduce the attention influence of the words to be replaced in the source prompt on irrelevant frame areas (see the first row in Fig. [2](#S3.F2)).
It is also applied to modulate attention between concept words and concept videos (see the second row in Fig. [2](#S3.F2)).
Specifically, for $K$ cross-attention layers in the UNet, we record cross-attention matrices $\mathbf{M}_{s}$ between the words and the video frame latents in each cross-attention layer.
To obtain ground truth for optimization, we use an off-the-shelf object detection network OWL-ViT (Minderer et al., ) to localize objects in video frames and generate corresponding binary pseudo-labels ${\mathbf{M}}^{\text{gt}}_{s}$.
We further apply max pooling to generate $K$ pseudo-labels, each with a designated resolution $P_{k}$.
The loss is then calculated as the mean absolute loss on irrelevant areas:

$$ $\mathcal{L}_{scam}=\frac{1}{K}\sum_{k=1}^{K}\sum_{i=1}^{P_{k}}\left[\|\mathbf{M}^{\text{gt}}_{s,k,i}-\mathbf{M}_{s,k,i}\|\cdot(1-\mathbf{M}^{\text{gt}}_{s,k,i})\right].$ (5) $$

The tcam loss is introduced to diminish the attention influence of the target words in the edited prompt to further promote the consistency of irrelevant areas before and after video editing (see the third row in Fig. [2](#S3.F2)).
Similar to the scam loss, we obtain cross-attention matrices $\mathbf{M}_{t}$ and pseudo-labels $\mathbf{M}^{\text{gt}}_{t}$ between the target words in the edited prompt and the video frame latents.
The loss is computed as:

$$ $\mathcal{L}_{tcam}=\frac{1}{K}\sum_{k=1}^{K}\sum_{i=1}^{P_{k}}\left[\|\mathbf{M}^{\text{gt}}_{t,k,i}-\mathbf{M}_{t,k,i}\|\cdot(1-\mathbf{M}^{\text{gt}}_{t,k,i})\right].$ (6) $$

Let the trainable parameters during the model tuning phase be denoted as $\epsilon_{\theta_{T}}$.
The noise prediction loss $\mathcal{L}_{noise}$ is then obtained by substituting $\epsilon_{\theta}$ in Eq. ([1](#S3.E1)) with $\epsilon_{\theta_{T}}$.
Given $\alpha$ and $\beta$ as the weighting coefficients for our proposed scam loss and tcam loss, respectively, the total loss for model tuning with dual prior supervision is formulated as:

$$ $\mathcal{L}=\mathcal{L}_{noise}+\alpha\mathcal{L}_{scam}+\beta\mathcal{L}_{tcam}.$ (7) $$

Inference.
As shown in Fig. [1](#S3.F1), the inference pipeline involves an inversion stage using the source text prompt, and an editing stage using the modified text prompt.
We cache the intermediate self-attention matrices and cross-attention matrices at each time step during inversion.
These matrices are then leveraged to manipulate attention during editing.
Specifically, we blend self-attention matrices to retain the semantic layout following FateZero Qi et al. () (Self-Attention Blending), and swap cross-attention matrices between the changed words and video latents similar to Prompt-to-Prompt Hertz et al. () (Cross-Attention Swap).

## 4 Experiments

### 4.1 Settings and Datasets

Our experiments are conducted on a machine equipped with an NVIDIA GeForce RTX 4090. During the concept augmented textual inversion stage, we set the learning rate for CLIP (Radford et al., ) word embeddings to $1\times 10^{-3}$, and the learning rate for LoRA modules inserted into the UNet to $1\times 10^{-5}$, with the number of training steps set to $5000$. Additionally, we randomly sample frame numbers within the range $[4,8]$ from the concept video during training, to prevent the inversion process from over-fitting to a fixed frame number.
For the video diffusion model fine-tuning stage, we empirically set $\alpha=0.1$ and $\beta=0.1$ in Eq. ([7](#S3.E7)). The training steps above all use AdamW (Loshchilov & Hutter, ) optimizer. In the inference stage of video editing, the guidance scale is set to $12.5$, the number of DDIM Inversion steps is $T=50$, and the self-attention blending and cross-attention swap steps are within the interval $[0,0.7T]$. To evaluate our proposed method, we used a portion of the DAVIS (Caelles et al., ) dataset and clip videos from the internet to construct video editing pairs, either with or without concept videos.

### 4.2 Metrics

Frame Consistency.
To compare the coherence of the video frames ${\mathbb{F}}$, we refer to the metric used in (Wu et al., ; Hessel et al., ), which calculates the average cosine distance $d$ between features $({\bm{v}}_{i},{\bm{v}}_{j})$ of each two different frames $({\bm{f}}_{i},{\bm{f}}_{j})$ encoded by the CLIP visual encoder (Radford et al., ), as Eq. ([8](#S4.E8)). Here, ${\bm{f}}_{i},{\bm{f}}_{j}\in{\mathbb{F}}$, ${\bm{f}}_{i}\neq{\bm{f}}_{j}$, and ${\mathbb{D}}$ denotes the set of the vector pairs $({\bm{v}}_{i},{\bm{v}}_{j})$.

$$ $d=\frac{1}{|{\mathbb{D}}|}\sum_{({\bm{v}}_{i},{\bm{v}}_{j})\in{\mathbb{D}}}\frac{{\bm{v}}_{i}\cdot{\bm{v}}_{j}}{\|{\bm{v}}_{i}\|\|{\bm{v}}_{j}\|}.$ (8) $$

**Table 1: Quantitative comparison of video editing results.**
| Method | Frame Consistency $\uparrow$ | M-PSNR $\uparrow$ | Concept Consistency $\uparrow$ |
| --- | --- | --- | --- |
| Tune-A-Video (Wu et al., ) | 0.9399 | 14.88 | 0.6982 |
| FateZero (Qi et al., ) | 0.9361 | 17.33 | 0.6822 |
| MotionDirector (Zhao et al., ) | 0.9446 | 13.48 | 0.7222 |
| RAVE (Kara et al., ) | 0.9339 | 17.67 | 0.6990 |
| Ours | 0.9464 | 19.90 | 0.7642 |

Masked Peek-Signal-Noise Ratio.
To compare the stability of the video non-target areas before and after target editing, we design a Masked Peak Signal-to-Noise Ratio (M-PSNR) metric. We use the OWL-ViT (Minderer et al., ) open-vocabulary object detection model with text pseudo-labels to estimate the bounding box mask ${\bm{M}}$ of the edited target. We then compare the average peek-signal-noise ratio of the original video frames and the edited video frames after applying this mask. The calculation formula for the specific function $f$ for the Mean Squared Error (MSE) used as input is as follows, where ${\bm{M}}\in\mathbb{R}^{H\times W}$, ${\bm{I}}^{s}\in\mathbb{R}^{H\times W\times C}$, and ${\bm{I}}^{e}\in\mathbb{R}^{H\times W\times C}$ refer to the mask value, the frame pixel value of video before and after editing, respectively.

$$ $f({\bm{I}}^{s},{\bm{I}}^{e},{\bm{M}})=\frac{1}{C}\frac{\sum_{k\in C}\sum_{i\in H}\sum_{j\in W}(I_{i,j,k}^{s}-I_{i,j,k}^{e})^{2}(1-M_{i,j})}{\sum_{i\in H}\sum_{j\in W}(1-M_{i,j})}.$ (9) $$

Concept Consistency.
To evaluate the correlation between the video editing results guided by the concept video and the concept video itself, while minimizing non-target areas interference, we employ a multi-step approach. First, we utilize a pre-trained OWL-ViT (Minderer et al., ) model in conjunction with pseudo-label prediction to generate object masks for both videos. Using these masks, we then extract pixel segments of the target objects from both the edited video and the concept video. Finally, we leverage the CLIP model to predict visual encoding vectors for these extracted segments and calculate the average cosine similarity between them.

Figure: Figure 3: Video generation with (Setting I) and without (Setting II) concept pairs. The top row of the figure contains the concept video with its prompt in the left part, and comparison settings in the right part. The second row is the source video frames to be edited and its prompt. The rows below show the editing results of the source video using the editing prompt, for (Wu et al., ), (Qi et al., ), (Zhao et al., ), (Kara et al., ) and our method, respectively, in which words with ‘$’ ahead mean concept words, and the same applies to subsequent results.
Refer to caption: /html/2410.12526/assets/x3.png

### 4.3 Comparisons with Existing Methods

Quantitative Evaluation.
As illustrated in Tab. [1](#S4.T1), we quantitatively assess text-driven video editing results in three aspects.
Compared with existing methods that extend and fine-tune the Stable Diffusion model, including Tune-A-Video (Wu et al., ), FateZero (Qi et al., ), RAVE (Kara et al., ), and MotionDirector (Zhao et al., ), our approach demonstrates superior inter-frame coherence in terms of the Frame Consistency Metric.
To evaluate the consistency of unrelated areas before and after video editing, we employ M-PSNR as a reference metric, and our method achieves the highest score by a large margin.
Concretely, our method outperforms MotionDirector (Zhao et al., ) by a noticeable 6.42 M-PSNR.
This is attributed to our proposed prior supervision mechanism, which effectively reduces the editing noise in non-target areas for both source and concept videos.
Furthermore, to evaluate the target fidelity in concept and edited videos,
we utilize Concept Consistency as a reference metric, and our method demonstrates greater fidelity compared to others.

Figure: Figure 4: Comparison of whether to use Concept Augmentation (CA) for textual inversion. Compared the text inversion results without and with concept augmentation for pairs (a), (b): ‘jeep’ $\to$ ‘$LAMBO’; and (c), (d): ‘jeep’ $\to$ ‘$CYBERTRUCK’, respectively, from the same source prompt “a jeep driving down a curvy road in the countryside”.
Refer to caption: /html/2410.12526/assets/x4.png

Qualitative Evaluation.
The visual comparison results of video editing with and without concept video guidance are shown in Fig. [3](#S4.F3).
As can be seen, our method can maintain content consistency in non-target areas before and after video editing with and without concept videos.
Particularly, when using concept videos, our method can effectively introduce the visual concept from the concept video into the edited video.
For example, in Fig. [3](#S4.F3) (Setting I), our method successfully replaces ‘man’ with the concept ‘SOPTIMUS’, while others either fail to preserve the background or cannot transfer the integral target shape.

On the other hand, other approaches commonly face instability in non-target areas of their edited videos.
Tune-A-Video (Wu et al., ) encounters the issue of dispersed cross-attention between word embeddings and video latents as it fine-tunes the model using only one video-text pair.
While FateZero (Qi et al., ) and RAVE (Kara et al., ) mitigate this issue by manipulating cross-attention matrices or shuffling noise in a zero-shot manner, these methods directly use concepts to drive video editing, which results in compromised non-target area consistency and degraded concept fidelity.
MotionDirector (Zhao et al., ) naturally supports extracting targets from concept videos via its trainable spatial path; however, the coupled spatial and temporal paths struggle to provide stable guidance, leading to noticeable inconsistencies in non-target areas.
In contrast, our proposed concept-augmented textual inversion and dual prior supervision can effectively maintain content consistency in non-target areas before and after video editing while accurately capturing specific attributes of user-provided concepts.

### 4.4 Ablation Study

Figure: Figure 5: The impact of dual prior supervision. From the first to the last row, using the editing example in Fig. [1](#S3.F1), we compare the average cross-attention maps and the editing results with and without the supervision mechanism of scam and tcam. Each case contains three pairs, and each pair consists of an average cross-attention map on the left and an edited frame on the right. The full comparisons are put in Fig. [11](#A1.F11) and Fig. [12](#A1.F12).
Refer to caption: /html/2410.12526/assets/x5.png

Concept Augmentation Alleviates Under-Fitting of Textual Inversion.
In this work, we draw on the idea of Textual Inversion (TI) from text-to-image generation and apply it to text-driven video editing to address the embedding of external concept words. However, simply applying TI may lead to under-fitting, resulting in a lack of realism. For instance, in the results shown in Fig. [4](#S4.F4)(a) and Fig. [4](#S4.F4)(c), where the keywords ‘jeep’ are altered to ‘LAMBO’ and ‘CYBERTRUCK’, although some attributes (e.g., shape) of the target concepts are partially retained, the results appear to ”drift” due to insufficient inductive bias. In contrast, the concept-augmented textual inversion can effectively capture the color, shape, and other attributes, as demonstrated in Fig. [4](#S4.F4)(b) and Fig. [4](#S4.F4)(d). The concept augmentation provides more detailed features to the target, significantly improving the fidelity of details in the inversion results.

Dual Prior Supervision Improves Stability and Fidelity.
In this work, we propose a Dual Prior Supervision strategy, which consists of two main components (See Sec. [3.2](#S3.SS2)): scam loss and tcam loss. Both components play crucial roles in maintaining the stability of the target generation. By comparing the attention regions in Fig. [5](#S4.F5) (a) (w/o tcam, w/o scam), Fig. [5](#S4.F5) (b) (w/o tcam, w/ scam), and Fig. [5](#S4.F5) (c) (w/ tcam, w/o scam), we can conclude that both scam and tcam (Fig. [5](#S4.F5) (d)) significantly reduce background disturbances and improve stability. However, the generated video results reveal that using either component alone cannot effectively reproduce the target object’s attributes, such as the car’s color. The proposed Dual Prior Supervision, which combines both components, not only enhances the stability of the background in the target results, but also captures the target object’s attributes more accurately, thereby improving the fidelity of the edited concept target.

Tuning w/ Concept Video Produces Stylized Results.
Recall that we construct the target videos in this work by templating the concept pairs to make the editing process more flexible. To explore the impact of the concept video in Setting I (Fig. [3](#S4.F3)), we conduct a simple experiment as shown in Fig. [6](#S4.F6). As shown in Fig. [6](#S4.F6)(a) and Fig. [6](#S4.F6)(b), tuning models with both concept video and concept prompt produce more stylized videos. The possible explanation for this is that the introduction of the concept video alleviates the over-fitting issue. Note that tuning with only the concept video (without the concept prompt) is not viable here, as we cannot analyze the intent without any textual guidance.

Figure: Figure 6: Comparison of whether to tune with the concept video. Compared the video editing results without and with tuning concept video for the left part: ‘car’ $\to$ ‘$GT3’; and the right part: ‘car’ $\to$ ‘$LAMBO’, from the source prompt “a car is drifting around a curve road with the background of a forest” and “a car is drifting in the snow”, respectively.
Refer to caption: /html/2410.12526/assets/x6.png

## 5 Limitations and Future Work

Mismatch when Significant Deformation.
Although our proposed method effectively mitigates the inconsistency in non-target areas caused by attention dispersion in video editing methods using attention replacement mechanisms, it may struggle when a single concept video guides target replacement in cases of significant deformation in the source video, such as running people. For instance, there may be insufficient detailed correspondences between the internal parts of the replacing and replaced targets during deformation, such as moving arms and legs. Potential solutions include ControlNet (Zhang et al., ) and OpenPose (Cao et al., ), which utilize motion conditions, like human pose, to guide the video editing process.

## 6 Conclusion

In this paper, we propose an improved concept-augmented video editing approach that generates diverse and stable target videos flexibly by devising abstract conceptual pairs. Specifically, the framework introduces a Concept-Augmented Textual Inversion (CATI), which extracts the target concept from user-customized videos. In practice, CATI enables plug-and-play guidance of stable diffusion for video shaping, effectively capturing target attributes for more stylized editing results. In addition, a Dual Prior Supervision (DPS) mechanism is designed to prevent unintended changes in non-target visual areas by crossing the attention between the sources and targets. Experimental results demonstrate that our method significantly improves the flexibility, consistency, and stability of text-guided video editing.

## Appendix A Appendix

### A.1 Training and Inference Details

Details of Concept Augmented Textual Inversion. During the stage of concept augmented textual inversion, we use AdamW optimizer (Loshchilov & Hutter, ), with the default betas set to $0.9$ and $0.999$, the weight decay set to $1\times 10^{-2}$ and the epsilon set to $1\times 10^{-8}$. Besides, LoRA modules inserted to UNet are without bias trainable parameters, in which the LoRA rank is set to $16$, the weight coefficient to scale LoRA output is set to $1.0$, and the dropout parameter is set to $0.1$. Additionally, we add a prefix to the head of concept prompts and use one embedding to represent a concept word as the same as in Textual Inversion (Gal et al., ).

Figure: Figure 7: Additional qualitative comparison of text-driven video editing without concept video. At the top of the figure are the source video frames to be edited and its corresponding descriptive prompt. The rows below show the editing results of the source video using the editing prompt, for (Wu et al., ), (Qi et al., ), (Zhao et al., ), (Kara et al., ), and our method, respectively.
Refer to caption: /html/2410.12526/assets/x7.png

Details of Dual Prior Supervision. During the stage of tuning the temporally extended diffusion model with dual prior supervision, we use the same optimizer settings in concept augmented textual inversion stage. For each training epoch, we sequentially use the pairs of source video with source prompt and concept video with concept prompt as input for one training iteration, predicting noise and computing the noise prediction loss. During one training iteration, we retain all the cross-attention probability matrices from the cross attention layers in the UNet and compute scam based on the pseudo ground-truth mask corresponding to each video. Before the end of each iteration, we then input the pair of source video and target prompt, also retaining the cross-attention probability matrices, and compute tcam based on the pseudo ground-truth mask. The number of training epochs are less than $250$. The input data frames have a resolution of $512\times 512$ pixels and a length of $6$ frames. The training time does not exceed $30$ minutes on an NVIDIA GeForce RTX 4090 device.

Figure: Figure 8: Additional qualitative comparison of text-driven video editing with concept video. At the top of the figure are the concept videos and the prompt containing the concept word. Below are the source video frames to be edited along with its descriptive prompt. The rows below show the editing results of the source video using the prompt containing the concept word, for (Wu et al., ), (Qi et al., ), (Zhao et al., ), (Kara et al., ), and our method, respectively.
Refer to caption: /html/2410.12526/assets/x8.png

Details of Inference. During the stage of inference, we use the device equipped with an NVIDIA GeForce RTX 4090 and store the model parameters and data inputs in fp16 format. During the denoising of the source video, we store intermediate variables including the self-attention and cross-attention probability matrices and source latents at each step in RAM, occupying approximately $100$ GB space. The attention control processes of self-attention blending and cross-attention swap consume most of the inference time. Depending on different attention control configurations, the inference time generally ranges between $1$ to $3$ minutes.

Additionally, to reduce the host memory overhead of intermediate variables during inference, we have designed and implemented a memory-saving inference scheme. This scheme requires about twice the inference time compared to the original but reduces the overhead of RAM to $5$ GB. The key difference between this scheme and the original is that it does not store the self-attention and cross-attention probability matrices; instead, it only stores the source latents at each denoising step and recalculates the self-attention and cross-attention probability matrices during attention control.

Figure: Figure 9: Additional qualitative comparison of text-driven video editing with concept video. The form is the same as Fig. [8](#A1.F8).
Refer to caption: /html/2410.12526/assets/x9.png

### A.2 Additional Comparisons with Existing Methods.

Text-driven Video Editing without Concept Video. We provide an additional set of comparison examples without concept video guidance for video editing, as shown in Fig. [7](#A1.F7).

Text-driven Video Editing with Concept Video. We provide six additional comparison examples with concept video guidance for video editing, as shown in Fig. [8](#A1.F8), Fig. [9](#A1.F9), and Fig. [10](#A1.F10).

Figure: Figure 10: Additional qualitative comparison of text-driven video editing with concept video. The form is the same as Fig. [8](#A1.F8).
Refer to caption: /html/2410.12526/assets/x10.png

### A.3 Additional Ablation Study Results

Additional Results for Dual Prior Supervision. We provide the full comparison of the editing results for Fig. [5](#S4.F5), as shown in Figure [11](#A1.F11), and the comparison of the full average cross attention map in Fig. [12](#A1.F12).

Figure: Figure 11: Full comparison of whether to tune with attention supervision. The first column shows frames from the source video to be edited. From the second column to the last column, using the editing pair in Fig. [1](#S3.F1) as the example, we compare the editing results with and without the attention supervision mechanism of scam and tcam.
Refer to caption: /html/2410.12526/assets/x11.png

Figure: Figure 12: Average cross attention probability map explorations. The first column shows frames from the source video to be edited. From the second column to the last column, using the editing pair in Fig. [1](#S3.F1) as the example, we compare the average cross attention maps with and without the attention supervision mechanism of scam and tcam, visualization of plasma colormap in matplotlib (Hunter, ).
Refer to caption: /html/2410.12526/assets/x12.png

Impact of Attention Supervision Weights.
Considering the impact of different attention supervision weights, Fig. [13](#A1.F13) shows that as the weight increases, there is a higher degree of overlap between the area of the replacement target and the edited target. Simultaneously, the consistency in non-target areas improves to a certain extent.

Figure: Figure 13: Hyperparameters of different attention supervision weights. The first column shows frames from the source video to be edited. From the second column to the last column, using the editing pair in Fig. [4](#S4.F4)(c) as the example, we compare the editing results with attention supervision weights from 0.05, 0.1 and 0.3.
Refer to caption: /html/2410.12526/assets/x13.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: Please see the weaknesses.
soundness: 2
strengths: * Quantitative/qualitative evaluation shows relatively improved performance.
* The two proposed methods intuitively make sense.
summary: This paper introduces an improved text-driven video editing approach using generative diffusion models. It overcomes limitations of traditional methods by incorporating a concept-augmented textual inversion technique and a dual prior supervision mechanism. For the concept-augmented textual inversion, LoRA is adopted, and for the dual prior supervision mechanism, a cross-attention-based masking technique is utilized.
weaknesses: * Writing needs improvement. It is difficult to distinguish what is being proposed and what are existing components in the method section.

* Lack of technical novelty. I think the use of LoRA with Textual Inversion (or other inversion/personalization technique) is already widely used as open-source, and compared to these, the use of LoRA in the proposed Concept-Augmented Textual Inversion does not appear to be significantly different. Also, although it is minor, it is not convincing why this technique is called Concept-Augmented Textual Inversion.

* In Dual Prior Supervision, a mask is obtained by thresholding the activated attention part corresponding to the text via cross-attention, and this is applied to the loss. This approach itself has been widely used in segmentation and image editing (e.g., masactrl) and does not appear technically novel. Even setting aside the issue of technical novelty, I have concerns about the approach of specifying areas corresponding to certain words through hard thresholding, which seems heuristic and may have a marginal effect outside situations where the object is explicitly referenced.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. What are the parameters that are finetuned during the scam and tcam loss optimization? 

2. What possible reasons do you think that the edited videos are not looking consistent enough? What possible solutions could you propose?
soundness: 2
strengths: 1. The overall presentation of the paper is easy to follow. Most of the details are clear and well-documented. 

2. Based on the quantitative metrics, the model can outperform the previous baselines.
summary: This paper proposes a text-driven video editing method using Stable Diffusion. It involves concept-augmented textual inversion and a dual prior supervision mechanism. The results show that the proposed method can outperform the baselines.
weaknesses: 1. My main concern is that the overall quality of the edited videos is not satisfactory. Mainly, there are obvious changes (shape and color) between frames, so the videos don’t look consistent. I think this may be the inherent weakness of the approach of inflating Stable Diffusion with temporal layers. The base model doesn’t have video prior knowledge compared to a video diffusion model which has been trained on video datasets. Possible ways to improve can be adding motion prior such as adopting optical flow [1] or propagating features among frames [2]. 

2. Besides the quality, the methodology is also not strong to me. Textual inversion comes with several variations, and one of them can be finetuning the model parameters instead of only finetuning the textual embedding. It seems concept-augmented textual inversion belongs to this variation by additionally finetuning V. The scam and tcam loss, although improves the results, does not seem novel or strong enough to boost the consistency of the videos. 

3. I appreciate that the authors also provide details of efficiency. I think the efficiency is also not optimal enough. More than 30 minutes (training + inference) to edit a 6-frame video seems inefficient and not practical. 

[1] Cong, Yuren, et al. "Flatten: optical flow-guided attention for consistent text-to-video editing." ICLR 2024.

[2] Geyer, Michal, et al. "Tokenflow: Consistent diffusion features for consistent video editing." ICLR 2024.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The pipeline for editing without concept video is not very clear for me, and in the code repository provided, a pretrained lora/concept model is required for all the cases including car to porsche. Can authors provide a more detailed explanation?
2. It would be interesting to know the range of concepts. More examples of diverse concepts or attributes could clarify the extent to which CATI handles generalization across different object classes and scenarios.
soundness: 3
strengths: 1. The motivation for introducing the concept of video editing is clear and good. 
2. The results are impressive, the concepts injected are consistent, and the backgrounds are generally consistent with the target videos.
3. The paper reads well and is easy to follow.
summary: This paper present a novel concept-based video editing framework. The framework includes a Concept-Augmented Textual Inversion (CATI) to extract the target concept from given video, and then use a Dual Prior Supervision (DPS) to manipulate the attention matrices to ensure the stability of output videos. For the CATI part, optimization is performed on both textual embeddings and LoRA modules added to the cross-attention layers. And for the DPS part, loss on both irrelevant parts and edited subject are used to prevent unwanted changes.

Results are conducted over several with or without concept video pairs and was compared with several methods. The overall results are good.
weaknesses: 1. How authors obtain the results from comparing methods is not explained in the paper. Some of those don’t accept concept video/images input, and if not given the concept video, then direct comparison is a bit unfair. 
2. The quantitative comparison table seems to be evaluated on both with or without concept videos. And the ablation for using or not using concept videos is missing. I suggest the author having separate dataset for using or not using concept videos, and this can further strengthen the contribution.
3. Missing references and some for comparisons:
    
    *[1] AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks
    [2] GenVideo: One-shot target-image and shape aware video editing using T2I diffusion models*
    
    *[3] Video Editing via Factorized Diffusion Distillation*
    
    *[4] VidToMe: Video Token Merging for Zero-Shot Video Editing*
    
    *[5] Pix2video: Video editing using image diffusion*
    
    *[6] Slicedit: Zero-Shot Video Editing With Text-to-Image Diffusion Models Using Spatio-Temporal Slices*
    
    *[7] TokenFlow: Consistent Diffusion Features for Consistent Video Editing*
    
    *[8] Video-P2P: Video Editing with Cross-attention Control*

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 2
questions: See question in weakness
soundness: 2
strengths: - Comprehensive experimental setup with both ablation and comparative studies.
- Clear writing that outlines the approach and methodology effectively.
summary: This paper introduces a novel pipeline for video editing that leverages spatial-temporal attention and dual prior supervision. The proposed method demonstrates potential through its structure and method of concept-augmented textual inversion and self-attention blending, albeit with certain limitations in innovation and effectiveness. The ablation study provides some insights, particularly around car color changes, but overall performance falls short of expectations.
weaknesses: - Limited innovation; the first proposed method (concept-augmented textual inversion) is not significantly novel, and the second (dual prior supervision mechanism) shows only moderate innovation.
- The quality of generated videos is subpar, with short video length and lackluster editing effects.
- Missing details on the training set used in the experiments.
- Evaluation metrics and comparison methods are unconventional and somewhat outdated, potentially impacting the rigor of performance comparisons.
- Ablation study results indicate some changes in attributes (e.g., car color) but are unimpressive in terms of visual quality and effect.

---

confidence: 4
contribution: 3
initial_rating: 3
presentation: 3
questions: Please refer to the weakness section.
soundness: 3
strengths: 1. The paper is easy to follow.
2. The clarity of the writing.
3. SCAM loss and TCAM loss sound novel.
4. Convincing metric for qualitative evaluation: Masked Peek-Signal-Noise Ratio.
summary: This paper addresses limitations in text-driven video editing using generative diffusion models, particularly the challenge of nuanced editing when constrained by pre-trained word embeddings. The authors propose a concept-augmented video editing approach that allows for flexible and stable video generation through the use of abstract conceptual pairs. The key components include concept-augmented textual inversion, which provides plug-and-play guidance for stable diffusion to better capture target attributes, and a dual prior supervision mechanism that enhances video stability and fidelity. By integrating these elements, the approach overcomes the disruptions caused by direct keyword alterations and allows for more nuanced, stylized editing. Experiments demonstrate that this method outperforms existing state-of-the-art techniques in generating stable and lifelike videos.
weaknesses: 1. Diffusion based video editing has developed quickly and Tune-A-Video and Fatezero are not the strong baselines at this point. Plus, how is this paper different from DreamVideo [1] and MotionBooth [2] paper? I think these works could serve as better baselines?
2. In L36, Zeroscope is not a video editing method but an open-sourced video diffusion model.
3. Is there a reason why image diffusion (stable diffusion) is used as a backbone? I don't see reasons that the method can't be applied to video diffusion models and compared to video diffusion based methods.
4. L242-245, can the authors show results?
5. CLP consistency fails to reflect temporal consistency. Using other metrics like DINOv2 (like the ones from VBench[3]) or LIPIPS would be better.
6. In terms of temporal consistency, the resulting videos of MotionDirector look better.
7. The concept augmented textual inversion lacks technical contribution.

[1] DreamVideo: Composing Your Dream Videos with Customized Subject and Motion, CVPR 2024

[2] MotionBooth: Motion-Aware Customized Text-to-Video Generation, Arxiv 2024

[3] VBench: Comprehensive Benchmark Suite for Video Generative Models, CVPR 2024

Meta-review: This paper receives reviews of 5,3,6,3,3. The AC follows the decisions of the reviewers to reject the paper. The main concerns of this paper are: 1) The need to improve on the writing to make the technical contributions clearer. 2) The lack of novelty since the use of LoRA with Textual Inversion is already widely used as open-source. 3)  The overall quality of the edited videos is not satisfactory. There are obvious inconsistencies between the frames of the videos. 4) Evaluation metrics and comparison methods are unconventional and somewhat outdated. On the other hand, the reviewer who gave a 6 did not give any strong reasons to overrule the majority of the other reviewers who reject the paper.

Decision: Reject