Towards Consistent Video Editing with Text-to-Image Diffusion Models
Authors: Zicheng Zhang
Bonan Li
Xuecheng Nie
Congying Han
Tiande Guo
Luoqi Liu
Venue: NeurIPS 2023 poster (2023)
Keywords: diffusion model
video editing
text-to-video diffusion model

Abstract: Existing works have advanced Text-to-Image (TTI) diffusion models for video editing in a one-shot learning manner. Despite their low requirements of data and computation, these methods might produce results of unsatisfied consistency with text prompt as well as temporal sequence, limiting their applications in the real world. In this paper, we propose to address the above issues with a novel EI$^2$ model towards Enhancing vIdeo Editing consIstency of TTI-based frameworks. Specifically, we analyze and find that the inconsistent problem is caused by newly added modules into TTI models for learning temporal information. These modules lead to covariate shift in the feature space, which harms the editing capability. Thus, we design EI$^2$ to tackle the above drawbacks with two classical modules: Shift-restricted Temporal Attention Module (STAM) and Fine-coarse Frame Attention Module (FFAM). First, through theoretical analysis, we demonstrate that covariate shift is highly related to Layer Normalization, thus STAM employs a Instance Centering layer replacing it to preserve the distribution of temporal features.  In addition, STAM employs an attention layer with normalized mapping to transform temporal features while constraining the variance shift.  As the second part, we incorporate STAM with a novel FFAM, which efficiently leverages fine-coarse spatial information of overall frames to further enhance temporal consistency. Extensive experiments demonstrate the superiority of the proposed EI$^2$ model.

Towards Consistent Video Editing with
Text-to-lmage Diffusion Models
Zicheng Zhang
Bonan Li
Xuecheng Nie²
Congying Han Tiande Guo
Luoqi Liu²
University of Chinese Academy of Sciences
MT Lab, Meitu Inc.
Video clip: "A woman is talking"
A woman talking, 3D rendering style
Prior work
"A wemen talking"
Figure 1: Comparisons of the proposed EI? and prior SOTA for text-driven video editing with Text-to-Image diffusion model. EIZ addresses problems on temporal and semantic inconsistencies existing in prior work, leading more consistent editing results. Better visual effect in color mode.
Abstract
Existing works have advanced Text-to-Image (TTI) diffusion models for video editing in a one-shot learning manner. Despite their low requirements of data and computation, these methods might produce results of unsatisfied consistency with text prompt as well as temporal sequence, limiting their applications in the real world. In this paper, we propose address the above issues with novel EI² model towards Enhancing vIdeo Editing consIstency of TTI-based frameworks. Specifically, we analyze and find that the inconsistent problem is caused by newly added modules into TTI models for learning temporal information. These modules lead to covariate shift in the feature space, which harms the editing capability. Thus, we design EI2 to tackle the above drawbacks with two classical modules: Shift- restricted Temporal Attention Module (STAM) and Fine-coarse Frame Attention Module (FFAM). First, through theoretical analysis, we demonstrate that covariate shift is highly related to Layer Normalization, thus STAM employs a Instance Centering layer replacing it preserve the distribution of temporal features. In addition, STAM employs an attention layer with normalized mapping to transform temporal features while constraining the variance shift. As the second part, we incorporate STAM with a novel FFAM, which efficiently leverages fine-coarse spatial information of overall frames to further enhance temporal consistency. Extensive experiments demonstrate the superiority of the proposed EI² model.
"Equal contribution Corresponning author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

1 Introduction
In light of the surging popularity of video applications in modern times, there has been a growing focus on developing editing techniques [18, 41, 54, 45] in research community to provide automatic and efficient video editing services for counterparts, e.g., content creators and media professionals. While existing techniques have explored the use of Variational Autoencoders (VAEs) [25, 32] and Generative Adversarial Networks (GANs) [12, 20, 57,51J, they are often limited to specific scenes or datasets, and struggle to provide a comprehensive solution. Thanks to the remarkable capabilities of diffusion models [15, 15, 48] in distribution learning, recent research [39, 38, 41, 36] has facilitated great progress in various fields including image synthesis, editing and restoration, posing significant promise for solving tasks in video domain. However, replicating the success in video domain is challenging, due to the higher-dimensional complexity, temporal consistency, and lack of high- quality datasets. In this paper, we develop diffusion model to tackle text-driven single video editing task [4, 56, 10], which utilizes text prompts to guide the change of style and objects in a given video.
From the perspective of diffusion models, prior works typically build Text-to- Video (TTV) frame- works and leverage the generative prior to address this task, wherein two primary approaches are exploited: The first approach [31, 10, 45] directly trains a TTV model on large corpus of text and video pairs to facilitate video editing. However, due to the lack of high-quality public video datasets, current methods mainly rely on in-house datasets and demand significant computational resources. The second approach [56, 28] extends the architecture of pre-trained Text-to- -oae-maeee (TTI) models [39, 38] for video editing. To capture motion in videos, various temporal modules, e.g., Pratial-Casud Attention (SCA) and Temporal Attention (TA), are introduced to TTI models. The temporal motion will be learned from video via tuning and DDIM inversion. This approach is more cost-efficient, easily accessible, and thus gaining more attention in recent studies [28, 44, 35, 55).
Nonetheless, methods inflating TTI models [56, 28] suffer from two critical issues: Temporal inconsistency, where the edited video exhibits cross-frame flickering in vision, and Semantic disparity, where videos are not altered in accordance with the semantics of given text prompts. Addressing these issues will considerably push the frontiers of text-driven video editing. In this paper, we reveal that the cause of the semantic disparity is not ane-ttunigg over-fitting to the given video, but rather the newly introduced temporal layer, designed to ensure consistency across frames, which may weaken or even eliminate the editing power of TTI models. We attribute this phenomenon to the covariate shift of generative feature space, as the extra added parameters of Temporal Attention (TA) without any constraint inevitably transform the statistics of features [21, 52, 3]. Moreover, while using SCA can help reduce computational overhead, it is a suboptimal choice given that the non-autoregressive nature of the diffusion model demands considering global relationships. Therefore, incorporating efficient global spatial-temporal attention [53, 47] is essential for enhancing temporal consistency.
We present novel approach called EI2 that enhances the capabilities of existing pre-trained TTI diffusion models [39] by incorporating well-grounded temporal modules for video editing task. EI² follows the one-shot tuning paradigm [56], while making significant contributions in both theory and practice: (1) Shift-restricted Temporal Attention Module (STAM) to resolve the semantic disparity: (i) We provide theoretical proof, under certain assumptions, that the covariate shift is unrelated to tuning. but is caused by newly introduced parameters in the TA module. This provides valuable guidance in addressing the problem. (ūi) On this basis, identify that the Layer Norm [3] in TA module is the prime cause of covariate shift. To mitigate this issue, we propose a simple yet effective replacement called Instance Centering to restrict distribution shift. (iū) Furthermore, we constrain the shift of variance by normalizing the weights in the vanilla attention within the 7A module. (2) Fine-coarse Frame Attention Module (FFAM) to enhance temporal consistency: We improve SCA with a novel fine-coarse interactive mechanism to establish an efficient spatial-temporal attention, which considers all frames while achieving low computational complexity. Instead of discarding information on the temporal dimension, we perform sampling along the spatial dimension. This preserves the overall structure of spatial-temporal data and reduces the data volumes for consideration. Concretely, FFAM respects the vanilla attention design to preserve fine information in the current frame, but downsamples non-current frames to obtain coarse features for interaction. In this way, FFAM can achieve improved modeling of motion while keeping comparable computational burden as SCA. (3) Extensive experiments validate that EI2 effectively addresses semantic disparity and enhances temporal consistency in video editing compared to the current state-of of-the--hteee methods.

---

2 Related works
Text-to-Image diffusion models. Diffusion models have demonstrated remarkable success in image generation by effectively approximating data distributions [15, 46, 48, 24, 9], outperforming previous generative models like GANs [6, 22, 43, 60]. Building upon the advancements of pre-trained language models [37]. Text-to-Image (TTT) diffusion models [38, 41, 39] have been developed to generate high-fidelity images conditioned on text descriptions. Prominent examples include recent works like DALLE2 [38], and Imagen [41] achieve impressive and controllable image generation. Recently, the Latent Diffusion Model (LDM) [39] has drawn more attention due to its efficiency and compelling results, which is used our base model in the experiments.
Texteto-Video diffusion models. TTV generation poses greater challenges compared to image generation due to its higher-dimensional complexity and the scarcity of high-quality datasets. Previous approaches mainly rely on autoregressive models [54, 19]. Pioneering works [18, 41] introduce new architectures using 3D U-Net [8] with factorized spatial-temporal attention [2, 16, 5J. Recent works attempt to leverage image diffusion priors in both modeling and data, such as Make-A- Video [45], which proposes fine-tuning from a pre-trained DALLE2 [38] with spatial-temporal modules. However, the challenges of collecting large-scale text-video datasets and the substantial computational overheads still hinder many researchers this field. Recently, two concurrent works [23, 1] propose adapting LDM to the video without fine-tuning, but still persist inconsistent issues.
Diffusion for text-driven editing. Due to clear mapping relationships between data and diffusion latent space, it is efficient to obtain DDPM [15] or DDIM [46] latent of a given image and then perform text-driven editing. To strengthen structural control, P2P [13] and Plug-and-Pla [50] use attention control to minimize changes to unrelated parts. Null-Text Inversion [30] fine-tunes the embedding of null text to improve real image editing. Some works [11, 40, 26] fine-tune TTI models to learn special tokens for personalized concepts and generate related images. While these methods can be applied to video editing frame by frame, is hard for them to ensure temporal consistency. [31, 10] train TTV models to achieve impressive editing performance but are not open-source. At carlier period, one-shot training and tuning approaches have been popular in GANs [43, 59, 58, 12] for various tasks. Tune-A-Viile (TAV) [56] proposes the first tuning strategy LDM [39] for video editting. Based on it, several concurrent works [28, 44, 35, 7, 55] are developed to improve the editing quality, e-g., speed and background preservation. By contrast, are concerned about the essential problem, i... the negative impact of introduced parameters on text-driven editing ability.
3 Preliminaries
Diffusion probability model. DDPM [15] establishes a relationship between complex data dis- tribution and the Gaussian distribution using forward and reverse Markov chains. Following the convention, we denote the random variable of data as zo. and the forward process generates latent variables ,iy through q<>x|=r variance schedule. The reverse process starts from 2r to sample the real data To sequentially through
N / Vo (1 - T)) where {or) is fixed
po 1x,, P) = N N: (1t,, 1, P), (,)).
(1)
Herein, P is an optional condition, Ep can be either predefined constants or trainable parameters associated with the variance schedule, and uo( t, P) Va. (z+ 1-a -o---ooee (z t,, P)), where
V1-a.
ā -i Oi, and Eg is parameterized function that is trained under diffusion loss supervision
Loss(Ea) Eza Ezzo-Paate..e--N(0 - 1tw..///O.......ooo1111111111111 ||e En (ate, ()))
(2)
This makes the reverse process Eq. (1) approach to the Evidence Lower Bound of data distribution. Latent Diffusion Model (LDM). LDM [39] first trains an Autoencoder network consisting of an encoder E and decoder D, where the composition D E approximates the identity mapping. This enables & to compress image data into a low-dimensional latent space. LDM then learns a diffusion model by optimizing Objective (2) in the latent space, with the corresponding text prompt P. In LDM, fg is equipped with a convolutional Transformer-based U-Net architecture, where the attention mechanism plays a critical role in interacting information from different modalities. Specifically, given two features E R8 xd; and E g/2xd5 the attention mechanism [53] obtains three elements Query Q. Key K and Value V by Q = zW,, g, K = cWk, V = cW.. where W,, E Rd xd Wa and W. E Rdd After that, they will interact to generate the transferred feature via
Attention(z. 2, c) VV where MQ,K softmax(QK")
(3)
3

---

Training
Covrariate shift
Diffusion loss
Module
to -mmage Diflusion Model
Module
Random noise
Inrlation
Feature
Restriction
Text-to-Vide Diffusion Model
FFAM
STAM
Module
Pest
"A car moving"
STAM
(a)
(c)
Fine-coorse
Inference
frame intoraction
Quarry
Key
DDIM noise
Text-to-Video Diffusion Model
Carrent
Others
*A sport car is moving, Van Gogh painting style"
Figure 2: Illustration of EI2 for text-driven video editing. (a) One-shot tuning paradigm: Given
video and text prompt, EI² first inflates pre-trained TTI model into TTV model, which is further tuned by minimizing the diffusion loss. (b) Video editing: EI2 uses noise from DDIM inversion [15] and custom text prompt to generate the edited video. (c) Model details: EI enhances the TTI model by upgrading its Self Attention (SA) module to a Fine-coarse Frame Attention Module (FFAM) and introducing novel Shift-restricted Temporal Attention Module (STAM). Unlike Temporal Attention (TA) Module which shifts the input distribution intensively, STAM constrains the output to better align with the subsequent modules. (d) Interactive mechanism in FFAM: For each patch (red star) in the latent, it interacts with the fine patches in the current frame and coarse patches in other frames.
Let define norm as the Layer Norm operator [3] and Linear(z) = =W, + b, where W,. E Rdxd, and æd, Transformer block in Eg has the residual structure shown below
Trausformer = Linear( Linear(Attention AAtenttio((nrrm(2) (orm())))))))))))))()))))
For convenience in later discussions, include the normalization operation of in the equation, although it may not be implemented in practice. LDM incorporates two types of Transformer blocks. The first Self Attention module only receives to compute Transformer(2 2), and the other Cross Attention module takes the latent feature and the text embedding to compute Eq. (4). We also overlook the Dropout [49] in the transformer block, as it freezes during the fine-tuning process.
Notations. For clarity, we abbreviate Self Attention, Cross Attention, and Temporal Attention as SA, CA, and TA, respectively. Accordingly, the Transformer blocks in Eq. (4) associated with these attention mechanisms are named SA Module, CA Module, and TA Module. In this paper, we variables n, I and d to represent the length of temporal, spatial, and feature dimensions, respectively.
4 Methodology
4.1 Overview of 2
Given video clip V (o....., v"} consisting of frames, and an associated text prompt P. text-driven video editing task leverages other text prompts guide style or object changes in the video. While existing large-scale TTI diffusion models [39, 38, 41] have shown exceptional behavior in image editing, they suffer from temporal inconsistency when applied to the video frames.
this paper, we propose a novel approach, named HI?, which inflates the frequent pre- tre-trained TTI diffusion model LDM [39] with well-designed temporal modules to create TTV diffusion model. As depicted Figure 2, the inflated model is optimized to minimize diffusion loss about data pair (V, P) so as capture the temporal motion in video V. while retaining the editing capability of TTI model. To edit the video, we use an altered prompt to guide diffusion reverse process based Eq. (I), where the initial latent zy is obtained by the DDIM inversion [15].
One-shot tuning for video editing.
Model Inflation. Technically, during the inflation process, the initial step involves converting all spatial convolutions in LDM into pseudo convolutions, which allows video data and latent to be processed in the same way batch of images. Furthermore, our approach differs from

---

previous work [56, 15, 35, 28] in two key aspects within Transformer blocks: (a) We introduce a novel Shift-restricted Temporal Attention Module (STAM) that learns motion while solving the covariate shift problem. This addresses the essential limitation in previous works [56, 28]. where the text-driven capability of the TTI model is weakened or deprived after inflation with a 7A Module. (b) Recognizing the complexities over time, we upgrade the existing SA Module into a new Fine- coarse Frame Attention Module (FFAM), Unlike SCA in previous approaches [56, 28] reduces the complexity of attention through causal sampling along the time dimension, our FFAM compresses information along spatial dimension while retaining temporal information to facilitate efficient and effective spatial-tempon attention interaction, leading to better results.
4.2 Theoretical analysis of covariate shift problem
As depicted in Figure I, is apparent that previous works [56, 28] can lead to semantic disparity, i... inconsistencies between the text prompt and the edited video. We attribute this disparity to the covariate shift [21] in the feature space, where modifying the prior distribution of a pre-trained generative model typically leads to a degradation in synthesis. In the case of Tune-A-Vidle [56]. the one-shot data poses difficulties in training meaningful parameters for aligning the output distribution from TA Module with the demands of the subsequent module. Therefore, our analysis investigates the transition of feature distribution in TTV model and provides evidence of covariate shift.
Since Tune-AA Video solely tunes the parameters of Transformers including W3 q in the vanilla modules, and all weights in TA Module, we abstract the problem as changes in distribution after passing through these weights. By considering the structures in Eq. (3) and Eq. (4), which serve as the basis for the SA, CA and TA Modules, we can obtain the following discoveries:
Proposition 1. We assume that for Transformers in Eq. (4), any input feature z E R (or (gnd) is a sample of i..d d-dimensional Gaussian random variables, denoted as rv(2) ~ N ((rro()), Ero(s)) and & = Norm(=), same notation for c. (i) For the output from Eq. (3), the i-th row vector vari- able, Atlention( 2, ")', follows the Gaussian distribution N (PrelV)) W., Ero((y)), (.Q.k-rrr())) where Pro(V)
W.. pro(e), Ere(V) = WTT Eru(e) W., and wQ.k = M[ x 112- (ûi) For the output from Eq. (4), each row vector variable in Transformer(z,, c) follows the Gaussian distribution N N(brote Er
where ('rv(e) Pru(=) WT (rr(V)) b, and E/m(=) -yn(s>) + wq.. ANQ.KWTEL. EV
The proof is provided in Supplementary Materials. We remark that (i) The Gaussian assumption is common and basic in modern neural networks [21, 52], as it is efficient to characterize feature dis- tributions. Here we leverage its properties, particularly additivity, to analyze changes in distributions. (ii) When tuning the weight matrix W, in SA and CA, the mean (1/(()) remains unaffected, while the covariance E E/r(2) is primarily influenced by the modification of wo,k. However, since W,, is well initialized from the pre-trained TTI model, its impact on the covariate shift is considered negligible. (ii) The weights in TA Module, which are randomly initialized and learned from one-shot data, have the potential to greatly influence the feature distribution, whereas it is crucial to ensure the output distribution resembles the input distribution to properly fit the following pre-trained modules.
4.3 Shift-restricted Temporal Attention Module
Based on the analysis, we can explicitly identify that TA Module leads to covariate shift. To address this issue, we propose to improve this module by restricting the distribution shift. In concrete, given that each unit in the spatial grid has temporal feature 2 E Rx when passing TA Module will be transformed by the operation Transformer(z, 2) in Eq. (4). Therefore, our main objective is to minimize the diffusion loss while ensuring TA Modules are constrained in distribution shift, i.e.
[Mro(s) + WT (ru(y) + - Pre(*) Ero(e) + WQ, W. " Pro(
min Loss(Ea); where TA Modules subject to 0
(5)
Eyo(s)
It is worth noting that there exist trivial solutions for TA Modules, such vanishing all parameters. However, we seek to identify the optimal configuration that maximizes the contribution to the task.
Plight from Layer Norm. As an essential operation, nevertheless, we find Layer Norm [3] in the TA Module significantly contributes to the issue of covariate shift. The computation of Layer
Norm across all dimensions, e.g., the mean value Mg nd Eis=1 2 fails to properly center the
5

---

feature along the temporal dimension. This results in zon- values for pro(z) and (ru(V)) leading to covariate shift and conflicts with the objective (5). Experimental results demonstrate a notable discrepancy between Mru(z) and Mzz with the former ranging from le - to 1, while the latter is typically less than le 1e-22 2. Therefore, we can conclude that Layer Norm inadequately centers the feature in the temporal dimension, leading to shift in the mean value. This conclusion is supported not only by theoretical analysis but also by compelling empirical evidence (see Section 5.2 and Figure 4), where even minor refinements in Layer Norm can yield noticeable improvements in the results.
Instance Centering layer. To address the issue of Layer Norm in TA Module, we propose Instance Centering (IC) an effective replacement, which operates temporal data E Raxd by
IC((z)
ym 1
(6)
When applied to batch of temporal data R it is defined to individually operate on each sample. Consequently, the IC' layer guarantees that (lv((),, (Vr(()), and WT (rr(() in Objective (5) vanish in principle. In this way, the new parameters in T', except for the bias, would not affect the mean of distribution. Importantly, IC layer preserves the temporal variance of the data. This deviation from the Instance Norm [52], which normalizes the data along the sequence dimensions, offers not only computational efficiency, resulting in about 20% increase in computational speed, but also plays role in controlling variance shifts associated with subsequent techniques.
Normalized linear mapping. Due to the interaction of multiple variables, the change of temporal variance in TA Module is unavoidable. We can establish the relation using norm properties:
(T/ Ere(e)) s ||&ll [IW,Il?)E II-
When considering the Frobenius norm, this equation indicates that the change in each element of the covariance is controlled by the rescaling coefficient in Norm, matrix norm of Wt in Linear, and W. in TA. Therefore, it implies that we can mitigate the variance shift by regularizing these values. From this we design the following improvements: (a) We do not employ the rescaling operator in IC' since the order of most temporal feature variances is varied from le 2 to 1e- 1, meaning the operation would amplify the variance and magnifying the covariate shift. This observation also supported by qualitative results (see Figure 4). (b) Instead of adding a regularization term to loss, we employ a more efficient technique called spectral normalization (SN) [29] to rescale W = mmmmmssW where max is the spectral norm of matrix. SN has been widely used in various tasks and offers better stability and theoretical guarantees compared to other weight normalization [42]. In our approach, we leverage SN to redefine the Attention and Linear layers as follows:
Attention 2, c) MQ. k zWe, and Lincar(2) zW + b.
(8)
STAM. The STAM layer is extension of Transformer in Eq. (4) and formulated as follows: STAM(2) + Liner((Atenntoon//((e)...C
We can prove that the feature vector in STAM(=) follows distribution N ((Pre(z) + E Ex)(s)) where |I < [ on Mre(s) is negligible under a small learning rate. Therefore, this design effectively alleviates the covariate shift in theory, enabling the improvement of semantic disparity in practice.
2|]Dro<>)] under the spectral norm. Since the bias is initialized from zero, the impact
4.4 Fine-coarse Frame Attention Module
Considering the set 2 {2d E Rixdyn j=1- which comprises features of all frames, Tune-A- Video introduces a Sparse-Casual Attention (SCA) mechanism to enhance the SA with sparse-temporal interaction, where only the first frame and its immediate previous frame are used for interaction, instead of considering all frames. This significantly reduces the computational complexity from On???) to (2nl?). While SCA is more effective than SA, it is not optimal for video editing tasks, in which global
relationships are crucial to ensure overall coherence. To address this issuc, we propose the Finc-coarse Frame Attention Module (FFAM) as a new solution. FFAM leverages a fine-coarse strategy to incorporate fine-grained information from the current frame with coarse features from other frames. Specifically, FFAM is defined as follows:
FFAM( ( Z) Attention zFc),
(10)
where ZFC = concat[2*, (downsample(2), r))ae) E R fixd

---

Table 1: Quantitative comparison with evaluated baselines. The "Training" refers to the process of
optimization and DDIM inversion [15], and "Memory" refers to the peak footprint of GPU device.
Frame consistency
Textual alignment
Runtime [min]
Memory [Gib]
Method
CLIP ScorefUser Vote
CT.IP Scoret User Vote
Training. Inference.
Trainigg Inference.
Tune-A-Vided [56]
96.64
25.6%
28.72
15.4%
11.0
0.5
9.6
5.3
Video-P2P [28]
96.84
28.3%
28.24
19.5%
19.2
2.0
27.3
19.5
Vid2Vide-zero [33]
95.17
11.2%
29.39
22.4%
2.5
2.5
17.2
23.3
EI
95.94
34.9% %
29.84
42.7%
12.5
0.5
11.0
5,3
where downsample(;. r) rescales the input with ratio 1/r². The design of FFAM respects the rule of SA to leverage data in the current frame, and uses coarse features sampled from other frames as an aid. contrast to SCA, which samples data over time to improve efficiency, FFAM adopts a robust way that condenses data along the spatial dimension, while retaining the structural relationships in both spatial and temporal dimensions. is worth noting that FFAM reduces the complexity to O 1("-1t"" n In experiments, when is set to 2 or 4, FFAM obtains improved temporal consistency while achieving comparable speed to SCA. This trade-off between SA and global spatial-temporal interaction enables FFAM to strike a balance between efficiency and consistency in video tasks.
5 Experiments
Implementation details. Our implementation of EI is based on the stable diffusion v1-4 1-4 frame- work'. We keep the Autoencoder of LDM frozen and tune We of the FFAM and CA Modules, and all parameters of STAMs. We follow previous works [56] to perform tuning on 8-frame videos of size 512 x 512. We utilize the A optimizer with a learning rate of 3e - for a total of 500 steps. During inference, we initialize the model from the DDIM inversion[46) and set the default classifier-free guidance [17] to 7.5. All experiments are conducted on an NVIDIA Tesla V100 GPU.
Baselines. In view of the limited accessible resource in this field, we select the three representative methods as baselines. (1) Tune-A-Vinn -Video [56]: The current SOTA in the field and the conventional baseline for related works. (2) Vid2Vid-zer [55]: A TTI-based zero-shot video editing method without fine-tuning. (3) Video-P2P [28]: An improvement method over Tune-A-Vidd via using P2P [13] and Nuxl-Text [30]. Experiments are conducted with their official codes and configurations.
Datasets.
Following previous works [56, 28], we collect videos from the DAVIS dataset [34] for comparison. We also gather face videos from the Pexels website to assess the fine-grained editing in the face domain. We utilize captioning model [27] to automatically generate the text prompts.
5.1 Comparison
Qualitative results. We present qualitative comparison results between EIe and baselines in Figure 3. Due to limited space, additional results are provided in the Supplementary Materials. We evaluate the performance of the methods using different general scenarios including object and style editing. The following conclusions can be drawn from the results: (1) Tune-A- A-Video [56] tends to performs better in coarse-level object replacement but poorly terms of style. This is evident in its attempt to forcefully fit the "surfing" action for object replacement, resulting in noticeable artifacts in the generated results. It also fails to produce the desired stylistic effects and still maintains the photorealistic style of the original videos. (2) Video-P2P [28], based on Tune-A- VVieoo, improves consistency with the original video through attention control but struggles with global style as before. (3) Vid2 Vid-zero [55] does not use the TA Module and adopts the parameters of the source model entirely, avoiding the issue of covariate shift. However, the lacking of fine- lin-ttnnig procedure leads to the edited results could not faithfully preserve the characteristics of source videos. (4) Compared to previous works, our approach combines the fidelity to the original video from Tune-A- (une-A-Video with the editing power of the TTI model from Vid2 For instance, in the case of "surfing", Er? effectively preserves the motion of source videos while creating frames suitable for characters, instead of blindly copying the source motion and introducing artifacts. Our experiments demonstrate that El2 exhibits significant visual advantages over the competitors.
Quantitative results. In line with previous studies [56, 28, 55], we evaluate methods using CLIP score [14] and user study to assess frame consistency and textual alignment. (1) CLIP score: To
Stable Diffusion: http://////////aggoooooooo/o///a http://giiaaatooooooo/oo////aaaa diw.lleo-dfffuino----eee
7

---

man--pider--on surfing, painting style
A man talking, Pixar 3D style
Video
Tune Aone-A-Mee Viieoo
Video PP
Vid2Vid virrr zero
E1
Figure 3: Comparison EI2 with Tune-A- Video [56], Video-P2P [28], and Vid2Vid-zero [55]. The black text in the first line represents the video caption, while the green text indicates the prompt for revision. The outputs generated by EI2 exhibit greater temporal continuity and semantic consistency.
evaluate frame consistency, we calculate CLIP [37] image embedding for all frames in the edited videos and report the average cosine similarity between pairs of video frames. For measuring textual faithfulness, we compute the average CLIP score between frames of output videos and corresponding edited prompts. We employ 15 videos from the dataset and edit them in terms of object and style, resulting in a total of 75 edited videos for each model. Table details the average results, which indicate that our method exhibits the strongest ability to achieve semantic alignment. Nevertheless, is important to note that the automatic evaluation may not align perfectly with human perception [3 1]] thus should only serve as reference scores. (2) User study: We present 30 pairs of videos and text prompts to volunteers and ask them to vote for edited videos with the best temporal consistency and those that best match the textual description. We collect valid votes from 50 volunteers. Table illustrates that EI2 receives the highest number of votes in both aspects, indicating that our method achieves superior editing quality and is preferred by users in practice.
Resource consumption. The practical resource consumption of video editing is significant con- sideration. The results in Table show that El2 introduces only marginal increase in computational overhead compared Tune-A-Vidde This indicates that our approach effectively balances the trade-off between resource utilization and editing performance, making a practical and efficient solution for video editing task.
while achieving a substantial improvement in performance.
5.2 Ablation Study
Restricting covariate shift. As the of our research, evaluate the impact of the covariate shift problem on video editing performance. According to the theoretic analysis in Sec. 4.2, we focus on the Layer Norm within the TA Module of Tume-A-Vider and make modifications while keeping the rest of the architecture unchanged. These modifications are performed based on the official code of Tune-A-Video. Figure 4 presents the results of these experiments. that the vanilla Tume-A-Vidle tends to restrain the textual guidance, leading to inadequate editing performance. Removing the rescaling parameters of the Layer Norm or completely eliminating the module itself does not effectively alleviate the covariate shift problem. However, when we replace the module with the Instance Norm [52], notable improvement performance achieved. Moreover, incorporating IC and weight normalization further enhance the results. These findings indicate significant relationship between covariate offset and semantic disparity, affirming the effectiveness of approach in mitigating these challenges.
the second column, is observed

---

Video
Layer Norm
Layer Norm"
Identity
Inst. Norm
Inst. Norm
Inst. Cent.
STAM
+ Painting
+ crochet
Figure 4: Ablation on the restriction of covariate shift. "*" means that the operator does not use learnable weights. Replacing Layer Norm with instance-based operator can obviously mitigate the covariate shift and has very distinct effect on improving textual guidance.
car driving the road cartoon style
Sparse-cusual Attention Module
Fine-coarse Frame Attention module
Fine-coarse Frame Attention Module r = 2
Figure Ablation on the spatial-tempont attention mechanism. Compared with previous SCA Module [56], our FFAM can enhance the temporal consistency even with a large scale ratio.
Fine-coarse Frame Attention. Figure 5 illustrates the effectiveness the fine-coarse interaction in improving temporal consistency. When combining the SCA Module with STAM, it cannot effectively learn temporal information, resulting in obvious flickering. this issue, ensuring superior editing quality and temporal coherence in our model. It is important to note that the scale ratio used in FFAM needs to be within the range of the L.DM feature map, which ranges from 8 to 64. Moreover, applying full approximately 32GiB memory footprint and a significant amount of time (~~45min) for training, which is not practical in application. In this regard, our FFAM provides flexible alternative to interaction, addressing the limitations associated with resource consumption while still achieving satisfactory results in terms of temporal consistency.
contrast, FFAM effectively addresses
attention to 8 frames requires
6 Conclusion
In this paper, we propose the EI2 model to tackle inconsistent editing results in expanding TTI models for video editing. Our analysis reveals that semantic inconsistency arises from new parameters in temporal attention, leading to covariate shift and compromising editing capability. The ET2 model comprises two key modules: the Shift-restricted Temporal Attention module (STAM) and the Fine- coarse Frame Attention module (FFAM). STAM employs simplified Instance Centering layer to climinate mean shift and normalized attention mapping to constrain variance shift. FFAM, integrated with STAM, effectively utilizes fine-coarse spatial information across frames to enhance temporal consistency, Extensive experiments confirm the validity and effectiveness our approach. We hope our work provides insights for future research in related fields.
Limitations. Our work has two main limitations. Firstly, while we believe that our analysis and proof of the covariate shift provides valuable insights into related issues, heavily relies on the Gaussian assumption, which cannot hold exactly in practical settings. Secondly, although our method demonstrates stronger editing capabilities, it still suffers from failures of temporal consistency in some cases, such as when replacing the "surfing man" with animals. We observe that El2 performs better in most style editing, but the replacement for objects limited to similar attributes. Further investigation and improvement are necessary address these limitations.

---

Acknowledgements
This paper is supported by the Strategic Priority Research Program of Chinese Academy of Sciences (XDA27000000), the National key research and development program of China (2021 YFA 1000403), the National Natural Science Foundation of China (Nos. U19B2040, 11991022) and the Fundamental Research Funds for the Central Universities.
References
[1] An, J., Zhang, S., Yang. H., Gupta, S., Huang, J.B., Luo, J., Yin, X.: Latent-shift: Latent diffu- sion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv: 2304.08477 (2023)
[2] Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., Schmid, C.: Vivit: A video vision transformer. In: ICCV (2021)
[3] Ba, J., Kiros, J.R., Hinton, G.E.; Layer normalization. arXiv preprint arXiv: 1607.06450 (2016)
[4] Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., Dekel, T.: Text2live: Text-driven layered image and video editing. arXiv prerpint arXiv: 2204.02491 (2022)
[5] Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video under- standing? In: ICML (2021)
[6] Brock, A., Donahue, J., Simonyan, K.: Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809. 1 1096 (2018)
[7] Ceylan, D., Huang, C.H.P., Mitra, N.J.: Pix2video: Video editing using image diffusion. arXiv preprint arXiv: 2303. 12688 (2023)
[8] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: MICCAI (2016)
[9] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. In: NeurIPS (2021)
[10] Esser, P., Chiu, J., Atighehchian, P., Granskog, J., Germanidis, A.: Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv: 2302.03011 (2023)
[11] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, An image is worth word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv: 2208.01615 (2022)
[12] Gur, S., Benaim, S., Wolf, L.: Hierarchical patch vae-gan: Generating diverse videos from a single sample. In: NeurlPS (2020)
[13] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to- prompt image editing with cross attention control. arXiv preprint arXiv: 2208.01626 (2022)
[14] Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv: 2104.08718 (2021)
[15] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020)
[16] Ho, J., Kalchbrenner, N., Weissenborn, D., Salimans, T.: Axial attention in multidimensional
transformers, arXiv preprint arXiv: 1912.12180 (2019)
[17] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv: 2207. (2022)
[18] Ho. J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, DJJ.: Video diffusion models. arXiv: 2204.03458 (2022)
[19] Hong, W., Ding, M., Zheng, W., Liu, X., Tang, J.: Cogvideo: Large-scale pretraining for
text-to-video generation via transformers. arXiv preprint arXiv: 2205.15868 (2022)
10

---

[20] Hu, Y., Luo, C., Chen, Z.: Make it move: Controllable image-to-vided generation with text descriptions. In: CVPR (2021)
[21] loffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: ICML (2015)
[22] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: CVPR (2020)
[23] Khachatryan, L.,, Movsisyan, A., Tadevosyan, V., Henschel, R., Wang, Z., Navasardyan, S., Shi, H.: Text2video-zeno: Text-to-image diffusion models zero-shot video generators. arXiv preprint arXiv: 2303. 13439 (2023)
[24] Kingma, D., Salimans, T., Poole, B., Ho, J.: Variational diffusion models. NeurIPS (2021)
[25] Kingma, D.P., Welling, M.: Auto-encoding variational bayes. arXiv preprint arXiv: 1312.6114 (2022)
[26] Kumari, N., Zhang, B., Zhang, R., Shechtman, E., Zhu, J.Y.: Multi-concept customization of text-to-image diffusion. arXiv preprint arXiv: 2212.04488 (2022)
[27] Li, J., Li, D., Savarese, S., Hoi, S.: 22ip-2: Bootstrapping angguageimmgge pre-training with frozen image encoders and large language models. arXiv preprint arXiv: 2301.12597 (2023)
[28] Liu, S., Zhang, Y., Li, W., Lin, Z., Jia, J.: Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv: 2303.04761 (2023)
[29] Miyato, T., Kataoka, T., Koyama, M., Yoshida, Y:: Spectral normalization for generative adversarial networks. arXiv preprint arXiv: 1802.05957 (2018)
[30] Mokady, R., Hertz, A., Aberman, K., Pritch, Y., Cohen-Or, D.: Null-text inversion for editing real images using guided diffusion models. arXiv preprint arXiv: 2211.09799 (2022)
[31] Molad, E., Horwitz, E., Valevski, D., Acha, A.R., Matias, Y., Pritch, Y., Leviathan, Y.,
Hoshen, Y.: Dreamix: Video diffusion models are general video editors. arXiv preprint arXiv: 2302.01329 (2023)
[32] van den Oord, A., Vinyals, O., Kavukcuoglu, K.: Neural discrete representation learning. In: NeurIPS (2018)
[33] Parmar, G., Singh, K.K., Zhang, R., Li, Y., Lu, Zhu, J.Y.: Zero-shot image-to-image translation. arXiv preprint arXiv: 2302.03027 (2023)
[34] Pont-Tuset, J., Perazzi, F., Caclles, S., Arbeláez, P., Sorkine-Homunn, A., Van Gool, L.: The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: 1704.00675 (2017)
[35] Qi, C., Cun, X., Zhang, Y., Lei, C., Wang, X., Shan, Y., Chen, Q.: Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv: 2303.09535 (2023)
[36] Qiu, X., Han, C., Zhang, Z., Li, B., Guo, T., Nie, X.: Diffbfrr: Bootstrapping diffusion model towards blind face restoration. arXiv preprint arXiv: 2305.04511 (2023)
[37] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell,
A., Mishkin, P., Clark, J., al.: Learning transferable visual models from natural language supervision. In: ICML (2021)
[38] Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M.: Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv: 2204.06125 (2022)
[39] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR (2022)
[40] Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., Aberman, K. Dreambooth: Fine tuning
text-to-image diffusion models for subject-driven generation, arXiv preprint arXiv: 2208. 12242 (2022)
11

---

[41] Saharia, C., Chan, W., Saxena, S., Li. Li. L., Whang. J., Denton, E., Ghasemipour, S.K.S., Ayan, B.K., Mahdavi. S.S., Lopes, R.G., al.: Photorealistic text-to-image diffusion models with deep language understanding. arXiv preprint arXiv: 2205. 1108711487 (2022)
[42] Salimans, T., Kingma, D.P.: Weight normalization: A simple reparameterizatioo to accelerate training of deep neural networks. In: NeurIPS (2016)
[43] Shaham, T.R., Dekel, T., Michaeli, T.: Singan: Learning a generative model from a single natural image. In: ICCV (2019)
[44] Shin, C., Kim, H., Lee, C.H., Lee, S.g., Yoon, S.: Edit-a-video: Single video editing with object-aware consistency. arXiv preprint arXiv: 2303.07945 (2023)
[45] Singer, U., Polyak, A., Hayes, T., Yin, X., An, J., Zhang, S., Hu, Q., Yang, H., Ashual, O., Gafni,
O., et al.: Make-a-video: Text-to-video generation without text-video data, arXiv preprint arXiv: 2209. 14092. (2022)
[46] Song. J., Meng, C., Ermon, S.: Denoising diffusion implicit models. arXiv preprint arXiv: 2010.02502 (2020)
[47] Song, S., Lan, C., Xing, J., Zeng, W., Liu, J.: An end-to-end spatio-temporal attention model for human action recognition from skeleton data. In: AAAI (2017)
[48] Song, Y., Sohl-Dickstein, J., Kingma, D.P., Kumar, A.. Ermon, S., Poole, B.: Score-based generative modeling through stochastic differential equations. In: ICLR (2021)
[49]
Srivastava, N., Hinton, G.E., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: simple way to prevent neural networks from overfitting. In: JMLR (2014)
[50] Tumanyan, N., Geyer, M., Bagon, S., Dekel, T.: Plug-and-play diffusion features for text-driven image-to-image translation. arXiv preprint arXiv: 2211 125522 (2022)
[51] Tzaban, R., Mokady, R., Gal, R., Bermano, A.H., Cohen-Or, D.: Stitch it in time: facial editing of real videos. In: SIGGRAPH Asia (2022)
[52] Ulyanov, D., Vedaldi, A., Lempitsky, V.S.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv: 1607.08022 (2016)
[53] Vaswani, A., Shazeer, N.M., Parmar, N., Uszkoreit, J., Jones, I.,, Gomez, A.N., Kaiser, L.,, Polosukhin, I.: Attention is all you need. In: NeurlPS (2017)
[54] Villegas, R., Babaeizadeh, M., Kindermans, P.J., Moraldo, H., Zhang, H., Saffar, M.T., Castro, S., Kunze, J., Erhan, D.: Phenaki: Variable length video generation from open domain textual descriptions. In: ICLR (2023)
[55] Wang, W., Xie, K., Liu, Z., Chen, H., Cao, Y., Wang, X., Shen, C.: Zero-shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv: 2303.17599
[56] Wu, J.Z., Ge, Y., Wang, X., Lei, W., Gu, Y., Hsu, W., Shan, Y., Qie, X., Shou, M.Z.: Tune-a- video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv: 2212. 1565 (2022)
[57] Yang, S., Jiang, L., Liu, Z., Loy, C.C.: Vtoonify: Controllable high-resolution portrait video style transfer. ACM TOG (2022)
[58] Zhang, Z., Han, C., Guo, T.: Exsingan: Learning an explainable generative model from a single image. In: BMVC (2021)
[59] Zhang, Z., Liu, Y., Han, C., Guo, T., Yao, T., Mei, T.: Generalized one-shot domain adaptation of generative adversarial networks. In: NeurlPS (2022)
[60] Zhang, Z., Liu, Y., Han, C., Shi, H., Guo, T., Zhou, B.: Petsgan: Rethinking priors for single image generation. In: AAAI (2022)
12


Reviews:
[{"review_id": "2MCLW0BvON", "reviewer": "Reviewer_k6zc", "summary": "This paper proposes $EI^2$ for the consistent text-driven video editing. To address the semantic disparity problem, it provides a detailed theoretical analysis and proposes Shift-restricted Temporal Attention Module (STAM) to tackle the covariate shift problem introduced by the temporal consistency module. To improve the temporal consistency, it provides a novel Fine-coarse Frame Attention Module (FFAM) to fully exploit the global temporal information while reduce the computational cost. Extensive comparisons with several available video editing methods shows its superiority and the ablation study also illustrates the effectiveness of the STAM and FFAM.", "questions": "- The duration of the generated video is quite short (about 1s-2s). How does the proposed methods perform when generating long video? (upto 5s-10s)\n- Will the trianing and inference code be released?", "limitations": "The authors have discussed the limitations in the main paper. More potential limirations please refer to Questions.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The authors provides a theoretical analysis about the semantic disparity problem in video editing, which makes the paper more convincing.\n- Based on the detailed analysis, the authors propose a novel Shift-restricted Temporal Attention Module to tackle the covariate shift, which can imrpove the semantic consistency in the edited video.\n- The design of the Fine-coarse Frame Attention Module makes sense, which can balance the performance and the computational cost.\n- The visual comparison shows its superiority over existing publicly available methods while the ablation study further illustrates the effectiveness of the proposed method.\n- The paper is well-presented.", "weaknesses": "- Lack of quantitative results of the ablation study. Though the authors provide the qualitative comparsion result for the abltion study to validate the effectiveness of the STAM and FFAM, it will make the paper more convincing if they provide the some quantitative results for the ablation study.\n- Lack of some technical details. For example, when conducting the temporal attention, how many frames does the model take into consideration?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1__: Lack of quantitative ablation study to validate the effectiveness of the STAM and FFAM.\n\n__A1__: Thanks for pointing out.  We have addressed this by including the quantitative results in the Table 1 of the provided PDF file. \n1. By replacing the TA module with the proposed STAM (+STAM) in the Tune-A-Video baseline, we observe a noticeable increase in the CLIP score for textual alignment. This demonstrates that STAM effectively alleviates the semantic disparity issue by addressing the covariate shift, thus enhancing the quality of video editing.\n2.  Substituting the SCA module in baseline with FFAM (+FFAM) results in a boost in the CLIP score for frame consistency. This indicates that FFAM better captures motion information from the given video frames.\n3. It is worth noting that the CLIP score (_i.e._,  CLIP similarity among frames), which is used to measure frame consistency, is not robust to pose and texture variations and does not accurately reflect temporal consistency. Therefore, we conduct an extra user study: Given 15 pairs of prompts and videos, where object changes are more considered, 10 volunteers vote for results with better temporal consistency between the ablated method and baseline. The results indicate that FFAM substantially improves the temporal consistency, while STAM slightly compromises temporal consistency due to the challenge of maintaining motion for more pronounced editing effects. This user study further reinforces  the effectiveness of FFAM to strengthen the temporal consistency.\n\nWe will add these in the revision.\n\n\n__Q2__: Lack of some technical details, _e.g._, how many frames does the model take into consideration?\n\n__A2__: Here we provide more details to make the technology clearer:\n\n1. Model details:  EI$^2$ inflates TTI to TTV model with the integration of STAM and FFAM modules, for an intuitive comprehension of the technique, we report details of these two modules: \t\n    a) FFAM strictly aligns the illustration in Eq. (10) and Figure 2, which half down-samples features of all other frames by the bilinear interpolation, thus does not introduce new parameters. \t\n    b) In STAM, the frame length is the same as the frames of video. Power iteration algorithm is used to fast approximate the max singular value, thus the spectral normalization does not bring extra time cost. For avoiding the illness of small singular value on normalization to magnify variance, we only normalize the matrix with spectral norm larger than 1 in practice.  \n2. Training details: We perform the training at resolution of 512 × 512. Following Tune-A-Video, we only fine-tune unfrozen parameters including query weight $W_q$ of FFAM and cross attention, and all weights in STAM. The optimization runs 500 steps on a learning rate 3e−5 (batch size = 1) with AdamW optimizer.  \n3. Inference details: We first use DDIM inversion to get the latent code the video, then the edited results are obtained via the DDIM sampling with given prompt after 50 steps. Thanks to STAM, we can set the classifier-free guidance scale to a small value, while Tune-A-Video works with a large value. In our paper, it is set to 7.5 same as the default value of LDM.      \n\n__Q3__: How does the proposed methods perform when generating long video?\n\n__A3__: EI$^2$ demonstrates impressive results for long videos. We have included examples of editing for a long video sequence (about 8s) in the attached rebuttal PDF, where the video is sampled with a sample rate 2, as same as the setting in Tune-A-Video, to get 36 frames in totally. These examples underscore the capability of EI^2 to maintain its semantic and temporal consistency when applied to longer video sequences and various tasks (_e.g._, object, style, background changes), surpassing the baseline Tune-A-Video.  We will provide more examples in the revision.\n\n__Q4__: Will the training and inference code be released?\n\n__A4__: Yes, we will make the code publicly available.\n\n\n\n"}, {"review_id": "7e0gMZLkqA", "reviewer": "Reviewer_eBTE", "summary": "The paper provides a comprehensive theoretical analysis of the covariate shift problem in current one-shot fine-tuning methods. In response, the authors propose a novel Shift-restrict Temporal Attention Module, designed to enhance the editing capability. Additionally, they introduce a Fine-coarse Frame Attention Module, which aims to improve temporal consistency. The visual examples presented in the paper demonstrate promising performance.", "questions": "1. In L184, the authors state that randomly initializing the Temporal Attention (TA) module could potentially compromise its effectiveness in terms of editing capability. Considering this, would it be beneficial to explore the possibility of initializing the TA module from well-trained Spatial Attention (SA) modules, as shown useful in other works?\n2. Upon reviewing the video comparisons provided in the supplementary materials, it becomes evident that several results from Video-p2p differ significantly from the demonstrations shown on their official website. For instance, the 'Spider-Man' riding case and the 'car driving' case display instances where the inversion step does not appear to have succeeded. I recommend investigating the code and corresponding results to address this discrepancy.\n3. A minor suggestion regarding Figure 2: it may be preferable to present the 'Inflation' operation without the surrounding box. Including the box might lead to the misconception that this is a parametric module, which could be misleading for readers.\n4. Only style manipulation cases are provided. Could the proposed method successfully address the pose/shape modification and object addition/remocal situations?", "limitations": "The authors have discussed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper offers a comprehensive analysis of the underlying causes of semantic disparity in fine-tuning-based video editing approaches. Through a rigorous examination under the Gaussian assumption, the authors effectively demonstrate the occurrence of covariate shift. Furthermore, the proposed solution put forth in the paper is both logical and well-founded.\n2. The paper showcases temporal consistency in the edited results, which is a noteworthy accomplishment.\n3. The paper demonstrates a commendable level of clarity and cohesiveness in its presentation, making it easily comprehensible and accessible to readers.", "weaknesses": "1. The analysis of covariate shift in this paper relies fundamentally on the assumption of i.i.d. Gaussian distribution. However, given the complex nature of real-world scenarios, this assumption may be overly restrictive. It would be valuable to incorporate a more generalized analysis to account for a broader range of situations.\n2. While it is acknowledged that commonly used CLIP-based metrics may not align perfectly with human perception, they remain the most objective evaluation metrics available. Table 1 reveals that the CLIP score for temporal consistency of the proposed method is inferior to that of previous works. Could you please provide further details on the user study, particularly in terms of ensuring sufficient objectivity and randomness?\n3. The visual examples depicted in Figure 3 highlight certain deficiencies in the fidelity of the proposed method to the original videos. For instance, the 3rd and 4th frames of the 'Spider-Man' case exhibit noticeable motion and pose disparities compared to the original frames. Additionally, the 4th frame in the 'man talking' case shows an open mouth, deviating from the original video.\n4. In addition to assessing temporal consistency and textual alignment, it is crucial to incorporate fidelity evaluation metrics such as PSNR/SSIM/LPIPS as reported in Fatezero and Video-p2p. It would be beneficial to include these metrics in the analysis to provide a more comprehensive assessment of the proposed method.\n5. The ablation study only includes one example to demonstrate the effectiveness of the proposed designs, which may not be sufficient. It is recommended to include a quantitative evaluation to strengthen the analysis and provide a more robust assessment of the proposed approach.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1__: It would be valuable to generalize analysis to account for a broader range of situations.\n\n__A1__: Thanks for pointing out. We recognize this limitation (Line 335-337), and explore two perspectives for generalization:\n\n1. Distribution Transition Perspective: In the proof, we approach the covariate shift by analyzing how the quasi-linear structure in the residual-based transformer block affects the distribution. This allows us to extend the analysis to various \"good\" distributions that exhibit \"linearity\" and \"additivity\" (Line 80). Thus, if the input follows a \"good\" distribution (_e.g._., Gaussian, Gamma, Uniform), according to the proof steps we can also derive a rigorous conclusion for that case.  \n\n2. Moment Change Perspective: Recognizing that moments encapsulate vital distribution information. we emphasize their role in the analysis.  By prioritizing the first two moments (mean and variance) and downplaying higher-order moments, we can align them to mitigate covariate shift. This simplifies STAM's goal from preserving distribution to preserving moments. This perspective applies beyond Gaussian distribution, directly generalizing our analysis to real-world scenarios.\n\nWe will add more about these in the revision.\n\n\n__Q2__:  CLIP score for Frame consistency, and user study details.\n\n__A2__:   We would like to clarify the distinction between the \"CLIP score for frame consistency\" and conventional \"temporal consistency (Please refer to A2 for Reviewer 8VBg). We try to align our user study with that of TAV:     \n- The edited videos from various methods are presented in a random order. The volunteers are required to select their favorite results that have the best temporal consistency and best aligns with the text prompt in vision.  Since previous works do not release their test prompts, we collect 30 commonly-used prompts in recent works for style, object and background changes.  We recruit 50 volunteers, which are from different regions on the networks. Compared with 5 participants in Tune-A-Video, we consider our survey has better randomness and objectiveness.  \n\n\n__Q3__: Deficiencies in the motion fidelity of the method to the original videos.\n\n__A3__: We consider this is a shared challenge across recent works (_e.g._, the \"dancing man\" in the teaser of TAV),  due to the inflated T2V models :\n1. inherit the limited editing ability of pre-trained LDM, which struggles with preserving delicate details. Even methods like DreamBooth for frame-by-frame editing encounter challenges in maintaining faithful poses, as observed in experiments with the \"surfing man\".\n2. cannot perfectly disentangle motion $m$ and content $c$ of video. When given a text $t=(t_c, t_m)$, TAV/EI$^2$ essentially \"generates\" the edited video, thus the fidelity of edited pose highly relies on whether the distribution $p(m|t_m)$ or $p(m|t)$ is deterministic.  The inflated model may learn $p(c,m|t) = p(m|c,t) p(c|t)=1$ under the diffusion loss, while it is hard to ensure $p(m|t)=1$, which is true if and only if $m$  and $c$  are totally independent in tuned LDM space.  This means that changes in $t_c$ could lead to fluctuations in the \"non-deterministic\" distribution $p(m|t)$ or $p(m|c,t)$, affecting $m$ during sampling.\n\nActually, our results appear natural, with motion closely resembling the originals, evident in surfing and mouth motion in SM video, also new examples in the provided PDF. As explained in L278-281, this validates that EI$^2$'s $p(c|t)$ maintains alignment with the pre-trained LDM, and $p(m|c,t)$ has effectively captured meaningful information.  In contrast, TAV may fail in $p(c|t)$ to produce artifacts or semantic disparity.\n\n\n__Q4__: Fidelity metrics PSNR/SSIM/LPIPS to provide a more comprehensive assessment.\n\n__A4__: Thanks for pointing out. We try to follow Video-P2P to compute LPIPS/SSIM for structure preservation, and PSNR for reconstruction quality. Here the PSNR scale may differ from that in Video-P2P. EI$^2$ can perform better than TAV and Vid2vid-zero in these metrics, and it is also extensible to couple with Video-P2P to enhance the fidelity.\n\n|       | Tune-A-Video | Vid2vid-zero |EI$^2$ |  Video-P2P | EI$^2$+P2P |\n| ----- | ------------ | ------------ | --------- | ------ | ---------- |\n| PSNR↑  | 21.9         | 15.5         | 23.7  | 28.4     | 28.4       |\n| LPIPS↓ | 0.40         | 0.42         | 0.38   | 0.35      | 0.34       |\n| SSIM↑  | 0.58         | 0.57         | 0.60   |0.64      |  0.64       |\n\n\n__Q5__: Quantitative evaluation of the ablation study.\n\n__A5__: Thanks for point out. We have provided this in PDF file. Please refer to A1 for Reviewer k6zc. \n\n\n\n__Q6__: Initializing the TA module from well-trained Spatial Attention (SA) modules.\n\n__A6__: Thanks for pointing out. We try to experiment with this initialization, but it seems to face similar shortcomings. We consider this does not conflict with our analysis of covariate shift.   Additionally, we aspire for the TA module to concentrate more on acquiring temporal information with random initialization, akin to Divided Space-Time Attention in TimeSformer[1], instead of utilizing spatial information as strong guidance. \n\n[1] Bertasius G, Wang H, Torresani L. Is space-time attention all you need for video understanding? ICML 2021.\n\n\n__Q7__: Investigating the code of Video-p2p.\n\n__A7__: After carefully checking, we ensure that we follows their official codes and settings. We think the diverse reproduction is probably caused by different devices and seeds. In actuality, for the sake of impartiality, the outcomes of all comparative approaches are derived from the optimal results under multiple random seeds. \n\n\n__Q8__: Better presentation for Figure 2.\n\n__A8__: Thanks for pointing out.  We will refine this in the revision.\n\n\n__Q9__: Could the proposed method address various situations?\n\n__A9__:  Yes, EI$^2$ can perform well, even better than TAV in these situations. We show some examples in the provided pdf. \n\n\n\n\n\n\n"}, {"review_id": "dxmcvSLoZI", "reviewer": "Reviewer_8VBg", "summary": "This paper claims that existing models suffer from inconsistent video editing results due to covariate shift caused by additional modules for learning temporal information. It proposes a novel model addressing this issue through two modules: Shift-restricted Temporal Attention (STAM) and Fine-coarse Frame Attention (FFAM). STAM replaces Layer Normalization with an Instance Centering layer and utilizes a normalized mapping attention layer to preserve temporal feature distribution and constrain variance shift. FFAM incorporates fine-coarse spatial information to further improve temporal consistency. Experimental results demonstrate the effectiveness of the proposed EI model.", "questions": "1. [Minor] The title might be a bit too high-level since this paper just points out and targets at one aspect of the cause in temporal consistency issue. Addressing the specific technical points might be a more proper and solid claim.", "limitations": "1. There are many latest related work on this topic emerging recently. Maybe many of them are concurrent work. It is not mandatory, but very good to include more comparisons with some most recent work, in terms of methodology, results and other technical differences, especially for those that have released all code and pretrained models.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper in detail analysed the issue of covariate shift in temporal consistency, which is a good perspective to cut in, and the proposed method accordingly tackled this issue.\n\n2. Figures 4 and 5 show ablation experiments that illustrate how each component contribute to the final performance.", "weaknesses": "1. Figure 3, as well as not few examples in the supplementary video (especially the first half almost), seems to show that the other methods aren't mostly doing well on text information, while having similar/comparable temporal consistency. This seems not to be consistent with the main target and point of the proposed method. In Table 1, the main metric, frame consistency in terms of CLIP score, doesn't show the proposed method's advantange either. I would like to learn more discussions here.\n\n2. The examples in the supplementary videos are better to be listed together with the same input and all other methods, instead of separately one by one. Current layout only has one comparing method per slide, which is not clear and hard to compare with all.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1__: Results in table1 and Figure 3 are not consistent with the main target and point of the proposed method. \n\n__A1__: In our work, the consistency refers to (L44-46) both semantic alignment of text and edited video, and temporal consistency in cross frames. \nIn this context, the results and your observations align with our intended point. Specifically, the observed semantic inconsistency in Tune-A-Video highlights a practical limitation that our approach seeks to overcome. To elaborate:   \n1. Our theoretical analysis associates this issue with covariate shift stemming from newly introduced temporal attention, prompting us to propose STAM as a solution.\n2. Besides, we introduce FFAM, an efficient enhancement to the prior sparse-casual attention module, which improves frame consistency.\n\nIn light of these contributions, the enhanced textual alignment indeed constitutes a part of our broader advancements.\n\n__Q2__: In Table 1, the main metric, frame consistency in terms of CLIP score, doesn't show the proposed method's advantage.\n\n__A2__: Aligned with the objectives of video editing, we aim to enhance semantic alignment of text prompts and edited results, while maintaining temporal consistency. Our proposed EI$^2$ method demonstrates advantageous performance in both these crucial aspects:\n\n1. Advancement in semantic consistency: Leveraging CLIP score (L287-288) and a comprehensive user study, our method consistently achieves superior results in terms of semantic alignment between edited videos and text prompts. This underscores the effectiveness of our STAM module.\n2. Advancement in temporal consistency:  Similarly, we conduct the automatic metric, _i.e._, CLIP score for frame consistency   (L285-286) and user study to evaluate the temporal consistency. We would like to clarify the inferior of CLIP score from the distinction between CLIP score and temporal consistency:\n   - **CLIP Score for Frame consistency vs. Temporal Consistency**. It is important to note that the CLIP score primarily evaluates high-level semantic consistency among frames. In contrast, \"temporal consistency\" traditionally assesses low-level details in cross-frame changes. Thus, the CLIP score may not fully account for artifacts or noise in frames, which are pivotal for visual perception, and can be influenced by multiple factors like content, pose, and texture. This also has been noted in previous works like Dreamix and vid2vid-zero. In this way, while the CLIP score remains a valuable objective evaluation metric, it might not capture temporal nuances accurately.\n\n    To account for this, additional user study is conducted to validate temporal consistency, and demonstrates the improvements of EI$^2$. We depict some examples in the provided PDF file to prove that our method can perform better than TAV in many cases.\n\nIn summary, with the effective improvement, EI$^2$ can achieve the better semantic and temporal consistency.  \n\n__Q3__: Examples in the supplementary videos is not clear and hard to compare with all.  \n\n__A3__: Thanks for pointing out.  We would have wanted to use this layout to provide high-resolution details, since the compressed file format might largely compromise the visual details. In the upcoming revision, we will take steps to enhance the clarity and ensure that the presented way effectively illustrates comparisons.\n\n__Q4__: About title problem.\n\n__A4__: Thanks for pointing out. We will refine this in the revision.\n\n\n__Q5__: It will be better to include more comparisons with some most recent work.\n\n__A5__: We agree that there are indeed many con-current and pre-print works after the submission, and we are actively incorporating these additional references into the revision. We consider that the methods we have currently compared are representatives of this field: the base method for model inflation and tuning (TAV), the extended approach for structure control (Video-P2P), and the method for zero-shot editing (vid2vid-zero). EI$^2$ is aligned with the first category, as it delves into exploring and revealing the way of inflation.\n\n\n\n"}, {"review_id": "GUmNxrEydP", "reviewer": "Reviewer_FCWG", "summary": "This paper carefully analyzes the problems of current one-shot video editing and proposes two techniques: Shift restricted Temporal Attention Module (STAM) and Fine-coarse Frame Attention Module (FFAM), to address these issues. Several quantitative and qualitative experiments are conducted to verify the effectiveness of the proposed method.", "questions": "see weakness", "limitations": "The authors have adequately addressed the limitation of the proposed method", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors meticulously analyze the cause of semantic disparity for video editing and attribute this to the covariate shift of generative feature space brought by newly added temporal attention. \n2. To address this problem, the authors propose STAM to alleviate the covariate shift and further improve the temporal consistency with another technique FFAM.\n3. The manuscript is well-organized and easy to follow.", "weaknesses": "1. Is there any quantitative ablation comparison for the proposed techniques? It would be better to quantitatively analyze these components besides qualitative visual comparison.\n2. From the visual ablation study, it seems that the normalized liner mapping contributes to the visual quality a lot. Is there any ablation that only applies normalized linear mapping?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1__: It would be better to quantitatively analyze components besides qualitative visual comparison.\n\n__A1__: Thanks for pointing out.  We have addressed this by including the quantitative results in the Table 1 of the provided PDF file. \n1. By replacing the TA module with the proposed STAM (+STAM) in the Tune-A-Video baseline, we observe a noticeable increase in the CLIP score for textual alignment. This demonstrates that STAM effectively alleviates the semantic disparity issue by addressing the covariate shift, thus enhancing the quality of video editing.\n2.  Substituting the SCA module in baseline with FFAM (+FFAM) results in a boost in the CLIP score for frame consistency. This indicates that FFAM better captures motion information from the given video frames.\n3. It is worth noting that the CLIP score (_i.e._,  CLIP similarity among frames), which is used to measure frame consistency, is not robust to pose and texture variations and does not accurately reflect temporal consistency. Therefore, we conduct an extra user study: Given 15 pairs of prompts and videos, where object changes are more considered,10 volunteers vote for results with better temporal consistency between the ablated method and baseline. The results indicate that FFAM substantially improves the temporal consistency, while STAM slightly compromises temporal consistency due to the challenge of maintaining motion for more pronounced editing effects. This user study further reinforces  the effectiveness of FFAM to strengthen the temporal consistency.\n\n__Q2__: Is there any ablation that only applies normalized linear mapping?\n\n__A2__: We illustrate the qualitative and quantitative experiments of normalized linear mapping in the provided pdf. \n1. As shown in Table 1, directly leveraging Normalized Linear Mapping  (the column w/o $IC$) can obviously improve the textual alignment score ($29.02$) compared with baseline model (+FFAM $28.50$). The results also show that $IC$ layer contributes more ($29.51$) to the alleviation, and the effects of two modules can add up to each other ($29.84$). \n2. As shown in Figure 4, Normalized Linear Mapping is also useful to enhance the visual effects, but the improvement is usually no more than $IC$ layer. And the effects of two modules can add up to each other.\n\nThese observations are supported and aligned with our theoretical analysis, where the Normalized Linear Mapping mainly alleviate the variance shift. Hence, in STAM it can be considered as an effective complementary of $IC$ layer. \n\n\n\n"}, {"review_id": "ccQkOOJyes", "reviewer": "Reviewer_ipyC", "summary": "Previous works on one-shot video editing might produce results of unsatisfied consistency with text prompt as well as temporal sequence. This paper presents that such inconsistency is caused by covariate shift in the feature space. Therefore, they propose Shift-restricted Temporal Attention Module (STAM) and Fine-coarse Frame Attention Module (FFAM) to improve the performance.", "questions": "My main concern is the frame consistency problem. The quantitative analysis of covariate shift and the explanation of spectral normalization are also important.", "limitations": "I think the frame consistency problem is a limitation that is ignored by the authors.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1 The theoretical analysis of covariate shift is comprehensive and clear.\n\n2 The proposed STAM can well address the style change problem while FFAM can improve the temporal consistency.", "weaknesses": "1 The motivation for STAM is that the inconsistency between text prompt and video lies in the covariate shift between the pretrained T2I model and the fine-tuned T2V model. This paper provides a theoretical analysis of this. However, as the output layer of temporal attn is initialized as zeros and fine-tuned by a small learning rate, the influence of covariate shift may not be much significant. It is better to analyze the covariance quantitatively.\n\n2 To address the covariance shift, in addition to the IC layer, spectral normalization is also proposed. But this paper does not provide an explanation and analysis of spectral normalization.\n\n3 **Frame Consistency**: Fitting to the one-shot video improves the video consistency while STAM aligns the feature to the T2I distribution. Therefore, the two parts are contradictory, and the frame consistency is degraded by this. This is a severe limitation of this paper. In Table 1 and Figure 3, the temporal consistency of this paper is much worse than Tune-A-Video and Video-P2P.\n\n4 The applications are limited. The proposed method mainly focuses on the style transfer problem. Other editing applications, such as background change, are not considered and compared with previous works.\n\n5 Despite providing ablation videos in the supplementary material, the ablation studies in the main paper are not comprehensive. More video frames should be provided to show the influence on frame consistency.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1__: The influence of covariate shift may be small. It is better to analyze the issue quantitatively.\n\n__A1__: Thanks for pointing out. We agree with your point that the implementations have potentially alleviated the covariate shift. But, due to the resonance of multiple variables, the issue is still present and significant. \n- Theoretical analysis. Prop. 1 shows $W_{L}$ and $\\mu_{rv(V)}$ jointly determine mean and value, making it essential to consider both their impacts. Moreover, the weight of output layer has a dimension $h^2$, where $h=320,640,1280$. Hence, its norm will quickly increase from zero even with a small learning rate $lr$, _e.g._, $O(h * lr)$ for F-norm at each iteration.\n\n- Quantitative analysis. With 16 TA modules inserted to the inflated LDM, we measured the L2 difference of mean and variance before and after the TA module. As the absolute value scales vary across training videos and layers, it is hard to provide meaningful statistic for them. We report the empirical range of relative changes (_e.g._, $||\\\\mu_{rv(z)}'-\\\\mu_{rv(z)}||/||\\\\mu_{rv(z)}||\\$) for 10 videos. \n  - For TAV: The relative change in mean reveals that the first 6 layers generally have minor shifts less than 10%. However, subsequent layers almost exhibit larger than 10%, and some even surpass 100%. Variances have similar trends, with the first 6 layers having roughly 1% changes, and subsequent layers having around 10% changes. *Given the cumulative impact of these layer-by-layer shifts*, the effect of covariate shift would be notable.\n  - For EI$^2$: The affect in mean largely decreases, with the first 6 layers changing nearly 1% and the later layers having changes less than 10%. Variance values also demonstrate about half reduction. These align with our theoretical analysis, where it is easier to control mean shift.\n\nHence, the covariate shift indeed exists. The results in the paper further confirm this point. We will add these in the revision.\n\n__Q2__: Explanation and analysis of spectral normalization (SN).\n\n__A2__: As the Line 223-225, SN, i.e., $\\bar{W} =W / \\sigma_{\\text{max}}(W)$, is a simple but effective way to normalize the matrix, where $\\sigma_{\\text{max}}(W)$ is the max singular value, or called the spectral norm. More details please refer to A2 for Reviewer k6zc.  \n\nWe mainly leverage SN to control the variance shift during STAM (Eq. 7). For TAV,  $\\sigma(W_L)$ concentrates in $(0,2)$, while $\\sigma(W_v)$  concentrates in $(3,6)$. After the square product and layer-by-layer cumulation, it will cause significant shift. Hence, SN plays a key role for controlling this no more than 1 in normalized linear mapping (A2 for Reviewer FCWG).\nWe will add these in the revision. \n\n__Q3__: The contradictory between learning video consistency and aligning the feature in STAM.\n\n__A3__: We clarify that the two aspects are not contradictory but rather complementary components that contribute to achieving impressive editing results. STAM still aims to fit the motion of video, but just under extra regularization of feature alignment. \n  1. TA module is crucial for consistency but faces covariate shift issue. Given that the removal of it largely degrades temporal consistency, STAM offers a grounded approach to extend it.\n  2. Our analysis provides insight based on \"certain assumption\". That implies we cannot entirely eliminate, but just \"alleviate\" the issue. For example, while the $IC$ layer theoretically outputs random variables with a zero mean, this process is biased in practice due to limited samples.\n  3. In the above analysis, we still observe shifted mean and variance in output features, indicating that STAM captures temporal motion from the video.\n\n__Q4__. The frame consistency degraded by STAM is a severe limitation of this paper. \n    \n__A4__: We recognize that  STAM may decrease the temporal consistency. Nevertheless, it is not a severe limitation for inconsistency in practice:\n1. It is important to emphasize the goals of text-driven video editing, where both the temporal and frame consistency should be considered. \n\n2. As shown in Table 1 of the PDF, STAM just slightly performs worse than vanilla TA, but brings considerable benefits to improve the textual alignment, thus enhance overall editing effect. \n\n3. The degradation might not solely result from STAM but could also the limitations of one-shot tuning. Evidenced by recent methods like TAV, greater divergence between the editing result and the original video tends to worsen temporal consistency. STAM can produce more pronounced effects, potentially leading to increased inconsistency. For simply edits such as converting \"car\" to \"sport car\", STAM can achieve comparable consistency as TA.\n\n4. The consistency can be further enhanced with other modules without degrading much semantic consistency, such as FFAM. Thus EI$^2$ strikes an improvement for semantic consistency, while still ensuring temporal consistency.\n\n__Q5__: In Table 1 and Figure 3, the temporal consistency of this paper is much worse than Tune-A-Video and Video-P2P.\n\n__A5__:  We would like to clarify the distinction between the \"CLIP score for frame consistency\" and conventional \"temporal consistency (Please refer to A2 for Reviewer 8VBg). And we respectfully disagree that the results of EI are much worse:\n  1. Neither TAV or Video-P2P achieves the desired effect, and artifacts even appear. \n  2. The videos in SM show that our edited videos are visually continuous and natural. \n  3. A large number of user studies to illustrate the advantages of our approach.\n\n__Q6__: Validating method in more general application.\n\n__A6__: We show some examples for object, background changes in the pdf.  EI$^2$ can perform well, even better than TAV in these situations.\n\n__Q7__: Ablation studies in the main paper are not comprehensive.\n\n__A7__: We will refine these in the revision. And we provide extra quantitative ablation study in the PDF. Please refer to our response to Reviewer FCWG. \n\n\n\n\n\n"}]

Author rebuttal:
We thank all the reviewers for their thoughtful reviews and insightful comments on our work!  It is encouraging to hear the feedback from reviewers that:   

1. The paper is commendably easy to follow. It is well-written and presented in terms of organization, logic, commendable clarity and cohesiveness (k6zc,FCWG,eBTE). 
2. The theoretical analysis is insightful. It elaborates the covariate shift problem to cause semantic inconsistency in video editing with a meticulous, clear and rigorous manner. (k6zc, FCWG, eBTE, 8VBg, ipyC). 
3. On this basis, the proposed EI$^2$ is novel and well-grounded, where STAM is well-founded to alleviate this semantic inconsistency (k6zc, FCWG, eBTE, 8VBg), and the way of FFAM to improve the temporal consistency while balancing the computational cost also makes sense (k6zc,FCWG,ipyC).
4. The experiments including comparison and ablation show promising performance and superiority over existing publicly available methods (k6zc,eBTE,8VBg,ipyC).  
  
Moreover, we appreciate and highly value the questions raised by the reviewers for their key guidance to consolidate this work. We have taken these questions seriously, and provided additional experiments and clarifications in responses below.  We hope these responses could address reviewers' concerns. 


Meta-review: This paper introduces the EI$^2$ model to enhance the consistency of Text-to-Image (TTI) diffusion models. The authors identify that added modules in state-of-the-art TTI methods,  such as temporal layers, induce a covariate shift that negatively impacts video edition. They offer a theoretical analysis of the covariate shift and introduce specific modules to mitigate this effect. These modules include the Temporal Attention Module (STAM) and the Fine-coarse Frame Attention Module (FFAM).

The paper received a predominantly positive evaluation, with four reviewers recommending acceptance and one recommending rejection. The main concerns raised by reviewers related to assumptions made in the theoretical analysis, the scale of the user study, and the need for clarifications in experiments, including aspects such as the quality of video editing results, chosen metrics, and potential conflicts between FFAM and STAM. The rebuttal adequately addressed these concerns, and the reviewers maintained their initial recommendations after the rebuttal. Reviewer RipyC maintains a borderline reject recommendation primarily due to concerns about the degraded temporal consistency of the proposed approach.

The AC has thoroughly reviewed the submission and the discussions. The AC considers that the paper offers an interesting analysis of covariate shift in video editing, presenting both theoretical results and relevant solutions to overcome it. They believe that the evidence on temporal consistency could have been strengthened by including comparisons to Neural Layered Atlases baselines such as [A], as suggested in the rebuttal. The FFAM and STAM components seem to introduce a trade-off between consistency and semantic alignment, but the AC considers that the provided results are overall convincing. Therefore, the AC recommends paper acceptance but highly encourages the authors to take into account reviewers' feedback when preparing the final version.

[A] Text2LIVE: Text-Driven Layered Image and Video Editing. O. Bar-Tal et al., ECCV 2022.
Decision: Accept (poster)