SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\left\| \Sigma_ {r v (z)} ^ {\prime} - \Sigma_ {r v (z)} \right\| \leq \left\| W _ {L} \right\| ^ {2} \left\| W _ {v} \right\| ^ {2} \left\| \Sigma_ {r v (\hat {z})} \right\|. \tag {7}
$$
When considering the Frobenius norm, this equation indicates that the change in each element of the covariance is controlled by the rescaling coefficient in Norm, matrix norm of  $W_{L}$  in Linear, and  $W_{v}$  in TA. Therefore, it implies that we can mitigate the variance shift by regularizing these values. From this we design the following improvements: (a) We do not employ the rescaling operator in IC since the order of most temporal feature variances is varied from  $1e - 2$  to  $1e - 1$ , meaning the operation would amplify the variance and magnifying the covariate shift. This observation is also supported by qualitative results (see Figure 4). (b) Instead of adding a regularization term to loss, we employ a more efficient technique called spectral normalization (SN) [29] to rescale  $\bar{W} = W / \sigma_{\max}(W)$ , where  $\sigma_{\max}$  is the spectral norm of matrix.  $SN$  has been widely used in various tasks and offers better stability and theoretical guarantees compared to other weight normalization [42]. In our approach, we leverage  $SN$  to redefine the Attention and Linear layers as follows:
$$
\text {A t t e n t i o n} (z, c) = M _ {Q, K} z \bar {W} _ {v}, \text {a n d L i n e a r} (z) = z \bar {W} _ {L} + b. \tag {8}
$$
STAM. The STAM layer is an extension of Transformer in Eq. (4) and formulated as follows:
$$
\operatorname {S T A M} (z) = z + \operatorname {L i n e a r} (\operatorname {A t t e n t i o n} (I C (z), I C (z))). \tag {9}
$$
We can prove that the feature vector in  $\mathrm{STAM}(z)$  follows a distribution  $N(\mu_{rv(z)} + b, \Sigma_{rv(z)}^{\prime})$ , where  $\| \Sigma_{rv(z)}^{\prime} \| \leq 2 \| \Sigma_{rv(z)} \|$  under the spectral norm. Since the bias  $b$  is initialized from zero, the impact on  $\mu_{rv(z)}$  is negligible under a small learning rate. Therefore, this design effectively alleviates the covariate shift in theory, enabling the improvement of semantic disparity in practice.
4.4 Fine-coarse Frame Attention Module
Considering the set  $\mathcal{Z} = \{z^j\in \mathbb{R}^{l\times d}\}_{j = 1}^n$ , which comprises features of all frames, Tune-A-Video introduces a Sparse-Casual Attention (SCA) mechanism to enhance the SA with sparse-temporal interaction, where only the first frame and its immediate previous frame are used for interaction, instead of considering all frames. This significantly reduces the computational complexity from  $\mathcal{O}(n^2 l^2)$  to  $\mathcal{O}(2nl^2)$ . While SCA is more effective than SA, it is not optimal for video editing tasks, in which global spatial-temporal relationships are crucial to ensure overall coherence. To address this issue, we propose the Fine-coarse Frame Attention Module (FFAM) as a new solution. FFAM leverages a fine-coarse strategy to incorporate fine-grained information from the current frame with coarse features from other frames. Specifically, FFAM is defined as follows:
$$
\operatorname {F F A M} \left(z ^ {i}; \mathcal {Z}\right) = \operatorname {A t t e n t i o n} \left(z ^ {i}, z _ {F C}\right), \tag {10}
$$
where  $z_{FC} = \mathrm{concat}[z^i, \{\mathrm{downsample}(z^j, r)\}_{j \neq i}] \in \mathbb{R}^{(\frac{n - 1 + r^2}{r^2} l) \times d}$ ,
Table 1: Quantitative comparison with evaluated baselines. The "Training" refers to the process of optimization and DDIM inversion [15], and "Memory" refers to the peak footprint of GPU device.
Method	Frame consistency	Textual alignment	Runtime [min]	Memory [Gib]
CLIP Score↑User Vote↑	CLIP Score↑User Vote↑	Training↓Inference↓	Training↓Inference↓
Tune-A-Video [56]	96.64	25.6%	28.72	15.4%	11.0	0.5	9.6	5.3
Video-P2P [28]	96.84	28.3%	28.24	19.5%	19.2	2.0	27.3	19.5
Vid2Vide-zero [33]	95.17	11.2%	29.39	22.4%	2.5	2.5	17.2	23.3
EI²	95.94	34.9%	29.84	42.7%	12.5	0.5	11.0	5.3
where  $\text{downsample}(\cdot, r)$  rescales the input with ratio  $1/r^2$ . The design of FFAM respects the rule of SA to leverage data in the current frame, and uses coarse features sampled from other frames as an aid. In contrast to SCA, which samples data over time to improve efficiency, FFAM adopts a robust way that condenses data along the spatial dimension, while retaining the structural relationships in both spatial and temporal dimensions. It is worth noting that FFAM reduces the complexity to  $\mathcal{O}\left(\frac{n-1+r^2}{r^2} n l^2\right)$ . In experiments, when  $r$  is set to 2 or 4, FFAM obtains improved temporal consistency while achieving comparable speed to SCA. This trade-off between SA and global spatial-temporal interaction enables FFAM to strike a balance between efficiency and consistency in video tasks.
Implementation details. Our implementation of  $\mathrm{EI}^2$  is based on the stable diffusion v1-4 framework<sup>3</sup>. We keep the Autoencoder of LDM frozen and tune  $W_{Q}$  of the FFAM and CA Modules, and all parameters of STAMs. We follow previous works [56] to perform tuning on 8-frame videos of size  $512 \times 512$ . We utilize the AdamW optimizer with a learning rate of  $3e - 5$  for a total of 500 steps. During inference, we initialize the model from the DDIM inversion[46] and set the default classifier-free guidance [17] to 7.5. All experiments are conducted on an NVIDIA Tesla V100 GPU.
Baselines. In view of the limited accessible resource in this field, we select the three representative methods as baselines. (1) Tune-A-Video [56]: The current SOTA in the field and the conventional baseline for related works. (2) Vid2Vid-zero [55]: A TTI-based zero-shot video editing method without fine-tuning. (3) Video-P2P [28]: An improvement method over Tune-A-Video via using P2P [13] and Null-Text [30]. Experiments are conducted with their official codes and configurations.
Datasets. Following previous works [56, 28], we collect videos from the DAVIS dataset [34] for comparison. We also gather face videos from the Pexels website to assess the fine-grained editing in the face domain. We utilize a captioning model [27] to automatically generate the text prompts.
5.1 Comparison
Qualitative results. We present qualitative comparison results between  $\mathrm{EI}^2$  and baselines in Figure 3. Due to limited space, additional results are provided in the Supplementary Materials. We evaluate the performance of the methods using different general scenarios including object and style editing. The following conclusions can be drawn from the results: (1) Tune-A-Video [56] tends to perform better in coarse-level object replacement but poorly in terms of style. This is evident in its attempt to forcefully fit the "surfing" action for object replacement, resulting in noticeable artifacts in the generated results. It also fails to produce the desired stylistic effects and still maintains the photorealistic style of the original videos. (2) Video-P2P [28], based on Tune-A-Video, improves consistency with the original video through attention control but struggles with global style as before. (3) Vid2Vid-zero [55] does not use the TA Module and adopts the parameters of the source model entirely, avoiding the issue of covariate shift. However, the lacking of fine-tuning procedure leads to the edited results could not faithfully preserve the characteristics of source videos. (4) Compared to previous works, our approach combines the fidelity to the original video from Tune-A-Video with the editing power of the TTI model from Vid2Vid-zero. For instance, in the case of "surfing",  $\mathrm{EI}^2$  effectively preserves the motion of source videos while creating frames suitable for characters, instead of blindly copying the source motion and introducing artifacts. Our experiments demonstrate that  $\mathrm{EI}^2$  exhibits significant visual advantages over the competitors.
Quantitative results. In line with previous studies [56, 28, 55], we evaluate methods using CLIP score [14] and user study to assess frame consistency and textual alignment. (1) CLIP score: To
3Stable Diffusion: https://huggingface.co/CompVis/stable-diffusion-v1-4
A man-spider-man is surfing, painting style
Figure 3: Comparison of  $\mathrm{EI}^2$  with Tune-A-Video [56], Video-P2P [28], and Vid2Vid-zero [55]. The black text in the first line represents the video caption, while the green text indicates the prompt for revision. The outputs generated by  $\mathrm{EI}^2$  exhibit greater temporal continuity and semantic consistency.
A man is talking, Pixar 3D style
evaluate frame consistency, we calculate CLIP [37] image embedding for all frames in the edited videos and report the average cosine similarity between pairs of video frames. For measuring textual faithfulness, we compute the average CLIP score between frames of output videos and corresponding edited prompts. We employ 15 videos from the dataset and edit them in terms of object and style, resulting in a total of 75 edited videos for each model. Table 1 details the average results, which indicate that our method exhibits the strongest ability to achieve semantic alignment. Nevertheless, it is important to note that the automatic evaluation may not align perfectly with human perception [31], thus should only serve as reference scores. (2) User study: We present 30 pairs of videos and text prompts to volunteers and ask them to vote for edited videos with the best temporal consistency and those that best match the textual description. We collect valid votes from 50 volunteers. Table 1 illustrates that  $\mathrm{EI}^2$  receives the highest number of votes in both aspects, indicating that our method achieves superior editing quality and is preferred by users in practice.
Resource consumption. The practical resource consumption of video editing is a significant consideration. The results in Table 1 show that  $\mathrm{EI}^2$  introduces only a marginal increase in computational overhead compared to Tune-A-Video, while achieving a substantial improvement in performance. This indicates that our approach effectively balances the trade-off between resource utilization and editing performance, making it a practical and efficient solution for video editing task.
Restricting covariate shift. As the core of our research, we evaluate the impact of the covariate shift problem on video editing performance. According to the theoretic analysis in Sec. 4.2, we focus on the Layer Norm within the TA Module of Tune-A-Video and make modifications while keeping the rest of the architecture unchanged. These modifications are performed based on the official code of Tune-A-Video. Figure 4 presents the results of these experiments. In the second column, it is observed that the vanilla Tune-A-Video tends to restrain the textual guidance, leading to inadequate editing performance. Removing the rescaling parameters of the Layer Norm or completely eliminating the module itself does not effectively alleviate the covariate shift problem. However, when we replace the module with the Instance Norm [52], a notable improvement in performance is achieved. Moreover, incorporating  $IC$  and weight normalization further enhance the results. These findings indicate a significant relationship between covariate offset and semantic disparity, affirming the effectiveness of our approach in mitigating these challenges.
e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e
Figure 4: Ablation on the restriction of covariate shift.  $*$  means that the operator does not use learnable weights. Replacing Layer Norm with instance-based operator can obviously mitigate the covariate shift and has a very distinct effect on improving textual guidance.
Figure 5: Ablation on the spatial-temporal attention mechanism. Compared with previous SCA Module [56], our FFAM can enhance the temporal consistency even with a large scale ratio.
Fine-coarse Frame Attention. Figure 5 illustrates the effectiveness of the fine-coarse interaction in improving temporal consistency. When combining the SCA Module with STAM, it cannot effectively learn temporal information, resulting in obvious flickering. In contrast, FFAM effectively addresses this issue, ensuring superior editing quality and temporal coherence in our model. It is important to note that the scale ratio used in FFAM needs to be within the range of the LDM feature map, which ranges from 8 to 64. Moreover, applying full spatial-temporal attention to 8 frames requires approximately 32GiB memory footprint and a significant amount of time ( $\sim$ 45min) for training, which is not practical in application. In this regard, our FFAM provides a flexible alternative to spatial-temporal interaction, addressing the limitations associated with resource consumption while still achieving satisfactory results in terms of temporal consistency.
In this paper, we propose the  $\mathrm{EI}^2$  model to tackle inconsistent editing results in expanding TTI models for video editing. Our analysis reveals that semantic inconsistency arises from new parameters in temporal attention, leading to covariate shift and compromising editing capability. The  $\mathrm{EI}^2$  model comprises two key modules: the Shift-restricted Temporal Attention module (STAM) and the Fine-coarse Frame Attention module (FFAM). STAM employs a simplified Instance Centering layer to eliminate mean shift and normalized attention mapping to constrain variance shift. FFAM, integrated with STAM, effectively utilizes fine-coarse spatial information across frames to enhance temporal consistency. Extensive experiments confirm the validity and effectiveness of our approach. We hope our work provides insights for future research in related fields.
**Limitations.** Our work has two main limitations. Firstly, while we believe that our analysis and proof of the covariate shift provides valuable insights into related issues, it heavily relies on the Gaussian assumption, which cannot hold exactly in practical settings. Secondly, although our method demonstrates stronger editing capabilities, it still suffers from failures of temporal consistency in some cases, such as when replacing the "surfing man" with animals. We observe that  $\mathrm{EI}^2$  performs better in most style editing, but the replacement for objects is limited to similar attributes. Further investigation and improvement are necessary to address these limitations.
This paper is supported by the Strategic Priority Research Program of Chinese Academy of Sciences (XDA27000000), the National key research and development program of China (2021YFA1000403), the National Natural Science Foundation of China (Nos. U19B2040, 11991022) and the Fundamental Research Funds for the Central Universities.
[1] An, J., Zhang, S., Yang, H., Gupta, S., Huang, J.B., Luo, J., Yin, X.: Latent-shift: Latent diffusion with temporal shift for efficient text-to-video generation. arXiv preprint arXiv: 2304.08477 (2023)
[2] Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lucic, M., Schmid, C.: Vivit: A video vision transformer. In: ICCV (2021)
[3] Ba, J., Kiros, J.R., Hinton, G.E.: Layer normalization. arXiv preprint arXiv: 1607.06450 (2016)
[4] Bar-Tal, O., Ofri-Amar, D., Fridman, R., Kasten, Y., Dekel, T.: Text2live: Text-driven layered image and video editing. arXiv preprint arXiv: 2204.02491 (2022)
[5] Bertasius, G., Wang, H., Torresani, L.: Is space-time attention all you need for video understanding? In: ICML (2021)
[6] Brock, A., Donahue, J., Simonyan, K.: Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809.11096 (2018)
[7] Ceylan, D., Huang, C.H.P., Mitra, N.J.: Pix2video: Video editing using image diffusion. arXiv preprint arXiv: 2303.12688 (2023)
[8] Çiçek, Ö., Abdulkadir, A., Lienkamp, S.S., Brox, T., Ronneberger, O.: 3d u-net: learning dense volumetric segmentation from sparse annotation. In: MICCAI (2016)
[9] Dhariwal, P., Nichol, A.: Diffusion models beat gans on image synthesis. In: NeurIPS (2021)
[10] Esser, P., Chiu, J., Atighehchian, P., Granskog, J., Germanidis, A.: Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv: 2302.03011 (2023)
[11] Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A.H., Chechik, G., Cohen-Or, D.: An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv: 2208.01618 (2022)
[12] Gur, S., Benaim, S., Wolf, L.: Hierarchical patch vae-gan: Generating diverse videos from a single sample. In: NeurIPS (2020)
[13] Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y., Cohen-Or, D.: Prompt-to-prompt image editing with cross attention control. arXiv preprint arXiv: 2208.01626 (2022)
[14] Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv: 2104.08718 (2021)
[15] Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. In: NeurIPS (2020)
[16] Ho, J., Kalchbrenner, N., Weissenborn, D., Salimans, T.: Axial attention in multidimensional transformers. arXiv preprint arXiv: 1912.12180 (2019)
[17] Ho, J., Salimans, T.: Classifier-free diffusion guidance. arXiv preprint arXiv: 2207.12598 (2022)
[18] Ho, J., Salimans, T., Gritsenko, A., Chan, W., Norouzi, M., Fleet, D.J.: Video diffusion models. arXiv: 2204.03458 (2022)
[19] Hong, W., Ding, M., Zheng, W., Liu, X., Tang, J.: Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv: 2205.15868 (2022)