Fuse Your Latents: Video Editing with Multi-source Latent Diffusion Models

Authors: instructpix2pix + vidrd
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 08:25:09

Abstract: Latent Diffusion Models (LDMs) are renowned for their powerful capabilities in image and video synthesis. 
Yet, compared to text-to-image (T2I) editing, text-to-video (T2V) editing suffers from a lack of decent temporal consistency and structure, due to insufficient pre-training data, limited model editability, or extensive tuning costs. To address this gap, we propose FLDM (Fused Latent Diffusion Model), a training-free framework that achieves high-quality T2V editing by integrating various T2I and T2V LDMs. Specifically, FLDM utilizes a hyper-parameter with an update schedule to effectively fuse image and video latents during the denoising process. 
This paper is the first to reveal that T2I and T2V LDMs can complement each other in terms of structure and temporal consistency, ultimately generating high-quality videos.
It is worth noting that FLDM can serve as a versatile plugin, applicable to off-the-shelf image and video LDMs, to significantly enhance the quality of video editing.
Extensive quantitative and qualitative experiments on popular T2I and T2V LDMs demonstrate FLDM's superior editing quality than state-of-the-art T2V editing methods.

Main paper: 
 introduction: 
 Recently, diffusion models [16, 42] have achieved significant success in image generation [7, 10, 20, 22, 29, 37, 38, 40] , image-to-image translation [39] , text-to-image editing [3, 14, 18, 29] and image inpainting [21, 25, 36] . The development of text-to-image (T2I) generation and editing models has inspired text-to-video (T2V) editing. A straightforward way of T2V editing is to utilize T2I models, considering temporal consistency modeling with motion conditions [4, 6] , temporal propagation [5, 11, 17, 30] and temporal modules [13, 32, 46] . Although T2I models have the advantage of video editing with high-quality structure and appearance, the direct adaptation from T2I models to T2V editing still has limitations in temporal consistency if the edited target has significant changes in shape [32] or motion [30] . Besides, the one-shot tuning strategy [1, 28, 46] suffers from low editability and training efficiency.Therefore, another option is to develop video generation diffusion models [2, 9, 15, 44] for video editing that can achieve decent temporal consistency. Compared to the images generated by T2I models, the videos generated by T2V generation models, despite having good temporal consistency, still lack structural integrity (e.g., textures, low resolution). The main reason is that obtaining high-quality video data is more challenging compared to image data, resulting in a smaller amount of video data available for training T2V generation models [44, 45] . Additionally, to achieve highquality video generation, T2V models require the incorporation of numerous conditional controls during training, such as depth and optical flow [9, 44] , which increases the training cost compared to T2I generation models.To this end, to achieve high-quality video editing, typically evaluated with temporal consistency, text alignment, and aesthetics, the question arises: Given T2I and T2V diffusion models, how can we maximize their potential for superior T2V editing with minimal operations? In this paper, for the first time, we investigate how to ensemble T2I and T2V diffusion models for video editing. We introduce the FLDM (Fused Latent Diffusion Model) by fusing multi-source LDMs, including T2I and T2V latent diffusion models. Drawing inspiration from Prompt2Prompt [14] , certain video editing methods such as FateZero [32] facilitate local editing through attention map manipulation, which can achieve fine-grained editing results. However, this way cannot guarantee robust temporal coherence and suffers from low editability. FLDM replaces attention injection with latent fusion, which realizes an overall and global editing.Concretely, we extract latents from both T2V and T2I LDMs. At every denoising timestep, we apply latent fusion with a hyperparameter to control the proportion of image to video latents. The underlying intuition is that this allows us to balance temporal consistency against structural integrity. The fused latents, enriched with both temporal and semantic information from the respective models, will undergo individual denoising by each model in the subsequent timestep. This suggests that both models can leverage supplemental information-namely, temporal modeling from T2V LDMs and superior generation quality from T2I LDMs. However, the excessive guidance from T2I LDMs may introduce image artifacts during the latent fusion process. To mitigate such side effects, we adjust the fusion ratio for FLDMs during the denoising process with an update schedule. Through extensive experiments with a range of T2I and T2V LDMs, we've found that while T2V LDMs struggle to maintain the structural integerity of the source video, they are greatly enhanced by the complementary strengths of T2I LDMs. Conversely, T2V LDMs excel in ensuring robust temporal consistency in the edited videos, thereby elevating the overall quality of edits performed by T2I LDMs. Figure 1 displays several video editing results using FLDM with InstructPix2Pix [3] and Control-Net [50] .To sum up, we make the following contributions.• We propose multi-source latent fusion, a straightforward yet effective inference strategy that can seamlessly integrate off-the-shelf T2I diffusion models with T2V diffusion models. • To the best of our knowledge, this work is the first to reveal that T2I and T2V diffusion models can complement each other effectively. Specifically, T2I models are essential for providing structural integrity, while T2V models are crucial for ensuring temporal consistency. • Without any tuning process, our method achieves better performance than other counterparts on both qualitative and quantitative evaluations. 
related work 2.1 text-to-image generation: 
 Before the advent of diffusion models, prominent text-to-image (T2I) and text-to-video (T2V) generation works predominantly adopted GAN [35, 49] or auto-regressive architectures [8, 34, 48] . Dhariwal et al. [7] were among the first to present comparisons between GANs and Diffusion Models in the context of image synthesis. Their findings suggested that diffusion models outperformed GANs in terms of diversity, stability of training objectives, and scalability. Subsequently, a series of text-to-image generation models based on the diffusion approach emerged [3, 18, 27, 29, 37, 38, 40, 50] , achieving high-fidelity generation. While T2I diffusion models have achieved remarkable performance in T2I generation, directly adapting these models for T2V editing proves insufficient due to the absence of temporal consistency modeling. Our method combines latents from both T2I and T2V models to enhance temporal consistency without compromising the editing capability of the T2I model. With no additional training or tuning required, the introduced latent fusion strategy is adaptable to any off-the-shelf T2I or T2V diffusion model, enhancing editing quality with minimal adjustments. A prevalent approach to video editing using diffusion models involves adapting T2I models to T2V models [41, 46] . This adaptation often incorporates temporal modules to guarantee temporal consistency. Tune-A-Video [46] integrates temporal attention layers into UNet and conducts one-shot tuning. Meanwhile, Make-A-Video [41] augments the network to encompass temporal information by extending it with spatial-temporal modules. However, fine-tuning target videos can lead to over-fitting source prompts, potentially diminishing the model's editing capabilities. To address this, several studies employ T2V diffusion models for video editing and demonstrate promising results. [28] introduces a mixed fine-tuning strategy for the Imagen Video model [15] that enhances motion editing. [9] presents a video diffusion model trained with depth information to govern video structure and content. Although these methods have yielded remarkable results in video editing, training T2V diffusion models poses a challenge due to the inherent difficulty in collecting high-quality video data for training. Furthermore, their primary focus is on enhancing T2V editing through mixed video-image fine-tuning and joint video-image training, neglecting the potential benefits of leveraging existing high-quality T2I models.Another avenue of research is inspired by Prompt2Prompt [14] and Plug-and-Play [43] , both of which facilitate local editing through attention map manipulation. [32] suggests blending self-attention maps with masks produced by cross-attention maps to support z 𝐼 𝑡 -1 ← DDIM T2I (z 𝐼 𝑡 , 𝑃, 𝑡);3:z 𝑉 𝑡 -1 ← DDIM T2V (z 𝑉 𝑡 , 𝑃, 𝑡); 4:if 𝑡 ≤ (𝑇 -𝜏) then 5:z * 𝑡 -1 = 𝛼 𝑡 * z 𝑉 𝑡 -1 + (1 -𝛼 𝑡 ) * z 𝐼 𝑡 -1 ; 6:𝛼 𝑡 -1 = 𝛼 𝑡 + (1 -𝛼 𝜏 )/(𝑇 -𝜏);7:z 𝐼 𝑡 -1 ← z * 𝑡 -1 ; 8: z 𝑉 𝑡 -1 ← z * 𝑡 -1 ; 9:end if 10: end for 11: return z * 0 zero-shot video editing. Meanwhile, [24] introduces a decoupledguidance attention control, adapting P2P to Video-P2P. However, these methods require extensive manual adjustment of the parameters in the attention map, making them challenging to apply in practical scenarios.In this paper, we introduce a simple-yet-effective latent fusion strategy that harnesses the strengths of both T2I and T2V diffusion models for T2V editing without requiring any tuning. For the first time, this study demonstrates that T2I and T2V diffusion models complement each other. 
method: 
 Our objective is to maximize the advantages of the existing textto-image (T2I) and text-to-video (T2V) diffusion models in terms of structure integrity and temporal consistency, aiming to achieve high-quality T2V editing. Given a video V containing 𝑚 frames, denoted as V = {x 𝑖 | 𝑖 ∈ [1, 𝑚]}, T2V editing aims to generate a new video conditioned on an editing text prompt 𝑃. In this section, we first introduce the preliminaries of the diffusion model in Section 3.1, including latent diffusion models [37] and DDIM inversion [42] . Then the detail design of FLDM (Fused Latent Diffusion Model) will be introduced in Section 3.2 and Section 3.3.Latent Diffusion Models (LDMs). LDMs denoise noisy latents in the latent space. First, an image x undergoes compression via an encoder E, producing a latent representation z = E (x), which is reconstructed back into an image through a decoder D. A U-Net 𝑈 𝜃 is applied as the denoising model to predict the noise, which is optimized by minimizing the following objective minEQUATIONwhere z 𝑡 is the noisy latent at timestep 𝑡 and 𝑃 is the conditional text embedding. DDIM Inversion. Deterministic DDIM sampling inverts the noisy latent z 𝑇 to a clean latent z 0 , the sampling process is expressed as where ε is the predicted noise and 𝛼 𝑡 is the multiplication of variances at timestep 𝑡. DDIM inversion is the reverse process of DDIM sampling, it converts a clean latent z 0 to a noised latent ẑ𝑇 which can preserve the structure information in real images.z 𝑡 -1 = √ ᾱ𝑡 -1 z 𝑡 - √ 1 -ᾱ𝑡 ε √ ᾱ𝑡 + √ 1 -ᾱ𝑡 -1 ε (2)EQUATIONAs depicted in Figure 2 , video editing conducted separately by T2I and T2V LDMs, without the integration of FLDM, reveals distinct drawbacks. T2I LDMs, although adept in frame-wise editing, tend to cause flickering issues, e.g., the tree in the background has obvious shape changes across frames. Similarly, T2V LDMs, despite their ability to maintain temporal consistency, often compromise on structural quality, e.g., the car shape is different from the original car (see the first row in Figure 1 ). FLDM addresses these issues by strategically fusing the latents of both T2I and T2V LDMs during the denoising process. As the denoising timesteps progress, there is an improvement in both the structural integrity and temporal consistency of the video edits. Figure 3 shows the FLDM pipeline of video editing with multisource latent fusion. Given the source video V src and a target prompt 𝑃, the VAE Encoder compresses the input video into a clean latent z 0 ∈ R 𝑓 ×𝑐 ×ℎ×𝑤 , where 𝑓 represents the number of frames in the source video. Then we apply DDIM inversion with T2V and T2I LDMs, which makes use of the source prompt, to invert the clean latent into noisy latents z 𝑉 𝑇 ∈ R 𝑓 ×𝑐 ×ℎ×𝑤 and z 𝐼 𝑇 ∈ R 𝑓 ×𝑐 ×ℎ×𝑤 . Afterward, T2V and T2I LDMs are applied to predict noise. Note that the T2I LDMs predict noise frame-by-frame and the denoised T2I latents are concatenated at the temporal dimension. Then we apply latent fusion by combining the denoised latents from T2V and T2I LDMs. As a result, a mixed noisy latent z * 𝑡 is generated for the next denoising timestepEQUATION)where the hyper-parameter 𝛼 controls the ratio of T2V and T2I latents. A larger 𝛼 ensures better temporal consistency but does harm to structure and text alignment, while if 𝛼 is too small, it would degrade to frame-wise editing with T2I LDMs. Notice that, our method is a versatile plugin that can work for latent diffusion models within the same latent space. We need to ensure that two LDMs share the same VAE Encoder and Decoder. Most LDMs are trained on the basis of Stable Diffusion [37] , where the weights of VAE are frozen and remain unchanged. As we utilize off-the-shelf T2I LDMs and T2V LDMs for joint denoising, the editing results depend on the fused models. Specifically, the temporal modeling capability of T2V LDMs affects the temporal consistency of edited video frames, while the spatial structure of the videos is related to T2I LDMs.In practice, we notice that fusing multiple latents at the early diffusion steps results in noisy output, since latent fusion may break the structure of videos. As a result, we let T2V and T2I LDMs infer separately in early 𝜏 steps and perform latents fusion at afterward steps. Although 𝛼 balances the ratio of temporal consistency and structure modeling, a fixed 𝛼 may bring too much guidance from T2I LDMs at the end of the denoising process, thus causing temporal inconsistency in generated videos. Since T2V LDMs are able to generate temporal coherent videos, we increase the ratio of T2V latents at the end of the denoising process. We propose an update schedule for the fusion ratio, wherein the ratio is incrementally increased after each latent fusion step as following:EQUATION)where 𝑇 is the total number of denoising steps, 𝛼 𝑡 is the fusion ratio at timestep 𝑡. In practice, we start to increase 𝛼 𝑡 at timestep 𝜏. The initial fusion ratio 𝛼 𝜏 ranges between [0, 1]. Formally, the algorithm is shown in Algorithm 1. FLDM aims to augment the capability of T2V models through T2I models, ensuring that T2V models retain a prominent position. So T2V latents should be allocated a larger propotion. The T2V EQUATIONEQUATIONAs 𝛼 𝑡 changes in a linear manner, we can calculate the difference between the T2V and T2I contributions:EQUATIONwhich means T2V signals take a dominant role. On the other hand, a linear update schedule ensures a smoother denoising process than a polynomial way. 
experiments 4.1 experiment settings: 
 Implementation Details. We attempt to apply FLDM to various T2I and T2V diffusion models to validate the effectiveness. For the diffusion model gallery, we consider popular T2I diffusion models including ControlNet [50] and InstructPix2Pix [3] loaded with pretrained weights from Stable Diffusion v1.5 [37] . Although there are some high-quality video diffusion models [9, 15, 45] , unfortunately, they can not be accessed. Therefore, we take the publicly available VidRD [12] and ZeroScope [26] with released pre-trained weights. We then sample 8 frames uniformly from input videos with a resolution of 256p for all models. For T2V models fused with ControlNet, we use ControlNet with canny edge condition and set the total timestep 𝑇 = 50 for DDIM schedule. For T2V models fused with InstructPix2Pix, we set the total timestep 𝑇 = 100 and the classifier-free guidance scale of text to 12.5 and of image to 1.5. We select 16 videos from DAVIS dataset [31] for evaluation, which are part of the TGVE benchmark [47] . For each video, there are three types of text prompts for video editing including style, object, and background. Evaluation Metrics. Following previous text-to-video editing works [9, 32, 46] , we conduct both quantitative and qualitative evaluations for FLDM. For quantitative evaluation, we utilize automatic metrics including frame consistency ('Tem-Con'), text alignment ('Text-Align'), user preference ('User-Pre') with CLIPScore [33] and PickScore [19] . For user study, three metrics (denoted as 'Edit', 'Image', and 'Temp') are conducted to evaluate editing quality, frame fidelity, and temporal consistency of edited video. Following [23] , we recruit 20 evaluators to pair-wisely compare our method with baselines, and present the preference percentage with "𝑝 1 /𝑝 2 " where 𝑝 1 denotes the preference percentage of baseline, 𝑝 2 denotes the preference percentage of our method.Baselines. We test two T2I models ControlNet (CN) and Instruct-Pix2Pix (IP2P), and two T2V models VidRD and ZeroScope (ZS) under the individual testing setting (T2I or T2V) and FLDMs testing setting (T2I + T2V). In addition, we compare our method against two state-of-the-art video editing methods including Tune-A-Video [46] and FateZero [32] . Applications. We show three example applications including 1) Object editing: Our method has a significant use case in altering objects by manipulating text prompts. This enables effortless replacement, addition, or removal of objects. For example, we can replace "goldfish" with "koi fish" or "sharks" (see Figure 5 ). When using our method with InstructPix2Pix, the instruction can be "replace the goldfish with koi fish" or just "turn them sharks". 2) Background change: Another application of our method is to change the background of video. When altering the location of the object, our method maintains the coherence of the object's motion. For example, we can modify the background of the rabbit in Figure 1 to be "on the floor", and add shadow of the rabbit that doesn't exist in the origin video. Figure 5 shows the results of editing "tank" to be "snowy-covered tank". 3) Style transfer: Through taking the knowledge from T2I models in open domains, FLDM can facilitate transforming videos into diverse styles that are challenging to acquire solely from video data. For example, we transform real-world videos into cartoon style (Figure 1 ), or Van Gogh style (Figure 5 ), by adding style descriptors to the prompt. Qualitative Results. We showcase visual comparisons of our method against other baselines from Figure 4 to Figure 6 . In Figure 5 and Figure 6 , we present the effects of fusing T2I models with VidRD and ZeroScope separately. In the case of goldfish, In-structPix2Pix has superior capabilities of frame-wise editing, while lacking consistency between frames (e.g., the appearance of koi fishes exhibits significant variations). In contrast, videos generated by VidRD individually have decent temporal consistency but lack original structure (e.g., background details). When applying FLDM to VidRD and InstructPix2Pix, both temporal consistency and structural integrity can be improved. In Figure 6 , we observed that ZeroScope (ZS) achieves satisfactory video editing results with DDIM inversion. However, FLDM can further enhance the visual aesthetics and fidelity of the generated videos (e.g., the cartoon rabbit generated by FLDM is more similar to the original rabbit). In Figure 4 , we compare FLDMs with two state-of-art video editing methods. We find Tune-A-Video struggles to keep the fidelity of the original video (e.g., the car is different from the one in the original video) since it lacks regional control. Besides, videos generated by FateZero fail to align with target prompts well (e.g., Vermeer painting style) without carefully tuning on word strength hyperparameters. It is worth noting that FLDM is more efficient than SOTAs since it does not require any model tuning [46] or heavy handcrafted hyper-parameter tuning [32] .Quantitative Results. We use automatic metrics to quantify our method against baselines in Table 1 . Results indicate that compared to editing with T2V models only, FLDM achieves higher user preference and better textual alignment at the cost of a slight loss of temporal consistency. FateZero demonstrates good frame consistency but struggles to achieve accurate text-guided editing. Tune-A-video excels in both temporal consistency and textual alignment. In comparison, our method outperforms these two methods in textual alignment and achieves comparable temporal consistency. The user study results are shown in Table 2 , which indicates our method is preferred by users across all metrics. For all the ablation experiments, we take VidRD as the T2V model, which is fused with T2I models (ControlNet and InstructPix2Pix) through FLDM if not other mentioned. Ablation of Initial Fusion Ratio 𝛼. We conduct extensive experiments with various fusion ratios 𝛼 as shown in Figure 7 and have the following observations: 1) For temporal consistency, the larger 𝛼 makes T2V latents have larger weight in fused latents which advances temporal consistency for generated videos. 2) For text alignment, there is one optimal value between [0, 1] which indicates that T2V and T2I models are complementary for text alignment, probably because the T2V model is trained with large-scale video datasets that have rich action concepts in complementary with T2I text prompts [45] . Figure 8 shows some samples generated with two T2I architectures fused with the T2V model. As we can see, the frame-wise editing (only T2I) performs worst since it ignores the temporal relation among frames. While with a T2V model and a suitable fusion ratio 𝛼, the temporal consistency can be improved clearly. Since training a T2V diffusion model is expensive in collecting datasets and computational cost, we can make the most use of existing T2I models to enhance T2V editing with FLDM. Considering the trade-off between temporal consistency and text alignment, we fuse a smaller proportion of T2I latents compared to T2V latents in practice. Ablation of 𝜏. Table 3 shows the quantitative results of different 𝜏. Figure 9 illustrates the effectiveness of hyper-parameter 𝜏 which decides from which denoising timestep we start to perform FLDM. When FLDM starts at an early timestep (e.g., 𝜏 <= 30 for InstructPix2Pix), it may destroy the edited video structure since T2V latents are incorporated into the denoising process too early. If FLDM starts too late (e.g., 𝜏 >= 45 for ControlNet), the T2I latents would take too much proportion throughout the denoising process, as a result, destroy temporal consistency for the edited video. As shown in Figure 9 , the car in the bottom row exhibits artifacts. Ablation of 𝛼 Update Schedule As we can see from Figure 10 , the 𝛼 update schedule can improve temporal consistency, e.g., the appearance of the car (Left: InstructPix2Pix + VidRD) and the beach (Right: ControlNet + VidRD) can be better preserved among video frames with the update schedule. This illustrates that, with decayed T2I latent proportion at the last 𝑇 -𝜏 timesteps, the update schedule further reduces T2I latents' negative effect on temporal consistency. 
conclusion: 
 In this paper, we propose FLDM (Fused Latent Diffusion Model), a simple-yet-effective strategy that achieves high video editing quality without any tuning cost. For each denoising timestep, we apply a hyper-parameter to adjust the latent ratio of T2V and T2I diffusion models, with an update schedule to alleviate image artifacts. For the first time, we reveal that T2V and T2I diffusion models are complementary to each other in temporal consistency and structure. Our method can serve as a versatile plugin for various off-the-shelf T2I and T2V models, which we believe will be valuable for real-world practice. 
 
 Appendix:  


Decision: This work proposes a fused latent diffusion model that combines T2I and T2V pretrained models for video editing. After the rebuttal, mouse reviewers are positive about this work. It is worth mentioning that the proposed method is simple yet effective, with comprehensive experimental results. Thus the AC recommends the acceptance of this paper. It is highly recommended to include more qualitative visual results in the revised version of the paper.

Reviews: review: paper_summary: paper_summary: This paper proposes the Fused Latent Diffusion Model, a video editing technique that uses T2I and T2V models together for better temporal consistency and structure fidelity. The latent fusion schedule is carefully designed to balance the influence of each independent component. The results show that the proposed method helps overcome the limitations of the baselines.
strength_weakness: strength_weakness: 1. The supplementary project page provides videos that help understand the method;
2. The method is technically clear and easy to follow;
3. The paper is well-constructed and easy to understand.
limitations: limitations: 1. The qualitative result is limited. The supplementary material only provides less than ten edited videos, making it hard to tell the generalizability of the proposed method;
2. According to all samples, the editing capability seems limited. Most results (especially the "car driving on the road" example) share almost the same structure as the input videos and only perform texture-level modification. Is the proposed method able to do some sematic-level editing, e.g., changing the car into a Lamborghini?
3. In Tab. 1, why FateZero achieve the highest temporal consistency and outperform the proposed method?
4. The compared baselines are not comprehensive. Should additionally consider other representative works such as TokenFlow, Pix2Vid, and CoDeF.
paper_summary: paper_summary: This work achieves high-quality T2V editing by integrating various T2I and T2V LDMs. Specifically, FLDM utilizes a hyper-parameter with an update schedule to effectively fuse image and video latents during the denoising process.
strength_weakness: strength_weakness: 1. This paper is the first to reveal that T2I and T2V LDMs can complement each other in terms of structure and temporal consistency, ultimately generating high-quality videos.

2. FLDM can serve as a versatile plugin, applicable to off-the-shelf image and video LDMs, to significantly enhance the quality of video editing.
limitations: limitations: 1. It is a natural idea to introduce the video diffusion model as the basic model into video editing. However, this article seriously ignores the focus of this type of improvement, that is, 1) it does not provide a detailed description of what motivation it is based on, 2) why it is introduced in this way, and 3) design multiple introduction methods and explore their differences.

2. In addition, the comparison method was too early, and there was even no comparison of papers in NeurIPS2023.
paper_summary: paper_summary: This paper proposes to tackle the problem of video editing via integrating various T2I and T2V LDMs, named FLDM. Specifically, FLDM utilizes a hyper-parameter with an update schedule to effectively fuse image and video latents during the denoising process. Experimental results show effectiveness of proposed method.
strength_weakness: strength_weakness: 1. The writing is clear, logical, and easy to understand. 
2. The proposed method is simple yet efficient.
3. Experimental results are complete and comprehensive.
limitations: limitations: 1. The proposed method is somehow tricky. Since the chosen T2I model is ControlNet and InstructPix2Pix as written in L515, it's pretty much like the ControlVideo [5] that can already preserve the structure information. The chosen video model is a pre-trained pure text-to-video model without any constraints of structure information. If we just use a structure preserve T2V model such as [3,5], then the hypothesis of T2V's non-structure preserving property is no longer valid.
2. About experimental results, lacking of comparison with other SOTA video editing methods [1-3].

Additional Information:
1. Is there any reason or experiment to explain that T2V cannot preserve structure? 


[1] Geyer M, Bar-Tal O, Bagon S, et al. Tokenflow: Consistent diffusion features for consistent video editing[J]. arXiv preprint arXiv:2307.10373, 2023.
[2] Kara O, Kurtkaya B, Yesiltepe H, et al. RAVE: Randomized Noise Shuffling for Fast and Consistent Video Editing with Diffusion Models[J]. arXiv preprint arXiv:2312.04524, 2023.
[3] Feng R, Weng W, Wang Y, et al. Ccedit: Creative and controllable video editing via diffusion models[J]. arXiv preprint arXiv:2309.16496, 2023.
[4] Zhang Y, Wei Y, Jiang D, et al. Controlvideo: Training-free controllable text-to-video generation[J]. arXiv preprint arXiv:2305.13077, 2023.
[5] Liew J H, Yan H, Zhang J, et al. Magicedit: High-fidelity and temporally coherent video editing[J]. arXiv preprint arXiv:2308.14749, 2023.
paper_summary: paper_summary: This paper proposes a train-free framework on fusing the T2V and T2I LDMs to edit the video. They use weighted sum to fuse these two. This fusion method tries to make a balance between structural modeling and temporal consistency. An update schedule on the weight $\alpha_t$ are also proposed to firstly get a good structure and then obtain temporal consistency. The results can show the trade-off between T2I and T2V models and show the effectiveness of the proposed methods.
strength_weakness: strength_weakness: 1.It’s a good exploration about cooperation between T2I and T2V LDMs, revealing the strength and weakness of the two kinds.
2.To a certain extent, the proposed method can integrate the advantages of T2I and T2V LDMs, without any training period.
3.A reasonable update schedule is designed to get the structural modeling firstly and then obtain the temporal modeling, which makes sense.
limitations: limitations: 1.According to the results shown in Tab. 1, performance of the proposed method is not very satisfactory for me. Compared with VidRD & DDIM and ZS & DDIM, the proposed method has a big drop (about 2 points) on CLIP Tem-Con and a negligible improvement on CLIP Text-Align. 
2.Lack the comparison with some SOTA video editing method, like Video-P2P, ControlVideo.
3.As for the qualitative results shown in Fig.6 left, the structural modeling of ZS+IP2P is not good (the “dog” is still similar to a rabit).
4.Lack the quantitative ablation study on the update schedule, I think it important to show this for comprehensive validity provement.