AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks
Authors: Max Ku
Cong Wei
Weiming Ren
Huan Yang
Wenhu Chen
Venue: Accepted by TMLR ()

Abstract: In the dynamic field of digital content creation using generative models, state-of-the-art video editing models still do not offer the level of quality and control that users desire. Previous works on video editing either extended from image-based generative models in a zero-shot
manner or necessitated extensive fine-tuning, which can hinder the production of fluid video edits. Furthermore, these methods frequently rely on textual input as the editing guidance, leading to ambiguities and limiting the types of edits they can perform. Recognizing these
challenges, we introduce AnyV2V, a novel tuning-free paradigm designed to simplify video editing into two primary steps: (1) employing an off-the-shelf image editing model to modify the first frame, (2) utilizing an existing image-to-video generation model to generate the edited video through temporal feature injection. AnyV2V can leverage any existing image editing tools to support an extensive array of video editing tasks, including prompt-based editing, reference-based style transfer, subject-driven editing, and identity manipulation, which were unattainable by previous methods. AnyV2V can also support any video length. Our evaluation shows that AnyV2V achieved CLIP Scores comparable to other baseline methods. Furthermore, AnyV2V significantly outperformed these baselines in human evaluations, demonstrating notable improvements in visual consistency with the source video while producing high-quality edits across all editing tasks.

Published in Transactions on Machine Learning Research (11/2024)
AnyV2V: A Tuning-Free Framework For Any Video-to-Video Editing Tasks
Max Ku", Cong Wei", * Weiming Ren", Harry Yang, * Wenhu Chen
* University of Waterloo, Veccoor Institute, Harmony.Al {m3ku, c58wei, w2ren, wathcchen)@uaaterloo..aa
Reviewed OpenReview: http:://opaaaeeresse.ee///rrtttt
Abstract
In the dynamic field of digital content creation using generative models, state-of-the-art video editing models still do not offer the level of quality and control that users desire. Previous works on video editing either extended from image- based generative models a zero-shot manner or necessitated extensive fine-tuning, which can hinder the production of fluid vidco edits. Furthermore, these methods frequently rely on textual input as the editing guidance, leading to ambiguities and limiting the types of edits they can perform. Recognizing these challenges, we introduce Any V2V, a novel tuning-free free paradigm designed to simplify video editing into two primary steps: (1) employing an off-the-shelf image editing model to modify the first frame, (2) utilizing an existing image-to-vider generation model to generate the edited video through temporal feature injection. Any can leverage any existing image editing tools to support an extensive array of video editing tasks, including style transfer, subject-driven editing, and identity
prompt- based editing, reference-basee manipulation, which were unattainable by previous methods. V2V can also support any video length. Our evaluation shows that AnyV2V achieved -coresscores comparable to other baseline methods. Furthermore, Vny significantly outperformed these baselines in human evaluations, demonstrating notable improvements in visual consistency with the source video while producing high-quality edits across all editing tasks. The code available at
AoG..////TTTIRR-ALLLaaaaaaaaaaa Vny V2V
1 Introduction
The development. of deep generative models (Ho et al. 2020) has led to significant advancements content.
creation and manipulation, especially in digital images (Rombach al. 2022 Nichol 2022 Brooks
ct al.
2023 Ku et al.] 2024 Chen et al.) 2023c Li ct al. 2023). However, video generation and editing
have not reached the same level of advancement as images (aag et al. 2023 Chen et al. 2023a 2024 Ho
et al. 2022). In the context of video editing, training a large-scale video editing model presents consideaabe challenges due to the scarcity of paired data and the substantial computational resources required.
To overcome these challenges, researchers proposed various approaches Geyer et al.
2023
Cong et al. 2023
Wu et al. 2023a b; Liu et al. 2023a Liang et al. 2023 Gu et al. 2023b: Cong et al.; 2023 Zhang al.
2023d Qi et al. 2023 Ceylan al., 2023 Yang et al. 2023 Jeong & Ye 2023 Guo al., 2023 , which
can be categorized into two types: (1) zero-shot adaptation from pre-trained text-to-image (T2I) models
(2) fine-tuned motion module from pre-trained T21 or text-to-video (T2V) models. The zero-shot
methods (Geyer et al.) 2023 Cong et al. 2023 Joong & Yo. 2023 Zhang et al. 2023d) usually suffer from flickering issues due to a lack of temporal understanding. On the other hand, fine-tuning methods (Wu et al., 2023a Chen et al.) 2023b Wu et al.) and computational overhead of edits. For example, a user might want to perform edits on visuals that are out-of-distribuus learned text encoder (e.g. change person to character from their artwork). Thus, highly customizable
2023b Gu et al. 2023b Guo et al. 2023) require more time edit vidcos. Morcover, all the methods can only adhere to certain types from the

---

Published in Transactions on Machine Learning Research (11/2024)
solution is more desired in video editing applications. It would be ideal if there were methods that allowed for seamless combination of human artistic input and the assistance provided by AT, allowing for synergy between huraan effort and AI, maintaining creators' creativity while producing high-quality outputs.
Editing Prompt
Source Video
*turn his hair whitt
InstructPix2Pix
Face Image
InstantID
Subject Image
AnyV2V
AnyDoor
Neural Style Transfer
Style Image
Any Black- Box Image Editing Method
1st frame only
Figure 1: AnyV2V is an evolving framework to handle all types of rideo-to-vides editing tasks without any parameter tuning. Vny disentangles video editing into two simpler problems: (1) Single image editing and (2) Image-to-video generation with video reference.
With the above motivations, we aim to develop video editing framework that requires no fine-tuning and caters to user demand. In this work, we present Any designed to enhance the controllability of zero-shot vidco editing by decomposing the task into two pivotal stages:
Apply image edit on the first frame with any off off-the-shelf image editing model or human effort.
2. Leverage the innate knowledge of the image-to-video (I2V) model to generate the edited video with the edited first frame, source video latent, and the intermediate lemporal features.
Our objective is to propagate the edited first frame across the entire video while ensuring alignment with the source vidco. To achieve this, we employ I2V models (Zhang et al. 2023c Chen et al. 2023d Ren et al.) 2024) for DDTM inversion to enable first-frame conditioning. With the inverted lalents as initial noise and the modified first frame the conditional signal, the I2V model can generate videos that are not only faithful the edited first frame but also follow the appearance and motion of the source video. To further enforce the consistency of the appearance and motion with the source video, perform feature injection in the I2V model. The two-stage editing process effectively offloads the editing operation to existing image editing tools. This design (detail in Section 4 helps Vny excel in:
Compatibility: provides a highly customized interface for a user to perform video edits on any modality. It can scamlessly integrate any image editing methods to perform diverse editing.
Simplicity: It does not require any fine-tuning nor additional video features like previous works (Gu et al. 2023b Wu al. 2023b Ouyang al. 2023) to achieve high appearance and temporal consistency for video editing tasks.
From our findings, without any fine-tuning, V2V can perform video editing tasks beyond the scope of current publicly available methods, such as reference- based style transfer, subject-driven editing, and identity manipulation. Vny also can perform prompt-based editing and achieve superior results on common vidco editing evaluation metries compared the baseline models (Geyer al. [2223 Cong et al. 2023 Wu et al. 2023b). We show both quantitatively and qualitatively that our method outperforms existing SOTA
2

---

Published in Transactions on Machine Learning Research (11/2024)
basclines in Section 5.3 and Appendix B Vny is favoured in 69.7% of samples for prompt alignment and 46.2% overall preference in human evaluation, while the best baseline only achieves 31.7% and 20.7%, respectively. Our method also reaches the highest CLIP-Text score of 0.2932 in text alignment and a competitive CLIP- CLIp-Image score of 0.9652 in temporal consistency.
All these achievements are thanks to the AnyV2V's design to harness the power of off-the-shelf image editing models from advanced image editing research. Through a comprehensive study and evaluation of the effectiveness of our design, our key observation is that the inverted noise latent and feature injection serve critical components to guide the video motion, and the 12V model itself has good capabilities in generating motions. We also found that by inverting long videos exceeding the I2V models' training frames, the inverted latents enable the 12V model to produce longer videos, making long video editing possible. To summarize, The main contributions of our work are three-fold:
We proposed Any as first fundamentally different solution for video editing, treating video editing a simpler image editing problem.
We showed that Vny can support long video editing by inverting videos that extend beyond the training frame lengths of 12V models.
Our extensive experiments showcased the superior performance of Any when compared to the existing SOTA methods, highlighting the potential of leveraging 12V models for video editing.
2
Related Works
Video generation has attracted considerable attention within the field (Chen et al. 2023a 2024
OpenA
Wang
et al.
2023 Hong et al. 2022 Zhang et al.] 2023a Henschel et al.] 2024 Wang et al., 2024c X Xing et al.
2023
Chen et al. 2023d Bar- Tal et al. 2024
Ren et al. 2024 Zhang et al. 2023c However, video manipulation
also represents a significant and popular area of interest. Initial attempts, such as Tunc-A- Video (Wu et al.,
2023b) and VideoP2P (Liu al. 2023b involved fine-tuning a text-to-image model achieve video editing
by learning the continuous motion. The concurrent works at that time such as Pix2Vidco (Ceylan et al. 2023 and Fate-Zero (Qi et al.) 2023 go for zero-shot approach, which leverages the inverted latent from text-to-image model to retain both structural and motion information. The progressively propagates to other frames edits. Subsequent developments have enhanced the results but generally follow the two paradigıns
/eyer et al. 2023 Wu et al.
2023a Cong et al.] 2023 Yang ct al. 2023
Ceylan et al. 2023 Ouyang et al. al. 2023b Esser et al. 2023 Chen el. al. 2023b Jeong & Ye 2023 Zhang
2023 Guo et al. 2023: Gu el
and ControlVideo (Zhang al, 2023d leveraged ControlNet (Zhang al. 2023b) for extra spatial guidance. TokenFlow (Geyer et al. 2023
et al. 2023d; Cheng et al.] 2023). Control-A-Vinn Viieeo (Chen et al.
2023b)
leveraged the nearest neighbor field and inverted latent to achieve temporally consistent edit. Fairy Wu et al. 2023a) followed both paradigms which they fine-tuned a text-to-image model and also cached the attention maps to propagate the frame edits. VideoSwap (Gu et al. 2023b) requires additional parameter tuning and video feature extraction (c.g. tracking, point correspondence, etc) to ensure appearance and temporal consistency. CoDeF (Ouyang et al. 2023) allows the first image edit to propagate the other frames with onc-shot tuning. UniEdit (Bai et al. 2024) leverages the inverted latent and fcature maps injection to achieve a wide range of video editing with a pre-trained text-to-video model (Wang et al. 2023).
However, none of the methods exact intentions or desired level of detail, often due to the ambiguity of natural language and the constraints of the model's capabilities. For example, VideoP2P (Liu et al. 2023a) is restricted to only word-swapping prompts due to the reliance on cross-attention. There is a clear need for a more precise and comprehensive solution for video editing tasks. Our work V2V the first work to empower a diverse array of video oditing tasks. We compare Vny with the existing methods in Table As can be seen, our method excels in its applicability and compatibility.
offer precise control to users, the edits may not align with the user's
3

---

Published in Transactions on Machine Learning Research (11/2024)
Table Comparison with different video editing methods and the type of editing tasks.
Prompt-based Reference-based Subject-Driven
Identity Manipulation
Method
Tuning- Free?
Backbone
Editing
Style Transfer
Editing
Tume-A- Saane-A-iidaan Video
Wu
Stable Diffusion
Pix2Viden Ceylan
2023
x
SD>-Depth
Gen-1
Resor ot al 2023
X
x
X
Stable Ditfusion
Taken low
Geyer ci
2025
X
X
X
Stable Diflusion
PLATTEN Cong et mi
2023
X
X
X
Stable Diffusion
Pairy (Wi
20254
X
Stable Ditfusion
ControlV
X
X
ControlNet
CoDeF Ouyang
X
X
ControlNet
VidenSwap (Gl of
2023h
X
X
X
AnimateDiff
UniEdit Bai ot al.
X
X
Any T2V Models
Any (Ours)
/
/
Any 12V Models
3 Preliminary
3.1 Image-to- Voddeeo--iide (12V) Generation Models
In this work, we focus on leveraging latent diffusion-based [Rombach et al. 2022 for video editing. Given an input first frame I1, a text prompt s and a noisy video latent 2, at time step t, 12V generation models recover less noisy latent Z₁-1 using a denoising model c0 4, s, t) conditioned on both I and The denoising model Eg contains a set of spatial and temporal self-attention layers, where the self-attention operation can be formulated as:
12V generation models
Q = @=we,,Ke K KW V WV Allention(Q,, K, V) = Softmax
(1)
QK
vd VV
(2)
where is the input hidden state to the self-attention layer and WQ. W and WV are learnable projection matrices that map onto query, key and value vectors, respectively. For spatial self-attention, represents a sequence of spatial tokens from each frame. For temporal self-attention, is composed of tokens located at the same spatial position across all frames.
3.2
DDIM Inversion
The denoising process for 12V generation models from Z, to Zt-1 can be achieved using the DDIM (Sng et al, sampling algorithm. The reverse process of DDIM sampling, known as DDIM inversion (Mokady et al.
2020)
2023;
Dhariwal Nichol 2021), allows obtaining 2 from z, such that Zt+1
Drt Z4
ocs1
1) co co(2,, zo, f), where is derived from the variance schedule of the diffusion process.
V
3.3
Plug-and-Play (PnP) Diffusion Features
Tumanyan et al. (2023a) proposed PnP diffusion features for image editing, based the observation that
intermediate convolutionn features f and self-attention scores A = Softmax QKT Vd in text-to-image (T2I)
denoising U-Net capture the semantic regions (e.g. legs torso of human body) during the image generation process.
Given an input source image S and target prompt P, PnP first performs DDIM inversion to obtain the image's corresponding noise {z, J|= at each time step t. It then collects the convolution features { and attention scores {A{} from some predefined layers at each time step of the backward diffusion process zi_ = cg(2, 2,t,, where Ø denotes the null text prompt during denoising.
To generate the edited image I", PnP starts from the initial noise of the source image (i.e. z x and performs feature injection during denoising: 2,_1 c0 P, 1, (st 411), where co('s's, UR. A1)) represents
4

---

Published in Transactions on Machine Learning Research (11/2024)
I2V Model
DDIM Inversion
Source Video ys
Original denoised
1st Frame 1
ZTT
CNN feature
ZT-1
Spatial-attention
Black-Box Image
Editing Model
Temp-attention
Condition
Prompt
Face Image
Subject Image
Style Image
Edited 1 Frame
Edited Video denoised
ZT
-
Any V2V takes a source video VS as input. In the first stage, we apply block-box image editing method on the first frame Jı according to the editing task. In the second stage, the source video inverted to initial noise zf, which is then denoised using DDIM sampling. During the sampling process, we extract spatial convolution, spatial attention, and temporal attention features from the I2V models' decoder layers. To generate the edited video, we perform DDIM sampling by fixing 2 2f and the edited first frame the conditional signal. During sampling, we inject the features and attention into corresponding layers of the model.
Figure 2:
the operation of replacing the intermediate feature and attention scores (s1", A/7} with (f1, This feature injection mechanism ensures I" to preserve the layout and structure from 1 while reflecting the description in P. To control the feature injection strength, PnP also employs two thresholds Ty and тд such that the feature and attention scores only injected in the first Ty and тΛ denoising steps. Our method extends this feature injection mechanism to 12V generation models, where we inject features in convolution, spatial, and temporal attention layers. We show the detailed design of V2V in Section 4
4 AnyV2V
Our method presents a two-stage approach to video editing, Given a source video V = {41, 12, Iz...., In). where is the frame at time and denotes the video length, we first extract the initial frame 1i and pass it into an image editing model ding to obtain edited first frame I; 0ang(1,, C). C' is the auxiliary conditions for image editing models such as text. prompt,, mask, style, etc. In the second stage, we feed the edited first frame I and a target prompt s* into an 12V generation model Eq and employ the inverted latent from the source vidco ys to guide the generation process such that the edited vidco V follows the motion of the source video Vs, the semantic information represented in the edited first frame If, and the target prompt An overall illustration of our video editing pipeline is shown in Figure In this section, we explain each core component of our method.
4.1 Flexible First Frame Editing
In visual manipulation, controllabilily is a key element in performing precise editing. Any enables more controllable video editing by utilizing image editing models to modify the video's first frame. This strategic approach enables highly accurate modifications the vidco and compatible with a broad spectrum of image oditing models, including other deep learning models that can perform image style transfer (Gatys al.
2015 Ghiasi al. 2017
Lötzsch et al. 2022 Wang al. 2024a), mask-based image editing (Nichol et al.
5

---

Published in Transactions on Machine Learning Research (11/2024)
2022
Avrahami et al. 2022); image inpainting (Suvorov et al. 2021 Ku et al.
2022)
editing (Wang et al. 2024b), and subject-driven image editing Chen et al. et al. 2023a).
image
2023c
Li et al.
2023
Gu
4.2
Structural Guidance using DDIM Inverison
To ensure the generated videos from the I2V generation model follow the general structure as presented in the source video, we employ DDIM inversion to obtain the latent noise of the source video at each time step 1. Specifically, we perform the inversion without text prompt condition but with the first. frame condition. Formally, given a source video ys = {4, 12, 13. ...,Ah we obtain the inverted latent noise for time step t as:
2 DDIM Inv(ea(Ze41 4,,,,, 2,)),
(3)
where DDIM Inv(-) denotes the DDIM inversion operation as described in Appendix In ideal cases, the latent noise zij. at the final time step T (initial noise of the source video) should be used the initial noise for sampling the edited videos. In practice, we find that due to the limited capability of certain T2V models, the edited videos denoised from the last time step are sometimes distorted. Following Li et al. (2023). we observe that starting the sampling from a previous time step T < T can be used as a simple workaround to fix this issue.
4.3 Appearance Guidance via Spatial Feature Injection
Our empirical obsorvation (Section 5.4 suggests that 12V generation models already have some editing capabilities by only using the edited first frame and DDIM inverted noise as the model input. However, we find that this simple approach is often unable to correctly preserve the background in the edited first frame and the motion in the source video, as the conditional signal from the source video encoded in the inverted noise is limited.
To enforce consistency with the source video, perform feature injection in both convolution layers and spatial attention layers in the denoising U-Net. During the video sampling process, we simultaneously denoise the source video using the previously collected DDIM inverted latents 2 at cach time step such that = = <z(2; J1, (). We preserve two types of features during source video denoising: convolution features f before skip connection from the ( residual block in the U-Net decoder, and the spatial self-attention scores (A/²) from l2 = (lown 1ow+ Ingg layers. We collect the queries (Q'+) and keys (K!²) instcad of directly collecting A!a as the attention score matrices are parameterized by the query and key vectors. We then replace the corresponding features during denoising the edited video in both the normal denoising branch and the negative prompt branch for bassifier-free guidance (Ho & Salimans, 2022). We use two thresholds Tcone and Tea to control the convolution and spatial attention injection to only happen in the first Toows and Tea steps during video sampling.
4.4 Motion Guidance through Temporal Feature Injection
The spatial feature injection mechanism described in Section 4.3 significantly enhances the background and overall structure consistency of the edited video. While it also helps maintain the source video motion to some degree, we observe that the edited videos will still have a high chance of containing incorrect motion compared to the source video. On the other hand, we notice that 12V generation models, or video diffusson models in general, are often initialized from pre-trained T2I models and continue to be trained on vidco data. During the training process, parameters in the spatial layers are often frozen set to lower learning rate such that the pre-trained weights from the T21 model are less affected, and the parameters in the temporal layers are more extensively updated during training. Therefore, it is likely that large portion of the motion information is encoded in the temporal layers of the 12V generation models. Concurrent work (Bai et al.) 2024) also observes that features in the temporal layers show similar characteristics with optical flow (Horn & Schunck 1981), a pattern that is often used to describe the motion the video.
To better reconstruct the source video motion in the edited video, we propose also inject the temporal attention features in the video generation process. Similar to spatial attention injection, collect the source
6

---

Published in Transactions on Machine Learning Research (11/2024)
Prompt-based Editing
old with party hat doing tai chi
A blue airplaine take off
couple public display affectionn snowing
Souree Video
Tine- A. Video
TokenFiow
FLATTEN
Any
Figure 3: Any is robust in a wide range of prompt-based editing tasks while preserving the background. The results align the most with the text prompt and maintain high motion consistency.
video temporal self-attention queries 0 and keys K/s from some U-Net decoder layers represented by la and inject them into the edited video denoising branches. We also only apply temporal attention injection in the first Tea steps during sampling.
4.5 Putting it Together
Overall, combining the spatial and temporal feature injection mechanisms, replace the editing branch features (frh, Q Kala Q Kr's} with the features from the source denoising branch:
_ = = 1", s Uh Q2 K'' Q K!))),
(4)
where co(: : 1 Q/; Ky. Q ,KK)) denotes the feature replacement operation across different layers 4, 42, I3- Our proposed spatial and temporal feature injection scheme enables tuning-free adaptation of I2V generation models for video editing. Our experimental results demonstrate that each component in our design is crucial the accurate editing of source videos. We showcase more qualitative results for the effectiveness of our
model components in Section 5
5
Experiments
5.1
Implementation Details
We employ Vny on three off-the-shelf I2V generation models: I2VGen-XII (Zhang al. 2023c Consist12V (Ren al. 2024) and SEINE (Cenn al., 2023d). For all 12V models, we use Toons 0.2T
Ts = 0.27 and The 0.5T, where T is the total number of sampling steps. We use the DDIM (Song al.
We the version provided http:://mnniinfffaccccalliiiiil

---

Published in Transactions on Machine Learning Research (11/2024)
Reference-based Style Transfer
Neural Style Transfer
Reference style image
Reference style image
Source Video
Any
Subject-driven Editing
AnyDoor
Reference subject image
Reference subject image
Sturce
Any
Identity Manipulation
InstantID
Reference face image
Reference face image
Any\2V
Figure 4: With different image editing models, Any V2V can achieve a wide range of editing tasks, including reference- based style transfer, subject-driven editing, and identity manipulation.
2020 sampler and set T to the default values of the selected 12V models. Following PnP Tumanyan et al. 2023b), we set 1 4 for convolution fcature injection and l2 = 133 = (4, 5, 6.., 11} for spatial and temporal attention injections. During sampling, we apply text classifier-fres guidance (CFG) (Ho & Salimans 2022 for models with the same negative prompt "Distorted, discontinuous, Ugly, blurry, low resolution, motionless, static, disfigured, disconnected limbs, Ugly faces, incomplete arms" " across all edits. To obtain the initial edited frames in our implementation, we use of image editing model candidates including prompt-based image editing model Pistris (Brooks et. al. 2023). style transfer model Neural Style Transfer (NST) (Galys

---

Published in Transactions on Machine Learning Research (11/2024)
Table Quantitative comparisons for our Any with baselines on prompt-based video editing. Alignment: prompt alignment; Overall: overall preference. Bold: best results; Underline: top-2.
Human Evaluation
CLIP Scores
Method
Aligmmet
Overall
"LIP-Text
CLIP CLIP-Image
Tune-A- A-Vide
15.2%
2.1%
0.2902
0.9704
Flow
31.7%
20.7%
0.2858
0.9783
FLATTEN
25.5%
16.6%
0.2742
0.9739
AnyV2V (SEINE)
28.9%
8.3%
0.2910
0.9631
V2y (ConsistI2V)
33.8%
11.7%
0.2896
0.9556
V2V (I2VGen-XL)
69.7%
46.2%
0.2932
0.9652
et al. 2015), subject-driven image editing model AnyDoor (Chen et al. 2023ch, and identity-driven image editing model InstantID (Wang et al. 2024b). We experiment with only the successfully edited frames, which is crucial for our method. We conducted
the experiments on a single Nvidia A6000 GPU. To edit a 16-frame video, it requires around 15G GPU memory and around 100 seconds for the whole inference process. We refer readers to Appendix A for more discussions implementation details and hyperparameter settings.
5.2 Tasks Definition
1. Prompt-based Editing: allows users to manipulate vidco content using only natural language. This can include descriplive prompts or instructions. With the prompl, Users can perform wide range of edits, such as incorporating accessories, spawning or swapping objects, adding effects, or altering the background.
Reference-based Style Transfer: In the realm of style transfer tasks, the artistic styles of Monet and Van Gogh are frequently explored, but in real-life examples, users might want to distinct style based on one particular artwork. In reference-based style transfer, we focus on using style image as reference to perform video editing. The edited video should capture the distinct style of the referenced artwork.
Subject-driven Editing: In subject-driven video editing, we aim at replacing an object in the video with a target subject based on given subject image while maintaining the video motion and persevering the background.
4. Identity Manipulation: Identity manipulation allows the user to manipulate video content by replacing a person with another person's 's identity in the video based on an input image of the target person.
5.3
Evaluation Results
As shown in Figure 3 and Any can perform the following vidco editing tasks: (1) prompt-based oditing, (2) reference-based style transfer, (3) subject-driven editing, and (4) identity manipulation. we compare Vny against three baseline models Mine-A-Vides (Wu et al. 2023b), TokenFlow (Geyer al. 2023 and FLATTEN (Cong et al. 2023) for (1) prompt-based based editing. Since there exists no publicly available baseline method for task (2) (3) (4), we evaluate the performance of three I2V generation models under V2y We
included more comprehensive evaluation in Appendix
Prompt-based Editing Unlike the baseline methods (Wu et al. 2023b Geyer et al. 2023 Cong et al., 2023) that often introduce unwarranted changes not specified in the text commands, Any V2V utilizes the procision of image oditing models to ensure only the targeted arcas of the scene are alterod, leaving the rest unchanged. Combining Vny with the cnorucctionguuiden image editing model InstructPix2Pix (Brooks)
et al. 2023). Any accurately places party hat elderly man's head and correctly paints the plane
9

---

Published in Transactions on Machine Learning Research (11/2024)
Long Video Editing
robot walking
Source Video
Frame
Frame 9
Frame 17
Frame 25
Frame 97
Frame 105
Frame 113
Frame 121
Figure 5: Any can edit video length beyond the training frame while maintaining motion consistency. The first row is the souree video frames while the second rows are the edited. The editing prompt of the image was "turn woman into a robot" using image model InstructPix2Pix (Brooks et al. 2023
in blue. Additionally, it maintains the original video's background and fidelity, whereas, in comparison, baseline methods often alter the color tone and shape of objects, as illustrated in Figure 3 Also, for motion tasks such as adding snowing weather, I2V models from Vny provide inherent support for animating the snowing while the bascline methods would result in flickering. For quantitative evaluations, we conduct a human evaluation to examine the degree of prompt alignment. and overall preference of the edited videos based on user voting, and also compute the metrics CLIP-Text for text alignment and CLiP-Image for temporal consistency. In detail, CLIP-Text is computed by the average cosine similarity between text embeddings from CLIP model (Radford et al. 2021) and CLIP-Image is computed in the same way but between image embeddings for every pair of consecutive frames. Table 2 shows that Any generally achieves high text-alignment and temporal consistency, while V2y with I2VGen-XL backbone is the most preferred method because it does not edit the video precisely.
Style Transfer, Subject-Driven Editing and Identity Manipulation For these novel tasks, we stress the alignment with the reference image instead of the text prompt. As shown in Figure 4 can observe that for task (2), V2V can capture one particular style that is tailor-made, and even if the style is not learned by the text encoder. In the examples, V2y captures the style of Vassily Kandinsky's artwork "Compasition VII" and Vincent Van Gogh's 's artwork "Chateau in Auvers at Sunset" accurately. For task (3), Any V2V can replace the subject in the video with other subjects even if the new subject differs from the original subject. In the examples, a cat is replaced with a dog according the reference image and maintains highly aligned motion and background as reflected in the source video. A car is replaced by our desired car while the wheel is still spimning in the edited video. For task (4), Any can swap a person's 's identity to anyone. We report both human evaluation results and find that V2y with 12VGen-XL backbone is the most preferred method in terms of reference alignment and overall performance, plotted in Table 5 which is in Appendix B
I2V Backbones We find that AnyV2V (IVVGen-XL) tends to be the most robust both qualitatively and quantitatively. has a good generalization ability to produce consistent motions in the video with high visual quality. Vny (ConsistI2V) can generate consistent motion, but sometimes the watermark would appear due to its training data, thus harming the visual quality. AnyV2V (SEINE)'s generalization ability is relatively weaker but still produces consistent and high-quality video if the motion is simple enough, such person walking.
10

---

Published in Transactions on Machine Learning Research (11/2024)
Ablation Study
Couple sitting on a couch while having fün playing with video game console, in the style of van gogh
Iron Man duncing ballet
Source Vidlo
Any
TLLAAA 1S-TLANZAAAy TG-TS: TO-TS-LLAAA TLAAAA
Figure 6: Visual comparisons of Any editing results after disabling temporal feature injection (T.I., spatial feature injection (S.L. and DDIM inverted initial noise (D.I.).
Editing Video beyond Training Frames of I2V model Current state-of-the-art T2V models (Chen et al.
2023d Ren et al. 2024 Zhang et al. 2023c are mostly trained on video data that contains only 16 frames. To edit videos that have length beyond the training frames of the I2V model, an intuitive approach would be generating videos in an ruto-regressive manner as used in Consist12V (Ren al. (024) and SEINE (Chen 2023d However, we find that such an experiment setup cannot maintain semantic consistency in our case. As many works in extending video generation exploit the initial latent to generate longer video QQiu et al.] 2023 Wu et al.) 2023cp, we leverage the longer inverted latent as the initial latent and force an 12V model to generate longer frames of output. Our experiments found that the inverted latent contains enough temporal and semantic information to allow the generated video to maintain temporal and semantic consistency, as shown Figure
ct al.
5.4 Ablation Study
To verify the effectiveness of our design choices, we conduct an ablation study by iteratively disabling the three core components in our model: temporal feature injection, spatial feature injection, and DDIM invertod latent as initial noise. We use V2V (I2VGen-XL) and a subset of 20 samples in this ablation study and report both the frame-wise consistency results using CLIP- score in Table R and qualitative comparisons in Figure G We provide more ablation analysis of other design considerations of our model in the Appendix.
Effectiveness of Temporal Feature Injection According to the results, after disabling temporal feature injection in V2V (12VGen-XL) XL), while we observe a slight increase in the CLIP-Image score value, the odited vidcos often demonstrate less adherence the motion presented in the souree vidco. For example, in the second frame of the "couple sitting" case (3rd row, 2nd column in the right panel in Figure 6), the motion of the woman raising her leg in the source video is not reflected in the edited video without applying
11

---

Published in Transactions on Machine Learning Research (11/2024)
Table 3: Ablation study results for Any (12VGen-XT.) Injection and S. Injection correspond temporal and spatial feature injection mechanisms, respectively.
Model
CLIP-Image
V2V (I2VGen-XL)
0.9648
Vny (l2VGen-XL) XL) w/o T. Injection
0.9652
V2V (I2VGen-XL) w/o T. Injection & S. Injection
0.9637
AnyV2V (I2VGen-XL) w/o T. Injection & S. Injection & DDIM Inversion
0.9607
temporal injection. On the other hand, even when the style of the video is completely changed, Any (12VGen-XL) with temporal injection still able to capture this nuance motion in the edited video.
Effectiveness of Spatial Feature Injection As shown in Table 3 we observe a drop in the CLIP-Image score after removing the spatial feature injection mechanisms from our model, indicating that the edited videos are not smoothly progressed across consecutive frames and contain more appearance and motion inconsistencies. Further illustrated the third row of Figure ( removing spatial feature injection will often result in incorrect subject appearance and pose (as shown in the "ballet dancing" case) and degenerated background appearance (evident in the "couple sitting" case). These observations demonstrale that directly generating edited videos from the DDIM inverted noise is often not enough fully preserve the source video structures, and the spatial feature injection mechanisms are crucial for achieving better editing results.
DDIM Inverted Noise Structural Guidance Finally, we observe further decrease in CLIP-Image scores and a significantly degraded visual appearance in both examples in Figure 6 after replacing the initial DDIM inverted noise with random noise during sampling. This indicates that the I2V generation models become less capable of animating the input image when the editing prompt is completely out-of-dommin and highlights the importance of the DDIM inverted noise as the structural guidance of the edited videos.
6 Conclusion
In this paper, we presented V2V, a novel unified framework for video editing. Our framework training- free, highly cost-effective, and can be applied to any image editing model and I2V generation model. To perform video editing with high precision, we propose two-stage approach to first edit the initial frame of the source video and then condition an 12V model with the edited first frame and the source video features and inverted latents to produce the edited video at any length. Comprehensive experiments have shown that our method achieves outstanding outcomes across broad spectrum of applications that are beyond the scope of existing SOTA methods while achieving superior results on both common video metrics and human evaluation. For future work, we aim to find a tuning-free free method to bridge the I2V properties into T2V models so that we can leverage existing strong T2V models for video editing.
12

---

Published in Transactions on Machine Learning Research (11/2024)
References
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogrition, pp- 2022.
Jianhong Bai, Tianyu He, Yuchi Wang, Junliang Guo, Haoji Hu, Zuozhu Liu, and Jiang Bian. Unicdit: A unified Luming-free framework for video motion and appearance editing. arXiv preprint or aixii:2422.. 13185, 2024.
Omer Bar- Tal, Hila Chefer, Omer Tov, Charles Herrmann, Roni Paiss, Shiran Zada, Ariel Ephrat, Junhwa Hur, Yuanzhen Li, Tomer Michaeli, et al. Lumiere: A space-time diffusion model for video generation. arXiv preprint arXiv:2401.12993 2024.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
Duygu Ceylan, Chun-Hao Paul Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. 2023.
Haoxin Chen, Menghan Xia, Yingqing He, Yong Zhang, Xiaodong Cun, Shaoshu Yang, Jinbo Xing, Yaofang Liu, Qifeng Chen, Xintao Wang, et al. Videocrafterl: Open diffusion models for high-quality video generation. arXiv preprinl Xiv:22330. 19512, 2023a.
Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.090, 2024.
Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models, 2023b.
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. Anydoor: Zero-shot object-level image customization. arXiv preprint arXix:2307.04481 2023c.
Xinyuan Chen, Yaohui Wang, Lingjun Zhang, Shaobin Zhuang, Xin Ma, Jiashuo Yu, Yali Wang, Dahua Lin, Yu Qiao, and Ziwei Liu. Seine: Short-to-long video diffusion model for generative transition and prediction. arXiv preprint Xrxiv:2810. 207000 2023d.
Jiaxin Cheng, Tianjun Xiao, and Tong He. Consistent vidco-to-vidoo transfer using synthetic dataset. arXiv preprinl Xiv:23311 00t:231..0022 002013, 2023.
Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Juan-Manue Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flatten: optical flow-guided attention for consistent text-to-video editing, arXiv preprint Xrx::233000005 0..0222, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Germanidis. Structure and content-guided video synthesis with diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7346 7356, 2023.
Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge. A neural algorithm of artistic style. CoRR, abs/1508.06511 06576, 2015. URL http:///axxi..org/abbba
Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arzic:2307. 2023.
Golnaz Ghiasi, Honglak Leo, Manjunath Kudlur, Vincent Dumoulin, and Jonathon Shlens. Exploring the structure of real-time, arbitrary neural artistic stylization network. CoRR, abs/1705.0668 2017. URL http://arxiv. wtwp://.wwwwaaaa/11....o.
13

---

Published in Transactions on Machine Learning Research (11/2024)
Jing Gu, Yilin Wang, Nanxuan Zhao, Tsu-Jui Fu, Wei Xiong, Qing Liu, Zhifei Zhang, He Zhang, Jianming Zhang, HyunJoon Jung, and Xin Eric Wang. Photoswap: Personalized subject swapping in images, 2023a.
Yuchao Gu, Yipin Zhou, Bichen Wu, Licheng Yu, Jia- Liu, Rui Zhao, Jay Zhangjie Wu, David Junhao Zhang, Mike Zheng Shou, and Kevin Tang. Vidcoswap: Customized video subject swapping with interactive semantic point correspondence. orXiv preprinl, 2023b.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023.
Roberto Henschel, Levon Khachatryan, Daniil Hayrapetyan, Hayk Poghosyan, Vahram Tadevosyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Streamingt2v: Consistent, dynamic, and extendable long video generation from text. arXiv preprint arXiv:2408.14773 2024.
Jonathan Ho and Tim Salimans. Classifier free diffusion guidance. arXiv preprint arXiv:2207.1259. 2022.
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. Advonces in newral information processing systems, 33:6840-688 6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXiv:2204.03455, 2022.
Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-vidoo generation via transformers. arXiv preprint arXiv:2205.1588 2022.
Berthold KP Horn and Brian G Schunck. Determining optical flow. Artificial intelligence, 17(1-3)1155-2010 1981.
Hyconho Jeong and Jong Chul Ye. Ground-a-vidco: Zero-shot grounded video editing using text-to-image diffusion models. arXiv preprint orXiv:2310.01100 01100077,00 2023.
Max Ku, Tianle Li, Kai Zhang, Yujie Lu, Xingyu Fu, Wenwen Zhuang, and Wenhu Chen. Imagenhub: Standardizing the evaluation of conditional image generation models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview http://pwrrrvii....oooe/ror///// net/foruuridd-DaVe2k...
Wing-Fung Ku, Wan-Chi Siu, Xi Cheng, and H. Authony Chan. Intelligent painter: Picture composition with resampling diffusion model, 2022.
Tianle Li, Max Ku, Cong Wei, and Wenhu Chen. Dreamedit: Subject-driven image editing, 2023.
Feng Liang, Bichen Wu, Jialiang Wang, Licheng Yu, Kunpeng Li, Yinan Zhao, Ishan Misra, Jia-Bin Huang, Peizhao Zhang, Peter Vajda, et al. Flowvid: Taming imperfect optical flows for consistent video-to-video synthesis. arXiv preprint arXiv:2312. 17681, 2023.
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.0m 04 761, 2023a.
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. arXiv:2309.0 04761, 2023b.
Winfried Lötzsch, Max Reimann, Martin Büßemeyer, Amir Semmn, Jürgen Döllner, and Matthias Trapp- Wise: Whitebox image stylization by example-based learning. ECCV, 2022.
Ron Mokady, Amir Hertz, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 6038 6047, 2023.
14

---

Published in Transactions on Machine Learning Research (11/2024)
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Tlya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text- guided diffusion models. In International Conference on Machine Learning, pp. 16784 16804. PMLR, 2022.
OpenAT. Video generation models as world simulators.
URL https://openai.ccmmreeesss
Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing. arXiv preprint orXi::2308.0783 07926, 2023.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv:2303.0940 09535, 2023.
Haonan Qiu, Menghan Xia, Yong Zhang, Yingqing He, Xintao Wang, Ying Shan, and Ziwei Liu. Freenoise: Tuning-free longer video diffusion via noise rescheduling, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Weiming Ren, Harry Yang, Ge Zhang, Cong Wei, Xinrun Du, Stephen Huang, and Wenhu Chen. Consisti2v: Enhancing visual consistency for image-to-vides generation. arXiv preprint arxiv:2402. 04324, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image senthesis with latent. diff'usion models. In Proceedings of the IEEE/CVP conference on compuler vision and pattern recognition, pp. 10684 10695, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint urXiv:2010.02500 2020.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-robust large mask inpainting with fourier convolutions. arXiv preprint arXiv:2109.077 071611 2021.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IREE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1921 1930, June 2023a.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven image-to-image translation. In Proceedings of the IEFE/CVF Conference on Compuler Vision und Pallern Recognition, pp. 1921-192299 1930, 2023b.
Haofan Wang, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. Instantstyle: Free lunch towards style-preserving in texl-to-image generation. arxiv preprinl arXiv:240402273. 2024a.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot identity-preserving generation in seconds. arXiv preprint arXiv:2401007519 2024b.
Xiang Wang, Hangjic Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion controllability. Advances in Neural Information Processing Systems, 36, 2024c.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang, Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. Xix preprint ar Xxx:23090.15000 2023.
15

---

Published in Transactions on Machine Learning Research (11/2024)
Bichen Wu, Ching- Yao Chuang, Xiaoyan Wang, Yichen Jia, Kapil Krishnakumar, Tong Xiao, Feng Liang, Licheng Yu, and Peter Vajda. Fairy: Fast parallelized instruction-guided video-to-video synthesis. arXiv preprint ar Xiv:22312 13834, 2023a.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tunc-a-video: Onc-shot tuning of image diffusion models for text-lovvideo generation. In Proceedings of the IEEE/CVE International Conference on Compuler Vision,
pp. 1923-7633, 2023b.
Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initialization gap in video diffusion models. arXiv preprint orXiv:2312.004 2023c.
Jinbo Xing, Menghan Xia, Yong Zhang, Haoxin Chen, Xintao Wang, Tien-Tsin Wong, and Ying Shan. Dynamicrafter: Animating open-domain images with video diffusion priors. arXiv preprint arXiv:2310.12190, 2023.
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. In ACM SIGGRAPH Asia Conference Proceedings, 2023.
David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent diffusion models for text-to-video generation. arXiv preprint Xr1:222309.158 2023a.
Lvmin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023b.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality imago- to-idoo synthesis via cascaded diffusion models. arXiv preprinl 2xx 2023c.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-Tree controllable text-to-video generation. aixic preprint arx::2305. 13077, 2023d.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Appendix
A Discussion on Model Implementation Details
When adapting AnyV2V to various I2V generation models, identify two sets of hyperparameters that are crucial to the final video editing results. They are (1) selection of U-Net decoder layers (41, le and 1a) to perform convolution, spatial attention and temporal attention injection and (2) Injection thresholds Teomes Tsa and Tta that control feature injections to happen in specific diffusion steps. In this section, we provide more discussions and analysis on the selection of these hyperparameters.
A.1 U-Net Layers for Feature Injection
To better understand how different layers in the I2V denoising U-Net produce features during video sampling, we perform a visualization of the convolution, spatial and temporal attention features for the three candidate 12V models 12VGen-XT. (Zhang et al. 2023c/, ConsistI2V [Ren et al. 2024) and SEINE (Chen et al.] 2023d]. Specifically, we visualize the average activation values across all channels in the output feature map from the convolution layers, and the average attention scores across all attention heads and all tokens (i.e. average attention weights for all other tokens attending to the current. token). The results shown in Figure 7
According to the figure, we observe that the intermediate convolution features from different 12V models show similar characteristics during video generation: carlier layers in the U-Net decoder produce features that represent the overall layout of the video frames and deeper layers capture the high-frequency details such as edges and textures. We choose to set 1 = 4 for convolution feature injection to inject background and layout guidance to the edited video without introducing too many high-frequency details. For spatial and temporal attention scores, we observe that the spatial attention maps tend to represent the semantic regions in the vidoo frames while the temporal attention maps highlight the foreground moving subjects (e.g. the running woman in Figure 7 One interesting observation for I2VGen-XL is that its spatial attention operations in deeper layers almost become hard attention, as the spatial tokens only attend to a single or very few tokens in each frame. We propose to inject features in decoder layers 4 to 11 (42 = la = {4, 5,.,,...]]
to preserve the semantic and motion information from the source video.
A.2 Ablation Analysis on Feature Injection Thresholds
We perform additional ablation analysis using different feature injection thresholds to study how these hyperparameters affect the edited vidco.
Effect of Spatial Injection Thresholds Teone, Tea We study the effect of disabling spatial feature injection or using different Tconv and Tsa values during video editing and show the qualitative results in Figure g We find that when spatial feature injection is disabled, the edited videos fail to fully adhere to the layout and motion from the source video. When spatial feature injection thresholds are too high, the edited videos are corrupted by the high-frequency details from the source video (e.g. textures from the woman's down jacket in Figure Setting Teone
Tsa 0.2T achieves desired editing outcome for our experiments.
Effect of Temporal Injection Threshold Tia We study the hyperparameter of temporal feature injection threshold Tta in different scttings and show the results in Figure D We observe that in circumstances where Tra < 0.5T (T is the total denoising steps), the motion guidance is too weak that it leads to only partly aligned motion with the source video, even though the motion itself is logical and smooth. At Tea > 0.52', the generated video shows a stronger adherence the motion but distortion occurs. We employ Tea 0.57 in our experiments and find that this value strikes the perfect balance on motion alignment, motion consistency, and video fidelity.
17

---

Published in Transactions on Machine Learning Research (11/2024)
12VGen-XL
Consisti2V
SEINE
Generated Video
Layer
Convolution Features
Layer 4
Layer 7
Layer 11
Layer 4
Spatial Attention Scores
Layer 7
Layer 11
Layer 4
Temporal Attention Scores
Layer 7
Layer 11
Low
High
Figure 7: Visualizations of the convolution, spatial attention and temporal attention features during video sampling for T2V generation models' decoder layers. We feed in the DDIM inverted noise to the T2V models such that the generated videos (first row) reconstructions of the source video.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Source
0
Tr = 0.2T
Tea 0.57
TGp = 0.77
T
Franc I
Fracso 10
Figure 8: Hyperparameter study on spatial feature injection. We find that Tea 0.2T is the best setting for maintaining the layout and structure in the edited video while not introducing unnccessary visual details from the source video. Te,s represents Tcone and Tsa (Editing prompt: teddy bear running. The experiment was conducted with the T2VGen-XT, backbone.
Source
Tc. 0.27
Tc.s 0.5T
Tc,s 0.7T
Tc
Freme 10
Frazs 10
Figure 9: Hyperparameter study on temporal feature injection. We find that Tra 0.5I to be the optimal setting as it balances motion alignment, motion consistency, and vidco fidelity. (Editing prompt: darth vader walking. The experiment was conducted with the SEINE backbone.
B Evaluation Detail
B.1 Quantitative Evaluations
Prompt-based Editing For (1) prompt-based editing, we conduct a human evaluation to examine the degree of prompt alignment and overall preference of the edited videos based user voting. We compare
19

---

Published in Transactions on Machine Learning Research (11/2024)
Any V2V against three baseline models: Tume-A-Video (Wu et al. 2023b J, TokenFlow (Geyer et al. 2023) and
FLATTEN (Cong et al. 2023). Human evaluation results in Table 2 demonstrate that our model achieves the best overall preference and prompt alignment among all methods, and V2y (12VGen-XL) is the most preferred method. We conjecture that the gain is coming from our compatibility with stato-of-the-art image editing models.
We also employ automatic evaluation metries on our edited video of the human evaluation datasets. Following previous works (Ceylan et al.; 2023 Bai et al. 2024], our automatic evaluation employs the CLIP (Radford et al. 2021) model to assess both text alignment and temporal consistency. For text alignment, we calculate the CLIP-Score, specifically by determining the average cosine similarity between the CLIP text emboddings derived from the editing prompt and the CLIP image embeddings across all frames. For temporal consistency, we evaluate the average cosine similarity between the CLIP image embeddings of every pair of consecutive frames. These two metrics are referred to as CLIP-Text and CLIP-Image, respectively. Our automatic evaluations in Table 2 demonstrate that our model competitive in prompt-bascd oditing compared to baseline methods.
Reference-based Style Transfer; Identity Manipulation and Subject-driven Editing For novel lasks (2), (3) and (4), we evaluate the performance of three 12V generation models using human evaluations and show the results in Table 5 As these tasks require reference images instead of text prompts, we focus on evaluating the reference alignment and overall preference of the edited videos. According to the results, we observe that Any (I2VGen-XL) is the best model across all tasks, underscoring its robustness and versatility in handling diverse video editing tasks. Any (SEINE) and AnyV2V (ConsistI2V) show varied performance across tasks. Any (SEINE) performs good reference alignment in pefeeence-based style transfer and identily manipulation, but falls short in subject-driven editing with lower scores. On the other hand, AnyV2V (ConsistI2V) shines in subject-driven editing, achieving second- best results in both reference alignment and overall preference. Since the latest image editing models have not yet reached a level of maturity that allows for consistent and precise editing (Ku et al.) 2024), we also report the image editing success rate in Table 5 to clarify that our method relies on good image frame edit.
B.2 Qualitative Results
Prompt-based Editing By leveraging the strength of image editing models, our V2V framework provides precise control of the edits such that the irrelevant parts in the scene are untouched after editing. In our experiment, used InstructPix2Pix (Brooks et al. 2023) for the first frame edit. Shown in Figure 3 our method correctly places party hat on an old man's 's head and successfully turns the color of an airplane to blue, while preserving the background and keeping the fidelity to the source video. Comparing our work with the three baseline models TokenFlow (Geyer et al. 2023), FLATTEN (Cong et al., 2023), and Tune-A-Videe A-ieo (Wu et al. 2023b), the baseline methods display either excessive or insufficient changes in the edited video to align with the editing text prompt. The color tone and object shapes are also tilted. It also worhh mentioning that our approach is far more consistent on some motion tasks such as adding snowing weather, due to the I2V model's inherent support for animating still scenes. The baseline methods, the other hand, can add snow to individual frames but cannot generate the effect of snow falling, as the per-frame or onc-shot editing methods lack the ability of temporal modeling.
Reference-basee based Style Transfer Our approach diverges from relying solely on textual descriptors for conducting style edits, using the style transfer model NST (Gatys et al., 2015) to obtain the edited frame. This level of controllability offers artists the unprecedented opportunity to use their art as reference for video editing, opening new avenues for creative expression. As demonstrated in Figure g our method captures the distinctive style of Vassily Kandinsky's 's artwork "C "Composition VII" and Vincent Van Gogh's 's artwork "Chateau in Auvers at Sunset" accurately, while such an edit is often hard to perform using existing text-guided vidco editing methods.
Subject-driven Editing In our experiment, we employed a subject-driven image editing model Any- Door (Chen et al. 2023c) for the first frame editing. AnyDoor allows replacing any object in the target image with the subject from only one reference image. We observe from Figure 4 that Any V2V produces highly
20

---

Published in Transactions on Machine Learning Research (11/2024)
motion-consistent videos when performing subject-driven object swapping. In the first example, AnyV2V successfully replaces the cat with a dog according to the reference image and maintains highly aligned motion and background as reflected in the source video. In the second example, the car is replaced by our desired car while maintaining the rotation angle in the edited vidco.
Identity Manipulation By integrating the identity-presereet image personalization model Instan- IID (Wang et al. 2024b) with ControlNet (Zhang et al., 2023b), this approach enables the replacement of an individual's 's identity to create an initial frame. Our V2V framework then processes this initial frame to produce an edited video, swapping the person's identity as showcased in Figure 4 To the best of our knowledge, InstantID with ControlNet method will alter the background due to its model properly. Tt. is possible to leverage other Seentity-preeever image personalization models and apply them to Any to preserve the background.
work is the first to provide such flexibility in the video editing models. Note that the
B.3
Human Evaluation
B.3.1
Dataset
Our human evaluation dataset contains a total samples that have been collected from https://www pexels. com For prompt-based editing, we employed InstructPix2Pix (Brooks et al., 2023) to compose the examples. Topics include swapping objects, adding objects, and removing objects. For subject-driven editing, employed AnyDoor (Chen et al. 2023c) to replace objects with reference subjects. For Neural Style
Transfer, employed NST (Galys et al. 2015 to compose the examples. For identity manipulation, we employed InstantIl) [Wang et al. 2024b to compose the examples. See Table for the statistic.
Table 4: Number of entries for Video Editing Evaluation Dataset
Category
Number of Entries
Image Editing Models Used
Prompt-based Editing
45
InstructPix2Pix
Reference-based Style Transfer
20
NST
Identity Manipulation
13
InstantID
Subject-driven Editing
11
AnyDoor
Total
89
B.3.2 Interface
In the evaluation setup, we provide the evaluator with generated images from both the baseline models and the AnyV2V models. Evaluators are tasked with selecting videos that best align with the provided prompt the reference image. Additionally, they are asked to express their overall preference for the edited videos. For detailed view of the interface used in this process, please Figure 10
C Discussion
C.1 Limitations
Inaccurate Edit from Image Editing Models. As our method relies on an initial frame edit, the image editing models are used. However, the current state-of-the-art models not mature enough to perform accurate edits consistently (Ku et al. 2024). For example, in the subject-driven video editing task, we found that Door (Chen et al. 2023c requires several tries to get good editing result. Efforts are required in manually picking a good edited frame. We expect that in the future better image editing models will minimize such effort.
Limited ability of I2V models. We found that the results from our method cannot follow the source video motion if the motion is fast (e.g. billiard balls hitting each other at full speed) complex (e.g.
21

---

Published in Transactions on Machine Learning Research (11/2024)
Soare Video
Video
2
Video
Viteo
Vike
Video
abgee willing promptt opply
Valco
Vilm
Valee
The oe bycire and ma dhale vay ball you dink are experied be edkedd vadevia book
Vile
Valco
iftt
Figure 10: The interface individual evaluation.
Table 5: Comparisons for three 12V models under V2V framework on novel video editing tasks. Align:
reference alignment; Overall: overall preference. Bold: best results; Underline: top-2.
Reference-based
Subject-driven
Identity
Task
Style Transfer
Editing
Manipulation
Image Editing Method
NST
AnyDoor
InstantID
Image Editing Success Rate
~00%
=10%
800
Human Evaluation
Align f
Overall f
Align
Overall
Aligu 1
Overall
V2V (SEINE)
92.3%
30.8%
48.4%
15.2%
72.7%
18.2%
V2y (ConsistI2V)
38.4%
10.3%
63.6%
42.4%
72.7%
27.3%
Any (I2VGen-XL)
100.0%
76.9%
93.9%
84.8%
90.1%
45.4%
person clipping her hair). One possible reason is that the current popular I2V models are generally trained slow-motion videos, such that lacking the ability to regenerate fast. complex motion even with motion guidance. We anticipate that the presence of robust 12V model can address this issue.
C.2 License of Assets
For image editing models, InstructPix2Pix Brooks et al. 2023) inherits Creative ML OpenRAIL-M License as it is built upon Stable Diffusion. Neural Style Transfer (Gatys et al.] 2015) under Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. InstantID (Wang et al, 2024bb is under Apache License 2.0. AnyDoor (Chen et al.) 2023c) is under the MIT License.
For baselines, Lime-A-Vides (Wu et al., 2023b) is under Apache License 2.0, TokenFlow (Geyer et al. is under MIT License andFLATTEN (Cong et al. 2023) is under Apache License 2.0.
2023
For the human evaluation dataset, the dataset. has been collected from https: http///wwwwwwwe...o///// com with all data governed by the terms outlined at attpp://www. wwwpw/wwwwwwww....ppppppp cww/license/
We decide release Vny code under the Creative Commons Attribution 4.0 License for casy access in the research community.
22

---

Published in Transactions on Machine Learning Research (11/2024)
C.3 Societal Impacts
Postive Social Impact. Any has the potential to significantly enhance the capabilities of video editing systems, making it easier for a wider range of users to manipulate images. This could have mumerous positive social impacts, as users would be able to achieve their editing goals without needing professional editing knowledge, such as using Photoshop or painting.
Misinformation spread and Privacy violations. As our technique allows for object manipulation, it can produce highly realistic yet completely fabricated videos of one individual or subject. There is a risk that harmful actors could exploit system to generate counterfeit videos to disseminate false information. Moreover, the ability to create convincing counterfeit content featuring individuals without their permission undermines privacy protections, possibly leading to the illicit use of a person's 's likeness for harmful purposes and damaging their reputation. These issues are similarly present DeepFake technologics. To mitigate the risk misuse, one proposed solution is the adoption unseen watermarking, a method commonly used to tackle such concerns in image gencration.
C.4 Safeguards
It crucial to implement proper safeguards and responsible frameworks when developing user-friendly video editing systems. For the human evaluation dataset, we manually collect a diverse range of images ensure balanced representation of objects from various domains. We only collect images that are considered safe.
C.5 Ethical Concerns for Human Evaluation
We believe our proposed human evaluation does not incur ethical concerns due to the following reasons: (1) the study does not involve any form of intervention or interaction that could affect the participants well-being. (2) Rating video content does not involve any physical psychological risk, nor does it expose participants to sensitive or distressing material. (3) The data collected from participants will be entirely anonymous and will not contain any identifiable private information. (4) Participation in the study is entirely voluntary, and participants can withdraw at any time without any consequences.
C.6 New Assets
Our paper introduces several new assets including a human evaluation dataset and demo videos generated by V2V. Each asset is thoroughly documented, detailing its creation, usage, and any relevant methodologies. The human evaluation dataset documentation includes details on how participant consent was obtained. The demo videos are provided as an anonymized zip file comply with submission guidelines, with detailed instructions for use. All assets are shared under an open license facilitate reuse and further research.
23

---

Published in Transactions on Machine Learning Research (11/2024)
D
More AnyV2V Showcases
Frame
Frame
Frame
Frame 16
Source
Robot
Source
Zebra
Source
Snow
Figure 11: V2V becomes an instruction bastreetion-based video editing tool when plugged with cnarnctiionguuided image editing models like InstructPix2Pix (Brooks et al. 2028 Prompt used "Turn the couple into robots", "Turn horse into zebra", and "Turn the sand into snow".
24

---

Published in Transactions on Machine Learning Research (11/2024)
Frame
Frame
Frame
Frame 16
Source
Figure 12: The more recent model InstantStyle (Wang et al. 2024a) can seamlessly plug in Any V2V to perform reference- based style transfer video editing. The reference art image in the bottom left corner is used to retrieve the first image edit.
25

---

Published in Transactions on Machine Learning Research (11/2024)
Frame 1
Frame
Frame
Frame 16
Source
Source
Figure 13: V2V can perform subject-driven video editing with a single image reference, using subject- driven image editing model like AnyDoor (Chen al. 2023c The subject image in the bottom left corner is used to retrieve the first image edit.
26


Reviews:
[{"review_id": "oVIdijkN3m", "reviewer": "Reviewer_oTJF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The framework's capability to support various video editing tasks, including prompt-based editing, reference-based style transfer, subject-driven editing, and identity manipulation, demonstrates its broad applicability.\n\nThe article presents experiments showing that AnyV2V significantly outperforms other baseline methods in both automatic and human evaluations, which is a testament to its effectiveness. By breaking down the video editing task into a single image editing and image-to-video generation, AnyV2V simplifies the editing process.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The introduction of AnyV2V as a novel tuning-free framework is a notable advancement in video editing research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Can you provide some case studies? In the four tasks mentioned in the experiment, which tasks did the AnyV2V model perform poorly? And in which scenarios did the generated videos perform poorly in these four tasks?\n\nIn addition to the prompt-based video editing task in Tab. 2, can you compare AnyV2V, which is based on the I2V model, with other video editing models that are based on the T2I model, such as CCEdit: Creative and Controllable Video Editing via Diffusion Models, presented at CVPR 2024, in the other three tasks?\n\nCan you provide more visual results of AnyV2V in video editing? For example, in the form of videos. Also, some visual comparison results with other video editing methods would help demonstrate the performance of AnyV2V in video editing.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dLbeR77u3X", "reviewer": "Reviewer_3TgA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. sufficient experimental data to demonstrate the flexibility and the quality of the proposed method.\n2. relatively simple and straightforward approach that does not require additional model training/tuning.\n3. open sourced\n\nWeaknesses:\n1. CLIP-image score does not seem to be a good metrics in this work, as author discussed in Section 5.4 - Temporal Feature Injection, i.e. while this injection is disabled and CLIP-score increased, author still believes the quality of the video is degraded. Similarly in Table 2, the three baseline models outperformed the proposed methods in CLIP-image scores but contradicted with human evaluation greatly. Including inconsistent metrics might cause confusions to the readers.\n\n2. The author refers the proposed method as an image editing framework mainly because the information is extracted from the source and injected into the edited video in a frame by frame fashion. However, if the goal is not to generate new content but only editing existing frames, some of the applications shown in this work could simply be achieved by applying image editing on every single frame (without invoking I2V and injections), e.g. style transfer and change the color of an object. Maybe the author can add a few sentences to elaborate the pros and cons vs this 'naive approach'.  \n\n3. In abstract, the last sentence claims that the proposed method \"significantly outperforms other baseline methods in automatic and human evaluations by a significant margin...\" However, results in Table 2 do not seem to support this claim. While Table 2 does show that Human Evaluations are significantly better than prior arts, CLIP scores are comparable or even slightly worse in some cases. Author may want to provide additional data to support, or consider revise, this claim.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed to a video editing method by combining existing image editing models with image-to-video (I2V) models. The main idea is to apply the existing, desired image editing models on the first frame, and then utilize the information extracted from the source video, including temporal and spatial features and DDIM inverted latent to create video with proper injections. Because all the editing and I2V models are found in previous works, no additional training or tuning is needed. Four types of editing approaches combined with three I2V models were demonstrated in this study. (not all the 4x3 combinations though.) Quantitative comparisons with previous works, using human evaluation scores and CLIP scores, show that the proposed method is superior in human evaluations and comparable in CLIP scores.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "please see weakness above", "broader_impact_concerns": "author already include this part in appendix C.3.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CuJAGBSOhr", "reviewer": "Reviewer_VExa", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n* The idea is surprisingly simple and it does not require further fine-tuning on existing model.\n\n***\n### Weaknesses:\n* The methodology parts is too abbreviating. It’s unclear how the frames other than the first frame is encoded into the latent and help the video editing. \n* From a high-level perspective, this paper can be viewed as an extension of the PnP diffusion features in video domain, by additionally injecting the temporal information into the decoding model. However, as I mentioned, this part is not presented clearly.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscripts proposed a tuning-free framework for video to video transformation, leveraging different kinds of existing pre-trained image editing model and image to video model. The core idea is first editing the first frame, then generating the video with the structural information as well as the spatial and temporal information obtained from the DDIM inversion.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please elaborate more on how the temporal information is leveraged during the decoding. Also please separate the diffusion steps and frame steps to make the presentation clear.", "broader_impact_concerns": "Not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers generally lean toward positive, noting that AnyV2V presents a simple yet effective tuning-free framework for video editing. The key strength highlighted is the method's broad applicability across various editing tasks without requiring additional training. While the technical implementation is straightforward, reviewers expressed concerns about limited scientific novelty, which is reflected in their weak opposition to including the work in the ICLR Journal-to-Conference track.

After the authors' responses and additional supplementary materials, it seems to me the reviewers' concerns about methodology clarity and empirical evidence were mostly addressed. The authors demonstrated the advantages of their approach over frame-by-frame editing and clarified the role of temporal feature injection. They also corrected their claims about automatic evaluation metrics.
Decision: Accept (poster)