Consistent Video-to-Video Transfer Using Synthetic Dataset
Authors: Jiaxin Cheng
Tianjun Xiao
Tong He
Venue: ICLR 2024 poster (2024)
Keywords: Computer Vision
Video Editing
Diffusion Model
Primary area: generative models
TL;DR: We've developed a synthetic dataset to train a text-based video editing model, eliminating the need for per-video fine-tuning, and introduced a method for seamless long video editing.

Abstract: We introduce a novel and efficient approach for text-based video-to-video editing that eliminates the need for resource-intensive per-video-per-model finetuning. At the core of our approach is a synthetic paired video dataset tailored for video-to-video transfer tasks. Inspired by Instruct Pix2Pix's image transfer via editing instruction, we adapt this paradigm to the video domain. Extending the Prompt-to-Prompt to videos, we efficiently generate paired samples, each with an input video and its edited counterpart. Alongside this, we introduce the Long Video Sampling Correction during sampling, ensuring consistent long videos across batches. Our method surpasses current methods like Tune-A-Video, heralding substantial progress in text-based video-to-video editing and suggesting exciting avenues for further exploration and deployment.

Published as a conference paper at ICLR 2024
CONSISTENT VIDEO-TO-VIDEO TRANSFER USING SYNTHETIC DATASET
Jiaxin Cheng, Tianjun Xiao & Tong He Amazon Web Services Shanghai Al Lab {cjiaxio iaxin, tianjux, httong)emmazzo.......
ABSTRACT
We introduce a novel and efficient approach for text-based video-to-video editing that eliminates the need for resource-intensive per-video-per-moot finetuning. At the core of our approach is synthetic paired video dataset tailored for video-to- video transfer tasks. Inspired by Instruct Pix2Pix's image transfer via editing instruction, adapt this paradigm to the video domain. Extending the Prompt- to-Prompt to vidcos, we efficiently generate paired samples, each with an input video and its edited counterpart. Alongside this, introduce the Long Video Sampling Correction during sampling, ensuring consistent long videos across batches. Our method surpasses current methods like Tune-A-Video heralding substantial progress in text-based video-to-video editing and suggesting excit- ing avenues for further exploration and deployment, https: http:///tt.tttts///ttttt......o amazon wwwzzonssceene///e hmmpor/ceecaaacccoco...c..ettete
Original video
Make the red Porsche and drive alone beach (Editing type: Multiple)
Make it snowy (Editing type: Background)
Make it watercolor (Editing type: Style)
Figure 1: InsV2V has versatile editing capabilities encompassing background, object, and stylistic changing. Our method adopts one-model-all-video strategy, achieving comparable performance while necessitating only inference. InsV2V eliminates the need to specify prompts for both original and target videos, simplifying the process by requiring only an edit prompt, thereby enhancing intuitiveness in video editing.

---

Published as a conference paper at ICLR 2024
INTRODUCTION
Text-based video editing Wu et al. 2022): : Zhao et al. (202)) Wang et al. (2023a); Oi et al. 2023)
Liu et al. (023) has recently garnered signilicant interest as versatile tool for multimedia content manipulation. However, existing approaches present several limitations that undermine their practi- cal utiliiy. Firstly, traditional methods typically require weieo--ee--eeeeee
finetuning, which im- poses a considerable computational burden. Furthermore, current methods require users to describe both the original and the target video Wu et al. (2022); Zhao et al. (2023): Wang et al. (2023a); Q et al. (2023 Liu ct al. 2023). This requirement is counterintuitive,
users generally only want to specify what edits they desire, rather than providing a comprehensive description of the original content. Moreover, these methods are constrained to individual video clips: if a video is too long to fit into model, these approaches fail to ensure transfer consistency across different clips.
To overcome these limitations, we introduce novel method with several distinctive features. Firstly, our approach offers a universal one model-all-vides transfer, freeing the process from per-video-per model finetuning. Moreover, our model simplifies user interaction by only necessitating intuitive editing prompt, rather than detailed descriptions of both the original and target videos, to carry out desired alterations. Secondly, we develop a synthetic dataset precisely crafted for video-to- video transfer tasks. Through rigorous pairing of text and video components, we establish an ideal training foundation for our models. Lastly, we introduce a sampling method specifically tailored for generating longer videos. By using the transferred results from preceding batches as reference, we achieve consistent transfers across extended video sequences.
We introduce Instruct Video-to-Video V2V), diffusion-based model that enables video editing using only an editing instruction, eliminating the need for per-video-per-molee bility is inspired by Instruct Pix2Pix [Brooks et al. (2023), which similarly allows for arbitrary image editing through textual instructions. A significant challenge in training such a model is the scarcity of naturally occurring paired video samples that can reflect an editing instruction. Such video pairs are virtually nonexistent in the wild, motivating us to create a synthetic dataset for training.
tuning. This capa-
Our synthetic video generation pipeline builds upon large language model (LLM) and the Prompt- to-Prompt |Hrrzz et al. (2022) method that is initially designed for image editing tasks (Figure 2 We use example-driven in-context learning approach guide the LLM produce these paired video descriptions. Additionally, we adapt the Prompt-to-Promp (PTP) method to the video domain by substituting the image diffusion model with a video counterpart Ho et al. (022c). This modifi- cation enables the generation of paired samples that consist of an input video and its edited version, precisely reflecting the relationships delineated by the editing prompts.
In addressing the limitations of long video editing in conventional video editing methods, we intro- duce Long Video Sampling Correction (LVSC). This technique mitigates challenges arising from fixed frame limitations and ensures seamless transitions between separately processed batches of a lengthy video. LVSC employs the final frames of the previous batch as a reference to guide the generation of subsequent batches, thereby maintaining visual consistency across the entire video. We also tackle issues related to global or holistic camera motion by introducing a motion compen- sation feature that uses optical flow. Our empirical evaluations confirm the effectiveness of LVSC and motion compensation in enhancing video quality and consistency.
2 RELATED WORK
Diffusion Models The advent of the diffusion model Sohl-Dickstein et al. 2015): Ho et al. (2020) has spurred significant advancements in the field of image generation. Over the course of just a few years, we have observed the diffusion model making groundbreaking progress in a variety of fields. This includes areas such as super-resolution Saharia et al. (202c)). colorization /aharia et al. 2022a). novel view synthesis Watson et al. (2022), style transfer Zhang et al. (2023), and 3D gen- eration Poole et al. (022); Tang et al. 2023h; Cheng et al. (023b). These breakthroughs have been achieved through various means. Some are attributable to the enhancements in network struc- tures such as Latent Diffusion Models (also known as Stable Diffusion) Rombach et al. (2022).
GLIDE Nichol et al. (2022), DALLE2 Ramesh et al. (222)) SDXL Podell et al. (2023), and Ima- gen Saharia et al. (2022b). Others are a result of improvements made in the training paradigm Nichol
GLIDENNichol
& Dhariwal (2021) Song & Ermon (2019): Dhhrrwal & Nichol (002): Song al. (2020b a). Pur-
2

---

Published as a conference paper at ICLR 2024
thermore, the ability to incorporate various conditions during image generation has played crucial role. These conditions include elements such as layout Cheng et al. (2023a); Rombach et al. (2022) segmentation Avrahami et al. (2023) 2022
the use Balaji et al. (2022)) Yang al. (2023a),
of an image as reference Mou et al. 2023) Ruiz et al.
b: Gal et al. (2022).
Diffusion-based Text-Guided Image Editing Image editing is process where we don't desire completely unconstrained generation but modifying an image under certain guidance (i.e.a reference image) during its generation. Various methods have been proposed address this task. Simple zero- shot image-to-iman translation methods, such as SDEdit |Meng et al. (2021) performed through diffusion and denoising on reference image. Techniques that incorporate degree of optimization, such as Imagic Kawar et al. (2023), which utilizes the concept of textual inversion Gal et al. (2022) and Null-text Inversion Mokady et al. 2023p, which leverages the Prompt-to-Prompt strategy Hertz et al. (2022) to control the behavior of cross-attention in the diffusion model for editing, have also been explored. These methods can impede the speed of editing due to the necessity for per-image- per-optimization. Models like Instruct Pix2Pix Brooks et al. (2023) have been employed to achieve image editing by training on synthetic data, This approach adeptly balances editing capabilities and fidelity to the reference image.
Diffusion-based Text-Guided Video Editing The success of the diffusion model in image genera- tion has been extended to vidco generation well Ho et al. (2022b): Harvey al. (2022) y; Blattmann et al. (2023 : Mei & Patel (2023; Ho et al. (2022a), and similarly, text-guided video editing has
sparked interest within the community. Techniques akin those in image editing have found appli-
cations in video editing. For instance, Dreamix Molad et al. (2023) uses diffusion and denoising for
video editing, reminiscent of the approach in SDEdit Meng et al. (202T). Strategies altering the be-
havior of the cross-attention layer to achieve editing, like the Prompt-to-Prompe Hertz et al.
2022
have been adopted by methods such as Vid2Vid Zero Wang al. (2023a), FateZero Qi et 2023
and rideo-p2ple Liu et al. (2023). Recent developments Wang et al. 2 Esser et al. (2023): Zhao
et al. (2023) leverage certain condition modalities extracted from the original video, like depth maps or edge sketches, to condition the video generation. The related, albeit non-diffusion-base method
Text2live Bar- Tal et al. (2022) also provides valuable perspectives on video editing.
User Inputs
Task Desoription
Examples
Task Specification
You will creatively generate
Here are some success examples: in Input: Rex
Generate triplet for following inpurs: In Fun clown 3d animation \n Happy family using laptoo on bed at home 'n
peired triplet for synthetic generation video clip...
Deanland, Charing Cross, watercolour, 9.12
make painting 'n Output:
Beanland Charing Croes ol painting, 9
Synthetic Video Pair
LLM Outputs
Imput: clown -3d
CLIP Filter
Video Diffusion Model + Parmpt-too-Prmmpt
Edit: change the clown to knight in shining armor,, Output: Fun knight shining 3d animation.
LLM
The pipeline for generating synthetic dataset using a large language model, whose out- puts include the prompt triplet consisting of input, edit, and edited prompts, as well as correspond- ing pair of videos. Visualization of generated videos can found Appendix B
Figure 2:
3 SYNTHETIC PAIRED VIDEO DATASET
A crucial element for training model capable of arbitrary video-to-video transfer, as opposed to a ger-video-per-mene
approach, lies in the availability of ample paired training data. Each pair consists of an input video and its corresponding edited version, providing the model with diverse range of examples essential for generalized performance. However, the scarcity of naturally oceur- ring video pairs with such correspondence poses significant challenge to the training process.
To address this, we introduce the concept a trade-off between initial training costs, including dataset creation, and long-term efficiency. We advocate for the use of synthetic data, which, while incurring an upfront cost, accurately maintains the required correspondence and fulfills the con- ditions for effective video-to-video transfer learning. The merit of this synthetic data generation approach underscored by its potential to offset the initial investment through substantial time savings and efficiency in the subsequent inference stages. This approach contrasts with 'per-vid-
3

---

Published as a conference paper at ICLR 2024
per-model' methods that necessitate repetitive fine-tuning for each new video, making strategy both cost-effective and practical in diverse real-world applications.
The potential of synthetic data generation, well-documented in the realm of text-guided edit- ing Brooks et al. (2023). is thereby extended to video-to-vidco transfer. This method allows us to construct the optimal conditions for the model to learn, offering a practical solution to the inherent obstacles associated with acquiring matching real-world video pairs.
3.1 DATASET CREATION:
In order to generate the synthetic dataset, we leverage the approach of Prompt-to-Promp (PTP) Hertz et al. (2022), proven method for producing paired samples in the field of image editing. The PTP employs both self-attention and cross-attention replacements to generate seman- tically aligned edited images. In self-attention, the post-softmax probability matrix of the input prompt replaces that of the edited prompt. The cross-attention replacement specifically swaps the text embedding of the edited prompt with that of the input prompt.
In the context of video-to-video transfer, we adapt PTP by substituting its underlying image diffu- sion models with video diffusion model. In addition, we extend the self-attention replacement to temporal attention layers, critical modification for maintaining structural coherence between input and edited videos. Figure 2 shows the overall pipeline for data generation. To guide the synthetic data generation,
employ set of paired text prompts, comprising input prompt, edited prompt, and an edit prompt. The input prompt corresponds to the synthetic original video, while the edited prompt and the edit prompt represent the desired synthetic edited video and the specific changes to be applied on the original video respectively.
3.2 PROMPT SOURCES
Our synthetic dataset is constructed using paired prompts from two differentiated sources, each serving specific purpose in the training process. The first source, LAION-IPTP, employs fine- tuned GPT-3 model to generate prompts based on 700 manually labeled captions from the LAION- Aesthetics dataset
2023 This yielded set of 450,000 prompt pairs, of which 304,168 were utilized for synthetic video creation. While the GPT-3-based prompts offer a substantial vol- ume of data, they originate from image captions and thus have limitations in their applicability to video generation. This led us to incorporate a second source, WebVid-MPT, which leverages video- specific captions from the Web Vid 10M dataset Bain et al. (202T). Using MPT-30B [ (2023) in a zero-shot manner, we devised a set of guidelines (see Appendix AA for details) to generate the needed paired prompts, adding an additional 100,000 samples to our dataset. Crucially, the WebVid-MPT source yielded threefold increase in the success rate of generating usable samples compared the LAION-IPTP source after sample filtering, reinforcing the need for video-specific captions in the training process. The LAION-IPTP prompt source demonstrated success rate of 5.49% (33,421 successful generations from 608,336 attempts). The WebVid-MPT prompt source showed higher success rate of 17.49% (34.989 successful generations from 200,000 attempts).
3.3
IMPLEMENTATION DETAILS AND SAMPLE SELECTION CRITERIA
Implementation Details: We use public available text-to-video mode for generating synthetic videos. Each video has 16 frames, processed over 30 DDIM Song et al. (220a) steps by the dif- fusion model. The self-attention and cross-attention replacements terminate at random step within the ranges of 0.3 to 0.45 and 0.6 to 0.85 respectively (out of 30 steps). The classifier-free guidance scale a random integer value between 5 and 12.
the Trompt-to-Promp model
Sample Selection Criteria:
To ensure the quality of our synthetic dataset, we employ CLIP-based filtering. For each prompt, generation is attempted with two random seeds, and three distinct clip scores are computed: CLIP Text Score: Evaluates the cosine similarity between each frame and the text. CLIP Frame Score: Measures the similarity between original and edited frames. CLIP Direction Score: Quantifies the similarity between the transition of iigiiall-famee ceaeeeett--diidd-iieee frame and triginal-eexxt-o-cddida-tttttt These scores are obtained for each of the 16 frames and averaged.

---

Published as a conference paper at ICLR 2024
Architecture of Instruct Video-to-Video
Edit Prompt: Make the car red Porsche and drive alone beach
Text Encoder
Preceding Batch
Concatenate Latent
Encoder
Decoder
LySC
Subsequent Batch
Denosing UNet
(b) Inflation
(c) Long Video Sampling Correction (LVSC) Noisy
Ref. Frames
Infiated Conv Attn
Temporal Atlention Dexx xloow
xxxxxfxhee Xhow
Reshape
gReshape x IR (Dbiww)cc
TM/
R xxxxxx
Infer Noise on
Ref. Frames
Convolution
Softmax
Subsequent Frames
O
Denosing UNet
Score Correction
MultiHead Attn
Feed Forward
Reshape
Reshape
[ Xxxw
Rebxck Xibow
Final Prediction:
Frozen Weights
Trainable Weights
Reference Frames from Preceding Batch
Figure 3: (a) The architecture of Ins V2V. For handling long videos processed in multiple batches, our approach leverages the proposed LVSC utilize the final frames from the preceding batch as reference frames for the subsequent batch. (b) The inflated convolutionals and attention layer, as well as temporal attention layer can handle 5D video tensors by dynamically reshaping them. (c) During each denoising iteration, the LVSC adjusts the predicted noise Eg(2)) based on reference ref frames =/e//
prior to executing the DDIM denoising.
sample preserved if meets the following conditions: CLIP Text Score > 0.2 for both original and edited videos, CLIP Direction Score > 0.2, and CLIP Frame Score >> 0.5. Samples that fail to meet these criteria are discarded.
MODEL ARCHITECTURE
4.1 PRELIMINARIES
Diffusion models learn to predict content in an image by iteratively denoising entirely random Gaussian noise. In the training process, the input image is first corrupted by the Gaussian noise, termed diffusion. The model's 's task is to restore diffused noisy image its original form. This process can considered as the optimization of the variational lower bound on the distribution p(x) of the image with T step Markovian inverse process. Various conditions c. such as text, images, and layouts, can be incorporated into diffusion models during the learning process. The model Ep aim to train is conditioned text and its training loss be expressed as
= (...s ||e E0( c)ll
Where Fe is the result diffusing the input image æ at timestep E [1, using random noise s. In practice, employ the Latent Diffusion Model (LDM) Rombach et al. (022) our backbone model and condition on input videos by concatenating the videos to the latent space, as illustrated in Figure (a). Instead of generating images in the RGB pixel space, LDM employs trained Vector Quantized Variational AutoEncoder (VQVAE) Esser et al. (202T) to convert images to visual codes
in latent space. This allows the model to achieve better results with the same training resources.
5

---

Published as a conference paper at ICLR 2024
Specifically. the image is first transformed into latent code VQEnc(z), and the model learns to predict predict p(z) from random noise. In the context of the video editing, two distinct conditions exist: the editing instructions and the reference video, represented by er and cy respectively. The model is designed to predict p(z) by optimizing the following loss function:
ExygBm(()) c0(2,, 1, (v,,,,, CT)
4.2 INFLATE IMAGE-TO TO-IMAGE MODEL To VIDEO-TO-VIDEO MODEL
Given the substantial similarities between image-to-image transfer and video-to-video transfer, model utilizes a foundational pre-trained 2D image-to-image transfer diffusion model. Using foun- dational model simplifies training but falls short in generating consistent videos, causing noticeable jitter when sampling frames individually. Thus, we transform this image-focused model into a video-compatible one for consistent frame production. We adopt model inflation as recommended in previous studies Wu et al. (022);; Guo et al. 2023). This method modifies the single image diffu- sion model to produce videos. The model now accepts 5D tensor input x E fghxexfxxxx Given its architecture was designed for a 4D input, we adjust the convolutional and attention layers in the model (Figure 5 (b)). Our inflation process involves: (1) Adapting convolutional and attention layers to process 5D tensor by reshaping temporarily to 4D. Once processed, it's reverted to SD. (2) Introducing temporal attention layers for frame consistency. When these layers handle 5D tensor, they reshape it to a 3D format, enabling pixel information exchange between frames via attention.
4.3 SAMPLING
During sampling, we employ an extrapolation technique named Classifier-Free Guidance (CFG) Ho
& Salimans (2022) to augment the generation quality. Given that we have two conditions, namely the conditional video and the editing prompt, we utilize the CFG method for these two conditions as proposed in Brooks al. (002)). Specifically, for each denoising timestep, three predictions made under different conditions: the unconditional inference ép(Ze, (Ze, Ø, Ø), where both the condi- tions are an empty string and an all-zero video; the video-conditioned prediction éø (Zz, ev, Ø): and the video and prompt-conditiones prediction 60(2,, cy, er). Here, we omit timestep for symbolic convenience. The final prediction is an extrapolation between these three predictions with video and text classifier-free guidance scale sy 2 1 and ST 2 1.
E0(21, CT) = Ea(z,, 2,
(3)
+ sv (Ea(Ze) cV 2)- - 0, 88 (E0(21, - E0(2,, cv, 2))
4.4 LONG VIDEO SAMPLING CORRECTION FOR EDITING LONG VIDEOS
In video editing, models often face limitations in processing extended video lengths in one go. Alter- ing the number of input frames can compromise the model's efficacy, as frame count is usually preset during training. To manage lengthy videos, we split them into smaller batches for independent sam- pling. While intra-batch frame consistency is preserved, inter-batch consistency isn't 't guaranteed, potentially resulting in visible discontinuities batch transition points.
To address this issue, we propose Long Video Sampling Correction (LVSC): during sampling, the results from the final N frames of the previous video batch can be used reference to guide the (c)). This technique helps to maintain visual consistency across different batches. Specifically, let re =0 =prev [. -N :] L.N.N.e denote the last N frames from the transfer result of the previous batch. Here, to avoid confusion with the batch size, we set the batch size to 1. The ensuing batch is the concatenation
generation of the next batch (Figure 3
noisy reference frames and subsequent frames [æres, 2 On the model's prediction (x))) = E0(2,, t, cv, cr), we implement score correction and the final prediction is the summation between prediction e0(2)) and correction term ejef_ - - where e!e is the closed-form inferred noise on reference frames. For notation simplicity, we use =(2Fe/) and E(x)) to denote the model's predictions on reference and subsequent frames, though they are processed together by the model instead of separate inputs.

---

Published as a conference paper at ICLR 2024
Egel =t ref
Vorzzes
&Rl.Nachh
(4)
-T- Cz.
(x()) EZ(Z)) + N
(ero [., ((s
(5)
We apply averaging on the correction term when there are multiple reference frames as shown in Equations 4 and (5) where di is the diffusion coefficient for timestep t (i.e.z; ( (1 T)). In our empirical observations, we find that when the video has global or holistic camera mo- tion, the score correction may struggle to produce consistent transfer results. To address this issue, we additionally introduce a motion compensation that leverages optical flow to establish correspon- dences between each pair of reference frames and the remaining frames in the batch. We then warp the score correction in accordance with this optical flow with details presented in Appendix D
5 EXPERIMENTS
5.1
EXPERIMENTAL SETUP
Dataset For our evaluation, we used the Text-Guided Video Editing (TGVE) competition datase D The TGVE dataset contains 76 videos that come from three different sources: Videov, Youtube, and
DAVIS Perazzi et al. (2016). Every video in the dataset comes with one original prompt that de- scribes the video and four prompts that suggest different edits for each video. Three editing prompts pertain to modifications in style, background, or object within the video. Additionally, a multiple editing prompt provided that may incorporate aspects of all three types of edits simultaneously.
Metries for Evaluation Given that our focus is on text-based video editing, we look at three critical aspects. First, we assess whether the edited video accurately reflects the editing instructions. Second, we determine whether the edited video successfully preserves the overall structure of the original video. Finally, we consider the aesthetics of the edited video, ensuring it is free of imperfections such as jittering. Our evaluation is based on user study and automated scoring metrics. In the user study, we follow TGVE competition to ask users three key questions. The Text Alignment question: Which video better aligns with the provided caption? The Structure question: Which video better retains the structure of the input video? The Quality question: Aesthetically, which video is superior? These questions aim to evaluate the quality of video editing, focusing the video's s alignment with editing instructions, its preservation of the original structure, and its aesthetic integrity. For objective metrics, incorporate PickScore Kirstain al. (023) that computes the average image-text alignment over all video frames and CLIP Frame (Frame Consistency) Radford al. (021), which measures the average cosine similarity among CLIP image embeddings across all video frames. We prefer PickScore over the CLIP text-image score since it's tailored to more closely align with human perception of image quality, which is also noticed by Podell al. (2033.
5.2 BASELINE METHODS
We benchmark Ins against leading text-driven video editing techniques: Tune-A- Video Wu al. (222)). Vid2Vid-Zen Wang et al. (023a), Pideo-P2P[iu (203)), 1 and ControlVideoo Zhao et al. (203). Tune-A- -ideeo has been treated as a de facto baseline this domain. Vid2Vid-Zero and Video-P2P adopt the cross-attention from Prompt-to-Prompe (PTP Hertz et al. (022). while ControlVideo leverages ControlNe
& Agrawala 2023). We test all methods for 32 frames, but PTP-based ones, due to their computational demand, are limited to 8 frames. Baselines are processed in a single batch to avoid inter-batch inconsistencies and use latent inversion Mokady et al. (222)) for structure preservation, which causes double inference time. Conversely, our method retains the video's 's structure more efficiently.
We also extend the comparison to include recent tuning- free video editing methods such as Token- Flow Geyer et al. 2023) Render-A-Videc Yang et al. (2023b), and Tix2VidecCeylaa et al. 2023)

---

Published as a conference paper at ICLR 2024
These methods, by eliminating the necessity for individual video model tuning, present compara- ble benchmark to our approach. To ensure frame-to- to-ffomef consistency, these methods either adopt cross-frame attention similar to Tune-A- Vided Wu et al. (2022), seen in Ceylan et al. (203); Yang et al. 2023b), or establish pixel-level correspondences between features and a reference key frame Geyer et al. (2023). This approach is effective in maintaining quality when there minor scene changes. However, in scenarios with significant differences between the key and reference frames, these methods may experience considerable degradation video quality, This limitation more clearly illustrated in Figure 12 in the Appendix.
5.3 MODEL DETAILS
Our model is adapted from a single image editing Stable Diffusion Brooks al. (2023) and we insert
temporal attention modules after each spatial attention layers as suggested by Guo et al. (2023) 3. Our training procedure makes use of the Adam optimizer with a learning rate set
10 The
model is trained with batch size of 512 over span of 2,000 iterations. This training process takes approximately hours to complete on four NVIDIA A10G GPUs.
During sampling. experiment with varying hyperparameters for video classifier-free guidance (VCFG) within the choice of [1.2, 1.5, 1.8], text classifier-free guidance to and video resolutions of 256 and 384. A detailed visual comparison using these differing hyperparameters can be found in the supplementary material (Appendix E The hyperparameters combination that achieves the highest PickScore is selected the final sampling result. Each video processed in three distinct batches using LVSC with fixed frame count of 16 within a batch, including reference frames from preceeding batch, and resulting in total frame count of 32.
5.4 LONG VIDEO SCORE CORRECTION AND MOTION COMPENSATION
Table I: Comparison of motion-awarr MSE and CLIP frame similarity between the last frame of the preceding batch and the first frame of the sub- sequent batch TGVE dataset.
To assess performance improvement, we em- ploy CLIP Frame Similarity and Motion-Aware Mean Squared Error (MAMSE) as evaluation metrics. Unlike traditional MSE. MAMSE accounts for frame-to-frame motion by utiliz- ing optical flow warp images, thereby en- suring loss computation in corresponding re- gions. Incorporating Long Video Score Correc- tion (LVSC) and Motion Compensation (MC) has led to enhanced performance as reflected in Table Further qualitative comparison, detailing the benefits of LVSC and MC, are provided in Appendices C and
LVSC
MC
MAMSE (%)
CLIPFrame f
2.02
0.9072
X
1.44
0,9093
1.37
0.9095
5.5
USER STUDY
Table 2: The first two columns display automated metrics concerning CLIP frame consistency and PickScore. The final four columns pertain to user study conducted under the TGVE protocol, where users were asked to select their preferred video when comparing the method against the TAV.
Method
CLIPFrame PickScore
Text Alignment Structure Quality Average
TAV*
0.924
20.36
CAMP
0.899
20.71
0.689
0.486
0.599
0.591
T
0.923
20.22
0.531
0.601
0.564
0.565
Vid2Vid-Zero
0.926
20.35
0.400
0.357
0.560
0.439
Video-P2P
0.935
20.08
0.355
0.534
0.536
0.475
ControlVideo
0.930
20.06
0.328
0.557
0.560
0.482
TokenFlow
0.940
20.49
0.287
0,563
0.624
0.491
Vixevideo
0.916
20.12
0.468
0,529
0.538
0,511
Aander-A-iide
0.909
19.58
0.326
0.551
0.525
0.467
V2V (Ours)
0.911
20.76
0.690
0.717
0.689
0.699
*: Scores from TGVE Isndrbbaad
8

---

Published as a conference paper at ICLR 2024
Alggmmet
Structure
Quality
Average
1
Preference (9)
0.8
0.6
0.0
STY
BKG
OBJ
MUL
STY
BKG
OBJ
MUL
OBJ
OBJ
Text Align Structure
Text Align
Quality
Structure Quality
Text A SaliinnSttutt+ttt
Poziiiee
Preference (%)
0.8
0.4
0.0
STY
BKG
OBJ
MUL
STY
BKG
OBJ
MUL
STY
BKG
OB]
MUL
STY
BKG
MUL
Figure 4: The abbreviations on x-axis indicate user preferences across four types of video editing within TGVE: Style, Background, Object, and Multiple. Each title specifies the evaluation metrics used for the corresponding figures. A "+" symbol signifies that the user vote meets multiple criteri. Additional qualitative results are presented in Appendix F
We conducted two separate user studies. The first followed the TGVE protocol, where we used Tune-A-Video the baseline and compared the outcomes of our method with this well-known approach. However, we realized that this method of always asking users to choose a better option might not fully represent the reality, where both videos could either perform well poorly. Thus, in the second user study, we compare our method with seven publicly available baselines. Instead of asking users choose better video, asked them vote for the quality of text alignment, structural preservation, and aesthetic quality for each transferred video. As evidenced by Table and and Figure g our approach excelled in all metries except CLIP Frame similarity. We contend that CLIP Frame similarity is not an entirely apt metric because it only captures semantic similarity between individual frames, which may not be constant throughout a well-edited video due to changing scenes.
In our evaluation protocol, we additionally introduce a multi-metric assessment, capturing cases where videos satisfy multiple evaluation criteria concurrently. This composite measure addresses a key shortcoming of single metrics, which may inadequately reflect overall editing performance. For example, high structure score might indicate that the edited video perfectly preserves the original, but such preservation could come at the expense of alignment with the editing instruction.
The results presented Figure further corroborate the advantages of our approach. While baseline methods demonstrate respectable performance when text alignment is not a prioritized criterion, they contrast, our method not only
fall short when this element is incorporated into the assessment. excels in aligning the edited vidco with the textual instructions but also maintains the structural integrity of the video, resulting in a high-quality output. This dual success underlines the robustness of our method in meeting the nuanced requirements of video editing tasks.
6 CONCLUSION
This research tackles the problem of text-based video editing. We have proposed a synthetic video generation pipeline, capable of producing paired data required for video editing. To adapt to the requirements of video editing, we have employed the model inflation technique to transform a single image diffusion model into video diffusion model. As innovation, we' ve introduced the Long Video Sampling Correction to ensure the generation of consistent long videos. Our approach is not only efficient but also highly effective. The user study conducted to evaluate the performance of our model yielded very high scores, substantiating the robustness and usability of our method in real-world applications.

---

Published as a conference paper at ICLR 2024
REFERENCES
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18208-18218, 2022.
Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski, Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable im- age generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18370-18380, 2023.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miik Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint 2XNv:22/1.013222 2022.
Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text- driven layered image and video editing. In European conference on computer vision, pp. 707-723.
Springer, 2022.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis, Align your latents: High-resolution video synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22563-22575, 2023.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image
editing instructions. In CVPR, pp. 18392-18402, 2023.
Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 23206-23217, 2023.
Jiaxin Cheng, Xiao Liang. Xingjian Shi, Tong He, Tianjun Xiao, and Mu Li. Layoutdif- fuse: Adapting foundational diffusion models for layout-to-image generation. arXiv preprint arXiv:2302.00 08900, 2023a.
Yen-Chi Cheng, Hsin- Hsin-Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Yan Gui. Sd- fusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4456-4465, 2023b.
Prafulla Dhariwal and Alexander Quinn Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems, 2021.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion, pp. 12873-12883 2021.
Patrick Esser, Johnathan Chiu, Parmida Atighehchian, Jonathan Granskog, and Anastasis Ger- manidis. Structure and content-guided video synthesis with diffusion models. arXiv preprint arXiv:2302.030//1 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2022.
Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10377 2023.
Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang. Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint M0Xiv:3007.04725. 2023.
10

---

Published as a conference paper at ICLR 2024
William Harvey, Saeid Naderiparizi, Vaden Masrani, Christian Weilbach, and Frank Wood. Flexible diffusion modeling of long vidcos. Advances in Neural Information Processing Systems, 35: 27953-27965, 2022.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-promp image editing with cross attention control. arXiv preprint arXiv:2208.0166: 2022.
Jonathan Ho and Tim Salimans. arXiv:2207. 125988 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-68521 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303,, 2022a.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. arXhv:2204.0343 2022b.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. URL https://arxiv. org/abs/2204.03458, 2022c.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a-pic: An open dataset of user preferences for text-to-image generation. arXiv preprint arXiv:2305.01569, 2023.
Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Vidco-p2p: Video editing with cross-attention control. arXiv preprint arXiv:2303.04760 2023.
Kangfu Mei and Vishal Patel. Vidm: Video implicit diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligncee, volume 37, pp. 9117-9125, 2023.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Yun Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In International Conference on Learning Representations, 2021.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
Eyal Molad, Eliahu Horwitz, Dani Valevski, Alex Rav Acha, Yossi Matias, Yacl Pritch, Yaniv Leviathan, and Yedid Hoshen. Dreamix: Video diffusion models are general video editors. arXiv preprint 23Xi:23302.01222 2023.
Chong Mou, Xintao Wang, Liangbin Xie, Jian Zhang, Zhongang Qi. Ying Shan, and Xiaohu Qie T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. arXiv preprint arXh::2302.0845. 2023.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-88 8171. PMLR, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text- text-guided diffusion models, In International Conference on Machine Learning, pp- 16784-16804. PMLR, 2022.
11

---

Published as a conference paper at ICLR 2024
Federico Perazzi, Jordi Pont-Tusee Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine- Horiiee- A benchmark dataset and evaluation methodology for video object segmen- tation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 724-732, 2016.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprini arXiv:2307.01952, 2023.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations, 2022.
Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. arXiv preprint arXiv:2303.09535. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06 06125, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Newral Informa- tion Processing Systems, 35:36479-386 964994,4444... 2022b.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4713 (005:4473-447224 4726, 2022c.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2020a.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2020b.
Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi. Lizhuang Ma, and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. arXiv preprint arXiv:2303. 14184, 2023.
12

---

Published as a conference paper at ICLR 2024
MosaicML NLP Team. Introducing 3pt-30b: Raising the bar for open-source foundation models, 2023. URL www.mosale
com/ www.blog//pppt.ccccccoooooooogop 30b Accessed: 2023-06-22.
Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part Il 16. pp. 402-419. Springer, 2020.
Wen Wang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang. and Chunhua Shen. Zero- shot video editing using off-the-shelf image diffusion models. arXiv preprint arXiv:2303. 17599, 2023a.
Xiang Wang, Hangjie Yuan, Shiwei Zhang, Dayou Chen, Jiuniu Wang, Yingya Zhang, Yujun Shen, Deli Zhao, and Jingren Zhou. Videocomposer: Compositional video synthesis with motion con- trollabilityy arXiv preprint arXiv:2306.02011 020188 2023b.
Daniel Watson, William Chan, Ricardo Martin Brualla, Jonathan Ho, Andrea Tagliasacchi, and Mohammad Norouzi. Novel view symthesis with diffusion models. In The Eleventh International Conference on Learning Representations, 2022.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for texi-to-video generation. arXiv preprint arXiv:2212.11566 2022.
Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun, Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing with diffusion models. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18381- 1839 2023a.
Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided video-to-video translation. arXiv preprint arXiv:2306.079555 2023b.
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint wxXiv23302.0554: 2023.
Yuxin Zhang, Nisha Huang, Fan Tang, Haibin Huang, Chongyang Ma, Weiming Dong, and Chang- sheng Xu. Inversion-based style transfer with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10146-10156, 2023.
Min Zhao, Rongzhen Wang, Fan Bao, Chongxuan Li, and Jun Zhu. Controlvideo: Adding condi- tional control for one shot text-to-video editing, arXiv preprint arXiv:2305.17008 2023.
13


Reviews:
[{"review_id": "oimHZdHbQu", "reviewer": "Reviewer_3EH3", "summary": "The paper tackles the problem of text-based video editing. The proposed method, namely InsV2V, is an extension of Instruct Pix2Pix to the domain of videos. InsV2V follows the same paradigm of first generating synthetic data containing videos before and after the editing, as well as the correspoinding text. These data can then be used for training the video editing model.\nThe synthetic data is generated using an off-the-shelf text-to-video model. To begin with, the text prompts are obtained from existing datasets and the instructions are generated using a pretrained LLM and in-context learning. Then the source videos are generated using the text-to-video model and the target videos are generated using Prompt-to-Prompt technique. Finally, these generated video and text pairs are filtered using CLIP scores.\nAfter the data is obtained, an video-to-video model is constructed based on a pretrained image-to-image LDM, and partially finetuned on the synthetic data. \nTo allow the generation of videos longer than the training data, the video-to-video model is conditioned on a few previous frames. Additionally, a score correction term based on optical flow is added to improve temporal consistency across consecutive batches of the same video.", "questions": "* Which LLM did you use for the in-context learning?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Text-based video editing is a difficult problem, yet the paper has managed to achieve.\n* The evaluation of the method is very comprehensive -- it has included both automatic metrics as well as user studies, demonstrating state-of-the-art performance.\n* The novel components proposed in the paper, namely long video score correction (LVSC) and motion compensation (MC) has significantly improved the consistency of the generated videos, as illustrated in Table 2 as well as in the supplementary material.", "weaknesses": "* The techniques used in the paper are not completely new -- a large portion of them has followed Instruct-Pix2Pix. \n* Even with all the measures in place, the generated videos are still far from being temporally consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YrDOuQb3W7", "reviewer": "Reviewer_99Pg", "summary": "This paper proposes an efficient text-based video-editing method by adapting Instruct Pix2Pix from image to video editing, eliminating the need for additional training. To enable long term video editing, a Long Video Sampling Strategy is proposed to maintain long video consistency. Experimental comparisons with other methods demonstrate the advantages of the proposed approach.", "questions": "1. In Section 3.1, the authors mentioned that the temporal attention layers are also replaced to adapt to video editing tasks. However, in the showcased results, most videos are style transferred frame by frame. Is there any attempt to simultaneously change the style of the video and modify the motion of the video, such as transforming a person walking towards the left to walking towards the right?\n2. In Section 3.2, how is the success rate calculated? Are there any other methods to improve the success rate?\n3. Does using video diffusion model based methods have any advantages over using optical flow in image diffusion models in video editing tasks?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow\n2. The paper proposed a very intersting idea for universal one-model-all-video transfer idea for vid-to-vid transfer\n3. It proposed a novel synthetic dataset fo vid-to-vid transfer task.\n4. The experiments were well conducted, showcasing detailed numerical indicators and a user study to evaluate the video editing method.", "weaknesses": "1. The proposed method in this paper lacks significant innovation. The majority of the content is derived from Instruct Pix2Pix by adapting image editing to video editing without much improvement. \n2. The proposed sampling method to maintain long video consistency is a variation of inpainting sampling methods, which is also widely used in image/video generation tasks. The experimental section provides detailed numerical metrics for various evaluation indicators and user study. However, the compared baseline lacks strength in video editing tasks, there already are some video editing models based on image diffusion models, such as Pix2Video[1],  Render A Video [2], TokenFlow[3], which also do not require fine-tuning on a single video. The proposed method in this paper did not compare itself with those models mentioned. \n3. From the generated video results in the provided supplementary, it seems that the proposed method does not achieve superior results.\n\n [1] Pix2Video: Video Editing using Image Diffusion\n\n [2] Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation\n\n [3] TokenFlow: Consistent Diffusion Features for Consistent Video Editing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BXU4Afgzrm", "reviewer": "Reviewer_BaKN", "summary": "This paper proposes a novel approach for text-based video-to-video editing that eliminates the need for resource-heavy finetuning for each video and model. The authors introduce a synthetic paired video dataset tailored for video-to-video transfer tasks, taking inspiration from image transfer methods such as Instruct Pix2Pix. This method translates the Prompt-to-Prompt model to videos and efficiently generates paired samples, each consisting of an input video and its edited counterpart. They also propose Long Video Sampling Correction (LVSC), ensuring consistent long videos across batches. Their method outperforms existing methods like Tune-A-Video in terms of text-based video-to-video editing, paving new avenues for exploration and deployment.", "questions": "1. It would be better to quantify the difference between the generated video dataset and some reference one, e.g., computing FID between them. It may also be helpful to validate the effectiveness of the given sampling criteria.\n2. How compatible is this approach with different types of video content, various editing instructions, and text-to-video generation methods?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method eliminates the need for per-video-per-model finetuning, potentially saving significant computational resources. The creation of a synthetic paired video dataset tailored for video-to-video transfer tasks is a novel approach that could prove beneficial for training models in this domain. The introduction of LVSC addresses the challenge of maintaining consistency in long videos across batches, a notable improvement over existing methods.\n2. Sufficient and comprehensive experiments (both quantitatively and qualitatively) on the comparisons are given with prior arts and ablations of key designs. The given method gives notable quantitative improvements and its visual results faithfully follow the given instructions compared with other approaches from the supp.", "weaknesses": "1. The generated video shows visual appeals in the given content with different styles while presenting severe jitter in the newly added content.\n2. The performance of the proposed method relies on the synthetic paired video dataset. Though it gives reasonable sampling strategies on the generated data, if this dataset doesn't closely match real-world scenarios, it may limit the model's utility.\n3. The paper does not talk about potential failure cases or limitations of their approach in the main paper, which could help us better understand the proposed system.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ldq3nRv0um", "reviewer": "Reviewer_NGWA", "summary": "The paper presents a method for text-driven video-to-video editing, eliminating the need for exhaustive per-video finetuning. Building on the Instruct Pix2Pix image transfer framework, the authors adapt the concept for videos, using a synthetic paired video dataset. They also introduce the Long Video Sampling Correction for consistency across longer video batches. Impressively, this approach outperforms existing techniques like Tune-A-Video, marking some advancements in the domain and opening doors for future research and application.", "questions": "Will the dataset be released?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper crates a synthetic dataset for training instruction-based video-to-video synthesis models. This is good and could potentially benefit the community.\n\n2. The paper is well written and easy to follow.", "weaknesses": "1. The paper is good at representation but some of the information is confusing. For example, in Table 1, the author claims that all baseline methods need fine-tuning and the proposed method does not need any fine-tuning. However, the proposed method also needs extra training and the cost of the dataset creation is also not reflected.\n\n2. The paper's technical contribution seems to be incremental. The proposed long video sampling strategy seems to ve pretty similar to the sliding window operation but stated in a more formal way.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)