NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editing
Authors: Ting-Hsuan Chen
Jiewen Chan
Hau-Shiang Shiu
Shih Han Yen
Changhan Yeh
Yu-Lun Liu
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Editing
Canonical Images
Diffusion Models
Temporal Consistency
Primary area: machine_vision

Abstract: We propose a video editing framework, NaRCan, which integrates a hybrid deformation field and diffusion prior to generate high-quality natural canonical images to represent the input video. Our approach utilizes homography to model global motion and employs multi-layer perceptrons (MLPs) to capture local residual deformations, enhancing the model’s ability to handle complex video dynamics. By introducing a diffusion prior from the early stages of training, our model ensures that the generated images retain a high-quality natural appearance, making the produced canonical images suitable for various downstream tasks in video editing, a capability not achieved by current canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA) fine-tuning and introduce a noise and diffusion prior update scheduling technique that accelerates the training process by 14 times. Extensive experimental results show that our method outperforms existing approaches in various video editing tasks and produces coherent and high-quality edited video sequences. See our project page for video results: [koi953215.github.io/NaRCan_page](https://koi953215.github.io/NaRCan_page/).

NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editing
Ting-Hsuan Chen
Jiewen Chan Hau-Shiang Shiu Yu-Lun Liu
Shih-Han Yen
Chang-Han Yeh
National Yang Ming Chiao Tung University
Abstract
We propose video editing framework, NaRCan, which integrates hybrid defor- mation field and diffusion prior to generate high-quality natural canonical images to represent the input video. Our approach utilizes homography model global motion and employs multi-layer perceptrons (MLPs) to capture local residual deformations, enhancing the model's s ability to handle complex video dynamics. By introducing diffusion prior from the early stages of training, our model en- sures that the generated images retain high-quality natural appearance, making the produced canonical images suitable for various downstream tasks in video editing, capability not achieved by current canonical -ased methods. Further- more, we incorporate low-rank adaptation (LoRA) fine-tuning and introduce a noise and diffusion prior update scheduling technique that accelerates the training process by 14 times. Extensive experimental results show that our method outper- forms existing approaches in various video editing tasks and produces coherent and high-qualit, edited video sequences. Code and video results are available at koi953215, 1..221...ittubiooo.. githhbbio/aaaCaa. pagee.
1
Introduction
Video editing has always been fascinating research area. For example, style transfer transforms the original video into completely new style, enriching the viewing experience. Other tasks include dynamic segmentation and handwriting, which all demonstrate the broad application value of video editing across various fields. Currently, diffusion model technology is hecoming increasingly mature and is known for its powerful generative capabilities and frequent use in video editing. However, in video-to-video tasks, maintaining temporal consistency presents significant challenge, particularly when applying image-based diffusion models to video editing tasks. When these models, originally designed for image generation, are applied trame-by-frame to videos, they often produce temporally inconsistent results due to their crame-independent processing nature. This limitation has motivated numerous research efforts to enhance temporal coherence through various techniques, such as optical flow guidance, latent space alignment, and cross-attention mechanisms, aiming to produce high-quality video sequences while preserving temporal consistency across frames.
Yet, even with solutions addressing temporal consistency, Liffusion-based methods [16, 4, 19, 8] still encounter significant challenges in precise localized editing tasks. This is where canonical- based methods demonstrate their unique advantages. By consolidating video content into a single representative image, canonical-basse methods [3, 5, 45] enable intuitive and precise spatial control over editing operations while inherently maintaining temporal consistency. This unified representation allows direct application of baaee editing techniques while ensuring that modifications propagate coherently throughout the video sequence, making these methods particularly effective for wide range of video editing tasks.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Video
[a] Video
Denoising U-Net
Representation
Editing
Video Reprosentation
|b] Dynamic
Denoising U Net
Segmentation
Video Representation
Style
Denoising U-Net
Transfer
Unnatural Canonical
Diffusion
Natural Canonical
Input Video Sequence
Propagated Edits
Video representation with diffusion prior. Given an RGB video, we can represent the video using a canonical image. However, the canonical image and reconstruction training process focuses only on reconstruction quality and could produce unnatural canonical image. This could cause problems with downstream tasks such as prompt-based video editing. In the bottom example, if the hand is distorted in the canonical image, the image editor, such as ControlNet [75], may not recognize it and could introduce an irrelevant object instead. In this paper, we propose introducing the diffiusion prior from LoRA [18] fine-tuned diffusion model to the training pipeline and constraining the canonical image to be natural. Our method facilitates several downstream tasks, such as (a) video editing, (b) dynamic segmentation, and (c) video style transfer.
Figure 1:
While previous canonical -asse approaches have established the fundamental utility of canonical im- ages in video editing, our observations suggest that enhancing the naturalness of these representations can significantly improve their effectiveness in various editing scenarios. We observe that existing methods primarily focus on reconstruction quality without explicit constraints to ensure natural canonical image generation. To address this opportunity for enhancement, we propose NaRCan, novel hybrid deformation field network architecture that incorporates diffusion priors (Figure 1) into the training pipeline (Figure 2). This integration enables our method to generate high-fidelity natural canonical images across diverse scenarios while maintaining temporal consistency. Through comprehensive experimental evaluation, we demonstrate that NaRCan achieves superior performance compared to state-of- of- the-art video editing methods in both qualitative and quantitative metrics.
Our main contributions are three-fold:
We have designed novel deformation field structure to represent object variations through- out an entire scene. Compared to other canonical-bare models, our model demonstrates superior expressive capability and faster convergence speed.
We have effectively integrated the diffusion prior into our pipeline, enabling our method to generate high-quality natural canonical images in any scenario. Additionally, we designed dynamic scheduling method that significantly accelerates the entire training process.
We thoroughly evaluate our method to show the state-of-the-an video editing performance.
2
Related Work
Implicit neural representation. Implicit neural representation [43] using coordinate-based MLP is an outstanding way represent video, capable of obtaining canonical image represent the entire video [45]. Recent methods employ hash grid encoding [42] or positional encoding [40] combined with MLP. The approach in [5] is more effective at handling spatial information, but the resulting canonical image exhibits severe distortion and warping with in-the-wild videos [11]. Therefore, we propose a hybrid deformation field method composed of homography [9] and residual deformation MI.P. This model design fits the deformation information in videos better than existing methods.
Consistent video editing. There are generally three approaches to video editing: (1) propagation- based, (2) layered representation-batt and (3) canonical- The first approach, propagation-

---

based, focuses on propagating information across frames [35, 20, 22, 23, 57, 63, 67]. This method can easily produce inaccurate results due to occlusions and error propagation. The second approach. layered representation-batt based,, separates video into foreground and background, obtaining canonical images for both [36, 25, 73, 34, 37, 38]. We can edit the entire video by editing these canonical images and then synthesizing the video afterward. However, this method heavily relies on masks. If the mask-RCND captures incorrect targets in the preprocessing stage, the fitted foreground can be incorrect, especially in scenes with large camera movement.
The third approach, canonical-basee based, typically uses MLP to obtain the deformation information of each pixel to form canonical image [45]. Transferring the video to canonical space maintains temporal consistency while editing and supports various downstream tasks, such as super-resolution and segmentation. CoDeF [45] uses this approach. but canonical images can deform severely in videos with significant camera or object movement. CoDeF suggests that using a group model could resolve these issues. However, using group CoDeF requires masks for training data obtained from SAM-track [12]. Incorrect masks for training data can result in an unnatural canonical image of the video. Even with correct masks for the foreground object, the canonical image might still be corrupted, rendering these images ineffective for video editing.
Video processing via generative models. Some works utilize GAN inversion [65, 78, 60, 74, 48, 26, 69] to edit images videos. Today, numerous generative models exist for editing images. Some methods, such as GLIDE [44], DALL-E [55, 54], stable diffusion [56], and Imagen [58], are trained on millions of images, resulting in incredible generative abilities. Other methods like SDEdit [39], ControlNet [75], and LDM [2] use conditions to achieve better editing results. Instruction-Nen baasd video editing methods like InstructPix2Pix [3] and another work [10] often yield sub-optimal results for different editing operations. Techniques like LoRA [18] can assist in fine-tuning to find better weights for editing. Additionally, many zero-shot diffusion-
methods [72, 16, 76, 4, 8, 19, 27] do not require model training but still need constraints to maintain temporal consistency. To address temporal consistency concerns, most methods like Tune-A- A-Veeo [70], Text2Video-222
[27], FateZero [52], and Vid2 [66] incorporate cross attennnn mechanisms. Some works propose training for video editing, such as Imagen Video [17] and Make-A- A- A-Video [59], but these require large datasets and significant computational resources. Unlike these methods, MeDM [13] uses a flow-coding algorithm to solve this problem. iunonical-based design does not require another mechanism to maintain temporal consistency, however. Once the canonical image is edited, the changes can be
propagated to every frame using a deformation field.
Lifting the naturalness of canonical image by diffusion models. The diffusion prior has been applied in various domains. Reconfusion [71] is few-shot novel view synthesis work that introduces a diffusion model before optimizing sampled novel views. Dreamfusion [50], a text-to-3D work, introduces score distillation sampling (SDS) loss, referencing 2D diffusion model to optimize 3D outputs. This approach inspires us to utilize the diffusion model to improve performance. Other methods like [6, 7, 21, 41, 33, 61, 64, 68, 79] also employ diffusion prior for text-to-3D tasks.
Several diffusion models focus on text-to-image generation [53, 56, 58, 62]. Additionally, some diffusion models can refine corrupted images to make them appear more natural. We propose adding a diffusion model to our pipeline (Figure 2) to enhance the naturalness of our canonical images. Our goal is improve the restoration of canonical images using the diffusion model. While create canonical images using a hybrid deformation field, this method might not always deliver optimal performance, especially in scenarios with dramatic motion changes severe non non-rigid transformations. The capabilities of the hybrid deformation field are still limited in such cases. Therefore, aim to introduce diffusion model to make our canonical images more natural, enhancing the effectiveness of video editing. This research has significant practical implications as it improve the quality and realism of video editing, benefiting various industries such as film production, advertising, and virtual reality.
3 Method
In this section, we first introduce our hybrid deformation modeling by combining homography and deformation MLP in Section 3.1. Subsequently, we elaborate on how we integrate the diffusion prior from LoRA fine-tuned latent diffusion model to ensure the naturalness of our canonical image representation in Section 3.2. Finally, we provide an additional way to improve the quality of our
3

---

"VV"
Text Encoderi
Denolsing U-Neet
(M)
TVV
Text Enceder
Dencising Un-iet
LoRA
Residual MUP
Canonical MLP
Styl Tranere
Scheduling
Natural
MSE Loss
Input Sequence
Homography
(4) LoRA Finetuning
(b) Video Representation
(4 Difission Prior
(4) Applications
Our proposed framework. Given an input video sequence, our method aims represent the video with natural canonical image, which is crucial representation for versatile downstream applications. (a) First, we fine-tune the LoRA weights of a pre-trained latent diffusion model on the input frames. (b) Second, we represent the video using a canonical MLP and deformation field, which consists homography estimation and residual deformation MLP for non-rigid residual deformations. By relying entirely on the reconstruction loss, the canonical MLP often fails to represent a natural canonical image, causing problems for downstream applications. E.g., image-to- image translation methods such as ControlNet [75] may not be able to recognize that there is a train the canonical image. (c) Therefore, we leverage the fine-tuned latent diffusion model to regularize and correct the unnatural canonical image into natural one. Specifically, we sophistically design noise scheduling corresponding to the frame reconstruction process. (d) The natural and artifacts-free canonical image then be facilitated to various downstream tasks such video style transfer, dynamic segmentation, and editing, such as adding handwritten characters of "NaRCan".
Figure 2:
video representation by separating multiple canonical images and describing the necessary changes for downstream tasks in Section 3.3. Figure shows our proposed framework.
3.1 Hybrid Deformation Field for Deformation Modeling
Traditional methods often rely on direct predictions of Au and Av, which means the displacement of pixel points time t, by using an MLP : and query the RGB color by another canonical image MLP ((.-)):
Au, Av = g(u, v, (), [R, G, B s(u + Au, + Av),
(1)
supplemented with a TVFlow regularization term to prevent overfitting by constraining the magnitude of spatial deformations. While this regularization effectively prevents extreme deformations, it merely imposes magnitude constraints without providing meaningful guidance for modeling complex spatial transformations. To address this limitation, we propose hybrid deformation field architecture composed of trainable homography matrix (u, v, t) and residual deformation MLP:
u', o' Hu,, v, t) + ((u, t); RR. G, B = flu', v').
Unlike the simple constraint mansee approach of TVFlow, our method leverages homography to provide global displacement information as structural guidance for the subsequent residual deformation MLP. This hierarchical design enables the residual deformation MLP to learn and express the deformation field more accurately and effectively by building upon the initial geometric transformation provided by the homography matrix.
3.2 Diffusion Prior
The canonical image encompasses all information within the entire video and can be reconstructed for each original video frame using the deformation field outlined in Section 1. Thus, editing solely the canonical image yields temporally coherent edited video. However, editing tasks such as drawing or writing on objects or editing based on ControlNet [75] require natural image as input to produce meaningful edited images. Existing canonical base methods do not incorporate mechanisms to ensure that the generated image is natural; instead, they rely solely on the model's ability to learn a natural image. However, when encountering scenarios with camera movement significant changes in video content, these existing techniques cannot adapt to such drastic variations. The model may generate canonical image that is nearly impossible to edit (rendering it devoid of any subsequent value).

---

Ganonical
1,000 Step S. 3,000
3,000 Step 5,000
Step 5,000
Early
(Updating DM prior larget every 10 steps)
(Updating DM prior Lunget every 100 steps)
[Uodeting DM prior larget every 2000 steps) Late
Adding
Adding
Adding slight
moderate noise
Diffusion Model
Target
Figure 3: Noise and diffusion prior update scheduling. Initially, our model fits object outlines before the fields converge and without the diffusion prior, resulting in unnatural elements in the canonical image due to complex non-rigid objects. Upon introducing the diffusion prior with increased noise and update frequency, the model learns to generate natural, high-quality images, leading to convergence. Thus, the strength of noise and the update frequency will also decrease. Moreover, it's worth mentioning that update scheduling cuts training time from 4.8 hours 20 minutes.
To address these challenges, we introduce diffusion priors, which successfully mitigate this issue. Our method can generate high-quality canonical images through diffusion priors, providing valuable inputs for various video editing tasks.
LoRA fine-tuning. To enhance the current diffusion model's ability to represent all video content better, we introduced special token specific this scene. We then fine-tuned the LoRA weight of the pre-trained diffusion model. This ensures that the diffusion model generates high-quality natural canonical images tailored to the testing sequence rather than randomly generating natural images that do not belong to the scene.
Noise and difflusion prior update scheduling. While the hybrid deformation field technique mentioned in Section 3.1 already produces better canonical images than other existing methods, it still needs to improve. As Figure 9b) depicts, canonical images generated solely relying on homography and residual deformation MLP still exhibit various degrees of distortion and unnatural characteristics. Therefore, integrating diffusion priors becomes imperative. We extract the canonical region currently observed by the model and calculate diffusion loss with the target image generated by the diffusion model to ensure the generation of natural canonical images. However, generating target image at each step would significantly prolong the training phase. Hence, we propose a hierarchical update scheduling to accelerate the process, which is shown Figure 3. In the initial stages of training, when the deformation field has not yet converged, more substantial noise is introduced to allow the diffusion model to dominate the scene of the canonical image. Simultaneously, the frequency of generating target images needs to be denser. As training progresses and the deformation field becomes more stable, the noise intensity and frequency of generating target images decrease accordingly. This hierarchical scheduling approach ensures that the final canonical image approaches the quality of per-step updates while speeding up the training process by 14 times (4.8 hours to 20 minutes.) We opt for diffusion loss over SDS because using SDS, mentioned in the Reconfusion [71] paper, is more prone to generate artifacts.
3.3 Separated NaRCan
When encountering overly complex scenes, relying solely on single natural canonical image representing the entire scenario is impractical and unrealistic. Hence, we need to segment the original video into multiple segments (Si.... S and train dedicated residual deformation MLPs [R,..., R& for each segment to obtain k natural canonical images [CI, C& J. It is worth noting that S, and S have an overlap, referred to as the overlap window. Frames within this region are obtained using linear interpolation shown in Figure 4. This method ensures that excellent temporal consistency maintained when switching from canonical image C, to C++1. Table demonstrates that our temporal consistency surpasses all existing methods even after segmentation. Additionally,
5

---

75%
50%
Canonical
Canonical image!
Frame sequence
Canonical image
Figure 4: Linear interpolation. After using the grid trick [18] to obtain the highly consistent canonical images C. and CE+1, we interpolate all frames within the overlap window. As time progresses, the weight for reconstructing each frame gradually shifts from referencing Cs solely referencing C++1- We achieve editing results with remarkable temporal consistency through this linear interpolation approach. Please refer our supplementary material for more video results.
we adopt different processing approaches for various downstream tasks to ensure that Separated NaRCan can adeptly adapt to these tasks.
Style transfer. With multiple canonical images obtained, utilize the grid trick [14, 24] to ensure sufficient consistency in style and content across the l canonical images. Specifically, we concat the canonical images into larger image (with 2x2 canonical images) and use ControlNet [75] to perform eext-guided image editing.
Video editing, When addressing video editing tasks such handwriting, we leverage the pre- trained optical flow models compute the flow between the k canonical images and use this flow to warp the editing content from C₁ to the C₁ image.
4
Experiments
4.1 Experimental Setup
We conduct experiments underscore the robustness and versatility of our proposed method. Our representation robust with a variety of deformations, encompassing rigid and non-rigid objects, as well as complex scenarios such smog and waves. We commence the introduction of the diffusion model at the 1000th iteration. From iteration 1000 to iteration 3000, the noise intensity is set at 0.4, and the target image generation frequency is every 10 iterations. Subsequently, spanning from iteration 3001 to iteration 5000, the noise intensity is adjusted to 0.3, with the target image generation frequency occurring every 100 iterations. Beyond the 5000th iteration mark. the noise intensity decreases to 0.2, and the target image is generated every 2000 iterations. The total iteration 12000 iterations, and Figure 3 is shown visualize the process of noise scheduling. On single NVIDIA RTX4090 GPU, the average training duration approximately 20 minutes when utilizing 100 video frames. When evaluating the temporal consistency in Table 1, we compare our separated NaRCan with other compared methods by setting k 3, i.e., we represent the sequence using three canonical images. By adjusting the training parameters accordingly, the optimization duration can be varied from 20 minutes to an hour.
4.2 Evaluation
Video editing, We run our method CoDeF [45], Hashing-nvd [5], CCEdit [15], and MeDM [13] on the DAVIS [49] and BalanceCC Benchmark that CCEdit proposed for evaluating the results of video editing. In Figure 5, we show the comparison of visual results. The BalanceCC Benchmark provides unified text prompt for each scene, and we ensure that all video clips from this benchmark are restricted to maximum of 100 frames. We utilize ControlNet [75] to edit the video using the unified text prompt provided by the BalanceCC Benchmark for fair comparison. We executed McDM and CCEdit within their provided environment settings and pre-trained models. Moreover,

---

Table 1: Quantitative results on the BalanceCC [15] dataset. There 100 videos in BalanceCC.
To ensure a representative distribution similar to BalanceCC, we randomly select 50 videos from BalanceCC and calculate warping and interpolation errors, which the metrics for temporal consistency. Our method outperforms these baseline methods in terms of temporal consistency.
Method
Venue
Short-term Ewarp +
Long-term Ewarp 4
Interpolation error
Hashing-nvd [5]
ICCV 2023
0.0070
0.0495
9.204
CoDeF [45]
CVPR 2024
0.0085
0.0785
8. 8.721
MeDM [13]
AAAI 2024
0.0072
0.0583
9.941
Ours
0.0065
0.0484
8.365
Camel
Input Video
Ours
CoDeF
Hashing-nvd
MeDM
CCEdit
Text Prompt: comel walking in an enclosure with a wooden fence and greenery in the
background, Minecraft world style.
Hot air balloon
Input Videa
Ours
CoDeF
Hashing-nvv
MeDM
CCEdit
Prompt: balloons adrift over an ancient desert, Chibi Animation style.
Train
Input Video
Ours
CoDeF
Hashing
MeDM
CCEdit
Text Prompt: model train moving along track miniature figures trees alongside, depicted claymation style.
Figure 5: Qualitative comparisons on text-guided video-to-video translation. Our method achieves prompt alignment, synthesis quality, and temporal consistency best. Zoom for the best view, and please refer to the supplementary materials for video comparisons. (a) In the camel scene, Medm [13] fails generate clear-textured images to ensure temporal consistency, while CCEdit [15] fails to correctly identify the second camel in the background. of person in the bottom right corner, Hashing-nve [5] exhibit noticeable contours due to masking, and both MeDM and CCEdit suffer from temporal inconsistency issues. For instance, in MeDM, the person transitions from wearing black clothes blue clothes. (c) MeDM and CCEdit still exhibit temporal inconsistency issues, such as significant color, texture, and structure changes. Other methods almost entirely lose the original train information or appear as unnatural artifacts.
(b) CoDeF [45] misses capturing the presence
Hashing-nvd nvd is video decomposition work that outputs two images representing foreground and background. To maintain the consistent style of these two images for video editing, We utilize the grid trick proposed by RAVE [24] tackle this problem. Finally, there also video-editing results on DAVIS [49], and the corresponding text prompts originated from the BalanceCC Benchmark. For the evaluation of temporal consistency, shown in Table 1, utilize short-term [29, 77] and long-term [30] warping errors, along with interpolation error [32], as our primary metrics.
Metries for evaluation. Since our main focus is text-guided video editing, we conduct user studies on edited vidcos compared with other methods, like CoDeF [45], McDM [13], and Hashing-mn -vd [5].

---

g
MeDM
Hashing-Iivd
80
20
Temporal Consistency
Text Alignment
Overall Quality
Figure 6: User Study. Our method achieves the highest user preference ratios across all three aspects,
compared with McDM [13], CoDeF [45], Hashing-my nvd [5].
Ours
CoDeF
Hashing nvd
Camel
Train
Cow
Butterfly
Figure 7: Qualitative comparisons on the canonical image. Our method generates more natural canonical images through a fine-tuned diffusion prior compared with CoDeF [45], Hashing-nvd [5]. The capability of the canonical image represent input frames plays crucial role in downstream applications. (Hashing-nve consists of two canonical images. Here, we have selected the canonical image representing the foreground.)
In the user study, 36 participants the text prompt used to edit the videos. There are three critical questions for users to answer. (I)
shown two edited videos, the original video each page, and
Which video has better temporal consistency? (2) Which video aligns better with the text prompt? (3)
Which video has better overall quality? Figure 6 summarizes the user study results, demonstrating that our method outperforms in all three dimensions.
Comparison of canonical images. In Figure 7, we run our method, CoDeF [45], and Hashing- nvd [5] on DAVIS [49] and BalanceCC [15]. Note that these are the canonical images of the input video, and we show the foreground atlas for Hashing-nväs nvd. There will be two output atlases for the background and foreground of Hashing-nvd. For comparison, we only show the foreground atlas for Hashing-nvd. As result, shown in Figure 7, the output canonical images of our method are more natural than others, even
scenes with the dramatic motion objects, such as scenes named "train" and "butterfly." We can clearly see that our canonical images preserve the original object information well in the above two scenarios.
Downstream video processing. To evaluate the performance of our method's handwritten video editing, we compare with CoDeF [45] and Hashing-nvd [5], which produce canonical images of the video for users write the characters the canonical image to accomplish video editing. We write "NaRCan" to test the performance these three methods on two scenes, "gold-fish fid-fiish" and "train", in the BalanceCC Benchmark [15] in Figure 8(a). We extract the same frame of videos for comparison.
For dynamic video segmentation, we segment the mask using the Segment Anything Model (SAM) [28] based on the learned canonical image of each method and propagate to the sequence.

---

Gold
Buiterlyy
Train
Ours
CoDeF
Hashing-nvd
Ours
CoDeF
Hashing nvd
(a) Adding handwritten characters
(b) Dynamic video segmentation.
Figure Qualitative comparisons on (a) adding handwritten characters and (b) dy dynamic video segmentation. Our method represents a natural image via diffusion prior, thus can achieve temporally consistent video editing and able precisely edit desired areas.
wola Rosidual MLP w/o Homography
w/o Diffusion prior
w/ Diffusion prior
Ours
Camell
Cows
cannddabbtuu
(8) Deformation modeling
(b) Diffusion prior
Figure 9: Ablation studies. (a) Deformation modeling: (Top) We show that canonical images without homography modeling fail to generate a faithful image as the capacity of residual deformation MIP could dominate the training process and still achieve near-perfect frame reconstruction. (Mid) On the contrary, without residual deformation MI.,, our method cannot model local non-rigid transformation, resulting in blurry foreground objects. (Bottom) Combining homography and residual deformation MLP has the best of both worlds and achieves the best canonical image representation. (b) Diffusion prior: (Top) Without diffusion prior to regularizing the canonical image, the training process relies only on the frame reconstruction and could sacrifice the faithfulness of the canonical image. (Bottom) Our fine-tuned diffusion prior effectively corrects the canonical image faithfully represent the input frames and results in natural canonical images.
The target of the mask in these two scenes are the clownfish named "coral-reef" and the flying butterfly in the scene called "butterfly." We use white to mark the mask for better visibility in Figure 8(b).
4.3 Ablation Study
Homography Residual Deformation MLP. In this section, we conducted ablation experiments focusing on both homography and residual deformation MLP. Figure 9()) clearly illustrates that using only MLP to fit the deformation field [31, 46, 47, 51] results in unsuitable canonical images for downstream tasks. Without the global information homography provides, the model encounters difficulties in converging the diffusion loss and instead focuses solely on optimizing the reconstruction loss. Conversely, if we rely solely on homography to express the deformation field, homography's expressive power is limited capturing detailed variations in non-rigid objects. As a result, only approximate and blureed outcomes can be obtained.
Diffusion prior. Relying on homography and the residual deformation MLP achieves relatively better canonical images than previous methods. However, the lack of assistance from the diffusion prior still prevents the stable generation of high-quality natural canonical images. Figure 9(b)
demonstrates the significance of supervising canonical images with the diffusion prior.

---

N separations
Training time(s)
PSNR f
SSIM f
short-term Ewarp
long-term Ewarp 4
771.50
23.814
0.6369
0.0016
0.0304
2
1530.79
24.423
0.6762
0.0019
0.0310
3
2275.20
24.852
0.7017
0.0021
0,0312
4
3015.32
25.185
0.7191
0.0022
0.0326
5
3761.40
25.398
0.7301
0.0022
0.0334
Figure 10: Trade-off between reconstruction quality and temporal consistency with varying separations. The visual results demonstrate that increasing the number of separations (from 1 to 5) improves the reconstruction quality of video frames. However, the table reveals a trade-off: as the number of separations increases, temporal consistency decreases. Our empirical findings suggest that using 3 separations typically achieves balance between reconstruction quality and temporal consistency for most scenarios.
breakdance
judo
breakdance-flare
Figure 11: Failure cases.
Impact of Separation Numbers. To investigate the optimal configuration of our Separated NaRCan, we conduct ablation studies by varying the number of canonical images from one five. Our analysis reveals a clear trade-off' between reconstruction quality and temporal consistency, as illustrated in Figure 10. While inereasing the number of separations improves frame reconstruction fidelity by allowing more detailed scene representation, it introduces additional transition regions that may compromise temporal coherence. Through extensive experimentation, we find that using three canonical images strikes an optimal balance between representational capacity and temporal stability most video scenarios.
5 Conclusion
In this paper, introduce NaRCan, video editing framework, integrating diffuusion priors and LoRA [18] fine-tuning to produce high-quality natural canonical images. This method tackles the challenges of maintaining the natural appearance of the canonical image and reduces training times with new boise-scheduling techniques. The results show NaRCan's advantage in managing complex video dynamics and its potential for wide in various multimedia applications.
Limitations. Our method relies on LoRA [18] fine-tuning to enhance the diffusion model's ability to represent the current scene. However, LoRA fine-tuning is time-consuming (about 30 minutes). Additionally, our training pipeline includes diffusion loss, which increases of extreme changes in video scenes, diffusion loss sometimes fails to guide the model in generating high-quality natural images (Figure 11). These limitations point out the challenge balancing model adaptability with computational efficiency and effectiveness in varied conditions.
training time. cases
10

---

Acknowledgments and Disclosure of Funding
This research was funded by the National Science and Technology Council, Taiwan, under Grants and 113-2628-E-A49-022 The authors are grateful to Google, NVIDIA, and MediaTek Inc. for generous donations. Yu-Lun Liu acknowledges the Yushan Young Fellow Program by the MOE in Taiwan.
NSTC 112-222-E--44-0044--
References
[1] Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In ECCV, 2022.
[2] Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhor, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In CVPR, 2023.
[3] Tim Brooks, Aleksander Holynski, and Alexci A Efros. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
[4] Duygu Ceylan, Chun-Hao P Huang, and Niloy J Mitra. Pix2video: Video editing using image diffusion. In ICCV, 2023.
[5] Cheng-Hung Chan, Cheng- Yang Yuan, Cheng Sun, and Swwann-Tzong Chen. Hashing neural video decomposition with multiplicative residuals in space-time. In ICCV, 2023.
Hsin-Ying [6] Dave Zhenyu Chen, Haoxuan Li, Hsin- Ying Lee, Sergey Tulyakov, and Matthias NieBner. Scenetex: High-qualig texture synthesis for indoor scenes via diffusion priors. arXiv preprint 1rXiv:231..12261, 2023.
[7] Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In ICCV, 2023.
[8] Weifeng Chen, Jie Wu, Pan Xie, Hefeng Wu, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. In ICLR, 2024.
[9] Yue Chen, Xingyu Chen, Xuan Wang. Qi Zhang, Yu Guo, Ying Shan, and Fei Wang. Local-to- global registration for bundle-adjusting neural radiance fields. In CVPR, 2023.
[10] Jiaxin Cheng, Tianjun Xiao, and Tong He. Consistent video-to-video transfer using synthetic dataset. In ICLR, 2024.
[11] Ka Leong Cheng, Qiuyu Wang. Zifan Shi, Kecheng Zheng, Yinghao Xu, Hao Ouyang, Qifeng Chen, and Yujun Shen. Learning naturally aggregated appearance for efficient 3d editing. arXiv preprint arXiv:2312. 2023.
[12] Yangming Cheng, Liulei Li, Yuanyou Xu, Xiaodi Li, Zongxin Yang, Wenguan Wang, and Yi Yang. Segment and track anything. arXiv preprint arXiv:2305.06558. 2023.
[13] Ernie Chu, Tzuhsuan Huang, Shuo- Yen Lin, and Jun Jun-Cheng Chen. Medm: Mediating image diffusion models for video-to-video translation with temporal correspondence guidance. In AAAI, 2024.
[14] Jan-Niklas Dihlmann, Andreas Engelhardt, and Hendrik Lensch. Signerf: Scene integrated generation for neural radiance fields. arXiv preprint arX1v:2401.01647, 2024.
[15] Ruoyu Feng, Wenming Weng, Yanhui Wang, Yuhui Yuan, Jianmin Bao, Chong Luo. Zhibo Chen, and Baining Guo. Ccedit: Creative and controllable video editing via diffusion models. In CVPR, 2024.
[16] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. In ICLR, 2024.

---

[17] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao. Alexey Gritsenko, Diederik P Kingma, Ben Poole. Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.03300 2022.
[18] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. 2021.
[19] Zhihao Hu and Dong Xu. Videocontrolnet: A motion-guided video-to-video translation frame- work by using diffusion model with controlnet. arXiv preprint arXiv:2307. 14073, 2023.
[20] Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as contrastive random walk. In NeurIPS, 2020.
[21] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text-guided object generation with dream fields. In CVPR, 2022.
[22] Varun Jampani, Raghudeep Gadde, and Peter Gehler. Video propagation networks, In CVPR, 2017.
[23] Ondfej Jamriška, Šárka Sochorová, Ondřej Texler, Michal Lukáč, Jakub Fišer, Jingwan Lu, Eli Shechtman, and Daniel Sýkora. Stylizing video by example. ACM TOG, 2019.
[24] Ozgur Kara, Bariscan Kurtkaya, Hidir Yesiltepe, James M Rehg, and Pinar Yanardag. Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models. In CVPR, 2024.
[25] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. ACM TOG, 2021.
[26] Kai Katsumata, Due Minh Vo, Bei Liu, and Hideki Nakayama. Revisiting latent space of gan inversion for robust real image editing. In WACV, 2024.
[27] Levon Khachatryan, Andranik Movsisyan, Vahram Tadevosyan, Roberto Henschel. Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to-image diffusion models are zero-shot video generators. In ICCV, 2023.
[28] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yer Lo, al. Segment anything. In ICCV, 2023.
[29] Wei-Sheng Lai, Jia-Bin Huang, Oliver Wang, Eli Shechtman, Ersin Yumer, and Ming-Hsus Yang. Learning blind video temporal consistency. In ECCV, 2018.
[30] Chenyang Lei, Yazhou Xing, and Qifeng Chen. Blind video temporal consistency via deep video prior. In NeurlPS, 2020.
[31] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, al. Neural 3d video synthesis from multi-view video. In CVPR, 2022.
[32] Xirui Li, Chao Ma. Xiaokang Yang, and Ming- Hsuan Yang. Vidtome: Video token merging for zero-shot video editing. In CVPR, 2024.
[33] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023.
[34] Sharon Lin, Matthew Fisher, Angela Dai, and Pat Hanrahan. Layerbuilder: Layer decomposition for interactive image and video color editing. arXiv preprint 1 2017.
[35] Yu-Lun Liu, Wei-Sheng Lai, Ming- Hsuan Yang, Yung- Chuang, and Jia- Ma-BBn Huang. Hybrid neural fusion for full-frame video stabilization. In ICCV, 2021.
12

---

[36] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In CVPR, 2023.
[37] Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, David Salesin, William T Freeman, and Michael Rubinstein. Layered neural rendering for retiming people in video. SIGGRAPH Asia, 2020.
[38] Erika Lu, Forrester Cole, Tali Dekel, Andrew Zisserman, William T Freeman, and Michael Rubinstein. Omnimatte: Associating objects and their effects in video. In CVPR, 2021.
[39] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
[40] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[41] Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, and Tiberiu Popa. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia, 2022.
[42] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM TOG, 2022.
[43] Seonghyeon Nam, Mareus A Brubaker, and Michael Brown. Neural image representations for multi-image fusion and layer separation. In ECCV, 2022.
[44] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In ICML, 2022.
[45] Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing. In CVPR, 2024.
[46] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualle Nerfies: Deformable neural radiance fields. In ICCV, 2021.
[47] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypemerrf: A nigher-dimensiona representation for topologically varying neural radiance fields. ACM TOG, 2021.
[48] Gaurav Parmar, Yijun Li, Jingwan Lu, Richard Zhang, Jun- Yan Zhu, and Krishna Kumar Singh. Spatially-adattive multilayer selection for gan inversion and editing. In CVPR, 2022.
[49] Jordi Pont-Tuset, Federico Perazzi, Sergi Caclles, Pablo Arbeláez, Alex Sorkine-Horunng, Hornung. and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017.
[50] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
[51] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021.
[52] Chenyang Qi, Xiaodong Cun. Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fusing attentions for zero-shot text-based video editing. In ICCV, 2023.
[53] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
13

---

[54] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Iya Sutskever. Zero-shot text-to-image generation. ICML, 2021.
[55] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 2022.
[56] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[57] Manuel Ruder, Alexey Dosovitskiy, and Thomas Brox. Artistic style transfer for videos. In GCPR, 2016.
[58] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
[59] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, al. Make-a-video: Text-to-video generation without text-video data. In ICLR, 2023.
[60] Haorui Song, Yong Du, Tianyi Xiang, Junyu Dong, Jing Qin, and Shengfeng He. Editing out-of-domain gan inversion via differential activations. In ECCV, 2022.
[61] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma. and Dong Chen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In ICCV, 2023.
[62] Luming Tang, Nataniel Ruiz, Qinghao Chu, Yuanzhen Li, Aleksander Holynski, David E Jacobs, Bharath Hariharan, Yael Pritch, Neal Wadhwa, Kfir Aberman, et al. Realfill: Reference-driven generation for authentic image completion. In CVPR, 2024.
[63] Ondřej Texler, David Futschik, Michal Kučera, Ondrej Jamriška, Šárka Sochorová, Menclei Chai, Sergey Tulyakov, and Daniel Sykora. Interactive video stylization using few-shot patch- based training. ACM TOG, 2020.
[64] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023.
[65] Tengfei Wang. Yong Zhang, Yanbo Fan, Jue Wang, and Qifeng Chen. High-fidelity gan inversion for image attribute editing. In CVPR, 2022.
[66] Wen Wang, Yan Jiang, Kangyang Xie, Zide Liu, Hao Chen, Yue Cao, Xinlong Wang, and Chunhua Shen. Zero-shot video editing using off-the-shelf image diffusion models. arXiv preprint 2033 2023.
[67] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency time. In CVPR, 2019.
[68] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS, 2023.
[69] Zhixiang Wang, Yu-Lun Liu, Jia-Bin Huang, Shin' Satoh, Sizhuo Ma, Gurunandan Krishnan, and Jian Wang. Disco: Portrait distortion correction with perspective-aware 3d gans. IJCV, 2024.
[70] Jay Zhangjie Wu, Yixiao Ge, Xintao Wang. Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. ICCV, 2023.
[71] Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P Srinivasan, Dor Verbin, Jonathan T Barron, Ben Poole, al. Reconfusion: 3d reconstruction with diffusion priors. In CVPR, 2024.
[72] Shuai Yang, Yifan Zhou, Ziwei Liu, and Chen Change Loy. Rerender a video: Zero-shot text-guided -idoo translation. In SIGGRAPH Asia, 2023.
14

---

[73] Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites for unsupervised video decomposition. In CVPR, 2022.
[74] Ahmet Burak Yildirim, Hamza Pehlivan, Bahri Batuhan Bilecen, and Aysegul Dundar, Diverse inpainting and editing with gan inversion. In /CCV, 2023.
[75] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
[76] Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang. Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free controllable text-to-video generation. ICLR, 2024.
[77] Shangchen Zhou, Peiqing Yang, Jianyi Wang, Yihang Luo, and Chen Change Loy. Upscale-a- video: censstell diffusion model for real-world video super-resolution. In CVPR, 2024.
[78] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In ECCV, 2020.
[79] Joseph Zhu and Peiye Zhuang. Hifa: High-fidelity text-to-3d with advanced diffusion guidance. arXiv preprint arXiv:2305. 18766, 2023.
15

---

A Appendix
A.1 Single NaRCan compared with Separated NaRCan
Dissecting canonical-based methods. this section, we will delve deeper into two different canonical-based methods: CoDeF and Hashing-nvd. As shown clearly in Figure 12, CoDeF generates poor-quality canonical images due to the lack of supervision and constraints from the diffusion prior. In contrast, method can consistently produce high-quality canonical images regardless of the video length, effectively preparing for subsequent downstream editing tasks. The drawbacks of Hashing-ny nvd are also apparent. This method relies heavily the Mask-RCNN technique, often resulting in inaccurate or incorrect foreground and background segmentation. Consequently, the final canonical images generated difficult to edit inapplicable techniques such ControlNet.
Parameter settings of Separated NaRCan. Subsequently, as shown in Figure 13, From our experiments, we found that when using Separated NaRCan, it is crucial to limit the number of segmentations in the video. This is because the editing information on the edited canonical image is propagated through warping using flows. If the flow is inaccurate contains errors, excessive warping can lead severe cumulative errors, significantly damaging the edited content.
82 frames
Kite-surf
27 frames
Coral-reef
Ours
CoDeF
Ours
Hashing-nvd foreground
Hashing-nvd background
Figure 12: Canonical analysis. (a) Compared to existing canonical-basse based methods, approach robustly generates high-quality natural canonical images regardless of the video length. (b) Our method accurately preserves the correct foreground and background information from the original scenes, avoiding severe distortions or warping and preventing generating content inconsistent with the scene. For example, in "kite-surfing", Hashing-nvd erroneously generates additional person.
A.2 Video Comparisons
We provide an interactive HTML interface to browse video results for comparisons. Specifically, provide video comparisons on three different tasks: (1) ControlNet style transfer, (2) dynamic segmentation, and (3) adding handwritten characters. We compare our proposed method, NaRCan, with of-te-of-the-ar methods: Hashing-nvd [5], CoDeF [45], and MeDM [13]. We also visualize the optimized canonical images if available for reference.
A.3
User Study
To conduct our user study, we employ GitHub Pages conjunction with Google Form to facilitate user evaluations of video quality. Each evaluation session comprises 15 scenes, each of which contains questions. For these evaluations, we randomly select 15 scenes from a pool 100 scenes in BalanceCC Benchmark [15]. Each evaluation page (Figure 14) presents a video edited using our method alongside a randomly selected video sourced from compared methods: MeDM [13], CoDeF [45], or Hashing-nvd [5].
16

---

Per-frame Separation 3
NsRCan
NaRCan
NaRCan
Noncon
Figure 13: Separated NaRCan and per-frame warping. Since we have the flexibility separate into multiple canonical images, we conduct an experiment to determine how many canonical images are optimal. We test using Separated NaRCan with segmentation equal and segmentation equal to N, where N equals the number of frames in the video. The results show that the editing information is damaged and has significant displacement due to inaccurate optical flow.
Oig Video
tongan
sectedly at the camera. mendered a alotch styla
Qt Wiech video has better conasterey?
bottar
QC Which vono las hetter cuncall qualy?
Widee
Wide
Figure 14: User Study Website. The video above the original video. The two videos below are ones being compared. We ask participants to determine which of the given videos best matches the description provided in the questions.
A.4
Grid Trick
We adopt the technique named "grid trick" from RAVE [24]. In that paper, method called grid trick is introduced, where multiple images are merged together and then fed to ControlNet [75] for editing obtain style transfer images with sufficient consistency content and color tones. Therefore, when using Separated NaRCan, we only need to apply the grid trick to our k canonical images to achieve this desired consistency among the canonical images easily [24].
Finally, coupled with the linear interpolation method mentioned in our paper, we can ensure that the video maintains sufficient temporal consistency, thereby successfully creating high-quality style transfer video.
17

---

Yacht
Bear
Two-swan
Figure Canonical images after using the grid trick. Using Separated NaRCan, we will obtain multiple canonical images, and through the grid trick, we can generate a high-quality and consistent style transfer canonical image. Therefore, Separated NaRCan still demonstrates excellent performance in this task.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and the introduction briefly introduce the motivation and the main contribution of our work. Our work utilizes the diffusion model to lift the performance video editing, which is also mentioned in the abstract and the Section introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We dicussed our limitations in 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
18

---

The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: First, we conducted a user study on our method and other baseline methods, including CoDeF [45], MeDM [13], Hashing-nvd [5], CCEdit [15]. We designed three questions focused on temporal consistency, text alignment. and quality and tested them on approximately 36 participants. Secondly, evaluated these methods, including ours, using three metrics to demonstrate that our performance is better.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
All assumptions should be clearly stated or referenced
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We expalin our experimental setting in Sec 4.1, and we will provide the metric and the main code. So, we think that will not be a concern. And the other hand, we will provide the link to our user study and the statistic we show is based on the people we tested, not created by us.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
19

---

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide it in the zip file
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/
httec//uuidee/ooeuuuuaeee wwt.iiiissssoiiiioooooooo for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: CCodeSubmiaaae /nips.cccc httppscccppbbbcccppccccece gwwpe///ge/eeeeeeeeeeeeeeee / ss onnooicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
20

---

Answer: [Yes]
Justification: Yes, we would show our hyperparameters in Sec 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: No, we didn't 't show it.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma eгror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We show the machine used in Sec 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
21

---

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics attpp:///neur/e http::pp....pp...cc//cettiiiiii.
Question:
Answer: [Yes]
Justification: We conducted in the paper conform, in every respect, with the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have improved the temporal consistency and quality in processing various video tasks, such as style transfer, dynamic segmentation, and other techniques typically applied images.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We don't release any data or models that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
22

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper explicitly mentions and respects the licenses and of use for all external assets. The creators of the code, data, and models are properly credited in the references section, and the terms of use for these assets are adhered to as specified by their respective licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, sepperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's ss creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The new assets introduced in the paper thoroughly documented in Sec/ 3.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We conduct user study in Figure and also provided screenshot in supplemen- tary material.
23

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [No]
Justification: Our paper doesn't 't bring potential risks incurred by study participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "qncYIF9tOl", "reviewer": "Reviewer_b5x9", "summary": "This paper proposes a novel approach for video editing in the scope of canonical image-based video editing. They argue the canonical images used in prior methods are unnatural, which degrades the performance a lot. To solve this problem, they propose to use a LoRA finetuned diffusion model to refine the unnatural canonical images to be natural. With this design, the performance is significantly improved. Experiments (comparison to baselines and ablation study) have demonstrated the effectiveness of this approach and the design. \n\nCurrently, I tend to accept this paper but I still have many questions about the details of this method. I will make a final decision depends on the rebuttal.", "questions": "- After you refine the canonical images, do you need to change the deformation fields? I think the contents are changed, so the correspondence should also be different, isn't it?\n\n- Why do you use LoRA, instead of other PEFT finetuning strategies or even full finetuning?\n\n- I am curious about the perceptual results of canonical images under large camera movement or significant changes. It is hard for me imagine such a canonical images. '\n\n- Using multiple canonical images is definite an important design. Can you discuss on that or provide more ablation experiments in the final version?\n\n- Some works like [1] also notice that the canonical images are flawed and use a different way to solve this problem (i.e., combine the original images and flawed canonical images), do we really have to make the canonical images natural?\n\n- [2] only considers short-term warping errors, you can consider using warping error in [3]. Besides, why the warping error scale is different with [2].\n\n[1] Blind Video Deflickering by Neural Filtering with a Flawed Atlas\n\n[2] Learning blind video temporal consistency\n\n[3] Blind video temporal consistency via deep video prior", "limitations": "The authors discuss the limitations in the conclusion part.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The motivation for their design is quite reasonable. Unnatural canonical images might degrade the image editing performance, which degrades the video editing performance then. And they do solve the problem by making the canonical image look more natural.\n\n- According to the user study, the performance looks significantly better. The ablation study is also good.\n\n- The analysis and presentation for the experiments part are clear and good.", "weaknesses": "- The writing is not clear enough. There are many miss details, which makes me confused. \n- The method should be slightly slower as they need to fine-tune the diffusion model for a scene. However, no detailed experiment is provided (only discussed in the conclusion). I believe providing such content can be insightful.\n- The comment, \"If the canonical image is not natural, it loses any editing value.\" is too rude, and wrong. Prior works have demonstrated the value of unnatural canonical images. As they train the models jointly, using unnatural canonical images is also reasonable (but not perfect). Please respect prior works.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thorough assessment and valuable feedback. We appreciate the positive comments on our motivation, performance improvements, and experimental analysis. We'll address the concerns and questions raised:\n\n### Strengths: \nWe're glad the reviewer found our motivation reasonable, performance improvements significant, and experimental analysis clear.\n\n### Weaknesses:\n> **W1. Writing clarity:**\n\nWe apologize for any lack of clarity and will improve the paper's writing in the final version, adding more details where needed.\n\n> **W2. Computational efficiency**\n\nWe've conducted additional experiments on computational efficiency:\n- LoRA fine-tuning takes ~30 minutes on two RTX 4090 GPUs.\n- Our full pipeline (including fine-tuning) processes 100 frames in ~50 minutes.\n\n> **W3. Strong wording**\n\nWe appreciate the reminder to respect prior work. We'll rephrase to acknowledge previous approaches while highlighting our improvement: \"While previous methods demonstrate the utility of canonical images, our approach of refining them to be more natural further enhances their editing value.\"\n\n### Questions:\n> **Q1. Deformation fields**\n\nOur training pipeline simultaneously optimizes both the canonical image and the deformation field. When diffusion loss influences the content of the canonical image, the deformation field is optimized during subsequent training steps. This mutual optimization continues throughout the training process. As a result, by the end of training, our deformation field has already adapted to and is fully compatible with the refined canonical image, requiring no additional adjustments.\n\n> **Q2. LoRA choice**\n\nWe chose LoRA for its efficiency and effectiveness in fine-tuning large models with limited data. LoRA's quick convergence and lower computational cost make it particularly suitable for our method. In response to your question, we conducted a comparative experiment with LoHA, another PEFT strategy. Our results show that LoRA achieved a similar visual quality to LoHA in significantly less time (30 minutes vs 70 minutes). This experiment further validated our choice of LoRA for its efficiency.\n\n> **Q3. Large camera movements**\n\nWe acknowledge that in scenarios with extremely drastic camera movements or significant changes, our method, like other approaches, does have limitations and may fail to produce ideal results. However, it's important to note that our approach can handle more complex and dynamic scenes compared to existing canonical-based methods. We've included the perceptual results of canonical images in the attached PDF file (Fig. 3(b)) to illustrate this capability.\n\n> **Q4. Multiple canonical images**\n\nWe agree this is an important aspect. We have conducted a more in-depth ablation study, the results of which are included in the attached PDF (Fig. 2). Our experiments with varying numbers of canonical images (1, 2, 3, 4, 5) reveal important trade-offs:\n- Too few canonical images: It is challenging to model the entire video, leading to poorer reconstruction quality, especially for complex scenes.\n- Too many canonical images: Decreased temporal consistency and significantly longer training times.\n\nBased on these findings, we empirically use 3 canonical images to strike a balance between editing quality and computational efficiency for most scenarios. This choice allows us to effectively model video content while minimizing the drawbacks associated with extreme numbers of canonical images.\n\n> **Q5. Alternative solutions**\n\nWhile methods like [A] offer valuable alternatives, our approach of refining canonical images to be natural provides several benefits:\n- Improved editability for downstream tasks.\n- Better preservation of scene semantics.\n- Simplified pipeline without the need for additional combination steps.\n\nTo illustrate this, we include visual examples in our attached PDF (Fig. 1) demonstrating why natural canonical images are necessary. These examples show that:\n- Non-natural, distorted canonical images can be challenging to edit in tasks like adding handwritten characters, potentially causing user difficulties.\n- Many current style transfer techniques, such as ControlNet, require natural images as input to function effectively.\n\n> **Q6. Warping error**\n\nThank you for your valuable suggestions. We implement the long-term warping error metric from [B] as recommended. The table below shows that our method achieves the best performance under this metric, demonstrating the long-term temporal consistency of NaRCan.\n\n|Method|short-term $E_{warp}$[28]$\\downarrow$|long-term $E_{warp}$ [B]$\\downarrow$|\n|-------------|:--------------:|:--------------:|\n| Hashing-nvd |     0.0070     |     0.0495     |\n| CoDeF       |     0.0085     |     0.0785     |\n| MeDM        |     0.0072     |     0.0583     |\n| Ours        |   **0.0065**   |   **0.0484**   |\n\nRegarding the scale difference with [28], we appreciate you pointing this out. It is due to an oversight in pixel value normalization (0-255 in our evaluations and 0-1 in [28]). We correct this misalignment and re-run our experiments. Our method continues to show the best performance, now with a more consistent scale with [28]. The remaining difference is due to the different tasks: [28] performs blind video temporal consistency enhancement, while NaRCan targets video editing.\n\nWe're grateful for your careful review, which has helped us improve our results' accuracy and consistency. These corrections and new evaluations will be included in the updated paper. We're committed to addressing all points raised to improve the final paper.\n\n[28] Lai, Wei-Sheng, et al. \"Learning Blind Video Temporal Consistency.\" ECCV. 2018.\n\n[A] Lei, Chenyang, et al. \"Blind Video Deflickering by Neural Filtering with a Flawed Atlas.\" CVPR. 2023.\n\n[B] Lei, Chenyang, et al. \"Blind Video Temporal Consistency via Deep Video Prior.\" NeurIPS. 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BeNKQKlbON", "reviewer": "Reviewer_4jq5", "summary": "In this study, the authors propose a hybrid deformation field and diffusion prior update scheduling to generate high-quality canonical images that maintain temporal consistency in video editing.", "questions": "- In Figure 2 (b) Video Representation, why is an accurate canonical image not generated despite applying the hybrid deformation field technique?\n\n- Additionally, why is an accurate canonical image not generated even after updating the MLP with reconstruction loss?\n\n- In 3.2 Diffusion Prior, what does the special token used for fine-tuning LoRA represent?\n\n- Also, what criteria were used to divide the steps for adding noise, and on what basis was the amount of noise added determined?\n\n- In 3.3 Separated NaRCan, how does linear interpolation differ from existing video frame interpolation methods?", "limitations": "- The authors clearly state the limitations of their research to inform the readers.", "rating": 6, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The use of homography, a transformation method from the existing image processing field, to generate accurate canonical images provides a strong foundation for the proposed method.", "weaknesses": "In the Method section, there is a lack of detailed information to support the authors' claims:\n  1) In 3.1 Hybrid Deformation Field for Deformation Modeling:\n     - The structure of Residual MLP and Canonical MLP\n     - How the hybrid deformation field composed of the homography matrix H and Residual deformation MLP is constructed\n  2) In 3.2 Diffusion Prior:\n     - The special token used for fine-tuning LoRA\n  3) In 3.3 Separated NaRCan:\n     - The criteria for dividing steps to add noise and the basis for determining the amount of noise to add\n\nCompleteness of the text:\n  - The order of Figures 3 and 4 does not match the order mentioned in the text.\n  - What does Fig.9(a) in line 123 refer to? Is it simply a typo?\n\nExperiment:\n  - The experimental results shown do not confirm whether the proposed method maintains the temporal consistency of the video through the generated canonical image.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thorough assessment and constructive feedback. We acknowledge the concerns raised and will address them point by point:\n\n### Strengths: \nWe're glad the reviewer recognizes the value of incorporating homography into our method.\n\n### Weaknesses and Questions:\n> **Q1. Method Details and Noise Scheduling Criteria**\n\nWe apologize for the lack of detail and will expand the Method section in the final version. Specifically:\n1. Hybrid Deformation Field (3.1):\n    - Homography: a trainable homography matrix.\n    - Residual MLP: 5-layer MLP with Sinusoidal activations, hidden dim 256.\n    - Canonical MLP: 5-layer MLP with Sinusoidal activations, hidden dim 256.\n    - Construction: $H(u,v,t)$ estimates global transformation, $g(u,v,t)$ learns residual deformations.\n2. Diffusion Prior (3.2) and Separated NaRCan (3.3):\n    * Special token: \"[V]\" prepended to text prompts representing video-specific features (L138-142). This is a widely used technique (DreamBooth [C], RealFill [58], etc.) to fine-tune the pre-trained diffusion model with customized data to represent/generate specific objects/scenes.\n    * Noise scheduling (Fig. 4 in our paper). These values were determined empirically to balance refinement and training efficiency. The criteria for dividing steps were based on extensive experiments and visual analysis, identifying three distinct stages in the model's convergence process:\n        * Early stage (Steps 1000-3000): Higher noise levels (40%, update every 10 steps) allow the diffusion model to guide the formation of natural canonical images, overcoming potential initial instabilities in the deformation field.\n        * Middle stage (Steps 3001-5000): Moderate noise levels (30%, update every 100 steps) are used as the model begins to stabilize, balancing refinement with preservation of learned features.\n        * Late stage (Steps 5001+): Lower noise levels (20%, update every 2000 steps) are applied to fine-tune the canonical image while maintaining the overall structure and content established in earlier stages.\n\n    This graduated approach ensures that the diffusion model can effectively guide the process throughout training, adapting its influence as the canonical image and deformation field improve.\n\n> **Q2. Figure Ordering**\n\nWe apologize for the confusion. We'll correct the figure ordering and references in the final version.\n\n> **Q3. Fig. 9(a) in line 123**\n\nThis reference is intentional but may appear misplaced due to formatting. The figure supports our discussion on MLP+TVFlow limitations and the importance of homography.\n\nFig. 9(a) shows our ablation study comparing MLP+TVFlow with our proposed Homography+MLP+TVFlow approach, which supports the claims made in this section. However, we failed to properly place the figure reference within parentheses, which may have caused confusion.\n\nWe apologize for any confusion caused by improper placement. In the revised version, we will correct the formatting, clarify the text-figure connection, and consider repositioning the figure for better alignment. We appreciate your attention to detail, which helps improve our paper's clarity and coherence.\n\n> **Q4. Temporal Consistency**\n\nWe'd like to argue that our method achieves superior temporal consistency, evidenced by:\n* Quantitative metrics: Table 1 of our paper shows that our method outperforms others in terms of warping error. New experiments using long-term warping error [B] further demonstrate our superior performance:\n\n|Method|short-term $E_{warp}$ [28]⭣|long-term $E_{warp}$ [B]⭣|\n|-|:-:|:-:|\n|Hashing-nvd|0.0070|0.0495|\n|CoDeF|0.0085|0.0785|\n|MeDM|0.0072|0.0583|\n|Ours|**0.0065**|**0.0484**|\n\n* Qualitative results: Videos in our supplementary materials show better semantic alignment with the text prompts and visual consistency compared to other methods.\n* User study (Fig. 6 in the paper): 72.2% of participants preferred our method for temporal consistency.\n\nWe attribute these results to our natural-refined canonical images. The revised paper will provide a more comprehensive analysis of temporal consistency performance.\n\n> **Q5. Canonical Image Quality**\n\nWhile hybrid deformation field and reconstruction loss achieve good reconstruction, they don't ensure a natural canonical image. The MLP alone can reconstruct the input video well even with unnatural canonical images, as reconstruction loss doesn't respect naturalness.\n\nOur attached PDF (Fig. 3(a)) shows visual evidence that excellent reconstruction can still result in unnatural canonical images without diffusion loss, limiting downstream task applicability. This highlights the importance of our diffusion-guided approach in producing natural, versatile canonical images.\n\n> **Q6. Linear interpolation**\n\nOur method differs fundamentally from traditional video frame interpolation:\n1. Our approach: We represent the video using multiple canonical images, applying linear interpolation between them to reduce flickering in overlap regions (Fig. 3 in our paper). This is a representation strategy, not frame generation.\n2. Traditional interpolation: Synthesizes intermediate frames between existing ones using techniques like optical flow.\n\nKey distinction: We blend different representations of the same content for smooth transitions, which is crucial for temporal consistency in video editing. Our goal is coherent video representation for editing, not frame rate increase or gap-filling like traditional methods.\n\nWe appreciate the reviewer's careful reading and insightful questions. We're committed to addressing all points raised to significantly improve the final paper. We believe these clarifications and additions will address the concerns about soundness and contribution, elevating the paper's quality and impact.\n\n[28] Learning Blind Video Temporal Consistency\n\n[58] RealFill: Reference-Driven Generation for Authentic Image Completion\n\n[B] Blind Video Temporal Consistency via Deep Video Prior\n\n[C] Dreambooth", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UpwTuAiEQz", "reviewer": "Reviewer_pLuV", "summary": "This paper proposes to solve the issue of temporal consistency in video editing. They propose a hybrid deformation field architecture, with a trainable homography matrix H(u, v, t) and residual deformation MLP, representing object variations throughout an entire scene. Combining it with the diffusion before the pipeline helps generate natural canonical images.", "questions": "1. What does it mean \"this regularization term restricts the model’s ability to express itself without providing additional\ninsights.\" (line 118-119) What's the additional \"insight\" you are looking for?\n2. How can diffusion prior ensure that the generated image is natural? The connection between the two is not clear to me. Is there any reason for that?\n3. Is the noise and diffusion prior update scheduling decided manually? If so, how did you decide it?", "limitations": "No, the authors did not address the limitations.\nI'm curious about the limitations of the motions that this method can handle. Can it deal with all kinds of videos, even with heavy motion blur? From the supplements, it is not clear to me.", "rating": 5, "confidence": 2, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The results are relatively temporal coherent.\n2. The converge speed is faster than other methods.", "weaknesses": "1. The paper is not well-written and lacks clarity for the reader. For example, diffusion loss has played an important role in this paper, yet it is not introduced. Some of the expressions are vague. Like in question 1.  What's the additional \"insight\" you are looking for?\n2. Things claimed are not reasoning enough like question 2.\n3. Limitations are not addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback and the opportunity to clarify our work. We acknowledge the concerns raised and will address them point by point:\n\n### Strengths: \nWe're glad the reviewer recognizes our method's temporal coherence and faster convergence.\n\n### Weaknesses and Questions:\n> **Q1. Writing clarity** \n\nWe apologize for the lack of clarity in our paper. We address these issues in our detailed responses below, including a clearer explanation of diffusion loss and clarification of vague expressions. The final version of our paper will incorporate all these improvements, ensuring a much clearer presentation of our work. We appreciate your feedback, which has been invaluable in enhancing our paper's quality.\n\n\n> **Q2. \"Additional insights\" (lines 118-119)**\n\nWe apologize for the unclear wording. By \"additional insights,\" we meant that traditional regularization terms like TVFlow limit model expressiveness without providing guidance on how to represent complex deformations. Our hybrid approach aims to provide this guidance through the homography matrix. We'll rephrase this section for clarity.\n\n\n> **Q3. How can diffusion prior ensure that the generated image is natural?**\n\nThe diffusion prior ensures the naturalness of the generated image through several mechanisms:\n- LoRA fine-tuning: We fine-tune the diffusion model using LoRA on various reference images from the specific video scene. This process allows the diffusion model to learn the characteristics of objects and environments unique to this video.\n\n- Leveraging pre-trained knowledge: The diffusion model comes with pre-trained knowledge of general image structure and content. This broad understanding helps guide the canonical image towards realistic representations.\n\n\n- Guiding the canonical image: During training, the diffusion prior minimizes the difference between the current canonical image and what the diffusion model considers natural for that scene. This process helps correct unnatural elements that might arise from the reconstruction process alone.\n\n- Restoration capability: If unnatural elements appear in the canonical image, the diffusion model, having seen natural scenes during fine-tuning, guides the image back towards a more realistic representation.\n\nIn essence, the diffusion prior acts as a learned naturalness constraint, ensuring that our canonical images remain faithful to the video content while adhering to the principles of natural image formation.\n\nWe'll expand this explanation in our revised paper, providing a more detailed description of the process and underlying principles to clarify the connection between the diffusion prior and the naturalness of the generated images.\n\n> **Q4. Noise and diffusion prior update scheduling**\n\nThe criteria for dividing steps were determined based on extensive experiments and visual analysis. We identified three distinct stages in the model's convergence process: early, middle, and late stages, each requiring different noise intensities, as detailed in our paper (lines 149-157 and Fig. 4).\n\n- Early stage: Higher noise levels (40%) are applied to allow the diffusion model to guide the formation of natural canonical images, overcoming potential initial instabilities in the deformation field.\n\n- Middle stage: Moderate noise levels (30%) are used as the model begins to stabilize, balancing refinement with preservation of learned features.\n\n- Late stage: Lower noise levels (20%) are applied to fine-tune the canonical image while maintaining the overall structure and content established in earlier stages.\n\nThis graduated approach ensures that the diffusion model can effectively guide the process throughout training, adapting its influence as the canonical image and deformation field improve.\n\n> **Q5. Limitations**\n\nWe address limitations in our original submission in the conclusion section (lines 250-255), including time-consuming LoRA fine-tuning (about 30 minutes) and challenges with extreme changes in video scenes. Additionally, we also include qualitative results of failure cases in the attached PDF (Fig. 3(b)) of this rebuttal to further illustrate these limitations.\n\n\nWe sincerely appreciate the reviewer's careful reading and insightful questions. We're committed to addressing all points raised to significantly improve the final paper's clarity, reasoning, and completeness. We believe these additions and clarifications will address the concerns about soundness and presentation, elevating the paper's quality and impact.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers and Area Chairs,

We sincerely thank all reviewers for their thorough assessments and valuable feedback. We appreciate the positive comments on our work:

Strengths:
1. Reasonable motivation and significant performance improvements (Reviewer b5x9).
2. Clear experimental analysis (Reviewer b5x9).
3. Superior temporal coherence in video editing results (Reviewer pLuV).
4. Faster convergence compared to existing methods (Reviewer pLuV).
5. Valuable incorporation of homography into our method (Reviewer 4jq5).

We have responded to each reviewer individually to address any comments. We would like to give a brief summary.

1. **Writing clarity and method details**: We expand explanations, particularly for the Hybrid Deformation Field and Diffusion Prior.
2. **Computational efficiency**: We provide specific runtime details for LoRA fine-tuning and the full pipeline.
3. **Justification for design choices**: We explain the selection of LoRA, noise scheduling, and the use of multiple canonical images.
4. **Limitations and failure cases**: We expand the discussion on the types of motions handled, performance on challenging videos, and potential failure scenarios.

We support these with additional experiments on challenging cases, providing both qualitative results in the individual rebuttal below and quantitative results in the attached PDF.

Again, we thank all reviewers and area chairs!

Best, Authors

Meta-review: The paper initially received some mixed reviews, with ratings $(3, 4, 6)$. However, following the rebuttal, all reviewers found the authors' responses adequately addressed their concerns and thus gave the final ratings with two *weak accept* and one *borderline accept*. 

Specifically:

- **Reviewer b5x9** found the rebuttal to be clear with a comprehensive explanation and results on computational efficiency, large camera movements, warping errors, and more discussion on other questions.

- **Reviewer 4jq5** confirmed that the proposed method showed better semantic alignment, and the difference between linear interpolation and video frame interpolation mentioned in the paper was clearly explained.

- **Reviewer pLuV** felt the detailed explanation addressed the concerns and believed the remaining obscured parts could be clarified in the revised paper,

Given the positive feedback and support from all reviewers, the area chair recommends accepting this paper.
Decision: Accept (poster)