NaRCan: Natural Refined Canonical Image with Integration of Diffusion Prior for Video Editing

Authors: ting-hsuan  chen
jiewen  chan
hau-shiang  shiu
shih-han  yen
chang-han  yeh
yu-lun  liu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-02 08:04:56

Abstract: We propose a video editing framework, NaRCan, which integrates a hybrid deformation field and diffusion prior to generate high-quality natural canonical images to represent the input video. Our approach utilizes homography to model global motion and employs multi-layer perceptrons (MLPs) to capture local residual deformations, enhancing the model’s ability to handle complex video dynamics. By introducing a diffusion prior from the early stages of training, our model ensures that the generated images retain a high-quality natural appearance, making the produced canonical images suitable for various downstream tasks in video editing, a capability not achieved by current canonical-based methods. Furthermore, we incorporate low-rank adaptation (LoRA) fine-tuning and introduce a noise and diffusion prior update scheduling technique that accelerates the training process by 14 times. Extensive experimental results show that our method outperforms existing approaches in various video editing tasks and produces coherent and high-quality edited video sequences. See our project page for video results: [koi953215.github.io/NaRCan_page](https://koi953215.github.io/NaRCan_page/).

Main paper: 
 introduction: 
 Video editing has always been a fascinating research area. For example, style transfer transforms the original video into a completely new style, enriching the viewing experience. Other tasks include dynamic segmentation and handwriting, which all demonstrate the broad application value of video editing across various fields. Currently, diffusion model technology is becoming increasingly mature and is known for its powerful generative capabilities and frequent use in video editing. However, in video-to-video tasks, maintaining temporal consistency presents a significant challenge, particularly when applying image-based diffusion models to video editing tasks. When these models, originally designed for image generation, are applied frame-by-frame to videos, they often produce temporally inconsistent results due to their frame-independent processing nature. This limitation has motivated numerous research efforts to enhance temporal coherence through various techniques, such as optical flow guidance, latent space alignment, and cross-attention mechanisms, aiming to produce high-quality video sequences while preserving temporal consistency across frames.Yet, even with solutions addressing temporal consistency, diffusion-based methods [16, 4, 19, 8] still encounter significant challenges in precise localized editing tasks. This is where canonicalbased methods demonstrate their unique advantages. By consolidating video content into a single representative image, canonical-based methods [3, 1, 5, 45] enable intuitive and precise spatial control over editing operations while inherently maintaining temporal consistency. This unified representation allows direct application of image-based editing techniques while ensuring that modifications propagate coherently throughout the video sequence, making these methods particularly effective for a wide range of video editing tasks. Given an RGB video, we can represent the video using a canonical image. However, the canonical image and reconstruction training process focuses only on reconstruction quality and could produce an unnatural canonical image. This could cause problems with downstream tasks such as prompt-based video editing. In the bottom example, if the hand is distorted in the canonical image, the image editor, such as ControlNet [75] , may not recognize it and could introduce an irrelevant object instead. In this paper, we propose introducing the diffusion prior from a LoRA [18] fine-tuned diffusion model to the training pipeline and constraining the canonical image to be natural. Our method facilitates several downstream tasks, such as (a) video editing, (b) dynamic segmentation, and (c) video style transfer.While previous canonical-based approaches have established the fundamental utility of canonical images in video editing, our observations suggest that enhancing the naturalness of these representations can significantly improve their effectiveness in various editing scenarios. We observe that existing methods primarily focus on reconstruction quality without explicit constraints to ensure natural canonical image generation. To address this opportunity for enhancement, we propose NaRCan, a novel hybrid deformation field network architecture that incorporates diffusion priors (Figure 1 ) into the training pipeline (Figure 2 ). This integration enables our method to generate high-fidelity natural canonical images across diverse scenarios while maintaining temporal consistency. Through comprehensive experimental evaluation, we demonstrate that NaRCan achieves superior performance compared to state-of-the-art video editing methods in both qualitative and quantitative metrics.Our main contributions are three-fold:• We have designed a novel deformation field structure to represent object variations throughout an entire scene. Compared to other canonical-based models, our model demonstrates superior expressive capability and faster convergence speed. • We have effectively integrated the diffusion prior into our pipeline, enabling our method to generate high-quality natural canonical images in any scenario. Additionally, we designed a dynamic scheduling method that significantly accelerates the entire training process. • We thoroughly evaluate our method to show the state-of-the-art video editing performance. 
related work: 
 Implicit neural representation. Implicit neural representation [43] using coordinate-based MLP is an outstanding way to represent a video, capable of obtaining a canonical image to represent the entire video [45] . Recent methods employ hash grid encoding [42] or positional encoding [40] combined with MLP. The approach in [5] is more effective at handling spatial information, but the resulting canonical image exhibits severe distortion and warping with in-the-wild videos [11] . Therefore, we propose a hybrid deformation field method composed of homography [9] and residual deformation MLP. This model design fits the deformation information in videos better than existing methods.Consistent video editing. There are generally three approaches to video editing: (1) propagationbased, (2) layered representation-based, and (3) canonical-based. The first approach, propagation-based, focuses on propagating information across frames [35, 20, 22, 23, 57, 63, 67] . This method can easily produce inaccurate results due to occlusions and error propagation. The second approach, layered representation-based, separates a video into foreground and background, obtaining canonical images for both [36, 25, 73, 34, 37, 38] . We can edit the entire video by editing these canonical images and then synthesizing the video afterward. However, this method heavily relies on masks. If the mask-RCNN captures incorrect targets in the preprocessing stage, the fitted foreground can be incorrect, especially in scenes with large camera movement.The third approach, canonical-based, typically uses MLP to obtain the deformation information of each pixel to form a canonical image [45] . Transferring the video to canonical space maintains temporal consistency while editing and supports various downstream tasks, such as super-resolution and segmentation. CoDeF [45] uses this approach, but canonical images can deform severely in videos with significant camera or object movement. CoDeF suggests that using a group model could resolve these issues. However, using group CoDeF requires masks for training data obtained from SAM-track [12] . Incorrect masks for training data can result in an unnatural canonical image of the video. Even with correct masks for the foreground object, the canonical image might still be corrupted, rendering these images ineffective for video editing.Video processing via generative models. Some works utilize GAN inversion [65, 78, 60, 74, 48, 26, 69] to edit images or videos. Today, numerous generative models exist for editing images. Some methods, such as GLIDE [44] , DALL-E [55, 54] , stable diffusion [56] , and Imagen [58] , are trained on millions of images, resulting in incredible generative abilities. Other methods like SDEdit [39] , ControlNet [75] , and LDM [2] use conditions to achieve better editing results. Instruction-based video editing methods like InstructPix2Pix [3] and another work [10] often yield sub-optimal results for different editing operations. Techniques like LoRA [18] can assist in fine-tuning to find better weights for editing. Additionally, many zero-shot diffusion-based methods [72, 16, 76, 4, 8, 19, 27] do not require model training but still need constraints to maintain temporal consistency. To address temporal consistency concerns, most methods like Tune-A-Video [70] , Text2Video-Zero [27] , FateZero [52] , and Vid2Vid-Zero [66] incorporate cross-attention mechanisms. Some works propose training for video editing, such as Imagen Video [17] and Make-A-Video [59] , but these require large datasets and significant computational resources. Unlike these methods, MeDM [13] uses a flow-coding algorithm to solve this problem. Canonical-based design does not require another mechanism to maintain temporal consistency, however. Once the canonical image is edited, the changes can be propagated to every frame using a deformation field.Lifting the naturalness of canonical image by diffusion models. The diffusion prior has been applied in various domains. Reconfusion [71] is a few-shot novel view synthesis work that introduces a diffusion model before optimizing sampled novel views. Dreamfusion [50] , a text-to-3D work, introduces score distillation sampling (SDS) loss, referencing a 2D diffusion model to optimize 3D outputs. This approach inspires us to utilize the diffusion model to improve performance. Other methods like [6, 7, 21, 41, 33, 61, 64, 68, 79] also employ diffusion prior for text-to-3D tasks.Several diffusion models focus on text-to-image generation [53, 56, 58, 62] . Additionally, some diffusion models can refine corrupted images to make them appear more natural. We propose adding a diffusion model to our pipeline (Figure 2 ) to enhance the naturalness of our canonical images. Our goal is to improve the restoration of canonical images using the diffusion model. While we create canonical images using a hybrid deformation field, this method might not always deliver optimal performance, especially in scenarios with dramatic motion changes or severe non-rigid transformations. The capabilities of the hybrid deformation field are still limited in such cases. Therefore, we aim to introduce a diffusion model to make our canonical images more natural, enhancing the effectiveness of video editing. This research has significant practical implications as it can improve the quality and realism of video editing, benefiting various industries such as film production, advertising, and virtual reality. 
method: 
 In this section, we first introduce our hybrid deformation modeling by combining homography and deformation MLP in Section 3.1. Subsequently, we elaborate on how we integrate the diffusion prior from a LoRA fine-tuned latent diffusion model to ensure the naturalness of our canonical image representation in Section 3.2. Finally, we provide an additional way to improve the quality of our By relying entirely on the reconstruction loss, the canonical MLP often fails to represent a natural canonical image, causing problems for downstream applications. E.g., image-toimage translation methods such as ControlNet [75] may not be able to recognize that there is a train in the canonical image. (c) Therefore, we leverage the fine-tuned latent diffusion model to regularize and correct the unnatural canonical image into a natural one. Specifically, we sophistically design a noise scheduling corresponding to the frame reconstruction process. (d) The natural and artifacts-free canonical image can then be facilitated to various downstream tasks such as video style transfer, dynamic segmentation, and editing, such as adding handwritten characters of "NaRCan".video representation by separating multiple canonical images and describing the necessary changes for downstream tasks in Section 3.3. Figure 2 shows our proposed framework.Traditional methods often rely on direct predictions of ∆u and ∆v, which means the displacement of pixel points u, v at time t, by using an MLP g(•, •, •) and query the RGB color by another canonical image MLP f (•, •):EQUATIONsupplemented with a TVFlow regularization term to prevent overfitting by constraining the magnitude of spatial deformations. While this regularization effectively prevents extreme deformations, it merely imposes magnitude constraints without providing meaningful guidance for modeling complex spatial transformations. To address this limitation, we propose a hybrid deformation field architecture composed of a trainable homography matrix H(u, v, t) and residual deformation MLP:EQUATIONUnlike the simple constraint-based approach of TVFlow, our method leverages homography to provide global displacement information as structural guidance for the subsequent residual deformation MLP. This hierarchical design enables the residual deformation MLP to learn and express the deformation field more accurately and effectively by building upon the initial geometric transformation provided by the homography matrix.The canonical image encompasses all information within the entire video and can be reconstructed for each original video frame using the deformation field outlined in Section 3.1. Thus, editing solely the canonical image yields a temporally coherent edited video. However, editing tasks such as drawing or writing on objects or editing based on ControlNet [75] require a natural image as input to produce meaningful edited images. Existing canonical base methods do not incorporate mechanisms to ensure that the generated image is natural; instead, they rely solely on the model's ability to learn a natural image. However, when encountering scenarios with camera movement or significant changes in video content, these existing techniques cannot adapt to such drastic variations. The model may generate a canonical image that is nearly impossible to edit (rendering it devoid of any subsequent value). Initially, our model fits object outlines before the fields converge and without the diffusion prior, resulting in unnatural elements in the canonical image due to complex non-rigid objects. Upon introducing the diffusion prior with increased noise and update frequency, the model learns to generate natural, high-quality images, leading to convergence. Thus, the strength of noise and the update frequency will also decrease. Moreover, it's worth mentioning that update scheduling cuts training time from 4.8 hours to 20 minutes.To address these challenges, we introduce diffusion priors, which successfully mitigate this issue. Our method can generate high-quality canonical images through diffusion priors, providing valuable inputs for various video editing tasks.LoRA fine-tuning. To enhance the current diffusion model's ability to represent all video content better, we introduced a special token specific to this scene. We then fine-tuned the LoRA weight of the pre-trained diffusion model. This ensures that the diffusion model generates high-quality natural canonical images tailored to the testing sequence rather than randomly generating natural images that do not belong to the scene.Noise and diffusion prior update scheduling. While the hybrid deformation field technique mentioned in Section 3.1 already produces better canonical images than other existing methods, it still needs to improve. As Figure 9 (b) depicts, canonical images generated solely relying on homography and residual deformation MLP still exhibit various degrees of distortion and unnatural characteristics. Therefore, integrating diffusion priors becomes imperative. We extract the canonical region currently observed by the model and calculate diffusion loss with the target image generated by the diffusion model to ensure the generation of natural canonical images. However, generating a target image at each step would significantly prolong the training phase. Hence, we propose a hierarchical update scheduling to accelerate the process, which is shown in Figure 3 . In the initial stages of training, when the deformation field has not yet converged, more substantial noise is introduced to allow the diffusion model to dominate the scene of the canonical image. Simultaneously, the frequency of generating target images needs to be denser. As training progresses and the deformation field becomes more stable, the noise intensity and frequency of generating target images decrease accordingly. This hierarchical scheduling approach ensures that the final canonical image approaches the quality of per-step updates while speeding up the training process by 14 times (4.8 hours to 20 minutes.) We opt for diffusion loss over SDS because using SDS, as mentioned in the Reconfusion [71] paper, is more prone to generate artifacts.When encountering overly complex scenes, relying solely on a single natural canonical image representing the entire scenario is impractical and unrealistic. Hence, we need to segment the original video into multiple segments {S 1 , ..., S k } and train dedicated residual deformation MLPs {R 1 , ..., R k } for each segment to obtain k natural canonical images {C 1 , ..., C k }. It is worth noting that S i and S i+1 have an overlap, referred to as the overlap window. Frames within this region are obtained using linear interpolation shown in Figure 4 . This method ensures that excellent temporal consistency is maintained when switching from canonical image C i to C i+1 . Table 1 demonstrates that our temporal consistency surpasses all existing methods even after segmentation. Additionally, Figure 4 : Linear interpolation. After using the grid trick [18] to obtain the highly consistent canonical images C k and C k+1 , we interpolate all frames within the overlap window. As time progresses, the weight for reconstructing each frame gradually shifts from referencing C k to solely referencing C k+1 . We achieve editing results with remarkable temporal consistency through this linear interpolation approach. Please refer to our supplementary material for more video results.we adopt different processing approaches for various downstream tasks to ensure that Separated NaRCan can adeptly adapt to these tasks.Style transfer. With multiple canonical images obtained, we utilize the grid trick [14, 24] to ensure sufficient consistency in style and content across the k canonical images. Specifically, we concat the canonical images into a larger image (with 2×2 canonical images) and use ControlNet [75] to perform text-guided image editing.Video editing. When addressing video editing tasks such as handwriting, we leverage the pretrained optical flow models to compute the flow between the k canonical images and use this flow to warp the editing content from C 1 to the C k image.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes]Justification: First, we conducted a user study on our method and other baseline methods, including CoDeF [45] , MeDM [13] , Hashing-nvd [5] , CCEdit [15] . We designed three questions focused on temporal consistency, text alignment, and quality and tested them on approximately 36 participants. Secondly, we evaluated these methods, including ours, using three metrics to demonstrate that our performance is better.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experimental setup: 
 We conduct experiments to underscore the robustness and versatility of our proposed method. Our representation is robust with a variety of deformations, encompassing rigid and non-rigid objects, as well as complex scenarios such as smog and waves. We commence the introduction of the diffusion model at the 1000th iteration. From iteration 1000 to iteration 3000, the noise intensity is set at 0.4, and the target image generation frequency is every 10 iterations. Subsequently, spanning from iteration 3001 to iteration 5000, the noise intensity is adjusted to 0.3, with the target image generation frequency occurring every 100 iterations. Beyond the 5000th iteration mark, the noise intensity decreases to 0.2, and the target image is generated every 2000 iterations. The total iteration is 12000 iterations, and Figure 3 is shown to visualize the process of noise scheduling. On a single NVIDIA RTX4090 GPU, the average training duration is approximately 20 minutes when utilizing 100 video frames. When evaluating the temporal consistency in Table 1 , we compare our separated NaRCan with other compared methods by setting k = 3, i.e., we represent the sequence using three canonical images. By adjusting the training parameters accordingly, the optimization duration can be varied from 20 minutes to an hour.Video editing. We run our method CoDeF [45] , Hashing-nvd [5] , CCEdit [15] , and MeDM [13] on the DAVIS [49] and BalanceCC Benchmark that CCEdit proposed for evaluating the results of video editing. In Figure 5 , we show the comparison of visual results. The BalanceCC Benchmark provides a unified text prompt for each scene, and we ensure that all video clips from this benchmark are restricted to a maximum of 100 frames. We utilize ControlNet [75] to edit the video using the unified text prompt provided by the BalanceCC Benchmark for a fair comparison. We executed MeDM and CCEdit within their provided environment settings and pre-trained models. Moreover, Hashing-nvd is a video decomposition work that outputs two images representing foreground and background. To maintain the consistent style of these two images for video editing, We utilize the grid trick proposed by RAVE [24] to tackle this problem. Finally, there are also some video-editing results on DAVIS [49] , and the corresponding text prompts originated from the BalanceCC Benchmark.For the evaluation of temporal consistency, as shown in Table 1 , we utilize short-term [29, 77] and long-term [30] warping errors, along with interpolation error [32] , as our primary metrics.Metrics for evaluation. Since our main focus is text-guided video editing, we conduct user studies on edited videos compared with other methods, like CoDeF [45] , MeDM [13] , and Hashing-nvd [5] . In the user study, 36 participants were shown two edited videos, the original video on each page, and the text prompt used to edit the videos. There are three critical questions for users to answer. (1) Which video has better temporal consistency? (2) Which video aligns better with the text prompt? (3) Which video has better overall quality? Figure 6 summarizes the user study results, demonstrating that our method outperforms in all three dimensions.Comparison of canonical images. In Figure 7 , we run our method, CoDeF [45] , and Hashingnvd [5] on DAVIS [49] and BalanceCC [15] . Note that these are the canonical images of the input video, and we show the foreground atlas for Hashing-nvd. There will be two output atlases for the background and foreground of Hashing-nvd. For comparison, we only show the foreground atlas for Hashing-nvd. As a result, shown in Figure 7 , the output canonical images of our method are more natural than others, even in the scenes with the dramatic motion of the objects, such as scenes named "train" and "butterfly." We can clearly see that our canonical images preserve the original object information well in the above two scenarios.To evaluate the performance of our method's handwritten video editing, we compare with CoDeF [45] and Hashing-nvd [5] , which produce canonical images of the video for users to write the characters on the canonical image to accomplish video editing. We write "NaRCan" to test the performance of these three methods on two scenes, "gold-fish" and "train", in the BalanceCC Benchmark [15] in Figure 8 (a). We extract the same frame of videos for comparison.For dynamic video segmentation, we segment the mask using the Segment Anything Model (SAM) [28] based on the learned canonical image of each method and propagate it to the sequence. The target of the mask in these two scenes are the clownfish named "coral-reef" and the flying butterfly in the scene called "butterfly." We use white to mark the mask for better visibility in Figure 8 (b).Homography & Residual Deformation MLP. In this section, we conducted ablation experiments focusing on both homography and residual deformation MLP. Figure 9 (a) clearly illustrates that using only MLP to fit the deformation field [31, 46, 47, 51] results in unsuitable canonical images for downstream tasks. Without the global information homography provides, the model encounters difficulties in converging the diffusion loss and instead focuses solely on optimizing the reconstruction loss. Conversely, if we rely solely on homography to express the deformation field, homography's expressive power is limited in capturing detailed variations in non-rigid objects. As a result, only approximate and blurred outcomes can be obtained.Diffusion prior. Relying on homography and the residual deformation MLP achieves relatively better canonical images than previous methods. However, the lack of assistance from the diffusion prior still prevents the stable generation of high-quality natural canonical images. Figure 9 (b) demonstrates the significance of supervising canonical images with the diffusion prior. 10 : Trade-off between reconstruction quality and temporal consistency with varying separations. The visual results demonstrate that increasing the number of separations (from 1 to 5) improves the reconstruction quality of video frames. However, the table reveals a trade-off: as the number of separations increases, temporal consistency decreases. Our empirical findings suggest that using 3 separations typically achieves a balance between reconstruction quality and temporal consistency for most scenarios. Impact of Separation Numbers. To investigate the optimal configuration of our Separated NaRCan, we conduct ablation studies by varying the number of canonical images from one to five. Our analysis reveals a clear trade-off between reconstruction quality and temporal consistency, as illustrated in Figure 10 . While increasing the number of separations improves frame reconstruction fidelity by allowing more detailed scene representation, it introduces additional transition regions that may compromise temporal coherence. Through extensive experimentation, we find that using three canonical images strikes an optimal balance between representational capacity and temporal stability for most video scenarios.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We expalin our experimental setting in Sec 4.1, and we will provide the metric and the main code. So, we think that will not be a concern. And on the other hand, we will provide the link to our user study and the statistic we show is based on the people we tested, not created by us. 
conclusion: 
 In this paper, we introduce NaRCan, a video editing framework, integrating diffusion priors and LoRA [18] fine-tuning to produce high-quality natural canonical images. This method tackles the challenges of maintaining the natural appearance of the canonical image and reduces training times with new noise-scheduling techniques. The results show NaRCan's advantage in managing complex video dynamics and its potential for wide use in various multimedia applications.Limitations. Our method relies on LoRA [18] fine-tuning to enhance the diffusion model's ability to represent the current scene. However, LoRA fine-tuning is time-consuming (about 30 minutes). Additionally, our training pipeline includes diffusion loss, which increases the training time. In cases of extreme changes in video scenes, diffusion loss sometimes fails to guide the model in generating high-quality natural images (Figure 11 ). These limitations point out the challenge of balancing model adaptability with computational efficiency and effectiveness in varied conditions. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: No, we didn't show it.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We don't release any data or models that have a high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper explicitly mentions and respects the licenses and terms of use for all external assets. The creators of the code, data, and models are properly credited in the references section, and the terms of use for these assets are adhered to as specified by their respective licenses.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The new assets introduced in the paper are thoroughly documented in Sec/ 3.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [Yes] Justification: We conduct user study in Figure 6 and also provided screenshot in supplementary material.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Single NaRCan compared with Separated NaRCan Dissecting canonical-based methods. In this section, we will delve deeper into two different canonical-based methods: CoDeF and Hashing-nvd. As shown clearly in Figure 12 , CoDeF generates poor-quality canonical images due to the lack of supervision and constraints from the diffusion prior. In contrast, our method can consistently produce high-quality canonical images regardless of the video length, effectively preparing for subsequent downstream editing tasks. The drawbacks of Hashing-nvd are also apparent. This method relies heavily on the Mask-RCNN technique, often resulting in inaccurate or incorrect foreground and background segmentation. Consequently, the final canonical images generated are difficult to edit or inapplicable to techniques such as ControlNet.Parameter settings of Separated NaRCan. Subsequently, as shown in Figure 13 , From our experiments, we found that when using Separated NaRCan, it is crucial to limit the number of segmentations in the video. This is because the editing information on the edited canonical image is propagated through warping using flows. If the flow is inaccurate or contains errors, excessive warping can lead to severe cumulative errors, significantly damaging the edited content.Hashing-nvd foreground We provide an interactive HTML interface to browse video results for comparisons. Specifically, we provide video comparisons on three different tasks: (1) ControlNet style transfer, (2) dynamic segmentation, and (3) adding handwritten characters. We compare our proposed method, NaRCan, with state-of-the-art methods: Hashing-nvd [5] , CoDeF [45] , and MeDM [13] . We also visualize the optimized canonical images if available for reference.To conduct our user study, we employ GitHub Pages in conjunction with a Google Form to facilitate user evaluations of video quality. Each evaluation session comprises 15 scenes, each of which contains 3 questions. For these evaluations, we randomly select 15 scenes from a pool of 100 scenes in BalanceCC Benchmark [15] . Each evaluation page (Figure 14 ) presents a video edited using our method alongside a randomly selected video sourced from compared methods: MeDM [13] ,CoDeF [45] , or Hashing-nvd [5] .Per-frame Separation = 3 t Figure 13 : Separated NaRCan and per-frame warping. Since we have the flexibility to separate into multiple canonical images, we conduct an experiment to determine how many canonical images are optimal. We test using Separated NaRCan with segmentation equal to 3 and segmentation equal to N, where N equals the number of frames in the video. The results show that the editing information is damaged and has significant displacement due to inaccurate optical flow.Figure 14 : User Study Website. The video above is the original video. The two videos below are the ones being compared. We ask participants to determine which of the given videos best matches the description provided in the questions.We adopt the technique named "grid trick" from RAVE [24] . In that paper, a method called grid trick is introduced, where multiple images are merged together and then fed to ControlNet [75] for editing to obtain style transfer images with sufficient consistency in content and color tones. Therefore, when using Separated NaRCan, we only need to apply the grid trick to our k canonical images to achieve this desired consistency among the canonical images easily [24] .Finally, coupled with the linear interpolation method mentioned in our paper, we can ensure that the video maintains sufficient temporal consistency, thereby successfully creating a high-quality style transfer video.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Justification: Yes, we would show our hyperparameters in Sec 4.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 


Decision: The paper initially received some mixed reviews, with ratings $(3, 4, 6)$. However, following the rebuttal, all reviewers found the authors' responses adequately addressed their concerns and thus gave the final ratings with two *weak accept* and one *borderline accept*. 

Specifically:

- **Reviewer b5x9** found the rebuttal to be clear with a comprehensive explanation and results on computational efficiency, large camera movements, warping errors, and more discussion on other questions.

- **Reviewer 4jq5** confirmed that the proposed method showed better semantic alignment, and the difference between linear interpolation and video frame interpolation mentioned in the paper was clearly explained.

- **Reviewer pLuV** felt the detailed explanation addressed the concerns and believed the remaining obscured parts could be clarified in the revised paper,

Given the positive feedback and support from all reviewers, the area chair recommends accepting this paper.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel approach for video editing in the scope of canonical image-based video editing. They argue the canonical images used in prior methods are unnatural, which degrades the performance a lot. To solve this problem, they propose to use a LoRA finetuned diffusion model to refine the unnatural canonical images to be natural. With this design, the performance is significantly improved. Experiments (comparison to baselines and ablation study) have demonstrated the effectiveness of this approach and the design. 

Currently, I tend to accept this paper but I still have many questions about the details of this method. I will make a final decision depends on the rebuttal.
strength_weakness: strength_weakness: - The writing is not clear enough. There are many miss details, which makes me confused. 
- The method should be slightly slower as they need to fine-tune the diffusion model for a scene. However, no detailed experiment is provided (only discussed in the conclusion). I believe providing such content can be insightful.
- The comment, "If the canonical image is not natural, it loses any editing value." is too rude, and wrong. Prior works have demonstrated the value of unnatural canonical images. As they train the models jointly, using unnatural canonical images is also reasonable (but not perfect). Please respect prior works.
questions: questions: - After you refine the canonical images, do you need to change the deformation fields? I think the contents are changed, so the correspondence should also be different, isn't it?

- Why do you use LoRA, instead of other PEFT finetuning strategies or even full finetuning?

- I am curious about the perceptual results of canonical images under large camera movement or significant changes. It is hard for me imagine such a canonical images. '

- Using multiple canonical images is definite an important design. Can you discuss on that or provide more ablation experiments in the final version?

- Some works like [1] also notice that the canonical images are flawed and use a different way to solve this problem (i.e., combine the original images and flawed canonical images), do we really have to make the canonical images natural?

- [2] only considers short-term warping errors, you can consider using warping error in [3]. Besides, why the warping error scale is different with [2].

[1] Blind Video Deflickering by Neural Filtering with a Flawed Atlas

[2] Learning blind video temporal consistency

[3] Blind video temporal consistency via deep video prior
limitations: limitations: The authors discuss the limitations in the conclusion part.
paper_summary: paper_summary: In this study, the authors propose a hybrid deformation field and diffusion prior update scheduling to generate high-quality canonical images that maintain temporal consistency in video editing.
strength_weakness: strength_weakness: In the Method section, there is a lack of detailed information to support the authors' claims:
  1) In 3.1 Hybrid Deformation Field for Deformation Modeling:
     - The structure of Residual MLP and Canonical MLP
     - How the hybrid deformation field composed of the homography matrix H and Residual deformation MLP is constructed
  2) In 3.2 Diffusion Prior:
     - The special token used for fine-tuning LoRA
  3) In 3.3 Separated NaRCan:
     - The criteria for dividing steps to add noise and the basis for determining the amount of noise to add

Completeness of the text:
  - The order of Figures 3 and 4 does not match the order mentioned in the text.
  - What does Fig.9(a) in line 123 refer to? Is it simply a typo?

Experiment:
  - The experimental results shown do not confirm whether the proposed method maintains the temporal consistency of the video through the generated canonical image.
questions: questions: - In Figure 2 (b) Video Representation, why is an accurate canonical image not generated despite applying the hybrid deformation field technique?

- Additionally, why is an accurate canonical image not generated even after updating the MLP with reconstruction loss?

- In 3.2 Diffusion Prior, what does the special token used for fine-tuning LoRA represent?

- Also, what criteria were used to divide the steps for adding noise, and on what basis was the amount of noise added determined?

- In 3.3 Separated NaRCan, how does linear interpolation differ from existing video frame interpolation methods?
limitations: limitations: - The authors clearly state the limitations of their research to inform the readers.
paper_summary: paper_summary: This paper proposes to solve the issue of temporal consistency in video editing. They propose a hybrid deformation field architecture, with a trainable homography matrix H(u, v, t) and residual deformation MLP, representing object variations throughout an entire scene. Combining it with the diffusion before the pipeline helps generate natural canonical images.
strength_weakness: strength_weakness: 1. The paper is not well-written and lacks clarity for the reader. For example, diffusion loss has played an important role in this paper, yet it is not introduced. Some of the expressions are vague. Like in question 1.  What's the additional "insight" you are looking for?
2. Things claimed are not reasoning enough like question 2.
3. Limitations are not addressed.
questions: questions: 1. What does it mean "this regularization term restricts the model’s ability to express itself without providing additional
insights." (line 118-119) What's the additional "insight" you are looking for?
2. How can diffusion prior ensure that the generated image is natural? The connection between the two is not clear to me. Is there any reason for that?
3. Is the noise and diffusion prior update scheduling decided manually? If so, how did you decide it?
limitations: limitations: No, the authors did not address the limitations.
I'm curious about the limitations of the motions that this method can handle. Can it deal with all kinds of videos, even with heavy motion blur? From the supplements, it is not clear to me.

Comments: title: The author's response address my concerns.
Post-rebuttal comments
Please let us know if you have additional questions after reading our response
Please let us know if you have additional questions after reading our response
comment: Dear Reviewer,

Thank you for your positive feedback on our rebuttal! We're glad our explanations addressed your concerns.

We appreciate your thorough review and remain open to any further questions.

Thank you! Authors
Dear Reviewer,

We sincerely thank you for evaluating our rebuttal. Your insightful comments have greatly improved our revised paper. We appreciate your expertise and dedication to the review process.

Best, Authors
Thanks for the detailed explanation. Given that the questions/concerns are addressed and the obscured parts will be clarified in the revised paper, I'll adjust my rating to borderline accept.
Thanks for your response regarding my concerns. I am glad to see that the author provides a comprehensive explanation and results on Computational efficiency, large camera movements, warping errors, and more discussion on other questions. The response is clear and decent. Hence, I will keep my rating and vote for weak acceptance of this paper.
Dear Reviewer,

Thank you for taking the time to review our rebuttal. We greatly appreciate your constructive feedback. Your suggestions have been invaluable in improving our revised paper.

Best regards, Authors
Thank you for the detailed explanation. I read the answer and referred to the existing papers. The author answered my questions by conducting additional performance evaluation using a new measurement method for temporal consistency. Through this, I can confirm that the method proposed by the author shows better semantic alignment. And the difference between linear interpolation and video frame interpolation mentioned in the paper is clearly explained. I am satisfied with the answer and have no follow-up questions. Therefore, I raise the score to weak accept.
Dear Reviewers,

We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further questions after reading our rebuttal.

We aim to address all the potential issues during the discussion period.

Thank you!

Best, Authors
Dear Reviewers,

We appreciate your reviews and comments. We hope our responses address your concerns. Please let us know if you have further questions after reading our rebuttal.

We aim to address all the potential issues during the discussion period.

Thank you!

Best, Authors