Title: Identifying and Solving Conditional Image Leakage in Image-to-Video Diffusion Model
Venue: NeurIPS 2024
Accepted: yes
Keywords: diffusion model

image-to-video diffusion model

video generation

video diffusion model
Abstract: Diffusion models have obtained substantial progress in image-to-video generation. However, in this paper, we find that these models tend to generate videos with less motion than expected. We attribute this to the issue called conditional image leakage, where the image-to-video diffusion models (I2V-DMs) tend to over-rely on the conditional image at large time steps. We further address this challenge from both inference and training aspects. First, we propose to start the generation process from an earlier time step to avoid the unreliable large-time steps of I2V-DMs, as well as an initial noise distribution with optimal analytic expressions (Analytic-Init) by minimizing the KL divergence between it and the actual marginal distribution to bridge the training-inference gap. Second, we design a time-dependent noise distribution (TimeNoise) for the conditional image during training, applying higher noise levels at larger time steps to disrupt it and reduce the model's dependency on it. We validate these general strategies on various I2V-DMs on our collected open-domain image benchmark and the UCF101 dataset. Extensive results show that our methods outperform baselines by producing higher motion scores with lower errors while maintaining image alignment and temporal consistency, thereby yielding superior overall performance and enabling more accurate motion control. The project page: \url{https://cond-image-leak.github.io/}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper argue that I2V-DMs tend to overly rely on the conditional image at large time steps, resulting in videos with less dynamic motion.\nTo address this, they propose using the KL divergence between the initial noise distribution and the actual marginal distribution to enhance the amplitude of motion while preserving video quality.\nBesides，the paper propose a time-dependent noise distribution for the conditional image during training process.\n\nSTRENGTHS:\nMotivation. The challenge of conditional image leakage presented in the paper is a credible issue and represents a significant academic problem.\n\nThe paper is exceptionally well-organized, making complex concepts easily digestible for the reader.\n\nWEAKNESSES:\n1. Claim. Regarding the conditional image and the diversity of content changes (referred to as 'motion' in this paper), it is, in fact, a matter of trade-off. The essence of image to video generation is to leverage the conditional image to enhance video quality. Therefore, the Proposition 1 and Training Strategy proposed by the authors are essentially fine-tuning techniques that favor the diversity of motion.\n\n2. Additionally, the foundation model of video generation guided only by the first frame inherently contains uncertainty in its output; it does not know whether the user wants to generate a video with significant or minimal motion. Some recent studies (last frame guided[1], motion control[2])have demonstrated that incorporating additional guidance signals can indeed achieve big and precise motion. \n\nParticularly, the works DragNUWA and MotionCtrl have shown that for the image-to-video generation task, it is still possible to generate significant motion. Therefore, the foundation model of SVD has the potential to directly generate substantial motion. In this context, the initial version of SVD, which could only generate minor motion, might be due to the lack of a specific condition or guidance signal for significant motion, rather than an inherent incapability. From a subjective standpoint, not all applications require substantial motion.\n\n[1] ToonCrafter: Generative Cartoon Interpolation\n[2] MotionCtrl: A Unified and Flexible Motion Controller for Video Generation\n[3] DragAnything: Motion Control for Anything using Entity Representation\n[4] DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory \n\n3. Experiments. The experiments appear to be somewhat confusing and not entirely convincing. \nRegarding the variables of the related training strategy displayed in Figure 8, the reviewer has not found the corresponding ablation study, which is a core contribution of this paper. Additionally, the author has not provided an ablation study on the 'M' of Proposition 1, such as testing performance changes between 0.6T and 1T, which is central to the first contribution. If the reviewer has overlooked it, please point it out.\n\nAdditionally, Figure 9 also appears to be somewhat confusing. Was a user study not conducted on the Analytic-Init? What does '-tune' signify—does it mean finetuning (training strategy)? If so, why does it appear in the inference phase?\n\n--------------------\nAfter reading the author’s and other reviewers' feedback, I have adjusted my score to 5.\n\nQUESTIONS:\nCan the authors provide more insights into the selection of hyperparameters (Line 230 Implementation Details) for the time-dependent noise distribution?\n\nLIMITATIONS:\nyes, the authors have discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper points out that existing image-to-video diffusion models can lead to videos without significant/desired amount of motion. Some evidence is presented to show that this is due to what the authors call \"conditional image leakage\" where the model places too much emphasis on the conditional image and all the generated frames look too similar to the conditional image. Two strategies are proposed to overcome this issue. The inference strategy starts the reverse diffusion process at an earlier time and uses improved initial noise distribution parameters using a closed-form expression to match the noise parameters at training time. The training strategy develops a time-dependent noise distribution to have higher noise at larger t to place lesser emphasis on the conditioning image for larger t values. Experiments indicate improved video generation using either or both of these strategies.\n\nSTRENGTHS:\n1. The methods presented in the paper are well-motivated and make intuitive sense (although there appear to be some technical issues as noted in the weaknesses below).\n\n2. Results: The quantitative results show clear improvements over baseline. The video examples submitted also show clear improvements over previous methods\n\n3. Relationship to existing works: the authors have done a good job presenting related works carefully and how existing methods may also inherently overcome some issues, if not all.\n\nWEAKNESSES:\n1. Issues with derivation: For deriving the expression of better noise initialization, the authors present their proof in Appendix A. The second and third terms of (7) are considered constants for the purpose of optimizing the noise distribution. I don't follow this and think there may be a technical error here. It is written that since $q_M(X_M)$ and $p_M(X_M)$ are independent, the entropy $H(\\mathcal{N}(X_M; \\mathbf{\\mu}_q, \\Sigma_q))$ and $H(q_M(X_M))$ can be considered constants for optimizing the parameters $\\mathbf{\\mu}_q$ and $\\Sigma_q$. I don't think this is correct as these parameters influence both $H(\\mathcal{N}(X_M; \\mathbf{\\mu}_q, \\Sigma_q))$ and $H(q_M(X_M))$. Can the authors please explain this in detail?\n\n2. Quantifying motion: The authors say that with previous works the models could in principle just output the conditioning image several times and still lead to lower error or improved quality metrics like FID and IS. However, they still use the same metrics for their experiments. Perhaps quantifying amount of motion or even better, quantifying realism of motion directly is important. \n\n3. Figure 9 is not clear. What are orange and blue colors indicating. Please explain this figure in detail and how it shows that the proposed method is considered better by the user study. I see very little differences going from <method> to <method>-tune in the figure.\n\nQUESTIONS:\nNo additional questions.\n\nLIMITATIONS:\nYes, authors have addressed some limitations of their method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates and proposes solutions for the problem of “conditional image leakage” in image-to-video generation. The authors claim that existing image-to-video diffusion models over-rely on the conditional image at large diffusion timesteps when the inputs are too noisy, leading to static video output. To address this issue, the authors propose an inference-time generation process that skips those large timesteps. The authors also propose adding noise to the conditional image during training to mitigate conditional image leakage.\n\nSTRENGTHS:\nThis paper identifies the problem of conditional image leakage and shows that existing methods suffer from static motion at large diffusion timesteps. The two proposed solutions empirically boost FVD and IS on UCF101 and bring significant improvements to motion magnitude while maintaining a similar image alignment and temporal consistency in the user study. These show that the proposed approach is indeed useful in generating more dynamic videos.\n\nWEAKNESSES:\nThere are a lot of prior works [a, b, c] that deal with noise initialization in video diffusion models that are highly related to the problem of conditional image leakage and the proposed inference technique. There are, however, no discussions on the differences between these techniques. Notably, FreeInit [a] and FrameInit [c] perverse the low-frequency component of the initial image – which is at odds with the proposed theory. I believe an in-depth discussion and a comparison with these approaches would help to establish the thesis of this paper. Additionally, PIA [d] also allows motion control – comparing the proposed motion-adding techniques with PIA’s motion control would also be helpful.\n\n[a] FreeInit: Bridging Initialization Gap in Video Diffusion Models\n\n[b] Preserve Your Own Correlation: A Noise Prior for Video Diffusion Models\n\n[c] ConsistI2V: Enhancing Visual Consistency for Image-to-Video Generation\n\n[d] PIA: Your Personalized Image Animator via Plug-and-Play Modules in Text-to-Image Models\n\nSecondly, I am not entirely convinced by the authors’ explanation of conditional image leakage. The authors claim that models learn a shortcut by copying the initial image at large diffusion timesteps, leading to static video output. However, models like VideoCrafter and SVD do produce outputs of diverse motion of some degree, as shown in their papers. What are the authors’ explanations for these motions? Where are they coming from? Trying to copy the conditional image when the input is otherwise noise is likely the optimal solution and would be learned – but that is not a sufficient condition for static output.\n\nThirdly, how are evaluations normalized for motion magnitude? Methods like SVD and PIA have motion control. Since the evaluation is focused on “dynamic motion”, perhaps choosing a different motion parameter for these methods would lead to different results in the user study. How are those parameters chosen? Do baseline SVD, SVD after finetuning, and SVD with the inference trick interpret the same motion scale parameter in the same way? One way to validate this is to plot the input motion scale parameter against the output motion magnitude (e.g., optical flow) for parameter selection.\n\nQUESTIONS:\nWhat are the motion magnitudes of SVD in Figure 3 and Figure 7? \n\nAdding different noises to the conditioning image at different time steps requires the conditioning image to be embedded multiple times. How does this affect inference time?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper presents an approach for image-to-video (I2V) generation. The authors start with a conditional image leakage problem in existing works where the conditional image significantly influences the generation process thereby impacting the dynamism of the video. The authors propose an inference time and a training time strategy to mitigate the problem. The approach is evaluated on UCF101 and the authors' ImageBench dataset. The results are impressive.

STRENGTHS:
1. The authors identified a crucial problem of conditional image leakage with I2V generation. 

2. The authors proposed two strategies to address conditional leakage. An inference time strategy that can be readily applied to existing approaches. Another approach is applied during the training time.

3. The paper is well-organized and easy to follow. 

4. Authors conducted a user study for qualitative evaluation. The results are impressive.

WEAKNESSES:
1. The authors are encouraged to provide more details about the ImageBench dataset.

2. While performing the user study, what qualities are evaluated by the users?

3. How is the accuracy of the generated videos estimated corresponding to the text prompt?  Do the users rate the matching score of the generated video corresponding to the text prompt?

4. The realism of the generated videos is lacking. This is common for diffusion-based generated visual content.  Have the authors considered how to improve the realism of generated videos?

QUESTIONS:
Please address the comments in the weaknesses section.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14