Motion-aware Latent Diffusion Models for Video Frame Interpolation

Authors: zhilin  huang
yijie  yu
ling  yang
chujun  qin
bing  zheng
xiawu  zheng
zikun  zhou
yaowei  wang
wenming  yang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 02:50:44

Abstract: With the advancement of AIGC, video frame interpolation (VFI) has become a crucial component in existing video generation frameworks, attracting widespread research interest. For the VFI task, the motion estimation between neighboring frames plays a crucial role in avoiding motion ambiguity. However, existing VFI methods always struggle to accurately predict the motion information between consecutive frames, and this imprecise estimation leads to blurred and visually incoherent interpolated frames. In this paper, we propose a novel diffusion framework, motion-aware latent diffusion models (MADiff), which is specifically designed for the VFI task. By incorporating motion priors between the conditional neighboring frames with the target interpolated frame predicted throughout the diffusion sampling procedure, MADiff progressively refines the intermediate outcomes, culminating in generating both visually smooth and realistic results. Extensive experiments conducted on benchmark datasets demonstrate that our method achieves state-of-the-art performance significantly outperforming existing approaches, especially under challenging scenarios involving dynamic textures with complex motion.

Main paper: 
 introduction: 
 Video frame interpolation (VFI) aims to generate intermediate frames between two consecutive video frames. It is commonly used to increase the frame resolution, e.g., to produce slow-motion content [29] . VFI has also been applied to video compression [69] , video generation [21] and animation production [57] .Existing VFI methods [29, 31, 37, 48, 71] are mostly based on deep neural networks. The early deep learning methods always rely on 3D convolution [28] or RNNs [9] to model the contextual correlations of conditional neighboring frames. Inspired by the success of generative adversarial networks (GANs) in image synthesis, more recent attempts have been made to develop video frame interpolation methods by incorporating the adversarial loss. Owing to the remarkable generative capabilities of GANs, these methods exhibit significant efficacy in predicting interpolated video frames that The encoder and decoder enable projection between image and latent spaces, and the diffusion processes take place in the latent space. 𝐼 2𝐸 denotes image-to-event generator [78] which have capability of generating event volume by taking two continuous frames as input. And 𝑚 denotes motion hints extracted from 𝐼 2𝐸.possess natural and consistent content, holding the state-of-the-art in the literature. However, these deep learning-based methods for VFI always tend to generate unrealistic texture, artifacts and lowperceptual results. The reason is that the primary contributor to the optimization objective -and consequently the final performance of the model -remains the L1/L2-based distortion loss between their outputs and the ground-truth interpolated frames [15] . This type of loss may not accurately reflect the perceptual quality of the interpolated videos. Consequently, [14] indicates that current techniques, despite attaining elevated PSNR scores, often fall short in terms of perceptual quality, particularly in demanding situations characterized by dynamic textures and complex motion patterns. Recently, de-noising diffusion probabilistic models (DDPMs) have been gaining widespread interest and have risen to prominence as state-of-the-art solutions across various areas [20, 21, 23, 24, 61, 72] . These diffusion models have demonstrated exceptional capabilities in creating realistic and perceptually-enhanced images and videos, reportedly outperforming other generative models. [15, 67] are among the early methods that have explored the application of diffusion models for VFI tasks. Specifically, they address the VFI tasks as a form of conditional image generation by taking neighboring frames into the de-noising network for the target interpolated frame generation.However, these methods fail to explicitly model the inter-frame motions between the interpolated frames and the given neighboring conditional frames, a crucial factor in preventing the generation of blurred interpolated frames due to motion ambiguity. This is particularly important in complex dynamic scenes that involve intricate motions, occlusions, or abrupt changes in brightness.To tackle these challenges, in this paper, we propose a novel latent diffusion framework, Motion-Aware latent Diffusion model (MADiff) for the video frame interpolation task. Specifically, we follow [15] to build upon MADiff by adopting recently proposed latent diffusion models (LDMs) [55] . LDMs consist of an autoencoder that maps images into a latent space and a de-noising U-net, which carries out the reverse diffusion process within that latent space, forming the foundation of our framework. To incorporate the interframe motion priors between given conditional neighboring frames with the interpolated frames into MADiff, we propose a novel vector quantized motion-aware generative adversarial network, named VQ-MAGAN. In particular, we initially utilize a pre-trained Event-GAN [78] to predict the event volume which reflects the pixel-level intensity changes between two consecutive frames. Subsequently, the event volumes between the interpolated frame and the two neighboring conditional frames are employed as motion hints to enhance image reconstructions within the decoder of VQ-MAGAN. In this design, VQ-MAGAN possesses the capability to predict the target interpolated frame by aggregating contextual details from the given neighboring frames under the guidance of inter-frame motion hints. Furthermore, for the de-noising process in LDMs, we also incorporate motion hints between the interpolated frame and the two neighboring frames as additional conditions.During the training process of both VQ-MAGAN and the denoising U-net, we directly utilize the ground-truth interpolated frame for extracting inter-frame motion hints between interpolated frame and neighboring conditional frames. Since the ground-truth interpolated frame is unknown during the sampling process of LDM, extracting motion hints between the interpolated frame and the conditional neighboring frames is not feasible. To eliminate the discrepancy of motion hints extraction between the sampling phase and the training phase, making the motion hints in the sampling process available, we propose a novel motion-aware sampling procedure (MA-Sampling). Specifically, during the sampling process, we use the coarse interpolated frame predicted in the previous time step to extract inter-frame motion hints in conjunction with the conditional neighboring frames. The extracted motion hints are then fed into both VQ-MAGAN and the de-noising U-net for the prediction of the interpolated frame in the current time step. By refining the interpolated frames progressively, our MADiff can effectively integrate the inter-frame motion hints into the sampling process, resulting in visually smooth and realistic frames.Extensive experiments on various VFI benchmark datasets, encompassing both low and high resolution content (up to 4K), demonstrate that our MADiff achieves the state-of-the-art performance significantly outperforming existing approaches, especially under challenging scenarios involving dynamic textures with complex motion. Our contributions are summarized as follows:• We propose a novel vector quantized motion-aware generative adversarial network, named VQ-MAGAN, which fully incorporates the inter-frame motion hints between the target interpolated frame and the given neighboring conditional frames into the prediction of the interpolated frame. • We propose a novel motion-aware sampling procedure, named MA-Sampling, to eliminate the discrepancy of motion hints extraction between the sampling phase and the training phase, making the extraction of motion hints in the sampling process feasible and refine the predicted interpolated frames progressively. • We demonstrate, through quantitative and qualitative experiments, that the proposed method achieves the state-of-theart performance outperforming existing approaches. 
related work 2.1 image-to-event generation: 
 Event cameras are a novel bio-inspired asynchronous sensor [18, 41] with advantages such as high dynamic range, high temporal resolution, and low power consumption [4, 18, 41] . Event cameras have the potential to provide solutions for a wide range of visually challenging scenarios and has already found extensive applications [2, 5, 22, 33, 38-40, 46, 54, 63, 64, 70, 77] . However, acquiring event stream data is expensive and requires specific devices. Recently, several studies have exploited the event simulation from the standard camera, i.e. simulating the event stream from continuous images or video sequences. Kaiser et al. [30] generate events by simply applying a threshold on the image difference. A positive or negative event is generated depending on the pixel's intensity difference. Pix2NVS [3] computes per-pixel luminance from conventional video frames.The technique generates synthetic events with inaccurate timestamps clustered to frame timestamps. [76, 78] propose a GAN-based method to generate the event volume by simply taking two continuous image frames as input. By modeling the correlation between continuous frames and event volumes, image-to-event generation models can capture inter-frame motion hints, which can serve as auxiliary guidance for the VFI task. In our MADiff, we directly utilized the pre-trained EventGAN [78] to extract motion hints for guiding the interpolation process.Current approaches to video frame interpolation (VFI) that leverage deep learning can typically be divided into two main categories: those that are based on flow estimation and those that rely on kernel methods. Existing flow-based methods [29, 35, 42, 44, 47, 48, 50, 51, 56, 71] primarily generate intermediate frames by analyzing the pixel motion between consecutive frames. Specifically, these methods use optical flow estimation to capture the movement of objects within the scene, and then synthesize intermediate frames based on these estimations. These methods can be founded on various assumptions and algorithms, such as assuming uniform motion within the scene or employing machine learning models to predict the optical flow field, thereby enhancing the accuracy and visual quality of the interpolated frames. On the other hand, Existing kernel-based video frame interpolation methods [7, 8, 11, 16, 37, 49] typically utilize kernel functions to estimate the relationships between pixels across different frames, generating intermediate frames by performing a weighted average of pixel values based on known frames. These methods can flexibly adapt to various motion patterns in the scene by selecting appropriate kernel sizes and shapes, thus effectively smoothing the visual transitions between frames while preserving details. In addition to these two main categories, there are also some methods integrate flow-based and kernel-based approaches to better synthesis performance [10, 31] . Meanwhile, several works attempt to introduce the event stream as a supplemental conditions for guiding accurate models between the neighboring frames and interpolated frames. However, almost event-based methods require taking the accurate ground-truth event streams as inputs which is inconvenient for practical application. In contrast, our MADiff directly utilizes off-the-shelf pre-trained image-to-event models for providing the motion hints in the subsequent interpolated frame generation process. Besides, our MADiff refines the target interpolated frame in a progressive manner, which is quite different from previous VFI methods that predict interpolated frame in a one-shot manner.Recently, denoising diffusion probabilistic models (DDPMs) have gained increasing attention and emerged as a new state-of-the-art in several areas of computer vision. [15, 67] are the most early methods exploiting the application of DDPMs for VFI tasks. Compared to previous VFI methods, diffusion-based methods show satisfactory perceptual performance, especially on dynamic textures with complex motions. However, existing diffusion-based methods simply define the VFI task as a conditional image generation task by considering the neighboring frames as conditions, neglecting to explicitly model inter-frame motions which are a crucial factor for generating realistic and visually smooth interpolated frames. Compared with these methods, we propose a novel diffusion framework, MADiff, to effectively incorporate motion hints from existing motion-related models into diffusion models in a progressive manner. 
preliminary 3.1 representation of event volume: 
 Each event e can be represented by a tuple (𝑥, 𝑦, 𝑡, 𝑝), where 𝑥 and 𝑦 represent the spatial position of the event, 𝑡 represents the timestamp, and 𝑝 = ±1 indicates its polarity.As described in [78] , the event are presented for easily processing: events are scattered into a fixed size 3D spatio-temporal volume, where each event, (𝑥, 𝑦, 𝑡, 𝑝) is inserted into the volume, which has 𝐵 = 9 temporal channels, with a linear kernel:EQUATIONThis retains the distribution of the events in 𝑥-𝑦-𝑡 space, and has shown success in a number of tasks [6, 53, 79] .Since the event volume generated by EventGAN [78] are the concatenation of event volumes of different polarity along the time dimension, the final event volume which is strictly non-negative. In our MADiff, we directly utilize the event volume generated by EventGAN as the inter-frame motion hints.Latent diffusion models (LDMs) [55] is variant of de-noising diffusion probabilistic models which executes the de-noising process in the latent space of an autoencoder, namely E (•) and D (•), implemented as pre-trained VQ-GAN [17] or VQ-VAE [65] . Compared with executing de-noising process in the pixel-level data, LDM can reduce computational costs while preserving high visual quality.For the training of LDM, the given latent code 𝑧 for a randomly sampled training image 𝑥 is converted to noise with a Markov process defined by the transition kernel:EQUATIONwhere 𝑡 = 1, 2, • • • ,𝑇 , 𝑧 0 = 𝑧, and 𝛼 𝑡 is a hyper-parameter that controls the rate of noise injection. When the amount of noise is sufficiently large, 𝑧 𝑇 becomes approximately distributed according to N (0, I). In order to convert noise back to data for sample generation, the reverse diffusion process is estimated by learning the reverse transition kernel:EQUATIONand then take it as an approximation to 𝑞(𝑧 𝑡 -1 | 𝑧 𝑡 ). Following Ho et al. [20] , 𝜇 𝜃 (𝑧 𝑡 ) is parameterized with a neural network 𝜖 𝜃 (𝑧 𝑡 , 𝑡) (called the score model [59, 60] ) and Σ 𝜃 is fixed to be a constant.The score model can be optimized with de-noising score matching [26, 66] . During sample generation, within each time step, the denoising U-net initially predicts the ẑ0 . Finally, the decoder of VQ-GAN or VQ-VAE generates the image Î0 from the de-noised latent representation ẑ0 , disregarding any contextual information. 
methods 4.1 motion hints extraction: 
 In MADiff, we utilize the pre-trained EventGAN [78] to capture inter-frame motion hints between the interpolated frame with the conditional neighboring frames. Specifically, given the interpolated frame 𝐼 0 ∈ R 𝐻 ×𝑊 ×3 and two conditional neighboring frames𝐼 -1 , 𝐼 +1 ∈ R 𝐻 ×𝑊 ×3, where 𝐼 -1 denotes the previous frame and 𝐼 +1 denotes the next frame. The motion hints extraction process are formulated as follows:EQUATIONEQUATIONwhere 𝑓 𝐼 2𝐸 (•) is the pre-trained EventGAN, 𝑚 𝑖 𝑗 denotes the extracted motion hints from the frame 𝑖 to the frame 𝑗. In practice, we directly use the predicted event volume 𝐸𝑉 𝑖 𝑗 ∈ R 𝐻 ×𝑊 × (2×𝐵) as 𝑚 𝑖 𝑗 . Besides, MADiff we proposed is a general framework which can incorporate different motion-related models easily. More details please refer to Section 5.5.2.Implementation Details. For the consideration of motion information between the interpolated frame and neighboring frames is crucial for the VFI task, we propose a novel VQ-GAN, namely VQ-MAGAN as presented in Figure 2 .The encoder E produces the latent encoding 𝑧 0 = E (𝐼 0 ) by taking the given ground-truth target frame 𝐼 0 ∈ R 𝐻 ×𝑊 ×3 as the input, where 𝑧 0 ∈ R 𝐻 𝑓 × 𝑊 𝑓 ×3 , and 𝑓 is a hyper-parameter. In practice, we set 𝑓 = 32 following [15] .Then the decoder D reconstruct target frame Î0 by taking 𝑧 0 and the feature pyramids 𝜙 -1 , 𝜙 +1 which are extracted by E from two neighboring frames 𝐼 -1 , 𝐼 +1 following [15] . Moreover, we utilize the motion hints extractor to capture the inter-frame motion hints 𝑚 -1 0 and 𝑚 0 +1 between the ground-truth target frame 𝐼 0 with the neighboring frames 𝐼 -1 and 𝐼 +1 . And then we take the 𝑚 -1 0 and 𝑚 0 +1 as an additional guidance for the layer-wisely contextual aggregation in the decoder D through a motion-aware warp (MA-Warp) module. Specifically, for the feature ℎ 𝑙 0 ∈ R 𝑈 ×𝑉 ×𝐶 of interpolated frame and the motion hints 𝑚 -1 0 , 𝑚 0 +1 ∈ R 𝐻 ×𝑊 × (2×𝐵) , the MA-Warp in 𝑙-th decoder layer firstly reshape the motion hints to the resolution of 𝑈 ×𝑉 , obtained 𝑚 𝑙 -1 0 , 𝑚 𝑙 0 +1 ∈ R 𝑈 ×𝑉 × (2×𝐵) . Then for each motion hints, a 2-channel offset map Ω 𝑙 -1 0 and Ω 𝑙 0 +1 , respectively, which reflects the pixel-level feature correlations from the neighboring frame to the target interpolated frame is generated through a learnable neural networks: 𝑓 Ω (•)EQUATIONEQUATIONAfter that, a warp function 𝑓 𝑤𝑎𝑟𝑝 (•) proposed in [25] is introduced and serves as a aggregation mechanism:EQUATIONEQUATIONThe MA-Warp also generates a gate map 𝑔 ∈ [0, 1] 𝑈 ×𝑉 ×1 to account for occlusion [29] , and a residual map 𝛿 ∈ R 𝑈 ×𝑉 ×𝐶 to further enhance the performance:EQUATIONEQUATIONEQUATION)where both 𝑓 𝑔 (•) and 𝑓 𝛿 (•) are learnable neural networks, h𝑙 0 is the output of MA-Warp in 𝑙-th decoder layer. By hierarchically applying MA-Warp in the decoder layer, VQ-MAGAN is able to fully utilize the motion hints for accurately aggregating pyramid contexts from neighboring frames. Compared with VQ-FIGAN proposed in [15] , our VQ-MAGAN has capability of incorporating the interframe motions between the target interpolated frames with the conditional neighboring frames.Training VQ-MAGAN. For the training of VQ-MAGAN, we follow the original training settings of VQGAN in [17, 55] , where the loss function consists of an LPIPS-based [75] perceptual loss, a patch-based adversarial loss [27] and a latent regularization term based on a vector quantization (VQ) layer [65] . Particularly, we use the ground-truth interpolated frame to extract motion hints with given conditional neighboring frames.Since the ground-truth target frame are provided for extracting motion hints during the training process of VQ-MAGAN, the reconstruction task may becomes much easier for VQ-MAGAN, potentially degrading the performance of reconstruction in the inference stage. To address this issue, we only utilize the motion hints with a probability of 0.5 during the training stage to assist in the reconstruction process of VQ-MAGAN. The pseudo code is provided in the Appendix. While we replace the ground-truth interpolated frame with the predicted one from the previous time step during the sampling procedure as described in Section 4.4.Implementation Details. The encoder of trained VQ-MAGAN allows us to access a compact latent space in which we perform forward diffusion by gradually adding Gaussian noise to the latent 𝑧 0 of the target frame 𝐼 0 according to a pre-defined noise schedule, and learn the reverse (de-noising) process to perform conditional generation. Moreover, compared with previous methods, we additionally incorporate the dynamic inter-frame motion hints into the denoising process. To this end, we adopt the noise-prediction parameterization [20] of diffusion models and train a de-noising U-Net by Figure 2 : The architecture of the vector quantized motion-aware generative adversarial network, VQ-MAGAN. In practice, the motion extractor is image-to-event generator [78] , 𝑚 𝑖 𝑗 denotes the inter-frame motion hints between frame 𝑖 and 𝑗.minimizing the re-weighted variational lower bound on the conditional log-likelihood log 𝑝 𝜃 (𝑧 0 |𝑧 -1 , 𝑧 +1 , 𝑚 -1 0 , 𝑚 0 +1 ),where 𝑧 -1 and 𝑧 +1 are the latent representations of the two conditional neighboring frames, and 𝑚 𝑖 𝑗 denotes the motion hints between frame 𝑖 and 𝑗 as described in Section 4.1. Specifically, the de-noising U-Net 𝜖 𝜃 takes as input the noisy latent representation 𝑧 𝑡 for the target frame 𝐼 0 (sampled from the 𝑡-th step in the forward diffusion process of length 𝑇 ), the diffusion step 𝑡, as well as the conditioning latent representations 𝑧 -1 , 𝑧 +1 for the neighboring frames 𝐼 -1 , 𝐼 +1 . It is trained to predict the noise added to 𝑧 0 at each time step 𝑡 by minimizingEQUATIONwhere 𝑡 ∼ U (1,𝑇 ). The derivation and full details of the training procedure of 𝜖 𝜃 are provide in Appendix. Intuitively, the training is performed by alternately adding a random Gaussian noise to 𝑧 0 according to a pre-defined noising schedule, and having the network 𝜖 𝜃 predict the noise added given the step 𝑡, conditioning on 𝑧 -1 , 𝑧 +1 and 𝑚 𝑖 𝑗 .Training of De-noising U-net. It is worth noting that during the training of de-noising U-net, we directly utilize the ground-truth interpolated frame for extracting inter-frame motion hints which is the same as the extraction process in the training of VQ-MAGAN.As described above, both VQ-MAGAN and de-noising U-net are conditioned on the inter-frame motion hints extracted from the interpolated frame and the conditional neighboring frames. During the training stage of both VQ-MAGAN and de-noising U-net we directly using the ground-truth interpolated frame for the motion hints extraction in a teach-forcing manner. However, the interpolated frame is unknown in the sampling phase, rendering the extraction of inter-frame motion hints between the interpolated frame and neighboring frames infeasible. While the motion hints directly extracted from given neighboring frames are often inaccurate and cannot provide sufficient guidance, leading to sub-optimal performance as described in Table 5 . For eliminating this discrepancy of motion hints extraction between the training and sampling phase, making the motion hints in the sampling process available, we propose a novel MA-Sampling.Before introducing MA-Sampling, we provide a review of the sampling process within traditional LDM for VFI tasks [15] : Within each time step, firstly the de-noising U-net 𝜖 𝜃 predicts the noise ε conditioned on the latent representations 𝑧 -1 , 𝑧 +1 of neighboring frames 𝐼 -1 , 𝐼 +1 . Then ẑ0|𝑡 is obtained as follows:EQUATION)where 𝜖 𝜃 (•) is the de-noising U-net, ẑ0|𝑡 denotes the predicted 𝑧 0 at time step 𝑡 (particularly, we denote ẑ0|1 as ẑ0 for simplification), ẑ𝑡 is the noisy latent representation of the predicted ẑ0|𝑡+1 obtained at previous time step 𝑡 + 1 during the sampling process. And ẑ𝑡 can be calculated by using ε and relevant parameters of the pre-defined forward process as eq (3) . Finally, the decoder of VQ-GAN produces the image Î0 from ẑ0|1 with the help of feature pyramids 𝜙 -1 , 𝜙 +1 extracted by the encoder E from 𝐼 -1 , 𝐼 +1 . Compared with the traditional sampling process mentioned above, our MA-Sampling has capability of incorporating accurate motion hints between the interpolated frame and the neighboring frames for progressively refining the predicted target frame. Specifically, at time step 𝑡, firstly the de-noising U-net 𝜖 𝜃 predicts the noise ε conditioned on the latent representations 𝑧 -1 , 𝑧 +1 of neighboring frames and additional motion hints m0 +1|𝑡 +1 , m-1 0|𝑡 +1 . Then ẑ0|𝑡 is obtained as follows:EQUATION)where m-1 0|𝑡 +1 , m0 +1|𝑡 +1 are extracted from the predicted interpolated frame Î0|𝑡+1 and neighboring frames 𝐼 -1 , 𝐼 +1 :EQUATIONAnd ẑ𝑡 -1 can be calculated using ε and relevant parameters of the pre-defined forward process as sampling process of previous methods eq (3) . Particularly, at time step 𝑇 , motion hints m-1 0|𝑇 +1 and m0 +1|𝑇 +1 are both replaced with empty features O ∈ R 𝐻 ×𝑊 × (2×𝐵) . Finally, the decoder D produces the interpolated frame Î0|1 (for simplification we denote it as Î0 ) from the de-noised latent representation ẑ0|𝑡+1 i.e. ẑ0 , by fully considering feature pyramids 𝜙 -1 , 𝜙 +1 extracted by the encoder E from 𝐼 -1 , 𝐼 +1 as contexts under the guidance of motion hints m-1 0|1 , m0 +1|1 . Full details and pseudo code are provided in the Appendix. 
experiments 5.1 implementation details: 
 Regarding the diffusion processes, following [15] , we adopt a linear noise schedule and a codebook size of 8192 for vector quantization in VQ-MAGAN. We sample from all baseline diffusion models with the DDIM [58] sampler for 200 steps. We also follow [15] to train the VQ-MAGAN using the Adam optimizer [34] and the de-noising U-net using the Adam-W optimizer [43] , with the initial learning rates set to 10 -5 and 10 -6 respectively. Single NVIDIA V100 GPU were used for all training and evaluation.In our experiments, we strictly follow the dataset configuration specified in [15] for both training and evaluation.For training MADiff, we utilize the widely adopted Vimeo90k dataset [71] . And additional samples from the BVI-DVC dataset [45] are adopted to better evaluate our VFI methods across a broader range of scenarios. The overall training set comprises 64,612 frame triplets from Vimeo90k-septuplets and 17,600 frame triplets from BVI-DVC, utilizing only the central three frames. For data augmentation, we randomly crop 256 × 256 patches and perform random flipping and temporal order reversing following [15] . For evaluation, MADiff is validated on four widely recognized VFI benchmarks, including Middlebury [1] , UCF-101 [62] , DAVIS [52] and SNU-FILM [10] . The resolutions of these test datasets vary from 225 × 225 up to 4K, covering various levels of VFI complexity.. Following [15] , we adopt a perceptual image quality metric LPIPS [75] , FloLPIPS [12] for performance evaluation. These metrics exhibit a stronger correlation with human assessments of VFI quality in contrast to the traditionally utilized quality metrics, such as PSNR and SSIM [68] . We also evaluate FID [19] which measures the similarity between the distributions of interpolated and ground-truth frames. FID was previously used as a perceptual metric for video compression [74] , enhancement [73] and colorization [32] . We also provide benchmark results based on PSNR and SSIM in the Appendix, noting that these are limited in reflecting the perceptual quality of interpolated content [14] and are therefore not the focus of this paper.MADiff was compared against 11 recent SOTA VFI methods, including BMBC [50] , AdaCoF [37] , CDFI [16] , XVFI [56] , ABME [51] , IFRNet [35] , VFIformer [44] , ST-MFNet [13] , FLAVR [31] , MCVD [67] and LDMVFI [15] . It it noted that MCVD and LDMVFI are diffusion-based VFI method.Table 1 shows the performance of the evaluated methods on the Middlebury, UCF-101 and DAVIS test sets. It can be observed that MADiff consistently outperforms all the other VFI methods including both non-diffusion or diffusion-based methods. Moreover, we evaluate the performance on the four splits of the SNU-FILM dataset (the average motion magnitude increases from Easy to Extreme), as summarized in Table 2 , which further demonstrates the superior perceptual quality of MADiff, especially in the scenes including complex motions (SNU-FILM-Hard and Extreme). Notably, the performance of other diffusion-based VFI methods, namely MCVD and LDMVFI, are generally unsatisfactory, which suggests that simply defining the VFI task as a conditional image generation task, without explicitly modeling inter-frame motions, may not be adequate for generating realistic and visually smooth results. As presented in Table 1 , the number of parameters in MADiff is large. This is because we adopted the existing de-noising U-net [55] designed for generic image generation following [15] and introduced adapters for fusing motion hints into the VQ-MAGAN and de-noising U-net.Figure 3 shows the comparison between example frames interpolated by MADiff with the competing methods. We can observe that, non-diffusion-based methods (BMBC [50] , VFIformer [44] , IFRNet [35] and ST-MFNet [13] ) tend to predict blurry results due to the L1/L2-based distortion loss. On the contrary, diffusion-based VFI method (LDMVFI [15] ) is able to generate more realistic results, while it also produces several artifacts leads by motion ambiguity. By incorporating inter-frame motion hints as the guidance, our MADiff has capability of predicting realistic results.In this section we experimentally validate and study the main components including the VQ-MAGAN and MA-Sampling in MADiff. 3 . By comparing Exp0, Exp1 and Exp2, we can observe that simply introducing MA-Sampling can only brings slightly performance improvements, while simply introducing motion hints to guide the contexts warping process in VQ-MAGAN can significantly improve the performance of MADiff. Moreover, the results Table 5 : Effectiveness of extracting motion hints between interpolated frames with neighboring frames instead of directly extracting motion hints from two continuous neighboring frames. For each column, we highlight the best result in red and the second best in blue.both VQ-MAGAN and MA-Sampling. The results indicate that our MADiff can effectively integrate various types of motion hints, thereby enhancing the performance of video frame interpolation. This demonstrates that our MADiff constitutes a flexible framework, allowing for the straightforward replacement of the motion hint extractor to incorporate diverse motion hints into diffusion models for the VFI task. Moreover, we conduct two additional experiments to ascertain the superiority of motion hints between the interpolated frame and the neighboring frames (referred to as Dynamic Motion Hints in Table 5 ) over motion hints between two neighboring frames (referred to as Global Motion Hints in Table 5 ). The results presented in Table 5 demonstrate that motion hints extracted between the interpolated frame and the neighboring frames offer more precise guidance and help to mitigate motion ambiguity in the VFI task, compared to motion hints directly extracted from two consecutive neighboring frames. 
conclusion: 
 In this paper, for the VFI task, we propose a novel motion-aware latent diffusion model (MADiff) which can fully leverage rich interframe motion priors from readily available and pre-trained motionrelated models during the generation of interpolated frames. Specifically, our MADiff consist of a vector quantized motion-aware generative adversarial network (VQ-MAGAN) and a de-noising U-net. And VQ-MAGAN is adept at aggregating contextual details under the guidance of inter-frame motion hints between the interpolated and given neighboring frames. Additionally, we propose a novel motion-aware sampling procedure (MA-Sampling) that progressively refines the predicted frame throughout the sampling process of the diffusion model. Comprehensive experiments conducted on benchmark datasets demonstrate that our MADiff achieves the state-of-the-art performance, significantly surpassing existing approaches, particularly in scenarios characterized by dynamic textures and complex motions. 
 
 Appendix:  


Decision: This paper presented a method for video frame interpolation (VFI) that uses motion priors between conditional neighbouring frames in a diffusion framework. The proposed diffusion sampling process that establishes motion prior could be a contribution to the community. This paper received comments from three reviewers. The strengths of this work include: the proposed idea of the motion-aware diffusion models, extensive experiments (including ablation studies) to validate the effectiveness of the proposed method, and the writing. There are also some limitations: the evaluation of the quality of the results, unclear description of some claims and statements, computational complexity, and performance inconsistency across different data. All three reviewers recommended positive scores, with two borderline accept and one weak accept.

Considering the above points, the key idea of this work might be of interest to a group of audience in ACM MM and worth seeing and discussing. As a result, the AC recommends Accept.

Reviews: review: paper_summary: paper_summary: The paper presents a novel approach to video frame interpolation (VFI) using Motion-Aware Latent Diffusion models (MADiff). It aims to address issues of motion ambiguity and imprecise motion estimation in existing VFI methods, which often result in blurred and visually incoherent interpolated frames. The proposed MADiff framework incorporates motion priors between neighboring frames to refine intermediate outcomes progressively, achieving state-of-the-art performance on benchmark datasets, particularly in challenging scenarios involving dynamic textures with complex motion.
strength_weakness: strength_weakness: 1) The introduction of motion-aware latent diffusion models is reasonable and somewhat innovative.
2) Extensive experiments conducted on various benchmark datasets, including high-resolution content up to 4K, demonstrate its performance improvements over the baselines.
3) The writing is good and the proposed method is detailed and logical.
limitations: limitations: 1) There is a heavy reliance on quantitative metrics (LPIPS, FloLPIPS, FID) for evaluating performance. These metrics, while useful, do not fully capture perceptual quality, especially in complex motion scenarios. The paper could benefit from more diverse evaluation metrics, including user studies or perceptual quality assessments.
2) The proposed method appears to be computationally expensive due to the use of latent diffusion models and VQ-MAGAN.
3) The failure cases or limitations of the proposed method are suggested to be analyzed.
paper_summary: paper_summary: This paper proposes MADIFF method to conduct video frame interpolation by incorporating motion priors among frames. The method establishes motion prior via a diffusion sampling procedure. However, additional clarification and comparisons may be needed for better understanding and evaluation.
strength_weakness: strength_weakness: 1. This paper explores the denoising diffusion probabilistic model, a recent emerging technology, that is known for its ability to yield realistic and perceptually-optimized images.
limitations: limitations: 1. In Figure 1, the mixed usage of subscripts with different meanings and no descriptions could introduce confusion. For example, the subscripts for I and z seem to have different meanings, i.e., some of them refer to frame index, while some of them refer to iterations of the diffusion process. Besides, this figure reminds me of Figure 1 of LDMVFI (AAAI 24), an earlier video frame interpolation method based on denoising diffusion probabilistic model. The proposed method is described as “significantly outperforming existing approaches”, but its qualitative results in Figure 6 seem to be very close to those of LDMVFI.
2. How to acquire z_t? Would the LDMs introduce randomness to the interpolation result? To clearly demonstrate the function of VQ-MAGAN in the diffusion process, is it possible to incorporate VQ-MAGAN in Figure 1?
3. How does the results compare to “Extracting motion and appearance via inter-frame attention for efficient video frame interpolation (CVPR 2023)”? The model size of the proposed model seems to be relatively large, e.g. 10 times larger compared with the existing methods.
paper_summary: paper_summary: The paper introduces an LDM-based method for video frame interpolation, addressing the challenge of accurately predicting motion information between consecutive frames. The proposed method incorporates motion priors between neighboring frames and refines intermediate outcomes through a diffusion sampling procedure. Experiments show the effectiveness of the proposed method.
strength_weakness: strength_weakness: 1.	The idea of introducing motion hints is interesting and will benefit the community.
2.	Extensive experiments demonstrate the effectiveness of proposed method.
3.	Ablation studies verify that each component is an optimal solution.
limitations: limitations: 1.	Misleading and overstatement of contribution. The authors claim that the proposed method achieves SOTA performance in complex scenarios involving dynamic textures. However, this assertion is potentially misleading due to the omission of the VFITex dataset in their evaluation. VFITex is a widely used and challenging dataset specifically designed to test video frame interpolation methods on various dynamic textures. The absence of results on VFITex leads to an incomplete comparison, particularly with LDMVFI, which includes corresponding results on this dataset. Consequently, the claim of SOTA performance is difficult to verify and may mislead readers. It is recommended to include evaluations on the VFITex dataset to provide a more comprehensive and accurate comparison of performance.
2.	Difference with other motion-aware works. It is necessary to add a more thorough discussion for polishing clear motivation (i.e., comparison with existing motion-aware techniques in video generation). Incorporating motion information is a well-established approach in this field, as evidenced by previous works [1,2]. Additionally, there are several motion-aware diffusion models (DM) [3,4] that, while not specifically designed for video frame interpolation, are relevant and should be considered. The authors should discuss these methods in the introduction or related work sections to provide a clearer context for their contributions.
3.	Event across datasets. Leveraging event-based data is a promising approach due to its dense and efficient nature. However, there are concerns regarding the performance consistency of EventGAN across different datasets. EventGAN, trained on image pairs to simulate events for existing image datasets, may not perform uniformly well across all evaluated datasets. This inconsistency is evident in the results presented in Table 4, where the proposed components using motion hints do not consistently improve performance. For instance, the FID gain (with/without MA-sampling) is -0.046 for Middlebury and 0.469 for DAVIS, while for UCF101, the FID gain (with/without VQ-MAGAN) is 0.388. That would be far more interesting to analyze the inconsistency across datasets by some visualization and discussion. And this can motivate the following works.

[1] Decomposing Motion and Content for Video Generation, CVPR18

[2] MOSO: Decomposing Motion, Scene and Object for Video Prediction, CVPR23

[3] Distribution Extrapolation Diffusion Model for Video Prediction, CVPR24

[4] Conditional Image-to-Video Generation with Latent Flow Diffusion Models, CVPR23