Frame Interpolation with Consecutive Brownian Bridge Diffusion

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 15:41:37

Abstract: Recent work in Video Frame Interpolation (VFI) tries to formulate VFI as a diffusion-based conditional image generation problem, synthesizing the intermediate frame given a random noise and neighboring frames. Due to the relatively high resolution of videos, Latent Diffusion Models (LDMs) are employed as the conditional generation model, where the autoencoder compresses images into latent representations for diffusion and then reconstructs images from these latent representations. Such a formulation poses a crucial challenge: VFI expects that the output is $\textit{deterministically}$ equal to the ground truth intermediate frame, but LDMs $\textit{randomly}$ generate a diverse set of different images when the model runs multiple times. The reason for the diverse generation is that the cumulative variance (variance accumulated at each step of generation) of generated latent representations in LDMs is large. This makes the sampling trajectory random, resulting in diverse rather than deterministic generations. To address this problem, we propose our unique solution: Frame Interpolation with Consecutive Brownian Bridge Diffusion. Specifically, we propose consecutive Brownian Bridge diffusion that takes a deterministic initial value as input, resulting in a much smaller cumulative variance of generated latent representations. Our experiments suggest that our method can improve together with the improvement of the autoencoder and achieve state-of-the-art performance in VFI, leaving strong potential for further enhancement.

Main paper: 
 related works 2.1 video frame interpolation: 
 Video Frame Interpolation can be roughly divided into two categories in terms of methodologies: flow-based methods [1, 7, 12, 18, 20, 24, 29, 32, 34, 39, 42, 47, 60] and kernel-based methods [4, 5, 9, 27, 36, 37, 46] . Flow-based methods assume certain motion types, where a few works assume non-linear types [7, 12] while others assume linear. Via such assumptions, flow-based methods estimate flows in two ways. Some estimate flows from the intermediate frame to neighboring frames (or the reverse way) and apply backward warping to neighboring frames and their features [1, 7, 12, 20, 29, 32, 39, 42, 60] . Others compute flows among the neighboring frames and apply forward splatting [18, 24, 34, 35, 47] . In addition to the basic framework, advanced details such as recurrence of inputs with different resolution level [24] , cross-frame attention [60] , and 4D-correlations [29] are proposed to improve performance. Kernel-based methods, introduced by [36] , aim to predict the convolution kernel applied to neighboring frames to generate the intermediate frame, but it has difficulty in dealing with large displacement. Following works [5, 9, 27] alleviate such issues by introducing deformable convolution. LDMVFI [11] recently introduced a method based on Latent Diffusion Models (LDMs) [43] , formulating VFI as a conditional generation task. LDMVFI uses an autoencoder introduced by LDMs to compress images into latent representations, efficiently run the diffusion process, and then reconstruct images from latent space. Instead of directly predicting image pixels during reconstruction, it takes upsampled latent representations in the autoencoder as inputs to predict convolution kernels in kernel-based methods to complete the VFI task.The diffusion model is introduced by DDPM [17] to image generation task and achieves excellent performance in high-fidelity and high-diversity image generation. The whole diffusion model can be split into a forward diffusion process and a backward sampling process. The forward diffusion process is defined as a Markov Chain with steps 𝑡 = 1, ...,𝑇 , and the backward sampling process aims to estimate the distribution of the reversed Markov chain. The variance of the reversed Markov chain has a closed-form solution, and and expected value of the reversed Markov chain is estimated with a deep neural network. Though achieving strong performance in image generation tasks, DDPM [17] requires 𝑇 = 1000 iterative steps to generate images, resulting in inefficient generation. Sampling steps cannot be skipped without largely degrading performance because the conditional distribution at step 𝑡 -2 needs to be computed with the conditional distribution at time 𝑡 -1 and 𝑡 due to its Markov property. To enable efficient and high-quality generation, DDIM [48] proposes a non-Markov formulation of diffusion models, where the conditional distribution at time 𝑡 -𝑘 (𝑘 > 0) can be directly computed with the conditional distribution at time 𝑡. Therefore, skipping steps does not largely degrade performance. Score-based SDEs [3, 49, 63] are also proposed as an alternative formulation of diffusion models by writing the diffusion process in terms of Stochastic Differential Equations [38] , where the reversed process has a closed-form continuous time formulation and can be solved with Eluer's method with a few steps [49] . In addition, Probability Flow ODE is proposed as the deterministic process that shares the same marginal distribution with the reversed SDE [49] . Following score-based SDEs, some works propose efficient methods to estimate the solution Probability Flow ODE [30, 31] . Instead of focusing on the nature of the diffusion process, DeepCache [33] proposes a feature caching and sharing mechanism in the denoising UNet, enabling parallel and skipping computation and further improving efficiency. To deal with high-resolution images, the Latent Diffusion Model [43] proposes an autoencoder with a Vector Quantization Layer (VQ Layer) that compresses and reconstructs images, and diffusion models run with compressed images. With such an autoencoder, high-resolution images can be generated efficiently. Other than accelerating generation, diffusion models are applied to conditional generation tasks [3, 6, 28, 43, 45, 61, 63] such as generation based on poses or skeletons, image inpainting, etc. 
methodology: 
 In this section, we will first go through preliminaries on the Diffusion Model (DDPM) [17] and Brownian Bridge Diffusion Model (BBDM) [28] and introduce the overview of our two-stage formulation: autoencoder and ground truth estimation (with consecutive Brownian Bridge diffusion). Then, we will discuss the details of our autoencoder method. Finally, we propose our solution to the frame interpolation task: consecutive Brownian Bridge diffusion.Diffusion Model. The forward diffusion process of Diffsuion Model [17] is defined as:EQUATIONWhen 𝑡 = 1, x 𝑡 -1 = x 0 is a sampled from the data (images). By iterating Eq. ( 1), we get the conditional marginal distribution of x 𝑡 [17] :EQUATIONwhere𝛼 𝑡 = 𝑡 𝑠=1 (1 -𝛽 𝑠 ).The sampling process can be derived with the Bayes' theorem [17] :EQUATIONwhere μ𝑡 =EQUATIONand β𝑡 = 1EQUATIONEq. ( 4) can be rewritten with Eq. ( 2) via reparameterization:EQUATIONBy Eq. ( 4) and ( 6), we only need to estimate 𝜖 to estimate 𝑝 𝜃 (x 𝑡 -1 |x 𝑡 ). Therefore, the training objective is:EQUATIONIt suffices to train a neural network 𝜖 𝜃 (x 𝑡 , 𝑡) predicting 𝜖.Brownian Bridge Diffusion Model. Brownian Bridge [44] is a stochastic process that transits between two fixed endpoints, which is formulated as𝑋 𝑡 = 𝑊 𝑡 |(𝑊 𝑡 1 ,𝑊 𝑡 2 ), where 𝑊 𝑡 is a standard Wiener process with distribution N (0, 𝑡). We can write a Brownian Bridge as 𝑋 𝑡 = 𝑊 𝑡 |(𝑊 0 ,𝑊 𝑇 ) to define a diffusion process. When 𝑊 0 = 𝑎,𝑊 𝑇 = 𝑏, it follows a normal distribution:EQUATIONBBDM [28] develops an image-to-image translation method based on the Brownian Bridge process by treating 𝑎 and 𝑏 as two images.The forward diffusion process is defined as:EQUATIONwhere 𝑚 𝑡 =andEQUATIONx 0 and y are two images, and 𝑠 is a constant that controls the maximum variance in the Brownian Bridge. The sampling process is derived based on Bayes' theorem [28] :EQUATION)where μ𝑡 = 𝑐 𝑥𝑡 x 𝑡 + 𝑐 𝑦𝑡 𝑦 + 𝑐 𝜖𝑡 (𝑚 𝑡 (y -x 0 ) + √︁ 𝛿 𝑡 𝜖), 𝑐 𝑥𝑡 = 𝛿 𝑡 -1 𝛿 𝑡 1 -𝑚 𝑡 1 -𝑚 𝑡 -1 + 𝛿 𝑡 |𝑡 -1 𝛿 𝑡 (1 -𝑚 𝑡 ), 𝑐 𝑦𝑡 = 𝑚 𝑡 -1 -𝑚 𝑡 1 -𝑚 𝑡 1 -𝑚 𝑡 -1 𝛿 𝑡 -1 𝛿 𝑡 , 𝑐 𝜖𝑡 = (1 -𝑚 𝑡 -1 ) 𝛿 𝑡 |𝑡 -1 𝛿 𝑡 , 𝛿 𝑡 |𝑡 -1 = 𝛿 𝑡 -𝛿 𝑡 -1 (1 -𝑚 𝑡 ) 2 (1 -𝑚 𝑡 -1 ) 2 .The goal of video frame interpolation is to estimate the intermediate frame 𝐼 𝑛 given the previous frame 𝐼 0 and the next frame 𝐼 1 . n is set to 0.5 to interpolate the frame in the middle of 𝐼 0 and 𝐼 1 . In latent diffusion models [43] , there is an autoencoder that encodes images to latent representations and decodes images from latent representations. The diffusion model is given a standard Gaussian noise, denoises it according to the sampling process, and decodes the denoised latent representation back to an image. Since the initial noise is random, the decoded images are diverse images when they are sampled repetitively with the same conditions such as poses. Instead of diversity, VFI looks for a deterministic ground truth, which is the intermediate frame. Such a ground truth frame is encoded to a ground truth latent representation by the encoder, and only the ground truth latent representation needs to be estimated since the decoder will decode it back to the frame. Therefore, LDMbased VFI can be split into two stages: autoencoder and ground truth estimation. The two stages are defined as:(1) Autoencoder. The primary function of the autoencoder is similar to image compression: compressing images to latent representations so that the diffusion model can be efficiently implemented. We denote x, y, z as encoded latent representations of 𝐼 𝑛 , 𝐼 0 , 𝐼 1 . In this stage, the goal is to compress 𝐼 𝑛 to x with an encoder and then reconstruct 𝐼 𝑛 from x with a decoder. x is provided to the decoder together with neighboring frames 𝐼 0 , 𝐼 1 and their features in the encoder at different down-sampling factors. The overview of this stage is shown in Figure 2 (a). However, to interpolate the intermediate frame, x is unknown, so we need to estimate this ground truth. (2) Ground truth estimation. In this stage, the goal is to accurately estimate x with a diffusion model. The diffusion model converts x to y, z with the diffusion process, and we train a UNet to predict the difference between the current diffusion state and x, shown in Figure 2 (b). The sampling process of the diffusion model will convert y, z to x with the UNet output.The autoencoder is modeled with VQModel [43] in Section 3.3, and the ground truth estimation is accomplished by our proposed (latent) consecutive Brownian Bridge diffusion in Section 3.4. During inference, both stages are combined as shown in Figure 2 (c) , where we decode diffusion-generated latent representation x. Via such formulation, we can have a more specific direction to improve VFI quality. If images decoded from x (Figure 2 (a) ) have similar visual quality to images decoded from x (Figure 2 (c) ), then the diffusion model achieves a strong performance in ground truth estimation, so it will be good to develop a good autoencoder. On the other way round, the performance of ground truth estimation can be potentially improved by redesigning the diffusion model.Diffusion models running in pixel space are extremely inefficient in video interpolation because videos can be up to 4K in real life [41] . Therefore, we can encode images into a latent space with encoder E and decode images from the latent space with decoder D. Features of 𝐼 0 , 𝐼 1 are included because detailed information may be lost when images are encoded to latent representations [11] . We incorporate feature pyramids of neighboring frames into the decoder stage as guidance because neighboring frames contain a large number of shared details. Given 𝐼 𝑛 , 𝐼 0 , 𝐼 1 , the encoder E will output encoded latent representation x, y, z for diffusion models and feature pyramids of 𝐼 0 , 𝐼 1 in different down-sampling rates, denoted {𝑓 𝑘 𝑦 }, {𝑓 𝑘 𝑧 }, where 𝑘 is down-sampling factor. When 𝑘 = 1, {𝑓 𝑘 𝑦 } and {𝑓 𝑘 𝑧 } represent original images. The decoder D will take sampled latent representation x (output of diffusion model that estimates x) and feature pyramids {𝑓 𝑘 𝑦 }, {𝑓 𝑘 𝑧 } to reconstruct 𝐼 𝑛 . In lines of equations, these can be expressed as:EQUATIONOur encoder shares an identical structure with that in LDMVFI [11] , and we slightly modify the decoder to better fit the VFI task.Decoding with Warped Features. LDMVFI [11] apply crossattention [54] to up-sampled x and 𝑓 𝑘 𝑥 , 𝑓 𝑘 𝑦 , but keeping feature of neighboring frames may preserve their original information (i.e. motion in previous and next frames). This is problematic since motion changes may be drastic in different frames. Therefore, we estimate optical flows from 𝐼 𝑛 to 𝐼 0 , 𝐼 1 with a flow estimation module and apply backward warping to the feature pyramids. Suppose x is generated by our consecutive Brownian Bridge diffusion, and it is up-sampled to ℎ 𝑘 where 𝑘 denotes the downsampling factor compared to the original image. Then, we apply 𝐶𝐴 ℎ 𝑘 , 𝐶𝑎𝑡 (𝑤𝑎𝑟𝑝 (𝑓 𝑘 𝑦 ), 𝑤𝑎𝑟𝑝 (𝑓 𝑘 𝑧 )) for 𝑘 > 1 to fuse the latent representation ℎ 𝑘 and feature pyramids 𝑓 𝑘 𝑦 and 𝑓 𝑘 𝑧 , where 𝐶𝐴(•, •), 𝐶𝑎𝑡 (•, •), and 𝑤𝑎𝑟𝑝 (•) denotes cross attention, channelwise concatenation, and backward warping with estimated optical flows respectively. Finally, we apply convolution layers to ℎ 1 to predict soft mask 𝐻 and residual 𝛿. The interpolation output is Î𝑛 = 𝐻 * 𝑤𝑎𝑟𝑝 (𝐼 0 ) + (1 -𝐻 ) * 𝑤𝑎𝑟𝑝 (𝐼 1 ) + 𝛿, where * holds for Hadamard product, and Î𝑛 is the reconstructed image. The detailed illustration of the architecture is shown in Figure 3 . The VQ layer is connected with the encoder during training, but it is disconnected from the encoder and receives the sampled latent representation from the diffusion model.Brownian Bridge diffusion model (BBDM) [28] is designed for translation between image pairs, connecting two deterministic points, which seems to be a good solution to estimate the ground truth intermediate frame. However, it does not fit the VFI task. In VFI, images are provided as triplets because we aim to reconstruct intermediate frames giving neighboring frames, resulting in three points that need to be connected. If we construct a Brownian Bridge between the intermediate frame and the next frame, then the previous frame is ignored, and so is the other way round. This is problematic because we do not know what "intermediate" is if we lose one of its neighbors. Therefore, we need a process that transits among three images. Given two neighboring images 𝐼 0 , 𝐼 1 , we aim to construct a Brownian Bridge process with endpoints 𝐼 0 , 𝐼 1 and additionally condition its middle stage on the intermediate frame 𝐼 𝑛 (𝑛 = 0.5 for 2× interpolation). To achieve this, the process starts at 𝑡 = 0 with value y, passes 𝑡 = 𝑇 with value x, and ends at 𝑡 = 2𝑇 with value z. To be consistent with the notation in diffusion models, x, y, z are used to represent latent representations of 𝐼 𝑛 , 𝐼 0 , 𝐼 1 respectively. It is therefore defined as 𝑋 𝑡 = 𝑊 𝑡 |𝑊 0 = y,𝑊 𝑇 = x,𝑊 2𝑇 = z. The sampling process starts from time 0 and 2𝑇 and goes to time 𝑇 . Such a process indeed consists of two Brownian Bridges, where the first one ends at x and the second one starts at x. We can easily 𝑠 ← 𝑈 𝑛𝑖 𝑓 𝑜𝑟𝑚(0,𝑇 ) 4:𝑤 𝑠 ← 𝑚𝑖𝑛{ 1 𝛿 𝑡 , 𝛾 } ⊲ 𝛾 is a pre-defined constant 5:𝜖 ← N (0, I) 6:x s 1 ← 𝑠 𝑇 x + (1 -𝑠 𝑇 )y + √︃ 𝑠 (𝑇 -𝑠 ) 𝑇 𝜖 7: x s 2 ← 𝑠 𝑇 x + (1 -𝑠 𝑇 )z + √︃ 𝑠 (𝑇 -𝑠 ) 𝑇 𝜖 8:r ← 𝑈 𝑛𝑖 𝑓 𝑜𝑟𝑚(0, 1) 9:if r < 0.5 then take a gradient step on 10:∇ 𝜃 ||𝜖 𝜃 (x 𝑠 1 ,𝑇 -𝑠, y, z) -(x 𝑠 1 -x)|| 2 2 11:else take a gradient step on 12:∇ 𝜃 ||𝜖 𝜃 (x 𝑠 2 ,𝑇 + 𝑠, y, z) -(x 𝑠 2 -x)|| 2 2 13: end if 14: until convergence Algorithm 2 Sampling 1: 𝑡 1 , 𝑡 2 ← 𝑇 , Δ 𝑡 ← 𝑇 sampling steps , x 𝑇 1 = y, x 𝑇 2 = z 2: repeat 3: 𝑠 1 , 𝑠 2 ← 𝑡 1 -Δ 𝑡 , 𝑡 2 -Δ 𝑡 4:𝜖 ← N (0, I) 5:EQUATION)According to Eq. ( 13), we can derive the distribution of our consecutive Brownian Bridge diffusion (details shown in supplementary materials):EQUATION)Cleaner Formulation. Eq. ( 11) is in a discrete setup (i.e. time = 0, 1, ...,𝑇 ), and the sampling process is derived via Bayes' theorem, resulting in a complicated formulation. To preserve the maximum variance, it suffices to have 𝑇 = 2𝑠 in Eq. ( 8) and discretize T for training and sampling. Our forward diffusion is defined as Eq. ( 14).To sample from time 𝑠 from 𝑡 (𝑠 < 𝑡), we rewrite Eq. ( 11) according to Eq. ( 13):EQUATIONNote that Eq. ( 11) is slightly different from ours in that it uses x 0 to represent x, but we directly use x. Since we have a closed-form solution of 𝑝 𝜃 (x 𝑠 |x 𝑡 , y) for 0 < 𝑠 < 𝑡 < 𝑇 , our method does not need DDIM [48] sampling for acceleration.Training and Sampling. According to Eq. ( 15), it suffices to have a neural network 𝜖 𝜃 estimating x 𝑡x 0 . Moreover, based on Eq. ( 14), we can sample 𝑠 from 𝑈 𝑛𝑖 𝑓 𝑜𝑟𝑚(0,𝑇 ) and compute 𝑡 = 𝑇 ± 𝑠 for 𝑡 > 𝑇 and 𝑇 < 𝑡. With one sample of 𝑠, we can obtain two samples at each side of our consecutive Brownian bridge diffusion symmetric at T. y, z are added to the denoising UNet as extra conditions. Therefore, the training objective becomes:EQUATIONwhereEQUATION)Optimizing Eq. ( 16) requires two forward calls of the denoising UNet, so to be more efficient in computation, we randomly select one of them to optimize during training. Moreover, [15] proposes 𝑚𝑖𝑛 -𝑆𝑁 𝑅 -𝛾 weighting for different time steps during training based on the signal-to-noise ratio, defined as 𝑚𝑖𝑛{𝑆𝑁 𝑅(𝑡), 𝛾 }. In DDPM [17] , we have 𝑆𝑁 𝑅(𝑡) = 𝛼 𝑡1-𝛼 𝑡 because the mean and standard deviation are scaled by √ 𝛼 𝑡 and √ 1 -𝛼 𝑡 respectively in the diffusion process. However, in our formulation, consecutive frames 𝐼 0 , 𝐼 1 share almost identical mean, and so as their encoded latent representations. Therefore, the mean is never scaled down. The SNR is defined as 1 , where 𝛿 𝑡 is the standard deviation of the diffusion process at time 𝑡. With the 𝑚𝑖𝑛 -𝑆𝑁 𝑅 -𝛾 weighting, the weighting of loss is defined as𝑤 𝑡 = 𝑚𝑖𝑛{ 1 𝛿 𝑡 , 𝛾 }.The training algorithm is shown in Algorithm 1. To sample from neighboring frames, we can sample from either of the two endpoints y, z with Eq. ( 14) and (15) , shown in Algorithm 2. After sampling, we replace x in Eq (12) with the sampled latent representations to decode the interpolated frame. Cumulative Variance. As we claimed, diffusion model [17] with conditional generation has a large cumulative variance while ours is much smaller. The cumulative variance for traditional conditional generation is larger than 1 + 𝑡 β𝑡 , which corresponds to 11.036 in experiments. However, in our method, such a cumulative variance is smaller than 𝑇 = 2 in our experiments, resulting in a more deterministic estimation of the ground truth latent representations. The detailed justification is in the supplementary materials. 
experiments 4.1 implementations: 
 Autoencoder. The down-sampling factor is set to be 𝑓 = 16 for our autoencoder, which follows the setup of LDMVFI [11] 1 . The flow estimation and refinement modules are initialized from pretrained VFIformer [32] and frozen for better efficiency. The codebook size and embedding dimension of the VQ Layer are set to 16384 and 3 respectively. The number of channels in the compact latent space (encoder output) is set to 8. A self-attention [54] is applied at 16× down-sampling latent representation (both encoder and decoder), and cross attentions [54] with warped features are applied on 2×, 4×, and 8× down-sampling factors in the decoder. Following LDMVFI, max-attention [53] is applied in all attention layers for better efficiency. The model is trained with Adam optimizer [25] with a learning rate of 10 -5 for 100 epochs with a batch size of 16. The autoencoder is still slowly converging after 100 epochs, but we stopped training to evaluate it.Consecutive Brownian Bridge Diffusion. We set 𝑇 = 2 (corresponding to maximum variance 1 2 ) and discretize 1000 steps for training and 50 steps for sampling. The denoising UNet takes the concatenation of x 𝑡 , y, z as input and is trained with Adam optimizer [25] with 10 -4 learning rate for 30 epochs with a batch size of 64. 𝛾 is set to be 5 in the 𝑚𝑖𝑛 -𝑆𝑁 𝑅 -𝛾 weighting.Training Sets. To ensure a fair comparison with most recent works [1, 7, 12, 18, 20, 24, 32, 34, 42, 47] , we train our models in Vimeo 90K triplets dataset [59] , which contains 51,312 triplets. We apply random flipping, random cropping to 256 × 256, temporal order reversing, and random rotation with multiples of 90 degrees as data augmentation. Test Sets. We select UCF-101 [50] , DAVIS [41] , SNU-FILM [8] , and Middlebury [2] to evaluate our method. UCF-101 and Middlebury consist of relatively low-resolution videos (less than 1K), whereas DAVIS and SNU-FILM consist of relatively high-resolution videos (up to 4K). SNU-FILM consists of four categories with increasing levels of difficulties (i.e. larger motion changes): easy, medium, hard, and extreme. Evaluation Metrics. Recent works [10, 11, 62] reveal that PSNR and SSIM [56] are sometimes unreliable because they have relatively low correlation with humans' visual judgments. However, deep-learning-based metrics such as FID [16] , LPIPS [62] , and FloLPIPS [10] are shown to have a higher correlation with humans' visual judgments in [11, 62] . Moreover, in our experiments, we also find such inconsistencies between PSNR/SSIM and visual quality, which will be discussed in Section 4.3. Therefore, we select FID, LPIPS, and FloLPIPS as our main evaluation metrics. LPIPS and FID measure distances in the space of deep learning features. FloLPIPS is based on LPIPS but takes the motion in the frames into consideration. Our methods evaluated with PSNR and SSIM will be included in the supplementary materials.Quantitative Results. Our method is compared with recent opensource state-of-the-art VFI methods, including ABME [40] , MCVD [55] , VFIformer [32] , IFRNet [26] , AMT [29] , UPR-Net [24] , EMA-VFI [60] , and LDMVFI [11] . The evaluation is reported in LPIPS/FloLPIPS/FID (lower the better), shown in Table 1 . We evaluate VFIformer, IFRNet, AMT, UPR-Net, and EMA-VFI with their trained weights, and other results are provided in the appendix of LDMVFI [11] . Models with different versions in the number of parameters are all chosen to be the largest ones. With the same autoencoder as LDMVFI [11] , our method (denoted as ours †) generally achieves better performance than LDMVFI, indicating the effectiveness of our consecutive Brownian Bridge diffusion. Moreover, with an improved autoencoder, our method (denoted as ours) generally achieves state-of-the-art performance. It is important to note that we achieve much better FloLPIPS than other SOTAs, indicating our interpolated results achieve stronger motion consistency. In a few datasets, our method does not achieve the best performance in FID or LPIPS because our autoencoder is still converging. Qualitative Results. In Table 1 , our consecutive Brownian Bridge diffusion with the autoencoder in LDMVFI [11] (denoted as our †) generally achieves better quantitative results than LDMVFI, showing our method is effective. We include qualitative visualization in Figure 5 to support this result. Moreover, as mentioned in Section 1, we find that the autoencoder in [11] usually reconstructs overlaid images, and therefore we propose a new method of reconstruction.We provide examples to visualize the reconstruction results with our autoencoder and LDMVFI's autoencoder for comparison, shown in Figure 4 . All examples are from SNU-FILM extreme [8] , which contains relatively large motion changes in neighboring frames.We have provided some visual comparisons of our method and recent SOTAs in Figure 1 . Our method achieves better visual quality because we have clearer details such as dog skins, cloth with folds, and fences with nets. However, UPR-Net [24] achieves better PSNR/SSIM in all the cropped regions (5 -10% better) than ours, which is highly inconsistent with the visual quality. More qualitative results are provided in the supplementary materials.As we discussed in Section 3.2, latent-diffusion-based VFI can be broken down into two stages, so we conduct an ablation study on the ground truth estimation capability of our consecutive Brownian Bridge diffusion. We compare the LPIPS/FloLPIPS/FID of decoded images with diffusion-generated latent representation x and ground truth x, which is encoded 𝐼 𝑛 . The results are shown in Table 2 . It is important to note that, fixing inputs as the ground truth, our autoencoder achieves a stronger performance than the autoencoder in LDMVFI [11] , indicating the effectiveness of our autoencoder. Also, fixing the autoencoder, our consecutive Brownian Bridge diffusion achieves almost identical performance with the ground truth, indicating its strong capability of ground truth estimation. However, the conditional generation model in LDMVFI [11] usually underperforms the autoencoder with ground truth inputs. Therefore, our method has a stronger ability in both the autoencoder and ground truth estimation stages. More ablation study is provided in the supplementary materials. [11] vs our method with the same autoencoder in LDMVFI (LDMVFI vs our † in Table 1 ). With the same autoencoder, our method can still achieve better visual quality than LDMVFI, demonstrating the superiority of our proposed consecutive Brownian Bridge diffusion. 
conclusion: 
 In this study, we formulate the latent-diffusion-based VFI as a twostage problem: autoencoder ground truth estimation. With this formulation, it is easy to figure out which part needs enhancements, guiding future research. We propose our consecutive Brownian Bridge diffusion that better estimates the ground truth latent representation due to its low cumulative variance. This method improves when the autoencoder is improved and achieves state-of-the-art performance with a simple yet effective design of the autoencoder, demonstrating its strong potential in VFI as a carefully designed autoencoder could potentially boost the performance by a large margin. Therefore, we believe our work will provide a unique research direction for diffusion-based frame interpolation.Limitations and Future Research. Our method uses a bisectionlike method to conduct multi-frame interpolation: we can interpolate 𝑡 = 0.5 between 𝑡 = 0, 1 and then interpolate 𝑡 = 0.25, 0.75. However, our method cannot directly interpolate 𝑡 = 0.1 from 𝑡 = 0, 1. Future research can be conducted to resolve the limitations mentioned above or to improve autoencoders or diffusion models for better interpolation quality.  
: 
 methods predict convolution kernels for pixels in the neighboring frames. Recent advances in flow estimation [19, 21-23, 51, 52, 57] make it more popular to adopt flow-based methods in VFI.Other than these two folds of methods, MCVD [55] and LD-MVFI [11] start formulating VFI as a diffusion-based image generation problem. LDMVFI considers VFI as a conditional generation task with Latent Diffusion Models (LDMs) [43] , where LDMs contain an autoencoder that compresses images into latent representations and reconstructs images from latent representations. Diffusion models [17] run in the latent space of the autoencoder. Though diffusion models achieve excellent performance in image generation, there remain challenges in applying them to VFI.(1) The formulation of diffusion models results in a large cumulative variance (the variance accumulated during sampling) of generated latent representations. The sampling process starts with standard Gaussian noise and adds small Gaussian noise to the denoised output at each step based on a pre-defined distribution. After the sampling process, images are generated, but these noises also add up to a large cumulative variance. Though such a variance is beneficial to diversity (i.e. repeated sampling results in different outputs), VFI requires that repeated sampling returns identical results, which is the ground truth intermediate frame. Therefore, a small cumulative variance is preferred in VFI. The relation of the cumulative variance and diversity is supported by the fact that DDIM [48] tends to generate relatively deterministic images than DDPM [17] . DDIM removes small noises at each sampling step, so the cumulative variance in DDIM is lower. LDMVFI [11] uses conditional generation as guidance, but this does not change the nature of large cumulative variance. In Section 3.4, we show that our method has a much lower cumulative variance than conditional generation. (2) Videos usually have high resolution, which can be up to 4K [41] ,resulting in practical constraints to apply diffusion models [17] in pixel spaces. It is natural to apply Latent Diffusion Models (LDMs) [43] to sample latent representations and reconstruct them back to images. LDMs apply VQModels in VQGAN [13] to compress images into latent representations and reconstruct images from latent representations. However, it does not take advantage of neighboring frames, which can be a good guide to reconstruction. LDMVFI designs reconstruction models that leverage neighboring frames, but it tends to reconstruct overlaid images when there is a relatively large motion between neighboring frames, possibly due to the cross-attention with features of neighboring frames, which is shown in Figure 1 .To tackle these challenges, we propose a consecutive Brownian Bridge diffusion model (in latent space) that transits among three deterministic endpoints for VFI. This method results in a much smaller cumulative variance, achieving a better estimation of the ground truth inputs. We can separate LDM-based VFI methods into two parts: autoencoder and ground truth estimation (with diffusion). It is different from the original LDMs [43] because the latent representation generated by diffusion does not aim to estimate some ground truth. It is also different from LDMVFI [11] because LD-MVFI does not consider the performance of autoencoder separately from the interpolation method. With such a two-stage separation, we evaluate them separately for specific directions of improvement. Moreover, we take advantage of flow estimation and refinement methods in recent literature [32] to improve the autoencoder. The feature pyramids from neighboring frames are warped based on estimated optical flows, aiming to alleviate the issues of reconstructing overlaid images. In experiments, our method improves by a large margin when the autoencoder is improved and achieves stateof-the-art performance. Our contribution can be summarized in three parts:• We propose a new consecutive Brownian Bridge diffusion model for VFI and justify its advantages over traditional diffusion models: lower cumulative variance and better ground truth estimation capability. Additionally, we provide a cleaner formulation of Brownian Bridges and also propose the loss weights among different times in Brownian Bridges. • We formulate the diffusion-based VFI as two stages: autoencoder and ground truth estimation. This is a novel interpretation of LDM-based VFI, which can provide specific directions for improvements. • Through extensive experiments, we validate the effectiveness of our method. Our method estimates the ground truth better than traditional diffusion with conditional generation. Moreover, the performance of our method improves when the autoencoder improves and achieves state-of-the-art performance with a simple yet effective autoencoder, indicating its strong potential in VFI. 
 
 Appendix: We follow their implementation and find that they achieve 16× down-sampling factor.2024-04-13 11:53. Page 6 of 1-10. 


Decision: The paper proposes a video frame interpolation method based on Brownian Bridge Diffusion. In particular, it proposes a consecutive Brownian bridge diffusion which operates on two endpoints with an additional middle point. It also warps features from reference frames and injects them via cross-attention, to the latent diffusion autoencoder architecture for increased performance. These modifications result in better performance over a wide range of datasets.

All reviewers agree that the proposed method is well-written and effective and the concerns are well-addressed during rebuttal.

I recommend acceptance.

Reviews: review: paper_summary: paper_summary: The authors propose a new video frame interpolation (VFI) method based on Brownian Bridge Diffusion. Due to the reduced variance of Brownian bridge diffusion, and because of the triplet pair nature of VFI, they propose a novel consecutive Brownian bridge diffusion which operates on two endpoints with an additional middle point. The authors also propose changes, namely warping features from reference frames and injecting them via cross attention, to the latent diffusion autoencoder architecture for increased performance. These modifications result in better performance over a wide range of datasets.
strength_weakness: strength_weakness: * Interesting and novel idea with the consecutive brownian bridge diffusion
* Comprehensive experiments
* Good results
limitations: limitations: * I do not believe that formulating VFI as two stages, autoencoder and ground truth estimation, is a novel concept. LDMVFI [1] also has this two stage approach, and while this can provide motivation to improve method in two separate ways I don't think this is a novel contribution.
* I find the writing of the manuscript unclear, specifically section 3.2, and generally the flow of the manuscript was somewhat difficult to fully understand unless reading many times.
* The main motivation was to reduce the variance of diffusion sampling. Why is DDIM on standard diffusion models not sufficient? It is briefly touched on lines 145-147 but not backed up with any evidence, and the cumulative variance experiment in the supplemental section 2.2 is performed on DDPM, not DDIM.
* Additional video examples would be appreciated, as there are only a few shown in the supplementary material and this is a video frame interpolation method

[1] Danier, Duolikun, Fan Zhang, and David Bull. "Ldmvfi: Video frame interpolation with latent diffusion models." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 2. 2024.
paper_summary: paper_summary: In this submission, the authors approach the problem of latent-diffusion-based video frame interpolation (VFI) as a two-stage process involving autoencoder ground truth estimation. This framework clearly identifies areas needing enhancement, providing a roadmap for future research. They introduce the consecutive Brownian Bridge diffusion technique, which better estimates the ground truth latent representation due to its low cumulative variance. This method shows improvement when the autoencoder is refined and achieves state-of-the-art performance with a straightforward yet effective autoencoder design.
strength_weakness: strength_weakness: 1. Clean and good formula, make it easy to read and follow.

2. The proposed method is based on some theories (Brownian Bridge), make this paper distinct from others.

3. Very clean illustration of the model architecture.

4. Extensive experiments results (both numerical and visualization) show its advantages.
limitations: limitations: 1. The motivation is not well enough. It is claimed that "VFI expects that the output is deterministically", while the generated images of LDM is diverse. Well I think for every diffusion generation tasks, the training images is also deterministically to the GT. So the motivation seems unclear to me.

2. The background about stochastic process is missing in the related works.

3. Minor: The font size in figures should match the main paper.
paper_summary: paper_summary: This paper introduces a novel method for video frame interpolation named Consecutive Brownian Bridge Diffusion. 

The authors formulate the VFI problem into two stages: autoencoder and ground truth estimation and optimize each stage accordingly. 

For the Autoencoder part, the authors use optical flow estimation and an additional loss for the enhancement, addressing issues related to reconstructing overlaid images. 

For the Diffusion part, a Brownian Bridge Diffusion method is proposed. BBDM is previously used for distribution transformation, and a well-known downstream task: image-to-image translation.  The optimization process for Brownian Bridge alternates randomly. The sampling method is similar to Euler A. 

Experimental results demonstrate that the method achieves state-of-the-art performance across multiple datasets and has the potential for further enhancement when the autoencoder is improved.
strength_weakness: strength_weakness: 1. This work clearly and logically presents both its motivations and solutions. These include using optical flow estimation to optimize autoencoders and employing the Brownian bridge for frame interpolation, both of which are well-motivated.

2. The experimental section effectively demonstrates the efficacy of the proposed solutions. Table 1 shows that both the Brownian bridge interpolation method and the enhanced autoencoders have positive impacts.

3. The ablation study validates the effectiveness of individual modules.
limitations: limitations: There are some small concerns:

1. The author appears to need to compare several solutions involving the Schrödinger bridge. Many Schrödinger bridge-based approaches have been applied to image translation. Given this, is there also potential for them to be used in VFI? Could we briefly analyze the advantages and disadvantages between BBDM and Schrödinger Bridges?

2. The summary of related work on VAE seems to have some inaccuracies. LDM utilizing a VAE does not include a VQ module. However, the VQ module and codebook structure are present in dVAE of Taming Transformers (VQ-GAN).

There is one significant concern regarding the complexity of the proposed solution. What are the inference time and resource requirements? These details are not provided in the paper, yet they are crucial for VFI, which is particularly sensitive to inference latency. Authors could include a discussion of these aspects.