VFIMamba: Video Frame Interpolation with State Space Models

Authors: guozhen  zhang
chunxu  liu
yutao  cui
xiaotong  zhao
kai  ma
limin  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:35:04

Abstract: Inter-frame modeling is pivotal in generating intermediate frames for video frame interpolation (VFI). Current approaches predominantly rely on convolution or attention-based models, which often either lack sufficient receptive fields or entail significant computational overheads. Recently, Selective State Space Models (S6) have emerged, tailored specifically for long sequence modeling, offering both linear complexity and data-dependent modeling capabilities. In this paper, we propose VFIMamba, a novel frame interpolation method for efficient and dynamic inter-frame modeling by harnessing the S6 model. Our approach introduces the Mixed-SSM Block (MSB), which initially rearranges tokens from adjacent frames in an interleaved fashion and subsequently applies multi-directional S6 modeling. This design facilitates the efficient transmission of information across frames while upholding linear complexity. Furthermore, we introduce a novel curriculum learning strategy that progressively cultivates proficiency in modeling inter-frame dynamics across varying motion magnitudes, fully unleashing the potential of the S6 model. Experimental findings showcase that our method attains state-of-the-art performance across diverse benchmarks, particularly excelling in high-resolution scenarios. In particular, on the X-TEST dataset, VFIMamba demonstrates a noteworthy improvement of 0.80 dB for 4K frames and 0.96 dB for 2K frames.

Main paper: 
 introduction: 
 Video Frame Interpolation (VFI), a fundamental task in video data processing, is gaining substantial attention for its ability to generate intermediate frames between consecutive frames (Liu et al., 2017) . Its utility spans many practical applications, including creating slow-motion videos through temporal upsampling (Jiang et al., 2018) , enhancing video refresh rates (Reda et al., 2022) , and generating novel views (Flynn et al., 2016; Szeliski, 1999) . VFI typically encompass two primary stages (Zhang et al., 2023) : firstly, conducting the inter-frame modeling of input consecutive frames; and secondly, leveraging the acquired information to estimate inter-frame motion and generate intermediate frame appearance. In practice, VFI often deals with high-resolution inputs (e.g., 4K) (Sim et al., 2021) , which results in significant object displacement and imposes high demands on the large receptive field of the modules that model information between frames. Additionally, since VFI is commonly applied to long-duration videos such as movies, model speed is also of paramount importance. Thus, striking a delicate balance between a sufficient receptive field and fast processing speed in modeling inter-frame information is the key in crafting effective VFI models. Current methods for modeling inter-frame information predominantly rely on convolutional neural networks (CNNs) (Liu et al., 2017; Kong et al., 2022; Huang et al., 2022) and attention-based models (Lu et al., 2022a; Zhang et al., 2023; Park et al., 2023; Liu et al., 2024a) . However, as illustrated in the first three rows of Table 1 , these methods either (1) lack flexibility and cannot adaptively model based on the input, (2) do not have sufficient receptive fields to capture inter-frame correlations at high resolutions, or (3) suffer from prohibitive computational complexity.On the other hand, Natural Language Processing (NLP) has recently witnessed the emergence of structured state space models (SSMs) (Gu et al., 2021) . Theoretically, SSMs combine the benefits of Recurrent Neural Networks (RNNs) and CNNs, leveraging the global receptive field characteristic of RNNs and the computational efficiency inherent in CNNs. One particularly notable SSM is the Selective State Space Model (S6), also known as Mamba (Gu & Dao, 2023) , which has garnered significant attention within the vision community. Mamba's novel feature of making SSM parameters time-variant (i.e., data-dependent) enables it to effectively select relevant context within sequences, a crucial factor for enhancing model performance. However, to the best of our knowledge, S6 has not yet been applied to low-level video tasks.To address the challenges faced by current VFI models and to explore the potential of the S6 model (Gu & Dao, 2023 ) in low-level video tasks, we propose VFIMamba, a novel frame interpolation method that adapts the S6 model for efficient and dynamic inter-frame modeling. As shown in Table 1 , VFIMamba provides the advantages of a global receptive field with linear complexity while maintaining data-dependent adaptability. Specifically, we introduce the Mixed-SSM Block (MSB) to replace existing modules for inter-frame information transfer. The original S6 model can only process a single sequence, so it is necessary to merge tokens from two frames into one sequence for effective inter-frame modeling. After thorough analysis and exploration, we figured out that interleaving tokens from both frames into a "super image" is more suitable for VFI. We then conduct multi-directional SSMs on this image to model inter-frame information. This interleaved approach facilitates interactions between adjacent tokens from different frames during sequence modeling and ensures that the intermediate tokens of any pair of tokens in the sequence are from their spatiotemporal neighborhood. By stacking MSB modules, our model effectively handles complex inter-frame information exchange. Finally, we use the extracted inter-frame features to estimate motion and generate the appearance of intermediate frames.While the S6 model boasts the advantages listed in Table 1 , it is crucial to employ appropriate training strategies to fully exploit its potential. Inspired by Bengio et al. (2009) , we propose a novel curriculum learning strategy that progressively teaches the model to handle inter-frame modeling across varying motion amplitudes. Specifically, while maintaining training on Vimeo90K (Xue et al 2019), we incrementally introduce large motion data from X-TRAIN (Sim et al., 2021) , increasing the motion amplitude over time. This learning strategy enables VFIMamba to perform well across a wide range of motion amplitudes, thereby fully unleashing the potential of the S6 model.To validate the effectiveness of VFIMamba across various types of video data, we conduct extensive testing on different benchmarks. As shown in Figure 1 , VFIMamba achieves the state-of-the-art (SOTA) performance across diverse datasets. This is particularly evident in high-resolution and large-motion datasets such as X-TEST (Sim et al., 2021) and SNU-FILM (Choi et al., 2020) .Contribution. In summary, the contributions of this paper are as follows:(1) We are the first to adapt the S6 model to the video frame interpolation task. To better adapt the model for this task, we introduce the Mixed-SSM Block (MSB), providing a solid foundation for future architectural exploration in frame interpolation.(2) We propose a novel curriculum learning strategy that incrementally introduces data with varying motion amplitudes, thereby fully harnessing the potential of the S6 model.(3) Our model achieves the state-of-the-art performance across a wide range of datasets, potentially sparking interest in the exploration of the S6 model within the video low-level community.2 Related work 
video frame interpolation: 
 The performance of VFI methods has seen significant advancements with the emergence of deep learning models.(1) CNNs-based approaches (Bao et al., 2019; Liu et al., 2017; Huang et al., 2022; Niklaus & Liu, 2018; Choi et al., 2020; Zhu et al., 2024b; Jia et al., 2022; Niklaus et al., 2017; Kalluri et al., 2023) : Initially, DVF (Liu et al., 2017) utilized a U-Net-like (Ronneberger et al., 2015) network to model two input frames and predicted the voxel flow for warping the two frames into the intermediate frame. Following this, CtxSyn (Niklaus & Liu, 2018) introduced ContextNet and RefineNet, where ContextNet extracts context information from each frame, and RefineNet further refines the coarse intermediate frame produced by warping. RIFE (Huang et al., 2022) proposed a novel, efficient framework that employs self-distillation to significantly reduce computational load and parameters while maintaining high performance. Due to its simplicity, many convolutional modeling works (Kong et al., 2022; Jia et al., 2022) have improved upon RIFE.(2) Attention-based approaches (Lu et al., 2022a; Zhang et al., 2023; Park et al., 2023; Liu et al., 2024a) : VFIFormer (Lu et al., 2022a) was the first to use attention to model inter-frame information, replacing the encoder part of U-Net with Transformer blocks. After that, EMA-VFI (Zhang et al., 2023) uses Swin-based (Liu et al., 2021) local attention to simultaneously capture local appearance and motion information. AMT (Li et al., 2023 ) used a multi-scale cost-volume construction similar to RAFT (Teed & Deng, 2020) to further enhance motion modeling capabilities. BiFormer (Park et al., 2023) introduced quasi-global bilateral attention to further increase the receptive field for large motions. SGM-VFI (Liu et al., 2024a) introduced sparse global matching to model motion between frames. However, current models struggle to balance sufficient receptive fields with computational overhead. In contrast, our method introduces the first interpolation model based on State Space Models (SSMs) (Gu & Dao, 2023) and further pushes the performance boundaries of VFI tasks.In the field of NLP, SSMs (Gu et al., 2021; Smith et al., 2022; Mehta et al., 2022; Fu et al., 2022) have recently emerged as one of the most promising contenders to challenge the dominance of Transformers. The Structured State Space Sequence Model (S4) (Gu et al., 2021) was initially introduced for linear complexity modeling of long sequences. Subsequent works have improved 4  3   2  1   4  3   2  1  3  3   1  1  4  4   2  2   1 1 2 2 3 3 4 4   4 4 3 3 2 2 1 1   1 3 1 3 2 4 2 4   4 2 4 2 3 1 3 its computational efficiency and model capacity. S5 (Smith et al., 2022) proposed a parallel scan and MIMO SSM, and GSS (Mehta et al., 2022) enhanced the model's capability by introducing gated mechanisms. Mamba (S6) (Gu & Dao, 2023) has recently stood out due to its data-dependent parameter generation and efficient hardware implementation, outperforming Transformers in longsequence NLP tasks. In the visual domain, Vim (Zhu et al., 2024a) was the first to permute 2D images into sequences for global modeling using bidirectional SSMs. Vmamba (Liu et al., 2024b) extended to four directions and introduced a hierarchical structural design. VideoMamba (Li et al., 2024) was the first to apply S6 in the video domain by permuting all frames into a spatiotemporal sequence. MambaIR (Guo et al., 2024) was the first to use the S6 model for image restoration tasks, achieving superior performance over Transformers. In this work, we explore the potential of the S6 model in VFI tasks, validating its effectiveness through detailed analysis and experimentation.3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We provide a discussion of limitations in the appendix A.2. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 SSMs are mainly inspired by the continuous linear time-invariant (LTI) systems, which apply an implicit latent state h(t) ∈ R N to map a 1-dimensional sequence or function x(t) ∈ R → y(t) ∈ R. Specifically, SSMs can be formulated as an ordinary differential equation (ODE):EQUATIONEQUATIONwhere contains evolution matrix A ∈ R N×N , projection parameters B ∈ R N×1 and C ∈ R 1×N . However, it is hard to solve the above differential equation in deep learning settings and needs to be approximated through discretization. Recent SSMs (Gu et al., 2021) propose to introduce a timescale parameter ∆ to transform the A, B to their discrete counterparts Ā, B, i.e.,EQUATIONEQUATIONThe above SSMs are performed for each channel separately and their parameters are data-independent, meaning that Ā, B and C are the same for any input in the same channel, limiting their flexibility in sequence modeling. Mamba (Gu & Dao, 2023) proposes the selective SSMs (S6), which dynamically generate the parameters for each input data x i ∈ R L using the entire x i :EQUATIONwhere S B ∈ R N×L , S C ∈ R N×L , S ∆ ∈ R L×L are linear projection layers. The B i and C i are shared for all channels of x i , ∆ i contains ∆ of L channels, and A are the same as previous SSMs. By the discretization in equations 5 and 6, Ā and B become different based on input data.Given two consecutive frames I 0 , I 1 ∈ R H×W×3 along with a timestep t, the objective of the frame interpolation task is to generate the intermediate frame I t ∈ R H×W×3 . As illustrated in Figure 2 , the overall pipeline of VFIMamba consists of three main components: frame feature extraction, interframe modeling, and frame generation. Firstly, we employ a set of lightweight convolutional layers to independently extract shallow features from each frame, progressively reducing the resolution to facilitate more efficient inter-frame modeling. This process can be formulated as:EQUATIONwhere L represents the set of convolutional layers, and F i l denotes the extracted low-level feature for I i . Next, we perform multi-resolution inter-frame modeling using the proposed Mixed-SSM Block (MSB). Each scale comprises N MSBs, and downsampling between scales is achieved through overlapping patch embedding (Wang et al., 2022) . We define the resulting inter-frame features as F i ssm . Finally, we utilize these high-quality inter-frame features for frame generation, which involves motion estimation between two frames and appearance refinement:EQUATIONwhere G denotes the frame generation network. Since this work primarily focuses on exploring the use of SSMs for inter-frame modeling, we largely follow the design from Zhang et al. (2023) and Huang et al. (2022) for the frame generation components, with detailed specifications provided in the appendix.Effective inter-frame modeling is crucial for frame interpolation tasks (Zhang et al., 2023) . Methods such as RIFE (Huang et al., 2022) and EMA-VFI (Zhang et al., 2023) employ simple convolution layers or local attention for inter-frame modeling, achieving high inference speeds but limiting receptive field. Conversely, SGM-VFI (Liu et al., 2024a) uses global inter-frame attention for motion estimation, which improves performance but sacrifices efficiency. In this work, we propose to use state space models (SSMs), specifically S6 (Gu & Dao, 2023) , to achieve both efficiency and effectiveness in inter-frame modeling.To facilitate more efficient inter-frame information exchange globally, we utilize SSMs for inter-frame modeling. As illustrated in Figure 2 , we introduce the Mixed-SSM Block (MSB) for integrate the S6 model into VFI frameworks. The overall design of the MSB is analogous to Transformer (Vaswani et al., 2017) blocks, but with two pivotal distinctions: (1) We substitute the attention mechanism with an enhanced S6 Block (Gu & Dao, 2023) , which could conduct global inter-frame modeling with linear complexity. (2) Drawing inspiration from Guo et al. (2024) and Behrouz et al. (2024) , which identified the lack of locality and inter-channel interaction in SSMs, we replace the multilayer perceptron (MLP) with a Channel-Attention Block (CAB) (Hu et al., 2018) .The original S6 model is limited to processing one-dimensional sequences, necessitating a strategy for scanning the feature maps of two input frames for inter-frame modeling. As depicted in Figure 3 , there are two primary methods to rearrange the two frames: sequential rearrangement, where the frames are concatenated into a single super image, and interleaved rearrangement, where the tokens of the two frames are interleaved to form a super image. Regardless of the rearrangement method, following Liu et al. (2024b) , the super image can be scanned in four directions: horizontally, vertically, and in their respective reverse directions. The S6 Block is then employed to model each direction independently, and the resulting sequences are rearranged and merged back. Here, we discuss which rearrangement method is better for inter-frame modeling in the context of frame interpolation. First, let us introduce a conclusion from (Ali et al., 2024) : the S6 layers can be approximated as hidden attention layers, with the attention weights given by:EQUATIONwhereEQUATIONIn this formulation, α i, j represents the hidden attention weight of the j-th token x j to the i-th token x i in the sequence. Unlike attention, which calculates weights based solely on the information from tokens x i (Q i ) and x j (K j ), the S6 model includes H i, j , which encompasses the contextual information between the i-th and j-th tokens in the sequence. Based on this conclusion, we observe that in the interleaved rearrangement, the intermediate tokens of any pair of tokens in the sequence are from their spatiotemporal neighborhood. This means that H i, j incorporates more local modeling, which is beneficial for low-level tasks like frame interpolation. Additionally, the number of intermediate tokens between spatiotemporally adjacent tokens is generally smaller in the interleaved rearrangement.In contrast, in the sequential rearrangement, even spatiotemporally adjacent tokens are separated by many unrelated tokens in the sequence. This can introduce noise and interfere with the modeling of the relationship between these tokens. A specific example can be seen in Figure 3 , where the tokens between the 6-th token of the first frame and the 11-th token of the second frame differs significantly between the two rearrangement methods. In summary, we believe that for video frame interpolation, the interleaved rearrangement method is more suitable for better local spatially-aware processing.Our experiments, detailed in Section 4.2, further validate this conclusion.Despite the advantageous characteristics of the S6 model, such as data dependence and global receptive field, it is crucial to fully exploit its potential through appropriate training strategies. Currently, two main training strategies are employed for frame interpolation: (1) Vimeo90K Only:Most methods training models exclusively on the Vimeo90K (Xue et al., 2019) . Although Vimeo90K offers a rich variety of video content, as analyzed by Liu et al. (2024a) and Kiefhaber et al. (2024) ,Table 2 : Quantitative comparison with SOTA methods on the low-resolution datasets, in terms of PSNR/SSIM (Wang et al., 2004) . The best results and the second best results are boldfaced and underlined respectively. FLOPs was calculated for 720p input. ⋆ indicates the results copied from Zhang et al. (2023) and Li et al. (2023) . In "Training Dataset", "V" stands for the triplet of Vimeo-90K and "X" stands for X-TRAIN. In "Runtime", we evaluate the inference speed of each method on 1024 × 1024 resolution inputs by a 2080Ti GPU. In "Average", we calculate the average performance of each method in terms of PSNR and SSIM. (Liu et al., 2024a; Park et al., 2023 ) further train the model on X-TRAIN (Sim et al., 2021) , a dataset characterized by large motions and high-resolution content, after initial training on Vimeo90K. While this approach successfully enhances the model's performance on high-resolution data, it often leads to the forgetting of the small-motion modeling capabilities acquired from Vimeo90K.To address these issues and fully exploit the potential of the S6 model, inspired by Bengio et al. (2009) , we propose a curriculum learning strategy for learning inter-frame modeling capabilities across varying motion magnitudes while maintaining the ability to model small motions. Specifically, while continuing training on Vimeo90K, we progressively incorporated data from X-TRAIN. The original size of X-TRAIN is 512 × 512, to co-train with Vimeo90K, we first resize the frames to S × S and then random crop to the same as Vimeo90K. Every T epochs, the resized size S is increased by 10% (starting from 256), and the temporal interval between selected frames is doubled (starting from 2), which means the motion magnitude increases as training progresses. This strategy enables the model to gradually learn inter-frame modeling capabilities across varying motion magnitudes, starting with smaller motions and progressing to larger ones.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We have no plans to open-source the source code at this time, but the training code and models will be open-sourced after publication.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 We provide two models: a lightweight model, VFIMamba-S, and a high-performance model, VFI-Mamba. Both models have N = 3; the only difference is that VFIMamba has twice the number of channels as VFIMamba-S. As described in Section 3.4, we employ a curriculum learning strategy in which T = 50 and train for 300 epochs in total. More model configurations and training details are provided in the appendix. Quantitative comparison. To validate the versatility of our proposed VFIMamba, we evaluated its performance (PSNR/SSIM) (Wang et al., 2004 ) across a variety of well-known benchmarks with different resolutions. The low-resolution datasets include Vimeo90K (448 × 256) (Xue et al., 2019) , UCF101 (256 × 256) (Soomro et al., 2012) , and SNU-FILM (1280 × 720) (Reda et al., 2022) . Notably, SNU-FILM is categorized into four levels of difficulty based on frame intervals: Easy, Medium, Hard, and Extreme. The high-resolution datasets include X-TEST (Sim et al., 2021) , X-TEST-L (a more challenging subset selected by Liu et al. (2024a) ), and Xiph (Montgomery, 1994) . Originally, these datasets are in 4K resolution, and following Zhang et al. (2023) , we also resize them to 2K for testing.For 8x interpolation, we followed the testing procedure of FILM (Reda et al., 2022) and used an iterative approach for frame interpolation. Specifically, we first generated an intermediate frame based on the input two frames, and then, using a divide-and-conquer strategy, we further divided the first frame and the generated intermediate frame, as well as the generated intermediate frame and the last frame, to iteratively generate the remaining frames.As shown in Tables 2 and 3 , VFIMamba achieves state-of-the-art performance on almost all datasets with FLOPs comparable to efficient models (Kong et al., 2022; Zhang et al., 2023) . Specifically, in large motion scenarios like X-TEST and X-TEST-L, VFIMamba demonstrates a noteworthy improvement compared with previous metod. This excellent performance underscores the potential of the S6 model in frame interpolation tasks, and we hope it will draw more attention to the application of SSMs in low-level video tasks.Qualitative comparison. To further validate the practical effectiveness of VFIMamba, we also present a visual comparison with other frame interpolation methods. As illustrated in Figure 4 , the arrows highlight areas where our method excels. VFIMamba demonstrates superior motion estimation and detail preservation in high-motion scenarios compared to other methods. This further substantiates that the incorporation of the S6 model enhances the performance of inter-frame interpolation tasks.Efficiency comparison. To validate the efficiency of VFIMamba, we compared the FLOPs and GPU memory usage required by various high-performance methods (Li et al. (2023) and Lu et al. (2022b) ) as the resolution increases. As shown in Figure 5 , VFIMamba requires significantly fewer FLOPs and GPU memory as the input resolution grows, demonstrating the effectiveness of the S6 model in the VFI task.In this section, we conduct ablation studies using the VFIMamba-S model for efficiency.Effect of the S6 for VFI. As a core contribution of this work, the S6 model balances computational efficiency and high performance for inter-frame modeling. To validate its effectiveness, as shown in (Full Attention). We observed that only removing the S6 model resulted in a parameter reduction of only 0.7M but led to a significant performance drop across various datasets, underscoring the importance of S6. In comparisons with Convolution and Local Attention, we found that although the S6 model is relatively slower due to its multiple scanning directions, it achieves substantial performance improvements. Compared to Full Attention, S6 not only surpasses its performance but also offers faster inference speed and lower memory consumption. In summary, the S6 model indeed achieves a balance between computational efficiency and performance compared to existing models.Frame rearrangement for inter-frame modeling. The rearrangement of input frames is crucial for inter-frame modeling using the S6 model. As analyzed in Section 3.3.2, we posit that interleaved rearrangement is more suitable for VFI tasks, and we provide experimental validation here. As shown in Table 5 , we experimented with two different rearrangement methods in both horizontal and vertical scans. The results demonstrate that using interleaved rearrangement consistently achieves the best performance across all datasets, with significant improvements over other methods. These findings further validate our analysis that interleaved rearrangement offers superior spatiotemporal local modeling capabilities for VFI. Explore different learning strategy. As described in Section 3.4, we proposed a curriculum learning strategy to fully harness the global modeling capabilities of the S6 model. In Figure 6 , we present the performance of different learning strategies over training epochs on both Vimeo90K and X-TEST. In addition to the Vimeo90K Only and Sequential Learning strategies mentioned in Section 3.4, we also compared a baseline approach where the two datasets were directly mixed for training (Mixed Learning). The results indicate that as epochs increase, the Vimeo90K Only strategy improves performance exclusively on Vimeo90K with negligible change on X-TEST. Sequential Learning, while eventually enhancing X-TEST performance, significantly degrades performance on Vimeo90K. Mixed Learning shows a gradual increase in performance on both datasets but fails to achieve competitive results. Our proposed curriculum learning strategy, however, achieves the best performance on both datasets simultaneously by the end of training.To validate the generalization capability of curriculum learning, we also trained the RIFE (Huang et al., 2022) and EMA-VFI (Zhang et al., 2023) from scratch using curriculum learning. As shown in Table 6 , after training, all models maintained their performance on the low-resolution dataset Vimeo90K while significantly improving performance on the X-TEST and SNU-FILM, fully verified the generalization of curriculum learning. Among these, our VFIMamba achieved the most significant improvement and the highest performance ceiling, further demonstrating the potential of the S6 model. 
conclusion: 
 In this paper, we have introduced VFIMamba, the first approach to adapt the SSM model to the video frame interpolation task. To achieve global inter-frame modeling with linear complexity, we devise the Mixed-SSM Block (MSB) for efficient inter-frame modeling using S6. We also explore various rearrangement methods to convert two frames into a sequence, discovering that interleaved rearrangement is more suitable for VFI tasks. Additionally, we propose a curriculum learning strategy to further leverage the potential of the S6 model. Experimental results demonstrate that VFIMamba achieves the state-of-the-art performance across various datasets, in particular highlighting the potential of the SSM model for VFI tasks with high resolution.  
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Training details and evaluation protocols are provided in the appendix A.6 and A.7.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We followed the same experiment protocols as in previously published methods for video frame interpolation; no error bars are included.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Video frame interpolation does not produce new semantic content.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cited the original papers we used and respected their license. License details are listed in Appendix A.8.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [NA] Justification: We have no plans to release the code and models at this time, but they will be open-sourced and well documented after publication. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this work, we introduce VFIMamba, the first video frame interpolation model based on SSMs.Video frame interpolation has wide-ranging applications in real-world video data processing, such as increasing the frame rate of AI-generated videos and generating slow-motion videos. Enhancing performance in various scenarios is crucial. However, as a research-oriented work, we trained our model on a very limited set of datasets (Vimeo90K (Xue et al., 2019) and X-TRAIN (Sim et al., 2021) ), which might result in some degree of overfitting. Consequently, there could be significant artifacts when applied in real-world usage. This issue can be mitigated by training on a more diverse and extensive set of datasets.As the first work to explore the application of SSM models in frame interpolation tasks, we have achieved high performance, but there are still some limitations. First, although our method is much faster than attention-based methods, it still falls short of real-time requirements. Future work on designing a more efficient SSMs would be highly valuable. Second, in this work, we primarily focused on the role of SSM in inter-frame modeling and did not explore its use in the frame generation module. In the future, directly using SSM for generating intermediate frames could also be a promising direction for exploration.To further evaluate the effective receptive field (ERF) of the S6 model in comparison with other efficient models (CNN, Local Attention) for inter-frame modeling, we used the method described by Luo et al. (2016) . Given a specific region in I 0 , we visualized the corresponding receptive fields in I 1 for different methods.As shown in Figure 7 , when the motion between I 0 and I 1 is significant, neither convolution nor local attention can focus on the corresponding region in I 1 before or after training. In contrast, the S6 model exhibits a larger global receptive field even before training, with notable concentration in both horizontal and vertical directions. We attribute this to the sequence rearrangement, where tokens closer together tend to have higher weights, a phenomenon also observed in VMamba (Liu et al., 2024b) .After training, the S6 model's focus becomes more concentrated on the horizontal region of I 1 , aligning better with the specified region in I 0 . This indicates that the S6 model can better capture dynamics even with significant motion between frames. As shown in Figure 8 , we provide more visualization comparisons. VFIMamba demonstrates better visual quality compared to other methods.As shown in Figure 9 , our frame feature extraction consists of multiple convolutional layers and PReLU (He et al., 2015) . The first convolution maps the image from 3 channels to C, with C = 16 for VFIMamba-S and C = 32 for VFIMamba. Each time patch embedding is applied, the image resolution is halved, and the number of channels is doubled. Finally, we obtain the shallow features F i l for each frame.As depicted in Figure 10 , our frame generation includes an iterative intermediate flow estimation, local flow refinement, and appearance refinement using RefinNet. First, the intermediate flow estimation module uses the features F i ssm obtained from inter-frame modeling with MSB for rough flow estimation. Specifically, we follow the design of EMA-VFI (Zhang et al., 2023) + ! , + " # %%& ! , # %%& " RefineNet + # 3×3 Conv, stride=1 PReLU ×3 ( # &→( , ( # '→( IFBlock ×2 ) ,,- & , ) ,,- ' # & , # ' PixelShuffle ×2 concat Downsample Upsample & (→& , & (→' , ' 3×3 Conv, stride=1 PReLU ×3 ) ,,- & , ) ,,- ' Backward Warp Backward Warp ̅ + # ! 16 × $ 16 ! 8 × $ 8Figure 10 : Details of frame generation. IFBlock is adopted from Huang et al. (2022) , which is used optionally to enhance the local detail generation performance.mask M by several convolutional layers. Then, we iteratively estimate the flow residual ∆ f and mask residual ∆M using the F i ssm from the H 8 × W 8 scale. After that, inspired by Jia et al. (2022) , which recognizes that the flow obtained through global inter-frame modeling may be coarse for high-resolution or large-motion scenes, we also introduce the IFBlock (Huang et al., 2022) to further enhance flow accuracy in local details. We then use the predicted motion to backward warp (Huang et al., 2022) the input frames to get the coarse intermediate frame Īt . Finally, we adopt a U-Netlike (Ronneberger et al., 2015) structure to predict the appearance residual using shallow features F i l and inter-frame features F i ssm , resulting in the final frame I t .Training loss We used the same training loss as Zhang et al. (2023) , which is a weighted combination of Laplacian loss (Niklaus & Liu, 2020) and warp loss (Liu et al., 2019) , with weights of 1 and 0.5, respectively.Training setting We used curriculum learning to train our model. For the data from Vimeo90K (Xue et al., 2019) , we randomly cropped the frames from 256 × 448 to 256 × 256. For the data from X-TRAIN (Sim et al., 2021) , since each sample contains 64 consecutive frames, we first randomly select two frames, starting with an interval of 1, which doubles every 50 epochs. Then, we randomly resized the frames from 512 × 512 to S × S , where S is initially 256 and increased by a factor of 1.1 every 50 epochs, and finally cropped them to 256 ×256 for alignment. The larger the resize size, the greater the motion magnitude of the generated data. The batch size for Vimeo90K is 32, and for X-TRAIN it is 8. We then applied time reversal and random rotation augmentations. We used AdamW as our optimizer with β 1 = 0.9, β 2 = 0.999, and a weight decay of 1 × 10 -4 . With warmup for 2,000 steps, the learning rate was gradually increased to 2 × 10 -4 , and then we used cosine annealing for 300 epochs to reduce the learning rate from 2 × 10 -4 to 2 × 10 -5 . Following Jia et al. ( 2022 In our paper, we primarily evaluated our methods on six benchmarks in terms of PSNR/SSIM (Wang et al., 2004 ): Vimeo90K (Xue et al., 2019) , UCF101 Soomro et al. (2012) , SNU-FILM (Choi et al., 2020) , Xiph (Montgomery, 1994) , X-TEST (Sim et al., 2021) , and X-TEST-L (Liu et al., 2024a) .We followed the test procedures of Huang et al. (2022) Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: All the reviewers unanimously recognize the novelty/significance of combining a new architecture (MAMBA) and the VFI task, the thorough experimental evaluation, the significance of a relatively simple curriculum learning strategy for VFI, and strong state-of-the-art performance of the proposed approach. The AC thus recommends to accept this paper. 

The authors are strongly encouraged to incorporate the new results into the final version of this paper.

Reviews: review: paper_summary: paper_summary: The paper introduces Mamba-based video frame interpolation. To fully incorporate the power of Mamba, the paper proposes an interleaving rearrangement method. Using this method, the SSM scans the same location tokens of 2 frames together instead of processing each frame separately. The paper also proposes curriculum learning with large motion data. The experiments show the improvement of using these ideas.
strength_weakness: strength_weakness: I have comments and questions about the final comparisons (Tables 2 and 3). 
>Most methods training models exclusively on the Vimeo90K (Xue et al., 2019). .... (2) Sequential Learning: To mitigate the limitations of training solely on Vimeo90K, some methods (Liu et al., 2024a; Park et al., 2023) further train the model on X-TRAIN (Sim et al., 2021) after initial training on Vimeo90K.

- Based on the description above, the proposed model used both Vimeo90K and X-TRAIN for training, but not all methods use both datasets. It's unclear which models are trained only on Vimeo90K or Vimeo90K+X-TRAIN. The improvement can be due to the increase of training datasets especially since the improvement is marginal compared to some models such as EMA-VFI (for low resolution), AMT-L, AMT-G, and SGM-VFI. I understand that curriculum learning improves performance compared to the simple data mix, but the current tables are not comparable if the trained datasets are not the same across the models. 
    - I ask authors to add which models are trained only on Vimeo90K or Vimeo90K+X-TRAIN in the tables.
    - If some comparable models are trained only on Vimeo90K, can authors add the comparisons of these models with Vimeo90K+X-TRAIN? I'm not sure if it's possible during the rebuttal period. 

- As some comparing models are better or similar to the proposed model, it's difficult to compare which ones are better. Adding the average of each model on the right column would be helpful. 

- How can the interleave rearrangement be efficiently implemented? It would be good to add details of how it is implemented in the paper. Also, is there any speed issue of using the interleave rearrangement instead of the sequential one?
questions: questions: See the weaknesses.
limitations: limitations: The limitation is discussed in the paper and is reasonable.
paper_summary: paper_summary: The paper presents a novel approach for video frame interpolation using Selective State Space Models (S6). The authors introduce VFIMamba, a method designed to efficiently and dynamically model inter-frame information. This method features the Mixed-SSM Block (MSB), which rearranges tokens from adjacent frames in an interleaved manner and applies multi-directional S6 modeling. Additionally, the paper proposes a curriculum learning strategy to progressively improve the model's ability to handle varying motion magnitudes. Experimental results show that VFIMamba achieves state-of-the-art performance on various benchmarks, especially in high-resolution scenarios.
strength_weakness: strength_weakness: Real-time Application: Although VFIMamba achieves high performance, it still falls short of real-time requirements. The paper could benefit from a discussion on potential strategies to improve inference speed.
Performance Gap Analysis: On low-resolution datasets, VFIMamba-S fails to yield second-best scores on most benchmarks and only outperform baselines with comparable FLOPs (e.g., EMA-VFI-S) by a relatively small margin.
questions: questions: 1. What specific optimizations could be applied to VFIMamba to make it suitable for real-time applications? Are there trade-offs between speed and accuracy that need to be considered?
2. Could the authors elaborate on the specific factors contributing to the performance gap, and suggest potential directions to address these issues?
limitations: limitations: The authors have acknowledged several limitations of their work, including the resource-intensive nature of training VFIMamba and the current inability to meet real-time requirements.
The authors suggest future work on designing more efficient SSMs and exploring the application of SSMs in the frame generation module.
paper_summary: paper_summary: This paper introduces a novel video frame interpolation (VFI) method called VFIMamba. VFIMamba is the first method that combines the State-Space Model Mamba with VFI architectures and therefore, it has the advantage of a linearly growing complexity w.r.t. the resolution while maintaining the ability to utilize global receptive fields similar to vision transformers. In order to apply the idea to VFI, architectural modifications have been proposed to handle 2 frames as input. Further, a novel curriculum learning strategy is used to increase the models performance across a large range of motions. The method has been evaluated on multiple datasets and w.r.t. to various other methods, achieving state-of-the-art PSNR values, especially improving the performance for high-resolution frames (2K and 4K).
strength_weakness: strength_weakness: - It is unclear how frames at arbitrary time steps are computed to perform 8x interpolation in table 3. It should also be directly clear from the caption of table 3, that 8x interpolation is evaluated (these details are only in the appendix).
- In general, a lot of important information is in the appendix. It is helpful to add at least a reference from the main paper to the appendix that there is more information, e.g., such as the evaluation and the experiment on generlalization of curriculum learning.
- Unclear how the FLOPs requirement scales with resolution. A plot/table showing the FLOPs compared to the input resolution for VFIMamba and other methods would be interesting to get a better feeling for the scaling w.r.t. frame resolution and a comparison of memory footprints might be interesting.
- Ideally, it should be mentioned in table 2 and 3 on which dataset the other methods have been trained on, especially for methods where the original paper proposed multiple versions such as XVFI.
- Some recent methods especially for high-resolution data are missing in Table 3 such as [A] and [B]. Especially [A] has only been included in Tab. 2 although their focus is also on high-resolution datasets and code for X-Test is directly available: https://github.com/feinanshan/M2M_VFI/blob/main/Test/bench_xtest.py


[A] Hu et al. Many-to-many Splatting for Efficient Video Frame Interpolation, 2022

[B] Nottebaum et al., Efficient Feature Extraction for High-resolution Video Frame Interpolation, 2022.
It would be nice to have longer sequences, and playing them a bit more fluently to get a feeling for the temporal and visual consistency.

Minor: 
- Some colors difficult to see in overlay in Figure 1.
- L. 126, Check sentence “where contains”
questions: questions: - Table 3: There is a discrepancy between many of the reported number in their respective original papers and in this paper on X-TEST for 2k and 4k, e.g.,   EMA-VFI-S reported 30.89dB instead of 29.91dB and BiFormer reported 31.32dB instead 31.18. Based on the experience of the reviewer it is possible to reproduce these numbers using the correct evaluation protocol.

- How is 8x interpolation performed?

- Unclear why there is no FLOPs measurement and no evaluation of SNU-FILM for SoftSplat given that the source code + trained models are publicly available (https://github.com/sniklaus/softmax-splatting)
limitations: limitations: Discussion of limitations is only in the appendix. It would be better to discuss them already in the main paper. 

Additionaly, has the method similar limitations as MAMBA (the base model for this entire work). Therefore, the compute requirements are still relatively high even though they do not have to compute an attention matrix anymore.
paper_summary: paper_summary: Based on the popular S6 model's advantages of linear computational complexity and data-independent modelling capability, this paper applies it to VFI. Specifically, this paper proposes a token rearrangement strategy to learn the information of adjust frames, in addition to introducing a curriculum learning strategy to dynamically learn various motion magnitudes between adjust frames through joint training of vimeo90K and X-TRAIN. The model achieves the highest performance on existing commonly used VFI datasets.
strength_weakness: strength_weakness: 1. It is recommended that the authors can demonstrate the accuracy of the interpolation results more intuitively by visualizing the error maps.

2. The experiments on table 4 are to demonstrate the computational validity of S6 model, and it is necessary to provide flops for the different models.

3. In limitations, the author says that the VFIMamba has faster speed, I would like to know the comparison of the runtime in table 2.

4. Lack of citations essential to the field, such as VFIT, TTVFI, ABME, EDSC, etc.
questions: questions: I would like to see more analysis on the model effectiveness, such as the runtimes in Table 2.
limitations: limitations: The authors discuss methodological limitations in the supplementary material.

Comments: title: response to the rebuttal
comment: We will be sure to further polish the final version based on your helpful suggestions. Thank you so much for your kind recognition of our work.
I thank the authors for the answers, and highly appreciate the extensive additional results and verification of the numbers. Minor detail, it would be nice if in the final version the FLOPs and runtime will also be added for 2K/4K. The additional provided details on training and additional ablations provided in the rebuttal pdf confirm the contribution of the paper and I still recommend the paper for acceptance.
We are truly grateful for your kind recognition of our work.
Thanks to the authors' reply, I raise my score to Accept since I am satisfied with the authors' response regarding performance gap. I am looking forward to the author's future work to further address on accelerating SSM.
We will be sure to incorporate your suggestions into the final version. Thank you so much for your kind acknowledgement of our work.
We will continue to make effort to enhance the efficiency of our model in the future. Thank you so much for your kind recognition of our work.
Thank you for the answers. 
My major concern was the unfair comparisons in Tables 2 and 3 as the paper uses an extra dataset. The authors' answers and additional information in Tables 2 and 3 (also Table 6) resolve my concern. Please add these details to the revised version. Also, please move Table 6 to the main paper; it's an important ablation. 
As the paper includes clear contributions and improvements, I increase my rating.
Thanks to the author's reply, I raise my score to Accept. Looking forward to the author's future work to further address on accelerating SSM.
Dear Reviewers,

Thank you very much again for your valuable service to the NeurIPS community.

As the authors have provided detailed responses, it would be great if you could check them and see if your concerns have been addressed. Your prompt feedback would provide an opportunity for the authors to offer additional clarifications if needed.

Best regards,

AC