VFIMamba: Video Frame Interpolation with State Space Models
Authors: Guozhen Zhang
Chunxu Liu
Yutao Cui
Xiaotong Zhao
Kai Ma
Limin Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Frame Interpolation
State Space Models
Low Level Vision
Primary area: machine_vision
TL;DR: A novel video frame interpolation method that leverages the S6 model for efficient and dynamic inter-frame modeling

Abstract: Inter-frame modeling is pivotal in generating intermediate frames for video frame interpolation (VFI). Current approaches predominantly rely on convolution or attention-based models, which often either lack sufficient receptive fields or entail significant computational overheads. Recently, Selective State Space Models (S6) have emerged, tailored specifically for long sequence modeling, offering both linear complexity and data-dependent modeling capabilities. In this paper, we propose VFIMamba, a novel frame interpolation method for efficient and dynamic inter-frame modeling by harnessing the S6 model. Our approach introduces the Mixed-SSM Block (MSB), which initially rearranges tokens from adjacent frames in an interleaved fashion and subsequently applies multi-directional S6 modeling. This design facilitates the efficient transmission of information across frames while upholding linear complexity. Furthermore, we introduce a novel curriculum learning strategy that progressively cultivates proficiency in modeling inter-frame dynamics across varying motion magnitudes, fully unleashing the potential of the S6 model. Experimental findings showcase that our method attains state-of-the-art performance across diverse benchmarks, particularly excelling in high-resolution scenarios. In particular, on the X-TEST dataset, VFIMamba demonstrates a noteworthy improvement of 0.80 dB for 4K frames and 0.96 dB for 2K frames.

VFIMamba: Video Frame Interpolation with State Space Models
Guozhen Zhangl.2. Chunxu Liu'
Yutao Cui² Xiaotong Zhao²
Kai Ma²
Limin Wang 1.31
State Key Laboratory for Novel Software Technology, Nanjing University
Platform and Content Group (PCG), Tencent
Shanghai AI Lab
Mambaa
Abstract
Inter-frame modeling pivotal in generating intermediate frames for video frame interpolation (VFI). Current approaches predominantly rely on convolution attention-basee models, which often either lack sufficient receptive fields or entail significant computational overheads. Recently, Selective State Space Models (S6) have emerged, tailored specifically for long sequence modeling, offering both linear complexity and data-dependent modeling capabilities. In this paper, we propose VFIMamba, a novel frame interpolation method for efficient and dynamic inter-frame modeling by harnessing the S6 model. Our approach introduces the Mixed-SSM Block (MSB), which initially rearranges tokens from adjacent frames in an interleaved fashion and subsequently applies multi-directional S6 model- ing. This design facilitates the efficient transmission of information across frames while upholding linear complexity. Furthermore, introduce novel currieulum learning strategy that progressively cultivates proficiency in modeling inter-frame dynamics across varying motion magnitudes, fully unleashing the potential of the S6 model. Experimental findings showcase that our method attains state-of-the performance across diverse benchmarks, particularly excelling in high-resolution scenarios. In particular, on the X- X-TEST dataset, VFIMamba demonstrates a note- worthy improvement of 0.80 dB for 4K frames and 0.96 dB for 2K frames.
Introduction
Video Frame Interpolation (VFI), fundamental task in video data processing, gaining substantial attention for its ability to generate intermediate frames between consecutive frames (Liu et al., 2017). Its utility spans many practical applications, including creating slow-motion videos through temporal upsampling (Jiang et al., 2018), enhancing video refresh rates (Reda et al., 2022), and generating novel views (Flynn al., 2016; Szeliski, 1999). VFI typically encompass two primary stages (Zhang et al., 2023): firstly, conducting the inter-frame modeling of input consecutive frames; and secondly, leveraging the acquired information estimate inter-frame motion and generate intermediate frame appearance. In practice, VFI often deals with high-resolution inputs (e.g., 4K) (Sim et al., 2021). which results in significant object displacement and imposes high demands on the large receptive field of the modules that model information between frames. Additionally, since VFI is commonly applied to long-duration videos such as movies, model speed is also of paramount importance. Thus, striking a delicate balance between a sufficient receptive field and fast processing speed in modeling inter-frame information is the key in crafting effective VFI models.
Work done during internship Tencent PCG. *Corresponding author (Imwang (anwang@nju.cdu.cm)
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

X-TEST-L
X-TEST
(2K)
(4K)
X-TEST-L
31.9
31.9
X-TEST
(2K)
30.8
32.9
30.7
Vimeo90K
36,4
35.7
35,5
365
Xiph
(2K)
39.6
35.6
29.8
40
25,4
SNU-FILM (easy)
36,2
30.6
SNU-FILM (extreme)
FILM
EMA-VFI
AMT-G
SNU-FILM (medium)
SNU-FILM (hard)
VFIMamba (Owrs)
Figure 1: Equipped with the S6 model, our VFIMamba achieves the state-of-the-eee performance on benchmarks across different input resolutions.
Current methods for modeling inter-frame information predominantly rely on convolutional neural networks (CNNs) (Liu et al., 2017; Kong et al., 2022; Huang al., 2022) and attention-based models (Lu et al., 2022a; Zhang et al., 2023; Park et al., 2023; Liu et al., 2024a). However, as illustrated in the first three rows of Table 1, these methods either (1) lack flexibility and cannot adaptively model based on the input, (2) do not have sufficient receptive fields to capture inter-frame correlations high resolutions, (3) suffer from prohibitive computational complexity.
On the other hand, Natural Language Processing (NLP) has recently witnessed the emergence of structured state space models (SSMs) (Gu et al., 2021). Theoretically, SSMs combine the benefits of Recurrent Neural Networks (RNNs) and CNNs, leveraging the global receptive field characteristic of RNNs and the computational efficiency inherent in CNNs. One particularly notable SSM is the Selective State Space Model (S6), also known as Mamba (Gu & Dao, 2023), which has garnered significant attention within the vision community. Mamba's novel feature of making SSM parameters tinevarriat (i.e., data detinddeeeddent) enables it to effectively select relevant context within sequences, a crucial factor for enhancing model performance. However, to the best of knowledge, S6 has yet been applied to low-level video tasks.
To address the challenges faced by current VFI models and to explore the potential of the S6 model (Gu & Dao, 2023) in low-level video tasks, we propose VFIMamba, a novel frame interpolation method that adapts the S6 model for efficient and dynamic inter-frame modeling. As shown in Table 1, VFIMamba provides the advantages of global receptive field with linear complexity while maintaining data-dependent adaptability.
Specifically, we introduce the Mixed-SSM Block (MSB) to replace existing modules for inter-frame information transfer. The original S6 model can only process single sequence, so it is necessary to merge tokens from two frames into one sequence for effective inter-frame modeling. After thorough analysis and exploration, we figured out that interleaving tokens from both frames into a "super image" is more suitable for VFI. We then conduct multi multi-directional SSMs on this image to model inter-frame information. This interleaved approach facilitates interactions between adjacent tokens from different frames during sequence modeling and ensures that the intermediate tokens of any pair of tokens in the sequence are from their spatiotemporal neighborhood. By stacking MSB modules, our model effectively handles complex inter-fran frame information exchange. Finally, we use the extracted inter-frame features to estimate motion and generate the appearance of intermediate frames.
While model boasts the advantages listed in Table I, crucial to employ appropriate training strategies to fully exploit its potential. Inspired by Bengio et al. (2009). we propose a novel curriculum learning strategy that progressively teaches the model handle inter-frame modeling across varying motion amplitudes. Specifically, while maintaining training Vimco90K (Xue et al.,

---

Table 1: Comparison of the model design for inter-frame modeling of VFIMamba and existing
methods. VFIMamba enjoys both the advantages of a large receptive field and linear complexity.
Model
Data-dependent
Linear complexity
Global receptive field
Representative method
CNN
X
/
X
RIFE (Huung et al., 2022) SGM-VFI (Liu et al., 2024a)
Attention
X
V
Local Atlention
X EMA-VH VFI (Zhang el al., 2023)
Mamba
V
V
VFIMamba (our work)
2019), incrementally introduce large motion data from X-TRAIN (Sim et al., 2021), increasing the motion amplitude over time. This learning strategy enables VFIMamba to perform well across a wide range of motion amplitudes, thereby fully unleashing the potential of the S6 model.
To validate the effectiveness of VFIMamba across various types of video data, we conduct extensive testing on different benchmarks. As shown in Figure I, VFIMamba achieves the state-of state-of-the-art a-rt (SOTA) performance across diverse datasets. This is particularly evident in high-resolution and large-motion datasets such as X-TEST (Sim et al., 2021) and SNU-FILM (Choi et al., 2020).
Contribution. In summary, the contributions of this paper are as follows: (1) We are the first to adapt the S6 model to the video frame interpolation task. To better adapt the model for this task, we introduce the Mixed-SSM Block (MSB). providing a solid foundation for future architectural exploration in frame interpolation. (2) We propose a novel curriculum learning strategy that incrementally introduces data with varying motion amplitudes, thereby fully harnessing the potential of the S6 model. (3) Our model achieves the state-of-the-art performance across wide range of datasets, potentially sparking interest in the exploration of the S6 model within the video low-level community.
2 Related work
2.1 Video frame interpolation
The performance of VFI methods has seen significant advancements with the emergence of deep learning models. (1) CNNs-based approaches (Bao et al., 2019; Liu et al., 2017: Huang et al., 2022;
Niklaus & Liu, 2018; Choi et al., 2020; Zhu et al., 2024b; Jía et al., 2022; Niklaus et al., 2017:
Kalluri et al., 2023): Initially, DVF (Liu et al., 2017) utilized U-Net-like (Ronneberger et al., 2015) network to model two input frames and predicted the voxel flow for warping the two frames into the intermediate frame. Following this, CixSyn (Niklaus & Liu, 2018) introduced ContextNet and RefineNet, where ContextNet extracts context information from each frame, and RefineNet further refines the coarse intermediate frame produced by warping. RIFE (Huang et al., 2022) proposed novel, efficient framework that employs self elf-distillation to significantly reduce computational load and parameters while maintaining high performance. Due to its simplicity, many convolutional modeling works (Kong et al., 2022; Jia et al., 2022) have improved upon RIFE. (2) Attention-based approaches (Lu et al., 2022a; Zhang et al., 2023; Park al., 2023; Liu al., 2024a): VFIFormer (Lu et al., 2022a) was the first to use attention to model inter-frame information, replacing the encoder part of U-Net with Transformer blocks. After that, EMA-VFI (Zhang et al., 2023) uses Swin-based (Liu et al., 2021) 1) local attention to simultancously capture local appearance and motion information. AMT (Li al., 2023) used a multi-scale cost-volume construction similar to RAFT (Teed & Deng, 2020) to further enhance motion modeling capabilities. BiFormer (Park et al., 2023) introduced quasi-global bilateral attention to further increase the receptive field for large motions. SGM-VFI (Liu et al., 2024a) introduced sparse global matching to model motion between frames. However, current models struggle to balance sufficient receptive fields with computational overhead. In contrast, our method introduces the first interpolation model based on State Space Models (SSMs) (Gu & Dao, 2023) and further pushes the performance boundaries of VFI tasks.
2.2 State space models
In the field of NLP, SSMs (Gu et al., 2021; Smith et al., 2022; Mehta et al., 2022; Fu et al., 2022) have recently emerged as one of the most promising contenders to challenge the dominance of Transformers. The Structured State Space Sequence Model (S4) (Gu et al., 2021) was initially introduced for linear complexity modeling of long sequences. Subsequent works have improved
3

---

Inte-FFane Modeling
Frume Festuee Exurstion
Fact
Mixed-SSM Black Xc
Pack
Mixed-SSM Block XVV
Frune
Modeling
Mixed-SSM Block
F2
DWCCame
Porch Pirbodding
CAB
Figure 2: Overall pipeline of VFIMamba. Firstly, lightweight feature extractor is employed to
encode the input frames into shallow features. Subsequently, we utilize the Mixed-SSM Block (MSB) conduct inter-frame modeling using S6. iterating V times at each scale. Finally, these inter-frame features are leveraged to generate the intermediate frame.
its computational efficiency and model capacity. S5 (Smith et al., 2022) proposed a parallel scan and MIMO SSM, and GSS (Mehta al., 2022) enhanced the model's capability by introducing gated mechanisms. Mamba (S6) (Gu & Dao, 2023) has recently stood out due to its data-dependent parameter generation and efficient hardware implementation, outperforming Transformers in long- sequence NLP tasks, In the visual domain, Vim (Zhu et al., 2024a) was the first to permute 2D images into sequences for global modeling using bidirectional SSMs. Vmamba (Liu al., 2024b) extended to four directions and introduced a hicrarchical structural design. VideoMamba (Li et al., 2024) was the first apply S6 the video domain by permuting all frames into a spatiotemporal sequence. MambalR (Guo et al., 2024) was the first to use the S6 model for image restoration tasks, achieving superior performance over Transformers. In this work, we explore the potential of the S6 model in VFI tasks, validating its effectiveness through detailed analysis and experimentation.
3 Method
3.1
Preliminaries
SSMs are mainly inspired by the continuous linear time-invariant (LTT) systems, which apply an implicit latent state h(t) e R to map a I-dimensional sequence or function x(() e R - y() e R. Specifically, SSMs be formulated as ordinary differential equation (ODB):
h' (1) Ah(t) Bx(1).
(1)
y() Ch(t),
(2)
where contains evolution matrix A NNN projection parameters B e Vx1 and C C RIXV However, it is hard to solve the above differential equation in deep learning settings and needs to be approximated through discretization. Recent SSMs (Gu et al., 2021) propose introduce timescale parameter A to transform the A, B to their discrete counterparts A, B, i.c.,
h, Åh,1 + Bx..
y, = Ch,,
(4)
Ā exp (AA).
(5)
B = (AA) (exp 7))
(6)
The above SSMs are performed for each channel separately and their parameters are data- data-independent, meaning that A, B and C are the same for any input in the same channel, limiting their flexibility in sequence modeling. Mamba (Gu & Dao, 2023) proposes the selective SSMs (S6), which dynamically

---

generate the parameters for each input data X; E R using the entire x;:
B: Bxi
S = Softpluss (Sxx;
where S RNN .S R .Sa !R are linear projection layers. The B, and C; are shared for all channels of x, A contains A of L channels, and A are the same as previous SSMs. By the discretization in equations and 6, Á and become different based on input data.
3.2 Overall pipeline
Given two consecutive frames To. J1 E RRxxwx3 along with a timestep 1, the objective of the frame interpolation task is to generate the intermediate frame E Hxxxx As illustrated in Figure 2, the overall pipeline of VFIMamba consists of three main components: frame feature extraction, inter- frame modeling, and frame generation. Firstly, we employ set of lightweight convolutional layers to independently extract shallow features from each frame, progressively reducing the resolution to facilitate more efficient inter-frame modeling. This process can be formulated as:
Fl = [())
(
where represents the of convolutional layers, and F denotes the extracted low-level feature for 4;. Next, we perform multi-resolution inter-frame modeling using the proposed Mixed-SSM Block (MSB). Each scale comprises N MSBs, and downsampling between scales is achieved through overlapping patch embedding (Wang et al., 2022). We define the resulting inter-frame features as F Fi Finally, we utilize these high-quality mnter-frame features for frame generation, which involves motion estimation between two frames and appearance refinement:
I, = G(F) sam F F.am)
where G denotes the frame generation network. Since this work primarily focuses on exploring the use of SSMs for inter-frame modeling, we largely follow the design from Zhang et (2023) and Huang et al. (2022) for the frame generation components, with detailed specifications provided in the appendix.
3.3 State space models for inter-frame modeling
Effective inter-frame modeling is crucial for frame interpolation tasks (Zhang et al., 2023). Methods such RIFE (Huang al., 2022) and EMA-VFI (Zhang al., 2023) employ simple convolution layers or local attention for anter-frame modeling, achieving high inference speeds but limiting receptive field. Conversely, SGM-VE VFI (Liu et al., 2024a) uses global inter-frame attention for motion estimation, which improves performance but sacrifices efficiency. In this work, we propose to use state space models (SSMs), specifically S6 (Gu & Dao, 2023), to achieve both efficiency and effectiveness in inter-frame modeling.
3.3.1 Mixed-SSM block
To facilitate more efficient inter-frame information exchange globally, we utilize SSMs for inter-frame modeling. As illustrated in Figure 2, we introduce the MiXed-SSM Block (MSB) for integrate the S6 model into VFI frameworks. The overall design of the MSB is analogous Transformer (Vaswani et al., 2017) blocks, but with two pivotal distinctions: (1) We substitute the attention mechanism with an enhanced S6 Block (Gu& Dao, 2023), which could conduct global inter-frame modeling with linear complexity. (2) Drawing inspiration from Guo et al. (2024) and Behrouz et al. (2024), which identified the lack of locality and inter-channel interaction in SSMs, we replace the multilayer perceptron (MLP) with Channel-Attention Block (CAB) (Hu et al., 2018).
The original S6 model is limited processing one-dimensional sequences, necessitating a strategy for scanning the feature maps of two input frames for inter-frame modeling. As depicted in Figure 3, there are two primary methods to rearrange the two frames: sequential rearrangement, where the frames are concatenated into a single super image, and interleaved rearrangement, where the tokens of the two frames are interleaved to form a super image. Regardless of the rearrangement method, following Liu et al. (2024b), the super image can be scanned in four directions: horizontally, vertically, and in their respective reverse directions. The S6 Block is then employed to model each direction independently, and the resulting sequences rearranged and merged back.
5

---

F1
Sequential Rearrangement
Interleaved Rearrangement
Horizontal Scan
12
12
12
13
16
13
13
15
16
16
Vertical Sean
S
8
8
8
11
9
I0
I1
12
10
II
12
12 12
13
L3
15 16
13
13
14
IS
IS
16 16
Figure Visualizations of different rearrangement methods and scan directions. The choice of rearrangement strategy impacts the information flow during inter-frame modeling with S6. For example, consider the 6-th token from la and the 11-th token from /, In sequential rearrangement, the intermediate tokens introduce too many irrelevant tokens, whereas interleaved rearrangement more effectively preserves the spatiotemporal locality.
3.3.2 Analysis on rearrangement strategies
Here, we discuss which rearrangement method better for inter-frame modeling in the context of frame interpolation. First, let us introduce conclusion from (Ali al., 2024): the S6 layers can be approximated as hidden attention layers, with the attention weights given by:
ai Q.K,Hur
(10)
where
Q Sc
Kj- =)=ReLU(SAs))SA Sxy,. HiJ exp( Σ (SAx)))
(11)
REG
S2x030
In this formulation, (ii represents the hidden attention weight of the j-th token xj to the i-th token x in the sequence. Unlike attention, which calculates weights based solely on the information from tokens (Q:) and xy (K,). the S6 model includes Ha,, which encompasses the contextual information between the i-th and j-th tokens the sequence. Based on this conclusion, we observe that in the interleaved rearrangement, the intermediate tokens of any pair of tokens in the sequence are from their spatiotemporal neighborhood. This means that H., incorporates more local modeling, which is beneficial for low-level tasks like frame interpolation. Additionally, the number of intermediate tokens between spatiotemporally adjacent tokens generally smaller in the interleaved rearrangement. spatiotemporally adjacent tokens separated by
In contrast, in the sequential rearrangement, many unrelated tokens the sequence. This can introduce noise and interfere with the modeling of the relationship between these tokens. A specific example can be seen in Figure 3. where the tokens between the 6-th token of the first frame and the 11-th token of the second frame differs significantly between the two rearrangement methods. In summary, we believe that for video frame interpolation, the interleaved rearrangement method is more suitable for better local spatially-aware processing. Our experiments, detailed in Section 4.2, further validate this conclusion.
3.4 Curriculum learning for VFIMamba
Despite the advantageous characteristics of the S6 model, such as data dependence and global receptive field, is crucial to fully exploit its potential through appropriate training strategies. Currently, two main training strategies are employed for frame interpolation: (1) Vimeo90K Only: Most methods training models exclusively on the Vimeo90K (Xue et al., 2019). Although Vimeo90K offers rich variety of video content, analyzed by Liu et al. (2024a) and Kiefhaber et al. (2024),

---

Table 2: Quantitative comparison with SOTA methods on the low-resolution datasets, in terms of PSNR/SSIM (Wang et al., 2004). The best results and the second best results are boldfaced and underlined respectively. FLOPs was calculated for 720p input. * indicates the results copied from Zhang et al. (2023) and Li et al. (2023). In "Training Dataset", "V" stands for the triplet of Vimeo-90K and "X" stands for X-TRAIN. In "Runtime", we evaluate the inference speed of each method on 1024 x 1024 resolution inputs by 2080Ti GPU. In "Average", we calculate the average performance of each method in terms of PSNR and SSIM.
Thenreg
SNE-CH (UTO 2000
Average
FLOPs
Rustims
(Coest 2019)
DAIN (Booe v., 2019
0987
00000 0780
112 9735
3045 97817
5.51
1977
CAIN+ (Thee 200m
10 99.30
75 97U1
31.3003 9:88
1.39
Softeker (Nicasa & Lin, 20209
36 15809906
51.00355557
40:26009911
36 007005966
30.530.00 0335
25 J60LS50A
31.920.5530
0.86
29644
cl.
30333009900
25:560 0276
17003450
MIM VST
35.25 30.20115504
31.596.552
0.26
shrechhong ot al, (177)
3X 61809777
14 5500
10 8000 9901
75 79319997
30. 100 M75
20005501
37.600 9518
11300
15.3
mRNed. (Kong stal, 2017)
99-35
128199997
30510.999 9258
TNTLSSOP
31.95499511
1179
115
IMA VIT G2angg 2019)
36 00.99.2.00 9910
1811 1901
0191
2988
AMILL dt 2023;
36 35009911
35 31
50:358.99133
36 1VV0555
307500054
31395630422
0.58
183.22
AMT-G at.
1200
(lin
3581003155
40.14009990
9795
50.800.9575 9575
SVDU8666
33.96009535
1.75
942.9
VETMonteeS
V+X
5606
20/100 9938
120 9600
30.800 9381
34,000 9535
0.26
178.0
Table 3: Quantitative comparison with SOTA methods on high-resolution datasets. "OOM" indicates "Out of Memory" on V100. All results are obtained through the same evaluation procedure.
Thining
X-TIST
20111
X-TISTLL
Xiph Ohtoegommry. 1990)
Aserage
2K
4K
2X
(Si al. 2031)
X
1500 91-44
10 12:0 01445
00.8233300 8011
2M 017223 8355
54. TAV7-9224 0254
11 201200011
MDM (Bl as.l.. 2022)
32.15 9258
30.99/099136
30:5001.0002
29-732099001
35.440094477
33.921089922
32.340.9151
RIFF:(hlang al. 30233
11.100
10.130.00
1900-4100
53 13-968189440
31 11.0709996
BRRNe (Koogeall
31.75/19147
38.06099150
30.76019933
24/74/055554
36.11009554
54 64.2550895466
32 03.230099661
FLDR (Nouchamm oial., 2022)
X
$1.1200 9002
30. 1044009941
25:50/0.0006
20. 00,3000.50 8200.8879
34 SUC.9200
33 33.66088822
31 11-420550L0
BiFommer (Pulk ot al. 2113)
V+X
9203501-200
30 90336019006
301 1440199900 9455
34 94330993335
33 33.45119555
31 31.5101115
EMA-VEL-S (Zhant 2023)
V
30.910000000
20.910.5951
29.510087755
28 M5-088733
36 36.55099421
54 $4.259090200
31 31.22005983
AMF-L/U eLi al. 2015)
91190091147
31 M115
17:01 9019
36 2710-40003
54 4901
12 919009113
AMIG stall.
32.35/192000 31.2500.9125 31.129.9157
13.90/99178
(Liu 20244)
VIX
29.90/89972
36.5700,9424
34 24123-10011
VFIMaalee5
V-X
9338
31.56/1.9199 31.73/092266
50.500 10.5000.000 9077
36, 72/69494
54.3220.0334
32.950092122
VIIMamte
V+X
33.3611061
32.15/9.9546
33.22/19189
913999991199
37.130:9451
54 54.5809699
33.42/9.926
its motions have limited magnitude. This restriction hampers the model's performance on inputs with large motions or high resolution. (2) Sequential Learning: To mitigate the limitations of training solely on Vimeo90K, some methods (Liu et al., 2024a; Park et al., 2023) further train the model on X-TRAIN (Sim et al., 2021), a dataset characterized by large motions and high-resolution content, after initial training on Vimeo90K. While this approach successfully enhances the model's s performance on high-resolution data, it often leads to the forgetting of the small-motion modeling capabilities acquired from Vimeo90K.
To address these issues and fully exploit the potential of the S6 model, inspired by Bengio et al. (2009), we propose a curriculum learning strategy for learning inter-frame modeling capabilities across varying motion magnitudes while maintaining the ability to model small motions. Specifically, while continuing training on Vimeo90K, we progressively incorporated data from X-TRAIN. The original size of X-TRAIN is 512x to co-train with Vimeo90K, we first resize the frames to S X and then random crop to the same as Vimeo90K. Every T epochs, the resized size S is increased by 10% (starting from 256), and the temporal interval between selected frames is doubled (starting from 2). which means the motion magnitude increases training progresses. This strategy enables the model to gradually learn inter-rrame modeling capabilities across varying motion magnitudes, starting with smaller motions and progressing to larger ones.
4 Experiments
We provide two models: a lightweight model, VFIMamba-S, and a high-performance model, VFI- Mamba. Both models have N = 3; the only difference is that VFIMamba has twice the number of channels as VFIMamba-S. As described in Section 3.4, we employ curriculum learning strategy in which T = 50 and train for 300 epochs in total. More model configurations and training details are provided in the appendix.
7

---

Overlay
RIFE
XVH
BiFormer
AMI-G
EMA-VII
SGM-VE
Ground Truch
(Chrs)
Figure 4: Visualizations from SNU-FILM (Reda et al., 2022) and X-TEST (Sim al., 2021).
VIIMamba
VFIMamba
ViIFermer
15
VFIFormer
AMT
AMT
10
2
GIP
1.87
2,0
256
512
758
1024
512
768
3004
Resolution
Resolation
Figure 5: Comparisons of FLOPs and GPU memory usage with increasing resolution input.
4.1 Comparison with the State-of-the-Art Methods
Quantitative comparison. To validate the versatility of our proposed VFIMamba, evaluated its performance (PSNR/SSIM) (Wang et al., 2004) across a variety of well-known benchmarks with different resolutions. The low-resolution datasets include Vimeo90K (448 x 256) (Xue al., 2019), UCF101 (256x 256) (Soomro et al., 2012), and SNU-FILM (1280 720) (Reda et al., 2022). Notably, SNU-FILM is categorized into four levels of difficulty based on frame intervals: Easy, Medium, Hard, and Extreme. The nigh-resolution datasets include X-TEST (Sim et al., 2021) 1), X-TEST-L (a more challenging subset selected by Liu et al. (2024a)), and Xiph (Montgomery, 1994). Originally, these datasets are in 4K resolution, and following Zhang et al. (2023), we also resize them to 2K for testing.
For 8x interpolation, iterative approach for frame interpolation. Specifically, we first generated an intermediate frame based on the input two frames, and then, using divide-and-conquer strategy, we further divided the first frame and the generated intermediate frame, as well as the generated intermediate frame and the last frame, to iteratively generate the remaining frames.
followed the testing procedure of FILM (Reda et al., 2022) and used an
As shown in Tables 2 and 3, VFIMamba achieves state-of- the-ttr-rrt performance on almost all datasets with FLOPs comparable to efficient models (Kong al., 2022; Zhang al., 2023). Specifically, in large motion scenarios like X-TEST and X-TEST-L, VFIMamba demonstrates a noteworthy improvement compared with previous metod. This excellent performance underscores the potential of the S6 model in frame interpolation tasks, and we hope will draw more attention to the application of SSMs in low-level video tasks.
Qualitative comparison. To further validate the practical effectiveness of VFIMamba, we also present visual comparison with other frame interpolation methods. As illustrated in Figure 4, the arrows highlight areas where method excels. VFIMamba demonstrates superior motion estimation

---

Table 4: Ablation on different models for inter-frame modeling. We the V100 GPU for evaluating
and "OOM" indicates "Out of Memory".
Model
Vimeo90K
X-TEST
SNU-FILM
Params (M)
720p Inference
2K
4K
hard
extreme
Time (ms)
w/o S6
35 62/0099777
28.9400.817
27.12/0.8466
30,41/0.9341
25. 25.14/0.8567
16.1
51
Convolation
35 35.8600.9790
31,58/09167
30.24/0.9044
30.61/0.9365
25.49/0.8631
23,4
55
Local Attention
35, 35.92/0.9790
30,49/0.8917
30.00/0.8845
30.47/0.93338
25.46/0.8625
15.6
59
Full Attention
36,04/0.9798
OOM
30.55/0.9367
25,35/0.8602
15.6
336
S6
36.12/009802
32.84/0.9328
31.73/0.2238
30.80/0.9381
25.59/0.8655
16.8
77
Table 5: Ablation on different rearrangement approachs. "Sequential"" means sequential rearrangement and "Interleaved" represents interleaved rearrangement.
X TEST
FIU-FILM
Horizontal Scan
Vertical Scan
Vimeo-90K
2K
4K
hand
extreme
Sequential
Sequential
35.55/0.9765
28.07/0.8327
26.75/0.8327
30.24/0.9319
25.03/0.8545
Sequential
Interlenved
35.76/0.9784
31.69/0.9226
30,45,00,0778
30.32/0.9342
25.21
Interleaved
Sequential
35.79/0.9785
31.49/0.9221
30.35/0.9053
30.12/0.9331
25.1 1,086602
Interleaved
Interleaved
36.12/0,9802
32.84/0,9328
31.73/0.9238
30,80/0,9381
25,59/0.8655
and detail preservation in high- nigh-motion scenarios compared other methods. This further substantiates that the incorporation of the S6 model enhances the performance of inter-frame interpolation tasks.
Efficiency comparison. To validate the efficiency of VFIMamba, we compared the FL.OPs and GPU memory usage required by various sigh-performance methods (Li et al. (2023) and Lu et al. (2022b)) as the resolution increases. As shown in Figure 5, VFIMamba requires significantly fewer FLOPs and GPU memory the input resolution grows, demonstrating the effectiveness of the S6 model in the VFI task.
4.2 Ablation Study
In this section, we conduct ablation studies using the VFIMamba-S model for efficiency.
Effect of the S6 for VFI. As core contribution of this work, the S6 model balances computational efficiency and high performance for inter-frame modeling. To validate its effectiveness, as shown in Table 4, we experimented by removing the SSM model from the MSB (w/o SSM), replacing the MSB with convolutions from RIFE (Huang et al., 2022) (Convolution), or local inter-frame attention from EMA-VFI (Zhang et al., 2023) (Local Attention), or global inter-frame attention (Liu et al., 2024a) (Full Attention). We observed that only removing the S6 model resulted in a parameter reduction of only 0.7M but led to significant performance drop across various datasets, underscoring the importance of S6. In comparisons with Convolution and Local Attention, we found that although the S6 model is relatively slower due to its multiple scanning directions, it achieves substantial performance improvements. Compared to Full Attention, S6 not only surpasses its performance but also offers faster inference speed and lower memory consumption. In summary, the S6 model indeed achieves a balance between computational efficiency and performance compared to existing models.
Frame rearrangement for inter-frame modeling. The rearrangement of input frames is crucial for inter-frame modeling using the S6 model. As analyzed in Section 3.3.2, we posit that interleaved rearrangement more suitable for VFI tasks, and we provide experimental validation here. As shown in Table 5, we experimented with two different rearrangement methods in both horizontal and vertical scans. The results demonstrate that using interleaved rearrangement consistently achieves the best performance across all datasets, with significant improvements over other methods. These findings further validate our analysis that interleaved rearrangement offers superior spatiotemporal local modeling capabilities for VFI.

---

VimcoFOX Ouly
32.5
Sequential Learning
270
Mixad Lexening
32.0
Loaming
219
31.5
31.0
A0
210
30.5
30.0
29.5
34:50
34.35
35.00
33.25
35.30
35.75
36.00
VimeeSOK (PSNR)
Figure 6: Performance of different learning methods, recorded every 30 epochs. Curriculum learning has the best performance in both the low-resolution and high-resolution benchmarks eventually.
Table 6: Performance of different methods without with curriculum learning.
Curriculum
X-TEST
SNU-FILM
Vimeo90K
Learning
2K
4K
hard
extreme
35.61/0.9797
31.10/0.8972
30.13/0.8927
30.36/0.9375
25.27/0.8601
RIFE
35.60/0.9797
31.40/0.9142
30.23/0.9011
30,47/0.9376
25.38/0.8619
X
36.07/0.9797
30.91/0.9000
29.91/0.8951
30.69/0.9375
25.47/0.8633
EMA VFI-S
36.05)0.9797
31.15/0.9083
29.98/0.8988
30.73/0.9379
25.53/0.8665
36.13/0.9802
30.82/0.8997
29.87/0.8949
30.58/0.9378
25.30/0.8620
VFIMamba-S
16.12/0.9802
32.84/0.9328
31.73/0.9238
30.80/0.9381 25.59/0.8655
Explore different learning strategy. As described in Section 3.4, we proposed a curriculum learning strategy to fully harness the global modeling capabilities of the S6 model. In Figure 6, we present the performance of different learning strategies over training epochs on both Vimeo90K and X-TEST. In addition to the Vimeo90K Only and Sequential Learning strategies mentioned in Section 3.4, we also compared baseline approach where the two datasets were directly mixed for training (Mixed Learning). The results indicate that as epochs increase, the Vimeo90K Only strategy improves performance exclusively on Vimeo90K with negligible change on X-TEST. Sequential Learning, while eventually enhancing X-TEST performance, significantly degrades performance on Vimeo90K. Mixed Learning shows a gradual increase in performance on both datasets but fails to achieve competitive results. Our proposed curriculum learning strategy, however, achieves the best performance on both datasets simultaneously by the end of training.
Generalization of curriculum learning To validate the generalization capability of curriculum learning, we also trained the RIFE (Huang et al., 2022) and EMA-VFI (Zhang et al., 2023) from scratch using curriculum learning, As shown in Table 6, after training, all models maintained their performance on the low-resolution dataset Vimeo90K while significantly improving performance on the X-TEST and SNU-FILM, fully verified the generalization of eurriculum learning. Among these, our VFIMamba achieved the most significant improvement and the highest performance ceiling, further demonstrating the potential of the S6 model.
5 Conclusion
In this paper, we have introduced VFIMamba, the first approach to adapt the SSM model to the video frame interpolation task. To achieve global inter-frame modeling with linear complexity, we devise the Mixed-SSM Block (MSB) for efficient fraer- modeling using S6. We also explore various rearrangement methods convert two frames into sequence, discovering that interleaved rearrangement is more suitable for VFI tasks. Additionally, we propose curriculum learning strategy further leverage the potential of the S6 model. Experimental results demonstrate that VFIMamba achieves the tate-of-the-arr performance across various datasets, in particular highlighting the potential of the SSM model for VFI tasks with high resolution.
10

---

Acknowledgement
This work supported by the National Key R&D Program China (No. 20222D0160900 the National Natural Science Foundation of China (No. 62076119), the Fundamental Research Funds for the Central Universities (No. 0202143801199 1998, the Nanjing University- China Mobile Communications Group Co., Ltd. Joint Institute, and the Collaborative Innovation Center of Novel Software Technology and Industrialization.
References
Ali, A.. Zimerman, I., and Wolf, L. The hidden attention of mamba models. arXiv preprint arXiv:2403.01590, 2024.
Bao, W., Lai, W.-S., Ma, C., Zhang, X., Gao, 7., and Yang, M.-H. Depth-aware video frame interpolation. In Proceedings of the IEEEJCVE Conference on Computer Vision and Pattern Recognition, pp. 3703-3712, 2019.
Behrouz, A., Santacatterina, M., and Zabih, R. Mambamixer: Efficient selective state space models with dual token and channel selection. arXiv preprint arXiv:2403.19888 2024.
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. International Conference Machine Learning, pp. 41-48, 2009.
Choi, M., Kim, H., Han, B., Xu, N., and Lee, K. M. Channel attention is all you need for video frame interpolation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 10663-10671, 2020.
Flynn, J., Neulander, I., Philbin, J.. and Snavely, N. Deepstereo: Learning predict new views from the world's imagery. In Proceedings of the IEEEJCVF Conference on Computer Vision and Pattern Recognition, pp. 5515-5524, 2016.
Fu, D. Y., Dao, T., Saab, K. K., Thomas, A. W., Rudra, A., and Re, C. Hungry hungry hippos: Towards language modeling with state space models. In International Conference on Learning Representations, 2022.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. 2023.
Gu, A., Goel, K., and Re, C. Efficiently modeling long sequences with structured state spaces. In International Conference Learning Representations, 2021.
Guo, H., Li, J., Dai, T., Ouyang, Z., Ren, X., and Xia, S.-T. Mambair: A simple baseline for image restoration with state-space model. arXiv preprint arXiv:2402.15665 2024.
He, K., Zhang, X., Ren, S., and Sun, Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1026-1034, 2015.
Hu, Shen, L., and Sun, G. Squeeze-and-excitatio networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 7132-7141, 2018.
Hu, P., Niklaus, S., Sclaroff, S., and Saenko, K. Many-to-many splatting for efficient video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3553 -5622, 2022.
Huang, Z., Zhang, T., Heng, W., Shi, B., and Zhou, S. Real-time intermediate flow estimation for video frame interpolation. In European Conference on Computer Vision, pp. 624-642. Springer, 2022.
Jia, 7.,, Lu, Y., and Li, H. Neighbor correspondence matching for flow-hased video frame synthesis. In ACM MM, pp. 5389-5397, 2022.
11

---

Jiang, H., Sun, D., Jampani, V., Yang, M.-H., Learned Miller,,,, E.. and Kautz, J. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9000-9008, 2018.
Kalluri, T., Pathak, D., Chandraker, M., and Tran, D. Flavr: Flow- Plow-agnostic video representations for fast frame interpolation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2071 2071-2082, 2023.
Kiefhaber, S., Niklaus, S., Liu, F., and Schaub-Meyer, S. Benchmarking video frame interpolation. arXiv preprint 2rXiv:2403.1777 17128, 2024.
Kong, L., Jiang, B., Luo, D., Chu, W., Huang, X., Tai, Y., Wang, C.,, and Yang, J. Ifrett: Intermediate feature refine network for efficient frame interpolation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pp. 1969-1978, 2022.
Lee, H., Kim, T., Chung, T.-y., Pak, D.,, Ban, Y., and Lee, S. Adacof: Adaptive collaboration of flows for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5316-5325, 2020.
Li, K., Li, X., Wang. Y., He, Y., Wang, Y., Wang, L.., and Qiao, Videomamba: State space model for efficient video understanding. arXiv preprint arXiv:2403.06977 069777 2024.
Li, Z., Zhu, Z.-L., Han, L.-H., Hou, Q., Guo, C.-L., and Cheng, M.-M. Amt: All-pairs multi- transforms for efficient frame interpolation. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 9801-9810, 2023.
Liu. C., Zhang, G., Zhao, R., and Wang, L. Sparse global matching for video frame interpolation with large motion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024a.
Liu, P., Lyu, M., King, I., and Xu, Selflow: Self-supervised learning of optical flow. In Proceedings of the IEEEJCVF Conference on Computer Vision and Pattern Recognition, pp. 4571-45800 2019.
Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y. Vmamba: Visual state space model. arXiv preprint arXiv:2401.1016 2024b.
Liu, Z., Yeh, R. A., Tang, X., Liu, Y., and Agarwala, A. Video frame synthesis using deep voxel flow. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 46311471, 2017.
Liu, Z.,, Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierar- chical vision transformer using shifted windows. Proceedings of the IEEEJCVF International Conference Computer Vision, pp. 10012- 10022, 2021.
Lu, L., Wu, R., Lin, H., Lu, J., and Jia, J. Video frame interpolation with transformer. In Proceedings of the IEEEJCVF Conference on Computer Vision and Pattern Recognition, pp. 3532-3542, 2022a.
Lu, L.,. Wu, R., Lin, H., Lu, and Jia, J. Video frame interpolation with transformer. In Proceedings of the IEEE|CVF Conference on Computer Vision and Pattern Recognition, pp. 3532-3542, 2022b.
Luo, W., Li, Y., Urtasun, R., and Zemel, R. Understanding the effective receptive field in deep convolutional neural networks. Advances in Neural Information Processing Systems, 29, 2016.
Mehta, H., Gupta, A., Cutkosky, A., and Neyshabur, B. Long range language modeling via gated state spaces, arXiv preprint arXiv:2206. 22244422.0..9 2022.
Montgomery, C.
Xiph.org video test media (derf's collection). In On- line, https:s//meii.iipppp... hrw.iiiooorvvio/ddrf... 1994.
Niklaus, S. and Liu, F. Context-aware synthesis for video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1701-1710, 2018.
Niklaus, S. and Liu, F. Softmax splatting for video frame interpolation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5437-5446, 2020.
12

---

Niklaus, S., Mai, L., and Liu, F. Video frame interpolation via adaptive separable convolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 261-270 2017.
Nottebaum, M., Roth, S., and Schaub-Meyer, S. Efficient feature extraction for high-resolution video frame interpolation. In British Machine Vision Conference BMVC, 2022.
Park, J., Kim, J., and Kim, C.-S. Biformer: Learning bilateral motion estimation via bilateral transformer for 4k video frame interpolation. In Proceedings of the IEEEJCVF Conference on Computer Vision and Pattern Recognition, pp. 1868-1577, 2023.
Reda, F., Kontkanen, J., Tabellion, E., Sun, D., Pantofaru, C., and Curless, B. Film: Frame interpolation for large motion. In European Conference on Computer Vision, pp. 250-266. Springer, 2022.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted InterventionMMICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp- 234-241. Springer, 2015.
Sim, H., Oh, J., and Kim, M. Xvfi: extreme video frame interpolation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14499814498, 2021.
Smith, J. T., Warrington, A., and Linderman, S. Simplified state space layers for sequence modeling. In International Conference on Learning Representations, 2022.
Soomro, K., Zamir, A. R., and Shah, M. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint 12Xiv:1212 12222402 2012.
Szeliski, R. Prediction error as a quality metric for motion and stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision, volume 2, pp. 781-788. IEEE, 1999.
Teed, Z. and Deng, J. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference Computer Vision, pp. 402-419. Springer, 2020.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, Attention all you need. Advances in Neural Information Processing Systems, 30, 2017.
Wang, W. Xie, E., Li, X., Fan, D.-P., Song, K., Liang, D., Lu, T., Luo, P., and Shao, L. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415 424, 2022.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 13(4):600-612, 2004.
Xue, T., Chen, B., Wu, J., Wei, D., and Freeman, W. T. Video enhancement with task-oriented flow. International Journal of Computer Vision, 127:1106-11 125, 2019.
Zhang, G., 7hu, Y., Wang, H., Chen, Y., Wu, G., and Wang, L.. Extracting motion and appearance via inter-frame attention for efficient video frame interpolation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5682-5692, 2023.
Zhu, L.. Liao, B., Zhang, Q. Wang, X., Liu, W., and Wang, X. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.094. 2024a.
Zhu, Y., Zhang, G., Tan, J., Wu, G., and Wang, L. Dual detrs for multi-label temporal action detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2024b.
13

---

A Appendix
Convottiinn (Befare)
Local Anceticn (Before)
Our So (Befnee)
Cunvolation (ABer)
Local Amemios (Aller)
Our 56 (Afle)
Figure 7: Visualizations of Effective Receptive Field (ERF) (Luo et al., 2016) of different models before and after training. We utilize the red area in Ia inspect its corresponding ERF in /, S6 model has a significantly larger receptive field and becomes more accurate after training.
A.1 Broader impact
In this work, we introduce VFIMamba, the first video frame interpolation model based on SSMs. Video frame interpolation has wide-ranging applications in real-world video data processing, such as increasing the frame rate of Al- Al-generated videos and generating slow-motion videos. Enhancing performance in various scenarios crucial. However, as a research-oriest model on very limited set of datasets (Vimeo90K (Xue et al., 2019) and X-TRAIN (Sim et al., 2021)), which might result some degree of overfitting. Consequently, there could significant artifacts when applied in real-world usage. This issue can be mitigated by training on more diverse and extensive set datasets.
work, trained our
A.2 Limitations and future work
As the first work explore the application SSM models in frame interpolation tasks, have achieved high performance, but there are still some limitations. First, although our method is much faster than aitention-based methods, it still falls short of real-time requirements. Future work on designing more efficient SSMs would be highly valuable. Second, in this work, we primarily focused on the role of SSM inter-frame modeling and did explore its use in the frame generation module. In the future, directly using SSM for generating intermediate frames could also be a promising direction for exploration.
A.3
Visualizations on effective receptive field
To further evaluate the effective receptive field (ERF) of the S6 model in comparison with other efficient models (CNN, Local Attention) for inter-frame modeling, we used the method described by Luo al. (2016). Given specific region in fo. visualized the corresponding receptive fields in /j for different methods.
As shown in Figure 7, when the motion between Io and I is significant, neither convolution nor local attention can focus on the corresponding region in J before or after training. contrast, the S6 model exhibits larger global receptive field even before training, with notable concentration in both horizontal and vertical directions. We attribute this to the sequence rearrangement, where tokens closer together tend to have higher weights, phenomenon also observed in VMamba (Liu et al., 2024b).
After training, the S6 model's focus becomes more concentrated on the horizontal region of I, aligning better with the specified region in f-- This indicates that the model can better capture dynamics even with significant motion between frames.
14

---

AARAAAAA
Overaay
VHIMamba
Figure 8: More Visualizations from SNU-FILM (Reda al., 2022) and X-TEST (Sim al., 2021).
Frume Feature Extrucion
1
Block
Block
Pauc
Cene Block
11/2xw
3x2 Canv, sustee1
3x3 Conv. stnde-2
3x3 Conss ariden
PReel
PReeUU
PRo.UCC
3x3
Figure 9: Details of frame feature extraction. The same color represents the same block structure.
A.4
More qualitative comparison
As shown in Figure 8, we provide more visualization comparisons. VFIMamba demonstrates better visual quality compared to other methods.
A.5
Model details
A.5.1
Frame feature extraction
As shown in Figure 9, our frame feature extraction consists of multiple convolutional layers and PReLU (He et al., 2015). The first convolution maps the image from channels C, with C 16 for VFIMamba-S and C = 32 for VFIMamba. Each time patch embedding is applied, the image resolution is halved, and the number of channels doubled. Finally, we obtain the shallow features F for each frame.
A.5.2 Frame generation
As depicted in Figure 10, our frame generation includes iterative intermediate flow estimation, local flow refinement, and appearance refinement using RefinNet. First, the intermediate flow estimation module uses the features Fi obtained from inter-frame modeling with MSB for rough flow estimation. Specifically, we follow the design of EMA-VFI (Zhang et al., 2023), first utilizing features F Sum from the H W scale and the original image I, for predicting the flow ff and occlusion
T5
15

---

Froms Genention
Purm Fumm 1
Incuelit Exiration
bremedudee Recibel
Woep
Bianation
Ir
Kin Flce
PludStullle
3 Can.eeeeee
Saro S
PadBSadlee
3K3 mndeen
Qparpl
A/. AM
Darmnample
Downncrple
4.
Figure 10: Details of frame generation. IFBlock is adopted from Huang et al. (2022), which is used optionally to enhance the local detail generation performance.
mask M by several convolutional layers. Then, we iteratively estimate the flow residual Af and mask residual AM using the F from the 8 scale. After that, inspired by Jia et al. (2022), which recognizes that the flow obtained through global inter-frame modeling may be coarse for high-resolution or large-motion scenes, we also introduce the IFBlock (Huang al., 2022) further enhance flow accuracy in local details, We then use the predicted motion to backward warp (Huang al.. 2022) the input frames to get the coarse intermediate frame I, Finally, we adopt a U-Net- like (Ronneberger et al., 2015) structure to predict the appearance residual using shallow features F) and frter-frame features Fy sam resulting in the final frame 4.
A.6 Training details
Training loss We used the same training loss as Zhang et al. (2023), which a weighted combina- tion of Laplacian loss (Niklaus & Liu, 2020) and warp loss (Liu et al., 2019), with weights of 1 and 0.5, respectively.
Training setting We used curriculum learning to train model. For the data from Vimeo90K (Xue et al., 2019), we randomly cropped the frames from 256 x 448 to 256 x 256. For the data from X-TRAIN (Sim et al., 2021), since each sample contains 64 consecutive frames, first randomly select two frames, starting with an interval of 1, which doubles every 50 epochs. Then, we randomly resized the frames from 512 512 to S x S, where S is initially 256 and increased by factor of 1.1 every 50 epochs, and finally cropped them to 256 x256 for alignment. The larger the resize size, the greater the motion magnitude of the generated data. The batch size for Vimeo90K 32, and for X-TRAIN it is 8. We then applied time reversal and random rotation augmentations. We used AdamW as our optimizer with 61 = 0.9, B2 0.999, and weight decay of x 10 With warmup for 2,000 steps, the learning rate was gradually increased to 2 x 10-4, and then we used cosine annealing for 300 epochs to reduce the learning rate from 10 4 2 x 10-5 Following Jia et al. (2022); Park et al. (2023); Liu et al. (2024a), we also trained the IFBlock separately on Vimeo90K for 100 epochs with same training setting to further improve the accuracy of local optical flow at high resolutions. The same procedure is followed for all ablation experiments.
Training time VFIMamba and VFIMamba-S were both trained on NVIDIA 32GB V100 GPUs. Training VFIMamba-S takes about 38 hours, while training VFIMamba takes about 108 hours.
A.7 Evaluation protocols
In our paper, we primarily evaluated our methods on six benchmarks in terms of PSNR/SSIM(Wang et al., 2004): Vimeo90K (Xue et al., 2019), UCF101 Soomro al. (2012), SNU-FILM (Choi et al., 2020), Xiph (Montgomery, 1994), X-TEST (Sim et al., 2021), and X-TEST-L (Liu et al., 2024a).
We followed the test procedures of Huang et al. (2022) for Vimeo90K and UCF101, Kong et al. (2022) for SNU-FILM, Niklaus Liu (2020) for Xiph, Reda (2022) for X-TEST with iterative 8x frame interpolation, and Liu et al. (2024a) for X-TEST-L with largest interval interpolation.
16

---

Table Licenses and URLs for every benchmark, code, and pretrained models used in this paper.
Assets
License
URL
http:://tthub..ommaac. L/toooo
ICCF10
noe-commencial research and oducational purposes
h/ips:///ttub.... com/lix 1991 pyttoocch-vvoell-eee flow
Benchmarks
SNU-FILM
MIT license
https//gichub..om//eee howmm//ygggahyyCC... CAIN
(4I)
research and education
cony OnngniinnngCC hrpp:.//media xiph siph.org/idooydert)
Xiph
freely celistributbbll
XVEI
for research and education only
hhtp://gitbbbbl wwt.ub.coom/iiiiiieeoe wtt.:iabbooeeiiyynggooXXXXX
FILM
Apache- 2.0 liceuse
hmpps//iibub com/gooogle htwp:///wooosot//////mmmmemooo/o
MIT
https//giibbb
www.:://..w/wwweeCCCCCCCCC RIFE
Codes and
FRNN
MIT license
Pretrained Models
BiFormer
Apache- 2.0 license
EMA VII
Apache 2.0 license
VFI
https///ittuu
SGM VFI
Apache- 2.0 license
http://sttaa.com//CCCCUUUUUUO
A.8 License of datasets and pre-trained models
All the dataset we used in the paper are commonly used datasets for academic purpose. All the licenses of the used benchmark, codes, and pretrained models are listed in Table 7.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: This work mainly focuses exploring the potential of S6 model in VFI, and the abstract and introduction document our methods and contributions in detail.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We provide discussion of limitations in the appendix A.2.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide model details in Section 3.1-3.4 and appendix A.5. Training details and evaluation protocols are also provided the appendix A.6 and A.7. The training code and models will be open-sourced after publication.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We have plans to open-source the source code at this time, but the training code and models will be open-sourced after publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
wwtesSubmiiiiiiii ssionPolccy cy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/public/gut tttpl/icggiiee//eeeeeeeeee /eed//CCeeubbii ssi ssionPolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training details and evaluation protocols are provided in the appendix A.6 and A.7.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We followed the same experiment protocols as in previously published methods for video frame interpolation; no error bars are included.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide setting of compute resources in the appendix A.6.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurrpgs httppp/ss.ccpclllccchcccccuuueee
Answer:
[Yes]
Justification: We've read carefully.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the broader impacts of our work in the appendix A.I.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
21

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Video frame interpolation does produce new semantic content.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cited the original papers used and respected their license. License details are listed in Appendix A.8.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We have plans to release the code and models at this time, but they will be open-sourced and well documented after publication.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not involved.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
23

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "B0yCtItXlZ", "reviewer": "Reviewer_A9TL", "summary": "The paper introduces Mamba-based video frame interpolation. To fully incorporate the power of Mamba, the paper proposes an interleaving rearrangement method. Using this method, the SSM scans the same location tokens of 2 frames together instead of processing each frame separately. The paper also proposes curriculum learning with large motion data. The experiments show the improvement of using these ideas.", "questions": "See the weaknesses.", "limitations": "The limitation is discussed in the paper and is reasonable.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper includes three contributions: 1) Use of Mamba for image interpolation, 2) Interleaved rearrangement, and 3) curriculum learning. \n- The idea is simple but effective for video frame interpolation. All these contributions are justified throughout the paper (especially Sections 3 and 4). \n- The paper is easy to follow.", "weaknesses": "I have comments and questions about the final comparisons (Tables 2 and 3). \n>Most methods training models exclusively on the Vimeo90K (Xue et al., 2019). .... (2) Sequential Learning: To mitigate the limitations of training solely on Vimeo90K, some methods (Liu et al., 2024a; Park et al., 2023) further train the model on X-TRAIN (Sim et al., 2021) after initial training on Vimeo90K.\n\n- Based on the description above, the proposed model used both Vimeo90K and X-TRAIN for training, but not all methods use both datasets. It's unclear which models are trained only on Vimeo90K or Vimeo90K+X-TRAIN. The improvement can be due to the increase of training datasets especially since the improvement is marginal compared to some models such as EMA-VFI (for low resolution), AMT-L, AMT-G, and SGM-VFI. I understand that curriculum learning improves performance compared to the simple data mix, but the current tables are not comparable if the trained datasets are not the same across the models. \n    - I ask authors to add which models are trained only on Vimeo90K or Vimeo90K+X-TRAIN in the tables.\n    - If some comparable models are trained only on Vimeo90K, can authors add the comparisons of these models with Vimeo90K+X-TRAIN? I'm not sure if it's possible during the rebuttal period. \n\n- As some comparing models are better or similar to the proposed model, it's difficult to compare which ones are better. Adding the average of each model on the right column would be helpful. \n\n- How can the interleave rearrangement be efficiently implemented? It would be good to add details of how it is implemented in the paper. Also, is there any speed issue of using the interleave rearrangement instead of the sequential one?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your feedback on the experimental and implementation details of our work. We would like to provide the following responses:\n\n**Q.1** *Regarding which models are trained only on Vimeo90K or Vimeo90K+X-TRAIN in the tables.*\n\n**R.1** Thank you for your suggestion. We would like to kindly remind you that the recent published papers (e.g., BiFormer [1] and SGM [2]) also did not specify the exact training dataset used for the compared methods. We have followed their settings, but **we agree that adding this detail will make the comparison more informative for the readers**. We will add the relevant information to Tables 2 and 3 as shown in the PDF of the global response.\n\n**Q.2** *Add the comparisons of these Vimeo90K-only models trained on Vimeo90K+X-TRAIN.*\n\n**R.2** As discussed in Table 6 and Appendix A.4 of the paper, we have already retrained the RIFE [3] and EMA-VFI-S [4] models on Vimeo90K+X-TRAIN. The results show that through curriculum learning, these two methods have achieved performance improvements on the high-resolution dataset, while maintaining their performance on Vimeo90K, demonstrating the generalization of our proposed curriculum learning training strategy. Meanwhile, **VFIMamba still exhibits the best performance under the same training setting**, indicating the higher upper bound of the VFIMamba structure.\n\n**Q.3** *Add the average of each model on the right column.*\n\n**R.3** Thanks for your suggestion. We will add the relevant information to Tables 2 and 3 as shown in the PDF of the global response.\n\n**Q.4** *Regarding the implementation of the interleave rearrangement.*\n\n**R.4** The implementation of the interleave rearrangement is very simple and efficient, which can be achieved through the following few lines of PyTorch code:\n\n```python\ndef interleave_merge(self, x1, x2):\n    # x1 is the first frame, x2 is the second\n    B, C, H, W = x1.shape\n    N = H * W\n    # Flatten the 2D tokens and transpose to B x N x C format\n    x1 = x1.view(B, C, -1).transpose(1, 2)\n    x2 = x2.view(B, C, -1).transpose(1, 2)\n    # Concatenate in the C dimension to get B x N x 2C, and then reshape to get B x 2N x C, and PyTorch will automatically interleave the tokens of the two frames\n    x = torch.cat([x1, x2], dim=-1).reshape(B, 2*N, C)\n    # Reshape back to B x C x 2N\n    return x.transpose(1, 2).contiguous()\n```\nBoth the interleave rearrangement and sequential rearrangement only involve reshaping and rearranging the token, which are linear-time operations. Therefore, the running efficiency of interleave rearrangement and sequential rearrangement can be considered approximately the same.\n\n> [1]Park, Junheum, Jintae Kim, and Chang-Su Kim. \"Biformer: Learning bilateral motion estimation via bilateral transformer for 4k video frame interpolation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n> [2]Liu, Chunxu, et al. \"Sparse Global Matching for Video Frame Interpolation with Large Motion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n> [3]Huang, Zhewei, et al. \"Real-time intermediate flow estimation for video frame interpolation.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n> [4]Zhang, Guozhen, et al. \"Extracting motion and appearance via inter-frame attention for efficient video frame interpolation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "deonE9ZGtd", "reviewer": "Reviewer_Bwpp", "summary": "The paper presents a novel approach for video frame interpolation using Selective State Space Models (S6). The authors introduce VFIMamba, a method designed to efficiently and dynamically model inter-frame information. This method features the Mixed-SSM Block (MSB), which rearranges tokens from adjacent frames in an interleaved manner and applies multi-directional S6 modeling. Additionally, the paper proposes a curriculum learning strategy to progressively improve the model's ability to handle varying motion magnitudes. Experimental results show that VFIMamba achieves state-of-the-art performance on various benchmarks, especially in high-resolution scenarios.", "questions": "1. What specific optimizations could be applied to VFIMamba to make it suitable for real-time applications? Are there trade-offs between speed and accuracy that need to be considered?\n2. Could the authors elaborate on the specific factors contributing to the performance gap, and suggest potential directions to address these issues?", "limitations": "The authors have acknowledged several limitations of their work, including the resource-intensive nature of training VFIMamba and the current inability to meet real-time requirements.\nThe authors suggest future work on designing more efficient SSMs and exploring the application of SSMs in the frame generation module.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: The introduction of the S6 model into video frame interpolation tasks is a novel contribution. The use of Mixed-SSM Blocks and the interleaved token arrangement are creative solutions to enhance inter-frame modeling.\nQuality: The paper presents thorough experiments and comparisons with state-of-the-art methods. The quantitative results demonstrate significant improvements in performance, particularly in high-resolution and large-motion scenarios.\nClarity: The methodology is clearly explained, with detailed descriptions of the proposed model components and training strategies. The visualizations and tables effectively support the claims made in the paper.\nSignificance: The VFIMamba method addresses key challenges in video frame interpolation, such as the need for large receptive fields and efficient computation.", "weaknesses": "Real-time Application: Although VFIMamba achieves high performance, it still falls short of real-time requirements. The paper could benefit from a discussion on potential strategies to improve inference speed.\nPerformance Gap Analysis: On low-resolution datasets, VFIMamba-S fails to yield second-best scores on most benchmarks and only outperform baselines with comparable FLOPs (e.g., EMA-VFI-S) by a relatively small margin.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the recognition and suggestions regarding our work, and our responses to your questions are as follows:\n\n**Q.1** *What specific optimizations could be applied to VFIMamba to make it suitable for real-time applications? Are there trade-offs between speed and accuracy that need to be considered?*\n\n**R.1** Thank you for this insightful question. The running speed of SSMs is mainly related to two aspects: the length of the input sequence and the number of scans performed on the sequence. Regarding the sequence length, we can reduce the number of tokens by fusing the neighboring tokens in the spatio-temporal position (e.g., through ToMe [1]). However, this may also have an impact on the performance of fine-grained spatio-temporal modeling. As for the number of scans, we currently follow VSSM [2] and use 4 scan directions for modeling. In the future, we can reduce the number of scans by dynamically selecting the necessary scan directions for video frame interpolation.\n\n**Q.2** *Could the authors elaborate on the specific factors contributing to the performance gap, and suggest potential directions to address these issues?*\n\n**R.2** Thank you for your question. First, we would like to kindly remind you that except for EMA-VFI-S, VFIMamba-S has a significant performance advantage over other models with similar FLOPS (AdaCof, XVFI, M2M, RIFE, etc.). As for why the performance gap with EMA-VFI-S cannot be widened, EMA-VFI-S belongs to the local attention-based method, and as shown in Table 1 of the paper, the biggest advantage of Mamba over local attention-based methods is the global receptive field, which may not have a significant impact in low-resolution cases. In the future, perhaps combining SSMs with some more fine-grained local modeling methods can further improve the performance at low resolutions.\n\n> [1] Bolya, Daniel, et al. \"Token merging: Your vit but faster.\" arXiv preprint arXiv:2210.09461 (2022).\n\n> [2] Liu, Yue et al. “VMamba: Visual State Space Model.” ArXiv abs/2401.10166 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3YYKr0H4gU", "reviewer": "Reviewer_9Hn5", "summary": "This paper introduces a novel video frame interpolation (VFI) method called VFIMamba. VFIMamba is the first method that combines the State-Space Model Mamba with VFI architectures and therefore, it has the advantage of a linearly growing complexity w.r.t. the resolution while maintaining the ability to utilize global receptive fields similar to vision transformers. In order to apply the idea to VFI, architectural modifications have been proposed to handle 2 frames as input. Further, a novel curriculum learning strategy is used to increase the models performance across a large range of motions. The method has been evaluated on multiple datasets and w.r.t. to various other methods, achieving state-of-the-art PSNR values, especially improving the performance for high-resolution frames (2K and 4K).", "questions": "- Table 3: There is a discrepancy between many of the reported number in their respective original papers and in this paper on X-TEST for 2k and 4k, e.g.,   EMA-VFI-S reported 30.89dB instead of 29.91dB and BiFormer reported 31.32dB instead 31.18. Based on the experience of the reviewer it is possible to reproduce these numbers using the correct evaluation protocol.\n\n- How is 8x interpolation performed?\n\n- Unclear why there is no FLOPs measurement and no evaluation of SNU-FILM for SoftSplat given that the source code + trained models are publicly available (https://github.com/sniklaus/softmax-splatting)", "limitations": "Discussion of limitations is only in the appendix. It would be better to discuss them already in the main paper. \n\nAdditionaly, has the method similar limitations as MAMBA (the base model for this entire work). Therefore, the compute requirements are still relatively high even though they do not have to compute an attention matrix anymore.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Combination of an emerging alternative to transformers (MAMBA) with VFI methods that allow for higher resolution frame interpolation due to linear complexity growth. The paper is interesting to read.\n - Discussion and evaluation of different sequence arrangements.\n- Introduction of a relatively simple curriculum learning strategy for VFI with experiments confirming that this strategy is beneficial for VFI methods in general when having to deal with large motions.\n- Reaches new state-of-the-art performance\n- Exhaustive ablation studies proving the effectiveness for each of their introduced modules\n- Supplement contains a video with qualitative comparisons, although only a few short sequences and given the coarse time steps, it is difficult to clearly judge the temporal consistency.", "weaknesses": "- It is unclear how frames at arbitrary time steps are computed to perform 8x interpolation in table 3. It should also be directly clear from the caption of table 3, that 8x interpolation is evaluated (these details are only in the appendix).\n- In general, a lot of important information is in the appendix. It is helpful to add at least a reference from the main paper to the appendix that there is more information, e.g., such as the evaluation and the experiment on generlalization of curriculum learning.\n- Unclear how the FLOPs requirement scales with resolution. A plot/table showing the FLOPs compared to the input resolution for VFIMamba and other methods would be interesting to get a better feeling for the scaling w.r.t. frame resolution and a comparison of memory footprints might be interesting.\n- Ideally, it should be mentioned in table 2 and 3 on which dataset the other methods have been trained on, especially for methods where the original paper proposed multiple versions such as XVFI.\n- Some recent methods especially for high-resolution data are missing in Table 3 such as [A] and [B]. Especially [A] has only been included in Tab. 2 although their focus is also on high-resolution datasets and code for X-Test is directly available: https://github.com/feinanshan/M2M_VFI/blob/main/Test/bench_xtest.py\n\n\n[A] Hu et al. Many-to-many Splatting for Efficient Video Frame Interpolation, 2022\n\n[B] Nottebaum et al., Efficient Feature Extraction for High-resolution Video Frame Interpolation, 2022.\nIt would be nice to have longer sequences, and playing them a bit more fluently to get a feeling for the temporal and visual consistency.\n\nMinor: \n- Some colors difficult to see in overlay in Figure 1.\n- L. 126, Check sentence “where contains”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive and constructive suggestions. We have the following responses to your concerns:\n\n**Q.1** *How is 8x interpolation performed?*\n\n**R.1** As mentioned in line 497, we followed the testing procedure of FILM [1] and used an iterative approach for frame interpolation. Specifically, we first generated an intermediate frame based on the input two frames, and then, using a divide-and-conquer strategy, we further divided the first frame and the generated intermediate frame, as well as the generated intermediate frame and the last frame, to iteratively generate the remaining frames. Thank you for the reminder, and we will further emphasize the specific testing procedure in the main text.\n\n**Q.2** *A lot of important information is in the appendix*\n\n**R.2** We sincerely appreciate your suggestion and will incorporate the mentioned content into the main text in the final version, with more references to the appendix.\n\n**Q.3** *How the FLOPs/memory scales with resolution*\n\n**R.3** We have included the comparison of FLOPs and memory usage of different methods at various resolutions in the global response PDF. As shown in Figure 10, our method has a clear advantage in FLOPs and memory usage at high resolutions.\n\n**Q.4** *It should be mentioned in Table 2 and 3 on which dataset the other methods have been trained on*\n\n**R.4** Thank you for your suggestion. We have updated the information in Tables 2 and 3 in the global response PDF to include the specific training datasets for each method. We will also add this information in the final version.\n\n**Q.5** *Some recent methods especially for high-resolution data are missing in Table 3*\n\n**R.5** Thank you for the reminder. We have added the results of these two papers under the same testing procedure in Table 3 of the global response PDF, and we will also include them in the final version.\n\n**Q.6** *A discrepancy between many of the reported numbers in their respective original papers and in this paper on X-TEST for 2k and 4k*\n\n**R.6** First, we would like to respectfully remind you that the results in different papers may have used different quantization methods (whether to round the network output) and test functions (the SSIM function in sklearn and the MATLAB-style SSIM can produce different results). **To make a fair comparison, we have re-tested all the open-source models under the same testing procedure, which may lead to minor differences in performance compared to their original results**. As for EMA-VFI-S, it used a model trained on the Vimeo90K **septuplet** dataset for testing X-TEST, while our method and all the other methods used models trained on the **triplet** dataset for testing. **For the fair comparison, we tested the EMA-VFI-S model trained on the triplet dataset** using our iterative frame interpolation method as in **Q.1**, resulting in a performance of 29.91 dB instead of 30.89 dB. Regarding the performance of BiFormer, BiFormer did not publicly release its test code and we used the test code of SGM [2]. The results were consistent with the SGM, so we initially thought we had successfully reproduced the correct results. Unfortunately, after careful examination and comparison,  we identified an inconsistency between the input processing of SGM's testing procedure and BiFormer's training procedure, making the results of BiFormer lower than it should be.  After re-conducting the proper testing, we also obtained a result of 31.32 dB, and we have also updated the results of BiFormer on other datasets in Table 3 of the global response PDF. **We also rechecked all the results in Table 3 and ensured their accuracy.** We are truly grateful for your reminder, and we will also remind the authors of SGM to correct the BiFormer results.\n\n**Q.7** *Performance of SoftSplat*\n\n**R.7** Thank you for the reminder. We sincerely apologize for not finding that SoftSplat has already been open-sourced. The performance of Softsplat under the same testing procedure has been updated in Table 2 of the global response PDF, and you are welcome to check it.\n\n**Q.8** *Minor errors*\n\n**R.8** We will further improve the color distinction in Figure 1 and carefully check the grammatical details of the paper. Thank you for the reminder.\n\n> [1] Reda, Fitsum, et al. \"Film: Frame interpolation for large motion.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n> [2] Liu, Chunxu, et al. \"Sparse Global Matching for Video Frame Interpolation with Large Motion.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k99AiizAvh", "reviewer": "Reviewer_YfRu", "summary": "Based on the popular S6 model's advantages of linear computational complexity and data-independent modelling capability, this paper applies it to VFI. Specifically, this paper proposes a token rearrangement strategy to learn the information of adjust frames, in addition to introducing a curriculum learning strategy to dynamically learn various motion magnitudes between adjust frames through joint training of vimeo90K and X-TRAIN. The model achieves the highest performance on existing commonly used VFI datasets.", "questions": "I would like to see more analysis on the model effectiveness, such as the runtimes in Table 2.", "limitations": "The authors discuss methodological limitations in the supplementary material.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is the first to adapt the S6 model to the VFI.\n\n2. Experimental results show that the proposed VFIMamba achieve performance while using competitive FLOPs.", "weaknesses": "1. It is recommended that the authors can demonstrate the accuracy of the interpolation results more intuitively by visualizing the error maps.\n\n2. The experiments on table 4 are to demonstrate the computational validity of S6 model, and it is necessary to provide flops for the different models.\n\n3. In limitations, the author says that the VFIMamba has faster speed, I would like to know the comparison of the runtime in table 2.\n\n4. Lack of citations essential to the field, such as VFIT, TTVFI, ABME, EDSC, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your recognition and feedback on our work. We would like to respond to your concerns as follows:\n\n**Q.1** Visualization of error maps\n\n**R.1** Thank you for your suggestion. Visualizing the error maps can indeed provide a more intuitive demonstration of the accuracy of the interpolation results. We have provided an illustrative example in Figure 11 of the global response PDF, where VFIMamba shows clear advantages over other methods. We will add more comparative examples in the final version.\n\n**Q.2** FLOPs of Table 4\n\n**R.2** Thank you for your suggestion. The FLOPs of the different models in Table 4 are reported in the table below. We will include this in the final version.\n| Model | FLOPs(T) |\n| --- | --- |\n| w/o S6 | 0.23 |\n| Convolution | 0.27 |\n| Local Attention | 0.23 |\n| Full Attention | 0.59 |\n| S6 | 0.24 |\n\n**Q.3** Runtime comparison\n\n**R.3** Thank you for your suggestion. We have provided the runtime comparison in Table 2 of the global response PDF, and we welcome you to check that. We would also like to kindly remind you that in the limitations, we stated that VFIMamba is faster compared to the attention methods like SGM, with a runtime of 311ms vs. 942ms for 1024x1024 inputs. **Our primary goal was to achieve high-performance video frame interpolation while maintain efficient processing, rather than solely pursuing the fastest runtime.** We have discussed in detail how to further improve the runtime of our method in the response to **Reviewer Bwpp's Q.1**.\n\n**Q.4** Additional citations\n\n**R.4** Thank you for the reminder and these works are indeed important for the VFI task. We will add these citations in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all reviewers' efforts in reviewing our paper and giving insightful comments as well as valuable suggestions. We are glad to find that the reviewers generally acknowledge the following contributions of our work.

* **Framework.** We are the first to adapt the S6 model to the VFI task [YfRu,A9TL,Bwpp,9Hn5] for addressing key challenges in video frame interpolation, such as the need for large receptive fields and efficient computation. The use of Mixed-SSM Blocks and the interleaved token arrangement are effective solutions for enhancing inter-frame modeling [A9TL,Bwpp,9Hn5].
* **Training strategy.** We propose a novel curriculum learning strategy that is beneficial for VFI methods when having to deal with large motions [A9TL, 9Hn5].
* **Experiments.** Our experimental results achieve state-of-the-art performance on most datasets, particularly in high-resolution and large-motion scenarios [Bwpp,9Hn5,YfRu]. Furthermore, we have conducted exhaustive ablation studies to demonstrate the effectiveness of our proposed method [A9TL, Bwpp, 9Hn5].

As suggested by the reviewers, we include the following contents in the revised manuscript to further polish our paper. The major revision is summarized as follows. Our detailed responses can be found in each response section to the reviewers.

* **Adding more detailed information to the main comparison Table 2 and Table 3.** This includes: *1)* The training dataset for each method [A9TL, 9Hn5]. *2)* The average performance of each model [A9TL]. *3)* Results for M2M, FLDR, BiFormer and SoftSplat [9Hn5]. *4)* Runtime [YfRu]. We have provided the updated Table 2 and Table 3 in the PDF below.
* **More visualization comparisons.** This includes visualizations on how the FLOPs/memory requirement scales with resolution [9Hn5], as well as error maps [YfRu]. We have provided these in the PDF below.
* **Transferring important information from the appendix to the main paper**. This includes detailed testing procedures and the experiment on the generalization of curriculum learning [A9TL, 9Hn5].

Meta-review: All the reviewers unanimously recognize the novelty/significance of combining a new architecture (MAMBA) and the VFI task, the thorough experimental evaluation, the significance of a relatively simple curriculum learning strategy for VFI, and strong state-of-the-art performance of the proposed approach. The AC thus recommends to accept this paper. 

The authors are strongly encouraged to incorporate the new results into the final version of this paper.
Decision: Accept (poster)