Generalizable Implicit Motion Modeling for Video Frame Interpolation
Authors: Zujin Guo
Wei Li
Chen Change Loy
Venue: NeurIPS 2024 poster (2024)
Keywords: Motion Modeling; Optical Flow; Video Frame Interpolation
Primary area: machine_vision

Abstract: Motion modeling is critical in flow-based Video Frame Interpolation (VFI). Existing paradigms either consider linear combinations of bidirectional flows or directly predict bilateral flows for given timestamps without exploring favorable motion priors, thus lacking the capability of effectively modeling spatiotemporal dynamics in real-world videos. To address this limitation, in this study, we introduce Generalizable Implicit Motion Modeling (GIMM), a novel and effective approach to motion modeling for VFI. Specifically, to enable GIMM as an effective motion modeling paradigm, we design a motion encoding pipeline to model spatiotemporal motion latent from bidirectional flows extracted from pre-trained flow estimators, effectively representing input-specific motion priors. Then, we implicitly predict arbitrary-timestep optical flows within two adjacent input frames via an adaptive coordinate-based neural network, with spatiotemporal coordinates and motion latent as inputs. Our GIMM can be easily integrated with existing flow-based VFI works by supplying accurately modeled motion. We show that GIMM performs better than the current state of the art on standard VFI benchmarks.

Generalizable Implicit Motion Modeling for Video Frame Interpolation
Zujin Guo, Wei Li, Chen Change Loy S S-Lab, Nanyang Technological University (zujin.gue) wei 1,, ccloy)@ntu. edu. edu. https://gseeandaat gi wttthb...o/rojjetss////oooooo
Abstract
Motion modeling is critical in flow-based Video Frame Interpolation (VFI). Exist- ing paradigms either consider linear combinations of bidirectional flows directly predict bilateral flows for given timestamps without exploring favorable motion priors, thus lacking the capability of effectively modeling spatiotemporal dynamics in real-world videos. To address this limitation, in this study, we introduce Gen- eralizable Implicit Motion Modeling (GIMM), novel and effective approach motion modeling for VFI. Specifically, to enable GIMM as an effective motion modeling paradigm, we design a motion encoding pipeline model spatiotemporal motion latent from bidirectional flows extracted from pre-trained flow estimators, effectively representing input-specific motion priors. Then, we implicitly predict arbitrary-timestep optical flows within two adjacent input frames via adaptive coordinate-base neural network, with spatiotemporal coordinates and motion latent as inputs. Our GIMM can be easily integrated with existing flow-based VFI works by supplying accurately modeled motion. We show that GIMM performs better than the current state of the art on standard VFI benchmarks.
Introduction
Video Frame Interpolation (VFI) is fundamental task in computer vision, which involves generating intermediate frames between two adjacent video frames. This technique is crucial for various practical applications, including novel view synthesis [57] 14 30), video generation [47]. and vidco compression [52]. This task highly challenging due to the complex motions typically found in real-world videos. To address this, recent research 29 frameworks, which have shown substantial success. Generally. these frameworks for VFI involve two main phases: 1) transforming the input frames based on estimated optical flows, and 2) merging and enhancing the warped frames to produce intermediate frames. Consequently, the accuracy of flow estimation is crucial for the fidelity of the synthesized frames.
55
20 has focused
dow-based
Accurately modeling flow from two distant frames is challenging due to the complexities capturing subtle and dynamic movements caused by varying motion speeds, object occlusions, and lighting conditions. A commonly used approach in the literature combines bidirectional flows derived 39] (Figure (aggure[)), This method assumes overlapped and linear motion for flow estimation, which does not accurately reflect real-world dynamics. Several studies directly predict bilateral flows based on intermediate discrete timestamps (Figure I b)). These approaches model the correlation between frame motions and timestamps without leveraging motion priors within input frames; they thus fall short of capturing complex spatial-temporal changes and handling occluded regions and unexpected deformations. Moreover, this disccrte--iime-aasee modeling paradigm ineffective for arbitrary-time interpolation.
from input frames 222 40
45
53
20
29
55 2
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Po-t
Genembabie impiet
Pioo
Fl.e
Pi-o
Floe Latiaate
Franne Systhess
Flow Eatmoker
Frams Systheds
Flo Eatmater
Frane Systhecs
0.75
Pono Fe-1
Pro
Pb-1
Fi
Fe+1
(a)
(b)
Figure 1: Schematic of motion modeling paradigms in video frame interpolation. (a) A naive linear combination of bidirectional flows Fo-1,, Fi-0 (i.c., flows between input frames) may lead to ambiguous and coarse motion estimation due strong overlapped and linear assumptions. (b) A cime-condition-busee -aaaiidio----ed modeling approach may predict suboptimal bilateral flows Fe-so, Fi-1 (i.e., flows between estimated and input frames), capturing spatiotemporal changes for moving objects ineffectively. (c) Our generalizable implicit motion modeling properly represents spatiotemporal dynamics across videos and predict better bilateral flows via adaptive coordinate-based neural network.
In this study, we explore a more effective and generalizable implicit approach to motion modeling for VFI. Inspired by the success of implicit neural representations 48] in encoding complex high- dimensional data such as 2D images [5]. 3D scenes [34], and videos 3] we propose to implicitly adjacent video frames using coordinate- based neural networks. This
model optical flows between implicit paradigm can directly take arbitrary spatiotemporal coordinates as inputs and effectively decode the desired space-time timee-time outputs, making it promising framework for learning highly dynamic optical flows in real-world videos. However, leveraging implicit neural networks for effective motion modeling poses challenges. First, standard implicit neural networks typically perform per-instance modeling, optimizing model parameters for a single specific input. This limitation restricts their applicability across different input video frames. Therefore, we need to develop a more adaptive implicit model capable of capturing motions in any given video. Second, efficiently integrating spatiotemporal information within implicit neural networks complex, especially when dealing with the intricate motions occurring between two video frames. This necessitates designing appropriate implicit neural architectures that can accurately predict and represent both the spatial and temporal dynamics in videos.
To this end, we propose a novel generalizable implicit flow encoding for motion modeling in VFI, called Generalizable Implicit Motion Modeling (GIMM) (Figure Our method only assumes the availability bidirectional flows (Fo-;1, Fi-o of two input frames obtained from a pre- pre-trained optical flow estimator (e.g., RAFT [50], FlowFormer [19). The input flows can be noisy as this prior will be refined estimate the bilateral flows (Fi+s, (r--); between arbitrary intermediate timestamps. Our method is unique in that it can represent input-specific motion priors effectively. For instance, it can accurately capture the complex dynamics of somersault. This is achieved by introducing motion encoding pipeline to extract spatiotemporal motion latent from the bidirectional flows. Our method can further enable frame interpolation at arbitrary timestamps, thanks to the adaptive coordinate-based neural network that takes spatiotemporal coordinates and motion latent as inputs. This capability allows our method to generate frames at various temporal granularities, providing flexibility and precision in video frame interpolation.
Our contributions are summarized as follows: We present an effective motion modeling paradigm for video frame interpolation characterized by a novel generalizable implicit motion modeling framework. Our GIMM is capable of accurately predicting optical flow for arbitrary timesteps between two adjacent video frames any resolution, allowing seamless integration with existing flow-based VFI methods. We demonstrate the advantages our GIMM in motion modeling for arbitrary-timester VFI tasks, achieving state-of-the-an performances various benchmarks.

---

2
Related Work
Video frame interpolation. Conventional VFI studies primarily rely on either direct frame synthesis
via convolutional networks T 24 or interpolation using dynamic kernels with learnable weights and offsets 37
28 Recent approaches have shifted towards flow-based methods to synthesize frames at desired timesteps, where motion modeling plays crucial role [20] [2 (54]
38
9
26
Some flow-based methods combine estimated bidirectional flows between input frames 39 I] often leading to inaccurate motion predictions, especially in occluded areas. This simplistic combination results in ambiguous and coarse motion estimation, causing object shifts in interpolated frames. Several recent approaches [20 22 55
49
17.
12
40
45
53
address these issues by directly predicting the desired motion within an end-to-end framework conditioned on timesteps, showing impressive results in synthesized frames. However, these methods mainly rely discrete-time-beee fitting of variable relationships between motion and timesteps, making it challenging to achieve consistent, continuous interpolation outcomes.
20, 32
Implicit neural representations. Implicit Neural Representations (INRs) have been shown effective in modeling complex, high-dimensional data for various upplications, including video compression
3 novel view synthesis [34] (30, and image super-resolution [5]. Typically, INRs learn a continuous mapping from set of coordinates to specific signal using coordinate-based neural network to encode data implicitly. The flexible and expressive modeling capabilities of INRs motivate us to explore their use to encode intricate motions and capture subtle, dynamic movements of objects in real-world videos. recent work related to ours is IFE [33. While both approaches consider implicit flow encoding, our method significantly differs from IFE. Unlike IFE, which focuses per-scene encoding- whhre each coordinate-basse network parameterized by a specific video instance-we aim to develop generalizable motion modeling approach that can be applied across different videos.
further extend INRs for generalizable encoding by conditioning coordinate-bass based neural networks with additional instance-speciff inputs. employ Transformers 51 as meta-learners to predict instance-specific weights or modulation features for coordinate-basee neural networks at high com- putational costs. Several notable studies to facilitate video interpolation and super- resouuton,, mainly focusing directly learning implicit space-time continuous neural representations from video. In contrast, we aim to explore effective motion modeling paradigms to improve intermediate frame synthesis for flow-based VFI. To our knowledge, we make the first attempt to utilize generalizable INRs for motion modeling in the context of VFI.
Generalizable INRs. Recent works 4F
10
For example, some approaches 6 27
25
45) leverage generalizable INRs for video encoding
3 Method
Given a pair of adjacent video frames Io. I, E "xW with timesteps f0, 1}, a general flow-based video frame interpolation process is defined as follows:
Fi = Glo, 41.t),
(1)
I, HFF,, To, IJ,, [0,
where G denotes motion modeling (or flow estimation) module, and H. indicates a frame interpolation process under the guidance of estimated motion F. at a timestep of t E [0. 1] for input frames To and JT. In this work, we mainly focus studying an effective motion modeling framework G for flow- VFI.
3.1 Generalizable Implicit Motion Modeling
The goal of GIMM is to estimate bilateral flows Fe-o, Fe-1 for any timestep E [0, 1]. deriving from initial bidirectional flows Fo--1, Fi
Fe Fi+11 G(Fo-11, F1-s0. f), Vt [0,
where Fo Fi-.0 are predicted from a pre-trained optical flow estimator (e.g., RAFT (50,, Flow- Former [9] with input frames To and 7, Figure depicts the overall generalizable motion mod- eling framework of GIMM. Motivated by the great success of INRs in modeling complex video data 4 34 GIMM uses adaptive coordinate- bassed neural network for continuous motion
3

---

3D Coordinates # Bi
Motion
Forward Warping
Latent Refince
(x.
Enooder
Motion
Coordinst--usedd
Bilateral
Network
Fb-s2; F1-12
Vi,
L
Vi
Figure 2: Our GIMM first transforms initial bidirectional flows Fo-+1 as normalized flows Vo, V1. The motion encoder extracts motion features Ko. K₁ from Vo. V1, respectively. Ko. Kı then forward warped at a given timestep t using bidirectional flows to obtain the warped features Ki-o: Ki-1- We pass the warped and initial motion features into Latent Refiner that outputs motion latent L,, representing motion information at tt Conditioned on (2. y), the coordinate based network go predicts the corresponding normalized flow V/2 with 3D coordinates interpolation usage, V, is then transferred into bilateral flows Pt Pt-1 through denormalization.
(xr y/ (). For
modeling. Unlike the existing IFE 13 that performs per-scene optimization and requires test-time learning different videos, our GIMM takes additional instance-specific motion latent inputs T,e to enhance model generalizability across various input videos.
Flow normalization. Following IFE flows as follows:
13 we perform normalization for the initial bidirectional
Vo = (Foo,1), = (b-Fi-o)).
Vi = o ((Fr,, Ff-20):
where & is a scale operator. This reversible normalization process aligns the scale and temporal direction of input bidirectional flows Fo Fi-00 with output bilateral flows Fi Fi-1, allowing the normalized flows Vú E [0, "|] x2 i E 0, 1 to be effectively encoded subsequent implicit motion modeling in GIMM.
Motion latent. To achieve generalizable implicit modeling of motion dynamics, we learn implicit neural representations for motions with conditions on ristance-specife motion latent inputs (1( v). which provides instance-specific motion priors at spatiotemporal coordinates x = (a, y: 8) for target motion (2 y). Specifically, we introduce a Motion Encoder to extract motion features K; from normalized flows V. To maintain spatiotemporal consistencies of modeled motion, we derive time- dependent motion features K, for given timestep t from both motion features Ko and K₁. To realize this, we apply forward warping w process to map every location motion features K; to the target timestep t:
K = wt (K, Fi-se, Zi).
(6)
Here, Ky-s are warped features with timesteps E {0,1) that integrate the correspondences from source (input) timesteps to the target timestep, Z: represents splatting weights, and Fi denotes scaled bidirectional flows at timestep t. We compute scaled bidirectional flows Fi as follows:
(1 (1-0) Fo-.. F = (1 -) 1) F1
i 0
(7)
if
To mitigate warping errors multi-to-one cases or occluded regions, we calculate splatting weights Z; using flow consistency U/ flow and variance U metrics (6] as follows:
Z, + a fow Ufoow + Over
(8)
U var
where a flow and Ovaa are learnable parameters. To obtain the final motion latent L,, we concatenate the warped features Concat Concat(Ku-t, K1-+1) as the coarse motion latent at timestep and further refine via a Latent Refiner module deal with potential information loss and ambiguous motion in forward warping process. Both the Latent Refiner and the Motion Encoder mentioned above are structured shallow convolutional networks. We provide their network configurations and detailed calculations for two flow metrics flow and U
var in the supplementary.

---

2, a
Io
Protrained
Frame Syuthesis Module
Generalizable Implicir Motion Modeling
Pius
Flow
Estimator
4
g
$
F.-0
+1
A,C
C
Figure 3: An overview of GIMM-VFI architecture. GIMM-VFI employs a pre-trained flow estimator, E, to predict bidirectional flows (Fo-1. Fi and extracts context features well correlation features C' from the input frames (To. I I).. Given the timestep f, generalizable implicit motion modeling (GIMM) module & (detailed in Figure takes the bidirectional flows as inputs and predicts bilateral flows (Fr-o, 1..1)), which are then passed into frame synthesis module S, together with extracted features (A, C), to synthesize the target frame It.
Implicit motion prediction. To realize INRs for generalizable motion modeling, we devise an adaptive coordinate-basee network 9e for implicit and continuous motion encoding, which maps spatiotemporal coordinates x E [-1, 1] xW x3 along with the corresponding D-dimension motion latent code L, E HxxD to predicted normalized flows Ve:
(9) The predicted normalized flow Vi can be converted into predicted bilateral flows Fe-2o Fi-11 via the reverse flow normalization process for VFI. See the supplementary for the details of the coordinate- based network.
V = 9o (x, L.).
Optimization. In practice, we optimize the GIMM module by minimizing a Mean-Square-Erro (MSE) loss between predicted (V)) and ground-tree truth (V₂) normalized optical flows:
H- W-1
fgimm (Vi, Vi) H W
I[V( V. 1)122
(10)
1=0
3.2 Integrating GIMM with Frame Interpolation
GIMM-VFL. Given two adjacent video frames with E 1). start with pre-trained optical flow estimator (e.g.. RAFT (50), FlowFormer [19] to extract context features A,, correlation features C₁ and initial bidirectional flows Fo-.1 and Fi-.o- We first predict bilateral flows Fr-.s for given timestep from initial bidirectional flows using a pre-trained GIMM module. The predicted bilateral flows Fi-ss context features A,, and correlation features C, are then passed into frame synthesis module S (adapted from the AMT (29]), which refines the input flows and generates warping masks M, E [0, x] Following previous flow-based VFI studies 29 55 20), we obtain warped images Ι via backward warping to under the guidance of predicted bilateral flows Fr-,i from input frames In and I
I = to (1,, F,..s):
(11)
(x v) = I,x + (Z, Fy,,( 9)).
(12)
where Plan Fl represents horizontal and vertical motion of Fi-io respectively. To generate the final interpolated image Ír, the warped images I,-,, It-.I are fused with the warping mask M, as follows:
i, = Mr 4,-00 + (1 M,) 41-11- See the supplementary for the details of the frame synthesis module. Overall objectives. We use the following objective function for VFI:
(13)
Linterp i.) Liap ( Te, i.) + Lebar (Je, (.) Loonsus (Jr, fe),
(14)
where Liaps Lehar, and Loensus denote the Laplacian loss (55 36), Charbonnier loss 29 and
census loss [29 33 respectively. In addition, to preserve the motion modeling in the pre-trained
5

---

Table 1: Comparisons of different motion modeling methods. We assess the modeled motion on Temeo-Triplet-loow (VTF) and Limeo-Spptuplet-ling (VSF) by employing PSNR and EPE metrics. Additionally, we demonstrate their impact on the interpolation task by presenting the PSNR values of their interpolation results on the 'hard' split of the SNU-FILM-ard dataset. We denote the calculated PSNR values as PSNR; for optical flows in the motion assessment and PSNR; for interpolated images the interpolation task.
Vimeo-Triplct-Foon (VTF)
Vimco-Septuplct-Foow (VSF)
SNU-FILMMarb-Haac
Motion method
PSNRtt
EPE/
PSNR,
EPE]
PSNR
Lincar
35.03
0.44
30.09
2.87
32.42
Forward Warp
32.80
0.47
28.22
3.38
32.31
End-to-End
29.23
1.02
25.99
5.12
32.28
BMBC 39
28.89
0.95
23.19
8.23
28.51
GIMM (- (FF-R)
37.56
0.34
30.45
2.68
32.62
GIMM module, we reconstruct and supervise the flows Vo, V with pseudo ground truth Vo, Ví. This objective Eyec is defined l|Vo
Vo + || V1ll2. The overall objective L is follows:
f = Linterp (1,, f.) Areehrect
(15)
where Arec is the hyperparameter. We optimize the entire GIMM-VFI model that contains the pre-trained flow estimator, pre-trained GIMM module, and frame synthesis module.
4 Experiments
We present quantitative and qualitative evaluations of motion modeling method GIMM in Section 4.1 and the corresponding interpolation method (GIMM-VFI) in Section Specifically, evaluate both motion quality and performance the downstream interpolation task. We compare GIMM-VFI with current state-of-the-ar VFI methods on arbitrary-timester interpolation.
Implementation details. We train the GIMM model on the training split of Vimeo90K [54) triplets dataset using optical flows extracted by off-the-shelf flow estimators. Our GIMM-VFF trained on the complete Vimeo90K septuplet dataset. Specifically, we implement two variants of GIMM-VFI using two different flow estimators: the RAFT (50) and FlowFormer [19], designated as GIMM-VER VFI-R and GIMM-VFI-F, respectively. More details are provided in the supplementury materials.
4.1
Motion Modeling
VTF and VSF benchmarks. To assess the modeled motion quality with GIMM, we use Flow- former [19] to produce pseudo ground truth motion. Specifically, establish two motion evaluation benchmarks: Vimeo- Vimeo-Triplet-
(VTF) and Vimeo-Septuplet-Flov (VSF). These benchmarks are derived from the triplet and septuplet splits of the Vimeo90K [54] and 6X motion modeling, respectively.
test set, tailored for evaluating 2X
Baselines. We compare GIMM with other motion modeling approaches, including Linear, Forward Warp, End-to-End, and BMBC 339. Specifically, Linear replaces our GIMM module with the linear approximation strategy (22) Similarly, Forward Warp substitutes GIMM with a forward warping strategy [35]. Meanwhile, the End-to-End denotes the strategy that directly predicts motion at arbitrary timesteps through end-to-end fitting. In this end-to-end setting, select the current state- of-the-art EMA- VVI (55) as the representative method. Additionally, we make further comparisons with BMBC 39), which is supposed model complex motion effectively through several stages of motion predictions and motion refinements. For fair comparisons, we employ RAFT (00 as the flow estimator for different motion modeling methods.
Evaluation settings. We measure the modeled motion quality on both VTF and VSF benchmarks by calculating PSNR normalized flows and End-Point-EError (EPE) on the unscaled flows. For the downstream interpolation task, we calculate PSNR on the 'hard' split of the M-r-FIL-M-ar benchmark Baainaark-SU-PPPPPS COU--IILM-arb-HHEE Details about SNU-FILM-arb presented in Section 4.4

---

Overlaid Input
Linear
Forward Warp
End-to-Fnd
BMBC
GIMM-VFI-R
Ground Truth
Figure 4: Qualitative comparisons of different motion modeling methods SNU-FILM --ar-------ee All the results are predicted at = 0.75, and ground truth flows are obtained by FlowFormer [9
Results. We provide quantitative comparisons of our GIMM and baselines in Table I We first report the results of motion quality on VTF and VSF We find that GIMM can continuously model motions in videos. GIMM predicts the best flows on both the VTF benchmark (37.56dB PSNR/ 0.34 EPE) and the VSF benchmark (30.45dB PSNR/ 2.68 EPE) that involves unseen timesteps during training. Moreover, our study highlights the importance of motion priors both motion modeling and the downstream interpolation task. The Linear, Forward Warp, and our GIMM methods that leverage
motion priors from RAFT [50) can provide better motion and interpolation performances than the End-to-End and BMBC
39 across all benchmarks. Notably, GIMM benefits the interpolation task most and achieves the highest PSNR of 32.62 on SNU-FILM-arb-HHmne While motion modeling plays critical role in interpolation, other modules also impact interpolation outcomes. The fad-to-End method has a narrower performance gap with other advanced motion modeling approaches the interpolation benchmark than motion modeling benchmarks, likely due to its more complex and extensive feature extraction and frame synthesis modules.
Visualizations. Besides the quantitative evaluation of motion modeling, we also qualitatively evaluate them on SNU-FILMM-rbb-Haie as shown Figure Since SNU-FILM aaa---r--aaaaa Hard requires 8X interpolation for input frames of large motion, classical methods like BMBC 39 may struggle to model the intermediate motion. contrast, we observe the capacity of GIMM for accurate motion modeling at arbitrary timesteps. Take the Somersault in Figure 4 for example, the ankle flow and corresponding interpolation generated by GIMM have the best consistency with the ground truth, pointing the same direction. Moreover, GIMM is capable of reducing ambiguities when interpolating moving objects. We can observe clearer silhouettes with fewer blurs in both the 'dog-leg' and 'dog-head' cases in Figure
4.2 Ablation Study on Motion Model Choices
As our core contribution, we present an effective motion modeling paradigm for video frame interpo- lation, characterized by a novel generalizable implicit motion modeling framework (GIMM), this section, we further investigate the necessity of implicit neural representations (INRs) and approaches
7

---

Table 2: Quantitative comparisons of different motion model choices.
Vimeo-
Triplet-Flow V
(VTF)
Vimeo Serro--ppppleetFFloe (VSF)
Model Variants
Param.(M)
PSNR
EPEL
PSNR
EPF.
Meta-learning (INR)
30.19
0.88
24.50
6.80
43.92
GIMM (U-Net)
36.96
0.39
29.96
2.90
4.27
GIMM (INR)
37.56
0.34
30.45
2.68
0.25
to generalizable INRs. The experiments are conducted on the motion modeling benchmarks, i.e. Vimeo-Triplet-e (VTF) and Vimeo-Septupllt-Floo (VSF).
Necessity of INRs. straightforward replacement of INRs in our GIMM is timestep-conddtioned U-Net as in diffusion literature [43]. We substitute the INRs with U-Net of a comparable number of channels to ensure fair comparisons. performance, especially on the 6X motion modeling benchmark VSF. This demonstrates the strong continuous modeling ability of INR. Besides, GIMM with INR has much lighter architecture, improving efficiency. Therefore, it is necessary and proper to use INR for our motion modeling.
Table 2 replacing INR with U-Net results in worse
Approaches generalizable INRs. To realize generalizable implicit modeling, alternative way is to modulate the weights of INRs with meta-learners [6 255 Following [25]. we employ Transformers [5]] as the meta-learner Compared GIMM, this meta-lcarning approach performs significantly worse across all benchmarks while introducing a model with over 170X more parameters.
4.3 Ablation Study on GIMM
In this section, we investigate the effectiveness of model designs in GIMM in Table The experiments conducted the VTF and VSF benchmarks.
Forward warping. We substitute forward warping operation GIMM with a straightforward linear combination of the input motion features (Non-Fwarp). In Table we observe significant performance drop with this modification. For example, the EPE on VSF increases by 0.20. This result highlights the importance of forward warping for GIMM's continuous motion modeling.
Implicit modeling, In GIMM, the motion can modeled even without the presence of implicit modeling. We conduct experiments without using any coordinates (Non-Imp). In Table [3] this variant yields performance gains, with reductions of 0.05 and 0.13 in the EPEs on VTF and VSF respectively, highlighting the importance of implicit modeling in GIMM.
Motion encoder. We leverage the motion encoder in our GIMM to extract features from flows, alleviating the negative impacts brought by the possible bias and noises in the estimated flows. To justify this design, remove the motion encoder for direct comparison (Non-ME), As a result, we observe increases in error on both benchmarks, e.g., the EPE on VSF rises 0.17. Therefore, the motion encoder benefits the motion modeling in GIMM.
Latent refinement. To validate the efficacy of latent refinement GIMM, we conducted experiments excluding the latent refiner (Non-Refiner). This leads to notable declines of 0.53 dB and 0.43 dB in PSNRs across both benchmarks. Thus, refining the motion latent proves essential for accurate motion modeling.
Spatial coordinates. Since the modeled motion consists of spatiotemporal changes, we use 3D spatiotemporal coordinates in the implicit modeling of GIMM. To assess the necessity of spatial coordinates, we replace 3D coordinates with temporal coordinates (T-coord only). This removal causes 0.16 dB drop of PSNR on Vimeo- Vimeo-Triplet-Fie Triplet-Flow and 0.06 increase of EPE Vimeo Saaeec-Spptuplt-Flooe Flow. This emphasizes the crucial role of spatial coordinates in implicit modeling.
4.4 Arbitrary-timestep Video Frame Interpolation
SNU-FILM-ard benchmark. We introduce SNU-FILM-arb benchmark for more generalized evaluation, to facilitate research on the arbitrary-timeste timesttep frame interpolation task. Specifically, we incorporate the SNU-FILM dataset [IT] that encompasses 310 clips from 31 videos captured 240 fps, with heights ranging from 384 to 1280 pixels and widths from 368 to 720 pixels. Different from

---

Table 3: Quantitative comparisons of different model variants. For each model variant, we evaluate its motion modeling performance on Vimeo- Triplet-Flow Vimeo-Triplet- Flow and Sameo-Septuplet-loowe respectively. We adopt EPE and PSNR as the metrics for motion quality.
Vimeo-Triplet-Flow (VTF)
Vineo-Septuplet-Flow (VSF)
Model Variants
PSNR
EPE!
PSNRT
EPE]
Non-Fwarp
37.07
0.37
30.09
2.88
Non-Imp
37.04
0.39
30.11
2.81
Non-ME
37.05
0.42
30.26
2,85
Non-Refiner
37.03
0.37
30.02
2.77
T-coord only
37.40
0.36
30.39
2.74
Full
37.56
0.34
30.45
2.68
Table 4: Quantitative results for arbitrary-timestep interpolation. We report the quantitative metrics reported as PSSNS/LPIPSS//IDD with the best results highlighted in boldface and the second best results in underline. The analysis categorizes methods into two groups: non-INR methods (first half) and INR-based methods (second half). We employ a '-' symbol to denote 'out-of- -ut-oo--mmoory scenarios, and dagger ('f') to indicate that the results are drawn from prior studies 20 18 55
SNU FILM-i
Method
2K
4K
Medium (4X)
Hard (8X)
Extreme (16X)
RIFE
20
31,43t /0, /26 11.99
30.58 10.15// 13.52
36.33/ 0.038 / 6.65
31.87/ /.0072 11.99
27.21 / 0044 19.82
M2M
18
32.13t /0.098 99.25
30.88t 0.158/ 8.67
36.56 / 0.036 / 5,98
31.92/ 0.061 10.13
27.14/0.112 17.37
IRNNet
26
31.531 0,008/ 23.93
30.46% 0.164 23.75
34.88/ 0.046/ 9.92
31.15 / 0.066 11.65
26.32/ 0..11/ 16.91
AMT
29
28.88
13.92
28.17 0.117////////// 13.97
34.49 0.072 / 9.25
31.03/ 0.089/ 10.34
26.44/0.136
14.72
UPR-Net 23
31.16/ 0.104 10.75
30.50 0.154 99445
36.78/ 0.033 / 6.09
31.96/ 0.064 9.93
27.14/0.111 16.76
EMA-VFI
32.53 0.097 /7.21
31.21 0.115// 8.61
36.65 6.041 / 77.07
32.28 0.074 12.17
27.72/0.130/ 19.58
CURE 45
30.24 0.14/1 26.42
36.09 / 0.035 6.98
31.32/ / 0.063 12.72
26.61 0 14.1 22.62
GIMM-VHI-R
32,71/ 0.113 6.52
3188/ 0.149/ 6.49
37.02/ 0.033 5.89
32.62/ 0.060/ 9.59
27.99 0.110/ 16.45
GIMM-VFL-F
32,91 /0,103/ 6.77
31,97 / 0,142/ / 6.58
37,03 / 0.031/ 5.86
32.56/ 0,059 19.95
28.01 0.109 16.79
its original setting of 2X interpolation, we conduct 4X, 8X, and 16X interpolation evaluations on the original 'Medium', 'Hard', and 'Extreme' subsets of SNU-FILM, respectively.
Evaluation settings. We calculate PSNR and perceptual indicators, e.g., LPIPS [56]. FID [16), both SNU-FILM-arb and X4K-1000FPS (XTest) [46]. XTest comprises 15 clips from 15 videos
for 8X interpolation. Following the established protocol from prior studies [18. 55), assess interpolation quality at both 2K and 4K resolutions. For fair comparisons, we disable test-time augmentations during evaluations.
Current advanced VFI methods. We compare GIMM-VFI with various advanced interpolation methods. For non-INR methods, include RIFE (20), IFRNet (26). M2M [8], AMT (2), UPR- Net 23 and EMA-VFI [55]. We also make comparisons with the INR-based method CURE 45
Results. We present the quantitative results of GIMM-VFI on arbitrary- interpolation benchmarks in Table Our proposed method achieves high-quality continuous interpolation across various timesteps (i.e., 4X, 8X, 16X). In terms of PSNR, we observe that our RAFT-based (50) method CIMM-VFI-R achieves significant improvements of 0.18 dB on XTest-2K, 0.67 dB on XTest-4K, and approximately 0.30 dB each subset of the SNU-FILM-arb, in comparison with the previous state-of-the-ee method EMMAVFII (55). For perceptual metrics, i.e., LPIPS and FID, our proposed GIMM-VEF still outperforms previous methods most benchmarks, delivering competitive performance. These observations demonstrate that our GIMM can offer effective motion modeling for the arbitrayy-timestep VFI task.
Visualizations. Besides the quantitative evaluations, we further qualitatively compare both GIMM- VFI-R and GIMM-VFI-F with existing VFI techniques, as illustrated in Figure [5] Our methods achieve better interpolation across various timesteps. For example, our methods maintain the integrity of moving object silhouettes (3rd case). Moreover, GIMM-VFI-R and GIMM-VFI-F preserve detailed textures within both occluded regions and moving objects from significant deformations, e.g. the rectangle pole tag(Ist case), truck (2nd case), swing case).

---

timestop
0.25
0.75
0.25
0.75
0.25
0.75
0.25
0.75
Overlaid Input
CURF
FMA-VFI
GIMM-VFIR
GUMM-VFI-R
Ground Truth
Figure 5: Qualitative comparisons of arbitrary-timestep interpolation on XTest-2K 46. Positions
pointed by the yellow arrow indicate the distinct performance of our method.
Limitation
There are several known limitations to our method. First all, the GIMM-VFL is closely related to the pre-trained flow estimator, which estimates bidirectional flows and extracts image features. Therefore. GIMM-VFI inherits the limitations of the chosen pre-trained flow estimators, which may include high computational costs. Additionally, GIMM-VFI only accepts two consecutive frames as inputs, which favorable for frame interpolations where larger and nonlinear motion exists.
Conclusion
We propose novel motion modeling method, GIMM, which performs continuous motion modeling for video frame interpolation. Our proposed GIMM is the first attempt to learn generalizable implicit neural representations for continuous motion modeling. The method can properly predict arbitrary- timestep optical flows within two adjacent video frames at any resolution, and be easily integrated with existing flow-based VFI approaches (e.g., AMT 29) without further modifications by supplying accurately modeled motion. Extensive experiments on the VFI benchmarks show that our GIMM can effectively perform generalizable motion modeling across videos.
Acknowledgement. This study supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, well as cash and inndind contributions from the industry ps)tner(s).
10

---

References
[1] Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuar Yang. Depth-aware video frame interpolation. In CVPR, 2019.
[2] Pierre Charbonnier, Laure Blanc-Feraud, Gilles Aubert, and Michel Barlaud. Two deterministic half-quadratic regularization algorithms for computed imaging. In ICIP, 1994.
[3] Hao Chen, Bo He. Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. NeRV: Neural representations for videos. In NeurlPS, 2021.
[4] Yi-Hsin Chen, Si-Cun Chen, Yen-Y Yu Lin, and Wen-Hsiao Peng. MoTIF: Learning motion trajectories with local implicit neural functions for continuous space-time video super-resolution. ICCV, 2023.
[5] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In CVPR, 2021.
[6] Yinbo Chen and Xiaolong Wang- Transformers meta-learners for implicit neural representa- tions. In ECCV, 2022.
[7] Zeyuan Chen, Yinbo Chen, Jingwen Liu, Xingqian Xu, Vidit Goel, Zhangyang Wang, Humphrey Shi, and Xiaolong Wang. VideoINR: Learning video implicit neural representation for continu- ous space-time super-resolution. In CVPR, 2022.
[8] Xianhang Cheng and Zhenzhong Chen. Video frame interpolation via deformable separable convolution. In AAAI, 2020.
[9] Xianhang Cheng and Zhenzhong Chen. Multiple video frame interpolation via enhanced deformable separable convolution. IEEE TPAMI, 2021.
[10] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In CVPR, 2020.
[11] Myungsub Choi, Heewon Kim, Bohyung Han, Ning Xu, and Kyoung Mu Lee. Channel attention all you need for video frame interpolation. AAAI, 2020.
[12] Tianyu Ding, Luming Liang, Zhihui Zhu, and Ilya Zharkov. CDFI: Compression-driven network design for frame interpolation. In CVPR, 2021.
[13] Pedro Figueiredo. Avinash Paliwal, and Nima Khademi Kalantari. Frame interpolation for dynamic scenes with implicit flow encoding. In WACV, 2023.
[14] John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. DeepStereo: Learning to predict new views from the world's imagery. CVPR, 2016.
[15] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Samna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In CVPR, 2020.
[16] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In NeurIPS, 2017.
[17] Xinyu Hou, Liming Jiang, Rui Shao, and Chen Change Loy. Video infilling with rich motion prior. In BMVC, 2023.
[18] Ping Hu, Simon Niklaus, Stan Sclaroff, and Kate Saenko. Many-to-many splatting for efficient video frame interpolation. In CVPR, 2022.
[19] Zhaoyang Huang, Xiaoyu Shi. Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. FlowFormer: A transformer architecture for optical flow. In ECCV, 2022.
[20] Zhewei Huang, Tianyuan Zhang, Wen Heng, Boxin Shi, and Shuchang Zhou. Real-time intermediate flow estimation for video frame interpolation. In ECCV, 2022.
11

---

[21] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias Nießner, and Thomas Funkhouser. Local implicit grid representations for 3d scenes. In CVPR, 2020.
[22] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming- -usunn Yang, Erik Learned-Mile and Jan Kautz. Super SloMo: High quality estimation of multiple intermediate frames for video interpolation. In CVPR, 2018.
[23] Xin Jin, Longhai Wu, Jie Chen, Youxin Chen, Jayoon Koo, and Cheul-Hee Hahm. A unified pyramid recurrent network for video frame interpolation. In CVPR, 2023,
[24] Tarun Kalluri, Deepak Pathak, Manmohan Chandraker, and Du Tran. FLAVR: Flow-agnostic video representations for fast frame interpolation. In WACV, 2023.
[25] Chiheon Kim, Doyup Lee, Sachoon Kim, Minsu Cho, and Wook-Shin Han. Generalizable implicit neural representations via instance pattern composers. In CVPR, 2023.
[26] Lingtong Kong, Boyuan Jiang, Donghao Luo, Wenqing Chu, Xiaoming Huang, Ying Tai, Chengjie Wang. and Jie Yang. IFRNet: Intermediate feature refine network for efficient frame interpolation. In CVPR, 2022.
[27] Doyup Lee, Chihcon Kim, Minsu Cho, and Wook-Shin Han. Locality-awarc generalizable implicit neural representation. In NeurlPS, 2023.
[28] Hyeongmin Lee, Tacoh Kim, Tae-Youme Young Chung, Dachyun Pak, Yuseok Ban, and Sangyoun Lee. AdaCoF: Adaptive collaboration of flows for video frame interpolation. In CVPR, 2020.
[29] Zhen Li, Zuo-Liang Zhu, Ling- Hao Han, Qibin Hou, Chun-Le Guo, and Ming-Ming Cheng. AMT: All-pairs multi-field transforms for efficient frame interpolation. In CVPR, 2023.
[30] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
[31] Chengxu Liu, Huan Yang, Jianlong Fu, and Xueming Qian. TTVFI: Learning trajectory-aware transformer for video frame interpolation. IEEE TIP, 2023.
[32] Liying Lu, Ruizheng Wu, Huaijia Lin, Jiangbo Lu, and Jiaya Jia. Video frame interpolation with transformer. In CVPR, 2022.
[33] Simon Meister, Junhwa Hur, and Stefan Roth. UnFlow: Unsupervised learning of optical flow with bidirectional census loss. In AAAI, 2018.
[34] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[35] Simon Niklaus, Ping Hu, and Jiawen Chen. Splatting synthesis for video frame interpola- tion. In WACV, 2023.
[36] Simon Niklaus and Feng Liu. Softmax splatting for video frame interpolation. In CVPR, 2020.
[37] Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive convolution. In CVPR, 2017.
[38] Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convolution. In ICCV, 2017.
[39] Junheum Park, Keunsoo Ko, Chul Lee, and Chang-Su Kim. BMBC: Bilateral motion estimation with bilateral cost volume for video interpolation. In ECCV, 2020.
[40] Junheum Park, Chul Lee, and Chang-Su Kim. Asymmetric bilateral motion estimation for video frame interpolation. In ICCV, 2021.
[41] Tomer Peleg, Pablo Szekely, Doron Sabo, and Omry Sendik. Im-net for high resolution video frame interpolation. In CVPR, 2019.
12

---

[42] Fitsum Reda, Janne Kontkanen, Eric Tabellion, Deqing Sun, Caroline Pantofaru, and Brian Curless. FILM: Frame interpolation for large motion. In ECCV, 2022.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[44] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. PIFu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, 2019.
[45] Wentao Shangguan, Yu Sun, Weijie Gan, and Ulugbek S Kamilov. Learning cross-video neural representations for high-quality frame interpolation. In ECCV, 2022.
[46] Hyeonjun Sim, Jihyong Oh, and Munchurl Kim. XVFI: extreme video frame interpolation. In ICCV, 2021.
[47] Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang. Oron Ashual, Oran Gafni, Devi Parikh, Sonal Gupta, and Yaniv Taigman. Make-A- A-Video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.
2022.
[48] Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NewrIPS, 2020.
[49] Li Siyao, Shiyu Zhao, Weijiang Yu, Wenxiu Sun, Dimitris Metaxas, Chen Change Loy, and Ziwei Liu. Deep animation video interpolation in the wild. In CVPR, 2021.
[50] Zachary Teed and Jia Deng. RAFT: Recurrent all all-pairs field transforms for optical flow. In ECCV, 2020.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017.
[52] Chao- Yuoo Wu, Nayan Singhal, and Philipp Krahenbuhl. Video compression through image interpolation. In ECCV, 2018.
[53] Xiangyu Xu, Li Siyao, Wenxiu Sun, Qian Yin, and Ming-Hsuan Yang. Quadratic video interpolation. In NeurlPS, 2019.
[54] Tianfan Xue, Baian Chen, Jiajun Wu, Donglai Wei, and William T Freeman. Video enhancement with task-oriented flow. I/C,, 2019.
[55] Guozhen Zhang, Yuhan Zhu, Haonan Wang, Youxin Chen, Gangshan Wu, and Limin Wang. Ex- tracting motion and appearance via inter-frame attention for efficient video frame interpolation. In CVPR, 2023.
[56] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[57] Tinghui Zhou, Shubham Tulsiani, Weilun Sun, Jitendra Malik, and Alexei A Efros. View synthesis by appearance flow. In ECCV, 2016.
13

---

7 Appendix
In this supplementary material, we first provide details of our method Section 7.1 including descriptions of flow metrics and reverse flow normalization. Besides, we present the network architectures of both the GIMM module and frame synthesis module in Section 7.2 Moreover, we provide details of our implementation and the corresponding training hyperparameters in Section 7.3 and Section 7.4 respectveely. We show qualitative results of motion modeled by GIMM and its model variants in Section 7.5 Furthermore, present integrations with other flow-based VFI methods in Section 7.6 Finally, we discuss the potential broader impacts Section 7.7
7.1 Method Details
Flow metrics. Following Splatting- based Synthesis [35,. the implementation of the forward warping operation within our GIMM module leverages flow metrics to enhance accuracy. More precisely, we employ flow consistency flow and flow variance U U For illustrative purposes, taking Fo--1 as representative example, we compute its associated metrics as follows:
Uito = Fo-+1 + to (Fi-so, Fo-+1)ll1,
(16)
U var IIVG((Fo-)))) ((Fo-+1)
(17)
where to is the backward warping operator, and G(.) denotes a x 3 Gaussian filter.
Reverse flow normalization. In GIMM-VFL VFI, we employ reverse flow normalization to convert the predicted normalized flow, V,, into bilateral flows for interpolation purposes. This transformation expressed through the following equations:
Fi = -
(18)
Fi = (1 -1-t)-6
(19)
In this context, ø-1 represents the scaling operation that extends the value range from [0, 1] to IR using an instance-specific scaling factor. This reverse flow normalization operation identical to the used in IFE [3].
7.2 Network Architecture
Concat Saaacat(Kgg-... K1-1t)
Cony 3x3 (16)
Conv 3x3 (32)
Ros. Unit (32)
Ros. Unit (32)
Ros. Unit (32)
Cony 3x3 (32)
Cony 3x3 (64)
Ros. Unit (64)
Vi)
(Ko, Ki)
Concat(K,,, Ki, Ko, Ki-s)
Conv
Motion Encoder
Latent Refiner
Lincar (35, 128)
Linear 1228, 128)
Linear 128, 128)
Linear 1228 128)
Liner (128,
Concet (x, L.)
Sin
Sm
Sin
V.
Coodinate-basee Network ge
Figure 6: Details of the network architectures within the GIMM Module. The numbers in parentheses denote the output channel count for convolutional layers or the dimensionality shift from input to output for linear layers.
14

---

(Ge, Ci)
AMT
Becirrational
Correlation
Uidaaee Block
(Po-s, Pan)
Decoder
Decoder
Refinement
Multi-feel
4
M
43)
Figure 7: Overview of the frame synthesis module. The frame synthesis module conducts multi- scale flow refinements and predicts warping masks for interpolation, including the intermediate warping mask M2 predicted by the initial decoder. It integrates the images and their pyramid features A Ai Bidirectional Correlation Volume, which is constructed with correlation features (Co, C1). Ultimately, the outputs from the Final Decoder to perform the final interpolation of through the Multi-fich fieldfidd Refinement block. This block is adapted from AMT-G 29]. Modified components are highlighted in green for clarity.
during decoding, updating predictions with correlation information from the
x3
x8
Coev 5xd PawdSShuflee
Chee 3x3 (64)
Cony (128)
Cont (128)
BrtchNomm
BiRBk (1285)
Canv Sed (32)
Conv 3x3 (64)
Conv 1xl (64)
RandiNorm
A))
Kel. U
Degeer
10 Mi)
4D)
Pla Commmnn
Desper Rell
RRBBlock
Multi-ficke Flow
Residuals
(To, I1 Fe Fest)
(Jo, F/ Fb.
Initial Decoder
Final Decoder
Figure 8: The network architectures of the modified decoders within the frame synthesis module. Unless specifically noted, each convolutional layer is activated by PReLU function. The Deeper IFRBlock (highlighted in red) is based on the decoder architecture (IFRBlock) introduced IFR- Net (26). In particular, enhance the original IFRBlock by adding additional residual blocks to create the Deeper IFRBlock.
GIMM module. The architectural details of the networks used in GIMM are shown Fig. 6 The Motion Encoder and Latent Refiner use similar convolutional network architecture with residual units. Each Residual Unit consists of two convolutional layers with skip connection. The major differences between them the number of channels and residual units. On the other hand, the coordinate- basedd network go is designed as a five-layer SIREN 48), incorporating the motion latent L, auxiliary input.
Frame synthesis module. Adapted from AMT-G 29,, the framework of the Frame synthesis module is illustrated in Fig. 7 This module integrates two unique decoding stages: adapted initial decoder and a revised final decoder, while the rest the module's ' architecture remains unchanged from AMT-G. The architectural specifies of these modified decoders are shown in Fig. g
7.3 Implementation Details
In addition to experiments in the main text, we present more detailed implementations of GIMM and GIMM-VFI in this section.
GIMM training, We begin by extracting optical flows from the training split of the Vimeo90K
triplets dataset (54) using Flowformer [19]. With these extracted flows, we train GIMM, randomly cropping the flows to resolution of 256 256. For each batch during training, we randomly select timestep / from the set (0, 0.5, 1) to supervise. We set the batch size to 64, and train the model for 240 epochs with a learning rate of 1 10-1 using NVIDIA V 100 GPUs.
15

---

GIMM-VFI training. We integrate the pre-trained GIMM module and flow estimator into the GIMM- VFI model for end-to-end training on the uibitrar-timmeste interpolation task. We implement two variants of GIMM-VFI, using two different flow estimators: the RAFT [5) and FlowFormer [1) designated as GIMM- -FI-R and GIMM- VFI-F, respectively. However, both versions of GIMM-VFI share the same training process. Similar to previous works (55] 20]. we train our model on the complete Vimeo90K septuplet split [54] for 60 epochs with a batch size of 32 and a learning rate of 8 x 10-5. We randomly select triple subsets for training from each septuplet, following the same sampling strategy as previous research (55] 20) We resize and randomly crop each frame into a resolution of 224 x 224 and perform a series of augmentations including rotation, flipping. temporal order reversing and channel order reversing. We train our model on 8 NVIDIA V100 GPUs.
7.4 Training hyperparameters
In addition to the implementation details in Sec. 7.3 we summarize the training settings for both GIMM and GIMM-VFI in Tab.
Table 5: Training settings for GIMM and GIMM-VFL.
Configuration
GIMM
GIMM-VFI
Optimizer
AdamW
Peak learning rate
le-4
8e-5
Minimum learning rate
le-4
8c-6
Epochs
240
60
Batch size per GPU
Weight decay
4e-5
Optimizer momentum
B1. 32 = 0.9, 0.999
Learning rate schedule
55
Cosine annealing
Warmip epochs
0
1
Training Resolution
256
256
224 x 224
Num. V100 GPUs
8
7.5 Qualitative Results of GIMM Motion Modeling
We present additional qualitative results of GIMM motion modeling in Figure 9 The visualization showcases the modeled motion of GIMM and variants in the ablation study. Our observation confirms the effectiveness of GIMM's design and is consistent with the results from the ablation study in the main text. For example, the head motion in Figure significantly deteriorates with noises across all timesteps when forward warping replaced or latent refinement is skipped.
7.6 Integration with other flow-based VFI methods
Our method GIMM focuses on continuous motion modeling, which further enables frame inter- polation arbitrary timesteps. We additionally integrate GIMM with other existing flow-based VFI approaches (e.g., TTVFI (31. IFRNet (26)), by supplying accurately modeled motion. For evaluations, we calculate PSNRs on arbitrary- interpolation benchmark SNU-FILM-arb in T We observed that integrating the GIMM module resulted in significant improvements. This further demonstrates the effectiveness of GIMM for continuous motion modeling when integrated with existing blow- VFI works.
7.7 Broader Impacts
Our proposed method, GIMM-VFI, demonstrates strong performance in continuous frame interpo- lation, with applications spanning real-world scenarios such as creating cinematic special effects and producing slow-motion videos. This technique not only enhances these use cases but also opens doors for further research in video technology. At its core, CIMM-VFI employs the GIMM module for continuous motion modeling in video frame interpolation, a feature that could also be adapted for related tasks like video compression and predictive video analysis. Despite its strengths, our approach shares limitations with other frame interpolation methods and, in some cases, may introduce artifacts that could pollute visual data on the Internet.
16

---

Overlaid Input Image
Overlaid Input Flows
10
2
80
21
@ on
Non-Fwarp
Non-Imp
Non-Refiner
T-coord only
Full
Ground Truth
Figure 9: Visual comparisons for 6X motion modeling on the Vimeo-septuplett-loox We compare GIMM with its model variants mentioned in the ablation study of the main text.
Table 6: Integrating GIMM with TTVFI and IFRNet.
SNU-FiLMMarb-Mdiium
SNU-FILM-arb-Hard
SNU-FILL-arb-EEtteenn
Motion method
PSNRT
PSNR1
PSNR1
TTVEI
34.48
30.39
26.24
TTVEL+GMM
35.55 (+1.07dB)
31.60 (+1.21dB)
27.40 (+1.16dB)
IFRNet
34.88
31.15
26.32
IFMM
36.46 (+1.58dB)
32.20 (+1.05dB)
27.73 (+1.41dB)
17

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have made clear claims in the abstract and introduction that accurately reflect paper's 's contributions the studied problem. Please refer to the abstract and introduction for more detailed information.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
18

---

Justification:
We have discussions on the limitations of our proposed GIMM-VFI method in Jf the main text.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details and training hyperparameters in the supplementary materials.
Guidelines:
19

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are unable to provide our code upon submission, but releasing the code to the public in the future is our plan.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nis.cc/public//uust http://///giiieee/eeeuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
20

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the general training information and evaluation settings in the main text. The full implementation details and training hyperparameters presented in the supplementary materials.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
This paper does not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the type and the number of GPUs used in experiments.
Guidelines:
21

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
We carefully follow the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
We discuss the potential impacts in the supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
22

---

Justification: Our paper focuses on the problem of motion modeling in the video frame interpolation task, and poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
In our paper, we have cited the related papers.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwwpssppppsssseeeeeeeeeeee thcode wwt.coo///aaaa comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: In our paper, we propose new method GIMM-VFL. Upon submission, we do not provide the source code. But we will make the code public in the future after acceptance.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
23

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "UMM0w35Bdg", "reviewer": "Reviewer_d4Ah", "summary": "The paper proposed a video frame interpolation model, starting from an optical flow, encoding flows to spatial-temporal motion latent. The motion prediction model GIMM, took the encoded initial motion latent to arbitrary-timestep interpolation motions. Finally, the motions are used to predict bilateral flows which can warp with input frames to recover the interpolation result.", "questions": "1. Is the flow normalization model necessary? What are potential temporal inconsistencies mentioned in line 146?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper proposed a novel flow estimating model, which can estimate bidirectional flows according to input time steps and input frames' flows. The model can be inserted with any flow-estimating VFI model.\n2. Quantitative and qualitative results show that the estimated flows are stable and sharp compared with baseline results. Providing a better interpolation result.", "weaknesses": "1. The GIMM starts from pretrained flows, which may limit the network performance. In fact, recent VFI networks still tend to estimate blurry results according to inaccurate flow estimating while facing hard cases. \n2. More perceptual quantitative indicators need to be shown in the paper, eg. FID or LPIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. Please find the following for our response.\n\n> **Q1:** The GIMM starts from pretrained flows, which may limit the network performance. In fact, recent VFI networks still tend to estimate blurry results according to inaccurate flow estimating while facing hard cases.\n\n**A1**: Please kindly refer to the \"Pretrained flow estimator\" section in the global response.\n\n> **Q2:** More perceptual quantitative indicators need to be shown in the paper, eg. FID or LPIPS.\n\n**A2**:  Thank you for the suggestion. We add the perceptual metrics FID and LPIPS to Table 3 (page 8). We report the best results in **boldface** and the second best with ***Italic boldface***. The results are listed below:\n\n| Method | XTest-2K (LPIPS/FID) | XTest-4K (LPIPS/FID) | SNU-FILM-arb-4x (LPIPS/FID) | SNU-FILM-arb-8x (LPIPS/FID) | SNU-FILM-arb-16x (LPIPS/FID) |\n| --- | --- | --- | --- | --- | --- |\n| RIFE | 0.126/11.99 | 0.152/13.52 | 0.038/6.65 | 0.072/11.99 | 0.134/19.82 |\n| IFRNet | 0.108/23.93 | 0.164/23.75 | 0.046/9.92 | 0.066/11.65 | 0.115/16.91 |\n| M2M | **0.098**/9.25 | 0.158/8.67 | 0.036/5.98 | 0.061/10.13 | 0.112/17.37 |\n| AMT | 0.153/13.92 | 0.187/13.97 | 0.072/9.25 | 0.089/10.34 | 0.136/14.72 |\n| UPR-Net | 0.104/10.75 | 0.154/9.45 | ***0.033***/6.09 | 0.064/***9.93*** | 0.111/***16.76*** |\n| EMA-VFI | ***0.097*** /7.21 | 0.156/8.61 | 0.041/7.07 | 0.074/12.17 | 0.130/19.58 |\n| CURE | 0.111/26.42 | OOM | 0.035/6.98 | 0.063/12.72 | 0.114/22.62 |\n| GIMMVFI-R | 0.113/**6.52** | ***0.149***/**6.49** | ***0.033***/***5.89*** | ***0.060***/**9.59** | ***0.110***/**16.45** |\n| GIMMVFI-F | 0.103/***6.74*** | **0.142**/***6.58*** | **0.031**/**5.86** | **0.059**/9.95 | **0.109**/16.79 |\n\nOur proposed method GIMM-VFI achieves competitive performance according to these perceptual quantitative indicators.\n\n> **Q3:** Is the flow normalization model necessary?\n\n**A3**: As described in the “Flow normalization” paragraph in Section 3.2 (page 4), the normalization process aligns temporal direction and scales the values of input flows. First, without temporal direction alignment of the inputs, it would be hard to define the direction of the output flow $V_t$. The direction alignment is thus necessary. Second, for the scale operation, we experiment by skipping it and present results on VTF and VSF as below:\n\n| Method | Scale Operation | VTF(PSNR) | VTF(EPE) | VSF(PSNR) | VSF(EPE) |\n| --- | --- | --- | --- | --- | --- |\n| GIMM | No | 32.07 | 1.28 | 22.23 | 10.72 |\n| GIMM | Yes |**37.56** | **0.34** | **30.45** | **2.68** |\n\nSkipping the scale operation performs much worse than the full GIMM settings across both benchmarks. It is therefore necessary to perform the complete flow normalization operation.\n\n> **Q4**: What are potential temporal inconsistencies mentioned in line 146?\n\n**A4**: The potential temporal inconsistencies refer to the possible bias and noises in the estimated flows between input frames. In order to mitigate their negative impacts on motion modeling, we employ a Motion Encoder (ME) to extract motion features. The ablation experiment on Motion Encoder can be found in the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2ZlQ3k8edO", "reviewer": "Reviewer_LUi9", "summary": "This paper aims to solve the Video Frame Interpolation task. To improve the capability of effectively modeling spatiotemporal dynamics, the paper proposes a Generalizable Implicit Motion Modeling (GIMM) module to leverage the implicit neural fields to estimate the flow field at an arbitrary time step. GIMM takes the spatial coordinates, time, and the latent motion code as the input of an INR, which makes it generalizable to new scenes.", "questions": "1. The paper is easy to read. \n2. The paper concatenates the motion latent code and the coordinates as the input to achieve generalizable INR. It seems the latent code dominates the input as it has a higher dimension. Will it make the INR less spatial-sensitive? \n3. A follow-up question is why not use meta-learning approaches or INR modulation [6]?", "limitations": "The paper discussed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The idea of using INR to achieve continuous time video frame interpretation is interesting but needs to be well investigated. \n2. The performance of arbitrary-timestep interpolation is good.", "weaknesses": "1. Novelty: The idea of using time-dependent generalizable INR is not new and has been explored in human motion synthesis [1]. [1] also uses the concatenation of spatial coordinates, time step and motion latent as the input for an INR MLP. \nMoreover, the way of using INR is very similar to CURE[41] in video interpolation. Can you elaborate more details on the difference between the two methods? It seems the performance improvement over CURE might be attributed to the pre-trained flow estimator. \n\n[1]. Wei, Dong, et al. \"NeRM: Learning Neural Representations for High-Framerate Human Motion Synthesis.\" ICLR 2024. \n\n2. Performance: In Table 1, GIMM (-VFI-R) only slightly outperforms a simple \"Linear\" approach on both Vimeo-Septuplet-Flow (VSF) and SNU-FILM-arb-Hard. \nAlso, in Table 2, without spatial coordinates, the performance slightly drops. Do you even conduct ablation on the time variable? Maybe with only motion code as the input, the performance still remains similar? \n\n3. The method relies on the pre-trained flow estimators, which may lead to a suboptimal solution. \n\n4. Typo: \"Latnet Refiner\" in Figure 2 caption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. Please find the following for our response.\n\n> **Q1**: Novelty: The idea of using time-dependent generalizable INR is not new and has been explored in human motion synthesis [1]. [1] also uses the concatenation of spatial coordinates, time step and motion latent as the input for an INR MLP.\n\n**A1**: We respectfully disagree with the reviewer that the idea of using time-dependent generalizable INR for motion modeling in Video Frame Interpolation lacks novelty due to the presence of NeRM. Although NeRM and our proposed method GIMM both leverage INR conditioned on coordinates and latent, they are quite different concerning the following main points. **1) GIMM performs the general type of motion modeling in the context of video frame interpolation.** Unlike NeRM which focuses on human motion, GIMM models any types of motion that may exist within the input frames. 2) While NeRM synthesizes **sparse** pose motion at each timestep, GIMM predicts **dense** motion in the form of optical flow for the usage of interpolation.  Consequently, **GIMM requires spatial coordinates** as the additional input to its INR for better modeling, akin to image-based INRs [5]. In contrast, NeRM only takes temporal coordinates. We plan to include NeRM in our references to enhance the comprehensiveness of our related work section. We will add this discussion in the revised manuscript.\n\n> **Q2**: Moreover, the way of using INR is very similar to CURE[41]... performance improvement over CURE might be attributed to the pre-trained flow estimator.\n\n**A2**: As described in the “Generalizable INRs.” paragraph in Section 2 (page 3), CURE directly learns generalizable INRs from video, while our method leverages generalizable INRs for motion modeling to improve intermediate frame synthesis for flow-based VFI. Besides, we would like to clarify that despite both CURE and our method have leveraged pre-trained flow estimators, i.e., RAFT, CURE still performs linear motion modeling with a strong assumption of motion overlapping that should lead to suboptimal results. As observed from Table 3, our GIMM-VFI-R consistently outperforms the CURE algorithm across all benchmarks, achieving PSNR improvements exceeding 1 dB. Additionally, GIMM-VFI-R effectively handles interpolation at 4K resolution where CURE encounters out-of-memory issues. Furthermore, CURE contains a heavier architecture of 51.28M parameters (31.49M larger than our GIMM-VFI-R) and requires a longer time in inference. \n\n> **Q3**. Performance: In Table 1, GIMM (-VFI-R) only slightly outperforms a simple \"Linear\" approach on both Vimeo-Septuplet-Flow (VSF) and SNU-FILM-arb-Hard.\n\n**A3**: As observed in Table 1 (page 7), the PSNR of our method is 0.36 dB and 0.20 dB higher than the \"Linear\" approach on the referred benchmarks, respectively. This improvement is significant, particularly considering that both methods utilize the pretrained flow estimator RAFT.\n\n> **Q4**: Also, in Table 2, without spatial coordinates, the performance slightly drops. \n\n**A4:** As shown in Table 2 (page 8), removing spatial coordinates causes a 0.16 dB **drop of PSNR** on VTF and a 0.06 **increase of End-Point-Error** on VSF. This demonstrates the crucial role of spatial coordinates.\n\n> **Q5**: Do you even conduct ablation on the time variable? Maybe with only motion code as the input, the performance still remains similar?\n\n**A5:** As presented in Table 2 (page 8) and analyzed in the “Implicit modeling.” paragraph in Section 4.2 (page 7), direct input motion latent code without using any coordinates results in a 0.52 dB **decrease of PSNR** on VTF and a 0.13 **increase of End-Point-Error** on VSF. This highlights the importance of implicit modeling in GIMM. \n\n> **Q6**: The method relies on the pre-trained flow estimators, which may lead to a suboptimal solution.\n\n**A6**: Due to the word limit, please refer to the \"Pretrained flow estimator\" section in the global response.\n\n> **Q7**: The paper concatenates the motion latent code and the coordinates as the input to achieve generalizable INR. It seems the latent code dominates the input as it has a higher dimension. Will it make the INR less spatial-sensitive?\n\n**A7**: Thank you for the suggestion. We reduce the dimension of motion latent from 32 to 8 and the results on VTF and VSF are listed below:\n\n| Latent Dim. | VTF(PSNR) | VTF(EPE) | VSF(PSNR) | VSF(EPE) |\n| --- | --- | --- | --- | --- |\n| 8 | 37.16 | 0.35 | 30.15 | 2.74 |\n| 32 |**37.56** | **0.34** | **30.45** | **2.68** |\n\nReducing latent dimension leads to worse performance, with reductions of **0.40dB** and **0.30dB** in the PSNRs on VTF and VSF respectively. This indicates that a proper choice of a higher dimension, e.g., 32, will help generalizable INR achieve better performance rather than impose negative effects on the implicit modeling. \n\n> **Q8**: A follow-up question is why not use meta-learning approaches or INR modulation [6]?\n\n**A8**: As suggested, we implement the Meta-learning approach [23] for modulating the weights of INR, specifically in modeling motion. We compare its performance on VTF and VSF with our method GIMM, and calculate their parameters. The results are listed below:\n\n| Method | VTF(PSNR) | VTF(EPE) | VSF(PSNR) | VSF(EPE) | Param. |\n| --- | --- | --- | --- | --- | --- |\n| Meta-learning approach [23] | 30.19 | 0.88 | 24.50 | 6.80 | 43.92M |\n| GIMM |**37.56** | **0.34** | **30.45** | **2.68** | **0.25M** |\n\nThe Meta-learning approach performs much worse than GIMM while adding more than 170x larger model parameters.\n\n> **Q9**: Typo: \"Latnet Refiner\" in Figure 2 caption.\n\n**A9**: We will correct the typo in our revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QereID7sD6", "reviewer": "Reviewer_rbyn", "summary": "This paper proposes a plug-and-play Generalizable Implicit Motion Modeling module to refine the optical flow in the task of video frame interpolation. Specifically, this module combines several core components of normalization, motion encoder, latent refiner, and coordinate-based network to achieve implicit motion modeling. Experimental results demonstrate the superiority of the optical flow obtained by the proposed GIMM approach.", "questions": "I'm most concerned about the efficiency of GIMM and the improvement it brings when plugged into other flow-based VFI methods.", "limitations": "The authors have discussed the limitations.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The accuracy of optical flow is an important issue that affects the effectiveness of video frame interpolation. It is valuable for the authors to model more accurate motion through implicit motion modeling.", "weaknesses": "1.The K, F, and Z symbols in Equations 6-7 should be labeled in Figure 2. In addition, the description of L150-151 for K is confusing to read, especially F is expressed as the difference between two coordinates.\n\n2.How does Equation 11 get the optical flow of GT? As far as I know, none of the existing VFI datasets have a corresponding optical flow.\n\n3.Table 1 allows for a comparison of some of the methods that model more complex motion than just modeling linear motion, such as QVI, ABME, BMBC.\n\n4.Table 3 should include parameter and runtime comparisons to demonstrate the effectiveness of the GIMM.\n\n5. As a plug-and-play motion modeling module, Table 3, the authors should add GIMM to other existing flow-based VFI methods for a fairer comparison, e.g., ABME, TTVFI, RIFE, and IFRNet. Because the design of the optical flow estimation network and synthesis network used in the different methods are not the same, and the use of a higher performance optical flow estimation network also has a significant performance improvement. In addition, validation on vimeo and UCF101 is necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. Please find the following for our response.\n\n> **Q1**: The K, F, and Z symbols in Equations 6-7 should be labeled in Figure 2. In addition, the description of L150-151 for K is confusing to read, especially F is expressed as the difference between two coordinates.\n> \n\n**A1**: Thanks for your suggestion. We will add the K, F, and Z symbols to Figure 2 and reformulate the description of L150-151 in our revised manuscript. \n\n> **Q2**: How does Equation 11 get the optical flow of GT? As far as I know, none of the existing VFI datasets have a corresponding optical flow.\n> \n\n**A2**: As described in Section 4 (page 6), we utilize Flowformer [17] to produce pseudo-ground truth optical flow for training and evaluation usages. \n\n> **Q3**: Table 1 allows for a comparison of some of the methods that model more complex motion than just modeling linear motion, such as QVI, ABME, BMBC.\n> \n\n**A3**: Thanks for your suggestion. We would like to clarify that our GIMM focus on continuous motion modeling for arbitrary-timestep interpolation with 2 frames as the input. While QVI takes 4 frames as input and ABME can only predict at the timestep of 0.5, BMBC is the only method that allows for continuous motion modeling and interpolation. Therefore, we extend Table 1 with experiments on BMBC. The results of BMBC’s motion modeling and interpolation compared with GIMM are listed below. \n\n| Method | VTF (PSNR) | VTF (EPE) | VSF (PSNR) | VSF (EPE) | SNU-FILM-arb-Hard |\n| --- | --- | --- | --- | --- | --- |\n| BMBC | 28.89 | 0.95 | 23.19 | 8.23 | 28.51 |\n| GIMM (-VFI-R) |**37.56** | **0.34** | **30.45** | **2.68** | **32.62** |\n\nOur GIMM outperforms BMBC concerning both motion modeling and interpolation. This further demonstrates the effectiveness of our method. We will add this to Table 1 (page 7), in our revised manuscript. \n\n> **Q4**: Table 3 should include parameter and runtime comparisons to demonstrate the effectiveness of the GIMM.\n> \n\n**A4**: Please kindly refer to the \"Parameters and Runtime\" section in the global response.\n\n> **Q5**: As a plug-and-play motion modeling module, Table 3, the authors should add GIMM to other existing flow-based VFI methods for a fairer comparison, e.g., ABME, TTVFI, RIFE, and IFRNet. Because the design of the optical flow estimation network and synthesis network used in the different methods are not the same, and the use of a higher performance optical flow estimation network also has a significant performance improvement. In addition, validation on vimeo and UCF101 is necessary.\n\n**A5-1**: We would like to clarify that our method GIMM focuses on **continuous** motion modeling, which further enables frame interpolation at **arbitrary timesteps**. Arbitrary-timestep interpolation relies more on continuous modeling while fixed-timestep interpolation relies more on model fitting at the specific timestep of 0.5. Notably, plugging in a better continuous modeling module doesn’t guarantee better model fitting since model fitting requires more on the model’s learning strategies and its overall design. Therefore, in terms of continuous modeling, we evaluate the plug-in ability of our proposed GIMM on arbitrary-timestep interpolation benchmark SNU-FILM-arb rather than fixed-timestep interpolation benchmarks, such as Vimeo90K and UCF101. Since there is a time limit for the rebuttal, we plug in the GIMM module to two of the representative existing flow-based VFI methods, TTVFI and IFRNet, for the experiment. Particularly, we plug in GIMM with a pretrained flow estimator RAFT to IFRNet, since there is not an existing flow estimator in the model. The results of PSNRs are listed below. \n\n| Method | SNU-FILM-arb-4X | SNU-FILM-arb-8X | SNU-FILM-arb-16X |\n| --- | --- | --- | --- |\n| TTVFI | 34.48 | 30.39 | 26.24 |\n| TTVFI+GIMM | **35.55 (+1.07dB)** | **31.60 (+1.21dB)** | **27.40 (+1.16dB)** |\n| IFRNet | 34.88 | 31.15 | 26.32 |\n| IFRNet+GIMM | **36.46 (+1.58dB)** | **32.20 (+1.05dB)** | **27.73 (+1.41dB)** |\n\nPlugging in the GIMM module results in significant improvements for arbitrary-timestep interpolation. This demonstrates the effectiveness of GIMM for continuous modeling when integrated with existing flow-based VFI works. We will add this experiment to the Supplementary of our revised manuscript.\n\n**A5-2**:In order to prove GIMM-VFI's effectiveness of the overall design for interpolation, we further provide evaluations on Vimeo90K and UCF101. Following EMA-VFI [50], we train GIMM-VFI for fixed-timestep interpolation. For evaluation, we calculate PSNRs and compare our method GIMM-VFI with the aforementioned and other state-of-the-art methods. The results are listed below:\n\n| Method | Vimeo90K (PSNR) | UCF101 (PSNR) |\n| --- | --- | --- |\n| IFRNet | 36.20 | 35.42 |\n| AMT | 36.53 | 35.45 |\n| UPR-Net | 36.42 | 35.47 |\n| ABME | 36.22 | 35.41 |\n| TTVFI | ***36.54***| ***35.51*** |\n| EMA-VFI | 36.50 | 35.42 |\n| CURE | 35.73 | 35.36 |\n| GIMM-VFI-R | ***36.54*** | ***35.51*** |\n| GIMM-VFI-F | **36.67** | **35.54** |\n\nTwo variants of GIMM-VF with different flow estimators, both achieve competitive performance. This further demonstrates our method's strong interpolation ability.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OQDCXOkBV9", "reviewer": "Reviewer_ZRiZ", "summary": "The authors propose GIMM to effectively model intermediate motions. Three core designs are: normalization over the initial bidirectional flows, motion encoding (spatiotemporal motion latent extraction from flows) and adaptive coordinate-based INR. \nThe framework first extracts bidirectional flows of the input frames via off-the-shelf optical flow models (e.g., RAFT) and normalizes them. Next, a Motion Encoder extracts motion features from the normalized flows. The motion feature maps are forward warped respective to the target timesteps. A Latent Refiner refines the warped motion features, and using the refined motion features, a coordinated-based MLP network predicts the normalized flow maps from the target timestep, which could be reversed to the bidirectional flows of original scales for frame synthesis. This implicit motion modeling framework is first individually trained with reconstruction loss on the predicted flow maps. Once it is pre-trained, a frame synthesis module is attached at the end of GIMM, which is jointly trained in an end-to-end manner with frame reconstruction loss.", "questions": "1. What is the input / output of coordinate-based MLP (INR), and why are the spatial coordinates necessary?\n- I had a hard time trying to understand this. To my understanding, the motion latent code L_t from the Latent Refiner would have a shape of H x W x D where H, W, D denote height, width and dimension, respectively. According to the main paper and the supplementary, the latent L_t and 3D coordinates x,y,t are concatenated and fed to the SIREN network. In that case, the concatenated representation would have a preserved spatial size of HxW, with each coordinate having different latent code L_t. \nIf that is the case, as mentioned in the weakness section above, I believe there is not much reason to include the spatial (x,y) coordinates as the input of the INR network as the network predicts the latent flow of its own spatial coordinate, and I even wonder why an INR form is necessary. For example, can't it be simply be in a form of a conditional U-Net, with target time-steps given as in diffusion literature? The experiments in Table 2 show that the use of spatial coordinates do affect the performance, but I failed to understand the reason for this. Could there be an explanation on the reason for the increase? There could possibly be some misunderstandings by me and wish for some clarifications, as it is a very important component of the paper.\n\n\n2. What is the input / output of the Latent Refiner?\n- According to the Fig.2 of the main paper, the output of the Motion Encoder K_i seems to be fed to the Latent Refiner. However, in the supplementary, Latent Refiner does not seem to take K_i as input, but only the warped motion features only. I wonder what is the right visualization.\n- In connection to the #1 question above, I understood that the output of Latent Refiner to be of shape H x W x D. I feel that it should be correct, but feel uncertain since there is not much description on it.\n\n\n3. How is the normalization of flows done? With maximum scalar values? With a large constant value? Or by log-scaling? I wonder how it is done, for the paper to be self-contained.\n\n4. How is the forward-warping method evaluated for motion modeling in Table 1? To my understanding, the backward flows from the target timestep, F_{t->0} and F_{t->1} are used for evaluation. However, forward warping could only provide F_{0->t} and F_{1->t}. which does not sound like a fair comparison. Did the authors use techniques such as flow reversal [48] or complementary flow reversal [42]?\n\n5. I wonder the computational costs of the framework, i.e., number of parameters, runtime, with comparison to state-of-the-art methods. The proposed framework seems to consist of many modules which could possibly require vast computational costs.", "limitations": "Yes, the authors have addressed their limitations.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Strong results: the visualized video in the supplementary video and the visualized motions fields look great in quality.\n2. To my knowledge, use of INRs for motion modeling in VFI, without training at test time is a novel approach.", "weaknesses": "1. Lack of Experiments\n\n1.1 Benchmarks\n- The authors use the test set from X4K1000FPS [42] and SNU-FILM-arb for evaluation, where SNU-FILM-arb is a dataset introduced by themselves. Although there are diverse public datasets available for evaluation, they do not report the experimental results on the public datasets prevalently used in the field. Namely, Vimeo90k [49]  is the most commonly used dataset, which they used for training and evaluation of modeled flows. I wonder the frame interpolation performance on Vimeo90k, as I believe there is no reason not to report them. In addition, I think there could have been other options for arbitrary-timestep frame interpolation benchmarks, e.g., Adobe240fps [52]. The proposed SNU-FILM-arb could be a useful benchmark for future studies, but I think they should have shown the validity of their method with public benchmarks before that.\n     \n1.2 Ablation studies\n\nI find the ablation studies of their method is a bit limited. The main points of their arguments are not experimentally verified in the ablation studies. I find the ablation studies in Table 2 to be a little off-topic, not focused on the main arguments.\n- Normalization: although they claim normalization of flows over scales and directions is one of the key designs of the framework in the abstract and introduction, the experimental results on its effectiveness cannot be found in Section 4.2 and Table 2.\n- Motion encoding: similar to normalization, the experimental studies on the effectiveness of motion encoding cannot be found in the paper. They only show how the Latent Refiner affects the performance, but not the motion encoding process, which is one of the contributions that the authors claim.\n- Generalizability: The title and the abstract emphasizes the 'generalizability' of the method, but do not show any experiments on generalizability. Although they argue that their method can be smoothly integrated with existing flow-based VFI works, there is no experiment on it. This part especially makes me consider the paper to be a bit over-claimed.\n\n2. Weak Analysis / Explanations\n\nThis is aligned with my concern aforementioned in 1.2 Ablation studies. The authors do not give a good explanation of the results or try deeper analysis on their main arguments.\n- Normalization: In the abstract and introduction, the authors claim that normalization of flows are one of the key designs, but neither do they experimentally show or intuitively explain its importance / role. I think there should have been at least an intuitive explanation of its necessity, along with an experimental result supporting it. The authors mention that they perform normalization following IFE[13], but if that is the case simply following prior work, it cannot be a contribution of the paper. Should the authors of the paper claim this to be a contribution, it needs further analysis.\n- INR: to my understanding, this part is the biggest contribution of the paper. However, I am not very convinced with the use of INRs. To my understanding, the INR takes the 3D coordinate as the input along with the motion latent codes at each pixel. I wonder why the  spatial coordinates are necessary, as it will always predict the flow of its own spatial position. Furthermore, in that case, the target timestep t is the only part that would make the difference, which makes me curious why we have to adopt the INR form rather than another form of conditioning mechanism. For instance, I think a conditional U-Net used in the diffusion literature could suffice. The authors do not give a solid explanation of why the INR form is necessary, and I wish to hear a clarified explanation.\n\n3. Presentation\n\nI feel that some important details, especially the core designs, are not described sufficiently. For instance, although the INR part is the largest contribution of the method, it is explained with only a couple of lines in the main manuscript, and the supplementary materials does not provide sufficient explanations. My main questions (#2 above & #1 in Questions) rise from the ambiguous description of the INR part. What is the input / output of the INR model, how are the input / output tensors shaped, etc. I had to make assumptions on this part, which made me fundamentally wonder why the INR form is necessary.\n\n[52] Su, Shuochen, et al. \"Deep video deblurring for hand-held cameras.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. Please find the following for our response.\n\n> **Q1**:The authors use the test set ... I wonder the frame interpolation performance on Vimeo90k,...\n\n**A1**: Due to the word limit, please refer to the **global response** and **A5-2** in our response to Reviewer **rbyn**.\n\n> **Q2**: ... other options for arbitrary-timestep frame interpolation benchmarks, e.g., Adobe240fps [52].\n\nThanks for your suggestion. We calculate PSNR on the Adobe240fps following the evaluation settings of IFRNet [24] and use the test split divided by VideoINR [7]. The results are listed below:\n\n| Method | Adobe240fps (PSNR) |\n| --- | --- |\n| IFRNet | 31.08 |\n| AMT | 30.70 |\n| UPR-Net | 32.01 |\n| CURE | 31.64 |\n| EMA-VFI | 31.26 |\n| GIMM-VFI-R | ***32.31*** |\n| GIMM-VFI-F | **32.33** |\n\nOur method achieves stronger performance. This further illustrates the effectiveness of our method for the arbitrary-timestep VFI task.\n\n> **Q3**: What is the input / output of coordinate-based MLP (INR)\n\n**A3**: We concatenate of the motion latent of $D \\times H \\times W$ and spatiotemporal coordinates of $3 \\times H \\times W$ as the input to the INR. The INR outputs the corresponding normalized flow. \n\n> **Q4**: ... why an INR form is necessary ... can't it be simply be in a form of a conditional U-Net, with target time-steps given as in diffusion literature?\n\n**A4**: As described in Section 2, the INRs have effective modeling ability for complex, high-dimensional data and can learn **continuous mapping** from a set of coordinates to a specific signal. Thus, we are motivated to leverage the INR for **continuous motion modeling** at arbitrary timesteps. We experiment by replacing the INR with a timestep-conditioned U-Net as in diffusion literature. The results on VTF, VSF and model parameters are listed below:\n\n| Method | VTF(PSNR) | VTF(EPE) | VSF(PSNR) | VSF(EPE) | Params |\n| --- | --- | --- | --- | --- | --- |\n| GIMM (U-Net) | 36.96 | 0.39 | 29.96 | 2.90 | 4.27M |\n| GIMM (INR) | **37.56** | **0.34** | **30.45** | **2.68** | **0.25M** |\n\nReplacing INR with U-Net results in worse performance, especially on the 6X motion modeling benchmark VSF. This demonstrates the strong continuous modeling ability of INR. Besides, GIMM with INR has a much lighter architecture which improves the efficiency. Therefore, it is necessary and proper to use INR for continuous motion modeling.\n\n> **Q5**: ... the use of spatial coordinates do affect the performance, but I failed to understand the reason for this ...\n\n**A5**: As explained in **A4**, we are motivated to use INR for its effective modeling ability, learning a **continuous mapping** from a set of coordinates to a specific signal. The coordinates provide positional information for INRs and help to learn continuous mapping since the coordinates are inherently continuous. In our motion modeling, we aim to model continuous motion between timesteps. The motion refers to dense optical flows, which consist of spatiotemporal changes. Therefore, it is necessary to use spatial coordinates for effective motion modeling. The experiment results in Table 2 demonstrate the necessity in practice. \n\n> **Q6**: why the spatial coordinates are necessary, as it will always predict the flow of its own spatial position.\n\n**A6**: In addition to A5, we would like to further clarify that **motion latent code serves as a function space for the implicit representations of different instances**. The latent code is used for making INRs generalizable which means no need for test-time optimization.  The spatiotemporal coordinates are still required for continuous modeling of spatiotemporal changes, such as dense optical flow in our case. Similar insights are shared in the related literature [5]. \n\n> **Q7**: Generalizability:...\n\n**A7**: As described in Section 2 (page 3) and Section 3.2 (page 4), “generalizability” means the generalizable modeling ability of the generalizable INRs (GINRs) across different instances. Unlike per-instance modeling INRs, GINRs do not require test-time learning. Our GIMM takes the motion latent code as an additional input to the INR and achieves generalizable motion modeling without the need for test-time optimization. A similar definition for generalizability can be found in the referenced paper [23].\n\n> **Q8**: How is the forward-warping method evaluated for motion modeling in Table 1?\n\n**A8**: The forward-warping method produces 'backward' flows $F_{t→0}$ and $F_{t→1}$ from estimated flows $F_{0→1}$ and $F_{1→0}$ without flow reversal techniques. For example,  $F_{t→0}$ can be easily obtained from $F_{t→0}=Fwarp(F_{1→0}\\cdot t,F_{1→0}\\cdot(1-t))$, where $Fwarp(a,b)$ indicates forward warping the objective $a$ with the referenced motion vector $b$. \n\n> **Q9**: I wonder the computational costs of the framework.\n\n**A9**: Due to the word limit, please kindly refer to the **global response**.\n\n> **Q10**: How is the normalization of flows done? \n\n**A10**: The normalization process follows IFE[13]. We agree with the reviewer and will delete this part from our key designs in our revised manuscript.\n\n> **Q11**: Motion encoding: … the experimental studies on the effectiveness of motion encoding cannot be found in the paper…\n\n**A11**: Thanks for your suggestion. Due to the word limit, please refer to the **global response**.\n\n> **Q12**: What is the input / output of the Latent Refiner?\n\n**A12**: The Latent Refiner refiner takes both the motion features and the coarse motion latent (warped motion features) as input and outputs the residual for the coarse motion latent. We will update the figure of Latnet Refiner’s architecture in the supplementary of our revised manuscript.\n\n> **Q13**: Presentation. I feel that some important details, ..., What is the input / output of the INR model, how are the input / output tensors shaped, etc….\n\n**A13**: Thanks for your suggestion. We will specify more details about the INR in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We would like to thank all reviewers for providing constructive feedback that helped improve the paper. Due to the word limit, we provide explanations and experiments for concerns shared by multiple reviewers in the following.

1. **Ablation on Motion Encoder (reviewer d4Ah and ZRiZ)**

We conducted experiments without the Motion Encoder for a direct comparison. The results of motion modeling are presented below.

| Method | with ME | VTF(PSNR) | VTF(EPE) | VSF(PSNR) | VSF(EPE) |
| --- | --- | --- | --- | --- | --- |
| GIMM | No | 37.05 | 0.42 | 30.26 | 2.85 |
| GIMM | Yes | **37.56** | **0.34** | **30.45** | **2.68** |

GIMM implemented with ME produces higher-quality flows, demonstrating that Motion Encoder is indeed helpful for our motion modeling. We will add these results in Table 2 (page 8) of the revised manuscript.

2. **Evaluations on Vimeo90K and UCF101 (reviewer rbyn and ZRiZ)**

We would like to clarify that our method focuses on **continuous motion** modeling, specifically for the **arbitrary timestep interpolation task**. Therefore, we didn’t include the evaluations on the fixed timestep interpolation benchmarks Vimeo90K and UCF101 in our submission. However, our proposed method still achieves competitive performance on the fixed-timestep interpolation benchmarks. We report the best results in **boldface** and the second best with ***Italic boldface***.

| Method | Vimeo90K (PSNR) | UCF101 (PSNR) |
| --- | --- | --- |
| IFRNet | 36.20 | 35.42 |
| AMT | 36.53 | 35.45 |
| UPR-Net | 36.42 | 35.47 |
| ABME | 36.22 | 35.41 |
| TTVFI | ***36.54*** | ***35.51*** |
| EMA-VFI | 36.50 | 35.42 |
| CURE | 35.73 | 35.36 |
| GIMM-VFI-R | ***36.54*** | ***35.51*** |
| GIMM-VFI-F | **36.67** | **35.54** |

Two variants of GIMM-VF with different flow estimators, both achieve competitive performance. This further demonstrates our method's strong interpolation ability.

3. **Pretrained flow estimator (reviewer d4Ah and LUi9)**

As shown in Table 1 (page 7) and discussed in this paragraph (line 225, page 7), many works [20,32] leverage motion priors from RAFT [45] can achieve better performance than method without motion priors [50].  Notably, our GIMM outperforms existing methods [20,32] that utilize the same motion priors.  This demonstrates that VFI methods can significantly benefit from pretrained flow estimators. Our GIMM offers the most substantial improvements. Furthermore, in Table 3 (page 8, GIMM-VFI-F vs. GIMM-VFI-R), we show that integrating a better flow estimator (FlowFormer [17]) can enhance model performance.

4. **Parameters and Runtime (reviewer rbyn and ZRiZ)**

Following RIFE [19], we collect the models of each paper and test them on an NVIDIA V100 GPU with the same hardware for 480P frame interpolation. We report the parameters and runtime of each model and list them as below:

| Method | Params. (M) | Runtime (s/f) |
| --- | --- | --- |
| RIFE | 10.71 | 0.01 |
| IFRNet | 19.7 | 0.03 |
| M2M | 7.61 | 0.01 |
| AMT | 2.99 | 0.03 |
| UPR-Net | 1.65 | 0.04 |
| EMA-VFI | 65.66 | 0.08 |
| CURE | 51.28 | 0.98 |
| GIMMVFI-R | 19.79 | 0.25 |
| GIMMVFI-F | 30.59 | 0.29 |

Compared with INR-based interpolation methods, our proposed method performs the fastest interpolation and maintains a relatively light architecture. However, there is still a runtime gap between the INR-based method and the non-INR-based interpolation method, we leave it for future research.

5. **Normalization (reviewer d4Ah and ZRiZ)**

As described in the “Flow normalization” paragraph in Section 3.2 (page 4), the normalization process aligns temporal direction and scales the values of input flows. First, without temporal direction alignment of the inputs, it would be hard to define the direction of the output flow $V_t$. The direction alignment is thus necessary. Second, for the scale operation, we experiment by skipping it and present results on VTF and VSF as below:

| Method | Scale Operation | VTF(PSNR) | VTF(EPE) | VSF(PSNR) | VSF(EPE) |
| --- | --- | --- | --- | --- | --- |
| GIMM | No | 32.07 | 1.28 | 22.23 | 10.72 |
| GIMM | Yes | **37.56** | **0.34** | **30.45** | **2.68** |

Skipping the scale operation performs much worse than the full GIMM settings across both benchmarks. It is therefore necessary to perform the complete flow normalization operation.

Although normalization is important, the normalization process follows IFE[13]. We agree with reviewer ZRiZ and will delete it from our key designs in our revised manuscript.

Meta-review: The paper presents a method for continuous motion modeling for video frame interpolation using implicit neural representations. The role of the pretrained flow estimator is questioned by d4Ah, which is sufficiently addressed by the rebuttal. Requested perceptual metrics are evaluated in the rebuttal and should be included in the main paper with further comment or analysis. LUi9 requires comparative discussion with respect to NeRM and CURE, which are explained in the rebuttal. LUi9 also points out small PSNR gains from GIMM - VFI-R compared to linear method, where the rebuttal disagrees to consider them significant. All the clarifications sought by rbyn are sufficiently addressed by the rebuttal. ZRiZ questions the use or INRs and the necessity of spatial coordinates, whose utility and correctness for continuous motion modeling is clarified by the authors over extensive post-rebuttal discussions. The use of INR in favor of U-Net is also empirically justified in the rebuttal, while additional results are also included on other benchmarks. In balance, while quality improvements for VFI are small in the supplementary videos, there is technical novelty in the implementation of INR for continuous motion modeling and quantitative results show positive trends for future work, so the paper is recommended for acceptance at NeurIPS.
Decision: Accept (poster)