Generalizable Implicit Motion Modeling for Video Frame Interpolation
Authors: name: Zujin Guo; name: Wei Li; name: Chen Change Loy
Published: 2024-07-11 17:13:15

Abstract: Motion modeling is critical in flow-based Video Frame Interpolation (VFI).
Existing paradigms either consider linear combinations of bidirectional flows
or directly predict bilateral flows for given timestamps without exploring
favorable motion priors, thus lacking the capability of effectively modeling
spatiotemporal dynamics in real-world videos. To address this limitation, in
this study, we introduce Generalizable Implicit Motion Modeling (GIMM), a novel
and effective approach to motion modeling for VFI. Specifically, to enable GIMM
as an effective motion modeling paradigm, we design a motion encoding pipeline
to model spatiotemporal motion latent from bidirectional flows extracted from
pre-trained flow estimators, effectively representing input-specific motion
priors. Then, we implicitly predict arbitrary-timestep optical flows within two
adjacent input frames via an adaptive coordinate-based neural network, with
spatiotemporal coordinates and motion latent as inputs. Our GIMM can be
smoothly integrated with existing flow-based VFI works without further
modifications. We show that GIMM performs better than the current state of the
art on the VFI benchmarks.