VMem: Consistent Interactive Video Scene Generation with Surfel-Indexed
  View Memory
Authors: name: Runjia Li, name: Philip Torr, name: Andrea Vedaldi, name: Tomas Jakab

We propose a novel memory mechanism to build video generators that can
explore environments interactively. Similar results have previously been
achieved by out-painting 2D views of the scene while incrementally
reconstructing its 3D geometry, which quickly accumulates errors, or by video
generators with a short context window, which struggle to maintain scene
coherence over the long term. To address these limitations, we introduce
Surfel-Indexed View Memory (VMem), a mechanism that remembers past views by
indexing them geometrically based on the 3D surface elements (surfels) they
have observed. VMem enables the efficient retrieval of the most relevant past
views when generating new ones. By focusing only on these relevant views, our
method produces consistent explorations of imagined environments at a fraction
of the computational cost of using all past views as context. We evaluate our
approach on challenging long-term scene synthesis benchmarks and demonstrate
superior performance compared to existing methods in maintaining scene
coherence and camera control.