VeGaS: Video Gaussian Splatting
Authors: name: Weronika Smolak-Dyżewska; name: Dawid Malarz; name: Kornel Howil; name: Jan Kaczmarczyk; name: Marcin Mazur; name: Przemysław Spurek
Published: 2024-11-17 10:02:36

Abstract: Implicit Neural Representations (INRs) employ neural networks to approximate
discrete data as continuous functions. In the context of video data, such
models can be utilized to transform the coordinates of pixel locations along
with frame occurrence times (or indices) into RGB color values. Although INRs
facilitate effective compression, they are unsuitable for editing purposes. One
potential solution is to use a 3D Gaussian Splatting (3DGS) based model, such
as the Video Gaussian Representation (VGR), which is capable of encoding video
as a multitude of 3D Gaussians and is applicable for numerous video processing
operations, including editing. Nevertheless, in this case, the capacity for
modification is constrained to a limited set of basic transformations. To
address this issue, we introduce the Video Gaussian Splatting (VeGaS) model,
which enables realistic modifications of video data. To construct VeGaS, we
propose a novel family of Folded-Gaussian distributions designed to capture
nonlinear dynamics in a video stream and model consecutive frames by 2D
Gaussians obtained as respective conditional distributions. Our experiments
demonstrate that VeGaS outperforms state-of-the-art solutions in frame
reconstruction tasks and allows realistic modifications of video data. The code
is available at: https://github.com/gmum/VeGaS.