V^3: Viewing Volumetric Videos on Mobiles via Streamable 2D Dynamic
  Gaussians
Authors: name: Penghao Wang; name: Zhirui Zhang; name: Liao Wang; name: Kaixin Yao; name: Siyuan Xie; name: Jingyi Yu; name: Minye Wu; name: Lan Xu
Published: 2024-09-20 16:54:27

Abstract: Experiencing high-fidelity volumetric video as seamlessly as 2D videos is a
long-held dream. However, current dynamic 3DGS methods, despite their high
rendering quality, face challenges in streaming on mobile devices due to
computational and bandwidth constraints. In this paper, we introduce
V3(Viewing Volumetric Videos), a novel approach that enables
high-quality mobile rendering through the streaming of dynamic Gaussians. Our
key innovation is to view dynamic 3DGS as 2D videos, facilitating the use of
hardware video codecs. Additionally, we propose a two-stage training strategy
to reduce storage requirements with rapid training speed. The first stage
employs hash encoding and shallow MLP to learn motion, then reduces the number
of Gaussians through pruning to meet the streaming requirements, while the
second stage fine tunes other Gaussian attributes using residual entropy loss
and temporal loss to improve temporal continuity. This strategy, which
disentangles motion and appearance, maintains high rendering quality with
compact storage requirements. Meanwhile, we designed a multi-platform player to
decode and render 2D Gaussian videos. Extensive experiments demonstrate the
effectiveness of V3, outperforming other methods by enabling
high-quality rendering and streaming on common devices, which is unseen before.
As the first to stream dynamic Gaussians on mobile devices, our companion
player offers users an unprecedented volumetric video experience, including
smooth scrolling and instant sharing. Our project page with source code is
available at https://authoritywang.github.io/v3/.