Splatter a Video: Video Gaussian Representation  for Versatile Processing
Authors: Yang-Tian Sun
Yi-Hua Huang
Lin Ma
Xiaoyang Lyu
Yan-Pei Cao
XIAOJUAN QI
Venue: NeurIPS 2024 poster (2024)
Keywords: Video Representation; Video Processing
Primary area: machine_vision
TL;DR: A method to represent casual videos using 3D Gaussians without estimating camera pose

Abstract: Video representation is a long-standing problem that is crucial for various downstream tasks, such as tracking, depth prediction, segmentation, view synthesis, and editing. However, current methods either struggle to model complex motions due to the absence of 3D structure or rely on implicit 3D representations that are ill-suited for manipulation tasks. To address these challenges, we introduce a novel explicit 3D representation—video Gaussian representation—that embeds a video into 3D Gaussians. 
Our proposed representation models video appearance in a 3D canonical space using explicit Gaussians as proxies and associates each Gaussian with 3D motions for video motion. This approach offers a more intrinsic and explicit representation than layered atlas or volumetric pixel matrices. To obtain such a representation, we distill 2D priors, such as optical flow and depth, from foundation models to regularize learning in this ill-posed setting.
Extensive applications demonstrate the versatility of our new video representation. It has been proven effective in numerous video processing tasks, including tracking, consistent video depth and feature refinement, motion and appearance editing, and stereoscopic video generation.

Splatter a Video: Video Gaussian Representation for Versatile Processing
Yang-Tian Sun
Yi-Hua Huang!*
Lin Ma
Xiaoyang Lyu
Yan-Pei Cao² Xiaojuan Qi
'The University of Hong Kong
VAST
Abstract
Video representation is long-standing problem that is crucial for various down- stream tasks, such as tracking, depth prediction, segmentation, view synthesis, and editing. However, current methods either struggle to model complex motions due to the absence of 3D structure or rely on implicit 3D representations that are ill-suited for manipulation tasks. To address these challenges, we introduce a embeds a
novel explicit 3D representation- Gaussian representation- video into 3D Gaussians. Our proposed representation models video appearance in a 3D canonical space using explicit Gaussians as proxies and associates each Gaussian with 3D motions for video motion. This approach offers more intrinsic and explicit representation than layered atlas or volumetric pixel matrices. To obtain such representation, from foundation models to regularize learning in this ill-posed setting. Extensive applications demonstrate the versatility of our video representation. It has been proven effective in numerous video processing tasks, including tracking, consistent video depth and feature refinement, motion and appearance editing, and stereoscopic video generation.
distill 2D priors, such optical flow and depth,
Introduction
Video processing, which encompasses a variety of tasks such video editing, enable numerous applications in fields like social media, filmmaking, and advertising [0] A video can be viewed as collection of spatiotemporal pixels. However, processing video directly in its pixel space, while maintaining temporal consistency, poses challenges due to the inherent complexities associated with appearance, motion, occlusions, and noise in the video data [14 27). Consequently, robust. video representation capable of abstracting and disentangling appearance and motion is crucial for facilitating various applications and overcoming these challenges.
Existing research on video representation for processing has primarily focused on 2D/2.5D techniques, employing methods such optical flow and tracking to associate pixels across frames [47 14] [88
These approaches often involve learning a canonical image [2] B3 30 or a layered atlas with persistent motion patterns [14] 24 9 to facilitate editing and then optical flow tracks to propagate edits throughout a video. The most recent work [33 utilizes hash grids combined with implicit functions to embed a video into learned canonical image for appearance and deformation field for motion. Despite achieving promising results in appearance editing tasks, these methods struggle handle occlusions of objects (see Fig. 3), leading erroneous propagation. Although layered 2.5D representation [14 24 4 9 can mitigate this issue, they still face challenges with
complex self-occlusions within layer. Moreover, these techniques have limited or no capability in
*Equal Contribution :Corresponding Author Project page: https::/ htttps://aumyangtiian github. wttpsptttteee../// w_vieee___vvvieeeeeeeeeee_eeeeee web/
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

addressing processing tasks that require 3D information, such as video representation with complex occlusions, consistent depth prediction, and stereoscopic video generation.
Video
Tracking
Consistent Depih/Feuture
Video Editing
Steresseopic Sputheris
Interpolation
Video Gaussian Representation
Figure 1: We propose approach to convert video into Video Gaussian Representation (VGR), which can be used for versatile video processing tasks conveniently.
Drawing inspiration from the fact that video is essentially projection of the dynamic 3D world onto the 2D image plane at different moments, we pose the question: is it possible to represent a video in its intrinsic 3D form? By doing so, could potentially bypass the limitations of 2D representations, such as occlusions, reduce the complexity of motion modeling, and support processing tasks that require 3D information. Recent work [48] has explored 3D representations, which employ implicit radiance field to model a canonical 3D space and leverage a bi-directional mapping network for associating 2D pixels with 3D representations. While this approach demonstrates promising performance in dense tracking, it falls short in faithfully representing video appearance, making it incapable of performing video processing tasks that require generating new videos, such as video editing. Moreover, its implicit nature limits its applicability to variety of video processing tasks that require explicit content or motion manipulations, such the removal or addition of objects and adjustments to the motion patterns of objects.
In this paper, introduce novel explicit video Gaussian representation (VGR) based 3D Gaussians [16]. Our core idea revolves around utilizing Gaussians in a canonical 3D space to model video appearance while associating each Gaussian with time-dependent 3D motion attributes to control its locations at different time steps for video motion. This 3D representation can then be employed process and render videos effectively. The subsequent challenge lies how map a video onto such 3D Gaussian representation. This is inherently difficult due to the loss of essential 3D information during 3D-to-2D projection, well as the entanglement of motion and appearance in videos. However, recent advancements in large models have facilitated the acquisition of high-quality monocular priors from images and videos, such as optical flow 44 I1] and monocular depth [54, 15 [33.. While these 2D priors may not be perfect, they can serve as regularization for learning through knowledge distillation. Consequently, we propose leveraging these priors in conjunction with our 3D motion regularization for learning. By doing so, we effectively lift 2D information- such as pixels, depth, and optical flow-into a unified and compact representation.
Upon learning, our video Gaussian representation can be used to support versatile video processing tasks, as shown in Fig. [ Here, we showcase efficacy in video-processing tasks: Specifically, can be used to obtain 1) dense tracking and 2) improve the consistency of monocular 2D prior across frames, leading to better video depth and feature consistency. Secondly, our representation facilitates range of video editing tasks. including 3) geometry editing and 4) appearance editing. Thirdly, it also proves useful in video interpolation, allowing for 5) the generation of smooth transitions between frames. Finally, our representation is inherently 3D, it opens up additional possibilities, such as 6) novel view synthesis (to a certain extent) and the creation of stereoscopic videos.
Related Work
As our method utilizes dynamic 3D Gaussians represent videos and supports versatile video processing, this section introduces related works on video editing, tracking, and dynamic Gaussian splatting. We brietly cover the most relevant works. For additional references, A.7

---

3D Representation
Editing Console
To
Ti
Tz
Video Gaussian
Interpoazion
Camera Space
Dynamic Video Gaussians
2D) Supervisions
Rasterizer
Flow
Mask
Video
Frame 0
Frame
Frame
RGB
Depih
Figure 2: Pipeline of approach. Given a video, we represent its intricate 3D content using video Gaussians in the camera coordinate space. By associating them with motion parameters, we enable video Gaussians capture the video dynamics. These video Gaussians are supervised by RGB image frames and 2D priors such as optical flow, depth, and label masks. This representation makes convenient for users to perform various editing tasks on video.
Video Editing Decomposing videos into layered representations facilitates advanced video editing
techniques. Kasten al. [4] introduced layered neural atlases, enabling efficient video propagation and editing. Further advancements include deformable sprites [188), di-dircctional warping fields [9]
and innovations in rendering lighting and color details 4 CoDeF [33 and GenDeF [49] focus on
multi-resolution hash grids and shallow MI.ss for frame-by-frame deformations. Latent diffusion
models 38 and methodologies like ControlVideo [6]] MaskINT
28 and VidToMe 20 have also
been employed for data-driven video editing.
Video Tracking Video tracking captures physical motion within video sequences. PIPs 8 and TAPIR [6] offer foundational approaches, while CoTracker [3] uses a sliding- window transformer for tracking. OminiMotion 48 and MFT (30 employ neural radiance fields and optical flow fields for dense tracking. State-of-the-art methods like RAFT 44 and FlowFormer flow estimations but struggle with long-term correspondences.
11 provide accurate
Dynamic Gaussian Splatting Gaussian Splatting [66 enhances rendering in radiance fields and has been extended to dynamic scenes 57. Methods like SC-GS [1O] and 3DGStream 44 offer novel approaches for scene dynamics. Our method targets monocular video representation. eliminating the need for camera pose estimations and facilitating robust long-term tracking and editing dynamic scenes.
3 3D Gaussian Splatting
Gaussian splatting 16 models 3D scenes using Gaussians learned from multiview images. Each Gaus- sian, G, is defined by a center μ and covariance matrix Σ: G(x)) exp (- - - μ)). Here, decomposed into RSST R" for optimization, with R as a rotation matrix parameterized by quaternion g and S scaling matrix parameterized by vector Each Gaussian also has an opacity a and spherical harmonic (SH) coefficients sh. Then 3D Gaussians can formulated as: (G3: H4j qj. 8j, Orj, shy} Rendering done via:
C(u) = Σ Tia,SH(shh, (a 1 az),
IEN
where is calculated by projecting Gaussian G; at the rendering pixel and v is the direction from view point to the Gaussian. Optimizing parameters {G, f4y qi Qy, shy) and adjusting densities allows for high-quality, real-time image synthesis. For a more detailed introduction to Gaussian Splatting, please refer to Sec. A.8 We extend 3D Gaussians to represent a video by adding attributes to Gaussians for versatile processing.

---

4 Method
Given video, goal is to use 3D Gaussians in canonical space to represent its appearance and associate Gaussians with 3D motions for video dynamics. To facilitate this mapping, we incorporate 2D priors extracted from existing 2D models and apply 3D motion regularization. This representation allows us to efficiently perform various downstream applications. The pipeline of our method is depicted in Fig. 2 In the following, we elaborate the video Gaussian representation in 4.1 Then, we discuss the learning objectives and optimization details in Sec 4.2 and Sec. respectively.
4.1 Video Gaussian Representation
Camera Coordinate Space Instead of utilizing an absolute 3D world coordinate system, we opt for the orthographic camera coordinate system to model a video's s 3D structure, as demonstrated in Omnimotion [48]. In this space, the video's width, height, and depth correspond to the X, Y, and Z axes, respectively. This enables us to circumvent the challenges associated with estimating camera poses or disentangling camera motion from scene dynamics, which can be not only time-consuming but also prone to failure in casually captured monocular videos with dynamic objects [34 59]. By modeling the scene as dynamic 3D Gaussians in the camera coordinate space, we intertwine camera motion with object motion and treat them as the same type of motion, eliminating the need for camera calibration. During the rendering process, the 3D Gaussians in the camera coordinate space are rasterized into images from an identity pose camera. This approach simplifies the representation of dynamics and avoids the challenges of estimating camera pose from monocular casual videos.
40 41
Video Gaussians Given a video V (I1, I2,..., In)) consisting of n. frames, our video Gaus- sian representation transforms it into a set of dynamic 3D Gaussians, parameterized as G {G1, G2...., Gm.- to simultaneously represent the appearance and motion dynamics of the video. Each Gaussian is characterized by its position 14, rotation quaternion q. scale spherical harmonics (SH) coefficients of appearance sh, and opacity a. In addition to these fundamental Gaussian proper- ties for appearance, dynamic attributes p. segmentation labels m, and image features f from any 2D base models (e.g.. DINOv2 [3 and SAM (17)) can also be associated with 3D Gaussians to depict the video's scene content. Consequently, a Gaussian can be expressed as G (μ. q, 8, a, sh, p, m, f). To learn these properties from a video, we enhance the differentiable 3D Gaussian renderer to render additional attributes beyond simple color, which denote as R(,, q, s, a, where represents the specific attribute to be rendered. The rendering function R. follows the same procedure as color rendering in the original Gaussian Splatting method [16].
Gaussian Dynamics When parameterizing motion, there is a trade-off between incorporating more regularization from motion priors and achieving high fitting capability [46). In line with recent popular methods [8), we employ a flexible set of hybrid bases comprising polynomials (22] and Fourier series f to model smooth 3D trajectories. Specifically, Fourier coefficients to each Gaussian, denoted as = {Fr) } U (P'm P'an) respectively. Here, n and I represent the order of coefficients. The position of a Gaussian at time can then be determined as follows:
assign learnable polynomial and
N
p(t) = po +
(pem cos((t) + Plonr sin (It)).
n==
Polynomial bases { are effective in modeling overall trends and local non-periodic variations in motion trajectories and are widely used in curve representation, such as in Bezier and B-spline curves 7 Fourier bases (cos( sin((t)} offer a frequency domain parameterization of curves, making them suitable for fitting smooth movements [[, and excel in capturing periodic motion components. The combination of these two bases leverages the strengths of both, providing comprehensive modeling, enhanced flexibility and accuracy, reduced overfitting, and robustness to noise. This equips Gaussians with the adaptability to fit various types of trajectories by adjusting the corresponding learnable coefficients. It is important to note that for each Gaussian, the associated parameters p = {pr] (//m Pooo) are learned from the video by optimizing the learning objective as described in Sec.

---

4.2 2D Monocular Priors and 3D Motion Regularization
Learning video Gaussians in the camera coordinate space to achieve consistency with real-world content using photometric loss is challenging and often ill-posed. There multiple solutions for video Gaussians to fit the observed 2D projections. For instance, relative depth orders among scene objects can be ambiguous without occlusion cues. Moreover, different Gaussians may sequentially represent the same object, and their motion may not precisely match the object's actual motion. Therefore, regularization required during the training process.
Thanks to advancements in 2D visual understanding methods, monocular 2D priors such as optical now accessible. Although not perfect, these priors can provide crucial cues to regularize learning. To stabilize our method's s training and ensure a real-world consistent solution, supervise the video Gaussians using priors from the estimated flow obtained from RAFT 44 and the estimated depth derived from Marigold [5].
flow 44 and depth estimation [54] [5 [3]
Flow Distillation Optical flow represents the 2D projection of 3D motion. Flow distillation serves to regularize the 2D projections of 3D Gaussian motions. To guarantee that the motion of video Gaussians aligns with the estimated optical flow, we project the 3D motion of Gaussians (u(t2) u(ti)) between frames & and f2 onto the 2D image plane and regularize it using the estimated optical flow:
Lrow = (t) (IR(((()),, g. a, x(u(t2)) ((((t)))) -
Here, π denotes the projection function that maps camera coordinates to image coordinates after projection, and -lo represents the optical flow estimated by RAFT [4] from ty to tz. This prior aids video Gaussians in learning the scene flow by ensuring that the 2D projection of their 3D motion on the X XY-plane is consistent with the optical flow instead of relying on relative depth changes along the Z-axis to fit frame colors.
Depth Distillation Monocular depth estimation provides the per-frame depth of a video. Although
these estimates may be inconsistent across long-range frames, they offer valuable cues for regularizing the scene geometry. As a result, we utilize depth maps estimated by Marigold (T5) to ensure a reasonable geometry for our video Gaussians. We employ the scale- and thift-trimmed loss proposed in MiDaS [37]:
ddpphh F. ((()))))(())))......).)))))>>))) , - +(D'),,(D*) = median D*), (4)
where 1 is the rendered depth of 3D Gaussians time I, and Te is the corresponding predicted depth. It is worth noting that, thanks to our 3D representation, our approach can, in turn, refine the inconsistent monocular depth estimations and yield consistent depth predictions for video.
In sum, flow distillation regularizes the projected 3D Gaussian motion on the 2D image plane, corresponding to the X-Y axes in the camera coordinate space. Meanwhile, depth distillation regularizes the relative video Gaussian positions corresponding to the Z-axis the camera coordinate space. Together, they offer comprehensive 3D supervision and complement each other, effectively regularizing the learning of 3D motion for video Gaussians.
3D Motion Regularization In addition to depth and flow distillation, we employ local rigidity regularization to prevent Gaussians from overfitting the rendering targets through non-rigid motions 10 26). This approach encourages the 3D motion of individual Gaussians to be as locally rigid as possible [42]. As a result, Gaussians form locally rigid structures, aligning with real-world dynamics. To constrain the local rigidity of Gaussian G; from time & to fz, we first identify the K nearest neighboring Gaussians G& (A: E N₁) using its 3D position at f1. Then, we apply the rigid loss to ensure that the edges between them (t,, (ee(t)) adhere rigid transformation:
Cara = 1().
Σ (1(m(tt))-ma R.((u((2)) - ( kEN:
(5)
where R is the estimated rigid rotation transformation given by
R = arg min
Σ KEM.
[|pa(tt) - (u R(ua(t)) (taa)11
(6)
RESO(3)
5

---

4.3 Optimization
In addition 2D priors and 3D regularization for learning 3D motion and geometry, we also incorporate a color rendering loss for appearance learning. Furthermore, we introduce an optional mask loss to facilitate the separation of background and foreground, which particularly useful for editing applications.
Color Rendering Loss Video Gaussian representation also learns fit the color of video frames {Ty at as in novel view synthesis methods [16] 29 with the rendering loss:
Crender = E, (II SH(sh, v) - -ill))-
Mask Loss Segmentation labels serve as crucial attribute for pixels, enabling the identification of groups of pixels belonging to foreground objects. our experiments, we separate pixels into foreground and background components by segmenting each frame and extracting the foreground mask M This mask is subsequently lifted to Gaussian, where associated with a learnable label attribute E 1). The label attributes of Gaussians are supervised by the image segmentation results:
Label E, (I) s, at, m) M
(8)
With the segmentation label, we can divide Gaussians into different parts and constrain their motion respectively, shown in Eq. 5 Our approach can also manipulate (remove/duplicate) and edit specific objects in a video, as shown in Fig 7
Total Learning Objective The total learning objective is the weighted sum of all the losses:
C = Avender Frender + Adteph + Cloww + Xurap Carap + Clabel-
(9)
Adaptive Density Control We initialize the video Gaussians by uniformly sampling points in the camera coordinate space of the first frame, and apply similar density control strategy as in vanilla Gaussian Splatting [6]. For more details, please refer Sec. A.
5
Video Processing Applications
With our video Gaussian representation, we can perform various video processing tasks, including 1) dense tracking, 2) consistent depth/feature prediction, 3) geometry editing, 4) appearance editing, 5) frame interpolation, 6) novel view synthesis. and 7) stereoscopic video creation. In this section, we detail these applications, highlighting the versatility of video Gaussians.
Dense Tracking Since the scene motion is captured by the dynamics of video Gaussians, we can project these dynamics onto the image plane as UV flow and rasterize the attributes flow maps. This method handles both short and long-frame gaps effectively. The pixel flow map dU, from to to calculated as:
dU2,- R(u(tt), (u((2)) - ((((t))))
(10)
The rendered dense flow map provides pixel correspondences, facilitating tracking across frames.
Consistent Depth/Feature Prediction Video Gaussians, supervised by monocular depth priors for each frame, conform to a reasonable geometry layout, providing consistent depth predictions across frames. Similarly, other image features can be distilled into video Gaussians; unifying per-frame features into consistent 3D form. To distill image features (c.g.. SAM [17] or DINOv2 [31)), we associate each video Gaussian with a feature attribute ff and rasterize them to match the feature map (Frt) from 2D models:
Lieature E, ()),,((()),,,,,))))),,.,),,,,,,,
(11)
Optimizing video Gaussians with Lieature unifies frame-wise 2D features a 3D form, enabling the rendering of view-consistent feature maps {F():
F. = R(n(t),()))))))))))(()) s, f).
(12)
Consistent feature prediction is crucial for applications like video segmentation and re-identification.

---

CoDeF
Omnimotion
Ours
GT
Figure 3: Qualitative comparison video reconstruction using our method and SOTA methods.
Figure 4: Dense tracking results on diverse complex motion patterns.
Geometry Editing the unified 3D space, geometry editing is straightforward. By distilling select Gaussians of the target identity and transform their positions μa quaternions q, and scales for translation, resizing, and rotation. Adjusting their opacities changes the transparency of the edited objects. It also facilitates easy object removal within a video and supports object copying both between and within videos.
segmentation labels into video Gaussians,
Appearance Editing Appearance editing with video Gaussians can also be easily achieved. Users can select a specific frame and perform painting, recoloring, or stylization. We all attributes except the SH coefficients representing Gaussian appearance and optimize them to fit the edited image T using:
Ledit R(((t)), s, SH(sh, v)) - 2
(13)
The edited results can propagate throughout the video, maintaining temporal consistency.
Frame Interpolation The learned smooth trajectories of video Gaussians enable interpolation of scene dynamics at any up-sampling rate. Interpolated Gaussians' dynamic attributes can render interpolated video frames. By re-mapping the timestep values (1) {t') with an arbitrary continuous function, we can freely adjust the video playback speed.
Novel View Synthesis Applying global rigid transformation T SE(3) to video Gaus- sians allows for camera position adjustments. The rendering results transformed Gaussians R(T(u(t)), T(g), 8, a, SH(sh,, v)) provide synthesized views from different perspectives.
Stereoscopic Video Creation Similar to the novel view synthesis application, we can achieve stereo- scopie frames by slightly translating video Gaussians horizontally by fixed distance, representing interocular distance. This application is crucial filmmaking and gaming.
Experiments
Evaluation We conducted experiments on the DAVIS dataset 336
as well as some videos used by
Omnimotion (48) and CoDeF [33]. Our approach is evaluated based on two criteria: 1) reconstructed video quality and 2) downstream video processing tasks. In addition to general video representation methods Deformable Sprites (58), Omnimotion 48 and CoDeF (33), we also compare with dynamic

---

Table Comparison with existing methods on Tap- Vid benchmark (DAVIS).
Methods
PSNR1
SSIM
LPIPS|
AJ
85ng 1
OAt
TCJ
Training Time
GPU Memory
FPSt
4DGS
511
18.12
0.5735
0.5130
10.2
75.45
mins
145.8
RoDynRF
23
24.79
0.723
0.394
>24 hours
24C
>Imin
Deformable Sprites
58
22.83
0.6983
0.3014
20.6
32.9
69.7
2.07
-30 mins
24G
1.6
Omnimotion
24.11
0.7145
0.3713
$1.7
67.5
85.3
>24 hours
24G
>Imin
CoDeF
33
26.17
0.8160
0.2905
7.6
13.7
78.0
7.56
-30 mins
10G
8.8
Ours
28.63
0.8373
0.2283
41.9
57.7
79.2
1.82
-30 mins
10G
149
Depth Marigold 15
Depth Ours
Feature SAM
Feature Ours
Figure 5: Qualitative comparison of video depth and features generated by our method and SOTA single-frame estimation methods. Our method yields more consistent estimations.
NeRF/3DGS methods, namely 4DGS [5]] and RoDynRF 23
Note that for 4DGS, we estimate
camera poses using monocular depth estimation method Unidepth 355 and DROID-SLAM [45]. Despite these efforts, the performance remains unsatisfactory. further highlighting the challenges of accurate camera pose estimation in causal dynamic videos. contrast, our approach demonstrates the capability to handle more complex motions and achieves significantly higher reconstruction quality. For downstream tasks, our method also shows comparable performance to those specifically designed for these tasks.
Video Reconstruction To demonstrate our method's ' fitting ability for casual videos, we compare it with Omnimotion [88] and CoDeF [33). Omnimotion tends to render blurred results due to the smooth bias of the MLP when modeling the canonical space, while CoDeF struggles with complex motions due to the limited representation ability of the 2D canonical image. We report the rendering quality metries and visualizations on the Tap- Vid DAVIS dataset [36] in Table Figure 3 More comparison with RyDynRF (23) and 4DGS [51] are provided in the supplementary materials.
6.1 Video Processing Applications
Dense Tracking, Our approach enables dense tracking by projecting the dynamics of Gaussians onto 2D image planes obtain correspondences. Tracking results are visualized in Fig. and evaluated in Table [ Despite Omnimotion's 's specialization in tracking, our approach supports a wider array of video processing tasks with higher computational and training efficiencies. achieves comparable results with better reconstruction quality using fewer resources. Tracking performance comparisons with similar-cost methods (CoDeF 4DGS) are shown in the supplementary materials 11 highlighting our superior outcomes.
Consistent Depth/ Feature Generation. We present the results of consistent video depth and features (using SAM [77) in Fig. 5 compared to per-frame prediction. Due to the unified 3D representation of video frames, the predicted depth and features exhibit significantly better consistency than those obtained from monocular predictions. We also evaluate the effectiveness of consistent SAM feature in the segmentation task in the supplementary materials 15 We recommend that readers watch the supplemental videos for better illustrations.
Geometry Editing By manipulating the Gaussians associated with specific labels, geometric editing of target identities, as demonstrated in Fig. 7 By deleting foreground Gaussians, remove foreground elements and render a clean background. Our approach also supports geometric edits such as duplicating, resizing, and translating. Additionally, the motion of these elements can be adjusted by setting different motion attributes.
achieve
Appearance Editing Users can edit the appearance specific frame by drawing, stylizing, or recoloring, and these edits will be propagated across the entire video with foss-frame consistency. In

---

Frames
Sketch
Cartoon
Figure 6: Appearance editing results using the 2D prompt editing method [60]
Fig 6 we demonstrate appearance editing using ControlNet 60 Appearance editing is user-friendly in our representation, as it only requires single-frame editing.
Frame
Figure Geometry editing results including object deleting, resizing, copying, and translating.
Novel View Synthesis & Stereoscopic Video Creation Benefitting from depth regularization, the 3D Gaussians maintain meaningful 3D structure, even from a monocular video. This facilitates novel view synthesis tasks, with examples provided in the supplemental video. Stereoscopic videos can also be produced, shown in Fig.
6.2 Ablation Study.
We perform ablation studies to validate the importance of the proposed modules, including camera model (perspective/orthograpiee))) flow loss, depth loss (1.2/scale-shift invariant). reported in Table 2
The results are
Table 2: Ablation of each module in our framework.
Methods
Ours
Perspective Camera w/o Flow Loss
w/o Depth Loss
L2 Depth Loss
PSNR 1
29.61
22.51
25.16
29. 18
28.15
SSIM
0.8624
0.6908
0.6937
0.8475
0.8214
I.PPS
0.1845
0.3958
0.4724
0.2449
0.3328
Using a predefined pinhole camera intrinsic led to unstable optimization, resulting in artifacts in both geometry and appearance. This instability likely stems from the rasterization process, where the denominators of Gaussians' screen coordinates UV include depth, complicating the gradients. Replacing the shift- and scale-invariant depth loss with absolute 1.2 loss degrades performance, as monocular depth cues ambiguous regarding scale and shift.

---

Moreover, the depth prior is crucial for maintaining the 3D structure of Gaussians. Without the depth prior, Gaussians collapse into a flat 2D plane, hindering novel view synthesis and resembling 2D>layerr methods.
We also demonstrate the significance of motion regularization (rigid loss) and the selection of motion coefficients and I, which effectively suppress unorganized Gaussian motion. Please refer to Sec. A.3 for more details.
Figure 8: Stereo view synthesis. One original frame is visualized the first column for comparison.
6.3 Limitations
Although achieving satisfying performance, there are still some limitations to be enhanced. First, our approach suffers from significant changes in the scene, since large deformation is hard optimize. Initializing the scene with dynamic point clouds might alleviate this problem. In addition, our approach still relies on existing correspondence estimation methods (e.g., RAFT), which might fail when processing rapid and highly non-rigid motion. Extending this representation to more general scenarios is still worth exploring. We have illustrated two scenarios in Fig
GT
Ours
Our 3D Scene
GT
Ours
Our 3D Scene
Figure 9: Our method may underperform in two scenarios. (1) Without camera estimation, it has to learn large camera rotations as scene motions, causing distant background blur (left). (2) It fails to track mast-n transients, as photometric loss insufficient for motion fitting (right).
7 Conclusion
In this paper, we introduced novel explicit video Gaussian representation (VGR) based 3D Gaussians to address the challenges of video processing. By modeling video appearance a canonical 3D space and associating each Gaussian with time-dependent 3D motion attributes, our approach effectively handles complex motions and occlusions. Leveraging recent advancements monocular priors, such optical flow and depth, lift 2D information into compact 3D representation. facilitating wide range of video-processing tasks. Our VGR method demonstrates efficacy in dense tracking, improving monocular 2D priors, video editing, interpolation, novel view synthesis, and stereoscopic video creation, providing robust and versatile framework for sophisticated video processing applications.
Acknowledgement
This work has been supported by Hong Kong Research Grant Council Early Career Scheme (Grant No. 27209621), General Research Fund Scheme (Grant No. 17202422), and RGC Matching Fund Scheme (RMGS). Part of the described research work is conducted in the JC STEM Lab of Robotics for Soft Materials funded by The Hong Kong Jockey Club Charities Trust. We would like to thank Ziyi Yang for the insightful discussion and generous help.
10

---

References
[1] Ijaz Akhter. Yaser Sheikh, Sohaib Khan, and Takeo Kanade. Trajectory space: A dual representation for nonrigid structure from motion. IEEE Trunsactions on Pattern Analysis and Machine Intelligence. 33(7):1442-1456, 2010.
[2] Alan C Bovik. Handbook image and video processing. Academic press, 2010.
[3] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. 2023.
[4] Cheng-Hung Chan, Cheng- Yang Yuan, Cheng Sun, and Hwann-Tzong Tzong Chen. Hashing neural video decomposition with multiplicative residuals in space-time. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7743 7753,773, 2023.
[5] Carl Doersch, Ankush Gupta, Larisa Markeeva, Adria Recasens, Lucas Smaira, Yusuf Aytar, Joao Carreira, Andrew Zisserman, and Yi Yang. Tap-vid: A benehmark for tracking any point in video, Advances Neural Information Processing Systems, 35:1 15:13310-111 13626, 2022.
[6] Carl Doersch, Yang. Mel Vecerik, Dilara Gokay, Ankush Gupta, Yusuf Aytar, Joao Carreira, and Andrew Zisserman, Tapir: Tracking any point with per-frame initialization and temporal refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10061-10072. 2023.
[7] William Gordon and Richard F Riesenfeld. B-spline curves and surfaces. In Computer aided geometric design, pages 92- Elsevier, 1974.
[8] Adam W Harley, Zhaoyuan Fang, and Katerina Fragkiadaki. Particle video revisited: Tracking through occlusions using point trajectories. European Conference on Computer Vision, pages 59-75. Springer, 2022.
[9] Jiahui Huang, Leonid Sigal, Kwang Moo Yi, Oliver Wang, and Joon- YoungY Lee. Inve: Interactive neural vidco editing. arXiv preprint 23Xi::3307.066633 2023.
[10] Yi-Hua Huang, Yang- Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse- controlled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312. 14937, 2023.
[11] Zhaoyang Huang, Xiaoyu Shi, Chao Zhang, Qiang Wang, Ka Chun Cheung, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Flowformer: A transformer architecture for optical flow. In European conference on computer vision, pages 668-685. Springer, 2022.
[12] Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk. Advances in neural information processing systems, 33:19545-1999 19560, 2020.
[13] Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635 2023.
[14] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases for consistent video editing. ACM Transactions Graphics (TOG), 40(6): 12, 2021.
[15] Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- based image generators for monocular depth estimation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[16] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4):1-14, 2023.
[17] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023.
[18] Agelos Kratimenos, Jiahui Lei, and Kostas Daniilidis. Dynmf: Neural motion factorization for real-time dynamic view synthesis with gaussian splatting. arXiv preprint arXiv:2112.00112. 2023.
[19] Maomao Li, Yu Li, Tianyu Yang, Yunfei Liu, Dongxu Yue, Zhihui Lin, and Dong Xu. video is worth 256 bases: Spatial-temporal pnpectation-mxximizaan inversion for zero-shot video editing. arXiv preprint arXiv:2312. 0550566, 2023.
[20] Xirui Li, Chao Ma. Xiaokang Yang, and Ming- Hsuan Yang. Vidtome: Video token merging for zero-shot video editing, arXiv preprint arXiv:2312.10656 2023.

---

[21] Zhengqi Li. Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4273-4284, 2023.
[22] Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. arXiv:23/2.05431, 2023.
[23] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang. Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[24] Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, David Salesin, William T Free- man, and Michael Rubinstein. Layered neural rendering for retiming people video. arXiv preprint vrXiv:2009.07833 2020.
[25] Tao Lu, Yu Mulin, Xu Linning, Xiangli Yuanbo, Wang Limin, Lin Dahua, and Dai. Bo Scaffold- gs: Structured 3d gaussians for view-adaptive rendering. Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[26] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.
[27] Xuan Luo, Jia- Bin Huang, Richard Szeliski, Kevin Matzen, and Johannes Kopt. Consistent video depth estimation. ACM Transactions on Graphics (ToG), 39(4):71-1, 2020.
[28] Haoyu Ma, Shahin Mahdizadehaghdam, Bichen Wu, Zhipeng Fan, Yuchao Gu, Wenliang Zhao, Lior Shapira, and Xiaohui Xie. Maskint: Video editing via interpolative non-autoregressive masked transformers. arXiv preprint 23..:231...2468, 2023.
[29] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[30] Michal Ncoral, Jonáš Serych, and Jiří Matas. Mft: Long-term tracking of every pixel. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 6837-6847, 2024.
[31] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V Vo, Mare Szafranicc, Vasil Khalidov, Pierre Fernandez. Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions Machine Learning Research, 2023.
[32] Halil Oruę and George M Phillips. b-bersstein polynomials and bézier curves. Journal of Computational and Applied Mathematics, 151 2003.
[33] Hao Ouyang, Qiuyu Wang, Yuxi Xiao, Qingyan Bai, Juntao Zhang, Kecheng Zheng, Xiaowei Zhou, Qifeng Chen, and Yujun Shen. Codef: Content deformation fields for temporally consistent video processing. arXiv preprint arxiv:2308. 07926, 2023.
[34] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ri- cardo Martin- Brualla, and Steven M Seitz. Hypernerf: a chigher-dimenssinnal representation for topologically varying neural radiance fields. ACM Trunsactions on Graphics (TOG), 40(6):1-12) 2021.
[35] Luigi Piccinelli, Hss Yang, Christos Sakaridis, Mattia Segu. Siyuan Li, Luc Van Gool, and Fisher Yu. UniDepth: Universal monocular metric depth estimation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[36] Jordi Pont-Tuses Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- Sorkine-Hornung, and Lue Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675, 2017.
[37] René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-datasel transfer. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44(3), 2022.
[38] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[39] Peter Sand and Seth Teller. Particle video: Long-range range motion estimation using point trajectories. Interna- tional journal of computer vision, 80:72 91,229912222 2008.
12

---

[40] Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from fron-mooe-roonnmmotion revisited. In Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
[41] Johannes Lutz Schönberger. Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In European Conference Computer Vision (ECCV), 2016.
[42] Olga Sorkine and Mare Alexa. As-rigid-as-posibble surface modeling. In Symposiun on Geometry processing, volume 4, pages 109- 116. Citeseer, 2007.
[43] Jiakai Sun, Han Jiao, Guangyuan Li. Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos. arXiv preprint arXiv:2403.01444 2024.
[44] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part 16, pages 402-419. Springer, 2020.
[45] Zachary Teed and Jia Deng. Droid-slamt Deep visual slam for monocular, stereo, and rgb-d cameras. In Neural Information Processing Systems, 2021.
[46] Chaoyang Wang, Ben Bekart, Simon Lucey, and Orazio Gallo, Neural trajectory fields for dynamic novel view synthesis, arXiv preprint arXiv:2/05.05999, 2021.
[47] John YA Wang and Edward H Adelson. Representing moving images with layers. IEEE transactions on image processing, (5)x625-633, 1994.
[48] Qianqian Wang, Yen-Yu Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all once. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19795-19806, 2023.
[49] Wen Wang, Kecheng Zheng, Qiuyu Wang. Hao Chen, Zifan Shi, Ceyuan Yang. Yujun Shen, and Chunhua Shen. Gendef: Learning generative deformation field for video generation. arXiv preprint
arXiv:2312.04561, 2023.
[50] Yao Wang. Jöm Ostermann, and Ya-Qu Zhang. Video processing and communicctiinns volume 1. Prentice hall Upper Saddle River, NJ, 2002.
[51] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.
[52] Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. arXiv preprint arXiv:2404.04319 2024.
[53] Guangkai Xu, Yongtao Ge. Mingyu Liu, Chengxiang Fan. Kangyang Xie, Zhiyue Zhao, Hao Chen, and Chunhua Shen. Diffusion models trained with large data are transferable visual models, arXiv preprint arXiv:2403006990, 2024.
[54] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint arXiv:2401.10891, 2024.
[55] Zeyu Yang, Hongye Yang. Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorcalistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv 23 10642, 2023.
[56] Ziyi Yang, Xinyu Gao, Yangtian Sun, Yihua Huang, Xiaoyang Lyu, Wen Zhou, Shaohui Jiao, Xiaojuan Qi, and Xiangang Jin. Spec-gaussian: Anisotropic view-dependent appearance for 3d gaussian splatting. arXiv preprint 24v: 15870, 2024.
[57] Ziyi Yang, Xinyu Gao. Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint 1xXx::2309.13001 2023.
[58] Vickie Ye, Zhengqi Li, Richard Tucker, Angjoo Kanazawa, and Noah Snavely. Deformable sprites for unsupervised video decomposition. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 2657-2666, 2022.
[59] Chao Yu, Zuxin Liu, Xin-Jun Liu, Fugui Xie, Yi Yang. Qi Wei, and Qiao Fei. Ds-slam: A semantic visual slam towards dynamic environments. In 2018 JEEE/RSJ international conference on intelligent robots and systems (IROS), pages 1168-1174 IEEE, 2018
13

---

[60] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-lo-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[61] Yabo Zhang. Yuxiang Wei, Dongsheng Jiang. Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Controlvideo: Training-free free controllable text-to-video generation. arXiv preprint arXiv:2305.13077, 2023.
[62] Youyuan Zhang. Xuan Ju, and James J Clark. Fastvideoedit: Leveraging consistency models for efficient text-to-vides -ideoo editing. arXiv preprint arXiv:2403.0620 2024.
14

---

A Appendix Supplemental Material
A.1 Implementation Details
Typically, we use video clip of about 50-100 frames and train the system iteratively for 20,000 steps. The training duration approximately 15-20 minutes on an NVIDIA 3090 GPU. The Gaussians are initialized as 10,0000 points randomly sampled in a [-1,1] orthographic camera for rendering for simplicity, which is fixed at the origin. We also modify the rasterization pipeline of 3DGS to support the orthographic projection by replacing the J in EWA
[-1,1] [0. box. We use an
projection with W/2
where W and H are the resolution of the image. For each
0
H/2
attribute attached to Gaussians, set different learning parameters and annealing strategies, list in T Note that the dynamics of Gaussians' rotation is also modelled in the same way as position.
During training, the number of video Gaussians is adaptively adjusted in vanilla Gaussian Splat- ting (16]. Every 100 steps, Gaussians with an accumulated gradient scale of positions above a threshold will be densified. Based on their projected size, they will be either split cloned. Con- currently. Gaussians with opacities below a threshold will be pruned. To avoid floaters, the opacity of Gaussians is reset to 0.01 every 3000 steps. After optimization, there around 10* 10th 3D Gaussian for a video containing 102 10 pixels (resolution x frame number).
The loss weights for render, depth, flow, motion regularization, and label are set to Xrender 5.0,
Aeepth 1.0, flow 2.0, Aerap = 0.1, and Aaaet = 1.0.
Table 3: Gaussian attributes table
Attributes
Position
Rotation
Scaling Se-3
SHo SH1,233
Polynomial
Fourier
Seg Label
SAM Feature
6e-5
le-3
2.5e-3 1.25e-4
1e-3
le-3
le-3
le-3
Annealed Ir
1.6e-6
/
1
le-5
le-5
/
/
A.2
More Visual Comparison
We have visualized the comparison with 4DGS [5T and RoDynRF 23 in Fig TO
We also visualized the tracking comparison in Fig 11 Note that due to the inaccurate camera pose for the wild scene, the performance of 4DGS is very limited.
4DGS 51
RoDynRE 23
Ours
Figure 10: Video reconstruction compared with RoDynRF higher PSNR and better visual quality.
23 and 4DGS
51
Our method has
15

---

CoDeF 33
4DGS 51
Ours
Figure 11:
Tracking results visualization shows our method outperforms CoDeF [? and 4DGS [ ].
especially in handling large-scale view changes and scene motions.
A.3 Ablation Study
Depth Regularization. Without the depth prior, Gaussians collapse into a 2D plane. Although overfitting ability remains largely unaffected, the 3D structure lost, and novel view synthesis is no longer possible, as shown in the right part of Fig. [2] Our approach then resembles 2D layer-based methods.
Rigid Loss. Without the rigid motion constraint, undesirable floaters appear, degrading rendering quality and reducing reconstruction PSNR by 1.51 dB, as illustrated in the left part of Fig. 12
w/o rigid
w/ rigid
w/o depth
w/ depth
Depth
RGB
Figure 12: The depth prior (w/ depth) ensures video Gaussians conform to realistic layout, while rigidity regularization (w/ rigid) eliminates floaters.
We also conducted ablation studies the selection of camera models and depth loss formats as shown in Fig 13
Ours
Perspective Cam
1.2 Depth Loss
Figure 13: Ablation for camera model L2 depth loss
We also evaluated the effect of different motion representation parameters (n/l Eq 2 reported in
Table and Fig. [4
16

---

Table 4: Ablation of n/l choice.
Metric
n=8/1=0 1=0
n=0 / 1=8
n=l=4
w/o n=1=8
n=1=12
PSNR t
28.02
26.87
28.46
29.61
27.47
SSIM 1
0.8392
0.7989
0,8512
0.8624
0,8357
LPIPS L
0.3099
0.3245
0.2271
0.1845
0.2532
n= 1-8
n=8, 1-0
n=l=4
n=
n=
GT
Figure 14: Ablation study on hyperparameters n and reveals optimal quality ( 8.
A.4 Consistent SAM Feature Evaluation
By adding SAM features to Gaussian points, our method can obtain consistent SAM features. Compared with SAM extracted from per single frame, this feature has better consistency for moving objects and removes the requirements of passing through the SAM image encoder, hence obtaining more accurate video segmentation results at a faster speed, as reported in the Table 5 and Fig. 15
Table 5: Comparision of pramfeame SAM feature and ours.
Metric
Per-frame SAM
Ours
IOU t Times f
0.753 0.513
0.827
0.025
A.5 Video Interpolation
Thanks to the continuous parameterization of dynamics, our approach can interpolate video frames over time. We present the interpolation results in Fig. [66 Our method supports any video interpolation using an arbitrary continuous time re-mapping function at any frame rate.
A.6 Multi-object Editing
By adding multi-channel mask attribute to each Gaussian point, our method can achieve separate editing of multiple objects. visualize an example of multi-object geometry appearance editing in
Fig. 17
A.7
Expanded Related Work
Video Editing Decomposing videos into layered representations facilitates advanced video editing techniques. Kasten et al. [14 introduced layered neural atlases that decompose an image into textured layers and learn corresponding deformation field, thereby enabling efficient video propagation and editing. Subsequent advancements have introduced more sophisticated models. Ye et [l. developed deformable sprites, segregating videos into distinct motion groups, each driven by an MLP-based representation. Huang et al. [9] proposed employing a bi-directional warping field to support extensive video tracking and editing capabilities over longer durations. Recent innovations have also focused on enhancing the rendering of lighting and color details. Chan et al. al.[4] extended this approach by incorporating additional layers and introducing residual color maps, enhancing the representation of illumination effects within the video. The most current development in this
17

---

Ours tl
Ours t2
SAM 12
1oU 0.827
1oU 0.753
Figure 15: Segmentation comparison reveals that our lifted SAM features outperform per-frame SAM segmentation, delivering higher resolution feature maps and superior IoU.
1+0.25
1+0.5
L+0.75
[+]
Figure 16: Video interpolation results. Please refer to the supplementary video for better visualization.
area is CoDeF 33 which leverages a multi-resolution hash grid and a shallow MLP to model
frame-by-frame deformations relative a canonical image. This approach allows for editing in the canonical space, with changes effectively propagated across the entire video. GenDel 49 similar representation generate controllable videos.
uses
Several studies have exploited the generative capabilities of latent diffusion models 38 for data- driven video editing. Control [6] adopts the methodology of ControlNet [60]. integrating control signals into the network during the video reconstruction process to guide editing. Employing related technique to manage control signals, MaskINT 28 utilizes frame interpolation generate edited videos from specifically edited keyframes. In contrast, VidToMe (00 implements a token merging approach to incorporate control signals into the editing process. Additionally, certain research efforts
[19 62
have explored using inversion solutions to achieve video editing.
Video Tracking. Video tracking is essential for capturing the physical motion of each point within video sequence (39. PIPs[8] track motion within fixed-size windows and include an occlusion branch, though they lack the ability to re-detect targets following prolonged occlusions. Building on temporal processing concepts from PIPs, TAPIR (6) introduces TAP-Net(5), which precisely locates per-frame points. CoTracker [33] advances this by tracking individual query points using a Blddig-windoww transformer approach. OminiMotion (48) pioneers the use of neural radiance fields 29] to model scene flow in NDC space. Its bijection network, which represents scene flow, is optimized for photometric consistency across frames, thereby enabling dense tracking. MFT[30] employs sequential and dense point tracking methodology using optical flow fields computed across varying time spans. SpatialTracker [52) transforms each frame into a triplane and estimates trajectories by iteratively predicting movements with transformer, facilitating 2D tracking within a 3D space. While state-of-the -f-the-arttttt optical flow methods such as RAFT (44) and FlowFormer[[I] provide accurate flow estimations for consecutive frames, they struggle with maintaining long-term frame correspondences.
Gaussian Splatting Gaussian Splatting [16] has emerged as a potent method for enhancing render- ing quality and speed in radiance fields. Following that, Lu al. (5 further organize the Gaussians distribution by introducing anchor points, and Yang et al. [56] enrich its fitting ability in the specular setting with anisotropic sphere gaussian. These approaches have been extended to dynamic scenes in various recent studies. Luiten al. (26) utilize frame-by-fie fr-mmme training, making it well-suited for
18

---

Frame/Mask
tl
NeuriS
Reunts
Figure 17:
Segmentation and editing results are displayed from left to right: multi-object segmen- tation and video editing across two frames. The first row illustrates geometry editing where two individuals are copied and repositioned to lie on the ground and stand aside. The second shows the appearance editing with "2024" and "NeurlPS" painted on two fish.
multi-view scenes. Yang al. [ advance this by segmenting scenes into 3D Gaussians coupled with a deformation field, particularly for monocular scenes. Building upon this work, Wu et al. have replaced the traditional MLP with multi-resolution hex-planes 33 and shallow MLP. Additionally, Yang et (55) integrate time as an additional dimension in their 4D Gaussian model. SC-GS[TO introduces novel approach using sparse control points learn a spatially compact representation of scene dynamics. 3DGStream [4]] offers a high-quality free viewpoint video (FVV) stream of dynamic scenes generated in real-time, though necessitates multi-view video streams as input. Gaussian Faassian-FFowwweo [22] hybrid the basis of polynomial and Fourier to represent the Gaussian motion. These methods typically rely on pre-estimated camera poses. Our approach specifically targets monocular video representation, obviating the need for camera pose estimations. This facilitates more robust long-term tracking and editing capabilities dynamic scenes.
A.8 Detailed Introduction to 3D Gaussian Splatting
Gaussian splatting [6] models 3D scenes using 3D Gaussians by learning from posed multiview images. Each Gaussian, denoted G, is defined by central point μ and a covariance matrix Σ,
G(z) = exp (- - (2-)).
(14)
The covariance matrix E undergoes decomposition into RSS RD for efficient optimization. Here, R represents a rotation matrix. parameterized by a quaternion from SO(3), and S is scaling matrix defined by a positive 3D vector s. Additionally, each Gaussian is assigned an opacity value to modulate its rendering impact and is equipped with spherical harmonic (SH) coefficients sh for capturing view-dependent effects. The collection of Gaussians represented as g {G; H4j fj 8j, aj, shz}. Rendering is achieved through the equation:
C(u) Σ Ta,,S((sh,, e.), where T; 11 o,).
(15)
VEN
Here, SH denotes the spherical harmonic function and u the viewing direction. The value of a; determined by evaluating the corresponding projection of Gaussian G; at pixel as follows:
1 7 14) E 1()),
a; ax; exp(
(16)
where u4 and XX represent the projected 2D center and covariance matrix of Gaussian G., respectively. By optimizing the Gaussian parameters {Gy Jjj Qj. 8j, oz shy} and dynamically adjusting Gaus- sian densities, high-quality and real-time image synthesis is facilitated. However, vanilla Gaussian splatting can only be used represent static scene. In this paper, integrate this representa- tion with video by assigning additional attributes to each Gaussian, enabling more versatile video processing.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper's contributions and scope are reflected.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed Sec. 6.3
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Full implementation details provided in Sec. A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [No]
Justification:
Code and data are not provided for now but will released to the public.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
cesssonnoiccyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctpppp..c/pulli///uuuiii gwtp///ide///eee/ CoodeSubeiiieeee ssi www.nnonsliiieeeeeee.o. onPolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
Full implementation details are provided in Sec. A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Results in tables are averaged over 3 runs with different random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in Sec. A.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
The paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: The statement about the impacts of the work to the society is provided in
See. 11
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets cited and corresponding licenses are respected.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "bI7uoQpFKB", "reviewer": "Reviewer_rMXd", "summary": "This paper presents a \"Video Gaussian Representation\" (VGR) to explicitly model the dynamic 3D scene contained in a monocular video. The VGR employs 3D Gaussian splatting as the backend, associating each Gaussian with time-dependent motion attributes. 2D priors, such as depth, optical flow, and optionally segmentation masks, are used to regularize the VGR optimization. The resulting VGR can be used for various video processing tasks, as claimed by the authors.", "questions": "1. Please address the concerns listed in the weakness section.\n2. How does the training/optimization time and model size compare with other methods?\n3. How are the losses weighted exactly?\n4. Do all videos are optimized according to same hyper-parameters?", "limitations": "Authors have discussed some limitations, but there are still other limitations that need to be addressed:\n1. There are too many hyperparameters involved in the method. How do changes in these hyperparameters influence the results? Is the algorithm robust enough to effectively represent the video?\n2. What are the potential failure cases? Some failure illustrations should be presented to provide readers with a better understanding.", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "1. The paper is well-structured and easy to follow.\n2. The visual illustrations are straightforward and clear.\n3. The authors explore numerous downstream video processing tasks, demonstrating the proposed method’s versatility to some extent.", "weaknesses": "### 1. Lack of Novelty:\nThe primary objectives and main components of the proposed method are stemmed from existing works:\n- Modeling video representation for video processing tasks is introduced by **CoDeF [CVPR'24]**:\n  > CoDeF: Content Deformation Fields for Temporally Consistent Video Processing\n- Motion dynamics modeled by polynomials and Fourier series is presented by **Gaussian-Flow [CVPR'24]**:\n  > Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particles\n- Depth loss is adopted from **MiDaS [TPAMI'22]**:\n  > Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer\n- 3D motion regularization can be found in numerous dynamic modeling works, including Dynamic **3D Gaussians [3DV'24]** and **SpatialTracker [CVPR'24]**:\n  > Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis\n\n  > SpatialTracker: Tracking Any 2D Pixels in 3D Space\n\n### 2. Unclear Method Description:\n- The **optimization procedure is missing** in the manuscript, which is crucial for understanding the methodology. Details such as how video frames are sampled and optimized, whether they are processed per-frame, batch-wise, or globally, and the approach to densification in new frames or under-reconstructed areas within the same frame, are not provided.\n- The settings for $n$ and $l$ in the polynomials and Fourier series are **not explained**. Additionally, an ablation study regarding these parameters should be included.\n\n### 3. Poor Evaluation:\nAccording to the authors' description in Line #255:\n> \"Our approach is evaluated based on two criteria: 1) reconstructed video quality and 2) downstream video processing tasks\"\nHowever:\n1. The reconstructed video quality is **only evaluated on 8 videos** selected from the DAVIS dataset, not the entire dataset (or test set), and only based on PSNR metrics, without considering other important metrics like LPIPS.\n2. The downstream video processing tasks are only presented as visual illustrations, with **no quantitative evaluation**. For example, tasks like dense tracking should be evaluated on a well-established benchmark like **TAP-Vid [NeurIPS'22]**.\n    > TAP-Vid: A Benchmark for Tracking Any Point in a Video\n\n### 4. Unvalidated Claim:\nThe authors claim in Lines #227 and #232 that 2D features are distilled to 3D and applied to tasks such as video segmentation and re-identification. However, **no experimental or theoretical evidence** is provided to demonstrate that the **rasterized features** can be successfully used for segmentation.\n\n### 5. Insufficient Literature Review:\nThe discussion of related works in dynamic 3D scene modeling only includes methods based on 3DGS, while NeRF-based methods such as **NR-NeRF [ICCV'21]** and **RoDynRF [CVPR'23]** should also be considered:\n> RoDynRF: Robust Dynamic Radiance Fields\n\n> Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your valuable time and the acknowledgment of our writing and illustrations. We are here to respond to the listed concerns. \n\n1. **Novelty**: \n\nWe would like to emphasize that our work is the first for dynamic GS without the camera dependency.  Our novelty doesn’t lie in any specific designs, but in the exploration to effectively lift casual videos from pixel to a more intrinsic 3D representation. \n\n- **Comparisons with CoDeF**: Our novelty lies in the exploration of representing **casual** videos by lifting to **3D** space, which is fundamentally different from CoDeF’s 2D representation.  To model a video,  CoDeF uses a canonical image and a network estimating the backward flow to model a video, which limits its flexibility in representing complex videos (Fig.3 of the main paper and Table.R1) and is incapable of video processing tasks requiring 3D like novel view synthesis and stereoscopic video creation.\n- **Comparison with 4D-GS methods like Gaussian-Flow**: Our key innovation is converting casual video pixels into intrinsic 3D representations without relying on camera pose estimation, which often fails in casually captured videos and is unachievable by current dynamic GS methods (refer to Fig.R1 and Table R1). Our approach can incorporate various motion-driving methods, though we primarily use Fourier and polynomial bases for their optimal balance of complexity and fitting ability, as discussed in our paper (Lines #135-143).\n- **Depth loss & 3D Motion Regularization**: For depth loss, the purposes and effects of the referred tasks and ours differ significantly. Our primary insight is to use monocular 2D clues to regularize the learning of 3D representations from videos, rather than to enhance depth estimation as in MiDaS.  In terms of motion regularization, we have cited related papers in lines #187-189 and did not emphasize this as a contribution. In summary, these losses themselves are not the focus; rather, it is the insights gained from employing such regularizers for learning in this ill-posed task that is crucial.\n2. **Method Description**\n- **Optimization procedure missing**: Basically, our optimization procedure includes the following steps:  1) randomly sample one frame from the video sequence in each training iteration to construct the loss (Equation 9),  2)  the densification of Gaussians follows the same strategy as 3DGS, as stated in the line #212 of the main paper. We will release the code to the community for reproduction.\n- **Unexplained n/l & ablation**: We chose n=l=8 in our experiments based on the ablation of varying combinations, as shown below. Please refer to the Fig.R6 of pdf for visual comparison.\n\n|  | n = 8 / l = 0 | n = 0 / l = 8 | n = l = 4 | n = l = 8 | n = l = 12 |\n| --- | --- | --- | --- | --- | --- |\n| PSNR ↑ | 28.02 | 26.87 | 28.46 | **29.61** | 27.47 |\n| SSIM ↑ | 0.8392 | 0.7989 | 0.8512 | **0.8624** | 0.8357 |\n| LPIPS ↓ | 0.3099 | 0.3245 | 0.2271 | **0.1845**| 0.2532 |\n3. **Evaluation**\n\n- **Insufficient Evaluation & Quantitative Evaluation on Dense Tracking**: Thank you for your feedback. Additional experiments on  the TAP-Vid DAVIS benchmark (480p) have been included, with both reconstruction and tracking metrics presented in Table.R1. \n\n    Despite Omnimotion's specialization in tracking, our approach supports a wider array of video processing tasks with higher computational and training efficiencies. It achieves comparable results with better reconstruction quality using fewer resources. Tracking performance comparisons with similar-cost methods (CoDeF / 4DGS) are shown in Fig. R2, highlighting our superior outcomes.\n\n4. **Claim**\n\nUsing the rasterized SAM feature with the SAM decoder has proven effective in image segmentation, as shown in Fig.R4. Our method ensures higher object integrity and also reduces the time needed for SAM encoding. The IoU and segmentation times, tested on an NVIDIA 3090GPU, are reported below.\n\n|  | Per-frame SAM | Ours |\n| --- | --- | --- |\n| IoU ↑ | 0.753 |0.827 |\n| Time/s | 0.513 | 0.025 |\n\n5. **Literature**\n\nThank you for your kind reminder. We will recognize the pioneering contributions of the paper \"Non-Rigid Neural Radiance Fields\" and the SOTA NeRF-based method RoDynRF. Additionally, we will include comparison results with RoDynRF in our analysis.\n\n6. **Other Questions**\n\n- **Training time & Model size**: The training time is approximately 30 minutes, which is comparable to image-based methods, slightly shorter than 4DGS (about 40 minutes), and significantly less than NeRF-based methods (Omnimotion, RoDyF). We have also detailed the training times in the table above. The model size differs scene by scene, typically less than 100 MB, comparable with 4DGS and much less than CoDeF (500M).\n\n- **Loss weight**: The loss weights for render, depth, flow, motion regularization, and label are set to $\\lambda_{render}$ = 5.0, $\\lambda_{depth}$ =1.0,$ \\lambda_{flow}$ = 2.0, $\\lambda_{arap}$ =0.1, and $\\lambda_{label}$ =1.0 respectively. We also ablate the importance of each loss and module of our method in Table R2.  We will add more experimental details in the final version.\n\n- **Optimization hyperparameters**: All of our experiments are performed with the same hyperparameters.\n\n7. **Limitations**\n\n- **Hyperparameters impact**: \n    1) The impact of hyperparameters of the deformation model (n/l) has been evaluated in the Table above and Fig.R6.\n\n    2) As for the hyperparameters of learning rate in Table 2 of the paper, most of them are derived from 3DGS. The learning rate of newly added features(dynamic, mask) is referred to original Gaussian features.\n\n    3) The effect of each loss is also evaluated in Table R2 and Fig. R3\n\n    All of our experiments are performed with the same hyperparameters. \n\n- **Potential failure cases**:  We have demonstrated some failure cases in Fig. R7. Our approach fails to handle large motions, like intense background rotation or objects that suddenly appear.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "es9ZpCHbjP", "reviewer": "Reviewer_8X9e", "summary": "This paper introduces a novel explicit 3D representation for video processing using video Gaussians. This method embeds videos into 3D Gaussians to model video appearance and motion in a 3D canonical space. By leveraging 2D priors such as optical flow and depth estimation to regularize the learning of video Gaussians, the approach ensures consistency with real-world content. The paper demonstrates the efficacy of this representation in various video processing tasks, including dense tracking, consistent depth and feature prediction, geometry and appearance editing, frame interpolation, novel view synthesis, and stereoscopic video creation. This method effectively handles complex motions and occlusions, offering a robust and versatile framework for sophisticated video processing applications.", "questions": "1. How do you design the loss weight for the 2D prior? Is it case by case hyper-parameters?", "limitations": "I believe the the authors adequately addressed the limitations. I believe this paper gives a good example and standard, especially in the experimental parts for the fields of combining Gaussian Splatting with monocular videos.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Utilizing dynamic 3DGS as video represenation is well-motivated and rarely explored before.\n2. The paper is generally well-written and easy to follow.\n3. The paper represents various video processing tasks to showcase  superiority the proposed VGR, which convinces me in the experiment   part.\n4. The limitation section includes a reasonable and honest analysis of the shortcomings of the current VGR.", "weaknesses": "1. More 4DGS-based methods, e.g,,[1],[2] should be compared and discussed in experiment. Note that I am not asking the author to compare the concurrent work like Gflow [3] or Mosca [4] but at least the aboved works presented in CVPR 2024 and ICLR 2024.\n2. More abalation about the VGR representation itself but not the 2D prior should be presented in the paper. For example, the design of dynamic attributes and hybrid bases of dynamic Gaussian positions.\n\n\n\n\n\n[1] Wu, Guanjun, et al. \"4d gaussian splatting for real-time dynamic scene rendering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2] Yang, Zeyu, et al. \"Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting.\" ICLR 2024\n\n[3] Wang, Shizun, et al. \"GFlow: Recovering 4D World from Monocular Video.\" arXiv preprint arXiv:2405.18426 (2024).\n\n[4] Lei, Jiahui, et al. \"MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds.\" arXiv preprint arXiv:2405.17421 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your support in our work! We are delighted to answer the concerns raised.\n\n**Compare with 4DGS based methods:** Thank you for your suggestions. We have incorporated a comparison with 4DGS [1] on Tap-Vid DAVIS benchmarks for reconstruction and tracking tasks respectively. The results are reported in Table R1 above.  We also visualize qualitative results in Fig. R1 and Fig.R2 of the attached pdf file. Note that due to the inaccurate camera pose for the wild scene, the performance of 4DGS is very limited. \n\nRegarding [2], as the method does not employ 'deformation-based' modeling but instead uses static Gaussians in 4D space, it can not be used to establish correspondences. We acknowledge both [3] and [4] as significant concurrent works and will include them in the related work section.\n\n**More ablation:** We have performed an ablation study on the design of VGR by varying the hyperparameters (n and l). It is important to note that when n=0 or l=0, our representation simplifies to Fourier-only and polynomial-only, respectively. We report the results below and visualize the comparison in Fig. R6\n\n|  | n = 8 / l = 0 | n = 0 / l = 8 | n = l = 4 | n = l = 8 | n = l = 12 |\n| --- | --- | --- | --- | --- | --- |\n| PSNR ↑ | 28.02 | 26.87 | 28.46 | **29.61** | 27.47 |\n| SSIM ↑ | 0.8392 | 0.7989 | 0.8512 | **0.8624** | 0.8357 |\n| LPIPS ↓ | 0.3099 | 0.3245 | 0.2271 | **0.1845** | 0.2532 |\n\n Thank you for the suggestion and we will include a discussion on this topic to enhance our paper. \n\n**Loss weight design:** The loss weights for render, depth, flow, motion regularization, and label are set to $\\lambda_{render}$ = 5.0,$\\lambda_{depth}$ =1.0, $\\lambda_{flow}$ = 2.0, $\\lambda_{arap}$ =0.1, and $\\lambda_{label}$ =1.0 respectively. We have also included more ablations regarding the camera model design and each loss in Table R2 and Fig. R3 of the attached pdf file.  We will add more experimental details in the final version.  All of our experiments are performed with the same hyperparameters, which demonstrate the robustness of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZIs9f9EcFt", "reviewer": "Reviewer_axg5", "summary": "The goal in this work is to represent a video with a set of 3D Gaussian primitives, rendered with the 3DGS pipeline. The approach is optimisation-based and outputs video-specific Gaussians following a pre-defined trajectory model — hybrid of a polynomial and a Fourier series. The optimisation involves the use of data-based priors, such as monocular depth and optical flow networks. While the breadth of the experiments is laudable, it is limited to a few qualitative examples. The only quantitative results are available for video reconstruction in terms of PSNR.", "questions": "The object masks are assumed available (from SAM). Do the static points corresponding to the camera motion also follow the trajectory formulation (c.f. Eq. (2))? Are there any assumptions about the camera intrinsic parameters?", "limitations": "Sec. 6.3 outlines some limitations. I would be curious to see a discussion on scaling the approach, especially in terms of handling multi-object videos.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "* The goal is quite ambitious — representing a video with 3D Gaussians to support a variety of downstream tasks, such as dense tracking, interpolation and video editing.\n* The approach follows quite naturally from the goals and makes use of available pre-trained models to supervise motion and depth estimation.\n* The scope of (albeit qualitative) experimentation is quite compelling; the quantiative results on novel view synthesis are very encouraging.", "weaknesses": "* Representing a video with a set of 3D Gaussians is a rather straightforward extension of the original 3DGS pipeline. The only novelty is the trajectory model (Eq. 2), which was also employed in concurrent work [22].\n* The experiments are predominantly qualitative. While it is reasonable for some tasks, such as editing, there are established benchmarks for the others (e.g. point tracking [8]). The only quantitative result in Tab. 1 is not an entirely fair comparison, since prior work does not use the specific monocular depth model used here.\n* The ablation study is quite limited. Indeed, there is little technical contribution to study, except perhaps for the trajectory model.\n\n*Post-rebuttal comment:* I thank the authors for their comprehensive response. If the additional results provided in the rebuttal are integrated into the revision, it will be a nice and interesting work to share with the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable advice!  We’d like to emphasize that using 3D representations to model and process casually captured in the wild videos, different from dynamic NeRF/GS,  is a much ill-posed problem.  Here, we design a system that integrates representations and regularization: 1) using ***a fixed orthographic camera***, alleviating the cumbersome camera pose estimation. 2) distilling monocular 2D priors for regularizing learning from such ill-posed scenarios with different regularizers from mono-depth and optical flow. We have illustrated some cases in which our model perfectly handles while dynamic NeRF/GS failed in Fig.R1 of the attached pdf file.  \n\nNote that our approach doesn’t rely on the specific motion module. We adopt Fouier and polynomial bases due to their good trade-off between fitting capability and complexity.   Indeed, we have experimented with other motion representations in our exploration, which also achieved good results. We will report more results and analysis on different motion representations for this task.\n\nMoreover,  the main purpose of our work is not for 4D reconstruction, the primary objective for GS, but rather to perform versatile video processing tasks in the pseudo-3D space, which is wildly demonstrated in the main paper and Fig. R5.\n\n**Predominantly qualitative experiments:**    As our primary intention is to show the versatility of the approach on a bunch of video processing tasks which mostly focus on visual appearance, we focus on qualitative comparisons.  Here, to more thoroughly evaluate our approach quantitatively, we add more experiments on the DAVIS test videos following the TAP-Vid benchmark(480p) and report both the reconstruction and tracking metric in Table R1.  It can be seen that our approach significantly outperforms other methods in reconstruction.\n\n**Unfair comparison**: CoDeF is a 2D image-based video representation, where the depth prior cannot be used. In terms of Omnimotion, we observed that adding our relative depth loss results in lower-quality final results, as detailed in the table below. This is due to the depth maps in Omnimotion do not correspond to physical depth, as stated in their own paper.\n\nComparison with Omnimotion with Depth Prior on camel dataset.\n\n|  | Omnimotion | Omnimotion + Depth | Ours |\n| --- | --- | --- | --- |\n| PSNR↑ | 23.9850 | 23.6139 | **30.48** |\n| SSIM↑ | 0.7055 | 0.6879 | **0.9299** |\n| LPIPS ↓ | 0.36447734 | 0.37741673 | **0.0849** |\n\n**Inadequate ablation:**  We conducted additional ablation studies focusing on the selection of coefficients n and l. The results are detailed below and Fig.R6. Our contribution extends beyond trajectory representation, and we have added more ablation studies regarding the camera model selection, different types of depth loss in Table.R2 and Fig.R6.\n\n|  | n = 8 / l = 0 | n = 0 / l = 8 | n = l = 4 | n = l = 8 | n = l = 12 |\n| --- | --- | --- | --- | --- | --- |\n| PSNR ↑ | 28.02 | 26.87 | 28.46 | **29.61** | 27.47 |\n| SSIM ↑ | 0.8392 | 0.7989 | 0.8512 | **0.8624** | 0.8357 |\n| LPIPS ↓ | 0.3099 | 0.3245 | 0.2271 | **0.1845** | 0.2532 |\n\nWe also conducted ablation studies on the selection of camera models and depth loss formats as shown in Table R2 and Fig.R3. Using a predefined pinhole camera intrinsic led to unstable optimization, resulting in artifacts in both geometry and appearance. This instability likely stems from the rasterization process, where the denominators of Gaussians’ screen coordinates UV include depth, complicating the gradients. Replacing the shift- and scale-invariant depth loss with absolute L2 loss degrades performance, as monocular depth cues are ambiguous regarding scale and shift.\n\n**About camera motion and intrinsic:**  Our method employs a stationary camera, effectively merging the intrinsic motion of points with their motion relative to the camera. Consequently, all points in the scene are in motion and adhere to the trajectory formulation described in Equation 2.  As stated in lines #458-461 of the paper, our method utilizes an orthographic projection camera. Orthographic projection decouples the screen coordinates of points from their depth coordinates, simplifying the optimization difficulty, as demonstrated in Fig. R6 and Table 2.\n\n**Multi-object Videos:** With multi-object mask labels, our approach can be extended directly by increasing the dimension of mask feature. We evaluated the reconstruction quality of our model under two different conditions: merging the masks of different objects into a single-channel mask and increasing the mask dimensions to fit multiple objects. We conducted evaluations in two different scenarios, judo, and goldfish, as shown in the table below. The results indicate that the outcomes are essentially consistent across both settings. Additionally, we visualized examples of multi-object scene editing in Fig. R5.\n\n|  | Judo |  |  | Golden-fish |  |  |\n| --- | --- | --- | --- | --- | --- | --- |\n|  | PSNR↑ | SSIM↑ | LPIPS ↓ | PSNR↑ | SSIM↑ | LPIPS ↓ |\n| Single Mask | 39.90 | 0.9706 | 0.088 | 33.17 | 0.9410 | 0.1431 |\n| Multi Masks | 39.47 | 0.9712 | 0.083 | 33.35 | 0.9412 | 0.1445 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M56SEJjNSd", "reviewer": "Reviewer_rPzr", "summary": "The work introduces a video Gaussian representation that leverages 2D priors, such as depth and optical flow, to regularize 3D Gaussians in for various 2D video tasks. This representation can be used for several downstream video processing and editing applications, including dense tracking, enhancing temporal consistency of 2D depth and features, geometry and appearance editing, video interpolation, novel video synthesis, and the creation of stereoscopic videos.", "questions": "- What is the resolution and fps of the dataset?\n- Compared to the baselines, does the proposed representation require more storage space?\n- Is there a tradeoff between temporal smoothness and spatial details in the proposed method?\n- How effective is the model on downstream tasks in terms of quantitative results?\n- How difficult is it to tune the hyperparameters?\n- To what extent can the novel synthesis work well?\n- What is the inference speed? Does the unified framework slow down the inference speed of 3DGS?", "limitations": "I agree with the limitation mentioned by the author.\n- One possible limitation is hard to capture long-duration clip or clip with long-range correspondence.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Unlike existing methods, this work learns Gaussian in a canonical 3D space (xyz) and reprojects them into the 2D domain as 2D frames for video tasks. This novel approach mitigates the negative impact of occlusions, facilitates the process of capturing complex motion, and enhances temporal consistency. To achieve these improvements, the work uses depth and optical flow to regularize the learning process. By incorporating label loss and non-rigid body constraints, the learned canonical 3D space supports several downstream tasks. The supplemental video clearly demonstrates the capabilities of the proposed representation.", "weaknesses": "- Learning one representation per clip (50-100 frames) takes 20 minutes, which is time-consuming. At 30 fps, encoding a one-minute clip requires 6 hours, and a ten-minute video takes approximately 2.5 days.\n\n- The work can support several image features, such as SAM. However, different image features may require varying types and amounts of 3D Gaussians. For instance, RGB images need a greater number of smaller Gaussians, while segmentation requires fewer but larger Gaussians. The method may overestimate the number of required Gaussians.\n\n- Although the method can enhance the temporal consistency for depth and SAM feature, it looks like the spatial domain is less smooth (more noisy) than these per-frame method in the video. Also, the per-frame method such as SAM can support different image per model, while the proposed method needs one model per clip.\n\n- The performance of the method on depth estimation, segmentation mask estimation, tracking, novel view synthesis, and geometry editing in terms of quantitative results is unknown. The work only provides the PSNR of RGB reconstruction. Additionally, during geometry editing, human contours are still visible in the video.\n\n- The model appears to be sensitive to the learning rate, requiring different learning rates for different losses (see Appendix Table 2).\n\n- The work does not discuss failure cases. For instance, it does not explain why the PSNR for the elephant and cow are worse than the baselines. This could be due to hyperparameter issues. Also, how bad is it on drastic motion data?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable feedback and address the concerns raised.\n\n**Training Speed**: While our method's training time relative to video duration is lengthy, it demonstrates significantly faster training speed compared to other methods, as shown in Table R1. We also report superior GPU memory and rendering efficiency (FPS). It's worth noting that once trained, these models can support various applications. Moreover, the speed may also potentially benefit from hardware advancements or parallel processing techniques.\n\n**Overestimation of Gaussians?**: This query touches on the different frequencies between RGB and feature images. While features with higher semantic content may skip low-level signals, our current approach didn't process these signals specially but focused on a universal video processing representation, which might lead to a over-parameterized Gaussians for features. Although a finer Gaussian representation might seem excessive, it prevents information loss and maintains sharp segmentation contours, ideal for handling the dense information in RGB images.\n\n**About SAM Feature**: The SAM feature appears smoother because it's interpolated from a lower-resolution tensor to full image resolution, whereas our rendering feature maintains the original resolution. Any high-frequency details smoothed during this process do not affect the final mask quality, as shown explicitly in Fig. R4 of the attached pdf file. Although SAM is more generalized, our method ensures better object integrity and segmentation quality across video frames, with more efficient processing time by skipping the image encoding step. The comparison is reported below:\n\n|  | Per-frame SAM | Ours |\n| --- | --- | --- |\n| IoU ↑ | 0.753 | 0.827 |\n| Time/s | 0.513 | 0.025 |\n\n**Sensitive to learning rate?**: All the videos share the same hyperparameters including loss weights and learning rate. We didn’t tune the parameters for different videos. Actually, most of the learning rates in Appendix Table 2 are derived from 3DGS.\n\n**Failure cases:**  We identify two scenarios where our method may underperform compared to dynamic GS methods or exhibit suboptimal performance, illustrated in Fig R7. \n\n-  The first scenario involves videos characterized by significant camera motion with few moving objects (e.g. elephant and cow in Table 1 of the paper). In these instances, the scene resembles a static environment where tools like COLMAP can effectively estimate precise camera poses. Dynamic GS methods model background motion using cameras extrinsic, whereas our method learns background trajectory during optimization. This approach imposes additional complexity on our training process, as our method must handle the bundle adjustment typically managed by COLMAP. In contrast, dynamic GS methods can fully use the camera prior, which is simpler for them to process. \n-  The second scenario involves videos with transient moving objects that appear only briefly, without continuous motion. In these cases, the estimated flow tends to be inaccurate, making it challenging for our method to learn pixel correspondence solely through the photometric consistency of Gaussian rendering.\n\n**Other Questions**\n\n**Resolution / FPS of dataset**:  We follow the Tap-Vid DAVIS benchmark (480p). The resolution of each video is 910 x 480 or 854 × 480, with a frame rate of 24 FPS.\n\n**Storage space**:  Our model size differs scene by scene, typically less than 100 MB, comparable with 4DGS, and much less than CoDeF (around 500M).\n\n**The trade-off between temporal smoothness & spatial details**:  Interesting observation. We feel there is no inherent trade-off between temporal smoothness and spatial details. This is due to the fact that natural motion patterns are often temporally smooth. Temporal smoothness helps prevent overfitting to the training data, avoiding fitting noise and thus it can even help improve spatial qualities. \n\n**Quantitative results on downstream tasks:** We have evaluated the tracking results on the Tap-Vid DAVIS benchmark. Our method outperforms most existing methods, except for Omnimotion. This is because Omnimotion is specifically designed for tracking tasks and performs poorly in terms of reconstruction and other metrics, and it does not support other editing tasks. In contrast, our method supports versatile video processing tasks, with higher computational efficiency and training efficiency. We also evaluate the segmentation results with our rasterized SAM feature on the gold-fish video, compared with SAM and reported in the above table.\n\n**Difficulty in tuning hyperparameters**:  We have not extensively fine-tuned hyperparameters such as learning rates. Our method shows robust performance and achieves the highest performance compared with existing methods with the same hyperparameters.\n\n**Extent of novel view synthesis**:  Our method effectively manages view changes in the application of generating stereoscopic videos. The specific boundary of view changes is determined by the accuracy of monocular estimation. With adequate depth supervision, our method can accommodate significant view warping. This important role of depth prior plays is also highlighted in the concurrent work \"Shape of Motion: 4D Reconstruction from a Single Video.\"\n\n**Inference speed**: The inference speed of our method is a little slower than 3DGS but still fast enough for real-time applications, approaching over 150 FPS on a resolution of 854×480 on a single NVIDIA RTX3090 GPU.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their valuable comments. We are glad and appreciate that the Reviews for the comments of “well structured, easy to follow” (**rMXd, 8X9e**),  “novel and well motivated” (**8X9e, rPzr**), and recognize that the versatility video processing ability is  “well demonstrated, very ambitious and very encouraging” (**rMXd, 8X9e, axg5, rPzr**). We will release the code to the community to facilitate the progress in this area.

Below we first clarify our key novelty:  our work represents an early exploration of 3D-based representations of ***casually captured*** videos, enabling a variety of applications. Our aim is not to claim any specific designs as major innovations, but rather to highlight the system that effectively integrates different components to learn such representations from casually captured videos, despite the highly ill-posed nature of the problem. This work aims to provide new insights into 3D representations for videos. 

We also greatly appreciate the reviewers' suggestions regarding the experimental section of our paper. In response, we have added experiments conducted on the **Tap-Vid DAVIS benchmark** (30 videos, 480p), providing a detailed report of various metrics. Additionally, we have included comparisons with dynamic NeRF/GS methods. It is noteworthy that in-the-wild videos often lack accurate camera poses, which significantly limits the performance of such methods.  The visual comparison is illustrated in the Fig.R1 of the attached PDF file.

Table R1.  Comprehensive comparison with existing methods on Tap-Vid benchmark (DAVIS). 

| Metric | PSNR ↑ | SSIM ↑ | LPIPS ↓ | AJ ↑ | <$\delta_{avg}^x ↑$ | OA ↑ | TC ↓ | Training Time | GPU Memory | FPS |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 4DGS | 18.12 | 0.5735 | 0.5130 | 5.1 | 10.2 | 75.45 | 8.11 | ~40 mins | 10G | 145.8 |
| RoDyF | 24.79 | 0.723 | 0.394 | \ | \ | \ | \ | > 24 hours | 24G | < 0.01|
| Deformable Sprites | 22.83 | 0.6983 | 0.3014 | 20.6 | 32.9 | 69.7 | 2.07 | ~30 mins | 24G | 1.6 |
| Omnimotion | 24.11 | 0.7145 | 0.3713 | **51.7** | **67.5** | **85.3** | **0.74** | > 24 hours | 24G | < 0.01 |
| CoDeF | 26.17 | 0.8160 | 0.2905 | 7.6 | 13.7 | 78.0 | 7.56 | ~30 mins | 10G | 8.8 |
| Ours | **28.63** | **0.8373** | **0.2283** | 41.9 | 57.7 | 79.2 | 1.82 | ~30 mins | 10G | 149 |

Table R2.  Ablation of each module in our framework. We have ablated the motion regularization loss in the supplementary material. The ablation study of other modules is reported below, carried out on 5 videos ("bike-packing", "blackswan", "kite-surf", "loading", "gold-fish"), including diverse scenarios.

|  | Ours | Perspective Camera | w/o Flow Loss | w/o Depth Loss | L2 Depth Loss |
| --- | --- | --- | --- | --- | --- |
| PSNR ↑ | **29.61** | 22.51 | 25.16 | 29.18 | 28.15 |
| SSIM ↑ | **0.8624** | 0.6908 | 0.6937 | 0.8475 | 0.8214 |
| LPIPS ↓ | **0.1845** | 0.3958 | 0.4724 | 0.2449 | 0.3328 |

Please also pay attention to the attached PDF file for more visualization results.

Meta-review: This paper received highly diverse ratings, where the discrepancy in the final rating boils down to a single question -- was the original submission too rough?

Extensive discussions were held between the AC, the SAC, and the reviewers, resulting in 64 forum replies (including reviews and the rebuttal), which far exceeds the typical number of posts for a single paper. All reviewers agreed that, in light of the rebuttal, the results seem solid, with the original claims in the paper holding. There was disagreement on whether the paper's presentation was clear enough, but this seems to have had mixed evaluations, making the AC believe that the quality was acceptable.

The contention was whether the original submission had enough results to support the claims made in the original submission. Two reviewers were in favour, as they were, although with qualitative highlights and only performed with a subset of the typical datasets used, demonstrated. One reviewer was initially negative but later changed their thoughts to conclude that it was enough, especially given that the original claims were supported by the rebuttal, and nothing changed concerning the claims. One reviewer remained negative, as the reviewer's opinion was that the original submission's results never had a "complete" set which would be required of a NeurIPS paper. An agreement was never met among the reviewers.

A significant concern raised by Reviewer rMXd was that the paper, according to the reviewer's point of view, was not finished at the time of submission, and accepting such work creates a precedent of unfair practice.  For example, an author could at the time of submission have the ideas written down, take the risk of not having the results, and then make them ready by the time of the rebuttal. This is clearly not what the NeurIPS FAQs suggest, as the FAQs state that the basis of the decision would be the original submission. The AC and the SAC also discussed whether this would be the case. It is both the AC and the SAC's opinion that **this is not the case.** The original submission, including the supplementary materials does provide at least a proof of concept result, which during the rebuttal phase was supported by numerical results. Within the reviewer discussions, it was also pointed out that NeurIPS does allow for an extra page for the camera ready, and also will have reviews public. It was the positive reviewers' consensus that these should be enough to have the revised version up to NeurIPS standards, without a major revision of the core of the paper, only adding the new experimental results provided in the rebuttal. To conclude, it is the AC's belief that the original submission had merit, which was further verified by the rebuttal, and that the original submission was perhaps incomplete **but not half-finished**.

Thus, the AC, in consultation with the SAC and extensive discussions with the reviewers, is recommending the paper for acceptance. The AC would further like to comment that the authors must include all of the results discussed in the rebuttal to the main paper.

The AC would further like to thank the extensive effort that the reviewers have taken to review the paper and participate in the discussions. The decision for this paper really boils down to philosophical debate and subjective evaluations of where we draw the line for a complete conference submission, especially related to empirical validation. The extensive debate that was held for this paper clearly demonstrates that this paper is on the borderline. While the current recommendation is to accept the paper, the AC would like to mention that either decision could have been justified. A critical component that was taken into account in the decision process was that the method remained identical, and the only things that were added were the numbers, including those on new data, and that the idea presented in the paper was interesting enough to excite reviewers to fight about it. Thus, sharing this with the broader research community would be valuable.
Decision: Accept (poster)