HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting
Authors: Qiankun Gao
Jiarui Meng
Chengxiang Wen
Jie Chen
Jian Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Gaussian Splatting
Streamable Dynamic Scene
Online 4D Resconstruction
Primary area: online_learning

Abstract: The online reconstruction of dynamic scenes from multi-view streaming videos faces significant challenges in training, rendering and storage efficiency. Harnessing superior learning speed and real-time rendering capabilities, 3D Gaussian Splatting (3DGS) has recently demonstrated considerable potential in this field. However, 3DGS can be inefficient in terms of storage and prone to overfitting by excessively growing Gaussians, particularly with limited views. This paper proposes an efficient framework, dubbed HiCoM, with three key components. First, we construct a compact and robust initial 3DGS representation using a perturbation smoothing strategy. Next, we introduce a Hierarchical Coherent Motion mechanism that leverages the inherent non-uniform distribution and local consistency of 3D Gaussians to swiftly and accurately learn motions across frames. Finally, we continually refine the 3DGS with additional Gaussians, which are later merged into the initial 3DGS to maintain consistency with the evolving scene. To preserve a compact representation, an equivalent number of low-opacity Gaussians that minimally impact the representation are removed before processing subsequent frames. Extensive experiments conducted on two widely used datasets show that our framework improves learning efficiency of the state-of-the-art methods by about 20% and reduces the data storage by 85%, achieving competitive free-viewpoint video synthesis quality but with higher robustness and stability. Moreover, by parallel learning multiple frames simultaneously, our HiCoM decreases the average training wall time to <2 seconds per frame with negligible performance degradation, substantially boosting real-world applicability and responsiveness.

HiCoM: Hierarchical Coherent Motion for Streamable Dynamic Scene with 3D Gaussian Splatting
Qiankun Gao Jiarui Meng', Chengxiang Wen', Jie Chen Jian Zhang 1.3
School of Electronic and Computer Engineering, Peking University, Shenzhen, China ?Pengcheng Laboratory, Shenzhen, China
Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology, Shenzhen Graduate School, Peking University, Shenzhen, China
300
HSCoM-in (ours) Hicpot (ours)
Training:
Training: 3.9
Rendaring 274 tps
Rendering 284 fps
Storage: 0.8 MB
Storage: 0.5 MB FSNR: 26,77
200
PSNR:
Rendering
1:00
SocamAF
Training Time (seconds)
Figure 1: The proposed HiCoM framework for streamable dynamic scene reconstruction achieves competitive rendering quality with significantly shorter training time, faster rendering speed, and substantially reduced storage and transmission requirements. The left figures show results of our HiCoM N3DV M and Meet Room 2 datasets, where "Res" indicates video resolution. The right figure is tested on the N3DV [ dataset, where the radius of the circle corresponds to the average storage per frame and the method the top left corner demonstrates the best performance.
Abstract
The online reconstruction of dynamic scenes from multi-view streaming videos faces significant challenges in training, rendering and storage efficiency. Har- nessing superior learning speed and real-time rendering capabilities, 3D Gaussian Splatting (3DGS) has recently demonstrated considerable potential in this field. However, 3DGS can be inefficient terms of storage and prone overfitting by excessively growing Gaussians, particularly with limited views. This paper pro- poses an efficient framework, dubbed HiCoM, with three key components. First, we construct compact and robust initial 3DGS representation using a perturbation smoothing strategy. Next, we introduce Hierarchical Coherent Motion mecha- nism that leverages the inherent non-uniform distribution and local consistency 3D Gaussians to swiftly and accurately learn motions across frames. Finally, continually refine the 3DGS with additional Gaussians, which are later merged into the initial 3DGS to maintain consistency with the evolving scene. To preserve compact representation,
equivalent number of low-opacity Gaussians that minimally impact the representation are removed before processing subsequent frames. Extensive experiments conducted our framework improves learning efficiency of the state-of-thh-art methods by about 20% and reduces the data storage by 85%, achieving competitive free-viewppint video synthesis quality but with higher robustness and stability. Moreover, by parallel learning multiple frames simultaneously, our HiCoM decreases the average training wall time to < 2 seconds per frame with negligible performance degrada- tion, substantially boosting real-world applicability and responsiveness. Code is avaliable at hattps://ge thub. com/ com/ggk//eae/ooooooooomooo HiCM
widely used datasets show that
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
The online reconstruction of dynamic scene from multi-view video streams essential for advancing applications such real-time free-vieppoint video (FVV) and virtual reality (VR), which are revolu- tionizing entertainment, education, and industry by providing immersive and interactive experiences. However, achieving high fidelity streamable dynamic scene poses significant challenges in training time, rendering speed, data storage and transmission efficiency.
Traditional methods for modeling and representing dynamic scenes, such as surface estimation 3 multi-sphere imaging 4 and depth mapping [5 6 via multi-view stereo technologies, struggle with the complex geometries and varied appearances of real-world scenarios. These techniques also face limitations in capturing the detailed and dynamic nature of practical environments.
Neural Radiance Fields (NeRFs) 7 throughs in 3D reconstruction and novel view synthesis by mapping spatial coordinates and viewing directions to color and density using a neural network. Dynamic NeRFs [[] [8] 19 20 21 [2 23 integrate temporal components capture scene changes demands of NeRF's volume rendering framework limit their feasibility in real-time applications.
g 10, 12 L3 14 1., [6] have made significant break-
time. However, the high computational
Acknowledging NeRF's limitations, researchers have introduced 3D Gaussian Splatting (3DGS) 24. an innovative method for fast 3D reconstruction and real-time rendering. In contrust to NeRF, 3DGS explicitly employs anisotropic 3D Gaussians primitive elements to represent 3D scenes, then rasterizes these Gaussians to render images from specific viewpoints, bypassing the complex and slow volume rendering pipeline. This not only speeds up the rendering process but also enhances the quality of the synthesized views. In a similar vein to NeRF, 3DGS has been adapted for dynamic scene reconstruction: several Dynamic Gaussian Splatting works 25
26 27 directly expand attributes of 33 34 focus on decoupling the dynamic scene into a base scene representation in canonical space and time-varying motion fields.
Gaussian primitives; other efforts [28] 29 30
31
32
Harnessing its efficiency, 3DGS naturally supports online learning of dynamic scenes. A recent development, 3DGStream (35), introduces pioneering framework that uses a Neural Transformation Cache derived from InstantNGP [36] to model the scene changes from previous frame to current frame, significantly reducing training time and storage requirements. However, this frame-by-frane online learning pipeline heavily relies on the quality of the initial 3DGS representation. Given that dynamic scenes are generally captured with a limited number of cameras, 3DGS can be prone to instability and overfitting when faced with sparse views. Furthermore, the number of Gaussian primitives in the initial 3DGS representation impacts the learning efficiency of subsequent frames. Despite employing multi-resolution hash encoding and lightweight MLP to alleviate inefficiency and speed up convergence, implicitly modeling the motion field still cannot fully capture critical aspects of the explicit and discrete nature of 3DGS. Although 3DGStream introduces new Gaussians to adapt to the appearance of new objects, these added Gaussians are not carried over to subsequent frames to maintain 3DGS compact. As real-world scenes evolve gradually, the discrepancies from the initial scene accumulate, necessitating more Gaussians accurately capture these changes.
In this paper, we introduce the Hierarchical Coherent Motion (HiCoM) framework, a novel approach designed to enhance the efficiency and stability of streamable dynamic scene online reconstruction. Our HiCoM framework begins with the learning of a compact and robust initial 3DGS representation through a perturbation smoothing strategy. This ensures reduced number of Gaussians, alleviates overfitting and establishes a foundation for consistent quality across frames. Then, we leverage the inherent non-uniform distribution and local consistency of 3D Gaussians to implement a hierarchical coherent motion mechanism. Specifically, we partition the scene into regions and recognize that only a few regions actually contain Gaussian primitives due the non-uniform distribution of Gaussians. We explicitly model the motion within these non-empty regions, allowing Gaussians in the same region to share identical motion patterns. These regions can be further divided into smaller areas, enabling the motion of each Gaussian to be determined by the combined motions of all levels of regions inhabits. This hierarchical coherent motion mechanism captures motions from coarse to fine granularity and requires only minimal set of parameters, facilitates rapid convergence. The inherent structure and consistency within and between regions thus support swift learning of scene changes across frames. We also introduce additional Gaussians to better accommodate significant updates in scene content. These new Gaussians are carefully integrated into the initial 3DGS representation to ensure continuous consistency with the evolving scene. To maintain the compactness of the 3DGS,

---

an equivalent number of low-opacity Gaussians, which no longer significantly contribute to the scene representation, will be removed before the learning of the next frame. This continual refinement to the initial 3DGS representation ensures it remains close as possible to the evolving scene, facilitating better subsequent learning. In addition, we introduce a parallel training strategy that enables simultaneous learning of multiple frames, significantly enhancing training efficiency with minimal impact on performance. The contributions of this paper are summarized as follows:
We introduce the HiCoM framework for online learning of dynamic scene from multi- view video streams, featuring a perturbation smoothing strategy for robust initial 3DGS representation learning, hierarchical coherent motion mechanism for efficient motion capture. and continual refinement to adapt evolving scene updates.
We devise a novel and concise hierarchical coherent motion mechanism that capitalizes on the inherent non-uniform distribution and local consistency of 3D Gaussians, efficiently capturing and modeling scene motions for swift and precise frame-to-frame adaptation.
Extensive experiments demonstrate that our HiCoM framework improves learning efficiency by about 20% and reduces data storage by 85% compared to state-of-the-art methods. Our parallel training strategy enables learning multiple frames simultaneously, substantially decreasing the training wall time with negligible effects on overall performance, further enhancing the practicality and responsiveness of our HiCoM for real-world applications.
2 Related Work
2.1 Dynamic Gaussian Splatting
3D Gaussian Splatting 24 has quickly garnered attention for dynamic scene reconstruction due its directly expand attributes of Gaussian primitives: 4D Gaussian Splatting (26) adds timestamp dimension to form 4D Gaussian primitives; Dynamic 3D Gaussians (5 incorporates positions and rotations at every decouple
efficiency and flexibility. 1) Several Dynamic Gaussian Splatting works [25] 26 27
timestamp into the Gaussian primitives. 2) Other works 28
29
30
31
32
33
the dynamic scene into base scene representation in canonical space and time- varying motion fields: [8] uses deep MLP to predict the motion of Gaussians based on their positions and the given timestamp. enabling detailed and real-time rendering of dynamic scenes; 4D-GS 29 refines this pipeline by employing a more efficient multi-resolution hex- hex-planes and a lightweight MLP, significantly enhancing efficiency and performance, particularly in high-resolution real-time rendering scenarios; SC-GS (30 assumes that scene motions are driven by small number of key point movements, leveraging sparse control points and dense Gaussians to distinctively represent motion and appearance in dynamic scenes, and employs a deformation MLP to predict time-varying 6 DoF' transformations for each control point, which are then locally interpolated through learned weights to create a motion field that influences the dense Gaussians, ensuring coherent and dynamic scene rendering.
Deformable 3D Gaussians
(28), inspired by dynamic NeRFs [
22 19 20
21
2.2 Online Learning of Streamable Dynamic Scene
Compared to the offline learning from fully captured multi-view videos, on-the-fly learning, i.e., frame-by-frame training, for constructing streamable dynamic scenes presents more challenges. Given the success of NeRFs 7 g 36
20 [9] in addressing both static and dynamic scenes, there has been growing interest in adapting and refining these techniques to solve the challenges of streamable dynamic scenes. StreamRF 2 tackles this by employing an incremental learning paradigm, where per-frame differences are modeled to efficiently adapt to changes, significantly speeding up the training process and reducing storage overhead than traditional methods. NeRFPlayer (38), on the other hand, decomposes the 4D spatiotemporal space based on temporal characteristics, optimizing for both speed and compactness in model representation, thus enabling fast reconstruction and streamable rendering. ReRF [39 extends these capabilities by modeling the residual information between adjacent timestamps, which allows for real-time free-viewpoint video (FVV) rendering of long-duration dynamic scenes with high fidelity. The emergence of 3D Gaussian Splatting (3DGS) has brought significant advancements in training efficiency and rendering speeds, making exceptionally suited for addressing streamable dynamic scenes. Building on superior 3DGS and inspired by previous NeRF works [2, 336, 3DGStream (35) introduces efficient framework that
3

---

uses a Neural Transformation Cache to manage the transformations of 3D Gaussians, drastically reducing the training time and storage requirements. This method also features an adaptive addition strategy for 3D Gaussians to better handle emerging objects in dynamic scenes.
3 Preliminaries
3D Gaussian Splatting explicitly represents scenes using anisotropic 3D Gaussian primitives, each defined by mean vector la and covariance matrix Σ, which respectively characterize the central position and geometric shape of the Gaussian in world space, mathematically formulated as:
G(x) = - (x-+) (x-())
The covariance matrix Σ is decomposed into scaling matrix S and a rotation matrix R to ensure physical meaning and facilitate optimization:
Σ RSSTR",
where S = diag(as 82) E TR3 and R E 8O(3) are parameterized as 3D scaling vector s and rotation quaternion q. respectively. Gaussian primitive is enriched with color and opacity, represented by spherical harmonic coefficients h and scalar a, respectively.
To render a novel viewpoint, Gaussian primitives are projected onto the camera plane using the view projection matrix W and the Jacobian J of the affine approximation of the projective transformation, the covariance matrix E' in camera coordinates given follows:
E' JWEW J
Rendering is performed by blending the contributions N overlapping Gaussian primitives at each pixel, taking into account their depth-ordering to ensure correct compositing, expressed
C= Σ C,O: I1)
(4)
KEN
1=1
where Ci aG represents the color and blending weight of the jth Gaussian, respectively.
The training of 3D Gaussian Splatting alternates between parameter optimization and density control.
Parameter optimization is supervised by the L1 loss and D-SSIM term:
L= (1 A)C1 + AL
(5)
where À typically set to 0.2. Meanwhile, density control manages Gaussian cloning and splitting to address overreeconttruction and under-reconssrructon.
4 Methodology
The online reconstruction of streamable dynamic scenes follows a sequential and iterative pipeline, crucial for applications requiring real-time rendering and interaction. Initially, a 3D Gaussian Splatting (3DGS) representation of the scene is constructed from the first frame of synchronized multi-view video streams. As each new frame is captured, this representation is updated and refined to align with the latest scene. This process demands high-quality scene representation at every timestamp to ensure visual fidelity, while also necessitating efficiency in learning, storage, and transmission to make feasible for real-world applications.
To achieve these goals, our framework (Figure 2 introduces three key designs: First, we develop compact and robust initial 3DGS representation using perturbation smoothing strategy, ensuring stable foundation for subsequent frame learning (Sec. 4.1) Second, we employ a hierarchical coherent motion mechanism, leveraging the inherent non- non-uniform distribution and local consistency of 3D Gaussians to model and rapidly learn scene motion from the previous frame to the current frame (Sec. 4.2p. Third, we implement continual refinement strategies to adapt to scene content updates missed by the motion mechanism, maintaining fidelity and accuracy the scene evolves (Sec. 4.3) Importantly, we develop a parallel training strategy that enables simultaneous learning multiple frames (Sec. 4.4). significantly enhancing the practicality and responsiveness of our framework.

---

t=
t=1
Herrrchical Coherent Motion
Perturbation
Rendening
Continual Refirement
(a) Initial Representation Leaming
Rendening
ieegge
Rendering
Ap Ag
(b) Hierarchical Coherent Motion
(c) Continual Refinement
Mations Ascociated with Regions
Operation Flow
Gradient Flow
Frazen
Training
Figure 2: Illustration of our HiCoM framework. We first construct compact and robust initial 3DGS representation from the first frame of the video streams with the perturbation smoothing strategy (a). Then, we learn each subsequent frame based on the previous 3DGS with proposed Hierarchical Coherent Motion mechanism (b) and Continual Refinement (c) with additional new Gaussians. This learning process continues until the final frame.
4.1 Initial 3DGS Representation Learning
The initial 3DGS representation forms the cornerstone of the entire dynamic scene reconstruction process, laying the groundwork for precise and efficient frame-hy-frane online learning.
In typical setups for capturing dynamic scenes, the number of cameras is often limited, contrasting sharply with static scenes that usually involve hundreds of multi-view images. This limitation can lead to instability and potential overfitting training views during optimization. Moreover, not adequately controlled, the number of Gaussians could grow unchecked, slowing down the rendering process and convergence speed the learning of subsequent frames.
To combat overfitting, prevalent issue in neural network training, various strategies such as dropout [40] and data augmentation [41
42 43 have been proposed. Drawing from these in- sights, we discover that simple perturbation smoothing strategy helps mitigating this issue in 3DGS, which has proven also effective in NeRF
44 As Figure
2 (a), add small gaussian noise the
position attribute 3D Gaussian during training, creating the random perturbed position:
Pp = f Anoise
where E N~0
(6)
thereby preventing the model from becoming too closely fitted the limited training views. Random perturbation not only stabilizes the learning process but also moderates the growth of 3D Gaussians. As a result, by the time the model converges, the number of Gaussians is significantly reduced, speeding up the convergence and enhancing the overall efficiency in subsequent frame learning.
4.2 Hierarchical Coherent Motion
Follow the online learning pipeline, we continuously fetch the latest views from the video streams and reconstruct the current scene representation. A straightforward method would be to use these new images to fine-tune the previous 3DGS. However, this naive approach is still insufficient in terms of learning speed and requires storing and transmitting significant amount of data.
Although recent efforts 28
35 have demonstrated the feasibility of using neural networks to predict the motion of each 3D Gaussian, showing promising results in adapting to scene changes, the implicit modeling of the motion field fails align well with the explicit and discrete nature of 3DGS. In contrast, we explicitly model the motion field through hierarchical coherent motion mechanism, allowing us to directly capture and adapt to scene dynamics in a structured and efficient manner.
As illustrated in Figure (b), the scene is first divided into regions with the same size. and we can determine the region in which each Gaussian is primarily located using the following formula:
{
0.5 }
=
5

---

where denotes the center coordinates of the region, pe is the center position of the 3D Gaussian, and represents the size of region.
During optimization, 3D) Gaussians naturally exhibit a non-uniform distribution, with areas of complex geometry and texture densely populated by smaller Gaussians, while simpler and smoother regions sparsely covered by larger Gaussians, resulting in a significant number of regions remaining empty. For those regions that contain Gaussians, we assign translation and rotation attributes, respectively parameterized by a 3D vector Aμ E I and a quaternion Åq E RA. These motion attributes are shared by the Gaussians within the region, ensuring local consistency.
However, motion within smaller subregions might exhibit subtle differences due to more localized dynamics. To accommodate these variations, further partition the regions and establish distinct motion parameters for these smaller areas. Consequently, the motion of each Gaussian is determined by the cumulative of motion from multiple hierarchical levels which it belongs:
Aug
Au',
Aq
Aq
(8)
where AHg E R3 and Aqg F+ represent the composed motion of Gaussian, Ap' indicates the motion contribution from the 1th level of regions, and L is the total number of motion levels involved.
This hierarchical coherent motion mechanism allows us to capture Gaussians' movements ranging from coarse to fine, ensuring a more detailed and accurate representation of dynamic changes across frames, as well as coherent motion between adjacent regions. By sharing motion attributes among multiple Gaussians, we decrease the number of individual computations required and the number of parameters we need to optimize, helping the learning converge faster and more reliably.
4.3 Continual Refinement
A 3DGS representation that is more closely aligned with the most recent scene can be obtained after applying the learned motion. However, this motion mechanism, focusing primarily on modest shifts to the positions and rotations of Gaussians from previous 3DGS, might not capture all the finer details of scene changes or adapt swiftly enough to significant scene content updates. This gap can be strategically bridged by adding new Gaussians in appropriate areas.
Specifically, during motion learning, some regions accumulate significant gradients indicating sub- stantial discrepancies between the learned and actual scene. Following established methods 24] B5)
we clone Gaussians in these regions that have accumulated gradients exceeding a predefined threshold. As shown in Figure (c), the newly cloned Gaussians then subjected further optimization and density control to better align with the most recent scene.
As real-world dynamic scenes typically exhibit coherent changes, newly added Gaussians tend to stay relevant across subsequent frames, making impractical to simply discard them. Thus, we retain these Gaussians, integrating them into the initial 3DGS to form the basis for learning the next frame. However, consistently merging new Gaussians in each frame can lead to a steady increase in the number of Gaussians, potentially slowing down rendering speeds, delaying convergence, and prolonging learning times. Therefore, we selectively remove an equivalent number of low-opacity Gaussians that minimally impact the overall visual integrity of the scene [45), to maintain the com- pactness and efficiency of 3DGS representation without compromising scene fidelity. Simultaneously, this continual refinement allows to address and enhance regions of the 3DGS that may have been underrepresented or insufficiently modeled in earlier learning stages, solidifying a comprehensive and adaptive representation as the scene evolves.
4.4 Parallel Training
Typically, our framework works in a sequential frame-by- pipeline. To further boost efficiency and responsiveness for practical applications, we draw inspiration from video encoding techniques where reference frames (such as I-frames in H.264/AVC) are used to predict multiple subsequent frames, leveraging temporal redundancy and reducing computational overload.
Considering that differences between consecutive frames are generally minor, we choose the 3DGS of the frame t as the reference simultaneously learn frames {t + 1,... ,t + k). After processing

---

Table Quantitative comparison results. "Storage" is reported without/with the initial frame. The method with is reproduced by us. StreamRF metrics on Meet Room only cover the discussion scene. experiments use original datasets without additional undistortion. Please also see Tables 6 and
N3DV
Meet Room
Method
PSNR
Storage
Train
Render
PSNR
Storage
Train
Render (FPS 1)
(dB f)
(MB ))
(s L)
(FPS f)
(dB 1)
(MB J)
(s J)
Naive 3DGS
30.69
95.8
337
217
26.37
46.9
152
275
StreamRF 2
30.68
17.7/31.4
10.0
13
26.72
5.7/9.0
6.8
155
3DGStream 135
30.15
7.6/7.8
8.2
210
25.96
4.0/4.
4.7
212
HiCoM (ours)
31.17
0.7/0.9
6.7
274
26.73
0,4/0.6
3.9
284
HiCoM-P4 (ours)
31.00
0.7/0.9
1.7
274
26.25
0.4/0.6
1.0
284
these frames, the 3DGS of the frame becomes the new reference for learning the next & frames. This parallel training strategy significantly reduce the average wall time cost per frame.
5 Experiment
5.1
Experimental Setup
For all of our experiments in this paper, we utilize two widely-used public datasets as follows.
Neural 3D Video (N3DV) [1] dataset comprises six dynamic indoor scenes featuring varying illuminations, view-dependent effects, and substantial volumetric details. Videos were captured at resolution of 2704 x 2078 and frame rate of 30 FPS. Video counts per scene range from 18 to 21. We follow prior works 2 35 29] to downsample the original video resolution by factor of two.
Meet Room [2] dataset was recorded with multi-view system consisting of 13 synchronized Azure Kinect cameras, covering three different indoor scenes. The resolution of captured videos is 1280 720 and the frame rate 30 FPS too.
For both datasets, the video captured by the center reference camera is used for testing, while other videos are employed for training. Both datasets already include camera pose information, so we only generate initial point clouds with COLMAP [46] following 4D-GS [2)]. Most scenes last only 10 seconds, thus, we restrict our experimentation to the first 300 frames to align with prior works.
We evaluate methods based on four metries, across all 300 frames: 1) video synthesis quality of the test views, assessed by the mean PSNR; 2) average training time per frame; 3) rendering speed, assessed by the frame rate of the synthesized video; 4) storage and transmission efficiency, evaluated by the average size of the representation parameters per frame.
Implementation. Our initial frame learning employs the standard 3DGS with fixed Anoise= 00.11, training for 15k and 10k steps on two datasets respectively, with splitting halted at the 5k step. The number of regions the smallest level is determined by dividing the number of Gaussians n in the initial 3DGS by the maximum number of Gaussians m= per region. The number of regions at each subsequent larger level is determined by further dividing by 23, and so forth, with only regions containing Gaussians having motion parameters, which are initialized at 0.6x of the previous frame's motion parameters. Each new frame undergoes 100 motion training steps, with an additional 100 steps after new Gaussians are added. We implement 3DGStream (35] in the same codebase as ours following its paper, and run all experiments 3 times on RTX 4090 GPUs, the mean metrics are reported. More details are provided in the Appendix.
5.2 Experimental Results
Quantitative Benchmark We conduct a comprehensive quantitative comparison with state-of-the- art online methods, StreamRF [2] and 3DGStream [35], presenting results in Table The Naive 3DGS serves as a baseline, training from scratch frame-by-frame with standard 3DGS method. The data of StreamRR are referenced from the 3DGStream paper, with training time and rendering speed estimated based on the compute capabilities of RTX 3090 and 4090 GPUs. Our HiCoM consistently achieves competitive video synthesis quality (PSNR), while significantly outpacing the
7

---

GT
3DG am
Ours
Figure 3: Qualitative results of Coffee Martini scene. Frames shown are the 1st, 61st. 121st, 181st 241*t, and 300th from the test video. Red boxes highlight areas with significant temporal motions. Our method achieves temporal coherence more closely matching the ground truth (GT).
two counterparts in training speed, with an average per-frame learning time (include the initial frame) reduced by more than 17% compared to 3DGStream on both datasets. This advantage becomes even more pronounced in complex scenes, such as Coffee Martini and Flame Salmon, where the learning speed improvement exceeds 20%, primarily due to our method's rapid convergence speed. Both our HiCoM and 3DGStream obtain rendering speeds over 200 fps, demonstrating real-time rendering capability. In terms of storage and transmission efficiency, our approach substantially surpasses the two competitors, requiring less than 10% of their average per-frame storage space after the initial frame. This is mainly due to two factors: firstly, our motion parameters are shared among multiple 3D Gaussians within local regions; secondly, the inherent -oinurrform distribution of the 3D Gaussians results in most areas being empty. thus requiring far fewer motion parameters. Due to the reduced amount of information needing transmission,
method is more advantagcous for streamable dynamic scenes. In the Appendix (Tables 6 7 we provide results for each scene and results of several ofate-of-the-art offline methods, offering more detailed quantitative comparison. We also include experimental results on two additional datasets, PanopticSports (25) and ParticleNeRF [47), in the Appendix. On the PanopticSports dataset, which features large- motion, method achieves better performance than the state-of-the-art Dynamic3DGS (25) method, while on the synthetic ParticleNeRF dataset, our HiCoM demonstrates good results as well. These additional experiments further validate the generalization capability of our approach across diverse datasets.
Qualitative Analysis Our HiCoM framework inherits the advantages of 3D Gaussian Splatting in representing static scenes but focuses more the changes in dynamic scenes over time. As shown in 3 the scene features a person making a drink, with his head, hands, and the liquid in the cup being the main dynamic areas. We select six frames, arranged in chronological order, from all 300 frames to analyze the rendering quality. It can be observed that our method produces results that are much closer to the ground truth in these dynamic arcas. Furthermore. our HiCoM exhibits more coherent temporal transitions because, in addition to updating the initial Gaussian positions and rotations, we carry over newly added Gaussians the subsequent frames instead of discarding them. This results in smoother and more natural transitions over time, such as the liquid gradually flowing into the glass. More qualitative results for other scenes provided in the Appendix.
5.3 Ablation Study
As shown in Table 2 we ablate three key components of our HiCoM framework. In the 2nd row, we do not use noise perturbation, i.e., the initial frame learning falls back standard 3DGS training. This causes many small Gaussians to split during the initial learning to fit the training views, resulting in a slight decrease in PSNR on the test views. By incorporating noise perturbation, our HiCoM reduces the number of Gaussians while making them larger, each contributing to the rendering of more pixels. Consequently, the training time was slightly longer (1*t row vs. 2nd row). In the 3rd row, we disable motion learning, using the first 100 steps solely for gradient accumulation to determine which Gaussians to clone. This resulted a significant performance drop, particularly in the relatively simple scenes Flame Steak and Discussion, indicating that this component is crucial for
8

---

Table 2: Ablation on main components of our HiCoM framework. Coffee Martini and Flame
Steak are from the N3DV dataset, while Discussion is from the Meet Room dataset.
Coffee Martini
Flame Steak
Discussion
Pertubation
Motion
Refinement
PSNR
Train
PSNR
Train
PSNR
Train ()
(dB
(s ()
(dB f)
(s ()
(dB f)
28.04
7.0
32.87
6.6
26.69
3.9
/
/
26.61
6.3
32.52
5.6
26.19
3.6
25.71
6.4
27.33
5.9
21.61
3.4
/ x 27.87
7.6
31.73
6.7
26.51
4.2
Table 3:
Ablation on convergence. "Em" " and "E," are number of motion learning steps and continual refinement steps, repectively.
N3DV
Meet Room
Fm
Fr
PSNR (dB 1)
Train (s
PSNR (dB 1)
Train (s J)
100
100
31.17
6.7
26.73
3.9
50
30.78
5.2
26.51
3.0
100
150
31.16
8.2
26.77
4.8
50
30.94
5.3
26.77
3.1
100
150
31.10
8.1
26.70
4.6
the overall framework. In the 4th row, we omit the refinement, extending motion learning instead. This leads to slight performance decrease, suggesting that the additional Gaussians further compensate for the intense and detailed scene changes that the motion learning might not capture.
A notable characteristic of our HiCoM is the explicit representation of motion field, which aligns well with the discrete, explicit nature of 3DGS. Our HiCoM requires very few motion parameters, reducing the complexity of motion representation and learning, theoretically accelerating the convergence speed. According to ablation results presented in Table increasing the motion learning steps from 50 to 100 (2 row vs. 1s row) still provides significant performance gains, i.e., 0.39 and 0.22 dB on two datasets, respectively. However, learning an additional 50 steps beyond 100 (1*t row vs. 3rd row) yields only minimal improvements. indicating that the model has mostly converged by 100 steps. Similar conclusions can drawn for learning newly added Gaussians. Ultimately, total of 200 learning steps is sufficient to achieve optimal results.
Our hierarchical coherent motion incorporates multiple levels of motion, with different levels corre- sponding to varying sizes of regions and reflecting different granularities of motion. In Table we present ablation studies on motion levels. We use three motion levels by default (the 5th row), refer to as "fine". "medium" and "coarse", respectively. As the results show, using only the "fine" level already achieves good performance. Adding the "medium" and "coarse" levels provides additional performance gains, e.g., the PSNR of Coffee Martini and Discussion scenes respectively boosts 0.25 dB and 0.59 dB, but the improvements diminish as the number of levels increases, indicating that HiCoM is effective and few motion levels sufficient achieve good results. It can also be observed that fewer motion levels and parameters do not necessarily imply shorter training time. This is because insufficient motion learning may lead to the generation of more Gaussians during continual refinement stage, slightly increasing the overall training time.
5.4 Parallel Training
We conduct experiments on three representative scenes from two datasets to validate the parallel learning. As shown in Table when the number of parallel frames is relatively small, such as 2 and the performance remains largely unchanged, with some scenes even exhibiting improved PSNR. However, as the number of parallel frames increases, the disparities between the learning and the reference frames grow, leading a decline in performance. Specifically, when the number of parallel frames reaches 16, there noticcable PSNR drop, i.e. 0.18, 1.82 and 1.32 dB, respectively.

---

Table Ablation on motion levels. Coffee Martini and Flame Steak are from the N3DV dataset, while Discussion is from the Meet Room dataset. The three motion levels correspond to regions of different sizes: "fine" for smaller regions, "medium" for intermediate regions, and "coarse" larger regions. Rows show results for different combinations of these levels.
Coffee Martini
Flame Steak
Discussion
Motion Levels
PSNR (dB f)
Train
PSNR (dB f)
Train
PSNR
Train
()
()
(dB f)
()
coarse medium fine fine medium fine + medium + coarse
26.67
6.77
30.28
6.26
22.89
3.83
27.24
6.60
32.14
6.26
24.91
3.80
27.79
6.65
32.76
6.33
26.10
3.72
27.94
6.77
32.88
6.37
26.61
3.77
28.04
7.02
32.87
6.62
26.69
3.89
Table 5: Parallel Training Performance on three scenes from N3DV and MeetRoom datasets.
Coffee Martini
Flame Steak
Discussion
#Parallel Frames
PSNR
Train
PSNR
Train
PSNR
Train
(dB 1)
(s ()
(dB f)
(s ()
(dB f)
(s ()
Non-Parallel
28.04
7.02
32.87
6.62
26.69
3.89
28.20
3.51
32.46
3.31
26.66
1.95
28.19
1.76
32.13
1.66
26.39
0.97
28.06
0.88
31.63
0.83
25.89
0.49
16
27.86
0.44
31.05
0.42
25.37
0.25
While parallel learning does not reduce the actual GPU computation time, it significantly reduces the wall time, making the system more practical for real-world applications. For example, with 4-frame parallel learning,
reduce the average learning time per frame to one-fourth of the original time while maintaining nearly the same quality. This means we can learn one frame in less than 2 seconds average, greatly enhancing the efficiency and responsiveness of the system.
Conclusion
This paper introduces HiCoM, a framework designed to tackle the challenges in online reconstruction of streamable dynamic scenes from multi-view videos. HiCoM reduces the number of 3D Gaussians for a more compact representation through the perturbation smoothing strategy, enhancing the robustness of initial frame learning. HiCoM distinguishes with its hierarchical coherent motion mechanism that explicitly represents motion between adjacent frames with minimal parameters, aligning well with the discrete and explicit nature of 3D Gaussian Splatting. This mechanism efficiently captures motion at different granularities, resulting in faster convergence and substantially reduced data storage. Additionally. HiCoM continuously refines the representation to adapt to evolving scene, facilitating better subsequent learning. Experimental results show that HiCoM boosts training efficiency by about 20% and reduces data storage by 85% compared state-of-the-art methods. Furthermore, the parallel training significantly reduces wall time cost with negligible performance degradation, elevating real-world applicability and responsiveness.
Limitations. Despite our HiCoM's significant improvements, the role of the initial 3DGS representa- tion remains crucial in the online learning pipeline, which is still not fully addressed. Future work could explore integrating advanced 3D Gaussian Splatting techniques to enhance the initial frame learning, which, combined with our hierarchical coherent motion mechanism, could further improve training efficiency while also reducing storage and transmission overhead. During online learning, reconstruction quality may decline over time due to error accumulation common issue across many online learning methods and one that requires further research to effectively address. Additionally, our experiments are conducted solely on indoor scenes, so the generalization our method outdoor more complex environments has not been verified and requires further validation.
10

---

Acknowledgements
This work supported in part by the National Key R&D Program of China (No. 2022/D01 18201). the Shenzhen Medical Research Funds China (No. B2302037), Natural Science Foundation of China (No. 61972217. 3207 62176249, 62006133, 133, 6227 1465). Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology (No.
and AI for Science Preffrrrd Program, Peking University Shenzhen Graduate School, China.
References
[1] Tianye Li, Mira Slaveheva, Michael Zollhocfer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2022.
[2] Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Ping Tan. Streaming radiance fields for video synthesis. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), 2022.
[3] Alvaro Collet, Ming Chuang, Patrick Sweeney, Don Gillett, Dennis Evseev, David J. Calabrese, Hugues Hoppe, Adam G. Kirk, and Steve Sullivan. High-quality streamable free-viewpoint video. ACM Transac- tions on Graphics (TOG), 2015.
[4] Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Vall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. Immersive light field video with a layered mesh representation. ACM Transactions Graphics (TOG), 2020.
[5] Yizhong Zhang, Jiaolong Yang. Zhen Liu. Ruicheng Wang, Guojun Chen, Tong, and Baining Guo. Virtualcube: An immersive vidco communication system. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2022.
[6] C. Lawrence Zitnick, Sing Bing Kang, Matthew T. Uyttendacle, Simon Winder, and Richard Szeliski. High-quality video view interpolation using layered representation. ACM Transactions on Graphics (TOG), 2004.
[7] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[8] Jonathan Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, Pratul Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[9] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf' 360: Unbounded anti aliased neural radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[10] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamic Shotton, and Julien Valentin. Fastnerf: High-fidelin neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[11] Ricardo Martin-Bann Brualla, Noha Radwan, Mchdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2021.
[12] Cheng Sun, Min Sun, and Hwann- Twann-Tzoog Chen. Direct voxel grid optimization: Super-fast convergence radiance fields reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[13] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti- aliased grid- neural radiance fields. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[14] Krishna Wadhwani and Tamaki Kojima. Squeezenerf: Further factorized fastnerf for memory-efficiae inference. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
11

---

[15] Alex Yu, Ruilong Li, Matthew Tancik. Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[16] Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. Merf: Memory-efficien radiance fields for real-time view synthesis in unhounded scenes, ACM Transactions on Graphics (TOG), 2023.
[17] Zhiwen Yan, Chen Li, and Gim Hee Lee. Nerf-ds: Neural radiance fields for dynamic specular objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[18] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noo -Noeuer D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[19] Ang Cao and Justin Johnson. Hexplane: fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2023.
[20] Sara Fridovich-Kell Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[21] Jiemin Fang, Taoran Yi, Xinggang Wang. Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. Proceedings of the ACM SIG- GRAPH Conference and Exhibition on Computer Graphics and Interactive Techniques in Asia (SIGGRAPH Asia), 2022.
[22] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu, Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[23] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph I Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), 2021.
[24] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 2023.
[25] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In International Conference on 3D Vision (3DV), 2024.
[26] Zeyu Yang, Hongye Yang. Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. In International Conference on Learning Representations (ICLR), 2024.
[27] Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang Hc, Wenzheng Chen, and Baoquan Chen. 4d gaussian splatting: Towards efficient novel view synthesis for dynamic scenes, arXiv preprint 20X1v:240..03307, 2024.
[28] Ziyi Yang. Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[29] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[30] Yi-Hua Huang, Yang-Tine Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse- controlled gaussian splatting for editable dynamic seenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[31] Agelos Kratimenos, Jiahui Lei, and Kostas Daniilidis. Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.
[32] Yiqing Liang, Numair Khan, Zhengqin Li. Thu Nguyen- Phuoo, Douglas Lanman, James Tompkin, and Lei Xiao. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis. arXiv preprint arXiv:2312.11458, 2023,
12

---

[33] Zhicheng Lu, Xiang Guo, Le Hui, Tianrui Chen, Min Yang, Xiao Tang. Feng Zhu, and Yuchao Dai. 3d geometry-aware deformable gaussian splatting for dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recoognition (CVPR), 2024.
[34] Jeongmin Bae, Seoha Kim, Youngsik Yun, Hahyun Lee, Gun Bang. and Youngjung Uh. Per Per-gaussian embedding- based deformation for deformable 3d gaussian splatting. Proceedings of the European Conference on Computer Vision (ECCV), 2024.
[35] Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the-fly fly training of 3d gaussians for efficient streaming of photo-realister free- free-viewpoint videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[36] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multircsolution hash encoding, ACM transactions on graphics (TOG), 2022.
[37] Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flowe 4d reconstruction with dynamic 3d gaussian particle. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[38] Liangchen Song, Anpei Chen. Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics (TVCG), 2023.
[39] Liao Wang. Qiang Hu, Qihan He, Ziyu Wang, Jingyi Yu, Tinne Tuytelaars, Lan Xu, and Minye Wu. Neural residual radiance fields for streamably free-viewpoint vidcos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[40] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 2014.
[41] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence (AAAT), 2020.
[42] Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations (ICLR), 2018.
[43] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[44] Tianlong Chen, Peihao Wang. Zhiwen Fan, and Zhangyang Wang. Aug-nerf: Training stronger neural radiance fields with triple-tevel eysiccally-roundde augmentations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[45] Joo Chan Lee, Daniel Rho, Xiangyu Sun, Jong Hwan Ko, and Eunbyung Park. Compact 3d gaussian representation for radiance field. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
[46] Johannes Lutz Schönherger and Jan-Michael Frahm. Structure-frommmoonee motion revisited. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[47] Jad Chauraa Feras Dayoub, and Niko Sünderhauf. Particlenerf: particle-bass encoding for online neural radiance fields. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), 2024.
[48] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
13

---

Appendix
This appendix provides additional material to supplement the main text. We will first introduce more implementation details in Sec. A Then we provide some additional experimental results in Sec. B
A More Implementation Details
Our code is primarily based on the open-source codes of 3DGS [4] and 4D-GS (29]. We keep the 4D-GS setting that does not reset opacity in the initial 3DGS learning because we observed that resetting opacity leads to the removal of larger Gaussians, which helpful when the number of Gaussians already significantly reduced by our noise perturbation smoothing mechanism. The hyperparameter Anoiee was empirically selected from the range of (0.001, 0.01, 0.13. Among these values, 0.01 consistently performed the best across multiple scenes, hence set it to 0.01 in all experiments. After applying the learned motion to 3DGS, we determine the Gaussians to cloned based on the accumulated gradients during motion learning, which is similar to standard 3DGS. The cloned Gaussians undergo alternating optimization and densification, with densification occurring every 40 steps. We found that this parameter is not sensitive and can be freely set between 20 and 50. The removal of low-opacity Gaussians happens before learning the next frame, so these Gaussians still contribute to the representation of the current frame.
At the time of conducting this work, the coderreleasedd by the authors 3DGStream still demonstrated instability and encountered issues during execution. Therefore, we reported the experimental results based on the code we implemented. Although, we attempted to faithfully reproduce the 3DGStream as described in its paper, inconsistencies remain in our implementation and experimental setup. The main difference lies in the dataset processing. We employ the original dataset following 4D-GS code, whereas 3DGStream utilizes processed datasets with modified resolutions and camera poses, resulting in about 50,000 fewer pixels per view image of N3DV [I] dataset. The spherical harmonics (SH) order is set to 1 as per the paper, but do not perform SH rotation because it increases the computational overhead without noticeable performance gains. Additionally, 3DGStream requires scene boundary being manually setup, whereas automatically compute them after removing some outlier Gaussians. These disparities lead to marginally lower metrics compared to those reported in the 3DGStream paper, but results align closely with other offline methods using the same dataset.
B Additional Experimental Results
3DGStream
3DGStream HiCoM (ours)
HiCoM (ours)
Number Gaussians aans (K)
493
32.16
32.36
PSNR (dB)
321
27.91
273
27.41
26.75
227
196
Coffee Martini
Flame Steak
Discussion
Coffee Martini
Flame Steak
Discussion
Figure 4: Comparison of initial 3DGS. 3DGStream utlizes the standard 3DGS training, HiCoM additionally incorporates noise perturbation.
14

---

Table 6: Per-scene quantitative results the N3DV dataset. The offiine and online methods separated into upper and lower sections, respectively. The pramframe metrics "Storage" and "Train" averaged over 300 frames, with those for offline methods 20 38
26
29
48 34 estimated based
on data reported in papers 22 48 34].
The data for Dynamic 3DGS are referenced from [48].
Coffee Martini
Cook Spinach
Cut Beef
Method
PSNR
Storage (MB
Train (s ()
PSNR (dB f)
Storage
Train
PSNR (dB f)
Storage (MB {)
Train
(dB f)
(MB .)
(s 4)
(s ()
KPlanes 20
29.99
1,0
14.4
32.60
1.0
14.4
31.82
1.0
14.4
NeRFPlayer
38
31.53
18.4
48.0
30.56
18.4
48.0
29.35
18.4
48.0
4DGS |26
28.33
29.0
76.0
32.93
29.0
76.0
33.85
29.0
76.0
4D-GS [29
27.34
0.3
5.3
32.46
0.3
5.3
32.90
0.3
5.3
Spacetime-GS
48
28.61
0.7
28.0
33.18
0.7
28.0
33.52
0.7
28.0
E-D3DGS 34
29.33
0.5
43.0
33.19
0.5
43.0
33.25
0.5
43.0
Dynamic 3DGS
25
26.49
9.2
560
32.97
9.2
560
30.72
9.2
560
3DGStream 35
26.73
7.6
9.26
31.38
7.6
7.79
31.36
7.6
7.73
HiCoM (ours)
28.04
0.8
7.02
32.45
0.6
6.47
32.72
0.6
6.56
HiCoM-P4 (ours)
28.19
0.8
1.76
32.34
0.6
1.62
32.36
0.6
1.64
Flame Salmon
Flame Steak
Sear Steak
PSNR
Storage
Train
PSNR
Storage
Train
PSNR
Storage
Train
(dB ))
(MB ()
(s 4)
(dB f)
(MB ()
(s J)
(dB ft
(MB ()
(s J)
KPlanes 20
30.44
1.0
14.4
32.38
1.0
14.4
32.52
1.0
14.4
NeRFPlayer
38
31.65
18.4
48.0
31.93
18.4
48.0
29.12
18.4
48.0
4DGS 226
29.38
29.0
76.0
34.03
29.0
76.0
33.51
29.0
76.0
4D-GS 29
29.20
0.3
5.3
32.51
0.3
5.3
32.49
0.3
5.3
Spacetime-Gis GS
48
29.48
0.7
28.0
33.64
0.7
28.0
33.89
0.7
28.0
E-D3DGS 34
29.72
0.5
43.0
33.55
0.5
43.0
33.55
0.5
43.0
Dynamic 3DGS
25
26.92
9.2
560
33.24
9.2
560
33.68
9.2
560
3DGStream (35
27.45
7.6
9.14
31.56
7.6
7.77
32.44
7.6
7.72
HiCoM (ours)
28.37
0.9
7.06
32.87
0.6
6.62
32.57
0.6
6.40
HiCoM-P4 (ours)
28.54
0.9
1.77
32.13
0.6
1.66
32.42
0.6
1.60
Table 7: Per-scene quantitative results on the Meet Room dataset. Rows marked with indicate experiments conducted on the undistorted version of the dataset.
Discussion
Trimming
VR Headset
Method
PSNR
Storage
Train
PSNR
Storage
Train
PSNR
Storage
Train
(dB
(MB J)
(s
(dB f)
(MB J)
(s 4)
(dB f)
(MB ()
(s ()
3DGStream [35
25.94
4.0
4.78
26.18
4.0
4.69
25.75
4.0
4.70
HiCoM (ours)
26.69
0.6
3.89
26.99
0.4
3.85
26.51
0.3
3,88
HiCoM-P4 (ours)
26.39
0.6
0.97
26.38
0.4
0.96
25.98
0.3
0.97
3DGStream (35
30.06
4.0
4.40
28.82
82.82
4.0
4.34
29.26
4.0
4.39
# (ours)
29.61
0.4
3.65
29.64
0.4
3.65
29.86
0.3
3.76
B.1
Inital 3DGS Comparison
The initial 3DGS representation is crucial for the overall performance of subsequent online learning. In Figure 4 we compare 3DGStream that utilizes the standard 3DGS training with our HiCoM that additionally incorporates noise perturbation. It is evident that our noise perturbation successfully reduces the number 3D Gaussians in the initial representation, while slightly enhancing the PSNR metric. This subtle improvement highlights the superiority of our method. However, seen in ablation experiments detailed the main text, when our method employs the same standard 3DGS training for the initial frame, it only achieves PSNR comparable to that of 3DGStream. This indicates the importance of fully leveraging the benefits latest advancements within the 3DGS domain to further improve the initial 3DGS representation.
15

---

Table 8: Performance stability three with the same random seed. Results reported as the mean and standard deviation from three runs on the N3DV dataset with seed 0.
Coffee Martini
Cook Spinach
Cut Beef
Flame
Flame Steak
Sear Steak
Method
Mean
Salmon
3DGStream 35
26.73 20.05
31.38 20.44
31.36 10.15
27.45 2001
31.56 20.50
30.15 20.13 32.57 20.2 31.17 20.25
32.44 2002
HiCoM (ours)
28.04 20.29
32.45 20.13
32.72 20.25
28.37 20.18
32.87 20.12
Table 9: Per-scene quantitative results the PanopticSports dataset. The asterisk (*) indicates that our method employed the same color trick as Dynamic 3DGS.
Method
Juggle
Boxes
Softball
Tennis
Football
Basketball
Mean
Dynamic 3DGS 25
29.48
29.46
28.43
28.11
28.49
28.22
28.70
3DGStream 35
24.68
23.69
22.44
23.09
24.97
20.01
23.15
HiCoM (ours)
27.82
27.37
28.06
27.80
27.80
27.29
27.69
HiCoM (ours)
29.55
28.96
29.08
29.02
29.25
28.60
29.08
B.2
Per-Scene Quantitative Results
We present the quantitative results for each scene from two datasets respectively in Tables to offine
provide a more detailed comparison. We also provide the results of several methods on N3DV dataset in Table as a reference. Among these, Coffee Martini and Flame Salmon are particularly complex scenes that require greater number of 3D Gaussians and motion parameters our HiCoM framework, resulting relatively higher storage requirements. However, they still require significantly less storage than training with the 3DGStream method.
Table 10: Quantitative results on the Particle-NeRF dataset.
Method
Particle-NeRF (25
Dynamic3DGS 25
HiCoM (ours)
PSNR (dB 1)
27.47
39.49
31.05
In addition, we conduct experiments on the PanopticSports dataset, which involves large-scale motions, and compare with the state-of-the-re Dynamic3DGS method, shown in Table 9 Notably, Dynamic3DGS heavily relies on segmentation masks, while our HiCoM does not require such additional supervision. To ensure fair comparison, we set the SH to Additionally, we observe that Dynamic3DGGS employs trick to mitigate potential multi-view color inconsistencies by learning scale and offset parameter for each color channel of each camera in the training views. To enhance performance, we incorporated this same color trick into HiCoM. The results demonstrate that our method significantly outperforms 3DGStream in terms of PSNR. While slightly behind Dynamic3DGS initially, with the same color trick applied, HiCoM clearly surpasses Dynamic3DGS, highlighting its superiority in handling large-scale motion scenarios.
All our experiments were conducted three times using the same random seed. In Table 8 we present the standard deviation of the three runs on the N3DV dataset, showing the fluctuation in PSNR metric, which is relatively minor. We also tried using different random seeds, and the performance fluctuation was still within the range shown in Table
Furthermore, we extended our evaluation to the Particle-NeRF 47 dataset, which features synthetic scenarios with uniform and periodic motions. The Table 10 shows that HiCoM outperforms Particle-NeRF, demonstrating its ability to effectively manage these types of motion.
B.3 Additional Qualitative Results
We provide additional qualitative results for other scenes in the N3DV dataset, three scenes from the Meet Room dataset and six scenes of PanopticSports dataset, as shown Figures 5 6 and 7
respectively. These results demonstrate the robustness and versatility of our HiCoM framework in capturing and representing dynamic scenes.
16

---

Gurs
(a) Cook Spinach
5008
(b) Cut Beef
3008imam
(c) Flame Salmon
3DGBtraam
(d) Flame Steak
10GStream
(e) Sear Steak
Figure 5: Qualitative results of additional scenes from N3DV dataset. Frames shown the 1st t1 121st 181st, 24 and th from the test video. Red boxes highlight areas with significant temporal motions.
17

---

Ours JoGStran GT
(a) Discussion
Ours 3DGStream GT
(b) Trimming
Ours 3DGStream GT
(c) VR Headset
Figure 6: Qualitative results of scenes from Meet Room dataset. Frames shown are the 1st t1 12 181st, 241 and th from the test video. Red boxes highlight areas with significant temporal motions.
18

---

3005emmn
(a) Juggle
30050eum
(b) Boxes
GT
3005emun
(c) Softball
(d) Tennis
Oan
(c) Football
20d5eran
Oat
Basketball
Figure 7: Visualization of PanopticSports dataset. Shown are frames 1, 31, 61, 91, 121, and 150 from the 0th test video.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
See conclusion section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
20

---

Justification:
All the theorems, formulas in the paper are numbered and cross-referenced.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification:
We will open source our code replicate our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

reproducible, they Answer: [Yes]
Justification:
The paper provides open access to both the data and the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
public/guidess www.eeoeaaiiiioooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ //nips httpcc/cpplllgggiuueee /ttpg/ddde///eebbbriiimmmm
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details of training and testing are provided in A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We ran all our experiments 3 times and reported the mean metrics. However, since some the results are directly cited from other papers, we did not include error bars in the main text to maintain consistency. Nevertheless, we reported the standard deviation of the available results in the appendix, please see Table X
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
22

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The details of compute resources are provided in 5.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpp:://euurpss... http///www.....oooooic///hhsaaaa
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legal to point out that an improvement in the quality of generative models could be used generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators or original owners of assets (e.g., code, data, models), used the paper, are properly credited, and the license and terms of explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "hyA6YxGZLC", "reviewer": "Reviewer_oHau", "summary": "This paper deals with the online reconstruction of dynamic scenes from multi view video input. The authors build their method on the popular 3D Gaussian Splatting technique. To tackle this setting, they propose 1. A perturbation smoothing that introduces small perturbations to the 3D positions of the Gaussians to improve the initial reconstruction learned on the first timestep, 2. A hierarchical coherent motion strategy that divides 3D Gaussians into a hierarchical voxel grid and assigns motion coefficients (3D translation and 4D Quaternion rotation) to each voxel, where the motion of each 3D Gaussian is the sum of all voxel-level motions it is associated with.", "questions": "It would be great if the authors could clarify my questions raised in the weaknesses section. Additionally, Equation 6 suggests that the noise is drawn from a normal distribution with variance 1, which seems quite large. Is this an error or is the perturbation actually that large? Are the reconstructions in metric scale?", "limitations": "The paper discusses the method limitations sufficiently.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The perturbation smoothing strategy is highly effective given the results in Tab.2 and Fig. 4\n- The method shows state-of-the-art performance on both N3DV and Meet Room\n- The method is simple and (partially) effective, the paper is well written", "weaknesses": "- The motion levels seem not to significantly affect the performance (cf. Tab. 5), which begs the question of whether the HiCoM strategy is critical for performance. Also, scene motion is piecewise constant, but has sharp boundaries. What if both dynamic foreground and static background fall into the same voxel?\n- In L274, the authors mention that mean results over 3 runs are used in the experiments. In L697, the authors mention they do not include error bars. What’s the reason for this?\n- Finally, I am curious about why online reconstruction is needed if both multi-view video and precomputed input poses are needed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the strengths of our work, particularly the perturbation smoothing strategy, the exceptional performance on N3DV and Meet Room datasets, and the quality of our writing. In the following, we address each of your comments and questions in detail.\n\n### 1. The impact of motion levels on performance.\n\nWe need to clarify the meaning of **motion levels** in Table 5. Our method uses three motion levels by default (the 3$^{rd}$ row), which we refer to as \"fine,\" \"medium,\" and \"coarse\", respectively. When motion levels is set to 1 (the 1$^{st}$ row), only the \"fine\" level is used. When set to 2 (the 2$^{nd}$ row), both \"fine\" and \"medium\" levels are applied. As the table shows, using only the \"fine\" level already achieves good performance. Adding the \"medium\" and \"coarse\" levels provides additional performance gains, *e.g.*, the PSNR of Coffee Martini and  Discussion scenes respectively boosts 0.23 dB and 0.59 dB, but the improvements diminish as the number of levels increases. This indicates that our HiCoM is effective and a few motion levels are sufficient to achieve good results. We will include this clarification in the revised manuscript and add experimental results for single-level motion (present in the table below) to provide a more comprehensive view.\n\n\n| Motion Level                | Coffee Martini | Flame Steak | Discussion |\n|-----------------------------|----------------|-------------|------------|\n| coarse                      | 26.67          | 30.28       | 22.89      |\n| medium                      | 27.24          | 32.14       | 24.91      |\n| fine                        | 27.79          | 32.76       | 26.10      |\n| fine + medium               | 27.94          | **32.88**   | 26.61      |\n| fine + medium + coarse      | **28.04**      | 32.87       | **26.69**  |\n\n\n### 2. The dynamic foreground and static background fall into the same voxel.\n\nYour observation is indeed insightful and highlights an important consideration in complex scenes. However, Gaussian primitives are not points but ellipsoids, representing small regions in space. We observed that Gaussian primitives for moving objects are generally smaller and densely packed, whereas those for background regions are usually larger and sparsely distributed. \nAs long as the finest motion level's coverage area is smaller than the radius of background Gaussian primitives, this issue can be substantially mitigated. \nAdditionally, after the motion learning, we will add some new Gaussian primitives, which can further help in correcting inadequately reconstructed regions. Future work could investigate methods to distinguish dynamic objects from the static background, such as filtering based on the size of Gaussian primitive.\n\n### 3. The absence of error bars.\n\nWe performed each experiment three times using the default random seed, but result variations were minimal. Given the number of metrics we reported, there was limited space in the tables. Additionally, some of the data were directly referenced from other papers that did not include standard deviations, so we omitted them for consistency.\nConsidering your suggestion, we will include all avaliable standard deviations for PSNR metrics (as shown in the table below) in Tables 6 and 7 in our revised manuscript. \n\n\n\n| Method              | Coffee Martini      | Cook Spinach        | Cut Beef           | Flame Salmon       | Flame Steak        | Sear Steak         | Mean              |\n|---------------------|---------------------|---------------------|--------------------|--------------------|--------------------|--------------------|--------------------|\n| 3DGStream           | 26.73~±~0.05       | 31.38~±~0.04       | 31.36~±~0.15       | 27.45~±~0.01       | 31.56~±~0.50       | 32.44~±~0.02       | 30.15~±~0.13         |\n| **HiCoM (ours)**    | 28.04~±~0.29       | 32.45~±~0.13       | 32.72~±~0.25       | 28.37~±~0.18       | 32.87~±~0.12       | 32.57~±~0.52       | 31.17~±~0.25         |\n\n\n\n\n\n### 4. The necessity of online reconstruction.\n\nMost methods for dynamic scene reconstruction work in an offline manner, but these methods may suffer from low storage efficiency, high resource demands, and insufficient real-time response for long-duration or real-time interactive dynamic scenes. For example, for a dynamic scene lasting 2 hours, methods using neural networks to model scene motion may require very large neural networks, substantial GPU memory, and may have slow convergence rates. The goal of online reconstruction is to mitigate these issues while achieving competitive reconstruction quality. Online reconstruction does not need to wait for all video capture to complete, and the cost for pose estimation is negligible. Real-time high-fidelity dynamic scene reconstruction would be valuable for modern applications such as AR/VR, free-viewpoint video streaming, and online meeting. \n\n### 5. The perturbation noise present in Equation 6.\n\nWe would like to clarify that Equation 6 also includes a coefficient $\\lambda_{noise}$, which adjusts the noise intensity. This coefficient allows us to control the perturbation magnitude, ensuring it is appropriate for the specific context of the reconstruction.\n\nThank you once again for your valuable feedback and insightful comments. We hope that our responses have addressed your concerns and clarified the points raised in your review. If you have any further questions or require additional clarifications, please feel free to leave a comment. We remain committed to providing any further information you may need.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UowugjFiIG", "reviewer": "Reviewer_QKxo", "summary": "This paper proposes a novel framework for online reconstruction of dynamic scenes based on 3DGS. The main contribution lies in its Hierarchical Coherent Motion, which gives a more compact motion representation. Results show its effectiveness and rendering improvement.", "questions": "1. Since the work focus on the online reconstruction of streamable videos, the efficiency is very important. I am wondering whether the cost of initialization from SfM (Colmap) should be considered.", "limitations": "1. No ethic issues.\n\n2. The authors kindly point out their limitations in the paper. The initialization of 3DGS in the first frame is significant and the paper is not fully addressed. It would be kind for authors to show how the initial frame learning influence the whole pipeline currently, as they claim it is not fully addressed.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The method is straightforward and easy to follow. \n\n2. The pertubation smoothing strategy is simple but effective. I think it maybe also work in sparse view rendering of 3DGS.\n\n3. The ablation is adequate.", "weaknesses": "1. Continual Refinement seems similar to the strategy in 3DGStream. It would be better to clarify the difference. \n\n2. Most parts of the scene is static, uniformally dividing the scenes to model the motion seems redudant.\n\n3. Embedding the motion in space alone may have limited modelling ability, especially when an object undergoes complex movements within a small region, such as intricate transformations back and forth. It might be better to incorporate the temporal dimension.\n\n4. Although the method is tailored for streamable videos, it would be better to compare with other dynamic 3DGS methods, such as 4DGaussians, GaussianFlow, spacetime Gaussian. The previous work 3DGStream also includes the comparison with offline or non-streamable methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate your positive feedback on the clarity and straightforward nature of our method. We are particularly encouraged by your recognition of the effectiveness of our perturbation smoothing strategy, as well as your appreciation of the simplicity. Your acknowledgment of the thoroughness of our ablation study is also highly valued. We have carefully reviewed your questions and concerns and address them in detail below.\n\n### 1. The difference between our continual refinement and 3DGStream.\n\nOur continual refinement may appear similar to the second stage of 3DGStream in form, but it fundamentally differs in approach. 3DGStream continuously applies learned motion to the Gaussian representation derived from the first frame, with newly added Gaussian primitives only used for rendering the current frame and not passed to subsequent frames. In contrast, our method merges new Gaussian primitives into the initial Gaussian representation, then removes the same number of less important Gaussian primitives based on importance metrics such as the opacity we used in this work or other indicators. If an object does not exist in the first frame, 3DGStream adds Gaussian primitives in every subsequent frame to fit the object, while our method can pass these Gaussian primitives to the next frame, eliminating the need to refit the object. \nWe believe our design is superior and will clearly state this difference in the revised manuscript to reduce any potential confusion for readers.\n\n### 2. The uniformly dividing the scene seems redudant.\n\nThank you for highlighting this issue. However, our method demonstrates that uniformly dividing the scene is simple and effective. We agree that adaptively dividing the scene based on the granularity of motion could further improve the motion representation: regions with minimal motion could be coarsely divided, while areas with complex motion could be finely partitioned. Based on your suggestion, we will explore more effective scene division methods in future work.\n\n### 3. Incorporate the temporal dimension into motion.\n\nOur online reconstruction framework focuses on representing and learning changes between adjacent frames in general real-world scenes, which keeps our method relatively simple but has proven effective in experiments. We agree that embedding the motion in space alone may have limited modeling ability, especially for complex movements such as intricate transformations within a small region. We acknowledge that some existing works specifically address such periodic motion patterns [1, 2]. Future work could enhance our framework by incorporating temporal information from observed frames to construct more comprehensive motion patterns.\n\n\n### 4. Comparison with offline reconstruction methods.\n\nWe included data reported by state-of-the-art offline dynamic scene reconstruction methods in the supplementary materials. Specifically, Table 8 in the Appendix shows that our method is comparable in reconstruction quality  across nine scenes from two datasets. We will highlight these comparisons more prominently and incorporate additional recent works in our revised manuscript.\n\n### 5. The cost of initialization from SfM.\n\nThe cost associated with SfM does not significantly affect the online reconstruction workflow. In dynamic scene reconstruction, the viewpoints are typically sparse, meaning that SfM needs to be applied to only a limited number of views from the initial frame. Generating the initial point cloud usually takes only a few minutes. In practical scenarios, camera positions for video capture are often pre-arranged, allowing the majority of the SfM work to be completed before the reconstruction process begins. \n\n### 6. The influence of the initial frame learning.\n\nOur online framework reconstructs the scene’s fundamental geometry and appearance through initial frame learning, and assumes that most objects in the dynamic scene will not undergo significant appearance changes. Subsequent learning focuses on representing and learning the motion of objects from the initial frame, adding a few new Gaussian primitives to handle previously unseen content and to improve motion learning. In contrast, if the initial scene's geometry and appearance are poorly reconstructed, it increases the burden on subsequent learning, leading to slower convergence and degraded reconstruction quality, which can accumulate over time. \n\nWe are grateful for your constructive feedback, which has been instrumental in enhancing the quality of our manuscript. If our clarifications have addressed your concerns, we kindly ask you to reconsider your evaluation of our manuscript. Should you have any further questions or require additional clarifications, we are happy to provide further details. Thank you once again for your time and consideration.\n\n[1] YutaoFeng,XiangFeng,YintongShang,YingJiang,ChangYu,ZeshunZong,TianjiaShao,HongzhiWu, Kun Zhou, Chenfanfu Jiang, et al. Gaussian splashing: Dynamic fluid synthesis with gaussian splatting. arXiv preprint arXiv:2401.15318, 2024.\n\n[2] Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1DTujg9GgI", "reviewer": "Reviewer_q4mz", "summary": "This paper proposes a novel online training framework for multi-view dynamic scenes. A HiCoM framework is introduced for online learning of dynamic scenes. To initialize a better 3D Gaussian, the authors propose adding noise because it may overfit in the forward-facing multiview setups. Modeling motion by a sparse and explicit structure looks very good and also preserves low storage consumption. Experiments also demonstrate that the results are better than previous methods. e.g. 3DGStream.", "questions": "1. How can you integrate the 3D Gaussians into the initial 3DGS? If you add 3D Gaussians at t(t>0) and how to compute its position at t=0.\n2. The reviewer expects to see the results on the Panopticsports dataset. If your code is based on 4D-GS, you can run the experiments directly.\n\nOverall, there are some written problems in the paper and there are lack of visualizations and discussions. The main concern of the reviewer is whether the HiCoM can model large motions such as Panopticsports. If the author can provide positive results on Panopticsports and more discussions, the reviewer may recommend accepting this paper.", "limitations": "The limitations are shown in the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A trick about adding noise to 3D Gaussians seems to contribute to forward-facing multiview setups.\n2. A hierarchical motion framework is introduced to model the motion in the multiview setups, which seems to work well.\n3. Managing deformation by a global and simple data structure is proven to save memory.", "weaknesses": "1. It seems that hierarchical coherent motion shows the potential to model large motions. So experiments in the PanopticSports dataset (used in [25]) should be included.\n2. Comparison/discussions with Particle-NeRF(Abou-Chakra et al, WACV24) should be included.\n3. You can add more comparison figures about ablation studies to demonstrate the results. The reviewer recommends authors submit a video to support a visualization, which is very important in 3D Vision.\n4. The introduction section is too long. The reviewer recommends the authors try to express themselves easily and efficiently. Only 3-4 paragraphs (70 lines) are enough to introduce your work. The writing could be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for positive assessment of adding noise to 3D Gaussian representation learning, the effectiveness of our hierarchical coherent motion, and the simple deformation data structure to save memory. We now address the main concerns you have raised as follows.\n\n### 1. The performance of HiCoM in reconstructing scenes with large motions.\n\nAs per your suggestion, we conducted experiments on the PanopticSports dataset. We set SH to 1 for fair comparison, and present results in the following table. Our HiCoM significantly surpasses the competitor 3DGStream in terms of PSNR, but slightly lower than Dynamic3DGS [3]. This is primarily because Dynamic3DGS utilizes segmentation masks as additional supervision, substantially boosts performance. According to Dynamic3DGS paper, removing segmentation masks leads to a significant performance drop (over 5dB), resulting in lower performance compared to our method.\nIn the attached PDF, we present rendering results of all six scenes from one of the test viewpoints. The 3DGStream struggles with large movements; people and objects in the scene generally remain near their initial positions and gradually become invisible. We believe this is due to the limitations of its grid-based hash encoding and MLP, which cannot effectively handle high-frequency temporal motion information. In contrast, our HiCoM captures large movements of people and objects in the scene well, demonstrates sufficient flexibility, but there is still considerable room for improvement in reconstruction quality, especially for smaller moving objects. \n\n\n| Method          | Juggle | Boxes | Softball | Tennis | Football | Basketball | Mean   |\n|-----------|--------|-------|----------|--------|----------|------------|--------|\n| Dyanmic3DGS      | 29.48  | 29.46 | 28.43    | 28.11  | 28.49    | 28.22      | 28.70  |\n| Dyanmic3DGS (w/o mask) | 24.14  | -     | -        | -      | -        | -          | -      |\n| 3DGStream          | 24.68  | 23.69 | 22.44    | 23.09  | 24.97    | 20.01      | 23.15  |\n| **HiCoM (ours)**     | 27.82  | 27.37 | 28.06    | 27.80  | 27.80    | 27.29      | 27.69  |\n\n\n### 2. Comparison with Particle-NeRF.\n\nAfter carefully reviewing Particle-NeRF paper, we found it to be an excellent piece of work. It uses discrete particles for scene representation instead of the uniform grids used in previous NeRF work (InstantNGP). By optimizing particle positions and features through gradients, it achieves a similar effect to 3DGS, with the main difference being in the rendering pipeline. Particle-NeRF models motion through particle movement, focusing more on periodic, regular, and micro-level object motions. In contrast, our method targets general, scene-level dynamic changes.  As we know, some Gaussian Splatting works  [1, 2] also specialize in periodic and regular object motions. We believe Particle-NeRF might not be as effective in modeling the dynamic changes that our method addresses. \n\nDespite this, we conducted experiments on the Particle-NeRF dataset, including five scenes, each with 40 viewpoints. We used 30 for training and the remaining 10 for testing. Please note that the Particle-NeRF paper mentions using only 20 training viewpoints and 10 testing viewpoints. As shown in the table below, our method achieves good reconstruction quality. Unfortunately, up to the time of submitting this response, we were unable to successfully run their open-source code, preventing us from conducting a more detailed comparison on the real-world dynamic scenes used in our paper.\n\n| Method  | PSNR  |\n|----|---|\n| Particle-NeRF       | 27.47 |\n| Dynamic3DGS         | 39.49 |\n| **HiCom (ours)**    | 31.05 |\n\n### 3. Add more comparison figures and visualization videos.\n\nDue to space limitations in the main manuscript, we have provided visual comparisons of all other scenes with the 3DGStream method in the supplementary materials. We have created videos to support our visualizations and are very willing to provide them. However, due to the review guidelines, we are unable to include external links here. We will consider including the comparison figures from the ablation studies in the revised manuscript and attach relevant visualization videos.\n\n### 4. The introduction section is too long.\n\nWe appreciate the reviewer's feedback on the length and clarity of the introduction section. Our introduction in the manuscript spans from line 23 to line 95, totaling 72 lines, which is in line with the 70 lines recommended by the reviewer. Nevertheless, we will further refine this section in the final version to ensure it is as concise and precise as possible.\n\n### 5. Integrate the 3D Gaussians into the initial 3DGS.\n\nIn our online reconstruction framework, we add some new Gaussian primitives to benefit the reconstruction and rendering of the current frame, but does not require computing their positions at $t=0$. Our primary objective is dynamic scene reconstruction and rendering, rather than object tracking. \n\nWe sincerely appreciate your detailed and constructive feedback. Your insights have been invaluable in refining our work, and we are committed to addressing your concerns to improve the overall quality of our paper. Thank you once again for your time and effort in reviewing our manuscript. We look forward to any further comments you may have.\n\n[1] Yutao Feng,Xiang Feng,Yintong Shang, Ying Jiang, Chang Yu, Zeshun Zong,Tianjia Shao,Hongzhi Wu, Kun Zhou, Chenfanfu Jiang, et al. Gaussian splashing: Dynamic fluid synthesis with gaussian splatting. arXiv preprint arXiv:2401.15318, 2024.\n\n[2] Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. In Proceedings of the European Conference on Computer Vision (ECCV), 2024.\n\n[3] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In International Conference on 3D Vision (3DV), 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all the reviewers for their thorough evaluation and valuable feedback on our manuscript. We are pleased that all reviewers recognized strengths of our proposed method and find it simple and effective. Reviewers 1 (oHau), 2 (QKxo) highlighted the effectiveness of our perturbation smoothing strategy, while Reviewers 2, 3 (q4mz) appreciated our hierarchical coherent motion framework. Reviewer 1 specifically noted the state-of-the-art performance of our method on N3DV and Meet Room datasets, and Reviewer 3 praised the memory efficiency of our motion modeling approach. We deeply appreciate these positive comments and have provided detailed responses to each reviewer's suggestions and concerns individually. The attached PDF includes visualization results on the PanopticSports dataset, which features large motion scenes, as suggested by Reviewer 3.

Meta-review: All reviewers agree that the submission proposes a simple and effective solution for radiance field reconstruction from multi-view streaming videos. The authors efficiently addressed the reviewers' concerns in the rebuttal and during the intense discussion. AC recommends acceptance. Please carefully address the concerns raised by the reviewers in the camera-ready version. For example, include the new comparison table with dynamic 3DGS after incorporating the image embedding trick, an ablation study on Motion Level, additional comparison figures and visualization videos, and the release of the code. We look forward to seeing these contributions from the open-source community.
Decision: Accept (poster)