HiCoM: Hierarchical Coherent Motion for Dynamic Streamable Scenes with 3D Gaussian Splatting

Authors: qiankun  gao
jiarui  meng
chengxiang  wen
jie  chen
jian  zhang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-26 06:19:15

Abstract: The online reconstruction of dynamic scenes from multi-view streaming videos faces significant challenges in training, rendering and storage efficiency. Harnessing superior learning speed and real-time rendering capabilities, 3D Gaussian Splatting (3DGS) has recently demonstrated considerable potential in this field. However, 3DGS can be inefficient in terms of storage and prone to overfitting by excessively growing Gaussians, particularly with limited views. This paper proposes an efficient framework, dubbed HiCoM, with three key components. First, we construct a compact and robust initial 3DGS representation using a perturbation smoothing strategy. Next, we introduce a Hierarchical Coherent Motion mechanism that leverages the inherent non-uniform distribution and local consistency of 3D Gaussians to swiftly and accurately learn motions across frames. Finally, we continually refine the 3DGS with additional Gaussians, which are later merged into the initial 3DGS to maintain consistency with the evolving scene. To preserve a compact representation, an equivalent number of low-opacity Gaussians that minimally impact the representation are removed before processing subsequent frames. Extensive experiments conducted on two widely used datasets show that our framework improves learning efficiency of the state-of-the-art methods by about 20% and reduces the data storage by 85%, achieving competitive free-viewpoint video synthesis quality but with higher robustness and stability. Moreover, by parallel learning multiple frames simultaneously, our HiCoM decreases the average training wall time to <2 seconds per frame with negligible performance degradation, substantially boosting real-world applicability and responsiveness.

Main paper: 
 introduction: 
 The online reconstruction of dynamic scene from multi-view video streams is essential for advancing applications such as real-time free-viewpoint video (FVV) and virtual reality (VR), which are revolutionizing entertainment, education, and industry by providing immersive and interactive experiences. However, achieving high fidelity streamable dynamic scene poses significant challenges in training time, rendering speed, data storage and transmission efficiency.Traditional methods for modeling and representing dynamic scenes, such as surface estimation [3] , multi-sphere imaging [4] , and depth mapping [5, 6] via multi-view stereo technologies, struggle with the complex geometries and varied appearances of real-world scenarios. These techniques also face limitations in capturing the detailed and dynamic nature of practical environments.Neural Radiance Fields (NeRFs) [7, 8, 9, 10, 11, 12, 13, 14, 15, 16] have made significant breakthroughs in 3D reconstruction and novel view synthesis by mapping spatial coordinates and viewing directions to color and density using a neural network. Dynamic NeRFs [1, 17, 18, 19, 20, 21, 22, 23] integrate temporal components to capture scene changes over time. However, the high computational demands of NeRF's volume rendering framework limit their feasibility in real-time applications. Acknowledging NeRF's limitations, researchers have introduced 3D Gaussian Splatting (3DGS) [24] , an innovative method for fast 3D reconstruction and real-time rendering. In contrust to NeRF, 3DGS explicitly employs anisotropic 3D Gaussians as primitive elements to represent 3D scenes, then rasterizes these Gaussians to render images from specific viewpoints, bypassing the complex and slow volume rendering pipeline. This not only speeds up the rendering process but also enhances the quality of the synthesized views. In a similar vein to NeRF, 3DGS has been adapted for dynamic scene reconstruction: several Dynamic Gaussian Splatting works [25, 26, 27] directly expand attributes of Gaussian primitives; other efforts [28, 29, 30, 31, 32, 33, 34] focus on decoupling the dynamic scene into a base scene representation in canonical space and time-varying motion fields.Harnessing its efficiency, 3DGS naturally supports online learning of dynamic scenes. A recent development, 3DGStream [35] , introduces a pioneering framework that uses a Neural Transformation Cache derived from InstantNGP [36] to model the scene changes from previous frame to current frame, significantly reducing training time and storage requirements. However, this frame-by-frame online learning pipeline heavily relies on the quality of the initial 3DGS representation. Given that dynamic scenes are generally captured with a limited number of cameras, 3DGS can be prone to instability and overfitting when faced with sparse views. Furthermore, the number of Gaussian primitives in the initial 3DGS representation impacts the learning efficiency of subsequent frames. Despite employing multi-resolution hash encoding and lightweight MLP to alleviate inefficiency and speed up convergence, implicitly modeling the motion field still cannot fully capture critical aspects of the explicit and discrete nature of 3DGS. Although 3DGStream introduces new Gaussians to adapt to the appearance of new objects, these added Gaussians are not carried over to subsequent frames to maintain 3DGS compact. As real-world scenes evolve gradually, the discrepancies from the initial scene accumulate, necessitating more Gaussians to accurately capture these changes.In this paper, we introduce the Hierarchical Coherent Motion (HiCoM) framework, a novel approach designed to enhance the efficiency and stability of streamable dynamic scene online reconstruction. Our HiCoM framework begins with the learning of a compact and robust initial 3DGS representation through a perturbation smoothing strategy. This ensures a reduced number of Gaussians, alleviates overfitting and establishes a foundation for consistent quality across frames. Then, we leverage the inherent non-uniform distribution and local consistency of 3D Gaussians to implement a hierarchical coherent motion mechanism. Specifically, we partition the scene into regions and recognize that only a few regions actually contain Gaussian primitives due to the non-uniform distribution of Gaussians. We explicitly model the motion within these non-empty regions, allowing Gaussians in the same region to share identical motion patterns. These regions can be further divided into smaller areas, enabling the motion of each Gaussian to be determined by the combined motions of all levels of regions it inhabits. This hierarchical coherent motion mechanism captures motions from coarse to fine granularity and requires only a minimal set of parameters, facilitates rapid convergence. The inherent structure and consistency within and between regions thus support swift learning of scene changes across frames. We also introduce additional Gaussians to better accommodate significant updates in scene content. These new Gaussians are carefully integrated into the initial 3DGS representation to ensure continuous consistency with the evolving scene. To maintain the compactness of the 3DGS, an equivalent number of low-opacity Gaussians, which no longer significantly contribute to the scene representation, will be removed before the learning of the next frame. This continual refinement to the initial 3DGS representation ensures it remains as close as possible to the evolving scene, facilitating better subsequent learning. In addition, we introduce a parallel training strategy that enables simultaneous learning of multiple frames, significantly enhancing training efficiency with minimal impact on performance. The contributions of this paper are summarized as follows:• We introduce the HiCoM framework for online learning of dynamic scene from multiview video streams, featuring a perturbation smoothing strategy for robust initial 3DGS representation learning, hierarchical coherent motion mechanism for efficient motion capture, and continual refinement to adapt evolving scene updates.• We devise a novel and concise hierarchical coherent motion mechanism that capitalizes on the inherent non-uniform distribution and local consistency of 3D Gaussians, efficiently capturing and modeling scene motions for swift and precise frame-to-frame adaptation.• Extensive experiments demonstrate that our HiCoM framework improves learning efficiency by about 20% and reduces data storage by 85% compared to state-of-the-art methods. Our parallel training strategy enables learning multiple frames simultaneously, substantially decreasing the training wall time with negligible effects on overall performance, further enhancing the practicality and responsiveness of our HiCoM for real-world applications.2 Related Work 
dynamic gaussian splatting: 
 3D Gaussian Splatting [24] has quickly garnered attention for dynamic scene reconstruction due to its efficiency and flexibility. 1) Several Dynamic Gaussian Splatting works [25, 26, 27] directly expand attributes of Gaussian primitives: 4D Gaussian Splatting [26] adds a timestamp dimension to form 4D Gaussian primitives; Dynamic 3D Gaussians [25] incorporates positions and rotations at every timestamp into the Gaussian primitives. 2) Other works [28, 29, 30, 31, 32, 34, 33, 37] decouple the dynamic scene into base scene representation in canonical space and time-varying motion fields: Deformable 3D Gaussians [28] , inspired by dynamic NeRFs [1, 22, 19, 20, 21, 18] , uses a deep MLP to predict the motion of Gaussians based on their positions and the given timestamp, enabling detailed and real-time rendering of dynamic scenes; 4D-GS [29] refines this pipeline by employing a more efficient multi-resolution hex-planes and a lightweight MLP, significantly enhancing efficiency and performance, particularly in high-resolution real-time rendering scenarios; SC-GS [30] assumes that scene motions are driven by a small number of key point movements, leveraging sparse control points and dense Gaussians to distinctively represent motion and appearance in dynamic scenes, and employs a deformation MLP to predict time-varying 6 DoF transformations for each control point, which are then locally interpolated through learned weights to create a motion field that influences the dense Gaussians, ensuring coherent and dynamic scene rendering.Compared to the offline learning from fully captured multi-view videos, on-the-fly learning, i.e., frame-by-frame training, for constructing streamable dynamic scenes presents more challenges. Given the success of NeRFs [7, 8, 9, 36, 20, 19] in addressing both static and dynamic scenes, there has been a growing interest in adapting and refining these techniques to solve the challenges of streamable dynamic scenes. StreamRF [2] tackles this by employing an incremental learning paradigm, where per-frame differences are modeled to efficiently adapt to changes, significantly speeding up the training process and reducing storage overhead than traditional methods. NeRFPlayer [38] , on the other hand, decomposes the 4D spatiotemporal space based on temporal characteristics, optimizing for both speed and compactness in model representation, thus enabling fast reconstruction and streamable rendering. ReRF [39] extends these capabilities by modeling the residual information between adjacent timestamps, which allows for real-time free-viewpoint video (FVV) rendering of long-duration dynamic scenes with high fidelity. The emergence of 3D Gaussian Splatting (3DGS) has brought significant advancements in training efficiency and rendering speeds, making it exceptionally suited for addressing streamable dynamic scenes. Building on superior 3DGS and inspired by previous NeRF works [2, 36] , 3DGStream [35] introduces an efficient framework that uses a Neural Transformation Cache to manage the transformations of 3D Gaussians, drastically reducing the training time and storage requirements. This method also features an adaptive addition strategy for 3D Gaussians to better handle emerging objects in dynamic scenes. 
preliminaries: 
 3D Gaussian Splatting explicitly represents scenes using anisotropic 3D Gaussian primitives, each defined by a mean vector µ and covariance matrix Σ, which respectively characterize the central position and geometric shape of the Gaussian in world space, mathematically formulated as:EQUATIONThe covariance matrix Σ is decomposed into a scaling matrix S and a rotation matrix R to ensure physical meaning and facilitate optimization:EQUATIONwhere S = diag(s x , s y , s z ) ∈ R 3 and R ∈ SO(3) are parameterized as a 3D scaling vector s and rotation quaternion q, respectively. Gaussian primitive is enriched with color and opacity, represented by spherical harmonic coefficients h and a scalar α, respectively.To render a novel viewpoint, Gaussian primitives are projected onto the camera plane using the view projection matrix W and the Jacobian J of the affine approximation of the projective transformation, the covariance matrix Σ ′ in camera coordinates given as follows:EQUATIONRendering is performed by blending the contributions of N overlapping Gaussian primitives at each pixel, taking into account their depth-ordering to ensure correct compositing, expressed as:EQUATIONwhere c i , α i represents the color and blending weight of the i th Gaussian, respectively.The training of 3D Gaussian Splatting alternates between parameter optimization and density control. Parameter optimization is supervised by the L 1 loss and D-SSIM term:EQUATION)where λ is typically set to 0.2. Meanwhile, density control manages Gaussian cloning and splitting to address over-reconstruction and under-reconstruction. 
methodology: 
 The online reconstruction of streamable dynamic scenes follows a sequential and iterative pipeline, crucial for applications requiring real-time rendering and interaction. Initially, a 3D Gaussian Splatting (3DGS) representation of the scene is constructed from the first frame of synchronized multi-view video streams. As each new frame is captured, this representation is updated and refined to align with the latest scene. This process demands a high-quality scene representation at every timestamp to ensure visual fidelity, while also necessitating efficiency in learning, storage, and transmission to make it feasible for real-world applications.To achieve these goals, our framework (Figure 2 ) introduces three key designs: First, we develop a compact and robust initial 3DGS representation using a perturbation smoothing strategy, ensuring a stable foundation for subsequent frame learning (Sec. 4.1). Second, we employ a hierarchical coherent motion mechanism, leveraging the inherent non-uniform distribution and local consistency of 3D Gaussians to model and rapidly learn scene motion from the previous frame to the current frame (Sec. 4.2). Third, we implement continual refinement strategies to adapt to scene content updates missed by the motion mechanism, maintaining fidelity and accuracy as the scene evolves (Sec. 4.3). Importantly, we develop a parallel training strategy that enables simultaneous learning of multiple frames (Sec. 4.4), significantly enhancing the practicality and responsiveness of our framework. The initial 3DGS representation forms the cornerstone of the entire dynamic scene reconstruction process, laying the groundwork for precise and efficient frame-by-frame online learning.In typical setups for capturing dynamic scenes, the number of cameras is often limited, contrasting sharply with static scenes that usually involve hundreds of multi-view images. This limitation can lead to instability and potential overfitting to training views during optimization. Moreover, if not adequately controlled, the number of Gaussians could grow unchecked, slowing down the rendering process and convergence speed in the learning of subsequent frames.To combat overfitting, a prevalent issue in neural network training, various strategies such as dropout [40] and data augmentation [41, 42, 43] have been proposed. Drawing from these insights, we discover that a simple perturbation smoothing strategy helps mitigating this issue in 3DGS, which has proven also effective in NeRF [44] . As Figure 2 (a), we add small gaussian noise to the position attribute of 3D Gaussian during training, creating the random perturbed position:EQUATIONthereby preventing the model from becoming too closely fitted to the limited training views. Random perturbation not only stabilizes the learning process but also moderates the growth of 3D Gaussians. As a result, by the time the model converges, the number of Gaussians is significantly reduced, speeding up the convergence and enhancing the overall efficiency in subsequent frame learning.Follow the online learning pipeline, we continuously fetch the latest views from the video streams and reconstruct the current scene representation. A straightforward method would be to use these new images to fine-tune the previous 3DGS. However, this naive approach is still insufficient in terms of learning speed and requires storing and transmitting a significant amount of data.Although recent efforts [28, 29, 35] have demonstrated the feasibility of using neural networks to predict the motion of each 3D Gaussian, showing promising results in adapting to scene changes, the implicit modeling of the motion field fails to align well with the explicit and discrete nature of 3DGS.In contrast, we explicitly model the motion field through hierarchical coherent motion mechanism, allowing us to directly capture and adapt to scene dynamics in a structured and efficient manner.As illustrated in Figure 2 (b), the scene is first divided into regions with the same size. and we can determine the region in which each 3D Gaussian is primarily located using the following formula:EQUATIONwhere r denotes the center coordinates of the region, µ is the center position of the 3D Gaussian, and e represents the size of region.During optimization, 3D Gaussians naturally exhibit a non-uniform distribution, with areas of complex geometry and texture densely populated by smaller Gaussians, while simpler and smoother regions sparsely covered by larger Gaussians, resulting in a significant number of regions remaining empty.For those regions that contain Gaussians, we assign translation and rotation attributes, respectively parameterized by a 3D vector ∆µ ∈ R 3 and a quaternion ∆q ∈ R 4 . These motion attributes are shared by the Gaussians within the region, ensuring local consistency.However, motion within smaller subregions might exhibit subtle differences due to more localized dynamics. To accommodate these variations, we further partition the regions and establish distinct motion parameters for these smaller areas. Consequently, the motion of each Gaussian is determined by the cumulative of motion from multiple hierarchical levels to which it belongs:EQUATIONwhere ∆µ g ∈ R 3 and ∆q g ∈ R 4 represent the composed motion of Gaussian, ∆µ l indicates the motion contribution from the l th level of regions, and L is the total number of motion levels involved.This hierarchical coherent motion mechanism allows us to capture Gaussians' movements ranging from coarse to fine, ensuring a more detailed and accurate representation of dynamic changes across frames, as well as coherent motion between adjacent regions. By sharing motion attributes among multiple Gaussians, we decrease the number of individual computations required and the number of parameters we need to optimize, helping the learning converge faster and more reliably.A 3DGS representation that is more closely aligned with the most recent scene can be obtained after applying the learned motion. However, this motion mechanism, focusing primarily on modest shifts to the positions and rotations of Gaussians from previous 3DGS, might not capture all the finer details of scene changes or adapt swiftly enough to significant scene content updates. This gap can be strategically bridged by adding new Gaussians in appropriate areas.Specifically, during motion learning, some regions accumulate significant gradients indicating substantial discrepancies between the learned and actual scene. Following established methods [24, 35] , we clone Gaussians in these regions that have accumulated gradients exceeding a predefined threshold. As shown in Figure 2 (c), the newly cloned Gaussians are then subjected to further optimization and density control to better align with the most recent scene.As real-world dynamic scenes typically exhibit coherent changes, newly added Gaussians tend to stay relevant across subsequent frames, making it impractical to simply discard them. Thus, we retain these Gaussians, integrating them into the initial 3DGS to form the basis for learning the next frame. However, consistently merging new Gaussians in each frame can lead to a steady increase in the number of Gaussians, potentially slowing down rendering speeds, delaying convergence, and prolonging learning times. Therefore, we selectively remove an equivalent number of low-opacity Gaussians that minimally impact the overall visual integrity of the scene [45] , to maintain the compactness and efficiency of 3DGS representation without compromising scene fidelity. Simultaneously, this continual refinement allows us to address and enhance regions of the 3DGS that may have been underrepresented or insufficiently modeled in earlier learning stages, solidifying a comprehensive and adaptive representation as the scene evolves.Typically, our framework works in a sequential frame-by-frame pipeline. To further boost efficiency and responsiveness for practical applications, we draw inspiration from video encoding techniques where reference frames (such as I-frames in H.264/AVC) are used to predict multiple subsequent frames, leveraging temporal redundancy and reducing computational overload.Considering that differences between consecutive frames are generally minor, we choose the 3DGS of the frame t as the reference to simultaneously learn frames {t + 1, • • • , t + k}. After processing 5 Experiment 
experimental setup: 
 For all of our experiments in this paper, we utilize two widely-used public datasets as follows.• Neural 3D Video (N3DV) [1] dataset comprises six dynamic indoor scenes featuring varying illuminations, view-dependent effects, and substantial volumetric details. Videos were captured at a resolution of 2704 × 2078 and a frame rate of 30 FPS. Video counts per scene range from 18 to 21. We follow prior works [2, 35, 29] to downsample the original video resolution by a factor of two.• Meet Room [2] dataset was recorded with a multi-view system consisting of 13 synchronized Azure Kinect cameras, covering three different indoor scenes. The resolution of captured videos is 1280 × 720 and the frame rate is 30 FPS too.For both datasets, the video captured by the center reference camera is used for testing, while other videos are employed for training. Both datasets already include camera pose information, so we only generate initial point clouds with COLMAP [46] following 4D-GS [29] . Most scenes last only 10 seconds, thus, we restrict our experimentation to the first 300 frames to align with prior works.We evaluate methods based on four metrics, across all 300 frames: 1) video synthesis quality of the test views, assessed by the mean PSNR; 2) average training time per frame; 3) rendering speed, assessed by the frame rate of the synthesized video; 4) storage and transmission efficiency, evaluated by the average size of the representation parameters per frame.Implementation. Our initial frame learning employs the standard 3DGS with a fixed λ noise =0.01, training for 15k and 10k steps on two datasets respectively, with splitting halted at the 5k th step.The number of regions at the smallest level is determined by dividing the number of Gaussians n in the initial 3DGS by the maximum number of Gaussians m=5 per region. The number of regions at each subsequent larger level is determined by further dividing by 2 3 , and so forth, with only regions containing Gaussians having motion parameters, which are initialized at 0.6× of the previous frame's motion parameters. Each new frame undergoes 100 motion training steps, with an additional 100 steps after new Gaussians are added. We implement 3DGStream [35] in the same codebase as ours following its paper, and run all experiments 3 times on RTX 4090 GPUs, the mean metrics are reported. More details are provided in the Appendix.Quantitative Benchmark We conduct a comprehensive quantitative comparison with state-of-theart online methods, StreamRF [2] and 3DGStream [35] , presenting results in two counterparts in training speed, with an average per-frame learning time (include the initial frame) reduced by more than 17% compared to 3DGStream on both datasets. This advantage becomes even more pronounced in complex scenes, such as Coffee Martini and Flame Salmon, where the learning speed improvement exceeds 20%, primarily due to our method's rapid convergence speed.Both our HiCoM and 3DGStream obtain rendering speeds over 200 fps, demonstrating real-time rendering capability. In terms of storage and transmission efficiency, our approach substantially surpasses the two competitors, requiring less than 10% of their average per-frame storage space after the initial frame. This is mainly due to two factors: firstly, our motion parameters are shared among multiple 3D Gaussians within local regions; secondly, the inherent non-uniform distribution of the 3D Gaussians results in most areas being empty, thus requiring far fewer motion parameters. Due to the reduced amount of information needing transmission, our method is more advantageous for streamable dynamic scenes. In the Appendix, we provide results for each scene and results of several state-of-the-art offline methods, offering a more detailed quantitative comparison. We also include experimental results on two additional datasets, PanopticSports [25] and ParticleNeRF [47] , in the Appendix. On the PanopticSports dataset, which features large-scale motion, our method achieves better performance than the state-of-the-art Dynamic3DGS [25] method, while on the synthetic ParticleNeRF dataset, our HiCoM demonstrates good results as well. These additional experiments further validate the generalization capability of our approach across diverse datasets.Qualitative Analysis Our HiCoM framework inherits the advantages of 3D Gaussian Splatting in representing static scenes but focuses more on the changes in dynamic scenes over time. As shown in Figure 3 , the scene features a person making a drink, with his head, hands, and the liquid in the cup being the main dynamic areas. We select six frames, arranged in chronological order, from all 300 frames to analyze the rendering quality. It can be observed that our method produces results that are much closer to the ground truth in these dynamic areas. Furthermore, our HiCoM exhibits more coherent temporal transitions because, in addition to updating the initial Gaussian positions and rotations, we carry over newly added Gaussians to the subsequent frames instead of discarding them. This results in smoother and more natural transitions over time, such as the liquid gradually flowing into the glass. More qualitative results for other scenes are provided in the Appendix.As shown in Table 2 , we ablate three key components of our HiCoM framework. In the 2 nd row, we do not use noise perturbation, i.e., the initial frame learning falls back to standard 3DGS training. This causes many small Gaussians to split during the initial learning to fit the training views, resulting in a slight decrease in PSNR on the test views. By incorporating noise perturbation, our HiCoM reduces the number of Gaussians while making them larger, each contributing to the rendering of more pixels. Consequently, the training time was slightly longer (1 st row vs. 2 nd row). In the 3 rd row, we disable motion learning, using the first 100 steps solely for gradient accumulation to determine which Gaussians to clone. This resulted in a significant performance drop, particularly in the relatively simple scenes Flame Steak and Discussion, indicating that this component is crucial for the overall framework. In the 4 th row, we omit the refinement, extending motion learning instead. This leads to a slight performance decrease, suggesting that the additional Gaussians can further compensate for the intense and detailed scene changes that the motion learning might not capture.A notable characteristic of our HiCoM is the explicit representation of motion field, which aligns well with the discrete, explicit nature of 3DGS. Our HiCoM requires very few motion parameters, reducing the complexity of motion representation and learning, theoretically accelerating the convergence speed. According to ablation results presented in Table 3 , increasing the motion learning steps from 50 to 100 (2 nd row vs. 1 st row) still provides significant performance gains, i.e., 0.39 and 0.22 dB on two datasets, respectively. However, learning an additional 50 steps beyond 100 (1 st row vs. 3 rd row) yields only minimal improvements , indicating that the model has mostly converged by 100 steps. Similar conclusions can be drawn for learning newly added Gaussians. Ultimately, a total of 200 learning steps is sufficient to achieve optimal results.Our hierarchical coherent motion incorporates multiple levels of motion, with different levels corresponding to varying sizes of regions and reflecting different granularities of motion. In Table 4 , we present ablation studies on motion levels. We use three motion levels by default (the 5 th row), refer to as "fine", "medium" and "coarse", respectively. As the results show, using only the "fine" level already achieves good performance. Adding the "medium" and "coarse" levels provides additional performance gains, e.g., the PSNR of Coffee Martini and Discussion scenes respectively boosts 0.25 dB and 0.59 dB, but the improvements diminish as the number of levels increases, indicating that our HiCoM is effective and a few motion levels are sufficient to achieve good results. It can also be observed that fewer motion levels and parameters do not necessarily imply shorter training time. This is because insufficient motion learning may lead to the generation of more Gaussians during continual refinement stage, slightly increasing the overall training time.We conduct experiments on three representative scenes from two datasets to validate the parallel learning. As shown in Table 5 , when the number of parallel frames is relatively small, such as 2 and 4, the performance remains largely unchanged, with some scenes even exhibiting improved PSNR. However, as the number of parallel frames increases, the disparities between the learning and the reference frames grow, leading to a decline in performance. Specifically, when the number of parallel frames reaches 16, there is a noticeable PSNR drop, i.e., 0.18, 1.82 and 1.32 dB, respectively. While parallel learning does not reduce the actual GPU computation time, it significantly reduces the wall time, making the system more practical for real-world applications. For example, with 4-frame parallel learning, we can reduce the average learning time per frame to one-fourth of the original time while maintaining nearly the same quality. This means we can learn one frame in less than 2 seconds on average, greatly enhancing the efficiency and responsiveness of the system. 
conclusion: 
 This paper introduces HiCoM, a framework designed to tackle the challenges in online reconstruction of streamable dynamic scenes from multi-view videos. HiCoM reduces the number of 3D Gaussians for a more compact representation through the perturbation smoothing strategy, enhancing the robustness of initial frame learning. HiCoM distinguishes with its hierarchical coherent motion mechanism that explicitly represents motion between adjacent frames with minimal parameters, aligning well with the discrete and explicit nature of 3D Gaussian Splatting. This mechanism efficiently captures motion at different granularities, resulting in faster convergence and substantially reduced data storage. Additionally, HiCoM continuously refines the representation to adapt to evolving scene, facilitating better subsequent learning. Experimental results show that HiCoM boosts training efficiency by about 20% and reduces data storage by 85% compared to state-of-the-art methods. Furthermore, the parallel training significantly reduces wall time cost with negligible performance degradation, elevating real-world applicability and responsiveness.Limitations. Despite our HiCoM's significant improvements, the role of the initial 3DGS representation remains crucial in the online learning pipeline, which is still not fully addressed. Future work could explore integrating advanced 3D Gaussian Splatting techniques to enhance the initial frame learning, which, combined with our hierarchical coherent motion mechanism, could further improve training efficiency while also reducing storage and transmission overhead. During online learning, reconstruction quality may decline over time due to error accumulation-a common issue across many online learning methods and one that requires further research to effectively address. Additionally, our experiments are conducted solely on indoor scenes, so the generalization of our method to outdoor or more complex environments has not been verified and requires further validation. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [No] Justification: . Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The creators or original owners of assets (e.g., code, data, models), used in the paper, are properly credited, and the license and terms of use explicitly are mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: This appendix provides additional material to supplement the main text. We will first introduce more implementation details in Sec. A. Then we provide some additional experimental results in Sec. B.Our code is primarily based on the open-source codes of 3DGS [24] and 4D-GS [29] . We keep the 4D-GS setting that does not reset opacity in the initial 3DGS learning because we observed that resetting opacity leads to the removal of larger Gaussians, which is helpful when the number of Gaussians already significantly reduced by our noise perturbation smoothing mechanism. The hyperparameter λ noise was empirically selected from the range of {0.001, 0.01, 0.1}. Among these values, 0.01 consistently performed the best across multiple scenes, hence we set it to 0.01 in all experiments. After applying the learned motion to 3DGS, we determine the Gaussians to be cloned based on the accumulated gradients during motion learning, which is similar to standard 3DGS. The cloned Gaussians undergo alternating optimization and densification, with densification occurring every 40 steps. We found that this parameter is not sensitive and can be freely set between 20 and 50. The removal of low-opacity Gaussians happens before learning the next frame, so these Gaussians still contribute to the representation of the current frame.At the time of conducting this work, the code1 released by the authors of 3DGStream still demonstrated instability and encountered issues during execution. Therefore, we reported the experimental results based on the code we implemented. Although, we attempted to faithfully reproduce the 3DGStream as described in its paper, inconsistencies remain in our implementation and experimental setup. The main difference lies in the dataset processing. We employ the original dataset following 4D-GS code, whereas 3DGStream utilizes processed datasets with modified resolutions and camera poses, resulting in about 50,000 fewer pixels per view image of N3DV [1] dataset. The spherical harmonics (SH) order is set to 1 as per the paper, but we do not perform SH rotation because it increases the computational overhead without noticeable performance gains. Additionally, 3DGStream requires scene boundary being manually setup, whereas we automatically compute them after removing some outlier Gaussians. These disparities lead to marginally lower metrics compared to those reported in the 3DGStream paper, but our results align closely with other offline methods using the same dataset. Table 6 : Per-scene quantitative results on the N3DV dataset. The offline and online methods are separated into upper and lower sections, respectively. The per-frame metrics "Storage" and "Train" are averaged over 300 frames, with those for offline methods [20, 38, 26, 29, 48, 34] estimated based on data reported in papers [29, 48, 34] . The data for Dynamic 3DGS are referenced from [48] .Method Coffee Martini Cook Spinach Cut Beef PSNR Storage Train PSNR Storage Train PSNR Storage Train (dB ↑) (MB ↓) (s ↓) (dB ↑) (MB ↓) (s ↓) (dB ↑) (MB ↓) (s ↓)KPlanes [20] 29.99 1.0 14. 4 (dB ↑) (MB ↓) (s ↓) (dB ↑) (MB ↓) (s ↓) (dB ↑) (MB ↓) (s ↓)KPlanes [20] 30 We present the quantitative results for each scene from two datasets respectively in Tables 6 and 7 to provide a more detailed comparison. We also provide the results of several state-of-the-art offline methods on N3DV dataset in Table 6 as a reference. Among these, Coffee Martini and Flame Salmon are particularly complex scenes that require a greater number of 3D Gaussians and motion parameters in our HiCoM framework, resulting in relatively higher storage requirements. However, they still require significantly less storage than training with the 3DGStream method. In addition, we conduct experiments on the PanopticSports dataset, which involves large-scale motions, and compare with the state-of-the-art Dynamic3DGS method, as shown in Table 9 . Notably, Dynamic3DGS heavily relies on segmentation masks, while our HiCoM does not require such additional supervision. To ensure a fair comparison, we set the SH to 1. Additionally, we observe that Dynamic3DGS employs a trick to mitigate potential multi-view color inconsistencies by learning a scale and offset parameter for each color channel of each camera in the training views. To enhance performance, we incorporated this same color trick into HiCoM. The results demonstrate that our method significantly outperforms 3DGStream in terms of PSNR. While slightly behind Dynamic3DGS initially, with the same color trick applied, HiCoM clearly surpasses Dynamic3DGS, highlighting its superiority in handling large-scale motion scenarios.(dB ↑) (MB ↓) (s ↓) (dB ↑) (MB ↓) (s ↓) (dB ↑) (MB ↓) (s ↓)All our experiments were conducted three times using the same random seed. In Table 8 , we present the standard deviation of the three runs on the N3DV dataset, showing the fluctuation in PSNR metric, which is relatively minor. We also tried using different random seeds, and the performance fluctuation was still within the range shown in Table 8 .Furthermore, we extended our evaluation to the Particle-NeRF [47] dataset, which features synthetic scenarios with uniform and periodic motions. The Table 10 shows that our HiCoM outperforms Particle-NeRF, demonstrating its ability to effectively manage these types of motion.We provide additional qualitative results for other scenes in the N3DV dataset, three scenes from the Meet Room dataset and six scenes of PanopticSports dataset, as shown in Figures 5, 6 and 7 , respectively. These results demonstrate the robustness and versatility of our HiCoM framework in capturing and representing dynamic scenes. • If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [No] Justification: . Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legal to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://github.com/SJoJoK/3DGStream 


Decision: All reviewers agree that the submission proposes a simple and effective solution for radiance field reconstruction from multi-view streaming videos. The authors efficiently addressed the reviewers' concerns in the rebuttal and during the intense discussion. AC recommends acceptance. Please carefully address the concerns raised by the reviewers in the camera-ready version. For example, include the new comparison table with dynamic 3DGS after incorporating the image embedding trick, an ablation study on Motion Level, additional comparison figures and visualization videos, and the release of the code. We look forward to seeing these contributions from the open-source community.

Reviews: review: paper_summary: paper_summary: This paper deals with the online reconstruction of dynamic scenes from multi view video input. The authors build their method on the popular 3D Gaussian Splatting technique. To tackle this setting, they propose 1. A perturbation smoothing that introduces small perturbations to the 3D positions of the Gaussians to improve the initial reconstruction learned on the first timestep, 2. A hierarchical coherent motion strategy that divides 3D Gaussians into a hierarchical voxel grid and assigns motion coefficients (3D translation and 4D Quaternion rotation) to each voxel, where the motion of each 3D Gaussian is the sum of all voxel-level motions it is associated with.
strength_weakness: strength_weakness: - The motion levels seem not to significantly affect the performance (cf. Tab. 5), which begs the question of whether the HiCoM strategy is critical for performance. Also, scene motion is piecewise constant, but has sharp boundaries. What if both dynamic foreground and static background fall into the same voxel?
- In L274, the authors mention that mean results over 3 runs are used in the experiments. In L697, the authors mention they do not include error bars. What’s the reason for this?
- Finally, I am curious about why online reconstruction is needed if both multi-view video and precomputed input poses are needed?
questions: questions: It would be great if the authors could clarify my questions raised in the weaknesses section. Additionally, Equation 6 suggests that the noise is drawn from a normal distribution with variance 1, which seems quite large. Is this an error or is the perturbation actually that large? Are the reconstructions in metric scale?
limitations: limitations: The paper discusses the method limitations sufficiently.
paper_summary: paper_summary: This paper proposes a novel framework for online reconstruction of dynamic scenes based on 3DGS. The main contribution lies in its Hierarchical Coherent Motion, which gives a more compact motion representation. Results show its effectiveness and rendering improvement.
strength_weakness: strength_weakness: 1. Continual Refinement seems similar to the strategy in 3DGStream. It would be better to clarify the difference. 

2. Most parts of the scene is static, uniformally dividing the scenes to model the motion seems redudant.

3. Embedding the motion in space alone may have limited modelling ability, especially when an object undergoes complex movements within a small region, such as intricate transformations back and forth. It might be better to incorporate the temporal dimension.

4. Although the method is tailored for streamable videos, it would be better to compare with other dynamic 3DGS methods, such as 4DGaussians, GaussianFlow, spacetime Gaussian. The previous work 3DGStream also includes the comparison with offline or non-streamable methods.
questions: questions: 1. Since the work focus on the online reconstruction of streamable videos, the efficiency is very important. I am wondering whether the cost of initialization from SfM (Colmap) should be considered.
limitations: limitations: 1. No ethic issues.

2. The authors kindly point out their limitations in the paper. The initialization of 3DGS in the first frame is significant and the paper is not fully addressed. It would be kind for authors to show how the initial frame learning influence the whole pipeline currently, as they claim it is not fully addressed.
paper_summary: paper_summary: This paper proposes a novel online training framework for multi-view dynamic scenes. A HiCoM framework is introduced for online learning of dynamic scenes. To initialize a better 3D Gaussian, the authors propose adding noise because it may overfit in the forward-facing multiview setups. Modeling motion by a sparse and explicit structure looks very good and also preserves low storage consumption. Experiments also demonstrate that the results are better than previous methods. e.g. 3DGStream.
strength_weakness: strength_weakness: 1. It seems that hierarchical coherent motion shows the potential to model large motions. So experiments in the PanopticSports dataset (used in [25]) should be included.
2. Comparison/discussions with Particle-NeRF(Abou-Chakra et al, WACV24) should be included.
3. You can add more comparison figures about ablation studies to demonstrate the results. The reviewer recommends authors submit a video to support a visualization, which is very important in 3D Vision.
4. The introduction section is too long. The reviewer recommends the authors try to express themselves easily and efficiently. Only 3-4 paragraphs (70 lines) are enough to introduce your work. The writing could be improved.
questions: questions: 1. How can you integrate the 3D Gaussians into the initial 3DGS? If you add 3D Gaussians at t(t>0) and how to compute its position at t=0.
2. The reviewer expects to see the results on the Panopticsports dataset. If your code is based on 4D-GS, you can run the experiments directly.

Overall, there are some written problems in the paper and there are lack of visualizations and discussions. The main concern of the reviewer is whether the HiCoM can model large motions such as Panopticsports. If the author can provide positive results on Panopticsports and more discussions, the reviewer may recommend accepting this paper.
limitations: limitations: The limitations are shown in the paper.

Comments: title: Discussion
comment: Dear Jiakai,

Thanks for the classification. To ensure a consistent comparison, we would either need you to provide an average score across all three scenes or have the HiCoM authors report a comparison for the "discussion" scene only.

@Jiakai, Do you think it’s possible to provide the average scores?

@HiCoM authors, please track this discussion.

- Your AC
Dear Reviewer,

Thank you very much for your feedback and for acknowledging our detailed explanations. We appreciate your recognition of the importance of including all results and discussions to benefit the research community, and we will **certainly do so** in the paper or appendix.

Thanks to the constructive comments from all reviewers, we have worked hard to improve our work. Particularly, driven by Reviewer q4mz’s insightful questions, we conducted additional experiments that demonstrated our method’s superior performance over the state-of-the-art Dynamic3DGS on the PanopticSports dataset with large motions. **Our experimental results and solid analysis has been validated by Reviewer q4mz, who has already raised his/her score based on our new results.**

We kindly suggest that you refer to our discussions with Reviewer  q4mz, or feel free to discuss the matter directly with he/she, to further assess the improvements we have made. If you have no further concerns, we would be grateful if you could consider raising your score as well.

Thank you again for your time and effort.

Best regards,  

The Authors
We appreciate your time and effort, as well as your final recognition of our improved work.

Sincerely.
Many thanks.

I will change my score to `weak accept`.

Best wishes.
Dear Reviewer,

Thank you for your encouraging words. We also deeply appreciate the efforts and contributions you and the other reviewers have made to improve our work.

We assure you that we will release the code upon acceptance of the paper. We have put significant effort into ensuring that our code is clean, easy to maintain, and extensible. We hope that once it’s open-sourced, it will be a useful resource for the research community and facilitate fair comparisons across different methods.

Thank you again for your support.

Best regards,  

The Authors
Thank you for your solid reply. I have no more questions regarding HiCoM.

I wish the HiCoM could be a new baseline for evaluating multiview & streamable dynamic scene NVS methods. 

Finally, I expect to see the code will be released if the paper is accepted for reproducing all experimental results. If the author promise to do that, I would like to change my score.
Dear Reviewer,

Thank you for your thoughtful consideration.

We acknowledge that the first point has been addressed to your satisfaction, and we will certainly include the results on both the synthetic animation dataset and the PanopticSports dataset as you requested in the third point. Regarding the forth point, we are always committed to releasing our code to benefit the community and advance research in this field.

Now, focusing on your second point. After further investigation, we identified **a key trick** that contributed to Dynamic3DGS's superior performance. Specifically, Dynamic3DGS learns **a scale and offset parameter** for each color channel for each camera in the training views. Although this trick was mentioned in its paper, it was not emphasized or subjected to ablation studies.

Initially, we overlooked this trick. However, after implementing this trick in our HiCoM, we observed **significant improvements**. For example, in the Football scene, our HiCoM achieved an average PSNR of **29.25 dB** over all 150 frames, which exceeds Dynamic3DGS's **28.49 dB**. Moreover, when this trick was removed from Dynamic3DGS, the PSNR dropped by **4 dB**. This further demonstrates that the reconstruction quality of the initial frame in online learning has a significant impact on overall performance. Our latest experimental results also validate our earlier response to you, indicating that the role of masks is more beneficial for learning overall foreground motion rather than being the key to achieving better foreground rendering quality. Of course, this aspect requires further in-depth study in future research. The table below presents more comprehensive experimental results. With the trick, our **HiCoM significantly outperforms Dynamic3DGS** across five out of the six scenes. We will further analyze these results and include in the revised paper.

| Method                 | Juggle | Boxes | Softball | Tennis | Football | Basketball | Mean  |
|------------------------|--------|-------|----------|--------|----------|------------|-------|
| Dynamic3DGS            | 29.48  | **29.46** | 28.43    | 28.11  | 28.49    | 28.22      | 28.70 |
| 3DGStream              | 24.68  | 23.69 | 22.44    | 23.09  | 24.97    | 20.01      | 23.15 |
| **HiCoM (ours)**       | 27.82  | 27.37 | 28.06    | 27.80  | 27.80    | 27.29      | 27.69 |
| **HiCoM (w/ trick)**   | **29.55**  | 28.96 | **29.08**    | **29.02**  | **29.25**    | **28.60**      | **29.08** |


We hope this new evidence addresses your concerns. We believe that your valuable feedback has significantly improved the **solidity** of our work, and we are very grateful for that. If you have no further concerns, we hope you will consider raising your score.

Best regards,

The Authors
Thanks for authors detailed explanation. Hope the authors can add all the results and discussions in the paper or the appendix, which really promotes the development of the community. According to the current rebuttal and other reviews, I would like to keep my score.
Dear Reviewer,

Thank you for your detailed comments. We greatly appreciate your insights.

We agree with your suggestion and will follow your advice to present the results of all methods in a single table, making it easier for readers to compare and analyze.

Regarding the discrepancy in the results of 3DGStream, we consulted with the 3DGStream authors before they released the code. They confirmed that they did not use the original camera poses provided by the dataset and applied distortion correction to the views. During this work, we followed the data processing practices used by 4D-GS (CVPR 2024, in Table 8) and several previous works using this dataset, including the StreamRF (the paper that originally proposed the Meet Room dataset) in our Table 1, all of which utilize the original camera poses. Additionally, we noted in the Appendix that the processed views in 3DGStream contain approximately 50,000 fewer pixels than the original views. However, from our observations of the N3DV dataset, we did not notice significant distortion. Therefore, to ensure a fair comparison, we chose to remain consistent with the practices used by previous methods, using the original view images.

After submitting our paper, we also conducted experiments using the undistorted dataset following the open-source code provided by the 3DGStream authors. On the Meet Room dataset, the average PSNR was **28.32 dB**, which is significantly lower than the **30.79 dB** reported in their paper. Meanwhile, our HiCoM achieved a PSNR of **31.02 dB** on the undistorted dataset. Following your suggestion, we will include these results in the paper to provide a more comprehensive comparison.

Thank you once again for your thoughtful feedback. If you have any further questions or concerns, please feel free to reach out, and we will do our best to address them. We will also carefully incorporate the discussions with you and other reviewers into the revised manuscript.

Best regards,

The Authors
Thanks for your valuable comments. These are my consideration.

1. I do not doubt that HiCoM can beat 3DGStream in most evaluation metrics and cases (positive).

2. Currently, Dynamic 3DGS is indeed better at solving large motions with foreground masks. And I think optimizing dynamic scenes with masks is a necessary step because it's easy and cheap to get a foreground mask. But HiCoM seems to be an online optimizing method, that enjoys less training times and less storage costs. If the author can demonstrate that HiCoM can beat dynamic 3DGS from any perspective I would raise my score. The current rebuttal can only answer my questions. (borderline)

3. Hope the author can add all the results in both the synthetic animation dataset and PanopticSports dataset in the revised version, both are important benchmarks in evaluating a method.

4. If possible, release the code.

Therefore, I would like to keep my score.
Dear Reviewer,

Thank you for your comments. We would like to acknowledge that our method  does indeed perform slightly worse than Dynamic3DGS on the two datasets you mentioned, where the appearance shows less variation and no dynamic addition or removal of content compared to the initial frame.

Beyond using masks, Dynamic3DGS includes the motion parameters for **each Gaussian primitive** at every time step, whereas our method allows **Gaussians within a local region** to share same motion parameters. While this may lead to a slight compromise in detail compared to Dynamic3DGS, it offers significant advantages in storage and training efficiency for online dynamic scene reconstruction.

We would like to further address your specific concerns as follows.

### 1. Why optimizing with masks doesn't improve the final PSNR of our HiCoM?

Masks represent the relative position of the foreground **as a whole** to the background, and they can be helpful in learning **the overall motion of foreground**. Our method already learns the overall foreground motion well without masks, demonstrated by the consistency between the rendered and the ground truth masks (will be included in the revised manuscript ). Therefore, using masks does not enhance the overall foreground motion learning. In contrast, Dynamic3DGS struggles with overall foreground motion without masks, highlights our method's strength in capturing large global motion.

While we effectively position the whole foreground, Dynamic3DGS’s ability to optimize each Gaussian’s position at different time steps leads to better internal detail and higher PSNR. 
However, we aim for our online dynamic reconstruction method to still perform well without masks, as in many cases, masks may not always be readily available.  We believe that applying finer division to the foreground region in our framework could further improve the quality.  While this is beyond the scope of the current work, it remains a promising direction for future research.

### 2. Performance comparison on the ParticleNeRF dataset.

The experimental results we presented show that our method performs better than ParticleNeRF, indicating that our approach is **indeed capable** of handling the uniform and periodic motions present in this synthetic dataset. The reviewer's expectation that our method should also surpass Dynamic3DGS may stem from a misunderstanding. As we previously explained, Dynamic3DGS benefits from its ability to optimize each Gaussian's position, which gives it an advantage, especially in these simpler synthetic scenarios.

We hope this clarification helps to address your concerns, and we sincerely appreciate your valuable feedback. We will make sure to thoroughly incorporate these clarifications into the revised version of our paper. Unfortunately, according to the **review guidelines**, we are unable to provide external links at this stage, but we would be happy to present the video results to you in a compliant manner at an appropriate time. 

Thank you once again for your consideration.

Best regards,  

The Authors
Dear authors, thanks for your reply. My concerns still exist.

1. As the authors addressed, if HiCoM can optimized with masks why it doesn't improve the final PSNR?  What's the reason? I don't think it would demonstrate HiCoM can achieve better foreground rendering quality compared with Dynamic 3DGS.

2. As the author said, Particle-NeRF shows its advantages in uniform motions, but I think it is the subset of real-world dynamics. I think a dynamic representation can handle the complex motions in real-world scenes, it can also handle uniform and periodic motion. Dynamic 3D Gaussians also demonstrate that it would achieve good results in both particleNeRF dataset and its used Sports dataset.  

Therefore, I think HiCoM should beat dynamic3DGS and particleNeRF in both synthetic animation datasets and panopticSports datasets, if evaluation metrics cannot demonstrate that, a video or figure is also convincible.
Thanks for authors' reply and effort. I have read the rebuttal and discussion with other reviewers. Most of my concerns are addressed. However, for the comparison with offline reconstruction methods, I hope the author can clearly present the comparison of methods in a table. Table 8 of the supplementary materials mentioned by the author only shows the results of offline methods, which made me spend more effort to compare the results of this method again. It is acceptable that offline methods perform better than online methods because they invest more time and even storage to optimize.

Besides, I have one more concern. I found the reported result for 3DGStream is much lower than its original paper, especially for the meet room dataset (For N3DV, the 3DGStream reports **31.67** PSNR while the authors' reimplemented verision reports **30.15** PSNR. For meet room, the 3DGStream reports **30.79** PSNR while the author's reimplemented verision reports **25.96** PSNR). I really appreciate authors' effort for reimplementation and I understand that the open-source code may not have been systematically maintained. However, such performance differences might make it difficult to conduct a fair comparison in the future works, as two works placed different results on the same dataset. Has the author tried contacting the authors of 3DGStream? I am not sure if more discussion is needed at this point. It is just my concern.
Dear Reviewer QKxo,

I hope you’re doing well.

We understand you may have a busy schedule and might not have had the chance to review our rebuttal yet. We hope that our responses have adequately addressed your concerns. If you have any further questions or need additional information, please feel free to let us know.

Thank you again for your time and effort in reviewing our submission.

Best regards,

The Authors of Submission 809
Dear Reviewer,

Thank you for your response, and we are pleased to hear that our rebuttal addressed most of your concerns.

Regarding the issue of random seeds, we have conducted an additional experiment with a different random seed. The results remain within the range we reported. Randomness is pervasive in experiments, and even with the same random seed, variations can occur due to factors such as GPU scheduling and optimization algorithms. In the field of Gaussian Splatting, almost all works use default random seed because neural networks are not extensively involved, the parameters of Gaussian primitives are not influenced by random seeds. For further discussion on this topic, please refer to issue 89 in the official 3DGS code on GitHub. We will further validate our results with different random seeds and emphasize this in the revised manuscript.

Thank you once again for your comment. We would be grateful if you could consider re-evaluating our work in light of these clarifications.

Best regards,

The Authors
I thank the authors for the response, most of my concerns are addressed. Could the authors elaborate on "using the default random seed"? Ideally, the experiments should be run with different random seeds to make sure the improvements are significant. Do the added error bars indicate a deviation of different runs with the same random seed? If so, I'd be curious as to why the results differ even when choosing the same random seed.
Thank you for your comment. We will now address the two questions you raised.

### 1. Can HiCoM be optimized with masks?

Our HiCoM is indeed extendable and can be optimized with masks. However, our experiments have shown that this does not yield additional performance gains. Specifically, we incorporated the loss between the rendered mask and the ground truth mask (referred to as $L_{mask}$), and constrained the background Gaussian primitives to remain as static as possible (referred to as $L_{static}$), as suggested by Dynamic3DGS.

Despite these efforts, our method has already shown an ability to learn the motion of the foreground and background reasonably well, as demonstrated in the attached PDF. The masks are rendered with sufficient accuracy, and further optimization using masks did not result in significant improvements. Moreover, since our method inherently learns minimal background motion, the $L_{mask}$ and $L_{static}$ made limited contributions.

### 2. "comparison with particle-NeRF" is strange.

We are unclear if the reviewer's concern is with the experimental results or our conclusion about ParticleNeRF. We will clarify both aspects briefly.

Firstly, regarding the experimental results, the ParticleNeRF dataset comprises synthetic objects with relatively simple but precise and regular motions (e.g., wheel rotating) and without changes in appearance. 
Our HiCoM potentially performs better than ParticleNeRF due to several factors, including the inherent advantages of Gaussian Splatting, the general compatibility of our motion learning mechanism with this dataset, and the additional Gaussian primitives introduced at each frame. Dynamic3DGS learns individual positions and rotations of Gaussian primitives at each time step, providing finer detail compared to our approach, where multiple Gaussian primitives share the same motion. Therefore, it is reasonable that Dynamic3DGS outperforms our HiCoM in this context.

Secondly, our conclusions about ParticleNeRF and its motion modeling are based on its paper and experiments. ParticleNeRF's position-based dynamics (PBD) algorithm tends to distribute particles more uniformly (but unlike the fixed grid of InstantNGP), while effective for simpler synthetic datasets with periodic, regular and micro motions, may struggle to handle the intricate details required in real-world dynamic scenes, where complex regions often need denser particles or Gaussian primitives and more detailed motion. ParticleNeRF's e experimental results demonstrate that it performs well on its own synthetic ParticleNeRF and Animated Blender datasets, both involving object-level motions, but the Figure S6 in its supplementary material shows less satisfactory results on a full scene. These observations support our conclusion.

Thank you again for your valuable feedback. We hope our responses meet your expectations. If there are any further questions or concerns, please do not hesitate to let us know.
Dear reviewer oHau,

We received diverse reviews for this submission, and you were initially negative. It would be extremely helpful if you could review the rebuttal and participate in the discussion over the next two days. Thank you very much!

- Your AC
Thanks for your comprehensive answers, most of my problems are solved. However, I still have two questions:

1. Can HiCoM be optimized with masks? In dynamic scene reconstruction, filtering dynamic objects with a foreground mask could be a normal and correct solution.

2.  ”comparison with particle-NeRF" is strange because I think rigid & regular motion is the subset of scene level and non-rigid motion. 

Therefore, I would like to expect the author's and other reviewer's comments on this issue.