Title: Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction
Venue: NeurIPS 2025
Accepted: yes
Keywords: Free-viewpoint video

3D Gaussian Splatting

Real-time transmission

4D reconstruction

Compression

Streaming
Abstract: 3D Gaussian Splatting (3DGS) has emerged as a high-fidelity and efficient paradigm for online free-viewpoint video (FVV) reconstruction, offering viewers rapid responsiveness and immersive experiences. However, existing online methods face challenge in prohibitive storage requirements primarily due to point-wise modeling that fails to exploit the motion properties. To address this limitation, we propose a novel Compact Gaussian Streaming (ComGS) framework, leveraging the locality and consistency of motion in dynamic scene, that models object-consistent Gaussian point motion through keypoint-driven motion representation. By transmitting only the keypoint attributes, this framework provides a more storage-efficient solution. Specifically, we first identify a sparse set of motion-sensitive keypoints localized within motion regions using a viewspace gradient difference strategy. Equipped with these keypoints, we propose an adaptive motion-driven mechanism that predicts a spatial influence field for propagating keypoint motion to neighboring Gaussian points with similar motion. Moreover, ComGS adopts an error-aware correction strategy for key frame reconstruction that selectively refines erroneous regions and mitigates error accumulation without unnecessary overhead. Overall, ComGS achieves a remarkable storage reduction of over 159 × compared to 3DGStream and 14 × compared to the SOTA method QUEEN, while maintaining competitive visual fidelity and rendering speed. Project page: https://chenjiacong-1005.github.io/ComGS/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper addresses the task of Streaming for Free-Viewpoint Video Reconstruction:\n\nFor this the frames are first grouped (GpF) with fixed size(?).\nThe first frame is reconstructed with 3DGS and the remaining frames of the group are modeled with deforming the Gaussians of the first frame.\nKeypoints with Gaussian influence fields are used to model motion in compact efficient manner.\nError aware masking is used to only make necessary update of parameters between key frames.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The proposed method achieves SOTA compression with very good visual quality\n- An extensive ablation study is performed to show the effect of each component \n- good visualization of components like key point selection which helps to understand the method and gives an intuition on how it works\n- easy to follow, compact mathematical notation\n\nWeaknesses:\n\n- Limitations are in appendix. IMO need to be moved (or short version of it) to main paper\n- No 3DGS compression methods in related work\n- Preliminary is missing the EWA splatting part\n- Evaluation datasets are limited. What about orbital camera datasets?\n\nQUESTIONS:\nI am not familiar with the prior works like QUEEN and 3DGSteam. I think parts of the paper are hard to understand since the proposed methods heavily builds on these works. \n\nNevertheless, most of the paper is well written and I was able to understand almost all of it. \nMy only concern is the section \"Error-Aware Corrector\". \nTo my understanding each keyframe is reconstructed individually (or just the very first one?). How are the Gaussian's matched between keyframes? \nIf the reconstruction is only done once for the first keyframe, then how does the densification work? Or this there none?\nImproving this part of the paper would definitively help to make the whole paper easier to understand. \n\nIf the authors address my concerns regarding that section I will consider raising the score.\n\nLIMITATIONS:\nLimitations are addressed but only in appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the ComGS framework, which optimizes online free-viewpoint video (FVV) reconstruction and transmission through keypoint-driven motion modeling. The core innovations include:\n1. Motion-sensitive keypoint selection, which locating dynamic regions based on viewpoint spatial gradient differences;\n2. Adaptive motion-driven mechanism through propagating keypoint motion to neighboring Gaussian points via a spatial influence field;\n3. Error-aware correction strategy which selectively optimizes error-prone regions in keyframes.\nExperiments show that ComGS achieves significant performance improvements and real-time rendering speed.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis paper proposes many effective methods to improve performance of FVV task, which includes:\n1. The keypoint selection strategy effectively focuses on dynamic regions; \n2. The spatial influence field replaces traditional KNN, adaptively accommodating varying motion scales;\n3. Error-aware correction effectively identifies the Gaussians need to be corrected.\n\nComprehensive comparisons with online/offline methods on mainstream datasets and ablation studies validate the performance of the proposed method and the necessity of each module.\n\n**Weakness**\n\n1. Relies on the rigid motion assumption, which may fail for non-rigid deformations. Besides, fixed number of keypoints is not adapted to varying motion complexity across scenes.\n2. For quality results, only PSNR is reported.  Including more evaluation (SSIM, LPIPS, etc.)metrics can better reflect the performance of the proposed method in the paper.\n\nQUESTIONS:\n1. While Section 3.4 examines selective residual updates, it lacks in-depth discussion regarding the handling of newly emerging objects. Can you further explain how this article addresses this issue?\n\n2. The current methods for FVV are largely confined to existing benchmark datasets and lack validation on challenging scenarios involving non-rigid objects, varying illumination conditions, and rapid motions. Can the method proposed in this article be applied to such scenarios?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a compact and motion-aware Gaussian Splatting framework for dynamic scene modeling. Existing methods suffer from excessive storage requirements due to redundant motion modeling. ComGS addresses this by (1) identifying sparse keypoints (≈200) in motion regions using viewspace gradient differences, avoiding redundant motion modeling; (2) propagating keypoint motion to neighboring Gaussian points via a spatial influence field, enabling shared motion representation and reducing storage; (3) periodically refines only error-prone Gaussian points to mitigate error accumulation without redundant updates. Experiments on N3DV and MeetRoom datasets show ComGS achieves a 159× storage reduction over 3DGStream and 14× over SOTA method QUEEN, while maintaining competitive PSNR and rendering speed.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. By focusing on motion regions and shared keypoint-driven motion, ComGS drastically reduces per-frame storage (0.049 MB/frame vs. 7.8 MB/frame for 3DGStream), enabling real-time transmission.\n2. ComGS maintains high visual quality (PSNR: 33.61 dB) and fast rendering (up to 147 FPS), outperforming baselines in both metrics.\n3. The ablation studies are adequate. \n\nWeakness\n1. The paper lacks the discussion and comparison with highly related works, i.e. Representing Long Volumetric Video with Temporal Gaussian Hierarchy[1], FreeTimeGS[2].\n\n[1] Xu, Zhen, et al. \"Representing long volumetric video with temporal gaussian hierarchy.\" ACM Transactions on Graphics (TOG) 43.6 (2024): 1-18.\n[2] Wang, Yifan, et al. \"FreeTimeGS: Free Gaussian Primitives at Anytime Anywhere for Dynamic Scene Reconstruction.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n2. Keypoint-driven motion works seems similar to KNN-based method like SC-GS, the ablation study for KNN and Adaptive control shows that the performance is comparable.\n\n3. Performance depends on key frame spacing (e.g., ComGS-s uses s=10, ComGS-l uses s=2). Optimal intervals may vary across scenes, requiring manual tuning.\n\n4. If keypoints fail to accurately identify motion regions (e.g., in low-contrast scenes), motion propagation could be inaccurate, degrading quality.\n\n5. The rendering speed drops a lot compared to 3DGStream, QUEEN.\n\nQUESTIONS:\n1. The motion-sensitive keypoint selection uses a gradient difference threshold. How is this threshold determined? Does it adapt to scene complexity?\n\n2. The paper uses 200 keypoints. What is the impact of varying this number (e.g., 100 vs. 300) on storage and fidelity? Is there an optimal range?\n\n3. The words ''spatial influence field'' is confusing. Is it just the covariance?\n\nCurrently my suggestion is boarderline, my concerns and questions are listed above. The comparison with Representing Long Volumetric Video with Temporal Gaussian Hierarchy is necessary, which also focuses on the efficiency of reconstructing dynamic multi-view videos.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces **Compact Gaussian Streaming (ComGS)**, a keypoint-driven method for novel view synthesis of dynamic scenes, applied to free-viewpoint video reconstruction.\nIn the first frame, a static model is trained, and the proposed **Motion-Sensitive Keypoint Selection** identifies which Gaussians should serve as keypoints using the next frame's image. For non-keyframes, scene motion is driven by these keypoints, significantly reducing data storage since only the motion of **N keypoints** (much fewer than all Gaussians) with 14 parameters each is used.\nTo address **accumulated errors**, a new keyframe is introduced every **s frames**, using **error-aware correction**, which updates Gaussians with significant errors. The residual of updated Gaussians will be transmitted.\nExperiments show that ComGS achieves **0.05-0.1 MB/frame storage**, **competitive rendering quality**, and a **7-14x compression ratio** compared to existing SOTA work, QUEEN.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. **Strong performance**: The proposed method significantly enhances the compactness of 3DGStream-like approaches, requiring only **0.05-0.1 MB/frame** for storage. This is a remarkable result with valuable implications for the transmission and distribution of free-viewpoint video (FVV).\n    \n2. **Reasonable and effective design**: The reviewer finds the proposed **Motion-Sensitive Keypoint Selection** and **Adaptive Motion-Driven Mechanism** both novel and effective, making them well-justified design choices.\n\n## Weakness\n1. **Lack of videos for dynamic scene free-viewpoint rendering**: For a paper on dynamic scene novel view synthesis, providing rendering videos is crucial to demonstrate subjective quality. Without them in the supplementary materials, the reviewer cannot assess whether the method is temporally stable.\n    \n2. **Comparison with baselines**: The reviewer noticed that in quantitative comparisons, the MeetRoom dataset does not include QUEEN's rendering results. Similarly, in qualitative comparisons, neither Neu3DV nor MeetRoom includes QUEEN's results. Since QUEEN is now open-sourced and serves as a strong baseline, the reviewer believes it is necessary to update the comparisons to include it.\n    \n3. **Compromised training complexity and rendering speed**: This method increases training time significantly—taking 40 seconds per frame—which suggests it cannot be considered an online reconstruction method. Additionally, rendering speed is noticeably slower, making it a major drawback. On low-end GPUs (e.g., RTX 2080), this method might not support real-time dynamic rendering.\n\nQUESTIONS:\n1. **Training time and rendering speed**: It would be helpful to understand why the training time and rendering speed are impacted. While we understand that the increased complexity may contribute to the performance tradeoff, could you clarify why this is acceptable in the context of the method's goals? \n2. **Support for random access**: Does the proposed method support random access, similar to conventional video playback, where users can start viewing from any arbitrary time point? Based on the description, it seems that decoding subsequent keyframes depends on previous non-keyframes, which might limit random access. If this is the case, could you share your thoughts on addressing this challenge?\n3. **Control points in the background (Figure 4)**: In Figure 4, some control points appear in the background. Could this suggest that the method for selecting control points might need further refinement to improve robustness?\n4. **Ablation study (Lines 261-272)**: The description of the ablation experiment is a bit difficult to follow, and some details seem to be missing. Could you kindly provide a more detailed explanation of the setup and results?\n5. **Rate-Distortion curve**: Since this is a compression-focused work, presenting a Rate-Distortion (RD) curve would be very insightful for evaluating the tradeoff between compression efficiency and reconstruction quality.\n\nIf the authors provide clear feedback for my concerns, I will consider upgrading my evaluation from borderline reject to borderline accept.\n\nLIMITATIONS:\nFrom a compression perspective, the method significantly improves compression efficiency but at the cost of substantially increased training time (which can also be viewed as encoding time). From a practical application standpoint, this cannot be considered an online reconstruction method (even though 3DGStream-like methods tend to make such claims). The reviewer believes that unless subsequent frames (after the first frame) can be reconstructed within 30ms, it cannot be classified as online reconstruction.\n    \nOther limitations are related to the writing and analysis, as detailed in the question section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}