Title: HoliGS: Holistic Gaussian Splatting for Embodied View Synthesis
Venue: NeurIPS 2025
Accepted: yes
Keywords: Embodied AI

Embodied View Synthesis

Gaussian Splatting
Abstract: We propose HoliGS, a novel deformable Gaussian splatting framework that addresses embodied view synthesis from long monocular RGB videos. Unlike prior 4D Gaussian splatting and dynamic NeRF pipelines, which struggle with training overhead in minute-long captures, our method leverages invertible Gaussian Splatting deformation networks to reconstruct large-scale, dynamic environments accurately. Specifically, we decompose each scene into a static background plus time-varying objects, each represented by learned Gaussian primitives undergoing global rigid transformations, skeleton-driven articulation, and subtle non-rigid deformations via an invertible neural flow. This hierarchical warping strategy enables robust free-viewpoint novel-view rendering from various embodied camera trajectories by attaching Gaussians to a complete canonical foreground shape (e.g., egocentric or third-person follow), which may involve substantial viewpoint changes and interactions between multiple actors. Our experiments demonstrate that HoliGS achieves superior reconstruction quality on challenging datasets while significantly reducing both training and rendering time compared to state-of-the-art monocular deformable NeRFs. These results highlight a practical and scalable solution for EVS in real-world scenarios. The source code will be released.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes HoliGS, a deformable Gaussian splatting framework for efficient embodied view synthesis (EVS) from long monocular RGB videos. It models scenes using a hierarchical decomposition into static backgrounds and dynamic foregrounds, incorporating rigid, articulated, and non-rigid deformations via an invertible neural model. By attaching Gaussians to a canonical shape, HoliGS enables robust rendering under complex 6-DOF camera trajectories. The method outperforms prior dynamic NeRF and 4D Gaussian approaches in both rendering quality and speed, achieving real-time performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper introduces a novel invertible neural flow to model non-rigid deformations, which allows for stable and consistent reconstructions over extended time periods\n2. Unlike prior dynamic NeRF methods that struggle with long sequences, HoliGS scales to minute-long videos with reduced training and rendering time.\n\nWeakness:\nMajor concern 1. The framework’s robustness and performance appear to rely on a large number of carefully chosen hyperparameters (in training objective, number of elements in the scenes etc.), which may limit reproducibility and raise my concerns about generalization. For example, in training loss 5 the author seem need to make a balance between different losses. And for each specific scene, the model seems to need to define the number of dynamic elements in the scene.\nMajor concern 2. HoliGS leverages a hybrid initialization involving rigid-body transformation, skeletons, background component transformation. And also use off-the-shelf model to provide depth supervision, optical flow, segmentation supervision etc. While effective, this complexity may hinder adoption and raise concerns about modularity and end-to-end usability. \n\nFor example, in table 4 we could observe that the without depth supervision the result degrades a lot. This makes me questionable about the effectiveness of the invertible deformation model, which should help mitigate the effect of depth supervision.  The model needs to reply on generic pose estimation network etc. to give a decent initialization. I wonder how well the model generalize with respect to these predefined prior of the scenes.\nMinor concern 3. it's also unclear to me how is the static component being initialized, and seperated from dynamic objects.\n\nQUESTIONS:\n1. Can the authors provide a more detailed analysis of hyperparameter sensitivity?\n2. How well does HoliGS generalize without extensive supervisions like depth, flow, semantics etc. which seems not be used in some baseline comparison methods.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents HoliGS, a novel framework for embodied view synthesis (EVS) from long monocular RGB videos capturing dynamic scenes with humans, animals, and complex interactions. It introduces a hierarchical deformable Gaussian representation combining (1) global rigid transformations, (2) skeleton-driven articulated motion, and (3) a fine-grained, invertible non-rigid deformation field. Empirically, the method achieves state-of-the-art performance in both reconstruction quality and speed on multiple challenging EVS benchmarks, including scenarios with complex viewpoint changes, long durations, and multiple deformable agents.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The hierarchical design of the paper is quite interesting. It effectively attempts to disentangle major components of dynamic scenes, including camera motion, rigid motion, and non-rigid deformation. As the authors note in the limitations, a similar disentanglement can be achieved using SMPL-based avatars through parametric modeling. However, the proposed method is notably more generalizable, as it does not rely on specific templates and can be applied to arbitrary dynamic scenes.\n2. The paper is well written, and the experiments are comprehensive. I appreciate the functional comparisons with strong baselines, as well as the inclusion of trajectory visualizations, rendering results, and thorough ablation studies. These collectively provide strong empirical support for the proposed method.\n\nWeaknesses:\n1. The training resolution of the input images is relatively low (480×360). It seems 3DGS frameworks are capable of handling higher-resolution inputs (e.g., 1K). The limited resolution introduces certain issues—for instance, the rendering results of all methods appear somewhat blurry, which makes it harder to assess the visual differences between competing methods.\n2. From the supplementary video, the hierarchical deformation pipeline appears to introduce floater-like artifacts around dynamic objects. Based on my understanding, this may be due to the rigid deformation component dominating the overall motion modeling, thereby making it harder for the non-rigid module to accurately fit the original appearance. While the input resolution could also be a contributing factor, other methods only exhibit general blurriness without such specific artifacts. This suggests that the issue may be inherent in the proposed hierarchical pipeline design. The authors are encouraged to discuss this issue in the rebuttal.\n\nQUESTIONS:\n1. What will the results look like if 1K images (high resolution) are input?\n2. Please cite SMPL in the section limitations.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors proposes a deformable dynamic Gaussian Splatting algorithm to tackle embodied view synthesis problem where inputs are long monocular videos with 6DoF cameras.\nHoliGS leverages invertible Gaussian Splatting deformation network to build scenes as a hierarchy of a background and multiple foreground articulated objects.\nThe experiments show state-of-the-art reconstruction quality over both previous evs algorithm total-recon and general monocular dynamic NeRF/Gaussian baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrength: \n- HoliGS is consistently outperforming total-recon and other monocular dynamic NeRF/Gaussian baselines in terms of reconstruction quality metrics on scenes.\n- close to 10 baselines, and detailed component ablations demonstrate substantial experimental effort \n- Loss terms, hyper-parameters and pipeline diagrams are clearly specified; code release is promised.\n\nWeakness:\n- The pipeline largely mirrors Total-Recon but swaps the NeRF renderer for Gaussian Splatting; the paper does not convincingly articulate what new insights enable the improvements.\n- The authors claimed efficiency advantage multiple times, but comparison with Total-Recon/GS/NeRF under identical hardware is not provided. (Line 48, for example)\n- HoliGS heavily replies on pseudo-labels. Although tracker-free, the method still relies on external monocular depth, optical-flow, and SAM masks (Eq. 3)\n- Key mathematical details (e.g., Mahalanobis weighting, RealNVP architecture) are deferred entirely to the supplement, hindering self-contained understanding.\n\nQUESTIONS:\n- Could you clearly state what's the difference between HoliGS and Total-Recon?\n- Have you seen downsides of replacing NeRFs with Gaussian Splatting?\n- Could you provide direct speed and memory benchmarks vs. Total-Recon and other important methods?\n\nLIMITATIONS:\nNot entirely in the scope of this work, but as a future direction I am really interested in seeing generative priors could help filling in the gap of the monocular video input.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a pipeline for modeling long-range dynamic 3D scenes, specifically targetted towards EVS. The method consists of multiple stages of networks: 1- estimating camera (to background or object canonical space) transformations 2- rigid skeletal transforms 3- soft invertable deformation field for modeling residual movements 3- and a SDF-based gaussian splatting backbone to model the scene geometry. Finally, the background and dynamic object models are composited to make the complete scene which is then optimized through photometric, depth, normal, SDF, segmentation and flow consistency losses.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The method achieves faithful 3D reconstruction of long-horizon dynamic 3D scenes\n- The method is relatively fast to converge compared to previous dynamic 3D reconstruction methods.\n- The method works with a monocular video capture \n- The method allows for EVS for humans and pets as an application useful for AR/VR.\n\nWeaknesses:\n- My main concern is the clarity of the paper in explaining the method in enough detail needed for reproducibility and understanding the full method. In what follows I pose a series of questions that I could not find the answer to in the paper, and I think should be made super clear for the paper to be clear.\n- Does the method need any form of input camera pose? The \"Global movements\" section in 3.1 seems to convey that all camera poses are predicted through the \"intrinsics MLP\", however if that is the case: 1- how are the background Gaussians initialized if not through SFM point cloud? 2- why is there no metrics reported on the accuracy of the recovered global transformations (camera extrinsics)?\n\n- Line 147: is PoseNet the articulation network in figure 3 or is it only used for initializing the rigid transforms? It is not clear if the articulation network is something that you train or just the off-the-shelf PoseNet. \n\n- What is the input to each of the networks depicted in figure 3? most importantly I am curious to know the input and output of the camera MLP, I am not sure what the \"differentiable rendering\" in this figure is outputting exactly, as the Gaussians seem to be sampled in the final stage and not before that?\n\n- In Line 35 you mention that using tracking networks makes most dynamic reconstruction methods heavy, however the proposed method uses depth estimation network, pose estimation network,  flow estimation network, etc. How do these models compare in memory and time consumption to the tracking networks used in G-Marbles for example.\n\n- The OOM reported in results shows a limitation of previous work but does not allow any comparison in terms of reconstruction quality, would it be possible to show a comparison on a shorter sequence where OOM doesnt happen? what is the memory limit for these methods?\n\n- The provided supplemental video is helpful in seeing better RGB quality than Total Recon but in terms of depth map, there seems to be nearly no improvement. Further, the two videos seem to be slightly out of sync (Total recon is a bit slower in speed, showing mismatch in the comparison slider movements)\n\nQUESTIONS:\nPlease refer to the previous section, most of my question are on the clarity of the method.\n\nLIMITATIONS:\nThe limitations are discussed to some extent. I think one main limitation that is left out is not explicitly modeling or analyzing how the model performs in the presence of rigidly moving objects such as cars.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}