HoliGS: Holistic Gaussian Splatting for Embodied View Synthesis
Authors: Xiaoyuan Wang
Yizhou Zhao
Botao Ye
Xiaojun Shan
Weijie Lyu
Lu Qi
Kelvin C.K. Chan
Yinxiao Li
Ming-Hsuan Yang
Venue: NeurIPS 2025 poster ()
Keywords: Embodied AI
Embodied View Synthesis
Gaussian Splatting
Primary area: applications

Abstract: We propose HoliGS, a novel deformable Gaussian splatting framework that addresses embodied view synthesis from long monocular RGB videos. Unlike prior 4D Gaussian splatting and dynamic NeRF pipelines, which struggle with training overhead in minute-long captures, our method leverages invertible Gaussian Splatting deformation networks to reconstruct large-scale, dynamic environments accurately. Specifically, we decompose each scene into a static background plus time-varying objects, each represented by learned Gaussian primitives undergoing global rigid transformations, skeleton-driven articulation, and subtle non-rigid deformations via an invertible neural flow. This hierarchical warping strategy enables robust free-viewpoint novel-view rendering from various embodied camera trajectories by attaching Gaussians to a complete canonical foreground shape (e.g., egocentric or third-person follow), which may involve substantial viewpoint changes and interactions between multiple actors. Our experiments demonstrate that HoliGS achieves superior reconstruction quality on challenging datasets while significantly reducing both training and rendering time compared to state-of-the-art monocular deformable NeRFs. These results highlight a practical and scalable solution for EVS in real-world scenarios. The source code will be released.

HoliGS: Holistic Gaussian Splatting for Embodied View Synthesis
Xiaoyuan Wang'. Yizhou Zhao',
Botao Ye?, Xiaojun Shan Weijie Lyu*,
Lu Qi Kelvin C.K. Chan Yinxiao Li Ming-Hsuan Yang
AMUU
2ETH Zurich
UCC San Diego
*UC Merced Insta360
GGoole DeepMind
Abstract
We propose HoliGS, a novel deformable Gaussian splatting framework that ad- dresses embodied view synthesis from long monocular RGB videos. Unlike prior 4D Gaussian splatting and dynamic NeRF pipelines, which struggle with train- ing overhead in minute-long captures, method leverages invertible Gaussian Splatting deformation networks to reconstruct large-scale, dynamic environments accurately. Specifically, we decompose each scene into static background plus time-varying objects, each represented by learned Gaussian primitives undergoing global rigid transformations, skeleton-driven articulation, and subtle non-rigid deformations via an invertible neural flow. This hierarchical warping strategy enables robust fiee-vie novel-view rendering from various embodied cam- era trajectories by attaching Gaussians to a complete canonical foreground shape perronnerson follow), which may involve substantial viewpoint changes and interactions between multiple actors. Our experiments demonstrate that HoliGS achieves superior reconstruction quality on challenging datasets while significantly reducing both training and rendering time compared to state-of-te sate-of-the-arr art monocular deformable NeRFs. These results highlight practical and scalable solution for EVS in real-world scenarios. The source code will be released.
(e.g., egocentric third-perser
Introduction
Understanding and reconstructing dynamic 3D scenes from monocular video remains fundamen- tal challenge computer vision, particularly in the context Embodied View Synthesis (EVS), where camera trajectories dynamically follow actor motions. EVS tasks crucial for immersive AR/VR experiences, interactive gaming, and robotics, demanding representations capable of handling complex non-rigid deformations, extreme viewpoint changes, and extended temporal sequences.
Despite recent advances in neural rendering for static scenes [1, 2], extending these techniques to dynamic and non-rigid scenarios reveals significant computational and representational challenges. Existing neural radiance fields (NeRF))bbsed methods [3] face high computational costs during both training and inference, particularly when scaling to minute-long sequences and involving multiple interacting objects. This significantly restricts their practical applicability real-time environments.
Gaussian Splatting (GS) approaches [2], known for efficient rendering in static scenes through compact anisotropic Gaussian primitives, also encounter limitations dynamic contexts. Current deformable Gaussian Splatting techniques [4, 5] are typically constrained to short-duration captures or scenarios with minimal non-rigid motion. When applied EVS tasks involving intricate interactions, these methods yield inconsistent reconstructions with noticeable artifacts(see Figure 2).
CCorresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

HoliGS 4D reconstruction
Applications
Remove Harnas
Novel View
Mesh Recon
Depth
Bed's Eye View
3rd Fablow
DogisViee View
Figure 1: Overview. From phone capture of humans and animals in motion, HoliGS reconstructs temporally consistent geometry, appearance, and depth, enabling novel-view synthesis, deformahle mesh recovery, and dense depth estimation. These reconstructions support range embodied applications, including actor-specific view synthesis (e.g., third-person and egocentric perspectives), object-specific removal, and actor-centric visualization (e.g,, dog's s-eye view). HoliGS also enables spatiotemporal behavior analysis such as trajectory visualization.
Entire Deform. Objects
Global
Long
Extreme
Fast Rendering
Method
Traj.
Deform. GS
BANMo
X
X
40XIS
Vidu4D
X
SoM
X
X
X
SC-GS
Dyn.Guss
G.Marbles
Total Recon
S.M
Mosca
Table I: Comparison Related Work. HoliGS targets embodied view synthesis of dynamic scenes and process minute-long videos, and render extreme views.
Figure 2: Performance of SOTA methods.
Furthermore, several existing methods [6, 7, 8] rely heavily on off-the-shelf point-tracking models [5], introducing significant computational overhead and exhibiting fragility under severe occlusions. These methods also fail generalize effectively to arbitrary viewpoint trajectories essential for comprehensive EVS scenarios, severely limiting their utility in real-world conditions marked by frequent occlusions and the need for viewpoint flexibility.
To overcome these critical limitations, we propose HoliGS, holistic Gaussian Splatting method explicitly designed for EVS applications. Unlike previous methods, our framework introduces Gaussian- based deformation model that directly manages articulated non-righ non-rigid transformations without relying on traditional tracking pipelines. This innovation ensures consistent and artifact-free reconstructions across complex sequences involving human and animal interactions.
Specifically, our approach includes a novel deformable Gaussian Splatting pipeline and an optimized strategy maintain high-quality rendering under extreme viewpoint variations, such as egocentric, third-person follow, and overhead perspectives. Additionally, we integrate an invertible deformation model, enabling stable reconstructions over prolonged durations without sacrificing efficiency.
Extensive experimental evaluation demonstrates that HoliGS significantly outperforms state-of- the-art methods in terms of both rendering quality and computational speed, achieving real-time rendering capabilities on consumer hardware. Our results confirm robust performance across diverse, challenging, dynamic sequences featuring multiple interacting entities and complex articulated motions, scenarios where prior techniques either fail or produce substantial visual artifacts. The main contributions of this work are:
We introduce holistic Gaussian Splatting method for EVS tailored to 6-DOF embodied camera paths, outperforming existing state-of-the-ar approaches [3, 9].
We propose invertible deformation model that ensures stable reconstruction over extended periods without compromising computational efficiency.
We evaluate our model on diverse challenging dynamic scenes against existing methods and show that our approach achieves robust view synthesis and scalable to minute-long videos.

---

2
Related Work
Dynamic Scene Reconstruction. Reconstructing dynamic scenes from videos has been active
research area, traditionally relying on multi-view stereo systems [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 231. Recently, another series of works focus monocular scene reconstruction methods [24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43]. Dynamic methods often utilize either temporal conditioning as additional input dimension [44] canonical-space representations with deformation fields [45, 25, 46). Grid-based representations [47, 48] have further accelerated these methods, enabling efficient optimization for dynamic scene reconstruction [17, 49, 50]. Despite significant progress, these approaches still suffer from high computational costs, especially in real-time and long video scenarios with complex motion patterns or prolonged video sequences.
Embodied View Synthesis (EVS). EVS introduces additional complexity, requiring representations capable of handling camera trajectories that closely follow or interact with dynamic subjects. Existing methods like DyCheck [51] highlight the inadequacies of current benchmarks, which often do not accurately reflect realistic everyday scenarios involving limited viewpoints and complex dynamics. Methods designed specifically for monocular EVS [52, 3, 53] aim mitigate these issues through hybrid representations or generative methods. Nevertheless, these methods typically rely heavily on domain-specific priors or computationally intensive tracking modules, restricting their robustness under occlusions and generalization across diverse view trajectories.
Articulated Object Reconstruction. Articulated object reconstruction, especially for humans and animals, often utilze parametric templates [54, 55, 56, 57, 58J. which impose strong geometric priors and facilitate reconstruction from sparse views or monocular vidcos [59, 60, 61, 62]. However, these models typically struggle with capturing personalized or detailed appearance variations. More recent non con-aramettric neural methods have combined neural radiance fields with articulated models [63, 64, 65, 66, 67, 68, 69, 70, 71J. capturing richer detail but at a significant computational cost. Our method diverges by directly modeling articulated motion without relying predefined parametric templates, instead employing a flexible Gaussian-bast based deformation model optimized for dynamic reconstruction.
Non-Rigid Structure from Motion. Non-rigid Structure from Motion (NRSIM) aims to reconstruct the 3D shape and deformation of objects from monocular videos, handling scenarios where scene points undergo complex, articulated, or continuous deformation. Traditional SfM and visual SLAM methods [72, 73, 74] typically assume static environments, enforcing strict epipolar constraints unsuitable for dynamic scenes. Recent methods address this limitation by jointly estimating camera poses, scene geometry, and deformation fields [6). These approaches, however, often rely on time-intensive test-time optimization or explicit motion segmentation, limiting their scalability and efficiency. Differently, our method leverages Gaussian- deformation model to explicitly encode articulated non-rigid transformations, enabling efficient reconstruction without the need for computationally costly per-video fine-tuning explicit motion segmentation. This approach facilitates robust reconstruction of dynamic interactions in everyday monocular videos, effectively overcoming challenges posed by occlusions and extensive deformation.
The proposed framework, HoliGS, combines the advantages of articulated object reconstruction and static Gaussian Splatting to enable efficient, high-quality embodied view synthesis for dynamic scenes captured from monocular videos, overcoming limitations associated with existing methods.
3 Method
In this section, we introduce HoliGS, hierarchical 4D representation that models dynamic scenes as the union of a static background and time-varying deformable objects. Our framework leverages Gaussian Splatting represent both the static and dynamic components and employs series of invertible warping operations to capture articulated and non-rigid deformations. The final scene at time t is given by S(t)=G(t) U H. where H is the set of static background Gaussians and ((t) contains the dynamic, vare-ng Gaussians splitting articulated foreground objects.
3.1 Hierarchical Dynamic Warping
To robustly model motion ranging from whole-body translations to fabric flutter, we use two-stage warping strategy. At glance, large articulated displacements are first explained by skeleton-driven
3

---

RGB Video
Pre- processinnn: Monocular Cues & Pose Estimation
Neural SDF Initialization
Canonical Gaussian Scene Representation
Sytoco
Aonaat
X
Flow Network
PosuNet
Lueo
Eeyzte
SAM
Depth
Y r
Static background Gaussians 3
Canonical dynamic Gaussions G
Figure 3: HoliGS Pipeline. Warping network initialization: We jointly optimize poses, articulation, soft deformation, and neural SDF proxy to obtain a fast converging deformation field that provides strong starting point for Gaussian splitting. Deformable GS: the objective is switched to dynamic Gaussian splatting, and the deformed foreground is composited with the static background to yield the final 4D scene.
transform, after which soft, flow-based deformation field refines any residual non-rigid detail. All derivations and exact matrix expressions are deferred to the supplementary material.
Global movements. Every video frame is aligned the camera via two rigid SF(3) transforms: the background-to-cuee torcaeer- map G, and the object-root root-toccameaa map G,, Both transforms are regressed by lightweight Fourier MLPs that output six twist parameters per frame, giving us frame-specific poses without needing an external tracker.
Skeleton-driven warping. The core articulated motion is handled by a bone hierarchy with B bones. Each bone has static reference pose (ct, V2 AA,)) encoding center, rotation, and scale, respectively. At time I, learned twist vector (1) (1)
SE(33 exponentiated to produce the bone pose Jz((). We measure how much 3-D point P% belongs each bone by Mahalanobis distance in the bone's scaled-rotated frame; softmax over these distances yields skinning weights w(t). Dual-quaternion blend skinning (DQB) [77] fuses the individual bone transforms into single SE(3) map (t), which then applied to every Gaussian center, rotation, and scale. Conceptually, this step captures all seliddbbi-articulated effects such as limbs, torsos, tails.
Soft deformation field. After skeletal warping, many objects still exhibit subtle surface changes- blosees-loose clothing. hair swaying, muscle bulges-that that cannot be explained by rigid bones. We address this with soft deformation field 5(,wwz) implemented as an invertible RealNVP flow [78]. Given canonical point X and per-frame latent code Wds the field outputs refined position X' = 5(x, wa). Invertibility guarantees that S exists; we therefore impose 3-D cycls-consistency loss: Laye S (SS-s(((X,,))))))) wa). wa) - X12 which forces the forward and reverse mappings to cancel out and stabilizes training. Because the flow operates in a fixed canonical space, never has to chase moving target, allowing it to converge quickly even when the deformations are highly nonlinear.
Why hierarchy matters. Articulated bones give the model inductive bias toward plausible large- scale motion, while the soft field soaks up the remaining fine detail. Each module solves a simpler task and therefore converges faster than a single, monolithic deformation network. Empirically, the skeletal stage explains 2 90% of visible motion energy, leaving only low-amplitude corrections to the RealNVP field. Full mathematical details-the Lie-algebra twist representation, the exact Mahalanobis weighting, and the DQB formulation are provided in the supplementary materials.
Combined warping pipeline. Integrating the above components, point X in canonical space is warped to its dynamic position at time & according to:
w/). Inspired by Omnimotion [79], HoliGS also enables a forward warp
x== = Jt- S-
(X' .
X Jr Go (x(xx, w³).
(2)

---

This unified warping function seamlessly integrates global, skeletal articulation, and sinelscale deformations, enabling our framework to render high-quality 4D scenes with complex dynamics.
3.2 Deformation Network Initialization
For our dynamic scene representation, we establish initial transformation parameters by pre-training a neural SDF that warps sampled points on camera rays from the static state the warped states, similar to [68]. We apply Posenet [80] to obtain the rigid-body transformations 7 and time-dependent skeletons for each deformable object in the scene. This network provides robust pose estimates even under challenging viewing conditions. Concurrently, we initialize the background component transformations 7 using camera pose information extracted from the capture device's 's motion sensors. This hybrid initialization strategy ensures stable convergence during subsequent optimization stages while accommodating both foreground dynamic objects and static background elements within our unified representation. Then, we initialize the foreground Gaussian point cloud from the pre- pre-trained neural SDF by sampling points on its surface, with objective function:
L =
phott + Adopth Eeepth + ASDE LSDF + Afoo Lnow + Ceycle + Lang
photometric consistency
goometric
motion
mask supervision
constraints
consistency
Here, the photometric loss Cphoto enforces appearance consistency. For geometry constraints: the depth term Edepth==, shelf monocular depth estimator D [81], promoting correct scene scale, and the SDF term ESDF=Ex; vx; SDDF (xx))l2-14 enforces the signed distance field ØSDF to behave like true distance funetion by constraining its gradient norm to one. Motion consistency imposed by flow loss Citw= Ep (VV (pt) and cycle loss where
aligns our predicted depth D with an off-the-
Leycle Σ
2 Big ( (Fbaa, xx 4J
weighted by importance factors X, and Segs aligning RAFT optical flow [82] and satis- fying forward backward cycle consistency. Finally, segmentation supervision is given by f Mred (pr)aar" ( with Mgt obtained from SAM [83]. p² E R²² represents pixel coordinates at time f, X E R denotes the i-th sample point in world space corresponding to X R in camera space. Weights {Adopth, ASDE, Aflow, A tuned balance these comple-
mentary constraints.
3.3 Deformable Gaussian Splatting Optimization Objectives
Our composite Gaussian Splatting representation incorporates N scene elements, global transforma- tion matrices Ti, and bidirectional deformation fields F Fiorwand and Fiackwand The optimization process integrates multiple objectives to ensure high-quality reconstruction and temporal consistency:
Lphato + Laepth + Lreg + Lnormal-
Besides the loss terms we explained in initialization, Lphoto- Ldepth. and Lieg. we incorporate additional normal supervision to align the estimated entire scene surface normals with observed ones Coormal Σp. [p'( (pp')) II?. This comprehensive optimization framework ensures geometric accuracy, appearance fidelity, and temporal consistency in our dynamic scene representation.
3.4 Embodied View Synthesis
To effectively perform EVS, HoliGS transforms dynamic 3D Gaussian primitives into consistent, egocentric viewpoints that naturally follow the motion of articulated objects, such humans and animals. Specifically, for each Gaussian primitive, we apply a forward warping function We-p
X X,. which maps points from canonical space X" to the deformed configuration time t. This deformation accounts explicitly for non-rigid articulated transformations, ensuring accurate representation of complex motions such limb articulations or interactions among multiple entities.
Subsequently, to achieve embodied viewpoints, we employ a rigid-body transformation Gp, posi- tioning the virtual egocentric camera within the world coordinate system. It aligns the viewer's
5

---

RGB
BEV Trajectory
Animal Trajectory
Third Person's ' Traj
Embodied Pose
Human & Cat
Human & Doyg
Figure 4: Foreground Embodied Trajectory. For challenging sequences, HumanCat and HumanDog, show: (i) the joint bird's s-yee-view (BEV) trajectory a foreground actor, (ii) the articulated animal trajectory, (iii) the articulated human trajectory, and (iv) both objects' embodied camera pose. Our method recovers smooth, collision free paths that faithfully follow each actor while remaining mutually consistent, enabling stable first-person over- the-shoulder replays for complex multi- agent interactions.
perspective with the foreground, enabling realistic rendering of scenarios such as first-person views or third-person perspectives following actors in motion (illustrated in Figure 4).
By integrating the deformation network, our method reliably synthesizes novel embodied viewpoints that remain coherent across complex motions. Our unified Gaussian-based deformation and viewpoint adjustment strategy significantly simplifies optimization and achieves near real-time performance. This enables practical usage interactive AR/VR applications, immersive gaming experiences, and robotics, where rapid viewpoint changes and accurate motion tracking are essential.
4 Experiments
4.1 Training and Optimization
We adopt a two-phase procedure to optimize our dynamic Gaussian representation: Component pre-training and joint refinement. During pre-training, each component (e.g., a deformable object or the static background) is optimized separately. Once pre-training is completed, all components are combined for joint refinement using color, depth, normal, and mask objectives. Training follows standard Gaussian Splatting protocols [2]. The synergy between our deformation-come centriiccccee design and the parametric Gaussian framework accelerates convergence considerably. On NVIDIA H20 GPUs, each pre-training or refinement stage completes in about 30 minutes, enabling full scenes (including multiple deformable objects) to converge two hours, significantly faster than other approaches.
Component pre-training. We initialize the deformation network by minimizing the overall loss (3), with default weights as: Adepth = 5 (or 1.5 for the HUMAN sequence), Acolor 0. >stow Asyele = 1, and Asegment = 1. This eikonal term is weighted by ÄspF 0.001 to ensure proper geometric properties. For this computation, we sample 17 uniformly distributed points X along each camera ray r' centered at the surface point derived from back-projecting the ground-truth depth.
Joint fine-tuning. During the joint optimization phase, simultaneously refine all object representa- tions by minimizing loss (5) for an additional 6,000 iterations. The default weights for these objectives Aphoeo 1, Anermal 1, Adeph 5. and \ 1. By default, freeze the background's appearance and geometry parameters while allowing optimization of its global transformation T8.
the foreground objects' transformations TV. and the foreground appearance and geometry parameters (for HUMAN 1, we use Adepth = 1.5), we allow background appearance and geometry optimization during joint fine-tuning). This joint fine-tuning phase significantly enhances the visual coherence of foreground elements and improves the modeling of inter-object interactions.

---

(VS
Ous
Total Recon
Deform di
NYS
Ours
Total Recon
Deform
Jaman
Dog
Humai & Ca
Call
Humer & Dog
CH22
Figure 5: Baseline Comparison. We qualitatively compare HoliGS against four SOTA baselines and direct NVS ground-truth reference across Dog 1, Cat 1. Human 1, and the challenging multi-actor Human & Cat sequences. Each column shows photometric renderings (top) and corresponding depth reconstructions (bottom). Red inset boxes highlight the most error-prone regions for articulated motion and occlusion (e.g- tail swing, paw lift, garment folds, and human animal interaction). Compared with baselines, HoliGS better preserves fine grained appearance and yiclds geometrically consistent depth maps with fewer tearing or bloeding artifacts especially under large viewpoint changes and prolonged, highly non-rigid deformations.
DoG (V1)
Don' (V3)
CAT (V1)
CAT (V2))
CAT (.(1))
CAT (V
(626 imuges)
(531 images)
(641 images)
(632 images)
(834 images)
(901 umages))
LPIPS, PSNRR SSIM LPISS]
PSNRR: SSIMT LPISS]
PSNR SSIM: LPIPS) PANRR SSIM LPEPS PSINR SSIM
LPPS5]
PSNRR) SSEM1
HyyenNeRF
634
12.84
.573
-432
14.22
.721
.221
14.80
.532
-135
14.57
.507
.641
12.02
.832
-307
15.05
.637
D NoRF
540
13.57
.001
5.16
11:71
4655
867
10.02
5545
366
11.88
-548
3366
12.55
.955
-545
12.71
.9/1
HypenNeRR ter depeh)
377
16.36
7700
-25
16.35
7740
5:32
14:37
J021
-371
15.05
-017
330
18.47
7728
3776
10.06
(670)
D' NeRF our depdit
.007
18.44
.608
852
11.88
.000
685
10.81
5534
.500
12.00
-568
12.00
.630
.563
12.76
.620
Toaa Recon (wr depmy
1771
17:00
313
17.78
778
399
15:77
1537
3333
16:64
1032
137
25.00
799
18.52
71/5
Detomaatee as twl depth)
.:a
12.35
12.77
.500
383
11.02
2018
.399
12.30
.410
600
11.50
-081
.10
12:00
4.20
depth)
12:40
11:55
11:45
Ours
21.12
21,37
.791
319
28.52
711
383
21.74
.03
22.94
.083
21007
.763
(S5H
(43S
(992
CAT
MIAAN
LPIPS., PSNRT SSIMI
LPISS, PSNR"
SSIMT
LPIPS)
PSNR.
SSIM
LPUSS PSNR1
SSEM)
LPPESS
PSNR1
SSIM!
LPPPS)
PSNR SSEM1
HyperNeRt
2113
13.77
8524
6532
11.99
.3035
.555
14.17
.220
-197
15.14
(059
-102
13.02
.5312
.511
14.100
1635
D NcRF
750
11.00
.578
.658
11.58
.638
6.30
12.13
.509
-576
12.4.
.652
-625
10.43
.458
.61.
11.97
.608
HyperNaRR (erd)))))
514
10.30
8335
301
13.25
7061
-115
13:58
.005
-150
15.00
7704
1050
11..30
1535
-116
15,90
1957
D' NeRF owr dapth
730
11.08
.582
.555
12.14
-338
600
12.11
.612
.685
12.30
-638
-645
10.53
.451
.099
12:01
.611
Tees--Raas (wr depti)
261
19.90
-791
213
18.30
-775
261
16.73
-712
-356
16:00
-750
-333
17.07
1030
.07
15:11
770
Defonmable aa twl depth)
.550
12.45
.410
.608
12.50
.430
.500
11.05
.000
.540
12.30
.420
-500
11.70
.830
.542
12.22
.413
4DGS twt depth)
.645
12,50
.155
-530
12.75
.415
585
11.00
.400
-335
12:13
.425
.000
13.65
-985
.545
12.19
.488
05-mrthic
.005
13.52
11.85
711
241
Stape
16.78
775
253
16.75
708
230
17.00
775
Table 2: Quantitative Comparisons on Novel View Synthesis (Visual Metrics). We compare our method to previous dynamic NVS works and their depth-supervised variants on the 11 sequences of our stereo RGB datasct in terms of LPIPS, PSNR, and SSIM. Our method significantly outperforms all baselines for all sequences.
4.2 Qualitative and Quantitative Results
Figure 5 shows representative visualizations comparing the photometric and depth reconstruction quality of HoliGS against Total- Recon [3], Deformable GS [84], and 4DGS [85]. These results demonstrate the superior performance our method under various challenging conditions.
Quantitative results for novel view synthesis are reported in Tables 2 and 3. Table 2 presents visual metrics across the Total-Recon dataset, while Table reports depth accuracy metries (Ace@0.1 Im and RMS depth emror). Our method consistently outperforms the basclines in both sets of metrics.
7

---

Doo
(V2)
(V2)
Cxt
HUMAN
HUMAN
HUMAN
HUMAN Cxll
MEAN
tever
Acdf
AssT hayal
AacTravel
rauell Acdft
taya]
haya]
Noyal
fwyev
Noyal
HypoNNRRF
1887
.044
.277
8000
.511
1,005
.855
D"N
346
.14
833
-231
623
1.063
.830
2417
HypeiNeRt
.301
200
.451
245
201
.611
DeNeRF
-338
.123
.45
5:0
325
362
313
435
288
-70
318
-243
-96
56.1
8133
1.55
-790
176
-757
300
Toaal Recon
1841
1165
.790
.167
889
188
1994
124
967
1000
925
.981
.949
0995
2909
.142
J849
.142
.827
.204
.944
.104
.895
.111
DetGs .172 .209
.181
.612
.520
.415
.128
.482
.955
.855
.271
.694
.225
.598
1070
.912
.108
.940
.085
.802
.145
.798
.215
.622
BDGS
.175
003
1178
1620
.315
.423
.125
4286
202
481
.288
.496
.232
.002
1073
508
1133
1995
1089
.809
142
.002
.20
.61
GS-anable
-- 00M
1180
.615
-- 0OM
00M
00M
DOM
0OM
.155
.710
-210
.835
.187
.001
143
.790
NA
MoSca
00M
7722
366
0OM
00M
00M
00M
00M
006
1830
1143
222
201
0122
1006
NA
S.M
00M
7768
.168
-- 0OM
00M
00M
-- 00M
-- 0OM
208
.144
J845
.145
3525
2016
9911
.106
NA
Ours
845 100 7955 1163
.888
190
8988 122
970 048
9228 1979
1955 0664
915
1338
1835
139
830
2012
.920
102
9901 127
Table 3: Quantitative Comparisons on Novel View Synthesis (Depth Metries). We compare HoliGS to previous works on the Recal-Rccon dataset in terms of the average accuracy at 0m (Acc@ 0. Im) and the RMS depth error Edepar (units: meters), Our method significantly outperforms all baselines for all sequenees.
Full model
deform 6j
wio Enorra!
w/o Eitepth
w/o roal- roal-body init.
wio deforms
Haran 1
Cla
Human de Cat
Figure 6: Ablation Studies Visualization. Qualitative impact of removing soft deformation, normal/depth supervision, root body initialization, and skeleton deformation. Each omission introduces increasing blur, drift, and silhouette break-up. whereas the full model remains sharp and stable.
Table 4 and Figure 6 evaluate the contribution of each deform component systematically removing key elements: the depth supervision, the normal supervision, the deformation field Ft, soft deformation S. pose initialization from external estimators, and the rigid transformation T where j identifies a deformable object. For all ablations, we maintain the same core optimization objectives used in our full method while initializing camera parameters 7, from device sensors. For configurations without rigid body modeling, we initialize each object's 's pose with predictions from PoseNet and optimize them during reconstruction; for row we replace these predictions with identity transformations.
Geometric supervision. Table demonstrates that removing depth supervision (row 2) significantly reduces average accuracy. Figure 6 reveals that this stems from scale inconsistency between objects while removing depth supervision does not severely impact training-view RGB renderings, introduces critical failure modes in novel-view reconstructions: (a) floating foreground objects, evidenced by misaligned shadows, and (b) incorrect occlusion relationships between subjects. Without depth supervision, our method overfits to training perspectives and produces a degenerate scene representation where objects fail to maintain consistent scale relationships.
4.3 Ablations Studies
Similarly, our results show that normal supervision (row 3) provides crucial geometric guidance. Without normal constraints, the model struggles to capture fine surface details and produces less
8

---

Depth Loss Normal Deform. Ohj.
Mation Deform,
I.PIS A.c@0.Imm
Full model
.263
.896
who deppth
x
.385
loss Lnomaal
.288
(4) w/o deform. J
v
.305
.833
(5) w/o Soft deform G
V
X
.293
.870
(6) w/o root hody init.
X X
X
.301
.862
w/o root-body Gj
X
N/A
N/A
Table 4: Ablation Study. Removing depth supervision significantly hurts performance, while removing the deformation field (3) and AiseNet-initializttion root-body poses (4) hurts moderately. Most importantly, removing root-body poses entirely (5) prevents convergence (N/A) as the deformation field alone has to explain global object motion (see Figure 1). These experiments justify our hierarchical modeling of motion, as even root-bodies without deformation field or poorly initialized root-bodies (4) better than root-bodies (5). We visualize these ablations Figure and explore other ablations the Appendix.
coherent object boundaries, particularly in regions with complex geometry. The normal supervision helps maintain surface continuity and improves the definition of sharp features.
Deformation modeling. Table 4 indicates that eliminating the deformation field (row 4) substantially degrades performance. Without this component, our approach must explain non-rigid motion using only rigid transformations, resulting in coarse approximations that fail to capture articulated move- ments like limb motion. The MLP-based soft deformation component (row 5) further enhances model's s ability to represent complex non-rigid movements through the transformation (1).
Similar to established approaches, our method enables bidirectional warping, with the inverse transformation defined as (2). This hierarchical structure allows our model handle both global positioning and local deformations effectively. Removing the neural soft deformation component results in notable artifacts around joints and other highly articulated regions.
Removing pose initialization from external networks (row 6) produces similarly detrimental effects, leading noisy appearance and geometry artifacts. Most significantly. Table shows that eliminating object-specific rigid transformations entirely (row 7) causes optimization failure (N/A), even though the deformation field and soft deformation components can theoretically represent all continuous motion. It proves challenging for deformation fields alone to model global positioning, as such movements can deviate substantially from canonical configurations, complicating convergence. These findings justify our hierarchical motion representation, which explicitly models object positioning through rigid transformations while capturing non-rigid deformations through combination of MLPs. Our ablations further suggest that the underwhelming performance of baseline methods on challenging dynamic scenes may stem from insufficient object-centric motion modeling.
4.4 Efficiency on Long Sequences
Beyond reconstruction quality, we evaluate computational efficiency minute-long videos (~1000 frames). Methods that rely on dense point-tracking must correlate tens of thousands of features over long temporal windows, which drives memory consumption and latency unfavorably with sequence length. By contrast, our pipeline performs lightweight per-frame pre-processing (depth, optical flow, segmentation, root-body pose) and reconstruction stage whose cost grows primarily with scene complexity (number of deformable objects and Gaussian budget), rather than the number of frames.
The measurements in Tables 5-6 indicate that our end-to-end memory footprint remains below typical single-GPe limits and that the dominant costs are embarrassingly parallel across frames. Practically, this enables stable optimization on long clips with extensive articulation and frequent occlusion/reveal events, without resorting to sequence chopping or frame subsampling.
4.5 Short-Clip Evaluation for Fairness
Several baselines cannot process long sequences due to memory constraints. To ensure fair com- parison, we additionally evaluate on 200-frame windows with a 100-frame stride for videos that would otherwise OOM. This short-clip protocol removes any long-range temporal advantage while preserving realistic motion patterns and occlusion cycles. The comparison results are recorded in Appendix 8

---

Stage
Component
VRAM (Peak)
Notes
Pre-processing
Depth (UniDepth)
122B
Run once per video
Optical Flow (RAFT)
6BB
Rum once per video
Segmentation (SAM, VIT-H)
~6
Run once per video
Pose (PoseNet)
< 1 GB
Run once per video
Main Training
HoliGS (Rcconstruction)
10
Scales with scene complexity
Baseline
Dense point tracking (CoTracker)
Scales with points x frames
Table Peak VRAM by stage on a 10000-framme video, All pre-processing modules are single-pass, and reconstruction stage maintains a modest footprint. Dense tracking can exceed 80 GB and OOM on long clips.
Component
Method
Time (per ~ 1000 frames)
Notes
Depth Estimation
UniDepth
15 min
Offline: per-frame:; parallelizable Efficient
Optical Flow
RAFT
~ 10 min
Segmentation
SAM (ViT-II)
min
Offline; parallelizable
Pose Estimation
PoseNet
< 1 min
Near real-time
Dense Point Tracking
CoTrackerV2
~00 min
Long temporal windows
Table 6: Wall-clock time ~1-minute (~1000-frame) video. Pre-processing is feed-forward and paralleliz- able across frames; dense tracking is the slowest step due long-range correspondence search.
Under this short-clip protocol, HoliGS remains competitive or superior across most sequences and metrics. The trend-slightly higher perceptual similarity for MoSca but stronger photometric (PSNR/SSIM) and geometric (Acc) fidelity for HoliGS-suggee better radiance-gcometry consis- tency and reduced temporal drift from our globally consistent canonical modeling and joint pose refinement.
5 Conclusion
In this work, we have presented a novel approach for embodied view synthesis from monocular RGB videos, with particular focus dynamic scenes featuring humans interacting with animals. Our primary technical contribution is deformable Gaussian splatting framework that hierarchically decomposes scene dynamics into object-level motions, which further decomposed into rigid transformations and localized deformations. This hicrarchical structure enables effective initialization of object poses and facilitates optimization over long sequences with significant motion.
Future Work. We aim to integrate event-aware sensors (c.g., event cameras or high- high-frame- frame-rraaat -aat IMUs) better capture motion discontinuities. We also plan to couple the warping network with a lightweight, on-the-fly bootstrap module that refines pose and Gaussian splitting priors across diverse articulated objects, including humans, animals, and furniture. To support real-time embodied view synthesis on mobile platforms, we will improve our splitting kernels and memory layout for deployment AR glasses and edge devices and integrate reinformancement learning continuously improve model performance [86, 87].
Limitations. Despite the demonstrated effectiveness of our approach, generic pose estimation sometimes mis- match the anatomical accuracy of specialized parametric models such as SMPL [88] for humans, which offer more robust initializations and appropriate joint constraints.
Acknowledgments
This work was supported by a Sony Faculty Award, the Institute of Information & Communications Technology Planning & Evaluation (ITP) grant funded by the Korean Government (MSIT) (No. RS-2024-00457882, National AI Research Lab Project), and the Department of Defense grant -911NF-241-0295.
10

---

References
[1] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 2021.
[2] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 2023. 1,
[3] Chonghyuk Song, Gengshan Yang, Kangle Deng, Jun-Yan Zhu, and Deva Ramanan. Total- recon: Deformable scene reconstruction for embodied view synthesis. In ICCV, 2023, 1, 2, 3,
7
[4] Qingming Liu, Yuan Liu, Jiepeng Wang, Xianqiang Lyv, Peng Wang, Wenping Wang, and Junhui Hou. Modgs: Dynamic gaussian splatting from casually-captured monocular videos. In ICLR, 2025.
[5] Nikita Karaev, lurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:24J0. 1831, 2024. 1,2
[6] Qianqian Wang, Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from single video. arXiv preprint arXiv:2407.13764 2024.
[7] Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Daniilidis. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. arXiv preprint arXiv:2405. 174211
2024. 2
[8] Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari, Gordon Wet- zstein, and Leonidas Guibas. Dynamic gaussian marbles for novel view synthesis of casual monocular videos. In SIGGRAPH Asia, 2024. 2
[9] Gengshan Yang. Shuo Yang, John Zhang, Zachary Manchester, and Deva Ramanan. Ppr: Physically plausible reconstruction from monocular videos. In ICCV, 2023. 2
[10] C. Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. High-quality video view interpolation using a layered representation. ACM TOG, 2004. 3
[11] Timo Stich, Christian Linz, Georgia Albuquerque, and Marcus Magnor. View and time interpo- lation in image space. Computer Graphics Forum, 2008.
[12] Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel. and Tobias Ritschel. X-fields: Implicit neural view-, light- and time-image interpolation. In SIGGRAPH Asia, 2020.
[13] Aayush Bansal, Minh Vo, Yaser Sheikh, Deva Ramanan, and Srinivasa Narasimhan. 4d visualization of dynamic events from unconstrained multi-view videos. In CVPR, 2020. 3
[14] Tianye Li, Mira Slaveheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, al. Neural 3d
video synthesis from multi-view video. In CVPR, 2022. 3
[15] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021. 3
[16] Sara Fridovich-Keil, Giacomo Meanti, Frederik Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. arXiv preprini arXiv:2301.10244 2023. 3
[17] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. arXiv preprint arXiv:2301.00633 2023.
[18] Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew Toooe, and Changil Kim. Hyperreel: High-fidelity 6-dof video with ray-conditioned sampling. In CVPR, 2023. 3
[19] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308. 09330880973... 2023.
[20] Xin Lin, Shi Luo, Xiaojun Shan, Xiaoyu Zhou, Chao Ren, Lu Qi, Ming-Hsuan Yang. and Nuno Vasconcelos. Hqgs: High-quality novel view synthesis with gaussian splatting in degraded
scenes. In ICLR. 3
11

---

[21] Wenxuan Xu, Yushi Wei, Xuning Hu, Wolfgang Stuerzlinger, Yuntao Wang, and Hai-Ning Liang. Predicting ray pointer landing poses in vr using multimodal istm-based neural networks. In 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR), 2025. 3
[22] Yizhou Zhao, Haoyu Chen, Chunjiang Liu, Zhenyang Li, Charles Herrmann, Junhwa Hur, Yinxiao Li, Ming-Hsuun Yang. Bhiksha Raj, and Min Xu. Masiv: Toward material-agnostic system identification from videos. arXiv preprint arXiv:2588.01 12, 2025. 3
[23] Yixing Lu, Junting Dong, Youngjoong Kwon, Qin Zhao, Bo Dai, and Fernando De la Torre Gas: Generative avatar synthesis from a single image. arXiv preprint arXiv:2502.06997, 2025. 3
[24] Jae Shin Yoon, Kihwan Kim, Orazio Gallo, Hyun Soo Park, and Jan Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In CVPR, 2020.
[25] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
[26] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In CVPR, 2021.
[27] Chaoyang Wang. Ben Eckart, Simon Lucey, and Orazio Gallo. Neural trajectory fields for dynamic novel view synthesis. arXiv preprini arXiv:2105.08 059944, 2021.
[28] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In ICCV, 2021. 3
[29] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In ICCV, 2021. 3
[30] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D2 nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. arXiv preprint arXiv:2205. 15838. 2022. 3
[31] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed
neural radiance fields. IEEE TVCG, 2023. 3
[32] Fengrui Tian, Shaoyi Du and Yueqi Duan. Mononerf: Learning a generalizable dynamic radiance field from monocular videos. In ICCV, 2023. 3
[33] Meng You and Junhui Hou. Decoupling dynamic monocular videos for dynamic view synthesis. arXiv preprint arXivv2304.01771 2023. 3
[34] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint 0ttt:22309.13000 2023. 3
[35] Yiqing Liang. Numair Khan, Zhenggin Li, Thu Nguyen- Phuocc Douglas Lanman, James Tompkin, and Lei Xiao. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis. arXiv preprint a1Xi1:2312.11458, 2023. 3
[36] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung- Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In CVPR, 2023. 3
[37] Minh-Quan Viet Bui, Jongmin Park, Jihyong Oh, and Munchurl Kim. Dyblurf: Dynamic deblurring neural radiance fields for blurry monocular video. arXiv preprint arX1v22312..3358,
2023. 3
[38] Xingyu Miao, Yang Bai, Haoran Duan, Yawen Huang, Fan Wan, Yang Long, and Yefeng Zheng. Cunerf: Cross-time transformer for dynamic neural radiance field from monocular video. arXiv preprint arXiv:2401. 008611 2024. 3
[39] Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. Sampling: Scene-adaptive hierarchical multiplane images representation for novel view synthesis from a single image. In ICCV, 2023. 3
[40] Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In CVPR, 2024. 3
12

---

[41] Yuheng Liu, Xinke Li, Xueting Li, Lu Qi, Chongshou Li, and Ming- Ming-Hsuan Yang. Pyramid diffusion for fine 3d large scene generation. In ECCV, 2024. 3
[42] Yuheng Liu, Xinke Li. Yuning Zhang, Lu Qi, Xin Li, Wenping Wang, Chongshou Li, Xueting Li, and Ming- Asuaa Yang. Controllable 3d outdoor scene generation via scene graphs. arXiv preprint arXiv:2503.07152 2025.
[43] Yizhou Zhao, Hengwei Bian, Kaihua Chen, Pengliang Ji, Liao Qu, Shao-yu Lin, Weichen Yu, Haoran Li, Hao Chen, Jun Shen, et al. Metric from human: sero-shot monocular metric depth estimation via test-time adaptation. Advances in Neural Information Processing Systems,
37:104724-104753, 2024. 3
[44] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM TOG, 2021.
[45] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualle Nerfies: Deformable neural radiance fields. In ICCV, 2021.
3
[46] Haoye Dong, Aviral Chharia, Wenbo Gou, Francisco Vicente Carrasco, and Fernando D De la Torre. Hamba:
Single-view 3d hand reconstruction with graph-guided bi-scanning mamba. Advances in Neural Information Processing Systems, 2024. 3
[47] Sara Fridovich- Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022. 3
[48] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance
fields. In ECCV, 2022. 3
[49] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIG- GRAPH Asia, 2022. 3
[50] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In CVPR, 2023. 3
[51] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. In NeurlPS, 2022. 3
[52] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In CVPR, 2023.
[53] Xuning Hu, Wenxuan Xu, Yushi Wei, Hao Zhang. Jin Huang, and Hai-Ning Liang. Optimizing moving target selection in vr by integrating proximity-based feedback types and modalities. In 2025 IEEE Conference Virtual Reality and 3D User Interfaces (VR), pages 52-62. IEEE, 2025.
3
[54] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM TOG, 2023. 3
[55] Silvia Zuffi, Angjoo Kanazawa, David Jacobs, and Michael J. Black. 3D menagerie: Modeling the 3D shape and pose of animals. In CVPR, 2017.
[56] Marc Badger, Yufu Wang, Adarsh Modh, Ammon Perkes, Nikos Kolotouros, Bernd G Pfrommer, Mare F Schmidt, and Kostas Daniilidis. 3d bird reconstruction: a dataset, model, and shape recovery from a single view. In ECCV, 2020. 3
[57] Nadine Rüeggg Shashank Tripathi, Konrad Schindler, Michael Black, and Silvia Zuffi. Bite: Beyond priors for improved three-d dog pose estimation. In CVPR, 2023. 3
[58] Yizhou Zhao, Tuanfeng Yang Wang, Bhiksha Raj, Min Xu, Jimei Yang, and Chun-Hao Paul Huang. Synergistic global-space camera and human reconstruction from videos. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1216-1226. 2024.
[59] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018. 3
[60] Nikos Kolotouros, Georgios Pavlakos, Michael Black, and Kostas Daniilidis. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In ICCV, 2019. 3
13

---

[61] Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4d: Reconstructing and tracking humans with transformers. arXiv preprint arXiv:2305.
2023. 3
[62] Yufu Wang and Kostas Daniilidis. Refit: Recurrent fitting network for 3d human recovery. In
ICCV, 2023.
[63] Sida Peng. Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
[64] Lingji Liu, Mare Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-view synthesis of human actors with pose control. ACM TOG, 2021.
[65] Chung-Yt Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In CVPR, 2022. 3
[66] Xu Chen, Yufeng Zheng, Michael Black, Otmar Hilliges, and Andreas Geiger. Snarf: Dif- ferentiable forward skinning for animating non-rigid neural implicit shapes. In ICCV, 2021.
[67] Xu Chen, Tianjian Jiang, Jie Song, Max Rietmann, Andreas Geiger, Michael J Black, and Otmar Hilliges. Fast-smarf: A fast deformer for articulated neural fields. IEEE TPAMI, 2023, 3
[68] Gengshan Yang, Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. Banmo: Building animatable 3d neural models from many casual videos. CVPR, 2022. 3,
[69] Gengshan Yang. Chaoyang Wang, N Dinesh Reddy, and Deva Ramanan. Reconstructing animatable categories from videos. In CVPR, 2023. 3
[70] Yikai Wang, Xinzhou Wang, Zilong Chen, Zhengyi Wang, Fuchun Sun, and Jun Zhu, Vidu4d: Single generated video to high-fidelity 4d reconstruction with dynamic gaussian surfels. In NeurIPS, 2024. 3
[71] Yizhou Zhao, Chunjiang Liu, Haoyu Chen, Bhiksha Raj. Min Xu, Tadas Baltrusaitis, Mitch Run- dle, Taiang Wu, and Kamran Ghasedi. Total-editing: Head avatar with editable appearance, motion, and lighting. arXiv preprint arXiv:2505. 20582, 2025. 3
[72] Johannes Lutz Schönberger and Jan-Michael Frahm. Stracture-from-mottie revisited. In CVPR, 2016.
[73] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. ORB-SLAM: a versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 2015. 3
[74] Zachary Teed and Jia Deng. DROID-SLAM: Deep visual SLAM for monocular, stereo, and
RGB-D cameras. In NeurlPS, 2021. 3
[75] Johannes Kopf, Xuejian Rong, and Jia-Bin Huang. Robust consistent video depth estimation. In CVPR, 2021.
[76] Zhoutong Zhang, Forrester Cole, Zhengqi Li, Michael Rubinstein, Noah Snavely, and William T. Freeman. Structure and motion from casual videos. In ECCV, 2022. 3
[77] Ladislav Kavan, Steven Collins, Jifí Žára, and Carol O'Sullivan. Skinning with dual quaternions.
SI3D, 2007.000.44 4
[78] Laurent Dinh, Jascha Sohl-Dickster and Samy Bengio. Density estimation using real nvp.
arXiv preprint arXiv:1605.1 1605.08800 2016. 4
[79] Qianqian Wang, Yen-Yu Yu Chang, Ruojin Cai, Zhengqi Li, Bharath Hariharan, Aleksander Holynski, and Noah Snavely. Tracking everything everywhere all at once. In ICCV, 2023. 4
[80] Alex Kendall, Matthew Grimes, and Roberto Cipolla. Posenet: A convolutional network for real-time 6-dof camera relocalization. In ICCV, 2015. 5
[81] Luigi Piccinelli, Yung-Hsu Yang. Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In CVPR, 2024.
[82] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow, In ECCV, 2020. 5
14

---

[83] Nikhila Ravi, Valentin Gabeur, Tuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.000
2024, 5
[84] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable
3d gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR, 2024. 7
[85] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024. 7
[86] Ji Liu, Beichen Ma. Qiaolin Yu, Ruoming Jin, Jingho Zhou, Yang Zhou, Huaiyu Dai, Haixun Wang, Dejing Dou, and Patrick Valduriez. Efficient federated learning with heterogeneous data and adaptive dropout. ACM Transactions on Knowledge Discovery from Data, 2025. 10
[87] Yuzhen Zhou, Jiajun Li, Yusheng Su, Gowtham Ramesh, Zilin Zhu, Xiang Long, Chenyang Zhao, Jin Pan. Xiaodong Yu, Ze Wang, et al. April: Active partial rollouts in reinforcement learning to tame long-tail generation. arXiv preprint arXiv:2509. 18521, 2025, 10
[88] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2. 2023. 10
15

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In section 5, we discuss the limitations of the work.
16

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our paper illustrates complete deformation field for articulated Gaussian splitting object.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide comprehensive details in section 4 and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
17

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code will be open- after paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. //wips.cc/publll/// https//cpppbclcguueeeeeee http://ooooSSSmssseeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
18

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify all the training and testing details in section and in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our paper focuses on 3D reconstruction and view synthesis tasks, where evaluation primarily relies visual quality metries (e.g.. PSNR, SSIM, LPIPS) computed on deterministic outputs. Since the models produce consistent results given fixed inputs, statistical significance tests like error bars
commonly used in this domain.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We record the compute resources in section 4 and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics pettps://neurtie cc/ppblicc/thhicssuuludeleee
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss these impacts in section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
If the authors
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
20

---

Our work does not involve the release of models or data that could pose significant risks of misuse. It does involve language models, content generators, or scraped datasets.
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the creators or original owners of assets in our paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswithcee thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
All the assets are well documented.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
21

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : http///eerp.../ee/ se//oonferences//200 /LLM) for what should or should be described.
22

---

A Appendix supplemental material
A.1 Implementation Details
Data Preprocessing All sequences are first normalised to common training resolution of 512x512 pixels. Following the protocol of BANMo, each 960 x 720 RGB frame is centre-cropped and downsampled, while its paired 256 192 depth image is bilinearly up-scaled. To stabilise early optimisation, we apply a global scale of 0.2 to both (i) the raw depth values and (ii) the translation component of the ARKit camera extrinsics that initialise the background root pose G.,, After training converges, this scale is reversed so that predicted depth and geometry return to metric units. All quantitative evaluations are finally performed on renderings resampled 480 360 resolution.
Dataset Details Our experiments are conducted on newly captured dataset comprising se- quences recorded with stereo camera setup at 30fps, featuring diverse scenes with complex interac- tions between humans and animals. Each sequence is approximately 0.5-1 minutes long, containing between 400 and 900 frames. We perform stereo rectification and use the left-camera frames for model training, reserving the right-camera frames exclusively for validation.
Evaluation Metrics We adopt standard visual quality metries (LPPIS, PSNR, SSIM) and depth accuracy metrics (Acc@0.1 Im and RMS depth error). For visual metrics, we compute results on novel views synthesized from withheld validation trajectories. Depth accuracy metrics utilize stereo-derived depth maps ground truth.
Metric Formulas
We provide precise formulations for the metrics used in quantitative evaluation:
. PSNR: PSNR 10 log10
MAXi
where MSE N (1:
MSE
SSIM: SSIM y) +e: (20. (o)
following standard definitions.
LPIPS: Utilizes pre-trained neural network to measure perceptual similarity.
Am Defined the proportion of predicted depth values within 0.1 meters of the ground truth.
RMS depth error:
V IN DD)²,, measuring mean depth deviation.
Deformation Network Initialization Dynamic Gaussian Splatting is notoriously sensitive its starting configuration: poorly placed Gaussians or mis-estimated skeletal poses readily trap optimisa- tion in severe local minima, producing results that hardly better than na've DEFORMABLE-GS baseline. To avoid this collapse we adopt the two-stage scheme described the main paper: (i) neural-SDF pre-fit jointly refines camera intrinsics. skeletal articulation, and soft deformation; (ii) Gaussians are then sampled on the resulting neural SDF canonical surface and the warping network continued to be optimized while we switch the objective dynamic Gaussian splatting. This warm-start supplies accurate joint positions, correct scale, and well- dell-disttibuted primitives, allowing subsequent learning to focus on fine non-rigid motion rather than coarse alignment. Ablations in Table 7 confirm that removing this initialisation causes up to a 35% drop in PSNR and depth accuracy on articulated human/animal sequences.
Network Architecture For the deformation networks, we adopt multi-layer perceptrons (MLPs) with sinusoidal Fourier features for positional encoding, Specifically, our global and object-root transformations use MLPs with hidden layers, each containing 256 neurons, activated with ReLU functions. The neural soft deformation network, modeled with a flow-based architecture inspired by RealNVP, comprises coupling layers to ensure invertibility.
Training and Optimization We implemented our model using PyTorch and optimized networks using Adam with an initial learning rate of 10-2, exponentially decayed by a factor of 0.5 every 2,000 iterations. For each optimization stage (initialization and joint refinement), we set the maximum number of iterations to 6,000, with early stopping criteria based on validation-set performance.
23

---

Method
PSNR
SSIM1
LPIPS]
Depth Acet
Depth Err.
Ours (full)
21.31
0.747
0.263 0.425
0.901 0.742
0.127 0.251
w/o initialization
17.30
0.552
Table 7: Effect of initialization. Higher is better for PSNR/ SSIM Depth Ace; lower is better for Depth Err.
Computational Cost Our proposed method significantly reduces computational requirements com- pared to NeRF-hased methods. On an NVIDIA H20 GPU, our initialization stage takes approximately 30 minutes, and joint refinement typically completes within 1.5 hours for sequences with around 800 frames. Inference for novel view synthesis operates at interactive frame rates (20fps average).
Because TOTAL-RECON reports training times RTX A6000, we re-ran our training on the same A6000. Under identical data and optimisation settings, our full pipeline required -1.2 hours, whereas TOTAL-RECON took -12 hours to reach comparable visual quality, confirming 10x speed-up while maintaining (and improving) reconstruction fidelity.
A.2 Additional Visual Qualitative Comparison
Previous work Dynamic Gaussian Splatting encompasses variety of architectures and settings. However, the main paper already demonstrates that our method surpasses these baselines in stability and fidelity across long, articulated sequences. Here, therefore focus on the most competitive prior art, TOTAL-RECOM which similarly targets long-range high-quality reconstructions. Comprehensive side-by-side renderings and depth maps (7, 8. 10, 11) show that our approach produces sharper silhouettes, fewer temporal artifacts, and consistently lower photometric and geometric error. The gap widens on challenging multi-actor scenes, confirming that the hierarchical deformation and articulated priors in our pipeline are critical for robust 4D reconstruction.
B Limitations and Future Work
Handling Discontinuous Motions Although our model effectively captures continuous articulated motions, handling abrupt discontinuities remains challenging due to smooth deformation field assumption. Future directions may explore explicit discontinuity modeling, possibly integrating event-based vision sensors for improved robustness in highly dynamic scenarios.
Improved Initialization Exploring advanced initialization methods, potentially leveraging para- metric body models (such SMPL for humans animal-specific skeletal models), could further enhance reconstruction quality and reduce sensitivity to initialization.
C Broader Impacts
Our method has potential positive impacts in AR/VR applications, enhancing realism in interactive systems. However, we acknowledge potential misuse risks, such as generating misleading synthetic content. We advocate responsible use and transparency in synthetic data usage, encouraging further research in detection and mitigation of malicious synthetic media.
24

---

GT
HoliGS
Total-Recon
GT
HoliGS
Total-Recon
Human & Dog
Depth
Human & Dog
Depth
Figure 7: NVS comparisons with Total-Recon.
25

---

GT
HoliGS
Total-Recon
GT
HoliGS
Tolal-Recon
Human & Cat
Depth
Human & Cat
Dopth
Figure 8: NVS comparisons with Total-Recon.
26

---

GT
HoliGS
Total-Recon
GT
HoliGS
Total-Recon
Cat2 (V1)
Depth
Cadl (VI)
Depth
Figure NVS comparisons with Tetal-Recon.
27

---

GT
HoliGS
Total-Recon
GT
HoliGS
Total-Recon
Cat2 (V2)
Depth
Dog! MD
Dapth
Figure 10: NVS comparisons with Total-Recon.
28

---

GT
HoliGS Total-Recon
GT
HoliGS
Total-Recon
Human
Depth
Human2
Depth
Figure 11: NVS comparisons with Total-Recon.
29

---

Sequence
Method
LPIPS
PSNR
SSIM1
Acc Ac.
Edopth (m)
HoliGS
0.270
19.94
0.784
0.838
0.172
MoSca
0.269
19.99
0.783
0.831
0.184
DOG 1 (VI)
Shape-of-Motion
0,288
19.58
0.770
0,826
0.189
GS-Marble
0.449
16.19
0.615
0.631
0.325
HoliGS
0.329
20.50
0.708
0.878
0.197
MoSca
0.328
20.40
0.698
0.866
0,214
CAT 1 (V1)
Shape-of-Motion
0.342
20.01
0.686
0,859
0,226
GS-Marble
0.525
15.74
0.531
0.662
0.369
HoliGS
0,293
21.69
0.693
0,894
0.126
MoSca
0.292
21.63
0.695
0.891
0.136
CAT 1 (V2)
Shape-of-Motion
0.298
21.44
0.686
0.887
0.141
GS-Marble
0,492
16.86
0.561
0.681
0.319
HoliGS
0,211
22.80
0.759
0.966
0.052
MoSea
0.210
22.80
0.756
0.964
0.058
2VI)
Shape-of-Motion
0.225
22.49
0.734
0.952
0,074
GS-Marble
0,418
18.06
0.609
0.718
0.281
HoliGS
0.271
22.08
0.759
0.929
0.082
MoSea
0.269
22.04
0.755
0.923
0.088
2V2)
Shape-of-Motion
0.281
21.81
0.743
0.913
0,100
GS-Marble
0,466
17.13
0.579
0.694
0.303
HoliGS
0.253
20.52
0.745
0.954
0.065
MoSca
0,250
20.34
0.724
0.931
0.089
CAT 3
Shape-of-Motion
0.271
19.83
0.710
0.920
0.106
GS-Marble
0,451
17.20
0.601
0.751
0.252
Mean
HoliGS
0.271
21.26
0.741
0.910
0.116
MoSca
0.270
21.20
0.735
0.901
0.128
Shape-of-Motion
0.284
20.86
0.722
0.893
0.139
GS-Marble
0,467
16.86
0.583
0.690
0.308
Table 8: Short-clip (200-frame) evaluation. Across six sequences, HoliGS wins or ties in 22/30 primary comparisons. MoSca slightly favors LPIPS (perceptual similarity), whereas HoliGS is stronger on PSNR/SSIM and depth accuracy (Ace@0. and achieves lower depth RMS (Eoepch).
30


Reviews:
[{"review_id": "2aoxUPbTDz", "reviewer": "Reviewer_kcQT", "summary": "The paper proposes a pipeline for modeling long-range dynamic 3D scenes, specifically targetted towards EVS. The method consists of multiple stages of networks: 1- estimating camera (to background or object canonical space) transformations 2- rigid skeletal transforms 3- soft invertable deformation field for modeling residual movements 3- and a SDF-based gaussian splatting backbone to model the scene geometry. Finally, the background and dynamic object models are composited to make the complete scene which is then optimized through photometric, depth, normal, SDF, segmentation and flow consistency losses.", "questions": "Please refer to the previous section, most of my question are on the clarity of the method.", "limitations": "The limitations are discussed to some extent. I think one main limitation that is left out is not explicitly modeling or analyzing how the model performs in the presence of rigidly moving objects such as cars.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The method achieves faithful 3D reconstruction of long-horizon dynamic 3D scenes\n- The method is relatively fast to converge compared to previous dynamic 3D reconstruction methods.\n- The method works with a monocular video capture \n- The method allows for EVS for humans and pets as an application useful for AR/VR.\n\nWeaknesses:\n- My main concern is the clarity of the paper in explaining the method in enough detail needed for reproducibility and understanding the full method. In what follows I pose a series of questions that I could not find the answer to in the paper, and I think should be made super clear for the paper to be clear.\n- Does the method need any form of input camera pose? The \"Global movements\" section in 3.1 seems to convey that all camera poses are predicted through the \"intrinsics MLP\", however if that is the case: 1- how are the background Gaussians initialized if not through SFM point cloud? 2- why is there no metrics reported on the accuracy of the recovered global transformations (camera extrinsics)?\n\n- Line 147: is PoseNet the articulation network in figure 3 or is it only used for initializing the rigid transforms? It is not clear if the articulation network is something that you train or just the off-the-shelf PoseNet. \n\n- What is the input to each of the networks depicted in figure 3? most importantly I am curious to know the input and output of the camera MLP, I am not sure what the \"differentiable rendering\" in this figure is outputting exactly, as the Gaussians seem to be sampled in the final stage and not before that?\n\n- In Line 35 you mention that using tracking networks makes most dynamic reconstruction methods heavy, however the proposed method uses depth estimation network, pose estimation network,  flow estimation network, etc. How do these models compare in memory and time consumption to the tracking networks used in G-Marbles for example.\n\n- The OOM reported in results shows a limitation of previous work but does not allow any comparison in terms of reconstruction quality, would it be possible to show a comparison on a shorter sequence where OOM doesnt happen? what is the memory limit for these methods?\n\n- The provided supplemental video is helpful in seeing better RGB quality than Total Recon but in terms of depth map, there seems to be nearly no improvement. Further, the two videos seem to be slightly out of sync (Total recon is a bit slower in speed, showing mismatch in the comparison slider movements)", "was_revised": true, "final_justification": "The paper presents strong results in EVS and camera pose refinement. I raised my score to BA as some methodological details were clarified in rebuttal period and the camera optimization seems to have a significant improvement on initial cameras. I am still unsure about the clarity of the paper in its current state as most of the essential details in the method section are excluded.", "rebuttal": "We sincerely thank the reviewer for their thorough and insightful review. Please find our point-by-point responses below.\n\n## [W1] Clarity\n\nThank you for your detailed comments. We hereby address all points raised and will revise the manuscript accordingly to ensure a thorough explanation of our method.\n\n## [W2] Input Camera Pose\n\nWe initialize our model using camera-to-world poses from onboard motion sensors (e.g., ARKit), which are then jointly refined during optimization by predicting per-frame delta SE(3) twists. Background Gaussians are initialized from these initial poses. For the foreground, we first pre-train a neural Signed Distance Function (SDF) and then initialize Gaussian centers by sampling directly on its learned surface. Since we lack independent ground-truth extrinsics beyond the noisy ARKit data, we evaluate performance based on reconstruction quality, such as Novel View Synthesis and depth accuracy, rather than a standalone pose-error metric.\n\n## [W3] PoseNet\n\nWe use PoseNet in an off-the-shelf manner, while further training it during optimization.\n\n## [W4] A Detailed Breakdown of Figure 3\n\nOur two-stage pipeline uses two different types of renderers. The \"differentiable rendering\" in Stage 1 is **volumetric rendering** used to learn the geometry; it is completely separate from the **Gaussian rasterization** in Stage 2. Here is a breakdown of the inputs and outputs for each module.\n\n### Stage 1: Geometry & Deformation Initialization\n\nThis stage learns the object's shape (as a Neural SDF) and its deformation model. It does **not** use Gaussian Splatting.\n\n#### Pose Estimation Network\n\n* Input: A single video frame at time $t$.\n* Output: The object's initial root pose and skeleton parameters for that frame.\n\n#### Articulation & Soft Deformation Networks\n\n* Input: A 3D point $X^*$ in the canonical (rest) space, plus the root pose and skeleton from the pose network.\n* Output: The warped 3D point $X^t$ in the live camera space for that frame.\n\n#### **Camera Intrinsics MLP**\n\n* Input: The frame index $t$ and initial camera intrinsic values (e.g., focal length derived from image dimensions).\n* Output: Refined camera intrinsics for that specific frame (focal lengths $f_x, f_y$ and principal point $c_x, c_y$). This allows the model to learn and correct for minor inaccuracies in the initial camera parameters, leading to a more consistent 3D reconstruction.\n\n#### Neural SDF Network\n\n* Input: A 3D point $X^*$ in the canonical space.\n* Output: A signed distance value, which defines the object's base shape.\n\n#### Differentiable Rendering (Volumetric Renderer)\n\n* Input: Camera parameters (from pose estimation + camera MLP), the learned deformation networks, and the Neural SDF.\n* Output: A rendered RGB image, depth map, and mask. These are compared against the ground-truth video to train the SDF and deformation networks.\n\n#### Sample GS (Final step of Stage 1)\n\n* Input: The fully trained Neural SDF.\n* Output: A point cloud of 3D points sampled from the SDF's surface. These points become the initial centers for the Gaussians in the next stage.\n\n### Stage 2: Deformable Gaussian Splatting\n\nThis is the main optimization stage that uses the efficient differentiable Gaussian rasterizer.\n\n* Input: The initial Gaussians from the \"Sample GS\" step, the pre-trained deformation networks from Stage 1, and a set of static background Gaussians.\n* Output: The final, high-fidelity rendered image, depth map, and normals, which are used to optimize the properties of the Gaussians (color, scale, rotation, opacity).\n\n## [W5] Memory and Time Comparison\n\nUnlike other state-of-the-art 4DGS methods (e.g., MoSca, Shape-of-Motion) that rely on dense point-tracking, our approach avoids its prohibitive resource costs. A dense tracker can exhaust even an 80GB A100 GPU on sequences with over 300 frames, making minute-long video reconstruction infeasible. In contrast, HoliGS employs a pipeline of lightweight networks, reducing memory and per-frame overhead to less than 25% of a dense tracker’s. This efficient design enables scalable reconstruction on long videos, as detailed by the modest VRAM usage shown in the table below.\n\n| Stage | Component | Method | VRAM (Peak Usage) | Notes |\n| :--- | :--- | :--- | :---: | :--- |\n| Pre-processing | Depth Estimation | UniDepth | ~12 GB | Run once per video. |\n| | Optical Flow | RAFT | ~6 GB | Run once per video. |\n| | Segmentation | SAM (ViT-H) | ~16 GB | Run once per video. |\n| | Pose Estimation | Posenet | < 1 GB | Run once per video. |\n| **Main Training** | **Reconstruction** | **HoliGS** | **~10 GB** | **Scales with scene complexity, not video length.** |\n| | | | | |\n| **Baseline** | Dense Point-Tracking | e.g., CoTracker | > 80 GB | Scales with number of points and frames; infeasible for ~1000 frames. |\n\n## [W6] Shorter Sequence Comparison\n\nTo ensure fair comparisons with baselines that fail on long sequences due to memory constraints, we evaluated all methods on shorter, 200-frame segments with 100-frame strides for long sequences that originally lead to OOM. The experimental results reveal that HoliGS consistently outperforms or remains highly competitive with specialized short-clip methods across nearly all evaluated sequences and metrics. We analyze the results as follows.\n\n**Strength of HoliGS.** The results do not show a trade-off. Instead, they highlight the comprehensive strengths of our method. Specifically, across all six test sequences, HoliGS wins or ties on 22 of the 30 primary metric comparisons (~73%). The superior performance of HoliGS even on these shorter clips underscores the benefits of our core design. By building a **globally consistent canonical model** and **jointly optimizing camera poses**, our method offers greater robustness to varied motion and complex object interactions. This validates that HoliGS is not only more scalable but also generally more accurate, making it a more reliable and powerful solution for \"in-the-wild\" dynamic scene reconstruction.\n\n**Specific strengths of baselines.** The baseline method MoSca consistently achieves the best LPIPS score, suggesting it excels at producing perceptually plausible results. However, this advantage in perceptual quality does not translate to better geometric or colorimetric accuracy, where HoliGS is demonstrably stronger.\n\n\n| Sequence | Method | LPIPS↓ | PSNR↑ | SSIM↑ | Acc↑ | $ε_\\text{depth}$↓ |\n|:---|:---|:---:|:---:|:---:|:---:|:---:|\n| **DOG 1 (V1)** | **HoliGS (Ours)** | .270 | 19.94 | **.784** | **.838** | **.172** |\n| | MoSca | **.269** | **19.99** | .783 | .831 | .184 |\n| | Shape-of-Motion | .288 | 19.58 | .770 | .826 | .189 |\n| | GS-Marble | .449 | 16.19 | .615 | .631 | .325 |\n| **CAT 1 (V1)** | **HoliGS (Ours)** | .329 | **20.50** | **.708** | **.878** | **.197** |\n| | MoSca | **.328** | 20.40 | .698 | .866 | .214 |\n| | Shape-of-Motion | .342 | 20.01 | .686 | .859 | .226 |\n| | GS-Marble | .525 | 15.74 | .531 | .662 | .369 |\n| **CAT 1 (V2)** | **HoliGS (Ours)** | .293 | **21.69** | .693 | **.894** | **.126** |\n| | MoSca | **.292** | 21.63 | **.695** | .891 | .136 |\n| | Shape-of-Motion | .298 | 21.44 | .686 | .887 | .141 |\n| | GS-Marble | .492 | 16.86 | .561 | .681 | .319 |\n| **CAT 2 (V1)** | **HoliGS (Ours)** | .211 | **22.80** | **.759** | **.966** | **.052** |\n| | MoSca | **.210** | **22.80** | .756 | .964 | .058 |\n| | Shape-of-Motion | .225 | 22.49 | .734 | .952 | .074 |\n| | GS-Marble | .418 | 18.06 | .609 | .718 | .281 |\n| **CAT 2 (V2)** | **HoliGS (Ours)** | .271 | **22.08** | **.759** | **.929** | **.082** |\n| | MoSca | **.269** | 22.04 | .755 | .923 | .088 |\n| | Shape-of-Motion | .281 | 21.81 | .743 | .913 | .100 |\n| | GS-Marble | .466 | 17.13 | .579 | .694 | .303 |\n| **CAT 3** | **HoliGS (Ours)** | .253 | **20.52** | **.745** | **.954** | **.065** |\n| | MoSca | **.250** | 20.34 | .724 | .931 | .089 |\n| | Shape-of-Motion | .271 | 19.83 | .710 | .920 | .106 |\n| | GS-Marble | .451 | 17.20 | .601 | .751 | .252 |\n| **Mean** | **HoliGS (Ours)** | .271 | **21.26** | **.741** | **.910** | **.116** |\n| | MoSca | **.270** | 21.20 | .735 | .901 | .128 |\n| | Shape-of-Motion | .284 | 20.86 | .722 | .893 | .139 |\n| | GS-Marble | .467 | 16.86 | .583 | .690 | .308 |\n\n## [W7] Supplemental Video\n\n**Depth map quality.** While overall error metrics are similar, our method produces qualitatively superior depth maps with noticeably sharper and more coherent details. For instance, in our supplement (Sec. S3), the results show crisper definition on the animal’s legs (Fig. S5) and cleaner surfaces on the table legs (Fig. S6) compared to Total Recon.\n\n**Video synchronization.** Both HoliGS and Total-Recon refine their initial camera poses during training. With different architectural biases, HoliGS and Total-Recon converges to a slightly different optimal camera trajectory. This \"representation-dependent drift\" means the novel-view videos, which are derived from these diverged paths, are not perfectly spatially aligned, leading to the illusion of a timing mismatch. The videos are temporally synchronized by frame count, which can be confirmed by observing specific actions within each render.\n \n## [L1] Handling Rigid Objects\n\nWe did not explicitly analyze performance on rigidly moving objects such as cars, furniture, or other non-articulated foreground elements due to the scope of this work. While our primary focus was not on rigid objects, our hierarchical framework inherently handles them as a simplified case of articulated motion. Our model can represent a rigid object by using a single-bone skeleton and disabling non-rigid deformations, letting the global SE(3) transformation capture all movement. Preliminary tests on a sequence with a rigidly moving car confirm this works well, achieving over 30 PSNR on training views. We agree that a full evaluation incorporating more diverse forms of objects is a valuable next step, and we look forward to exploring this promising direction in future work."}, {"review_id": "JWaXyqJKzr", "reviewer": "Reviewer_K3QB", "summary": "The authors proposes a deformable dynamic Gaussian Splatting algorithm to tackle embodied view synthesis problem where inputs are long monocular videos with 6DoF cameras.\nHoliGS leverages invertible Gaussian Splatting deformation network to build scenes as a hierarchy of a background and multiple foreground articulated objects.\nThe experiments show state-of-the-art reconstruction quality over both previous evs algorithm total-recon and general monocular dynamic NeRF/Gaussian baselines.", "questions": "- Could you clearly state what's the difference between HoliGS and Total-Recon?\n- Have you seen downsides of replacing NeRFs with Gaussian Splatting?\n- Could you provide direct speed and memory benchmarks vs. Total-Recon and other important methods?", "limitations": "Not entirely in the scope of this work, but as a future direction I am really interested in seeing generative priors could help filling in the gap of the monocular video input.", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength: \n- HoliGS is consistently outperforming total-recon and other monocular dynamic NeRF/Gaussian baselines in terms of reconstruction quality metrics on scenes.\n- close to 10 baselines, and detailed component ablations demonstrate substantial experimental effort \n- Loss terms, hyper-parameters and pipeline diagrams are clearly specified; code release is promised.\n\nWeakness:\n- The pipeline largely mirrors Total-Recon but swaps the NeRF renderer for Gaussian Splatting; the paper does not convincingly articulate what new insights enable the improvements.\n- The authors claimed efficiency advantage multiple times, but comparison with Total-Recon/GS/NeRF under identical hardware is not provided. (Line 48, for example)\n- HoliGS heavily replies on pseudo-labels. Although tracker-free, the method still relies on external monocular depth, optical-flow, and SAM masks (Eq. 3)\n- Key mathematical details (e.g., Mahalanobis weighting, RealNVP architecture) are deferred entirely to the supplement, hindering self-contained understanding.", "was_revised": true, "final_justification": "The authors' rebuttal has addressed my concern so i am giving a positive score.", "rebuttal": "We sincerely thank the reviewer for their thorough and insightful review. Please find our point-by-point responses below.\n\n## [W1, Q1] Key Insights\n\nWhile with a piepline similar to Total-Recon, HoliGS does not simply replace the NeRF representation in Total-Recon with GS. A naive swap fails to converge because GS is notoriously sensitive to its starting configuration. Based on this observation, we introduce a pre-training stage that optimizes a Neural SDF proxy. This step, which is not present in Total-Recon's pipeline, serves two crucial purposes: it first optimizes our deformation networks to a robust state, and then produces a high-quality surface from which we sample points to provide an effective initialization for the Gaussians. As our ablation study confirms, removing this initialization causes a severe drop in performance, validating it as a necessary and novel component of our method.\n\n| Method | PSNR↑ | SSIM↑ | LPIPS↓ | Depth Acc↑ | Depth Err↓ |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| **Ours (full)** | **21.31** | **0.747** | **0.263** | **0.901** | **0.127** |\n| w/o initialization | 17.30 | 0.552 | 0.425 | 0.742 | 0.251 |\n\n## [W2] Efficiency Comparison\n\nWe benchmarked all methods on a single NVIDIA RTX A6000 GPU. As summarized in the table below, HoliGS offers a significant training efficiency advantage, especially when compared to high-fidelity NeRF-based methods. This improved efficiency makes the reconstruction of long, dynamic scenes practical, avoiding the prohibitive training times of previous state-of-the-art approaches.\n\n| Sequence | HoliGS (Ours) | Total-Recon | Dynamic GS | 4DGS |\n|:---|:---:|:---:|:---:|:---:|\n| DOG 1 (V1) | `01:15:32` | `12:05:10` | `00:33:15` | `00:38:45` |\n| DOG 1 (V2) | `01:10:41` | `11:45:20` | `00:31:50` | `00:36:10` |\n| CAT 1 (V1) | `01:18:25` | `12:10:30` | `00:34:05` | `00:39:20` |\n| CAT 1 (V2) | `01:16:50` | `12:08:15` | `00:33:40` | `00:38:55` |\n| CAT 2 (V1) | `01:35:10` | `13:30:55` | `00:38:20` | `00:44:15` |\n| CAT 2 (V2) | `01:40:22` | `13:55:40` | `00:40:10` | `00:46:00` |\n| CAT 3 | `01:28:18` | `13:02:11` | `00:36:50` | `00:42:30` |\n| HUMAN 1 | `01:12:30` | `11:50:00`  | `00:32:10` | `00:37:05` |\n| HUMAN 2 | `01:05:55` | `11:30:15`  | `00:30:45` | `00:35:20` |\n| HUMAN-DOG | `02:02:14` | `14:10:25`  | `00:41:30` | `00:48:10` |\n| HUMAN-CAT | `01:55:48` | `13:45:30`  | `00:39:50` | `00:45:40` |\n\n## [W3] Reliance on Off-the-shelf Models\n\nWe would like to clarify that our methodology is common practice. The use of external priors from networks for depth, flow, and segmentation is a cornerstone of today's leading monocular 4D reconstruction pipelines, including MoSca (CVPR 2025) and Shape-of-Motion (ECCV 2025). We chose this established, tracker-free design for its efficiency, which allows HoliGS to scale to minute-long videos where dense-tracking methods like G-Marbles (Siggraph Asia 2024) would fail due to memory constraints. Although fully self-supervised methods are a valuable research goal, this pragmatic design is what currently makes high-quality, long-duration dynamic reconstruction a tractable problem. While a purely self-supervised approach is an important future goal, this pragmatic design is what enables HoliGS to achieve the high-quality, minute-long reconstructions demonstrated in our work.\n\n\n## [W4] Key Mathematical Details\n\nWe agree that adding more mathematical details will improve the paper's clarity, and we will incorporate them into the main paper in the revised manuscript.\n\n## [Q2] Downsides of Replacing NeRF with GS\n\nWe acknowledge there are several downsides of GS compared to NeRFs. GS is less robust, being highly sensitive to initialization (requiring our SDF bootstrap to prevent a 35% PSNR drop) and demanding more complex optimization heuristics. As an explicit representation, it also faces memory scalability issues when the scene is complex and can struggle with the topology of fine structures like fur. However, we found these challenges to be a worthwhile trade-off for the paramount benefits of 10$\\times$ faster training and real-time rendering capabilities.\n\n## [L1] Generative Priors\n\nIntegrating generative priors represents an promising future direction for this field. Such models could be used to create truly holistic scene reconstructions by completing geometry for unobserved regions, like the back of a sofa or the underside of a table, thus enabling true free-viewpoint rendering. Furthermore, generative priors could operate in the temporal domain to synthesize plausible dynamics during moments of heavy motion blur or occlusion, leading to smoother, more realistic animations. One could even leverage cross-modal, text-conditioned models to inject high-level semantic details, such as using the prompt \"wooden table\" to realistically generate its unseen legs and texture. We are excited to explore it in the future."}, {"review_id": "PpCqknCqaw", "reviewer": "Reviewer_wfcZ", "summary": "This paper presents HoliGS, a novel framework for embodied view synthesis (EVS) from long monocular RGB videos capturing dynamic scenes with humans, animals, and complex interactions. It introduces a hierarchical deformable Gaussian representation combining (1) global rigid transformations, (2) skeleton-driven articulated motion, and (3) a fine-grained, invertible non-rigid deformation field. Empirically, the method achieves state-of-the-art performance in both reconstruction quality and speed on multiple challenging EVS benchmarks, including scenarios with complex viewpoint changes, long durations, and multiple deformable agents.", "questions": "1. What will the results look like if 1K images (high resolution) are input?\n2. Please cite SMPL in the section limitations.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The hierarchical design of the paper is quite interesting. It effectively attempts to disentangle major components of dynamic scenes, including camera motion, rigid motion, and non-rigid deformation. As the authors note in the limitations, a similar disentanglement can be achieved using SMPL-based avatars through parametric modeling. However, the proposed method is notably more generalizable, as it does not rely on specific templates and can be applied to arbitrary dynamic scenes.\n2. The paper is well written, and the experiments are comprehensive. I appreciate the functional comparisons with strong baselines, as well as the inclusion of trajectory visualizations, rendering results, and thorough ablation studies. These collectively provide strong empirical support for the proposed method.\n\nWeaknesses:\n1. The training resolution of the input images is relatively low (480×360). It seems 3DGS frameworks are capable of handling higher-resolution inputs (e.g., 1K). The limited resolution introduces certain issues—for instance, the rendering results of all methods appear somewhat blurry, which makes it harder to assess the visual differences between competing methods.\n2. From the supplementary video, the hierarchical deformation pipeline appears to introduce floater-like artifacts around dynamic objects. Based on my understanding, this may be due to the rigid deformation component dominating the overall motion modeling, thereby making it harder for the non-rigid module to accurately fit the original appearance. While the input resolution could also be a contributing factor, other methods only exhibit general blurriness without such specific artifacts. This suggests that the issue may be inherent in the proposed hierarchical pipeline design. The authors are encouraged to discuss this issue in the rebuttal.", "was_revised": true, "final_justification": "The author has presented the results on 1K images as well as an analysis of the artifacts. This addresses my concerns, and I therefore maintain my original rating.", "rebuttal": "We sincerely thank the reviewer for their thorough and insightful review. Please find our point-by-point responses below.\n\n## [W1, Q1] Handling Higer-resolution Inputs\n\nWe initially used a lower resolution to ensure fair comparisons with baselines like Total-Recon. To address your point, we re-trained our HoliGS on the original high-resolution (1K) inputs, which improves performance across all metrics. This confirms that our model successfully leverages the high-frequency details available in the 1K inputs to produce more accurate reconstructions. Further, we believe the more modest gains in structural metrics (SSIM and depth) are primarily limited by the significant motion blur present in the casually captured source videos, which serves as an imperfect ground truth.\n\n| Resolution | PSNR↑ | SSIM↑ | LPIPS↓ | Acc↑ | εdepth↓ |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n| Low (480p) | 21.31 | .747 | .263 | .901 | .127 |\n| High (1K) | **22.48** | **.751** | **.248** | **.905** | **.124** |\n\n\n## [W2] Floater-like Artifacts\n\nWe agree with the reviewer that the floater artifacts are an inherent limitation of our current design. Our analysis points to three root causes, each suggesting a clear direction for future work.\n\n**Skinning weight discontinuities.** Artifacts appear at articulation boundaries (e.g., tails, hands) where conflicting motion signals from different bones pull Gaussians into physically implausible positions. This could be addressed with adaptive, time-varying skinning weights that adjust based on local motion.\n\n**Discrete primitive limitations.** Unlike continuous fields, discrete Gaussians cannot smoothly \"stretch\" across high-deformation regions. During rapid motion, they can land in mathematically averaged, non-physical intermediate states. A promising solution is targeted densification, spawning new Gaussians in high-deformation zones to better capture the surface.\n\n**Hierarchical motion dominance.** Large global motions can sometimes dominate the local deformation field, especially at extremities, as the model struggles to fully counteract the root transformation. Introducing physics-informed constraints could penalize non-physical states and help disentangle global and local motion.\n\n## [Q2] Citing SMPL\n\nThank you for the suggestion. We will ensure a proper citation for SMPL is included in the limitations section of the revised manuscript."}, {"review_id": "jvlNSh9KFv", "reviewer": "Reviewer_cTCM", "summary": "This paper proposes HoliGS, a deformable Gaussian splatting framework for efficient embodied view synthesis (EVS) from long monocular RGB videos. It models scenes using a hierarchical decomposition into static backgrounds and dynamic foregrounds, incorporating rigid, articulated, and non-rigid deformations via an invertible neural model. By attaching Gaussians to a canonical shape, HoliGS enables robust rendering under complex 6-DOF camera trajectories. The method outperforms prior dynamic NeRF and 4D Gaussian approaches in both rendering quality and speed, achieving real-time performance.", "questions": "1. Can the authors provide a more detailed analysis of hyperparameter sensitivity?\n2. How well does HoliGS generalize without extensive supervisions like depth, flow, semantics etc. which seems not be used in some baseline comparison methods.", "limitations": "yes", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The paper introduces a novel invertible neural flow to model non-rigid deformations, which allows for stable and consistent reconstructions over extended time periods\n2. Unlike prior dynamic NeRF methods that struggle with long sequences, HoliGS scales to minute-long videos with reduced training and rendering time.\n\nWeakness:\nMajor concern 1. The framework’s robustness and performance appear to rely on a large number of carefully chosen hyperparameters (in training objective, number of elements in the scenes etc.), which may limit reproducibility and raise my concerns about generalization. For example, in training loss 5 the author seem need to make a balance between different losses. And for each specific scene, the model seems to need to define the number of dynamic elements in the scene.\nMajor concern 2. HoliGS leverages a hybrid initialization involving rigid-body transformation, skeletons, background component transformation. And also use off-the-shelf model to provide depth supervision, optical flow, segmentation supervision etc. While effective, this complexity may hinder adoption and raise concerns about modularity and end-to-end usability. \n\nFor example, in table 4 we could observe that the without depth supervision the result degrades a lot. This makes me questionable about the effectiveness of the invertible deformation model, which should help mitigate the effect of depth supervision.  The model needs to reply on generic pose estimation network etc. to give a decent initialization. I wonder how well the model generalize with respect to these predefined prior of the scenes.\nMinor concern 3. it's also unclear to me how is the static component being initialized, and seperated from dynamic objects.", "was_revised": true, "final_justification": "The sensitivity of hyperparameters across scenes raises concerns about the method’s generalizability and ease of transfer to new scenarios. Additionally, several key technical components are not clearly explained. As such, I remain unsure on recommending this paper for publication in its current form and remain my previous score.", "rebuttal": "We sincerely thank the reviewer for their thorough and insightful review. Please find our point-by-point responses below.\n\n## [W1, Q1] Hyperparameter Sensitivity\n\nWe present sensitivity analysis of the key hyperparameters in our two-stage training process. Our framework consists of a **Component Pre-training** stage to establish a robust geometric and motion foundation, followed by a **Joint Fine-tuning** stage to refine the appearance and inter-object interactions. The analysis is therefore split into two parts. The results are averaged across all sequences in our dataset. Our chosen default parameters are highlighted and represent the best-performing configuration.\n\n### Stage 1: Component Pre-training Sensitivity\n\nIn this initial stage, the goal is to learn a stable canonical shape (SDF) and a coherent deformation field. The most critical hyperparameters are the weights for the depth, color, and optical flow losses.\n\n| $\\lambda_\\text{depth}$ | $\\lambda_\\text{color}$ | $\\lambda_\\text{flow}$ | LPIPS↓ | PSNR↑ | SSIM↑ | Acc↑ | $\\epsilon_\\text{depth}$↓ |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 1.0 | 0.1 | 1.0 | .285 | 20.95 | .735 | .855 | .158 |\n| **5.0** | **0.1** | **1.0** | **.263** | **21.31** | **.747** | **.901** | **.127** |\n| 10.0| 0.1 | 1.0 | .272 | 21.10 | .740 | .891 | .135 |\n| 5.0 | 0.01 | 1.0 | .279 | 21.05 | .738 | .898 | .130 |\n| 5.0 | 1.0 | 1.0 | .288 | 20.85 | .725 | .885 | .141 |\n| 5.0 | 0.1 | 0.1 | .291 | 20.70 | .720 | .870 | .155 |\n| 5.0 | 0.1 | 5.0 | .282 | 20.90 | .730 | .875 | .148 |\n\nThe results show that a careful balance is required during pre-training.\n* **Depth loss ($\\lambda_\\text{depth}$).** This is the most critical parameter in Stage 1. A lower weight (`1.0`) fails to establish a strong geometric foundation, resulting in significantly worse depth accuracy. Conversely, a very high weight (`10.0`) can cause the model to overfit to potentially noisy pseudo-ground-truth depth, creating geometrically rigid but less visually coherent surfaces, which slightly hurts all metrics. Our default of `5.0` provides the best balance.\n* **Color loss ($\\lambda_\\text{color}$).** We found that a relatively small color weight (`0.1`) is crucial in this initial stage. A higher weight (`1.0`) can cause the model to \"bake in\" appearance and lighting details prematurely, which conflicts with the primary goal of learning a clean, underlying geometry and leads to degraded performance.\n* **Flow loss ($\\lambda_\\text{flow}$).** The flow loss acts as a regularizer for motion. A low weight (`0.1`) results in less temporally consistent reconstructions. A very high weight (`5.0`) can cause overfitting to the estimated optical flow, introducing jitter and artifacts.\n\n### Stage 2: Joint Fine-tuning Sensitivity\n\nUsing the best model from Stage 1, this stage refines the entire scene with a focus on high-fidelity appearance and correct object interactions. The key parameters are the weights for the final photorealistic, normal, and depth losses.\n\n| $\\lambda_\\text{photo}$ | $\\lambda_\\text{normal}$ | $\\lambda_\\text{depth}$ | LPIPS↓ | PSNR↑ | SSIM↑ | Acc↑ | $\\epsilon_\\text{depth}$↓ |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 0.1 | 1.0 | 5.0 | .281 | 20.90 | .730 | .898 | .131 |\n| **1.0** | **1.0** | **5.0** | **.263** | **21.31** | **.747** | **.901** | **.127** |\n| 5.0 | 1.0 | 5.0 | .270 | 21.15 | .741 | .895 | .134 |\n| 1.0 | 0.1 | 5.0 | .275 | 21.18 | .740 | .880 | .145 |\n| 1.0 | 5.0 | 5.0 | .269 | 21.25 | .745 | .885 | .140 |\n| 1.0 | 1.0 | 1.0 | .278 | 21.00 | .735 | .865 | .152 |\n| 1.0 | 1.0 | 10.0| .268 | 21.22 | .742 | .895 | .130 |\n\nIn this refinement stage, the balance shifts towards achieving high visual quality while preserving the strong geometry from Stage 1.\n* **Photometric and normal losses ($\\lambda_\\text{photo}$, $\\lambda_\\text{normal}$).** We observe that an equal weighting of `1.0` for the photometric and normal losses provides the best trade-off. Over-emphasizing the photo loss (`5.0`) can introduce minor visual artifacts as the model sacrifices geometric smoothness for pixel-perfect colors. Over-emphasizing the normal loss (`5.0`) can create overly sharp, non-photorealistic surfaces that hurt the LPIPS score.\n* **Depth loss ($\\lambda_\\text{depth}$).** While appearance is key in this stage, maintaining a strong depth weight (`5.0`) is still vital. A lower weight (`1.0`) allows the geometry to drift from the metric ground truth, significantly impacting depth accuracy. A higher weight (`10.0`) offers diminishing returns and can slightly compromise the visual metrics by making the surfaces overly rigid.\n\nIn conclusion, our chosen hyperparameters represent a carefully tuned balance between geometric and appearance-based losses, tailored to the specific goals of each training stage. While the model shows some sensitivity, the performance remains robust within a reasonable range of our selected default values.\n\n## [W2, Q2] Generalization without Extensive Supervisions\n\nWe conducted an ablation study to validate the effectiveness of supervision components, with results below. Note that semantic masks are indispensable, as the pipeline fails without them. The results demonstrate that all supervision components are crucial for achieving our state-of-the-art performance, as ablating any of them leads to a significant degradation in quality. This analysis clarifies that our invertible deformation model and the geometric supervisions (like depth) play distinct, complementary roles: the depth loss establishes the scene's foundational 3D structure, while the invertible model ensures that the learned motion is stable and consistent over time. The large drop from ablating depth supervision does not mean our invertible model is ineffective. Rather, it shows that even a perfect motion model can't fix a scene with fundamentally incorrect geometry. You need both: **correct geometry** (from depth) and **consistent motion** (from our invertible deformation model). While our method relies on these priors, this is a standard and necessary approach (also adopted in MoSca, Shape-of-Motion, etc.) to constrain the fundamentally ill-posed problem of monocular dynamic reconstruction, and our ablation shows how each component contributes to the final high-quality result.\n\n| Ablation | LPIPS↓ | PSNR↑ | SSIM↑ | Acc↑ | $\\epsilon_\\text{depth}$↓ |\n|:---|:---:|:---:|:---:|:---:|:---:|\n| **Full Model (Ours)** | **.263** | **21.31** | **.747** | **.901** | **.127** |\n| w/o Depth Loss ($\\mathcal{L}_\\text{depth}$) | .385 | 18.52 | .651 | .847 | .248 |\n| w/o Normal Loss ($\\mathcal{L}_\\text{normal}$) | .288 | 20.47 | .720 | .832 | .163 |\n| w/o Flow Loss ($\\mathcal{L}_\\text{flow}$) | .310 | 19.81 | .702 | .861 | .184 |\n| w/o Cycle Loss ($\\mathcal{L}_\\text{cycle}$) | .325 | 19.55 | .693 | .852 | .196 |\n\n## [M3] Static Component Initialization\n\nOur method first decomposes the scene into static and dynamic components using foreground masks from SAM. The static background is then initialized by creating a 3D point cloud from initial camera poses (e.g., from ARKit) and estimated depth. The static components are initialized from this point cloud."}]

Author rebuttal:

Meta-review: The paper makes significant technical and practical contributions to embodied view synthesis. Three reviewers are positive. The comprehensive rebuttals successfully addressed major concerns about generalizability, efficiency claims, and method clarity. The extensive per-scene ablations convincingly demonstrate robustness rather than sensitivity. While clarity could be improved, the authors' commitment to open-sourcing code and revising the manuscript addresses reproducibility concerns. The work represents a substantial advance in making long-duration dynamic scene reconstruction practical, which is valuable for the community
Decision: Accept (poster)