Orientation-anchored Hyper-Gaussian for 4D Reconstruction from Casual Videos
Authors: Junyi Wu
Jiachen Tao
Haoxuan Wang
Gaowen Liu
Ramana Rao Kompella
Yan Yan
Venue: NeurIPS 2025 poster ()
Keywords: 4D Reconstruction
Primary area: applications
TL;DR: We present Orientation-anchored Gaussian Splatting (OriGS), a novel framework for high-quality 4D reconstruction from casually captured monocular videos.

Abstract: We present Orientation-anchored Gaussian Splatting (OriGS), a novel framework for high-quality 4D reconstruction from casually captured monocular videos.
While recent advances extend 3D Gaussian Splatting to dynamic scenes via various motion anchors, such as graph nodes or spline control points, they often rely on low-rank assumptions and fall short in modeling complex, region-specific deformations inherent to unconstrained dynamics.
OriGS addresses this by introducing a hyperdimensional representation grounded in scene orientation.
We first estimate a Global Orientation Field that propagates principal forward directions across space and time, serving as stable structural guidance for dynamic modeling.
Built upon this, we propose Orientation-aware Hyper-Gaussian, a unified formulation that embeds time, space, geometry, and orientation into a coherent probabilistic state.
This enables inferring region-specific deformation through principled conditioned slicing, adaptively capturing diverse local dynamics in alignment with global motion intent.
Experiments demonstrate the superior reconstruction fidelity of OriGS over mainstream methods in challenging real-world dynamic scenes.

Orientation-anchore Hyper-Gausiian for 4D Reconstruction from Casual Videos
Junyi Wu
Jiachen Tao
Haoxuan Wang'
Gaowen Liu?
Ramana Rao Kompella²
Yan Yan
University of Illinois Chicago Cisco Research
Abstract
We present Orientation-anchore Gaussian Splatting (OriGS), novel frame- work for high-quality 4D reconstruction from casually captured monocular videos. While recent advances extend 3D Gaussian Splatting to dynamic scenes via various motion anchors, such as graph nodes or spline control points, they often rely on low-rank assumptions and fall short in modeling complex, region-specifie deforma- tions inherent to unconstrained dynamics. OriGS addresses this by introducing a hyperdimensional representation grounded in scene orientation. We first estimate a Global Orientation Field that propagates principal forward directions across space and time, serving stable structural guidance for dynamic modeling. Built upon this, we propose Orientation-aware Hapss-Ga a unified formulation that embeds time, space, geometry, and orientation into coherent probabilistic state. This enables inferring region-specific deformation through principled con- ditioned slicing, adaptively capturing diverse local dynamics in alignment with global motion intent. Experiments demonstrate the superior reconstruction fidelity of OriGS over mainstream methods in challenging real-world dynamic scenes.
Introduction
Videos offer a window into the continuous flow of forms, light, and transformations that weave experience of space and time. Among various sources, casually captured monocular videos stand as the most ubiquitous and accessible. These videos are typically recorded by smartphones, handheld cameras, consumer drones, spanning broad spectrum of everyday scenes with diverse object compositions and unconstrained motion patterns. Recovering the underlying dynamic reality from such inputs remains foundational pursuit computer graphics and 3D vision [33], boosting applications in autonomous driving [72, 21], robotics [30, 53], and virtual/augmentee reality [81, 10].
At its core, 4D reconstruction aims recover the time-varying geometry, appearance, and motion of real-world environments from videos [82], bridging visual observations with underlying scene dynamics. 3D Gaussian Splatting (3DGS) [37] has recently emerged powerful solution for static scene reconstruction, offering a compact and expressive point-basee representation that enables high-fidelity modeling and real-time rendering. Building its success, substantial efforts have been devoted to extending 3DGS to dynamic scenes [49, 76, 71, 64]. These methods primarily capture temporal evolution by applying per-Caussian deformations, such translation and rotation, across time. Recent works are further pursuing robust reconstruction from casually captured videos by modeling deformation through various motion anchors. Representative approaches include learning motion bases and weighting coefficients [67], constructing discrete 3D node graphs [48, 39, 44], and employing continuous cubic Hermite splines with sparse control points [77, 54].
Yet, reconstructing scene dynamics from casually captured monocular videos remains challenging due to the complexity of local motion. Regions within the scene often exhibit diverse motion
"Corresoondigg Author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

patterns, influenced by object articulation, interaction context, or progression through different action stages [61]. Although recent approaches incorporate explicit, structured deformation fields, they largely reduce complex dynamics to low-rank motion anchors, assuming that neighboring regions exhibit similar motion. While effective for smooth or rigid transformations, such deformation anchors are inherently fragile under unconstrained dynamics in real-world scenarios, where motion patterns may vary significantly across regions. Consequently, these formulations often struggle to capture complex local dynamics, resulting in spatial drift, structural fragmentation, or temporal inconsistency.
To address these challenges, we are motivated by the perspective that the diversity of local motion often reflects more than just positional or geometric variation; it suggests an underlying regularity in how different regions evolve over time. Intuitively, local deformations are not solely determined by spatial position or object shape, but also by how each part of a scene is expected to move within a broader dynamic context. Fortunately, while casual monocular videos lack explicit multi-view supervision, they do preserve object-centric motion cues that implicitly reflect these region-specffic motion tendencies. Among them, we focus our exploration on a natural and grounded signal: orientation. In the physical world, while absolute position is sensitive to noise and scale ambiguity, orientation evolves under the smoother governance of local angular momentum and inertia [16, 27]. This makes it a more stable proxy for how motion is expected to unfold in complex dynamic scenes, indicating the forward tendencies across regions. In this work, we embrace orientation as a dynamic anchor for 4D reconstruction. Our key insight is to treat orientation as structural signal that organizes how different parts of the scene deform and interact, thereby capturing coherent evolution.
Building on this perspective, we introduce a novel framework for high-quality 4D reconstruction from casual monocular videos. Our OriGS comprises two synergistic components for dynamic modeling. We first construct a Global Orientation Field that captures long-range scene evolution by estimating and propagating principal forward directions over time via localized structure alignment. This orientation field provides stable oriented anchors that extend across space and time, which help organize diverse motion patterns in Hyper-
iiientation-anchored Gaussian Splatting (OriGS),
different regions. @ On top of this structured foundation, we propose Orientation-awan Gaussian, a hyperdimensional representation to model complex local dynamics throughout the scene. We associate each Gaussian primitive with unified probabilistic state defined over structured manifold of time, space, geometry, and orientation. By conditioning this representation on the scene's evolving orientation field, we infer region-specific deformations through principled conditioned slicing strategy. This allows OriGS to adaptively model how different parts of the scene deform over time and how their motions evolve in alignment with the underlying dynamic intent. Together, these two components enable OriGS to robustly reconstruct dynamic scenes from casual videos, bridging global coherence and local motion diversity in unified and principled manner.
We validate the effectiveness of OriGS on a range of casual monocular videos, including DAVIS [59], OpenAl SORA [6]. YouTube-VOL VOS [74], and the DyCheck benchmark [19]. OriGS consistently recovers sharper geometry and more coherent motion compared to recent state-of of-te-of-the-art methods, demonstrating its superior reconstruction fidelity in real-world dynamic scenes.
2 Related Works
Dynamic 3D Representations. Dynamic scene modeling has been widely studied by extending
static 3D representations with deformation mechanisms [20, 9]. Neural Radiance Fields (NeRF) [51] represent scenes as volumetric functions, and are adapted to dynamic settings by learning deformation fields that warp canonical points over time [55, 60, 56, 17, 34, 70J. Later works introduce temporal embeddings [57, 40], or hybrid representations for faster rendering [2, 18, 8, 62]. However, their implicit volumetric nature limits real-time inference and structure-an aware modeling. Recently, 3D Gaussian Splatting (3DGS) [37] has emerged as a powerful alternative for scene representation
[4, 82]. Dynamic extensions incorporate deformation fields [49, 71, 76, 15, 3, 31], apply trajectory interpolation [41, 38, 77], or directly embed time into Gaussian primitives [14, 75]. Yet, lacking explicit guidance from underlying structures, these methods are generally incapable of faithfully recovering complex motion in real-world dynamic scenes.
Hyperdimensional Gaussians. Recent work has enhanced the expressiveness of Gaussian represen- tations by embedding auxiliary signals into higher-dimensional spaces. 4DGS [75, 14) incorporates time to model spatio-temporal volumes, while N-DG [11] generalizes to a latent space embedding po-
2

---

sition, view direction, and material cues. To capture rie-deependent effects, 6DGS [22, 24] integrates angular information and modulates opacity and color accordingly, and 7DGS [23] further extends this with temporal modeling. In this work, we explore hyperdimensional Gaussians as structured probabilistic states, modeling space, time, geometry, and orientation in a unified representation.
4D Reconstruction from Casual Monocular Videos. Recovering dynamic representations from casually captured monocular videos is highly ill-posed. Early efforts enhance NeRFs by specialized deformation fields [60, 55, 56], long-range feature aggregation [43], or point trajectory prediction [66].
While accounting for scene changes, they predominantly consider controlled settings with teleporting cameras or quasi-static scenes [19]. Recent advances shift toward Gaussian- based representations, leveraging 2D priors enable reconstruction in the wild. Methods like Shape-of-Motion [67], Gaussian Marbles [64], MoSca [39], and MoDGS [46] guide per-Gaussian deformation through various motion anchors. Others adopt compact motion models, such as spline interpolation [54] or hierarchical structures [44]. to trace temporal trajectories. Nonetheless, their reliance on low-rank motion priors limits the capacity to capture the complex, spatially varying dynamics present in real- world environments. In contrast, we propose to organize deformation within global orientation field and condition dynamic modeling on a unified hyperdimensional representation, enabling coherent reconstruction under diverse motion patterns.
3 Preliminaries
3D Gaussian Splatting. 3DGS [37] represents scenes set of explicit 3D Gaussian primitives, each specified by a spatial center Up E TR3S and an anisotropic covariance matrix E E R333,
G(x) = exp 1-((x (x xx
where x E R33 denotes a 3D spatial location. To ensure positive semi-definiteness, E is factorized as Σ RSS RJ in practice, where S is diagonal scaling matrix and is a rotation matrix aligning the Gaussian with the global coordinate system. Moreover, each 3D Gaussian associates an opacity value IR and color vector E RRS During rendering, contributions from all Gaussians overlapping a pixel are composited via alpha blending to provide the final color.
In this work, adopt 3DGS as differentiable and compact representation for dynamic scenes, which can be efficiently rendered into images with a rasterization pipeline.
3D Trajectories from Monocular Videos. Given a monocular video of TT frames, our first step toward 4D reconstruction is recovering lang-range 3D trajectories of scene points over time. Each trajectory is defined as a sequence {T'}/=>> where T' E R3 denotes the 3D position at frame t. However, estimating such trajectories from monocular inputs inherently ill ill-posed due to depth ambiguity from 3D>to-2D projection. To miti mitigate this, we combine metric depth estimation [28, 29, 58] with long-range pixel tracking [12, 36, 73]. We predict per-frame depth maps d' RR2 + R4 and extract 2D trajectory {u') [u'} where u' E R denotes the pixel location at frame t. Consistent with common practices [64, 73, 39], we then lift each 2D trajectory into 3D world space, aided by the depth priors:
W T K (u', d' (u')).
(2)
where (-(-) denotes the projection function from camera to image space defined by intrinsics K. and W is the camera pose at frame t. When camera parameters are unavailable, we estimate them via bundle adjustment [45, 39].
The resulting sequence {T' provides a 3D trajectory across the scene. We further extract principal forward directions from these trajectories construct oriented anchors, which provide structured and temporally coherent guidance for dynamic modeling in our framework.
Method
Gaussian Splatting (OriGS), novel framework for 4D reconstruction from casual videos. OriGS built upon two synergistic components: (i) a Global Orientation Field (Section 4.1) that captures long-range scene evolution via propagated orientation cues; and (ii) an Orientation-aware Hyper-Gaussin (Section 4.2) that models complex region- specific dynamics via orientation-conddiionne inference in unified hyperdimensional representation.
We introduce Orientation-ancee
3

---

Initial Principal Orientation Estimation
Temporal Orientation Propagation
A
Oriemed Anchor
30 rrelecde
Video
Neighboring 6x Anchor Set
Inilial
Temporal Window
Principal Direction
Orientation
Figure I: Illustration Global Orientation Field. Given trajectorics recovered from monocular video, we extract structured set of oriented anchors. (Left) Anchors' positions are initialized along tracked points across frames. (Middle) We estimate initial principal orientations by applying PCA over short temporal windows, reflecting the dominant forward direction per anchor. (Right) These initial orientations are then propagated over time via localized Procrustes alignment, producing temporally coherent orientation field across the scene.
4.1 Global Orientation Field
{T'} of scene points (Section 3), we convert these raw translation paths into structured set of oriented anchors, each carrying spatial position and an associated local orientation. These anchors serve as the basis for constructing our Global Orientation Field.
Initial Principal Orientation Estimation. Given the recovered 3D trajectory
To initialize orientation, we estimate the principal direction of each anchor by analyzing the dominant motion in the early portion of its trajectory. Concretely, we unfold each trajectory and extract the first W frames as temporal window, forming { Jt=1- We center the points using the temporal mean == w W r', and define &f = T' F. We then compute the covariance matrix the centered points and apply principal component analysis (PCA) extract the dominant direction:
PCA(C) MVi, V2, val. where C
The leading eigenvector vi reflects the dominant motion axis, which we assign the anchor's forward direction to construct its initial local orientation O E (0(3).
Temporal Orientation Propagation. To extend the initial principal orientation across time, we estimate time-varying orientation sequence {O'}2 for each anchor, capturing its evolving motion direction throughout the scene. This can be formulated as a Procrustes alignment problem [26, 25]. At
each time step we identify local neighborhood {T around each anchor based on Euclidean distance and align it to its reference configuration by optimizing:
K
min TT
Σ
[| - 7²) -T' +1)
(4)
where 7' and 5 spatial centroids of the anchor set, and T represents the rotation transformation.
This alignment is solved by applying singular value decomposition (SVD) to the covariance matrix of the localized anchor set. The optimal rotation transformation is then given by T VUT, where V and U are obtained from SVD. As a correction step. we adjust the last column based on the sign of its determinant. Finally, compute the anchor's orientation Of by composing the optimal transformation with the initial principal orientation:
-T= O
Repeating this process yields temporally coherent orientations {0')T These propagated orienta- tions constitute our Global Orientation Field, which encodes motion tendencies across the scene and provides stable anchors for modeling complex dynamics. We illustrate this process in Figure 1, where AQ (0) T!' (O), T;'- denotes the relative anchor transformation in SE(3) across time.
4.2
Orientation-awanl HapssaGaussian
Dynamic scenes often exhibit complex, region-specific motion that cannot be fully explained by position or geometry alone. While casual monocular videos lack multi-view supervision, they retain

---

Posttion P
(t, Condrioned Siicing
Pood
S
Blundoo
Inference & Modulation
Rotation I'
Opecity
(Ap,Ag,t,0) (M Eig)
(Ap.Ag r'' r² N (faood Mraud)
Global Orientation Field
Oriented Anchor
Hyper-Gaussian
3D Gaussian Primitive
3DGS Pipeline
Figure 2: Illustration of Orientation-awart Hyper-Gaussian. We model complex dynamics using high- dimensional Gaussians defined over a structured manifold spanning time, space, geometry, and orientation. Given target time t' and its local orientation O" (interpolated from the Global Orientation Field via SO(3) blending), we perform conditioned slicing on the hyper- Gaussaan to infer the expected dynamic states. The predicted fAodd modulates 3D Gaussian primitives, which is then compatible with the 3DGS pipeline.
coherent orientation cues that implicitly reveal how different regions are expected to move. We lever- age this structural signal and introduce Orientation-awan Hyper-Gaussin,, hyperdimensional representation that models local dynamics over a unified manifold spanning time, space, geome- try, and orientation. By conditioning on the evolving orientation field, this formulation facilitates principled deformation inference aligned with the dynamic intent.
Multivariate Modeling of Dynamic States. To reflect complex motion patterns across space and time, we represent each region of the scene with multivariate dynamic state that jointly encodes temporal progression and structural changes. Specifically, let p' E R denote the position of a point time f. We define its local dynamic state as:
c (Ap. 0) E M,,
where Ap and Ag represent deviations in position and geometry (e.g., scale and rotation), O E SO((3) the orientation derived from our Global Orientation Field, and M denotes a manifold spanning time, space, geometry, and orientation.
This formulation captures the intrinsic interdependence among these factors in dynamic scenes, treating them as jointly evolving rather than isolated variables. To model their correlations in unified and flexible manner, we then represent each dynamic state & as probabilistic entity with a multivariate Gaussian over M:
~ C~NN Eg), μe E M,, ΣS EXD
where μe is the canonical dynamic state, and Eg captures intra- and inter-correlations. By grounding this probabilistic state in the evolving global orientation field, our model is able to learn region-specific variability that aligns with the scene's 's underlying dynamic intent.
Hyper- Gause-Gn Representation. To realize our formulation, we build upon 3D Gaussian Splat- ting [37] and equip each primitive with the multivariate Gaussian distribution defined in Eq. (7). This results in dynamic representation that encodes the localized motion and geometric variability within a unified probabilistic entity.
Specifically, we begin by initializing a set of 3D Gaussians {G₁} across the scene, each parameterized by its position Mp,, rotation R,, scale Sj. opacity aj. and color cj (Section 3). To enable dynamic modeling, we associate each Gj with state mean llg and covariance Σg:
E(spids) jgg = (Map) Hag, Me, HO), Σe = Σ E(ApAAR).I..))
Eppp, E1,0)
(8)
where MAp E 33 represents the spatial offset, Mdd E R2 SO(3) describes local variations in scale and rotation, μt within the global field. The covariance matrix captures both intra- (e.g., temporal or geometric) and cross-gromd group (e.g., spatial opatial-oriennationaa)) dependencies to model complex dynamics.
the temporal coordinate, and μo SO(3) represents the orientation
For numerical stability, we parameterize Σg via Cholesky decomposition [11, 22, 23]: = L where Le is a lower-triangular matrix. Both وlg and Lg are treated as learnable parameters and jointly optimized with other Gaussian attributes. More details are provided the supplementary materials.
Conditioned Slicing for Dynamic State Inference. Given the Syper-Gaussioe representation, we seck to infer concrete dynamic behavior by conditioning on local motion context. Our core intuition
5

---

is that each part of the scene undergoes diverse localized deformation in response to how the region expected to evolve under temporal and orientational cues.
Specifically, we define query (t', OO')) IR (O(3), representing the target time and its associated orientation. Given the probability P(E) = (N,, Σe), we derive the conditional distribution over the spatial and geometric subspace:
P(Ap, Ag L', VO') - (Msond Exand),
(9)
with conditional mean and covariance given by Gaussian slicing rules:
Meond (HAP. M&g) + (Ap. 2T.0, Me, O θ Po):
(10)
cond = ((p)AR ((A. (1.00 E(..O).Ap.AR))
(11)
where O denotes the relative rotation operator in SO(3). Practically, rather than sampling (Ap. Ag) from the conditional distribution, which may introduce optimization instability, we adopt the maximun-likelihood estimation principle and use the conditional mean Moodd as the determinis- tic prediction:
(12) This inferred local dynamic state captures the expected position and geometry variations given temporal and orientation contexts. Through this slicing process, the hyper-Gaussian can be adaptively projected onto various scene regions, enabling temporally and spatially coherent reconstruction.
P := E[Ap, Ag 11,011 Mlcood
Modulation of Hyper-Gaussian Attributes. To integrate our dynamic modeling into the efficient and differentiable rendering pipeline, we modulate 3D Gaussian primitives based the inferred dynamic state. For target time t' and local orientation O can obtain Poood from the conditioned distribution using Eq. (12) and apply it to the 3D Gaussian parameters accordingly:
(13) where MAp: Pas: PAR denote the inferred variations in position, scale, and rotation, respectively, and @ is quaternion- -base rotation composition. In parallel, we modulate each primitive's 's opacity based on its proximity to the canonical state in both time and orientation. This confidence-weighted visibility is defined as:
P = u5 + PAp. Sc' S MAs: È = R 4 PAR.
8d
exp (-2 Σイ -
-1)2 (o" e ('O ) Σo' (o' Q vO))
(14)
Together, these modulations encourage each Syper-Gaussian to adaptively express geometry and appearance consistent with the temporal and orientational cues, while suppressing noisy contributions from out-of-support regions.
Anchor-Guided Deformation and Conditioning. Recall that to query the hyper-Gaussian conditioned slicing defined by Eq. (9), we must first estimate each Gaussian's 's local motion context, including its position and orientation under the evolving scene. We achieve this through anchor-driven deformation within our Global Orientation Field, drawing inspiration from Embedded Deformation Graphs [65, 69] and their extensions in dynamic modeling [5, 13, 44, 39, 52, 83].
Gaussian via
Specifically, each Gaussian Gj is softly associated with a set of K nearby oriented anchors via skinning weights [Way] } computed from spatial proximity. Each anchor i carries time-varying pair (O), +(). representing its orientation and position at time f. We express the relative anchor transformation in (3) across time as: AQ -q' (O)" , 7. (O/) 7/)-7,1111 which is further converted into a dual quaternion - TD. To deform 93 towards any target time t', we aggregate the relative transformations of its neighboring anchors using a weighted dual quaternion blend [52]: q!-r"
Blendn (wwz) )_ which then yields the updated attributes:
RR (T,,4,))) -; -) -R)) (pp)).
(15)
Importantly, this deformation scheme provides access to the local orientation Of required for state conditioning Eq. (9). However, this requires additional care since the deformed Gaussian may not coincide with any specific anchor. Therefore, we interpolate O' from surrounding anchors:
O Blendgo(3)
{uey. O ),))
(16)

---

MoSca [39]
(Trining View>
OriGS (Ours) (Triining View)
MoSca [39] (Novel View>
OriGS (Ours) (Novel View)
Figure 3: Visual comparisons of training sequence reconstruction and novel view synthesis in-the-wild videos. Please Q zoom for more details.
This produces a smoothly varying orientation field that respects both the global structure and local dynamics. By grounding the conditioned slicing on these anchor-driven estimates, our Gyuss-Gaussian representation can adaptively capture complex motion across different regions.
Experiments
5.1 Experimental Protocol
Datasets. We evaluate the 4D reconstruction and novel view synthesis performance of our OriGS on diverse set of in-the-wild monocular videos, emphasizing scenes with complex motion and object interactions. Our primary evaluation includes videos from DAVIS [59], OpenAl SORA [6]. and YouTube- VOS [74], which reflect the casual and unconstrained nature of our target setting. For quantitative benchmarking, we additionally report results on the DyCheck dataset [19], which contains seven scenes with multi-camera captures for novel view synthesis evaluation.
Evaluation. For in-the-wild videos, where ground-truth novel views are inherently unavailable, we conduct qualitative evaluations assess the visual fidelity of reconstructed scenes. For the DyCheck dataset [19]. which provides multi-camera setups for quantitative assessment, we evaluate our method using standard image quality metries: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) [68], and Learned Perceptual Image Patch Similarity (LPIPS) [78]. The DyCheck dataset uses three synchronized cameras: one hand-held mobile view and two stationary references. Following [19], we measure reconstruction quality from the reference viewpoints.
5.2 Experimental Results
In-the-wild. We qualitatively evaluate OriGS on range of casually captured monocular videos that exhibit diverse motion patterns. Figure presents side-by-side comparisons with MoSca [39], recent state-of-te thettrrr system for automatic 4D reconstruction in real-world scenarios. Across
7

---

Table 1: Quantitative comparisons of novel view synthesis on DyCheck. We report PSNR, SSIM, and LPIPS metrics across seven scenes, Markers and denote with and without ground-Iruth camera pose, respectively.
The best second best and third best results are highlighted.
Apple
Block
Paper Windmill
Space Out
Method
PSNRT
SSIMT
LPIPS)
PSNR1
SSIMT
LPIPS]
PSNRT
SSIMI
LPIPS]
PSNRT
SSIM
LPIPS.
o T-NeRF [19]
17.43
0.728
0,508
17.52
0.669
0,346
17.55
0.367
0.258
17.71
0.591
0.377
(SFFF(42) [42]
16.47
0.754
0.414
14.71
0.606
0.438
14.94
0.272
0.348
17.65
0.636
0.341
Nerfies [55]
17.54
0.750
0.478
16.61
0.639
0,389
17.34
0.378
0.211
17.79
0.622
0.303
HyperNeRF [56]
17.64
0,743
0.478
17.54
0.670
0,331
17.38
0.382
0.209
17.93
0.605
0.320
RoDynRF [47]
18.73
0.722
0.552
18.73
0.634
0.513
16.71
0.321
0.482
18.56
0.594
0.413
DynPoint [80]
17.78
0.743
17.67
0.667
17.32
0,366
17.78
0.603
o PGDVS [79)
16.66
0.721
0.411
16.38
0.601
0.293
17.19
0,386
0.277
16.49
0.592
0.326
-DyBluRF [7]
18.00
0.737
0.488
17.47
0.665
0.349
18.19
0.405
0.301
18.83
0.643
0.326
o CTNNRF [50]
19.53
0.691
19.74
0.626
17.66
0,346
0.346
18.11
0.601
Dyya GS [49]
7.65
0.766
7.55
0,684
6.24
0.729
6.79
0.733
4DGS [71]
15.41
0.450
11.28
0.633
15.60
0.297
14.60
0.372
DD-NCC [35]
16.83
0.752
0,469
15.53
0.632
0,350
18.01
0.432
0.209
18.69
0.640
0.246
Marbles [64]]
16.50
0.499
16.11
0,363
16.19
0.454
15.97
0.437
SoM [67]
MoSca [39]
15.99
0.705
0.503
18.20
0.665
0.324
21.37
0.645
0.175
22.57
0.750
0.198
OriGS (Ours)
17.21
0.739
0.428
18.67
0.676
0.317
21.48
0.646
0.171
22.78
0.754
0.194
o OriGS (Ours)
19.46
0.807
0.341
18.76
0.690
0.319
22.46
0.751
0.152
20.87
0.672
0.249
Spin
Teddy
Wheel
Average
Method
PSNR1
SSIMT
LPIPS)
PSNR1
SSIM1
LPIPS
PSNRT
SSIM1
LPIPS
PSNRT
SSIM
LPIPS.
T-NeRF [19]
19.16
0.567
0.443
13.71
0.570
0.429
15.65
0.548
0.292
16.96
0.577
0.379
NSFF [42]
17.26
0.540
0.371
12.59
0.537
0.527
14.59
0,511
0.331
15.46
0.551
0.396
Nerfies [55]
18.38
0.585
0.309
13.65
0.557
0.372
13.82
0.458
0.310
16.45
0.570
0.339
HyperNeRF [56]
19.20
0.561
0.325
13.97
0,568
0,350
13.99
0.455
0,310
16.81
0.569
0.332
RoDynRF [47]
17.41
0.484
0.570
14.33
0.536
0.613
15.20
0.449
0.478
17.10
0.534
0.517
DynPoint [80]
19.04
0,564
13.95
0,551
14.72
0,515
16.89
0.573
PGDVS [79]
18.49
0.590
0.247
13.29
0,516
0.399
12.68
0.429
0,429
15.88
0.548
0,340
DyBBuRR [?]
18.20
0.400
0.572
0.435
16.26
0.575
0.325
17.37
0.373
-CTNNRF [50]
19.79
0,516
14.51
0.509
14.48
0,430
17.69
0.531
Dyyn. GS [49]
8.08
0.651
7.41
0.690
7.28
0.593
7.29
0.692
+4GG [71]
14.42
0.339
12.36
0,466
11.79
0.436
13.64
0.428
à DD-NCC [35]
17.78
0.585
0.309
12.19
0.536
0,503
13.27
0.549
0.349
16.04
0.589
0.348
MMablees [64]
17.51
0,424
13.68
0.443
14.58
0.389
15.79
0,428
SoM [67]
17.32
0.598
0.296
MoSca [39]
20.71
0.677
0.229
15.54
0.624
0.356
17.74
0.667
0.238
18.84
0.676
0.289
-OriSS (Ours)
20.92
0.670
0.215
16.11
0.639
0.334
17.92
0.670
0.233
19.30
0.685
0.270
OOiGS (Ours)
21.62
0.759
0.177
16.45
0.649
0.325
18.19
0.684
0.226
19.69
0.716
0.256
both training sequence reconstruction and novel view synthesis, OriGS consistently recovers cleaner geometry and preserves structural integrity under complex dynamics. Notably, our method remains robust in challenging scenarios, such as fast movement (e.g., the spinning pig and the parrot). These results demonstrate that OriGS generalizes effectively to complex real-world videos, enabling faithful 4D reconstruction and compelling view synthesis without relying on multi-view constraints.
DyCheck. We compare OriGS against recent of of-the-an art methods, including NeRF- [19, 42, 55, 56, 47, 7, 50, 80, 79] and 3DGS-based [49, 71, 35, 64, 67, 39]. Table 1 reports quantitative results across all seven scenes using PSNR, SSIM, and LPIPS. We indicate whether ground-trutt camera poses are used by markers (with) and + (without). Our OriGS achieves superior reconstruction fidelity, especially in scenes involving rotational deformation (e.g., the "Apple" and "Paper Windmill" scenes), where prior methods tend to produce artifacts. Notably. OriGS outperforms all baselines in the average across all three metrics, even without access to ground-truth camera pose. When provided with accurate poses, performance is further improved in most scenes, demonstrating the effectiveness of our framework. A minor exception is the "Space Out" scene, which is limited by inaccurate pose metadata provided in the dataset, as previously observed in [67].
Figure 4 showcases the qualitative advantages of OriGS in novel view synthesis. Compared to mainstream methods, our OriGS can produce sharper surface boundaries and more coherent geometry under challenging motion. For example, in the "Paper Windmill" scene (top row), other methods typically struggle with blurred or broken structures due to the rapid spinning motion. In contrast, our ÖriGS, by anchoring complex dynamics to a coherent orientation field, maintains the integrity of the rotating blades. Compared to OriGS, bascline results often exhibit severe ghosting artifacts and
8

---

GT
D3DGS [76]
Marbles [64]
SoM [67]
MoSca [39]
OriGS (Ours)
Figure 4: Visual comparisons of novel view synthesis DyCheck. Markers and denote with and without ground-truth camera pose, respectively. Our OriGS can recover sharper geometry and more coherent motion.
rigid-body distortion in scenes with hand-object occlusion (e.g., the "Block" scene (bottom row)), further emphasizing the superiority of our method in handling complex motion.
5.3
Ablation Study
To analyze the impact of each design component in OriGS, we conduct progressive ablation study from minimal baseline to our full model. Table reports quantitative results on the "Apple" scene
from DyCheck [19] using the provided camera pose, and Figure presents qualitative comparisons on the "Libby" scene from DAVIS [59]. We follow the same experimental setting for novel view synthesis as in Scction 5.2. (i) We begin with 3DGS-MLP as the baseline, where each 3D Gaussian primitive up- dated individually across time using shared deformation MLP, without explicit motion modeling. This baseline can only capture simple movement and struggles with structural coherence. (ii) Next, we consider Deform w/ GOF, which replaces the MLP with anchor-driven deformation guided by our Global Orientation Field. This variant enables Gaussian primitives capture global motion, yet lacks the expressiveness model complex dynamics across regions. (iii) To enable localized, time-adaptive behavior, we augment the model with Haper-Gaussian w/ t, where each Gaussian encodes probabilistic state over time, space, and geometry (i.e., & (Ap, Ag, t) ). The deformation is inferred by conditioning this hyper-Gaussian on the target time step, enabling smoother and temporally adaptive reconstruction. (iv) Finally, our OriGS further incorporates orientation as a conditioning signal. By anchoring local dynamics both temporal and orientational cues, it enables the model capture direction-senan sensitive motion patterns and better preserve coherent evolution across regions with diverse dynamics.
Table 2: Ablation study on OriGS variants on the "Apple"
scene from DyCheck [19].
Method
PSNRT
SSIM
LPIPSJ
(i) 3DGS-MLD (Baseline)
13.47
0.532
0.586
(ii) Deform w/ GOF
16.28
0.694
0,476
(iii)
w/
18.71
0.750
0.393
(iv) OriGS (Full)
19.46
0.807
0.341
Original
3DGS-MLP
Deform w/ GOF
Hyper- Gaussian w/ L
OriGS
Figure Ablation study on OriGS variants on "Libby" scene from DAVIS [59]. We show a frame from the original input video (leftmost) and zoomed-in novel view synthesis results from different ablated variants.
The main region interest the dog behind the pillar which exhibits fast movement and heavy occlusion.

---

6 Conclusion
In this work, we presented Orientation-an -anchoor Gaussian Splatting (OriGS), novel framework for 4D reconstruction from casually captured monocular videos. Our core insight is to embrace orientation dynamic anchor that organizes how different parts of the scene deform and interact. By introducing Global Orientation Field and Hyper-Gaussian Gaussian representation, OriGS models long-range evolution and local motion variation unified and principled manner, thereby capturing coherent dynamics across the scene. Extensive experiments demonstrate the superiority of our OriGS, enabling high-fidelity reconstruction and view synthesis under challenging real-world scenarios.
Acknowledgments: This research is supported by NSF IIS-2525840, CNS-2432533 BCCS-25145744 NIH IRF MMHI and Cisco Research unrestricted gift. This article solely reflects opinions and conclusions of authors and not funding agencies.
References
[1] Mare Alexa, Daniel Cohen-Or, and David Levin. As-rigid-as -st-sssiblo shape interpolation. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 165-172. 2023.
[2] Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf. Matthew O' O'Toole, and Changil Kim. Hyperreel: High-fidelity 6-dof video with ray-conditioned sampling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16610-16620, 2023.
[3] Jeongmin Bae, Seoha Kim, Youngsik Yun, Hahyun Lee, Gun Bang, and Youngjung Uh. Per- gaussian embedding- baased 321-335. Springer, 2024.
based deformation for deformable 3d gaussian splatting. In ECCV, pages
[4] Yanqi Bao, Tianyu Ding, Jing Huo, Yaoli Liu, Yuxin Li, Wenbin Li, Yang Gao, and Jiebo Luo. 3d gaussian splatting: Survey, technologies, challenges, and opportunities. In IEEE TCSVT, 2025.
[5] Aljaz Bozic, Pablo Palafox, Michael Zollhöfer, Angela Dai, Justus Thies, and Matthias Nießner. Neural non-rigid tracking. Advances in Neural Information Processing Systems, 33:18727- 18737, 2020.
[6] Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024.
[7] Minh-Quan Viet Bui, Jongmin Park, Jihyong Oh, and Munchurl Kim. Dyblurf: Dynamic deblurring neural radiance fields for blurry monocular video. arXiv preprint arXiv:2312.13528, 2023.
[8] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 130-141 141, 2023.
[9] Guikun Chen and Wenguan Wang. A survey on 3d gaussian splatting, arXiv preprint arXiv:2401.03890, 2024.
[10] Jianmei Dai, Zhilong Zhang, Shiwen Mao, and Dampu Liu. A view synthesis-bart based 360 vr caching system over mec-enabled c-ran. IEEE Transactions on Circuits and Systems for Video Technology, 30(10)33843-33 3855, 2019.
[11] Stavros Diolatzis, Tobias Zirr, Alexander Kuznetsov, Georgios Kopanas, and Anton Kaplanyan. N-dimensional gaussians for fitting of high dimensional functions. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[12] Carl Doersch, Pauline Luc, Yang, Dilara Gokay, Skanda Koppula, Ankush Gupta, Joseph Heyward, Ignacio Rocco, Ross Goroshin, João Carreira, et al. Bootstap: Bootstrapped training for macking-any-poine Proceedings of the Asian Conference on Computer Vision, pages 3257-3274, 2024.
10

---

[13] Mingsong Dou, Jonathan Taylor, Henry Fuchs, Andrew Fitzgibbon, and Shahram Izadi. 3d scanning deformable objects with single rgbd sensor. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 493-501, 2015.
[14] Yuanxing Duan, Fangyin Wei, Qiyu Dai, Yuhang He, Wenzheng Chen, and Baoquan Chen. 4d-rotor gaussian splatting: towards efficient novel view synthesis for dynamic scenes. In ACM SIGGRAPH 2024 Conference Papers, pages 1-11, 2024.
[15] Bardienus Duisterhof, Zhao Mandi, Yunchao Yao, Jia- Wei Liu, Mike Zheng Shou, Shuran Song, and Jeffrey Ichnowski. Md-splatting: Learning metric deformation from 4d gaussians in highly deformable scenes. arXiv preprint arXiv:2312.00588 2(3), 2023.
[16] Denis Evans. On the representation of orientation space. Molecular physics. 34(2)311-3325, 1977.
[17] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIC- GRAPH Asia 2022 Conference Pupers, pages 1-9, 2022.
[18] Sara Keil,, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12488, 2023.
[19] Hang Gao, Ruilong Li, Shubham Tulsiani, Bryan Russell, and Angjoo Kanazawa. Monocular dynamic view synthesis: A reality check. Advances in Neural Information Processing Systems, 35:33768-3333 33780, 2022.
[20] Kyle Gao, Yina Gao, Hongjie He, Dening Lu, Linlin Xu, and Jonathan Li. Nerf: Neural radiance field in 3d vision, comprehensive review. arXiv preprint arXivv2210.00377, 2022.
[21] Shenyuan Gao, Jiazhi Yang, Li Chen, Kashyap Chitta, Yihang Qiu, Andreas Geiger, Jun Zhang, and Hongyang Li. Vista: A generalizable driving world model with high fidelity and versatile controllability. arXiv preprint arXiv: 17398 2024.
[22] Zhongpai Gao, Benjamin Planche, Meng Zheng, Anwesa Choudhuri, Terrence Chen, and Ziyan Wu. 6dgs: Enhanced direction-aware gaussian splatting for volumetric rendering. In ICLR, 2025.
[23] Zhongpai Gao, Benjamin Planche, Meng Zheng, Anwesa Choudhuri, Terrence Chen, and Ziyan Wu. 7dgs: Unified ceattalltemporal-angee gaussian splatting, arXiv preprint arXiv:2503.0790 077446 2025.
[24] Zhongpai Gao, Meng Zheng, Benjamin Planche, Anwesa Choudhuri, Terrence Chen, and Ziyan Wu. Render-fm: foundation model for real-time photorealistic volumetric rendering. arXiv preprint arXiv:2505. 17338, 2025.
[25] Colin Goodall. Procrustes methods in the statistical analysis of shape. Journal of the Royal Statistical Society: Series B (Methodological), 53(2):285-321, 1991.
[26] John C Gower. Generalized procrustes analysis. Psychometrika, 40(1):33-51, 1975.
[27] David Halliday, Robert Resnick, and Jearl Walker. Fundamentals of physics. John Wiley & Sons, 2013.
[28] Mu Hu, Wei Yin, Chi Zhang, Zhipeng Cai, Xiaoxiao Long, Hao Chen, Kaixuan Wang, Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero-shot metric depth and surface normal estimation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
[29] Wenbo Hu, Xiangjun Gao, Xiaoyu Li, Sijie Zhao, Xiaodong Cun, Yong Zhang, Long Quan, and Ying Shan. Depthcrafter: Generating consistent long depth sequences for open-wordd videos. arXiv preprint 2afi::229000099 2024.

---

[30] Wenlong Huang, Chen Wang. Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. arXiv preprint arXiv:2307.0597. 2023.
[31] Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang. Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 4220-4230, 2024.
[32] Takeo Igarashi, Tomer Moscovich, and John F Hughes.
shape manipulation.
ACM transactions on Graphics (TOG), 24(3): 134-11144,1,,,,, 2005.
[33] Anastasia loannidou, Elisavet Chatzilari, Spiros Nikolopoulos, and loannis Kompatsiaris. Deep learning advances in computer vision with 3d data: A survey. ACM computing surveys (CSUR), 50(2):1-38, 2017.
[34] Wei Jiang, K Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from a single video. In European Conference on Computer Vision, pages 402-418. Springer, 2022.
[35] Moritz Kappel, Florian Hahlbohm, Timon Scholz, Susana Castillo, Christian Theobalt, Martin Eisemann, Vladislav Golyanik, and Marcus Magnor. D-npe: Dynamic neural point clouds for non-rigid view synthesis from monocular video. In Computer Graphics Forum, page e70038. Wiley Online Library, 2024.
[36] Nikita Karaev, lurii Makarov, Jianyuan Wang, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker3: Simpler and better point tracking by pseudo-labelling real videos. arXiv preprint arXiv:2410.1118. 2024.
[37] Bernhard Kerbl, Georgios Kopanas, Thomas Leimküühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. In ACM Trans. Graph., 2023.
[38] Junoh Lee, Chang Yeon Won, Hyunjun Jung, Inhwan Bae, and Hae-Gon Jeon. Fully explicit dynamic gaussian splatting. NewrlPS, 37:5384-540 54044,540,4440 2024.
[39] Jiahui Lei, Yijia Weng, Adam Harley, Leonidas Guibas, and Kostas Danilidis. Mosca: Dynamic gaussian fusion from casual videos via 4d motion scaffolds. In CVPR, 2025.
[40] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 5521 2022.
[41] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8508-8520, 2024.
[42] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-tima view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498 6508, 2021.
[43] Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4273-4284, 2023.
[44] Yiming Liang, Tianhan Xu, and Yuta Kikuchi. Himor: Monocular deformable gaussian reconstruction with hierarchical motion representation. In CVPR, 2025.
[45] Chen-Hsuan Lin. Wei-Chiu Ma, Antonio Torralba, and Simon Lucey. Barf: Bundle-adjusting neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5741-575: 2021.
[46] Qingming Liu, Yuan Liu, Jiepeng Wang, Xianqiang Lyv, Peng Wang, Wenping Wang, and Junhui Hou. Modgs: Dynamic gaussian splatting from castally-captured monocular videos. In ICLR, 2025.
12

---

[47] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf, Changil Kim, Yung Yu Chuang, Johannes Kopf, and Jia-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13-23, 2023.
[48] Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold- gs: Structured 3d gaussians for view-adaptive rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20654-20664, 2024.
[49] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan, Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 2024 International Conference on 3D Vision (3DV), pages 800-809. IEEE, 2024.
[50] Xingyu Miao, Yang Bai, Haoran Duan, Fan Wan, Yawen Huang. Yang Long, and Yefeng Zheng. Cinerf: Cross-time transformer for dynamic neural radiance field from monocular video. Pattern Recognition, 156: 10729, 2024.
[51] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, (5(1):99-106, 2021.
[52] Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 343-352, 2015.
[53] Abby O'Neill, Abdul Rehman, Abhiram Maddukuri, Abhishek Gupta, Abhishek Padalkar, Abraham Lee, Acorn Pooley, Agrim Gupta, Ajay Mandlekar, Ajinkya Jain, et al. Open x- embodiment: Robotic learning datasets and rt-x models: Open x-embodiment collaboration O. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 6892-6903. IEEE, 2024.
[54] Jongmin Park, Minh-Quan Viet Bui, Juan Luis Gonzalez Bello, Jaeho Moon, Jihyong Oh, and Munchurl Kim. Splinegs: Robust notion-adaptive spline for real-time dynamic 3d gaussians from monocular video. In CVPR, 2025.
[55] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualle Nerfies: Deformable neural radiance fields. In ICCV, 2021.
[56] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. In SIGGRAPH Asia, 2021.
[57] Sungheon Park, Minjung Son, Seokhwan Jang, Young Chun Ahn, Ji- Yeon Kim, and Nahyup Kang. Temporal interpolation is all you need for dynamic neural radiance fields. In Proceedings of the IEEE/CVF conference computer vision and paltern recognition, pages 4212-4221, 2023.
[58] Luigi Piccinelli, Yung-Hsu Yang, Christos Sakaridis, Mattia Segu, Siyuan Li, Luc Van Gool, and Fisher Yu. Unidepth: Universal monocular metric depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10106-10116, 16, 2024.
[59] Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine- MokkingHHoruug,. and Lue Van Gool. The 2017 davis challenge video object segmentation. arXiv preprint 1704v01704.00675 2017.
[60] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021.
[61] Ahmed A Shabana. Computational dynamics. John Wiley & Sons, 2009.
[62] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 16632-16642, 2023.
13

---

[63] Olga Sorkine and Mare Alexa. As-rigid-ar as as-posiible surface modeling. In Symposium on Geometry processing, volume 4, pages 109-116. Citeseer, 2007.
[64] Colton Stearns, Adam Harley, Mikaela Uy, Florian Dubost, Federico Tombari, Gordon Wet- zstein, and Leonidas Guibas. Dynamic gaussian marbles for novel view synthesis of casual monocular videos. In SIGGRAPHI Asia, 2024.
[65] Robert W Summer, Johannes Schmid, and Mark Pauly. Embedded deformation for shape manipulation. In ACM siggraph 2007 papers, pages 80-es, 2007.
[66] Fengrui Tian, Shaoyi Du, and Yueqi Duan. Mononerf: Learning generalizable dynamic radi- ance field from monocular videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17903-17913, 2023.
[67] Qianqian Wang. Vickie Ye, Hang Gao, Jake Austin, Zhengqi Li, and Angjoo Kanazawa. Shape of motion: 4d reconstruction from a single video. arXiv preprint wrXiv:240..33764 2024.
[68] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Fero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, (3(4):600- 612, 2004.
[69] Ofir Weber, Olga Sorkine, Yaron Lipman, and Craig Gotsman. Context-aware skeletal shape deformation. In Computer Graphics Forum, pages 265-274, Wiley Online Library, 2007.
[70] Chung- Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher- Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF conference computer vision and pattern Recognition, pages 16210-16220, 2022.
[71] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie. Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 20310-20320, 2024.
[72] Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, et al. Mars: An instance-award aware, modular and realistic simulator for autonomous driving. In CAAI International Conference on Artificial Intelligence, pages 3-15. Springer, 2023.
[73] Yuxi Xiao, Qianqian Wang, Shangzhan Zhang, Nan Xue, Sida Peng, Yujun Shen, and Xiaowei Zhou. Spatialtracker: Tracking any 2d pixels in 3d space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 20406-20417, 2024.
[74] Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint
arXtv:1809.03327. 2018.
[75] Zeyu Yang, Hongye Yang. Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint 10.1422300.1064., 2023.
[76] Ziyi Yang. Xinyu Gao. Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20331-20341 2024.
[77] Jihwan Yoon, Sangbeom Han, Jaeseok Oh, and Minsik Lee. Splinegs: Learning smooth trajectories in gaussian splatting for dynamic scene reconstruction. In ICLR, 2025.
[78] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
[79] Xiaoming Zhao, Alex Colburn, Fangchang Ma, Miguel Angel Bautista, Joshua M Susskind, and Alexander Schwing. Pseudo -eneralieer dynamic view synthesis from a video. arXiv preprint arXiv:2310.0885 2023.
14

---

[80] Kaichen Zhou, Jia-Xing Zhong, Sangyun Shin, Kai Lu, Yiyuan Yang, Andrew Markham, and Niki Trigoni. Dynpoint: Dynamic neural point for view synthesis. Advances in Neural Information Processing Systems, 36:69532-69545, 2023.
[81] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnifi- cation: Learning view synthesis using multiplane images. arXiv preprint 2018.
[82] Jiaxuan Zhu and Hao Tang. Dynamic scene reconstruction: Recent advance in real-time rendering and streaming, arXiv preprinl arXiv:2503.08/ 06, 2025.
[83] Michael Zollhöfer, Matthias Nießner. Shahram Izadi, Christoph Rehmann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, al. Real-time non-rigid reconstruction using an rgb-d camera. ACM Transactions on Graphics (ToG), 33(4):1-12, 2014.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims in the abstract and introduction accurately reflect the contributions the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer supplementary materials.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Please refer Section 5.1 and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification:
The code will be released publicly.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeececceccccccct httpa//ggiiieeeeeeSuuuuiiiiii /CodeSubmie ssionnollll onPolicy))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
18

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please refer supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /ttp///eerpp../// cc
Answer: [Yes]
Justification:
The research conducted in this paper adheres the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer supplementary materials.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
19

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All external assets such as datasets and code libraries used in the research are properly credited, and their licenses are respected and clearly mentioned.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
20

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact core methodology, scientific rigorousness, originality of the research, declaration is not required.
21

---

Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to our LLM policy (https / //n/ur/iss.p////e eeeeeeeeeen22 /LLM) for what should or should not be described.
22

---

A Evaluation on Point Tracking
Following the DyCheck benchmark [19] and prior works [64, 391, we perform a quantitative evalua- tion on point tracking accuracy. We compare with both tracking and reconstruction methods. Results shown in Table indicate the superiority of our OriGS.
Table 3: Correspondence Evaluation on DyCheck with PCK-T @0.05%.
Method
Nerfies [55]
HyperNeRF [56]
Dyn. GS [49]
4DGS [71]
PCK-T f
0,400
0,453
0,079
0.073
Method
CoTracker [36]
Marbles [64]
MoSca [39]
OriGS (Ours)
PCK-T f
0,803
0.806
0.824
0.851
B
Demo Videos
To further showcase the effectiveness of OriGS in real-world scenarios, we include additional video demonstrations of in-the-wild 4D reconstruction in the supplementary material. Following the same experimental setup as in the main paper, we present visualizations of both training sequence reconstruction and novel view synthesis, comparing OriGS with the recent state-of-the 4D provide video demonstrations of the OriGS variants designed
reconstruction system. In addition, in our ablation study, illustrating how individual components affect reconstruction quality. These results collectively highlight the temporal consistency and structural fidelity of OriGS across diverse scenes and motion patterns.
C Implementation Details
Oriented Anchor Initialization. We follow similar initialization pipeline provided in recent open-source codebases of dynamic reconstruction frameworks [64, 39J. Specifically, we first extract long-range 2D point trajectories using SpatialTracker [73] and obtain per-frame depth maps using DepthCrafter [29]. These 2D correspondences are lifted into 3D space using estimated camera parameters from bundle adjustment [39]. The resulting 3D trajectories are then converted into oriented anchors, whose initial principal directions are estimated by applying PCA over faray-frame motion. In our experiments, we set the temporal window size W = 5. This step provides a stable estimate of forward direction, which is then temporally propagated to form Global Orientation Field.
Hapsr-Gaussian Parameterization. We adopt the 3D Gaussian Splatting (3DGS) [37] as the base representation and extend each Gaussian primitive with hyper-Gaussian parameterized by a canonical state jg and Cholesky-decomposel covariance Σς = Lg where Le is a lower- triangular matrix [22, 23, 11, 24]. To improve computational efficiency, we avoid constructing the full covariance matrix Σg. Instead, we further factor it into two learnable subcomponents: (i) marginal covariance Ext.o) over temporal and orientational variables, which inherits the Cholesky- decomposed parameterization to ensure positive semi- deiiiidefiitenesss and (ii) cross- covariance term ((Ap.Ag)((t.) that captures the correlation between dynamic geometry (Ap, Ag) and temporal- orientational context. During training, we jointly optimize these covariance terms along with the canonical mean jl and the original 3DGS parameters (position, scale, rotation, color, and opacity). This factorization strategy circumvents the overhead of full matrix parameterization while retaining the necessary statistical coupling for efficient conditioned slicing.
Optimization. We optimize the model using the differentiable issterzaation-bssed rendering pipeline from 3DGS [37], adapted to support digh-dimensional slicing and dynamic modulation. The loss function combines: (i) photometric loss [37], an RGB reconstruction loss between rendered and ground-trute -ruth images, (ii) 2D correspondence loss, alignment to long-range 2D tracks and depth priors from foundation models, as in [64, 39, 44, 54], and (iii) deformation regularization, an as- as-rigid-as possible constraint [73, 63, 1, 32] applied to anchor-guided transformations. For scalability and high-fidelity reconstruction, we also design a pruning-and-densiffen. densiiinsiiiation scheme: Gaussian primitives with low opacity are pruned, while spatial regions exhibiting high response gradients with respect to up and Map are densified through local duplication. All experiments are conducted on a single
23

---

NVIDIA RTX A6000 GPU, and the full optimization of a typical scene takes approximately 0.5-2 hours, depending on video length and complexity.
D Limitations
While OriGS demonstrates promising results, it also presents certain limitations. (i) In scenes dominated by simple motion, such as near-rigid translation along a fixed axis, orientation tends to remain approximately constant over time. Consequently, conditioning cues may offer limited additional benefit, yet our framework still incurs unnecessary optimization complexity of high-dimensional modeling. Future work could explore adaptive mechanisms that orientation cues based on the complexity of motion in the
these orientational
modulate the use or dimensionality scene. (ii) OriGS leverages 2D priors such as point trajectories and depth from vision foundation models to initialize and optimize the orientation field. The reliability of these priors can affect the quality of 4D reconstruction. This reflects a deeper challenge tied to the development of reliable visual priors, which has long been a cornerstone of progress in computer vision research.
E Broader Impacts
OriGS provides unified framework for reconstructing dynamic scenes from casual monocular videos, which can benefit various real-world applications. For instance, in virtual and augmented reality, OriGS can reconstruct dynamic environments or actors from consumer- graeeerr-rrdeee-e video input alone, lowering the barrier to immersive content creation. Our work can also support robotics and behavioral analysis, especially where low-cost monocular capture is the only viable option. However, as with other scene reconstruction techniques, OriGS could potentially be misused for unauthorized replication of environments or individuals. While our framework is not designed for such misconduct, responsible usage should consider privacy and ethical concerns.
24


Reviews:
[{"review_id": "aqwyWtm2yE", "reviewer": "Reviewer_p9KW", "summary": "This is a paper on 4D reconstruction. The basic concept of the paper is that orientation of scene components changes smoothly due to physics constraints like preservation of the angular momentum.\nTo achieve this the authors introduce the notions of the Global Orientation Field and the Orientation-aware Hyper-Gaussian. \nAn initial dominant orientation is derived from the PCA using the temporal mean of trajectories over an initial window. Propagation happens through Procrustes on the mean compensated trajectories. A state consisting of position and orientation deviations, time, and the field orientation is modeled as a Gaussian distribution. The state components are dependent, the deviations are conditioned in time and field orientation. Deformation is modeled through anchors as in the Embedded Deformation Graph frameworks.\nTrajectory estimation leverages foundation models on depth and tracking. Bundle adjustment is applied to compute camera intrinsics and extrinsics.\nExperiments include evaluation on DAVIS, SORA, VOS, and DyCheck.", "questions": "Q1: Why does Procruses guarantee a smoothness? Is it because of the overlap in temporal windows? One would expect that a Kalman Filter would be more effective on that.\n\nQ2: Explain the difference between the anchors in this paper and the scaffolds used in MOSCA [36].\n\nQ3: It is awkard to include time in a state vector (reminds more of Galilean or Lorentz transformations), please explain why.\n\nQ4: It is not clear what the authors mean with hyper-state in line 192, isn't it just the mean of the states?\n\nQ4: Explain the special minus sign in line 209 and eq. (14), what is its exact definition? (the special plus in line 221 is clear because it refers to quaternions).\n\nQ5: Loss function belongs to the main text not the appendix.\n\nQ6: Which exactly foundation models are used for depth and 2D correspondence (many are listed in l. 35 of the appendix). \n\nQ6: Please explain the ablation. The jump from Deform with GOF to Hyper-Gaussian with t does include only the incorporation of t but no probabilistic formulation?\n\nQ7: The performance drops only slightly when bundle adjustment is used to estimate poses rather than the ground truth (behavior observed in MOSCA) as well. This is surprising given that camera motions are quite ambiguous and one would expect that an out of the box method like BARF would introduce errors. \n\nElaborate on the weaknesses.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "S1: The main strength of the paper is in the introduction of a dynamics model based on the smoothness of orientation. \n\nS2: Results are outperforming all other methods on DyCheck.\n\nS3: The ablation is helpful but has to be better explained (see Q below).\n\nI see only few major weaknesses in the paper (the rest will be in questions) and I am willing to increase the score.\n\nW1: The introduction of time in the state is intriguing but there is no prior in the science literature of not relativistic systems. There is no 3D object or other dynamic tracking that includes time and there is no motivation given. Does this result in a time-variant system? How can we have the time in the state when it appears also as a superscipt in the propagation?\n\nW2: The conditioning of the deviation on the dominant orientation (9) is also not explained.\nAt this point the reader wonders if $O^{t'}$ is one per scene or a real field.\nI think that (9-11) could be easily written in a clean Kalman Filter formulation. \n\nW3: Because of the introduction of probabilities the math became very messy. Several covariances appear that are not defined. Is the covariance with respect to $\\Delta O$ based on quaternions? \nHow is a Gaussian defined on the SE(3) manifold? What does a covariance in time mean (is there noise in time?). Pages 5 and 6 need significant mathematical re-work.\n\nVery minor weakness: $VU^T$ is in $O(3)$ it has to be multiplied with diag(1,1,det(UV^T)) to make it SO(3).", "was_revised": true, "final_justification": "The authors made a significant effort in clarifying the mathematics of the approach and I hope that they will incorporate it in their camera ready. The writing would have been clearer if the authors had attended a class on differential geometry which I really recommend them to do as soon as possible.", "rebuttal": "# **W1 & Q3: Time in the State**\n\n### **a. Motivation**\n**(1) Practical Goal.** Our motivation for introducing time (alongside orientation) is not to formulate a relativistic system, but to capture temporally varying motion patterns in a practical way for 4D reconstruction. By jointly conditioning on orientation and time, our model enables time-adaptive deformation in a unified slicing-based framework.\n\n**(2) Disambiguating Orientations.** While orientation provides valuable cues, it may not fully resolve temporal ambiguity. In real-world scenes, identical or similar orientations can arise at different time, yet the local dynamics or scene context can differ. For instance, an object returning to a familiar orientation may still be undergoing a different trajectory or surrounding geometry, depending on the phase of motion. Incorporating time allows the model to distinguish such cases and adapt to the appropriate context.\n\n\n### **b. Time in the State & Superscipt in Propagation**\nTime $t'$ is involved in two processes:\n\n**(1) Indexing Global Orientation Field.** We propagate the initial orientations through the video to build a Global Orientation Field. The superscript of $O^{t'}$ denotes that the orientation field is queried at frame $t'$.\n\n**(2) Conditioning Hyper‑Gaussian.** Time $t'$ is included in the conditioning variable for deformation.\n\nThe workflow is sequential and consistent: we first query the orientation field to obtain $O^{t'}$, and then use both $t'$ and $O^{t'}$ in conditioned slicing for local deformation.\n\n\n### **c. Validation from Prior Studies**\nRecent advances in dynamic scene modeling, such as 4DGS [1] and 4D-Rotor GS [2], have shown the benefits of embedding time into Gaussian primitives for learning spatio-temporal patterns (line 85 in our paper).\n\nOur method is conceptually aligned with this direction, but extends it through a Hyper-Gaussian, modeling space, time, geometry, and orientation in a unified representation.\n\n\n# **W2: Clarifying $O^{t'}$**\n**(1) $O^{t'}$ is a Field.** In Eq.9, $O^{t'}$ refers to the local orientation queried from Global Orientation Field at a specific 3D position and time step $t'$ (line 228 & 240). For each Hyper‑Gaussian, we first deform its position and rotation by **Eq.15**; we then query the orientation field at the deformed position and $t'$ to obtain a unique $O_{t'}$ by **Eq.16**. We will explain the query mechanism earlier (around line 205).\n\n\n**(2) Kalman Filter.** We appreciate the insightful suggestion and agree that Kalman Filter offers a principled perspective to our framework. We believe Kalman-based approach could provide additional benefits, such as refining motion consistency or modeling uncertainty in dynamic scenes. We consider this a promising avenue for future exploration.\n\n\n# **W3: Clarification on Math**\n### **a. Covariances Not Defined**\nWe define a multivariate Gaussian: $\\xi = (\\Delta p, \\Delta g, t, O).$ The covariance matrix $\\Sigma_{\\xi}$ captures the correlation among these variables. All covariance blocks appearing in Eq.8–14, such as $\\Sigma_{(t,O)}$ and $\\Sigma_{(\\Delta p,\\Delta g),(t,O)}$, are **submatrices** of $\\Sigma_{\\xi}$, which select the rows and columns corresponding to the respective variable groups in the subscript.\n\n\n### **b. Covariance in $O$**\nThe orientation $O \\in \\mathrm{SO}(3)$ is modeled using quaternions. When performing conditioned slicing (e.g., Eq.10), we adopt the logarithmic map and define the relative rotation $O^{t'} \\ominus \\mu_O$ as a tangent vector in $\\mathfrak{so}(3)$ (Please also refer to **Q4b: Special Minus Sign** below). Thus, the covariance over $O$ lies in the tangent space at $\\mu_O$, and we operate in Euclidean coordinates during slicing.\n\n\n### **c. Gaussian on Manifold**\nOur formulation does not place a Gaussian directly on the full $SE(3)$ manifold. Instead, the dynamic state is modeled on a product space. The Euclidean components are handled with a standard multivariate Gaussian, while the rotational component is handled via a quaternion-based relative rotation followed by a logarithmic map into the tangent space $\\mathfrak{so}(3)$. This Euclidean approximation allows us to apply closed-form Gaussian conditioning.\n\n**Established Principle.** Recent high-dimensional Gaussian splatting methods, such as N-DG [3] and 6DGS [4], embed additional cues into Euclidean spaces and perform Gaussian conditioning accordingly. Specifically, N-DG augments Gaussians with albedo and roughness, while 6DGS integrates view direction to model view-dependent effects. Building on this principled strategy, we extend the formulation to model space, time, geometry, and orientation in a unified representation.\n\n\n### **d. Covariance in Time**\nIncluding time during slicing allows each Hyper‑Gaussian to modulate its response by temporal proximity, acting as a soft activation kernel along the time axis. While we do not explicitly model \"noise in time\", including time as a slicing dimension naturally gives rise to a conditional variance along the temporal axis. Intuitively, its covariance serves to regulate how temporally sensitive the dynamic state is to deviations in the target time.\n\n\n# **W4: $VU^T$**\nIn our implementation, we have applied the standard correction step to make it $SO(3)$. We will clarify this.\n\n\n---\n\n\n# **Q1: Procruses**\n**(1) Clarification.** Our use of Procrustes is not intended to explicitly optimize for smoothness. Instead, we estimate the initial orientation over a temporal window in early frames, and propagate it via frame-to-frame Procrustes from a geometric perspective.\n\n**(2) Implicit Smoothness.** While not designed to enforce smoothness, we perform Procrustes within local neighborhoods (line 153), avoiding abrupt discontinuities. Additionally, we apply interpolation (line 241) over surrounding anchors to support arbitrary querying, which introduces smoothness.\n\n**(3) Kalman Filter.** We acknowledge that Extended Kalman Filter on $SO(3)$ offers a principled alternative, particularly under noisy or ambiguous motion. We view this as a promising direction for future exploration.\n\n\n# **Q2: Difference with Scaffold**\nOur oriented anchors and MoSca's scaffolds share conceptual similarities, as both are derived from Embedded Deformation Graphs. However, there are important differences:\n\n**(1) Lack of Orientation in Scaffold.** Scaffold nodes are not designed to encode the actual motion orientation. Their \"orientation\" are initialized as identity matrices and only **relative** rotations are estimated between adjacent nodes over time.\n\n**(2) Scaffold** serves as a deformation support structure to drive 3D Gaussian primitives, while relying heavily on the primitives themselves to capture complex local dynamics. **Oriented anchors** capture the forward direction across time, serving as conditioning signal for local deformation.\n\n\n# **Q3: Time in the State**\nPlease see **W1 & Q3: Time in the State** above.\n\n\n# **Q4a: Hyper-state**\nSorry for the confusion. The \"hyper-state\" in lines 192 and 222 should simply be \"state\", as it is just the mean of the states.\n\n\n# **Q4b: Special Minus Sign**\nThe $\\ominus$ aims to compute the relative rotation between two orientations. Given two unit quaternions $O^{t'}$ and $\\mu_O$, the relative rotation is:\n$$q_{\\text{rel}} = O^{t'} \\cdot \\mu_O^{-1}.$$\nBecause a unit quaternion's inverse is simply its conjugate, this can be calculated efficiently by:\n$$q_{\\text{rel}} := O^{t'} \\cdot \\overline{\\mu_O}.$$\nTo incorporate this relative rotation into our conditioned slicing, we next apply the standard $SO(3)$ logarithmic map. The $\\ominus$ is then defined as:\n$$O^{t'} \\ominus \\mu_O = \\log(q_{\\text{rel}}) \\in \\mathfrak{so}(3).$$\nThis produces an axis-angle vector representing the minimal rotation from $\\mu_O$ to $O^{t'}$, and ensures that the result lies in an Euclidean space suitable for Gaussian slicing.\n\n\n# **Q5: Loss Function**\nWe will move the optimization detail to main paper as Section 4.3.\n\n\n# **Q6a: Foundation Models**\nWe use DepthCrafter [5] and SpatialTracker [6].\n\n\n# **Q6b: Clarifying Ablation**\nThe variant \"Hyper-Gaussian with t\" still involves a probabilistic formulation. Specifically:\n\n- This variant defines a Hyper-Gaussian over $\\xi = (\\Delta \\mathbf{p}, \\Delta \\mathbf{g}, t)$ (line 304).\n\n- At inference, we perform conditioned slicing on the Hyper-Gaussian to obtain time-dependent deformation, following the mechanism in Eq.9–14.\n\n- This variant removes orientation from both the state modeling and the conditioning process, thus relying solely on time for deformation inference.\n\n\n# **Q7: Slight Performance Drops**\nThank you for pointing out this interesting observation. We agree that one might intuitively expect larger degradation with estimated poses, especially in monocular settings where camera motion can be ambiguous.\n\nHowever, we find that this phenomenon is closely tied to the **design of DyCheck benchmark** [7]. DyCheck is specifically constructed to evaluate 4D reconstruction under casual monocular conditions, featuring:\n- **Slow camera motion** to preserve the monocular setting\n- **Fast and complex object motion** for scene-level dynamic challenges\n\nThe major bottleneck on DyCheck is not camera ambiguity, but rather the difficulty of modeling complex scene dynamics. Thus, even when using poses estimated, the performance drop may be relatively slight, as the limiting factor lies elsewhere.\n\n\n---\n[1] Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting.\n\n[2] 4D-Rotor Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes.\n\n[3] N-Dimensional Gaussians for Fitting of High Dimensional Functions.\n\n[4] 6DGS: Enhanced Direction-Aware Gaussian Splatting for Volumetric Rendering.\n\n[5] DepthCrafter: Generating Consistent Long Depth Sequences for Open-World Videos.\n\n[6] SpatialTracker: Tracking Any 2D Pixel in 3D Space.\n\n[7] Monocular Dynamic View Synthesis: A Reality Check."}, {"review_id": "GJEhAQZe6t", "reviewer": "Reviewer_2Vka", "summary": "The paper presents a new method for dynamic Gaussians Splatting reconstruction from monocular RGB videos. The main contribution is in making the orientation of motion as first-class citizen of the method. First, a Global Orientation Field is estimated that captures main forward directions of the motion. Then, Orientation-aware Hyper-Gaussians are defined that can model complex local motion in a robust way. Quantitative results show improvement over previous methods.", "questions": "Overall, I view the paper and the idea novel and original. However, the limited experimental validation prevents me from giving a higher score. Additional evaluation on 2D/3D tracking accuracy could be valuable. Moreover, the evaluation on DyCheck is questionable, and it's not clear if the comparison is fair. The qualitative results are not satisfactory.", "limitations": "Yes, the limitations are discussed in the supplementary files.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths: \n* An idea of basing everything on the motion orientation is indeed novel and interesting.\n* A new type of Gaussian Splats are proposed, called Orientation-aware Hyper-Gaussian.\n* The paper is mostly well-written, including rigorous derivation of new Hyper-Gaussians.\n* The proposed method improves the novel-view synthesis results on the DyCheck dataset. \n* Ablation study supports some design choices.\n\nWeaknesses:\n* The supplementary video is hard to explore as the folder contains many subfolders and many different videos. It makes it almost impossible to compare different methods. Instead, the authors should have prepared a single video with side-by-side comparisons. Regarding the reconstruction quality in the video: it seems quite low-quality, and the qualitative results are underwhelming. \n* Only novel view synthesis evaluation is presented. Why wasn't the 2D/3D tracking accuracy evaluated (as in Shape-of-Motion)? Dynamic mask segmentation could have also been evaluated.\n* In Table 1, were all the methods run by the authors of this paper, or were the numbers just copied from the papers? This is very important, because some papers use different subsets of the DyCheck dataset (e.g. Shape-of-Motion doesn't use all the scenes in Table 1), so if some numbers were just copied, the comparison is invalid. And given that the results in Table 1 for SoM do not include per-scene scores, most likely the number were copied.", "was_revised": true, "final_justification": "I appreciate additional experiments on 2D point tracking. It would still be beneficial for the paper to include evaluation on 3D point tracking and dynamic mask segmentation tasks. I still view the paper as a borderline one, but I'm willing to raise my score to Borderline Accept.", "rebuttal": "# **W1: Supplementary Video**\nWe sincerely thank the reviewer for the suggestion on organizing the supplementary materials! We realize that the current folder-based structure may hinder efficient comparison. We will restructure the supplementary materials to showcase side-by-side comparisons in the revised version.\n\nRegarding the perceived reconstruction quality, we would like to provide further context to better frame our results:\n\n\n**(1) Challenge of Monocular 4D Reconstruction.** 4D reconstruction from casual monocular videos is highly ill-posed and under-constrained. This challenge is particularly acute when rendering novel views under spatiotemporal configurations that differ **significantly** from training views, such as rotating the camera at frozen time or introducing aggressive viewpoint movements, as demonstrated in our videos. These scenarios require models to generalize beyond observed motion paths and to complete occluded or unobserved geometry, often with minimal depth cues. While current methods [1,2,3,4] may produce visually appealing results under **mild** view changes, they typically struggle with such out-of-distribution (OOD) conditions, leading to collapse in visual quality.\n\nWe believe it is important to evaluate reconstruction methods not only in easy settings but also under these more extreme cases to distinguish the advancement. Therefore, rather than claiming perfection in absolute quality, our goal is to push the boundary of what is achievable under casual, monocular, and unconstrained capture.\n\n**(2) Comprehensive Evaluation and Method Robustness.** Our supplementary video provides comprehensive qualitative results, including standard training views, rotating camera at frozen time, and moving camera across video. These three protocols range from in-distribution to highly challenging OOD conditions. In all cases, our method demonstrates clear advantages over existing SOTA approaches. This robustness is further suggested by qualitative superiority in side-by-side comparisons (Figure 3) and quantitative gains on DyCheck (Table 1).\n\n**(3) Peer Validation.** We would also like to highlight that multiple reviewers acknowledged our method's superiority. For instance, Reviewers p9KW highlighted \"results are outperforming all other methods on DyCheck\", Reviewer QZBh observed \"the proposed method surpasses existing baselines\", and Reviewer 1JMD appreciated that \"the method is straightforward and elegant with good results\". We believe these remarks reflect a broader recognition of our contributions.\n\n\n# **W2: More Evaluation**\nWe thank the reviewer for the helpful suggestion.\n### **a. Point Tracking**\nWe have extended our experiments to include dense 2D point tracking on the DyCheck benchmark, using the provided PCK-T@5% metric (percentage of correctly tracked keypoints at a 5% interval), which is adopted in recent works [1,4]. The results are shown below:\n\n| Methods             | PCK-T ↑ |\n|---------------------|---------|\n| Nerfies             | 0.400   |\n| HyperNeRF           | 0.453   |\n| Dyn. Gauss.         | 0.079   |\n| 4D Gauss.           | 0.073   |\n| CoTracker           | 0.803   |\n| Gauss. Marbles      | 0.806   |\n| BootsTAPIR          | 0.779   |\n| MoSca               | 0.824   |\n| **Ours**            | **0.851**  |\n\nOur OriGS significantly outperforms prior approaches in point tracking, further validating the strength of our orientation-anchored representation in capturing complex motion patterns.\n\n### **b. Dynamic Mask Segmentation**\nTo the best of our knowledge, this is not a standard evaluation protocol in the literature for dynamic scene reconstruction [1,2,3,4]. Most recent methods focus on view synthesis but do not incorporate semantic mask evaluation. Nevertheless, we agree that evaluating segmentation is an interesting direction, and our OriGS could potentially benefit such tasks.\n\n\n# **W3: Validity of Table 1 Comparisons**\nWe thank the reviewer for pointing out the potential ambiguity regarding Table 1. We would like to clarify that the comparisons presented are **fair and valid**.\n\n**(1)** All methods in Table 1 were evaluated over the same scene set, following recent work like MoSca [4]. For consistency, we adopted the same evaluation protocol as in MoSca and **included the average result of Shape-of-Motion (SoM) as reported in the MoSca paper.**\n\n**(2)** While we do not report per-scene results for SoM (since they do not release preprocessed data for the DyCheck scenes we use), the reported average result is consistent and directly comparable, given that it **comes from the exact same subset** used across all methods.\n\nWe will clarify this point in the main paper to ensure transparency. We hope this explanation resolves the reviewer's concern about the validity of the comparison.\n\n[1] Dynamic Gaussian Marbles for Novel View Synthesis of Casual Monocular Videos.\n\n[2] Shape of Motion: 4D Reconstruction from a Single Video.\n\n[3] MoDGS: Dynamic Gaussian Splatting from Casually-captured Monocular Videos with Depth Priors.\n\n[4] MoSca: Dynamic Gaussian Fusion from Casual Videos via 4D Motion Scaffolds."}, {"review_id": "b2dXPFdETK", "reviewer": "Reviewer_QZBh", "summary": "This paper proposes a framework, called OriGS, for 4D reconstruction from monocular videos. The key to the proposed method is: 1) a Global Orientation Field, that estimates global forward moving directions, 2) an Orientation-aware Hyper-Gaussian representation, that queries local deformations by time and global orientations.", "questions": "Why are local deformations queried from global anchor orientations only, rather than from global anchor positions as well?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength: \n* Decomposition of motions into global and local components is reasonable.\n* The proposed method surpasses existing baselines in quantitative evaluation.\n\nWeakness:\n* The advantages of the Orientation-aware Hyper-Gaussian representation is not clearly demonstrated. For example, in Fig. 5, I cannot the difference between the results w/ (col 4&5) and w/o (col 3) Hyper-Gaussian. The authors are suggested to highlight the difference in qualitative results and explain why different variants lead to such changes.\n* The presentation of Hyper-Gaussian in Sec 4.2 is difficult to follow. I struggled a lot to figure out the design of this module.", "was_revised": false, "final_justification": "", "rebuttal": "# **W1: Advantages of Orientation-aware Hyper-Gaussian**\nWe appreciate the reviewer's suggestion to highlight the qualitative differences and analyze the behavior of these variants.\n\n### **a. Visual Comparison**\n**(1) Difference.** In the \"Libby\" scene (Fig. 5), the most notable differences among the variants occur in **the running dog**, especially in highly articulated and fast-rotating parts such as the **legs, tail, and head**.\n\n**(2) Analysis on the \"Libby\" scene.** The \"Libby\" scene is particularly challenging due to both **fast motion** and **severe occlusions**. Most of the frame is dominated by large trees and pillars in the foreground, behind which the dog is moving rapidly. This makes the key dynamic regions (dog) subtle and **easily overlooked** in still images visualization in the paper.\n\nThanks for the reviewer's suggestion, we will highlight these regions using **segmentation masks** in the revised figure and clarify the visual differences in the caption.\n\n**(3) Video Demonstration.** We have also included rendered videos of this case in the supplementary materials, where the differences in motion fidelity and shape consistency among the variants become much more apparent. Please refer to \"supp_video/ablation study/libby\".\n\n### **b. Analysis of Variants (Col. 3-5)**\n**(1) Col. 3:** Uses anchors in Global Orientation Field to drive Gaussian deformation. While this captures overall motion direction and recover mainly the **dog's torso**, it lacks local adaptivity and thus fails to reconstruct subtle body parts such as the **legs** and **tail**.\n\n**(2) Col. 4:** Incorporates time as a conditioning variable, allowing temporally-adaptive modeling of motion. This recovers more articulated motion details.\n\n**(3) Col. 5 (full OriGS):** Uses local orientation as an additional condition, enabling the model to adaptively slice the Hyper-Gaussian based on fine-scale changes in motion direction. As a result, **leg and tail movements** involving rapid local directional shifts are reconstructed with much greater fidelity and consistency.\n\n\n# **W2: Presentation of Method Section**\nWe sincerely appreciate your feedback regarding the clarity of Section 4.2. We plan to refine this section from two perspectives:\n\n**(1) Add a brief roadmap at the start of Section 4.2 to outline the logical connections of key components.**\n- **Multivariate Modeling of Dynamic States:** We introduce a probabilistic dynamic state to capture local motion variations.\n\n- **Hyper-Gaussian Representation:** We extend 3D Gaussians into higher-dimensional to encode the probabilistic dynamic state.\n\n- **Conditioned Slicing for Dynamic State Inference:** The probabilistic dynamic state is inferred via conditioned slicing, which obtains deformations.\n\n- **Modulation of Hyper-Gaussian Attributes:** The inferred deformations is then used to modulate Gaussian attributes (position, scale, etc.).\n\n- **Anchor-Guided Deformation and Conditioning:** We explain how the conditioning variables in slicing (e.g., orientation) are obtained through interpolation over the Global Orientation Field.\n\n**(2) Restructuring for better flow.** To prevent overloading, we will move the explanation of anchor-based interpolation (currently starting from line 227) earlier into Section 4.1. This provides a self-contained treatment of the Global Orientation Field, while allowing Section 4.2 to focus entirely on the Hyper-Gaussian module.\n\nWe appreciate this opportunity to improve clarity. In parallel, we are grateful that multiple reviewers found our exposition largely effective. For example:\n- Reviewer 2Vka noted \"the paper is mostly well-written, including rigorous derivation of new Hyper-Gaussians\"\n- Reviewer 1JMD commented \"enjoyed reading the paper in general; well-written and very clear in its exposition; the method is straightforward and elegant\".\n\n\n## **Q1: Local deformations queried by global anchor positions (as well)**\nWe thank the reviewer for this insightful question! While global position may seem intuitively useful, it does not align well with our representation. Our findings are summarized below:\n\n**(1) Slicing in high-dimensional Gaussian favors abstract conditioning.** As formulated in Eq.14, each Hyper-Gaussian is activated most strongly near its learned mean along the conditioning axes. Thus, choosing a conditioning variable that reflects **underlying motion pattern**, rather than **absolute location**, leads to more meaningful and coherent slicing behavior.\n\n**(2) Global anchor position leads to over-localization and reduced generalization.** When using global position as part of the query, the model tends to overfit to spatial anchors. Each primitive becomes tightly \"pinned\" to its specific location, making it harder to generalize deformation behaviors across regions with similar motion patterns.\n\n**(3) Orientation offers more abstract and transferable motion priors, serving as a generalized indicator of local motion direction.** Conditioning on orientation enables the model to capture **shared** deformation modes across space and time, such as consistent leg motion aligned with walking direction, regardless of **absolute position**. As our task involves modeling dynamic deformation fields, orientation is a natural and expressive cue."}, {"review_id": "5NEcnUtqzj", "reviewer": "Reviewer_1JMD", "summary": "This paper presents a novel gaussian splats optimization framework for dynamic 4D scene reconstruction from casually captured videos. This novel framework composes of two key parts: \n- a method to estimate a global orientation field from consecutive frames using its metric depth estimation\n- a probabilistic formulation of gaussian splats that conditions each guassian splat on the local orientation and time, which is able to capture more diverse and complex region-specific deformations through time.", "questions": "Mostly stated above. Among them, I care most about\n- adding some form of visualization of the estimated global orientation field, or slices of it along with the results.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "I really enjoyed reading the paper in general! It's well-written and very clear in its exposition. The method is straightforward and elegant with good results.  \n\nI only find the visualization a bit lacking given the insights the paper focuses on. The paper showed very little visualization of the actual orientation field and the conditioning/slicing mechanism with gaussian splats. Some form of this visualization that's not just abstract didactic ones could be quite nice to add to the paper in addition to the results.\n\nIn section 4.1 titled as Global Orientation Field, it doesn’t really mention how is the field interpolated in places where there is no anchor points estimated. It’s mentioned a bit later in line 241. I think it would really help with the paper flow if it’s explained in 4.1 how is the orientation field interpolated at places with no anchor points, as the gaussian splats can be placed anywhere in space. \n\nOther minor things:\n- line 210: would love to see some quantitative results on the difference between sampling from the distribution v.s using the expected mean ? Is one more expensive than the other?\n- In the ablation study, I’m a bit confused on 3DGS + MLP v.s. Deform w/ GOF. Seems in the first case, there is a shared MLP being optimized, does the second one optimize anything to guide the deformation? Or it’s purely using the inferred gof to update splats? Could you provide more details of these ablated studies in supplement?", "was_revised": true, "final_justification": "I have reviewed authors' responses and keeping my score.", "rebuttal": "We thank the reviewer for the kind comments and for highlighting important aspects to improve clarity and insight.\n# **W1: More Visualization**\nWe appreciate the suggestion of visualizing the orientation field and the Gaussian slicing process, which would be meaningful. In the revised version, we plan to include the following visualizations:\n\n### a. Global Orientation Field\n**(1) Anchor Tracking.** We will track oriented anchors and project them onto 2D to produce motion trajectories over time. Overlaying these trajectories on the RGB video frames offers an intuitive view of how the orientations follow the motion in dynamic scenes.\n\n**(2) Interpolated Orientation.** To visualize the interpolated orientation field, we will densely query orientations using deformed Gaussian positions. We then reduce these orientations to 3D-vector using PCA and apply a color-mapping scheme similar to optical flow to illustrate local directionality and temporal evolution.\n\n### b. Slicing Mechanism\n**(1) 4D Primitive.** Using the Viser tool, we can directly visualize Gaussian primitives in the full 4D space, before and after applying the orientation-conditioned slicing. This helps understand how high-dimensional structures are sliced under different motion conditions.\n\n**(2) Rendering Quality.** We will also provide rendered image comparisons that show how the orientation-aware slicing affects the final visual quality.\n\nUnfortunately, due to **NeurIPS policies this year**, we are **NOT** permitted to include additional figures or visual materials during rebuttal. We sincerely regret this limitation, but we will ensure that the above visualizations are presented in the final supplementary materials.\n\n\n# **W2: Paper Flow Regarding Section 4.1**\nWe sincerely appreciate the reviewer's constructive suggestion to improve the narrative structure of Section 4.1.\n\n**(1) Current Logit.** In the current draft, we **first** introduce the Hyper-Gaussian formulation and **only then** motivate the need to query arbitrary positions within the Global Orientation Field, which **in turn** leads to the introduction of the interpolation mechanism.\n\n**(2) Rethinking.** Upon revisiting this structure in light of the reviewer's comment, we realize that the **interpolation mechanism** is in fact more **foundational**, and not inherently tied to the Hyper-Gaussian slicing process. Rather, it is an **intrinsic property** of the Global Orientation Field, which by design should support querying at arbitrary 3D locations.\n\n**(3) Better Structure.** With this in mind, we agree that it would be more coherent to present the interpolation strategy directly within **Section 4.1**, as part of the core definition of the field. This adjustment will make the exposition more self-contained and improve the overall flow of the method section.\n\n\n# **W3: Sampling from the Distribution**\nWe explored an alternative of our Hyper-Gaussian representation by sampling from the learned distribution, rather than conditioning on its mean.\n\n**(1) Implementation.** To ensure differentiability, we apply the reparameterization trick:\n$$z = \\mu + \\sigma \\cdot \\epsilon, \\epsilon \\sim \\mathcal{N}(\\mathbf{0}, \\mathbf{I}). $$\nThe deformation is then obtained using the sampled latent $z$ instead of the mean. We perform single-sample inference per primitive without Monte Carlo averaging.\n\n**(2) Performance Comparison.** We compare \"sampling from the distribution\" v.s. \"using the expected mean\" using the DyCheck benchmark present in Table 1 in our paper. We observe that the sampling variant results in inferior reconstruction quality compared to mean-based conditioning:\n\n| Methods             | PSNR↑ | SSIM↑ | LPIPS↓ |\n| - | - | - | - |\n| sampling | 18.93 | 0.621 | 0.271 |\n| mean | 19.69 | 0.716 | 0.256 |\n\nWe hypothesize two reasons:\n- **Reason 1. 4D reconstruction from monocular video is fundamentally deterministic.** The learned mean of the Hyper-Gaussian corresponds to the most representative deformation mode, as it aligns with the highest-probability activation slice. Sampling may introduce unnecessary variance that disrupts the coherence of the deformation field and hinders stable optimization.\n\n- **Reason 2. Lack of meaningful uncertainty in monocular settings.** While stochastic sampling introduces diversity, the intrinsic ambiguity of monocular 4D reconstruction still remains. Handling such uncertainty would likely require a generative model capable of synthesizing the missing information from multiple views. Nonetheless, we believe this direction is orthogonal and complementary to our work, which focuses on building a robust 4D representation that could be integrated into such generative frameworks. We hope our contribution will benefit the broader goal of advancing 4D modeling from casual video.\n\n**(3) Efficiency.** From a computational standpoint, using a sampled latent vector incurs negligible overhead compared to using the mean, because both rely on a single forward pass through the same deformation query pipeline and the reparameterization operation itself is lightweight.\n\n\n# **W4: Clarification on the Ablation.**\nThank you for pointing out this potential ambiguity. We will include a more detailed explanation in the supplementary material to improve clarity.\n\n**(1) 3DGS-MLP** introduces a shared MLP that is trained to predict per-primitive deformation.\n\n**(2) Deform w/ GOF**, as described in the main paper (line 300), removes the learnable MLP entirely. Instead, it purely uses the Global Orientation Field to directly drive deformation. Specifically, it adopts the Anchor-Guided Deformation detailed in lines 227–244, which shifts each Gaussian primitive along the oriented anchor trajectory. This variant allows Gaussian primitives to capture global motion trends, but lacks the expressiveness needed to model complex, region-specific dynamics."}]

Author rebuttal:

Meta-review: This paper presents OriGS, a novel framework for high-quality 4D reconstruction that introduces an elegant orientation-anchored representation to model complex scene dynamics from monocular videos. The work received strong support from the reviewers, who were convinced by its originality, superior performance on challenging benchmarks, and the clarity of its formulation. The authors' thorough rebuttal, including new experiments, successfully addressed initial concerns and solidified the consensus for acceptance.
Decision: Accept (poster)