Title: Orientation-anchored Hyper-Gaussian for 4D Reconstruction from Casual Videos
Venue: NeurIPS 2025
Accepted: yes
Keywords: 4D Reconstruction
Abstract: We present Orientation-anchored Gaussian Splatting (OriGS), a novel framework for high-quality 4D reconstruction from casually captured monocular videos.
While recent advances extend 3D Gaussian Splatting to dynamic scenes via various motion anchors, such as graph nodes or spline control points, they often rely on low-rank assumptions and fall short in modeling complex, region-specific deformations inherent to unconstrained dynamics.
OriGS addresses this by introducing a hyperdimensional representation grounded in scene orientation.
We first estimate a Global Orientation Field that propagates principal forward directions across space and time, serving as stable structural guidance for dynamic modeling.
Built upon this, we propose Orientation-aware Hyper-Gaussian, a unified formulation that embeds time, space, geometry, and orientation into a coherent probabilistic state.
This enables inferring region-specific deformation through principled conditioned slicing, adaptively capturing diverse local dynamics in alignment with global motion intent.
Experiments demonstrate the superior reconstruction fidelity of OriGS over mainstream methods in challenging real-world dynamic scenes.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel gaussian splats optimization framework for dynamic 4D scene reconstruction from casually captured videos. This novel framework composes of two key parts: \n- a method to estimate a global orientation field from consecutive frames using its metric depth estimation\n- a probabilistic formulation of gaussian splats that conditions each guassian splat on the local orientation and time, which is able to capture more diverse and complex region-specific deformations through time.\n\nSTRENGTHS AND WEAKNESSES:\nI really enjoyed reading the paper in general! It's well-written and very clear in its exposition. The method is straightforward and elegant with good results.  \n\nI only find the visualization a bit lacking given the insights the paper focuses on. The paper showed very little visualization of the actual orientation field and the conditioning/slicing mechanism with gaussian splats. Some form of this visualization that's not just abstract didactic ones could be quite nice to add to the paper in addition to the results.\n\nIn section 4.1 titled as Global Orientation Field, it doesn’t really mention how is the field interpolated in places where there is no anchor points estimated. It’s mentioned a bit later in line 241. I think it would really help with the paper flow if it’s explained in 4.1 how is the orientation field interpolated at places with no anchor points, as the gaussian splats can be placed anywhere in space. \n\nOther minor things:\n- line 210: would love to see some quantitative results on the difference between sampling from the distribution v.s using the expected mean ? Is one more expensive than the other?\n- In the ablation study, I’m a bit confused on 3DGS + MLP v.s. Deform w/ GOF. Seems in the first case, there is a shared MLP being optimized, does the second one optimize anything to guide the deformation? Or it’s purely using the inferred gof to update splats? Could you provide more details of these ablated studies in supplement?\n\nQUESTIONS:\nMostly stated above. Among them, I care most about\n- adding some form of visualization of the estimated global orientation field, or slices of it along with the results.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-13"}

comments: SUMMARY:
This paper proposes a framework, called OriGS, for 4D reconstruction from monocular videos. The key to the proposed method is: 1) a Global Orientation Field, that estimates global forward moving directions, 2) an Orientation-aware Hyper-Gaussian representation, that queries local deformations by time and global orientations.

STRENGTHS AND WEAKNESSES:
Strength: 
* Decomposition of motions into global and local components is reasonable.
* The proposed method surpasses existing baselines in quantitative evaluation.

Weakness:
* The advantages of the Orientation-aware Hyper-Gaussian representation is not clearly demonstrated. For example, in Fig. 5, I cannot the difference between the results w/ (col 4&5) and w/o (col 3) Hyper-Gaussian. The authors are suggested to highlight the difference in qualitative results and explain why different variants lead to such changes.
* The presentation of Hyper-Gaussian in Sec 4.2 is difficult to follow. I struggled a lot to figure out the design of this module.

QUESTIONS:
Why are local deformations queried from global anchor orientations only, rather than from global anchor positions as well?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 1
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new method for dynamic Gaussians Splatting reconstruction from monocular RGB videos. The main contribution is in making the orientation of motion as first-class citizen of the method. First, a Global Orientation Field is estimated that captures main forward directions of the motion. Then, Orientation-aware Hyper-Gaussians are defined that can model complex local motion in a robust way. Quantitative results show improvement over previous methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n* An idea of basing everything on the motion orientation is indeed novel and interesting.\n* A new type of Gaussian Splats are proposed, called Orientation-aware Hyper-Gaussian.\n* The paper is mostly well-written, including rigorous derivation of new Hyper-Gaussians.\n* The proposed method improves the novel-view synthesis results on the DyCheck dataset. \n* Ablation study supports some design choices.\n\nWeaknesses:\n* The supplementary video is hard to explore as the folder contains many subfolders and many different videos. It makes it almost impossible to compare different methods. Instead, the authors should have prepared a single video with side-by-side comparisons. Regarding the reconstruction quality in the video: it seems quite low-quality, and the qualitative results are underwhelming. \n* Only novel view synthesis evaluation is presented. Why wasn't the 2D/3D tracking accuracy evaluated (as in Shape-of-Motion)? Dynamic mask segmentation could have also been evaluated.\n* In Table 1, were all the methods run by the authors of this paper, or were the numbers just copied from the papers? This is very important, because some papers use different subsets of the DyCheck dataset (e.g. Shape-of-Motion doesn't use all the scenes in Table 1), so if some numbers were just copied, the comparison is invalid. And given that the results in Table 1 for SoM do not include per-scene scores, most likely the number were copied.\n\nQUESTIONS:\nOverall, I view the paper and the idea novel and original. However, the limited experimental validation prevents me from giving a higher score. Additional evaluation on 2D/3D tracking accuracy could be valuable. Moreover, the evaluation on DyCheck is questionable, and it's not clear if the comparison is fair. The qualitative results are not satisfactory.\n\nLIMITATIONS:\nYes, the limitations are discussed in the supplementary files.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis is a paper on 4D reconstruction. The basic concept of the paper is that orientation of scene components changes smoothly due to physics constraints like preservation of the angular momentum.\nTo achieve this the authors introduce the notions of the Global Orientation Field and the Orientation-aware Hyper-Gaussian. \nAn initial dominant orientation is derived from the PCA using the temporal mean of trajectories over an initial window. Propagation happens through Procrustes on the mean compensated trajectories. A state consisting of position and orientation deviations, time, and the field orientation is modeled as a Gaussian distribution. The state components are dependent, the deviations are conditioned in time and field orientation. Deformation is modeled through anchors as in the Embedded Deformation Graph frameworks.\nTrajectory estimation leverages foundation models on depth and tracking. Bundle adjustment is applied to compute camera intrinsics and extrinsics.\nExperiments include evaluation on DAVIS, SORA, VOS, and DyCheck.\n\nSTRENGTHS AND WEAKNESSES:\nS1: The main strength of the paper is in the introduction of a dynamics model based on the smoothness of orientation. \n\nS2: Results are outperforming all other methods on DyCheck.\n\nS3: The ablation is helpful but has to be better explained (see Q below).\n\nI see only few major weaknesses in the paper (the rest will be in questions) and I am willing to increase the score.\n\nW1: The introduction of time in the state is intriguing but there is no prior in the science literature of not relativistic systems. There is no 3D object or other dynamic tracking that includes time and there is no motivation given. Does this result in a time-variant system? How can we have the time in the state when it appears also as a superscipt in the propagation?\n\nW2: The conditioning of the deviation on the dominant orientation (9) is also not explained.\nAt this point the reader wonders if $O^{t'}$ is one per scene or a real field.\nI think that (9-11) could be easily written in a clean Kalman Filter formulation. \n\nW3: Because of the introduction of probabilities the math became very messy. Several covariances appear that are not defined. Is the covariance with respect to $\\Delta O$ based on quaternions? \nHow is a Gaussian defined on the SE(3) manifold? What does a covariance in time mean (is there noise in time?). Pages 5 and 6 need significant mathematical re-work.\n\nVery minor weakness: $VU^T$ is in $O(3)$ it has to be multiplied with diag(1,1,det(UV^T)) to make it SO(3).\n\nQUESTIONS:\nQ1: Why does Procruses guarantee a smoothness? Is it because of the overlap in temporal windows? One would expect that a Kalman Filter would be more effective on that.\n\nQ2: Explain the difference between the anchors in this paper and the scaffolds used in MOSCA [36].\n\nQ3: It is awkard to include time in a state vector (reminds more of Galilean or Lorentz transformations), please explain why.\n\nQ4: It is not clear what the authors mean with hyper-state in line 192, isn't it just the mean of the states?\n\nQ4: Explain the special minus sign in line 209 and eq. (14), what is its exact definition? (the special plus in line 221 is clear because it refers to quaternions).\n\nQ5: Loss function belongs to the main text not the appendix.\n\nQ6: Which exactly foundation models are used for depth and 2D correspondence (many are listed in l. 35 of the appendix). \n\nQ6: Please explain the ablation. The jump from Deform with GOF to Hyper-Gaussian with t does include only the incorporation of t but no probabilistic formulation?\n\nQ7: The performance drops only slightly when bundle adjustment is used to estimate poses rather than the ground truth (behavior observed in MOSCA) as well. This is surprising given that camera motions are quite ambiguous and one would expect that an out of the box method like BARF would introduce errors. \n\nElaborate on the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}