HEIR: Learning Graph-Based Motion Hierarchies
Authors: Cheng Zheng
William Alexander Koch
Baiang Li
Felix Heide
Venue: NeurIPS 2025 poster ()
Keywords: Hierarchical Motion Modeling
Dynamic Scene Understanding
Video Motion Editing
Scene Deformation
Primary area: optimization

Abstract: Hierarchical structures of motion exist across research fields, including computer vision, graphics, and robotics, where complex dynamics typically arise from coordinated interactions among simpler motion components. Existing methods to model such dynamics typically rely on manually-defined or heuristic hierarchies with fixed motion primitives, limiting their generalizability across different tasks. In this work, we propose a general hierarchical motion modeling method that learns structured, interpretable motion relationships directly from data. Our method represents observed motions using graph-based hierarchies, explicitly decomposing global absolute motions into parent-inherited patterns and local motion residuals. We formulate hierarchy inference as a differentiable graph learning problem, where vertices represent elemental motions and directed edges capture learned parent-child dependencies through graph neural networks. We evaluate our hierarchical reconstruction approach on three examples: 1D translational motion, 2D rotational motion, and dynamic 3D scene deformation via Gaussian splatting. Experimental results show that our method reconstructs the intrinsic motion hierarchy in 1D and 2D cases, and produces more realistic and interpretable deformations compared to the baseline on dynamic 3D Gaussian splatting scenes. By providing an adaptable, data-driven hierarchical modeling paradigm, our method offers a formulation applicable to a broad range of motion-centric tasks.

HEIR: Learning Graph-Based Motion Hierarchies
Cheng Zheng'* William Koch ! Princeton University chengzh, william koch, baiang 11, fheide@princceton.com edu
Baiang Li "Tore Robotics
Felix Heide 1.2
Abstract
Hierarchical structures of motion exist across research fields, including computer vision, graphics, and roboties, where complex dynamics typically arise from co- ordinated interactions among simpler motion components. Existing methods to model such dynamics typically rely on conmually-definee or heuristic hierarchies with fixed motion primitives, limiting their generalizability across different tasks. In this work, we propose general hierarchical motion modeling method that learns structured, interpretable motion relationships directly from data. Our method rep- resents observed motions using graph-based hierarchies, explicitly decomposing global absolute motions into parent-inherin We formulate hierarchy inference as a differentiable graph learning problem, where vertices represent elemental motions and directed edges capture learned parent- child dependencies through graph neural networks. We evaluate our hicrarchical reconstruction approach on three examples: 1D translational motion, 2D rotational motion, and dynamic 3D scene deformation via Gaussian splatting. Experimental results show that method reconstructs the intrinsic motion hierarchy in ID and 2D cases, and produces more realistic and interpretable deformations com- pared the baseline dynamic 3D Gaussian splatring scenes. By providing an adaptable, data-driven hierarchical modeling paradigm, our method offers a formulation applicable broad range of motion-centric tasks. Project Page: https://light. http:iigttttpiintoo.t.ooo princeton...o www.icettooe.d........ee edu/HEIR/
patterns and local motion residuals.
Introduction
Many natural and artificial motion systems comprise simple motion primitives that coordinate to produce complex behaviors. Understanding these structured relationships commonly called motion hicrarchics - is fundamental across multiple research areas, from action recognition in computer vision [5] to verifiable prediction in roboties [1, 31]. By capturing multi-scale dependencies, hierarchical models tame the combinatorial explosion when generating, predicting, or controlling motion.
In computer vision, hierarchical representations decompose raw trajectories into semantically mean- ingful layers. Early work introduced fixed hierarchies for activity recognition and motion cap- ture [5, 1, 31], while more recent methods learn multi-level encodings that bridge low-level displace- ments and high-level actions [18, 19]. These approaches identify repeated motion patterns and enable reasoning over sub-actions, improving tasks like video generation and pose estimation. In robotics, hierarchical structures guide both planning and exccution. Global objectives such navigating to a goal - are dccomposed into coordinated limb or joint movements via modular controllers. Hierarchical reinforcement learning architectures separately optimize high-level navigation and local motor policies [10], and modular schemes coordinate limb-specific controllers under whole-bodd objectives [42]. This multi-scale decomposition enhances adaptability and robustness in dynamic environments.
'These authors contributed equally to this work. Listing order is random.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent advances move beyond manually defined templates toward data-driven hierarchy discovery. Hierarchical VAEs introduce multi-level latent spaces that explicitly model coarse global trajectories and fine local adjustments [20]. Diffusion-based frameworks employ semantic graphs to generate motions at successive abstraction levels, offering fine control without hand-crafted priors [13]. These methods aim to learn interpretable, generalizable motion representations directly from data, closing the gap between fixed kinematic models and the rich variability of real-world behaviors.
Despite differences in objectives and formulations, these research areas still face common technical challenges in modeling motion hierarchies. Most approaches represent motion structures either with crad-crafted heuristics or non non- con-iterpreetbll neural modules [21, 32, 37, 18], making them difficult to transfer across tasks. These models also struggle to find the appropriate coarseness, i.e., balancing the granularity of motion primitives against the expressiveness needed for the task. As such, automatically discovering interpretable, transferable representations of motion hierarchies is an open challenge, requiring adaptively selecting the right level of abstraction for diverse downstream applications.
To address these challenges, we propose a ceneral-purposs hierarchical motion modeling method that learns interpretable, structured motion relationships through a graph-based network. We introduce a learnable motion graph, where vertices correspond subsets of discrete motion elements (e.g., Gaus- sian splats, tracked keypoints), and directed edges define learned hierarchy of motion dependencies. To enable this, we learn edge weights on a proximity graph to infer parent-child relationships: the resulting edge weights define the parent probability distribution for any motion element, from which we sample a discrete motion hierarchy. This allows us to model both global constraints and local variations in a unified, data-driven way.
We validate the generality and effectiveness of our method three challenging tasks: two synthetic benchmarks involving structured point motion to show expressiveness (one translational and one rotational), as well as dynamic 3D Gaussian splatting for high-fidelity scene deformation. In all settings, the proposed approach effectively captures and exploits the underlying hierarchical structures. Across representative dynamic scenes, our method achieves improvements over existing methods for all scenarios in scene deformation for pixel error and perceptual metrics, highlighting its effectiveness in producing perceptually and structurally faithful deformations.
2 Related Work
We review relevant work in two areas: motion representations and 3D scene deformation.
Hierarchical Motion Representation and Decomposition. Motion representations capture structure in dynamic systems to support analysis, generation, and control. We existing works have explored explicit or implicit motion models.
Conventional explicit approaches impose predefined structure on motion to improve control, analysis or generation. In humanoid settings, skeletons define natural hierarchical relationships which have been used for various downstream tasks such as SDF learning or motion re-targeting [2]. For video understanding, motion programs [18] represent human behavior sequences of symbolic motion primitives. The FineGym dataset [27] takes another approach and develops a three-layer semantic hierarchy of pre-defined actions, which has been shown to be useful in action recognition [27, 19]. These methods are largely limited to a domain, however, as defining an explicit structure relies on domain-specific assumptions.
On the implicit side, unsupervised methods such as spectral clustering [5] and Moving Poselets [32] learn task- specific motion patterns that improve recognition without relying on predefined semantics. More closely related to our work, in physics- based settings, relationships emerge by learning how entities interact with each other [6, 26]. Similarly, Neural Relational Inference [16] recovers the underlying interaction graph even when the connectivity is unknown a priori.
At the intersection, we find methods that rely explicit structures, but learn the relevant components. MovingParts [38] factorizes dynamic NeRF scenes into rigidly moving segments for part-level editing, and an unsupervised video approach clusters pixels by principal motion axes to animate articulated objects [28]. Both recover meaningful parts but also do not infer parent-child dependencies, leaving the full hierarchy unspecified. Unlike prior work, we do not assume any prior domain, dimensionality or naturally occurring structures in the data. Our method is capable of decomposing the motion on its own at the same time, it provides an interpretable and controllable structure to the motion.
2

---

3D Scene Deformation and Editing, Traditional methods for scene deformation explicitly define geo- metric transformations using mesh-based deformation energies or cage-based constraints. Approaches such as Laplacian coordinates [29, 30] preserve local geometric details by formulating deformation as energy minimization. Cage-based methods [24, 14, 40] encapsulate an object within coarse control mesh, allowing intuitive deformation through sparse user manipulation. A Although effective for explicit geometric editing, these methods typically assume structured mesh representations.
Recent progress in 3D scene deformation and editing has been largely driven by the emergence of neural implicit representations [22]. NeMF [8] proposes continuous spatio-temporal representation to model motion as a continuous function over time, enabling smooth interpolation and editing within neural fields. MovingParts [38] discovers object parts from motion cues in dynamic radiance fields for part-level animation. NeRFShop [11] integrates cage- transformations within NeRFs, facilitating user-driven interactive deformations. Similarly, Wang el al. [34] use coarse mesh guidance to impose semantically controllable deformations on neural representations. These approaches primarily focus on achieving visually coherent edits but neglect the underlying hierarchical motion structure.
A few existing methods are addressing structured motion explicitly. CAMM [17] models mesh dynamics using kinematic chains but limited to occlusion-free settings and mesh-based priors. SC-GS [9] proposes sparse control points for deforming Gaussian splats, yet the neural network- based mapping from control points to Gaussian splats causes entangled deformation effects. Explicit hierarchical modeling of motion in 3D editing remains relatively underexplored, but there are some concurrent efforts. HiMoR [21] employs a manually defined motion tree with pre-set basis to decompose motion, lacking the flexibility to discover scene-dependent hierarchy. MB-GS [43] represents Gaussian splat dynamics using sparse motion graphs with dual quaternion skinning and learnable weight painting. While providing more structured control, its motion structure is still predefined per-object rather than learned from data.
In contrast to these works, our method learns a hierarchical motion representation from observed scene dynamics. By inferring structured, data-driven parent-child relationships between motion elements, our method achieves interpretable, flexible, and consistent scene deformation and editing.
3 Hierarchical Motion Learning
In this section, we introduce the proposed hierarchical motion learning approach. We first define the problem setup Section 3.1, then briefly overview our method in Section 3.2. Next, we describe the details in learning of motion hierarchies based on a proximity graph in Section 3.3, and show it can also be extended to rotational motion in Section 3.4. Following, in Section 3.5, we describe how we apply our method to the deformation 3D Gaussian Splat scenes.
3.1 Problem Setup
We tackle a new problem of hicrarchical motion modeling that decomposes observed absolute motion into parent-inherited motion and residual local motion, structured by learnable directed graph hierarchy. Figure provides a graphical reference to this problem. Given N-dim motion elements with observed positions X E R xd on d-dim over time steps t O.... T. define the absolute velocity or deformation at time step t as the trame-to-fanme difference between consecutive time steps A X - X E R N Our objective is to infer hierarchy matrix H E {0, 1) NNN such that absolute deformations A can be decomposed into a parent- -nhertti motion and relative
motion 8"
A HA' + 5'.
Specifically, Hyg 1 means element j is the parent of clement i. To make the above equation valid, the hierarchy matrix H should satisfy the following: (1) Each element has exactly one parent, meaning each row has only one non-zero entry; (2) The element cannot be the parent of itself, so the diagonal entries are zero; (3) The hierarchy must not contain cycles, meaning there exists no sequence of distinct indices 11, ..,.... ... is (with le > 1), such that H,, Hizis
H,,,,
The hierarchy matrix H here exactly represents a directed acyelic graph (DAG), which consists of vertices and edges with each edge directed from one vertex to another, such that following those directions will never form a closed loop. Note that the hierarchy matrix defined here is an adjacency matrix with binary values [23, 41], where indicates edge between the two vertices and 0 otherwise.
3

---

Encoder (-)
Decoder D(-)
f
Nluttenn socisi
Observed Absolute Motion
Predicted Relative Motion &
Predicted Aosolute Motion &
Bumbei--Stttan
Observed
Leeraalee Edge Weights W Sempled Hiararchy Matrix
Figure Learning Motion Hierarchies. Given a sequence of observed positions X over time (left), we predict absolute motions and candidate graphs based on local spatial proximity. A graph neural network processes this structure to predict edge weights to infer a probabilistic parent-child hierarchy over motion elements (bottom path). The encoder computes the prediction of the relative motion based on these weighted parent candidates. The absolute motion of each motion element then recursively aggregated from its parent using residual composition process (top path) and hicrarchy matrix sampled from the edge weights using Coubell-Softmal We learn the hierarchy by minimizing the difference between the observed and predicted absolute motions across all time steps.
Other adjacency matrices might have values between 0 and represent the weight (or cost) of the edge between two vertices.
3.2 Overview
We introduce a hierarchy motion learning method to tackle the above problem, illustrated in Fig. 1.
We define a proximity directed graph Go Ev) to learn the optimal hierarchy matrix H. The vertices of the graph / ! E IR N x denotes the set of N motion elements. For each vertex i, V as the set its k nearest neighbors in Euclidean space. This neighborhood serves as the parent candidate of the vertex, where (i,j) E Eo if and only A E & The attributes the edge wij represent the possibility that vertex j could serve as
we define neighborhood vertices (
the parent of vertex i. We obtain the predicted value of relative motions 8' via message passing and aggregation in the graph, and reconstruct the full motion Á using H sampled from a normalized distribution of W. The corresponding optimization problem then to find the edge weights W that minimize the difference between the predicted motion and ground truth motion
T-1 Coase (
W arg min W
Σ
(2)
where G (-) denotes the message passing, aggregation, and vertex update in the graph, and D (.)) represents the decoding from relative motion to absolute motion.
3.3 Learning Motion Hierarchies
We next describe our model structure and the training objective. The model contains an encoder module G (-) and a decoder module D (-) that propose to learn the motion hierarchies. We use graph neural network as the encoder that takes the observed absolute motions to predict local dynamics (A : W), and define decoder module reconstruct the global dynamics Á = given a hierarchical relationship H and local dynamics. The hierarchy matrix H is sampled from W the encoder and passed into the decoder for absolute motion extraction. We supervise this model to learn motions faithfully and explain the observed deformations A across all f O...., T 1.

---

Sparse Message Passing (Encoder 9). With the absolute- field A = iA..... A' as vertex features, a single passage-passingg layer operates on the proximity graph Gg. We compute learnable logit for each edge (ij) E & based on the input features A; and A,. Specifically, we use graph attention layer [33] that applies LeakyRel.U
the dot product of a learnable attention vector with the concatenated linear projections of input vertex features, followed by softmax normalization over all the j's that are in the neighborhood of vertex i as
W = softmax (LeakyReILU(a" [WA, (A
where a is the attention vector and W is the weighted matrix for linear transform. The relative velocity of the vertex in Go can then be estimated the difference between the absolute velocity of themselves, and the weighted subtraction of the absolute velocities of their parent candidates, i.e.,
8 A Ej wiy A;-
Sampling Hierarchies. During training, we draw S candidate hierarchy matrices (H E {0, 1) N ) from learned edge weights W. To ensure differentiability in the discrete space, we apply the Gumbel-Softmax trick [12]. For each vertex i, we sample noise lik ~ Gumbel(0, 1) and compute soft parent probabilities
(s)
(3) exp( + 9 JEN(()
/
Σ
(6) exp((wi5 + D6g )/T)
(4)
where 7 > 0 is a temperature annealed during training. For the forward pass, we use the hard (straight- through) variant, setting (1 if j arg max; Wijk and 0 otherwise. At the same time, gradients are back-propagated through the soft weights w(+), At inference, we use the maximun-likelihood hierarchy, with Hij if and only if arg max; Syk.
=(s)
Absolute Motion Reconstruction (Decoder D). Given a hierarchy matrix и E {0, 1J and the predicted relative velocities 8 RR- at time I, we reconstruct the absolute velocities Á by accumulating the relative velocities along the hierarchy. Specifically, the absolute velocity bmaax H' where H' can be expressed as the truncated Neumann series '' Ei H' = -i=0 corresponds ancestor relationships at depth I of the hierarchy, and Lmaa S N is the maximum
depth of the tree. In practice, we also include an early stopping condition 1' < & for some 8 E R in cases where the depth of the tree is lower than Lmax-
We note that hierarchy matrix H satisfying our conditions acyelic and therefore nilpotent, meaning we could write the Neumann series closed-form expression À =
- -1--) However, in practice, rely the truncated series for stability purposes. This series converges at the deepest branch while remaining well-defined even the sampling method occasionally produces cycles.
Training Objective. We learn W by minimizing two E objectives
W arg min EIP(G(A::) - + II (A
(5)
where the first term encourages the reconstructed absolute velocities to match the ground-trute -ruth deformations, and the second term regularizes the magnitude of the relative velocity field, incentivizing the model minimize local velocities and explain motion primarily through parents. Without this regularizer, the model would trivially minimize the reconstruction loss with the solution 8' A f and a hierarchy corresponding to a star topology. bypassing any meaningful hierarchical structure. Here, À is a hyperparameter to balance these objectives.
3.4 Enabling Rotation Inheritance
With minor modifications, our method is also capable of inheriting rotations. The key idea is to modify the encoder G to predict the relative velocity in polar coordinates rather than Cartesian coordinates. For each candidate parent-child pair (i, j). the encoder decomposes motion into radial velocity component ii measuring the rate of change distance |rfy) between nodes, and an angular velocity component 0/ capturing the rate of change in orientation of rly. The relative velocity in
5

---

Cartesian coordinates ij can be easily reconstructed from the polar components. We denote the
aggregated values returned by G as i?. 0j and 8%. For details, we refer the supplemental material. These aggregated predictions are obtained as before, by weighting edge contributions with learned attention scores. Given Lasee from equations 2 and 5, we modify our learning objective as follows:
N (ALA(I)) + = 1 )ol(ā111))
W arg min
Σ t=
Crase (A', (G'
(6)
where Ar. A and XA are hyperparameters for regularizing radial velocity, angular velocity, and connectivity, respectively. The term L (H) is connectivity prior based on the graph Laplacian of the symmetrized hierarchy: let A max(H + HH, 1) and L D A with D = diag(A1). Its second if and only if the graph is connected. We therefore penalize low connectivity with a hinge loss L = relu(T, >2 ()), encouraging well-formed, non-fragmented hierarchies. The graph Laplacian is widely used for analyzing graph connectivity properties.
eigenvalue 2₂ (L),, known as the algebraic connectivity [4], is strictly positive
3.5 3D Scene Deformations and Editing
Next, we descrihe how the proposed method applies to 3D Gaussian Splat scene deformations. This setting presents significant challenge due to the large number of Gaussians involved. Realistic scenes often contain up to hundred thousands of Gaussians, each with spatial and temporal attributes, making scalable and structured deformation non-trivial. We represent motions in a scene using dynamic 3D Gaussian splatting and treat each Gaussian in the scene as a vertex in the graph, At inference time, we deform the 3D scene with learned hierarchy under the as-rigid-as-possible constraints, and use the deformed Gaussians to generate the new scene.
Preliminaries. Gaussian splatting represents a 3D scene using set of 3D Gaussians, where each Gaussian Gj is defined by parameters {リJ Enj; oj, [15]. the 31) covariance matrix, σj is opacity, and ej denotes spherical harmonic coefficients encoding view-dependent colors. The covariance matrix Σ, can be decomposed as Σ; R RJ. where R E (3)3) a rotation matrix parameterized by a quaternion qj. and Sy- diag(s,) E pax3 is diagonal matrix encoding the axis- axis-aligned scaling.
is the 3D center location, Σ, is
To model temporal dynamics, of time f. These include translations 8/15 (t), quaternion-base based rotations (t) and anisotropic scalings (t). The deformation field can be implemented as an implicit neural network, as in 4D-Gaussians [36] and Deformable-Gr [39], or as a composition of shared motion bases and per- Gaussian coefficients, as in Shape-Of-Motion [35] and SC-GS [9]. These models are typically trained via photometric reconstruction losses between rendered outputs and ground-truth video frames.
a deformation field predict per-Gaussian offsets function
Our approach builds on these advances but introduces explicit, learnable motion hierarchy among Gaussians. That means, rather than modeling each motion independently or relying on fixed bases, we infer structured parent-child dependencies that enable interpretable and flexible scene deformation.
Scene Deformation. We infer the deformation of the 3D scene based on user-specified inputs and the learned hierarchy matrix H in Sec. 3.2. We select a Gaussian splat G₁,, and trace the hierarchy matrix to get all its descendants Desc (Gx). These positions are treated as "handles" with constrained positions from user input (either translation or rotation around the center). The deformation for the rest of the Gaussian splats is calculated via an weriiid-a--possibbll (ARAP) solver [30] to preserve local structure. The ARAP solver optimizes the deformed position well as the rotations of the Gaussian splats by minimizing the energy given defined handles
N
E(G') =
Σ W
wall p³) - R (p: - -p))l.
JEN,
where the p; and p are the Gaussian center locations before and after optimization, Here, R/ is the optimized rigid rotation matrix E 80(3), and wi and way are the Gaussian- and edge-dependent weights, which are set to and the cotangent weights according to the original paper. We then apply

---

Node Positions X
Node Velocities A
Volurty A.
Time
Tinee
Reconstructed Hierarchy
Relative Node Velocities 6;
Ground Trufhh
Figure 2: Learning of Hierarchical Relations in 1D Trajectory. We evaluate the proposed hierarchical learning method for a ID motion trajectory where individual nodes are moving hierarchical manner (see Ground Truth motion hierarchy in bottom left inset), but each adding its own unknown motion. Top left to bottom right: (1) raw node positions X; of the hierarchical trajectories over time, (2) absolute node velocities A. (3) reconstructed hierarchy from inferred relationships with ground-trume -ouut hierarchy in the inset, and relative velocities 6, with respect each node parent, given the reconstructed hierarchy (3). We find that the method is able to correctly identify all motions (bottom left) with the two core motions through the orange and green nodes.
the obtained translation p P, and the rotation matrix R to the 3D center location and quaternion of the Gaussian, respectively. By re-rendering the scene with updated Gaussian parameters, we obtain an image of the deformed scene. Note that we learn the hierarchy matrix on downsampled Gaussians and skinning weights apply deformation all Gaussians. We illustrate the details of this process in the supplemental material.
4 Experiments
We first validate the proposed method toy benchmark that contains 1D motions constructed with known hierarchies, as well as a synthetic planetary orbit dataset. Following, we move to evaluation the method for high-dimensional task of 3D Gaussian Splat deformations with complex motion patterns.
4.1 1D Toy Example
To validate the expressiveness of the method, we construct synthetic dataset with known hierarchies. We create minimal point X RR- with N = 11 nodes and E (0..... 7), T 200 frames. Node 0 is fixed at the origin (r) = 0, Vt) and serves as the root. Node follows low-frequency sine motion
x1 = sin(woo + do) + v/.
Ao = 2, wo 10, 7 ~0~(00,0²))
where o = 5x 10 and off ~ (0, ) is perturbation. Node 2 a child of node 1, and adds figger-fr component,
2/ 2 + A sin (wit Φ1) + n/2 A 1, W1 20.
The remaining nodes z; for i 3, ..... 10 inherit the oow-frequency motion from node 1, and with probability of p 0.5 also inherit the high-frequence term from node 2. Their initial positions are drawn uniformly from [-1, and perturbed at each time step by o/ ~ (0, Fig. top-left shows the trajectories of the nodes over time. By construction, we obtain ground-truth hierarchy matrix H shown in the bottom-left inset of Fig. 2 with three layers: the root, a low-frequency layer, and high-freee fregueery layer.
7

---

Ak
Learned Edge Weights W
Reconstructed Hierarchy
Ka -0
M4y
Podtos
Figure 3: Learning of hierarchical relations in a planetary system. We evaluate on a synthetic
dataset with rotational hierarchies, simplified synthetic planetary dataset. From left to right: (I) illustration of the pairwise metrics used for regularization between two timesteps; for clarity, only subset of possible parent-child relations is shown. Solid arrows indicate potential parent-child vectors, with the color corresponding to the parent candidate. (2) Learned edge weights, where entries with green border correspond to correct reconstructions, (3) The observed data shown with the reconstructed hierarchy; we note that the "moons" correctly inherit motion from their "planets".
Evaluation. We train 1,000 independent models for 1,000 epochs each, annealing the Gumbel- Softmax temperature from 1.5 to 0.3. A custom softmax implementation keeps every parent probability strictly positive, avoiding premature collapse.
After training of the model, we validate the correctness. However, as there are many valid hierarchy reconstructions, we cannot simply match against the ground truth. We validate hierarchy matrix H by comparing depth-1 parent-child clusters. As they describe the same motion, the same cluster should be found in the ground-truth tree H*, up to some permutation within the cluster. If this is satisfied for all subgroups, we consider the hierarchy to be valid.
Across all runs obtain 73% success rate, indicating that the method reliably recovers the three- layer structure despite noise. We highlight that this is significant result, as there are 10 potential candidate hierarchies, but only 50 valid ones. There are permutations for the first motion group, for the second as well as potential orderings of the groups.
Method
Accuracy (%)
Proposed
73.0
Monte-Carlo Estimate
5.0 10-7
Table 1: Hieearchy- Accuracy on the 1D Hierarchical Motion Task. Models are trained with Gumbel-Softmax annealing and a custom softmax to ensure stable parent assignment. A hierarchy is counted as correct if all depth-1 motion groups match ground-truth clusters up to permutation. Our method recovers a valid hicrarchy matrix in 73% of cases, whereas a random Monte-Carlo estimate reconstructs valid one << 1%.
This synthetic dataset demonstrates that method is capable of disentangling nested motions and discover the correct parent structure, validating the theory before moving to 3D scene deformations. Please refer to the supplement material for additional details.
4.2 Planetary System
To further test the rotational extension, we construct a synthetic planctary system dataset with 11 nodes and T' = 100 timesteps. Node 0 represents the star (root), with planets and moons attached as its descendants as shown in Fig. 3. In this synthetic dataset, we assume circular motion and do not consider gravitational influences the purpose is to show the expressiveness of the system in capturing rotations.
Evaluation. We correctly reconstruct 100% of hierarchies when there is no noise present in the data, and 73.6% with Gaussian noise (a = 0.05). In both cases, we train 1,000 independent models for 500 epochs each. Validation against the ground truth hierarchy is easier than in the ID toy example,

---

Deformation
Deformation
Ours
SC-GS
Ours
SC-GS
Excavator
Hook
Warreer Jumpingjacks
Figure 4: Qualitative Evaluation of Gaussian Scene Deformation on the D-NeRF [25] dataset. We evaluate the method for hierarchical relationship learning on Gaussian splitting scenes, with thousands of nodes. Specifically, we show scene deformation for the "Fxcavator", "Hook", "Jumppngjacks"" and "Warrior" scenes from the D-NeRF [9] dataset. The arrows show the user-defined deformation on the faded original scene in two different scenarios. We overlay the resulting deformed scenes for the proposed method and SC-GS [9] on the original scene The proposed method produces more realistic and physically coherent deformations, preserving structural rigidity, while SC-GS introduces unnatural distortions and misaligned body geometry.
as there is only a single valid hicrarchy reconstruction. We use A = 12.0 to penalize deviations in distance, As = 0.8 to regularize relative velocity, and XB 0.0 we do not want penalize relative angular velocity here, on the contrary. The Laplacian connectivity prior is enforced with weight AA 6.0. We refer to the supplementary material for additional ablations, in particular the impact of noise.
4.3 Dynamic Gaussian Splatting Scene Deformation
We next validate the method on 3D dynamic Gaussian splatting with experiments on D-NeRF dataset [25], which contains a variety of rigid and non-rigid deformations of various objects. The scenes are usually represented with hundred thousands of Gaussian with spatial and temporal parameters. We compare to SC-GS [9] as the only very recent method capable of tacking this problem. Note that while there are other recent approaches [21, 43, 7, 3] that tackle this problem, code was not available for any of them at the time of this study. Fig. 4 reports qualitative comparisons to SC-GS [9] with two different interactive deformations on four different D-NeRF scenes.
We find that our method achieves more realistic and coherent deformations, effectively preserving meaningful structural relationships and maintaining scene integrity. Specifically, in the Excavator example, SC-GS introduces unnatural bending and distortion on the shovel-body connections and the ground, while our method realistically adjusts the shovel's 's position only to preserve the excavator's rigid geometry. For the Hook example and Warrior example. SC-GS produces exaggerated body distortions, whereas our method maintains a plausible body posture and natural limb alignment. Simi- larly, in the Jumpingiucks example, SC-GS generates physically unrealistic leg and arm deformations, whereas our method produces smooth, physically plausible limb movements consistent with human body and motion constraints.

---

Table 2: Quantitative Evaluation on the D-NeRF [25] dataset. We quantitatively assess our method for the task of scene deformation on dynamic scenes from D-NeRF dataset [25]. We evaluate scene reconstruction metrics of known dynamic poses, like shovel lifting, person punching, and jumping. The proposed method improves existing methods across all scenes on perceptual scene quality and similarity metrics.
Excavator
Hook
Jumpingjacks
Warrior
Ours
SC-GS [9]
Ours
SC-GS [9]
Ours
SC-GS [9]
Ours
SC-GS [9]
PSNR t
21.56
19.91
18.3
15.7
21.54
21.12
16.17
15.29
SSIM t
0.917
0.88
0.93
0.92
0,952
0,944
0,934
0,926
CLIP-I f
0.978
0.978
0.971
0.958
0.975
0.948
0.985
0,965
LPIPS 4
0.0383
0.065
0.0617
0.0954
0.0507
0.0748
0.0567
0.089
Table 2 quantifies the quality of the deformed scene using peak signal-to-noise ratio (PSNR), CLIP image-image similarity (CLIP-I), structural similarity (SSIM), and learned perceptual image patch similarity (LPIPS). Note these are not the reconstruction scores against ground-truth views as in the standard D-NeRF benchmark, we compute these metrics by projecting the deformed 3D scenes into 2D images and comparing them against the original scene under the same camera view. Our method consistently outperforms the baseline across all metrics and scenes, indicating improved perceptual and structural fidelity. See the supplement video for the full deformation process of the scenes, and the supplemental material for additional experimental results and ablation experiments.
Conclusion
We introduce a general- general-purpose purpose method for hierarchical motion modeling that learns structured motion relationships directly from data. By formulating motion decomposition as a differentiable graph learning problem, our method infers interpretable parent-child dependencies and disentangles global and local motion behaviors. We validate the method on 1D) hierarchical motion reconstruction, simplified planetary orbit and dynamic 3D scene deformation of Gaussian splitting scenes. We confirm that our approach compares favorably to existing methods and produces more coherent, realistic, and semantically structured deformations. Unlike prior work that relies on fixed motion bases or heuristic hierarchies, our method adapts flexibly to scene dynamics while maintaining interpretability.
While our method effectively captures hierarchical motion structure from data, it inherits several limitations from the same data. It depends on the presence and observability of motion in the input and cannot infer latent or task-driven semantics that are not reflected in the motion trajectories- for example, it cannot recover the motion of an object part (e.g., truck's 's shovel) if it remains static in the training data. Moreover, the current formulation assumes each motion element has a single parent, which may restrict expressiveness in systems with overlapping or multi-source motion influences.
Limitations:
Despite these limitations, our results suggest that data-driven motion hierarchies offer a promising foundation for structured and generalizable motion modeling. We can strengthen long-range depen- dency detection by replacing k-NN with global-connectin attention layer, dilated-radin neighbours, or small set of randomly initialized long- rangeennee edges. The learned explicit hierarchy also allows us to selectively add local rigidity during deformation to further avoid unwanted deformation artifacts. We hope this work encourages further exploration of learnable hierarchical representations across domains.
variants like sparsely sampled global
Acknowledgments and Disclosure of Funding
We thank Guangyuan Zhao and Congli Wang for their help with the paper writing, and Jan Philipp Schneider for his input and ideas. Felix Heide was supported by an NSF CAREER Award (2047359), a Packard Foundation Fellowship. a Sloan Research Fellowship, Disney Research Award, Sony Young Faculty Award, a Project X Innovation Award, and an Amazon Science Research Award. Cheng Zheng and Felix Heide were supported by Bosch Research Award.
10

---

References
[1] Andrew Benton, Eugen Solowjow, and Prithvi Akella. Verifiable learned behaviors via motion primitive composition: Applications scooping granular media. 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 2549-2555. IEEE, 2024.
[2] Sourav Biswas, Kangxue Yin, Maria Shugrina, Sanja Fidler, and Sameh Khamis. Hierarchical neural implicit pose network for animation and motion retargeting. arXiv preprint m7Xiv22112.00955 2021. 2
[3] Jiahua Dong and Yu-Xiong Wang. 3DGS-Drag: Dragging gaussians for intuitive point-based 3D editing. In Proceedings of the International Conference Learning Representations, 2025.
[4] Miroslav Fiedler. Algebraic connectivity of graphs. Czechoslovak Mathematical Journal, 23(2):298-305,
1973. Publisher: Institute of Mathematics, Academy of Sciences of the Czech Republic. 6
[5] Adrien Gaidon, Zaid Harchaoui, and Cordelia Schmid. Activity representation with motion hierarchies. International Journal of Computer Vixion, 05 2013. 1.2
[6] Artur Grigorev, Bernhard Thomaszewski, Michael Black, Otmar Hilliges. HOOD: Hierarchical graphs for generalized modelling of clothing dynamics. 2023.
[7] Xiao Han, Runze Tian, Yifei Tong, Fenggen Yu, Dingyao Liu, and Yan Zhang. ARAP-GS: Drag-driven berigid-a--possbble 3D gaussian splatting editing with diffusion prior. arXiv preprint ar 1rX1v:2504.17788, 2025. 9
[8] Chengan He, Jun Saito, James Zachary, Holly Rushmeier, and Yi Zhou. NeMF: Neural motion fields for kinematic animation. Advances in Neural Information Processing Systems, 35:4244 2022. 3
[9] Yi-Hua Huang, Yang- Tian Sun, Ziyi Yang. Xiaoyang Lyu. Yan-Pei Cao, and Xiaojuan Qi. SC-GS: Sparse- controlled gaussian splatting for editable dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1220-4230, 2024. 3, 6, 9, 10
[10] Deepali Jain, Atil Iscen, and Ken Caluwaerts. Hierarchical reinforcement learning for quadruped lo- comotion. In 2019 JEFE/RSJ international conference on intelligent robots and systems (IROS), pages 7551-7557. IEFF, 2019.
[11] Clément Jambon, Bernhard Kerbl, Georgios Kopanas, Stavros Diolatzis. Thomas Leimkühher, and George Drettakis. Nerfshop:
Interactive editing of neural radiance fields. Proceedings of the ACM Computer Graphics and Interactive Techniques, 6(1), 2023.
[12] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, 2017. 5
[13] Peng Jin, Yang Wu, Yanbo Fan, Zhongqian Sun, Wei Yang, and Li Yuan. Act as you wish: Fine-grained control of motion diffusion model with hicrarchical semantic graphs. Advances in Neural Information Processing Systems, 16:15597-5 2023. 2
[14] Tao Ju, Qian-Yi Yi Zhou, Michiel Van De Panne, Daniel Cohen-Or, and Ulrich Neumann. Reusable skinning templates using cage-based deformations. ACM Transactions on Graphics (ToG), 27(5):1-10, 2008. 3
[15] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for
real-time radiance field rendering. ACM Trans. Graph., +2(4):139-1, 2023. 6
[16] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In Proceedings of the International Conference on Machine Learning. pages 16888-6697 PMLR, 10-15 Jul 2018. 2
[17] Tianshu Kuai, Akash Karthikeyan, Yash Kant, Ashkan Mirzaci, and Igor Gilitschenski. Camm: Building category-agnostic and animatable 3D models from monocular videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6587-6597, 2023.
[18] Sumith Kulal, Jiayuan Mao, Alex Aiken, and Jiajun Wu. Hierarchical motion understanding via motion programs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 6568-6576, 2021. 1,2
[19] Mei Chee Leong, Hui Li Tan, Haosong Zhang, Liyuan Li, Feng Lin, and Joo Hwee Lim. Joint learning on the hierarchy representation for fine grained human action recognition. In 2021 IEEE International Conference on Image Processing (ICIP), pages 1059-106 1063. IEEE, 2021. 1,
11

---

[20] Jiaman Li. Ruben Villegas, Duygu Ceylan, Jimei Yang. Zhengfei Kuang, Hao Li, and Yajie Zhao. Task- generic hierarchical human motion prior using VAEs. In 2021 International Conference 3D Vision (3DV), pages 771-781, IEEE, 2021.
[21] Yiming Liang, Tianhan Xu, Yuta Kikuchi. Himor: Monocular deformable gaussian reconstruction with hierarchical motion representation. arXiv preprint arXiv:2504.062 10, 2025. 2, 3,
[22] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020. 3
[23] MARK EJ Newman, Networks: An introduction, 2010. 3
[24] Jesús R Nicto and Antonio Susín. Cage based deformations: A survey. In Deformation Models: Tracking.
Animation and Applications, pages 75-99. Springer, 2012. 3
[25] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer, D-nerf: Neural radiance fields for dynamic seenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318 10318-10327, 2021. 9, 10
[26] Alvaro Sanchez- Gonzalez. Jonathan Godwin, Tobias Pfaff, Rex Ying. Jure Leskovec, and Peter W. Battaglia. Learning simulate complex physics with graph networks. In International Conference on Machine Learning, 2020. 2
[27] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. FineGym: A hierarchical video dataset for fine-grained action understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2616-2625 2020. 2
[28] Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion represen- tations for articulated animation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13653-13662, 2021.
[29] Olga Sorkine. Laplacian mesh processing. Eurographics (State of the Art Reports), 4(4):1, 2005, 3
[30] Olga Sorkine and Marc Alexa. As-rigid-as as- assossiile surface modeling. In Symposium on Geometry processing, volume 4, pages 109-116 116. Citeseer, 2007. 3,6
[31] Wataru Takano, Hirotaka Imagawa, and Yoshihiko Nakamura. Prediction of human behaviors in the future through symbolic inference. In IEEE International Conference on Roboticx and Automation, pages 1970-1975. IEEE, 2011.
[32] Lingling Tao and René Vidal. Moving poselets: A discriminative and interpretable skeletal motion representation for action recognition. In Proceedings of the IEEE International Conference Computer Vision Workshops, pages 61 69, 2015.
[33] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.1090:: 2017.
[34] Can Wang, Mingming Hc, Menglei Chai, Dongdong Chen, and Jing Liao. Mesh-guided neural implicit field editing, arXiv preprint arXiv:2112.02157 2023. 3
[35] Qianqian Wang, Vickie Ye, Hang Gao. Jake Austin, Zhenggi Li, and Angjoo Kanazawa. Shape of motion: 4D reconstruction from single video. arXiv preprint arXiv:2407.13764 2024. 6
[36] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4D gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 203 1030002020, 2024. 6
[37] Jingwei Xu, Huazhe Xu, Binghing Ni, Xiaokang Yang, Xiaolong Wang, and Trevor Darrell. Hierarchical style-based networks for motion synthesis. In Proceedings of the European Conference on Computer Vision, pages 178- 194. Springer, 2020. 2
[38] Kaizhi Yang, Xiaoshuai Zhang, Zhiao Huang. Xuejin Chen, Zexiang Xu, and Hao Su. MovingParts: Motion-based 3D part discovery in dynamic radiance field. arXiv preprint arX1::2303.05703 2023. 2, 3
[39] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3D gaussians for high-fidelig monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20331-2034, 2024. 6
12

---

[40] Wang Yifan, Noam Aigerman, Vladimir G Kim, Siddhartha Chaudhuri, and Olga Sorkine- Sorkine-Homung Homunn. Neural cages for feetailprrsseving 3D deformations. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 75-83, 2020.
[41] Zhitao Ying. Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchical graph representation learning with differentiable pooling, Advances Neural Information Processing Systems, 31, 2018. 3
[42] Kai Yuan, Noor Sajid, Karl Friston, and Zhibin Hicrarchical generative modelling autonomous robots. Nature Machine Intelligence, 5(12):1402 1414, 2023.
[43] Xinyu Zhang. Haonan Chang. Yuhan Liu, Abdeslam Boularias. Motion blender gaussian splatting for dynamic reconstruction. arXiv preprint (rxxiv:503.0904: 2025. 3,9
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outline the problem of hierarchical motion modeling across domains and introduce the proposed graph-based learning method. The paper delivers its stated contributions by formalizing the hierarchy learning problem, presenting the proposed method, and validating it on ID and 3D motion modeling tasks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well hy the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the method are explicitly discussed in the conclusion section, noting challenges such as dependence on observed motion data, and the assumption of single parent hierarchy per motion element.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g., independence assumptions, noiseless settings, model well- soll-specification
asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would
The authors should reflect the scope of the claims made, c.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how
they scale with dataset size.
If a'applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should that individual actions in favor of transparency play an important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
their best judgment and recognize
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper is primarily algorithmic. While it formalizes the problem and provides mathematical formulations of the model and training procedures, it does not present new theoretical proof's requiring validation.
14

---

Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The method section specifies key model components, equations and settings for both ID motion and 3D Gaussian splatting tasks. The training and implementation details are discussed in the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making paper reproducible important, regardless whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution a new model (c.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide closed-source models, it may be that access to the model is limited some way (e.g.. to registered users). but it should be possible for other researchers to have some path to reproducing verifying the results.
reprodueibility. In the case of
5. Open access to data and code
Question: Does the paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: The code and data will be released upon publication. Supplemental material includes detailed information datasets, model configurations, and instructions for running the experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines httt::://niss.ccppuutt ouiider//ooeeuubmiiessoooooo... for more details.
15

---

While we encourage the release of code and data, we understand that this might not be possible. "No" acceptable answer. Papers cannot be rejected simply for not including code, unless this central to the contribution (e.g.. for a new open-soure benchmark).
The instructions should contain the exact command and environment needed the results. See the NeurlPS code and data submission guidelines (httpe: : /wwps/ips...oe.o cc cc/public/ Stles///oooeSuuiiiiiiooooo
reproduce
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should releasc anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental selting/details
Question: Does the paper specify all training and test details (e-g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The supplementary material list training parameters (e.g., learning rates, optimizers, epochs), data sources, and evaluation procedures.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: In the ID motion hicrarchy reconstruction task, we conduct 1000 independent trials and report aggregate accuracy. This provides statistically meaningful evaluation of the model's s robustness and generalization in controlled setting.
Guidelines:
answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error hars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (c.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
16

---

Answer: [Yes]
Justification: The supplemental material specifies hardware (NVIDIA RTX 3090) approximate runtimes for different experiments, which are sufficient to estimate required computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethies
Question: Does the research conducted the paper conform, every respect, with NeurIPS Code of Ethics attps://neures cc/public/Bthi httppp/lce/chhcccuuueeeeeeeeeeee cowuidelinessssseeoeeeeeeeeee
Answer: [Yes]
Justification: The research does not involve human subjects, sensitive data, or applications with clear misuse risks. The contributions focus on improving motion modeling in computer vision and graphics, with no evident ethical violations.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Codc of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: While the primary focus is technical, the conclusion briefly discusses the method's applicability and limitations.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate point out that an improvement the quality generative models could used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release models, providing defenses in addition to attacks, mechanisms monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
17

---

Answer: [NA]
The method and datascts used do not pose high misuse risks. The models are task- specifcc and do not involve generative models with sensitive outputs.
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards allow for controlled use of model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites all external methods (c.g., s--rigid-a-posssible datasets (D-NeRF),, and baseline models used in experiments. References to all these included.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or datasct.
The authors should state which version of the asset is used and, if possible, include URL..
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
assets released, the license, copyright information, and terms of the package should be provided. For popular datasets, paperswi thcode. com/ /ataasta has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
We introduce new motion hierarchy learning method and a set of reproducible synthetic motion scenarios for evaluation. The paper and the supplemental material provide detailed documen- tation of the algorithmic pipeline, model design, data generation process, and implementation setup. Code relcase will include usage instructions and configuration files to support external use.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
18

---

Justification:
The paper does not involve any human subject experiments crowdsourcing compo- nents.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country institution) were obtained?
Question:
Answer: [NA]
Justification: No human subject research conducted, and IRB review necessary.
Guidelincs:
The answer NA that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it important, original, non-standard component
core methods this research? Note that if the LLM used only for writing, editing. or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs are not used in the method or experiments. Any use of language models was only for text editing and not relevant to the scientific content.
Guidelines:
The answer NA means that the method development in this rescarch does not involve LLMs any important, original, or non-standard components.
Please refer LLM policy (ht http://///pppppppp//////a/e/ee// should or should described.
for what
19


Reviews:
[{"review_id": "xT38OPTzED", "reviewer": "Reviewer_r1v9", "summary": "The authors present a hierarchical motion modelling method that learns motion structures directly from data. This approach decomposes global movements into parent-inherited patterns and local residual motions using a graph-based hierarchy. They treat hierarchy inference as a differentiable graph learning problem, where vertices indicate motions and directed edges capture parent-child relationships through graph neural networks.", "questions": "How many points can your method work with? Would that limit the method's performance?\n\n**Note**\nIn line 232, the first $w_{ij}$ should be $w_i$.", "limitations": "- The method relies on the presence and observability of motion in the input and cannot infer occluded parts or static object parts in the training data.\n- The method appears to be restricted to handling only a limited number of graph vertices because of its computational complexity, which may lead to poorer performance compared to other dynamic approaches.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strength**\n- The method learns motion structures without relying on hand-crafted heuristics. \n- The method can accommodate different graph vertices (tracked points, Gaussian splats or keypoints).\n\n**Weaknesses**\n- The result appears to be weak; many papers on dynamic NeRF and GS report significantly higher PSNR and SSIM on the D-NeRF dataset, achieving results better than (PSNR: 32.668, SSIM: 0.971, LPIPS: 0.041). They are also absent from the comparison and discussion in the related work. How do these methods differ from the current approach? And why should we choose the proposed methods when others produce better results?\n\n- The authors stated, \"... we propose a general-purpose hierarchical motion modelling method that learns interpretable, structured motion relationships through a graph-based network.\" However, the manuscript lacks a demonstration of interpretability.", "was_revised": true, "final_justification": "The authors addressed my point regarding low PSNR/SSIM values and clarified that the goal is to learn an editable, already reconstructed dynamic scene. For completeness, they shared their PSNR/SSIM scores (PSNR 40.72, SSIM 0.998, and LPIPS 0.021). Regarding the lack of editability, the authors mentioned they will add an illustrative figure visualising the learned hierarchy with nodes overlaid. Therefore, I raise my score.", "rebuttal": "We thank Reviewer r1v9 for the thoughtful and constructive feedback. In the following, we clarify that the reported metrics differ from standard benchmarks due to the nature of our editing task, and include standard reconstruction scores to facilitate direct comparison. We also clarify the interpretability of our learned motion hierarchy, now supported with visualization together with expanded explanation and an additional application. Finally, we address concerns around occlusion handling and graph scalability by detailing how our method observes the full 3D scenes and leverages a coarse-to-fine motion representation. We will also correct the typo the reviewer kindly pointed out.\n\n**[W1: Low PSNR/SSIM values]**\n\n- We thank the reviewer for pointing this out. Dynamic-scene reconstruction methods aim to reconstruct the views or 3D model of a scene within the observable timeframe, whereas **our goal is to learn an editable, interpretable motion hierarchy, given an already reconstructed dynamic scene**. Accordingly, the PSNR and SSIM numbers in Table 2 are **not** reconstruction scores against ground-truth views (as in the standard D-NeRF benchmark). Instead, they are computed between the manually deformed scene and the original non-deformed one (explained in line 293-294). Because our edits deliberately introduce pixel-level warps, full-reference metrics that assume perfect spatial alignment inevitably drop; we keep them as a measurement to confirm that no gross photometric artifacts appear.\n\n- To assess quality in a way that is robust to geometric displacement, we report **LPIPS** and **CLIP-I** because both compare deep feature embeddings that remain stable under moderate warps. They reflect perceptual and semantic fidelity, respectively. **For additional reference-free evaluation we now also provide FID** (Frechet Inception Distance) scores of **68.65 (our method)** v.s. 107.03 (SC-GS). Note that this number is relatively high compared to values reported in generative models as the D-NeRF subset contains few images.\n\n- For completeness, we report the reconstruction component of SC-GS that we use as input for our method, which achieves **PSNR 40.72**, **SSIM 0.998** and **LPIPS 0.021** before performing any scene deformation with our test-time method. \n\n**[W2: Lack of interpretability]**\n\n- We claim that the hierarchy graph we learned is interpretable because **each node corresponds to an observed element (Gaussian or point), and the parent-child links correspond to semantically meaningful motion groups**. In our toy example, the learned hierarchy recovers this: nodes with similar inherited motion are clustered under common parents. The edge weights reflect the likelihood of parent-child relationships, making the resulting graph an interpretable representation of the motion composition. In the dynamic 3D GS scene, we see the resulting groups of the hierarchy tree coincide with intuitive semantic parts like the shovel of the excavator and the limbs of the person. In the camera-ready version, **we will add one illustrative figure visualizing the learned hierarchy with nodes overlaid** on the excavator scene, showing the learned shovel-body relative motion. We will also expand the accompanying text to make this interpretability evidence explicit. \n\n- **Separately, we demonstrate the interpretability with an additional application**, where we recover motion hierarchies in a synthetic orbital system by correctly identifying which bodies orbit around which. The learned graph reflects the expected structure of the solar system, with moons linked as children to their respective planets, and planets to a star, mirroring simplified orbital dependencies. **We will include a visualization of the recovered hierarchy alongside details of the model adaptations** required in the camera-ready version.\n\n**[L1: Reliance on observability of motion]**\n\n- Our method is a test-time learning approach that, by design, does not rely on priors. We agree with the reviewer that we cannot infer static objects in the training data as we stated in the paper limitation section. However, **we are able to tackle occluded parts** since we first reconstruct the full 3D scene from multiple views, like all the NeRF and GS methods. It gives us a full understanding of all the existing motion components in the scene, even if the object is occluded in some of the views.\n\n**[L2, Q1: Clarification on the limit of the number of points]**\n\n- We agree with the reviewer that we are limited by the graph size so we cannot deal with every Gaussian splat in large-scale scenes. In our experiments, we work with ~5K GSs per scene and use weight-based skinning to apply deformation on the remaining GSs representing the scene. We emphasize that the **hierarchical structures in motion tend to be on a more macroscopic level, which means we are not sacrificing quality** with these interpolations."}, {"review_id": "hSkN7qiAjw", "reviewer": "Reviewer_HD6y", "summary": "The authors propose a method to learn a hiearchy of moving elements, giving the position of each element at each time step. Based on a nearest-neighbour graph, an encoder predicts for each edge whether one element is the parent of the other. A decoder reconstructs the motion field based on the predicted motion hierarchy. Training the system to minimize the overall independent motion is shown to lead to a meaningful hierarchy. The authors demonstrate their method both on a simple toy example and show scalability by application to dynamic gaussian splatting with several 100k motion elements.", "questions": "- How many time points are required for training? Is there a trade-off between compute and performance? Further, it would be helpful to mention the computer requirements in the main paper.\n- What advantage does predicting the parent-relationship using a neural network offer compared to optimizing it directly?", "limitations": "The authors adequately discuss limitations of their model.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper is very well written. The description of the method and experiments is well supported by formulas and figures.\n- The proposed model is well motivated and shows good results in the toy setting as well as the ability to scale to a large, real world setting.\n- The method follows a simple approach and doesn't add unneccessary complexity\n\n**Weaknesses**\nI am unsure about the relevance of this problem setting. The model has to be optimized for each instance, and requires observing all motion patterns in the input data. Further, it is not clear to me how the number of available timepoints in the training data influences the quality of the results.", "was_revised": true, "final_justification": "I raised only minor concerns in my review, which where addressed by the authors in the rebuttal. Further, the other reviews and respective rebuttals did not yield any additional points that justify rejecting the paper in my view. Therefore I'll keep my rating of the paper.", "rebuttal": "We thank Reviewer HD6y for their feedback and constructive questions. In the following, we clarify the relevance of our problem setting by highlighting its generality beyond standard 3D scene applications, supported by an additional planetary orbit reconstruction application. While the model is optimized per scene, this follows the established paradigm in NeRF-based pipelines and enables generalization to moderate unseen deformations. We address the reviewer’s questions by providing new ablations on the influence of numbers of training timepoints, and by detailing the computational and modeling advantages of using a neural network for hierarchy prediction over direct optimization.\n\n**[W1: Unclear problem relevance, additional training data quantity analysis]** \n\n- Our setting targets general hierarchical motion modelling: we learn a directed parent-child graph that decomposes every observed velocity into a motion inherited from its parent plus a local residual. **It operates on generic motion elements** (e.g., Gaussian splats, tracked points) **and makes no assumptions about meshes or object class**. As shown by our toy example, the approach applies equally to other fields, be it planetary orbits or rigid body motions. Existing rigging or dynamic-NeRF editors focus on character skeleton or 3D scenes, and are not able to tackle problems outside their domain. Thus, our method is able to cover a **wider range of practical motion scenarios** and provides relevance for motion analysis across diverse domains. \n\n- We have tested our model on an additional application of planetary movements by expanding the scope to arbitrary affine transformations, with only minor modifications to the model. In this setting, **the method successfully identifies relations in planetary orbits**, i.e., which bodies orbit around which, allowing us to disentangle planets and moons. As we are unable to include new figures here, we will include them in the camera-ready version, alongside details on our synthetic dataset and the training setup.\n\n- Regarding the reviewer’s concern that the **model is optimized per scene**, we note that this is **by design**, and akin to the standard test-time learning phase in NeRF and dynamic-GS pipelines. The method is fully self-supervised, optimized once and then ready to be reused for any subsequent deformations or queries. \n\n- We acknowledge that the method depends on the observability of motion as we stated in the limitation section. Although the hierarchy is learned from the motions observed, it **generalizes to moderate, previously unseen deformations**. However, extreme edits outside the training distribution may yield less predictable semantics, similar to other data-driven methods.\n\n- We appreciate the reviewer for bringing up how the number of available timepoints influence the quality of results. **We have run an ablation in which we train on only a fraction of the timesteps (T = 200 total)** and measure hierarchy-reconstruction accuracy over 100 iterations. Accuracy climbs steadily, reaching roughly 45% accuracy when half the frames are available, and **begins to plateau once approx. 70% of the timeframes are included**. We will add the plot and a brief discussion in the supplementary material. \n\n**[Q1: Additional training data quantity analysis]** \n\n- Please refer to the last paragraph in W1.\n\n**[Q2: Clarification on advantage of using neural networks]** \n\n- The motion hierarchy we learned is a discrete tree over *N* elements, and the number of possible trees grows super-exponentially with *N*. Direct optimization therefore becomes a **hard combinatorial search**: even with greedy agglomerative clustering one must evaluate all combinations of edge scores repeatedly and still risks getting trapped in local optima. \n\n- **If formulated as a minimum cost arborescence problem**, we could find an optimal hierarchy with $O(N^2 \\log(N))$ complexity using the Chu-Liu/Edmonds algorithm. However, this approach is not differentiable (if needed for other downstream applications) and is not fully parallelizable. **Our GNN approach has a better time complexity** of $O(N^2)$, **and furthermore turns this discrete problem into a continuous, differentiable relaxation**: it produces soft parent probabilities that are refined jointly with the motion-decomposition loss. Gradient descent then updates all edges in parallel without the need to enumerate all possible trees."}, {"review_id": "fDVJcvXReX", "reviewer": "Reviewer_3bfi", "summary": "This paper introduces a novel deformation field to manipulate 3D Gaussian objects. Specifically, the authors introduce a general hierarchical motion modeling framework that captures motion through a learned tree-structured dependency among Gaussian primitives. Both qualitative and quantitative results demonstrate that the proposed method outperforms SC-GS.", "questions": "1.It would be beneficial to benchmark against SC-GS on a larger number of scenes and report average performance metrics.\n\n2.Figure 3 is visually confusing. Adopting a clearer visualization that better illustrates the intended object deformations would be beneficial.\n\n3.Additional visualizations would help clarify how the method captures structural hierarchies, e.g., showing different hierarchical levels of Gaussians using different colors or visualizing all nodes in a subtree.\n\n4.The method may struggle to capture long-range dependencies, such as interactions between distant object parts. This limitation should be discussed.\n\n5.Some results still exhibit unwanted deformation artifacts, such as stretching or compression in regions that should remain rigid.", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.The research problem is meaningful: modeling physically-dependent relationships within a scene has broad applications in 3D animation.\n\n2.The motivation is well-grounded: the method learns hierarchical tree-structured dependencies tailored to the scene, and employs GNNs to model it.\n\n3.Experimental results show that the proposed method achieves superior performance compared to SC-GS and is able to preserve structural rigidity generally.\n\nWeaknesses\n1.It would be beneficial to benchmark against SC-GS on a larger number of scenes and report average performance metrics.\n\n2.Figure 3 is visually confusing. Adopting a clearer visualization that better illustrates the intended object deformations would be beneficial.\n\n3.Additional visualizations would help clarify how the method captures structural hierarchies, e.g., showing different hierarchical levels of Gaussians using different colors or visualizing all nodes in a subtree.", "was_revised": true, "final_justification": "The authors' rebuttal addressed most of my concerns. And revisions and discussions will be included in the final draft. So I keep my initial rating.", "rebuttal": "We thank Reviewer 3bfi for their assessment and detailed suggestions. In response, we have extended our benchmarks to include full D-NeRF comparisons with SC-GS and clarified performance gains with averaged metrics. In the following, we commit to improved figures that better illustrate deformation effects, and also expand the discussion of current limitations, including long-range dependencies and deformation artifacts, and outline straightforward extensions to address these in future work.\n\n**[W1, Q1: Additional benchmark scenes]** \n\n- We have extended our evaluation to the full D-NeRF dataset. The **proposed method outperforms SC-GS on average over the full dataset across all metrics**. Specifically, the average metrics of our method versus SC-GS are **PSNR(↑) 20.35/19.30**, **SSIM(↑) 0.9438/0.9366**, **CLIP-I(↑) 0.9635/0.9621**, **LPIPS(↓) 0.0453/0.0659** and **FID(↓) 68.65/107.03**. These metrics will be added to the revised Table 2.\n\n**[W2, Q2: Improvements to Figure 3]** \n\n- We thank the reviewer for this suggestion. For better illustration, **we will overlay the scene before and after deformation and use arrows to indicate the user-defined position and direction of the motion for Figure 3**. We are not allowed to attach the revised figure in the rebuttal but it will be added to the camera-ready version.\n\n**[W3, Q3: Illustration of hierarchy levels]** \n\n- We thank the reviewer for this suggestion. In the revised version, **we will include a figure to show different hierarchical levels of Gaussians** as what we have done in the toy example.\n\n**[Q4: Clarification on long-range dependencies]** \n\n- We agree with the reviewer that our current implementation which builds the initial edges in the graph using k-NN *may miss dependencies between widely separated parts*. We made this choice to keep message-passing efficient, and it empirically yields strong results on the scenes we target. \n- **The method itself, however, is not limited to local edges**: replacing k-NN with a sparsely sampled global attention layer, dilated‐radius neighbours, or a small set of randomly initialized long-range edges would allow information to propagate across distant regions without altering the loss or training procedure. We will add a discussion of this and note that exploring these global-connectivity variants is a straightforward avenue for future work.\n\n**[Q5: Clarification on unwanted deformation artifacts]** \n\n- We thank the reviewer for pointing this out. We do not impose hard rigidity constraints here to allow non-rigid, natural deformations, especially for the characters. We would like to note that **as the first method to capture structured motions without manually designed hierarchies, we already achieved less stretching or compression artifacts than SC-GS**. Moreover, the learned parent-child graph now provides an explicit motion structure, which allows us to selectively add local rigidity. For example, edges whose nodes maintain near-constant relative pose over time could be strictly rigidity-regularized during deformation. This would be an interesting future work for us and we will add this discussion in the revised conclusion section."}, {"review_id": "f6FV2kOcuV", "reviewer": "Reviewer_s1zH", "summary": "The paper presents a method of modeling hierarchical motions, which learns hierarchical motion relationships from data, that can be generalized to any type of motion including 1D data to 3DGS based scenes and objects. The paper's method learns to build a DAG, similar to a skeleton, and decompose the global motion into parent-induced motion and local motion residuals.", "questions": "Although this paper tackles the important problem of building generalizable motion hierarchies, there are several unclear points that can be improved: (1) should include more discussion and comparison with prior works on rigging and learning skeleton/skinning weights (2) analysis on error accumulation analysis as depth increases (3) needs experiments showing how training data quantity affects the learned hierarchy and motion quality. I would first like to listen to other reviewers' opinons and the author response.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The approach tackles the important problem of learning hierarchical motion relationships with broad applicability beyond human subjects to arbitrary objects.\n\n2. The 3D GS experiments demonstrate visually impressive improvements over baseline methods, showing clear performance in motion quality.\n\nWeaknesses\n\n1. The distinction between graph-based skeletal hierarchy modeling and established character rigging techniques (which builds skeleton and skeletal weights from 3D characters) remains unclear. (please see references below) While this method generalizes to arbitrary scenes unlike rigging methods focused on specific forms (humans, quadrupeds), the paper needs discussion and comparison demonstrating why this hierarchical motion approach outperforms existing methods.\n\n2. Local relative motion delta_t inherently contain errors that accumulate through the skeletal graph. Please also include the skeleton graph depths of 3DGS inputs and quantify how much increasing depth leads to accumulated errors.\n\n3. As mentioned by the authors, the learning process is dependent on training data, and semantic hierarchies not included from training data cannot be captured. Ablation studies showing how training data quantity affects hierarchy quality and learned skeleton variations would improve understanding the robustness of the proposed method.\n\n\n(rigging related)\n\nRigNet: Neural Rigging for Articulated Characters, SIGGRAPH 2020\n\nOne Model to Rig Them All: Diverse Skeleton Rigging with UniRig, SIGGRAPH 2025\n\n(the following methods include building skeleton+skinning weights from video inputs, a more challenging setting)\n\nBANMo: Building Animatable 3D Neural Models from Many Casual Videos, CVPR 2022\n\nDreaMo: Articulated 3D Reconstruction From A Single Casual Video, WACV 2024", "was_revised": true, "final_justification": "The authors mostly clarified the concerns in the rebuttal, and explained the difference of the current work compared to other methods that produce similar results. Therefore I raise my rating.", "rebuttal": "## Response\nWe thank Reviewer s1zH for highlighting the relevance of our work and for raising valuable points regarding its distinction from rigging methods. In the following, we clarify the differences in objectives and supervision between our method and existing rigging techniques, and outline how the two can be complementary. We address the concerns on error accumulation through a depth-based analysis and provide new ablation results showing how training data quantity affects the accuracy of hierarchy reconstruction. These additions will be reflected in the revised version.\n\n**[W1, Q1: Comparison with rigging techniques]** \n\nWe thank the reviewer for directing us to rigging techniques, which is an established field very relevant to our method. We will add discussion and corresponding references to the paper to adequately situate the work in the landscape of existing approaches. Below we summarize the key distinctions:\n\n**1.** As the reviewer mentioned, compared to these methods, **we generalize to arbitrary scenes and are not limited to mesh-representation or point-cloud**. To further demonstrate this, we tested the model on an additional application involving arbitrary affine transformations. With minor architecture modifications, we were able to successfully recover motion hierarchies in a planetary system. The model identified parent-child relationships corresponding to orbital structure, correctly disentangling planets from moons. We will include visualizations and further details in the camera-ready version.\n\n**2.** Although **RigNet** and **UniRig** generate a skeleton that has a very similar structure to our hierarchy graph, **RigNet’s BoneNet** and **UniRig’s Skeleton-Tree GPT** are both trained on thousands of meshes with ground-truth joints and skinning weights. Our hierarchy graph is directly learned from the motion from the scene via a self-supervised reconstruction loss; no pre-rigged data is required. \n\n**3.** **BANMo/DreaMo** learn where to place deformable handles (‘neural bones’ in the paper) so the object can be rendered consistently across video frames whereas our method learns how those parts influence each other over time. This fundamental difference in objective makes their skeleton (only available in DreaMo) adjacency-based, and doesn’t yield the parent-child relationships we target.\n\n**4.** **Our goal is to provide a motion-centric hierarchy, which is, in fact, complementary to many rigging techniques**. For example, a supervised rigger could seed an initial topology, while our optimization refines parent-child directions using dynamics. It is especially useful for long-range dependencies that K-NN initialization may miss. We will add these discussions to the revised version of the paper.\n\n**[W2, Q2: Additional error accumulation analysis]** \n\n- We thank the reviewer for highlighting this point and have conducted an explicit depth analysis. As we are unable to add figures here, we describe the results, and will provide an additional figure for the camera-ready version. We measured the normalized absolute velocity error $ (|v_{abs}^{pred}(i) - v_{abs}^{gt}(i)| / |v_{abs}^{gt}(i)|)$ for each node on the excavator scene and aggregated the mean ± std values at each successive hierarchy depth. \n- **The mean value of the error stays bounded within the range of 1-3%**, which indicates that our error doesn’t explode along deeper branches. This is because each hierarchy level is trained under the same end-to-end reconstruction loss, and the local motion term (observed motion = parent-inherited motion + local motion) re-anchors every child to the observed velocity field, correcting any errors induced by its parent.\n\n**[W3, Q3: Additional training data quantity analysis]** \n\n- We appreciate the reviewer for bringing up how the number of available timepoints influence the quality of results and have run an ablation in which we train on only a fraction f of the timesteps (T = 200 total) and measure hierarchy-reconstruction accuracy over 100 iterations. \n- Accuracy climbs steadily, reaching roughly 45% accuracy when half the frames are available, and begins to **plateau once approx. 70% of the timeframes are included**. We will add the plot and a brief discussion in the supplementary material."}]

Author rebuttal:

Meta-review: This paper introduces a general-purpose method for learning interpretable, hierarchical motion from dynamic scenes, representing motion as a graph-based structure that is learned at test-time. All reviewers recognized the significance of the problem and the novelty of the proposed solution. The method uses Heuristic-Free Design, it's general and applies to various motion representations. The results are strong. There were some concerns that were very well addressed in the rebuttal. Hence the decision is to accept the manuscript! Congrats!
Decision: Accept (poster)