Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling

Authors: mingtong  zhang
kaifeng  zhang
yunzhu  li
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 11:59:53

Abstract: Videos of robots interacting with objects encode rich information about the objects' dynamics. However, existing video prediction approaches typically do not explicitly account for the 3D information from videos, such as robot actions and objects' 3D states, limiting their use in real-world robotic applications. In this work, we introduce a framework to learn object dynamics directly from multi-view RGB videos by explicitly considering the robot's action trajectories and their effects on scene dynamics. We utilize the 3D Gaussian representation of 3D Gaussian Splatting (3DGS) to train a particle-based dynamics model using Graph Neural Networks. This model operates on sparse control particles downsampled from the densely tracked 3D Gaussian reconstructions. By learning the neural dynamics model on offline robot interaction data, our method can predict object motions under varying initial configurations and unseen robot actions. The 3D transformations of Gaussians can be interpolated from the motions of control particles, enabling the rendering of predicted future object states and achieving action-conditioned video prediction. The dynamics model can also be applied to model-based planning frameworks for object manipulation tasks. We conduct experiments on various kinds of deformable materials, including ropes, clothes, and stuffed animals, demonstrating our framework's ability to model complex shapes and dynamics. Our project page is available at \url{https://gaussian-gbnd.github.io/}.

Main paper: 
 introduction: 
 Humans naturally grasp the dynamics of objects through observation and interaction, allowing them to intuitively predict how objects will move in response to specific actions [1] . This predictive capability enables humans to plan their behavior to achieve specific goals. In robotics, developing predictive models of object motions is critical for model-based planning [2] [3] [4] . However, learning such models from real data is challenging due to the complex physics involved in robot-object interactions. Previous works have used simulation environments to learn dynamics models where ground truth 3D object states are available [5] [6] [7] . In contrast, real interaction data, such as videos, tend to be difficult to extract 3D information from, thus leading to inefficiencies in model learning.Recent advancements in 3D Gaussian Splatting have introduced a novel approach to 3D reconstruction. 3D Gaussian Splatting uses a collection of 3D Gaussians with optimizable parameters as object particles. A direct extension of this framework is to optimize the temporal motions of 3D Gaussians, leading to dynamic scene fitting. Nevertheless, such reconstruction methods only fit given dynamic scenes but can not predict object motions into the future.In this work, we propose a novel method that combines dynamic 3D reconstruction with dynamics modeling. Our method learns a neural dynamics model from real videos for 3D action-conditioned video prediction and model-based planning. To achieve this, we first follow previous dynamic 3D reconstruction approaches [8] to obtain a particle-based representation for dynamic scenes. We extract dense correspondence from long-horizon robot-object interaction videos, which serve as the training data for a dynamics model based on Graph Neural Networks (GNNs). Operating on a spatial graph of control particles, the model predicts object motions under external actions such as robot interactions. To enable dense motion prediction, we design an interpolation scheme to calculate the transformations of 3D Gaussians from sparse control particles, enabling action-conditioned video prediction. The dynamics model can also be incorporated into a model-based planning pipeline, e.g. model predictive control, for object manipulation tasks.We perform experiments on various objects of deformable materials including ropes, cloths, and toy animals. Results show our method accurately reconstructs 3D Gaussians and maintains coherent correspondences across frames, even with occlusions. The learned dynamics model simulates the physical behaviors of the deformable materials truthfully and generalizes well to unseen actions. Our method outperforms other system identification approaches (such as parameter optimization in physics-based simulators) in motion prediction accuracy and video prediction quality. We also demonstrate the model-based planning performance using our learned model in a range of object manipulation tasks. 
related work: 
 Dense Correspondence from Videos. Dense correspondence is usually extracted from videos using pixel-wise tracking [9] [10] [11] [12] [13] . Such tracking methods are usually formulated as a 2D prediction problem and require large datasets to train. Our approach is more related to another line of work, which focuses on lifting RGB or RGB-D images to 3D and track in the 3D space [14] [15] [16] . Recently, Dynamic 3D Gaussian Splatting approaches [8, 17] reconstruct objects as Gaussian particles and optimize per-sequence tracking using rendering loss. Building on this line of work, our method uses 3D point scans of objects as initialization and extracts correspondence for dynamics model training.A primary incentive for examining point tracking is its potential application to robotics [18, 19] . For instance, RoboTAP [20] demonstrates that pre-trained point tracking models enhance the sample efficiency in visual imitation learning, and ATM [21] shows that predictions of point trajectories provide control guidance for robots. Our method, in contrast, learns neural dynamics on top of particle-based tracking, which generalizes to unseen robot actions and can be naturally integrated with a model-based planning framework.Neural Dynamics Modeling of Real-World Objects. Modeling the dynamics of real-world objects is extremely challenging due to the high complexity of the state space and the variance of physical properties, especially for deformable materials. Using physics-based simulators, previ- Q h a W X m n G o O l g m Z u 6 D Z Z x v n O N E z E = " > A A A B 6 H i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S J 4 K o l I 9 V j 0 4 r E F + w F t K J v t p l 2 7 2 Y T d i V B C f 4 E X D 4 p 4 9 S d 5 8 9 + 4 b X P Q 1 g c D j / d m m J k X J F I Y d N 1 v Z 2 1 9 Y 3 N r u 7 B T 3 N 3 b P z g s H R 2 3 T J x q x p s s l r H u B N R w K R R v o k D J O 4 n m N A o k b w f j u 5 n f f u L a i F g 9 4 C T h f k S H S o S C U b R S A / u l s l t x 5 y C r x M t J G X L U + 6 W v 3 i B m a c Q V M k m N 6 X p u g n 5 G N Q o m + b T Y S w 1 P K B vT I e 9 a q m j E j Z / N D 5 2 S c 6 s M S B h r W w r J X P 0 9 k d H I m E k U 2 M 6 I 4 s g s e z P x P 6 + b Y n j j Z 0 I l K X L F F o v C V B K M y e x r M h C a M 5 Q T S y j T w t 5 K 2 I h q y t B m U 7 Q h e M s v r 5 L W Z c W r V q q N q 3 L t N o + j A K d w B h f g w T X U 4 B 7 q 0 A Q G H J 7 h F d 6 c R + f F e X c + F q 1 r T j 5 z A n / g f P 4 A 4 7 u N A w = = < / l a t e x i t > We first achieve dense 3D tracking of long-horizon robot-object interactions using multi-view videos and Dyn3DGS optimization. We then learn the object dynamics through a graph-based neural network. This approach enables applications such as (i) action-conditioned video prediction using linear blend skinning for motion prediction, and (ii) model-based planning for robotics. ous works have manually specified physical parameters [22] , or performed gradient-free parameter search [23] and gradient-based methods [24] [25] [26] to optimize low-dimensional physical parameters in a continuous domain. However, these methods require accurate perception models to transfer objects into simulatable assets and thus are limited to constrained, manually designed environments. Neural network-based simulators can also learn object dynamics without requiring the analytical model [27] [28] [29] [30] [31] . Especially, graph-based networks can learn the dynamics of various kinds of models such as plasticine [32, 33] , cloth [6, 34] , fluid [5, 35] , etc. Our method builds upon these previous works but reduces the requirement of large-scale offline training data, enabling direct learning from videos, thus potentially eliminating sim-to-real gaps.Video Prediction for Robotics. Autoregressive [36] [37] [38] and diffusion models [39] [40] [41] trained on Internet-scale data have demonstrated significant capabilities in video generation. Actionconditioned video prediction methods initially showed their capability in predicting future video frames conditioned on actions in simple scenarios, such as atari games [42, 43] and object push [44] . Recently, diffusion-based methods for video prediction [45] [46] [47] [48] [49] have created realistic videos of robotic or human manipulations using large-scale data. These models can be applied to generating robot actions by solving robot poses from videos [50] or learning goal-conditioned policies [46, 51] . Compared to these approaches, our model learns action-conditioned video prediction in 3D, thus making it compatible with model-based planning frameworks for various manipulation tasks. 
method: 
 3.1 Preliminary: 3D Gaussian Splatting 3D Gaussian Splatting [52] optimizes a dense set of 3D Gaussians as explicit scene representation. Each Gaussian is defined by a 3D covariance matrix Σ centered at µ. The matrix Σ decomposes into a rotation matrix R and a scale matrix S by Σ = RSS T R T . For image rendering, 3D Gaussians are projected to 2D using viewing transformation W . The covariance matrix in camera coordinates Σ ′ is computed as Σ ′ = JW ΣW T J T , where J is the Jacobian of the affine approximation of the projective transformation.To optimize the 3D Gaussians, they are projected onto the image plane, where each pixel's color C is determined by a weighted blending:EQUATIONwhere c i is the color, and α i is the opacity value of each 3D Gaussian. The loss function is defined between the rendering and the ground truth image, as the L 1 error with a D-SSIM term: L = (1 -λ)L 1 + λL D-SSIM , where we take λ = 0.2 in our experiments.3D Gaussians have been proven effective in modeling continuously changing dynamic scenes. For instance, Dynamic 3D Gaussians (Dyn3DGS) [8] optimizes the spatial transformation of a fixed set of oriented Gaussian kernels over time to fit multiview video sequences. We build upon Dyn3DGS to extract dense correspondence of 3D Gaussians for deforming objects. Our key insight is that maintaining uniform Gaussian attributes improves modeling accuracy and physical consistency. Specifically, we keep the color, opacity, and scale of Gaussians constant during optimization, while allowing position and orientation change. We empirically found that enforcing color-consistent ellipsoids reduces optimization parameters and speeds up training, resulting in improved correspondence quality, especially when the video contains partial occlusion. Additionally, we only retain high-opacity Gaussians during training.In line with Dyn3DGS, we use physics-inspired optimization objectives to guide optimization, ensuring physical plausibility and accurate long-term dense correspondence. The non-rigid physical modeling principles capture natural scene dynamics, enhancing the fidelity and stability of tracking over time. To overcome the challenges of our 4-view configuration, we strengthen the local rigidity and isometry objectives, improving the ability to handle limited observations.To isolate the objects of interest, we use GroundingDINO [53] and Segment Anything [54] models to obtain masks for objects the robot interacts with. This allows us to filter out the interference of background objects, thus enhancing the optimization efficiency.The dense tracking of object particles facilitates action-conditioned dynamics learning. Consider the scenario of an external robotic action applied to an object. The action can be represented as a sequence of end-effector positions A = {a t } 0≤t≤T . From the video sequence of the action, we can fit 3D Gaussians to the video using our modified Dyn3DGS, resulting in a collection of dynamic Gaussians:X = {X t } 0≤t≤T = {µ t i } 1≤i≤N,0≤t≤T, where X t denotes the set of Gaussians at time t, and µ t i denotes the position of a single Gaussian indexed i at time t. The underlying physical dynamics function could then be depicted asEQUATIONfor any timestep t. We approximate this dynamics function with a Graph Neural Network (GNN) architecture [55] , parameterized by θ. To construct the graph, we apply the farthest point sampling algorithm on the Gaussians X to extract a sparse subset of control particles X = { Xt } 0≤t≤T = {μ t i } 1≤i≤n,0≤t≤T , with n particles and pairwise distance threshold d v , and use them as graph vertices. The robot end-effector position a t is also considered as a graph vertex. A bidirectional edge is connected if two vertices' distance is below a threshold d e , resulting in the edge set Ê.The vertex encoder takes as input the motion of each particle μt i -μt-1 i in the previous k time steps (k = 3 in our experiments) and a one-hot vector indicating whether the vertex belongs to the object or the robot end-effector. To model the relationship between the object and the table surface, the distance of every vertex to the surface is also included in the input. The edge encoder takes as input the 3-dimensional position difference of the two vertices and a one-hot vector indicating whether the edge is an object-object relation or an object-robot relation.The graph encodes vertices and edges into latent features using shared vertex and edge encoders, performs message passing for p timesteps (p = 3 in our experiments), and uses a shared decoder to map vertex latent features back to a 3-dimensional output indicating the motion at time t. The entire process can be represented as Xt+1,pred = Xt + f θ ( Xt-k:t , Êt ).(Training. The main loss function to train our dynamics network is the MSE prediction error:EQUATION)where τ is the look-forward horizon size indicating how many steps we recurrently perform future prediction. We take τ = 5 in our experiments to get a balance between prediction stability and computational cost. During training, we randomly sample time steps t from the training sequences to serve as the starting frame.Empirically, we also find adding regularization terms to be helpful in some objects with complex shapes. These include an edge length regularization term penalizing the mean squared difference of edge lengths between subsequent frames, and a rigidity regularization term, applied to objects that are rigid or close to rigid, penalizing the mean squared difference between the predicted motions and a rigid transformation estimated from the predictions [56] .The flexibility of 3D Gaussians allows us to perform motion densification from the sparse subset of particle X to the entire collection of Gaussians X while maintaining the ability to render photorealistic videos. Taking an initial static collection of Gaussians X = {X 0 } and the action sequence A = {a t } 0≤t≤T as input, we first perform recurrent future prediction on the subsampled graph vertices, outputting X0:T . Then, for each timestep t, we perform the following interpolation scheme to derive the center motions and rotations of each Gaussian:Graph Vertex Transformations. In the first step, we need to calculate the 6-DoF transformation for each graph vertex. The outputs of the GNN are per-vertex motions, which directly serve as the 3D translations. For the 3D rotation, for each vertex μt i , we solve for the rotation R t i according to the motion of its neighbors from time t to t + 1:EQUATION)Gaussian Transformations. We adopt Linear Blend Skinning (LBS) [57, 58] to calculate the motions of Gaussian centers. Concretely,EQUATIONwhere w t ib is the weight associated between Gaussian i and graph vertex b; R t b , r t b are the matrix and quaternion representations of vertex b's rotation at time t; T t b is the translation of vertex b, which is directly predicted by our dynamics model. The blending weight between a Gaussian and a vertex is inversely proportional to their 3D distance:EQUATIONthus assigning larger weights to vertices that are spatially closer to the Gaussians. We start from the initial Gaussian centers and rotations at t = 0 and repeatedly calculate the blending weights and updated Gaussian centers and rotations, resulting in dense and smooth Gaussian motions over the entire video sequence. The dynamics network can also be used to perform model-based planning in robotics tasks. Given the multi-view RGB-D image of the object, we apply segmentation and point fusing to get the complete point cloud of the object. We then construct graph vertices and edges with farthest point sampling. Given a target configuration, we use a Model Predictive Control (MPC) [59] framework to plan for robot actions. The optimization objective is defined as the mean squared difference between the target state and the predicted object state if an action is applied. In our experiments, we apply the Model-Predictive Path Integral (MPPI) [60] trajectory optimization algorithm. 
experiment setup: 
 We perform experiments on 3D tracking, action-conditioned video prediction, and model-based planning. To achieve this, we collect a real-world dataset, with a focus on acquiring multiview images synchronized with corresponding actions at each timestep. The dataset consists of 3 types of deformable objects: rope, cloth, and toy animals, totaling 8 object instances. For all objects, we capture RGBD images and record robot actions at 15 FPS. The depth is used to initialize the point cloud for Gaussian representation. Our data collection setup includes four cameras, strategically positioned at the corners of the workspace and oriented to provide a comprehensive top-down perspective. This setup allows us to collect synchronized multiview data necessary for our experiments.We initialize Gaussian tracking with a point cloud from four views at the first timestep and optimize our Dyn3DGS for dense correspondence. To evaluate our method, we compare it with advanced 2D tracking approaches by projecting 4-view results into 3D using our depth data and evaluate the best results. The unmodified Dyn3DGS is considered as a baseline to show our improvements within the 4-view real-world experiment setup. Following prior work [61] , we evaluate median trajectory error (MTE) in millimeters, position accuracy (δ) at thresholds of 2, 4, 8, and 16 millimeters (reporting the average), and survival rate with a threshold of 0.5 meters. For a fair comparison, we also project our 3D tracking results into 2D pixels and evaluate the same metrics, allowing comprehensive assessment in both 2D and 3D contexts.The quantitative results in Tab. 1 demonstrate that our 3D tracking method outperforms all baselines, including SpatialTracker [15] , which uses depth as a 3D prior. The results highlight our method's capability across various scenes and objects. Additionally, the qualitative results in Fig. 3 show that our Dynamic 3D Gaussian tracking provides precise dense correspondence, even under challenging conditions such as occlusions from robot and object deformations. Table 1 : Quantitative Results on Dynamic 3D Gaussian Tracking. We labeled the ground truth for 200 frames per episode in 3D space for 1 or 2 object instances in each category, covering two episodes per object. Our Dyn3DGS-based tracking method outperforms all baselines, including the unmodified Dyn3DGS, in both 2D and 3D metrics. Using dynamic 3D Gaussian reconstruction with tracking, we train the graph-based neural dynamics models with downsampled control particles. These particle motions are used to interpolate dense Gaussian kernel motion and rotations. To evaluate the quality of our dynamics prediction and 3D Gaussian rendering, we assess action-conditioned video prediction performance.Baselines. Synthesizing action-conditioned object motions requires physics priors, and existing text-conditioned video prediction methods are not compatible with taking 3D robot actions as input.In this experiment, we consider 2 physics simulator-based baselines, MPM and FleX.MPM is based on the Material Point Method simulation framework [62, 63] . Our baseline MPM uses the same simulation setting as previous works [22, 26] , while adding support for two types of robot end-effectors: cylindrical pusher and gripper. For each object instance, we set the friction coefficient µ and Young's modulus E (assumed uniform) to be two learnable parameters that are optimized from the training data using the CMA-ES algorithm. FleX is based on the NVIDIA FleX simulator. For all objects, we adopt the soft body simulator in FleX, reconstruct the initial object mesh from 3D Gaussians using alpha-shape mesh reconstruction. Similarly, we optimize the friction coefficient µ and stiffness coefficient s (assumed uniform) of each object instance using CMA-ES.Metrics. To evaluate the 3D video prediction performance, we adopt 3D distance metrics (3D Chamfer Distance, 3D Earth Mover's Distance), 2D segmentation similarity metrics (J &F score), Results. The results are listed in Tab. 2. Across all metrics, our method outperforms both baselines significantly, proving that we learn a more accurate dynamics model and give more realistic video prediction results. Our neural-based dynamics model effectively avoids the sim-to-real gap by learning from real-world data, thus improving the motion prediction accuracy under unseen robot actions and object configurations. This is also supported by the qualitative visualizations of predicted videos, shown in Fig. 4 . The action-conditioned dynamics model enables deployment in real-world robot planning tasks, demonstrating its generalizability to unseen object configurations. We choose rope straightening and toy animal relocating tasks to show our model's ability to manipulate highly deformable objects to target configurations. The quantitative results in Fig. 5 demonstrate that our approach effectively reduces error and achieves high success rates. The computational complexity and inefficiency of MPM and FleX baselines make them impractical to perform real-world planning tasks. 
conclusion and limitation: 
 In this paper, we introduce a novel approach for graph-based neural dynamics modeling from realworld data for 3D action-conditioned video prediction and planning. Our method reconstructs 3D Gaussians with cross-frame correspondences and learns a neural dynamics model that facilitates action-conditioned future prediction. The method outperforms baseline approaches in motion prediction and video prediction accuracy. We also showcase our planning performance for object manipulation tasks, highlighting our framework's effectiveness in real-world robotic applications.Limitation Although our approach learns object dynamics directly from videos, collecting realworld data remains costly, and the perception module may fail when there are large occlusions or textureless objects. In future work, we aim to develop a more efficient data collection pipeline and a more robust perception system to reduce the learning cost and enhance the method's applicability to all kinds of real videos. 
 
 Appendix:  


Decision: Thank you for your submission to CoRL 2024. The reviewers liked many aspects of the paper and felt it was a promising line of research. They specifically pointed out that they liked the direction of combining Gaussian splatting and particle-base dynamics, the well-written paper, the effectiveness of the method, and the real-world learning and experiments.

The reviewers raised concerns about unclear points of novelty, wanting clarification of limitations, concerns about the experiments (e.g., fairness and single-view vs multiview), a few unsupported claims, and more methodological contributions.

The authors' thorough responses to the initial reviews were compelling and appreciated by the reviewers and the AC. While there are lingering concerns (which we encourage the authors to address for camera-ready), they found that the response was sufficient to raise the recommendation.

Reviews: review: title: title: Overall, I think it's a good paper that presents a technically sound approach for training particle-based dynamics models with real-world videos.
paper_summary: paper_summary: The paper proposes a new framework that combines a particle-based dynamics model with Gaussian splatting, enabling the training of dynamics models using unlabeled multi-view (posed) video observations. The authors evaluated the proposed model on real-world videos of a robot arm interacting with deformable objects, demonstrating strong results in 3D point tracking, video prediction, and model-based planning.
main_review: main_review: Strengths
1. I like the overall direction very much. Combining Gaussian Splatting (GS) with a particle-based dynamics model is a natural way to enable dynamics model training without using dense particle trajectory labels, which are hard to obtain in real-world videos. The proposed approach to achieve this goal is technically sound. Additionallly, to my knowledge, this is the first work to present a framework that combines GS and dynamics models in the context of robot manipulation.
2. The paper is easy to follow, although minor edits are needed to correct a few typos and clarify some points.
3. The effectiveness of the proposed method is demonstrated across point tracking, video prediction, and model-based planning.

Weaknesses
1. Novel contributions aren't clearly presented in Section 3.2. I believe most of Sec. 3.2., including equations (1)-(4), is from prior work [17], so I am not convinced that this much space is necessary to present Sec. 3.2. The difference between [17] and this paper in enabling dense point tracking needs to be clarified. For the same reason, Table 1 might be moved to the supplementary material if there wasn't much change between [17] and this paper in dense point tracking, as Table 1 would just show the results of [17] on the new datasets in that case. If there were any design changes between [17] and this paper, I encourage the authors to add ablation results in Table 1 to show any performance difference made by new changes.

2. It would be beneficial to include more clarifications in the paper regarding the following potential limitations:
- I believe the dynamics model cannot be generalized over object categories, meaning the dynamics model needs to be trained for each object category separately. 
- The objects used during training and testing appear to be the same, so it is unclear whether the method can generalize to novel object instances within the same category (i.e., new toy dolls that the dynamics model hasn't seen during training).

3  Typos or missing details.
- In line 100, what is J?
- Inconsistent point neighborhood notations in Eq. (2)-(4), and (8): knn(i,k), knn_{i,k}, N(i)
- what is T^t_b in Eq (9)?
- Mu^t_i instead of Mu^t_j in Eq (9)?
- In line 188, what does "with FPS" mean?
questions: questions: I would like to see answers to the questions I raised in points 1 and 2 of the Weaknesses section above.
limitations: limitations: 2
review_summary: review_summary: Although the paper needs some polishing to fix a few typos and clarify the points I raised in my review, overall, I think it's a good paper with strong results. It successfully demonstrates the use of particle-based dynamics models trained with rendering supervision for robot manipulation. After rebuttal: I thank the authors for all their significant efforts in addressing the reviewers' concerns. I am happy with the answers provided by the authors, so I will keep my original "weak accept" rating.
title: title: Interesting idea but lack of methodological contributions and convincing experiments.
paper_summary: paper_summary: The authors propose a novel approach for learning 3D object dynamics from real-world videos of robot interactions. The method involves generating ground-truth object states with Gaussian Splatting (GS) and learning a Graph Neural Network (GNN) from a set of control points of the labeled data. The pre-trained GNN can then be used to predict 3D object states conditioned on robot actions and render future object states to achieve action-conditioned video prediction.
main_review: main_review: The paper proposes an interesting idea to learn 3D action-conditioned models of objects from real-world videos leveraging recent advances in point-tracking. Moreover, the real-world evaluation is extensive. 

My primary concern relates to the novelty of the proposed dense tracking with Gaussian Splatting (GS). Neither the GS technique nor the physics-inspired regularization losses are a contribution of this paper, as presented also in the original paper [1]. However, the authors claim this to be “their proposed tracking method” (see lines 135-136). Moreover, the authors further claim that their  Dynamic 3D Gaussian tracking provides precise, dense correspondence, even under challenging conditions such as occlusions from the robot and object deformations. However, with multiple views, the robot occlusions are limited, and the deformations shown in the evaluation are not particularly strong. The paper would be much stronger if the method could account for larger deformations, including self-occlusions of the object. 

Regarding the experiments, all the tracking baselines are single-view baselines. Did the author lift these baselines to multi-view? If not, I would suggest at least reporting the results from the best view or providing the results of the proposed method from only one view. Moreover, Dynamic 3D Gaussians[1] is missing.

The authors put some emphasis on the action-conditioned video prediction, claiming that compared to related work on video prediction for robotics, their work “facilitates dynamics learning to support more complex robotic manipulation tasks.”. However, this statement is not supported by experimental results. I wonder why the authors do not compare with other data-driven approaches, such as visual foresight [2], but instead claim that “existing text-conditioned video prediction methods are not compatible with taking 3D robot actions as input.”

My final comment relates to Figure 4. In the rope experiments, I wonder how it is possible that in the second frame of the MPM simulation, the pusher moves the wrong side of the rope (which in the previous frame is far from the starting position of the push). This looks to me like either the wrong frame or an implementation error. 

[1] Luiten, J., Kopanas, G., Leibe, B. and Ramanan, D., 2023. Dynamic 3d Gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.09713.

[2] Finn, C. and Levine, S., 2017, May. Deep visual foresight for planning robot motion. In 2017 IEEE International Conference on Robotics and Automation (ICRA) (pp. 2786-2793). IEEE.

### Clarity
The paper is clear and easy to follow. The only section that I believe requires some revision is related to the Gaussian transformations in section 3.4. Equation (9) should be revised. Should the term $\mu_j^t$ be $\mu_i^t$? What is $T_b^t$? Moreover, it is unclear how the blending weights are calculated. It is mentioned that it is inversely proportional to the 3D distance of the Gaussian i and graph vertex b but more details could be provided in order to ensure replicability. 

### Strengths
* Combining 3D tracking methods to learn object dynamics from real-world objects is a novel application.
* The authors provide extensive real-world experiments.
* The paper is well-written and easy to follow.


### Weaknesses
* I challenge the novelty of the tracking approach proposed in this work.
* The claimed benefits of the Dynamic 3D Gaussian tracking under challenging conditions (occlusions and deformations) are not convincingly supported. The evaluations do not show strong deformations, and multiple views minimize robot occlusions.
* Some relevant baselines are missing in the current evaluation.
* Minor methodological contribution.
questions: questions: * Would the learned model generalize to unseen objects?
* Clarify how the tracking method differs from Dyn3DGS and, if it is different, integrate it as a baseline.
* Clarify how the tracking results with respect to monocular baselines are conducted.
* Clarify Figure 4.
* Support the statement that your method “facilitates dynamics learning to support more complex robotic manipulation tasks.” by comparing it with other learned dynamics models (e.g., Visual Foresight).
* MINOR: Can you expand on the computational complexity of the overall method and the planning? The videos showed that it required 20x, so I wonder where the computational bottleneck is.
* MINOR: Revise equation 9.
* MINOR: Expand on the computation of the blending weights.
* CURIOSITY: How do you label ground truths for real-world evaluation? This is quite a challenging task, especially under deformations. It would be of great value to share the labeling approach.
limitations: limitations: 3
review_summary: review_summary: The proposed idea is interesting and relevant for robotics, but I challenge the real contribution of the work. Moreover, some claims of the authors are not well supported by the experiments.
title: title: Combination of existing parts to an interesting and novel method
paper_summary: paper_summary: The paper focuses on learning action-conditioned dynamics models from short sequences of RGB-D images that capture a robot interacting with a deformable object.  Initially, the authors use these observations for a vision-based 4D reconstruction of the object based on  Dyn3DGS. They then select a subset of the learned 3D Gaussians, which, combined with known robotic actions, are used to train a GNN-based dynamics model. The effectiveness of this method is demonstrated by evaluating the tracking performance of the 4D reconstruction and by assessing the dynamics model for planning and action-conditioned video prediction.
main_review: main_review: ### Strengths

**Novelty and potential**
Training an action-conditioned motion-model using 3D point tracking as supervision is novel and has great application potential for many different robotic applications. 

**Evaluation**
The quantitative and qualitative evaluation is convincing and shows the efficacy of the proposed method. In particular, during the rebuttal, the authors presented further results on more complex scenes showing the performance of the method under the influence of shadows or occlusions.

**Rebuttal**
During the rebuttal phase, the authors were able to address most of my concerns regarding the paper with *extensive* answers and presented an impressive amount of additional results.

**Clarity** 
The paper is overall well written, with easily comprehensible figures which support the general understanding.

### Weaknesses

**Methodological contribution**
While the presented method is overall novel, the novelty of the individual components—tracking and dynamics modelling—is limited, and the primary contribution comes from their combination. For me it seems that especially the tracking part of the method as described in 3.2 is just a slightly modified version of Dyn3DGS, with the key difference that the parameters of the Gaussians (except the mean) are kept constant over time and that the weights of the regularization losses are tuned for the method to function in a sparsely observed environment.

**Straightened start configuration** 
The method shares the limitation of Dyn3DGS, that only parts of the scene can be tracked which are visible in the initial frame. This requires for example cloths to be in a straightened state in order to be properly modelled.

**Weaknesses resolved during rebuttal**
- Lacking a sufficient limitations section.
- Missing qualitative examples on complex scenes which contain shadows and occlusion.
- Model-based planning experiments without baselines. (Still no baselines, but sufficient justification why not)
questions: questions: **Issues**
- Evaluate tracking and video prediction on more complex scenes (shadows, more deformation, ...) *resolved*
- Add the unmodified Dyn3DGS as tracking baseline/ablation *resolved*
- Put the planning results into perspective by comparing to baselines *resolved*
- Extend limitations discussion *resolved*

**Questions**
- Since there is already a dependency on using depth cameras for the initialization of the 3D Gaussians, have the authors also considered using depth to guide the optimization of the tracking? *answered*
- What is used as the ground truth state for the calculation of the tracking metrics? *answered*
- Since the color of the 3D Gaussians is kept constant over time, wouldn't there be issues in the tracking if there were strong dynamic shadows, for example from the robot occluding the light source? *answered*
- How long does it take to train the dense tracking, especially in comparison to the baselines? It would be a strong limitation if the tracking took for example 10min to train, while the baselines were able to yield results after a few seconds. *answered*
- L149: "In the case where an external obstacle (for example, a table surface) is in the scene, the distance and direction of every vertex to the obstacle are also included in the input." Since this is not further explained in the paper: How are obstacles detected/modeled? *answered*
limitations: limitations: 3
review_summary: review_summary: After the rebuttal I changed my recommendation to "weak accept". Although I am convinced of the novelty, potential and evaluation of the method, it has some shortcomings when it comes to a methodological contribution.