Dynamic 3D Gaussian Tracking for Graph-Based Neural Dynamics Modeling
Authors: name: Mingtong Zhang, name: Kaifeng Zhang, name: Yunzhu Li

Videos of robots interacting with objects encode rich information about the
objects' dynamics. However, existing video prediction approaches typically do
not explicitly account for the 3D information from videos, such as robot
actions and objects' 3D states, limiting their use in real-world robotic
applications. In this work, we introduce a framework to learn object dynamics
directly from multi-view RGB videos by explicitly considering the robot's
action trajectories and their effects on scene dynamics. We utilize the 3D
Gaussian representation of 3D Gaussian Splatting (3DGS) to train a
particle-based dynamics model using Graph Neural Networks. This model operates
on sparse control particles downsampled from the densely tracked 3D Gaussian
reconstructions. By learning the neural dynamics model on offline robot
interaction data, our method can predict object motions under varying initial
configurations and unseen robot actions. The 3D transformations of Gaussians
can be interpolated from the motions of control particles, enabling the
rendering of predicted future object states and achieving action-conditioned
video prediction. The dynamics model can also be applied to model-based
planning frameworks for object manipulation tasks. We conduct experiments on
various kinds of deformable materials, including ropes, clothes, and stuffed
animals, demonstrating our framework's ability to model complex shapes and
dynamics. Our project page is available at https://gs-dynamics.github.io.