Physically Embodied Gaussian Splatting: A Visually Learnt and Physically Grounded 3D Representation for Robotics

Authors: jad  abou-chakra
krishan  rana
feras  dayoub
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 05:53:47

Abstract: For robots to robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation  -- modelling geometry, physics, and visual observations -- that informs perception, planning, and control algorithms. We propose a novel dual "Gaussian-Particle" representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics system to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through a splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates "visual forces" that correct the particle positions while respecting known physical constraints. By integrating predictive physical modeling with continuous visually-derived corrections, our unified representation reasons about the present and future while synchronizing with reality. We validate our approach on 2D and 3D tracking tasks as well as photometric reconstruction quality. Videos are found at https://embodied-gaussians.github.io/

Main paper: 
 introduction: 
 The real world is governed by many well-understood physical priors -matter cannot occupy the same space, scenes comprise rigid and deformable objects, gravity acts on all objects, robots have known kinematic structures. Incorporating such priors into a world representation can constrain how it evolves over time, enforcing adherence to the laws of physics. However, most representations like pointclouds, images [1] , or latent descriptors [2, 3, 4] cannot explicitly encode and reason over these priors. Consequently, their ability to predict future states lacks critical physical constraints.Particle-based physics simulators [5, 6, 7] elegantly capture the physical priors that are often known in robotic scenarios and enable forward simulation of dynamics. This makes them attractive for modeling the physical world. To use them in a robotics context, we propose a method to initialize particles from RBGD observations and to periodically correct the errors accrued over time using only RGB observations from the real world.To enable continuous state correction through visual feedback, we add a visual aspect to the particles which allows a corrective force to be calculated. Recent work has shown 3D Gaussians [8, 9] are a differentiable, performant, and expressive representation of visual state that can render images from any viewpoint. Our contribution is to couple these Gaussians to the particles. With this dual Gaussian-Particle representation, we can simulate future states using a physics system. We can also predict the visual appearance by rendering the attached Gaussians. Observations are compared to the rendered images to compute a photometric loss which drives the movement of the Gaussians and subsequently generates "visual forces" that correct the positions of the attached particles. Our key contributions are thus: (i) A novel dual Gaussian-Particle representation that captures geometry, physics, and visual appearance in a unified way. (ii) A method of initializing this representation from RGBD data and instance maps. (iii) A real-time method to correct the particle states using visual feedback. An overview of our system is shown in Fig. 1 . 2), the predicted visual state of the world (3), and the corrective "visual" forces (5) being applied as a result of a visual discrepancy between the rendered state and the image from the camera (4). 
preliminaries: 
 Our approach builds upon Position-Based Dynamics [5, 6] (PBD) and Gaussian Splatting [8] . This section provides an overview of these two components separately, while the subsequent section details our contribution that interconnects them for a robotics setting.Particle-Based Physics Simulation PBD is a physics simulation technique well-suited for our robotics application, which requires real-time operation and robust performance. In our formulation, PBD acts on oriented particles where each particle i is defined by its position p i ∈ R 3 , velocity v i ∈ R 3 , orientation q i ∈ S 3 , angular velocity ω i ∈ R 3 , external force f i ∈ R 3 , radius r i ∈ R + , and mass m i ∈ R + . A particle may belong to a shape S j and thus has its resting position pi as an additional attribute.At the core of the PBD framework are the various constraints that govern the behavior of the simulation. These constraints are defined as cost functions that operate on the particle positions, ensuring the simulation adheres to the desired physical properties. The general form of a PBD constraint is a cost function C(p 0 , ..., p n , µ r ) ∈ R + , where µ r is a relaxation factor. The constraints are minimized at each simulation step using a Jacobi solver. The solver iteratively updates the positions of the particles by aggregating the required positional changes ∆p to locally satisfy the constraint. Generally, the change required is given by:EQUATIONIn our system, we use ground, collision, and shape constraints. Their associated ∆p are outlined below:PBD's ground constraint is used to prevent particles from penetrating the ground plane given by (n, d):EQUATIONPBD's collision constraint which operates on a pair of particles i and j is used to model collisions:EQUATIONLastly, to ensure a group of particles belonging to a particular object (either deformable or rigid) maintain their structure throughout the simulation, we use the shape matching algorithm which is explained in detail in [7, 10] . Briefly, the shape matching algorithm requires the computation of the following matrix for each shape S:A S = i∈S 1 5 m i R i +p i pT i -M c S cT S , c S = i∈S m i p i M , cS = i∈S m i pi M , M = i∈S m i (4)where R i ∈ SO(3) is the matrix form the quaternion q i . A S can be decomposed into R S S and thus the changes in particles positions required to maintain structure are given by:EQUATIONHere, k S is the stiffness parameter of the shape. Both rigid and deformable objects can be modelled with shape matching. A rigid object is composed of a single shape. A deformable object, however, is composed of multiple shapes where each shape is composed of a particle and its neighbours.For a sample implementation of PBD and the shape matching algorithm, we refer the reader to [11] .In our work, we build on Warp's [12] PBD implementation and extend it to incorporate the shape matching algorithm. In our experiments, each physics step takes approximately 5 ms to complete. More details on PBD can be found in our supplementary.Gaussian Splatting Gaussian splatting [8] has emerged as a powerful rendering technique that can capture the state of the visual world with a discrete set of 3D Gaussians. Each Gaussian i is parameterized by its positiong i ∈ R 3 , orientation R i ∈ SO(3), scale s i ∈ R 3 , opacity α i ∈ R + , and color c i ∈ R 3 .Given a viewpoint whose transform relative to the world frame is denoted by V ∈ SE(3) and projection function from the 3D world to the view's screenspace is defined by π(x), the color at a pixel coordinate u can be calculated by sorting the Gaussians in increasing order of their viewspace z-coordinate and then using the splatting formula in Eq. (6) .EQUATIONα i (u) = a i e -gi(u) , g i (u) = x T i Σ -1 i x i , x i = u -π(g i ) Σ i = JVΣ i V T J Tis the covariance of the Gaussian i projected into the viewpoint's screenspace where J is the Jacobian of the projection function π(x) andΣ i = R i diag(s 2 i )R T i .For the full details of this process, the reader is referred to [8] .The rendering equation is not limited to color. In our method, we also associate each Gaussian with a segmentation id o i (as is done in [9] ). Note, however, this is only needed for our initialization scheme and not our prediction and corrective steps. Rendering segmentation is done using:EQUATIONSince the splatting process is differentiable, the attributes defining the 3D Gaussians can be learnt to represent a specific scene by minimizing the photometric loss L rgb between a set of groundtruth images and their corresponding splatted renders. Our initialization procedure also makes use of L seg . EQUATION 
method: 
 Our method creates a model of the world that can at realtime rates be (i) forward simulated, (ii) regularized with physical priors, and (iii) corrected through visual observations from 3 cameras. The model comprises two tightly integrated representations: a set of N particles that represent the physical world and are acted upon by a PBD physics system, and M Gaussians that visually depict the world through efficient splatting-based rendering. The key novelty lies in the introduction of "Gaussian-Particle" bonds that synergistically couple these two representations, creating a bridge between the physical and visual aspects of the modeled environment. A bond is a rigid transform that links a Gaussian to its parent particle. In Sec. 3.1, we describe how to initialize the particles, Gaussians, and their interconnecting bonds. In Sec. 3.2, we describe how the simulated representation is kept synchronized with the real world using the observations from the cameras.To initialize the particles, we compute a loose 3D bounding box around each object using the depth data and instance masks. The handful of instance masks required are user-generated with Cutie [13] (a mask labelling tool) in only a few seconds with minimal effort, however this could plausibly be automated using VLMs or SAM [14, 15, 16] . The instance and depth information is only required at initialization. The bounding boxes are filled with evenly spaced spherical Gaussians whose radius matches the smallest relevant geometric feature (4 to 7 mm). Gaussians that do not project into the instance mask are pruned. The Gaussian positions, colors, and opacities are optimized by iteratively solving for collision and ground constraints using the Jacobi solver mentioned in (Sec. 2) and minimizing the photometric and segmentation reconstruction loss (Eq. ( 8)) using Adam [17] . Thus for this stage only, the Gaussians act as though they are also particles. Gaussians with an opacity lower than 0.3 (emperically chosen) are pruned and their size is upper bounded to roughly the size of the particle. Particles are initialized at the locations of the remaining Gaussians. The particles belonging to each object are then connected to each other using shape matching constraints (2) .The user also indicates whether each object is rigid or deformable. We envision that in the future, a VLM could automatically make this determination. These particles represent a collision-free, ground-aligned approximation of the object geometries, filling up the observed shapes. While this approach does not model cavities, this could be addressed in the future by incorporating depth-based losses or by replacing the grid initialization with a more sophisticated 3D initialization scheme.Subsequently, we continue optimizing the Gaussians without imposing collision constraints and allow the scale to change. We also introduce new Gaussians by enabling the densification procedure detailed in [8] . Finally, each Gaussian is parented to the closest particle and its location relative to the particle is stored as a bond. Any Gaussian that is farther from a threshold to a particle is discarded. This reconstruction process as well as its results are visualized in Fig. 2 . A typical scene contains around a thousand particles and ten thousand Gaussians. After the initialization phase, our approach employs a combination of Position-Based Dynamics (PBD) and Gaussian splatting optimization to predict the current state of our representation. Our method can be decomposed into two stages: a prediction stage and a correction stage. These two stages are called sequentially and are illustrated in Fig. 3 .Prediction The PBD physics system acts upon the particles in our system at a rate of 30Hz. The prediction step constitutes setting the particles associated with the robot to the positions calculated by the forward kinematics, running a single PBD physics step that forward projects the particles and resolves physical constraints (as described in Sec. 2), and finally rigidly moving the Gaussians bonded to particles to their new predicted locations. In our system, this takes approximately 5 ms to perform on an NVIDIA 3090, leaving 28 ms for the subsequent correction stage.Correction After the physics prediction, the current state is rendered from known camera viewpoints. The renders are compared to the images received from the cameras and the photometric reconstruction loss is reduced by optimizing the parameters of the Gaussians. Note that only RGB data is required for the optimization. All Gaussians are allowed to learn new colors and opacities at a low learning rate. This allows shadows imposed by the robot and by the objects to be explained by changing the colors of the Gaussians. Only Gaussians attached to objects are allowed to change their positions and orientations. After 5 optimization steps -a number tuned to meet realtime constraints -the desired positions of the Gaussians are stored and all Gaussians are reset to their original position before the optimization started. The difference in the desired positions of the Gaussians and the original positions of the Gaussians is used to compute a force which is imposed on the connected particles. This force is calculated as f i = K p j o j (g j -g 0 j ), where g j and g 0 j are the final and initial positions of the Gaussians connected to particle i, o j is the opacity, and K p is a proportional gain that has to be tuned (We found K p = 60 to be a good value for our experiments). The Gaussians are thus never directly moved by the correction step. Rather, the correction step is used to generate corrective external forces on the particles which are ultimately resolved by the physics system. The resulting movement of the particles, as orchestrated by the physics, causes the Gaussians bonded to them to move. Consequently, the system is never in a physically infeasible state. Moving the Gaussians in this manner ensures that they always remain in place relative to their bonded particles. 
experiments: 
 We rigorously evaluate the performance of our proposed system across several key metrics to determine its efficacy in dynamic object tracking and photometric reconstruction from novel viewpoints.Datasets To evaluate our method, we utilize both a simulated dataset and a real dataset, each comprising a tabletop scene. The simulated dataset (Fig. 4 ) consists of 25 scenarios designed to highlight various dynamic conditions, including single object pushing (5 scenes), multiple object pushing (5 scenes), object pickup (5), object pushover (5) , and pushing a deformable rope (5) . The real dataset (Fig. 5 ) contains 25 scenarios exhibiting similar variations in dynamic conditions. The realworld experiments have Aruco markers attached to the objects. The markers are used to extract groundtruth 2D and 3D trajectories using a combination of Aruco detection, manual labelling, and factor-graph based optimization [18] . For both datasets, we employ 3 cameras to derive the visual forces, while 2 cameras are used for evaluation purposes only. This diverse set of experiments allows us to assess the performance of our method under a range of challenging scenarios, encompassing both rigid and deformable objects, as well as interactions involving single and multiple objects.Baselines A baseline that is realtime, correctable and serves as world model could not be foundtherefore, we compare our method with baselines that can separately track and predict. We note that our main contribution is that we can do both at the same time. Therefore, we compare our method as a 3D tracker against Dynamic 3D Gaussians (D3DGS) [9] , a 2D tracker against Cotracker [19] , and as a forward model against only using a physics simulator [5] without visual forces.Unlike our approach, D3DGS incorporates physical priors directly into the Gaussian optimization process through auxiliary losses. D3DGS cannot be used as world model like our method since it cannot project what may happen to the Gaussians if forces act on the system. It also requires a foreground mask to be provided at each timestep. In [9] , background subtraction is used to acquire those masks. However, we found that under our challenging realtime constraints where only 3 cameras are used -this leads to catastrophic failure in tracking as visualized in Fig. 6 . Therefore, to make it competitive, we augment the baseline (D3DGS*) by giving it groundtruth hand-labelled masks and by forcing the Gaussians associated with the robot to move according to the forward kinematics of the robot rather than through the optimization process. Furthermore, we found D3DGS auxilliary losses make it half as slow as our visual training iteration, but we assume that optimizations can be made to match the speed of ours and thus allow them six training iterations to match our five training iterations and one physics step. For our method, the full parameters are listed in the supplementary.Metrics We evaluate our method on the mean error in the 2D and 3D trajectories of known points. We also evaluate the foreground photometric reconstruction quality (which includes only the objects on the table) from unseen viewpoints. The predicted 3D trajectory of a query point is obtained by tracking the frame of the Gaussian that was closest to that query at the first timestep. This procedure is consistent with the approach used in [9] . 3D trajectories are projected into each camera to obtain the 2D trajectories and their initial points are used to query Cotracker [19] . The trajectory error is calculated as the mean difference between the groundtruth and the predicted trajectories of several points sampled on the objects. Results Fig. 4 and 5 show 5 of the 25 scenarios tested for each of the simulated and real datasets. The 3D tracking error is plotted over time and shows our method robustly tracking the objects in the scene. Tables 1 and 2 summarize our metrics over each scenario. Our method outperforms all baselines on all experiments except the simulated Pickup tasks. The Pickup tasks are highly dynamic environments where our physical priors were significantly different to the physics exhibited in the simulated scene (see videos on website). This highlights an expected weakness of our system where significantly misaligned physical priors can degrade performance. Nevertheless, our system is able to recover and acquire the final state of the pickup task as shown in Fig. 4 . In all other experiments (45/50), physical priors significantly improve tracking performance. Fig. 6 shows qualitative results. 
related work: 
 To the best of our knowledge, we are the first to create a representation consisting of both particles and 3D Gaussians for the purposes of fusing physical and visual priors within a correctable robotics world model. However, the use of a particle-based physics system alongside a visual component other than Gaussians has been at the core of other works [20, 21, 22, 23] . Moreover, regularizing Gaussian splatting with physical priors through means other than a physics framework has also been explored [9] (the baseline used in our experiments).ParticleNeRF [20] uses particles that are acted upon by a physics system and can be rendered from any viewpoint using a Neural Radiance Field [24] formulation. Particles are associated with latent features which can be decoded into a radiance field by a small neural network. While ParticleNeRF introduced the use of a particle-based physics system (PBD) to incorporate physical constraints and deviations from the NeRF's reconstruction loss relative to particle positions, it only utilized collision constraints and did not fully explore the idea of employing various physical priors to regularize the reconstruction. In part, due to the slower NeRF formulation and the requirement of upwards of 10 cameras to constrain the optimization, ParticleNeRF could not be deployed in a real world robotics setting with realtime constraints. In contrast, our work uses the much faster Gaussian splatting to represent the visual world and further makes heavy use of physical priors which allow visual corrections from as little as 3 cameras in the scene.Particles within a PBD framework paired with corrections from observed pointclouds have been used to model soft human tissue within the domain of surgical robotics [21, 22, 23] . Our method is more general, realtime, and uses visual feedback achieved through the fast Gaussian splatting rather than pointclouds and SDFs. Figure 6 : A scene's predicted visual state with different methods. Our method synchronizes with the real state using a combination of physical prediction and visual correction. Physical prediction alone can estimate the state for about 2 seconds but will eventually desynchronize. Visual correction alone (D3DGS) allows the Gaussians to move in physically infeasible ways, creating scenarios where objects split and Gaussians move freely within the object despite auxiliary structural losses.D3DGS [9] adds physical priors to the Gaussian optimization process with the aim of regularizing the movement of the Gaussians. The significant difference with our work is that the physical priors are not strictly enforced by a physics system -rather auxiliary losses are added to the photometric reconstruction loss. Importantly, this means that D3DGS cannot act as a world model because it cannot be used to predict future states. Nevertheless, D3DGS has shown good tracking and reconstruction performance when groundtruth masks are provided or when upwards of 20 well-distributed cameras are observing the scene and the optimization is given ample time to converge. While these requirements are difficult to achieve in a robotics setting, this visually-driven optimization serves as our baseline and ablates the visual component of our system as well as outlines the importance of grounding the optimization with a physics system. 
limitations and conclusions: 
 Our method was demonstrated on a table-top scenario where heavy use of physical priors can be made. The extension to an open-world setting is left to future work. We make an assumption that the predicted state will be close enough to the groundtruth so that visual forces can be effective. This assumption is broken when significant errors in modelling cause the physics system to accrue errors at a faster rate than can be corrected. This can occur when an object is moved very quickly by the robot. Consequently, a method of closing the loop using global visual information is needed -plausibly using a system like [25] . We foresee that a more sophisticated initialization procedure that uses learnt shape priors would be more powerful. Lastly, our method does not alter the physical structure or parameters of the objects once the modelling is done. Therefore, new observations are not used to correct modelling mistakes. A means of online structure correction and system identification would extend this framework.In conclusion, we have presented a hybrid representation consisting of particles and 3D Gaussians that together represent the physical and visual state of the world. In conjunction, they can be used to predict future states and correct the predicted state from observed data. This synergy makes them suitable for use as a world model in future robotic works. The world model can then be used to extract object state for a reinforcement-based or an imitation-based policy or to plan for future actions using model predictive control. for all images and masks I do Adam step 5:L ← L rgb + L seg 6:L.backward() 7:g, a, c = optimizer.step() 8:for k iterations do Jacobi step 9:g = solveCollisionConstraints(g) 10:g = solveGroundConstraints(g) 11: p = g Create particles at Gaussian locations 12: Initialize particle mass and velocities 13: Create particle shape constraints 14: for m iterations do for all images and masks I do p 0 , q 0 ← p i , q i 3: q i ← [ ωi |ωi| sin θ, cos θ]q i 6: for k solver iterations do Resolve constraints 7:p i ← p i + ∆tv i + ∆t 2 mi (f i + gravity) 4: θ ← |wi|∆tfor all particles i do 8:p i ← groundConstraints(i) 9:for all collision pairs i, j do 10:p i ← collisionConstraints(i, j) 11:for all shapes s do 12:for particles i in s do 13: p i , q i ← shapeMatching(i, s) 14: for all particles i do Update velocities 15:v i ← (p i -p 0 )/∆t 16:ω i ← axis(q i q -1 0 ).angle(q i q -1 0 )/∆t Algorithm 3 Visual Forces f k ← f k + K p (g i -g prev i ) 12: g ← g prevReset positionsRobot Initialization The robot's particles are manually fitted to the links using Blender.The link each particle belongs to is stored so that forward kinematics can be used to appropriately change its position. Furthermore, the robot is rendered in Blender from multiple viewpoints, and Gaussians are trained to reconstruct these renders. These Gaussians are then bonded to the closest particle on the robot. The combination of particles, Gaussians, and bonds is inserted at the start of every scenario. The same parameters used for training the static scene are also applied to the robot.Object Initialization For each object, the 3D bounding box is calculated from its pointcloud, which is extracted from the depth and instance masks. The initialization process is described in Algorithm 1. We use n = 80 and m = 250. All particles are initialized to a mass of 0.1 kg with the exception of the real and simulated rope which are set to 0.2 kg and 0.3 kg. The higher the mass of the particle, the less the influence of the visual force. The mass acts as both a physical and a visual inertia. These concepts can be separated in future work if fine-grained tuning is needed. For scenarios involving rope, the corrective forces are less reliable than that of larger bodies because they occupy less pixels in the image and the physical priors are less constraining because of the deformability. We compensate for the increased noise in the corrective forces by increasing the visual inertia. Note that Algorithm 1 is repeated for each object. Future work may choose to build all the objects simultaneously rather than sequentially to reduce the overall duration of the initialization. In the current implementation, object modeling takes approximately 20 to 40 seconds, which we found acceptable given that it is only done once per scenario. 
hz: 
 Figure 3 : Our real-time correction method illustrated in its different steps.The robot is modelled similarly using renders from its known meshes and inserted into each scene. The background elements (a white table in our experiments) are modelled only using Gaussians with the conventional training regime [8] . 
 
 Appendix: Step The Position-Based Dynamics (PBD) physics system is used to predict the locations of the particles and the Gaussians at each timestep. It runs at a fixed frequency of 30 Hz (33.33 ms per step). The physics step is described in Algorithm 2. We use 20 substeps. At each substep, the velocities and forces are integrated, and then the constraints are solved using a Jacobi solver. Four Jacobi iterations are employed to sufficiently solve the physical constraints. After every physics step, the particle velocities are multiplied by 0.9 (an empirically chosen value). This damping contributes to system stability. Supplementary Figure 8 : The 3D tracking performance of our system and its baselines on all scenarios (simulated and real) 


Decision: The following strengths and weaknesses summarize the reviewers' initial comments for this submission.

Strengths:
- The proposed approach is novel and interesting.
- The paper is mostly well written and clear.
- Experimental evaluation of 2D and 3D tracking performance demonstrate better performance than provided baselines.
- An implementation with real-time tracking performance is demonstrated.

Weaknesses:
- The learning aspect of this paper is questionable, as the PBD simulator is not learnable/adapted.
- The paper lacks ablations for understanding design choices.
- The claimed usability as a world model is not substantiated in experiments.

The author response and discussion phase well addressed the reviewer concerns. The approach is mainly dynamic 3D reconstruction and updating a PBD simulator to follow the dynamic reconstruction. The learning aspect of the approach is minor as it only enables future predictions in the observed scene and the approach does not adapt properties of the dynamics models (e.g. physics parameters). Nevertheless, the reviewers find the approach relevant and interesting for the CoRL community, hence, recommending acceptance of the paper.

The revised paper must not violate the page limit for submission.

Reviews: review: title: title: Very interesting work in an important direction
paper_summary: paper_summary: The authors propose an innovative interactive scene representation for robotic manipulation using particles and Gaussian splatting. This approach employs Gaussian splatting to correct particle positions in the Position Based Dynamics (PBD) simulation while integrating other PBD priors to appropriately model robot-object interactions using particles. The method shows promising results for 3D tracking and has numerous potential downstream applications.
main_review: main_review: - The concept of an interactive scene representation is crucial for robotics, particularly in manipulation tasks. The core idea of updating particles using Gaussian splats is both novel and logical. The authors have incorporated extensive physical priors, such as gravity, ground, and collision constraints, enhancing the realism of the simulation.

- While the core methodology is clear, some details and notations are missing or unclear, particularly in the preliminaries section.

-  The visualizations provided are excellent and effectively explain the method. 

- Although a comparison with Dynamic 3D Gaussians has been made, it would be interesting to compare this approach to other classical particle-based representations that attempt to localize object particles using 3D feature matching or similar techniques.

- It would be beneficial for future comparisons if the authors released the tracking data and object meshes.

- This paper has the potential to initiate a new direction for scene representations in robotics.
questions: questions: - How sensitive is the method to the type of shape-matching algorithm used? Different algorithms may impact the system's performance, and insights into this aspect would be valuable.

- The correction step requires clarification. It's not immediately apparent why the Gaussians aren't moved with the particles assigned accordingly instead of the particles moving due to the visual loss. The authors should explain their reasoning behind this choice and its implications.
limitations: limitations: 3
review_summary: review_summary: Tthis research presents an innovative approach to interactive scene representation for robotic manipulation, with the potential to significantly impact the field. There are details and clarifications missing in the current manuscript, especially in the preliminaries section, which the authors should address.
title: title: Questionable contribution
paper_summary: paper_summary: This paper proposed a particle-based physically enabled Gaussian Splatting approach to integrate the predictive physical modeling and visual corrections.
main_review: main_review: This paper proposed a novel approach for achieving the robotic world model. A particle-based physics system to anticipate the future plausible states and a Gaussian Splatting module to leverage visual observations to online correct the predictions. The proposed method is evaluated on 2D and 3D tracking tasks and photometric reconstruction quality.

### Strengths:
* This paper proposed a `Gaussian-Particle` bonds to couple the two representations to enable the visual-driven physical prediction correction, which is interesting and novel to the best of my knowledge.
* The proposed approach leverages the particle-based physics simulation to introduce several constrains to ensure the prediction is physically feasible.
* The proposed method is evaluated on 3D tracking, 2D tracking, and physical simulation tasks. Authors delicately designed several baselines to compare with since there's no direct comparable existing works.
* In the supplementary materials, authors verified the choice of physical constraints by ablation studies.

### Weakness:
* In my understanding, the PBD system is not learnable so the visual forces only corrects the past predictions but will not help future predictions in similar cases. In other words, this system is just trying to sync with the real-world visual perceptions but not actually "learning" to predict it. In the case when we do not have access to visual data (e.g., serving as a simulator or world model), this system will not perform better than the PBD system itself. This is not a "world model" in my opinion, it would not perform better than pure physical simulator if used as "world model".
questions: questions: * Does the number of Gaussians always more than particles? Is there any insights behind it? For a large Gaussian, will it be better to associate multiple particles to it?
limitations: limitations: 2
review_summary: review_summary: If the prediction is not learnable, I strongly doubt the contribution and impact of this work.
title: title: Interesting work, needs more ablations
paper_summary: paper_summary: This paper proposes a representation based on particle-based dynamics (PBD) and Gaussian splatting that is both predictive and can be corrected by visual observations in real-time. A detailed initialisation scheme is proposed to initialise both the particles and Gaussians along with the rigid "bonds" that connect them. Prediction is done via particle-based dynamics, and corrections use a few Gaussian splatting optimisation steps to computed a "visual force" to be used within the PBD to correct the particle positions. Experiments demonstrate better tracking than baselines, and demos/videos demonstrate impressive tracking of rigid and linearly deformable objects (e.g. ropes).
main_review: main_review: Strengths:
- The idea of combining the strengths of Gaussian Splatting and PBD is an interesting combination that allows for predictive capabilities and visual corrections. The proposed method demonstrates this quite well, and quantitative experiments show that these physical predictions can help tracking in scenarios where robots are interacting with rigid/deformable objects. As mentioned in the paper, no baseline exists that is both physically predictive and capable of being corrected via visual feedback, thus the representation's novelty is significant.
- The real-time demo system for tracking is quite impressive. It has been well-engineered. Additionally, the conducted experiments demonstrate better tracking in both 2D and 3D compared to a baseline without visual feedback (which demonstrates the utility of the visual feedback component to the system), and against the D3DGS method.
- The paper is well-written and easy to follow.


Weaknesses:
- While the current ablation demonstrates the utility of the visual feedback component of the system, there are many more components in the system that are not ablated. The main weakness is that the paper lacks ablations to understand many of these design choices. For example, the initialisation step has many details (such as optimising Gaussians as particles then separating them) that are not well-understood, and it's not clear how important this initialisation step is for performance. Readers cannot decouple these many design choices to the overall performance of the tracking. Other examples include: why is the visual feedback distilled into a visual force? Why not just quickly optimise the particle positions based on the Gaussian deformations?
- Minor weakness: as the title of the paper mentions the representation is a world model, it would have been nice to see downstream applications of this representations such as reinforcement learning.
questions: questions: Questions/Comments:
- What are the details of the shape matching constraints?
- For the prediction step (PBD), I assume there is a given joint torque applied to the robot by a user? Can the authors detail this?
- Line 147: What are "features" here? did authors mean the positions/orientations/scale/color?
- While the supplement has some ablations that show results on a single video, quantitative results could be helpful here.
limitations: limitations: 3
review_summary: review_summary: POST-REBUTTAL UPDATE: Really interesting. The work has been scoped more properly, and more ablations in the supplement provide insight into the strength of the initialisation. I do wish there was more insight into the design choices of the system, but I think the contributions are important enough to be shown at CoRL.