Physically Embodied Gaussian Splatting: A Visually Learnt and Physically Grounded 3D Representation for Robotics
Authors: Jad Abou-Chakra
Krishan Rana
Feras Dayoub
Niko Suenderhauf
Venue: CoRL 2024 (2024)
Keywords: 3D Representation
Gaussian Splatting
Robotics
Tracking
Physics
TL;DR: We use particles to represent the physical state of a robot's world and 3D Gaussians for its visual state, combining them to enable forward modeling and realtime visual-based correction from images coming from 3 cameras.

Abstract: For robots to robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation  -- modelling geometry, physics, and visual observations -- that informs perception, planning, and control algorithms. We propose a novel dual "Gaussian-Particle" representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics system to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through a splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates "visual forces" that correct the particle positions while respecting known physical constraints. By integrating predictive physical modeling with continuous visually-derived corrections, our unified representation reasons about the present and future while synchronizing with reality. We validate our approach on 2D and 3D tracking tasks as well as photometric reconstruction quality. Videos are found at https://embodied-gaussians.github.io/

Physically Embodied Gaussian Splatting: A Visually Learnt and Physically Grounded 3D Representation for Robotics
Jad Abou-Chakra Queensland University of Technology
Krishan Rana
Queensland University of Technology
Feras Dayoub The University of Adelaide
Niko Sünderhauf Queensland University of Technology
Abstract: For robots robustly understand and interact with the physical world, it is highly beneficial to have a comprehensive representation - modelling geom- etry, physics, and visual observations that informs perception, planning, and control algorithms. We propose novel dual "Gaussian-Particle representation that models the physical world while (i) enabling predictive simulation of future states and (ii) allowing online correction from visual observations in a dynamic world. Our representation comprises particles that capture the geometrical aspect of objects in the world and can be used alongside a particle-based physics sys- tem to anticipate physically plausible future states. Attached to these particles are 3D Gaussians that render images from any viewpoint through splatting process thus capturing the visual state. By comparing the predicted and observed images, our approach generates "visual forces" that correct the particle positions while re- specting known physical constraints. By integrating predictive physical modeling with continuous visually-derivee corrections, about the present and future while synchronizing with reality. We validate our ap- proach on 2D and 3D tracking tasks as well as photometric reconstruction quality. Videos are found at https:/emmbodied-gaussee www.ssiinnnnittuu.o......
unified representation reasons
INTRODUCTION
The real world is governed by many well-understood physical priors matter cannot occupy the same space, scenes comprise rigid and deformable objects, gravity acts on all objects, robots have known kinematic structures. Incorporating such priors into a world representation can constrain how it evolves over time, enforcing adherence to the laws of physics. However, most representations like pointclouds, images [1], or latent descriptors [2, 3, 4] cannot explicitly encode and reason over these priors. Consequently, their ability to predict future states lacks critical physical constraints.
Particle-based physics simulators [5, 7] elegantly capture the physical priors that often known in robotic scenarios and enable forward simulation of dynamics. This makes them attractive for modeling the physical world. To use them in robotics context, we propose method to initialize particles from RBGD observations and to periodically correct the errors accrued over time using only RGB observations from the real world.
To enable continuous state correction through visual feedback, we add a visual aspect to the particles which allows corrective force be calculated. Recent work has shown 3D Gaussians [8, 9] are a differentiable, performant, and expressive representation of visual state that can render images from any viewpoint. Our contribution is to couple these Gaussians to the particles. With this dual Caussian-Particle representation, we can simulate future states using a physics system. We can also predict the visual appearance by rendering the attached Gaussians. Observations are compared to the rendered images to compute a photometric loss which drives the movement of the Gaussians and subsequently generates "visual forces" that correct the positions of the attached particles. Our key contributions are thus: (i) A novel dual Gaussian-Partiche representation that captures geometry, physics, and visual appearance in unified way. (ii) method of initializing this representation from RGBD data and instance maps. (iii) A real-time method to correct the particle states using visual feedback. An overview of our system is shown in Fig.
8th Conference on Robot Learning (CoRI. 2024), Munich, Germany.

---

1. Visual and Physical Priors
2. Dual Gaussian- Representation
Bond
Kinematies
Gaussian
Partiele
Object
Collisions
Gravity
3. Splatting
4. Visual Loss
5. Visual Forces
Figure 1: Image from a real-world experiment showing the available physical priors (1), the dual Caussian-Particha representation (2), the predicted visual state of the world (3), and the corrective "visual" forces (5) being applied as a result of visual discrepancy between the rendered state and the image from the camera (4).
2 PRELIMINARIES
Our approach builds upon Position-Basee Dynamics [5, 6] (PBD) and Gaussian Splatting [8]. This section provides an overview of these two components separately, while the subsequent section details our contribution that interconnects them for robotics setting.
Particle-Based Physics Simulation PBD physics simulation technique well-suited for our robotics application, which requires real-time operation and robust performance. In our formulation, PBD acts on oriented particles where each particle is defined by its position p; E TR3, velocity v; E R3, orientation E $ angular velocity w; E RR3, external force f; E R radius E R and mass m, + A particle may belong to a shape S, and thus has resting position P; as an additional attribute.
At the core of the PBD framework are the various constraints that govem the behavior of the simu- lation. These constraints are defined as cost functions that operate on the particle positions, ensuring the simulation adheres to the desired physical properties. The general form of PBD constraint a cost function C'.Po....Pxa. Pxi Hr) E IR+,, where is relaxation factor. The constraints are min-
imized each simulation step using Jacobi solver. The solver iteratively updates the positions of the particles by aggregating the required positional changes Ap to locally satisfy the constraint. Generally, the change required is given by:
C Ap. -4 Σ, w. V Cp
Cp,C where
w;
m;
In our system, we use ground, collision, and shape constraints. Their associated Ap are outlined below:
PBD's ground constraint is used to prevent particles from penetrating the ground plane given by
(n, d):
App ground Cgrownd T2, d) n, Cgrondd (p d) min(n" T P, + d r,,
(2)

---

PBD's 's collision constraint which operates on pair of particles and j is used to model collisions:
102 P: Pj UN Wy
Appa
C.olpp. Cool min(lp, Pyl
Tjj 0)
(3)
Pp: P2ll
Lastly, to ensure a group of particles belonging to a particular object (either deformable or rigid) maintain their structure throughout the simulation, explained in detail in [7, 10]. Briefly, the shape matching algorithm requires the computation of the following matrix for each shape S:
the shape matching algorithm which is
As = Σ 5 M, R +P.P, - Mcsc., Cs Eies m.P. M
CS Eies maP: M
M
Σ my
(4)
iES
SE
where R. E (O(3) is the matrix form the quaternion Q. As can be decomposed into Rs and thus the changes in particles positions required to maintain structure are given by:
Dpp Kg Rs(P; 0g) + -c-- - Pal
(5)
Here, ky is the stiffness parameter of the shape. Both rigid and deformable objects can be modelled with shape matching. A rigid object is composed of single shape. A deformable object, however, is composed of multiple shapes where each shape is composed of a particle and its neighbours.
For sample implementation of PBD and the shape matching algorithm, we refer the reader to [11]. In our work, we build on Warp's [12] PBD implementation and extend it to incorporate the shape matching algorithm. In our experiments, each physics step takes approximately 5 ms complete. More details on PBD can be found in our supplementary.
Gaussian Splatting Gaussian splatting [8] has emerged as a powerful rendering technique that can capture the state of the visual world with a discrete set of 3D Gaussians. Each Gaussian is parameterized by its position 9; E R orientation R. SO(3), scale s, R2, opacity as E Rt. and color & E 1
Given a viewpoint whose transform relative to the world frame is denoted by V E SE(3) and projection function from the 3D world to the view's screenspace defined by x xx), the color at pixel coordinate u can be calculated by sorting the Gaussians in increasing order of their viewspace z-coordinate and then using the splatting formula in Eq. (6).
Crgb(u) =
Σ (u) II (1-a,(u))
(6)
iEN
ax (u) = aee Bi(u), (u = x Σ x,
(g,)
= =VE is the covariance of the Gaussian projected into the viewpoint's screenspace where J is the Jacobian of the projection function (xx and Σ, = R,diag(s (s?) R For the full details of this process, the reader referred [8].
The rendering equation is not limited to color. In our method, we also associate each Gaussian with segmentation id o (as is done in [9]). Note, however, this is only needed for our initialization scheme and not our prediction and corrective steps. Rendering segmentation is done using:
i-1 II - -1- ay)) j=1
S(u) Σ o,a (u) TEA
Since the splatting process is differentiable, the attributes defining the 3D Gaussians can be learnt to represent a specific scene by minimizing the photometric loss Ligb between set of groundtruth images and their corresponding splatted renders. Our initialization procedure also makes use of L
Tighh =
Σ (rgg(()) - (n)( and Treg
Σ
SS( (y))
(8)
3

---

Initialization Procedure
Results
Q
2
Doll
Initialize pointeloud from posed cameras
Segment points with masks and compute BBox
Fill BBox with spherical
Mug
Gaussions and remove
outside masks
Shape Marching
Ball
4. Train with reconstraction loss and collision avoidance cosntraint
5. Initialize particles Gaussian locations and create shape construints
6. Continue training Gaussians and connect nearest particles
Figure 2: The initialization procedure (left) and example results from real-world data (right).
3 METHOD
Our method creates a model of the world that can at realtime rates be (i) forward simulated, (ii) regularized with physical priors, and (iii) corrected through visual observations from 3 cameras. The model comprises two tightly integrated representations: set of N particles that represent the physical world and are acted upon by PBD physics system, and M Gaussians that visually depict the world through efficient splatting- rendering. The key novelty lies the introduction of "Gaussian- Parttccle" bonds that synergistically couple these two representations, creating bridge between the physical and visual aspects of the modeled environment. A bond is a rigid transform that links Gaussian to its parent particle. In Sec. 3.1, we describe how initialize the particles, Gaussians, and their interconnecting bonds. In Sec. 3.2, we describe how the simulated representa- tion is kept synchronized with the real world using the observations from the cameras.
3.1 Initialization
To initialize the particles, we compute a loose 3D bounding box around each object using the depth data and instance masks. The handful of instance masks required are user-gennt user-generated with Cutie [13] (a mask labelling tool) in only few seconds with minimal effort, however this could plausibly be automated using VLMs or SAM [14, 15, 16). The instance and depth information is only required initialization. The bounding boxes are filled with evenly spaced spherical Gaussians whose radius matches the smallest relevant geometric feature (4 to 7 mm). Gaussians that do not project into the instance mask are pruned. The Gaussian positions, colors, and opacitics are optimized by iteratively solving for collision and ground constraints using the Jacobi solver mentioned in (Sec. 2) and min- imizing the photometric and segmentation reconstruction loss (Eq. (8)) using Adam [17]. Thus for this stage only, the Gaussians act as though they are also particles. Gaussians with opacity lower than 0.3 (emperically chosen) are pruned and their size is upper bounded to roughly the size of the particle. Particles initialized at the locations of the remaining Gaussians. The particles belonging to each object are then connected to each other using shape matching constraints (2).
The user also indicates whether each object is rigid or deformable. We envision that in the future, a VLM could automatically make this determination. These particles represent collision- freee, ground-aligned approximation of the object geometries, filling up the observed shapes. While this approach does not model cavities, this could be addressed in the future by incorporating depth-based losses by replacing the grid initialization with more sophisticated 3D initialization scheme.
Subsequently, we continue optimizing the Gaussians without imposing collision constraints and allow the scale to change. We also introduce new Gaussians by enabling the densification procedure detailed in [8]. Finally, each Gaussian is parented to the closest particle and its location relative to the particle is stored bond. Any Gaussian that is farther from threshold particle is discarded. This reconstruction process as well as its results are visualized in Fig. 2. typical scene contains around thousand particles and ten thousand Gaussians.

---

Prodiction
Comectiin
latesi imagss and juizt puotiums
nubuii juints rus step
Ostinzz Gausiina
Compuse visu! foress Gurrian
5.Aal siveal
Plysics Stop
Steps
30 Hz
Figure 3: Our real-time correction method illustrated in its different steps.
The robot is modelled similarly using renders from its known meshes and inserted into each scene. The background elements (a white table in our experiments) are modelled only using Gaussians with the conventional training regime [8].
3.2 Online Prediction and Correction
After the initialization phase, approach employs combination of Position- -aaoe Dynamics (PBD) and Gaussian splatting optimization to predict the current state of our representation. Our method can be decomposed into two stages: prediction stage and correction stage. These two stages are called sequentially and are illustrated in Fig. 3.
Prediction The PBD physics system acts upon the particles in system at rate of 30Hz. The prediction step constitutes setting the particles associated with the robot to the positions calculated by the forward kinematics, rumning single PBD physics step that forward projects the particles and resolves physical constraints (as deseribed in Sec. 2), and finally rigidly moving the Gaussians bonded to partieles to their new predicted locations. In system, this takes approximately 5 ms to perform on an NVIDIA 3090, leaving 28 ms for the subsequent correction stage.
Correction After the physics prediction, the current state is rendered from known camera view- points. The renders are compared to the images received from the cameras and the photometric reconstruction loss is reduced by optimizing the parameters of the Gaussians. Note that only RGB data is required for the optimization. All Gaussians are allowed to learn new colors and opacities at a low learning rate. This allows shadows imposed by the robot and by the objects to be explained by changing the colors of the Gaussians. Only Gaussians attached to objects are allowed to change their positions and orientations. After 5 optimization steps number tuned to meet realtime constraints the desired positions the Gaussians are stored and all Gaussians are resef to their original posi- tion before the optimization started. The difference in the desired positions of the Gaussians and the original positions of the Gaussians is used to compute a force which is imposed on the connected particles. This force is calculated as f² K, Ej و5(g gy). where 9, and ge the final and initial positions of the Gaussians connected to particle i, 05 is the opacity, and Kp is a proportional gain that has to be tuned (We found K₂ = 60 to be a good value for our experiments). The Gaussians thus never directly moved by the correction step. Rather, the correction step is used generate corrective external forces on the particles which are ultimately resolved by the physics system. The resulting movement of the particles, as orchestrated by the physics, causes the Gaussians bonded to them to move. Consequently, the system is never in a physically infeasible state. Moving the Gaussians in this manner ensures that they always remain in place relative their bonded particles.
4 EXPERIMENTS
We rigorously evaluate the performance of our proposed system across several key metrics to deter- mine its efficacy in dynamic object tracking and photometric reconstruction from novel viewpoints.
Datasets To evaluate our method, we utilize both a simulated dataset and a real dataset, each com- prising a tabletop scene. The simulated dataset (Fig. 4) consists of 25 scenarios designed to highlight various dynamic conditions, including single object pushing (5 scenes), multiple object pushing (5 scenes), object pickup (5), object pushover (5), and pushing a deformable rope (5). The real dataset (Fig. 5) contains 25 scenarios exhibiting similar variations dynamic conditions. The real- world experiments have Aruco markers attached to the objects. The markers are used to extract
5

---

Single
Multiple
Falling
Pickup
Rope
3D Tracking Ervor (cm)
Ours
Physics Only
D3DGS
20
Time
(6)
Figure 4: Tracking error set of points attached to moving objects synthetic scenes showing
different dynamic conditions that include pushing, falling, picking up, and deformable objects.
Table 1: Tracking error and photometric reconstruction quality from unseen views the simulated dataset for our full method, physics only, augmented D3DGS and Cotracker [19]
4 3D Tracking error (cm)
4 2D Tracking error (px)
1 PSNR
Ours Physics
D3DGS*
Ours Physics
D3DGS*
[19]
Ours
Physics
D3DGS*
Single
0.65
9.87
4.05
5.46
81.44
36.19
29.13
17.32
14.16
16.52
Multiple
0.52
6.70
2.71
5.92
52.84
14.25
17.40
17.52
14.34
17.07
Falling
1.52
11.00
7.64
14.50
117.10
62.93
48.33
16.79
12.88
16.26
Pickup
2.25
2.58
0.93
27.61
31.49
9.95
14.24
13.88
13.17
15.84
Rope 1.02
4.50
2.71
7.08
25.55
22.50
28.75
14.63
11.97
15.47
groundtruth and 3D trajectories using a combination of Aruco detection, manual labelling, and factor-graph based optimization [18]. For both datasets, we employ 3 cameras to derive the visual forces, while cameras are used for evaluation puposes only. This diverse set of experiments allows us assess the performance of our method under a range of challenging scenarios, encompassing both rigid and deformable objects, as well as interactions involving single and multiple objects.
Baselines A baseline that is realtime, correctable and serves as world model could not be found therefore, we compare our method with baselines that can separately track and predict. We note that our main contribution is that we can do both at the same time. Therefore, we compare our method as a 3D tracker against Dynamic 3D Gaussians (D3DGS) [9], a 2D tracker against Cotracker [19], and as a forward model against only using physics simulator [5] without visual forces.
Unlike our approach, D3DGS incorporates physical priors directly into the Gaussian optimization process through auxiliary losses. D3DGS cannot be used as world model like our method since it cannot project what may happen to the Gaussians if forces act on the system. It also requires fore- ground mask to be provided at each timestep. In [9], background subtraction is used to acquire those masks. However, we found that under our challenging realtime constraints where only 3 cameras are used - this leads to catastrophic failure in tracking as visualized in Fig. 6. Therefore, to make it competitive, we augment the baseline (D3DGS*) by giving it groundtruth hand- labelled masks and by forcing the Gaussians associated with the robot to move according to the forward kinematics of the robot rather than through the optimization process. Furthermore, we found D3DGS auxilliary losses make it half as slow as our visual training iteration, but we assume that optimizations can be made to match the speed of ours and thus allow them six training iterations to match our five training iterations and one physics step. For our method, the full parameters are listed in the supplementary.
Metries We evaluate our method on the mean error in the 2D and 3D trajectories of known points. We also evaluate the foreground photometric reconstruction quality (which includes only the objects on the table) from unseen viewpoints. The predicted 3D trajectory of query point is obtained by tracking the frame of the Gaussian that was closest to that query at the first timestep. This procedure consistent with the approach used in [9]. 3D trajectories are projected into each camera to obtain the 2D trajectories and their initial points are used to query Cotracker [19]. The trajectory error is calculated as the mean difference between the groundtruth and the predicted trajectories of several points sampled on the objects.

---

Single
Multiple
Falling
Pickup
Rope
3D Tracking Eroor (cm)
Ours
Physics Only
D3DGS
Time (8)
Figure 5: The tracking error on a set of points attached moving objects is recorded on real scenes.
Table 2:
Tracking error and photometric reconstruction quality from unseen views on the real
dataset for our full method, physics only, augmented D3DGS and Cotracker [19].
3D Tracking error (cm)
4 2D Tracking error (px)
t PSNR
Ours
Physics
D3DGS
Ours Physics
D3DGS*
[19]
Ours
Physics
D
Single
1.40
31.71
5.96
11.90
264.20
57.10
88.42
16.49
10.46
12.05
Multiple
1.84
26.07
10.09
17.00
238.42
57.75
110.79
16.99
10.64
14.18
Falling
2.06
7.80
5.90
16.51
62.56
56.98
85.33
17.05
13.56
15.87
Pickup 0.68
3.36
3.36
6.65
37.34
7.80
8.21
16.38
13.40
15.22
Rope 1.01
11.24
11.24
6.68
76.98
82.45
41.87
15.81
13.73
12.11
Results Fig. 4 and show 5 of the 25 scenarios tested for each of the simulated and real datasets. The 3D tracking error plotted over time and shows our method robustly tracking the objects in the scene. Tables and summarize our metrics over each scenario. Our method outperforms all baselines on all experiments except the simulated Pickup tasks. The Pickup tasks are highly dynamic environments where our physical priors were significantly different to the physics exhibited in the simulated scene (see videos on website). This highlights an expected weakness of our system where significantly misaligned physical priors can degrade performance. Nevertheless, our system is able recover and acquire the final state the pickup task as shown in Fig. 4. In all other experiments (45/50), physical priors significantly improve tracking performance. Fig. 6 shows qualitative results.
5 RELATED WORK
To the best of our knowledge, we are the first to create representation consisting of both particles and 3D Gaussians for the purposes of fusing physical and visual priors within correctable robotics world model. However, the use of particle-based physics system alongside visual component other than Gaussians has been at the core of other works [20, 21, 22, 23]. Moreover, regularizing Gaussian splatting with physical priors through means other than physics framework has also been explored [9] (the baseline used in our experiments).
ParticleNeRF [20] uses particles that are acted upon by physics system and can be rendered from any viewpoint using Neural Radiance Field [24] formulation. Particles are associated with latent features which can be decoded into radiance field by a small neural network. While ParticleNeRF introduced the use a particle-based physics system (PBD) to incorporate physical constraints and deviations from the NeRF's reconstruction loss relative to particle positions, it only utilized collision constraints and did not fully explore the idea employing various physical priors to regularize the reconstruction. In part, due to the slower NeRF formulation and the requirement of upwards of 10 cameras to constrain the optimization, ParticleNeRF could not be deployed real world robotics setting with realtime constraints. In contrast, our work uses the much faster Gaussian splatting to represent the visual world and further makes heavy use physical priors which allow visual corrections from as little as 3 cameras in the scene.
Particles within a PBD framework paired with corrections from observed pointclouds have been used to model soft human tissue within the domain of surgical robotics [21, 22, 23]. Our method is more general, realtime, and uses visual feedback achieved through the fast Gaussian splatting rather than pointclouds and SDFs.
7

---

Os
27s
GT
Our
Physics
D3DGS
Figure 6: A scene's predicted visual state with different methods. Our method synchronizes with the real state using combination of physical prediction and visual correction. Physical prediction alone can estimate the state for about seconds but will eventually desynchronize. Visual correction alone (D3DGS) allows the Gaussians to move in physically infeasible ways, creating scenarios where objects split and Gaussians move freely within the object despite auxiliary structural losses.
D3DGS [9] adds physical priors to the Gaussian optimization process with the aim of regularizing the movement of the Gaussians. The significant difference with work is that the physical priors are not strictly enforced by a physics system rather auxiliary losses are added to the photometric reconstruction loss. Importantly, this means that D3DGS cannot cannot be used to predict future states. Nevertheless, D3DGS has shown good tracking and recon- struction performance when groundtruth masks are provided when upwards of 20 well-distributed cameras are observing the scene and the optimization is given ample time to converge. While these requirements are difficult to achieve in robotics setting, this visually-driven optimization serves as our baseline and ablates the visual component of our system as well as outlines the importance of grounding the optimization with physics system.
world model because
LIMITATIONS AND CONCLUSIONS
Our method was demonstrated on table-top scenario where heavy use of physical priors can be made. The extension to an open-worke setting left to future work. We make an assumption that the predicted state will be close enough to the groundtruth so that visual forces can be effective. This assumption broken when significant errors in modelling cause the physics system to accrue enrors at faster rate than can be corrected. This can occur when an object is moved very quickly by robot. Consequently, a method of closing the loop using global visual information needed plausibly using system like [25]. We foresee that more sophisticated initialization procedure that uses learnt shape priors would be more powerful. Lastly, our method does not alter the physical structure parameters of the objects once the modelling is done. Therefore, observations are not used to correct modelling mistakes. A means of online structure correction and system identification would extend this framework.
In conclusion, we have presented hybrid representation consisting of particles and 3D Gaussians that together represent the physical and visual state of the world. In conjunction, they can be used to predict future states and correct the predicted state from observed data. This synergy makes them suitable for use as world model in future robotic works. The world model can then be used to extract object state for a reinforcement-basee or an imitation-based policy or to plan for future actions using model predictive control.

---

ACKNOWLEDGMENT
The authors acknowledge continued support from the Queensland University of Technology (QUT) through the Centre for Robotics. This work was partially supported by the Australian Gov- ernment through the Australian Research Council's Discovery Projects funding scheme (Project DP220102398).
References
[T] C. Finn, I. J. Goodfellow, and S. Levine. Unsupervised learning for physical inter- action through video prediction. ArXiv, abs/1605.07157, 2016. URL https://api. semanticscholar woo.ooog/go....... Perg/CorpusID:2220
:265911. 2659157.
[2] Y. Ze, N. Hansen, Y. Chen, M. Jain, and X. Wang. Visual reinforcement learning with self- supervised 3d representations. IEEE Robotics and Automation Letters, 86)):889-22897, 2023. doi:10. 109...00000RR.222.22555555
[3] D. Driess, I. Schubert, P. Florence, Y. Li, and M. Toussaint. Reinforcement learning with neural radiance fields. Advances in Neural Information Processing Systems, 35:16931-16945, 2022.
[4] D. Driess, Z. Huang, Y. Li, R. Tedrake, and M. Toussaint. Learning multi-object dynamics with compositional neural radiance fields. In Proceedings of The 6th Conference on Robot Learning. volume 205 of Proceedings of Machine Learning Research, pages 1755-1768, PMLR, 14-18 Dec 2023.
[5] M. Müller, B. Heidelberger, M. Hennix, and J. Ratcliff. Position based dynamics. Journal of Visual Communication and Image Representation, 18(2):109-118, 2007. ISSN 1047-3203. com/
200.00000.....000000000.... .005. URL
[6] M. Macklin, M. Müller, and N. Chentanez. Xpbd: Position- simulation of compliant con- strained dynamics. Proceedings of the 9th International Conference on Motion Games, MIG '6, page 49-54, New York, NY, USA, 2016. Association for Computing Machinery. ISBN 978 97814503459277. 101110.1145799922588228 2994272. URL 2994258. 2994272.
wtt.doo..ogg/.... 1145/
[7] M. Müller and N. Chentanez. Solid simulation with oriented particles. ACM Trans. Graph., 30(4), jul 2011. ISSN 0730-0301. doi: 15/0114//200324 10. 1D..1//22010324 1964987.
URL https://doi.orgg
[8] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL https: //repo-sam inria. www.ii...f..iiiaeaae...e. http://uggpp////gggasssaaaaaa/// gaaussannsslatting.
[9] Luiten, G. Kopanas, B. Leibe, and Ramanan. Dynamic 3d gaussians: Tracking by persis- tent dynamic view synthesis. In 3DV, 2024.
[10] M. Müller, Heidelberger, M. Teschner, and M. Gross. Meshless deformations based on shape matching. ACM Truns. Graph., 24(3):471-47) jul 2005. ISSN 0730-0301.. doi: 10. 1145/1073204.1 1073216. URI http://ts:// 110..111//7730.11077777 1073216.
[11] J. Bender.
PosttionBaseDDynamiie, Dec. 2022.
URL https://github... com/
[12] M. Macklin. Warp: A high-performanne python framework for gpu simulation and graphics.
https://githube com/nvidia/warpp http::///tttt...////////////s/s March 2022. NVIDIA GPU Technology Conference (GTC).
[13] H. K. Cheng, S. W. Oh, B. Price, J.-Y. Lee, and A. Schwing. Putting the object back into video object segmentation. In arXiv, 2023.

---

[14] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and Girshick. Segment anything. arXiv:2304.02643; 2023.
[15] Liu, Z. Zeng, Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprini Axiv:23033055999 2023.
[16] T. Ren, S. Liu, A. Zeng. J. Lin, K. Li, H. Cao. J. Chen, X. Huang, Y. Chen, F. Yan, Z. Zeng, H. Zhang, F. Li, J. Yang, H. Li, Q. Jiang, and L. Zhang. Grounded sam: Assembling open- world models for diverse visual tasks, 2024.
[17] D. P. Kingma and J. Ba. arXiv:1412.6980, 2014.
Adam: A method for stochastic optimization. arXiv preprint
[18] H. Martiros, A. Miller, N. Bucki, Solliday, Kennedy, J. Zhu, T. Dang, D. Pattison, H. Zheng, T. Tomic, P. Henry, G. Cross, J. VanderMey, A. Sun, S. Wang, and K. Holtz. Sym- Force: Symbolic Computation and Code Generation for Roboties. In Proceedings of Robotics: Science and Systems, 2022. doi: 10. 10..15607/SS..22222 2022.X1I...
[19] N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker: It is better to track together. arXiv:2307.07635, 2023.
[20] J. Abou-Chakra, F. Dayoub, and N. Sünderhauf. Particlenerf: A particle-based encoding for online neural radiance fields. In Proceedings of the IEEE/CVF Winter Conference on Applica- lions of Computer Vision ( (WACV), pages 5975-5984, January 2024.
[21] F. Liu, Z. Li, Y. Han, J. Lu, F. Richter, and M. C. Yip. Real-to-sim registration of deformable soft tissue with position-based dynamics for surgical robot autonomy. In 2021 IEEE Inter- national Conference on Robolics and Aulomation (ICRA), pages -2324-1234, 2021. doi: N0.1109/CCA448066 2021 .95611177
[22] Liu, E. Su, Lu, M. Li, and M. C. Yip. Robotic manipulation of deformable rope-like ob- jects using differentiable compliant cosition-bassd dynamics. IEEE Robotics and Automation Letters, 8(7):3964 1971, 2023. doi: 109:0.1199//R..20333 33667766..
[23] X. Liang, F. Liu, Y. Zhang, Y. Li, S. Lin, and M. Yip. Real-to-sim deformable object manipu- lation: Optimizing physics models with residual mappings for robotic surgery. arXiv preprint arXiv:2309. 1656, 2023.
[24] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[25] C. Doersch, Y. Yang, M. Vecerik, Gokay, A. Gupta, Aytar, J. Carreira, and A. Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 1006 10062- 2023.
10

---

Supplementary Material
Physically Embodied Gaussian Splatting: A Visually Learnt and Physically Grounded 3D Representation for Robotics
Jad Abou-Chakra', Krishan Rana², Feras Dayoub', Niko Sünderhauf
Queensland University of Technology University of Adelaide
A Experimental Setup
The real-world experiments are conducted us- ing the tabletop setup shown Supp. Fig- ure 1. The setup employs Franka Emika robot equipped with two end-effectors: a standard gripper for pick-up scenarios and a pusher for other scenarios. The tabletop and robot are ob- served by five cameras: three Intel RealSense D455 cameras and two D435 cameras. These cameras jointly calibrated using hand- eye calibration technique. During operation, all five cameras are utilized for system initializa- tion. However, only the three D455 cameras are employed during the prediction and correction stages. In all scenarios, the robot is teleoper- ated to manipulate objects on the tabletop. The datasets are captured by recording the image stream from the cameras and encoding them as HEVC videos. These videos are scaled to resolution 640x360 and decoded real-time during evaluation to mimic live operation. Ad- ditionally, the robot's 's joint positions are times- tamped and saved during the recording process and replayed during the evaluation.
Evaluation
Training
Supplementary Figure 1: The tabletop setup used the real experiments showing the robot, some of the objects used in the scenarios, and the position of the cameras used.
B
Implementation
The system follows a two-step process: initialization and prediction/correction.. During initializa- tion, particles and Gaussians are generated for each detected object the scene. Subsequently, the system enters the prediction and correction stage, where the particles are simulated using Position- Based Dynamics (PBD) physics system, while corrective forces are calculated hased on the Gaus- sians attached to the particles. This section elaborates on the implementation and parameterization details of each phase.
Static Scene Initialization The tabletop is modeled using the five RGBD cameras in the scene, employing the standard Gaussian Splatting technique. However, to avoid interference with object placed on the table, the Gaussians are initialized as thin disks. Additionally, the table's pointcloud utilized calculate the ground plane. The Gaussians are trained using the Adam optimizer for 500 steps, with position learning rate of le color learning rate of 2.5e 33 scaling learning rate of -3 opacity learning rate of 1e -2, and rotation learning rate of 1e -3 The scale clamped between 1 mm and cm.

---

Robot Initialization The robot's particles are manually fitted the links using Blender.
Algorithm Dual Gaussian- Initialization
The link each particle belongs to is stored so that forward kinematics can be used to ap- propriately change its position. Furthermore, the robot is rendered in Blender from multi- ple viewpoints, and Gaussians are trained to reconstruct these renders. These Gaussians
1: Fill BBox with Spherical Gaussians
2: Prune Gaussians Not In Instance Masks
3:
for n iterations do
4:
for all images and masks I do Adam step
5:
I. +- Ligh + +Tegg L.backward()
then bonded to the closest particle on the robot. The combination of particles, Gaus- sians, and bonds is inserted at the start of ev- ery scenario. The same parameters used for training the static scene are also applied to the robot.
7:
g. a, c optimizer.step0
8:
for k iterations do
> Jacobi step
g = solvecColissonnCoosttrnnttg
10:
g
11: p= g Create particles Gaussian locations
12: Initialize particle mass and velocities
13: Create particle shape constraints
Object Initialization For each object, the 3D bounding box is calculated from its point- cloud, which is extracted from the depth and instance masks. The initialization process is described in Algorithm 1. We use n 80 and m 250. All particles are initialized to a mass of 0.1 kg with the exception of the real and simulated rope which 0.2 kg and 0.3 kg. The higher the mass of the particle, the less the influence of the vi- sual force. The mass acts as both a physical and a visual inertia. These concepts can be tun-
14: for m iterations do
15: for all images and masks 7 do
16:
Hrgh.backwar(()
17:
g. a, c, optimizer.stepO) g,a, = density(g, a, a,
18:
19:
for each Gaussian i do gi-parent findClosestParticeelp
20:
Algorithm 2 PBD Physics Step
1: for all particles do
IItegraate particles
2:
Po qn - Pa q:
separated in future work if fine-graine ing is needed. For scenarios involving rope, the corrective forces are less reliable than that of larger bodies because they occupy less pix- els in the image and the physical priors are less constraining because of the deformabil- ity. We compensate for the increased noise in the corrective forces by increasing the visual inertia. Note that Algorithm is repeated for each object. Future work may choose to build all the objects simultaneously rather than se- quentially to reduce the overall duration of the initialization. In the current implemen- tation, object modeling takes approximately 20 to 40 seconds, which we found acceptable
3:
p; + P, + Alv; +
A m, (f: grrvityy)
0 welAr
5:
Q: [. sin 0, cos 0|g
6:
for k solver iterations do Resolve constraints
7:
for all particles do
P. groundConstraints(()
9:
for all collision pairs i, do
10:
P. collisiinCConstraintsi,,,
11:
for all shapes s do
12:
for particles i in s do
13:
Pi: Q: +- shapeMatching(i..)
14:
for all particles do
>> Update velocities
15:
v. f (p; Pu)/
given that it is only done once per scenario.
16:
wi + axis ")) aangge(g.ggo
Prediction Step The Position- -Baedd Dy- namies (PBD) physies system is used pre- dict the locations of the particles and the Gaussians each timestep. It runs at fixed frequency of 30 Hz (33.33 ms per step). The physics step is described in Algorithm 2. We use 20 substeps. At each substep, the veloc- ities and forces are integrated, and then the constraints are solved using a Jacobi solver. Four Jacobi iterations are employed to suf- ficiently solve the physical constraints. Af- ter every physics step, the particle velocities are multiplied by 0.9 (an empirically chosen value). This damping contributes to system stability.
Algorithm 3 Visual Forces
grvv { 9
> Save positions
2: o AdamOptimizer()
for n iterations do
4:
Choose random image /
5:
Ligh(1) backkardd
6:
ginot objects].grad
7:
g. c, o, + o.step()
8:
for every Gaussian i do
k gt.parent
10:
if' k is not None then
11:
fx - f& Kp g ploy)
12: g - gre
> Reset positions
12

---

Other
Physics (5ms)
Visual Forces (22ms)
Other
Physics
Vis
dam Adam
GUI,
Adam Adam 2
Adam
Adam
Adam
(5ms)
Adam
Adam
33.33ms
Supplementary Figure 2: The various functions called during the prediction and the correction step profiled. In the 'Other' phase, the GUI is drawn and new sensor observations are read. The physics step takes 5 ms and is followed by approximately 22 of Adam optimizations that are used to compute the visual forces.
Effect of Physical Priors on Tracking Error
Physical Priors
Collision
3.5
No Collisions
No Ground
No Gravity
Tricking Error (cm)
2.5
No Robot
2.0
1.5
1.0
0.5
0.0
Time (8)
Supplementary Figure 3: An ablation showing the effect of different physical priors the 3D tracking error of 12 points located on two objects on a tabletop. The scene used for this ablation is "Multiplel" from the simulated dataset. Using all physical priors produces average the lowest tracking error over time.
Correction Step Visual forces are computed in the correction step using Algorithm 1. Gaussian displacements are calculated using 5 iterations of the Adam optimizer. The scales of the Gaussians fixed, while the positions, rotations, opacities, and colors are allowed to change. Adam's 's internal parameters are reset at every new physics step. Gaussian displacements below 2 mm are ignored to increase stability. The position learning rate is set between 1 and -3 depending on the scene, while the rotation, color, and opacity learning rates are set to 1e le-4, 5e - and 5e respectively. Allowing colors, opacities, and rotations to change gives the system more ways to explain lighting variations that should not be explained by the motion of the Gaussians. A Kp of 60 is used in all scenarios.
The prediction and correction steps are profiled in Supp. Figure 2.
C
Ablations
We evaluate the effectiveness of our system's embedded physical priors by sim- ulating a scene with two objects, as illustrated in Supp. Figure 3 and summarized Supp. Figure 5. The scenarios highlight how our system's 's performance is enhanced by incorporating various phys- ical constraints: (i) With all physical priors enabled, our system accurately captures the objects' (ii) When collisions between particles are ignored, the objects' states deviate from the ground truth, particularly during intense collision events (Collisions 2 and 3). (iii) Disabling the ground plane and gravity causes the objects' motions to oscillate continuously, as their movements are no longer properly regulated. (iv) Even with the ground plane intact, disabling gravity leads to similar oscillatory behavior, as the objects are not subjected to the expected downward force. By adding physical priors, our system achieves better predictions that more closely match the groundtruth.
Physical Priors
dynamics, including collisions and interactions with the environment.
13

---

Supp. Figure 4 ablates (i) the number of cameras used to compute visual forces, (ii) the resolution of the images used for the reconstruction loss, (iii) the effect of visual gain, (iv) the Gaussian position learning rate, and (v) the number of Adam iterations. The mean tracking error across the entire trajectory also reported in Supp. 5.
Cameras The ablations reveal that increasing the number of cameras yields diminishing returns within our framework. We observe that higher resolutions lead to lower tracking error. There is, however, only a slight difference between 1280x720, 640x360, 320x 1280x720 comes at a significant computational cost with visual force computation taking approximately 40 ms compared to the 20 ms for the lower resolutions. Below 640x360, the factor limiting performance is no longer resolution and thus there is no performance gain. For these reasons, we choose the 640x360 as the image size with which we calculate the visual forces.
Effect of Number of Camerus on Tracking Error
Number of Camerus
Hffect of Image Resolution 3D Tracking Emrr
Camera RGB Resolution
1280x720
640x360
320x180
160x90
Effect Visuul Gain (Kp) on 3D Tracking Error
Tracking Liror (cm)
Visuul Gain (Kp)
30
120
Effect Increasing Gaussian Position Learning
Tracking Error
Gaussian Position Learning Rate
0.5c-3
1.0c-3
2.0c-3
3.00-3
4.0c-3
Ma.
Effect of Increasing Number of Gaussian Optimization Steps 3D Tracking Error
Adam Iterations Per Physics Step
1.5
1.0
0.5
10
15
25
30
Time (s)
Supplementary Figure 4: The effect of varying the parameters of our system on 3D tracking perfor- mance.
14

---

Physical Priors
All
No Collision
No Gravity
No Robot
No Ground
3D Tracking error (cm)
0.3640.27
0.52+0.58
0.52+0.33
0.57+0.61
0.84+0.72
(a) Effect Physical Priors on the 3D Tracking Error em reported as meantstandard deviation.
Number of Cameras
5
3D Tracking Error (cm) 1.16 +0.82
0.41+0.47
0.37=0.27
0.4
0.39=0.28
(b) Effect Number of Cameras on the Tracking Error in em reported a meantstandard deviation.
Image Resolution
1280x720
640x360
320x180
160x90
3I) Tracking error (cm)
0.58#0.49
0.59#0.49
0.65+0.50
0.87+0.56
(c) Effect of Number of Cameras on the 3D Tracking Error in em reported meantstandard deviation.
Visual Forces Gain
30
60
90
120
3D Tracking error (cm)
0.41#0.41
0.40#0.33
0.460=0.37
0.750#0.57
(d) Effect of Visual Gain (Kp) the 3D Tracking Error in em reported as meansstandard deviation.
Learning Rate
0.5e-3
le-3
2e-3
3e-3
4e-3
3D Tracking error (cm)
7.10#8.53
0.55+0.48
0.40+0.36
0.3940.31
0.40=0.31
(e) Effect of Increasing Gaussian Position Leamingg Rate the 3D Tracking Error in reported meandstandard deviation.
Number of Adam Iterations
5
7
11
3D Tracking error (cm)
7.1+8.53
0.62+0.59
0.40+0.36
0.40+0.38
0.43+0.46
(I) Effect of Increasing Number of Gaussian Optimization Steps on the 3D Tracking Error em reported as mcantstandard deviation.
Supplementary Figure 5: The effect of varying the parameters of system the mean 3D track- ing performance.
Visual Forces Our framework uses visual forces create the corrective actions needed to keep the Caussian-Partiche representation synchronized. This results in smooth corrections but it also creates dynamic effects that, without careful tuning, creates oscillations. These oscillations are similar to the behaviour of an undamped spring system. Future work may look into removing the oscillatory effect by possibly adding a derivative term the visual force calculation. For this work, we tune our system and find a balance between an acceptable amount of oscillations and tracking ability. The ablations in Supp. Figure 4 show that a high gain (and/or high Gaussian position learning rate) produces high oscillation and that low gain (and/or low learning rate) has a detrimental effect on tracking. The number of Adam iterations chosen that realtime constraints are met. The ablation shows that reducing the number of Adam iterations is trade off that can be made when the physics timestep takes longer than expected without significant impact on overall synchronization of the world model.
Initialization We evaluate our initialization method by comparing it with particle positions man- ually modeled using Blender. We refer the manual initialization as the "oracle" initialization because the known object meshes are used to manually model the objects with particles. We show examples of the oracle initialization alongside our method's initialization in Supp. Figure 6. We evaluate the tracking performance of our system using the oracle initialization and the initialization scheme described our method. The results, presented Table demonstrate that our automated initialization method achieves performance comparable to the oracle.
15

---

Our Initialization
Oracle Initialization
Our Initialization
Oracle Initialization
Mug
T-Block
Supplementary Figure 6: Particle placements of our initialization method compared to the oracle initialization.
Supplementary Table Tracking error and photometric reconstruction quality from unseen views
from scene from each scene category. The table compares our automatic initialization with manual modelling procedure (oracle), showing similar tracking errors.
2D Tracking Error (px)
3D Tracking Error (cm)
PSNR (FG)
Ours
Oracle
Ours
Oracle
Ours
Oracle
Single
5.31
4.27
0.54
0..4
17.69
17.74
Multiple
4.38
6.69
0.39
0.55
17.73
17.65
Falling 13.04
14.61
1.18
1.52
16.18
16.65
Pickup
25.92
28.89
2.09
2.25
15.11
14.65
Rope 10.96
9.62
1.39
1.07
14.73
15.55
D
Failure Modes
The Gaussian-Parie Particle representation deviate from the groundtruth in several ways. If the ren- dered state of the scene significantly differs from the groundtruth image, the visual forces will not create meaningful corrections.
Additionally, the physical modelling is significantly different real world counterpart, the physical priors will have detrimental effect on the tracking performance of the system. This can be seen in the real scenario title "Pushover 5"r Supp. Figure where T-Block could not be pushed over and thus escaped the radius of convergence of the visual forces.
In some instances, both the texture and the geometry of the object are simultaneously ambiguous. In the simulated "Rope I" scenario, the rope can rotate around its spine without impacting either the geometry or the texture thus allowing for slight steady state error occur.
E Design Choices
The primary utility our representation that it can be forward simulated at faster than realtime rates and corrected from visual observations at realtime rates. Our design choices were thus primar- ily guided by the need to lower the computational overhead of both the prediction and the correction step.
We opted for Position-Basee Dynamics (PBD) despite its lower physical accuracy compared to other simulators because of its superior real-time performance and stability. Additionally, chose Gaussian Splatting over other differentiable rendering techniques due to its status as the fastest differentiable renderer available.
The decision use visual forces for state correction was also driven by the need to reduce com- putational demands. Although alternative methods, such as integrating additional shape matching constraints between the particles and the Gaussians, could allow Gaussians to influence particle po- sitions, these approaches would be more computationally expensive than our method. They would require resolving constraints over multiple iterations every PBD substep and would couple the physics system with the correction system, making performance of the physics loop dependent on
16

---

1. Physical Prediction Failure
2. Visual Symmetries
Pushover
Rope
GT Trajectory
Tracked Trajectory
Supplementary Figure 7:
The first image shows highly dynamic scenario where the physics failed to push the TBlock into a location where visual forces could correct it. The second image shows scenario where both visual and geometrical symmetries allowed the rope to rotate around its central axis and created steady state error in tracking.
the number of Gaussians in the scene. In contrast, our method uses visual forces that are simple to implement, fast, and maintain clear separation between prediction and correction systems.
F Experimental Results
The 3D tracking performance of our system on all scenarios are shown in Supp. Figure 8.
17

---

31 Tracking Error on All Simulated and Real-World Scenarios
Ours Single 4
Plysics Only
DIDGS
Single
Single 2
Single 3
Single 5
10
20
Multiple
Mulriple 2
Multiple
Multiple
Multiple 5
20
20
0
0
Pushover
Pushover
Pushover
Pushover
Pushover
10
10
0
Pickup
Pickup
Pickup
Pickup
Pickup
Rope
Rope
Rape
Rape 4
Rupe
25
10
Trockngg Froo
Single
Single 2
Single 3
Single
Single 5
50
50
50
40
10
30
20
40
10
20
30
10
20
30
Multiple
Multiple 2
Multiple
Multiple
Multiple 5
25
20
25
10
20
30
20
40
10
20
30
20
40
20
40
Pushover
Pushover
Pushover
Pushover
Pushover
20
20
10
10
20
15
10
15
Pickup
Pickup
Pickup
Pickup
Pickup
10
10
Rope
Rope
Rape
Rope
Rope
50
50
10
20
30
10
10
20
Time (s)
Supplementary Figure 8: The 3D tracking performance of our system and its baselines on all sce- narios (simulated and real)
18


Reviews:
[{"review_id": "czIfV17CiA", "reviewer": "Reviewer_pZSN", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors propose an innovative interactive scene representation for robotic manipulation using particles and Gaussian splatting. This approach employs Gaussian splatting to correct particle positions in the Position Based Dynamics (PBD) simulation while integrating other PBD priors to appropriately model robot-object interactions using particles. The method shows promising results for 3D tracking and has numerous potential downstream applications.", "summary_of_recommendation": "Tthis research presents an innovative approach to interactive scene representation for robotic manipulation, with the potential to significantly impact the field. There are details and clarifications missing in the current manuscript, especially in the preliminaries section, which the authors should address.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- How sensitive is the method to the type of shape-matching algorithm used? Different algorithms may impact the system's performance, and insights into this aspect would be valuable.\n\n- The correction step requires clarification. It's not immediately apparent why the Gaussians aren't moved with the particles assigned accordingly instead of the particles moving due to the visual loss. The authors should explain their reasoning behind this choice and its implications.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "d2NOZvdCbf", "reviewer": "Reviewer_ud25", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a particle-based physically enabled Gaussian Splatting approach to integrate the predictive physical modeling and visual corrections.", "summary_of_recommendation": "If the prediction is not learnable, I strongly doubt the contribution and impact of this work.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "* Does the number of Gaussians always more than particles? Is there any insights behind it? For a large Gaussian, will it be better to associate multiple particles to it?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LyNsJXNGy4", "reviewer": "Reviewer_iPN3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a representation based on particle-based dynamics (PBD) and Gaussian splatting that is both predictive and can be corrected by visual observations in real-time. A detailed initialisation scheme is proposed to initialise both the particles and Gaussians along with the rigid \"bonds\" that connect them. Prediction is done via particle-based dynamics, and corrections use a few Gaussian splatting optimisation steps to computed a \"visual force\" to be used within the PBD to correct the particle positions. Experiments demonstrate better tracking than baselines, and demos/videos demonstrate impressive tracking of rigid and linearly deformable objects (e.g. ropes).", "summary_of_recommendation": "POST-REBUTTAL UPDATE: Really interesting. The work has been scoped more properly, and more ablations in the supplement provide insight into the strength of the initialisation. I do wish there was more insight into the design choices of the system, but I think the contributions are important enough to be shown at CoRL.", "technical_quality": 4, "clarity_of_presentation": 5, "potential_impact": 4, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Questions/Comments:\n- What are the details of the shape matching constraints?\n- For the prediction step (PBD), I assume there is a given joint torque applied to the robot by a user? Can the authors detail this?\n- Line 147: What are \"features\" here? did authors mean the positions/orientations/scale/color?\n- While the supplement has some ablations that show results on a single video, quantitative results could be helpful here.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The following strengths and weaknesses summarize the reviewers' initial comments for this submission.

Strengths:
- The proposed approach is novel and interesting.
- The paper is mostly well written and clear.
- Experimental evaluation of 2D and 3D tracking performance demonstrate better performance than provided baselines.
- An implementation with real-time tracking performance is demonstrated.

Weaknesses:
- The learning aspect of this paper is questionable, as the PBD simulator is not learnable/adapted.
- The paper lacks ablations for understanding design choices.
- The claimed usability as a world model is not substantiated in experiments.

The author response and discussion phase well addressed the reviewer concerns. The approach is mainly dynamic 3D reconstruction and updating a PBD simulator to follow the dynamic reconstruction. The learning aspect of the approach is minor as it only enables future predictions in the observed scene and the approach does not adapt properties of the dynamics models (e.g. physics parameters). Nevertheless, the reviewers find the approach relevant and interesting for the CoRL community, hence, recommending acceptance of the paper.

The revised paper must not violate the page limit for submission.
Decision: Accept