GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulation
Authors: Junhao Cai
Yuji Yang
Weihao Yuan
Yisheng HE
Zilong Dong
Liefeng Bo
Hui Cheng
Qifeng Chen
Venue: NeurIPS 2024 oral (2024)
Keywords: Object Property Identification
Gaussian-inform Continuum
Primary area: machine_vision
TL;DR: We propose a novel hybrid pipeline that takes advantage of the 3D Gaussian representation of the object to both acquire 3D shapes and empower the simulated continuum to render 2D shapes for physical property estimation.

Abstract: This paper studies the problem of estimating physical properties (system identification) through visual observations. To facilitate geometry-aware guidance in physical property estimation, we introduce a novel hybrid framework that leverages 3D Gaussian representation to not only capture explicit shapes but also enable the simulated continuum to render object masks as 2D shape surrogates during training. We propose a new dynamic 3D Gaussian framework based on motion factorization to recover the object as 3D Gaussian point sets across different time states. Furthermore, we develop a coarse-to-fine filling strategy to generate the density fields of the object from the Gaussian reconstruction, allowing for the extraction of object continuums along with their surfaces and the integration of Gaussian attributes into these continuum. In addition to the extracted object surfaces, the Gaussian-informed continuum also enables the rendering of object masks during simulations, serving as 2D-shape guidance for physical property estimation. Extensive experimental evaluations demonstrate that our pipeline achieves state-of-the-art performance across multiple benchmarks and metrics. Additionally, we illustrate the effectiveness of the proposed method through real-world demonstrations, showcasing its practical utility. Our project page is at  https://jukgei.github.io/project/gic.

GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulation
Junhao Cai
Yuji Yang?*
Weihao Yuan
Yisheng He
Liefeng Bo The Hong Kong University of Science and Technology, Sun Yat-sen University, Allibaba Group Equal contribution, order determined by coin toss. Corresponding author.
Zilong Dong
Hui Cheng" Qifeng Chen
Abstract
This paper studies the problem of estimating physical properties (system identi- fication) through visual observations. To facilitate geometry-awand guidance in physical property estimation, we introduce a novel hybrid framework that leverages 3D Gaussian representation to not only capture explicit shapes but also enable the simulated continuum to render object masks as 21) shape surrogates during training. We propose new dynamic 3D Gaussian framework based motion factorization to recover the object as 3D Gaussian point sets across different time states. Furthermore, we develop coarse-to-fine filling strategy generate the density fields of the object from the Gaussian reconstruction, allowing for the extraction of object continuums along with their surfaces and the integration of Gaussian attributes into these continuums. In addition to the extracted object surfaces, the Faussian-informe continuum also enables the rendering of object masks during simulations, serving as 2D-shape guidance for physical property estimation. Extensive experimental evaluations demonstrate that our pipeline achieves state-of-the-art performance across multiple benchmarks and metrics. Additionally, we illustrate the effectiveness of the proposed method through real- world demonstrations, showcasing its practical utility. Our project page is at http://pp....iiuuuubbooooppcc/io
Introduction
Identifying the physical properties of objects (i.e., system identification) is essential for numerous applications such as games, digital twins, and robotic manipulation []. Although humans can intuitively deduce the underlying physical properties with single glance when the object under- goes deformation, estimating the properties with only visual observations remains challenging for computational perceptual algorithms.
To tackle this challenge, many established methods 4 adopt the assumption of elastic material and perform physics-based modeling based on mass-spring systems (MSS) finite element method (FEM) to model and simulate the dynamics of the objects. Such an assumption inevitably restricts the ability to simulate more general types beyond elastic materials, such as fluids granular media. Another problem of previous methods lies that many methods [8-10] require the ground-truth full knowledge of object geometry for the identification, which limits their practicality. Some subsequent methods (5) 4 turn to recover the geometrics and physical properties from observations in decoupled manner. Specifically, these methods first extract object geometries by making use of stereo observations or dynamic neural reconstruction [T]] from RGB video sequences, and then perform simulation directly the point clouds after the tetrahedral mesh conversion. While these methods introduce explicit geometries to guide the estimation of physical properties, the noisy reconstruction results usually lead degraded system identification performance.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Recently. PAC-NeRF 12] integrates neural radiance fields (NeRF) [3] with a continuum dynamic model to tackle the above problems. The object geometries and physical properties are captured in unified framework. Despite its effectiveness, this method possesses two limitations. Firstly, the implicit shapes represented by NeRF often lead to inferior geometries, which might cause inaccurate trajectories during simulation. Secondly, PAC-NeRF renders the novel views of deformed objects based on the appearance radiance field reconstructed from the static scene, which might introduce texture distortion, particularly when objects undergo significant deformations, resulting in discrepancies between the rendered and the observed images [4..
To address these limitations, this paper proposes novel hybrid solution based on 3D Gaussians [15] [6] and material point method (MPM) [17] The core strength of this work is that we make use of both 3D shapes from dynamic 3D Gaussian reconstruction and 2D shapes rendered by the Gaussian-informn continuum for physical property estimation.
To generate more precise shapes to reason physical property, we first propose a factoie- dynamic 3D Gaussian network to conduct dynamic scene reconstruction. We then extract the continuum from the recovered 3D Gaussians at each frame by leveraging a coarse-to-fine filling strategy to generate the density field of the object progressively. The resulting density fields can be used to sample continuum particles for simulation and extract object surfaces as 3D-shape supervision in physical property estimation. To eliminate the appearance distortion caused by large deformation in PAC-NeRF, we further assign Gaussian attributes to the continuum particles where the opacity and scale attributes are evaluated from the density field. Such Gaussian- -nneromedd continuum are able to render object masks during simulation, which can be regarded as 2D-shape representation to guide the estimation and effectively avoid using inferior rendering results for learning physical properties.
To demonstrate the superiority of the proposed method over other baselines, we conduct three types of experiments, including evaluations of physical properties, dynamic reconstruction, and future state simulation. We also demonstrate a real-world application in digital twins and robotic manipulation, showing the applicability of the proposed method in real-world scenarios.
Our contributions are summarized as follows.
We propose novel hybrid pipeline that takes advantage of the 3D Gaussian representation of the object to both acquire 3D shapes and empower the simulated continuum to render 2D shapes for physical property estimation.
We propose a novel dynamic 31) Gaussian framework with motion factorization to achieve more precise dynamic reconstruction. We also propose a coarse-to-fing filling strategy to generate the density field of the object, which can be utilized to extract object surfaces and obtain Gaussian- informed continuum particles.
Extensive experiments show that our pipeline attains state-of-the-art performance on existing benchmarks with wide range of metrics, We also present a real-world demonstration to show the efficiency of the proposed method.
2
Related Work
Dynamic reconstruction. Reconstructing dynamic scenes from monocular or multi-view video(s) is long-standing problem in the computer vision community [19 20). Previous works exploit neural implicit representation (21 (2) for non-rigid reconstruction. These methods either reconstruct the
maintain a canonical shape and model the deformation While effective for novel view synthesis, these methods often require extensive training time and can result in noisy deformations owing to the implicit representation, which may compromise the utility of the recovered geometries for physical property estimation [1]. Recent progress in 3D Gaussian Splatting (3DGS) technique (T5) stands out to be prevalent method for 3D reconstruction and novel view synthesis because of the abilities of explicit shape modeling and extremely fast view rendering. Similar to non- rigid NeRF, many follow-up works extend the 3DGS into 4D by treating each frame separately [28] or decomposing scene into a canonical 3D Gaussian point cloud and a deformation model that warps the canonical shape into specific scene [16 29 30). In this paper, we draw upon these prior studies [16] 29] and
scene in a frame-wise manner 23 24
with a neural network 25 26
propose a novel motion-factorize dynamic 3D Gaussian network to achieve better performance on reconstruction and novel view synthesis.
2

---

System identification. Understanding the physics laws of the 3D world is beneficial for simula- tion 33 and manipulation 2 38). However, unveiling these properties from visual information is an extremely difficult task due to the ambiguity introduced by incomplete obser- vation and the high degrees of freedom of the scene. Early works [39, 40] study the problem by learning physical properties via interactions. With recent improvements in differentiable physics simulation [17 [8 41H45], many methods turn to evaluate the physical properties by comparing the rendering results with 2D ground truth given the prior knowledge about the object geometry. VEO [S] presents a differentiable simulator to learn patterns from 4D reconstruction and force-displacement measurements. Another approach [4] eliminates the dependence of captured forces by proposing an iteration framework between deformation tracking and parameter optimization. While these methods demonstrate promising results, the inferior reconstruction might lead to degraded performance, and the assumption of elastic material restricts the applicability. PAC-NeRF [12] instead proposes single framework to recover both the unknown geometry and physical properties of deformable objects from multi-view video sequences. However, the inferior geometries and blurry rendered images might have detrimental effects on physical property reasoning. In this work, adopt MPM as simulation framework following the approach used in PAC-NeRF due to its ability to simulate a variety of materials (6 46.48). Unlike previous approaches, we utilize dynamic 3D Gaussians to reconstruct explicit 3D geometries and generate simulatable continuum particles. Furthermore, we enhance the particles with Gaussian attributes, facilitating the rendering of 2D shapes, and thereby improving physical parameter estimation.
3 Preliminary
In this section, we briefly review the core idea of 3D Gaussian Splatting (3DGS) 15 and introduce its point-based alpha blending to render depth maps and foreground masks. Typically, 3DGS utilizes 3D Gaussians, each defined by a central point Ho. covariance matrix Σg. density value ø,, and color attribute to efficiently render images from specific viewpoints. Each point is denoted as
G(x) = exp( 2 (ar mO) 2 (x (40));
where Eo can be factorized as Eg So Rg. in which Ro a rotation matrix represented by quaternion vector To E R4, and So is a diagonal scaling matrix characterized by 3D vector so E 18. If we consider isotropic Gaussian representation, the scaling matrix can be written as spl, where is scalar and / is the identity matrix. When performing splatting, the 3D Gaussians are projected into 2D with the covariance matrix defined as Σn JWE, WTTTT JT, where J is the Jacobian of affine approximation of the projective transformation matrix. The rendered color I(u) with its foreground mask A(u) at pixel u are then evaluated by integrating N ordered slatted Gaussians via the point-base alpha blending. Since the depth of each Gaussian point at a specific view can be obtained according to its transformation matrix, we can further render the depth map D using the same blending method [16 50)
49 and W is the viewing transformation
I(u)= TTaac
A(u)= Σ TTa
D(u) > T,a,d,
where T; = -1; is the accumulated transmittance, as is the probability of termination at point i, and de is the depth of the Gaussian point at the specific view.
Method
4.1 Problem Definition and Overview
In this work, we aim reconstruct the geometries and the physical properties of various object types from multi-view videos. Formally, given set of video sequences (V/ 1...n} with moving object and the corresponding camera extrinsic and intrinsic parameters {(T,, K;)li = 1...), the goal of this task is to recover the explicit geometries of the object represented by continuum particles P(t) and its corresponding physical parameters (e.g., Young's modulus E and Poisson's ratio v for elastic objects). We follow the assumption in PAC-NeRF and PhysGaussian [12 51 that the object types (e.g., elastic, granular, Destonnian/oon-Newtooiia plastic) are known and the physical phenomenon follows continuum mechanics [[7 52
3

---

G Gaussian- Gaussian-informae Continuum Generation
Motion- factorized Dynamic Gaussian
geterstion strilegy
6
Network
Caussaan attin lEt octtion
(a)
Multi-view Video Capture
Gaussians
Continuums
Surfacos
(b)
Physical Parameter
Material Point
Method (MPM)
Loss
P(O)
forward
Simuleted Continuums
Simulated Surfaces and Masks
Target Surfaces and Masks
word
Given a series multi-view images capturing a moving object, the motion-factorized dynamic 3D Gaussian network is trained reconstruct the dynamic object as Gaussian point sets across different time states. From the reconstructed results, we emplay coarse-to-fine strategy to generate density fields recover the continuums and extract object surfaces. The continuum is endowed with Gaussian attributes allow mask rendering. (b) Identification: The MPM simulates the trajectory with the initial continuum P(0) and the physical parameters The simulated object surfaces and the rendered masks are then compared against the previously extracted surfaces (colored in blue) and the corresponding masks from the dataset. The differences are quantified to guide the parameter estimation process. (c) Simulation: Digital twin demonstrations are displayed. Simulated objects (colored by stress increasing from blue red), characterized by the properties estimated from observation, exhibit behavior consistent with real-world objects.
Figure Overview. (a) Continuum Generation:
The overview of the proposed pipeline is illustrated in Fig. 1 which consists of three modules: motion-factorized dynamic 3D Gaussian network (Sec. 4.2) for 4D reconstruction of the object, coarse-to-fine density field generation strategy (Sec. 4.3) for continuum generation, surface extraction. and Gaussian attribute assignment, and a procedure (Sec. 4.4) showing how we leverage Gaussian- informed continuum and extracted surfaces to estimate physical properties.
4.2
Motion-factorized Dynamic 3D Gaussian Network
Our dynamic 3D Gaussian network follows existing frameworks [16 29 30 that simultancously maintain a canonical 3D Gaussian set and a deformation field modeled by a neural network to warp the canonical shape into object states at specific times. The core idea of this pipeline, presented in Fig. 2 is that the motion of every point in the object can be decomposed into a small range of motion bases.
Architecture. We first factorize the entire motion into Nm, bases that are modeled by a fully connected neural network, where every basis shares common backbone except the final layer. The output of each basis consists of the deformations at position dtr(() E IR and at scale ds; (t) E R. To model the exact deformation for each position, next propose lightweight coefficient network that maps the positions at canonical space with specific time to their corresponding motion coefficients w(Po, t) E N Therefore, the deformed position and the scale for each Gaussian point are evaluated by the linear combination of the motion basis according to the motion coefficients:
No
Nm w tt)ss
p(t)= = 40 + Σ
wi t)dpa(t)
s(t) =
In this work, we regard the Gaussians as isotropic kernels, which has been demonstrated as an effective way to simplify the model and better reconstruct the scene 66. 53). We should note that

---

Motion-factorized Dynamic 3D Gaussian Network
y(t)
Motion Network
d Hey
d Payy
di
ds,
ds
dsy
Sp
Coefficient Network
W,
714))
Multiply
Add
Figure 2: The pipeline of the proposed dynamic 3D Gaussian network. The motion network backbone consists 8 fully connected (FC) layers. The output the motion block is fod to N.. heads to generate motion residuals. The coefficient network contains 4 FC layers.
54 also perform motion decomposition modeling, pipeline shows two major differences: 1) instead of modeling each basis with an independent neural network, our module shares common backbone. Our key observation is that for reconstructing a dynamic object, all points on the object should follow a similar moving tendency, and the final heads of the neural network sufficient model the details of different parts of the object; 2) to increase the ability to fit high rank of the dynamic scene [16], we model the motion coefficients as time-variant variables rather than constant Gaussian attributes 29
although previous works 29
Optimization. We employ the same setting in [6] to train our pipeline. Concretely, the canonical 3D Gaussians initialized with points randomly sampled from the given bounding box of the scene. We start training the deformation network after 3,000 iterations of warm-up for the 3D Gaussians. Similar to previous works [16] 29]. we optimize the pipeline by computing the L1 norm and Structural Similarity Index Measure (SSIM) between the rendered image / and the ground truth image Moreover, since large scales may lead to inaccurate reconstructed shapes [55), we thus perform L1 norm the scale attributes of all the points recover more fine-grand shapes of the object. Therefore, the overall loss function is defined as:
Lgs (I, + A (7, ř) + (
(4)
where A and A2 are balancing hyperparameters. More in-depth analysis of the proposed pipeline. including implementation details and effects of scale regularization, presented in Appendix A.
4.3 Gaussian-informed Continnum Generation
Coarse-to-fine density field generation. Since the reconstructed Gaussian particles are served for rendering only, meaning that they are not evenly distributed the objects, they cannot be directly used for simulation [51]. Therefore, we propose a novel coarse-to- to-sse- filling strategy to iteratively generate density fields of the object based on the reconstructed Gaussian particles from Equ. Eqn the internal particles filtered by the rendered depth maps. The proposed strategy is presented in Alg. The implementation details and visual results illustrated in Appendix A.2
Concretely, the internal particles, initialized by uniform sampling from the bounding box of Gaussian particles, are filtered by projecting the particles to various images to compare the projected depth with rendered depth values (lines 1-6 in Alg. T.. The resulting particles can roughly represent the shape of the object. However, denoted in Eqn. the rendered depth maps evaluated in an accumulated manner, making them less precise in representing the object surface.
Therefore, We employ coarse-to- to-fie filling strategy by iteratively upsampling the density field and reassigning the densities on the indices computed from both the Gaussian and internal particles (lines 8-16 in Alg. J. Fig. 33 provides sketch illustration of the proposed strategy. Specifically, due to the large grid size the initial stage, the object is completely inside the voxels with high densities. Next, sequentially perform upsampling (line 10), mcan filtering (line 13), and reassigning the
5

---

(a)
(b)
(c)
(d)
(e)
Figure 3: Sketch illustration the coarse-to-fin filling strategy. Gaussian and internal particles are depicted in green and blue, respectively. (a) Voxels containing particles are assigned high densities. (b) Following the upsampling and smoothing of the field, densities near boundaries become blurred (indicated in light yellow). (c) The particles again used to correct the voxels that contain particles with high densities. (d) and (e) repeat the previous operations to achieve a more detailed shape.
field (line 14) at each iteration. The first two operations produce more fin--gaiied shapes, and the reassigning operation ensures high densities at the surface to avoid over-erosion caused by the first two steps. Finally, the continuum particles with the corresponding object surfaces can be extracted by thresholding the density field (lines 16- 17 in Alg.
Algorithm Pscudo code for coarsc-to-fine filling Input:
Gaussian particles at time t: PP((t) = {(u(t), s(t(t),
pairs of camera extrinsic and intrinsic parameters: (J]i I...::: parameters: grid size Aur; number of upsampling steps n. thresholds thmins Aomin's Output:
Continuum particles P(t) and corresponding surface Š(t);
1: Randomly sample an initial particle set P.. from the bounding box of {n(t)}:
2: for i + 1, do
3: D: GausssinnSllatting@Po
Tú, Ky):
render depth map at view i obtain image indices and depths of P.. at view i filter out particles that outside the object
4: (uin, Vin), diy +- Proj( Tú, K:):
Pin P...D.(ua Via) $ dun];
end for
Initialize the zero-value density field F(t) with Ar and the bounding box of (n(t));
8:
for 1, do if of then
9:
10:
F(t) + Tritinear ItterpoaatoorrrFF1.. F(Olp. q.r] 1, where p. Discretize(Pa.. [n(O)):
b upsample F(t) with scale factor 2
11:
12: end
13: Fit) <- MieaaniiteiiiggFFtt
14:
F(Ol..g.)] 1, where p.y. r.r +- Discrelize(Paa u (p(O)):
15: end for
16: P(t) <- GetPosition (thmmn $ F())):
17: 5(() GetPosition (thomi S F(t) S thtous);;
Caussian-informel continuum. In PAC-NeRF, the particles equipped with appearance features to enable image rendering for the continuum at different states. We can also achieve this function by treating the particles Gaussian kernels and re-train the particles using the visual data. However, this process is cumbersome and will also face the same issue PAC-NeRF where distorted RGB images will be rendered when large deformation occurs. Therefore, instead of injecting appearance attributes, we opt to assign density and scale attributes to the particles where the densities originate from the density field, and the scale attributes can be directly obtained by the field grid size, The Gaussian-informed continuum is defined as a set of triplets:
P (6ū 8Az, oF))
(5)
where & P. SAz = Ar/2"., and or F(Discrelize(() (we neglect in the notation for simplicity). Therefore, only render object masks 2D shape surrogates for supervision.

---

4.4 Geometry-aware Physical Property Estimation
With the Gaussian-informel continuum at initial state (0) and the extracted surfaces S(t) in place, we can employ MPM to perform simulation on the continuum and evaluate the difference in terms of both the 3D and 2D shapes. Concretely, after rollout by MPM given the current estimation of physical parameters, we obtain a trajectory P(t) with corresponding object surfaces (t). We thus can render object masks over the trajectory. Then the loss of the current rollout can be computed as:
Cppe
1 Σ (cc)(((t)))((tt) 1=1
Σ ((A,(t)) j=1
(6)
where LCD and L1 are chamfer distance and L1 norm respectively, S(t,) denotes the simulated surface at time 44 (1,) is the rendered mask at view j. and (44 represents the object mask of the image extracted from video V time t,. Due to the differential property of the simulator, the evaluated loss is used to optimize the target physical parameters O
5 Experiments
Datasets. To thoroughly assess our proposed method, employ two sources of data introduced by PAC-NeRF [12 and Spring-Gan Gaus (6. Concretely, PAC-NeRF contributes two synthetic datasets generated by MLS-MPM framework [18] Each object in both datasets includes RGB images from 11 distinct viewpoints, with approximately 14 frames per viewpoint. The datasets feature range of materials, including elastic and plastic objects, granular media, and both Newtonian and non- Newtonian fluids. The first dataset contains 45 cross-shape objects with different initial conditions and ground-truth values of physical properties, while the second one consists of 9 objects with different shapes. The interpretation of the physical parameters is listed in Appendix A.9 A. Spring-Gaus generates a synthetic dataset of elastic objects and collects a real-world dataset containing both static and dynamic scenes. The synthetic data contains 30 frames in each of 10 viewpoints. While the real-world data only contains viewpoints for each object the dynamic scene, captures 50-70 images from various viewpoints for the static scene. Moreover, we follow previous works [12 6 and use the off-the-shelff matting [56] or segmentation [57] techniques to obtain object masks.
Baselines. For dynamic reconstruction, we compare with PAC-NeRF and the current state-of-the-art deformable 3D Gaussian method DefGS [16] the PAC-NeRF synthetic dataset. More comparison of our dynamic 3D Gaussian pipeline on other widely-used datasets such as D-NeRF [55] is presented in Appendix [A..3] For system identification, performance using the two datasets introduced in PAC-NeRF. To further demonstrate the precision of the proposed method in terms of geometry recovery and future prediction, we perform experiments on the Spring-Gaus synthetic dataset and compare the results with PAC-NeRF and Spring-Gaus
employ PAC-NeRF as the baseline and evaluate the
Metrics. The evaluation metrics in the experiments include 1) Chamfer Distance (CD), with units expressed in 10 2) Earth Mover's 's Distance (EMD): 3) Peak Signal-to-Noiss Ratio (PSNR); 4) Structural Similarity Index Metric (SSIM) [58); and 5) Mean Absolute Error (MAB), with values scaled by a factor of 100. The first two metrics are used to evaluate discrepancies between the reconstructed and ground-truth point clouds. PSNR and SSIM are leveraged the Spring- Gaus dataset to validate the precision of future state prediction. We compute the mean absolute error for the evaluation of physical property estimation.
5.1 Evaluation on PAC-NeRF Synthetic Dataset
Comparison on dynamic reconstruction. In this experiment, first perform dynamic Gaussian reconstruction on the cross-shaped object dataset using DetGS and our proposed method, respectively. We then employ the same filling strategy on the reconstructed Gaussians at each time state to generate the continuum, which is regarded as the final recovered geometry of the object and used to make comparisons with the oracle shape to compute CD and EMD. Since PAC-NeRF jointly recovers both geometries and physical parameters, we use the final estimated results to generate the trajectory for evaluation.
The results, reported in Tab. [ show that our method outperforms the baselines on both metrics and achieves more precise reconstruction performance most objects. Specifically, we find that
7

---

(a)
(
Figure Comparison between rendered and ground-truth images. (a) Rendered RGB images by PAC-NeRF, (b)
Rendered masks by our method. (c)-(d) Ground-truth RGB images and masks. The mask-hased supervision can
introduce fewer discrepancies compared with the RGB-based guidance when the estimated shapes are correct.
Table 1: Dynamic Reconstruction on PAC-NeRF Dataset
Metries
CD 4
EMD
Methods
PAC-NeRF 12
DefGS [T6]
Ours
PAC-NeRF 12
DefGS (T6
Ours
Newtonian
0.277
0.269
0.243
0,027
0,027
0.025
Non-Newtonian
0.236
0.216
0.195
0.025
0.024
0.022
Elasticity
0.238
0.191
0.178
0.025
0.022
0.02
Plasticine
0.429
0.213
0.196
0,029
0,024
0.022
Sand
0.212
0.281
0.25
0,025
0.028
0.025
Mean
0.278
0.234
0.212
0.026
0.025
0.023
the NeRF representation used by PAC-NeRF usually leads overly large shape generation. While DefGS performs well on elastic objects, its performance degenerates when modeling objects with large deformations. such as granular media and fluids. Our method can better handle these objects due to the flexibility of trajectory representation.
Comparison system identification.
We evaluate the performance of system identification of the two datasets proposed by PAC-NeRF. For the first dataset, we compute the MAE of the parameters for each type of object. To demonstrate the ef- fectiveness of the 2D shape representation, we also conduct experiments on the second dataset by only using masks for supervi- sion on our method, namely "Ours*", For the second dataset, we execute 10 times of our method with different random seeds for each object instance and report the mean value of the estimation results. The training details are illustrated in Appendix
Table System identification performance PAC-NeRF cross-shaped object Datasct
Type
Parameters
PAC-NeRF
Ours*
Ours
(0g1o(p)
11.616.60
1.53
1.53
Newtonian
16.7+5.37 16.0#22.4
19.8+19.2
0
0.86#1.45 0.20+00.88
0.20
1pg1o(p)
24.1::21.9
32.9114.6
13.5
(K)
44.0:1-26.3
17.7+:20.2 12.9-16..8
Non- Newtonian
logia (TY)
5.09-17.41 3.74 4.80
log1o
28.7+23.3
34.9#22.1 40.7=24.6
0.29:10.13
0.6810022 0.1910.09
A.3
logia F)
3.02+:3.72
3.27+4.11 2.43+:3.29
Flasticity
4.35+55.08 3.10+:2.00 2.52+2..3
The results, reported Tab. 2 and Tab. show that the proposed hybrid pipeline can achieve more accurate estimation a wide range of entries and objects, which demonstrate the effectiveness of the geometry-aware guidance. Fig. visual- izes the RGB images rendered by PAC- NeRF and the masks rendered by our method. We can see that when large de- formation occurs, the rendered RGB image
1
0.50-10.23
0.78.10.26 0.82-1032
log E)
83.8-6884
28.1422.4
25.6+29.4
logio (T))
11.2+14.5 1.24+0.90
1.67+11.1
Plasticine
18.9+15.7 10.2+5.34
9.59+5..0
0.56+0.17
0.13+00.04
0.22
Oviic
4.89 4.21
4.18
Sand
0.21
0.24+00.08 0.17=:0.05
becomes distorted, while the rendered mask can effectively reduce such effect and get better perfor-

---

Table 3: System Identification Performance PAC-NeRF Dataset
PAC-NeRF
Ours
Ground Truth
Droplet
2.09 x 102,
1,08 10 10"
2.01 x 10a, x= 0.18 10P
200 10" #
Letter
A 6
J 95.05.
1.00
100, x 102
Cream
# 1.21 x 10% == 1.57 x 10",
==
1.03 10" R= 1.48 x 10".
# =
10" x== 10"
- 3.16 x 10". 5.6
2.98 x 10". 6.6
x 10",
Toothpase
10?, ==
== 4.19
10
109
10 10P.
Ty = 228,288 y 9.77
TV 1- 9.1
226.
by =
200.
Torus
E = 1,04 x 10F 0.322
E
0.99
10" 2 0.295
E 10% 0.3
Bird
8 2.78 10". 0.273
3.08
10
0.284
10
Playdoh
E 3.81 10f.
0,272 =168
10
5
1.58
10" 0= 0.322 TV = 1.56 x 10
2= x 10%, v= 0.3, by 1.1 10 6= 10" 0.3. 3.85 10
Cat
== 1.61 x 107, 0.203, 3.57
10
E
0.98 x 10" > 0.296. = 3.76 x 10
Tropby
9'... 36.1"
P/-- 38.0"
8% 40
Table 4: Future State Simulation Spring- Gaus Synthetic Dataset
torus
cross
cream
apple
paste
chess
banana
Mean
Spring-Gaus
2.38
1.57
2.22
1.87
7.03
2.59
18.48
5.16
PAC-NeRF
12
2.47
3.87
2.21
4.69
37.7
8.2
66.43
17.94
Ours
0.75
1.09
0.94
0.22
2.79
0.77
0.12
0.95
SSIM PSNR~ EMD.
Spring- Gaus
6
0,087
0.051
0.094
0,076
0,126
0.095
0.135
0.095
PAC-NeRF
12
0.055
0.111
0.083
0.108
0,192
0.155
0.234
0.134
Ours
0.034
0.058
0.050
0.030
0.096
0.059
0.017
0.049
Spring- Opring-Gaus
6
16.83
16.93
15.42
21.55
14.71
16.08
17.89
17.06
PAC-NeRF 12
17.46
14.15
15.37
19.94
12.32
15.08
16.04
15.77
Ours
20.24
30.51
19.15
26.89
16.31
18.44
29.29
22.98
Spring- Doring-Gaus
0.919
0.940
0.862
0.902
0.872
0.881
0.904
0.897
PAC-NeRF 12
0.913
0,906
0.858
0.878
0.819
0.848
0.886
0.870
Ours
0.942
0.939
0.909
0.948
0.894
0.912
0.964
0.930
mance. By leveraging both 3D and 2D shape guidance, our method obtains the best results on most entries. More qualitative results are available the supplementary video.
5.2 Evaluation on Spring-Gaus Synthetic Dataset
Comparison on future state simulation. To further demonstrate the performance of our proposed method, we follow the setting in Spring-Gaus [6] that uses the first 20 frames training data and the subsequent 10 frames for evaluation. Concretely, we first perform system identification based on our method and then use the estimated physical parameters and the continuum to simulate trajectory that includes the states of the 30 frames. Therefore, we can compute CD and EMD between the simulated continuum and the ground-truth point cloud. Since we know the exact position of the continuum at each time state after estimation, we can assign time-invariant Gaussian attributes by training Gaussians on the continuum using the first 20 frames of RGB images, which enable image rendering at novel views and states. Therefore, we can compute PSNR and SSIM at any time state.
The results of future state prediction are presented in Tab. 4 and the results of reconstruction on the training states are reported in Appendix [A.4] We observe that our method significantly outperforms the baselines of our method for both geometry recovery and system identification. The results of PSNR and SSIM show that leveraging dynamic visual data to train the Gaussian attributes on the continuum improves rendering quality. This further reveals that the generated trajectories are precise such that the particles consistent to contribute to the rendering for the same region of the object at different time states.
CD and EMD metrics over almost all object instances, which shows the superiority
5.3 Real-world Application: Digital Twins in Robotic Grasping Scenario
To demonstrate the efficacy of the proposed method in real-world scenarios, we perform system identification on the real-world dataset collected by Spring-Gau Gaus 66. as shown in Fig. 5 Since the real-world dataset consists of static and dynamic scenes for each object, we follow the procedure introduced by Spring-Gaus to progressively 1) reconstruct Gaussian set of the object from the

---

Syring Gaus
Outs
Real world
Idantification
Future simulltioon
Grasp demonstraaions
Figure 5: Real-world application. Left: Identification and future state simulation. Right: Grasping simulation.
The the simulated object indicated blue (low) red (high). The gripper widths from top bottom are set to 6cm, 4.5cm, and 3.5cm, respectively.
static scene, 2) transform the static Gaussian set to the initial state of the dynamic scene based registration network similar as iNeRF [9], and 3) perform system identification from the dynamic observation by our method "Ours*" due to the lack of sufficient images for dynamic reconstruction. Subsequently, we establish robotic platforms in both simulated and real-world environments, each equipped with URI 10 robot arms configured identically. We then execute grasp attempts both the reconstructed objects with the estimated properties in the simulation and the corresponding real- world objects under the same configuration. The results of more objects, and more details about the training and the experiment setting are presented Appendix A.5] From the results shown Fig. [ we see that our method demonstrates its capability to effectively model the deformation experienced by the objects upon impact with a surface. Furthermore, by applying identical gripper forces to both the simulated and real-world versions of the objects, observe similar deformation behaviors. This consistency in deformation under identical conditions supports that the estimated physical parameters closely mirror the real-world properties of the objects.
Conclusion and Limitations
This paper proposes a novel solution that leverages the 3D Gaussian representation of objects to acquire explicit shapes while concurrently enabling the simulated continuum to render 2D shapes facilitate the estimation of physical properties. A novel factoried dynamic 3D Gaussian framework is proposed to reconstruct precise dynamic scenes. Object surfaces and Gaussian-infomme continuum are obtained by utilizing the proposed coarse-to-fine density field generation strategy. Extensive experiments demonstrate the efficacy and applicability of our method.
Despite the performance we achieve, this method still suffers from limitations, such as the assumption of continuum mechanics, the requirements of multi-view images with known camera poses, and the need for prior knowledge of object constitutive models. Integrating the pose-free method [60] or generalized constitutive [6] model with our method will be an interesting direction for future work.
From the perspective of application, while this method can yield accurate estimations, it may pose risks for fragile objects, as the interaction required for property inference could potentially cause damage. Moreover, the computational demands of our framework are substantial which require at least 1.5 hours to simultaneously recover both the geometry and physical properties of each object. Future work could explore leveraging multi-model large language models [62) and large reconstruction models 633 66 to facilitate the recovery process.
7 Acknowledgements
This research was supported by the Research Grant Council of the Hong Kong Special Administrative Region under grant number 16212623. We thank Licheng Zhong for providing us with details about real data collection and links for purchasing objects for real-world experiments.
10

---

References
[1] Hang Yin, Anastasia Varava, and Danica Kragic. Modeling, learning, perception, and control methods for deformable object manipulation. Science Robotics, 6(54))cabd8803, 2021.
[2] Haochen Shi, Huazhe Xu, Samuel Clarke, Yunzhu Li, and Jiajun Wu. Robocook: Long-horizon clasto-plastic object manipulation with diverse tools. In Proceedings of the Conference on Robot Learning (CoRL), pages 642-660. PMLR, 2023.
[3] Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, and Jiajun Wu. Robocraft: Learning to see, simulate, and shape plasto-plastic objects 3d with graph networks. The International Journal of Robotics Research (IJRR), 43(4):533-549, 2024.
[4] Bin Wang, Longhua Wu, KangKang Yin, Uri Ascher, Libin Liu, and Hui Huang. Deformation capture and modeling of soft objects. ACM Transactions Graphics (TOG), 34(4):1-12, 2015.
[5] Hsiao-yu Chen, Edith Tretschk, Tuur Stuyck, Petr Kadlecek, Ladislav Kavan, Etienne Vouga, and Christoph Lassner. Virtual elastic objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15827-15837, 2022.
[6] Licheng Zhong, MloggXXing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mase mass 3d gaussians. arXiv preprint Attl:2403309433.. 2024.
[7] Matthias Müller and Markus H Gross. Interactive virtual materials. In Graphics interface. volume 2004, pages 239-246, 2004.
[8] Miguel Jaques, Michael Burke, and Timothy Hospedales. Physics as aas-invrree inverse-graphics: Unsuper- vised physical parameter estimation from video. In Proceedingy of the International Conference on Learning Representations (ICLR), 2020.
[9] Pingchuan Ma, Tao Du, Joshua Tenenbaum, Wojciech Matusik, and Chuang Gan. Risp: Gendering-invarianl state predictor with differentiable simulation and rendering for cross- domain parameter estimation. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[10] Pingchuan Ma, Peter Yichen Chen, Bolei Deng, Joshua B Tenenbaum, Tao Du, Chuang Gan, and Wojciech Matusik. Learning neural constitutive laws from motion observations for generalizable pde dynamics. In International Conference on Machine Learning (ICML), pages 23279-23300.
PMLR, 2023.
[11] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference Computer Vision (CVPR), pages 12959-12970, 2021.
[12] Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chen- fanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for coometry-agnostic system identification. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
[13] Cheng Sun, Min Sun, and Hwann-Tzone -zong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5459-5469, 2022.
[14] Yutao Feng, Yintong Shang, Xuan Li, Tianjia Shao, Chenfanfu Jiang, and Yin Yang. Pie- nerf: Physics-based interactive elastodynamics with nerf. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[15] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 42(4):1- 14, 2023.
[16] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF International Conference on Computer Vision 2024.
11

---

[17] Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The material point method for simulating continuum materials. In Acm siggraph 2016 courses. pages 1-52. ACM New York, NY, USA, 2016.
[18] Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG), 37(4):1-14, 2018.
[19] Li Zhang, Brian Curless, and Steven M Seitz. Spacetime stereo: Shape recovery for dynamic scenes, In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), volume 2, pages II-367. IEEE, 2003.
[20] Richard A Newcombe, Dieter Fox, and Steven M Seitz. Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time. In Proceedings of the IEEE/CVF International Conference on Computer Vision (CVPR), pages 343-352, 2015.
[21] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[22] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems (NeurIPS), 34:27171-27183, 2021.
[23] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5521-5531, 2022.
[24] Yiming Wang, Qin Han, Mare Habermann, Kostas Danilidis, Christian Theobalt, and Lingjie Liu. Neus2: Fast learning of neural implicit surfaces for multi-view reconstruction. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 3295-3306, 2023.
[25] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10318-10322 2021.
[26] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Brauiln,ruuala,, and Steven M Seitz. Hypernerf: a higher-dimensionall representation for topologically varying neural radiance fields. ACM Transactions Graphies (TOG), 40(6) 2021.
[27] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 130-141, 2023.
[28] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In Proceedings of the International Conference on 3D Vision (3DV), 2024.
[29] Agelos Kratimenos, Jiahui Lei, and Kostas Daniilidis. Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting. arXiv preprint 1rX1v:2312.00112,
2023.
[30] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[31] Junbang Liang, Ming Lin, and Vladlen Koltun. Differentiable cloth simulation for inverse problems. Advances in Neural Information Processing Systems (NeurIPS), 32, 2019.
12

---

[32] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[33] Priya Sundaresan, Rika Antonova, and Jeannette Bobgl. Diffcloud: Real-to-sim from point clouds with differentiable simulation and rendering of deformable objects. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11828-10835 IEEE, 2022.
[34] Yifei Li, Tao Du Kui Wu, Jie Xu, and Wojciech Matusik. Diffcloth: Differentiable cloth simulation with dry frictional contact. ACM Transactions on Graphics (TOG), 42(1):1-20, 2022.
[35] Jinxi Li, Ziyang Song, and Bo Yang. Nvfi: Neural velocity fields for 3d physies learning from dynamic videos. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
[36] Xiao Liang, Fei Liu, Yutong Zhang, Yuelei Li, Shan Lin, and Michael Yip. Real-to-sim deformable object manipulation: Optimizing physics models with residual mappings for robotic surgery. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), pages 15471-15477. IEEE, 2024.
[37] Dongzhe Zheng, Siqiong Yao, Wenqiang Xu, and Cewu Lu. Differentiable cloth parameter identification and state estimation in manipulation. IEEE Robotics and Automation Letters (RA-L), 2024.
[38] Yi-Ling Qiao, Alexander Gao, and Ming Lin. Neuphysics: Editable neural geometry and physics from monocular videos. Advances in Neural Information Processing Systems (NeurIPS), 35: 35:12841-12854, 2022.
[39] Barbara Frank, Rüdiger Schmedding, Cyrill Stachniss. Matthias Teschner, and Wolfram Bur- gard. Learning the elasticity parameters of deformable objects with a manipulation robot. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1877-1883. IEEE, 2010.
[40] Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B Tenenbaum, and Shuran Song. Densephys- net: Learning dense physical object representations via multi-step dynamic interactions. In Proceedings of the Robotics: Science and Systems, 2019.
[41] J Krishna Murthy, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jérôme Parent-I Lévesque, Kevin Xie, Kenny Erleben, et al. gradsim: Differentiable simulation for system identification and visuomotor control. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[42] Moritz Geilinger, David Hahn, Jonas Zehnder, Moritz Bächer, Bernhard Thomaszewski, and Stelian Coros. Add: Analytically differentiable dynamics for multi-body systems with frictional contact. ACM Transactions on Graphics (TOG), 39(6):1-15, 2020.
[43] Eric Heiden, Miles Macklin, Yashraj Narang, Dieter Fox, Animesh Garg, and Fabio Ramos. Disect: A differentiable simulation engine for autonomous robotic cutting. In Proceedings of the Roboties: Science and Systems, 2021.
[44] Tao Du, Kui Wu, Pingchuan Ma, Sebastien Wah, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. Diffpd: Differentiable projective dynamics. ACM Transactions on Graphics (TOG), 41(2):1-21, 2021.
[45] Yiling Qiao, Junbang Liang, Vladlen Koltun, and Ming Lin. Differentiable simulation of soft multi-body systems. Advances in Neural Information Processing Systems (NeurIPS). 34:17123-17132 2021.
[46] Chenfanfu Jiang, Craig Schroeder, Andrew Selle, Joseph Teran, and Alexey Stomakhin. The affine particle-in-cell method. ACM Transactions Graphics (TOG), 34(4):1-10, 2015.
13

---

[47] Yonghao Yue, Breannan Smith, Christopher Batty, Changxi Zheng, and Eitan Grinspun. Contin- uum foam: material point method for shear-dependent flows. ACM Transactions on Graphies (TOG), 34(5):1-20, 2015.
[48] Gergely Klär, Theodore Gast, Andre Pradhana, Chuyuan Fu, Craig Schroeder, Chenfanfu Jiang. and Joseph Teran. Drucker-prager elastoplasticity for sand animation. ACM Transactions on Graphics (TOG), 35(4):1-122 2016.
[49] Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Surface splatting. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 371 2001.
[50] Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-ftow: 4d reconstruction with dynamic 3d gaussian particle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[51] Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[52] Eduardo WV Chaves. Notes on continuum mechanics. Springer Science & Business Media, 2013.
[53] Vladimir Yugay, Yue Li, Theo Gevers, and Martin Oswald. Gaussian-slam: Photo-realistic dense slam with gaussian splatting. arXiv preprint arXiv:2312. 2023.
[54] Kai Katsumata, Due Minh Vo, and Hideki Nakayama. An efficient 3d gaussian representation for monocular/multi-vier dynamic scenes, arXiv preprint arXiv:2311. 2023.
[55] Hanlin Chen, Chen Li, and Gim Hee Lee. Neusg: Neural implicit surface reconstruction with 3d gaussian splatting guidance. arXiv preprint arXiv:23 2023.
[56] Shanchuan Lin, Andrey Ryabtsev, Soumyadip Sengupta, Brian Curless, Steven M Seitz, and Ira seamelmacher-Shhizzemannn Real-time high-resolution background matting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8762 8771, 2021.
[57] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015-4026, 2023.
[58] Zhou Wang, Alan C Bovik, Hamid Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
[59] Lin Yen-Chen, Pete Florence, Jonathan T Barron, Alberto Rodriguez, Phillip Isola, and Tsung- Yi Lin. inerf: Inverting neural radiance fields for pose estimation. In Proceedings of the JEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1323-1330. IEEE, 2021.
[60] Yang Fu, Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A Efros, and Xiaolong Wang. Colmap- free 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[61] Haozhe Su, Xuan Li, Tao Xue, Chenfanfu Jiang, and Mridul Aanjaneya. A generalized constitutive model for versatile mpm simulation and inverse learning with differentiable physics. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(3):1-20, 2023.
[62] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
14

---

[63] Dmitry Tochilkin, David Pankratz. Zexiang Liu, Zixuan Huang, Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from single image. arXiv preprint arXiv:2403.0211 2024.
[64] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. European Conference Computer Vision (ECCV), pages 1-18. Springer, 2024.
[65] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu, Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arXiv:2433.05054, 2024.
[66] Jiale Xu, Weihao Cheng, Yiming Gao, Xintao Wang, Shenghua Gao, and Ying Shan. In- stantmesh: Efficient 3d mesh generation from a single image with sparse-view large reconstruc- tion models. arXiv preprint MrXiv:2404.0719/ 2024.
[67] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. In Proceedings of the International Conference Learning Representations (ICLR), 2015.
[68] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 16632-16642, 2023.
[69] Sara Fridovich-Keil. Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12479-12488, 2023.
[70] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIG- GRAPH Asia 2022 Conference Pupers, pages 1-9, 2022.
[71] Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, et al. Isaac gym: High performance gpu- epu-based physics simulation for robot learning. arXiv preprint arXiv:2108.104 10470, 2021.
[72] Isabella Huang, Yashraj Narang, Clemens Eppner, Balakumar Sundaralingam, Miles Macklin, Ruzena Bajesy, Tucker Hermans, and Dieter Fox. Defgraspsim: Physics- based simulation of grasp outcomes for 3d deformable objects. IEEE Robotics and Automation Letters, 7(3):6274- 6281, 2022.
[73] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM SIGGRAPH Computer Graphics, page 163-169, 1998.
[74] Yixin Hu, Teseo Schneider, Bolun Wang, Denis Zorin, and Daniele Panozzo. Fast tetrahedral meshing in the wild. ACM Transactions Graphics (TOG), (9 2020.
15

---

A Appendix
A.1 Motion-factorized Dynamic 3D Gaussian Network
A.1.1 Implementation details
We employ temporal and positional encoding to the time t and position μo. respectively, to introduce features with various frequencies. Specifically, the encoding module denoted as (x)) (sin(2*<<)), cos(2% x.)) ( 4-1
k=0 where L 10 for both and μoo
All the modules within the proposed network are composed of fully connected layers. The intermedi- ate layers are uniformly designed, featuring both input and ouiput channels configured to 256, and employ ReLU activation. For training, we adhere the protocol established in [T6,, utilizing the Adam optimizer [67] with the same learning rate as specified in [6]] The total number of iterations set at 40,000, with densification and pruning operations conducted every 500 steps until reaching 15,000 iterations. Additionally, the number of motions Nm, is set to 8 for all objects in our network. A and 22 Eqn. are all set to All the experiments are conducted on a single A10 GPU.
A.1.2 Effects of scale regularization
Figure 6: Visualization trophy sequences. Row rendering results from the network trained without scale regularization. Row rendering results from the network trained with scale regularization.
When addressing the deformation of objects such as fluids granular media, the network may struggle to fit transformations accurately due to significant discrepancics between the canonical and target shapes. As compensatory mechanism, the network may employ Gaussians with enlarged scales to mitigate shape distortions during image rendering. This effect visualized the top row of Fig. 6 To rectify this issue, implement scale regularization during network training. which enforces Gaussian kernels to maintain smaller scales. The efficacy of this operation is demonstrated in the second of Fig. [6] where it is evident that scale regularization enables the reconstruction of more precise shapes for rendering.
A.1.3 Evaluation on D-NeRF Dataset
To further evaluate the performance our method in terms of novel view synthesis, we conduct the experiment on the D-NeRF (5 dataset, which is a widely used benchmark consisting of moving items with data captured by monocular camera. compute PSNR on the D-NeRF test set and compare our method with previous dynamic approaches, including Tensor4D 688, K-Planes [69].
Vox 770,, and DefGS [16. The results, reported in Tab. 5 demonstrate the proposed dynamic 3D Gaussian pipeline can also achieve superior performance rendering.
16

---

Table 5: Results of PSNR (1) on D-NeRF 25
Dataset
Hell Warrior
Bouncing Balls
Jumping Jacks
Method
Mutant
Hook
T-Rex
Stand Up
Mean
Tensor4D
68
31.26
29.11
28.63
24.47
23.86
30.56
24.2
27.44
K-Planes
69
24.58
32.5
28.12
40.05
30.43
33.1
31.11
31.41
TiNeu Vox 770
27.1
31.87
30.61
40.23
31.25
34.61
33.49
32.74
DefGS [[6
41.54
42.63
37.42
41.01
38.1
44.62
37.72
40.43
Ours
41.97
42.93
38.04
41.26
37.54
45.32
38.86
40.85
A.2
Gaussian-informed Continnum Generation
A.2.1
Implementation details
In Alg. the number of iterations, denoted Yhus is uniformly set to 4 for all objects. We the initial grid size Ar according to the volume of the object. For most objects, Ar = 0.1, while for small items such as toothpaste in PAC-NeRF dataset, Ar 0.01. The parameters thmin and thmor are set to 0.5 and 0.8, respectively. The resulting particle count ranges from approximately 50,000 to 100.000.
A.2.2 Visualization of coarse-to-fine filling
Fig. visualizes the filling results of our proposed coarse-to- torse-tt--fiee strategy with different numbers of iterations, along with the results from PAC-NeRF and ground-truth shapes. The qualitative results show that our method can generate more accurate shapes compared with PAC-NeRF, which tends to recover over-large shapes. We should note that we cannot recover the cat-shaped object as [1]. though we use the code officially implemented by PAC-NeRF without any modification.
A.3 Training details PAC-NeRF Dataset
The training process is divided into two cub-processes, where we perform system identification after estimating the initial velocity of the object using the first three frames of data. Both processes use Adam optimizer to tune the parameters.
A.4 More Experiments on Spring-Gaus Synthetic Dataset
Besides performing evaluation on the simulated future states in Sec. 5.2 EMD on states existing in the training data, and the results are reported in Tab. [6] obvious to see that method outperforms the baselines by large margin, which further demonstrates the performance of our method in terms of reconstruction and identification.
we also evaluate CD and
Table 6: Dynamic Reconstruction on Spring-Gaud Synthetic Dataset
torus
cross
cream
apple
chess
banana
Mean
Spring- Douing-Gaus 6
0.17
0.48
0.36
0.38
0.19
1.80
2.60
0.85
PAC-NeRF 12
4.92
1.10
0.77
1.11
3.14
0.96
2.77
2.11
Ours
0.13
0.13
0.14
0.15
0.41
0.03
0.17
EMDJ
Spring Gaus
0.040
0.037
0.031
0.033
0.022
0.063
0.052
0.040
PAC-NeRF
12
0.056
0.052
0.041
0.045
0.054
0.052
0.062
0.052
Ours
0.020
0.020
0.019
0.020
0.025
0.036
0.011
0.022
A.5 Experiment Setting for Spring-Gaus Real-world Dataset
Training details. The dynamic scenes in Spring-Gaus 6 contain only three viewpoints, which are insufficient for dynamic 3D Gaussian reconstruction. Conversely, the static scenes incorporate 50 to 70 images captured from various viewpoints. Following the protocol established in Spring- Gaus, we reconstruct 3D Gaussian points from the static scenes using the traditional 3D Gaussian Splatting
17

---

(a) nl 2
(b) n, =3 3
The 4 (d) nu
(e) PAC-NeRF
(f) Oracle
Figure 7: Visualization Coarse-to-fine Filling. (a)-(d) are the filling results by our method with different times of upsampling operations. (e) visualize the point clouds recovered by PAC-NeRF. (T) shows the ground-truth shapes.
18

---

(3DGS) technique [1].. Subsequently, we transform the static Gaussian set to the initial configuration of the dynamic scene, guided by the relative pose between the two scenes. The pose is estimated iteratively based on the discrepancies observed between the rendered images and the actual images at the initial state of the dynamic scene. After pose estimation, we implement our methodology. which leverages only implicit shape guidance, to conduct system identification.
Experimental setting. We conducted grasping experiments using the URIO robotic arm equipped with the Robotiq I 40 dexterous gripper in both simulated and real-world settings, ensuring consistency in the mass of the objects and their grasping poses across both environments, For the simulations, we employed the FEM- based Isaac Gym simulator [7]] for its advanced capabilities in realistically simulating deformable objects [72]. To facilitate the simulation of deformable objects, we apply
the Marching Cubes algorithm 773 to the generated density fields to derive the object meshes.
Subsequently, we utilize fTetWild 74 for the tetrahedralization of these meshes.
More results. Qualitative results of grasp demonstrations on pig and dog objects are shown in Fig.
Figure 8: The stress on the simulated objects is indicated by blue (low) to red (high). The gripper widths from left to right for each object are set 5.5em, 4.5cm, and 3.5em, respectively.
A.6 System Identification Result on PAC-NeRF Dataset
Table System Identification Result on PAC-NeRF Dataset
Noekaain (laa, Grcec 20.
LD 02
Prodiction
21000.80
148398.15
A-
327746-31
-
223551.23
45362118 18
30033.24
2445644
129903.10
A- 10L3S
A.
AT $1.2
J 32.44
225.33
002.5
192095.35
357566-85
Not Chii, Gaesc
P- 18M53-65
A- 1761 16
A--
Mi 11000100
20117 30
H- 35322-55
1500655
SUCRMEAD AS
Prsfiction
1215.55
207622
177746-01
4239
5768.52
Ty 1007
16008.64
28080 20
3604.00
1411.0
24480-41
1000 36
R- 4.4
8- 21:21
12.85
5.56
0 04.55
5
1.07
5- 1.75
14309.25
10051.6
09922700
15006.73
200221 72
52548-45
120005.0
50770.50 50
Goand Trats
- 501066 50
12105403
249656:44
154761355
121439 33
1400E 50
57001 06
33858.99
122995
21.27
22.31
27.24
20.43
112T
21.00
Ulunscy Der: Caros: 12
T7. 0.251
Proliction
023000
02776
0.3527
1.1250
0.2211
0.2757
10006GLS
LLLVORS
122000
132066
0.2076
0L3597
Plaricn dann. Oapi 5 10060
0.2%
D900
124258.02
6332336-43
Prodiction
2.1327
1.3300
21731
0.3717
1333-11
Ty 36873.08
37167.93
Ty 33417.20
12686 94
TV 28773333
23621 08
09507102
E144508-59
Gooand Traia
v- 0000
-- 1566
in 0.17
0.3324
1821.02
ry 64806.21
Ty 26085.20
Ty 29
Ty
41
Ty 14756.01
ST
Ty 2000L57
by 2181 80
24307.29
Gansc
8/.4
Phidiction
e6 33
4716
15,3300
Geozad Touts
15.. = 1217077
%). " 3023783
1... = 20 2810
M... 290543
%.a 42.2562
Each moterial contains 10 instanoes (5 for granuler material) with virious object crientations, inttaal velocities and physical parameters.
19

---

Table 8: Notation of Algorithm
Operator symbol
Explanation
(t(t)
Gaussian partiele set at time
P(t)
Sampled continuum particles time f
S(O)
Sampled surface particles at time 1, S()) C P(()
F(L)
3D Density field at time
Proj
Operation projecting 3D particles into 2D image indices according to the camera parameters
Discretize Positioon
Operation mapping particle positions to voxel indices on the density field Operation returning 3D positions the binary field
A.7 Notation of Algorithm
A.8 Necessity of 2D mask supervision
To evaluate the necessity of 2D mask supervision, we perform system identification 45 cross- shaped object instances the PAC-NeRF dataset by our method but with only object surface supervision. The results reported in Tab. It is obvious to that combining both 2D and 31) shapes as supervision can achieve more accurate performance compared to using 3D shapes only. Therefore, we believe that utilizing 2D mask supervision some extent makes up for the errors introduced by the 3D object surfaces extracted from dynamic 3D Gaussians.
Table 9: System identification with/without mask supervision
Type
Parameters
w/o masks
w/ masks
10)
2.19+2.90
1.53=:1.31
Newtonian
10g1o(k)
24.2422..
14.8
27
0.20+10.08
0.20-10.07
logıo (1)
19.4:1:7.77
13.5 18.2
log (k)
24.0::24.8
12.9:16.88
Non-
(oy)
4.58119.11
4.80
Newtonian
log (n)
49.11:40.5
40.71124.6
1.33++0.54
0.19+00.09
log (
2.85111994
2.43.33229
Elasticity
V
3.97.1..64
2.52.12.33
V
0.22.10.10
0.82.10.32
log 1 E))
25.6+:27.4
25 25.64129.4
Plasticine
log (TY)
9.04+22.37
1.67
1.16.10.00
0.22
frric
2.55++2.03
4.18:
Sand
0.31+:0188
0.17#00.5
A.9 Physical Properties
In this work, we simulate five types materials, including elasticity, plasticine, granular media, Newtonian fluids, and non-Newtonian fluids. Each material exhibits distinct physical properties. We provide brief introduction to the properties of each material.
Elassticty:: The Young's modulus (F) a measure of the stiffness of solid material, quantifying the relationship between stress and strain in material under elastic deformation. The Poisson's ratio (v) describes the tendency of a material to expand or contract along its width when it is stretched or compressed along its length.
The yield stress (ry) is the minimum stress that material requires to transition from elastic deformation to plastic deformation, marking the onset of permanent deformation. Both Young's modulus (E) and Poisson's ratio (v) exhibit characteristics similar to those of elastic materials.
Plasticine:
Granular Media: The friction angle (0fric) is a measure of the inherent resistance of granular material to sliding or shearing, directly related to the angle at which material can be piled without slumping.
20

---

Newtonian fluids: The bulk modulus (R) is measure of a material' 's resistance to uniform compression. quantifying how much it compresses under given amount of external pressure. Fluid viscosity (μ) describes a fluid's resistance to flow, quantifying how much it resists deformation at given rate.
Non-Newtonian fluids: The plasticity viscosity (7)) refers the measure of a viscoplastic material's resistance to deformation, which defines how it behaves under stress beyond its yield point. The bulk modulus (x) and fluid viscosity (1)) are comparable those of Newtonian fluids, while the yield stress (Ty) is akin to that of plasticine.
A.10 Constitutive Models
A constitutive model describes how material responds to stress, strain, other external forces. It defines the material's behavior by relating stress and strain through constitutive equations, which can capture complex behaviors such elasticity, plasticity, and fracture. The MPM simulator is capable of modeling a diverse range of materials by employing various constitutive models. In this work, we have implemented simulations for five distinct types of materials: elasticity, plasticine, granular, Newtonian fluids, and non-Newtonian fluids.
Elasticity. We use the Neo- Weo-Hookeen model, which is common nonlinear hyperelastic model, to simulate the elasticity materials and predict deformations. The Cauchy stress for this model is defined by
Jo = μ (FFT) + [o)) -+|
(7)
where the F is the deformation gradient, J = del (F) and μa, À are the Lamé parameters, which related to the material properties of Young's modulus (E) and Poisson's 's ratio (v) as:
E
EV
#r 2(1 21
X (1+ 2v)
(8)
Plasticine. We use the Saint Conant-Kirchhofd Model (StVK) together with von Mises yield criterion to simulate the plasticine. For this model, the stess is defined as:
Jo F [2p + FTT
where G (FTF- I) is the Green strain. The von Mises yield criterion serves as a tool to assess whether the deformation exceeds the recoverable limit. The deformation gradient will mapped back onto the boundary of elastic region using the following projection:
[F F {
87 S 0
Z(F)
(10)
Uuexple रلंद - V
otherwise
where the 8ry ||²|| Ty obtained by performing Singular Value Decomposition (SVD) on deformation gradient F.
2n log(E) is the normalized Hencky strain. The U, E and V can be
Granular Media. Similar to plasticine, the StVK constitutive model is used to simulate granular media. Brucker-Prager yield criteria [48) is selected as the yielding condition. It is defined as follows:
(d) 2p) > 0,
Tr(E) > O, or 87 = =1[[||] + +a
(11)
2p
where d is the spatial dimension, = gradient return mapping defined by
prie. and fric is the friction angle. The deformation 33 3-3-in Oyrso
UVT
Tr(() > 0 6y s 0, Tr()( 00. otherwise
((F) =
F
(12)
Uexp (c- by
V
[illl
Newtonian Fluid. We adopt the approach used in PAC-NeRF [1], which employs J-based fluid
model combined with a viscosity term to simulate Newtonian fluids. The stress for this model is defined by
1 Jo = "VV VVT) + x(J -
VvT)
(13)
21

---

where jr and y. represent the fluid viscosity and the bulk modulus, respectively.
Non-Newtonian Fluid. We employ the viscoplastic model 447 to simulate non-Newtonian fluids. Although we continue to utilize the von Mises criteria to delineate the elastic region, the presence of viscoplasticity implies that deformation will not be immediately reverted onto the yield surface. It is defined as follows:
by S otherwise
2(F)
(14)
U exp 2p + [T)(
A d (IE
2jeé
(15)
b7
|||||
+
2/At
where d is the spatial dimension. The U, Σ and V can be obtained by performing Singular Value Decomposition (SVD) on deformation gradient F.
22

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiicton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction sections already reflect the paper's contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
23

---

Justification:
The limitations have been discussed in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have tried to include all the details and referenced work for reproduction. We will also release the code of our method.
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
The code is not included for now. But we will release the code to the public soon.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
Dablic//gides/Codeee.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //niss.ccppulic//guus http://////iiiee///deuuubiiiiiii
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
25

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
These details are described in the experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The error bars are reported in the experiments section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
This is described in the appendix section.
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This work conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
27

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The used assets are properly cited in the experiments section.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
28

---

Answer:
[NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "8TY7b31Tc1", "reviewer": "Reviewer_FGDU", "summary": "This paper presents an approach for estimating the geometry and physical properties of objects through visual observations using 3D Gaussian representations. The method employs a dynamic 3D Gaussian framework to reconstruct objects as point sets over time and a coarse-to-fine filling strategy to generate density fields. This facilitates the extraction of object continuums and integrates Gaussian attributes, aiding in rendering object masks during simulations for implicit shape guidance.The extracted geometries are then used to guide physical property estimation through differentiable MPM simulation.", "questions": "1. Table 3: Can you include the ground truth values in the table so that readers understand what to expect?\n2. System Identification: How do you set the initial parameters for the system identification? Are the optimizations sensitive to initial conditions?\n3. Figure 4: Would mask-based supervision fail when the estimated shapes are significantly different from the ground truth? Have you observed any cases where this occurs?\n4. Figure 1a: There is a typo in the figure caption. Change \"caption\" to \"capture.\"\n5. References: It would be beneficial to include references to works on differentiable cloth simulation for system identification and inverse problems, such as \"Differentiable Cloth Simulation for Inverse Problems\" by Liang et al. and \"DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact\" by Li et al.", "limitations": "The authors have discussed the limitations of the method regarding known camera parameters, assumption of known material models, and continuum mechanics. However, I do wonder about the failure cases of the method, if there are any. Understanding specific scenarios where the method does not perform well would provide valuable insights and help guide future improvements.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The experiments in this paper demonstrate improvements over prior works such as PAC-NeRF and Spring-Gaus. The introduction of a novel hybrid framework leveraging 3D Gaussian representations for physical property estimation is straightforward and easy to understand, and experiments have confirmed their effectiveness.  Overall, the paper makes meaningful contributions to the problem of geometry + physical property estimation from multi-view videos.", "weaknesses": "Some of the technical terms are misused, making the paper confusing to read. For example, \"implicit shape representation\" is repeatedly used to refer to the rendered object image masks, but \"implicit\" generally refers to using functions (parametric or neural networks) to represent shapes, where shapes must be retrieved through function evaluations, hence \"implicit\". Please correct this terminology, as the geometries in this work are represented using GS, which are explicit representations. Additionally, some of the results presentations are confusing and could benefit from clearer explanations and more organized presentation (see below). Clarifying these aspects would greatly enhance the paper's readability and overall impact.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following questions raised about our work. Please let us know if there’s anything we can clarify further.\n\n---\n\n> 0. Some of the technical terms are misused, making the paper confusing to read. For example, \"implicit shape representation\" is repeatedly used to refer to the rendered object image masks, but \"implicit\" generally refers to using functions (parametric or neural networks) to represent shapes, where shapes must be retrieved through function evaluations, hence \"implicit\". …\n\n**Reply:** Sorry for the confusion caused. Since our method employs both 3D surface particles and 2D object masks for supervision, we originally intended to use \"implicit shape representation\" to describe 2D object masks in order to distinguish them from 3D surface particles, but we overlooked the ambiguity it introduced. We will correct the description by directly using 2D object masks in the revised version.\n\n---\n\n> 1. Table 3: Can you include the ground truth values in the table so that readers understand what to expect?\n\n**Reply:** Thanks for this constructive advice. For more details about the estimated and ground truth values, please refer to the attached PDF in the \"Author Rebuttal\" session. We will also add the table to the Appendix in the revised version. \n\n---\n\n> 2. System Identification: How do you set the initial parameters for the system identification? Are the optimizations sensitive to initial conditions?\n\n**Reply:** The table in the attached PDF also lists each instance's initial guess. To make a fair comparison, we followed the setting in PAC-NeRF [1] to assign the same initial values for the instances with the same material. The results show that our method is robust to initial conditions even when they are significantly different from the ground truth. \n\n---\n\n> 3. Figure 4: Would mask-based supervision fail when the estimated shapes are significantly different from the ground truth? Have you observed any cases where this occurs?\n\n**Reply:** Under extreme conditions, the mask-based supervision might fail when the simulated trajectory is completely out of view for all viewpoints. However, we did not find any failure cases in our experiments, because we performed initial velocity estimation before system identification. With the initial velocity available and multiple viewpoints located at proper positions, we observed that it's unlikely that the simulated trajectory is outside the field of view, and the estimated shapes will always converge to the ground truth shapes even if they have significant discrepancies at the initial stage. \n\n---\n\n> 4. Figure 1a: There is a typo in the figure caption. Change \"caption\" to \"capture.”\n\n**Reply:** Thank you for pointing this out. We will correct the typo in the revised version. \n\n---\n\n> 5. References: It would be beneficial to include references to works on differentiable cloth simulation for system identification and inverse problems, such as \"Differentiable Cloth Simulation for Inverse Problems\" by Liang et al. and \"DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact\" by Li et al.\n\n**Reply:** Thank you for this suggestion. These two works also tackle the inverse problems with differentiable simulators, particularly in cloth material. We will cite the related works you mentioned in our revised manuscript.\n\n---\n\n> 6. Limitations: I do wonder about the failure cases of the method if there are any.\n\n**Reply:** In the synthetic experiments, we didn't encounter any failure cases on both the PAC-NeRF [1] and SpringGaus [2] synthetic datasets, since each scenario includes video sequences from 10 or 11 distinct viewpoints, which are sufficient for dynamic Gaussian reconstruction. However, the proposed dynamic Gaussian module would fail to reconstruct the trajectory on the SpringGaus real dataset because it only contains 3 viewpoints to capture the dynamic scene. That's why we only use object masks for the supervision of system identification (More details are illustrated in Section 5.3 and Appendix A.5.). Therefore, our method might not perform well when **fewer views** of the scenario are available. Using fewer views to recover both geometry and system identification is more practical and will be an interesting direction for future work. \n\n---\n\n[1] Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chen-fanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostic system identification. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.\n\n[2] Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. arXiv preprint arXiv:2403.09434, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5AJ847fDIY", "reviewer": "Reviewer_a7go", "summary": "The manuscript proposes a novel hybrid framework that leverages 3D Gaussian representations for system identification from visual observations. The framework captures both explicit and implicit shapes using dynamic 3D Gaussian reconstruction and a coarse-to-fine filling strategy to generate density fields. These fields are used to sample continuum particles for simulation and extract object surfaces, which can render object masks during simulations to guide physical property estimation.", "questions": "1. I wonder what makes the difference between the proposed method and PAC-NeRF in the infilled particle generation.\n2. If I understood correctly, the motion network only takes time as input. I wonder if it helps by combining the temporal encoding from DiffAqua [1] to further capture the low and high frequencies.\n3. The infilling algorithm seems expensive since the complexity grows exponentially. Did you try using Octree [2] or similar algorithm to speed it up?\n\n[1] Ma, Pingchuan, et al. \"Diffaqua: A differentiable computational design pipeline for soft underwater swimmers with shape interpolation.\" ACM Transactions on Graphics (TOG) 40.4 (2021): 1-14.\n\n[2] Meagher, Donald JR. Octree encoding: A new technique for the representation, manipulation and display of arbitrary 3-d objects by computer. Electrical and Systems Engineering Department Rensseiaer Polytechnic Institute Image Processing Laboratory, 1980.", "limitations": "Yes.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The presentation is clear. The figures look high-quality.\n2. I personally appreciate the real-world experiments. I am happy to see the proposed method also works in real life.\n3. The proposed method solves one of the most interesting problem in the intersection of guassian splatting and physical simulation, where physical simulation requires volumetric representation but 3dgs outputs surfaces.", "weaknesses": "1. In the real-world experiment, I found the authors switched to FEM for deformable body simulation, which conflicts with the MPM simulator used in their pipeline. I think it needs justifications.\n2. Some wordings are confusing: e.g., line 166, do you mean effective instead of efficient?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following questions about our work. Please let us know if there’s anything we can clarify further.\n\n---\n\n> 1. In the real-world experiment, I found the authors switched to FEM for deformable body simulation, which conflicts with the MPM simulator used in their pipeline. I think it needs justifications.\n\n**Reply:** Sorry for the confusion caused. Switching to FEM is because most of the widely-used robotic simulators incorporate FEM for deformable object simulation, including Isaac Gym, the simulator we used in our experiments. Moreover, estimating physical parameters using MPM and then applying them to FEM should be a feasible way for this application, since\n\n(1) both the Material Point Method (MPM) and the Finite Element Method (FEM) originate from Galerkin methods [3]. In theory, MLS-MPM can achieve first-order consistency in simulations [3], which is comparable to the consistency achieved by linear FEM simulations, \n\n(2) the material properties (e.g., Young's modulus and Poisson's ratio) are independent of the numerical methods.\n\nWe will clarify this in the revised version. \n\n---\n\n> 2. Some wordings are confusing: e.g., line 166, do you mean effective instead of efficient?\n\n**Reply:** Sorry for the confusion caused. Indeed, we should use \"effective\" in line 166. We will correct the expression in the revised version. \n\n---\n\n> 3. I wonder what makes the difference between the proposed method and PAC-NeRF in the infilled particle generation.\n\n**Reply:** Sorry for the lack of clarity.\n\n(1) PAC-NeRF samples particles by directly selecting NeRF field voxels whose alpha values are greater than the predefined threshold and then performing uniform sampling 4 times on each extracted voxel. Since PAC-NeRF defined the alpha threshold with **a small value** to make sure that a solid continuum can be obtained, they usually tend to recover over-large shapes. \n\n(2) On the contrary, our method turns to generate a density field based on the Gaussian and initial particles {$\\mu(t)$} $\\cup P_{in}$. Among these particles, the Gaussian particles are prone to be located at the object surface, which can guarantee high density at the surface region, while the initial ones incorporated with the coarse-to-fine operations can ensure a solid internal region. With this module, we can extract the continuum from the density field, which serves only for **representing the object shape** instead of the NeRF field, which is utilized for **rendering**. Please refer to Figure 7 in Appendix A.2.2 to see the qualitative results of the proposed filling algorithm, along with ones from PAC-NeRF.\n\nWe will add more explanation in the revised version.\n\n---\n\n> 4. If I understood correctly, the motion network only takes time as input. I wonder if it helps by combining the temporal encoding from DiffAqua [1] to further capture the low and high frequencies.\n\n**Reply:** Sorry for the lack of clarity. We do employ temporal and positional encoding to the time $t$ and position $\\mu_0$, respectively, to introduce features with various frequencies. Specifically, the encoding module is denoted as $\\gamma(x) = \\left( \\sin(2^k \\pi x), \\cos(2^k \\pi x) \\right)_{k=0}^{L-1}$, where $L=10$ for both $t$ and $\\mu_0$, which is exactly the same as the setting in DiffAqua. We will add the notations to Figure 2 and the implementation details to Appendix A.1.1 in the revised version. \n\n---\n\n> 5. The infilling algorithm seems expensive since the complexity grows exponentially. Did you try using Octree [2] or similar algorithm to speed it up?\n\n**Reply:** Although the memory requirements for processing the volumetric data scales cubically with the grid resolution, we still use the naive volumetric data structure for our algorithm because \n\n(1) in practice, only **four** iterations are required to achieve sufficient accuracy (the implementation details are also available in Appendix A.2.1), \n\n(2) such a data structure can be efficiently implemented with GPU acceleration based on PyTorch, where the trilinear interpolation and mean filter operations are implemented by \"grid_sample\" and \"conv3d\" functions, respectively. \n\nTherefore, we can almost achieve real-time performance (more than 10 fps on a single Nvidia 3090 GPU) on our infilling algorithm.\n\n---\n\n[3] Hu, Yuanming, et al. \"A moving least squares material point method with displacement discontinuity and two-way rigid body coupling.\" ACM Transactions on Graphics (TOG) 37.4 (2018): 1-14.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0tBtc6MTTa", "reviewer": "Reviewer_HGBq", "summary": "This paper introduces a novel hybrid method that leverages 3D Gaussian representation and continuum to estimate physical properties of deformable objects. From multi-view video, the Gaussian- informed continuum can be extracted and then combined with material point method (MPM) simulation to train the whole pipeline by using both 3D shape and 2D as supervision. The experiments show that the proposed method outperforms previous approaches based on continuum dynamics or 3D Gaussian representation for dynamic reconstruction, system identification or other real-world applications.", "questions": "1. It would be nice to see a  comparison between the choice of Motion-factorized dynamic 3D Gaussian network and previous architectures ?\n\n2. It is important that the experimentation on complex motions must be performed in order to truly understand the strength of the proposed  method.", "limitations": "To some extent, yes.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper introduces an efficient motion-factorized dynamic 3D Gaussian network to reconstruct the object states as a linear combination of basis motions, in which the estimated motions and coefficients share the same backbone.\n\n2. The generated Gaussian-informed continuum consists of the density and grid size scale given by the proposed coarse-to-fine filling strategy, which is further used as supervision in training together with the MPM simulation. This tackles the issue of using quantised Gaussian particles for simulation of continuous structures.\n\n3. The experiments show that the method can achieve SoTA performance compared to pre- vious works among various deformable objects, especially when large deformation occurs. The method is moreover applicable to real-word scenarios.", "weaknesses": "1. The authors stated that such a lightweight architecture of the motion-factorized dynamic 3D Gaussian network is sufficient for complex motions rather than modeling each basis with an independent network (line 168-171) while lacking an experimental proof. \n\n2. The representation in Section 4.3 is lacking in elaboration and should accompany with more details in the supplementary. Please give some detailed elaboration for Section 4.3 about Gaussian-informed continuum, e.g., notations used in Algorithm 1, dimensions of the variables, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following questions about our work. Please let us know if there’s anything we can clarify further.\n\n---\n\n> 1. The authors stated that such a lightweight architecture of the motion-factorized dynamic 3D Gaussian network is sufficient for complex motions rather than modeling each basis with an independent network (line 168-171) while lacking an experimental proof.\n\n**Reply:** Thank you for pointing out this question. We conducted two ablation analyses to empirically demonstrate the arguments mentioned in lines 168-171.\n\n(1) We first use our method to perform dynamic Gaussian reconstructions on 45 cross-shaped objects in the PAC-NeRF dataset, except each motion basis is modeled with an independent network. Specifically, each basis takes the encoded time as input and contains 8 fully connected layers. The output is the residuals of position and scale of this basis. We use the setting mentioned in Sec. 5.1 to evaluate the CD and EMD on the above method variants and compare them with our method. The results are reported in the table below. The results show that our backbone-shared architecture (**Ours**) slightly outperforms the independent-basis networks (**Ind.**) in terms of dynamic reconstruction, which empirically demonstrates the strength that the reduced order modeling of dynamic Gaussians is sufficient for motion reconstruction tasks. Moreover, our method can achieve less training time compared with the independent design (around 15 minutes vs. 45 minutes for a dynamic scene on a single 3090 GPU).\n\n|                | CD $\\downarrow$ (Ind.)    | CD $\\downarrow$ (**Ours**) | EMD $\\downarrow$ (Ind.)    | EMD $\\downarrow$ (**Ours**) |\n|----------------|---------------------------|----------------------------|----------------------------|-----------------------------|\n| Newtonian      | 0.250                     | **0.243**                  | 0.026                      | **0.025**                   |\n| Non-Newtonian  | 0.204                     | **0.195**                  | 0.023                      | **0.022**                   |\n| Elasticity     | 0.188                     | **0.178**                  | 0.022                      | **0.020**                   |\n| Plasticine     | 0.215                     | **0.196**                  | 0.024                      | **0.022**                   |\n| Sand           | 0.273                     | **0.250**                  | 0.028                      | **0.025**                   |\n| Mean           | 0.226                     | **0.212**                  | 0.025                      | **0.023**                   |\n\n(2) To validate the second argument in lines 171-172, we compared our method with DynMF [1], which also uses neural networks as learnable bases while considering motion coefficients as time-invariant Gaussian attributes, by evaluating the PSNR on the D-NeRF dataset. The results are reported in the table below. The results show that modeling the motion coefficients as time-variant variables does increase the ability to fit the dynamic scenes. \n\n| Method    | Hell Warrior | Mutant | Hook  | Bouncing Balls | T-Rex  | Stand Up | Jumping Jacks | Mean  |\n|-----------|--------------|--------|-------|----------------|--------|----------|---------------|-------|\n| DynMF [1] | 36.60        | 41.00  | 31.30 | 41.01          | 35.10  | 41.16    | 35.75         | 37.42 |\n| Ours      | **41.97**    | **42.93** | **38.04** | **41.26** | **37.54** | **45.32** | **38.86** | **40.85** |\n\n---\n\n> 2. The representation in Section 4.3 is lacking in elaboration and should accompany with more details in the supplementary. Please give some detailed elaboration for Section 4.3 about Gaussian-informed continuum, e.g., notations used in Algorithm 1, dimensions of the variables, etc.\n\n**Reply:** Thank you for this constructive advice. As suggested, we added a table to the attached PDF in the \"Author Rebuttal\" session to clarify the operators and symbols in detail. We will also add the table to the Appendix in the revised version.\n\n---\n\n> 3. It would be nice to see a comparison between the choice of Motion-factorized dynamic 3D Gaussian network and previous architectures ?\n\n**Reply:** Please refer to the first reply. \n\n---\n\n> 4. It is important that the experimentation on complex motions must be performed in order to truly understand the strength of the proposed method.\n\n**Reply:** Thank you for this suggestion. We conduct an additional experiment on a scenario with a more complex boundary condition and motion trajectory. Specifically, we use our method to perform system identification on an elastic rope falling onto two rigid cylinders. The data format is the same as PAC-NeRF. The estimated property is reported in the table below, and the simulated trajectory is visualized in the attached PDF. The results show that our method can also generalize to scenarios with more complex boundary conditions and motions.\n\n|        | Initial Guess | PacNeRF         | Ours               | Ground Truth |\n|--------|---------------|-----------------|--------------------|--------------|\n| $E$    | $10^3$        | $1.12 \\times 10^5$ | $\\mathbf{1.03 \\times 10^5}$ | $10^5$       |\n| $\\nu$  | $0.4$         | $0.22$          | $\\mathbf{0.23}$    | $0.3$        |\n\n---\n\n[1] Kratimenos, Agelos, Jiahui Lei, and Kostas Daniilidis. \"Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting.\" arXiv preprint arXiv:2312.00112 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GFLodZ2y8M", "reviewer": "Reviewer_mZdJ", "summary": "The paper proposes an improvement of PAC-NeRF for the task of estimating material properties from multiview video using 3D Gaussian Splatting (3DGS). Instead of estimating geometry solely based on the first frame like PAC-NeRF, the proposed method uses 4D Gaussian Splatting (4DGS) with reduced order modeling to construct 4D geometry, enabling the use of 3D supervision. A coarse-to-fine internal filling strategy is introduced to ensure that the simulation operates on a solid volume. 2D mask loss is used for additional supervision.", "questions": "If the 4D geometry is accurate enough, is it adequate to only use 3D supervision? Ablation studies are needed to validate the necessity of 2D mask supervision.\n\nCoarse-to-fine density field creation: At the beginning, the reconstruction contour is much larger than the object. How does it shrink to the actual boundary? The $TrilinearInterpolation$ operator will not shrink the contour, and there is no operator to assign zeros.\n\nIn the simulation, how do the Gaussian kernel scales evolve? The paper seems to assume isotropic kernels, but physical deformation can transform the sphere into an ellipse. How is this addressed?", "limitations": "Limitations are well discussed.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The reduced order modeling of 4DGS is a good fit for the reconstruction task with a limited number of fixed views. Directly applying full-order 4DGS seems hard to optimize due to the high number of DOFs.\n\nThe experiment results are promising.\n\nA real-world application is provided.", "weaknesses": "Some symbols are not defined clearly, making it hard to follow at times. For example, $Discretize$ operator in Line 214; $\\tilde{P}$ and $F$ are not defined in the text. I need to guess from Alg 1.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed reading of our paper and constructive suggestions! We hope our responses adequately address the following questions about our work. Please let us know if there’s anything we can clarify further.\n\n---\n\n> 1. Some symbols are not defined clearly, making it hard to follow at times. For example, $Discretize$ operator in Line 214; $\\tilde{P}$ and $F$ are not defined in the text. I need to guess from Alg 1.\n\n**Reply:** Sorry for the lack of clarity. $Discretize$ denotes the operation mapping particle positions to voxel indices on the density field. \\tilde{P}$ and $F$ are the sampled particles and the density field. We add a table to the attached PDF in the \"Author Rebuttal\" session to clarify the operators and symbols in detail. We will also add the table to the Appendix in the revised version.\n\n---\n\n> 2. If the 4D geometry is accurate enough, is it adequate to only use 3D supervision? Ablation studies are needed to validate the necessity of 2D mask supervision.\n\n**Reply:** Thank you for this constructive advice. \n\n(1) To answer the first question, we perform system identification on the torus object, which is the only instance that provides a ground truth mesh model in the PAC-NeRF dataset. Specifically, we use the ground truth point clouds as the continuum for simulation and utilize the mesh model to extract surface particles from the point cloud as 3D supervision. The experiment, performed with this configuration and other settings unchanged, achieves $E = 1,002,907.25$ and $\\mu = 0.2991$, which are close to the ground truth ($E = 1,000,000$ and $\\mu = 0.3$). Therefore, we believe that **it should be sufficient to use 3D object surfaces as supervision once the recovered geometry is accurate enough**. \n\n(2) To evaluate the necessity of 2D mask supervision, we perform system identification on 45 cross-shaped object instances in the PAC-NeRF dataset by our method but with only object surface supervision. The results are reported in the table below. It is obvious to see that combining both 2D and 3D shapes as supervision can achieve more accurate performance compared to using 3D shapes only. Therefore, we believe that **utilizing 2D mask supervision to some extent makes up for the errors introduced by the 3D object surfaces extracted from dynamic 3D Gaussians**. We will add the analysis to the revised version. \n\n| **Type**          | **Parameters**   | **w/o masks**            | **w/ masks**             |\n|-------------------|------------------|--------------------------|--------------------------|\n| **Newtonian**     | $\\log_{10}(\\mu)$ | $2.19 \\pm 2.90$          | $\\mathbf{1.53 \\pm 1.31}$ |\n|                   | $\\log_{10}(\\kappa)$ | $24.2 \\pm 22.2$       | $\\mathbf{14.8 \\pm 19.2}$ |\n|                   | $v$              | $0.20 \\pm 0.08$          | $\\mathbf{0.20 \\pm 0.07}$ |\n| **Non-Newtonian** | $\\log_{10}(\\mu)$ | $19.4 \\pm 27.7$          | $\\mathbf{13.5 \\pm 18.2}$ |\n|                   | $\\log_{10}(\\kappa)$ | $24.0 \\pm 24.8$       | $\\mathbf{12.9 \\pm 16.8}$ |\n|                   | $\\log_{10}(\\tau_Y)$ | $\\mathbf{4.58 \\pm 9.11}$ | $4.80 \\pm 3.92$      |\n|                   | $\\log_{10}(\\eta)$ | $49.1 \\pm 40.5$         | $\\mathbf{40.7 \\pm 24.6}$ |\n|                   | $v$              | $1.33 \\pm 0.54$          | $\\mathbf{0.19 \\pm 0.09}$ |\n| **Elasticity**    | $\\log_{10}(E)$   | $2.85 \\pm 1.94$          | $\\mathbf{2.43 \\pm 3.29}$ |\n|                   | $\\nu$            | $3.97 \\pm 2.64$          | $\\mathbf{2.52 \\pm 2.03}$ |\n|                   | $v$              | $\\mathbf{0.22 \\pm 0.10}$ | $0.82 \\pm 0.32$          |\n| **Plasticine**    | $\\log_{10}(E)$   | $\\mathbf{25.6 \\pm 27.4}$ | $25.6 \\pm 29.4$          |\n|                   | $\\log_{10}(\\tau_Y)$ | $9.04 \\pm 2.37$       | $\\mathbf{1.67 \\pm 1.21}$ |\n|                   | $v$              | $1.16 \\pm 0.00$          | $\\mathbf{0.22 \\pm 0.10}$ |\n| **Sand**          | $\\theta_{fric}$  | $\\mathbf{2.55 \\pm 2.03}$ | $4.18 \\pm 0.52$          |\n|                   | $v$              | $0.31 \\pm 0.18$          | $\\mathbf{0.17 \\pm 0.05}$ |\n\n---\n\n> 3. Coarse-to-fine density field creation: At the beginning, the reconstruction contour is much larger than the object. How does it shrink to the actual boundary? The $TrilinearInterpolation$ operator will not shrink the contour, and there is no operator to assign zeros.\n\n**Reply:** Sorry for the lack of clarity. Although all the operations cannot assign voxels outside an object to zeros, the trilinear interpolation and mean filter operations can reduce the density of the voxels outside the object boundary. By iteratively performing mean filtering and particle voxel reassigning, the densities outside the boundary will be sufficiently small while the object boundary and internal region keep high-density values, and we thus can extract the object particles by thresholding the density field. \n\n---\n\n> 4. In the simulation, how do the Gaussian kernel scales evolve? The paper seems to assume isotropic kernels, but physical deformation can transform the sphere into an ellipse. How is this addressed?\n\n**Reply:** Sorry for the lack of clarity. In this work, we use the grid size of the density field as scale attributes of Gaussian kernels and fix them during the simulation. We admit that a physics-informed scale transformation such as PhysGaussian [1] allows a more realistic rendering. In future work, we will integrate this function into our method to enable kernel transformation during simulation. \n\n---\n\n[1] Xie, Tianyi, et al. \"Physgaussian: Physics-integrated 3d gaussians for generative dynamics.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to express our sincere gratitude to all the reviewers for their time and their valuable feedback. We deeply appreciate their recognition of our work, such as 
"The experiments in this paper demonstrate improvements over prior works" (FGDU), 
"I am happy to see the proposed method also works in real life" (a7go), 
"The proposed method solves one of the most interesting problems in the intersection of Gaussian splatting and physical simulation" (a7go), 
"The experiments show that the method can achieve SoTA performance" (HGBq), and 
"The reduced order modeling of 4DGS is a good fit for the reconstruction task" (mZdJ). 
We hope that our work indeed "makes meaningful contributions to the problem of geometry + physical property estimation" (FGDU).

Inspired by their thoughtful comments, we have incorporated the following changes in the revision of our paper:

- We conducted four additional experiments, including 
	- a comparison of our proposed network and independent basis baselines in terms of dynamic reconstruction, 
	- system identification on a rope instance with more complex motion and boundary conditions, 
	- system identification on the torus instance with ground truth point cloud and surface supervision, and
	- system identification on 45 cross-shaped object instances with only 3D surface supervision, 
	to address the concerns of the reviewers. 
- We updated our manuscript to fix typos and misused terminology to reduce the potential for misunderstandings.
- We added a table to the appendix to provide the estimated and ground truth values for Table 3 in the main manuscript (see Figure 1 in the attached PDF).
- We added a table to the appendix to clarify the operators and symbols in Algorithm 1 in detail (see Figure 2 in the attached PDF).

We hope our responses adequately address the questions raised about our work. Please let us know if there is anything else we can clarify further.

Meta-review: This paper has received unanimous acceptance recommendation (2 WA, 2A), it is addressing an important problem and shows potential for real world applications. It is recommended for oral presentation for its high impact.
Decision: Accept (oral)