GIC: Gaussian-Informed Continuum for Physical Property Identification and Simulation

Authors: junhao  cai
yuji  yang
weihao  yuan
yisheng  he
zilong  dong
liefeng  bo
hui  cheng
qifeng  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 15:14:43

Abstract: This paper studies the problem of estimating physical properties (system identification) through visual observations. To facilitate geometry-aware guidance in physical property estimation, we introduce a novel hybrid framework that leverages 3D Gaussian representation to not only capture explicit shapes but also enable the simulated continuum to render object masks as 2D shape surrogates during training. We propose a new dynamic 3D Gaussian framework based on motion factorization to recover the object as 3D Gaussian point sets across different time states. Furthermore, we develop a coarse-to-fine filling strategy to generate the density fields of the object from the Gaussian reconstruction, allowing for the extraction of object continuums along with their surfaces and the integration of Gaussian attributes into these continuum. In addition to the extracted object surfaces, the Gaussian-informed continuum also enables the rendering of object masks during simulations, serving as 2D-shape guidance for physical property estimation. Extensive experimental evaluations demonstrate that our pipeline achieves state-of-the-art performance across multiple benchmarks and metrics. Additionally, we illustrate the effectiveness of the proposed method through real-world demonstrations, showcasing its practical utility. Our project page is at  https://jukgei.github.io/project/gic.

Main paper: 
 introduction: 
 Identifying the physical properties of objects (i.e., system identification) is essential for numerous applications such as games, digital twins, and robotic manipulation [1] [2] [3] . Although humans can intuitively deduce the underlying physical properties with a single glance when the object undergoes deformation, estimating the properties with only visual observations remains challenging for computational perceptual algorithms.To tackle this challenge, many established methods [4] [5] [6] adopt the assumption of elastic material [7] and perform physics-based modeling based on mass-spring systems (MSS) or finite element method (FEM) to model and simulate the dynamics of the objects. Such an assumption inevitably restricts the ability to simulate more general types beyond elastic materials, such as fluids or granular media. Another problem of previous methods lies in that many methods [8] [9] [10] require the ground-truth full knowledge of object geometry for the identification, which limits their practicality. Some subsequent methods [5, 4] turn to recover the geometries and physical properties from observations in a decoupled manner. Specifically, these methods first extract object geometries by making use of stereo observations or dynamic neural reconstruction [11] from RGB video sequences, and then perform simulation directly on the point clouds or after the tetrahedral mesh conversion. While these methods introduce explicit geometries to guide the estimation of physical properties, the noisy reconstruction results usually lead to degraded system identification performance.Recently, PAC-NeRF [12] integrates neural radiance fields (NeRF) [13] with a continuum dynamic model to tackle the above problems. The object geometries and physical properties are captured in a unified framework. Despite its effectiveness, this method possesses two limitations. Firstly, the implicit shapes represented by NeRF often lead to inferior geometries, which might cause inaccurate trajectories during simulation. Secondly, PAC-NeRF renders the novel views of deformed objects based on the appearance radiance field reconstructed from the static scene, which might introduce texture distortion, particularly when objects undergo significant deformations, resulting in discrepancies between the rendered and the observed images [14] .To address these limitations, this paper proposes a novel hybrid solution based on 3D Gaussians [15, 16] and material point method (MPM) [17, 18] . The core strength of this work is that we make use of both 3D shapes from dynamic 3D Gaussian reconstruction and 2D shapes rendered by the Gaussian-informed continuum for physical property estimation.To generate more precise shapes to reason physical property, we first propose a motion-factorized dynamic 3D Gaussian network to conduct dynamic scene reconstruction. We then extract the continuum from the recovered 3D Gaussians at each frame by leveraging a coarse-to-fine filling strategy to generate the density field of the object progressively. The resulting density fields can be used to sample continuum particles for simulation and extract object surfaces as 3D-shape supervision in physical property estimation. To eliminate the appearance distortion caused by large deformation in PAC-NeRF, we further assign Gaussian attributes to the continuum particles where the opacity and scale attributes are evaluated from the density field. Such Gaussian-informed continuum are able to render object masks during simulation, which can be regarded as a 2D-shape representation to guide the estimation and effectively avoid using inferior rendering results for learning physical properties.To demonstrate the superiority of the proposed method over other baselines, we conduct three types of experiments, including evaluations of physical properties, dynamic reconstruction, and future state simulation. We also demonstrate a real-world application in digital twins and robotic manipulation, showing the applicability of the proposed method in real-world scenarios.Our contributions are summarized as follows.• We propose a novel hybrid pipeline that takes advantage of the 3D Gaussian representation of the object to both acquire 3D shapes and empower the simulated continuum to render 2D shapes for physical property estimation.• We propose a novel dynamic 3D Gaussian framework with motion factorization to achieve more precise dynamic reconstruction. We also propose a coarse-to-fine filling strategy to generate the density field of the object, which can be utilized to extract object surfaces and obtain Gaussianinformed continuum particles.• Extensive experiments show that our pipeline attains state-of-the-art performance on existing benchmarks with a wide range of metrics. We also present a real-world demonstration to show the efficiency of the proposed method. 
related work: 
 Dynamic reconstruction. Reconstructing dynamic scenes from monocular or multi-view video(s) is a long-standing problem in the computer vision community [19, 20] . Previous works exploit neural implicit representation [21, 22] for non-rigid reconstruction. These methods either reconstruct the scene in a frame-wise manner [23, 24] or maintain a canonical shape and model the deformation with a neural network [25, 26, 11, 27] . While effective for novel view synthesis, these methods often require extensive training time and can result in noisy deformations owing to the implicit representation, which may compromise the utility of the recovered geometries for physical property estimation [12] . Recent progress in 3D Gaussian Splatting (3DGS) technique [15] stands out to be a prevalent method for 3D reconstruction and novel view synthesis because of the abilities of explicit shape modeling and extremely fast view rendering. Similar to non-rigid NeRF, many follow-up works extend the 3DGS into 4D by treating each frame separately [28] or decomposing a scene into a canonical 3D Gaussian point cloud and a deformation model that warps the canonical shape into a specific scene [16, 29, 30] . In this paper, we draw upon these prior studies [16, 29] and propose a novel motion-factorized dynamic 3D Gaussian network to achieve better performance on reconstruction and novel view synthesis.System identification. Understanding the physics laws of the 3D world is beneficial for simulation [31] [32] [33] [34] [35] 6] and manipulation [2, 3, [36] [37] [38] . However, unveiling these properties from visual information is an extremely difficult task due to the ambiguity introduced by incomplete observation and the high degrees of freedom of the scene. Early works [39, 40] study the problem by learning physical properties via interactions. With recent improvements in differentiable physics simulation [17, 18, [41] [42] [43] [44] [45] , many methods turn to evaluate the physical properties by comparing the rendering results with 2D ground truth given the prior knowledge about the object geometry. VEO [5] presents a differentiable simulator to learn patterns from 4D reconstruction and force-displacement measurements. Another approach [4] eliminates the dependence of captured forces by proposing an iteration framework between deformation tracking and parameter optimization. While these methods demonstrate promising results, the inferior reconstruction might lead to degraded performance, and the assumption of elastic material restricts the applicability. PAC-NeRF [12] instead proposes a single framework to recover both the unknown geometry and physical properties of deformable objects from multi-view video sequences. However, the inferior geometries and blurry rendered images might have detrimental effects on physical property reasoning. In this work, we adopt MPM as our simulation framework following the approach used in PAC-NeRF due to its ability to simulate a variety of materials [6, [46] [47] [48] . Unlike previous approaches, we utilize dynamic 3D Gaussians to reconstruct explicit 3D geometries and generate simulatable continuum particles. Furthermore, we enhance the particles with Gaussian attributes, facilitating the rendering of 2D shapes, and thereby improving physical parameter estimation. 
preliminary: 
 In this section, we briefly review the core idea of 3D Gaussian Splatting (3DGS) [15] and introduce its point-based alpha blending to render depth maps and foreground masks. Typically, 3DGS utilizes 3D Gaussians, each defined by a central point µ 0 , a covariance matrix Σ 0 , a density value σ, and a color attribute c, to efficiently render images from specific viewpoints. Each point is denoted asEQUATIONwhere Σ 0 can be factorized as Σ 0 = R 0 S 0 S T 0 R T 0 , in which R 0 is a rotation matrix represented by a quaternion vector r 0 ∈ R 4 , and S 0 is a a diagonal scaling matrix characterized by a 3D vector s 0 ∈ R 3 . If we consider isotropic Gaussian representation, the scaling matrix can be written as s 0 I, where s 0 is a scalar and I is the identity matrix. When performing splatting, the 3D Gaussians are projected into 2D with the covariance matrix defined as Σ ′ 0 = JW Σ 0 W T J T , where J is the Jacobian of affine approximation of the projective transformation [49] , and W is the viewing transformation matrix. The rendered color I(u) with its foreground mask A(u) at pixel u are then evaluated by integrating N ordered slatted Gaussians via the point-based alpha blending. Since the depth of each Gaussian point at a specific view can be obtained according to its transformation matrix, we can further render the depth map D using the same blending method [16, 50] , asEQUATIONwhereT i = i-1 j=1 (1 -α j )is the accumulated transmittance, α i is the probability of termination at point i, and d i is the depth of the Gaussian point at the specific view. 
problem definition and overview: 
 In this work, we aim to reconstruct the geometries and the physical properties of various object types from multi-view videos. Formally, given a set of video sequences {V i |i = 1...n} with moving object and the corresponding camera extrinsic and intrinsic parameters {(T i , K i )|i = 1...n}, the goal of this task is to recover the explicit geometries of the object represented by continuum particles P (t) and its corresponding physical parameters Θ (e.g., Young's modulus E and Poisson's ratio ν for elastic objects). We follow the assumption in PAC-NeRF and PhysGaussian [12, 51] that the object types (e.g., elastic, granular, Newtonian/non-Newtonian, plastic) are known and the physical phenomenon follows continuum mechanics [17, 52] . The overview of the proposed pipeline is illustrated in Fig. 1 , which consists of three modules: a motion-factorized dynamic 3D Gaussian network (Sec. 4.2) for 4D reconstruction of the object, a coarse-to-fine density field generation strategy (Sec. 4.3) for continuum generation, surface extraction, and Gaussian attribute assignment, and a procedure (Sec. 4.4) showing how we leverage Gaussianinformed continuum and extracted surfaces to estimate physical properties.Our dynamic 3D Gaussian network follows existing frameworks [16, 29, 30] that simultaneously maintain a canonical 3D Gaussian set and a deformation field modeled by a neural network to warp the canonical shape into object states at specific times. The core idea of this pipeline, presented in Fig. 2 , is that the motion of every point in the object can be decomposed into a small range of motion bases.Architecture. We first factorize the entire motion into N m bases that are modeled by a fully connected neural network, where every basis shares a common backbone except the final layer. The output of each basis consists of the deformations at position dµ i (t) ∈ R 3 and at scale ds i (t) ∈ R. To model the exact deformation for each position, we next propose a lightweight coefficient network that maps the positions at canonical space with specific time to their corresponding motion coefficients w(µ 0 , t) ∈ R Nm . Therefore, the deformed position and the scale for each Gaussian point are evaluated by the linear combination of the motion basis according to the motion coefficients:EQUATIONIn this work, we regard all the Gaussians as isotropic kernels, which has been demonstrated as an effective way to simplify the model and better reconstruct the scene [6, 53] . We should note that although previous works [29, 54] also perform motion decomposition modeling, our pipeline shows two major differences: 1) instead of modeling each basis with an independent neural network, our module shares a common backbone. Our key observation is that for reconstructing a dynamic object, all points on the object should follow a similar moving tendency, and the final heads of the neural network are sufficient to model the details of different parts of the object; 2) to increase the ability to fit high rank of the dynamic scene [16] , we model the motion coefficients as time-variant variables rather than constant Gaussian attributes [29] .Optimization. We employ the same setting in [16] to train our pipeline. Concretely, the canonical 3D Gaussians are initialized with points randomly sampled from the given bounding box of the scene. We start training the deformation network after 3,000 iterations of warm-up for the 3D Gaussians. Similar to previous works [16, 29] , we optimize the pipeline by computing the L1 norm and Structural Similarity Index Measure (SSIM) between the rendered image I and the ground truth image Ĩ. Moreover, since large scales may lead to inaccurate reconstructed shapes [55] , we thus perform L1 norm on the scale attributes of all the points to recover more fine-grand shapes of the object. Therefore, the overall loss function is defined as:EQUATIONwhere λ 1 and λ 2 are balancing hyperparameters. More in-depth analysis of the proposed pipeline, including implementation details and effects of scale regularization, are presented in Appendix A.1.Coarse-to-fine density field generation. Since the reconstructed Gaussian particles are served for rendering only, meaning that they are not evenly distributed on the objects, they cannot be directly used for simulation [51] . Therefore, we propose a novel coarse-to-fine filling strategy to iteratively generate density fields of the object based on the reconstructed Gaussian particles from Eqn. 3 and the internal particles filtered by the rendered depth maps. The proposed strategy is presented in Alg. 1.The implementation details and visual results are illustrated in Appendix A.2.Concretely, the internal particles, initialized by uniform sampling from the bounding box of Gaussian particles, are filtered by projecting the particles to various images to compare the projected depth with rendered depth values (lines 1-6 in Alg. 1). The resulting particles can roughly represent the shape of the object. However, as denoted in Eqn. 2, the rendered depth maps are evaluated in an accumulated manner, making them less precise in representing the object surface.Therefore, We employ a coarse-to-fine filling strategy by iteratively upsampling the density field and reassigning the densities on the indices computed from both the Gaussian and internal particles (lines 8-16 in Alg. 1). Fig. 3 provides a sketch illustration of the proposed strategy. Specifically, due to the large grid size at the initial stage, the object is completely inside the voxels with high densities.Next, we sequentially perform upsampling (line 10), mean filtering (line 13), and reassigning the The particles are again used to correct the voxels that contain particles with high densities. (d) and (e) repeat the previous operations to achieve a more detailed shape.field (line 14) at each iteration. The first two operations produce more fine-grained shapes, and the reassigning operation ensures high densities at the surface to avoid over-erosion caused by the first two steps. Finally, the continuum particles with the corresponding object surfaces can be extracted by thresholding the density field (lines 16-17 in Alg. 1). (u in , v in ), d in ← P roj(P in , T i , K i ); ▷ obtain image indices and depths of P in at view i 5:P in ← P in [ Di (u in , v in ) ≤ d in ];▷ filter out particles that are outside the object 6: end for 7: Initialize the zero-value density field F (t) with ∆x and the bounding box of {µ(t)}; 8: for j ← 1, n u do 9:if j ̸ = 1 then 10:F (t) ← T rilinearInterpolation(F (t), 2)▷ upsample F (t) with scale factor 2 11:F (t)[p, q, r] = 1, where p, q, r ← Discretize(P in ∪ {µ(t)});12:end if 13:F (t) ← M eanF iltering(F (t)); 14:F (t)[p, q, r] = 1, where p, q, r ← Discretize(P in ∪ {µ(t)}); 15: end for 16: P (t) ← GetP osition(th min ≤ F (t)); 17: S(t) ← GetP osition(th min ≤ F (t) ≤ th max );Gaussian-informed continuum. In PAC-NeRF, the particles are equipped with appearance features to enable image rendering for the continuum at different states. We can also achieve this function by treating the particles as Gaussian kernels and re-train the particles using the visual data. However, this process is cumbersome and will also face the same issue in PAC-NeRF where distorted RGB images will be rendered when large deformation occurs. Therefore, instead of injecting appearance attributes, we opt to assign density and scale attributes to the particles where the densities originate from the density field, and the scale attributes can be directly obtained by the field grid size. The Gaussian-informed continuum is defined as a set of triplets:EQUATIONwhere p ∈ P , s ∆x = ∆x/2 nu , and σ F = F [Discretize(p)] (we neglect t in the notation for simplicity). Therefore, we only render object masks as 2D shape surrogates for supervision.With the Gaussian-informed continuum at initial state P P (0) and the extracted surfaces S(t) in place, we can employ MPM to perform simulation on the continuum and evaluate the difference in terms of both the 3D and 2D shapes. Concretely, after a rollout by MPM given the current estimation of physical parameters, we obtain a trajectory P (t) with corresponding object surfaces S(t). We thus can render object masks over the trajectory. Then the loss of the current rollout can be computed as:EQUATIONwhere L CD and L 1 are chamfer distance and L1 norm respectively, S(t i ) denotes the simulated surface at time t i , A j (t i ) is the rendered mask at view j, and Ãj (t i ) represents the object mask of the image extracted from video V j at time t i . Due to the differential property of the simulator, the evaluated loss is used to optimize the target physical parameters Θ. 
experiments: 
 Datasets. To thoroughly assess our proposed method, we employ two sources of data introduced by PAC-NeRF [12] and Spring-Gaus [6] . Concretely, PAC-NeRF contributes two synthetic datasets generated by MLS-MPM framework [18] . Each object in both datasets includes RGB images from 11 distinct viewpoints, with approximately 14 frames per viewpoint. The datasets feature a range of materials, including elastic and plastic objects, granular media, and both Newtonian and non-Newtonian fluids. The first dataset contains 45 cross-shape objects with different initial conditions and ground-truth values of physical properties, while the second one consists of 9 objects with different shapes. The interpretation of the physical parameters is listed in Appendix A.9 and A.10. Spring-Gaus generates a synthetic dataset of elastic objects and collects a real-world dataset containing both static and dynamic scenes. The synthetic data contains 30 frames in each of 10 viewpoints. While the real-world data only contains 3 viewpoints for each object in the dynamic scene, it captures 50-70 images from various viewpoints for the static scene. Moreover, we follow previous works [12, 6] and use the off-the-shelf matting [56] or segmentation [57] techniques to obtain object masks.Baselines. For dynamic reconstruction, we compare with PAC-NeRF and the current state-of-the-art deformable 3D Gaussian method DefGS [16] on the PAC-NeRF synthetic dataset. More comparison of our dynamic 3D Gaussian pipeline on other widely-used datasets such as D-NeRF [25] is presented in Appendix A.1.3. For system identification, we employ PAC-NeRF as the baseline and evaluate the performance using the two datasets introduced in PAC-NeRF. To further demonstrate the precision of the proposed method in terms of geometry recovery and future prediction, we perform experiments on the Spring-Gaus synthetic dataset and compare the results with PAC-NeRF and Spring-Gaus.Metrics. The evaluation metrics in the experiments include 1) Chamfer Distance (CD), with units expressed in 10 3 mm 2 ; 2) Earth Mover's Distance (EMD); 3) Peak Signal-to-Noise Ratio (PSNR); 4) Structural Similarity Index Metric (SSIM) [58] ; and 5) Mean Absolute Error (MAE), with values scaled by a factor of 100. The first two metrics are used to evaluate discrepancies between the reconstructed and ground-truth point clouds. PSNR and SSIM are leveraged on the Spring-Gaus dataset to validate the precision of future state prediction. We compute the mean absolute error for the evaluation of physical property estimation.Comparison on dynamic reconstruction. In this experiment, we first perform dynamic Gaussian reconstruction on the cross-shaped object dataset using DefGS and our proposed method, respectively. We then employ the same filling strategy on the reconstructed Gaussians at each time state to generate the continuum, which is regarded as the final recovered geometry of the object and used to make comparisons with the oracle shape to compute CD and EMD. Since PAC-NeRF jointly recovers both geometries and physical parameters, we use the final estimated results to generate the trajectory for evaluation.The results, reported in Tab. 1, show that our method outperforms the baselines on both metrics and achieves more precise reconstruction performance on most objects. Specifically, we find that the NeRF representation used by PAC-NeRF usually leads to overly large shape generation. While DefGS performs well on elastic objects, its performance degenerates when modeling objects with large deformations, such as granular media and fluids. Our method can better handle these objects due to the flexibility of trajectory representation. Comparison on system identification.We evaluate the performance of system identification of the two datasets proposed by PAC-NeRF. For the first dataset, we compute the MAE of the parameters for each type of object. To demonstrate the effectiveness of the 2D shape representation, we also conduct experiments on the second dataset by only using masks for supervision on our method, namely "Ours*". For the second dataset, we execute 10 times of our method with different random seeds for each object instance and report the mean value of the estimation results. The training details are illustrated in Appendix A.3.The results, reported in Tab. 2 and Tab. 3, show that the proposed hybrid pipeline can achieve more accurate estimation over a wide range of entries and objects, which demonstrate the effectiveness of the geometry-aware guidance. Fig. 4 visualizes the RGB images rendered by PAC-NeRF and the masks rendered by our method. We can see that when large deformation occurs, the rendered RGB image becomes distorted, while the rendered mask can effectively reduce such effect and get better perfor- mance. By leveraging both 3D and 2D shape guidance, our method obtains the best results on most entries. More qualitative results are available in the supplementary video.Comparison on future state simulation. To further demonstrate the performance of our proposed method, we follow the setting in Spring-Gaus [6] that uses the first 20 frames as training data and the subsequent 10 frames for evaluation. Concretely, we first perform system identification based on our method and then use the estimated physical parameters and the continuum to simulate a trajectory that includes the states of the 30 frames. Therefore, we can compute CD and EMD between the simulated continuum and the ground-truth point cloud. Since we know the exact position of the continuum at each time state after estimation, we can assign time-invariant Gaussian attributes by training Gaussians on the continuum using the first 20 frames of RGB images, which enable image rendering at novel views and states. Therefore, we can compute PSNR and SSIM at any time state.The results of future state prediction are presented in Tab. 4, and the results of reconstruction on the training states are reported in Appendix A. 4 . We observe that our method significantly outperforms the baselines on CD and EMD metrics over almost all object instances, which shows the superiority of our method for both geometry recovery and system identification. The results of PSNR and SSIM show that leveraging dynamic visual data to train the Gaussian attributes on the continuum improves rendering quality. This further reveals that the generated trajectories are precise such that the particles are consistent to contribute to the rendering for the same region of the object at different time states.To demonstrate the efficacy of the proposed method in real-world scenarios, we perform system identification on the real-world dataset collected by Spring-Gaus [6] , as shown in Fig. 5 . Since the real-world dataset consists of static and dynamic scenes for each object, we follow the procedure introduced by Spring-Gaus to progressively 1) reconstruct a Gaussian set of the object from the static scene, 2) transform the static Gaussian set to the initial state of the dynamic scene based on a registration network similar as iNeRF [6, 59] , and 3) perform system identification from the dynamic observation by our method "Ours*" due to the lack of sufficient images for dynamic reconstruction. Subsequently, we establish robotic platforms in both simulated and real-world environments, each equipped with UR10 robot arms configured identically. We then execute grasp attempts on both the reconstructed objects with the estimated properties in the simulation and the corresponding real-world objects under the same configuration. The results of more objects, and more details about the training and the experiment setting are presented in Appendix A.5. From the results shown in Fig. 5 , we see that our method demonstrates its capability to effectively model the deformation experienced by the objects upon impact with a surface. Furthermore, by applying identical gripper forces to both the simulated and real-world versions of the objects, we observe similar deformation behaviors. This consistency in deformation under identical conditions supports that the estimated physical parameters closely mirror the real-world properties of the objects.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The code is not included for now. But we will release the code to the public soon.Guidelines:• The answer NA means that paper does not include experiments requiring code. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion and limitations: 
 This paper proposes a novel solution that leverages the 3D Gaussian representation of objects to acquire explicit shapes while concurrently enabling the simulated continuum to render 2D shapes to facilitate the estimation of physical properties. A novel motion-factorized dynamic 3D Gaussian framework is proposed to reconstruct precise dynamic scenes. Object surfaces and Gaussian-informed continuum are obtained by utilizing the proposed coarse-to-fine density field generation strategy. Extensive experiments demonstrate the efficacy and applicability of our method.Despite the performance we achieve, this method still suffers from limitations, such as the assumption of continuum mechanics, the requirements of multi-view images with known camera poses, and the need for prior knowledge of object constitutive models. Integrating the pose-free method [60] or generalized constitutive [61] model with our method will be an interesting direction for future work.From the perspective of application, while this method can yield accurate estimations, it may pose risks for fragile objects, as the interaction required for property inference could potentially cause damage. Moreover, the computational demands of our framework are substantial which require at least 1.5 hours to simultaneously recover both the geometry and physical properties of each object. Future work could explore leveraging multi-model large language models [62] and large reconstruction models [63] [64] [65] [66] to facilitate the recovery process.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: These details are described in the experiment section and appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: The error bars are reported in the experiments section.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: This is described in the appendix section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Justification: There is no societal impact of the work performed. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper poses no such risks.Answer: [NA] Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Motion-factorized Dynamic 3D Gaussian NetworkWe employ temporal and positional encoding to the time t and position µ 0 , respectively, to introduce features with various frequencies. Specifically, the encoding module is denoted as γ(x) = sin(2 k πx), cos(2 k πx) L-1 k=0 , where L = 10 for both t and µ 0 . All the modules within the proposed network are composed of fully connected layers. The intermediate layers are uniformly designed, featuring both input and output channels configured to 256, and employ ReLU activation. For training, we adhere to the protocol established in [16] , utilizing the Adam optimizer [67] with the same learning rate as specified in [16] . The total number of iterations is set at 40,000, with densification and pruning operations conducted every 500 steps until reaching 15,000 iterations. Additionally, the number of motions N m is set to 8 for all objects in our network. λ 1 and λ 2 in Eqn. 4 are all set to 1. All the experiments are conducted on a single A10 GPU. When addressing the deformation of objects such as fluids or granular media, the network may struggle to fit transformations accurately due to significant discrepancies between the canonical and target shapes. As a compensatory mechanism, the network may employ Gaussians with enlarged scales to mitigate shape distortions during image rendering. This effect is visualized in the top row of Fig. 6 . To rectify this issue, we implement scale regularization during network training, which enforces Gaussian kernels to maintain smaller scales. The efficacy of this operation is demonstrated in the second row of Fig. 6 , where it is evident that scale regularization enables the reconstruction of more precise shapes for rendering.To further evaluate the performance of our method in terms of novel view synthesis, we conduct the experiment on the D-NeRF [25] dataset, which is a widely used benchmark consisting of moving items with data captured by a monocular camera. We compute PSNR on the D-NeRF test set and compare our method with previous dynamic approaches, including Tensor4D [68] , K-Planes [69] , TiNeuVox [70] , and DefGS [16] . The results, reported in Tab. 5, demonstrate the proposed dynamic 3D Gaussian pipeline can also achieve superior performance on rendering. In Alg. 1, the number of iterations, denoted as n u , is uniformly set to 4 for all objects. We set the initial grid size ∆x according to the volume of the object. For most objects, ∆x = 0.1, while for small items such as toothpaste in PAC-NeRF dataset, ∆x = 0.01. The parameters th min and th max are set to 0.5 and 0.8, respectively. The resulting particle count ranges from approximately 50,000 to 100,000.Fig. 7 visualizes the filling results of our proposed coarse-to-fine strategy with different numbers of iterations, along with the results from PAC-NeRF and ground-truth shapes. The qualitative results show that our method can generate more accurate shapes compared with PAC-NeRF, which tends to recover over-large shapes. We should note that we cannot recover the cat-shaped object as in [12] , though we use the code officially implemented by PAC-NeRF without any modification.The training process is divided into two sub-processes, where we perform system identification after estimating the initial velocity of the object using the first three frames of data. Both processes use Adam [67] optimizer to tune the parameters.Besides performing evaluation on the simulated future states in Sec. 5.2, we also evaluate CD and EMD on states existing in the training data, and the results are reported in Tab. 6. It is obvious to see that our method outperforms the baselines by a large margin, which further demonstrates the performance of our method in terms of reconstruction and identification. Each material contains 10 instances (5 for granular material) with various object orientations, initial velocities, and physical parameters. To evaluate the necessity of 2D mask supervision, we perform system identification on 45 crossshaped object instances in the PAC-NeRF dataset by our method but with only object surface supervision. The results are reported in Tab. 9. It is obvious to see that combining both 2D and 3D shapes as supervision can achieve more accurate performance compared to using 3D shapes only. Therefore, we believe that utilizing 2D mask supervision to some extent makes up for the errors introduced by the 3D object surfaces extracted from dynamic 3D Gaussians. In this work, we simulate five types of materials, including elasticity, plasticine, granular media, Newtonian fluids, and non-Newtonian fluids. Each material exhibits distinct physical properties. We provide a brief introduction to the properties of each material.Elasticity: The Young's modulus (E) is a measure of the stiffness of a solid material, quantifying the relationship between stress and strain in a material under elastic deformation. The Poisson's ratio (ν) describes the tendency of a material to expand or contract along its width when it is stretched or compressed along its length.Plasticine: The yield stress (τ Y ) is the minimum stress that a material requires to transition from elastic deformation to plastic deformation, marking the onset of permanent deformation. Both Young's modulus (E) and Poisson's ratio (ν) exhibit characteristics similar to those of elastic materials.Granular Media: The friction angle (θ f ric ) is a measure of the inherent resistance of a granular material to sliding or shearing, directly related to the angle at which a material can be piled without slumping.Newtonian fluids: The bulk modulus (κ) is a measure of a material's resistance to uniform compression, quantifying how it a given amount of external pressure. Fluid viscosity describes a fluid's resistance to flow, quantifying how much it resists deformation at a given rate.Non-Newtonian fluids: The plasticity viscosity (η) refers to the measure of a viscoplastic material's resistance to deformation, which defines how it behaves under stress beyond its yield point. The bulk modulus (κ) and fluid viscosity (µ) are comparable to those of Newtonian fluids, while the yield stress (τ Y ) is akin to that of plasticine.A constitutive model describes how a material responds to stress, strain, or other external forces. It defines the material's behavior by relating stress and strain through constitutive equations, which can capture complex behaviors such as elasticity, plasticity, and fracture. The MPM simulator is capable of modeling a diverse range of materials by employing various constitutive models. In this work, we have implemented simulations for five distinct types of materials: elasticity, plasticine, granular, Newtonian fluids, and non-Newtonian fluids.Elasticity. We use the Neo-Hookean model, which is a common nonlinear hyperelastic model, to simulate the elasticity of materials and predict deformations. The Cauchy stress for this model is defined byEQUATIONwhere the F is the deformation gradient, J = det(F) and µ, λ are the Lamé parameters, which are related to the material properties of Young's modulus (E) and Poisson's ratio (ν) as:EQUATIONPlasticine. We use the Saint Venant-Kirchhoff Model (StVK) together with von Mises yield criterion to simulate the plasticine. For this model, the stess is defined as:EQUATIONwhere G = 1 2 (F ⊺ F -I) is the Green strain. The von Mises yield criterion serves as a tool to assess whether the deformation exceeds the recoverable limit. The deformation gradient will be mapped back onto the boundary of elastic region using the following projection:EQUATIONwhere the δγ = ∥ε∥ -τ Y 2µ , ϵ = log(Σ) is the normalized Hencky strain. The U, Σ and V can be obtained by performing Singular Value Decomposition (SVD) on deformation gradient F. Granular Media. Similar to plasticine, the StVK constitutive model is used to simulate granular media. Drucker-Prager yield criteria [48] is selected as the yielding condition. It is defined as follows:EQUATION)where d is the spatial dimension, α = 2 3 2 sin θ f ric 3-sin θ f ric and θ f ric is the friction angle. The deformation gradient return mapping is defined byEQUATION)Newtonian Fluid. We adopt the approach used in PAC-NeRF [12] , which employs a J-based fluid model combined with a viscosity term to simulate Newtonian fluids. The stress for this model is defined byEQUATIONwhere µ and κ represent the fluid viscosity and the bulk modulus, respectively.Non-Newtonian Fluid. We employ model to simulate non-Newtonian Although we continue to utilize the von Mises criteria to delineate the elastic region, the presence of viscoplasticity implies that deformation will not be immediately reverted onto the yield surface. It is defined as follows:EQUATIONEQUATION)where d is the spatial dimension. The U, Σ and V can be obtained by performing Singular Value Decomposition (SVD) on deformation gradient F.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: This paper has received unanimous acceptance recommendation (2 WA, 2A), it is addressing an important problem and shows potential for real world applications. It is recommended for oral presentation for its high impact.

Reviews: review: paper_summary: paper_summary: This paper presents an approach for estimating the geometry and physical properties of objects through visual observations using 3D Gaussian representations. The method employs a dynamic 3D Gaussian framework to reconstruct objects as point sets over time and a coarse-to-fine filling strategy to generate density fields. This facilitates the extraction of object continuums and integrates Gaussian attributes, aiding in rendering object masks during simulations for implicit shape guidance.The extracted geometries are then used to guide physical property estimation through differentiable MPM simulation.
strength_weakness: strength_weakness: Some of the technical terms are misused, making the paper confusing to read. For example, "implicit shape representation" is repeatedly used to refer to the rendered object image masks, but "implicit" generally refers to using functions (parametric or neural networks) to represent shapes, where shapes must be retrieved through function evaluations, hence "implicit". Please correct this terminology, as the geometries in this work are represented using GS, which are explicit representations. Additionally, some of the results presentations are confusing and could benefit from clearer explanations and more organized presentation (see below). Clarifying these aspects would greatly enhance the paper's readability and overall impact.
questions: questions: 1. Table 3: Can you include the ground truth values in the table so that readers understand what to expect?
2. System Identification: How do you set the initial parameters for the system identification? Are the optimizations sensitive to initial conditions?
3. Figure 4: Would mask-based supervision fail when the estimated shapes are significantly different from the ground truth? Have you observed any cases where this occurs?
4. Figure 1a: There is a typo in the figure caption. Change "caption" to "capture."
5. References: It would be beneficial to include references to works on differentiable cloth simulation for system identification and inverse problems, such as "Differentiable Cloth Simulation for Inverse Problems" by Liang et al. and "DiffCloth: Differentiable Cloth Simulation with Dry Frictional Contact" by Li et al.
limitations: limitations: The authors have discussed the limitations of the method regarding known camera parameters, assumption of known material models, and continuum mechanics. However, I do wonder about the failure cases of the method, if there are any. Understanding specific scenarios where the method does not perform well would provide valuable insights and help guide future improvements.
paper_summary: paper_summary: The manuscript proposes a novel hybrid framework that leverages 3D Gaussian representations for system identification from visual observations. The framework captures both explicit and implicit shapes using dynamic 3D Gaussian reconstruction and a coarse-to-fine filling strategy to generate density fields. These fields are used to sample continuum particles for simulation and extract object surfaces, which can render object masks during simulations to guide physical property estimation.
strength_weakness: strength_weakness: 1. In the real-world experiment, I found the authors switched to FEM for deformable body simulation, which conflicts with the MPM simulator used in their pipeline. I think it needs justifications.
2. Some wordings are confusing: e.g., line 166, do you mean effective instead of efficient?
questions: questions: 1. I wonder what makes the difference between the proposed method and PAC-NeRF in the infilled particle generation.
2. If I understood correctly, the motion network only takes time as input. I wonder if it helps by combining the temporal encoding from DiffAqua [1] to further capture the low and high frequencies.
3. The infilling algorithm seems expensive since the complexity grows exponentially. Did you try using Octree [2] or similar algorithm to speed it up?

[1] Ma, Pingchuan, et al. "Diffaqua: A differentiable computational design pipeline for soft underwater swimmers with shape interpolation." ACM Transactions on Graphics (TOG) 40.4 (2021): 1-14.

[2] Meagher, Donald JR. Octree encoding: A new technique for the representation, manipulation and display of arbitrary 3-d objects by computer. Electrical and Systems Engineering Department Rensseiaer Polytechnic Institute Image Processing Laboratory, 1980.
limitations: limitations: Yes.
paper_summary: paper_summary: This paper introduces a novel hybrid method that leverages 3D Gaussian representation and continuum to estimate physical properties of deformable objects. From multi-view video, the Gaussian- informed continuum can be extracted and then combined with material point method (MPM) simulation to train the whole pipeline by using both 3D shape and 2D as supervision. The experiments show that the proposed method outperforms previous approaches based on continuum dynamics or 3D Gaussian representation for dynamic reconstruction, system identification or other real-world applications.
strength_weakness: strength_weakness: 1. The authors stated that such a lightweight architecture of the motion-factorized dynamic 3D Gaussian network is sufficient for complex motions rather than modeling each basis with an independent network (line 168-171) while lacking an experimental proof. 

2. The representation in Section 4.3 is lacking in elaboration and should accompany with more details in the supplementary. Please give some detailed elaboration for Section 4.3 about Gaussian-informed continuum, e.g., notations used in Algorithm 1, dimensions of the variables, etc.
questions: questions: 1. It would be nice to see a  comparison between the choice of Motion-factorized dynamic 3D Gaussian network and previous architectures ?

2. It is important that the experimentation on complex motions must be performed in order to truly understand the strength of the proposed  method.
limitations: limitations: To some extent, yes.
paper_summary: paper_summary: The paper proposes an improvement of PAC-NeRF for the task of estimating material properties from multiview video using 3D Gaussian Splatting (3DGS). Instead of estimating geometry solely based on the first frame like PAC-NeRF, the proposed method uses 4D Gaussian Splatting (4DGS) with reduced order modeling to construct 4D geometry, enabling the use of 3D supervision. A coarse-to-fine internal filling strategy is introduced to ensure that the simulation operates on a solid volume. 2D mask loss is used for additional supervision.
strength_weakness: strength_weakness: Some symbols are not defined clearly, making it hard to follow at times. For example, $Discretize$ operator in Line 214; $\tilde{P}$ and $F$ are not defined in the text. I need to guess from Alg 1.
questions: questions: If the 4D geometry is accurate enough, is it adequate to only use 3D supervision? Ablation studies are needed to validate the necessity of 2D mask supervision.

Coarse-to-fine density field creation: At the beginning, the reconstruction contour is much larger than the object. How does it shrink to the actual boundary? The $TrilinearInterpolation$ operator will not shrink the contour, and there is no operator to assign zeros.

In the simulation, how do the Gaussian kernel scales evolve? The paper seems to assume isotropic kernels, but physical deformation can transform the sphere into an ellipse. How is this addressed?
limitations: limitations: Limitations are well discussed.

Comments: comment: Thanks for your feedback and we are pleased that our response has successfully addressed your concerns.
Thanks for your feedback and we are pleased that our response has successfully addressed your concerns.
Thank you for the rebuttal. I do not have further questions.
Thank you for the reply. The replies addressed my concerns and there I'll raise my score.
Thanks for the reviewer's feedback. We are pleased that our response has addressed the concerns. We would appreciate that if the reviewer could re-evaluate the review score. (Note: since the openreview system had issue earlier that reviewers cannot receive the email after posting comments, we delete the old comment and resend it.)
I thank the authors for the rebuttal.

I don't have any questions at the moment.
Thanks for the reviewer's feedback. We are pleased that our response has addressed the concerns. We admit that although the orthogonality assumption works on our application, there might be practical challenges when aligning MPM and FEM, especially on more complex tasks. We'll moderate our stance in the revised version.
I appreciate the authors' responses, which addressed most of my concerns and questions. Regarding the first question, I suggest moderating the stance on the orthogonality between the numerical method and the corresponding physical parameters. While the claim is theoretically sound, practitioners often encounter significant misalignment between MPM and FEM, and I fear this statement might be misleading. It would be beneficial to see dedicated work on FEM due to its realism and applicability in robotics-related tasks. However, I understand this would require considerable separate effort and merits its own publication. Therefore, I will raise my score to 7 to advocate for acceptance.