Title: Gaussian-Augmented Physics Simulation and System Identification with Complex Colliders
Venue: NeurIPS 2025
Accepted: yes
Keywords: System Identifications from Videos

Gaussian Splatting

Physics Simulation
Abstract: System identification involving the geometry, appearance, and physical properties from video observations is a challenging task with applications in robotics and graphics. Recent approaches have relied on fully differentiable Material Point Method (MPM) and rendering for simultaneous optimization of these properties. However, they are limited to simplified object-environment interactions with planar colliders and fail in more challenging scenarios where objects collide with non-planar surfaces. We propose AS-DiffMPM, a differentiable MPM framework that enables physical property estimation with arbitrarily shaped colliders. Our approach extends existing methods by incorporating a differentiable collision handling mechanism, allowing the target object to interact with complex rigid bodies while maintaining end-to-end optimization. We show AS-DiffMPM can be easily interfaced with various novel view synthesis methods as a framework for system identification from visual observations.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of system identification using arbitrarily shaped colliders. To achieve this, it enhances an existing differentiable Material Point Method (MPM) by incorporating a differentiable collision handling mechanism. This addition enables the estimation of physical properties using differentiable renderers and simulators. Experiments conducted on several synthetic benchmarks demonstrated the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. The paper is well written and easy to follow.\n\n2. It extends the existing differentiable MPM simulator with support for a differentiable collision handling strategy that accommodates arbitrarily shaped colliders.\n\n3. The proposed collision handling strategy is generalized to primitives with associated normals, e.g., 2D Gaussian disks.\n\n**Weaknesses**:\n\n1. The proposed method aims to address system identification with complex colliders. While it did present results of non-planar colliders, it still lacks *variety* and *complexity* regarding the (a) location, (b) size, (c) shape, and (d) number of colliders in the considered scenarios. \n\n2. The experiments only show results of a limited range of material types, i.e., only consider fluids and granular media. However, previous work [1] also considers elastic and plastic materials for experiments. This observation may indicate *a lack of generality* of the proposed framework.\n\n3. The method claims that an ad hoc solution for system identification with complex colliders is necessary, yet it only presents results based on synthetic data. In a synthetic environment, we can actually estimate the material properties of objects by using a planar collider to collect simulation trajectories (since we have full control of the simulation). Consequently, a more practical application of the proposed method should involve using multi-view videos captured from *real-world environments*, which show objects colliding with complex colliders. However, the method lacks results that reflect this practical scenario.\n\nQUESTIONS:\n1.\tPer Weakness 1, could the authors present more results, including but not limited to the following scenarios: (1) move the collider to a corner of the floor instead of directly below the object, (2) make the size of the collider (i.e., the bounding-box size) much greater than that of the object, (3) use an open-surface collider, e.g., a bowl-shaped mesh or a cylinder open at the ends, and (4) place multiple colliders side by side?\n\n2.\tPer Weakness 2, could the authors present results of system identification on elastic and plastic objects?\n\n3.\tPer Weakness 3, could the authors conduct at least one real-world experiment to verify its practical usefulness?\n\n4.\tHow long does it take (on average) to finish an identification task for the proposed method and the baseline approaches?\n\nLIMITATIONS:\nThe authors list several limitations in the paper, including the requirement for rigid-body reconstruction and the use of static colliders. \n\nThe objects considered in the paper appear to have only one shape (i.e., the cross shape) and uniform material properties. It would be interesting to have some results on objects with other shapes and heterogeneous material compositions.\n\nAnother interesting experiment would be to replace the colliders of rigid objects with those of deformable objects.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AS-DiffMPM, a differentiable Material Point Method framework that supports system identification of continuum objects colliding with arbitrarily shaped rigid bodies. Unlike prior works that only handle planar boundaries, AS-DiffMPM incorporates a particle-wise collision handling mechanism—based on Compatible Particle-in-Cell (CPIC)—to resolve interactions with complex meshes or 2D Gaussian splat primitives. The method is integrated with both voxel-based NeRF and point-based Gaussian splatting renderers to optimize physical parameters from multi-view videos. Extensive experiments demonstrate its effectiveness on Newtonian, non-Newtonian, and granular materials, using both ground-truth particle trajectories and visual observations for supervision\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1.Extends differentiable MPM to arbitrary collider geometries via CPIC, enabling accurate, particle-wise velocity corrections during P2G and G2P stages.\n\n2.Provides seamless interfaces to voxel-grid NeRFs and 2D Gaussian Splatting methods, supporting end-to-end gradient flow from photometric losses to physical parameters .\n\n3.Benchmarks across three material types (Newtonian, non-Newtonian, granular) and three colliders (Box, Bunny, Armadillo), reporting both trajectory-based and vision-based identification results, with clear error statistics and comparisons against two baselines (GOP-DiffMPM, RP-DiffMPM) and multiple rendering backends.\n\n4.Authors plan to release code, facilitating reproducibility and further research in differentiable physics and system identification.\n\n\nWeaknesses\n1. MPM is fundamentally a Galerkin-based method—yet the paper does not analyze whether the differentiable formulation preserves energy conservation, admits closed-form solutions, or under what conditions it remains stable.\n\n2. All P2G/G2P and collision computations occur on the background grid; the results are therefore highly dependent on grid resolution, with coarser grids potentially introducing aliasing or inaccuracy.\n\n3. The paper positions CPIC-based collision handling as a more efficient alternative to hard-clamp methods, compensating for approximation errors with speed. However, it never quantifies whether the computational effort saved by avoiding per-particle constraints is fully realized once the overhead of scatter/gather operations and backpropagation through the grid is accounted for. This raises the question of whether any real efficiency advantage survives in practical, high-resolution scenarios.\n\n4. There is no discussion of the method’s numerical properties: e.g., under what time step or grid refinement regimes the optimization converges, whether the gradient signals remain well-conditioned, or if any spurious oscillations arise during backpropagation.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nSee Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces AS-DiffMPM, a differentiable Material Point Method (MPM) framework that supports system identification in scenes involving interactions between deformable objects and arbitrarily shaped rigid body colliders. The authors extend differentiable MPM techniques with a particle-wise collision handling mechanism inspired by CPIC, enabling gradient-based physical parameter estimation in the presence of non-planar, complex rigid bodies. AS-DiffMPM is evaluated in combination with various rendering pipelines (voxel-based NeRF, 2DGS, MDyn-3DGS), and both qualitative and quantitative results demonstrate its effectiveness across multiple material types and collider\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. [Exploration of non-planar colliders] The paper addresses a previously underexplored problem—system identification with non-planar rigid body colliders—by extending differentiable MPM to handle complex geometry interactions. This is a novel and necessary advancement over prior works like PAC-NeRF and GIC. By supporting arbitrarily shaped colliders, AS-DiffMPM enables more realistic physics-based analysis and simulation. This has clear value for downstream applications in robotics, graphics, and 3D perception.\n2. [Good compatibility] AS-DiffMPM is designed to be compatible with multiple rendering pipelines, increasing its adaptability and potential for broad adoption.\n3. [Technical novelty of Integrating CPIC] The proposed method integrate CPIP to project the rigid body onto a grid-based distance field so that it can handle collisions in a particle-wise manner and apply velocity corrections on particles in local neighborhood. \n4. [Extensive experimental evaluation] The authors conduct extensive evaluations on both particle-trajectory-based and vision-based system identification tasks across diverse material types and collider geometries, showing superior or competitive performance.\n\n\nWeaknesses\n1. [No comparison with existing method on planar colliders] Although the paper emphasizes AS-DiffMPM’s capability to handle interactions with arbitrarily shaped rigid bodies, it does not report results on planar colliders—a standard experimental setting used in prior work such as PAC-NeRF. This omission makes it difficult to assess whether the proposed method remains competitive in the regimes where earlier methods perform well, and prevents a direct comparison under identical conditions.\n2. [Lack of qualitative comparison with other methods] The qualitative visualizations provided do not include side-by-side comparisons between AS-DiffMPM, baseline methods (e.g., GOP-DiffMPM), and ground truth. Such comparisons are critical for visually assessing the fidelity of simulated trajectories and would significantly strengthen the empirical claims of the paper.\n\nQUESTIONS:\n1. In Table 1 (System Identification from Particle Trajectories), AS-DiffMPM appears to outperform the baselines predominantly in the case of Non-Newtonian materials, whereas for Newtonian and Granular materials, the baseline methods achieve lower errors in many settings. Could the authors elaborate on the underlying reasons for this discrepancy? What characteristics of Non-Newtonian materials make AS-DiffMPM particularly effective?\n2. In Table 3 (Geometric Error Evaluation), AS-DiffMPM only outperforms baselines in the case of Granular materials, which does not fully align with the trends observed in Table 1 for System Identification. Could the authors provide a deeper analysis of under what conditions (material type, collider complexity, or metric) AS-DiffMPM provides superior performance compared to GOP-DiffMPM or RP-DiffMPM?\n3. The proposed method is titled Any-Shape Differentiable Material Point Method, yet no experimental results are reported for interactions with planar colliders—scenarios where prior works (e.g., PAC-NeRF, GIC) are typically evaluated. Could the authors clarify whether AS-DiffMPM maintains competitive performance in this condition?\n4. The method is evaluated in conjunction with several rendering pipelines (DVGO, 2DGS, and MDyn-3DGS), which differ substantially in complexity and representation. Could the authors quantify the differences in training time and rendering latency among these models in the context of the system identification task? A breakdown of computational efficiency would be helpful for practitioners deciding between rendering backends.\n\nLIMITATIONS:\nYes. The paper explicitly addresses its limitations, most notably the assumption of static rigid bodies and the absence of real-world evaluations. The authors also mention that their approach could be extended to moving colliders but defer such experimentation to future work. This upfront discussion is appreciated.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to recover physical properties from video for fluid or granular media. The approach combines existing MP methods for physics simulation with differrentiable rendering. Different to prior work, arbitrary colliders (beyond planes) are supported. The contributions are on how to relate these colliders with the grid&point representations in MPM. Notably the reconstruction is done jointly with 3D reconstruction of the shape through differentiable rendering, without an intermediate step to track visual features.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- the implementation and experiments are extensive, including both Voxel-based NeRF and Gaussian Splatting as point-based methods.\n- experiments are on established benchmarks and i comparison to existing methods\n\n- the addition of arbitrary colliders appears as rather a technical detail yet there are a few considerations to be made and the evaluation is decent, so there is still value for the community.\n\nWeaknesses:\n- experiments are only on synthetic sequences, would you think it generalizes to real scenes? Should be included in limitation section.\n- It would be better to up-front explain that the focus is on Newtonian fluids, non-Newtonian fluids, and granular media. From the intro I was more expecting deformable solid objects to be tackled, as it was very general.\n- in the eperiment section authors refer largely to prior work. To be self contained, it would be beneficial to know how close the initial parameter guesses are to the GT etc.\n\nQUESTIONS:\n- how important are tne #of sequences used, when does it degrade?\n- are there other limitations, have other material types been tested? Would certain shapes be problematic? \n- why is the collider shape imported and not reconstructed from multi-view?\n\nLIMITATIONS:\n- See questions above, I find the current limitation section a little shallow as it adresses only once conceptual issue but leavs out other.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}