NeuMA: Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics
Authors: Junyi Cao
Shanyan Guan
Yanhao Ge
Wei Li
Xiaokang Yang
Chao Ma
Venue: NeurIPS 2024 poster (2024)
Keywords: Intuitive Physics
Differentiable Renderer
Neural Simulation
Primary area: neuroscience_and_cognitive_science
TL;DR: We propose a residual adaptation paradigm for the visual grounding of intrinsic dynamics.

Abstract: While humans effortlessly discern intrinsic dynamics and adapt to new scenarios, modern AI systems often struggle. Current methods for visual grounding of dynamics either use pure neural-network-based simulators (black box), which may violate physical laws, or traditional physical simulators (white box), which rely on expert-defined equations that may not fully capture actual dynamics. We propose the Neural Material Adaptor (NeuMA), which integrates existing physical laws with learned corrections, facilitating accurate learning of actual dynamics while maintaining the generalizability and interpretability of physical priors. Additionally, we propose Particle-GS, a particle-driven 3D Gaussian Splatting variant that bridges simulation and observed images, allowing back-propagate image gradients to optimize the simulator. Comprehensive experiments on various dynamics in terms of grounded particle accuracy, dynamic rendering quality, and generalization ability demonstrate that NeuMA can accurately capture intrinsic dynamics. Project Page: https://xjay18.github.io/projects/neuma.html.

NeuMA: Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics
Junyi Cao
Shanyan Guan Yanhao Ge
Wei Li
Xiaokang Yang' Chao Ma
MoE Key Lab of Artificial Intelligence, Al Institute, Shanghai Jiao Tong University
2 vivo Mobile Communication Co., Ltd. {junyicao, xkyang, chaoma)@ejju.edae edu. {guanshanyan, halege, liwei. Swwei.yygh@@vivo.coo
Abstract
While humans effortlessly discern intrinsic dynamics and adapt to new scenarios, modern Al systems often struggle. Current methods for visual grounding of dynamics either use pure neural-network. meall--ewwoorbbbeee simulators (black box), which may violate physical laws, or traditional physical simulators (white box). which rely on expert-defined equations that may not fully capture actual dynamics. We propose the Neural Material Adaptor (NeuMA), which integrates existing physical laws with learned corrections, facilitating accurate learning of actual dynamics while maintaining the generalizability and interpretability of physical priors. Additionally, we propose Particle-Ge GSticleeGSs,, particle-driven 3D Gaussian Splatting variant that bridges simulation and observed images, allowing prok-propagate image gradients to optimize the simulator. Comprehensive experiments on various dynamics in terms of grounded particle accuracy, dynamic rendering quality, and generalization ability demonstrate that NeuMA can accurately capture intrinsic dynamics. Project Page: https::/xjay/l github. wtt/projects//enneee html
1 Introduction
Teaching a machine to "see, understand, and reason" the physical world like humans has been a fundamental pursuit of machine learning and cognitive science. Imagine the flexibility, robustness, and generalizability of human intelligence: simply by observing an object falling on the ground and bouncing up, a young child make a plausible guess of its intrinsic dynamies (by telling what material it is made of), adapt to new scenarios (involving new objects and initial conditions) and predict interactions with other objects. However, modern AI systems still fail match this cognitive ability, known as visual grounding, of humans.
Many efforts have thus been made impart the ability of visual grounding of dynamics AI systems [5] 25, 26 28 45], typically by training a differentiable simulator [14] 3T 34] 72 90 with pixel supervision from differentiable renderer [37 58). Depending on the formulation of the simulator, current works can be categorized into two types: black box or white box. Black box approaches [26, 89) directly
neural network model the dynamic transition. Due the deep coupling among extrinsic attributes (e.g., geometry) and intrinsic (physical) motion during rendering, black box approaches are prone to violating physical laws and have limited generalization capability without any physics constraints on the transition process [25] 53
In contrast, white box methods 45
traditional physical simulators (e.g., Material Point Method 34) to approximate object dynamics explicitly with diffialadifereeniaal equations (PDE), a.k.a., motion equations. These methods back-propagate pixel differences from renderer to the
60 95
'This work done during internship vivo.
Project lead.
*Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Expert-designed Physical Models
Actual Intrinsic Dynamics mi hontor's Notooss Du P Dt sqpaltion
P = #CF= (P9 + (eg Netellssinnn Faeidyy
Cx
Neural Adaptor
(Z von Hiss Hazieity
Pf
GAP
Correction
Grounded Dynamics
Visual Observation
Figure The core idea of NeuMA:
Learning to correct existing expert knowledge on object
motions by fitting neural material adaptor to ground-truth visual observations.
physical coefficients (e.g., Young's modulus, Poisson's ratio) of an analytical simulator. Naturally, the estimated physical coefficients can be applied to new scenes. However, the motion equations are expert-designed and may not perfectly align with the actual dynamies. This raises the key question we ask in this work: how to accurately infer the actual intrinsic dynamics from the visual observations?
To answer this question, we propose the Neural Material Adaptor (NeuMA), learning-based model with
priors. As shown in Figure L the core idea of NeuMA is to formulate the learning of intrinsic dynamics specified by physical law M residual adaptation paradigm: M == Mo + AM, where .M is the expert-designed physical models, and AM represents the correction term grounding the observed images. This paradigm enjoys two advancements: one hand, unlike white-box methods solely relying on Ma. NeuMA can model the actual intrinsic dynamics by optimizing AM align with observations (more accurate and flexible); the other hand, unlike black-box methods ignoring any physical priors, NeuMA fits the actual dynamics based on commonly-agreed physical models Mo (more generalizable and physically interpretable). Specifically, built upon the progress in physics simulation, NeuMA uses the Neural Constitutive Laws (NCLaw) [53] to formulate Mo. which is network that encodes existing physical priors and constraints. As for A,, we use low-rank adaptor (30] that enjoys the efficient adaptation and preservation of the prior My. Then, to supervise the simulation module via rendering, we propose Particle-GS, differentiable renderer in the form of particle-driven 3D Gaussian Splatting (3DGS) [37 variant. It leverages the predicted motion of particles to drive Gaussian kernels through pre-defined relationship between particles and kernels. We use Particle-GS as the bridge from simulation to visual images, which allows marrying image gradients to optimize the simulator.
We evaluate NeuMA on various dynamic scenes with different materials and initial conditions. It shows competitive results object dynamics grounding and dynamic scene rendering while achieving good generalization to novel shapes, multi-object interactions, and extended-time prediction.
2 Problem Formulation
We tackle the problem of grounding the intrinsic dynamics of object from sequence of visual observation / = { 2 IT). Following common practice [18] [53 60, 97], in this work, we adopt the elastodynamic equation [22) to describe the dynamical systems:
(1) where V P is the divergence of the stress tensor P. p is the object density, and b is the given body force. Besides, Φb denotes the displacement field describe the object's deformation, and お is its acceleration. To solve Equation (I) material models M (see Appendix for the background), which delineate how the object responds (depicted by P) under deformations (depicted by Φ), should be prescribed. To realize differentiable grounding from observed videos, we then parameterize M with learnable parameters 9 (i.e., Ma) 332 [3 79]. In summary, the dynamical system governed by Equation (T] can be described by a transition model S: 8,41 = ((ss;MM) where s, are the particle states g.. positions and velocities) at the 1-th time step.
pŌ= - . pb.
To achieve dynamics grounding using only visual data, we develop a differentiable renderer R. This enables pixel supervision to be used for backpropagation to the transition model S, allowing for the

---

Stage I. Initial State Acquisition
Stage III. Dynamic Scene Rendering
N
Reconstruction
Binding Basad Kernel Deformation
Binding
Eq. (7-8)
Particle Sampling
t=t
Splatting
1+3
Pei Vt
Stage Physics Simulation
Pi+1 V++1
St
F
AM
Pt
pettrial
AM
Fe
r+1
St+1
p = V pb
Mi
M
Eastic Model
PDE-Based Time Integration
Plastic Model
Figure 2: The pipeline of NeuMA for visual grounding. During Stage first reconstruct the 3D Gaussian kernels of the foreground object using masked multi-view images. Then, we uniformly sample the initial physical particles from the object volume and bind them the reconstructed Gaussian kernels. In Stage II, we integrate the neural material adaptor into the PDE-based simulation framework estimate the actual dynamics. In Stage III, we deform the Gaussian kernels according to the binding relationship (pre-computed in Stage I) and then render 2D images. The neural material adaptor is trained end-to-end using the difference between the rendered and observed images.
optimization of the neural material model Ma- Specifically, R. learns to synthesize an image I given the state 8t, i.e., I, (s;;; Kr, Pt), where K,, Pr denotes the camera's intrinsic and extrinsic matrix at the 1-th time step. The optimal weights 0* of MM is obtained by minimizing the visual difference Ly between predicted images and the ground-truv -tuth observations:
0" arg min Ly arg min | Iell.-
Method
We introduce NeuMA, ceural-network-bee based material adaptor, to learn the intrinsic dynamics specified by the material model M from only visual observations 7. Our key insight is to implicitly represent the material model learnable, residual term AM3 based widely accepted physical priors Mo (e.g., neo-Hookeat elasticity for elastic objects [83, etc.). This gives:
Mg = Mo + AMos
where implement AM using the low-rank adaptation (LoRA) 30 restrict our correction not to overturn the physical priors. We embed the whole material model Ma into a differentiable simulator S, followed by a differentiable renderer R to enable supervised training directly the outputs (ie.,, synthesized images) of R. In view of recent advances in particle-based physical simulation, which achieve satisfactory performance for visual dynamics grounding [26 45
instantiate S as the well-known Material Point Method (MPM) [31 34 (2) since it can handle complex motion simulation for various materials and easily back-propagate gradients for end-to-end training (26] 45 (85). Please refer to Appendix D for more details about MPM. We further embrace ti-the-art art scene representation technique, 3D Gaussian Splatting (3DGS) 37], build up our differentiable renderer, Particle-Gr GS due to its explicit and flexible representation for efficient deformation. Please see Appendix for details about 3DGS.
the state-of- ffate-of-the-wee
The overall pipeline for NeuMA to conduct visual dynamics grounding, as shown in Figure includes three steps: initial state acquisition, physics simulation, and dynamic scene rendering.
3.1 Initial State Acquisition
... Start with a set of calibrated multi-view observations To = (T) .,...,, depicting an object the initial time step (i.e., f. = 0), we are interested in obtaining (1) the 3D Gaussian kernels
X(i) (x(1), (i), A(i), c(i)) representing the object, where æ(1), (i), A(i), c(i) denote the

---

center, opacity, covariance matrix, and spherical harmonic coefficients of the i-th Gaussian kemnl;; and (2) the initial state So (Po: Vo. Fr) for physics simulation, where Po: vo. F are the initial particle positions, velocities, and elastic deformation gradients, respectively.
To obtain the 3D representation, interest out of the set of initial image frames by running an image matting algorithm (e.g.. [38] 33)).
first follow PAC-NeRF 45 separate foreground objects of
We then adopt the standard training pipeline of 3DGS (optionally with the anisotropy regularizer 85 to encourage Gaussian kernels to be more evenly shaped to reduce spiky artifacts) to
used in get a dedicated object reconstruction represented by a set of Gaussian kemelss (K(i)).
Regarding the initial partiele state, a precise de- scription of object boundaries is critical for the sim- ulator to properly handle the interaction between objects. Thus, we explicitly reconstruct a surface mesh for the foreground object via multi-view sur- face reconstruction [92]. We uniformly sample points inside the mesh volume as the initial particle positions Pu- To acquire the initial particle veloc- ities un. we additionally use first few frames from the set of visual observations I and proppaate the visual difference loss L. through the differen- tiable renderer R and simulator S to optimize vo while keeping other parameters fixed. The initial elastic deformation gradients Fū are set to the iden- tity matrices. In addition, we compute the mass of particles as the product of a constant density p and their volume, which is estimated by taking average the object volume over particles inside.
Algorithm 1: Particle Binding
Input: Gaussian centers (z{(t) (
Gaussian covariance { {A
particle positions (po ( j=1
confidence threshold 7
Output: Binding matrix B
1 zeros(NK. Np); 2 for + 1 to NN do for j < to Np do // Difference vector
dp Po(i) (i);
// Mahalanobis distance
dm = dØ (()
// Check the threshold
if dm S chi2(r) then
B(i,)) =1;
Particle-GS. Although prior works [17
85
95
end
demonstrated the feasibility of direct adaptation of 9 3DGS to particle-based physics with an identical mapping between 3D Gaussian kernels and phys- ical particles, we argue that such kernel-particle relationship is sub-optimal. In particular, the scene reconstruction achieved by 3DGS is primarily optimized for visual appearance, leading to an unbal- anced kernel distribution ssprbbutioon-spaee in texture-less regions and dense in texture-rich areas. Thus, direct simulation based on the original Gaussian kernels may result unrealistic outcomes. To ameliorate this issue, we propose Baricle-GSS, which models the hierarchical relationship between Gaussian kernels and simulation particles via novel particle binding mechanism.
end
// Normalize for each row (sm((((L)))) B(i,:)
end
Particle Binding. Refer to Algorithm | start by binding each Gaussian kernel IC(i) with set of nearby particles based on the Mahalanobis distance dm [10] [57] between the kernel and particle position (). We pre-define confidence threshold and use the chi-squared test chi2(.) [24 to check whether dm Schi2(r) for each kernel-particle pair. We record the relationship between Gaussian kernels and simulation particles as (sparse) binding matrix B, which is used in Stage III to deform Gaussian kernels according to the physical states of multiple internal particles at subsequent time steps (Section 3.3p. Our binding mechanism offers two advantages. First, it makes the visual dynamics grounding more robust to the initial visual representations (i.e., 3DGS reconstruction) of the object of interest. Second, since Gaussian kernels generally grow around object surfaces, each kernel carries local object-part information. The proposed binding mechanism acts as a smooth filter force the particles belonging to the same object part (i.e., Gaussian kernel) to have relatively uniform physical states, thus facilitating realistic simulation.
3.2 Physics Simulation
Given the particle state 80 = (Po, vo. Fi) at the initial time step, we follow the standard MPM practice to perform the time integration scheme I for physics simulation from / = 1 to / = T. To correct expert-defined material models, i.e., Mo. to better align with visual observations, we implement NeuMA as Mú, := M6 + AMM where i {e,p} denotes either the elastic or the plastic
4

---

material model, and B; is the associated trainable parameters. We adopt NCLaw's [3 optimized material models as fixed Mo and leverage LoRA [30] fulfill the material adaptation process during physics simulation. We embed NeuMA into the differentiable MPM simulator S and iteratively predict the next physical states 81+1 based on the current state &t. Specifically, the simulation process of NeuMA for single time step can be decomposed into three successive stages:
1. Computing the first Piolar-Kirchhof' stress P: (equivalently, the Cauchy stress or or the Kirchhoff stress T) using the clastic material model M
P. = Ma (FF)
2. Obtaining the updated positions P-+1- velocities V-+1, and trial elastic deformation gradient Festrial of material points via the time integration of MPM (refer to Appendix E.6 for details):
[+1
P++1 vt+1, Foitrall I(p,, U2 Fi, Pt).
(5)
3. Post-processing the trial clastic deformation gradient by projecting back to the admissible elastic region using the plastic material model M
F ++1 Ma. (Fi+1
(6)
3.3 Dynamic Scene Rendering
In this subsection, we detail the dynamic scene rendering process achieved by our Particle-GS, which completes our pipeline of dynamics grounding from pixel-level video data. We first augment the static Gaussian kernels to have time-dependent kernel centers and covariance matrices, Lee we use K.(()= = (t) (i), (i), c(i)) denote the i-th Gaussian kernel at I-th time step. Following PhysGaussian 85 we assume that the opacity and the spherical harmonic coefficients are invariant over time. To render the predicted frame at (t + 1)-th time step for supervised training, we deform the Gaussian kernels according to the binding matrix B obtained at the initial time step and the particle states output by the simulator S. Concretely, we calculate the deformed Gaussian centers 1 as:
A0 = B x (Pr+1 PP);
Ft++1 8t + A0
We adopt a similar update scheme for the Gaussian covariance like 85 but additionally consider the binding relationship between kernels and particles:
F
E+1 = B x F I+1+
(8)
A++1 F F
Ao Ao(Fit1)
After obtaining the updated Gaussian kernels {Ke+1 (i)], }. splat these kernels onto 2D image plane as in Equation (9) obtain the synthesized image I++1
3.4 Training Details
We perform supervised training upon the output of our differentiable renderer R to optimize NeuMA in end-to-end manner based on the rendering loss L. defined in Equation (2). Dur- ing the initial state acquisition stage, we actually run the particle binding two times. We obtain the indices of Gaussian kernels without attached particles in the first run and then initialize new physical particles at these kernel centers to ensure each kernel at least binds to one particle. Otherwise, some kernels will remain static across the timeline and lead to inaccurate visual grounding. We then run the final particle binding to record the binding matrix. We optimize the neural material adaptor A using RAdam (49 optimizer with cosine learning rate scheduler for 1, 000 iterations for each scene. We choose r = 95% and p = 1, 000 for all experiments unless otherwise specified.
4 Experiments
4.1 Experimental Setup
Baselines. Given that NeuMA is pioneering study aimed at inferring intrinsic dynamics from camera observations alone, making fair comparison with existing methodologies is inherently challenging. Consequently, we make best-effort comparison with the combinations of existing related works to evaluate the proposed method from different aspects.
5

---

Table 1: Quantitative comparison in object dynamics grounding in Chamfer distance.
Method
BouncyBall
JellyDuck
RubberPawn
ClayCat
SandFish
HoneyBottle
Average
PAC-NeRF 45
516.30
137.73
15.47
15.38
1.71
2.21
114,80
NCLaw (53
56.13
6.32
3.31
2,45
2.61
2.26
12.18
NeuMA
1.19
3.03
1.27
1.00
0.65
0.73
1.31
NeuMA w/ P.S.
1.45
3.74
1.25
1,00
0.80
0.84
1.51
NeuMA w/o Bind
3.34
28.42
4.62
1.26
0.97
1.01
6,60
NeuMA w/o A/
1.87
3.63
1.42
1.36
1.21
1.23
1.79
PAC-NeRF 45 is the most relevant work to research, which inverts material parameters (e.g..
Young's modulus) of heuristic simulator, i.e., MPM [34,, from multi-view videos.
NCLaw [53] relies pre-defined particle data to train general material model without considering adaptation to new kinds of material. We adopt its pre-trained model as the basic material model
(ie.,Ms in Equation (3) and compare method with it to show the effect of material adaptation.
NCLaw + Particle-Ga GS (NCLaw+R): Beyond particle-level comparison with NCLaw, we also
conduct visual comparisons by integrating pre-trained NCLaw with our renderer, Particle-GS.
NeuMA w/ Particle Supervision (NeuMA w/ P.S.): To evaluate the effect of visual supervision, we also report the performance of NeuMA trained with ground-truth 3D particles.
NeuMA w/o AMg We ablate the motion correction term in NeuMA and directly train Mo. in order to verify our core idea the residual motion adaptation paradigm. This variant equivalent to finetuning NCLaw with Particle-GS using visual observation.
NeuMA w/o Bind: We ablate the particle binding procedure by directly treating the 3D Gaussian kernels as physical particles, which is commonly used in previous works [1 85).
Spring-Gaud Gautn (96] is a concurrent work that models elastic objects with Spring-Mass 3D Gaussians. It achieves dynamics grounding by tuning learnable material parameters (e.g.. the spring stiffness)
with an analytical simulator. We compare with our method in real-world experiments.
Dataset. For comprehensive evaluation, we consider both synthetic and real-world data.
Synthetic Data. Following PAC-NeRF 45 and NCLaw [53]. we use MPM 34 to simulate 6 kinds of dynamics with different initial conditions (object shape, velocities, positions). materials, and time intervals between simulation steps. We use Blender [9] to render high-fidelity and realistic images. The generated 6 benchmarks are named "BouncyBall", "JellyDuck", "RubberPawn' "ClayCat", "HoneyBottle", and "SandFishh" We report the simulation details in Appendix [B] Real-world Data. We adopt the real-world data provided by Spring-Gaus (96] assess the visual grounding performance in the real world. We choose 4 scenes from the released dataset, i.e., "Bun", "Burger", "Dog" and "Pig", for experiments. Please refer to Section 4.1 of [96] for details.
Metrics. Following previous works 26 78 we calculate the 1.2-Chamfer distance 15 52 between the grounded and ground-truth particles to measure the accuracy of object dynamics grounding. Note that we scale the values by 10 in all experiments unless otherwise specified. We follow 3DGS (37 and use PSNR 29]. SSIM [8], and LPIPS [4]] as the metrics for dynamic scene rendering.
Implementation Details. In the initial state acquisition stage. we generally obtain 10k~30k Gaussian kernels after static scene reconstruction. We cull Gaussians whose opacities fall behind 0.02 before training NeuMA. The number of initial particles achieved for physics simulation is about 30k. We use 1- and 3- v-veww videos on synthetic and real-world data, respectively, as ground-truth observations for dynamics grounding. In implementing NeuMA, we follow 55 to explicitly regularize neural networks to adhere to two physical priors, i.e., frame indifference and undeformed state equilibrium. Please refer to [53] for the implementation. In addition, we set the rank and value of A.Ms to 16 by default. Our MPM simulator operates in a [o, 1] cube with a fixed resolution of 32 and 7 for synthetic and real-world data unless otherwise specified. All the experiments are conducted single NVIDIA A100 GPU.
4.2 Performance on Object Dynamics Grounding
Here, we compare NeuMA with considered baselines on visual dynamics grounding using synthetic data. We report the Chamfer distance between the predicted and the ground-trus trutt particle positions in T It is observed that NeuMA achieves satisfactory results on each scene and the least Chamfer
Pawn by Poly by Google [CC-BY] via Poly Pizza.

---

NCLaw
NeuMA
NeuMA W/
W.P.S P.S.
NeuMA w// Bind
NeuMA w/o 2M
Chamiar Distance
BouncyBall
Chariier Distance
DellyDuck
Chamifer Distance
RubberPawn
96
231
340 320
340
300
Time Step
Time Step
Time Step
Chamierr Detance
ClayCat
Chamifer Detance
SandFish
Chamifer Distance
HoneyBottte
150
170
100
Time Step
Time Step
Time
Step
Figure 3: Comparison object dynamics grounding over the entire simulation sequence.
PAC-NeRF
NCLaw+R
NeuMA
NeuMA A.M
NeuMA
NeuMM w/o Bind
LPTPS
0.15
0.95
0.90
25
0.05
20
Ball
Duck
Pawn
Cat
Bottle
0.85 Ball
Duck
Pawn
Bottle
Fish
Ball
Duck
Pawn
Bottle
Fish
Figure 4: Quantitative comparison in dynamic scene rendering.
distance average, suggesting the superiority of our overall pipeline for visual dynamics grounding. From the first three rows in T compared to PAC-NeRF [45], which requires an accurate initial guess of the physical parameters, and NCLaw [53], which uses pre-defined fixed material models, NeuMA introduces learnable residual term AM to flexibly correct prior knowledge to align with current observations, significantly improving performance.
We also find that NeuMA even outperforms variants with particle-level supervision (i.e., NeuMA w/ P.S.) in some cases. This is very critical result, proving that the model's 's reasoning ability can match the human common sense of estimating object dynamics purely from visual observation. We attribute this to the introduction of AMa, which preserves the expert prior of Mo while fine- tuning to the given dynamic scene. Moreover, results in the fifth row indicate that properly binding physical particles and Gaussian kernels is critical optimize the residual term Mo. To explain, partiele binding scheme, unlike previous work's one-to-one mapping between particles and Gaussian kernels (85), ensures that particles within a Gaussian kernel share relatively uniform physical states, thereby reducing the degrees of freedom during optimization.
We further show the Chamfer distance at each time step in Figure We observe that the Chamfer distance NeuMA consistently remains lower than that of other competitors with the simulation ongoing. This verifies that NeuMA accurately learn the intrinsic dynamics laws of the actual observations. Please refer to the particle visualizations of JellyDuck in Figure 12 for an intuitive com- parison. In summary, these results convincingly validate the effectiveness of our main contribution-a learnable, residual neural material adaptor that can correct physical priors from visual observations.
4.3 Performance Dynamic Scene Rendering
we present averaged visual quality metrics synthetic data to compare view synthesis performance over the entire simulation time. Our method generally outperforms all baselines, often by large margin. Furthermore, we present the rendering results in Figure 5 Despite using only single-view video data to ground intrinsie dynamics, our model consistently generates high-fidelity, physically plausible image sequences. In the RubberPawn scene with delicate geometries, PAC-
In Figure 4
7

---

Reference NCLaw* PAC-NeRF NeuMA |Reference NCLaw+ PAC-NeRF NeuMA: Reference NCLawt3 PAC-NeRF NeuMA
Time
BouncyBall
RubberPawn
SandFish
Figure 5: The visual results for dynamic scene rendering.
Spring-Gaus
Sprng-Geus
Sorng-Grun
Spring-Gaus
Bun
18.72 0.08 21.75/ 0.8
Bunger
15.17 20.01/ 0.05
Dog
X0.30 0.00
23.13/ 0.06
Plg
22.02/0.05 24.32/ 0.05
Figure 6: Comparisons on real-world samples. We also present quantitative results (i.e., PSNR / LPIPS) between predictions and observations (with background filtered) the bottom line.
NeRF 45 trained with multi-view data produces blurry artifacts due limited implicit scene resolution. Although NCLaw [53] combined with our Particle-Gr GS yields better visual quality, it fails to capture material motion accurately. These results verify that our residual adaptation can achieve accurate dynamics grounding and that Particle-GS can handle complex geometries and deformations.
We further evaluate NeuMA real data in Figure It is that the rendered sequences of NeuMA are more aligned with the observations than those of Spring-Gaus (96] both qualitatively and quantitatively. The results confirm the effectiveness of NeuMA in real-world scenarios.
4.4 Experimental Analysis
Dynamics Interpolation. Recall that AM acts residual term correct the expert-designed material model MM to match the actual dynamics. To study the ficxibility of the residual design, apply different compositional weights resulting dynamics Figure 7 It is observed that NeuMA can smoothly interpolate between the dynamics specified by material priors and by observed images, indicating A&M indeed learns the difference between the actual dynamics and the dynamics described by the -esert-designed motion equations. These visual results verify the flexibility of our proposed residual adaptation paradigm (ie.., Ma = MM + AMa)) for grounding intrinsic dynamics.
(30] to AMa (by adjusting a) and visualize the
Dynamics Generalization. To study whether NeuMA has learned intrinsic dynamics, we directly apply the trained NeuMA to predict the dynamics given a new object with various initial conditions (e.g., initial velocities and locations). We use the letters of "NeurIPS" the test target, and the results are shown Figure S a). The used material model is listed at the top. observed that new objects also have similar motion patterns as the applied dynamics, verifying that NeuMA indeed learns the intrinsic dynamics that can generalize to novel conditions. Furthermore, we ask: can the learned NeuMA be directly applied to multi-object interaction? The visual results in Figure g b), particularly the challenging case of the SandFish colliding with the JellyDuck, show that two trained NeuMA instances seamlessly interact with each other produce physically plausible dynamics.
We also quantify the generalization performance these two scenarios by (1) applying the learned Ms to new object, i.e., the letter "N", and (2) incorporating two learned NeuMAs for multi-

---

Obsenvation
Obsenation
Figure 7: The visual results for dynamics interpolation. By applying different weights to the residual term AMg NeuMA can generate diverse dynamics that smoothly translate prior dynamics visual observation.
Bouncy
Rubber
P
S
N
P
P
S
Bouncy
S
N
8
N
e
S
Jely
(b)
Figure 8: The visual results for dynamics generalization. The blue indicates the applied material for each object. (a) Generalization to new objects (i.e. the letters of "NeurlPS"); (b) Generalization multi-object interactions.
Granninng
Ground Tust
0.000
E-0
t- 130
t- 200
t- 300
400 t- 500
600
t- 700
8=800
t- 900
0:33
1000 t= 1100 t= 1200 1300 t=
1400 t= 1500 t-
1600 t= 1700 c= 1800 to 1900
500
1200
1000
(n) Figure 9: Results for dynamics prediction. Given visual data from 0 to 400, NeuMA can generate physically plausible prediction for longer period. (a) Visualization of predicted images; (b) Comparison of the mean height of the BouncyBall between ground truth and our prediction.
Fart:Pokk Hodel
Oodert
Rorti MO06
Toeiin (b) RubberfPown
(6) BouncySall
Figure 10: Gradient norm analysis. Y-axis: average gradient norms across the training stage. Higher norms indicate more significant changes in the parameters of the corresponding material model.
object interactions. The results in terms of Chamfer distance are shown in Table 2 We observe that NeuMA achieves favorable results over other variants. summary, the generalization ability of NeuMA largely stems from its refinement of an expert-designed motion model Mo, which ensures physical interpretability and enables effective application in estimating multi-object interactions.
Dynamics Prediction. The hallmark a good physics model is its ability to predict future dynamics accurately (25). In Figure(9) we investigate NeuMA's predicted dynamics for extended time. It is seen that NeuMA achieves effective prediction for a period that is three times longer than the given observation sequence, which validates its potential generate long-duration dynamic videos.
Prior (Mo) Correction. Figure 10 presents the gradient norms the elastic and plastic material models during the training process. As shown in Figure 5 the reference dynamics (i.e., visual

---

Table Quantitative results on generalization.
Observtion
Gro.nding
Method
Bouncy
Rubber
Sand
Ball Cat
NeuMA
0.99
0.36
0.33
0.71
NeuMA w/ P.S.
1.16
0.32
0.30
0.73
NeuMA w/o Bind
4.26
14.99
0.48
1.19
Figure 11: Grounding result an object with uneven mass.
NeuMA LoRA
1.78
0.45
0.36
0.91
Table Quantitative results given different physical prior Ma.
Grounding RubherPawn
Generalization
Setting
M5
ME
ClayCat
Rubber + "N" Clay + "N"
StVK
Mises
1.27
1.00
0.36
1.23
II
Peo-Ilookean
Mises
1.94
0,91
0.36
1.04
III Fixed Corotated
von Mises
1.95
1.60
0.36
1.47
IV
Fixed Corotated
Identity
3.64
3,22
0,70
1.31
V
SIVK
Drucken- Drucken-Prager
30.26
12.91
3,91
3.31
observations) of the BouncyBall deviate from NCLaw (i.e., the prior Mo in our experiment) that the reference exhibits more pronounced deformation before returning to its original shape. This deviation is primarily driven by the elastic material model. Interestingly, NeuMA's learning process shows a tendency to adjust the elastic model to better align with the ground-truth observations, as illustrated in Figure a0)a). For the RubberPawn, the main discrepancy from the prior lies in the degree of plastic deformation. As shown in Figure [O),), NeuMA opts to adjust the plastic model more significantly than the elastic model. In conclusion, this analysis highlights NeuMA's interpretability in adaptively correcting priors specific material models to better match the visual observations.
Uneven Mass Distribution. In our implementation, we, by default, assume uniform mass distribu- tion for each object. Here, we analyze how NeuMA performs when applied to objects with uneven mass. shown in Figure 11] we assign particles with different mass densities p to obtain ClayCat with uneven mass. We follow the similar pipeline depicted in Section X for visual grounding and display the qualitative results. We also quantify the Chamfer distance between the grounded and ground-truth particles, and the result is 1.33. From these results, it is seen that NeuMA can handle objects with uneven mass, which validates its potential for visual grounding in complex scenarios.
Inaccurate Application of Mo- Here, we study the effect of inaccurate application of Mo using two plastic objects from our synthetic dataset, i.e., RubberPawn and ClayCat. The specific settings and quantitative results (in terms of Chamfer distance) are shown in Table Setting I is our default, where the correct material models are used as the physical prior. Settings II and III introduce inaccurate elastic models, but the resulting motion still follows plastic behavior. Settings IV and are more challenging, as they are typically used for simulating elastic and granular objects. Our method handles moderate deviations from correct models (Settings II and III), but when the prior is completely incorrect (Settings IV and V), performance drops significantly. To address this, Large Vision Language Models like GPT-4o may be used to infer plausible material models given keyframes.
Conclusion and Limitations
Conclusion. We introduce Neural Material Adaptor (NeuMA), a novel framework to infer intrinsic dynamics from visual data. By integrating data-driven corrections with established physical laws, NeuMA combines the interpretability of white box models with the adaptability of black box models. Extensive experiments verify that NeuMA enhances the accuracy and generalization of physical simulations, offering a significant advancement in AI's ability to understand and predict dynamic scenes.
object dynamics grounding, dynamic rendering, and its generalizability
Limitations. Despite its strengths, NeuMA has several limitations. First, requires acquiring initial particles via multi-view surface reconstruction technique, which, however, requires calibrated cameras and does not perform well in complex scenes. Second, cumulative errors in the forward simulation have a negative effect on future predictions, requiring the design of new physics-informed constraints in the particle space. Additionally, since NeuMA relies on visual supervision, motion blur is a key factor that can degrade the grounding performance.
10

---

Acknowledgments
This work was supported in part by NSFC (62322113, 62376156), Shanghai Municipal Science and Technology Major Project (2021SH22D2X01022) and the Fundamental Research Funds for the Central Universities.
References
[1] Christopher Bates, Peter W Battaglia, IIker Yildirim, and Joshua B Tenenbaum. Humans predict liquid dynamics using probabilistic simulation. In CogSci, 2015.
[2] Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as engine of physical scene understanding. Proceedings of the National Academy of Sciences, 2013.
[3] Ronaldo Borja. Plasticity. Springer Berlin Heidelberg. doi, 10:978-3, 2013.
[4] Junyi Cao, Zhichao Li, Naiyan Wang, and Chao Ma. Lightning NeRF: Efficient hybrid scene representation for autonomous driving. In ICRA, 2024.
[5] Pradyumna Chari, Chinmay Talegaonkar, Yunhao Ba, and Achuta Kadambi. Visual physics: Discovering physical laws from videos. CoRR:1911.1 18193, 2019.
[6] Honglin Chen, Wanhee Lee, Hong- Xing Yu, Rahul Mysore Venkatesh, Joshua B Tenenbaum, Daniel Bear, Jiajun Wu, and Daniel LK Yamins. Unsupervised 3D scene representation learning via movable object inference. TMLR, 2022.
[7] Wenzheng Chen, Joey Litalien, Jun Gao, Zian Wang, Clement Fuji Tsang, Sameh Khamis, Or Litany, and Sanja Fidler. DIB-R++: Learning to predict lighting and material with hybrid differentiable renderer. In NeurIPS, 2021.
[8] Paul J Cohen. The independence of the continuum hypothesis. Proceedings of the National Academy of Sciences, 10(6):1143-1144 1963.
[9] Blender Online Community. Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018
[10] Roy De Maesschalck, Delphine
and Désiré L Massart. The Mahalanobis distance. Chemometrics and intelligent laboratory systems, 50(1):1-18, 2000.
[11] Eduardo A de Souza Neto, Djordje Peric, and David RJ Owen. Computational methods for plasticity: theory and applications. John Wiley & Sons, 2011.
[12] Johannes Dornheim, Lukas Morand, Hemanth Janarthanam Nallani, and Dirk Helm. Neural networks for constitutive modeling: From universal function approximators advanced models and the integration of physics. Archives of Computational Methods in Engineering, (2):1100 1127, 2024.
[13] Jiafei Duan, Arijit Dasgupta, Jason Fischer, and Cheston Tan. A survey on machine learning approaches for modelling intuitive physics. In IJCAI, 2022.
[14] Mathieu Dubied, Mike Yan Michelis, Andrew Spielberg, and Robert Kevin Katzschmann. Sim- to-real for soft robots using differentiable FEM: Recipes for meshing, damping, and actuation. IEEE Roboties and Automation Letters, 7(2)50055 552500255-22,,,----1122153 2022.
[15] Philipp Erler, Paul Guerrero, Stefan Ohrhallinger, Niloy Mitra, and Michael Wimmer. Points2Surf learning implicit surfaces from point clouds. In ECCV, 2020.
[16] Qijun Feng, Zhen Xing, Zuxuan Wu, and Yu-Gang Jiang. FDGaussian: Fast gaussian splatting from single image via geometric-aware diffusion model. CoRR:2403.10242, 2024.
[17] Yutao Feng, Xiang Feng, Yintong Shang, Ying Jiang, Chang Yu, Zeshun Zong, Tianjia Shao, Hongzhi Wu, Kun Zhou, Chenfanfu Jiang, et Gaussian splashing: Dynamic fluid synthesis with Gaussian splatting. CoRR:2401.15318, 2024.

---

[18] Yutao Feng, Yintong Shang. Xuan Li. Tinjia Shao, Chenfanfu Jiang. and Yin Yang. PIE-NeRF: Physics-based interactive clastodynamics with NeRF. In CVPR, 2023.
[19] Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning visual predictive models of physics for playing billiards. In ICLR, 2016.
[20] Alexander Fuchs, Yousef Heider, Kun Wang, WaiChing Sun, and Michael Kaliske. DNN2: A hyper-parameter reinforcement learning game for self-design of neural network based elasto- plastic constitutive descriptions. Computers & Structures, 249: 249:10 2021.
[21] David L Gilden and Dennis R Proffitt. Heuristic judgment of mass ratio in two-body collisions. Perception & Psychophysics, 56:708-720, 1994.
[22] Oscar Gonzalez and Andrew M Stuart. A first course in continuum mechanics, volume 42. Cambridge University Press, 2008.
[23] Dominique Gravel, Charles D Canham, Marilou Beaudet, and Christian Messier. Reconciling niche and neutrality: The continuum hypothesis. Ecology letters, 9(4):399-409, 2006.
[24] Priscilla E Greenwood and Michael S Nikulin. A guide to chi-squared testing, volume 280. John Wiley & Sons, 1996.
[25] Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. In NeurIPS, 2019.
[26] Shanyan Guan, Huayu Deng, Yunbo Wang, and Xiaokang Yang. NeuroFluid: Fluid dynamics grounding with particle-driven neural radiance fields. In ICML, 2022.
[27] Mary Hegarty. Mechanical reasoning by mental simulation. Trends in cognitive sciences, 8(6):280-285, 2004.
[28] Florian Hofherr, Lukas Koestler, Florian Bernard, and Daniel Cremers. Neural implicit repre- sentations for physical parameter inference from single video. In WACV, 2023.
[29] Alain Hore and Djemel Ziou. Image quality metrics: PSNR vs. SSIM. ICPR, 2010.
[30] Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et LoRA: Low-rank adaptation of large language models. In ICLR, 2021.
[31] Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana. and Chenfanfu Jiang, moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM TOG, 2018.
[32] Daniel Z Huang, Kailai Xu, Charbel Farhat, and Eric Darve. Learning constitutive relations from indirect observations using deep neural networks. Journal of Computational Physics, 416:109491, 2020.
[33] Chenfanfu Jiang, Craig Schroeder, Andrew Selle, Joseph Teran, and Alexey Stomakhin. The affine particle-in-cell method. ACM TOG, 2015.
[34] Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The material point method for simulating continuum materials, In SIGGRAPH 2016 courses. ACM, 2016.
[35] Ying Jiang, Chang Yu, Tianyi Xie, Xuan Li, Yutao Feng, Huamin Wang, Minchen Li, Henry Lau, Feng Gao, Yin Yang, et al. VR-GS: A physical dynamics-aware interactive gaussian splatting system in virtual reality. In SIGGRAPH, 2024.
[36] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3D mesh renderer. CVPR, 2018.
[37] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3D Gaussian splatting for real-time radiance field rendering. ACM TOG, 2023.
12

---

[38] Tachun Kim, Kunhee Kim, Joonyeong Lee, Dongmin Cha. Jiho Lee, and Daijin Kim. Revisiting image pyramid structure for resolution salient object detection. In ACCV, 2022.
[39] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yet Lo, et al. Segment anything. In ICCV, 2023.
[40] Gergely Klár, Theodore Gast, Andre Pradhana, Chuyuan Fu, Craig Schroeder, Chenfanfu Jiang, and Joseph Teran. Drucker-prager elastoplasticity for sand animation. ACM TOG, 2016.
[41] Dominik K Klein, Mauricio Fernández, Robert J Martin, Patrizio Neff, and Oliver Weeger. Polyconvex anisotropic hyperelasticity with neural networks. Journal of the Mechanics and Physics of Solids, 159: 104703, 2022.
[42] Krishna Kumar and Joseph Vantassel. GNS: A generalizable graph neural network-based simulator for particulate and fluid modeling. Open Source Softw., 8(88):5025, 2023.
[43] Simon Le Cleac'h, Hong-Xing Yu, Michelle Guo, Taylor Howell, Ruohan Gao, Jiajun Wu, Zachary Manchester, and Mac Schwager. Differentiable physies simulation of dynamics- augmented neural objects. IEEE Robotics and Automation Letters, 8(5):2780-278) 2023.
[44] Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. Differentiable monte carlo ray tracing through edge sampling. ACM TOG, 2018.
[45] Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chen- fanfu Jiang, and Chuang Gan. PAC-NeRF: Physics augmented continuum neural radiance fields for geometry-agnoste system identification. In ICLR, 2022.
[46] Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel Yamins, Jiajun Wu, Joshua Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In ICML, 2020.
[47] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2018.
[48] Burigede Liu, Nikola Kovachki, Zongyi Li, Kamyar Azizzadenesheli, Anima Anandkumar, -aased based multiscale method and its
Andrew M Stuart, and Kaushik Bhattacharya. learning application to inelastic impact problems. Journal of the Mechanics and Physics of Solids, 158: 104668, 2022.
[49] Liyuan Liu, Haoming Jiang. Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In ICLR, 2020.
[50] Shichen Liu, Tianye Li. Weikai Chen, and Hao Li. Soft rasterizer: differentiable renderer for basee 3D reasoning, In ICCV, 2019.
[51] Matthew M Loper and Michael J Black. OpenDR: An approximate differentiable renderer. In ECCV, 2014.
[52] Baorui Ma, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Neural-Pull, Learning signed distance function from point clouds by learning pull space onto surface. In ICML, 2021.
[53] Pingchuan Ma, Peter Yichen Chen, Bolei Deng, Joshua B Tenenbaum, Tao Du, Chuang Gan, and Wojciech Matusik. Learning neural constitutive laws from motion observations for generalizable PDE dynamics. In ICML, 2023.
[54] Pingchuan Ma, Tao Du, Joshua B Tenenbaum, Wojciech Matusik, and Chuang Gan. RISP: Rendering-invarian state predictor with differentiable simulation and rendering for cross- domain parameter estimation. In ICLR, 2021.
[55] Robert Maier, Kihwan Kim, Daniel Cremers, Jan Kautz, and Matthias NieBner. Intrinsic3D: High-quality 3D reconstruction by joint appearance and geometry optimization with spatially- varying lighting. In ICCV, 2017.
[56] Michael McCloskey. Intuitive physics. Scientific American, 1983.
13

---

[57] Goeffrey J McLachlan. Mahalanobis distance. Resonance, 4(6):20-20 1999.
[58] B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[59] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG, 2022.
[60] J. Krishma Murthy, Miles Macklin, Florian Golemo, Vikram Voleti, Linda Petrini, Martin Weiss, Breandan Considine, Jérôme Parent-Lévesque, Kevin Xie, Kenny Erleben, Liam Paull, Florian Shkurti, Derek Nowrouzezahrai, and Sanja Fidler. gradSim: Differentiable simulation for system identification and visuomotor control. In ICLR, 2021.
[61] Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. Large steps in inverse rendering of geometry. ACM TOG, 2021.
[62] Merlin Nimier-David Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. ACM TOG, 2019.
[63] Yicong Peng. Yichao Yan, Shengqi Liu, Yuhao Cheng, Shanyan Guan, Bowen Pan, Guangtao Zhai, and Xiaokang Yang. CageNeRF: Cage-based neural radiance field for generalized 3D deformation and animation. In NeurlPS, 2022.
[64] Tobias Pfaff, Meire Fortunato, Alvaro Manchez-Gonzalle and Peter Battaglia. Learning mesh- based simulation with graph networks. In ICLR, 2020.
[65] Lukas Prantl, Benjamin Ummenhofer, Vladlen Koltun, and Nils Thuerey. Guarantced conserva- tion of momentum for learning particle-based fluid dynamics. In NeurlPS, 2022.
[66] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks:
A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-707, 2019.
[67] Sverker Runeson, Peter Juslin, and Henrik Olsson. Visual perception of dynamic properties: Cue heuristics versus direct-perceptual competence. Psychological review, 107(3):525, 2000.
[68] Jan Rychlewski. On Hooke's ' law. Journal of Applied Mathematics and Mechanics, 48(3):303- 314, 1984.
[69] Adam N Sanbor, Vikash K Mansinghka, and Thomas L Griffiths. Reconciling intuitive physics and newtonian mechanics for colliding objects. Psychological review, 120(2):411, 2013.
[70] Yidi Shao, Chen Change Loy, and Bo Dai. Transformer with implicit edges for particle-based physics simulation. In ECCV, 2022.
[71] Alexey Stomakhin, Craig Schroeder, Lawrence Chai, Joseph Teran, and Andrew Selle. material point method for snow simulation. ACM TOG, 2013.
[72] Deborah Sulsky, Zhen Chen, and Howard L Schreyer. A particle method for history- materials. Computer methods applied mechanics and engineering. 1 118(1-2):199-199 196, 1994.
[73] Deborah Sulsky, Shi-Jian Zhou, and Howard Schreyer. Application of a particle-in-cell method to solid mechanics. Computer physics communications, 1995.
[74] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. LGM:: Large multi-view gaussian model for high-resolution 3D content creation. In ECCV, 2024.
[75] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. DreamGaussian: Generative gaussian splatting for efficient 3D content creation. In ICLR, 2023.
[76] Fei Tao, Xin Liu, Haodong Du, and Wenbin Yu. Learning composite constitutive laws via coupling abaqus and deep neural network. Composite Structures, 272:114137, 2021.
[77] Tomer D Ullman, Elizabeth Spelke, Peter Battaglia, and Joshua B Tenenbaum. Mind games: Game engines as an architecture for intuitive physics. Trends in cognitive sciences, 21 (9)6694 665, 2017.
14

---

[78] Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In ICLR, 2019.
[79] Bin Wang, Yuanmin Deng, Paul Kry, Uri Ascher, Hui Huang, and Baoquan Chen. Learning elastic constitutive material and damping models. In Computer Graphics Forum. Wiley Online Library, 2020.
[80] Yiming Wang, Qin Han, Mare Habermann, Kostas Daniilidis, Christian Theobalt, and Lingjie Liu. NeuS2: Fast learning of neural implicit surfaces for multi-view reconstruction. In ICCV, 2023.
[81] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from emror visibility to structural similarity. IEEE TIP, 13(4):600-612, 2004.
[82] Zian Wang. Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojeic, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representations for inverse rendering of urban scenes. In CVPR, 2023.
[83] Alan Wineman. Some results for generalized cco-hookean elastic materials. International Journal of Non-Linear Mechanics, 40(2-3):271-217 2005.
[84] Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, and Josh Tenenbaum. Learning to see physics via visual de-animation. In NeurIPS, 2017.
[85] Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. PhysGaussian: Physics-integrated 3D gaussians for generative dynamics. In CVPR, 2024.
[86] Hongyi Xu and Jernej Barbič. Example-based damping design. ACM TOG, 2017.
[87] Hongyi Xu, Funshing Sin, Yufeng Zhu, and Jernej Barbič. Nonlinear material design using principal stretches. ACM TOG, 2015.
[88] Yinghao Xu, Zifan Shi, Wang Yifan, Hansheng Chen, Ceyuan Yang. Sida Peng, Yujun Shen, and Gordon Wetzstein. GRM: Large gaussian reconstruction model for efficient 3D reconstruction and generation. In ECCV, 2024.
[89] Haotian Xue, Antonio Torralba, Josh Tenenbaum, Dan Yamins, Yunzhu Li, and Hsiao- Yu Tung. 3D-Inthhys: Towards more generalized 3D-grounded visual intuitive physics under challenging scenes. In NeurlPS, 2023.
[90] Tianju Xue, Shuheng Liao, Zhengtao Gan, Chanwook Park, Xiaoyu Xie, Wing Kam Liu, and Jian Cao. JAX-FEM: A differentiable GPU-accelerated 3D finite element solver for automatic inverse design and mechanistic data science. Computer Physics Communications, 291:108802, 2023.
[91] Shunyu Yao, Tzu Ming Hsu, Jun-Ya Zhu, Jiajun Wu, Antonio Torralba, Bill Freeman, and Josh Tenenbaum. 3D-aware scene manipulation via inverse graphics. In NeurIPS, 2018.
[92] Zehao Yu, Torsten Sattler, and Andreas Geiger. Gaussian opacity fields: Efficient high-quality compact surface reconstruction in unbounded scenes. CoRR:2404. 10772, 2024.
[93] Yonghao Yue, Breannan Smith, Christopher Batty, Changxi Zheng, and Eitan Grinspun. Contin- uum foam: A material point method for shear-dependent flows. ACM TOG, 2015.
[94] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as a perceptual metric. In CVPR, 2018.
[95] Tianyuan Zhang. Hong-Xing Yu, Rundi Wu, Brandon Y Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T Freeman. PhysDreamer: Physics- bassic interaction with 3D objects via video generation. In ECCV, 2024.
[96] Licheng Zhong, XlongXXing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3D Gaussians. In ECCV, 2024.
[97] Zeshun Zong, Xuan Li, Minchen Li, Maurizio M Chiaramonte, Wojciech Matusik, Eitan Grinspun, Kevin Carlberg, Chenfanfu Jiang, and Peter Yichen Chen. Neural stress fields for reduced-order elastoplasticity and fracture. In SIGGRAPH Asia, 2023.
15

---

A Background
Material Models. Fundamental to continuum mechamics, material models (also known as constitutive models) define the relationship between material response and the forces or deformations applied. One example is Hooke's 's law i.e., F., krr 688, which relates the spring force with deformation. These models not only encapsulate the material's 's intrinsic properties but also dictate how it behaves under various loading conditions. the context of an elastodynamic system following Equation (T. two types of constitutive relationships must be pre-defined. One is the elastic material model MM that describes the stress response P given the elastic deformation gradient F (i.e., the elastic part of the deformation gradient V
33 40) The other is the plastic material model .MP that defines return mapping projecting the trial elastic deformation gradient Fe.trial onto the plastic yield constraint fy
[3 I Traditionally, material models have been constructed through nonlinear polynomial
bases 86 87 Recently, researchers have adopted seamingg-based approaches [12] 20 41 48 76
to
use neural networks to encode the constitutive relationships for different materials.
3D Gaussian Splatting. 3DGS 37 is an optimization based rasterization approach for time-efficient 3D scene reconstruction. It represents the object of interest as set of Gaussian kernels {IC(i) NK with learnable parameters ((i) {=x()) a(i), A(),, c(i)), where 2(1), ((i), A(t), denote the center, opacity, covariance matrix, and spherical harmonic coefficients of the i-th Gaussian kernel. 3DGS splats these kernels onto the 2D image plane to synthesize an image according to the camera
viewing matrix. The final color of each pixel is calculated as:
C Σ sH(d()):(()) a'(1) II (I--a'())
(9)
where a' ai) = G())a(i) is the z-depth ordered effective opacity (with G(i) denotes the 2D Gaussian weight of the i-th kernel), and d(i) denotes the viewing direction from the camera z(i). This process is fully differentiable. Hence, 3DGS uses L loss and structural similarity index measure (SSIM) loss to supervise the optimization of Gaussian kernels.
B
More Details about the Synthetic Dataset
Table 4: Typical geometric and physical properties related to the observed dynamics.
Benchmark
Initial Shape
Dynamics
Step-size (s) Initial Velocity (m/s)
Initial Height (m)
Boul
Ball
Bouncy
le 3
(0, 1.92,, 0)
0.28
JellyDuck
Duck
Jelly
1c 3
(0. -1.62, 0)
0.42
RubberPawn
Pawn
Rubber
5e 4
(0, -1.57,0 0)
0.39
ClayCat
Cat
Clay
5e - 4
(0. 00.21.1,00)
0.32
Honey Bottle
Bottle
Honey
(0, 00.-1.19,0 0)
0.42
SandFish
Fish
Sand
5e 4
(0, 00.60, 0)
0.28
to generate ground-truth particle states for 400 steps and use Blender [9] to render photo-realistic videos at 10 different camera views with a resolution of 800 800. From these views, we choose one frontal viewpoint of the object for visual grounding on the synthetic data. To ensure an accurate object appearance and geometry, we additionally render 50 uniformly sampled viewpoints at O, with the cameras evenly spaced on a sphere covering the object of interest, and use these data for static reconstruction. We also report typical geometric and physical properties of the ground tround-truth observation in Table 4 Our synthetic dataset features different materials, from elastic objects to granular substances, that exhibit various dynamics and diverse shapes. Please refer to the project page for video demonstrations of the synthetic dataset.
For each benchmark, we use MPM 31 34 72
C More Experimental Results
Influence of the Rank of AMa. Recall that the dynamics corrector AM2 is implemented low-rank adaptor (refer to Equation K3). Here, we evaluate the influence the rank, ie., r, of AM on three synthetic benchmarks. The results reported in Table Upon reviewing the results, it is
16

---

Table 5: Ablation study on the rank of A.Mo. The default rank value is 16.
Rank of AM
BouncyBall
RubberPawn
SandFish
Average
1.35
1.50
0.68
1.18
1.23
1.41
0.67
1.10
16
1.19
1.27
0.65
1.04
1.20
1.28
0.66
1.05
t= 100
== 200
t = 400
Reference
NCLaw
NeuMA
NeuMA w/P.. NeuMA w/o Bind
NeuMA w/o A.Mo
Figure 12: Visualization of the simulated particles on the JellyDuck benchmark.
evident that the Chamfer distance increases significantly as the rank decreases. This likely due to the lower model fitting capacity at lower ranks. However, increasing the rank to a larger value, e,g., 24, does not decrease the chamfer distance further, and even increases slightly. This suggests that the network capacity becomes saturated when the rank exceeds a certain threshold. Therefore, we set the rank as 16 by default our experiments.
Visualization of the Physical Particles. We present the visualization of simulated particles in Fig- 12 We choose the JellyDuck benchmark from our synthetic dataset for the following analysis. (a) Pre-trained NCI aaw fails to generalize to new types of materials that deviate from previously learned priors, observed by the unrealistic simulation of the motion of the JellyDuck, especially around the duck's s neck. This result suggests the necessity of using NeuMA for adaptation to new observations: (b) NeuMA w/ P.S. achieves simulation that is aligned with the reference at several initial steps. However, it suffers from unrealistic simulation afterward, as observed by the tail of the yellow duck at 400. A possible reason is that simulation at the particle level incurs high freedom during optimization, leading to large cumulative errors as time goes on, (c) NeuMA w/o Bind exhibits poor results since directly treats 3D Gaussians underlying physical particles. However, the distribution of Gaussian kernels may not conform to the continuum hypothesis 8 (3 adopted by particle-based simulators. Thus, driving these kernels directly would result in unrealistic motion; (d) Regarding NeuMA w/o A.Mo direct modification may overturn the physical priors, as observed by the unrealistic simulation of the green duck's mouth. The result indicates that NeuMA w/o AMa is sub-optimal way for grounding intrinsic dynamics. This particle-level visualization, from another aspect, demonstrates the superiority of NeuMA in achieving intrinsic dynamics grounding from video data.
the parameters of pre-trained material models
Experiments on complex objects. Here, we show additional visual results for dynamics grounding on several objects with delicate geometries in Figure [3] Specifically, we consider (a) a large object, "Machine Man (b) an open container. "Crate". and (c) non-convex object, "Ring"
While these objects feature different geometric properties, the dynamic rendering results achieved by NeuMA validate its effectiveness on complex scenes.
Machine Man by Vaporworks [CC-BY] via Poly Pizza. Diamond ring by Poly by Google [CC-BY] via Poly Pizza.

---

Oesenvation
Observation
Osservatoon
Grounding
Grounding
Grounding
(e)
(b)
Figure 13: Visualization of dynamics grounding on complex objects.
Imput (mage imge-to-30)
(by image-to-33))
(by No.MA)
Figure 14: Visualization of the simulation results on 3D content from image-to-3D model.
Compatibility with Image-to-3D Models. Recent advancements in 3D content generation [16 75 88) offer tremendous potential for various industrial applications, including digital games, virtual reality, and computational design. In this context, an easily conceivable question is whether NeuMA can drive the generated 3D content to produce physically realistic animations. In this subsection, we integrate a state-of-the-an image-to-33 model [88] with NeuMA to investigate such new pipeline for 4D generation. Concretely, we use single image as input to the image-to-3D method to obtain 3D model represented by 3DGS [37 (and optionally do some post- post-processing to cull the isolated kernels and reduce the number of kernels). We use Blender [9] to transform the 3D) representation into a surface mesh for our particle sampling procedure. At this point, we can directly apply trained AsuMAs to generate physically plausible animations, as demonstrated in Section 4.4 We present the visual results of this experiment in Figure [4] From the top to the bottom in the rightmost figure, separately apply the trained NeuMA on JelkyDuck, ClayCat, and HoneyBottle for dynamics generation. As observed, NeuMA is capable driving the 3D representation produced by the image-to-3D model to generate physically plausible dynamic sequences. These results show NeuMA's potential for physically realistic 4D generation from a single image.
D Related Work
Intuitive Physics. Intuitive physics studies how humans acquire cognitive capabilities and how to
build machine systems with similar capabilities [3
556. Researchers explore this topic from various
perspectives, including rule-based models 21
67 69 probabilistic mental simulation [ 27), and
cognitive intuitive physics engines 2 5 77 With advancements in deep learning and differentiable
rendering, intuitive physics has shown promising applicability reasoning about 3D geometry [55]
61 [2]
scene representation (6|91] physical parameters [7] 28 46), and object dynamics [19
from images or vidcos. However, most works focus on rigid objects. Some system identification works 45, 54 60] related to ours but either entangled with geometric shape [54], require
extra rendering parameters (60), or rely on expert-definee dynamics models (55. An open question remains: how can neural networks estimate actual intrinsic dynamics solely from observed images?
Differentiable Simulation. Recent works have introduced deep learning- based methods into the physics simulation field, enabling faster and more stable simulations and solving inverse problems. fully embrace neural networks (NN), directly learning an end-to- end neural network to implicitly model physical laws, such as material models [71). Despite their efficiency, these purely NN-based models cannot guarantee physical correctness and have limited generalizability, Another approach explicitly incorporates expert-designed simulation priors as
Some works 42 64
70 78
18

---

inductive bias in the network architecture 32 53 65 79. These approaches show great flexibility and generalizability to new shapes and initial/boundary conditions. However, the reliance on known expert knowledge limits their application to arbitrary materials, which yet is the target of our NeuMA.
Differentiable Rendering. We modified 3D Gaussian Splatting (3DGS) 37 serve our differ- entiable renderer, enabling image gradient computation relative to simulation results. Unlike other differentiable renderers like mesh-based rasterizers 36 50 51] ray tracers 44
62 and neural
radiance fields 44 558
59
63
801, 3DGS, using explicit 3D Gaussian kernels, is more compatible
with physical simulations. Some studies I7
96 combine 3DGS with physical simulators to generate realistic motions, treating Gaussian kernels as physical particles. However, Gaussian kernels, meant for rendering, can render objects effectively even when unevenly distributed. In contrast, physical simulations require particles to evenly fill objects for accurate representation. Our Particle-GS method calculates the Gaussian kernel and the neighboring relationships of physical particles, adjusting the Gaussian kernel based on nearby physical particles. This decouples property representations, ensuring both accurate rendering and physical simulation.
85
95
E
Introduction to the Material Point Method
E.1
Introduction
The Material Point Method (MPM) is numerical technique for solving continuum mechanics problems. Since its first publication in 1994 (72). MPM has greatly matured and become more accurate and stable for simulating the behavior of wide range of continuum materials. To be self-contained, we provide here an overview of the derivation of MPM from the fundamental partial differential equations (PDEs), which completes the definition of the time integration scheme I used in Equation (
For detailed derivations, please refer to previous literature 33
93
E.2
Governing Equations
MPM is derived from the conservation laws of mass and momentum in their Eulerian form:
dp pV
(10)
dl
pa V a + pb,
(11)
where p represents density, v is velocity, a denotes acceleration, a is the Cauchy stress tensor, and stands for body forces. Mass conservation is inherently maintained by advecting the Lagrangian particles in MPM.
E.3 Weak Formulation
To obtain the weak form of the momentum conservation equation, multiply it by test function w and integrate over a domain Ω:
I pa wdlt / (v
(V.o) .r) wdl pb wdlt pd-
(12)
This equation must hold for any test function and any domain ft Using integration by parts, convert the weak form into:
[
pa wah = =-
n o: Awd? + Janr w To-TdS+ +
/
/ pb wdd? pbdddl,
(13)
o
where T is the traction on the boundary ØΩT.
E.4 Spatial Discretization
MPM discretizes the weak form using basis functions for both the material points and the grid. This results in the following discretized system:
No
No
No
(b)a(b) = Σ
(())).)(())(()
(14)
bm
19

---

Algorithm 2: Time integration scheme of MPM
Input:
Positions (F)()) velocities me(3), and elastic deformation gradients (i) of each material point ,,...,,y, Np time
Output: Positions Ex+1(() (t), velocities (1)) trial elastic deformation gradients F
(t) of
each material point -Np at time t + 1
Particle-to-Grid Transfer: Transfer the Lagrangian particle data to the Eulerian grid by computing for b = ,,...,N:
(b
Σ
No No (z)
Np
m((bvu((b) = Σ
No x (i)) M (i)os())
(b) = (F)
Nr
PO
(F
f1(b) Σ
((x. (o) (x) M(1)
Po
2 Solve Governing Equations on Grid: Update the grid quantities by computing for == I,...,NG
14+1 (b) 1
me(b) (f7 - fi(()
Ave+1 (b) (it1 (b)
U-+1 (b) (b) + 10++1(b)
3 Grid-tt-Parricle Transfer: Update the particle velocities and deformation gradients by computing for .p
NG No (ox (b))
(1)
b=1
No 0b41(b) (N(( Fi(1)
F trial Fortial
I
Update Particle Positions: Update the positions of the particles.
(1) x ve+1 (t)
where Mas = EN, M(i) (a(iii)) (z(i)) is the full mass matrix. Here, Vo(1), (1), (1). af1), and b(i) represent the initial volume, mass, Kirchhoff stress, current position, and body force of material point i, respectively. N₂ and atb) denote the basis function and acceleration at the Eulerian grid node with NG being the number of grid nodes.
E.5 Temporal Discretization
For time discretization, we use the explicit Euler method with time step size At:
No Mab
1 (b) vr(b)
Nº
No
Σ
+ Σ (Z)) (15)
Al
20

---

E.6 MPM Algorithm
We summarize the time integration scheme of the MPM algorithm, implemented under the MLS- MPM framework by [II, in Algorithm 2 used in our framework.
This completes the procedure of the physics simulation
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We highlighted our contributions and novelties in the Abstract and the second to last paragraph in the Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. It is fine include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of our work in Section 5.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
21

---

3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The related theories about physics simulation are well-explored in other works, and we clearly state their work in the manuscript.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The manuscript clearly describes the implementation, training, and evaluation details, while the dataset details are summarized in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
22

---

reproducible, they In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We have released our synthetic benchmark data and code through the project page shown the end of the Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
http:///bbb//ggiiiee/ecoceeuaaaa Swt.///oodSSssss.eoooea wttpsuummmsssisnooii sswiiisiiccyiiiiiic for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Refer to training details in Section 3.4 and evaluation setup in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Error bar is not available to our task.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
23

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer to Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: : newrips..... httpripp..cppppcpeeeeeeeee cc.ppulicetthiicccddee.........
Answer: [Yes]
Justification:
We have read and fully agree with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is an intersection cognitive science and machine learning. We cannot imagine the negative impact this work may have society the moment.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work does not involve this issue.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We stated the license in the submission.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/dataseta
datasets.papersstt
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Question:
Answer: [Yes]
Justification: The six synthetic benchmarks are introduced in Section 4.1 and have been released project page.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [No]
Justification: We used objective evaluation metrics (e.g. PSNR, SSIM, LPIPS and Chamfer distance) and did not conduct human study.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
26

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "CEHZLPeVsv", "reviewer": "Reviewer_se8S", "summary": "NeuMA is a technique to learn residuals on top of physics models to better capture intrinsic dynamics of non-rigid materials.  The paper uses Gaussian splatting (GS) to obtain differentiable rendering, and update the NeuMA model by minimizing reconstruction error of visual observations.  The paper also proposes the use of a particle alignment step that allows simulating more uniformly through the material with minimal impact on the position of the Gaussians.  Evaluations show improved performance over similar approaches and ablations.", "questions": "1. Emphasis is placed throughout the paper on single-camera inputs, but the initial shape creation requires multiple views.  Is this not contradictory?  I see that distinctions are required because previous approaches require multiple views of the full dynamic trajectory.\n1. Does the material need to be known for correct application of $\\mathcal{M}_0$?  What is the effect of a misalignment of actual material and the heuristically selected model?\n1.  Does the binding between simulated and GS particles need to be updated as timesteps increase and the material deforms substantially?", "limitations": "Limitations adequately addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well structured, and the methods explained well.  The diagrams are informative and clear.  The results seem to be qualitatively interesting, and quantitatively superior to the baselines considered.  The particle-GS step is a clever method to bind particle-based simulation with Gaussian splatting.", "weaknesses": "1. It is a bit unclear to me how the generalization to novel objects and object interaction (in section 4.4) should be evaluated.  This is one of the more exciting uses of the NeuMA approach, but gets very little attention in the paper.  I would appreciate a bit more exploration of how this generalization.  Can these experiments be quantified and compared to other methods?  See the next point for one additional possible ablation.\n1. NeuMA interpolates between black- and white-box approaches arbitrarily with a parameter, $\\alpha$.  It is unclear what role $\\alpha$ plays in the NeuMA approach.  From the provided examples in Figures 8 and 9, it seems that (qualitatively), reconstruction improves as more weight is given to $\\Delta\\mathcal{M}_\\theta$.  This raises the question, what is the benefit of $\\mathcal{M}_0$?  Have the authors done experiments with only the learned model?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your positive and constructive comments, and try to address your concerns below.\n\n> W1: It is a bit unclear to me how the generalization to novel objects and object interaction (in section 4.4) should be evaluated ... Can these experiments be quantified and compared to other methods?\n\nWe supplement quantitative results on dynamics generalization by evaluating the L2-Chamfer distance (L2CD) between grounded and ground-truth particles in the simulation space. Note that we scale L2CD by $10^4$ in all experiments (same below).\n\n| Method | Bouncy -> \"N\" | Rubber -> \"N\" | Sand -> \"N\" | Ball & Cat |\n| :--- |:---:|:---:|:---:|:---:|\n| NeuMA |   **0.99**    |     0.36      |    0.33     |  **0.71**  |\n| NeuMA *w/* P.S.  |     1.16      |   **0.32**    |  **0.30**   |    0.73    |\n| NeuMA *w/o* Bind |     4.26      |     14.99     |    0.48     |    1.19    |\n| NeuMA *w/o* LoRA |     1.78      |     0.45      |    0.36     |    0.91    |\n\nIn this table,\n-  `Bouncy -> \"N\"` means that we apply the learned NeuMA on BouncyBall directly to the letter \"N\" and evaluate the generalization performance (the same with `Rubber -> \"N\"` and `Sand -> \"N\"`). \n- `Ball & Cat` is the quantitative result of the interaction between the ball and the cat in Figure 7 in the manuscript. \n\nFrom the table, we observe that NeuMA achieves favorable performance over other methods in dynamics generalization.\n\n> W2: It is unclear what role $α$ plays in the NeuMA approach. From the provided examples in Figures 8 and 9, it seems that (qualitatively), reconstruction improves as more weight is given to $Δ\\mathcal{M}_𝜃$. This raises the question, what is the benefit of $\\mathcal{M}_0$? Have the authors done experiments with only the learned model?\n\nWe implement the neural material adaptor $Δ\\mathcal{M}_\\theta$ using the low-rank adaptation (LoRA) and $r,α$ are two hyperparameters for LoRA. Specifically, $r$ is the rank of the trainable weights and $α$, which is commonly set to equal to $r$ during training, can be tuned during inference as a scaling parameter to modify the influence of LoRA on the base model. In our case, $α$ is like a weight coefficient on the adaptor: When $α=0$, it means we do not alter our prior (*i.e.,* the base model $\\mathcal{M}_0$); when $α$ gets larger, the generated dynamics will become more similar to the given observation. In general, LoRA could not be evaluated alone without the base model. Please refer to `Q2` below, where we study the benefit of $\\mathcal{M}_0$ by changing different $\\mathcal{M}_0$ for dynamics grounding.\n\n> Q1: Emphasis is placed throughout the paper on single-camera inputs, but the initial shape creation requires multiple views. Is this not contradictory?\n\nWe emphasize the use of a single camera as this eases the burden of camera synchronization for capturing the full dynamic trajectory, which is required by previous works like PAC-NeRF. During the initial state acquisition, we can use a single camera to capture multi-view images to ensure an accurate modeling of the object appearance and geometry for later visual grounding. Moreover, thanks to the physical prior $\\mathcal{M}_0$, we have a rough guess of the object motion and could perform dynamics grounding given single-view camera observation. We will clarify our setting in the revision.\n\n> Q2: Does the material need to be known for correct application of $\\mathcal{M}_0$? What is the effect of a misalignment of actual material and the heuristically selected model?\n\nIn previous experiments, we assume a correct material model is known as the prior $\\mathcal{M}_0$ for dynamics grounding. It should be noted that the underlying material parameters (*e.g.,* Young’s modulus and Poisson’s ratio) are unknown. Here, we study the effect of inaccurate application of $\\mathcal{M}_0$. We choose two plastic objects, RubberPawn and ClayCat, for this experiment. The specific settings and L2CD results are shown below. We also show the visual grounding results for RubberPawn in **Figure B in the attached PDF**.\n\n| Setting | $\\mathcal{M}_0^e$ | $\\mathcal{M}_0^p$ | RubberPawn | ClayCat | Rubber -> \"N\" | Clay -> \"N\" |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n|    I    |       StVK        |     von Mises     |    1.27    |  1.00   |     0.36      |    1.23     |\n|   II    |    Neo-Hookean    |     von Mises     |    1.94    |  0.91   |     0.36      |    1.04     |\n|   III   |  Fixed Corotated  |     von Mises     |    1.95    |  1.60   |     0.36      |    1.47     |\n|   IV    |  Fixed Corotated  |     Identity      |    3.64    |  3.22   |     0.70      |    1.31     |\n|    V    |       StVK        |  Drucker-Prager   |   30.26    |  12.91  |     3.91      |    3.31     |\n\nIn this table, \n- Setting `I` refers to our previous experimental setting in which the correct material model is set as the physical prior. \n- Settings `II` and `III` adopt inaccurate elastic material models, but the resulting motion still conforming plastic material. \n- Settings `IV` and `V` are more challenging, as the former is commonly used for simulating elastic objects and the latter for granular objects.\n\nFrom the table, we can see that our method can handle moderate deviation from correct material models (*e.g.*, Setting II and III). However, when the physical prior is completely wrong, (*i.e.*, Setting IV and V), the performance would undergo an obvious decrease. As a remedy, it may be helpful to leverage Large Language Vision Models (LLVM) like GPT-4o to give a plausible guess of the material models given some key frames of the visual observation.\n\n> Q3: Does the binding between simulated and GS particles need to be updated as timesteps increase and the material deforms substantially?\n\nFor efficiency, we only compute the binding matrix during the initial stage. This strategy also works well for materials with large deformations (*e.g.*, SandFish from our synthetic data).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wz9kftEMCX", "reviewer": "Reviewer_TxrR", "summary": "The authors propose a new framework to introduce motion dynamics with residuals using a single view on top of a 3D Gaussian splatting based reconstruction of an object obtained from multiple views from the same camera. This approach models motion using physical laws and learned residuals which allows for interpretability while also allowing for transferring material properties to new objects. The authors evaluate their method against existing methods on a variety of materials and geometry and demonstrate a consistent improvement in both dynamics and novel view quality.", "questions": "- Uniformly sampling points inside the object to gather the initial state seems to assume that the geometry is a closed volume. What would happen if the given object is not guaranteed to be closed? Have you considered other sampling methods - for example near the surface? How would these other sampling methods impact quality?\n- The initial state of Gaussian kernels is constructed from multiple views but the intrinsic dynamics are inferred from a single view. How is this view chosen?\n- How would the technique behave when applied to geometry with uneven mass distribution?\n- What assumptions are made when applying a pretrained NeuMA to a new object?\n- What are some failure cases where the initial state estimate may not produce good results?", "limitations": "The authors list a few limitations including the need for calibrated multi-view cameras for the initial 3D reconstruction, the accumulation of errors over time and susceptibility to errors due to motion blur.\n\nThe objects used in the simulations appear to be small in size and of uniform material properties. It would be interesting to have some results on larger objects where motion from one end of the object doesn't directly affect the other end or the material varies through the volume.\n\nAnother interesting set of evaluations would be on non-convex objects that change topology over time. These would likely highlight limitations of the initial 3D reconstruction which stays constant over time.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is mostly well written and easy to follow but a few parts gloss over details relevant to the design decisions.  The experiments are well designed, compared to multiple relevant existing methods and cover both the dynamics and novel view synthesis aspect. Ablations of various aspects of the technique demonstrate their respective importance. The technique goes from modeling motion directly to modeling motion residuals which improves the quality of the reconstruction over time and reduces accumulated errors visible in prior works.", "weaknesses": "The method appears to not be evaluated on real-world data where assumptions about motion dynamics may not hold perfectly. It would be interesting to visualize the motion residuals and quantify the errors to better understand their characteristics over time. The paper glosses over some relatively minor design decisions such as how is the single view for grounding dynamics chosen, dimensions of the geometry volume and applying the method to a new object.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for identifying our work and providing valuable comments. We try to address your concerns below.\n\n> W1: The method appears to not be evaluated on real-world data where assumptions about motion dynamics may not hold perfectly. It would be interesting to visualize the motion residuals and quantify the errors to better understand their characteristics over time. The paper glosses over some relatively minor design decisions such as how is the single view for grounding dynamics chosen, dimensions of the geometry volume and applying the method to a new object.\n\n**Real-world data.** We supplement the real-world experiment using data captured by Spring-Gaus and present the result in **Figure A in the attached PDF**. We observe that NeuMA's grounded dynamics are more accurate than the competitor in terms of view synthesis metrics, which verifies the effectiveness of NeuMA in the real world. Note that ground-truth particles are unavailable for real data, so we do not present the Chamfer distance here.\n\n**Visualization of motion residuals.** We show visual results of motion residuals in **Figure D in the attached PDF** to quantify the dynamics grounding errors. The color of each particle indicates its sided Chamfer distance to the ground truth. Please also refer to Figure 3 of the manuscript, where we display the average motion residuals per time step over the simulation trajectory. These results verify the superiority of our method over baselines.\n\n**Single view Choice.** Please refer to `Q2` below.\n\n**Geometry volume.** We use 3D geometry volume, and the grid size is $32^3$.\n\n**Generalization procedure.** The procedure of applying NeuMA to a new object is: (1) Acquire the 3D Gaussian and surface mesh of the new object from multi-view images (details refer to Section 3.1); (2) Set a user-defined initial velocity and select a material adaptor $Δ\\mathcal{M}_\\theta$ with the physical prior $\\mathcal{M}_0$ to generate the physical-plausible animation.\n\n> Q1: Uniformly sampling points inside the object to gather the initial state seems to assume that the geometry is a closed volume. What would happen if the given object is not guaranteed to be closed? Have you considered other sampling methods - for example near the surface? How would these other sampling methods impact quality?\n\n**Open volume.** Thoughtful viewpoint! We use the Material Point Method (MPM) for differentiable simulation in 3D space and require the 3D volume to be set for each particle. Thus, it is difficult for our method to handle curves or thin surfaces without 3D volume. However, if the object has a well-defined 3D volume, our method should work for either closed or open objects. To verify this, we present the dynamics grounding result of an open container in **Figure F in the attached PDF**. It is observed our method can address open volume.\n\n**Particle sampling.** We assume the object is filled with mass, so we use volume sampling instead of surface sampling, since the latter could lead to unrealistic simulations if the interior space is large. For an intuitive understanding, please refer to the results of \"NeuMA *w/o* Bind\" shown in Figure 10 of the manuscript. In previous experiments, NeuMA *w/o* Bind could be considered as using the surface sampling since we directly treat Gaussian kernels (sprinkled near the object surface) as physical particles for dynamics grounding. For objects with thin structures, like the open container in Figure F, surface and volume sampling yield similar results, and thus their dynamic grounding results are comparable. We also present the L2-Chamfer distance (L2CD) between the grounded and ground-truth particles in this case.\n\n| Open Container | Volume Sampling | Surface Sampling |\n| :---: | :---: | :---: |\n|L2CD ($\\times 10^{-4}$)| 1.28 | 1.51 | \n\n> Q2: The initial state of Gaussian kernels is constructed from multiple views but the intrinsic dynamics are inferred from a single view. How is this view chosen?\n\nWe use multi-view static observations required by vanilla 3D Gaussian Splatting (3DGS) to accurately capture the object's appearance and geometry. For the single-view dynamic observation, we typically select a frontal view of the object and try to minimize self-occlusion. Note that our method also supports multi-view dynamic observation if such data is available.\n\n> Q3: How would the technique behave when applied to geometry with uneven mass distribution?\n\nWe conduct an experiment on an object with uneven mass shown in **Figure E in the attached PDF** by assigning particles with different densities, *i.e.*, $\\rho$. We also quantify L2CD in this case, and the result is $1.33\\times 10^{-4}$. These results show that NeuMA can handle objects with uneven mass.\n\n> Q4: What assumptions are made when applying a pretrained NeuMA to a new object?\n\nWe assume the new object's 3D representations (*i.e.,* 3D Gaussian kernels and surface mesh) are available. Alternatively, we can do 3D reconstruction to get these representations given dense multi-view images. The initial velocity is also required but is user-defined. \n\n> Q5: What are some failure cases where the initial state estimate may not produce good results?\n\nWhen acquiring the 3D representations, our framework inherits the common failure cases from multi-view reconstruction techniques, *e.g.,* (1) the camera parameters are inaccurate, and (2) the static captures are too sparse. Besides, the prediction of initial velocity may be inaccurate if the object exhibits complex motions at the beginning (*e.g.*, different parts have different initial velocities), as we currently assume a uniform initial velocity.\n\n> Limitation: Large objects and non-convex objects.\n\nNice suggestion! In **Figure G, H in the attached PDF**, we show the dynamics grounding results of a large object and a non-convex object separately. The quantitative results are $0.91\\times 10^{-4}$ and $0.51 \\times 10^{-4}$. The results verify the effectiveness of NeuMA on complex geometries.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q2xU6b5bHq", "reviewer": "Reviewer_zoZv", "summary": "This paper proposes the Neural Material Adaptor (NeuMA), a framework which integrates existing physical laws with learned corrections, thus facilitating the accurate learning of actual dynamics, while also maintaining generalizability and interpretability of physical priors. The framework also proposes Particle-GS, a particle-driven 3D Gaussian Splatting variant, bridging between simulation and observed images, and thus permitting to back-propagate image gradients to optimize the simulator. Various experiments on different dynamics (in terms of grounded particle accuracy, novel-view quality, and generalization ability), demonstrate that NeuMA can accurately capture intrinsic dynamics", "questions": "Does your framework involve the same number of particles if the objects do not deform in normal situations?", "limitations": "1.Acquisition of the initial-state particles requires calibrated cameras not performing well on complex scenes.\n2. Reducing cumulative errors in the forward particles simulation.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. New correction term for the material law.\n2. New binding mechanism between particles and Gaussian kernels.\n3. Integration with simulation and rendering.\n4. Increased accuracy and generalizability.", "weaknesses": "1.Acquisition of the initial-state particles requires calibrated cameras not performing well on complex scenes.\n2. Reducing cumulative errors in the forward particles simulation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your supportive evaluation of our work. We provide the answers to your question below.\n\n> Q1: Does your framework involve the same number of particles if the objects do not deform in normal situations?\n\nYes. Following the common practice in differentiable physics [25,44,50,88], our method assumes the number of physical particles is invariant across the entire simulation trajectory.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HBmKJbebL5", "reviewer": "Reviewer_Rwta", "summary": "This paper introduces a novel pipeline to train the neural constitutive model via image supervisions. It corrects the existing physical material law with a kind of residual learning, and supervise the parameters by aligning groundtruth images the rendered images which are pictured through differentiable renderer techniques.", "questions": "See weakness", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The work conceptually belongs to neural physics dynamics field, which could be somewhat encouraged. It combines the conventional physics knowledge with currently popular neural rendering techniques, might inspire both vision and physics area. \n\n1.\tThis paper is well-written and is friendly to those who don’t have enough background in material mechanics.\n\n2.\tThis paper chooses to learn the correction terms via LoRA instead of training the neural constitutive law from scratch, which maintains the original physical priors well and decrease the training difficulties. \n\n3.\tThe learned material laws can somewhat generalize to large-gap scenes and produce physically plausible motions with various materials.", "weaknesses": "Here are still some technical doubts about this method. \n\n1.\tThe dataset should be described in more detail. For example, the number of views to initialize the scene, and the number of views for the subsequent optimization. How many episodes are used to train each kind of material? These factors are important because they will tell readers the efficiency of the learning-based material law. \n\n2.\tIn Appendix C, the authors claim that they provide videos in supplementary materials, but I haven’t found them in the submission system. \n\n3.\tIn Table 1, Fig. 3, and Fig. 4, the results show that the NeuMA overall outperforms the NeuMA w/ PS. I think this could be a bit weird. The groundtruth image sequences are generated by the the groundtruth 3D particle tracks. Thus, the information contained in the particle supervision should not be less than that in the 2D image supervision. However, the results shown by the authors illustrate that only using 2D image supervisions is better than using the 3D particle correspondence. I think the metrics of the model trained by the 3D labels should be the upper limitation of the methods and the result of 2D supervised methods should try to approximate it but cannot entirely surpass it so much. Therefore I think the author did not train the 3D baseline thoroughly, which may be due to the improper selection of training strategy or base model.\n\n4.\tThe lack of real-world examples could be a small but not severe issue to prove the practicability of the proposed method. If possible, the author should provide some realistic examples. If not, the author should claim the reason for the limitation. \n\nI will reconsider the score according to the answers of the author", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work. Your feedback is instrumental in strengthening our paper. Here are our responses to your concerns.\n\n> W1: The dataset should be described in more detail. For example, the number of views to initialize the scene, and the number of views for the subsequent optimization. How many episodes are used to train each kind of material? These factors are important because they will tell readers the efficiency of the learning-based material law. \n\nThank you for your thoughtful consideration. Each scene from our synthetic data contains 50 uniformly sampled viewpoints for initial state estimation, with the cameras evenly spaced on a sphere covering the object of interest. In addition, we record 400 frames depicting the object’s subsequent motion from a single view. We use the first 20 frames from the dynamic observation to infer the object's initial velocity and the entire trajectories for material adaptor optimization. We train each material adaptor for 1,000 iterations.\n\n> W2: In Appendix C, the authors claim that they provide videos in supplementary materials, but I haven’t found them in the submission system. \n\nWe apologize for not managing to upload the video demonstrations of our synthetic dataset at submission. As a remedy, we supplement some key frames to show the object motion in **Figure C in the attached PDF**. We will make sure to release the synthetic data later. \n\n> W3: In Table 1, Fig. 3, and Fig. 4, the results show that the NeuMA overall outperforms the NeuMA w/ PS. I think this could be a bit weird. The groundtruth image sequences are generated by the the groundtruth 3D particle tracks. Thus, the information contained in the particle supervision should not be less than that in the 2D image supervision. However, the results shown by the authors illustrate that only using 2D image supervisions is better than using the 3D particle correspondence. I think the metrics of the model trained by the 3D labels should be the upper limitation of the methods and the result of 2D supervised methods should try to approximate it but cannot entirely surpass it so much. Therefore I think the author did not train the 3D baseline thoroughly, which may be due to the improper selection of training strategy or base model.\n\nThoughtful viewpoint! The mentioned results may seem weird at first glance but can actually be attributed to the following aspects.\n- **Texture information**: Although the image sequences are generated from ground-truth particles, they are rendered with color under a given light source, which contains information about texture and shading that helps regularize the optimization of the neural material adaptor.\n- **Particle binding**: Note that NeuMA *w/* P.S. does not use the differentiable renderer and, therefore, cannot leverage the particle binding scheme proposed in Section 3.1 of the manuscript. In contrast, our method explicitly models the relationship between 3D Gaussian kernels and simulation particles via the binding mechanism. Since Gaussian kernels generally grow around object surfaces, each kernel encapsulates local object-part information. Our binding mechanism ensures that particles within a kernel share relatively uniform physical states, thereby reducing the degrees of freedom during optimization. NeuMA *w/* P.S., however, exhibits a higher degree of freedom as it is directly supervised by the 3D positions of tens of thousands of particles. This increased complexity can make the optimization of NeuMA *w/* P.S. challenging, especially given that the trainable parameters are of low rank.\n\nPlease note that in our previous experiments, we adopt the same hyperparameter settings and the number of training iterations for both NeuMA and NeuMA *w/* P.S. To investigate whether the baseline model is not thoroughly tuned, we conduct additional ablation studies using the BouncyBall benchmark on the learning rate of NeuMA *w/* P.S. in the table below. We report the L2-Chamfer distance (L2CD) between the grounded and ground-truth particles here. These values are scaled by $10^4$.\n\n ori. lr $\\times 0.25$ | ori. lr $\\times 0.5$ | ori. lr | ori. lr $\\times 1.5$ | ori. lr $\\times 2.0$ |\n :---: | :---: | :---: |:---:|:---:|\n 1.69                  | 1.62                 | 1.45    | 2.00                 | 2.32                 | \n \nFrom the table, it is observed that the results obtained by these variants still underperform our method (with a value of 1.19).\n\nIt is also worth noting that a previous method, PAC-NeRF [44], also observes a similar phenomenon: A 2D pixel-level supervision better optimizes the physical parameters as opposed to a 3D metric. Based on the above analysis, we argue the experimental results should be plausible.\n\n> W4: The lack of real-world examples could be a small but not severe issue to prove the practicability of the proposed method. If possible, the author should provide some realistic examples. If not, the author should claim the reason for the limitation. \n\nFollowing your suggestions, we supplement dynamics grounding results on real-world data. Specifically, we use the data collected by Spring-Gaus and adopt its experimental setting for experiments. Please kindly refer to **Figure A in the attached PDF** where we show the grounding results achieved by NeuMA and Spring-Gaus. Note that for real-world data, the ground-truth particles are unavailable and thus we do not present the Chamfer distance here. It is observed that the rendered sequences of NeuMA are more aligned with the observations than those of Spring-Gaus both qualitatively and quantitatively. The results confirm the effectiveness of NeuMA in real-world scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their time and efforts on reviewing the paper. We are excited to see that reviewers recognized the novelty of our technical contribution (Reviewer TxrR, zoZv, Rwta), acknowledged a better performance achieved by our method over baselines (Reviewer se8S, TxrR, zoZv), and found the paper well-structured and easy-to-follow (Reviewer se8S, TxrR, Rwta).

We also appreciate the reviewers for their constructive comments and concerns. In the attached PDF file, we provide additional visualizations for more details. We summarize the contents in the attached file below.

- **Figure A (to Reviewer TxrR, Rwta)** presents comparisons on dynamics grounding using real-world data captured by Spring-Gaus [a], showing that NeuMA achieves favorable performance on real-world dynamics over the baseline.
- **Figure B (to Reviewer se8S)** illustrates the effect of inaccurate application of $\mathcal{M}_0$, showing that NeuMA can tolerate a wrong material prior to some extent.
- **Figure C (to Reviewer Rwta)** displays our synthetic dataset.
- **Figure D (to Reviewer TxrR)** visualizes the motion residuals achieved by different methods to better demonstrate their grounding performance.
- **Figure E (to Reviewer TxrR)** presents NeuMA's dynamics grounding result on an object with uneven mass distribution.
- **Figure F (to Reviewer TxrR)** presents NeuMA's dynamics grounding result on an open container without closed volume.
- **Figure G (to Reviewer TxrR)** presents NeuMA's dynamics grounding result on a large object where motion from one part does not directly affect another part.
- **Figure H (to Reviewer TxrR)** presents NeuMA's dynamics grounding result on a non-convex object with topology changing over time.

Note: Data used in Figures F, G, and H are from Poly Pizza.

[a] Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3D
gaussians. In ECCV, 2024.

Meta-review: Most reviewers agreed that this paper is impactful and advances the state of the art. I believe the concerns of the one hold-out were adequately addressed during the rebuttal period, although they did not respond. For these reasons I recommend acceptance.
Decision: Accept (poster)