OmniPhysGS: 3D Constitutive Gaussians for General Physics-Based Dynamics Generation
Authors: Yuchen Lin
Chenguo Lin
Jianjin Xu
Yadong MU
Venue: ICLR 2025 Poster (2025)
Keywords: Physics-based Modeling
3D Dynamics
3D Gaussian Splatting
Video Score Distillation
Primary area: generative models
TL;DR: We introduce a novel framework for general physics-based 3D dynamic scene synthesis, which can automatically and flexibly model various materials with domain-expert constitutive models in a physics-guided network.

Abstract: Recently, significant advancements have been made in the reconstruction and generation of 3D assets, including static cases and those with physical interactions. To recover the physical properties of 3D assets, existing methods typically assume that all materials belong to a specific predefined category (e.g., elasticity). However, such assumptions ignore the complex composition of multiple heterogeneous objects in real scenarios and tend to render less physically plausible animation given a wider range of objects. We propose OmniPhysGS for synthesizing a physics-based 3D dynamic scene composed of more general objects. A key design of OmniPhysGS is treating each 3D asset as a collection of constitutive 3D Gaussians. For each Gaussian, its physical material is represented by an ensemble of 12 physical domain-expert sub-models (rubber, metal, honey, water, etc.), which greatly enhances the flexibility of the proposed model. In the implementation, we define a scene by user-specified prompts and supervise the estimation of material weighting factors via a pretrained video diffusion model. Comprehensive experiments demonstrate that OmniPhysGS achieves more general and realistic physical dynamics across a broader spectrum of materials, including elastic, viscoelastic, plastic, and fluid substances, as well as interactions between different materials. Our method surpasses existing methods by approximately 3% to 16% in metrics of visual quality and text alignment.

Published as a conference paper at ICLR 2025
OMNIPHYSGS: 3D CONSTITUTIVE GAUSSIANS FOR GENERAL PHYSICS-BASED DYNAMICS GENERATION
Yuchen Lin', Chenguo Lin Jianjin Xu², Yadong Mu!!
'Peking University, "Carnegie Mellon University
https: / //wgsx/..mmmm/.////../////.///./ github. /o/projjetss/mmillyyggs
ABSTRACT
Recently, significant advancements have been made in the reconstruction and gen- eration of 3D assets, including static cases and those with physical interactions. To recover the physical properties of 3D assets, existing methods typically as- sume that all materials belong to a specific predefined category (e.g., elasticity). However, such assumptions ignore the complex composition of multiple heteroge- neous objects in real scenarios and tend to render less physically plausible anima- tion given wider range of objects. We propose OMNIPHYSGS for synthesizing a physics- based 3D dynamic scene composed of more general objects. A key de- sign of OMNIPHYSGS treating each 3D asset as a collection of constitutive 3D Gaussians. For each Gaussian, its physical material is represented by an ensem- ble of 12 physical domain-expert sub-models (rubber, metal, honey, water, etc.), which greatly enhances the flexibility of the proposed model. In the implementa- tion, we define a scene by user-specifies sser-specified prompts and supervise the estimation of material weighting factors via pretrained video diffusion model. Comprehensive experiments demonstrate that OMNIPHYSGS achieves more general and realistic physical dynamics across broader spectrum of materials, including elastic, vis- coelastic, plastic, and fluid substances, as well as interactions between different materials. Our method surpasses existing methods by approximately 3% to 16% in metrics of visual quality and text alignment.
INTRODUCTION
Synthesizing realistic 3D dynamic (i.e., 4D) scenes has emerged as attractive task with the devel- opment of 3D reconstruction and video generation techniques. Recent advances in 3D differentiable rendering (Mildenhall et al., 2020; Kerbl et al., 2023) establish effective grounds for learning-basee dynamic scene synthesis. Although some methods (Ren et al., 2023; Zhao et al., 2023; Yin et al., 2023b; Jiang et al., 2024b; Ling al., 2024) have presented vivid 3D dynamics, the non-physical nature of data-driven approaches inevitably leads to artifacts and inconsistencies since the learned models are not strictly constrained by the physical laws governing the real world. To this end, we advocate that the incorporation of physical priors is essential for 4D scene synthesis, which guides the learning process to generate more realistic and physically plausible results.
In this work, we choose to use 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) to represent the scene due to its particle nature, allowing the assignment of physical properties to each Gaussian par- ticle. Related works (Zhong et al., 2024; Fu et al., 2024; Lin et al., 2024; Qiu et al., 2024; Borycki et al., 2024; Feng et al., 2024) have investigated the integration of physical priors in the context of Gaussian Splatting. PhysGaussian (Xie al., 2024) initially introduces the Material Particle Method (MPM) (Stomakhin et al., 2013; Jiang et al., 2016) to simulate the dynamics of 3DGS. Subsequent studies (Zhang et al., 2024b; Liu et al., 2024; Huang et al., 2024) extend MPM to learning-based dynamic 3D scene synthesis by modeling physical properties with video supervision (Poole et al., 2023). However, some methods require manual tuning of physical properties (Xie et al., 2024), which is time-consuming and may lead to suboptimal results, as estimating material properties re- quires expert knowledge. Other methods are limited to specific material types, such as pure elastic materials (Zhang et al., 2024b; Huang et al., 2024) or viscoelastic materials (Liu et al., 2024), but do not support multiple materials within a single scene, As a result, they fail to offer a general and automatic solution for dynamic scene synthesis.
t: Project lead; t Corresponding author.

---

Published as a conference paper at ICLR 2025
Existing methods (fixed physical models): Fails to generalize
Folling
PhysDreamer (Zhang et al.)
Elastic
On not modd
Model
Render
Elastic Strength
Supervise
Video Dillusion
Our method (learnable physical models): Generalizes
! Folling
OmniPhysGS (Ours)
Elastic Gaussians
Non-elastic Gaussians
Randomly Init Constitutive
Render
Physical Model
Estimation
Supervise
Video Diffusian
Deformation
"A rubber duck falls on a soft mud pile"
Learned Constitutive Gausslans
Figure I: Comparison with Previous Methods. Existing methods rely on handcrafted or narrowly restrictive physical models (e.g., pure elasticity) that limit generalizability. Our method introduces Constitutive Gaussians to better represent physical materials, thus achieving more automatic and physically plausible dynamic synthesis of various materials within unified framework.
Our task is to synthesize general physics-based 3D dynamics, where general dynamics is defined as generating dynamics for a wide range of materials, including pure elastic, viscoelastic, plastic, and fluid substances, as well as interactions between different materials within a single scene. To achieve this, ideal physical guidance should (1) provide strong physical constraint for the dynam- ies, (2) be flexible handling various kinds of materials and objects, and (3) be capable of learning without the burden of manual modeling. Constitutive models (Gonzalez. & Stuart, 2008; Jiang et al., 2016) are physical models that describe the material responses to different mechanical conditions, providing the relationship between two physical quantities such as stress and strain. These models, which are typically derived from experts' observations of material mechanisms, are decisive factors in determining the dynamics of different materials. To enhance the flexibility and automation of dynamic synthesis, it's crucial to generalize the constitutive model instead of manually setting it as previous works (Xie al., 2024; Zhang et al., 2024b; Liu al., 2024; Huang et al., 2024) did.
In light of this, we propose OMNIPHYSGS as a novel framework for general maysics-basce 3D dy- namic synthesis. Our method starts with obtaining static Gaussian particles from multi-view images, and we adopt MPM to simulate the dynamics of 3DGS. To generalize the representation of physical materials, we extend vanilla Gaussians to the proposed Constitutive Gaussians by introducing learnable constitutive model for each Gaussian particle. Constitutive Gaussians are designed as physics-guided neural network, where features of Gaussian kernels are extracted and processed by physical-aware decoder to predict strain and stress. These predictions are incorporated with set of expert-designed constitutive models to handle various materials and enhance the physical plausibil- ity of dynamic scenes. The simulated states of Constitutive Gaussians are rendered into video frames and passed a pre-trained video diffusion model with text prompt. Finally, learnable Constitutive Gaussians are optimized with the Score Distillation Sampling (SDS) (Poole et al., 2023).
To the best our knowledge, OMNIPHYSGS the first method to introduce learnable constitu- tive models for physics-based 3D dynamic synthesis (Xie et al., 2024; Zhang et al., 2024b; Huang et al., 2024; Liu et al., 2024). As shown in Figure 1, the core contribution of our method, general bassed 3D dynamic synthesis, refers to the automatic synthesis of dynamics and interac- tions between heterogeneous materials, all while maintaining physical plausibility. This is achieved by leveraging MPM's physical constraints, the generalizability of Constirutiee Gaussians, and the
2

---

Published as a conference paper at ICLR 2025
material knowledge encoded in the video diffusion model. Extensive experiments demonstrate that OMNIPHYSGS can generate physically plausible motions and interactions of various materials, without any manual tuning of physical properties. Comprehensive comparisons with state-of ooftth--artt methods and ablation studies validate the effectiveness of each component of our method.
2 RELATED WORK
4D Generation Most efforts in 4D generation leverage text-to-image and video diffusion mod- els to distill 4D objects using SDS (Poole et al., 2023), employing various dynamic 3D repre- sentations, such as HexPlane (Singer et al., 2023), multi-scale 4D grids (Zhao et al., 2023), K- plane (Jiang et al., 2024b), multi-resolution hash encoding (Bahmani et al., 2024), disentangled canonical NeRF (Zheng et al., 2024) or 3D Gaussians (Ling et al., 2024) with a deformation field, and warped Gaussian surfels (Wang et al., 2024a). To facilitate nult apatial-tempore consis- tency modeling, recent works (Zhang et al., 2024a; Jiang et al., 2024a) concentrate on multi-view video generative models to provide enhanced gradients for distillation. In addition to SDS supervi- sion, other studies propose generating videos first as direct references for appearance and motion to optimize 3D representations (Ren et al., 2023; Yin et al., 2023b; Pan al., 2024; Zeng et al., 2024) or utilizing generalizable reconstruction model to avoid the time-consuming distillation process (Ren al., 2024). However, none of these approaches guarantee the physical fidelity of generated 4D contents, due to the lack of physical constraints during optimization.
Interactive Dynamies Generation Existing works have investigated interactive dynamics gener- ation for both 2D and 3D content with respect to user preferences constraints. For image anima- tion, various initial conditions have been utilized to guide the generation process, such as driving videos (Siarohin et al., 2019a;b; 2021; Karras et al., 2023), keypoint trajectories (Hao et al., 2018; Blattmann et al., 2021; Chen et al., 2023a; Yin et al., 2023a; Li et al., 2024), or text prompts (Ho et al., 2022; Yang et al., 2023; Chen et al., 2023b;e; Zhang et al., 2023). Recent works (Jiang al., 2024a; Ling et al., 2024) extend the interactive dynamics generation to 3D content. To ensure the physical plausibilitty of the generated dynamics, a series of works (Li et al., 2023; Qiu et al., 2024; Borycki et al., 2024; Zhong et al., 2024; Fu et al., 2024; Feng et al., 2024) have proposed to introduce physical constraints. Among these, PhysGaussian (Xie al., 2024) married physical simulation method, MPM, with Gaussian representations. However, since current 3D representations are not naturally endowed with material properties, PhysGaussian requires manually specifying material properties for each particle. Subsequent works (Huang et al., 2024; Zhang et al., 2024b; Liu et al., 2024) attempt to learn physical parameters from diffusion models but are restricted to specific types of materials, such as elasticity. In contrast, our method aims to synthesize 3D dynamics with physi- cal fidelity for various materials and complex object combinations.
METHOD
3.1 PROBLEM STATEMENT
3D Gaussian Splatting (Kerbl et al., 2023) is employed to represent 3D scenes, offering a good trade-off between simplicity and expressivity. A set of 3D Gaussian kernels (xp: Ep, shpp (p) hEP used as a representation, where X, R& the center of the kernel, Ep its covariance matrix, shp represents spherical harmonic coefficients, and Op is the kernel opacity. These kernels can be splatted onto 2D image planes for different cameras to render 3D scene from arbitrary viewpoints.
We are addressing the task of synthesizing physically plausible 3D dynamics for heterogeneous objects. Given 3D scene S represented by a set of 3D Gaussian kernels (xp: Σp, shp, Mppp and text prompt y describing the motion or material of different objects in the scene, our goal is to synthesize the 3D dynamics that align with the prompt and simultancously adhere to physical laws.
3.2 OMNIPHYSGS
As shown in Figure 2, our method simulates the dynamic scene with the Material Point Method and renders a video clip V = pre-trained text-to-video diffusion model & using Score Distillation Sampling (Poole et al., 2023).
I'N 2] via the 3DGS renderer, which is then optimized by a
3

---

Published as a conference paper at ICLR 2025
Apply Forces
Elastic Goussians
Non elactic Gaussians
Physical Model Prediction
Differentiable MPM Simulation
Gaussians
Constitutive Model
gray rubber walf bouncing the surface.
Material Prompt
SDS
Loss
Video Diffusion
Simulated Video
Figure 2: Method Overview. OMNIPHYSGS extends 3D Gaussians with learnable constitutive models, introducing Constitutive Gaussians to the differentiable Material Point Method (MPM). A pre-trained video diffusion model is used to guide the training with Score Distillation Sampling.
During the training process, we propose Constitutive Gaussians that are trained to predict the optimal material properties of each Gaussian particle, enabling an automatic simulation of general 3D dynamics that align with given text prompts. denotes the learnable parameters of Constitutive Gaussians, which can vary for objects in different materials within the same scene as exemplified in Figure 1. Our objective is to find the optimal parameters 0* that maximize the log-likelihood of the distribution modeled by the large pre-trained text-to-video models as:
= arg max log P (V(0))S,y:
3.2.1 CONSTITUTIVE GAUSSIAN
Constitutive models are expert-designed functions characterizing how materials deform under spe- cific mechanical conditions. These models are widely adopted in the field of continuum mechan- ics (Gonzalez& Stuart, 2008; Jiang et al., 2016) to solve the conservation equations of mass and momentum:
Dp + pV v 0, DI
Dv DE
V 7
where
a (xx 1) Fet(F) OF
1 OV
FF F
3333
represents the Cauchy stress tensor. V 3x33 - R3x3 is the hyperelastic energy density function and FE E IR3x3 denotes the elastic part of the deformation gradient F 3R333 In practice, the elastic part of the deformation gradient needs to be corrected as F = (FF), where & R3x3 + g3x3 a return function that applies plasticity constraints FE Both and are referred as constitutive models, which describe the material behavior.
In the Material Point Method (see Algorithm in Appendix A.2), the only factors that determine the material properties of each particle are constitutive models (-( and (b(-) and the physical parameters Y E A in functions Festress and ReturnMap, with the rest of the MPM pipeline being independent of the material properties. To this end, we propose to extend vanilla Gaussian kernels to Constitutive Gaussian kernels, generalizing the learning process of material properties in the MPM simulation. We introduce learnable constitutive model for each Constitutive Gaussian particle
V Vogr
1 f- 120ye
"Oppy:
(4)
where fet. 6pt. and Ophy are the learnable parameters of neural networks that represent the hyper- clastic energy density function, the plasticity return function, and the physical parameters of the material, respectively. Therefore, the training target can be formulated as follows:
min
L (S,y; 1),
(5)
82t.0pt.0phh
4

---

Published as a conference paper at ICLR 2025
(a) Neural Network-only Prediction
Convergence Issue
Force
Physical
MLP
Latent Embeddings
Response
(a)
(b) Consititutive Prediction
Physical aware
Feature
Encoder
(b)
Physical Model Prediction
Figure 3: Constitutive Gaussian Network. The network architecture of Constitutive Gaussians consists of 3D feature encoder and physical-aware decoder. Expert-designed constitutive models are integrated into the decoder to guide the learning process, effectively avoiding the convergence issues faced by vanilla neural networks.
where C is the loss function be introduced in Section 3.2.3.
Different from previous works (Zhang et al., 2024b; Liu et al., 2024; Huang et al., 2024) that keep the constitutive models fixed, Constitutive Gaussians enable the model to capture various material behaviors from both elastic and plastic deformations, thus providing a more flexible and expressive representation of the material properties. Critically, we endow physically plausible simulating framework with the learning capability of material properties, thus achieving the pursuit of physical fidelity, general representation of diverse, heterogeneous materials, and automatic prompt-driven synthesis. The effectiveness of Constitutive Gaussians is demonstrated in experiments in Section 4.2.
3.2.2 PHYSICS GUIDED CONSTITUTIVE NETWORK
A naive approach to generalizing constitutive models involves training neural network from scratch, with the expectation that the pre-trained diffusion model provides sufficient prior knowl- edge for the network to recover the material properties. Several previous works (Ma et al., 2023; Zong et al., 2023) have investigated replacing parts of the MPM simulator generalizing the consti- tutive models with simple neural networks. However, illustrated in Figure 3 and Section 4.3, our preliminary experiments show that neural networks without any physical priors can face difficulties fitting the highly nonlinear constitutive models. Without ground truth for each simulated state, the NN-based MPM simulator becomes unstable and difficult to converge under the guidance of the video diffusion model. To address this issue, we propose to integrate an ensemble of expert-designed constitutive models into the neural network structure, thus utilizing more physical priors to guide the learning process. The physically guided architecture of Constitutive Gaussians consists of two main components: 3D feature encoder backbone to extract features of 3D scenes and physical-aware decoder to predict the material properties of Constitutive Gaussians.
3D Feature Encoder We adopt a similar 3D backbone structure to PointBert (Yu ct al., 2022). Our feature encoder first utilizes the Farthest Point Sampling (FPS) and k-Nearest Neighbors (kNN) algo- rithm to partition the particles into a set of local neighborhoods, and then applies a mini-PointNet (Qi al., 2017a;b) encode the information (xp: Ep. shp, ap) PEN, of Gaussian kernels within each neighborhood M; to a feature vector the MPM simulation, based on the initial positions of the Gaussian kernels, thus avoiding the noise introduced by the unoptimized material properties and improving training efficiency.
Rd. The partition process is only performed once before
Physical-aware Decoder The encoded features f, are then decoded physical- decoder. First, an MLP employed to predict the material category logits for each neighborhood from a set of expert-designed constitutive models. Assuming homogeneous material properties within each neighborhood, a single expert constitutive model is assigned to all particles within each neighbor- hood based on the highest logit value, using hardmax operation with ttraight-through estima- tor (Bengio et al., 2013) as:
TPEN; V/, (FpEN)); FPEN; (Patial PEN.), Ji, ki arg max (f.), MLP) (6)
5

---

Published as a conference paper at ICLR 2025
where the pre-defined functions (-) and (b))) of domain-exper models utilized to calculate the Cauchy stress tensor o and the corrected deformation gradient F, respectively. We collect 12 com- binations of constitutive models, including 3 hyperelastic density functions and 4 plasticity return functions, to cover a wide range of materials, such as pure elasticity, viscoelasticity, plasticity, and fluidity (see Appendix A.4 for details). These expert-designed constitutive models capture various material behaviors, thus providing priors that guide the neural network in learning material proper- ties while ensuring that intermediate results adhere to physical constraints.
The intuition behind our physics-guide constitutive network is that complex scenes can be decom- posed into local, homogeneous neighborhoods consisting of similar fundamental materials, which can be effectively described by depern-designe constitutive models. The hardmax operation en- forces strict adherence to the physical laws, preventing the network from producing physically im- plausible outcomes. The physical-aware decoder is designed to be differentiable, thus enabling the end-to-end training of the entire network. Ablation studies on the network structure are provided in Section 4.3.
3.2.3 DIFFUSION GUIDANCE
Following previous common practice, we adopt the Score Distillation Sampling (SDS) (Poole et al., 2023) as the guidance for training Constitutive Gaussians. SDS distills 3D priors from large pre- trained text-conditioned 2D diffusion models to generate 3D content from text prompts (Lin et al., 2023; Wang al., 2024b; Tang al., 2023). During each training iteration, the MPM simulator starts from the initial state s = {x v F ppEP
JPEP and steps N - times to obtain simulated
states [5 : 8 8',,... ... , 1]. The evolutions of generated positions x and deformation gradients F are then passed to MPM-compatible 3DGS renderer (see Appendix A.3) to generate a video clip -= [1% _4 i j(x-1], J. ModelScope (Wang et al., 2023) is used as the pre-trained 2D diffusion prior & supervise the video clip V. To clarify, noticing that each & and F is dependent on the model parameters 0 = {Oet, Opt, (phy) when the learnable Constitutive Gaussians are integrated into the MPM simulator, the gradient of can be formulated as:
VoLSDS = Eg,e w (E) (Eo (V:&.x) -
av OR. 0x, F 00
3.2.4 TRAINING STRATEGY
We propose a novel strategy for optimizing the long simulation sequence with two key components.
Grouping The MPM steps are first-order Markov process, where the state at time ln only de- pends on the state at time 1n- To stabilize the simulating process, the time interval At must be small enough, thus resulting in large number of steps (N 1e3) during each training iteration. This leads to high memory consumption and gradient aaploding/vanishien chain of gradient propagation. Moreover, off-the-shelf video diffusion models are not designed for such long sequences. To mitigate these issues, we first sample subset of the simulation states uniformly and then group them into mini-batches for diffusion guidance as:
issues given the long
samplle = -f flo
+ (M (M-1)ma) 1jmm:
(G (MMMMMMMMMMMMMMMMMMMMMMMMMM-- 11mm
(8)
Group 0
Group G-1
where the original video clip D is sampled every m frames and grouped into G mini-batches with M frames (M < N) in each mini-batch.
Multiple Mini-Bated Training The first-order Markov property of the Material Point Method determines that each mini-batch is optimized based on the previous one. Since the video diffusion model does not directly provide the ground truth for each simulation state, the training can start from an incorrect state if the previous mini-batchis is not well-optimized. Therefore, different from previous works (Liu et al., 2024; Huang et al., 2024), which optimize through the stages directly, to train each mini batch multiple times before proceeding to the next mini- bini-batch in each training iteration. This strategy Ablation studies in Section 4.3 demonstrate the effectiveness the proposed training strategy, and provide more details of the training strategy in code snippet in Appendix B.4.
propose
enhance training efficiency by gradually correcting the simulation states.

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative evaluations of MPM solvers. We compare the performance of our solver with the MPM solver from NCLaw (Ma et al., 2023) on the material estimation task. We report the average time and memory consumption of train or test iteration on 10 particles with I 10 time steps. Better results are in bold.
MPM Solver
Train MemorymB
Test MemorymB |
Train Time, |
Test Time, +
NCLaw
48,073
2,383
21.3
9.24
Ours
13,889
2,637
47.8
7.21
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Simulator Details Off-the-shelf MPM simulators (Ma et al., 2023; Xie et al., 2024) are based on warp (Macklin, 2022), Python library for high-performance simulation. Despite warp's com- patibility with torch, the communication between these two libraries is both time-consuming and memory-intensive To facilitate end-to-end training, we reorganized the MPM algorithm, transform- ing it into highly vectorized torch expressions. Table shows the performance of the original MPM solvers and our implementation, demonstrating the efficiency of our method, which significantly reduces the training memory consumption, saving 75% of GPU memory. Our implementation will be released to the public for future research.
Datasets We collect several static 3DGS scenes from the public dataset of PhysGaussian (Xie et al., 2024). We also utilize BlenderNerf (Raafat, 2024) to generate more scenes with different materials and objects. Each 3D scene generated from 100 multi-view renderings.
Baselines We compare method with three state-of- of-the--tttt- art 3D chysical-plausible dynamic syn- thesis methods: (1) PhysDreamer (Zhang et al., 2024b), which utilizes generated video from dif- fusion models to supervise Young's modulus field for 3D objects; (2) Physics3D (Liu et al., 2024) and (3) DreamPhysies (Huang et al., 2024), which adopt Score Distillation Sampling to optimize Young's modulus, Poisson's 's ratio, and damping coefficient. Further implementation details of our method and the baselines are provided in Appendix B. and B.2.
Evaluation Metrics Since there is no ground touth data for dynamic 3D scenes, we propose to evaluate the performance of our method in two aspects. First, measure the alignment between generated videos and text prompts using CLIPSIM (Wu et al., 2021). CLIPSIM is derived from the average cosine similarity between the text embedding and each frame embedding. Second, DiMf and DiffCT.IP are proposed to evaluate the expressiveness and robustness of our method by measuring the difference between the video generated by trained model and randomly ini- tialized model with SSIM and CLIPSIM, respectively. Higher CLIPSIM, Diff and DiffeLIP indicate better performance. We provide calculation methods of evaluation metrics Appendix B.3.
4.2 RESULTS AND COMPARISONS
Single Object in Different Materials Given prompts describing different physical properties of the same object, OMNIPHYSGS can generate videos with the corresponding dynamic behaviors. For example, the prompt tree swinging the wind leads to video where the tree is swaying back and forth, while the prompt a tree collapsing like sand results in a video where the tree collapses into a pile of sand. Table 2 shows the quantitative results of our method and the baselines. Our method achieves better performance in modeling all kinds of materials, surpassing current state-of-the-an methods by about 3% 16% across different cases in text-alignment metrics. Specifically, these baselines achieve close performance to that of our method in modeling pure elasticities, but they struggle to model the behaviors of other materials (e.g., plasticity, viscoelasticity. fluid), which have different properties from those of their fixed constitutive models. This demonstrates the effectiveness and generalizability of our learnable Constitutive Gaussians in capturing the complex behaviors of different materials. Visualizations of the qualitative results are shown in Figure 4 and Appendix C.3.

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative evaluations of generating different materials for a single object. We compare our method with PhysDreamer (Zhang et al., 2024b), Physics3D (Liu et al., 2024), and Dream- Physics (Huang al., 2024) on several cases of synthesizing different materials. Higher DiffssIM: DiffCI. and CLIPSIM indicate better performance. The best results are highlighted in bold.
Swinging Ficus
Collapsing Ficus
Rubber Bear
Sand Bear
Jelly Cube
Water Cuhe
Average
Diffsara
0.0815
0.0014
0.0620
0.0620
0.0031
0.0022
0.0303
PhysDreamer
Difletar
0.9771
1.0021
0.9589
1,0004
0.9590
1.0062
0.9840
CLIPSIMg
22.293
21.891
17,859
14.422
22,573
18.706
19.634
DiffssIM
0.0523
0.0058
0.0640
0.0639
0.0054
0.0045
0.0327
Pbysics3D
DifEcui
0.9574
0.9814
0.9561
1.0155
0.9658
1.0061
0.9804
CLIPSIM
21.845
21.438
17,806
14.701
22,734
18.704
19.538
DiffsarM
0.0816
0.0017
0,0622
0.0627
0.0038
0.0024
0.0307
DreamPhysies
Difierar
0.9800
0.9955
0,9522
0.9876
0.9606
1.0101
0.9810
CLIPSIMs
22.361
21.747
17.734
14.297
22.609
18.778
19.588
DiffssIM
0,0698
0.1497
0.0763
0.1100
0.0124
0.0203
0,0731
Ours
DilICUI
0.9929
1,0481
1,0060
1.1534
1.0006
1,0628
1,0:440
CI JPSMMM
22.653
22.896
18.736
16.698
23.552
19.758
20.716
Table 3: Quantitative evaluations of generating different materials for multi-object scenes.
Scene
Rubber and Sand
Duck and Pile
Rubber hits Metal
Bear into Water
Average
Method
DiffssIM
0.0737
0.0321
0.0056
0.0325
0.0360
PhysDreamer
DiffeLIP
1.0397
1.0399
1.0240
1.0036
1.0268
CLIPSIMg
27.4 17.441
28.564
27.075
24.036
26.779
DiffssIM
0.0566
0.0250
0.0103
0.0274
0.0298
Physics3D
DiffetIP
1.0333
1.0364
1.0222
0.9995
1.0228
CLIPSIMg
27.271
28.467
27.026
23.938
26.675
DiffssrM
0.0819
0.0266
0.0261
0.0289
0.0409
DreamPhysics
Differar
1.0351
1.0418
1.0436
0.9944
1.0287
CLIPSIM
27.319
28.616
27.593
23.816
26.836
Diff
0.1129
0.0714
0.0494
0.0951
0.0822
Ours
DiffetIP 1.0570
1.0488
1.0803
1.0413
1.0569
CLIPSIMg,
27.897
28.809
28.564
24.939
27.552
Multiple Objects in Different Materials Facilitated by the learnable Constitutive Gaussians, OMNIPIYSSG can extract the features of each Gaussian particle within a scene and predict the material properties of each particle. This enables the model to generate dynamics with multiple objects made of different materials. Table shows the quantitative results of our method and the baselines. Our method outperforms baselines in all cases, surpassing them by about 4% in text-alignment metrics, demonstrating the ability to distinguish and model different materials for multiple objects from arbitrary prompts. Figure and Appendix C.3 show the qualitative results of method in generating dynamics with multiple objects in different materials. Notably, all baselines tend to predict homogeneous material properties for all objects in the scene, which leads to failure in modeling the dynamics of multiple objects with different materials. In contrast, method can predict the material properties of each object in the scene according to the text prompt, highlighting the expressiveness of our Constitutive Gaussians.
Generalization of Motion The Material Point Method capable simulating various kinds of deformation behaviors for the same scene under different boundary conditions. This endows our method with the zero-shot generalization ability to generate different motions after training on a single scene. By applying initial impulse, adding colliders, changing the gravity direction, method can synthesize dynamics exhibiting a wide range of material behaviors. We present qualita- tive results of our method's generalization ability in Appendix C.I.
8

---

Published as a conference paper at ICLR 2025
Table 4: Ablation studies on network architectures (w/o physical prior) and training strategies (wo grouping or multi-batch) of Constitutive Gaussians. We report the average SSMM DifECLIP, and CLIPSIM on both the single object cases in Table and the multiple object cases in Table
Scene
Single Object Different Materials
Multiple Objects in Different Materials
Method
DiffssIM
DiffcurP
CLIPSIMg
DiffssIM t Diffcure f CLIPSIMg
w/o Grouping
Out of Memory
Out of Memory
w/o Multi-Batch
0.0638
1.0224
20.241
0.0444
1.0067
26.248
w/o Physical Prior
0.0536
1.0241
20.218
0.0126
1.0025
23.251
Ours
0.0731
1.0440
20.716
0.0822
1.0569
27.552
Ours
"A swinging ficus."
"A collapsing ficus." "
PhysDreamer
Figure 4: Qualitative visualizations of 3D dynamic synthesis for single object in different materi- als. We present the results of our method and PhysDreamer. Other baselines share the same problem (see Appendix C.3). The prompt is provided as the caption of each subfigure.
More qualitative results of our method and the baselines are presented in Appendix C, showing the generalization ability of our method in generating dynamics with different materials and objects. We also provide rendered videos the supplementary material.
4.3 ABLATION STUDY
Network Architecture We conduct ablation studies to demonstrate the necessity of introducing the expert-designed constitutive models to learnable Constitutive Gaussians. Following NCLaw (Ma et al., 2023), we replace the physical-aware decoder of Constitutive Gaussians with simple MLP to directly predict the Cauchy stress tensor and the deformation gradient tensor (both are 3 x 3 matrices). Table 4 presents the quantitative results, showing that a simple MLP has difficulty fit- ting the highly non-linear and complex constitutive models. This underscores the importance of incorporating pre-defined physical constitutive models into the learnable Constitutive Gaussians.
Training Strategy We also conduct ablation studies to investigate the effectiveness of the pro- posed training strategy. Specifically, we compare the performance of our method with (1) training over the whole rendered video without grouping the frames into stages and (2) training without the multi-batch strategy. Quantitative results are shown in Table 4, where training without group- ing leads to Out of Memory due to the long chain of gradient propagation. Additionally, training without the multi-batch strategy results in significant drop in performance due to the optimiza- tion difficulty for the first-order Markov chain. These results demonstrate that our training strategy enhances training efficiency and stability.

---

Published as a conference paper at ICLR 2025
PhysDreamer
Ours
"A lego excavator is digging soil."
"A rubber wolf falling on water."
Coca-Cola
Coca-Cola
Coca-Cola
Coca-Cola
Coca-Cola
Coca-Cola
"A hard duck colliding create a dent in a breakable metal can."
Figure 5: Qualitative visualizations of 3D dynamic synthesis for multiple objects in different mate- rials. More comparison results are provided in Appendix C.3.
5 CONCLUSION
By introducing learnable Constitutive Gaussians, we propose a novel framework, OMNIPHYSGS, for general physics-based 3D dynamic scene synthesis. Facilitated by incorporating domain exppett constitutive models in the poyssiss-gudeds network, our method can automatically and flexibly model various materials for each Gaussian particle within a scene. The supervision of pretrained text-to- video models builds user-friendly interface for generating physically plausible and visually real- istic dynamic scenes aligned with text prompts. We hope our work could be beneficial in practical scenarios, such as immersive video games, hyper-realistic virtual reality experiences, robotic simu- lations, and computer-aides design tools.
Limitations and Future Work this work, we pick expert-designed constitutive models limited to several representative materials, and the per-scene SDS optimization is time-intensive. Future work could consider collecting more kinds of materials to scale up the diversity of the materials and designing more efficient optimization algorithm to build an amortized model for instant inference.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To enhance the reproducibility of our work, we provide detailed experimental settings, including dataset sources in Section 4.1, baseline implementations Appendix B.2, and evaluation metrics in Appendix B.3. Training details, such as the hyperparameters, are outlined in Appendix B.1, and the code snippets for the training process are provided Appendix B.4. A detailed description of the Material Point Method algorithm is available in Appendix A. Our code and data will be released after curation.
ACKNOWLEDGMENTS
The work was supported by National Key R&D Program of China (20222D0160030), an internal grant of Peking University (2024JK28), and grant from Kuaishou (No. DJHL-20040000--155)
REFERENCES
Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024.
Jernej Barbič and Doug L James. Real-time subspace integration for st. venant-kirchhoff deformable models. ACM transactions graphics (TOG), 2005.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: (00i:v:300.332,, 2013.
Andreas Blattmann, Timo Milbich, Michael Dorkenwald, and Bjöm Ommer. ipoke: Poking a still image for controlled stochastic video synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Javier Bonet and Richard D Wood. Nonlinear continuum mechanics for finite element analysis. Cambridge university press, 1997.
Piotr Borycki, Weronika Smolak, Joanna Waczyńska, Marcin Mazur, Stawomir Tadeja, and Prze- mystaw Spurek. Gasp: Gaussian splatting for physic-based based simulations. arXiv preprint arXhv:2409.05819, 2024.
Peter Yichen Chen, Maytee Chantharayukhonthom. Yonghao Yue, Eitan Grinspun, and Ken Kam- rin. Hybrid discrete-continung modeling of shear localization in granular media. Journal of the Mechanics and Physies of Solids, 2021.
Tsai-Shien Chen, Chich Hubert Lin, Hung- Yu Tseng, Tsung- Yi Lin, and Ming-Hsuan Yang. Motion- conditioned diffusion model for controllable video synthesis. arXiv preprint arXiv:2304. 14404, 2023a.
Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840 2023b.
Xi Chen, Zhiheng Liu, Mengting Chen, Yutong Feng. Yu Liu, Yujun Shen, and Hengshuang Zhao. Livephoto: Real image animation with text-guided motion control. arXiv preprint arXiv:2312.02928, 2023c.
Daniel Charles Drucker and William Prager. Soil mechanics and plastic analysis limit design. Quarterly of applied mathematics, 1952.
Yutao Feng, Xiang Feng, Yintong Shang, Ying Jiang, Chang Yu, Zeshun Zong, Tianjia Shao, Hongzhi Wu, Kun Zhou, Chenfanfu Jiang, and Yin Yang. Gaussian splashing: Unified parti- cles for versatile motion synthesis and rendering, 2024.

---

Published as a conference paper at ICLR 2025
Zhoujie Fu, Jiacheng Wei, Wenhao Shen, Chaoyue Song. Xiaofeng Yang, Fayao Liu, Xulei Yang, and Guosheng Lin. Syne4d: Video guided controllable dynamics for physics-basee 4d generation. arXiv preprint arXiv:2405.16849, 2024.
Ming Gao, Xinlei Wang, Kui Wu, Andre Pradhana, Eftychios Sifakis, Cem Yuksel, and Chenfanfu Jiang. Gpu optimization of material point methods. ACM Transactions on Graphics (TOG), 2018.
Oscar Gonzalez and Andrew M Stuart. A first course in continuum mechanics. Cambridge University Press, 2008
Zekun Hao, Xun Huang, and Serge Belongie. Controllable video generation with sparse trajectories. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions on Graphics (TOG), 2018.
Tianyu Huang, Yihan Zeng, Hui Li, Wangmeng Zuo, and Rynson WH Lau. Dreamphysics: Learn- ing physical properties of dynamic 3d gaussians with video diffusion priors. arXiv preprint arXiv:2406.01476, 2024.
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. arXiv preprint arXiv:2104.033/1 2021.
Chenfanfu Jiang, Craig Schroeder, Andrew Selle, Joseph Teran, and Alexey Stomakhin. The affine particle-in-cell method. ACM Transactions Graphics (TOG), 2015.
Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The mate- rial point method for simulating continuum materials, 2016.
Yanqin Jiang, Chaohui Yu, Chenjie Cao, Fan Wang, Weiming Hu, and Jin Gao. Animate3d: Animat- ing any 3d model with multi-view video diffusion. In Advances in neural information processing systems (NeurIPS), 2024a.
Yanqin Jiang, Li Zhang, Jin Gao, Weiming Hu, and Yao Yao. Consistent4d: Consistent 360 dy- namic object generation from monocular video. In International Conference on Learning Repre- sentations (ICLR), 2024b.
Johanna Karras, Aleksander Holynski, Ting-Chun Wang, and Ira Kemelmacchrr-Shizzeman Dream- pose: Fashion image-to-vine video synthesis via stable diffusion. In IEEE/CVF International Confer- ence on Computer Vision (ICCV), 2023.
Bernhard Kerbl, Georgios Kopanas. Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions Graphics (TOG), 2023.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6900 2014.
Gergely Klár, Theodore Gast, Andre Pradhana, Chuyuan Fu, Craig Schroeder, Chenfanfu Jiang, and Joseph Teran. Drucker-prager elastoplasticity for sand animation. ACM Transactions on Graphics (TOG), 2016.
Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for scometry-agnoster system identification. arXiv preprint arXiv:2303.05512 2023.
Zhengqi Li, Richard Tucker. Noah Snavely, and Aleksander Holynski. Generative image dynam- ics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024.
12

---

Published as a conference paper at ICLR 2025
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-3d con- tent creation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Jiajing Lin, Zhenzhong Wang, Yongji Hou, Yuzhou Tang, and Min Jiang. Phy 124: Fast physics- driven 4d content generation from a single image. arXiv preprint urXiv:2409.07179 179, 2024.
Huan Ling, Seung Wook Kim, Antonio Torralba, Sanja Fidler, and Karsten Kreis. Align your gaussians: Text-to-4d with dynamic 3d gaussians and composed diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Fangfu Liu, Hanyang Wang, Shunyu Yao, Shengjun Zhang, Jie Zhou, and Yueqi Duan. Physics3d: Learning physical properties of 3d gaussians via video diffusion. arXiv preprint arXiv:2406.000 04338, 2024.
Pingchuan Ma, Peter Yichen Chen, Bolei Deng. Joshua B Tenenbaum, Tao Du, Chuang Gan, and Wojciech Matusik. Learning neural constitutive laws from motion observations for generalizable pde dynamics. International Conference on Machine Learning (ICML), 2023.
Miles Macklin. Warp: A high-performance python framework for gpu simulation and graph-
ics. sttps://githii htt///tttub//./////////tt www/ /warp, 2022. NVIDIA GPU Technology Conference (GTC).
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. European Conference Computer Vision (ECCV), 2020.
Mises. Mechanik der festen körper im plastisch- deformablen zustand. Nachrichten von der Gesellschaft der Wissenschaften zu Göttingen, Mathhenarisch--yyssklllsshe Klasse, 1913.
Zijie Pan, Zeyu Yang. Xiatian Zhu, and Li Zhang. Fast dynamic 3d object generation from single- view video. arXiv preprint ar Xrxiv:2001.00 088442, 2024.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Lconidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning on point sets in a metric space. Advances in neural information processing systems (NeurIPS), 2017b.
Ri-Zhao Qiu, Ge Yang. Weijia Zeng, and Xiaolong Wang. Language-driven physics-based scene synthesis and editing via feature splatting. In European Conference Computer Vision (ECCV), 2024.
Maxime Raafat. BlenderNeRF, 2024. URL BleR..rNeRF.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning (ICML), 2021.
Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dream- 2023.
gaussian4d: Generative 4d gaussian splatting. arXiv preprint
Jiawei Ren, Kevin Xie, Ashkan Mirzaei, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, et al. L4gm: Large 4d gaussian reconstruction model. In Advances in neural information processing systems (NeurIPS), 2024.
13

---

Published as a conference paper at ICLR 2025
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. Animat- ing arbitrary objects via deep motion transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019a.
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Advances in neural information processing systems (NeurIPS), 2019b.
Aliaksandr Siarohin, Oliver J Woodford, Jian Ren, Menglei Chai, and Sergey Tulyakov. Motion rep- resentations for articulated animation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
Uriel Singer, Shelly Sheynin, Adam Polyak, Oron Ashual, Jurii Makarov, Filippos Kokkinos, Naman Goyal, Andrea Vedaldi, Devi Parikh, Justin Johnson, al. Text-to-4d dynamic scene generation. In International Conference on Machine Learning (ICML), 2023.
Alexey Stomakhin, Russell Howes, Craig A Schroeder, and Joseph M Teran. Energetically consis- tent invertible elasticity. In Symposium on Computer Animation, 2012.
Alexey Stomakhin, Craig Schroeder, Lawrence Chai, Joseph Teran, and Andrew Selle. A material point method for snow simulation. ACM Transactions on Graphics (TOG), 2013.
Alexey Stomakhin, Craig Schroeder, Chenfanfu Jiang, Lawrence Chai, Joseph Teran, and Andrew Selle. Augmented mpm for phase-change and varied materials. ACM Transactions on Graphics (TOG), 2014.
Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309, 16653, 2023.
Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Mod- elscope texi-to-video technical report. arXiv preprint arXiv:2308.06:3 06571, 2023.
Yikai Wang, Xinzhou Wang. Zilong Chen, Zhengyi Wang, Fuchun Sun, and Jun Zhu. Vidu4d: Single generated video to high-fidelity 4d reconstruction with dynamic gaussian surfels. arXiv preprint arXiv:2405.1663 2024a.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Pro- lifiedreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Neural Information Processing Systems (NeurIPS), 2024b.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing. 2004.
Chenfei Wu, Lun Huang, Qianxi Zhang. Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Generating open-domain videos from natural descriptions. arXiv preprint arXiv:2104. 14806, 2021.
Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-interrated 3d gaussians for generative dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Ruihan Yang, Prakhar Srivastava, and Stephan Mandt. Diffusion probabilistic modeling for video
generation. Entropy, 2023.
Shengming Yin, Chenfei Wu, Jian Liang, Jie Shi, Houqiang Li, Gong Ming, and Nan Duan. Drag- nuwa: Fine-grained control in video generation by integrating text, image, and trajectory. arXiv preprint arXiv:2308.00800 2023a.
Yuyang Yin, Dejia Xu, Zhangyang Wang, Yao Zhao, and Yunchao Wei. 4dgen: Grounded 4d content generation with spatial-temporal consistency. arXiv preprint arXiv:2312.17225, 2023b.
Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition (CVPR), 2022.
14

---

Published as a conference paper at ICLR 2025
Yonghao Yue, Breannan Smith, Peter Yichen Chen, Maytee Chantharayukhonnomm, Ken Kamrin, and Eitan Grinspun. Hybrid grains: Adaptive coupling of discrete and continuum simulations of granular media. ACM Transactions Graphics (TOG), 2018
Yifei Zeng, Yanqin Jiang, Siyu Zhu, Yuanxun Lu, Youtian Lin, Hao Zhu, Weiming Hu, Xun Cao, and Yao Yao. Stag4d: Spatial-temporal anchored generative 4d gaussians. In European Conference on Computer Vision (ECCV), 2024.
Haiyu Zhang, Xinyuan Chen, Yaohui Wang, Xihui Liu, Yunhong Wang, and Yu Qiao. 4diffusion: Multi-view video diffusion model for 4d generation. arXiv preprint arXiv:2405.20674, 2024a.
Shiwei Zhang, Jiayu Wang, Yingya Zhang, Kang Zhao, Hangjie Yuan, Zhiwu Qin, Xiang Wang, Deli Zhao, and Jingren Zhou. 12vgen-xl: High-quality image-to-video synthesis via cascaded diffusion models. arXiv preprint 2..1:2311.04444 2023.
Tianyuan Zhang, Hong-Xing Yu, Rundi Wu, Brandon Feng, Changxi Zheng. Noah Snavely, Jiajun Wu, and William T Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. European Conference on Computer Vision (ECCV), 2024b.
Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Animate 124: Animating image to 4d dynamic scene. arXiv preprint drXiv:2311.1600) 2023.
Yufeng Zheng, Xueting Li, Koki Nagano, Sifei Liu, Otmar Hilliges, and Shalini De Mello. A unified approach for text-and image-guided 4d scene generation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR). 2024.
Licheng Zhong, Hong-Xinn Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass mass 3d gaussians. European Conference on Computer Vision (ECCV), 2024.
Zeshun Zong, Xuan Li, Minchen Li, Maurizio M Chiaramonte, Wojciech Matusik, Eitan Grinspun, Kevin Carlberg, Chenfanfu Jiang, and Peter Yichen Chen. Neural stress fields for reduced-order clastoplasticity and fracture. In SIGGRAPH Asia 2023 Conference Papers, 2023.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A MATERIAL POINT METHOD
A.1 CONTINUUM MECHANICS
Continuum mechanics (Gonzalez & Stuart, 2008; Jiang al., 2016) models the behavior of materials
as a continuous medium by a deformation map x = (XX,t) where X denotes the undeformed configuration and denotes the deformed configuration. The deformation gradient F V V t) describes the local deformation of the material. The key governing equations for the deformation of material are the conservation of mass and momentum:
Dp - Dt
Dv
P Dr - fox
(9)
where p(x, t) is the density, v(x,t t) is the velocity field, ox, t) is the Cauchy stress tensor, and (xx is the external force. The Cauchy stress tensor is defined
ON (F (EE)EE E R
oxx,) t) det(F) OF
(10)
where R3x3 + R3x3 is the hyperelastic energy density function, F E 3x33 is the elastic part of the deformation gradient, since the total deformation gradient can be decomposed as F FFF" where F" E TR3x3 is the plastic part of the deformation gradient. In practice, the decomposition equation of the deformation gradient R3x3 is a return function that applies the plasticity constraints to F
be reparameterized as F (FFF) where tà R333
The hyperelastic energy density function (-) and the plasticity return function (+)) are the con- stitutive models that describe the material behavior. They map the elastic part of the deformation gradient F to the Cauchy tively. These models are typically designed by experts in the field to describe the material properties, determining how the material deforms under certain circumstances. To match the material behav- ior, these functions are highly nonlinear and can be of great complexity. Appendix A.4 provides some examples of these models, showing that constitutive models can be applied a wide range of materials.
tensor and the corrected deformation gradient
RR333 respec-
A.2 MPM ALGORITHM
MPM (Jiang et al., 2016; Stomakhin et al., 2013) is a hybrid Lagrangian method that dis- cretizes the continuum into set of particles and utilizes background grid to solve the conservation equations. As shown in the pseudo-code velocity, impulse, external force, etc.), MPM performs a particle-to-grid (P2G) step transfer the particle information (i.e. the mass and momentum) to the grid and a grid-to-particle (G2P) step to transfer the grid information back to the particles in the simulation loop.
Algorithm 1, given boundary conditions b (e.g. initial
Algorithm A Moving Least Squares Material Point Method (MLS-MPM) Step
Input: s [xt vfn Fln Ci IPEp
Output: gia+1 = {xp
fat1 bnts
F Ch Jpep
Vp
1: for all p E do
2: stress, fu+1 + F2 pp yap)
3: x vfn f- App Agply b))
4: Xp .vp fut1 Ff mial Cp Particlle2rrdaaaatttidee vt. Ch stress(
5: Fiut + ReturnMap p, trial 7p)
6:
end for
MPM traces particles' states stn = (xf v Frr heEP in every time step tn. where x!- e R3 is the position of the particle, va- E R3 is the velocity, F e R3x3 is the deformation gradient, and C R3x3 is the affine momentum. The hyperelastic energy density function V, (.) and the
16

---

Published as a conference paper at ICLR 2025
plasticity return function dp (-) are utilized as the constitutive models to calculate the stress tensor and to correct the trial deformation gradient, respectively. Physical parameters 7/p EK such as Young's modulus and Poisson's ratio, are included the constitutive models.
We denote my E R and V E I as the mass of particle p and the volume of the particle, respectively, which are invariant during the dynamics. For each grid node G, denote m R and v E 3 as the mass and velocity of the node at time i, respectively. The position of each grid node is denoted as x; E IR, which is fixed during the simulation. Mass and momentum are transferred between particles and grid nodes, according the interpolation functions (x). We denote the interpolation result of particle grid node i as whp R.
In practice we prefer to express the strain-stress relationship using deformation gradient F₃ and first Piola-Kirchoff stress tensor Pp E RR3x3 of particle p as:
OV p P
(11)
OF, p
because they are more naturally expressed in the material space (Jiang et al., 2016). For simplic- ity, we omit the partial derivative symbol in the following equations. The MPM algorithm is summarized follows (items I~5 illustrate single MPM time step):
0. Initialization Initialize the particle state from the static Gaussians as:
p
0, F I, C = 0, V
(12)
and set the grid mass and velocity to zero as:
m to 0,
O, ViEg
(13)
1. Stress Update Calculate the stress tensor P as:
P Vp (F Ep
(14)
2. Particle to Grid (P2G) Transfer particle information to grids using APIC scheme (Jiang al., 2015) as:
m Σ
wh mps V
(15)
pEP
m
W mp (v'r + C (x: x²x))
(16)
PEP
3. Grid Update Apply internal and external forces to the grid nodes as:
At
vín
Σ
PP Atfeu ViEg ip
(17)
m
PEP
4. Grid to Particle (G2P) Transfer grid information to particles as:
vhn+1 p
Σ
tntz
xfn+1 = x + Atv
(18)
iEg
Cha+1
12
up v
In lntt (x, - -xtx x²= VpEp
(19)
Ar (b+1)
tEg
Vytn41
Σ fm V
Fma1 trial
= (I A Fr,
(20)
iEG
where is the B-spline degree and Ar the grid spacing. 5. Plasticity Correction Apply the plasticity return function to correct the trial deformation gradient as:
Fla+1 dhp
(Ffret p. trial
Vp eper
(21)
Boundary conditions can be applied to particles before P2G and to grid nodes before G2P on the positions or velocities. Readers can refer to (Jiang et al., 2016; Stomakhin et al., 2013) for more details about the MPM algorithm.
17

---

Published as a conference paper at ICLR 2025
A.3 UPDATES FOR RENDERING PROCESS
Due to the point-wise nature of the MPM, it is well-suited for simulating 3D Gaussian particles given the evolutions the Gaussian kemels' positions and deformation gradients. Following Phys- Gaussian (Xie et al., 2024), we update Gaussian kernels at each time step to ensure that the kernels follow the Gaussian distribution after deformation as:
An prrender
+ x
En p.render
F... Ep (F(a)²
(22)
The rendering view direction dp from the viewpoint to each Gaussian kernel is also modified by the deformation gradient as:
d dprennder
-R/x)) dp.
(23)
where Rp is derived from the polar decomposition of the deformation gradient Fş R. Other necessary attributes the Gaussian kernels, such as spherical harmonic coefficients and opacity, are kept unchanged during the simulation process. We render the Gaussian kernels at each time step using the official implementation of Gaussian Splatting renderer (Kerbl et al., 2023)'.
A.4 CONSTITUTIVE MODELS
We collect expert-designed constitutive models from previous works (Ma et al., 2023; Zong et al., 2023; Xie et al., 2024), which describe several representative materials including materials with elasticity (e.g., rubber, branches, and cloth), plasticity (e.g., snow, metal, and clay), viscoelasticity (e.g., honey and mud), and fluidity (e.g., water, oil, and lava). Table 5 lists some necessary physical parameters used in the constitutive models.
Table 5: Physical Parameters for Constitutive Models.
Notation
Description
Young's 's Modulus
Poisson's ' Ratio
Shear Modulus
Lamé Parameter
Value
2(1+v)
(1++1(1 2n)
A.4.1 HYPERELASTIC ENERGY DENSITY FUNCTION
We use the first Hola-KKrchoofi stress tensor P 8 listing the map from deformation gradient to P.
oF to express the stress-strain relationship by
Fixed Corotated Elasticity ticity
We follow (Stomakhin et al., 2012) to define the fixed corotated elas-
P (F) 2p (F R) + ()-1)
(24)
where R from the polar decomposition F = RS and J is the determinant of F. Fixed corotated elasticity is suitable for simulating rubber-like materials.
Neo-Hookean Elasticity
We follow (Bonet & Wood, 1997) to define the Neo-Hookeen elasticity
as:
( = μ (F- + Alog (J)
(25)
Neo-Hookean elasticity is suitable for simulating clasticities like springs,
StVK Elasticity We follow (Klár et al., 2016; Barbič & James, 2005) to define the StVK elasticity
P(F) = U
+ Atr (In E) Σ) VT
(26)
where U. Σ, and V are derived from the singular value decomposition of F UEVT. StVK clasticity is suitable for simulating plasticity like sand and metal.
hhttps://gi thub. wwt..coo//rraphecc.... in anrriaaaggsiiiilsslsaatatii
18

---

Published as a conference paper at ICLR 2025
A.4.2 PLASTICITY RETURN FUNCTION
We use the plasticity return function (-) correct the trial deformation gradient F Fiaal the final deformation gradient F. For simplicity, we omit the subscript trial in the following equations.
Identity Plasticity Most pure elastic materials adopt the identity plasticity as:
FF =.
(27)
Drucker-Prager Plasticity We follow (Drucker & Prager, 1952; Klár et al., 2016; Yue et al., 2018; Chen et al., 2021) to define the Mucker-Prager plasticity as:
I.
sum (e) 0,
Uz 1 UZ VT,
2
Σ,
8y S 0 and sum (c) S 0,
(28)
exp
by (
||ll
otherwise,
where 6 = log Σ). Drucker-Prager plasticity is suitable for simulating plasticity like snow and sand.
von Mises Plasticity
We follow (Mises, 1913; Hu et al., 2018; Huang et al., 2021) to define the von Mises plasticity
Σ,
by S 0, otherwise.
(F)
UZ Σ VT,
(
(29)
exp
6
87
Von Mises plasticity is suitable for simulating plasticity like metal and clay.
Fluid Plasticity We follow (Stomakhin et al., 2014; Gao et al., 2018) define the fluid plasticity
as:
FF)
J
(30)
Fluid plasticity is suitable for simulating fluidity like water and lava.
B
MORE DETAILS ON IMPLEMENTATION
B.1 HYPERPARAMETERS AND TRAINING SETTINGS
Simulating Details We set the simulating timestep At to x 1-00=fo for all experiments. In training, the simulated states are sampled every 10 steps and rendered into images with a fixed camera. The video length is set 150 frames with frame rate of 30 fps, resulting in total of 5 seconds. We load all Gaussian particles The simulating world is equipped with gravity of 9.8 m/s². In the training phase, do not apply any other initial velocity or external force to the particles for all quantitative experiments except the swinging ficus case.
1 x x 1 bounding box, which is divided into 25 x 25 x 25 grids.
Training Details All experiments are conducted on single NVIDIA A6000 (48GB) GPU. We train our model using the Adam optimizer (Kingma, 2014) with a learning rate of x 10-5 for 5 iterations of each scene. In each iteration, the whole simulating process is divided into 10 stages sequentially where 15 frames are rendered for each stage. Video clips of each stage are optimized by pretrained text-to-video diffusion model (Wang al., 2023) for 30 iterations before we proceed to the next stage. For the learnable Constitutive Gaussians, we first adopt FPS to sample 8192 centers from the original Gaussian particles and then use convolutional layer encode the features within a neighborhood and a simple MLP to predict the category of the material, Each neighborhood includes 32 particles.
B.2 BASELINE DETAILS
We reimplement the baseline methods according to their GitHub repositories.
19

---

Published as a conference paper at ICLR 2025
PhysDreamer (Zhang et al., 2024b)², which utilizes diffusion generated reference videos to optimize Young's s modulus in constitutive combinations Fixed Corotated Elasticity and Identity Plasticity. We use our text prompts to generate the reference videos training data for PhysDreamer.
Physics3D (Liu et al., 2024)', which adopts StVK Elasticity and dissipation term con- stitutive models. SDS is used optimize Young's modulus, Poisson's 's ratio, and damping coefficient in Physics3I..
DreamPhysics (Huang et al., 2024)*, which leverages SDS to optimize Young's modulus and Poisson's s ratio Fixed Corotated Elasticity and Identity Plasticity.
We use the same experimental settings across all methods. Since our multi-batch training strategy actually trains the model multiple times in a single iteration, we train the baselines for number of tterati ons x number of internal iterations iterations to ensure a fair comparison.
B.3 EVALUATION DETAILS
We use the Vit-L/14 version of CLIP (Radford et al., 2021) to calculate the CLIPSIM (Wu et al., 2021) 1) score as:
1 N
Σ CLIP(Ia,))
CLIPSIM
(31)
where In is the n-th frame the generated video and y the text prompt. Higher CLIPSIM indicates better alignment between the video and the text. Diffssrar and DiffeLIP are derived from:
1 Σ SSIM in), Diffcus CLIPSIM'
CLIPSIM
DiffsSIM N
(32)
where 1, is the n-th frame of the video generated by a randomly initialized model, SSTM is the structural similarity index (Wang et al., 2004), and CLIPSIM' is the CLIPSIM of the random model. These two metrics, sSIM and DiffcLIP. measure the improvement of the model during training and eliminate the influence of initialization. Higher Diff and DifferLP indicate better learning ability and robustness.
B.4 TRAINING STRATEGY
Code snippet illustrates our training strategy of first dividing the whole simulating process into multiple stages, grouping the frames each stage into mini-batches, and then optimizing the video clips of each mini-batch multiple times. Note that each frame is sampled every fixed number of steps.
C ADDITIONAL RESULTS
C.I VISUALIZATIONS OF MOTION GENERALIZATION
Given different boundary conditions, our trained model can be generalized to synthesize diverse dynamic behaviors. We present additional visualizations motion generalization in Figure taking the rubber wolf as example.
C.2 VISUALIZATIONS OF OMNIPHYSSS
We present additional visualizations of 3D dynamic synthesis for single object in different materi- als in Figure and for multiple objects Figure 8.
cs30
Ahttps: :// http:///gg.gt..coooo//ggggaa////
20

---

Published as a conference paper at ICLR 2025
Listing I: Training Strategy
Train an iteration X, V, F, с initialize()
for stage in range (num_stages):
# Grouping
# Save the end state of the last stage for multi-bate x_ckpt, v_ckpt, F_ckpt, C_ckpt = V, F, C for internal
in range (num iteraiaa
# Multi-Batch
# Start from the end state of the last stage v, F, = x_ckpt, v_ckpt, F_ckpt, C_ckpt stage_frames = []
for frame in range (num frames)
for step in range moum_steps_pee_t per__srmee____
# MPM steps
x, v, F, C mpm_step (x, v, F, C)
rendering render (x, v, C)
stage_frames. append (rendering)
loss score_distillatinn samplimmmlgggg (stage_frames) loss bass.backward()
optimizer.step ()
C.3 VISUAL COMPARISONS WITH BASELINES
We present visual comparisons with baselines 3D syntheiss for a single object in different materials in Figures 10, and 11, and for multiple objects in Figures 12 and 13. Remarkably, all baselines tend to predict the same material regardless of the different input prompts, resulting in dynamic synthesis outputs that appear quite similar. This highlights that merely tuning physical parameters is insufficient for synthesizing diverse dynamic behaviors, constrained by the limited expressiveness of the fixed physics model. In contrast, our method effectively synthesizes wide range of dynamic behaviors by learning the material properties from pretrained video diffusion, thus achieving better generalizability and expressiveness.
"fall"
"flip"
"spin"
"tear"
Figure 6: Visualization results of motion generalization.
21

---

Published as a conference paper at ICLR 2025
CocaCCol
Coca-Cola
"A crushed metal can."
"A rubber duck falling.
"A rubber wolf bouncing."
"A sand wolf collapsing."
"A jelly bouncing."
"Water flowing."
"A collapsing mud pile."
"A soccer ball hits the ground." "
Figure 7: Qualitative visualizations of 3D dynamic synthesis for a single object with our method.
22

---

Published as a conference paper at ICLR 2025
"A rubber duck bouncing on the ground and a sand pile collapsing into ruins."
"Two rubber ducks colliding."
"A golden rubber wolf bouncing and grey sand wolf collapsing."
"A rubber wolf falling on water."
Coca-Cola
CocaCCoaa
Coca-Col
"A hard duck colliding to create a dent in breakable metal can." "
"A duck falling on a soft mud pile." "
"A lego excavator is digging soil."
"Two kinds of water flowing." "
Figure& 8: Qualitative visualizations of 3D dynamic synthesis for multiple objects with our method.
23

---

Published as a conference paper at ICLR 2025
Ours
"A swinging ficus." "
PhysDreamer
DreamPhysics Physics3D
Ours
"A collapsing ficus." "
PhysDreamer
DreamPhysics Physics3D
Figure 9: Qualitative visualizations of 3D dynamic synthesis for a single object in different materi-
als. We present the results of our method and the baselines.
24

---

Published as a conference paper at ICLR 2025
Ours
"A rubber wolf bouncing." "
PhysDreamer
DreamPhysics Physics3D
Ours
"A sand wolf collapsing."
PhysDreamer
DreamPhysics P
Figure 10: Qualitative visualizations of 3D dynamic synthesis for a single object in different mate-
rials. We present the results of our method and the baselines.
25

---

Published as a conference paper at ICLR 2025
Ours
jelly bouncing."
PhysDreamer
DreamPhysics Physics3D
Ours
"Water flowing."
PhysDreamer
DreamPhysics Physics3D
Figure 11: Qualitative visualizations of 3D dynamic synthesis for a single object in different mate- rials. We present the results of method and the baselines.
26

---

Published as a conference paper at ICLR 2025
Ours
"A golden rubber wolf bouncing and grey sand wolf collapsing."
PhysDreamer
DreamPhysics Physics3D
Ours
"A rubber wolf falling on water."
PhysDreamer
DreamPhysics Physics3D
Figure 12: Qualitative visualizations of 3D dynamic synthesis for multiple objects in different ma- terials. We present the results of our method and the baselines.
27

---

Published as a conference paper at ICLR 2025
Ours
"A duck falling soft mud pile."
PhysDreamer
DreamPhysies Physics3D
Ours
CocaCCla
Coca-Cola
Coco-Cola
"A hard duck colliding to create a dent in a breakable metal can."
DreamPhysics P PhysDreamer
Coca-Cll
Coca-Cola
Coca-Cola
Coca-Col
Coca-Cola
CocaCola
CocaCola
Coca-Cola
Coca-Cola
Coca-Cola
Coca-oola
Coca-Cola
PocaCCoa
Coca-Cola
Coca-Cola
Figure 13: Qualitative visualizations of 3D dynamic synthesis for multiple objects in different ma-
terials. We present the results of our method and the baselines.
28


Reviews:
[{"review_id": "Su8IZZA1MR", "reviewer": "Reviewer_7Xoc", "summary": "The paper presents a framework for text-driven physical parameter learning for 3DGS-reconstructed objects. Compared to previous methods, the main novelty lies in making discrete constitutive model types learnable based on given text prompts. For each particle, the physics-aware decoder predicts a material type from 12 predefined material classes. The simulated and rendered results are fed into a pretrained text-to-video diffusion model to evaluate SDS for guiding both material class and material parameter learning. Efficiency enhancements are also proposed, including optimizing the long sequence in chunks.", "questions": "- How is the physical-aware decoder trained? The material classes are discrete and the sampling of $j_i, k_i$ in Eq.6 is non-differentiable. How does the model learn to change material class? This is the key novelty of the paper, a detailed discussion is needed.\n\n- The following references are highly related, which also design learnable constitutive models:\n    - Nagasawa, Kentaro, et al. \"Mixing sauces: a viscosity blending model for shear thinning fluids.\" ACM Trans. Graph. 38.4 (2019): 95-1.\n    - Su, Haozhe, et al. \"A generalized constitutive model for versatile mpm simulation and inverse learning with differentiable physics.\" Proceedings of the ACM on Computer Graphics and Interactive Techniques 6.3 (2023): 1-20.\n   \n\n- Warp can create arrays from Torch tensor without copying. I am wondering why it consumes much more memory than implementing MPM in Pytorch.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The overall presentation of the framework is clear.\n- Comparisons and ablations are extensive.", "weaknesses": "- The controls of the scene—such as initial velocity and external forces—are not learnable parameters. For this reason, the paper is best summarized as a framework for learning physical parameters, where interactions between objects remain fixed and are not directly controllable. Consequently, text descriptions must be crafted based on these known, predefined interactions.\n- In reality, objects cannot be both elastic and sandy. The physical plausibility is questionable. With such heuristic blending of materials within a single object, the dynamics may conform to unrealistic motions encoded in the video model that is impossible in reality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jFvn5hAyrt", "reviewer": "Reviewer_h6D1", "summary": "This paper proposes a physics-based dynamics generation method that includes different physical material properties. The authors model the object as constitutive 3D Gaussians that exhibit multiple physical material properties by an ensemble of physical domain-expert sub-models. The dynamics can be described and input as a user prompt, the learnable constitutive models would be optimized by a pre-trained video diffusion model with SDS loss. The authors designed a 3D feature extractor and physical-aware decoder to model ordinary Gaussians to constitutive Gaussians. Due to the limited generation length of existing video diffusion models, the authors designed two training strategies, grouping and multiple mini-batch training to allow the training on MPM simulation steps.", "questions": "The experiments primarily compare the same object using two distinctly different materials, such as rubber versus sand or jelly versus fluids. However, in practical applications, it is unusual to utilize the same object with radically different materials. Moreover, modifying the material type in PhysGaussian [1] is relatively straightforward, requiring changes to only three fields in the configuration file: the material itself, Young’s modulus, and Poisson’s ratio. This simplicity contrasts sharply with the complexity of optimizing a model using video diffusion models for similar tasks. From my perspective, it is beneficial to explore the optimization of physical parameters for a single type of material. For instance, metals like gold exhibit high plasticity, whereas aluminum alloys display low plasticity. Similarly, wood varieties may vary significantly in flexibility. As demonstrated in Figure 6 of PhysGaussian [1], jelly can show varying degrees of stiffness and volume preservation by adjusting Young's modulus and Poisson's ratios. These parameters are challenging to adjust manually and merit further exploration for optimization via video diffusion models.\n\nAs stated in line 234 by the authors, “Unlike previous works (Zhang et al., 2024b; Liu et al., 2024; Huang et al., 2024) that maintained fixed constitutive models, Constitutive Gaussians allow the model to capture diverse material behaviors, encompassing both elastic and plastic deformations, thus offering a more dynamic and comprehensive representation of material properties.” Additionally, Equation 4 highlights that the hyperelastic energy density function, the plasticity return function, and the physical parameters are all learnable parameters within the constitutive models. Therefore, as the authors assert, this approach should facilitate the optimization of material properties within the same material category.\n\nI am eager to see experimental results that demonstrate this capability of the model. For instance, comparing a \"hard rubber bear\" with a \"stretchy rubber bear,\" or a \"hard metal can\" with a \"soft metal can\" would illustrate the model’s effectiveness. Presenting such results would prompt me to raise my score.\n\n[1] Xie, T., Zong, Z., Qiu, Y., Li, X., Feng, Y., Yang, Y., & Jiang, C. (2024). Physgaussian: Physics-integrated 3d Gaussians for generative dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 4389-4398).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This approach uses 3D Gaussians as constitutive models, enabling the representation of diverse physical material properties. It facilitates the optimization of these properties via a learnable MPM simulation, which integrates video SDS loss.\n\n2. Domain-expert constitutive models are incorporated to steer the learning process across various materials, functioning similarly to a Mixture of Experts methodology.\n\n3.  Two training strategies are introduced, grouping and multi-mini-batch training, to address the challenges posed by numerous MPM simulation steps and the limited number of frames produced by video diffusion models,", "weaknesses": "1. The authors limit their comparisons to the same object modeled with vastly different materials. This approach is less persuasive because altering materials in PhysGaussian [1] is straightforward, whereas optimizing appropriate physical parameters is challenging. More convincing evidence would come from showing varied parameter results within the same material. I will elaborate on this point in the questions section.\n\n2. The collision experiments appear to lack physical contact, consistently showing a gap between the colliding objects, whereas collisions modeled in PhysGaussian [1] are depicted as more substantial. The overall visual quality of the experiments still falls short of what is achieved with PhysGaussian.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZoYDGKykC9", "reviewer": "Reviewer_ktNn", "summary": "This paper focuses on generating dynamic 3D objects with physics-based simulation. The paper aims to relax the assumption of fixed constitutive models from previous works such as PhysGaussian and PhysDreamer. In particular, this paper introduces the OmniPhysGS framework with the learnable Constitutive Gaussians at the core. The main technical idea is to decompose an object into many small particle groups and use SDS optimization to assign a particular material to each group. The material assignment is a classification process that selects one of twelve predefined constitutive models with learnable physical parameters. Experiments are performed on a set of simple synthetic objects, showing that the proposed approach can generate different types of material dynamics.", "questions": "Please see weaknesses above. I'm open to change my mind if there are evidences to rebut my points, though.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The problem is relevant and important. Current methods (e.g., PhysDreamer, Physics3D) are very much limited to elastic objects due to fixed constutive model. The proposed approach adds the flexibility of choosing from different materials.\n- The experiment results of generating different types of material dynamics for a single input are interesting.\n- Implementing a memory-efficient MPM solver is useful.", "weaknesses": "- There is no real experiments such as the ones in PhysDreamer and Physics3D.\n- Some experiment results look weird, such as in 0:49 of the video, the can hit by the toy duck. It looks like the can has a very weird material (it does not look like metal at all), such that it keeps shrinking after being lightly hit by a toy.\n- The metrics (e.g. CLIP similarity score) applying to videos do not seem to be very convincing to me. I'm not sure if CLIP score can be used to measure motion quality. I think the motion realism would be better judged by human preference.\n- I'm a bit concerned about the technical design of dividing an object into small particle groups and allowing them to have different predefined constitutive models. That may not align with real-world physics. For example, it may give arbitrary material prediction in the interior of an object. I think the can example is one illustration of generating unreasonable results. Would this give reasonable results all the time?\n- It is not clear to me why a neural network is needed at all. It seems there is no training, but per-scene test-time optimization. So there is no generalization issue. Then one may simply optimize a 12-way one-hot softmax vector for the material classification. What's the advantage of having to train a scene-specific neural network?\n- Some notations are confusing, e.g., in L207, the physical parameters are denoted as a single real scalar value, yet the supplementary material says it includes Young’s modulus and Poisson’s ratio.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0poHAw41V0", "reviewer": "Reviewer_S54K", "summary": "In this work, the authors propose omniphysgs, a novel method for creating physics-based 3D dynamic scenes with diverse objects by modeling each asset as collections of 3D Gaussians and using multiple material sub-models. This method allows for complex material compositions, enhancing realism and flexibility in physical interactions. Experimental results show omniphysgs outperforms existing methods in visual quality and text alignment.", "questions": "I am curious about how this method handles interactions between a single object and an entire scene, as opposed to interactions between two objects. Additionally, it would be better to understand the method's performance and behavior in scenarios involving more than two objects.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Performance: The proposed method omniphysgs achieves state-of-the-art results. The experiments well validate the effectiveness of the proposed methods.\n\n2. Clarity: The paper is well-written and clearly structured, making the methodology and results easy to understand and follow.\n\n3. Technical Novelty: The main contributions of this paper are twofold: 1) They propose a novel framework, which models each 3D asset as a collection of 3D Gaussians and represents physical materials using an ensemble of 12 domain-specific sub-models. This design significantly enhances the flexibility and realism of the synthesized dynamic scenes. 2) They define a scene by user-specified prompts and use a pretrained video diffusion model to supervise the estimation of material weighting factors, enabling the synthesis of more general and physically plausible interactions across a diverse range of materials.", "weaknesses": "Figure 1 is unclear regarding the material properties of the objects. It is confusing when the mountain is depicted as non-elastic while the duck toy is elastic, but the mountain collapses after the duck toy falls, which seems inconsistent with the assigned material properties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F3pqeXYovU", "reviewer": "Reviewer_kAsd", "summary": "The OmniPhysGS framework can simulate general physics-based 3D dynamics. It works with many types of materials like elastic, plastic, and fluid. Authors use Constitutive Gaussians and video diffusion models to make more realistic animations. It uses MPM for accurate physics simulation. The paper shows better results than PhysDreamer, Physics3D, and DreamPhysics.", "questions": "A discussion comparing the method to 'A generalized constitutive model for versatile MPM simulation and inverse learning with differentiable physics.' [2023] would be good. This would provide more context on how OmniPhysGS performs in comparison to recent advancements in MPM simulation and the capabilities of inverse learning. Authors should say more about the limits of the constitutive models. It is not clear what they assume about materials in each Gaussian particle. Would the proposed method be able to do material mixture as in this paper?\n\nIn experiments, it would be good to say how OmniPhysGS parameters are chosen. This is not clear for multi-object scenes where material differences are important.\n\nMore explanation needed about permanent deformation. How does OmniPhysGS handle it for viscoelastic or plastic materials? Would there be failure during training? For example, a bad parameter causes a 3D model to break down to pieces. The optmization may get stuck, right?\n\nIn Figure 3, could add some notes. Use a scene with multiple model categories, and show how the classification decoder assigns constitutive models to different areas of the scene. \n\nThe wolf on water scene, how is the shape of the water container specified in the pipeline?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is the first method to use learnable constitutive models in Gaussian Splatting. It makes material simulation more flexible.\n\nIt supports many different materials. Other methods usually focus on one type.\n\nThe method uses text prompts and video diffusion models. It is very user-friendly and simple to use.", "weaknesses": "The quality depends on pre-trained diffusion models. This could make it difficult to simulate new materials or very specific materials.\n\nThe method might lack robustness when dealing with more complex or unusual physical scenarios, especially when the guidance models do not adequately capture the specific material properties.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)