DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Rendering
Authors: Jiaxu Wang
Jingkai SUN
Ziyi Zhang
Junhao He
Qiang Zhang
Mingyuan Sun
Renjing Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Particle Dynamics; Learning 3D from 2D; Physics-augmented Graph Network
Primary area: machine_vision

Abstract: Learning-based simulators show great potential for simulating particle dynamics when 3D groundtruth is available, but per-particle correspondences are not always accessible. The development of neural rendering presents a new solution to this field to learn 3D dynamics from 2D images by inverse rendering. 
However, existing approaches still suffer from ill-posed natures resulting from the 2D to 3D uncertainty, for example, specific 2D images can correspond with various 3D particle distributions. To mitigate such uncertainty, we consider a conventional, mechanically interpretable framework as the physical priors and extend it to a learning-based version. In brief, we incorporate the learnable graph kernels into the classic Discrete Element Analysis (DEA) framework to implement a novel mechanics-informed network architecture. In this case, the graph networks are only used for approximating some specific mechanical operators in the DEA framework rather than the whole dynamics mapping. By integrating the strong physics priors, our methods can effectively learn the dynamics of various materials from the partial 2D observations in a unified manner. Experiments show that our approach outperforms other learned simulators by a large margin in this context and is robust to different renderers, fewer training samples, and fewer camera views.

DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Rendering
Jiaxu Wang' Jingkai Sun Junhao He' Ziyi Zhang'
Qiang Zhang! Mingyuan Sun Renjing Xu!
Hong Kong University of Science and Technology, Guangzhou, China Beijing Innovation Center of Humanoid Robotics Co. Ltd, Beijing, China Northheatern University, Shenyang. China
(jaang457, qzhang749, [Sun444)@connec www.hhusttge. edu. cn
mingyuansun@stumail neu. www.nuu.ucce.eeeeomm cn {junhaohe, ziyizhang, renjingxu www.t-gz. edu. cn
Abstract
Learning-basee simulators show great potential for simulating particle dynam- ics when 3D groundtruth is available, but per-particle correspondences are not always accessible. The development of neural rendering presents new solution this field to learn 3D dynamics from 2D images by inverse rendering. However, existing approaches still suffer from ill-posed natures resulting from the 2D 3D uncertainty, for example, specific 2D images can correspond with various 3D particle distributions. To mitigate such uncertainty, we consider a conventional, mechanically interpretable framework as the physical priors and extend it to a learning-basd based version. In brief, we incorporate the learnable graph kernels into the classic Discrete Element Analysis (DEA) framework to implement novel reehhanics-integrated learning system. In this case, the graph network kernels are only used for approximating some specific mechanical operators in the DEA framework rather than the whole dynamics mapping. By integrating the strong physics priors, methods effectively learn the dynamics of various materials from the partial 2D observations in a unified manner. Experiments show that our approach outperforms other learned simulators by a large margin in this context and is robust to different renderers, fewer training samples, and fewer camera views.
Introduction
Simulating complex physical dynamics and interactions of different materials is crucial in areas including graphics, robotics, and mechanical analysis. While conventional numerical tools offer plausible predictions, they are computationally expensive and need extra inputs like material specifications. In contrast, sorming-based simulators have recently garnered significant attention as they offer more efficient solutions. Previous works primarily simulate object dynamics in 2D. They either treat pixels as grids [ or map the images into low latent space 2 3 and predict future latent states. However, these 2D-centric approaches possess limitations. The world is inherently 3D, and 2D methods struggle to reason about physical processes because they rely on view-dependent features, and are hard understand real object geometries. To address these, researchers have incorporated multi-view 3D perceptions into simulations such as Neural Radiance Field (NeRF) [4] which is an implicit 3D-aware representation. Some studies extract fiw-iinariiant representations and 3D structured priors by NeRFs to learn 3D-aware dynamics 6]. They either represent the whole scene as single vector learn compositional object features by foreground masks. However, they require heavy computational demands and struggle with objects with high degrees of freedom.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Particle-base based learned simulators show impressive results in modeling 3D dynamics. The success mainly attributed to the popularity of Graph Neural Networks (GNNs). In general, objects are represented as particles that are regarded as nodes in graphs, and their interactions are modeled by edges. Previous studies 7 g adapt GNN to predict particle tracks and achieve good results. Recent research has made strides in improving GNN simulators [ [O]. They require particle correspondences across times for training. However, 3D positions of particles across time are not always accessible and learning dynamics solely from visual input is still big challenge. The reasons can be summarized as follows. Determining particle positions from 2D observations leads to uncertainty since different particle distributions can produce similar 2D images. Moreover, previous GNN-based simulators aim to learn how to infer the entire dynamics, which are fully uninterpretable and result in hard optimization. Several studies [[] reconstruct 3D from 2D images and then learn from it, but they are not directly trained end-to-end with pixel supervision. One feasible way is to employ inverse rendering. For example, [13 14] use NeRF-based inverse rendering learn dynamics from 2D images. However, they are either constrained to simulate specific material or incapable of dealing with the 2D-3D ambiguities, thereby damaging their generalization ability. Furthermore, existing approaches only evaluate their methods on simple datasets. Their synthetic dataset often contained a limited variety of materials (usually rigid bodies), rarely involved collisions between objects, and featured very regular initial shapes.
To address the above challenges, this work incorporates strong mechanical constraints in the learning- based simulation system to effectively learn the 3D particle dynamics from 2D observations. Discrete Element Analysis (DEA) [[], also known Discrete Element Method [16], is traditional numerical method to simulate particle dynamics in mechanical analysis. This method computes interaction forces between particles predict how the entire assembly of particles behaves over time. However, traditional DEA heavily relies on the user- peer-predefiee mechanical relations between particles, such as constitutive mapping or dissipation modeling, which often involve several material-specifie hyperparameters. Moreover, the results often deviate from the actual mechanical responses because the constitutive equations are overly idealized.
In this work, we combine GNNs with the DEA theory to implement a lator, called the Discrete Element Learner (DEL), aiming to enhance the robustness, generalization, and interoperability of GNN architectures. In detail, we use GNNs as kernels to learn the mechanical operators in DEA rather than adopting user-define equations. On the other hand, the graph networks only need to fit some specific physical equations in the DEA framework instead of learning the whole evolution of dynamics, which largely reduces the optimization difficulties. Therefore, combining the conventional mechanical framework and GNN can reduce the 2D-to-3D uncertainty and alleviate the ill-posed nature. The main contributions of this work can be summarized follows:
neural simu-
We propose novel inttegiid neural simulation system called DEL which incor- porates graph networks into the conventional Discrete Element Analysis framework to effectively learn the 3D particle dynamics from 2D observations in physically constrained and interpretable manner.
We design the network architecture under the guidance of the DEA theory. In detail, we use learnable GNN kernels to only fit several specific mechanical operators in the classic DEA framework, instead of learning the entire dynamics to make GNNs and the DEA mutually benefit from each other, significantly reducing the ill-posed nature of this task.
We evaluate our method on synthetic datasets that contains various materials and complex initial shapes compared to existing ones. Extensive experiments show that our method surpasses all previous ones in terms of robustness and performance.
2 Related Work
2.1 GNN- particle dynamics simulator
There has been many works [7 18 9 19 20 21 to develop GNN-based particle simulators to predict
3D dynamical systems. This is because representing 3D scenes as particles perfectly matches the graph structure via particles as nodes and interactions as edges. GNS 7 shows plausible simulations on multiple materials by multi-step message passing. DPI X adds one level of hierarchy to the rigid and predicts the rigid transformation via generalized coordinates. EGNN [17] maintains the
2

---

render
Neural Reconstruction
GT
2
Time-0
Dynamic Predictor Recurrent Dynamic Inference
Time
Particles Initialization
(b)
The paradigm of the dynamics learning via inverse rendering. (a) Particles Initialization Process. The scene is initialized as particles. (b)Rrcurrent Dynamic Inference Process. The generated particle set is fed into dynamic predictor to infer the next state iteratively.
Figure 1:
equivariance of graphs by passing scalar and vector messages separately, and explicitly assumes the direction of vector message passing along with the edges. SGNN (9] proposes the subequivariant simulator, which has strong generalization box models and non-interpretable, thus complicating the optimization. There are some works incorporating basic physical priors into neural networks [22 23 24 25 (2),, whereas they perform well either on toy examples or specific topologies such as rigid hinges. All the above-mentioned learned simulators require full 3D particle tracks as labels for training. They cannot learn from pixel-level supervision because of the large solution spaces caused by the 2D-to-3D uncertainties, which experimentally proved in Section Our approach reduces ambiguities by integrating GNNs into a mechanical analysis framework. Our method not only yields impressive results supervised by 3D labels but also effectively learns realistic physics under 2D supervision.
long-term predictions. Most of them are black-
2.2 Learning dynamics from 2D images
Learning dynamics from merely visual observations is vital for many domains. Previous works
[ map images into low dimensional space and learn dynamic models to infer the evolution of
latent vectors. However, the general latent approach [27 28 29
33 makes things like pixel-level video prediction rather than real physical inference 34 The biggest reason the
30
31
32
gap between 2D observations and 3D worlds [0]. To address this challenge, recent works consider 3D-invariant representation to build latent states. NeRF used by (5] to encode multiview images as view-independent features. But it serves the whole scene single vector, and cannot handle scenes with multiple objects. [6] encodes compositional multi-object environments into implicit neural
scatter functions, while it only handles rigid objects. Similarly, (33] and (35) compositional
implicit representation, but cannot simulate objects with large deformations. Some other methods
need additional signals, such as Lidar data. The 3D-aware latent dynamics also lack generalizability to unobserved scenarios and cannot work with complex topologies and varying materials. Moreover, latent dynamics models are fully non interpeetable..
36 37 38
Very recently, with the development of differentiable neural rendering, a few studies have attempted to train 3D dynamic models from visions via inverse rendering [33 [4] 39]. They bridge images and 3D scenes with a differentiable renderer to minimize the renderings and groundtruth. However, [13 and [14] only simulate fluids because they require fluid properties input. 3PD learns 3D particle dynamics directly from images and can simulate various solid materials. However, requires jointly training its own particle renderer and latent simulator, which leads to black-box nature and is hard to be adopted by other renderers. Conversely, our DEL can seamlessly integrate into any point-based renderers with satisfactory performance and is physically interpretable well as can simulate various materials in unified manner.
3 Methodology
3.1 Preliminaries and Problem Statement
This task is formulated as inferring particle dynamics via inverse neural rendering. Similar to other inverse graphics, the scene can be represented by 3D primitives, and then the dynamical module infers the future state of these primitives. Once this future state is inferred, it can be effectively transformed into visual images by neural renderers. The dynamical module can be trained from

---

the eгror between the renderings and observations. Figure T illlstrates the general paradigm. In this formula, 3D scenes should be represented as particles, and then, the renderer should be able to render particles into images with given camera viewpoint, According to the above discussion, we choose the Generalizable Point Field [40 (GPF) as our renderer, which can represent 3D scene as particles, change its content by moving particles, and render images with arbitrary views. Notably, other particle-based renderers such as [ or recently prevailing [42], can also be used arbitrarily as long as they are fully differentiable and represent objects particles, The renderer module can iteratively produce updated images after the dynamic module moves particles at each timestamp.
The dynamical module operates as graph network simulator. Consider a physical system with N particles to represent M objects, the simulator models its dynamics by mapping the current state to consequent future states, usually the positions of particles. Assume X; E X are particle states at time I, X usually includes the coordinate xf, the velocity v and particle's intrinsic attributes A; such as the material type and mass. The learnable GNN simulator S considers particles as nodes and dynamically constructs connections at independent time steps when the distance between two particles is smaller than a threshold (E' { ||x; Xy ll2 <= r)) The GNN maps all the information at the current state to the positions at the next timestamp by passing messages on the graph, i.e. x Sg(x,, v a, E). Different GNN simulators mainly lie in the different designs of pesssae--asssinge networks. As we claim in Section [ the Sgs in most previous approaches aim to learn the entire dynamics process, which leads to hard optimization and the risk of overfitting. Moreover, they are non-interpretable black boxes. Therefore, the learning target would be very ill-posed because the solution space very large when only visual supervisions are given.
We propose a mechanics-encodee architecture that combines the GNN with the typical DEA to reduce uncertainty and improve interpretability. In the following section, we first introduce the general DEA method and its potential to be enhanced by the learning-base kernels. Second, we present how we incorporate the graph networks into DEA to replace the traditional operators.
3.2 The General Discrete Element Analysis Theory
In this section, we introduce the general framework of Discrete Element Analysis, also known as the Discrete Element Method, and its drawbacks which potentially can be enhanced by our learnable kernels. Here we only cover the general knowledge that we need to design our architecture, we recommend readers refer to [15] 43 [6] for deeper knowledge of DEA. In the framework, the whole scene represented as particles and the DEA used simulate the behavior and interactions of these particles. Generally, in this framework, the movement of an individual particle is governed by the Newton-Fuu -uler motion equation:
d'u
a
Σ F + + +0+++) FFF
dt2
where is the movement vector, m, the mass of particle i. Fy refers to the gravity. FGJ and Flj are the interaction forces between particle and j. the former marks potential interaction force. and the latter marks dissipative (viscous) contributions. The potential interactions primarily arise from physical contact between elements [5]. The dissipative contributions take into account kinetic energy dissipation mechanisms concerned with the dispersion of elastic waves (this dissipation is general for all materials) 49). Given this context, the potential contributions to interactions assume a significant role while the dissipative contribution merely influences the energy dissipation within the system. Therefore, the fundamental problem is to formulate a general form of potential interactions between particles, which would apply to materials with different features of mechanical responses.
Besides, F.. and Fú, can be decomposed into the normal and tangential directions, which represented by the superscript n and in Equation 2
F
F
F
+F
(2)
Substituting Equation 2 into Equation and omitting the gravity terms for simplicity, we can derive:
duu di
m,
+
> [F + FF
(3)
j=1
4

---

where the first term is the normal constituent and the second term is the tangential constituent. Here we discuss the potential interaction forces within the two directions respectively. According to [[]] The Fy can be considered as the composition of contact force f and bond force Sty. The contact forces are activated when two particles physically contact and collide. The bond forces mean if the two particles belong to the same object, they are connected by a bond that will provide attraction or repulsion based on their relative positions to maintain the fundamental properties of the material. We depict the mechanism in Figure 2 Furthermore, in DEA, a physical quantity called intrusion scalar is commonly used to compute the two forces:
ddn = (ri + r2) ||x -xil2
where ris ry are the radius of particle X1, Xj. ōd, > 0 means they contact and vice versa. We use visual aid Fig. aaa to depict ōd, intuitively. The red hard sphere intrudes into the blue soft sphere in the figure. The deformation length on the blue surface refers to the ōd,, The normal contact force fi should be related to the ōd,, because the particle tends to recover its initial shape. In addition, the direction of for is the normal direction between i and j. On the other side, as stated in Figure (bb) pbn acts as a linkage between two particles belonging to the same object, akin to bond. f also relates to ddn. According to the above discussion, the total normal potential interaction forces can be formulated as:
(F Ars) Fb' (ōdn. Ai))) ) (ödn; Arj )
i.j E Ok: i.j $ O
where F% F are two functions map the dd,, and Aij = [Ai, to the contact and bond forces respectively. Og is the Object k. Here A, is the material properties in the particles' small surrounding vicinity. n xxy-Xi specified by users, which are often simple linear polynomial functions. For example, the most simple way to compute the total interaction force is the linear spring model Foto kōd,,- Some more complex functions such as Mirdz-Mindlin (Eq. ( are also commonly used.
x,-X,_ is the normal unit vector. In DEA, the F Ai, Aj usually require to be
4 EV EV Rōdn, Fg (ōd.)) ki ödn 3
P7 (odx)) -
(6)
In Eq. 6 E, R,, hú are human defined material parameters. However, the above-introducet handcrafted
mapping functions only roughly approximate real cases and always deviate from the realistic natures of materials, leading to inaccurate simulations of DEA. Likewise, the tangential interactions can be analogous to the above. Notably, the normal direction contributes mostly to the total potential interactions because the tangential deformation is small due to the friction constraints [4].. Therefore, general DEA often approximates it by the instantaneous displacement within timestamp At, i.c. ddl = || At|l2. vi is the tangential velocity of j relative to i.
f = (ode, Auj. (IGG)ID DI + F. (ōde, Anj,
Similar to n, the t= v /IvV/ is the tangential unit vector to determine the force direction. F Fb are the dser-defined simple functions. Furthermore, the tangential magnitude is also affected by the normal force 45) thus ||vř,|] included. The DEA theory includes the dissipative contribution (15), also called the viscosity term [45] as it is only to simulate energy dissipation to prevent the system from exhibiting perpetual motion. In general, large velocities lead to large dissipation. In the general DEA, the term is often modeled by:
Fris = 11 Cerit Urel
(8)
where Corit is a material properties called critical damping, Uret refers to Un (normal velocity), v (tangential velocity) correspondingly.
Based on the above discussion, DEA requires multiple user-speeific mechanical operators including F Fy. F F. FG, Ft. wis. which would be inaccurate and fully different for various materials. Moreover, we cannot precisely estimate the real properties of materials when only videos are available. This can be considered another impracticability of the DEA framework. On the contrary, the benefit of DEA is that we only need to solve the magnitudes of these decomposed forces because these directions are physically constrained in this framework. Therefore, we keep the advantages of the physical priors while remedying the defects of DEA by replacing these mechanical operators with trainable network kernels.
5

---

(a) Different objects
(b) Same objects
Objest
Treperr
Object
Bond
Felocky
Dh, n Rd,, v
Farce
3d
V
Impac?
&
Intistion
D-v 1
Normal
Figure 2: Two cases of particle interactions. (a) contact forces, affected by the intrusion ōd,, (b) The bond force exists between two particles of the same object, affected by the bond length.
Tangenial selvar
MLP
Figure 3: The main pipeline of message-passing network
3.3
Mechanics-informed Graph Network Architecture
This subsection introduces the proposed Discrete Element Learner which replaces these human- specific operators in DEA with learnable graph kernels. In other words, integrate physics prior knowledge into the network design to make the entire AI system differentiable and can be optimized through image sequences. We visualize such mechanics-ing mechanies-integrated network architecture in Fig. 3 Furthermore, our method implicitly encodes the material properties into embedding vectors during the unsupervised training. Similar to other GNN-based simulators, we first construct subgraphs by searching neighbors with a fixed radius, and each subgraph can be viewed as a collective of particles involved in interactions. Then we convert all physical variables including velocities and positions to each particle-centric coordinate system when analyzing associated particles.
We define the -EA-incorporated peassage-psssing network as follows and the symbols previously used maintain their consistent meanings. First, we encode each particle attribute A such as material types into latent embedding h; 2200 [-1,1] 1] via Eq. 9
h, N Norm (MLP(A.))
(9)
Second, the following four equations in GNN are used to implement Eq.
n4, n, Cij 45 (ōdn, hi)
(10)
fij ReLU R2,, ei))
(11)
f = Ho,,, Eij)
(12)
f1j 1j + fly
(13)
where 4 a graph network kernel, n, ny. and Cij are node and edge features. na. encode the properties of the small regions around particle j. eij encodes their interaction. H.s and H, are two heads to regress the magnitude of the two forces. Due to the previous discussion, the contact force f can only act from towards i, it must be positive value, therefore we apply ReLU activation. While the bond force frb can either positive or negative, thereby no activation is used, As for the dissipative effect, we consider the dissipation value because always diminishes the potential interaction force. In our network, we use an MLP p" activated by Sigmoid model the normal dissipative phenomenon in Eq. 14
reduction coefficient rather than directly regress its
f = o66 (11 ff n
(14)
Likewise, we apply another kernel of (Eq. 15 to replace Eq. 7 A minor difference is that we omit the tangential bond force because when the particle undergoes very small relative displacement tangentially, the bond length remains nearly unchanged (ry) x 0). In this way, fo C
S4 क5 (1lvtyl2: Jij eij, ni, ni,ri)
(15)
According to the discussion in the previous section, the quantity ftJ relates to particle properties (n,). the tangential relative displacement [lof,]12238 and the precomputed normal pressure (ft)) ') Similar to σ", of models tangential dissipation in Eq. 16
f (1e
(16)

---

3Dintphts
EGNN
VPD
Ours
GT
Figure 4: Qualitative Comparisons of dynamics prediction between DEL and baselines in the particle-view on test sequences.
Notably, all graph kernels output the magnitudes of these mechanical vectors because our mechanical framework precomputes their directions by and t, which largely reduces the ambiguities when learning dynamics. It can be observed that our approach deeply integrates the mechanical analysis framework, and is partially interpretable. The design of this architecture is strongly inspired by the physical knowledge to reduce the learning burden, we vividly demonstrate it in Fig.
3.4 Training Strategy
We train the DEL from only visions. Assuming cameras record many dynamic sequences including the materials but with different initial conditions, first initialize scenes particles by GPF at timestamp to. Then we adopt the first three frames to approximate the initial velocities, similar [46). Next, the dynamic module tries move particles to the next state. Then the GPF renders the after-moving particles into images. If the positions are correctly predicted, the rendered images should align with the recordings. We use L2 loss to supervise
Σ | R(D(xx-1) cam))] cam
(17)
where D refers to the dynamic module, X,-1 is particle states at the last timestamp, R is the GPF rendering module, and I.am is the observed image view cam and time t. Moreover, we apply the same L2 loss to supervise the images produced by the particles updated by using only the f." (the normal force) stated in Figure 3 This aims to amplify the contribution of the normal direction since it claimed the previous section that the normal direction the main influencing factor for collisions. The formulation of this L2 loss is referred to C, In addition, the gradients of rendering loss OC. 8x can be interpreted as the scaled velocities of particles. Thus propose the gradient loss to constrain the direction of the total velocity of each particle.
aL, Ox:
(1
Cy =
(18)
da;
The final loss L CT + C, 3C.. More implementation details are contained in the Appendix.
4 Experiments
Dataset. Existing datasets in this field contain only a few types of materials or even only rigidity, lack rich interactions between objects, and have simple initial shapes. Therefore, we create more
7

---

Table 1: Quantitative Comparisons between and benchmarks five scenarios in render views.
Plasticine SSIMT
SandFFall
Multi-Objj
FLuidR
Bear
Method
PSNR1
LPIPS|
PSNR1
SSIM
LPIPS|
PSNR
SSIMT
LPIPS|
PSNR"
SSIM1
LPIPS,
PSNR"
SSIM
LPIPS]
SGNN*
25.27
0.925
0,143
21.61
0.886
0.216
24.76
0.909
0.166
28.88
0.935
0.168
27.61
0.949
0.132
NeRF-dy dy
21.09
0.893
0.225
22.58
0.879
0.216
19.61
0,826
0.318
25.79
0.925
0.270
22.83
0.873
0.232
EGNN 17
26.27
0,944
0.119
25.17
0,918
0.178
26.38
0,928
0.144
30.28
0.951
0.123
29.13
0.953
0.117
VPD 39
27.06
0.941
0.101
24.61
0.926
0.127
25.62
0.921
0.136
30.06
0,947
0.126
30.52
0.964
0.102
Ours
28.09
0.959
0.091
26.65
0.945
0.113
27.06
0.939
0.128
30.53
0,944
0.122
30.08
0.964
0.105
Table 2: Quantitative comparisons between ours and baselines on five scenarios in particle views.
Plasticine
SandFall
Multi-Objs
FluidR
Bear
Method
CD4
EMD4
CD1
EMD4
CD4
EMD1
CD4
EMD4
CD4
EMD
SGNN# [9
35.91
26.4
2.47
2.69
20.3
26.9
3.98
5.02
4.69
5.01
3DIntphys [1
26.99
22.61
3.17
3.35
16.55
17.61
6.92
8.01
6.69
6.01
EGNN* [1
16.20
14.61
2.13
2.56
13.21
13.77
2.58
3.01
3.95
4.16
VPD 39
16.96
12.77
1.99
2.35
14.26
14.57
3.22
2.94
3,41
3.71
Ours
7.54
7.10
1.73
1.90
8.48
9.13
1.72
1.88
3.54
3.33
challenging dataset that includes various materials (rigid, elastic, plastic, fluid, and sandy soil). The dataset includes six main scenarios. Each scenario contains 128 training and 12 testing dynamic sequences with different initial conditions such as velocities and shapes. These sequences generated by our MPM simulator, then Blender is used to render them to produce high-fidelity multiview images. We use 4 cameras to observe each dynamic episode. The six scenarios are: Plasticine illustrates the collision of an elastic ball with a plasticine toy. Multi-Objs includes rigid and elastic objects. SandFall depicts sand descending onto elasticities. Fluids contains fluid with different viscosity. FluidR describes that Newtonian fluid flows onto rigid bodies. Bear involves interactions of elastic, plastic, and rigid objects.
Baselines. We compare the proposed approach with the two most recent baselines, i.e. VPD 39 and 3DIniphys []]. These two methods focus on learning 3D particle dynamics from 2D images. VPD must jointly train its specific point renderer and particle dynamics module, which are highly coupled. Therefore, it cannol use other point-based renderers. In contrast, our DEL does not contain any assumption of the neural renderer and thus can be directly adapted to different renderers. We choose the GPF [40] our renderer. 3DIntphys first reconstructs series of NeRF's for the image sequences at all timestamps, then extracts the point cloud from each NeRF, and finally trains its dynamics module with point similarity loss across time. Hence, it cannot be trained in end-to-end manner and cannot render novel views from the updated particle sets.
Moreover, to evaluate the effectiveness of our dynamics prediction. We set two additional baselines in which we replace our dynamic model with two prevailing GNN-based simulators, i.e. EGNN [17 and SGNN p while keeping the same point-based render with us. In their original paper, they require 3D particle correspondence across time for training. But in our setting, we also utilize inverse rendering to train them from images. We mark these two variants via an upright We also compare with the NeRF-dy (5). fully implicit dynamics predictor. Next, we additionally retrain NeuroFluid [K] on the Fluids scenario for comparisons because it only supports inferring fluid dynamics.
Metrix. We use the Chamber Distance (CD) and Earth Mover Distance (EMD) to measure the similarities between the predicted particle distributions and the groundtruth because no per-particle correspondence is available. The reported CD and EMD are multiplied by 102 in all tables for clarity. We compare the PSNR, SSIM, and LPIPS (AlexNet) between the renderings and 2D labels. We also provide qualitative results and comparisons for better visual assessment.
4.1 Results and Comparisons
Results in Rendering View. We also present the comparisons of rendering qualities to further evaluate the effectiveness because we adopt inverse rendering to bridge 2D and 3D. The quantitative results are listed in Table [] The VPD overall achieves the second- designed specifically for high-quality rendering. However, due to the robust physical priors, our approach more easily learns the underlying physical rules, resulting in a more accurate dynamics prediction, consequently, rendering more plausible images. The other two graph-based simulators perform mediocrely. We additionally show visual examples in Figure Our approach gives better renderings than baselines due to better dynamic predictions. Due to page constraints, only give
performance because it is
8

---

EGNN
VPD
Outs
GT
Figure 5: Qualitative Comparisons of rendered images between ours and baselines.
GT
Outs
Narrolad
Figure 6: Qualitative comparisons between neurofluid and ours.
examples of the Plasticine scenario. More results on other scenarios can be seen in the Appendix. Results in Partiele View. In this section, report the simulation results generated by differ- ent approaches in the particle view. Table 2 and Figure 4 show the quantitative and qualitative comparisons respectively.
is observed from Table 2 that our method delivers the most satis- factory results across all scenarios. One interesting finding is that the EGNN' overall outper- forms the SGNN", but in their respective original papers. the SGNN performs better than EGNN when 3D labels are available. The reason might be EGNN benefits from predefining message- passing directions, but SGNN simultaneously determines both directions and values, which hard to optimize when only 2D images are given. From this figure, VPD, 3DIntphys, EGNN*, and SGNN" cannot predict precise interactions while our method shows steady long-term simulation.
Table 3: Ablation studies of four components
Table 4: Quantitative results Fluids scene
SandFall
Multi-Obis
Particle-view
Render-view
Method
CDJ
EMD1
CD)
FMDJ
Method
CDL
EMD1
PSNR1
SSIM
LPIPS
NofE,
4.09
29.7
32.9
Neurofluid
10.7
11.0
25.36
0.930
0.175
NofE.
2.26
2.78
11.4
18.6
SGNN*
11.87
11.32
27.68
0.946
0.182
Nofderomp
3.15
3.04
16.5
13.3
EGNN
10.76
9.78
29.01
0.962
0.108
N,
2.65
2.91
10.27
12.38
Ours
4.18
2.97
30.02
0.962
0.104
Full
8.48
4.2 Additional Comparisons and Analysis
Comparisons with Neuroftuid. Neurofluid [13] another unsupervised method for learning fluid dynamics. It uses the particle-based PhysNeRF as the renderer and employs DI.F [47 as the dynamic modules. We compare our approach with on the Fluids. The results are reported in Table 4 and
Fig. 6
The results show that Neurofluid cannot work well on test data because its dynamic module

---

lacks enough physical priors. Another reason is that it jointly trains the renderer and dynamics modules which makes them compensate for each other, causing overfitting of training data.
Ablation Studies. We evaluate some significant components of our method. First, we ablate the gradient loss (marked as Cy) Second, we report the contribution of the tangential decomposition constituent f (No/ff)). Third, we make the graph network fully regress the direction and magnitude of the interaction forces (N (No/ oedcompp) instead of using the priors encoded in the DEA framework, i.e. the output of the graph is a force vector. Next, we ablate the L loss term, further proving the significance of normal components. The quantitative results are listed in Table which shows that the Lg and C, contribute simplifying the optimization. In addition, the mechanical decomposition is important as well. Even though the main direction of message passing is along the directions of edges, the tangential components indeed make the simulation results more realistic. Furthermore, we evaluate the effect of different renders, different training data sizes, different numbers of cameras used to capture scenes, and different points. We also test the Rollout MSE of the three methods when the 3D labels are available. Both of the results can be seen in our appendix, which shows that our method is satisfactory and robust under all the above ablation conditions.
Conclusion and Limitation
Conclusion. We propose the DEL which combines the Discrete Element Analysis framework with graph networks to effectively learn 3D particle dynamics from only 2D images with various materials. The main idea is integrate strong physical priors to reduce 2D 3D uncertainties. Existing GNN-based simulators, which are designed for learning from 3D particle correspondence. try to model the whole dynamics of particles. Differently, the DEL only adopts graph networks as learnable kernels to model some specific mechanical operators in the DEA framework, while keeping its mechanical priors, such as the direction of forces and decompositions of forces. We also evaluate our approach on synthetic data with various materials, initial shapes, and extensive interactions. The experiments show our method outperforms baselines when only 2D supervision is accessible. We also show the robustness of our methods to the renderers, training data sizes, and 3D labels.
Limitation. Currently, studies in this field, including this work, are conducted on synthetic datasets due to the impracticability of collecting multiview dynamic videos. Hence, "learning from few data" could potentially help address the problem of learning 3D dynamics from a single realistic video. We include them in our future work.
References
[1] Haozhi Qi, Xiaolong Wang, Deepak Pathak, Yi Ma, and Jitendra Malik. Learning long-term visual dynamics with region proposal interaction networks. In International Conference on Learning Representations, 2020.
[2] Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. pages 2555-2566 PMLR, 2019.
[3] Rohit Girdhar, Laura Gustafson, Aaron Adcock, and Laurens van der Maaten, Forward prediction for physical reasoning. In International Conference on Machine Learning Workshop. 2021.
[4] B Mildenhall, PP Srinivasan, M Tancik. JT Barron, R Ramamoorthi, and R Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. European conference on computer vision, 2020.
[5] Yunzhu Li, Shuang Li. Vincent Sitzmann, Pulkit Agrawal, and Antonio Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pages 1 112-133, 2022.
[6] Stephen Tian, Yancheng Cai, Hong Xing Yu, Sergey Zakharov, Katherine Liu, Adrien Gaidon, Yunzhu Li, and Jiajun Wu. Multi-object manipulation via object-centric neural scattering functions. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9021-9031, 2023.
[7] Alvaro Sanchez-Gonzale Jonathan Godwin, Tohias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. pages 8459-8463 2020.
[8] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralaa Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, 2018
[9] Jiaqi Han, Wenbing Huang. Hengbo Ma, Jiachen Li, Josh Tenenbaum, and Chuang Gan. Learning physical dynamics with subequivariant graph neural networks. Advances in Neural Information Processing Systems, 35:26256-22 2222688.2668.... 2022.
10

---

[10] Daniel Bear, Elias Wang, Damian Mrowca, Felix Jedidja Binder, Hsiao-Yo Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin A Smith, Yun-Yun Sun, et al. Physion: Evaluating physical prediction from vision in humans and machines. 2021.
[11] Haotian Xue, Antonio Torralba, Joshua Tenenhaum, Damiel Yamins, Yunzhu Li, and Hsiao-Ya Yu Tung. 3d-intphys: Towards more generalized 3d-grounded visual intuitive physics under challenging scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3624-3634,
2023.
[12] Lili Li, Toru Lin, Kexin Yi, DavidM. Bear, Daniel Yamins, Jiajun Wu, Jo Teneabaum, and Antonio Torralba. Visual grounding of learned physical models. arXiv: Learning,axXiv: Learning, Apr 2020.
[13] Shanyan Guan, Huayu Deng, Yunho Wang, and Xiaokang Yang. Neurofluid: Fluid dynamics grounding with particle-driven neural radiance fields. International Conference on Machine Learning, pages 7919-7929. PMLR, 2022.
[14] Jinxian Liu, Ye Chen, Bingbing Ni, Jiyao Mao, and Zhenbo Yu. Inferring fluid dynamics via inverse rendering, arXiv preprint arXiv:2304, 04446, 2023.
[15] Peter Wriggers and B Avci. Discrete element methods: basics and applications in engineering. Modeling in engineering using innovative numerical methods for solids and fluids, pages 1-30, 2020.
[16] Du-Min Kuang, Zhi-Lin Long, Ikechukwu Ogwu, and Zhuo Chen. A discrete element method (dem) approach to simulating particle breakage. Acta Geotechnica, 17(7):251122744 2022.
[17] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks.
pages 9323-9332, 2021.
[18] Wenbing Huang, Jiaqi Han, Yu Rong. Tingyang Xu, Fuchun Sun, and Junzhou Huang. Equivariant graph mechanics networks with constraints. In International Conference on Learning Representations, 2021.
[19] KelseyR. Allen, Yulia Rubanova, Tatiana
WilliamDwight Whitney, Alvaro Sanchez- Gonzalez, PeterW. Battaglia, and Tobias Pfaff. Learning rigid dynamics with face interaction graph networks. Cornell University DrrXixCCoredd University arXiv, Dec 2022.
[20] Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez. and PeterW. Battaglia. Learning mesh-based simulation with graph networks. arXiv: Learning arxiv:: Learning, Oct 2020.
[21] Kelsey R Allen, Tatiana Lopez Guevara, Yulia Rubanova, Kim Stachenfeld, Alvaro Sanchez- Gonzalz, Peter Battaglia, and Tobias Pfaff. Graph network simulators can learn discontinuous, rigid contact dynamics. In Conference on Robot Learning, pages 1157-1167. PMLR, 2023.
[22] Yulia Rubanova, Alvaro Sanzhez-Gonzalez, Tobias Pfaff, and Peter Battaglia. Constraint-basee graph network simulator. arXiv preprint arXiv:21
2021.
[23] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty, Extending lagrangian and hamiltonian neural networks with differentiable contact models. Advances in Neural Information Processing Systems, 34:21910-21922, 2021.
[24] Wenbing Huang, Jiaqi Han, Yu Rong. Tingyang Xu, Fuchun Sun, and Junzhou Huang. Equivariant graph mechanics networks with constraints. arXiv preprint arXiv:2203.0644 064442, 2022.
[25] Suresh Bishnoi, Ravinder Bhattoo, Sayan Ranu, and NM Krishnan. Enhancing the inductive biases of graph neural ode for modeling dynamical systems. arXiv preprint arXiv:2209.10744 2022.
[26] Guangsi Shi, Daokun Zhang. Ming Jin, and Shirui Pan. Towards complex dynamic physics system
simulation with graph neural odes. arXiv preprint arXiv:2305. 12334, 2023.
[27] Nicholas Watters, Daniel Zoran, Theophane Weber, Peter Battaglia, Razvan Pascanu, and Andrea Tacchetti. Visual interaction networks: Learning a physics simulator from video. 30, 2017.
[28] Yufei Yc, Mancesh Singh, Abhinav Gupta, and Shubham Tulsiani. Compositional video prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10353-10362, 2019.
[29] Yilun Dai, Mengjia Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Josh Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text- guided video generation. Jan 2023.
[30] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301 04104, 2023.
[31] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300
2022.
[32] Haochen Shi, Huazhe Xu, Zhiao Huang, Yunzhu Li, and Jiajun Wu. Robocraft: Learning to see, simulate, and shape clasto-plastic objects with graph networks. arXiv preprint rrXiv:2205.02909 2022.
[33] Danny Driess, Zhiao Huang. Yunzhu Li, Russ Tedrake, and Mare Toussaint. Learning mujei-object dynamics with compositional neural radiance fields. In Conference on Robot Learning, pages 1755-1766 2023.
11

---

[34] Haotian Xue, Antonio Torralba, Joshua Tenenbaum. Daniel Yamins, Yunzhu Li, and Hsiao- Yu Tung. 3d-iniphys: Towards generalized 3d-grounded visual intuitive physics under challenging scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3624-3634,
2023.
[35] Danny Driess, Jung-Su Su Ha, Marc Toussaint, and Russ Tedrake. Learning models functionals of genedddistanee fields for manipulation planning. In Conference on Robot Learning, pages 245-255, 2022.
[36] Zhenjia Xu, Zhanpeng He, Jiajun Wu, and Shuran Song. Learning 3d dynamic scene representations for robot manipulation. In Conference on Robot Learning, pages 126-142, 2021.
[37] Jonas Linkerhägner. Niklas Freymuth, Paul Maria Schcikl, Franziska Mathis-Ullrich. and Gerhard Neu- mann. Grounding graph network simulators using physical sensor observations, In International Conference Learning Reprexentations, 2022.
[38] Lucas Manuelli, Yunzhu Li, Pete Florence, and Russ Tedrake. Keypoints into the future: Self-supervised correspondence in model-based reinforcement learning. In Conference on Robot Learning, pages 693-710,
2021.
[39] William Whitney, Tatiana Lopez-Guevara, Tobias PfafT, Yulia Rubanova, Thomas Kipf. Kimberly Stachenfeld, and Kelsey R Allen. Learning 3d particle-based simulators from rgb-d videos. arXiv preprint arXiv:2312.05353 2023.
[40] Jiaxu Wang. Ziyi Zhang, and Renjing Xu. Learning robust generalizable radiance field with visibility and feature augmented point representation. arXiv preprint UrXiv:2401.13344 2024.
[41] Jad Abou-Chakra, Feras Dayoub, and Niko Sünderhauf. Particlenerf: Partiele based encoding for online neural radiance fields dynamic scenes. arXiv preprint arXiv:221
2022.
[42] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-tine real-time radiance field rendering. ACM Trans. Graph., 42(4):139- 2023.
[43] Willy Leclere. Discrete element method to simulate the elastic hchavior 3d heterogeneous continuous media. International Journal of Solids and Structures, 121:86-102, 2017.
[44] Florian Fleissner, Timo Gaugele, and Peter Eberhard. Applications of the discrete element method in mechanical engineering. Multibody system dynamicx, 18:81-94, 2007.
[45] Federico A Tavarez and Michael E Plesha. Discrete element method for modelling solid and particulate materials. International journal for numerical methods in engineering, 70(4):379-404, 2007.
[46] Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. arXiv preprint arXiv:2311.12198. 2023.
[47] Benjamin Ummenhofer, Lukas Prantl, Nils Thürey, Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. International Conference on Learning Representations, Apr 2020.
12

---

Appendix
A
Impact Statements
This work introduces a novel deep learning architecture tightly integrated with a classic mechanical analysis framework, the Discrete Element Method, to efficiently learn 3D particle dynamics from only 2D observations. The success provides insights into the combination of physics- augieened deep learning and 3D neural rendering for areas, e.g. physical simulations, engineering mechanical analysis, and computer graphics.
B Nomenclature
list all symbols and marks used in the main paper in the following Table for the convenience of reference.
Table 5: Nomenclature. This table is split across pages
Variable
Description
i
i-th particle
X;
position vector of particle velocity vector of particle acceleration vector of particle movement vector of particle potential interaction force vector
V;
a,
u,
F
F
viscous force vector gravity vector
F
F
potential interaction force vector along the normal direction
F
potential interaction force vector along the tangential direction
F
viscous interaction force vector along the normal direction viscous interaction force vector along the tangential direction
Fet
dd
intrusion scalar
ddg
instantancous tangential displacement time interval
At
T
radius of particle
A,
attribute of i-th particle
F
bond particle between particles which belong to the same object mapping from intrusion to normal contact force mapping from intrusion to tangential contact force
F
F
mapping from intrusion to normal viscous force mapping from intrusion to tangential viscous force mapping from intrusion to normal bond force
F
continued page
13

---

continued from previous page
Variable
Description
Ok
set of particles belonging the same k-th object the tangential velocity vector normal unit vector tangential unit vector
v!
t
o"
the normal dissipative model
of
the tangential dissipative model normal graph kernel
h
tangential graph kernel prediction head to regress the magnitude of the contact forces
He
Hy
prediction head to regress the magnitude of the bond forces sigmoid activation function the normal bond force the tangential bond force the normal contact force the tangential contact force precomputed normal force vector normal force vector
fom
tht
fo
S
fl
precomputed tangential force tangential force vector the frictional coefficient i-th node features
f
n4
(t)
velocity of i-th particle at timestep t latent embedding from i-th particle attribute
h4
eij
edge features between node and
C Detailed architecture
We describe the detailed architecture used in the DEL. The entire graph operator depicted in Equation too Equation i6 in the main paper.
The embedding layer is an MLP with 2 hidden layers. The output embedding latent vector is normalized [-1.1]. The A; and A, refer to the embedding vectors of the per-particle attribute, which we define as = [mat, do, ||a;--1 "12]]. Here the mat denotes the type materials, d, the distance between the particle to the mass center at the current timestamp, and ||af- represents the value of the acceleration at the last timestamp of this particle. In our learning framework, the mainly represents the specific material and its properties. During training, the properties this material encoded into the embedding of A.
The x and & aim to map related physical quantities the abstracted node and edge features which implement by graph neural networks. åd, in d is the initial edge features, we also consider it as
14

---

Cij. Therefore, Tn can be described as:
temply = 21 (ely² ;hi,h)s))
e/j = (2emmmplyy ct5
temp's =
Σel cl
(19)
N
NT
T4 = Norm((a((eempss,)) h:))
where temps are temporary intermediate variables. ni, and e'g are the final output of the Pn. 1/1,2,3 are three 2-layer MLPs with residual connection. e
is the edge feature from the last network layer. Norm refers to learnable Layer Normalization. The reason why we perform an aggregation operation before computing the normal forces the following. We assume that the mechanical behavior of a certain particle should be related to the external intrusion and the properties of its surrounding vicinity. Therefore, we use this graph aggregation to encode the information of its vicinity (n;) and the external influence (eiz). The abstracted features then are input to two different output heads Hs. He to produce the final magnitude of the forces. The two heads are implemented as small MLPs with 2 layers and 200 hidden dimensions.
As for 4 in Equation 15
|u foj and Egy are input edge features. In addition, n, and nj are the concatenation of the n. from Equation I0 and the h2 from Equation because we aim emphasize the original particle attributes which affect the tangential force.
templ, = 1 (eij. n4, nz))
c #222temply "
templ 1
Σ
c
(20)
N
NO
f/ th c templ, temp,)
where ety cat([e);] -I 22,, mm 1), n, cat([n) h2]). The final outputs are fü and ety. The rest of the architecture remains the same with Equation 20, Besides, use simple MLP with two hidden layers (each layer includes 200 neurons) to model ø² and Φ" Also, the sigmoid activation used before outputting the coefficients because the viscous forces are manifested reduction in potential interaction forces. After flj in Equation 16 and f. in Equation 14 are obtained, we aggregate the forces for each particle:
f
Σ
( + +
(21)
NO
Thus we can update their velocities and positions by the Euler integration:
f
V, Vt-1 At
my
(22)
x; = X,-1 + (V.-+
At)
D Implementation Details
In this section, we introduce the implementation details of our experiment setup. We first use pretrained GPF to initialize the scene as particles.
For the dynamic module, we build the graph at each timestamp via the k-nearest neighbor search with fixed radius of 0.025. The particle radius (r for different materials) is equal the search radius initially and will be optimized through the training process to be property of the material. In addition, we place a heavy rigid table the bottom of each scenario, which is also represented by particles but we do not update its position, and its velocity is constantly set to zero. The only function of to support the moving objects above. Additionally, the mass parameter for each particle is set to initialize at and optimized during the training phase as well. All models are trained via Adam optimizer with 5e-4 learning rate. We adopt the StepLR schedule to adjust the learning rate with the increasing step. After 10,000 iterations, multiply the learning rate by 0.9.
15

---

Sev! Sand Randar
Elasiic
Hosticin
Figure 7: Examples about material swapping SandFall
E
Swapping Materials.
Our method has a unique advantage. We can swap materials used in simulation by swapping the material embeddings and graph kernels because different materials share the same mechanical framework and the graph network in it is only responsible for mapping physical quantities and material embeddings the forces. We show example illustrate this application. As shown in Figure 7 the first and second rows are the predicted rendered views and partiele views of the SandFall respectively. We change the graph kernel and material embedding of the sand the elastic and plasticine ones respectively. We can observe corresponding changes in the mechanical behavior of particles.
F Detailed Dataset Description
In this section, we thoroughly describe the data generation process for our experiments, employing Material Point Method (MPM) to simulate interactions among various materials. We designed six distinct scenarios to encompass diverse range of material combinations: Plasticine, Multi-Objs, Bear, FluidR. SandFall, and Fluids. Each scenario is represented through 128 dynamic episodes, differentiated by unique initial conditions such shapes and velocities, while maintaining consistent material properties, including elasticity and viscosity coefficients, within each scenario.
For the reconstruction of meshes from simulation data, we utilized SplashSurf, followed by rendering multiview dynamic image sequences using Blender. At every timestep, the motion is captured from four distinct camera angles. The objective of this research is to derive material interaction behaviors from these 2D observational sequences.
This scenario features interactions between two distinct materials: a red elastic object and blue elastoplastic material. We initialize the elastoplastic materials in various shapes, including duck and bunny configurations, which are then subjected to impacts from the red clastic ball from multiple directions.
Plasticine.
This scenario encompasses interactions among five objects composed three distinct materials. The blue ball and cuboid categorized rigid. In contrast, the cylinder and rainbow- colored ring are elastic, each characterized by unique values elastic modulus and Poisson's ratio. The experimental setup initiates with the rigid ball colliding with the rigid cuboid, triggering sequence of subsequent collisions. Each episode is distinguished by varying arrangements and initial configurations of the objects, showcasing diversity in shape and positioning.
Multi-Objs.
Bear. This scenario investigates interactions among objects composed of three distinct materials: plastic, elastic, and rigid. The setup includes brown bear modeled as plastic toy, a triangular ship constructed from an clastic material, and a heavy, rigid, yellow box. The experimental design involves the ship and box colliding with the plastic bear from various directions and positions, aiming to study the resultant material behavior and object interactions under different impact scenarios. Each collision is designed to explore the dynamic responses of plastic, elastic, and rigid materials when subjected to varying forces and angles of impact.
Fluids. The Fluids scenario examines the dynamics of fluid behavior it impacts a tabletop. with the fluid initially shaped into complex geometries, including forms reminiscent of bunny, Pokémon, or duck, each varying in size. This setup facilitates observations of the fluid's response upon collision
16

---

Muiti-objs
Multi-objs
3.25
3.09
3.12
3.08
3.01
EGMI
SGNN
Durs
3.50
EGNN SGNN
3.00
3.25
2,82
2.72
2.82
72
2.87
3,00
2.85
2.59
2,02
3.07
3.04
3.01
2.99
2.75
2.74
2.37
2.50
2,12
2.25
2.12
2.1
2,1
2.07
2.1
2,00
1.75
Input view
1.50
32
64
128
142
Training Data
Figure 8: Experiments of model evaluation on the SandFall with different numbers of input views. The coordinate is the number of input views, and the coordinate is the log Chamber Distance.
Figure 9: Experiments of the number of training episodes on Multi-objs dataset. The coordinate is the number of training data, and the y coordi- nate is the log Chamber Distance.
Muiti-objs
Multi-obis
EGNN
EGNN SGNN
3.15
3,06
SGNN"
27.01
2.89
2.76
[og[CD)
2.71
PSNR
25.8
24.83
2.27
2.23
24.06
PhysNeRF
ParticleNeRF
GPF
PhysNeRF
ParticieNeRF
GPF
Figure 10: Particle-view results of the dynamic prediction on Multi-objs dataset by using differ- ent particle-based renderers. The y coordinate refers to the log Chamber Distance metrics.
Figure 11: Rendering results of the three meth- ods on Multi-objs dataset by using different ren- derers. The y coordinate refers the PSNR metrics.
with the ground, encapsulated within a scenario that includes invisible boundaries to constrain the fluid's spread. Upon contacting these boundaries, fluid particles alter their trajectories, enabling detailed study fluid dynamics and boundary interactions. This scenario allows for the exploration of fluid behavior under varied initial conditions, contributing to understanding of fluid dynamics in controlled environments.
FluidR. In this scenario, we explore the dynamics of liquid flow over hollow shelf with a complex geometry. Across different episodes, we systematically vary initial configurations altering both its shape and the height from which it is released atop the shelf. The shelf itself is constructed from hard clastic material, characterized by a very high Young's ' modulus, to study the interaction between the liquid's 's fluid dynamics and the shelf's structural response. This setup provides a unique opportunity to observe the behavior of liquids contact with elastic materials under varying initial conditions, offering insights into fluid-structure interaction phenomena.
the liquid,
FandFall. This scenario investigates the dynamic interaction between a life buoy, composed sand soil, and an elastic toy duck. Upon impact, the toy duck undergoes deformation and subsequently recovers its original shape, causing the sand soil life buoy to rebound. significant challenge in modeling the dynamics of this interaction lies in the partial obscuration of the toy duck by the sand soil during impact. This obscuration results in incomplete information regarding the duck's deformation and response, complicating the task of accurately learning the system's 's dynamics. The study focuses
17

---

Multi-objs
Plasticine
40
EGNN
EGNN
30
Rollout CD
Ours
Rollout CD
30
20
SGNN
20
SGNN
10
0-
25
50
75
100
125
20
40
60
80
Figure 12: Accumulated rollout chamfer distance two scenes
on understanding and overcoming the limitations imposed by insufficient visual information on the elastic response of the toy under the impact of granular material.
G Additional Results and Comparisons
G.1
The establishment of Rollout metrics
We in addition show two line diagrams of the accumulated Rollout Chamfer Distance in the two sequences in Plasticine and Multi-Ohj, to further indicate the superiority of our dynamics module
(Fig. 12
G.2
The influence of different renderer
The default renderer utilized in the main paper the GPF. In this section, we evaluate our methods and the baselines with the other two particle-based renderens, i.e. PhysNeRF [33, ParticleNeRF [4]] both of which can render dynamic particles. PhysNeRF is employed in Neurofluid to represent and render scenes. However, in the original paper, it is jointly trained with the dynamics module and requires more camera views. For a fair comparison, we pretrain the PhysNeRF from scratch and freeze its parameters for all models. In addition, and ParticleNeRI because they are difficult to initialize the scenes as point clouds from sparse camera views (4 in our experiments), while GPF is capable of initialization due its depth estimation module. Figure I1 and Figure 10 show the rendering and dynamic prediction results of different renders. We can
provide initialized particle set for PhysNeRF
method achieves better performance regardless of which renderer is employed. Moreover, rendering capability scales proportionally with the accuracy of learned dynamics. As GPF produces better rendering quality, more accurate dynamics can be learned by utilizing it. This example illustrates that our method is robust to the renderer selection because physical priors always guide the model toward reasonable learning target.
Huid
4
H
#
Flid
Plastrine
H
H
H
Sand Stone
Figure 13: Detailed example of swapping materials without retraining the model the FluidR scenario
G.3 The influence of the number of camera views in training
In this section, we use various numbers of camera views to train the models. In the main paper, four camera views are deployed across four corners the scene. Here we additionally evaluate if 1, 2, 8, 10 cameras are used, what will happen? The ablation results on Multi-Objs dataset reported
18

---

Send rendar
Sand
Hlsiiiiee
Elastic
Figure 14: Detailed example of swapping materials without retraining the model the SandFall scenario
Multi-view
Input
Simulaation Remove Rond
Figure 15: Visualization of the remove the bond force.
in Figure 8 observed that our approach sensitive the number of cameras, even if only one camera is used, due to the physical constraints posed by our mechanics framework. The search space contracted by the physical knowledge injected into the model. Therefore our method be effectively trained via only visible particles while the invisible ones are constrained through physics. On the contrary, the other two baselines are severely affected by the number of cameras. They cannot learn well from sparse cameras. With the increase in the number of cameras, their performance also improved and the upward trend shows no signs of abating. We can assume that the maximum upper bound of such improvement should be the performance when the 3D labels are used to train them.
G.4 The influence of different sizes of training samples
We evaluate how the training set size affects the performance and the results are reported in Figure 9 With increasing sizes of the training set, the log Chamber Distance of EGNN" and SGNN" experience slight rises. However our performance remains stable and always stays on top, which means our method is not sensitive to the training sizes and can learn from sparse data.
G.5 What will happen if we remove the bond force?
In this subsection, we claim the importance of the bond force in the simulation system and the significance of using two independent networks to model contact and bond force separately. We did two additional experiments. The first is that remove the bond foree from the entire system, i.c. remove Eq. The second is that we use a single network to predict the bond force and contact force simultaneously, i.e. merge Eq. 12 and IT together. We observe that the simulator cannot keep the original shape of rigid and elastic bodies in either case. We show a visual example in Fig. [1 and Fig. to illustrate this. These two experiments prove the effectiveness and necessity of this design to separately model the two forces by distinct networks.
G.6 How does the performance when 3D groundtruth particle labels is used
We here report the evaluation of training these models with 3D particle tracks. Even though our approach aims to learn 3D dynamics from images, the mechanics-encoded paradigm is also helpful for learning from 3D labels. Table 6 show the rollout mean square errors of the three models in all scenarios. From this comparison, SGNN's performance has rapidly risen and is roughly par
19

---

Multi-view
Input
Simulation
Laiform
Train Bond
Figure 16: Visualization of the case to jointly model the contact and bond force by single network.
Table 6: Quantitative comparisons between our method and benchmarks on five scenarios in particle
views.
Given 3D GT Rollout MSE
10^2
Method Plasticine SandFall Multi-Objs
FluidR Bear Fluids
Ours 0.275
0.0414
0.239
0.136 0.301 0.051
SGNN* 0.268
0.0468
0.252
0.121 0.282 0.062
EGNN
0.614
0.127
0.568
0.318 0.705 0.143
with our method. Both of them outperform EGNN by a considerable margin, which is consistent with that reported in their original paper. This further proves that the reason why SGNN fails to perform well under pixel supervision is caused by the uncertainty of 2D to 3D. EGNN is better than SGNN under 2D supervision because it predefines the direction of message passing which reduces the learning space as well. More importantly, our promising performance illustrates the effectiveness of incorporating strong mechanical priors for both 2D and 3D labels used. Our approach seems to excel in simulating solids, particularly elastic and rigid bodies. However, the SGNN method outperforms in simulating particulate matter like sand and viscous liquids. This discrepancy may stem from the presence of bond forces in our mechanical framework, constraining particles belonging to the same material.
G.7 Additional Demonstrations of Swapping Materials
Our methodology introduces a novel capability for dynamic material pair substitution within pre- existing simulation environments. By predefining mechanical responses and employing a GNN solely for mapping deformations to interaction forces, we facilitate the modification of material interactions with minimal adjustments. This process involves substituting the parameters of the GNN kernel with those derived from alternative scenarios and altering the input Atj. representing the adjacency matrix or interaction terms.
Figure 13 illustrates this concept by substituting the original fluid-elastic interaction pair with a plastic- elastic pair in the Bear scenario, and a sand-elastic pair in the SandFall scenario, demonstrating the adaptability of our approach to simulate varied mechanical behaviors. Similarly, Figure [4] presents another application of our method, where the sand soil material is replaced with elastic and plastic materials, leveraging GNN kernels trained in distinct contexts. These examples underscore our method's versatility simulating diverse material behaviors through strategic parameter adjustments.
H Visualization of Learned Particle Interaction Forces
In this section, we visualize the contact forces and bond forces to validate the physical meaning and
interoperability of the learnable GNN kernel. After training the DEL, the d", He and Ho in Eq. IO
I2 should correctly map intrusion into the contact and bond force magnitudes. We extracted these GNN kernels from the simulation system separately to evaluate their responses and outputs to different intrusion inputs. For simplicity, we only evaluate the normal direction. The recorded results are visualized in Fig. 17 In this figure, the x x-axis refers to the intrusion value (od in the paper), the y-axis denotes the normalized output of Eq. [T] and 12 The solid line denotes the contact force and the dashed line denotes the bond forces. We represent different materials in various colors. We can see from this figure that for rigid bodies, even very small displacements can result in significant resistance, preventing the object from deforming, For sand and water, since they do not need to
20

---

rigid
contact
plastic bond
elastic band
plastic contac
elastic contact
sand bond
normalized force
fluid band
sand bond
0.6
0.4
0.0
-0.001
0.002
0.006
6d
Figure 17: Visualization of the learned constitutive mapping. The x-axis refers to the intrusion and the y-axis denotes normalized force magnitude.
Figure 18: Visualization of the learned material embeddings on Multi-Obj scenes.
maintain original shapes, our network adaptively learns to set the bond forces to very small values. For plastic materials, network has also learned to break the bond links at appropriate times (the bond forces are close to O). Through this visualization, we would like to claim that our GNN kernel has indeed learned real physical meaning i.e. the forces between particles for simulating different materials.
Visualization of Learned Material Embeddings
Furthermore, following the suggestion of the reviewer, we visualize the learned feature vector, i.e. ha in Eq. 99] by using the t-SNE method, which shown in Fig. 18 In this figure, the red scatter points are projected by the feature of Rigid particles. The green and blue scatter points are produced by the feature of particles belonging two different types of elastic bodies. The features belonging to the same object have clustered together. The reason they do not overlap completely is because their positions relative to the object are also encoded in the A,, As shown the Figure, the red points are closer together, which is because for rigid bodies, regardless of where the points are on the object, they have strong resistance to deformation. We believe this visualization demonstrates that our framework has learned the material-specific features.
21

---

Ground
Truth
Our
Simulation
Figure 19; Failure examples of the proposed method when simulating non-Newtonian fluids.
J More Qualitative Results in both render and particle views
In this section, we report additional qualitative results from Fig. 21 to Fig. 30 These figures include both particle view and render view of our method, EGNN", VPD, and 3DIntphys (only available for particle views). More detailed results can be seen in our supplementary video. For the EGNN" and our method, we demonstrate both render and particle views faithfully compare the predicted dynamics.
be observed from these figures that our method can generalize well different initial shapes and conditions, especially when predicting long-term dynamics. While the VPD and EGNN" cannot precisely predict the mechanical behaviors between materials. Or they only produce plausible results on some certain materials. For instance, in some cases, such as BEAR and FluidR, the EGNN can deliver plausible results the early stage interactions, but the performance degrades drastically with the progressing deformation.
We in addition show comparison of baselines for the Fluids dataset in Figure 20 preserves the basic trend of water flow, which other methods do not.
Our method
K Failure Cases
In this section, discuss failure cases of the proposed method. We observe that this method struggles to simulate non-Newtonian fluids because fundamental assumption of this method is that the material properties are consistent throughout the training dataset. However, the properties of non-Newtonian fluids change with stress variation. Here we show an example in Fig. [19 If we use our method to simulate oon-Newtonian fluid, a large gap between the prediction and groundtruth can be observed.
This method is also suited simulate smoking. First, smoke consists of extremely small water vapor particles, with the particle size being very small and the number of particles being vast. Simulating smoke by particles requires significant computational resources. Second, smoke simulation not only involves the exchange of momentum between particles but also is governed by thermodynamics, making the simulation smoke with particles complex topic.
22

---

BENNN
SGNNN
Neurofld
Figure 20: Qualitative Comparisons of all baselines on Fluids dataset both rendering and particle views.
3Dintphy
EGNN
VPD
Ours
Gr
Figure 21: Long-term predictions of 3DIntphys, VPD, EGNN and our method on Plasticine scenarios in particle view.
23

---

EGNN
VPD
Outs
GT
Figure 22: Long-term predictions of VPD, EGNN and our method on Plasticine scenarios in render view.
DDinppays
EGNN
VPD
Ours
Figure 23: Long-term predictions of 3DIntphss, VPD, EGNN and our method Multi-objs scenarios in particle view.

---

EGNN
VPD
Outs
Figure 24: Long-ten term predictions VPD, EGNN and our method on Multi-objs scenarios in render view.
3DInkphs
EGNN
VPD
Ouns
Figure 25: Long-term predictions of 3DInthhys, VPD, EGNN and method on Bear scenarios in particle view.
25

---

BGNN
VPD
OuIs
CT
Figure 26: Long-term predictions of VPD, EGNN and method Bear scenarios in render view.
EGNN
VPD
Ows
GT
Figure 27: Long-term predictions of 3DIntphys, VPD, EGNN and our method FluidR scenarios in particle view.
26

---

EGNN
VPD
Ours
GT
Figure 28: Long-term predictions of VPD, EGNN and our method on FluidR scenarios in render view.
Diinppyss
EGNN
VPD
OuuSS
GT
Figure 29: Long-term predictions of 3DInthys,, VPD, EGNN and method on SandFall scenarios in particle view.
27

---

BGNN
VPD
Ous
Figure 30: Long-term predictions of VPD, EGNN and our method on SandFall scenarios in render view.
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We briefly and accurately conclude the contribution and scope of our proposed paper in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discussed the limitations of proposed method in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well- well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
28

---

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: This paper provides the full set of assumptions and complete proof and
introduction in Section 3 and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: this paper. we give a detailed introduction about how we construct the dataset. The proposed method, training, and testing paradigm, and data collection method are completely reproducible and easy follow.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may
29

---

be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code and dataset can be obtained by emailing the author only for research collaborations. One can also easily reproduce the proposed physics-informed GNN on the guidance of the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
Sublic/guides /oooe//ummiiiiioooooooooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
/ wpips.cc/public/eeeecc http://///uiiee///ddeeeabbaaaaas
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
30

---

Answer: [Yes]
Justification: We give implementation details, comparative experiments, and ablation studies in Section and our Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We give qualitative and quantitative experimental results to prove our proposed method outperforms existing dynamic learners across multiple scenarios in Section 4 and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on computing resources is concluded in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
31

---

9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http://s///arppp...cooa/aaaa/ata
Answer: [Yes]
Justification: Our proposed methods, code, datasets, and experiment paradigms fully comply with NeurIPS' Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential impact of this work in the Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for the responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: In this paper, we do not utilize datasets with high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
32

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators and original owners of assets used in the paper are all credited and the license and terms of use are explicitly mentioned and properly respected.
Justification:
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the batasets.paperssit paperswi thcode. com datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not contain crowdsourcing experiments and research with human subjects.
Guidelines:
33

---

The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
34


Reviews:
[{"review_id": "GEmeVfEozM", "reviewer": "Reviewer_LfJV", "summary": "The paper considers the problem of physical modeling the dynamics of 3D objects in space using only 2D observations. The authors propose to solve this problem by viewing objects as sets of interacting points and using differentiable rendering of these point clouds. In this pipeline, neural models are used to directly predict the dynamics of 3D particles. Unlike existing methods, the authors propose to use a physically principled framework (Discrete Element Analysis) to develop a partially interpretable neural model that can be used to predict the target dynamics. This framework decomposes the forces applied to the points into gravity, potential, and viscous interaction forces (and further into normal and tangent components). The proposed model thus is constrained to predict certain components from these force decompositions resulting in a more principled, interpretable, and effective dynamics prediction.\n\nThe authors propose new more challenging synthetic datasets containing variable materials and objects, evaluate on this data, and compare the results to several existing baselines. The proposed method shows improvements over the baselines in all the scenarios. The authors also provide an ablation study showing the importance of different components.", "questions": "How are CD and EMD computed? Are any of the intermediate states used to compute the metrics or are they only computed for the final state? If the proposed model is interpretable does it make sense to compare the predicted particle forces in all the intermediate steps with the ground truth forces from the simulations?\n\nSome examples show that objects can be «torn» over the course of the considered scenario. Currently, the resulting distinct parts of the torn object will still be considered as one single object by the model, is not it? Does it cause problems? Have you considered any solutions for it?", "limitations": "The limitations are properly addressed in the text.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "* The proposed model seems like an effective combination of a physically principled framework and neural model, which potentially can be further explored and improved.\n* The proposed method significantly outperforms all baselines.\n* The authors propose a new dataset.", "weaknesses": "* The written text requires some additional polishing (citations should not be treated as nouns, sometimes the layout of figures and text is too tight, and there are some grammatical errors and not correctly formulated sentences).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness: Polishing Language**\n\nWe thank the reviewer for pointing out this. Following this comment, we have thoroughly revised the language and grammar to enhance the overall quality of the text, carefully checked the citation, and optimized the layout. The updated version is fully improved for the academic language.\n\n> **Question 1: Metrics Calculation**\n\nAs stated by the reviewer, our goal is to predict a sequence of particle trajectories, therefore there are lots of intermediate states. We would claim that **all the metrics** in this work, including *CD, EMD, PSNR, SSIM, and LPIPS*, are derived by averaging over all timestamps of all sequences in the test set. Hence the evaluated metrics contain **all intermediate states and all sequences**. We will clarify this point in *Section 4* of our revised manuscript following this comment. \n\n> **Question 2: Interpretable**\n\nThanks for this comment. Yes, the proposed method is partially interpretable because we predict the interaction forces between particles by our physical neural operators. The predicted force is directly applied to update the velocity and position of particles by Euler Integration. Therefore, the next accurate positions can **only be obtained by correct force prediction**. However, we do not have the groundtruth of the particle interaction forces. The reason is that all the synthetic data are generated by the Material Point Method (MPM). In MPM, the interaction forces are not explicitly and directly derived. Instead, it simulates the dynamics by transferring the momentum between particles and background grids. Therefore, it's hard to obtain the groundtruth of the interaction forces. \n\nNevertheless, we conducted additional analysis to validate **the physical meaning of the forces** predicted by our DEL. In *Section H and Figure 17 in the Appendix*, we visualize the **relationship between the predicted forces and particle deformations** for each type of material by accessing the learned graph operators with different inputs. These curves indeed reveal the inherent mechanical properties of these different materials. For example, for plastic materials, the force initially increases with displacement. However, when the deformation reaches a certain extent, the material undergoes yielding, and the force no longer increases with further displacement or even decreases. This phenomenon closely mirrors real-world plastic materials. Moreover, for the rigid body, even very small deformations can lead to a significant increase in force, which is also in agreement with real-world situations.\n\n> **Question 3: Torning Objects**\n\nIn fact, the proposed method recognizes particles in terms of the **material level** instead of the object level. If an object is torn apart, **the material of these parts remains unchanged**, therefore they have the same mechanical properties. For example, a piece of plastic, even when torn into two halves, remains plastic. This phenomenon looks like the DEL recognizes them as the same objects, but as we stated above, these particles just belong to the same material with constant properties, such as mechanical parameters, colors, etc. Hence DEL does not meet these object-level issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0QsotGVexU", "reviewer": "Reviewer_qBCD", "summary": "The paper proposes a framework to learn 3D dynamics from 2D observations. DEM uses hand-designed kernel functions to model interaction force between particles, which may vary significantly for different material types. The paper changes these kernels to learnable GNN kernels. The time integration still follows DEM. Combined with a differentiable particle renderer, these kernels can be trained to match 2D observations.", "questions": "[Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics](https://arxiv.org/abs/2304.14369) should be cited, since it solves a similar task.\n\nAre the material types in particle attribute A pre-known? Are the initial particle distributions known? The known assumptions should be listed in the paper.\n\nHow are particle colors assigned? In reality, objects usually have textures. Manually assigning colors may be impractical. \n\nThe particle initialization step is illustrated in Fig. 1, but no technical details are discussed in the text. It seems that the initial geometry is provided in the experiments. In that case, the known-geometry assumption should be illustrated in the figure instead. However, geometry reconstruction is important for real data applications.", "limitations": "Limitations are discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The framework is quite general, with force-based time integration providing a strong physics prior. The use of learnable kernels to model point interactions enables the DEM to effectively fit observations.\n\nNumerous comparisons are conducted, and the results are promising.", "weaknesses": "The evaluations are conducted only on synthetic data.\n\nThe learned particle interactions are black-boxes, which are not explainable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness 1: The evaluations are conducted only on synthetic data.**\n\nThanks for this comment. At present, the proposed method is indeed only evaluated on synthetic data. This is because capturing multiview videos for a dynamic process in the real world is difficult. Because it is hard to **simultaneously capture multiple dynamic videos** of the same object. Additionally, some objects, such as plastic materials, may become damaged via collision when shooting the first video. So it is **hard to be repeated**. We have also pointed out this issue in *the Limitation section* of our original paper.\n\nIn the future, we are going to investigate potential solutions for that. First, transferring the model trained on synthetic data to real-world situations might be possible. Second, we are going to explore few-shot or one-shot learning to learn dynamics with a single real-world video, which might be achieved by integrating more explicit physics and geometry priors into learning-based frameworks.\n\n> **Weakness 2: The learned particle interactions are black-boxes, which are not explainable.**\n\nThe DEL is proposed by **integrating graph neural operators into a classic mechanics analysis framework**, we intentionally make these operators to approximate the mapping from particle deformation to particle interaction forces. Therefore, we would like to claim that this approach is **partially interpretable**. We list the reasons as follows: \n\n**First**, we directly adopt the predicted particle interaction forces to update the velocities and positions of all particles, and achieve accurate results, which can indicate **the predicted forces are correct** since **incorrect force cannot result in precise deformations** under the framework of Euler integration.\n\n**Second**, since we employed the framework of the Discrete Element Method (DEM), it naturally satisfies **the conservation of momentum and energy**. In addition, given our assumption that particle mass remains constant and particles neither vanish nor are created spontaneously, our method also adheres to the law of **conservation of mass**.\n\n**Third**, we conduct additional analysis to evaluate the traits of these neural operators in *Section H and Figure 17* in the Appendix. We visually **study how the predicted forces relate to the particle deformation** of different materials. The relationships are implicitly learned by our DEL. We observe that the **force-deformation curves indeed reflect the real-world properties** of these materials. \nThere are two examples to illustrate. **(1)** With plastic materials, the force grows as they stretch at the beginning. But after stretching to a certain extent, they begin to yield, and more displacement would not increase forces further. This behavior is truly like real plastic materials. **(2)** For rigid solid materials, even tiny shape changes can cause extremely large force to maintain their initial shapes, which also matches what we see in the real world.\n\n> **Question 1: Cite a relevant paper**\n\nWe appreciate this reviewer for bringing such a relevant paper to us. We cite the paper in a proper place in our revised paper.\n\n> **Question 2 and 4: Known Assumptions and Particle Initialization**\n\nWe would like to present that both the material types and initial particle distribution are also **unknown** in our DEL framework. In *particle attribute A*, we only tell the DEL which particles belong to the same material, but we **do not provide the certain material type** for them. Only the image sequences and their corresponding camera poses are needed to train the model. The material types, including their mechanical properties, can be **implicitly encoded into the graph neural operators** during training. \n\nThe initial particle distribution can be obtained by using the Generalizable Point Field (**GPF**) [1] in this work, which is a differentiable PointNeRF-like 3D reconstruction method. We introduce this step in *Section 3.1*. We would like to claim that our method can be seamlessly combined with other point-based renderers, as long as they can **(1)** render the deformed scene according to the point movement and **(2)** propagate the gradients. The simulation starts only after the initialization of the particles is finished. Moreover, we evaluate our method on the other two different differentiable point-based renderers, i.e. ParticleNeRF and PhysNeRF in Section G2 and Figures 10 and 11 in the Appendix, the results show that our DEL can also **deliver the best performance no matter what renderers are used**. \n\nFollowing the reviewer's suggestion, we add detailed descriptions of the **assumption and initialization** of particles in *the first paragraph of Section 3* in our revised paper. Furthermore, to improve the readability, we add some basic background knowledge of the GPF in our revised Appendix. \n\n*[1] Wang J, et al.. Learning robust generalizable radiance field with visibility and feature augmented point representation. International Conference of Learning Representations (ICLR) 2024*\n\n> **Question 3: Color and Texture**\n\nAs we stated in the answer to Question 1, the colors of these particles are faithfully reconstructed from images by using GPF. These colors and textures would remain unchanged during the simulation process. We agree with the review that the textures in reality could be various and complicated. However, we would argue that **complex textures are more beneficial for learning dynamics because they better reveal the direction of deformation at specific points on an object's surface due to the color gradients in the image**. This helps in reducing uncertainty. However the uniform color does not have large gradients. Conversely, textures of an object with uniform color make it difficult to ascertain the exact direction of deformation at a point on the surface, because there are no color gradients around that point.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AcafrIM0hn", "reviewer": "Reviewer_fjMr", "summary": "This paper proposes to incorporate the neural network with Discrete Element Analysis framework for particle-based simulation. The method adopts GPF as differentiable renderer and is trained through multi-view videos. The proposed model delivers faithful rollouts and outperforms the baselines.", "questions": "1. To enable the training on different scenes through the images, are different pertained GPF needed for different scenes?\n2. How is the performance of the method for long-term predictions? For example, a trajectory with 150 frames.\n3. At L253, the author claims to use L2 loss. However, for equation 17, it seems the loss is L1. Is this a typo?", "limitations": "Please refer to the weaknesses and limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "* The proposed method deeply integrates the DEA and obtains robust performance.\n* The model can be trained through multi-view videos in an end-to-end manner.", "weaknesses": "1. To simulate different scenes, such as deformable objects and fluid, it seems that one has to train different models on different cases, indicating scene-specific design with limited generalisation abilities.\n2. The results of verifying the generalisation abilities are missing. For example, after trained on the deformable bunny as shown in Figure 5, can the model simulate objects of other shapes, such as a car, or objects composed of more particles?\n3. The method seems to be capable of dealing with objects with limited particles and struggle to simulate large amount of particles.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness 1: Scene Specific**\n\nThanks for this comment. The proposed approach indeed needs to be trained for modeling different materials. In fact, we prefer to describe it as **material-specific** training rather than scene-specific training. This is because DEL can implicitly encode the mechanical properties of a certain material into the graph neural operator during training, and after training, the trained graph neural operator can be directly inserted **into any unseen scenes with different initialization conditions**, such as shapes and velocities. \n\nWe also conducted additional experiments and analysis for this point in *Section E, Figure 7, Figure, 13, and Figure 14* in Appendix. It can be observed that various materials can be simulated by **simply exchanging the graph operators trained in different scenes**. We believe that this also shows the generalization ability of the DEL. An ideal situation to use this method is that one can train the graph kernel for a specific material in a certain dataset in advance, and then invoke it when one needs to simulate the material. \n\n> **Weakness 2: Generalization Ability**\n\nWe would like to argue that all visualized results in our paper are from the **test set**, including the bunny in Figure 5 and the duck in Figure 22. Both of the initial shapes have never been seen by the model during training. As we stated in the *Dataset* part in Section 4, for each specific material pair, for example, the plastic and the rigid pairs in the **Plasticine** scenario (Figure 5), there are 128 different training sequences and 12 testing sequences. All of them have different initial shapes, velocities, and locations. We would claim that **the \"Plasticine\" scenario does not refer to the specific \"bunny\" or \"duck\", it refers to the same plastic material that makes up them**. \n\n For example, in the training set, the plasticine might be initialized as a \"sphere\" or a \"dog\", but in the test set, it can be a \"bunny\" or a \"duck\". Therefore, we could claim that our model can be generalized well to different initial conditions that have never appeared in the training phase as long as the same type of material is simulated. Also, the dataset and experimental setups are identical across all models discussed in the paper. To conclude, **our DEL can effectively generalize to unobserved scenarios with different initial conditions** when simulating the same materials.\n\nFurthermore, in *Figure 8, 9 and Section G.3, G.4* in the Appendix, we evaluate the training efficiency of our model. The results show that even when trained with a limited amount of data, such as **1/8** training dataset, our model still achieves impressive performance. In contrast, other baselines that do not adopt physical priors exhibit a rapid decline in performance as the training data is reduced. This also indicates the generalization ability of our DEL. \n\n> **Weakness 3: The Large Number of Particles**\n\nThanks for this comment. We indeed have not conducted experiments in particularly large scenes because representing large scenes with particles requires a substantial amount of VRAM. However, unlike other methods that model the dynamics across the entire spatial domain, our approach models only the interactions between particles, analyzing each pair individually. This way significantly **reduces the computational overhead** when simulating relatively large scenes, and is **scale-independent**. Due to its scale-invariant properties, our method should outperform others when simulating large scenes. In the future, we plan to explore hierarchical simulation techniques to reduce memory consumption, thereby improving the efficiency of simulating larger scenes. \n\n> **Question 1: GPF related**\n\nThanks for this question. The GPF does not need to be retrained for different scenes, because GPF is a **generalizable point-based NeRF** method, it has already been pretrained on large 3D reconstruction datasets and can directly convert multiview images into a point-based NeRF representation without backpropagation. Moreover, we slightly fine-tuned the GPF on our training set based on their provided checkpoints for better reconstruction quality. \n\nTo improve the readability of this paper, we introduce the above basic preliminaries about GPF in *the first paragraph of Section 3* following this comment. Moreover, we add detailed descriptions of the point-based renderer that we use in our revised Appendix. \n\nBesides, we also evaluate the performance of our method when different point-based renderers are adopted in *Section G2, Figures 10 and 11* in the Appendix. The results illustrate that DEL is robust to different renderers. As long as the renderer meets the following requirements: **1.** Point-based: can change the content with the movement of points. **2.** Differentiable: can propagate gradient, it can be used to train DEL. \n\n> **Question 2: Long-term prediction**\n\nThe average timestamps for each scenario in the test set can be seen in the following Table:\n\n| Scenario | plasticine | SandFall | Multi-Obj | FluidR | Bear | Fluids |\n|-|-|-|-|-|-|-|\n|Avg Step Number | 86 | 94 | 132 | 155 | 124 | 138 |\n\nAmong them, the Plasticine and SandFall predict relatively short-term dynamics, while **the Multi-Obj, FluidR, and Fluids provide the results for long-term predictions** whose timestamps are approximately **from 130 to 160**. In addition, all metrics reported in this paper including *CD, EMD, PSNR, SSIM, and LPIPS* are averaged across all timestamps of all sequences in the test set. This indicates that our method can perform better than these counterparts in both long-term and short-term predictions. We are also going to explore more super-long examples of these learned simulators in the follow-up plan. \n\n> **Question 3: A typo in the definition of loss function**\n\nWe thank the reviewer's careful check very much. This is a typo in *Line 253*, the loss should be L1 loss and we have corrected it in our revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Vn5w5Irw6w", "reviewer": "Reviewer_uaqu", "summary": "This paper presents a new method to learn 3D particle dynamics from sparse 2D observations using inverse rendering.\nIn constrast to previous work, it does not learn a fully unconstrained model but makes use of known physical priors.\nThe method learns graph network kernels to model the particle interation forces in the DEA framework, which are classically designed domain experts.\nThese graph networks are trained from 2D observations using a differentiable renderer without any 3D supervision.\nThe method is evaluated on different scenes and using different physical materials.\nIt surpasses the provided baseline comparisons in the test shown in the paper.", "questions": "- 3.3: the $v_{ij}^{n}$ or $v_{ij}^{t}$ used in equations 14 to 16 do not match any of the definitions in appendix B.\n- For training (3.4), how is the scene initialization done with the renderer? And how accurate is the estimated initial velocity based on this initialization? \n- Closely related: how sensitive is your method to inaccuracies in the initial velocity?\n- How long are the sequences used in your experiments?\n- How long is your training (number of iterations, epochs, or variable updates)?\n- Are the views shown in the figures the ones that were used for training or are they novel views?\n\nThere are several grammatical errors throughout the paper, for example:\n- 291 \"metrix\"\n- 291 \"cham**b**er distance\" should be \"cham**f**er distance\"\n- 306 There seems to be a new paragraph missing before \"Results in particle view\"\n- 311 \"we claim the reason that...\"\n- 343 \"while keep the priors regulated...\"\nI would encourage the authors to do a complete pass over the paper to fix any writing mistakes.\nThe tables and figure on page 9 are also very squeezed.", "limitations": "Limitations are briefly discussed.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper presents a novel combination of graph networks and classical particle-based simulation that, in conjunction with a differentiable renderer, allows it to recover 3D particle dynamics from images.\n\nThe presented method is technically sound, well explained, and easy to follow.\nThe evaluation is sensible and very extensive, encompassing different material configurations, and shows a siginficant improvement over the SoTA.\n\nThe source code and dataset will be made available to the public.", "weaknesses": "There are several things that are not clear to me, especially related to scene initialization and the initial velocities. These points are detailed in the questions below. I don't think these are fundamental problems, but I nonetheless hope the authors can clarify these aspects in the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **Weakness and Question 2 and 3: Scene and Velocity Initialization**\n\nThanks for the reviewer pointing out these ambiguities. We explain these details in the following and will add them to our revised manuscript for better readability. \n\n**Scene Initialization.** As we stated in the first paragraph of *Section 3*, we adopt the Generalizable Point Field (**GPF**) [1] to initialize the scene. GPF is a point-based NeRF-like approach, which can directly convert multiview images into a point-based NeRF representation in a generalizable way because GPF is pretrained on large 3D reconstruction datasets. In detail, GPF first projects 2D images into 3D points by predicting their depth maps. Second, it hierarchically aggregates features from images to the point scaffold to obtain separate appearance and geometry features. GPF can render changeable content by moving these featured points. We slightly fine-tuned the GPF on our training set based on the authors' provided checkpoints for better reconstruction. \n\nMoreover, we also evaluate some other point-based renderers in *Section G2, Figures 10 and 11* in the Appendix, the results illustrate that **DEL is also robust to these renderers**. We would like to claim that all point-based renderers can be adopted to train DEL as long as **(1)** they can render different content with the point movement **(2)** they are fully differentiable to propagate gradients. \n\n**Velocities Initialization.** We follow the methods in PAC-NeRF [2] and PhysGaussian [3] to use images by using the first three frames to optimize the initial velocities. We assume that the initial velocities for all particles of the moving objects are identical. Hence, we only need to run the following loop:\n1. optimize the velocity, \n2. move the particle set according to it, \n3. render novel views based on the new position of the particles \n4. compute the loss between the rendered views and the real views\n5. update the velocity to align the renderings with the real views\n\nAfter the error is smaller than a predefined threshold, the loop is broken. \nFollowing the reviewer's advice, we additionally evaluate the effect of the approximated velocities. We train two models on the Plasticine scene using estimated velocities and another using actual velocities. Then, we test each model on two test sets, one provides the real velocities and another does not. The results are shown below:\n\n| Test/Train | w | w/o |\n|-------|-------|-------|\n| w | 7.37 | 7.48 |\n| w/o | 7.82 | 7.54 |\n\nThis table reports the Mean Rollout Chamfer Distance for each situation. \"w\" refers to \"with real velocities\" and \"w/o\" refers to \"without velocities\". \nFrom the Table, we can see that the results do not exhibit significant fluctuations, which may be due to the simplistic nature of the initial velocity estimations. How to estimate initial velocities in a more complex environment remains a worthy topic for future research.\n\n*[1] Wang J, et al.. Learning robust generalizable radiance field with visibility and feature augmented point representation. International Conference of Learning Representations (ICLR) 2024*\n\n*[2]Li X, Qiao Y L, Chen P Y, et al. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnostic system identification.  International Conference of Learning Representations (ICLR) 2023*\n\n*[3]Xie T, Zong Z, Qiu Y, et al. Physgaussian: Physics-integrated 3d gaussians for generative dynamics.  Conference on Computer Vision and Pattern Recognition. (CVPR) 2024*\n\n> **Question 1: Notation**\n\nWe thank the reviewer for the careful check very much. The $v_{ij}^n$ in Equation 14 represents the relative velocity of particle i with respect to j, actually, the velocity could be a vector, so it should be modified as  **v**$_{ij}^n$ . (bond font represents a vector)\n\nSimilarly, the $v_{ij}^n$ which denotes the tangential velocity should be modified as **v**$_{ij}^t$. We thoroughly check all notations in the paper and correct them all, and we update the Nomenclature in Appendix B accordingly. \n\n> **Question 4: The length of the sequences**\n\nFor different simulation scenarios we have different sequence lengths. We report the simulation steps (i.e. the sequence lengths), which are averaged over the entire test set for each scenario, in the following Table.\n\n| Scenario | plasticine | SandFall | Multi-Obj | FluidR | Bear | Fluids |\n|-------|-------|-------|-------|-------|-------|-------|\n|Avg Step Number | 86 | 94 | 132 | 155 | 124 | 138 |\n\nTo better clarify this, we add this description to Section F in our revised manuscript. \n\n> **Question 5: Training time**\n\nThanks for this comment. As we stated in the answer of Weakness, we first finetuned the GPF by using their default learning rate on our training set for 500 iterations, this step only takes 12 minutes. Then we train our model on a certain scenario, for example the Plasticine,  for about 24000 iterations on a single NVIDIA RTX3090, and the loss has been steady.  The training time for a certain scenario is about 3.6~4 hours. We add the above training information to the *Section D* in our revised paper for the convenience of readers. \n\n> **Question 6: Test views**\n\nThanks for this question. We would like to clarify that all camera views and initial conditions in the test set have never been seen in the training set. Because we believe that this should be better for evaluating the generalization ability of these models. We add this illustration in the first paragraph of Section 4 in our revised paper for better readability. \n\n> **Question 7: grammar errors and layout**\n\nWe appreciate a lot the reviewer for pointing out these typos and errors. Following these suggestions, we correct them in the revised version. Moreover, we carefully review and improve the text's language and grammar, and optimize the layout to boost the quality of the paper. Now the revised version is more readable and clearer.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers are generally positive on the paper.  Some questions were raised around generalization, evaluation, and scaling and some minor presentation issues were presented.  Overall, these questions have been resolved in the rebuttal and discussion phase and the presentation issues are minor and able to be resolved in the final version.

As such, the AC recommends acceptance as a poster and encourages the authors to carefully revise the paper based on the feedback from the review process.
Decision: Accept (poster)