DEL: Discrete Element Learner for Learning 3D Particle Dynamics with Neural Rendering

Authors: jiaxu  wang
jingkai  sun
junhao  he
ziyi  zhang
qiang  zhang
mingyuan  sun
renjing  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 09:16:10

Abstract: Learning-based simulators show great potential for simulating particle dynamics when 3D groundtruth is available, but per-particle correspondences are not always accessible. The development of neural rendering presents a new solution to this field to learn 3D dynamics from 2D images by inverse rendering. 
However, existing approaches still suffer from ill-posed natures resulting from the 2D to 3D uncertainty, for example, specific 2D images can correspond with various 3D particle distributions. To mitigate such uncertainty, we consider a conventional, mechanically interpretable framework as the physical priors and extend it to a learning-based version. In brief, we incorporate the learnable graph kernels into the classic Discrete Element Analysis (DEA) framework to implement a novel mechanics-informed network architecture. In this case, the graph networks are only used for approximating some specific mechanical operators in the DEA framework rather than the whole dynamics mapping. By integrating the strong physics priors, our methods can effectively learn the dynamics of various materials from the partial 2D observations in a unified manner. Experiments show that our approach outperforms other learned simulators by a large margin in this context and is robust to different renderers, fewer training samples, and fewer camera views.

Main paper: 
 introduction: 
 Simulating complex physical dynamics and interactions of different materials is crucial in areas including graphics, robotics, and mechanical analysis. While conventional numerical tools offer plausible predictions, they are computationally expensive and need extra user inputs like material specifications. In contrast, learning-based simulators have recently garnered significant attention as they offer more efficient solutions. Previous works primarily simulate object dynamics in 2D. They either treat pixels as grids [1] or map the images into low latent space [2, 3] and predict future latent states. However, these 2D-centric approaches possess limitations. The world is inherently 3D, and 2D methods struggle to reason about physical processes because they rely on view-dependent features, and are hard to understand real object geometries. To address these, researchers have incorporated multi-view 3D perceptions into simulations such as Neural Radiance Field (NeRF) [4] which is an implicit 3D-aware representation. Some studies extract view-invariant representations and 3D structured priors by NeRFs to learn 3D-aware dynamics [5, 6] . They either represent the whole scene as a single vector or learn compositional object features by foreground masks. However, they require heavy computational demands and struggle with objects with high degrees of freedom.Particle-based learned simulators show impressive results in modeling 3D dynamics. The success is mainly attributed to the popularity of Graph Neural Networks (GNNs). In general, objects are represented as particles that are regarded as nodes in graphs, and their interactions are modeled by edges. Previous studies [7, 8] adapt GNN to predict particle tracks and achieve good results. Recent research has made strides in improving GNN simulators [9, 10] . They require particle correspondences across times for training. However, 3D positions of particles across time are not always accessible and learning dynamics solely from visual input is still a big challenge. The reasons can be summarized as follows. Determining particle positions from 2D observations leads to uncertainty since different particle distributions can produce similar 2D images. Moreover, previous GNN-based simulators aim to learn how to infer the entire dynamics, which are fully uninterpretable and result in hard optimization. Several studies [11, 12] reconstruct 3D from 2D images and then learn from it, but they are not directly trained end-to-end with pixel supervision. One feasible way is to employ inverse rendering. For example, [13, 14] use NeRF-based inverse rendering to learn dynamics from 2D images. However, they are either constrained to simulate specific material or incapable of dealing with the 2D-3D ambiguities, thereby damaging their generalization ability. Furthermore, existing approaches only evaluate their methods on simple datasets. Their synthetic dataset often contained a limited variety of materials (usually rigid bodies), rarely involved collisions between objects, and featured very regular initial shapes.To address the above challenges, this work incorporates strong mechanical constraints in the learningbased simulation system to effectively learn the 3D particle dynamics from 2D observations. Discrete Element Analysis (DEA) [15] , also known as Discrete Element Method [16] , is a traditional numerical method to simulate particle dynamics in mechanical analysis. This method computes interaction forces between particles to predict how the entire assembly of particles behaves over time. However, traditional DEA heavily relies on the user-predefined mechanical relations between particles, such as constitutive mapping or dissipation modeling, which often involve several material-specific hyperparameters. Moreover, the results often deviate from the actual mechanical responses because the constitutive equations are overly idealized.In this work, we combine GNNs with the DEA theory to implement a physics-integrated neural simulator, called the Discrete Element Learner (DEL), aiming to enhance the robustness, generalization, and interoperability of GNN architectures. In detail, we use GNNs as kernels to learn the mechanical operators in DEA rather than adopting user-define equations. On the other hand, the graph networks only need to fit some specific physical equations in the DEA framework instead of learning the whole evolution of dynamics, which largely reduces the optimization difficulties. Therefore, combining the conventional mechanical framework and GNN can reduce the 2D-to-3D uncertainty and alleviate the ill-posed nature. The main contributions of this work can be summarized as follows:• We propose a novel physics-integrated neural simulation system called DEL which incorporates graph networks into the conventional Discrete Element Analysis framework to effectively learn the 3D particle dynamics from 2D observations in a physically constrained and interpretable manner.• We design the network architecture under the guidance of the DEA theory. In detail, we use learnable GNN kernels to only fit several specific mechanical operators in the classic DEA framework, instead of learning the entire dynamics to make GNNs and the DEA mutually benefit from each other, significantly reducing the ill-posed nature of this task.• We evaluate our method on synthetic datasets that contains various materials and complex initial shapes compared to existing ones. Extensive experiments show that our method surpasses all previous ones in terms of robustness and performance.2 Related Work 
gnn-based particle dynamics simulator: 
 There has been many works [17, 18, 9, 19, 20, 21] to develop GNN-based particle simulators to predict 3D dynamical systems. This is because representing 3D scenes as particles perfectly matches the graph structure via particles as nodes and interactions as edges. GNS [7] shows plausible simulations on multiple materials by multi-step message passing. DPI [8] adds one level of hierarchy to the rigid and predicts the rigid transformation via generalized coordinates. EGNN [17] maintains the equivariance of graphs by passing scalar and vector messages separately, and explicitly assumes the direction of vector message passing along with the edges. SGNN [9] proposes the subequivariant simulator, which has a strong generalization to long-term predictions. Most of them are blackbox models and non-interpretable, thus complicating the optimization. There are some works incorporating basic physical priors into neural networks [22, 23, 24, 25, 26] , whereas they perform well either on toy examples or specific topologies such as rigid hinges. All the above-mentioned learned simulators require full 3D particle tracks as labels for training. They cannot learn from pixel-level supervision because of the large solution spaces caused by the 2D-to-3D uncertainties, which we experimentally proved in Section 4. Our approach reduces ambiguities by integrating GNNs into a mechanical analysis framework. Our method not only yields impressive results supervised by 3D labels but also effectively learns realistic physics under 2D supervision.Learning dynamics from merely visual observations is vital for many domains. Previous works [1, 3] map images into low dimensional space and learn dynamic models to infer the evolution of latent vectors. However, the general latent approach [27, 28, 2, 29, 30, 31, 32, 33] makes things like pixel-level video prediction rather than real physical inference [34] . The biggest reason is the gap between 2D observations and 3D worlds [10] . To address this challenge, recent works consider 3D-invariant representation to build latent states. NeRF is used by [5] to encode multiview images as view-independent features. But it serves the whole scene as a single vector, and cannot handle scenes with multiple objects. [6] encodes compositional multi-object environments into implicit neural scatter functions, while it only handles rigid objects. Similarly, [33] and [35] use compositional implicit representation, but cannot simulate objects with large deformations. Some other methods [36, 37, 38] need additional signals, such as Lidar data. The 3D-aware latent dynamics also lack generalizability to unobserved scenarios and cannot work with complex topologies and varying materials. Moreover, latent dynamics models are fully non-interpretable.Very recently, with the development of differentiable neural rendering, a few studies have attempted to train 3D dynamic models from visions via inverse rendering [13, 14, 39] . They bridge images and 3D scenes with a differentiable renderer to minimize the renderings and groundtruth. However, [13] and [14] can only simulate fluids because they require fluid properties as input. VPD [39] learns 3D particle dynamics directly from images and can simulate various solid materials. However, it requires jointly training its own particle renderer and latent simulator, which leads to a black-box nature and is hard to be adopted by other renderers. Conversely, our DEL can seamlessly integrate into any point-based renderers with satisfactory performance and is physically interpretable as well as can simulate various materials in a unified manner. 
preliminaries and problem statement: 
 This task is formulated as inferring particle dynamics via inverse neural rendering. Similar to other inverse graphics, the scene can be represented by 3D primitives, and then the dynamical module infers the future state of these primitives. Once this future state is inferred, it can be effectively transformed into visual images by neural renderers. The dynamical module can be trained from the error between the renderings and observations. Figure 1 illustrates the general paradigm. In this formula, 3D scenes should be represented as particles, and then, the renderer should be able to render particles into images with a given camera viewpoint. According to the above discussion, we choose the Generalizable Point Field [40] (GPF) as our renderer, which can represent a 3D scene as particles, change its content by moving particles, and render images with arbitrary views. Notably, other particle-based renderers such as [41, 13] or recently prevailing [42] , can also be used arbitrarily as long as they are fully differentiable and represent objects as particles. The renderer module can iteratively produce updated images after the dynamic module moves particles at each timestamp.The dynamical module operates as a graph network simulator. Consider a physical system with N particles to represent M objects, the simulator models its dynamics by mapping the current state to consequent future states, usually the positions of particles. Assume X t i ∈ X are particle states at time t, X t i usually includes the coordinate x t i , the velocity v t i and particle's intrinsic attributes A i such as the material type and mass. The learnable GNN simulator S considers particles as nodes and dynamically constructs connections at independent time steps when the distance between two particles is smaller than a threshold (E = {i, j : ||x i -x j || 2 <= r}). The GNN maps all the information at the current state to the positions at the next timestamp by passing messages on the graph, i.e.x t+1 i = S θ (x t i , v t i , a i , E).Different GNN simulators mainly lie in the different designs of message-passing networks. As we claim in Section 1, the S θ s in most previous approaches aim to learn the entire dynamics process, which leads to hard optimization and the risk of overfitting. Moreover, they are non-interpretable black boxes. Therefore, the learning target would be very ill-posed because the solution space is very large when only visual supervisions are given.We propose a mechanics-encoded architecture that combines the GNN with the typical DEA to reduce uncertainty and improve interpretability. In the following section, we first introduce the general DEA method and its potential to be enhanced by the learning-based kernels. Second, we present how we incorporate the graph networks into DEA to replace the traditional operators.In this section, we introduce the general framework of Discrete Element Analysis, also known as the Discrete Element Method, and its drawbacks which potentially can be enhanced by our learnable kernels. Here we only cover the general knowledge that we need to design our architecture, we recommend readers refer to [15, 43, 16] for deeper knowledge of DEA. In the framework, the whole scene is represented as particles and the DEA is used to simulate the behavior and interactions of these particles. Generally, in this framework, the movement of an individual particle is governed by the Newton-Euler motion equation:EQUATION)where u is the movement vector, m i is the mass of particle i. F g i refers to the gravity. F p ij and F v ij are the interaction forces between particle i and j, the former marks potential interaction force, and the latter marks dissipative (viscous) contributions. The potential interactions primarily arise from physical contact between elements [15] . The dissipative contributions take into account kinetic energy dissipation mechanisms concerned with the dispersion of elastic waves (this dissipation is general for all materials) [43] . Given this context, the potential contributions to interactions assume a significant role while the dissipative contribution merely influences the energy dissipation within the system. Therefore, the fundamental problem is to formulate a general form of potential interactions between particles, which would apply to materials with different features of mechanical responses.Besides, F p ij and F v ij can be decomposed into the normal and tangential directions, which are represented by the superscript n and t in Equation 2.EQUATIONSubstituting Equation 2into Equation 1 and omitting the gravity terms for simplicity, we can derive:EQUATIONwhere the first term is the normal constituent and the second term is the tangential constituent. Here we discuss the potential interaction forces within the two directions respectively. According to [16] ,The F pn ij can be considered as the composition of contact force f cn ij and bond force f bn ij . The contact forces are activated when two particles physically contact and collide. The bond forces mean if the two particles belong to the same object, they are connected by a bond that will provide attraction or repulsion based on their relative positions to maintain the fundamental properties of the material. We depict the mechanism in Figure 2 . Furthermore, in DEA, a physical quantity called intrusion scalar is commonly used to compute the two forces:EQUATIONwhere r i , r j are the radius of particle x i , x j . δd n > 0 means they contact and vice versa. We use visual aid Fig. 2 (a) to depict δd n intuitively. The red hard sphere intrudes into the blue soft sphere in the figure. The deformation length on the blue surface refers to the δd n . The normal contact force f cn ij should be related to the δd n because the particle tends to recover its initial shape. In addition, the direction of f cn ij is the normal direction between i and j. On the other side, as stated in Figure 2 (b), f bn ij acts as a linkage between two particles belonging to the same object, akin to a bond. f bn ij also relates to δd n . According to the above discussion, the total normal potential interaction forces can be formulated as:EQUATION)where F n c , F n b are two functions to map the δd n and A ij = [A i , A j ] to the contact and bond forces respectively. O k is the Object k. Here A * is the material properties in the particles' small surrounding vicinity.n = xj -xi ∥x j -x i ∥ 2is the normal unit vector. In DEA, the F n c , F n b , A i , A j usually require to be specified by users, which are often simple linear or polynomial functions. For example, the most simple way to compute the total interaction force is the linear spring model F n c+b = k δd n . Some more complex functions such as Hertz-Mindlin (Eq. 6) are also commonly used.EQUATIONIn Eq. 6, E, R, k b are human-defined material parameters. However, the above-introduced handcrafted mapping functions only roughly approximate real cases and always deviate from the realistic natures of materials, leading to inaccurate simulations of DEA. Likewise, the tangential interactions can be analogous to the above. Notably, the normal direction contributes mostly to the total potential interactions because the tangential deformation is small due to the friction constraints [44] . Therefore, general DEA often approximates it by the instantaneous displacement within a timestamp ∆t, i.e. δd t = ||v t ij ∆t|| 2 . v t ij is the tangential velocity of j relative to i.EQUATIONSimilar to n, the t = v t ij /||v t ij || is the tangential unit vector to determine the force direction. F t c , F t b are the user-defined simple functions. Furthermore, the tangential magnitude is also affected by the normal force [45] thus ||v t ij || is included. The DEA theory includes the dissipative contribution [15] , also called the viscosity term [45] , as it is only to simulate energy dissipation to prevent the system from exhibiting perpetual motion. In general, large velocities lead to large dissipation. In the general DEA, the term is often modeled by:EQUATION)where c crit is a material properties called critical damping, v rel refers to v n (normal velocity), v t (tangential velocity) correspondingly.Based on the above discussion, DEA requires multiple user-specific mechanical operators including F n c , F n b , F n vis , F t c , F t b , F t vis , which would be inaccurate and fully different for various materials. Moreover, we cannot precisely estimate the real properties of materials when only videos are available. This can be considered another impracticability of the DEA framework. On the contrary, the benefit of DEA is that we only need to solve the magnitudes of these decomposed forces because these directions are physically constrained in this framework. Therefore, we keep the advantages of the physical priors while remedying the defects of DEA by replacing these mechanical operators with trainable network kernels. Impact Object A Object B Velocity Force Intrusion Normal v f n d  n ij v t ij v n ij f ij f t ij f (a) Different objects (b) Same objects Bond n ij f ij f t ij f n ij v t ij v Impact i j i n d  jn ij n j i v d h h  ] , , , , [ t ij n ij ij j i v f h h h n ij h t ij f i x GT n i x MLP n ij f    i v   n i v  ij h Normal solver Tangential solver cn ij f bn ij f Figure 3:The main pipeline of message-passing networkThis subsection introduces the proposed Discrete Element Learner which replaces these humanspecific operators in DEA with learnable graph kernels. In other words, we integrate physics prior knowledge into the network design to make the entire AI system differentiable and can be optimized through image sequences. We visualize such mechanics-integrated network architecture in Fig. 3 . Furthermore, our method implicitly encodes the material properties into embedding vectors during the unsupervised training. Similar to other GNN-based simulators, we first construct subgraphs by searching neighbors with a fixed radius, and each subgraph can be viewed as a collective of particles involved in interactions. Then we convert all physical variables including velocities and positions to each particle-centric coordinate system when analyzing associated particles.We define the DEA-incorporated message-passing network as follows and the symbols previously used maintain their consistent meanings. First, we encode each particle attribute A i such as material types into latent embedding h i ∈ R 200 , [-1, 1] via Eq. 9.EQUATIONSecond, the following four equations in GNN are used to implement Eq. 5EQUATIONEQUATIONEQUATIONEQUATION) where Φ n is a graph network kernel, n i , n j , and e ij are node and edge features. n i,j encode the properties of the small regions around particle i, j. e ij encodes their interaction. H c , and H b are two heads to regress the magnitude of the two forces. Due to the previous discussion, the contact force f cn ij can only act from j towards i, it must be a positive value, therefore we apply ReLU activation. While the bond force f nb ij can be either positive or negative, thereby no activation is used. As for the dissipative effect, we consider the dissipation as a reduction coefficient rather than directly regress its value because it always diminishes the potential interaction force. In our network, we use an MLP ϕ n activated by Sigmoid σ to model the normal dissipative phenomenon in Eq. 14.EQUATIONLikewise, we apply another kernel Φ t (Eq. 15) to replace Eq. 7. A minor difference is that we omit the tangential bond force because when the particle undergoes very small relative displacement tangentially, the bond length remains nearly unchanged (f bt ij ≈ 0). In this way,EQUATIONAccording to the discussion in the previous section, the quantity f t ′ ij relates to particle properties (n * ), the tangential relative displacement (||v t ij || 2 ∆t), and the precomputed normal pressure (f n ′ ij ). Similar to ϕ n , ϕ t models tangential dissipation in Eq. 16. Notably, all graph kernels output the magnitudes of these mechanical vectors because our mechanical framework precomputes their directions by n and t, which largely reduces the ambiguities when learning dynamics. It can be observed that our approach deeply integrates the mechanical analysis framework, and is partially interpretable. The design of this architecture is strongly inspired by the physical knowledge to reduce the learning burden, we vividly demonstrate it in Fig. 3 .EQUATIONWe train the DEL from only visions. Assuming cameras record many dynamic sequences including the same materials but with different initial conditions, we first initialize scenes as particles by GPF at timestamp t 0 . Then we adopt the first three frames to approximate the initial velocities, similar to [46] . Next, the dynamic module tries to move particles to the next state. Then the GPF renders the after-moving particles into images. If the positions are correctly predicted, the rendered images should align with the recordings. We use L2 loss to superviseEQUATIONwhere D refers to the dynamic module, X t-1 is particle states at the last timestamp, R is the GPF rendering module, andI cam tis the observed image at view cam and time t. Moreover, we apply the same L2 loss to supervise the images produced by the particles updated by using only the f n i (the normal force) as stated in Figure 3 . This aims to amplify the contribution of the normal direction since it is claimed in the previous section that the normal direction is the main influencing factor for collisions. The formulation of this L2 loss is referred to as L n r . In addition, the gradients of rendering loss ∂Lr ∂x can be interpreted as the scaled velocities of particles. Thus we propose the gradient loss to constrain the direction of the total velocity of each particle.EQUATIONThe final loss is L = L n r + L r + βL g . More implementation details are contained in the Appendix.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: This paper provides the full set of assumptions and a complete proof and introduction in Section.3, and 4. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiments: 
 Dataset. Existing datasets in this field contain only a few types of materials or even only rigidity, lack rich interactions between objects, and have simple initial shapes. Therefore, we create a more Baselines. We compare the proposed approach with the two most recent baselines, i.e. VPD [39] and 3DIntphys [11] . These two methods focus on learning 3D particle dynamics from 2D images. VPD must jointly train its specific point renderer and particle dynamics module, which are highly coupled. Therefore, it cannot use other point-based renderers. In contrast, our DEL does not contain any assumption of the neural renderer and thus can be directly adapted to different renderers. We choose the GPF [40] as our renderer. 3DIntphys first reconstructs a series of NeRFs for the image sequences at all timestamps, then extracts the point cloud from each NeRF, and finally trains its dynamics module with point similarity loss across time. Hence, it cannot be trained in an end-to-end manner and cannot render novel views from the updated particle sets.Moreover, to evaluate the effectiveness of our dynamics prediction. We set two additional baselines in which we replace our dynamic model with two prevailing GNN-based simulators, i.e. EGNN [17] and SGNN [9] while keeping the same point-based render with us. In their original paper, they require 3D particle correspondence across time for training. But in our setting, we also utilize inverse rendering to train them from images. We mark these two variants via an upright * . We also compare with the NeRF-dy [5] , a fully implicit dynamics predictor. Next, we additionally retrain NeuroFluid [13] on the Fluids scenario for comparisons because it only supports inferring fluid dynamics.Metrix. We use the Chamber Distance (CD) and Earth Mover Distance (EMD) to measure the similarities between the predicted particle distributions and the groundtruth because no per-particle correspondence is available. The reported CD and EMD are multiplied by 10 2 in all tables for clarity. We compare the PSNR, SSIM, and LPIPS (AlexNet) between the renderings and 2D labels. We also provide qualitative results and comparisons for a better visual assessment.Results in Rendering View. We also present the comparisons of rendering qualities to further evaluate the effectiveness because we adopt inverse rendering to bridge 2D and 3D. The quantitative results are listed in Table 1 . The VPD overall achieves the second-best performance because it is designed specifically for high-quality rendering. However, due to the robust physical priors, our approach more easily learns the underlying physical rules, resulting in a more accurate dynamics prediction, consequently, rendering more plausible images. The other two graph-based simulators perform mediocrely. We additionally show visual examples in Figure 5 . Our approach gives better renderings than baselines due to better dynamic predictions. Due to page constraints, we only give Results in Particle View. In this section, we report the simulation results generated by different approaches in the particle view. Table 2 and Figure 4 show the quantitative and qualitative comparisons respectively. It is observed from Table 2 that our method delivers the most satisfactory results across all scenarios. One interesting finding is that the EGNN * overall outperforms the SGNN * , but in their respective original papers, the SGNN performs better than EGNN when 3D labels are available. The reason might be EGNN benefits from predefining messagepassing directions, but SGNN simultaneously determines both directions and values, which is hard to optimize when only 2D images are given. From this figure, VPD, 3DIntphys, EGNN * , and SGNN * cannot predict precise interactions while our method shows steady long-term simulation. Comparisons with Neurofluid. Neurofluid [13] is another unsupervised method for learning fluid dynamics. It uses the particle-based PhysNeRF as the renderer and employs DLF [47] as the dynamic modules. We compare our approach with it on the Fluids. The results are reported in Table 4 and Fig. 6 . The results show that Neurofluid cannot work well on test data because its dynamic module lacks enough physical priors. Another reason is that it jointly trains the renderer and dynamics modules which makes them compensate for each other, causing overfitting of training data.Ablation Studies. We evaluate some significant components of our method. First, we ablate the gradient loss (marked as N o/L g ). Second, we report the contribution of the tangential decomposition constituent f t ij (N o/f t ij ). Third, we make the graph network fully regress the direction and magnitude of the interaction forces (N o/decomp) instead of using the priors encoded in the DEA framework, i.e. the output of the graph is a force vector. Next, we ablate the L r n loss term, further proving the significance of normal components. The quantitative results are listed in Table 3 , which shows that the L g and L r n contribute to simplifying the optimization. In addition, the mechanical decomposition is important as well. Even though the main direction of message passing is along the directions of edges, the tangential components indeed make the simulation results more realistic. Furthermore, we evaluate the effect of different renders, different training data sizes, different numbers of cameras used to capture scenes, and different points. We also test the Rollout MSE of the three methods when the 3D labels are available. Both of the results can be seen in our appendix, which shows that our method is satisfactory and robust under all the above ablation conditions.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: In this paper, we give a detailed introduction about how we construct the dataset. The proposed method, training, and testing paradigm, and data collection method are completely reproducible and easy to follow. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
conclusion and limitation: 
 Conclusion. We propose the DEL which combines the Discrete Element Analysis framework with graph networks to effectively learn 3D particle dynamics from only 2D images with various materials. The main idea is to integrate strong physical priors to reduce 2D to 3D uncertainties. Existing GNN-based simulators, which are designed for learning from 3D particle correspondence, try to model the whole dynamics of particles. Differently, the DEL only adopts graph networks as learnable kernels to model some specific mechanical operators in the DEA framework, while keeping its mechanical priors, such as the direction of forces and decompositions of forces. We also evaluate our approach on synthetic data with various materials, initial shapes, and extensive interactions. The experiments show our method outperforms baselines when only 2D supervision is accessible. We also show the robustness of our methods to the renderers, training data sizes, and 3D labels.Limitation. Currently, studies in this field, including this work, are conducted on synthetic datasets due to the impracticability of collecting multiview dynamic videos. Hence, "learning from few data" could potentially help address the problem of learning 3D dynamics from a single realistic video. We include them in our future work.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: The code and dataset can be obtained by emailing the author only for research collaborations. One can also easily reproduce the proposed physics-informed GNN on the guidance of the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We discuss the potential impact of this work in the Appendix. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for the responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: In this paper, we do not utilize datasets with a high risk of misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: This work introduces a novel deep learning architecture tightly integrated with a classic mechanical analysis framework, the Discrete Element Method, to efficiently learn 3D particle dynamics from only 2D observations. The success provides insights into the combination of physics-augmented deep learning and 3D neural rendering for areas, e.g. physical simulations, engineering mechanical analysis, and computer graphics.we list all symbols and marks used in the main paper in the following Table for the convenience of reference. Variable Description i i-th particle x i position vector of particle v i velocity vector of particle a i acceleration vector of particle u i movement vector of particle F p ij potential interaction force vector F v ij viscous force vector F g i gravity vector F pn ij potential interaction force vector along the normal direction F pt ij potential interaction force vector along the tangential direction F vn ij viscous interaction force vector along the normal direction F vt ij viscous interaction force vector along the tangential direction n i i-th node features v i (t)velocity of i-th particle at timestep t h i latent embedding from i-th particle attribute e ij edge features between node i and jWe describe the detailed architecture used in the DEL. The entire graph operator is depicted in Equation 9 to Equation 16 in the main paper.The embedding layer is an MLP with 2 hidden layers. The output embedding latent vector is normalized to [-1,1]. The A i and A j refer to the embedding vectors of the per-particle attribute, which we define asA = [mat, d o , ||a t-1 i || 2 ].Here the mat denotes the type of materials, d o is the distance between the particle to the mass center at the current timestamp, and ||a t-1 i || 2 represents the value of the acceleration at the last timestamp of this particle. In our learning framework, the A mainly represents the specific material and its properties. During training, the properties of this material are encoded into the embedding of A.The Φ n and Φ t aim to map related physical quantities to the abstracted node and edge features which we implement by graph neural networks. δd n in Φ n is the initial edge features, we also consider it as e ij . Therefore, Φ n can be described as:EQUATIONwhere temps are temporary intermediate variables. n i and e l ij are the final output of the Φ n . ψ 1,2,3 are three 2-layer MLPs with residual connection. e l-1 ij is the edge feature from the last network layer. N orm refers to learnable Layer Normalization. The reason why we perform an aggregation operation before computing the normal forces is the following. We assume that the mechanical behavior of a certain particle should be related to the external intrusion and the properties of its surrounding vicinity. Therefore, we use this graph aggregation to encode the information of its vicinity (n i ) and the external influence (e ij ). The abstracted features then are input to two different output heads H b , H c to produce the final magnitude of the forces. The two heads are implemented as small MLPs with 2 layers and 200 hidden dimensions.As for Φ t in Equation 15,||v t ij || 2 , f n ′ ij, and e ij are input edge features. In addition, n i and n j are the concatenation of the n i from Equation 10 and the h i from Equation 9because we aim to emphasize the original particle attributes which affect the tangential force.EQUATIONwheree ij = cat([e l-1 ij , ||v t ij || 2 , f n ′ ij ]), n i = cat([n l-1 i , h i ]). The final outputs are f t ′ ij and e l ij . The rest of the architecture remains the same with Equation 20. Besides, we use a simple MLP with two hidden layers (each layer includes 200 neurons) to model ϕ t and ϕ n . Also, the sigmoid activation is used before outputting the coefficients because the viscous forces are manifested as a reduction in potential interaction forces. After f t ij in Equation 16 and f n ij in Equation 14 are obtained, we aggregate the forces for each particle:EQUATIONThus we can update their velocities and positions by the Euler integration:EQUATIONIn this section, we introduce the implementation details of our experiment setup. We first use a pretrained GPF to initialize the scene as particles.For the dynamic module, we build the graph at each timestamp via the k-nearest neighbor search with a fixed radius of 0.025. The particle radius (r for different materials) is set to equal the search radius initially and will be optimized through the training process to be a property of the material. In addition, we place a heavy rigid table at the bottom of each scenario, which is also represented by particles but we do not update its position, and its velocity is constantly set to zero. The only function of it is to support the moving objects above. Additionally, the mass parameter for each particle is set to initialize at 1 and optimized during the training phase as well. All models are trained via AdamW optimizer with 5e-4 learning rate. We adopt the StepLR schedule to adjust the learning rate with the increasing step. After 10,000 iterations, we multiply the learning rate by 0.9. Our method has a unique advantage. We can swap materials used in simulation by swapping the material embeddings and graph kernels because different materials share the same mechanical framework and the graph network in it is only responsible for mapping physical quantities and material embeddings to the forces. We show an example to illustrate this application. As shown in Figure 7 , the first and second rows are the predicted rendered views and particle views of the SandFall respectively. We change the graph kernel and material embedding of the sand to the elastic and plasticine ones respectively. We can observe corresponding changes in the mechanical behavior of particles.Sand Render Sand Elastic PlasticineIn this section, we thoroughly describe the data generation process for our experiments, employing the Material Point Method (MPM) to simulate interactions among various materials. We designed six distinct scenarios to encompass a diverse range of material combinations: Plasticine, Multi-Objs, Bear, FluidR, SandFall, and Fluids. Each scenario is represented through 128 dynamic episodes, differentiated by unique initial conditions such as shapes and velocities, while maintaining consistent material properties, including elasticity and viscosity coefficients, within each scenario.For the reconstruction of meshes from simulation data, we utilized SplashSurf, followed by rendering multiview dynamic image sequences using Blender. At every timestep, the motion is captured from four distinct camera angles. The objective of this research is to derive material interaction behaviors from these 2D observational sequences.Plasticine. This scenario features interactions between two distinct materials: a red elastic object and a blue elastoplastic material. We initialize the elastoplastic materials in various shapes, including duck and bunny configurations, which are then subjected to impacts from the red elastic ball from multiple directions.Multi-Objs. This scenario encompasses interactions among five objects composed of three distinct materials. The blue ball and cuboid are categorized as rigid. In contrast, the cylinder and rainbowcolored ring are elastic, each characterized by unique values of elastic modulus and Poisson's ratio.The experimental setup initiates with the rigid ball colliding with the rigid cuboid, triggering a sequence of subsequent collisions. Each episode is distinguished by varying arrangements and initial configurations of the objects, showcasing diversity in shape and positioning.Bear. This scenario investigates interactions among objects composed of three distinct materials: plastic, elastic, and rigid. The setup includes a brown bear modeled as a plastic toy, a triangular ship constructed from an elastic material, and a heavy, rigid, yellow box. The experimental design involves the ship and box colliding with the plastic bear from various directions and positions, aiming to study the resultant material behavior and object interactions under different impact scenarios. Each collision is designed to explore the dynamic responses of plastic, elastic, and rigid materials when subjected to varying forces and angles of impact.Fluids. The Fluids scenario examines the dynamics of fluid behavior as it impacts a tabletop, with the fluid initially shaped into complex geometries, including forms reminiscent of a bunny, Pokémon, or duck, each varying in size. This setup facilitates observations of the fluid's response upon collision with the ground, encapsulated within a scenario that includes invisible boundaries to constrain the fluid's spread. Upon contacting these boundaries, fluid particles alter their trajectories, enabling detailed study of fluid dynamics and boundary interactions. This scenario allows for the exploration of fluid behavior under varied initial conditions, contributing to our understanding of fluid dynamics in controlled environments.FluidR. In this scenario, we explore the dynamics of liquid flow over a hollow shelf with a complex geometry. Across different episodes, we systematically vary the initial configurations of the liquid, altering both its shape and the height from which it is released atop the shelf. The shelf itself is constructed from a hard elastic material, characterized by a very high Young's modulus, to study the interaction between the liquid's fluid dynamics and the shelf's structural response. This setup provides a unique opportunity to observe the behavior of liquids in contact with elastic materials under varying initial conditions, offering insights into fluid-structure interaction phenomena.SandFall. This scenario investigates the dynamic interaction between a life buoy, composed of sand soil, and an elastic toy duck. Upon impact, the toy duck undergoes deformation and subsequently recovers to its original shape, causing the sand soil life buoy to rebound. A significant challenge in modeling the dynamics of this interaction lies in the partial obscuration of the toy duck by the sand soil during impact. This obscuration results in incomplete information regarding the duck's deformation and response, complicating the task of accurately learning the system's dynamics. The study focuses We in addition show two line diagrams of the accumulated Rollout Chamfer Distance in the two sequences in Plasticine and Multi-Obj, to further indicate the superiority of our dynamics module (Fig. 12 ).The default renderer utilized in the main paper is the GPF. In this section, we evaluate our methods and the baselines with the other two particle-based renderers, i.e. PhysNeRF [13] , ParticleNeRF [41] , both of which can render dynamic particles. PhysNeRF is employed in Neurofluid to represent and render scenes. However, in the original paper, it is jointly trained with the dynamics module and requires more camera views. For a fair comparison, we pretrain the PhysNeRF from scratch and freeze its parameters for all models. In addition, we provide an initialized particle set for PhysNeRF and ParticleNeRF because they are difficult to initialize the scenes as point clouds from sparse camera views (4 in our experiments), while GPF is capable of initialization due to its depth estimation module. Figure 11 and Figure 10 show the rendering and dynamic prediction results of different renders. We can see our method achieves better performance regardless of which renderer is employed. Moreover, rendering capability scales proportionally with the accuracy of learned dynamics. As GPF produces better rendering quality, more accurate dynamics can be learned by utilizing it. This example illustrates that our method is robust to the renderer selection because physical priors always guide the model toward a reasonable learning target. with our method. Both of them outperform EGNN by a considerable margin, which is consistent with that reported in their original paper. This further proves that the reason why SGNN fails to perform well under pixel supervision is caused by the uncertainty of 2D to 3D. EGNN is better than SGNN under 2D supervision because it predefines the direction of message passing which reduces the learning space as well. More importantly, our promising performance illustrates the effectiveness of incorporating strong mechanical priors for both 2D and 3D labels used. Our approach seems to excel in simulating solids, particularly elastic and rigid bodies. However, the SGNN method outperforms in simulating particulate matter like sand and viscous liquids. This discrepancy may stem from the presence of bond forces in our mechanical framework, constraining particles belonging to the same material.Our methodology introduces a novel capability for dynamic material pair substitution within preexisting simulation environments. By predefining mechanical responses and employing a GNN solely for mapping deformations to interaction forces, we facilitate the modification of material interactions with minimal adjustments. This process involves substituting the parameters of the GNN kernel with those derived from alternative scenarios and altering the input A ij , representing the adjacency matrix or interaction terms.Figure 13 illustrates this concept by substituting the original fluid-elastic interaction pair with a plasticelastic pair in the Bear scenario, and a sand-elastic pair in the SandFall scenario, demonstrating the adaptability of our approach to simulate varied mechanical behaviors. Similarly, Figure 14 presents another application of our method, where the sand soil material is replaced with elastic and plastic materials, leveraging GNN kernels trained in distinct contexts. These examples underscore our method's versatility in simulating diverse material behaviors through strategic parameter adjustments.In this section, we visualize the contact forces and bond forces to validate the physical meaning and interoperability of the learnable GNN kernel. After training the DEL, the ϕ n , H c and H b in Eq. 10, 11, 12 should correctly map intrusion into the contact and bond force magnitudes. We extracted these GNN kernels from the simulation system separately to evaluate their responses and outputs to different intrusion inputs. For simplicity, we only evaluate the normal direction. The recorded results are visualized in Fig. 17 . In this figure, the x-axis refers to the intrusion value (δd in the paper), the y-axis denotes the normalized output of Eq. 11, and 12. The solid line denotes the contact force and the dashed line denotes the bond forces. We represent different materials in various colors. We can see from this figure that for rigid bodies, even very small displacements can result in significant resistance, preventing the object from deforming. For sand and water, since they do not need to maintain original shapes, our network adaptively learns to set the bond forces to very small values. For plastic materials, our network has also learned to break the bond links at appropriate times (the bond forces are close to 0). Through this visualization, we would like to claim that our GNN kernel has indeed learned real physical meaning i.e. the forces between particles for simulating different materials.Furthermore, following the suggestion of the reviewer, we visualize the learned feature vector, i.e. h i in Eq. 9, by using the t-SNE method, which is shown in Fig. 18 . In this figure, the red scatter points are projected by the feature of Rigid particles. The green and blue scatter points are produced by the feature of particles belonging to two different types of elastic bodies. The features belonging to the same object have clustered together. The reason they do not overlap completely is because their positions relative to the object are also encoded in the A i . As shown in the Figure, the red points are closer together, which is because for rigid bodies, regardless of where the points are on the object, they have a strong resistance to deformation. We believe this visualization demonstrates that our framework has learned the material-specific features. J More Qualitative Results in both render and particle viewsIn this section, we report additional qualitative results from Fig. 21 to Fig. 30 . These figures include both particle view and render view of our method, EGNN * , VPD, and 3DIntphys (only available for particle views). More detailed results can be seen in our supplementary video. For the EGNN * and our method, we demonstrate both render and particle views to faithfully compare the predicted dynamics.It can be observed from these figures that our method can generalize well to different initial shapes and conditions, especially when predicting long-term dynamics. While the VPD and EGNN * cannot precisely predict the mechanical behaviors between materials. Or they only produce plausible results on some certain materials. For instance, in some cases, such as BEAR and FluidR, the EGNN * can deliver plausible results at the early stage of interactions, but the performance degrades drastically with the progressing deformation.We in addition show a comparison of baselines for the Fluids dataset in Figure 20 . Our method preserves the basic trend of water flow, which other methods do not.In this section, we discuss failure cases of the proposed method. We observe that this method struggles to simulate non-Newtonian fluids because a fundamental assumption of this method is that the material properties are consistent throughout the training dataset. However, the properties of non-Newtonian fluids change with stress variation. Here we show an example in Fig. 19 . If we use our method to simulate non-Newtonian fluid, a large gap between the prediction and groundtruth can be observed.This method is also not suited to simulate smoking. First, smoke consists of extremely small water vapor particles, with the particle size being very small and the number of particles being vast. Simulating smoke by particles requires significant computational resources. Second, smoke simulation not only involves the exchange of momentum between particles but also is governed by thermodynamics, making the simulation of smoke with particles a complex topic. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 


Decision: The reviewers are generally positive on the paper.  Some questions were raised around generalization, evaluation, and scaling and some minor presentation issues were presented.  Overall, these questions have been resolved in the rebuttal and discussion phase and the presentation issues are minor and able to be resolved in the final version.

As such, the AC recommends acceptance as a poster and encourages the authors to carefully revise the paper based on the feedback from the review process.

Reviews: review: paper_summary: paper_summary: The paper considers the problem of physical modeling the dynamics of 3D objects in space using only 2D observations. The authors propose to solve this problem by viewing objects as sets of interacting points and using differentiable rendering of these point clouds. In this pipeline, neural models are used to directly predict the dynamics of 3D particles. Unlike existing methods, the authors propose to use a physically principled framework (Discrete Element Analysis) to develop a partially interpretable neural model that can be used to predict the target dynamics. This framework decomposes the forces applied to the points into gravity, potential, and viscous interaction forces (and further into normal and tangent components). The proposed model thus is constrained to predict certain components from these force decompositions resulting in a more principled, interpretable, and effective dynamics prediction.

The authors propose new more challenging synthetic datasets containing variable materials and objects, evaluate on this data, and compare the results to several existing baselines. The proposed method shows improvements over the baselines in all the scenarios. The authors also provide an ablation study showing the importance of different components.
strength_weakness: strength_weakness: * The written text requires some additional polishing (citations should not be treated as nouns, sometimes the layout of figures and text is too tight, and there are some grammatical errors and not correctly formulated sentences).
questions: questions: How are CD and EMD computed? Are any of the intermediate states used to compute the metrics or are they only computed for the final state? If the proposed model is interpretable does it make sense to compare the predicted particle forces in all the intermediate steps with the ground truth forces from the simulations?

Some examples show that objects can be «torn» over the course of the considered scenario. Currently, the resulting distinct parts of the torn object will still be considered as one single object by the model, is not it? Does it cause problems? Have you considered any solutions for it?
limitations: limitations: The limitations are properly addressed in the text.
paper_summary: paper_summary: The paper proposes a framework to learn 3D dynamics from 2D observations. DEM uses hand-designed kernel functions to model interaction force between particles, which may vary significantly for different material types. The paper changes these kernels to learnable GNN kernels. The time integration still follows DEM. Combined with a differentiable particle renderer, these kernels can be trained to match 2D observations.
strength_weakness: strength_weakness: The evaluations are conducted only on synthetic data.

The learned particle interactions are black-boxes, which are not explainable.
questions: questions: [Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics](https://arxiv.org/abs/2304.14369) should be cited, since it solves a similar task.

Are the material types in particle attribute A pre-known? Are the initial particle distributions known? The known assumptions should be listed in the paper.

How are particle colors assigned? In reality, objects usually have textures. Manually assigning colors may be impractical. 

The particle initialization step is illustrated in Fig. 1, but no technical details are discussed in the text. It seems that the initial geometry is provided in the experiments. In that case, the known-geometry assumption should be illustrated in the figure instead. However, geometry reconstruction is important for real data applications.
limitations: limitations: Limitations are discussed.
paper_summary: paper_summary: This paper proposes to incorporate the neural network with Discrete Element Analysis framework for particle-based simulation. The method adopts GPF as differentiable renderer and is trained through multi-view videos. The proposed model delivers faithful rollouts and outperforms the baselines.
strength_weakness: strength_weakness: 1. To simulate different scenes, such as deformable objects and fluid, it seems that one has to train different models on different cases, indicating scene-specific design with limited generalisation abilities.
2. The results of verifying the generalisation abilities are missing. For example, after trained on the deformable bunny as shown in Figure 5, can the model simulate objects of other shapes, such as a car, or objects composed of more particles?
3. The method seems to be capable of dealing with objects with limited particles and struggle to simulate large amount of particles.
questions: questions: 1. To enable the training on different scenes through the images, are different pertained GPF needed for different scenes?
2. How is the performance of the method for long-term predictions? For example, a trajectory with 150 frames.
3. At L253, the author claims to use L2 loss. However, for equation 17, it seems the loss is L1. Is this a typo?
limitations: limitations: Please refer to the weaknesses and limitations.
paper_summary: paper_summary: This paper presents a new method to learn 3D particle dynamics from sparse 2D observations using inverse rendering.
In constrast to previous work, it does not learn a fully unconstrained model but makes use of known physical priors.
The method learns graph network kernels to model the particle interation forces in the DEA framework, which are classically designed domain experts.
These graph networks are trained from 2D observations using a differentiable renderer without any 3D supervision.
The method is evaluated on different scenes and using different physical materials.
It surpasses the provided baseline comparisons in the test shown in the paper.
strength_weakness: strength_weakness: There are several things that are not clear to me, especially related to scene initialization and the initial velocities. These points are detailed in the questions below. I don't think these are fundamental problems, but I nonetheless hope the authors can clarify these aspects in the rebuttal.
questions: questions: - 3.3: the $v_{ij}^{n}$ or $v_{ij}^{t}$ used in equations 14 to 16 do not match any of the definitions in appendix B.
- For training (3.4), how is the scene initialization done with the renderer? And how accurate is the estimated initial velocity based on this initialization? 
- Closely related: how sensitive is your method to inaccuracies in the initial velocity?
- How long are the sequences used in your experiments?
- How long is your training (number of iterations, epochs, or variable updates)?
- Are the views shown in the figures the ones that were used for training or are they novel views?

There are several grammatical errors throughout the paper, for example:
- 291 "metrix"
- 291 "cham**b**er distance" should be "cham**f**er distance"
- 306 There seems to be a new paragraph missing before "Results in particle view"
- 311 "we claim the reason that..."
- 343 "while keep the priors regulated..."
I would encourage the authors to do a complete pass over the paper to fix any writing mistakes.
The tables and figure on page 9 are also very squeezed.
limitations: limitations: Limitations are briefly discussed.

Comments: title: Re: Rebuttal by Authors
comment: We sincerely thank the reviewer's constructive suggestions, time, and attention in the review and discussion periods, which help us a lot in polishing our manuscript and updating the revision version.
Thank you for these valuable comments and encouraging feedback. We are sure that they improve the quality of our manuscript by a large margin. Thanks again for your time and attention in the review and discussion periods.
Thank you for the rebuttal. I do not have further questions.
Thank you for the clarifications. Trusting that these unclear parts will be clarified in a final version I'd be happy to still support an "accept" for this paper.