Learning Physical Dynamics with Subequivariant Graph Neural Networks
Authors: Jiaqi Han
Wenbing Huang
Hengbo Ma
Jiachen Li
Joshua B. Tenenbaum
Chuang Gan
Venue: NeurIPS 2022 Accept (2022)
Keywords: physical dynamics
graph neural networks
symmetry

Abstract: Graph Neural Networks (GNNs) have become a prevailing tool for learning physical dynamics. However, they still encounter several challenges: 1) Physical laws abide by symmetry,  which is a vital inductive bias accounting for model generalization and should be incorporated into the model design. Existing simulators either consider insufficient symmetry, or enforce excessive equivariance in practice when symmetry is partially broken by gravity. 2) Objects in the physical world possess diverse shapes, sizes, and properties, which should be appropriately processed by the model. To tackle these difficulties, we propose a novel backbone, called Subequivariant Graph Neural Network, which 1) relaxes equivariance to subequivariance by considering external fields like gravity, where the universal approximation ability holds theoretically; 2) introduces a new subequivariant object-aware message passing for learning physical interactions between multiple objects of various shapes in particle-based representation; 3) operates in a hierarchical fashion, allowing for modeling long-range and complex interactions. Our model achieves on average over 3% enhancement in contact prediction accuracy across 8 scenarios on Physion and 2$\times$ lower rollout MSE on RigidFall compared with state-of-the-art GNN simulators, while exhibiting strong generalization and data efficiency.

Learning Physical Dynamies with Subequivariant Graph Neural Networks
Jiaqi Han
Wenbing Huang"
Tsinghua University
Gaoling School of Artificial Intelligence,
Renmin University of China
Beijing Key Laboratory of Big Data Management and Analysis Methods
Hengbo Ma University of California, Berkeley
Jiachen
Joshua B. Tenenbaum MIT BCS, CBMM, CSAIL
Stanford University
Chuang Gan
UMass Amherst MIT-IBM Watson AI Lab
Abstract
Graph Neural Networks (GNNs) have become prevailing tool for learning physi- cal dynamics. However, they still encounter several challenges: 1) Physical laws abide by symmetry, which is vital inductive bias accounting for model generaliza- tion and should be incorporated into the model design. Existing simulators either consider insufficient symmetry, enforce excessive equivariance in practice when symmetry is partially broken by gravity. 2) Objects in the physical world possess diverse shapes, sizes, and properties, which should be appropriately processed by the model. To tackle these difficulties, we propose a novel backbone, Subequivari- ant Graph Neural Network, which 1) relaxes equivariance to subequivariance by considering external fields like gravity, where the universal approximation ability holds theoretically; introduces new subequivariant object-aware message pass- ing for learning physical interactions between multiple objects of various shapes in the particle-based representation; 3) operates hierarchical fashion, allowing for modeling long-range and complex interactions. Our model achieves on average over 3% enhancement in contact prediction accuracy across scenarios on Phys- ion and 2x lower rollout MSE on RigidFall compared with mute-off-theaat GNN simulators, while exhibiting strong generalization and data efficiency. Code and videos are available at project page: https//hanjgll.giiuubbooO
Introduction
Learning and predicting the complicated physical dynamics and interactions between objects are vital to many tasks including physical reasoning [8, 38, 7, 19], scene understanding [3], and model-based planning and control [20, 23, 24, 37, 1]. Several learning-based differentiable simulators [31, 30, 22, 27] have been proposed, and they obtain promising achievements in simulating various kinds of interacting objects including rigid and fluids. The success mainly relies on the prevalence of graph neural networks [14, 30, 15], which are desirable tools for physical simulation, by modeling particles as nodes, physical relations as edges, and their interactions as the message passing thereon.
"Corresponding author: Wenbing Huang.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Notably, physical world exhibits certain symmetries. For example, the way dominoes fall from left to right will exactly be preserved when the system is rotated horizontally to another direction. Inherently. humans reason about the dynamics with the preservation of such symmetry, and this inductive bias has been endorsed by the physical laws that also abide by the symmetries in our 3D world. Yet, regardless of this inductive bias, current differentiable simulators like GNS [30] and DPI [22] would fail to learn the real dynamics that can generalize to all directions. In the example of dominoes, if the data in training are positioned from left to right, GNS can predict well during testing when the dominoes also fall in the same direction, but performs poorly if the scene is rotated horizontally (see demo in the Supplementary Video). This observation implies that GNS overfits training samples without learning the true dynamics that obey the symmetry, limiting its generalization.
With this consideration, there have been a number of works, named geometrically equivariant graph neural networks [16], that leverage symmetry as an inductive bias in learning to simulate. These models are designed such that their outputs will rotate/rranslate//eelec in the same way as the inputs, hence retaining the symmetry. However, models like EGNN [32] and GMN [17] are exerted E(3)- equivariance, the full symmetry of the 3D Euclidean space. Such constraint is so strong that it penalizes all directions in the 3D space, which cannot be applied to scenarios with external fields, like gravity. The existence of gravity breaks the symmetry in the vertical direction, reducing E(3) to its subgroup. We formally characterize this phenomenon of equivariance relaxation as subequivariance.
In general, simulating physics on datasets like Physion [4] are highly challenging, due to the scale of the system (on average thousands of particles per system), the diversity of the interactions (e.g.. collision, friction, gravity), as well as multiple shapes, materials, or even rigidness of the objects. These factors require unique efforts in designing the simulators, which brings another weakness of current GNN-based methods: From a practical point of view, they seldom explicitly involve the geometric object information into message passing. Moreover, when modeling multiple interacting objects of different shapes, the interactions between within objects are usually different. For example, the former aims to exchange momentum or energy across objects, while the latter usually accounts for the geometrical constraint. It is thus important to involve objectness to distinguish interactions between particles within objects from those across objects.
In this work, we propose Subequivariant Graph Neural Networks (SGNN) that consist of several features to tackle the above challenges. 1. We relax equivariance to subequivariance and design subequivariant functions to physical scenarios with the existence of gravity, and excitingly, we have proved that our designed form inherits the approximation universality. 2. We formulate novel subequivariant object-aware message passing framework for modeling dynamics and interactions between objects of varying shapes. 3. We incorporate the subequivariant message passing into a hierarchical model to deal with long-range and complicated object interactions. We demonstrate the efficacy of our SGNN for learning physical dynamics on a large-scale challenging dataset Physion with 8 different scenarios, and a 3-object RigidFall dataset. Experimental results show that model is capable of yielding more accurate dynamics prediction, is highly data-efficient, and has strong generalization compared with the state-of- the-art learning- based differentiable physical simulators.
2 Related Work
GNN-based physical dynamies simulators. There have been many works that employ graph neural networks as physical simulators of dynamical systems [2, 26, 18). Graph Network Simulator (GNS) proposed by [30] has been showcased a simple yet powerful tool in simulating large systems in particle-based representation of rigid and fluids by dynamically constructing interaction graph and performing multiple steps of information propagation. DPI [22] adds one-level of hierarchy to the rigid and predicts the rigid transformation via generalized coordinates, with applications to manipulation. There are also works that incorporate strong physical priors like Hamiltonian mechanics [29, 39], geometrical constraints [28] and shapes [27]. Despite the promising empirical results, these works have not given sufficient considerations on symmetry especially when external force like gravity presents, leaving room for enhancing their generalization to unseen testing data.
Geometrically equivariant graph neural networks. Equivariant graph neural networks [16] are a family of GNNs that are specifically designed to meet the constraint of certain symmetry, mostly involving translations, rotations, and/or reflections in Fuclidean space. This goal is approached by several measures, including solving group convolution with irreducible representation [34, 12] or

---

leveraging invariant scalarization [35] like taking the inner product [32, 17]. Our approach also belongs to the scalarization family together with EGNN [32] and GMN [17]. Specifically, EGNN models the interaction with the invariant distance as input, computed as an inner product of the relative positions, and GMN generalizes to multi-channel version by taking into consideration a stack of multiple vectors. These equivariant GNNs operate on particle graphs or point clouds, while we additionally consider a combination of both particle- and object-level information for physical simulation. More importantly they are assumed a full Euclidean symmetry with strong equivariance constraint, while we elaborate how to relax the constraint in the existence external fields like gravity by leveraging subequivariance.
Equivariance on subgroups. H(n) E(te--lte CNNs [36, 6] develop convolutional kernels that meet equivariance on E(3)/E(2) and their subgroups by leveraging restricted representation. EMLP [10] obtains equivariance on arbitrary matrix groups. However, these approaches require specifying the particular group/subgroup to perform equivariance, while our goal here is to relax the equivariance on subgroups of E(3) by considering external force field, having more physical implications. Moreover, these works rely on computationally expensive operations like irreducible representation or solving group constraints, while our formulation resorts to scalarization, which is easy to implement, efficient to compute, and also comes with necessary universality guarantee.
3 Subequivariant Graph Neural Networks
3.1 Background
GNN-based simulators. We start by introducing the mechanism of GNN-based simulators for learning physical dynamics. We consider the particle-based representation of a physical system with N particles consisting of M objects. At time t, each particle within the system possesses some state information, including 1. the position a!!) E R3 and the velocity J.") E RR, which are both directional vectors; 2. some attributes h; E R" without geometric context, such as the rigidness; 3. the dynamic spatial connections with other particles, where an edge will be constructed if the distance between two particles is smaller than threshold namely, g()) {(i,j): || o <). We denote the object category of particle i as o(i) E Z satisfying o(i) = k if particle i belongs to object k. The goal here is to predict the position of the next step xt (t(++1) given the above system information at time 1, which can be favorably modeled by GNNs, i.e., a (:((-1) = PPNN ((=!")) ), (o(1), ), (he), E
Since the prediction different time shares the same model, we will henceforth omit the temporal superscript & for all variables for brevity.
The conventional GNN simulators şGNN [22, 30, 26] offer a favorable solution by leveraging message- passing on the interaction graph, which computes:
mij db (Z:; Bi, 2;,, Jj hi, hg))
N 6, h1 1/ ESENTO) mij. Bi, 8g h:
where (ii) = [j= (i.j) E 8) is the neighbors of node i, and vi are the edge message function and node update function, respectively. The prediction is obtained by conducting several iterations of message passing. Nevertheless, such form does not guarantee the desirable symmetry context with common choices of ó and vi. and meanwhile the object information has not been claborated. leaving room for a more exquisite message passing scheme for learning complex physical dynamics. Recent works such as EGNN [32] and GMN [17] have actually considered ((3))-qqivariance in the design of the functions 6 and v/ to pursue symmetry. However, they are not applicable to the case when symmetry is partially violated by gravity. which motivates our proposal in $ 3.2. Prior to introducing our work, we first provide necessary preliminaries related to equivariance.
Equivariance. In this paper, we basically focus on equivariance in terms of E(3) transformations: translation, rotation, and reflection. The formal definition is provided below.
Definition 1 (Equivariance and Invariance). We call that the function f gaxm x R + 33xm'
is E( -equivaai,,, if for any transformation g in E(3), f(g- Z, h) = g- S(Ž, h), VŽ E 2
wh e R", Similarly, f is invariant if flg Z. h) f Z h), Vg E E(3).
3

---

R.
R.
2, - 11,
o
Z 2 O
my oR. -2, B/)
m
(3) - 1,000 =,1l")
mky Zz)
Z, Z, G C.n 12, 6 20
R. 12 0) (Z.
2, - 13
GNS, DPI
EGNN
GMN
SGNN
Object (
Figure 1: Comparison between different message passing. We omit the scalars h, for simplicity.
In Definition 1, the group action is instantiated as g -= Ož for the orthogonal transformation (rotation and reflection) where O E O(3) = {0 T= = I], and g 2 z + t for translation where & E IR3. Note that for the input of f. we have added the the right-arrow superscript on Z to distinguish it from the scalar h that is unaffected by the transformation, akin to our notations for the position Z; velocity Ü, and attribute h,
It is non-trivial to derive equivariant function particularly for orthogonal transformations. GMN [17] proposes multichannel scalarization form, which yields,
f(Z,)) = žv, S.L. = o Z,h),
(3)
where the inner product ZT 7 Rmxm is firstly computed and concatenated with h, the resultant invariant term is then transformed by function (usually a Multi-LLayer Perceptron (MLP)) a pmxm+n + Rmxm' producing V E Rmxm' and the directional output is acquired by taking a matrix multiplication of the basis Z with V. By joining the analyses in GMN along with [35], we have the universality of the formulation in Eq. (3), which provided in Appendix A.
3.2 Subequivariant Object-aware Message Passing
To simultaneously benefit from symmetry and object-aware information, we develop Subequivariant Object-aware Message Passing (SOMP). It characterizes information aggregation between multi- channel vectors and scalars, while taking into account both the particle and object information. We
use Z, E 3 to indicate stack of m multi-channel 3D vectors; particularly, Z; is initialized as [ G.]. S I N by involving both the position and velocity. We additionally initialize the object features by pooling its particles: for object k, we have C₁ = [)o())= [(to())=)) Z,, Cs-
Σ [() h; 1 < k < M. We also introduce a binary operation "0" which yields Z, e Z,
[Z,- Z,, ū. T... It transforms the translation-equiiariann vectors to be invariant and stacks those invariant vectors together, resulting in a translationinvariiat representation while enriching the input vectors with more channels. The direction of gravity is set to be along the vertical axis.
In high-level overview, denote message passing as the function y that updates each particle given the input states of all particles, objects, and graph connectivity:
[Z) KO + (1(Z., (1(Z,,))),, {(Cx, m.)l
(4)
Specifically, yp is unfolded the following message passing and aggregation computations:
Z, (Z. @ CoG)))(( e C.t)) Žz),
(5)
hy h. (he (1,)//oo()
(6)
Myj Mij dg (Zigs hij
(() (Z., hi) + thg (ESENOO M e Co()), E SENNO m.j) m.:))l/allcot))
where || is the concatenation along the channel dimension, Ni(i L (i,j) E {} is the neighbors of node i, and both Φg and vor are subequivariant and will be defined in detail in Eq. (9). In detail, we first derive the multi-channel vector input Z., for the interaction between particle i and in Eq. (5)

---

by stacking the three terms along the channel dimension, including the relative information of the particle with respect to its belonged object: Z, e Cote) and Z, e Co()). and the relative information between particles Ż @ Z, By this means, we arrive at a mnnnslatiooninvariin representation Z, with rich object-aware geometric information. For the invariant features simply concatenate them in Eq. (6). Subsequently, in Eq. (7) we feed the interactions Zig and hij into the subequivariant message function $g. yielding the vector message M.j and scalar message m.y². Finally, Eq. (8) first performs message aggregation and then updates the states by another subequivariant function dgg. obtaining the eventual result. With the updated Z and h4, one can readily obtain the output with equivariance or invariance as desired, which, in our case, implies Z = [<e]] by setting m' = 1.
Comparison with existing GNN-based simulators. The core of satisfying equivariance lies in taking the invariant inner product before feeding the vectors into the MLP. Such design shares a similar spirit in nature with existing works including EGNN [32] and GMN [17]. The interaction modeling of EGNN only considers the relative position ã. zi and its inner product || E which might have limited expressivity while tackling complex physical interactions and dynamics between objects. GMN further extends multi-channel interaction Z, e Zj where Z, = ( ūt]. Nevertheless, it lacks the necessary object information, while we explicitly involve the particle-object correlation Z, é Coto) in our SOMP. Besides, GNS [30] and DPI [22] only enforces translation equivariance by directly feeding [ zÿ ū, أy] into an MLP without taking the inner product. They fail to preserve the e33-vivuivae and consequently have weaker generalization as we will illustrate in our experiments. Fig. summarizes and compares the message- passiing schemes of GNS, EGNN, GMN and our SOMP. Particularly, our design of SOMP takes careful considerations: 1. Equivariance is still permitted with both geometric and scalar object features involved; 2. The object information can also be constantly updated during message passing; 3. The expressivity of SOMP enhanced over EGNN and GMN with object information considered. We provide detailed theoretical comparisons in Appendix A.2 by showing that EGNN and GMN are indeed special cases of SGNN.
We now present the detailed formulations of 0g in Eq. (7) and Vy in Eq. (8). Since the full symmetry is violated by gravity g E R and the dynamics of the system will naturally preserve a gravitational acceleration in the vertical direction. By this means, the orthogonal symmetry is no longer maintained in every direction but only restricted to the subgroup 03(3) == {O E E(3) Og = g). that is, the rotations/reflections around the gravitational axis. We term such reduction of equivariance novel notion: subequivariance. To reflect this special symmetry, we augment Eq. (3) by:
l ( h) = [ g. V3-
S.L.V ([Z,g] [ h))
where a: (m+)+))(m++)) - (m)+) is an MLP. Compared with Eq. (3), here we just augment the directional input with g. Interestingly, such a simple augmentation is universally expressive:
Theorem 1. Let fa h) be defined by Eq. (9). Then, fg is -equivarra, More importantly, For any 0g(3)) -qquivrrqvv function f( [2, h), there always exists an MLP a satisfying I|ʃ for arbitrarily small positive value c.
Sall <6
The proof is non-straightforward and deferred to Appendix A.I. The detailed architectural view of the proposed SOMP is depicted in Fig. 8 in Appendix A.3, We leverage our designed subequivariant function with such desirable properties for dg and vig in our subequivariant object-aware message passing. We immediately have the following theorem guaranteeing the validity of our design.
Theorem 2.
The message passing yp (Eq. 4) is (g(3) (3))-)))vaarian.
3.3 Application to Physical Scenes with Hierarchical Modeling
This subsection introduce the entire architecture of our subequivariant GNN. Many physical scenes are complicated, possibly involving contact, collision, and friction amongst multiple objects. In light of this, we further incorporate our message passing into multi-stage hierarchical modeling framework. One of our interesting findings here is the edge separation. This is indeed motivated by the consideration that the interactions between or within objects are usually different, similar to
For both dy and eig. we expand more output channels besides V7 of in Eq. (9), and assign it to the invariant message myj in Eq. (7) and h² in Eq. (8), respectively.
5

---

Particle-level
Object-level
Particle-level
Einter
& obj
Eiuner
Inter-object Message Passing
Object-level Message Pussing
Inner-object Message Passing
Figure 2:
The overall architecture with hierarchical modeling. specified as three stages.
automorphism graph networks [9, 25, 33] where the neighboring objects formulate the isomorphism group of edges. The former serves as bridge of exchanging momentum and energy via interaction forces, while the latter usually accounts for maintaining the rigid or plastic constraints. Therefore, would be beneficial to disentangle these two kinds of interactions in message passing, which also distinguishes our hierarchical modeling from DPI [22]. The overall flowchart is provided Fig. 2. To be specific, alternate between the following three stages that implement yp in Eq. (4) distinctly.
Particle-level inter-object message passing. We start by modeling the local interactions from the particle-level, where we only involve those between different objects. That is,
( KO = = (e (1(2,, (((,,,))) {(Ck, 1
(10)
where Emner {(i,)) (1.5) E,) # o(j)}.
Object-level message passing, Given the renewed particle-level information produced by the first stage, are now ready to construct the object-level message passing graph. The object-level
message passing is proceeded as
422 (((Ck, cx), (0).
(11)
where Eujj comprises the edges between interacted objects. indicating Eubj = (k), (ii j) Einer: oii = o(j) 1}. As a specific instantiation of yo Eq. (4), here each object considered as node and the original object information in Eq. (5-8) is omitted (this is why denote the second input of se as the empty set 0). Furthermore, we leverage pooling of the updated particle-level in- formation from the first stage the object-level interactions, or formally, we re-define the calculation
of "0" for two different objects indexed by k, I as C. e C. Moan-Pool(..))e.... similarly, c |c Mcan-Pool EL)) (h/l1h)).
2 e 2:) and
3. Particle-level inner-object message passing. Finally, given the updated object state information, we carry out the message passing for the particles only along the inner-object edges Einner = ((i,j)
(i,5) E o()) o(j)), namely,
( M = (((2,,))) ((C), ()
k= Einner)
(12)
We use N as the proposal of the positions in the next time step, akin to the way in $ 3.2.
4 Experiments
We conduct evaluations on Physion [4] and RigidFall [21]. Physion [4] is large scale dataset created by the ThreeDWorld simulator [13]. which consists of eight different scenarios, including Dominoes, Contain, Collide, Drop, Roll, Link, Support, and Drape. These scenarios possess diverse physical scenes with complicated object interactions, serving as a challenging benchmark for evaluating dynamics models. Particularly, Drape also involves simulating deformable objects. RigidFall [21] is simulation dataset whose scenes involve several cubes falling and colliding under varying magnitude of gravitational acceleration.

---

Table 1: Contact prediction accuracy (%) on Physion. Results are averaged across 3 runs.
Dominoes
Contain
Link
Drape
Support
Drop
Collide
Roll
GNS* [30]
78. 78.6219
71.6216
66.7+15
58.8210
68.2
65.3
86.1
8
DPI* [22]
82.3+13
72.3+18
63.7+22
53.3209
64.8+20
70.7=08
84,4
82.3
GNS [30]
53,3+18
70.7+22
58,0+17
58.0+13
61.3+11
65.0+07
76.7+112
80.0+0.
DPI [22]
57.3+22
71,9+13
63.0+
52.3+12
58,0+1.
68.7++07
78.7+1.3
80.5
GNS-Rot [30]
74.7=12
72.7+20
63. 2
57.3+10
65.5
64.7=1
84.0=10
79.4
DPI-Rot [22]
72.7+17
69.4+22
65.2+
52.8+10
66.7+ax
72.3+005
83,2+1.
80.1
EGNN [32]
61.34111
66,0+19
52.7+12
54.7+08
60.0
63.3+a9
76.7+114
79.8+04
EGNN-S
72.0121
64,6118
55.3:21
55.3210
60.5
69.3:20
79.321.
81.6209
GMN [17]
54.7100
57.6122
54,5:16
57.6111
55.1
54.2:111
19.5
81.3
GMN-S
55.6.113
65.3
55.1
57.0
59.3122
57.3
81.2112
79.3
Our SGNN
89.1
78.1+15
73.3
60.6
71.2+t09
7..3
85.3
84.2
Dominoes
Contain
Link
10a
Rolluu 8SE
30
Rollour MSSE
10'
Rollour MSE
107
30-1
201
10-
GNS-RRot
GNS-Rot
100
GNS-RRt
DPS-Rot
Dft-Rut
DPL-Rot
10a4
EGNN
EGNN
EGNN
(Ours)
(Durs)
20
100
120
Figure 3: The rollout-MSE curves on Dominoes, Link, and Contain. Our model achieves the lowest MSE. Colored areas indicate 95% confidence intervals. Full results in Appendix C.2.
4.1 Baselines
We compare SGNN with several SOTA particle-based GNN simulators, including two non-equivariant models GNS [30] and DPI [22], as well as two equivariant models EGNN [32] and GMN [17]. Notably, the data preprocessing provided by Physion [4] for particle-based methods does not consider the camera-angle for the rotation of 3D coordinates, leading to the situation that in some scenarios, e.g., Dominoes, Link, Support, and Collide, there exist directional bias (from left to right horizontally) in the trajectories. This motivates us consider three evaluation protocols: 1. Use the original training and testing set with directional bias. 2. Train on the original set and test on the randomly-rotater testing set. 3. Train on the randomly-rotate training set (akin to applying data augmentation of rotations [5]). and test on the randomly-rotane
testing set. Here all rotations are restricted to the ones around gravity. We dub the results obtained from the three cases GNS*, GNS, and GNS-Rot for GNS, and similarly for DPI. For EGNN and GMN, we also propose to adapt them to be subequivariant (EGNN-S and GMN-S) by adding extra vector along the gravity in their update of velocities. Details are in Appendix B. Note that EGNNs, GMNs, and our SGNN always produce exactly the same result in the three scenarios due to their equivariance (or subequivariance). For RigidFall, such bias is not observed since the cubes are falling vertically driven by gravity.
4.2 Evaluation on Physion
Experimental setup. We strictly follow the training and evaluation protocol proposed in [4] for the particle-based simulators. We employ two evaluation metries: 1. Contact prediction accuracy. This metric, as also adopted by [4], evaluates whether two targeted objects labeled in the dataset will contact during the model rollout. 2. Rollout MSE, the mean squared error between model rollout and the ground truth. We reuse the training protocol and for baselines and our model to ensure fair comparison. In detail, all MLPs are initialized with projection layers and a hidden dimension of 200. The networks are trained with an Adam optimizer, using an initial learning rate 0.0001 and an early-stopping of 10 epochs on the validation loss. We use 4 iterations in each message passing of our model. We run RANSAC [11] during testing on all models to help enforce the rigid constraint as suggested by Physion. More experimental details including a comparison of computational complexity are deferred to Appendix B.
suggested in Physion
7

---

step
Step
Ground Truth
Ours
GNS
EGNN
Figure 4: Visualization examples on Physion. Our model yields accurate long-tenm predictions.
Results. We present the average contact prediction accuracy with std in Table 1, and the rollout MSE in Fig. 3. The detailed MSE on all 8 scenarios deferred to Appendix C.2 due to space limit. We interpret results by answering the following questions. Q1. Is maintaining the symmetry important to models that learn to simulate? For the non-equivariant models GNS and DPI, their performance usually encounters significant drop if they are trained the original data but tested the rotated, particularly on the scenarios with bias in directions. For example, the accuracy of GNS drops from 78.6% to 53.3% Dominoes when comparing GNS* with GNS. Such observation also holds for DPI and on more scenarios like Collide and Support. Using data augmentation would help in relieving the issue, but still hard recover the performance. The accuracy on Dominoes yielded by GNS-Rot is 74.7%, still worse than 78.6% by GNS* Our SGNN instead leverages subequivariance to generalize all horizontal directions, and is always guaranteed to produce the same prediction regardless of any rotations around gravity, and there is also no need to apply any data augmentation of rotations during training. Interestingly, EGNN-S and GMN-S offers improvements over EGNN and GMN, implying the necessity of subequivariance, but are still outperformed by SGNN by large margin. Q2. Does SGNN learn the physics and simulate the dynamics better than other models? As displayed in Table 1, SGNN achieves the highest prediction accuracy on 7 out of the 8 scenarios, and is also very competitive on Collide. The improvements are very significant (>5% accuracy enhancement over the second best) on scenarios including Dominoes, Contain, and Link, where there are multiple objects contacting, colliding, and interacting. This observation is also supported by the rollout MSE curves in Fig. 3, showing SGNN consistently yielding lower prediction error along the trajectories. Q3. How does SGNN perform compared with other equivariant models? For EGNN and GMN, enforcing such strong constraint of -eui-aauivee leads to inferior performance in scenes with gravity. For instance, in Dominoes, Contain, and Link, the accuracy of EGNN and GMN only stays around 60%, while SGNN achieves promising results with subequivariant object-aware message passing.
Generalization across scenarios. We further evaluate the generalization capability of our model across various scenarios. To fulfill this goal, we leverage models trained in certain scenarios to test them in others. We summarize the re- sults in Fig. 5, where the models are trained in the scenarios indexed by the rows and evaluated on those indexed by the columns. It is observed that model exhibits signif- icantly stronger generalization than GNS, the best-performed baseline Physion. For example, our model, although trained with the dynamics of Contain or Link, still yields high accuracy (>78%) when tested on Drop, showcasing that our model is more advantageous in learning the dynam- ies and interactions of various objects.
GNS
Ours
90
64.7
54.7
68.8
62.0
74.3
64.0
71.5
65.7
80
B
70
66.2
78.7
87.
72.7
65
62.3
56.7
88.0
63.5
65.3
60.0
74.7
71.2
-55
A: Drop
B: Contuin
C: Link
D: Support
Figure 5: Generalization across tasks. Row/column indicates the training/testing scenario, respectively.
8

---

Ablation studies. Here we conduct several ab- lations to inspect how our proposed components contribute to the overall performance, and the results are given in Table Subequivariance. We replace our subequivariant formulation of Øg and vig by the Equivaauia counterpart without gravity. It is clear that, in this way, the model restricted by strong equivariance con- straints, which leads to significant degradation in performance. This verifies the necessity of relaxing the full-equivariant model with our introduced method for physical scenarios with gravity. Moreover, we extend the Steerable E(2)-CNNS [36, 6] to GNNs as an alternative to obtain equivari- on the SE(2) subgroup (more details results are in Appendix C.5). Although better than EGNN, the adapted Steerable SH(2)-GNN is still inferior to SGNN, implying our physics-inspired SOMP is more advantageous on simulating physical dynamics. 2. Object-aware message passing. We replace C2 and C₁ in Eq. (10) by zeros, eliminating the object information from message passing. Without this necessary information, the model might fail to capture useful geometric vectors like 2; - which is closely related to physical quantities like torque. 3. Hierarchy. We compare our model with its counterpart without hierarchy. The flat model encounters average of 4% drop in the prediction accuracy, showcasing the advantages of leveraging the particle- and object-level message passing for modeling complex object interactions in physical scenes. 4. Edge separation. We employ different sets of edges the inter-object and inner-object message passing. We argue that it relieves the learning complexity of the cessage-passing with different types of relations. It is observed that removing edge separation leads to detriment the accuracy in various scenarios.
Table 2: Ablation studies.
Dominoes
Contain
Drop
SGNN
89.1
78.1
74.1
eqquivarrreeeee w/o object-aware
79.6
65.3
1
81.2
68.7
70.1
w/o hierarchy
83.5
72.5
71.3
w/o edge separation
85.0
74.8
72.0
Steer-SE(2)-GNN [36]
59.1
66.7
66.7
Generalization toward other rotations. To further investigate whether incorporating our subequiv- ariance truly helps the model to learn the effect of gravity, we apply a rotation around a non non- gravity axis, resulting in scenarios in Fig. 6 where domi- noes are placed on an incline while gravity still points downwards vertically. Interestingly, SGNN well generalizes to this novel scenario and reasonably simulates the effect of gravity. Particularly, the domino at the bottom starts to slide down along the table driven by gravity. More visualizations are provided in the vidco.
Table
Gravity
Figure 6: Rotation around non- non-gravity axis.
Qualitative comparisons. We provide visual- ization samples on Physion in Fig. 4. More cases are shown by our Supplementary Video. Our model consistently yields accurate predictions across various scenarios.
4.3 Evaluation on RigidFall
Table 3: Rollout MSE (x10-2) on RigidFall. [Train| denotes the number of training samples.
Train
200
|Taiin| 500
|Train| 1000 = 40
5000 |Triin|
= 20
= 40
t 20
== 40
t 20
t 20
= 40
GNS [30]
2.40+1132
6.794343
1.894087
5.41
1.09
3.38
0.65
2.38
DPI [22]
1.71
5.374226
1,48
4.67
1.24
3.97+233
0.52+a.3
2,40+1.58
EGNN [32]
1.894124
5,664281
1.07
3,94
0.95+044
2.72+115
0.78
2.63
GMN [17]
2.74
7,08
2.50+a98
6.34
1.91
4.50+221
1.36+0xs
3.50+1.98
Our SGNN 0.721000
2.44
0.39
1.47
0.31
1.09
0.20
1.01
Experimental setup. We use the code provided by [22]. We record the rollout MSE t 20 and 40. To compare the data efficiency of different models, we consider four cases with the size of training set ranging from 200 5000. More details in Appendix B.
Results. The results are presented in Table and visualized in Fig. 7. Our model consistently yields the best predictions regardless of the time step and the size of training set. Notably, the rollout

---

step
step
Ground
Truth
Ours
Figure 7: Visualization examples on RigidFall. Our SGNN provides accurate simulation trajectories.
error still maintains very low even if only 200 training samples are provided, verifying the strong data-efficiency and generalization of SGNN.
Discussion
Limitations and Future Work. Our model relies on the particle-based representation of the physical systems, which, in real-world scenarios, might be difficult to obtain, or usually with noise. Our model does not explicitly consider self-contact. Future directions include augmenting our dynamics model with strong physical priors like Hamiltonian [29, 39], or combining it with groundings of visual prior [21] with application to more sophisticated tasks like physical reasoning or robot manipulation.
Conclusion. We propose Subequivariant Graph Neural Networks for modeling physical dynamics of multiple interacting objects. We inject appropriate symmetry framework, and takes into account both particle- and object-level state messages. In particular, subequivariance is novel concept for characterizing the relaxation of equivariance from a group to its subgroup. We show that relaxation from full equivariance to subequivariance can be applied to systems with gravity involved. Experiments verify that SGNN accurately captures the dynamics with the guarantee of the desirable symmetry, and exhibits strong generalization with high data- data-efficiency.
hierarchical message passing
Acknowledgments and Disclosure of Funding
We thank the anonymous reviewers for their constructive suggestions. This project was in part supported by MIT-iBM Watson AI Lab, Amazon Research Award Mitsubishi Electric. Dr. Wen- bing Huang was supported by the following projects: the National Natural Science Foundation of China (No.62 NNo.62006137); Guoqiang Research Institute General Project, Tsinghua University (No. 2021GQG1012); Beijing Outstanding Young Scientist Program (No. BJWZZJHO0
References
[1] Pulkit Agrawal, Ashvin V Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to poke by poking: Experiential learning of intuitive physics. Advances in neural information processing systems, 29, 2016.
[2] Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and koray kavukcuoglu. Interaction networks for learning about objects, relations and physics. Advances in Neural Information Processing Systems, volume 29, 2016.
[3] Peter Battaglia, Jessica B Hamrick, and Joshua Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy of Sciences, 110(45):1327- 18332, 2013.
[4] Daniel Bear, Elias Wang, Damian Mrowca, Felix Jedidja Binder, Hsiao- Yu Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin A. Smith, Fan- Yun Sun, Li Fei-Fei, Nancy Kanwisher, Joshua B. Tenenbaum, Daniel LK Yamins, and Judith E Fan. Physion: Evaluating physical pre- diction from vision in humans and machines. Thirty-fftth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
10

---

[5] Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew G Wilson. Learning invariances in neural networks from training data, In H. Larochelle, M. Ranzato, Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 17605-17616. Curran Associates, Inc., 2020.
[6] Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build e (n)-equivariant steerable cnns. In International Conference on Leurning Representations, 2021.
[7] Zhenfang Chen, Jiayuan Mao, Jiajun Wu, Kwan- Yee Kenneth Wong, Joshua B. Tenenbaum, and Chuang Gan. Grounding physical concepts of objects and events through dynamic visual reasoning. In International Conference on Learning Representations, 2021.
[8] Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B. Tenenbaum, and Chuang Gan. Comphy: Compositional physical reasoning of objects and events from videos. In International Conference on Learning Representations, 2022.
[9] Pim de Haan, Taco S Cohen, and Max Welling. Natural graph networks. Advances in Neural Information Processing Systems, 33:3636-3646, 2020.
[10] Mare Finzi, Max Welling, and Andrew Gordon Wilson A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. In International Conference on Machine Learning, pages 2318-3328. PMLR, 2021.
[11] Martin A Fischler and Robert C Bolles. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):31 1395, 1981.
[12] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. teansftrasssoommrs:: 3d roto-translation equivariant attention networks. In Advances in Neural Information Processing Systems, volume 33, pages 1970-1981. Curran Associates, Inc., 2020.
[13]
Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas,, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Tyler Feigelis, Daniel Bear, Dan Gutfreund, David Daniel Cox, Antonio Torralba, James DiCarlo, Joshua B. Tenenbaum, Josh Medermott, and Daniel LK Yamins. ThreeDWorld: A platform for interactive multi-modal physical simulation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
[14] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pages 1 PMLR, 2017.
[15] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
[16] Jiaqi Han, Yu Rong, Tingyang Xu, and Wenbing Huang. Geometrically equivariant graph neural networks: A survey. arXiv preprint arXiv:2202.072.33 2022.
[17] Wenbing Huang, Jiaqi Han, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang, Equiv- ariant graph mechanics networks with constraints. In International Conference on Learning Representations, 2022.
[18] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In Proceedings the the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2688-
2697. PMLR, 10-15 Jul 2018
[19] Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example, 2016.
[20] Sizhe Li, Zhiao Huang, Tao Du, Hao Su, Joshua B. Tenenbaum, and Chuang Gan. Contact points discovery for soft-body manipulations with differentiable physics. International Conference Learning Representations, 2022.

---

[21] Yunzhu Li, Toru Lin, Kexin Yi, Daniel Bear, Daniel Yamins, Jiajun Wu, Joshua Tenenbaum, and Antonio Torralba. Visual grounding of learned physical models. In International conference on machine learning, pages 5927-5936. PMLR, 2020.
[22] Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In International Conference on Learning Representations, 2018.
[23] Xingyu Lin, Zhiao Huang, Yunzhu Li, David Held, Joshua B. Tenenbaum, and Chuang Gan, Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools. In International Conference on Learning Representations, 2022.
[24] Pingchuan Ma, Tao Du, Joshua B Tenenbaum, Wojciech Matusik, and Chuang Gan. Risp: bondering-invariaan state predictor with differentiable simulation and rendering for cross- domain parameter estimation. ICLR, 2022.
[25] Joshua Mitton and Roderick Murray-Smin Smith.. Local permutation equivariance for graph neural networks. arXiv preprint arXiv:2111. 1e:::2211.11844 2021.
[26] Damian Mrowea, Chengxu Zhuang, Elias Wang, Nick Haber, Li F Fei-Fei, Josh Tenenbaum, and Daniel L Yamins. Flexible neural representation for physics prediction. Advances in neural information processing systems, 31, 2018.
[27] Tobias Pfaff, Meire Fortunato, Alvaro Janchez-Gonzalee and Peter Battaglia. Learning mesh- based simulation with graph networks, In International Conference on Learning Representations. 2021.
[28] Yulia Rubanova, Alvaro Sanchez-Gonzalez, Tobias Pfaff, and Peter Battaglia. Constraint graph network simulator, 2022.
[29] Alvaro Sanehez-Gonzalez, Victor Bapst, Kyle Cranmer, and Peter Battaglia. Hamiltonian graph networks with ode integrators. arXiv preprint arXiv: 12790, 2019.
[30] Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Conference Machine Learning, pages 8459-8468. PMLR, 2020.
[31] Alvaro Sunchez-Gonzaled Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Ried- miller, Raia Hadsell, and Peter Battaglia. Graph networks learnable physies engines for inference and control. In Proceedings of the 35th International Conference on Machine Learn- ing. volume 80 of Proceedings of Machine Learning Research, pages 4470-4479, PMLR. 10-15 Jul 2018.
[32] Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural networks. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9323-9332. PMLR, 18-24 Jul 2021.
[33] Erik Henning Thiede, Wenda Zhou, and Risi Kondor. Autobahn: Automorphism-basee graph neural nets. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021.
[34] Nathaniel Thomas, Tess Smidt, Steven Kearne,, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equiiariaan neural networks for point clouds. arXiv preprint ar arXiv: 1802:082192099999 19, 2018
[35] Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars universal: Equivariant machine learning, structured like classical physics. In Advances in Neural Information Processing Systems, 2021.
[36] Maurice Weiler and Gabriele Cesa. General (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems, 32, 2019.
12

---

[37] Yufei Ye, Dhiraj Gandhi. Abhinav Gupta, and Shubham Tulsiani. Object-centric forward modeling for model predictive control. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura, editors, Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 100-109. PMLR, 30 Oct-01 Nov 2020.
[38] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Toralla,, and Joshua Tenenbaum. CLEVRER: Collision events for video representation and reasoning. ICLR, 2019.
[39] Yaofeng Desmond Zhong, Biswadip Dey, and Amit Chakraborty. Extending lagrangian and hamiltonian neural networks with differentiable contact models. In A. Beygelzimer, Y. Dauphin, Liang, and Wortman Vaughan, editors, Advances in Neural Information P Processing Systems, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] Our contributions stated in the introduction.
(b) Did you deseribe the limitations of your work? [Yes] In Sec. 5.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In Appendix.
(b) Did you include complete proofs of all theoretical results? [Yes] In Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material URL)? [Yes] See our supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] In Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] In Sec. 4.
(b) Did you mention the license of the assets? [Yes] In Appendix.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "qg9kA51oOC", "reviewer": "Reviewer_bh6f", "summary": "The paper introduces a new model for learning physical dynamics. It introduces a concept that is termed subequivariance, which appears similar to restricted representations from group theory. The paper also introduces a hierarchical message passing to better enable long range interactions between particles. The paper also expands the input feature space by computing geometric features of the input particles and their objects. The paper experiments on a range of dynamics datasets, namely Physion and RigidFall. ", "questions": "* How does subequivariance differs from using the restricted representation?\n* How does including object information differ from being a specific feature space choice?\n* How does the choice of using different edges to pass between objects/particles differ from an automorphism equivariant network?\n* Why are only rotations applied around the gravity axis in the experiments?", "limitations": "N/A", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper adds a \"binary operation\" which is the difference in position between adjacent features which adds a translation invariant representation to the input feature space, although the benefits and justification for this are not adequately detailed. \n* The experiments seem thorough and multiple baselines are used, although I am not famailir with these baselines so I cannot comment on whether any relevant baselines have been missed.\n\nWeaknesses:\n\n[major]\n* A key concept of the paper is characterising \"subequivariance\" as a relaxation of equivariance focusing on the group E(3). Although this concept has already been considered previously in [1,2] by using restricted representations of the group rather than the regular representation.\n* A criticism of GNNs made in the paper is that they seldom explicitly involve the object information into the message passing, but this feels like a specific decision choice of the input features rather than some fundamental issue of GNNs which a paper can solve. Also, there are papers which consider specific object information such as [3].\n* The concept of having different edges which pass different information appears strongly related to a relevant sub-field of GNNs exploring automorphism equivariance which has been ignored in this paper [4,5,6].\n* In the experiments only rotations around the gravity axis are used. Surely, all rotations should be possible and all the models which do not correctly account for the break in symmetry caused by gravity should produce unrealistic results, while if this model works correctly it should break the symmetry and produce realistic results. I feel like this would be a far stronger result proving the correctness of the model. \n\n[minor]\n* Multiple typos / grammatical mistakes makes reading the paper difficult. (L44, 59)\n* The object features appear to be the pooled features of its particles, which is used to create a feature space by subtracting these from each of the particles features. This seems to be a complexly worded way of saying subtracting the mean from the particle features which is a common approach in ML.\n\n\n[1] Weiler, M. and Cesa, G., 2019. General e (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems, 32.\n[2] Cesa, G., Lang, L. and Weiler, M., 2021, September. A Program to Build E (N)-Equivariant Steerable CNNs. In International Conference on Learning Representations.\n[3] Pfaff, T., Fortunato, M., Sanchez-Gonzalez, A. and Battaglia, P.W., 2020. Learning mesh-based simulation with graph networks. arXiv preprint arXiv:2010.03409.\n[4] de Haan, P., Cohen, T.S. and Welling, M., 2020. Natural graph networks. Advances in Neural Information Processing Systems, 33, pp.3636-3646.\n[5] Thiede, E., Zhou, W. and Kondor, R., 2021. Autobahn: Automorphism-based graph neural nets. Advances in Neural Information Processing Systems, 34, pp.29922-29934.\n[6] Mitton, J. and Murray-Smith, R., 2021. Local Permutation Equivariance For Graph Neural Networks. arXiv preprint arXiv:2111.11840.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "lT_sXJXICqP", "reviewer": "Reviewer_tCa3", "summary": "This paper targets an interesting question of embedding equivariance into graph neural networks for physical dynamics recovery. The main focus is to 1) relax the strict constraint for cases with gravity and 2) consider the differences in self- and mutual interactions during learning.", "questions": "The review has some questions about the work.\n1.\tIn Sec. 3.1. the authors define the problem target as to predict the position of the next time step (x^{t+1), where the position of the current time step is input (x^t). The reviewer would like to know if the proposed method is generalizable to a broader problem setup. There are many physical systems whose modeling can be seen as a mapping from one input quantity to one output quantity (xy). The graph structure is also applicable. How to generalize the proposed method then?\n2.\tThe structure of the proposed Subequivariant Graph Neural Networks needs to be clarified. The reviewer suggests adding a flow chart around Sec. 3.2 -3.3. \n\n", "limitations": "Please see the questions above.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This work targets the learning of physical system dynamics, which is an important topic and needs more attention in the community. The authors provide a clear illustration of the relevance to existing works on 1) graph neural networks to model interactions among particles/objects and 2) using physical constraints as an inductive bias to improve generalizability. The authors mainly propose subequivariance against the case with gravity and design multi-stage modeling to account for differences in object properties such as shape. To the best of the review’s knowledge, this work is new. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NBB1xZ-9r2G", "reviewer": "Reviewer_Y6Jq", "summary": "The manuscript improves the equivalent graph neural network (GNN) to tackle the possible inefficiency when the model predicts the dynamics of a physical system when the system symmetry is partially broken by an external force such as gravity.  The subequivariant GNN (SGNN) proposed in this work introduces the hierarchical architecture in terms of the particles and objects where the subequivariant message passing is incorporated into the model to deal with complicated object interactions. The approach with the additional freedom can be more accurate in the task to evaluate the physical dynamic of objects from vision and achieve an impressive generalization compared with GNS model etc. ", "questions": "Q1: As mentioned previously, the author should make a better introduction to make people easy to understand the background and motivation as well as the challenge of the task. \n\nQ2: It seems the work is similar to the development of the machine learning force field (MLFF) in molecular dynamic simulation. Probably, the task predicting the physical dynamic of the objects has less information from the vision system than that can be obtained in an atomic system. It would be better if the author could clarify the difference between the two tasks. \n\nQ3: If the SGNN mode is capable of predicting the force field of the atomic system, it would be interesting to compare with the MLFF models. There are many recently developed models such as Nequip and TorchMD incorporating the rotation equivariance and achieving impressive high accuracy while predicting the atomic force field.\n\nQ4:  Besides Physion, is there any other dataset for further verifying the performance of the model? Unlike SGNN, it seems the compared models, such as GMN and EGNN are general models and not specifically developed for this task. The comparison may not be fair. It would be better to compare with the model which is dedicated to this task. Otherwise, the SGNN should be compared with other models in different tasks.\n", "limitations": "Yes. This is a work dedicated to developing the machine learning model for science. There would be no potential negative societal impact.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:  The work introduces the concept of subequivariance based on the hierarchical network to extend the capability of GNN in the task of learning the physical dynamics from vision information. The work has very good originality. \n\nQuality: The model is developed rigorously based on the understanding of the rotation equivariant group. The design and test of SGNN with Physion dataset are thoughtful and convincing. \n\nClarity: The presentation of the manuscript is clear. However, the background and the motivation are confusing. It is a bit hard to understand the challenge of the problem until I read the online introduction of Physion dataset. \n\nSignificance: Though I may not consider this work as a breakthrough of GNN model development in the focused task, the fresh idea, such as the subequivariance described in the manuscript, indeed helps the neural network achieve better accuracy and generalization. The work can bring people to pay attention to the importance of developing a model with appropriate physical constrain for a similar task.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7ovg1IKQrEs", "reviewer": "Reviewer_cLqG", "summary": "In this work, authors present a new formulation of the equivariant graph neural network, namely, subequivariant GNN (SGNN), which allows the modeling of systems with symmetry breakage such as gravity. In addition, to model systems with different shapes and geometry, an additional feature that represent the object type is added so as to distinguish the intra-object (elasticity/rigidity/plasticity) and inter-object interactions (collision/repulsion/weak attraction). A hierarchical modeling approach is implemented to address particle- and object-level interactions separately. By considering different physical scenarios, for instance, collision and contact prediction, the superior performance of SGNN is demonstrated.", "questions": "Some of the questions that naturally follow from the previous sections and some additional questions the authors should address are mentioned below.\n1) Since one of the main aim of the present work is to simulate realistic contact models, additional baselines which employ contact models in the Lagrangian/Hamiltonian neural network framework may be considered. Indeed, they may not scale as the GNNs, but they can potentially give improved conservation of physical laws than purely data-driven approaches. Also, this will provide insights into the deficiencies of SGNN.\n2) Authors have not evaluated how realistic the trajectory is with respect to the physical laws. For instance, is the energy and momentum conserved in the collision, is the coefficient of restitution 1, etc. These are important to analyze, since the claim is that the SGNN provides improved dynamics (Q2 in results). \n3) Again, since the contact is of main concern in the present work, authors should evaluate how the model performs in situations where there is self contact. At present, it is not clear whether the formulation is capable of dealing with such scenarios.", "limitations": "There are several limitations for the present work, which the authors should consider.\n1) Although a particle-based approach is employed, unlike previous works such GNS, there are no discussions on deformable systems. It is not clear how much improvement SGNN can give for deformable systems.\n2) As mentioned in the questions, discussions on self-contact and how this is incorporated is missing.\n3) Performance of SGNN on systems with drag and other dissipative forces are lacking.\n4) In the case of a deformable system, consider a scenario where a particle from a given object breaks away, gets reflected by the wall, and then comes back to interact with the same object. In such case, if the particle comes with the \\varepsilon cutoff distance, does SGNN model this as a contact or as a particle of the object. In other words, does the particle \"heal\" with the object or not. If yes, this is unphysical. It is not clear if the model can address this issue.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, the work is well-written and clearly presented. It builds on the equivariant GNNs and modifies it to present the idea of SGNN, which can include directional symmetry breaking such as gravity. TSome of the main comments regarding the work are as follows. \n1) While the idea is useful, the implementation and proofs are fairly straightforward extension from the equivariant case. Also, no additional inductive biases to preserve the physics (as in the case of Hamiltonian or other physics-informed GNNs) are implemented. This raises a question on the validity of the trajectory predicted. Specifically, no comments on whether the trajectories represent a physically feasible realization is not discussed. This is important because one of the major advantages the authors claim for SGNN is the ability to \"learn\" the dynamics. \n2) Authors refer to previous works such as Hamiltonian GNNs, and mention that they do not consider rotational equivariance. This is incorrect. In Hamiltonian GNNs, the edge embeddings can be modified to have the distance (L2 norm of the difference in positions), instead of giving simply the vectorial difference. Since the functional form to be learned in the case of a Hamiltonian is scalar, this approach also works very well and is both translationally and rotationally invariant. \n3) The examples demonstrated in the work are those, where contact seems to be of main interest. It is not clear how much better the model would perform in other cases where contact is not necessarily the primary interest but dynamics is. Now, if the main focus of the work is simulate contact, then there has been several works which attempted to do this (for instance, Zhong, Y.D., Dey, B. and Chakraborty, A., 2021. Extending lagrangian and hamiltonian neural networks with differentiable contact models. Advances in Neural Information Processing Systems, 34, pp.21910-21922.), which employ a similar to idea of cutoff-based contact detection. Indeed, they do not employ a graph-based approach. However, in the present case although the particles are considered, there do not seem to be any deformation simulated and hence, these approaches referred should also stand equally valid as SGNN.\n4) Also, the experiments chosen seem to be favorable for SGNN in comparison to the other baselines. For instance, implementing EGNN and GMN, exactly as they are, with gravity is expected to yield poor performance as the architecture expects the data to be rotationally invariant. Similarly, GNS and DPI learns purely from data and hence are unaware of symmetry unless trained specifically for it. More interesting examples where the situations where GNS and EGNN have been shown to yield SOTA performance could give a more realistic representation of how better SGNN is in comparison to these models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept