Autoregressive Graph Network for Learning Multi-step Physics
Authors: Sakthi Kumar Arul Prakash
Conrad Tucker
Venue: Submitted to ICLR 2023 (2023)
Keywords: graph network
autoregressive model
physics simulation
forward model
inverse model

Abstract: In this work, we propose an Autoregressive Graph Network~(AGN) that learns forward physics using a temporal inductive bias. Currently, temporal state space information is provided as additional input to a GN when generating roll-out physics simulations. While this relatively increases the network's predictive performance over multiple time steps, a temporal model enables the network to induce and learn temporal biases. In dynamical systems, the arrow of time simplifies possible interactions in the sense that we can assume current observations to be dependent on preceding states. The autoregressive property naturally induces the arrow of time and can further constrain physics-induced GNs to conserve symmetries over long time-steps. Our proposed GN encodes temporal state information using an autoregressive encoder that can parallelly compute latent temporal embeddings over multiple time steps during a single forward pass. We perform case studies that compare multi-step forward predictions against baseline data-driven GNs across diverse datasets that feature different particle interactions. Our approach outperforms the state of the art GN and physics-induced GNs in 9 out of 10 and in 7 out of 10 particle physics datasets when conditioned on optimal historical states. 

Under review as a conference paper at ICLR 2023
AUTOREGRESSIVE GRAPH NETWORK FOR LEARNING MULPI-STEP PHYSICS
Anonymous authors Paper under double-blind review
ABSTRACT
In this work, we propose a Autoregressive Graph Network (AGN) that learns for- ward physics using temporal inductive bias. Currently, temporal state space in- formation provided as additional input to GN when generating roll-out physics simulations. While this relatively increases the network's 's predictive performance over multiple time steps, a temporal model enables the network to induce and learn temporal biases. In dynamical systems, the arrow of time simplifies possible interactions in the sense that we can assume current observations to be dependent preceding states. Our proposed GN encodes temporal state information using an autoregressive encoder that can parallelly compute latent temporal embeddings over multiple time steps during a single forward pass. We perform case studies that compare multi-step forward predictions against baseline data-driven one-step GNs as well multi-step sequential models across diverse datasets that feature different particle interactions. Our approach outperforms the baseline GN and physics-induced GNs in 8 out of 10 and in 8 out of 10 particle physics datasets respectively when conditioned on optimal historical states. Further, through an energy analysis we find that our method not only accumulates the least roll-out error but also conserves energy more efficiently than baseline Graph Transformer Network while having an order magnitude lesser parameters.
INTRODUCTION
In the recent years, there has been a growing interest in learning physics with the help of deep learning coupled with other techniques such as inductive biases, physics informed loss functions and meta-learning (Fragkiadaki et al.) (2016); |Battagiaa et al. (2016); Xu et al. (2019); HHll et al. (202T). ) Relational networks such as Graph Networks (GNs) can decompose and learn the dynamics of physics system on the basis of particle interactions within their neighborhoods (Battaglia et al. (2016); Li et al. (2018); Sanchez-Gonzalez et al. (2020)) Across science and engineering, particle states often contain system and particle specific properties such mass, density, velocity, particle system.
type, etc. that are required to approximate the dynamics
In general, given the current state of system of particles along with particle specific local prop- erties and global system properties, it is possible to apply GNs to predict the trajectory of the sys- tem (Sanchez-Gonzaz Gonzalez al. (2018: 2020 Often referred as the forward problem, it assumes knowledge about the physical properties of particles and therefore utilizes the observations to con- struct a suitable model that predicts the trajectory of the system of partieles. The solution typical forward dynamics problem governed by an ODE involving particles can be parameterized using a GNN by learning from the current state or by using a history of previous particle states. There are strong benefits to training on entire sequences or multiple time-steps (Mobajerin) (2017); Xu et al. (219)) as one-step GNs tend to be unstable and accumulate error the long-run. While prior work has shown that concatenating history of previous states enables a trained simulator such as GNN to predict the next state more accurately, sequential model captures certain symmetries, e.g., arrow of time, conservation of energy, momentum .etc. Sequential models such as RNN, LSTM, GRU and Transformers have been applied to ID time series and N-body systems (Chen et al. (018);; Zhang et al. (2020); Han et al. )2022). While appealing choices model dynamical systems due to their implicit memory mechanisms, they require sequential computations that come with significant memory overhead as the lookback length and/or the dimensionality of the problem increases.

---

Under review as a conference paper at ICLR 2023
To avoiding sequential computations while retaining relational information about previous states, masking has been employed as a successful strategy in enabling ferd-forward neural networks to enjoy the best of both worlds (Germain et al. (2015); Van Den Oord et al. 2016); Papamakarios P et al.
(2077). Autoregressive models have been widely used across Machine Learning to learn conditional dependencies to model distributions Rezende & Mohamed (2015); Papamakarios et al. (2017) and
learn long range predictions using masked/causal Transformer like models (Ghazvininejad et al. (2019) Shi et al. (020); Han al. (2022). Further, to improve predictive performance. the original feature space is often mapped to hidden latent space on which Transformers and the family of sequential models operate. The map from the orginal feature space latent space at each instant in time is often performed using linear layers such as MLP. Much of the previous work on Physics based deep learning has found success learning a latent linear representation latent feature space. While MLPs can model the relational information between features as an undirected and fully connected graph, a recurrent or autoregressive learns a causal graph between them. In particle based system, such causal graph can encode numerous dependencies between state space variables. Fig. Fig. Tillustrates the common causal relations learned by sequential models and the structural differences in graphs encoded in the latent space. The key difference between using an autoregressive and non-autoregressive graph to map a state space to a latent space condenses dimensional feature vector mapping & embeddings. Each of these additional embeddings capture different causal relation between state space variables. Our central hypothesis therefore is that latent space encoded with a causal graph enables particle based ML surrogate models to learn long range dependencies that obey conservation principles much better than an undirected graph. Therefore, in the absence of structure on the state space variables, inductive biases such as the next states of a system of particles affecting the initial states are learned, when in practice, such scenarios are not encountered when learning to approximate the dynamics of a forward simulation problem.
the dynamies in
between mapping particle's single
latent space instead of
Autoregressive Graph
Non-Autoregresssvv Graph
Vector Autoregressive Graph
Legend
Vartable Node
Temponal Shate Noda
Directed Edge
Undirected Edge
Poslion allng soda time 15
Posiiio along sods 2 lime
Scalar Autoregressive Graph
Velccity along tics time
#
Vecotyy along axis time
Figure 1: Structure of the state space with/without auto-regressive property. The temporal state nodes constitute an asymmetric or DAG-like temporal graph which further contains a state space graph within each node. The directed edges do not allow interactions between the previous and the next state-variables, while the undirected edges allows such interactions.
The key contributions of the proposed approach are as follows:
An Autoregressive Graph Encoder (AGN) that explicitly induces the arrow of time (i.e. previous states affect the future states) bias capture causal relations between state space variables on the latent space.
The induced temporal bias enables the Graph Network to achieve superior energy conser- vation and roll-out error accumulation performance across long time steps.
Comparable multi-step prediction performance against a Graph Transformer model whilst requiring order of magnitude lower parameters.
2

---

Under review as a conference paper at ICLR 2023
2 RELATED WORK
Engineering simulations developed for prediction and control of complex physical systems can be built based on empirical or theoretical findings. Compared analytical models, learned simulator can be more efficient at predicting complex phenomena (He et al. (2019); Sanchez-Gonzalez et al. (2020). Further, domain knowledge can be encoded as physics priors in the form of loss function regularization (Raissi et al. (2019)) and data featurization to improve accuracy and performance. However, explicitly providing knowledge may limit generalizability to different datasets that are fundamentally governed by the same underlying dynamics. Data-driven approaches learn the dy- namics of system without explicitly solving an ODE or PDE. The physical interactions between particles are learned implicitly by nature of the model construction (Battaglia et al. (2018); Chen ct al. (2018); Greydanus et al.] (2019); Cranmer et al. (2020a b):
et al. (2020);
Pfaff et al. (2021); 1); Rubanova et al. (2022)) therefore enabling generlization across similar class of problems. Such neural networks have shown to approximate solutions of ordinary differential equations (ODE) that govern the dynamics of a system and have also shown be compatible with most off-the-shelf ODE explicit solvers (i.e., Euler, Lunge-Kutta)) While successful, such meta-
models have only been trained to learn the forward dynamics of the system based on the current state information or by concatenating history of states.
Deep implicit models (Amos & Kolter (017);; Bai et al. (2019 2020 are powerful class of implicit learning methods proposed for learning constraints as well as learning to solve constraints. Often these models work around the computational expense of computing first and higher order gradients required by numerical optimization solvers such as gradient descent, Newton's 's method, .etc. Classical mechanics present a mechanistic understanding of reality that can be captured using such implicit or constrained models. Symmetries such as conservation of energy and momentum can be implicitly learned by constraining the gradients of the networks to satisfy LagrrnLagrange or Hamilton's 's equations (Chen et al. (20 8);; Lutter et al. (2019;; Greydanus et al (2019); Cranmer al. (020a)). Recent work ( (Yang et al. (2020); Rubaanva et al. (2022) have proposed implicit neural network strategies to learn physics-based based constraints. These methods frame the forward problem as constrained optimization problem. The Neural Projections method iteratively proposes a prediction of the next state of system with explicit Buler step to solve a root-finding problem, then projects the prediction onto a learned constraint manifold that is implemented as a multilayer perceptron (MLP). A recent work (Han et al. (2022) proposes a sequential model with a transformer operating in the latent space to perform autoregression on latent temporal embeddings of particles. While their approach helps manage the dimensionality of the problem so as limit memory consumption of the sequence model, it still requires & sequential computations.
In this work, we induce an arrow of time constraint on the encoder parameterization by modifying it as an autoregressive encoder (Germain et al. (2015) that encodes the nodes and edges sepa- rately with modifications to suit n dimensional physics problems. The Masked Autoencoder for Distribution Estimation (MADE) model proposed by (Germain et al. (2015)) only enforces scalar autoregressive property that may not suit vectors denoting a single state of a particle. By changing the manner in which the masking scheme is implemented, we generalize the MADE model to have a vector autoregressive property that can be used to generate temporal latent node and edge embed- dings. Put together with message passing network, we show that our model is able to model time dependencies better and lead to slower error accumulation over long time-steps.
3 APPROACH
3.1 LEARNED SIMULATOR OVERVIEW
The dynamics of a system of N particles can be described by a coupled system of Ordinary Differ- ential Equations (ODEs) that takes the following general form:
d x(t = f(x(t),t,w) पi
(I)
where, x(t) [r r|7 represents the state of the system at time t with E Nxd denoting the position vectors and i E IR Nxd denoting the velocity vectors of N particles in d dimensions. The particles also possesses physical property such mass, charge, density, etc. and is denoted by
3

---

Under review as a conference paper at ICLR 2023
Rue Sumulation
Construct
B
xc(() |e 11 1 +5)
A
(1(1,, x(1 k
(() [e', rr
(1 2: 21
Exnmple: Gravitrionally Influenced Partccls
Ausorageendi
Nodi
Fdge Teaader
GNN GNN Block
Emoxien (MLF)
Eacoder (MT
Message Passing
ONN Block
Edox
Connections
mrasks
C
GNN Block
Rasodier (MT
Tdge
Made
F(E)
Figure 2: Schematics: A) Illustrates data and corresponding particle graph generation, B) Vector Au- toregressive Graph Network, C) A single layer Vector Autoregressive Node Encoder with masks in the output layer that generates 4z dimensional latent vector for every node i. The directed colored arrows denote dependence.
RN, In general this problem requires an approximate numerical solution that is acquired by solving the ODE subject to the initial value condition x(0). In this paper, we are interested in learning the dynamics of the system (f) which can typically be complex using parameterized function approximator (f(.)), where (-) denotes the parameters to be learned. Further, since we adopt a graph-based representation of the system of particles, we consider particles as nodes (V) in graph (G) such that every particle has first-order neighborhood as well as hops of higher-order neighborhoods. Hence, every system state (x(t) has a corresponding adjacency matrix (A(t) which matrix representation of (t)) EV.
To solve the forward problem, the learned simulator fo x(t) + x(() maps the current state of the system of particles time t to its first-order derivative which is the velocity and acceleration vectors of the system particles. We note that the training is supervised on acceleration labels only. Then, using choice of numerical integrator (i.e., Euler, Runge-Kute the next state x(t 1) can be obtained from (()) = [t ij..
3.2
LEARNING FORWARD DYNAMICS
3.2.1 VECTOR AUTOREGRESSIVE (VAR) ENCODER
Following the established notations, we implement the learnable function fg using Graph Net- works (GN) (Battaglia et al. (218))) arranged in the Encode-Process. Process-Decodee architecture, similar to previous work on GN-based learned simulators Maachez-GGonzalee et a.l. (2020): |Pafff at. (202T) In this paper, we generalize the notion of autoregression on scalars to vectors via Vector autoregres- sion (VAR). Specifically, they explicitly induce the notion of arrow time wherein the future states do not affect the past states. However, a non-auto-reeresssve graph captures dependencies wherein the past and future states affect each other, which typically not the case in forward simulation problems. As result, the network
whole tends leam biases that may not hold beyond train- ing. In the simple case 2D problem, the motion of each particle a trajectory can be described through vector ([r), r2, F1, 122 2) in dimensional state space. Fach element of the vector is related
(2)nn

---

Under review as a conference paper at ICLR 2023
to another as well as the next state's vector through unknown coefficients (eg. parame- terized as Neural Nets). This can compactly be captured using linear model such as VAR. In the example below, show a multivariate VAR model of order 2 with noise term (e').
W
wi,
wf,
wi
wf.
w(.
wf.r
w
22.1
w2.2
w2.3
22.4
w2.5
wi.8
w1.7
w/
Wa.1
W4.3
33,4
wi,
wi.7
wi
Wa.1
wh.s
Wi.
w1.7
wi.
+
(2)
u
u5,2
35,4
W6.1
W6,1
0
0
W7.1
W17.
88,1
188,2
18,33
18,4
In Eqn. 2, W W M", where W represents parameterizable autoregressive coefficient matrix such that elements (wrss, 1:4) and M is mask the size of W that be used to induce the desired dependencies generalzzable manner between input and output variables of interest. the case above, the matrix M creates the following temporal dependencies: r' r'((r'-), , -'- Since seek to parameterize such VAR process using neural network, analogy in the context of a an autoencoder with a single hidden layer. We drop the noise term in our NN formulation. Let the output of the first hidden layer be denoted as H(x) = act(b (b+ +-W+(WW)) (WoM )x), which is further used computing the prediction (X), such that & = act(c + (V M Hx))). In the expressions above, act implies an activation function (typically ReLU), M and MV are the masks that impose the autoregressive property while V and W the weight matrices that parameterize the autoregressive coefficient matrix for a VAR based encoder order k..
show similar
To design set of masks that preserve the expected input-output relations, begin by assigning each element x of the dimensional state vector x()) an integer / corresponding to the 1hh time step and each hidden unit in the hidden layer integer p as function (m(s, ()) of and Hence, the 8th hidden unit's dependence can be computed as follows: m(s,t)
mod d + (d (s mod d)) + dt d(1 + t) p gives the maximum number of input units which can be connected. Further, we allow m(s,t) = P, where P = kd, while not allowing m(s, f) = so as ensure that the current state as well as the 1 lag state are conditioned upon when generating nigh-dimensional temporal latent vectors as opposed to probabilisitic conditionals or prior parameterization in generative models.
The conditional temporal latent vectors can be generated using deep autoencoder parameterized with the help of the following masking rules. In the single hidden layer case, the constraints on the maximum number of inputs each hidden unit are encoded in the matrix masking the connections between the input and hidden units:
[m(ss))2p: if m(, 2p otherwise
MW s.p
0.
for E {d, 2d,...,,kdf kd]. (l,..,, 5} and (0....,k: 16-1,, Further, we encode the constraint that the th network output is connected to all m(s.t) p so that the output weights can connect the tdh network output to hidden units with m(s,t) t) I These constraints can be encoded in the output mask matrix follows:
( 1p22ms..)) p,a 10.
if 2 m(s,t) otherwise
M
(4)
The proposed encoder architecture naturally generalizes to deep architectures. To impose the same constraint on the succeeding hidden layer, we just have to ensure that each hidden unit s' in the subsequent layer gets connected to the preceeding layers such that m (s, < ml|1 (s', ((s,,)) t), where I denotes the layer number. The rule given by Eqn. (3) can be generalized
any layer I. The last hidden layer (L') should preserve the natural ordering (vector autoregressive graph) by enfore- ing Eqn. (4). Further, we are interested to generate latent temporal embeddings of varying size (2). To do so, we create copies of final layer masks such that each mask generates an embedding the size of the input vector (i.e. if the input of node has size of |xs], then the vector autoregressive
5

---

Under review as a conference paper at ICLR 2023
output of the encoder will be of size |x,). We pick value by matching the encoder output size with that of a typical autoencoder implemented in a GN.
We have vector autoregressive node encoder (V.ARy) and an edge encoder (VAARg) whose cor- responding output node and edge embeddings are denoted as H and H respectively. The input to V.ARg [x,(t - k) (t k), (t - k) xt(t k).... (t) (t),), (t() - (t) repre- senting the relative position and velocity information between particles across & thyper-parameee time-steps, while that of V.ARy [r,(t- - k), (t - k)..... rt()), it; (t)) representing the full state vector between particles across le (hyper-parameter) time-steps.
3.2.2 PROCESSOR AND DECODER
Having described the graph encoders, we now briefly describe the graph processor and decoder along with their respective inputs and outputs. The graph processor computes interactions among nodes through I. message passing blocks arranged in sequence using standard MIP edge encoders & and node encoders εv. Message-passing preserves constraints while allowing information to propagate between nodes and edges. The processor takes as input the latent temporal embeddings (H) (0) and H!)) generated by the autoregressive node and edge encoders to perform message passing compu- tations. In the subsequent sections, we drop / for brevity.
The edge encoder (Ex) performs the following node-to-edge (V + E) message passing operation,
( E' & (II H
H H 1 H (1-1)
(5)
where, [. denotes concatenation and denotes the I-th graph network block. Following the edge encoding operation, the node encoder (Ev) takes input the edge embedding and performs the following edge-to-node (E V) message passing operation,
E V H (+1)
e()) (1) ( @ H
JEN.
The node encoder aggregates the edge features for target node across its neighboring nodes (N),, concatenates the target node features and finally adds the previous block's 's latent graph features using residual connections. Following the processing step, the output of the last message passing block gets passed to an MLP decoder that predicts the acceleration (f). See Supplementary Materials for more details implementation.
DISCUSSION
In this section, we discuss the key findings from our experiments. In order to evaluate the rigorous- ness of the proposed approach, we keep a number of hyper-parameter choices such as the number of hidden layers, hidden neurons, etc. constant. See Supplementary Materials for full details on datasets and choice of hyper-parametert
We benchmark our approach against the following work
one-step forward prediction: Graph Network (GN) (Sanchez- Gonzalez et al. (020)), Graph Lagrangian Network (GLN) (Cranmer et al. (2020a)), Graph Hamiltonian Network (GHN) (Grey-
danus et al. (019 Cranmer al. (2020b)). Graph Transformer Network (GTN) (Shi al.
(2020).
We also benchmark against sequential forward prediction models such as Gated Graph Recurrent Neural Networks (GGRNN) (Seo et al. (2018)) and by adapting the base Graph Transformer ar- chitecture of Haan et al. Han el al. (2022) without regularizing the latent space nor pre-computing the latent embeddings. While Plaff al. (2020) propose a GN for learning physics simulations, we note that their approach when applied to particle based datasets simply assumes the architecture as proposed by (Sanchez-Gonzales et al. (2020). See Supplementary Materials for more details on time-complexity analysis, model parameter count and baseline implementations.
4.1 PERFORMANCE OF FORWARD DYNAMICS PREDICTION
We train all baselines (except the sequential models) as well as our models on one-step target accel- eration predictions (i.e., Given x(t + k)), predict F(t Æ)) using 70:30 train, test ratio. Once trained, we sample 30 initial conditions that were not a part of the train/test distribution to generate 30 simulations over 1000 time-step roll-out using RK4 integrator.

---

Under review as a conference paper at ICLR 2023
The optimum lookback length (k*) is function of the model well as the dataset. While discuss the impact of l on the stability and forward prediction performance later, tables | and 4 report the Mcan Squared Error (MSE) multi step predictions when optimum k* is chosen for each model/dataset. See Supplementary Materials for more details on look back length analysis. We impose translation invariance by construction for all baseline models by explicitly not providing ab- solute positions as input. Through our experiments, we find that our approach performs substantially better when provided with absolute positions while baselines perform worse with absolute positions as input. While this is surprising, we provide reasoning for why this is the case. Please refer to the analysis in the supplementary material for more details.
partial
4.1.1 FORWARD SIMULATION COMPARISON WITH SEQUENTIAL MODELS
We arbitrarily fix k and compare AGN, along with other one-step and multi-step methods. We use GTN to perform both single-step step predictions (no temporal recurrence) as well as multi-step pre- dictions under the many-to-many setting wherein we input & states and predict i acceleration targets corresponding to each state. Table Showss the forward simulation performance across the methods with GTN (MS) outperforming the AGN (scalar) and AGN (vector) on 6 out of 10 datasets. Despite the slightly better performance of GTN (MS), AGN has an order of magnitude fewer parameters to learn and therefore can be trained faster to achieve GTN (MS) like performance. When comparing GTN (MS) with GTN (SS), the performance of GTN (MS) is substantial across all datasets except on 2D/3D Gravity (2) ) This difference in performance can clearly be attributed to the temporal bi- ases learned by temporal model that are simply not captured by non-temporal model. However, in comparison to GTN (SS), AGN performs substanially better across the board. While GGRNN being temporal model, it's performance is quite poor across the datasets and is outperformed by all single step methods that have been considered in this paper. This difference in performance can likely be attributed to the better expressiveness of the other methods (temporal and non-temporal), While we hypothesize that AGN could possible show comparable performance with GTN (MS) on different kt we instead analyze the energy conservation property of the models in the next section. See Supplementary Materials for additional forward prediction comparisons.
Table I: Mean roll-our error compar- Table 2: Roll-out error comparison between sequential and ison between one-step methods for an sequennial models when 5. SS-Single Step, MS- optimum A
Multi Step
Dutasets
GN
GIN
MGN (vec-
AGN (scalla)) (Ouno
Datasets
GN (8S)
GTN (55)
GTN (MS)
GGRNN (MS)
MGX (vaxtor)
AGX (scalar) 188)
(Ours)
nor]
(88)
(Ours)
Spring
1.77
1.36
Spring
3.9
2.68
1.36
20 Dumped
9.37
11.04
3.17
7,25
2D Damped
19:39
977.89
6.89
70
12:75
7:25
30 Gravity (2)
10.55
165.39
9.41
6.95
3B Gravity (:)))
14.96
165.39
4.67
33.78
27.58
17.59
Gravity (5)
3.58
Gravity
4781.57
61.30
Charge
Charge
515
Spring
10.57
4.76
7.4
Sprinn
6.34
2744
3D Dumped
16.13
21.52
13.46
12.23
3D Damped
202.71
251.98
13,92
887.01
15.75
20.26
3D Gravity (3)
2.19
0.45
7.38
9.97
3D Gravity (.5)
2.09
1266
0.55
224.04
7.38
19.55
Grasiity (5)
091
Gravity
800.004
1.70
1.60
Chargo
Charge
1.05
4.1.2 ENERGY CONSERVATION COMPARISON WITH SEQUENTIAL MODELS
Roll-out errors unfortunately do not narrate the entire story. Since energy and roll-out error share an intimate relationship, we compute and report (see 3 the energy MSE between the predicted trajectories and the ground-truth trajectories by measuring the total energy deviation. While we notice that GTN's (MS) roll-out eгror is superior across the datasets, we do not notice a similar trend when comparing their corresponding energy MSEs. We find AGN models, specifically the scalar AGN (SS) model to outperform GTN (MS) by conserving energy more efficiently across the trajectory. This difference is due to the fact that GTN (MS) utilizes linear encoder that does not capture conditional relationships between states (i.e., xf is not dependent on any of x x() However, as result of AGN models capturing such relations, we notice that it conserves energy much better than all baselines, especially on non-conservative systems (2D/3D damped spring). We hypothesize that energy continues to accrue, the roll-out MSE of GTN (MS) may reach higher-values than AGN the simulation is continued for longer time-steps. F and 4 further illustrate the dif- ference in roll-out and energy MSE for different k on non-conservative and conservative system
7

---

Under review as a conference paper at ICLR 2023
respectively. While we expected to see better prediction performance by the AGN (vector) model due to their added structure that captures state space relations, the simple and slightly less structured AGN (scalar) consistently outperforms the AGN (vector) model. See Supplementary Materials for additional energy analysis on single-step methods.
Table 3: Roll-out energy accumulation error comparison between sequential and non-sequential models when k- 5. SS-Single Step Model, MS-Multi Step Model; indicates an extremely large number (> than lc7)
Dnitanets
GNTSS)
GTN I88)
GEN
GORNN (MS)
MGN (vector)
MGN (scalar)
(SS))Ound)
(S8) (Our)
215 Spring
290.86
354.5
RG2.14
393,19
206.04
20 Damped
208.38
749412.46
S4597
55.93
371.3
Gravity
1884.96
83541.03
118145
2406.0
903.16
Gruvity (2)
93926622
66334.99
3181,42
2D Charge
65987244
265108.12
44.21
9053.59
190091.34
21687
Spring
701.13
255.47
734.28
1244.72
563.2
Dampped
3714.1
21768
401413
303 Gruvity
004.51
365.97
325.00
792164001
2345.27
5730.01
51 Gravity
4771.97
72661.00
483.10
15742.91
106459.40
Charge
21107500
12532542
22441.99
27209
2
200
600
175
130
600
100
5200
75
0-
200
400
600
800
2000
Lenghck
LoRRbOOK Longts
Figure 3: (3D Damped) is non-conservative system. Left: Roll-out error across models when k = 5. Middle: Roll-out error look back comparison. Right: Roll-out energy accumulation look back comparison. We crop the Y axis of the bar graphs using the median MSE across models for better visualization.
Fram at LSB bo (fu
AON DDPL
5030
2.75
CN
3.50
4030
3.0
3.25
3000
8.75
8:50
8:00
0.0
200
403
60C
300
1000
Loodbacc Length
Timestops
Figure 4: (3D Gravity (r2)) is a conservative system. Left: Roll-out error across models when k 5. Middle: Roll-out error look back comparison. Right: Roll-out energy accumulation look back comparison. We crop the Y axis of the bar graphs using the median MSE across models for better visualization.
4.1.3 PHYSICS CONSTRAINTS
Next, we perform experiments by modifying a GN and AGN to have physics biases (Lagrangian and Hamiltonian). Since these constraints require computing the gradients with respect to rd and
8

---

Under review as a conference paper at ICLR 2023
momentum at time t, they require absolute positions of the particles as input. Like AGN, em- play an autoregressive node and edge encoder to capture spatio-temporal latent embeddings that are then used as input approximate the Lagrangian and Hamiltonian of the parameteric learnable function (ft)) From table we can see that AGIN (vector and scalar variants) outperforms the baseline Graph Lagrangian Network in 8 out of 10 datasets while AGHN (vector and scalar vari- ants) outperforms the baseline Graph Hamiltonian Network in 9 out of 10 datasets when trained with the optimum k. Contrary the dominating performance of AGN (scalar) variant in the earlier comparison, AGLN/AGHN (vector) model seems to outperform AGLN/AGHN (scalar) variant in a majority of the cases. This shows that physics- induced models are able to exploit the state space structure to gain advantage over non-physics models. Unlike the non-physies induced models, the major argument underlying the success of Lagrangian and Hamiltonian networks is that they are able to conserve energy much more efficiently and for longer time-steps. To validate how well the proposed physics-induced models conserve energy, we perform a similar energy analysis (see Ta- 5 on the trajectories predicted by all physics-ing physies-indueed models. Specifically, we report the energy MSE corresponding to the optimum k therefore allowing us to infer its relationship with the roll-out MSE. In general, we find that GLN and AGLN conserve energy much better than GHN, AGHN and non-physics induced models. While there seems to be a tie between AGLN and GLN models in terms of energy MSE, we note that GLN is subject to higher deviation in energy accumulation, as observed from its performance on the 2D/3D charge and 3D Gravity (,2) datasets. Hence, we find that autoregressive models do indeed help already prysscs-biased models to preserve energy over long time steps. We report additional results for choice of other k values in the supplementary mate- rial. During training, we notice that both GLN and GHN are sensitive to choice of lookback length and a sub-optimal lookback length affects the prediction performance of error (instability). On the contrary, we find that our AGLN and AGHN models like their AGN counterpart are quite stable even when using sub-optimal lookback length.
well as causes a blow-up
Table 4: Mean roll-out error comparison across Physics Constrained GNs for an optimum }:*
Dutasets
GIN
GHNN
AGEN
AGLN
ACHI tor) (Ours)
AGHN
tor) (Ours)
(scalar)
(scalar) (Ours)
(Ours)
2D Spring
0.84
2,05
0.70
0.89
127
20 Damppe
654
425.80
231
16:00
3.88
Gravity
13.51
18.16
11.08
23 Gravity (5)
5:70
52.78
1.01
1.99
1.29
6.84
21 Charge
42.4
0.98
361
414
5.66
Spring
1,76
10:08
4:0
Dampped
46.22
2006
313 Gravity (5)
2.49
3.52
1.61
1.68
3.86
6.00
Gravity
9.24
1.41
672
0.86
0.75
1:00
Charge
Table 5: Mean energy accumulation comparison across Physics Constrained GNs for an optimum
k*
Datasets
GLN
GHN
(vec-
AGHN (scalar) (Ours)
tor) (Ours)
(scalar) (Ours)
bor) (Ours)
20 Spring
140,04
19374
274.46
232.19
124.1
SOF.T
Damped
29.04
44.73
333.88
80.6
Gravity
171.84
485.29
127,83
177.53
1160
1141.59
213 Gruvity
163.5
01908005
139300
161.13
211.10
87.81
2D Charge
10443.70
67054 94
8.51
192.64
350
303.63
Spring
246.02
203497
Damped
109.23
207.1
MD Gravity (1)
88.40
1297 12
262.81
150.54
31883
3742.82
Gravity
4080.84
453004
471.41
474.49
Charge
24641.9
131.86
11:35
2,07
CONCLUSION
In this paper, we introduce a Graph Network with temporal inductive bias to learn forward physics simulations. The temporal bias is enforced by an Autoregressive encoder that attends to the previous state inputs of system of particles. The proposed model circumvents sequential computations required to compute latent vectors and therefore does not sacrifice speed for accuracy and instead allows GN to enjoy the best of both worlds. Further, by capturing diverse state relations from trajectory of length k, it offers the best roll-out and energy MSE performances. This bias enables our model to perform robust multi-step predictions when compared against the baselines.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Brandon Amos and Zico Kolter. Optnet: Differentiable optimization as a layer in neural networks.
In International Conference on Machine Learning, pp. 136-145. PMLR, 2017. ISBN 2640-3498.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. Deep equilibrium models. Advances in Neural
Information Processing Systems, 32, 2019.
Shaojie Bai, Vladlen Koltun, and J Zico Kolter. Multiscale deep equilibrium models. Advances in Neural Information Processing Systems, 33:5238-52500 2020.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. Interac- tion networks for learning about objects, relations and physics. Advances in Neural Information Processing Systems, pp. 4509-4517, 2016. ISSN 10495258.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanzhez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre. Francis Song. Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 0t/.//00..02221 2018.
Ricky T Q Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural Ordinary Differential Equations. In S Bengio, H Wallach, H Larochelle, K Grauman, N Cesa- Resa-Bianchi, and R Garnett (cds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates, Inc., 2018 http:::: http::::///pceceeee iig.eeiiiigg.... wew.ssneuriii ips. wwtppppppppp....ceee 2201//fill.. pdf
69386f 6523
Miles Cranmer, Sam Greydanus, Stephan Hoyer, Peter Battaglia, David Spergel, and Shirley Ho. Lagrangian neural networks. arXiv,, pp. 1-9, 2020a. ISSN 23318422.
Miles D Cranmer, Alvaro Sanchez-Gonzalez, Peter W Battaglia, Rui Xu, Kyle Cranmer, David N Spergel, and Shirley Ho. Discovering Symbolic Models from Deep Learning with Induc- tive Biases. In Hugo Larochelle. Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Bal- can, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6- 12, 2020, virtual, 2020b. URL hash/c9f2f99 707 wwd8dddd112222 Bww.BBbbb2e22cee1eeeeee wbw.sssattbbbbsattt....oo. wwwtraat.tttt.............aa.. .tml
cc/ wttp:cppppppp...////////
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning Visual Predic- tive Models of Physics for Playing Billiards. In Yoshua Bengio and Yann LeCun (eds.), 4th In- fernational Conference on Learning Representations, (ICLR) 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http::/arxiv...am.//////// 07404
http://xxxvv..rgg...... org/ abs/1111
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. Made: Masked autoencoder for distribution estimation. In International conference on machine learning, pp. 881-889. PMLR, 2015.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. Constant-time machine translation with conditional masked language models. arXiv preprint
2019.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian Neural Networks. In H Wallach, H Larochelle, A Beygelzimer, F d'' texttuuuueee Alché-Bue, E Fox, and R Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL hntpss://proccedingg. nwwr.is....eu................... wttp:eewwpp.ss.eeeeeeeeeoeoo hctps//ccccaape/////1eeee 22ccBecaaceedeeett6cc33222222222
Paper. pdf
Eric J. Hall, Søren Taverniers, Markos A. Katsoulakis, and Daniel M. Tartakovsky. GINNs: Graph- Informed Neural Networks for multiscale physics. Journal of Computational Physics, 433:1-20, 2021. ISSN 10902716. doi: 10. 20..11/.1
Xu Han, Han Gao, Tobias Pffaf, Jian-Xun Wang, and Li-Ping Liu. Predicting Physics in Mesh- reduced Space with Temporal Attention. arXiv preprint a2Xiv:2201.09/ 13, 2022.
10

---

Under review as a conference paper at ICLR 2023
Siyu He, Yin Li, Yu Feng. Shirley Ho, Siamak Ravanbakhsh, Wei Chen, and Barnabás Póczos. Learning to predict the cosmological structure formation. Proceedings of the National Academy of Sciences, 116(28):13225-1388
2019. ISSN 0027-8424.
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning par- ticle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.01166, 2018.
Michael Lutter, Christian Ritter, and Jan Peters. Deep Lagrangian Networks: Using Physies as Model Prior for Deep Learning. BT - 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019. URL httpps::/ppenrevierr.. openreview. 1i
Nima Mohajerin. Modeling dynamic systems for multi-step prediction with recurrent neural net- works. 2017.
George Papamakarios, Theo Pavlakou, and lain Murray. Masked Autoregressive Flow for Density Estimation. In Proceedings of the 3/ International Conference on Neural Information Process- ing Systems, NIPS 17, pp. 2335-2344, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalez and Peter W Battaglia. Learning mesh- based simulation with graph networks. arXiv preprint atXiv:20/0.03409 2020.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzalerr and Peter W Battaglia. Learning Mesh- Based Simulation with Graph Networks. In 9th International Conference on Learning Repre- sentations, (ICLR) 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL net forum
https: htttps://ppenreeee
XP
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informe neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational Physics, 378:686-707, 2019. ISSN 0021-9991.
Danilo Jimenez Rezende and Shakir Mohamed. Variational Inference with Normalizing Flows. In Francis R Bach and David M Blei (eds.), Proceedings of the 32nd International Con- ference on Machine Learning, {ICML} 2015, Lille, France, 6-11 July 2015, volume 37 of (JMLR) Workshop and Conference Proceedings, pp. 1530-1538. JMLR.org, 2015. URL URLhttp: html
//proceedings.. .wr. wwt.ppeess/v3/reeeeeeess
Yulia Rubanova, Alvaro Sanchez-Gonzake Tobias Pfaff, and Peter Battaglia. Constraint- graph network simulator. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning. volume 162 of Proceedings of Machine Learning Research, pp. 18844 18870. PMLR, 2022. / wwwpppeesssi... http://ss/veee//aaaaaee2a22222a wttp:/rraanovaaaaam html
Alvaro Sanchez-Gonzaler Nicolas Heess, Jost Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. 35th International Conference on Machine Learning, ICML 2018, 10:7097-7117, 2018
Alvaro Sanchez-Gonzalez Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Confer- ence on Machine Learning, pp. 8459 8468. PMLR, 2020. ISBN 2640-3498.
Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In International conference on neural information processing, pp. 362-373. Springer, 2018.
Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong Wenjin Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009.03509 2020.
Aäron Van Den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In International conference on machine learning, pp. 1747-1756. PMLR, 2016.

---

Under review as a conference paper at ICLR 2023
Zhenjia Xu, Jiajun Wu, Andy Zeng, Joshua B Tenenbaum, and Shuran Song. DensePhysNet. Learning Dense Physical Object Representations Via Multi-Step Dynamic Interactions. In An- tonio Bicchi, Hadas Kress-Gazit, and Seth Hutchinson (eds.), Robotics: Science and Systems XV, University of Freiburg, Freiburg im Breisgau, Germany, June 22-26, 2019, 2019. doi: 11.5607/RSS.2211.....80
hellitttts://of..cmm
15607.160770 / /SSS. 201 9. XV. 64.
Shuqi Yang, Xingzhe He, and Bo Zhu. Learning physical constraints with neural projections. Ad- vances in Neural Information Processing Systems, 33:5178-511 51888851199110 2020.
Ruiyang Zhang, Yang Liu, and Hao Sun. Physics-informet multi-LSTM networks for metamodeling of nonlinear structures. Computer Methods Applied Mechanics and Engineering. 369:1 13322, 2020. ISSN 0045-7825.
12


Reviews:
[{"review_id": "cy4x1RrZZ7", "reviewer": "Reviewer_xSTt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to add a linear node and edge encoding layer before the message passing steps of a graph network in order to improve inference performance. This encoder is employed for physics problems where the networks have the task to infer accelerations for time integration of the system under consideration, and its main job seems to be a selection of features from current and previous time step information. This approach is demonstrated for four different, but relatively simple types of particle-based systems.\n", "summary_of_the_review": "Based on my current understanding of the architecture, I'm not clear whether the improved error measurements stem from more weights or inherent advantages of the encoding layers. So, based on my current understanding, I don't think I can recommend accepting this paper. I would recommend that the authors spend time to more clearly analyze and explain why their method leads to improvements, rather than long lists of graphs. This would also potentially make it possible to assess how well proposed changes would carry over to other settings, e.g., other types of physics simulations.\n", "strength_and_weaknesses": "The main goal of the paper - more accurate predictions for learned physical systems - is an important and relevant one. Also, the evaluations seem to show improvements for the proposed architectures. However, I see quite a few problems and unclear points with the algorithm and manuscript:\n\nAmong others, I was surprised about the focus and attention that is given to the \"autoregressive\" label. This is a fairly standard approach for time integration methods, and the necessity for multiple previous steps is an inherent drawback of learned algorithms: from classical physics we know that the evolution of a deterministic physics system is uniquely prescribed by a single state. The necessity for working with multiple time steps (\"lookback\" as its called here) is not necessarily a positive aspect. The use of linear encoder/decoder blocks for graph networks is also not exactly new, e.g., see Stachenfeld et al. 2021 \"Learned coarse models\".\n\nIn addition, I was missing an explanation for why these additional linear layers should have a significant impact on the learning task (beyond adding more weights). This is likewise something that could be handled by the MLPs in the message passing layers. Here the paper also does not make clear how the weight count changes when including the proposed encoders.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "WTxr5In9atK", "reviewer": "Reviewer_hzQp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work presents an Autoregressive Graph Network (AGN) that learns physics models. Given the state of a system for $k$ time-steps, an autoencoder is used to generate node (and edge) embeddings for each particle (and pairs of particles) in the system. Importantly, the autoencoder is constrained to impose an autoregressive structure that captures temporal dependencies. The node and edge embeddings are then input to a message-passing network, whose output is then used to predict the target state.  \nFinally, experiments are performed to study the performance of the AGN at multi-step forward prediction on datasets featuring different particle interactions.", "summary_of_the_review": "This paper presents a Graph Network that imposes an autoregressive structure on its input embeddings to capture temporal dependencies.  Experimental results indicate that the approach outperforms baseline GNs on multiple datasets. However, a few important aspects of the approach were unclear. Improving the presentation, for section 3 in particular, would help the reader better understand these aspects of the approach.", "strength_and_weaknesses": "The experimental results demonstrate that the AGN outperforms the baseline GNs on most datasets presented.\n\nHowever, some aspects of the approach were not completely clear. \n- In the introduction, the authors mention that the encoder learns the arrow of time. But based on the discussion following Eq 2, it looks like the mask M explicitly induces the desired temporal dependencies? \n- In the last paragraph of section 3.2.1 (VAR encoder), the input to the VAR edge encoder includes the derivatives of the full state: $\\dot{\\boldsymbol{x}_i}(t) - \\dot{\\boldsymbol{x}_j}(t)$. According to the definition following Eq 1, these derivative terms already include the target $\\ddot{\\boldsymbol{r}}$. Is this a typo or intended? Also, why does Eq 5 include both $H_i - H_j$ and $|H_i - H_j|$ terms?\n- If the temporal dependence is already captured by the embeddings, why are there $L$ message-passing steps? How much more computationally efficient is the AGN compared to the baseline GNs used in the experiments? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Vw_m24RzyI", "reviewer": "Reviewer_n2LJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an additional autoregressive module to the vanilla graph network-based simulator (GNS), which can outperform GNS while maintaining high efficiency. Specifically, it learns spatiotemporal embedding for simulating spatial-temporal trajectories, which somehow solves the problem of high computational costs caused by long-term memory. It conducts experiments on multiple scenarios such as gravity, spring, charge, etc.,  to demonstrate the effectiveness of the proposed method.   ", "summary_of_the_review": "I think the problem is important and the proposed model seems effective, however, my main concern is about the advantage of this model over other sequential models, which is not supported by experiments. ", "strength_and_weaknesses": "### Strength\n1. The problem of designing graph networks for spatial-temporal simulation is important. \n2. It somehow solves the problem of high computational costs caused by long-term memory. \n3. It outperforms vanilla GNS while maintaining high efficiency.\n\n### Weakness\n1. The authors claim that this autoregressive model is better than other sequential models, such as RNN and Transformer, but there is no such comparison. I think the comparisons of both accuracy and efficiency are necessary. I am not fully convinced that this model is better than other commonly-used sequential models. \n2. The idea of the autoregressive model seems not very novel. \n3. The writing can be improved. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Afj1ajpRti", "reviewer": "Reviewer_ucoP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new graph network (GN) architecture for learning dynamics models of particle-based systems. Their architecture is structured as a GN applied autoregressively in the temporal direction. They compare their method to various alternative GN architectures (vanilla, physics-inspired) on a few different dynamical systems.", "summary_of_the_review": "Combining the concerns about novelty (see \"Strengths & Weaknesses\") and the sparse explanation of the baselines (see \"Clarity, Quality, Novelty And Reproducibility\"), I have a hard time understanding why the proposed method outperforms the baselines. Based on the provided results, it does seem to perform well empirically, but I'm wondering if there is some other feature of the proposed method (e.g. some architectural choice) or some overlooked flaw of the baselines or that explains this? Without a clear understanding of the contributions nor the empirical success, I would have a hard time recommending this paper for acceptance.", "strength_and_weaknesses": "Strengths:\n1. The proposed method is extensively evaluated, with many experiments on a range of tasks.\n\nWeaknesses:\n1. The proposed method does not feel especially novel. One of the main contributions discussed, the concept of \"vector autoregression\" feels like a trivial extension of existing work, either of [1] by extending it to be autoregressive over time, or of [2] by extending it to particle systems via a GN.\n2. Some of the arguments about benefits of the proposed method feel tenuous at best. For example, I have a hard time following the discussion about translation invariance. The authors claim that \"By providing absolute positions as input, the autoregressive encoder learns translation invariant features.\" -> Why exactly does this happen?\n\n\n[1] Learning to Simulate Complex Physics with Graph Networks. Sanchez-Gonzalez et al, https://arxiv.org/abs/2002.09405.\n[2] Prediction and control with temporal segment models. Mishra et al, https://arxiv.org/abs/1703.04070.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject