Neural Constraint Inference: Inferring Energy Constraints in Interacting Systems
Authors: Armand Comas
Yilun Du
Sandesh Ghimire
Christian Fernandez Lopez
Mario Sznaier
Joshua B. Tenenbaum
Octavia Camps
Venue: Submitted to ICLR 2023 (2023)
Keywords: relational inference
energy-based models
energy constraints
trajectory prediction
graph neural networks

Abstract: Systems consisting of interacting agents are prevalent in the world, ranging from dynamical systems in physics to complex biological networks. To build systems which can interact robustly in the real world, it is thus important to be able to infer the precise interactions governing such systems.  Existing approaches typically discover such interactions by explicitly modeling the feedforward dynamics of the trajectories. In this work, we propose Neural Constraint Inference (NCI) model as an alternative approach to discover such interactions: it discovers a set of relational constraints, represented as energy functions, which when optimized reconstruct the original trajectory. We illustrate how NCI can faithfully predict future trajectory dynamics, achieving  more consistent long-rollouts than existing approaches. We show that the constraints discovered by NCI are disentangled and may be intermixed with constraints from other trajectories. Finally, we illustrate how those constraints enable the incorporation of external test-time constraints.

Under review as a conference paper at ICLR 2023
NEURAL CONSTRAINT INFERENCE: INFERRING EN- ERGY CONSTRAINTS IN INTERACTING SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Systems consisting of interacting agents are prevalent in the world, ranging from dynamical systems in physics to complex biological networks. To build systems which can interact robustly in the real world, it is thus important to be able to infer the precise interactions governing such systems. Existing approaches typically discover such interactions by explicitly modeling the feedforward dynamics of the trajectories. In this work, we propose Neural Constraint Inference (NCI) model an alternative approach to discover such interactions: it discovers set relational constraints, represented as energy functions, which when optimized reconstruct the original trajectory. We illustrate how NCI can faithfully predict future trajectory dynamics, achieving more consistent long-rollouts than existing approaches. We show that the constraints discovered by NCI disentangled and may be intermixed with constraints from other trajectories. Finally, we illustrate how those constraints enable the incorporation of external test-time constraints.
INTRODUCTION
Dynamical systems ubiquitous in both nature and everyday life. Such systems emerge naturally in scientific settings such chemical pathways and particle dynamics as well as everyday settings such as in sports teams or social events. Such dynamical systems may be decomposed as a set of different interacting components, where the interactions with respect to each other lead to complex dynamics. Modeling the underlying dynamics such systems is hard: often times we only have access example trajectories, without knowledge of the underlying interactions the dynamics that govern them.
Zone defense / Push
Attraction player
Attraction to goal
Team
Team
Ball
Consider the scenario given in Figure 1, consisting of set of NBA players playing basketball game. While the motion of individual players may appear stochastic in nature, each player aims score the basket on the opposite team's ' side of the court. Thus, we may utilize sets of interactions explain their behaviors a group of players on the defensive team serve as zone defense, preventing players from the opposite team from getting close the basket. Simultaneously. group of offensive players moves towards the goal, while a group of defensive players moves to intercept them and prevent them from scoring. By applying our underlying knowledge of these interactions between players, we may forecast the future dynamics of the basketball game significantly more accurately.
Figure Interactions between NBA players. Com- plex dynamics, such as the player trajectories in the NBA, may be explained using simple set of inter- actions. In this setting, one team of players aims block a separate team from scoring.
Most works modeling such complex dynamics do not explicitly disentangle individual interactions between objects. Instead, they rely on learned network to implicitly disentangle them (Battaglia et al., 2016; Gilmer et al., 2017; van Steenkiste et al., 2018). In contrast, Kipf et al. (2018) propose Neural Relation Inference (NRI), which learns structured set of explicit interaction models between objects and show how such explicit interaction modeling enables more effective downstream predictions. In this work, argue that should instead model and disentangle interactions between objects

---

Under review as a conference paper at ICLR 2023
as set of learned relational constraints, with dynamical prediction corresponding to constraint satisfaction problem. To this end, we propose Neural Constraint Inference (NCT), where we encode each of these constraints as an energy function (Du et al., 2021).
To predict future dynamics with NCI, we then solve a constraint satisfaction problem, where we optimize for trajectory prediction which minimizes our predicted energy. Prior work implicit physical simulation has suggested that such implicit physics modeling (i.e. modeling dynamics through a constraint satisfaction problem) is significantly more accurate at simulating strong inter- actions in dynamics than explicit physics models (i.e. modeling dynamics as explicit feed-forward roll-outs) (Rubanova et al., 2022).
In different experiments, we illustrate how our constraint based decomposition of interactions provides unique benefits over prior learned approaches for decomposing dynamics. First, we illustrate how such a decomposition improves the temporal consistency, achieving significantly lower long-term temporal prediction error. We show that the decomposition disentangled, enabling us to intermix interactions between separate trajectories together. We further show that constraints can linearly be decoded into underlying ground-truth interactions. Finally, we illustrate that such a decomposition enables us to add flexible test-time constraints to incorporate new changes in the environment,
In summary, this work, we contribute the following: (i). We propose Neural Constraint Inference (NCI, which discovers, in unsupervised manner, the underlying interactions between particles in a system as a set of energy constraints. (ii). We illustrate how such a constraint decomposition of interactions enables more accurate long-horizon trajectory prediction performance over prior methods. And (iii). we illustrate how such a constraint decompositions of interactions is disentangled and enables the recombination of constraints between separate trajectories, as well the addition of novel test-time constraints.
2 LITERATURE
Dynamics and Relational Inference Several works in the past years have studied the problem of learning dynamics of a physical system from simulated trajectories with graph neural networks (GNNs) (Guttenberg et al., 2016; Gilmer et al., 2017; van Steenkiste et al., 2018; Lu et al., 2021; Li et al., 2018; Yang et al., 2022). As an extension of the foundational work of Battaglia et al. (2016), interaction networks, Kipf et al. (2018) proposes to infer an explicit interaction structure while simultaneously learning the dynamical model of the interacting systems in unsupervised manner, by inferring edge classes with a classifier. Selecting models based on observed trajectories is also the base of Alet et al. (2019); Goyal et al. (2019); Graber & Schwing (2020). Graber & Schwing (2020) extends Kipf et al. (2018) to temporally dynamic edge constraints, which yields better results in real-world datasets. NCI differs from these approach as the generation procedure uses an optimization solver to satisfy a set of soft constraints. Recent work Rubanova et al. (2022) also explores combining graph networks with energy optimization. However, lacks the modularity of NCI, and the ability to infer edge types from observation. Instead, global energy function is learned for all nodes, by leveraging ground truth attributes such as mass. Thus, it has no mechanism to predict trajectories in the absence of those attributes nor when different types of relations are present.
Energy-Based Models Energy-based models have a long history in machine learning. Early work focuses on density modeling Hinton (2002); Du & Mordatch (2019); Nijkamp et al. (2020) by aiming to learn a function that assigns low energy values to data that belongs to the input distribution. To successfully sample data-points, EBMs have recently relied gradient- Langevin dynamics Du Mordatch (2019). Recent works have illustrated that such gradient- based optimization procedure can enable the composition of energy functions representing different concepts Du et al. (2020)
and successfully high-dimensional domains such as images Liu et al. (2021); Nie et al. (2021).
Unsupervised discovery of composable energy functions on images was explored in Du et al. (2021).
In this work, we extend ideas of unsupervised concept learning in EBMs to constraints and apply them to dynamical modelling and relational inference.
3 CONSTRAINTS AS ENERGY BASED MODELS
We will consider constraints as specifying a set X of underlying trajectories & E RTxD which have underlying property we desire. In section Section 3.1, we discuss how can represent constraints
2

---

Under review as a conference paper at ICLR 2023
on trajectories using EBM. We further discuss how may compose multiple constraints together as EBMs in Section 3.2.
3.1 ENERGY-BASED MODELS
Definition. An Energy-Basee Model (EBM) is defined probabilistically using the Boltzmann distribution pe(x) exp( (a()) with an underlying partition function Z(0) = exp( Eg Eg(x)) )dx, where denotes the weights that parameterize the energy function Ég. We will represent constraint EBM, defined using a neural network parameterized energy function Ex(x:) R - R that maps each datapoint to a scalar value representing an energy. A constraint then corresponds to the set of datapoints in which the assigned energy is low. Thus, datapoints satisfying our constraint have high likelihood, and all other datapoints have low likelihood. Constraint satisfaction then corresponds sampling from the EBM distribution pa (2)).
2(0)
Solving Constraints. In our framework, solving constraint corresponds to sampling from the EBM which defines it, and thus finding high-likelihood data points under pe (z). We follow existing works and utilize gradient based MCMC procedure, Langevin Dynamics (Welling & Tch, 201 Du Mordatch, 2019) to sample from the EBM distribution. In particular, to solve a constraint, we initialize trajectory x0 from uniform noise. We then run M iterative steps following:
x = m
ExEs (xm w", wm
~ ~((,o).
(1)
2
where each step we iteratively optimize the trajectory with respect to the energy function, using an underlying gradient step size of À and noise scale of a. We include hyperparameter details for sampling Section A.1 of the appendix, and heuristically set the noise scale of 0.
3.2 COMPOSING CONSTRAINTS
Next, we discuss how we may compose different sets of constraints together, where each constraint is parameterized by separate EBM E (a)). Our composition operator builds on existing works on composing EBMs representing concepts Du et al. (2021).
Sampling Composed Constraints. Given set of separate constraints, we wish to solve for a set of trajectories æ which jointly satisfy each of the constraints. our EBM formulation, this corresponds to finding a trajectory æ which is low energy under each of the specified energy functions bY (z).
Such setting is equivalent to finding trajectory which has high likelihood under each EBM probability distribution (x). This corresponds to sampling from the distribution defined by the product the individual EBM distributions.
[x) x E, c (x)())
which corresponds to a new EBM with energy function F (x) (an analogous approach can be applied to generate images subject to set of concepts (Du et al., 2020)). Thus, we may sample from the composition of set of constraints using a sampling procedure as Equation 1, using the new energy function E (x). defined as the sum of each individual energy function. Intuitively, this corresponds to continuous optimization procedure on each energy function together.
In our setting, different energy functions (x) are constructed by conditioning an energy function on separate latent vectors. These latents are directly inferred unsupervised from input trajectories by training an encoder jointly with the energy function parameters.
NEURAL CONSTRAINT INFERENCE
Next, we discuss Neural Constraint Inference (NCI), our unsupervised approach to decompose a trajectory x(I...)); consisting of N separate nodes at each timestep, into a set of separate
EBM Ex constraints. Neural Constraint Inference (NCI) is composed by two steps: (i) an encoder for obtaining a set of energy constraints and (ii) a sampling process which optimizes for a predicted trajectory, given the inferred energy constraints. Energy functions in NCI are trained using autoencoding, similar to Du et al. (2021). We provide an illustration our approach in Figure 2 and pseudocode in Algorithm 1.
3

---

Under review as a conference paper at ICLR 2023
Observed trajectory
Relational Energy Constraints
baase Sampling
Final Prediction
RG.
Constraints
(x(::)) 100
(x 50
ZR.G
edge encoding
- (N By Ey Fine fan++ (x":Fa)) -
Figure 2: Overview of NCL In the left, portion x (1... of the input trajectory is observed by Encs and encoded by GNN into relational energy constraints, in the form of a set of latent vectors for each edge in the graph. In the right, energy functions parametrized as GNNs for each edge latent vector z are constructed. Energy functions are trained so that optimizing trajectory x" from uniform noise into final trajecotry x reconstructs the future states of the observed trajectory. This refinement process uses Langevin Dynamies (Eq. 4). Given the full trajectory x" at sampling step m. we update it by summing the gradient contributions of the energy function associated to each edge, resulting in x=1
4.1 RELATIONAL ENERGY CONSTRAINTS
To effectively parameterize different energy constraints for separate interactions, we learn latent conditioned energy function Ex z): RDD x IR - R. Then, inferring a set different energy constraints corresponds to inferring latent E R that conditions an energy function.
Given a trajectory x(...T)); we infer a set of L different latent vectors for each each pair of interacting nodes in trajectory. Thus, given set of N different nodes, this corresponds to set of N(N 1) different energy functions.
To generate trajectory, optimize the en- (x: ergy function [xx) Eny,1 (x; 2ija). across node indices and j from 1 to N and latent vectors / from 1 to L. However, assign- ing one energy function each latent code becomes prohibitively expensive as the num- ber of nodes in trajectory increases. Thus, reduce this computational burden, we param- eterize L energy functions as shared message passing graph networks, grouping all edge contributions ij in a single network. The en- ergy is then computed individual node energies after message pass- ing. To evaluate the energy corresponding a single edge factor Zig.1 we mask out the con- tributions of all other edges the final node energies. Architecture and further details can be found in Section A.2 of the appendix.
Algorithm Training algorithm for NCL
Iopot: injectories x. Observed Imjectories x( I... 7T,, condi- tions x(I.. Lo), step size X. number of gradient steps M. encoder Ency, energy functions Fig noise " 0, mae data distribution Po while converged do
x. PD
Encude components from x(L...')
fu) Enca (x(1...2"))
Optimize sumple x, via gradient descent:
x! 24(0,
for gradient step No do
2 - vV E
(3"
end for
Optimize objective Corn wi
20 Volls" x||
summation over all
Update based using optimizer end while
Figure 3: Training Algorithm. NCI is trained to infer set of constraints, represented energy functions, using trajectory reconstruction objective. A set of latents (x inferred from the beginning of trajectory x(I...T)). and define different constraints. A trajectory is optimized w.n.t. to energy functions and supervised with the trajectory x.
To condition to message passing shared graph network on each inferred latent Z.j. each edge eii, j) in the graph is conditioned by the corresponding encoded edge latent code Zij.t. by means of FiLM modulation (Perez et al., 2018).
4.2 INFERRING ENERGY CONSTRAINTS
We utilize Ency (x) TxD RR Dx to encode the observed trajectories x into latent representa- tions per edge in the observation. We utilize fully connected GNN with pesssag--passigg to infer latents using the encoder module Kipf et al. (2018). Instead of classifying edge types and using them gate ouputs, utilize continuous latent code Zij.o allowing for higher flexibility.
4

---

Under review as a conference paper at ICLR 2023
4.3 TRAINING OBJECTIVE
To train NCI, we infer a set of different EBM constraints by auto-encoding the underlying trajectory. In particular, given a trajectory x(I...)); (x(1) x(T),), we split the trajectory into initial conditions x(I...b), corresponding to the first To states of the trajectory and x(To...), correspond- ing the subsequent states of the trajectory, where each state of the trajectory consists of N different
nodes. The edge constraints are encoded by observing a portion of the overall trajectory x x(1..."".. where T" T.
We infer a set of different L latents per edge of input observations utilizing the observed states x(I....") using the encoder specified in Section 4.2, generating set of latents (z}. We then aim to train energy functions so that the following unnormalized distribution assigns low energy and high likelihood the full trajectory x:
[z)) x II
p(x|z,,,) = exp (-E Ence Ence (x(L...T"))ss..))
S.J.Vis
where Zij,
and E is the energy function linked to the In constraint of the encoded edge between nodes and j, respectively.
Since wish to learn set of constraints with high likelihood for the observed trajectory x, tractable supervised manner to learn such a set of valid constraints, we directly supervise that sample using Equation 1 corresponds the original trajectory x, similar Du al. (2021). In particular, we sample M steps of Langevin sampling starting from x², which is initialized from uniform noise and the initial conditions fixed the ground-truth cl....T)):
x Th
Vx
Σ 17.
Exj. (xm=11;;3)) (ijaa) + w", wnth ~ A(,,
(4)
where is the mih step and is the step size. We then compute MSE objective with XM, which is the result of M sampling iterations and the ground truth trajectory x:
(MSE = || x||2.
(5)
We optimize both x and the parameters 0 with automatic differentiation. The overall training algorithm is provided in Algorithm 1.
EXPERIMENTS
In this section we firstly describe our datasets (Section 5.1) and baselines (Section 5.2). Following, we describe the quantitative results (Section 5.3). In Section 5.4, we show experiments on (i.) recombination, (fi.) edge classification and (iii.) contribution of the constraints. Next, in Section 5.5, we describe OOD) sample detection. Finally, show how incorporate test-time constraints in Section 5.6. In the appendix we give implementation details (Section I) experimental details (Section A.3) and provide an ablation study (Section A.5) and additional examples (Section A.4).
5.1 DATASETS
We tested our model in three different domains. First, we carry on experiments in two simulated environments: (i.) Particles connected by springs, and (ii.) Particles with charges. Next, we test several properties of our model in (fii.) NBA SportVu motion dataset, which displays real motion from tracked basketball players along several NBA games. Finally, we test our performance (iv.) JPL Horizons, a physics- based realistic dataset.
Simulated data. Following the experimental setting described in Kipf et al. (2018), we generate states (position and velocity) of dynamical system for N 5 particles for 70 time-steps. Our model observes the first 49 states, fixes one state and predicts the following 20. We generate 50k training samples and 10k for validation and test splits.
This setting is interesting because the rules by which particles interact are known and simple. However, they can generate very complex behaviour.
Springs:
The particles move inside a box with elastic collisions. They are connected by a spring with probability 0.5, and interact according to Hooke's 's law.
Charged: The particles move inside box as in Springs. They are assigned a positive or negative charge di E (#tg) with probability 0.5 and interact via Coulomb forces.
5

---

Under review as a conference paper at ICLR 2023
Springs
Charged
Prediction steps
10
20
10
20
Static
7.93e-5
7.59e-3
2.82e-2
5.09e-3
2.26e-2
5.42e-2
IN
1.32c-5
1.28c-3
4.71c-3
2.46e-4
1.06c-2
2.15c-2
LSTM (single)
2.27c-6
4.69c-4
4.90c-3
2.71e-3
7.05c-3
1.65e-2
LSTM (joint)
4.13e-8
2.19c-5
7.02e-4
1.68e-3
6,45e-3
1.49c-2
Cond. GNN
9.00e-6
6.13e-5
3.14e-4
3.67e-3
5.61e-3
1.0Se-2
NRI (full graph)
1.66c-5
1.64c-3
6.31o-3
1.09c-3
3.78c-3
9.24c-3
NRI (learned)
3.12c-8
3.29c-6
2.13e-5
1.05e-3
3.21e-3
7.06e-3
NCT (Ours)
2.34e-7
1.57e-6
1.74e-5
8.85e-4
3.33e-3
6,54e-3
Table I: Mean squared error (MSE) in predicting future states for simulations with 5 interacting objects. NCT outperforms the baselines in mid to long-term prediction error.
NBA Sp SportVU is an automated ID and tracking service that collects data of NBA players and the ball (N = 11) during game. The dataset is generated by splitting each of the labeled events into 65 steps trajectories of coordinates x.y. We compute the velocities to generate the states. The dataset is composed of 50k samples for training and Ik samples for validation and test.
JPL Horizons The JPI. Horizons on-line ephemeris system provides access to solar system data. characterizes the 3D location and velocity of solar system objects (targets) as a function of time, as seen from locations within the solar system (origins). This dataset consists captured between 1800 and 2022, with one datapoint every 10 days. We define the nodes as = 12 targets of the solar system: planets, 3 moons and the Sun. This data is captured from 13 origins: each one of the targets plus the solar system barycenter (SSB). We gather 1880 trajectories of 43 timesteps split as 1504/188 188/188 188 for train, validation and test.
the trajectories
5.2 BASELINES
We consider a Static baseline, which copies the previous state vector, LSTM (single), an LSTM trained to predict the state vector difference at every timestep. We further compare with LSTM (joint) which differs from single in that it concatenates input representations from all objects after passing them through MLP. For the synthetic experiments, where the edge types are known priori, we also evaluate NRI, the architecture presented in Kipf et al. (2018) to infer the interaction graph (learned), and with a fully connected graph of single edge type (full graph). We further add GNN conditioned to the observed trajectories, and an interaction network (IN) Battaglia et al. (2016). For NBA SportsVU we will also evaluate on two social interaction- basod methods: Social-LSTM (S-LSTM) Alahi et al. (2016) and Directional-L.ST (TrajNet++) Kothari et al. (2021).
5.3 QUANTITATIVE COMPARISON
For all datasets, we will observe a portion of the trajectory and predict 20 timesteps.
We first test our approach in Springs and Charged datasets. We evaluate the Mean-Squared Error (MSE) against Kipf et al. (2018), their chosen baselines, a generic Conditional-GNN and an IN. Our models observes 49 timesteps and fixes the 50g, initial conditions for prediction. We can see in Table I that NCI achieves better long-term prediction in both datasets, and slightly worse short-term prediction error. This likely caused by the nature of the generative process. The predictions are temporally one-shot, while they are refined by several optimization iterations.
Similarly, for NBA (Table (left)) the model observes 40 timesteps and fixes the following 5 as initial conditions for prediction. NCI outperforms the baselines in terms of prediction error. The models designed for social interaction perform poorly long-term prediction, while they have shown to excel in other tasks such as collision avoidance.
For the JPL Horizons dataset in Table 2 (right), NCI outperforms the baselines substantially. Models have access to 23 timesteps. NCI observes 20 timesteps and fixes 3 as initial conditions for prediction. JPL. Horizons is a challenging dataset given the unknown masses of the bodies involved, as well as effects from unobserved smaller bodies nearby them that introduce noise to the trajectories.
5.4 DISENTANGLEMENT OF ENERGY CONSTRAINTS
NCI assigns an energy function to each of the representations learned by the encoder. Those constrain the generation process by conditioning the features of their associated EBM. The optimiza- tion procedure, hence, aims to satisfy all constraints. We train our model ensure that different

---

Under review as a conference paper at ICLR 2023
GT Springs
GT Charged
Recombined
0.2
0.0
0.2
-0.2
-0.4
-0.2
0.0
0.0
+
0.0
-0.2
-0.2
-0.4
-0.2
-0.4 -0.2 0.0
0.7
0.2
0.0
-0.2
00.4
0.0 0.2
Figure 4: NCT can recombine encoded energy constraints at test-time learned from different datasets. Illustrated, samples from Springs (Col. 1) and Charged (Col. 2) and their recombinations (Col. 3). NCI encodes both trajectories. NCI is able to reconstruct trajectories framed in green with the initial conditions marked in blue, while swapping the edge constraints associated to the nodes in the red dashed box. Recombinations look semantically plausible and smooth.
NBA VU
JPL Horizons
Prediction steps
10
20
10
20
S-LSTM
6.60e:5
6.67e-3
2.57e-2
TrajNet++
5.30c-5
5.88c-3
2.33c-2
Static
2.13e-4
3.04e-3
1.07e-2
3.33e-3
5.54e-2
9.05e-2
I.STM (joint)
8.07e-5
1.42e-3
5.31e-3
1.97e-6
3.98e-5
1.09e-4
Cond. GNN
1.71c-4
1.12c-3
3.11c-3
4.57c-6
4.66c-6
5.96c-6
NRI
3.56e-6
7.46e-4
2.74e-3
2.67e-7
7.35e-7
1.16e-6
ANRI
7.97e-6
1.07e-3
4.520-3
1.35e-5
5.12c-5
1.64e-4
NCI (Ours)
1.276-5
3.46e-4
1.86e-3
4.05e-7
4.70e-7
8.60e-7
Table 2: Mean squared error (MSE) in predicting future states for NBA dataset and JPL. Horizons dataset, with 11 and 12 interacting objects respectively. NCI performs better than the baselines at short and long terms.
constraints contribute individually to the generative process by addition of their associated gradients. We argue that this procedure both (i.) aids discovery of disentangled edge representations and (ii.) allows composition among disjoint training distributions.
Recombination To verify claims, show how NCI can compose energy constraints learned from two different distributions, at test-time. Figure 4 shows qualitative results of recombinations from Springs and Charged datasets. The process is as follows: we train two instances of our model (NCIs, NCIc) to reconstruct Springs and Charged trajectories respectively. Given sample trajectories drawn from each dataset (Col. for Springs and 2 for Charged), we encode them into their relational energy constraints. For each row, we aim to reconstruct the trajectory framed in green while swapping one of the relational constraints (red dashed box) with an instance drawn from the other dataset. As an example, in the first row of the figure, we encode the Springs trajectory with NCIg and the Charged trajectory with NCT.. Next, we fix the initial conditions corresponding to the Charged trajectory (blue dots) and sample by optimizing the relational energy functions. To achieve recombination. each model targets specific edges. We satisfy the constraints encoded by NCIş for the mutual edges corresponding to the nodes in red dashed boxes. The rest of edge constraints are encoded by NCIg- The sampling process is done jointly by both models, each satisfying their corresponding edge constraints. The result a natural combination of the two datasets, which constrain only the targeted edges. Reconstructed trajectories in Figure 4 (col. 3) are smooth and semantically reasonable.
7

---

Under review as a conference paper at ICLR 2023
Outlier Delection Energies per Node
Eval/ Train
Springs
0.2
0.1
Springs
0.00 =0.01
Energy
Energy
Charged
0.18 #0.30
0.43
0.12
S&C (eval all)
0.14 #0.30
0.08
0.04
0.0
0.06
0.03
S&C (eval S)
0.09 #0.18
0.06
0.03
S&C (eval C)
0.19 20.37
0.04
0.0
0.1
0.0
0.1
Quantitative evalua- tion of out-of-distribution detec- tion. S&C rows correspond to the Springs and Charged mixed dataset. For this dataset we eval- uate individually and jointly the Springs and Charged nodes.
Table 4:
Figure 5: We can use NCT for outlier detection. A model trained with certain relation lypes can detect when trajectory exhibits a new type of relation. The illustrated trajectories show the energy associated each one of the nodes. Red trajectories: Charged particles, Blue-Green trajectories: Springs particles. We train NCI in Springs dataset. Our model assigns higher energies to those nodes that behave differently than the training set.
Edge classification To test the proper edge disen- tanglement capabilities of NCI, we propose to eval- uate quantitatively the representation value of our inferred constraints. We measure their ability to clas- sify correctly the edge type that they correspond to.
Model
Springs
Charged
5 objects
Corr. (path)
52.4=0.
55.8
Corr. (LSTM)
52. 7
54.2+20
NRI (sim.)
99.8100
59.6208
We train a linear layer to decode the constraints dis- covered by the encoder into edge types. The objec- tive used is Binary Cross-Entropy w.r.t. the ground truth edges. Table 3 shows the accuracy obtained. In the table. Corr. (path) estimates the interactions by thresholding the matrix of correlations between trajectory features. In Corr. LSTM each trajectory modeled individually and calculates correlations between output hidden states. For NRI, the accuracy is the result of their encoder's 's edge classification.
NRI (Icarned)
99. 99.9100
82. 1206
NCI (linear dec.)
97.4#t.01
69. 69.4#1..
Table 3: Accuracy (in %) of true interaction recovery. We train a linear decoder to predict edge types from constraints. Without supervision NCI learns meaningful representations that can often recover true interactions. Corr. indicates evaluation by means of the matrix of correlations between feature vectors.
We recover the true edge types with high accuracy in Springs and with moderate success for Charged dataset. For comparison, note that our evaluation setting for this experiment is considerably different from that of NRI. Unlike NRI, NCI's edge representations discovered without the purpose of edge classification. We provide additional comparisons in Section A.4 of the appendix.
Contribution of Energy Constraints Each of the disentangled constraints controls different aspects of the interactions. A qualitative example shown in Figure depicts the gradient orientation of two energy constraints model has been trained with. We can see how each constraint pushes the player trajectory into different directions, each one of them pointed to a different player of the rival team.
Gradients function #1
Gradients for function #2
Figure 6: Energy constraints discovered by our approach control different aspects of the trajec- tories. For model trained with 2 energy functions, this illustration shows the gradients associated each energy function applied to a ground -tuttth sam- ple. Each constraint pushes the player of interest into one of the opponents.
5.5 OUT-OR-DISTRIBUTION DETECTION
We further utilize the energy value at each constraint of NCI to detect out-of-distribution interactions in trajectory. In our proposed architecture, energy is evaluated at the node level. Therefore, if NCI has been trained with a specific dataset, the constraints associated to diitr--dittiitttinn type of edges are expected to have higher energy.
We design a new dataset Shharedd-Springee as a combination of Springs and Charged interaction types. In simulation, nodes are assigned both roles of Charged and Springs particles, but all the forces they receive correspond to one of the two types with probability p = 0.5. We train a model with the Springs dataset and evaluate the energies in the proposed mixed setting.
Figure shows qualitatively how the energy is considerably higher for the nodes with Charged-type forces (drawn in red). Quantitative results are summarized in Table 4 for 1k test samples. We can
8

---

Under review as a conference paper at ICLR 2023
Edge Constraints
(Edge Velocity) Constraints
Edge Constraints
(Edge- - Velocity) Constraints
(Edge 2x Velociiyy Constraints
(Edge + Attraction) Constraints
Figure 7: NCI is able to incorporate energy constraints in test-time. We depicted reconstructions of NBA samples with added constraints. Left: (Col. Row 1): Reconstruction of the encoded trajectory. (Col. 1, Row 2): Decrease of velocity. (Col. 2, Row 1): Low increase velncity. (Col. 2, Row 2): High increase of velocity. Right: (Col. 3, Row 1): Reconstruction of the encoded trajectory, (Col. 3, Row 2): Attraction of the players to goal point (blue dot). Painted orange, the ground-truth ball trajectory.
see that energies corresponding to Spring-type nodes are considerably lower than for Charged-type nodes, indicating that constraints are correctly capturing the behavior of the desired interactions.
5.6 FLEXIBLE GENERATION
Another advantage of our approach is that it can flexibly incorporate test-time user specified con- straints. For this experiment, we investigate two different sets of constraints in Figure when reconstructing given 40 step trajectory of the NBA dataset.
Velocity Constraints We incorporate the following velocity energy constraint: E
e Ei, (V',,) (v = E mod(v!), for particle i time t. The weight À = le 2/N scales the effect of this constraint over the rest and is multiplicative constant that indicates the strength and direction of the constraint. Figure 7 (left), we show (i.) e = 0: Reconstruction (top-left); (ii.) 6 = Decrease of velocity (bottom-left); (iii.) -5: Low increase of velocity (top-right); and (iv.) 10: High increase of velocity (bottom-right). Results satisfy test constraints.
Goal Constraints We also add at test-time attraction energy constraint of the form: E = eA Ei, Φf. Here = le - 4/N, ø is the angle between the velocity vector v of object at time-step I and the orientation of the vector p¹q. where P is the location of object at time and the point of attraction goal. Figure (right) illustrates the scenarios (i.) O: Reconstruction (top); (ii.) e = 3: Attraction to the goal (bottom, goal in blue). The reconstructed trajectory follows the new constraint, while maintaining the constraints of the encoded trajectories.
DISCUSSION
Conclusion. In this work we introduced Neural Constraint Inference (NCI) which infers relational constraints specified as energy functions to model the dynamics of an interacting system. We illustrate how NCI can also faithfully predict future trajectory dynamics, achieving more accurate long-rollouts than the baselines. We further illustrate the disentanglement of the discovered constraints by intermixing them with constraints from other distributions of trajectories and show that constraints interpretable. We also show that constraints obtained by NCI may discover ditt----isttiuttionn node interactions even within a scene. Finally, we illustrate the flexibility of modeling relational constraints enabling the incorporation of external hand-crafted constraints at test-time.
Limitations and Future Work. Our current solution is currently limited to energy constraints associated to edges, and it would be beneficial to further explore its application to other type of constraints. Future work will focus generalizing the use of the energy functions, for instance, targeting individual nodes in the graph.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robicquet, Li Fei-Fei, and Silvio Savarese. Social Istm: Human trajectory prediction in crowded spaces. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 961-971,201 6
Ferran Alet, Erica Weng, Tomas Lozano-Perez, and Leslie Pack Kaelbling. Neural relational inference with fast modular meta-learning. In NeurlPS, 2019. 2
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction networks for learning about objects, relations and physics. NIPS, 2016. 6, 14
Yilun Du and Igor Mordatch. Implicit generation and modeling with energy based models. In NeurIPS, 2019. 3
Yilun Du, Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In NeurlPS, 2020. 2,
Yilun Du, Shuang Li, Yash Sharma, Joshua B. Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. NeurlPS, 2021. 3,5
Justin Gilmer, Samuel Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. ICML, 2017. 1,2
Anirudh Goyal, Alex Lamb, Jordan Hoffmann, Shagun Sodhani, Sergey Levine, Yoshua Bengio, and Bernhard Schölkopf. Recurrent independent mechanisms. ICLR, 10893,, 2019.
Colin Graber and Alexander G. Schwing. Dynamic neural relational inference. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2020. 2
Nicholas Guttenberg, Nathaniel Virgo, Olaf Witkowski, Hidetoshi Aoki, and Ryota Kanai. Permutation-cquivariant neural networks applied to dynamics prediction. ArXiv, abs/1612.04530,, 2016.7 2
Geoffrey Hinton. Training products of experts by minimizing contrastive divergence. Neural
Computation, 14:1771-1800, 2002. 2, 12
Thomas Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard S. Zemel. Neural relational inference for interacting systems. ICML, 2018. 1, 2, 4, 5, 6, 14
Parth Kothari, Sven Kreiss, and Alexandre Alahi. Human trajectory forecasting in crowds: A deep learning perspective. IEEE Transactions Intelligent Transportation Systems, pp. 1-15, 2021. doi: 10. N..100/TTTS.200
6, 14, 15
Yunzhu Li, Jiajun Wu, Russ Tedrake, Joshua B Tenenbaum, and Antonio Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv: 181000060 2018. 2
Nan Liu, Shuang Li, Yilun Du, Joshua B. Tenenbaum, and Antonio Torralba. Learning to compose visual relations. NeurlPS, 2021. 2
Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis. Learning nonlinear operators via deeponet based on the universal approximation theorem of operators. Nature Machine Intelligence, 3(3):218-229, 2021. 2
Weili Nie, Arash Vahdat, and Anima Anandkumar. Controllable and compositional generation with
latent-space energy-basse based models. NeurlPS, 2021. 2
Erik Nijkamp, Mitch Hill, Tian Han, Song-Chun Zhu, and Ying Nian Wu. On the anatomy of meme-based maximum likelihood learning of energy-based models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5272-5280, 2020.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville, Film: Visual reasoning with general conditioning layer. In AAAI, 2018. 4, 12
10

---

Under review as a conference paper at ICLR 2023
Yulia Rubanova, Alvaro Sanchez-Gonzen Gonzalez, Tobias Pfaff, and Peter Battaglia. Constraint-based graph network simulator. ICML, 2022.
Sjoerd van Steenkiste, Michael Chang, Klaus Greff, and Jürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. ICLR, 2018.
1,2
Max Welling and Yee Whye Teh. Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.3 3
Liu Yang, Constantinos Daskalakis, and George E Karniadakis. Generative ensemble regression: Learning particle dynamics from observations of ensembles with physics-informed deep generative models. SIAM Journal on Scientific Computing, 44(1):B80-B99) 2022.

---

Under review as a conference paper at ICLR 2023
A APPENDIX
In this appendix, we present implementation details in Section A. Details about the architecture and training procedure can be found in Section A.2. Next, experiment settings are described in Section A.3. Following, additional image generation results in Section A.4, together with additional quantitative results. We further run an ablation study of each of our proposed components in Section A.5, together with an ablation study in test-time. Following, we evaluate the energy of NCI in longer time roll-outs in Section A.6. Finally we discuss the broader impact of our work Section A.7.
A.1 IMPLEMENTATION DETAILS
Software: We implemented this method using Ubuntu 18.04, Python 3.6, Pytorch 1.10, Cuda 11.2 and several additional libraries which will be provided environment requirements file.
Hardware: For each of our experiments we used GPU RTX 2080 Ti (Blower Edition) with 12.8GB of memory. Models are trained for approximately day.
A.2 ARCHITECTURE AND TRAINING DETAILS
In this section we discuss depth the architecture of the main modules of our method. We also discuss the idiosyncrasies of our training procedure.
Architecture The architecture of NCI composed of main modules: (i.) The encoder in Figure A3 is composed by convolutional and multi-layer perceptron blocks, with ELU activation functions. encodes the observable trajectory x(1...T') into set of latent codes per edge, with total of N x (N 1) edges. (ii.) The short-term energy function in Figure A2 processes the trajectory in chunks 5 time-steps. (iii.) The long-term energy function in Figure Al processes the trajectory with several convolutional filters, while reducing its temporal resolution. It finally temporally pools the whole trajectory. It is designed to observe the overall shape of the trajectory. Both energy functions make of the Swish activation function. The resulting energy is the summation of the short and long-term energies E = ELr + EST- The terms node - edge and edge + node correspond to the different steps of message passing procedure. In node edge, information from connected node pair is concatenated an edge representation. edge - node represents the summated contribution of all edge features connected every node. The conditioning blocks modulate the energy function features by means of FiLM modulation Perez et al. (2018).
An illustration of the overall architecture can be seen in Figure A1.
Constraint Splitting To generate a trajectory, we optimize the energy function E(x) = Eiga E (x; Z.j,),, across node indices i and j from to N and latent vectors I from 1 to L. Explicitly computing one energy function per edge becomes prohibitively expensive as the number of nodes trajectory increases. As introduced, the computational burden reduced by utilizing shared message passing graph network to compute fixed set of features for all edges (Tables A1, A2, A3 and Figure A1). Hence, the energy corresponding masking out the contributions of all other edges. However, in order to recombine edge types across multiple datasets With this objective, in training time we randomly split the encoded edge constraints into two disjoint subsets. The generated trajectory is a product of joint optimization of two energy functions, each one conditioned to of the subsets. Each energy function observes edge constraint subset while masking out the contributions of the rest edges.
single edge factor Zij.1 obtained by
desirable to train
model to combine multiple energy function contributions.
Regularization
To speed up training and regularize the energy values, we found useful to add the Contrastive Divergence loss LCD Hinton (2002):
LCD = (x)(x)
Σ E Ex;
(x; Ziga))
Estop grad(gu (8))
Σ 8
Ed (x: 2ig.1)
(6)
where (x) is the true distribution of the data, and go the distribution approximated by NCI. We also regularize the energy values by penalizing the squared energy resulting from above. These
12

---

Under review as a conference paper at ICLR 2023
L MSE
XM
X
X
a (l,,...,?? T)
Fix al,,...,
F
(aco( T)
xL
Constrains
Zij,
C V
x0 1(011)
Σ E
v
e
v-te
M
Figure Al: Architecture and methodology our approach. In the left, Enc, observes portion of the input trajectory and encodes them into constraints, in the form of latent vectors Z,,.. In the right, a set of energy functions parametrized as GNNs are conditioned by Zij,t at the edge level. We initialize a trajectory x" as uniform noise and the ground-truth initial conditions x()..T)) and update by minimizing the constrained energy functions. We sample by means Langevin Dynamics. supervise reconstructed trajectories with an MSE objective with respect to the ground -rrnth trajectory.
regularizations are necessary for the successful training of our model, however, they helpful to stabilize training and therefore used for all experiments. For out-of-distribution detection experiments, regularization is remarkably useful. In-distribution samples are trained have an energy close hence out-of out-of-distriben distribution samples are easily detected. Both regularizations are added to the primary objective (MSH) with a weight of Areg le
Node
Edge
Node - Figge
Node - Edge
5x1 CNN Block Down (2) 64
5x1 CNN Block Down (2) 64
5x1 CNN Block Down (3) 64
CNN Conditioning Block (2) 64
Unfold Trajectory K:5, S:1
Temporal Avg. Pool
CNN Conditioning Block (2)
Dense
MLP (2) 64
Temporal Avg. Pool
MI.P Conditioning Block (2) 64
Edge - Node
Edge + Node
MLP Conditioning Block (2) 64
MLP (2) 64
MLP
Edge - Node
Node - Edge
Dense I: Eur
MLP (2) 64
MLP (2) 64
Long-Tern Energy
Dense 1: EST
Dense + LN + (LxNum. edges)
Short- Term Energy
Table Al: Architectrre for the long-term energy function.. The energy computed eval- the whole trajectory by leveraging ID convolational layers with final temporal av- erage pooling. Numher layers specified surenbeess.
Table A3: Architecture for the encoder.
Table A2: Architecture for the short-term energy function. The energy computed eval- clunks steps
Number layers specified parentheses.
trajectory. ob- tained with strides of size I. Number of lay- ers specified in pareatheses.
13

---

Under review as a conference paper at ICLR 2023
A.3 EXPERIMENT DETAILS
In the following section we discuss the specific setting for each one of the experiments. In all cases, NCI uses Adam optimizer and a learning rate of LR 3e with scheduled decay of y 0.5 every 100k iterations ( 80 epochs). The illustrated trajectories shown in the figures have been plotted by accumulating the velocity dimensions of each predicted state to an initial ground-truth point.
Baselines We choose the baselines described Kipf et al. (2018), together with their main contribution, NRI. NRI is considered the de facto standard for relational inference, and LSTM the main baseline architecture for trajectory forecasting.
Static: Copies previous state vectors.
LSTM (single): LSTM model trained predict the state vector difference at every time-step. It consists of a tow-layer LSTTM with shared parameters and 256 hidden units. The input to the model is passsed through two-layer MLP with ReLU activations before it is passed to the LSTM. The last hidden vector of the LSTM for each time-steps is also passed through a two-layer MLP with ReLU activations, which outputs predicted state difference. This is done individually for each particle. The LSTM has access to the ground-trute input states until prediction starts.
This model similar to LSTM (single), with the difference that node states are concatenated before being processed by the model. This allows communication across node trajectories.
LSTM (joint):
NRI (learned graph): For this model, an encoder infers interactions while simultaneously learning the dynamics from observational data. The encoder outputs a latent code that represents the underlying interaction graph and the reconstruction is based on graph neural
networks.
NRI (full graph):
This instanciation of NRI is similar to the one above, with the differ- ence that the latent graph fixed. The encoder only allowed to output type of edge representation.
dNRI: Extension of NRI to dynamic relation setting. The encoder infers separate relation graphs for every time-step. The reconstruction based on graph neural networks, also in dynamic fashion.
Conditional GNN: For this model, we encode the edges similarly as in NCI, with the observed part of the trajectory. We decode them one shot by means of GNN with message passing.
IN: This follows the official implementation provided by Battaglia et al. (2016). In this case, objects and relations are encoded separately, and propagated in time with GNN. This model does not observe any part trajectory (unconditional).
Social LSTM and Directional LSTM (TrajNet++): This two models are implemented by Kothari al. (2021) and executed following instructions in their github https:/ usbasel lines. The architectures are LSTM-based and use different types pooling functions to gather information surrounding each node. The code is addapted to a state with dimensionality 4 instead of 2.
gi thub. com vv ta teiitaa-elll / jnetpl
Quantitative Comparison We utilize the same setting and simulated datasets as detailed Kipf et al. (2018), including their baselines: Static, LSTM (single), LSTM (joint) and variations of NRI. For the C-GNN, we train for 400 epochs with batch size of 40 and D. = 64. We encode 50 time-steps and predict the following 20.
For Springs and Charged datasets, NCI is trained with 2 energy functions and latent size per edge constraint of D_ 64. We encode 49 time-steps into set of constraints, fixes 1 time-step from the ground-truth and predicts the following 20. Number of sampling steps M varies from 3 to 6 along the first 300k iterations and step-size of X 0.4. We use batch size of 40 and train for 400 epochs.
Similarly, we run experiments in NBA Sports dataset with the same baseline setting for LSTM (joint) and C-GNN and the same data normalization scheme. C-GNN trained similarly as before
14

---

Under review as a conference paper at ICLR 2023
with batch size of 8 for 25 epochs. For prediction in NBA experiments, NCI encodes 40 time-steps into a set of constraints of dimension D2 = 64, fixes time-step from the ground- -oound-truth and predicts the following 20. In training and testing, only the 20 predicted time-steps are generated and supervised with the ground-truth trajectory. Our model is trained for 25 epochs, with a batch size of 6 and single set of edge constraints. Number of sampling steps M varies from 3 to 5 along the first 200k iterations and step-size of X = 2.
For this dataset, results in social interaction-based networks (S-LSTM, D-LSTM) are very poor in long term. This behavior has been discussed with the authors of D-LSTM (Kothari et al. (2021)), I),, and concluded that it is expected when solving a task of prediction. This model is usually employed for tasks such as pedestrian collision avoidance with teacher forcing training. After switching the objective function to a purely MSE loss and changing the training strategy with the authors' help, the results improved very slightly. The model converged after 25 epochs.
NRI and dNRI are trained with a batch size of 40 and 32 respectively, learning rate of LR= 5e and hidden sizes of 256. We train them for 35 epochs, until convergence which which takes approximately 24h. For both architectures, we use 2 edge types as recommended in the respective papers. Specifically, in NRI they try for higher number of edge types and conclude that the model is overfitting.
Finally, experiments in JPL Horizons dataset are with the same baseline setting NBA Sports In this case, we train LSTM (joint), C-GNN and NCI for 2000 epochs (given the small size of the dataset). In none of the cases we see signs of overfitting. C-GNN and NCI use batch size of 10 and 6 respectively. NCI has a number of sampling steps M that varies from to 5 along the first 300k iterations and a step-size of A 2. In this experiment NRI and dNRI are trained for 2000 epochs with the same hyperparameters as in the NBA experiments. For both experiments we observe that dNRI performs worse than NRI. We tried the best of our ability to obtain the expected results, but after hyperparameter tuning including modifications in the number of edge types, we conclude that the differences are negligible and therefore we keep the default settings.
Recombination While not necessary, we found recombinations slightly more natural-looking by leveraging variation of the original architecture. In this case, each EBM has branch that evaluates trajectory energies unconditionally. That branch is solely used by edges that have been masked out (i.e. conditioned in another EBM). The unconditional architecture is therefore very similar to that in Al and A2 but disregarding the conditioning blocks. We train NCI for 120 epochs both in Charged and Springs datasets separately, with latent size per edge constraint of of D = 8. For this experiment, NCI is trained to reconstruct 30 observed time-steps and predict the following 10, as we find prediction helpful for proper constraint learning. We use a dataset instance with double sampling frequency than for the other experiments. For data augmentation, we select randomly the initial point of the trajectories in the range To = [1, (1..... 50). velocity distributions diverge along the trajectories. The encoder observes the input data both rotated and mustance-normaliin In test-time, we sample M- times with a step-size of À 16.
Out-of-distrie distributiion Detection We utilize the same Springs and Charged dataset variations as for the recombination experiments, For evaluation we also utilize the Charged-Springs dataset explained in the main body of the paper. The energy values are obtained at the node level by evaluating ground- truth trajectory. The hyperparameters of the model are those of the recombination experiments.
We found especially useful to use the regularization in Equation 6. This maintains the in-distribution energy close to 0, while increasing energy from -itor--distribution nodes.
Flexible Generation For this experiment, we train model in NBA VU dataset with the same hyperparameters as for quantitative comparison. However, in this case we only reconstruct 40 time-steps. The formulation of the hand-crafted constraints is described with detail in the main body of the paper. À corresponding to the weight of the new constrain is found by grid search by means of visual inspection. However, it is fixed for new instances of the experiment.
Edge Classification For edge classification, we decode the constraints into edge types (1 or 0) with a dense layer. In all cases, we use models trained for prediction and decode the latent codes product of observing the 49 initial time-steps with the model. We train the dense layer with BCE
15

---

Under review as a conference paper at ICLR 2023
Model
Springs
Charged
objects
LSTM (linear dec.)
61.4
60.3
NCI (linear dec.)
97.4
69.4
Table A4: Accuracy (in %) true interaction recovery. latent codes. For LSTM, we linearly project the last hidden vector pairs associated to an edge between two nodes, excluding self-loops.
linear decoder to predict edge types from
(binary cross-entropy) loss and LR with shared parameters and 64 hidden units that models each trajectory individually. It receives ground-truth inputs until step 49 and is conditioned to the previous time steps. For this experiment, we keep the last hidden state of the second layer of the LSTM 128 dimensional edges by concatenation the node representations and decode them linearly into an edge type, training the layer for 75 epochs. Table A4 is discussed in Section A.4. Baseline setting for results in Table are detailed in the main body of the paper.
4. For the LSTT in Table A4, we train two-layer LSTTM
node representation. We generate
16

---

Under review as a conference paper at ICLR 2023
GT Springs
GT Charged
Recombined
Figure A2: More examples of recombinations. Illustrated, samples from Springs (Col. 1) and Charged (Col. 2) and their recombinations (Col. 3). NCI encodes both trajectories. NCI is able to reconstruct trajectories framed in green with the initial conditions marked in blue, while swapping the edge constraints associated to the nodes in the red dashed box. Recombinations look semantically plausible and smooth.
A.4 MORE EXAMPLES AND ADDITIONAL RESULTS
In this Section, we illustrate more examples of the main experiments and additional quantitative
results.
Qualitative Examples Figure A2 illustrates cross-dataset combinations of edge constraints. We encode ground-truth trajectories (columns 1 and 2) separately into their corresponding energy constraints. We train different model for each data distribution. Next, fix initial conditions xo (blue points) of the scene framed in green and aim to reconstruct them. In test-time we generate trajectory by minimizing simultancously the energy functions corresponding to the two models. Each model targets specific edges. Particularly, one model adds constraints corresponding to the mutual edges of the particles highlighted by red dashed box, while the other aims reconstruct the rest of the trajectories.
Next, Figures A3 and A4 show qualitatively the predictions of NCI compared the ground truth. In both cases, the initial 49 steps are the ground-truth trajectory. The black dot indicates the beginning of our model's predictions (the trajectory color gets lighter with time). Predictions are often accurate. In cases where there is significant difference with the ground-truth (e.g. green node in the center of Figure A4), the predicted trajectory looks semantically plausible.
Figure A6 illustrates the ability of NCI to add hand-crafted constraints in test-time. In the top row, trajectory reconstruction together with added goals that the nodes (players) are attracted to. In the bottom row, trajectory reconstruction followed by the same trajectory with variations of the agent velocities. The resulting plots show how both the edge constraints and the newly added hand- cradtcra constraints are respected.
17

---

Under review as a conference paper at ICLR 2023
Springs: + 20 predictions
20 Pred. NCI
Ground Truth
Figure A3: Qualitative example 20 predictions in the future of NCI in Springs partieles dataset. In both rows, first steps are the ground-iruth iruth. The black indicates the beginning of predictions row all cases, predictions look almost identical to the ground-truth.
Charged: 50 GT 20 predictions
20 Pred. NCI
Ground Truth
predictions the future NCI the Charged particles dataset. both rows, first 50 steps are the ground-truth The cases, predictions are fairly close to the ground-truth. In the cases where they differ (green node center) the predictions smooth and look reasonable.
Figure A4: Qualitative example
black indicates the heginning of predictions row all
Finally, Figure A5 illustrates an example of the sampling procedure. By leveraging Langevin Dynamics sampling we refine our predictions iteratively in gradient-based optimization procedure. The model quickly understands the general shape of the trajectories and refines them locally using the final sampling steps.
Edge Linear Decoding Results We show in Table A4 the quantitative comparison of NCI with respect to the LSTM (single) baseline for linear edge type decoding. Note that NRI learned edge representations are already a classification of explicit edge types and therefore do not allow for linear decoding. The resulting accuracy of NCI surpasses that of the LSTM by significant margin.
18

---

Under review as a conference paper at ICLR 2023
Sampling Procedure in Trajectory Reconstruction
Steps:
Ground Truth
Figure AS: Examples of trajectory reconstruction procedure time-steps Charged dataset. initialize the velocity as uniform noise and sample times (in this case) using Langevin Dynamics. We obtain faithful reconstruction of the ground truth trajectories.
A.5 ABLATION STUDY
Following, we add an ablation study carried on in the Charged dataset, with 370k iterations and batch size of 64. Here, we analyse NCT's performance through a variety design choices. Those are the following:
Latent size: We explore different sizes for the edge constraints. The choices LS E ( (16, 32, 64, 128).
Langevin step size. The choices are À {0.1, 0.2, 2.0, 6.0, 10.0, 14.0).
Objective: We evaluate the impact of adding the contrastive divergence objective to the MSE reconstruction loss.
Edge masking strategy: In setting with 2 EBMs, we evaluate 1) a random masking strategy, 2) masking strategy based the edge contribution specific node and 3) no masking strategy.
Decoder baseline: We substitute the iterative energy minimization by a feed-forward graph decoder. This is equivalent to the Conditional GNN baseline.
For this experiment, we follow the setting in the quantitative comparison and simply modify the variable interest. Tha analysis shows how small langevin step desirable terms performance (A 0.2). The model seems to be robust to the latent size choice, although there is slight preference for LS 32. When it comes to the objective, we show quantitatively that regularizing training with contrastive divergence term improves performance. We can also observe how masking strategy is better than none. Finally, we show by comparing to a decoder baseline how our sampling strategy is erucial for competitive results. Our ablation study is summarized in Table A5.
Similarly, we perform an ablation at test-time, with our best trained model for the Charge dataset. We analyse NCT's performance under the following variations:
19

---

Under review as a conference paper at ICLR 2023
Number of Langevin steps: We train our model with M 6 langevin steps and test it with
M [1,2,3, 3, 4, 5, 6, 8, 9, 10, 20).
Number of nodes: We train NCI with 5 nodes and test in datasets with N = and N
Results are summarized in Table A6. "Train" indicates the setting used in training.
A.6 LONGER- ROLL-OUTS
We evaluate our model against NRI in longer term prediction. Given the chaotic nature of trajectories, we evaluate the energy among consecutive states (squared pairwise difference) for 100 timesteps. This time, we employ our model ruto-regressively in the Charged dataset, while it has been trained to predict 20 steps one-shot. RI indicates that iteratively predict 1 time-step. Results in Table A7 show how NCI preserves approximately more than times of the energy than NRI as time unfolds.
A.7 BROADER IMPACT
Understanding interactions across agents a trajectory fundamental to explain their present behavior and predict their future. The importance of such understanding is higher when we do not have access to the true interaction types or they are simply not discrete set. those cases, being able to learn representations of interactions from observational data provides a window into the physics of the world we live in. These property is desirable in AI for applications such as molecular dynamies modeling autonomous vehicles, which have huge impact our lives. Despite the fact that there a wide range of approaches for inferring interactions and predicting trajectories, there relatively little work inferring these in interpretable manner. Our model aims learn these interaction constraints that allow for higher degree of manipulation over the learned representations. This interpretability and manipulability properties are important to AI, but might raise concerns of abuse. Our approach, similar to many other approaches, may capture the implicit biases present in data. There is also the potential threat of attacks to systems that rely on interpretable models, which can be more easily targeted than those which are opaque.
20

---

Under review as a conference paper at ICLR 2023
Reconstruction: Edge Constraints
(Edge + Goal 1) Constraints
(Edge + Goal 2) Constraints
0.0
-0.2
-0.2
-0.2
-0.4
-0.4
-0.4
-0.6
-0.6
-0.6
-0.8
-0.8
-0.8
-0.50
-0.25
-0.75
-0.25
Reconstruction: Edge Constraints
(Edge- Velocity) Constraints
(Edge Velocity) Constraints
0.0
-022
-0.2
-0.2
-0.4
-0.4
-0.4
-0.6
-0.6
-0.6
-0.8
-0.8
-0.8
Figure A6: More examples of NBA reconstructed trajectories with added constraints in test-time. NCI can generate realistic trajectories by respecting both (i). The learned edge constraints (ii). new hand-crafted constraints added in test-time. Illustrated (top row) we see the reconstruction the trajectory with an added goal. We can see that the trajectories have the tendency to attracted in the direction indicated by the bluc arrow. The actual goal is located outside the frame. We can see (bottom row) constraints of higher and lower velocity than the reconstruction. In all cases, the trajectories follow the new constraints without loosing their original constraints.
Ablation Study
step
10 step
20 step
Latent Size
16
1.20e-3
3.75e-3
7.06e-3
32
9.87e-4
3.60e-3
6.82e-3
64
1.15e-3
3,82c-3
7.00e-3
128
1.15e-3
3,91e-3
7.24e-3
Sampling step size
0.1
9.54e-
3.53e-3
6.81e-3
0.2
9.07e-4
3.36e-3
6.62e-3
2.0
1.16e-3
3,82e-3
7.09e-3
6.0
1.15e-3
3,82e-3
7.00e-3
10.0
1.14e-3
3.78e-3
7.07e-3
14.0
1.19e-3
3.91e-3
7.29e-3
Masking type
Mask random
1.15e-3
3.82e-3
7.02e-3
Mask by node
1.15e-3
3.78e-3
7.03e-3
No masking
1.17e-3
3,82e-3
7.09e-3
Objective
MSE+CD
1.15e-3
3,82e-3
7.02e-3
MSE (no CD)
1.36e-3
4.02e-3
7.47e-3
Decoder (no sampling)
3.27e-3
5.31e-3
9.65e-3
Table A5: Ablation study investigating effects of different factors like latent size, sampling step size, masking type and objective results.
21

---

Under review as a conference paper at ICLR 2023
Ablation Study in Test
step
10 step
20 step
Number of Langevin steps M
1.06e-1
1.18e-1
2.47e-1
3,23e-2
4.34e-2
8.22e-2
7. 7.00e-3
1.16e-2
2.68e-2
1 1.86e-3
4.63e-3
1.00e-2
1.03e-3
3.51e-3
7.05e-3
6 (Train)
8.85e-4
3.33e-3
6.54e-3
9.00e-4
3.33e-3
6.45e-3
8
8.93e-4
3.36e-3
6.47e-3
9.00e-4
3.39e-3
6.60e-3
10
9.10e-4
3.44e-3
6.66e-3
20
1.03e-3
3.64e-3
7.04e-3
Number of nodes N
1.79e-3
3.71e-3
6.74e-3
(Train)
8.85e-4
3.33e-3
6.54e-3
1.34e-3
4.66e-3
1.01e-2
Table A6: Ablation study investigating effects different factors test-time: Number of langevin steps and
number of nodes.
Energy Evaluation (1e-3
1-2
10-11
20
-21
30-31
40-41
50-51
60-61
70-71
80-81
90-91
100-101
Baseline
5.0
4,9
5.0
5.0
4.9
4.9
5.0
4.9
5.0
5.0
4,9
NCI (R1)
2.4
2.0
1.9
NRI
1.2
0.75
0.65
0.60
0.68
0.62
0.61
0.70
0.59
0.57
0.65
Table A7: Energy evaluation NCI preserves more energy in long rollouts than NRI
22


Reviews:
[{"review_id": "aVVj9nvlHz", "reviewer": "Reviewer_ShW2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an implicit method based on an energy model for discovering the underlying model in a dynamical system and forecasting agents' future dynamical processes. Instead of using a VAE-like solution, this paper proposed a EBM-based solution. On a variety of simulated data, the authors demonstrate the effectiveness of their algorithm and achieve superior results compared to a reasonable baseline.", "summary_of_the_review": "Overall, I tend to vote for negative results and my main concern comes from innovation. I am not convinced that the proposed method is innovative enough, especially compared to previous work such as NRI. There is still room for further exploration of the proposed method in terms of its practicality, training efficiency, and robustness.", "strength_and_weaknesses": "[Strength]\n* S1: The research topic's inspiration is vivid and captivating.\n\n[Weeknesses]\n* W_1: The paper was somewhat challenging to follow, and the logical connection between several lines was unclear. The entirety of section 2 did not aid comprehension of literature.\n* W_2: The empirical result is insignificant, and some baseline are absence(e.g. the recent work in section 2)\n\n* W_other: some misused of cite and citep(Intro, para3)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UhKhsOp-sB", "reviewer": "Reviewer_m9s9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a way to learn the dynamics between interacting particles by learning the constraints between them. It shows it can predict trajectories on synthetic and real datasets.", "summary_of_the_review": "The two main issues I have with this paper are (1) It is unclear how the main concept of constraints is implemented besides the use of a EBM. (2)  The empirical evaluation is weak due to missing comparisons to more relevant and challenging baselines.", "strength_and_weaknesses": "Strength:\n- The paper is well-written and easy to follow\n- Paper shows good empirical results.\n\nWeaknesses:\n- Limited novelty, the main contribution is replacing the VAE in NRI with an EBM.\n- Unclear use of the term constraint. The so-called inferred constraints are simply the energy in the EMB. There is no real connection to any physical or mathematical constraint. I fail to see how this connects to constraints on the system, besides the fact that any generative model limits low-density areas. \n- Missing comparisons in the experimental section. Tab. 2 does not include NRI results, and both tables do not show recent relevant baselines for comparison, e.g. Graber & Swhing \"Dynamic Neural Relational Inference\", Salzmann et al \"Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data\", and Rubanova et al \"Constraint-based graph network simulator.\".\n- Regarding Rubanova et al, it is not \"Concurrent to our work\" as it has been on arxiv for almost a year now and published in ICML 2022.\n- It is not clear how the \"constraints\" related to actual relations and the lackluster accuracy in prediction is not promising.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "T9KkLt5r2A", "reviewer": "Reviewer_1k6S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to learn an energy-based conditional generative model from trajectories of multiple particles. Specifically, the model (i) uses a graph neural network to encode partial trajectories into typed relations (each type is equivalently seen as a constraint) and then (ii) uses such information to define an energy score function by summing all the energy contributions of each relation. A standard gradient-based procedure for energy-based models is used to sample the full trajectory given the extracted relations, thus acting as a decoding process. The proposed model is learnt by maximising the conditional log-likelihood.\n\nFrom a conceptual point of view, the work differs from related approaches, including the neural relational inference (NRI), in the sense that it tackles the same problem of NRI using an energy-based formulation instead of a VAE one. From a computational point of view, the work replaces the explicit decoder in NRI with a sampling procedure (involving the use of second-order derivatives). From a task point of view, the energy-based formulation enables to tackle some challenges currently out of reach for NRI, including the OOD detection and the conditional generation of additional user-defined constraints (e.g. constraints defined as differentiable functions on the trajectories) introduced at test time.\n\nExperiments are conducted on 2 toy datasets, namely springs and charged particles, and 2 real datasets, including NBA SportVU and JPL Horizons.", "summary_of_the_review": "------- POST REBUTTAL -------------\n\nThe answers in the rebuttal partially address my concerns. Doubts still remain on the nature of the method and the experimental methodology (both in terms of predictive performance and computational cost).\n\nThe proposed model learns a conditional distribution $p(x|x_{partial})$, where $x$ and $x_{partial}$ are the full and partial trajectories, respectively. The definition in Section 4.3 Eq. (3) is not appropriate. The model is not conditioned on the constraints, which are learnt and not given, but it is conditioned on the partial trajectories. This is not only a minor technical issue, but it has also an important practical consequence. In fact, the model is, in all effect, autoregressive and the experiments should take into account this fact.\nMy original concern about the comparisons with neural relational inference (NRI) still remain after the rebuttal. In both old and new experiments, NRI outperforms the proposed method in terms of predictive performance for step 1 (both on toy and real data, see for example \"General comments 2/2 part. 2.2\"). However, when considering larger future steps predictions, namely 10 and 20 (for real), the phenomenon is inverted. This sheds some lights on the unfair methodology used in the comparisons. The proposed method has been trained to deal with predictions (up to 20 steps) contrarily to NRI, which deal with them in an autoregressive way. Therefore, the authors should consider the same conditions for comparisons and treat their model as an autoregressive one.\n\nAdditionally, the analysis in terms of computation for the proposed approach provided in the rebuttal is not informative. The authors have shown an empirical comparison against NRI by considering fixed values of batch size and trajectory length. What does it happen when increasing the number of particles, their feature dimensionality or the trajectory length? I expect that the computation of second-order derivatives (required by the training/decoding component) becomes quickly intractable in these cases, whereas NRI is still able to handle them. However, such analysis is still elusive and missing.\n\nOverall, I think that the idea of using an energy-based model is nice as enabling to handle new tasks, including OOD detection and dealing with new constraints at test time. However, I feel that the experimental analysis is still not complete and the main claims are not well supported (see e.g. in the abstract \"achieving more consistent long-rollouts than existing\napproaches\"). Consequently, I decrease my score.", "strength_and_weaknesses": "**Strengths**\n- The paper is well-written. I enjoyed reading it.\n- The idea of learning an energy-based model in the dynamics setting is to my knowledge new. Maybe you can also consider some recent related work [1].\n- The proposed solution enables to tackle new challenges for the toy cases (OOD and conditional generation with user-defined constraints) and therefore has a good potential.\n\n**Weaknesses**\n- Several experimental comparisons against NCI are missing and should be included (see below for further details).\n- Code is not available.\n\nRegarding the first weakness:\n- Experiments (general) Computationally speaking, it is not clear how the proposed model compares against NRI. Specifically, while NRI uses an autoregressive model for the decoding process, NCI uses a gradient-based procedure for sampling, which requires the computation of second-order derivatives during training. How are these derivatives handled during training? And can you provide a discussion and perhaps a quantitative comparison both for training and inference with NRI?\n- Experiments (general). The proposed model is in essence a conditional generative model. How does the model generalise to predictions at future time steps larger than the ones seen during training? As far as I understand, the model can be iteratively applied by setting the initial state of the sampling process to the last generated portion of the trajectory, thus enabling to see further in the future. Is this correct? If so, why not comparing on this task against NRI?\n- Experiments (general). How does the model generalise to a different number of objects seen during training?\n- Experiments (toy) on traditional generation and edge classification tasks. In Table 1, NCI seems to achieve similar performance to NRI. In which sense is the model outperforming NRI, can you elaborate more on that? Additionally, I found a bit confusing the experiments on edge classification, as NCI is clearly inferior to NRI. What is the message here?\n- Experiments (real) why there is no comparison with NRI?\n\nRegarding the second weakness:\n\nAre you planning to integrally release the code?\n\nSome minor comments/curiosities:\n- The maximization of the conditional log-likelihood requires to minimize the energy score for training data while maximising it for generated data. Why this second optimisation term is not used in all the experiments, but only for OOD detection (as mentioned in the Appendix A)?\n- In Table 1, please add the lower bound baseline with known graph.\n- Can you think of other real cases other than the NBA dataset where additional constraints can be specified?\n- Last paragraph in the introduction: decompositionS of interactions is disentagleMENT -> decomposition of interactions is disentangled.\n\n[1] ModLaNets: Learning Generalisable Dynamics via Modularity and Physical Inductive Bias (ICML 2022)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HRZBST7R2bT", "reviewer": "Reviewer_C8J2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes neural constrain inference (NCI), a dynamics model which predicts future trajectories by inferring interactions between particles represented as sets of energy constraints, and sampling from the resulting energy-based model (EBM) using Langevin dynamics. It is demonstrated on various dynamics task that NCI yields lower rollout errors compared to methods based on standard feed-forward prediction, such as interaction networks. Additionally, various useful features of the method are demonstrated, namely disentanglement of interactions, the possibility of the recombination of different constraints, out-of-distribution prediction, and the introduction of novel hand-made constraints at test time.", "summary_of_the_review": "Overall, this is a well written paper presenting an interesting idea, in a way that is well supported by experimental evaluations. While it could be argued that it offers limited novelty over Rubanova et al., I am of the opinion that when different aspects of a genuinely novel idea are concurrently developed by two papers, there is value in considering both perspectives. I am therefore leaning towards acceptance.", "strength_and_weaknesses": "Strengths:\n - The paper addresses an important problem in an interesting way, based on a clear motivation.\n - The method is sensible and clearly presented.\n - The quantitative empirical results are convincing and support the authors' claims.\n - The method's capabilities are thoroughly explored through additional experiments.\n\nWeaknesses:\n - The novelty of the paper is put into question by a concurrent work (see below).\n - The qualitative results seem to generally match the claims in the text, but are somewhat hard to read from static images. It would be useful if the authors would provide video comparisons of the predicted trajectories.\n - The claim of long-term stability of the rollouts could be explored further, beyond the 20 timesteps measured in Table 1. While due to the chaotic nature of the systems considered, it cannot be expected that predicted trajectories exactly match ground truth beyond this point, long term stability may still be evaluated. For instance, by providing long video sequences, or by measuring the degree to which physical conservation laws hold (as in https://arxiv.org/pdf/1910.02425.pdf Fig. 5).\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject