Roto-translated Local Coordinate Frames For Interacting Dynamical Systems
Authors: Miltiadis Kofinas
Naveen Shankar Nagaraja
Efstratios Gavves
Venue: NeurIPS 2021 Poster (2021)
Keywords: Trajectory forecasting
interacting dynamical systems
graph neural networks
roto-translation equivariance
equivariance
invariance
geometric graphs

Abstract: Modelling interactions is critical in learning complex dynamical systems, namely systems of interacting objects with highly non-linear and time-dependent behaviour. A large class of such systems can be formalized as $\textit{geometric graphs}$, $\textit{i.e.}$ graphs with nodes positioned in the Euclidean space given an $\textit{arbitrarily}$ chosen global coordinate system, for instance vehicles in a traffic scene. Notwithstanding the arbitrary global coordinate system, the governing dynamics of the respective dynamical systems are invariant to rotations and translations, also known as $\textit{Galilean invariance}$. As ignoring these invariances leads to worse generalization, in this work we propose local coordinate systems per node-object to induce roto-translation invariance to the geometric graph of the interacting dynamical system. Further, the local coordinate systems allow for a natural definition of anisotropic filtering in graph neural networks. Experiments in traffic scenes, 3D motion capture, and colliding particles demonstrate the proposed approach comfortably outperforms the recent state-of-the-art.

Roto-translated Local Coordinate Frames For Interacting Dynamical Systems
Miltiadis Kofinas
Naveen Shankar Nagaraja
Eftratios Gavves University of Amsterdam egavves@uva.nl
University of Amsterdam loffinssuvaa.n
BMW Group
Naveen-Shankar Nagaaaaaaaaaaaaaaamm....
Abstract
Modelling interactions is critical in learning complex dynamical systems, namely systems of interacting objects with highly non-linear and time-dependent behaviour. A large class of such systems can be formalized as geometric graphs, i.e., graphs with nodes positioned in the Euclidean space given an arbitrarily chosen global coordinate system, for instance vehicles in a traffic scene. Notwithstanding the arbitrary global coordinate system, the governing dynamics of the respective dy- namical systems are invariant rotations and translations, also known Galilean invariance. As ignoring these invariances leads to worse generalization, in this work we propose local coordinate frames per node-object induce roto-tran -to-translation invariance to the geometric graph of the interacting dynamical system. Further, the local coordinate frames allow for natural definition of anisotropic filtering graph neural networks. Experiments in traffic scenes, 3D motion capture, and col- liding particles demonstrate that the proposed approach comfortably outperforms the recent state-of-the-art.
Introduction
Modelling interacting dynamical systems -systems of interacting objects with highly non-linear and time-dependent behaviour- with neural networks attracting a significant amount interest 227 3 [19 for its potential to learn long-term behaviours directly from observations. A large class of these systems consists of objects in the physical space, for instance pedestrians a traffic scene 31 37] colliding subatomic particles (5). These systems can be formalized geometric graphs. in which the nodes describe the physical coordinates of the objects among other features. Kipf et al. [27] introduced Neural Relational Inference (NRT) to learn geometric graph dynamical systems using the
variational autoencoding framework 25 38 Following [27) dynamic NRI 19 advocated sequential latent variable models encode time- trnesiransient behaviours. Both approaches and the majority of learning al algorithms for dynamical systems assume an arbitrary global coordinate system to encode time-transient interactions and model complex behaviours. In this work, posit that taking into account the relative nature of dynamics is key to accurate modelling of interacting dynamical systems.
Represented by geometric graphs, learning algorithms of dynamical systems subscribe themselves to the Newtonian space. The absolute notion Newtonian space and mechanics, however, determines that there exist infinite inertial frames that connect with each other by rotation and translation. Each of these inertial frames is equivalent in that they can all serve as global coordinate frames and, thus, arbitrary choice is made. Notwithstanding this arbitrariness, the dynamics of the system invariant to the choice global coordinate frame up to a rotation and translation, in what is also known Galilean invariance. Put otherwise, geometric graphs of interacting dynamical systems often exhibit symmetries that if left to their own devices lead models to subpar learning.
Inspired by the notion of Galilean invariance, we focus on inducing roto-translation invariance in interacting dynamical systems and their geometric graphs to sustain the effects of underlying
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

pathological symmetries. Symmetries, invariances and equivariances have attracted an increased 54
interest with learning algorithms in the late years [T] 53 increasing complexity of new tasks and data, exploiting the symmetries improves sample efficiency 17 41 by requiring fewer data points and gradient updates. To date the majority of works on exploiting symmetries or data augmentations are with static data [1] 44 (54). We argue and show that invariance in representations of dynamic data is just as important, if not more, as is critical in accounting for the inevitable increased pattern complexity and non-stationarity.
46 T7..
The reason that with the
We induce roto-trame tron-raanlltiinn invariant representations in graphs by local coordinate frames. Each local coordinate frame is centered at node-object in the geometric graph and rotated to match its angular position -yaw, pitch, and roll. Since all intermediate operations are performed on the local coordinate frames, the graph neural network is roto-trane trto-translation invariant and the final transformed output is roto-translation equivariant. We obtain equivariance to global noto-translations by an inverse rotation that transforms the predictions back to the global coordinates.
We make the following three contributions. First, introduce canonicalized roto-translated local coordinate frames for interacting dynamical systems formalized in geometric graphs. Second, by operating solely on these coordinate frames, we enable roto-translation invariant edge prediction and roto-translation equivariant trajectory forecasting. Third, we present a novel methodology for natural anisotropic continuous filters based on relative linear and angular positions of neighboring objects in the canonicalized local coordinate frames. We continue with a brief introduction of relevant background and then the description of our method. We present related work and finish with experiments and ablation studies on number of settings, including modelling pedestrians in 2D) traffic scenes, 3D particles colliding, and 3D human motion capture systems.
2 Background
2.1 Interacting dynamical systems and geometric graphs
An interacting dynamical system consists of i = 1,..., N₂ objects, whose position p (Pa. py: P=)) and velocity u dp (ug, Ny. u in the Euclidean space are recorded over time t. The state of the i-th object at timestep is described by x = [p uf], adopting for clarity a column vector notation and using -] to denote vector concatenation.
dt
Over the past few years, natural way that has emerged for organizing interacting dynamical systems by geometric graphs
19 through space and time, 9 = (g) ( where G' (V*,E*) is the snapshot of the graph at timestep t. The nodes V = [rf ., v. ) of the graph correspond to the objects in the dynamical system, with of corresponding to the state of the i-th object, x/. The edges& Er C {ce.,- = (vf, uf) (u), vf) E yt x V*} of the graph, correspond to the interactions from node-object j to node-object We use Miii to denote the graph neighbours of node 2/2 In the absence of domain knowledge about how objects connect, for instance the links between atoms in molecules, the graph is fully connected. Explicit inference of the graph structure can be achieved by using latent edges xf,; corresponding to the edges ef..-
Graph neural networks 42 32 18 exchange messages between neighbors and update the vertex and edge embeddings per layer, commonly referred to as message passing
(1) = f!())
([n (11) h (1-1) by
11)
(1)
h = f(!)
h 1-1)
(2)
(E
where h,C' is the embedding of node e at layer 1 and h 1. is the embedding of edge Cj,a at layer I. fe: fo denote differentiable functions such ML.s and O commonly a summation or an average. Many graph neural networks rely on isotropic filters (26).
denotes permutation invariant function,
although various ways 47 36] to cireumvent this constraint have also been explored.

---

(2)
Figure 1: In (a), objects positioned in an arbitrary global 2D coordinate frame; arrows represent orientations, In (b)-(e), objects in the canonicalized local coordinate frames, translated match the larget object's position and rotated to match its orientation
2.2 Neural relational inference
Kipf et al. proposed neural relational inference (NRI), a variational autoencoding inference model [25 38 that explicitly infers the graph structure over discrete latent graph and simultaneously
learns the dynamical system. Given the input trajectories the encoder learns to infer interactions
as latent edges Zj.a E [0, 1]K sampled from concrete distribution 34, 23 with edge types. The decoder receives the inferred interactions and the past trajectories, and learns the dynamical
system, Pp (x I (x +1 |x!' z). The encoder is regular graph neural network without explicitly taking time into account and learns to infer latent interactions by maximizing the evidence lower bound for the next time step. The decoder is another graph neural network that either assumes
Markovian dynamics po (x*+1|x 1it .2) Po 1 x z)) or is recurrent through time. As the prior and encoder in NRI assume static interactions (e.g. whether forces between charged particles are attractive repulsive), dynamic NRI [19 replaces them with sequential relation prior based on past states, P (z) x) I Po (z' x1:1 2 211t-1), and approximate relation posterior based on both the past and future states. The decoder is also reformulated as pe (x(2) = I[ (x ++1
( | [x2: 2
taking into account the dynamic nature of interactions.
2.3 Invariance and equivariance
Last, we give very brief introduction invariance and equivariance. A function f X + y is cquivariant [54] under group of transformations if every transformation π E 1l of the input x E X can be associated with a transformation v E & of the output, ([f(x)] = fxx(x)). A special case is invariance, where V {I}, the identity transformation, f(x) = f
In this work, t, and rotation invariance/equivariinee,, iee, Q = f(Qx) using the rotation matrix Q.
interested in translation. i.e., f(x) + T f(x 7) with the translation vector
3 Roto-translation invariance with local coordinate frames
In this section present method, termed LoCS (Local Coordinate frameS). We start with the derivation of roto-translated local coordinate frames and continue with the formulation of graph networks and continuous anisotropic filters operating in these frames.
3.1 Local coordinate frames
Starting from the spatio-tempont graph G. we focus for clarity pairs of node-objects in the same time step, x x the real world, objects are not point particles and have a spatial extension.
Central to our method is the use pitch and roll, that describe the orientation of a rigid body with respect to the axes of the coordinate system. We, thus, augment the states xf with the angular positions, using v = [pf. wf, uf] to denote the augmented state that captures the angular position well the linear position and velocity.
angular positions w = (0, d; vb) otherwise known as yaw,
3

---

Our method draws inspiration from Galilean invariance and inertial frames of reference that capture the relative locations and motions of all the objects in a system. We introduce N, local coordinate frames, one per object in the system. For the i-th object, a local coordinate frame is one in which both the linear and the angular position lie on the origin. By the adoption of the local coordinates with respect to object-centric frames, the behaviors between objects will not depend on the arbitrary positions of objects in the absolute Newtonian space, In other words, the local coordinates offer invariance to global translations and rotations and do not bias the learning algorithm. Our goal, thereby, is to compute the relative local coordinates of all objects j = l..... Nr, while iterating over the i-th reference object.
The transformation from global to local coordinate systems is in two steps. Per target node, we first translate the origin to match its linear position by translation transformation. Since velocities and angular positions are translation invariant, we only need to perform the translation to the linear positions. This gives us the relative positions rf. [p] P Then, we canonicalize the local coordinate frame to match the target object's orientation by a rotation transformation, described by the rotation matrix Q(w:). The coordinates of all other objects are analogously transformed given the i-th local coordinate frame. The rotations are performed independently and equivalently to the state components of the j-th object, namely the relative -due to the translation transformation performed first- linear positions, the angular positions and the velocities. A schematic overview of the proposed transformation in 2D setting is presented in down the transformed state as:
fig. I Using tensor operations, compactly write
R(w) = Q(w) e Q(w) e Q(w) Vil= R [r] wo u
(3)
(4)
where @ denotes the direct sum operator that concatenates two matrices along the diagonal resulting in block diagonal matrix and RęT R (w²) reduce notation clutter. The rotation matrix in
eq. (3) has three entries. Each ((w) independently transforms the relative linear positions, the angular positions and the velocities.
Local coordinate frames in 2D We start with the simpler case where the dynamical system resides in the 2D Euclidean space, for instance pedestrians in a traffic scene. In dimensions, the angular position is scalar value, namely the yaw angle 0. Thus, the rotation matrix for a target node o is:
(cos 0; -inn 0 Q(w:) = (().) = sin 0: cos 0:
(5)
Local coordinate frames in 3D When in the 3D Newtonian space the chain of transformations is the same; first, we translate the origin match each target node's linear position, and then rotate the local coordinate frames to match each object's orientation. The translation transformation is identical the 2D case. However, the rotation transformation is more involved. For the 3D case, we must decompose Q(w) into 3 chained elemental rotations, described by the matrices (#), (g) and Qur (1). Q (0) describes rotation around the z-axis by an angle 6, Q2 (Ø) describes a rotation around the y-axis by an angle é and Q. describes a rotation around the axi by angle th. We use the following convention that dictates the order of rotations, Q(w) = Q₂ Qu (Ø) (1b). Each elemental rotation matrix has similar structure to leq. (5) We provide the complete description of all rotation matrices in appendix A.2 After the computation of the rotation matrix, the states are transformed identically toleq. (4)
The local coordinate frames are invariant with respect global translations and rotations, either in 2 or 3 dimensions. We provide the detailed derivations of Q(w) for the 2D and 3D case in appendices A.I and A.22 respectively, and a detailed proof in infappendix A.3
3.2 Approximate angular positions
In practice, we do not always have perfect information about the object states, such as the angular positions. this case, we can approximate them using the angles of the velocity vector as a proxy. Specifically, in 2 dimensions, the angular position is a scalar value and is approximated using the azimuth angle of the polar representation of the velocity vector, 0 = tan (an-n(u,) -1,y/ (..)). In 3 dimensions, we transform velocities to spherical coordinates (ups ug Uz) and use these angles to rotate the local

---

coordinate frame and approximate 2 out of the 3 angles. The angle 0 is computed as above and cos - (u=/ /l/l))/ this case, we retain invariance for the coordinates for which we have perfect knowledge, while we will have an invariance leakage for the approximate ones. That said, experiments show there is little to no consequences and accurate predictions are still attained.
3.3 Local coordinate frame graph neural networks
Having canonicalized the object states in the interacting system, we obtain representations that are invariant to global translations and rotations. Following [9], formulate the core of the network as a variational autoencoder [25 38) with latent edge types that change dynamically over time. The network receives the canonicalized representations as input and operates solely on the local coordinate systems, We infer the graph structure over discrete latent graph and simultancously learn the dynamical system. Learning the graph structure roto-transe tr-tsaanlaaionn invariant task; we want to predict the same edge distribution for each pair of vertices regardless of the global rotation of translation. In contrast, trajectory forecasting is ooto-translation equivariant task; a global translation and rotation to the input trajectories should affect the output trajectories equivalently, Following [19] we maximize the evidence lower bound, L( 0) Ega(sxx) llog pa (x/z)] KL|g (2) (x, We provide the exact form of the loss components in appendix B.1
Encoder and prior Our encoder and prior closely follow 19, described in section 2.2 First, we compute the local coordinate frame representations Vis per pair j, é (including self-loops) and per timestep t according eq. (4) We then perform number of message passing steps and obtain feature vector per object pair. Omitting time indices for clarity, we have
h (1) (()) ([v vale))
(6)
h ,!!) (vas) + WOOI JEN())
Σn js
(2)
(8)
(1) (2) are MLPs and ge is a linear layer. We feed the embeddings h into 2 The functions 1e" 1
(1)
LSSTSs [2): one forward time that computes the prior and one backwards time for the encoder. The hidden state from the forward LSTM is used to compute the prior distribution, while the hidden states from both the forward and the backward LSTM are concatenated to compute the encoder distribution. The formulation is identical to [9] the exact details can be found in appendix B.1
During training, we sample interactions zi,, from qs (z/,,is (x) using Gumbel-Softmal 34 23]. We perform teacher-forcie forcing during the whole training, and task the model to predict the trajectories only for one step ahead. During inference, we sample interactions from the prior distribution.
Decoder As mentioned in section 2.2, the decoder pe (x|z) = =II² Pe (x*++Ix* [x 2 ( is tasked with predicting future trajectories given past and present trajectories well as the predicted relations. As proposed by [19 (27) we can have either a Markovian or a recurrent decoder, depending on the governing dynamics. In many settings, like colliding elementary particles in physics, the governing dynamics satisfy the Markov property, Px(x'+ +11 |x
1x = px +x*+1 xx", z²). In this case, the decoder is implemented with graph neural network similar [19]. In many real-world applications, however, the Markovian assumption does not hold. In that case, the graph neural network also features GRU unit that learns recurrent hidden state during the message passing.
We can use local coordinates frames with both types of decoders, as defined in 19 27 with the difference that the message passing is performed with the local coordinate frame representations fjo so that attain noto-translation invariance,
m Σ A ((vie.
(9)
mf = f(3)
g! (vit)) + [o(1)l
Σmf, m jEN())
(10)
5

---

The output of the model per time step comprises position and velocity predictions for the next time step. As common the literature [9) we predict the difference in position and velocity from the previous time step, which equals the velocity and acceleration respectively, and numerically integrate to make predictions. While computations up to the output layer are roto-translation invariant, the predictions must be roto-translation equivariant, so that global roto-translations to the inputs affect the outputs equivalently, To transform predictions back to the global coordinate frame and achieve roto-translation equivariance, we do an inverse rotation by R(w,) Q(w() + Q(w)), and then integrate numerically, i.e., x x - R(w() Ax;+1 (+1 We provide the definitions for both decoders in appendix B.1 and a detailed proof on equivariance to global ooto-translations in lappendix A.3)
3.4 Anisotropic filtering
One of the main reasons why filters in graph neural networks are isotropic is the inherent absence of an invariant coordinate frame. In a geometric graph dynamical system, object positions can serve this role. We, thus, use the local iio-trranslation invariant coordinate frames for anisotropic filtering, using weights that depend on the relative linear positions and angular positions of objects given the central i-th object. Similar to [45], our filter generating network, implemented by an MLP, is a matrix field that maps relative positions and orientation tuples to graph network filters, iee weight matrices, WF: RD x Slw) + RDam xDi,, The anisotropic filters replace the isotropic ones in updating the latent edge representations, weighing neighbors according to their positions, h/.
W> ([Q (w!) ry,.0 Q (w))) w|])
Vi Vile
3.5 Data normalization
While the graph neural networks are roto-translation invariant and equivariant in the intermediate and the output layers respectively, we must also make sure that the pre- and post-pne post-processing of data are appropriate. The common practices of min-mar max normalization or z-score normalization are unsuitable because they anisotropically scale and translate the input and output position and velocities. That is, these transformations change the directions of the velocity vectors non-equivalently. This is counter-intuitive since velocities are not treated as geometric entities but as generic additional dimensions to the features. For instance, as translation equals a vector subtraction changing the magnitude and the direction of vectors, translating the velocities removes any notion of speed from the input to the neural network. What is more, the scaling operations apply anisotropic transformations, affecting each axis differently.
We instead opt for a much simpler data normalization scheme that more geometrically oriented and suitable for troossaanlltion invariance with local coordinate frames. This scheme does not perform any translation operations, since local coordinate frames naturally tend center data around the origin; besides, they are invariant to a mere isotropic translation to the node positions. For the scaling operation, we opt for a simple isotropic transformation that shrinks relative positions and velocities equivalently across all axes. We scale the inputs, both positions and velocities, by the maximum speed (velocity norm) in the training set, Smax = max; ||u:ll ll. that is x' = S S x, where = dtiag($mux 1). During post-processing, we can convert our predictions to actual units, e.g. m and m/s. by applying the inverse transformation, x = Sx'. We term this operation speed normalization.
4 Related work
Learning dynamical systems & trajectory forecasting In the late years, and alongside NRI 27 and dNRI [19), many have studied learning dynamical systems [0]. Further, many works have focused on the problems of pedestrian motion prediction and traffic scene trajectory forecasting 21, 21 35 339. A number of works 39] uses distance-based heuristics to create the graph adjacency and estimate interactions. Kosaraju et al. (28) use self-attention to predict the influence of neighbouring nodes. Both approaches are different from our work, since we explicitly predict the latent graph structure and perform inference on it.
have risen in popularity over the past few years, demonstrating high effectiveness and parameter efficiency. Schütt et 444 use radial basis functions pair-wisc node distances to generate continuous filters and perform mes-
Equivariant deep learning Equivariant neural networks I|
53 54
46

---

sage passing using depth-wise separable convolutions. Fuchs et al. [17 introduce SE(3)- transformerss by incorporating spherical harmonics in a transformer network, resulting in 3D roto- translationn equivariant attention network. de Haan et al. K propose anisotropic gauge equivariant kemnlss for graph networks on meshes based on neighbouring vertex angles and parallel transport. Walters et al. [50] propose rotationally equivariant continuous convolutions for 2D trajectory prediction. Closer to our work is the work of Satorras et al. [41]. They propose a graph network that leverages the rotation equívariant relative position and tronsraanlttion invariant euclidean distance between node pairs in novel message passing scheme that updates node features as well as node coordinate embeddings. Different from our work, they do not capitalize on orientations and velocities of neighbouring objects. Our work leverages these impactful quantities expressed in local coordinate frames to make more reliable predictions.
Anisotropic filtering Graph neural networks 42
32 8
have been used extensively for modeling
dynamical systems [55 35 19
Several works 355 incorporate isotropic graph filters 226 20
as part of the graph network. However, these isotropic filters have global weight sharing scheme, L. they use single weight matrix for all neighbours, which amounts a linear transformation over the aggregated neighbour information. Velickovic et al. [47] use self-attention 2 and [36 use Gaussian Mixture Models (GMMs) based on relative neighbour positions.
Other works address this issue by proposing continuous filters on graphs and point clouds 24] 51 50 44 16]. Highly related to our work is the work of Simonovsky and Komodakis 455 They generalize convolution to arbitrary graphs and introduce dynamically generated filters based on edge attributes to perform continuous anisotropic convolutions on graph signals and point clouds. Differently, though, they do not operate under roto-translated local coordinate systems. This results in diminished parameter efficiency and weight sharing that they compensate for with data augmentation.
Experiments
We evaluate the proposed method, LoCS, on 2D and 3D geometric graph dynamical systems from the literature. In 2D, we evaluate on a synthetic physics simulation dataset proposed by dNRI [19] and on traffic trajectory forecasting 4. In 3D, we evaluate on an 3D-extended version of the charged
particles 27 and on motion capture dataset [IO]. We compare with NRI [27] dNRI 19 and the
very recent EGNN 41 For all methods we use publicly available code from 19 41 For EGNN,
we autoregressively feed the output as the input to the next timestep. The full implementations details are in apppeddx B.3 Our code, data, and models will be available online
Our architecture closely follows dNRI 26 19]
Unless otherwise specified, all common layers have the same structure, and we use the same number of latent edge types. Following (26 19 we report the mean squared eтror of positions and velocities over time. We compute errors in the original unnormalized data space for a fair comparison across different data normalization techniques and
scales, E(t) = ND > ||x[ - 12.. We also report the L2 norm errors for positions (displacement errors), Et(() N Σ m== Ip -filly- and velocities, (t) & Em=ullus Ilu', 16 separately to gain insights. L2 norm errors are in the same scale with the respective variables (position and velocity) and, thus, more interpretable than MSE. We always ran experiments using 5 different random initialization seeds and report the mean and the standard deviation. We plot each method with a different color, as well as a different marker for color-blind friendly visualizations. The x-coordinates of the markers are chosen for aesthetic purposes; they differ across settings and they meant convey extra information. Here report the main results and visualizations and provide much more extensive examples in the appendix.
5.1 Synthetic dataset
On the synthetic 2D physics simulation we use the same experimental settings as in 19 The dataset comprises scenes with three particles. Two of the particles move with constant, ran- domly initialized velocity and the third partiele is initialized with random velocity but pushed away when close to one of the others. Scenes last for 50 timesteps. For evaluation, we use the first 25 timesteps as input and the models are tasked with predicting the following
7

---

(Qurs)
INRI
Position Errors
Velocity Errors
Total Erros
0.4
0.000
0.020
0.3
0.015
1o B100
1g Brro
MSE
0.2
0.000
0.010
0.1
0.005
0.065
0.0
16
21
16
21
11
16
21
Step
Step
Step
Figure 2: Results on synthetic dataset
(a)
(b)
(c)
Figure 3: LoCS predictions on (a) synthetic dataset, (b) charged particles and (c) inD.
25 timesteps. We report the results in lig. and plot qualitative examples fig. 3al more in fig. 7 in lappnddx C.I. We also report the average Fl score for relation prediction in table In all visualizations, each color denotes different object. Table 1: Relation prediction F1 score on synthetic dataset
Semi-transparent trajectories indicate the groundtruth future trajectories. Present timesteps denoted by small black circles. Markers denote the timesteps, increasing in size as trajectories evolve through time. We observe that method can reliably model interactions and outperforms competing methods in predicting future trajectories.
Method
NRI
dNRI
LoCS
F1
26.5
60.8
88.9
5.2 Charged particles
In the charged particles datasets the particles interact with one another via electrostatic forces. We extend the dataset by 27 from 2 dimensions and, following a similar approach [7 41 we remove the virtual boxes that confine the particle trajectories. We generate 30,000 scenes for training, 5,000 for validation and 5,000 for testing. Each scene comprises trajectories of particles negative charge. The forces are either attractive repulsive according to physical laws. Following (27). training and validation scenes last for timesteps. For evaluation, test scenes last for 20 additional timesteps (50 for visualization). We plot the MSE in fīg. 4a and L2 errors in Figure 11 in aappendix D LoCS has consistently lower errors, in total as well as individually for positions and velocities. We, further, provide qualitative results for 50 future time steps in fig. 3b see more in fig. 8 in lppendix C.2
that carry either a positive
5.3 Traffic trajectory forecasting
The inD dataset a real-world 2D traffic trajectory forecasting dataset of pedestrians, vehicles, and cyclists, recorded at 4 different German traffic intersections. Traffic scenes contain a varying number of participants also changing over time. We follow the exact same setting as in dNRI. The dataset contains 36 recordings; we split them in 19/7/10 for training, validation and testing. We divide each scene into 50-step sequences. We use the first 5 timesteps as input and the model has predict the remaining 45 time steps. We compare with dNRI, EGNN, and a GRU [ baseline, but not with NRI since there are varying number of nodes. We plot the MSE fig. 4b and T.2 errors in Figure 12 in appendix D and as before, LoCS outperforms competing methods consistently.

---

LoCS (Ours)
dNRI
NRI
EGNN
GRU
Total Brrors
Total Errors
Total Emoor
0.8
60
MSE
0.4
MST
MSB
0.2
0.0
0
0.0
13
17
20
1
10
19
28
37
45
9
17
25
33
41
48
Step
(a)
(b)
(c)
Figure Total error curves in: (a) charged particles, (b) inD, (c) motion #35
LaCS LoCSS (Qus)
UNRI (Minemax sorm)
NRI
EGNN
GRU
LoCSS (No norm)
(Sporil norm)
(hottopicc)
LaCS (Tranchation only)
Toral Froors
Total
Total
DX030
3.5
D1135
0.5
0000
D1305
0.2
0.0
0000
11
16
10
37
Stop
S0ep
3tpp
(a)
(b)
(c)
(d)
Figure 5: Total error curves in ablation experiments: (a) on highly interactive charged particles, (b) on the impact of speed normalization, (c) on the impact of isotropic filters, (d) on the impact of rotation.
5.4 Motion capture
Last, we experiment with the CMU motion capture database [10] with 3D data, following the exact same setting as [9] and studying the motion of subject #35. We train the models using sequences of 50 timesteps as input and evaluate on sequences of 99 time steps. We plot the MSE in lig. 4c and
L2 errors in fig. 13 in inppeendix D and observe that LoCS is attaining consistently lower errors.
5.5 Ablation experiments
High-intensity interactions We assess LoCS in fig. 5a in highly interactive scenarios by creating subset the charged particles test set (819 scenes 16.38% of the original test set) in which simple constant velocity model performs poorly (L2 error >> 1.5). With more and stronger interactions the proposed local coordinate frames performs even better relatively competitors.
Speed normalization impact We assess the impact of speed normalization on the synthetic dataset in [ig. 5b We evaluate LoCS with and without speed normalization, as well dNRI with speed normalization instead of min-max normalization. Results are shown in fig. 5b) We observe that when the inputs are normalized using min-mas normalization, LoCS underperforms. Without any normalization, LoCS already performs better than other baselines. Using speed normalization improves the performance of LoCS even further. Finally, speed normalization not the main cause for the improvements for LoCS. If it were, it should also benefit dNRI. We conclude that speed normalization is important for local coordinate frames to make sure equivariance maintained after un-normalization LoCS attains the highest accuracies when combined with speed normalization, although works quite well even without it, while methods that are not equivariant like dNRI do not benefit from it.

---

(c)
(d)
translated-only local coordinate frames for object A in 3 different dynamical systems #1-#3. In dynamical system #3, A's roto-translated local coordinate frame
Figure 6: In a
b
Impact of anisotropic filtering We compare anisotropic and isotropic filtering inD in fig. 5c Be it with isotropic or anisotropic filters, the local coordinate frames outperform the competitors. That said, anisotropic filters give a clear advantage over isotropic ones.
Impact of rotation in spherical symmetries Even though particles do not have intrinsic orienta- tions, they do have the direction of their velocity. We postulate that toto-translated local coordinate frames allow for more efficient learning, as long as the coordinate frames are consistently invariant. To motivate this, consider the simple case of a 2D system shown in fig. 6a with three objects A, B, C with no intrinsic orientation. In fig. 6a B is far above left of A, while C is near below right of A. Rotating the system by л, see fig. 6b A still lies at the same origin, however, C is now in the top left and B is in the bottom right quadrant. Without canonicalizing with respect to rotations, the description of the two systems is very different, and subsequent neural networks will have learn to account for this underlying symmetry. On the other hand, different dynamical systems should yield different representations. For example, consider the dynamical systems shown in figs. 6b and 6cc While they only differ in A's velocity, their dynamics vary greatly: in fig. 6c, A and B are moving perpendicularly to one another and may crush due to attractive forces. In the canonicalized frame in fig. 6d the neighbour representations are indeed very different from fig. 6b
In the end, we care that our inputs are represented consistently (invariantly) if their relative differences (translations or rotations) are the same according to the system at hand, regardless of how we obtain the reference axis for the rotation (intrinsic angular position or another invariant quantity like the angles of acceleration vectors). Thus, applying both translation and rotation transformation helps even for objects with no intrinsic viewpoint and orientation, like point masses. We confirm the hypothesis in an ablation experiment with charged particles, where using only the translation transformation to form local coordinate frames leads to decreased accuracy, see fig.. 5d
Conclusion
In this work introduced LoCS, method that introduces canonicalized roto-translated local coordinate frames for all objects in interacting dynamical systems formalized in geometric graphs. These coordinate frames grant us global invariance to iro-trranslation and naturally allow for anisotropic continuous filtering. We demonstrate the effectiveness of our method in a range of 2D and 3D settings, outperforming recent state-of-the-ard works.
Limitations Many dynamical systems in nature are not formalized as geometric graphs (e.g. social networks), or are not intrinsically described by angular positions (e.g. elementary particles), which case the proposed method is not applicable. Furthermore, although we approximate angular positions using velocities, our method guarantees full invariance/equivatiance to global cro-trranllatins only in the 2D case, while in 3 dimensions we have an equivariance leakage. We have identified two modes where local coordinate frames do not exhibit the same large improvements. First, when the data setting relies, in fact, on a global coordinate frame, like different positions in basketball court [56). Second, when the direction of the local coordinate frames cannot be well-defined, like objects with 0 velocity (although simply and arbitrarily setting the orientation to 0 works just as well). Last, a theoretical limitation that Fuler angles, which LoCS also uses, are prone to singularities and gimbal lock, although in practice observed no problem.
10

---

Acknowledgments
The project is funded by the NWO LIFT grant 'FLORA'.
References
[1] Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Li, F., and Savarese, Social LSTM: Human Trajectory Prediction in Crowded Spaces. In 2016 IEEE Conference on Computer Vision and Pattrn Recognition (CVPR), 2016.
[2] Bahdanau, D., Cho, K.. and Bengio, Y. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference on Learning Representations (ICLR), 2015.
[3] Battaglia, P. W., Pascanu, R., Lai, M., Rezende, D. J., and Kavukcuoglu, K. Interaction Networks for Learning about Ohjects, Relations and Physics. In Advances in Neural Information Processing Systems 29 (NIPS), 2016.
[4] Bock, J., Krajewski, R., Moers, T., Runde, S., Vater, L.,, and Eckstein, L. The inD dataset: A drone dataset of naturalistic road user trajectories at german intersections. In 2020 IEEE Intelligent Vehicles Symposium (IV), 2020.
[5] Calafiura, P., Farrell. S., Gray, H., Vlimant, J.RR., Innocente. V., Salzburger. A., Amrouche, S.. Golling. T., Kiehn, M., Estrade, V., et al. TrackML: A High Energy Physics Particle Tracking Challenge. In 2018 IEEE 14th International Conference on e-Science (e-Science). 2018.
[6] Chang, M.. Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang. D., Cam, P., Lucey, S.. Ramanan, D., and Hays, J. Argoverse: 3D Tracking and Forecasting With Rich Maps. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Cho, K., van Merriënboer, B., Gulechre, C., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Learning Phrase Representations using RNN Encoder Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
[8] Chung, J., Kastner, K., Dinh, L., Goel, K., Courville, A. C., and Bengio, Y. A Recurrent Latent Variable Model for Sequential Data. In Advances in Neural Information Processing Systems 28 (NIPS), 2015.
[9] Clevert, D., Unterthiner, T., and Hochreiter, S. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In 4th International Conference Learning Representations (ICLR), 2016.
[10] CMU. Carnegie-Mellon Motion Capture Database, 2003. URL http:/tttt/pmp......o/ www.:uaccc.cccccmmmmmmmmoo cmm.cmm..... edu
[11] Cohen, and Welling, Group Equivariant Convolutional Networks. Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016.
[12] Cohen, and Welling, M. Steerable CNNs. International Conference on Learning Representations (ICLR), 2017.
[13] de Haan, P., Weiler, M., Cohen, T., and Max, W. Gauge equivariant mesh CNNs: Anisotropic convolutions on geometric graphs. In 9th International Conference on Learning Representations (ICLR), 2021.
[14] Deng, H., Birdal, T., and Ilie, S. Ppf-foldnct: Unsupervised learning of rotation invariant 3d local descriptors. Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[15] Dwivedi, V. P., Joshi, C. K., Laurent, T., Bengio, Y., and Bresson, X. Benchmarking graph neural networks. arXiv preprint 2003.20003,,,98,, 2020.
[16] Fey, M., Lenssen, E., Weichert, F. and Müller, H. SplineCNN: Fast Geometric Deep Learning With Continuous B-Spline Kernels. In 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[17] Fuchs, F., Worrall. E., Fischer, V., and Welling, M. SE(3) Transforrnss: Roto-Transhe Translation Equivariant Attention Networks. In Advances in Neural Information Processing Systems 33 (NeurIPS), 2020.
[18] Gilmer, J.. Schoenholz, S. S., Riley, F., Vinyals, O., and Dahl, G. E. Neural Message Passing for Quantum Chemistry. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
11

---

[19] Graber, C. and Schwing, A. G. Dynamic Neural Relational Inference. In 2020 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[20] Hamilton, W. L.. Ying, Z.. and Leskovec, J. Inductive Representation Learning on Large Graphs. In Advances in Neural Information Processing Systems 30 (NIPS), 2017.
[21] Helbing, D. and Molnar, P. Social foree model for pedestrian dynamics. Physical review E, 51(5), 1995.
[22] Hochreiter, S. and Schmidhuber, J. Long short-term memory. Neural computation, 9(),, 1997.
[23] Jang, E., Gu, S., and Poole, B. Categorical Reparameterization with Gumbel-Softmax. In Sth International Conference on Learning Representations (ICLR), 2017.
[24] Jia, X., Brabandere, B. D., Tuytelaars. T., and Gool, L. V. Dynamic Filter Networks. In Advances in Neural Information Processing Systems 29 (NIPS), 2016.
[25] Kingma, D. and Welling, M. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations (ICIR), 2014.
[26] Kipf, T. N. and Welling, M. Semi-Supervised Classification with Graph Convolutional Networks. In Sth International Conference on Learning Representations (ICLR), 2017. 2017.
[27] Kipf, T. N., Fetaya, E.,, Wang, K., Welling, M., and Zemel, R. S. Neural Relational Inference for Interacting Systems. In Proceedingy of the 35th International Conference on Machine Learning (ICML), 2018.
[28] Kosaraju. V. Sadeghian, A., Martín-
R., Reid, 1. D., Rezatofighi. H., and Savarese, S. BiCiAI-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAM and Graph Attention Networks. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019.
[29] Kothari, P., Kreiss, S., and Alahi, A. Human trajectory forecasting in crowds: A deep learning perspective. arXiv preprint arXiv:2007.0366 033699 2020.
[30] Krishnan, R. G., Shalit, U., and Sontag, D. Deep kalman filters. arXiv preprint arXiv: 2015.
[31] Lerner, A., Chrysanthou, Y., and Lischinski, D. Crowds by example. In Computer graphics forum, volume 26. Wiley Online Library, 2007.
[32] Li, Y., Tarlow, D., Brockschmidt, M., and Zemel, R. S. Gated Graph Sequence Neural Networks. In 4th International Conference on Learning Representations (ICLR), 2016.
[33] Ma, Y., Zhu, X., Zhang, S., Yang, R., Wang, W., and Manocha, D. TrafficPredict: Trajectory Prediction for Hcterogencous Traffic-Agents. In The Thirty-Thing Conference on Artificial Intelligence (AAAI), 2019.
[34] Maddison, C. J., Mnih, A., and Teh, W. The Concrete Distribution: Continuous Relaxation of Discrete Random Variables. In 5th International Conference on Learning Representations (ICLR), 2017.
[35] Mohamed, A., Qian, K., Elhoseiny, M., and Claudel, G. Docial-STGCNN: A Social Temporal Graph Convolutional Neural Network for Human Trajectory Prediction. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[36] Monti, F., Boscaini, D., Masci, J., Rodolà, E., Svoboda, J., and Bronstein, M. M. Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs. In 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[37] Pellegrini, S., Ess, A., Schindler, K., and Gool, L. V. You 'l never walk alone: Modeling social behavior for multi-target tracking. In IEEE 12th International Conference on Computer Vision (ICCV), 2009.
[38] Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31st International Conference on Machine Learning (ICMI.), 2014.
[39] Salzmann, T., Ivanovic, B., Chakravarty, P., and Pavone, M. Trajectron++: Dynamically-feasee fcasiblee trajectory forecasting with heterogeneous data. In European Conference on Computer Vision (ÉCCV), 2020.
[40] Sanchez- Gonzzllz, A., Godwin, J., Pfaff, T., Ying, R., Leskovec, J., and Battaglia, P. W. Learning to Simulate Complex Physics with Graph Networks. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
[41] Satorras, V. G., Hoogeboom. E., and Welling, M. E(n) Equivariant Graph Neural Networks, In Proceedings of the 38th International Conference Machine Learning (ICML), 2021.
12

---

[42] Scarselli, F. Gori, M., Tsoi, A. C., Hagenbuchner, M., and Monfardini, G. The graph neural network model. IEEE transactions neural networks, 20(1), 2008.
[43] Schäller, C., Aravantinos, V., Lay, F., and Knoll, A. What the constant velocity model can teach us about pedestrian motion prediction. IEEE Robotics and Automation Letters, 5(2), 2020.
[44] Schütt, K., Kindermans, P., Felix, H. E. S., Chmiela, S., Tkatchenko, A., and Müller, K. SchNet: A continuous-fiter convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems 30 (NIPS), 2017.
[45] Simonovsky, M. and Komodakis, N. Dynamic Slge--ooniitioee Filters Convolutional Neural Networks on Graphs. In 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[46] Thomas, N., Smidt, T., Kearnes, S., Yang, L.,, Li, L.., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation-and and translation equivariant neural networks for 3d point clouds, arXiv preprint 1022v:180..082) 2018.
rrassation-equivaaiaat
[47] Velickovic, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., and Bengio, Graph Attention Networks. In 6th International Conference on Learning Representations (ICLR), 2018.
[48] Vemula. A., Muelling. K., and Oh, Social attention: Modeling attention human crowds. In 2018 IEEE international Conference on Robotics and Automation (ICRA). 2018.
[49] Vignac, C., Loukas, A., and Frossard, P. Building powerful and equivariant graph neural networks with message-passing. arXiv preprint arXiv:2006. 151007 2020.
[50] Walters, R., Li, J., and Yu, Trajectory Prediction using Equivariant Continuous Convolution. International Conference on Learning Representations (ICLR), 2021.
[51] Wang, S., Suo, S., Ma, W., Pokrovsky, A., and Urtasun, R. Deep Parametric Continuous Convolutional Neural Networks. In 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[52] Wang, Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M., and Solomon, J. M. Dynamic graph enn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5), 2019.
[53] Weiler, and Cesa, General Equivvaraa Stecrable CNNs. Advances Neural Information Processing Systems 32 (NeurIPS), 2019.
[54] Worrall, D. E., Garbin, S. J., Turmukhambetov, D., and Brostow, G. Harmonic Networks: Deep Translation and Rotation Equivariance. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[55] Yan, S., Xiong, Y., and Lin, D. Spatial Temporal Graph Convolutional Networks for Skeleton-Baud -assd Action Recognition. In Proceedings of the Thirly-Second Conference Artificial Intelligence (AAAI), 2018.
[56] Yue, Y., Lucey, P., Carr, P., Bialkowski, A., and Matthews, I. Learning fine-grained spatial models for dynamic sports play prediction. In 2014 IEEE international conference data mining, 2014.
13


Reviews:
[{"review_id": "z2GImQ5hC1J", "reviewer": "Reviewer_QAhm", "summary": "This paper proposes a to use local coordinate system per object in a spatio-temporal graph (with a GNN) of multiple object in order to learn and predict the dynamics of the objects interaction. This enables adding invariance, equivariance, and use anisotropic filters between GNN layers to improve learning performance.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n+ The addition seems simple and useful for a wide range of problems, at least ones that use graph node features as euclidean states of position and velocity.\n+ Experiments demonstration notable improvement over baselines.\n+ Paper is mostly well motivated (exploit invariance and symmetry for dynamic tasks, prior work is focused on static datasets) and clearly explained.\n\nWeaknesses\n- The overall contribution falls a bit on the weaker side. Besides the insight to swap to a local coordinate system, Section 3.1 is mostly simple linear algebra and rigid body mechanics, commonly employed in graphics, vision, and robotics. Related to that, since rotation matrices are already used why not consider working with SE2 / SE3 Lie groups and make the math less clunky?\n\n- Section 3.2, 'In practice, we do not always have perfect information about the object states,' what application is this with respect to? Is the idea to find the next state from the current velocity i.e. the temporal model assumes a linear dynamical system?\n\n- An example in the introduction would help ground where symmetry arises and why it leads to subpar leaning. '... exhibit symmetries that if left to their own devices lead models to subpar learning.'\n\n- More insight with examples would be helpful in explaining how the paper converged to idea of speed normalization?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations were adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "lqCHQY0EZ1i", "reviewer": "Reviewer_fJBY", "summary": "This paper presents an approach to  enable roto-translation invariant edge prediction and roto-translation equivariant trajectory forecasting. Rotation and translation invariants are achieved by transforming global coordinate into local coordinate frames of each object in the graph neural network. Experiments are done on forecasting trajectories of charged particles, traffics and human motions.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pro:\n\n1. The idea of representing neighbors coordinate in its own local coordinate frame is simple and straightforward. And the improvement seems substantial in the experiments presented.\n\n2. Detailed derivation provided to support the soundness of the method.\n\nQuestions\n\n1. It is not clear to me how you get the angular position approximation when the velocity is 0.\n\n2. I am also not sure how you can represent full 3D orientation using euler angle only since it will encounter singularity at some point.\n\n3. Figure 3 doesn't really tell us anything without the ground truth. Even though more data are provided in the appendix, figure 3 seems redundant. \n\n4. I don't follow why we are doing a speed normalization ablation on dNRI, shouldn't it perform on LoCS? i.e, compare LoCS with or without speed normalization.\n\n5. What does GRU in Figure 4b, Figure 5c refer to?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors address the limitation.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "jCB_rckAyTb", "reviewer": "Reviewer_Yjm7", "summary": "This paper proposes the use of local coordinate frames for dynamics modelling in graph neural networks. 6dof pose, with angles estimated using translational velocity, is used to determine object pose, and graphs constructed in coordinate frames of these objects. Standard GNN dynamics models can then be trained, with the addition of an anisotrophic filter. Results on a range of experimental settings (synthetic, charged particles, mocap) show that dynamics modelling in local coordinate frames is highly effective and results in improved performance when compared with the arbitrary global frames currently in use.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nThe idea of dynamics modelling in local coordinate frames is not new, but this is the first work I have seen to do so in a graph neural network.\n\nQuality:\n\nThe paper is clear and well structured, and experiments are concise but thorough, clearly highlighting the value of the proposed approach. Ablations show that coordinate frame changes bring about the greatest improvement in performance. This was the best paper in my set.\n\nClarity:\n\nThe paper is clearly laid out, but I found it extremely formally written, and in many cases simple ideas are described using jargon and terminology that \"...obfuscates and impresses rather than clarifies.\"  to quote troubling trends in machine learning scholarship, Lipton 18. This is an extremely simple, but clearly effective idea, but I found that the paper hides this under layers of unnecessary complexity.\n\nSignificance:\n\nI believe the performance improvements obtained by using local coordinate frames are significant enough to warrant publication, and that there is enough in this paper for interested parties to build on.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper briefly discussed limitations, although I would have liked more detail on failure modes, downsides - when will it fail, is it vulnerable to gimbal lock, is the pose estimation affected by symmetry's, are there better orientation representations, etc.\n\nThere may be potential societal impacts for tracking applications which could adopt this work, but I believe this work is still in such an early research stage that this is unlikely to be the case. ", "review_text": "", "extra_scores": {"ethical_concerns": "I have no ethical concerns with this work.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "fLg3DEWBtgR", "reviewer": "Reviewer_49Zp", "summary": "This paper investigates the problem of simultaneous learning of interactions and dynamics of dynamical systems of multiple interacting objects. The proposed method is built upon dynamics NRI and incorporate a design Roto-translated invariance. The key of the Roto-translated invariance design is a local coordinate frame design - one for each object. The feature vectors that are fed into GNN are based on local coordinate frame representations instead of global coordinate frame representations. The results shows a clear advantage over previous methods. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper leverages symmetries and built it into the computation graph to improve the learning of interactions and dynamics. The idea is straight forward, but the writing of the paper need to be improved to make it easier for readers to appreciate it. For example, Figure 1 is very helpful for people to understand the high-level idea, but it is too far away from the presentation of the idea (Section 3.1) and it is not referred to in the text. The authors refer to Figure 6 in Section 5.2 and Figure 7 in Section 5.3, but those figures are not in the main text. It seems they are not referring to Figure 6 and 7 in the supplementary materials either. \n\nI have one key question: Your augmented state only consists of linear position, angular position and linear velocity. You are not including angular velocity. I guess this is because you assume they are point mass and cannot rotate. In your examples, the particles indeed cannot rotate. If this is the case, I wonder why including the angular position and the rotation transformation is necessary. In other words, what if you only have the translation transformation to form each local frame? In your framework, this reduces to the case that you don't have angular position data and you approximate the angular positions as the global x direction instead of the direction of velocity vector. I think it would be good to add it as an ablation study so that we know the importance of rotation translation. Intuitively I can't think of any reason rotation transformation would help if you only have particles instead of 3D rigid bodies. \n\nOne nitpik: Figure 1 is a little misleading since the visual orientation of the cartoons are fixed across figures. For example, the polygon is visually at the same orientation even though the arrow indicates the orientations should be different. It's easier to change them all to circles but I guess the authors use different shapes to be color-blind friendly. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have addressed the limitations. Societal impact doesn't apply in this case. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: As all reviewers agree, the paper discusses an interesting approach to the problem of learning complex dynamical systems with notable experimental supports. Although some reviewers have concerns on the technical part, the authors' responses have resolved those. The paper still have some weak points, especially in the presentation, but I think I can expect the authors modify the paper in the camera-ready by reflecting the discussion. Based on these, I recommend acceptance (poster) for this paper.

Decision: Accept (Poster)