Neural Field Discovery Disentangles Equivariance in Interacting Dynamical Systems
Authors: Miltiadis Kofinas
Erik J Bekkers
Naveen Shankar Nagaraja
Efstratios Gavves
Venue: Submitted to ICLR 2023 (2023)
Keywords: Interacting Dynamical systems
Graph Neural Networks
Neural Fields
Equivariance

Abstract: Systems of interacting objects often evolve under the influence of underlying field effects that govern their dynamics, \emph{e.g.} electromagnetic fields in physics, or map topologies and traffic rules in traffic scenes. While the interactions between objects depend on local information, the underlying fields depend on global states. Pedestrians and vehicles in traffic scenes, for example, follow different traffic rules and social norms depending on their absolute geolocation. The entanglement of global and local effects makes recently popularized equivariant networks inapplicable, since they fail to capture global information. To address this, in this work, we propose to \emph{disentangle} local object interactions --which are equivariant to global roto-translations and depend on relative positions and orientations-- from external global field effects --which depend on absolute positions and orientations. We theorize the presence of latent fields, which we aim to discover \emph{without} directly observing them, but infer them instead from the dynamics alone. We propose neural fields to learn the latent fields, and model the interactions with equivariant graph networks operating in local coordinate frames. We combine the two components in a graph network that transforms field effects in local frames and operates solely there. Our experiments show that we can accurately discover the underlying fields in charged particles settings, traffic scenes, and gravitational n-body problems, and effectively use them to learn the system and forecast future trajectories.

Under review as a conference paper at ICLR 2023
NEURAL FIELD DISCOVERY DISENTANGLES EQUIV- ARIANCE IN INTERACTING DYNAMICAL SYSTEMS
Anonymous authors Paper under double-blind review
ABSTRACT
Systems of interacting objects often evolve under the influence of underlying field effects that govern their dynamics, egg. electromagnetic fields in physics, or map topologies and traffic rules in traffic scenes. While the interactions between objects depend on local information, the underlying fields depend on global states. Pedes- trians and vehicles in traffic scenes, example, follow different traffic rules and social norms depending their absolute geolocation. The entanglement of global and local effects makes recently popularized equivariant networks inapplicable, since they fail to capture global information. To address this, in this work, we propose disentangle local object interactions -which are equivariant to global roto-translations and depend on relative positions and orientations- from external global field effects -which depend on absolute positions and orientations. We theorize the presence of latent fields, which aim to discover without directly observing them, but infer them instead from the dynamics alone. We propose neural fields to learn the latent fields, and model the interactions with equivariant graph networks operating in local coordinate frames. We combine the two components in a graph network that transforms field effects local frames and operates solely there. Our experiments show that we can accurately discover the underlying fields in charged particles settings, traffic scenes, and gravitational n-body problems, and effectively use them to learn the system and forecast future trajectories.
INTRODUCTION
Systems of interacting objects are omnipresent in nature, with examples ranging from the sub- atomic to the astronomical scale -including col- liding particles and n-body systems of celestial objects- well human-centric settings like traffic scenes, governed by social dynamics. The majority of these systems does not evolve in vacuum, they instead evolve under the influences of underlying fields. For example, electromag- netic fields may govern the dynamics of charged particles. In traffic scenes, the road network and traffic rules govern the actions of traffic scene participants. N-body systems might swirl around supermassive black holes that create gravitational fields. Earlier work on learning interacting systems proposed graph networks (Kipf et al., 2018
Ippal Trajestoriss
Dieowerool Fiet:
Figure I: N-body system simulation with underly- ing gravitational field. We uncover fields that un- derlie interacting systems using trajectories only.
Battaglia et al.) 2016 Sanchez- Gonzalez al. 2020). while sfate-of-the- -hetthr methods for interacting systems propose equivarianl graph networks (Walters et al., 2021 Satorras et al. 2021 Kofinas et al. 202T Brandstetter et al.] (202) to model dynamics while respecting their underlying symmetries. These networks exhibit increased robustness and performance, while maintaining parameter efficiency due to weight sharing. They are, however, not compatible with underlying field effects, since they can only capture local states, such as relative positions, while fields depend absolute states (positions or orientations). In other words, global fields violate the strict equivariance hypothesis. We are, thus, in need of augmented notion of equivariance that encapsulates both local and global effects.
In many real- world settings, strict SE(3) equivariance, -equivariance to the special Euclidean group of translations and rotations- does not hold. A function that predicts trajectories in interacting systems

---

Under review as a conference paper at ICLR 2023
is equivariant translations if f(x) + f(x + +) 7) for translation vector and equivariant to rotations if Qx(x) = fx) for a rotation matrix Q. That is, even if the symmetries exist in a particular setting, they only manifest themselves in local interactions, yet they are entangled with global effects that stem from absolute states. For instance, pedestrians and vehicles in traffic scenes operate within egocentric perspectives; individual local coordinate frames allow for differentiating objects based on relative states. Objects, however, behave differently depending on their absolute geolocation, e.g. people in different countries cities follow different traffic rules and social norms, and exhibit different driving habits. In such cases, global transformations are not even properly defined; would be illogical to perform translation of the Earth, since this would coincide with a different location on the globe.
rotation within the global coordinate frame
Even within local region of space, though, certain effects may only depend on global object states. N-body systems from physics, for example, exhibit roto-translational symmetries, since gravitational forces only depend on relative positions. Dynamies, however, may be influenced by external gravitational fields, which are either unknown not subject to transformations. Thus, strict equivariance violated, since equivariant object interactions are entangled with global field effects.
In this work, we disentangle the interactions between objects -which are equivariant to global roto- translations and depend on relative positions and orientations- from external field effects -which propose neural fields to model the
depend absolute positions and orientations. Furthermore, underlying field effects. Neural fields depend on absolute positions, and potentially orientations, and predict latent force fields. We model discrete object interactions with equivariant local coordinate frame graph networks, and continuous field effects with neural fields.
We make the following contributions. First, introduce the notion of entangled equivariance that intertwines global and local effects, and propose a novel architecture that disentangles equivariant local object interactions from global field effects. Second, we introduce neural fields to discover global latent fields in interacting dynamical systems, and infer them by observing the dynamics alone. Third, we propose an approximately equivariant graph network that extends local coordinate frame graph networks by introducing an auxiliary origin node, resulting in mixture of global and local coordinate frames. Finally, we conduct experiments on number of field settings, and observe that explicitly modelling fields is mandatory for effective future forecasting, while their unsupervised discovery opens a window for model explainability.
We term our method Aether, inspired by the postulated medium that permeates all throughout space and allows for the propagation of light.
2
BACKGROUND
In this section, we introduce background knowledge interacting systems, local coordinate frame graph networks, and neural fields, which will serve as a foundation for our method.
2.1 INTERACTING DYNAMICAL SYSTEMS
An interacting dynamical system comprises trajectories of N objects, recorded for T timesteps. The snapshot of the i-th object at timestep describes the state x = [p], uf], i E , N). (1....,7) T}. where denotes the position and u denotes the velocity, using [ to denote vector concatenation along the feature dimension. Interacting dynamical systems can be naturally formalized as temporl geometric graphs (Battaglia et al. 2016 Kipf et al. 2018 Graber & Schwing
2020), G (0) with graph snapshots g' = (V*s Et) different time steps. The set of graph nodes V [ef,- J describes the objects in the system; of corresponds to x/. The set of edges
g C (tt vf) (u). E yt x V* describes pair- wise object interactions; (1f, off) corresponds
to an interaction from node to node i. Finally, we denote the graph neighbors of node is with (i)..
2.2 LOCAL COORDINATE FRAME GRAPH NETWORKS
Local coordinate frame graph networks have been popularized in recent years (Kofinas al. 2021
Luo et al. 2022) as a method to achieve SE(3) equivariance, due to their low computational overhead and high performance. Kofinas al. (202T) proposed LoCS and introduced local coordinate frames
2

---

Under review as a conference paper at ICLR 2023
for all node-objects at all timesteps. They define augmented node states v = [p wt ug]. where w denotes the angular position of node at timestep t. Each local coordinate frame is translated to match the target object's position and rotated to match its orientation. Considering the representation of node j in the local coordinate frame of node i, they first compute the relative positions rj, = P P and then they rotate the state using the matrix representation of the angular position Q(w)):
R(w!) [F)... w/. u²]
(1)
where R(w() Q(w() (w)) @ Q(w!). and @ denotes direct sum. LoCS then proposes graph neural network (Scarselli et al. 2008 Li et al., 2016) Gilmer et al. 2017) that uses local states:
h1 fe ([vJen virl).
Ax Ax fo 9o (vis)) + (W(1)
Enj
(3)
SEN()
where Se, and Oo MLPs. The output of this graph network comprises differences in positions and velocities from the previous time step, in the local frame of each object. Since these outputs are invariant, LoCS performs an inverse transformation convert them back the global coordinate frame and achieve equivariance, x =x + +(w,)) Axtt! where R(w() = Q(w)) @ Q(w)).
2.3 NEURAL FIELDS
Finally, we make brief introduction neural ficlds. Neural fields, coordinate-based MLPs, a class of neural networks that parameterize fields using neural networks (Xie et al., 2022). They take input spatial temporal coordinates and predict prior behaviors and generalize to new fields via conditioning on a latent variable z that encodes the properties of specific field. |erez al. (2018) proposed Feature-wise Linear Modulation (FiLM), a conditioning mechanism that modulates a signal. It comprises two sub-networks a, B that perform multiplicative and additive modulation the input signal, and can be described by FiLM z) = a(z) 3 h + B(z), where is the conditioning latent variable, h is the signal to be modulated, and a, are MLPs that scale the signal, and add bias term, respectively.
quantity. Neural fields learn
METHOD
In this section, we present our method, termed Aether. First, we describe the notion of entangled cquivariance. and introduce our architecture that disentangles global field effects from local object interactions. Then, continue with the description of the neural field that infers latent fields by observing the dynamics alone. Finally, we formulate approximately equivariant global-local coordinate frame graph networks. We note that throughout this work, focus on 2-dimensional settings, and fields that are unaffected by the observable objects and their interactions thereof.
3.1 AETHER
Interacting dynamical systems rarely evolve in a vacuum, rather they evolve under the influence of external field effects. While object interactions depend on local infor- mation, the underlying fields depend on global states. On the one hand, locality object interactions stems from the fact that dynamics obey a number of symmetries. By ex- tension, object interactions are equivariant to a particular group of transformations. On the other hand, field effects are non-local; they depend on absolute object states, Thus, strict equivariance is violated, since equivariant object in- teractions are entangled with global field effects. We refer to this phenomenon as entangled equivariance.
Fun East
Two objects in gravitational field. We only observe the total force exerted at each particle, i.e. the sum of equivariant particle forces and global field effects.
Figure 2:
As an example, in fig. 2 we observe a system of 2 objects that evolve under the infiuence of gravitational field. The
3

---

Under review as a conference paper at ICLR 2023
arrows positioned the objects represent the forces exerted on them. One constituent of the net force is caused by the object interactions, and is thus equivariant, while the other can be attributed to the gravitational pull. However, can only observe the net force exerted at each particle, iee the sum of equivariant partiele forces and non-equivariant force field effects. Hence, in this system, we say that equivariance is entangled.
Neural Field F(p,w) w x)
Groundtruth Trajectories
Predicted Field
Graph Aggregation
FMN
Gruph Network
Input Trajectories
Predicted Trajectories
Figure The pipeline of our method, Aether.
We now propose our architecture that disentangles local object interactions from global field effects. We model the object interactions with local coordinate frame graph networks (Kofinas et al. 2021). and field effects with neural fields. During training, and given multitude of input systems, neural fields will, in principle, be able to isolate global from local effects, since only global effects are recurring phenomena. We hypothesize that field effects can be attributed force fields, and therefore. our neural fields learn to discover latent force fields. The pipeline of our method is shown in fīg. 3 Our inputs comprise states x for trajectories N objects for T timesteps. Since neural fields model global field effects, they depend on absolute states. Thus, we feed positions P and potentially orientations w/. of the trajectories as input to neural field that predicts latent forces f = f(p w!).
The predicted field forces can be now considered part of the node states, and further, they can be treated similarly to other state variables like velocities; as vectors, forces are translation invariant treat the problem setup if were once
and rotation equivariant. Thus, moving onward, again back in the strict equivariance regime. We append the predicted forces f for each node-object i and each timestep to the node states, and transform them corresponding local coordinate frames, similarly toleq. (1) Namely, the force exerted on node j. expressed in the local coordinate frame of node computed as:
f/:
Q (w.
(4)
We feed the new local node states to local coordinate frame graph network as follows:
h1
Vile
Vale
f
(5)
([viee
A Ax
1/1
So
9v
fijs
[V(1))
Σbi jEN(()
(6)
x R(w!)
Ax+11
where R(w() Q(w)) e Q(w)). The equations above are similar to eqs. (2) and 33) with the
addition of the highlighted parts that denote the predicted forces expressed at local coordinate frames. In practice, closely follow (Kipf et al. 2018 Graber & Schwing 2020 Kotinas et al. 2021 and
formulate our model as a variational autoencoder (Kingma & Welling 20 2:14:00112200 Rezende et al. 2014
with latent edge types. The exact details are presented in appenddxx A.2.2
4

---

Under review as a conference paper at ICLR 2023
3.2 FIELD DISCOVERY
In many settings, fields might not be directly observable for us to probe them at will and use them for supervision. For example, astronomical observations of solar systems and galaxies might not include black holes, yet we can observe their effects. Moreover, fields are often not even measurable or quantifiable, or they are not explicitly defined. For instance, social "fields" " that guide traffic, cannot be measured or defined explicitly, but we can safely assume they exist. Motivated by these observations, we design an architecture that performs unsupervised field discovery, while solving the surrogate supervised task of trajectory forecasting.
In this work, we aim to discover two different types of fields, which we term "static" and "dynamic" fields. Static fields refer to settings in which have single field shared throughout the whole dataset. On the other hand, dynamic fields refer to settings in which we have a different field for each input system, and fields also differ between train, validation, and test sets.
We now describe neural fields, used in this work, to model the underlying field effects. Neural fields depend on absolute positions, and potentially orientations, and predict latent force fields. When dealing with static fields, we use an unconditional neural field, i.e. a neural field that is a function only of the input resitions-orientationn contrast, for dynamic fields, we use a conditional neural field, i.e. neural field that also depends on a latent vector z E TR. that represents the underlying field. The latent will be inferred from the input trajectories and can be thought of as representing unusual non-equivariant dynamics. We use to explicitly condition the neural field, and thus, its general form is f IR2 x x R. IR?.
as the field values are common across data samples. In
We start with the description of unconditional neural fields used in static field settings, since conditional neural fields share the same backbone. First, we encode the input positions using Gaussian random Fourier features (Gaussian RFF) (Tancik et al. (2020). They are defined as:
Static fields
<p)) = [cos (2ro)) sin(2rBp)]]
(8)
where p E RDa are the input coordinates, and B E RD.DDa is matrix with entries sampled from a Gaussian distribution, B& ~ _N( The variance o² can be chosen per task with a hyperparameter sweep.
We use unit vector representation for the orientations 0, 0 [cos 0,so0, 0] Following Kofinas et al.] (2021), we use the angles of the velocity vectors as a proxy for the orientations. Then, we use a linear layer encode the orientation vectors, 5(0) W_. e. We finally concatenate the encoded positions and orientations in single vector that is being fed as input to the neural field.
After encoding the input coordinates, we use a 3-layer MI. with SiLU (Ramachandran et al. 2018 activations in-between, that outputs latent force field, f(p, 0) = (IPP([(()),(0))
Dynamic fields The neural fields used to model the dynamic fields are conditioned on latent vector representation z E R. that describes prior knowledge about the underlying field, and are defined as f(p, 0 z). In our case, the latent representation should "summarize" the input graph such that it isolates only global effects from the field. To that end, we employ simple global spatio-temporal attention mechanism, similar to Li et al. (2016), that aggregates the input system in latent vector representation. First, we define object embeddings W, a matrix used to linearly transform the inputs, and GRU is the Gated Recurrent Unit (Cho) et al. all]||2011 2014).. We also define temporal embeddings t = PE((), where PE are positional encodings /Vaswani et al. 2017). The aggregation is then defined as follows:
= GRU (W ( where
softmax (s)( So (sf), with sf =
=
ox +
z =
1.1
where fa R R, fo R. R are 2-layer MLPs with SiLU activations Ramachandran et al. (018)) in between.
After having obtained latent vector representation that summarizes the input system, condition the neural field using FiLM (Perez et al. 2018). We include FiLM layers after the first two linear layers of the neural field. The exact details are presented in appendix A.2.T
5

---

Under review as a conference paper at ICLR 2023
3.3 APPROXIMATE EQUIVARIANCE WITH GLOBAL-LOCAL COORDINATE FRAMES
Equivariant neural networks cannot capture non-local information, such as global field effects. In this work, explicitly aim to discover these fields and disentangle them from local object interactions. An alternative, or rather complementary approach, would be to directly combine global and local information, following the recently proposed notion of approximate equivariance (Wang et al.) 2022). Starting from LoCS (Kotinas et al. 2021). we can combine global information and still operate in local coordinate frames by defining an auxiliary node-object corresponding to the global coordinate frame, i.e. an object positioned at the origin, and oriented to match the x-axis.
Similar to all objects the system, the full state of the origin node O comprises the concatenation of its position and velocity, Xo = [po: uo] = (0, 0, 1, 1, We temporarily add this node in the geometric graph, and we further connect with edges to all other objects. Thus, the new vertex set is V V U {O). and the new edge set is E = E U (co, Vi E V}. The origin state can be expressed in local coordinate frames using similar transformations toleq. (I) The origin transformed in the local
coordinate frame of the i-th object is computed as follows:
VOl: R [p's P/: ub! R (-P' u'].
(10)
Since graph networks are permutation equivariant, have distinguish between this auxiliary
origin node and the other nodes, otherwise, the origin would be treated as an actual object, and there
no way for the network understand which the origin is. Hence, we augment each object's state
with the origin node information expressed in local coordinate frames, extending egs. (2) and (3) to
h2.a- = fo (( Vale
v
(11)
Ax In
f+ = fo
Vou
+
Σ h JEN(1)
(12)
[OO
This approach allows us to remove the origin node from the actual graph, and push the information in the node states. We term this method G-LoCS (Global-Local Coordinate FrameS). In practice, similar to Aether, we formulate our model as a variational autoencoder (Kingma & Welling 2014
Rezende et al. 2014) with latent edge types. The full details presented in lappendix A.2.3 Finally, in practice, we integrate G-LoCS in Aether, since it can enhance the performance of our method.
4 RELATED WORK
Equivariant graph networks The seminal works of (Cohen & Welling 2016 2017 Worrall
al. 2017) introduced equivariant convolutional neural networks and demonstrated effectiveness, robustness, and increased parameter efficiency. Recently, many works have proposed equivariant
graph networks (Schütt et al., [017 Thomas al. [018 Fuchs et al.) [220 Walters et al., 202T
Satorras et al., 2021 Kotinas et al. 2021: Brandstetter et al. [022]] Luo et al. 2022). Walters et al. (2021) propose rotationally equivariant continuous convolutions for trajectory prediction. Satorras et al. (202T) propose computationally efficient equivariant graph network that leverages invariant euclidean distances between node pairs. Kofinns et al. (2021) introduce ooto-translated local coordinate frames for all objects in an interacting system and propose equivariant local coordinate frame graph networks. |Bransttttter et al. (2022) generalize equivariant graph networks using steerable MLPs (Thomas et al.) 2018) and incorporate geometric and physical information in message passing. Equivariant graph networks differ from our work since they cannot capture non-local information, while our work disentangles equivariant local interactions from global effects and captures them both.
Approximate equivariance Recently, a number of works has proposed to shift away from strict cquivariance, in what (Wang et al., 2022) termed as approximate equivariance. WWang et al. (002) propose approximately equivariant networks for dynamical systems, by relaxing equivariance con- straints in group convolutions and steerable convolutions. van der Ouderaa et al. (2022) propose to relax strict equivariance by interpolating between equivariant and non equivariia operations, using non-stationary kernels that also depend on the absolute input group element. Romero & Lohit (2021) propose Partial G-CNNs that learn layer-wise partial equivariances from data. We note that

---

Under review as a conference paper at ICLR 2023
though these works and approximate equivariance share similarities with our work, our notion of disentangled equivariance is conceptually different. That is because related work uses the term approximate equivariance to denote that equivariance is "broken" due to noise imperfections, while our work disentangles the system dynamics that are actually equivariant, from the global field effects that are not, and in fact, might be unaffected by such transformations.
Neural fields Neural fields have recently exploded in popularity in 3D computer vision, popularized by NeRF (Mildenhall et al. 202T). Since MLPs are universal function approximators (Hornik et al.) 1989), neural fields parameterized by MLPs can, in principle, encode continuous signals arbitrary resolution. However, neural networks can suffer from "spectral bias" (Rahaman et al., 2019) Basni et al. 2020), Lee. they are biased to fit functions with low spatial frequency. To address this issue, number of solutions have been proposed. Tancik et al. (2020) leverage Neural Tangent Kernel (NTK) theory and propose Random Fourier Features (RFF), showing that they can overcome the spectral bias. They also show that RFF are generalization of positional encodings, popularized in recent years in natural language processing by Transformers (Vaswani al. (017). Concurrently, [Sitzmann et al. (2200) proposed SIREN, neural networks with sinusoidal activation functions. While neural fields have been used extensively in computer vision problems including 3D scene reconstruction (Park) et al. 2019, Mescheder et al. 2019) and differentiable rendering (Sitzmann et al.) 2019) Mildenhall et al. 2021). , they have not seen wide usage in dynamical systems. Notably, [Raissi al. (2019)
proposed Physics-Informer Neural Networks (PINNs), neural PDE solvers based neural fields.
EXPERIMENTS
We evaluate our proposed method, Aether, settings that include static as well as dynamic fields. First, we explore charged particles (Kipf et al.] 2018) that evolve under the effect of a static elec- trostatic field. Then, evaluate our method a single location, and thus a static field as well. Finally, we explore gravitational n-body problems (Brandstetter et al. 2022) with dynamic fields.
subset of inD (Bock et al. 2020) that contains
We compare our method against dNRI (Graber & Schwing 2020) and LoCS (Kofinas et al. 2021)) DNRI (Graber & Schwing 2020)
graph network operating in global coordinates, and is, in principle, able to uncover both the global and the local dynamics. It is formulated as a VAE (Kingma & Welling 2014 Rezende et al. 2014) with latent edge types and explicitly infers a latent graph structure. LoCS (Koinnas et al. (021)) on the other hand, operates in local coordinates, and is, thus, unable to uncover the global dynamics. For all methods, we use their publicly available source code. Our source code, data, and models will be shared upon publication.
Our architecture and experimental setup closely follow Graber & Schwing 2020); Kofinas al. (202I). Unless specified differently, neural field has a hidden size of 512. In charged particles and in n-body problems, we only use positions as input to the neural field, while in traffic scenes we also orientations. The full implementation details presented in mappenddxx A.2.2 In all settings, report the mean squared error of positions and velocities over time, Ft(t) ND Σn ||x] * 112-
Here we demonstrate indicative visualizations, and provide more extensive qualitative results in appendix C
5.1 CHARGED PARTICLES
First, study the effect of static fields, i.e. single field across all train, validation, and test simulations. We extend the charged particles dataset from frip et al. (2018) by adding a number of immovable sources. Overall, these sources act like regular particles, exerting forces the observable particles, except we ignore any forces exerted to them, and fix their positions and velocities to zero. We use M 20 "source" particles and N 5 "observable" particles. Sources particles principle have different charge magnitudes than the observable particles, but for simplicity we set all magnitudes [q|
in
All particles carry either positive or negative charges with equal probability. Following Satorras et al. (202)) Fuchs et al. (2020); Koñnas et al. (202T), we remove virtual borders that cause elastic collisions. We generate dataset of 50,000 simulations for training, 10,000 for validation and 10,000 for testing. Following Kipf et al. (2018), each simulation lasts for 49 timesteps. During inference, we use the first 29 steps as input and predict the remaining 20 steps.
7

---

Under review as a conference paper at ICLR 2023
dNRI
LoCS
G-LoCS
Aether (Ours)
Total Errors
Total Erross
Total Ervors
2.5
0.5
2.0
2,0
MSE
0.6
MSE
1.5
0.4
1.0
1.0
0.5
0.0
Step
Stop
Stop
(a)
(b)
Figure 4: Results on (a) charged particles, (b) inD, and (c) gravity.
Figure 6: Discovered field on inD (Bock al. 2020). For simplicity, we only visualize the field for discrete input orientations in C4 = {0, 2 x. 2
We compare method against dNRI and LoCS, well as G-LoCS, which should in principle be able to model both local and global dynamics effectively. We plot MSE in fng. 4a and Le errors in fig.. 20, and visualize the learned field in fig. 5 We visualize sample predictions in fig. 8al and showcase more pre- dicted trajectories in inlappendix C.I We observe that equivariant methods like LoCS perform poorly, while the approximately equivariant G-LoCS performs much better than equivariant and non-equivariant methods. Aether outperforms all other methods, demonstrating that it can disentangle equivariance. Furthermore, as shown in fig. 5] and and 12 in lappendix C.I.I Aether can effectively discover the underlying field.
Prodicted
Groundrruth
Figure 5: Learned Field (left) in charged particle settings compared to groundtruth (right).
5.2 TRAFFIC SCENES
Next, we study the effectiveness of static field discovery in traffic scenes. We use inD (Bock et al. 2020), dataset with real-world traffic scenes that comprises trajectories of pedestrians, vehicles, and cyclists, recorded different locations. We create subset that contains scenes from a single location. Namely, we choose "Frankenburg. Aachen", since it is the location with most interactions in the dataset. The subset corresponds to 12 recordings; we use for training, 2 for validation, and for testing. We follow similar experimental setting with Graber & Schwwng (2020); Kofinas et al. (202T). We divide each scene into 18-step sequences. We use the first 6 time steps as input and predict the next 12 time steps. We plot MSE in fig. 4bl and L2 errors in fig. 21 and visualize the learned field in ffi.. 6) and in fig. 17 in lappeddxx C.2. Since the learned field is a function of positions and orientations, only visualize it for 4 discrete orientations, namely the group C4 {0, 3, म, 3 ). We visualize predictions in fig. 8b) and showcase more in inlapeendix C.2 Again, we see that Aether outperforms all other methods. The discovered field, while hard interpret, shows high activations that coincide with road locations and directions, indicating that it can guide objects through the topology of the road network.
5.3 GRAVITATIONAL FIELD
8

---

Under review as a conference paper at ICLR 2023
(a)
(c)
Figure 8: Aether predictions (a) charged particles, (b) inD, and (c) gravity.
We now study the task of field discovery with dynamic fields, i.e. fields that are different across simulations. We extend the gravity dataset by Brandstetter et al. (022) by adding gravitational sources. We create dataset of 5,000 simulations for training. 1,000 for validation and 1,000 for testing. We use N 5 particles and M = 1 source. We set the masses of particles to mp = while the source has mass of m, = 10. We generate trajectories of 49 timesteps. We use the first 44 timesteps as input and predict the remaining 5 steps.
Predected Feed
Groundrruhh Rold
Preckcted
We plot MSE fig. 4c and L2 errors in fig. 22, and visualize the learned field in fig. 7 We visualize sample predictions in fig...g.. 8c and showcase more in inpppendix C.3] We observe that Aether outpertorms other methods, and can accurately discover the correct "shape" of the force fields, as well as the location of the sources.
Figure 7: Learned dynamic fields (left) in n-body problem vs groundtruth (right).
5.4 ABLATION EXPERIMENTS
Significance of discovered field simulated environment like the charged particles setting, have access the simulator, and by exten- sion, to the groundtruth fields and the sources that generate them. We can leverage the simulator to study the significance of the discovered field in the task of trajectory forecasting, and establish an upper bound to our performance. To that end, create two "oracle" models that have access to the groundtruth fields, force oracle and a source oracle. The force oracle identical Aether, but the groundtruth forces from the simulator at each point instead of predicting them with neural field. The source oracle assumes knowledge of the "field sources". Thus, there is no longer need for disentanglement, and the problem is strictly equivariant graph, add include
Total Erross
1.0
Particl Oraale
Onade
Anther
0.6
0.2
0.0
550p
Figure 9: Results the importance the learned field in charged particles.
again. We include the sources as virtual nodes edges from the sources to the particles. We modify LoCS to use virtual nodes, and predict future states for all the "observable" particles, but not for the "source" particles. We describe this oracle in detail in appendix A.2.4 We plot the MSE in fig. 9and the L2 errors in fig. 23 in appendix D.4] We observe that our model's performance closely follows the oracle models, and in fact, par, for roughly 10 steps. This demonstrates that the discovered field is almost as helpful as the groundtruth.
CONCLUSION
In this work, we introduced Aether, method that disentangles local object interactions from global field effects in interacting dynamical systems. We proposed neural fields to discover such field effects, and infer them from the dynamics alone. We combined neural fields with equivariant graph networks to learn the dynamical systems. We demonstrate that our method can accurately discover the underlying fields in range of settings with fixed and dynamic fields, and effectively use them to forecast future trajectories.
Limitations
Fields not always passive, but they may react the observable environment.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Alexandre Alahi, Kratarth Goel, Vignesh Ramanathan, Alexandre Robiequet, Fei-Fei Li, and Silvio Savarese. Social LSTM: Human Trajectory Prediction in Crowded Spaces. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In 3rd International Conference Learning Representations (ICLR), 2015.
Ronen Basri, Meirav Galum, Amnon Geifman, David Jacobs, Yomi Kasten, and Shira Kritchman. Frequency Bias in Neural Networks for Input of Non-Uniform Density. Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, and Koray Kavukcuoglu. Interaction Networks for Learning about Objects, Relations and Physics. In Advances in Neural Information Processing Systems 29 (NIPS), 2016.
Julian Back, Robert Krajewski, Tobias Moers, Steffen Runde, Lennart Vater, and Lutz Eckstein. The inD dataset: A Drone Dataset of Naturalistic Road User Trajectories at German Intersections. In 2020 IEEE Intelligent Vehicles Symposium 2020.
Johannes Brandstetter, Rob Hesselink, Elisc der Pol, Erik Bekkers, and Max Welling, Geometric and Physical Quantities Improve E(3) equivariant Message Passing. In Tenth International Conference on Learning Representations (ICLR), 2022.
Polo Calafiura, Steven Farrell, Heather Gray, Jean-Roch Vlimant, Vincenzo Innocente, Andreas Salzburger, Sabrina Amrouche, Tobias Golling, Moritz Kiehn, Victor Estrade, al. TrackML: A High Energy Physics Particle Tracking Challenge. In 2018 IEEE 14th International Conference on e-Science (e-Science), 2018
Ming-Fang Chang. John Lambert, Patsorn Sangkloy, Jagject Singh, Slawomir Bak, Andrew Hartnett, De Wang, Peter Carr, Simon Lucey, Deva Ramanan, and James Hays. Argoverse: Tracking and Forecasting With Rich Maps. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference Empirical Methods in Natural Language Processing (EMNLP), 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C. Courville, and Yoshua Bengio. Recurrent Latent Variable Model for Sequential Data. In Advances in Neural Information Processing Systems 28 (NIPS), 2015.
Ajork-Aré Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs). In 4th International Conference on Learning Representations (ICLR). 2016.
CMU. Carnegie-Mellon Motion Capture Database, 2003. URL http::/mocap.csscm.
Taco Cohen and Max Welling. Group Equivariant Convolutional Networks. In Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016.
Taco S. Cohen and Max Welling. Steerable CNNs. In 5th International Conference on Learning Representations (ICLR), 2017.
Pim de Haan, Maurice Weiler, Taco Cohen, and Welling Max. Gauge Equivariant Mesh CNNs: Anisotropic convolutions on geometric graphs. In 9th International Conference on Learning Representations (ICLR), 2021.
Haowen Deng, Tolga Birdal, and Slobodan llie. PPF-Foldnet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors. In Proceedings the European Conference on Computer Vision (ECCV), 2018.
Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. SplineCNN: Fast Geometric Deep Learning With Continuous B-Spline Kernels. In 2018 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
Fabian Fuchs, Daniel F. Worrall, Volker Fischer, and Max Welling. trans-rranssss:mes::: 3D Roto- coto-Translation Equivariant Attention Networks. In Advances in Neural Information Processing Systems 33 (NeurIPS), 2020.
10

---

Under review as a conference paper at ICLR 2023
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. Proceedings of the 34th International Conference on Machine Learning (ICML), 2017.
Colin Graber and Alexander G. Schwing. Dynamic Neural Relational Inference. In 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
William Hamilton, Zhitao Ying, and Jure Leskovec. Inductive Representation Learning on Large Graphs. In Advances Neural Information Processing Systems 30 (NIPS). 2017.
Dirk Helbing and Peter Molnar. Social Force Model for Pedestrian Dynamics. Physical review 51(5), 1995.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-term Memory. Neural computation, 98),, 1997.
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks universal approximators. Neural networks, 2(5):359-366, 1989.
Fric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel-Softmax. In 5th International Conference on Learning Representations (ICI.R), 2017.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Lue Van Gool. Dynamic Filter Networks. In Advances in Neural Information Processing Systems 29 (NIPS), 2016.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations (ICLR), 2014.
Thomas N. Kipt and Max Welling, Semi Supervieee Classification with Graph Convolutional Networks. In 5th International Conference on Learning Representations (ICLR), 2017.
Thomas N. Kipf, Ethan Fetaya, Kuan-Chich Wang, Max Welling, and Richard S. Zemel. Neural Relational Inference for Interacting Systems. Proceedings of the 35th International Conference on Machine Learning (ICML), 2018.
Miltiadis Kofinas, Naveen Shankar Nagaraja, and Efstratios Gavves. Roto-translated Local Coordinate Frames For Interacting Dynamical Systems. In Thirty-Fifih Conference on Neural Information Processing Systems (NeurIPS), 2021.
Alon Lerner, Yiorgos Chrysanthou, and Dani Lischinski. Crowds by Example. In Computer graphics forum, volume 26. Wiley Online Library, 2007.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard S. Zemel. Gated Graph Sequence Neural Networks. In 4th International Conference on Learning Representations (ICLR), 2016.
Shitong Luo, Jiahan Li, Jiaqi Guan, Yufeng Su, Chaoran Cheng, Jian Peng. and Jianzhu Ma. Equivariant Point Cloud Analysis via Learning Orientations for Message Passing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Conerete Distribution: A Continuous Relaxation of Discrete Random Variables. In 5th International Conference Learning Representations (ICLR), 2017.
Lars Mescheder, Michael Oechsle. Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition (CVPR), 2019.
Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi. and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. Communications of the ACM, 65 (1):99-106, 2021.
Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, and Christian G. Claudel. Social-STes STGCNN: A Social Spatio-Temmn Temporal Graph Convolutional Neural Network for Human Trajectory Prediction. In 2020 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), 2020.
Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodolà, Jan Svoboda, and Michael M. Bronstein. Geometric Deep Learning on Graphs and Manifolds Using Mixture Model CNNs. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Nigamaa Nayakanti, Rami Al- Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion Forecasting via Simple & Efficient Attention Networks, arXiv preprint ar X2v: 2022.
11

---

Under review as a conference paper at ICLR 2023
Jeong Joon Park. Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (CVPR), 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, perforeeferm deep learning library. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019.
Stefano Pellegrini, Andreas Ess, Konrad Schindler, and Luc Van Gool. You'll Never Walk Alone: Modeling Social Behavior for Multi-target Tracking. IEEE 12th International Conference on Computer Vision (ICCV), 2009.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with a General Conditioning Layer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the Spectral Bias of Neural Networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.
Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informan neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686 707, 2019.
Prajit Ramachandran, Barret Zoph, and Quoe Le. Searching for Activation Functions. 6th International Conference on Learning Representations, (ICLR), 2018
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31st International Conference on Machine Learning (ICMI.), 2014.
David W Romero and Suhas Lohit. Learning Equivariances and Partial Equivariances from Data. arXiv preprint arXiv:21 10211, 2021.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron++:: Dynamically-Feasibbe Trajectory Forecasting With Heterogeneous Data. In European Conference Computer Vision (ECCV), 2020.
Alvaro Sanchez- Sanchez-Gonzake Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter W. Battaglia. Learning Simulate Complex Physics with Graph Networks. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) Equivariant Graph Neural Networks. In Proceedings of the 38th International Conference on Machine Learning (ICML), 2021.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The Graph Neural Network Model. IEEE transactions on neural networks, 20(1), 2008.
Kristof Schüitt, Picter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmicla, Alexandre Tkatchenko, and Slluus-Rober Müller. SchNet: A continuous-filee -fittt convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems 30 (NIPS), 2017.
Martin Simonovsky and Nikos Komodakis. Dynamic Edge-Conditionel Filters in Convolutional Neural Networks on Graphs. In 2017 IEEE Conference on Computer Vision and Patter Recognition (CVPR). 2017.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene Representation Networks: Continuous 3D-Structure-AAare Neural Scene Representations. Advances in Neural Information Processing Systems 32 (NeurIPS), 2019.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Implicit Neural Representations with Periodic Activation Functions. Advances in Neural Information Processing Systems 33 (NeurlPS), 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. In Fourth Conference on Neural Information Processing Systems (NeurIPS), 2020.
12

---

Under review as a conference paper at ICLR 2023
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equiiariatt neural networks for 3D point clouds. arXiv preprint arXiv: 1802.08219, 2018.
Tycho FA van der Ouderaa, David Romero, Mark van der Wilk. Relaxing Equivariance Constraints with Non-stationary Continuous Filters. arXiv preprint arXiv:2204.07178. 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. Advances in Neural Information Processing Systems 30 (NeurIPS), 2017.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. In 6th International Conference on Learning Representations (ICLR), 2018.
Anirudh Vemula, Katharina Muelling, and Jean Oh. Social Attention: Modeling Attention in Human Crowds. In 2018 IEEE international Conference Robotics and Automation (ICRA). 2018.
Robin Walters, Jinxi Li, and Rose Yu. Trajectory Prediction using Equivariant Continuous Convolution. In 9th International Conference on Learning Representations (ICLR), 2021.
Rui Wang, Robin Walters, and Rose Yu. Approximately Equivariant Networks for Imperfectly Symmetric Dynamics. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.
Shenlong Wang. Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep Parametric Continuous Convolutional Neural Networks. In 2018 IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic Graph Can for Learning on Point Clouds. Acm Transactions On Graphics (tog), 38(5), 2019.
Maurice Weiler and Gabriele Cesa. General B(2) Eqquivarra Steerable CNNs. Advances in Neural Information Processing Systems 32 (NeurIPS), 2019.
Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow, Harmonic Networks: Deep Translation and Rotation Equivariance. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany. Shigin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural Fields in Visual Computing and Beyond. In Computer Graphics Forum, 2022.
Yisong Yue, Patrick Lucey, Peter Carr, Alina Bialkowski. and lain Matthews. Learning Fine- Grained Spatial Models for Dynamic Sports Play Prediction. 2014 IEEE international conference on data mining, 2014,
13


Reviews:
[{"review_id": "yewJ7zOSfqz", "reviewer": "Reviewer_DdmJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces Aether, a new method that disentangles local object interactions from global field effects in interacting dynamic systems. The new approach leverages the neural field and VAE model to infer from the dynamics along. Then combining neural fields with equivariant graph networks learns dynamic systems.  A set of examples are used to demonstrate the methods compared with several baseline methods. ", "summary_of_the_review": "Overall, the paper is good to show the idea with solid experiments. However, the core idea is not very clear to show. As I mentioned in the weakness, \n\n1. what's the core difference between aether and previous SOTA work？ \n2. what's the major contribution that makes entanglement to disentanglement? \n3. What's high-level intuition? \n4. The method is VAE-based disentanglement. Any previous related works? \n5. What's the current limitation, except the domain limitation? ", "strength_and_weaknesses": "- Strength\n\n1. The paper is well-written and organized so that it is easy to follow.\n2. Background section provides many necessary details and information\n3. The overall idea of disentangling local and global is very interesting  \n\n- Weakness\n\n1. The novelty is not clear to show. \n2. Lack of high-level intuition why the proposed method works better, specifically disentanglement property \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "lRnCU5fSoP", "reviewer": "Reviewer_HEK4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the notion of disentangled equivariance for systems of interactive systems, as well as a framework that disentangles the effects of underlying global field from the local system dynamics that are equivariant to transformations in SE(3). They are also able to learn a neural field to predict underlying field effects. ", "summary_of_the_review": "See strength and weakness section.\n------------\nThe authors' response address many of my concerns and questions. I will raise the score.", "strength_and_weaknesses": "Strengths:\n\n1. This work proposes a novel notion of disentangled equivariance, and proposes a framework to successfully disentangle global field effects from local system interactions that are equivariant to translations and rotations.\n2. They are able to predict global field effects that are latent by observing the system dynamics using neural fields.\n\nWeaknesses:\n\n1. The paper need more work to explain the motivation and their method. Most of the explanation for base concepts is referred to in related papers. \n2. The experimental setup is not clear. Again, an explanation of datasets and learning problem should have been provided, either in the main paper or if not, in the supplementary.\n3. The relationship between the notion of approximate equivariance and disentangled equivariance is unclear from their description. In section 3.3, they mention that the difference lies in the fact that approximate equivariance tries to combine global field and local field effects. But it seems from their model description in section 3.1 that they are doing the same thing (especially from eqs. 5-7). Hence, the novelty of their notion of disentangled equivariance is unclear.\n4. The experiments are also conducted on fairly limited settings (in two dimensional systems) and in scenarios where the underlying global fields remain fixed throughout the interaction. This makes it difficult to assess the impact of this work.\n5. The results are also difficult to interpret and require more details about what is the ideal result and how the method compares to the baselines. For eg., in Figure 7, it is unclear what the figures represent and why the predicted and ground-truth fields look so different.\n\nOther things that could improve the paper:\n\n1. A more detailed description of their experimental setup, and datasets used in the supplementary would be helpful to making the method and its impact more clear.\n2. Deeper explanation of the model architecture by using the supplementary material would also be helpful.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Xby6YcWgxD", "reviewer": "Reviewer_Gz6i", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper considers the task of discovering the underlying force field of the interacting dynamical systems. In particular, this paper focus on the setting where the observed trajectories of particles are driven by the interacting forces between the particles as well as a global force field. The entanglement of global and local effects makes recently popularized equivariant networks inapplicable, since they fail to capture global information. To address this issue, this work proposes to disentangle local object interactions from external global field effects. Specifically, a neural field is used to approximate the latent field and the interactions between particles are modeled with equivariant graph networks operating in local coordinate frames. Experiments show that the proposed approach accurately discovers the underlying fields in several interesting application.", "summary_of_the_review": "This is nice paper that studies the force field discovery in a more practical setting. A question I have for the authors is that in the more interesting dynamic field setting, why can we expect the dynamic force and the interacting forces to be disentangled using this approach? Intuitively, this is not possible without further assumptions.", "strength_and_weaknesses": "Strength:\n\nWhile previous works focus on the discovery of the interacting force between particles from the observable trajectory of particles, this work takes a step further by considering the existence of an underlying global force field. This generalized setting cannot be captured by the equivariant networks as they fail to capture global information. By incorporating an additional neural field, the authors design a new model, Aether, which in principle is able to automatically learn the interacting force and global force separately, which hence disentangles these two forces.\n\nWeakness:\n\n1. The global field is assumed to be independent of the observed objectives, but this seems to be a strong assumption, e.g. when the blackhole is considered as the source of the global field, its position and velocity are also affected by the planets around it. Therefore, this assumption may not hold, especially in a large time interval.\n2. While authors do consider the setting of dynamic field, it is clear how the interacting force can be disentangled from this time-varying dynamic field. In the worst case scenario, all forces can be described by a dynamic field.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ht5XelNCfs7", "reviewer": "Reviewer_T3HQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Please provide a brief summary of the paper and its contributions.\nThe authors proposed a disentanglement approach to local object interactions from external global ﬁeld effects, which depend on absolute positions and orientations. \nThe contribution of this paper is as follows: \n* They introduced the idea of entangled equivariance that intertwines global and local effects, and proposed a new architecture that disentangles equivariant local object interactions from global ﬁeld effects. \n* They introduced neural ﬁelds to discover global latent ﬁelds in interacting dynamical systems, and infer them by observing the dynamics alone. \n* They proposed an approximately equivariant graph network that extends local coordinate frame graph networks by introducing an auxiliary origin node.\n* They conducted experiments on a number of ﬁeld settings, and observed that explicitly modeling ﬁelds was mandatory for effective future forecasting, while their unsupervised discovery provided model explainability.\n", "summary_of_the_review": "Based on the comment above, I consider that the strength of this paper outperformed the weakness, but I had some concerns at this stage, so I cannot give a higher rating.", "strength_and_weaknesses": "Strength: \n* The ideas of the above three contributions are interesting and may have a novelty.\n* Overall, the paper was well-written and I can understand most of the ideas.\n* The experimental results clearly show the superiority of the proposed approach.\n\nWeakness:\n* Although the presentation was overall clear, some points in methods and experimental results were unclear to me (described as the specific comments).\n\nSpecific comments\n* About the name of the proposed method: Aether. The existence of Aether in modern physics may be denied, but what is the standpoint of the authors? I want to know the motivation and the opinion from the viewpoint of physics. \n* In section 3.3, the author said that “Finally, in practice, we integrate G-LoCS in Aether” but in the experiments I did not find the integration. Where is this?\n* First sentence in Section 5.2: InD (Bock…\n* Fig 6: I understand “the learned ﬁeld is a function of positions and orientations”, but how did the author define C4 in the inD experiment? I did not understand how we should see Fig 6. The definition of the orientations here and qualitative explanations may be required.\n* Section 5.4: In the ablation experiments, I did not understand why the oracle simulations accumulated errors. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject