Latent Field Discovery in Interacting Dynamical Systems with Neural Fields
Authors: Miltiadis Kofinas
Erik J Bekkers
Naveen Shankar Nagaraja
Efstratios Gavves
Venue: NeurIPS 2023 poster (2023)
Keywords: Graph Neural Networks
Neural Fields
Field Discovery
Equivariance
Interacting Dynamical Systems
Geometric Graphs
TL;DR: We discover global fields in interacting systems, inferring them from the dynamics alone, using neural fields.

Abstract: Systems of interacting objects often evolve under the influence of underlying field effects that govern their dynamics, yet previous works have abstracted away from such effects, and assume that systems evolve in a vacuum. In this work, we focus on discovering these fields, and infer them from the observed dynamics alone, without directly observing them. We theorize the presence of latent force fields, and propose neural fields to learn them. Since the observed dynamics constitute the net effect of local object interactions and global field effects, recently popularized equivariant networks are inapplicable, as they fail to capture global information. To address this, we propose to disentangle local object interactions --which are SE(3) equivariant and depend on relative states-- from external global field effects --which depend on absolute states. We model the interactions with equivariant graph networks, and combine them with neural fields in a novel graph network that integrates field forces. Our experiments show that we can accurately discover the underlying fields in charged particles settings, traffic scenes, and gravitational n-body problems, and effectively use them to learn the system and forecast future trajectories.

Latent Field Discovery In Interacting Dynamical Systems With Neural Fields
Miltiadis Kofinas
Erik J. Bekkers University of Amsterdam e.j bekkersduva.nl
Naveen Shankar Nagaraja
University of Amsterdam
BMW Group
m. koffinssuva...
Naveen- Laveen-Shanaar.Waaaaaaaaaabt .wwgaaaaaabam.........aaaaaaaa.. de
Efstratios Gavves University of Amsterdam egavves@uva.nl
Abstract
Systems of interacting objects often evolve under the influence of field effects that govern their dynamics, yet previous works have abstracted away from such effects, and assume that systems evolve in a vacuum. on discovering these fields, and infer them from the observed dynamics alone, without directly observing them. We theorize the presence of latent force fields, and propose neural fields to learn them. Since the observed dynamics constitute the net effect of local object interactions and global field effects, recently popularized equivariant networks are inapplicable, as they fail to capture global information. To address this, we propose to disentangle local object interactions -which are SE(n) equivariant and depend relative states- from external global field effects -which depend on absolute states. We model interactions with equivariant graph networks, and combine them with neural fields in novel graph network that integrates field forces. Our experiments show that we can accurately discover the underlying fields in charged particles settings, traffic scenes, and gravitational n-body problems, and effectively use them to learn the system and forecast future trajectories.
this work, we focus
Introduction
Systems of interacting objects are omnipresent in nature, with examples ranging from the sub- atomic to the astronomical scale -including col- liding particles and n-body systems of celestial objects- as well as settings that involve human activities, governed by social dynamics, like traf- fic scenes. The majority of these systems does not evolve in vacuum; instead, systems evolve under the influences of underlying fields. For example, electromagnetic fields may govern the dynamics charged particles, while galaxies swirl around supermassive black holes that create gravitational fields. In traffic scenes, the road network and traffic rules govern the actions of traffic scene participants. Despite the ubiquity of fields, previous works on modelling interacting systems have only focused on the in vitro case of systems evolving in vacuum.
Figure I: N-body system with underlying gravita- tional field. We uncover fields that underlie inter- acting systems using only the observed trajectories.
Earlier work learning interacting systems proposed graph networks [3, 23, 40). Recently, state-of- the-art methods for interacting systems propose equivariant graph networks [50, 41, 24, 5, 10] to
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

model dynamics while respecting the symmetries that often underlie them. These networks exhibit increased robustness and performance. while maintaining parameter efficiency due weight sharing. They are, however, not compatible with underlying field effects, since they can only capture local states, such as relative positions, while fields depend on absolute states (e.g. positions or orientations). In other words, global fields violate the strict equivariance hypothesis.
Within the context of modelling interacting systems, function f that predicts future trajectories is SE(3) equivariant -equivariant to the special Euclidean group of translations and rotations- if (Rx+ = (x) + for a translation vector T and a rotation matrix R. While strict equivariance holds in idealized settings, it does not hold in many worldd settings. That is, even if the symmetries exist in a particular setting, they only manifest themselves in local interactions, yet they are entangled with global effects that stem from absolute states. N-body systems from physics, for example, exhibit F(3) symmetries, since gravitational forces only depend on relative positions. Dynamies, however, may be influenced by external force fields, e.g. black holes, which are either unknown or not subject to transformations. Thus, strict equivariance violated, since equivariant object interactions are entangled with global field effects.
We make the following contributions. First, we introduce neural fields to discover global latent force fields in interacting dynamical systems, and infer them by observing the dynamics alone. Second, we introduce the notion of entangled equivariance that intertwines global and local effects, and propose a novel architecture that disentangles equivariant local object interactions from global field effects. Third, we propose an approximately equivariant graph network that extends equivariant graph networks by using a mixture of global and local information. Finally, we conduct experiments on a number of field settings, including real-world traffic scenes, and extending state-of- -he-trt-tt setups from the literature. We observe that explicitly modelling fields is mandatory for effective future forecasting, while their unsupervised discovery opens a window for model explainability.
We term our method Aether, inspired by the postulated medium that permeates all throughout space and allows for the propagation of light.
Background
Interacting dynamical systems An interacting dynamical system comprises trajectories of N objects in d dimensions, E [2,3], recorded for timesteps. The snapshot of the i-th object at timestep / describes the state x² = (p/, uf]. i E (1,..., N), 1 (1,..., .T). where p E Rd denotes Rd denotes the velocity, using I+ -J to denote vector concatenation along the feature dimension. We are interested in forecasting future trajectories, i.e. predict the future states for all objects and for a number of timesteps. Interacting dynamical systems can be naturally formalized
the position and
T with graph snapshots g = (V", (*) at different time steps. The set of graph nodes yr = (uf, ) describes the objects in the system; of corresponds to x/. The set of edges Ex S ((r). uf) (of uf) V" V* describes pair- wise object
as spatio-temporte geometric graphs [3, 23, 14), )
1==:
interactions; (vf, ef) corresponds to an interaction from node j to node i. Finally, (i) denotes the neighbors of node Ui
Local coordinate frame graph networks Local coordinate frame graph networks have been popu- larized in recent years [24, 26, 10, 52, 20, 30] as method to achieve SE(3) oor E(3)- equivariance, due to their low computational overhead and high performance. Kofinas et al. [24] proposed LoCS and introduced local coordinate frames for all node-objects all timesteps. They define augmented node states v = [p] w!, ug]. where w/ denotes the angular position of node at timestep f. Kofinas et al. [24] use velocities that predicts latent orientations. Each local coordinate frame is translated to match the target object's position and rotated match its orientation. Considering the representation of node j in the local coordinate frame of node i, denoted as y they first compute the relative positions ry.: = =P} pf
proxy to angular positions, while Luo et al. [26] use another network
and then they rotate the state using the matrix representation of the angular position Q(w/):
VSH+= R(w!) T [F w/, u
(I)
2

---

where R(w!) Q(w{)
( @ (ww))) @ Qw and @ denotes direct sum. LoCS then proposes graph neural network [42, 25, 13] that uses local states:
hy, = fo (e (1) vi.))
(2)
Ax Ax 1/1 = fo Ir (((xx)
+
En JEN())
(3)
where fe, fer and Ie MIPs, and C= 1/((())- The output of this graph network comprises differences in positions and velocities from the previous time step, in the local frame each object. Since these outputs are invariant, LoCS performs inverse transformation to convert them back to the global coordinate frame and achieve equívariance, x x + R(w!) R(w() = Q(w!) + Q(w!).
++1 Ax where
i/s
Neural fields Finally, we make brief introduction to neural fields. Neural fields, or coordinate- based MI.Ps, are a class of neural networks that parameterize fields using neural networks (see Xie et al. [54] for survey). They take as input states like spatial coordinates and predict some quantity. Neural fields can learn prior behaviors and generalize to new fields via conditioning on latent variable that encodes the properties of field. Perez et al. [33] proposed Feature-wise Linear Modulation (FiLM,, conditioning mechanism that modulates a signal. comprises two sub-networks a, 3 that perform multiplicative and additive modulation to the input signal, and can be described by (h, z) = (zz) O h (()), where z is the conditioning variable, h is the signal to be modulated, and are MLPs that scale the signal, and add bias term, respectively.
3 Method
In this section, we present our method, termed Aether. First, we describe the notion of entangled equivariance, and introduce architecture that disentangles global field effects from local object interactions. Then, we continue with the description of the neural field that infers latent fields by observing the dynamics alone. Finally, coordinate frame graph networks. We note that throughout this work, we focus on fields that are unaffected by the observable objects and their interactions thereof.
formulate approximately equivariant global-local
3.1 Aether
Interacting dynamical systems rarely evolve in a vacuum, rather they evolve under the influence of external field effects. While object interactions depend on local information, the underlying fields depend on global states. On the one hand, locality in object interactions stems from the fact that dynamics obey number of symmetries. By extension, object interactions equivariant to particular group of transformations. On the other hand, field effects are non-local; they depend on absolute object states. Thus, strict equivariance is violated, since equivariant object interactions entangled with global field effects. We refer to this phenomenon as entangled equivariance.
As example, in Figure we observe a system of two objects that evolve in a gravitational field. The arrows positioned the objects represent the forces exerted on them. One constituent of the net force is caused by object interactions, and is thus equivariant, while the other can be attributed to the gravitational pull. However, we can only observe the net force each partiele, i.e. the sum of equivariant pairwise forces and non-equivariant field effects. Hence, in this system, we say that equivariance entangled.
Fusd Ball faisk
Two objects in gravitational field. We only observe the total force exerted at each particle, i.e. the sum of equivariant pairwise particle forces and global field effects.
Figure 2:
We now propose our architecture that disentangles local object interactions from global field effects. We model object interactions with local coordinate frame graph net- works [24], and field effects with neural fields. During training, and given multitude of input systems, neural fields will. in principle, be able to isolate

---

global from local effects, since only global effects are recurring phenomena. We hypothesize that field effects can be attributed to force fields, and therefore, our neural fields learn to discover latent force fields. The pipeline of our method is shown in Figure 3. Our inputs comprise augmented states {v]) for trajectories of N objects for T timesteps. Since neural fields model global fields, they depend on absolute states. Thus, we feed the states of the trajectories v -or subset of state variables- as input to a neural field that predicts latent forces f f(v)).
Prodisted frecs
Query states
Predicted field
Groundtruth trajectories
Latent Neural Field
FiLM
MLP
Loss
FILM
f
MLP
Graph Aggregation
Graph
Einooding
Q
Neural Network
Ixpul unjesturics
Quary
Input trajectories
Predicted trajectories
(a) Latent Neural Field
(b) Aether pipeline
Figure 3: The pipeline of our method, Aether. In the latent neural field (a). a graph aggregation module summarizes the input trajectories in latent variable z. Query states from input trajectories, alongside z, are fed to a neural field that predicts a latent force field. In (b), a graph network integrates predicted forces with input trajectories predict future trajectories. The graph aggregation module and the FiLM layers exist only in a dynamic field setting.
The predicted field forces can be now considered part of the node states, and further, they can be treated similarly to other state variables like velocities; as vectors, forces are unaffected by the action of translations, while they covariantly transform with rotations. Thus, moving onward, we treat the problem setup as we were once again back in the strict equivariance regime. We append the predicted forces f for each node-object and each timestep to the node states, and transform them to corresponding local coordinate frames, similarly to Equation (1). Namely, the force exerted on node expressed in the local coordinate frame of node computed as: fie Q (W() We feed the new local node states local coordinate frame graph network as follows:
hí,a fe
Vija
Gife
(4)
))
Ax!11 1+1 fo 9r (
Vilēn
fijs
+ + JEN(()
(5)
x (+1 = x + R(w()
1+1
(6)
where R(w!) = =(w))=Q(w") @ Q(w)) C 1/V0O)) The equations above are similar to Equations (2) and (3), with the addition of the highlighted parts that denote the predicted forces expressed at local coordinate frames. In practice, in most experiments, we closely follow [23, 14, 24] and formulate our model variational autoencoder [21, 37] with latent edge types. The exact details are presented in Appendix A.1.2.
3.2 Field discovery
Oftentimes, fields might not be directly observable for us to probe them at will and use them for supervision. For example, astronomical observations of solar systems and galaxies might not include black holes, yet we can observe their effects. Moreover, fields are often not even measurable or quantifiable, or they are defined implicitly. For instance, "social fields" that guide traffic, cannot be measured or defined explicitly, but we can safely assume they exist. Motivated by these observations, we design an architecture that performs unsupervised field discovery, while solving the surrogate supervised task of trajectory forecasting.

---

In this work, we aim to discover two different types of fields, which we term "static" and "dynamic" fields. Static fields refer to settings in which we have a single field shared throughout the whole dataset. On the other hand, dynamic fields refer to settings in which we have different field for each input system, and consequently, fields also differ between train, validation, and test sets.
We now describe neural fields, used in this work, to model the underlying field effects. Neural fields depend on absolute states and predict latent force fields. When dealing with static fields, we use unconditional neural fields, i.e. neural fields that are functions only of the query states, as the field values are common across data samples. Note that unconditional neural fields are not functions of the input states; they will make the same predictions regardless of the inputs. In contrast, for dynamic fields, we use a conditional neural field, i.e. neural field that also depends on latent vector z E R.,, that represents the underlying field. The latent z will be inferred from the input trajectories and can be thought of as representing unusual non- coneaaiiaai dynamics. We use z to explicitly condition the neural field, and thus, its general form is f: IRd x SO(d) x Rd x RDD + Rd, where d E (2, 3), depending on the setting.
During training, both for conditional and unconditional neural fields, we only sample the field at query states that coincide with the states of the input objects, since we only have supervision about their future trajectories there.
Static fields We start with the description of unconditional neural fields used in static field settings, since conditional neural fields share the same backbone. First, we encode the query positions using where Rd are the query coordinates, and B E R d xd is matrix with entries sampled from Gaussian distribution, Bur ~ (N(0, The variance o² can be chosen per task with a hyperparameter sweep.
Gaussian random Fourier features [46], as follows: >()) = (cos(2nBp), sin(2mBp))
We encode velocities using a simple linear layer ((u) = W.,. u. For orientations, in d = 3 dimensions, use unit vector representation for each angle in w (0, o, (5) a [cos w, sin w] In d 2 dimensions, we use the same encoding, except that we now have a single angle w = 8. Then, we use linear layer to encode the orientation vectors, 8(w) W. We finally concatenate the encoded positions, orientations, and velocities in a single vector that is being fed as input to the neural field. The neural field is 3-layer MLP with SiLU [36] activations in-between, and outputs latent force field, f(v)= = MLP(( ((w). ((n))).
The neural fields used to model the dynamic fields are conditioned on a latent vector representation IR that describes prior knowledge about the underlying field, and are defined as f(v] z). In our case, the latent representation should "summarize" the input graph such that it isolates only global effects from the field. To that end, we employ a simple global spatio-temporal attention mechanism, similar to Li et al. [25], that aggregates the input system in a latent vector representation. First, we define object embeddings O, GRU (W xx! !:"T), where W,
Dynamic fields
matrix used to linearly transform the inputs, and GRU is the Gated Recurrent Unit [7]. We also define temporal embeddings PE(t), where PE are positional encodings [49]. Using these embeddings. we augment the input as s = [x), o, + t. The aggregation is then defined as follows:
Σ
softmax (fatmaxxfa(ff ($)) fo())),
where fo D. + R. D. R are 2-layer MLPs with SiLU activations nn-between.
After having obtained latent vector representation z that summarizes the input system, we condition the neural field using FiLM [33]. We include FiLM layers after the first two linear layers of the neural field. The exact details are presented in Appendix A.I.I.
3.3 Approximate equivariance with global-local coordinate frames
Equivariant neural networks cannot capture nocal information, such as global field effects. In this work, we explicitly aim to discover these fields and disentangle them from local object interactions. An alternative, or rather complementary approach, would be to directly combine global and local information, following the recently proposed notion of approximate equivariance [51]. Starting from LoCS [24], we can integrate global information and still operate in local coordinate frames
5

---

by defining an auxiliary node-object corresponding to the global coordinate frame, i.e. an object positioned at the origin, and oriented to match the x-axis.
Similar to all objects in the system, the full state of the origin node O comprises the concatenation of its position and velocity, xo [po; uo]. We an "artificial" velocity that matches the z-axis in order to compute non-degenerate frame. As such, we have x = [0, &J. The origin state can be expressed in the local coordinate frame of the i-th object similarly Equation (I), as follows:
O R PP' p/, _b! R u']
(8)
Since graph networks are permutation equivariant, we need to explicitly distinguish between the origin node and other nodes. We circumvent that by augmenting each object's state with the origin node information expressed in local coordinate frames, extending Equations (2) and (3) to
([
h3 = Se
Vil Vij
vo
Ax ++1 fo
gu
V
Vol
[N(1]
ΣnS h JEN(A)
(10)
This approach pushes the information in the node states, and removes the need to add the origin node to the actual graph. We term this method G-LoCS (Global-Local Coordinate FrameS). In practice, similar to Acther, we formulate G-LoCS as a variational autoencoder [21, 37] with latent edge types. The full details are presented in Appendix A.2. Finally, in practice, we integrate G-LoCS in Aether, since it can enhance the performance of our method.
Related work
Equivariant graph networks The seminal works of [8, 9, 53] introduced equivariant convolutional neural networks and demonstrated effectiveness, robustness, and increased parameter efficiency. Recently, many works have proposed equivariant graph networks [43, 47, 12, 50, 41, 24, 5, 26, 18]. Walters et al. [50] propose rotationally equivariant continuous convolutions for trajectory prediction. Satorras al. [41] propose computationally efficient equivariant graph network that leverages invariant euclidean distances between node pairs. Kofinas et al. [24] introduce roto-translated local coordinate frames for all objects in interacting system and propose equivariant local coordinate frame graph networks. Brandstetter et al. [5] generalize equivariant graph networks using steerable MLPs [47] and incorporate geometric and physical information in message passing. Equivariant graph networks differ from our work since they cannot capture non-local information, while our work disentangles equivariant local interactions from global effects and captures them both.
Approximate equivariance Recently, a number of works has proposed to shift away from strict equivariance, in what Wang al. [51] termed approximale equivariance. Wang al. [51] propose approximately equivariant networks for dynamical systems, by relaxing equivariance constraints in group convolutions and stecrable convolutions. van der Ouderaa al. [48] propose relax strict equivariance by interpolating between equivariant and non-equivariant operations. using non- stationary kernels that also depend on the absolute input group element. Romero and Lohit [38] propose Partial G-CNNs that learn layer-wise partial equivariances from data. We note that even though approximately equivariant networks share similarities with our work, our notion of disentan- gled equivariance is conceptually different. That is because related work uses the term approximate equivariance to denote that equivariance "broken" due to noise or imperfections, while our work disentangles the system dynamics that are actually equivariant, from the global field effects that are not, and in fact, might be unaffected by such transformations. Further, to the hest of our knowledge, approximate equivariance has only been studied in the context of convolutional networks, not in the context of graph networks and interacting systems. Tangentially, Han et al. [15] propose subequiv- ariant graph networks, and relax equivariance to subequivariance by considering external fields like gravity. However, they assume priori known fields that do not require to inferred by the model.
Neural fields Neural fields have recently exploded in popularity in 3D computer vision, popularized by NeRF [29]. Since MI.s are universal function approximators [17], neural fields parameterized by MLPs can, in principle, encode continuous signals at arbitrary resolution. However, neural

---

networks can suffer from "spectral bias" [34, 2], i.e. they are biased to fit functions with low spatial frequency. To address this issue, a number of solutions have been proposed. Tancik al. [46] leverage Neural Tangent Kernel (NTK) theory and propose Random Fourier Features (RFF), showing that they can overeome the spectral bias. They also show that RFF are a generalization of positional encodings, popularized in recent years in natural language processing by Transformers [49]. Concurrently, Sitzmann et al. [45] proposed SIREN, neural networks with sinusoidal activation functions. While neural fields have been used extensively in computer vision problems including 3D scene reconstruction [31, 28] and differentiable rendering [44, 29), they have not seen wide usage in dynamical systems. Notably, Raissi al. [35] proposed Physics-Informed Neural Networks (PINNs), neural PDE solvers based on neural fields. Finally, Dupont et al. [11] and Zhuang et al. [56] propose generative models of neural fields.
5 Experiments
We evaluate our proposed method, Aether, on settings that include static as well as dynamic fields. First, we explore 2D) charged particles that evolve under the effect of a static electrostatic field, as well as 3D particles that evolve under Lorentz. force field [10]. Then, we evaluate our method on subset of inD [4] that contains a single location, and thus a static field as well. Finally, we explore 3D gravitational n-body problems [5] with dynamic fields. Our code, data, and models will be open-sourced online'.
In most experiments, we compare our method against dNRI [14] and LoCS [24] two state-of-the-an sequence trajectory forecasting, as well G-LoCS. DNRI [14] is graph network operating in global coordinates, and is, in principle, able to uncover both the global and the local dynamics. is formulated as VAE [21, 37] with latent edge types and explicitly infers a latent graph structure. LoCS [24], on the other hand, operates in local coordinates, and is, thus, unable to uncover the global dynamics. Finally, G-LoCS is in principle able model both local and global dynamics effectively. For all methods, use their publicly available source code.
networks for sequence-to seuueenc-to-sequence
Our architecture and experimental setup closely follow Graber and Schwing [14], Kofinas et al. [24]. Unless specified differently, our neural field has hidden size of 512. In charged particles and in n-body problems, we only use positions as input to the neural field, while traffic scenes we also use orientations. The full implementation details are presented in Appendix A.1.2. In all settings, we report the mean squared error (MSE) of positions and velocities over time. Here we demonstrate indicative visualizations, and provide more extensive qualitative results in Appendix D).
For the Lorentz force field experiment, we the official source code from ClofNet [10], and follow their exact setup. We compare our method against SE((3) Transformers [12], EGNN [41], and ClofNet [10]. We evaluate methods using the mean squared error between predicted and groundtruth positions. Since this setting is not a sequence-ro-sequenee task, we use simplified network architecture without VAE, and following baselines, we make sure that the number of parameters of our model is approximately equal to other methods. The full implementation details are presented in Appendix A.1.3.
dNRI
+
LoCS
G-LoCS
Aether (Ouns
Total Eirors
Total Erors
1.0
2.5
0.5
1.5
0.10
0.06
0.2
0.5
7
9
3
Step
(a)
(b)
(c)
Figure 4: Results on (a) electrostatic field, (b) inD, and (c) gravity.
7

---

5.1 Electrostatic field
First, we study the effect of static fields, i.e. single field across all train, validation, and test simulations. We extend the charged particles dataset from Kipf et [23] by adding a num- ber of immovable sources. These sources act like regular particles, exerting forces on the ob- servable particles, except we ignore any forces exerted to them, and keep their positions fixed. We use M = 20 "source" particles and N = 5 "observable" particles. We generate 50,000 sim- ulations for training. 10,000 for validation and 10,000 for testing. Following Kipf et al. [23], each simulation lasts for 49 timesteps. During inference, we use the first 29 steps as input and predict the remaining 20 steps. The full dataset details presented in Appendix B.I.
Predicted Field
Groundtruth Field
Figure 5: Learned Field (left) in electrostatic field
setting compared to groundtruth (right).
We compare our method against dNRI, LoCS, and G-LoCS. We plot MSE in Figure 4a and L2 errors in Figure 18, and visualize the learned field in Figure 5. We showcase predicted trajectories in Figure Appendix D.I. We observe that equivariant methods like LoCS perform poorly, while the approximately equivariant G-LoCS performs much better than equivariant and non-equivariant methods. Acther outperforms all other methods, demonstrating that it can disentangle equivariance. Furthermore, as shown in Figure 5, and Figure 10 in Appendix D.I.1, Aether can effectively discover the underlying field.
5.2 Lorentz force field
Du et al. [10] introduced dataset of 3D charged particles evolving under the influence of a Lorentz force field. Each simulation contains 20 particles. We use the official source code and follow the exact experimental setup with Du et al. [10]. We show quantitative results in Table Our method can clearly outperform all other methods by large margin, reducing the error by 48.6%. We also note that our method has fewer parameters than ClofNet, and is thus more efficient.
Table 1: Position prediction MSE on Lorentz force field. Results marked with f were taken from ClofNet [10].
Method
MSE (L)
No. parameters
GNN І
0.0908
104,387
SE(3) Transformer [12]
0. 0.1438
1,763,134
EGNN i [41]
0.0368
134,020
ClofNet [10]
0.0251
160,964
Aether (ours)
0.0129
132,822
5.3 Traffic scenes
Next, we study the effectiveness of static field discovery in traffic scenes. We use inD [4], a dataset with real-world traffic scenes that comprises trajectories of pedestrians, vehicles, and cyclists. We create a subset that contains scenes from a single location. The full dataset details are presented in Appendix B.2. We divide scenes into 18-step sequences; we use the first 6 time steps as input and predict the next 12 time steps. We plot MSE in Figure 4b and L2 errors in Figure 19, and visualize the learned field in Figure and in Figure 15 in Appendix D.2.1. Since the learned field is a function of positions and orientations, we only visualize it for 4 discrete orientations, namely the 3m We showcase predictions in Figure 11 Appendix D.2. Again, Aether
group C2 = (0, x n, 2 outperforms all other methods. The discovered field, while hard to interpret, shows high activations that coincide with road locations and directions, indicating that it can guide objects through the topology of the road network.
8

---

Figure 6: Discovered field inD [4]. For clarity, we only visualize the field for discrete input orientations in C4 = {0, ,,
5.4 Gravitational field
We now study the task of dynamic field discovery, i.e. fields that are different across simulations. We extend the gravity dataset by Brandstetter et [5] by adding gravitational sources. We create a dataset of 50,000 simulations for training. 10,000 for validation and 10,000 for testing. We use N particles and M source. We set the masses of particles to mp = while the source's mass is m,, 10. We generate trajectories of 49 timesteps. We use the first 44 steps as input and predict the remaining steps. We plot the MSE in Figure 4c and L2 errors in Figure 20. We observe that once again, Aether clearly outperforms other methods.
5.5 Ablation experiments
Table 2: (a) Ablation study on the importance of the learned field. (b) Ablation study on the importance of sequential architecture. (c) Ablation study on the choice of equivariant GNN backbone.
(a) Electrostatic field
(b) Lorentz force field
(c) Lorentz force field
Method
MSE@ 10 (L)
Method
MSE (L)
Method
MSE (L)
Particle Oracle
0.1847
LoCS [24]
0.0238
EGNN [41]
0.0368
Force Oracle
0.1883
Aether
0.0129
EGNN+
0.0254
Aether
0.2015
Parallel Aether
0.0211
Significance of discovered field In simulated environment like the electrostatic field setting, we have access the grounduruth fields and the sources that generate them. We leverage the simulator to study the significance of the discovered field in the task of trajectory forecasting, and establish an upper bound to our performance. To that end, we create two "oracle" models that have access to the groundtruth information, a force oracle and source oracle. The force oracle is identical to Aether, but uses the groundtruth forces from the simulator instead of predicting them with a neural field. The source oracle assumes knowledge of the "field sources". Thus, there is no longer need for disentanglement, and the problem strictly equivariant again. We include the sources as virtual nodes in the graph, add include edges from the sources to the particles. We describe this oracle in detail in Appendix A.3. We show the MSE in Table 2a and plot the MSE and L2 errors in Figure 22 in Appendix E.5. We observe that Aether closely follows the two oracle models, and is par, for roughly 10 steps. This demonstrates that the discovered field is almost as helpful as the groundtruth.
Learning the global field separately Our architecture connects the neural field with the graph network sequentially, i.e. the output of neural field is given as input the graph network. We believe that this is integral for effective field discovery, as well as overall modelling, since it enables the graph network learn to isolate local interactions from the observed net dynamics. We this hypothesis with an ablation study, in which we connect the neural field and the graph network in parallel. The two networks are now working independently. and we only add their predictions at the output. We term this model Parallel Aether. We provide implementation details in Appendix A.4. We perform the experiment on the Lorentz field setting, and show results in Table 2b. We also compare both methods against LoCS, as it is common backbone in both methods, to demonstrate the performance gain due the discovered field. We can see that even though the parallel architecture boosts performance, clearly not effective the sequential approach, which verifies our hypothesis.

---

Table 3: Ablation study on using conditional neural fields for static fields. Experiment on Lorentz force field setting. Method
MSE (4)
No. parameters
Inference Time
LoCS [24]
0.0238
130,307
0,0033
Aether
0.0129
132,822
0,0037
Conditional Aether
0.0131
142,807
0.0047
Choice of equivariant network Our method is agnostic to the choice of equivariant graph network; we expect that it would be beneficial for a number of strictly equivariant networks. To test this hypothesis, combine
method with EGNN [41]. We start from the velocity formulation of EGNN and modify the message and velocity equations to incorporate the predicted forces for each node. We describe the model in detail in Appendix A.5. We train and evaluate this method on the Lorentz force field setting, and report the results in Table 2c. EGNN combined with Aether reduces the error by 30.9%, compared to vanilla EGNN, which enhances our hypothesis. We further include comparisons with more equivariant and non-equivariant graph networks in Appendix C.
Conditional neural fields for static settings Conditional neural fields generalize unconditional neural fields, and could, in principle, be used to learn static fields. In that case, the neural field should learn to ignore the latent vector, since the generated field should be identical regardless of the input system; we expect its performance to match the unconditional field. This, however, can come at the cost of increased training and inference time, well redundant computational resources and model parameters. We verify this hypothesis with an ablation study on the Lorentz force field, where we train and evaluate our method using a conditional field. In Table 3, we report the MSE, as well as the training time per minibatch, the inference time, and the number of parameters for each model. While the conditional model performs almost on par with the original unconditional model, this comes at the cost of 27% higher inference time and 9,985 more parameters. We conclude that the unconditional neural field is the preferred choice when there is expert knowledge that the field at hand is a static field. In the absence of such knowledge, e.g. on an exploratory analysis for underlying fields, then the conditional neural field would be preferable.
Conclusion
In this work, introduced Aether, method that discovers global fields in interacting systems. We propose neural fields to discover latent force fields, and infer them from the dynamics alone. Furthermore, disentangle global fields from local object interactions, and combine neural fields with equivariant graph networks to learn the systems. We show that our method can accurately discover the underlying fields in range of settings with static and dynamic fields, and effectively use them to forecast future trajectories. To the best of our knowledge, Aether is the first work that discovers fields in interacting systems, and the first that is able model systems with equivariant interactions and global fields. We hope that this work will inspire the community and bootstrap a line of works that explores field discovery, since fields are omnipresent in all scientific tasks.
Limitations In this work, we have only considered fields that do not react to the observable environment. While this setting is often true, in other scenarios, active fields might be crucial for effective modelling of the system dynamics. Furthermore, in the dynamic field setting, we assume that the input trajectories are descriptive enough summarize the field we are trying to discover. While this hypothesis often holds, it might not always be true. Future work can explore these very interesting research directions.
Acknowledgments
The project is funded by the NWO LIFT grant 'FLORA',
10

---

References
[1] Bahdanau, D., Cho, K., and Bengio, Y. Neural Machine Translation by Jointly Learning Align and Translate. 3rd International Conference Learning Representations (ICLR), 2015. 16
[2] Basri, R., Galun, M., Geifiman, A., Jacobs, D., Kasten, Y., and Kritchman, S. Frequency Bias in Neural Networks for Input of Machine Learning (ICML), 2020.
Density. Proceedings of the 37th International Conference on
[3] Battaglia, P. W. Pascanu, R., Lai, M., Rezende, D. J., and Kavukcuoglu, K. Interaction Networks for Learning about Objects, Relations and Physics. In Advances in Neural Information Processing Systems 29 (NIPS), 2016. 1,2
[4] Bock, J., Krajewski, R., Moers, T., Runde, S., Vater, L., and Eckstein, L. The inD dataset: A Drone Dataset of Naturalistic Road User Trajectorics at German Intersections. In 2020 IEEE Intelligent Vehicles Symposium (IV), 2020. 7, 8, 9, 20, 21, 24, 28
[5] Brandstetter, J., Hesselink, R., van der Pol, E., Bekkers, E., and Welling, M. Geometric and Physical Quantities Improve E(3) Equivariant Message Passing. In 10th International Conference Learning Representations (ICLR), 2022. 6, 7, 20
[6] Chang, M., Lambert, J., Sangkloy, P., Singh, J., Bak, S., Hartnett, A., Wang, D., Carr, P., Lucey, S., Ramanan, D., and Hays, Argoverse: 3D Tracking and Forecasting With Rich Maps. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Cho, K., van Merriënboer, B., Gulechre, C.,, Bahdanau, D., Bougares, F,, Schwenk, H., and Bengio, Y. Learning Phrase Representations using RNN Encoder-Decods for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014. 5, 14
[8] Cohen, T. and Welling, M. Group Equivariant Convolutional Networks. In Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016. 6
[9] Cohen, T. S.S. and Welling, M. Stecrable CNNs. In 5th International Conference on Learning Representations (ICLR), 2017. 6
[10] Du, W., Zhang, H., Du, Y., Meng, Q., Chen, W., Zheng, N., Shao, B., and Liu, T.-Y. SE(3) Equivariant Graph Neural Networks with Complete Local Frames. In Proceedings of the 39th International Conference on Machine Learning (ICML), 2022.
16, 17
[11] Dupont, F.,, Kim, H., Eslami, S., Rezende, D., and Rosenbaum, D. From data functa: Your data point is function and you Learning (ICMI.), 2022.7
it like one. Proceedings of the 39th International Conference on Machine
[12] Fuchs, Worrall, E., Fischer, V., and Welling, M. Transfrrrsssssmrrs:: 3D Roto- Roto-Tooassee Translation Equivariant Attention Networks. Advances in Neural Information Processing Systems 33 (NeurIPS), 2020. 6, 7,8, 8,
[13] Gilmer, J., Schoenholz, S. S., Riley, E,, Vinyals, O., and Dahl, G. E. Neural Message Passing for Quantum Chemistry. In Proceedings of the 34th International Conference on Machine Learning (ICML), 2017. 3
[14] Graber, C. and Schwing, A. G. Dynamic Neural Relational Inference. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 2, 4, 15, 16, 20, 22
[15] Han, J., Huang, W., Ma, H., Li, J., Tenenbaum, J., and Gan, C. Learning Physical Dynamics With Subequivariant Graph Neural Networks. In Advances in Neural Information Processing Systems 35 (NeurlPS), 2022. 21
[16] Hochreiter, S. and Schmidhuher, J. Long Short-term Memory. Neural computation, 9(8), 1997. 15
[17] Hornik, K., Stinchcombe, M., and White, H. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366, 1989. 6
[18] Huang, W., Han, J., Rong, Y., Xu, T., Sun, F,, and Huang, J. Equivariant Graph Mechanics Networks with Constraints. In 10th International Conference Learning Representations (ICLR), 2022. 6, 21
[19] Jang, E., Gu, S., and Poole, B. Categorical Reparameterization with Gumbel-Softman In 5th International Conference Learning Representations (ICLR), 2017. 15

---

[20] Kaba, S.-O.. Mondal, A. K., Zhang, Y.. Bengio. Y., and Ravanbakhsh, S. Equivariance with Learned Canonicalization Functions. In Proceedings the 40th International Conference Machine Learning (ICML), 2023. 2
[21] Kingma, D. and Welling, M. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations (ICIR), 2014. 4, 6, 7, 14, 15, 16
[22] Kipf. T. N. and Welling, M. Superviuee Classification with Graph Convolutional Networks, Sth International Conference on Learning Representations (ICLR), 2017. 16
[23] Kipf, N., Fetaya, E.,, Wang, K., Welling, M., and 7emel, R. S. Neural Relational Inference for Interacting Systems. In Proceedings of the 35th International Conferencee on Machine Learning (ICML., 2018. 1, 1, 4, 8, 16, 19, 22
[24] Kofinas, M., Nagaraja, N. S., and Gavves, E. Roto- coto-trannslted Local Coordinate Frames For Interacting Dynamical Systems. In Advances in Neural Information Processing Systems 34 (NeurIPS), 2021. 1, 1, 4, 5, 6, 7, 10, 14, 15, 17, 19, 20, 22, 30
[25] Li, Y., Tarlow, D., Brockschmidt, M., and Zemel, S. Gated Graph Sequence Neural Networks. In 4th International Conference on Learning Representations (ICLR), 2016. 3,5, 14
[26] Luo, S., Li, J., Guan, J., Su, Y., Cheng, C., Peng, J., and Ma. Equivariant Point Cloud Analysis via Learning Orientations for Message Passing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 2,
[27] Maddison, C. J. Mnih, A., and Teh, Y. W. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In 5th International Conference on Learning Representations (ICLR), 2017. 15
[28] Mescheder, L... Oechsle, M. Niemeyer. M., Nowozin, S., and Geiger. A. Occupancy Networks: Learning 3D Reconstruction in Function Space. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019. 7
[29] Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. NeRF: Represent- ing Scenes as Neural Radiance Fields for View Synthesis. Communications of the ACM, 65(1):99-066
2021. 6,7 7
[30] Morchead, A. and Cheng, J. Geometry-Complet Perceptron Networks for 3D Molecular Graphs, AAAI Workshop on Deep Learning on Graphs: Methods and Applications, 2023. 2
[31] Park, J. J., Florence, P., Straub, J., Newcombe, R., and Lovegrove, S. DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 7
[32] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.,, Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S.. Steiner, B., Fang, L., Bai, J., and Chintala, PyTorch: An Imperative Style, High- PegghPPeffrmancce Deep Learning Library. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. 18
[33] Perez, F., Strub, F., De Vries, H., Dumoulin, V., and Courville, A. FILM: Visual Reasoning with a General Conditioning Layer. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. 3, 5, 15, 22
[34] Rahaman, N., Baratin, A., Arpit, D., Draxler, F., Lin, M., Hamprecht, F., Bengio, Y., and Courville, A. On the Spectral Bias of Neural Networks. In Proceedings of the 36th International Conference on Machine Learning (ICML), 2019. 7
[35] Raissi, M., Perdikaris, P., and Karniadakis, G. F. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlincar partial differential equations. Journal of Computational physics. 378:686-707, 2019.
[36] Ramachandran, P., Zoph, B., Le, Q. Searching for Activation Functions. In 6th International Conference on Learning Representations, (ICLR), 2018. 5, 14, 15, 17
[37] Rezende, D. J., Mohamed, S., and Wierstra, D. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31st International Conference on Machine Learning (ICML), 2014, 4, 6, 7, 14, 15
[38] Romero, D. W. and Lohit. S. Learning Partial Equivariances from Data, In Advances in Neural Information Processing Systems 35 (NeurIPS), 2021.
12

---

[39] Salzmann, T., Ivanovic, B., Chakravarty, P., and Pavone, M. Trajectron++:: Dynamically-Fesibeee Trajectory Forecasting With Heterogeneous Data. In European Conference Computer Vision (ECCV), 2020.
[40] Sanchez-Gonzalez, A., Godwin, J., Pfaff, T., Ying, R., Leskover, J., and Battaglia, P. W. Learning to Simulate Complex Physics with Graph Networks. In Proceedings of the 37th International Conference on Machine Learning (ICML), 2020.
[41] Satorras, V. G., Hoogeboom, E., and Welling, M. E(n) Equivariant Graph Neural Networks. In Proceedings of the 38th International Conference on Machine Learning (ICML). 2021. 1, 9, 10, 18, 19
[42] Scarselli, F., Gori, M.. Tsoi, A. C.. Hagenbuchner, M., and Monfardini, G. The Graph Neural Network Model. IEEE Irunsactions on neural networks, 20(1), 2008. 3
[43] Schütt, K., Kindermans, P., Felix, H. E. S., Chmiela, S., Tkatchenko, A., and Müller, K. SchNet: Continuous- filter convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems 30 (NIPS), 2017. 6
[44] Sitzmann, V., Zollhöfer, M., and Wetzstein, Scene Representation Networks: Continuous 3D-Structure- Aware Neural Scene Representations. In Advances in Neural Information Processing Systems 32 (NeurIPS), 2019. 7
[45] Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G. Implicit Neural Representations with Periodic Activation Functions. In Advances in Neural Information Processing Systems 33 (NeurIPS), 2020.
[46] Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Kar S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., and Ng. R. Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains. Advances in Neural Information Processing Systems 33 (NeurIPS), 2020. 5,
[47] Thomas, N., Smidt, T., Kearnes, S., Yang, L., Li, L., Kohlhoff, K., and Riley, P. Tensor field networks: Rotation-ant and translation -qquiviion--ivviiiatttt neural networks for 3D point clouds. arXiv preprint 182..0880,,,8219 2018. 6
[48] van der Ouderaa, T. F., Romero, W., van Wilk, M. Relaxing Equivariance Constraints with Non-stationary Continuous Filters. In Advances in Neural Information Processing Systems 35 (NeurlPS), 2022. 6
[49] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, A. N., Kaiser, L., and Polosukhin, L Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (NeurIPS), 2017. 5, 7, 14
[50] Walters, R., Li, J., and Yu, R. Trajectory Prediction using Equivariant Continuous Convolution. In 9th International Conference Learning Representations (ICLR), 2021. 1,6
[51] Wang, R., Walters, R., and Yu, R. Approximately Equivariant Networks for Imperfectly Symmetric Dynamies. In Proceedings the 39th International Conference Machine Learning (ICML) 2022. 5, 6
[52] Wang, X. and Zhang, M. Graph Neural Network with Local Frame for Molecular Potential Energy Surface. In Learning Graphs Conference (LoG), 2022. 2
[53] Worrall, D. E.,, Garbin, S. J., Turmukhambetov, D., and Brostow, G. J. Harmonic Networks: Deep Translation and Rotation Equivariance. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017,
[54] Xie, Y., Takikawa, T., Saito, S., Litany, O., Yan, S., Khan, N., Tombari, F., Tompkin, J., Sitzmann, V., and Sridhar, S. Neural Fields in Visual Computing and Beyond. In Computer Graphics Forum, 2022.
[55] Xu, C., Tan, R. T., Tan, Y., Chen, S., Wang, Y. G., Wang, X., and Wang, Y. EqMotion: Equivariant Multi ageti- Motion Prediction with Invariant Interaction Reasoning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 21
[56] Zhuang, P., Abnar, S., Gu, J., Schwing, A., Susskind, J. M., Bautista, M. Á. Diffusion Probabilistic Fields. In / /th International Conference on Learning Representations (ICLR), 2022. 7
13

---

A Implementation details
A.1 Aether
Here we present the full Aether architecture. We first describe the details the neural field used for field discovery, and then we describe graph network formulated as variational autoencoder [21, 37].
A.1.1 Neural field
In its general form, the neural field takes as inputs query states that comprise positions p E R orientations w E SO(d), and velocities u E RR as well as a latent code E R: used to condition the field, and predicts latent forces the query states. Thus, it is defined as f R SO IR xII. Ed where d {2,3]. 3). Depending on the task hand, can omit the latent code z, e.g. if are modelling static field, or the orientations w, if we have prior knowledge that the field is independent them.
Encoding positions We encode the query positions using Gaussian random Fourier features [46], (P) (2o(2BBBp) sin Bp)] where B E R 4 xd is matrix with entries sampled from Gaussian distribution, Bąr ~ (0, 2²2). Throughout the experiments, and unless otherwise specified,
we use unit variance x² 1, and 2 256. Thus, the encoded positions have dimension of 512.
Encoding orientations In dimensions, for the orientations w (0, vb) T, we follow [24], and use the angles of the velocity vectors as proxy. We represent each angle in w as unit vector, @ [cos sin sin and encode them with linear layer 8(@) W. where W., E 2 dimensions, we use the same encoding, except that we now have a single angle w = 0.
In
Encoding velocities For velocities, we simply encode them using a linear layer ((u) = W. u, where W. E RRP Pex, Finally, we concatenate the encoded positions, orientations, and velocities in a single vector before feed them input to the neural field.
The latent code z "summarizes" the input graph such that isolates global field effects. We employ simple global spatio-temport attention mechanism, similar to Li et al. [25], that aggregates the input system in a latent vector representation. First, we define object embeddings O GRU(WW xx where W, Poxd is a matrix used to linearly transform the inputs, and
Latent code
GRU is the Gated Recurrent Unit [7]. We also define temporal embeddings PE(t), where PE
positional encodings [49], defined
PF(()2: = sin (/110000//D))
(11)
PE(()2+1 cos (0o$t//10000 (/1000022/D.)
(12)
where is the i-th dimension.
The aggregation is then defined as follows:
softmax(fa(s)) fo(), with of = [x|, o,
(13)
where fa : D. + R. fo R. + RD are 2-layer MLPs with SiLU activations [36] in-between. They can be summarized as:
fa [Linear(D,, D,) - SiLU + Linear (inear(Dr,11 1)}, fo [Linear( Ds, D2) + SiLU + Linear(D, D,))
(14)
(15)
In experiments, we use D. 512, D, = 512, and D, = D, + 2d 516 in d dimensions, or D, 518 in d = dimensions.
14

---

Neural field conditioning We condition the neural field using FiLM [33]. Following the imple- mentation details of FiLM, in practice, we use the following equation for a FiLM layer:
h' (h, z) (1 a(z)) c h B(z),
(16)
where h is the encoded imput in the first FiLM layer, or the conditioned input in subsequent FiLM
layers, and a : R. + R S. 3: R. - R th are MLPs. This equation deviates slightly from Section 2, since it predicts the residual of multiplicative modulation. This approach can be beneficial during the carly stages of training, since it initially defaults to an identity transformation for zero-initialized weights, while the alternative "zero out" the network outputs. For both and (3 we use 2-layer MLPs with SiLU activations in- between.
a = [Linear( D,, Dp) + SiLU + Linear Dn, Dn)} 3 = [Linear(Dr, Db) SiLU Linear Dn, Dn) Unless specified otherwise, in all experiments, we use Dn = 512.
(17)
(18)
Full neural field The full neural field a 3-layer MLP with SiLU [36] activations in-between, and FiLM layers after the first two linear layers, and outputs latent force field. The neural field can be summarized as:
f(v z) [Linear - FiLM SiLU Linear -> FiLM SiLU Linear).
(19)
A.1.2 Aether as variational autoencoder
Here we present our graph network architecture that closely follows Graber and Schwing [14], Kofinas al. [24]. The model formulated as variational autoencoder [21, 37] with latent edge types that infers latent graph structure. The encoder is tasked with predicting interactions between object pairs, while the decoder uses the sampled graph structure to make predictions. As mentioned in Section 3.3, our full architecture also integrates G-LoCS, i.e. the augmented node states include the predicted forces exerted at the target node, as well as the state of the auxiliary origin-node, expressed in the local frame of the target node.
Encoder Equations (20) to (22) describe the message passing steps of our graph network. these equations, we process each timestep independently. Then, in Equations (23) and (24) we compute evolution of edge embeddings over time with LSTMs [16], and Equations (25) and (26) we estimate the posterior and the learned prior over our edges.
h 33
G
YOil)
(20)
h (1).e = p!(
g0
Nis
voia 1)
+
1
Inge
ma
(21)
[V(1))
JEN()
h (2),t = )!2) (In
(1).
(1).e
b (
(22)
já
h prioo = LSTM prior
(2). 1-1 (1.)).
(23)
h/ (.n)..nc = LSTM
h s+1 (..)).me))
(24)
ps (= 21:111 softmax (Sproo (his.))
(25)
g (zz)../2z) softmax fenc [ML.. p kmne) )))
(26)
The functions f!!)) (() f(2) (
(I) fpriio fene denote MLPs.
Decoder The decoder samples z(,,) using Gumbel-Softme [27, 19]. The following equations formalize message passing scheme performed for the current timestep, and another one performed for the hidden node states. Both are used update the hidden node states, and to make predictions for the next timestep. As mentioned in Section 2, we make predictions in the local coordinate frame
15

---

of each node. Thus, we perform an inverse transformation for each node to transform the predictions back the global coordinate frame.
([rino
YOy 1)
mj.
Σ
Fije
Vijo
fils
(27)
m f( g.50) (viicc fijs vou1))
Emj
(28)
WOl
JEN()
h Σ Go ((b),
(29)
n [W(1)
(30)
JEN(()
h GRU ([n): mi'), ht))
(31)
1 x + R 114) (hat2)
(32)
p(x 1:t _21 NV (1 [1 I))
(33)
Our GRU [1] is identical to the one used [23].
Training Our full VAE model is trained by minimizing the negative Evidence Lower Bound (ELBO), which comprises the reconstruction loss of the predicted trajectories (positions and velocities) and the KT. divergence.
( 0) = Eys xx) Pog (x|=)] (L
(34)
Following Graber and Schwing [14], the reconstruction loss and the KL divergence take the following form:
E
= =-Σ 20
log loog(2ro*)) (2TT²))
(35)
2
(L(g((<xx)\lpp((/bs)] =
Σ c x) log ;
(36)
where HH denotes the entropy operator. In all experiments, set the variance 0² 10-5 We train Aether using Adam [21]. Unless stated otherwise, all experiments, we use a learning rate of 5e-4.
A.1.3 Aether architecture in Lorentz force field setting
The Lorentz force field setting, proposed by [10] uses only single timestep input and the task is to predict the positions for a single timestep in the future. Thus, we have to modify our architecture for this setting. This way, we also ensure fairer comparison with other methods. We do not use an NRI [22] or dNRI [14] backbone in this setting.
h0 (())
Vilés
fAs
Telss
||rs,:/122 s2llle)
(37)
f!)) 6,th
9v
Vilas fal
+ [N(1)]
Σ hys JEN(()
(38)
1 0 3a
[n! (1( h
(1-1) hn n
(39)
0 [
h (1-1) +
[(1i Σn
(40)
JEN(
p: p. R; fo (h!
(41)
16

---

with / E (,..... LJ. We use 4 layers, L.e. -= 4. The functions 1. denote 2-layer MLPs with SiLU [36] activations after each layer. The functions !! denote 2-layer MLPs with SiLU activations in-between, and doubling the dimensionality in-between. Finally, Iu denotes a linear layer. while fo denotes a 3-layer MLP with SiLU activations in between. Following [10], we use hidden dimension of 64. For this experiment, use a learning rate of le 3.
f [Linear + SiLU + Linear)
(42)
f [Linear SiLU + Linear SiLU}
(43)
fo [Linear + SiLU + Linear - SiLU + Linear]
(44)
For the neural field, we use the input positions and velocities as input, well as the particle charges, i.e. f= (p, u. q). We do not use any input encoding for positions or velocities in this setting, but we use embedding for the charges that maps them to 16 dimensions. We concatenate the charge embeddings with positions and velocities and feed them as input to the neural field. The neural field is a 3-layer MLP with SiLU activations in-between. We use hidden dimension of 32 in the neural field.
f = [Linear + SiLU - Linear - SiLU - Linear}
(45)
A.2 G-LoCS
Encoder
h (1)a f!l) ((v) Vojo vorr))
3,
(46)
h (1) f("))
9.60) (.. ([vi)e ([ + W(OI)
(47)
JEN())
h 5 f!2)
[ (1).e h
a . h 1
(48)
h .rior
LSTM prior
(n (2) h (.dt.
(49)
h (ns), LSTM
(2).c h (.0). ji
(50)
Pe x=' 1: 2 11-11) softmax Sprior h ((sr))))niir
(51)
9o (2),,1xx) softmax feas ((..))poo b(,)),o 1)
(52)
Decoder
m Σ E.stert ([viie Vhio vokl)
(53)
mf = f(3) gt.)) ([virr vbil)
1
m/
(54)
[V(0)l
JEN())
h/ Σ En ((h). ht])
(55)
nf = W(O
Σ ha JEN())
(56)
h. = GRU(( m; h()
(57)
1 41 = x + R (b((tt))
(58)
p(x++1
1xt
,2 21:1)= = (utt _0*1)
(59)
A.3 Source oracle
The source oracle modifies LoCS [24] to use virtual nodes. Since graph networks are permutation invariant, cannot just include the sources nodes of the graph and perform message passing,
17

---

as the network would not be able to distinguish particles from sources. Thus, we treat the sources separately in the message passing so that the network can identify them. More specifically, we introduce new message function that computes field source -> particle messages. Furthermore, we introduce a separate aggregation function in the update step that only aggregates the messages from field sources. We denote the set of field sources as S. The state of a field source E S is denoted as v,, while the same state expressed in the local coordinate frame of node is denoted as Valé- The source oracle graph network is defined as follows:
h5, = fe ((vi Vaje )) h/,5 = La (L((v vije )
(60)
(61)
Ax = So (vis) + (W(i)
bie JEN((()
+ S||
Σ h
(62)
JES
where fa is an MLP. We predict future states for all the "observable" particles, but not for the "source" particles.
A.4 Parallel aether architecture
(1) fe ([V3/t Valas 9:93, [|15)
(63)
1,th f!
g0 (vaa)) + (V(0) JEN())
1
(64)
hg 1
[n 111 h
1 n2 1)
(65)
h 1
h (1-1)
[W(i)] Σ h0
(66)
JEN(1)
p. P. + R, fo + f,
(67)
A.5 Aether with EGNN backbone
Our method is agnostic to the choice of equivariant graph network; we expect that it would be beneficial for a number of strictly equivariant networks. To test this hypothesis, we combine EGNN [41] with our method; starting from the velocity formulation of EGNN, modify the message and velocity equations to incorporate the predicted forces for each node, as follows:
mj.i = de [h:,
h
Ip'
p ajs f. F3
1
(68)
u = dv h fz
u
+ C
Σ (p') - -P!) dz (,
(69)
jz
P = =p + u [+1
(70)
m: Σ mji
(71)
jEA(()
h [+1 = (h m.).
(72)
The remaining EGNN components remain unaltered.
A.6 Computing resources
All experiments were performed on single GPUs. We used 2 different GPU models, namely the Nvidia RTX 2080 Ti, and Nvidia GTX 1080 Our source code was written in PyTorch [32], version 1.4.0, and CUDA 10.0.
18

---

B Dataset details
B.1 Electrostatic field
Kipf et al. [23] introduced a dataset of interacting charged particles. Charged particles interact via electrostatic Coulomb forces. We assume set of N particles, and each particle has a position P E IR' and a charge E R. The force fi exerted from particle to particle is computed as
follows:
P/. Pj P.:
(73)
PIA D
(74)
|
f/. = C 4:gp
(75)
||PS
Since forces only depend on positions at the current timestep, in the following equations, we omit the time indices to reduce clutter. The total force exerted particle is:
N
f,
Σ j=L.jæfi
f3.
C.
Σ
(76)
|
j= Lig
The electric field is a vector field, whose value at the test position assumes a positive test charge qi and is defined as:
Dj
E =
-C.
(77)
|p..il 2
N
N
E =
Σ J=L
E
C.
Σ
P
(78)
|P
j=1
Our first experiment aims to study the effect of static fields, i.e. a single field across all train, validation, and simulations. We extend the charged particles dataset by adding number of immovable sources. Overall, these sources act like regular particles, exerting forces on the observable particles, except we ignore any forces exerted them, and fix their positions and velocities to zero. We use N = "observable" particles and M 20 "source" particles. In all experiments, we assume unit charges, qi +1, and C 1. The probabilities of positive or negative charges are equal. Then, the forces and the electric field can be simplified as:
D
fg. = sign(4:0)
(79)
|
D
E
Σ
sign(q;)
(80)
||
j=1,4$1
The net force exerted at particle i E [I....,N) is computed as:
N+M
N+M
f,
Σ
f=1=
Σ f
fyx
(81)
j=1
j=1.j
N+1
particles
Following Satorras al. [41], Fuchs al. [12], Kofinas al. [24], we remove virtual borders that cause elastic collisions. We generate dataset of 50,000 simulations for training, 10,000 for validation and 10,000 for testing. The datasets contains only the positions and velocities for the "observable" particles, while the field sources are only used for visualization. Following Kipf et al. [23]. each simulation lasts for timesteps. During inference, we use the first 29 steps as input and predict the remaining 20 steps.
19

---

Figure 7: Visualization of the static field in the electrostatic field setting
B.2 Traffic scenes inD
InD [4] real-world traffic scenes dataset that comprises trajectories of pedestrians, vehicles, and cyclists. It contains 33 recordings, recorded different locations in Aachen, Germany. We hypothesize that discovering latent traffic force field will be beneficial for trajectory forecasting in traffic scenes. For simplicity, we focus on static field discovery in traffic scenes. We create subset that contains scenes from a single location. Namely, we choose "Frankenburg, Aachen", since is the location with most interactions the dataset. The subset corresponds to 12 recordings; we use for training, 2 for validation, and 2 for testing. We follow a similar experimental setting with Graber and Schwing [14], Kofinas al. [24]. We divide each scene into 18- 18-step sequences. We use the first 6 time steps as input and predict the next 12 time steps.
B.3 Gravitational n-body dataset
In this experiment, we study the influence of dynamic fields, i.e. fields that are different across simulations. Similar to the electrostatic field setting, extend the gravitational n-body dataset by Brandstetter et al. [5] by adding gravitational sources. The equation that describes the forces is similar Equation (75). Namely, have
P |
fr = C. m,
(82)
where m,, m, are the particle masses. We create a dataset 50,000 simulations for training, 10,000 for validation and 10,000 for testing. We use N = 5 particles and M = source. We set the masses of particles to mp 1, while the source has a mass of m, = 10. Similarly to the electrostatic field experiment, the datasets contains only the positions and velocities for the "observable" particles, while the field source is only used for visualization. We generate trajectories 49 timesteps. We use the first 44 timesteps as input and predict the remaining 5 steps. All other dataset details are identical to Brandstetter al. [5].
B.3.1 2D gravitational n-body dataset
We also experiment with smaller variant of the dynamic gravitational fields, using a 2D setting. We create dataset of 5,000 simulations for training, 1,000 for validation and 1,000 for testing. All other
20

---

dataset details are the same with the full 3D dataset. We report results in Figure 21. We showcase predicted trajectories Appendix D.3, and the learned fields in Appendix D.3.I.
C Extra experiments
C.1 Alternative equivariant network backbones
To further test the applicability of our method, we combine it with different equivariant graph network backbones. First, combine method with GMN [18]. Similar to Equations (4) and (5), concatenate the predicted forces for each node with the message Zy in [Z,, f, fz]. The formulation above is further motivated by SGNN [15], which extends GMN by including gravity as external force term, as well as object-aware information (see appendix A.3 in [15] for a comparison). In our case, replace the gravity term with the predicted forces per node. Thus, instead of (Zja- ji::0 g). have [Zz; f,, f,]-
We train and evaluate GMN the Lorentz force field setting. We then add the force terms using the formulation above. show the results table below. Indeed, using Aether greatly enhances performance of GMN, which further enhances our hypothesis.
Table Ablation study on the choice of equivariant GNN backbone. Position prediction MSE on Lorentz force field.
Method
MSE (4)
GMN [18]
0,0365
+MN+ee (ours)
0.0261
Next, we integrate method in EqMotion [55], recent equivariant method with state-of-the-arr performance on trajectory forecasting. incorporate Aether Eq by treating the predicted forces as geometric features similar to velocities. Namely, after computing the forces for each object at each timestep, we compute the magnitudes the force vectors and the force angle sequence, i.e. the angles between forces in consecutive timesteps. We concatenate these quantities to the existing features for the feature initialization step.
We train and evaluate EqMotion and Aether with EqMotion on inD [4] following our experimental setup. We report the results in Figure 8. We see that Aether is beneficial even for state-of-the-an trajectory forecasting method, which further strengthens our claims.
EgMotion
EgMootion (Ouns)
Position Errors
Velocity Errors
Total Errors
0.25
1.0
0.20
0.8
Lg lime
0.4
Hiror
0.15
0.6
0.10
0.1
0.2
Step
Step
Step
Figure 8: Ablation study on the choice of equivariant GNN backbone. Results inD.
C.2 Non-equivariant network with neural field
Our ability to capture global components with neural field stems from our overall architecture, which promotes disentanglement, Using a graph network that respects the underlying symmetries and has inductive bias towards using local interactions, iee. any equivariant graph network, allows
21

---

the neural field to "solve for" the global components, by "subtracting" the local interactions from the observable net effects.
The choice of an equivariant network is crucial here: a non-equivariant graph network like NRI [23] dNRI [14] would merely gather "redundant" information from the neural field. We demonstrate this mathematically in the following equations for the static field, in which we first compute the force f at a target position p, and then compute the node embedding using equations from NRI/CNRI, including the forces.
f(p) = (LP: h s(p, f. (L,
(83)
(84)
We can see that the node embeddings depend on positions twice, one explicit and one through another MLP, an architecture similar a concatenated residual connection. In this case, we do not expect the neural field to isolate global forces, or to be helpful for future forecasting. We test this hypothesis with an ablation experiment on the electrostatic field setting, by combing our neural field with dNRI, instead of an equivariant network. In Table 5, we report the MSE at the final prediction timestep, ie. MSE@20 20. We can see that adding a neural field to a non-equivariant network does not enhance performance, and in fact, it results in performance degradation, which enhances our hypothesis.
Table Ablation study on the suitability of non-equivariant networks with Aether. Combining dNRI -a non-equivarian graph network- with a neural field does not enhance performance. Results on the electrostatic field setting.
Method
MSE@ 20 (J)
dNRI [14]
1.20
dether
1.37
Aether
0.69
C.3 Choice of conditioning mechanism
FiLM [33] is used in the dynamic field setting to condition neural fields. To examine its influence on performance, we perform ablation study the 3D gravitational setting, where we replace FiLM layers with conditioning by concatenation, a very simple and successful conditioning mechanism. We term this model Concal Aether, and report the results in Table 6. We see that conditioning by concatenation underperforms, scoring almost on par with LoCS. This is perhaps expected, since concatenation is a rather weak form of conditioning, and our task is very challenging. On the other hand, FiLM is powerful mechanism and is able to condition effectively.
Table 6: Ablation study the choice of conditioning mechanism. Results the 3D gravitational field setting.
Method
MSE@5 (L)
LoCS [24]
0.1308
Aether
0.0660
Concat Aether
0.1474
22

---

D Qualitative results
D.1 Electrostatic field
Figure shows qualitative results on the electrostatic field setting.
(a) Acther
(b) G-LoCS
(c) LoCS
(d) dNRI
Figure 9: Predictions on the electrostatic field setting. Lighter colors indicate predictions, and darker colors indicate the groundtruth. Predictions start where markers have black edges. Markers get bigger and more opaque as trajectories evolve time. The background streamplots indicate the groundtruth field, and are not given input to the networks. Similarly, the blue @ markers and the red e markers, are merely shown for illustrative purposes, indicating the charges the field sources, and are not given input to the networks. Best viewed in color.
D.1.1 Discovered electrostatic field
In Figure 10 we visualize the discovered electrostatic field compared to the groundtruth one.
Predicted Field
Groundtruth Field
Figure 10: Learned Field (left) in electrostatic field setting compared to groundtruth (right).
23

---

D.2 InD
Figure 11 shows qualitative results of our method on inD [4]. Figures 12, 13 and 14 show qualitative results for G-LoCS, LoCS, and dNRI, respectively.
(a)
(b)
Figure 11: Aether predictions (right) on inD, compared to groundtruth (left). Predictions start where markers are colored black. Best viewed in color.

---

Figure 12: G-LoCS predictions (right) on inD, compared to groundtruth (left). Predictions start where markers are colored black. Best viewed in color.
25

---

Figure 13: LoCS predictions (right) on inD, compared to groundtruth (left). Predictions start where markers are colored black. Best viewed in color.
26

---

Figure 14: dNRI predictions (right) on inD, compared to groundtruth (left). Predictions start where markers are colored black. Best viewed in color.
D.2.1 Discovered traffic force field
In Figure 15 we visualize the discovered traffic force field on inD. the supplementary material, we provide vidco visualizations of the learned field, with input orientations evolving over time.
27

---

Figure 15: Discovered field on inD [4]. For simplicity, we only visualize the field for discrete input orientations in C₁ {0, n. n, 2 }. Best viewed in color.
3xt
D.3 2D gravity
Figure 16 shows qualitative results on the 2D gravitational n-body problem.
(a) Acther
(b) G-LoCS
LoCS
(d) dNRI
Figure 16: Predictions on gravity. Lighter colors indicate predictions, and darker colors indicate the groundtruth. Predictions start where markers have black edges. Markers get bigger trajectories evolve. Best viewed in color.
D.3.1 Discovered 2D gravitational fields
Figure 17 shows examples of discovered fields compared to the groundtruth ones.
28

---

Predicted Field
Groundtruth Field
Predicted Field
Groundtruth Field
Figure 17: Learned dynamic fields (left) in 2D gravitational field setting vs groundtruth (right).
29

---

E
Quantitative results
In all settings, we report the total errors, i.e. the mean squared errors of positions and velocities
over time, E(t) ND En |x - & 112 Following Kofinas et al. [24], we also separately
1
report the Lg norm position errors, Ep(t) = N > - - -2ll2- and velocity errors, Et) =
1 N E 12
N
E.1 Electrostatic field
dNRI
LoCS
G-LoCS
Acther
Pusition Errors
Velucity Errors
Total
1:25
0.8
0.6
1.00
11 Eavo
0.6
0.4
0.75
0.4
0.50
0.2
0.25
Step
Step
Figure 18: Results in the electrostatic field setting.
E.2 InD
aNRI
Aether
G-LaCS
LoCS
Position Errors
Velocity Emors
Total Errors
2.0
0.5
2.5
0.4
2.0
1.5
6 Etrer
Lg Error
0.5
0.1
0.3
0.0
0.0
0.0
9
11
12
11
12
Step
Step
Step
Figure 19: Results in inD.
30

---

E.3
Gravity
dNRI
LaCS
G-LeCS
Aether
Position Errors
Velocity Ervors
Total Errors
0.08
0.3
0.04
5, liter
F. Ertor
0.03
0.2
MSE
0.02
0.05
0.1
0.01
0.00)
0.0
0.00
Step
Step
Step
Figure 20: Results the dynamic gravitational field setting.
E.4 2D gravity
aNRI
Aether
G-LoCS
+
LaCS
Position Errors
Velocity Ervors
Total Errors
0.30
1.5
0.20
5, Etter
MSE
0.15
0.10
0.05
0.00
0.0
Step
Step
Step
Figure 21: Results in the dynamic 2D gravitational field setting.
E.5
Significance of the discovered field
Partile Oracle
Force Oracle
Aether
Position Errors
Velocity Errors
Total Errors
0.5
1.0
0.6
5o EEter
0.3
0.6
0.4
0.2
0.4
0.1
0.0
0.0
13
20
13
17
5
20
Step
Step
Step
Figure 22: Ablation study on the significance of the discovered field. Results in the electrostatic field setting.
31


Reviews:
[{"review_id": "vZgeOLnUQt", "reviewer": "Reviewer_hUPP", "summary": "This paper proposes to learn the latent fields governing the dynamics of interacting objects in a system without observations of the field. The authors propose unconditional and conditional latent field models to learn the static and dynamic fields respectively. The learned latent field signals are represented in local coordinate representations and treated as additional node features in graph learning to predict future node trajectories. Specifically, the paper adopts a recent VAE framework to make predictions in an autoregressive way.", "questions": "Please see comments above.", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The general idea of discovering latent neural fields is very interesting and plays an important role in modeling many real-world problems and scientific tasks.\n* The proposed idea looks simple yet effective in learning the latent fields. It is flexible and potentially can be applied to many graph learning frameworks.\n* The quantitative and qualitative results showcase the proposed idea and clearly show the performance gains compared with the previous works.", "weaknesses": "Unclear notations:\n* The shape of different variables are often not explained clearly. It’s not easy to infer from the context and therefore not self-contained.\n* Eq (1): how to understand the shape of angular position $\\boldsymbol{Q(w_i^t)}$ and $\\boldsymbol{\\tilde{R}(w_i^t)}$? The notations do not seem to be self-contained.\n* Eq (2) - (3) what are the definitions of $\\boldsymbol{h_{j,i}^t}$?\n* Definition of angle in L169 uses the symbol $\\boldsymbol{\\omega}$, is it the same as in L74?\n* L94 uses the symbol $\\boldsymbol{h}$, is it related to h in Eq (2)?\n* Latent code $\\boldsymbol{z}$ and the latent graph edge type $\\boldsymbol{z}$ in the appendix are using the same letter, which is very confusing.\n\nL44 and below: In the example of the N-body system, if the E(3) symmetry holds, then SE(3) symmetry must hold since SE(3) is a special case of E(3) without reflection. To the reviewer, this example doesn’t explain the pitched point that SE(3) equivariance doesn’t always hold.\n\nAbout the combination of latent field learning and VAE:\n* The proposed models use latent edge-based VAE to predict the future node states. It’d be better to contain the VAE formulation in the main paper briefly to be self-contained.\n* The idea of learning latent fields can be applied in non-VAE frameworks for trajectory predictions as well. It’d be more convincing to show these experimental results.\n\nAbout neural field modeling.\n* How important is the assumption of static and dynamic fields for learning? Can the model learn different fields in a data-driven manner?\n* To the reviewer, either conditional or unconditional neural fields modeling methods can be used to learn both static or dynamic fields. It’s not convincing without ablation experiments to justify the model design w.r.t. types of fields.\n* L155-L162: the naming of unconditional and conditional neural fields look confusing to the reviewer, as both are functions of input states essentially. Although they are different in architecture design, both are neural functions of the input trajectories.\n* Figure 3 shows the conditional neural field setup for dynamic fields, while most of the experiments are using static fields. To the reviewer, this is a bit misleading and does not showcase the proposed model properly. Please add both static and dynamic fields setup in the figure.\n\nAbout G-LoCS baseline:\n* How is the artificial velocity defined in the experiments? The details are vague to the reviewer.\n* This method is essentially an additional baseline with node feature augmentation and is not a novel contribution to the reviewer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insightful comments. We appreciate that they find the idea of field discovery very interesting, and our method simple and effective. Please find our answers below.\n\n```\nUnclear Notation\n```\n- We describe most shapes in detail in appendix A. We will incorporate the variable sizes in our equations for the updated version of the manuscript.\n- The matrix $\\mathbf{Q}$ constitutes the matrix representation of the angular position $\\boldsymbol{\\omega}$, i.e. it is a rotation matrix in SO(d), d=2, 3. Hence, depending on the dimensionality of the problem, it is a $2 \\times 2$ or $3 \\times 3$ matrix. The matrix $\\mathbf{\\tilde{R}}$ is the direct sum of three rotation matrices $\\mathbf{Q}$, i.e. it is a block diagonal matrix where each block is a matrix $\\mathbf{Q}$. Its dimensions are $6 \\times 6$ in 2 dimensions or $9 \\times 9$ in 3 dimensions.\n- $\\mathbf{h}^t_{j,i}$ are the intermediate edge features (from node $j$ to node $i$) or “messages” in the message passing network nomenclature. The superscript $t$ denotes the actual timestep.\n- Yes, the definition of angle or angular position $\\boldsymbol{\\omega}$ in lines 74 and 169 is the same. In line 169, we define $\\boldsymbol{\\omega}$ for the more general case of 3 dimensions; in 2 dimensions we only have one angle and $$\\boldsymbol{\\omega}$  = \\theta$.\n- Indeed, we use the same letter $\\mathbf{z}$ to describe two different variables, the latent code and the latent graph edge type. We will fix the notation for the camera ready version of the paper.\n\n```\nIn the example of the N-body system, if the E(3) symmetry holds, then SE(3) symmetry must hold since SE(3) is a special case of E(3) without reflection. To the reviewer, this example doesn’t explain the pitched point that SE(3) equivariance doesn’t always hold.\n```\n\nThe point here is that the strict E(3) symmetry does not always hold, because it might be violated by external force fields, which are either unknown or not subject to transformation. As an example, an n-body system might evolve under the influence of an unobserved black hole that generates a gravitational field. Then, the observed dynamics would not be strictly equivariant any more, as we cannot transform the unobserved quantities.\n\n```\nThe proposed models use latent edge-based VAE to predict the future node states. It’d be better to contain the VAE formulation in the main paper briefly to be self-contained.\n\nThe idea of learning latent fields can be applied in non-VAE frameworks for trajectory predictions as well. It’d be more convincing to show these experimental results.\n```\n\nThank you for the suggestion. The VAE formulation is orthogonal to our method; it serves as an implementation detail, and the method can also work without the VAE component. Case in point, our model in the Lorentz force field setting does not include a VAE (this also enables a fairer comparison with other baselines). See appendix A.1.3 for the non-VAE implementation of our method. For the camera ready version, we will clarify that the VAE is orthogonal to our method, and highlight that non-VAE is also possible.\n\n\n```\nTo the reviewer, either conditional or unconditional neural fields modeling methods can be used to learn both static or dynamic fields. It’s not convincing without ablation experiments to justify the model design w.r.t. types of fields.\n```\n\nUnconditional neural fields cannot be used to learn dynamic fields, since they would only learn an average force field across the dataset. On the other hand, a conditional neural field could be used to learn a static field. In that case, the neural field should learn to ignore the latent vector $\\mathbf{z}$, since the generated field should be identical regardless of the input system; we expect its performance to match the unconditional field. We verify this hypothesis with an ablation study on the Lorentz force field, where we train and evaluate our method using a conditional field. The MSE is now 0.0131, while the original unconditional method had an MSE 0.0129.\n\n```\nL155-L162: the naming of unconditional and conditional neural fields look confusing to the reviewer, as both are functions of input states essentially.\n```\n\nWe borrow the terminology of conditional and unconditional neural fields from [1]. Unconditional neural fields are not functions of the input states, on the contrary, they are independent of the input states. We can sample the neural field in arbitrary positions; in practice we sample it at the input states because we have supervision there. On the other hand, the conditional neural field is a function of the input states, since they are used to generate the latent vector $\\mathbf{z}$ used to condition the neural field. We will clarify terminology in the introduction.\n\n```\nAbout G-LoCS baseline, how is the artificial velocity defined in the experiments?\n```\n\nAs we mention in the background section of the paper, LoCS [2] uses velocities as a proxy to estimate angular positions. G-LoCS introduces an auxiliary node that corresponds to the global coordinate frame. We define its velocity to match the x-axis, e.g. $\\mathbf{u} = (1,0)^T$ in 2 dimensions, such that we can compute a non-degenerate angular position (frame).\n\n```\nThe G-LoCS baseline is essentially an additional baseline with node feature augmentation and is not a novel contribution to the reviewer.\n```\n\nG-LoCS is an approximately equivariant graph network that retains the strengths of equivariance through local coordinate frames, but it can also capture global phenomena through the global coordinate frame. We consider it to be a conceptually simple yet novel approximately equivariant graph network with minimum computational overhead, as well as an additional baseline.\n\n### References\n\n[1] Xie, Yiheng, et al. \"Neural fields in visual computing and beyond.\" 2022.\n\n[2] Kofinas, Miltiadis, et al. \"Roto-translated local coordinate frames for interacting dynamical systems.\" NeurIPS, 2021."}, {"review_id": "HMt4r3ByAj", "reviewer": "Reviewer_8BnH", "summary": "The main assumption of this paper is that local object interactions generally happen within external global fields. Based on this assumption, the paper then proposes to disentangle the local from the global interactions, respectively, by learning both an equivariant graph network, capturing the local interactions, and a neural network, capturing the global interactions. The latter network modifies the behavior of the graph network. The experimental results of the paper demonstrate that one can accurately discover the underlying fields in charged particles settings, traffic scenes, and gravitational n-body problems, and effectively use them to learn the system and forecast future trajectories.", "questions": "Is it possible to generalize the ideas in this paper to a structuring mechanism relevant to a more general class of ML applications?", "limitations": "This work only considers fields that do not react to the observable environment.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of considering local interactions happening in the presence of global fields is an interesting one, and seems to be new. So is the approach, learning the external fields, and the experimental validation demonstrate an increased accuracy of objects-trajectory prediction. ", "weaknesses": "The results of this work seem to be confined to physics applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for acknowledging the novelty of our idea and approach. Please find our answers below.\n```\nIs it possible to generalize the ideas in this paper to a structuring mechanism relevant to a more general class of ML applications?\n```\nThe notion of the field is tightly linked with a global phenomenon that occurs in the continuous (Euclidean) space. This space could be underlying a geometric structure such as a geometric graph or an image grid, or be a continuous relaxation of such a structure. \nWe can draw a lot of qualitative similarities with works on graph networks that explore virtual global nodes. For example, the work of Battaglia et al. [2] explored graphs with global attributes. While they can only model constant global attributes, our fields go a step further, since we now have a global function that can take different values based on the node states. In other words, global attributes can be seen as a special case of our method, one where our predicted field is constant everywhere.\nFurthermore, in [2], global properties of the graph can be derived from global attributes. Similarly, we expect to be able to derive graph properties from operating on the global neural field, by using a method that operates on neural fields, e.g. the recently popularized Functa [1].\n\nWe believe that this approach could be used to approximate and replace graph sub-structures or image grids, and we are excited to see the community explore this research direction.\n\n### References\n\n[1] Dupont, Emilien, et al. \"From data to functa: Your data point is a function and you can treat it like one.\" ICML, 2022.\n\n[2] Battaglia, Peter W., et al. \"Relational inductive biases, deep learning, and graph networks\". 2018.\n"}, {"review_id": "QccX7AlVeI", "reviewer": "Reviewer_DhsC", "summary": "In this paper, the authors have introduced a novel method, Aether, for discovering the latent field within dynamical systems in order to better model their dynamical evolutions. This is achieved by disentangle the interactions between objects and the effect of the external global field, conditioned on the states for the static field case or an extra latent state vector for the dynamic field case. The experiments on charged particles with external fields, Lorentz force field, traffic scenes, and gravitational field demostrate that Aether can learn to discover the latent external field and leverage the field information to better model the dynamics of the entire system.", "questions": "Q1. Most of the datasets used in the paper are artificial/simulation data. Are there any practical application of the field discovery method on more real-world data (besides the traffic data)? Adding more real-world datasets will largely enhance the credibility of the proposed approach.\n\nQ2. Even for the traffic data, it would be of great significance to compare the method with state-of-the-art traffic prediction approaches, while in this paper only weak baselines are considered for this dataset. It is unclear that to what extent the current method would impact these fields of practical applications.\n\nQ3. Results in Figure 4 may require additional justifications. The reported MSEs of all methods seem to be around 0.6 at timestep 20 on charged NRI dataset, while in the original paper [1] the numbers were around the scale of 1e-3 when timestep = 20. Are all models properly trained? Moreover, it would be helpful to include ground truth trajectories in the visualizations to see whether the models can offer visually reasonable predictions, as per previous works.\n\n[1] Kipf et al. Neural Relational Inference for Interacting Systems. 2018.\n", "limitations": "\nThe authors have sufficiently discuss the limitations of their method.\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper identifies an important problem of latent field discovery which is of good originality.\n\n2. The presentation of the paper is clear and easy to read. The method is well-designed and easy to follow.\n\n3. The experiments are carefully designed which cover a wide range of datasets including physical systems and even traffic scences data.", "weaknesses": "1. The experiments only comprise relatively simple scenarios and even methods compared against (see Q1 and Q2). It would be better to showcase the effectiveness and impact of the method on more challenging and complex datasets/benchmarks.\n\n2. Some experiment results seem confusing and might require further explanations (see Q3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insightful comments. We appreciate that they find our problem statement original and our method well-designed. Please find our answers below.\n\n```\nQ1. Most of the datasets used in the paper are artificial/simulation data. Are there any practical application of the field discovery method on more real-world data (besides the traffic data)? Adding more real-world datasets will largely enhance the credibility of the proposed approach.\n```\n\nWhile fields underlie many real-world systems, to the best of our knowledge, there are no established real-world datasets that include fields. As the community moves beyond strict equivariance, we expect that more and more real-world datasets that incorporate fields will be established.\n\n```\nQ2. Even for the traffic data, it would be of great significance to compare the method with state-of-the-art traffic prediction approaches, while in this paper only weak baselines are considered for this dataset. It is unclear that to what extent the current method would impact these fields of practical applications.\n```\n\nThank you for your suggestion. We perform a new experiment, where we integrate our method in EqMotion [5], a recent equivariant method with state-of-the-art performance on trajectory forecasting. We incorporate Aether in EqMotion by treating the predicted forces as geometric features similar to velocities. Namely, after computing the forces for each object at each timestep, we compute the magnitudes of the force vectors and the force angle sequence, i.e. the angles between forces in consecutive timesteps. We concatenate these quantities to the existing features for the feature initialization step.\n\nWe train and evaluate EqMotion and Aether+EqMotion on inD following our experimental setup. Here we report the final total error, i.e. the MSE at timestep 12. EqMotion has an MSE of 1.104, while Aether+EqMotion has an MSE of 0.962. We see that Aether is beneficial even for a state-of-the-art trajectory forecasting method, which further strengthens our claims.\n\n```\nQ3. Results in Figure 4 may require additional justifications. The reported MSEs of all methods seem to be around 0.6 at timestep 20 on charged NRI dataset, while in the original paper [1] the numbers were around the scale of 1e-3 when timestep = 20. Are all models properly trained? Moreover, it would be helpful to include ground truth trajectories in the visualizations to see whether the models can offer visually reasonable predictions, as per previous works.\n```\n\nThere are three very important differences between our electrostatic field experiment and the original charged particles dataset [1]. First, these are two different datasets; our electrostatic field dataset extends the original dataset by adding an electrostatic field. Trajectory forecasting in the electrostatic field is a much harder task, since the unobserved field can drastically impact future trajectories. Second, the y-axis between the two figures is actually different; both measure the MSE, but NRI reports the normalized error, i.e. the error between min-max normalized groundtruth and predictions, while we report the unnormalized error following [4], i.e. the error in the original data scale, to ensure fair comparison across different normalization functions. Finally, as we mention in appendix B.1, following [2, 3], we have removed virtual borders that cause elastic collisions.\n\nPlease see appendix D.1 for visualizations and comparison with groundtruth future trajectories, as well as videos in the electrostatic_predictions directory in the supplementary material. They demonstrate that our method make reasonable predictions.\n\n### References\n\n[1] Kipf et al. “Neural Relational Inference for Interacting Systems”. ICML, 2018.\n\n[2] Satorras, Vıctor Garcia, et al. \"E(n) equivariant graph neural networks\". ICML, 2021.\n\n[3] Fuchs, Fabian, et al. \"SE(3)-transformers: 3d roto-translation equivariant attention networks.\" NeurIPS, 2020.\n\n[4] Kofinas, Miltiadis, et al. \"Roto-translated local coordinate frames for interacting dynamical systems.\" NeurIPS, 2021.\n\n[5] Xu, Chenxin, et al. \"EqMotion: Equivariant Multi-agent Motion Prediction with Invariant Interaction Reasoning\". CVPR, 2023."}, {"review_id": "HLb1oWZGeJ", "reviewer": "Reviewer_HXpt", "summary": "In this paper, the authors address the often-overlooked influence of underlying field effects on the evolution of interacting systems, postulating the existence of latent force fields. They suggest utilizing neural fields for learning these dynamics. Equivariant networks, although commonly used, fail to capture the global information, a crucial shortcoming the authors aim to rectify. They propose to separate local object interactions (SE(3) equivariant and dependent on relative states) from the global field effects (dependent on absolute states).\n\nThrough the use of equivariant graph networks to model these interactions and their integration with neural fields, the authors construct a novel graph network to incorporate field forces. The authors' experimental results demonstrate their ability to successfully identify underlying fields within various scenarios, including charged particle settings, traffic scenes, and gravitational n-body problems. Furthermore, these findings prove the method's effectiveness in system learning and trajectory forecasting.", "questions": "There are a few open question which are key to the formulation of the training objective of the proposed approach. What are the reasons for choosing a deterministic training objective (eg. L2 reconstruction) for a problem that is inherently non-deterministic. Wouldn't a model that predicts a distribution over future trajectories be conceptually preferred to model that predicts a single point estimate? I am happy to update my score after discussing these questions with authors. ", "limitations": "Limitations and societal impact of the proposed approach is properly discussed", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The structure and presentation of the paper are appropriately executed. The authors have used clear and precise language, which aids in the understanding of the complex concepts discussed. This ease of readability can prove beneficial for readers across various levels of familiarity with the subject matter, contributing to the paper's usefulness.\n\nThe concept of entangled equivariance, as introduced in this work, is a novel approach in system dynamics. This method, which focuses on factorizing out the influence of global fields from local interactions, provides a fresh insight into the understanding of interacting systems. \n\nI really enjoyed the range of experiments conducted in the paper, involving diverse systems of interacting entities at different scales. The application of the proposed methodology across different contexts, such as charged particles, traffic scenes, and gravitational n-body problems, illustrates the method's broad applicability. ", "weaknesses": "I found it interesting that there's no  explicit mention of the training objective. This omission can potentially lead to ambiguity in understanding the complete process and methodology involved. For a comprehensive study, it is crucial to explicitly specify each component, including the training objective, to ensure a thorough understanding of the research.\n\nThe training objective appears to be a reconstruction term (given Fig. 3), which may prove problematic in non-deterministic like trajectory forecasting. As it currently stands, the proposed approach will always predict the same future trajectory given the same input trajectory, a characteristic that is incongruous with the inherent non-deterministic nature of forward predictions in time, especially when the complete dynamical system is not fully known and explicitly defined. In addition, the approach is evaluated primarily in the context of future trajectory prediction, which only accentuates these concerns. I\n\nLastly, there seems to be an omission in the citations pertaining to probabilistic models of neural fields. Specifically, works such as Functa [1] and Diffusion Probabilistic Fields [2] are not referenced. Including these references would enhance the completeness of the literature review and underscore the authors' comprehensive understanding of the existing body of research.\n\n[1] Dupont, Emilien, et al. \"From data to functa: Your data point is a function and you can treat it like one.\" arXiv preprint arXiv:2201.12204 (2022).\n\n[2] Zhuang, Peiye, et al. \"Diffusion probabilistic fields.\" The Eleventh International Conference on Learning Representations. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their insightful comments. We appreciate that they find our method novel and insightful with broad applicability. Please find our answers below.\n\n```\nI found it interesting that there's no explicit mention of the training objective. This omission can potentially lead to ambiguity in understanding the complete process and methodology involved. For a comprehensive study, it is crucial to explicitly specify each component, including the training objective, to ensure a thorough understanding of the research.\n```\n\nThank you for the recommendation. We discuss the training objective in detail in appendix A.1.2, lines 612-620. However, we acknowledge the importance of explicitly specifying each component, and we will include details about the training objective in the main document.\n\n```\nWhat are the reasons for choosing a deterministic training objective (eg. L2 reconstruction) for a problem that is inherently non-deterministic. Wouldn't a model that predicts a distribution over future trajectories be conceptually preferred to model that predicts a single point estimate?\n```\n\nIn most experimental settings (except the Lorentz force field setting), our method is formulated as a VAE with latent edge types. The training objective is minimizing the negative ELBO, which comprises a reconstruction term (Gaussian log likelihood) and a KL divergence regularization term. As such, our method can predict multiple different trajectories by sampling different latent variables, or sampling from the output Gaussian distribution, instead of using just the mean as a point estimate.\n\nIn the Lorentz force field setting, we indeed train only using the reconstruction term (MSE). While we agree that a probabilistic approach is more suitable to the task of future forecasting, we note, that this is in line with the majority of state-of-the-art works in the field, see [3, 4] as examples, as well as [5] that introduced this dataset.\n\n```\nLastly, there seems to be an omission in the citations pertaining to probabilistic models of neural fields. Specifically, works such as Functa [1] and Diffusion Probabilistic Fields [2] are not referenced. Including these references would enhance the completeness of the literature review and underscore the authors' comprehensive understanding of the existing body of research.\n```\n\nThank you for the suggestions. We agree that the line of works that includes Functa [1] and DPF [2] are related to our work, and we will gladly include these references in the related work for the camera ready version.\n\n### References\n\n[1] Dupont, Emilien, et al. \"From data to functa: Your data point is a function and you can treat it like one.\" ICML, 2022.\n\n[2] Zhuang, Peiye, et al. \"Diffusion probabilistic fields\". ICLR, 2022.\n\n[3] Satorras, Vıctor Garcia, et al. \"E(n) equivariant graph neural networks\". ICML, 2021.\n\n[4] Du, Weitao, et al. \"SE(3) equivariant graph neural networks with complete local frames\". ICML, 2022.\n\n[5] Brandstetter, Johannes, et al. \"Geometric and physical quantities improve E(3) equivariant message passing\". ICLR, 2022."}]

Author rebuttal:
### Reponse to question by Reviewer hUPP
```
Figure 3 shows the conditional neural field setup for dynamic fields, while most of the experiments are using static fields. To the reviewer, this is a bit misleading and does not showcase the proposed model properly. Please add both static and dynamic fields setup in the figure.
```

Thank you for the suggestion. We acknowledge that this can be misleading, and we will update the figure for the camera ready version of the paper. The architecture for the static field settings is almost the same, with the exception of the graph aggregation module that generates the latent vector $z$ and the FiLM layers that modulate the neural field.
Please see the update figure attached in the pdf.

Meta-review: The paper extends previous works on learning latent interactions for observations of the dynamics of interacting objects to include learning latent external fields. This is an important extension to the existing literature and the paper shows applications in both physical systems as well as traffic scenes. The authors use a novel architecture to address this problem that takes into account the appropriate symmetries. As such we decided that paper should be accepted to publication. However, there were many remarks on the writing/notations and we hope the writters will take the time to incoporate these remarks into the final version.
Decision: Accept (poster)