SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Liming Wu $^{1,2*}$ ; Zhichao Hou $^{3*}$ ; Jirui Yuan $^{4}$ , Yu Rong $^{5}$ , Wenbing Huang $^{1,2\dagger}$
$^{1}$ Gaoling School of Artificial Intelligence, Renmin University of China
$^{2}$ Beijing Key Laboratory of Big Data Management and Analysis Methods, Beijing, China
$^{3}$ Department of Computer Science, North Carolina State University
$^{4}$ Institute for AI Industry Research (AIR), Tsinghua University  $^{5}$ Tencent AI Lab
{manliowu,hwenbing}@ruc.edu.cn zhou4@ncsu.edu
yu.rong@hotmail.com yuanjirui@air.tsinghua.edu.cn
Learning to represent and simulate the dynamics of physical systems is a crucial yet challenging task. Existing equivariant Graph Neural Network (GNN) based methods have encapsulated the symmetry of physics, e.g., translations, rotations, etc, leading to better generalization ability. Nevertheless, their frame-to-frame formulation of the task overlooks the non-Markov property mainly incurred by unobserved dynamics in the environment. In this paper, we reformulate dynamics simulation as a spatio-temporal prediction task, by employing the trajectory in the past period to recover the Non-Markovian interactions. We propose Equivariant Spatio-Temporal Attentive Graph Networks (ESTAG), an equivariant version of spatio-temporal GNNs, to fulfill our purpose. At its core, we design a novel Equivariant Discrete Fourier Transform (EDFT) to extract periodic patterns from the history frames, and then construct an Equivariant Spatial Module (ESM) to accomplish spatial message passing, and an Equivariant Temporal Module (ETM) with the forward attention and equivariant pooling mechanisms to aggregate temporal message. We evaluate our model on three real datasets corresponding to the molecular-, protein- and macro-level. Experimental results verify the effectiveness of ESTAG compared to typical spatio-temporal GNNs and equivariant GNNs.
It has been a goal to represent and simulate the dynamics of physical systems by making use of machine learning techniques [37, 8, 12]. The related studies, once pushed forward, have great potential to facilitate a variety of downstream scientific tasks including Molecular Dynamic (MD) simulation [19], protein structure prediction [1], virtual screening of drugs and materials [30], model-based robot planning/control [35], and many others.
Plenty of solutions have been proposed, amongst which the usage of Graph Neural Networks (GNNs) [41] becomes one of the most desirable directions. GNNs naturally model particles or unit elements as nodes, physical relations as edges, and the latent interactions as the message passing thereon. More recently, a line of researches [38, 11, 21, 9, 32, 20] have been concerned with generalizing GNNs to fit the symmetry of our physical world. These works, also known as equivariant GNNs [18], ensure that translating/rotating/reflecting the geometric input of GNNs results in the output transformed in the same way. By handcrafting such Euclidean equivariance, the
Figure 1: Comparison of the problem setting between previous methods and our paper. Here, we choose the dynamics of the Aspirin molecular with time lag as 1 for illustration.
models are well predictable to scenarios under arbitrary coordinate systems, giving rise to enhanced generalization ability.
In spite of the fruitful progress, existing methods overlook a vital point: the observed physical dynamics are almost non-Markovian. In previous methods, they usually take as input the conformation of a system at a single temporal frame and predict as output the future conformation after a fixed time interval, forming a frame-to-frame forecasting problem. Under the Markovian assumption, this setting is pardonable as future frames are independent of all other past frames given the input one. Nevertheless, the Markovian assumption is rather unrealistic when there are other unobserved objects interacting with the system we are simulating [40]. For instance, when we consider simulating the dynamics of a protein that is interacting with an unobserved solvent (such as water), the Markovian property no longer holds; in other words, even conditional on the current frame, the future dynamics of the protein depends on the current state of the solvent which, however, is influenced by the past states of the protein itself, owing to the interaction between the protein and the solvent. The improper Markovian assumption makes current works immature in dynamics modeling.
To relieve from the Markovian assumption, this paper proposes to employ the states in the past period to reflect the latent and unobserved dynamics. In principle, we can recover the non-Markovian behavior (e.g. interacting with a solvent) if the past period is sufficiently long. We collect a period of past system states as spatio-temporal graphs, and utilize them as the input to formulate a spatiotemporal prediction task, other than the frame-to-frame problem as usual (see Figure 1). This motivates us to leverage existing Spatio-Temporal GNNs (STGNNs) [44] to fulfill our purpose, which, unfortunately, are unable to conform to the aforementioned Euclidean symmetry and the underlying physical laws. Hence, the equivariant version of STGNNs is no doubt in demand. Another point is that periodic motions are frequently observed in typical physical systems [3]. For example, the Aspirin molecular exhibits clear periodic thermal vibration when binding to a target protein. Under the spatio-temporal setting, we are able to model periodicity of dynamics, which, however, is less investigated before.
- We reveal the non-Markov behavior in physical dynamics simulation by developing equivariant spatio-temporal graph models. The proposed model dubbed Equivariant Spatio-Temporal Attentive Graph network (ESTAG) conforms to Euclidean symmetry and alleviates the limitation of the Markovian assumption.
- We design a novel Equivariant Discrete Fourier Transform (EDFT) to extract periodic features from the dynamics, and then construct an Equivariant Spatial Module (ESM), and an Equivariant Temporal Module (ETM) with forward attention and equivariant pooling, to process spatial and temporal message passing, respectively.
- The effectiveness of ESTAG is verified on three real datasets corresponding to the molecular-, protein- and macro-level. We prove that, involving both temporal memory and equivariance are advantageous compared to typical STGNNs and equivariant GNNs with adding trivial spatio-temporal aggregation.
GNNs for Physical Dynamics Modeling Graph Neural Networks (GNNs) have shown great potential in physical dynamics modeling. IN [2], NRI [23], and HRN [29] are a series of works to learn physical
Figure 2: Schematic overview of ESTAG. After inputting historical graph trajectories  $\mathcal{G}_0, \dots, \mathcal{G}_{T-1}$ , Equivariant Discrete Fourier Transform (EDFT) extracts equivariant frequency features  $\vec{f}$  from the trajectory. We process them into the invariant node-wise feature  $c$  and adjacency matrix  $\mathbf{A}$  to be adopted for the next stage. Then we stack Equivariant Spatial Module (ESM) and Equivariant Temporal Module (ETM) alternatively for  $L$  times to explore spatial and temporal dependencies. After the equivariant temporal pooling layer, we obtain the estimated position  $\vec{x}^*(T)$ .
system interaction and evolution. Considering the energy conservation and incorporating physical prior knowledge into GNNs, HNN [15], HOGN [31] leverage ODE and Hamiltonian mechanics to capture the interactions in the systems. However, all of the above-mentioned models don't take into account the underlying symmetries of a system. In order to introduce Euclidean equivariance, Tensor-Field networks (TFN) [38] and SE(3)-Transformer [11] equip filters with rotation equivariance by irreducible representation of the SO(3) group. LieTransformer [21] and LieConv [9] leverage the Lie group to enforce equivariance. Besides, EGNN [32] utilized a simpler E(n)-equivariant framework which can achieve competitive results without computationally expensive information. Based on EGNN, GMN [20] further proposes an equivariant and constraint-aware architecture by making use of forward kinematics information in physical system. Nevertheless, all of these methods ignore the natural spatiotemporal patterns of physical dynamics and model it as a frame-to-frame forecasting problem.
Spatio-temporal graph neural networks STGNNs [22] aim to capture the spatial and temporal dependency simultaneously and are widely investigated in various applications like traffic forecasting and human recognition. DCRNN [26] and GaAN [45] are RNN-based methods which filter inputs and hidden states passed to recurrent unit using graph convolutions. However, RNN-based approaches are time-consuming and may suffer from gradient explosion/vanishing problem. CNN-based approaches, such as STGCN [44] and ST-GCN [43], interleave 1D-CNN layers with graph convolutional layers to tackle spatial temporal graphs in a non-recursive manner. Besides, attention mechanism, an important technique STGCNs, is employed by GaAN [45], AGL-STAN [36] and ASTGCN [16] to learn dynamic dependencies in both space and time domain. The aforementioned approaches are targeted to the applications in 2D graph scenarios such as traffic networks and human skeleton graphs, and may not be well applicable to 3D physical systems in which geometric equivariance is a really important property.
Equivariant spatio-temporal graph neural networks There are few previous works designing spatio-temporal GNNs while maintaining equivariance. Particularly, by using GRU [6] to record the memory of past frames, LoCS [24] additionally incorporates rotation-invariance to improve the model's generalization ability. Different from the recurrent update mechanism used in LoCs, EqMotion [42] distills the history trajectories of each node into a multi-dimension vector, by which the spatio-temporal graph is compressed as a spatial graph, then it designs an equivariant module and an interaction reasoning module to predict future frames. However, both of LoCS and EqMotion are still defective in exploring the interactions among history trajectories, while in this paper we propose a transformer-alike architecture to fully leverage the spatio-temporal interactions based on equivariant attentions.
3 Notations and Task Definition
The dynamics of physical objects (such as molecules) can be formulated with the notion of spatiotemporal graphs, as shown in Figure 1 (left). In particular, a spatiotemporal graph of node number  $N$  and
temporal length  $T$  is denoted as  $\{\mathcal{G}_t = (\mathcal{V}_t,\mathcal{E})\}_{t = 0}^{T - 1}$ . Here, the nodes  $\nu_{t}$  are shared across time, but each node  $i$  is assigned with different scalar feature  $h_i(t)\in \mathbb{R}^c$ , position vector  $\vec{x}_i(t)\in \mathbb{R}^3$  at different temporal frame  $t$ ; the edges  $\mathcal{E}$  are associated with an identical adjacency matrix  $\pmb {A}\in \mathbb{R}^{N\times N\times m}$  whose element  $A_{ij}\in \mathbb{R}^m$  defines the edge feature between node  $i$  and  $j$ . We henceforth denote by the matrices  $\pmb {H}(t)\in \mathbb{R}^{c\times N}$  and  $\vec{\pmb{X}} (t)\in \mathbb{R}^{3\times N}$  the collection of all nodes in  $\mathcal{G}_t$ . Here for simplicity, we specify the time lag as  $\Delta t = 1$  which could be valued more than 1 in practice. In general, the  $t$ -th frame corresponds to time  $T - t\Delta t$ .
Task Definition This paper is interested in predicting the physical state, particularly the position of each node at frame  $T$  given the historical graph series  $\{\mathcal{G}_t\}_{t=0}^{T-1}$ . In form, we learn the function  $\phi$ :
$$
\{(H (t), \vec {X} (t), A) \} _ {t = 0} ^ {T - 1} \xrightarrow {\phi} \vec {X} (T). \tag {1}
$$
Although previous approaches such as EGNN [32] and GMN [20] also claim to tackle physical dynamics modeling, they neglect the application of spatiotemporal patterns and only accomplish frame-to-frame prediction, where the input of  $\phi$  is reduced to a single frame, e.g.,  $\mathcal{G}_{T-1}$  in Eq. 1.
Equivalence A crucial constraint on physical dynamics is that the function  $\phi$  should meet the symmetry of our 3D world. In other words, for any translation/rotation/reflection  $g$  in the group E(3),  $\phi$  satisfies:
$$
\phi \left(\left\{\left(\boldsymbol {H} (t), g \cdot \vec {\boldsymbol {X}} (t), \boldsymbol {A}\right) \right\} _ {t = 0} ^ {T - 1}\right) = g \cdot \vec {\boldsymbol {X}} (T), \tag {2}
$$
where the group action  $\cdot$  is instantiated as  $g\cdot \vec{X} (t)\coloneqq \vec{X} (t) + b$  for translation  $\pmb {b}\in \mathbb{R}^3$  and  $g\cdot \vec{X} (t)\coloneqq O\vec{X} (t)$  for rotation/reflection  $\pmb {O}\in \mathbb{R}^{3\times 3}$ .
4 Our approach: ESTAG
Discovering informative spatiotemporal patterns from the input graph series is vital to physical dynamics modeling. In this section, we introduce ESTAG that pursues this goal in a considerate way: We first extract the frequency of each node's trajectory by EDFT (§ 4.1), which captures the node-wise temporal dynamics in a global sense and returns important features for the next stage; We then separately characterize the spatial dependency among the nodes of each input graph  $\mathcal{G}_t$  via ESM (§ 4.2); We finally unveil the temporal dynamics of each node through the attention-based mechanism ETM and output the estimated position of each node in  $\mathcal{G}_T$  after an equivariant temporal pooling layer (§ 4.3). The overall architecture is shown in Figure 2.
4.1 Equivariant Discrete Fourier Transform (EDFT)
The Fourier Transform (FT) gives us insight into the wave frequencies contained in the input signal that is usually periodic. With the extracted frequencies, we are able to view the global behavior of each node  $i$  in different frequency domains. Conventional multidimensional FT employs distinct Fourier bases for different input dimensions of the original signals. Here, to ensure equivariance, we first translate the signals by the mean position and then adopt the same basis over the spatial dimension. To be specific, we compute equivariant DFT as follows:
$$
\vec {\boldsymbol {f}} _ {i} (k) = \sum_ {t = 0} ^ {T - 1} e ^ {- i ^ {\prime} \frac {2 \pi}{T} k t} \left(\vec {\boldsymbol {x}} _ {i} (t) - \overline {{\vec {\boldsymbol {x}} (t)}}\right), \tag {3}
$$
where,  $i'$  is the imaginary unit,  $k = 0,1,\dots ,T - 1$  is the frequency index,  $\overrightarrow{\pmb{x}} (t)$  is the average position of all nodes in the  $t$ -th frame  $\mathcal{G}_t$ , and the output  $\vec{f_i} (k)\in \mathbb{C}^3$  is complex. The frequencies calculated by Eq. 3 are then utilized to formulate two crucial quantities: the frequency cross-correlation  $\pmb{A}_{ij}\in \mathbb{R}^{T}$  between node  $i$  and  $j$ , and the frequency amplitude  $\pmb {c}_i\in \mathbb{R}^T$  of node  $i$ .
In signal processing, cross-correlation measures the similarity of two functions  $f_{1}$  and  $f_{2}$ . It satisfies  $\mathcal{F}\{f_1\star f_2\} = \overline{\mathcal{F}\{f_1\}}\cdot \mathcal{F}\{f_2\}$ , where  $\mathcal{F}$  and  $\star$  denote the FT and the cross-correlation operator, respectively, and  $\overline{\mathcal{F}}$  indicates the complex conjugate of  $\mathcal{F}$ . Borrowing this idea, we compute the cross-correlation in the frequency domain by Eq. 3 as:
$$
\boldsymbol {A} _ {i j} (k) = w _ {k} \left(\boldsymbol {h} _ {i}\right) w _ {k} \left(\boldsymbol {h} _ {j}\right) | \langle \vec {\boldsymbol {f}} _ {i} (k), \vec {\boldsymbol {f}} _ {j} (k) \rangle |, \tag {4}
$$
where  $\langle \cdot, \cdot \rangle$  defines the complex inner product. Notably, we have added two learnable parameters  $w_{k}(\pmb{h}_{i})$  and  $w_{k}(\pmb{h}_{j})$  dependent on node features, which act like spectral filters of the  $k$ -th frequency and enable us to select related frequency for the prediction. In the next subsection, we will apply  $\pmb{A}_{ij}$  as the edge feature to capture the relationship between different nodes. We use Aspirin as an example and visualize the  $\pmb{A}$  in Figure 3.
Figure 3: Visualization of cross-correlation  $A$  on Aspirin. EDFT can not only identify strongly-connected nodes (e.g. Node 8 and Node 11), but also discover latent relationship between two nodes which are disconnected yet may have similar structures or functions (e.g. Node 8 and Node 10).
We further compute for node  $i$  the amplitude of the frequency  $\vec{f}_i(k)$  along with the parameter  $w_{k}(\pmb{h}_{i})$ :
$$
\boldsymbol {c} _ {i} (k) = w _ {k} \left(\boldsymbol {h} _ {i}\right) \| \vec {\boldsymbol {f}} _ {i} (k) \| ^ {2}. \tag {5}
$$
This term will be used in the update of the hidden features in the next subsection.
A promising property of Eq. 3 is that it is translation invariant and rotation/reflection equivariant. Therefore, both  $A_{ij}$  and  $c_i$  are E(3)-invariant, which will facilitate the design of following modules.
4.2 Equivariant Spatial Module (ESM)
For each graph  $\mathcal{G}_t$ , our ESM is proposed to encode its spatial geometry through equivariant message passing. ESM is built upon EGNN [33] which is a prevailing kind of equivariant GNNs, but it has subtly involved the FT features from the last subsection for enhanced performance beyond EGNN.
The  $l$ -th layer message passing in ESM is as below:
$$
\boldsymbol {m} _ {i j} = \phi_ {m} \left(\boldsymbol {h} _ {i} ^ {(l)} (t), \boldsymbol {h} _ {j} ^ {(l)} (t), \| \vec {\boldsymbol {x}} _ {i j} ^ {(l)} (t) \| ^ {2}, \boldsymbol {A} _ {i j}\right), \tag {6}
$$
$$
\boldsymbol {h} _ {i} ^ {(l + 1)} (t) = \boldsymbol {h} _ {i} ^ {(l)} (t) + \phi_ {h} \left(\boldsymbol {h} _ {i} ^ {(l)} (t), \boldsymbol {c} _ {i}, \sum_ {j \neq i} \boldsymbol {m} _ {i j}\right), \tag {7}
$$
$$
\vec {\boldsymbol {a}} _ {i} (t) = \frac {1}{| \mathcal {N} (i) |} \sum_ {j \in \mathcal {N} (i)} \vec {\boldsymbol {x}} _ {i j} ^ {(l)} (t) \phi_ {x} (\boldsymbol {m} _ {i j}), \tag {8}
$$
$$
\vec {\boldsymbol {x}} _ {i} ^ {(l + 1)} (t) = \vec {\boldsymbol {x}} _ {i} ^ {(l)} (t) + \vec {\boldsymbol {a}} _ {i} (t), \tag {9}
$$
where,  $\phi_{m}$  computes the message  $m_{ij}$  from node  $j$  to  $i$ ,  $\phi_{h}$  updates the hidden representation  $h_{i}$ ,  $\phi_{x}$  returns a one-dimensional scalar for the update of  $\vec{a}_i(t)$ , and all the above functions are Multi-Layer Perceptrons (MLPs);  $\vec{x}_{ij}(t) = \vec{x}_i(t) - \vec{x}_j(t)$  is the relative position and  $\mathcal{N}(i)$  denotes the neighborhoods of node  $i$ .
Notably, we leverage the cross-correlation  $\mathbf{A}_{ij}$  as the edge feature in Eq. 6 to evaluate the connection between node  $i$  and  $j$  over the global temporal window, since it is computed from the entire trajectory. We also make use of  $c_{i}$  as the input of the update in Eq. 7. The benefit of considering these two terms will be ablated in our experiments.
4.3 Equivariant Temporal Module (ETM)
Forward Temporal Attention Inspired by the great success of Transformer [39] in sequence modeling, we develop ETM that describes the self-correspondence of each node's trajectory based on the forward attention mechanism, and more importantly, in an E(3)-equivariant way.
In detail, each layer of ETM conducts the following process:
$$
\alpha_ {i} ^ {(l)} (t s) = \frac {\exp \left(\boldsymbol {q} _ {i} ^ {(l)} (t) ^ {\top} \boldsymbol {k} _ {i} ^ {(l)} (s)\right)}{\sum_ {s = 0} ^ {t} \exp \left(\boldsymbol {q} _ {i} ^ {(l)} (t) ^ {\top} \boldsymbol {k} _ {i} ^ {(l)} (s)\right)}, \tag {10}
$$
$$
\boldsymbol {h} _ {i} ^ {(l + 1)} (t) = \boldsymbol {h} _ {i} ^ {(l)} (t) + \sum_ {s = 0} ^ {t} \alpha_ {i} ^ {(l)} (t s) \boldsymbol {v} _ {i} ^ {(l)} (s), \tag {11}
$$
$$
\vec {\boldsymbol {x}} _ {i} ^ {(l + 1)} (t) = \vec {\boldsymbol {x}} _ {i} ^ {(l)} (t) + \sum_ {s = 0} ^ {t} \alpha_ {i} ^ {(l)} (t s) \vec {\boldsymbol {x}} _ {i} ^ {(l)} (t s) \phi_ {x} \left(\boldsymbol {v} _ {i} ^ {(l)} (s)\right), \tag {12}
$$
where,  $\alpha_{ts}$  is the attention weight between time  $t$  and  $s$ , computed by the query  $\mathbf{q}_t$  and key  $\mathbf{k}_s$ ; the hidden feature  $\pmb{h}_i(t)$  is updated as a weighted combination of the value  $\pmb{v}_s$ ; the position vector  $\vec{\pmb{x}}_i(t)$  is derived from a weighted combination of a one-dimensional scalar  $\phi_x(\pmb{v}_s)$  multiplied with the temporal displace vector  $\vec{\pmb{x}}_i^{(l)}(ts) = \vec{\pmb{x}}_i^{(l)}(t) - \vec{\pmb{x}}_i^{(l)}(s)$ . Specifically,  $\pmb{q}_i^{(l)}(t) = \phi_q(\pmb{h}_i^{(l)}(t))$ ,  $\pmb{k}_i^{(l)}(t) = \phi_k(\pmb{h}_i^{(l)}(t))$  and  $\pmb{v}_i^{(l)}(t) = \phi_v(\pmb{h}_i^{(l)}(t))$  are all E(3)-invariant functions. Notably, we derive a particle's next position in a forward-looking way, to keep physical rationality as the derivation of current state should not be dependent on future positions.
Equivariant Temporal Pooling We alternate one-layer ESM and one-layer ETM over  $L$  layers, and finally attain the updated coordinates  $\vec{\pmb{x}}_i^{(L)}\in \mathbb{R}^{T\times 3}$  for each node  $i$ . Then the predicted coordinates at time  $T$  is given by the following equivariant linear pooling:
$$
\vec {\boldsymbol {x}} _ {i} ^ {*} (T) = \hat {\boldsymbol {X}} _ {i} \boldsymbol {w} + \vec {\boldsymbol {x}} _ {i} ^ {(L)} (T - 1), \tag {13}
$$
where the parameter  $\pmb{w} \in \mathbb{R}^{(T-1)}$  consists of learnable weights, and  $\hat{\pmb{X}}_i = [\vec{\pmb{x}}_i^{(L)}(0) - \vec{\pmb{x}}_i^{(L)}(T-1), \vec{\pmb{x}}_i^{(L)}(1) - \vec{\pmb{x}}_i^{(L)}(T-1), \dots, \vec{\pmb{x}}_i^{(L)}(T-2) - \vec{\pmb{x}}_i^{(L)}(T-1)]$  is translated by  $\vec{\pmb{x}}_i^{(L)}(T-1)$  to allow translation invariance.
We train ESTAG end-to-end via the mean squared error (MSE) loss:
$$
\mathcal {L} = \sum_ {i = 1} ^ {N} \| \vec {\boldsymbol {x}} _ {i} (T) - \vec {\boldsymbol {x}} _ {i} ^ {*} (T) \| _ {2} ^ {2}. \tag {14}
$$
By the design of EDFT, ESM and ETM, we have the following property of our model ESTAG.
Theorem 4.1. We denote ESTAG as  $\vec{X}(T) = \phi\left(\{(H(t), g \cdot \vec{X}(t), A)\}_{t=0}^{T-1}\right)$ , then  $\phi$  is  $E(3)$ -equivariant.
Although we mainly exploit EGNN as the backbone (particularly in ESM), our framework is general and can be easily extended to other equivariant GNNs, such as GMN [20], the multi-channel version of EGNN. In general, the extended models deal with multi-channel coordinate  $\vec{Z} \in \mathbb{R}^{3 \times m}$  instead of  $\vec{x} \in \mathbb{R}^3$ . The most significant feature of these models is to replace the invariant scalar  $\|\vec{x}\|^2$  in the formulations of the message  $m_{ij}$  (Eq. 6) with the term  $\vec{Z}^\top \vec{Z}$ . It is easily to prove that this term is equivariant to any orthogonal matrix  $O$ , i.e.,  $(O\vec{Z})^\top (O\vec{Z}) = \vec{Z}^\top \vec{Z}, \forall O \in \mathbb{R}^{3 \times 3}, O^\top O = I$ . Besides, it can be reduced to invariant scalar  $\|\vec{x}\|^2$  when  $m = 1$ . Empirically, we add the normalization term in order to achieve more stable performance:  $\frac{\vec{Z}^\top \vec{Z}}{\|\vec{Z}^\top \vec{Z}\|_F}$ , where  $\| \cdot \|_F$  is the Frobenius norm. In the above derivation, we only display the formulation on EGNN, the details of multi-channel ESTAG are shown in Appendix C.1.
Table 1: Prediction error  $(\times 10^{-3})$  on MD17 dataset. Results averaged across 3 runs. We do not display the standard deviation due to its small value.
ASPIRIN	BENZENE	ETHANOL	MALONALDEHYDE	NAPHTHALENE	SALICYLIC	TOLUENE	URACIL
Pt-s	15.579	4.457	4.332	13.206	8.958	12.256	6.818	10.269
Pt-m	9.058	2.536	2.688	6.749	6.918	8.122	5.622	7.257
Pt-t	0.715	0.114	0.456	0.596	0.737	0.688	0.688	0.674
EGNN-s	12.056	3.290	2.354	10.635	4.871	8.733	3.154	6.815
EGNN-m	6.237	1.882	1.532	4.842	3.791	4.623	2.516	3.606
EGNN-t	0.625	0.112	0.416	0.513	0.614	0.598	0.577	0.568
ST_TFN	0.719	0.122	0.432	0.569	0.688	0.684	0.628	0.669
ST_GNN	1.014	0.210	0.487	0.664	0.769	0.789	0.713	0.680
ST_SE(3)TR	0.669	0.119	0.428	0.550	0.625	0.630	0.591	0.597
ST_EGNN	0.735	0.163	0.245	0.427	0.745	0.687	0.553	0.445
EQMOTION	0.721	0.156	0.476	0.600	0.747	0.697	0.691	0.681
STGCN	0.715	0.106	0.456	0.596	0.736	0.682	0.687	0.673
AGL-STAN	0.719	0.106	0.459	0.596	0.601	0.452	0.683	0.515
ESTAG	0.063	0.003	0.099	0.101	0.068	0.047	0.079	0.066