An Exploration of Conditioning Methods in Graph Neural Networks
Authors: Yeskendir Koishekenov
Erik J Bekkers
Venue: Submitted to ICLR 2023 (2023)
Keywords: graph neural networks
geometric deep learning
deep learning

Abstract: The flexibility and effectiveness of message passing based graph neural networks (GNNs) induced considerable advances in deep learning on graph-structured data. In such approaches, GNNs recursively update node representations based on their neighbors and they gain expressivity through the use of node and edge attribute vectors. E.g., In computational tasks such as physics and chemistry usage of edge attributes such as relative position or distance proved to be essential. In this work, we address not what kind of attributes to use, but how to condition on this information to improve model performance. We consider three types of conditioning; weak, strong, and pure, which respectively relate to concatenation-based conditioning, gating, and transformations that are causally dependent on the attributes. This categorization provides a unifying viewpoint on different classes of GNNs, from separable convolutions to various forms of message passing networks. We provide an empirical study on the effect of conditioning methods in several tasks in computational chemistry.

Under review as a conference paper at ICLR 2023
AN EXPLORATION OF CONDITIONING METHODS IN GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The flexibility and effectiveness of message passing based graph neural networks (GNNs) induced considerable advances in deep learning on graph-structured data, In such approaches, GNNs recursively update node representations based their neighbors and they gain expressivity through the use of node and edge attribute vectors. E.g., in computational tasks such physics and chemistry usage of edge attributes such as relative position or distance proved to be essential. In this work, we address not what kind of attributes to use, but how condition on this infor- mation to improve model performance. We consider three types of conditioning; weak, strong, and pure, which respectively relate to concatenation-bant tioning, gating, and transformations that are causally dependent on the attributes. This categorization provides unifying viewpoint on different classes of GNNs, from separable convolutions to various forms of message passing networks. We provide an empirical study on the effect of conditioning methods several tasks in computational chemistry.
condi-
INTRODUCTION
Graph neural networks (GNNs) are a family of neural networks that can learn from graph- structrrd data. Starting with the success of GCN (Kipf & Welling, 2016) in achieving state-of-the per- formance on semi-supervised classification,, several variants of GNNs have been developed for this task, including Grap (Hamilton al. 2017). GAT (Veličković et al., 2017), GATv2 (Brody ct al. 2021), EGNN (atorras et al. 2021 to
few most recent ones.
Most of the models based on the message-passingg framework utilize conditional linear layers. We define "conditioning" " as using additional information together with feature vectors from its neigh- bors' nodes. For example, EGNN (Satorras et al. 2021) conditions message vectors on the distance between two nodes DimeNet (Gasteiger Many neural network models use conditioning in their layers without exploring their different vari- ants. Therefore, improving upon the type of conditioning could still improve most state-of-the-an models. We believe that this is the first work that analyzes different conditioning methods in GNNs.
al. 2020b) additionally utilizes angle information.
In this paper, we categorize three conditioning methods: weak, strong, and pure. They differ in the level of dependency on the a given quantity, such as an edge attributes, and differ in complexity. Message passing neural network (MPNN) using weak conditioning method concatenates attributes with node features. In this scenario. linear layers effectively gain an attribute-dependee de-endenteet bias, which we consider weak type of conditioning as this does not guarantee that the attribute is actually utilized, i.e., it could be ignored. On the other hand, we have pure conditioning method which forces the model to always use the attributes by letting them causally parametrize transformation matrices. However, from practical perspective pure conditioning is computationally expensive and it can be simplified to strong conditioning method, which corresponds to an attribute-dependent gating of the outputs of linear layers. We experiment with these three conditioning methods in variations of the EGNN model (Satorras et al.] (021)) on computational chemistry datasets QM9 (Ramakrishnan et al.) 2014) and MD17 (Chmiela et al. 2017) and show the advantage of strong conditioning weak conditioning in performance, and over pure conditioning in training time.
The main contributions of this paper are:

---

Under review as a conference paper at ICLR 2023
(i) A unifying analysis of geometric message passing by formulating conditional transforma- tions in terms of various forms of conditional linear layers.
(ii) An intuitive exposition of different conditioning methods in the context of convolutional message passing.
(iii) Empirical studies that show the benefit of strong conditioning methods, as well as the ben- efit of deep conditioning in multi-layer perceptron-based message functions.
In this work, address not what kind of attributes use, but how to condition this information to improve model performance. As such, we focus on an intuitive analysis and in the experimental section do not intend to achieve the best performance, but focus on ablation studies in order to obtain general take-home messages.
2
PRELIMINARIES
In this section, we introduce the relevant materials on graph neural networks on top of which we will later complement our analysis and definitions of conditioning methods.
2.1 GRAPH NEURAL NETWORK
In this work, we consider the graph regression task example. A graph is represented by g (V, E) with nodes E V and edges eij E E. A typical message passing layer (Gilmer et al.
2017 is defined as:
mij = Se a)))
(1)
m: Σ my
JEN(1)
h = on (h'hh,m
Where h is the embedding of node o at layer aij is the edge attribute of nodes U; and vy. and (()) is the set of neighbors of the node uy Finally, the and Φn are the message (edge) and update (node) functions respectively which are commonly parametrized by Multilayer Perceptrons (MLPs).
2.2 GEOMETRIC GRAPH NEURAL NETWORKS
When the graphs have an embedding in Euclidean space, i.e., each node has an associated position x; E TR" we want to leverage this geometric information whilst preserving stability/invariance to rigid-body transformations. That is, many tasks are invariant to Euclidean distance preserving transformations in E(n). E.g.. the prediction of energy of a system of atoms in invariant to its global position and orientation in space. Several works have shown how to build equivariant message passing based graph neural networks for such geometric graphs.
Central in those works is the conditioning of the message and update function invariant geomet- ric attributes, such as the pairwise distance a, = ||x, x;||. as popularized in (Satorras et al. 2021), or covariant spherical/circular harmonic embeddings of relative position a.; Y(x; x; as is common steerable group convolution- graph NNs (Brandstetter et al.] 2021). Here we consider attributes that transforms predictably via representations of E(n) as covariants, and those that remain invariant as invariants. Such covariants typically contain more (directional) information. but require specialized operations such the Clebsch- Gordan tensor product (Thomas al. 2018
Anderson et al.) 2019) in order to preserve equivariance of the graph NNs. [Saooraas et al. (2021) show that with a simple recipe based on invariant attributes, one can often obtain equally powerful graph NNs. As such, we focus this paper the use of a.j ||x; x; as sufficiently expressive attribute, and model the message and update functions &e and on as regular MLPs.
Our objective then is to understand what is the most effective way of utilizing attributes in geometric graph NNs. To make this notion of conditioning explicit, we will denote the message and update function
de (b), h² any) and on (h |as), where we note that, although uncommon, it is possible to define invariant or covariant geometric node attributes a; (Brandstetter al. 202T).
2

---

Under review as a conference paper at ICLR 2023
3 ANALYSIS OF CONDITIONING METHODS
In the subsequent, we unify several conditioning methods used in literature through the notion of conditional linear layers, and by discussing them relation to the prevalent convolution layer and its variations. As a starting point, we use the fact that convolution is simple form of message passing with linear message functions conditioned on relative position, i.e.,
mij (f) X x,) = W(x; - x;
(4)
and the following update function typically is the application of point-wise activation function d, ie., 6n (f.) a (f.), possibly with a skip connection as ResNets He al. (2016). In general, geometric graph NN do not just linearly transform node features, but generally do this non-linearly message/update functions parametrized by MLPs, leading a notion of non-linear convolutions when the attributes are invariant/covariant quantities (Brandstetter et al. 2021
Importantly, these MLPs themselves are parametrized by linear layers, intertwined with non-linear activation functions, and nothing prevents from conditioning each of these linear layers the at- tributes. It the purpose this paper to categorize several options for conditioning and understand what effect this has performance.
3.1 CONDITIONAL LINEAR LAYERS
We propose the following modifications of the linear layer, to make them conditional on attributes
Wah == Wh
(5)
Wah == W (h @a)
weak
(6)
Wa h = (W"a) o (W"h)
strong
(7)
W,h = W(a)h
pure
where keep the similarity the common notation for linear transformations, we use the notation W, to denote that the linear transformation is conditioned on We further use to notation h @ a to
denote the concatenation of vectors h and a, and use @ to denote element- multiplication, i.e., ④ b (a,bı, azb2....
We stress the hierarchy terms of the dependence the transformation on a. The most direct de- pendence is in the pure method, in which the transformation is causally parametrized by a, followed by strong conditioning which a standard unconditional transformation W is gated by a vector W a. Both the pure and strong methods are by construction forced to utilize the attribute, as, in particular in the strong case, the transformation would not exists without attribute a. We refer to equation as weak conditioning, of the attribute, and moreover, if the dimensionality of is much larger than that of a, the transfor- mation of the attribute only contributes to small extent to the output of this layer. We hypothesize that this hierarchy correlates with performance and experimentally test this hypothesis in Sec. What follows brief analysis of these types of conditioning.
in principle the transformation would still exist in the absence
3.2 PURE CONDITIONING CORRESPONDS TO BI-LINEAR LAYERS
As common for implementations of convolutions, one typically expands the convolution kernel W (x, - x,) R.ox,; i.e., transformation matrix with elements Wos - x;) that depend on relative position, in basis (Ør: .R. R) via
de
W Wor(x/ - x,) Σ
Whor di (x, -x))
(9)
The basis could be the usual 3 x 3 pixel basis, or it could be continuous basis for when the continuous structure of the data is to be respected, such as circular or spherical harmonics (Worrall
et al. 2017 Weiler & Cesa 2019 Thomas et al. [2018] Anderson et al. 2019), B-splines (Bekkers) 2020 Fey et al. 2018), or hermite polynomials (Sosnovik et al.) 2000). recent works on the
parametrization of continuous functions Neural Fields (Xie et al. 2022 in transformer-based
3

---

Under review as a conference paper at ICLR 2023
methods (Vaswani et al. 2017). such basis functions often referred to coordinate embeddings or position encodings.
An important observation is that, given such parametrization through basis functions, the pure conditioning layer corresponds to a bi-linear layer
ΣΣ (a) Whot by > b
h²+1
((a)
bilsnear
W h'
(10)
where we use i to index the input feature vector, o the output feature vector, and the basis functions.
Such bilinear layers are often implicitly used in convolutional architectures, where the expansion in the basis function is typically hard-coded or pre-computed. On continuous data, such as point cloud methods, the bilinear layer is ubiquitous. Notably, in the context of steerable group equivariant convolutions, the transformations happen via bilinear operators called the Glebsch-Gordan tensor product, in combination with spherical harmonic embeddings of relative position (Brandstetter et al., 2021). Outside of the (group) convolution literature, bilinear layers are often used to explicitly model conditioning geometric attributes, for which relevant example to current work is DimeNet (Gasteiger et al.] 2020b). DimeNet uses an advanced message passing framework in which messages and updates are conditioned geometric quantities (embedded as spherical harmonies and radial basis functions) via combinations of weak, strong, and pure conditioning.
3.3 STRONG CONDITIONING CORRESPONDS TO DEPTH-WISE SEPARABLE CONVOLUTIONS
In later works, DimeNet was improved by DimeNet++ (Gasteiger al.) 2020a). both in perfor- mance and speed, by replacing the compute-heavy bilinear layers (pure conditioning) with the more efficient gating type conditioning (strong conditioning). The computational bottleneck of pure con- ditioning motives the use of strong conditioning, route that has proven successful in convolutional architectures computer vision as well, via the use of so-called depth-wise separable convolutions Sifree & Mallat] 2014] Chollet (2077).
Chollet (017)) shows huge efficiency gains, both in terms of computing and performance, when fac- torizing convolution kernels in two parts. One part does the channel mixing, which does not depend on the relative position, while another part depends on the relative position which scales/gates the
output. That is if the kernel is given by
(xi - xx)) = W. - w.
(11)
the convolution boils down to message passing with conditional linear layers of the strong type, as we can write
h' = (W"(x; x.)) O (W"h),
(12)
where we can define W - x.) = W"a, as the linear transformation of a coordinate em- bedding a.j Φxx, x;) if we want the connection to equation explicit. The fact that the transformation overall is linear and that it splits into a part that does and a part that doesn' 't depend on pair-wise attributes allows for very efficient implementations that first perform a group- wise convolution, followed by channel mixing. This principle at the core of the recently popularized ConvNeXt architecture (Liu al. 2022) and plays essential role in equivariant graph NNs on (molecular) point clouds in order to be able to scale up (Thomas et al.) 2018). Separability recently also proved necessary in order for equivariant convolutional NNs scale up large groups, such as the ccale-roatioon-trasllae parametrize neural fields (Xie et 2022), strong conditioning commonly appear in the form of so-cal so-called film layers Perez et al. (2018).
group (Knigge et al.) 2022). In the context of conditional NN to
3.4 WEAK CONDITIONING CORRESPONDS TO LINEAR LAYERS WITH A CONDITIONAL BIAS
Weak conditioning (cquation 6 corresponds to standard linear layer with an adaptive bias:
Wah W (h@a)= w h +
W
a w' h b(a).
(13)
no coodition
conditional bias
where W' are the first d, rows of W that applied the h Rd. of the concatenated vector, and w the last de rows of W that are applied Rdd. This simple form of conditioning
4

---

Under review as a conference paper at ICLR 2023
most used in literature to condition any type of NN on the conditioning vector a. from message passing methods (Gilmer et al. 2017 Satorras et al. (022), to conditional neural fields, e.g. as a simple but effective form of modulation in sirens (Dupont et al. 2022). to conditional variational auto-encoders (Sohn et al. 2015).
3.5
CONDITIONAL MLPS
With the various forms of conditional linear layers given in equations 6 and we can build conditional MLPs, by simply replacing the usual linear layers with the conditional ones. Such MI.ss could e.g. he denoted with MLP (a). Usually, only the first layer of such a conditional MLP is conditional and usually of the weak type, as with EGNN (Satorras et al.) 202T). In the experiments we show that this simple choice is sub-optimal in the context of geometric message passing à la EGNN, and show that improvements can be made by either by conditioning more layers, or switching to strong conditioning.
4 RELATED WORK: GEOMETRIC MESSAGE PASSING FOR COMPUTATIONAL CHEMISTRY
In the previous section, we discussed several options for conditioning message/update functions for us message passing graph neural networks, as well as their use in other fields of deep learning- In our experiments benchmark the three conditioning methods (equations 6 7 8) in the context geometric message passing for computational chemistry. Recent works this category, and the types of conditioning used in those works, are as follows.
EGNN (Satorras et al. 202T is message passing neural network (MPNN) that uses a weak condi- tioning method to utilize the distance between nodes. DimeNet (Gasteiger al. 2020b) is the type of MPNN, where message embeddings interact based on the distance between atoms and the angle directions. Pure conditioning is adapted utilize angle directions in message update and aggrega- tion. Tensor Field Network (Thomas et al.) 2018) is a neural network for 3D point clouds. Each point in TFN is associated with vector in representation of (O To condition one representa- tion on another, a tensor product of representations is used which in its general form corresponds to pure conditioning. However, many works of the steerable message passing kind, including TFN and NequlP (Batzner et al. (2022), implement a separable variation (strong conditioning) for the sake of computational efficiency. The exception is steerable EGNN (Brandstetter et al., (02T), which uses steerable MIPs with pure conditioning in each layer, i.e., not only the first layer of the message function as in EGNN.
DimeNet++ (Klicpera et al. 2020) is an extension of DimeNet which changed conditioning method from pure to strong to increase efficiency. Klicpera et al. (2020) showed that changing the condi- tioning method decreased the runtime time of the original DimeNet by factor of 5. SchNet (Schütt) ct al., 2017) is another example of MPNN that utilizes atom locations using a strong conditioning method. PaiNN (Schütt et al. (202I) further extends SchNet by projecting the interatomic distances via radial basis functions and iteratively updating the vectors along with the scalar features but also using a strong conditioning method.
EXPERIMENTS
In this section, we design experiments to evaluate the effectiveness of three conditioning methods shown in equation 6 7 and g on two real-world datasets: QM9 (Ramakrishnan et al., 2014) and MD17 (Chmiela et al. 2017). We want to demonstrate the effect of the choice of the conditioning method and present the results of some other models on benchmarks to provide context. In our experiments, report Mean Absolute Error (MAE) between model predictions and ground truth.
Implementation details As a baseline architecture, we use the EGNN model that consists of 7 lay- ers, 128 features per hidden layer, and 2-layer message (edge) function O. MLP @hy ||x, x2) with only weak conditioning via equation the first layer. We will test modifications of this model by changing the weak conditioning, to strong or pure, and explore the effect of conditioning multiple layers in the edge function MI.P.
5

---

Under review as a conference paper at ICLR 2023
We found it useful to pass geometric distances through two-layer MLP, as form of vectoriza- tion/coordinate embedding of the pairwise distance, for the QM9 dataset and Random Fourier Fea- ture layer (Rahimi & Recht 2007) for the MD17 dataset.
We use the same values for hyperparameterss from the EGNN paper (Satorras et al. 202T) for both datasets: trained each model on QM9/MD17 dataset for total of 1.000/500 epochs, used Adam optimizer, batch size 96 (64 for pure-EGNN), weight decay le-16, and cosine decay for the learning rate starting at 5e-4.
QM9 The QM9 (Ramakrishnan et al. 2014) dataset consists of small molecules represented as set of atoms (up 29 atoms per molecule), each atom having a 3D position associated and a five- dimensional one-hot node embedding that describe the atom type (H, C, N, O, F). The QM9 dataset has 3D coordinate locations of each atom and we use the distance between two atoms (nodes) as an edge attribute. The dataset comprises 12 quantum properties for each of the molecules. We use 100k molecules for training, 18k for validation, and 13k for testing.
Table shows the Mean Absolute Error for the prediction of 12 molecular properties for weak and strong conditioning methods. We can clearly see that using strong conditioning shows lower MAE than weak conditioning for all properties by an average of 10%. Due to large number of data points and our limited computational resources we limit our experiments to weak and strong conditioning methods. Training one epoch of pure-EGNN was on average 267x longer than training strong-EGNN which made intractable to validate pure conditioning.
Table I: Mean Absolute Error for the molecular property prediction benchmark in QM9 dataset.
Task
Ar meV
<номо meV
ELUMO meV
H
R²
U mcV
U/ meVV
ZPV E meV
Units
D
cal/mol K
meV
meV
boky"
SchNet
.235
69
43
38
.030
.040
17
.180
20
20
1.50
DimeNet++
044
33
25
20
.030
.023
7
.331
6
1.21
EGNN (baselinc)
.071
48
29
.029
.031
.106
12
1.55
weak-EGNN (ours)
.067
50.52
28.11
25.74
.032
.031
9.81
10.51
.138
11.14
9.94
1.455
strong EGNN (ours)
.061
44.52
27.48
23.83
.023
.029
9.79
10.29
.088
9.39
9.88
1.45
The advantage in the performance of strong conditioning over weak conditioning is clearly shown in Table We also hypothesize that the advantage of the strong conditioning method will be more evident in constrained settings of shallow networks. In order test whether strong conditioning provide a more efficient parametrization then weak conditioning, we repeat experiments from Table Ton three molecular properties, but with more shallow network. Table 2a shows the MAE for the prediction of three molecular properties for weak and strong conditioning methods for models with 7 and 3 layers. On average improvement of strong conditioning over weak conditioning was more highlighted for the shallow network than for the deep network (10.8% vs 7.7%). It demonstrates the importance of conditioning methods in models with less capacity.
We define conditioning depth as the number of conditional layers in the message function. In experiments, the conditioning depth was 2 as we conditioned all two layers in the message function. We test the effect of conditioning depth for different conditioning methods. Table 2b shows the MAE for the prediction of five molecular properties for weak and strong conditioning methods for models with conditioning depths 1 and 2. The weak-EGNN shows better performance with single conditional layer while strong-EGNN EGNN benefits more from two conditional layers.
MD17 MD17 (Chmiela et al., 2017) dataset of eight small organic molecules containing up to 17 total atoms composed of the atoms H, C. N, O, F. For each molecule, an ab-initio molecular dynamics simulation was run using DFT to calculate the ground state energy and forces. At inter- mittent timesteps, the energy, forces, and configuration (positions of each atom) were recorded. We uniformly sample 50k molecules for training, 10k for validation, and 10k for testing.
Table 3 shows the Mean Absolute Error for the prediction of energies for 8 molecules for three conditioning methods. From Table we can see that strong conditioning improved the performance of 5 molecules by average of 14.5%. It did not show improvement in molecules that initially had high MAE. In experiments with pure conditioning, to decrease computational cost we used half of the model: 2 conditional layers, hidden embedding size of 128 64, and 7 3 layers. With

---

Under review as a conference paper at ICLR 2023
Table 2: Mean Absolute Error (MAE) for the molecular property prediction benchmark in QM9 dataset for different numbers of layers and conditioning depths. A is the percentage difference of MAE, lower is better.
(a) shallow vs. deep models
(b) 1 vs. conditional layers
Task
a boby
de meV
CROMO meV
Task
Ac mcV
Eapur meV
ELUINO meV
Units
Units
bohr
Weak-EGNN (7 layers)
067
50.52
28.11
weak- (cond. depthwi)
.061
46.48
28.59
22.76
.024
strong EONN layers)
.661
44.52
27.48
weak EGNN (cond. depih=:)
.067
50.52
28.11
25.74
.022
-9.22%
-11.88%
-2.24
strong- EGNN (cond. depth=1)
45.72
27.16
22.41
0126
weak Weak-EGNN (3 layers)
0766
61.26
36,48
Strong-EGNN (cond. dopth=2)
.061
44.52
27.48
23.83
.023
EGNN layes)
.699
53.25
32,9
these changes, a pure-EGNN model was 14.3x and 16x times slower than strong-EGNY and weak- EGNN respectively. The negative impact of decreasing the number of layers on the performance of BoNn on the QM9 dataset can be seen in Table 22 Considering that we reduced by half the number of layers, hidden embedding size, and conditioning depth, pure-EGNN shows competitive performance with Beok/strong-FGNi on 5 molecules and outperforms it some of them. This potential gain in performance might, however, not outweigh the computation costs of pure over strong weak conditioning.
Table 3: Mean Absolute Error (MAB) for the conformational energies (meV) prediction benchmark on MD17 dataset.
Molecule
Aspirin
Benzene
Ethanol
Malonaldehyde
Naphthalene
Salicylic acid
Toluene
Uracil
Cormorant
4.25
0.997
1.171
1.778
1.258
2.862
1.474
0.997
sGDML
8.239
4.336
3.035
4.336
5.204
5.204
4,336
4.77
SchNet
5,204
3.035
2.168
3,469
4.77
4.336
3.903
4.336
weak- -GNN
25.608
4.262
2.576
11,755
13.693
19.132
16.447
12.468
strong- strong-EGNN EGNN
29.168
3.265
2.278
10.454
13.794
15.131
18.804
11.726
pure-EGNN*
34.706
3.513
4.582
12,403
21.421
27.221
14.692
12.249
pure-FEGNN network is smaller than weak/strong-EGNNI
CONCLUSION
In this work, we explore how graph neural networks can recursively update node embeddings with edge attribute information such as geometric distance. We provide unifying analysis of several works literature that utilize attributes, through a notion of conditional linear layers. We present three conditioning methods this end: weak, strong, and pure. Weak conditioning method catenates edge attributes to node features, strong conditioning method gates node features, and in pure conditioning, edge attributes causally parametrize transformation matrices. We explain the in- tuition of each method and apply them to the EGNN model to empirically show their difference in performance and computational cost on QM9 and MD17 datasets.
Our conclusion is that strong conditioning (gating) generally beats weak conditioning (concatena- tion) the message passing framework. We also conclude that pure conditioning computationally prohibitive in geometric message passing, whilst it can achieve competitive performance with weak and strong conditioning methods with smaller network. This confirms the impact observed by other works on separable convolutions, s.a. in Depth-wise separable convolutions (Chollet, (017) and ConvNeXt (Liu et al. 2022), and justifies the use of separable (group) convolutions of steerable tensor-based methods such TFN (Thomas et al. 2018). While these methods can be formulated as linear message passing methods of the convolutional form, we show that performance gains can be achieved through multi-layer conditional message functions, (Brandstetter et al. 202T). this setting, we show that it can beneficial to condition all layers in the conditional MLP. rather than only condition the first layer is the convention.
form of non non-linear convolution

---

Under review as a conference paper at ICLR 2023
We believe that categorization of conditioning methods, combined with empirical findings can be used as a guideline in designing the next generation of geometrie graph neural network architectures.
REFERENCES
Brandon Anderson, Truong Son Hy, and Risi Kondor. Cormorant: Covariant molecular neural networks. Advances in neural information processing systems, 32, 2019.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Ko- rnbluth, Nicola Molinari, Tess E Smidt, and Boris Kozinsky. E (3)=equivariant graph neural networks for data-efficient accurate interatomic potentials. Nature communications, 13(1):
1-11,2022
Erik Bekkers. B-spline cnns on lie groups. In International Conference Learning Representa- tions, 2020. URL WRRLtttps://ppeeeriiie
Johannes Brandstetter, Rob Hesselink, Elise van der Pol, Erik Bekkers, and Max Welling. Ge- ometric and physical quantities improve e (3) equivariant message passing. arXiv preprint arXiv:2110.02905, 2021.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? arXiv preprint arXiv:2 / /4491, 2021.
Stefan Chmiela, Alexandre Tkatchenko, Huziel E Sauceda, Igor Poltavsky, Kristof T Schütt, and Klaus-Robert Müller. Machine learning of accurate energy-conserving molecular force fields. Science advances, 3(5)ee 2017.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Emilien Dupont, Hyunjik Kim, SM Ali Eslami, Danilo Jimenez Rezende, and Dan Rosenbaum. From data to functa: Your data point is a function and you can treat it like one. In International Conference on Machine Learning. pp. 5694-5725. PMLR, 2022.
Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. Splinecnn: Fast geomet- ric deep learning with continuous b-spline kernels. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 869-877, 2018.
Johannes Gasteiger, Shankari Giri, Johannes T. Margraf, and Stephan Günnemann. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. In Machine Learn- ing for Molecules Workshop. NeurIPS, 2020a.
Johannes Gasteiger, Janck GroB, and Stephan Günnemann. Directional message passing for molecular graphs. In International Conference on Learning Representations, 2020b. URL
https://openreview. net www.:/ooommmmdieeaaaaee
Justin Gilmer, Samuel Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning. pp- 1263-1272. PMLR, 2017.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems, 30, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint arXiv: wet..:.09000900 2016.
Johannes Klicpera, Shankari Giri, Johannes T Margraf, and Stephan Günnemann. Fast and uncertainty-awart directional message passing for non-equilibrium molecules. arXiv preprint 1rXiv:2011.141155 2020.
8

---

Under review as a conference paper at ICLR 2023
David M Knigge, David W Romero, and Erik Bekkers. Exploiting redundancy: Separable group convolutional networks on lic groups. In International Conference on Machine Learning, pp- 11359-11386 PMLR, 2022.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 1 11976-11999 1986, 2022.
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in neural information processing systems, 20, 2007.
Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp. and O Anatole Von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):177, 2014.
Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural net- works. In International Conference on Machine Learning, pp. 9323-9332. PMLR, 2021.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Rober Müller. Schnet: continuous filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In International Conference on Machine Learning, pp. 9377-93888 PMLR, 2021.
Laurent Sifre and Stéphane Mallat. Rigid-motion scattering for texture classification. arXiv preprint arXiv:140311687 2014.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran As- sociates, Inc., 2015. httpthttttttt////ptte URL/httpss:/pprrceeiinns neur sesuripps.. wttea///pps..cpppe/////111/ ccc/paper/2011/11ee pdf
Ivan Sosnovik, Michal Szmaja, and Amold Smeulders. Scale-equivariant steerable networks. In International Conference on Learning Representations, 2020. URL http:tttt///peeee/////// htte/Eoouum.id-HJgpug
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equiiariann neural networks for 3d point clouds. arXiv preprint arXiv: /r2ivv/802.08219, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint 100.0037,,,,0003, 2017.
Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems, 32, 2019.
Daniel E Worrall, Stephan Garbin, Daniyar Turmukhanbetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2037,2011 2017.
Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual comput- ing and beyond. In Computer Graphics Forum, volume 41, pp. 641-676. Wiley Online Library, 2022.


Reviews:
[{"review_id": "iy31cltijZv", "reviewer": "Reviewer_j1mL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors summarize and analyze the existing ways (or called conditioning methods) that use edge attribute information together with node embeddings in GNN. The intuition of each conditioning method is explained for comparison. In experiments, the authors use a GNN model as the baseline to show the difference between these conditioning methods on benchmark datasets.", "summary_of_the_review": "Even though the existing conditioning methods in GNNs are summarized and analyzed, the authors only provide limited experiments. It would be better to use more powerful GNN as baseline and more comprehensively evaluate the conditioning methods.", "strength_and_weaknesses": "Pros:\n1. The existing ways of using additional information when updating node embeddings in GNNs are summarized and analyzed.\n2. The paper is easy to follow and well-organized.\n\nCons:\n1. The addressed methods are all based on existing works without having new conditioning methods.\n2. The experiments are only done based on EGNN, which is not a well strong model in the related field. Besides, the empirically compared conditioning methods are only focusing on geometric distances.\n3. The models except for EGNN and EGNN variants in Table 1 and Table 3 seem to be unrelated to the discussions. These models perform much better than the EGNN variants and the authors do not mention them at all.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WimyDPnzLhW", "reviewer": "Reviewer_xVZv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors analyze the three different message-passing schemes of Graph Neural Networks conditional on the geometric property, namely weak, strong, and pure methods, and empirically study their effect.", "summary_of_the_review": "This paper studies an interesting problem in geometric GNN design. However, it has limited novelty and has drawn few meaningful conclusions.", "strength_and_weaknesses": "In this paper, the authors analyze the three message-passing schemes of Graph Neural Networks conditional on the geometric property, namely weak, strong, and pure methods, which relate to concatenation, gating, and transformations depending on the geometric attributes. To compare the effects of these methods, the authors perform experiments on molecular datasets and conclude that the strong method, i.e., the gating method, not only has higher computational efficiency but also achieves better performance in most cases.\n\nStrength:\n1. The paper is well-written and easy to read.\n2. The paper focuses on an interesting problem, i.e., how to employ geometric attributes in GNNs properly.\n\nWeaknesses:\n1. The novelty and significance of this paper are concerned. As mentioned in the paper, the three methods have already been raised in previous work. And this paper only summarizes these methods and only gets some trivial results through experiments without theoretical analysis. The paper proposes an interesting question, but it doesn't lead to a meaningful conclusion.\n2. The experiment is not sufficient. The authors only compare the three message-passing methods based on the EGNN model. In order to get a more reasonable conclusion, it is necessary to conduct experiments on several basic models.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "EhUaqNqfbJH", "reviewer": "Reviewer_QMHG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes three types of conditioning (weak, strong, and pure) to unify some conditioning methods in the literature. Two empirical testing studies show that strong conditioning can perform better than weak conditioning and more efficiently than pure conditioning.", "summary_of_the_review": "Overall, it is insightful to unify current models into a framework. However, more numeric and theoretical supports are encouraged. ", "strength_and_weaknesses": "Strengths:\n\n1. A general framework of conditioning methods is proposed.\n\n\nWeaknesses:\n\n1. In the experimental section, it would be great if the authors could show more statistics on the results, like standard deviation, so that it can further show that improvement made by strong conditioning is statistically significant. Also, some experimental setups are missing, like the number of runs for the two tasks.\n2. Even though the authors empirically justify that for the EGNN instance, conditioning methods help boost performance, can any theory be developed to verify it?\n3. Since the authors only show the advantage of strong conditioning over weak conditioning on the EGNN model, it is not convincing enough to assert that this will still hold for other models. Therefore, It is encouraged and interesting to see the applications of these conditioning methods on a few more models.\n\n\nQuestions:\n\n1. Are these conditioning methods designed only for geometric GNNs, i.e., GNNs with graph data input with Euclidean embedding?\n2. In Table 2(b), is there any reason why strong-EGNN is much worse than weak-EGNN in predicting $\\mu$?\n3. Is there any reason why EGNN is not compared on the MD17 dataset?\n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject