Symmetry-driven graph neural networks
Authors: Francesco Farina
Emma Slade
Venue: ICLR 2022 Submitted (2022)

Abstract: Exploiting symmetries and invariance in data is a powerful, yet not fully exploited, way to achieve better generalisation with more
efficiency.  In this paper, we introduce two graph network architectures that are equivariant to several types of transformations affecting the node coordinates.  First, we build equivariance to any transformation in the coordinate embeddings that preserves the distance between neighbouring nodes, allowing for equivariance to the Euclidean group. Then, we introduce angle attributes to build equivariance to any angle preserving transformation - thus, to the conformal group.  Thanks to their equivariance properties, the proposed models can be vastly more data efficient with respect to classical graph architectures, intrinsically equipped with a better inductive bias and better at generalising.  We demonstrate these capabilities on a synthetic dataset composed of $n$-dimensional geometric objects.  Additionally, we provide examples of their limitations when (the right) symmetries are not present in the data.

Under review as a conference paper at ICLR 2022
SYMMETRY-DRIVEN GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Exploiting symmetries and invariance in data is powerful, yet not fully exploited, way to achieve better generalisation with more efficiency. In this paper, we in- troduce two graph network architectures that are equivariant to several types of transformations affecting the node coordinates. First, we build equivariance to any transformation in the coordinate embeddings that preserves the distance between neighbouring nodes, allowing for equivariance to the Euclidean group. Then, we introduce angle attributes to build equivariance to any angle preserving transfor- mation thus, to the conformal group. Thanks to their equivariance properties, the proposed models can be vastly more data efficient with respect to classical graph architectures, intrinsically equipped with a better inductive bias and better at generalising. We demonstrate these capabilities on a synthetic dataset com- posed of n-dimensional geometric objects. Additionally, we provide examples of their limitations when (the right) symmetries are not present in the data.
INTRODUCTION
Symmetries exist throughout nature. All the fundamental laws of physics are built upon the frame- work of symmetries, from the gauge groups describing the Standard Model of particle physics, to Einstein's 's theories of general and special relativity. Once understands the symmetry of a certain system, powerful predictions can be made. A notable example is that of Gell-Mann's is eightfold- way (Gell-Mann) [96T). built upon the symmetries observed in hadrons, that led his prediction of the Ω baryon, which was subsequently observed 3 years later (Barnes et al. 1964). The study of symmetries and invariance deep learning has recently become field of interest to the community (see, e.g., (Bronstein et al. 2021) for a comprehensive overview), and rapid progress has been made in constructing architectures with group theoretic structures embedded within. Two fundamental architectures in machine learning, the convolutional and graph neural networks, are invariant to the translation and permutation groups respectively.
Graph networks in particular are designed to learn from graph-structured data and are by construc- tion invariant to permutations of the input nodes. They were originally proposed (Gori et al.) 2005 Scarselli al. (2008) and have received a lot of attention in the last years (see, e.g., (Battaglia ct al. 2018 Hamilton 2020; Wu et al. 2020) for a comprehensive overview). Due to their prop- erties, they find application in broad range of problems like learning the dynamics of complex physical systems (Sanchez-Gonzalez et al., 2020 [faff et al. 2021), particle identification in parti- cle physics (Dreyer & Qui 2021). learning causal and relational graphs (Kipf al. 2018 Li et al.) 2020), discovering symbolic models (Cranmer et al. 2220)), as well as quantum chemistry (Gilmer al. 2017) and drug discovery (Stokes al. (0200)
In this work, we decouple node coordinates from other node attributes to obtain invariance (and possibly equivariance) many transformations affecting the node coordinates, possibly belonging to important groups. in addition to permutation invariance. First, we define the distance preserving graph network (DGN) whose updated node, edge and global attributes are invariant to any transfor- mation in the coordinate embeddings that preserves the distance between neighbouring nodes, while node coordinates can be updated an equivariant way. Examples of such transformations are rota- tions and translations, but also transformations the Hoberman sphere, whose 3D shape changes while preserving distances between connected nodes. Equivariance to dilations in the coordinates be obtained via the conformal orthogonal group with the inclusion of an additional input layer. Then, we define the angle preserving graph network (AGN), whose updated node, edge and global attributes (resp. node coordinates) are invariant (resp. equivariant) any transformation preserving

---

Under review as a conference paper at ICLR 2022
the angles between triples of neighbouring nodes in the graph (a notable example being molecular conformations) and, thus, to the n-dimensional conformal group on R"..
By constructing such architectures, enable wide range of possible transformations to be per- formed on graph-structured data, with the only requirement being that distances or angles between coordinates of neighbouring nodes preserved. This means that the updated attributesccoordinats of both networks are invarian//quivaaiiat to translations, rotations, reflections (i.e., the Euclidean group, En)),, with the AGN allowing also invariancee/quivariane orthogonal rotations. In practice, this means the networks are able to filter out many copies of the same input whose coordinate embeddings have been transformed and therefore learn more efficiently than architectures which consider the inputs as distinct. In other words, a single sample contains the same information as many copies of obtained by appropriately transforming it.
dilations, inversions and non-
Finally, while the two architectures we present are partially overlapping in terms of symmetries, it important to consider specific use cases where the use of one architecture over another would be preferred. We test the architectures on synthetic dataset consisting of n-dimensional geometric shapes and other benchmark datasets. Moreover, we show examples of cases in which using our architectures is counterproductive due to the lack of symmetries in the data.
2
BACKGROUND
2.1 GROUP THEORY AND EQUIVARIANCE
Group theory provides the mathematical formulation for symmetries of systems, with symmetry operations represented by individual group elements. A group can be defined as a set G equipped with a binary operation, which enables one to combine two group elements to form a third, whilst preserving the group axioms (associativity, identity, closure and inverse). A function unaffected by a group action said to be invariant, which is particular instance of equivariance. Formally, let X C R. and 4gg X + X be transformation on X for group element g G. Then, the linear map += X Y, Y C IR,, is equivariant to G if Vi E G. 3. + Y such that
(x)((x)) = (x))))). Mpg X
When vy is the identity, say that & is invariant to G. The functional form of O is determined by the specific group of interest. It is worth pointing out that, whilst not often expressed in group theo- retic notation, equivariance exists in common deep learning architectures; the convolution operation used in CNNs equivariant under the translation group and approximate invariance is typically achieved via pooling operations. Graph neural networks invariant the permutation group.
Euclidean group The translation and orthogonal groups (T(n) and (n)) respectively) act through translations, orthogonal rotations and reflections. The semidirect product of these two groups E(n) = T(n) x O(n) is known as the Euclidean group, which consists of transformations that preserves distances. A function that is invariant under E(n) is |xx x1122, for any x; X, E E" (see Appendix A for proof).
Conformal group The conformal group Con is the group of transformations pp R Rn that preserve angles. Formally, for any triple of vectors Xj, Xi, X E IR.", let denote by L(x,, Xx, Xs) the angle centred on x; with rays given by x, X; and X X;, Then, conformal transformation to satisfies
L(x,,
<(x((x)), sxxx)) L(x,, X1, X1,
By definition, the transformations of the conformal group include translations, rotations, reflections (which collectively form the Fuclidean group E(n), as well as dilations, inversions and other fea- tures. An important subgroup the conformal orthogonal group, which requires transformations to be orthogonal (and thus, inversions not allowed).
2.2 GRAPHS AND COORDINATE EMBEDDINGS
A graph is defined as g(V, E) where V/ = (I.....N) N. is the set of nodes and È = ((j.))} [ CVVV the set of (directed) edges connecting nodes in > (where j and denote the source and target nodes
2

---

Under review as a conference paper at ICLR 2022
Il Iy yil
clbuan sil dlywyuys)
K x]1 by -1
Figure I: Representation of relative distance (left) and angle (right) preserving maps.
respectively). We define NG a {i (i,z) E E] as the set of (in- Jneighbours of node i. Moreover, we associate node and edge features, V, E R=, Vi E V, and ejs R V(j E E respectively, to each node and edge in the graph as well as a global attribute E TRu We assume that node features V, consist of node coordinates x; E Rd* and additional features h; E IRa (unrelated to coordinates), so that V; = [x,, h.]. n2e = n2 + Ri. With a slight abuse of notation, we denote Gx(V,E) E.) and (V,E) as different coordinate embeddings of the same graph, meaning that they differ only in their node coordinates x,, while having the same node, edge and global attributes h, Cj and u. Finally, let E V V V be the set of (ordered) triples of nodes in graph G that form an angle, i.e., A é (G.i,A) j.k NT", jj & k, Vi E V} with N₁" & i (i.t) V (i.j) 8} being the set of in- and out-neighbors of node i.
We define relative distance and angle preserving maps as follows (see Figure T
Definition. Let Gx (V, E) and GY (V. E) be such that ||x: x lly; yj 11 V(i,j) E E. Let
x (i, and y rbii, Gy ) for some function IR + E" K E Zt. We say that sp is
relative distance preserving map if ||x; -x311² Ily y 112, V(i.j) EE
Definition.
yi, v(j,,k) E A. Let x (i, and y vi(i. for some function R + R", K Zt. We say that ri relative angle preserving map if (xx X xx) / (y) y yt), v(j,i,k) E A.
Let 9x (V,E) and Gr (V,E) be such that L(x,, x,, xx)
The simplest map satisfying the above definitions is the identity function
rb 9x) =
which keeps the coordinates unchanged during the update step. Updating all the coordinates in the same way also trivially preserves both relative distances and angles. In the case where the coordinate embedding X is a Conformal orthogonal transformation of a possible map y is defined by
+
Σ JEN.
ax(xx; - x:).
with azi possibly being parametric function of other graph attributes. A thorough discussion and proof' are reported in in Appendix
EQUIVARIANT GRAPH NETWORKS
We start this section by recalling the structure standard graph network block. Then, we intro- duce two novel architectures with additional invariance and equivariance properties with respect to different node coordinate embeddings. We stress that while we build our discussion upon equiv- ariance with respect node coordinates (since there is, in general, sense of equivariance of the network's 's node or edge embeddings), there may be cases in which our discussion can be extended to node or edge embeddings.
3

---

Under review as a conference paper at ICLR 2022
(a) GN
(b) DGN
(c) AGN
Figure 2: Graphical representation of the graph architectures
3.1
THE STARTING POINT GRAPH NETWORK BLOCK (GN)
general standard graph network block can be defined in terms of edge, node and global updates
ej 0² (V Vs. ejt, es,
v(Ji)
(4a)
v = o² (Vi. (e)))
ViEV
(4b)
ut (0" (tv, hev),, p (1e Hetth..ee)) u)
(4c)
where d² tn. update functions (usually defined as neural networks whose parameters are to be learned) and po pp-+", pm- aggregation functions reducing of elements of variable length single one via an input's permutation equivariant operation like element wiise summation or mean.
_ 0² tn + R d" RT + +n, - Re
3.2 DISTANCE PRESERVING GRAPH NETWORK BLOCK (DGN)
By decoupling the updates of node coordinates and features (x; and h,), the DGN block is defined through the following updates
et のه (e h hy, || 15xxx,1 x))...u. h Φ² (p ((e ((et)ieNN)) hi, u),
v(i
(5a)
ViEV
(5b)
x 20t (i,xx),
ViEV
(5c)
u ("(p="
((e (nne)) ( hev), p² (fllxt xx 11) u)
(5d)
where de 2n.+2na+nut1 R", dh R +ma - R"A 6² Ro. +n +n
R
the update functions, p pr n are aggregation functions and vith RRK - R= E Z some, possibly parametric, relative distance preserving map.
Because of the way in which node coordinates are processed to update edge and global embeddings (i.e., only through their relative distances), it can be easily seen that e h and ut are, by con- struction, invariant to any transformation of the input coordinates that locally maintains their relative distances along the edges defined by the graph structure notable example being the one of Eu- clidean transformations; see Appendix D for proof). Moreover, updated node coordinates x; can be invariant or equivariant to (some of) those transformations, depending on the particular structure of ohah. For example, using equation 3 or the trivial identity function (equation 2 would result in coordinates being updated in an equivariant way with respect to Euclidean transformations, with ajı possibly being parametric functions (e.g., ajt d" (ex))
3.3 ANGLE PRESERVING GRAPH NETWORK BLOCK (AGN)
Given a graph G(V let ajik R be the angle embedding associated to each angle (j, k) e A and assume that C'i contain no information about the angles /(xj, X,, Xx). Moreover, define A;
4

---

Under review as a conference paper at ICLR 2022
as the set of (ordered) couples of nodes forming an angle whose vertex is node i, i.e., A, _ {(i
k) E A. = i}. Then, the AGN block is then characterised by the following updates
j (h, by, hys jeck, X,, x4, xx), u),
v(j,i E A (6a)
ej of (h,, ha, ejt, u)
v(i. E (6b)
h Ø² "h: ph (be p jus (GNTA))))))))
Vi WEV (6c)
x (i,)))
Vi WEV (6d)
u = o""p"
((ee))
p (H
(6c)
where ,0,0", ,0, Φ², vttt, 6" are the update functions] and ph pa-th ph ,p pu are the
aggregation functions, with 2h being (possibly parametric) relative angle preserving map. Vari- ants of this network that still preserve its properties can be easily constructed by, egg, using edge embeddings to update the angle embeddings, using angles to update edges or not considering an- gle attributes at all. Some examples are reported in Appendix F.I It can be casily seen that jik ej,, h,"",, ut are invariant to any transformation of the coordinate embeddings that preserves the angles created by neighbouring nodes in the graph, while updated coordinates x can be invari- ant equivariant to (some of) the same transformations, depending on the structure of r/"". As with the DGN, equivariance to the conformal orthogonal group can be achieved by using equation 3 A particular case of transformations the AGN can deal with is the one of conformal transfomations (see Appendix E for a formal discussion).
DISCUSSION
In the previous section we introduced two novel graph architectures. To construct them we have mainly built upon the actions on a coordinate system of two transformations that preserves distances and angles between neighbouring nodes in a graph. These transformations include as particular cases those in the Euclidean and conformal groups. Equivariance under E(n) means equivariance to orthogonal rotations and translations, while by considering the full conformal group, we can fur- neural network architecture. Conformal n--dimensiona transformations consist
ther generalise of the groups containing translations, dilations, rotations and inversions with respect to an n sphere. A conformal transformation is therefore powerful tool for mapping data points onto each other, and hence, building a neural network architecture invariant to the conformal group enables the architecture be invariant wide selection of interesting subgroups. By introducing distance and angle preserving transformations that take into account the structure of the graph, invariance and equivariance under more general transformations have been achieved. A notable example is that the DGN architecture, whose updates are invariant to any transformation of a Hoberman sphere, while both the DGN and AGN can be invariant to transformations affecting only subsets of their nodes, as shown below.
In summary, by taking advantage of the powerful tools of group theory, and performing operations on vector space which are invariant or equivariant under group transformations, we can build a neural network architecture which can take advantage of these group properties. Such an architecture will be able to deal with rotated, translated, dilated (or more generally transformed) data more efficiently than a standard graph network which does not have the above group properties built into it.
4.1 BEYOND GLOBAL TRANSFORMATIONS
So far, we have talked about global transformations in the coordinate space. In general, global symmetry Pg. as defined in Section 2.1 acts on a function vb(x) as
(x)) + (x).
(7)
local group symmetry (x) is instead defined as
(x) + (x))
(8)
'The update functions in equation live in the function spaces da tra R d² R tnett ~ 5² R + to the 1 t R R"= for appropriate K E 2 depending the actual number of parameters, and d² RR 7 +ut +n tm. R=
5

---

Under review as a conference paper at ICLR 2022
DGN
AGN
Figure 3: Equivariance of the two networks. The green/red square denotes the base graph and the
blue/black ones alternative coordinate embeddings, equivariant for the DGN (red) or AGN (green).
where now the group action can differ all points.
The architectures we present are also able to deal with some local symmetries. Namely, the DGN and AGN preserve local distances and angles respectively where a suitable subgraph must be defined. As an example consider transformation that rotates only some nodes of graph; all nodes which are in the neighbourhoods of the nodes will, in the angular architecture, have their angles preserved, and in the distance-prest preserving architecture, the distances between nodes will be preserved. A pertinent use-case for such a symmetry is that of molecular conformations, where the spatial arrangement of the atoms can be interconverted by rotations about formally single bonds). Both the DGN and the AGN updates would be invariant to such a transformation at the level of the two distinct subgraphs separated by the bond in question, thus enabling one to learn conformation iiviooiinnnniiin properties.
4.2 COMPARING AND EXTENDING THE TWO ARCHITECTURES
As mentioned, the two architectures we presented are partially overlapping in terms of equivariance features. Whilst both of them contain invariance to E(n), the DGN can deal with non-orthogonal transformations that preserve the distance between neighbouring nodes (such as with the Hoberman sphere) while the AGN is invariant to the conformal group which includes non-orthogonal transfor- mations that preserve angles but not distances (see Figure 3
The DGN can be equipped with a sense of scale invariance through an input scaling layer that makes invariant to the conformal orthogonal group. Let 7 = a/ max((.))e ||x;- - x;|l for some E IR, where we identify y satisfying the dilation condition required under the conformal or- thogonal group. Then, conformal orthogonal invariance can simply be obtained by computing scale- normalised coordinates X, 7x; and using them as the coordinates in equation
Introducing coordinate scaling therefore enables the DGN to be scale invariant, under the condition that all transformations remain orthogonal. With the AGN, invariant under the full conformal group. This larger group invariance enables us to work with very powerful transformations on data, which can also be dangerous. For example, if the scale is an important property of the dataset, the AGN will not recognise it and so will learn incorrect information about the data. Similarly, one must also not use the DGN blindly and irrespective of the properties of the dataset; as the architecture is invariant to transformations that preserve distance between neighbouring nodes but not angles it is able to, for example, map a square onto line. The two architectures we presented can be also combined together at the price of losing some features, while generalising to both distance and angle preserving transformations.
relax this condition and be
5 RELATED WORK
The study and formulation of group equivariant neural networks have flourished in the last years and they have proven to be extremely powerful in many tasks. The first example is probably that of convolutional neural networks (LeCun et al., 1990). which explots translation equivariance and invariance, thanks to the convolution and pooling operations respectively, and have led to break- throughs in most vision tasks. CNNs have been generalised to exploit larger groups of symmetries. G-CNNs are proposed in (Cohen& & Welling (016)) to deal with more general symmetries. includ- ing rotations and translations, and extended (Bekkers) 2020 Finzi et al. 2020) to deal with Lie groups. Equivariance to arbitrary symmetry groups can be achieved via self-attention mechanisms in (Romero & Cordonnier 2021). Continuous convolutions are used in SchNet (Schütt et al. 2017)

---

Under review as a conference paper at ICLR 2022
to achieve E(n) invariance, and (3) equivariance is achieved in Thomas al. (2018) via the use of spherical harmonics. The drawback of many of these methods is their limited applicability due to computational complexity.
There has been some work on constructing general MLPs that are equivariant to different groups; layer equivariant to general matrix groups is presented in Finzi et al. (202T) whilst equivariance to the Lorentz group for physics applications has recently been explored (Bogatskiy et al.) 2020). Further applications include (Mattheakis et al., 2019), where physical symmetries are embedded in neural networks via embedding physical constraints in the structure of the network, and in (Baren-
boim et al. 202I) where the ability of neural networks to discover and learn symmetries in data is
explored.
Graph neural networks are, by construction, permutation invariant (Scarselli al. 2008 Battaglia et al. 2018). Recently, there has been lot of work on building equivariance to other interesting groups in GNNs. There has been particular interest the Euclidean group with results for the
subgroups SE( and E(3) obtained in (Thoma et al. 20188 Fuchs et al. 2020; Köhler et al. 2020
Finzi et al. 2020: Batzner et al. 2021 Yang et [ 2020). Extending these architectures, a message- passing convolutional graph layer is proposed in (Satorras et al. 2021 which implements E(n) invariance for edge and node features and equivariance for node coordinates, leading to state of the art results in variety of tasks, including n-body particle systems simulations and molecular property predictions. A similar idea has been also proposed in (Farina & Slade 2021), where a general ((n) equivariant GNN is presented, and in (Horie et al. 2021), in which equivariance isometric transformations is considered. These (mn) equivariant networks can be seen as special instances of our DGN architecture, where particular aggregation functions are applied and particular choice is made for the distance preserving map x/" (see Appendix G In (Satorras et al.] [202]). for example, the map 1 is chosen so that relative distances between all possible pairs of node in the graph are preserved, thus allowing only for rigid transformations in E(n).
Angular information is used in (Smith et al., (017) but no attention explicitly paid equivari- ance. DimeNET is proposed in (Klicpera et al. (020)), where both distance and angle embeddings computed through embeddings in novel orthogonal basis functions are used in a message passing graph network to achieve equivariance. State of the art results are obtained on molecular property and dynamics prediction datasets. We note, however, that while DimeNET produces state of the art results, it is restricted to atomic data, due to the requirement of Gaussian radial and Bessel functions as the basis representations. Additionally, angular embeddings computed in the isometric invariant layer in (Horie et al. 2021) corresponds to the extraction of both relative distances and angles of each pair of vertices. Again, these approaches are special instances of our networks where particular aggregation and learnable functions or hand engineered distance and/or angle embeddings (see Appendix G). In this sense, our network is unconstrained and can learn what it needs. More- over, thanks to its general form can achieve invariance the conformal group (and more) which, to the best of our knowledge, does not hold for known architectures.
used
In summary, most of the existing equivariant graph architectures are at most equivariant to the Eu- clidean group (represented as the intersection set in Figure 3), whilst the architectures we propose allow, in their general form, more general transformations.
EXPERIMENTAL RESULTS
In this section we give an insight into the performance of the architectures we proposed and on their limitations. First, we demonstrate that perfect generalisation to unseen data can be achieved on datasets with a large number of symmetries, possibly accompanied by a faster convergence rate (as shown for QM9 in Appendix H.44). Then, we show that, when used on datasets with few sym- metries, using our architectures can be counterproductive. are present, equivariant architectures experience both increased accuracy and convergence speed, as demonstrated by state of the art results obtained on a number of tasks by architectures that can be obtained as particular instances of the ones proposed here. Examples include n-body system dynamics prediction, molecular dynamics and molecular property prediction tasks (Satorras et al.) 2021 Klicpera et al. 2020 Horie et al. 2021).
general, when the right symmetries
7

---

Under review as a conference paper at ICLR 2022
6.1 POLYTOPES CLASSIHICATION
We consider a nn-dimensional polytopes classification problems for n = 3,4, 2,4.5. The datasets are composed of graph representations of regular polytopes and the number of classes varies with n. Simplexes, hypercubes and orthoplexes are considered for all values of n. For n = 3 also dodeca- hedra and icosahedra are added, while for n we consider 24-. 120- and 600- cell polytopes. Additional results are reported in Appendix polytope where the graph is specified in terms node coordinates x; (so that the resulting polytope is inscribed in hypersphere of radius 1) and list of edges & (and possibly angles A); node, edge and possible angle attributes (h,, ejá, ((ja)) are fixed to be equal to 0 for every node in the graph. This makes the problem intrinsically harder since only coordinates can be used to discern between polytopes. The test set is composed of randomly transformed versions of those in the training set, whose coordinates are obtained x; yAx; + q for some Y I, A IR"x and q E R". Note that, while even a standard GN could learn to correctly classify transformed polytopes, a much larger training dataset is needed in order do so, as shown in Appendix H
The training dataset consists of a single graph per
We compare graph networks built with AGN, DGN and standard (GN) blocks, as well as EGNN (Satorras et al. 202T), DimeNET (Kliepera et al.. (220)) and NE3-Transformer (Fuchs al. 2020), the latter only for n 3. The DGN block is also combined with a scaling layer (SDGN) as in Section 4.2 to obtain equivariance dilations. Architectures based AGN, DGN and GN blocks employ sum aggregation functions p and 20 set to equation (other cases are considered in the appendix). Results are reported Table
AGN
SUGGN
DGN
GN
SES-Trans
DimeNer
BGNN
tminnin acouncy
Orthogonal
0.44
0.15
Onthogooal dilation
0.45
0.36
0.02
Non-orthagoual fy- 0.5)
0.44
0.47
0.15
0.34
0.33
0.08
Non-erihugooull (xx = 1.5)
0.93 +: 0.07
0.41
: 0.07
0.44
0.17
1
0.33 + 0.10
0.33
0.12
Non-orthagyeal (jr 3.01
0.83 + 0.15
0.37 : 0.04
0.43
0.15
0.31
0.03
0.31
0.07
crinning accunacy
NoW BOO SUCCEN tott 3xcuniyy
Orthogonal
1
0.53 0.04
Orthogonal + dilation
0.61 1 0.05
0.51
0.03
0.52 1 0.06
0.55 E 0.07
Non- Non-orthogoual
0.96
0.60
0.30
0.66
Non orthogonal
1.5)
0.83
0.61
0.06
0.51
0.54
Non-orthogceal (y+ 3.0)
0.77 + 0.17
0.59 + 0.08
0.51
0.04
0.53 + 0.03
0.63 + 0.11
(minning
Ohtlogonal
Orthogonal dilation
0.46 1 0.05
0.55
0.04
0.58 0 06
0.55
0.06
Non- Non-erihogonal (fr = 0.51
0.99 1: 0.01
0.41
1 0.05
0.60
0 0.13
0.44
: 0 03
0.41
0.04
Non orthogonal
1.5)
0.98
0.02
0.39
0.61
0.16
0.45
0.
0.03
orihogonal
0.02
0.08
0.04
Table 1: Polytopes classification: training and test accuracy (mean + standard deviation over 10 runs) for different transformations in the test set.
We note that all models reach a perfect accuracy on the training set. As for the test accuracy, results vary across the different models, in line with what one expects from their invariance properties. The standard GN cannot generalise to any type of transformation. When only orthogonal transformations are performed in the test set (i.e., A 7 = 1) all models (except GN) perfectly generalise to unseen data. However, as expected, when also adding dilations (i.e., A A = yI, 7 e R), only AGN and SDGN (and the Transfooffrrm for adding random non-orthogonal transformations (i.e., μ = B[| possibly unexpected occurs. The AGN (and partially also the SDGN and the TranSrrasfrrrrr for n 3) perfectly generalises to unseen polytopes whose neither angles nor edge lengths have been preserved (due to the non-orthogonality of A). This is probably due the networks having learnt to distinguish polytopes based on the sum (or mean) of their angles which is preserved under non- orthogonal transformations due the Gram Euler theorem. While this behaviour turns out to be useful in the task at hand, one can easily envisage cases in which it may cause issues.
3) can generalise well. Finally, when
A[]| A - [[]r]] > 0. 7 E R), something
6.2 BENCHMARK DATASETS
Now we consider benchmark datasets from (Dwivedi et al. 2020 for which node coordinates are provided. In particular, we consider MNIST, CIFARIO as graph classification problems and TSP as an edge classification one. When treated as images, MNIST and CIFARIO are usually examples of datasets containing symmetries. However, when represented as graphs in (Dwivedi al. 2020
8

---

Under review as a conference paper at ICLR 2022
those symmetries are mostly lost. This is due to, for example, the fact that both the foreground and the background of the images are embedded the graphs. Furthermore, in TSP there are no apparent symmetries. In these cases, one may expect, using equivariant network would not produce any benefits. Rather, as we show, negative effects can appear.
We trained our models and standard GNN these datasets. All models have roughly the same number of parameters and the results are reported in Table 2 As can be seen, treating node co- ordinates if they can posses some symmetric between among different samples turns out to be counterproductive both in terms of convergence speed and accuracy. In particular we see that while the standard GN has the best performance, the DGN slightly better than the AGN. This is probably due to the fact that, while some non- -onooroooon transformations mostly preserving local distances may be present the dataset, angle preserving ones are much more rare. On TSP in particular, using angle related properties results in significant drop in performance.
MNIST
CIFARIO
TSP
aoc
lest acc
train auc
ace
(positie)
lest (positive)
0.943
0.004
0.932
0.004
0.644
0.004
0.530
0.004
0.632
0.020
DGN
0.957
+: 0.001
0.945
0.006
0.857
4: 0.004
0 0.592
1: 0.002
0.748
0.013
0.771 1: 0.014
GN
0.982
+ 0.001
0.977
+ 0.002
0.719
+ 0.007
0.687 + 0.001
0.772 + 0.041
0.793 + 0.033
Table 2: Benchmark datasets. Train and test accuracy reported for MNIST and CIFARIO. For TSP, due to the high class unbalance, the train and test F1 score for the positive class is reported.
6.2.1 COMPUTATIONAL COMPLEXITY
All the experiments have been run on an NVIDIA V100 GPU. The average time required for each training step (i.e., forward pass, back-pro propagation and optimiser step for a batch of data) is reported in Table for the architectures presented in this paper, normalised with respect to the batch-size. For the polytopes classification experiment we report the results obtained by using equation and equa- tion 3 parentheses). Using the scaling layer for the DGN drastically increase the computational complexity. Moreover, it can be seen that while the AGN and DGN have similar complexities for dataset in which the number of edges (and hence angles) is relatively low, when graphs are (nearly) complete (like in TSP) highly connected, the AGN has an overhead with respect to the DGN (due to the need of updated and computing angles in addition to edges).
Time training step (ms)
polytopes (n=))
polylupes (ran))
polyopes (n=3)
MNIST
CIFARTO
TSP
QMV
batch-size
128
512
AGN
3.3.338)
3:3.8)
(63)6
1.64
35
0.23
SDGN DGN
(.4
14.5 (15:2)
(8)
7.81
12.5
333))
(3.7)
5.3 (6.3)
0.78
1.09
16
0.19
3.2
52
0.77
1.02
Table 3: Time per training step in the various experiments, normalised with respect to the batch-size.
7 CONCLUSION
In this paper we have presented novel deep learning architectures which are equivariant to distance and angle preserving transformations in graph coordinate embeddings. In particular, we have shown invariance or cquivariance to the En)), CO" Q) and Conf( groups in addition to permuta- tion invariance. Invariance to local symmetries can also be achieved when the different transforma- tions applied suitable subgraphs. We have applied our models to a synthetic dataset composed of n--iimnnsional regular polytopes as well as to several benchmark datasets. We have shown that the architectures we propose are significantly more accurate and data efficient than a standard graph network on datasets where there are large numbers of symmetries in the data. We also explicitly show examples where our architecture produces unexpected results or would not be applicable, due to lack of symmetry in the data.
REFERENCES
Gabriela Barenboim, Johannes Him, and Veronica Sanz. Symmetry meets AL. SciPost Phys., 11:14,
2021.

---

Under review as a conference paper at ICLR 2022
V. E. Barnes, P. L. Connolly, D. J. Crennell, B. B. Culwick, W. C. Delaney, W. B. Fowler, P. E.
Hagerty, E. L. Hart, N. Horwitz, P. V. Hough, J. E. Jensen, K. Kopp, K. W. Lai. J. Leitner, L. Lloyd, G. W. London, T. W. Morris, Y. Oren, R. B. Palmer, A. G. Prodell, D. RadojičiC, D. C.
Rahm, C. R. Richardson, N. P. Samios, J. R. Sanford, R. P. Shutt, J. R. Smith, D. L. Stonehill, R. C. Strand, A. M. Thorndike, M. S. Webster, J. Willis, and S. S. Yamamoto. Observation of Hyperon with Strangeness Minus Three Physical Review Letters, 12(8):204 206, 1964.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre. Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals. Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. 2018.
Simon Batzner, Tess E. Smidt, Lixin Sun, Jonathan P. Mailoa, Mordechai Kornbluth, Nicola Moli- nari, and Boris Kozinsky. -euuivaquv graph neural networks for data-efficient and accurate interatomic potentials. 2021.
Erik J Bekkers. B-spline CNNs on Lie groups. In International Conference on Learning Represen- tations, 2020.
Alexander Bogatskiy, Brandon Anderson, Jan Offermann, Marwah Roussi, David Miller, and Risi Kondor. Lorentz group equivariant neural network for particle physics. International Confer- ence on Machine Learning, pp. 992-1002. PMLR, 2020.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13778 2021.
Taco Cohen and Max Welling, Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999. PMLR, 2016
Miles Cranmer, Alvaro Sanchez Gonzalez, Peter Battaglia, Rui Xu, Kyle Cranmer, David Spergel, and Shirley Ho. Discovering symbolic models from deep learning with inductive biases. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 17429-17442. Curran Associates, Inc., 2020.
Frédéric A. Dreyer and Huilin Qu Jet tagging in the Lund plane with graph networks. JHEP, 03: 052, 2021. doi: 10.1007/HHEP03222100522
Vijay Prakash Dwivedi, Chaitanya K Joshi, Thomas Laurent, Yoshua Bengio, and Xavier Bresson. Benchmarking graph neural networks. arXiv preprint arXiv:2003.00982) 2020.
Francesco Farina and Emma Slade. Data efficiency in graph networks through equivariance. arXiv preprint arXiv:2 106. 13786, 2021. Presented at the ICML. 2021 Workshop on Subset Selection in Machine Learning: From Theory to Practice.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolu- tional neural networks for equivariance to lie groups on arbitrary continuous data. In International Conference Machine Learning, pp. 3165-3176 PMLR, 2020.
Marc Finzi, Max Welling, and Andrew Gordon Gordon Wilson. A practical method for constructing equivariant multilayer perceptrons for arbitrary matrix groups. Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 3318-331 33288. PMLR, 18-24 Jul 2021.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se(3)-transformer:: 3d roto- translation equivariant attention networks. H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 1970-1981. Curran Associates, Inc., 2020.
M Gell-Mann. The eightfold way: A theory of strong interaction symmetry. 3 1961.
10

---

Under review as a conference paper at ICLR 2022
Justin Gilmer, Samuel Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pp- 1263-1272. PMLR, 2017.
Marco Gori, Gabriele Monfardini, and Franco Scarselli, new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference Neural Nenworks, 2005., volume 2, pp. 729-734. IEFE, 2005.
William L. Hamilton. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning, 14(3):1-159, 2020.
Masanobu Horie, Naoki Morita, Toshiaki Hishinuma, Yu Ihara, and Naoto Mitsume. Isometric trans- formation invariant and equivariant graph convolutional networks. In International Conference Learning Representations, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR (Poster), 2015.
Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems. In International Conference on Machine Learning, pp. 2688- 2697. PMLR, 2018
Johannes Klicpera, Janek GroB, and Stephan Günnemann. Directional message passing for molec- ular graphs. In International Conference on Learning Representations, 2020.
Jonas Köhler, Leon Klein, and Frank Noe. Equivariant flows: Exact likelihood generative learning for symmetric densities. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th In- ternational Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5361 133000377.. PMLR, 13-18 Jul 2020.
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence D Jackel. Handwritten digit recognition with a back precppprppggtiinn net- work. Advances in neural information processing systems, pp- 396 196-404, 1990.
Yunzhu Li, Antonio Torralba, Anima Anandkumar, Dieter Fox, and Animesh Garg, Causal discov- ery in physical systems from videos. In H. Larochelle, M. Ranzato, R. Hadsell, F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 9180-9192. Curran Associates, Inc., 2020.
Marios Mattheakis, Pavlos Protopapas, David Sondak, Marco Di Giovanni, and Efthimios Kaxiras. Physical symmetries embedded in neural networks. arXiv preprint arXiv:/904.008 2019.
Song Mei, Theodor Misiakiewicz, and Andrea Montanari. Learning with invariances in random features and kernel models. arXiv preprint urXiv:2102.13211 2021.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez-Gonzaleez, and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference on Learning Representationss 2021.
David W. Romero and Jean-Baptiste Cordonnier. Group equivariant stand-alone self-attention for vision. In International Conference on Learning Representations, 2021.
Alvaro Sanchez-Gonzalez, Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In International Confer- on Machine Learning, pp. 8459 8468. PMLR, 2020.
Víctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E(n) equivariant graph neural net- works. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference on Machine Learning, volume 139 Proceedings of Machine Learning Research, pp- 9323-9332. PMLR, 18-24 Jul 2021.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEFE transactions neural networks,
2008

---

Under review as a conference paper at ICLR 2022
Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko. insights from deep tensor neural networks. Nature communications, 8(1):1-8,
2017.
Justin S Smith, Olexandr Isayev, and Adrian E Roitberg. Ani-l: extensible neural network potential with dft accuracy at force field computational cost. Chemical science, 8(4):3192-3203, 2017.
Jonathan M. Stokes, Kevin Yang, Kyle Swanson, Wengong Jin, Andres Cubillos-Ruiz. Nina M. Donghia, Craig R. MacNair, Shawn French, Lindsey A. Carfrae, Zohar Bloom-Ackermann, Vic- toria M. Tran, Anush Chiappino-Pepe. Ahmed H. Badran, Ian W. Andrews, Emma J. Chory, George M. Church, Eric D. Brown, Tommi S. Jaakkola, Regina Barzilay, and James J. Collins. A deep learning approach antibiotic discovery. Cell, 180(4))688-7022 e13, 2020.
Nathanicl Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and -nquiaiion-euiiaaiian neural networks for 3d point 2018.
clouds. arXiv preprint arXiv:/802.0080
Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay S. Pande. Moleculenet: A benchmark for molecular machine learning. CoRR, abs/1703.00564, 2017.
Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long. Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions on neural networks and learning systems, 2020.
Qin Yang, Chenglin Li, Wenrui Dai, Junni Zou, Guo-Jun Qi, and Hongkai Xiong. Rotation equiv- ariant graph convolutional network for spherical image classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
A
E(n) INVARIANCE OF ||x - x,112
To show that ||x, xyl12 is invariant under E(n), it is sufficient to that, under translation, one has
x x x =
2 E E"
while under rotation
x Qx,
Q On).
Now, by using the above relations and the fact that E(n) is the semidirect product of 7(n) and O(n), it is easy show that ||x; x,112 is invariant under F(n) since
|| x211 + | - x 11 = (Q(x: + 2) Q(x, + TQQ(x, + +2) - -Q(xx + 2))
(9a)
= (Qx: - Qx,) (Qx: - Qx,)
(9b)
(xi x,) TTQ(x= -x1)
(9c)
= (xi xx)" T(x; - -xi),
(9d)
where we used the fact that Q = IAQ E ((n).
B INVARIANCE UNDER THE CONFORMAL ORTHOGONAL GROUP
In this section we will derive the result from Section 2.1 in the main text, that dilations and orthogo- nal rotations are transformations under CO(R", Q). The group is defined for vector space V with a quadratic form Q. The group contains the linear transformations defined as
+ V. The group action is
2(Tx)=
(10)
12

---

Under review as a conference paper at ICLR 2022
where T is the set of linear transformations that need to define and is scalar. For purposes, we can consider a positive-defing quadratic form on R"; as we restrict our discussion to Euclidean space then the relevant quadratic form is
(11)
and so inserting equation 11 into equation 10 we have the condition
[(Tu)) Σ
we consider, as above, an orthogonal transformation Q, and in addition, a dilation 2; -> y24, then is simple show that, with T =QQ
= - Σ x
C RELATIVE DISTANCE AND ANGLE PRESERVING MAPS
Many different relative distance and/or angle preserving maps to used in the coordinate update can be obtained in different ways, eventually requiring further assumptions on the data.
The simplest one is the identity function
x = (i. 9x) = x;
(12)
which keeps the coordinates unchanged during the update step. In this case both relative distances and angles are trivially preserved. Updating all the coordinates in the same way also trivially pre- serves both relative distances and angles. Examples include
axis
E R
(13)
x X; a,
R
(14)
x Qx + a.
Q
E Q
(15)
where a, a, Q can be learnable parameters or parametric functions of other network parameters, e,g., o= (u). The matrix Q can also be non orthogonal, provided that the obtained transformation preserves distances or angles.
Devising more complex forms for v for arbitrary coordinate embeddings X, Y satisfying the relative distance or angle preserving property, is quite tricky and further assumptions are in general required. To see this, consider relative distance preserving transformation. Any transformation X -> Y. such that ||x; - 11 = lly; - 1 2 can be obtained as x; 7: A,y; qi where %i A, and qi. i V, are solutions the system
A,y q. - -3 9i1l = lly. - y1 (1, E
with E R, 1, E R" Vi E V. In this general case, defining (non trivial) map vi lly; y (i) E E is hard without any
such that, after its application, one has ||x -x 1|2 assumption on, at least, the topology of the graph Gx (or, equivalently Gy). A similar reasoning can be applied also to relative angle preserving maps.
we restrict ourselves case which " = 7, A; = A, q, = q. E V. this results X being Conformal orthogonal transformation of Y. In this case, possible map ve is defined by
+ Σ jEN.
ax(xx xx))
(16)
with azs possibly being a parametric function of node/edge/angle/glbbbll attributes, e.g., ajs o" (efe VJ v .u). Notice that, under conformal orthogonal transformations (hence also E(n) transformations) y: = (x) = Ax AAT T, has that equation 16 is equivariant map.
13

---

Under review as a conference paper at ICLR 2022
In fact.
y = 4 Σ
Eajlyy - v.)
JEN
= 7Ax; + Σ
ax((1 + - 7Ax, + b)
JEN;
aji(x; x3) +
-AA
X;
JEN.
yAx
Thus, when this map used 1/2 in the DGN AGN block, node coordinates be updated in an equivariant way with respect to conformal orthogonal transformations. Using the same argument, is easy show that equation 13 is also equivariant, but equation 14 is not.
Now we show that equation 16 is both relative distance and angle preserving, For the sake of notation, assume 7 1, however the arguments can be applied when 7 * 1.
Relative distance preservation equation 16
To show that equation [6 satisfies the definition
of distance distance-preserving preserving map it is sufficient to show that, since x; Ay; + one has
x = x;
Σ JEN
axj(x; - xi)
Ay, + +q+ +
Σ
air(Ax, +9 + - 9)
jEN:
Ay, q Σ
Oij( (Ax; -Ax,)
Oij(
JEN:
A
Y: + Σ
(X, -
xx)) +q
JEM,
Ay, +
(17)
which implies
|x - x 12 ||A q AAj q1l2
||A(y! -1,
(yy yt)TA AAAAYY ))
= =Ily² x1?
where in the last line we used the fact that A A=
While equation 17 implies that angles are preserved since is an E(n) transformation of y.. can show this explicitly by recalling that the angle between two vectors x, - x, and X x; can be computed as
Relative angle preservation of equation 16
(x, - "0x=xx) (x4 x,)
cosL(x,, X1,
||x; X; Xx X;
14

---

Under review as a conference paper at ICLR 2022
Then, one has
x
x;
( k
x;
(Ay] 9 Ay; 9) (Ay +q- Ay, q)
|x]
x
x
x
] q Ay; 9 Ay + +q Ay -ql
[AyJ AyT) A ]AAYT - -Ay))
| AyT Ayy - A
(y)) -yt)TA A A(y - yy))
Vet y! AT AAy) xt) Vott -y,) -Y TA Aly y!)
(y) y.) (yyt - -yt)
[y - y 'llyt -yell
where in the last line we used the fact that A AT=11
Local symmetry transformation of equation 16 Suppose we have a local symmetry transforma-
tion, A(x) which only acts on a subgraph 9 E g. such that v = (V1, V2... Va. Vn+1,... V.n). and similarly for the edge, coordinate and angle features. The action of A(x) is then
A(X)x = (A(*) A(x)x,,.... A())&m, A(x)xx+1,.... A(x)xm)
(A(x))) ( ... A(X)xn, Xn+1..... Xm)
By defining the 2 subgraphs containing the coordinate features which are and are not affected by the symmetry transformation, we can therefore write a coordinate update equation [6] for both subgraphs; Eq. equation 16 for the x; and, for the X;.
Σ axlli, x2). jEM
+
We have shown above that this coordinate update preserves distances and angles for global group transformations; by defining the 2 subgraphs as above, we can promote the local symmetry trans- formation A(x) to global transformations 2 subgraphs, one can generalise the argument to any number of local transformations so long as the subgraphs are defined as above. As we have subdivide the graph into subgraphs. we note that these local transformations cannot be defined arbitrarily as there must be a sense of neigh- bourhood of nodes within each subgraph. A local transformation which affects unrelated nodes identically (which is valid class of local symmetry) is not valid for this reason.
subgraphs. Whilst here have shown this for only
D EUCLIDEAN GROUP EQUIVARIANCE OF THE DGN BLOCK
To show equivariance to F(n) transformations of the input coordinates for the DGN block, we begin with the edge update in equation 5a Since ||x; x, 1/ is invariant under F(n)) transformation, x + Qx + z, for some rotation matrix Q E O(n) and translation vector z E IR", then
S (eno Vs, VJ, ||x x, 112, u) - Φe (oja Vi, v,, Qx: + z Qx z u)
= 00 (ejo, V4, Vj, |x: - x, 112.u).
Invariance of the node and global updates in equation 5 follows naturally as they are composed of invariant quantities. The coordinate update in equation 5c can be invariant equivariant under E(n) depending on the structure of sha
CONFORMAL. GROUP INVARIANCE OF THE AGN BLOCK
As discussed in Section 2.I. the conformal group consists of transformations that preserve angles between all possible triples of coordinates (see equation []. To show that the AGN block is invariant
15

---

Under review as a conference paper at ICLR 2022
to conformal transformations it is sufficient note that given a conformal transformation, (p, one has that the angle update equation 6a is invariant to it since
(vi, Vjs Vl: jik, ((x,, x xs), u) 7 (vi, V,, Vas aii (x)()),, (x(x),,, (p(x)) )), u) (v,,, Vj, VR. jikk [(x,, x,, X4), u)
Invariance of the other updates is trivially satisfied by construction. The coordinate updates can also be constructed to equivariant to conformal orthogonal (and hence Euclidean) transformations, with an appropriate choice of v0ah We stress that equivariance to the conformal group includes by definition equivariance to the Euclidean group as we are only concerned with transformations on R". The orthogonal rotations and translations of the Euclidean group therefore a subset of possible conformal transformations on IR" as they are angle-conserving transformations. Angular information be extremely powerful in tasks where classical (or distance-basede GNNs fail, like in graph isomorphism tests where, e.g., a hexagon not distinguished from two triangles.
F ADDITIONAL. FORMULATIONS
F.1 ALTERNATIVE FORMULATIONS FOR THE ANGLE PRESERVING GRAPH NETWORK
number of variations can be proposed for the angle preserving graph network:
Edge attributes can be used in angle updates
a Jj "0 (Vis Vj, Vk: eij. eik: ajiks 4(x,, Xb), u),
V(,i,k) E..
Relative distances can be used in the angle updates
Ox (-..,, || xy112: ||x xx (x X Xk),
v(jii,)) E..
Angle attributes be used in edge updates
e/ の² (eij. Vs, Vy.p" "" (fa
pm (1a
v(i.j) E
where A.j = { (y.2 E A, = i, = j} the set of angles whose first ray is defined by (i.j).
Angle embeddings can be ignored and node attributes can be updated with the angles them selves
oe = (Vy,
V(,1) E
x" p (fotthie)) (1<(x), xs, xx) u))
VIEV
x (i, u+ = (e
ViEV
F.2 COMBINED ARCHITECTURE
The DGN and AGN architectures can be combined in a single architecture, An example
(V,,,, Vj, Vk, jiik,, Xi, u), of oz (V v., Cjn |I: 1x,-xx, 1,, u),
v(j A
v(J.) EE 3
v x" (vi. (fet)))eN)), p'-
"Ha (Hatt
Vi V
x vz (i, n =
Vie
( (unee)))
where the global update can contain aggregated information about angles and distances and also the other updates can be generalised shown above. This architecture is by construction equivariant to transformations the coordinate embeddings for which both relative distances and angles are preserved.
16

---

Under review as a conference paper at ICLR 2022
G OBTAINING OTHER ARCHITECTURES AS SPECIAL INSTANCES
In this appendix we will represent some of the architectures discussed in the main text explicitly as instances of our architecture.
G.1 DIMENET KLICPERA ET AL. 2020)
Dimenet be obtained from the AGN by considering edge and distance information in the angle update and using sum aggregation functions as
(ejej | (x, x,, xx)). v(Ji,,) E A
jk
ej d' Σ aggh);
v(ii) EE
iEN;
x Xi:
ViE
v. o" (Vi.
Σ
VieV
JEN;
where ||x; x5l1 is defined represented within a set of orthogonal basis functions ensr and the angles (xx,, xj. Xx) within basis defined as aSBF.
G.2
EGNN (SATORRAS ET AL. 2021
The EGNN network is obtained from the DGN by selecting specific form for the coordinate update function via using the sum aggregation function as de and not propagating updated edges, i.c.
e = o² (vj, Vi,
in jis |x; x11 Vú,i)
x = x; + Σ (x - (0)) ViEV,
VJ = o" (v:
Σ JEN,
Viev,
where of are the edge attributes the input data (implying that eft not propagated to any subse- quent layer, in message passing networks).
G.3 ISOGNN (HORIE ET AL.. 2021
The IsoGNN architecture defined for tensors of rank-n; to compare with the other architectures presented here, we show below the architecture for rank-1 tensors.
Σ
TyjRi(Xe - xx)
>20) ) V,, V(
AEEVAA
x
Xi:
VEE
vt d² (v:.
ViEV
where Tijai is untrainable 2-dimensional matrix which is translation and rotation invariant, and determined offline from the data for each class problem.
G.4 OTHER NETWORKS
Standard graph networks be obtained from ours by skipping some updates not considering
equivariant information. Also other variants, including SchNet (Schütt et al. (017) or TFN (Thomas et al. 2018), be cast message passing architectures shown in (Saoorras al. 2021
17

---

Under review as a conference paper at ICLR 2022
H ADDITIONAL EXPERIMENTS AND IMPLEMENTATION DETAILS
H.1 COMMON IMPLEMENTATION DETAILS
The update functions of the networks all implemented as MLPs. After the graph layers, the produced node embeddings are passed through another MLP, a global pooling layer and a final MLP with output dimension equal to the number of classes for graph classification tasks. For edge classification tasks (TSP), the architecture after the graph layers is single MLP taking as input source and target nodes and predicting the class of each edge. Each network is trained starting from 10 different initial conditions. The results in the tables contains the mean and standard deviation resulting from the 10 initialisations.
H.2 POLYTOPES CLASSIFICATION
H.2.1 SPECIFIC IMPLEMENTATION DETAILS
All the MLPs have one hidden layer containing 64 neurons and swish activation function. We used graph layers for AGN and DimeNet and 3 for the DGN, GN and EGNN with embeddings in the hidden layers having dimension 32. Aggregation functions and the pooling layer implements mean or sum operations (and are specified in the results' tables) for our architectures. As for DimeNet, we use distances and angles directly instead of their RBF embeddings since we do not consider atomistic quantities. Adam (Kingma & Ba 2015) used train the all the models with learning rate 0.001 and regularisation for 1000 epochs. The batch-size is equal to the number of training samples (so 5, 6, 3 respectively, for n = 3, 5)..
H.2.2 ADDITIONAL EXPERIMENTS
the main paper we reported results when the identity function equation 12 was used to perform the coordinate update together with sum aggregation function. In Tables 4 and 6 we report also the results obtained when using equation 16 for the coordinate updates and possibly mean aggregation function. As can be seen, using the mean aggregation function usually causes drop in performance. In particular, the SDGN with mean aggregation function is unable to correctly classify the polytopes training time. This is because, after rescaling the coordinates, all edges have the same length and, employing a mean aggregation results in always the same output for all polytopes. This is partially alleviated by using equation 16 as the node coordinate update function, which allows one to differently remap coordinates of different polytopes. The other results are qualitatively similar.
lest
Orthogonal
Orthogonal dilation
(f 0.5)
1.6)
[jr 3.01
192281
0.98 + 0.04
ACN
30420
16
1
1
1
0.97 + 0.03
AGN
1
SDGN
10
0.2
0.2
0.2
0.2
0.2
0.2
SDGN
11
0.8
0.8
0.8
0.65 1: 0.05
0.40 + 0.08
0.36 + 0.06
SDGN
0.93 + 0.07
0.83 1 0.15
DXN
0.83
0.22 0.02
0.22 0.03
0.20
0.22 1 0.04
DGN
T0022
16
0.29 2 0.02
0.25 1: 0.02
0.02 0.24
0.20 1 0.03
DGN
0.45 0.05
0.14 0.12
0.41 0.07
0.37 1 0.04
GN
0.25
0.04
0.20 0.05
0.21 0.03
0.20
0.02
0.22 0.04
GN
5am
0.44
0.15
0.46 0.14
0.47
0.15
0.44 4 0.17
0.43 1.34100 0.15
Table 4: Polytopes classification: training and test accuracy (mean + standard deviation over 10 runs) for n 3, for different transformations the test set.
Adding not all-identical node features The experiments run so far used datasets containing only information about node coordinates and the presence edges. If we add (not-all-identical) node features, then also SDGN with mean aggregation function is able correctly classify all the poly- topes. This happens thanks to the additional features breaking a symmetry making the graphs the simplex and the orthoplex look identical to the SDGN layer.
18

---

Under review as a conference paper at ICLR 2022
Iest accumey
bloc
Orthogonal
Orthogonal dilation
Non-orlhayeaal 0.5)
Non-orilegenal
Nen-onlaguna
3.0)
0.98 + 0.04
AGN
0.97 = 0.04
AGN
900h
12
AGN
40000
0.17
0.57 0.13
SDGN
50100
0.96 + 0.03
0.83 2 0.13
0.77 + 0.17
SDGN
0.92 0.08
0.78
0.14
0.65 4 0.20
DGN
16
0.34 0.02
0.40 0.04
0.38
0.03
0.30 1 0.03
DON
10V
0.61 = 0.05
0.60 0.08
0.61
0.06
0.59 + 0.08
DXN
0.60 0.04
0.61 0.03
0.57 2 0.08
0.55 + 0.07
ON
W0m
0.53 0.04
0.03
0.48 0.12
0.49
+ 0.04
Table 5: Polytopes classification: training and test accuracy (mean + standard deviation over 10 runs) for 4, for different transformations in the set.
accumeyy Non-crllayeen
Nam-orihogomal (a = 1.5)
Non-onhoggma (u 3.01
blok
train ace
Orthogonal
Orthogonal dilation
(gi = 0.5)
AGN
20002
10
0.99 + 0.03
0.86 1 0.11 0.84 1 0.11
0.77 0.13
AGN
0.99 0.03
0.75 1 0.11
AGN
100h
16
1
SDGN
10
0:33
0.33
0.33
0.33
0.33
0.33
SDGN
0.60 + 0.25
0.48
0.15
0.37 + 0.
SDGN
100m
16
0.16 0.83
0.75
0.14
1 0.18
DGN
20222
1
0.49 0.01
0.00 0.33
0.49
+ 0.02
0.37 + 0.01
DGN
0.51 0.03
0.40 0.04
0.41
0.06
0.39 1 0.05
DGN
10
0.37 0.10
0.45 0.12
0.54
0.13
0.49
0.10
GN
0.38 +:
0.05
0.42
0 0.06
0.34 0.04
0.38
0.08
0.43 0.06
ON
0.64 + 0.07
0.55 = 0.04
0.60
0.13
0.61 0.16
0.58 4 0.08
Table 6: Polytopes classification: training and test accuracy (mean + standard deviation 10 runs) for 5, for different transformations in the test set.
Data efficiency To emphasise the advantage of having a network that is able to exploit symmetries in the dataset in terms of data efficiency, we study how many samples in the training set are necessary for a standard GNN to reach reasonable generalisation performance. For the set of transformations we considered the previous sections, we augmented the training set with {2, ... 100} randomly transformed (as in the respective test set) copies of each polytope. We trained standard GNN on these augmented datasets and observed the resulting test accuracy after 1000 epochs. Results are reported in Figure for each set of transformations in terms of mean and standard deviation over 10 random initialisations. can be seen that samples per polytope may be sufficient when only or- thogonal transformations
considered. Adding also dilations and non-orthogonal transformations further increases the number of data points that are required. This shows that while data augmen- tation can be successfully exploited, it is provably sub-optimal in terms of sample complexity (Mei) et al. 2021) and architectures with built-in equivariance properties represent more efficient strategy consider.
Orthoguna
Otteopaai
Non-orthogoaal (o 0.5)
0.8
Son-orihogooal (p 1.5)
Non oothorlhooooal
0.2
10
30
50 60 70 90 00000
Samplex polytoope texining
Figure 4: Test accuracy vs samples per polytope in the training set for a standard GNN (n = 3).
H.3 MNIST, CIFAR10, TSP
MNIST and CIFARIO are classical image classification datasets converted into graphs using super- pixels and assigning the super-pixel coordinates node coordinates and the intensity node fea-
19

---

Under review as a conference paper at ICLR 2022
tures. TSP (based the Travelling Salesman Problem) tests link prediction 2D Euclidean graphs to identify edges belonging to the optimal TSP solution given by the Concorde solver.
For this datasets we use the same splits as specified in (Dwivedi al. 2020
H.3.1 SPECIFIC IMPLEMENTATION DETAILS
The MLPs have one hidden layer containing 64 neurons, swish activation function and dropout layers with dropout rate 0.01. We used 2 graph layers for AGN and 3 for both the DGN and GN with embeddings in the hidden layers having dimension 64. Aggregation functions and the pooling layer implements sum operations. The models are trained with Adam with learning rate = 0.001 and no regularisation for 100 epochs. Batch-size 128 is used for MNIST and CIFARTO, and 8 for TSP.
H.4 MOLECULAR PROPERTY PREDICTION QM9
The QM9 dataset (Wu et al. 2017) is comprised of small molecules (hydrogen, carbon, nitrogen, oxygen, flourine) with the target properties being 12 chemical properties, As the target properties are equivariant to Euclidean transformations of the atoms' coordinates, and also to the order in which atoms are processed, QM9 is an excellent benchmarking dataset for a GNN, especially if E(n) invariant. Indeed, state of the art results have been achieved on this dataset by EGNN, Dimenet, SE3-Transformer (Satorras et al., [021]] Klicpera et al., 2020): Fuchs et al.) 2020). Here we show that, in addition to leading better accuracy, employing equivariant networks give significant improvement in convergence speed.
Figure 5 show the evolution of the mean squared error on the both architectures. We see that both the AGN and DGN outperform the standard GN in two aspects; the model trains more rapidly, and also reaches lower loss.
all the target properties for
CHOMA
$12/M0
2/1
1.54
1:48
(R7)
ZPVE
100
20
AGN
DGN
GN
Figure 5: QM9: test MSE loss the 12 target properties.
H.4.1 SPECIFIC IMPLEMENTATION DETAILS
The QM9 dataset is composed of roughly 134k molecules: we used 100k for training and the re- maining for testing. The AGN and DGN networks receive the embedding of the atomic properties as initial node features and the coordinates of each atom coordinate features, while in the standard GNN they are stacked and provided as node features. Edge embeddings representing bond types are also provided.
The networks consist of graph layers (3 for the AGN) with each MLP having hidden layers of 128 nodes, swish activation function and dropout rate of 0.01. The node embeddings of the last graph layer are passed through an MLP, a global mean pooling layer and a last MLP to map them to the target chemical property. The last two MI.Ps each have single hidden layer of 128 nodes, swish
20

---

Under review as a conference paper at ICLR 2022
activation and dropout rate of 0.01. The target chemical properties are all standardised by subtracting the mean and dividing by the standard deviation for each target. The networks are trained for 1000 epochs using ADAM with learning rate of 0.0005 and mean squared error loss.
21


Reviews:
[{"review_id": "UopJEuuP_pi", "reviewer": "Reviewer_CQdG", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n  + The attempt to develop invariance and equivariant graph neural\n  networks is valuable. \n\nConcerns:\n\n  - The proposed approach makes very strong assumptions, such as: 1)  node\n    coordinates (in an ambient Euclidean space) are given; 2) the\n    other noe features are  coordinate-independent.  This greatly\n    limits the applicability of the proposed approach.\n\n  -  With the above assumptions, in essence the authors explicitly \"hand-code\"\n     specific \"invariance\" and \"equivariance\" properties, namely,    rotation/translation invariance and angle-preserving\n     transformations as captured by Eucliean and conformal groups, in AGN. \n\n - The evaluation is done only on very limited graph datasets, namely\n   the Dwivedi et al., 2020 benchmark datasets where graphs are\n   generated from, e.g., 2D Euclidean datasets such as images, where\n   node coordinates are well defined. \n   \n\nOther comments:\n\n   - What are \"global\" attributes $\\vec{u}$ defined on? I presume that\n     they are not defined on per-node or per-edge basis. This is never\n     explained.\n\n   - Even under the assumption that node coordinates are available as\n     part of the node features, you are basically assuming that the\n     Euclidean distance is the right \"distance\" metric for the\n     underlying graphs. This may not hold in general at all. For\n     example, the nodes may lie in a sphere in an ambient 3-D\n     Euclidean space, and the \"right\" distance between the nodes should be the geodesic distance on the sphere, not the ambient Euclidean distance.\n\n - In your model eqs.(5a) - (5b), you are making a lot of implicit\n   assumptions. For example, in using 5a, you are assuming that edge\n   features are independent of node coordinates, but depend only on \"distances\"\n   between them. The same comments also apply to other equations, such\n   as the transformation of the node coordinates will be based only\n   node coordinates, but not other node features. \n   \n - Without any \"formal\" mathematical context on how graph data are\n   generated or sampled from, why would   rotation/translation/angle-preserving transformations would be the\n   right \"symmetry\" in the data? In particular, why would preserving \"angle\" between neighboring (three) nodes actually capture?\n\n - How does your model apply to more  general    graph datasets, e.g., molecular biological networks or social\n    networks such as co-authorship networks, where node coordinates\n    are not available, and the whole goal of graph representation via\n    node embedding is to find  \"appropriate\" node coordinates in a\n    proper embedding space.\n\nMinor nitpicks:\n   - in your definition of equivarance, you need to first fix the group action  on Y,  $\\phi'_g$, otherwise the definition does not make sense.\n\n  - I am not sure what value the materials in Sections A-E in the appendix really add. You are basically restating the translation-invariance and angle preserving properties of Euclidean groups and conformal groups, and sections D and E hold because you explicitly construct AGA to be distance-invariant and angle preserving.", "summary_of_the_paper": "The paper seeks to develop invariance and equivariant graph neural networks. \nBased on the assumption that node feautures can be separated into a\nnode coordinate feature vector and coordinate-independent node feature\nvector, the authors develop a GNN architecture, called \"angle\npreserving graph network\" (AGN), that is  equivariant to  the\nEuclidean group and conformal group. The method is evaluated using the\n\"benchmark\" graph datasets produced in (Dwivedi et al., 2020) from,\ne.g., MNIST and CIFAR10 datasets.", "summary_of_the_review": "The paper seeks to develop invariance and equivariant graph neural\n networks. The proposed approach makes strong assumptions. As such, it\n has very limited applicability. The evaluation is also very limited.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "JoLkF-NBwLm", "reviewer": "Reviewer_W7xU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes DGN and AGN that can update graph representation equivariantly w.r.t. any transformation preserving distance and angles, respectively. The authors conduct experiments on synthetic data to demonstrate the effectiveness of considering symmetries.\n\n#####Pros#####\n\n1. It is intuitively sound and necessary to consider the symmetries of data. The motivation of this work is clearly explained and convincing.\n\n\n2. The proposed DGN and AGN, shown in Figure 2 (b) and (c), are technically sound and general enough. They are developed based on the general framework GN (Figure 2 (a)). Most existing methods, such as SchNet and DimeNet included in Section 5, can be viewed as special cases of the proposed frameworks. Hence, the proposed DGN and AGN are unified well.\n\n\n3. The experiments on synthetic data are well-designed, which can show the effectiveness of exploiting symmetries. This can support the methodology strongly.\n\n\n#####Cons#####\n\n1. The main concern is that there are no empirical results to demonstrate that DGN or AGN is effective on real-world tasks, instead of specifically designed synthetic tasks. It is strongly desired that the proposed method can be evaluated on real datasets.\n", "summary_of_the_paper": "This paper proposes two graph networks that are equivariant to distance and angle preserving transformations in graph coordinates. It moves one step forward based on GN (Graph Network) by decoupling node coordinates from other node attributes. The experiments on synthetic datasets shows the capabilities of the proposed symmetry-driven graph networks.", "summary_of_the_review": "Overall, I think the current version of this paper is slightly above the acceptance threshold since its motivation and proposed methods are technically sound and convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GtWFT9e2VuH", "reviewer": "Reviewer_yqP4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Initial Recommendation: Marginal Accept\n\nI have specified the reasons for my recommendation below:\n\nStrengths:\n1. The paper is clearly structured and well written, and I found it easy to read.\n2. Analogous to E(N) GNN - the model is simple and computationally efficient (along with quantitative results which show computational complexity) - without the use of special functions like Bessel functions, etc to capture equivariance to the desired transformation actions.\n3. Strong experimental results on the synthetic polytope classification datasets when the symmetries are present - and also provide the reader examples of when the model should not be used.\n \nWeakness:\n1. The authors discuss the ability to capture molecular conformers as one of the main selling points of the network being equivariant to the conformal group but do not show results on the QM9/ other macromolecular datasets in comparison to E(N) GNN, but without using a fully connected graph (Apart from rate of convergence in the appendix). For instance, in molecules, only when covalent/ hydrogen bonds are present does the distance between atoms need to be preserved, right?.\n2. The authors do not present other use cases (or motivations of this work) of the symmetries to the conformal group or distance preserving transformations of an object - for example results on point cloud datasets (where the objects are not rigid bodies) would do well for this work.\n\n\nMinor:\n1. Introduce $a_{ji}$ earlier rather than in page 4 as it is used in eq 3 (in pg 3)", "summary_of_the_paper": "The authors propose distance preserving Graph Network and Angle Preserving Graph Networks which are equivariant to node permutation as well distance preserving transformations/ angle preserving transformations of the coordinates associated with the nodes. The authors generalize the E(N) GNN work to include group symmetries to the conformal group and to the case when not all nodes in the graph are connected. Experiments on the  polytopes classification dataset show that the model is effective when symmetries are present in the data, and ineffective when they are not (MNIST, CIFAR and TSP).", "summary_of_the_review": "The paper is well written and the proposed solution is simple and computationally efficient. The claims are sound and the empirical evaluation is partly adequate but partly needs work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "02gwI7FTyz", "reviewer": "Reviewer_ECjd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1.\tThis paper is well written. I enjoy the reading. The authors have introduced the necessary conceptions in a comfortable way. It is easy to follow the whole idea and the methodology.\n\n2.\tIt is good that the proposed DGN and AGN can generalize the current methods including EGNN and DIMENET, by implementing different choices of certain components. The validity of preserving the distance or angles is clearly justified (mainly in appendix).\n\n3.\tThe experiments on polytopes classification, though seems over-simplified, are able to demonstrate the superiority of the proposed AGN over other methods (GN, SE3-Trans, DimeNET, EGNN) particularly when non-orthogonal symmetry is considered. \n\nWeaknesses:\n\n1.\tThe novelty is somehow limited. It is indeed that Equations (5-6) have extended the design of GN (Eq. (4)) by involving equivariance w.r.t. node coordinates and invariance w.r.t. other embedding features. But the main step contributed to this property lies in the update of the node coordinates, namely, Eq. (5c) and Eq. (6d), which is previously developed by EGNN (Satorras et al., 2021) and other related works. This can be checked by comparing Eq. (3) used in this paper with Eq. (4) in the EGNN paper. Even the authors have additionally proved that Eq. (3) is equivariant to Conformal orthogonal transformations which is not discussed in EGNN, such equivariance is trivial to derive. Specifically, for DGN, it is very similar to EGNN, unless that DGN is built upon GN (with global vector u) while EGNN is upon MPNN.\n\n2.\tThe proposed AGN and DGN show merits only on the simulated dataset. For the real datasets (MNIST, CIFAR10, and TSP), the performance of AGN and DGN are even worse than GN. The authors attribute this detriment to the lack of symmetry in the data. This seems somewhat questionable. Since the images for example in MNIST are actually rotation/translation equivariant, DGN is supposed to perform better than GN, given that DGN is a rotation/translation equivariant version of GN by the design in Eq.(5). Previous studies such as group CNN have also demonstrated further taking the rotation equivariance into account delivers desirable enhancement. The authors are suggested to provide more explanations on this issue. \n\n3.\tIn terms of evaluations on polytopes classification, all methods achieve accuracy 1 for training. Does this mean that the simulated dataset is too simple? What is the number of the training/testing samples? Is it too small?\n\n4.\tThere remain several confusing presentations in the current version:\n\n4.1\tIn the first paragraph, the authors state that graph neural networks, are invariant to the symmetry groups. The “symmetry group” here is too general and includes other cases besides the permutation group. \n\n4.2\tIt seems AGN is only equivariant with regard to the Conformal orthogonal group (a subgroup of Conformal transformations) according to the derivations in Appendix C. So why has the claim that AGN is equivariant to the Conformal group?\n\n4.3\tThe authors are suggested to provide formal definitions of these groups: E(n), CO(Rn;Q) and Conf(Rn;0) groups. \n", "summary_of_the_paper": "This paper proposes two kinds of equivariant GNNs, including the distance preserving graph network (DGN) that is equivariant to the Euclidean transformations and angle preserving graph network (AGN) that is equivariant to the Conformal group. The proposed frameworks generalize several typical previous architectures. The advantage is experimentally evaluated on a synthetic dataset composed of n-dimensional geometric objects. ", "summary_of_the_review": "Overall, there are certain merits for the proposed architectures, but both the originality and the experimental significance are insufficient. I initially suggest weak rejection. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to extend the invariance/equivariance properties of GNNs by focusing on distance-preserving and angle-preserving transformations, given respectively by the Euclidean and Conformal group. Preliminary experiments are reported that demonstrate the advantage of such architectures. 
Reviewers found this work generally interesting, tackling an important problem and proposing a valid solution. However, they also raised important concerns, namely the relatively minor novelty relative to recent models (such as EGNN), as well as the lack of convincing real-world experiments that would validate the modeling assumptions. Taking all these considerations into account, the AC recommends rejection at this time, and encourages the authors to address the points raised by reviewers in a revision.
Decision: Reject